Observability, The Four Golden Signals & SRE Graduation Exam

පිටුව 20: Observability, The Four Golden Signals, Toil 50% Rule සහ අවසන් විභාගය

Telemetry මූලිකාංග (Metrics, Logs, Traces), Google හි Four Golden Signals Dashboard, Toil අඩු කිරීමේ 50% නීතිය සහ SRE අවසන් විභාගය!

21 Sept 20264 min readSelf Guide · Stage 12#Observability#Golden signals#SRE Labs

Lab 20 of 20 · Observability & Graduation · 5 hands-on activities · bilingual (English + සිංහල)

පිටුව 20: Observability, The Four Golden Signals, Toil 50% Rule සහ අවසන් විභාගය

💡 සරල සිංහලෙන්: Four Golden Signals (මූලික සංඥා 4) සහ Toil 50% නීතිය

ICU එකක ලෙඩෙකුගේ හෘද ස්පන්දන මොනිටරය වගේ (The 4 Golden Signals):

1. Latency (ප්‍රමාදය): වෙබ් සයිට් එකේ Page එකක් Load වෙන්න තත්පර කීයක් යනවද?
2. Traffic (සෙනඟ): එක තත්පරයකට Users ලා කී දෙනෙක් සයිට් එකට එනවද (Requests Per Second)?
3. Errors (දෝෂ): 100 දෙනෙක් එනකොට කී දෙනෙකුට Error පේනවද? (0.1% ට වඩා අඩු වෙන්න ඕන).
4. Saturation (පිරීම): Server එකේ CPU සහ RAM එකෙන් කොච්චර පිරිලාද? (80% පැන්නොත් අනතුරක්!).

Google එකේ 50% නීතිය:
හැමදාම එකම දේ අතින් කරන කරදරකාරී වැඩ (Toil) වලට ඔබේ කාලයෙන් 50%කට වඩා යොදවන්න තහනම්! ඉතිරි 50% අනිවාර්යයෙන්ම ඒ වැඩේ ස්වයංක්‍රීයව වෙන Software සහ Scripts ලියන්න ඕන.

Architectural Foundation & Systems Diagram

Google SRE defines the Four Golden Signals as the essential production telemetry vitals of any service-oriented architecture:

Live Telemetry Gauges Dashboard

🛡️ Google's 50% Toil Rule — Operational work (tickets, manual restarts) must NEVER exceed 50% of an SRE's time. The remaining ≥50% MUST be spent writing software and automation.
📝 Blameless Postmortems — When outages happen, you do not blame humans. You document timeline, root cause, impact, and action items so the system itself prevents that failure mode forever.
🎓 SRE Graduation — You have mastered: Linux hardware, syscalls, bash, inodes, permissions, processes, memory, systemd, TCP networking, SSH, AWS, DevOps culture, Git, Docker internals, Kubernetes clusters, Jenkins CI/CD, and SRE math!

Step-by-step practical lab

Step 01: Query Prometheus Endpoint for Service Metrics

සිංහලෙන් — Prometheus Metrics Endpoint එකෙන් Live Requests ප්‍රමාණය කියවමු.

Scrape metrics endpoint to read live request counters for Latency, Traffic, Errors, and Saturation.

ExampleShell

curl -s http://localhost:8080/metrics | grep http_requests_total

/metrics : Prometheus scraper එකට දත්ත ලබා දෙන standard OpenMetrics endpoint එකයි.

Expected output

Output

# HELP http_requests_total Total number of HTTP requests processed
# TYPE http_requests_total counter
http_requests_total{code="200",handler="/api"} 84201
http_requests_total{code="500",handler="/api"} 14

SRE Tip: The Four Golden Signals are Latency (time to respond), Traffic (demand), Errors (rate of failed requests), and Saturation (system fullness).

Hands-on practice activities

Work through these yourself — tick them off as you go.

Activity 96: Probe Prometheus Application Metrics Endpoint (Drill) — Inspect Prometheus exposition format metrics (counters, gauges, histograms).
Activity 97: Four Golden Signals Audit (Checkpoint) — Memorize the 4 Signals: Latency, Traffic, Errors, Saturation.
Activity 98: Google 50% Rule Toil Cap (Checkpoint) — Ensure that at least 20 hours of a 40-hour work week are reserved for software engineering projects.
Activity 99: Blameless Postmortem Structure (Checkpoint) — Every incident report requires: Executive Summary, Timeline, Root Cause, Trigger, Impact, and Preventative Action Items.
Activity 100: Final Comprehensive SRE Masterclass Certification Exam (Quiz) — Score 100% on the final assessment to unlock the Elite SRE Architect badge!