Observability, Monitoring & Alerting

Observability is the ability to infer the internal health and execution state of complex distributed systems based solely on external telemetry outputs (Logs, Metrics, and Traces).

1. The Three Pillars of Observability

Pillar	Definition & Character	Popular Tooling
Logs	Immutable, timestamped textual records of discrete system events.	ElasticSearch/Logstash/Kibana (ELK), Grafana Loki, Splunk
Metrics	Numeric measurements aggregated over time intervals (counters, gauges, histograms).	Prometheus, Datadog, AWS CloudWatch, StatsD
Distributed Tracing	End-to-end request lifecycle graph following requests across microservice boundaries.	OpenTelemetry (OTel), Jaeger, Zipkin

2. Structured Logging Best Practices

Always emit structured JSON logs rather than raw unformatted strings to enable fast querying and indexing:

{
  "timestamp": "2026-08-16T09:42:00.124Z",
  "level": "ERROR",
  "service": "payment-orchestrator",
  "trace_id": "9f82b71a-e83c",
  "span_id": "4a12c8e",
  "user_id": "usr_99812",
  "event": "payment_charge_failed",
  "error_code": "INSUFFICIENT_FUNDS",
  "amount": 149.99,
  "currency": "USD",
  "duration_ms": 342
}

3. The Four Golden Signals (Google SRE)

Signal	Definition	Threshold / Alert Rule Example
Latency	Time taken to serve a request. Focus on p95 / p99 rather than averages.	Alert if `p99 latency > 800ms` for > 3 minutes.
Traffic	Total operational load or demand on the system (e.g., HTTP req/s, I/O rate).	Alert if `RPS drops by > 40%` unexpectedly.
Errors	Rate of requests that fail explicitly (HTTP 5xx, timeouts, exceptions).	Alert if `5xx error rate > 1%` of total traffic.
Saturation	Fraction of the most constrained resource currently utilized (CPU, memory, DB pool).	Alert if `DB connection pool > 85%` full.

4. Distributed Tracing Flow

flowchart TD U["👤 User Request\n(Trace ID: trace-abc-77)"] --> GW["🚪 API Gateway\nTotal: 220ms"] GW --> AUTH["🔐 Auth Service\n15ms"] GW --> UPS["👤 User Profile Service\n45ms"] UPS --> REDIS["⚡ Redis Cache\n2ms (HIT)"] GW --> ORD["📦 Order Service\n140ms"] ORD --> INV["🗄️ Inventory DB Query\n25ms"] ORD --> PAY["⚠️ Payment Gateway HTTP\n105ms (Bottleneck)"] GW --> KAFKA["📨 Kafka Producer\n5ms (Async)"]

5. Modern Telemetry Architecture