Reliability & Fault Tolerance

Designing reliable systems means ensuring continuous correct operation even when individual hardware nodes, network links, or external third-party APIs fail.

1. High Availability SLAs ("The Nines")

Availability is measured as the percentage of total uptime over a given period:

$\text{Availability} = \frac{\text{Uptime}}{\text{Uptime} + \text{Downtime}} \times 100\%$

Availability SLA	Downtime per Year	Downtime per Month	Downtime per Day
99% (Two Nines)	3.65 days	7.31 hours	14.4 minutes
99.9% (Three Nines)	8.77 hours	43.8 minutes	1.44 minutes
99.99% (Four Nines)	52.6 minutes	4.38 minutes	8.64 seconds
99.999% (Five Nines)	5.26 minutes	26.3 seconds	864 milliseconds

2. Redundancy & Failover Architectures

Active-Passive Failover:
┌──────────────┐     Heartbeat Ping      ┌──────────────┐
│ Active Node  │ <.....................> │ Standby Node │ (Takes over if active fails)
└──────────────┘                         └──────────────┘

Active-Active Failover:
┌──────────────┐                         ┌──────────────┐
│ Active Node 1│ <═════ Shared Load ════>│ Active Node 2│ (Both serve traffic concurrently)
└──────────────┘                         └──────────────┘

3. Retry Strategies & Exponential Backoff with Jitter

When calling remote services over a network, immediate retries can cause a thundering herd and crash an already struggling downstream server.

Exponential Backoff with Full Jitter:

$\text{Sleep} = \text{random}(0, \min(\text{MaxSleep}, \text{Base} \times 2^{\text{Attempt}}))$

Attempt 1: wait random(0, 1.0s)
Attempt 2: wait random(0, 2.0s)
Attempt 3: wait random(0, 4.0s)
Attempt 4: wait random(0, 8.0s)
Attempt 5: Max retries reached ──> Fallback / Raise Alert

4. Rate Limiting Algorithms

Rate limiting protects APIs against abusive scrapers, credential stuffing, and DDoS attacks.

Rate Limiter Algorithm	How It Works	Strengths	Trade-offs
Token Bucket	Tokens added to bucket at constant rate; each request consumes 1 token.	Handles temporary traffic bursts gracefully.	Memory overhead for tracking tokens per user.
Leaky Bucket	Requests enter a queue and leak out at a constant, smooth rate.	Guarantees perfectly constant egress rate.	Bursty traffic is delayed in the queue.
Fixed Window Counter	Counts total requests within fixed discrete windows (e.g., per minute).	Extremely low memory footprint.	Traffic spikes at window boundaries can allow double limit.
Sliding Window Log	Records timestamps of all requests in a sorted set (Redis ZSET).	100% accurate enforcement.	High memory consumption for high-volume endpoints.

5. Graceful Degradation & Fallbacks

When non-essential microservices fail, the overall system should degrade non-critical functionality rather than throwing full 500 error pages:

Normal Operation:
- Personalized AI Recommendations
- Real-time Stock Ticker
- Live Chat Support

Degraded Mode (AI Service Outage):
- Static "Popular Today" Fallback List
- Cached Stock Data with "Delayed" Banner
- Asynchronous Support Email Form