Distributed Systems & Consensus

A distributed system is a collection of autonomous computing entities (nodes) communicating over a network that coordinates actions to appear to the end-user as a single coherent system.

1. The 8 Fallacies of Distributed Computing

L. Peter Deutsch formulated the 8 false assumptions developers commonly make when transitioning from monolithic to distributed systems:

The network is reliable: Connections drop, packets get lost, and firewalls reset sockets.
Latency is zero: Every remote RPC / HTTP call adds measurable network delay.
Bandwidth is infinite: Network saturation causes packet buffering and bottlenecks.
The network is secure: Data in transit must always be encrypted via TLS.
Topology does not change: Nodes, switches, and IPs dynamically change in cloud environments.
There is one administrator: Different microservices are managed by different teams.
Transport cost is zero: Serialization/deserialization (JSON/Protobuf) costs CPU cycles.
The network is homogeneous: Nodes run different OS versions, architectures, and network stacks.

2. Distributed Consensus (Paxos & Raft)

Consensus is the process of getting multiple distributed nodes to agree on a consistent value or state transition, even when some nodes fail or experience network partitions.

Comparison of Consensus Protocols:

Protocol	Used By	Characteristics
Paxos	Google Chubby, Spanner	Proven mathematically correct; notoriously complex to implement
Raft	etcd (Kubernetes), HashiCorp Consul, CockroachDB	Understandable leader-based consensus protocol
ZAB	Apache ZooKeeper	Primary-backup atomic broadcast algorithm

Raft Protocol Workflow:

flowchart TD A["All nodes begin as Followers"] --> B{"Missed heartbeat?\n(Election Timeout)"} B -->|"Yes"| C["Transition to Candidate\n& request votes"] C --> D{"Received majority\nvotes (> N/2)?"} D -->|"Yes"| E["🏆 Becomes Leader"] E --> F["Leader accepts client writes\n& appends to local Log"] F --> G["Leader replicates log entries\nto Followers via AppendEntries RPC"] G --> H{"Majority of Followers\nacknowledge?"} H -->|"Yes"| I["✅ Leader commits entry\n& responds to client"]

3. Distributed Locking (Redis Redlock & Zookeeper)

When multiple instances of a service must coordinate exclusive access to a shared resource (e.g., claiming a scheduled batch job, booking a seat):

Redis Single-Instance Distributed Lock:

# Acquire Lock with 30-second TTL (atomic command):
SET resource_lock "unique_random_uuid" NX EX 30

# NX: Set only if key does not exist
# EX: Auto-expire in 30 seconds to prevent deadlocks

Safe Release with Lua Script (Atomic check-and-delete):

if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
else
    return 0
end

4. Idempotency in Distributed Systems

An operation is idempotent if applying it multiple times produces the exact same side effects and state as applying it once.

HTTP Method / Action	Idempotent?	Reason
`GET /orders/123`	✅ Yes	Read operation; state remains unchanged
`PUT /users/123`	✅ Yes	Entire resource is replaced with identical state
`DELETE /orders/123`	✅ Yes	Deleting an already deleted record has same final state
`POST /payments`	❌ No	Submitting twice charges the customer twice without idempotency protection

Implementing Idempotency with Idempotency Keys:

flowchart TD A["1. Client generates UUID\nIdempotency-Key: 7b34b1a8-d9f2-..."] --> B["2. Client sends POST /payments\nwith header"] B --> C{"3. Server checks\nDB / Redis"} C -->|"Key exists &\nstatus = COMPLETED"| D["✅ Return cached\nresponse immediately"] C -->|"Key exists &\nstatus = PROCESSING"| E["⚠️ Return 409 Conflict"] C -->|"Key not found"| F["Save key with status PROCESSING\n& execute payment"]