Distributed systems project
A Dynamo-style distributed
key-value store
Data is sharded across nodes with consistent hashing, replicated three times, and served with per-request STRONG (quorum) or EVENTUAL consistency. It keeps working when nodes crash or get partitioned, and every failure scenario is tested automatically in CI.
Try it: break the cluster
Write a key, kill or partition its replicas, and watch quorums fail, hints queue up and the Kafka log bring replicas back. Replica placement uses the same MD5 ring (256 virtual nodes) as the Java code, so these are exactly the replicas the real cluster picks.
How a write works
There is no leader. The node that receives the request becomes the coordinator for it.
- Partition: the key is hashed onto a ring with 256 virtual nodes per server; the next 3 distinct servers clockwise own it.
- Replicate: the coordinator writes to all 3 in parallel and answers when 2 confirm (STRONG) or when Kafka has durably stored the write (EVENTUAL).
- Recover: a replica that missed the write gets it from a hint as soon as it is back, or from the Kafka log, even if the coordinator crashed meanwhile.
- Detect failures: nodes refresh TTL keys in Redis every 500 ms; a missing key marks a node DOWN, and HTTP heartbeats take over if Redis itself fails.
What's inside
Each piece maps to a question asked in system design interviews, and is implemented rather than described.
Consistent hashing
Adding a 5th node moves ~20% of keys instead of ~80% with modulo hashing, which a unit test measures. The virtual-node count was chosen from measurements.
Tunable consistency
STRONG uses R + W > N majority quorums; EVENTUAL waits for one durable acknowledgement. With 2 of 3 replicas down, STRONG returns 503 and EVENTUAL keeps serving: CAP in practice.
Kafka replication log
Writes are keyed by data key so each key stays ordered in one partition. Every node is its own consumer group with static membership and resumes from committed offsets after a crash.
MySQL storage per node
Each node owns its own database. Last-write-wins is enforced atomically in SQL, so a write from a hint, a peer and the log at the same time can never go back to an older version.
Failure handling
Heartbeat failure detection, hinted handoff, read repair and a partition simulator that isolates a node from peers, Redis and Kafka at once.
Production habits
Prometheus metrics (latency percentiles, replication lag, hint backlog), structured JSON logs, Kubernetes StatefulSet, and a CI pipeline that deploys and breaks the cluster on every push.
Proven under failure
On the real system, these scenarios run automatically on every push: in JUnit, on a Docker Compose stack (5 nodes + Kafka + MySQL + Redis), and on Kubernetes.
| Scenario | Expected behaviour | Result |
|---|---|---|
| Kill 1 of a key's 3 replicas | STRONG reads and writes keep succeeding | ✓ 200 |
| Kill 2 of 3 replicas | STRONG write rejected, EVENTUAL write accepted | ✓ 503 / 200 |
| Restart the killed replicas | They receive every write they missed | ✓ converged |
| Network partition around a replica | Cluster routes around it; it catches up after healing | ✓ ~1 s |
| Coordinator dies while a replica is down | The replica still gets the write from Kafka | ✓ |
| Delete a Kubernetes replica pod | Strong writes continue; replacement pod converges | ✓ |
Show real CI output (Docker Compose: 5 nodes + Kafka + MySQL + Redis)
[ 35s] cluster healthy - stack: {"replicationLog":"kafka:kv-replication","storage":"mysql","membership":"redis"}
[ 35s] key chaos:1789297733 -> replicas node5 node1 node3, coordinator node2
[ 35s] --- scenario 1: one replica crashes
[ 35s] ok - strong PUT v1 (HTTP 200)
[ 36s] killed node5
[ 36s] ok - strong GET with 2/3 replicas (HTTP 200)
[ 36s] ok - strong PUT v2 with 2/3 replicas (HTTP 200)
[ 36s] --- scenario 2: quorum lost (two replicas down)
[ 36s] killed node1
[ 37s] ok - strong PUT rejected without quorum (HTTP 503)
[ 37s] ok - eventual PUT accepted with 1/3 replicas (HTTP 200)
[ 37s] ok - eventual GET with 1/3 replicas (HTTP 200)
[ 37s] --- scenario 3: recovery via hinted handoff
[ 38s] restarted node5
[ 39s] restarted node1
[ 51s] ok - node5 converged to 'v3'
[ 51s] ok - node1 converged to 'v3'
[ 51s] --- scenario 4: network partition around a replica
[ 51s] ok - isolate node3 (HTTP 200)
[ 51s] ok - isolated node refuses traffic (HTTP 503)
[ 51s] ok - strong PUT during partition (HTTP 200)
[ 51s] ok - heal node3 (HTTP 200)
[ 51s] ok - node3 converged to 'v4'
[ 51s] --- evidence from the infrastructure
[ 51s] ok - node3's own MySQL database (kv_node3) holds 'v4'
[ 51s] Kafka consumer group of node5 (replication log offsets and lag):
GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
kv-node5 kv-replication 5 5 5 0
[ 53s] ALL FAILURE SCENARIOS PASSED
Benchmarks
5 nodes, Kafka, MySQL and Redis sharing one laptop with the load generator; 5,000 operations per row, 32 concurrent clients. Absolute numbers are a floor; the ratio between the modes is the point.
| Consistency | Operation | Throughput (ops/s) | p50 (ms) | p99 (ms) | Errors |
|---|---|---|---|---|---|
| STRONG | PUT | 328 | 86.7 | 265.3 | 0 |
| STRONG | GET | 530 | 46.7 | 230.7 | 0 |
| EVENTUAL | PUT | 573 | 48.0 | 161.5 | 0 |
| EVENTUAL | GET | 1,086 | 20.1 | 169.6 | 0 |
Design decisions
The trade-offs behind the implementation. The README has the full list.
Why leaderless instead of Raft?
No election and no failover pause: any 2 of 3 replicas can serve a STRONG request, and consistency becomes a per-request choice. Raft would be the right call for linearizable compare-and-set.
Why isn't Kafka on the STRONG path?
STRONG promises that 2 replicas stored the write, which needs their direct acknowledgements. For EVENTUAL, the durable log is the guarantee, so the write is acknowledged once Kafka has it.
Why can Redis fail safely?
Redis only speeds up failure detection (one read for the whole cluster). If it is unreachable, nodes fall back to HTTP heartbeats, so it is not a single point of failure.
How are conflicts resolved?
Last-write-wins on a hybrid logical clock, with tombstones for deletes. Honest limit: clock skew can drop a concurrent write; vector clocks would detect it.
Is a failed STRONG write rolled back?
No, as in Dynamo and Cassandra: a 503 means "unknown outcome". Writes are idempotent, so clients retry safely.
What would come next?
Dynamic membership with range streaming, Merkle-tree anti-entropy, vector clocks and an LSM storage engine.
Delivery pipeline
Every push to main runs this in GitHub Actions.
Testcontainers: Kafka, MySQL, Redis → Chaos tests
Docker Compose, full stack → Deploy to Kubernetes
kind + pod-kill smoke test → Publish image
GHCR