Distributed Key-Value Store

What it is for

Shared state across a distributed system has two ways to go wrong. A node whose copy falls out of sync starts acting on the wrong context, and the drift spreads. And when nodes lose connection, the store still has to answer. This system trades speed for both: it is meant for propagating system state and configuration changes, not for raw sensor data or high-rate telemetry, where a time-series database or an append-only log is the better tool.

Consensus

It implements Raft, from Ongaro and Ousterhout. One elected leader owns the replicated log; writes sent to a follower are forwarded to it. Every entry the leader sends carries the index and term of the entry before it, and a follower only accepts it if its own log matches there. A follower whose log is too short reports where it ends and the leader jumps straight to that point; one with a conflicting term at the same index walks back with the leader until they agree, and everything after is overwritten.

Elections go through a pre-vote first. A follower whose election deadline passes asks its peers whether they would vote for it, and they only say yes if they have also not heard from the leader and its log is at least as up to date as theirs. Only with a majority of pre-votes does it start a real election. The cluster stays up as long as ⌊n/2⌋ + 1 nodes can reach each other.

Changing the cluster

Nodes can be added and removed through /cluster/members while the cluster runs. A new node joins as a learner: it receives the log but cannot vote and does not count toward quorum until it has caught up, so adding a node never stalls commits behind a node with an empty log.

Removing the leader is the special case. It stays leader until its own removal commits, then picks the voter whose log most closely matches its own, brings it fully up to date, steps down, and has that node start the next election immediately.

Node IDs are random at startup, so an operator only has to give each node its peers’ addresses; each node asks its peers for their IDs through the API.

Log compaction

Left alone, a Raft log grows forever. Each node periodically snapshots its state machine: the last applied index and term, the voter, learner and leader configuration, and a copy of the key space, and drops the log before it, keeping only a short tail. A follower that has fallen behind the start of the leader’s log is sent the cached snapshot in chunks instead. With a data directory set, snapshots and logs are written to disk and the directory is fsynced, so a node survives a crash.

Telemetry

GET /cluster/metrics returns Prometheus text and GET /cluster/events returns the last 256 notable events: node starts and stops, leader changes, peers going down. Data comes from three places. Client requests are bucketed by status in axum middleware; three failed RPCs in a row mark a peer as down, and that log line is intercepted as an event; and a failed disk write emits a persistence-failure event, with a recovery event once writes succeed again.

A/B testing put the overhead in the noise: PUTs to the leader went from 5,719 to 5,692 requests a second, and PUTs to followers actually rose, from 5,557 to 5,730.

Testing & Validation

The test that found the most bugs was a Jepsen-style chaos test. Every 0.1 to 1.2 seconds it kills or freezes a node, while simulated clients send a 90/10 mix of writes and reads to random nodes and check that they never read a stale value. After each run it verifies that every acknowledged write reads back correctly, every node ends with the same data and agreeing logs, no term ever had two leaders, and no node crashed on its own.

Across a 3.8-hour campaign — 3.26M writes, 3,808 leader terms, 196 log truncations — all 213 runs passed. Median write latency held at about 1.4 ms whether zero or four of five nodes were down, and median recovery after a fault was under 0.3 seconds, 13 seconds at worst. What moved was the tail and throughput: p99 rose with more faults, throughput fell sharply once a majority could be down, and a 30-minute soak with aggressive compaction settings (every 500 entries, 4 KB chunks) saw p99 climb from 19 ms to 213 ms, where the default settings held flat at about 11 ms.

Alongside the chaos test there are 6 unit tests and 17 cluster integration tests covering replication, quorum, membership, snapshotting and telemetry.