One Raft group per range

In TeselaDB, each key range is a Raft group of its own. Its replicas agree on the order of their writes among themselves, without going through a central coordinator, and a cluster with thousands of ranges has thousands of groups that each move at their own pace. Almost every guarantee your application sees comes from that decision, from what a write waits for to what happens when a node is cut off from the others.

What a range is

A range is an interval [start, end) of the ordered key space. A collection starts in one range and splits when it holds more than 64 MiB of live data or 512 MiB on disk with its versions, or when it takes too much load. Each range has its replicas on several nodes. The cluster replicates data range by range. It also spreads ranges across nodes and splits them with that same unit.

   range 42                 range 43
 ┌─────────────┐         ┌─────────────┐
 │ leader  n1  │         │ leader  n2  │
 │ replica n2  │         │ replica n3  │
 │ replica n3  │         │ replica n1  │
 └─────────────┘         └─────────────┘

How a write commits

One replica of each range is the leader. It proposes writes in its log, and an entry is applied once a majority of the replicas have it on disk. When you get an answer depends on the write concern you ask for. With w: "majority", the answer comes once the majority has stored the entry with fdatasync and the leader has applied it. With w: 1 it is enough for the entry to be in the leader's log. If you add j: true, it also has to be on its disk. Asking for more replicas than the range has fails with UnsatisfiableWriteConcern (100), as in MongoDB. An orderly shutdown of the node flushes to disk everything its log already had, also what was confirmed with w: 1, while a crash can lose it.

Reading without asking

While its lease lasts, the leader knows no other replica can have committed writes, so it answers reads without sending a single message. A newly elected leader has no lease yet and confirms its leadership with ReadIndex before reading. linearizable reads always confirm it, even with a lease, as MongoDB confirms the majority on every read of that kind. A follower can also serve reads, up to its closed timestamp, which the leader publishes 3 seconds behind its clock.

Writes that can be repeated

When a driver loses the answer to a write, for example because the connection dropped right after the node received it, it sends the same write again with the same session and statement numbers. So that it is not applied twice, each range keeps a table with the session id, the transaction number and the statement number of each write, with its result. If the same write arrives again, the range returns the stored result without repeating it. The table is purged after 30 minutes and travels inside the range's snapshots, so a new replica knows it too.

When the leader is cut off

If a network partition separates the leader from the rest, it can no longer commit writes and its lease expires. The majority elects another leader among the remaining replicas. Drivers do not need to know which one it is. The node they reach answers NotLeader with a hint about the new leader, and TeselaDB's internal client follows that hint or tries the next replica. For your application, the change shows as an operation that takes a little longer or is retried.

Many groups on each core

A node carries many ranges. Each core of each node runs many groups at once, and its Raft log is also the write-ahead log of storage, so a write is not stored twice on disk. An idle group sends no heartbeats, which is why thousands of quiet ranges generate no Raft traffic between the nodes. A Raft entry holds up to 64 MiB, but it travels over the network in pieces of 1 MiB at most, just like the snapshot a new replica catches up with. The cluster metadata, with the nodes, the catalogue and the list of ranges, lives in another Raft group, Group 0.

What you gain

Ranges are independent of each other. A problem in one does not slow the others, because each group elects its leader and commits its writes without waiting for any other. If a node goes down, each range it led elects a new leader among its replicas and keeps taking writes while a majority answers. A write confirmed with w: "majority" survives the loss of any minority of the replicas of its range.

What it costs

A range needs a majority of its replicas to accept writes, so each write waits for the disk of that majority. If you spread the replicas across distant data centres, that wait shows in the latency of each write, because every commit includes the round trip to the replicas that make up the majority. The Raft in every range and Ranges and replicas pages tell the same as reference.