A transaction across nodes

A TeselaDB transaction can touch documents of several ranges that live on different nodes. Even so it commits atomically and with snapshot isolation, just like a MongoDB transaction. To see how, we follow one purchase from its first write until its changes become ordinary versions in each range, and then we look at what happens when another operation crosses its path or when the node that coordinates it goes down halfway.

The example

range A (node 1): stock      { _id: "red-chair", available: 3 }
range B (node 2): orders     { customer: "Ana", sku: "red-chair", units: 1 }

The purchase takes one unit from the stock and creates the order. The stock document lives in a range whose leader is on node 1. The new order lands in another range, led by node 2. Neither node can commit the purchase on its own, because each one only sees half of it.

If everything fell in one range

Before going on, the easy case is worth a look. When all the keys of a transaction fall in the same range, its leader commits it with a single write replicated by Raft, with the same conflict checks as the long path. There is no separate record and no provisional versions. It is the fastest case, and the reason why a good key keeps together the documents that are written at the same time.

The transaction record

In our example there are two ranges, so the transaction needs an arbiter. It is the transaction record, an internal document that lives in the range covering its key and is therefore replicated with Raft like any other data. It stores the state (Pending, Committed or Aborted), the read timestamp, the minimum commit timestamp, the commit timestamp, the coordinator's last heartbeat and the key intervals where the transaction left writes. While the record says Pending, nobody else sees the purchase.

Writing without committing

Each write of the transaction leaves a provisional version in its range, an intent. Node 1 writes the stock intent with available: 2 and node 2 writes the order's. Before accepting it, each leader checks two things. If there is a committed version newer than the transaction's read timestamp, or an intent of another transaction that is still alive, the write fails with WriteConflict and the TransientTransactionError label. A write on its own, outside any transaction, does not fail in that case. It waits for the transaction that owns the intent to finish, within its maxTimeMS, as MongoDB does.

Committing

With both intents written, the coordinator asks the record to commit. The commit timestamp is the greatest of three: the one the coordinator proposes, the minimum commit timestamp the record keeps and the highest timestamp of its intents, which a leader may have raised. That way no version lands below a read already served. The purchase exists from the moment that change of state commits in Raft. Before answering, the coordinator waits until its wall clock reaches the commit timestamp, so that any transaction starting after the answer sees the purchase.

After the answer

Your application gets the confirmation as soon as the record changes state. The intents stay provisional in their ranges for a moment, because turning them into ordinary versions is background work. If a reader gets there before that work and finds an intent whose record has already finished, it resolves the intent itself on the spot. In no case does the conversion hold the client back.

When two operations cross

Suppose another read with a greater timestamp finds the stock intent while the purchase is still Pending. It does not wait. It pushes the purchase's minimum commit timestamp above its own. Then it looks at what the record returns. If it is still Pending, already pushed, or Aborted, it reads the previous version, available: 3. If it is already Committed with a timestamp lower than or equal to its own, it reads the value of the intent. The read never stands still waiting for a commit, and the purchase, if it commits, will do so above that read.

If the coordinator goes down

The coordinator sends heartbeats to the record. A transaction without a heartbeat for 5 seconds counts as abandoned, and anyone who stumbles on one of its intents aborts it with a push before going on. If the coordinator died before creating the record, that push creates it already Aborted, and when the coordinator comes back and tries to create it, it gets NoSuchTransaction with TransientTransactionError. A transaction therefore never ends up aborted for one reader and committed for another. A collector on each core also goes through the pending records of its ranges and aborts the abandoned ones.

Limits

A transaction is aborted after 60 seconds by default. If its state does not fit in the share of memory its coordinator has, it fails with TransactionTooLarge (257). A read at a timestamp older than the history the node keeps fails with SnapshotTooOld (239). If the range of the record splits during the transaction, the record travels with its key and the transaction does not notice. To see it in code, the guide Inventory without overselling takes stock and creates an order in a single transaction.