Nodes
How to add nodes to the cluster and retire the ones you no longer need without stopping the service.
Every node runs the same binary. The cluster spreads replicas and leaders across them, so adding or removing one needs no manual data movement.
Requirements
- Access to
teseladb adminon a node of the cluster. - To add one, a machine with the binary installed and port 27100 open towards the other nodes.
List the nodes
teseladb admin nodesAdd a node
teseladb admin token create --ttl 1h
teseladb start --dbpath /var/lib/teseladb --join node-01:27100 --enroll-token <token>The first command runs on any node of the cluster and creates a single-use enrolment token; the second starts the binary on the new machine with that token. The new node gets its certificate signed by the cluster CA and starts receiving range replicas.
Drain a node
teseladb admin node drain <node_id>You cannot drain the last active node. Drain the node before retiring its machine. The node goes to Draining and the balancer takes away its replicas and its
leaderships and, last, its vote in Group 0; once it has no replica left, it closes its client port and the drivers go to
another node.
Remove a node
teseladb admin node remove <node_id>Remove the node once it has no replicas and no vote in Group 0. If it still has replicas, the command fails, says how many are left and asks you to finish the drain first.
Rotate a node's certificate
teseladb admin cert rotateWithout restarting, the node generates a new key and certificate signed by the cluster CA and starts using them.
Check
teseladb admin nodes shows the new node as active and, after a while, with replicas. A drained node shows up as
Draining until it has no replicas left. As soon as you remove it, it leaves the list.
Common problems
node drainfails on the last active node, because a cluster needs at least one node serving.node removefails and says how many replicas the node has left. Wait for the drain to finish and try again.
See also
- Deploy with systemd, with adding a node step by step.
- Ranges and replicas, to see how replicas are spread.