Production checklist
What to check before taking a TeselaDB cluster to production.
Go through these points before your applications use the cluster in production, from the machines to the backups.
Requirements
- Linux on x86_64 or aarch64. macOS is for development, with no latency guarantees.
- NVMe disks. TeselaDB writes with
O_DIRECTand uses io_uring; if the kernel lacks it, it uses I/O threads and says so in the log. - At least three nodes, on different machines, so each range keeps answering if one goes down.
- Clocks synchronised with NTP. A node whose clock drifts more than 125 ms from the cluster median hands over its leaderships and takes none until it is back within the margin.
Resources
| Option | Default | What it does |
|---|---|---|
--cores | logical cores | One thread per core, the same on every node |
--cpu-list | CPUs allowed to the process | Which CPU each thread is pinned to |
--memory-limit | 75% of the lower of physical memory and the cgroup limit | Total memory of the node |
--io-executor | auto | uring or threads |
--direct-io | auto | on or off |
Network
Open port 27100 between the nodes, 27017 to your applications and, if you use it, the REST API on 27080. The administration port 27200 should only be reachable by your operations team and your metrics system; Ports says which credentials each one asks for.
Security
- Do not use
--insecure. - Create one user per application with its roles and keep the
rootpassword off the server. - Turn on encryption at rest if your disks do not already do it, as Encryption at rest explains.
Operations
- Schedule backups with
teseladb admin backup, following Backup and restore. - Send
/metricsto Prometheus and/healthzto your load balancer, as Monitoring describes. - Before you retire a machine, drain its node with the steps in Nodes.
Check
teseladb admin nodes lists every node as active. /healthz answers 200 on each of them. If your load balancer
queries /healthz, it stops sending traffic to a node as soon as the node stops answering.
Common problems
- If a node hands over all its leaderships and accepts none, its clock has drifted more than 125 ms from the cluster median. Check NTP on that machine; the node accepts them again once the clock is back within the margin.
- If a node starts with I/O threads and not with io_uring, its kernel has no io_uring. The start-up log says so.
See also
- Ports, with what each port asks for.
- Deploy with systemd, to set up the three nodes.