Troubleshoot performance
Slow queries, concentrated writes and nodes without io_uring.
These are the performance problems seen most often. For each one you have the cause, how to confirm it, how to fix it and what to do so it does not happen again.
A query reads many more documents than it returns
The query is slow and its time grows with the size of the collection. It returns few documents, but the node goes through many more to find them.
Cause
No index covers the filter or the sort of the query, so the planner walks the whole collection or an index that only serves part of the filter.
How to confirm it
Compare two figures. Run the query with explain("executionStats") and, if totalDocsExamined is much greater than
nReturned, the node is reading documents it then throws away.
Fix
Create an index with the equality fields of the filter first, then the sort fields and last the range fields. Run
explain again to see that it uses it.
How to avoid it
Put new queries through the profiler before taking them to production, with a copy of the production data, because with a small collection any query looks fast and the profiler does not warn you about the documents it reads in excess.
Writes pile up on one range
One node works much harder than the others. Nearly all the writes of a collection go to the same range, and that range splits again and again.
Cause
The key of the documents, or the key of an index, always grows: an ObjectId, a creation date or a counter. Every new
entry lands at the end of the interval.
How to confirm it
Run teseladb admin ranges --collection <database.collection>. You will see one range that splits again and again at
its end while the others barely change.
Fix
Change the key. Use a key for the documents that does not always grow, such as a random UUID, and create the indexes on
a date or a counter with teselaBuckets, which spreads their entries across hash buckets.
How to avoid it
Before you create a collection with heavy writes, choose its key with Keys and hotspots at hand.
One node is slower than the others
The operations one node leads take longer than the same operations on the other nodes, with the same load and the same data.
Cause
There are two possible causes. The kernel of that machine has no io_uring and the node uses I/O threads, or its disk is not NVMe.
How to confirm it
Start with the log. The node's start-up line says which disk executor it chose, and if it says I/O threads you have the cause; if it says io_uring, check the model of the machine's disk.
Fix
Update the kernel to a version with io_uring and use NVMe disks. Start the node again so it picks the io_uring executor.
How to avoid it
Prepare every machine of the cluster the same way, following Hardware and OS requirements before installing the binary.