Alec Matusis wrote:
We have encountered an unexpected performance impact only by moving an LMDB to a different linux machine with similar hardware characteristics.
We have a 93GB LMDB environment with 3 databases. The database in question is 13GB. The test executable loops over key/value pairs in the 13GB database with a read-only cursor. For the same executable, we observe two different behaviors on different machines (the same lmdb environment was copied to the machines with scp). First machine has 148GB RAM and the second has 105GB RAM and the same CPU.
The expected and desired behavior on linux kernel 3.13 / eglibc 2.19 shows that the process takes 13GB of shared memory (seen by top and confirmed with /proc/<pid>/smaps below)). In the alternate behavior on kernel 4.15 / glibc 2.27 the process reads from disk 83GB into shared memory (which takes 16 minutes instead of 8 minutes on the correct machine in the corresponding initial runs).
glibc version is irrelevant, only the kernel version matters, since no library calls of any kind are invoked in a read operation.
Try repeating the test with MDB_NORDAHEAD set on the environment.