https://bugs.openldap.org/show_bug.cgi?id=10262
Issue ID: 10262 Summary: Feature request: configurable memory alignment of LMDB keys and values Product: LMDB Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Keywords: needs_review Severity: normal Priority: --- Component: liblmdb Assignee: bugs@openldap.org Reporter: sascha@brawer.ch Target Milestone: ---
When creating an LMDB table, it would be nice if an application could request how its keys and values will be aligned in memory.
Currently, LMDB seems to gives 2-byte alignment; see LMDB issue 10260. On most non-Intel CPUs, unaligned reads will cause SIGBUS errors, so any data with 32-bit or 64-bit values has to be accessed in multiple 16-bit chunks (which is inefficient), or copied out of LMDB-mapped memory into a custom, properly aligned buffer (which is inefficient, too). To prevent such performance degradation, it would be nice if applications could request alignment of keys and/or values to 8-byte boundaries. Then, LMDB data would have the same alignment guarantees as malloc().
The Linux kernel has a nice description of alignment: https://www.kernel.org/doc/html/latest/core-api/unaligned-memory-access.html
Even on Intel CPUs, being able to specify alignment would be useful. For example, AVX-512 benefits from data being aligned to 64-byte boundaries. If an application could request 64-byte alignment for a given table, its values could be loaded direclty into AVX-512 registers. This would be useful for applications whose tables contain bitvectors or other data suitable for SIMD proceesing. https://www.intel.com/content/www/us/en/developer/articles/technical/data-al...
Of course, padding comes at a cost. It increases storage space and reduces cache effectiveness. It would be wasteful to align each key and value in every table to some boundary. Hence the suggestion to make this configurable per table, perhaps with additional flags for mdb_dbi_open().
One could argue that memory alignment is out of scope for LMDB, leaving it up to applications to deal with misalignments. However, because of the cost of workarounds, it would make LMDB (significantly) less efficient than it could be, even on Intel CPUs. Thus, many thanks for considering this feature request.