Hi,
I've just spent about two days chasing down a bug in my application using LMDB, where if I ran the code on an ARM (and ARM only) platform, data returned from mdb_get() would appear to be corrupted.
The symtomps were quite bizarre, I'm including them here in case someone else sees something similar; you can skip to the conclusion if you're in a hurry:
a) data written by an application process using mdb_put() appeared correct and consistent b) the same data read by a different application process using mdb_cursor_get() would appear consistent when examined with a live GDB on the target c) however, when actually running the application code, even under GDB, some data would mysteriously be corrupted d) single-stepping through one case of corruption lead me to a single LDR instruction which appeared to load bogus data from memory. GDB showed the same data (at the exact same memory address) as correct, i.e. not corrupted. e) the problem only occured on ARM and was always reproducible.
Much experimenting and googling later I found [1], the upshot of which is that on some ARM platforms unaligned word accesses to data using LDR produce *bogus* data. No exception, nothing, just silent data corruption.
My question is: Given that I'm storing C structs directly in LMDB, I need to get at least word-aligned pointers back from LMDB in order to be able to access the data safely.
I've not found anything in the documentation or the source about being able to tweak alignment of key and data values; I *think* that all I need is an option where LMDB would guarantee a minimum word (4 byte in this case) alignment of data.
How hard would this be to implement? Would you consider such a feature?
The only workaround I can think of is explicitly copying all data I get back from LMDB into aligned struct pointers; for obvious performance reasons I don't want to do that.
Martin
[1] http://stackoverflow.com/questions/21488501/strange-pointer-arithmetic See the second answer, and the link to the ARM documentation for the LDR instruction.
Martin Lucina wrote:
Hi,
I've just spent about two days chasing down a bug in my application using LMDB, where if I ran the code on an ARM (and ARM only) platform, data returned from mdb_get() would appear to be corrupted.
The symtomps were quite bizarre, I'm including them here in case someone else sees something similar; you can skip to the conclusion if you're in a hurry:
a) data written by an application process using mdb_put() appeared correct and consistent b) the same data read by a different application process using mdb_cursor_get() would appear consistent when examined with a live GDB on the target c) however, when actually running the application code, even under GDB, some data would mysteriously be corrupted d) single-stepping through one case of corruption lead me to a single LDR instruction which appeared to load bogus data from memory. GDB showed the same data (at the exact same memory address) as correct, i.e. not corrupted. e) the problem only occured on ARM and was always reproducible.
Much experimenting and googling later I found [1], the upshot of which is that on some ARM platforms unaligned word accesses to data using LDR produce *bogus* data. No exception, nothing, just silent data corruption.
My question is: Given that I'm storing C structs directly in LMDB, I need to get at least word-aligned pointers back from LMDB in order to be able to access the data safely.
I've not found anything in the documentation or the source about being able to tweak alignment of key and data values; I *think* that all I need is an option where LMDB would guarantee a minimum word (4 byte in this case) alignment of data.
How hard would this be to implement? Would you consider such a feature?
LMDB guarantees 2-byte alignment of keys, and data is generally stored contiguously with keys. If you want 4-byte alignment, it's your responsibility to pad your keys to 4-byte boundaries. If you pad appropriately, your alignment will be preserved.
The only workaround I can think of is explicitly copying all data I get back from LMDB into aligned struct pointers; for obvious performance reasons I don't want to do that.
Martin
[1] http://stackoverflow.com/questions/21488501/strange-pointer-arithmetic See the second answer, and the link to the ARM documentation for the LDR instruction.
hyc@symas.com said:
My question is: Given that I'm storing C structs directly in LMDB, I need to get at least word-aligned pointers back from LMDB in order to be able to access the data safely.
I've not found anything in the documentation or the source about being able to tweak alignment of key and data values; I *think* that all I need is an option where LMDB would guarantee a minimum word (4 byte in this case) alignment of data.
How hard would this be to implement? Would you consider such a feature?
LMDB guarantees 2-byte alignment of keys, and data is generally stored contiguously with keys. If you want 4-byte alignment, it's your responsibility to pad your keys to 4-byte boundaries. If you pad appropriately, your alignment will be preserved.
In this case the keys are size_t (4 bytes on the target) and I'm using MDB_INTEGERKEY. I'm not sure what you mean by padding the keys to 4-byte boundaries given that LMDB copies my data into its memory map and I have no control over where it gets placed.
Martin
martin@lucina.net said:
hyc@symas.com said:
My question is: Given that I'm storing C structs directly in LMDB, I need to get at least word-aligned pointers back from LMDB in order to be able to access the data safely.
I've not found anything in the documentation or the source about being able to tweak alignment of key and data values; I *think* that all I need is an option where LMDB would guarantee a minimum word (4 byte in this case) alignment of data.
How hard would this be to implement? Would you consider such a feature?
LMDB guarantees 2-byte alignment of keys, and data is generally stored contiguously with keys. If you want 4-byte alignment, it's your responsibility to pad your keys to 4-byte boundaries. If you pad appropriately, your alignment will be preserved.
In this case the keys are size_t (4 bytes on the target) and I'm using MDB_INTEGERKEY. I'm not sure what you mean by padding the keys to 4-byte boundaries given that LMDB copies my data into its memory map and I have no control over where it gets placed.
Ah. What you forgot to mention (and I tracked down in a Reddit comment somewhere) is that the (key + data) size (and not just the key size) should be a multiple of the required alignment. The problematic records in question are using VLAs to store variable length strings; if I pad the allocated space to the nearest word boundary that seems to fix the issue.
Martin
Martin Lucina wrote:
martin@lucina.net said:
hyc@symas.com said:
My question is: Given that I'm storing C structs directly in LMDB, I need to get at least word-aligned pointers back from LMDB in order to be able to access the data safely.
I've not found anything in the documentation or the source about being able to tweak alignment of key and data values; I *think* that all I need is an option where LMDB would guarantee a minimum word (4 byte in this case) alignment of data.
How hard would this be to implement? Would you consider such a feature?
LMDB guarantees 2-byte alignment of keys, and data is generally stored contiguously with keys. If you want 4-byte alignment, it's your responsibility to pad your keys to 4-byte boundaries. If you pad appropriately, your alignment will be preserved.
In this case the keys are size_t (4 bytes on the target) and I'm using MDB_INTEGERKEY. I'm not sure what you mean by padding the keys to 4-byte boundaries given that LMDB copies my data into its memory map and I have no control over where it gets placed.
Ah. What you forgot to mention (and I tracked down in a Reddit comment somewhere) is that the (key + data) size (and not just the key size) should be a multiple of the required alignment.
That is what "data is stored contiguously with keys" means.
The problematic records in
question are using VLAs to store variable length strings; if I pad the allocated space to the nearest word boundary that seems to fix the issue.
Martin
openldap-technical@openldap.org
-
Howard Chu -
Martin Lucina