Hi,
1. Is it possible to disable memory mapping in LMDB?
2. If I create a brand-new database and write data into it (but haven't read it yet) can I be assured it won't be memory-mapped yet?
The purpose of these question is to get more consistent/reproducible time benchmarks for some stuff I'm doing. (As you know the drive vs. memory-mapped performance varies heavily.)
Thanks, Sam
Sam Dave wrote:
Hi,
- Is it possible to disable memory mapping in LMDB?
No, that is fundamental to LMDB's design. The name is "Lightning Memory-Mapped Database", it means what it says.
- If I create a brand-new database and write data into it (but haven't read it yet) can I be assured it won't be memory-mapped yet?
The question makes no sense.
The purpose of these question is to get more consistent/reproducible time benchmarks for some stuff I'm doing.
There are two ways to get consistent timings - use a test DB that fits entirely in RAM, or use a test DB that is several times larger than RAM.
For an example of the latter, see http://www.lmdb.tech/bench/hyperdex/
When testing LMDB itself these tests tend to be not very interesting; they only show you that LMDB runs as fast as your storage device has IOPS. When comparing against other DBs these tests show how badly other DB designs fare under heavy memory pressure...
(As you know the drive vs. memory-mapped performance varies heavily.)
Memory mapping has nothing to do with variance. Physical I/O determines that, and it would be the same variance even if you were using standard read/write calls (although it would be slower overall).
Thanks, Sam
Thanks for the info!
- If I create a brand-new database and write data into it (but haven't read it yet) can I be assured it won't be memory-mapped yet?
The question makes no sense.
I have noticed that reading the database (iterating through with a cursor) the 2nd time tends to be much faster than the 1st time (sometimes the speedup happens after the 3rd or 4th time). I presumed this was because the memory mapping happened not immediately.
That's the background of my 2. question. What was I missing?
Thanks, Sam
Mar 9, 2020, 01:07 by hyc@symas.com:
Sam Dave wrote:
Hi,
- Is it possible to disable memory mapping in LMDB?
No, that is fundamental to LMDB's design. The name is "Lightning Memory-Mapped Database", it means what it says.
- If I create a brand-new database and write data into it (but haven't read it yet) can I be assured it won't be memory-mapped yet?
The question makes no sense.
The purpose of these question is to get more consistent/reproducible time benchmarks for some stuff I'm doing.
There are two ways to get consistent timings - use a test DB that fits entirely in RAM, or use a test DB that is several times larger than RAM.
For an example of the latter, see http://www.lmdb.tech/bench/hyperdex/
When testing LMDB itself these tests tend to be not very interesting; they only show you that LMDB runs as fast as your storage device has IOPS. When comparing against other DBs these tests show how badly other DB designs fare under heavy memory pressure...
(As you know the drive vs. memory-mapped performance varies heavily.)
Memory mapping has nothing to do with variance. Physical I/O determines that, and it would be the same variance even if you were using standard read/write calls (although it would be slower overall).
Thanks, Sam
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
I guess he means "cached" instead of "memory mapped" ?
Geert
On Mon, Mar 09, 2020 at 00:07:25 +0000, Howard Chu wrote:
Sam Dave wrote:
Hi,
- Is it possible to disable memory mapping in LMDB?
No, that is fundamental to LMDB's design. The name is "Lightning Memory- Mapped Database", it means what it says.
- If I create a brand-new database and write data into it (but haven't
read it yet) can I be assured it won't be memory-mapped yet?
The question makes no sense.
The purpose of these question is to get more consistent/reproducible time benchmarks for some stuff I'm doing.
There are two ways to get consistent timings - use a test DB that fits entirely in RAM, or use a test DB that is several times larger than RAM.
For an example of the latter, see http://www.lmdb.tech/bench/hyperdex/
When testing LMDB itself these tests tend to be not very interesting; they only show you that LMDB runs as fast as your storage device has IOPS. When comparing against other DBs these tests show how badly other DB designs fare under heavy memory pressure...
(As you know the drive vs. memory-mapped performance varies heavily.)
Memory mapping has nothing to do with variance. Physical I/O determines that, and it would be the same variance even if you were using standard read/write calls (although it would be slower overall).
Thanks, Sam
openldap-technical@openldap.org
-
Geert Hendrickx -
Howard Chu -
Sam Dave