https://bugs.openldap.org/show_bug.cgi?id=9475
Issue ID: 9475 Summary: Add support for MAP_POPULATE Product: LMDB Version: unspecified Hardware: All OS: Linux Status: UNCONFIRMED Severity: normal Priority: --- Component: liblmdb Assignee: bugs@openldap.org Reporter: aa531811820@gmail.com Target Milestone: ---
In some case (such as cloud computing platforms), the reading speed of large files is very fast while the small files is very slow, and we have enough memory, so we hope to prefetch the entire LMDB file into the memory during MMAP through the MAP_POPULATE flag . According to our test, this is faster than using readahead flag. Here are some test data:
# mmap with no readahead read one sample: 0.2s total time: 4800s
# mmap with readahead read one sample: 0.0001s~0.03s total time: 95.86s
# mmap with MAP_POPULATE db init: 20s read one sample: 0.0001s total time: 78s
https://bugs.openldap.org/show_bug.cgi?id=9475
--- Comment #1 from Howard Chu hyc@openldap.org --- (In reply to leslie from comment #0)
In some case (such as cloud computing platforms), the reading speed of large files is very fast while the small files is very slow, and we have enough memory, so we hope to prefetch the entire LMDB file into the memory during MMAP through the MAP_POPULATE flag . According to our test, this is faster than using readahead flag. Here are some test data:
# mmap with no readahead read one sample: 0.2s total time: 4800s
# mmap with readahead read one sample: 0.0001s~0.03s total time: 95.86s
# mmap with MAP_POPULATE db init: 20s read one sample: 0.0001s total time: 78s
Have you tested the performance by simply dd'ing the entire file into memory before running your test? `dd if=data.mdb of=/dev/null` would be sufficient to preload the data into memory.
https://bugs.openldap.org/show_bug.cgi?id=9475
--- Comment #2 from leslie aa531811820@gmail.com --- Read the entire file before running can be as fast as MAP_POPULATE. But I think using MAP_POPULATE is better, by providing a 'map_populate' parameter in 'env_new' function, users can clearly know this feature.
openldap-its@openldap.org 于2021年2月22日周一 下午9:59写道：
https://bugs.openldap.org/show_bug.cgi?id=9475
--- Comment #1 from Howard Chu hyc@openldap.org --- (In reply to leslie from comment #0)
In some case (such as cloud computing platforms), the reading speed of
large
files is very fast while the small files is very slow, and we have enough memory, so we hope to prefetch the entire LMDB file into the memory
during
MMAP through the MAP_POPULATE flag . According to our test, this is
faster
than using readahead flag. Here are some test data:
# mmap with no readahead read one sample: 0.2s total time: 4800s
# mmap with readahead read one sample: 0.0001s~0.03s total time: 95.86s
# mmap with MAP_POPULATE db init: 20s read one sample: 0.0001s total time: 78s
Have you tested the performance by simply dd'ing the entire file into memory before running your test? `dd if=data.mdb of=/dev/null` would be sufficient to preload the data into memory.
-- You are receiving this mail because: You reported the issue.
-
openldap-its＠openldap.org