https://bugs.openldap.org/show_bug.cgi?id=9769
Issue ID: 9769 Summary: Patch new feature batch get Product: LMDB Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Keywords: needs_review Severity: normal Priority: --- Component: liblmdb Assignee: bugs@openldap.org Reporter: rouzier@gmail.com Target Milestone: ---
Created attachment 859 --> https://bugs.openldap.org/attachment.cgi?id=859&action=edit New functionality mdb_cursor_get_batch
New functionality mdb_cursor_get_batch
mdb_cursor_get_batch retrieves a page worth of key/values. This is to reduce the number of function calls when doing a scan of the database.
https://bugs.openldap.org/show_bug.cgi?id=9769
--- Comment #1 from rouzier@gmail.com rouzier@gmail.com --- I, James Rouzier, hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice.
https://bugs.openldap.org/show_bug.cgi?id=9769
--- Comment #2 from Howard Chu hyc@openldap.org --- (In reply to rouzier@gmail.com from comment #0)
Created attachment 859 [details] New functionality mdb_cursor_get_batch
New functionality mdb_cursor_get_batch
mdb_cursor_get_batch retrieves a page worth of key/values. This is to reduce the number of function calls when doing a scan of the database.
That's what MDB_GET_MULTIPLE is for.
https://bugs.openldap.org/show_bug.cgi?id=9769
--- Comment #3 from rouzier@gmail.com rouzier@gmail.com --- MDB_GET_MULTIPLE only works on MDB_DUPFIXED.
mdb_cursor_get_batch retrieves a page worth of key/values on a "regular" db. This is to reduce the number of function calls when scanning the database.
https://bugs.openldap.org/show_bug.cgi?id=9769
--- Comment #4 from Howard Chu hyc@openldap.org --- (In reply to rouzier@gmail.com from comment #3)
MDB_GET_MULTIPLE only works on MDB_DUPFIXED.
mdb_cursor_get_batch retrieves a page worth of key/values on a "regular" db. This is to reduce the number of function calls when scanning the database.
Please provide benchmarks showing the call overhead that this patch saves.
https://bugs.openldap.org/show_bug.cgi?id=9769
--- Comment #5 from rouzier@gmail.com rouzier@gmail.com --- Created attachment 860 --> https://bugs.openldap.org/attachment.cgi?id=860&action=edit simple bench mark
A simple benchmark that counts the number of records using mdb_cursor_get_batch and mdb_cursor_get.
Please let me know if you would like a better benchmark.
https://bugs.openldap.org/show_bug.cgi?id=9769
rouzier@gmail.com rouzier@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Attachment #859 is|0 |1 obsolete| |
--- Comment #6 from rouzier@gmail.com rouzier@gmail.com --- Created attachment 862 --> https://bugs.openldap.org/attachment.cgi?id=862&action=edit Updated patch
I rework the feature and merged the functionality into mdb_get_cursor.
https://bugs.openldap.org/show_bug.cgi?id=9769
rouzier@gmail.com rouzier@gmail.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Attachment #860 is|0 |1 obsolete| |
--- Comment #7 from rouzier@gmail.com rouzier@gmail.com --- Created attachment 863 --> https://bugs.openldap.org/attachment.cgi?id=863&action=edit Updated benchmark
It looks like on my setup it is about twice as fast.
The following is the output of bench.c
Adding 1000000 values count: A 1000 full scans using cursor_next_batch took 5.93687200 seconds (5936872) count: A 1000 full scans using cursor_next took 13.50153900 seconds (13501539) noop: A 1000 full scans using cursor_next_batch took 6.31800600 seconds (6318006) noop: A 1000 full scans using cursor_next took 13.50907800 seconds (13509078)
https://bugs.openldap.org/show_bug.cgi?id=9769
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|--- |1.0.0 Keywords|needs_review |
-
openldap-its＠openldap.org