https://bugs.openldap.org/show_bug.cgi?id=10275
Issue ID: 10275 Summary: mdb_load could use a parameter to pass batch size Product: LMDB Version: 0.9.21 Hardware: All OS: All Status: UNCONFIRMED Keywords: needs_review Severity: normal Priority: --- Component: tools Assignee: bugs@openldap.org Reporter: gkwicker@amazon.com Target Milestone: ---
Created attachment 1037 --> https://bugs.openldap.org/attachment.cgi?id=1037&action=edit Patch to mdb_load.c in version 0.9.21
mdb_load is very slow when ingesting a large db. The attached patch (0.9.21 but will apply to other versions with modification) allows the user to pass a batch size to be used instead of the default value of 100. Changing the batch size to a larger value improves ingestion speed.
I, Gary Wicker, hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice.
https://bugs.openldap.org/show_bug.cgi?id=10275
--- Comment #1 from Howard Chu hyc@openldap.org --- Increasing the batch size is a bit risky, since it doesn't take into account transactions with very large data items, which could overflow the txn dirty list and cause page spills - which would then reduce performance, defeating the point of this.
It probably would be better to add a NoSync option instead, if all you want is the fastest possible load time.