https://bugs.openldap.org/show_bug.cgi?id=10275
Issue ID: 10275 Summary: mdb_load could use a parameter to pass batch size Product: LMDB Version: 0.9.21 Hardware: All OS: All Status: UNCONFIRMED Keywords: needs_review Severity: normal Priority: --- Component: tools Assignee: bugs@openldap.org Reporter: gkwicker@amazon.com Target Milestone: ---
Created attachment 1037 --> https://bugs.openldap.org/attachment.cgi?id=1037&action=edit Patch to mdb_load.c in version 0.9.21
mdb_load is very slow when ingesting a large db. The attached patch (0.9.21 but will apply to other versions with modification) allows the user to pass a batch size to be used instead of the default value of 100. Changing the batch size to a larger value improves ingestion speed.
I, Gary Wicker, hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice.
https://bugs.openldap.org/show_bug.cgi?id=10275
--- Comment #1 from Howard Chu hyc@openldap.org --- Increasing the batch size is a bit risky, since it doesn't take into account transactions with very large data items, which could overflow the txn dirty list and cause page spills - which would then reduce performance, defeating the point of this.
It probably would be better to add a NoSync option instead, if all you want is the fastest possible load time.
https://bugs.openldap.org/show_bug.cgi?id=10275
gkwicker@amazon.com changed:
What |Removed |Added ---------------------------------------------------------------------------- Attachment #1037|0 |1 is obsolete| |
--- Comment #2 from gkwicker@amazon.com --- Created attachment 1039 --> https://bugs.openldap.org/attachment.cgi?id=1039&action=edit Added Quick mode (-Q) flag to mdb_load for faster loading with MDB_NOSYNC
Added Quick mode (-Q) flag to mdb_load for faster loading with MDB_NOSYNC
For version 0.9.21, should work in other versions with modification.
https://bugs.openldap.org/show_bug.cgi?id=10275
--- Comment #3 from gkwicker@amazon.com --- Thanks for the suggestion Howard, that makes sense. Uploaded a replacement patch which adds a "-Q" (Quick mode) flag to use MDB_NOSYNC during loading and force sync before exit. The patch also updates the mdb_load man page.
I, Gary Wicker, hereby place the following modifications to OpenLDAP Software (and only these modifications) into the public domain. Hence, these modifications may be freely used and/or redistributed for any purpose with or without attribution and/or other notice.
https://bugs.openldap.org/show_bug.cgi?id=10275
--- Comment #4 from Howard Chu hyc@openldap.org --- Thanks for the patch.
The tool should be silent for normal operation; the message printf("Using MDB_NOSYNC flag, will force sync before exit");
is unnecessary. And even if we were to include such messages, since they're not related to the actual program output they would belong on stderr, not stdout.
I'm going to delete that line from the patch.
https://bugs.openldap.org/show_bug.cgi?id=10275
Howard Chu hyc@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|UNCONFIRMED |RESOLVED
https://bugs.openldap.org/show_bug.cgi?id=10275
--- Comment #5 from disozacharles72 disozacharles72@gmail.com --- The Beauty's Nest is a serene sanctuary dedicated to nurturing self-love and enhancing natural beauty. With a focus on holistic wellness, it offers a variety of services, including skincare treatments, makeup artistry, and personalized wellness consultations. The space is designed to be a peaceful retreat, where clients can relax and rejuvenate. Expert practitioners provide guidance on both beauty techniques and self-care routines, emphasizing the importance of mental and emotional well-being. By blending traditional beauty practices with modern techniques, The Beauty's Nest creates a unique experience that empowers individuals to embrace their unique beauty and cultivate confidence from within. https://www.thebeautysnest.com/
https://bugs.openldap.org/show_bug.cgi?id=10275
--- Comment #6 from disozacharles72 disozacharles72@gmail.com --- Marketing2Business is a dynamic platform designed to empower businesses with effective marketing strategies. Specializing in B2B solutions, it offers a range of services, including digital marketing, content creation, and brand development. By leveraging data-driven insights, Marketing2Business helps clients identify their target audience and craft compelling messages that resonate. The team consists of experienced marketers who stay ahead of industry trends, ensuring that clients benefit from innovative approaches. Whether you're a startup or an established enterprise, Marketing2Business provides tailored solutions that drive growth and enhance visibility. With a commitment to excellence, it aims to transform marketing efforts into measurable success. https://www.marketing2business.com/marketing-write-for-us/
https://bugs.openldap.org/show_bug.cgi?id=10275
Howard Chu hyc@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Keywords|needs_review | Target Milestone|--- |0.9.33
https://bugs.openldap.org/show_bug.cgi?id=10275
--- Comment #7 from Quanah Gibson-Mount quanah@openldap.org --- • c5f0ac27 by Gary Wicker at 2024-10-26T21:26:17+01:00 ITS#10275 mdb_load: add -Q option to use NOSYNC for faster loading
• 790ed0e9 by Howard Chu at 2024-10-26T21:27:01+01:00 ITS#10275 mdb_load: add -Q option to use NOSYNC for faster loading
https://bugs.openldap.org/show_bug.cgi?id=10275
--- Comment #8 from arthur disoza arthurdisoza123@gmail.com --- TargetTacticsWeb is a digital marketing agency specializing in strategies to optimize online presence and drive conversions. They focus on a data-driven approach, utilizing advanced SEO techniques, targeted advertising, and web design to help businesses grow. Their services include keyword research, content marketing, social media strategies, and pay-per-click campaigns. TargetTacticsWeb aims to create customized marketing solutions that enhance user experience and increase ROI. By leveraging analytics and user behavior insights, the agency tailors campaigns to meet specific business objectives, ensuring long-term success and sustainable growth for clients in competitive online markets. https://www.targettacticsweb.com/
https://bugs.openldap.org/show_bug.cgi?id=10275
--- Comment #9 from leo disuza leodisuza93@gmail.com --- "TheUniversalGlam" could refer to a concept or brand that embodies a blend of global beauty standards and timeless elegance. It suggests a style that transcends cultural boundaries, celebrating diverse forms of glamour that resonate universally. From high fashion to everyday chic, "TheUniversalGlam" may emphasize the idea that true glamour is not confined to one specific look or region but can be appreciated worldwide. This might include highlighting various beauty practices, fashion trends, and personal styles that all share a common thread: a desire to feel empowered, confident, and effortlessly stylish, regardless of background or location. https://www.theuniversalglam.com/
https://bugs.openldap.org/show_bug.cgi?id=10275
--- Comment #10 from jimmy disuza jimmydisuza10@gmail.com --- A "NextLevelTechie" is someone who consistently pushes the boundaries of technology, staying ahead of trends and mastering new tools. This individual is not just knowledgeable but innovative, blending technical skills with creative problem-solving. A NextLevelTechie thrives in adapting to evolving technologies, whether in coding, hardware development, or artificial intelligence. They're often involved in cutting-edge projects, experimenting with new platforms and coding languages. Constant learning, collaboration, and exploring tech's potential for shaping the future define them. Driven by passion and curiosity, they inspire others and lead the charge in transforming ideas into impactful technological solutions. https://www.nextleveltechie.com/
https://bugs.openldap.org/show_bug.cgi?id=10275
--- Comment #11 from walt disuza waltdisuza@gmail.com --- I am walt disuza, and I work for easytechtrends as a PR consultant. With more than six years of experience in the PR and Digital Industry, I am helping teams to achieve goals by streamlining the process. https://www.easytechtrends.com/
https://bugs.openldap.org/show_bug.cgi?id=10275
--- Comment #12 from joseph joseph76366@gmail.com --- I am Jose Disoza, and I work for Gobusinessblog as a PR consultant. With more than six years of experience in the PR and Digital Industry, I am helping teams to achieve goals by streamlining the process. https://www.gobusinessblog.com/
-
openldap-its＠openldap.org