openldap 2.4.57
On 4/14 I ran mdb_copy -c to compact the DB and remove free pages. At that point I had 736k pages/2.9G file size. After compact the database is 41k pages and 140M. This solved my ldap request latency problem (seeing 30 second delays in simple queries when ldap updates are happening).
22 days later, we're at 244k pages and auth latency is up to 2 seconds during updates. The database is now 1G.
What accounts for MDB free/unused page growth? We make lots of incremental inserts and removals (Add new user, add user to group, remove user from group, remove user). Removal actions seem to trigger the query latency.
Why does a large amount of free pages impact single user removals from large groups?
mdb_stat -e /opt/slapd/data Environment Info Map address: (nil) Map size: 17179869184 Page size: 4096 Max pages: 4194304 Number of pages used: 244657 Last transaction ID: 58158791 Max readers: 126 Number of readers used: 18 Status of Main DB Tree depth: 1 Branch pages: 0 Leaf pages: 1 Overflow pages: 0 Entries: 22
--On Thursday, May 6, 2021 12:47 PM -0700 Zetan Drableg zetan.drableg@gmail.com wrote:
What accounts for MDB free/unused page growth? We make lots of incremental inserts and removals (Add new user, add user to group, remove user from group, remove user). Removal actions seem to trigger the query latency.
Your described write traffic pattern is known to cause fragmentation in the database (which causes there to be a large number of free pages). OpenLDAP 2.5 has features to address this, specifically the multival settings. See slapd-mdb(5) for OpenLDAP 2.5.
Generally you would likely want to configure both that and sortvals on your deployment once you're on OpenLDAP 2.5.
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
On 5/7/21 5:14 PM, Quanah Gibson-Mount wrote:
--On Thursday, May 6, 2021 12:47 PM -0700 Zetan Drableg zetan.drableg@gmail.com wrote:
What accounts for MDB free/unused page growth? We make lots of incremental inserts and removals (Add new user, add user to group, remove user from group, remove user). Removal actions seem to trigger the query latency.
Your described write traffic pattern is known to cause fragmentation in the database (which causes there to be a large number of free pages). OpenLDAP 2.5 has features to address this, specifically the multival settings. See slapd-mdb(5) for OpenLDAP 2.5.
Generally you would likely want to configure both that and sortvals on your deployment once you're on OpenLDAP 2.5.
Would it make sense to run a maintenance script at regular intervals which 1. stops slapd, 2. runs mdb_copy -c, 3. moves .mdb files, 4. re-starts slapd?
Ciao, Michael.
really sorry for the lack of details because I'm really outside my field.
I was just observing in openldap-2.5.4 the pbkdf2 module tries to install to /usr/local/libexec/openldap despite the
--prefix=/opt/openldap
in the configure on the src root.
I am pretty sure it worked fine on openldap-2.4.* (at the beginning it didn't but then it was fixed).
It's not a big deal, thank you for your great work,
franz
--On Saturday, May 8, 2021 7:09 PM +0200 Francesco Malvezzi francesco.malvezzi@unimore.it wrote:
really sorry for the lack of details because I'm really outside my field.
I was just observing in openldap-2.5.4 the pbkdf2 module tries to install to /usr/local/libexec/openldap despite the
--prefix=/opt/openldap
in the configure on the src root.
I am pretty sure it worked fine on openldap-2.4.* (at the beginning it didn't but then it was fixed).
No, it is a contrib module, it has never been tied to configured. Read the makefile for how to set the prefix correctly.
Regards, Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
On 09/05/21 05:23, Quanah Gibson-Mount wrote:
--On Saturday, May 8, 2021 7:09 PM +0200 Francesco Malvezzi francesco.malvezzi@unimore.it wrote:
really sorry for the lack of details because I'm really outside my field.
I was just observing in openldap-2.5.4 the pbkdf2 module tries to install to /usr/local/libexec/openldap despite the
--prefix=/opt/openldap
in the configure on the src root.
I am pretty sure it worked fine on openldap-2.4.* (at the beginning it didn't but then it was fixed).
No, it is a contrib module, it has never been tied to configured. Read the makefile for how to set the prefix correctly.
thank you for your time: yes you are right.
I got what let assume me wrongly it did work on previous releases (very easy: if you upgrade in-place, from the second time on the old libraries are already in $PREFIX/libexec/openldap and everything works smooth even if new libraries are installed in /usr/local/libexec/openldap),
franz
On Fri, May 07, 2021 at 08:14:52 -0700, Quanah Gibson-Mount wrote:
Your described write traffic pattern is known to cause fragmentation in the database (which causes there to be a large number of free pages). OpenLDAP 2.5 has features to address this, specifically the multival settings. See slapd-mdb(5) for OpenLDAP 2.5.
Quanah
I suppose multival also makes sense for very large single-valued attributes? Do we set olcDbMultival to 0,0 then for such attribute?
Geert
openldap-technical@openldap.org
-
Francesco Malvezzi -
Geert Hendrickx -
Michael Ströder -
Quanah Gibson-Mount -
Zetan Drableg