Hi Quanah, We have 80 GB RAM on the system and 300 GB SSD disk allocated for the directory.
Regards Bhanush
On Wed, Jan 18, 2023 at 7:20 PM Quanah Gibson-Mount quanah@fast-mail.org wrote:
--On Wednesday, January 18, 2023 6:31 PM +0530 Bhanush Mehta bhanush.mehta@flipkart.com wrote:
Hi Quanah
The current mdb data file is GB on disk. We are using XFS, we tried with ext2 and ext4, we saw the same behavior for slow mods.
2376455:/var/lib/ldap$ sudo du -sh data.mdb 11G data.mdb 2376455:/var/lib/ldap$ sudo du -s data.mdb 10519776 data.mdb
The dump from slapcat is 200 MB approx. Whenever we add it to a new instance, it balloons up to 10 GB and then stays around the same size. local-backups$ du -sh ldapdump202301180000.ldif204M ldapdump202301180000.ldif
Can DB size be a reason for slow mods? And, how do we investigate the slow mod further?
One is a flat text file export of the database (.ldif)
The other is a set of binary data based off of the flat text file export, but includes things such as indices. You cannot correlate the two size wise.
If your binary database after an import with slapadd -q is ~11GB, than that's the "real" size of your database.
How much memory do you have on the system you're using?
I would note that it is not advised to use XFS with back-mdb.
--Quanah
--On Wednesday, January 18, 2023 7:30 PM +0530 Bhanush Mehta bhanush.mehta@flipkart.com wrote:
Hi Quanah, We have 80 GB RAM on the system and 300 GB SSD disk allocated for the directory.
I'm pruning the huge list of people from flipkart, it's not really appropriate for the list, you can send them any answers you think are relevant or point them at the list archives.
I obviously don't know how your data is actually in use in your environment, but from the behavior you're describing you have multi-valued attributes that have many values. There are fixes to help with write operations in this case, but it'll require you to upgrade to a supported release stream, OpenLDAP 2.5 at a minimum to implement them. If you're able to confirm my assumptions, then I would recommend the following:
* Move any attribute that has a large number of values to a sub-database with multival. In the back-mdb DATABASE configuration: olcDbMultival: default 100,10 * Use a sorted index for attributes that have many values. This is done in the FRONTEND configuration via olcSortvals. It will require you to know what attribute(s) you're using that have large value sets. If you're not able to determine this by examining your data (although it should be fairly apparent) you can get some idea from the mdb_stat command, such as: mdb_stat -a /path/to/database, i.e., mdb_stat -a var/ldap/db/ . Then look at the number of 'Entries' a given indexed attribute is shown as having. For example, the database I work on has ~110k total objects in it. One of the attributes we have that has many values is 'uniqueMember'. In this case, the mdb_stat -a output shows that for the uniqueMember index database has 4.15 million entries, which indicates that it is a multi-valued attribute that is used in database entries with many values.
Those two options used together can significantly reduce the amount of time it takes to update entries where multi-valued attributes are being modified.
Other things that could help overall performance but are not related to this specific issue:
* Enable 64-bit index hashes (olcIndexHash64: TRUE) * Configure a back-mdb backend section, and increase the default IDL exponent (olcBkMdbIdlExp). I use a value of 18 for the environment I work with.
Regards, Quanah
openldap-technical@openldap.org