Hi!
Trying to convert my syncrepl configuration to a delta syncrepl configuration, the slave runs out of accesslog space during initial sync. For the main database about 30MB should be enough, but I configured 50MB, also for the accesslog files. It seems sync continues even if accesslog periodically complains like this:
slapd[13951]: => mdb_idl_insert_keys: c_put id failed: MDB_MAP_FULL: Environment mapsize limit reached (-30792) slapd[13951]: conn=-1 op=0 accesslog_response: got result 0x50 adding log entry reqStart=20250314110032.000187Z,cn=changelog-1
From stat I guess that accesslog needs at least three times as much space as the main database does: # stat /var/lib/ldap/data.mdb /var/lib/ldap/changelog-1/data.mdb File: /var/lib/ldap/data.mdb Size: 22888448 Blocks: 44704 IO Block: 4096 regular file Device: 33h/51d Inode: 259467 Links: 1 Access: (0600/-rw-------) Uid: ( 76/ ldap) Gid: ( 70/ ldap) Access: 2025-03-14 11:57:46.303713632 +0100 Modify: 2025-03-14 12:01:08.417328586 +0100 Change: 2025-03-14 12:01:08.417328586 +0100 Birth: 2025-03-14 11:57:46.303713632 +0100 File: /var/lib/ldap/changelog-1/data.mdb Size: 52424704 Blocks: 102392 IO Block: 4096 regular file Device: 33h/51d Inode: 259471 Links: 1 Access: (0600/-rw-------) Uid: ( 76/ ldap) Gid: ( 70/ ldap) Access: 2025-03-14 11:57:46.323713395 +0100 Modify: 2025-03-14 12:01:08.425328492 +0100 Change: 2025-03-14 12:01:08.425328492 +0100 Birth: 2025-03-14 11:57:46.323713395 +0100
Shouldn't (assuming the sync happens entry-by-entry) the accesslog have about the same size as the main database after initial sync (when all databases were empty)? When setting the size of the accesslog database to twice the size of the database it succeeded which stats like this:
# stat /var/lib/ldap/data.mdb /var/lib/ldap/changelog-1/data.mdb File: /var/lib/ldap/data.mdb Size: 22888448 Blocks: 44704 IO Block: 4096 regular file Device: 33h/51d Inode: 259514 Links: 1 Access: (0600/-rw-------) Uid: ( 76/ ldap) Gid: ( 70/ ldap) Access: 2025-03-14 12:39:01.278545521 +0100 Modify: 2025-03-14 12:45:03.558274733 +0100 Change: 2025-03-14 12:45:03.558274733 +0100 Birth: 2025-03-14 12:39:01.278545521 +0100 File: /var/lib/ldap/changelog-1/data.mdb Size: 89657344 Blocks: 175112 IO Block: 4096 regular file Device: 33h/51d Inode: 259518 Links: 1 Access: (0600/-rw-------) Uid: ( 76/ ldap) Gid: ( 70/ ldap) Access: 2025-03-14 12:39:01.298545286 +0100 Modify: 2025-03-14 12:45:03.566274639 +0100 Change: 2025-03-14 12:45:03.566274639 +0100 Birth: 2025-03-14 12:39:01.298545286 +0100
So the blocks in the accesslog are roughly four times (3.91 ) as much as in the original database. Is there any doc making statements about the proportion? (I'm a fan of "small is beautiful" instead of "bigger is better")
Kind regards, Ulrich Windl
--On Friday, March 14, 2025 12:51 PM +0000 "Windl, Ulrich" u.windl@ukr.de wrote:
Hi!
Trying to convert my syncrepl configuration to a delta syncrepl configuration, the slave runs out of accesslog space during initial sync.
For the main database about 30MB should be enough, but I configured 50MB, also for the accesslog files.
There's zero correlation between the size of the main DB and the size of the accesslog DB.
--Quanah
Hi!
Why should the number of entries in (size of) the main database not be correlated with the size of the accesslog regarding the initial sync? The more entrties in the main DB, the more entires in the accesslog, I guess. As a matter of fact I suspect the unexpected size of my accesslog is due to the number of indices being configured (I have configured a few more than absolutely required). Maybe the question is: What is the minimum size per entry for an "eq" index? Is it two times 64 bit, or is it even more?
Kind regards, Ulrich Windl
-----Original Message----- From: Quanah Gibson-Mount quanah@fast-mail.org Sent: Friday, March 14, 2025 10:31 PM To: Windl, Ulrich u.windl@ukr.de; openldap-technical@openldap.org Subject: [EXT] Re: accesslog seemingly needs a lot of space
--On Friday, March 14, 2025 12:51 PM +0000 "Windl, Ulrich" u.windl@ukr.de wrote:
Hi!
Trying to convert my syncrepl configuration to a delta syncrepl configuration, the slave runs out of accesslog space during initial sync.
For the main database about 30MB should be enough, but I configured
50MB,
also for the accesslog files.
There's zero correlation between the size of the main DB and the size of the accesslog DB.
--Quanah
On Mon, Mar 17, 2025 at 08:44:55AM +0000, Windl, Ulrich wrote:
Hi!
Why should the number of entries in (size of) the main database not be correlated with the size of the accesslog regarding the initial sync? The more entrties in the main DB, the more entires in the accesslog, I guess.
Hi Ulrich, the size of an accesslog entry has to be considerably larger than the size of the original entry, for starters it has to communicate a lot of additional information/metadata about the operation. And that's assuming you only enabled accesslog to log the bare minimum needed for delta-syncrepl use.
If you enabled any more (e.g. logsuccess false), enabling accesslog and doing a live bootstrap would never strike me as a good combination. In that case I would either slapadd from a recent backup or configure accesslog afterwards.
As a matter of fact I suspect the unexpected size of my accesslog is due to the number of indices being configured (I have configured a few more than absolutely required).
Unless you created any substring indexes or went really overboard, I doubt it should cost that much.
Regards,
Ondřej,
I'm aware that doing the initial sync via accesslog is not the best idea, but first I wanted to check whether it works at all, and it also seems to be a good indicator whether the size of the databases is large enough (I'm new to mdb, and unfortunately SLES does not ship the mdb tools with OpenLDAP). And I'm using 11 equality only indices.
Mit freundlichen Grüßen Ulrich Windl
-----Original Message----- From: Ondřej Kuzník ondra@mistotebe.net Sent: Monday, March 17, 2025 1:29 PM To: Windl, Ulrich u.windl@ukr.de Cc: Quanah Gibson-Mount quanah@fast-mail.org; openldap- technical@openldap.org Subject: [EXT] Re: Re: accesslog seemingly needs a lot of space
On Mon, Mar 17, 2025 at 08:44:55AM +0000, Windl, Ulrich wrote:
Hi!
Why should the number of entries in (size of) the main database not be correlated with the size of the accesslog regarding the initial sync? The more entrties in the main DB, the more entires in the accesslog, I guess.
Hi Ulrich, the size of an accesslog entry has to be considerably larger than the size of the original entry, for starters it has to communicate a lot of additional information/metadata about the operation. And that's assuming you only enabled accesslog to log the bare minimum needed for delta-syncrepl use.
If you enabled any more (e.g. logsuccess false), enabling accesslog and doing a live bootstrap would never strike me as a good combination. In that case I would either slapadd from a recent backup or configure accesslog afterwards.
As a matter of fact I suspect the unexpected size of my accesslog is due to the number of indices being configured (I have configured a few more than absolutely required).
Unless you created any substring indexes or went really overboard, I doubt it should cost that much.
Regards,
-- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP
--On Monday, March 17, 2025 3:17 PM +0000 "Windl, Ulrich" u.windl@ukr.de wrote:
Ondřej,
I'm aware that doing the initial sync via accesslog is not the best idea, but first I wanted to check whether it works at all, and it also seems to be a good indicator whether the size of the databases is large enough (I'm new to mdb, and unfortunately SLES does not ship the mdb tools with OpenLDAP). And I'm using 11 equality only indices.
Indices on the main DB have no correlation to the size of the accesslog DB, which has its own set of required indices. The reason why the accesslog tends to be *larger* than the main DB is it records all change traffic (at least with a minimum of just delta-syncrepl and/or session log). Now, if you never have changes, or you have very few changes in your environment, and you had a very large main database, then it's possible the accesslog DB would be smaller than the main DB. Additionally, the accesslog is generally configured to hold all changes over a span of time (I've generally done 1-2 weeks).
So the size of the accesslog DB is determined by a number of things:
* What operations you specifically have logged into it (changes (delta-syncrepl) or more) * How long it's configured to keep those changes in the accesslog DB * The rate at which recorded operations occur on your primary database
--Quanah
-----Original Message----- From: Quanah Gibson-Mount quanah@fast-mail.org Sent: Monday, March 17, 2025 6:08 PM To: Windl, Ulrich u.windl@ukr.de; Ondřej Kuzník ondra@mistotebe.net Cc: openldap-technical@openldap.org Subject: [EXT] RE: Re: Re: accesslog seemingly needs a lot of space
--On Monday, March 17, 2025 3:17 PM +0000 "Windl, Ulrich" u.windl@ukr.de wrote:
Ondřej,
I'm aware that doing the initial sync via accesslog is not the best idea, but first I wanted to check whether it works at all, and it also seems to be a good indicator whether the size of the databases is large enough (I'm new to mdb, and unfortunately SLES does not ship the mdb tools with OpenLDAP). And I'm using 11 equality only indices.
Indices on the main DB have no correlation to the size of the accesslog DB, which has its own set of required indices. The reason why the accesslog
[Windl, Ulrich] Of course those 11 indoces are on the accesslog database.
tends to be *larger* than the main DB is it records all change traffic (at least with a minimum of just delta-syncrepl and/or session log). Now, if you never have changes, or you have very few changes in your environment, and you had a very large main database, then it's possible the accesslog DB would be smaller than the main DB. Additionally, the accesslog is generally configured to hold all changes over a span of time (I've generally done 1-2 weeks).
So the size of the accesslog DB is determined by a number of things:
- What operations you specifically have logged into it (changes
(delta-syncrepl) or more)
- How long it's configured to keep those changes in the accesslog DB
- The rate at which recorded operations occur on your primary database
[Windl, Ulrich] Thanks for explaining!
--Quanah
--On Tuesday, March 18, 2025 9:59 AM +0000 "Windl, Ulrich" u.windl@ukr.de wrote:
[Windl, Ulrich] Of course those 11 indoces are on the accesslog database.
I'm not sure what 11 indices you are referring to. An accesslog database only requires the following indices (all eq):
entryCSN objectClass reqDN reqEnd reqResult reqStart
For a total of 6 indices.
--Quanah
-----Original Message----- From: Quanah Gibson-Mount quanah@fast-mail.org Sent: Tuesday, March 18, 2025 4:51 PM To: Windl, Ulrich u.windl@ukr.de; Ondřej Kuzník ondra@mistotebe.net Cc: openldap-technical@openldap.org Subject: [EXT] RE: RE: Re: Re: accesslog seemingly needs a lot of space
--On Tuesday, March 18, 2025 9:59 AM +0000 "Windl, Ulrich" u.windl@ukr.de wrote:
[Windl, Ulrich] Of course those 11 indoces are on the accesslog database.
I'm not sure what 11 indices you are referring to. An accesslog database only requires the following indices (all eq):
entryCSN objectClass reqDN reqEnd reqResult reqStart
For a total of 6 indices.
As I read the accesslog with separate programs, too, I'm using olcDbIndex: entryCSN eq olcDbIndex: objectclass eq olcDbIndex: reqStart eq olcDbIndex: reqEnd eq olcDbIndex: entryUUID eq olcDbIndex: reqResult eq olcDbIndex: reqDN eq olcDbIndex: reqMod sub olcDbIndex: reqType eq olcDbIndex: reqAuthzID eq olcDbIndex: modifyTimestamp eq
but as the actual accesslog only contains successful writes, I could drop reqResult.
Kind regards, Ulrich Windl
openldap-technical@openldap.org
-
Ondřej Kuzník -
Quanah Gibson-Mount -
Windl, Ulrich