Hi,
we have ldap servers with many ldap entries (3 million and more) these are replicating with other instances (Master-Master + Read slaves). When we need to restart and ldap due to maintenance or other problems those are taking a lot of time before they serving a requests again. It looks like that in most cases it does a full refresh of the whole tree (printed to the log) but there are is also a timespan where nothing is printed to the log but the slapd process is really busy and consuming more than one core.
We are currently using 2.5.16.1 of openldap but have experienced this with older version too. Any hint about how to improve this?
Thanks, Norbert
--On Wednesday, August 28, 2024 6:05 PM +0200 Norbert openldap@freakix.de wrote:
Hi,
we have ldap servers with many ldap entries (3 million and more) these are replicating with other instances (Master-Master + Read slaves). When we need to restart and ldap due to maintenance or other problems those are taking a lot of time before they serving a requests again. It looks like that in most cases it does a full refresh of the whole tree (printed to the log) but there are is also a timespan where nothing is printed to the log but the slapd process is really busy and consuming more than one core.
We are currently using 2.5.16.1 of openldap but have experienced this with older version too. Any hint about how to improve this?
What type of replication are you using? Also useful to see your replication configuration for both provider and consumer (minus passwords).
--Quanah
Hi,
Am 28.08.24 um 18:40 schrieb Quanah Gibson-Mount:
What type of replication are you using? Also useful to see your replication configuration for both provider and consumer (minus passwords).
It is N-Way Multi provider replication with following configuration
olcSyncrepl: {0}rid=201 provider="ldaps://1.1.1.1" type=refreshAndPersist retry="10 +" searchbase="o=root" bindmethod=simple binddn="cn=syncuser,ou=config,o=root" credentials="xxx" tls_reqcert=allow tls_cacert=/etc/ssl/certs/ca-certificates.crt olcSyncrepl: {1}rid=202 provider="ldaps://1.1.1.2" type=refreshAndPersist retry="10 +" searchbase="o=root" bindmethod=simple binddn="cn=syncuser,ou=config,o=root" credentials="xxx" tls_reqcert=allow tls_cacert=/etc/ssl/certs/ca-certificates.crt olcSyncrepl: {2}rid=101 provider="ldap://1.1.2.1" type=refreshAndPersist retry="10 +" searchbase="o=root" bindmethod=simple binddn="cn=syncuser,ou=config,o=root" credentials="xxx" olcSyncrepl: {3}rid=102 provider="ldap://1.1.2.2" type=refreshAndPersist retry="10 +" searchbase="o=root" bindmethod=simple binddn="cn=syncuser,ou=config,o=root" credentials="xxx"
Regards, Norbert
--On Wednesday, August 28, 2024 9:41 PM +0200 Norbert openldap@freakix.de wrote:
Hi,
Am 28.08.24 um 18:40 schrieb Quanah Gibson-Mount:
What type of replication are you using? Also useful to see your replication configuration for both provider and consumer (minus passwords).
It is N-Way Multi provider replication with following configuration
This is some of the configuration... what is your syncprov overlay configuration?
--Quanah
Am 28.08.24 um 21:19 schrieb Quanah Gibson-Mount:
This is some of the configuration... what is your syncprov overlay configuration?
For the overlay nothing is configured:
dn: olcOverlay={0}syncprov,olcDatabase={1}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: {0}syncprov
Regards, Norbert
--On Thursday, August 29, 2024 11:31 AM +0200 Norbert openldap@freakix.de wrote:
Am 28.08.24 um 21:19 schrieb Quanah Gibson-Mount:
This is some of the configuration... what is your syncprov overlay configuration?
For the overlay nothing is configured:
dn: olcOverlay={0}syncprov,olcDatabase={1}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: {0}syncprov
Hi Norbert,
This explains the behavior you're seeing then. As there is no sessionlog configured, the servers will need to do a full REFRESH after coming back up if any changes have occurred while they were offline as they will be unable to know if any DELETE ops happened from the other servers as they have no stored history.
In OpenLDAP 2.5 and 2.6, you have two options for a sessionlog. You can use the ephemeral in-memory one or use an accesslog database as permanent (up to the point of accesslog cleanup) one. I have recently moved to the second option.
You're also missing a checkpoint configuration on your syncprov overlay, which is also advised.
I personally have moved to using the accesslog based sessionlog, as you can still hit REFRESHes if you need to take your providers down for maintenance and one or more get write ops other ones didn't because they were already shut down.
This is my configuration. For the primary DB, I have both syncprov and accesslog overlays configured, and then I have an accesslog database with its own coresponding syncprov overlay. Theoretically I could switch over to using delta-syncrepl if I ever had a need for that as well. You may need to adjust the maxsize on the accesslog DB given the size of your database, if it has frequent write operations.
dn: olcOverlay={0}syncprov,olcDatabase={1}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: {0}syncprov olcSpCheckpoint: 500 15 olcSpSessionlogSource: cn=accesslog
dn: olcOverlay={1}accesslog,olcDatabase={1}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcAccessLogConfig olcOverlay: {1}accesslog olcAccessLogDB: cn=accesslog olcAccessLogOps: writes olcAccessLogPurge: 7+00:00 1+00:00 olcAccessLogSuccess: TRUE
dn: olcDatabase={2}mdb,cn=config objectClass: olcDatabaseConfig objectClass: olcMdbConfig olcDatabase: {2}mdb olcDbDirectory: /var/lib/ldap/accesslog olcSuffix: cn=accesslog olcAcces: .... olcLimits: ... olcDbCheckpoint: 1024 5 olcDbIndex: default eq olcDbIndex: entryCSN olcDbIndex: objectClass olcDbIndex: reqDN olcDbIndex: reqEnd olcDbIndex: reqResult olcDbIndex: reqStart olcDbMaxSize: 32212254720 olcDBMode: 0600
dn: olcOverlay={0}syncprov,olcDatabase={2}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: {0}syncprov olcSpNoPresent: TRUE olcSpReloadHint: TRUE
Regards, Quanah
openldap-technical@openldap.org
-
Norbert -
Quanah Gibson-Mount