Hi ,

I am trying to configure and use the openldap 2-WAY Multimaster Replication setup for high availability(HA), but the HA solution used to freeze very often. Using ansible tool I have automated the installation and configuration steps. I tried with manual steps too before using the ansible for the actual deployment

Following are the env. details used for installation and configuration

Two nodes ldap-test1 and ldap-test2 are running with base OS RHEL7

LDAP rpms installed:

openldap-clients-2.4.39-6.el7.x86_64 openldap-servers-2.4.39-6.el7.x86_64 openldap-2.4.39-6.el7.x86_64

Configuration steps:

1. Added the nis, cosine schemas ldapadd -Y EXTERNAL -H ldapi:/// -D "cn=config" -f cosine.ldif ldapadd -Y EXTERNAL -H ldapi:/// -D "cn=config" -f nis.ldif

2. Send the new global configuration settings to slapd 'ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/global_config.ldif

# cat global_config.ldif

dn: cn=module{0},cn=config objectClass: olcModuleList cn: module{0} olcModuleLoad: syncprov

dn: olcOverlay=syncprov,olcDatabase={0}config,cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: syncprov

dn: olcDatabase={2}hdb,cn=config changetype: modify replace: olcSuffix olcSuffix: dc=example,dc=com

dn: olcDatabase={2}hdb,cn=config changetype: modify replace: olcRootDN olcRootDN: cn=Manager,dc=example,dc=com

dn: olcDatabase={2}hdb,cn=config changetype: modify replace: olcRootPW olcRootPW: {SSHA}vAp3OToPGMYnWEkh+76RJEVyfCIdnsDg

dn: cn=config changetype: modify replace: olcTLSCACertificateFile olcTLSCACertificateFile: /etc/openldap/certs/cacert.pem

dn: cn=config changetype: modify replace: olcTLSCertificateFile olcTLSCertificateFile: /etc/openldap/certs/slapdcert.pem

dn: cn=config changetype: modify replace: olcTLSCertificateKeyFile olcTLSCertificateKeyFile: /etc/openldap/certs/slapdkey.pem

dn: cn=config changetype: modify replace: olcLogLevel olcLogLevel: -1

dn: olcDatabase={1}monitor,cn=config changetype: modify replace: olcAccess olcAccess: {0}to * by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" read by dn.base="cn=Manager,dc=example,dc=com" read by * none

dn: cn=config changeType: modify add: olcServerID olcServerID: 1

dn: olcDatabase={0}config,cn=config changetype: modify add: olcRootDN olcRootDN: cn=admin,cn=config

dn: cn=config changetype: modify

dn: olcDatabase={0}config,cn=config changetype: modify replace: olcRootPW olcRootPW: {SSHA}vAp3OToPGMYnWEkh+76RJEVyfCIdnsDg

dn: cn=config changetype: modify replace: olcServerID olcServerID: 1 ldaps://ldap-test1 olcServerID: 2 ldaps://ldap-test2

3. Load base.ldif ldapadd -x -w redhat7 -D cn=Manager,dc=example,dc=com -f /etc/openldap/base.ldif

# cat base.ldif dn: dc=example,dc=com dc: example objectClass: top objectClass: domain

dn: ou=People,dc=example,dc=com ou: People objectClass: top objectClass: organizationalUnit

dn: ou=Group,dc=example,dc=com ou: Group objectClass: top objectClass: organizationalUnit

4. Load hdb_config.ldif ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/hdb_config.ldif

#cat hdb_config.ldif

dn: olcDatabase={2}hdb,cn=config changetype: modify replace: olcSuffix olcSuffix: dc=example,dc=com

dn: olcDatabase={2}hdb,cn=config changetype: modify replace: olcRootDN olcRootDN: cn=Manager,dc=example,dc=com

dn: olcDatabase={2}hdb,cn=config changetype: modify replace: olcRootPW olcRootPW: {{ ldap_root_password.stdout }}

dn: olcDatabase={2}hdb,cn=config changetype: modify replace: olcDbIndex olcDbIndex: entryCSN eq olcDbIndex: entryUUID eq

5. Load replication.ldif ldapadd -Y EXTERNAL -H ldapi:/// -f /etc/openldap/replication.ldif

cat replication.ldif

dn: olcOverlay=syncprov,olcDatabase={2}hdb,cn=config changetype: add objectClass: olcOverlayConfig objectClass: olcSyncProvConfig olcOverlay: syncprov

dn: olcDatabase={2}hdb,cn=config changetype: modify replace: olcSyncRepl olcSyncRepl: rid=101 provider=ldaps://ldap-test1 binddn="cn=Manager,dc=example,dc=com" bindmethod=simple credentials=redhat7 searchbase="dc=example,dc=com" type=refreshAndPersist interval=00:00:00:10 retry="5 5 300 5" timeout=1 olcSyncRepl: rid=102 provider=ldaps://ldap-test2 binddn="cn=Manager,dc=example,dc=com" bindmethod=simple credentials=redhat7 searchbase="dc=example,dc=com" type=refreshAndPersist interval=00:00:00:10 retry="5 5 300 5" timeout=1 - replace: olcMirrorMode olcMirrorMode: TRUE

dn: olcDatabase={0}config,cn=config changetype: modify replace: olcSyncRepl olcSyncRepl: rid=101 provider=ldaps://ldap-test1 binddn="cn=admin,cn=config" bindmethod=simple credentials=redhat7 searchbase="cn=config" type=refreshAndPersist retry="5 5 300 5" timeout=1 olcSyncRepl: rid=102 provider=ldaps://ldap-test2 binddn="cn=admin,cn=config" bindmethod=simple credentials=redhat7 searchbase="cn=config" type=refreshAndPersist retry="5 5 300 5" timeout=1 - replace: olcMirrorMode olcMirrorMode: TRUE

Above configuration steps 1 to 4 are executed parallely on both nodes , only step 5 ie. replication.ldif was executed serially one node after the other because parallel execution of step 5 causing the solution to freeze

Parallel execution on both nodes : 1. Added the nis, cosine schemas 2. Send the new global configuration settings to slapd 3. Load base.ldif

4. Load hdb_config.ldif , executed on any one of two nodes assuming that content will replicated automatically on other node once the servers are replicated

Serial execution : Executed on both nodes one after the other 5. Load replication.ldif

Sometimes LDAP replication is causing the solution to freeze and it may hung during the deployment , after deployment or while executing some basic ldap operations like ldapadd/modify/delete etc

1. First thing I would like to know Is there any specific order we need to follow to avoid solution freeze or ldapadd command hung with two nodes configuring parallely ?

2. Anything wrong with the configuration attributes used ? If so what attributes I need to add/update to avoid the command/service hung during configuration or after the deployment ?

3. To verify the high availability I used to stop ldap service any one of the two nodes and send the ldap requests to other node, but sometimes the restart of the service not bringing back the two nodes in Sync. And for replication I am verifying based on number of connections established between two servers ( min. 4 , 2 for config replication and 2 for db replication ) And unittest to verify db replication by creating an ldap user on one node and search & delete operations on the other node.

4. Whenever the basic ldap commands add/modify etc hung on any one node , I used to restart the slapd service on the corresponding node, Is it the right way ? HA solution is working fine by configuring above steps in the given order, ldap services are restarted and in Sync, but sometimes the solution used to freeze when executing basic ldapadd commond on any one node

There are no specific error messages in the logs to understand the reason for hung. May 15 22:20:17 ldap-test2 slapd[8538]: daemon: epoll: listen=7 active_threads=0 tvp=zero May 15 22:20:17 ldap-test2 slapd[8538]: daemon: epoll: listen=8 active_threads=0 tvp=zero

Always a single restart is also not helping to bring the servers back to sync and available, sometimes the number of tcp connections are 3 instead of required 4 connections

[root@ldap-test1 openldap]# netstat -a | grep ldaps tcp 0 0 ldap-test1:ldaps 0.0.0.0:* LISTEN tcp 0 0 ldap-test1:34854 ldap-test2:ldaps ESTABLISHED tcp 0 0 ldap-test1:ldaps ldap-test2:48493 ESTABLISHED tcp 0 0 ldap-test1:34856 ldap-test2:ldaps ESTABLISHED

Used to restart couple of times until 4 tcp connections are established and servers are replicated properly

5. Can someone please help me to get rid of this situation ? I didn't find any clue in the archived messages etc for similar kind of problems.

Thanks & Regards, Shashi