On May 21, 2019, at 7:22 PM, Quanah Gibson-Mount quanah@symas.com wrote:
Additionally, without the specific syncrepl configurations from the consumers (do they listen to both? do they only listen to one?) there's additional levels of variability. If they listen to both masters, then it wouldn't be uncommon for them to receive the change from master A and ignore the change from master B.
They are listening to both masters. The problem is not so much the “error” message in the log as the fact that they haven’t processed the change from either master and so the objects are out of sync.
With so much redacted from both the config and the change operations, it makes it fairly difficult to comment further. For example, you could be a victim of ITS#8990, but you would have to provide unredacted results of the two sets of changes as they appear in the accesslog DBs from both masters (In ITS#8990, a change propagates correctly between to MMR servers, but is written incorrectly into the accesslog DB of one of the masters).
I have just read ITS#8990. While that is a concern for us, that did not come into play in the particular example that I cited (values were being added to an attribute).
Having made that changes that you recommended, the config now looks like:
# # See slapd-config(5) for details on configuration options. # This file should NOT be world readable. # dn: cn=config objectClass: olcGlobal cn: config # # Set the serverId giving explict producer URL’s — these lines only on producers # olcServerID: 1 ldap://producer1.umd.edu olcServerID: 2 ldap://producer2.umd.edu # # # Set the args & pid files # olcArgsFile: /var/run/openldap/slapd.args olcPidFile: /var/run/openldap/slapd.pid # # Set the idle timeout # olcIdleTimeout: 300 # # Sample security restrictions # Require integrity protection (prevent hijacking) # Require 112-bit (3DES or better) encryption for updates # Require 64-bit encryption for simple bind #olcSecurity: ssf=1 update_ssf=112 simple_bind=64 # olcSaslSecProps: noanonymous,passcred # # TLS configuration olcTLSCACertificatePath: /local/ssl/certs/ca olcTLSCACertificateFile: /etc/openldap/certs/slapd-cacert.pem olcTLSCertificateFile: /etc/openldap/certs/slapd-cert.pem olcTLSCertificateKeyFile: /etc/openldap/certs/slapd-key.pem olcTLSVerifyClient: never # olcRootDSE: /etc/openldap/root_dse.ldif # # # For ProxyAuthorization olcAuthzPolicy: to # # # Threading configuration olcToolThreads: 2 # olcLogLevel: stats sync
# # Load dynamic backend modules: # dn: cn=module,cn=config objectClass: olcModuleList cn: module olcModulepath: /usr/lib64/openldap olcModuleload: accesslog.la olcModuleload: auditlog.la #olcModuleload: back_dnssrv.la #olcModuleload: back_ldap.la olcModuleload: back_mdb.la #olcModuleload: back_meta.la #olcModuleload: back_null.la #olcModuleload: back_passwd.la #olcModuleload: back_relay.la #olcModuleload: back_shell.la #olcModuleload: back_sock.la #olcModuleload: collect.la #olcModuleload: constraint.la #olcModuleload: dds.la #olcModuleload: deref.la #olcModuleload: dyngroup.la olcModuleload: dynlist.la olcModuleload: memberof.la #olcModuleload: pcache.la olcModuleload: ppolicy.la #olcModuleload: refint.la #olcModuleload: retcode.la #olcModuleload: rwm.la #olcModuleload: seqmod.la #olcModuleload: smbk5pwd.la #olcModuleload: sssvlv.la olcModuleload: syncprov.la #olcModuleload: translucent.la #olcModuleload: unique.la #olcModuleload: valsort.la # olcModuleload: pw-sha2.la
dn: cn=schema,cn=config objectClass: olcSchemaConfig cn: schema
include: file:///etc/openldap/schema/core.ldif include: file:///etc/openldap/schema/cosine.ldif include: file:///etc/openldap/schema/inetorgperson.ldif include: file:///etc/openldap/schema/nis.ldif include: file:///etc/openldap/schema/dyngroup.ldif include: file:///etc/openldap/schema/ppolicy.ldif include: file:///etc/openldap/schema/other.ldif include: file:///etc/openldap/schema/eduPerson-201602.ldif include: file:///etc/openldap/schema/umPerson.ldif include: file:///etc/openldap/schema/umGeneric.ldif include: file:///etc/openldap/schema/attributeSet.ldif
# Frontend settings # dn: olcDatabase=frontend,cn=config objectClass: olcDatabaseConfig objectClass: olcFrontendConfig olcDatabase: frontend # # Sample global access control policy: # Root DSE: allow anyone to read it # Subschema (sub)entry DSE: allow anyone to read it # Other DSEs: # Allow self write access # Allow authenticated users read access # Allow anonymous users to authenticate # #olcAccess: to dn.base="" by * read #olcAccess: to dn.base="cn=Subschema" by * read #olcAccess: to * # by self write # by users read # by anonymous auth # # if no access controls are present, the default policy # allows anyone and everyone to read anything but restricts # updates to rootdn. (e.g., "access to * by * read") # # rootdn can always read and write EVERYTHING! # olcSizeLimit: 50 olcTimeLimit: 240 # # INSERT-ACL frontend
# Config settings # dn: olcDatabase={0}config,cn=config objectClass: olcDatabaseConfig olcDatabase: {0}config olcAccess: {0}to dn.sub=cn=config by group.exact=cn=directory-admin,cn=groups,ou=ldap,dc=umd,dc=edu read by * none olcAddContentAcl: TRUE olcLastMod: TRUE olcMaxDerefDepth: 15 olcReadOnly: FALSE olcSyncUseSubentry: FALSE olcMonitoring: FALSE olcRootDN: cn=admin,cn=config olcRootPW: ADMIN_PW
dn: olcDatabase={1}monitor,cn=config objectClass: olcDatabaseConfig objectClass: olcMonitorConfig olcDatabase: {1}monitor olcAccess: {0}to dn.sub=cn=monitor,cn=config by dn=“cn=admin,dc=umd,dc=edu" read by group.exact=cn=directory-admin,cn=groups,ou=ldap,dc=umd,dc=edu read olcLimits: group/groupOfNames/member="cn=directory-admin,cn=groups,ou=ldap,dc=umd,dc=edu" size=unlimited time=unlimited
####################################################################### # LMDB database definitions ####################################################################### # dn: olcDatabase={2}mdb,cn=config objectClass: olcDatabaseConfig objectClass: olcMdbConfig olcDatabase: {2}mdb olcSuffix: dc=umd,dc=edu olcRootDN: cn=admin,dc=umd,dc=edu # Cleartext passwords, especially for the rootdn, should # be avoided. See slappasswd(8) and slapd-config(5) for details. # Use of strong authentication encouraged. olcRootPW: ADMIN_PW # The database directory MUST exist prior to running slapd AND # should only be accessible by the slapd and slap tools. # Mode 700 recommended. olcDbDirectory: /var/lib/ldap/umd-edu olcDbMaxReaders: 126 olcDbSearchStack: 16 olcDbMaxSize: 10000000000 # Indices to maintain olcDbIndex: cn eq,approx,sub olcDbIndex: eduPersonPrincipalName eq olcDbIndex: employeeNumber eq olcDbIndex: givenName eq,sub olcDbIndex: mail pres,eq olcDbIndex: member eq olcDbIndex: memberOf eq olcDbIndex: objectClass pres,eq olcDbIndex: ou pres,eq,sub olcDbIndex: sn eq,approx,sub olcDbIndex: uid eq,sub olcDbIndex: umAccountType eq,sub olcDbIndex: umAdminId eq olcDbIndex: umAffiliate eq olcDbIndex: umAlternateMail eq olcDbIndex: umEmployee eq olcDbIndex: umExpirationdate eq olcDbIndex: umGroup pres,eq olcDbIndex: umId eq olcDbIndex: umInactiveDate pres,eq olcDbIndex: umInstitution eq olcDbIndex: umInstitutionActive eq olcDbIndex: umLibraryBarcode eq olcDbIndex: umMailAlias pres,eq olcDbIndex: umMailFwd pres,eq olcDbIndex: umNameComponent eq,sub olcDbIndex: umNickname eq,sub olcDbIndex: umOwnerId eq olcDbIndex: umRegcourse eq olcDbIndex: umRegStatus eq olcDbIndex: umServices eq olcDbIndex: umServiceStatus pres,eq olcDbIndex: umStudentStatus pres,eq # Indices for syncrepl olcDbIndex: entryUUID eq olcDbIndex: entryCSN eq # # Search Limits # olcLimits: group/groupOfNames/member="cn=directory-admin,cn=groups,ou=ldap,dc=umd,dc=edu" size=unlimited time=unlimited olcLimits: group/groupOfNames/member="cn=replciation-auth,cn=groups,ou=ldap,dc=umd,dc=edu" size=unlimited time=unlimited olcLimits: group/groupOfNames/member="cn=search-unlimited,cn=groups,ou=ldap,dc=umd,dc=edu" size=5000 size.pr=5000 size.prtotal=unlimited time=unlimited olcLimits: group/groupOfNames/member="cn=search-limit-3000,cn=groups,ou=ldap,dc=umd,dc=edu" size=3000 time=300 olcLimits: group/groupOfNames/member="cn=search-limit-2000,cn=groups,ou=ldap,dc=umd,dc=edu" size=2000 time=300 olcLimits: group/groupOfNames/member="cn=search-limit-5,cn=groups,ou=ldap,dc=umd,dc=edu" size=5 time=300 olcLimits: dn.subtree="cn=auth,ou=ldap,dc=umd,dc=edu" size=1000 time=300 olcLimits: dn.subtree="ou=people,dc=umd,dc=edu" size=100 size.prtotal=disabled time.soft=30 time.hard=240 olcLimits: anonymous size=50 time.soft=30 size.prtotal=disabled time.hard=240 # # INSERT-ACL umd # # # Replication Configs # # The rid=1 block is not on producer1 # The rid=2 block is not on producer2 # olcSyncrepl: rid=1 provider=ldaps://producer1.umd.edu type=refreshAndPersist scope=sub searchbase="dc=umd,dc=edu" bindmethod=simple binddn="uid=sync.consumer,dc=umd,dc=edu" credentials=“SYNC_PW" schemachecking=off retry="5 5 300 +" interval=00:00:00:30 timeout=10 keepalive="240:10:30" tls_cacert=/etc/openldap/certs/slapd-cacert.pem tls_reqcert=never logbase="cn=accesslog" syncdata=accesslog logfilter="(&(objectClass=auditWriteObject)(reqResult=0))" olcSyncrepl: rid=2 provider=ldaps://useast1c-openldap-eng-main1.it-eng-aaa.aws.umd.edu type=refreshAndPersist scope=sub searchbase="dc=umd,dc=edu" bindmethod=simple binddn="uid=sync.consumer,dc=umd,dc=edu" credentials="SYNC_PW" schemachecking=off retry="5 5 300 +" interval=00:00:00:30 timeout=10 keepalive="240:10:30" tls_cacert=/etc/openldap/certs/slapd-cacert.pem tls_reqcert=never logbase="cn=accesslog" syncdata=accesslog logfilter="(&(objectClass=auditWriteObject)(reqResult=0))” # # These lines only on consumers # olcUpdateRef: ldaps://master.directory.umd.edu olcMirrorMode: FALSE # # This line only on producers # olcMirrorMode: TRUE
dn: olcOverlay={1}accesslog,olcDatabase={2}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcAccessLogConfig olcOverlay: {1}accesslog olcAccessLogDB: cn=accesslog olcAccessLogOps: writes olcAccessLogSuccess: TRUE olcAccessLogPurge: 5+00:00 1+00:00
dn: olcOverlay={2}dynlist,olcDatabase={2}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcDynamicList olcOverlay: {2}dynlist olcDlAttrSet: {0}groupOfURLs memberURL member olcDlAttrSet: {1}umGroupOfURLs memberURL member
dn: olcOverlay={3}ppolicy,olcDatabase={2}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcPPolicyConfig olcOverlay: {3}ppolicy olcPPolicyDefault: cn=default,ou=policies,ou=ldap,dc=umd,dc=edu olcPPolicyHashCleartext: TRUE olcPPolicyUseLockout: FALSE olcPPolicyForwardUpdates: FALSE
dn: olcOverlay={5}memberof,olcDatabase={2}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcMemberOf olcOverlay: {5}memberof olcMemberOfDangling: ignore olcMemberOfRefInt: FALSE olcMemberOfGroupOC: groupOfNames olcMemberOfMemberAD: member olcMemberOfMemberOfAD: memberOf
dn: olcOverlay={6}memberof,olcDatabase={2}mdb,cn=config objectClass: olcOverlayConfig objectClass: olcMemberOf olcOverlay: {6}memberof olcMemberOfDangling: ignore olcMemberOfRefInt: FALSE olcMemberOfGroupOC: umGroupOfNames olcMemberOfMemberAD: member olcMemberOfMemberOfAD: memberOf
dn: olcDatabase={3}mdb,cn=config objectClass: olcDatabaseConfig objectClass: olcMdbConfig olcDatabase: {3}mdb olcSuffix: cn=accesslog olcDbDirectory: /var/lib/ldap/accesslog olcDbMaxReaders: 126 olcDbSearchStack: 16 olcDbMaxSize: 10000000000 olcRootDN: cn=admin,cn=accesslog olcRootPW: ADMIN_PW # # INSERT-ACL accesslog # olcLimits: * size=unlimited time=unlimited olcDbIndex: objectClass eq olcDbIndex: reqStart eq olcDbIndex: reqType eq olcDbIndex: reqAuthzId eq olcDbIndex: reqDN eq olcDbIndex: reqEnd eq olcDbIndex: reqResult eq # Indices for syncrepl olcDbIndex: entryUUID eq olcDbIndex: entryCSN eq
#### END ####
I have omitted the voluminous access rules (sometimes being at a university lead to really squirrelly policies) but I don’t believe that they are part of the problem since replication will work completely fine for a while before getting sideways on some, but not necessarily all, of the consumers.
// John Pfeifer Division of Information Technology University of Maryland, College Park