Hello list,
I have an OpenLDAP provider/consumer installation on two SLES10 systems. One is set up as a provider LDAP, the second is a consumer LDAP using "refreshOnly" synrepl synchronization. The LDAP provides user information for a Samba installation.
The initial synchronization of the consumer works as expected. All LDAP entries are copied to the consumer directory. But after some time, usually when users log in into the Samba running with the provider LDAP, nearly 50% of all LDAP entries on the consumer are deleted. This happens without any change on the provider LDAP!
Checking the logs I found, that delete-messages can be found in the consumers system log.
I do not understand the source of the problem. No entries in the provider LDAP are deleted, so no entries should be deleted in the consumer.
To check the installation, I set up a second consumer in a VMWare environment. And also in this system, which had been set up from scratch and only holds the OpenLDAP-consumer, the entries are deleted at the same time when they are deleted in the "real" OpenLDAP consumer system.
When I change an attribute of an entry in the provider LDAP which has been deleted from the consumer by this process, like adding a description, this change is forwarded to th consumer and the entry "reappears" in the LDAP of the consumer.
Any help or suggestion is appreciated.
Thanks in advance Joachim Hergeth
The providers slapd.conf is: # # See slapd.conf(5) for details on configuration options. # This file should NOT be world readable. # include /etc/openldap/schema/core.schema include /etc/openldap/schema/cosine.schema include /etc/openldap/schema/inetorgperson.schema #include /etc/openldap/schema/rfc2307bis.schema include /etc/openldap/schema/nis.schema include /etc/openldap/schema/yast.schema include /etc/openldap/schema/samba3.schema # Define global ACLs to disable default read access.
# Do not enable referrals until AFTER you have a working directory # service AND an understanding of referrals. #referral ldap://root.openldap.org
pidfile /var/run/slapd/slapd.pid argsfile /var/run/slapd/slapd.args
# Load dynamic backend modules: modulepath /usr/lib/openldap/modules # moduleload back_ldap.la # moduleload back_meta.la # moduleload back_monitor.la # moduleload back_perl.la
# Sample security restrictions # Require integrity protection (prevent hijacking) # Require 112-bit (3DES or better) encryption for updates # Require 63-bit encryption for simple bind # security ssf=1 update_ssf=112 simple_bind=64
# Sample access control policy: # Root DSE: allow anyone to read it # Subschema (sub)entry DSE: allow anyone to read it # Other DSEs: # Allow self write access to user password # Allow anonymous users to authenticate # Allow read access to everything else # Directives needed to implement policy: ## Yast2 samba hack ACL ## allow the "ldap admin dn" access, but deny everyone else access to attrs=SambaLMPassword,SambaNTPassword by dn="cn=Administrator,dc=master-ldap,dc=gts-systems.de" write by dn="cn=repladmin,dc=master-ldap,dc=gts-systems.de" write by * none ## Yast2 samba hack ACL done access to dn.base="" by dn="cn=repladmin,dc=master-ldap,dc=gts-systems.de" write by self write by * auth
access to dn.base="cn=Subschema" by dn="cn=repladmin,dc=master-ldap,dc=gts-systems.de" write by * read
access to attrs=userPassword,userPKCS12 by dn="cn=repladmin,dc=master-ldap,dc=gts-systems.de" write by self write by * auth
access to attrs=shadowLastChange by self write by dn="cn=repladmin,dc=master-ldap,dc=gts-systems.de" write by * read
access to * by dn="cn=repladmin,dc=master-ldap,dc=gts-systems.de" write by * read by anonymous auth
# if no access controls are present, the default policy # allows anyone and everyone to read anything but restricts # updates to rootdn. (e.g., "access to * by * read") # # rootdn can always read and write EVERYTHING!
####################################################################### # BDB database definitions #######################################################################
schemacheck on idletimeout 30 loglevel 131 backend bdb database bdb suffix "dc=master-ldap,dc=gts-systems.de" rootdn "cn=Administrator,dc=master-ldap,dc=gts-systems.de" rootpw <some working password> directory /var/lib/ldap/ checkpoint 1024 5 cachesize 10000 index objectClass,entryCSN,entryUUID,uidNumber,gidNumber eq index member,mail eq,pres index cn,displayname,uid,sn,givenname sub,eq,pres index memberUID eq index sambaSID eq index sambaPrimaryGroupSID eq index sambaDomainName eq index default sub overlay syncprov syncprov-checkpoint 100 1 syncprov-sessionlog 1 ---------------------------------------------------------------------------
the consumers slapd.conf is: # # See slapd.conf(5) for details on configuration options. # This file should NOT be world readable. # include /etc/openldap/schema/core.schema include /etc/openldap/schema/cosine.schema include /etc/openldap/schema/inetorgperson.schema #include /etc/openldap/schema/rfc2307bis.schema include /etc/openldap/schema/nis.schema include /etc/openldap/schema/samba3.schema include /etc/openldap/schema/yast.schema # Define global ACLs to disable default read access.
# Do not enable referrals until AFTER you have a working directory # service AND an understanding of referrals. #referral ldap://root.openldap.org
pidfile /var/run/slapd/slapd.pid argsfile /var/run/slapd/slapd.args
# Load dynamic backend modules: modulepath /usr/lib/openldap/modules # moduleload back_ldap.la # moduleload back_meta.la # moduleload back_monitor.la # moduleload back_perl.la
# Sample security restrictions # Require integrity protection (prevent hijacking) # Require 112-bit (3DES or better) encryption for updates # Require 63-bit encryption for simple bind # security ssf=1 update_ssf=112 simple_bind=64
# Sample access control policy: # Root DSE: allow anyone to read it # Subschema (sub)entry DSE: allow anyone to read it # Other DSEs: # Allow self write access to user password # Allow anonymous users to authenticate # Allow read access to everything else # Directives needed to implement policy: access to dn.base="" by dn="cn=repladmin,dc=master-ldap,dc=gts-systems.de" write by self write by * auth
access to dn.base="cn=Subschema" by dn="cn=repladmin,dc=master-ldap,dc=gts-systems.de" write by * read
access to attrs=userPassword,userPKCS12 by dn="cn=repladmin,dc=master-ldap,dc=gts-systems.de" write by self write by * auth
access to attrs=shadowLastChange by dn="cn=repladmin,dc=master-ldap,dc=gts-systems.de" write by self write by * read
access to * by dn="cn=repladmin,dc=master-ldap,dc=gts-systems.de" write # by anonymous auth by * read
# if no access controls are present, the default policy # allows anyone and everyone to read anything but restricts # updates to rootdn. (e.g., "access to * by * read") # # rootdn can always read and write EVERYTHING!
####################################################################### # BDB database definitions #######################################################################
schemackeck on idletimeout 30 loglevel 15288 # TLSCertificateFile /etc/ssl/servercerts/servercert.pem # TLSCACertificatePath /etc/ssl/certs/ # TLSCertificateKeyFile /etc/ssl/servercerts/serverkey.pem
database bdb suffix "dc=master-ldap,dc=gts-systems.de" rootdn "cn=Administrator,dc=master-ldap,dc=gts-systems.de" rootpw "<some working password>" directory /var/lib/ldap checkpoint 1024 5 cachesize 10000
index objectClass,uidNumber,gidNumber,entryCSN,entryUUID eq index member,mail eq,pres index cn,displayname,uid,sn,givenname sub,eq,pres index memberUID eq index sambaSID eq index sambaPrimaryGroupSID eq index sambaDomainName eq index default sub
syncrepl rid=321 provider=ldap://192.168.1.210:389 type=refreshOnly interval=00:00:00:30 searchbase="dc=master-ldap,dc=gts-systems.de" scope=sub schemachecking=off bindmethod=simple binddn="cn=repladmin,dc=master-ldap,dc=gts-systems.de" credentials=<another nice password>
---------------------------------------------------------------------------
On Thursday 12 April 2007 10:38, Joachim Hergeth wrote:
Hello list,
I have an OpenLDAP provider/consumer installation on two SLES10 systems. One is set up as a provider LDAP, the second is a consumer LDAP using "refreshOnly" synrepl synchronization. The LDAP provides user information for a Samba installation.
The initial synchronization of the consumer works as expected. All LDAP entries are copied to the consumer directory. But after some time, usually when users log in into the Samba running with the provider LDAP, nearly 50% of all LDAP entries on the consumer are deleted. This happens without any change on the provider LDAP!
I guess that you are testing this with the OpenLDAP Version that shipped with SLES10? Would you mind try a newer Version, e.g. the RPMs from http://software.opensuse.org/download/OpenLDAP/SLE_10/ and check if the problem is there as well? Note, that we have an update in the queue for SLES10 to bring it to a more recent version.
Some general comments regarding you configuration (I guess you special problem is not related to those): - The provider config has a line "schemacheck on" this is not a valid slapd.conf statement (IIRC it is from OpenLDAP 2.0.X times or even older) - To debug syncrepl Problems it is most helpful to have the loglevel "sync" enabled. You can to that by just adding "sync" to you "loglevel" line. - The "backend bdb" statement is superfluous - The "syncprov-sessionlog 1" begins with whitespaces, might be a copy 'n paste error in the mail. If it also begins with whitespaces in your slapd.conf you should remove the whitespace. Otherwise it would be treated as the continuation of the previous line (which is wrong). Additionally a sessionlog of "1" operation doesn't make much sense IMO. I suggest you either remove that option or set it to a more reasonable value. - I don't know how large you database is (how many entries) but I should make sure that the syncrepl consumer does not hit the sizelimit of your provider. As you have not configure any sizelimit the default is used which is 500. You can adjust the sizelimit with the "sizelimit" or the "limits" directive in slapd.conf (see slapd.conf man-page for details).
Checking the logs I found, that delete-messages can be found in the consumers system log.
I do not understand the source of the problem. No entries in the provider LDAP are deleted, so no entries should be deleted in the consumer.
Checking if it works with a more recent Version and logfiles with syncrepl logging enabled might help to clear up the issue.
To check the installation, I set up a second consumer in a VMWare environment. And also in this system, which had been set up from scratch and only holds the OpenLDAP-consumer, the entries are deleted at the same time when they are deleted in the "real" OpenLDAP consumer system.
When I change an attribute of an entry in the provider LDAP which has been deleted from the consumer by this process, like adding a description, this change is forwarded to th consumer and the entry "reappears" in the LDAP of the consumer.
Joachim Hergeth wrote:
The initial synchronization of the consumer works as expected. All LDAP entries are copied to the consumer directory. But after some time, usually when users log in into the Samba running with the provider LDAP, nearly 50% of all LDAP entries on the consumer are deleted. This happens without any change on the provider LDAP!
I am experiencing a similar problem, database entries being deleted when they shouldn't be, on a number of replicas. This is causing disruption to users and we'd really like to get it cleared up.
Here's the environment: OpenLDAP 2.3.32 running on Debian 3.1 (Sarge) compiled with sync logging patch discussed about 4 months ago loglevel config sync on all servers BDB 4.2 backend Syncrepl replication all round A "master" server (com) - holds the master copy of the database A number of servers that replicate directly from com An "intermediate" server (wwsv04) that - is on the same LAN and subnet as com - replicates from com - acts as provider for all other servers 88 servers/replicas in total Approx 9000 records All replicas are supposed to be complete copies Nothing particularly fancy or clever going on
I have spent today dissecting the logs from two incidents this week in which entries were erroneously deleted. Although the circumstances of the two incidents are quite different, from examining the logs I believe it is the same thing happening in each case.
Incident 1: A test server ran out of space in /var. After cleaning up, rebooting and doing db_recover, 244 entries were erroneously deleted. This server is on the same building network as com and wwsv04 (different subnet), and replicates from wwsv04.
Incident 2: A production server on the other side of the world lost contact with its provider, and after reconnecting, erroneously deleted 249 entries. This server is connected to the main network via an OpenVPN tunnel across the internet, and replicates from com. (The really odd thing about this is that the network link came back up over an hour 1hr before the first timestamp of this incident, but that is most likely a separate issue.)
I have attached three files (I hope they're not too big): - log_analysis.ods - an OpenOffice spreadsheet containing correlated log entries and my comments for both incidents - incident_1.tgz - syslogs relating to incident 1 - and a CSV version of my analysis of incident 1 - incident_2.tgz - likewise for incident 2
Replication configs are as follows: --------------------------------- com --- # Syncrepl provider
overlay syncprov syncprov-checkpoint 10 5 syncprov-sessionlog 100 syncprov-reloadhint TRUE ------------------------------------- wwsv04 ------ # Syncrepl provider
overlay syncprov syncprov-checkpoint 10 5 syncprov-sessionlog 100 syncprov-reloadhint TRUE
# syncrepl consumer
syncrepl rid=123 provider=ldap://com.example.co.nz type=refreshAndPersist searchbase="dc=example,dc=co,dc=nz" scope=sub schemachecking=off bindmethod=simple binddn="cn=root,dc=example,dc=co,dc=nz" credentials=secret retry=5,5,30,5,60,5,300,+ ------------------------------------- zzsv01 ------ syncrepl rid=123 provider=ldap://wwsv04.example.co.nz type=refreshAndPersist searchbase="dc=example,dc=co,dc=nz" scope=sub schemachecking=off bindmethod=simple binddn="cn=root,dc=example,dc=co,dc=nz" credentials=secret retry=5,5,30,5,60,5,300,+ ------------------------------------- nosv01 ------ syncrepl rid=123 provider=ldap://com.example.co.nz type=refreshAndPersist searchbase="dc=example,dc=co,dc=nz" scope=sub schemachecking=off bindmethod=simple binddn="cn=root,dc=example,dc=co,dc=nz" credentials=secret retry=5,5,30,5,60,5,300,+ -------------------------------------
On Wed, 18 Apr 2007, Lesley Walker wrote:
OpenLDAP 2.3.32 running on Debian 3.1 (Sarge)
[...]
A production server on the other side of the world lost contact with its provider, and after reconnecting, erroneously deleted 249 entries. This
syncprov with a session log drops entries. Makes you wonder if there could be something wrong with the presence list like the numerous other ITS on point...
syncprov.c 1.167 Thu Jan 25 01:06:33 2007 UTC; 2 months, 3 weeks ago by hyc ITS#4813 don't limit the presence list to the current snapshot. add some debug msgs for the sessionlog list.
So...try again with 2.3.35, and let us know.
Aaron Richton wrote:
syncprov.c 1.167 Thu Jan 25 01:06:33 2007 UTC; 2 months, 3 weeks ago by hyc ITS#4813 don't limit the presence list to the current snapshot. add some debug msgs for the sessionlog list.
This would seem to match at least part of the problem.
So...try again with 2.3.35, and let us know.
I can put it on a test server, but I'm not sure how I'll know if it's fixed, since it doesn't happen all the time and I haven't yet been able to identify any common triggers.
As discussed previously, I won't be putting into production any version that hasn't been declared stable.
I would like to make a (late) conclusion to my initial question and the answers I got.
Ralf Haferkamp suggested an LDAP update but also pointed to some flaws in the slapd.conf file. We did not check the update (internal reasons...) but I experimented with the problems in slapd.conf.
And after some trials it seems, that the erroneous entry "syncprov-sessionlog 1" starting a continuation line was the reason for the problems......
After fixing this, all entries in the consumer LDAP remained stable.
Thanks for your help J.Hergeth
Ralf Haferkamp schrieb:
On Thursday 12 April 2007 10:38, Joachim Hergeth wrote:
Hello list,
I have an OpenLDAP provider/consumer installation on two SLES10 systems. One is set up as a provider LDAP, the second is a consumer LDAP using "refreshOnly" synrepl synchronization. The LDAP provides user information for a Samba installation.
The initial synchronization of the consumer works as expected. All LDAP entries are copied to the consumer directory. But after some time, usually when users log in into the Samba running with the provider LDAP, nearly 50% of all LDAP entries on the consumer are deleted. This happens without any change on the provider LDAP!
I guess that you are testing this with the OpenLDAP Version that shipped with SLES10? Would you mind try a newer Version, e.g. the RPMs from http://software.opensuse.org/download/OpenLDAP/SLE_10/ and check if the problem is there as well? Note, that we have an update in the queue for SLES10 to bring it to a more recent version.
Some general comments regarding you configuration (I guess you special problem is not related to those):
- The provider config has a line "schemacheck on" this is not a valid slapd.conf statement (IIRC it is from OpenLDAP 2.0.X times or even older)
- To debug syncrepl Problems it is most helpful to have the loglevel "sync" enabled. You can to that by just adding "sync" to you "loglevel" line.
- The "backend bdb" statement is superfluous
- The "syncprov-sessionlog 1" begins with whitespaces, might be a copy 'n paste error in the mail. If it also begins with whitespaces in your slapd.conf you should remove the whitespace. Otherwise it would be treated as the continuation of the previous line (which is wrong). Additionally a sessionlog of "1" operation doesn't make much sense IMO. I suggest you either remove that option or set it to a more reasonable value.
- I don't know how large you database is (how many entries) but I should make sure that the syncrepl consumer does not hit the sizelimit of your provider. As you have not configure any sizelimit the default is used which is 500. You can adjust the sizelimit with the "sizelimit" or the "limits" directive in slapd.conf (see slapd.conf man-page for details).
Checking the logs I found, that delete-messages can be found in the consumers system log.
I do not understand the source of the problem. No entries in the provider LDAP are deleted, so no entries should be deleted in the consumer.
Checking if it works with a more recent Version and logfiles with syncrepl logging enabled might help to clear up the issue.
To check the installation, I set up a second consumer in a VMWare environment. And also in this system, which had been set up from scratch and only holds the OpenLDAP-consumer, the entries are deleted at the same time when they are deleted in the "real" OpenLDAP consumer system.
When I change an attribute of an entry in the provider LDAP which has been deleted from the consumer by this process, like adding a description, this change is forwarded to th consumer and the entry "reappears" in the LDAP of the consumer.
openldap-software@openldap.org
-
Aaron Richton -
Joachim Hergeth -
Lesley Walker -
Ralf Haferkamp