openldap-bugs

openldap-bugs@openldap.org

1 participants
21001 discussions

Re: (ITS#6482) slapcat doesn't continue with -c after the first corrupted database entry
by sysadmin＠htl-leonding.ac.at 18 Apr '10

18 Apr '10

This is a multi-part message in MIME format. --------------020204070007000004050507 Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit The main point wasn't to tell you how important the data was, but that tools like ldapsearch or other tools connecting to slapd using port 389 still saw the good data. And slapd never complained about a corrupted database. As a result of that the data corruption was hidden (all other programs worked, only three non-important entries were not accessible) until I tried to use slapcat to export the data for an upgrade. Regarding "If the underlying database fails, there's little slapcat can do.": As long as slapd sees the data, because it sends it in responses to ldap-queries, the database hasn't failed so much, that slapcat couldn't retrieve those data. Regarding "any garbage coming out of a corrupted database": slapd never complained about the database being corrupted, it started without any problems (except that three entries were unreadable), the entries after those damaged entries made sense and weren't garbage. Berkeley's db_tools claimed that there were some errors in the db, but slapd worked without problems, only slapcat (and ldapdelete when trying to delete those entries) complained that it couldn't decode them (which seems to be rather a problem of the application layer than of the database layer, but might have been caused by some little db corruption). I tried tools like "db_recover", but they didn't help, so I wanted to extract the good ldap entries (using slapcat ...) to drop the database and recreate it with the exported ldif, which failed because slapcat stopped after the first failure (even with -c). Using the submitted patch I was able to export all readable data into a ldif-file and to recreate the database. summary: * slapd never complained, ldapsearch listed the data, only few entries were missing (because for them following was true: <= entry_decode: slap_str2undef_ad(object�..!p): AttributeDescription contains inappropriate characters) --> it seems that some attribute-values were damaged, but not the dn? * if ldapsearch is able to retrieve the data, slapcat should also do * I don't know how you want to recover an corrupted database. I tried to recover all good entries and then to recreate it. And for that I wanted to use slapcat (or is there any other tool for exporting data from the ldap database than slapcat with option -c)? * maybe the way -c works shouldn't be changed, but there should be an option for trying data recovery harder without having to patch openldap (which is complicated for users of other distros not working directly with the sources, but with precompiled packages) --------------020204070007000004050507 Content-Type: text/html; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <meta http-equiv="content-type" content="text/html; charset=ISO-8859-15"> </head> <body bgcolor="#ffffff" text="#000000"> The main point wasn't to tell you how important the data was, but that tools like ldapsearch or other tools connecting to slapd using port 389 still saw the good data. And slapd never complained about a corrupted database. As a result of that the data corruption was hidden (all other programs worked, only three non-important entries were not accessible) until I tried to use slapcat to export the data for an upgrade. Regarding "If the underlying database fails, there's little slapcat can do.": As long as slapd sees the data, because it sends it in responses to ldap-queries, the database hasn't failed so much, that slapcat couldn't retrieve those data. Regarding "any garbage coming out of a corrupted database": slapd never complained about the database being corrupted, it started without any problems (except that three entries were unreadable), the entries after those damaged entries made sense and weren't garbage. Berkeley's db_tools claimed that there were some errors in the db, but slapd worked without problems, only slapcat (and ldapdelete when trying to delete those entries) complained that it couldn't decode them (which seems to be rather a problem of the application layer than of the database layer, but might have been caused by some little db corruption). I tried tools like "db_recover", but they didn't help, so I wanted to extract the good ldap entries (using slapcat ...) to drop the database and recreate it with the exported ldif, which failed because slapcat stopped after the first failure (even with -c). Using the submitted patch I was able to export all readable data into a ldif-file and to recreate the database. summary: * slapd never complained, ldapsearch listed the data, only few entries were missing (because for them following was true: <pre><= entry_decode: slap_str2undef_ad(object&#65533;..!p): AttributeDescription contains inappropriate characters) --> it seems that some attribute-values were damaged, but not the dn? </pre> * if ldapsearch is able to retrieve the data, slapcat should also do * I don't know how you want to recover an corrupted database. I tried to recover all good entries and then to recreate it. And for that I wanted to use slapcat (or is there any other tool for exporting data from the ldap database than slapcat with option -c)? * maybe the way -c works shouldn't be changed, but there should be an option for trying data recovery harder without having to patch openldap (which is complicated for users of other distros not working directly with the sources, but with precompiled packages) </body> </html> --------------020204070007000004050507--

1 0

Re: (ITS#6482) slapcat doesn't continue with -c after the first corrupted database entry
by masarati＠aero.polimi.it 18 Apr '10

18 Apr '10

> Maybe I haven't described the situation correctly: > The database was used for managing user accounts (via nss_ldap). The > database seems to had three corrupted entries in the middle (good > entries were following after that). The command "getent passwd" i.e. > still listed all users except those three with corrupted entries, but > export via slapcat and reimport of the data would have led to loss of > many more accounts. Lossing half your data is not the expected behaviour > for a tool, which should list all the existing (or at least somehow > readable) database entries. I'm not questioning the importance of the data slapcat couldn't export. > The important point is that slapcat cancelled the export at the first > corrupted database entry and didn't even try to get to the non-corrupted > database entries (which were seen by the other LDAP-tools). This is not correct. Slapcat tries. It asks the underlying backend to provide the subsequent entries. If the underlying database fails, there's little slapcat can do. > It didn't > even try in "continuation" mode, so there was no way to rescue all the > readable data via slapcat. I've tested slapcat in continuation mode by forcibly having be_entry_get() return NULL, and it works as expected: it continues to the next one, as soon as be_entry_next() is able to return a valid ID. if it can't, then anything you'd get from be_entry_next() with an arbitrary ID is unreliable. > For slapcat all entries occuring after the > first corrupt entry were lost. As the remaining good entries were still > accessible and visible via the other tools communicating with slapd via > the port 389, the expected behaviour for slapcat would have been to try > harder to ignore the damaged entries and to recover the remaining > readable entries after them. Maybe slapcat really failed to extract the > next entries id, but in "continuation"-mode it should try hard to > workaround any errors caused by non-readable entries. In case of an > non-readable entry my patch continues stupidly incrementing the id by 1 > and trying to read that entry. As soon as it can extract an entry, it > continues processing. If by "try hard" you mean that any garbage coming out of a corrupted database would be fine for you, then you can fix your patch by having it stop when id == NOID. To me, it sounds a bit too hard. What about recovering the database first, did you try? p.

1 0

Re: (ITS#6528) test020 with hdb: Query 15: ldapsearch failed (4)
by masarati＠aero.polimi.it 18 Apr '10

18 Apr '10

Thanks, I'm working at this. p. > Full_Name: Michael Ströder > Version: HEAD > OS: > URL: > Submission from: (NULL) (84.163.121.239) > > > Note: It seems to work with back-bdb > > $ ./run -b hdb test020 > Cleaning up test run directory leftover from previous run. > Running ./scripts/test020-proxycache for hdb... > Starting master slapd on TCP/IP port 9011... > Using ldapsearch to check that master slapd is running... > Using ldapadd to populate the master directory... > Starting proxy cache on TCP/IP port 9012... > Using ldapsearch to check that proxy slapd is running... > Making queries on the proxy cache... > Query 1: filter:(sn=Jon) attrs:all (expect nothing) > Query 2: filter:(|(cn=*Jon*)(sn=Jon*)) attrs:cn sn title uid > Query 3: filter:(sn=Smith*) attrs:cn sn uid > Query 4: filter:(sn=Doe*) attrs:cn sn title uid > Query 5: filter:(uid=johnd) attrs:mail postaladdress telephonenumber cn > uid > Query 6: filter:(mail=*@mail.alumni.example.com) attrs:cn sn title uid > Query 7: filter:(mail=*) attrs:cn sn title uid > Query 8: filter:(mail=*example.com) attrs:cn sn title uid > ldapsearch failed (4) > Query 9: filter:(uid=b*) attrs:mail > ldapsearch failed (4) > Query 1 not cacheable > Query 2 cacheable > Query 3 cacheable > Query 4 cacheable > Query 5 cacheable > Query 6 cacheable > Query 7 not cacheable > Query 8 cacheable > Query 9 cacheable > Successfully verified cacheability > Query 10: filter:(|(cn=*Jones)(sn=Jones)) attrs:cn sn title uid > Query 11: filter:(sn=Smith) attrs:cn sn title uid > Query 12: filter:(uid=bjorn) attrs:mail postaladdress telephonenumber cn > uid > Query 13: filter:(mail=jaj@mail.alumni.example.com) attrs:cn sn title uid > Query 14: filter:(mail=*example.com) attrs:cn sn title uid > ldapsearch failed (4) > Query 15: filter:(uid=b*) attrs:mail > ldapsearch failed (4) > Query 10 answerable > Query 11 answerable > Query 12 answerable > Query 13 not answerable > Query 14 not answerable > Query 15 answerable > Successfully verified answerability > Filtering ldapsearch results... > Filtering original ldif... > Comparing filter output... > Comparison failed > > >

1 0

Re: (ITS#6527) Build errors in function � slap_cf_aux_table_unparse�
by masarati＠aero.polimi.it 18 Apr '10

18 Apr '10

Thanks, Quanah is working at this. p. > Full_Name: Michael Ströder > Version: RE24 > OS: openSUSE 11.2 Linux > URL: > Submission from: (NULL) (84.163.121.239) > > > HEAD builds and make test works. > > RE24 fails to build since recent check-ins (anon CVS) yesterday: > > make[3]: Leaving directory > `/usr/src/michael/openldap/OPENLDAP_REL_ENG_2_4/openldap/servers/slapd/overlays' > ../../build/mkversion -v "2.4.X" -s -n Versionstr slapd > version.c > cc -g -O2 -DSLAP_SCHEMA_EXPOSE -I../../include -I. -I./slapi -I. > -I../../include > -c -o main.o main.c > cc -g -O2 -DSLAP_SCHEMA_EXPOSE -I../../include -I. -I./slapi -I. > -I../../include > -c -o globals.o globals.c > cc -g -O2 -DSLAP_SCHEMA_EXPOSE -I../../include -I. -I./slapi -I. > -I../../include > -c -o bconfig.o bconfig.c > cc -g -O2 -DSLAP_SCHEMA_EXPOSE -I../../include -I. -I./slapi -I. > -I../../include > -c -o config.o config.c > config.c: In function �slap_cf_aux_table_unparse�: > config.c:1575: error: duplicate case value > config.c:1547: error: previously used here > make[2]: *** [config.o] Error 1 > make[2]: Leaving directory > `/usr/src/michael/openldap/OPENLDAP_REL_ENG_2_4/openldap/servers/slapd' > make[1]: *** [all-common] Error 1 > make[1]: Leaving directory > `/usr/src/michael/openldap/OPENLDAP_REL_ENG_2_4/openldap/servers' > make: *** [all-common] Error 1 > > >

1 0

(ITS#6528) test020 with hdb: Query 15: ldapsearch failed (4)
by michael＠stroeder.com 18 Apr '10

18 Apr '10

Full_Name: Michael Ströder Version: HEAD OS: URL: Submission from: (NULL) (84.163.121.239) Note: It seems to work with back-bdb $ ./run -b hdb test020 Cleaning up test run directory leftover from previous run. Running ./scripts/test020-proxycache for hdb... Starting master slapd on TCP/IP port 9011... Using ldapsearch to check that master slapd is running... Using ldapadd to populate the master directory... Starting proxy cache on TCP/IP port 9012... Using ldapsearch to check that proxy slapd is running... Making queries on the proxy cache... Query 1: filter:(sn=Jon) attrs:all (expect nothing) Query 2: filter:(|(cn=*Jon*)(sn=Jon*)) attrs:cn sn title uid Query 3: filter:(sn=Smith*) attrs:cn sn uid Query 4: filter:(sn=Doe*) attrs:cn sn title uid Query 5: filter:(uid=johnd) attrs:mail postaladdress telephonenumber cn uid Query 6: filter:(mail=*@mail.alumni.example.com) attrs:cn sn title uid Query 7: filter:(mail=*) attrs:cn sn title uid Query 8: filter:(mail=*example.com) attrs:cn sn title uid ldapsearch failed (4) Query 9: filter:(uid=b*) attrs:mail ldapsearch failed (4) Query 1 not cacheable Query 2 cacheable Query 3 cacheable Query 4 cacheable Query 5 cacheable Query 6 cacheable Query 7 not cacheable Query 8 cacheable Query 9 cacheable Successfully verified cacheability Query 10: filter:(|(cn=*Jones)(sn=Jones)) attrs:cn sn title uid Query 11: filter:(sn=Smith) attrs:cn sn title uid Query 12: filter:(uid=bjorn) attrs:mail postaladdress telephonenumber cn uid Query 13: filter:(mail=jaj@mail.alumni.example.com) attrs:cn sn title uid Query 14: filter:(mail=*example.com) attrs:cn sn title uid ldapsearch failed (4) Query 15: filter:(uid=b*) attrs:mail ldapsearch failed (4) Query 10 answerable Query 11 answerable Query 12 answerable Query 13 not answerable Query 14 not answerable Query 15 answerable Successfully verified answerability Filtering ldapsearch results... Filtering original ldif... Comparing filter output... Comparison failed

1 0

Re: (ITS#6482) slapcat doesn't continue with -c after the first corrupted database entry
by sysadmin＠htl-leonding.ac.at 18 Apr '10

18 Apr '10

Maybe I haven't described the situation correctly: The database was used for managing user accounts (via nss_ldap). The database seems to had three corrupted entries in the middle (good entries were following after that). The command "getent passwd" i.e. still listed all users except those three with corrupted entries, but export via slapcat and reimport of the data would have led to loss of many more accounts. Lossing half your data is not the expected behaviour for a tool, which should list all the existing (or at least somehow readable) database entries. The important point is that slapcat cancelled the export at the first corrupted database entry and didn't even try to get to the non-corrupted database entries (which were seen by the other LDAP-tools). It didn't even try in "continuation" mode, so there was no way to rescue all the readable data via slapcat. For slapcat all entries occuring after the first corrupt entry were lost. As the remaining good entries were still accessible and visible via the other tools communicating with slapd via the port 389, the expected behaviour for slapcat would have been to try harder to ignore the damaged entries and to recover the remaining readable entries after them. Maybe slapcat really failed to extract the next entries id, but in "continuation"-mode it should try hard to workaround any errors caused by non-readable entries. In case of an non-readable entry my patch continues stupidly incrementing the id by 1 and trying to read that entry. As soon as it can extract an entry, it continues processing.

1 0

(ITS#6527) Build errors in function �slap_cf_aux_table_unparse�
by michael＠stroeder.com 18 Apr '10

18 Apr '10

Full_Name: Michael Ströder Version: RE24 OS: openSUSE 11.2 Linux URL: Submission from: (NULL) (84.163.121.239) HEAD builds and make test works. RE24 fails to build since recent check-ins (anon CVS) yesterday: make[3]: Leaving directory `/usr/src/michael/openldap/OPENLDAP_REL_ENG_2_4/openldap/servers/slapd/overlays' ../../build/mkversion -v "2.4.X" -s -n Versionstr slapd > version.c cc -g -O2 -DSLAP_SCHEMA_EXPOSE -I../../include -I. -I./slapi -I. -I../../include -c -o main.o main.c cc -g -O2 -DSLAP_SCHEMA_EXPOSE -I../../include -I. -I./slapi -I. -I../../include -c -o globals.o globals.c cc -g -O2 -DSLAP_SCHEMA_EXPOSE -I../../include -I. -I./slapi -I. -I../../include -c -o bconfig.o bconfig.c cc -g -O2 -DSLAP_SCHEMA_EXPOSE -I../../include -I. -I./slapi -I. -I../../include -c -o config.o config.c config.c: In function slap_cf_aux_table_unparse: config.c:1575: error: duplicate case value config.c:1547: error: previously used here make[2]: *** [config.o] Error 1 make[2]: Leaving directory `/usr/src/michael/openldap/OPENLDAP_REL_ENG_2_4/openldap/servers/slapd' make[1]: *** [all-common] Error 1 make[1]: Leaving directory `/usr/src/michael/openldap/OPENLDAP_REL_ENG_2_4/openldap/servers' make: *** [all-common] Error 1

1 0

Re: (ITS#6458) syncrepl fails if attribute has no equalty maching rule
by masarati＠aero.polimi.it 17 Apr '10

17 Apr '10

Should now be fixed in HEAD; please test. Thanks, p.

1 0

Re: (ITS#6458) syncrepl fails if attribute has no equalty maching rule
by masarati＠aero.polimi.it 17 Apr '10

17 Apr '10

> I see the issue, and I've isolated it to the case where you're removing a > value. In that case, in attr_cmp(), nn == 0 and no < o; the original test > for modifying an attribute without matching rule fails. I've fixed it > according to this patch, not sure whether it's appropriate for other cases > > <ftp://ftp.openldap.org/incoming/pierangelo-masarati-2010-04-17-syncrepl-no-…> > > Please test and review. Thanks, p. Please disregard, this patch is breaking other things (basically, producer's and consumer's entries are equivalent, but attrs are listed in a different order; this conforms to the LDAP model but breaks the original, preferable behavior). Working at a better fix. p.

1 0

Re: (ITS#6458) syncrepl fails if attribute has no equalty maching rule
by masarati＠aero.polimi.it 17 Apr '10

17 Apr '10

> Full_Name: Patrick Cernko > Version: 2.4.21 > OS: Linux > URL: > http://www.mpi-inf.mpg.de/~pcernko/syncrepl_fails_if_attribute_has_no_equal… > Submission from: (NULL) (139.19.3.67) > > > Hi OpenLDAP-Programmers, > > although some related bug reports were posted in the past years, this > issue > still seems not to be fixed: > > OpenLDAP's syncrepl replication fails if it comes to removing one of a > multi-value attribes of an object if this attribute does not have a > equality > matching rule defined in the schema. Modification works on the master > server but > fails to replicate to the slave(s) leaving them with the dit version just > before > and not replicating any other changes any more. > > I compiled a small example how to reproduce the problem with the (maybe > "well-known") nisNetgroupTriple attribute from nis.schema as shipped with > openldap-2.4.21: > > - extract the files from > http://www.mpi-inf.mpg.de/~pcernko/syncrepl_fails_if_attribute_has_no_equal… > into an empty working directory > - download the tarball to that working dir and configure similar to the > just > extracted configure.sh > - build and install it to get an install subdir in the working dir > - start the master using sh master.sh in the working dir > This will create the "master" database in your working dir, slapadd the > dit.ldif and start a master slapd with a unix socket /tmp/master. > - start the slave using sh slave.sh > This will create "slave" database in your working dir and start the > slave > slapd with a unix socket /tmp/slave which will immediately start syncing > from > the running master > - if you want you can search the 2 servers with > install/bin/ldapsearch -H ldapi://%2ftmp%2fmaster/ -b ou=openldap -LLL > resp. > install/bin/ldapsearch -H ldapi://%2ftmp%2fslave/ -b ou=openldap -LLL > - use "python mod.py" to remove one of the nisNetgroupTriple attributes > from the > netgroup object > - you will see a debug message from the slave server like this: > > bdb_modify_internal: delete nisNetgroupTriple > bdb_modify_internal: 18 modify/delete: nisNetgroupTriple: no equality > matching > rule > send_ldap_result: err=18 matched="" text="modify/delete: > nisNetgroupTriple: no > equality matching rule" > null_callback : error code 0x12 > slap_graduate_commit_csn: removing 0x29931c0 > 20100120124712.028626Z#000000#000#000000 > syncrepl_entry: rid=001 be_modify cn=netgroup,ou=openldap (18) > syncrepl_entry: rid=001 be_modify failed (18) > > if not, use "python mod.py add" to re-add the removed nisNetgroupTriple > and > try again. Sometimes, it seems to work (race condition?). > - you can also verify the difference with: > install/bin/ldapsearch -H ldapi://%2ftmp%2fmaster/ -b ou=openldap -LLL > cn=netgroup > install/bin/ldapsearch -H ldapi://%2ftmp%2fslave/ -b ou=openldap -LLL > cn=netgroup > - from now on, the slave will fail to sync any succeeding changes, as it > continuously tries to sync the modification above first. :-( > > Now, obviously having an attribute without equality matching rule is not a > good > idea in the first place. However, the (official?) nis.schema comes with > such an > attribute which we must use like this (e.g. we would have to patch the > Solaris > kernel to change it for the Solaris NFS server). > > It is possible to modify a nisNetgroup object like this: according to the > debugging output, I guess the master removes simply ALL nisNetgroupTriple > attributes and adds only the attributes that should remain. When the > corresponding syncrepl_entry is handled by the slave, it seems the it in > contrast tries to really only remove the 2-be-removed attribute which does > not > work as it cannot identify the right one due to a missing eq-match > operator. > > I suggest the best way to solve this issue is to try to get syncrepl do > exactly > the same actions (not resulting changes) as the server does. I see the issue, and I've isolated it to the case where you're removing a value. In that case, in attr_cmp(), nn == 0 and no < o; the original test for modifying an attribute without matching rule fails. I've fixed it according to this patch, not sure whether it's appropriate for other cases <ftp://ftp.openldap.org/incoming/pierangelo-masarati-2010-04-17-syncrepl-no-…> Please test and review. Thanks, p.

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs