- openldap-bugs - openldap.org

Re: (ITS#8875) [Patch] Performance problems in back-mdb with large DITs and many aliases
by quanah＠symas.com 28 Jun '19

28 Jun '19

--On Sunday, April 07, 2019 10:15 AM +0000 henrik.bohnenkamp(a)ionos.com wrote: > On Thu, Apr 04, 2019 at 05:32:16PM +0100, Howard Chu wrote: > Hi Howard, > > thanks for your comments. There is now a new version of patches available > in my github repository. Apart from addressing your comments, I have > rebased the patches against the current master branch (quite a lot > activity in the last two months, I have noticed). In particular, the new > function mdb_get_aliases now also uses the global size variables for IDL > dimensions, rather than the previous CPP constants. Had a customer who was hitting this issue try out these patches -- It greatly decreases the search time (from unknown/infinite to 1 minute). Unfortunately slapd then segv's. Working on getting a test database to reproduce the issue with for a good backtrace. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#8427) Incorrect value of tls_reqcert in syncrepl
by quanah＠symas.com 27 Jun '19

27 Jun '19

Hello, Unfortunately this patch introduces a regression and breaks existing configurations where the protocol in use is ldaps:///. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: Regression after ITS#8427 fix with back-ldap
by quanah＠symas.com 27 Jun '19

27 Jun '19

--On Thursday, June 27, 2019 11:51 PM +0200 Abdelkader Chelouah <a.chelouah(a)gmail.com> wrote: > > I tried with with > > > olcDbStartTLS: ldaps > > > with no success. > > > Logs on the proxy > > > 2019-06-27T20:46:07.602925+00:00 arrakis slapd-2.4-bb[17262]: conn=1001 > fd=11 ACCEPT from IP=[::1]:46462 (IP=[::1]:10020) > 2019-06-27T20:46:07.603476+00:00 arrakis slapd-2.4-bb[17262]: conn=1001 > op=0 BIND dn="uid=alice,ou=people,dc=local" method=128 > 2019-06-27T20:46:07.609212+00:00 arrakis slapd-2.4-bb[17262]: conn=1001 > op=0 ldap_back_retry: retrying URI="ldaps://arrakis.local:10011" DN="" > 2019-06-27T20:46:07.613435+00:00 arrakis slapd-2.4-bb[17262]: conn=1001 > op=0 RESULT tag=97 err=52 text=Proxy operation retry failed > 2019-06-27T20:46:07.613968+00:00 arrakis slapd-2.4-bb[17262]: conn=1001 > op=1 UNBIND > 2019-06-27T20:46:07.614264+00:00 arrakis slapd-2.4-bb[17262]: conn=1001 > fd=11 closed > > Logs on the backend ldap server > > > 2019-06-27T20:46:07.609971+00:00 arrakis slapd-2.4-aa[14682]: conn=1011 > fd=12 ACCEPT from IP=172.18.0.2:47156 (IP=172.18.0.2:10011) > 2019-06-27T20:46:07.613718+00:00 arrakis slapd-2.4-aa[14682]: conn=1011 > fd=12 closed (TLS negotiation failure) Thanks! Please include the openldap-its list in your replies so that they properly get associated with the relevant ITS. I'll raise this up as a priority for the 2.4.48 release. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: Regression after ITS#8427 fix with back-ldap
by quanah＠symas.com 27 Jun '19

27 Jun '19

--On Thursday, June 27, 2019 9:08 PM +0000 a.chelouah(a)gmail.com wrote: > This is a multi-part message in MIME format. > --------------93F3FA89632EC27DC6224304 > Content-Type: text/plain; charset=utf-8; format=flowed > Content-Transfer-Encoding: 7bit > > Hello, > > Commit 6f623dfa1ca65698c19ccc6c058cd170e633384e fixing ITS#8427 (Set up > TLS settings on each reconnection) introduce a regression when the proxy > connect to the**Backend ldap server via ldaps:// Thanks for the follow up! It may be useful in the future to participate in the testing calls that are noted on the openldap-devel list as well. Does the issue persist if you set: olcDbStartTLS: ldaps as documented in the slapd-ldap(5) man page? Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Regression after ITS#8427 fix with back-ldap
by a.chelouah＠gmail.com 27 Jun '19

27 Jun '19

This is a multi-part message in MIME format. --------------93F3FA89632EC27DC6224304 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hello, Commit 6f623dfa1ca65698c19ccc6c058cd170e633384e fixing ITS#8427 (Set up TLS settings on each reconnection) introduce a regression when the proxy connect to the**Backend ldap server via ldaps:// The relevent part of my config is: dn: olcDatabase={2}ldap,cn=config objectClass: olcDatabaseConfig objectClass: olcLDAPConfig olcDatabase: {2}ldap olcSuffix: dc=local olcDbURI: ldaps://ldap.local olcDbChaseReferrals: TRUE olcDbRebindAsUser: TRUE olcDbIDAssertBind: bindmethod=none tls_cacert=/etc/pki/tls/certs/ca.crt olcDbIDAssertAuthzFrom: "*" (I also tried by setting LDAPTLS_CACERT env var when starting slapd) On backend ldap server logs, I get the message "TLS negociation failure" Regards --------------93F3FA89632EC27DC6224304 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> </head> <body text="#000000" bgcolor="#FFFFFF"> Hello, Commit 6f623dfa1ca65698c19ccc6c058cd170e633384e fixing ITS#8427 (Set up TLS settings on each reconnection) introduce a regression when the proxy connect to the Backend ldap server via <a class="moz-txt-link-freetext" href="ldaps://">ldaps://</a> The relevent part of my config is: dn: olcDatabase={2}ldap,cn=config objectClass: olcDatabaseConfig objectClass: olcLDAPConfig olcDatabase: {2}ldap olcSuffix: dc=local olcDbURI: <a class="moz-txt-link-freetext" href="ldaps://ldap.local">ldaps://ldap.local</a> olcDbChaseReferrals: TRUE olcDbRebindAsUser: TRUE olcDbIDAssertBind: bindmethod=none tls_cacert=/etc/pki/tls/certs/ca.crt olcDbIDAssertAuthzFrom: "*" (I also tried by setting LDAPTLS_CACERT env var when starting slapd) On backend ldap server logs, I get the message "TLS negociation failure" Regards </body> </html> --------------93F3FA89632EC27DC6224304--

1 0

Re: (ITS#8977) Make IDL sizes configurable in back-mdb
by quanah＠symas.com 27 Jun '19

27 Jun '19

--On Thursday, June 27, 2019 8:35 PM +0000 hyc(a)symas.com wrote: > No, because order is irrelevant for these. Cool, thanks! I'll continue on with deeper testing then. :) --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#8977) Make IDL sizes configurable in back-mdb
by hyc＠symas.com 27 Jun '19

27 Jun '19

Quanah Gibson-Mount wrote: > --On Tuesday, June 25, 2019 7:07 PM +0000 quanah(a)symas.com wrote: >=20 >> c) That conversion from slapd.conf to cn=3Dconfig be fixed so that it = works. >=20 >=20 > I'm not entirely sure that the fix that went in for this in ec411582d66= 3667d6b638162db51dfa70f5263d3 is correct.=C2=A0 Specifically, unlike ever= ything else, there is > no associated weight.=C2=A0 If in the future other backends (such as SQ= L, LDAP, etc) support server-wide settings via an olcBackend configuratio= n, the weight would > need to exist. >=20 > root@anvil4:/tmp/q/slapd.d/cn=3Dconfig# ls -l > total 64 > -rw------- 1 root root=C2=A0=C2=A0 448 Jun 27 12:19 'cn=3Dmodule{0}.ldi= f' > drwxr-x--- 2 root root=C2=A0 4096 Jun 27 12:19 'cn=3Dschema' > -rw------- 1 root root 39646 Jun 27 12:19 'cn=3Dschema.ldif' > -rw------- 1 root root=C2=A0=C2=A0 435 Jun 27 12:19 'olcBackend=3Dmdb.l= dif' > -rw------- 1 root root=C2=A0=C2=A0 596 Jun 27 12:19 'olcDatabase=3D{-1}= frontend.ldif' > -rw------- 1 root root=C2=A0=C2=A0 584 Jun 27 12:19 'olcDatabase=3D{0}c= onfig.ldif' > -rw------- 1 root root=C2=A0=C2=A0 859 Jun 27 12:19 'olcDatabase=3D{1}m= db.ldif' >=20 >=20 > I.e., I was rather expecting: >=20 > olcBackend=3D{1}mdb.ldif >=20 > or similar. No, because order is irrelevant for these. --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8977) Make IDL sizes configurable in back-mdb
by quanah＠symas.com 27 Jun '19

27 Jun '19

--On Tuesday, June 25, 2019 7:07 PM +0000 quanah(a)symas.com wrote: > c) That conversion from slapd.conf to cn=config be fixed so that it works. I'm not entirely sure that the fix that went in for this in ec411582d663667d6b638162db51dfa70f5263d3 is correct. Specifically, unlike everything else, there is no associated weight. If in the future other backends (such as SQL, LDAP, etc) support server-wide settings via an olcBackend configuration, the weight would need to exist. root@anvil4:/tmp/q/slapd.d/cn=config# ls -l total 64 -rw------- 1 root root 448 Jun 27 12:19 'cn=module{0}.ldif' drwxr-x--- 2 root root 4096 Jun 27 12:19 'cn=schema' -rw------- 1 root root 39646 Jun 27 12:19 'cn=schema.ldif' -rw------- 1 root root 435 Jun 27 12:19 'olcBackend=mdb.ldif' -rw------- 1 root root 596 Jun 27 12:19 'olcDatabase={-1}frontend.ldif' -rw------- 1 root root 584 Jun 27 12:19 'olcDatabase={0}config.ldif' -rw------- 1 root root 859 Jun 27 12:19 'olcDatabase={1}mdb.ldif' I.e., I was rather expecting: olcBackend={1}mdb.ldif or similar. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9043) Improve loglevel sync logging
by ondra＠openldap.org 26 Jun '19

26 Jun '19

Full_Name: Ondrej Kuznik Version: master OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (82.10.24.68) The existing replication logs are quite hard to follow. In syncprov, messages about different sessions mingle with no indication where they belong and many important replication decisions are not logged at all. And what is logged is very hard to follow and understand why the provider/consumer does what it does. The following could be done to improve matters: - prepend the request information to syncprov messages - add information to existing messages, especially human-readable commentary that can be followed as a story, giving context in terms of "X happened so doing Y" - log when checkpoints happen, what cookies are generated and why, ... Most of the above is drafted here: https://github.com/mistotebe/openldap/tree/its9043

1 0

Re: (ITS#8977) Make IDL sizes configurable in back-mdb
by quanah＠symas.com 25 Jun '19

25 Jun '19

--On Monday, May 06, 2019 9:28 PM +0000 quanah(a)symas.com wrote: To further document the issues after further testing with OpenLDAP master as of 6/25/2019 a) If a value for idlexp is set in a slapd.conf file that is not allowed (< 16 or > 31), converting to cn=config using slaptest will incorrectly report that the target "slapd.d" directory doesn't exist, like: /usr/local/etc/openldap# /usr/local/sbin/slaptest -f slapd.conf -F /tmp/slapd.d slaptest: bad configuration directory! ls -ld /tmp/slapd.d drwxr-xr-x 2 root root 40 Jun 25 17:59 /tmp/slapd.d b) If the idlexp value is corrected to be within the allowed range, the converted cn=config database loses the backend configuration and the idlexp setting: /usr/local/sbin/slaptest -f slapd.conf -F /tmp/slapd.d config file testing succeeded cd /tmp/slapd.d/ find . -type f | xargs grep -i idlexp ./cn=config/cn=schema.ldif:olcAttributeTypes: ( OLcfgBkAt:12.1 NAME 'olcBkMdbIdlExp' DESC 'Power of 2 u ./cn=config/cn=schema.ldif: onfiguration' SUP olcBackendConfig STRUCTURAL MAY olcBkMdbIdlExp ) No olcBackend... file exists, etc. Suggested remedies: a) The man page documentation be updated to note the limitations on valid ranges for the idlexp setting (16<=x<=31), at least for 64-bit systems. b) slaptest provides a valid error if the idlexp setting is not within the valid range (as opposed to complaining the target directory does not exist, when in fact it does). c) That conversion from slapd.conf to cn=config be fixed so that it works. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#9042) loglevel setting that allows seeing attribute values for MOD operations
by quanah＠symas.com 25 Jun '19

25 Jun '19

--On Tuesday, June 25, 2019 5:04 PM +0000 quanah(a)openldap.org wrote: > Currently there is no way to see what values a client is providing for a > MOD operation outside of the "packets" debug level. It would be helpful > for debugging purposes where one does not have control of the client to > be able to set a loglevel that would expose this information. STATS2 would be the appropriate loglevel setting. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9042) loglevel setting that allows seeing attribute values for MOD operations
by quanah＠openldap.org 25 Jun '19

25 Jun '19

Full_Name: Quanah Gibson-Mount Version: HEAD OS: N/A URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (47.208.128.44) Currently there is no way to see what values a client is providing for a MOD operation outside of the "packets" debug level. It would be helpful for debugging purposes where one does not have control of the client to be able to set a loglevel that would expose this information.

1 0

Re: (ITS#9041) cyrus.c includes limits.h twice
by quanah＠symas.com 25 Jun '19

25 Jun '19

--On Friday, June 21, 2019 10:58 PM +0000 quanah(a)openldap.org wrote: > This should be cleaned up. Fixed in b02807ea2f5eaf85e57e67e5851931a116947b94 --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

Re: (ITS#7996) Tighten race in ldap_int_initialize
by Armin.Tueting＠tueting-online.com 23 Jun '19

23 Jun '19

Am Freitag, den 21.06.2019, 13:31 +0200 schrieb Ond=C5=99ej Kuzn=C3=ADk: [...] > Hi Armin, Hi Ondrej, > thank you very much for your report, this should now be fixed in master > with commit b2f4cacd4783cfe49370accc712863f9537f9924. I'm not following 'master' - I'm afraid! But, the commit has been merged into 'OPENLDAP_REL_ENG_2_4' and slapd is now serving our requests! So far, so good! Thanks for your effort! >=20 > Regards, Regards, Armin.

1 0

(ITS#9041) cyrus.c includes limits.h twice
by quanah＠openldap.org 21 Jun '19

21 Jun '19

Full_Name: Quanah Gibson-Mount Version: RE24 OS: N/A URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (47.208.128.44) There appear to have been dueling commits on April 10, 2005 and April 9, 2005, so that we get: 3e800f20bd (Kurt Zeilenga 2005-04-10 19:32:14 +0000 28)#ifdef HAVE_LIMITS_H 3e800f20bd (Kurt Zeilenga 2005-04-10 19:32:14 +0000 29)#include <limits.h> 3e800f20bd (Kurt Zeilenga 2005-04-10 19:32:14 +0000 30)#endif 3e800f20bd (Kurt Zeilenga 2005-04-10 19:32:14 +0000 31) bc0cc3272c (Kurt Zeilenga 2003-02-21 17:48:03 +0000 32)#include "ldap-int.h" bc0cc3272c (Kurt Zeilenga 2003-02-21 17:48:03 +0000 33) bc0cc3272c (Kurt Zeilenga 2003-02-21 17:48:03 +0000 34)#ifdef HAVE_CYRUS_SASL bc0cc3272c (Kurt Zeilenga 2003-02-21 17:48:03 +0000 35) eca819d866 (Howard Chu 2005-04-09 06:41:39 +0000 36)#ifdef HAVE_LIMITS_H eca819d866 (Howard Chu 2005-04-09 06:41:39 +0000 37)#include <limits.h> eca819d866 (Howard Chu 2005-04-09 06:41:39 +0000 38)#endif This should be cleaned up.

1 0

Re: (ITS#9037) observing crash in mdb_cursor_put()
by myk＠mykzilla.org 21 Jun '19

21 Jun '19

After stepping through the test program in a debugger, I see that mdb_page_search returns MDB_NOTFOUND if mc->mc_db->md_root is P_INVALID because the tree is empty. And mdb_cursor_put sets rc to MDB_NO_ROOT if mc->mc_db->md_root is P_INVALID. So that can't be related to the crash, at least not in the instances I'm investigating, because insert_key and insert_data (and hence rc) is always MDB_NOTFOUND in their minidumps when the crash occurs, which can't happen when rc is set to MDB_NO_ROOT. -myk

1 0

Re: (ITS#7996) Tighten race in ldap_int_initialize
by ondra＠mistotebe.net 21 Jun '19

21 Jun '19

On Tue, Jun 18, 2019 at 07:33:48AM +0000, Armin.Tueting(a)tueting-online.com wrote: > Am Montag, den 17.06.2019, 15:40 -0700 schrieb Quanah Gibson-Mount: >> --On Monday, June 17, 2019 7:16 PM +0200 Armin T=C3=BCting=20 >> <Armin.Tueting(a)tueting-online.com> wrote: >>>> I.e., it started and then got as far as reading your ldap.conf file. >>>> What is the contents of ldap.conf? >>> Attached 'ldap.conf'. Nothing unusual... >>> >>>> Have you run the test suite (make test)? Does it pass? fail? >>> Attached 'make_test.txt'. As far as I can see - it has been passed. >> >> Ok, so make test passes without issue, so it would appear there's something >> specific with your configuration that is triggering the problem. Would you >> be able to provide your slapd configuration (minus any passwords and the >> like)? > I'll privately send it to you in a seperate email. > >> >> Additionally, if you could get a full gdb backtrace of the hung slapd=20 >> process that would be useful as well. I.e.: >> >> start up slapd >> gdb /path/to/slapd <pid #> > (gdb) thr apply all bt full > #3 0x00007ffa4a92a135 in ldap_pvt_thread_mutex_lock > #4 0x00007ffa4a944e6e in ldap_set_option > #5 0x00007ffa4a943ed3 in openldap_ldap_init_w_conf > #6 0x00007ffa4a944318 in openldap_ldap_init_w_sysconf > #8 0x00007ffa4a944e38 in ldap_set_option > #9 0x0000000000413cd6 in main () Hi Armin, thank you very much for your report, this should now be fixed in master with commit b2f4cacd4783cfe49370accc712863f9537f9924. Regards, -- Ondřej Kuzník Senior Software Engineer Symas Corporation http://www.symas.com Packaged, certified, and supported LDAP solutions powered by OpenLDAP

1 0

Corrected vulnerabilities (CVE's) versus ITS# number
by claude.cheret＠external.thalesgroup.com 21 Jun '19

21 Jun '19

Full_Name: Claude CHERET Version: 2.4.46 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (92.163.84.130) Hi, Can you tel me where I can find information about corrected CVE versus OpenLdap version ? Bests Regards. -- Claude Chéret claude.cheret(a)external.thalesgroup.com

1 0

Re: (ITS#9037) observing crash in mdb_cursor_put()
by myk＠mykzilla.org 21 Jun '19

21 Jun '19

myk(a)mykzilla.org wrote on 2019-06-20 23:39: > myk(a)mykzilla.org wrote on 2019-06-20 23:35: >> I'm unsure if it's related, but with this change to mdb.c, the following >> test program (which resembles Firefox's usage) asserts in >> mdb_cursor_init because rc is MDB_NOTFOUND. > Erm, here's the program without the errant characters: My apologies, I'm unsure why the characters appeared the second time as well. (I tested first by manually removing them, sending the message to myself, and then confirming they didn't appear.) I've saved the program to this GitHub gist: https://gist.github.com/mykmelez/ac75a7ab0a0d09a7d709be63806d01d7 -myk

1 0

Re: (ITS#9037) observing crash in mdb_cursor_put()
by myk＠mykzilla.org 21 Jun '19

21 Jun '19

myk(a)mykzilla.org wrote on 2019-06-20 23:35: > I'm unsure if it's related, but with this change to mdb.c, the following > test program (which resembles Firefox's usage) asserts in > mdb_cursor_init because rc is MDB_NOTFOUND. Erm, here's the program without the errant characters: #include <stdio.h> #include <stdlib.h> #include "lmdb.h" #define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr) #define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \ "%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort())) int main(int argc,char * argv[]) { int rc; MDB_env *env; MDB_dbi dbi; MDB_val key, data; MDB_txn *txn; char sval[] = "foo"; char dval[] = "bar"; E(mdb_env_create(&env)); E(mdb_env_set_maxdbs(env, 2)); E(mdb_env_open(env, "./mytestdb", 0, 0664)); E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_dbi_open(txn, "subdb", MDB_CREATE, &dbi)); E(mdb_txn_commit(txn)); key.mv_size = 3; key.mv_data = sval; data.mv_size = 3; data.mv_data = dval; E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_put(txn, dbi, &key, &data, 0)); E(mdb_txn_commit(txn)); mdb_dbi_close(env, dbi); mdb_env_close(env); return 0; }

1 0

Re: (ITS#9037) observing crash in mdb_cursor_put()
by myk＠mykzilla.org 21 Jun '19

21 Jun '19

hyc(a)symas.com wrote on 2019-06-19 09:55: > Maybe. For testing purposes you could try something like this > diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c > index 692feaa38b..e41f3bc36a 100644 > --- a/libraries/liblmdb/mdb.c > +++ b/libraries/liblmdb/mdb.c > @@ -7620,7 +7620,8 @@ mdb_cursor_init(MDB_cursor *mc, MDB_txn *txn, MDB_dbi dbi, MDB_xcursor *mx) > mc->mc_xcursor = NULL; > } > if (*mc->mc_dbflag & DB_STALE) { > - mdb_page_search(mc, NULL, MDB_PS_ROOTONLY); > + int rc = mdb_page_search(mc, NULL, MDB_PS_ROOTONLY); > + mdb_cassert(mc, !rc); > } > } I'm unsure if it's related, but with this change to mdb.c, the following test program (which resembles Firefox's usage) asserts in mdb_cursor_init because rc is MDB_NOTFOUND. #include <stdio.h> #include <stdlib.h> #include "lmdb.h" #define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr) #define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \ "%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort())) int main(int argc,char * argv[]) { int rc; MDB_env *env; MDB_dbi dbi; MDB_val key, data; MDB_txn *txn; char sval[] = "foo"; char dval[] = "bar"; E(mdb_env_create(&env)); E(mdb_env_set_maxdbs(env, 2)); E(mdb_env_open(env, "./mytestdb", 0, 0664)); E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_dbi_open(txn, "subdb", MDB_CREATE, &dbi)); E(mdb_txn_commit(txn)); key.mv_size = 3; key.mv_data = sval; data.mv_size = 3; data.mv_data = dval; E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_put(txn, dbi, &key, &data, 0)); E(mdb_txn_commit(txn)); mdb_dbi_close(env, dbi); mdb_env_close(env); return 0; } Without the change to mdb.c, however, the program appears to work correctly. -myk

1 0

Re: (ITS#9039) key corruption using mdb_put and differing data
by hyc＠symas.com 20 Jun '19

20 Jun '19

lmdb(a)pluckeye.net wrote: > Full_Name: Jonny Wilkes > Version: 2.4.2 > OS: Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (67.160.82.31) > > > The following code shows key corruption. Yes, because you've misused the API. > > I am probably using the API wrong, but I don't see how. I apologize in advance > for not having a clue. Read the Note http://www.lmdb.tech/doc/group__mdb.html#ga8bf10cd91d3f3a83a34d04ce6b07992d Further comments below. > And now, on to the test. It can be compiled and run like so: > > gcc t1.c -llmdb && ./a.out > > --------- t1.c starts here --------- > /* > > The output of this program using liblmdb0 v0.9.21-1 on Ubuntu is: > > --- dump --- > key12345 > key67890 > --- dump --- > key12345 > y12345 > > Why? What happened to key67890? > > */ > > #include <lmdb.h> > #if USE_SOURCE > #include "midl.c" > #include "mdb.c" > #endif > #include <stdio.h> > #include <string.h> > #include <unistd.h> > > #define P(K) printf("%s\n", (char *)K.mv_data) > > #define X(x) do { int _x = x; if (_x) fprintf(stderr, "%d %s\n", _x, > mdb_strerror(_x)); } while (0) > > const char *path = "/tmp/lmdb-why.tmp"; > MDB_env *env; > > /* > * setup creates /tmp/lmdb-why.tmp with 2 keys with the data "hey": > * > * 1. key12345 : hey > * 2. key67890 : hey > */ > int > setup(void) > { > unlink(path); > if (!env) { > X(mdb_env_create(&env)); > X(mdb_env_open(env, path, MDB_NOSUBDIR, 0664)); > } > > MDB_txn *txn; X(mdb_txn_begin(env, NULL, 0, &txn)); > MDB_dbi dbi; X(mdb_dbi_open(txn, NULL, 0, &dbi)); > > const char *keys[] = { > "key12345", > "key67890" > }; > for (unsigned i = 0; i < 2; ++i) { > MDB_val key, data; > key.mv_data = (void *)keys[i]; > key.mv_size = strlen(keys[i])+1; > data.mv_data = "hey"; > data.mv_size = 4; > X(mdb_put(txn, dbi, &key, &data, 0)); > } > X(mdb_txn_commit(txn)); > return 0; > } > > int > dump(void) > { > printf("--- dump ---\n"); > > MDB_txn *txn; X(mdb_txn_begin(env, NULL, 0, &txn)); > MDB_dbi dbi; X(mdb_dbi_open(txn, NULL, 0, &dbi)); > MDB_cursor *cursor; X(mdb_cursor_open(txn, dbi, &cursor)); > > MDB_val key; > int rc; > while ((rc = mdb_cursor_get(cursor, &key, NULL, MDB_NEXT)) == 0) { > P(key); > } > > mdb_cursor_close(cursor); > if (MDB_NOTFOUND != rc) > return rc; > mdb_txn_abort(txn); > return 0; > } > > /* > * change_data change the data for each key to "there" using mdb_put. > * After it completes, the keys are corrupt. > */ > int > change_data(void) > { > MDB_txn *txn; X(mdb_txn_begin(env, NULL, 0, &txn)); > MDB_dbi dbi; X(mdb_dbi_open(txn, NULL, 0, &dbi)); > MDB_cursor *cursor; X(mdb_cursor_open(txn, dbi, &cursor)); > > MDB_val key; > int rc; > while ((rc = mdb_cursor_get(cursor, &key, NULL, MDB_NEXT)) == 0) { > MDB_val data; > data.mv_data = (void *)"there"; > data.mv_size = 6; /* ! */ > X(mdb_put(txn, dbi, &key, &data, 0)); As documented, the key returned by mdb_cursor_get is only valid until the next update operation. Since you've called mdb_put here, the key is altered while the put is executing, and so you get your corruption. You have to copy the key if you want this particular code to work. But even so, you're not using the API optimally. Since you have a cursor already, you should be using mdb_cursor_put instead. > } > mdb_cursor_close(cursor); > if (MDB_NOTFOUND != rc) > return rc; > > X(mdb_txn_commit(txn)); > return 0; > } > > int main(void) > { > X(setup()); > X(dump()); > X(change_data()); > X(dump()); > return 0; > } Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9039) key corruption using mdb_put and differing data
by lmdb＠pluckeye.net 20 Jun '19

20 Jun '19

Full_Name: Jonny Wilkes Version: 2.4.2 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (67.160.82.31) The following code shows key corruption. I am probably using the API wrong, but I don't see how. I apologize in advance for not having a clue. And now, on to the test. It can be compiled and run like so: gcc t1.c -llmdb && ./a.out --------- t1.c starts here --------- /* The output of this program using liblmdb0 v0.9.21-1 on Ubuntu is: --- dump --- key12345 key67890 --- dump --- key12345 y12345 Why? What happened to key67890? */ #include <lmdb.h> #if USE_SOURCE #include "midl.c" #include "mdb.c" #endif #include <stdio.h> #include <string.h> #include <unistd.h> #define P(K) printf("%s\n", (char *)K.mv_data) #define X(x) do { int _x = x; if (_x) fprintf(stderr, "%d %s\n", _x, mdb_strerror(_x)); } while (0) const char *path = "/tmp/lmdb-why.tmp"; MDB_env *env; /* * setup creates /tmp/lmdb-why.tmp with 2 keys with the data "hey": * * 1. key12345 : hey * 2. key67890 : hey */ int setup(void) { unlink(path); if (!env) { X(mdb_env_create(&env)); X(mdb_env_open(env, path, MDB_NOSUBDIR, 0664)); } MDB_txn *txn; X(mdb_txn_begin(env, NULL, 0, &txn)); MDB_dbi dbi; X(mdb_dbi_open(txn, NULL, 0, &dbi)); const char *keys[] = { "key12345", "key67890" }; for (unsigned i = 0; i < 2; ++i) { MDB_val key, data; key.mv_data = (void *)keys[i]; key.mv_size = strlen(keys[i])+1; data.mv_data = "hey"; data.mv_size = 4; X(mdb_put(txn, dbi, &key, &data, 0)); } X(mdb_txn_commit(txn)); return 0; } int dump(void) { printf("--- dump ---\n"); MDB_txn *txn; X(mdb_txn_begin(env, NULL, 0, &txn)); MDB_dbi dbi; X(mdb_dbi_open(txn, NULL, 0, &dbi)); MDB_cursor *cursor; X(mdb_cursor_open(txn, dbi, &cursor)); MDB_val key; int rc; while ((rc = mdb_cursor_get(cursor, &key, NULL, MDB_NEXT)) == 0) { P(key); } mdb_cursor_close(cursor); if (MDB_NOTFOUND != rc) return rc; mdb_txn_abort(txn); return 0; } /* * change_data change the data for each key to "there" using mdb_put. * After it completes, the keys are corrupt. */ int change_data(void) { MDB_txn *txn; X(mdb_txn_begin(env, NULL, 0, &txn)); MDB_dbi dbi; X(mdb_dbi_open(txn, NULL, 0, &dbi)); MDB_cursor *cursor; X(mdb_cursor_open(txn, dbi, &cursor)); MDB_val key; int rc; while ((rc = mdb_cursor_get(cursor, &key, NULL, MDB_NEXT)) == 0) { MDB_val data; data.mv_data = (void *)"there"; data.mv_size = 6; /* ! */ X(mdb_put(txn, dbi, &key, &data, 0)); } mdb_cursor_close(cursor); if (MDB_NOTFOUND != rc) return rc; X(mdb_txn_commit(txn)); return 0; } int main(void) { X(setup()); X(dump()); X(change_data()); X(dump()); return 0; }

1 0

Re: (ITS#9037) observing crash in mdb_cursor_put()
by hyc＠symas.com 19 Jun '19

19 Jun '19

myk(a)mykzilla.org wrote: > hyc(a)symas.com wrote on 2019-06-18 01:30: >> There is no issue with mdb_cursor_put then. The question is why didn't mdb_page_search find the DBI's root node? > Would it be useful for mdb_cursor_init to return the result code of > mdb_page_search, and for mdb_put to then return that code if it isn't > MDB_SUCCESS? Maybe. For testing purposes you could try something like this diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c index 692feaa38b..e41f3bc36a 100644 --- a/libraries/liblmdb/mdb.c +++ b/libraries/liblmdb/mdb.c @@ -7620,7 +7620,8 @@ mdb_cursor_init(MDB_cursor *mc, MDB_txn *txn, MDB_dbi dbi, MDB_xcursor *mx) mc->mc_xcursor = NULL; } if (*mc->mc_dbflag & DB_STALE) { - mdb_page_search(mc, NULL, MDB_PS_ROOTONLY); + int rc = mdb_page_search(mc, NULL, MDB_PS_ROOTONLY); + mdb_cassert(mc, !rc); } } I suspect the problem will simply be that you've used an invalid DBI. > > That wouldn't resolve the problem, but it might avoid the crash and > provide some insight into the failure (if mdb_page_search is actually > returning a failure code when it doesn't find the DBI's root node). > > -myk > > > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9037) observing crash in mdb_cursor_put()
by myk＠mykzilla.org 19 Jun '19

19 Jun '19

hyc(a)symas.com wrote on 2019-06-18 01:30: > There is no issue with mdb_cursor_put then. The question is why didn't mdb_page_search find the DBI's root node? Would it be useful for mdb_cursor_init to return the result code of mdb_page_search, and for mdb_put to then return that code if it isn't MDB_SUCCESS? That wouldn't resolve the problem, but it might avoid the crash and provide some insight into the failure (if mdb_page_search is actually returning a failure code when it doesn't find the DBI's root node). -myk

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs