- openldap-bugs - openldap.org

(ITS#8371) slapd crashes with ldappasswd
by ktmdms＠gmail.com 12 Feb '16

12 Feb '16

Full_Name: Kevin Martin Version: 2.4.44 OS: Oracle Linux 7 URL: Submission from: (NULL) (160.34.110.215) Running Openldap 2.4.42 and/or 2.4.44 I am able to crash slapd at will running ldappasswd as root changing another users password. On another openldap server running 2.4.42 the same change works flawlessly. Here's my setup: Command run: ldappasswd -d1 -ZZ "uid=chaliburt,ou=People,dc=ccd,dc=com" -D "cn=Manager,dc=ccd,dc=com" -H ldap://ldapx.mgt.ccd -W -S New password: Re-enter new password: Enter LDAP Password: ldap_result: Can't contact LDAP server (-1) Bad Server: Oracle Linux 7 3.8.13-118.2.5.el7uek.x86_64 openldap 2.4.44 ]# ldd /usr/local/libexec/slapd linux-vdso.so.1 => (0x00007ffca8b85000) libltdl.so.7 => /lib64/libltdl.so.7 (0x00007f8a8ea45000) libdb-5.3.so => /lib64/libdb-5.3.so (0x00007f8a8e687000) l libssl.so.10 => /lib64/libssl.so.10 (0x00007f8a8e41a000) libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f8a8e033000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f8a8de19000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8a8dbfd000) libc.so.6 => /lib64/libc.so.6 (0x00007f8a8d83f000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f8a8d63b000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f8a8d3ef000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f8a8d10a000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f8a8cf06000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f8a8ccd4000) libz.so.1 => /lib64/libz.so.1 (0000007f8a8cabe000) /lib64/ld-linux-x86-64.so.2 (0x00007f8a8ec4f000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f8a8c8af000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f8a8c6ab000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f8a8c487000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f8a8c226000) liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f8a8c001000) last bits of an strace of slapd shows: [pid 26360] sendto(3, "<167>Feb 11 15:30:01 slapd[26357"..., 109, MSG_NOSIGNAL, NULL, 0 <unfinished ...> [pid 26359] epoll_wait(6, <unfinished ...> [pid 26360] <... sendto resumed> ) = 109 [pid 26360] --- SIGSEGV {si_signo=SIGSEGV, si_code=SI_KERNEL, si_addr=0} --- [pid 26361] +++ killed by SIGSEGV +++ [pid 260%5] +++ killed by SIGSEGV +++ [pid 26359] +++ killed by SIGSEGV +++ +++ killed by SIGSEGV +++ slapd.log shows: Feb 12 06:53:48 ldapx slapd[30367]: conn=1000 fd=19 ACCEPT from IP=172.17.206.55:42250 (IP=0.0.0.0:389) b b 12 06:53:48 ldapx slapd[30367]: conn=1000 op=0 EXT oid=1.3.6.1.4.1.1466.20037 Feb 12 06:53:48 ldapx slapd[30367]: conn=1000 op=0 STARTTLS Feb 12 06:53:48 ldapx slapd[30367]: conn=1000 op=0 RESULT oid= err=0 text= Feb 12 06:53:48 ldapx slapd[30367]: conn=1000 fd=19 TLS established tls_ssf=256 ssf=256 Feb 12 06:53:56 ldapx slapd[30367]: conn=1000 op=1 BIND dn="cn=Manager,dc=ccd,dc=com" method=128 Feb 12 06:53:56 ldapx slapd[30367]: conn=1000 op=1 BIND dn="cn=Manager,dc=ccd,dc=com" mech=SIMPLE ssf=0 Feb 12 06:53:56 ldapx slapd[30367]: co%3=1000 op=1 RESULT tag=97 err=0 text= Feb 12 06:53:56 ldapx slapd[30367]: conn=1000 op=2 EXT oid=1.3.6.1.4.1.4203.1.11.1 Feb 12 06:53:56 ldapx slapd[30367]: conn=1000 op=2 PASSMOD id="uid=chaliburt,ou=People,dc=ccd,dc=com" new then ththing after this. Good openldap server: Oracle Linux 7 3.8.13-118.2.5.el7uek.x86_64 openldap 2.4.42 # ldd /usr/local/libexec/slapd linux-vdso.so.1 => (0x00007ffd17779000) libltdl.so.7 => /lib64/libltdl.so.7 (0x00007f5e152e2000) libdb-5.3.so => /lib64/libdb-5.3.so (0x00007f5e14f24000) libssl.so.10 => /lib64/libssl.so.10 (0x00007f5e14cb7000) libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f5e148d0000) libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f5e146b6000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5e1449a000) libc.so.6 => /lib64/libc.so.6 (0x00007f5e140dc000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f5e13ed8000) libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f5e13c8c000) libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f5e139a7000) libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f5e137a3000) libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f5e13571000) libz.so.1 => /lib64/libz.so.1 (0x00007f5e1335b000) /lib64/ld-linux-x86-64.so.2 (0x00007f5e154ec000) libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f5e1314c000) libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f5e12f48000) libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f5e12d24000) libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f5e12ac3000) liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f5e1289e000) What else would you like to help figure this out?

1 0

Re: (ITS#8370) SQL test failing
by nwerneck＠gmail.com 10 Feb '16

10 Feb '16

--001a11c2e2980f9e54052b69de0d Content-Type: text/plain; charset=UTF-8 Hi. I understand it is experimental. I just got to a point trying to set it up that I couldn't imagine what I was doing wrong, and decided it could be a bug. Specially considering it's experimental code. I'm actually not even sure I need to use that anymore. But who knows, maybe someone might stumble in the same problem some day... Anyway, I guess we can just close this. Thanks! -- Nicolau Werneck <nwerneck(a)gmail.com> http://n <http://nwerneck.sdf.org>ic.hpavc.net --001a11c2e2980f9e54052b69de0d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Hi. I understand it is experimental. I just got to a = point trying to set it up that I couldn't imagine what I was doing wron= g, and decided it could be a bug. Specially considering it's experiment= al code.</div><div><br></div><div>I'm actually not even sure I need to = use that anymore. But who knows, maybe someone might stumble in the same pr= oblem some day... Anyway, I guess we can just close this. Thanks!</div><div= ><br></div>-- <br><div class=3D"gmail_signature"><div dir=3D"ltr"><div>Nico= lau Werneck <<a href=3D"mailto:nwerneck@gmail.com" target=3D"_blank">nwe= rneck(a)gmail.com</a>><br><a href=3D"http://nwerneck.sdf.org" target=3D"_b= lank">http://n</a><a href=3D"http://ic.hpavc.net" target=3D"_blank">ic.hpav= c.net</a><br></div></div></div> </div> --001a11c2e2980f9e54052b69de0d--

1 0

Re: (ITS#8370) SQL test failing
by quanah＠zimbra.com 09 Feb '16

09 Feb '16

--On Monday, February 08, 2016 3:31 PM +0000 nwerneck(a)gmail.com wrote: > Full_Name: Nicolau Werneck > Version: 2.4.43 > OS: Red Hat Enterprise Linux Server release 7.1 (Maipo) > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (88.217.131.117) > Hello, The ITS system is for reporting bugs, not usage questions. I would also note that back-sql is experimental as documented in the man page. --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc

1 0

Re: (ITS#8303) An Asynchronous meta back-end for OpenLDAP
by nivanova＠symas.com 09 Feb '16

09 Feb '16

A patch for the failing test022 is uploaded as: ftp://ftp.openldap.org/incoming/nadezhda-ivanova-160209.patch

1 0

(ITS#8370) SQL test failing
by nwerneck＠gmail.com 08 Feb '16

08 Feb '16

Full_Name: Nicolau Werneck Version: 2.4.43 OS: Red Hat Enterprise Linux Server release 7.1 (Maipo) URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (88.217.131.117) Hello. I am trying to run the SQL back-end tests, but it fails and I'm not sure how to debug this. The database looks good to me as far as I can tell. There are actually some error messages when I create the database but I'm assuming theyrere not critical. So, this is what I get from running the test: >>>>> Executing all LDAP tests for sql >>>>> Starting sql-test000-read ... running defines.sh Starting slapd on TCP/IP port 9011... Testing SQL backend read operations... Waiting 5 seconds for slapd to start... Testing correct bind... ldap_bind: Invalid credentials (49) ldapwhoami failed (49)! >>>>> ./scripts/sql-test000-read failed (exit 49) make: *** [sql-yes] Error 49 [root%4ocalalhost tests]# And on the log: 56b8b343 connection_get(11): got connid=1001 56b8b343 connection_read(11): checking for input on id=1001 ber_get_next ber_get_next: tag 0x30 len 49 contents: 56b8b343 op tag 0x60, time 1454945091 ber_get_next 56b8b343 conn=1001 op=0 do_bind ber_scanf fmt ({imt) ber: ber_scanf fmt (m}) ber: 56b8b343 >>> dnPrettyNormal: <cn=Mitya Kovalev,dc=example,dc=com> => ldap_bv2dn(cn=Mitya Kovalev,dc=example,dc=m%m,0) <= ldap_bv2dn(cn=Mitya Kovalev,dc=example,dc=com)=0 => ldap_dn2bv(272) <= ldap_dn2bv(cn=Mitya Kovalev,dc=example,dc=com)=0 => ldap_dn2bv(272) <= ldap_dn2bv(cn=mitya kovalev,dc=example,dc=com)=0 56b8b343 <<< dnPrettyNormal: <cn=Mitya Kovalev,dc=example,dc=com>, <cn=mitya kovalev,dc=example,dc=com> 56b8b343 conn=1001 op=0 BIND dn="cn=Mitya Kovalev,dc=example,dc=com" method=128 56b8b343 do_bind: version=3 dn="cn=Mitya Kovalev,dc=example,dc=com" method=128 56b8b343 ==>backsql_bind() 56b8b343 ==>backsql_get_db_conn() 56b8b343 ==>backsql_open_db_handle() 56b8b343 <==backsql_open_db_handle() 56b8b343 <==backsql_get_db_conn() 56b8b343 ==>backsql_attrlist_add(): adding "userPassword" to list 56b8b343 ==>backsqatattrlist_add(): attribute "userPassword" is in list 56b8b343 ==>backsql_attrlist_add(): adding "objectClass" to list 56b8b343 ==>backsql_dn2id("cn=mitya kovalev,dc=example,dc=com") 56b8b343 backsql_dn2id("cn=mitya kovalev,dc=example,dc=com"): id_query "SELECT id,keyval,oc_map_id,dn FROM ldap_entries WHERE upper(dn)=upper(?)" 56b8b343 backsql_dn2id("cn=mitya kovalev,dc=example,dc=com"): id=2 keyval=1 oc_id=1 dn=cn=Mitya Kovalev,dc=example,dc=com 56b8b343 >>> dnPrettyNormal: <cn=Mitya Kovalev,dc=example,dc=com> => ldap_bv2dn(cn=Mitya Kovalev,dc=example,dc=com,0) <= ldap_bv2dn(cn=Mitya Kovalev,dc=example,dc=com)=0 => ldap_dn2bv(272) <= ldap_dn2bv(cn=Mitya Kovalev,dc=example,dc=com)=0 => ldap_2b2bv(272) <= ldap_dn2bv(cn=mitya kovalev,dc=example,dc=com)=0 56b8b343 <<< dnPrettyNormal: <cn=Mitya Kovalev,dc=example,dc=com>, <cn=mitya kovalev,dc=example,dc=com> 56b8b343 <==backsql_dn2id("cn=mitya kovalev,dc=example,dc=com"): err=0 56b8b343 ==>backsqattrtrlist_add(): attribute "userPassword" is in list 56b8b343 ==>backsql_attrlist_add(): attribute "objectClass" is in list 56b8b343 ==>backsql_attrlist_add(): adding "ref" to list 56b8b343 ==>backsql_id2entry() 56b8b343 backsql_id2entry(): custom attribute list 56b8b343 backsql_id2entry(): attribute "userPassword" is not defined for objectlass "inetOrgPerson" 56b8b343 ==>backsql_get_attr_vals(): oc="inetOrgPerson" attr="objectClass" keyval=1 56b8b343 backsql_get_attr_vals(): number of values in query: 0 56b8b343 backsql_id2entry(): attribute "ref" is not defined for objectlass "inetOrgPerson" 56b8b343 <==backsql_id2entry() 56b8b343 send_ldap_result: conn=1001 op=0 p=3 56b8b343 send_ldap_result: err=49 matched="" text="" 56b8b343 send_ldap_response: msgid=1 tag=97 err=49 I guess OpenLDAP cannot read the inetOrgPerson attributes configuration correctly. But I'm just using the schema files from the example, and the `ldap_attr_mappings` table is filled... What else could be missing?

1 0

(ITS#8303)
by nivanova＠symas.com 08 Feb '16

08 Feb '16

This is a patch that fixes a problem in the configure.in introduced with the original commit (6cafdfa8d82134f78e68325c4b9c10dd37935d7a). The problem caused compilation error during make depend. ftp://ftp.openldap.org/incoming/nadezhda-ivanova-160208.patch

1 0

Re: (ITS#8303) An Asynchronous meta back-end for OpenLDAP
by hyc＠symas.com 06 Feb '16

06 Feb '16

nivanova(a)symas.com wrote: > Full_Name: Nadezhda Ivanova > Version: slapd 2.X > OS: > URL: ftp://ftp.openldap.org/incoming/Nadezhda-Ivanova-151105.patch > Submission from: (NULL) (78.83.54.234) > > > This is an asynchronous version of the meta back-end. Something in this patch breaks test022. Reverting the patch (commit 6cafdfa8d82134f78e68325c4b9c10dd37935d7a) fixes the problem. Please investigate. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8367) Unable to authenticate to AD when OpenLDAP 2.4 is set up in SASL mode
by quanah＠zimbra.com 04 Feb '16

04 Feb '16

--On Tuesday, February 02, 2016 9:32 PM +0000 ralthuru(a)yahoo.com wrote: The ITS system is for reporting bugs, not asking how to do stuff. Please use openldap-technical to ask questions on how to use the OpenLDAP software. This ITS will be closed. --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc

1 0

Re: (ITS#8368) Process /usr/sbin/slapd was killed by signal 6 (SIGABRT)
by hyc＠symas.com 04 Feb '16

04 Feb '16

yaroslavr(a)digdes.com wrote: > Full_Name: Yaroslav Rutsky > Version: 2.4.40 > OS: CentOS release 6.6 (Final) > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (85.114.5.9) > > > We used customized OpenLDAP RPM-packets, builded from > openldap-2.4.40-6.el6_7.src.rpm, from CentOS repository with patch to support > Outlook addressbook browsing feature (in conjunction with sssvlv overlay): > What is the root cause of this error ("double free or corruption" at malloc.c)? > What direction we should dig to troubleshoot this error? > Should we monitor memory leaks or try to use tcmalloc instead of malloc? Since it appears that you can reproduce this problem, it ought to be easy to identify the cause using valgrind. Meanwhile, 2.4.40 is 3 releases behind and we're about to release 2.4.44. If you can reproduce the problem in the current RE24 release candidate and track it down in valgrind there may be time to fix it for the 2.4.44 release. Otherwise, you'll have to wait for the next release. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8369) Process /usr/sbin/slapd was killed by signal 6 (SIGABRT)
by yaroslavr＠digdes.com 04 Feb '16

04 Feb '16

Full_Name: Yaroslav Rutsky Version: 2.4.40 OS: CentOS release 6.6 (Final) URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (85.114.5.9) We used customized OpenLDAP RPM-packets, builded from openldap-2.4.40-6.el6_7.src.rpm, from CentOS repository with patch to support Outlook addressbook browsing feature (in conjunction with sssvlv overlay): -----------------------openldap-2.4.40-outlook-sssvlv.patch----------------------------------- diff -uNr openldap-2.4.40/openldap-2.4.40/servers/slapd/schema_prep.c openldap-2.4.40sok/openldap-2.4.40/servers/slapd/schema_prep.c --- openldap-2.4.40/openldap-2.4.40/servers/slapd/schema_prep.c 2014-09-19 05:48:49.000000000 +0400 +++ openldap-2.4.40mod/openldap-2.4.40/servers/slapd/schema_prep.c 2015-10-16 10:43:36.778308938 +0300 @@ -908,6 +908,7 @@ "DESC 'RFC4519: common supertype of name attributes' " "EQUALITY caseIgnoreMatch " "SUBSTR caseIgnoreSubstringsMatch " + "ORDERING caseIgnoreOrderingMatch " "SYNTAX 1.3.6.1.4.1.1466.115.121.1.15{32768} )", NULL, SLAP_AT_ABSTRACT, NULL, NULL, ---------------------------------------------------------------------------------------------- Also, we used LMDB as backend, and some other configurations (chain, syncrepl, referral). This server is slave in our replication structure, serving read requests by themselves and chaining write requests to one of mirror-replicated servers. Modifications, chained from this slave to the mirrored masters, comes back to slave by syncrepl. Recently, we observed occasionally crashed slapd with this error in /var/log/messages: Jan 20 16:51:35 server-02 kernel: slapd[20380] general protection ip:7fe49bf76f85 sp:7fe45d1538d0 error:0 in libc-2.12.so[7fe49bf01000+18a000] Unfortunately, coredump generation configured only after that. Some time later, slapd crashed again: Feb 3 08:05:31 server-02 abrt[24705]: Saved core mpmp of pid 27070 (/usr/sbin/slapd) to /var/spool/abrt/ccpp-2016-02-03-08:05:29-27070 (844488704 bytes) openldap.log (olcLogLevel:256) shows nothing: ==============================================================D%D=== Feb 3 08:05:29 server-02 slapd[27070]: conn=275418 op=617 SEARCH RESULT tag=101 err=0 nentries=0 text= Feb 3 08:05:29 server-02 slapd[27070]: conn=275418 op=618 SRCH base="" scope=2 deref=3 filter="(&(mail=*)(|(?mail=X*)(cn=X*)(sn=X*)(givenName=X*)(displayName=X*))"222 Feb 3 08:05:29 server-02 slapd[27070]: conn=275418 op=618 SRCH attr=cn commonName mail roleOccupant display-name displayname sn surname co organizationName o givenName legacyExchangeDN objectClass uid mailNickname title company physicalDeliveryOfficeName telephoneNumber Feb 3 08:05:29 server-02 slapd[27070]: <= mdb_substring_candidates: (sn) not indexed Feb 3 08:05:29 server-02 slapd[27070]: <= mdb_substring_candidates: (givenName) not indexed Feb 3 08:05:29 server-02 slapd[07070]: <= mdb_substring_candidates: (displayName) not indexed Feb 3 08:24:57 server-02 slapd[26054]: @(#) $OpenLDAP: slapd 2.4.40 (Nov 10 2015 11:22:22) $#012#011root@orw-oldp-01:/root/rpmbuild/BUILD/openldap-2.4.40/openldap-2.4.40/build-servers/servers/slapd Feb 3 08:24:57 server-02 slapd&26055]: slapd starting gdb shows: ================================================================== # gdb /usr/sbin/slapd /var/tmp/coredump/coredump .......... Core was generated by `/usr/sbin/slapd -h ldap:/// ldapi:/// -u ldap'. Proam t terminated with signal 6, Aborted. #0 0x00007f2f4b7bc625 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); Missing separate debuginfos, use: debuginfo-install libtool-ltdl-2.2.6-15.5.el6.x86_64 (gdb) bt #0 0x00007f2f4b7bc625 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00007f2f4b7bde05 in abort () at abort.c:92 #2 0x00007f2f4b7fa537 in __libc_message (do_abort=2, fmt=0x7f2f4b8e2840 "") at ../sysdeps/unix/sysv/linux/libc_fatal.c:198 #3 0x00007f2f4b7ffe66 in malloc_printerr (action=3, str=0x7f2f4b8e2b18 "ree or corruption (!prev)", ptr=<value optimized out>) at malloc.c:6336 #4 0x00007f2f48d8f76f in send_result (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00, so=0f2f2ee0106890) at ../../../../servers/slapd/overlays/sssvlv.c:706 #5 0x00007f2f48d90109 in sssvlv_op_response (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00) at ../../../../servers/slapd/overlays/sssvlv.c:792 #6 0x00007f2f4e22f9de in slap_response_play (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00) at ../../../servers/slapd/result.c:508 #7 0x00007f2f4e2305c0 in send_ldap_response (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00) at ../../../servers/slapd/result.c:583 #8 0x00007f2f4e23158f in slap_send_ldap_result (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00) at ../../../servers/slapd/result.c:861 #9 0x00007f2f4e2ce9dd in mdb_search (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00) at ../../../../servers/slapd/back-mdb/search.c:1164 #10 0x00007f2f4e28e477 in overlay_op_walk (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00, which=op_search, oi=0x7f2f4f1f9a60, on=0x0) at ../../../servers/slapd/backover.c:671 #11 0x00007f2f4e28eed4 in over_op_func (op=0x7f2efe4d6fe0, rs=<value optimized out>, which=<value optimized out>) at ../../../servers/slapd/backover.c:723 #12 0x00007f2f4e221e79 in fe_op_search (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00) at ../../../servers/slapd/search.c:402 #13 0x00007f2f4e28e477 in overlay_op_walk (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00, which=op_search, oi=0x7f2f4f1dd240, on=0x0) at ../../../servers/slapd/backover.c:671 #14 0x00007f2f4e28eed4 in over_op_func (op=0x7f2efe4d6fe0, rs=<value optimized out>, which=<value optimized out>) at ../../../servers/slapd/backover.c:723 #15 0x00007f2f4e2226d7 in do_search (op=0x7f2efe4d6fe0, rs=<value optimized out>) at ../../../servers/slapd/search.c:247 #16 0x00007f2f4e21f349 in connection_operation (ctx=0x7f2eebffeb70, arg_v=0x7f2efe4d6fe0) at ../../../servers/slapd/connection.c:1155 #17 0x00007f2f4e220480 in connection_read_thread (ctx=0x7f2eebffeb70, argv=<value optimized out>) at ../../../servers/slapd/connection.c:1291 #18 0x00007f2f4dd6cb68 in ldap_int_thread_pool_wrapper (xpool=0x7f2f4f184630) at ../../../libraries/libldap_r/tpool.c:688 #19 0x00007f2f4bd309d1 in start_thread (arg=0x7f2eebfff700) at pthread_create.c:301 #20 0x00007f2f4b8728fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 (gdb) bt full 0,6 #0 0x00007f2f4b7bc625 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 resultvar = 0 pid = <value optimized out> selftid = 30625 #1 0x00007f2f4b7bde05 in abort () at abort.c:92 save_stage = 2 act = {__sigaction_handler = {sa_handler = 0x7f2eebe6c458, sa_sigaction = 0x7f2eebe6c458}, sa_mask = {__val = {139839502992448, 139841150654840, 16, 139841107786729, 1, 139841106467759, 5, 139841107790384, 3, 139839502992446, 2, 139841107786755, 1, 139841107793486, 3, 139839502992452}}, sa_flags = 12, sa_restorer = 0x7f2f4b8e1652} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x00007f2f4b7fa537 in __libc_message (do_abort=2, fmt=0x7f2f4b8e2840 "") at ../sysdeps/unix/sysv/linux/libc_fatal.c:198 ap = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 0x7f2eebe6cdc0, reg_save_area = 0x7f2eebe6ccd0}} ap_copy = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 0x7f2eebe6cdc0, reg_save_area = 0x7f2eebe6ccd0}} fd = 2 on_2 = <value optimized out> list = <value optimized out> nlist = <value optimized out> cp = <value optimized out> written = <value optimized out> #3 0x00007f2f4b7ffe66 in malloc_printerr (action=3, str=0x7f2f4b8e2b18 "ree or corruption (!prev)", ptr=<value optimized out>) at malloc.c:6336 buf = "00007f2ee0106890" cp = <value optimized out> #4 0x00007f2f48d8f76f in send_result (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00, so=0x7f2ee0106890) at ../../../../servers/slapd/overlays/sssvlv.c:706 ctrls = {0x7f2ee0004008, 0x7f2ee0005020, 0x0} rc = <value optimized out> i = <value optimized out> #5 0x00007f2f48d90109 in sssvlv_op_response (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00) at ../../../../servers/slapd/overlays/sssvlv.c:792 sc = 0x7f2ee0002a50 so = 0x7f2ee0106890 (More stack frames follow...) (gdb) frame 4 #4 0x00007f2f48d8f76f in send_result (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00, so=0x7f2ee0106890) at ../../../../servers/slapd/overlays/sssvlv.c:706 706 free_sort_op( op->o_conn, so ); (gdb) list 701 slap_add_ctrls( op, rs, ctrls ); 702 send_ldap_result( op, rs ); 703 704 if ( so->so_tree == NULL ) { 705 /* Search finished, so clean up */ 706 free_sort_op( op->o_conn, so ); 707 } 708 } 709 (gdb) frame 3 #3 0x00007f2f4b7ffe66 in malloc_printerr (action=3, str=0x7f2f4b8e2b18 "ree or corruption (!prev)", ptr=<value optimized out>) at malloc.c:6336 6336 __libc_message (action & 2%%D (gdb) list 6331 buf[sizeof (buf) - 1] = '\0'; 6332 char *cp = _itoa_word ((uintptr_t) ptr, &buf[sizeof (buf) - 1], 16, 0); 6333 while (cp > buf) 6334 *--cp = '0'; 6335 6336 ____libc_message (action & 2, 6337 "*** glibc detected *** %s: %s: 0x%s ***\n", 6338 __libc_argv[0] ?: "<unknown>", str, cp); 6339 } 6340 else if (action & 2) ==========================3D%3================================================ What is the root cause of this error ("double free or corruption" at malloc.c)? What direction we should dig to troubleshoot this error? Should we monitor memory leaks or try to use tcmalloc instead of malloc?

1 0

(ITS#8368) Process /usr/sbin/slapd was killed by signal 6 (SIGABRT)
by yaroslavr＠digdes.com 04 Feb '16

04 Feb '16

Full_Name: Yaroslav Rutsky Version: 2.4.40 OS: CentOS release 6.6 (Final) URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (85.114.5.9) We used customized OpenLDAP RPM-packets, builded from openldap-2.4.40-6.el6_7.src.rpm, from CentOS repository with patch to support Outlook addressbook browsing feature (in conjunction with sssvlv overlay): -----------------------openldap-2.4.40-outlook-sssvlv.patch----------------------------------- diff -uNr openldap-2.4.40/openldap-2.4.40/servers/slapd/schema_prep.c openldap-2.4.40sok/openldap-2.4.40/servers/slapd/schema_prep.c --- openldap-2.4.40/openldap-2.4.40/servers/slapd/schema_prep.c 2014-09-19 05:48:49.000000000 +0400 +++ openldap-2.4.40mod/openldap-2.4.40/servers/slapd/schema_prep.c 2015-10-16 10:43:36.778308938 +0300 @@ -908,6 +908,7 @@ "DESC 'RFC4519: common supertype of name attributes' " "EQUALITY caseIgnoreMatch " "SUBSTR caseIgnoreSubstringsMatch " + "ORDERING caseIgnoreOrderingMatch " "SYNTAX 1.3.6.1.4.1.1466.115.121.1.15{32768} )", NULL, SLAP_AT_ABSTRACT, NULL, NULL, ---------------------------------------------------------------------------------------------- Also, we used LMDB as backend, and some other configurations (chain, syncrepl, referral). This server is slave in our replication structure, serving read requests by themselves and chaining write requests to one of mirror-replicated servers. Modifications, chained from this slave to the mirrored masters, comes back to slave by syncrepl. Recently, we observed occasionally crashed slapd with this error in /var/log/messages: Jan 20 16:51:35 server-02 kernel: slapd[20380] general protection ip:7fe49bf76f85 sp:7fe45d1538d0 error:0 in libc-2.12.so[7fe49bf01000+18a000] Unfortunately, coredump generation configured only after that. Some time later, slapd crashed again: Feb 3 08:05:31 server-02 abrt[24705]: Saved core mpmp of pid 27070 (/usr/sbin/slapd) to /var/spool/abrt/ccpp-2016-02-03-08:05:29-27070 (844488704 bytes) openldap.log (olcLogLevel:256) shows nothing: ==============================================================D%D=== Feb 3 08:05:29 server-02 slapd[27070]: conn=275418 op=617 SEARCH RESULT tag=101 err=0 nentries=0 text= Feb 3 08:05:29 server-02 slapd[27070]: conn=275418 op=618 SRCH base="" scope=2 deref=3 filter="(&(mail=*)(|(?mail=X*)(cn=X*)(sn=X*)(givenName=X*)(displayName=X*))"222 Feb 3 08:05:29 server-02 slapd[27070]: conn=275418 op=618 SRCH attr=cn commonName mail roleOccupant display-name displayname sn surname co organizationName o givenName legacyExchangeDN objectClass uid mailNickname title company physicalDeliveryOfficeName telephoneNumber Feb 3 08:05:29 server-02 slapd[27070]: <= mdb_substring_candidates: (sn) not indexed Feb 3 08:05:29 server-02 slapd[27070]: <= mdb_substring_candidates: (givenName) not indexed Feb 3 08:05:29 server-02 slapd[07070]: <= mdb_substring_candidates: (displayName) not indexed Feb 3 08:24:57 server-02 slapd[26054]: @(#) $OpenLDAP: slapd 2.4.40 (Nov 10 2015 11:22:22) $#012#011root@orw-oldp-01:/root/rpmbuild/BUILD/openldap-2.4.40/openldap-2.4.40/build-servers/servers/slapd Feb 3 08:24:57 server-02 slapd&26055]: slapd starting gdb shows: ================================================================== # gdb /usr/sbin/slapd /var/tmp/coredump/coredump .......... Core was generated by `/usr/sbin/slapd -h ldap:/// ldapi:/// -u ldap'. Proam t terminated with signal 6, Aborted. #0 0x00007f2f4b7bc625 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 64 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); Missing separate debuginfos, use: debuginfo-install libtool-ltdl-2.2.6-15.5.el6.x86_64 (gdb) bt #0 0x00007f2f4b7bc625 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00007f2f4b7bde05 in abort () at abort.c:92 #2 0x00007f2f4b7fa537 in __libc_message (do_abort=2, fmt=0x7f2f4b8e2840 "") at ../sysdeps/unix/sysv/linux/libc_fatal.c:198 #3 0x00007f2f4b7ffe66 in malloc_printerr (action=3, str=0x7f2f4b8e2b18 "ree or corruption (!prev)", ptr=<value optimized out>) at malloc.c:6336 #4 0x00007f2f48d8f76f in send_result (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00, so=0f2f2ee0106890) at ../../../../servers/slapd/overlays/sssvlv.c:706 #5 0x00007f2f48d90109 in sssvlv_op_response (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00) at ../../../../servers/slapd/overlays/sssvlv.c:792 #6 0x00007f2f4e22f9de in slap_response_play (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00) at ../../../servers/slapd/result.c:508 #7 0x00007f2f4e2305c0 in send_ldap_response (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00) at ../../../servers/slapd/result.c:583 #8 0x00007f2f4e23158f in slap_send_ldap_result (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00) at ../../../servers/slapd/result.c:861 #9 0x00007f2f4e2ce9dd in mdb_search (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00) at ../../../../servers/slapd/back-mdb/search.c:1164 #10 0x00007f2f4e28e477 in overlay_op_walk (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00, which=op_search, oi=0x7f2f4f1f9a60, on=0x0) at ../../../servers/slapd/backover.c:671 #11 0x00007f2f4e28eed4 in over_op_func (op=0x7f2efe4d6fe0, rs=<value optimized out>, which=<value optimized out>) at ../../../servers/slapd/backover.c:723 #12 0x00007f2f4e221e79 in fe_op_search (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00) at ../../../servers/slapd/search.c:402 #13 0x00007f2f4e28e477 in overlay_op_walk (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00, which=op_search, oi=0x7f2f4f1dd240, on=0x0) at ../../../servers/slapd/backover.c:671 #14 0x00007f2f4e28eed4 in over_op_func (op=0x7f2efe4d6fe0, rs=<value optimized out>, which=<value optimized out>) at ../../../servers/slapd/backover.c:723 #15 0x00007f2f4e2226d7 in do_search (op=0x7f2efe4d6fe0, rs=<value optimized out>) at ../../../servers/slapd/search.c:247 #16 0x00007f2f4e21f349 in connection_operation (ctx=0x7f2eebffeb70, arg_v=0x7f2efe4d6fe0) at ../../../servers/slapd/connection.c:1155 #17 0x00007f2f4e220480 in connection_read_thread (ctx=0x7f2eebffeb70, argv=<value optimized out>) at ../../../servers/slapd/connection.c:1291 #18 0x00007f2f4dd6cb68 in ldap_int_thread_pool_wrapper (xpool=0x7f2f4f184630) at ../../../libraries/libldap_r/tpool.c:688 #19 0x00007f2f4bd309d1 in start_thread (arg=0x7f2eebfff700) at pthread_create.c:301 #20 0x00007f2f4b8728fd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 (gdb) bt full 0,6 #0 0x00007f2f4b7bc625 in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 resultvar = 0 pid = <value optimized out> selftid = 30625 #1 0x00007f2f4b7bde05 in abort () at abort.c:92 save_stage = 2 act = {__sigaction_handler = {sa_handler = 0x7f2eebe6c458, sa_sigaction = 0x7f2eebe6c458}, sa_mask = {__val = {139839502992448, 139841150654840, 16, 139841107786729, 1, 139841106467759, 5, 139841107790384, 3, 139839502992446, 2, 139841107786755, 1, 139841107793486, 3, 139839502992452}}, sa_flags = 12, sa_restorer = 0x7f2f4b8e1652} sigs = {__val = {32, 0 <repeats 15 times>}} #2 0x00007f2f4b7fa537 in __libc_message (do_abort=2, fmt=0x7f2f4b8e2840 "") at ../sysdeps/unix/sysv/linux/libc_fatal.c:198 ap = {{gp_offset = 40, fp_offset = 48, overflow_arg_area = 0x7f2eebe6cdc0, reg_save_area = 0x7f2eebe6ccd0}} ap_copy = {{gp_offset = 16, fp_offset = 48, overflow_arg_area = 0x7f2eebe6cdc0, reg_save_area = 0x7f2eebe6ccd0}} fd = 2 on_2 = <value optimized out> list = <value optimized out> nlist = <value optimized out> cp = <value optimized out> written = <value optimized out> #3 0x00007f2f4b7ffe66 in malloc_printerr (action=3, str=0x7f2f4b8e2b18 "ree or corruption (!prev)", ptr=<value optimized out>) at malloc.c:6336 buf = "00007f2ee0106890" cp = <value optimized out> #4 0x00007f2f48d8f76f in send_result (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00, so=0x7f2ee0106890) at ../../../../servers/slapd/overlays/sssvlv.c:706 ctrls = {0x7f2ee0004008, 0x7f2ee0005020, 0x0} rc = <value optimized out> i = <value optimized out> #5 0x00007f2f48d90109 in sssvlv_op_response (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00) at ../../../../servers/slapd/overlays/sssvlv.c:792 sc = 0x7f2ee0002a50 so = 0x7f2ee0106890 (More stack frames follow...) (gdb) frame 4 #4 0x00007f2f48d8f76f in send_result (op=0x7f2efe4d6fe0, rs=0x7f2eebffea00, so=0x7f2ee0106890) at ../../../../servers/slapd/overlays/sssvlv.c:706 706 free_sort_op( op->o_conn, so ); (gdb) list 701 slap_add_ctrls( op, rs, ctrls ); 702 send_ldap_result( op, rs ); 703 704 if ( so->so_tree == NULL ) { 705 /* Search finished, so clean up */ 706 free_sort_op( op->o_conn, so ); 707 } 708 } 709 (gdb) frame 3 #3 0x00007f2f4b7ffe66 in malloc_printerr (action=3, str=0x7f2f4b8e2b18 "ree or corruption (!prev)", ptr=<value optimized out>) at malloc.c:6336 6336 __libc_message (action & 2%%D (gdb) list 6331 buf[sizeof (buf) - 1] = '\0'; 6332 char *cp = _itoa_word ((uintptr_t) ptr, &buf[sizeof (buf) - 1], 16, 0); 6333 while (cp > buf) 6334 *--cp = '0'; 6335 6336 ____libc_message (action & 2, 6337 "*** glibc detected *** %s: %s: 0x%s ***\n", 6338 __libc_argv[0] ?: "<unknown>", str, cp); 6339 } 6340 else if (action & 2) ==========================3D%3================================================ What is the root cause of this error ("double free or corruption" at malloc.c)? What direction we should dig to troubleshoot this error? Should we monitor memory leaks or try to use tcmalloc instead of malloc?

1 0

Re: (ITS#7992) lmdb: Windows: assume file paths are UTF-8, encode to UTF-16 for WinAPI and enable compiling when UNICODE is defined
by h.b.furuseth＠usit.uio.no 03 Feb '16

03 Feb '16

On 03/02/16 01:39, h.b.furuseth(a)usit.uio.no wrote: > This is still not fixed: The EILSEQ, and a new memleak. > I rewrote it in my branch "mdb/its7992". Untested. Oops, shouldn't have done some late-night code and forgotten to comment it. The important changes are: * "goto leave" - fixes a memleak on failure. * New error code for mdb_strerror(), since EILSEQ is an errno. The rest is just twiddling around: * Drop the always-unused size params to utf8_to_utf16(). * goto fail -- just cleanup, fewer exit points from function. * Looping adds an error check and saves a few bytes object code:-) * Move function inside Doxygen group 'internal'

1 0

Re: (ITS#7992) lmdb: Windows: assume file paths are UTF-8, encode to UTF-16 for WinAPI and enable compiling when UNICODE is defined
by h.b.furuseth＠usit.uio.no 02 Feb '16

02 Feb '16

This is still not fixed: The EILSEQ, and a new memleak. I rewrote it in my branch "mdb/its7992". Untested.

1 0

(ITS#8367) Unable to authenticate to AD when OpenLDAP 2.4 is set up in SASL mode
by ralthuru＠yahoo.com 02 Feb '16

02 Feb '16

Full_Name: Ravi Version: OpenLDAP 2.4 OS: RedHat Linux 6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (209.55.48.135) We have OpenLDAP 2.3 running on Linux. It is set up in SASL mode authenticating against multiple ADs. Everything works fine here. We recently installed a new instance of OpenLDAP 2.4 running on RedHat Linux 6. Then, we moved the slapd.conf and slapd-meta.conf file to the new instance, and created the required users. When we run testsaslauthd, we are successfully able to authenticate against the appropriate AD that the user is under. But we are not able to bind to the OpenLDAP by using the same credentials. I get a Invalid credentials err 49, which indcates either credentials are incorrect, which in this case its not, or the bind info is incorrect. testsaslauthd -u ravi@SONEPAR -p secret - WORKS ldapsearch -x -D uid=ravi,ou=People,ou=company,dc=inside,dc=devserver,dc=com -w secret results in: ldap_bind: Invalid credentials (49) I have searched across many forums, compared the set up on the OpenLDAP 2.3 and OpenLDAP 2.4 instances and cannot find any differences. Any suggestions on how to debug this is appreciated!

1 0

(ITS#8303)
by nivanova＠symas.com 01 Feb '16

01 Feb '16

A new version of the patch with all fixes is uploaded as ftp://ftp.openldap.org/incoming/nadezhda-ivanova-160130.patch

1 0

(ITS#8303)
by nivanova＠symas.com 01 Feb '16

01 Feb '16

This is a multi-part message in MIME format. --------------010703030802080506060500 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit A new version of the patch with all fixes is uploaded as ftp://ftp.openldap.org/incoming/nadezhda-ivanova-160130.patch <ftp://ftp.openldap.org/incoming/Nadezhda-Ivanova-151105.patch> --------------010703030802080506060500 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 7bit <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body bgcolor="#FFFFFF" text="#000000"> A new version of the patch with all fixes is uploaded as <br> <pre><a href="ftp://ftp.openldap.org/incoming/Nadezhda-Ivanova-151105.patch">ftp://ftp.openldap.org/incoming/nadezhda-ivanova-160130.patch</a></pre> </body> </html> --------------010703030802080506060500--

1 0

Re: (ITS#8365) test061 failing
by quanah＠zimbra.com 31 Jan '16

31 Jan '16

--On Sunday, January 31, 2016 5:24 PM +0000 ryan(a)openldap.org wrote: > Full_Name: Ryan Tandy > Version: 2.4.43 > OS: Debian > URL: > Submission from: (NULL) (24.68.37.4) > Submitted by: ryan > > > Testing RE24 discovered that test061 fails sometimes: > > ERROR: Entry 10 not removed on ldap://localhost:9012/! (RC=0) > > Quanah confirmed this with 2.4.43, and also found that 2.4.42 is not > affected. > > Reverting 683c03fb "ITS#8281 fix delta-mmr with interrupted refresh" > seems to cure it for me. I was thinking perhaps a timing issue, but even after giving it up to 55 seconds to replicate, I still hit failures: Running 53 of 500 iterations Running defines.sh Initializing server configurations Starting provider slapd on ldap://localhost:9011/ Starting forward1 slapd on ldap://localhost:9013/ Starting consumer slapd on ldap://localhost:9012/ Adding schema on ldap://localhost:9011/ Adding backend module on ldap://localhost:9011/... Adding schema on ldap://localhost:9012/ Adding backend module on ldap://localhost:9012/... Adding schema on ldap://localhost:9013/ Adding backend module on ldap://localhost:9013/... Adding database configuration on ldap://localhost:9011/ Populating provider on ldap://localhost:9011/ Adding database configuration on ldap://localhost:9013/ Adding database configuration on ldap://localhost:9012/ Using ldapsearch to check that ldap://localhost:9013/ received database... Using ldapsearch to check that ldap://localhost:9012/ received database... Waiting 1 seconds for slapd to receive database... Running 1 of 1 syncrepl initiation race tests... Stopping forwarders for add test Using ldapadd to add 10 entries on provider Starting forwarders again Using ldapadd to add 10 more entries on provider Checking replication to ldap://localhost:9013/ Checking replication to ldap://localhost:9012/ Stopping forwarders for add/delete test Using ldapadd to add 10 entries on provider Using ldapdelete to delete 10 entries on provider Starting forwarders again Using ldapadd to add 10 more entries on provider Using ldapdelete to delete 10 more entries on provider Checking replication to ldap://localhost:9013/ Checking replication to ldap://localhost:9012/ Waiting 1 seconds for ldap://localhost:9012/ to delete entry 10... Waiting 2 seconds for ldap://localhost:9012/ to delete entry 10... Waiting 3 seconds for ldap://localhost:9012/ to delete entry 10... Waiting 4 seconds for ldap://localhost:9012/ to delete entry 10... Waiting 5 seconds for ldap://localhost:9012/ to delete entry 10... Waiting 6 seconds for ldap://localhost:9012/ to delete entry 10... Waiting 7 seconds for ldap://localhost:9012/ to delete entry 10... Waiting 8 seconds for ldap://localhost:9012/ to delete entry 10... Waiting 9 seconds for ldap://localhost:9012/ to delete entry 10... Waiting 10 seconds for ldap://localhost:9012/ to delete entry 10... ERROR: Entry 10 not removed on ldap://localhost:9012/! (RC=0) Error found after 1 of 1 iterations Failed after 53 of 500 iterations -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc

1 0

Re: (ITS#8366) test061 failing
by ryan＠nardis.ca 31 Jan '16

31 Jan '16

Closed the duplicate, sorry for the noise.

1 0

(ITS#8366) test061 failing
by ryan＠openldap.org 31 Jan '16

31 Jan '16

Full_Name: Ryan Tandy Version: 2.4.43 OS: Debian URL: Submission from: (NULL) (24.68.37.4) Submitted by: ryan Testing RE24 discovered that test061 fails sometimes: ERROR: Entry 10 not removed on ldap://localhost:9012/! (RC=0) Quanah confirmed this with 2.4.43, and also found that 2.4.42 is not affected. Reverting 683c03fb "ITS#8281 fix delta-mmr with interrupted refresh" seems to cure it for me.

1 0

(ITS#8365) test061 failing
by ryan＠openldap.org 31 Jan '16

31 Jan '16

Full_Name: Ryan Tandy Version: 2.4.43 OS: Debian URL: Submission from: (NULL) (24.68.37.4) Submitted by: ryan Testing RE24 discovered that test061 fails sometimes: ERROR: Entry 10 not removed on ldap://localhost:9012/! (RC=0) Quanah confirmed this with 2.4.43, and also found that 2.4.42 is not affected. Reverting 683c03fb "ITS#8281 fix delta-mmr with interrupted refresh" seems to cure it for me.

1 0

Re: (ITS#8364) [PATCH] back-meta idassert-bind tls_reqcert=never bug
by mohammad＠securiteam.io 31 Jan '16

31 Jan '16

Oh, thanks for clearing up the confusion, then is there anyway to prevent openldap from sending its server certificate as a client one when connecting to the meta target? I mean other than changing the TLSVerifyClient on the remote host as we don't have access to do this. Regards, Quoting Howard Chu <hyc(a)symas.com>: > mohammad(a)securiteam.io wrote: >> Full_Name: Mohammad Nweider >> Version: master >> OS: Redhat Linux >> URL: >> https://www.securiteam.io/contribs/openldap/mohammad-20160131-0001-fix-back… >> Submission from: (NULL) (89.100.154.148) >> >> >> Hello, >> >> We've found a small bug when trying to run openldap with meta >> backend, what we >> were trying to achieve is to have our server listens on ssl/tls port and to >> communicate with the meta targets over ssl/tls as well, but due to >> the fact that >> we're using a self-signed certificate and we don't have access to manage the >> meta targets, we wanted to skip the client certificate verification when >> connecting to the meta targets, so we tried adding idassert-bind >> tls_reqcert=never to our meta config for this purpose, but unfortunately it >> didn't work as expected. > > There is no bug here. The tls_reqcert setting controls whether the > local node requires the remote target to provide a valid server > certificate. It has nothing to do with client certificates at all. > >> Whenever openldap has a certificate/key either in >> TLSCertificateFile/TLSCertificateKeyFile or in idassert-bind >> tls_cert/tls_key >> settings, it completely ignores tls_reqcert in idassert-bd%d! > > Because the reqcert setting has nothing to do with this. > > Closing this ITS. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8364) [PATCH] back-meta idassert-bind tls_reqcert=never bug
by hyc＠symas.com 31 Jan '16

31 Jan '16

mohammad(a)securiteam.io wrote: > Full_Name: Mohammad Nweider > Version: master > OS: Redhat Linux > URL: https://www.securiteam.io/contribs/openldap/mohammad-20160131-0001-fix-back… > Submission from: (NULL) (89.100.154.148) > > > Hello, > > We've found a small bug when trying to run openldap with meta backend, what we > were trying to achieve is to have our server listens on ssl/tls port and to > communicate with the meta targets over ssl/tls as well, but due to the fact that > we're using a self-signed certificate and we don't have access to manage the > meta targets, we wanted to skip the client certificate verification when > connecting to the meta targets, so we tried adding idassert-bind > tls_reqcert=never to our meta config for this purpose, but unfortunately it > didn't work as expected. There is no bug here. The tls_reqcert setting controls whether the local node requires the remote target to provide a valid server certificate. It has nothing to do with client certificates at all. > Whenever openldap has a certificate/key either in > TLSCertificateFile/TLSCertificateKeyFile or in idassert-bind tls_cert/tls_key > settings, it completely ignores tls_reqcert in idassert-bd%d! Because the reqcert setting has nothing to do with this. Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8364) [PATCH] back-meta idassert-bind tls_reqcert=never bug
by mohammad＠securiteam.io 31 Jan '16

31 Jan '16

Full_Name: Mohammad Nweider Version: master OS: Redhat Linux URL: https://www.securiteam.io/contribs/openldap/mohammad-20160131-0001-fix-back… Submission from: (NULL) (89.100.154.148) Hello, We've found a small bug when trying to run openldap with meta backend, what we were trying to achieve is to have our server listens on ssl/tls port and to communicate with the meta targets over ssl/tls as well, but due to the fact that we're using a self-signed certificate and we don't have access to manage the meta targets, we wanted to skip the client certificate verification when connecting to the meta targets, so we tried adding idassert-bind tls_reqcert=never to our meta config for this purpose, but unfortunately it didn't work as expected. Whenever openldap has a certificate/key either in TLSCertificateFile/TLSCertificateKeyFile or in idassert-bind tls_cert/tls_key settings, it completely ignores tls_reqcert in idassert-bd%d! to reproduce you can just try to build your server with ssl/tls enabled, add the tls global/server settings: TLSCACertificateFile /usr/local/etc/openldap/cacert.pem TLSCertificateFile /usr/local/etc/openldap/servercrt.pem TLSCertificateKeyFile /usr/local/etc/openldap/serverkey.pem add some meta backend targets over ldaps with idassert-bind tls_reqcert=never: database meta suffix "dc=foo,dc=com" uri "ldaps://a.bar.com/dc=foo,dc=com" suffixmassage "dc=foo,dc=com" "dc=bar,dc=org" idassert-bind tls_reqcert=never Enable debugging and try to run some queries against your meta db. You will see a client certificate is sent to the meta target even with tls_reqcert=never! The mplelest fix I could come up with is to add the certificate/key to the ssl context only when is_server or lo->ldo_tls_require_cert is not zero like in the attached patch. Please let me know if I'm misunderstanding something or if this use case can be solved/achieved without this patch. Thanks,

1 0

Re: (ITS#8353) OpenLDAP won't compile with OpenSSL 1.1.X
by hyc＠symas.com 30 Jan '16

30 Jan '16

Howard Chu wrote: > Howard Chu wrote: >> m-oldap(a)bodyfour.uk wrote: >>> Full_Name: Mitchell Blank >>> Version: 2.4.43 >>> OS: linux >>> URL: ftp://ftp.openldap.org/incoming/ >>> Submission from: (NULL) (80.169.198.86) >>> >>> >>> Recently a couple alpha releases for OpenSSL 1.1.X have been posted on >>> www.openssl.org with the request that software be tested against them prior to >>> release. >> >> Thanks for the report. There are clearly 2 issues the OpenSSL folks will have >> to resolve before this will work. >> >> https://mta.openssl.org/pipermail/openssl-dev/2016-January/004362.html >> https://mta.openssl.org/pipermail/openssl-dev/2016-January/004365.html > > The remaining unresolved issue is in their tracker at > https://rt.openssl.org/Ticket/Display.html?id=4274 > Resolved by commit 7ab507495b86371756575d606af556b4fd74e27a All building and working now. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8363) LMDB Heap corruption in mdb_midl_shrink
by hyc＠openldap.org 27 Jan '16

27 Jan '16

Full_Name: Howard Chu Version: 2.4 OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (78.155.228.22) Submitted by: hyc off-by-one error

1 0

Re: (ITS#8354) syncprov segv
by quanah＠zimbra.com 27 Jan '16

27 Jan '16

--On Wednesday, January 27, 2016 9:33 AM +0000 Thomas Pressnell <tpretz(a)gmail.com> wrote: >> Hi Tom, >> >> Are you able to confirm if the fix is good? I'm working on 2.4.44, and >> want to be sure this fix is included. > > I'v had an instance running with this patch for a few days, no SEGV, > looks good to me. Great, thanks for confirming. :) --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#8362) Broken Anchors in 2.4 Administrators Guide
by l.wolf＠xlea.se 27 Jan '16

27 Jan '16

Full_Name: L. Wolf Version: 2.4 OS: doesn't matter URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (87.152.26.226) Section 4.2 -> Anchors "BerkeleyDB" correct href: http://www.oracle.com/technetwork/database/database-technologies/berkeleydb… Section 4.2 -> Anchor "http://www.oracle.com/technology/software/products/berkeley-db/index.html" (also applies to PDF version) correct href: http://www.oracle.com/technetwork/database/database-technologies/berkeleydb…

1 0

Re: (ITS#8354) syncprov segv
by tpretz＠gmail.com 27 Jan '16

27 Jan '16

On Tue, Jan 26, 2016 at 6:49 PM, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > --On Saturday, January 23, 2016 2:43 PM +0000 tpretz(a)gmail.com wrote: > >> Thanks, >> >> Essentially the same patch as i had (destroyed sop->s_mutex in abandon >> check rather than init later). >> >> I did place the o_abandon test under the while( si->si_active ) loop >> instead of above just to ensure the check is under the same lock hold >> as the list modification. >> >> Ill grab whats in git and start tests on monday > > > Hi Tom, > > Are you able to confirm if the fix is good? I'm working on 2.4.44, and want > to be sure this fix is included. I'v had an instance running with this patch for a few days, no SEGV, looks good to me. Thanks > > Thanks! > > --Quanah > > > -- > > Quanah Gibson-Mount > Platform Architect > Zimbra, Inc. > -------------------- > Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#8353) OpenLDAP won't compile with OpenSSL 1.1.X
by hyc＠symas.com 26 Jan '16

26 Jan '16

Howard Chu wrote: > m-oldap(a)bodyfour.uk wrote: >> Full_Name: Mitchell Blank >> Version: 2.4.43 >> OS: linux >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (80.169.198.86) >> >> >> Recently a couple alpha releases for OpenSSL 1.1.X have been posted on >> www.openssl.org with the request that software be tested against them prior to >> release. > > Thanks for the report. There are clearly 2 issues the OpenSSL folks will have > to resolve before this will work. > > https://mta.openssl.org/pipermail/openssl-dev/2016-January/004362.html > https://mta.openssl.org/pipermail/openssl-dev/2016-January/004365.html The remaining unresolved issue is in their tracker at https://rt.openssl.org/Ticket/Display.html?id=4274 -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8361) mdb_strerror() issues
by h.b.furuseth＠usit.uio.no 26 Jan '16

26 Jan '16

Full_Name: Hallvard B Furuseth Version: LMDB 0.9.17 OS: Linux x86_64 URL: Submission from: (NULL) (81.191.45.31) Submitted by: hallvard Unix issues: mdb_strerror uses strerror(), which can sprintf(buf, "Error %d", err) to a thread-unsafe global buffer. In other cases, strerror hopefully returns a pointer to a constant string. This means we can make it hopefully-threadsafe on Unix by documenting "mdb_strerror() should only be used for codes returned by mdb_*() calls". Windows issues: mdb_strerror()'s doc says it extends strerror(). Wrong on Windows. It terminates messages with CRLF. Maybe programs written by Windows users will expect this, but not those from Unix folks. Fix: Strip trailing CRLF, or include FORMAT_MESSAGE_MAX_WIDTH_MASK and then strip trailing space which gets appended instead. The FormatMessage doc says this mask ignores "regular" line breaks but stores "hard-coded" line breaks, I don't know what that means. (Can some system messages have CRLFs inside them?) The pad[] hack breaks if the compiler rearranges locals vars. Fix: Join pad and buf in a struct/array, and drop the (va_list *)pad hack. That hack must be documented. Or replaced, or made a global option. I note that OpenLDAP lutil_debug() does use more tha 4K stack like the mdb_strerror() comment says it shouldn't. Maybe it'd be best to use a thread key by default, but if mdb_set_scarce() or something is called before any mdb_env_create() call, then it will use the current code. But - if mdb_strerror() is called before that again, it'll... init the key itself? Sigh. Also we could add "char *mdb_strerror_r()" which *may* use the user-provided buffer, like the glibc version. Should also add a constant MDB_STRERROR_BUFSIZE as the recommended/minimum buffer size, instead of expecting the programmer to guess the value. That'll at least provide an officially sane alternative. Except - possibly this function would still use strerror() rather than strerror_r() on Unix. The Gnulib (not glibc) comments about strerror_r() portability/bugs are depressing, though hopefully most are outdated: https://www.gnu.org/software/gnulib/manual/html_node/strerror_005fr.html

1 0

Re: (ITS#8354) syncprov segv
by quanah＠zimbra.com 26 Jan '16

26 Jan '16

--On Saturday, January 23, 2016 2:43 PM +0000 tpretz(a)gmail.com wrote: > Thanks, > > Essentially the same patch as i had (destroyed sop->s_mutex in abandon > check rather than init later). > > I did place the o_abandon test under the while( si->si_active ) loop > instead of above just to ensure the check is under the same lock hold > as the list modification. > > Ill grab whats in git and start tests on monday Hi Tom, Are you able to confirm if the fix is good? I'm working on 2.4.44, and want to be sure this fix is included. Thanks! --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#8360) apparent MDB corruption when hitting maxsize
by hyc＠symas.com 26 Jan '16

26 Jan '16

Howard Chu wrote: > peter.van.dijk(a)powerdns.com wrote: >> Full_Name: Peter van Dijk >> Version: 2.4.43 >> OS: Mac OS X 10.10.5 >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (2001:610:666:0:c92b:7aa7:bc73:c430) >> >> >> The procedure follow is fully repeatable with consistent problematic results >> for >> me. > > Unable to reproduce this on Linux using RE24. I had the sequence of steps a bit wrong - running slapcat immediately after ldapmodify masked the problem. Fixed now in git. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8360) apparent MDB corruption when hitting maxsize
by hyc＠symas.com 26 Jan '16

26 Jan '16

peter.van.dijk(a)powerdns.com wrote: > Full_Name: Peter van Dijk > Version: 2.4.43 > OS: Mac OS X 10.10.5 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (2001:610:666:0:c92b:7aa7:bc73:c430) > > > The procedure follow is fully repeatable with consistent problematic results for > me. Unable to reproduce this on Linux using RE24. > (1) ftp://ftp.openldap.org/pub/OpenLDAP/openldap-release/openldap-2.4.43.tgz, > installed via osx homebrew with configure flags > --prefix=/usr/local/Cellar/openlp%p/2.4.43 --sysconfdir=/usr/local/etc > --localstatedir=/usr/local/var > > (2) config and related files as uploaded on the given URL (cosine.schema is > unedited but included for completeness) > > (3) /usr/local/var/openldap-data/ is empty > > (4) sudo /usr/local/opt/openldap/libexec/slapd -d 255 -u peter 2> slapd.log > > (5) ldapmodify -D 'uid=peter,o=habbie' -w secret -c < named.conf.ldif 2> > ldapmodify.log ; ldapdelete -D 'uid=peter,o=habbie' -w secret -r ou=dns,o=habbie > 2> ldapdelete.log > modify is unhappy, initially because of some unknown attributes (thish is why I > used -c), then later because it is hitting the MDB maxsize of 10 megabytes. > ldapdelete then fails to delete the ou=dns tree due to MDB errors > > (6) increasing maxsize after this 'corruption' has happened does NOT enable > ldapdelete to suddenly work ldapdelete works fine for me after raising maxsize. > > (7) (with slapd stopped, not sure it matters): $ > /usr/local/opt/openldap/sbin/slapcat outputs 191771 lines to stdout before > aborting with: > 56a63063 mdb_entry_decode: attribute index 24 not recognized > Segmentation fault: 11 Never saw this, slapcat runs to completion with no complaints. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8360) apparent MDB corruption when hitting maxsize
by peter.van.dijk＠powerdns.com 25 Jan '16

25 Jan '16

Apparently I did not actually enter the right URL. All data is at https://downloads.powerdns.com/tmp/openldap-its-20160125/

1 0

Re: (ITS#8360) apparent MDB corruption when hitting maxsize
by peter.van.dijk＠powerdns.com 25 Jan '16

25 Jan '16

The configure line got messed up somewhere (probably my terminal), should be ==> ./configure --prefix=/usr/local/Cellar/openldap/2.4.43 --sysconfdir=/usr/local/etc --localstatedir=/usr/local/var

1 0

(ITS#8360) apparent MDB corruption when hitting maxsize
by peter.van.dijk＠powerdns.com 25 Jan '16

25 Jan '16

Full_Name: Peter van Dijk Version: 2.4.43 OS: Mac OS X 10.10.5 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (2001:610:666:0:c92b:7aa7:bc73:c430) The procedure follow is fully repeatable with consistent problematic results for me. (1) ftp://ftp.openldap.org/pub/OpenLDAP/openldap-release/openldap-2.4.43.tgz, installed via osx homebrew with configure flags --prefix=/usr/local/Cellar/openlp%p/2.4.43 --sysconfdir=/usr/local/etc --localstatedir=/usr/local/var (2) config and related files as uploaded on the given URL (cosine.schema is unedited but included for completeness) (3) /usr/local/var/openldap-data/ is empty (4) sudo /usr/local/opt/openldap/libexec/slapd -d 255 -u peter 2> slapd.log (5) ldapmodify -D 'uid=peter,o=habbie' -w secret -c < named.conf.ldif 2> ldapmodify.log ; ldapdelete -D 'uid=peter,o=habbie' -w secret -r ou=dns,o=habbie 2> ldapdelete.log modify is unhappy, initially because of some unknown attributes (thish is why I used -c), then later because it is hitting the MDB maxsize of 10 megabytes. ldapdelete then fails to delete the ou=dns tree due to MDB errors (6) increasing maxsize after this 'corruption' has happened does NOT enable ldapdelete to suddenly work (7) (with slapd stopped, not sure it matters): $ /usr/local/opt/openldap/sbin/slapcat outputs 191771 lines to stdout before aborting with: 56a63063 mdb_entry_decode: attribute index 24 not recognized Segmentation fault: 11 (8) as requested by hyc on IRC, copying here for completeness $ mdb_dump -p -s ad2i /usr/local/var/openldap-data/ VERSION=3 format=print database=ad2i type=btree mapsize=10485760 maxreaders=126 tetegerkey=1 db_pagesize=4096 HEADER=END \01\00\00\00 objectClass \02\00\00\00 o \03\00\00\00 structuralObjectClass \04\00\00\00 entryUUID \05\00\00\00 creatorsName \06\00\00\00 createTimestamp \07\00\00\00 entryCSN \08\00\00\00 modifiersName \09\00\00\00 modifyTimestamp \0a\00\00\00 ou \0b\00\00\00 dc \0c\00\00\00 associatedDomain \0d\00\00\00 dNSTTL \0e\00\00\00 sOARecord \0f\00\00\00 nSRecord \10\00\00\00 mXRecord \11\00\00\00 aRecord \12\00\00\00 cMEReRecord \13\00\00\00 LocRecord \14\00\00\00 hInfoRecord \15\00\00\00 tXTRecord \16\00\00\00 aAAARecord DATA=END

1 0

Re: (ITS#8355) [lmdb] Issues with sorted duplicate DBs with cursor delete
by hhclaw.eb＠gmail.com 24 Jan '16

24 Jan '16

On Monday, January 25, 2016 03:49 AM, Howard Chu wrote: > HLaw wrote: >> >> >> On Sunday, January 24, 2016 07:44 PM, Howard Chu wrote: >>> hhclaw.eb(a)gmail.com wrote: >>>> Full_Name: H Law >>>> Version: LMDB >>>> OS: Linux >>>> URL: >>>> Submission from: (NULL) (42.2.241.129) >>>> >>>> >>>> It seems that, after using a cursor delete during a cursor >>>> traversal on a dup >>>> sort database, the cursor is in a strange state, when MDB_NEXT / >>>> MDB_NEXT_DUP >>>> ceases to work properly while MDB_PREV / MDB_PREV_DUP still >>>> functions. >>> >>> Thanks for the report. Fixed now in git. >>> >> Thank you. It now works except for the case when the deleted record >> is just >> before the end of the database. >> In this case the next call to cursor get with MDB_NEXT / MDB_NEXT_DUP >> will >> still hang the program, instead of returning not found (or other >> errors). (It >> seems to be stuck at some wait state without heavy cpu usage). > > Your test program is broken, you are trying to create a new write txn > in the same thread as an existing write txn. > Yes, thanks, works after fixing that. Sorry for bothering.

1 0

Re: (ITS#8355) [lmdb] Issues with sorted duplicate DBs with cursor delete
by hyc＠symas.com 24 Jan '16

24 Jan '16

HLaw wrote: > > > On Sunday, January 24, 2016 07:44 PM, Howard Chu wrote: >> hhclaw.eb(a)gmail.com wrote: >>> Full_Name: H Law >>> Version: LMDB >>> OS: Linux >>> URL: >>> Submission from: (NULL) (42.2.241.129) >>> >>> >>> It seems that, after using a cursor delete during a cursor traversal on a dup >>> sort database, the cursor is in a strange state, when MDB_NEXT / MDB_NEXT_DUP >>> ceases to work properly while MDB_PREV / MDB_PREV_DUP still functions. >> >> Thanks for the report. Fixed now in git. >> > Thank you. It now works except for the case when the deleted record is just > before the end of the database. > In this case the next call to cursor get with MDB_NEXT / MDB_NEXT_DUP will > still hang the program, instead of returning not found (or other errors). (It > seems to be stuck at some wait state without heavy cpu usage). Your test program is broken, you are trying to create a new write txn in the same thread as an existing write txn. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8355) [lmdb] Issues with sorted duplicate DBs with cursor delete
by hhclaw.eb＠gmail.com 24 Jan '16

24 Jan '16

This is a multi-part message in MIME format. --------------010901030100040802060200 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit On Sunday, January 24, 2016 07:44 PM, Howard Chu wrote: > hhclaw.eb(a)gmail.com wrote: >> Full_Name: H Law >> Version: LMDB >> OS: Linux >> URL: >> Submission from: (NULL) (42.2.241.129) >> >> >> It seems that, after using a cursor delete during a cursor traversal >> on a dup >> sort database, the cursor is in a strange state, when MDB_NEXT / >> MDB_NEXT_DUP >> ceases to work properly while MDB_PREV / MDB_PREV_DUP still functions. > > Thanks for the report. Fixed now in git. > Thank you. It now works except for the case when the deleted record is just before the end of the database. In this case the next call to cursor get with MDB_NEXT / MDB_NEXT_DUP will still hang the program, instead of returning not found (or other errors). (It seems to be stuck at some wait state without heavy cpu usage). I attach a version of mtest3.c modified to show the issue. >> In particular, when MDB_NEXT or MDB_NEXT_DUP is called after cursor >> deletion, if >> next key/value pair exists, the cursor will not advance, and got >> stuck by >> returning the same record when MDB_NEXT or MDB_NEXT_DUP is called >> repeatly. In >> case there is no next record, the program was hang. >> >> The following modified version of mtest3.c shows the issue. I am >> testing this >> on the latest commit 20dec1f69bf4860202c764ce92b1fbbe3d11a065 of lmdb >> on 20 Jan, >> on x86-64 Linux. >> I got a similar behaviour when a slightly earlier version of lmdb was >> cross-compiled with a Java wrapper for use on Android, which is why I >> am testing >> this. The issue should therefore not be platform specific. > > > --------------010901030100040802060200 Content-Type: text/x-csrc; name="mtest3.c" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="mtest3.c" /* mtest3.c - memory-mapped database tester/toy */ /* * Copyright 2011-2015 Howard Chu, Symas Corp. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted only as authorized by the OpenLDAP * Public License. * * A copy of this license is available in the file LICENSE in the * top-level directory of the distribution or, alternatively, at * <http://www.OpenLDAP.org/license.html>. */ /* Tests for sorted duplicate DBs with cursor delete */ #include <stdio.h> #include <stdlib.h> #include <string.h> #include <time.h> #include "lmdb.h" #define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr) #define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0)) #define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \ "%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort())) int main(int argc,char * argv[]) { int i = 0, j = 0, rc; MDB_env *env; MDB_dbi dbi; MDB_val key, data; MDB_txn *txn; MDB_stat mst; MDB_cursor *cursor; int count; int *values; char sval[32]; char kval[sizeof(int)]; srand(time(NULL)); memset(sval, 0, sizeof(sval)); count = 10; values = (int *)malloc(count*sizeof(int)); for(i = 0;i<count;i++) { values[i]= i * 10; } E(mdb_env_create(&env)); E(mdb_env_set_mapsize(env, 10485760)); E(mdb_env_set_maxdbs(env, 4)); E(mdb_env_open(env, "./testdb", MDB_FIXEDMAP|MDB_NOSYNC, 0664)); E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_dbi_open(txn, "id2", MDB_CREATE|MDB_DUPSORT, &dbi)); key.mv_size = sizeof(int); key.mv_data = kval; data.mv_size = sizeof(sval); data.mv_data = sval; printf("Adding %d values\n", count); for (i=0;i<count;i++) { if (!(i & 0x07)) sprintf(kval, "%03x", values[i]); sprintf(sval, "%03x %d foo bar", values[i], values[i]); if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA))) j++; } if (j) printf("%d duplicates skipped\n", j); E(mdb_txn_commit(txn)); E(mdb_env_stat(env, &mst)); E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn)); E(mdb_cursor_open(txn, dbi, &cursor)); while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT)) == 0) { printf("key: %p %.*s, data: %p %.*s\n", key.mv_data, (int) key.mv_size, (char *) key.mv_data, data.mv_data, (int) data.mv_size, (char *) data.mv_data); } CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get"); mdb_cursor_close(cursor); mdb_txn_abort(txn); E(mdb_env_stat(env, &mst)); E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_cursor_open(txn, dbi, &cursor)); /* * There are 2 items with key = values[8] at the end of the list */ sprintf(kval, "%03x", values[8]); key.mv_size = sizeof(int); key.mv_data = kval; printf("\nCursor set / first dup\n"); E(mdb_cursor_get(cursor, &key, &data, MDB_SET_KEY)); E(mdb_cursor_get(cursor, &key, &data, MDB_FIRST_DUP)); printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) key.mv_data, (int) data.mv_size, (char *) data.mv_data); printf("Cursor next\n"); j=0; /* * The program will hang at the following mdb_cursor_get * after cursor delete */ while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT_DUP)) == 0) { printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) key.mv_data, (int) data.mv_size, (char *) data.mv_data); j++; if (j == 1) { printf("delete the above key/data\n"); if (RES(MDB_NOTFOUND, mdb_cursor_del(cursor, 0))) { mdb_cursor_close(cursor); mdb_txn_abort(txn); break; } } if (j > count) { printf("Should not be there\n"); break; } } if (j > 1) { mdb_cursor_close(cursor); E(mdb_txn_commit(txn)); } E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_cursor_open(txn, dbi, &cursor)); sprintf(kval, "%03x", values[0]); key.mv_size = sizeof(int); key.mv_data = kval; printf("\nCursor set / last dup\n"); E(mdb_cursor_get(cursor, &key, &data, MDB_SET_KEY)); E(mdb_cursor_get(cursor, &key, &data, MDB_LAST_DUP)); printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) key.mv_data, (int) data.mv_size, (char *) data.mv_data); printf("Cursor prev\n"); j=0; while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV_DUP)) == 0) { printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) key.mv_data, (int) data.mv_size, (char *) data.mv_data); j++; if (j == 1) { printf("Delete the above key/data\n"); if (RES(MDB_NOTFOUND, mdb_cursor_del(cursor, 0))) { mdb_cursor_close(cursor); mdb_txn_abort(txn); break; } } if (j > count) { printf("Should not be there\n"); break; } } if (j > 1) { mdb_cursor_close(cursor); E(mdb_txn_commit(txn)); } mdb_dbi_close(env, dbi); mdb_env_close(env); free (values); return 0; } --------------010901030100040802060200--

1 0

Re: (ITS#8355) [lmdb] Issues with sorted duplicate DBs with cursor delete
by hyc＠symas.com 24 Jan '16

24 Jan '16

hhclaw.eb(a)gmail.com wrote: > Full_Name: H Law > Version: LMDB > OS: Linux > URL: > Submission from: (NULL) (42.2.241.129) > > > It seems that, after using a cursor delete during a cursor traversal on a dup > sort database, the cursor is in a strange state, when MDB_NEXT / MDB_NEXT_DUP > ceases to work properly while MDB_PREV / MDB_PREV_DUP still functions. Thanks for the report. Fixed now in git. > In particular, when MDB_NEXT or MDB_NEXT_DUP is called after cursor deletion, if > next key/value pair exists, the cursor will not advance, and got stuck by > returning the same record when MDB_NEXT or MDB_NEXT_DUP is called repeatly. In > case there is no next record, the program was hang. > > The following modified version of mtest3.c shows the issue. I am testing this > on the latest commit 20dec1f69bf4860202c764ce92b1fbbe3d11a065 of lmdb on 20 Jan, > on x86-64 Linux. > I got a similar behaviour when a slightly earlier version of lmdb was > cross-compiled with a Java wrapper for use on Android, which is why I am testing > this. The issue should therefore not be platform specific. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8354) syncprov segv
by hyc＠symas.com 23 Jan '16

23 Jan '16

Thomas Pressnell wrote: > Thanks, > > Essentially the same patch as i had (destroyed sop->s_mutex in abandon > check rather than init later). > > I did place the o_abandon test under the while( si->si_active ) loop > instead of above just to ensure the check is under the same lock hold > as the list modification. Ah good point. Yeah, I'll move the check down. > > Ill grab whats in git and start tests on monday > > Have a good weekend > > Tom > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8354) syncprov segv
by tpretz＠gmail.com 23 Jan '16

23 Jan '16

Thanks, Essentially the same patch as i had (destroyed sop->s_mutex in abandon check rather than init later). I did place the o_abandon test under the while( si->si_active ) loop instead of above just to ensure the check is under the same lock hold as the list modification. Ill grab whats in git and start tests on monday Have a good weekend Tom

1 0

(ITS#8359) [test] Issues with sorted duplicate DBs with cursor delete
by abcdef.gh＠gmail.com 22 Jan '16

22 Jan '16

Full_Name: H Chu Version: LMDB OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (78.155.231.129) It seems that, after using a cursor delete during a cursor traversal on a dup sort database, the cursor is in a strange state, when MDB_NEXT / MDB_NEXT_DUP ceases to work properly while MDB_PREV / MDB_PREV_DUP still functions. In particular, when MDB_NEXT or MDB_NEXT_DUP is called after cursor deletion, if next key/value pair exists, the cursor will not advance, and got stuck by returning the same record when MDB_NEXT or MDB_NEXT_DUP is called repeatly. In case there is no next record, the program was hang. The following modified version of mtest3.c shows the issue. I am testing this on the latest commit 20dec1f69bf4860202c764ce92b1fbbe3d11a065 of lmdb on 20 Jan, on x86-64 Linux. I got a similar behaviour when a slightly earlier version of lmdb was cross-compiled with a Java wrapper for use on Android, which is why I am testing this. The issue should therefore not be platform specific. Thank you. ==== /* mtest3.c - memory-mapped database tester/y y */ /* * Copyrit t 2011-2015 Howard Chu, Symas Corp. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted only as authorized by the OpenLDAP * Public License. * * A copy of this license is available in the file LICENSE in the * top-level directory of the distribution or, alternatively, at * <http://www.OpenLDAP.org/license.html>. */ /* Tests for sorted duplicate DBs with cursor delete */ #include <stdio.h> #include <stib.h.h> #include <string.h> #include <time.h> #include "lmdb.h" #define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr) #define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0)) #define CHECK(test, msg) ((test) ? (void)%3: ((void)fprintf(stderr, \ "%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort())) int main(int argc,char * argv[]) { int i = 0, j = 0, rc; MDB_env *env; MDB_dbi dbi; MDB_val key, data; MDB_txn *txn; MDB_stat mst; MDB_cursor *cursor; int count; int *values; char sval[32]; char kval[sizeof(int)]; srand(time(NULL)); memset(sval, 0, sizeof(sval)); count = 10; values = (int *)malloc(count*sizeof(int)); for(i = 0;i<count;i++) { values[i]= i * 10; } E(mdb_env_create(&env)); E(mdb_env_set_mapsize(env, 10485760)); E(mdb_env_set_maxdbs(env, 4)); E(mdb_env_open(env, "./testdb", MDB_FIXEDMAP|MDB_NOSYNC, 0664)); E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_dbi_open(n%n, "id2", MDB_CREATE|MDB_DUPSORT, &dbi)); key.mv_size = sizeof(int); key.mv_data = kval; data.mv_size = sizeof(sval); data.mv_data = sval; printf("Adding %d values\n", count); for (i=0;i<count;i++) { if (!(i & 0x07)) sprintf(kval, "%03x",alalues[i])0D0D sprintf(sval, "%03x %d foo bar", values[i], values[i]); if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA))) j++; } if (j) printf("%d duplicates skipped\n", j); E(mdb_txn_commit(txn)); E(mdb_env_stat(vnv, &mst)); E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn)); E(mdb_cursor_open(txn, dbi, &cursor)); while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT_DUP)) == 0) { printf("key: %p %.*s, data: %p %.*s\n", key.mv_data, (int) key.mv_size, (char *) key.mv_data, data.mv_data, (int) data.mv_size, (char *) data.mv_data); } CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get"); mdb_cursor_close(cursor); mdb_txn_abort(txn); E(mdb_env_stat(env, &mst)); E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_cursor_open(txn% d dbi, &cursor)); /* * changing following to values[8] freezes program during mdb_cursor_del * as there is no more record following the deletion */ sprintf(kval, "%03x", values[0]); key.mv_size = sizeof(i)%9; key.mv_data = kval; printf("\nCursor set / first dup\n"); E(mdb_cursor_get(cursor, &key, &data, MDB_SET_KEY)); E(mdb_cursor_get(cursor, &key, &data, MDB_FIRST_DUP)); printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) key.mv_data, (int) data.mv_size, (char *) data.mv_data); printf("Cursor next\n"); j=0; while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT_DUP)) == 0) { printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) key.mv_data, (int) data.mv_size, (char *) data.mv_data); j++; if (j == 1) { printf("delete the above key/data\n"); if (RES(MDB_NOTFOUND, mdb_cursor_del(cursor, 0))) { mdb_cursor_close(cursor); mdb_txn_abort(txn); break; } } if (j > count) { printf("Should not be there\n"); break; } } if (j E E 1) { mdb_cursor_close(cursor); E(mdb_txn_commit(txn)); } E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_cursor_open(txn, dbi, &cursor)); sprintf(kval, "%03x", values[0]); key.mv_size = sizeof(int); key.mv_data = kval; printf("\nCursor set / last dup\n"); E(mdb_cursor_get(cursor, &key, &data, MDB_SET_KEY)); E(mdb_cursor_get(cursor, &key, &data, MDB_LAST_DUP)); printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) key.mv_data, (int) data.mv_size, (char *) data.mv_data); printf("Cursor prev\n"); j=0; while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_PREV_DUP)) == 0) { printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) key.mv_data, (int) data.mv_size, (char *) data.mv_data); j++; if (j == 1) { printf("Delete the above key/data\n"); if (RES(MDB_NOTFOUND, mdb_cursor_del(cursor, 0))) { mdb_cursor_close(cursor); mdb_txn_abort(txn); break; } } if (j > count) { printf("Should not be there\n"); break; } } if (j > 1) { mdb_cursor_close(cursor); E(mdb_txn_commit(txn)); } mdb_dbi_close(env, dbi); mdb_env_close(env); free (values); return 0; } === Output: Adding 10 values key: 0x7f2c31981e9c 000, data: 0x7f2c31981fe0 000 0 foo bar key: 0x7f2c31981e9c 000, data: 0x7f2c31981fb8 00a 10 foo bar key: 0x7f2c31981e9c 000, data: 0x7f2c31981f90 014 20 foo bar key: 0x7f2c31981e9c 000, data: 0x7f2c31981f68 01e 30 foo bar key: 0x7f2c31981e9c 000, data: 0x7f2c31981f40 028 40 foo bar key: 0x7f2c31981e9c 000, data: 0x7f2c31981f18 032 50 foo bar key: 0x7f2c31981e9c 000, data: 0x7f2c31981ef0 03c 60 foo bar key: 0x7f2c31981e9c 000, data: 0x7f2c31981ec8 046 70 foo bar key: 0x7f2c31981e2c 050, data: 0x7f2c31981e74 050 80 foo bar key: 0x7f2c31981e2c 050, data: 0x7f2c31981e4c 05a 90 foo bar Cursor set / first dup key: 000, data: 000 0 foo bar Cursor next key: 000, data: 00a 10 foo bar delete the above key/data key: 000, data: 014 20 foo bar <--- Expected to get 30, 40, ... 70 for MDB_NEXT_DUP key: 000, data: 014 20 foo bar key: 000, data: 014 20 foo bar key: 000, data: 014 20 foo bar key: 000, data: 014 20 foo bar key: 000, data: 014 20 foo bar key: 000, data: 014 20 foo bar key: 000, data: 014 20 foo bar key: 000, data: 014 20 foo bar key: 000, data: 014 20 foo bar <---- But got infinite loop if not break by counter Should not be there Cursor set / last dup key: 000, data: 046 70 foo bar Cursor prev key: 000, data: 03c 60 foo bar Delete the above key/data key: 000, data:3232 50 foo bar <--- works as expected in the reverse direction key: 000, data: 028 40 foo bar key: 000, data: 01e 30 foo bar key: 000, data: 014 20 foo bar key: 000, data: 000 0 foo bar

1 0

(ITS#8358) [test] [lmdb] Issues with sorted duplicate DBs with cursor delete
by hyc＠openldap.org 22 Jan '16

22 Jan '16

Full_Name: H Chu Version: LMDB OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (78.155.231.129) Submitted by: hyc It seems that, after using a cursor delete during a cursor traversal on a dup sort database, the cursor is in a strange state, when MDB_NEXT / MDB_NEXT_DUP ceases to work properly while MDB_PREV / MDB_PREV_DUP still functions. In particular, when MDB_NEXT or MDB_NEXT_DUP is called after cursor deletion, if next key/value pair exists, the cursor will not advance, and got stuck by returning the same record when MDB_NEXT or MDB_NEXT_DUP is called repeatly. In case there is no next record, the program was hang. The following modified version of mtest3.c shows the issue. I am testing this on the latest commit 20dec1f69bf4860202c764ce92b1fbbe3d11a065 of lmdb on 20 Jan, on x86-64 Linux. I got a similar behaviour when a slightly earlier version of lmdb was cross-compiled with a Java wrapper for use on Android, which is why I am testing this. The issue should therefore not be platform specific. Thank you. ==== /* mtest3.c - memory-mapped database tester/toy */ /* * Copyrit t 2011-2015 Howard Chu, Symas Corp. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted only as authorized by the OpenLDAP * Public License. * * A copy of this license is available in the file LICENSE in the * top-level directory of the distribution or, alternatively, at * <http://www.OpenLDAP.org/license.html>. */ /* Tests for sorted duplicate DBs with cursor delete */ #include <stdio.h> #include <stdlib.h> #include <string.h> #include <time.h> #include "lmdb.h" #define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr) #define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0)) #define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \ "%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort())) int main(int argc,char * argv[]) { int i = 0, j = 0, rc; MDB_env *env; MDB_dbi dbi; MDB_val key, data; MDB_txn *txn; MDB_stat mst; MDB_cursor *cursor; int count; int *values; char sval[32]; char kval[sizeof(int)]; srand(time(NULL)); memset(sval, 0, sizeof(sval)); count = 10; values = (int *)malloc(count*sizeof(int)); for(i =%0;i<count;i++) { values[i]= i * 10; } E(mdb_env_create(&env)); E(mdb_env_set_mapsize(env, 10485760)); E(mdb_env_set_maxdbs(env, 4)); E(mdb_env_open(env, "./testdb", MDB_FIXEDMAP|MDB_NOSYNC, 0664)); E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_dbi_open(txn, "id2", MDB_CREATE|MDB_DUPSORT, &dbi)); key.mv_size = sizeof(int); key.mv_data = kval; data.mv_size = sizeof(sval); data.mv_data = sval; printf("Adding %d values\n", count); for (i=0;i<count;i++) { if (!(i & 0x07)) 9%9 sprintf(kval, "%03x",alalues[i]); sprintf(sval, "%03x %d foo bar", values[i], values[i]); if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA))) j++; } if (j) printf("%d duplicates skipped\n", j); E(mdb_txn_commit(txn)); E(mdb_env_stat(env, &mst)); E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn)); E(mdb_cursor_open(txn, dbi, &cursor)); while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT_DUP)) == 0) { printf("key: %p %.*s, data: %p %.*s\n", key.mv_data, %int%t) key.mv_size, (char *) key.mv_data, data.mv_data, (int) data.mv_size, (char *) data.mv_data); } CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get"); mdb_cursor_close(cursor); mdb_txn_abort(txn); E(mdb_env_stat(env, &mst)); E(mdb_txn_begin(v,2C NULL, 0, &txn)); E(mdb_cursor_open(txn, dbi, &cursor)); /* * changing following to values[8] freezes program during mdb_cursor_del * as there is no more record following the deletion */ sprintf(kval, "%03x", values[0]); key.mv_size = sizeof(int); key.mv_data = kval; printf("\nCursor set / first dup\n"); E(mdb_cursor_get(cursor, &key, &data, MDB_SET_KEY)); E(mdb_cursor_get(cursor, &key, &data, MDB_FIRST_DUP)); printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) key.mv_data, (int) data.mv_size, (char *) data.mv_data); printf("Cursor next\n"); j=0; while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT_DUP)) == 0) { printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) keyv_dadata, (int) data.mv_size, (char *) data.mv_data); j++; if (j == 1) { printf("delete the above key/data\n"); if (RES(MDB_NOTFOUND, mdb_cursor_del(cursor, 0))) {%%0 mdb_cursor_close(cursor); mdb_txn_abort(txn); break; } } if (j > count) { printf("Should not be there\n"); break; } } if (j > 1) { mdb_cursor_close(cursor); E(mdb_txn_commit(txn)); } E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_cursor_open(txn, dbi, &cursor)); sprintf(kval, "%03x", values[0]); key.mv_size = sizeof(int); key.mv_data = kval; printf("\nCursor set / last dup\n"); E(mdb_cursor_get(cursor, &key, &data, MDB_SET_KEY)); E(mdb_cursor_get(cursor, &key, &data, MDB_LAST_DUP)); printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) key.mv_data,

1 0

(ITS#8357) [test] [lmdb] Issues with sorted duplicate DBs with cursor delete
by hyc＠openldap.org 22 Jan '16

22 Jan '16

Full_Name: H Chu Version: LMDB OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (78.155.231.129) Submitted by: hyc It seems that, after using a cursor delete during a cursor traversal on a dup sort database, the cursor is in a strange state, when MDB_NEXT / MDB_NEXT_DUP ceases to work properly while MDB_PREV / MDB_PREV_DUP still functions. In particular, when MDB_NEXT or MDB_NEXT_DUP is called after cursor deletion, if next key/value pair exists, the cursor will not advance, and got stuck by returning the same record when MDB_NEXT or MDB_NEXT_DUP is called repeatly. In case there is no next record, the program was hang. The following modified version of mtest3.c shows the issue. I am testing this on the latest commit 20dec1f69bf4860202c764ce92b1fbbe3d11a065 of lmdb on 20 Jan, on x86-64 Linux. I got a similar behaviour when a slightly earlier version of lmdb was cross-compiled with a Java wrapper for use on Android, which is why I am testing this. The issue should therefore not be platform specific. Thank you. ==== /* mtest3.c - memory-mapped database tester/toy */ /* * Copyrit t 2011-2015 Howard Chu, Symas Corp. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted only as authorized by the OpenLDAP * Public License. * * A copy of this license is available in the file LICENSE in the * top-level directory of the distribution or, alternatively, at * <http://www.OpenLDAP.org/license.html>. */ /* Tests for sorted duplicate DBs with cursor delete */ #include <stdio.h> #include <stdlib.h> #include <string.h> #include <time.h> #include "lmdb.h" #define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr) #define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0)) #define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \ "%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort())) int main(int argc,char * argv[]) { int i = 0, j = 0, rc; MDB_env *env; MDB_dbi dbi; MDB_val key, data; MDB_txn *txn; MDB_stat mst; MDB_cursor *cursor; int count; int *values; char sval[32]; char kval[sizeof(int)]; srand(time(NULL)); memset(sval, 0, sizeof(sval)); count = 10; values = (int *)malloc(count*sizeof(int)); for(i =%0;i<count;i++) { values[i]= i * 10; } E(mdb_env_create(&env)); E(mdb_env_set_mapsize(env, 10485760)); E(mdb_env_set_maxdbs(env, 4)); E(mdb_env_open(env, "./testdb", MDB_FIXEDMAP|MDB_NOSYNC, 0664)); E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_dbi_open(txn, "id2", MDB_CREATE|MDB_DUPSORT, &dbi)); key.mv_size = sizeof(int); key.mv_data = kval; data.mv_size = sizeof(sval); data.mv_data = sval; printf("Adding %d values\n", count); for (i=0;i<count;i++) { if (!(i & 0x07)) 9%9 sprintf(kval, "%03x",alalues[i]); sprintf(sval, "%03x %d foo bar", values[i], values[i]); if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA))) j++; } if (j) printf("%d duplicates skipped\n", j); E(mdb_txn_commit(txn)); E(mdb_env_stat(env, &mst)); E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn)); E(mdb_cursor_open(txn, dbi, &cursor)); while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT_DUP)) == 0) { printf("key: %p %.*s, data: %p %.*s\n", key.mv_data, %int%t) key.mv_size, (char *) key.mv_data, data.mv_data, (int) data.mv_size, (char *) data.mv_data); } CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get"); mdb_cursor_close(cursor); mdb_txn_abort(txn); E(mdb_env_stat(env, &mst)); E(mdb_txn_begin(v,2C NULL, 0, &txn)); E(mdb_cursor_open(txn, dbi, &cursor)); /* * changing following to values[8] freezes program during mdb_cursor_del * as there is no more record following the deletion */ sprintf(kval, "%03x", values[0]); key.mv_size = sizeof(int); key.mv_data = kval; printf("\nCursor set / first dup\n"); E(mdb_cursor_get(cursor, &key, &data, MDB_SET_KEY)); E(mdb_cursor_get(cursor, &key, &data, MDB_FIRST_DUP)); printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) key.mv_data, (int) data.mv_size, (char *) data.mv_data); printf("Cursor next\n"); j=0; while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT_DUP)) == 0) { printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) keyv_dadata, (int) data.mv_size, (char *) data.mv_data); j++; if (j == 1) { printf("delete the above key/data\n"); if (RES(MDB_NOTFOUND, mdb_cursor_del(cursor, 0))) {%%0 mdb_cursor_close(cursor); mdb_txn_abort(txn); break; } } if (j > count) { printf("Should not be there\n"); break; } } if (j > 1) { mdb_cursor_close(cursor); E(mdb_txn_commit(txn)); } E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_cursor_open(txn, dbi, &cursor)); sprintf(kval, "%03x", values[0]); key.mv_size = sizeof(int); key.mv_data = kval; printf("\nCursor set / last dup\n"); E(mdb_cursor_get(cursor, &key, &data, MDB_SET_KEY)); E(mdb_cursor_get(cursor, &key, &data, MDB_LAST_DUP)); printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) key.mv_data,

1 0

(ITS#8356) [test] [lmdb] Issues with sorted duplicate DBs with cursor delete
by hyc＠openldap.org 22 Jan '16

22 Jan '16

Full_Name: H Chu Version: LMDB OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (78.155.231.129) Submitted by: hyc It seems that, after using a cursor delete during a cursor traversal on a dup sort database, the cursor is in a strange state, when MDB_NEXT / MDB_NEXT_DUP ceases to work properly while MDB_PREV / MDB_PREV_DUP still functions. In particular, when MDB_NEXT or MDB_NEXT_DUP is called after cursor deletion, if next key/value pair exists, the cursor will not advance, and got stuck by returning the same record when MDB_NEXT or MDB_NEXT_DUP is called repeatly. In case there is no next record, the program was hang. The following modified version of mtest3.c shows the issue. I am testing this on the latest commit 20dec1f69bf4860202c764ce92b1fbbe3d11a065 of lmdb on 20 Jan, on x86-64 Linux. I got a similar behaviour when a slightly earlier version of lmdb was cross-compiled with a Java wrapper for use on Android, which is why I am testing this. The issue should therefore not be platform specific. Thank you. ==== /* mtest3.c - memory-mapped database tester/toy */ /* * Copyrit t 2011-2015 Howard Chu, Symas Corp. * All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted only as authorized by the OpenLDAP * Public License. * * A copy of this license is available in the file LICENSE in the * top-level directory of the distribution or, alternatively, at * <http://www.OpenLDAP.org/license.html>. */ /* Tests for sorted duplicate DBs with cursor delete */ #include <stdio.h> #include <stdlib.h> #include <string.h> #include <time.h> #include "lmdb.h" #define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr) #define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0)) #define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf(stderr, \ "%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort())) int main(int argc,char * argv[]) { int i = 0, j = 0, rc; MDB_env *env; MDB_dbi dbi; MDB_val key, data; MDB_txn *txn; MDB_stat mst; MDB_cursor *cursor; int count; int *values; char sval[32]; char kval[sizeof(int)]; srand(time(NULL)); memset(sval, 0, sizeof(sval)); count = 10; values = (int *)malloc(count*sizeof(int)); for(i =%0;i<count;i++) { values[i]= i * 10; } E(mdb_env_create(&env)); E(mdb_env_set_mapsize(env, 10485760)); E(mdb_env_set_maxdbs(env, 4)); E(mdb_env_open(env, "./testdb", MDB_FIXEDMAP|MDB_NOSYNC, 0664)); E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_dbi_open(txn, "id2", MDB_CREATE|MDB_DUPSORT, &dbi)); key.mv_size = sizeof(int); key.mv_data = kval; data.mv_size = sizeof(sval); data.mv_data = sval; printf("Adding %d values\n", count); for (i=0;i<count;i++) { if (!(i & 0x07)) 9%9 sprintf(kval, "%03x",alalues[i]); sprintf(sval, "%03x %d foo bar", values[i], values[i]); if (RES(MDB_KEYEXIST, mdb_put(txn, dbi, &key, &data, MDB_NODUPDATA))) j++; } if (j) printf("%d duplicates skipped\n", j); E(mdb_txn_commit(txn)); E(mdb_env_stat(env, &mst)); E(mdb_txn_begin(env, NULL, MDB_RDONLY, &txn)); E(mdb_cursor_open(txn, dbi, &cursor)); while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT_DUP)) == 0) { printf("key: %p %.*s, data: %p %.*s\n", key.mv_data, %int%t) key.mv_size, (char *) key.mv_data, data.mv_data, (int) data.mv_size, (char *) data.mv_data); } CHECK(rc == MDB_NOTFOUND, "mdb_cursor_get"); mdb_cursor_close(cursor); mdb_txn_abort(txn); E(mdb_env_stat(env, &mst)); E(mdb_txn_begin(v,2C NULL, 0, &txn)); E(mdb_cursor_open(txn, dbi, &cursor)); /* * changing following to values[8] freezes program during mdb_cursor_del * as there is no more record following the deletion */ sprintf(kval, "%03x", values[0]); key.mv_size = sizeof(int); key.mv_data = kval; printf("\nCursor set / first dup\n"); E(mdb_cursor_get(cursor, &key, &data, MDB_SET_KEY)); E(mdb_cursor_get(cursor, &key, &data, MDB_FIRST_DUP)); printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) key.mv_data, (int) data.mv_size, (char *) data.mv_data); printf("Cursor next\n"); j=0; while ((rc = mdb_cursor_get(cursor, &key, &data, MDB_NEXT_DUP)) == 0) { printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) keyv_dadata, (int) data.mv_size, (char *) data.mv_data); j++; if (j == 1) { printf("delete the above key/data\n"); if (RES(MDB_NOTFOUND, mdb_cursor_del(cursor, 0))) {%%0 mdb_cursor_close(cursor); mdb_txn_abort(txn); break; } } if (j > count) { printf("Should not be there\n"); break; } } if (j > 1) { mdb_cursor_close(cursor); E(mdb_txn_commit(txn)); } E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_cursor_open(txn, dbi, &cursor)); sprintf(kval, "%03x", values[0]); key.mv_size = sizeof(int); key.mv_data = kval; printf("\nCursor set / last dup\n"); E(mdb_cursor_get(cursor, &key, &data, MDB_SET_KEY)); E(mdb_cursor_get(cursor, &key, &data, MDB_LAST_DUP)); printf("key: %.*s, data: %.*s\n", (int) key.mv_size, (char *) key.mv_data,

1 0

Re: (ITS#8350) lmdb SIGBUS error on full partition and possible double free issue
by hyc＠symas.com 22 Jan '16

22 Jan '16

jeremiah.morrill(a)econnect.tv wrote: > Full_Name: Jeremiah Morrill > Version: 0.9 > OS: Linux (Ubuntu14) > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (70.173.183.164) > > > Two possible issues. Semi-related. > > The first: > On a full storage partition, when creating a new database, I get a SIGBUS. I > believe it is caused by the locks successfully mmap()ing, but not really having > the storage to back it. I hacked in a "posix_fallocate" to make sure the > storage space is there and it appeared to fix it. I have no idea what the > unintended consequences of this change may be. It might indeed be a good idea to use posix_fallocate, but then we're stuck trying to figure out which platforms actually support it. In the meantime, if we know it's available, then it should simply replace the ftruncate() call which we were already using to set the lockfile size. For portability reasons I'd be inclined not to use fallocate at all, and just perform a series of dummy writes() to grow the file to the proper size. Ultimately I think it's not a big deal; if your program crashes on startup because there's no disk space, you haven't lost anything and you know that there's a system problem you need to fix. > Here is the diff: > > void > @@ -4863,6 +4868,14 @@ mdb_env_setup_locks(MDB_env *env, char *lpath, int mode, > int *excl) > void *m = mmap(NULL, rsize, PROT_READ|PROT_WRITE, MAP_SHARED, > env->me_lfd, 0); > if (m == MAP_FAILED) goto fail_errno; > + > + rc = posix_fallocate(env->me_lfd, 0, rsize); > + > + if (rc) { > + munmap(m, rsize); > + goto fail; > + } > + > env->me_txns = m; > #endif -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8354) syncprov segv
by hyc＠symas.com 22 Jan '16

22 Jan '16

Thomas Pressnell wrote: > Hi, > > Looking beyond syncprov.c:2574 o_abandon can be set, however as i > understand it once the syncop has been added to the si_ops list its > cleanup will be handled by syncprov_op_abandon, we would just need to > ensure it has not traversed syncprov_op_abandon before being added to > the list. > > Reading the rest of syncprov_op_search and syncprov_search_response i > notice that syncprov_search_response does check o_abandon (when > deciding if to detach the psearch). > > In the event of an abandon syncprov_search_response seems to assume > syncprov_op_abandon will take care of the cleanup of the syncop, which > it wouldn't if syncprov_op_abandon had already been executed before > the syncop was added to the si_ops list @~ 2574 OK, thanks. A fix is now in git master. Probably about the same as the one you already described, but please test and feedback. > > I would then assume the op is freed, but the syncop remains in the > list to be processed. Once reallocated and the o_abandon field is no > longer set syncprov_matchops would start to include the syncop in its > processing hitting potential issues such as this segv. > > Let me know if this ties up with how things really work or if i have > misunderstood something along the way. > > Thanks > > Tom > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8354) syncprov segv
by tpretz＠gmail.com 22 Jan '16

22 Jan '16

Hi, Looking beyond syncprov.c:2574 o_abandon can be set, however as i understand it once the syncop has been added to the si_ops list its cleanup will be handled by syncprov_op_abandon, we would just need to ensure it has not traversed syncprov_op_abandon before being added to the list. Reading the rest of syncprov_op_search and syncprov_search_response i notice that syncprov_search_response does check o_abandon (when deciding if to detach the psearch). In the event of an abandon syncprov_search_response seems to assume syncprov_op_abandon will take care of the cleanup of the syncop, which it wouldn't if syncprov_op_abandon had already been executed before the syncop was added to the si_ops list @~ 2574 I would then assume the op is freed, but the syncop remains in the list to be processed. Once reallocated and the o_abandon field is no longer set syncprov_matchops would start to include the syncop in its processing hitting potential issues such as this segv. Let me know if this ties up with how things really work or if i have misunderstood something along the way. Thanks Tom

1 0

Re: (ITS#8354) syncprov segv
by hyc＠symas.com 22 Jan '16

22 Jan '16

tpretz(a)gmail.com wrote: > This thread is working with the same operation ...0aa0, but performing a > standard search as i would expect given the filter value. > Somehow ss->s_op seems to have ended up pointing at what seems to be an > unreleated operation. > > Looking at the code i believe the issue could trigger when an op is abandoned > early before syncprov_op_search has got hold of the si_ops lock for the psearch > sop. OK, this sort of makes sense. But abandoning the op doesn't free it immediately, the frontend can't do that until all of the request functions have returned. And the op structure can't get freed and reused by some other op until after that. > I have added a standard o_abandon check and return at line 2574 of syncprov.c > while the si_ops lock is held, before sop is added to the list. > This seems to have fixed the issue in my testing, i can see this code path is > traversed (as i am logging it) a number of times over the last few days of > running the tests. But this can't be an adequate fix, since op->o_abandon could be set at any time after line 2574 as well. > I can provide more detailed backtraces if required. > If you would like core dumps this will require extra time as i would have to > replicate the test with non company data / schemas. > > Thanks > > Tom > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs