openldap-bugs

openldap-bugs@openldap.org

1 participants
21001 discussions

Re: (ITS#7715) SIGBUS when mdb is configured with writemap
by hyc＠symas.com 04 Oct '13

04 Oct '13

Željko Nejašmić wrote: > On Fri, Oct 4, 2013 at 10:26 AM, <hyc(a)symas.com> wrote: >> >> Thanks. So far the output shows nothing out of the ordinary. Can you also >> provide the output for: >> ls -l /opt/openldap/var/openldap-data > > root@spr101:~ $ ls -ahl /opt/openldap/var/openldap-data > total 100K > drwxr-xr-x. 2 openldap openldap 4.0K Oct 4 07:54 . > drwxr-xr-x. 4 openldap openldap 4.0K May 28 08:49 .. > -rw-------. 1 openldap openldap 88K Oct 4 07:44 data.mdb > -rw-------. 1 openldap openldap 8.0K Oct 4 07:54 lock.mdb Something is flaky with your installation. As I wrote in my previous email, LMDB always sets the file size to match your mapsize, to avoid this SIGBUS issue. Your mapsize is 38654705664 (36GB) but the data.mdb filesize is only 88K, it should be 36GB. You must have some other process running that is truncating the file; there is nothing in OpenLDAP's code that would cause this behavior. > root@spr101:~ $ ps -efa | grep slapd > openldap 44012 1 0 07:54 ? 00:00:01 > /opt/openldap/libexec/slapd -h ldap:/// ldapi:/// -F > /opt/openldap/etc/openldap/slapd.d -g openldap -u openldap > > root@spr101:~ $ mount | grep sdb1 > /dev/sdb1 on /opt type ext4 (rw,noatime,nobarrier,data=ordered) > Do note that we did load testing on ext2 and ext4, with both being > tweaked and not, and as far as I recall, every combination of > parameters causes the problem; I will try to reproduce it on > ext{2,3,4} with default mounting parameters. Almost forgot, we did > tests using tmpfs also, with the same outcome. The whole /opt was used > as a mount point for the tmpfs. > > root@spr101:~ $ fdisk -lu /dev/sdb1 > Disk /dev/sdb1: 400.0 GB, 400027127808 bytes > 224 heads, 56 sectors/track, 62284 cylinders, total 781302984 sectors > Units = sectors of 1 * 512 = 512 bytes > Sector size (logical/physical): 512 bytes / 512 bytes > I/O size (minimum/optimal): 512 bytes / 512 bytes > Disk identifier: 0x00000000 > > gparted doesn't seem to be complaining about SSD block alignment: > root@spr101:~ $ parted /dev/sdb1 align-check opt 1 > > Another thing I'll be trying is making a partition with different > alignment parameters to see if any change occurs; tried using a > different scheme, but that hasn't produced a a change in behavior. > >> >> Note from the mmap(2) manpage: >> >> Use of a mapped region can result in these signals: >> >> SIGSEGV >> Attempted write into a region mapped as read-only. >> >> SIGBUS Attempted access to a portion of the buffer that does not >> correspond to the file (for example, beyond >> the end of the file, including the case where another process >> has truncated the file). >> >> LMDB always sets the size of the file when using the WRITEMAP option, so the >> SIGBUS condition should never happen. The pointers in your gdb output are all >> within the bounds of the map and mapsize you configured. >> >> What filesystem type is the data directory stored on? This is looking more >> like a kernel or filesystem bug than anything else. >> >> -- >> -- Howard Chu >> CTO, Symas Corp. http://www.symas.com >> Director, Highland Sun http://highlandsun.com/hyc/ >> Chief Architect, OpenLDAP http://www.openldap.org/project/ > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7715) SIGBUS when mdb is configured with writemap
by nejasmicz＠gmail.com 04 Oct '13

04 Oct '13

On Fri, Oct 4, 2013 at 10:26 AM, <hyc(a)symas.com> wrote: > > Thanks. So far the output shows nothing out of the ordinary. Can you also > provide the output for: > ls -l /opt/openldap/var/openldap-data root@spr101:~ $ ls -ahl /opt/openldap/var/openldap-data total 100K drwxr-xr-x. 2 openldap openldap 4.0K Oct 4 07:54 . drwxr-xr-x. 4 openldap openldap 4.0K May 28 08:49 .. -rw-------. 1 openldap openldap 88K Oct 4 07:44 data.mdb -rw-------. 1 openldap openldap 8.0K Oct 4 07:54 lock.mdb root@spr101:~ $ ps -efa | grep slapd openldap 44012 1 0 07:54 ? 00:00:01 /opt/openldap/libexec/slapd -h ldap:/// ldapi:/// -F /opt/openldap/etc/openldap/slapd.d -g openldap -u openldap root@spr101:~ $ mount | grep sdb1 /dev/sdb1 on /opt type ext4 (rw,noatime,nobarrier,data=ordered) Do note that we did load testing on ext2 and ext4, with both being tweaked and not, and as far as I recall, every combination of parameters causes the problem; I will try to reproduce it on ext{2,3,4} with default mounting parameters. Almost forgot, we did tests using tmpfs also, with the same outcome. The whole /opt was used as a mount point for the tmpfs. root@spr101:~ $ fdisk -lu /dev/sdb1 Disk /dev/sdb1: 400.0 GB, 400027127808 bytes 224 heads, 56 sectors/track, 62284 cylinders, total 781302984 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 gparted doesn't seem to be complaining about SSD block alignment: root@spr101:~ $ parted /dev/sdb1 align-check opt 1 Another thing I'll be trying is making a partition with different alignment parameters to see if any change occurs; tried using a different scheme, but that hasn't produced a a change in behavior. > > Note from the mmap(2) manpage: > > Use of a mapped region can result in these signals: > > SIGSEGV > Attempted write into a region mapped as read-only. > > SIGBUS Attempted access to a portion of the buffer that does not > correspond to the file (for example, beyond > the end of the file, including the case where another process > has truncated the file). > > LMDB always sets the size of the file when using the WRITEMAP option, so the > SIGBUS condition should never happen. The pointers in your gdb output are all > within the bounds of the map and mapsize you configured. > > What filesystem type is the data directory stored on? This is looking more > like a kernel or filesystem bug than anything else. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7715) SIGBUS when mdb is configured with writemap
by hyc＠symas.com 04 Oct '13

04 Oct '13

Thanks. So far the output shows nothing out of the ordinary. Can you also provide the output for: ls -l /opt/openldap/var/openldap-data Note from the mmap(2) manpage: Use of a mapped region can result in these signals: SIGSEGV Attempted write into a region mapped as read-only. SIGBUS Attempted access to a portion of the buffer that does not correspond to the file (for example, beyond the end of the file, including the case where another process has truncated the file). LMDB always sets the size of the file when using the WRITEMAP option, so the SIGBUS condition should never happen. The pointers in your gdb output are all within the bounds of the map and mapsize you configured. What filesystem type is the data directory stored on? This is looking more like a kernel or filesystem bug than anything else. Željko Nejašmić wrote: > On Thu, Oct 3, 2013 at 2:57 PM, <hyc(a)symas.com> wrote: >> >> Željko Nejašmić wrote: >>> On Wed, Oct 2, 2013 at 6:43 PM, <hyc(a)symas.com <mailto:hyc@symas.com>> wrote: >>> I ran the whole test suite for mdb, and as far as I can see, every test >>> returned OK. >>> >>> Found a new way to reproduce the SIGBUS using ldapadd on Ubuntu 12.04 firing >>> to openldap on RH 6.3: >>> ldapadd -h 172.17.101.150 -p 389 -D "cn=admin,dc=test" -w test -f test.ldif -- >>> ldif file is the same as the previous ldclt command. Doubt it matters, but the >>> ldif file is 1M adds. >> >> Can you provide your slapd config and this LDIF? > > > LDIF file is just 1M adds of the following format: > dn: uid=XXXXXXX,ds=USERS,o=STANDARD,dc=spr > objectClass: sprUser > > I've uploaded the config here: http://hastebin.com/nicecafaci.ldif > >>> On the RH box: >>> - compiled openldap with -g -O0 and previously used flags >>> gdb `find /root/openldap/ -type d -printf '-d %p '` --args >>> /opt/openldap/libexec/slapd -h "ldap:/// ldapi:///" -F >>> /opt/openldap/etc/openldap/slapd.d -g openldap -u openldap -d 0 >>> >>> gdb output: >>> bt -- http://hastebin.com/hefikekaxi.sh >>> bt 10 full -- http://hastebin.com/vudocosuka.sh >> >> Can you also provide the gdb output for >> print *env > > (gdb) print *env > $1 = {me_fd = 15, me_lfd = 14, me_mfd = 15, me_flags = 940113920, > me_psize = 4096, > me_maxreaders = 126, me_numreaders = 2, me_numdbs = 13, me_maxdbs = > 130, me_pid = 43974, > me_path = 0x8c19a0 "/opt/openldap/var/openldap-data", me_map = > 0x7ff6f721f000 "", > me_txns = 0x7ffff7ffb000, me_metas = {0x7ff6f721f010, > 0x7ff6f7220010}, me_txn = 0x7ff6dc10afd0, > me_mapsize = 38654705664, me_size = 0, me_maxpg = 9437184, me_dbxs = 0x8eed10, > me_dbflags = 0x8ceb40, me_txkey = 1, me_pgstate = {mf_pghead = > 0x7ff6dc109d58, mf_pglast = 44}, > me_dpages = 0x0, me_free_pgs = 0xaf75e8, me_dirty_list = > 0x7ffff721f010, me_maxfree_1pg = 509, > me_nodemax = 2040} > >> -- Howard Chu >> CTO, Symas Corp. http://www.symas.com >> Director, Highland Sun http://highlandsun.com/hyc/ >> Chief Architect, OpenLDAP http://www.openldap.org/project/ > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7715) SIGBUS when mdb is configured with writemap
by nejasmicz＠gmail.com 03 Oct '13

03 Oct '13

On Thu, Oct 3, 2013 at 2:57 PM, <hyc(a)symas.com> wrote: > > Željko Nejašmić wrote: > > On Wed, Oct 2, 2013 at 6:43 PM, <hyc(a)symas.com <mailto:hyc@symas.com>> wrote: > > I ran the whole test suite for mdb, and as far as I can see, every test > > returned OK. > > > > Found a new way to reproduce the SIGBUS using ldapadd on Ubuntu 12.04 firing > > to openldap on RH 6.3: > > ldapadd -h 172.17.101.150 -p 389 -D "cn=admin,dc=test" -w test -f test.ldif -- > > ldif file is the same as the previous ldclt command. Doubt it matters, but the > > ldif file is 1M adds. > > Can you provide your slapd config and this LDIF? LDIF file is just 1M adds of the following format: dn: uid=XXXXXXX,ds=USERS,o=STANDARD,dc=spr objectClass: sprUser I've uploaded the config here: http://hastebin.com/nicecafaci.ldif > > On the RH box: > > - compiled openldap with -g -O0 and previously used flags > > gdb `find /root/openldap/ -type d -printf '-d %p '` --args > > /opt/openldap/libexec/slapd -h "ldap:/// ldapi:///" -F > > /opt/openldap/etc/openldap/slapd.d -g openldap -u openldap -d 0 > > > > gdb output: > > bt -- http://hastebin.com/hefikekaxi.sh > > bt 10 full -- http://hastebin.com/vudocosuka.sh > > Can you also provide the gdb output for > print *env (gdb) print *env $1 = {me_fd = 15, me_lfd = 14, me_mfd = 15, me_flags = 940113920, me_psize = 4096, me_maxreaders = 126, me_numreaders = 2, me_numdbs = 13, me_maxdbs = 130, me_pid = 43974, me_path = 0x8c19a0 "/opt/openldap/var/openldap-data", me_map = 0x7ff6f721f000 "", me_txns = 0x7ffff7ffb000, me_metas = {0x7ff6f721f010, 0x7ff6f7220010}, me_txn = 0x7ff6dc10afd0, me_mapsize = 38654705664, me_size = 0, me_maxpg = 9437184, me_dbxs = 0x8eed10, me_dbflags = 0x8ceb40, me_txkey = 1, me_pgstate = {mf_pghead = 0x7ff6dc109d58, mf_pglast = 44}, me_dpages = 0x0, me_free_pgs = 0xaf75e8, me_dirty_list = 0x7ffff721f010, me_maxfree_1pg = 509, me_nodemax = 2040} > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#7717) Sudden memory increase leading to Master LDAP crash
by marcon.bruno＠free.fr 03 Oct '13

03 Oct '13

Full_Name: Marcon Bruno Version: 2.4.36 OS: Ubuntu URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (185.23.92.11) Hi. We have deployed OpenLDAP on our system, using the following architecture : One primary master One failover master (mirror mode) 4 replicas It works very well until the master LDAP receive two many writes /seconds. With 40 writes/seconds, the master LDAP work well about several minutes, then the memory process (slapd) started to increase very quickly, about 20Mo/s ! When all memory and swap is consumed, the OS kill Slapd. Notes : - Even in disabling all overlays except syncprov, the problem persists - If disabling only syncprov, there is no more problem - If all slaves and mirror master are stopped, there is no more problem This problem is very annoying. We are looking to a Microsoft solution, even if we have already deployed our solution but obviously, we would like to keep an OpenLDAP solution. The problem is very easy to replay : one master and one slave are sufficient. Then run 9 windows with a script adding an entry and modifying an entry.

1 0

Re: (ITS#7715) SIGBUS when mdb is configured with writemap
by hyc＠symas.com 03 Oct '13

03 Oct '13

Željko Nejašmić wrote: > On Wed, Oct 2, 2013 at 6:43 PM, <hyc(a)symas.com <mailto:hyc@symas.com>> wrote: > Without any other clues, this feels like ASLR is messing with us but that's > just a wild guess. I can no longer reproduce the SIGBUS in slapd regardless of > compile options, while ldclt itself keeps dying. If you can find some more > reliable way to reproduce the issue that would help. Perhaps using the client > in test060. > > > I ran the whole test suite for mdb, and as far as I can see, every test > returned OK. > > Found a new way to reproduce the SIGBUS using ldapadd on Ubuntu 12.04 firing > to openldap on RH 6.3: > ldapadd -h 172.17.101.150 -p 389 -D "cn=admin,dc=test" -w test -f test.ldif -- > ldif file is the same as the previous ldclt command. Doubt it matters, but the > ldif file is 1M adds. Can you provide your slapd config and this LDIF? > > On the RH box: > - compiled openldap with -g -O0 and previously used flags > gdb `find /root/openldap/ -type d -printf '-d %p '` --args > /opt/openldap/libexec/slapd -h "ldap:/// ldapi:///" -F > /opt/openldap/etc/openldap/slapd.d -g openldap -u openldap -d 0 > > gdb output: > bt -- http://hastebin.com/hefikekaxi.sh > bt 10 full -- http://hastebin.com/vudocosuka.sh Can you also provide the gdb output for print *env > > I am begging to doubt that the problem could be on my end as the bt seems to > point to schema problems (although, I haven't analyzed it in great detail yet). > > > Zeljko -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7715) SIGBUS when mdb is configured with writemap
by nejasmicz＠gmail.com 03 Oct '13

03 Oct '13

--001a11c253c0f33b8604e7d59850 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Wed, Oct 2, 2013 at 6:43 PM, <hyc(a)symas.com> wrote: > hyc(a)symas.com wrote: > > =C5=BDeljko Neja=C5=A1mi=C4=87 wrote: > >> Here you go http://hastebin.com/fukecejuje.tex > > > > Interestingly enough, I got the same result as you on an initial > compile/run > > of slapd. Unfortunately, with optimization, the backtrace wasn't all th= at > > useful. Recompiling back-mdb with just -g, no optimization, gets a > different > > result though - slapd is fine, and ldclt dies with a heap corruption or > > double-free. > > And now I cannot reproduce the original SIGBUS at all. But still getting > various crashes in ldclt. > > ldclt -h localhost -p 9011 -D cn=3Dmanager,dc=3Dexample,dc=3Dcom -w secre= t -b > ou=3Dpeople,dc=3Dexample,dc=3Dcom -e > object=3Dxx.txt,rdn=3D'uid:[A=3DINCRNNOLOOP(200000;999999;6)]' -e > add,commoncounter > -I 68 > ldclt version 4.23 > ldclt[30207]: Starting at Wed Oct 2 09:39:10 2013 > > *** glibc detected *** ldclt: double free or corruption (fasttop): > 0x00007fc2180021e0 *** > =3D=3D=3D=3D=3D=3D=3D Backtrace: =3D=3D=3D=3D=3D=3D=3D=3D=3D > /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7fc223ff9b96] > /usr/lib/x86_64-linux-gnu/libnspr4.so(+0x142d1)[0x7fc224f0b2d1] > /usr/lib/x86_64-linux-gnu/libnspr4.so(+0x1ae74)[0x7fc224f11e74] > /usr/lib/x86_64-linux-gnu/libnspr4.so(PR_Malloc+0x49)[0x7fc224f0d0b9] > /usr/lib/x86_64-linux-gnu/libnspr4.so(+0x129f4)[0x7fc224f099f4] > /usr/lib/x86_64-linux-gnu/libnspr4.so(+0x12068)[0x7fc224f09068] > /usr/lib/x86_64-linux-gnu/libnspr4.so(PR_vsmprintf+0x38)[0x7fc224f09b28] > /usr/lib/x86_64-linux-gnu/libnspr4.so(PR_smprintf+0x8c)[0x7fc224f09bec] > ldclt(+0x663a)[0x7fc22556663a] > ldclt(+0x74a4)[0x7fc2255674a4] > ldclt(+0x9d48)[0x7fc225569d48] > ldclt(threadMain+0x329)[0x7fc2255735d9] > /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fc224341e9a] > /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fc22406ecbd] > > Without any other clues, this feels like ASLR is messing with us but that= 's > just a wild guess. I can no longer reproduce the SIGBUS in slapd > regardless of > compile options, while ldclt itself keeps dying. If you can find some mor= e > reliable way to reproduce the issue that would help. Perhaps using the > client > in test060. I ran the whole test suite for mdb, and as far as I can see, every test returned OK. Found a new way to reproduce the SIGBUS using ldapadd on Ubuntu 12.04 firing to openldap on RH 6.3: ldapadd -h 172.17.101.150 -p 389 -D "cn=3Dadmin,dc=3Dtest" -w test -f test.= ldif -- ldif file is the same as the previous ldclt command. Doubt it matters, but the ldif file is 1M adds. On the RH box: - compiled openldap with -g -O0 and previously used flags gdb `find /root/openldap/ -type d -printf '-d %p '` --args /opt/openldap/libexec/slapd -h "ldap:/// ldapi:///" -F /opt/openldap/etc/openldap/slapd.d -g openldap -u openldap -d 0 gdb output: bt -- http://hastebin.com/hefikekaxi.sh bt 10 full -- http://hastebin.com/vudocosuka.sh I am begging to doubt that the problem could be on my end as the bt seems to point to schema problems (although, I haven't analyzed it in great detail yet). Zeljko --001a11c253c0f33b8604e7d59850 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><br><div class=3D"gmail_extra"><br><br><div class=3D"gmail= _quote">On Wed, Oct 2, 2013 at 6:43 PM, <span dir=3D"ltr"><<a href=3D"m= ailto:hyc@symas.com" target=3D"_blank">hyc(a)symas.com</a>></span> wrote:<= br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;bord= er-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:soli= d;padding-left:1ex"> <div><a href=3D"mailto:hyc@symas.com" target=3D"_blank">hyc(a)symas.com</a> w= rote:<br> > =C5=BDeljko Neja=C5=A1mi=C4=87 wrote:<br> >> Here you go <a href=3D"http://hastebin.com/fukecejuje.tex" target= =3D"_blank">http://hastebin.com/fukecejuje.tex</a><br> ><br> > Interestingly enough, I got the same result as you on an initial compi= le/run<br> > of slapd. Unfortunately, with optimization, the backtrace wasn't a= ll that<br> > useful. Recompiling back-mdb with just -g, no optimization, gets a dif= ferent<br> > result though - slapd is fine, and ldclt dies with a heap corruption o= r<br> > double-free.<br> <br> </div>And now I cannot reproduce the original SIGBUS at all. But still gett= ing<br> various crashes in ldclt.<br> <div><br> ldclt -h localhost -p 9011 -D cn=3Dmanager,dc=3Dexample,dc=3Dcom -w secret = -b<br> ou=3Dpeople,dc=3Dexample,dc=3Dcom -e<br> object=3Dxx.txt,rdn=3D'uid:[A=3DINCRNNOLOOP(200000;999999;6)]' -e a= dd,commoncounter<br> -I 68<br> ldclt version 4.23<br> </div>ldclt[30207]: Starting at Wed Oct =C2=A02 09:39:10 2013<br> <div><br> *** glibc detected *** ldclt: double free or corruption (fasttop):<br> </div>0x00007fc2180021e0 ***<br> =3D=3D=3D=3D=3D=3D=3D Backtrace: =3D=3D=3D=3D=3D=3D=3D=3D=3D<br> /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7fc223ff9b96]<br> /usr/lib/x86_64-linux-gnu/libnspr4.so(+0x142d1)[0x7fc224f0b2d1]<br> /usr/lib/x86_64-linux-gnu/libnspr4.so(+0x1ae74)[0x7fc224f11e74]<br> /usr/lib/x86_64-linux-gnu/libnspr4.so(PR_Malloc+0x49)[0x7fc224f0d0b9]<br> /usr/lib/x86_64-linux-gnu/libnspr4.so(+0x129f4)[0x7fc224f099f4]<br> /usr/lib/x86_64-linux-gnu/libnspr4.so(+0x12068)[0x7fc224f09068]<br> /usr/lib/x86_64-linux-gnu/libnspr4.so(PR_vsmprintf+0x38)[0x7fc224f09b28]<br= > /usr/lib/x86_64-linux-gnu/libnspr4.so(PR_smprintf+0x8c)[0x7fc224f09bec]<br> ldclt(+0x663a)[0x7fc22556663a]<br> ldclt(+0x74a4)[0x7fc2255674a4]<br> ldclt(+0x9d48)[0x7fc225569d48]<br> ldclt(threadMain+0x329)[0x7fc2255735d9]<br> /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fc224341e9a]<br> /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fc22406ecbd]<br> <br> Without any other clues, this feels like ASLR is messing with us but that&#= 39;s<br> just a wild guess. I can no longer reproduce the SIGBUS in slapd regardless= of<br> compile options, while ldclt itself keeps dying. If you can find some more<= br> reliable way to reproduce the issue that would help. Perhaps using the clie= nt<br> in test060.</blockquote><div><br></div><div>I ran the whole test suite for = mdb, and as far as I can see, every test returned OK.</div><div><br></div><= div>Found a new way to reproduce the SIGBUS using ldapadd on Ubuntu 12.04 f= iring to openldap on RH 6.3:<br> </div><div>ldapadd -h 172.17.101.150 -p 389 -D "cn=3Dadmin,dc=3Dtest&q= uot; -w test -f test.ldif -- ldif file is the same as the previous ldclt co= mmand. Doubt it matters, but the ldif file is 1M adds.</div></div><br></div= > <div class=3D"gmail_extra">On the RH box:</div><div class=3D"gmail_extra">-= compiled openldap with -g -O0 and previously used flags</div><div class=3D= "gmail_extra"> gdb `find /root/openldap/ -type d -printf '-d %p '` --args /opt/ope= nldap/libexec/slapd -h "ldap:/// ldapi:///" -F /opt/openldap/etc/= openldap/slapd.d -g openldap -u openldap -d 0<br></div><div class=3D"gmail_= extra"> <br></div><div class=3D"gmail_extra">gdb output:</div><div class=3D"gmail_e= xtra">bt --=C2=A0<a href=3D"http://hastebin.com/hefikekaxi.sh" target=3D"_b= lank">http://hastebin.com/hefikekaxi.sh</a></div><div class=3D"gmail_extra"= >bt 10 full --=C2=A0<a href=3D"http://hastebin.com/vudocosuka.sh" target=3D= "_blank">http://hastebin.com/vudocosuka.sh</a></div> <div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">I am beggin= g to doubt that the problem could be on my end as the bt seems to point to = schema problems (although, I haven't analyzed it in great detail yet).<= /div> <div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra"><br></div><= div class=3D"gmail_extra">Zeljko</div> </div> --001a11c253c0f33b8604e7d59850--

1 0

Re: (ITS#7715) SIGBUS when mdb is configured with writemap
by hyc＠symas.com 02 Oct '13

02 Oct '13

hyc(a)symas.com wrote: > Željko Nejašmić wrote: >> Here you go http://hastebin.com/fukecejuje.tex > > Interestingly enough, I got the same result as you on an initial compile/run > of slapd. Unfortunately, with optimization, the backtrace wasn't all that > useful. Recompiling back-mdb with just -g, no optimization, gets a different > result though - slapd is fine, and ldclt dies with a heap corruption or > double-free. And now I cannot reproduce the original SIGBUS at all. But still getting various crashes in ldclt. ldclt -h localhost -p 9011 -D cn=manager,dc=example,dc=com -w secret -b ou=people,dc=example,dc=com -e object=xx.txt,rdn='uid:[A=INCRNNOLOOP(200000;999999;6)]' -e add,commoncounter -I 68 ldclt version 4.23 ldclt[30207]: Starting at Wed Oct 2 09:39:10 2013 *** glibc detected *** ldclt: double free or corruption (fasttop): 0x00007fc2180021e0 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7fc223ff9b96] /usr/lib/x86_64-linux-gnu/libnspr4.so(+0x142d1)[0x7fc224f0b2d1] /usr/lib/x86_64-linux-gnu/libnspr4.so(+0x1ae74)[0x7fc224f11e74] /usr/lib/x86_64-linux-gnu/libnspr4.so(PR_Malloc+0x49)[0x7fc224f0d0b9] /usr/lib/x86_64-linux-gnu/libnspr4.so(+0x129f4)[0x7fc224f099f4] /usr/lib/x86_64-linux-gnu/libnspr4.so(+0x12068)[0x7fc224f09068] /usr/lib/x86_64-linux-gnu/libnspr4.so(PR_vsmprintf+0x38)[0x7fc224f09b28] /usr/lib/x86_64-linux-gnu/libnspr4.so(PR_smprintf+0x8c)[0x7fc224f09bec] ldclt(+0x663a)[0x7fc22556663a] ldclt(+0x74a4)[0x7fc2255674a4] ldclt(+0x9d48)[0x7fc225569d48] ldclt(threadMain+0x329)[0x7fc2255735d9] /lib/x86_64-linux-gnu/libpthread.so.0(+0x7e9a)[0x7fc224341e9a] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fc22406ecbd] Without any other clues, this feels like ASLR is messing with us but that's just a wild guess. I can no longer reproduce the SIGBUS in slapd regardless of compile options, while ldclt itself keeps dying. If you can find some more reliable way to reproduce the issue that would help. Perhaps using the client in test060. > All subsequent runs give the same result. Still looking into this; getting > away from the Ubuntu bundled libldap will probably help. >> >> >> Zeljko >> >> >> On Wed, Oct 2, 2013 at 11:34 AM, <hyc(a)symas.com <mailto:hyc@symas.com>> wrote: >> >> nejasmicz(a)gmail.com <mailto:nejasmicz@gmail.com> wrote: >> > Full_Name: Željko Nejašmić >> > Version: latest git pull of OPENLDAP_REL_ENG_2_4 >> > OS: RedHat 6.3 >> > URL: >> > Submission from: (NULL) (213.147.123.33) >> > >> >> > Hardware underneath all of that is: >> > 1) HP ProLiant BL460c Gen8, dual Xeon E5-2658 with attached >> storage blade >> > D2200sb with SSD raid, 256GB RAM -- RedHat 6.3 tests >> > 2) Intel server blade S2400BB, dual Xeon E5-2403, 48GB RAM -- >> Ubuntu 12.04 >> > tests >> > >> > If anything more is required to assist you in troubleshooting, please >> let me >> > know. >> >> Can you also post the gdb output for "bt 5 full" ? >> > >> > >> > Zeljko >> > >> > >> -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7716) slapd crashes after search immediately followed by (abandon+) unbind
by michael＠stroeder.com 02 Oct '13

02 Oct '13

michael.vishchers(a)7p-group.com wrote: > Full_Name: Michael Vishchers > Version: 2.4.23 > OS: Red Hat Enterprise Linux Server release 6.2 Did you check whether this problem still remains with the recent release 2.4.36? (I vaguely remember issues like this being fixed after 2.4.23 but I'm not sure.) Ciao, Michael.

1 0

(ITS#7716) slapd crashes after search immediately followed by (abandon+) unbind
by michael.vishchers＠7p-group.com 02 Oct '13

02 Oct '13

Full_Name: Michael Vishchers Version: 2.4.23 OS: Red Hat Enterprise Linux Server release 6.2 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (178.15.66.50) slapd, running as a proxy to rewrite incoming connections based on user dn for later routing to different back ends, dies sporadically after receiving a (network delayed) search request that is "immediately" followed by an (optional) abandon request and an unbind request. We suspect that the abandon or unbind code tries to clean up data structures that belong to a not yet completely initialized search operation. The problem can unfortunately not easily be reproduced. Last time we had to wait at least two weeks before it appeared. It may be a timing problem between two or more threads. This is the stacktrace, core and other files could be provided if necessary. Program terminated with signal 11, Segmentation fault. #0 0x00007f299c3e71bc in ?? () #0 0x00007f299c3e71bc in ?? () No symbol table info available. #1 0x00007f2999bbd983 in rwm_op_rollback (op=0x7f2984002190, rs=<value optimized out>, ros=0x7f298c003570) at ../../../../servers/slapd/overlays/rwm.c:107 __PRETTY_FUNCTION__ = "rwm_op_rollback" #2 0x00007f2999bbe988 in rwm_op_search (op=0x7f2984002190, rs=0x7f2995bafaa0) at ../../../../servers/slapd/overlays/rwm.c:984 on = 0x7f299fb95210 rwmap = 0x7f299fb94f70 rc = <value optimized out> dc = {rwmap = 0x7f2984002190, conn = 0x7f298c003468, ctx = 0x12 <Address 0x12 out of bounds>, rs = 0x7f299e7963b0} fstr = {bv_len = 0, bv_val = 0x0} f = 0x0 an = 0x0 text = <value optimized out> roc = 0x7f298c003550 #3 0x00007f299e7fe02a in overlay_op_walk (op=0x7f2984002190, rs=0x7f2995bafaa0, which=op_search, oi=0x7f299fb95030, on=0x7f299fb95210) at ../../../servers/slapd/backover.c:659 func = 0x7f299fb95268 rc = 32768 #4 0x00007f299e8d29a1 in slapi_op_func (op=0x7f2984002190, rs=0x7f2995bafaa0) at ../../../../servers/slapd/slapi/slapi_overlay.c:647 pb = 0x7f298c1051b0 which = op_search opinfo = <value optimized out> rc = <value optimized out> oi = <value optimized out> on = <value optimized out> cb = {sc_next = 0x7f2995bae7e0, sc_response = 0x7f299e8d1fc0 <slapi_over_response>, sc_cleanup = 0x7f299e8d1ed0 <slapi_over_cleanup>, sc_private = 0x7f298c1051b0} internal_op = 0 preop_type = <value optimized out> postop_type = 503 be = 0x7f2995bae800 #5 0x00007f299e7fe02a in overlay_op_walk (op=0x7f2984002190, rs=0x7f2995bafaa0, which=op_search, oi=0x7f299fb95030, on=0x7f299fb9e8c0) at ../../../servers/slapd/backover.c:659 func = 0x7f299fb9e918 rc = 32768 #6 0x00007f299e7feb6b in over_op_func (op=0x7f2984002190, rs=<value optimized out>, which=<value optimized out>) at ../../../servers/slapd/backover.c:721 oi = <value optimized out> on = <value optimized out> be = 0x7f299fb940b0 db = {bd_info = 0x7f299fb95210, bd_self = 0x7f299fb940b0, be_ctrls = "\000", '\001' <repeats 17 times>, '\000' <repeats 14 times>, "\001", be_flags = 257, be_restrictops = 0, be_requires = 5, be_ssf_set = {sss_ssf = 0, sss_transport = 0, sss_tls = 0, sss_sasl = 0, sss_update_ssf = 0, sss_update_transport = 0, sss_update_tls = 0, sss_update_sasl = 0, sss_simple_bind = 0}, be_suffix = 0x7f299fb94ed0, be_nsuffix = 0x7f299fb94f00, be_schemadn = {bv_len = 0, bv_val = 0x0}, be_schemandn = {bv_len = 0, bv_val = 0x0}, be_rootdn = {bv_len = 0, bv_val = 0x0}, be_rootndn = {bv_len = 0, bv_val = 0x0}, be_rootpw = {bv_len = 0, bv_val = 0x0}, be_max_deref_depth = 15, be_def_limit = {lms_t_soft = 3600, lms_t_hard = 0, lms_s_soft = 500, lms_s_hard = 0, lms_s_unchecked = -1, lms_s_pr = 0, lms_s_pr_hide = 0, lms_s_pr_total = 0}, be_limits = 0x0, be_acl = 0x0, be_dfltaccess = ACL_READ, be_update_ndn = {bv_len = 0, bv_val = 0x0}, be_update_refs = 0x0, be_pending_csn_list = 0x7f299fc738f0, be_pcl_mutex = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}, be_syncinfo = 0x0, be_pb = 0x7f299fb9eaa0, be_cf_ocs = 0x7f299eb6da00, be_private = 0x7f299fb94240, be_next = {stqe_next = 0x0}} cb = {sc_next = 0x0, sc_response = 0x7f299e7fdd40 <over_back_response>, sc_cleanup = 0, sc_private = 0x7f299fb95030} sc = <value optimized out> rc = 32768 __PRETTY_FUNCTION__ = "over_op_func" #7 0x00007f299e794999 in fe_op_search (op=0x7f2984002190, rs=0x7f2995bafaa0) at ../../../servers/slapd/search.c:366 bd = 0x7f299eb72760 #8 0x00007f299e795177 in do_search (op=0x7f2984002190, rs=<value optimized out>) at ../../../servers/slapd/search.c:217 base = {bv_len = 55, bv_val = 0x7f298c11fef9 "vfsid=491722472236,ou=subscriber,ou=mmo,c=de,o=vodafone"} siz = 0 off = 0 i = <value optimized out> #9 0x00007f299e7920f9 in connection_operation (ctx=0x7f2995bafb70, arg_v=0x7f2984002190) at ../../../servers/slapd/connection.c:1109 rc = 80 cancel = <value optimized out> op = 0x7f2984002190 rs = {sr_type = REP_RESULT, sr_tag = 0, sr_msgid = 0, sr_err = 80, sr_matched = 0x0, sr_text = 0x7f2999bc4122 "Rewrite error", sr_ref = 0x0, sr_ctrls = 0x0, sr_un = {sru_search = {r_entry = 0x0, r_attr_flags = 0, r_operational_attrs = 0x0, r_attrs = 0x0, r_nentries = 0, r_v2ref = 0x0}, sru_sasl = {r_sasldata = 0x0}, sru_extended = {r_rspoid = 0x0, r_rspdata = 0x0}}, sr_flags = 0} tag = 99 opidx = SLAP_OP_SEARCH conn = 0x7f2996db74d0 memctx = 0x7f298c002820 memctx_null = 0x0 memsiz = 1048576 __PRETTY_FUNCTION__ = "connection_operation" #10 0x00007f299e892678 in ldap_int_thread_pool_wrapper (xpool=0x7f299fae9ae0) at ../../../libraries/libldap_r/tpool.c:685 pool = 0x7f299fae9ae0 task = 0x7f2988000a20 work_list = <value optimized out> ctx = {ltu_id = 139816582448896, ltu_key = {{ltk_key = 0x7f299e790d50, ltk_data = 0x7f298c002d40, ltk_free = 0x7f299e790e30 <conn_counter_destroy>}, {ltk_key = 0x7f299e7eaf70, ltk_data = 0x7f298c002820, ltk_free = 0x7f299e7eae50 <slap_sl_mem_destroy>}, {ltk_key = 0x7f299e7a6b70, ltk_data = 0x0, ltk_free = 0x7f299e7a6940 <slap_op_q_destroy>}, {ltk_key = 0x0, ltk_data = 0x0, ltk_free = 0} <repeats 29 times>}} kctx = <value optimized out> keyslot = <value optimized out> hash = <value optimized out> __PRETTY_FUNCTION__ = "ldap_int_thread_pool_wrapper" #11 0x00007f299c91e7f1 in ?? () No symbol table info available. #12 0x00007f2995bb0700 in ?? () No symbol table info available. #13 0x0000000000000000 in ?? () No symbol table info available.

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs