openldap-bugs

openldap-bugs@openldap.org

1 participants
20964 discussions

(ITS#8916) MDB_VL32 in readonly causes MDB_CORRUPTED
by pierre.chambart＠ocamlpro.com 19 Sep '18

19 Sep '18

Full_Name: Pierre Chambart Version: OS: URL: Submission from: (NULL) (185.194.234.133) We have been trying LMDB compiled with the MDB_VL32 flag and encountered some MDB_CORRUPTED error that didn't occur without MDB_VL32. We could produce a minimal reproduction case: https://raw.githubusercontent.com/chambart/lmdb/vl32_readonly_fix/libraries… This is basically writing a key, reading it, writing something else then reading it again. After some investigation, this is how I understand what is happening: * The chunk being mapped for the first read contains 3 pages instead of the 16 pages that would be mapped without readonly. * The second write extend the size of the base to 5 pages * Finally the last read retrieve the mapped chunk from the first read, which has size 3, but assumes that it has size 5, hence reading some random value in memory past the end of the pages. It can be verified by asserting that id3.mcnt is larger than rem at the end of mdb_rpage_get, around this line: ok: p = (MDB_page *)((char *)id3.mptr + rem * env->me_psize); This does not seem to be the case. I made a dirty fix that seems to work. It's simply to remove those lines: /* don't map past last written page in read-only envs */ if ((env->me_flags & MDB_RDONLY) && pgno + MDB_RPAGE_CHUNK-1 > txn->mt_last_pgno) id3.mcnt = txn->mt_last_pgno + 1 - pgno; else This seems to be some optimization that avoids mapping the 16 pages of the chunk. But given that those pages are initialized, there doesn't seem to be much harm in mapping a few more pages in that relatively rare situation. The patch is provided in https://github.com/chambart/lmdb/commit/52e57a392e91935efe6d57337da09f23e40… If you want to reproduce the bug you can use the https://github.com/chambart/lmdb/tree/vl32_readonly_fix which also contains the patch to the makefile to build in 32bits with MDB_VL32. Thanks a lot, hoping that this can be of some help.

1 0

Re: (ITS#8650) EAGAIN from gnutls_handshake not respected
by ryan＠nardis.ca 18 Sep '18

18 Sep '18

Made some good progress on this one this evening. The original issue this ITS is about is that gnutls_handshake() can, in some versions of GnuTLS, return GNUTLS_E_AGAIN even when the socket is blocking. Specifically, this happens in the case I described with a large CA list sent by the server. For slapd, the patch I committed is unfortunately completely wrong. It has been using non-blocking sockets forever, EAGAIN is expected and handled robustly -- or it was, until I introduced the busy-loop. For clients I'm still working on figuring out the right path forward. There is some EAGAIN handling conditional on LDAP_USE_NON_BLOCKING_TLS which itself is behind LDAP_DEVEL. However this code is meant for non-blocking sockets, and in my case it ends up stuck in poll() waiting for a notification that never arrives. In 2.4, ret == 1 simply falls into the success case and proceeds to send data without completing the handshake first. It's possible that what I actually want here is a (ret > 0) case in ldap_int_tls_start for when LDAP_USE_NON_BLOCKING_TLS is absent and ldap_int_tls_connect returns 1. (I'd also need to adapt the non-blocking path to be able to handle a blocking socket as well.) But it's also possible that gnutls_handshake() returning GNUTLS_E_AGAIN with a blocking socket is simply a GnuTLS bug that was introduced at some point. I still need to determine exactly when and why its behaviour changed. (It is still happening with 3.5.19.) In any case, my patch has to be reverted, as its impact (making slapd busy-loop) is obviously worse than the status quo (misbehaving clients in a specific case). I have pushed that revert now and will continue digging as time permits.

1 0

(ITS#8915) OpenLDAP and OpenSSL v1.1.1
by neal.brown＠threesl.com 13 Sep '18

13 Sep '18

Full_Name: Neal Brown Version: 2.4.46 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (88.98.37.245) OpenLDAP will not configure for building against OpenSSL v1.1.1 due to SSLv3 being disabled.

1 0

(ITS#8914) OpenLDAP and OpenSSL v1.1.1
by nea.brown＠threesl.com 13 Sep '18

13 Sep '18

1 0

Re: (ITS#8898) mdb_get on corrupted LMDB file crash the process
by hyc＠symas.com 12 Sep '18

12 Sep '18

Irad K wrote: > Hi Howard and thanks for the info.=C2=A0 >=20 > I've read your insights about the corruption nature and I'd like to see= if I can write some sort of integrity check before I try to open the fil= e.=C2=A0 > Perhaps you could guide me where to look for the file format, where can= I find all the meta-pages and in which offset can I find the used pages = ? Read the source yourself if you want to know the underlying format. Other= wise, just use mdb_env_info(). =C2=A0 > Basically, I wish to check the latest meta-page and see if the number o= f pages correspond with the file actual size.=C2=A0 >=20 > Second, as I've stated, my target operating system is macOS.. are you f= amiliar of any known issues with file atomicity in this particular OS ?=C2= =A0 > Perhaps you could tell me how it's implemented under the hood to achiev= e data consistency if I'm using the database using MDB_NOMETASYNC, so I f= urther > investigate this issue and address it to Apple if necessary.=20 I've seen other reports of corruptions on MacOSX. Since they're in SQLite= I haven't investigated further. http://sqlite.1065341.n5.nabble.com/SQLITE-vs-OSX-mmap-inevitable-catalog= -corruption-td85620.html http://sqlite.1065341.n5.nabble.com/Re-Database-corruption-and-PRAGMA-ful= lfsync-on-macOS-td95366i20.html >=20 > Irad >=20 > On Mon, Sep 10, 2018 at 8:44 PM Howard Chu <hyc(a)symas.com <mailto:hyc@s= ymas.com>> wrote: >=20 > iradization(a)gmail.com <mailto:iradization@gmail.com> wrote: > > Full_Name: Irad.k > > Version: recent from GitHub. > > OS: macOS > > URL: ftp://ftp.openldap.org/incoming/ > > Submission from: (NULL) (82.81.84.130) > > > > > > Hi, > > > > I'm working with LMDB with the following flags :=C2=A0 MDB_NOMETA= SYNC | MDB_NOSUBDIR > > | MDB_NOTLS. > > > > Somehow, even though the DB should be ACI(without the D),=C2=A0 i= t got corrupted > > after recovering from kernel panic, and It crashes my process whe= n trying to > > access one of the records (see crash log below). > > > > Here's a link to the file : > > > > https://drive.google.com/file/d/12Q3KYYrapiJOgiaccnDL3tQACkqrM3C5= /view?usp=3Dsharing >=20 > This file is only 6 pages long, but the latest meta page claims tha= t it's using 11 pages. > Looking at the previous meta page, it claims that 9 pages were used= . >=20 > Clearly your OS failed to write the complete contents out before it= panicked. At the > very least the file should have been 9 pages long. Sounds like you = have an OS bug. >=20 > > According to the crash log from the process, It can clearly be se= en that the > > invalid address reside inside the mapped file region which is the= lmdb mapped > > file, but still I get KERN_MEMORY_ERROR on that address. > > > >>From what I know, an attempt to access address within the mapped = range can > > either retrieve the page contents directly from memory (if it's a= lready there), > > or trigger page fault trap that eventually lead to reading the mi= ssing data from > > disk and return it to process as well. > > > > One thing that raise some concerns is that the file size is only = 24k and the > > mapping spans over 256M. However, the file's meta data seems to b= e coherent to > > file contents. > > > > Any idea how did it happen, and what exactly in the file cause th= is corruption ? > > > > > > CRASH LOG : > > -----------------------------------------------------------------= ----- > > Exception Type:=C2=A0 =C2=A0 =C2=A0 =C2=A0 EXC_BAD_ACCESS (SIGBUS= ) > > Exception Codes:=C2=A0 =C2=A0 =C2=A0 =C2=A0KERN_MEMORY_ERROR at 0= x000000010648800a > > Exception Note:=C2=A0 =C2=A0 =C2=A0 =C2=A0 EXC_CORPSE_NOTIFY > > > > Termination Signal:=C2=A0 =C2=A0 Bus error: 10 > > Termination Reason:=C2=A0 =C2=A0 Namespace SIGNAL, Code 0xa > > Terminating Process:=C2=A0 =C2=A0exc handler [0] > > > > VM Regions Near 0x10648800a: > > > > __LINKEDIT=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00000000= 106464000-000000010647f000 [=C2=A0 108K] r--/rwx SM=3DCOW > >=C2=A0 ^Z^C [/usr/lib/dyld] > > --> mapped file=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 00000001= 0647f000-000000011647f000 [256.0M] r--/rwx > > SM=3DPRV=C2=A0 Object_id=3D9034edd9 > > STACK GUARD=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 000070000f2b= 3000-000070000f2b4000 [=C2=A0 =C2=A0 4K] ---/rwx SM=3DNUL > >=C2=A0 stack guard for thread 1 > > > > Thread 0 Crashed:: Dispatch queue: com.apple.main-thread > > 0=C2=A0 =C2=A0myprog=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00x0000000101666756 mdb_page_search_root + = 39 > > 1=C2=A0 =C2=A0myprog=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00x00000001016660f7 mdb_page_search + 182 > > 2=C2=A0 =C2=A0myprog=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00x00000001016614de mdb_cursor_set + 88 > > 3=C2=A0 =C2=A0myprog=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A00x0000000101661476 mdb_get + 134 > > > > > > >=20 >=20 > --=20 > =C2=A0 -- Howard Chu > =C2=A0 CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0htt= p://www.symas.com > =C2=A0 Director, Highland Sun=C2=A0 =C2=A0 =C2=A0http://highlandsun= .com/hyc/ > =C2=A0 Chief Architect, OpenLDAP=C2=A0 http://www.openldap.org/proj= ect/ >=20 --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8898) mdb_get on corrupted LMDB file crash the process
by iradization＠gmail.com 11 Sep '18

11 Sep '18

--000000000000d405310575a6de5e Content-Type: text/plain; charset="UTF-8" Hi Howard and thanks for the info. I've read your insights about the corruption nature and I'd like to see if I can write some sort of integrity check before I try to open the file. Perhaps you could guide me where to look for the file format, where can I find all the meta-pages and in which offset can I find the used pages ? Basically, I wish to check the latest meta-page and see if the number of pages correspond with the file actual size. Second, as I've stated, my target operating system is macOS.. are you familiar of any known issues with file atomicity in this particular OS ? Perhaps you could tell me how it's implemented under the hood to achieve data consistency if I'm using the database using MDB_NOMETASYNC, so I further investigate this issue and address it to Apple if necessary. Irad On Mon, Sep 10, 2018 at 8:44 PM Howard Chu <hyc(a)symas.com> wrote: > iradization(a)gmail.com wrote: > > Full_Name: Irad.k > > Version: recent from GitHub. > > OS: macOS > > URL: ftp://ftp.openldap.org/incoming/ > > Submission from: (NULL) (82.81.84.130) > > > > > > Hi, > > > > I'm working with LMDB with the following flags : MDB_NOMETASYNC | > MDB_NOSUBDIR > > | MDB_NOTLS. > > > > Somehow, even though the DB should be ACI(without the D), it got > corrupted > > after recovering from kernel panic, and It crashes my process when > trying to > > access one of the records (see crash log below). > > > > Here's a link to the file : > > > > > https://drive.google.com/file/d/12Q3KYYrapiJOgiaccnDL3tQACkqrM3C5/view?usp=… > > This file is only 6 pages long, but the latest meta page claims that it's > using 11 pages. > Looking at the previous meta page, it claims that 9 pages were used. > > Clearly your OS failed to write the complete contents out before it > panicked. At the > very least the file should have been 9 pages long. Sounds like you have an > OS bug. > > > According to the crash log from the process, It can clearly be seen that > the > > invalid address reside inside the mapped file region which is the lmdb > mapped > > file, but still I get KERN_MEMORY_ERROR on that address. > > > >>From what I know, an attempt to access address within the mapped range > can > > either retrieve the page contents directly from memory (if it's already > there), > > or trigger page fault trap that eventually lead to reading the missing > data from > > disk and return it to process as well. > > > > One thing that raise some concerns is that the file size is only 24k and > the > > mapping spans over 256M. However, the file's meta data seems to be > coherent to > > file contents. > > > > Any idea how did it happen, and what exactly in the file cause this > corruption ? > > > > > > CRASH LOG : > > ---------------------------------------------------------------------- > > Exception Type: EXC_BAD_ACCESS (SIGBUS) > > Exception Codes: KERN_MEMORY_ERROR at 0x000000010648800a > > Exception Note: EXC_CORPSE_NOTIFY > > > > Termination Signal: Bus error: 10 > > Termination Reason: Namespace SIGNAL, Code 0xa > > Terminating Process: exc handler [0] > > > > VM Regions Near 0x10648800a: > > > > __LINKEDIT 0000000106464000-000000010647f000 [ 108K] > r--/rwx SM=COW > > ^Z^C [/usr/lib/dyld] > > --> mapped file 000000010647f000-000000011647f000 [256.0M] > r--/rwx > > SM=PRV Object_id=9034edd9 > > STACK GUARD 000070000f2b3000-000070000f2b4000 [ 4K] > ---/rwx SM=NUL > > stack guard for thread 1 > > > > Thread 0 Crashed:: Dispatch queue: com.apple.main-thread > > 0 myprog 0x0000000101666756 mdb_page_search_root + > 39 > > 1 myprog 0x00000001016660f7 mdb_page_search + 182 > > 2 myprog 0x00000001016614de mdb_cursor_set + 88 > > 3 myprog 0x0000000101661476 mdb_get + 134 > > > > > > > > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ > --000000000000d405310575a6de5e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr">Hi Howard and thanks for= the info.=C2=A0<div><br><div>I've read your insights about the corrupt= ion nature and I'd like to see if I can write some sort of integrity ch= eck before I try to open the file.=C2=A0</div><div>Perhaps you could guide = me where to look for the file format, where can I find all the meta-pages a= nd in which offset can I find the used pages ?=C2=A0</div><div>Basically, I= wish to check the latest meta-page and see if the number of pages correspo= nd with the file actual size.=C2=A0<br></div><div><br></div><div>Second, as= I've stated, my target operating system is macOS.. are you familiar of= any known issues with file atomicity in this particular OS ?=C2=A0</div></= div><div><div>Perhaps you could tell me how it's implemented under the = hood to achieve data consistency if I'm using the database using MDB_NO= METASYNC, so I further investigate this issue and address it to Apple if ne= cessary.=C2=A0</div></div><div><br></div><div>Irad</div></div></div></div><= br><div class=3D"gmail_quote"><div dir=3D"ltr">On Mon, Sep 10, 2018 at 8:44= PM Howard Chu <<a href=3D"mailto:hyc@symas.com">hyc(a)symas.com</a>> w= rote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex= ;border-left:1px #ccc solid;padding-left:1ex"><a href=3D"mailto:iradization= @gmail.com" target=3D"_blank">iradization(a)gmail.com</a> wrote:<br> > Full_Name: Irad.k<br> > Version: recent from GitHub.<br> > OS: macOS<br> > URL: <a href=3D"ftp://ftp.openldap.org/incoming/" rel=3D"noreferrer" t= arget=3D"_blank">ftp://ftp.openldap.org/incoming/</a><br> > Submission from: (NULL) (82.81.84.130)<br> > <br> > <br> > Hi, <br> > <br> > I'm working with LMDB with the following flags :=C2=A0 MDB_NOMETAS= YNC | MDB_NOSUBDIR<br> > | MDB_NOTLS.<br> > <br> > Somehow, even though the DB should be ACI(without the D),=C2=A0 it got= corrupted<br> > after recovering from kernel panic, and It crashes my process when try= ing to<br> > access one of the records (see crash log below). <br> > <br> > Here's a link to the file : <br> > <br> > <a href=3D"https://drive.google.com/file/d/12Q3KYYrapiJOgiaccnDL3tQACk= qrM3C5/view?usp=3Dsharing" rel=3D"noreferrer" target=3D"_blank">https://dri= ve.google.com/file/d/12Q3KYYrapiJOgiaccnDL3tQACkqrM3C5/view?usp=3Dsharing</= a><br> <br> This file is only 6 pages long, but the latest meta page claims that it&#39= ;s using 11 pages.<br> Looking at the previous meta page, it claims that 9 pages were used.<br> <br> Clearly your OS failed to write the complete contents out before it panicke= d. At the<br> very least the file should have been 9 pages long. Sounds like you have an = OS bug.<br> <br> > According to the crash log from the process, It can clearly be seen th= at the<br> > invalid address reside inside the mapped file region which is the lmdb= mapped<br> > file, but still I get KERN_MEMORY_ERROR on that address.<br> > <br> >>From what I know, an attempt to access address within the mapped ra= nge can<br> > either retrieve the page contents directly from memory (if it's al= ready there),<br> > or trigger page fault trap that eventually lead to reading the missing= data from<br> > disk and return it to process as well.<br> > <br> > One thing that raise some concerns is that the file size is only 24k a= nd the<br> > mapping spans over 256M. However, the file's meta data seems to be= coherent to<br> > file contents.<br> > <br> > Any idea how did it happen, and what exactly in the file cause this co= rruption ?<br> > <br> > <br> > CRASH LOG :<br> > ----------------------------------------------------------------------= <br> > Exception Type:=C2=A0 =C2=A0 =C2=A0 =C2=A0 EXC_BAD_ACCESS (SIGBUS)<br> > Exception Codes:=C2=A0 =C2=A0 =C2=A0 =C2=A0KERN_MEMORY_ERROR at 0x0000= 00010648800a<br> > Exception Note:=C2=A0 =C2=A0 =C2=A0 =C2=A0 EXC_CORPSE_NOTIFY<br> > <br> > Termination Signal:=C2=A0 =C2=A0 Bus error: 10<br> > Termination Reason:=C2=A0 =C2=A0 Namespace SIGNAL, Code 0xa<br> > Terminating Process:=C2=A0 =C2=A0exc handler [0]<br> > <br> > VM Regions Near 0x10648800a:<br> > <br> > __LINKEDIT=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0000000010646= 4000-000000010647f000 [=C2=A0 108K] r--/rwx SM=3DCOW<br> >=C2=A0 ^Z^C [/usr/lib/dyld]<br> > --> mapped file=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0000000106= 47f000-000000011647f000 [256.0M] r--/rwx<br> > SM=3DPRV=C2=A0 Object_id=3D9034edd9<br> > STACK GUARD=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 000070000f2b3000-= 000070000f2b4000 [=C2=A0 =C2=A0 4K] ---/rwx SM=3DNUL<br> >=C2=A0 stack guard for thread 1<br> > <br> > Thread 0 Crashed:: Dispatch queue: com.apple.main-thread<br> > 0=C2=A0 =C2=A0myprog=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A00x0000000101666756 mdb_page_search_root + 39<br> > 1=C2=A0 =C2=A0myprog=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A00x00000001016660f7 mdb_page_search + 182<br> > 2=C2=A0 =C2=A0myprog=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A00x00000001016614de mdb_cursor_set + 88<br> > 3=C2=A0 =C2=A0myprog=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A00x0000000101661476 mdb_get + 134<br> > <br> > <br> > <br> <br> <br> -- <br> =C2=A0 -- Howard Chu<br> =C2=A0 CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"= http://www.symas.com" rel=3D"noreferrer" target=3D"_blank">http://www.symas= .com</a><br> =C2=A0 Director, Highland Sun=C2=A0 =C2=A0 =C2=A0<a href=3D"http://highland= sun.com/hyc/" rel=3D"noreferrer" target=3D"_blank">http://highlandsun.com/h= yc/</a><br> =C2=A0 Chief Architect, OpenLDAP=C2=A0 <a href=3D"http://www.openldap.org/p= roject/" rel=3D"noreferrer" target=3D"_blank">http://www.openldap.org/proje= ct/</a><br> </blockquote></div> --000000000000d405310575a6de5e--

1 0

Re: (ITS#8898) mdb_get on corrupted LMDB file crash the process
by hyc＠symas.com 10 Sep '18

10 Sep '18

iradization(a)gmail.com wrote: > Full_Name: Irad.k > Version: recent from GitHub. > OS: macOS > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (82.81.84.130) > > > Hi, > > I'm working with LMDB with the following flags : MDB_NOMETASYNC | MDB_NOSUBDIR > | MDB_NOTLS. > > Somehow, even though the DB should be ACI(without the D), it got corrupted > after recovering from kernel panic, and It crashes my process when trying to > access one of the records (see crash log below). > > Here's a link to the file : > > https://drive.google.com/file/d/12Q3KYYrapiJOgiaccnDL3tQACkqrM3C5/view?usp=… This file is only 6 pages long, but the latest meta page claims that it's using 11 pages. Looking at the previous meta page, it claims that 9 pages were used. Clearly your OS failed to write the complete contents out before it panicked. At the very least the file should have been 9 pages long. Sounds like you have an OS bug. > According to the crash log from the process, It can clearly be seen that the > invalid address reside inside the mapped file region which is the lmdb mapped > file, but still I get KERN_MEMORY_ERROR on that address. > >>From what I know, an attempt to access address within the mapped range can > either retrieve the page contents directly from memory (if it's already there), > or trigger page fault trap that eventually lead to reading the missing data from > disk and return it to process as well. > > One thing that raise some concerns is that the file size is only 24k and the > mapping spans over 256M. However, the file's meta data seems to be coherent to > file contents. > > Any idea how did it happen, and what exactly in the file cause this corruption ? > > > CRASH LOG : > ---------------------------------------------------------------------- > Exception Type: EXC_BAD_ACCESS (SIGBUS) > Exception Codes: KERN_MEMORY_ERROR at 0x000000010648800a > Exception Note: EXC_CORPSE_NOTIFY > > Termination Signal: Bus error: 10 > Termination Reason: Namespace SIGNAL, Code 0xa > Terminating Process: exc handler [0] > > VM Regions Near 0x10648800a: > > __LINKEDIT 0000000106464000-000000010647f000 [ 108K] r--/rwx SM=COW > ^Z^C [/usr/lib/dyld] > --> mapped file 000000010647f000-000000011647f000 [256.0M] r--/rwx > SM=PRV Object_id=9034edd9 > STACK GUARD 000070000f2b3000-000070000f2b4000 [ 4K] ---/rwx SM=NUL > stack guard for thread 1 > > Thread 0 Crashed:: Dispatch queue: com.apple.main-thread > 0 myprog 0x0000000101666756 mdb_page_search_root + 39 > 1 myprog 0x00000001016660f7 mdb_page_search + 182 > 2 myprog 0x00000001016614de mdb_cursor_set + 88 > 3 myprog 0x0000000101661476 mdb_get + 134 > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8901) slapd segfaults in mdb_env_reader_dest
by hyc＠symas.com 10 Sep '18

10 Sep '18

ckowalczyk(a)suse.com wrote: > Full_Name: Chris Kowalczyk > Version: 2.4.41 > OS: SLE12 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (2620:113:80c0:5::2222) > > > Hello, > I have been experiencing segfaults in mdb_env_reader_dest happening randomly, > only from time to time, during shutting off openldap. > > After some investigation I narrowed the problem down to mdb_env_reader_dest > destructor function being called after its object had been deleted while > powering off the application. mdb_env_reader_dest was called automatically, > because it was agginned to the object in pthread_key_create call: > https://github.com/openldap/openldap/blob/master/libraries/liblmdb/mdb.c#L4… > > The problem was not happening often, and always during powering off, so no data > was lost etc. However, due to its annoyance, I prepared a simple patch, which > prevents mdb_env_reader_dest function being called if the object was already > deleted. The patch works fine, I haven't seen the problem since it was > introduced. > > I am using OpenLDAP 2.4.41, but the code seems to be the same in other versions > as well. Would you like me to create a pull request for it? We've discussed this extensively in private emails. There is no documented reason why the provided patch would have the claimed effect. Deferring this until a valid explanation for why this patch is correct can be shown. > > ========================== > === The backtrace: > ========================== > > Program terminated with signal 11, Segmentation fault. > #0 mdb_env_reader_dest (ptr=0x7f4a9c71c080) at > ../../../libraries/liblmdb/mdb.c:4050 > 4050 reader->mr_pid = 0; > > #thread apply all bt > > Thread 2 (LWP 20920): > #0 0x00007f4a9ac31d9d in close () at ../sysdeps/unix/syscall-template.S:84 > #1 0x000055902eaeae8c in mdb_env_close1 (env=env@entry=0x55902fce50e0) > at ../../../libraries/liblmdb/mdb.c:4756 > #2 0x000055902eaeb578 in mdb_env_close1 (env=0x55902fce50e0) at > ../../../libraries/liblmdb/mdb.c:4779 > #3 mdb_env_close (env=0x55902fce50e0) at ../../../libraries/liblmdb/mdb.c:4778 > #4 0x000055902ea69114 in mdb_db_close (be=0x55902faeb300, cr=<optimized out>) > at init.c:340 > #5 0x000055902ea3815b in over_db_close (be=0x55902faeb300, cr=0x0) at > backover.c:182 > #6 0x000055902e9d8c0b in backend_shutdown (be=0x55902faeb300, be@entry=0x0) at > backend.c:376 > #7 0x000055902e9fa258 in slap_shutdown (be=be@entry=0x0) at init.c:232 > #8 0x000055902e9af12f in main (argc=<optimized out>, argv=<optimized out>) at > main.c:1027 > > Thread 1 (LWP 20928): > #0 mdb_env_reader_dest (ptr=0x7f4a9c71c080) at > ../../../libraries/liblmdb/mdb.c:4050 > #1 0x00007f4a9ac2a512 in __nptl_deallocate_tsd () at pthread_create.c:176 > #2 0x00007f4a9ac2a767 in start_thread (arg=0x7f4a97acc700) at > pthread_create.c:347 > #3 0x00007f4a9a968aad in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 > > # frame 0 > #0 mdb_env_reader_dest (ptr=0x7f4a9c71c080) at > ../../../libraries/liblmdb/mdb.c:4050 > 4050 reader->mr_pid = 0; > #p reader > $1 = (MDB_reader *) 0x7f4a9c71c080 > #p *reader > Cannot access memory at address 0x7f4a9c71c080 > > ========================== > === The proposed patch: > ========================== > > diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c > index 6bdf3151d..56212151b 100644 > --- a/libraries/liblmdb/mdb.c > +++ b/libraries/liblmdb/mdb.c > @@ -4692,6 +4692,11 @@ mdb_env_close0(MDB_env *env, int excl) > > if (env->me_flags & MDB_ENV_TXKEY) { > pthread_key_delete(env->me_txkey); > + > + // No need to call desctructor anymore, as all pid > + // values are cleared below. > + env->me_txkey = NULL; > + > #ifdef _WIN32 > /* Delete our key from the global list */ > for (i=0; i<mdb_tls_nkeys; i++) > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8913) Add meson build system to lmdb
by nacho.resa＠gmail.com 05 Sep '18

05 Sep '18

--0000000000006c1a190575262da9 Content-Type: text/plain; charset="UTF-8" not super important since it is anyway quite fast but takes half of the time running meson + ninja than running make -j6... :) real 0m3.070s user 0m7.031s sys 0m0.493s vs real 0m1.501s user 0m2.138s sys 0m0.446s In any case it is not the end of the world to keep a fork for gvsbuild and maybe fedora, at the end of the day this library does not change that often to make the build system painful to maintain out of upstream. On Wed, Sep 5, 2018 at 7:15 PM Howard Chu <hyc(a)symas.com> wrote: > Ignacio Casal Quinteiro wrote: > > FWIW any chance to reconsider having both build systems? > > I have no desire to support multiple build systems. The Project does not > support > any tools that we don't use ourselves. > > The files you add significantly increase the line count, and don't even > contain any comments. > In contrast, the current Makefile is mostly comments. > > > with meson it is easy to build on windows. Plus I'd like you > > to see at these numbers: > > > > time make: > > real 0m5.629s > > user 0m5.197s > > sys 0m0.399s > > > > time meson build > > real 0m0.452s > > user 0m0.335s > > sys 0m0.131s > > > > time ninja -C build > > real 0m0.992s > > user 0m1.812s > > sys 0m0.362s > > > > You can see that configuring and building with meson is faster than just > using make > > On my system the majority of build time is in gcc itself. The proportion > of time that > make itself uses is trivial. I would guess the only reason you're seeing a > speedup is > because meson parallelizes the build. You would get the same effect using > "make -jXX". > > > > > On Wed, Sep 5, 2018 at 6:31 PM Ignacio Casal Quinteiro < > nacho.resa(a)gmail.com <mailto:nacho.resa@gmail.com>> wrote: > > > > yeah... knowing my previous tries with lmdb I did not have much > hope... > > > > On Wed, Sep 5, 2018 at 6:21 PM Howard Chu <hyc(a)symas.com <mailto: > hyc(a)symas.com>> wrote: > > > > nacho.resa(a)gmail.com <mailto:nacho.resa@gmail.com> wrote: > > > Full_Name: Ignacio Casal Quinteiro > > > Version: > > > OS: > > > URL: ftp://ftp.openldap.org/incoming/ > > > Submission from: (NULL) (54.239.6.177) > > > > > > > > > Meson is the new thing out there, > > > > No. > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ > -- Ignacio Casal Quinteiro --0000000000006c1a190575262da9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div dir=3D"ltr"><div>not super important= since it is anyway quite fast but takes half of the time running meson + n= inja than running make -j6... :)</div><div>real=C2=A0=C2=A0=C2=A0 0m3.070s<= br>user=C2=A0=C2=A0=C2=A0 0m7.031s<br>sys=C2=A0=C2=A0=C2=A0 0m0.493s<br></d= iv><div><br></div><div>vs</div><div><br></div><div>real=C2=A0=C2=A0=C2=A0 0= m1.501s<br>user=C2=A0=C2=A0=C2=A0 0m2.138s<br>sys=C2=A0=C2=A0=C2=A0 0m0.446= s<br></div><div><br></div><div>In any case it is not the end of the world t= o keep a fork for gvsbuild and maybe fedora, at the end of the day this lib= rary</div><div>does not change that often to make the build system painful = to maintain out of upstream.<br></div></div></div></div><br><div class=3D"g= mail_quote"><div dir=3D"ltr">On Wed, Sep 5, 2018 at 7:15 PM Howard Chu <= <a href=3D"mailto:hyc@symas.com">hyc(a)symas.com</a>> wrote:<br></div><blo= ckquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #c= cc solid;padding-left:1ex">Ignacio Casal Quinteiro wrote:<br> > FWIW any chance to reconsider having both build systems?<br> <br> I have no desire to support multiple build systems. The Project does not su= pport<br> any tools that we don't use ourselves.<br> <br> The files you add significantly increase the line count, and don't even= contain any comments.<br> In contrast, the current Makefile is mostly comments.<br> <br> > with meson it is easy to build on windows. Plus I'd like you<br> > to see at these numbers:<br> > <br> > time make:<br> > real=C2=A0=C2=A0=C2=A0 0m5.629s<br> > user=C2=A0=C2=A0=C2=A0 0m5.197s<br> > sys=C2=A0=C2=A0=C2=A0 0m0.399s<br> > <br> > time meson build<br> > real=C2=A0=C2=A0=C2=A0 0m0.452s<br> > user=C2=A0=C2=A0=C2=A0 0m0.335s<br> > sys=C2=A0=C2=A0=C2=A0 0m0.131s<br> > <br> > time ninja -C build<br> > real=C2=A0=C2=A0=C2=A0 0m0.992s<br> > user=C2=A0=C2=A0=C2=A0 0m1.812s<br> > sys=C2=A0=C2=A0=C2=A0 0m0.362s<br> > <br> > You can see that configuring and building with meson is faster than ju= st using make<br> <br> On my system the majority of build time is in gcc itself. The proportion of= time that<br> make itself uses is trivial. I would guess the only reason you're seein= g a speedup is<br> because meson parallelizes the build. You would get the same effect using &= quot;make -jXX".<br> <br> > <br> > On Wed, Sep 5, 2018 at 6:31 PM Ignacio Casal Quinteiro <<a href=3D"= mailto:nacho.resa@gmail.com" target=3D"_blank">nacho.resa(a)gmail.com</a> &lt= ;mailto:<a href=3D"mailto:nacho.resa@gmail.com" target=3D"_blank">nacho.res= a(a)gmail.com</a>>> wrote:<br> > <br> >=C2=A0 =C2=A0 =C2=A0yeah... knowing my previous tries with lmdb I did n= ot have much hope...<br> > <br> >=C2=A0 =C2=A0 =C2=A0On Wed, Sep 5, 2018 at 6:21 PM Howard Chu <<a hr= ef=3D"mailto:hyc@symas.com" target=3D"_blank">hyc(a)symas.com</a> <mailto:= <a href=3D"mailto:hyc@symas.com" target=3D"_blank">hyc(a)symas.com</a>>&gt= ; wrote:<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"mailto:nacho.resa@gmail.co= m" target=3D"_blank">nacho.resa(a)gmail.com</a> <mailto:<a href=3D"mailto:= nacho.resa(a)gmail.com" target=3D"_blank">nacho.resa(a)gmail.com</a>> wrote:= <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> Full_Name: Ignacio Casal Quintei= ro<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> Version:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> OS:<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> URL: <a href=3D"ftp://ftp.openld= ap.org/incoming/" rel=3D"noreferrer" target=3D"_blank">ftp://ftp.openldap.o= rg/incoming/</a><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> Submission from: (NULL) (54.239.= 6.177)<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0><br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0> Meson is the new thing out there= ,<br> > <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0No.<br> <br> -- <br> =C2=A0 -- Howard Chu<br> =C2=A0 CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"= http://www.symas.com" rel=3D"noreferrer" target=3D"_blank">http://www.symas= .com</a><br> =C2=A0 Director, Highland Sun=C2=A0 =C2=A0 =C2=A0<a href=3D"http://highland= sun.com/hyc/" rel=3D"noreferrer" target=3D"_blank">http://highlandsun.com/h= yc/</a><br> =C2=A0 Chief Architect, OpenLDAP=C2=A0 <a href=3D"http://www.openldap.org/p= roject/" rel=3D"noreferrer" target=3D"_blank">http://www.openldap.org/proje= ct/</a><br> </blockquote></div><br clear=3D"all"><br>-- <br><div dir=3D"ltr" class=3D"g= mail_signature" data-smartmail=3D"gmail_signature">Ignacio Casal Quinteiro<= /div> --0000000000006c1a190575262da9--

1 0

Re: (ITS#8913) Add meson build system to lmdb
by hyc＠symas.com 05 Sep '18

05 Sep '18

Ignacio Casal Quinteiro wrote: > FWIW any chance to reconsider having both build systems? I have no desire to support multiple build systems. The Project does not = support any tools that we don't use ourselves. The files you add significantly increase the line count, and don't even c= ontain any comments. In contrast, the current Makefile is mostly comments. > with meson it is easy to build on windows. Plus I'd like you > to see at these numbers: >=20 > time make: > real=C2=A0=C2=A0=C2=A0 0m5.629s > user=C2=A0=C2=A0=C2=A0 0m5.197s > sys=C2=A0=C2=A0=C2=A0 0m0.399s >=20 > time meson build > real=C2=A0=C2=A0=C2=A0 0m0.452s > user=C2=A0=C2=A0=C2=A0 0m0.335s > sys=C2=A0=C2=A0=C2=A0 0m0.131s >=20 > time ninja -C build > real=C2=A0=C2=A0=C2=A0 0m0.992s > user=C2=A0=C2=A0=C2=A0 0m1.812s > sys=C2=A0=C2=A0=C2=A0 0m0.362s >=20 > You can see that configuring and building with meson is faster than jus= t using make On my system the majority of build time is in gcc itself. The proportion = of time that make itself uses is trivial. I would guess the only reason you're seeing = a speedup is because meson parallelizes the build. You would get the same effect using= "make -jXX". >=20 > On Wed, Sep 5, 2018 at 6:31 PM Ignacio Casal Quinteiro <nacho.resa@gmai= l.com <mailto:nacho.resa@gmail.com>> wrote: >=20 > yeah... knowing my previous tries with lmdb I did not have much hop= e... >=20 > On Wed, Sep 5, 2018 at 6:21 PM Howard Chu <hyc(a)symas.com <mailto:hy= c(a)symas.com>> wrote: >=20 > nacho.resa(a)gmail.com <mailto:nacho.resa@gmail.com> wrote: > > Full_Name: Ignacio Casal Quinteiro > > Version: > > OS: > > URL: ftp://ftp.openldap.org/incoming/ > > Submission from: (NULL) (54.239.6.177) > > > > > > Meson is the new thing out there, >=20 > No. --=20 -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs