openldap-bugs

openldap-bugs@openldap.org

1 participants
21065 discussions

Re: (ITS#9037) observing crash in mdb_cursor_put()
by vporof＠mozilla.com 13 Aug '19

13 Aug '19

--Apple-Mail=_F8BCAB64-A19B-4B00-B41B-03BE06C12A1F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On 6. Aug 2019, at 13:43, Victor Porof <vporof(a)mozilla.com> wrote: >=20 > Here's everything we know about: = https://crash-stats.mozilla.org/report/index/5d77bd19-41ce-459f-9c1c-7f9fb= 0190324 See the "details" and "telemetry environment" sections for a = breakdown. Since that link is now expired, here=E2=80=99s an up to date place to = see all recent crash reports involving `mdb_cursor_put`: = https://crash-stats.mozilla.org/signature/?signature=3Dmdb_cursor_put#repo= rts = <https://crash-stats.mozilla.org/signature/?signature=3Dmdb_cursor_put#rep= orts>=20 Victor= --Apple-Mail=_F8BCAB64-A19B-4B00-B41B-03BE06C12A1F Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D""><br = class=3D""><div><blockquote type=3D"cite" class=3D""><div class=3D"">On = 6. Aug 2019, at 13:43, Victor Porof <<a = href=3D"mailto:vporof@mozilla.com" class=3D"">vporof(a)mozilla.com</a>> = wrote:</div><div class=3D""><div class=3D""><br class=3D"">Here's = everything we know about: <a = href=3D"https://crash-stats.mozilla.org/report/index/5d77bd19-41ce-459f-9c= 1c-7f9fb0190324" = class=3D"">https://crash-stats.mozilla.org/report/index/5d77bd19-41ce-459f= -9c1c-7f9fb0190324</a> See the "details" and "telemetry environment" = sections for a breakdown.<br class=3D""></div></div></blockquote><div><br = class=3D""></div>Since that link is now expired, here=E2=80=99s an up to = date place to see all recent crash reports involving = `mdb_cursor_put`: <a = href=3D"https://crash-stats.mozilla.org/signature/?signature=3Dmdb_cursor_= put#reports" = class=3D"">https://crash-stats.mozilla.org/signature/?signature=3Dmdb_curs= or_put#reports</a> <div class=3D""><br class=3D""></div><div = class=3D"">Victor</div></div></body></html>= --Apple-Mail=_F8BCAB64-A19B-4B00-B41B-03BE06C12A1F--

1 0

Re: (ITS#9062) Please have /build/mkdep respect TMPDIR if set
by quanah＠symas.com 13 Aug '19

13 Aug '19

--On Sunday, August 11, 2019 7:34 PM +0000 zagarc(a)oclc.org wrote: > Would you please consider changing this to: > > TMP=${TMPDIR-/tmp}/mkdep$$ Thanks for the report, this is fixed in OpenLDAP master. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9063) tls_reqcert in slapd.conf/slapd-config man pages should be bold
by quanah＠openldap.org 12 Aug '19

12 Aug '19

Full_Name: Quanah Gibson-Mount Version: 2.4.48 OS: N/A URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (47.208.128.44) When looking at the man pages, the tls_reqcert description in the section of the man page about syncrepl is missing a BOLD tag, unlike the other keywords. Trivial fix.

1 0

(ITS#9062) Please have /build/mkdep respect TMPDIR if set
by zagarc＠oclc.org 11 Aug '19

11 Aug '19

Full_Name: Chris Zagar Version: 2.4.48 OS: Linux URL: Submission from: (NULL) (68.98.212.84) /build/mkdep contains this line: TMP=/tmp/mkdep$$ that forces the use of the /tmp directory. The /tmp directory is vulnerable to race conditions. The rest of OpenLDAP obeys the TMPDIR environment variable if it exists as a mitigation to this risk. Would you please consider changing this to: TMP=${TMPDIR-/tmp}/mkdep$$ so this will obey TMPDIR as well? Thank you. Chris Zagar zagarc(a)oclc.org

1 0

Re: (ITS#9061) LDAP replication issue on a big network
by quanah＠symas.com 07 Aug '19

07 Aug '19

--On Wednesday, August 07, 2019 10:35 AM +0000 alex.s(a)wildix.com wrote: > Full_Name: Alex > Version: 2.4.44+dfsg-5+deb9u2 Hello, The ITS system is for bug reports, not usage questions. Additionally, if you're doing replication of any type, please upgrade to the latest release (2.4.48). Ensure you are using delta-syncrepl. For any other questions, please use the correct forum which is the openldap-technical(a)openldap.org mailing list: <https://www.openldap.org/lists/mm/listinfo/openldap-technical>. I believe there is a backport of 2.4.48 for Debian9 from the backports repo. This ITS will be closed. Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

(ITS#9061) LDAP replication issue on a big network
by alex.s＠wildix.com 07 Aug '19

07 Aug '19

Full_Name: Alex Version: 2.4.44+dfsg-5+deb9u2 OS: Debian 9 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (154.41.3.130) Hello I have some issues with replication between Master and Slave LDAP servers. Preconditions data: I have a Master server with a huge LDAP data And also I have about 200 Slave servers around the world which have a data replica in their databases. What happened: I have applied a bckup on Master server. (Actually data have not changed except entryCSN) entryCSN has been deleted from backup previously, before apply, for actualise data in LDAP database. As a result I had a situation when all Slave servers starts replication. The first question is: how I can avoid full replication after apply backup on Master? I understand that entryCSN changed and LDAP should sync some objects. But can I use another way to actualize the data in LDAP database instead of entryCSN? The second question is: after apply backup and restarts the Master server I had a problem with local LDAP because all Slave servers start connecting and start their replication. In this case LDAP on Master server not responding even on localhost via ldapsearch. May I change some parameters to increase concurrent connections? Because LDAP starts not responding if the quantity of simultaneous connected of Slave servers exceeds 10-15 Thank you in advance Best regards, Alex

1 0

Re: (ITS#9060) lmdb corruption after many memory map resizes
by ruan.declercq＠netronome.com 06 Aug '19

06 Aug '19

--00000000000075fbb5058f76dbf0 Content-Type: text/plain; charset="UTF-8" The documentation states: "It may be called at later times if no transactions are active in this process. Note that the library does not check for this condition, the caller must ensure it explicitly." Maybe I'm missing something, but my code aims to have no active transactions in this process when I do the resize. That's why I first do the txn_commit() and then the resize. On Tue, Aug 6, 2019 at 6:08 PM Howard Chu <hyc(a)symas.com> wrote: > ruan.declercq(a)netronome.com wrote: > > Full_Name: Ruan de Clercq > > Version: > > OS: Ubuntu 19.04 > > URL: ftp://ftp.openldap.org/incoming/ > > Submission from: (NULL) (155.93.214.171) > > > > > > Hi, > > > > I am using 0.9.23-0ubuntu1 on ubuntu 19.04. > > > > I have an application where the exact db size isn't known beforehand. > Therefore, > > I set the memory map size beforehand, and then increase the size when I > run out > > of space. > > This is incorrect usage of LMDB. The docs explicitly state to use a large > size at > the beginning and leave it alone. > > > However, when I use lmdb in concurrent processes, and frequently > > increase the size of the memory map, I eventually get a corrupt database > > (MDB_CORRUPTED). > > Your code is incorrect. The docs explicitly state that there must not be > any outstanding > activity when using the set_mapsize call. > > Closing this ITS. > > > Here's a minimal code example: > > > > #include <stdio.h> > > #include <stdlib.h> > > #include <inttypes.h> > > #include <time.h> > > #include "lmdb.h" > > #include <cstring> > > > > int resize(MDB_env *env) { > > // read map size and increase > > int val; > > MDB_envinfo stat; > > MDB_txn *txn; > > if ((val = mdb_env_info(env, &stat) != MDB_SUCCESS) || > > (val = mdb_env_set_mapsize(env, stat.me_mapsize + 4096) != > MDB_SUCCESS) > > || > > (val = mdb_env_info(env, &stat) != MDB_SUCCESS)) { > > printf("set new mapsize %d\n", val); > > return val; > > } > > printf("Resized to %lu\n", stat.me_mapsize); > > if (((val = mdb_txn_begin(env, NULL, 0, &txn) != MDB_SUCCESS)) || > > ((val = mdb_txn_commit(txn)))) { > > printf("txn %d\n", val); > > } > > return val; > > } > > > > int main(int argc,char * argv[]) > > { > > timespec tp; > > clock_gettime(CLOCK_MONOTONIC, &tp); > > srand(tp.tv_sec + tp.tv_nsec); > > > > MDB_env *env = NULL; > > MDB_txn *txn = NULL; > > MDB_dbi dbi; > > int val; > > if ((val = mdb_env_create(&env) != MDB_SUCCESS) || > > (val = mdb_env_set_maxreaders(env, 126) != MDB_SUCCESS)) { > > printf("Create error %d", val); > > goto out; > > } > > > > if ((val = mdb_env_set_mapsize(env, 4096 * 128) != MDB_SUCCESS) || > > (val = mdb_env_set_maxdbs(env, 1) != MDB_SUCCESS) || > > (val = mdb_env_open(env, "./data/", MDB_MAPASYNC | MDB_WRITEMAP, > 0664) != > > MDB_SUCCESS) || > > (val = mdb_txn_begin(env, NULL, 0, &txn) != MDB_SUCCESS) || > > (val = mdb_dbi_open(txn, "test", MDB_CREATE, &dbi) != MDB_SUCCESS) > || > > (val = mdb_txn_commit(txn) != MDB_SUCCESS)) { > > printf("create db %d\n", val); > > goto out; > > } > > > > for (int i = 0; i < 1000; i++) { > > char sval[32]; > > sprintf(sval, "%03x %d foo bar", 32, rand()); > > MDB_val mdb_key, mdb_val; > > mdb_key.mv_size = strlen(sval);; > > mdb_key.mv_data = sval; > > mdb_val.mv_size = strlen(sval); > > mdb_val.mv_data = sval; > > > > if ((val = mdb_txn_begin(env, NULL, 0, &txn) != MDB_SUCCESS)) { > > printf("mdb_txn_begin %d\n", val); > > goto out; > > } > > if ((val = mdb_put(txn, dbi, &mdb_key, &mdb_val, 0)) != MDB_SUCCESS) > { > > mdb_txn_commit(txn); > > if (val == MDB_MAP_FULL) { > > val = resize(env); > > continue; > > } > > printf("mdb_put %d\n", val); > > goto out; > > } > > if ((val = mdb_txn_commit(txn)) != MDB_SUCCESS) { > > if (val == MDB_MAP_FULL) { > > val = resize(env); > > continue; > > } > > printf("mdb_txn_commit %d\n", val); > > goto out; > > } > > } > > out: > > if (env) { > > mdb_dbi_close(env, dbi); > > mdb_env_close(env); > > } > > printf("OK\n"); > > } > > > > After compiling I run the following command a couple of times: > > for i in {1..20}; do echo $i; (lmdb_test &); done > > > > > > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/ > -- Regards, Ruan de Clercq --00000000000075fbb5058f76dbf0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><div>The documentation states:</div><div>= "It may be called at later times if no transactions are active in=C2= =A0this process. Note that the library does not check for this condition, t= he caller must ensure it explicitly."</div><div>Maybe I'm missing = something, but my code aims to have no active transactions in this process = when I do the resize. That's why I first do the txn_commit() and then t= he resize.</div><div><br></div><div>On Tue, Aug 6, 2019 at 6:08 PM Howard C= hu <<a href=3D"mailto:hyc@symas.com">hyc(a)symas.com</a>> wrote:</div><= div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"margin= :0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-lef= t-color:rgb(204,204,204);padding-left:1ex"><a href=3D"mailto:ruan.declercq@= netronome.com" target=3D"_blank">ruan.declercq(a)netronome.com</a> wrote:<br> > Full_Name: Ruan de Clercq<br> > Version: <br> > OS: Ubuntu 19.04<br> > URL: <a href=3D"ftp://ftp.openldap.org/incoming/" rel=3D"noreferrer" t= arget=3D"_blank">ftp://ftp.openldap.org/incoming/</a><br> > Submission from: (NULL) (155.93.214.171)<br> > <br> > <br> > Hi,<br> > <br> > I am using 0.9.23-0ubuntu1 on ubuntu 19.04. <br> > <br> > I have an application where the exact db size isn't known beforeha= nd. Therefore,<br> > I set the memory map size beforehand, and then increase the size when = I run out<br> > of space.<br> <br> This is incorrect usage of LMDB. The docs explicitly state to use a large s= ize at<br> the beginning and leave it alone.<br> <br> > However, when I use lmdb in concurrent processes, and frequently<br> > increase the size of the memory map, I eventually get a corrupt databa= se<br> > (MDB_CORRUPTED).<br> <br> Your code is incorrect. The docs explicitly state that there must not be an= y outstanding<br> activity when using the set_mapsize call.<br> <br> Closing this ITS.<br> ><br> =C2=A0Here's a minimal code example:<br> > <br> > #include <stdio.h><br> > #include <stdlib.h><br> > #include <inttypes.h><br> > #include <time.h><br> > #include "lmdb.h"<br> > #include <cstring><br> > <br> > int resize(MDB_env *env) {<br> >=C2=A0 =C2=A0// read map size and increase<br> >=C2=A0 =C2=A0int val;<br> >=C2=A0 =C2=A0MDB_envinfo stat;<br> >=C2=A0 =C2=A0MDB_txn *txn;<br> >=C2=A0 =C2=A0if ((val =3D mdb_env_info(env, &stat) !=3D MDB_SUCCESS= ) ||<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0(val =3D mdb_env_set_mapsize(env, stat.me_ma= psize + 4096) !=3D MDB_SUCCESS)<br> > ||<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0(val =3D mdb_env_info(env, &stat) !=3D M= DB_SUCCESS)) {<br> >=C2=A0 =C2=A0 =C2=A0printf("set new mapsize %d\n", val);<br> >=C2=A0 =C2=A0 =C2=A0return val;<br> >=C2=A0 =C2=A0}<br> >=C2=A0 =C2=A0printf("Resized to %lu\n", stat.me_mapsize);<br> >=C2=A0 =C2=A0if (((val =3D mdb_txn_begin(env, NULL, 0, &txn) !=3D M= DB_SUCCESS)) ||<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0((val =3D mdb_txn_commit(txn)))) {<br> >=C2=A0 =C2=A0 =C2=A0printf("txn %d\n", val);<br> >=C2=A0 =C2=A0}<br> >=C2=A0 =C2=A0return val;<br> > }<br> > <br> > int main(int argc,char * argv[])<br> > {<br> >=C2=A0 =C2=A0timespec tp;<br> >=C2=A0 =C2=A0clock_gettime(CLOCK_MONOTONIC, &tp);<br> >=C2=A0 =C2=A0srand(tp.tv_sec + tp.tv_nsec);<br> > <br> >=C2=A0 =C2=A0MDB_env *env =3D NULL;<br> >=C2=A0 =C2=A0MDB_txn *txn =3D NULL;<br> >=C2=A0 =C2=A0MDB_dbi dbi;<br> >=C2=A0 =C2=A0int val;<br> >=C2=A0 =C2=A0if ((val =3D mdb_env_create(&env) !=3D MDB_SUCCESS) ||= <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0(val =3D mdb_env_set_maxreaders(env, 126) != =3D MDB_SUCCESS)) {<br> >=C2=A0 =C2=A0 =C2=A0printf("Create error %d", val);<br> >=C2=A0 =C2=A0 =C2=A0goto out;<br> >=C2=A0 =C2=A0}<br> > <br> >=C2=A0 =C2=A0if ((val =3D mdb_env_set_mapsize(env, 4096 * 128) !=3D MDB= _SUCCESS) ||<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0(val =3D mdb_env_set_maxdbs(env, 1) !=3D MDB= _SUCCESS) ||<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0(val =3D mdb_env_open(env, "./data/&quo= t;, MDB_MAPASYNC | MDB_WRITEMAP, 0664) !=3D<br> > MDB_SUCCESS) ||<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0(val =3D mdb_txn_begin(env, NULL, 0, &tx= n) !=3D MDB_SUCCESS) ||<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0(val =3D mdb_dbi_open(txn, "test",= MDB_CREATE, &dbi) !=3D MDB_SUCCESS) ||<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0(val =3D mdb_txn_commit(txn) !=3D MDB_SUCCES= S)) {<br> >=C2=A0 =C2=A0 =C2=A0printf("create db %d\n", val);<br> >=C2=A0 =C2=A0 =C2=A0goto out;<br> >=C2=A0 =C2=A0}<br> > <br> >=C2=A0 =C2=A0for (int i =3D 0; i < 1000; i++) {<br> >=C2=A0 =C2=A0 =C2=A0char sval[32];<br> >=C2=A0 =C2=A0 =C2=A0sprintf(sval, "%03x %d foo bar", 32, rand= ());<br> >=C2=A0 =C2=A0 =C2=A0MDB_val mdb_key, mdb_val;<br> >=C2=A0 =C2=A0 =C2=A0mdb_key.mv_size =3D strlen(sval);;<br> >=C2=A0 =C2=A0 =C2=A0mdb_key.mv_data =3D sval;<br> >=C2=A0 =C2=A0 =C2=A0mdb_val.mv_size =3D strlen(sval);<br> >=C2=A0 =C2=A0 =C2=A0mdb_val.mv_data =3D sval;<br> > <br> >=C2=A0 =C2=A0 =C2=A0if ((val =3D mdb_txn_begin(env, NULL, 0, &txn) = !=3D MDB_SUCCESS)) {<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0printf("mdb_txn_begin %d\n", val);= <br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0goto out;<br> >=C2=A0 =C2=A0 =C2=A0}<br> >=C2=A0 =C2=A0 =C2=A0if ((val =3D mdb_put(txn, dbi, &mdb_key, &m= db_val, 0)) !=3D MDB_SUCCESS) {<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0mdb_txn_commit(txn);<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0if (val =3D=3D MDB_MAP_FULL) {<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0val =3D resize(env);<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0continue;<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0}<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0printf("mdb_put %d\n", val);<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0goto out;<br> >=C2=A0 =C2=A0 =C2=A0}<br> >=C2=A0 =C2=A0 =C2=A0if ((val =3D mdb_txn_commit(txn)) !=3D MDB_SUCCESS)= {<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0if (val =3D=3D MDB_MAP_FULL) {<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0val =3D resize(env);<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0continue;<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0}<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0printf("mdb_txn_commit %d\n", val)= ;<br> >=C2=A0 =C2=A0 =C2=A0 =C2=A0goto out;<br> >=C2=A0 =C2=A0 =C2=A0}<br> >=C2=A0 =C2=A0}<br> > out:<br> >=C2=A0 =C2=A0if (env) {<br> >=C2=A0 =C2=A0 =C2=A0mdb_dbi_close(env, dbi);<br> >=C2=A0 =C2=A0 =C2=A0mdb_env_close(env);<br> >=C2=A0 =C2=A0}<br> >=C2=A0 =C2=A0printf("OK\n");<br> > }<br> > <br> > After compiling I run the following command a couple of times:<br> > for i in {1..20}; do echo $i; (lmdb_test &); done<br> > <br> > <br> <br> <br> -- <br> =C2=A0 -- Howard Chu<br> =C2=A0 CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"= http://www.symas.com" rel=3D"noreferrer" target=3D"_blank">http://www.symas= .com</a><br> =C2=A0 Director, Highland Sun=C2=A0 =C2=A0 =C2=A0<a href=3D"http://highland= sun.com/hyc/" rel=3D"noreferrer" target=3D"_blank">http://highlandsun.com/h= yc/</a><br> =C2=A0 Chief Architect, OpenLDAP=C2=A0 <a href=3D"http://www.openldap.org/p= roject/" rel=3D"noreferrer" target=3D"_blank">http://www.openldap.org/proje= ct/</a><br> </blockquote></div><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"= class=3D"gmail_signature"><div dir=3D"ltr"><div>Regards,</div><div>Ruan de= Clercq<br></div></div></div></div></div> --00000000000075fbb5058f76dbf0--

1 0

Re: (ITS#9060) lmdb corruption after many memory map resizes
by hyc＠symas.com 06 Aug '19

06 Aug '19

ruan.declercq(a)netronome.com wrote: > Full_Name: Ruan de Clercq > Version: > OS: Ubuntu 19.04 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (155.93.214.171) > > > Hi, > > I am using 0.9.23-0ubuntu1 on ubuntu 19.04. > > I have an application where the exact db size isn't known beforehand. Therefore, > I set the memory map size beforehand, and then increase the size when I run out > of space. This is incorrect usage of LMDB. The docs explicitly state to use a large size at the beginning and leave it alone. > However, when I use lmdb in concurrent processes, and frequently > increase the size of the memory map, I eventually get a corrupt database > (MDB_CORRUPTED). Your code is incorrect. The docs explicitly state that there must not be any outstanding activity when using the set_mapsize call. Closing this ITS. > Here's a minimal code example: > > #include <stdio.h> > #include <stdlib.h> > #include <inttypes.h> > #include <time.h> > #include "lmdb.h" > #include <cstring> > > int resize(MDB_env *env) { > // read map size and increase > int val; > MDB_envinfo stat; > MDB_txn *txn; > if ((val = mdb_env_info(env, &stat) != MDB_SUCCESS) || > (val = mdb_env_set_mapsize(env, stat.me_mapsize + 4096) != MDB_SUCCESS) > || > (val = mdb_env_info(env, &stat) != MDB_SUCCESS)) { > printf("set new mapsize %d\n", val); > return val; > } > printf("Resized to %lu\n", stat.me_mapsize); > if (((val = mdb_txn_begin(env, NULL, 0, &txn) != MDB_SUCCESS)) || > ((val = mdb_txn_commit(txn)))) { > printf("txn %d\n", val); > } > return val; > } > > int main(int argc,char * argv[]) > { > timespec tp; > clock_gettime(CLOCK_MONOTONIC, &tp); > srand(tp.tv_sec + tp.tv_nsec); > > MDB_env *env = NULL; > MDB_txn *txn = NULL; > MDB_dbi dbi; > int val; > if ((val = mdb_env_create(&env) != MDB_SUCCESS) || > (val = mdb_env_set_maxreaders(env, 126) != MDB_SUCCESS)) { > printf("Create error %d", val); > goto out; > } > > if ((val = mdb_env_set_mapsize(env, 4096 * 128) != MDB_SUCCESS) || > (val = mdb_env_set_maxdbs(env, 1) != MDB_SUCCESS) || > (val = mdb_env_open(env, "./data/", MDB_MAPASYNC | MDB_WRITEMAP, 0664) != > MDB_SUCCESS) || > (val = mdb_txn_begin(env, NULL, 0, &txn) != MDB_SUCCESS) || > (val = mdb_dbi_open(txn, "test", MDB_CREATE, &dbi) != MDB_SUCCESS) || > (val = mdb_txn_commit(txn) != MDB_SUCCESS)) { > printf("create db %d\n", val); > goto out; > } > > for (int i = 0; i < 1000; i++) { > char sval[32]; > sprintf(sval, "%03x %d foo bar", 32, rand()); > MDB_val mdb_key, mdb_val; > mdb_key.mv_size = strlen(sval);; > mdb_key.mv_data = sval; > mdb_val.mv_size = strlen(sval); > mdb_val.mv_data = sval; > > if ((val = mdb_txn_begin(env, NULL, 0, &txn) != MDB_SUCCESS)) { > printf("mdb_txn_begin %d\n", val); > goto out; > } > if ((val = mdb_put(txn, dbi, &mdb_key, &mdb_val, 0)) != MDB_SUCCESS) { > mdb_txn_commit(txn); > if (val == MDB_MAP_FULL) { > val = resize(env); > continue; > } > printf("mdb_put %d\n", val); > goto out; > } > if ((val = mdb_txn_commit(txn)) != MDB_SUCCESS) { > if (val == MDB_MAP_FULL) { > val = resize(env); > continue; > } > printf("mdb_txn_commit %d\n", val); > goto out; > } > } > out: > if (env) { > mdb_dbi_close(env, dbi); > mdb_env_close(env); > } > printf("OK\n"); > } > > After compiling I run the following command a couple of times: > for i in {1..20}; do echo $i; (lmdb_test &); done > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#9037) observing crash in mdb_cursor_put()
by vporof＠mozilla.com 06 Aug '19

06 Aug '19

> On 1. Aug 2019, at 19:47, Howard Chu <hyc(a)symas.com> wrote: >=20 > vporof(a)mozilla.com wrote: >> Hey folks. >>=20 >> =3D46rom Myk=3DE2=3D80=3D99s investigations in the previous followup, = it seems =3D >> that the suggested changes to `mdb_cursor_init` to avoid using an =3D >> invalid DBI might not be solving the actual issue, given the = behaviour =3D >> of `mdb_page_search`. >=20 > Agreed, that assert that I suggested isn't catching what we want. FWIW, here's my findings when attempting to look into what was happening = with regards to that test case: = https://gist.github.com/victorporof/d1f98b8a52f79c7ff99f361d3a2adc3e It's unclear how much overlap there is with the previous findings, and = whether or not calling `mdb_put` should assert with the DBI previously = opened via `mdb_dbi_open` with MDB_CREATE. Let me know if what I'm = observing here is expected. >=20 >> It=3DE2=3D80=3D99s also causing the seemingly correct test program to = assert =3D >> when it wasn=3DE2=3D80=3D99t before. It=3DE2=3D80=3D99s unclear = whether this should =3D >> be the case or not, perhaps Howard can confirm the expected = behaviour. >>=20 >> In any case, we=3DE2=3D80=3D99re wondering if there=3DE2=3D80=3D99s = been any other =3D >> progress, or if someone managed to reproduce this issue? Shipping new = =3D >> features built on top of LMDB in Firefox is currently blocked due to = =3D >> these failures, so any additional info would be helpful. >=20 > Sorry, still not seeing this over here. What else do you know about = the > systems where this is occurring? RAM size, storage on HDD / SSD / USB = ? Here's everything we know about: = https://crash-stats.mozilla.org/report/index/5d77bd19-41ce-459f-9c1c-7f9fb= 0190324 See the "details" and "telemetry environment" sections for a = breakdown. Victor >=20 > --=20 > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#9060) lmdb corruption after many memory map resizes
by ruan.declercq＠netronome.com 06 Aug '19

06 Aug '19

Full_Name: Ruan de Clercq Version: OS: Ubuntu 19.04 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (155.93.214.171) Hi, I am using 0.9.23-0ubuntu1 on ubuntu 19.04. I have an application where the exact db size isn't known beforehand. Therefore, I set the memory map size beforehand, and then increase the size when I run out of space. However, when I use lmdb in concurrent processes, and frequently increase the size of the memory map, I eventually get a corrupt database (MDB_CORRUPTED). Here's a minimal code example: #include <stdio.h> #include <stdlib.h> #include <inttypes.h> #include <time.h> #include "lmdb.h" #include <cstring> int resize(MDB_env *env) { // read map size and increase int val; MDB_envinfo stat; MDB_txn *txn; if ((val = mdb_env_info(env, &stat) != MDB_SUCCESS) || (val = mdb_env_set_mapsize(env, stat.me_mapsize + 4096) != MDB_SUCCESS) || (val = mdb_env_info(env, &stat) != MDB_SUCCESS)) { printf("set new mapsize %d\n", val); return val; } printf("Resized to %lu\n", stat.me_mapsize); if (((val = mdb_txn_begin(env, NULL, 0, &txn) != MDB_SUCCESS)) || ((val = mdb_txn_commit(txn)))) { printf("txn %d\n", val); } return val; } int main(int argc,char * argv[]) { timespec tp; clock_gettime(CLOCK_MONOTONIC, &tp); srand(tp.tv_sec + tp.tv_nsec); MDB_env *env = NULL; MDB_txn *txn = NULL; MDB_dbi dbi; int val; if ((val = mdb_env_create(&env) != MDB_SUCCESS) || (val = mdb_env_set_maxreaders(env, 126) != MDB_SUCCESS)) { printf("Create error %d", val); goto out; } if ((val = mdb_env_set_mapsize(env, 4096 * 128) != MDB_SUCCESS) || (val = mdb_env_set_maxdbs(env, 1) != MDB_SUCCESS) || (val = mdb_env_open(env, "./data/", MDB_MAPASYNC | MDB_WRITEMAP, 0664) != MDB_SUCCESS) || (val = mdb_txn_begin(env, NULL, 0, &txn) != MDB_SUCCESS) || (val = mdb_dbi_open(txn, "test", MDB_CREATE, &dbi) != MDB_SUCCESS) || (val = mdb_txn_commit(txn) != MDB_SUCCESS)) { printf("create db %d\n", val); goto out; } for (int i = 0; i < 1000; i++) { char sval[32]; sprintf(sval, "%03x %d foo bar", 32, rand()); MDB_val mdb_key, mdb_val; mdb_key.mv_size = strlen(sval);; mdb_key.mv_data = sval; mdb_val.mv_size = strlen(sval); mdb_val.mv_data = sval; if ((val = mdb_txn_begin(env, NULL, 0, &txn) != MDB_SUCCESS)) { printf("mdb_txn_begin %d\n", val); goto out; } if ((val = mdb_put(txn, dbi, &mdb_key, &mdb_val, 0)) != MDB_SUCCESS) { mdb_txn_commit(txn); if (val == MDB_MAP_FULL) { val = resize(env); continue; } printf("mdb_put %d\n", val); goto out; } if ((val = mdb_txn_commit(txn)) != MDB_SUCCESS) { if (val == MDB_MAP_FULL) { val = resize(env); continue; } printf("mdb_txn_commit %d\n", val); goto out; } } out: if (env) { mdb_dbi_close(env, dbi); mdb_env_close(env); } printf("OK\n"); } After compiling I run the following command a couple of times: for i in {1..20}; do echo $i; (lmdb_test &); done

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs