openldap-bugs August 2014

openldap-bugs@openldap.org

16 participants
39 discussions

Re: (ITS#7915) pcache and translucent crashes OL
by quanah＠zimbra.com 04 Aug '14

04 Aug '14

--On Monday, August 04, 2014 10:16 PM +0300 Nikos Voutsinas=20 <nvoutsin(a)gmail.com> wrote: > I am aware of #7587, however based on the release dates shouldn't > this=C2=A0fix be=C2=A0included in 2.4.39; I just told you specifically that the fix was made after 2.4.39 was=20 released. Again, please test current RE24. Thanks! --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#7915) pcache and translucent crashes OL
by nvoutsin＠gmail.com 04 Aug '14

04 Aug '14

--001a11364318a34ed604ffd1be1c Content-Type: text/plain; charset=UTF-8 I am aware of #7587, however based on the release dates shouldn't this fix be included in 2.4.39; On Monday, August 4, 2014, Quanah Gibson-Mount <quanah(a)zimbra.com> wrote: > --On Monday, August 04, 2014 1:04 AM +0000 nvoutsin(a)gmail.com wrote: > > Full_Name: Nikos Voutsinas >> Version: 2.4.39 >> OS: Debian >> > > This looks like a duplicate of ITS#7587. Can you please re-test with > current code from RE24 where this is fixed? > > Thanks! > > --Quanah > > > > -- > > Quanah Gibson-Mount > Server Architect > Zimbra, Inc. > -------------------- > Zimbra :: the leader in open source messaging and collaboration > --001a11364318a34ed604ffd1be1c Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I am aware of #7587, however based on the release dates shouldn't this= =C2=A0fix be=C2=A0<span></span>included in 2.4.39;<br><br>On Monday, August= 4, 2014, Quanah Gibson-Mount <<a href=3D"mailto:quanah@zimbra.com">quan= ah(a)zimbra.com</a>> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex">--On Monday, August 04, 2014 1:04 AM +0000 <= a>nvoutsin(a)gmail.com</a> wrote:<br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Full_Name: Nikos Voutsinas<br> Version: 2.4.39<br> OS: Debian<br> </blockquote> <br> This looks like a duplicate of ITS#7587. =C2=A0Can you please re-test with = current code from RE24 where this is fixed?<br> <br> Thanks!<br> <br> --Quanah<br> <br> <br> <br> --<br> <br> Quanah Gibson-Mount<br> Server Architect<br> Zimbra, Inc.<br> --------------------<br> Zimbra :: =C2=A0the leader in open source messaging and collaboration<br> </blockquote> --001a11364318a34ed604ffd1be1c--

1 0

Re: (ITS#7915) pcache and translucent crashes OL
by quanah＠zimbra.com 04 Aug '14

04 Aug '14

--On Monday, August 04, 2014 1:04 AM +0000 nvoutsin(a)gmail.com wrote: > Full_Name: Nikos Voutsinas > Version: 2.4.39 > OS: Debian This looks like a duplicate of ITS#7587. Can you please re-test with current code from RE24 where this is fixed? Thanks! --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#7915) pcache and translucent crashes OL
by nvoutsin＠gmail.com 03 Aug '14

03 Aug '14

Full_Name: Nikos Voutsinas Version: 2.4.39 OS: Debian URL: http://users.uoa.gr/~nvoutsin/OpenLdap_ITS/add-trasnlucent(bdb)-pcache(bdb)… http://users.uoa.gr/~nvoutsin/OpenLdap_ITS/add-translucent(bdb)-pcache(bdb)… http://users.uoa.gr/~nvoutsin/OpenLdap_ITS/add-trasnlucent(bdb)-pcache%mdmd… http://users.uoa.gr/~nvoutsin/OpenLdap_ITS/add-translucent(bdb)-pcache(mdb)… http://users.uoa.gr/~nvoutsin/OpenLdap_ITS/add-trasnlucent(mdb)-pcache(mdb)… http://users.uoa.gr/~nvoutsin/OpenLdap_ITS/add-translucent(mdb)-pcache(mdb)… http://users.uoa.gr/~nvoutsin/OpenLdap_ITS/add-trasnlucent(mdb)-pcache(bdb)… http://users.uoa.gr/~nvoutsin/OpenLdap_ITS/add-translucent(mdb)-pcache(bdb)… Submission from: (NULL) (5.54.11.83) The combination of translucent and pcache overlays crashes OL or makes it unresponsive. Any attempt to modify an object on the replica by using the following ldif: dn: uid=joe,dc=foo,dc=com changetype: modify add: description description: first - dn: uid=joe,dc=foo,dc=com changetype: modify replace: description description: second - cause OL to fail either during the modification or on the subsequent operation. Specifically, OL behaves differently depending on the type of database backends that are used for translucent and pcache DBs, as it is shown below: translucent db,pcache db: comment bdb,bdb: freezes on the second modify op bdb,mdb: crashes on the secend modify op (invalid pointer) mdb,mdb: completes the modify ops. crashes on the first search on object mdb,bdb: completes the modify ops. crashes, if not on the first, then on subsequent op Attached files: a)the cn=config ldif for each of the above cases and b)the corresponding debug output during the modification of the user object or if that completes during the subsequent search on it.

1 0

Re: (ITS#7914) LMDB: MDB_DUPSORT and mdb_cursor_put(.., MDB_CURRENT) can put DB in illegal state
by h.b.furuseth＠usit.uio.no 03 Aug '14

03 Aug '14

On 03/08/14 20:13, hyc(a)symas.com wrote: > The recent doc changes are a bit over-zealous here. The only requirement is > that the new value would naturally sort into the same position. So yes, you > can store a value different than the old value. No, MDB_CURRENT does not update branch keys. That was the reason for those latest doc changes.

1 0

Re: (ITS#7914) LMDB: MDB_DUPSORT and mdb_cursor_put(.., MDB_CURRENT) can put DB in illegal state
by hyc＠symas.com 03 Aug '14

03 Aug '14

Max Bolingbroke wrote: > On 2 August 2014 17:43, Howard Chu <hyc(a)symas.com <mailto:hyc@symas.com>> wrote: > > Max Bolingbroke wrote: > > OK, that does make sense. So MDB_CURRENT is totally useless with > MDB_DUPSORT? > Since you are either pointing at the (k, v) pair already (so the put > will do > nothing) or you aren't (so the put will break the DB). > > > The feature would not exist if it were useless. > > > An alternative explanation for the existence of MDB_CURRENT could be that it > is useful without MDB_DUPSORT (which is certainly is!) but is useless with it > on. I take it from this remark that you do see some use for the combination of > the two options but I guess I must be too stupid to see what that is! > > I suppose that MDB_CURRENT might be useful with MDB_DUPSORT if you could use > it to replace the (key, value) pair being pointed to by the cursor with a > different (key, value) pair, so long as the LMDB user knew that the > replacement would not violate the LMDB invariant that duplicates must be > sorted and distinct. Though this use case "seems" to work with the current > implementation the docs certainly do not promise that this is supported. What defines a "match" depends on the comparison function. If you use a comparator that only uses a portion of the value then of course you can store different values that still match. Prefix comparison is the most obvious example. The recent doc changes are a bit over-zealous here. The only requirement is that the new value would naturally sort into the same position. So yes, you can store a value different than the old value. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7894) LMDB: MDB_DUPSORT and mdb_cursor_put(.., MDB_CURRENT) can put DB in illegal state
by batterseapower＠hotmail.com 03 Aug '14

03 Aug '14

--001a11c2f26695c81a04ffbb03f9 Content-Type: text/plain; charset=UTF-8 On 2 August 2014 17:43, Howard Chu <hyc(a)symas.com> wrote: > Max Bolingbroke wrote: > >> OK, that does make sense. So MDB_CURRENT is totally useless with >> MDB_DUPSORT? >> Since you are either pointing at the (k, v) pair already (so the put will >> do >> nothing) or you aren't (so the put will break the DB). >> > > The feature would not exist if it were useless. An alternative explanation for the existence of MDB_CURRENT could be that it is useful without MDB_DUPSORT (which is certainly is!) but is useless with it on. I take it from this remark that you do see some use for the combination of the two options but I guess I must be too stupid to see what that is! I suppose that MDB_CURRENT might be useful with MDB_DUPSORT if you could use it to replace the (key, value) pair being pointed to by the cursor with a different (key, value) pair, so long as the LMDB user knew that the replacement would not violate the LMDB invariant that duplicates must be sorted and distinct. Though this use case "seems" to work with the current implementation the docs certainly do not promise that this is supported. > Do you think it would be better to error out when the supplied key/data >> doesn't match the data pointed to by the key? That would protect the DB >> from >> this kind of corruption in the presence of bugs in the application. >> > > We don't attempt to protect from user errors; we expect competent > programmers. Is that really true? For example your docs from MDB_WRITEMAP say: """Use a writeable memory map unless MDB_RDONLY is set. This is faster and uses fewer mallocs, but loses protection from application bugs like wild pointer writes and other bad updates into the database.""" So LMDB's default behaviour is actually to take a perf hit in the name of protecting programmers from themselves. My proposal is consistent with that philosophy. > p.s. you may want to add a note to the docs that MDB_RESERVE is not >> allowed >> with MDB_DUPSORT. The combination doesn't seem to make a lot of sense >> > > Indeed it makes no sense at all. And no, we don't attempt to protect from > nonsensical usage. Except that in e.g. mdb_put you explicitly check for nonsensical flags making it through to mdb_cursor_put: if ((flags & (MDB_NOOVERWRITE|MDB_NODUPDATA|MDB_RESERVE|MDB_APPEND|MDB_APPENDDUP)) != flags) return EINVAL; A check for the nonsensical combination of MDB_RESERVE and MDB_DUPSORT would be in that spirit. --001a11c2f26695c81a04ffbb03f9 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 2= August 2014 17:43, Howard Chu <span dir=3D"ltr"><<a href=3D"mailto:hyc@= symas.com" target=3D"_blank">hyc(a)symas.com</a>></span> wrote:<br><blockq= uote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-wi= dth:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-= left:1ex"> <div class=3D"">Max Bolingbroke wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> OK, that does make sense. So MDB_CURRENT is totally useless with MDB_DUPSOR= T?<br> Since you are either pointing at the (k, v) pair already (so the put will d= o<br> nothing) or you aren't (so the put will break the DB).<br> </blockquote> <br></div> The feature would not exist if it were useless.</blockquote><div><br></div>= <div>An alternative explanation for the existence of MDB_CURRENT could be t= hat it is useful without MDB_DUPSORT (which is certainly is!) but is useles= s with it on. I take it from this remark that you do see some use for the c= ombination of the two options but I guess I must be too stupid to see what = that is!</div> <div><br>I suppose that MDB_CURRENT might be useful with MDB_DUPSORT if you= could use it to replace the (key, value) pair being pointed to by the curs= or with a different (key, value) pair, so long as the LMDB user knew that t= he replacement would not violate the LMDB invariant that duplicates must be= sorted and distinct. Though this use case "seems" to work with t= he current implementation the docs certainly do not promise that this is su= pported.</div> <div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px = 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-l= eft-style:solid;padding-left:1ex"><div class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> Do you think it would be better to error out when the supplied key/data<br> doesn't match the data pointed to by the key? That would protect the DB= from<br> this kind of corruption in the presence of bugs in the application.<br> </blockquote> <br></div> We don't attempt to protect from user errors; we expect competent progr= ammers.</blockquote><div><br></div><div>Is that really true? For example yo= ur docs from MDB_WRITEMAP say:</div><div><br></div><div><div>""&q= uot;Use a writeable memory map unless MDB_RDONLY is set. This is faster</di= v> <div>and uses fewer mallocs, but loses protection from application bugs</di= v><div>like wild pointer writes and other bad updates into the database.&qu= ot;""</div></div><div><br></div><div>So LMDB's default behavi= our is actually to take a perf hit in the name of protecting programmers fr= om themselves. My proposal is consistent with that philosophy.</div> <div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px = 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-l= eft-style:solid;padding-left:1ex"><div class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> p.s. you may want to add a note to the docs that MDB_RESERVE is not allowed= <br> with MDB_DUPSORT. The combination doesn't seem to make a lot of sense<b= r> </blockquote> <br></div> Indeed it makes no sense at all. And no, we don't attempt to protect fr= om nonsensical usage.</blockquote><div><br></div><div>Except that in e.g. m= db_put you explicitly check for nonsensical flags making it through to mdb_= cursor_put:</div> <div><br></div><div><div>=C2=A0 =C2=A0 if ((flags & (MDB_NOOVERWRITE|MD= B_NODUPDATA|MDB_RESERVE|MDB_APPEND|MDB_APPENDDUP)) !=3D flags)</div><div><s= pan class=3D"" style=3D"white-space:pre"> </span>return EINVAL;</div></div= ><div><br></div> <div>A check for the nonsensical combination of MDB_RESERVE and MDB_DUPSORT= would be in that spirit.</div></div></div></div> --001a11c2f26695c81a04ffbb03f9--

1 0

Re: (ITS#7914) LMDB: MDB_DUPSORT and mdb_cursor_put(.., MDB_CURRENT) can put DB in illegal state
by hyc＠symas.com 02 Aug '14

02 Aug '14

Due to an error in the transition from old to new OpenLDAP.org server, your ITS was erroneously assigned #7894 which was already allocated. Just following up here to let you know the correct ITS#. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#7894) LMDB: MDB_DUPSORT and mdb_cursor_put(.., MDB_CURRENT) can put DB in illegal state
by batterseapower＠hotmail.com 01 Aug '14

01 Aug '14

Full_Name: Max Bolingbroke Version: LMDB HEAD OS: OS X URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (149.254.58.67) It is possible to insert a (key, value) pair into an MDB_DUPSORTed LMDB database twice using mdb_cursor_put(..., MDB_CURRENT). If you attempt to insert the same (key, value) pair twice with mdb_put then the second insert has no effect. This seems like a bug in mdb_cursor_put. Furthermore, you can use mdb_cursor_put(..., MDB_CURRENT) to insert duplicates in the opposite of their correctly sorted order. This is certainly a bug. The following C program demonstrates both issues: #define _XOPEN_SOURCE 500 #include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <stdarg.h> #include "lmdb.h" void panic(const char * restrict format, ...) { va_list argp; va_start(argp, format); vprintf%foformat, argp); va_end(argp); exit(1); } void mdb_check(char *context, int rc) { if (rc != 0) { panic("%s: %s", context, mdb_strerror(rc)); } } void mdb_check_notfound(char *context, int rc) { if (rc != MDB_NOTFOUND) { panic("%s: %s", context, mdb_strerror(rc)); } } int BE(int x) { return (x & 0x000000FF) << 24 | (x & 0x0000FF00) << 8 | (x & 0x00FF0000) >> 8 | (x & 0xFF000000) >> 24; } int main(int argc, char **argv) { srand(1337); MDB_e * *env; mdb_check("mdb_env_create", mdb_env_create(&env)); mdb_check("mdb_env_set_maxdbs", mdb_env_set_maxdbs(env, 1)); mdb_check("mdb_env_set_mapsize", mdb_env_set_mapsize(env, 1024 * 1024)); mdb_check("mdb_env_open", mdb_env_open(env, "testdb", MDB_NOSUBDIR, 0644)); MDB_txn *txn; mdb_check("mdb_txn_begin", mdb_txn_begin(env, NULL, 0, &txn)); MDB_dbi dbi; mdb_check("mdb_dbi_open", mdb_dbi_open(txn, "root", MDB_CREATE | MDB_DUPSORT, &dbi)); int key; int value; MDB_val keyBuffer; keyBuffer.mv_size = sizeof(int); keyBuffer.mv_data = &key; MDB_val valueBuffer; valueBuffer.mv_size = sizeof(int); valueBuffer.mv_data = &value; { key = BE(2); value = BE(200); mdb_check("mdb_put", mdb_put(txn, dbi, &keyBuffer, &valueBuffer, 0)); key = BE(2); value = BE(300); mdb_check("mdb_put", mdb_put(txn, dbi, &keyBuffer, &valueBuffer, 0)); // Inserting the same (key, value) pair twice: second insert is ignored key = BE(2); value = BE(300); mdb_check("mdb_put", mdb_put(txn, dbi, &keyBuffer, &valueBuffer, 0)); } { MDB_cursor *cursor; mdb_check("mdb_cursor_open", mdb_cursor_open(txn, dbi, &cursor)); size_t count; mdb_check("mdb_cursor_get", mdb_cursor_get(cursor, &keyBuffer, &valueBuffer, MDB_FIRST))0D0D mdb_check("mdb_cursor_count", mdb_cursor_count(cursor, &count)); if (count != 2) panic("count != 2"); if (0) { // Inserting the same (key, value) pair twice: second insert is not ignored. // While this is different to mdb_put, I can't tell from the docs whether this is a bug or not. keyBuffer.mv_size = sizeof(int); valueBuffer.mv_size = sizeof(int); keyBuffer.mv_data = &key; valueBuffer.mv_data = &value; key = BE(2); value = BE(300); mdb_check("mdb_cursor_put2,2C mdb_cursor_put(cursor, &keyBuffer, &valueBuffer, MDB_CURRENT)); mdb_check("mdb_cursor_count", mdb_cursor_count(cursor, &count)); if (count != 1) panic("count != 1"); } else { // This on the other hand is definitely a bug: after doing this the duplicates are not sorted. keyBuffer.mv_size = sizeof(int); valueBuffer.mv_size = sizeof(int); keyBuffer.mv_data = &key; valueBuffer.mv_data = &value; key = BE(2); value = BE(400); mdb_check("mdb_cursor_put", mdb_cursor_p(c8cursor, &keyBuffer, &valueBuffer, MDB_CURRENT)); mdb_check("mdb_cursor_count", mdb_cursor_count(cursor, &count)); if (count != 2) panic("count != 2"); mdb_check("mdb_cursor_get", mdb_cursor_get(cursor, &keyBuffer, &valueBuffer, MDB_FIRST)); printf("%d\n", BE(*((int*)valueBuffer.mv_data))); mdb_check("mdb_cursor_get", mdb_cursor_get(cursor, &keyBuffer, &valueBuffer, MDB_NEXT)); printf("%d\n", BE(*((int*)valueBuffer.mv_data))); mdb_check_notfound("mdb_cursor_get", mdb_cursor_get(cursor, &keyBuffer, &valueBuffer, MDB_NEXT)); } mdb_cursor_close(cursor); } mdb_check("mdb_txn_commit", mdb_txn_commit(txn)); return 0; }

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs August 2014