openldap-bugs

openldap-bugs@openldap.org

1 participants
20960 discussions

Re: (ITS#8490) changes not written to accesslog, causing replicas to loop syncing
by quanah＠zimbra.com 01 Sep '16

01 Sep '16

--On Thursday, September 01, 2016 8:05 AM +0000 quanah(a)zimbra.com wrote: > --On Thursday, September 01, 2016 7:52 AM +0000 quanah(a)openldap.org wrote: > >> Full_Name: Quanah Gibson-Mount >> Version: OpenLDAP 2.4.44 >> OS: Linux 2.6 >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (75.111.52.177) >> >> >> In a 2-node MMR setup. Node 1 is getting a lot of write traffic. Both >> node 1 and node 2 have 3 replicas each. At some point, a change is >> received by node 1, which writes the change to its accesslog DB and its >> primary DB. It's 3 replicas are all correctly updated. MMR node 2 >> receives the change, updates its primary DB, but *fails* to write the >> change to the accesslog DB. However, it *does* write the CSN update to >> the accesslog DB successfully. This causes all of its replicas to also >> update their CSN. Then a change comes in triggering a constraint >> violation on the replicas, but fully accepted by their master. > > So the above summary is incorrect. While 3 replicas did go out of > sync... 2 belonged to the primary master (node1), and 1 belonged to the > secondary master (node 2). So really, 4 systems didn't log the change > (MMR node 2, ldap05, ldap07, ldap09). Ok, so that's not correct either. I now have the correct topography: ldap01 has the following replicas: ldap02, ldap05, ldap07, ldap09 ldap02 has the following replicas: ldap01, ldap06, ldap08, ldap10 So the replicas of ldap01 received the change and rejected it. ldap02 just skipped writing the entry to the accesslog, and as a result, none of its replicas ever got the change, and thus they never hit the failure issue of err 19, but they all are now lacking this modification entirely. I would note that every server was loaded today from the same ldap backup, so they were all perfectly in sync. In looking at the LDAP accesslog, what I see is that what should have been a modRDN op was stored in the accesslog as a MOD op (the one I noted before). This seems particularly bizarre, because ldap01 should have rejected this change as well. It appears we may have a problem where the accesslog DB is updated, but then the change got rejected by the unique overlay. --Quanah -- Quanah Gibson-Mount

1 0

Re: (ITS#8490) changes not written to accesslog, causing replicas to loop syncing
by quanah＠zimbra.com 01 Sep '16

01 Sep '16

--On Thursday, September 01, 2016 7:52 AM +0000 quanah(a)openldap.org wrote: > Full_Name: Quanah Gibson-Mount > Version: OpenLDAP 2.4.44 > OS: Linux 2.6 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (75.111.52.177) > > > In a 2-node MMR setup. Node 1 is getting a lot of write traffic. Both > node 1 and node 2 have 3 replicas each. At some point, a change is > received by node 1, which writes the change to its accesslog DB and its > primary DB. It's 3 replicas are all correctly updated. MMR node 2 > receives the change, updates its primary DB, but *fails* to write the > change to the accesslog DB. However, it *does* write the CSN update to > the accesslog DB successfully. This causes all of its replicas to also > update their CSN. Then a change comes in triggering a constraint > violation on the replicas, but fully accepted by their master. So the above summary is incorrect. While 3 replicas did go out of sync... 2 belonged to the primary master (node1), and 1 belonged to the secondary master (node 2). So really, 4 systems didn't log the change (MMR node 2, ldap05, ldap07, ldap09). So I have no explanation at the moment why ldap06, ldap08, and ldap10 were not affected. --Quanah -- Quanah Gibson-Mount

1 0

(ITS#8490) changes not written to accesslog, causing replicas to loop syncing
by quanah＠openldap.org 01 Sep '16

01 Sep '16

Full_Name: Quanah Gibson-Mount Version: OpenLDAP 2.4.44 OS: Linux 2.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (75.111.52.177) In a 2-node MMR setup. Node 1 is getting a lot of write traffic. Both node 1 and node 2 have 3 replicas each. At some point, a change is received by node 1, which writes the change to its accesslog DB and its primary DB. It's 3 replicas are all correctly updated. MMR node 2 receives the change, updates its primary DB, but *fails* to write the change to the accesslog DB. However, it *does* write the CSN update to the accesslog DB successfully. This causes all of its replicas to also update their CSN. Then a change comes in triggering a constraint violation on the replicas, but fully accepted by their master. Node 1 records the following change in its accesslog DB: dn: reqStart=20160901025119.904382Z,cn=accesslog objectClass: auditModify structuralObctctClass: auditModify reqStart: 20160901025119.904382Z reqEnd: 20160901025119.906625Z reqType: modify reqSession: 7780 reqAuthzID: uid=zimbra,cn=admins,cn=zimbra reqDN: uid=renamed-trackp,ou=people,dc=xxxxxxxx,dc=co,dc=za reqResult: 0 reqM%3: zimbraMailWhitelistMaxNumEntries:= 0 reqMod: zimbraPrefMailLocalDeliveryDisabled:= FALSE reqMod: zimbraHideInGal:= TRUE reqMod: userPassword:= {SSHA} reqMod: mail:= renamed-trackp(a)xxxxxxx.co.za reqMod: displayName:= trackp(a)xxxxxxxx.co.za reodod: objectClass:= inetOrgPerson reqMod: objectClass:= zimbraAccount reqMod: objectClass:= amavisAccount reqMod: zimbraMailHost:= store328-dc01.cm.xxxxxxxxxxx reqMod: zimbraPasswordMustChange:= FALSE reqMod: cn:= trackp(a)xxxxxxxxxxxxx.co.za reqMod: zimbraPasswordModifiedTime:= 20160121142933Z reqMod: zimbraMailStatus:= enabled reqMod: zimbraIsDelegatedAdminAccount:= FALSE reqMod: zimbraPrefOutOfOfficeReplyEnabled:= FALSE reqMod: zimbraMailBlacklistMaxNumEntries:= 0 reqMod: zimbraAccountStatus:= locked reqMod: zimbraPrefTimeZoneId:= Africa/Harare reqMod: zimbraId:= 403c7b67-eea0-4315-9b10-a2853c6cf70b reqMod: zimbraMailTransport:= lmtp:store328-dc01.cm.xxxxxxxxxx:7025 reqMod: sn:= trackp(a)xxxxxxxxxxx.co.za reqMod: zimbraCOSId:= 1daeb979-365f-46b7-bb3f-2ad46f4576c1 reqMod: zimbraCreateTimestamp:= 20160121142933Z reqMod: zimbraMailDeliveryAddress:= renamed-trackp(a)xxxxxxx.co.za reqMod: entryCSN:= 20160901025119.904589Z#000000#002#000000 reqMod: modifiersName:= uid=zimbra,cn=admins,cn=zimbra reqMod: modifyTimestamp:= 20160901025119Z reqEntryUUID: 2293af38-5497-1035-8bd6-1d34a55b9433 entryUUID: b46d8728-043a-1036-8ab7-9bf2e127fe3b creatorsName: cn=config createTimestamp: 20160901025119Z entryCSN: 20160901025119.904589Z#000000#002#000000 modifiersName: cn=config modifyTimestamp: 20160901025119Z On the other master, we find in the accesslog two sequential CSN updates: dn: reqStart=20160901025119.904218Z,cn=accesslog objectClass: auditModify structuralObjectClass: auditModify reqStart: 20160901025119.904218Z reqEnd: 20160901025119.905035Z reqType: modify reqSession: 201 reqAuthzID: cn=config reqDN: reqResult: 0 reqMod: contextCSN:= 20160901025119.902062Z#000000#002#000000 reqMod: contextCSN:= 20160901015920.689225Z#000000#003#000000 reqEntryUUID: 78143fa2-7af5-48cc-8d5e-6e15f68dd3b2 entryUUID: b46d48a8-043a-1036-81a5-b3dead5e3175 creatorsName: cn=config createTimestamp: 20160901025119Z entryCSN: 20160901025119.902062Z#000000#002#000000 modifiersName: cn=config modifyTimestamp: 20160901025119Z dn: reqStart=20160901025119.908257Z,cn=accesslog objectClass: auditModify structuralObjectClass: auditModify reqStart: 20160901025119.908257Z reqEnd: 20160901025119.908439Z reqType: modify reqSession: 201 reqAuthzID: cn=config reqDN: reqResult: 0 reqMod: contextCSN:= 20160901025119.904589Z#000000#002#000000 reqMod: contextCSN:= 20160901015920.689225Z#000000#003#000000 reqEntryUUID: 78143fa2-7af5-48cc-8d5e-6e15f68dd3b2 entryUUID: b46dcdaa-043a-1036-81a6-b3dead5e3175 creatorsName: cn=config createTimestamp: 20160901025119Z entryCSN: 20160901025119.904589Z#000000#002#000000 modifiersName: cn=config modifyTimestamp: 20160901025119Z Note the CSN update in the second case is for 20160901025119.904589Z. Note that this was the CSN of the change that in node 1 was a real update. The atribute with a constraint in this case is ZimbraID, which is colliding with the zimbraID of the original entry it was renamed from.

1 0

Re: (ITS#8489) LMDB: C_EOF not cleared on cursor in mdb_cursor_put
by hyc＠symas.com 01 Sep '16

01 Sep '16

juerg.bircher(a)gmail.com wrote: > Full_Name: Juerg Bircher > Version: LMDB (master) > OS: MacOS / Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (178.82.36.179) > > > > The function mdb_cursor_put does not clear possibly set C_EOF flag on cursor. > Therefore an successive mdb_cursor_get call using MDB_NEXT may get an > MDB_NOTFOUND result even thought not having put the new value at the end of the > database. > > The following example should illustrate the issue: Thanks for the report, fixed now in mdb.master -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8489) LMDB: C_EOF not cleared on cursor in mdb_cursor_put
by juerg.bircher＠gmail.com 31 Aug '16

31 Aug '16

Full_Name: Juerg Bircher Version: LMDB (master) OS: MacOS / Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (178.82.36.179) The function mdb_cursor_put does not clear possibly set C_EOF flag on cursor. Therefore an successive mdb_cursor_get call using MDB_NEXT may get an MDB_NOTFOUND result even thought not having put the new value at the end of the database. The following example should illustrate the issue: Assume an database having two committed values "0" and "5". Begin a new transaction and open an cursor for the next steps: step 1 put 1 get next => returns 5 ==> ok step 2 put 6 get next 3E r returns MDB_NOTFOUND as at end of the database (sets C_EOF on cursor) ==> ok step 3 put 2 get next => should return 5 but due to the set C_EOF flag returns MDB_NOTFOUND ==> failure The fix could be to clear C_EOF on any successful mdb_cursor_put calls. The following code can be used to verify the issue. #include "lmdb.h" #include <stdio.h> #include <stdlib.h> #include <unistd.h> #include <string.h> static char *env_name = "/tmp/testdb"; static char *db_name = "tt"22; #define MAX_MAP_SIZ (1024 * 1024) #define E(expr) CHECK((rc = (expr)) == MDB_SUCCESS, #expr) #define RES(err, expr) ((rc = expr) == (err) || (CHECK(!rc, #expr), 0)) #define CHECK(test, msg) ((test) ? (void)0 : ((void)fprintf%2tdtderr, \ "%s:%d: %s: %s\n", __FILE__, __LINE__, msg, mdb_strerror(rc)), abort())) static MDB_env *env; static MDB_dbi dbi; static void createTestValue(int num, MDB_val *key, MDB_val *val) { static char buf[16]; sprintf(buf, "%03d", num); key->mv_data = (void *)buf; key->mv_size = strlen((const char *)buf); val->mv_data = key->mv_data; val->mv_size = key->mv_size; } static void setup() { int rc; MDB_txn *txn; MDB_cursor *cursor; MDB_val key, val; E(mdb_env_create(&env)); E(mdb_env_set_maxreaders(env, 2)); E(mdb_env_set_maxdbs(env, 10)); E(mdb_env_set_mapsize(env, MAX_MAP_SIZ)); E(mdb_env_open(env, env_name, 0, 0664)); E(mdb_txn_begin(env, NULL,%0, &txn)); E(mdb_dbi_open(txn, db_name, MDB_CREATE, &dbi)); E(mdb_cursor_open(txn, dbi, &cursor)); createTestValue(0, &key, &val); E(mdb_cursor_put(cursor, &key, &val, 0)); createTestValue(5, &key, &val); E(mdb_cursor_put(cursor, &key, &val, 0)); mdb_cursor_close(cursor); E(mdb_txn_commit(txn)); } static int step1(MDB_cursor *cursor) { int rc; MDB_val key, val, rkey, rval; createTestValue(1, &key, &val); E(mdb_cursor_put(cursor, &key, &val, MDB_NOOVERWRITE)); E(mdb_cursor_get(cursor, &rkey, &rval, MDB_NEXT)); // ==> returns 5 return 0; } static int step2(MDB_cursor *cursor) { int rc; MDB_val key, val, rkey, rval; createTestValue(6, &key, &val); E(mdb_cursor_put(cursor, &key, &val, MDB_NOOVERWRITE)); rc = mdb_cursor_get(cursor, &rkey, &rval, MDB_NEXT); // ==> return nothing as 6 is the last one if (rc != MDB_NOTFOUND) { fprintf(stderr, "step2 failed\n"); return -1; } return 0; } static int step3(MDB_cursor *cursor) { int rc; MDB_val key, val, rkey, rval; createTestValue(2, &key, &val); E(mdb_cursor_put(cursor, &key, &val, MDB_NOOVERWRITE)); E(mdb_cursor_get(cursor, &rkey, &rval, MDB_NEXT)); // ==> should return 5 return 0; } static void test() { int rc; MDB_txn *txn; MDB_cursor *cursor; E(mdb_txn_begin(env, NULL, 0, &txn)); E(mdb_cursor_open(txn, dbi, &cursor)); step1(cursor); step2(cursor); step3(cursor); mdb_cursor_close(cursor); E(mdb_txn_commit(txn)); } static void cleanup() { mdb_dbi_close(env, dbi); mdb_env_close(env); char name[1024]; sprintf(name% "22%s/data.mdb", db_name); unlink(name); sprintf(name, "%s/lock.mdb", db_name); unlink(name); } int main(int argc,char * argv[]) { setup(); test(); cleanup(); }

1 0

Re: (ITS#7682) LMDB: [PATCH] mdb_env_copy should retry open() if O_DIRECT fails
by h.b.furuseth＠usit.uio.no 31 Aug '16

31 Aug '16

Sorry, looks like I dropped this message. Yes, #ifdef F_NOCACHE / #elif defined O_DIRECT / #endif seems better. Included in branch "mdb/fopen" in my repo at UiO. On 24. des. 2013 18:18, Howard Chu wrote: > h.b.furuseth(a)usit.uio.no wrote: >> This was fixed in LMDB 0.9.9, except: If both O_DIRECT and >> F_NOCACHE are #defined, both are used. Looks a bit strange. >> Maybe just one should be used in that case. (Which?) Also I'm >> still wondering if env_open needs to fail if F_NOCACHE fails. >> >> Anyway, maybe it should be something like this. The "else {}" >> is in case O_DIRECT is an unsupported dummy definiton: > > Isn't that in the wrong order then? What you suggest here will use O_DIRECT > even if it's a dummy. > > I don't see any reason to fail the operation. They just want to make a > copy/backup; even if the optimization of avoiding pollution of the buffer > cache can't be done, they still want the copy to happen. > >> #ifdef O_DIRECT >> if ((rc = fcntl(newfd, F_GETFL)) != -1) >> (void) fcntl(newfd, F_SETFL, rc | O_DIRECT); >> else >> #endif >> { >> #ifdef F_NOCACHE /* __APPLE__ */ >> (void) fcntl(newfd, F_NOCACHE, 1); >> #endif >> } -- Hallvard

1 0

Re: (ITS#8488) OpenSSL 1.1.0
by hyc＠symas.com 30 Aug '16

30 Aug '16

norm.green(a)gemtalksystems.com wrote: > Full_Name: Norm Green > Version: 2.4.43 > OS: Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (50.240.19.49) > > > Please support OpenSSL 1.1.0. Current versions of OpenLDAP will not build with > OpenSSL 1.1.0 due to OpenSSL API changes. > > Already supported in git master. Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8488) OpenSSL 1.1.0
by norm.green＠gemtalksystems.com 30 Aug '16

30 Aug '16

Full_Name: Norm Green Version: 2.4.43 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (50.240.19.49) Please support OpenSSL 1.1.0. Current versions of OpenLDAP will not build with OpenSSL 1.1.0 due to OpenSSL API changes.

1 0

Re: (ITS#8487) Properly handle realloc() failures
by hyc＠symas.com 30 Aug '16

30 Aug '16

cissp(a)bryongloden.com wrote: > Full_Name: Bryon Gloden, CISSP. > Version: > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (161.133.194.55) > > > Greetings. On line numbers 123 and 134 of liblmdb/midl.c > (https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/midl.c) there > are two common realloc() mistakes. Passing one pointer into realloc() and > assigning the result directly into that same pointer variable can cause a memory > leak if the reallocation fails, because the original allocation will still > exist. The correct way to do this is to use a temporary pointer variable. > > Credit: http://stackoverflow.com/a/11548901 > > Found by https://github.com/bryongloden/cppcheck > > This is all false. On line number 123 we are shrinking the memory region - this will never fail. At worst it can only return the original pointer unchanged. On line number 134 we are leaving the original pointer unchanged on failure, it is not leaked. The caller can free it if it wants to. Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#8487) Properly handle realloc() failures
by cissp＠bryongloden.com 30 Aug '16

30 Aug '16

Full_Name: Bryon Gloden, CISSP. Version: OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (161.133.194.55) Greetings. On line numbers 123 and 134 of liblmdb/midl.c (https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/midl.c) there are two common realloc() mistakes. Passing one pointer into realloc() and assigning the result directly into that same pointer variable can cause a memory leak if the reallocation fails, because the original allocation will still exist. The correct way to do this is to use a temporary pointer variable. Credit: http://stackoverflow.com/a/11548901 Found by https://github.com/bryongloden/cppcheck

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs