openldap-bugs

openldap-bugs@openldap.org

1 participants
20966 discussions

Re: (ITS#5668) Invalid entryCSN generated, and slapd will not restart
by hyc＠symas.com 10 Sep '08

10 Sep '08

emmanuel.duru(a)atosorigin.com wrote: > I am running windows 32 bits. > Here is what I get from my debugging session: > After QueryPerformanceCounter(&count ), count.QuadPart is worth > 4243738405535005 > So the overflow comes from count.QuadPart *= 1000000; I see. There's no reason we should be computing against the full value of the count; since we're only looking for the microsecond portion we should have stripped off everything greater. (E.g., count.QuadPart %= cFreq.QuadPart). I'll have to play with this code a bit more, there's probably a better way to isolate the microseconds. PS: I guess that means your Windows machine had an uptime of several days. Quite impressive, for Windows... ;) > > >> -----Message d'origine----- >> De : Howard Chu [mailto:hyc@symas.com] >> Envoyé : mercredi 27 août 2008 13:08 >> À : openldap-its(a)openldap.org; Emmanuel Duru >> Objet : Re: (ITS#5668) Invalid entryCSN generated, and slapd will not >> restart >> >> ando(a)sys-net.it wrote: >>> emmanuel.duru(a)atosorigin.com wrote: >>>> I see that tm->tm_usub is negative, there seems to be overflows between >>>> LARGE_INTEGER and int variables. >> It would take over 4 billion operations in a single timer tick (on the >> order >> of nanoseconds) to make tm_usub overflow. That seems pretty unlikely. >> >>> If the problem disappears by initializing the static variables in >>> lutil_gettime(), then it might be a compiler issue. >> I suppose that's always possible... >> >> The original post shows that the tm_usec field is negative. That could >> happen >> if the offset we computed between the SYSTEMTIME and the >> PerformanceCounter >> was wrong, or if the SYSTEMTIME was adjusted while the process was >> running. >> >> What version of Windows are you running? 32 or 64 bit? Can you singlestep >> through this function with a debugger and verify all of the values? I >> haven't >> run this code on Windows in a long time, would take a bit of effort to >> resurrect my build environment. >> >> -- >> -- Howard Chu >> CTO, Symas Corp. http://www.symas.com >> Director, Highland Sun http://highlandsun.com/hyc/ >> Chief Architect, OpenLDAP http://www.openldap.org/project/ > > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5694)
by ando＠sys-net.it 10 Sep '08

10 Sep '08

Works for me, after Howard's fix. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ----------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Fax: +39 0382 476497 Email: ando(a)sys-net.it -----------------------------------

1 0

Re: (ITS#5694) Usage of -w with slapadd breaks delta-syncrepl
by ando＠sys-net.it 09 Sep '08

09 Sep '08

Quanah Gibson-Mount wrote: > Just to be clear here, I'm slapcating a provider while it is running, > then stopping the provider and the replica, wiping both their dbs, then > using slapadd -w to reload the *provider* and syncrepl to load the > consumer. Thus I don't see whether or not the CSN is in memory at the > time on the *provider* matters much. Right, it shouldn't matter, since either it is equal to what -w would generate, or -w would overwrite it with the correct value anyway. The only case I see is if the last operation was a delete, then the contextCSN on the provider would be greater than what a slapadd -w would generate. > Particularly given that if I stop > the reloaded provider and restart it (i.e., flushing the context CSN), I > still get the same behavior problems on the replica. I.e., the replica > has never been loaded via slapadd, only the provider, and the provider > has been stopped started since the slapadd. In fact, I could easily reproduce the behavior you pointed out. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ----------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Fax: +39 0382 476497 Email: ando(a)sys-net.it -----------------------------------

1 0

Re: (ITS#5694) Usage of -w with slapadd breaks delta-syncrepl
by quanah＠zimbra.com 09 Sep '08

09 Sep '08

--On Tuesday, September 09, 2008 8:59 PM +0200 Pierangelo Masarati <ando(a)sys-net.it> wrote: > Quanah Gibson-Mount wrote: >> --On Tuesday, September 09, 2008 8:50 PM +0200 Pierangelo Masarati >> <ando(a)sys-net.it> wrote: >> >>> I thuink the issue is related to the fact that slapadd -w uses >>> bi_tool_entry_modify() and bi_tool_sync(). However, slapo-accesslog(5) >>> does not implement those hooks. As a consequence, any modification to >>> the context entry is not reflected in slapo-accesslog(5). >>> >>> However, I don't see how this should affect replication, since a >>> consistent database, after slapcatt'ing, should generate a ldif with the >>> correct contextCSN. However, if you slapcat before stopping the >>> producer, the resulting ldif will not contain the contextCSN, because >>> after a fresh slapadd without -w it's kept in memory until the producer >>> is stopped. You should try Just to be clear here, I'm slapcating a provider while it is running, then stopping the provider and the replica, wiping both their dbs, then using slapadd -w to reload the *provider* and syncrepl to load the consumer. Thus I don't see whether or not the CSN is in memory at the time on the *provider* matters much. Particularly given that if I stop the reloaded provider and restart it (i.e., flushing the context CSN), I still get the same behavior problems on the replica. I.e., the replica has never been loaded via slapadd, only the provider, and the provider has been stopped started since the slapadd. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5694) Usage of -w with slapadd breaks delta-syncrepl
by quanah＠zimbra.com 09 Sep '08

09 Sep '08

--On Tuesday, September 09, 2008 12:19 PM -0700 Howard Chu <hyc(a)symas.com> wrote: > quanah(a)zimbra.com wrote: >> --On Tuesday, September 09, 2008 8:59 PM +0200 Pierangelo Masarati >> <ando(a)sys-net.it> wrote: >> >> >>>> I'll give what you suggested a try, but this is still a serious bug. >>> There's something I don't clearly get. As far as I understand, you are >>> saying that consumers lose sync when the log database is purged before >>> they have a chance to sync (because they were down). I don't see how >>> they could sync, then. Of course, there should be a means for >>> slapo-syncprov(5) to tell that, and force a refresh. >> >> Because the consumer is supposed to do a full refresh if they find their >> context CSN doesn't match the contextCSN of the master. > > Not quite. More precisely, the provider is supposed to tell the consumer > that the consumer's cookie is out of date if the cookie is older than > anything in the provider, and the consumer asked for the reload Hint. > Otherwise the provider just sees the consumer is stale and returns a full > dump implicitly. Well, the consumer's definitely not acting on a full dump in either case. :/ In my latest test, after the purge, I have: # accesslog dn: cn=accesslog contextCSN: 20080909190440.484524Z#000000#000#000000 dn: contextCSN: 20080909190440.484524Z#000000#000#000000 on the master and contextCSN: 20080909183254.258851Z#000000#000#000000 on the replica prior to starting it. SO the replica is clearly behind. The new entry on the master is: dn: uid=qtest7,cn=admins,cn=zimbra entryCSN: 20080909190440.484524Z#000000#000#000000 which matches the CSN on both the main DB and accesslog DB. now, staring the replica, we find: contextCSN: 20080909190440.484524Z#000000#000#000000 So, it believes it has sync'd now, but uid=qtest7 doesn't exist! --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5694) Usage of -w with slapadd breaks delta-syncrepl
by hyc＠symas.com 09 Sep '08

09 Sep '08

quanah(a)zimbra.com wrote: > --On Tuesday, September 09, 2008 8:59 PM +0200 Pierangelo Masarati > <ando(a)sys-net.it> wrote: > > >>> I'll give what you suggested a try, but this is still a serious bug. >> There's something I don't clearly get. As far as I understand, you are >> saying that consumers lose sync when the log database is purged before >> they have a chance to sync (because they were down). I don't see how >> they could sync, then. Of course, there should be a means for >> slapo-syncprov(5) to tell that, and force a refresh. > > Because the consumer is supposed to do a full refresh if they find their > context CSN doesn't match the contextCSN of the master. Not quite. More precisely, the provider is supposed to tell the consumer that the consumer's cookie is out of date if the cookie is older than anything in the provider, and the consumer asked for the reload Hint. Otherwise the provider just sees the consumer is stale and returns a full dump implicitly. In plain syncrepl the consumer never asks for the reload hint. In delta-sync the consumer does ask for a reload hint, because that tells it to do a sync from the main DB instead of the log DB. It sounds like if there's a bug here, it's caused by the logdb's provider not sending a reload hint when it should. > See ITS#5376, ITS#5378, ITS#5465, ITS#5493, ITS#5282 -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#5694) Usage of -w with slapadd breaks delta-syncrepl
by quanah＠zimbra.com 09 Sep '08

09 Sep '08

--On Tuesday, September 09, 2008 8:59 PM +0200 Pierangelo Masarati <ando(a)sys-net.it> wrote: >> I'll give what you suggested a try, but this is still a serious bug. > > There's something I don't clearly get. As far as I understand, you are > saying that consumers lose sync when the log database is purged before > they have a chance to sync (because they were down). I don't see how > they could sync, then. Of course, there should be a means for > slapo-syncprov(5) to tell that, and force a refresh. Because the consumer is supposed to do a full refresh if they find their context CSN doesn't match the contextCSN of the master. See ITS#5376, ITS#5378, ITS#5465, ITS#5493, ITS#5282 --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5694) Usage of -w with slapadd breaks delta-syncrepl
by ando＠sys-net.it 09 Sep '08

09 Sep '08

Quanah Gibson-Mount wrote: > --On Tuesday, September 09, 2008 8:50 PM +0200 Pierangelo Masarati > <ando(a)sys-net.it> wrote: > >> I thuink the issue is related to the fact that slapadd -w uses >> bi_tool_entry_modify() and bi_tool_sync(). However, slapo-accesslog(5) >> does not implement those hooks. As a consequence, any modification to >> the context entry is not reflected in slapo-accesslog(5). >> >> However, I don't see how this should affect replication, since a >> consistent database, after slapcatt'ing, should generate a ldif with the >> correct contextCSN. However, if you slapcat before stopping the >> producer, the resulting ldif will not contain the contextCSN, because >> after a fresh slapadd without -w it's kept in memory until the producer >> is stopped. You should try > > It should be possible to restore a working provider/slave relationship > from a hot slapcat taken at any point in time. Well, actually slapadd -w should suffice. As an alternative, slapcat could support -w as well, for symmetry, so that consistent ldifs can be generated. > I'll give what you suggested a try, but this is still a serious bug. There's something I don't clearly get. As far as I understand, you are saying that consumers lose sync when the log database is purged before they have a chance to sync (because they were down). I don't see how they could sync, then. Of course, there should be a means for slapo-syncprov(5) to tell that, and force a refresh. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ----------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Fax: +39 0382 476497 Email: ando(a)sys-net.it -----------------------------------

1 0

Re: (ITS#5694) Usage of -w with slapadd breaks delta-syncrepl
by quanah＠zimbra.com 09 Sep '08

09 Sep '08

--On Tuesday, September 09, 2008 8:50 PM +0200 Pierangelo Masarati <ando(a)sys-net.it> wrote: > I thuink the issue is related to the fact that slapadd -w uses > bi_tool_entry_modify() and bi_tool_sync(). However, slapo-accesslog(5) > does not implement those hooks. As a consequence, any modification to > the context entry is not reflected in slapo-accesslog(5). > > However, I don't see how this should affect replication, since a > consistent database, after slapcatt'ing, should generate a ldif with the > correct contextCSN. However, if you slapcat before stopping the > producer, the resulting ldif will not contain the contextCSN, because > after a fresh slapadd without -w it's kept in memory until the producer > is stopped. You should try It should be possible to restore a working provider/slave relationship from a hot slapcat taken at any point in time. I'll give what you suggested a try, but this is still a serious bug. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#5694) Usage of -w with slapadd breaks delta-syncrepl
by ando＠sys-net.it 09 Sep '08

09 Sep '08

I thuink the issue is related to the fact that slapadd -w uses bi_tool_entry_modify() and bi_tool_sync(). However, slapo-accesslog(5) does not implement those hooks. As a consequence, any modification to the context entry is not reflected in slapo-accesslog(5). However, I don't see how this should affect replication, since a consistent database, after slapcatt'ing, should generate a ldif with the correct contextCSN. However, if you slapcat before stopping the producer, the resulting ldif will not contain the contextCSN, because after a fresh slapadd without -w it's kept in memory until the producer is stopped. You should try (i) stop the master and replica (j) do a slapcat of the master (k) delete the database on the master and replica (l) Use slapadd -w to restore your slapcat of 5 users (m) start the replica, verify the contents are the same (n) stop the replica (o) add a user on the master (p) wait for the accesslog to expire (q) start the replica In the meanwhile, you should check what contextCSN your consumers have after syncing from a producer loaded without -w. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ----------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Fax: +39 0382 476497 Email: ando(a)sys-net.it -----------------------------------

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs