--On Friday, January 30, 2009 9:45 AM -0500 Francis Swasey <Frank.Swasey(a)uvm.edu&gt; wrote: > I am running OpenLDAP 2.3.39 (locally built) on Red Hat Enterprise Linux > 4 servers with several replicas. We use delta-syncrepl to keep the > replicas in sync with the master server. > > The issue we have now experienced a few times is that if there are a LOT > of updates in the nightly batch update process, that not all of the > updates make it to the replicas but the contextcsn stays in sync, so we > see strange errors that eventually lead us to see that the replicas are > not current even though they think they are. Yes, this was a known issue, and both the replicas and the master must all be on 2.3.43, and all the replicas reloaded from a fresh dump of the master. Once you've done that, it should take care of the problems you are seeing.

--On Friday, January 30, 2009 9:45 AM -0500 Francis Swasey <Frank.Swasey(a)uvm.edu&gt; wrote: > I am running OpenLDAP 2.3.39 (locally built) on Red Hat Enterprise Linux > 4 servers with several replicas. We use delta-syncrepl to keep the > replicas in sync with the master server. Yes, this was a known issue, and both the replicas and the master must all be on 2.3.43, and all the replicas reloaded from a fresh dump of the master. Once you've done that, it should take care of the problems you are seeing.

> Is there anything that should be logged that would help identify the > failure (I'm currently using loglevel of "stats sync" on the master and > all the replicas) ? Some further digging into this and I see that the changes this morning to at least one of these entries are not present in the accesslog database. No wonder the change didn't make it to the replica's, it didn't even make it into the accesslog on the master (although auditlog sees the change and the dc=uvm,dc=edu database on the master has the change). Any suggestions on where to look in the accesslog overlay to see why these modify operations are not being recorded?

On 2/18/09 12:07 PM, Quanah Gibson-Mount wrote: > --On Wednesday, February 18, 2009 9:55 AM -0500 Francis Swasey > <Frank.Swasey(a)uvm.edu&gt; wrote: > >>> Is there anything that should be logged that would help identify the >>> failure (I'm currently using loglevel of "stats sync" on the master and >>> all the replicas) ? >> >> Some further digging into this and I see that the changes this >> morning to >> at least one of these entries are not present in the accesslog database. >> No wonder the change didn't make it to the replica's, it didn't even >> make >> it into the accesslog on the master (although auditlog sees the change >> and the dc=uvm,dc=edu database on the master has the change). >> >> Any suggestions on where to look in the accesslog overlay to see why >> these modify operations are not being recorded? > > Well, I'd generally check db_stat -c to make sure you didn't run out > of locks/lockers/lock objects in the accesslog DB to start. I assume > you have reasonable logging on the master, so that you can see if any > errors were thrown when the MODs that didn't get written out occurred, > etc. The number of locks, lockers, and lock objects are all still at the default 1000. The maximum numbers are 54, 124, and 40 (respectively). So, I think I'm safe there. As I said, I'm logging stats and sync. The modify that didn't make it into the accesslog happened at 1234942530 (2:35:30 EST this morning). The only "interesting" thing logged around that time was: connection_input: conn=139425 deferring operation: too many executing which happened at 2:34:58, 2:35:01, 2:35:04, 2:35:09, 2:35:11, 2:35:12, 2:35:16, 2:35:21, 2:35:27, 2:35:31, and 2:35:35. (a total of 11 times) conn 139425 was the ldapmodify command which was connected from 2:34:57 until 2:35:38 that was performing the 1321 changes (559 adds, 1 delete, and 761 modifications). Assuming my loglevel is high enough to catch the problem -- that looks like it.

Francis Swasey wrote:

> I assume this is some kind of timing bug. What info can I provide to > help find this? > > I am running 2.3.43 on Red Hat Enterprise Linux v4. At this point, not much until you upgrade to 2.4. 2.3 hasn't been updated in 8 months and is about to be retired.

> You may consider the 2.4 branch stable -- but I'm still seeing way too > many problems being reported to consider any release of 2.4 for > production work. Well, the amount of problems being reported might be higher for 2.4. But this does not say anything about stability of your particular configuration. One of the reasons for more issues being reported for 2.4 probably is that many deployments were already migrated to 2.4. So naturally if less people are using 2.3 there will be less issues reported.

I believe the changes are not being placed in the delta database (cn=accesslog). I'll be sure to check the accesslog database the next time the replicas miss changes and see if those changes are listed there or not.

--On Friday, March 06, 2009 7:30 AM -0500 Francis Swasey <Frank.Swasey(a)uvm.edu&gt; wrote: > This hypothesis has been confirmed with the modifications that didn't > make it to the replicas yesterday morning and this morning. > > Looks like the race conditions in accesslog/syncrepl that are being > worked on in the ITS's against 2.4 were present in the later 2.3 systems > as well. I'm not sure what race conditions you're referring to here. Using accesslog serializes writes, therefore you can't hit the race conditions about updating the CSN that are being worked on in current RE24. I'll note that my 2.3.43 builds are actually a hybrid with some 2.4 code. I don't believe there's anything accesslog specific in it, but there is some connection code reworking that is a part of it (the lightweight dispatcher from 2.4, most notably). I've yet to see this with any of our deployments and some of them are quite large and extremely busy.

--On Tuesday, March 10, 2009 10:49 AM -0400 Francis Swasey

> To be specific, there are changes that make it into the master server, > and the auditlog overlay logs them, but the accesslog overlay does NOT > put them in the accesslog database, so they do not get sent to the > replica servers. > > It seems to be some kind of race condition. I haven't figured out a way > to reproduce the failure yet. Oh, you have auditlog in place too? I don't believe you mentioned that before. I bet it is related to them both being enabled.

--On Tuesday, March 10, 2009 1:08 PM -0400 Francis Swasey <Frank.Swasey(a)uvm.edu&gt; wrote: > On 3/10/09 12:20 PM, Quanah Gibson-Mount wrote: >> --On Tuesday, March 10, 2009 10:49 AM -0400 Francis Swasey > >>> To be specific, there are changes that make it into the master server, >>> and the auditlog overlay logs them, but the accesslog overlay does NOT >>> put them in the accesslog database, so they do not get sent to the >>> replica servers. >>> >>> It seems to be some kind of race condition. I haven't figured out a >>> way >>> to reproduce the failure yet. >> >> Oh, you have auditlog in place too? I don't believe you mentioned that >> before. I bet it is related to them both being enabled. > > It is the first time I've mentioned it in this thread, but I've mentioned > it in previous threads. > > So -- why would having auditlog and accesslog (and syncprov) all used > with a database cause accesslog to miss some of the changes? I don't know. :/ But I do know I've never seen it in very write-intensive environments, and that is a major difference. Do writes to go auditlog before accesslog in your configuration? Maybe there's a bug in auditlog under a high-write load where the changes don't get cleaned up properly. If auditlog is being written first, try reversing the order, so that accesslog gets the changes first (I assume that's possible).

Why not just have another process suck out the contents of accesslog you're interested in auditing and write it to a log file. Having both overlays is kinda redundant.

This morning, my replica's were out of sync again, so I have now restarted

On 3/10/09 2:17 PM, Francis Swasey wrote: > If I remember correctly that the overlays get control in the reverse > order of their listing (ie, the last one listed is the first one > executed), then yes, the auditlog is run first, then accesslog, then > syncprov. > > I'll do some experimenting with the order and see if I can list auditlog > first so it runs last. I had a chance last night to put the re-ordered overlay list into production. Having auditlog listed first, syncprov second, and accesslog listed last. This morning, my replica's were out of sync again, so I have now restarted the master server with the auditlog overlay not used (didn't even load the module). If my replica's go out of sync again, then there is definitely a bug in accesslog -- and since the developers are still fighting with replication bugs in 2.4.15 -- I don't see much relief coming any time soon.