Hi, I have huge LDIF file from openldap 2.3.30, which I try to load in a 2.4.19 mirrormode setup. I've tried different ways to load it. 1) Load the LDIF on server 1 and wait for server 2 to replicate it. - it takes several days and server-2 never seems to get all the way and catch up with server 1. 2) Load the LDIF on both servers and start slapd. - Afterwards not all entries created on server 1 is replicated to server-2. So the first thing to rule out would be to ensure that I've loaded the LDIF correctly: The entryCSN from the 2.3.30 server is in this format: 20071214130312Z#000000#00#000000 The two servers (server1/server2) in the 2.4.19 setup have sid 1, 2 and rid 3,4 (cn=config replication has rid 1,2). I loaded the LDIF like this on both servers: $ slapadd -q -w -l backup-2.3.30.ldif Is that enough to make replication start from a known state? Could someone please exemplify a scenario where the -S option for slapadd is needed?

masarati(a)aero.polimi.it wrote: > - load server 1 using slapadd with option -S (the SID of server 1) and > -w > > - slapcat server 1 > > - slapadd server 2 using the slapcat from server 1 > > this ensures you have consistent entryCSN and contextCSN Ahh... That's of course right. But that will also more than double the time needed to load a backup on a mirrormode setup. It already takes over 2 hours to load the database on server 1. Is there really no way around this load-dump-load procedure?

Two hours? Are you sure your DB_CONFIG file is tuned, tool-threads is set to cpu#, and -q is being used with slapadd?

Must be one heck of a large DB otherwise. :P It takes me two hours to load a 3 million entry DB that ends up being a 12GB db on disk.

Peter Mogensen wrote: > masarati(a)aero.polimi.it wrote: >> - load server 1 using slapadd with option -S (the SID of server 1) and -w >> >> - slapcat server 1 >> >> - slapadd server 2 using the slapcat from server 1 >> >> this ensures you have consistent entryCSN and contextCSN > > Ahh... > That's of course right. > But that will also more than double the time needed to load a backup on > a mirrormode setup. This procedure should only be needed if the LDIF doesn't already contain correct CSNs. If you're loading a backup from a 2.4 slapcat you can just slapadd it on all servers at once.

Pierangelo Masarati wrote: > Howard Chu wrote: >> Peter Mogensen wrote: >>> masarati(a)aero.polimi.it wrote: >>>> - load server 1 using slapadd with option -S (the SID of server 1) >>>> and -w >>>> >>>> - slapcat server 1 >>>> >>>> - slapadd server 2 using the slapcat from server 1 >>>> >>>> this ensures you have consistent entryCSN and contextCSN >>> >>> Ahh... >>> That's of course right. >>> But that will also more than double the time needed to load a backup >>> on a mirrormode setup. >> >> This procedure should only be needed if the LDIF doesn't already contain >> correct CSNs. If you're loading a backup from a 2.4 slapcat you can just >> slapadd it on all servers at once. > > My understanding is that he was loading LDIF from 2.3, which has a > different format for CSN. So the first run with -S and -w was > intended to initialize CSN info in 2.4 format with the SID of the > first master. This would probably require to remove entryCSN values > from the original LDIF. I've done as above. "slapadd -S 1 -q -w" on server-1 (Server-ID 1) Then slapcat on server-1 I would have expected the entryCSN values in the output to now be with SID 1, but they look like this: entryCSN: 20071214130312.000000Z#000000#000#000000 Then contextCSN is also with SID 0: contextCSN: 20090929120520.000000Z#000000#000#000000 Though that surpised me I impirted the LDIF to server-2 (SID 2) and replication seems to work. However, after the first change from server-1 has been replicated to server-2, there are now 2 contextCSN's on server-2: entryCSN: 20071214130312.000000Z#000000#000#000000 contextCSN: 20090929120520.000000Z#000000#000#000000 contextCSN: 20091112161735.074445Z#000000#001#000000 ... the last one with SID 1. This is not the behaviour I would have expected.

Pierangelo Masarati wrote: > Peter Mogensen wrote: >> Pierangelo Masarati wrote: >>> Howard Chu wrote: >>>> Peter Mogensen wrote: >>>>> masarati(a)aero.polimi.it wrote: >>>>>> - load server 1 using slapadd with option -S (the SID of server >>>>>> 1) and -w >>>>>> >>>>>> - slapcat server 1 >>>>>> >>>>>> - slapadd server 2 using the slapcat from server 1 >>>>>> >>>>>> this ensures you have consistent entryCSN and contextCSN >>>>> >>>>> Ahh... >>>>> That's of course right. >>>>> But that will also more than double the time needed to load a >>>>> backup on a mirrormode setup. >>>> >>>> This procedure should only be needed if the LDIF doesn't already >>>> contain >>>> correct CSNs. If you're loading a backup from a 2.4 slapcat you can >>>> just >>>> slapadd it on all servers at once. >>> >>> My understanding is that he was loading LDIF from 2.3, which has a >>> different format for CSN. So the first run with -S and -w was >>> intended to initialize CSN info in 2.4 format with the SID of the >>> first master. This would probably require to remove entryCSN values >>> from the original LDIF. >> >> I've done as above. >> "slapadd -S 1 -q -w" on server-1 (Server-ID 1) >> >> Then slapcat on server-1 >> >> I would have expected the entryCSN values in the output to now be >> with SID 1, but they look like this: >> entryCSN: 20071214130312.000000Z#000000#000#000000 >> >> Then contextCSN is also with SID 0: >> contextCSN: 20090929120520.000000Z#000000#000#000000 >> >> Though that surpised me I impirted the LDIF to server-2 (SID 2) and >> replication seems to work. >> However, after the first change from server-1 has been replicated to >> server-2, there are now 2 contextCSN's on server-2: >> entryCSN: 20071214130312.000000Z#000000#000#000000 >> contextCSN: 20090929120520.000000Z#000000#000#000000 >> contextCSN: 20091112161735.074445Z#000000#001#000000 >> >> ... the last one with SID 1. >> >> This is not the behaviour I would have expected. > > What happened is that slapadd simply converted the existing 2.3 CSNs > to 2.4 format while keeping their value. The fact the first > contextCSN (generated by slapadd) has SID 000 is expected, since the > contextCSN is computed as the largest entryCSN (one for each SID that > appears in the database's entryCSN). > > At this point, you should: > > - take the LDIF slapcat from server-1 > - manually modify all SIDs to 001 (e.g. using sed or whatever) > - reload the LDIF into server-1 > > Now you have a properly initialized server-1. Ahh... -S is only for generated CSN's.

But if I'm loading the same data into both servers in a mirromode setup, then I shouldn't really have any use for the old CSN values, should I? So instead of sed/perl chaing the CSN's I could just remove them from the LDIF and let sladadd generate new ones?

It strikes my that there should be an FAQ about this (loading a backup from one server setup into another with different SID/RIDs).

Peter Mogensen wrote: > masarati(a)aero.polimi.it wrote: >> - load server 1 using slapadd with option -S (the SID of server 1) and -w >> >> - slapcat server 1 >> >> - slapadd server 2 using the slapcat from server 1 >> >> this ensures you have consistent entryCSN and contextCSN > > Ahh... > That's of course right. > But that will also more than double the time needed to load a backup on > a mirrormode setup. This procedure should only be needed if the LDIF doesn't already contain correct CSNs. If you're loading a backup from a 2.4 slapcat you can just slapadd it on all servers at once.

On Thu, Nov 12, 2009 at 4:26 AM, <masarati(a)aero.polimi.it <mailto:masarati@aero.polimi.it>> wrote: > Could someone please exemplify a scenario where the -S option for > slapadd is needed? - load server 1 using slapadd with option -S (the SID of server 1) and -w - slapcat server 1 - slapadd server 2 using the slapcat from server 1 this ensures you have consistent entryCSN and contextCSN So can i say that in mirrormode, there is never a need to provide -S <sid> unless you are initializing the first server (sid 1) from a ldif without valid CSN's etc, eg. by providing the "-S 1 -w" options to slapadd ?

If this is true, then would i also be correct is saying that -S <sid> only sets a parameter to enable the -w option to set the SID correctly in CSN's etc? and that the value provided by -S has no effect if -w is not provided ?

I imagine therefore the same thing would apply when initializing a set of multimaster servers, where the <sid> provided by -S just happened to be the first one you are loading. The others 2-N nodes would be initialized from a slapcat of the first server you loaded (using the contextCSN's etc., slapadd "-S <sid> -w" created), still using slapadd but without needing to provide either the -S <sid> option or the -w option.

Just trying to clarify my understanding before contributing a paragraph of doc to explain this.

On Sat, Nov 21, 2009 at 7:10 PM, Howard Chu <hyc(a)symas.com <mailto:hyc@symas.com>> wrote: > If this is true, then would i also be correct is saying that -S <sid> > only sets a parameter to enable the -w option to set the SID correctly > in CSN's etc? and that the value provided by -S has no effect if -w is > not provided ? No. hmmm.. csnsid in slapadd.c defaults to 0, unless the -S option is given, but let us assume it is given -S 1, so csnsid=1 if the -w option is specified, then update_ctxcsn is >0 it looks like slapadd will create the berval csn (in memory) which takes the value of csnsid,

but it will only write related data if (update_ctxcsn)

but for multimaster the csnsid is used (thus must be provided) to find it's own CSN, but unless update_ctxcsn is >0, it wont change any state?

so maybe -S <sid> is required (with or without -w) to initialize multimaster to find the correct CSN (as other master's CSN's could be present? - i dont quite get why though if we are re-loading the database), but for mirrormode -S <sid> is not required (if there is no -w), but only because any other <sid>'s are likely to be shadow contexts in the initial case of a data load ?

On Sun, Nov 22, 2009 at 5:42 AM, Howard Chu <hyc(a)symas.com&gt; wrote:> but it will only write related data if (update_ctxcsn) > > The manpage clearly states that -S sets the SID used in generated > entryCSNs. > > It also clearly states that the -w option writes the contextCSN based on > the > greatest entryCSN in the database. > > Period, end of story. > Indeed. So the consequence is that the -S <sid> option is used to initialize the entryCSN's, regardless of if the -w option being provided. If the -w option is provided, then the entryCSN's are searched for the maximal entryCSN, which becomes the contextCSN (either recently generated, or provided as part of the LDIF being loaded) If the -S is not provided, it defaults to 0.