--On Tuesday, October 15, 2013 11:47 PM +0200 Bruno Marcon
<marcon.bruno(a)free.fr> wrote:
> I have found the origin of the memory increase. The problem occur with a
> huge group and the RefreshAndPersist mode replication without using DELTA
> Sync. So each slaves pull from the master the entire entry.
>
> The LDAP contains about 100.000 users. All of them are member of a group
> name "XXX". The group XXX is a group with 100.000 attributes "member",
> one for each user.
>
> So when I assign 20 users per seconds to the XXX group, the master LDAP
> push (or the slaves pull ?) the group XXX 20 times per seconds for each
> slaves (there is 5 slaves).
>
> 5x20x100.000 = 10.000.000 attributes per seconds... The master LDAP handle
> this rate for several seconds, perhaps one minute, then suddenly increase
> in memory and is killed by the OS at 2Go of memory.
>
> To avoid this, i have set the DELTA sync replication to only send the
> modification : for each assignition of a user in the group, only the
> attribute member for the user is replicate.
>
> A question is why the LDAP is increasing in memory ? it should be slowed
> rather than store something in memory after having reach a limit ?
>
> Our main problem now is : we have two masters in mirror mode, and 4
> replicas. For the replicas, I can use the DELTA sync mode. For the
> recovery master (for fail over), I can't use DELTA sync because this mode
> is not supported in Mirror mode.
> So we have configure the masters replication in RefreshOnly mode with a
> polling period of 5 seconds, to avoid this memory increase in
> RefreshAndPersist mode without DELTA sync.
> Doing this, the recovery master have 5s of delay before replicate the
> primary master, so if the primary master crash for any reason, the
> recovery master may have missed up to 5s of modifications (users creation
> for example). BUT replicas are up to date with the old primary master are
> more up to date than the recovery master and may have more users, because
> there replication is immediate (RefreshAndPersist mode).
>
> If we can restart the "old" primary master, the recovery master get the
> missing users thanks to replication. If we can't restart the primary
> master because it is KO, we have a recovery master with less users than
> slave, so we have some onconsistency between replicas and the recovery
> master.
>
> Do you have any solution to this problem ? It occures with big LDAP
> database and hight rate of modifications on the master because we work on
> a huge project with thousands of users (and perhaps in the future
> millions of users).
>
> Regards,
> Bruno Marcon.
>
>
> -----Message d'origine-----
> De : MARCON Bruno [mailto:Bruno.MARCON@thalesgroup.com]
> Envoyé : mercredi 9 octobre 2013 14:04
> À : Bruno Marcon
> Objet : RE: (ITS#7717) Sudden memory increase leading to Master LDAP
> crash
>
>
>
> Bruno Marcon
> ThereSIS Innovation lab, ICT Security Unit
> - IT Security Expert
> - Software Architect
>
> Thales Research & Technology
> Campus Polytechnique
> 1, avenue Augustin Fresnel
> 91767 Palaiseau cedex
> France
> Office: +33 (0)1 69 41 60 96
> Fax: +33 (0)1 69 41 55 63
>
>
> -----Message d'origine-----
> De : Bruno Marcon [mailto:marcon.bruno@free.fr] Envoyé : mardi 8
> octobre 2013 18:05 À : MARCON Bruno Objet : TR: (ITS#7717) Sudden
> memory increase leading to Master LDAP crash
>
>
>
> -----Message d'origine-----
> De : Quanah Gibson-Mount [mailto:quanah@zimbra.com]
> Envoyé : lundi 7 octobre 2013 16:56
> À : marcon.bruno(a)free.fr; openldap-its(a)openldap.org
> Objet : Re: (ITS#7717) Sudden memory increase leading to Master LDAP
> crash
>
> --On Thursday, October 03, 2013 1:14 PM +0000 marcon.bruno(a)free.fr wrote:
>
>> The problem is very easy to replay : one master and one slave are
>> sufficient. Then run 9 windows with a script adding an entry and
>> modifying an entry.
>
> Please provide full configuration details.
>
> --Quanah
>
>
> --
>
> Quanah Gibson-Mount
> Architect - Server
> Zimbra Software, LLC
> --------------------
> Zimbra :: the leader in open source messaging and collaboration
>
>
--
Quanah Gibson-Mount
Architect - Server
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration