--1251447850-1386810138-1349783312=:99375 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable
Hi,=0ATo try and overcome this issue, we tried two fixes:=0A=0A1. Every tim= e a user was deleted from a group, we force-updated the user=A0=0Aobject ma= nually to make sure its entryCSN got updated and it got=A0=0Areplicated pro= perly. This is an expensive operation and did not scale=A0=0Awell for big g= roup sizes (10-20k), and did not work out.=0A=0A2. We then tried to do the = same thing in OpenLDAP. We noticed in the=A0=0Amemberof.c commits that ther= e were a couple of patches to force the=A0=0AentryCSN of the user object to= get updated. (http://tinyurl.com/8k4qrdj=A0=0Aand http://tinyurl.com/9akqg= fl)These have since been reverted because of=A0=0Aaccess log and some repli= cation issues, but for us, speed was a higher=A0=0Apriority. I reapplied th= ese patches back to the code. This solved the=A0=0Amember-of replication is= sue, but we noticed that occasionally under a=A0=0Aheavy load, there was a = sudden surge in OpenLDAP's memory usage going up=A0=0Ato whatever memory wa= s available and finally crashing.=0A=0AWe have gone back to option (1) thou= gh (2) would be the preferred option.=0A=0AAny help on figuring out why (2)= caused the memory bloat would be really=A0=0Agreat. I can provide more det= ails/memory traces if needed.=0A=0AWe will be glad to contribute any fixes = once we are able to nail down=A0=0Athe issue.=0A=0AThanks,=0AArunkumar=0A= =0A________________________________=0A From: Howard Chu hyc@symas.com=0AT= o: arun s arunkumar_1123@yahoo.com =0ACc: "openldap-its@openldap.org" <op= enldap-its@openldap.org> =0ASent: Monday, 1 October 2012 6:59 PM=0ASubject:= Re: (ITS#7400) Memberof and Syncrepl incompatibility=0A =0Aarun s wrote:= =0A> Hi,=0A> Yes, I am able to reproduce the issue with 2.4.32=0A> =0A> Mak= ing sense of the logs for the exact reproduction is hard since it needs a= =0A> lot of operations in a short time. But this will probably help:=0A> = =0A> 1. At the start of the test, the group temp_group existed.=0A> =0A> 2.= I created a user temp_user and added temp_user to temp_group.=0A> =0A> 3. = During replication, the group was replicated first and I got an error 32=0A=
(NO_SUCH_OBJECT) when it tried to modify the memberOf of the temp_user ob=
ject=0A> (This does not exist in the readslave yet).=0A> =0A> 4. The temp_u= ser object was replicated next, and as you see, querying it does=0A> show a= memberOf attribute, proving that this field was replicated. (Note that=0A>= I have run OpenLDAP with debug as -1 and verified that the replicated data= has=0A> the memberOf field in it. I can provide this too if needed.)=0A=0A= I see. The current code drops the memberOf attribute if it was not explicit= ly requested in the search. However, by default the consumer requests "+" w= hich means "all operational attributes" and so slapd considers memberOf to = have been requested. We need to reconsider this aspect of the design.=0A> = =0A> 5. The more serious problem occurs when the sequence is reversed and t= he group=0A> has been deleted as a last operation - The user is replicated = first, but since=0A> the group is deleted, it is never replicated and a sta= le memberOf entry stays=0A> with the user.=0A=0A--=A0 -- Howard Chu=0A=A0 = CTO, Symas Corp.=A0 =A0 =A0 =A0 =A0 http://www.symas.com=0A=A0 Director, Hi= ghland Sun=A0 =A0 http://highlandsun.com/hyc/=0A=A0 Chief Architect, OpenLD= AP=A0 http://www.openldap.org/project/ --1251447850-1386810138-1349783312=:99375 Content-Type: text/html; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable
<html><body><div style=3D"color:#000; background-color:#fff; font-family:ti= mes new roman, new york, times, serif;font-size:12pt"><div style=3D"font-fa= mily: 'times new roman', 'new york', times, serif; font-size: 12pt; "><span=
=0A=0A=0A=0A=0A=0A=0A=0A<div class=3D"p1">Hi,</div></span></div><div><div>=
To try and overcome this issue, we tried two fixes:</div><div><br></div><di= v>1. Every time a user was deleted from a group, we force-updated the user&= nbsp;</div><div>object manually to make sure its entryCSN got updated and i= t got </div><div>replicated properly. This is an expensive operation a= nd did not scale </div><div>well for big group sizes (10-20k), and did= not work out.</div><div><br></div><div>2. We then tried to do the same thi= ng in OpenLDAP. We noticed in the </div><div>memberof.c commits that t= here were a couple of patches to force the </div><div>entryCSN of the = user object to get updated. (http://tinyurl.com/8k4qrdj%C2%A0;</div><div>and= http://tinyurl.com/9akqgfl)These have since been reverted because of = </div><div>access log and some replication issues, but for us, speed was a = higher </div><div>priority. I reapplied these patches back to the code= . This solved the </div><div>member-of replication issue, but we noticed that occas= ionally under a </div><div>heavy load, there was a sudden surge in Ope= nLDAP's memory usage going up </div><div>to whatever memory was availa= ble and finally crashing.</div><div><br></div><div>We have gone back to opt= ion (1) though (2) would be the preferred option.</div><div><br></div><div>= Any help on figuring out why (2) caused the memory bloat would be really&nb= sp;</div><div>great. I can provide more details/memory traces if needed.</d= iv><div><br></div><div>We will be glad to contribute any fixes once we are = able to nail down </div><div>the issue.</div><div><br></div><div>Thank= s,</div><div>Arunkumar</div></div> <div style=3D"font-family: 'times new r= oman', 'new york', times, serif; font-size: 12pt; "> <div style=3D"font-fam= ily: 'times new roman', 'new york', times, serif; font-size: 12pt; "> <div = dir=3D"ltr"> <font size=3D"2" face=3D"Arial"> <hr size=3D"1"> <b><span style=3D"font-weight:bold;">From:</span></b> Howard Chu <hyc@symas.com&= gt;<br> <b><span style=3D"font-weight: bold;">To:</span></b> arun s <aru= nkumar_1123@yahoo.com> <br><b><span style=3D"font-weight: bold;">Cc:</sp= an></b> "openldap-its@openldap.org" <openldap-its@openldap.org> <br> = <b><span style=3D"font-weight: bold;">Sent:</span></b> Monday, 1 October 20= 12 6:59 PM<br> <b><span style=3D"font-weight: bold;">Subject:</span></b> Re= : (ITS#7400) Memberof and Syncrepl incompatibility<br> </font> </div> <br>a= run s wrote:<br>> Hi,<br>> Yes, I am able to reproduce the issue with= 2.4.32<br>> <br>> Making sense of the logs for the exact reproductio= n is hard since it needs a<br>> lot of operations in a short time. But t= his will probably help:<br>> <br>> 1. At the start of the test, the g= roup temp_group existed.<br>> <br>> 2. I created a user temp_user and= added temp_user to temp_group.<br>> <br>> 3. During replication, the group was replicated first and I got an error 32<br>> (NO_SUCH_OBJECT) = when it tried to modify the memberOf of the temp_user object<br>> (This = does not exist in the readslave yet).<br>> <br>> 4. The temp_user obj= ect was replicated next, and as you see, querying it does<br>> show a me= mberOf attribute, proving that this field was replicated. (Note that<br>>= ; I have run OpenLDAP with debug as -1 and verified that the replicated dat= a has<br>> the memberOf field in it. I can provide this too if needed.)<= br><br>I see. The current code drops the memberOf attribute if it was not e= xplicitly requested in the search. However, by default the consumer request= s "+" which means "all operational attributes" and so slapd considers membe= rOf to have been requested. We need to reconsider this aspect of the design= .<br>> <br>> 5. The more serious problem occurs when the sequence is = reversed and the group<br>> has been deleted as a last operation - The user is replicated first, but since<br>> the group is deleted, it= is never replicated and a stale memberOf entry stays<br>> with the user= .<br><br>-- -- Howard Chu<br> CTO, Symas Corp. &n= bsp; <a href=3D"http://www.symas.com/" target=3D"_blank">htt= p://www.symas.com</a><br> Director, Highland Sun <a hre= f=3D"http://highlandsun.com/hyc/" target=3D"_blank">http://highlandsun.com/= hyc/</a><br> Chief Architect, OpenLDAP <a href=3D"http://www.op= enldap.org/project/" target=3D"_blank">http://www.openldap.org/project/</a>= <br><br><br> </div> </div> </div></body></html> --1251447850-1386810138-1349783312=:99375--
-
arunkumar_1123＠yahoo.com