Re: (ITS#8800) MMR: out of date master will ignore history of its own changes
by quanah＠symas.com
--On Tuesday, January 30, 2018 9:04 PM +0000 quanah(a)openldap.org wrote:
> Full_Name: Quanah Gibson-Mount
> Version: 2.4.45
> OS: Linux
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (47.208.148.239)
>
>
> Did the following test:
>
> 4-way MMR setup, database populated from an initial DB that has history
> to it
Per Howard's suggestion, I commented out lines 2390-2396 in syncprov.c,
which allows the master to get fed back its own operations on startup.
On the plus side, the database indeed get all the operations sent back.
There were 1187 entries in the accesslog DBs of all 4 nodes.
On the minus side, while server IDs 2,3, and 4 all agreed on the final
resulting contextCSN, serverID 1 did not. Which then broke the ability for
the nodes to communicate with each other (err=53)
Last entry on serverIDs 1/2/3/4:
dn: reqStart=20180130233019.000017Z,cn=accesslog
objectClass: auditModify
reqStart: 20180130233019.000017Z
reqEnd: 20180130233019.000018Z
reqType: modify
reqSession: 1
reqAuthzID: cn=ldaproot,dc=xxx,dc=yyy
reqDN: dc=xxx,dc=yyy
reqResult: 0
reqMod: contextCSN:= 20180130233019.035885Z#000000#001#000000
reqMod: contextCSN:= 20171130222521.056018Z#000000#002#000000
reqMod: contextCSN:= 20171130222318.939265Z#000000#003#000000
reqMod: contextCSN:= 20171203041258.811473Z#000000#004#000000
reqEntryUUID: 156eb8cc-18e9-1027-80e5-d3f2010890dc
contextCSNs on 2/3/4:
base
contextCSN: 20180130233019.035885Z#000000#001#000000
contextCSN: 20171130222521.056018Z#000000#002#000000
contextCSN: 20171130222318.939265Z#000000#003#000000
contextCSN: 20171203041258.811473Z#000000#004#000000
accesslog
contextCSN: 20180130233019.035885Z#000000#001#000000
contextCSN: 20171130222521.056018Z#000000#002#000000
contextCSN: 20171130222318.939265Z#000000#003#000000
contextCSN: 20171203041258.811473Z#000000#004#000000
contextCSNs on 1:
base
contextCSN: 20180130233019.035885Z#000000#001#000000
contextCSN: 20171130222521.056018Z#000000#002#000000
contextCSN: 20171130222318.939265Z#000000#003#000000
contextCSN: 20171203041258.811473Z#000000#004#000000
accesslog
contextCSN: 20180130233016.137867Z#000000#001#000000
contextCSN: 20171130222521.056018Z#000000#002#000000
contextCSN: 20171130222318.939265Z#000000#003#000000
contextCSN: 20171203041258.811473Z#000000#004#000000
Note that the contextCSN is correct on the database root, but incorrect in
the accesslog entry.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 10 months
(ITS#8800) MMR: out of date master will ignore history of its own changes
by quanah＠openldap.org
Full_Name: Quanah Gibson-Mount
Version: 2.4.45
OS: Linux
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (47.208.148.239)
Did the following test:
4-way MMR setup, database populated from an initial DB that has history to it
Make several thousand MODs to serverID 1 only
Stop serverID 1
wipe its database
reload serverID 1 from the initial DB
Expected result:
serverID 1 REFRESHes from the other servers to sync up its database
Actual result:
serverID 1 updates its contextCSN to the last change op, but does not sync any
actual changes back, leaving it out of sync with the rest of the cluster.
5 years, 10 months
Re: (ITS#8100) Empty accesslog causes issues with delta-syncrepl MMR configurations
by quanah＠symas.com
--On Monday, January 29, 2018 10:23 AM -0800 Quanah Gibson-Mount
<quanah(a)symas.com> wrote:
> I'll continue testing for the other half of the fix (Deleting all but the
> most recent entry from the accesslog during purge)
This part appears to work as desired. I set the purge interval to 10
minutes, checking every 5 minutes. Made changes.
All entries but the most recent one were removed after 15 minutes went by.
Made more changes, did the same wait period, and again, all entries but the
most recent were removed during the next cleanup interval.
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 10 months
Re: (ITS#8100) Empty accesslog causes issues with delta-syncrepl MMR configurations
by quanah＠symas.com
--On Friday, January 26, 2018 8:23 PM +0000 hyc(a)symas.com wrote:
> A patch which skips deleting the final entry, and creates an initial
> dummy log entry if needed, is available in
> https://github.com/quanah/openldap-scratch/tree/its8100 for testing.
Hi Howard,
When reinstalling a 4-way MMR system from scratch, we still end up in
REFRESH mode. In the database I'm loading, there are 4 contextCSN values,
one per active master:
contextCSN: 20171203010043.825769Z#000000#001#000000
contextCSN: 20171130222521.056018Z#000000#002#000000
contextCSN: 20171130222318.939265Z#000000#003#000000
contextCSN: 20171203041258.811473Z#000000#004#000000
When I start up the first master (serverID 4 in this case), a contextCSN
value is properly written for it to the underlying db:
Jan 29 10:06:06 anvil4 slapd[1949]: slapd starting
Jan 29 10:06:06 anvil4 slapd[1949]: slap_queue_csn: queueing 0x7f54d4104220
20171203041258.811473Z#000000#004#000000
Jan 29 10:06:06 anvil4 slapd[1949]: slap_queue_csn: queueing 0x7f54d4104cc0
20171203041258.811473Z#000000#004#000000
Jan 29 10:06:06 anvil4 slapd[1949]: slap_graduate_commit_csn: removing
0x7f54d4104cc0 20171203041258.811473Z#000000#004#000000
Jan 29 10:06:06 anvil4 slapd[1949]: slap_graduate_commit_csn: removing
0x7f54d4104220 20171203041258.811473Z#000000#004#000000
But when I start the other 3 masters, they do not write an entry for their
CSN, and since there's no CSN value for them on the other masters either,
they all fall back to REFRESH_DELETE:
Jan 29 10:06:26 anvil4 slapd[1949]: do_syncrep2: rid=003
LDAP_RES_INTERMEDIATE - REFRESH_DELETE
Even worse, they do this for every master that comes online.
I think the code needs to add an entry to the accesslog for every
contextCSN value, not just the final contextCSN?
I'll continue testing for the other half of the fix (Deleting all but the
most recent entry from the accesslog during purge)
Thanks!
--Quanah
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 10 months
Re: (ITS#8100) Empty accesslog causes issues with delta-syncrepl MMR configurations
by hyc＠symas.com
quanah(a)symas.com wrote:
> --On Thursday, April 09, 2015 5:42 AM +0000 quanah(a)openldap.org wrote:
>
>> Full_Name: Quanah Gibson-Mount
>> Version: 2.4.39
>> OS: Linux 2.6
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (50.25.188.166)
>>
>>
>> When one has an MMR setup using delta-syncrepl, and the masters get into a
>> situation where one is out of sync, or adding a new MMR node to an
>> existing cluster, things will be broken until the new/reloaded node has a
>> write op that goes to the accesslog DB. In an existing cluster, where a
>> node is being reloaded, it causes all nodes to go into an endless looping
>> fallback sync until that write occurs.
>
> One possible fix for this, would be to refuse to delete the final entry in
> the accesslog during the purge phase. That way, the accesslog would never
> be empty. I'm not sure how difficult this would be to implement, code wise.
A patch which skips deleting the final entry, and creates an initial dummy log
entry if needed, is available in
https://github.com/quanah/openldap-scratch/tree/its8100 for testing.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
5 years, 10 months
Re: (ITS#8791) OpenSSL 1.1.1 compat issue
by quanah＠symas.com
Hi Bradley,
I believe it's waiting on a review from Howard. I also plan on throwing it
into my scratch repo and testing when I get the time, but my primary focus
at the moment is migrating the OpenLDAP project to new infrastructure and a
new bug tracking system. ;)
--Quanah
--On Wednesday, January 24, 2018 10:23 PM +0000 bbaetz(a)google.com wrote:
> --089e082f9ab494ea2405638d1cae
> Content-Type: text/plain; charset="UTF-8"
>
> Is there anything else I need to do in order to get this committed?
>
> Bradley
>
> On Fri, 15 Dec 2017 at 12:08 Bradley Baetz <bbaetz(a)google.com> wrote:
>
>> Done in ftp://ftp.openldap.org/incoming/bradley-baetz-20171215.patch
>>
>>
>> On Fri, 15 Dec 2017 at 04:36 Howard Chu <hyc(a)symas.com> wrote:
>>
>>> bbaetz(a)google.com wrote:
>>> > Full_Name: Bradley Baetz
>>> > Version: 2.4.45
>>> > OS: linux
>>> > URL: ftp://ftp.openldap.org/incoming/bradley-baetz-20171214.patch
>>> > Submission from: (NULL) (2401:fa00:9:11:7ac0:58b5:299c:bebb)
>>>
>>> Thanks for the patch. The initialization of the static tlso_bio_method
>>> is racy. One-time initializations should be done in tlso_init, and the
>>> allocated
>>> memory should be freed in tlso_destroy.
>>>
>>> >
>>> > ITS#8533 added support for the OpenSSL's hiding of the bio_method_st
>>> struct.
>>> >
>>> > However, it did this by re-defining the now-private structure, using
>>> the OpenSSL
>>> > 1.0 version. That will fail when OpenSSL changes their structure,
>>> > which
>>> they
>>> > have already done for v1.1.1 - see
>>> >
>>> https://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=include/internal
>>> /bio.h;hb=e1dd8fa00a1e06d27c8b024dac7657a8d8a9b451#l16
>>> >
>>> > It also fails with BoringSSL, which has v1.0's OPENSSL_VERSION_NUMBER
>>> define,
>>> > but has not yet hidden the struct definition.
>>> >
>>> > The attached file is derived from OpenLDAP Software. All of the
>>> modifications to
>>> > OpenLDAP Software represented in the following patch(es) were
>>> > developed
>>> by
>>> > Google, LLC. Google, LLC has not assigned rights and/or interest in
>>> this work to
>>> > any party. I, Bradley Baetz am authorized by Google, LLC, my employer,
>>> to
>>> > release this work under the following terms.
>>> >
>>> > The attached modifications to OpenLDAP Software are subject to the
>>> following
>>> > notice:
>>> > Copyright 2017 Google, LLC.
>>> > Redistribution and use in source and binary forms, with or without
>>> modification,
>>> > are permitted only as authorized by the OpenLDAP Public License.
>>> >
>>> >
>>>
>>>
>>> --
>>> -- Howard Chu
>>> CTO, Symas Corp. http://www.symas.com
>>> Director, Highland Sun http://highlandsun.com/hyc/
>>> Chief Architect, OpenLDAP http://www.openldap.org/project/
>>>
>>
>
> --089e082f9ab494ea2405638d1cae
> Content-Type: text/html; charset="UTF-8"
> Content-Transfer-Encoding: quoted-printable
>
> <div dir=3D"ltr">Is there anything else I need to do in order to get this
> c= ommitted?<div><br></div><div>Bradley</div></div><br><div
> class=3D"gmail_quo= te"><div dir=3D"ltr">On Fri, 15 Dec 2017 at 12:08
> Bradley Baetz <<a href=
> =3D"mailto:bbaetz@google.com">bbaetz(a)google.com</a>>
> wrote:<br></div><bl= ockquote class=3D"gmail_quote" style=3D"margin:0 0 0
> .8ex;border-left:1px #= ccc solid;padding-left:1ex"><div
> dir=3D"ltr"><span style=3D"font-size:small= ">Done in=C2=A0</span><a
> href=3D"ftp://ftp.openldap.org/incoming/bradley-ba= etz-20171215.patch"
> style=3D"font-size:small" target=3D"_blank">ftp://ftp.o=
> penldap.org/incoming/bradley-baetz-20171215.patch</a><br><br
> class=3D"m_906=
> 2438285945864329inbox-inbox-Apple-interchange-newline"></div><br><div
> class= =3D"gmail_quote"><div dir=3D"ltr">On Fri, 15 Dec 2017 at 04:36
> Howard Chu &= lt;<a href=3D"mailto:hyc@symas.com"
> target=3D"_blank">hyc(a)symas.com</a>>= wrote:<br></div><blockquote
> class=3D"gmail_quote" style=3D"margin:0 0 0 .8= ex;border-left:1px #ccc
> solid;padding-left:1ex"><a href=3D"mailto:bbaetz@go= ogle.com"
> target=3D"_blank">bbaetz(a)google.com</a> wrote:<br>
> > Full_Name: Bradley Baetz<br>
> > Version: 2.4.45<br>
> > OS: linux<br>
> > URL: <a
> href=3D"ftp://ftp.openldap.org/incoming/bradley-baetz-20171214= .patch"
> rel=3D"noreferrer" target=3D"_blank">ftp://ftp.openldap.org/incomin=
> g/bradley-baetz-20171214.patch</a><br>
> > Submission from: (NULL) (2401:fa00:9:11:7ac0:58b5:299c:bebb)<br>
> <br>
> Thanks for the patch. The initialization of the static tlso_bio_method
> is<b= r>
> racy. One-time initializations should be done in tlso_init, and the
> allocat= ed<br>
> memory should be freed in tlso_destroy.<br>
> <br>
> ><br>
> > ITS#8533 added support for the OpenSSL's hiding of the
> bio_method_= st struct.<br>
> ><br>
> > However, it did this by re-defining the now-private structure, using
> t= he OpenSSL<br>
> > 1.0 version. That will fail when OpenSSL changes their structure,
> whic= h they<br>
> > have already done for v1.1.1 - see<br>
> > <a
> href=3D"https://git.openssl.org/gitweb/?p=3Dopenssl.git;a=3Dblob;f=
> =3Dinclude/internal/bio.h;hb=3De1dd8fa00a1e06d27c8b024dac7657a8d8a9b451#l
> 16= " rel=3D"noreferrer"
> target=3D"_blank">https://git.openssl.org/gitweb/?p=3D=
> openssl.git;a=3Dblob;f=3Dinclude/internal/bio.h;hb=3De1dd8fa00a1e06d27c8b
> 02= 4dac7657a8d8a9b451#l16</a><br>
> ><br>
> > It also fails with BoringSSL, which has v1.0's
> OPENSSL_VERSION_NUM= BER define,<br>
> > but has not yet hidden the struct definition.<br>
> ><br>
> > The attached file is derived from OpenLDAP Software. All of the
> modifi= cations to<br>
> > OpenLDAP Software represented in the following patch(es) were
> develope= d by<br>
> > Google, LLC. Google, LLC has not assigned rights and/or interest in
> th= is work to<br>
> > any party. I, Bradley Baetz am authorized by Google, LLC, my
> employer,= to<br>
> > release this work under the following terms.<br>
> ><br>
> > The attached modifications to OpenLDAP Software are subject to the
> fol= lowing<br>
> > notice:<br>
> > Copyright 2017 Google, LLC.<br>
> > Redistribution and use in source and binary forms, with or without
> mod= ification,<br>
> > are permitted only as authorized by the OpenLDAP Public License.<br>
> ><br>
> ><br>
> <br>
> <br>
> --<br>
> =C2=A0 =C2=A0-- Howard Chu<br>
> =C2=A0 =C2=A0CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a
> hr= ef=3D"http://www.symas.com" rel=3D"noreferrer"
> target=3D"_blank">http://www= .symas.com</a><br>
> =C2=A0 =C2=A0Director, Highland Sun=C2=A0 =C2=A0 =C2=A0<a
> href=3D"http://hi= ghlandsun.com/hyc/" rel=3D"noreferrer"
> target=3D"_blank">http://highlandsun= .com/hyc/</a><br>
> =C2=A0 =C2=A0Chief Architect, OpenLDAP=C2=A0 <a
> href=3D"http://www.openldap= .org/project/" rel=3D"noreferrer"
> target=3D"_blank">http://www.openldap.org= /project/</a><br>
> </blockquote></div></blockquote></div>
>
> --089e082f9ab494ea2405638d1cae--
>
>
>
--
Quanah Gibson-Mount
Product Architect
Symas Corporation
Packaged, certified, and supported LDAP solutions powered by OpenLDAP:
<http://www.symas.com>
5 years, 10 months
Re: (ITS#8791) OpenSSL 1.1.1 compat issue
by bbaetz＠google.com
--089e082f9ab494ea2405638d1cae
Content-Type: text/plain; charset="UTF-8"
Is there anything else I need to do in order to get this committed?
Bradley
On Fri, 15 Dec 2017 at 12:08 Bradley Baetz <bbaetz(a)google.com> wrote:
> Done in ftp://ftp.openldap.org/incoming/bradley-baetz-20171215.patch
>
>
> On Fri, 15 Dec 2017 at 04:36 Howard Chu <hyc(a)symas.com> wrote:
>
>> bbaetz(a)google.com wrote:
>> > Full_Name: Bradley Baetz
>> > Version: 2.4.45
>> > OS: linux
>> > URL: ftp://ftp.openldap.org/incoming/bradley-baetz-20171214.patch
>> > Submission from: (NULL) (2401:fa00:9:11:7ac0:58b5:299c:bebb)
>>
>> Thanks for the patch. The initialization of the static tlso_bio_method is
>> racy. One-time initializations should be done in tlso_init, and the
>> allocated
>> memory should be freed in tlso_destroy.
>>
>> >
>> > ITS#8533 added support for the OpenSSL's hiding of the bio_method_st
>> struct.
>> >
>> > However, it did this by re-defining the now-private structure, using
>> the OpenSSL
>> > 1.0 version. That will fail when OpenSSL changes their structure, which
>> they
>> > have already done for v1.1.1 - see
>> >
>> https://git.openssl.org/gitweb/?p=openssl.git;a=blob;f=include/internal/b...
>> >
>> > It also fails with BoringSSL, which has v1.0's OPENSSL_VERSION_NUMBER
>> define,
>> > but has not yet hidden the struct definition.
>> >
>> > The attached file is derived from OpenLDAP Software. All of the
>> modifications to
>> > OpenLDAP Software represented in the following patch(es) were developed
>> by
>> > Google, LLC. Google, LLC has not assigned rights and/or interest in
>> this work to
>> > any party. I, Bradley Baetz am authorized by Google, LLC, my employer,
>> to
>> > release this work under the following terms.
>> >
>> > The attached modifications to OpenLDAP Software are subject to the
>> following
>> > notice:
>> > Copyright 2017 Google, LLC.
>> > Redistribution and use in source and binary forms, with or without
>> modification,
>> > are permitted only as authorized by the OpenLDAP Public License.
>> >
>> >
>>
>>
>> --
>> -- Howard Chu
>> CTO, Symas Corp. http://www.symas.com
>> Director, Highland Sun http://highlandsun.com/hyc/
>> Chief Architect, OpenLDAP http://www.openldap.org/project/
>>
>
--089e082f9ab494ea2405638d1cae
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Is there anything else I need to do in order to get this c=
ommitted?<div><br></div><div>Bradley</div></div><br><div class=3D"gmail_quo=
te"><div dir=3D"ltr">On Fri, 15 Dec 2017 at 12:08 Bradley Baetz <<a href=
=3D"mailto:bbaetz@google.com">bbaetz(a)google.com</a>> wrote:<br></div><bl=
ockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #=
ccc solid;padding-left:1ex"><div dir=3D"ltr"><span style=3D"font-size:small=
">Done in=C2=A0</span><a href=3D"ftp://ftp.openldap.org/incoming/bradley-ba=
etz-20171215.patch" style=3D"font-size:small" target=3D"_blank">ftp://ftp.o=
penldap.org/incoming/bradley-baetz-20171215.patch</a><br><br class=3D"m_906=
2438285945864329inbox-inbox-Apple-interchange-newline"></div><br><div class=
=3D"gmail_quote"><div dir=3D"ltr">On Fri, 15 Dec 2017 at 04:36 Howard Chu &=
lt;<a href=3D"mailto:hyc@symas.com" target=3D"_blank">hyc(a)symas.com</a>>=
wrote:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8=
ex;border-left:1px #ccc solid;padding-left:1ex"><a href=3D"mailto:bbaetz@go=
ogle.com" target=3D"_blank">bbaetz(a)google.com</a> wrote:<br>
> Full_Name: Bradley Baetz<br>
> Version: 2.4.45<br>
> OS: linux<br>
> URL: <a href=3D"ftp://ftp.openldap.org/incoming/bradley-baetz-20171214=
.patch" rel=3D"noreferrer" target=3D"_blank">ftp://ftp.openldap.org/incomin=
g/bradley-baetz-20171214.patch</a><br>
> Submission from: (NULL) (2401:fa00:9:11:7ac0:58b5:299c:bebb)<br>
<br>
Thanks for the patch. The initialization of the static tlso_bio_method is<b=
r>
racy. One-time initializations should be done in tlso_init, and the allocat=
ed<br>
memory should be freed in tlso_destroy.<br>
<br>
><br>
> ITS#8533 added support for the OpenSSL's hiding of the bio_method_=
st struct.<br>
><br>
> However, it did this by re-defining the now-private structure, using t=
he OpenSSL<br>
> 1.0 version. That will fail when OpenSSL changes their structure, whic=
h they<br>
> have already done for v1.1.1 - see<br>
> <a href=3D"https://git.openssl.org/gitweb/?p=3Dopenssl.git;a=3Dblob;f=
=3Dinclude/internal/bio.h;hb=3De1dd8fa00a1e06d27c8b024dac7657a8d8a9b451#l16=
" rel=3D"noreferrer" target=3D"_blank">https://git.openssl.org/gitweb/?p=3D=
openssl.git;a=3Dblob;f=3Dinclude/internal/bio.h;hb=3De1dd8fa00a1e06d27c8b02=
4dac7657a8d8a9b451#l16</a><br>
><br>
> It also fails with BoringSSL, which has v1.0's OPENSSL_VERSION_NUM=
BER define,<br>
> but has not yet hidden the struct definition.<br>
><br>
> The attached file is derived from OpenLDAP Software. All of the modifi=
cations to<br>
> OpenLDAP Software represented in the following patch(es) were develope=
d by<br>
> Google, LLC. Google, LLC has not assigned rights and/or interest in th=
is work to<br>
> any party. I, Bradley Baetz am authorized by Google, LLC, my employer,=
to<br>
> release this work under the following terms.<br>
><br>
> The attached modifications to OpenLDAP Software are subject to the fol=
lowing<br>
> notice:<br>
> Copyright 2017 Google, LLC.<br>
> Redistribution and use in source and binary forms, with or without mod=
ification,<br>
> are permitted only as authorized by the OpenLDAP Public License.<br>
><br>
><br>
<br>
<br>
--<br>
=C2=A0 =C2=A0-- Howard Chu<br>
=C2=A0 =C2=A0CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a hr=
ef=3D"http://www.symas.com" rel=3D"noreferrer" target=3D"_blank">http://www=
.symas.com</a><br>
=C2=A0 =C2=A0Director, Highland Sun=C2=A0 =C2=A0 =C2=A0<a href=3D"http://hi=
ghlandsun.com/hyc/" rel=3D"noreferrer" target=3D"_blank">http://highlandsun=
.com/hyc/</a><br>
=C2=A0 =C2=A0Chief Architect, OpenLDAP=C2=A0 <a href=3D"http://www.openldap=
.org/project/" rel=3D"noreferrer" target=3D"_blank">http://www.openldap.org=
/project/</a><br>
</blockquote></div></blockquote></div>
--089e082f9ab494ea2405638d1cae--
5 years, 10 months
