(ITS#5545) Inconsistency in multi-master setup
by st＠itsc.cuhk.edu.hk
Full_Name: SiuTo Wong
Version: 2.4.10
OS: CentOS 5 (Final)
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (137.189.29.184)
Have setup 2-node multi-master configuration. Found that when operating on same
set of users at high loading, there will be consistency between the masters.
The problem can be simulated by running copies of perl script to add/delete the
same set of users simultaneously, e.g.
add user a001 - a100 &
delete user a001 - a100 &
or
add user a001 - a100 &
add user a001 - a100 &
There is no problem if multiple threads of add/delete are running,
when operating on different sets of users.
Attached please find snap of syncrepl setting of one of the servers. All
attributes are the same on both masters, except serverId, rid, and provider.
Thanks a lot.
====================== cut here ======================
index entryCSN,entryUUID eq
lastmod on
serverId 2
syncrepl rid=002
provider=ldap://192.168.28.70
starttls=yes
tls_reqcert=never
tls_cacert=/etc/CA/cacert.pem
type=refreshAndPersist
retry="5 + 5 +"
searchbase="dc=mydomain,dc=hk"
schemachecking=off
bindmethod=simple
binddn="cn=srAdmin,ou=People,dc=mydomain,dc=hk"
credentials=youguess
interval=00:00:00:05
mirrormode true
overlay syncprov
syncprov-checkpoint 100 10
syncprov-sessionlog 100
limits dn.exact="cn=srAdmin,ou=People,dc=cuhk,dc=edu,dc=hk" time.soft=unlimited
time.hard=unlimited size.soft=unlimited size.hard=unlimited
====================== cut here ======================
15 years, 5 months
Re: (ITS#5517) add superclass of existing class fails
by Kurt＠OpenLDAP.org
On Jun 5, 2008, at 6:31 AM, h.b.furuseth(a)usit.uio.no wrote:
> And when they add the absent A and that fails, but we can fix that.
I think the best is option is disallow addition of an unlisted subject
and disallow the removal of a unlisted subclass (except by replace).
This will provide more a more consistent and symmetric behavior.
In your initial report, it was this inconsistent and non-symmetric
behavior that was your most significant concern.
As far as RFC conformance goes, I argue that with this change, the
behavior would be fully conformant. No where in the RFCs does it
require servers to return implicitly added values, no where in the
RFCs does it preclude servers from removing implicitly added values
upon removal of values which caused there addition.
The concern I have is impact on clients which add a class and then
delete that class, expecting it just to work. But if that class is
subclass, and the add causes the insertion of some other class, then
the delete will either leave that other class there or lead to an
error, both undesirable and unexpected behaviors.
I think the mistake that LDAP designers made (long ago) was not to
require LDAP CLIENTS to provide all (non-top) classes, as is required
of DAP clients.
Regarding objectClass being special (especially with regard to its
equality matching rule): consider (objectClass=top) and X.500 "top may
be omitted" statement.
-- Kurt
15 years, 5 months
Re: (ITS#5517) add superclass of existing class fails
by h.b.furuseth＠usit.uio.no
(I seem to be arguing myself into two different corners here, but
anyway...)
Kurt(a)OpenLDAP.org writes:
>On May 24, 2008, at 2:37 PM, h.b.furuseth(a)usit.uio.no wrote:
>> ando(a)sys-net.it writes:
>>> The issue right now is caused by the fact that comparing present
>>> values with the asserted one causes objectSubClassMatch() to check
>>> for match including superiors.
>>
>> This looks like a bug to me.
>
> Maybe in the RFCs.
Yes, to my eyes it cleary says what you say was not intended.
>> objectClass has EQUALITY matching rule
>> objectIdentifierMatch, which does not follow the object class
>> inheritance chain.
>
> Yes, but objectClass is quite special (despite being a
> userApplications attribute).
Hpmh. Not very, according to RFCs 4511-12. Maybe we all "knew" it so
well that we never noticed, during several IETF WGs.
> For instance, (objectClass=*) is to match DSEs which don't have an
> objectClass attribute.
That's explicitly mentioned in RFC 4512 for the root DSE, as is the
(objectClass=subschema) special case.
Section RFC 4512 section 3.3 says each entry in the DIT has an
'objectClass' attribute. Are other DSEs not in the DIT?
There is RFC 4511 section 4.5.1 (Search Request)'s statement that List
and Read operations can be emulated with a search for (objectClass=*).
Nowhere else can I find that (objectClass=*) is special.
> Likewise, objectClass=foo can match objectClass attributes which
> don't explicitly contain foo.
Not according what the RFCs say to my eyes, (below):
>> See RFC 4517 section 4.2.26. RFC 4512 section 3.3
>> (the 'objectClass' attribute) does not special-case this.
>>
>> What makes slapd work is that it also disobeys RFC 4512 section 3.3:
>> "When creating an entry or adding an 'objectClass' value to an
>> entry,
>> all superclasses of the named classes SHALL be implicitly added as
>> well if not already present."
>
>> This doesn't say the classes shall be considered to be implicitly
>> present, like slapd does.
>
> You are reading 3.3 as if it said "SHALL be added (by the server)".
> It doesn't say "SHALL be added (by the server). It says "SHALL be
> implicitly added" (presumedly by the server).
I don't see how that changes anything, but then I'm not a native English
speaker.
> This was discussed during LDAPbis discussions. As the Editor of RFC
> 4512, I can say that it was my intent to allow servers to merely treat
> superclasses as being present.
Hard do argue with that. I couldn't find these discussions though. Do
you remember where/when? All I found was this, which seems to imply the
opposite:
http://www.openldap.org/lists/ietf-ldapbis/200403/msg00023.html
>> Instead slapd should add the missing classes to the actual object,
>
> Doing so is problematic. A client which adds B then deletes B would
> be surprised by non-net-zero result of the operation(s).
Clients will be surprised anyway. Currently clients are being
surprised when (objectClass=A) matches even though A is absent.
And when they add the absent A and that fails, but we can fix that.
>> just like it adds some other attribute/values
>> implicitly (createTimestamp, subschemaSubentry, etc).
>
> These are operational attributes. ObjectClass is an userApplications
> attribute. Servers should not muck with the values of userApplication
> attributes.
Well, it does when it adds missing naming attributes (since RFC4511,
ITS#5412). I guess that is a better example. And it "mucks with"
objectClass in a different way anyway, as you've just pointed out.
--
Hallvard
15 years, 5 months
Re: (ITS#5540) Normalization assertion in attr.c
by unix.gurus＠gmail.com
Hi,
On Wed, Jun 4, 2008 at 2:20 PM, Hallvard B Furuseth
<h.b.furuseth(a)usit.uio.no> wrote:
> I haven't follwed this ITS, but a few notes anyway:
>
> unix.gurus(a)gmail.com writes:
>> monitorCounter
>> monitorCounter does not have an equality matching rule,
>
> Yes it does. back-monitor/init.c line 1710. What are you looking at?
> (Hmm, I too had the impression it had no EQUALITY rule...)
I'm working with OpenLDAP 2.4.9.
You're right. integerMatch has an equality rule but no normalizing
function. The logic that determines whether a normalized value should
exist says:
have_norm = a->a_desc->ad_type->sat_equality != NULL &&
a->a_desc->ad_type->sat_equality->smr_normalize != NULL;
...
new_nval = nvals != NULL;
...
if ( have_norm != new_nval ) {
assert()
So we don't need nvals for integers. For this reason I was right to
remove the nval for monitorCounter, monitorOpInitiated,
monitorOpCompleted, but my reasoning was wrong.
>> monitorOpCompleted
>> monitorOpInitiated
>> These attributes do not have equality matching rules,
>
> These do too, inherited from monitorCounter.
See above.
> it doesn't work right in 2.3 however:
>
> ldapsearch -LLL -bcn=monitor -s one '(monitorCounter=0)' monitorCounter
> dn: cn=Operations,cn=Monitor
> monitorOpInitiated: 231542983
> monitorOpCompleted: 231542982
I wonder if val is being maintained and nval is being left at 0. The
code at the end of monitor_subsys_ops_update() maintains a_vals but
not a_nvals, so that is probably the problem:
/* NOTE: no minus sign is allowed in the counters... */
UI2BV( &a->a_vals[ 0 ], nInitiated );
ldap_pvt_mp_clear( nInitiated );
...
/* NOTE: no minus sign is allowed in the counters... */
UI2BV( &a->a_vals[ 0 ], nCompleted );
ldap_pvt_mp_clear( nCompleted );
>> namingContexts
>> configContext
>> dynamicSubtrees
>>
>> These attributes do not have equality matching rules, but probably
>> should. I add 'EQUALITY distinguishedNameMatch' to them in
>> schema_prep.c.
>
> No, that breaks the definitions in the RFCs they come from. (See their
> DESC strings.) I don't know why they lack EQUALITY rules, maybe we just
> forgot when we defined them, but that's the way it is. Same with
> supportedControl.
Ok, so we should not define nvals for them?
> monitorContext, readOnly and the olc* attributes are defined by OpenLDAP
> (their OIDs start with 1.3.6.1.4.1.4203) and can be modified if we feel
> like it. Personally I prefer attrs to have the matching rules they can
> have unless there is a reason not to, but I didn't write these modules
> so I don't know if there _is_ a reason not to.
--
Thanks,
Sean Burford
15 years, 5 months
Re: (ITS#5540) Normalization assertion in attr.c
by h.b.furuseth＠usit.uio.no
I haven't follwed this ITS, but a few notes anyway:
unix.gurus(a)gmail.com writes:
> monitorCounter
> monitorCounter does not have an equality matching rule,
Yes it does. back-monitor/init.c line 1710. What are you looking at?
(Hmm, I too had the impression it had no EQUALITY rule...)
> monitorOpCompleted
> monitorOpInitiated
> These attributes do not have equality matching rules,
These do too, inherited from monitorCounter.
it doesn't work right in 2.3 however:
ldapsearch -LLL -bcn=monitor -s one '(monitorCounter=0)' monitorCounter
dn: cn=Operations,cn=Monitor
monitorOpInitiated: 231542983
monitorOpCompleted: 231542982
> namingContexts
> configContext
> dynamicSubtrees
>
> These attributes do not have equality matching rules, but probably
> should. I add 'EQUALITY distinguishedNameMatch' to them in
> schema_prep.c.
No, that breaks the definitions in the RFCs they come from. (See their
DESC strings.) I don't know why they lack EQUALITY rules, maybe we just
forgot when we defined them, but that's the way it is. Same with
supportedControl.
monitorContext, readOnly and the olc* attributes are defined by OpenLDAP
(their OIDs start with 1.3.6.1.4.1.4203) and can be modified if we feel
like it. Personally I prefer attrs to have the matching rules they can
have unless there is a reason not to, but I didn't write these modules
so I don't know if there _is_ a reason not to.
--
Hallvard
15 years, 5 months
Re: (ITS#5540) Normalization assertion in attr.c
by unix.gurus＠gmail.com
HI,
On Fri, May 30, 2008 at 2:40 AM, <hyc(a)symas.com> wrote:
> hyc(a)symas.com wrote:
>> We'll probably need to discuss on the -devel list what steps to take from here.
>>
> I wrote the attached patch for the original issue. Unfortunately it asserted
> right away:
>
...
>
> It was adding the createTimestamp attribute, without providing normalized
> values. slap_add_opattrs was written before the generalizedTimeNormalize
> function was written... I suspect there will be a fair number of cases that
> need to be cleaned up.
I've identified these attributes as having differences between their
nval population and their schema.
I've been using Google coredumper instead of assertions to mark places
where normalization looks bad. In this way, I get a bunch of core
files to look through after a run instead of a single assertion. Once
all of the attributes that have bad normalization have been identified
in this way, I'll go back over the source and look for other places
where they are used.
I haven't found all of the normalization differences yet, but I wanted
to check in to ensure I'm on the right path. I would like your
comments on whether these look like the right changes to make. You
can refer to ftp://ftp.openldap.org/incoming/sburford-initial-normalization-20080604.p...
for the details.
createTimestamp
createTimestamp does have an equality matching rule, as I think it should.
The buffer timestamp is populated with a generalized time by
slap_timestamp(), so I add ×tamp as the nval for the calls to
attr_merge_one in add.c slap_add_opattrs() and schema.c schema_info().
I also use a slight variation of this for createTimestamp in
slapadd.c slapadd().
modifyTimestamp
Same arguments and modifications as createTimestamp.
modifyTimestamp is also used in modify.c slap_mods_opattrs(), where I
ch_malloc a new BerVarray for the mod->sml_nvalues. I dont use the
same BerVarray for vals and nvals in case whatever frees mod entries
frees them as separate pointers (double free). Does something free
all of the vals and nvals in the modify list?
entryCSN
entryCSN does have an equality matching rule, as I think it should.
In add.c slap_add_opattrs() I change the attr_merge_one into an
attr_merge_normalize_one with op->o_tmpmemctx as the memory context.
I haven't looked into the memory context argument, but assume it
contains a list of things to free when the context is finished with?
In slapadd I also change the attr_merge_one into an
attr_merge_normalize_one but I pass a NULL memory context, since
slapadd is short lived and there are other examples like this in the
nearby source code.
In modify.c slap_mods_opattrs() I ch_malloc a new BerVarray for the
entryCSN nval and use attr_normalize_one to populate it. Again I
assume something frees the components of the modify list later.
contextCSN
contextCSN does have an equality matching rule, as I think it should.
In ctxcsn.c slap_create_context_csn_entr() I change attr_merge_one
into attr_merge_normalize_one with a NULL memory context.
In overlays/syncprov.c syncprov_checkpoint() and syncrepl.c
syncrepl_updateCookie() it is a bit more difficult because a local
modlist is created without any attr_merge calls, so I ch_malloc an
array for the nvals, and populate it with attr_normalize and use
ber_array_bvfree to free it.
monitorCounter
monitorCounter does not have an equality matching rule, so I remove
the nval argument from attr_merge_one() in back-monitor/conn.c
monitor_subsys_conn_init() (affects counters for max FD, total conns
and current conns).
monitorOpCompleted
monitorOpInitiated
These attributes do not have equality matching rules, so I remove the
nval argument from attr_merge_one() in back-monitor/operation.c
monitor_subsys_ops_init().
monitorContext
namingContexts
configContext
dynamicSubtrees
These attributes do not have equality matching rules, but probably
should. I add 'EQUALITY distinguishedNameMatch' to them in
schema_prep.c.
olcRootDN
olcSchemaDN
These attributes do not have equality matching rules, but probably
should. I add 'EQUALITY distinguishedNameMatch' to them in bconfig.c.
supportedControl
This attribute does not have an equality matching rule, but probably
should. I add 'EQUALITY objectIdentifierMatch' to it in
schema_prep.c.
readOnly
This attribute does not have an equality matching rule, I don't know
if it should. I NULLed the nval passed to attr_merge_one in
back-monitor/database.c.
default_referral:
In rootdse.c root_dse_info() I change attr_merge to
attr_merge_normalize for default_referral, since it has an equality
matching rule. There doesn't seem to be a memory context in use here.
I suspect this normalization might result in memory leakage unless
followed by a free in root_dse_info()?
Also, bconfig.c config_build_attrs() contains this logic which calls
attr_merge_normalize() for values that don't need normalization.
attr_merge_normalize won't normalize them though, so it should be
fine:
if ( c->rvalue_nvals )
attr_merge(e, ct[i].ad, c->rvalue_vals, c->rvalue_nvals);
else
attr_merge_normalize(e, ct[i].ad, c->rvalue_vals, NULL);
> I don't have time at the moment to chase them all down.
> Anyone else want to jump in here? If not, we may have to push this back a bit.
> Note that this patch will probably require a fair number of databases to be
> reloaded.
The attributes listed above tend to agree with that, but if they
really were not being normalized before storage I'm surprised that the
original assertion wasn't hit more often?
--
Thanks,
Sean Burford
15 years, 5 months
Re: (ITS#5544) libRSAglue not found
by hyc＠symas.com
quanah(a)zimbra.com wrote:
> --On Tuesday, June 03, 2008 9:01 PM +0000 steven(a)vaningelgem.be wrote:
>
>> ------=_Part_3208_8915130.1212526882765
>> Content-Type: text/plain; charset=UTF-8
>> Content-Transfer-Encoding: 7bit
>> Content-Disposition: inline
>>
>> Hi Howard,
>>
>>
>> I did read the INSTALL file, and I know about the LIBS variable... Let's
>> see... I added a LIBS="-lssl -lsasl2" to it... So in the config.log that
>> changed the line to include (at the end) -lssl -lsasl2, as it's supposed
>> to do...
>> But it didn't changed the fact that -lRSAglue was still included, which is
>> the original reason I posted this.
>>
>> When I check out the openSSL sources, I see that the v0.9.6 branch op
>> OpenSSL is the only branch having that RSAglue in it (with the latest
>> version from March 2004)... All later versions (0.9.7 and higher) don't
>> have this RSAglue in it anymore...
>>
>> Would that mean that I should build OpenLDAP against a version of OpenSSL
>> of 4 years ago?
>>
>> That's why I submitted the bug in the first place! Because this RSAglue
>> library cannot be found anymore...
As you've already noted, no current version of OpenSSL uses libRSAglue. If
you're running into this, then something is flaky in your OpenSSL build. That
is not an OpenLDAP issue. However, if you examine the error messages in
config.log, you should be able to see what tests are failing with libssl.
> I build OpenLDAP against OpenSSL 0.9.8g all the time without problem in my
> own custom location (/opt/zimbra/openssl/lib). You simply need to define
> your various variables to gcc correctly.
>
> I suggest reading the OpenLDAP FAQ, which has an article or two on linking
> against non-standard locations. My guess is you have some other library
> pulling in libRSAglue.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
15 years, 5 months
Re: (ITS#5544) libRSAglue not found
by quanah＠zimbra.com
--On Tuesday, June 03, 2008 9:01 PM +0000 steven(a)vaningelgem.be wrote:
> ------=_Part_3208_8915130.1212526882765
> Content-Type: text/plain; charset=UTF-8
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline
>
> Hi Howard,
>
>
> I did read the INSTALL file, and I know about the LIBS variable... Let's
> see... I added a LIBS="-lssl -lsasl2" to it... So in the config.log that
> changed the line to include (at the end) -lssl -lsasl2, as it's supposed
> to do...
> But it didn't changed the fact that -lRSAglue was still included, which is
> the original reason I posted this.
>
> When I check out the openSSL sources, I see that the v0.9.6 branch op
> OpenSSL is the only branch having that RSAglue in it (with the latest
> version from March 2004)... All later versions (0.9.7 and higher) don't
> have this RSAglue in it anymore...
>
> Would that mean that I should build OpenLDAP against a version of OpenSSL
> of 4 years ago?
>
> That's why I submitted the bug in the first place! Because this RSAglue
> library cannot be found anymore...
I build OpenLDAP against OpenSSL 0.9.8g all the time without problem in my
own custom location (/opt/zimbra/openssl/lib). You simply need to define
your various variables to gcc correctly.
I suggest reading the OpenLDAP FAQ, which has an article or two on linking
against non-standard locations. My guess is you have some other library
pulling in libRSAglue.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
15 years, 5 months