Re: (ITS#5376) Full sync fails with delta-syncrepl, refreshAndPersist, and high modifications
by hyc＠symas.com
quanah(a)zimbra.com wrote:
> With a build of 2.3 with the (adjusted) patches from HEAD, it looks like
> this issue is resolved. I fully sync'd a replica from the master, and
> after it was done, I did a slapcat of them.
>
> A direct diff shows a lot of entries out of order (because the replica gets
> them in multiple refresh phases), however, if I sort the two ldif files,
> and do a diff, I get:
>
> [zimbra@smtp-2 tmp]$ sort slave.ldif>slave.ldif.s
> [zimbra@smtp-2 tmp]$ sort master.ldif>master.ldif.s
> [zimbra@smtp-2 tmp]$ diff -u master.ldif.s slave.ldif.s
> [zimbra@smtp-2 tmp]$
>
> No diff. So I believe this is fixed. Sound correct?
Sounds good to me. I guess this should go into both RE23 and RE24.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
15 years, 7 months
Re: (ITS#5370) race condition in slap_op_time()
by h.b.furuseth＠usit.uio.no
hyc(a)symas.com writes:
> Except that this will take a lot more restructuring in accesslog. Going with
> the quick fix for now.
Looks like I didn't notice the first use of o_tincr in accesslog:-(
I guess accesslog can set an "I need o_tincr" flag somewhere, which
slapd will then obey.
--
Hallvard
15 years, 7 months
Re: (ITS#5376) Full sync fails with delta-syncrepl, refreshAndPersist, and high modifications
by quanah＠zimbra.com
--On February 13, 2008 3:14:37 AM +0000 quanah(a)zimbra.com wrote:
>
>
> --On February 13, 2008 3:03:46 AM +0000 quanah(a)zimbra.com wrote:
>
>>
>> And, with the patch for ITS#5282, all replication stops (probably a good
>> thing).
>
> More specifically, the replica stops accepting any new changes, since it
> can't proceed on the MODs to a non-existent entry. The master of course
> continues to accept and record changes, and any non-problematic replicas
> continue to receive changes.
With a build of 2.3 with the (adjusted) patches from HEAD, it looks like
this issue is resolved. I fully sync'd a replica from the master, and
after it was done, I did a slapcat of them.
A direct diff shows a lot of entries out of order (because the replica gets
them in multiple refresh phases), however, if I sort the two ldif files,
and do a diff, I get:
[zimbra@smtp-2 tmp]$ sort slave.ldif >slave.ldif.s
[zimbra@smtp-2 tmp]$ sort master.ldif >master.ldif.s
[zimbra@smtp-2 tmp]$ diff -u master.ldif.s slave.ldif.s
[zimbra@smtp-2 tmp]$
No diff. So I believe this is fixed. Sound correct?
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
15 years, 7 months
Re: (ITS#5376) Full sync fails with delta-syncrepl, refreshAndPersist, and high modifications
by quanah＠zimbra.com
--On February 13, 2008 3:03:46 AM +0000 quanah(a)zimbra.com wrote:
>
> And, with the patch for ITS#5282, all replication stops (probably a good
> thing).
More specifically, the replica stops accepting any new changes, since it
can't proceed on the MODs to a non-existent entry. The master of course
continues to accept and record changes, and any non-problematic replicas
continue to receive changes.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
15 years, 7 months
Re: (ITS#5376) Full sync fails with delta-syncrepl, refreshAndPersist, and high modifications
by quanah＠zimbra.com
--On February 13, 2008 2:57:39 AM +0000 quanah(a)zimbra.com wrote:
> Full_Name: Quanah Gibson-Mount
> Version: 2.3.40
> OS:
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (24.23.156.219)
>
>
> If you start a replica with an empty DB, using delta-sync replication,
> and the master is receiving a high rate of changes, the Refresh phase
> will skip entries that have been modified. When it moves to the
> "persist" phase, delta-syncrepl will send modification diffs to those
> entries rather than the full entries, so entries end up missing, having
> never been sent in a full form.
And, with the patch for ITS#5282, all replication stops (probably a good
thing).
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
15 years, 7 months
(ITS#5376) Full sync fails with delta-syncrepl, refreshAndPersist, and high modifications
by quanah＠zimbra.com
Full_Name: Quanah Gibson-Mount
Version: 2.3.40
OS:
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (24.23.156.219)
If you start a replica with an empty DB, using delta-sync replication, and the
master is receiving a high rate of changes, the Refresh phase will skip entries
that have been modified. When it moves to the "persist" phase, delta-syncrepl
will send modification diffs to those entries rather than the full entries, so
entries end up missing, having never been sent in a full form.
15 years, 7 months
Re: (ITS#5339) wrong referral from back-bdb
by hyc＠symas.com
h.b.furuseth(a)usit.uio.no wrote:
> Howard Chu writes:
>>> Both RFCs disagree.
>> They are wrong. Or at least, under-specified. In X.501 section 17.3
>> "Directory Distribution Model" it's quite clear that all of the
>> components of a distributed directory must belong to a single DIT.
>
> Which is not true in the LDAP world, and I don't know about today's
> X.500 world. Nameflow died and Dante was unable to resurrect it. Maybe
> the X.500 world has also switched to 'dc' structure, I don't know.
>
> Anyway, LDAP is not X.500.
RFC4510 Section 2 "Relationship to X.500"
This technical specification defines LDAP in terms of [X.500] as an
X.500 access mechanism. An LDAP server MUST act in accordance with
the X.500 (1993) series of International Telecommunication Union -
Telecommunication Standardization (ITU-T) Recommendations when
providing the service.
> Remember, LDAP referrals need not even be
> LDAP URLs. My guess is that's one reason for the difference from X.500:
> When you can even return a HTTP URL, why should you _not_ be allowed
> to return a far smaller change from the original URL?
As I've said many times before, LDAP referrals are a poorly designed piece of
junk. The fact that the spec leaves it open to return other types of URLs but
nobody in the world knows what that should actually mean to an LDAP client is
just further proof of the abysmal lack of thought that went into the design.
> Also LDAP started out a lot sloppier than X.500.
That's self-evident. ;)
> Getting some response
> which hopefully resembled what you wanted was better than none:-)
Getting a wrong answer is worse than getting no answer, because the client
will just assume that everything is OK and keep going, rather than reporting
an error to the user.
>> And again, the service model requires the directory to store the
>> client's data without any alteration. If the client wants to create
>> the entry "cn=me,o=we" either that exact entry must be created, or the
>> operation must fail.
>
> Well, the operation does fail in the original server. Then the client
> may try another operation at another server... I don't know if that's a
> valid interpretation of the service model or not. In any case, it's a
> detail compared to the mess
No, in this case the client is not being told to retry the *same* operation in
another server, it's being told to try a *very different* operation in another
server. Once again, the client made a specific request to create an entry with
a specific DN. If the server can't honor the request as specified then it
should fail and the operation should halt.
Anyway, the very existence of LDAP referrals is a violation of the service
model. X.501(1993) section 18.2:
It is a requirement of the Directory that, for particular modes of
user interaction, the distribution of the directory be rendered
transparent, thereby giving the effect that the whole of the DIB
appears to be within each and every DSA.
LDAPv1 had no referrals. When they were introduced in LDAPv2 it's clear that
nobody knew what they were doing, or nobody wanted to tackle the glaring
absence of an analogue to X.500 DSP. They should never have been introduced.
We're stuck with them for now, but we can at least try to make them make sense.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
15 years, 7 months
Re: (ITS#5370) race condition in slap_op_time()
by hyc＠symas.com
hyc(a)symas.com wrote:
> h.b.furuseth(a)usit.uio.no wrote:
>> However as far as I can tell, only the accesslog overlay uses
>> op->o_tincr, the value which needs the mutex. And accesslog
>> calls slap_op_time() itself when it needs that value.
>> So maybe we should remove the op->o_tincr field. Other calls
>> to slap_op_time() can be replaced with slap_get_time().
>
> Sounds like a good idea. It was somewhat redundant with the CSN counter
> (except that this was for all operations, and CSN was only for write
> operations) and it never became generally useful. (Shifted to microsecond
> timestamps in 2.4, because tincr isn't useful across multiple servers.) I
> guess it makes more sense to make it specific to accesslog.
>
Except that this will take a lot more restructuring in accesslog. Going with
the quick fix for now.
--
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
15 years, 7 months
Re: (ITS#5374) accesslog+hdb crash with test043-delta-syncrepl
by quanah＠zimbra.com
--On February 12, 2008 6:12:08 PM +0000 quanah(a)zimbra.com wrote:
> (and it passed 10 times for me successfully).
At Hallvard's request, I patched accesslog thusly:
diff -u accesslog.c.orig accesslog.c
--- accesslog.c.orig 2008-02-12 10:18:16.000000000 -0800
+++ accesslog.c 2008-02-12 10:21:46.000000000 -0800
@@ -914,6 +914,7 @@
ldap_pvt_thread_mutex_lock( &li->li_log_mutex );
old = li->li_old;
li->li_old = NULL;
+ assert(li->li_unlock == (int) ldap_pvt_thread_self());
li->li_unlock = 0;
ldap_pvt_thread_mutex_unlock( &li->li_op_mutex );
}
@@ -1262,7 +1263,8 @@
* overlays like refint to keep working.
*/
ldap_pvt_thread_mutex_lock( &li->li_op_mutex );
- li->li_unlock = 1;
+ /* li->li_unlock = 1; */
+ li->li_unlock = (int) ldap_pvt_thread_self();
if ( li->li_oldf && ( op->o_tag == LDAP_REQ_DELETE ||
op->o_tag == LDAP_REQ_MODIFY )) {
int rc;
Multiple runs of the test continued to be successful.
--Quanah
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
15 years, 7 months
Re: (ITS#5374) accesslog+hdb crash with test043-delta-syncrepl
by quanah＠zimbra.com
(and it passed 10 times for me successfully).
--On February 12, 2008 5:59:22 PM +0000 quanah(a)zimbra.com wrote:
>
>
> --On February 12, 2008 12:11:50 PM +0000 h.b.furuseth(a)usit.uio.no wrote:
>
>> Full_Name: Hallvard B Furuseth
>> Version: RE23
>> OS: Linux
>> URL:
>> Submission from: (NULL) (129.240.6.233)
>> Submitted by: hallvard
>>
>>
>> /configure \
>> --disable-dynamic --disable-aci --enable-crypt --enable-lmpasswd \
>> --enable-spasswd --enable-rlookups --enable-slapi --enable-wrappers \
>> --enable-backends --disable-sql --enable-overlays \
>> CPPFLAGS='-DLDAP_THREAD_DEBUG -DLDAP_RDWR_DEBUG'
>
> I built as:
>
> LD_LIBRARY_PATH="/usr/local/lib" CC=gcc CXX=g++ CFLAGS='-g -O0'
> CXXFLAGS='-g -O0' CPPFLAGS='-DLDAP_THREAD_DEBUG -DLDAP_RDWR_DEBUG' sh
> configure --datadir='${prefix}/lib' --libexecdir='${prefix}/lib'
> --sharedstatedir='${prefix}/lib' --prefix=/usr/local
> --disable-ipv6 --with-cyrus-sasl --with-tls
> --enable-dynamic --enable-slapd --enable-modules
> --enable-spasswd --enable-rewrite
> --enable-rlookups --enable-wrappers
> --enable-backends=mod --disable-shell
> --disable-sql --enable-overlays=mod --enable-slapi=yes
>
>
> Thread package selected is:
>
> -pthread
>
> s,@LTHREAD_LIBS@, -pthread,;t t
>
>
> --Quanah
>
>
> --
>
> Quanah Gibson-Mount
> Principal Software Engineer
> Zimbra, Inc
> --------------------
> Zimbra :: the leader in open source messaging and collaboration
>
>
--
Quanah Gibson-Mount
Principal Software Engineer
Zimbra, Inc
--------------------
Zimbra :: the leader in open source messaging and collaboration
15 years, 7 months
