7:27 a.m.
I am updating an OpenLDAP installation at present, and one of the
improvements is to introduce a pair of master servers running in
mirror mode. There will be several read-only servers as well, and I
would like those to replicate from whichever master is currently
available.
I did the obvious thing, and put two syncrepl clauses in the
read-only server's config - one for each master server.
Starting with an empty database, the read-only server chews CPU
badly and generates thousands of log messages like this:
Sep 2 14:57:34 nis0 slapd[9257]: => bdb_idl_insert_key: c_put id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
Sep 2 14:57:38 nis0 slapd[9257]: => bdb_idl_delete_key: c_del id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
Sep 2 14:57:38 nis0 slapd[9257]: conn=-1 op=0: attribute "entryCSN" index
delete failure
Sep 2 14:57:38 nis0 slapd[9257]: => bdb_idl_insert_key: c_put id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
Sep 2 14:57:38 nis0 slapd[9257]: => bdb_idl_delete_key: c_del id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
Sep 2 14:57:38 nis0 slapd[9257]: conn=-1 op=0: attribute "entryCSN" index
delete failure
....
Sep 2 14:56:10 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
Sep 2 14:56:10 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
Sep 2 14:56:11 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
Sep 2 14:56:11 nis0 slapd[4953]: conn=-1 op=0: attribute "localAttribute" index
add failure
Sep 2 14:56:11 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
Sep 2 14:56:11 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
It is very likely that each syncrepl connection will be retrieving
the same entries at about the same time, so there is clearly potential
for collisions.
Question: is this topology sensible? If it is expected to work I
will gather some debug data for an ITS. If not, I will have to
drop back to plan B...
OpenLDAP 2.4.22 with BerkeleyDB.4.8 running on SLES 10.3 i586
Thanks
Andrew
--
-----------------------------------------------------------------------
| From Andrew Findlay, Skills 1st Ltd |
| Consultant in large-scale systems, networks, and directory services |
| http://www.skills-1st.co.uk/ +44 1628 782565 |
-----------------------------------------------------------------------
4:41 a.m.
Andrew,
Andrew Findlay schrieb am 02.09.2010 16:27 Uhr:
I am updating an OpenLDAP installation at present, and one of the
improvements is to introduce a pair of master servers running in
mirror mode. There will be several read-only servers as well, and I
would like those to replicate from whichever master is currently
available.
I did the obvious thing, and put two syncrepl clauses in the
read-only server's config - one for each master server.
I'm not sure, this
is the "obvious". :)
I'm not sure either, if it is possible to define more than one provider
in the syncrepl option.
The best would be to use a unified service address and a single name for
both servers.
Marc
5:45 a.m.
On Fri, Sep 03, 2010 at 01:41:32PM +0200, Marc Patermann wrote:
> I did the obvious thing, and put two syncrepl clauses in the
> read-only server's config - one for each master server.
I'm not sure, this is the "obvious". :)
I'm not sure either, if it is possible to define more than one provider in
the syncrepl option.
I don't think it is, but what I did was to create two syncrepl
options: that is certainly supported as it is needed for
multi-master setups.
The best would be to use a unified service address and a single name
for
both servers.
That is one of the various 'plan Bs'....
Andrew
--
-----------------------------------------------------------------------
| From Andrew Findlay, Skills 1st Ltd |
| Consultant in large-scale systems, networks, and directory services |
| http://www.skills-1st.co.uk/ +44 1628 782565 |
-----------------------------------------------------------------------
7:35 a.m.
Hi Andrew,
Le 02/09/2010 16:27, Andrew Findlay a écrit :
I am updating an OpenLDAP installation at present, and one of the
improvements is to introduce a pair of master servers running in
mirror mode. There will be several read-only servers as well, and I
would like those to replicate from whichever master is currently
available.
I did the obvious thing, and put two syncrepl clauses in the
read-only server's config - one for each master server.
Starting with an empty database, the read-only server chews CPU
badly and generates thousands of log messages like this:
Sep 2 14:57:34 nis0 slapd[9257]: => bdb_idl_insert_key: c_put id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
Sep 2 14:57:38 nis0 slapd[9257]: => bdb_idl_delete_key: c_del id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
Sep 2 14:57:38 nis0 slapd[9257]: conn=-1 op=0: attribute "entryCSN" index
delete failure
Sep 2 14:57:38 nis0 slapd[9257]: => bdb_idl_insert_key: c_put id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
Sep 2 14:57:38 nis0 slapd[9257]: => bdb_idl_delete_key: c_del id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
Sep 2 14:57:38 nis0 slapd[9257]: conn=-1 op=0: attribute "entryCSN" index
delete failure
....
Sep 2 14:56:10 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
Sep 2 14:56:10 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
Sep 2 14:56:11 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
Sep 2 14:56:11 nis0 slapd[4953]: conn=-1 op=0: attribute "localAttribute"
index add failure
Sep 2 14:56:11 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
Sep 2 14:56:11 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
It is very likely that each syncrepl connection will be retrieving
the same entries at about the same time, so there is clearly potential
for collisions.
DB_LOCK_DEADLOCK errors are only a warning: retries should occur until
the operation completes. Of course, if they can be avoided, best avoid!
Question: is this topology sensible? If it is expected to work I
will gather some debug data for an ITS. If not, I will have to
drop back to plan B...
This is an interesting configuration. I would not have proceeded like
this but, as Marc Patermann suggested, I would set up a virtual address
that points to the currently available master, and configure one
syncrepl clause using this address (and all other LDAP clients, in
fact). Could this approach work for you?
Looking at the OpenLDAP configuration issue, I note your remark that
several syncrepl statements are used in multi-master setups. I'm not
entirely sure why this works better, but it may well be due to all
servers having the syncprov overlay, which serializes modifications when
a persistent search is in progress.
Using two syncrepl statements is certainly suboptimal, as all
modifications will be replicated twice to all read-only servers.
However, I don't see any reason why it shouldn't work, off the top of my
head. Does slapd end up synchronizing everything?
Jonathan
--
==========================================
Jonathan CLARKE
------------------------------------------
Normation
44 rue Cauchy, 94110 Arcueil, France
------------------------------------------
Telephone: +33 (0)1 83 62 26 96
------------------------------------------
Web: http://www.normation.com/
==========================================
8:18 a.m.
On Fri, Sep 03, 2010 at 04:35:24PM +0200, Jonathan CLARKE wrote:
DB_LOCK_DEADLOCK errors are only a warning: retries should occur
until the
operation completes. Of course, if they can be avoided, best avoid!
> Question: is this topology sensible? If it is expected to work I
> will gather some debug data for an ITS. If not, I will have to
> drop back to plan B...
This is an interesting configuration. I would not have proceeded like this
but, as Marc Patermann suggested, I would set up a virtual address that
points to the currently available master, and configure one syncrepl clause
using this address (and all other LDAP clients, in fact). Could this
approach work for you?
That is what I have done now, and it does work. I am still a little
uncertain about it though: when the normal server fails and the DNS
entry or routing changes to point to the hot standby, will this
confuse the consumer slapd? We are effectively telling it that the
second machine *is* the original one, yet it will have a different
serverID and possibly different contents.
Looking at the OpenLDAP configuration issue, I note your remark that
several syncrepl statements are used in multi-master setups. I'm not
entirely sure why this works better, but it may well be due to all servers
having the syncprov overlay, which serializes modifications when a
persistent search is in progress.
Sounds plausible.
Using two syncrepl statements is certainly suboptimal, as all
modifications
will be replicated twice to all read-only servers. However, I don't see any
reason why it shouldn't work, off the top of my head. Does slapd end up
synchronizing everything?
Not sure - there were only 25000 entries but I gave up and stopped the
consumer server after 30 minutes as it still had not synchronised.
Good point about the double replication, though if it had worked
cleanly it would be OK in the (low modification) environment that I
have. The advantage is that nothing else is needed to manage the
failover / fail-back cases.
Andrew
--
-----------------------------------------------------------------------
| From Andrew Findlay, Skills 1st Ltd |
| Consultant in large-scale systems, networks, and directory services |
| http://www.skills-1st.co.uk/ +44 1628 782565 |
-----------------------------------------------------------------------
11:06 a.m.
Le 03/09/2010 17:18, Andrew Findlay a écrit :
On Fri, Sep 03, 2010 at 04:35:24PM +0200, Jonathan CLARKE wrote:
> DB_LOCK_DEADLOCK errors are only a warning: retries should occur until the
> operation completes. Of course, if they can be avoided, best avoid!
>
>> Question: is this topology sensible? If it is expected to work I
>> will gather some debug data for an ITS. If not, I will have to
>> drop back to plan B...
>
> This is an interesting configuration. I would not have proceeded like this
> but, as Marc Patermann suggested, I would set up a virtual address that
> points to the currently available master, and configure one syncrepl clause
> using this address (and all other LDAP clients, in fact). Could this
> approach work for you?
That is what I have done now, and it does work. I am still a little
uncertain about it though: when the normal server fails and the DNS
entry or routing changes to point to the hot standby, will this
confuse the consumer slapd? We are effectively telling it that the
second machine *is* the original one, yet it will have a different
serverID and possibly different contents.
Actually, I just set up a few servers to test this out.
I don't have any problems using the 2 syncrepl statements side-by-side
on the slave. When one master goes offline, replication continues from
the other, etc.
Could your problem be due to an unrelated configuration problem?
If not, in the hypothesis that the simultaneous-ness of changes is
causing problems, maybe try the seqmod overlay? (random idea, I don't
know this overlay very well)
My testing using one syncrepl statement for a single virtual address
also works fine in general (replication picks up where it left off),
except in one case:
- The slave server has a newer CSN for one of the serverIDs than the
master it's talking to. In this case, replication just fails with a
"LDAP_RES_SEARCH_RESULT" message.
Of course, this case can only occur if a modification was made, let's
say, on master1, and master2 didn't replicate it before master1 became
unavailable, and then master2 was then promoted to use the virtual
address (despite it not being up-to-date with master1). But still...
The reason for this error is that the syncprov overlay on master2
detects that one of the slave's CSNs is newer than it's own (the first
in this case), and closes the persistent search (syncprov.c, label
bailout:), even though the other CSN could be older, and thus syncprov
could provide updates.
I'm not sure if this can be considered a bug, but I think so. However,
what to do in this case, from syncprov's point of view, is unclear to me...
> Using two syncrepl statements is certainly suboptimal, as all
modifications
> will be replicated twice to all read-only servers. However, I don't see any
> reason why it shouldn't work, off the top of my head. Does slapd end up
> synchronizing everything?
Not sure - there were only 25000 entries but I gave up and stopped the
consumer server after 30 minutes as it still had not synchronised.
Good point about the double replication, though if it had worked
cleanly it would be OK in the (low modification) environment that I
have. The advantage is that nothing else is needed to manage the
failover / fail-back cases.
Makes sense, and seems like rather an attractive architecture.
Jonathan
--
==========================================
Jonathan CLARKE
------------------------------------------
Normation
44 rue Cauchy, 94110 Arcueil, France
------------------------------------------
Telephone: +33 (0)1 83 62 26 96
------------------------------------------
Web: http://www.normation.com/
==========================================
9:42 a.m.
On Fri, Sep 03, 2010 at 08:06:31PM +0200, Jonathan CLARKE wrote:
I don't have any problems using the 2 syncrepl statements
side-by-side on
the slave. When one master goes offline, replication continues from the
other, etc.
I have done some more tests with a new slave machine having two
syncrepl clauses to talk to two mirrormode masters. This time I
pre-loaded the database from a recent LDIF backup and sync was achieved
in a few seconds. There were a few grumbles of the form:
Sep 10 17:05:41 nis2 slapd[6571]: conn=-1 op=0: attribute "mail" index delete
failure
Sep 10 17:05:42 nis2 slapd[6571]: => bdb_idl_delete_key: c_get failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
I suppose these might lead to trouble later, but what is more worrying
is that I then ran slapcat backups of the two masters and the new
slave, and there is a difference. The two masters have identical data,
but...
The slave has one entry where pwdAccountLockedTime is missing. This
is an account that was locked by admin action between the initial load
data being dumped from the master server and the new slave being started
up, so it should have been replicated from one master or the other by
syncrepl. Every other attribute in the entry is identical, including the
modifyTimestamp which records when the pwdAccountLockedTime attribute
was added. I know that the entry did not change after that, as I have
a full changelog on both masters.
Andrew
--
-----------------------------------------------------------------------
| From Andrew Findlay, Skills 1st Ltd |
| Consultant in large-scale systems, networks, and directory services |
| http://www.skills-1st.co.uk/ +44 1628 782565 |
-----------------------------------------------------------------------
1 a.m.
Hi Andrew,
On 10/09/2010 18:42, Andrew Findlay wrote:
On Fri, Sep 03, 2010 at 08:06:31PM +0200, Jonathan CLARKE wrote:
> I don't have any problems using the 2 syncrepl statements side-by-side on
> the slave. When one master goes offline, replication continues from the
> other, etc.
I have done some more tests with a new slave machine having two
syncrepl clauses to talk to two mirrormode masters. This time I
pre-loaded the database from a recent LDIF backup and sync was achieved
in a few seconds. There were a few grumbles of the form:
Sep 10 17:05:41 nis2 slapd[6571]: conn=-1 op=0: attribute "mail" index delete
failure
Sep 10 17:05:42 nis2 slapd[6571]: => bdb_idl_delete_key: c_get failed:
DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
I suppose these might lead to trouble later, but what is more worrying
is that I then ran slapcat backups of the two masters and the new
slave, and there is a difference. The two masters have identical data,
but...
The slave has one entry where pwdAccountLockedTime is missing. This
is an account that was locked by admin action between the initial load
data being dumped from the master server and the new slave being started
up, so it should have been replicated from one master or the other by
syncrepl. Every other attribute in the entry is identical, including the
modifyTimestamp which records when the pwdAccountLockedTime attribute
was added. I know that the entry did not change after that, as I have
a full changelog on both masters.
This is most likely a separate issue. Updates to the ppolicy operational
attributes are not replicated like "standard" changes, but instead
written directly into the local database. So it's to be expected that
you see differences on these attributes between syncrepl
consumers/providers.
See the ppolicy_forward_updates option in slapo-ppolicy(5) for details
and a possible workaround.
Jonathan
--
==========================================
Jonathan CLARKE
------------------------------------------
Normation
44 rue Cauchy, 94110 Arcueil, France
------------------------------------------
Telephone: +33 (0)1 83 62 26 96
Mobile: +33 (0)6 99 60 03 10
------------------------------------------
Web: http://www.normation.com/
==========================================
9:07 a.m.
On Wed, Sep 22, 2010 at 10:00:29AM +0200, Jonathan CLARKE wrote:
> The slave has one entry where pwdAccountLockedTime is missing.
This
> is an account that was locked by admin action between the initial load
> data being dumped from the master server and the new slave being started
> up, so it should have been replicated from one master or the other by
> syncrepl. Every other attribute in the entry is identical, including the
> modifyTimestamp which records when the pwdAccountLockedTime attribute
> was added. I know that the entry did not change after that, as I have
> a full changelog on both masters.
This is most likely a separate issue. Updates to the ppolicy operational
attributes are not replicated like "standard" changes, but instead written
directly into the local database. So it's to be expected that you see
differences on these attributes between syncrepl consumers/providers.
That is true for updates that result from local Bind operations
(recording password failures, lockouts due to password failures etc).
In my case the missing update reflected administrative action taken
on the master server, and thus it should have propagated. The
modifyTimestamp did propagate, but the actual admin action did not...
See the ppolicy_forward_updates option in slapo-ppolicy(5) for
details and
a possible workaround.
That only applies to a Bind-induced change propagating against the
normal flow of replication. My case was the reverse. Unfortunately the
machines concerned are now in production service so it will be hard to
replicate the circumstances.
Andrew
--
-----------------------------------------------------------------------
| From Andrew Findlay, Skills 1st Ltd |
| Consultant in large-scale systems, networks, and directory services |
| http://www.skills-1st.co.uk/ +44 1628 782565 |
-----------------------------------------------------------------------
4564
days inactive
4584
days old
openldap-technical@openldap.org
8 comments
3 participants
participants (3)
-
Andrew Findlay -
Jonathan CLARKE -
Marc Patermann