I am updating an OpenLDAP installation at present, and one of the improvements is to introduce a pair of master servers running in mirror mode. There will be several read-only servers as well, and I would like those to replicate from whichever master is currently available.
I did the obvious thing, and put two syncrepl clauses in the read-only server's config - one for each master server.
Starting with an empty database, the read-only server chews CPU badly and generates thousands of log messages like this:
Sep 2 14:57:34 nis0 slapd[9257]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994) Sep 2 14:57:38 nis0 slapd[9257]: => bdb_idl_delete_key: c_del id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994) Sep 2 14:57:38 nis0 slapd[9257]: conn=-1 op=0: attribute "entryCSN" index delete failure Sep 2 14:57:38 nis0 slapd[9257]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994) Sep 2 14:57:38 nis0 slapd[9257]: => bdb_idl_delete_key: c_del id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994) Sep 2 14:57:38 nis0 slapd[9257]: conn=-1 op=0: attribute "entryCSN" index delete failure .... Sep 2 14:56:10 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994) Sep 2 14:56:10 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994) Sep 2 14:56:11 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994) Sep 2 14:56:11 nis0 slapd[4953]: conn=-1 op=0: attribute "localAttribute" index add failure Sep 2 14:56:11 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994) Sep 2 14:56:11 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
It is very likely that each syncrepl connection will be retrieving the same entries at about the same time, so there is clearly potential for collisions.
Question: is this topology sensible? If it is expected to work I will gather some debug data for an ITS. If not, I will have to drop back to plan B...
OpenLDAP 2.4.22 with BerkeleyDB.4.8 running on SLES 10.3 i586
Thanks
Andrew
Andrew,
Andrew Findlay schrieb am 02.09.2010 16:27 Uhr:
I am updating an OpenLDAP installation at present, and one of the improvements is to introduce a pair of master servers running in mirror mode. There will be several read-only servers as well, and I would like those to replicate from whichever master is currently available.
I did the obvious thing, and put two syncrepl clauses in the read-only server's config - one for each master server.
I'm not sure, this is the "obvious". :) I'm not sure either, if it is possible to define more than one provider in the syncrepl option. The best would be to use a unified service address and a single name for both servers.
Marc
On Fri, Sep 03, 2010 at 01:41:32PM +0200, Marc Patermann wrote:
I did the obvious thing, and put two syncrepl clauses in the read-only server's config - one for each master server.
I'm not sure, this is the "obvious". :) I'm not sure either, if it is possible to define more than one provider in the syncrepl option.
I don't think it is, but what I did was to create two syncrepl options: that is certainly supported as it is needed for multi-master setups.
The best would be to use a unified service address and a single name for both servers.
That is one of the various 'plan Bs'....
Andrew
Hi Andrew,
Le 02/09/2010 16:27, Andrew Findlay a écrit :
I am updating an OpenLDAP installation at present, and one of the improvements is to introduce a pair of master servers running in mirror mode. There will be several read-only servers as well, and I would like those to replicate from whichever master is currently available.
I did the obvious thing, and put two syncrepl clauses in the read-only server's config - one for each master server.
Starting with an empty database, the read-only server chews CPU badly and generates thousands of log messages like this:
Sep 2 14:57:34 nis0 slapd[9257]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994) Sep 2 14:57:38 nis0 slapd[9257]: => bdb_idl_delete_key: c_del id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994) Sep 2 14:57:38 nis0 slapd[9257]: conn=-1 op=0: attribute "entryCSN" index delete failure Sep 2 14:57:38 nis0 slapd[9257]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994) Sep 2 14:57:38 nis0 slapd[9257]: => bdb_idl_delete_key: c_del id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994) Sep 2 14:57:38 nis0 slapd[9257]: conn=-1 op=0: attribute "entryCSN" index delete failure .... Sep 2 14:56:10 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994) Sep 2 14:56:10 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994) Sep 2 14:56:11 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994) Sep 2 14:56:11 nis0 slapd[4953]: conn=-1 op=0: attribute "localAttribute" index add failure Sep 2 14:56:11 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994) Sep 2 14:56:11 nis0 slapd[4953]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
It is very likely that each syncrepl connection will be retrieving the same entries at about the same time, so there is clearly potential for collisions.
DB_LOCK_DEADLOCK errors are only a warning: retries should occur until the operation completes. Of course, if they can be avoided, best avoid!
Question: is this topology sensible? If it is expected to work I will gather some debug data for an ITS. If not, I will have to drop back to plan B...
This is an interesting configuration. I would not have proceeded like this but, as Marc Patermann suggested, I would set up a virtual address that points to the currently available master, and configure one syncrepl clause using this address (and all other LDAP clients, in fact). Could this approach work for you?
Looking at the OpenLDAP configuration issue, I note your remark that several syncrepl statements are used in multi-master setups. I'm not entirely sure why this works better, but it may well be due to all servers having the syncprov overlay, which serializes modifications when a persistent search is in progress.
Using two syncrepl statements is certainly suboptimal, as all modifications will be replicated twice to all read-only servers. However, I don't see any reason why it shouldn't work, off the top of my head. Does slapd end up synchronizing everything?
Jonathan
On Fri, Sep 03, 2010 at 04:35:24PM +0200, Jonathan CLARKE wrote:
DB_LOCK_DEADLOCK errors are only a warning: retries should occur until the operation completes. Of course, if they can be avoided, best avoid!
Question: is this topology sensible? If it is expected to work I will gather some debug data for an ITS. If not, I will have to drop back to plan B...
This is an interesting configuration. I would not have proceeded like this but, as Marc Patermann suggested, I would set up a virtual address that points to the currently available master, and configure one syncrepl clause using this address (and all other LDAP clients, in fact). Could this approach work for you?
That is what I have done now, and it does work. I am still a little uncertain about it though: when the normal server fails and the DNS entry or routing changes to point to the hot standby, will this confuse the consumer slapd? We are effectively telling it that the second machine *is* the original one, yet it will have a different serverID and possibly different contents.
Looking at the OpenLDAP configuration issue, I note your remark that several syncrepl statements are used in multi-master setups. I'm not entirely sure why this works better, but it may well be due to all servers having the syncprov overlay, which serializes modifications when a persistent search is in progress.
Sounds plausible.
Using two syncrepl statements is certainly suboptimal, as all modifications will be replicated twice to all read-only servers. However, I don't see any reason why it shouldn't work, off the top of my head. Does slapd end up synchronizing everything?
Not sure - there were only 25000 entries but I gave up and stopped the consumer server after 30 minutes as it still had not synchronised.
Good point about the double replication, though if it had worked cleanly it would be OK in the (low modification) environment that I have. The advantage is that nothing else is needed to manage the failover / fail-back cases.
Andrew
Le 03/09/2010 17:18, Andrew Findlay a écrit :
On Fri, Sep 03, 2010 at 04:35:24PM +0200, Jonathan CLARKE wrote:
DB_LOCK_DEADLOCK errors are only a warning: retries should occur until the operation completes. Of course, if they can be avoided, best avoid!
Question: is this topology sensible? If it is expected to work I will gather some debug data for an ITS. If not, I will have to drop back to plan B...
This is an interesting configuration. I would not have proceeded like this but, as Marc Patermann suggested, I would set up a virtual address that points to the currently available master, and configure one syncrepl clause using this address (and all other LDAP clients, in fact). Could this approach work for you?
That is what I have done now, and it does work. I am still a little uncertain about it though: when the normal server fails and the DNS entry or routing changes to point to the hot standby, will this confuse the consumer slapd? We are effectively telling it that the second machine *is* the original one, yet it will have a different serverID and possibly different contents.
Actually, I just set up a few servers to test this out.
I don't have any problems using the 2 syncrepl statements side-by-side on the slave. When one master goes offline, replication continues from the other, etc.
Could your problem be due to an unrelated configuration problem? If not, in the hypothesis that the simultaneous-ness of changes is causing problems, maybe try the seqmod overlay? (random idea, I don't know this overlay very well)
My testing using one syncrepl statement for a single virtual address also works fine in general (replication picks up where it left off), except in one case: - The slave server has a newer CSN for one of the serverIDs than the master it's talking to. In this case, replication just fails with a "LDAP_RES_SEARCH_RESULT" message.
Of course, this case can only occur if a modification was made, let's say, on master1, and master2 didn't replicate it before master1 became unavailable, and then master2 was then promoted to use the virtual address (despite it not being up-to-date with master1). But still...
The reason for this error is that the syncprov overlay on master2 detects that one of the slave's CSNs is newer than it's own (the first in this case), and closes the persistent search (syncprov.c, label bailout:), even though the other CSN could be older, and thus syncprov could provide updates.
I'm not sure if this can be considered a bug, but I think so. However, what to do in this case, from syncprov's point of view, is unclear to me...
Using two syncrepl statements is certainly suboptimal, as all modifications will be replicated twice to all read-only servers. However, I don't see any reason why it shouldn't work, off the top of my head. Does slapd end up synchronizing everything?
Not sure - there were only 25000 entries but I gave up and stopped the consumer server after 30 minutes as it still had not synchronised.
Good point about the double replication, though if it had worked cleanly it would be OK in the (low modification) environment that I have. The advantage is that nothing else is needed to manage the failover / fail-back cases.
Makes sense, and seems like rather an attractive architecture.
Jonathan
On Fri, Sep 03, 2010 at 08:06:31PM +0200, Jonathan CLARKE wrote:
I don't have any problems using the 2 syncrepl statements side-by-side on the slave. When one master goes offline, replication continues from the other, etc.
I have done some more tests with a new slave machine having two syncrepl clauses to talk to two mirrormode masters. This time I pre-loaded the database from a recent LDIF backup and sync was achieved in a few seconds. There were a few grumbles of the form:
Sep 10 17:05:41 nis2 slapd[6571]: conn=-1 op=0: attribute "mail" index delete failure Sep 10 17:05:42 nis2 slapd[6571]: => bdb_idl_delete_key: c_get failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
I suppose these might lead to trouble later, but what is more worrying is that I then ran slapcat backups of the two masters and the new slave, and there is a difference. The two masters have identical data, but...
The slave has one entry where pwdAccountLockedTime is missing. This is an account that was locked by admin action between the initial load data being dumped from the master server and the new slave being started up, so it should have been replicated from one master or the other by syncrepl. Every other attribute in the entry is identical, including the modifyTimestamp which records when the pwdAccountLockedTime attribute was added. I know that the entry did not change after that, as I have a full changelog on both masters.
Andrew
Hi Andrew,
On 10/09/2010 18:42, Andrew Findlay wrote:
On Fri, Sep 03, 2010 at 08:06:31PM +0200, Jonathan CLARKE wrote:
I don't have any problems using the 2 syncrepl statements side-by-side on the slave. When one master goes offline, replication continues from the other, etc.
I have done some more tests with a new slave machine having two syncrepl clauses to talk to two mirrormode masters. This time I pre-loaded the database from a recent LDIF backup and sync was achieved in a few seconds. There were a few grumbles of the form:
Sep 10 17:05:41 nis2 slapd[6571]: conn=-1 op=0: attribute "mail" index delete failure Sep 10 17:05:42 nis2 slapd[6571]: => bdb_idl_delete_key: c_get failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30994)
I suppose these might lead to trouble later, but what is more worrying is that I then ran slapcat backups of the two masters and the new slave, and there is a difference. The two masters have identical data, but...
The slave has one entry where pwdAccountLockedTime is missing. This is an account that was locked by admin action between the initial load data being dumped from the master server and the new slave being started up, so it should have been replicated from one master or the other by syncrepl. Every other attribute in the entry is identical, including the modifyTimestamp which records when the pwdAccountLockedTime attribute was added. I know that the entry did not change after that, as I have a full changelog on both masters.
This is most likely a separate issue. Updates to the ppolicy operational attributes are not replicated like "standard" changes, but instead written directly into the local database. So it's to be expected that you see differences on these attributes between syncrepl consumers/providers.
See the ppolicy_forward_updates option in slapo-ppolicy(5) for details and a possible workaround.
Jonathan
On Wed, Sep 22, 2010 at 10:00:29AM +0200, Jonathan CLARKE wrote:
The slave has one entry where pwdAccountLockedTime is missing. This is an account that was locked by admin action between the initial load data being dumped from the master server and the new slave being started up, so it should have been replicated from one master or the other by syncrepl. Every other attribute in the entry is identical, including the modifyTimestamp which records when the pwdAccountLockedTime attribute was added. I know that the entry did not change after that, as I have a full changelog on both masters.
This is most likely a separate issue. Updates to the ppolicy operational attributes are not replicated like "standard" changes, but instead written directly into the local database. So it's to be expected that you see differences on these attributes between syncrepl consumers/providers.
That is true for updates that result from local Bind operations (recording password failures, lockouts due to password failures etc). In my case the missing update reflected administrative action taken on the master server, and thus it should have propagated. The modifyTimestamp did propagate, but the actual admin action did not...
See the ppolicy_forward_updates option in slapo-ppolicy(5) for details and a possible workaround.
That only applies to a Bind-induced change propagating against the normal flow of replication. My case was the reverse. Unfortunately the machines concerned are now in production service so it will be hard to replicate the circumstances.
Andrew
openldap-technical@openldap.org
-
Andrew Findlay -
Jonathan CLARKE -
Marc Patermann