--On Wednesday, April 14, 2010 8:47 AM +0900 Jorgen Lundman lundman@lundman.net wrote:

Perhaps it is time to go to a newer version of OpenLDAP. The 2.3.41 version was recommended to us about 3 years ago on this list. What is the recommended/most-stable version at the moment? Should we also upgrade BerkeleyDB?

The latest, most stable release is always found at:

http://www.openldap.org/software/download/

OpenLDAP 2.4 requires BDB 4.4 or later. The only reliable replication mechanism in OpenLDAP 2.3 is using delta-syncrepl.

You can also read http://www.openldap.org/software/release/changes.html and search for "syncrepl" and "syncprov" to see all the fixes to it since OpenLDAP 2.3.

--Quanah

--

Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

Jorgen Lundman wrote:

Jorgen Lundman wrote:

...

openldap-2.3.41 db-4.2.52.NC-PLUS_5_PATCHES Solaris 10 x86

/var/log/slaplog.20100413.gz:Apr 13 15:07:31 ldapslave03.unix slapd[27475]: [ID 561622 local4.debug] syncrepl_del_nonpresent: rid 329 be_delete DNSHostName=www,DNSZoneName=$customer.com,ou=dns,dc=$DC (66)

Just information sharing, no current issues...

I spun up some XENs worlds, one for master and one for client, for my own test case.

I did the operations of adding a DNS entry A-TYPE to a zone ("roger.test-this.com" in this case), then deleting said record and watching the sync log output of the slave.

Any number of these operations, add->del->add->del, appeared to work correctly.

However, when I did:

1) add "roger.test-this.com" 2) stop slapd on slave. 3) delete "roger.test-this.com" 4) start slapd on slave.

I get the following entries: (I only start slapd, no other operations)

May 11 14:35:47 ldapslave00.unix slapd[28256]: [ID 542995 local4.debug] slapd sh utdown: waiting for 1 threads to terminate May 11 14:35:48 ldapslave00.unix slapd[28256]: [ID 486161 local4.debug] slapd st opped. May 11 14:36:15 ldapslave00.unix slapd[28264]: [ID 702911 local4.debug] @(#) $Op enLDAP: slapd 2.3.41 (May 7 2010 11:24:55) $ May 11 14:36:15 ldapslave00.unix slapd[28265]: [ID 100111 local4.debug] slapd st arting May 11 14:36:16 ldapslave00.unix slapd[28265]: [ID 804257 local4.debug] do_syncr ep2: rid 279 LDAP_RES_INTERMEDIATE - SYNC_ID_SET May 11 14:36:16 ldapslave00.unix slapd[28265]: [ID 561622 local4.debug] syncrepl _del_nonpresent: rid 279 be_delete DNSHostName=roger,DNSZoneName=test-this.com,o u=dns,$DC (66)

So I have a test-case to work with.

As a side note, I do notice that if I then further executed:

add "www.test-this.com"

.. the slave ignores it, then ..

del "www.test-this.com"

the slave syncs this fine and we are back in business and consistent again.

I started working on upgrade path, I picked:

db-4.8.30 openldap-2.4.21

I first upgraded the LDAP master. I tried various techniques, since I can't use db_upgrade directly. But the new 2.4.21 is happy to sync against the old 2.3.41, so I think I will simply run it in parallel until I am sure it has caught up. (You guys have a clever way to test and know when LDAP syncrepl has caught up?)

With LDAP master running 2.4.21, and LDAP slave running 2.3.41, the test-case still works. Ie, it still breaks.

I then also upgraded LDAP slave to 2.4.21 and I can confirm the problem no longer happens. So it has indeed already been fixed. And I have an upgrade path to follow...

You should use the supported tools to migrate your database -- slapcat the master, then slapadd that to the replica... Same with rebuilding the master. That is the only officially supported upgrade path from 2.3 -> 2.4.

I did try that too. 'slapcat' took 6 minutes, and 'slapadd' to an empty directory wanted 15hours. If I move the ldif file to the x4500 where I have 16GB of /tmp, I can 'slapadd' it all in 13 minutes, then rsync the data back to the master and disk (3 minutes).

I just thought the whole rsyncing the binary db files to be 'uglier' than having 2.4 syncrepl against a 2.3 until it was up to date.

Lund

Sounds to me like you haven't properly tuned your database then. I suggest you read up on properly tuning a DB_CONFIG file (and possibly using a shared memory key as part of that), and I'm guessing you didn't use the -q flag with slapadd?

That is entirely possible, I'm entirely self-taught on LDAP.

I have the DB_CONFIG lines in slapd.conf so that they are automatically put into DB_CONFIG file. I just assumed that slapadd would do that. (Just tested, and it does).

I did not use -q.

# time slapadd -q -f slapd-import.conf -l ../full-dump.txt bdb_monitor_db_open: monitoring disabled; configure monitor database to enable *## 10.41% eta 49m44s elapsed 05m46s spd 1.5 M/s

That could be done in a maintenance window of 2 hours at very least.

Is then the official OpenLDAP policy is to do that, even though having 2.4 syncrepl from 2.3 would 'mean less than 5 minutes down-time' ? Clearly that part is attractive to me at 3am..

For amusement sake, /tmp import on the x4540:

-## 10.08% eta 10m24s elapsed 01m09s spd 2.0 M/s

And the DB_CONFIG settings:

dbconfig set_lk_detect DB_LOCK_DEFAULT dbconfig set_lg_max 52428800 dbconfig set_cachesize 0 67108864 1 dbconfig set_flags db_log_autoremove dbconfig set_lk_max_objects 1500 dbconfig set_lk_max_locks 1500 dbconfig set_lk_max_lockers 1500

...

dbconfig set_lk_detect DB_LOCK_DEFAULT dbconfig set_lg_max 52428800 dbconfig set_cachesize 0 67108864 1 dbconfig set_flags db_log_autoremove dbconfig set_lk_max_objects 1500 dbconfig set_lk_max_locks 1500 dbconfig set_lk_max_lockers 1500

You don't provide relevant data on the size of your database to make any comment one way or the other on whether your "set_cachesize" value is correct.

I would suggest you provide the output of du -c -h *.bdb in your OpenLDAP 2.3 database directory.

Coming right up! I appreciate your time in doing some sanity checks for me.

2.8M DNSData.bdb 3.9M DNSHostName.bdb 2.4M DNSIPAddr.bdb 1.4M DNSType.bdb 1.5M accountStatus.bdb 258K amavisSpamLover.bdb 642K deliveryMode.bdb 1.2G dn2id.bdb 62M entryCSN.bdb 84M entryUUID.bdb 13M gecos.bdb 898K gidNumber.bdb 2.5G id2entry.bdb 9.0M mail.bdb 7.6M mailAlternateAddress.bdb 5.6M o.bdb 3.9M objectClass.bdb 1.0M radiusGroupName.bdb 14M uid.bdb 12M uidNumber.bdb (no -c in Solaris, the whole dir is: ) 9.0G .

If it is at all interesting, on the live master (not the test server with identical data) the running data look like:

# /usr/local/BerkeleyDB.4.2/bin/db_stat -m 80MB 2KB 912B Total cache size. 1 Number of caches. 80MB 8KB Pool individual cache size. 0 Requested pages mapped into the process' address space. 163M Requested pages found in the cache (98%). 3324522 Requested pages not found in the cache. 25800 Pages created in the cache. 3324522 Pages read into the cache. 1893069 Pages written from the cache to the backing file. 3333040 Clean pages forced from the cache. 832 Dirty pages forced from the cache. 0 Dirty pages written by trickle-sync thread. 16450 Current total page count. 16303 Current clean page count. 147 Current dirty page count. 8191 Number of hash buckets used for page location. 169M Total number of times hash chains searched for a page. 14 The longest hash chain searched for a page. 383M Total number of hash buckets examined for page location. 479M The number of hash bucket locks granted without waiting. 4563 The number of hash bucket locks granted after waiting. 62 The maximum number of times any hash bucket lock was waited for. 12M The number of region locks granted without waiting. 11491 The number of region locks granted after waiting. 3350398 The number of page allocations. 6704823 The number of hash buckets examined during allocations 4161 The max number of hash buckets examined for an allocation 3333872 The number of pages examined during allocations 2064 The max number of pages examined for an allocation

Also, master LDAP slapd.conf has this cache related line:

lastmod on checkpoint 128 15 cachesize 5000

Now, live slapd consumes about 800MB of memory when it is running, which is probably acceptable as the server is dedicated to LDAP master. (Has 2GB RAM).