----- ldap@mm.st wrote:

I am rebuilding our aging pre 2.2 openldap servers that ran ldbm backend and slurpd. We ran this setup without any issues for many years.

The new setup is: RH5 openldap 2.3.43 (Stock RH) bdb backend 4.4.20 (Stock RH) Entries in db- about 1820 LDIF file is about 1.2M Memory- Master 4GB Slave 2GB (will add two more slaves)

Database section of slapd.conf: database bdb suffix "o=example.com" rootdn "cn=root">cn=root">cn=root">cn=root,o=example.com" rootpw {SSHA} ..... cachesize 1900 checkpoint 512 30 directory /var/lib/ldap index objectClass,uid,uidNumber,gidNumber,memberUid,uniqueMember

eq index cn,mail,surname,givenname

eq,subinitial

DB_CONFIG: set_cachesize 0 4153344 1 set_lk_max_objects 1500 set_lk_max_locks 1500 set_lk_max_lockers 1500 set_lg_regionmax 1048576 set_lg_bsize 32768 set_lg_max 131072 set_lg_dir /var/lib/ldap set_flags DB_LOG_AUTOREMOVE

This new setup appeared to work great for the last 10 days or so. I was able to authenticate clients, add records etc. Running slapd_db_stat -m and slapd_db_stat -c seem to indicate everything was ok. Before I put this setup into production, I got slurpd to function. Then decided to disable slurpd to use syncrepl in refreshonly mode. This also seemed to work fine. I'm not sure if the replication started this or not, but wanted to include all the events that let up to this.

Replication should not be related at all.

I have started to get: bdb(o=example.com): PANIC: fatal region error detected; run recovery On both servers at different times. During this time slapd continues to run which seems to confuse clients that try to use it and they will not try the other server that is listed in ldap.conf. To recover I did: service ldap stop, slapd_db_recover -h /var/lib/ldap, service ldap start.

I then commented all the replication stuff out in the slapd.conf and restarted ldap. It will run for a while (varies 5 minutes - ?) then I get the same errors and clients are unable to authenticate. On one of the servers I deleted all the files (except DB_CONFIG) and did a slapadd of a ldif file that I generated every night (without stopping slapd).

You imported while slapd was running? This is a recipe for failure. You can import to a different directory, stop slapd, and switch directories, and then start, but importing in the directory while slapd is running is a bad idea.

Same results once I started slapd again. I have enabled debug for slapd and have not seen anything different, I attached gdb to the running slapd and no errors are noted. I even copied a backup copy of slapd.conf prior to the replication settings (even though they are commented out) thinking that maybe something in there was causing it..

Then after several recoveries as described above the systems seem to be working again. One has not generated the error for for over 5.5 hours the other has not had any problems for 2 hours. For some reason after that period when the errors showed up for a while, things seem to be working again, at least for now.

I'm nervous about putting this into production until I can get this to function properly without these issues. During the 10 day period with everything working good, the slave would occasional (rarely) get the error and I would do a recovery, but we thought this was due to possible hardware problems. Now I'm not so sure.

I have a monitor script that runs slapd_db_stat -m and -c every 5 minutes and nothing seems wrong there, I far as I can tell. I'm hoping someone can help me determine possible causes or things to look at.

I would recommend that any server that hasn't had a clean import while slapd is *NOT* running, get one. Run it for a few days, and see if you see any problems.

I have been running 2.3.43 for years (my own packages on RHEL4, then my own packages on RHEL5, now some boxes run the RHEL packages), and never seen *this* issue, with a *much* larger directory (with about 8 replicas, though not all replicas have all databases).

Usually database corruption is due to hardware failure, unclean shutdown, or finger trouble.

Regards, Buchan