Problem: For a customer we use LDAP for many years. Last year suddenly the slapd service just stopped without any traces in the logfiles. After a restart of slapd everything works fine again. But the problem was there: it was not an incident, now and then slapd just stops and always without any traces in the logfiles. Sometime three times a day, sometime a week without a failure. I can't find a pattern or any relation to any other service on the linux server.
Environment: - Several (debian squeeze) servers , several windows servers. We use bdb database backend. - There is one master LDAP server which provides syncprov and two replica's LDAP servers (syncrepl). The master server is most intens used (mainly samba as primary domain controller: a few hundred useraccounts, lot of groupaccounts, workstations, acl's, etc.), one of the replica's is not very busy but handles the mail for all users (lookup: amavis, postfix, courier-imap, mailaccount settings etc). The third replica is not busy at all, it is a remote location. - Total LDAP is 3700 dn's, slapcat produces a file of 7,3 Mb. - It is only the master LDAP with stops suddenly. I have never seen a failure of a replica LDAP.
Because I have no clear idea about the problem I have no idea which technical details are relevant: DB_CONFIG =========== set_cachesize 0 10485760 1 set_lk_max_objects 10000 set_lk_max_locks 10000 set_lk_max_lockers 10000 set_lg_dir /home/ldap-dbd The database is stored on a ext3 filesystem, kernel 2.6.32. The server has no problems, plenty of memory and a fast diskarray (SAS->SATA). Never technical problems with this server. And it worked without problems for a long period. Nothing has changed to the environment or the LDAP setup (except of course with the upgrade to debian squeeze but the problem was already there).
What we have tried: - upgrade from openldap 2..4.17 (debian lenny+backports) to openldap 2.4.23 (debian squeeze). I saw in the release notes that problems related to syncrepl were solved. Therefor we waited for version 2.4.23 te become available in debian. This upgrade made no difference. - reindex, rebuilt the directory. When I rebuilt the LDAP with a clean LDIF file on the master LDAP or an other machine with ldapadd there is not one error or warning.
The workaround for the moment: I have written a process monitor (perl daemon) which monitors the slapd daemon and if it suddenly stops, slapd is restarted. It is of course not a solution but the 300 user can work. If slapd stops without a restart within 1 minute a few hundred people can't work because samba stops working.
I would like to receive suggestions what we can do to find the problem. Because there is no pattern, nothing in the logfiles I don't know where to start.