We have a 3-way multimaster configuration running on CentOS 5.11, OpenLDAP 2.4.40. All three have been up for years, until the other day:
Slapd is running on two of the three (server names: ldapserver1, ldapserver2, and ldapserver3). Slapd stopped and won't restart on ldapserver2.
From Logs on ldapserver2:
May 10 04:02:13 gp42-admin4 slapd[4541]: slapd shutdown: waiting for 0 operations/tasks to finish
May 10 04:02:19 gp42-admin4 slapd[15633]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2. 4.40/servers/slapd
May 10 04:02:19 gp42-admin4 slapd[15633]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 10 04:02:19 gp42-admin4 slapd[15633]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 10 04:02:21 gp42-admin4 slapd[15634]: bdb_db_open: database "cn=accesslog": database already in use.
May 10 04:02:21 gp42-admin4 slapd[15634]: backend_startup_one (type=bdb, suffix="cn=accesslog"): bi_db_open failed! (-1)
May 10 04:02:21 gp42-admin4 slapd[15634]: slapd stopped.
May 10 04:02:22 gp42-admin4 slapd[4541]: slapd stopped.
When attempting to restart slapd on server2:
May 13 10:13:54 gp42-admin4 slapd[12085]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2. 4.40/servers/slapd
May 13 10:13:54 gp42-admin4 slapd[12085]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:13:54 gp42-admin4 slapd[12085]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:13:56 gp42-admin4 slapd[12086]: slapd starting
May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=002 (4096) Content Sync Refresh Required
May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=001 (4096) Content Sync Refresh Required
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_add 0xfc6: parent (cn=accesslog) insert failed: -30995
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_delete_key: c_del id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_delete 0xf50: parent (cn=accesslog) delete failed: -30995
May 13 10:15:55 gp42-admin4 slapd[12106]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2. 4.40/servers/slapd
May 13 10:15:55 gp42-admin4 slapd[12106]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:15:55 gp42-admin4 slapd[12106]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:15:55 gp42-admin4 slapd[12106]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:15:57 gp42-admin4 slapd[12106]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:15:58 gp42-admin4 slapd[12106]: slapd starting
May 13 10:28:49 gp42-admin4 slapd[12255]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2. 4.40/servers/slapd
May 13 10:28:49 gp42-admin4 slapd[12255]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:28:49 gp42-admin4 slapd[12255]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database "dc=example,dc=com": unclean shutdown detected; attempting recovery.
May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:28:52 gp42-admin4 slapd[12255]: slapd starting
May 13 10:29:24 gp42-admin4 slapd[12264]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2. 4.40/servers/slapd
May 13 10:29:24 gp42-admin4 slapd[12264]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:29:24 gp42-admin4 slapd[12264]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:29:24 gp42-admin4 slapd[12264]: slapd starting
May 13 10:29:53 gp42-admin4 slapd[12280]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2. 4.40/servers/slapd
May 13 10:29:53 gp42-admin4 slapd[12280]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:29:53 gp42-admin4 slapd[12280]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:29:53 gp42-admin4 slapd[12280]: slapd starting
May 13 10:32:35 gp42-admin4 slapd[12345]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2. 4.40/servers/slapd
Attempting to restart slapd from the command-line:
5735ed50 slapd starting
5735ed50 => bdb_entry_get: ndn: "cn=accesslog"
5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
5735ed50 bdb_idl_fetch_key: %cn=accesslog
5735ed50 bdb_idl_fetch_key: [b49d1940]
5735ed50 bdb_idl_fetch_key:
5735ed50 send_ldap_result: err=0 matched="" text=""
5735ed50 => bdb_entry_get: ndn: "dc=example,dc=com"
5735ed50 => bdb_entry_get: oc: "(null)", at: "contextCSN"
ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN
ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
5735ed50 => bdb_entry_get: ndn: "uid=jdoe,ou=Users,dc=example,dc=com"
5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
Aborted
I have run db_recover on the dbase(s) on ldapserver2 but to no avail.
Does anyone have any suggestions?
Thank you in advance for any assistance.
John D. Borresen (Dave)
Linux/Unix Systems Administrator
MIT Lincoln Laboratory
Humanitarian Assistance and Disaster Relief (HADR) Systems
244 Wood St
Lexington, MA 02420
Email: mailto:john.borresen@ll.mit.edu john.borresen@ll.mit.edu
I've noticed this error/warning that keeps standing out when starting slapd on ldapserver2:
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
In my google searches I found this post from Quanah as a possible bug in 2.4.44 (we're running 2.4.40):
This list is for discussing reported issues in OpenLDAP Software ()
headers
quanah | 27 Apr 16:56 2016
(ITS#8413) Assertion in back-mdb/search.c during replication
Full_Name: Quanah Gibson-Mount
Version: 2.4.44
OS: Linux
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (75.111.52.177)
During replication from the accesslog DB, in a 4-way MMR setup, various masters
periodically crash with slapd: search.c:1246: oc_filter: Assertion `f != ((void
*)0)' failed.
This is back-mdb/search.c, not slapd-search.c
This is triggered when a NULL filter is passed through. However, it should be
impossible for the filter generated by str2filter to ever fail.
Permalink | Reply |
Navigate
Go to gmane.network.openldap.bugs.
Topic
Go to the topic.
Advertisement
Project Web Page
This list is for discussing reported issues in OpenLDAP Software ()
Search Archive
Language
Change language
Options
Current view: Threads only / Showing whole messages / Not hiding cited text.
Change to All messages, shortened messages, or hide cited text.
Post a message
NNTP Newsgroup
Classic Gmane web interface
XML RSS Feed
List Information
About Gmane
Gmane
Again, if anyone has any suggestions as to a workaround or a resolution that would be most appreciative.
Thanks,
John D. Borresen (Dave)
Email: john.borresen@ll.mit.edu
From: openldap-technical [mailto:openldap-technical-bounces@openldap.org] On Behalf Of Borresen, John - 0444 - MITLL Sent: Friday, May 13, 2016 11:13 AM To: openldap-technical@openldap.org Subject: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
We have a 3-way multimaster configuration running on CentOS 5.11, OpenLDAP 2.4.40. All three have been up for years, until the other day:
Slapd is running on two of the three (server names: ldapserver1, ldapserver2, and ldapserver3). Slapd stopped and won't restart on ldapserver2.
From Logs on ldapserver2:
May 10 04:02:13 gp42-admin4 slapd[4541]: slapd shutdown: waiting for 0 operations/tasks to finish
May 10 04:02:19 gp42-admin4 slapd[15633]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2. 4.40/servers/slapd
May 10 04:02:19 gp42-admin4 slapd[15633]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 10 04:02:19 gp42-admin4 slapd[15633]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 10 04:02:21 gp42-admin4 slapd[15634]: bdb_db_open: database "cn=accesslog": database already in use.
May 10 04:02:21 gp42-admin4 slapd[15634]: backend_startup_one (type=bdb, suffix="cn=accesslog"): bi_db_open failed! (-1)
May 10 04:02:21 gp42-admin4 slapd[15634]: slapd stopped.
May 10 04:02:22 gp42-admin4 slapd[4541]: slapd stopped.
When attempting to restart slapd on server2:
May 13 10:13:54 gp42-admin4 slapd[12085]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2. 4.40/servers/slapd
May 13 10:13:54 gp42-admin4 slapd[12085]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:13:54 gp42-admin4 slapd[12085]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:13:56 gp42-admin4 slapd[12086]: slapd starting
May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=002 (4096) Content Sync Refresh Required
May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=001 (4096) Content Sync Refresh Required
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_add 0xfc6: parent (cn=accesslog) insert failed: -30995
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_delete_key: c_del id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_delete 0xf50: parent (cn=accesslog) delete failed: -30995
May 13 10:15:55 gp42-admin4 slapd[12106]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2. 4.40/servers/slapd
May 13 10:15:55 gp42-admin4 slapd[12106]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:15:55 gp42-admin4 slapd[12106]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:15:55 gp42-admin4 slapd[12106]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:15:57 gp42-admin4 slapd[12106]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:15:58 gp42-admin4 slapd[12106]: slapd starting
May 13 10:28:49 gp42-admin4 slapd[12255]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2. 4.40/servers/slapd
May 13 10:28:49 gp42-admin4 slapd[12255]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:28:49 gp42-admin4 slapd[12255]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database "dc=example,dc=com": unclean shutdown detected; attempting recovery.
May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:28:52 gp42-admin4 slapd[12255]: slapd starting
May 13 10:29:24 gp42-admin4 slapd[12264]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2. 4.40/servers/slapd
May 13 10:29:24 gp42-admin4 slapd[12264]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:29:24 gp42-admin4 slapd[12264]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:29:24 gp42-admin4 slapd[12264]: slapd starting
May 13 10:29:53 gp42-admin4 slapd[12280]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2. 4.40/servers/slapd
May 13 10:29:53 gp42-admin4 slapd[12280]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:29:53 gp42-admin4 slapd[12280]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:29:53 gp42-admin4 slapd[12280]: slapd starting
May 13 10:32:35 gp42-admin4 slapd[12345]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2. 4.40/servers/slapd
Attempting to restart slapd from the command-line:
5735ed50 slapd starting
5735ed50 => bdb_entry_get: ndn: "cn=accesslog"
5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
5735ed50 bdb_idl_fetch_key: %cn=accesslog
5735ed50 bdb_idl_fetch_key: [b49d1940]
5735ed50 bdb_idl_fetch_key:
5735ed50 send_ldap_result: err=0 matched="" text=""
5735ed50 => bdb_entry_get: ndn: "dc=example,dc=com"
5735ed50 => bdb_entry_get: oc: "(null)", at: "contextCSN"
ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN
ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
5735ed50 => bdb_entry_get: ndn: "uid=jdoe,ou=Users,dc=example,dc=com"
5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
Aborted
I have run db_recover on the dbase(s) on ldapserver2 but to no avail.
Does anyone have any suggestions?
Thank you in advance for any assistance.
John D. Borresen (Dave)
Linux/Unix Systems Administrator
MIT Lincoln Laboratory
Humanitarian Assistance and Disaster Relief (HADR) Systems
244 Wood St
Lexington, MA 02420
Email: john.borresen@ll.mit.edu
That was specific to back-mdb. Your logs showed corruption with BDB. Are you using mdb, bdb, or both?
--Quanah
--On Monday, May 16, 2016 3:22 PM +0000 "Borresen, John - 0444 - MITLL" John.Borresen@ll.mit.edu wrote:
I've noticed this error/warning that keeps standing out when starting slapd on ldapserver2:
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
In my google searches I found this post from Quanah as a possible bug in 2.4.44 (we're running 2.4.40):
This list is for discussing reported issues in OpenLDAP Software ()
headers
quanah | 27 Apr 16:56 2016
(ITS#8413) Assertion in back-mdb/search.c during replication
Full_Name: Quanah Gibson-Mount
Version: 2.4.44
OS: Linux
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (75.111.52.177)
During replication from the accesslog DB, in a 4-way MMR setup, various masters
periodically crash with slapd: search.c:1246: oc_filter: Assertion `f != ((void
*)0)' failed.
This is back-mdb/search.c, not slapd-search.c
This is triggered when a NULL filter is passed through. However, it should be
impossible for the filter generated by str2filter to ever fail.
Permalink | Reply |
Navigate
Go to gmane.network.openldap.bugs.
Topic
Go to the topic.
Advertisement
Project Web Page
This list is for discussing reported issues in OpenLDAP Software ()
Search Archive
Language
Change language
Options
Current view: Threads only / Showing whole messages / Not hiding cited text.
Change to All messages, shortened messages, or hide cited text.
Post a message
NNTP Newsgroup
Classic Gmane web interface
XML RSS Feed
List Information
About Gmane
Gmane
Again, if anyone has any suggestions as to a workaround or a resolution that would be most appreciative.
Thanks,
John D. Borresen (Dave)
Email: john.borresen@ll.mit.edu
From: openldap-technical [mailto:openldap-technical-bounces@openldap.org] On Behalf Of Borresen, John - 0444 - MITLL Sent: Friday, May 13, 2016 11:13 AM To: openldap-technical@openldap.org Subject: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
We have a 3-way multimaster configuration running on CentOS 5.11, OpenLDAP 2.4.40. All three have been up for years, until the other day:
Slapd is running on two of the three (server names: ldapserver1, ldapserver2, and ldapserver3). Slapd stopped and won't restart on ldapserver2.
From Logs on ldapserver2:
May 10 04:02:13 gp42-admin4 slapd[4541]: slapd shutdown: waiting for 0 operations/tasks to finish
May 10 04:02:19 gp42-admin4 slapd[15633]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap -2.4.40/servers/slapd
May 10 04:02:19 gp42-admin4 slapd[15633]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 10 04:02:19 gp42-admin4 slapd[15633]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 10 04:02:21 gp42-admin4 slapd[15634]: bdb_db_open: database "cn=accesslog": database already in use.
May 10 04:02:21 gp42-admin4 slapd[15634]: backend_startup_one (type=bdb, suffix="cn=accesslog"): bi_db_open failed! (-1)
May 10 04:02:21 gp42-admin4 slapd[15634]: slapd stopped.
May 10 04:02:22 gp42-admin4 slapd[4541]: slapd stopped.
When attempting to restart slapd on server2:
May 13 10:13:54 gp42-admin4 slapd[12085]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap -2.4.40/servers/slapd
May 13 10:13:54 gp42-admin4 slapd[12085]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:13:54 gp42-admin4 slapd[12085]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:13:56 gp42-admin4 slapd[12086]: slapd starting
May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=002 (4096) Content Sync Refresh Required
May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=001 (4096) Content Sync Refresh Required
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_add 0xfc6: parent (cn=accesslog) insert failed: -30995
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_delete_key: c_del id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_delete 0xf50: parent (cn=accesslog) delete failed: -30995
May 13 10:15:55 gp42-admin4 slapd[12106]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap -2.4.40/servers/slapd
May 13 10:15:55 gp42-admin4 slapd[12106]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:15:55 gp42-admin4 slapd[12106]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:15:55 gp42-admin4 slapd[12106]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:15:57 gp42-admin4 slapd[12106]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:15:58 gp42-admin4 slapd[12106]: slapd starting
May 13 10:28:49 gp42-admin4 slapd[12255]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap -2.4.40/servers/slapd
May 13 10:28:49 gp42-admin4 slapd[12255]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:28:49 gp42-admin4 slapd[12255]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database "dc=example,dc=com": unclean shutdown detected; attempting recovery.
May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:28:52 gp42-admin4 slapd[12255]: slapd starting
May 13 10:29:24 gp42-admin4 slapd[12264]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap -2.4.40/servers/slapd
May 13 10:29:24 gp42-admin4 slapd[12264]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:29:24 gp42-admin4 slapd[12264]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:29:24 gp42-admin4 slapd[12264]: slapd starting
May 13 10:29:53 gp42-admin4 slapd[12280]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap -2.4.40/servers/slapd
May 13 10:29:53 gp42-admin4 slapd[12280]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:29:53 gp42-admin4 slapd[12280]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:29:53 gp42-admin4 slapd[12280]: slapd starting
May 13 10:32:35 gp42-admin4 slapd[12345]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openldap -2.4.40/servers/slapd
Attempting to restart slapd from the command-line:
5735ed50 slapd starting
5735ed50 => bdb_entry_get: ndn: "cn=accesslog"
5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
5735ed50 bdb_idl_fetch_key: %cn=accesslog
5735ed50 bdb_idl_fetch_key: [b49d1940]
5735ed50 bdb_idl_fetch_key:
5735ed50 send_ldap_result: err=0 matched="" text=""
5735ed50 => bdb_entry_get: ndn: "dc=example,dc=com"
5735ed50 => bdb_entry_get: oc: "(null)", at: "contextCSN"
ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN
ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
5735ed50 => bdb_entry_get: ndn: "uid=jdoe,ou=Users,dc=example,dc=com"
5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
Aborted
I have run db_recover on the dbase(s) on ldapserver2 but to no avail.
Does anyone have any suggestions?
Thank you in advance for any assistance.
John D. Borresen (Dave)
Linux/Unix Systems Administrator
MIT Lincoln Laboratory
Humanitarian Assistance and Disaster Relief (HADR) Systems
244 Wood St
Lexington, MA 02420
Email: john.borresen@ll.mit.edu
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
We're using bdb.
John D. Borresen (Dave) Ph: (781) 981-1609 Email: john.borresen@ll.mit.edu
-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Monday, May 16, 2016 11:55 AM To: Borresen, John - 0444 - MITLL; openldap-technical@openldap.org Subject: Re: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
That was specific to back-mdb. Your logs showed corruption with BDB. Are you using mdb, bdb, or both?
--Quanah
--On Monday, May 16, 2016 3:22 PM +0000 "Borresen, John - 0444 - MITLL" John.Borresen@ll.mit.edu wrote:
I've noticed this error/warning that keeps standing out when starting slapd on ldapserver2:
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
In my google searches I found this post from Quanah as a possible bug in 2.4.44 (we're running 2.4.40):
This list is for discussing reported issues in OpenLDAP Software ()
headers
quanah | 27 Apr 16:56 2016
(ITS#8413) Assertion in back-mdb/search.c during replication
Full_Name: Quanah Gibson-Mount
Version: 2.4.44
OS: Linux
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (75.111.52.177)
During replication from the accesslog DB, in a 4-way MMR setup, various masters
periodically crash with slapd: search.c:1246: oc_filter: Assertion `f != ((void
*)0)' failed.
This is back-mdb/search.c, not slapd-search.c
This is triggered when a NULL filter is passed through. However, it should be
impossible for the filter generated by str2filter to ever fail.
Permalink | Reply |
Navigate
Go to gmane.network.openldap.bugs.
Topic
Go to the topic.
Advertisement
Project Web Page
This list is for discussing reported issues in OpenLDAP Software ()
Search Archive
Language
Change language
Options
Current view: Threads only / Showing whole messages / Not hiding cited text.
Change to All messages, shortened messages, or hide cited text.
Post a message
NNTP Newsgroup
Classic Gmane web interface
XML RSS Feed
List Information
About Gmane
Gmane
Again, if anyone has any suggestions as to a workaround or a resolution that would be most appreciative.
Thanks,
John D. Borresen (Dave)
Email: john.borresen@ll.mit.edu
From: openldap-technical [mailto:openldap-technical-bounces@openldap.org] On Behalf Of Borresen, John - 0444 - MITLL Sent: Friday, May 13, 2016 11:13 AM To: openldap-technical@openldap.org Subject: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
We have a 3-way multimaster configuration running on CentOS 5.11, OpenLDAP 2.4.40. All three have been up for years, until the other day:
Slapd is running on two of the three (server names: ldapserver1, ldapserver2, and ldapserver3). Slapd stopped and won't restart on ldapserver2.
From Logs on ldapserver2:
May 10 04:02:13 gp42-admin4 slapd[4541]: slapd shutdown: waiting for 0 operations/tasks to finish
May 10 04:02:19 gp42-admin4 slapd[15633]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 10 04:02:19 gp42-admin4 slapd[15633]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 10 04:02:19 gp42-admin4 slapd[15633]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 10 04:02:21 gp42-admin4 slapd[15634]: bdb_db_open: database "cn=accesslog": database already in use.
May 10 04:02:21 gp42-admin4 slapd[15634]: backend_startup_one (type=bdb, suffix="cn=accesslog"): bi_db_open failed! (-1)
May 10 04:02:21 gp42-admin4 slapd[15634]: slapd stopped.
May 10 04:02:22 gp42-admin4 slapd[4541]: slapd stopped.
When attempting to restart slapd on server2:
May 13 10:13:54 gp42-admin4 slapd[12085]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 13 10:13:54 gp42-admin4 slapd[12085]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:13:54 gp42-admin4 slapd[12085]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:13:56 gp42-admin4 slapd[12086]: slapd starting
May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=002 (4096) Content Sync Refresh Required
May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=001 (4096) Content Sync Refresh Required
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_add 0xfc6: parent (cn=accesslog) insert failed: -30995
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_delete_key: c_del id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_delete 0xf50: parent (cn=accesslog) delete failed: -30995
May 13 10:15:55 gp42-admin4 slapd[12106]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 13 10:15:55 gp42-admin4 slapd[12106]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:15:55 gp42-admin4 slapd[12106]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:15:55 gp42-admin4 slapd[12106]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:15:57 gp42-admin4 slapd[12106]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:15:58 gp42-admin4 slapd[12106]: slapd starting
May 13 10:28:49 gp42-admin4 slapd[12255]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 13 10:28:49 gp42-admin4 slapd[12255]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:28:49 gp42-admin4 slapd[12255]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database "dc=example,dc=com": unclean shutdown detected; attempting recovery.
May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:28:52 gp42-admin4 slapd[12255]: slapd starting
May 13 10:29:24 gp42-admin4 slapd[12264]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 13 10:29:24 gp42-admin4 slapd[12264]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:29:24 gp42-admin4 slapd[12264]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:29:24 gp42-admin4 slapd[12264]: slapd starting
May 13 10:29:53 gp42-admin4 slapd[12280]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 13 10:29:53 gp42-admin4 slapd[12280]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:29:53 gp42-admin4 slapd[12280]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:29:53 gp42-admin4 slapd[12280]: slapd starting
May 13 10:32:35 gp42-admin4 slapd[12345]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
Attempting to restart slapd from the command-line:
5735ed50 slapd starting
5735ed50 => bdb_entry_get: ndn: "cn=accesslog"
5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
5735ed50 bdb_idl_fetch_key: %cn=accesslog
5735ed50 bdb_idl_fetch_key: [b49d1940]
5735ed50 bdb_idl_fetch_key:
5735ed50 send_ldap_result: err=0 matched="" text=""
5735ed50 => bdb_entry_get: ndn: "dc=example,dc=com"
5735ed50 => bdb_entry_get: oc: "(null)", at: "contextCSN"
ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN
ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
5735ed50 => bdb_entry_get: ndn: "uid=jdoe,ou=Users,dc=example,dc=com"
5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
Aborted
I have run db_recover on the dbase(s) on ldapserver2 but to no avail.
Does anyone have any suggestions?
Thank you in advance for any assistance.
John D. Borresen (Dave)
Linux/Unix Systems Administrator
MIT Lincoln Laboratory
Humanitarian Assistance and Disaster Relief (HADR) Systems
244 Wood St
Lexington, MA 02420
Email: john.borresen@ll.mit.edu
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
I've run db_recover on ldapserver2, to no avail:
# db_recover -v -h /var/lib/openldap/accesslog Finding last valid log LSN: file: 5 offset 4659816 Recovery starting from [5][4655494] Recovery complete at Mon May 16 13:03:19 2016 Maximum transaction ID 8000000c Recovery checkpoint [5][4660438]
# db_recover -v -h /var/lib/openldap/openldap-data Finding last valid log LSN: file: 3 offset 7169862 Recovery starting from [3][7169734] Recovery complete at Mon May 16 13:03:33 2016 Maximum transaction ID 800035d9 Recovery checkpoint [3][7169862]
Same error: # /usr/local/openldap/libexec/slapd -u ldap -h ldap://ldapserver2.example.come -F /usr/local/openldap/etc/openldap/slapd.d -d 256 5739fe11 @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $ clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2.4.40/serv ers/slapd 5739fe11 bdb_db_open: database "dc=group42,dc=ldap": unclean shutdown detected; attempting recovery. 5739fe11 bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery. 5739fe11 slapd starting slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed. Aborted
I found references to remove *.bdb in the accesslog and openldap-data directories and the __db.*. Could I run a slapcat of the dbase on ldapserver1 and copy that over to ldapserver2? Is that a viable option?
John D. Borresen (Dave) Ph: (781) 981-1609 Email: john.borresen@ll.mit.edu
-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Monday, May 16, 2016 11:55 AM To: Borresen, John - 0444 - MITLL; openldap-technical@openldap.org Subject: Re: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
That was specific to back-mdb. Your logs showed corruption with BDB. Are you using mdb, bdb, or both?
--Quanah
--On Monday, May 16, 2016 3:22 PM +0000 "Borresen, John - 0444 - MITLL" John.Borresen@ll.mit.edu wrote:
I've noticed this error/warning that keeps standing out when starting slapd on ldapserver2:
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
In my google searches I found this post from Quanah as a possible bug in 2.4.44 (we're running 2.4.40):
This list is for discussing reported issues in OpenLDAP Software ()
headers
quanah | 27 Apr 16:56 2016
(ITS#8413) Assertion in back-mdb/search.c during replication
Full_Name: Quanah Gibson-Mount
Version: 2.4.44
OS: Linux
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (75.111.52.177)
During replication from the accesslog DB, in a 4-way MMR setup, various masters
periodically crash with slapd: search.c:1246: oc_filter: Assertion `f != ((void
*)0)' failed.
This is back-mdb/search.c, not slapd-search.c
This is triggered when a NULL filter is passed through. However, it should be
impossible for the filter generated by str2filter to ever fail.
Permalink | Reply |
Navigate
Go to gmane.network.openldap.bugs.
Topic
Go to the topic.
Advertisement
Project Web Page
This list is for discussing reported issues in OpenLDAP Software ()
Search Archive
Language
Change language
Options
Current view: Threads only / Showing whole messages / Not hiding cited text.
Change to All messages, shortened messages, or hide cited text.
Post a message
NNTP Newsgroup
Classic Gmane web interface
XML RSS Feed
List Information
About Gmane
Gmane
Again, if anyone has any suggestions as to a workaround or a resolution that would be most appreciative.
Thanks,
John D. Borresen (Dave)
Email: john.borresen@ll.mit.edu
From: openldap-technical [mailto:openldap-technical-bounces@openldap.org] On Behalf Of Borresen, John - 0444 - MITLL Sent: Friday, May 13, 2016 11:13 AM To: openldap-technical@openldap.org Subject: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
We have a 3-way multimaster configuration running on CentOS 5.11, OpenLDAP 2.4.40. All three have been up for years, until the other day:
Slapd is running on two of the three (server names: ldapserver1, ldapserver2, and ldapserver3). Slapd stopped and won't restart on ldapserver2.
From Logs on ldapserver2:
May 10 04:02:13 gp42-admin4 slapd[4541]: slapd shutdown: waiting for 0 operations/tasks to finish
May 10 04:02:19 gp42-admin4 slapd[15633]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 10 04:02:19 gp42-admin4 slapd[15633]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 10 04:02:19 gp42-admin4 slapd[15633]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 10 04:02:21 gp42-admin4 slapd[15634]: bdb_db_open: database "cn=accesslog": database already in use.
May 10 04:02:21 gp42-admin4 slapd[15634]: backend_startup_one (type=bdb, suffix="cn=accesslog"): bi_db_open failed! (-1)
May 10 04:02:21 gp42-admin4 slapd[15634]: slapd stopped.
May 10 04:02:22 gp42-admin4 slapd[4541]: slapd stopped.
When attempting to restart slapd on server2:
May 13 10:13:54 gp42-admin4 slapd[12085]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 13 10:13:54 gp42-admin4 slapd[12085]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:13:54 gp42-admin4 slapd[12085]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:13:56 gp42-admin4 slapd[12086]: slapd starting
May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=002 (4096) Content Sync Refresh Required
May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=001 (4096) Content Sync Refresh Required
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_add 0xfc6: parent (cn=accesslog) insert failed: -30995
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_delete_key: c_del id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_delete 0xf50: parent (cn=accesslog) delete failed: -30995
May 13 10:15:55 gp42-admin4 slapd[12106]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 13 10:15:55 gp42-admin4 slapd[12106]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:15:55 gp42-admin4 slapd[12106]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:15:55 gp42-admin4 slapd[12106]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:15:57 gp42-admin4 slapd[12106]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:15:58 gp42-admin4 slapd[12106]: slapd starting
May 13 10:28:49 gp42-admin4 slapd[12255]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 13 10:28:49 gp42-admin4 slapd[12255]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:28:49 gp42-admin4 slapd[12255]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database "dc=example,dc=com": unclean shutdown detected; attempting recovery.
May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:28:52 gp42-admin4 slapd[12255]: slapd starting
May 13 10:29:24 gp42-admin4 slapd[12264]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 13 10:29:24 gp42-admin4 slapd[12264]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:29:24 gp42-admin4 slapd[12264]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:29:24 gp42-admin4 slapd[12264]: slapd starting
May 13 10:29:53 gp42-admin4 slapd[12280]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 13 10:29:53 gp42-admin4 slapd[12280]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:29:53 gp42-admin4 slapd[12280]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:29:53 gp42-admin4 slapd[12280]: slapd starting
May 13 10:32:35 gp42-admin4 slapd[12345]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
Attempting to restart slapd from the command-line:
5735ed50 slapd starting
5735ed50 => bdb_entry_get: ndn: "cn=accesslog"
5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
5735ed50 bdb_idl_fetch_key: %cn=accesslog
5735ed50 bdb_idl_fetch_key: [b49d1940]
5735ed50 bdb_idl_fetch_key:
5735ed50 send_ldap_result: err=0 matched="" text=""
5735ed50 => bdb_entry_get: ndn: "dc=example,dc=com"
5735ed50 => bdb_entry_get: oc: "(null)", at: "contextCSN"
ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN
ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
5735ed50 => bdb_entry_get: ndn: "uid=jdoe,ou=Users,dc=example,dc=com"
5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
Aborted
I have run db_recover on the dbase(s) on ldapserver2 but to no avail.
Does anyone have any suggestions?
Thank you in advance for any assistance.
John D. Borresen (Dave)
Linux/Unix Systems Administrator
MIT Lincoln Laboratory
Humanitarian Assistance and Disaster Relief (HADR) Systems
244 Wood St
Lexington, MA 02420
Email: john.borresen@ll.mit.edu
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
Actually, never mind, it is not specific to mdb, I was thinking of the wrong thing. You'll have to build OpenLDAP 2.4.44 + that fix to recover.
--Quanah
--On Monday, May 16, 2016 6:21 PM +0000 "Borresen, John - 0444 - MITLL" John.Borresen@ll.mit.edu wrote:
I've run db_recover on ldapserver2, to no avail:
# db_recover -v -h /var/lib/openldap/accesslog Finding last valid log LSN: file: 5 offset 4659816 Recovery starting from [5][4655494] Recovery complete at Mon May 16 13:03:19 2016 Maximum transaction ID 8000000c Recovery checkpoint [5][4660438]
# db_recover -v -h /var/lib/openldap/openldap-data Finding last valid log LSN: file: 3 offset 7169862 Recovery starting from [3][7169734] Recovery complete at Mon May 16 13:03:33 2016 Maximum transaction ID 800035d9 Recovery checkpoint [3][7169862]
Same error: # /usr/local/openldap/libexec/slapd -u ldap -h ldap://ldapserver2.example.come -F /usr/local/openldap/etc/openldap/slapd.d -d 256 5739fe11 @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $
clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2.4.40/s erv ers/slapd 5739fe11 bdb_db_open: database "dc=group42,dc=ldap": unclean shutdown detected; attempting recovery. 5739fe11 bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery. 5739fe11 slapd starting slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed. Aborted
I found references to remove *.bdb in the accesslog and openldap-data directories and the __db.*. Could I run a slapcat of the dbase on ldapserver1 and copy that over to ldapserver2? Is that a viable option?
John D. Borresen (Dave) Ph: (781) 981-1609 Email: john.borresen@ll.mit.edu
-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Monday, May 16, 2016 11:55 AM To: Borresen, John - 0444 - MITLL; openldap-technical@openldap.org Subject: Re: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
That was specific to back-mdb. Your logs showed corruption with BDB. Are you using mdb, bdb, or both?
--Quanah
--On Monday, May 16, 2016 3:22 PM +0000 "Borresen, John - 0444 - MITLL" John.Borresen@ll.mit.edu wrote:
I've noticed this error/warning that keeps standing out when starting slapd on ldapserver2:
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
In my google searches I found this post from Quanah as a possible bug in 2.4.44 (we're running 2.4.40):
This list is for discussing reported issues in OpenLDAP Software ()
headers
quanah | 27 Apr 16:56 2016
(ITS#8413) Assertion in back-mdb/search.c during replication
Full_Name: Quanah Gibson-Mount
Version: 2.4.44
OS: Linux
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (75.111.52.177)
During replication from the accesslog DB, in a 4-way MMR setup, various masters
periodically crash with slapd: search.c:1246: oc_filter: Assertion `f != ((void
*)0)' failed.
This is back-mdb/search.c, not slapd-search.c
This is triggered when a NULL filter is passed through. However, it should be
impossible for the filter generated by str2filter to ever fail.
Permalink | Reply |
Navigate
Go to gmane.network.openldap.bugs.
Topic
Go to the topic.
Advertisement
Project Web Page
This list is for discussing reported issues in OpenLDAP Software ()
Search Archive
Language
Change language
Options
Current view: Threads only / Showing whole messages / Not hiding cited text.
Change to All messages, shortened messages, or hide cited text.
Post a message
NNTP Newsgroup
Classic Gmane web interface
XML RSS Feed
List Information
About Gmane
Gmane
Again, if anyone has any suggestions as to a workaround or a resolution that would be most appreciative.
Thanks,
John D. Borresen (Dave)
Email: john.borresen@ll.mit.edu
From: openldap-technical [mailto:openldap-technical-bounces@openldap.org] On Behalf Of Borresen, John - 0444 - MITLL Sent: Friday, May 13, 2016 11:13 AM To: openldap-technical@openldap.org Subject: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
We have a 3-way multimaster configuration running on CentOS 5.11, OpenLDAP 2.4.40. All three have been up for years, until the other day:
Slapd is running on two of the three (server names: ldapserver1, ldapserver2, and ldapserver3). Slapd stopped and won't restart on ldapserver2.
From Logs on ldapserver2:
May 10 04:02:13 gp42-admin4 slapd[4541]: slapd shutdown: waiting for 0 operations/tasks to finish
May 10 04:02:19 gp42-admin4 slapd[15633]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 10 04:02:19 gp42-admin4 slapd[15633]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 10 04:02:19 gp42-admin4 slapd[15633]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 10 04:02:21 gp42-admin4 slapd[15634]: bdb_db_open: database "cn=accesslog": database already in use.
May 10 04:02:21 gp42-admin4 slapd[15634]: backend_startup_one (type=bdb, suffix="cn=accesslog"): bi_db_open failed! (-1)
May 10 04:02:21 gp42-admin4 slapd[15634]: slapd stopped.
May 10 04:02:22 gp42-admin4 slapd[4541]: slapd stopped.
When attempting to restart slapd on server2:
May 13 10:13:54 gp42-admin4 slapd[12085]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 13 10:13:54 gp42-admin4 slapd[12085]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:13:54 gp42-admin4 slapd[12085]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:13:56 gp42-admin4 slapd[12086]: slapd starting
May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=002 (4096) Content Sync Refresh Required
May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=001 (4096) Content Sync Refresh Required
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_add 0xfc6: parent (cn=accesslog) insert failed: -30995
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_delete_key: c_del id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_delete 0xf50: parent (cn=accesslog) delete failed: -30995
May 13 10:15:55 gp42-admin4 slapd[12106]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 13 10:15:55 gp42-admin4 slapd[12106]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:15:55 gp42-admin4 slapd[12106]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:15:55 gp42-admin4 slapd[12106]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:15:57 gp42-admin4 slapd[12106]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:15:58 gp42-admin4 slapd[12106]: slapd starting
May 13 10:28:49 gp42-admin4 slapd[12255]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 13 10:28:49 gp42-admin4 slapd[12255]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:28:49 gp42-admin4 slapd[12255]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database "dc=example,dc=com": unclean shutdown detected; attempting recovery.
May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:28:52 gp42-admin4 slapd[12255]: slapd starting
May 13 10:29:24 gp42-admin4 slapd[12264]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 13 10:29:24 gp42-admin4 slapd[12264]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:29:24 gp42-admin4 slapd[12264]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:29:24 gp42-admin4 slapd[12264]: slapd starting
May 13 10:29:53 gp42-admin4 slapd[12280]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
May 13 10:29:53 gp42-admin4 slapd[12280]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:29:53 gp42-admin4 slapd[12280]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:29:53 gp42-admin4 slapd[12280]: slapd starting
May 13 10:32:35 gp42-admin4 slapd[12345]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/openl dap -2.4.40/servers/slapd
Attempting to restart slapd from the command-line:
5735ed50 slapd starting
5735ed50 => bdb_entry_get: ndn: "cn=accesslog"
5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
5735ed50 bdb_idl_fetch_key: %cn=accesslog
5735ed50 bdb_idl_fetch_key: [b49d1940]
5735ed50 bdb_idl_fetch_key:
5735ed50 send_ldap_result: err=0 matched="" text=""
5735ed50 => bdb_entry_get: ndn: "dc=example,dc=com"
5735ed50 => bdb_entry_get: oc: "(null)", at: "contextCSN"
ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN
ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
5735ed50 => bdb_entry_get: ndn: "uid=jdoe,ou=Users,dc=example,dc=com"
5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
Aborted
I have run db_recover on the dbase(s) on ldapserver2 but to no avail.
Does anyone have any suggestions?
Thank you in advance for any assistance.
John D. Borresen (Dave)
Linux/Unix Systems Administrator
MIT Lincoln Laboratory
Humanitarian Assistance and Disaster Relief (HADR) Systems
244 Wood St
Lexington, MA 02420
Email: john.borresen@ll.mit.edu
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc.
Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
I have upgraded to 2.4.44, ran:
# db_recover -v -c -h /var/lib/openldap/accesslog Finding last valid log LSN: file: 5 offset 4694358 Recovery starting from [4][28] Recovery complete at Mon May 16 15:12:49 2016 Maximum transaction ID 8000329e Recovery checkpoint [5][4694980]
Then attempted to start and received the following: 573a1bb7 bdb_db_open: database "dc=example,dc=com": unclean shutdown detected; attempting recovery. 573a1bb7 bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery. 573a1bb7 slapd starting slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed. Aborted
Same error(s) as before.
John D. Borresen (Dave) Ph: (781) 981-1609 Email: john.borresen@ll.mit.edu
-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Monday, May 16, 2016 1:47 PM To: Borresen, John - 0444 - MITLL; openldap-technical@openldap.org Subject: RE: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
Actually, never mind, it is not specific to mdb, I was thinking of the wrong thing. You'll have to build OpenLDAP 2.4.44 + that fix to recover.
--Quanah
--On Monday, May 16, 2016 6:21 PM +0000 "Borresen, John - 0444 - MITLL" John.Borresen@ll.mit.edu wrote:
I've run db_recover on ldapserver2, to no avail:
# db_recover -v -h /var/lib/openldap/accesslog Finding last valid log LSN: file: 5 offset 4659816 Recovery starting from [5][4655494] Recovery complete at Mon May 16 13:03:19 2016 Maximum transaction ID 8000000c Recovery checkpoint [5][4660438]
# db_recover -v -h /var/lib/openldap/openldap-data Finding last valid log LSN: file: 3 offset 7169862 Recovery starting from [3][7169734] Recovery complete at Mon May 16 13:03:33 2016 Maximum transaction ID 800035d9 Recovery checkpoint [3][7169862]
Same error: # /usr/local/openldap/libexec/slapd -u ldap -h ldap://ldapserver2.example.come -F /usr/local/openldap/etc/openldap/slapd.d -d 256 5739fe11 @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $
clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2.4.4 0/s erv ers/slapd 5739fe11 bdb_db_open: database "dc=group42,dc=ldap": unclean shutdown detected; attempting recovery. 5739fe11 bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery. 5739fe11 slapd starting slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed. Aborted
I found references to remove *.bdb in the accesslog and openldap-data directories and the __db.*. Could I run a slapcat of the dbase on ldapserver1 and copy that over to ldapserver2? Is that a viable option?
John D. Borresen (Dave) Ph: (781) 981-1609 Email: john.borresen@ll.mit.edu
-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Monday, May 16, 2016 11:55 AM To: Borresen, John - 0444 - MITLL; openldap-technical@openldap.org Subject: Re: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
That was specific to back-mdb. Your logs showed corruption with BDB. Are you using mdb, bdb, or both?
--Quanah
--On Monday, May 16, 2016 3:22 PM +0000 "Borresen, John - 0444 - MITLL" John.Borresen@ll.mit.edu wrote:
I've noticed this error/warning that keeps standing out when starting slapd on ldapserver2:
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
In my google searches I found this post from Quanah as a possible bug in 2.4.44 (we're running 2.4.40):
This list is for discussing reported issues in OpenLDAP Software ()
headers
quanah | 27 Apr 16:56 2016
(ITS#8413) Assertion in back-mdb/search.c during replication
Full_Name: Quanah Gibson-Mount
Version: 2.4.44
OS: Linux
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (75.111.52.177)
During replication from the accesslog DB, in a 4-way MMR setup, various masters
periodically crash with slapd: search.c:1246: oc_filter: Assertion `f != ((void
*)0)' failed.
This is back-mdb/search.c, not slapd-search.c
This is triggered when a NULL filter is passed through. However, it should be
impossible for the filter generated by str2filter to ever fail.
Permalink | Reply |
Navigate
Go to gmane.network.openldap.bugs.
Topic
Go to the topic.
Advertisement
Project Web Page
This list is for discussing reported issues in OpenLDAP Software ()
Search Archive
Language
Change language
Options
Current view: Threads only / Showing whole messages / Not hiding cited text.
Change to All messages, shortened messages, or hide cited text.
Post a message
NNTP Newsgroup
Classic Gmane web interface
XML RSS Feed
List Information
About Gmane
Gmane
Again, if anyone has any suggestions as to a workaround or a resolution that would be most appreciative.
Thanks,
John D. Borresen (Dave)
Email: john.borresen@ll.mit.edu
From: openldap-technical [mailto:openldap-technical-bounces@openldap.org] On Behalf Of Borresen, John - 0444 - MITLL Sent: Friday, May 13, 2016 11:13 AM To: openldap-technical@openldap.org Subject: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
We have a 3-way multimaster configuration running on CentOS 5.11, OpenLDAP 2.4.40. All three have been up for years, until the other day:
Slapd is running on two of the three (server names: ldapserver1, ldapserver2, and ldapserver3). Slapd stopped and won't restart on ldapserver2.
From Logs on ldapserver2:
May 10 04:02:13 gp42-admin4 slapd[4541]: slapd shutdown: waiting for 0 operations/tasks to finish
May 10 04:02:19 gp42-admin4 slapd[15633]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open l dap -2.4.40/servers/slapd
May 10 04:02:19 gp42-admin4 slapd[15633]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 10 04:02:19 gp42-admin4 slapd[15633]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 10 04:02:21 gp42-admin4 slapd[15634]: bdb_db_open: database "cn=accesslog": database already in use.
May 10 04:02:21 gp42-admin4 slapd[15634]: backend_startup_one (type=bdb, suffix="cn=accesslog"): bi_db_open failed! (-1)
May 10 04:02:21 gp42-admin4 slapd[15634]: slapd stopped.
May 10 04:02:22 gp42-admin4 slapd[4541]: slapd stopped.
When attempting to restart slapd on server2:
May 13 10:13:54 gp42-admin4 slapd[12085]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open l dap -2.4.40/servers/slapd
May 13 10:13:54 gp42-admin4 slapd[12085]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:13:54 gp42-admin4 slapd[12085]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:13:56 gp42-admin4 slapd[12086]: slapd starting
May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=002 (4096) Content Sync Refresh Required
May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=001 (4096) Content Sync Refresh Required
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_add 0xfc6: parent (cn=accesslog) insert failed: -30995
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_delete_key: c_del id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_delete 0xf50: parent (cn=accesslog) delete failed: -30995
May 13 10:15:55 gp42-admin4 slapd[12106]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open l dap -2.4.40/servers/slapd
May 13 10:15:55 gp42-admin4 slapd[12106]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:15:55 gp42-admin4 slapd[12106]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:15:55 gp42-admin4 slapd[12106]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:15:57 gp42-admin4 slapd[12106]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:15:58 gp42-admin4 slapd[12106]: slapd starting
May 13 10:28:49 gp42-admin4 slapd[12255]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open l dap -2.4.40/servers/slapd
May 13 10:28:49 gp42-admin4 slapd[12255]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:28:49 gp42-admin4 slapd[12255]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database "dc=example,dc=com": unclean shutdown detected; attempting recovery.
May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:28:52 gp42-admin4 slapd[12255]: slapd starting
May 13 10:29:24 gp42-admin4 slapd[12264]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open l dap -2.4.40/servers/slapd
May 13 10:29:24 gp42-admin4 slapd[12264]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:29:24 gp42-admin4 slapd[12264]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:29:24 gp42-admin4 slapd[12264]: slapd starting
May 13 10:29:53 gp42-admin4 slapd[12280]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open l dap -2.4.40/servers/slapd
May 13 10:29:53 gp42-admin4 slapd[12280]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:29:53 gp42-admin4 slapd[12280]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:29:53 gp42-admin4 slapd[12280]: slapd starting
May 13 10:32:35 gp42-admin4 slapd[12345]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open l dap -2.4.40/servers/slapd
Attempting to restart slapd from the command-line:
5735ed50 slapd starting
5735ed50 => bdb_entry_get: ndn: "cn=accesslog"
5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
5735ed50 bdb_idl_fetch_key: %cn=accesslog
5735ed50 bdb_idl_fetch_key: [b49d1940]
5735ed50 bdb_idl_fetch_key:
5735ed50 send_ldap_result: err=0 matched="" text=""
5735ed50 => bdb_entry_get: ndn: "dc=example,dc=com"
5735ed50 => bdb_entry_get: oc: "(null)", at: "contextCSN"
ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN
ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
5735ed50 => bdb_entry_get: ndn: "uid=jdoe,ou=Users,dc=example,dc=com"
5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
Aborted
I have run db_recover on the dbase(s) on ldapserver2 but to no avail.
Does anyone have any suggestions?
Thank you in advance for any assistance.
John D. Borresen (Dave)
Linux/Unix Systems Administrator
MIT Lincoln Laboratory
Humanitarian Assistance and Disaster Relief (HADR) Systems
244 Wood St
Lexington, MA 02420
Email: john.borresen@ll.mit.edu
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc.
Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
As I said:
You will need to build 2.4.44 PLUS THE PATCH from head.
--Quanah
--On Monday, May 16, 2016 8:16 PM +0000 "Borresen, John - 0444 - MITLL" John.Borresen@ll.mit.edu wrote:
I have upgraded to 2.4.44, ran:
# db_recover -v -c -h /var/lib/openldap/accesslog Finding last valid log LSN: file: 5 offset 4694358 Recovery starting from [4][28] Recovery complete at Mon May 16 15:12:49 2016 Maximum transaction ID 8000329e Recovery checkpoint [5][4694980]
Then attempted to start and received the following: 573a1bb7 bdb_db_open: database "dc=example,dc=com": unclean shutdown detected; attempting recovery. 573a1bb7 bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery. 573a1bb7 slapd starting slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed. Aborted
Same error(s) as before.
John D. Borresen (Dave) Ph: (781) 981-1609 Email: john.borresen@ll.mit.edu
-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Monday, May 16, 2016 1:47 PM To: Borresen, John - 0444 - MITLL; openldap-technical@openldap.org Subject: RE: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
Actually, never mind, it is not specific to mdb, I was thinking of the wrong thing. You'll have to build OpenLDAP 2.4.44 + that fix to recover.
--Quanah
--On Monday, May 16, 2016 6:21 PM +0000 "Borresen, John - 0444 - MITLL" John.Borresen@ll.mit.edu wrote:
I've run db_recover on ldapserver2, to no avail:
# db_recover -v -h /var/lib/openldap/accesslog Finding last valid log LSN: file: 5 offset 4659816 Recovery starting from [5][4655494] Recovery complete at Mon May 16 13:03:19 2016 Maximum transaction ID 8000000c Recovery checkpoint [5][4660438]
# db_recover -v -h /var/lib/openldap/openldap-data Finding last valid log LSN: file: 3 offset 7169862 Recovery starting from [3][7169734] Recovery complete at Mon May 16 13:03:33 2016 Maximum transaction ID 800035d9 Recovery checkpoint [3][7169862]
Same error: # /usr/local/openldap/libexec/slapd -u ldap -h ldap://ldapserver2.example.come -F /usr/local/openldap/etc/openldap/slapd.d -d 256 5739fe11 @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $
clement@localhost.localdomain:/home/clement/build/BUILD/openldap-2.4.4 0/s erv ers/slapd 5739fe11 bdb_db_open: database "dc=group42,dc=ldap": unclean shutdown detected; attempting recovery. 5739fe11 bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery. 5739fe11 slapd starting slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed. Aborted
I found references to remove *.bdb in the accesslog and openldap-data directories and the __db.*. Could I run a slapcat of the dbase on ldapserver1 and copy that over to ldapserver2? Is that a viable option?
John D. Borresen (Dave) Ph: (781) 981-1609 Email: john.borresen@ll.mit.edu
-----Original Message----- From: Quanah Gibson-Mount [mailto:quanah@zimbra.com] Sent: Monday, May 16, 2016 11:55 AM To: Borresen, John - 0444 - MITLL; openldap-technical@openldap.org Subject: Re: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
That was specific to back-mdb. Your logs showed corruption with BDB. Are you using mdb, bdb, or both?
--Quanah
--On Monday, May 16, 2016 3:22 PM +0000 "Borresen, John - 0444 - MITLL" John.Borresen@ll.mit.edu wrote:
I've noticed this error/warning that keeps standing out when starting slapd on ldapserver2:
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
In my google searches I found this post from Quanah as a possible bug in 2.4.44 (we're running 2.4.40):
This list is for discussing reported issues in OpenLDAP Software ()
headers
quanah | 27 Apr 16:56 2016
(ITS#8413) Assertion in back-mdb/search.c during replication
Full_Name: Quanah Gibson-Mount
Version: 2.4.44
OS: Linux
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (75.111.52.177)
During replication from the accesslog DB, in a 4-way MMR setup, various masters
periodically crash with slapd: search.c:1246: oc_filter: Assertion `f != ((void
*)0)' failed.
This is back-mdb/search.c, not slapd-search.c
This is triggered when a NULL filter is passed through. However, it should be
impossible for the filter generated by str2filter to ever fail.
Permalink | Reply |
Navigate
Go to gmane.network.openldap.bugs.
Topic
Go to the topic.
Advertisement
Project Web Page
This list is for discussing reported issues in OpenLDAP Software ()
Search Archive
Language
Change language
Options
Current view: Threads only / Showing whole messages / Not hiding cited text.
Change to All messages, shortened messages, or hide cited text.
Post a message
NNTP Newsgroup
Classic Gmane web interface
XML RSS Feed
List Information
About Gmane
Gmane
Again, if anyone has any suggestions as to a workaround or a resolution that would be most appreciative.
Thanks,
John D. Borresen (Dave)
Email: john.borresen@ll.mit.edu
From: openldap-technical [mailto:openldap-technical-bounces@openldap.org] On Behalf Of Borresen, John - 0444 - MITLL Sent: Friday, May 13, 2016 11:13 AM To: openldap-technical@openldap.org Subject: SLAPD WON'T START ON ONE OF THE MULTIMASTERS
We have a 3-way multimaster configuration running on CentOS 5.11, OpenLDAP 2.4.40. All three have been up for years, until the other day:
Slapd is running on two of the three (server names: ldapserver1, ldapserver2, and ldapserver3). Slapd stopped and won't restart on ldapserver2.
From Logs on ldapserver2:
May 10 04:02:13 gp42-admin4 slapd[4541]: slapd shutdown: waiting for 0 operations/tasks to finish
May 10 04:02:19 gp42-admin4 slapd[15633]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open l dap -2.4.40/servers/slapd
May 10 04:02:19 gp42-admin4 slapd[15633]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 10 04:02:19 gp42-admin4 slapd[15633]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 10 04:02:21 gp42-admin4 slapd[15634]: bdb_db_open: database "cn=accesslog": database already in use.
May 10 04:02:21 gp42-admin4 slapd[15634]: backend_startup_one (type=bdb, suffix="cn=accesslog"): bi_db_open failed! (-1)
May 10 04:02:21 gp42-admin4 slapd[15634]: slapd stopped.
May 10 04:02:22 gp42-admin4 slapd[4541]: slapd stopped.
When attempting to restart slapd on server2:
May 13 10:13:54 gp42-admin4 slapd[12085]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open l dap -2.4.40/servers/slapd
May 13 10:13:54 gp42-admin4 slapd[12085]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:13:54 gp42-admin4 slapd[12085]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:13:56 gp42-admin4 slapd[12086]: slapd starting
May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=002 (4096) Content Sync Refresh Required
May 13 10:13:56 gp42-admin4 slapd[12086]: do_syncrep2: rid=001 (4096) Content Sync Refresh Required
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_insert_key: c_put id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_add 0xfc6: parent (cn=accesslog) insert failed: -30995
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_idl_delete_key: c_del id failed: DB_LOCK_DEADLOCK: Locker killed to resolve a deadlock (-30995)
May 13 10:13:57 gp42-admin4 slapd[12086]: => bdb_dn2id_delete 0xf50: parent (cn=accesslog) delete failed: -30995
May 13 10:15:55 gp42-admin4 slapd[12106]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open l dap -2.4.40/servers/slapd
May 13 10:15:55 gp42-admin4 slapd[12106]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:15:55 gp42-admin4 slapd[12106]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:15:55 gp42-admin4 slapd[12106]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:15:57 gp42-admin4 slapd[12106]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:15:58 gp42-admin4 slapd[12106]: slapd starting
May 13 10:28:49 gp42-admin4 slapd[12255]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open l dap -2.4.40/servers/slapd
May 13 10:28:49 gp42-admin4 slapd[12255]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:28:49 gp42-admin4 slapd[12255]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database "dc=example,dc=com": unclean shutdown detected; attempting recovery.
May 13 10:28:50 gp42-admin4 slapd[12255]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:28:52 gp42-admin4 slapd[12255]: slapd starting
May 13 10:29:24 gp42-admin4 slapd[12264]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open l dap -2.4.40/servers/slapd
May 13 10:29:24 gp42-admin4 slapd[12264]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:29:24 gp42-admin4 slapd[12264]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:29:24 gp42-admin4 slapd[12264]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:29:24 gp42-admin4 slapd[12264]: slapd starting
May 13 10:29:53 gp42-admin4 slapd[12280]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open l dap -2.4.40/servers/slapd
May 13 10:29:53 gp42-admin4 slapd[12280]: nss-ldap: do_open: do_start_tls failed:stat=-1
May 13 10:29:53 gp42-admin4 slapd[12280]: nss_ldap: reconnected to LDAP server ldap://ldapserver1.example.com
May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database "dc=example,dc=ldap": unclean shutdown detected; attempting recovery.
May 13 10:29:53 gp42-admin4 slapd[12280]: bdb_db_open: database "cn=accesslog": unclean shutdown detected; attempting recovery.
May 13 10:29:53 gp42-admin4 slapd[12280]: slapd starting
May 13 10:32:35 gp42-admin4 slapd[12345]: @(#) $OpenLDAP: slapd 2.4.40 (Sep 30 2014 16:49:45) $#012#011clement@localhost.localdomain:/home/clement/build/BUILD/open l dap -2.4.40/servers/slapd
Attempting to restart slapd from the command-line:
5735ed50 slapd starting
5735ed50 => bdb_entry_get: ndn: "cn=accesslog"
5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
5735ed50 bdb_idl_fetch_key: %cn=accesslog
5735ed50 bdb_idl_fetch_key: [b49d1940]
5735ed50 bdb_idl_fetch_key:
5735ed50 send_ldap_result: err=0 matched="" text=""
5735ed50 => bdb_entry_get: ndn: "dc=example,dc=com"
5735ed50 => bdb_entry_get: oc: "(null)", at: "contextCSN"
ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN
ldap_build_search_req ATTRS: reqDN reqType reqMod reqNewRDN reqDeleteOldRDN reqNewSuperior entryCSN
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com,0)
<= ldap_bv2dn(uid=jdoe,ou=Users,dc=example,dc=com)=0
=> ldap_dn2bv(272)
<= ldap_dn2bv(uid=jdoe,ou=Users,dc=example,dc=com)=0
5735ed50 => bdb_entry_get: ndn: "uid=jdoe,ou=Users,dc=example,dc=com"
5735ed50 => bdb_entry_get: oc: "(null)", at: "(null)"
slapd: search.c:1125: oc_filter: Assertion `f != ((void *)0)' failed.
Aborted
I have run db_recover on the dbase(s) on ldapserver2 but to no avail.
Does anyone have any suggestions?
Thank you in advance for any assistance.
John D. Borresen (Dave)
Linux/Unix Systems Administrator
MIT Lincoln Laboratory
Humanitarian Assistance and Disaster Relief (HADR) Systems
244 Wood St
Lexington, MA 02420
Email: john.borresen@ll.mit.edu
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc.
Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc.
Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc
openldap-technical@openldap.org
-
Borresen, John - 0444 - MITLL -
Quanah Gibson-Mount