Hello everyone,
I have a bit of an issue with my ldap database. I have a Zimbra community edition which uses openldap. A server crashed and I am unable to start the ldap services after the reboot. The description of my problem, after some digging about is:
the initial error indicated problem with the ldap
Starting ldap...Done. Search error: Unable to determine enabled services from ldap. Enabled services read from cache. Service list may be inaccurate.
Having investigated the issue, I noticed the following errors in the zimbra.log
slapd[31281]: mdb_entry_decode: attribute index 560427631 not recognized
I also noticed that the /opt/zimbra/data/ldap/mdb/db/data.mdb is actually 81Gb in size and had reached the limit imposed by the ldap_db_maxsize variable. so over the weekend, the LDAP mdb file became no longer sparse.
I tried following the steps described in https://syslint.com/blog/tutorial/solved-critical-ldap-primary-mdb-database-... but with no success, as the slapcat segfaults with the following message.
/opt/zimbra/common/sbin/slapcat -ccc -F /opt/zimbra/data/ldap/config -b "" -l /opt/zimbra/RECOVERY/SLAPCAT/zimbra_mdb.ldiff 5c583982 mdb_entry_decode: attribute index 560427631 not recognized Segmentation fault (core dumped)
the mdb_copy produces a file of 420 mb in size, but it still contains the "mdb_entry_decode: attribute index 560427631 not recognized" error. I've also tried mdb_dump, but had the same issues after using the mdb_load command.
I found a post ( http://www.openldap.org/its/index.cgi/Software%20Bugs?id=8360 ) in the openldap community that the mdb gets corrupted if it reaches the maximum defined size. but no solution of how to fix it.
any advice on getting the database recovered and working again?
Cheers
Andrei
Andrei Mikhailovsky wrote:
Hello everyone,
I have a bit of an issue with my ldap database. I have a Zimbra community edition which uses openldap. A server crashed and I am unable to start the ldap services after the reboot. The description of my problem, after some digging about is:
the initial error indicated problem with the ldap
Starting ldap...Done. Search error: Unable to determine enabled services from ldap. Enabled services read from cache. Service list may be inaccurate.
Having investigated the issue, I noticed the following errors in the zimbra.log
*slapd[31281]: mdb_entry_decode: attribute index 560427631 not recognized*
I also noticed that the /opt/zimbra/data/ldap/mdb/db/data.mdb is actually 81Gb in size and had reached the limit imposed by the ldap_db_maxsize variable. so over the weekend, the LDAP mdb file became no longer sparse.
I tried following the steps described in https://syslint.com/blog/tutorial/solved-critical-ldap-primary-mdb-database-... but with no success, as the slapcat segfaults with the following message.
/opt/zimbra/common/sbin/slapcat -ccc -F /opt/zimbra/data/ldap/config -b "" -l /opt/zimbra/RECOVERY/SLAPCAT/zimbra_mdb.ldiff 5c583982 mdb_entry_decode: attribute index 560427631 not recognized Segmentation fault (core dumped)
the mdb_copy produces a file of 420 mb in size, but it still contains the "mdb_entry_decode: attribute index 560427631 not recognized" error. I've also tried mdb_dump, but had the same issues after using the mdb_load command.
I found a post ( http://www.openldap.org/its/index.cgi/Software%20Bugs?id=8360%C2%A0) in the openldap community that the mdb gets corrupted if it reaches the maximum defined size. but no solution of how to fix it.
That's from over 3 years ago and has subsequently fixed. If you're running on such an old release, there's likely not much that can be done. Ordinarily it's possible to back up to the immediately preceding transaction, in case the last transaction is corrupted, but with that particular bug it's likely that the corruption occurred in an earlier transaction and has been carried forward in all subsequent ones.
any advice on getting the database recovered and working again?
You could try using the preceding transaction and see if it's in any better shape. The code for this is not released in LMDB 0.9. You can compile the mdb.master branch in git to obtain it. Then use the "-v" option with mdb_copy and see if that copy of the database is usable.
Hi Howard,
You could try using the preceding transaction and see if it's in any better shape. The code for this is not released in LMDB 0.9. You can compile the mdb.master branch in git to obtain it. Then use the "-v" option with mdb_copy and see if that copy of the database is usable.
I have compiled liblmdb using the mdb.master branch and used the mdb_copy as you've suggested. It didn't produce any errors. However, when I copy the data.mdb back to the Zimbra server it still produces the same error:
Feb 13 18:30:03 mail-archive slapd[8830]: mdb_entry_decode: attribute index 560427631 not recognized
and when I am trying to reindex I get:
zimbra@mail-archive:~/data/ldap/mdb/db$ /opt/zimbra/libexec/zmslapindex -vv indexing id=00000001 indexing id=00000002 indexing id=00000003 5c64631c mdb_entry_decode: attribute index 560427631 not recognized /opt/zimbra/libexec/zmslapindex: line 67: 11205 Segmentation fault (core dumped) /opt/zimbra/common/sbin/slapindex -q -F /opt/zimbra/data/ldap/config -b "" $KEY
My broken ldap data folder contains the following files:
zimbra@mail-archive:~/data/ldap/mdb/db$ ls -la /opt/zimbra/RECOVERY/ldap-original/mdb/db/ total 83886096 drwxr-xr-x 1 zimbra zimbra 60 Feb 4 12:12 . drwxr-xr-x 1 zimbra zimbra 4 Sep 7 10:37 .. -rw------- 1 zimbra zimbra 85899345920 Feb 4 10:38 data.mdb -rw------- 1 zimbra zimbra 8192 Feb 6 17:37 lock.mdb -rw-r----- 1 root root 10485759 Feb 4 12:12 log.0000000001
I can see there is the log file log.0000000001. I don't know how do I check or roll back data using this log file? I haven't managed to find anything useful by googling. Any idea?
Cheers
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
--On Wednesday, February 13, 2019 6:37 PM +0000 Andrei Mikhailovsky andrei@arhont.com wrote:
Hi Howard,
You could try using the preceding transaction and see if it's in any better shape. The code for this is not released in LMDB 0.9. You can compile the mdb.master branch in git to obtain it. Then use the "-v" option with mdb_copy and see if that copy of the database is usable.
I have compiled liblmdb using the mdb.master branch and used the mdb_copy as you've suggested. It didn't produce any errors. However, when I copy the data.mdb back to the Zimbra server it still produces the same error:
Did you use mdb_copy with the -v flag as Howard noted? It is helpful to be precise about exactly what steps you took.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Hi Quanah,
Yes, indeed. I've used the ./mdb_copy -v <source> <destination> to get the copy of the database. It has created 227MB file, which I've copied over to the ldap data folder and replaced the original 80GB file.
Cheers
----- Original Message -----
From: "Quanah Gibson-Mount" quanah@symas.com To: "Andrei Mikhailovsky" andrei@arhont.com, "Howard Chu" hyc@symas.com Cc: "openldap-technical" openldap-technical@openldap.org Sent: Thursday, 14 February, 2019 01:04:26 Subject: Re: help with mdb database recovery after crash
--On Wednesday, February 13, 2019 6:37 PM +0000 Andrei Mikhailovsky andrei@arhont.com wrote:
Hi Howard,
You could try using the preceding transaction and see if it's in any better shape. The code for this is not released in LMDB 0.9. You can compile the mdb.master branch in git to obtain it. Then use the "-v" option with mdb_copy and see if that copy of the database is usable.
I have compiled liblmdb using the mdb.master branch and used the mdb_copy as you've suggested. It didn't produce any errors. However, when I copy the data.mdb back to the Zimbra server it still produces the same error:
Did you use mdb_copy with the -v flag as Howard noted? It is helpful to be precise about exactly what steps you took.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Andrei Mikhailovsky wrote:
Hi Howard,
You could try using the preceding transaction and see if it's in any better shape. The code for this is not released in LMDB 0.9. You can compile the mdb.master branch in git to obtain it. Then use the "-v" option with mdb_copy and see if that copy of the database is usable.
I have compiled liblmdb using the mdb.master branch and used the mdb_copy as you've suggested. It didn't produce any errors. However, when I copy the data.mdb back to the Zimbra server it still produces the same error:
Feb 13 18:30:03 mail-archive slapd[8830]: mdb_entry_decode: attribute index 560427631 not recognized
Then unfortunately you just have bogus attribute data in the DB. Seems unlikely that you'll be able to sort this out, it may actually be a disk media failure.
and when I am trying to reindex I get:
zimbra@mail-archive:~/data/ldap/mdb/db$ /opt/zimbra/libexec/zmslapindex -vv indexing id=00000001 indexing id=00000002 indexing id=00000003 5c64631c mdb_entry_decode: attribute index 560427631 not recognized /opt/zimbra/libexec/zmslapindex: line 67: 11205 Segmentation fault (core dumped) /opt/zimbra/common/sbin/slapindex -q -F /opt/zimbra/data/ldap/config -b "" $KEY
My broken ldap data folder contains the following files:
zimbra@mail-archive:~/data/ldap/mdb/db$ ls -la /opt/zimbra/RECOVERY/ldap-original/mdb/db/ total 83886096 drwxr-xr-x 1 zimbra zimbra 60 Feb 4 12:12 . drwxr-xr-x 1 zimbra zimbra 4 Sep 7 10:37 .. -rw------- 1 zimbra zimbra 85899345920 Feb 4 10:38 data.mdb -rw------- 1 zimbra zimbra 8192 Feb 6 17:37 lock.mdb -rw-r----- 1 root root 10485759 Feb 4 12:12 log.0000000001
I can see there is the log file log.0000000001. I don't know how do I check or roll back data using this log file? I haven't managed to find anything useful by googling. Any idea?
LMDB doesn't use log files. Apparently you've let some other software overwrite your database.
Cheers
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
openldap-technical@openldap.org
-
Andrei Mikhailovsky -
Howard Chu -
Quanah Gibson-Mount