Hi,
I have a large database with several million entries, which for a few specific entries takes very long (several minutes) to search.
Looking at the debug output the search suddenly goes into a mode where it tests a lot of completely irrelevant entries and outputs a lot of "bdb_search: <number> scope not okay" messages - which takes very long time before the query ends.
I've tried deleting all indexes and re-indexing with slapdindex, but the problem persists - also with 2.4.28. It's BDB 4.8.26 with back_bdb.
However... loading the entire database with slapadd made the problem go away. So I suspect the id2entry or dn2id database is broken.
So now I'm wondering... how do these situations occur and is there anyway to fix the database without reloading. It takes several hourse to reload, so it's not an ideal situation to have to do that every time such an event occurs. They are rather rare compared to the total number of entries, but they pop up once in a while, so it'll be a rutine job to start the database over from ldif, if that's the solution.
Is there any tool to inspect the BDB database, find the problem, fix it and re-index?
/Peter
Hi Peter
possibly tools in the db4-utils package?
which are:
/usr/bin/db_archive /usr/bin/db_checkpoint /usr/bin/db_deadlock /usr/bin/db_dump /usr/bin/db_dump185 /usr/bin/db_hotbackup /usr/bin/db_load /usr/bin/db_printlog /usr/bin/db_recover /usr/bin/db_sql /usr/bin/db_stat /usr/bin/db_upgrade /usr/bin/db_verify
suomi
On 2011-12-09 13:14, Peter Mogensen wrote:
Hi,
I have a large database with several million entries, which for a few specific entries takes very long (several minutes) to search.
Looking at the debug output the search suddenly goes into a mode where it tests a lot of completely irrelevant entries and outputs a lot of "bdb_search: <number> scope not okay" messages - which takes very long time before the query ends.
I've tried deleting all indexes and re-indexing with slapdindex, but the problem persists - also with 2.4.28. It's BDB 4.8.26 with back_bdb.
However... loading the entire database with slapadd made the problem go away. So I suspect the id2entry or dn2id database is broken.
So now I'm wondering... how do these situations occur and is there anyway to fix the database without reloading. It takes several hourse to reload, so it's not an ideal situation to have to do that every time such an event occurs. They are rather rare compared to the total number of entries, but they pop up once in a while, so it'll be a rutine job to start the database over from ldif, if that's the solution.
Is there any tool to inspect the BDB database, find the problem, fix it and re-index?
/Peter
--On Friday, December 09, 2011 1:14 PM +0100 Peter Mogensen apm@one.com wrote:
Hi,
I have a large database with several million entries, which for a few specific entries takes very long (several minutes) to search.
Looking at the debug output the search suddenly goes into a mode where it tests a lot of completely irrelevant entries and outputs a lot of "bdb_search: <number> scope not okay" messages - which takes very long time before the query ends.
I've tried deleting all indexes and re-indexing with slapdindex, but the problem persists - also with 2.4.28. It's BDB 4.8.26 with back_bdb.
4.8.26 was pulled by Oracle. The only version available for download is 4.8.30. I would advise updating BDB as a first step.
Also, confirming you are using back-bdb not back-hdb as your backend?
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
On 2011-12-09 20:40, Quanah Gibson-Mount wrote:
4.8.26 was pulled by Oracle. The only version available for download is 4.8.30. I would advise updating BDB as a first step.
I have actually reproduced what I suspect is the same problem (it's the same entries affected anyway) with an installation upgraded to 4.8.30: http://www.openldap.org/its/index.cgi/Incoming?id=6491
Also, confirming you are using back-bdb not back-hdb as your backend?
Yes... it's back-bdb, not hdb
/Peter
--On Friday, December 09, 2011 8:44 PM +0100 Peter Mogensen apm@one.com wrote:
On 2011-12-09 20:40, Quanah Gibson-Mount wrote:
4.8.26 was pulled by Oracle. The only version available for download is 4.8.30. I would advise updating BDB as a first step.
I have actually reproduced what I suspect is the same problem (it's the same entries affected anyway) with an installation upgraded to 4.8.30: http://www.openldap.org/its/index.cgi/Incoming?id=6491
Also, confirming you are using back-bdb not back-hdb as your backend?
Yes... it's back-bdb, not hdb
Ok. Interesting both that ITS and your report are using BDB 4.8. I've used BDB 4.7 with patches for DBs with many millions of entries without ever hitting this problem.
If you have a test server to try and reproduce this, can you try BDB 4.7.25+patches from Oracle instead of 4.8?
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
openldap-technical@openldap.org