openldap-bugs

openldap-bugs@openldap.org

1 participants
21001 discussions

(ITS#7213) back-mdb corruption with quickmode
by quanah＠OpenLDAP.org 20 Mar '12

20 Mar '12

Full_Name: Quanah Gibson-Mount Version: 2.4.30 OS: Linux 2.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (75.108.184.39) Using quickmode with slapadd can result in a corrupt database.

1 0

Re: (ITS#7210) back-mdb endless database growth
by quanah＠zimbra.com 20 Mar '12

20 Mar '12

--On Tuesday, March 20, 2012 2:16 PM +0000 hyc(a)symas.com wrote: > hyc(a)symas.com wrote: >> quanah(a)OpenLDAP.org wrote: >>> Full_Name: Quanah Gibson-Mount >>> Version: 2.4.30 >>> OS: Linux 2.6 >>> URL: ftp://ftp.openldap.org/incoming/ >>> Submission from: (NULL) (75.108.184.39) >>> >>> >>> If you create a script to add& delete the same 50,000 users, and then >>> re-run that script multiple times, the MDB database on disk continues >>> to grow, despite the fact that we are only adding and deleting the same >>> data repeatedly. There should be zero growth to the database. >>> >>> Instead, the data.mdb file went from 108MB in size to over 1.2GB after >>> 5-6 runs. Time to add/delete all the users went from 3 minutes 32 >>> seconds on the first run to over 90 minutes on the final run (5th or >>> 6th, I lost count). >> >> At present, MDB is working as intended. In your test a number of overflow >> pages are being used. Currently libmdb always allocates overflow pages >> from new space, because it's not smart enough to search the free list for >> contiguous pages. Can treat this ITS as an enhancement request for that >> purpose, but this is not a new issue nor is it unexpected in the current >> version of MDB. >> > Hm, I take this back. The overflow pages are not enough to account for > the difference. Still looking. If this were truly the expected behavior, it would make using back-mdb with delta-syncrepl useless, as the accesslog would simply grow without bound. ;) Even back-bdb/hdb don't do that. While the DB will never be smaller than the maximum size ever reached without reloading, it won't unnecessarily grow past that either. :P --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: (ITS#7210) back-mdb endless database growth
by hyc＠symas.com 20 Mar '12

20 Mar '12

hyc(a)symas.com wrote: > quanah(a)OpenLDAP.org wrote: >> Full_Name: Quanah Gibson-Mount >> Version: 2.4.30 >> OS: Linux 2.6 >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (75.108.184.39) >> >> >> If you create a script to add& delete the same 50,000 users, and then re-run >> that script multiple times, the MDB database on disk continues to grow, despite >> the fact that we are only adding and deleting the same data repeatedly. There >> should be zero growth to the database. >> >> Instead, the data.mdb file went from 108MB in size to over 1.2GB after 5-6 runs. >> Time to add/delete all the users went from 3 minutes 32 seconds on the first >> run to over 90 minutes on the final run (5th or 6th, I lost count). > > At present, MDB is working as intended. In your test a number of overflow > pages are being used. Currently libmdb always allocates overflow pages from > new space, because it's not smart enough to search the free list for > contiguous pages. Can treat this ITS as an enhancement request for that > purpose, but this is not a new issue nor is it unexpected in the current > version of MDB. > Hm, I take this back. The overflow pages are not enough to account for the difference. Still looking. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7210) back-mdb endless database growth
by hyc＠symas.com 20 Mar '12

20 Mar '12

quanah(a)OpenLDAP.org wrote: > Full_Name: Quanah Gibson-Mount > Version: 2.4.30 > OS: Linux 2.6 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (75.108.184.39) > > > If you create a script to add& delete the same 50,000 users, and then re-run > that script multiple times, the MDB database on disk continues to grow, despite > the fact that we are only adding and deleting the same data repeatedly. There > should be zero growth to the database. > > Instead, the data.mdb file went from 108MB in size to over 1.2GB after 5-6 runs. > Time to add/delete all the users went from 3 minutes 32 seconds on the first > run to over 90 minutes on the final run (5th or 6th, I lost count). At present, MDB is working as intended. In your test a number of overflow pages are being used. Currently libmdb always allocates overflow pages from new space, because it's not smart enough to search the free list for contiguous pages. Can treat this ITS as an enhancement request for that purpose, but this is not a new issue nor is it unexpected in the current version of MDB. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#7212) slapmodify support for back-config
by ondrej.kuznik＠acision.com 20 Mar '12

20 Mar '12

Full_Name: Ondrej Kuznik Version: HEAD OS: Linux URL: ftp://ftp.openldap.org/incoming/Ondrej-Kuznik-20120319-back-config-slapmodi… Submission from: (NULL) (62.168.56.1) For slapmodify to be usable on a database, that database needs to implement the bi_tool_dn2id_get and bi_tool_entry_modify functions, which is not the case of the back-config. I've put together a minimal (and maybe too naive) implementation of these for back-config and back-ldif at the above link. Currently, there is no validation whether the modification passes back-config checks since config_modify_internal takes a modify operation, while bi_tool_entry_modify receives only the new version of the entry to be modified. Also, for slapmodify to be really useful, it would have to allow entry deletion, for which there is no tool mode callback. To implement deletes, do you think it is better that deletion of an entry with children fail or delete the entire subtree? While the former seems both easier to implement and more sensible to me, I would like your know opinion. The attached file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Acision. Acision has not assigned rights and/or interest in this work to any party. I, Ondrej Kuznik am authorized by Acision, my employer, to release this work under the following terms. The attached modifications to OpenLDAP Software are subject to the following notice: Copyright 2012 Acision Redistribution and use in source and binary forms, with or without modification, are permitted only as authorized by the OpenLDAP Public License.

1 0

Re: (ITS#7211) mdb inconsistency without LAZY_LOCKS
by h.b.furuseth＠usit.uio.no 19 Mar '12

19 Mar '12

I wrote: > I'm sleeping for write transactions only since that's when the > inconsistent copy is written back to disk. Please ignore that, that was another too-paranoid patch -- Hallvard

1 0

(ITS#7211) mdb inconsistency without LAZY_LOCKS
by h.b.furuseth＠usit.uio.no 19 Mar '12

19 Mar '12

Full_Name: Hallvard B Furuseth Version: master, af3a23a1ec93220be392a8ea6372475b7357c171 OS: Linux x86_64 URL: Submission from: (NULL) (195.1.106.125) Submitted by: hallvard A write transaction can overwrite env->me_metas[]->mm_dbs, me_numdbs and me_dbxs while another txn is copying them in mdb_txn_renew0(), resulting in an inconsistent copy. '#define LAZY_LOCKS 0' fixes it. Without it, test054 dumps core with this patch: diff --git a/libraries/libmdb/mdb.c b/libraries/libmdb/mdb.c index b714119..5e22827 100644 --- a/libraries/libmdb/mdb.c +++ b/libraries/libmdb/mdb.c @@ -1519,2 +1519,4 @@ mdb_txn_renew0(MDB_txn *txn) memcpy(txn->mt_dbs, env->me_metas[txn->mt_toggle]->mm_dbs, 2 * sizeof(MDB_db)); + if ((rand() & 1)) usleep(100000); + assert(!memcmp(txn->mt_dbs, env->me_metas[txn->mt_toggle]->mm_dbs, 2 * sizeof(MDB_db))); if (txn->mt_numdbs > 2) I'm sleeping for write transactions only since that's when the inconsistent copy is written back to disk.

1 0

(ITS#7210) back-mdb endless database growth
by quanah＠OpenLDAP.org 19 Mar '12

19 Mar '12

Full_Name: Quanah Gibson-Mount Version: 2.4.30 OS: Linux 2.6 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (75.108.184.39) If you create a script to add & delete the same 50,000 users, and then re-run that script multiple times, the MDB database on disk continues to grow, despite the fact that we are only adding and deleting the same data repeatedly. There should be zero growth to the database. Instead, the data.mdb file went from 108MB in size to over 1.2GB after 5-6 runs. Time to add/delete all the users went from 3 minutes 32 seconds on the first run to over 90 minutes on the final run (5th or 6th, I lost count).

1 0

Re: (ITS#5915) Some man pages still only talk about slapd.conf
by ghenry＠suretecsystems.com 19 Mar '12

19 Mar '12

It is on the list for updating and a migration section. -- Kind Regards, Gavin Henry. Managing Director. T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry(a)suretecsystems.com Open Source. Open Solutions(tm). http://www.suretecsystems.com/ Suretec Systems is a limited company registered in Scotland. Registered number: SC258005. Registered office: 24 Cormack Park, Rothienorman, Inverurie, Aberdeenshire, AB51 8GL. Subject to disclaimer at http://www.suretecgroup.com/disclaimer.html On 19 Mar 2012, at 14:51, Bruno Schrade <bs(a)schwarzwald-software.de> wrote: > Sehr geehrter Herr Henry, > > i just moved an OpenLDAP-Server from Debian-5 to Debian-6. Expecting it would simply run using the same configuration file i just copied slapd.conf. This resulted in error messages along the line "missing credentials". So i looked up the QuickStartGuide and continued debugging the authorization process - finally found the reason and cure by reading /etc/init.d/slapd. > > I would like to suggest two changes: > > 1 May i suggest a quick fix: add something like "don't read this" > to the top of the QuickStartGuide. > 2 If possible change your mind on dropping slapd.conf-Support. > - Dropping it will break existing configurations. > - People are quite accustomed to /etc/.../.. .conf-files > > I would like to add that i really appreciate the work you are doing. > > Mit freundlichen Grüssen > > Schwarzwald Software Engineering * Bruno Schrade > Theodor-Kreuz-Strasse 6 * D-79868 Feldberg * Tel. +49 (0)7655 618 > http://www.schwarzwald-software.de >

1 0

Re: (ITS#5915) Some man pages still only talk about slapd.conf
by bs＠schwarzwald-software.de 19 Mar '12

19 Mar '12

Sehr geehrter Herr Henry, i just moved an OpenLDAP-Server from Debian-5 to Debian-6. Expecting it would simply run using the same configuration file i just copied slapd.conf. This resulted in error messages along the line "missing credentials". So i looked up the QuickStartGuide and continued debugging the authorization process - finally found the reason and cure by reading /etc/init.d/slapd. I would like to suggest two changes: 1 May i suggest a quick fix: add something like "don't read this" to the top of the QuickStartGuide. 2 If possible change your mind on dropping slapd.conf-Support. - Dropping it will break existing configurations. - People are quite accustomed to /etc/.../.. .conf-files I would like to add that i really appreciate the work you are doing. Mit freundlichen Grüssen Schwarzwald Software Engineering * Bruno Schrade Theodor-Kreuz-Strasse 6 * D-79868 Feldberg * Tel. +49 (0)7655 618 http://www.schwarzwald-software.de

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs