Hello,
because of problems with bdb (virtual memory using and glibc) and limitiations (IDL), I want migrate to mdb.
So my first question: Does mdb have limitations like bdb it have aka BDB_IDL_LOGN?
Second, I set up an small lab for tests with mdb and don't get the slapd started with larger mdb size (10GB).
Here the lab and test:
Start slapd with an empty /var/lib/ldap/ldap.mdb directory I want reserve 10GByte disk space for mdb (maxsize 10737418240)
error message in syslog ============================================================== @(#) $OpenLDAP: slapd 2.4.33 $ opensuse-buildservice@opensuse.org daemon: IPv6 socket() failed errno=97 (Address family not supported by protocol) mdb_db_open: database "ou=demo" cannot be opened, err 12. Restore from backup! backend_startup_one (type=mdb, suffix="ou=demo"): bi_db_open failed! (12) slapd stopped.
created files from slapd while using ============================================================== ~# du -b /var/lib/ldap/ldap.mdb/* 12288 /var/lib/ldap/ldap.mdb/data.mdb 8192 /var/lib/ldap/ldap.mdb/lock.mdb
I "straced" the start of slapd, here I think is the importand part: ============================================================== . . . stat("/var/lib/ldap/ldap.mdb", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 mmap(NULL, 2101248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe38f915000 brk(0x7fe3948ae000) = 0x7fe3948ae000 open("/var/lib/ldap/ldap.mdb/lock.mdb", O_RDWR|O_CREAT, 0600) = 9 fcntl(9, F_GETFD) = 0 fcntl(9, F_SETFD, FD_CLOEXEC) = 0 fcntl(9, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) = 0 lseek(9, 0, SEEK_END) = 0 ftruncate(9, 8192) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_SHARED, 9, 0) = 0x7fe3941e4000 open("/var/lib/ldap/ldap.mdb/data.mdb", O_RDWR|O_CREAT, 0600) = 11 read(11, "", 4096) = 0 mmap(NULL, 10737418240, PROT_READ, MAP_SHARED, 11, 0) = -1 ENOMEM (Cannot allocate memory) close(11) = 0 munmap(0x7fe3941e4000, 8192) = 0 close(9) = 0 . . .
So I tried to start slapd with the default of maxsize -- every thing is ok -- ============================================================== If i comment out maxsize 10737418240 (use the default of 10M), start slapd every thing is fine, slapd starts and create files:
~# du -b /var/lib/ldap/ldap.mdb/* 12288 /var/lib/ldap/ldap.mdb/data.mdb 8192 /var/lib/ldap/ldap.mdb/lock.mdb
As I understand, at first slapd start, the database file will be created with the full size, but as sparse file? So why I don't see this, if I use the default (maxsize 10485760)
Here my slapd.configuration file ============================================================== include /etc/openldap/schema/core.schema include /etc/openldap/schema/cosine.schema include /etc/openldap/schema/inetorgperson.schema include /etc/openldap/schema/yast.schema include /etc/openldap/schema/rfc2307bis.schema pidfile /var/run/slapd/slapd.pid argsfile /var/run/slapd/slapd.args
modulepath /usr/lib/ldap moduleload back_mdb moduleload back_monitor
sizelimit -1 timelimit 300 disallow bind_anon require authc
gentlehup on tool-threads 4 serverID 001
database mdb suffix "ou=demo" rootdn "cn=admin" rootpw password directory /var/lib/ldap/ldap.mdb loglevel 256 maxsize 10737418240 index objectClass,entryUUID,entryCSN eq
mounted filesystem for database ============================================================== debld02:/etc/openldap # mount | grep ldap /dev/sdb5 on /var/lib/ldap type ext3 (rw,relatime)
free space on filsystem ============================================================== debld02:/etc/openldap # df -h | grep ldap /dev/sdb5 30G 5.6G 23G 20% /var/lib/ldap
So I tried to create the sparse file manual ============================================================== ~# su -- ldap ~> id uid=76(ldap) gid=70(ldap) groups=70(ldap)
~> dd if=/dev/zero of=/var/lib/ldap/ldap.mdb/test.sparse bs=1 count=0 seek=10G 0+0 records in 0+0 records out 0 bytes (0 B) copied, 1.3413e-05 s, 0.0 kB/s
~> du -b /var/lib/ldap/ldap.mdb/test.sparse 10737418240 /var/lib/ldap/ldap.mdb/test.sparse
What is going wrong with my test lab?
Thanks Meike
Meike Stone wrote:
Hello,
because of problems with bdb (virtual memory using and glibc) and limitiations (IDL), I want migrate to mdb.
So my first question: Does mdb have limitations like bdb it have aka BDB_IDL_LOGN?
Yes. back-mdb is ~60% the same code as back-bdb/hdb, its indexing functions are basically identical.
Second, I set up an small lab for tests with mdb and don't get the slapd started with larger mdb size (10GB).
Check your ulimits. MDB is much simpler than anything else but you are still required to understand how your own OS/platform works.
stat("/var/lib/ldap/ldap.mdb", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0 mmap(NULL, 2101248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fe38f915000 brk(0x7fe3948ae000) = 0x7fe3948ae000 open("/var/lib/ldap/ldap.mdb/lock.mdb", O_RDWR|O_CREAT, 0600) = 9 fcntl(9, F_GETFD) = 0 fcntl(9, F_SETFD, FD_CLOEXEC) = 0 fcntl(9, F_SETLK, {type=F_WRLCK, whence=SEEK_SET, start=0, len=1}) = 0 lseek(9, 0, SEEK_END) = 0 ftruncate(9, 8192) = 0 mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_SHARED, 9, 0) = 0x7fe3941e4000 open("/var/lib/ldap/ldap.mdb/data.mdb", O_RDWR|O_CREAT, 0600) = 11 read(11, "", 4096) = 0 mmap(NULL, 10737418240, PROT_READ, MAP_SHARED, 11, 0) = -1 ENOMEM (Cannot allocate memory) close(11) = 0 munmap(0x7fe3941e4000, 8192) = 0 close(9) = 0 .
So my first question: Does mdb have limitations like bdb it have aka BDB_IDL_LOGN?
Yes. back-mdb is ~60% the same code as back-bdb/hdb, its indexing functions are basically identical.
Thanks for information, .. it was not that what I expected, so I think for a lot of users with larger database this is a problem and they use patched versions...
To find out, what the problem was, why *suddenly* the database was extreme slowly, in spite of indexes, it took a long time for us.
So it would be great, to notice this in the admin guide and make a page in the Faq-O-Matic.
Second, I set up an small lab for tests with mdb and don't get the slapd started with larger mdb size (10GB).
Check your ulimits. MDB is much simpler than anything else but you are still required to understand how your own OS/platform works.
Thank for pointing me, this was the right hint (would be nice, to notice this in the manual page too), because at this point I did not realized, that I have to increase the size of the virtual memory!
(I tried to generate a sparse manual (see my last post), to check if there are any problems related to this.)
I use SLES and as discussed in my last thread with Quanah (http://www.openldap.org/lists/openldap-technical/201301/msg00179.html), there is limited the virtual memory to the physical memory. Thats bad, but it happened. I don't know the limits of other distributions.
I hope, that for mdb I don't must have at least the same size of RAM, like the database is..?
Thank you very much!
Meike
Meike Stone wrote:
So my first question: Does mdb have limitations like bdb it have aka BDB_IDL_LOGN?
Yes. back-mdb is ~60% the same code as back-bdb/hdb, its indexing functions are basically identical.
Thanks for information, .. it was not that what I expected,
Read the MDB docs/papers/presentations.
--On Friday, January 18, 2013 12:29 PM +0000 Howard Chu hyc@symas.com wrote:
Meike Stone wrote:
Hello,
because of problems with bdb (virtual memory using and glibc) and limitiations (IDL), I want migrate to mdb.
So my first question: Does mdb have limitations like bdb it have aka BDB_IDL_LOGN?
Yes. back-mdb is ~60% the same code as back-bdb/hdb, its indexing functions are basically identical.
However, I never got mdb to work successfully by modifying these values.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
openldap-technical@openldap.org
-
Howard Chu -
Meike Stone -
Quanah Gibson-Mount