I posted this to openldap-bugs but I didn't see it actually posted to the archive so I'll try in openldap-technical. I have LMDB integrated and it's working smoothly except for this issue.
---- Original Message ----
I'm an author of an open source database called Sky (http://skydb.io/) and I'm interested in porting the backend off LevelDB and move it over to LMDB. I pulled down the LMDB code using the instructions on Ferenc Szalai's gomdb (https://github.com/szferi/gomdb) README:
git clone -b mdb.master --single-branch git://git.openldap.org/openldap.git
When I ran "make test" against the code it seems to run through most of the tests but I get a "Resource Busy" error at one point. Here's the full output:
https://gist.github.com/benbjohnson/5628725
I'm running Mac OS X 10.8.3 and here's the output of my gcc -v:
$ gcc -v Using built-in specs. Target: i686-apple-darwin11 Configured with: /private/var/tmp/llvmgcc42/llvmgcc42-2336.11~182/src/configure --disable-checking --enable-werror --prefix=/Applications/Xcode.app/Contents/Developer/usr/llvm-gcc-4.2 --mandir=/share/man --enable-languages=c,objc,c++,obj-c++ --program-prefix=llvm- --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib --build=i686-apple-darwin11 --enable-llvm=/private/var/tmp/llvmgcc42/llvmgcc42-2336.11~182/dst-llvmCore/Developer/usr/local --program-prefix=i686-apple-darwin11- --host=x86_64-apple-darwin11 --target=i686-apple-darwin11 --with-gxx-include-dir=/usr/include/c++/4.2.1 Thread model: posix gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
Let me know if you need any other info from me.
Ben Johnson ben@skylandlabs.com
Ben Johnson ben@skylandlabs.com
Ben Johnson wrote:
I posted this to openldap-bugs but I didn't see it actually posted to the archive so I'll try in openldap-technical. I have LMDB integrated and it's working smoothly except for this issue.
---- Original Message ----
I'm an author of an open source database called Sky (http://skydb.io/) and I'm interested in porting the backend off LevelDB and move it over to LMDB. I pulled down the LMDB code using the instructions on Ferenc Szalai's gomdb (https://github.com/szferi/gomdb) README:
git clone -b mdb.master --single-branch git://git.openldap.org/openldap.git
When I ran "make test" against the code it seems to run through most of the tests but I get a "Resource Busy" error at one point. Here's the full output:
https://gist.github.com/benbjohnson/5628725
I'm running Mac OS X 10.8.3 and here's the output of my gcc -v:
MacOSX and FreeBSD are somewhat dicy. If you can run the failing command ("mdb_stat testdb") by hand, successfully, it's probably fine.
$ gcc -v Using built-in specs. Target: i686-apple-darwin11 Configured with: /private/var/tmp/llvmgcc42/llvmgcc42-2336.11~182/src/configure --disable-checking --enable-werror --prefix=/Applications/Xcode.app/Contents/Developer/usr/llvm-gcc-4.2 --mandir=/share/man --enable-languages=c,objc,c++,obj-c++ --program-prefix=llvm- --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib --build=i686-apple-darwin11 --enable-llvm=/private/var/tmp/llvmgcc42/llvmgcc42-2336.11~182/dst-llvmCore/Developer/usr/local --program-prefix=i686-apple-darwin11- --host=x86_64-apple-darwin11 --target=i686-apple-darwin11 --with-gxx-include-dir=/usr/include/c++/4.2.1 Thread model: posix gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
Let me know if you need any other info from me.
Ben Johnson ben@skylandlabs.com mailto:ben@skylandlabs.com
Ben Johnson ben@skylandlabs.com mailto:ben@skylandlabs.com
Running "mdb_stat testdb" gives me the same "Resource Busy" error.
It looks like mdb.c:2941 is checking if the return address from mmap() is the same as the hint passed in. Is there a problem with just using the mmap() return address? It looks like the mmap() is successful but just allocating to a different place.
Ben Johnson ben@skylandlabs.com
On May 23, 2013, at 4:55 PM, Howard Chu hyc@symas.com wrote:
Ben Johnson wrote:
I posted this to openldap-bugs but I didn't see it actually posted to the archive so I'll try in openldap-technical. I have LMDB integrated and it's working smoothly except for this issue.
---- Original Message ----
I'm an author of an open source database called Sky (http://skydb.io/) and I'm interested in porting the backend off LevelDB and move it over to LMDB. I pulled down the LMDB code using the instructions on Ferenc Szalai's gomdb (https://github.com/szferi/gomdb) README:
git clone -b mdb.master --single-branch git://git.openldap.org/openldap.git
When I ran "make test" against the code it seems to run through most of the tests but I get a "Resource Busy" error at one point. Here's the full output:
https://gist.github.com/benbjohnson/5628725
I'm running Mac OS X 10.8.3 and here's the output of my gcc -v:
MacOSX and FreeBSD are somewhat dicy. If you can run the failing command ("mdb_stat testdb") by hand, successfully, it's probably fine.
$ gcc -v Using built-in specs. Target: i686-apple-darwin11 Configured with: /private/var/tmp/llvmgcc42/llvmgcc42-2336.11~182/src/configure --disable-checking --enable-werror --prefix=/Applications/Xcode.app/Contents/Developer/usr/llvm-gcc-4.2 --mandir=/share/man --enable-languages=c,objc,c++,obj-c++ --program-prefix=llvm- --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib --build=i686-apple-darwin11 --enable-llvm=/private/var/tmp/llvmgcc42/llvmgcc42-2336.11~182/dst-llvmCore/Developer/usr/local --program-prefix=i686-apple-darwin11- --host=x86_64-apple-darwin11 --target=i686-apple-darwin11 --with-gxx-include-dir=/usr/include/c++/4.2.1 Thread model: posix gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
Let me know if you need any other info from me.
Ben Johnson ben@skylandlabs.com mailto:ben@skylandlabs.com
Ben Johnson ben@skylandlabs.com mailto:ben@skylandlabs.com
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Ben Johnson wrote:
Running "mdb_stat testdb" gives me the same "Resource Busy" error.
It looks like mdb.c:2941 is checking if the return address from mmap() is the same as the hint passed in. Is there a problem with just using the mmap() return address? It looks like the mmap() is successful but just allocating to a different place.
That's treated as a failure since mtest uses the FIXEDMAP flag. In most applications you won't be using FIXEDMAP so you can ignore this error.
Ben Johnson ben@skylandlabs.com mailto:ben@skylandlabs.com
On May 23, 2013, at 4:55 PM, Howard Chu <hyc@symas.com mailto:hyc@symas.com> wrote:
Ben Johnson wrote:
I posted this to openldap-bugs but I didn't see it actually posted to the archive so I'll try in openldap-technical. I have LMDB integrated and it's working smoothly except for this issue.
---- Original Message ----
I'm an author of an open source database called Sky (http://skydb.io/) and I'm interested in porting the backend off LevelDB and move it over to LMDB. I pulled down the LMDB code using the instructions on Ferenc Szalai's gomdb (https://github.com/szferi/gomdb) README:
git clone -b mdb.master --single-branch git://git.openldap.org/openldap.git
When I ran "make test" against the code it seems to run through most of the tests but I get a "Resource Busy" error at one point. Here's the full output:
https://gist.github.com/benbjohnson/5628725
I'm running Mac OS X 10.8.3 and here's the output of my gcc -v:
MacOSX and FreeBSD are somewhat dicy. If you can run the failing command ("mdb_stat testdb") by hand, successfully, it's probably fine.
$ gcc -v Using built-in specs. Target: i686-apple-darwin11 Configured with: /private/var/tmp/llvmgcc42/llvmgcc42-2336.11~182/src/configure --disable-checking --enable-werror --prefix=/Applications/Xcode.app/Contents/Developer/usr/llvm-gcc-4.2 --mandir=/share/man --enable-languages=c,objc,c++,obj-c++ --program-prefix=llvm- --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib --build=i686-apple-darwin11 --enable-llvm=/private/var/tmp/llvmgcc42/llvmgcc42-2336.11~182/dst-llvmCore/Developer/usr/local --program-prefix=i686-apple-darwin11- --host=x86_64-apple-darwin11 --target=i686-apple-darwin11 --with-gxx-include-dir=/usr/include/c++/4.2.1 Thread model: posix gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
Let me know if you need any other info from me.
Ben Johnson ben@skylandlabs.com mailto:ben@skylandlabs.com mailto:ben@skylandlabs.com
Ben Johnson ben@skylandlabs.com mailto:ben@skylandlabs.com mailto:ben@skylandlabs.com
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Howard-
The meta.mm_address is getting set to env->me_map only if it's a new environment and FIXEDMAP is used. If I open up an existing database then newenv would be 0 and the FIXEDMAP would be ignored, right?
Ben Johnson ben@skylandlabs.com
On May 23, 2013, at 5:04 PM, Howard Chu hyc@symas.com wrote:
Ben Johnson wrote:
Running "mdb_stat testdb" gives me the same "Resource Busy" error.
It looks like mdb.c:2941 is checking if the return address from mmap() is the same as the hint passed in. Is there a problem with just using the mmap() return address? It looks like the mmap() is successful but just allocating to a different place.
That's treated as a failure since mtest uses the FIXEDMAP flag. In most applications you won't be using FIXEDMAP so you can ignore this error.
Ben Johnson ben@skylandlabs.com mailto:ben@skylandlabs.com
On May 23, 2013, at 4:55 PM, Howard Chu <hyc@symas.com mailto:hyc@symas.com> wrote:
Ben Johnson wrote:
I posted this to openldap-bugs but I didn't see it actually posted to the archive so I'll try in openldap-technical. I have LMDB integrated and it's working smoothly except for this issue.
---- Original Message ----
I'm an author of an open source database called Sky (http://skydb.io/) and I'm interested in porting the backend off LevelDB and move it over to LMDB. I pulled down the LMDB code using the instructions on Ferenc Szalai's gomdb (https://github.com/szferi/gomdb) README:
git clone -b mdb.master --single-branch git://git.openldap.org/openldap.git
When I ran "make test" against the code it seems to run through most of the tests but I get a "Resource Busy" error at one point. Here's the full output:
https://gist.github.com/benbjohnson/5628725
I'm running Mac OS X 10.8.3 and here's the output of my gcc -v:
MacOSX and FreeBSD are somewhat dicy. If you can run the failing command ("mdb_stat testdb") by hand, successfully, it's probably fine.
$ gcc -v Using built-in specs. Target: i686-apple-darwin11 Configured with: /private/var/tmp/llvmgcc42/llvmgcc42-2336.11~182/src/configure --disable-checking --enable-werror --prefix=/Applications/Xcode.app/Contents/Developer/usr/llvm-gcc-4.2 --mandir=/share/man --enable-languages=c,objc,c++,obj-c++ --program-prefix=llvm- --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib --build=i686-apple-darwin11 --enable-llvm=/private/var/tmp/llvmgcc42/llvmgcc42-2336.11~182/dst-llvmCore/Developer/usr/local --program-prefix=i686-apple-darwin11- --host=x86_64-apple-darwin11 --target=i686-apple-darwin11 --with-gxx-include-dir=/usr/include/c++/4.2.1 Thread model: posix gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
Let me know if you need any other info from me.
Ben Johnson ben@skylandlabs.com mailto:ben@skylandlabs.com mailto:ben@skylandlabs.com
Ben Johnson ben@skylandlabs.com mailto:ben@skylandlabs.com mailto:ben@skylandlabs.com
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Hi,
nearly every 2 weeks the slapd stops answering on search requests. The process is still running(socket is open as well) but every questions runs into a timeout. When that happen, I need to kill the process with "-9" manually. After that I can start the process again.
We have configured a Master(Read-Write)-Slave(Read-Only) cluster, actually it happend on the Slave but it also had happend on the Master, but not that often. When the process on the Master crashes, the DB is broken and I need to restore a backup via slapcat ...
Objects stored in the BDB: 40.000 Search-Requests on Slave: 3.000.000 per day Search-Requests on Master: 100.000 per day Virtualized on VMWare 5.0
Master: Debian 6
Slave: Happend on Debian 6 as well as on Debian7(now) root@ldap085:/var/openldapdb07# slapd -V @(#) $OpenLDAP: slapd (Apr 23 2013 12:16:04) $ root@lupin:/tmp/buildd/openldap-2.4.31/debian/build/servers/slapd
root@ldap085:/var/openldapdb07# uname -r 3.2.0-4-amd64
Regards Carsten
--On Wednesday, June 12, 2013 8:31 AM +0200 Carsten Czerner carsten.czerner@leuphana.de wrote:
Happend on Debian 6 as well as on Debian7(now) root@ldap085:/var/openldapdb07# slapd -V @(#) $OpenLDAP: slapd (Apr 23 2013 12:16:04) $ root@lupin:/tmp/buildd/openldap-2.4.31/debian/build/servers/slapd
http://www.openldap.org/faq/data/cache/1456.html
I would note there are issues with the way in which Debian builds BDB as well.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
On Wed, 12 Jun 2013 07:35:56 -0700 Quanah Gibson-Mount quanah@zimbra.com wrote
--On Wednesday, June 12, 2013 8:31 AM +0200 Carsten Czerner carsten.czerner@leuphana.de wrote:
Happend on Debian 6 as well as on Debian7(now) root@ldap085:/var/openldapdb07# slapd -V @(#) $OpenLDAP: slapd (Apr 23 2013 12:16:04) $ root@lupin:/tmp/buildd/openldap-2.4.31/debian/build/servers/slapd
http://www.openldap.org/faq/data/cache/1456.html
I would note there are issues with the way in which Debian builds BDB as well.
Is this still true today? Which issues are this in detail?
Ciao, Michael.
--On Wednesday, June 12, 2013 4:59 PM +0200 Michael Ströder michael@stroeder.com wrote:
I would note there are issues with the way in which Debian builds BDB as well.
Is this still true today? Which issues are this in detail?
Yes, it is still true today. They do not set the --with-mutex=POSIX/pthreads flag.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
On Wed, 12 Jun 2013 08:35:37 -0700 Quanah Gibson-Mount quanah@zimbra.com wrote
--On Wednesday, June 12, 2013 4:59 PM +0200 Michael Ströder michael@stroeder.com wrote:
I would note there are issues with the way in which Debian builds BDB as well.
Is this still true today? Which issues are this in detail?
Yes, it is still true today. They do not set the --with-mutex=POSIX/pthreads flag.
We looked at the db4.8 source package. It seems they used configure option
--with-mutex=POSIX/pthreads/library
See build log here: https://buildd.debian.org/status/fetch.php?pkg=db4.8&arch=amd64&ver=...
Anything else which could be missing?
Ciao, Michael.
--On Wednesday, June 12, 2013 5:57 PM +0200 Michael Ströder michael@stroeder.com wrote:
On Wed, 12 Jun 2013 08:35:37 -0700 Quanah Gibson-Mount quanah@zimbra.com wrote
--On Wednesday, June 12, 2013 4:59 PM +0200 Michael Ströder michael@stroeder.com wrote:
I would note there are issues with the way in which Debian builds BDB as well.
Is this still true today? Which issues are this in detail?
Yes, it is still true today. They do not set the --with-mutex=POSIX/pthreads flag.
We looked at the db4.8 source package. It seems they used configure option
--with-mutex=POSIX/pthreads/library
See build log here: https://buildd.debian.org/status/fetch.php?pkg=db4.8&arch=amd64&ver=... 0-2&stamp=1283126079
Anything else which could be missing?
Yep. --enable-posixmutexes. Those two options need to go together.
--Quanah
--
Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
Quanah Gibson-Mount wrote:
--On Wednesday, June 12, 2013 5:57 PM +0200 Michael Ströder michael@stroeder.com wrote:
On Wed, 12 Jun 2013 08:35:37 -0700 Quanah Gibson-Mount quanah@zimbra.com wrote
--On Wednesday, June 12, 2013 4:59 PM +0200 Michael Ströder michael@stroeder.com wrote:
I would note there are issues with the way in which Debian builds BDB as well.
Is this still true today? Which issues are this in detail?
Yes, it is still true today. They do not set the --with-mutex=POSIX/pthreads flag.
We looked at the db4.8 source package. It seems they used configure option
--with-mutex=POSIX/pthreads/library
See build log here: https://buildd.debian.org/status/fetch.php?pkg=db4.8&arch=amd64&ver=... 0-2&stamp=1283126079
Anything else which could be missing?
Yep. --enable-posixmutexes. Those two options need to go together.
I have to nitpick here:
Analyzing db-4.8.30/dist/configure shows that option --enable-posixmutexes sets the variable db_cv_posixmutexes which in turn sets db_cv_mutex=posix_only if db_cv_mutex is "no" before.
If --with-mutex=POSIX/pthreads/library is used db_cv_mutex is set to this value. So it seems to me that --enable-posixmutexes is a no-op if --with-mutex=POSIX/pthreads/library was set explicitly.
Simply invoking:
$ cd build_unix $ ../dist/configure --with-mutex=POSIX/pthreads/library [..] checking if --enable-posixmutexes option specified... no [..] checking if --with-mutex=MUTEX option specified... POSIX/pthreads/library [..] checking for mutexes... (cached) POSIX/pthreads/library checking pthread.h usability... yes checking pthread.h presence... yes checking for pthread.h... yes checking for main in -lpthread... yes [..] checking for pthread_self... yes checking for pthread_yield... yes [..]
And I examined the differences of all generated files under build_unix/ if --enable-posixmutexes is set or not - of course always setting --with-mutex=POSIX/pthreads/library): There are no significant differences.
While I personally would prefer to explicitly mention --enable-posixmutexes just for completeness it technically does not seem to be a difference. So to me the Debian package libdb4.8 is not responsible for possible hangs.
Am I overlooking something?
Ciao, Michael.
openldap-technical@openldap.org
-
Ben Johnson -
Carsten Czerner -
Howard Chu -
Michael Ströder -
Quanah Gibson-Mount