- openldap-bugs - openldap.org

Re: (ITS#7653) Test Build Version 2.4.36 hangs
by quanah＠zimbra.com 07 Aug '13

07 Aug '13

--On July 31, 2013 6:51:18 AM +0000 howard.allison(a)pensionsversicherung.at wrote: > Full_Name: Howard Allison > Version: 2.4.36 > OS: AIX > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (194.153.217.248) > > > AIX 6.1 Using IBM VAC Compiler > Configure and Build works, test hangs at test005 Hi, thanks for the report. Unfortunately, your stack trace only shows a single thread out of multiple threads. This thread is waiting on another thread to complete a task. Please submit a full backtrace of all threads. Thanks! --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#7656) Build: Installing LMDB on OS X
by john＠jahewson.com 05 Aug '13

05 Aug '13

Full_Name: John Hewson Version: 2.4.35 OS: OS X 10.8.4 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (76.175.89.135) I can't install liblmdb on OS X because its Makefile assumes that cp can handle multiple source files, rather than the basic POSIX cp, which can't. This leads to the situation on OS X where: cp source1 source2 dest throws an error. The breaks the LMDB Makefile, because its install attempts to do: ILIBS = liblmdb.a liblmdb.so ... cp $(IPROGS) $(DESTDIR)$(prefix)/bin Most makefiles I've encountered use a for loop for copying multiple files, so the fix for the Makefile is to change `install` to: install: $(ILIBS) $(IPROGS) $(IHDRS) for f in $(IPROGS); do cp $$f $(DESTDIR)$(prefix)/bin; done for f in $(ILIBS); do cp $$f $(DESTDIR)$(prefix)/lib; done for f in $(IHDRS); do cp $$f $(DESTDIR)$(prefix)/include; done for f in $(IDOCS); do cp $$f $(DESTDIR)$(prefix)/man/man1; done Then LMDB builds and installs successfully on OS X.

1 0

Re: (ITS#7655) segfault during initial mirror of multimaster delta replication
by quanah＠zimbra.com 05 Aug '13

05 Aug '13

--On Sunday, August 04, 2013 4:27 PM +0000 hans.freitag(a)entiretec.com wrote: > Full_Name: Hans Freitag > Version: 2.4.35 and 33 > OS: SLES 11SP2 > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (193.200.138.3) > > > I have a Multimaster Delta replication setup here with bdb on a 18 GB > Database. > > After a crash due to a full disk I made a new database on one node ans > started over. > > The empty node started to replicate, from the full one but after a while > (approx. 2GB) it crashed with a segfault: > > Aug 4 11:45:32 mhr-dd-lda-01 kernel: [52189.476209] slapd[10158]: > segfault at 20 ip 00007ff97ebfabc0 sp 00007ff6e57e6b38 error 4 in > libc-2.11.1.so[7ff97eb79000+155000] > > So i thought, maybe it is not e good Idea to put in a package for SP2 in a > machine running SP1 so my first attempt to solve was an upgrade. After the > upgrade I got this: > > Aug 4 12:46:29 mhr-dd-lda-01 kernel: [ 1414.757587] slapd[3704]: > segfault at 20 ip 00007fc82eee6182 sp 00007fc592e0acf0 error 4 in > slapd[7fc82ee7a000+1e6000] > > So I created a brandnew openldap RPM 2.4.35 rpm to try out if the problem > is maybe related to the 2.4.33 version I am running. But fail: > > Aug 4 13:47:19 mhr-dd-lda-01 kernel: [ 5063.074410] slapd[8749]: > segfault at 20 ip 00007fcbc1b537dc sp 00007fc92624fb88 error 4 in > slapd[7fcbc1ac8000+1ea000] > > At the moment I deactivated the accesslogging on the node which seems to > work. I will know for sure in a few hours. ;-) I can try to reproduce > that on a backup node next week. Whenn all the main nodes are up and > running again. :) I would suggest you build with debugging symbols, enable core files, and provide a backtrace of the problem. What you have provided does not give any useful information for debugging purposes. You also fail to state the backend you are using (back-bdb or back-hdb). For information on how to provide a backtrace: <http://www.openldap.org/faq/data/cache/59.html> Regards, Quanah -- Quanah Gibson-Mount Lead Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

(ITS#7655) segfault during initial mirror of multimaster delta replication
by hans.freitag＠entiretec.com 04 Aug '13

04 Aug '13

Full_Name: Hans Freitag Version: 2.4.35 and 33 OS: SLES 11SP2 URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (193.200.138.3) I have a Multimaster Delta replication setup here with bdb on a 18 GB Database. After a crash due to a full disk I made a new database on one node ans started over. The empty node started to replicate, from the full one but after a while (approx. 2GB) it crashed with a segfault: Aug 4 11:45:32 mhr-dd-lda-01 kernel: [52189.476209] slapd[10158]: segfault at 20 ip 00007ff97ebfabc0 sp 00007ff6e57e6b38 error 4 in libc-2.11.1.so[7ff97eb79000+155000] So i thought, maybe it is not e good Idea to put in a package for SP2 in a machine running SP1 so my first attempt to solve was an upgrade. After the upgrade I got this: Aug 4 12:46:29 mhr-dd-lda-01 kernel: [ 1414.757587] slapd[3704]: segfault at 20 ip 00007fc82eee6182 sp 00007fc592e0acf0 error 4 in slapd[7fc82ee7a000+1e6000] So I created a brandnew openldap RPM 2.4.35 rpm to try out if the problem is maybe related to the 2.4.33 version I am running. But fail: Aug 4 13:47:19 mhr-dd-lda-01 kernel: [ 5063.074410] slapd[8749]: segfault at 20 ip 00007fcbc1b537dc sp 00007fc92624fb88 error 4 in slapd[7fcbc1ac8000+1ea000] At the moment I deactivated the accesslogging on the node which seems to work. I will know for sure in a few hours. ;-) I can try to reproduce that on a backup node next week. Whenn all the main nodes are up and running again. :)

1 0

ITS#7599 libldap and libclntsh
by hyc＠symas.com 02 Aug '13

02 Aug '13

This is an Oracle bug, please contact Oracle for a fix. Their library should not be exporting symbols from the ldap_* namespace; this space has been owned by libldap since the 1990s. Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#7654) [PATCH] back-meta/config.c: #include "ldif.h"
by fumiyas＠osstech.co.jp 02 Aug '13

02 Aug '13

Full_Name: SATOH Fumiyasu Version: master OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (240b:10:2640:bf0:222:5fff:fe95:3cda) >From 9d009b4890586354e94be92f4ddb2462e9d18158 Mon Sep 17 00:00:00 2001 From: SATOH Fumiyasu <fumiyas(a)osstech.co.jp> Date: Fri, 2 Aug 2013 18:20:05 +0900 Subject: [PATCH] back-meta/config.c: #include "ldif.h" --- servers/slapd/back-meta/config.c | 1 + 1 file changed, 1 insertion(+) diff --git a/servers/slapd/back-meta/config.c b/servers/slapd/back-meta/config.c index 0be9ed8..a23f5ef 100644 --- a/servers/slapd/back-meta/config.c +++ b/servers/slapd/back-meta/config.c @@ -30,6 +30,7 @@ #include "slap.h" #include "config.h" #include "lutil.h" +#include "ldif.h" #include "../back-ldap/back-ldap.h" #include "back-meta.h" -- 1.8.3.2

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by h.b.furuseth＠usit.uio.no 01 Aug '13

01 Aug '13

Howard Chu writes: > h.b.furuseth(a)usit.uio.no wrote: >> I'd call this two doc bugs: >> >> Something should mention that once a page has been used, the file will >> never shrink below that page. (Can't quite say that the file never >> shrinks, since WRITEMAP and Windows will set it to exact re-configurable >> mapsize in some cases. If I remember correctly.) > > You cannot shrink the mapsize below that last used page, and the last used > page is a non-decreasing number. Yes. Duh, that's already roughly stated under mdb_env_set_mapsize(). >> mdb_env_copy() and mdb_env_copyfd() should document that >> >> @node This call can trigger siginficant file size growth if run in >> parallel with write transactions, because it employs a read-only >> transaction. See long-lived transactions under Caveats. I'll add this one though. >> I wonder that means these functions should have a flag parameter. >> For some databases, users might prefer a _write_ transaction while >> reading the DB, blocking writes while copying. > > No one has ever wanted a hot-backup function that halts writes. You > might as well not do a hot-backup at all. Yes, nevermind about a mdb_env_copy() flag. An mdb_txn_begin timeout would make it a viable choice - but if that is implemented someday, those two behaviors together could go in an MDB_env flag. -- Hallvard

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by hyc＠symas.com 31 Jul '13

31 Jul '13

h.b.furuseth(a)usit.uio.no wrote: > I'd call this two doc bugs: > > Something should mention that once a page has been used, the file will > never shrink below that page. (Can't quite say that the file never > shrinks, since WRITEMAP and Windows will set it to exact re-configurable > mapsize in some cases. If I remember correctly.) You cannot shrink the mapsize below that last used page, and the last used page is a non-decreasing number. > mdb_env_copy() and mdb_env_copyfd() should document that > > @node This call can trigger siginficant file size growth if run in > parallel with write transactions, because it employs a read-only > transaction. See long-lived transactions under Caveats. > > I wonder that means these functions should have a flag parameter. > For some databases, users might prefer a _write_ transaction while > reading the DB, blocking writes while copying. No one has ever wanted a hot-backup function that halts writes. You might as well not do a hot-backup at all. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by h.b.furuseth＠usit.uio.no 31 Jul '13

31 Jul '13

I'd call this two doc bugs: Something should mention that once a page has been used, the file will never shrink below that page. (Can't quite say that the file never shrinks, since WRITEMAP and Windows will set it to exact re-configurable mapsize in some cases. If I remember correctly.) mdb_env_copy() and mdb_env_copyfd() should document that @node This call can trigger siginficant file size growth if run in parallel with write transactions, because it employs a read-only transaction. See long-lived transactions under Caveats. I wonder that means these functions should have a flag parameter. For some databases, users might prefer a _write_ transaction while reading the DB, blocking writes while copying. -- Hallvard

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by hyc＠symas.com 31 Jul '13

31 Jul '13

Dimitrij Denissenko wrote: > Makes sense, but out of interest: shouldn't the file size drop again, > when locked pages are released? No. All allocated pages get reused internally, so there's no point giving them back to the OS. > > On 31/07/13 15:34, Howard Chu wrote: >> dimitrij.denissenko(a)blacksquaremedia.com wrote: >>> Hi again, >>> >>> I was able to recreate the scenario in a test case. I must have mixed >>> something up previously. After more tests it seems that the memory >>> growth only appears when transactions are reset within the child. >>> >>> Here's the full code: >>> https://gist.github.com/dim/6122131 >>> >>> Output on my machine: >>> >>> $ rm -f testdb/* && mkdir -p testdb && make mtest8 && ./mtest8 >>> gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast >>> -c mtest8.c >>> gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast >>> mtest8.o liblmdb.a -o mtest8 >>> NO FORK : 1699840 >>> FORK & OPEN : 1699840 >>> FORK & COPY : 6225920 >> >> This behavior is expected. Your child process is holding a read txn >> while it performs the env_copyfd(). During that time writers are not >> able to reuse old data pages, so all writes consume new pages. When >> the copyfd completes and releases its read txn the writers will resume >> reusing old pages. >> >> There's no bug here, this ITS will be closed. >> > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by dimitrij.denissenko＠blacksquaremedia.com 31 Jul '13

31 Jul '13

Makes sense, but out of interest: shouldn't the file size drop again, when locked pages are released? On 31/07/13 15:34, Howard Chu wrote: > dimitrij.denissenko(a)blacksquaremedia.com wrote: >> Hi again, >> >> I was able to recreate the scenario in a test case. I must have mixed >> something up previously. After more tests it seems that the memory >> growth only appears when transactions are reset within the child. >> >> Here's the full code: >> https://gist.github.com/dim/6122131 >> >> Output on my machine: >> >> $ rm -f testdb/* && mkdir -p testdb && make mtest8 && ./mtest8 >> gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast >> -c mtest8.c >> gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast >> mtest8.o liblmdb.a -o mtest8 >> NO FORK : 1699840 >> FORK & OPEN : 1699840 >> FORK & COPY : 6225920 > > This behavior is expected. Your child process is holding a read txn > while it performs the env_copyfd(). During that time writers are not > able to reuse old data pages, so all writes consume new pages. When > the copyfd completes and releases its read txn the writers will resume > reusing old pages. > > There's no bug here, this ITS will be closed. >

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by hyc＠symas.com 31 Jul '13

31 Jul '13

dimitrij.denissenko(a)blacksquaremedia.com wrote: > Hi again, > > I was able to recreate the scenario in a test case. I must have mixed > something up previously. After more tests it seems that the memory > growth only appears when transactions are reset within the child. > > Here's the full code: > https://gist.github.com/dim/6122131 > > Output on my machine: > > $ rm -f testdb/* && mkdir -p testdb && make mtest8 && ./mtest8 > gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast > -c mtest8.c > gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast > mtest8.o liblmdb.a -o mtest8 > NO FORK : 1699840 > FORK & OPEN : 1699840 > FORK & COPY : 6225920 This behavior is expected. Your child process is holding a read txn while it performs the env_copyfd(). During that time writers are not able to reuse old data pages, so all writes consume new pages. When the copyfd completes and releases its read txn the writers will resume reusing old pages. There's no bug here, this ITS will be closed. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by dimitrij.denissenko＠blacksquaremedia.com 31 Jul '13

31 Jul '13

Hi again, I was able to recreate the scenario in a test case. I must have mixed something up previously. After more tests it seems that the memory growth only appears when transactions are reset within the child. Here's the full code: https://gist.github.com/dim/6122131 Output on my machine: $ rm -f testdb/* && mkdir -p testdb && make mtest8 && ./mtest8 gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast -c mtest8.c gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast mtest8.o liblmdb.a -o mtest8 NO FORK : 1699840 FORK & OPEN : 1699840 FORK & COPY : 6225920 Thanks, Dimitrij On 30/07/13 23:34, Hallvard Breien Furuseth wrote: > Whoops, too hasty reply. Didn't notice your mdb_env_close(env) > before the fork. But do you mdb_env_create() again after? > mdb_env_close() destroys the MDB_env. >

1 0

(ITS#7653) Test Build Version 2.4.36 hangs
by howard.allison＠pensionsversicherung.at 31 Jul '13

31 Jul '13

Full_Name: Howard Allison Version: 2.4.36 OS: AIX URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (194.153.217.248) AIX 6.1 Using IBM VAC Compiler Configure and Build works, test hangs at test005 Stack Trace reading symbolic information ... stopped in __fd_select at 0xd0262f2c ($t2) 0xd0262f2c (__fd_select+0xcc) 80410014 lwz r2,0x14(r1) (dbx) where __fd_select(??, ??, ??, ??, ??) at 0xd0262f2c daemon.select(__fds = 0, __readlist = (nil), __writelist = (nil), __exceptlist = (nil), __timeout = (nil)), line 234 in "time.h" unnamed block in slapd_daemon_task(ptr = (nil)), line 2536 in "daemon.c" unnamed block in slapd_daemon_task(ptr = (nil)), line 2536 in "daemon.c" slapd_daemon_task(ptr = (nil)), line 2536 in "daemon.c"

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by dimitrij.denissenko＠blacksquaremedia.com 31 Jul '13

31 Jul '13

--001a11c21408c992a604e2c913b8 Content-Type: text/plain; charset=UTF-8 Yes, sorry, my example wasn't very good. I call mdb_env_create in both, the parent and the child. I will create an mtest8.c today to illustrate the problem. On 30 Jul 2013 23:34, "Hallvard Breien Furuseth" <h.b.furuseth(a)usit.uio.no> wrote: > Whoops, too hasty reply. Didn't notice your mdb_env_close(env) > before the fork. But do you mdb_env_create() again after? > mdb_env_close() destroys the MDB_env. > > -- > Hallvard > --001a11c21408c992a604e2c913b8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Yes, sorry, my example wasn't very good. I call mdb_env_= create in both, the parent and the child. I will create an mtest8.c today t= o illustrate the problem. <div class=3D"gmail_quote">On 30 Jul 2013 23:34, "Hallvard Breien Furu= seth" <<a href=3D"mailto:h.b.furuseth@usit.uio.no">h.b.furuseth@usi= t.uio.no</a>> wrote: <blockquote class=3D"gmail_= quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1= ex"> Whoops, too hasty reply. =C2=A0Didn't notice your mdb_env_close(env)<br= > before the fork. =C2=A0But do you mdb_env_create() again after? mdb_env_close() destroys the MDB_env. -- Hallvard </blockquote></div> --001a11c21408c992a604e2c913b8--

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by h.b.furuseth＠usit.uio.no 31 Jul '13

31 Jul '13

Whoops, too hasty reply. Didn't notice your mdb_env_close(env) before the fork. But do you mdb_env_create() again after? mdb_env_close() destroys the MDB_env. -- Hallvard

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by h.b.furuseth＠usit.uio.no 31 Jul '13

31 Jul '13

dimitrij.denissenko(a)blacksquaremedia.com writes: > I found an interesting issue with LMDB. I have populated the DB with a bunch of > records and it uses ~30M on disk (after sync). Then I added a background process > to my app and populated the database again with the same record set. mdb_env_create(); fork(); mdb_env_open(); is invalid. An MDB_env may only be used in the process which created it. See Caveats in lmdb.h. This is because mdb_env_create() sets the pid which the MDB_env will use in the lockfile. We could delay this to mdb_env_open(), then your particular program would work but fork() after mdb_env_open() would not. But I don't see much point in doing so. -- Hallvard

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by hyc＠symas.com 30 Jul '13

30 Jul '13

dimitrij.denissenko(a)blacksquaremedia.com wrote: > --001a11c2ed64c222e504e2c14561 > Content-Type: text/plain; charset=UTF-8 > > Hi, > > All writes occur in the parent process only. The child (normally) only > reopens the environment and performs a few short reads. > > But, it's the actual opening of the env in the forked child that is causing > the database growth. I tried to close the env straight after opening it in > the child (without performing any reads), and have encountered the same > issues. I don't see anything like that here. There's nothing in mdb_env_open(MDB_NOSYNC) that can even affect the size of the DB file. I don't think there's anything we can investigate without sample code that reproduces the situation. > > Hope that makes sense, > Dimitrij > On 30 Jul 2013 21:19, "Howard Chu" <hyc(a)symas.com> wrote: > >> dimitrij.denissenko(a)**blacksquaremedia.com<dimitrij.denissenko(a)blacksquaremedia.com>wrote: >> >>> Full_Name: Dimitrij Denissenko >>> Version: >>> OS: Ubuntu 12.04 >>> URL: >>> Submission from: (NULL) (62.30.100.0) >>> >>> >>> Hi, >>> >>> I found an interesting issue with LMDB. I have populated the DB with a >>> bunch of >>> records and it uses ~30M on disk (after sync). Then I added a background >>> process >>> to my app and populated the database again with the same record set. >>> Surprisingly. the resulting size on disk was >70M. >>> >>> The background process is forked periodically to perform some maintenance >>> tasks, >>> here is my (simplified) code: >>> >>> /* Close env before forking */ >>> mdb_env_close(env); >>> >>> if ((childpid = fork()) == 0) { >>> /* Child */ >>> rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); >>> ... >>> } else { >>> /* Parent */ >>> rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); >>> ... >>> } >>> >>> I could narrow it down to the mdb_env_open call in the child. If I add >>> exit(0) >>> before the mdb_env_open line, the DB size remains consistently at ~30M. >>> The data >>> size seems to grow proportionally to the number of forks performed during >>> data >>> load. What could be causing the growth? What can I do to prevent it? >>> >>> Thanks in advance >>> >>> PS: I tried it with MDB_FIXMAP and without, same result. >>> >> >> Without seeing more of your code, it's impossible to tell. Are you adding >> the data on both sides of the fork? In the above code snippet, where are >> your mdb_put calls occurring? Are both the parent and child processes >> writing identical data? >> >> -- >> -- Howard Chu >> CTO, Symas Corp. http://www.symas.com >> Director, Highland Sun http://highlandsun.com/hyc/ >> Chief Architect, OpenLDAP http://www.openldap.org/**project/<http://www.openldap.org/project/> >> > > --001a11c2ed64c222e504e2c14561 > Content-Type: text/html; charset=UTF-8 > Content-Transfer-Encoding: quoted-printable > > Hi, > All writes occur in the parent process only. The child (norm= > ally) only reopens the environment and performs a few short reads. > But, it's the actual opening of the env in the forked ch= > ild that is causing the database growth. I tried to close the env straight = > after opening it in the child (without performing any reads), and have enco= > untered the same issues. > > Hope that makes sense, > Dimitrij > <div class=3D"gmail_quote">On 30 Jul 2013 21:19, "Howard Chu" &lt= > ;<a href=3D"mailto:hyc@symas.com">hyc(a)symas.com</a>> wrote: ttribution"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo= > rder-left:1px #ccc solid;padding-left:1ex"> > <a href=3D"mailto:dimitrij.denissenko@blacksquaremedia.com" target=3D"_blan= > k">dimitrij.denissenko@blacksquaremedia.com</a> wrote: > <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= > x #ccc solid;padding-left:1ex"> > Full_Name: Dimitrij Denissenko > Version: > OS: Ubuntu 12.04 > URL: > Submission from: (NULL) (62.30.100.0) > > > Hi, > > I found an interesting issue with LMDB. I have populated the DB with a bunc= > h of > records and it uses ~30M on disk (after sync). Then I added a background pr= > ocess > to my app and populated the database again with the same record set. > Surprisingly. the resulting size on disk was >70M. > > The background process is forked periodically to perform some maintenance t= > asks, > here is my (simplified) code: > > /* Close env before forking */ > mdb_env_close(env); > > if ((childpid =3D fork()) =3D=3D 0) { > =C2=A0 =C2=A0 =C2=A0/* Child */ > =C2=A0 =C2=A0 =C2=A0rc =3D mdb_env_open(env, ".", MDB_NOSYNC, 064= > 4); > =C2=A0 =C2=A0 =C2=A0... > } else { > =C2=A0 =C2=A0 =C2=A0/* Parent */ > =C2=A0 =C2=A0 =C2=A0rc =3D mdb_env_open(env, ".", MDB_NOSYNC, 064= > 4); > =C2=A0 =C2=A0 =C2=A0... > } > > I could narrow it down to the mdb_env_open call in the child. If I add exit= > (0) > before the mdb_env_open line, the DB size remains consistently at ~30M. The= > data > size seems to grow proportionally to the number of forks performed during d= > ata > load. What could be causing the growth? What can I do to prevent it? > > Thanks in advance > > PS: I tried it with MDB_FIXMAP and without, same result. > </blockquote> > > Without seeing more of your code, it's impossible to tell. Are you addi= > ng the data on both sides of the fork? In the above code snippet, where are= > your mdb_put calls occurring? Are both the parent and child processes writ= > ing identical data? > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by dimitrij.denissenko＠blacksquaremedia.com 30 Jul '13

30 Jul '13

--001a11c2ed64c222e504e2c14561 Content-Type: text/plain; charset=UTF-8 Hi, All writes occur in the parent process only. The child (normally) only reopens the environment and performs a few short reads. But, it's the actual opening of the env in the forked child that is causing the database growth. I tried to close the env straight after opening it in the child (without performing any reads), and have encountered the same issues. Hope that makes sense, Dimitrij On 30 Jul 2013 21:19, "Howard Chu" <hyc(a)symas.com> wrote: > dimitrij.denissenko(a)**blacksquaremedia.com<dimitrij.denissenko(a)blacksquaremedia.com>wrote: > >> Full_Name: Dimitrij Denissenko >> Version: >> OS: Ubuntu 12.04 >> URL: >> Submission from: (NULL) (62.30.100.0) >> >> >> Hi, >> >> I found an interesting issue with LMDB. I have populated the DB with a >> bunch of >> records and it uses ~30M on disk (after sync). Then I added a background >> process >> to my app and populated the database again with the same record set. >> Surprisingly. the resulting size on disk was >70M. >> >> The background process is forked periodically to perform some maintenance >> tasks, >> here is my (simplified) code: >> >> /* Close env before forking */ >> mdb_env_close(env); >> >> if ((childpid = fork()) == 0) { >> /* Child */ >> rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); >> ... >> } else { >> /* Parent */ >> rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); >> ... >> } >> >> I could narrow it down to the mdb_env_open call in the child. If I add >> exit(0) >> before the mdb_env_open line, the DB size remains consistently at ~30M. >> The data >> size seems to grow proportionally to the number of forks performed during >> data >> load. What could be causing the growth? What can I do to prevent it? >> >> Thanks in advance >> >> PS: I tried it with MDB_FIXMAP and without, same result. >> > > Without seeing more of your code, it's impossible to tell. Are you adding > the data on both sides of the fork? In the above code snippet, where are > your mdb_put calls occurring? Are both the parent and child processes > writing identical data? > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/**project/<http://www.openldap.org/project/> > --001a11c2ed64c222e504e2c14561 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, All writes occur in the parent process only. The child (norm= ally) only reopens the environment and performs a few short reads. But, it's the actual opening of the env in the forked ch= ild that is causing the database growth. I tried to close the env straight = after opening it in the child (without performing any reads), and have enco= untered the same issues. Hope that makes sense, Dimitrij <div class=3D"gmail_quote">On 30 Jul 2013 21:19, "Howard Chu" &lt= ;<a href=3D"mailto:hyc@symas.com">hyc(a)symas.com</a>> wrote: <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo= rder-left:1px #ccc solid;padding-left:1ex"> <a href=3D"mailto:dimitrij.denissenko@blacksquaremedia.com" target=3D"_blan= k">dimitrij.denissenko@blacksquaremedia.com</a> wrote: <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Full_Name: Dimitrij Denissenko Version: OS: Ubuntu 12.04 URL: Submission from: (NULL) (62.30.100.0) Hi, I found an interesting issue with LMDB. I have populated the DB with a bunc= h of records and it uses ~30M on disk (after sync). Then I added a background pr= ocess to my app and populated the database again with the same record set. Surprisingly. the resulting size on disk was >70M. The background process is forked periodically to perform some maintenance t= asks, here is my (simplified) code: /* Close env before forking */ mdb_env_close(env); if ((childpid =3D fork()) =3D=3D 0) { =C2=A0 =C2=A0 =C2=A0/* Child */ =C2=A0 =C2=A0 =C2=A0rc =3D mdb_env_open(env, ".", MDB_NOSYNC, 064= 4); =C2=A0 =C2=A0 =C2=A0... } else { =C2=A0 =C2=A0 =C2=A0/* Parent */ =C2=A0 =C2=A0 =C2=A0rc =3D mdb_env_open(env, ".", MDB_NOSYNC, 064= 4); =C2=A0 =C2=A0 =C2=A0... } I could narrow it down to the mdb_env_open call in the child. If I add exit= (0) before the mdb_env_open line, the DB size remains consistently at ~30M. The= data size seems to grow proportionally to the number of forks performed during d= ata load. What could be causing the growth? What can I do to prevent it? Thanks in advance PS: I tried it with MDB_FIXMAP and without, same result. </blockquote> Without seeing more of your code, it's impossible to tell. Are you addi= ng the data on both sides of the fork? In the above code snippet, where are= your mdb_put calls occurring? Are both the parent and child processes writ= ing identical data? -- =C2=A0 -- Howard Chu =C2=A0 CTO, Symas Corp. =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"http:= //www.symas.com" target=3D"_blank">http://www.symas.com</a> =C2=A0 Director, Highland Sun =C2=A0 =C2=A0 <a href=3D"http://highlandsun.c= om/hyc/" target=3D"_blank">http://highlandsun.com/hyc/</a> =C2=A0 Chief Architect, OpenLDAP =C2=A0<a href=3D"http://www.openldap.org/p= roject/" target=3D"_blank">http://www.openldap.org/project/</a> </blockquote></div> --001a11c2ed64c222e504e2c14561--

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by hyc＠symas.com 30 Jul '13

30 Jul '13

dimitrij.denissenko(a)blacksquaremedia.com wrote: > Full_Name: Dimitrij Denissenko > Version: > OS: Ubuntu 12.04 > URL: > Submission from: (NULL) (62.30.100.0) > > > Hi, > > I found an interesting issue with LMDB. I have populated the DB with a bunch of > records and it uses ~30M on disk (after sync). Then I added a background process > to my app and populated the database again with the same record set. > Surprisingly. the resulting size on disk was >70M. > > The background process is forked periodically to perform some maintenance tasks, > here is my (simplified) code: > > /* Close env before forking */ > mdb_env_close(env); > > if ((childpid = fork()) == 0) { > /* Child */ > rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); > ... > } else { > /* Parent */ > rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); > ... > } > > I could narrow it down to the mdb_env_open call in the child. If I add exit(0) > before the mdb_env_open line, the DB size remains consistently at ~30M. The data > size seems to grow proportionally to the number of forks performed during data > load. What could be causing the growth? What can I do to prevent it? > > Thanks in advance > > PS: I tried it with MDB_FIXMAP and without, same result. Without seeing more of your code, it's impossible to tell. Are you adding the data on both sides of the fork? In the above code snippet, where are your mdb_put calls occurring? Are both the parent and child processes writing identical data? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7652) LMDB: mdb_copy fails on files > 2GB
by hyc＠symas.com 30 Jul '13

30 Jul '13

dimitrij.denissenko(a)blacksquaremedia.com wrote: > Full_Name: Dimitrij Denissenko > Version: > OS: Ubuntu 12.04 (64bit) > URL: > Submission from: (NULL) (62.30.100.0) > > > Hi, > > The latest mdb.master (acf20f7) fails on files that are larger than 2G, even on > 64bit systems. > > I could narrow it down to: Thanks for the report. The bogus parts of those commits have been reverted. > > --- a/mdb/libraries/liblmdb/mdb.c > +++ b/mdb/libraries/liblmdb/mdb.c > @@ -4070,7 +4070,7 @@ > else > w2 = wsize; > wres = write(fd, ptr, w2); > - rc = wres == (ssize_t)w2 ? MDB_SUCCESS : wres < 0 ? ErrCode() : EIO; > + rc = wres > 0 ? MDB_SUCCESS : ErrCode(); > if (rc) break; > wsize -= wres; > ptr += wres; > > After this change the environment was copied correctly. > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#7652) LMDB: mdb_copy fails on files > 2GB
by dimitrij.denissenko＠blacksquaremedia.com 30 Jul '13

30 Jul '13

Full_Name: Dimitrij Denissenko Version: OS: Ubuntu 12.04 (64bit) URL: Submission from: (NULL) (62.30.100.0) Hi, The latest mdb.master (acf20f7) fails on files that are larger than 2G, even on 64bit systems. I could narrow it down to: --- a/mdb/libraries/liblmdb/mdb.c +++ b/mdb/libraries/liblmdb/mdb.c @@ -4070,7 +4070,7 @@ else w2 = wsize; wres = write(fd, ptr, w2); - rc = wres == (ssize_t)w2 ? MDB_SUCCESS : wres < 0 ? ErrCode() : EIO; + rc = wres > 0 ? MDB_SUCCESS : ErrCode(); if (rc) break; wsize -= wres; ptr += wres; After this change the environment was copied correctly.

1 0

(ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by dimitrij.denissenko＠blacksquaremedia.com 30 Jul '13

30 Jul '13

Full_Name: Dimitrij Denissenko Version: OS: Ubuntu 12.04 URL: Submission from: (NULL) (62.30.100.0) Hi, I found an interesting issue with LMDB. I have populated the DB with a bunch of records and it uses ~30M on disk (after sync). Then I added a background process to my app and populated the database again with the same record set. Surprisingly. the resulting size on disk was >70M. The background process is forked periodically to perform some maintenance tasks, here is my (simplified) code: /* Close env before forking */ mdb_env_close(env); if ((childpid = fork()) == 0) { /* Child */ rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); ... } else { /* Parent */ rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); ... } I could narrow it down to the mdb_env_open call in the child. If I add exit(0) before the mdb_env_open line, the DB size remains consistently at ~30M. The data size seems to grow proportionally to the number of forks performed during data load. What could be causing the growth? What can I do to prevent it? Thanks in advance PS: I tried it with MDB_FIXMAP and without, same result.

1 0

Re: (ITS#7631) Overlay to rewrite value of attribute using regex
by bart＠icm.edu.pl 29 Jul '13

29 Jul '13

On Mon, 29 Jul 2013 08:30:23 -0700, Howard Chu wrote: > > I presume we still need a formal IPR notice in the ITS? Is it enough: The attached patch file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch were developed by Dominik Bartkiewicz bart(a)icm.edu.pl. I have not assigned rights and/or interest in this work to any party. DB

1 0

Re: (ITS#6856) OpenLDAP 2.4.23 / db 4.7.25 lockup 100% CPU
by hyc＠symas.com 29 Jul '13

29 Jul '13

requate(a)univention.de wrote: > Looks like the patch for ITS#7222 fixed this. > Thanks for the followup. Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs