July 2013 - openldap-bugs

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes

by hyc＠symas.com

h.b.furuseth(a)usit.uio.no wrote: > I'd call this two doc bugs: > > Something should mention that once a page has been used, the file will > never shrink below that page. (Can't quite say that the file never > shrinks, since WRITEMAP and Windows will set it to exact re-configurable > mapsize in some cases. If I remember correctly.) You cannot shrink the mapsize below that last used page, and the last used page is a non-decreasing number. > mdb_env_copy() and mdb_env_copyfd() should document that > > @node This call can trigger siginficant file size growth if run in > parallel with write transactions, because it employs a read-only > transaction. See long-lived transactions under Caveats. > > I wonder that means these functions should have a flag parameter. > For some databases, users might prefer a _write_ transaction while > reading the DB, blocking writes while copying. No one has ever wanted a hot-backup function that halts writes. You might as well not do a hot-backup at all. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

10 years, 1 month

1
0
0 / 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes

by h.b.furuseth＠usit.uio.no

I'd call this two doc bugs: Something should mention that once a page has been used, the file will never shrink below that page. (Can't quite say that the file never shrinks, since WRITEMAP and Windows will set it to exact re-configurable mapsize in some cases. If I remember correctly.) mdb_env_copy() and mdb_env_copyfd() should document that @node This call can trigger siginficant file size growth if run in parallel with write transactions, because it employs a read-only transaction. See long-lived transactions under Caveats. I wonder that means these functions should have a flag parameter. For some databases, users might prefer a _write_ transaction while reading the DB, blocking writes while copying. -- Hallvard

10 years, 1 month

1
0
0 / 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes

by hyc＠symas.com

Dimitrij Denissenko wrote: > Makes sense, but out of interest: shouldn't the file size drop again, > when locked pages are released? No. All allocated pages get reused internally, so there's no point giving them back to the OS. > > On 31/07/13 15:34, Howard Chu wrote: >> dimitrij.denissenko(a)blacksquaremedia.com wrote: >>> Hi again, >>> >>> I was able to recreate the scenario in a test case. I must have mixed >>> something up previously. After more tests it seems that the memory >>> growth only appears when transactions are reset within the child. >>> >>> Here's the full code: >>> https://gist.github.com/dim/6122131 >>> >>> Output on my machine: >>> >>> $ rm -f testdb/* && mkdir -p testdb && make mtest8 && ./mtest8 >>> gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast >>> -c mtest8.c >>> gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast >>> mtest8.o liblmdb.a -o mtest8 >>> NO FORK : 1699840 >>> FORK & OPEN : 1699840 >>> FORK & COPY : 6225920 >> >> This behavior is expected. Your child process is holding a read txn >> while it performs the env_copyfd(). During that time writers are not >> able to reuse old data pages, so all writes consume new pages. When >> the copyfd completes and releases its read txn the writers will resume >> reusing old pages. >> >> There's no bug here, this ITS will be closed. >> > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

10 years, 1 month

1
0
0 / 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes

by dimitrij.denissenko＠blacksquaremedia.com

Makes sense, but out of interest: shouldn't the file size drop again, when locked pages are released? On 31/07/13 15:34, Howard Chu wrote: > dimitrij.denissenko(a)blacksquaremedia.com wrote: >> Hi again, >> >> I was able to recreate the scenario in a test case. I must have mixed >> something up previously. After more tests it seems that the memory >> growth only appears when transactions are reset within the child. >> >> Here's the full code: >> https://gist.github.com/dim/6122131 >> >> Output on my machine: >> >> $ rm -f testdb/* && mkdir -p testdb && make mtest8 && ./mtest8 >> gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast >> -c mtest8.c >> gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast >> mtest8.o liblmdb.a -o mtest8 >> NO FORK : 1699840 >> FORK & OPEN : 1699840 >> FORK & COPY : 6225920 > > This behavior is expected. Your child process is holding a read txn > while it performs the env_copyfd(). During that time writers are not > able to reuse old data pages, so all writes consume new pages. When > the copyfd completes and releases its read txn the writers will resume > reusing old pages. > > There's no bug here, this ITS will be closed. >

10 years, 1 month

1
0
0 / 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes

by hyc＠symas.com

dimitrij.denissenko(a)blacksquaremedia.com wrote: > Hi again, > > I was able to recreate the scenario in a test case. I must have mixed > something up previously. After more tests it seems that the memory > growth only appears when transactions are reset within the child. > > Here's the full code: > https://gist.github.com/dim/6122131 > > Output on my machine: > > $ rm -f testdb/* && mkdir -p testdb && make mtest8 && ./mtest8 > gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast > -c mtest8.c > gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast > mtest8.o liblmdb.a -o mtest8 > NO FORK : 1699840 > FORK & OPEN : 1699840 > FORK & COPY : 6225920 This behavior is expected. Your child process is holding a read txn while it performs the env_copyfd(). During that time writers are not able to reuse old data pages, so all writes consume new pages. When the copyfd completes and releases its read txn the writers will resume reusing old pages. There's no bug here, this ITS will be closed. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

10 years, 1 month

1
0
0 / 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes

by dimitrij.denissenko＠blacksquaremedia.com

Hi again, I was able to recreate the scenario in a test case. I must have mixed something up previously. After more tests it seems that the memory growth only appears when transactions are reset within the child. Here's the full code: https://gist.github.com/dim/6122131 Output on my machine: $ rm -f testdb/* && mkdir -p testdb && make mtest8 && ./mtest8 gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast -c mtest8.c gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast mtest8.o liblmdb.a -o mtest8 NO FORK : 1699840 FORK & OPEN : 1699840 FORK & COPY : 6225920 Thanks, Dimitrij On 30/07/13 23:34, Hallvard Breien Furuseth wrote: > Whoops, too hasty reply. Didn't notice your mdb_env_close(env) > before the fork. But do you mdb_env_create() again after? > mdb_env_close() destroys the MDB_env. >

10 years, 1 month

1
0
0 / 0

(ITS#7653) Test Build Version 2.4.36 hangs

by howard.allison＠pensionsversicherung.at

Full_Name: Howard Allison Version: 2.4.36 OS: AIX URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (194.153.217.248) AIX 6.1 Using IBM VAC Compiler Configure and Build works, test hangs at test005 Stack Trace reading symbolic information ... stopped in __fd_select at 0xd0262f2c ($t2) 0xd0262f2c (__fd_select+0xcc) 80410014 lwz r2,0x14(r1) (dbx) where __fd_select(??, ??, ??, ??, ??) at 0xd0262f2c daemon.select(__fds = 0, __readlist = (nil), __writelist = (nil), __exceptlist = (nil), __timeout = (nil)), line 234 in "time.h" unnamed block in slapd_daemon_task(ptr = (nil)), line 2536 in "daemon.c" unnamed block in slapd_daemon_task(ptr = (nil)), line 2536 in "daemon.c" slapd_daemon_task(ptr = (nil)), line 2536 in "daemon.c"

10 years, 1 month

1
0
0 / 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes

by dimitrij.denissenko＠blacksquaremedia.com

--001a11c21408c992a604e2c913b8 Content-Type: text/plain; charset=UTF-8 Yes, sorry, my example wasn't very good. I call mdb_env_create in both, the parent and the child. I will create an mtest8.c today to illustrate the problem. On 30 Jul 2013 23:34, "Hallvard Breien Furuseth" <h.b.furuseth(a)usit.uio.no> wrote: > Whoops, too hasty reply. Didn't notice your mdb_env_close(env) > before the fork. But do you mdb_env_create() again after? > mdb_env_close() destroys the MDB_env. > > -- > Hallvard > --001a11c21408c992a604e2c913b8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <p dir=3D"ltr">Yes, sorry, my example wasn't very good. I call mdb_env_= create in both, the parent and the child. I will create an mtest8.c today t= o illustrate the problem.</p> <div class=3D"gmail_quote">On 30 Jul 2013 23:34, "Hallvard Breien Furu= seth" <<a href=3D"mailto:h.b.furuseth@usit.uio.no">h.b.furuseth@usi= t.uio.no</a>> wrote:<br type=3D"attribution"><blockquote class=3D"gmail_= quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1= ex"> Whoops, too hasty reply. =C2=A0Didn't notice your mdb_env_close(env)<br= > before the fork. =C2=A0But do you mdb_env_create() again after?<br> mdb_env_close() destroys the MDB_env.<br> <br> --<br> Hallvard<br> </blockquote></div> --001a11c21408c992a604e2c913b8--

10 years, 1 month

1
0
0 / 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes

by h.b.furuseth＠usit.uio.no

Whoops, too hasty reply. Didn't notice your mdb_env_close(env) before the fork. But do you mdb_env_create() again after? mdb_env_close() destroys the MDB_env. -- Hallvard

10 years, 1 month

1
0
0 / 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes

by h.b.furuseth＠usit.uio.no

dimitrij.denissenko(a)blacksquaremedia.com writes: > I found an interesting issue with LMDB. I have populated the DB with a bunch of > records and it uses ~30M on disk (after sync). Then I added a background process > to my app and populated the database again with the same record set. mdb_env_create(); fork(); mdb_env_open(); is invalid. An MDB_env may only be used in the process which created it. See Caveats in lmdb.h. This is because mdb_env_create() sets the pid which the MDB_env will use in the lockfile. We could delay this to mdb_env_open(), then your particular program would work but fork() after mdb_env_open() would not. But I don't see much point in doing so. -- Hallvard

10 years, 1 month

1
0
0 / 0

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs July 2013