openldap-bugs

openldap-bugs@openldap.org

1 participants
20961 discussions

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by dimitrij.denissenko＠blacksquaremedia.com 31 Jul '13

31 Jul '13

Hi again, I was able to recreate the scenario in a test case. I must have mixed something up previously. After more tests it seems that the memory growth only appears when transactions are reset within the child. Here's the full code: https://gist.github.com/dim/6122131 Output on my machine: $ rm -f testdb/* && mkdir -p testdb && make mtest8 && ./mtest8 gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast -c mtest8.c gcc -pthread -O2 -g -W -Wall -Wno-unused-parameter -Wbad-function-cast mtest8.o liblmdb.a -o mtest8 NO FORK : 1699840 FORK & OPEN : 1699840 FORK & COPY : 6225920 Thanks, Dimitrij On 30/07/13 23:34, Hallvard Breien Furuseth wrote: > Whoops, too hasty reply. Didn't notice your mdb_env_close(env) > before the fork. But do you mdb_env_create() again after? > mdb_env_close() destroys the MDB_env. >

1 0

(ITS#7653) Test Build Version 2.4.36 hangs
by howard.allison＠pensionsversicherung.at 31 Jul '13

31 Jul '13

Full_Name: Howard Allison Version: 2.4.36 OS: AIX URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (194.153.217.248) AIX 6.1 Using IBM VAC Compiler Configure and Build works, test hangs at test005 Stack Trace reading symbolic information ... stopped in __fd_select at 0xd0262f2c ($t2) 0xd0262f2c (__fd_select+0xcc) 80410014 lwz r2,0x14(r1) (dbx) where __fd_select(??, ??, ??, ??, ??) at 0xd0262f2c daemon.select(__fds = 0, __readlist = (nil), __writelist = (nil), __exceptlist = (nil), __timeout = (nil)), line 234 in "time.h" unnamed block in slapd_daemon_task(ptr = (nil)), line 2536 in "daemon.c" unnamed block in slapd_daemon_task(ptr = (nil)), line 2536 in "daemon.c" slapd_daemon_task(ptr = (nil)), line 2536 in "daemon.c"

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by dimitrij.denissenko＠blacksquaremedia.com 31 Jul '13

31 Jul '13

--001a11c21408c992a604e2c913b8 Content-Type: text/plain; charset=UTF-8 Yes, sorry, my example wasn't very good. I call mdb_env_create in both, the parent and the child. I will create an mtest8.c today to illustrate the problem. On 30 Jul 2013 23:34, "Hallvard Breien Furuseth" <h.b.furuseth(a)usit.uio.no> wrote: > Whoops, too hasty reply. Didn't notice your mdb_env_close(env) > before the fork. But do you mdb_env_create() again after? > mdb_env_close() destroys the MDB_env. > > -- > Hallvard > --001a11c21408c992a604e2c913b8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Yes, sorry, my example wasn't very good. I call mdb_env_= create in both, the parent and the child. I will create an mtest8.c today t= o illustrate the problem. <div class=3D"gmail_quote">On 30 Jul 2013 23:34, "Hallvard Breien Furu= seth" <<a href=3D"mailto:h.b.furuseth@usit.uio.no">h.b.furuseth@usi= t.uio.no</a>> wrote: <blockquote class=3D"gmail_= quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1= ex"> Whoops, too hasty reply. =C2=A0Didn't notice your mdb_env_close(env)<br= > before the fork. =C2=A0But do you mdb_env_create() again after? mdb_env_close() destroys the MDB_env. -- Hallvard </blockquote></div> --001a11c21408c992a604e2c913b8--

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by h.b.furuseth＠usit.uio.no 31 Jul '13

31 Jul '13

Whoops, too hasty reply. Didn't notice your mdb_env_close(env) before the fork. But do you mdb_env_create() again after? mdb_env_close() destroys the MDB_env. -- Hallvard

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by h.b.furuseth＠usit.uio.no 31 Jul '13

31 Jul '13

dimitrij.denissenko(a)blacksquaremedia.com writes: > I found an interesting issue with LMDB. I have populated the DB with a bunch of > records and it uses ~30M on disk (after sync). Then I added a background process > to my app and populated the database again with the same record set. mdb_env_create(); fork(); mdb_env_open(); is invalid. An MDB_env may only be used in the process which created it. See Caveats in lmdb.h. This is because mdb_env_create() sets the pid which the MDB_env will use in the lockfile. We could delay this to mdb_env_open(), then your particular program would work but fork() after mdb_env_open() would not. But I don't see much point in doing so. -- Hallvard

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by hyc＠symas.com 30 Jul '13

30 Jul '13

dimitrij.denissenko(a)blacksquaremedia.com wrote: > --001a11c2ed64c222e504e2c14561 > Content-Type: text/plain; charset=UTF-8 > > Hi, > > All writes occur in the parent process only. The child (normally) only > reopens the environment and performs a few short reads. > > But, it's the actual opening of the env in the forked child that is causing > the database growth. I tried to close the env straight after opening it in > the child (without performing any reads), and have encountered the same > issues. I don't see anything like that here. There's nothing in mdb_env_open(MDB_NOSYNC) that can even affect the size of the DB file. I don't think there's anything we can investigate without sample code that reproduces the situation. > > Hope that makes sense, > Dimitrij > On 30 Jul 2013 21:19, "Howard Chu" <hyc(a)symas.com> wrote: > >> dimitrij.denissenko(a)**blacksquaremedia.com<dimitrij.denissenko(a)blacksquaremedia.com>wrote: >> >>> Full_Name: Dimitrij Denissenko >>> Version: >>> OS: Ubuntu 12.04 >>> URL: >>> Submission from: (NULL) (62.30.100.0) >>> >>> >>> Hi, >>> >>> I found an interesting issue with LMDB. I have populated the DB with a >>> bunch of >>> records and it uses ~30M on disk (after sync). Then I added a background >>> process >>> to my app and populated the database again with the same record set. >>> Surprisingly. the resulting size on disk was >70M. >>> >>> The background process is forked periodically to perform some maintenance >>> tasks, >>> here is my (simplified) code: >>> >>> /* Close env before forking */ >>> mdb_env_close(env); >>> >>> if ((childpid = fork()) == 0) { >>> /* Child */ >>> rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); >>> ... >>> } else { >>> /* Parent */ >>> rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); >>> ... >>> } >>> >>> I could narrow it down to the mdb_env_open call in the child. If I add >>> exit(0) >>> before the mdb_env_open line, the DB size remains consistently at ~30M. >>> The data >>> size seems to grow proportionally to the number of forks performed during >>> data >>> load. What could be causing the growth? What can I do to prevent it? >>> >>> Thanks in advance >>> >>> PS: I tried it with MDB_FIXMAP and without, same result. >>> >> >> Without seeing more of your code, it's impossible to tell. Are you adding >> the data on both sides of the fork? In the above code snippet, where are >> your mdb_put calls occurring? Are both the parent and child processes >> writing identical data? >> >> -- >> -- Howard Chu >> CTO, Symas Corp. http://www.symas.com >> Director, Highland Sun http://highlandsun.com/hyc/ >> Chief Architect, OpenLDAP http://www.openldap.org/**project/<http://www.openldap.org/project/> >> > > --001a11c2ed64c222e504e2c14561 > Content-Type: text/html; charset=UTF-8 > Content-Transfer-Encoding: quoted-printable > > Hi, > All writes occur in the parent process only. The child (norm= > ally) only reopens the environment and performs a few short reads. > But, it's the actual opening of the env in the forked ch= > ild that is causing the database growth. I tried to close the env straight = > after opening it in the child (without performing any reads), and have enco= > untered the same issues. > > Hope that makes sense, > Dimitrij > <div class=3D"gmail_quote">On 30 Jul 2013 21:19, "Howard Chu" &lt= > ;<a href=3D"mailto:hyc@symas.com">hyc(a)symas.com</a>> wrote: ttribution"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo= > rder-left:1px #ccc solid;padding-left:1ex"> > <a href=3D"mailto:dimitrij.denissenko@blacksquaremedia.com" target=3D"_blan= > k">dimitrij.denissenko@blacksquaremedia.com</a> wrote: > <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= > x #ccc solid;padding-left:1ex"> > Full_Name: Dimitrij Denissenko > Version: > OS: Ubuntu 12.04 > URL: > Submission from: (NULL) (62.30.100.0) > > > Hi, > > I found an interesting issue with LMDB. I have populated the DB with a bunc= > h of > records and it uses ~30M on disk (after sync). Then I added a background pr= > ocess > to my app and populated the database again with the same record set. > Surprisingly. the resulting size on disk was >70M. > > The background process is forked periodically to perform some maintenance t= > asks, > here is my (simplified) code: > > /* Close env before forking */ > mdb_env_close(env); > > if ((childpid =3D fork()) =3D=3D 0) { > =C2=A0 =C2=A0 =C2=A0/* Child */ > =C2=A0 =C2=A0 =C2=A0rc =3D mdb_env_open(env, ".", MDB_NOSYNC, 064= > 4); > =C2=A0 =C2=A0 =C2=A0... > } else { > =C2=A0 =C2=A0 =C2=A0/* Parent */ > =C2=A0 =C2=A0 =C2=A0rc =3D mdb_env_open(env, ".", MDB_NOSYNC, 064= > 4); > =C2=A0 =C2=A0 =C2=A0... > } > > I could narrow it down to the mdb_env_open call in the child. If I add exit= > (0) > before the mdb_env_open line, the DB size remains consistently at ~30M. The= > data > size seems to grow proportionally to the number of forks performed during d= > ata > load. What could be causing the growth? What can I do to prevent it? > > Thanks in advance > > PS: I tried it with MDB_FIXMAP and without, same result. > </blockquote> > > Without seeing more of your code, it's impossible to tell. Are you addi= > ng the data on both sides of the fork? In the above code snippet, where are= > your mdb_put calls occurring? Are both the parent and child processes writ= > ing identical data? > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by dimitrij.denissenko＠blacksquaremedia.com 30 Jul '13

30 Jul '13

--001a11c2ed64c222e504e2c14561 Content-Type: text/plain; charset=UTF-8 Hi, All writes occur in the parent process only. The child (normally) only reopens the environment and performs a few short reads. But, it's the actual opening of the env in the forked child that is causing the database growth. I tried to close the env straight after opening it in the child (without performing any reads), and have encountered the same issues. Hope that makes sense, Dimitrij On 30 Jul 2013 21:19, "Howard Chu" <hyc(a)symas.com> wrote: > dimitrij.denissenko(a)**blacksquaremedia.com<dimitrij.denissenko(a)blacksquaremedia.com>wrote: > >> Full_Name: Dimitrij Denissenko >> Version: >> OS: Ubuntu 12.04 >> URL: >> Submission from: (NULL) (62.30.100.0) >> >> >> Hi, >> >> I found an interesting issue with LMDB. I have populated the DB with a >> bunch of >> records and it uses ~30M on disk (after sync). Then I added a background >> process >> to my app and populated the database again with the same record set. >> Surprisingly. the resulting size on disk was >70M. >> >> The background process is forked periodically to perform some maintenance >> tasks, >> here is my (simplified) code: >> >> /* Close env before forking */ >> mdb_env_close(env); >> >> if ((childpid = fork()) == 0) { >> /* Child */ >> rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); >> ... >> } else { >> /* Parent */ >> rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); >> ... >> } >> >> I could narrow it down to the mdb_env_open call in the child. If I add >> exit(0) >> before the mdb_env_open line, the DB size remains consistently at ~30M. >> The data >> size seems to grow proportionally to the number of forks performed during >> data >> load. What could be causing the growth? What can I do to prevent it? >> >> Thanks in advance >> >> PS: I tried it with MDB_FIXMAP and without, same result. >> > > Without seeing more of your code, it's impossible to tell. Are you adding > the data on both sides of the fork? In the above code snippet, where are > your mdb_put calls occurring? Are both the parent and child processes > writing identical data? > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/**project/<http://www.openldap.org/project/> > --001a11c2ed64c222e504e2c14561 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, All writes occur in the parent process only. The child (norm= ally) only reopens the environment and performs a few short reads. But, it's the actual opening of the env in the forked ch= ild that is causing the database growth. I tried to close the env straight = after opening it in the child (without performing any reads), and have enco= untered the same issues. Hope that makes sense, Dimitrij <div class=3D"gmail_quote">On 30 Jul 2013 21:19, "Howard Chu" &lt= ;<a href=3D"mailto:hyc@symas.com">hyc(a)symas.com</a>> wrote: <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo= rder-left:1px #ccc solid;padding-left:1ex"> <a href=3D"mailto:dimitrij.denissenko@blacksquaremedia.com" target=3D"_blan= k">dimitrij.denissenko@blacksquaremedia.com</a> wrote: <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Full_Name: Dimitrij Denissenko Version: OS: Ubuntu 12.04 URL: Submission from: (NULL) (62.30.100.0) Hi, I found an interesting issue with LMDB. I have populated the DB with a bunc= h of records and it uses ~30M on disk (after sync). Then I added a background pr= ocess to my app and populated the database again with the same record set. Surprisingly. the resulting size on disk was >70M. The background process is forked periodically to perform some maintenance t= asks, here is my (simplified) code: /* Close env before forking */ mdb_env_close(env); if ((childpid =3D fork()) =3D=3D 0) { =C2=A0 =C2=A0 =C2=A0/* Child */ =C2=A0 =C2=A0 =C2=A0rc =3D mdb_env_open(env, ".", MDB_NOSYNC, 064= 4); =C2=A0 =C2=A0 =C2=A0... } else { =C2=A0 =C2=A0 =C2=A0/* Parent */ =C2=A0 =C2=A0 =C2=A0rc =3D mdb_env_open(env, ".", MDB_NOSYNC, 064= 4); =C2=A0 =C2=A0 =C2=A0... } I could narrow it down to the mdb_env_open call in the child. If I add exit= (0) before the mdb_env_open line, the DB size remains consistently at ~30M. The= data size seems to grow proportionally to the number of forks performed during d= ata load. What could be causing the growth? What can I do to prevent it? Thanks in advance PS: I tried it with MDB_FIXMAP and without, same result. </blockquote> Without seeing more of your code, it's impossible to tell. Are you addi= ng the data on both sides of the fork? In the above code snippet, where are= your mdb_put calls occurring? Are both the parent and child processes writ= ing identical data? -- =C2=A0 -- Howard Chu =C2=A0 CTO, Symas Corp. =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"http:= //www.symas.com" target=3D"_blank">http://www.symas.com</a> =C2=A0 Director, Highland Sun =C2=A0 =C2=A0 <a href=3D"http://highlandsun.c= om/hyc/" target=3D"_blank">http://highlandsun.com/hyc/</a> =C2=A0 Chief Architect, OpenLDAP =C2=A0<a href=3D"http://www.openldap.org/p= roject/" target=3D"_blank">http://www.openldap.org/project/</a> </blockquote></div> --001a11c2ed64c222e504e2c14561--

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by hyc＠symas.com 30 Jul '13

30 Jul '13

dimitrij.denissenko(a)blacksquaremedia.com wrote: > Full_Name: Dimitrij Denissenko > Version: > OS: Ubuntu 12.04 > URL: > Submission from: (NULL) (62.30.100.0) > > > Hi, > > I found an interesting issue with LMDB. I have populated the DB with a bunch of > records and it uses ~30M on disk (after sync). Then I added a background process > to my app and populated the database again with the same record set. > Surprisingly. the resulting size on disk was >70M. > > The background process is forked periodically to perform some maintenance tasks, > here is my (simplified) code: > > /* Close env before forking */ > mdb_env_close(env); > > if ((childpid = fork()) == 0) { > /* Child */ > rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); > ... > } else { > /* Parent */ > rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); > ... > } > > I could narrow it down to the mdb_env_open call in the child. If I add exit(0) > before the mdb_env_open line, the DB size remains consistently at ~30M. The data > size seems to grow proportionally to the number of forks performed during data > load. What could be causing the growth? What can I do to prevent it? > > Thanks in advance > > PS: I tried it with MDB_FIXMAP and without, same result. Without seeing more of your code, it's impossible to tell. Are you adding the data on both sides of the fork? In the above code snippet, where are your mdb_put calls occurring? Are both the parent and child processes writing identical data? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7652) LMDB: mdb_copy fails on files > 2GB
by hyc＠symas.com 30 Jul '13

30 Jul '13

dimitrij.denissenko(a)blacksquaremedia.com wrote: > Full_Name: Dimitrij Denissenko > Version: > OS: Ubuntu 12.04 (64bit) > URL: > Submission from: (NULL) (62.30.100.0) > > > Hi, > > The latest mdb.master (acf20f7) fails on files that are larger than 2G, even on > 64bit systems. > > I could narrow it down to: Thanks for the report. The bogus parts of those commits have been reverted. > > --- a/mdb/libraries/liblmdb/mdb.c > +++ b/mdb/libraries/liblmdb/mdb.c > @@ -4070,7 +4070,7 @@ > else > w2 = wsize; > wres = write(fd, ptr, w2); > - rc = wres == (ssize_t)w2 ? MDB_SUCCESS : wres < 0 ? ErrCode() : EIO; > + rc = wres > 0 ? MDB_SUCCESS : ErrCode(); > if (rc) break; > wsize -= wres; > ptr += wres; > > After this change the environment was copied correctly. > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#7652) LMDB: mdb_copy fails on files > 2GB
by dimitrij.denissenko＠blacksquaremedia.com 30 Jul '13

30 Jul '13

Full_Name: Dimitrij Denissenko Version: OS: Ubuntu 12.04 (64bit) URL: Submission from: (NULL) (62.30.100.0) Hi, The latest mdb.master (acf20f7) fails on files that are larger than 2G, even on 64bit systems. I could narrow it down to: --- a/mdb/libraries/liblmdb/mdb.c +++ b/mdb/libraries/liblmdb/mdb.c @@ -4070,7 +4070,7 @@ else w2 = wsize; wres = write(fd, ptr, w2); - rc = wres == (ssize_t)w2 ? MDB_SUCCESS : wres < 0 ? ErrCode() : EIO; + rc = wres > 0 ? MDB_SUCCESS : ErrCode(); if (rc) break; wsize -= wres; ptr += wres; After this change the environment was copied correctly.

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs