openldap-bugs July 2013

openldap-bugs@openldap.org

23 participants
85 discussions

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by hyc＠symas.com 30 Jul '13

30 Jul '13

dimitrij.denissenko(a)blacksquaremedia.com wrote: > --001a11c2ed64c222e504e2c14561 > Content-Type: text/plain; charset=UTF-8 > > Hi, > > All writes occur in the parent process only. The child (normally) only > reopens the environment and performs a few short reads. > > But, it's the actual opening of the env in the forked child that is causing > the database growth. I tried to close the env straight after opening it in > the child (without performing any reads), and have encountered the same > issues. I don't see anything like that here. There's nothing in mdb_env_open(MDB_NOSYNC) that can even affect the size of the DB file. I don't think there's anything we can investigate without sample code that reproduces the situation. > > Hope that makes sense, > Dimitrij > On 30 Jul 2013 21:19, "Howard Chu" <hyc(a)symas.com> wrote: > >> dimitrij.denissenko(a)**blacksquaremedia.com<dimitrij.denissenko(a)blacksquaremedia.com>wrote: >> >>> Full_Name: Dimitrij Denissenko >>> Version: >>> OS: Ubuntu 12.04 >>> URL: >>> Submission from: (NULL) (62.30.100.0) >>> >>> >>> Hi, >>> >>> I found an interesting issue with LMDB. I have populated the DB with a >>> bunch of >>> records and it uses ~30M on disk (after sync). Then I added a background >>> process >>> to my app and populated the database again with the same record set. >>> Surprisingly. the resulting size on disk was >70M. >>> >>> The background process is forked periodically to perform some maintenance >>> tasks, >>> here is my (simplified) code: >>> >>> /* Close env before forking */ >>> mdb_env_close(env); >>> >>> if ((childpid = fork()) == 0) { >>> /* Child */ >>> rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); >>> ... >>> } else { >>> /* Parent */ >>> rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); >>> ... >>> } >>> >>> I could narrow it down to the mdb_env_open call in the child. If I add >>> exit(0) >>> before the mdb_env_open line, the DB size remains consistently at ~30M. >>> The data >>> size seems to grow proportionally to the number of forks performed during >>> data >>> load. What could be causing the growth? What can I do to prevent it? >>> >>> Thanks in advance >>> >>> PS: I tried it with MDB_FIXMAP and without, same result. >>> >> >> Without seeing more of your code, it's impossible to tell. Are you adding >> the data on both sides of the fork? In the above code snippet, where are >> your mdb_put calls occurring? Are both the parent and child processes >> writing identical data? >> >> -- >> -- Howard Chu >> CTO, Symas Corp. http://www.symas.com >> Director, Highland Sun http://highlandsun.com/hyc/ >> Chief Architect, OpenLDAP http://www.openldap.org/**project/<http://www.openldap.org/project/> >> > > --001a11c2ed64c222e504e2c14561 > Content-Type: text/html; charset=UTF-8 > Content-Transfer-Encoding: quoted-printable > > Hi, > All writes occur in the parent process only. The child (norm= > ally) only reopens the environment and performs a few short reads. > But, it's the actual opening of the env in the forked ch= > ild that is causing the database growth. I tried to close the env straight = > after opening it in the child (without performing any reads), and have enco= > untered the same issues. > > Hope that makes sense, > Dimitrij > <div class=3D"gmail_quote">On 30 Jul 2013 21:19, "Howard Chu" &lt= > ;<a href=3D"mailto:hyc@symas.com">hyc(a)symas.com</a>> wrote: ttribution"><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo= > rder-left:1px #ccc solid;padding-left:1ex"> > <a href=3D"mailto:dimitrij.denissenko@blacksquaremedia.com" target=3D"_blan= > k">dimitrij.denissenko@blacksquaremedia.com</a> wrote: > <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= > x #ccc solid;padding-left:1ex"> > Full_Name: Dimitrij Denissenko > Version: > OS: Ubuntu 12.04 > URL: > Submission from: (NULL) (62.30.100.0) > > > Hi, > > I found an interesting issue with LMDB. I have populated the DB with a bunc= > h of > records and it uses ~30M on disk (after sync). Then I added a background pr= > ocess > to my app and populated the database again with the same record set. > Surprisingly. the resulting size on disk was >70M. > > The background process is forked periodically to perform some maintenance t= > asks, > here is my (simplified) code: > > /* Close env before forking */ > mdb_env_close(env); > > if ((childpid =3D fork()) =3D=3D 0) { > =C2=A0 =C2=A0 =C2=A0/* Child */ > =C2=A0 =C2=A0 =C2=A0rc =3D mdb_env_open(env, ".", MDB_NOSYNC, 064= > 4); > =C2=A0 =C2=A0 =C2=A0... > } else { > =C2=A0 =C2=A0 =C2=A0/* Parent */ > =C2=A0 =C2=A0 =C2=A0rc =3D mdb_env_open(env, ".", MDB_NOSYNC, 064= > 4); > =C2=A0 =C2=A0 =C2=A0... > } > > I could narrow it down to the mdb_env_open call in the child. If I add exit= > (0) > before the mdb_env_open line, the DB size remains consistently at ~30M. The= > data > size seems to grow proportionally to the number of forks performed during d= > ata > load. What could be causing the growth? What can I do to prevent it? > > Thanks in advance > > PS: I tried it with MDB_FIXMAP and without, same result. > </blockquote> > > Without seeing more of your code, it's impossible to tell. Are you addi= > ng the data on both sides of the fork? In the above code snippet, where are= > your mdb_put calls occurring? Are both the parent and child processes writ= > ing identical data? > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by dimitrij.denissenko＠blacksquaremedia.com 30 Jul '13

30 Jul '13

--001a11c2ed64c222e504e2c14561 Content-Type: text/plain; charset=UTF-8 Hi, All writes occur in the parent process only. The child (normally) only reopens the environment and performs a few short reads. But, it's the actual opening of the env in the forked child that is causing the database growth. I tried to close the env straight after opening it in the child (without performing any reads), and have encountered the same issues. Hope that makes sense, Dimitrij On 30 Jul 2013 21:19, "Howard Chu" <hyc(a)symas.com> wrote: > dimitrij.denissenko(a)**blacksquaremedia.com<dimitrij.denissenko(a)blacksquaremedia.com>wrote: > >> Full_Name: Dimitrij Denissenko >> Version: >> OS: Ubuntu 12.04 >> URL: >> Submission from: (NULL) (62.30.100.0) >> >> >> Hi, >> >> I found an interesting issue with LMDB. I have populated the DB with a >> bunch of >> records and it uses ~30M on disk (after sync). Then I added a background >> process >> to my app and populated the database again with the same record set. >> Surprisingly. the resulting size on disk was >70M. >> >> The background process is forked periodically to perform some maintenance >> tasks, >> here is my (simplified) code: >> >> /* Close env before forking */ >> mdb_env_close(env); >> >> if ((childpid = fork()) == 0) { >> /* Child */ >> rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); >> ... >> } else { >> /* Parent */ >> rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); >> ... >> } >> >> I could narrow it down to the mdb_env_open call in the child. If I add >> exit(0) >> before the mdb_env_open line, the DB size remains consistently at ~30M. >> The data >> size seems to grow proportionally to the number of forks performed during >> data >> load. What could be causing the growth? What can I do to prevent it? >> >> Thanks in advance >> >> PS: I tried it with MDB_FIXMAP and without, same result. >> > > Without seeing more of your code, it's impossible to tell. Are you adding > the data on both sides of the fork? In the above code snippet, where are > your mdb_put calls occurring? Are both the parent and child processes > writing identical data? > > -- > -- Howard Chu > CTO, Symas Corp. http://www.symas.com > Director, Highland Sun http://highlandsun.com/hyc/ > Chief Architect, OpenLDAP http://www.openldap.org/**project/<http://www.openldap.org/project/> > --001a11c2ed64c222e504e2c14561 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, All writes occur in the parent process only. The child (norm= ally) only reopens the environment and performs a few short reads. But, it's the actual opening of the env in the forked ch= ild that is causing the database growth. I tried to close the env straight = after opening it in the child (without performing any reads), and have enco= untered the same issues. Hope that makes sense, Dimitrij <div class=3D"gmail_quote">On 30 Jul 2013 21:19, "Howard Chu" &lt= ;<a href=3D"mailto:hyc@symas.com">hyc(a)symas.com</a>> wrote: <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bo= rder-left:1px #ccc solid;padding-left:1ex"> <a href=3D"mailto:dimitrij.denissenko@blacksquaremedia.com" target=3D"_blan= k">dimitrij.denissenko@blacksquaremedia.com</a> wrote: <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Full_Name: Dimitrij Denissenko Version: OS: Ubuntu 12.04 URL: Submission from: (NULL) (62.30.100.0) Hi, I found an interesting issue with LMDB. I have populated the DB with a bunc= h of records and it uses ~30M on disk (after sync). Then I added a background pr= ocess to my app and populated the database again with the same record set. Surprisingly. the resulting size on disk was >70M. The background process is forked periodically to perform some maintenance t= asks, here is my (simplified) code: /* Close env before forking */ mdb_env_close(env); if ((childpid =3D fork()) =3D=3D 0) { =C2=A0 =C2=A0 =C2=A0/* Child */ =C2=A0 =C2=A0 =C2=A0rc =3D mdb_env_open(env, ".", MDB_NOSYNC, 064= 4); =C2=A0 =C2=A0 =C2=A0... } else { =C2=A0 =C2=A0 =C2=A0/* Parent */ =C2=A0 =C2=A0 =C2=A0rc =3D mdb_env_open(env, ".", MDB_NOSYNC, 064= 4); =C2=A0 =C2=A0 =C2=A0... } I could narrow it down to the mdb_env_open call in the child. If I add exit= (0) before the mdb_env_open line, the DB size remains consistently at ~30M. The= data size seems to grow proportionally to the number of forks performed during d= ata load. What could be causing the growth? What can I do to prevent it? Thanks in advance PS: I tried it with MDB_FIXMAP and without, same result. </blockquote> Without seeing more of your code, it's impossible to tell. Are you addi= ng the data on both sides of the fork? In the above code snippet, where are= your mdb_put calls occurring? Are both the parent and child processes writ= ing identical data? -- =C2=A0 -- Howard Chu =C2=A0 CTO, Symas Corp. =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <a href=3D"http:= //www.symas.com" target=3D"_blank">http://www.symas.com</a> =C2=A0 Director, Highland Sun =C2=A0 =C2=A0 <a href=3D"http://highlandsun.c= om/hyc/" target=3D"_blank">http://highlandsun.com/hyc/</a> =C2=A0 Chief Architect, OpenLDAP =C2=A0<a href=3D"http://www.openldap.org/p= roject/" target=3D"_blank">http://www.openldap.org/project/</a> </blockquote></div> --001a11c2ed64c222e504e2c14561--

1 0

Re: (ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by hyc＠symas.com 30 Jul '13

30 Jul '13

dimitrij.denissenko(a)blacksquaremedia.com wrote: > Full_Name: Dimitrij Denissenko > Version: > OS: Ubuntu 12.04 > URL: > Submission from: (NULL) (62.30.100.0) > > > Hi, > > I found an interesting issue with LMDB. I have populated the DB with a bunch of > records and it uses ~30M on disk (after sync). Then I added a background process > to my app and populated the database again with the same record set. > Surprisingly. the resulting size on disk was >70M. > > The background process is forked periodically to perform some maintenance tasks, > here is my (simplified) code: > > /* Close env before forking */ > mdb_env_close(env); > > if ((childpid = fork()) == 0) { > /* Child */ > rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); > ... > } else { > /* Parent */ > rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); > ... > } > > I could narrow it down to the mdb_env_open call in the child. If I add exit(0) > before the mdb_env_open line, the DB size remains consistently at ~30M. The data > size seems to grow proportionally to the number of forks performed during data > load. What could be causing the growth? What can I do to prevent it? > > Thanks in advance > > PS: I tried it with MDB_FIXMAP and without, same result. Without seeing more of your code, it's impossible to tell. Are you adding the data on both sides of the fork? In the above code snippet, where are your mdb_put calls occurring? Are both the parent and child processes writing identical data? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#7652) LMDB: mdb_copy fails on files > 2GB
by hyc＠symas.com 30 Jul '13

30 Jul '13

dimitrij.denissenko(a)blacksquaremedia.com wrote: > Full_Name: Dimitrij Denissenko > Version: > OS: Ubuntu 12.04 (64bit) > URL: > Submission from: (NULL) (62.30.100.0) > > > Hi, > > The latest mdb.master (acf20f7) fails on files that are larger than 2G, even on > 64bit systems. > > I could narrow it down to: Thanks for the report. The bogus parts of those commits have been reverted. > > --- a/mdb/libraries/liblmdb/mdb.c > +++ b/mdb/libraries/liblmdb/mdb.c > @@ -4070,7 +4070,7 @@ > else > w2 = wsize; > wres = write(fd, ptr, w2); > - rc = wres == (ssize_t)w2 ? MDB_SUCCESS : wres < 0 ? ErrCode() : EIO; > + rc = wres > 0 ? MDB_SUCCESS : ErrCode(); > if (rc) break; > wsize -= wres; > ptr += wres; > > After this change the environment was copied correctly. > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

(ITS#7652) LMDB: mdb_copy fails on files > 2GB
by dimitrij.denissenko＠blacksquaremedia.com 30 Jul '13

30 Jul '13

Full_Name: Dimitrij Denissenko Version: OS: Ubuntu 12.04 (64bit) URL: Submission from: (NULL) (62.30.100.0) Hi, The latest mdb.master (acf20f7) fails on files that are larger than 2G, even on 64bit systems. I could narrow it down to: --- a/mdb/libraries/liblmdb/mdb.c +++ b/mdb/libraries/liblmdb/mdb.c @@ -4070,7 +4070,7 @@ else w2 = wsize; wres = write(fd, ptr, w2); - rc = wres == (ssize_t)w2 ? MDB_SUCCESS : wres < 0 ? ErrCode() : EIO; + rc = wres > 0 ? MDB_SUCCESS : ErrCode(); if (rc) break; wsize -= wres; ptr += wres; After this change the environment was copied correctly.

1 0

(ITS#7651) LMDB: Uncontrolled database when opened from multiple processes
by dimitrij.denissenko＠blacksquaremedia.com 30 Jul '13

30 Jul '13

Full_Name: Dimitrij Denissenko Version: OS: Ubuntu 12.04 URL: Submission from: (NULL) (62.30.100.0) Hi, I found an interesting issue with LMDB. I have populated the DB with a bunch of records and it uses ~30M on disk (after sync). Then I added a background process to my app and populated the database again with the same record set. Surprisingly. the resulting size on disk was >70M. The background process is forked periodically to perform some maintenance tasks, here is my (simplified) code: /* Close env before forking */ mdb_env_close(env); if ((childpid = fork()) == 0) { /* Child */ rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); ... } else { /* Parent */ rc = mdb_env_open(env, ".", MDB_NOSYNC, 0644); ... } I could narrow it down to the mdb_env_open call in the child. If I add exit(0) before the mdb_env_open line, the DB size remains consistently at ~30M. The data size seems to grow proportionally to the number of forks performed during data load. What could be causing the growth? What can I do to prevent it? Thanks in advance PS: I tried it with MDB_FIXMAP and without, same result.

1 0

Re: (ITS#7631) Overlay to rewrite value of attribute using regex
by bart＠icm.edu.pl 29 Jul '13

29 Jul '13

On Mon, 29 Jul 2013 08:30:23 -0700, Howard Chu wrote: > > I presume we still need a formal IPR notice in the ITS? Is it enough: The attached patch file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch were developed by Dominik Bartkiewicz bart(a)icm.edu.pl. I have not assigned rights and/or interest in this work to any party. DB

1 0

Re: (ITS#6856) OpenLDAP 2.4.23 / db 4.7.25 lockup 100% CPU
by hyc＠symas.com 29 Jul '13

29 Jul '13

requate(a)univention.de wrote: > Looks like the patch for ITS#7222 fixed this. > Thanks for the followup. Closing this ITS. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#6856) OpenLDAP 2.4.23 / db 4.7.25 lockup 100% CPU
by requate＠univention.de 29 Jul '13

29 Jul '13

Looks like the patch for ITS#7222 fixed this. -- Arvid Requate Open Source Software Engineer Univention GmbH be open. Mary-Somerville-Str.1 28359 Bremen Tel. : +49 421 22232-52 Fax : +49 421 22232-99 requate(a)univention.de http://www.univention.de Geschäftsführer: Peter H. Ganten HRB 20755 Amtsgericht Bremen Steuer-Nr.: 71-597-02876

1 0

Re: (ITS#7631) Overlay to rewrite value of attribute using regex
by hyc＠symas.com 29 Jul '13

29 Jul '13

bart(a)icm.edu.pl wrote: > --=_c864110a076fc513331e0370bcb9adbe > Content-Transfer-Encoding: 7bit > Content-Type: text/plain; charset=UTF-8; > format=flowed > > > I put patch here because the ftp server is full. > DB Kurt, the source file in the patch contains this header: @@ -0,0 +1,467 @@ +/* rewatt.c - Overlay to rewrite some attribute */ +/* Redistribution and use in source and binary forms, with or without + * modification, are permitted only as authorized by the OpenLDAP + * Public License. + * + * A copy of this license is available in the file LICENSE in the + * top-level directory of the distribution or, alternatively, at + * <http://www.OpenLDAP.org/license.html>. + */ +/* ACKNOWLEDGEMENTS: + * This work was originally developed by the Dominik Bartkiewicz bart(a)icm.edu.pl for + * inclusion in OpenLDAP Software. + */ I presume we still need a formal IPR notice in the ITS? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

← Newer
1
2
3
4
5
6
7
8
9
Older →

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs July 2013