openldap-bugs

openldap-bugs@openldap.org

1 participants
20965 discussions

Re: (ITS#8420) lmdb Windows sizeof(off_t) == sizeof(int32_t)
by hyc＠symas.com 10 May '16

10 May '16

clayton.stangeland(a)gmail.com wrote: > Full_Name: Clayton Stangeland > Version: lmdb master a04aad31c2 > OS: Windows 7 64 bit > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (170.54.58.5) > > > This is an lmdb bug. I see no bug here. We fixed this back in January in commit 20dec1f69bf4860202c764ce92b1fbbe3d11a065 > > mdb_page_flush fails as database size nears 2 GB on Windows. The reason is off_t > == int32_t on Windows. So the off_t pos variable becomes negative and this > section of code in mdb_page_flush enters the error condition as ov.OffsetHigh is > also negative: > > ov.Offset = pos & 0xffffffff; > ov.OffsetHigh = pos >> 16 >> 16; > if (!WriteFile(env->me_fd, dp, size, NULL, &ov)) { > rc = ErrCode(); > DPRINTF(("WriteFile: %d", rc)); > return rc; > } > > Adding23dedefine off_t int64_t in an #ifdef _WIN32 region solves this. > > > -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8421) LMDB broken for files larger than 2^31 bytes under Windows
by lukaswhl＠gmail.com 10 May '16

10 May '16

--94eb2c123f7c8355130532869195 Content-Type: text/plain; charset=UTF-8 Looks like a duplicate indeed. Didn't see ITS#8420, sorry. 2016-05-11 12:56 GMT+12:00 Quanah Gibson-Mount <quanah(a)zimbra.com>: > --On Wednesday, May 11, 2016 1:47 AM +0000 lukaswhl(a)gmail.com wrote: > > Full_Name: Lukas W >> Version: mdb.master >> OS: >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (2407:7000:9a07:8600::2) >> > > Isn't this a dupe of ITS#8420? > > --Quanah > > > > -- > > Quanah Gibson-Mount > Platform Architect > Zimbra, Inc. > -------------------- > Zimbra :: the leader in open source messaging and collaboration > A division of Synacor, Inc > --94eb2c123f7c8355130532869195 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr">Looks like a duplicate indeed. Didn't see ITS#8420, so= rry.<br></div><div class=3D"gmail_extra"><br><div class=3D"gmail_quote">201= 6-05-11 12:56 GMT+12:00 Quanah Gibson-Mount <span dir=3D"ltr"><<a href= =3D"mailto:quanah@zimbra.com" target=3D"_blank">quanah(a)zimbra.com</a>></= span>:<br><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;bord= er-left:1px #ccc solid;padding-left:1ex">--On Wednesday, May 11, 2016 1:47 = AM +0000 <a href=3D"mailto:lukaswhl@gmail.com" target=3D"_blank">lukaswhl@g= mail.com</a> wrote:<br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Full_Name: Lukas W<br> Version: mdb.master<br> OS:<br> URL: <a href=3D"ftp://ftp.openldap.org/incoming/" rel=3D"noreferrer" target= =3D"_blank">ftp://ftp.openldap.org/incoming/</a><br> Submission from: (NULL) (2407:7000:9a07:8600::2)<br> </blockquote> <br> Isn't this a dupe of ITS#8420?<br> <br> --Quanah<br> <br> <br> <br> --<br> <br> Quanah Gibson-Mount<br> Platform Architect<br> Zimbra, Inc.<br> --------------------<br> Zimbra ::=C2=A0 the leader in open source messaging and collaboration<br> A division of Synacor, Inc<br> </blockquote></div><br></div> --94eb2c123f7c8355130532869195--

1 0

Re: (ITS#8421) LMDB broken for files larger than 2^31 bytes under Windows
by quanah＠zimbra.com 10 May '16

10 May '16

--On Wednesday, May 11, 2016 1:47 AM +0000 lukaswhl(a)gmail.com wrote: > Full_Name: Lukas W > Version: mdb.master > OS: > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (2407:7000:9a07:8600::2) Isn't this a dupe of ITS#8420? --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration A division of Synacor, Inc

1 0

(ITS#8421) LMDB broken for files larger than 2^31 bytes under Windows
by lukaswhl＠gmail.com 10 May '16

10 May '16

Full_Name: Lukas W Version: mdb.master OS: URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (2407:7000:9a07:8600::2) off_t is defined as a long int, which is 4 bytes long in MSVC, but 8 bytes long in GCC. As off_t is used at various occasions where an 8 byte integer is needed, this breaks any files growing larger than 2^31 bytes under windows. The following patch fixes it. diff --git a/libraries/liblmdb/mdb.c b/libraries/liblmdb/mdb.c index 54170b8..652ad6e 100644 --- a/libraries/liblmdb/mdb.c +++ b/libraries/liblmdb/mdb.c @@ -1340,7 +1340,7 @@ struct MDB_env { MDB_txn *me_txn; 909/**< crerent write transaction */ MDB_txn *me_txn0; /**< prealloc'd write transaction */ mdb_size_t me_mapsize; /**< size of the data memory map */ - off_t me_size; /**< current file size */ + mdb_size_t me_size; /**< current file size */ pgno_t me_maxpg;%%0 /**< _m_mapsize / me_psize */ MDB_dbx *me_dbxs; /**< array of static DB info */ uint16_t *me_dbflags; /**< array of flags from MDB_db.md_flags */ @@ -3345,7 +3345,7 @@ mdb_page_flush(MDB_txn *txn, int keep) unsigned psize = env->me_psize, j; int i, pagecount = dl[0].mid, rc; size_t size = 0; - off_t pos = 0; + size_t pos = 0; pgno_t pgno = 0; MDB_page *dp = NULL; #ifdef _WIN32 @@ -3353,7 +3353,7 @@ mdb_page_flush(MDB_txn *txn, int keep) #else ruruct iovec iov[MDB_COMMIT_PAGES]; ssize_t wsize = 0, wres; - off_t wpos = 0, next_pos = 1; /* impossible pos, so pos != next_pos */ + ssize_t wpos = 0, next_pos = 1; /* impossible pos, so pos != next_pos */ int n = 0; #endif @@ -3860,7 +3860,7 @@ mdb_env_write_meta(MDB_txn *txn) MDB_meta meta, metab, *mp; unsigned flags; mdb_size_t mapsize; - off_t off; + mdb_size_t off; int rc, len, toggle; char *ptr; HANDLE mfd; @@ -4568,7 +4569,7 @@ mdb_hash_val(MDB_val *val, mdb_hash_t hval) static const char mdb_a85[]= "0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz!#$%&()*+-;<=>?@^_`{|}~"; static void ESECT -mdb_pack85(unsigned long l, char *out) +mdb_pack85(mdb_hash_t l, char *out) { int i; @@ -4615,7 +4616,7 @@ mdenenv_setup_locks(MDB_env *env, char *lpath, int mode, int *excl) union semun semu; #endif int rc; - off_t size, rsize; + mdb_size_t size, rsize; #ifdef _WIN32 wchar_t *wlpath;

1 0

(ITS#8420) lmdb Windows sizeof(off_t) == sizeof(int32_t)
by clayton.stangeland＠gmail.com 07 May '16

07 May '16

Full_Name: Clayton Stangeland Version: lmdb master a04aad31c2 OS: Windows 7 64 bit URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (170.54.58.5) This is an lmdb bug. mdb_page_flush fails as database size nears 2 GB on Windows. The reason is off_t == int32_t on Windows. So the off_t pos variable becomes negative and this section of code in mdb_page_flush enters the error condition as ov.OffsetHigh is also negative: ov.Offset = pos & 0xffffffff; ov.OffsetHigh = pos >> 16 >> 16; if (!WriteFile(env->me_fd, dp, size, NULL, &ov)) { rc = ErrCode(); DPRINTF(("WriteFile: %d", rc)); return rc; } Adding23dedefine off_t int64_t in an #ifdef _WIN32 region solves this.

1 0

Re: (ITS#8419) mdb_env_stat crashes when the env argument hasn't been opened yet
by hyc＠symas.com 07 May '16

07 May '16

On Sat, May 07, 2016 at 10:39:29PM +0000, aschorr(a)telemetry-investments.com wrote: > Hi, FYI, mdb_env_info has the same issue. Dup of ITS#8401, closing this ITS. > > Thanks, > Andy > > >

1 0

Re: (ITS#8419) mdb_env_stat crashes when the env argument hasn't been opened yet
by aschorr＠telemetry-investments.com 07 May '16

07 May '16

Hi, FYI, mdb_env_info has the same issue. Thanks, Andy

1 0

(ITS#8419) mdb_env_stat crashes when the env argument hasn't been opened yet
by aschorr＠telemetry-investments.com 07 May '16

07 May '16

Full_Name: Andrew Schorr Version: lmdb 0.9.18 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (38.76.0.51) I am implementing a gawk lmdb API. As a silly test, I ran the following code: gawk -l lmdb ' BEGIN { env = mdb_env_create() rc = mdb_env_stat(env, f) }' It catches a SIGSEGV. It is apparently illegal to call mdb_env_stat before calling mdb_open. Please consider updating the documentation or updating the function to protect against this situation. Thanks, Andy

1 0

Re: (ITS#8418) mdb_load performance
by aschorr＠telemetry-investments.com 06 May '16

06 May '16

Hi, sorry, please close this issue. I am unable to duplicate my initial results. The use of a cursor seems to be marginally helpful, and I no longer see any slowdown from batching the commits. I guess my server must have had some other loads on it that impacted my previous test results. I do see one odd result of batching -- the process ru_maxrss value reported by getrusage is much higher (10 GB vs 3 GB for my test data). I'm not sure why that's the case. Perhaps that resulted in some swapping during my initial test. Sorry for wasting your time. Regards, Andy On Thu, May 05, 2016 at 09:56:10PM +0000, openldap-its(a)OpenLDAP.org wrote: > > *** THIS IS AN AUTOMATICALLY GENERATED REPLY *** > > Thanks for your report to the OpenLDAP Issue Tracking System. Your > report has been assigned the tracking number ITS#8418. > > One of our support engineers will look at your report in due course. > Note that this may take some time because our support engineers > are volunteers. They only work on OpenLDAP when they have spare > time. > > If you need to provide additional information in regards to your > issue report, you may do so by replying to this message. Note that > any mail sent to openldap-its(a)openldap.org with (ITS#8418) > in the subject will automatically be attached to the issue report. > > mailto:openldap-its@openldap.org?subject=(ITS#8418) > > You may follow the progress of this report by loading the following > URL in a web browser: > http://www.OpenLDAP.org/its/index.cgi?findid=8418 > > Please remember to retain your issue tracking number (ITS#8418) > on any further messages you send to us regarding this report. If > you don't then you'll just waste our time and yours because we > won't be able to properly track the report. > > Please note that the Issue Tracking System is not intended to > be used to seek help in the proper use of OpenLDAP Software. > Such requests will be closed. > > OpenLDAP Software is user supported. > http://www.OpenLDAP.org/support/ > > -------------- > Copyright 1998-2007 The OpenLDAP Foundation, All Rights Reserved. > -- Andrew Schorr e-mail: aschorr(a)telemetry-investments.com Telemetry Investments, L.L.C. phone: 917-305-1748 545 Fifth Ave, Suite 1108 fax: 212-425-5550 New York, NY 10017-3630

1 0

(ITS#8418) mdb_load performance
by aschorr＠telemetry-investments.com 05 May '16

05 May '16

Full_Name: Andrew Schorr Version: lmdb 0.9.18 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (38.76.0.51) I patched mdb_load to add an option to set the mapsize with -T, as per open issue #8417. I then used it to load a 7 GB text file, and I found that the calls to mdb_txn_commit after every 100 mdb_cursor_put calls really slows down performance. If one comments out the batching, it goes much faster. And even faster is to avoid the cursor and just call mdb_put in a loop. Is there a reason mdb_load commits after every 100 puts?

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs