openldap-bugs

openldap-bugs@openldap.org

1 participants
20963 discussions

(ITS#8803) [PATCH] Add -L option for using external locking
by nirsof＠gmail.com 10 Feb '18

10 Feb '18

Full_Name: Nir Soffer Version: master OS: Fedora 27 URL: https://github.com/nirs/lmdb/commit/e9ffc44797f9884590c480d0f160d8f745ee8ff9 Submission from: (NULL) (217.132.180.29) A user may prefer to provide external locking using the MDB_NOLOCK when using the lmdb library. To use the mdb_dump, mdb_load, and mdb_stat with the same locking, we need an option to disable the library internal locking when running these tools. The attached patch file is derived from lmdb Software. All of the modifications to lmdb represented in the following patch(es) were developed by Nir Soffer nirsof(a)gmail.com. I have not assigned rights and/or interest in this work to any party. The attached modifications to OpenLDAP Software are subject to the following notice: Copyright 2018 Nir Soffer Redistribution and use in source and binary forms, with or without modification, are permitted only as authorized by the OpenLDAP Public License.

1 0

Re: (ITS#6268) multi-master sync replication ldap_add error code 68 bug
by quanah＠symas.com 09 Feb '18

09 Feb '18

Hi Barry, I had a few questions to ask in relation to this ITS: a) Can you reproduce this problem with current RE24? and b) Could you provide copies of your shell scripts so I can create a generalized regression test case for this? (And, if this issue is not resolved as of yet, it will help with getting it fixed). Thanks! Regards, Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>

1 0

AW: (ITS#8777) [LMDB] Closing read cursor uses already freed transaction (MDB_VL32)
by markus＠greenrobot.de 09 Feb '18

09 Feb '18

It's not that trivial. We recycle cursors because they *might* be reused later. At the point when the txn is done it is unknown whether the cursor will be used in the future.

1 0

Re: (ITS#8777) [LMDB] Closing read cursor uses already freed transaction (MDB_VL32)
by hyc＠symas.com 09 Feb '18

09 Feb '18

Howard Chu wrote: > Markus Junginger wrote: >>> There was never a bug here. The doc is quite explicit >>> http://www.lmdb.tech/doc/group__mdb.html#ga73a5938ae4c3239ee11efa07eb22b882 >>> >>> After txn_abort the only valid call on a still-open cursor is >>> cursor_renew, not cursor_close. >> >> However you want to call it, are you open to improve this workflow? >> >> API-wise, it would be so much simpler just to free a cursor after it became >> obsolete. It's counter-intuitive to open a new txn, renew the cursor, close >> the cursor, and abort the txn - just to close a cursor. > > API-wise, it is much simpler to just close the cursor before closing its > transaction, if you have no need to reuse the cursor. To be explicit - a typical workflow, as outlined in the Getting Started guide: Open an env Open a txn inside the env Open a cursor inside the txn When finished: Close cursor inside txn Close txn inside env Close env The only reason the API allows a read cursor to exist outside of its creating read txn is to allow it to be reused in a later read txn. If you're not reusing it then you should just close it in the proper order. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8777) [LMDB] Closing read cursor uses already freed transaction (MDB_VL32)
by hyc＠symas.com 09 Feb '18

09 Feb '18

Markus Junginger wrote: >> There was never a bug here. The doc is quite explicit >> http://www.lmdb.tech/doc/group__mdb.html#ga73a5938ae4c3239ee11efa07eb22b882 >> >> After txn_abort the only valid call on a still-open cursor is >> cursor_renew, not cursor_close. > > However you want to call it, are you open to improve this workflow? > > API-wise, it would be so much simpler just to free a cursor after it became > obsolete. It's counter-intuitive to open a new txn, renew the cursor, close > the cursor, and abort the txn - just to close a cursor. API-wise, it is much simpler to just close the cursor before closing its transaction, if you have no need to reuse the cursor. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: (ITS#8777) [LMDB] Closing read cursor uses already freed transaction (MDB_VL32)
by markus＠greenrobot.de 09 Feb '18

09 Feb '18

> There was never a bug here. The doc is quite explicit > http://www.lmdb.tech/doc/group__mdb.html#ga73a5938ae4c3239ee11efa07eb22b882 > > After txn_abort the only valid call on a still-open cursor is > cursor_renew, not cursor_close. However you want to call it, are you open to improve this workflow? API-wise, it would be so much simpler just to free a cursor after it became obsolete. It's counter-intuitive to open a new txn, renew the cursor, close the cursor, and abort the txn - just to close a cursor. And without MDB_VL32, plain cursor closing after txn abort seems to work just fine (I did not consider "closing" as "using" when I read the docs). One could say that the inconsistent behavior is a result of the complexity MDB_VL32 introduces. If you don't want to change the current behavior, it would still be nice for future developers to null the obsolete txn ptr in the cursor on txn abort to make the unwanted close call crash always, not just occasionally.

1 0

Re: AW: AW: (ITS#8777) [LMDB] Closing read cursor uses already freed transaction (MDB_VL32)
by hyc＠symas.com 09 Feb '18

09 Feb '18

Markus Junginger wrote: > I perfectly see that MDB_VL32 is the superior as a feature. On the other > hand I'm a bit worried that it may hold up LMDB development for use cases > you refer to as obsolete yourself. > > While there are workarounds for this particular bug (which we applied), There was never a bug here. The doc is quite explicit http://www.lmdb.tech/doc/group__mdb.html#ga73a5938ae4c3239ee11efa07eb22b882 After txn_abort the only valid call on a still-open cursor is cursor_renew, not cursor_close. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

AW: AW: (ITS#8777) [LMDB] Closing read cursor uses already freed transaction (MDB_VL32)
by markus＠greenrobot.de 09 Feb '18

09 Feb '18

I perfectly see that MDB_VL32 is the superior as a feature. On the other hand I'm a bit worried that it may hold up LMDB development for use cases you refer to as obsolete yourself. While there are workarounds for this particular bug (which we applied), I wouldn't be surprised if MDB_VL32 still needs some stabilization after it will be officially released to get real exposure in the real world. Also, for future changes, the MDB_VL32 "fork" will have to be adjusted as well.

1 0

Re: AW: (ITS#8777) [LMDB] Closing read cursor uses already freed transaction (MDB_VL32)
by hyc＠symas.com 09 Feb '18

09 Feb '18

markus(a)greenrobot.de wrote: >> When using MDB_VL32 to access a 64bit DB on a 32bit CPU, the DB cannot fit >> into a single mmap. >> [...] > > Long story short: I did not understand MDB_VL32 in the first place. I > thought it would just enable 64 bit sizes. Pointless, if you're still limited to 2GB. It would only eat up 4 extra bytes overhead in every internal integer. >> Ideally, you should quit using obsolete 32bit CPUs. > > Of course. Think about mobile and IoT however: Here, there are still 32 bit > CPUs and it's not uncommon to even make 64 bit ARM CPUs operate in 32 bit > mode to reduce binary size etc. > > I'd like to propose a different approach to 64 bits on 32 bit CPUs; > something like this: > #ifdef MDB_SIZE64 > typedef uint64_t mdb_size_t; > ... > > Why? MDB_VL32 adds complexity and it would take additional coding and > testing efforts to get it production ready. If you had used the API as intended, with txn_reset/renew as I originally suggested, there would be no problem. > > I think MDB_SIZE64 might be a better trade-off. Of course this would only > work with smaller DBs files (< 2GB/4GB). But I don't see a huge scenario for > 32 bit CPUs being able to work with huge files anyway. The reason MDB_VL32 exists at all is because of exactly this - people wanted to use 32 bit CPUs with 64 bit databases. > The main reason I see > to use 64 bits consistently across CPU architecture is binary compatibility. > This is a big thing for Android (shipping apps with prefilled DBs). Also it > enables getting DB files from 32 bit devices and open those on the desktop, > etc. > > What do you think? Bad idea. MDB_VL32 provides *full* compatibility between 32bit and 64bit CPUs *regardless of filesize*. Your suggestion will only cause confusion, as people wonder why their DB dies after reaching 2GB in size. It offers none of the benefits of 64bit capability, while still paying all of the cost in terms of storage overhead. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

AW: (ITS#8777) [LMDB] Closing read cursor uses already freed transaction (MDB_VL32)
by markus＠greenrobot.de 09 Feb '18

09 Feb '18

> When using MDB_VL32 to access a 64bit DB on a 32bit CPU, the DB cannot fit > into a single mmap. > [...] Long story short: I did not understand MDB_VL32 in the first place. I thought it would just enable 64 bit sizes. > Ideally, you should quit using obsolete 32bit CPUs. Of course. Think about mobile and IoT however: Here, there are still 32 bit CPUs and it's not uncommon to even make 64 bit ARM CPUs operate in 32 bit mode to reduce binary size etc. I'd like to propose a different approach to 64 bits on 32 bit CPUs; something like this: #ifdef MDB_SIZE64 typedef uint64_t mdb_size_t; ... Why? MDB_VL32 adds complexity and it would take additional coding and testing efforts to get it production ready. I think MDB_SIZE64 might be a better trade-off. Of course this would only work with smaller DBs files (< 2GB/4GB). But I don't see a huge scenario for 32 bit CPUs being able to work with huge files anyway. The main reason I see to use 64 bits consistently across CPU architecture is binary compatibility. This is a big thing for Android (shipping apps with prefilled DBs). Also it enables getting DB files from 32 bit devices and open those on the desktop, etc. What do you think? Thanks, Markus PS.: Here's the full patch (adding MDB_SIZE64), which runs fine at least with our test suite: lmdb.h @@ -187,7 +187,15 @@ # define MDB_FMT_Z "z" /**< printf/scanf format modifier for size_t */ #endif -#ifndef MDB_VL32 +#if defined(MDB_VL32) && defined(MDB_SIZE64) +#error either define MDB_VL32 OR MDB_SIZE64 +#endif +#ifdef MDB_SIZE64 +typedef uint64_t mdb_size_t; +# define MDB_SIZE_MAX SIZE_MAX /**< max #mdb_size_t */ +# define MDB_PRIy(t) PRI##t##64 +# define MDB_SCNy(t) SCN##t##64 +#elif !defined(MDB_VL32)) /** Unsigned type used for mapsize, entry counts and page/transaction IDs. * * It is normally size_t, hence the name. Defining MDB_VL32 makes it

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs