openldap-bugs

openldap-bugs@openldap.org

1 participants
21000 discussions

[Issue 9574] New: mdb_drop missing from .gitignore
by openldap-its＠openldap.org 07 Jun '21

07 Jun '21

https://bugs.openldap.org/show_bug.cgi?id=9574 Issue ID: 9574 Summary: mdb_drop missing from .gitignore Product: LMDB Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: --- Component: liblmdb Assignee: bugs(a)openldap.org Reporter: max(a)davitt.me Target Milestone: --- The .gitignore file contains entries for mdb_copy, mdb_stat, mdb_dump, and mdb_load, but not mdb_drop. This is mildly annoying while building. Normally I would've just created a merge request directly on GitLab to fix something like this but I'm having issues with it (see #9573). Sorry for the inconvenience! -- You are receiving this mail because: You are on the CC list for the issue.

1 2

[Issue 9027] LMDB: API to retrieve map address
by openldap-its＠openldap.org 05 Jun '21

05 Jun '21

https://bugs.openldap.org/show_bug.cgi?id=9027 --- Comment #4 from Howard Chu <hyc(a)openldap.org> --- (In reply to doug from comment #3) > Hi! I would like to voice my support for this feature, and explain how this > API would help me. Sorry but I don't find any of these arguments to be compelling. > > First of all, a bit of background: I'm maintaining C++ bindings for LMDB > (https://github.com/hoytech/lmdbxx), and have been a contributor to the perl > bindings and others. I have published two interface layers that use LMDB, > https://github.com/hoytech/quadrable and > https://github.com/hoytech/rasgueadb and as my colleagues would attest, am a > relentless LMDB advocate. > > Application 1: Testing zero-copy > > That previous link, RasgueaDB, is an indexing and query layer for LMDB. It > uses flatbuffers for its data format, which allows zero-copy access to > fields. Since there are several layers involved, I wanted to make sure that > no copying was happening, and also to add a test to the test-suite to make > sure this remains the case. > > To do this, I have a library called assert_zerocopy: > https://github.com/hoytech/hoytech-cpp/blob/master/hoytech/assert_zerocopy.h > (and also a perl equivalent here: https://metacpan.org/pod/Test::ZeroCopy ) > > However, in order to verify the returned value is actually a zero-copy > reference, I need the pointer to the memory map (and the map's size). I have > added a terrible hack to the lmdbxx bindings to retrieve this: > > https://github.com/hoytech/lmdbxx/blob/ > 08eddafcc4613c7fc8ebd88f5db87c7d7bfb9f52/lmdb%2B%2B.h#L1115-L1118 > > But this is obviously not a good approach (as mentioned by Nic in this > thread) and it would be much better if there was an API to retrieve this > value (like there is for me_mapsize). No. The only thing you need to do, to assert that no copying has occurred, is to also retrieve the record thru the standard LMDB API, and compare value pointers. If they are the same, then no copying has occurred. At any rate, this sounds like only a debugging feature, with no actual production use. > Application 2: Virtual memory controls > > In general I agree that mlock()ing large files like databases is > counter-productive. However, there are other controls that can be applied to > the virtual memory of an application. > > For example, rsync attempts to preserve the filesystem cache state so that > rsync invocations have as small an impact as possible on the cache (since > it's a shared resource): https://insights.oetiker.ch/linux/fadvise/ > > Another example, before restarting a server, Instagram uses my utility > vmtouch to snapshot the virtual memory state (basically the set of "hot" > files). After rebooting, the state is restored before the server is added > back to the active pool of servers. > > I have generalised this with my application/library vmprobe: > https://vmprobe.com/filesystem-cache > > If I had access to me_map within my application, I would be able to > integrate libvmprobe more easily and portably. For instance, this would > allow me to take a snapshot of which pages are resident in memory, perform > some large read query that touches many pages, then restore the original VM > page residency set after the query completes. > > Another variant of this: If you have multiple customers who use the same > server (at different times) then you can restore a customer's previous VM > state when they login. If you know approximately the pages that will be > accessed in advance, sequentially pre-paging them in can have significant > performance benefits. I did some experiments with Postgres pre-paging you > can see here: https://vmprobe.com/database-speedup If you want control over the address that is mmap'd just use MDB_FIXEDMAP and pass in your desired address. The actions you're talking about here are highly system-dependent, you may as well just read /proc/<PID>/maps yourself and work from there. Again, this is a pretty niche case. > > Conclusion > > Although certainly a user could abuse me_map if it were exposed, one of the > reasons why I enjoy LMDB so much is that it generally assumes its users know > what they are doing and doesn't compromise on flexibility or performance for > the sake of users who don't. For example, although it is not recommended to > use MDB_WRITEMAP in most cases, it nevertheless exists. > > As explained above, I have several use-cases for me_map that I feel are > legitimate. I am already accessing it in a hacky way but naturally would > prefer a more portable and future-proof API. > > If this API is added, the documentation should of course make clear that > accessing or modifying (if MDB_WRITEMAP) data through me_map is not > supported in any way, and regular use-cases have no need for this value and > should stick to the supported APIs. > > Thank you! -- You are receiving this mail because: You are on the CC list for the issue.

1 0

[Issue 9027] LMDB: API to retrieve map address
by openldap-its＠openldap.org 05 Jun '21

05 Jun '21

https://bugs.openldap.org/show_bug.cgi?id=9027 --- Comment #3 from doug(a)hcsw.org --- Hi! I would like to voice my support for this feature, and explain how this API would help me. First of all, a bit of background: I'm maintaining C++ bindings for LMDB (https://github.com/hoytech/lmdbxx), and have been a contributor to the perl bindings and others. I have published two interface layers that use LMDB, https://github.com/hoytech/quadrable and https://github.com/hoytech/rasgueadb and as my colleagues would attest, am a relentless LMDB advocate. Application 1: Testing zero-copy That previous link, RasgueaDB, is an indexing and query layer for LMDB. It uses flatbuffers for its data format, which allows zero-copy access to fields. Since there are several layers involved, I wanted to make sure that no copying was happening, and also to add a test to the test-suite to make sure this remains the case. To do this, I have a library called assert_zerocopy: https://github.com/hoytech/hoytech-cpp/blob/master/hoytech/assert_zerocopy.h (and also a perl equivalent here: https://metacpan.org/pod/Test::ZeroCopy ) However, in order to verify the returned value is actually a zero-copy reference, I need the pointer to the memory map (and the map's size). I have added a terrible hack to the lmdbxx bindings to retrieve this: https://github.com/hoytech/lmdbxx/blob/08eddafcc4613c7fc8ebd88f5db87c7d7bfb… But this is obviously not a good approach (as mentioned by Nic in this thread) and it would be much better if there was an API to retrieve this value (like there is for me_mapsize). Application 2: Virtual memory controls In general I agree that mlock()ing large files like databases is counter-productive. However, there are other controls that can be applied to the virtual memory of an application. For example, rsync attempts to preserve the filesystem cache state so that rsync invocations have as small an impact as possible on the cache (since it's a shared resource): https://insights.oetiker.ch/linux/fadvise/ Another example, before restarting a server, Instagram uses my utility vmtouch to snapshot the virtual memory state (basically the set of "hot" files). After rebooting, the state is restored before the server is added back to the active pool of servers. I have generalised this with my application/library vmprobe: https://vmprobe.com/filesystem-cache If I had access to me_map within my application, I would be able to integrate libvmprobe more easily and portably. For instance, this would allow me to take a snapshot of which pages are resident in memory, perform some large read query that touches many pages, then restore the original VM page residency set after the query completes. Another variant of this: If you have multiple customers who use the same server (at different times) then you can restore a customer's previous VM state when they login. If you know approximately the pages that will be accessed in advance, sequentially pre-paging them in can have significant performance benefits. I did some experiments with Postgres pre-paging you can see here: https://vmprobe.com/database-speedup Conclusion Although certainly a user could abuse me_map if it were exposed, one of the reasons why I enjoy LMDB so much is that it generally assumes its users know what they are doing and doesn't compromise on flexibility or performance for the sake of users who don't. For example, although it is not recommended to use MDB_WRITEMAP in most cases, it nevertheless exists. As explained above, I have several use-cases for me_map that I feel are legitimate. I am already accessing it in a hacky way but naturally would prefer a more portable and future-proof API. If this API is added, the documentation should of course make clear that accessing or modifying (if MDB_WRITEMAP) data through me_map is not supported in any way, and regular use-cases have no need for this value and should stick to the supported APIs. Thank you! -- You are receiving this mail because: You are on the CC list for the issue.

1 0

[Issue 9570] New: Roadmap needs updating now that OpenLDAP 2.5 is released
by openldap-its＠openldap.org 04 Jun '21

04 Jun '21

https://bugs.openldap.org/show_bug.cgi?id=9570 Issue ID: 9570 Summary: Roadmap needs updating now that OpenLDAP 2.5 is released Product: website Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: --- Component: website Assignee: bugs(a)openldap.org Reporter: quanah(a)openldap.org Target Milestone: --- Need to update the roadmap. -- You are receiving this mail because: You are on the CC list for the issue.

1 3

[Issue 9557] New: syncrepl refreshonly sometimes finishes too early
by openldap-its＠openldap.org 03 Jun '21

03 Jun '21

https://bugs.openldap.org/show_bug.cgi?id=9557 Issue ID: 9557 Summary: syncrepl refreshonly sometimes finishes too early Product: OpenLDAP Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Keywords: replication Severity: normal Priority: --- Component: slapd Assignee: bugs(a)openldap.org Reporter: ondra(a)mistotebe.net Target Milestone: --- In a delta-MMR test environment where conflicts are intentionally introduced, the fallback sessions sometimes cut off, not sending all entries they ought to. Instead LDAP_SUCCESS is sent with the snapshot cookie at some point. That cookie represents the correct contextCSN at the time of the search, so the missing changes are skipped over forever. As of now, the cause of the cutoff is under investigation. Currently, this was only run with 2.5 + persistent sessionlog enabled but there are indications this might not be limited to 2.5, tests with other versions/configs are pending. -- You are receiving this mail because: You are on the CC list for the issue.

1 7

[Issue 9551] New: dnSubtreeMatch and others do not handle empty DN as asserted value
by openldap-its＠openldap.org 03 Jun '21

03 Jun '21

https://bugs.openldap.org/show_bug.cgi?id=9551 Issue ID: 9551 Summary: dnSubtreeMatch and others do not handle empty DN as asserted value Product: OpenLDAP Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: --- Component: slapd Assignee: bugs(a)openldap.org Reporter: ondra(a)mistotebe.net Target Milestone: --- AFAIK '(reqDN:dnSubtreeMatch:=)' should be equivalent to 'reqDN=*', however it only seems to match the empty dn right now. -- You are receiving this mail because: You are on the CC list for the issue.

1 4

[Issue 9537] New: slap_timestamp() can give a duplicated timestamp across restarts
by openldap-its＠openldap.org 03 Jun '21

03 Jun '21

https://bugs.openldap.org/show_bug.cgi?id=9537 Issue ID: 9537 Summary: slap_timestamp() can give a duplicated timestamp across restarts Product: OpenLDAP Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: --- Component: slapd Assignee: bugs(a)openldap.org Reporter: ondra(a)mistotebe.net Target Milestone: --- On busy sites, when a slapd restart takes <1s, accesslog can fail to log changes with LDAP_ALREADY_EXISTS. This is because slap_timestamp() only logs timestamps with a 1s precision, disambiguating the rest with a counter that's forgotten across restarts. It is possible my analysis in ITS#9487 is partially invalidated because of this. -- You are receiving this mail because: You are on the CC list for the issue.

1 7

[Issue 9526] New: slapadd -w crashes
by openldap-its＠openldap.org 03 Jun '21

03 Jun '21

https://bugs.openldap.org/show_bug.cgi?id=9526 Issue ID: 9526 Summary: slapadd -w crashes Product: OpenLDAP Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: --- Component: slapd Assignee: bugs(a)openldap.org Reporter: grapvar(a)gmail.com Target Milestone: --- Let slapd.conf is: > database mdb > suffix "o=Foo" > sync_use_subentry database is blank and we are adding this foo.ldif: > dn: o=FOO > objectClass:organization Let's load: > slapd -T add -v -l foo.ldif -w then on Solaris: > added: "o=FOO" (00000001) > Segmentation Fault (core dumped) ... on Linux: > added: "o=FOO" (00000001) > => mdb_next_id: get failed: Invalid argument (22) > => mdb_tool_next_id: next_id failed: Invalid argument (22) > => mdb_tool_entry_put: txn_aborted! Invalid argument (22) > slapadd: couldn't create context entry > Closing DB... This is because: * mdb_tool_next_id() takes dead global [tools.c`static MDB_cursor *mcp] for further operations * cursor is dead because mdb_tool_entry_put() didn't initialized it * mdb_tool_entry_put() didn't initialized cursor because it thinks it is initialized, because there is an active global [tools.c`MDB_txn *mdb_tool_txn] * transaction was initialized by mdb_tool_dn2id_get(), which doesn't care about cursors. Long story short: the global state in tools.c is not managed consistently and needs rethinking. -- You are receiving this mail because: You are on the CC list for the issue.

1 9

[Issue 6467] syncrepl enhancements
by openldap-its＠openldap.org 03 Jun '21

03 Jun '21

https://bugs.openldap.org/show_bug.cgi?id=6467 Quanah Gibson-Mount <quanah(a)openldap.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |VERIFIED -- You are receiving this mail because: You are on the CC list for the issue.

1 0

[Issue 9552] New: slapo-accesslog should record new DN after rename
by openldap-its＠openldap.org 03 Jun '21

03 Jun '21

https://bugs.openldap.org/show_bug.cgi?id=9552 Issue ID: 9552 Summary: slapo-accesslog should record new DN after rename Product: OpenLDAP Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: --- Component: overlays Assignee: bugs(a)openldap.org Reporter: ondra(a)mistotebe.net Target Milestone: --- To use accesslog as a sessionlog source, one needs to be able to resolve whether a modrdn moved an entry in/out of the search scope. Syncprov would either have to examine every modrdn request or, if accesslog were to log the final DN, it could check for entries which crossed the scope. A new attribute 'reqNewDN' should be tracked for modrdn ops. -- You are receiving this mail because: You are on the CC list for the issue.

1 4

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-bugs