openldap-devel May 2015

openldap-devel@openldap.org

12 participants
10 discussions

Fwd: LMDB and text encoding
by Timur Kristóf 07 Jun '17

07 Jun '17

> I just had a look at how BDB handled this. As you can see they used a > TO_TSTRING macro to convert incoming pathnames from UTF8 to UTF16. > > https://gitorious.org/berkeleydb/berkeleydb/source/347d239a1e44ed4f773ae927… > > https://gitorious.org/berkeleydb/berkeleydb/source/347d239a1e44ed4f773ae927… > > (And a FROM_TSTRING for the reverse, as well.) (Mea culpa, I accidentally hit "reply" instead of "reply all". Sorry. Now reposting to the mailing list.) Since we only need to do this on Windows, we could use MultiByteToWideChar with CP_UTF8. (That's what TO_TSTRING does, too.) I do not think we would ever need to do any such conversion on UNIX. https://msdn.microsoft.com/en-us/library/windows/desktop/dd319072%28v=vs.85… I'm not sure if we can just copy-paste BDB's code. Probably not, that would lead to licensing issues, wouldn't it?

4 16

Does a LMDB database file ever shrink, after row deletion, if so when?
by Tinker 23 May '15

23 May '15

Hi! Is there any mechanism in LMDB to give back disk space after some particular amount of row deletion, how does this work, if not are there any plans for it now? If there's not, as a user I need 2x disk space and some mechanism to be able to migrate, so automatic shrinking would be a great thing. Thanks!! Tinker

2 1

*prio* SQLightning/LMDB OpenBSD problem: ./sqlite3 fails with "unknown error" on start and sqlite3_open() fails with 2 = SQLITE_INTERNAL. Did I build it incorrectly or do we need to resolve a bug?
by Tinker 23 May '15

23 May '15

Hi! On OpenBSD 5.5 AMD64, I compile SQLightning as follows: > git clone https://github.com/LMDB/lmdb.git > cd lmdb > git checkout mdb.RE/0.9 > cd libraries/liblmdb > make > sudo make install > (On Linux: sudo ldconfig) > cd ../../.. > > git clone https://github.com/LMDB/sqlightning.git > cd sqlightning > git checkout mdb > ./configure (trying "make" here complained that "mdb.c" doesn't exist on Linux and that "mdb.h" doesn't exist on OpenBSD, so I fixed this by "cp ../lmdb/libraries/liblmdb/* src/" and then): > make -j2 > sudo make install > (On Linux: sudo ldconfig) I then try the newly built sqlite3 binary: > ./sqlite3 /tmp/zz > SQLite version 3.7.17 2013-05-20 00:56:22 > Enter ".help" for instructions > Enter SQL statements terminated with a ";" > sqlite> create table a(b); > Error: unable to open database "/tmp/zz": UNKNOWN ERROR Equally so, sqlite3_open() fails with 2 = SQLITE_INTERNAL , http://www.sqlite.org/rescode.html#internal [1] . Both SQLightning, LMDB and SQLite are built specifically to work on OpenBSD (except for in the read-only mmap+fwrite()+fsync() mode, how to switch that off or on is beyond my current knowledge). How fix? Thank you very much! Tinker Links: ------ [1] http://www.sqlite.org/rescode.html#internal

1 0

lmdb: is read operation permitted inside the write transaction?
by Dmytro Milinevskyy 19 May '15

19 May '15

Hello, this might be a trivial question for you but I didn't find anything in the documentation nor any mention in the examples. Basically I need to have an opportunity to call mdb_get while in write transaction. The sequence is smth like: - txn = mdb_txn_begin(flags=0) for i in 0..x: - v = mdb_get(txn, i) - mdb_put(txn, x+i, v) - mdb_put(txn, i, v+1) - mdb_txn_commit(txn) Will it be always valid data? I'm running a stress test and didn't spot any issues so far. Though I might be extremely lucky today. The DB is used in multi-threaded environment but the code illustrated above is protected with a mutex. I tried to dig a bit into the lmdb internals but unfortunately didn't get any glue. Thanks for your help, Dmytro

4 5

LMDB-Patches and SQLightning-Issues
by Johannes Munk (P5) 17 May '15

17 May '15

Dear Howard and OpenLDAP-team! First of all: thanks a lot for the great LMDB and all the effort you have put into it. In the last couple of weeks we have been trying to get LMDB/SQLightning to work as backend for our semantic network software. On that road we made LMDB compatible with Win8.1/MSVC2013 (see patches here <https://gitlab.com/lukas-w/lmdb/commits/p5>), fixing scrambled filenames <https://gitlab.com/lukas-w/lmdb/commit/e10837272788f87ce43049bbbd3c9bcc0071…> and made SQLightning work with files greater than 4 GB <https://gitlab.com/lukas-w/sqlightning/commit/e52a9949fd5c30b7ac45b0909259c…>. But after some simple tests that worked, we ran into some issues, which we can't get a handle on. We have reported most of our issues here: https://gitlab.com/mdb/sqlightning/issues It would be great, if you could take a peek! Thx a lot in advance, Johannes

2 4

ITS#8102, syncrepl concurrency
by Howard Chu 13 May '15

13 May '15

We should probably change the consumer such that when multiple consumers are configured on a context, only one of them can be active at a time. In particular, in refresh phase, you'll get multiple redundant copies of the refresh data. All providers should be equivalent, so whichever one we connect to first should be used and any others should be left idle. Or, each session should be established completely sequentially - do a complete refresh from provider #1, then do the same for provider #2, etc... In a quiescent setup, the refreshes after provider #1 will be no-ops because the consumer is already up to date. In a busy setup, each refresh will still have work to do but it will be a smaller volume after provider #1's refresh completes. Simplest would be to just establish the first connection and ignore other providers unless the first connection breaks. Unfortunately this wouldn't be reliable in MMR - you could have two nodes pointed at each other, ignoring any other nodes in the configuration. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

3 2

syncrepl consumer is slow
by Howard Chu 12 May '15

12 May '15

One thing I just noticed, while testing replication with 3 servers on my laptop - during a refresh, the provider gets blocked waiting to write to the consumers after writing about 4000 entries. I.e., the consumers aren't processing fast enough to keep up with the search running on the provider. (That's actually not too surprising since reads are usually faster than writes anyway.) The consumer code has lots of problems as it is, just adding this note to the pile. I'm considering adding an option to the consumer to write its entries with dbnosync during the refresh phase. The rationale being, there's nothing to lose anyway if the refresh is interrupted. I.e., the consumer can't update its contextCSN until the very end of the refresh, so any partial refresh that gets interrupted is wasted effort - the consumer will always have to start over from the beginning on its next refresh attempt. As such, there's no point in safely/synchronously writing any of the received entries - they're useless until the final contextCSN update. The implementation approach would be to define a new control e.g. "fast write" for the consumer to pass to the underlying backend on any write op. We would also have to e.g. add an MDB_TXN_NOSYNC flag to mdb_txn_begin() (BDB already has the equivalent flag). This would only be used for writes that are part of a refresh phase. In persist mode the provider and consumers' write speeds should be more closely matched so it wouldn't be necessary or useful. Comments? -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

5 16

ITS8100: What to do about a fresh accesslog DB when in delta-sync MMR node
by Quanah Gibson-Mount 11 May '15

11 May '15

There's a general issue with using delta-sync MMR that needs some resolution (It's causing some considerable pain for customers): It is possible to put the entire system into endless fallbacks until a new and/or reloaded node gets a write operation. The general problem is that when an existing master queries the new/reloaded master's accesslog DB, zero entries are returned. This then triggers the fallback. This happens up util such a time as the new/reloaded master gets a direct write op. I've worked around it in general by immediately doing a no-op on the primary db (ldapmodify/replace an attribute value with its own value), but it would be nice to be able to bring new MMR nodes online or be able to reload MMR nodes for if they get out of sync, etc, without causing this sync fallback issue. There is no clear solution at the moment on what to do about zero results from the accesslog in the delta-sync MMR scenario. Proposals welcome. ;) --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 1

RE24 testing call #4 (2.4.41), LMDB RE0.9 testing call #4 (0.9.15)
by Quanah Gibson-Mount 01 May '15

01 May '15

OpenLDAP 2.4.41 Engineering Fixed ldapsearch to explicitly flush its buffer (ITS#8118) Fixed libldap async connections (ITS#8090) Fixed libldap double free of request during abandon (ITS#7967) Fixed libldap error string for LDAP_X_CONNECTING (ITS#8093) Fixed libldap segfault in ldap_sync_initialize (ITS#8001) Fixed libldap ldif-wrap off by one error (ITS#8003) Fixed libldap handling of TLS in async mode (ITS#8022) Fixed libldap null pointer dereference (ITS#8028) Fixed libldap mutex handling with LDAP_OPT_SESSION_REFCNT (ITS#8050) Fixed slapd slapadd onetime leak with -w (ITS#8014) Fixed slapd sasl auxprop crash with invalid config (ITS#8092) Fixed slapd syncrepl delta-mmr issue with overlays and slapd.conf (ITS#7976) Fixed slapd syncrepl mutex for cookie state (ITS#7968) Fixed slapd syncrepl memory leaks (ITS#8035) Fixed slapd syncrepl to free presentlist at end of refresh mode (ITS#8038) Fixed slapd syncrepl to streamline presentlist (ITS#8042) Fixed slapd syncrepl concurrency when CHECK_CSN is enabled (ITS#8120) Fixed slapd rootdn checks for hidden backends (ITS#8108) Fixed slapd segfault when using matched values control (ITS#8046) Fixed slapd-mdb minor case typo (ITS#8049) Fixed slapd-mdb one-level search (ITS#7975) Fixed slapd-mdb heap corruption (ITS#7965) Fixed slapd-mdb crash after deleting in-use schema (ITS#7995) Fixed slapd-mdb minor code cleanup (ITS#8011) Fixed slapd-mdb to return errors when using incorrect env flags (ITS#8016) Fixed slapd-mdb to correctly update search candidates (ITS#8036, ITS#7904) Fixed slapd-mdb when there were more than 65535 aliases in scope (ITS#8103) Fixed slapd-meta TLS initialization with ldaps URIs (ITS#8022) Fixed slapd-perl conversion to cn=config (ITS#8105) Fixed slapo-collect segfault (ITS#7797) Fixed slapo-constraint with 0 count constraint (ITS#7780,ITS#7781) Fixed slapo-deref with empty attribute list (ITS#8027) Fixed slapo-memberof to correctly reject invalid members (ITS#8107) Fixed slapo-sock result parser for CONTINUE (ITS#8048) Fixed slapo-syncprov synprov_matchops usage of test_filter (ITS#8013) Fixed slapo-syncprov segfault on disconnect/abandon (ITS#5452,ITS#8012) Fixed slapo-syncprov memory leak (ITS#8039) Fixed slapo-syncprov segfault on disconnect/abandon (ITS#8043) Fixed slapo-syncprov deadlock when autogroup is in use (ITS#8063) Fixed slapo-syncprov potential loss of changes when under load (ITS#8081) Fixed slapo-unique enforcement of uniqueness with manageDSAit control (ITS#8057) Build Environment Fixed libdb detection with gcc 5.x (ITS#8056) Enhanced contrib modules build paths (ITS#7782) Fixed contrib/autogroup internal operation identity (ITS#8006) Fixed contrib/autogroup to skip internal ops with accesslog (ITS#8065) Fixed contrib/passwd/sha2 compiler warning (ITS#8000) Fixed contrib/noopsrch compiler warning (ITS#7998) Fixed contrib/dupent compiler warnings (ITS#7997) Test suite: Added vrFilter test (ITS#8046) Contrib Added pbkdf2 sha256 and sha512 schemes (ITS#7977) Fixed autogroup modification callback responses (ITS#6970) Fixed nssov compare with usergroup (ITS#8079) Fixed nssov password change behavior (ITS#8080) Fixed nssov updated to 0.9.4 (ITS#8097) Documentation Added ldap_get_option(3) LDAP_FEATURE_INFO_VERSION information (ITS#8032) Added ldap_get_option(3) LDAP_OPT_API_INFO_VERSION information (ITS#8032) Fixed slapd-config(5), slapd.conf(5) tls_cipher_suite option (ITS#8099) Fixed slapd-meta(5), slapd-ldap(5) tls_cipher_suite option (ITS#8099) LMDB 0.9.15 Release Engineering Fix txn init (ITS#7961,#7987) Fix MDB_PREV_DUP (ITS#7955,#7671) Fix compact of empty env (ITS#7956) Fix mdb_rebalance collapsing root (ITS#8062) Fix mdb_load with large values (ITS#8066) Fix to retry writes on EINTR (ITS#8106) Fix mdb_cursor_del on empty DB (ITS#8109) Fix MDB_INTEGERDUP key compare (ITS#8117) Added workaround for fdatasync bug in ext3fs Build Don't use -fPIC for static lib Update .gitignore (ITS#7952,#7953) Cleanup for "make test" (ITS#7841) Misc. Android/Windows cleanup Documentation Fix MDB_APPEND doc Clarify mdb_dbi_open doc Thanks! --Quanah -- Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

3 3

MDB_INTEGERDUP in back-mdb's indexer for attr-tables.
by Леонид Юрьев 30 Apr '15

30 Apr '15

Hi, Howard. I would like to refine my question. Currently I develop a mdb_chk tool for deep check LMDB-files, like a fsck tool. It is needed for us, but let this stay aside for now. On some step I plan to check the size of each key in each DBI with MDB_INTEGERKEY and/or MDB_INTEGERDUP flags. Initially I assumed - if one of these flags is set, the the length of all keys must be sizeof(int) or sizeof(long). As will be used the corresponding comparison functions (mdb_cmp_int, mdb_cmp_long) , which can handle only keys of such sizes. But this approach has failed. A dbi-tables created for attr by mdb_attr_dbs_open() have MDB_DUPSORT|MDB_DUPFIXED|MDB_INTEGERDUP flags and may contain records with a 5-byte keys. For instance as the 'entryCSN'. Later you have written that MDB_INTEGERDUP is required. Now I do not understand how it all works: - MDB_INTEGERDUP is set for attr tables. - indexer could create a 5-byte keys. - mdb-backend set custom comparison function only for MDB_DN2ID table. - under debugger I see that md_dcmp is pointed to the default. - assert for key-size in default comparators was not failed while "make test". - all test are passed without MDB_INTEGERDUP. - but you told MDB_INTEGERDUP is necessary So, - how and when MDB_INTEGERDUP will have an impact? - in which cases keys in the table definitely should be sizeof(int) or sizeof(long)? Thank in advance, Leonid.

2 4

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel May 2015