openldap-devel

openldap-devel@openldap.org

1420 discussions

Valgrind report an error in sample-mdb.c
by yi huang 05 Dec '14

05 Dec '14

I'm not sure how this happens, is it a problem? $ make $ gcc sample-mdb.c -L. -llmdb $ valgrind ./a.out ==13010== Memcheck, a memory error detector ==13010== Copyright (C) 2002-2012, and GNU GPL'd, by Julian Seward et al. ==13010== Using Valgrind-3.8.1 and LibVEX; rerun with -h for copyright info ==13010== Command: ./a.out ==13010== ==13010== Syscall param pwrite64(buf) points to uninitialised byte(s) ==13010== at 0x42319D3: pwrite (pwrite.c:46) ==13010== by 0x403AB25: mdb_page_flush (mdb.c:3160) ==13010== by 0x4043AD0: mdb_txn_commit (mdb.c:3416) ==13010== by 0x8048A13: main (in /home/yihuang/src/mdb/libraries/liblmdb/a.out) ==13010== Address 0x43c35b4 is 4,084 bytes inside a block of size 4,096 alloc'd ==13010== at 0x402BE18: malloc (vg_replace_malloc.c:270) ==13010== by 0x403AE47: mdb_page_malloc.isra.13 (mdb.c:1615) ==13010== by 0x403D281: mdb_page_alloc.isra.22 (mdb.c:2120) ==13010== by 0x403D446: mdb_page_touch (mdb.c:2252) ==13010== by 0x403EEED: mdb_cursor_touch (mdb.c:6019) ==13010== key: 0x4643fdc 020 , data: 0x4643fe0 020 3141592 foo bar ==13010== ==13010== HEAP SUMMARY: ==13010== in use at exit: 0 bytes in 0 blocks ==13010== total heap usage: 15 allocs, 15 frees, 1,586,385 bytes allocated ==13010== ==13010== All heap blocks were freed -- no leaks are possible ==13010== ==13010== For counts of detected and suppressed errors, rerun with: -v ==13010== Use --track-origins=yes to see where uninitialised values come from ==13010== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

2 1

LMDB Getting transaction ID (feature request)
by David Barbour 05 Dec '14

05 Dec '14

At the moment, LMDB provides a means to view the last committed transaction on the environment, but does not provide any means to reliably acquire the ID for the 'current' transaction. I cannot check that the `me_last_txnid` returned from mdb_envinfo() is the same snapshot I'm looking at. I can think of several scenarios where accessing the transaction ID specific to the snapshot a transaction is viewing could be very useful. First, if I want to promote a read-only transaction to a read-write transaction, a very simple mechanism is to abort the read, create the read-write transaction and compare the transaction IDs. If they're the same, I can immediately continue since there is no risk of any changes. Otherwise, I can restart the transaction with write enabled. Second, if I want to build a layer above LMDB modeling concurrent read-write transactions, I can potentially track (in several read-only transactions) intended writes together with a bloom-filter for the keys I've read. (There are other ways to do this, too, e.g. via a proper STM layer.) It is feasible to logically serialize and merge a bunch of non-conflicting write transactions as one larger write transaction. Such a layer can potentially improve throughput and amortize synchronization costs, albeit with a tradeoff for memory overheads and latency. Third, if I want to poll the database, the ability to peek at the transaction ID and see that nothing has changed could be very useful. In an HTTP server, the transaction ID could serve as an (admittedly coarse grained) e-tag. Fortunately, implementing this feature request is trivial. It uses only information we already have: mt_txnid. I'm asking for access to this value, e.g. via: txnid_t mdb_txn_id(MDB_txn* txn); Will you make this happen? Regards, Dave

2 1

Re: Antw: Passwords, Hashing, and Binds
by Quanah Gibson-Mount 25 Nov '14

25 Nov '14

--On Monday, November 24, 2014 12:22 PM +0100 Onno van der Straaten <onno.van.der.straaten(a)gmail.com> wrote: > sudo make install I'd generally advise you really read over the options to configure, and build a better set of binaries. For example, leave out back-bdb/hdb, and enable building things modularly. My options are: --with-cyrus-sasl \ --with-tls=openssl \ --enable-dynamic \ --enable-slapd \ --enable-modules \ --enable-backends=mod \ --disable-shell \ --disable-sql \ --disable-bdb \ --disable-hdb \ --disable-ndb \ --enable-overlays=mod \ --enable-debug \ --enable-spasswd \ --enable-crypt; \ > Make the sha2 module > cd ~/openldap/contrib/slapd-modules/passwd/sha2 > sed -i.bak s/-Wall -g/-Wall -g fPIC/g Makefile > make I do: (cd openldap-$(LDAP_VERSION)/contrib/slapd-modules/passwd/sha2; \ $(MAKE) prefix=/usr/local LIBS="-L$(LDAP_LIB_DIR) -lldap_r -llber" install STRIP=""; \ ) And then it installs it for me in the same location. Just make sure you use the same prefix here. > This results in a number of files pw-sha2.la sha2.lo sha2.o slapd-sha2.lo slapd-sha2.o > > The question now is how to install this on my target OpenLDAP server. I > put the files in /usr/lib64/openldap en dan tried to add the following > dn: cn=module{0},cn=config > changetype: modify > replace: olcModuleLoad > olcModuleLoad: slapd-sha2.la I'm not sure that replacing olcModuleLoad is correct. If you already have values in there, you probably want to keep them. I generally *add* an additional values. In any case, your value for the attribute is incorrect. The .la file is named, as in your email, pw-sha2.la, not slapd-sha2.la . If you want to add it as an additional module to load, then you would do changetype: modify add: olcModuleLoad olcModuleLoad: pw-sha2.la My loaded modules are: dn: cn=module{0} objectClass: olcModuleList cn: module{0} olcModulePath: /opt/zimbra/openldap/sbin/openldap olcModuleLoad: {0}back_mdb.la olcModuleLoad: {1}back_monitor.la olcModuleLoad: {2}syncprov.la olcModuleLoad: {3}accesslog.la olcModuleLoad: {4}dynlist.la olcModuleLoad: {5}unique.la olcModuleLoad: {6}noopsrch.la olcModuleLoad: {7}pw-sha2.la for example. now, if you want to make something like say, SHA512 the default, then you need to modify the frontend config db: dn: olcDatabase={-1},cn=config changetype: modify replace: olcPasswordHash olcPasswordHash: {SSHA512} --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: [LMDB] Lockups with robust mutexes and crashing processes
by Marcos-David Dione 25 Nov '14

25 Nov '14

Marcos-David Dione/NCE/AMADEUS wrote on 24/11/2014 11:10:42: > Seen like that I'm not sure if there's a defined behaviour > for that. I'll ask in the glibc and/or kernel MLs and I'll come > back with the answer. and here's the answer: > On 11/24/2014 03:34 PM, Marcos Dione wrote: > > We found a situation where a robust mutex cannot be recovered > > from a stale lock and we're wondering if it's simply an undefined > > situation or a bug in the kernel. Attached you will find the sample > > code, which is loosely based on a glibc's test case.The gist of it is as > > follows: > > > > 1. we open a file. > > 2. we mmap it and use that mem to store a robust mutex. > > 3. we lock the mutex. > > 4. we munmap the file. > > 5. we close the file. > > Undefined behaviour. > > This results in undefined behaviour since the allocated storage for > the mutex object has been lost. You need to keep that storage around > for the robust algorithms to work with. Without any data you can't > do anything. Full answer: https://sourceware.org/ml/libc-help/2014-11/msg00035.html -- Marcos Dione Astek Sud-Est R&D-SSP-DTA-TAE-TDS for Amadeus SAS T: +33 (4)4 9704 1727 marcos-david.dione(a)amadeus.com

2 3

[LMDB] Lockups with robust mutexes and crashing processes
by Marcos-David Dione 24 Nov '14

24 Nov '14

I already posted this to the IRC channel, but there was no response, so I repost this here. I'm trying out lmdb from master, including the robust mutex code. We're experiencing lock ups after the process holding the lock dies, as if the robust lock was not recovered. I tried to come up with an lmdb example that shows it and I got it, just a few lines. It uses fork() just to automate it; see that the environment is opened in both children. Here's the code: http://pastebin.com/Cbbri6az If I run this, I see that one of the children waits for the write lock and is not awakened when the other child dies without closing the txn (but notice I close the env). This is on purpose, to simulate a crashing process.The worst part is that I can't reproduce it using directly libpthread and mmap. Here is the code I came up with: http://pastebin.com/ybR5L4cP It's a little bit more verbose because I based it on a glibc test case. Are we missing anything? It seems to us that the code follows does not break any of LMDB's caveats (specially the one about creating the envs before fork()'ing. Is it wrong to assume that the waiting process should recover the lock from staleness? -- Marcos Dione Astek Sud-Est R&D-SSP-DTA-TAE-TDS for Amadeus SAS T: +33 (4)4 9704 1727 marcos-david.dione(a)amadeus.com

2 2

Accessing entry attributes from pwdcheckmodule
by Daly Chikhaoui 23 Nov '14

23 Nov '14

Hello I need to access, from the pwdcheckmodule script, to the current entry (the one we are checking it's password complexity) attributes. Is that possible ? Regards, Daly

2 1

Don't Use Copy Control (1.3.6.1.4.1.4203.666.5.15 vs. 1.3.6.1.1.22)
by Michael Ströder 30 Oct '14

30 Oct '14

HI! Is there any issue with using OID 1.3.6.1.1.22 as controlType for Don't Use Copy Control as defined in RFC 6171? Currently an experimental OpenLDAP .666 OID s used which makes this control rather unusable outside OpenLDAP implementation. Ciao, Michael.

3 8

test
by Quanah Gibson-Mount 30 Oct '14

30 Oct '14

testing... Earlier email never showed up. --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

slapd as an appliance
by Howard Chu 29 Oct '14

29 Oct '14

A couple different random thoughts all collided so I thought I'd jot them down here... We can have back-mdb/LMDB set the PROT_EXEC permission on the mmap, and then store executable code in LDAP entries and execute them directly. The idea came to me while reading about just-in-time C compilers and remembering my previous plans for liblforth. At first I was thinking only of small plugins (e.g. syntax parsers/normalizers) but really there's no reason that you couldn't handle larger executables the same way. I.e., we could store all of slapd's dynamically loadable modules in a back-mdb backend, change olcModuleLoad to use a DN, and dynamically insert and delete code on the fly, without needing any filesystem interactions. Once you have a system where all administrative operations can be done through LDAP, without requiring any other admin interface, it becomes pretty trivial to drop it into a sealed black box and deploy an "appliance." This would be a pretty key advantage for a small, self-contained embedded processor. Of course, it doesn't stop there - if the only system interface needed is the LDAP port, we don't even need a full OS distro any more. We could run slapd as process #1 (who needs systemd?) - a hardened system with zero unexpected attack surfaces, only one network port open to the world. One of the current in-progress items for LMDB is to enable using it on raw disk partitions. Adding this feature would mean you no longer have to worry about what type of filesystem to choose, or how to format it before setting up a database. Again, it would be ideal for an appliance because you'd just have to plugin a storage device and then create a back-mdb backend pointing to it. Another item for LMDB, once raw partition support is working, is in-kernel LMDB. Initially my motivation was to reproduce the KBDBFS work - an in-kernel filesystem built on top of BerkeleyDB (see here http://www.fsl.cs.sunysb.edu/docs/kbdbfs-msthesis/ ) but there's no need to stop there. We could port all of slapd into the kernel, and have a turnkey appliance with no user processes at all. Food for thought... -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 1

write-scaling problems in LMDB
by Luke Kenneth Casson Leighton 20 Oct '14

20 Oct '14

http://symas.com/mdb/inmem/scaling.html i just looked at this and i have a sneaking suspicion that you may be running into the same problem that i encountered when accidentally opening 10 LMDBs 20 times by forking 30 processes *after* opening the 10 LMDBs... (and also forgetting to close them in the parent). what i found was that when i reduced the number of LMDBs to 3 or below, the loadavg on the multi-core system was absolutely fine [ok, it was around 3 to 4 but that's ok] adding one more LMDB (remember that's 31 extra file handles opened to a shm-mmap'd file) increased the loadavg to 6 or 7. by the time that was up to 10 the loadavg had completely unreasonably jumped to over 30. i could log in over ssh - that was not a problem. editing a file was ok (opening it) but trying to create new files resulted in applications (such as vim) was blocked so badly that i often could not even press ctrl-z in order to background the task, and had to kill the entire ssh session. in each test run the amount of work being done was actually relatively small. basically i suspect a severe bug in the linux kernel which these extreme circumstances (32 or 16 processes accessing the same mmap'd file for example) have never been encountered, so the bug is simply... unknown. can i make the suggestion that, whilst i am aware that it is generally not recommended for production environments to run more processes than there are cores, you try running 128, 256 and even 512 processes all hitting that 64-core system, and monitor its I/O usage (iostats) and loadavg whilst doing so? the hypothesis to test is that the performance, which should scale reasonably linearly downwards as a ratio of the number of processes to the number of cores, instead drops like a lead balloon. l.

2 13

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel