openldap-devel

openldap-devel@openldap.org

1418 discussions

Re: Antw: Passwords, Hashing, and Binds
by Quanah Gibson-Mount 25 Nov '14

25 Nov '14

--On Monday, November 24, 2014 12:22 PM +0100 Onno van der Straaten <onno.van.der.straaten(a)gmail.com> wrote: > sudo make install I'd generally advise you really read over the options to configure, and build a better set of binaries. For example, leave out back-bdb/hdb, and enable building things modularly. My options are: --with-cyrus-sasl \ --with-tls=openssl \ --enable-dynamic \ --enable-slapd \ --enable-modules \ --enable-backends=mod \ --disable-shell \ --disable-sql \ --disable-bdb \ --disable-hdb \ --disable-ndb \ --enable-overlays=mod \ --enable-debug \ --enable-spasswd \ --enable-crypt; \ > Make the sha2 module > cd ~/openldap/contrib/slapd-modules/passwd/sha2 > sed -i.bak s/-Wall -g/-Wall -g fPIC/g Makefile > make I do: (cd openldap-$(LDAP_VERSION)/contrib/slapd-modules/passwd/sha2; \ $(MAKE) prefix=/usr/local LIBS="-L$(LDAP_LIB_DIR) -lldap_r -llber" install STRIP=""; \ ) And then it installs it for me in the same location. Just make sure you use the same prefix here. > This results in a number of files pw-sha2.la sha2.lo sha2.o slapd-sha2.lo slapd-sha2.o > > The question now is how to install this on my target OpenLDAP server. I > put the files in /usr/lib64/openldap en dan tried to add the following > dn: cn=module{0},cn=config > changetype: modify > replace: olcModuleLoad > olcModuleLoad: slapd-sha2.la I'm not sure that replacing olcModuleLoad is correct. If you already have values in there, you probably want to keep them. I generally *add* an additional values. In any case, your value for the attribute is incorrect. The .la file is named, as in your email, pw-sha2.la, not slapd-sha2.la . If you want to add it as an additional module to load, then you would do changetype: modify add: olcModuleLoad olcModuleLoad: pw-sha2.la My loaded modules are: dn: cn=module{0} objectClass: olcModuleList cn: module{0} olcModulePath: /opt/zimbra/openldap/sbin/openldap olcModuleLoad: {0}back_mdb.la olcModuleLoad: {1}back_monitor.la olcModuleLoad: {2}syncprov.la olcModuleLoad: {3}accesslog.la olcModuleLoad: {4}dynlist.la olcModuleLoad: {5}unique.la olcModuleLoad: {6}noopsrch.la olcModuleLoad: {7}pw-sha2.la for example. now, if you want to make something like say, SHA512 the default, then you need to modify the frontend config db: dn: olcDatabase={-1},cn=config changetype: modify replace: olcPasswordHash olcPasswordHash: {SSHA512} --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

Re: [LMDB] Lockups with robust mutexes and crashing processes
by Marcos-David Dione 25 Nov '14

25 Nov '14

Marcos-David Dione/NCE/AMADEUS wrote on 24/11/2014 11:10:42: > Seen like that I'm not sure if there's a defined behaviour > for that. I'll ask in the glibc and/or kernel MLs and I'll come > back with the answer. and here's the answer: > On 11/24/2014 03:34 PM, Marcos Dione wrote: > > We found a situation where a robust mutex cannot be recovered > > from a stale lock and we're wondering if it's simply an undefined > > situation or a bug in the kernel. Attached you will find the sample > > code, which is loosely based on a glibc's test case.The gist of it is as > > follows: > > > > 1. we open a file. > > 2. we mmap it and use that mem to store a robust mutex. > > 3. we lock the mutex. > > 4. we munmap the file. > > 5. we close the file. > > Undefined behaviour. > > This results in undefined behaviour since the allocated storage for > the mutex object has been lost. You need to keep that storage around > for the robust algorithms to work with. Without any data you can't > do anything. Full answer: https://sourceware.org/ml/libc-help/2014-11/msg00035.html -- Marcos Dione Astek Sud-Est R&D-SSP-DTA-TAE-TDS for Amadeus SAS T: +33 (4)4 9704 1727 marcos-david.dione(a)amadeus.com

2 3

[LMDB] Lockups with robust mutexes and crashing processes
by Marcos-David Dione 24 Nov '14

24 Nov '14

I already posted this to the IRC channel, but there was no response, so I repost this here. I'm trying out lmdb from master, including the robust mutex code. We're experiencing lock ups after the process holding the lock dies, as if the robust lock was not recovered. I tried to come up with an lmdb example that shows it and I got it, just a few lines. It uses fork() just to automate it; see that the environment is opened in both children. Here's the code: http://pastebin.com/Cbbri6az If I run this, I see that one of the children waits for the write lock and is not awakened when the other child dies without closing the txn (but notice I close the env). This is on purpose, to simulate a crashing process.The worst part is that I can't reproduce it using directly libpthread and mmap. Here is the code I came up with: http://pastebin.com/ybR5L4cP It's a little bit more verbose because I based it on a glibc test case. Are we missing anything? It seems to us that the code follows does not break any of LMDB's caveats (specially the one about creating the envs before fork()'ing. Is it wrong to assume that the waiting process should recover the lock from staleness? -- Marcos Dione Astek Sud-Est R&D-SSP-DTA-TAE-TDS for Amadeus SAS T: +33 (4)4 9704 1727 marcos-david.dione(a)amadeus.com

2 2

Accessing entry attributes from pwdcheckmodule
by Daly Chikhaoui 23 Nov '14

23 Nov '14

Hello I need to access, from the pwdcheckmodule script, to the current entry (the one we are checking it's password complexity) attributes. Is that possible ? Regards, Daly

2 1

Don't Use Copy Control (1.3.6.1.4.1.4203.666.5.15 vs. 1.3.6.1.1.22)
by Michael Ströder 31 Oct '14

31 Oct '14

HI! Is there any issue with using OID 1.3.6.1.1.22 as controlType for Don't Use Copy Control as defined in RFC 6171? Currently an experimental OpenLDAP .666 OID s used which makes this control rather unusable outside OpenLDAP implementation. Ciao, Michael.

3 8

test
by Quanah Gibson-Mount 30 Oct '14

30 Oct '14

testing... Earlier email never showed up. --Quanah -- Quanah Gibson-Mount Server Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

1 0

slapd as an appliance
by Howard Chu 29 Oct '14

29 Oct '14

A couple different random thoughts all collided so I thought I'd jot them down here... We can have back-mdb/LMDB set the PROT_EXEC permission on the mmap, and then store executable code in LDAP entries and execute them directly. The idea came to me while reading about just-in-time C compilers and remembering my previous plans for liblforth. At first I was thinking only of small plugins (e.g. syntax parsers/normalizers) but really there's no reason that you couldn't handle larger executables the same way. I.e., we could store all of slapd's dynamically loadable modules in a back-mdb backend, change olcModuleLoad to use a DN, and dynamically insert and delete code on the fly, without needing any filesystem interactions. Once you have a system where all administrative operations can be done through LDAP, without requiring any other admin interface, it becomes pretty trivial to drop it into a sealed black box and deploy an "appliance." This would be a pretty key advantage for a small, self-contained embedded processor. Of course, it doesn't stop there - if the only system interface needed is the LDAP port, we don't even need a full OS distro any more. We could run slapd as process #1 (who needs systemd?) - a hardened system with zero unexpected attack surfaces, only one network port open to the world. One of the current in-progress items for LMDB is to enable using it on raw disk partitions. Adding this feature would mean you no longer have to worry about what type of filesystem to choose, or how to format it before setting up a database. Again, it would be ideal for an appliance because you'd just have to plugin a storage device and then create a back-mdb backend pointing to it. Another item for LMDB, once raw partition support is working, is in-kernel LMDB. Initially my motivation was to reproduce the KBDBFS work - an in-kernel filesystem built on top of BerkeleyDB (see here http://www.fsl.cs.sunysb.edu/docs/kbdbfs-msthesis/ ) but there's no need to stop there. We could port all of slapd into the kernel, and have a turnkey appliance with no user processes at all. Food for thought... -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 1

write-scaling problems in LMDB
by Luke Kenneth Casson Leighton 20 Oct '14

20 Oct '14

http://symas.com/mdb/inmem/scaling.html i just looked at this and i have a sneaking suspicion that you may be running into the same problem that i encountered when accidentally opening 10 LMDBs 20 times by forking 30 processes *after* opening the 10 LMDBs... (and also forgetting to close them in the parent). what i found was that when i reduced the number of LMDBs to 3 or below, the loadavg on the multi-core system was absolutely fine [ok, it was around 3 to 4 but that's ok] adding one more LMDB (remember that's 31 extra file handles opened to a shm-mmap'd file) increased the loadavg to 6 or 7. by the time that was up to 10 the loadavg had completely unreasonably jumped to over 30. i could log in over ssh - that was not a problem. editing a file was ok (opening it) but trying to create new files resulted in applications (such as vim) was blocked so badly that i often could not even press ctrl-z in order to background the task, and had to kill the entire ssh session. in each test run the amount of work being done was actually relatively small. basically i suspect a severe bug in the linux kernel which these extreme circumstances (32 or 16 processes accessing the same mmap'd file for example) have never been encountered, so the bug is simply... unknown. can i make the suggestion that, whilst i am aware that it is generally not recommended for production environments to run more processes than there are cores, you try running 128, 256 and even 512 processes all hitting that 64-core system, and monitor its I/O usage (iostats) and loadavg whilst doing so? the hypothesis to test is that the performance, which should scale reasonably linearly downwards as a ratio of the number of processes to the number of cores, instead drops like a lead balloon. l.

2 13

LMDB app-level crash consistency
by Howard Chu 18 Oct '14

18 Oct '14

For posterity, this is the conversation we had with the University of Wisconsin researchers behind this Usenix OSDI paper: https://www.usenix.org/conference/osdi14/technical-sessions/presentation/pi… > On Thu, Jul 24, 2014 at 8:10 AM, Howard Chu <hyc(a)symas.com > <mailto:hyc@symas.com>> wrote: > > Hello Thanu, > I just saw your slides from > http://wisdom.cs.wisc.edu/__index.htm#news-2 > <http://wisdom.cs.wisc.edu/index.htm#news-2> and was wondering if you > could give more details on the crash vulnerabilities you found in LMDB. > Thanks. 25/07/14 09:42 Thanumalayan Sankaranarayana Pillai wrote: > > Hi Howard, > > > > Thanks for your interest in the slides. We appreciate you caring about the > > vulnerability. > > The single crash vulnerability that we discovered in LMDB concerns the > 106-byte update to the header while inserting a key-value pair (within the > mdb_txn_commit function). Specifically, if this overwrite is not done > atomically (i.e., if only a part of the 106 bytes goes to disk, and then a > crash happens), then LMDB reports an error if we try to further modify the > database after reboot. The error reported is "mdb_page_touch: Assertion > `mp->mp_p.p_pgno != pgno' failed". > > We also did some manual analysis by making the loss in atomicity correspond to > field boundaries within the information contained in those 106-bytes (such as > free list data and transaction ID), and found that LMDB loses the previous > transaction sometimes. For example, if the database contains Txn-1, Txn-2, and > Txn-3, and a crash happens while inserting Txn-4, the rebooted-database might > lose Txn-3. I have not included any such manual analysis results in the slides. > > I have attached a text file with a more detailed description of the > vulnerability. If you have any questions on the details, I have CC'ed Ram, who > actually tested LMDB for the vulnerabilities and might be able to help you > better than me. > > Thus, the vulnerability in LMDB relates to the atomicity of a file overwrite > that is within a 512-byte sector. With current (ext3/ext4/btrfs/xfs/zfs) file > systems running atop hard drives and SSDs, this is probably not a problem. > (The only place we have heard about that indicates atomicity might not be > maintained here is [1]). > > However, byte-accessible NVRAMs in the future will probably expose this > vulnerability. We are also not sure what happens with the multi-level storage > stacks many people use these days (Virtual machines atop Fusion-IO-card > connected via networks to a NAS?). In our understanding, even for these > situations, LMDB can solve this vulnerability easily, since it has two root > pointers. > > All said, since we are not particularly experts at using LMDB, there is a > slight chance we made a mistake in how we used LMDB while testing it. Please > let us know if you suspect this to be the case. Hope this helps! > > [1] http://lkml.iu.edu//hypermail/linux/kernel/0908.3/00155.html > > -- > Thanks, > Thanu 25/07/14 11:38 Howard: > Thanks much for your reply. The results you describe are not too surprising; > we assume atomic sector writes. This is an area we've spent quite a lot of > time researching ourselves. I summarized our findings here last year > http://www.openldap.org/its/index.cgi/Incoming?id=7668 > > Currently all hard drive/SSD manufacturers guarantee that 512 byte sector > writes are atomic. As such, failure to write the 106 byte header is not > something we account for in current LMDB releases. Also, failures of this type > should result in ECC errors in the disk sector - it should be impossible to > successfully read a sector that was written incorrectly in the ways you describe. > > Even in extreme cases, the probability of failure to write the leading 128 out > of 512 bytes of a sector is nearly nil - even on very old hard drives, before > 512-byte sector write guarantees. We would have to go back nearly 30 years to > find such a device, e.g. > > http://bitsavers.informatik.uni-stuttgart.de/pdf/quantum/Quantum_ProDrive_4… > > Page 23, Section 2.1 > "No damage or loss of data will occur if power is applied or removed during > drive operation, except that data may be lost in the sector being written at > the time of power loss." > > From the specs on page 15, the data transfer rate to/from the platters is > 1.25MB/sec, so the time to write one full sector is 0.4096ms; the time to > write the leading 128 bytes of the sector is thus 1/4 of that: 0.10ms. You > would have to be very very unlucky to have a power failure hit the drive > within this .1ms window of time. Fast-forward to present day and it's simply > not an issue. > > But as you say, byte-addressable storage is coming soon. The LMDB 1.x versions > will add a checksum to the two headers, and will fallback to using the > previous header if a checksum error is found. > > Thanks again for your work, looking forward to reading your full paper in the > future. 25/07/14 13:14 Thanu: > Hi Howard, > > Great to know that so much thought was already put into this issue. (We usually see people assume things about the storage stack that are simply false, and without any notion of when they will become false if they aren't already.) I am waiting for all the cool new things that will be on LMDB 1.0. > > Thanks, > Thanu 25/07/14 13:15 Howard: > Hi again, > looking over these vulnerabilities, they all seem unrealistic even given > the existence of byte-addressable storage devices. In particular, even if you > assume writes may be non-atomic, why do you assume that writes will be > non-sequential within a sector? > > For example, if we were talking about an HDD that did not guarantee atomic > sector writes, we still know that bytes are written sequentially from > beginning of sector to end of sector; this is one reason why the LMDB meta > page is laid out in the way that it is - the transaction ID is last because if > a partial write occurs and the txn ID doesn't make it to disk, this meta page > will be ignored and the previous one will be used automatically. > > In the tests you describe here, you have broken this sequential assumption, > but there are no storage devices in existence that can fail in the manners you > describe. And even in byte-addressable memories, DMA or memcpy routines will > always start at a low-numbered address and work sequentially higher. Do you > really believe it is important for software to protect against this type of > failure scenario? >> > Found using automatic analysis >> > ------------------------------ >> > >> > Write (106 bytes) at lib/mdb.c:3227[mdb_txn_commit]. Split into three, first part (first 36 bytes) alone does not go to disk. >> > >> > Error: lib/mdb.c:1921: mdb page touch: Assertion ‘mp->mp p.p pgno != pgno’ failed. >> > >> > Exact situations where the vulnerabilities arise, found using manual analysis >> > ----------------------------------------------------------------------------- >> > >> > 1. Only txn id gets into metapage and no other data: >> > >> > Previous transaction is lost. >> > >> > 2. Free list data, txn id and last page goes in but not main db details: >> > >> > Previous transaction is lost. Error: lib/mdb.c:1921: mdb page touch: Assertion ‘mp->mp p.p pgno != pgno’ failed. >> > >> > 3. only main db - 2 fields no free pointer: >> > >> > Error: lib/mdb.c:1921: mdb page touch: Assertion ‘mp->mp p.p pgno != pgno’ failed. >> > >> > 4. Everything gets in except database pointer - it is 48 bytes - but a crash happens such that second 24 bytes (which contains the root) is missed: >> > >> > Previous transaction is lost. Error: lib/mdb.c:1921: mdb page touch: Assertion ‘mp->mp p.p pgno != pgno’ failed. 25/07/14 14:03 Thanu: > Hi Howard, > > To be honest, we are not yet sure whether it is important for software to protect sequential-assumption-breaking under practical circumstances. While byte-addressable devices might only persist things sequentially, a byte-addressable file system itself might re-order bytes (what if the memcpy doesn't start from the lower address?); at this point however, I am probably being cynical and hypothesizing too much about the future. > > However there is currently no standard/guarantee that (file-system-level) writes should be sequential or atomic as far as we know. One of our goals in doing this project is to show the entire community how non-standardized the interaction between the application and the file system is. It is fully possible that, by the time NVMs become main-stream, standards arise and sector-size non-sequential (or even non-atomic) writes are disallowed. (It might also be possible that someone invents an NVM with high parallel bandwidth, and memcpy() explicitly parallelizes updates to contiguous bytes for higher throughput.) > > Personally, with software such as LMDB, I feel that we should simply add a CRC, just in case future storage stacks behave in "reasonably weird" ways. I do understand that it might be better to not do this for either readability or performance and simply keep an eye out for any new storage stacks. > > If it is not too much trouble, can you point me to any discussions about byte-addressable storage being sequential? I have heard this mentioned once before, but I cannot find any solid references to it. Any references to sequentiality within HDDs will be useful as well. > > Thanks, > Thanu On Fri, Jul 25, 2014 at 8:50 AM, Howard Chu <hyc(a)symas.com wrote: >>> Regarding sequentiality within HDDs, that's a natural consequence of the >>> physical construction/operation of HDDs. The platters rotate under the R/W >>> head, the R/W head spits out a stream of bits in sequential order. The >>> platter always rotates in one direction, at essentially constant speed, so >>> the R/W head cannot skip back and forth within 1 sector and write out of >>> order. It's physically impossible for HDDs to write non-sequentially >>> within a single sector. >>> >>> For your reference re: SSDs, here's an old Samsung NAND Flash data sheet. >>> http://pdf1.alldatasheet.com/__datasheet-pdf/view/84340/__SAMSUNG/K9F1G08U0… >>> <http://pdf1.alldatasheet.com/datasheet-pdf/view/84340/SAMSUNG/K9F1G08U0A.ht…> >>> >>> The relevant info is page 31 "Page Program" >>> "The device is programmed basically on a page basis, but it does allow >>> multiple partial page programing of a word or consecutive >>> bytes up to 2112, in a single page program cycle. The number of >>> consecutive partial page programming operation within the same >>> page without an intervening erase operation must not exceed 4 times for >>> main array(1time/512byte) and 4 times for spare >>> array(1time/16byte). The addressing should be done in sequential order in >>> a block." >>> >>> For byte-addressable storage I suppose no hard rules apply. If we're >>> talking about NVDIMMs, they are treated basically the same as DRAM. Still, >>> in Linux today you're most likely to see a block-device style layer >>> imposed above them. DMA controllers tend to be programmed for >>> auto-incrementing addresses. memcpy will tend to do the same, since CPU >>> prefetchers are also designed with at least cache-line sequentiality. >>> I.e., it may not be important to write to the destination in sequential >>> order, but there is a performance benefit to reading from the source in >>> sequential order, and there is no benefit to holding the read and >>> reordering it before pushing to the destination. >>> 25/07/14 15:46 Thanu: > Hi Howard, > > Thanks for the explanation and the pointers. They are really helpful. > > While you certainly have more working knowledge of byte-addressable > storage than me, I still believe there is an (extremely) small chance of non-sequential updates on Linux. Let us say that the cache block is 64 bytes in size, and the block-device layer modifies the mapped memory addresses corresponding to all user writes. For argument's sake, let us also say that writes are written back to the actual storage if there is a change in the (coherence) ownership of a cache block. If we consider two adjacent 64-byte blocks, and the second block is written twice by different cores, it is possible that the second block reaches the storage first. > > Of course, even if such non-sequential updates become possible in the future, it would require changes to the file-system journaling machinery itself, and is bound to be well-publicized. I am only justifying why we considered non-sequential updates in our study (to measure the impact of such scenarios on applications). We are hoping that if someone in the future (ever) debates about introducing non-atomic updates, they fully understand the application-level consequences by looking at our study. > > Thanks, > Thanu 25/07/14 16:46 Howard: > Fair enough. Certainly, if the question ever arose, I would be glad we have > your study to point to as a reference. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

question about use of buffers in lmdb
by Luke Kenneth Casson Leighton 17 Oct '14

17 Oct '14

howard, et al, hi, i notice that LMDB stores data in an "append only" manner (from the Q&A). i was thinking of recommending that in python the buffers be handed around with *direct* and persistent pointers to the data within the LMDB shm/memory-map. the reason that i believe this will be successful is because the circumstances are rather unusual: once data is entered (added) it is *never* overwritten, updated, changed, or modified. there simply is no need [in this application]. so there will only ever be insertion (once and only once) or controlled and monitored deletion (by age of record). under these circumstances is it perfectly reasonable to pass around the pointers to the buffers returned from value look-ups *indefinitely*? if so that would be awesome. l.

2 2

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel