openldap-technical

openldap-technical@openldap.org

14 participants
9796 discussions

LMDB space amplification
by Shu, Xinxin 27 Apr '15

27 Apr '15

Hi list, Recently I have integrated lmdb into ceph , got intial results , I dump all records of lmdb, there are several different size key-value pairs, 1. key size : 45 byte, value size = 124 byte 2. key size : 85 byte, value size = 1187 byte 3. key size : 57 byte, value size = 135 byte 4. key size : 64 byte, value size = 182 byte 5. key size : 33 byte, value size = 4096 byte (used for real data, about 80 %) this db contains about ~ 10000000 records, about 80 percent of records are the last key-value pairs (key size 33 byte, value size 4096 byte), I calculate size of all key-value pairs, about 33GB , but on-disk size is about 63GB, so space amplification is ~ 2, is this amplification reasonable ? In which situation can lmdb get such large space amplification? How can I reduce this amplification? If you need any other details , please email me , any tips will be appreciate. Cheers, xinxin

2 3

[WIP] WiredTiger Backend
by HAMANO Tsukasa 27 Apr '15

27 Apr '15

Hi, I'm writing OpenLDAP WiredTiger backend. It will get high write throughput and high concurrency performance. Now this backend module is incomplete but basic feature is working, So I publish the code. https://www.osstech.co.jp/download/hamano/openldap/0001-OpenLDAP-WiredTiger… Some note: * slapadd, slapcat, slapindex will works. * LDAP BIND, ADD, DELETE, SEARCH will works. * MODIFY, MODRDN does not implement yet. * deref search does not implement yet. * WiredTiger is not support multiprocess access yet. It mean that we can't to do slatcat while running slapd. It will be supported in the future. * This backend module may have a issue about licensing when distributing whole binary under OpenLDAP Public Licence. Now WiredTiger, Inc are working cooperatively toward a resolve the issue. I'm glad if you get to try it and give me advice. Thank you. -- Open Source Solution Technology Corporation HAMANO Tsukasa <hamano(a)osstech.co.jp> fingerprint = 2285 2111 6D34 3816 3C2E A5B9 16BE D101 6069 BE55

2 1

modifying cn=config with ldapmodify
by Robert Munn 26 Apr '15

26 Apr '15

I have been trying to replace the SSL cert settings on my OpenLDAP instance running on Ubuntu using ldapmodify. I followed directions on the Ubuntu wiki: https://help.ubuntu.com/lts/serverguide/openldap-server.html#openldap-tls using a modified ldif file for the replace: dn: cn=config changetype: modify replace: olcTLSCACertificateFile olcTLSCACertificateFile: /etc/ssl/certs/cacert.pem - replace: olcTLSCertificateFile olcTLSCertificateFile: /etc/ssl/certs/ldap01_slapd_cert.pem - replace: olcTLSCertificateKeyFile olcTLSCertificateKeyFile: /etc/ssl/private/ldap01_slapd_key.pem When it didn’t work on my existing instance I built a new instance in a new Ubuntu VM (14.04) and tried the original directions from Ubuntu. That did not work either. The ldapmodify command executes correctly but it seems that the change is not registered by the server. This is the case in both the new instance and the old instance of OpenLDAP. I ended up replacing the values (or adding them in the new instance) in the /etc/ldap/slapd.d/cn=config.ldif file manually. Making the changes manually and restarting slapd works, but my understanding was that changes to cn=config should be made through ldapmodify. I also found a tech note at CentOS: https://www.centos.org/docs/5/html/CDS/cli/8.0/Configuration_Command_File_R… in section 2.2.2.2 that indicates changes to cn=config will be ignored: "If an attribute is added to cn=config, the server ignores it." So am I mistaken? Do I need to do something different? I would prefer to manage the config with ldapmodify, but since I don’t change cn=config that often, I can change it manually. Robert

2 1

EXOP patches for PHP LDAP module
by Emmanuel Dreyfus 25 Apr '15

25 Apr '15

Hello For whoever is interested, I updated Pierangelo Masarati's patch to support EXOP in PHP LDAP module. Patches for PHP 5.4, 5.5 and 5.6 can be found here: http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/databases/php-ldap/files/ Previous patch for PHP 5.3 and PHP 5.2 is in the Attic. -- Emmanuel Dreyfus manu(a)netbsd.org

3 5

Re: Antw: LMDB usage with long or variable length keys
by Alex V. 25 Apr '15

25 Apr '15

On 4/24/2015 10:45 AM, Ulrich Windl wrote: >>>> "Alex V."<alex.v.odesk(a)gmail.com> schrieb am 24.04.2015 um 08:54 in Nachricht > <5539E88C.3080006(a)gmail.com>: >> On 4/24/2015 9:02 AM, Ulrich Windl wrote: >>> I think this depends very much on your access patterns (which you didn't >> explain). >>> Regards, >>> Ulrich >> What required is `get(key)` and `set(key, val)`. And, optional but not >> unwanted, range lookup. > Still unclear: What is the expected ratio between get and set for a specific key? Do you ever delete keys? If so, how many accesses do you expect before deleting a key? > > You could hash your keys using MD5 or SHA, and use the first 16 bytes as key. As your access pattern does not include enumerating the original keys, that would work, saving space vs. consuming some extra CPU cycles. I can think of other algorithms, but you were not specific with your access pattern. > > Regards, > Ulrich > I haven't figured all the details yet, I am just looking for different possibilities. number of `get`s is probably many times higher than number of `set`s or `delete`s. One possible usage case is database that stores source files metadata(symbols) for autocompletion/navigation, something like "source navigator ng"(it uses bdb). Another use case is almost the same, with urls as keys and word index as metadata. Looks like basename/url parts hierarchy will do, I am going to test it some day.

1 0

Re: Different contextCSN on multi-master replication servers
by NICOLAS KOVAC NEUMANN 24 Apr '15

24 Apr '15

Thank you both Howard and Leonid. Yes, you're right, it happened the other way around; the modification was made on the second server and propagated back to the first one. However, I don't know why the change was in turn returned to the second server - SIDs are ok as far as I know. However, as Leonid mentioned, both servers weren't synchronized correctly anyway. Turns out that yesterday we upgraded to 2.4.39-6 (as I stated in my first mail). Previously, we were using 2.4.39-3 and it seemed to work fine. We also noted that 2.4.39-6 produced some additional issues (like client and syncrepl sockets dying without any apparent reason), so today we downgraded back to 2.4.39-3 and everything seems to work just fine again. We had a look at the changelog from 2.4.39-3 to 2.4.39-6 and no change seems to be explicitly syncrepl related, but rather related to LDAPS (strange, as we use the LDAP protocol for syncrepl instead of LDAPS). Anyway, we'll keep version 2.4.39-3 as far as it works well. Thanks. Regards. 2015-04-21 23:53 GMT+01:00 Леонид Юрьев <leo(a)yuriev.ru>: > Hi Nicolás, > > 1) If contextCSN(s) are differs on servers, then are still not > syncronized (or has a glitches). > http://www.openldap.org/lists/openldap-technical/201108/threads.html#00001 > > 2) Replication takes a some time. Therefore contextCSN(s) may be equals > only when some time was no any changes. > > 3) Make sure that the time is synchronized on servers (e.g. by using > ntpdate). > > 4) Unfortunatelly, all current releases (include 2.4.39 and 2.4.40) have > enough bugs in replication code. > For example, by ITS#8081 ( > http://www.openldap.org/its/index.cgi/Software%20Bugs?id=8081) you could > get segfault, but also lost (like undo) some changes by a replication. > > 5) We made a fork of OpenLDAP project for our usecase (highload > TELCO-aware multi-master), it called ReOpenLDAP. > If you decide to build slapd from sources, I recommend use our ReOpenLDAP > ;) > > New features yet not documented in english man-pages, by you can translate > by Google: > https://github.com/ReOpen/ReOpenLDAP/releases/tag/ReOpenLDAP-2.4.41-rc > > https://github.com/ReOpen/ReOpenLDAP/commit/4fc4bc18dd4bd80909aa80700c5c19b… > > https://github.com/ReOpen/ReOpenLDAP/commit/95808b156ee36a886523b7096a75d50… > > https://github.com/ReOpen/ReOpenLDAP/commit/1c94bc17ec285388e8a8299399ed537… > > Leonid. > > 2015-04-21 16:01 GMT+03:00 Nicolás Kovac Neumann <nkovacne(a)ull.edu.es>: > >> Hi, >> >> We're currently using N-way multimaster replication on two servers for >> our LDAP directory, both for the config and the hdb databases. It's working >> fine mostly, but we've run into a possible issue with the syncrepl engine >> which we would like to cast light on. We're using CentOS 7 with >> openldap-servers version 2.4.39-6. >> >> We made an update on one of the entries (server1, in this case), so >> server2 replicated that change (as shown below in the log): >> >> ==> server1/ldap.log <== >> Apr 21 13:38:55 server1 slapd[1835]: do_syncrep2: rid=002 >> cookie=rid=002,sid=002,csn=20150421123855.643239Z#000000#002#000000 >> Apr 21 13:38:55 server1 slapd[1835]: syncrepl_message_to_entry: >> rid=002 DN: uid=user1,cn=subtree,dc=example,dc=org, UUID: >> 18a2436c-73ce-1030-95dd-b52dc05ced13 >> Apr 21 13:38:55 server1 slapd[1835]: syncrepl_entry: rid=002 >> LDAP_RES_SEARCH_ENTRY(LDAP_SYNC_MODIFY) >> Apr 21 13:38:55 server1 slapd[1835]: syncrepl_entry: rid=002 >> be_search (0) >> Apr 21 13:38:55 server1 slapd[1835]: syncrepl_entry: rid=002 >> uid=user1,cn=subtree,dc=example,dc=org >> Apr 21 13:38:55 server1 slapd[1835]: slap_queue_csn: queing >> 0x7ff8f42789f0 20150421123855.643239Z#000000#002#000000 >> Apr 21 13:38:55 server1 slapd[1835]: slap_graduate_commit_csn: >> removing 0x7ff8f435e770 20150421123855.643239Z#000000#002#000000 >> Apr 21 13:38:55 server1 slapd[1835]: syncrepl_entry: rid=002 >> be_modify uid=user1,cn=subtree,dc=example,dc=org (0) >> Apr 21 13:38:55 server1 slapd[1835]: syncprov_sendresp: >> cookie=rid=001,sid=001,csn=20150421123855.643239Z#000000#002#000000 >> Apr 21 13:38:55 server1 slapd[1835]: slap_queue_csn: queing >> 0x7ff8f42789f0 20150421123855.643239Z#000000#002#000000 >> Apr 21 13:38:55 server1 slapd[1835]: slap_graduate_commit_csn: >> removing 0x7ff8f41b7b90 20150421123855.643239Z#000000#002#000000 >> >> ==> server2/ldap.log <== >> Apr 21 13:38:55 server2 slapd[1948]: slap_queue_csn: queing >> 0x7f897affb220 20150421123855.643239Z#000000#002#000000 >> Apr 21 13:38:55 server2 slapd[1948]: syncprov_sendresp: to=001, >> cookie=rid=002,sid=002,csn=20150421123855.643239Z#000000#002#000000 >> Apr 21 13:38:55 server2 slapd[1948]: slap_graduate_commit_csn: >> removing 0x7f89307f42a0 20150421123855.643239Z#000000#002#000000 >> >> Nothing strange up to now, however, if we query the contextCSN, it >> differs on both servers. >> >> For server1, we have: >> >> contextCSN: 20150421123523.281736Z#000000#001#000000 >> contextCSN: 20150421123417.889477Z#000000#002#000000 >> >> For server2, the value for server ID 001 differs: >> >> contextCSN: 20150421115324.003502Z#000000#001#000000 >> contextCSN: 20150421123417.889477Z#000000#002#000000 >> >> However, the entry seems to replicate the entryCSN correctly on both >> servers: >> >> entryCSN: 20150421123417.889477Z#000000#002#000000 >> >> Is this the expected behavior? Shouldn't both contextCSN values match on >> both servers? >> >> Thanks! >> >> Regards, >> >> Nicolás >> >> >

2 1

LMDB usage with long or variable length keys
by Alex V. 24 Apr '15

24 Apr '15

I am new to key-value stores. I would like to use it like big persistent associative array and I want to be able to use long length keys. For example file path or URL. What technique should I use to implement such dictionary on top of LMDB? Probably some kind of hashing and collision resolution? Or just recompile with MDB_MAXKEYSIZE=2047 or something like this? Or LMDB is a wrong tool for this job?

5 7

ldap configuration question
by Ross, Daniel B. 24 Apr '15

24 Apr '15

I have looked a couple options, and read the openldap documentation, but I really dont know how to accomplish what I need to do. Here is what I am trying to do. I have a greater organization that is stuck on using Microsoft products namely Microsoft LDS. To make matters worse they present the data to my linux servers in a completely non-standard way. Its driving my solaris and linux box nuts and they simply dont want to work with it. What i need to do is continue to use the campus usernames and passwords but present the Data in a format that my linux/unix hosts can use. Is this possible? i.e. userid would still be samwise but instead of a bizzarre OU=monkeypeople,dc=example,dc=com I want it to present as people,dc=example,dc=com. The mismatches are far more and I need to add things like nfs mounts and it would be nice to allow users on only certain hosts, none of which their LDS can do, or even if it can they are not willing to do. I looked at referral and aliasing but it does not seem to be doing what I am trying to do. Passthrough authentication looks close but I cant find sufficient documentation to actually configure a system to use it. So what I want to do is use campus for the username and password authentication, but the local ldap for everything else, groups, dns, etc, etc. Thanks Daniel

1 0

Re: Antw: Re: Different contextCSN on multi-master replication servers
by Nicolás Kovac Neumann 24 Apr '15

24 Apr '15

Hi Ulrich, El 24/04/15 a las 07:07, Ulrich Windl escribió: > [...] > After haveing had such problems, I recommand exporting the databases of both > servers and compare the contents: If the mismatch, I'd import one database from > the other server's dump... > > Regards, > Ulrich That's the approach we've been using years ago but as this is a patchy workaround we rather preferred get syncrepl working correctly. After downgrading the package to 2.4.39-3 again it seems to work again, so I think we'll apply the old and good advise "If it works, don't fix it". At least not until a certain time. Thanks. Regards, Nicolás.

1 0

moznss, gnutls, openssl
by Chuck Theobald 24 Apr '15

24 Apr '15

What is the current wisdom regarding which tls library to use? I've got a version 2.4.39 installation on RHEL 6.6 for which I cannot get tls to work. I end up with the "TLS: can't connect: TLS error -5938:Encountered end of file." error. Likely a misconfiguration of moznss, though I followed one set of directions using certutil, but lack the proper setting for my ldap TLSCACertificateFile. My Debian-based ldap servers run with either openssl or gnutls. Thanks, -- Chuck Theobald System Administrator The Robert and Beverly Lewis Center for Neuroimaging University of Oregon P: 541-346-0343 F: 541-346-0345

3 2

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical