openldap-technical

openldap-technical@openldap.org

3 participants
9898 discussions

back-mdb and alias dereferencing
by Henrik Bohnenkamp 07 Jul '18

07 Jul '18

Hi, I would like to bring up the problem with back-mdb and alias-dereferening again. There is Bug ITS#7657 for this already, and in http://www.openldap.org/lists/openldap-technical/201504/msg00008.html the problem was also addressed. Probably in other places as well, but I did not find anything else in the archives. I would like to reconfigure all my LDAP clusters to use back-mdb, however, I can't: my LDAP tree has about 2 Mio entries, 600000 of which are alias entries. I observe that when using back-mdb and making search request with search scope "sub" and deref=always, than this search request will take seconds before a response... in contrast, with deref=never, the same search request takes only millisecs. It does not matter whether there are aliases in the search scope or not... with deref=always it always takes seconds, each request keeping one CPU thread very busy. So if many requests come in with sub/always (which unfortunately happens a lot because many LDAP clients just set deref=always as a default) it is a sure way to bring down an LDAP cluster configured with back-mdb. The problem has haunted me for a while (incidents, incidents :-), and I have tried to find out where it comes from. I would like to share my conjectures here. A search request with sub/always eventually ends up in the function search_aliases(..), located in back-mdb/search.c (code/linenumbers is from v2.4.46 ): 157 /* Find all aliases in database */ 158 MDB_IDL_ZERO( aliases ); 159 rs->sr_err = mdb_filter_candidates( op, isc->mt, &af, aliases, 160 curscop, visited ); 161 if (rs->sr_err != LDAP_SUCCESS || MDB_IDL_IS_ZERO( aliases )) { 162 return rs->sr_err; 163 } 164 oldsubs[0] = 1; 165 oldsubs[1] = e_id; 166 167 MDB_IDL_ZERO( visited ); 168 MDB_IDL_ZERO( newsubs ); 169 170 cursoro = 0; 171 ido = mdb_idl_first( oldsubs, &cursoro ); 172 173 for (;;) { 174 /* Set curscop to only the aliases in the current scope. Start with 175 * all the aliases, then get the intersection with the scope. 176 */ 177 rs->sr_err = mdb_idscope( op, isc->mt, e_id, aliases, curscop ); After adding a bit of extra debug code, I could see in the logs that the time is actually spent in line 177, i.e. the mdb_idscope function. What I believe is happening is the following: in line 159 the index list of all aliases in the DB is constructed and put into the "aliases" list. The list comes from the objectClass index, but since there are so many alias entries, the index contains not an explicit list, but a range... a range that covers basically *all* 2 Mio entries in the DB. (this happens also with the hdb backend, so the fact that "aliases" is a range is perhaps part, but not cause of the problem). Then the for loop is entered in 173, and mdb_idscope in 177 is called. The idea there is to compute the intersection of alias list "aliases" and search scope, i.e. get all aliases that are in the actual search scope. This is done by walking through the 2 millions indexes represented by the range in "aliases", and deciding whether the id is in the scope or not. This, if I understand correctly, by walking up the LDAP tree from leaf towards root (actually walking up the structure of the dn2id DB) ... if we encounter the search base on the way, the alias is part of the search scope, if we reach the root, then not. Computing this intersection is what costs time, and it does not depend on whether the intersection is empty or not. Apparently it is done differently in back_hdb and by extension, back_bdb. I have not analysed that code there in detail but I believe the intersection is computed by actually looking at 2 different ID lists, one containing the aliases (range) again, the other one the actual IDs of the search scope. Which, it appears, is more efficient. So the problem seems to be to compute the relevant aliases to start expanding from. The way it is done now the complexity seems to be linear in the number of aliases, if not number of entries on the LDAP tree. For my own use cases at least it would be much more efficient to just do a tree traversal of the search scope, pick up the relevant aliases and put them in the list. I believe even for large trees this would work quite well... if I search my whole tree for entries with objectClass=alias (deref=never, of course), this is amazingly fast. I am currently looking into how something like that could be implemented, but I need to dive much deeper in the code before I could come up with something worth considering. What are the views of the developers on this? Does the analysis sound plausible? Could a fix be easily obtained? I'm willing to try out any patch thrown my way. Somewhere is written that mdb will replace hdb/bdb eventually ... I believe this problem needs a fix before that can happen. Henrik -- Henrik Bohnenkamp Senior Sysadmin Linux & Applications Operation United Internet Corporate Services GmbH | Brauerstraße 50 | 76135 Karlsruhe | Germany Phone: +49 721 91374 - 4159 E-mail: hbohnenkamp atatatat united-internet.de | Web: www.united-internet.de Amtsgericht Montabaur / HRB 23031 Geschäftsführer: Verena Amann, Markus Kadelke, Frank Krause, Guido Mannshausen Member of United Internet This e-mail may contain confidential and/or privileged information. If you are not the intended recipient of this e-mail, you are hereby notified that saving, distribution or use of the content of this e-mail in any way is prohibited. If you have received this e-mail in error, please notify the sender and delete the e-mail.

2 2

Fixing slapo-nops (ITS#8759)
by manu＠netbsd.org 06 Jul '18

06 Jul '18

Hello Following ITS#8759, I am looking for ideas about how to fix slapo-nops. This overlays suppress nilpotent operations that replace an attribute values by the same values. The code is in contrib/slapd-modules/nops/nops.c The problem is that slapo-nops assumes it can free the nilpotent struct Modifications after removing them from the list, but for instande slapo-memberof uses struct Modifications allocated on the stack. Not freeing in slapo-nops is not a fix, since unlinked struct Modifications will not be freed anywhere else, and we will get a memory leak. Would it make sense to add a SLAP_MOD_ONSTACK flag in struct Modification's sm_flags so that we can tell the difference? Any other idea? -- Emmanuel Dreyfus http://hcpnet.free.fr/pubz manu(a)netbsd.org

2 1

slapd crashes on startup
by BÖSCH Christian 06 Jul '18

06 Jul '18

Hi, I have 2 nodes in multimaster setup. Node1 doesn’t startup anymore and crashes. In debug mode the last lines are: 5b3f6b78 <= index_entry_del( 1121725, "uid=jst9897,ou=ais2ldap,o=abc.at" ) success ./../../../libraries/liblmdb/mdb.c:5276: Assertion 'NUMKEYS(mp) > 1' failed in mdb_page_search_root() Abort What does that mean, and what to do to get online again and have the DB in sync with Node2? Thanks in advance, Christian

2 4

Client IP in Loadbalanced configuration
by Pascal kolijn 06 Jul '18

06 Jul '18

List, I have a haproxy loadbalanced ldap service. I would like to see the 'original' client IP in the openldapserver log files, because now I only see the haproxies IP addresses for all sessions. I understand that haproxy has a 'feature' called the PROXY protocol [https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt] that adds the original IP as metadata somewhere in the request, I had hopes that this would work with openldap >= 2.3, because they tested such a PROXY protocol signature packet. But it is not, yet(?), implemented in openldap. I also understand that the slapd-ldap backend has a option called session-tracking-request so some serverside/backend side stuff seems to be available somewhere in openldap.. So any pointers or tips to achieve this, or use a different setup? -- Pascal Kolijn Vrije Universiteit Amsterdam

2 1

time-base release schedule?
by David Magda 03 Jul '18

03 Jul '18

Hello, Would it be possible to moved to a timed release schedule? Looking at the ChangeLog, the current release 'schedule' seems to be a bit... non-deterministic: http://www.openldap.org/software/release/changes.html Moving towards a fixed-date schedule (once or twice a year?) would allow downstream maintainers to update things like Linux distributions in a more predictable fashion. The "maintenance releases" would be nothing more than tagging all of the patches since the last release and creating a tarball. If there are zero patches since the last releases, then a new tarball may not be needed (or a "NOP release" could be done just to keep people in-practice). Otherwise, regardless of whether is is one or one-dozen patches, a new version is revved. If there is a security issue, a release could be issued outside of the fixed schedule. Ideally (IMHO) security releases would only include the security fix so people can be confident that it is a low-risk change that won't change / break other behaviours. Given that most users of OpenLDAP "consume" it via their distribution-of-choice's package, this would (IMHO) increase the chances of more recent versions of OpenLDAP being used (especially in Debian and Ubuntu, but also Fedora). Recent Debian releases occur in Q2, and Ubuntu does April/October (LTSes are April), while Fedora (roughly) does May/November. Perhaps a February/September cycle? Not sure whether this is something for 2.4, or for 2.5 when it comes out. Any thoughts? Regards, David

5 8

LMDB docs
by KS Sreeram 03 Jul '18

03 Jul '18

Hello folks On this page: http://www.lmdb.tech/doc/starting.html The following passage occurs: > Because LMDB is very efficient (and usually zero-copy), the data returned in an MDB_val structure may be memory-mapped straight from disk. In other words look but do not touch (or free() for that matter). Once a transaction is closed, the values can no longer be used, so make a copy if you need to keep them after that. The second sentence above is a bit misleading. Returned data cannot be used after a subsequent update. Not mentioning that is misleading. Perhaps the second sentence can be replace with this: > The values can no longer be used after a subsequent update or after the transaction ends. Make a copy if you need to keep them after that. The documentation for `mdb_get` is correct and does mention the constraints. It’ll be nice to update this page too. Best KS Sreeram

1 0

test059-slave-config schema replication
by Richard Gray 03 Jul '18

03 Jul '18

Hi, I am trying to get cn=config replication working as demonstrated by the test059-slave-config test script, but have run into a problem with schema replication. This is the same issue raised on the list previously by Christopher Strider Cook [1], but I thought I'd bring it up again as it's been seven years, so perhaps there are new ideas in this space. Furthermore, I can demonstrate how to reproduce the problem with a simpler configuration than that described by Christopher. [1]: https://www.openldap.org/lists/openldap-technical/201106/msg00157.html test059-slave-config describes two configuration trees, cn=config as used by a master server, and cn=config,cn=slave which is replicated to any number of replica servers. Furthermore, there is a syncrepl agreement defined on the cn=config,cn=slave database that replicates the master schema from cn=schema,cn=config into the replica configuration tree as cn=schema,cn=config,cn=slave. If you make any changes to the cn=config,cn=slave tree however, the next time the master is restarted, it will complain that the "consumer state is newer than provider". At this point, I'm considering ditching the schema replication, and maintaining it independently on the master and replica servers. It's probably not much of an imposition seeing as I will need to maintain the rest of the master and replica configuration independently anyway. Nevertheless, I thought I'd raise the issue here to see if anyone had any clever solutions to this problem. Here's how to reproduce the issue (OpenLDAP 2.4.46 on Linux 4.15.12): 1. Download and build OpenLDAP 2. Run test059-slave-config: ./run test059-slave-config 3. Start a slapd using the provider config created by the test script: /home/richard/working/openldap-2.4.46/tests/../servers/slapd/slapd -F /home/richard/working/openldap-2.4.46/tests/testrun/cfpro.d -h ldap://localhost:9011/ -d 0x4105 4. Make a change under cn=config,cn=slave: $ ldapadd -x -h localhost -p 9011 -D cn=config -w EMWRueBU dn: olcBackend=bdb,cn=config, cn=slave objectClass: olcBackendConfig olcBackend: bdb adding new entry "olcBackend=bdb,cn=config, cn=slave" ^C 5. Restart the slapd instance: ^C /home/richard/working/openldap-2.4.46/tests/../servers/slapd/slapd -F /home/richard/working/openldap-2.4.46/tests/testrun/cfpro.d -h ldap://localhost:9011/ -d 0x4105 ...SNIP... 5b3b0202 conn=1005 op=1 SRCH base="cn=schema,cn=config" scope=2 deref=0 filter="(objectClass=*)" 5b3b0202 conn=1005 op=1 SRCH attr=* + 5b3b0202 send_ldap_result: conn=1005 op=1 p=3 5b3b0202 send_ldap_result: err=0 matched="" text="" 5b3b0202 send_ldap_result: conn=1005 op=1 p=3 5b3b0202 send_ldap_result: err=53 matched="" text="consumer state is newer than provider!" 5b3b0202 send_ldap_response: msgid=2 tag=101 err=53 ber_flush2: 52 bytes to sd 16 read1msg: ld 0x7fd46c109ba0 msgid 2 all 0 ber_get_next 5b3b0202 conn=1005 op=1 SEARCH RESULT tag=101 err=53 nentries=0 text=consumer state is newer than provider! ber_get_next: tag 0x30 len 50 contents: read1msg: ld 0x7fd46c109ba0 msgid 2 message type search-result ber_scanf fmt ({eAA) ber: read1msg: ld 0x7fd46c109ba0 0 new referrals read1msg: mark request completed, ld 0x7fd46c109ba0 msgid 2 request done: ld 0x7fd46c109ba0 msgid 2 res_errno: 53, res_error: <consumer state is newer than provider!>, res_matched: <> ldap_free_request (origid 2, msgid 2) 5b3b0202 do_syncrep2: rid=001 LDAP_RES_SEARCH_RESULT ldap_parse_result ber_scanf fmt ({iAA) ber: ber_scanf fmt (}) ber: ldap_err2string 5b3b0202 do_syncrep2: rid=001 LDAP_RES_SEARCH_RESULT (53) Server is unwilling to perform ldap_err2string Any suggested workarounds would be appreciated. Regards, -- *Richard Gray* | Operations Technical Lead *DDI:* +64 9 950 2196 *Fax:* +64 9 302 0518 *Mobile:* +64 21 050 8178 *Freephone:*0800 SMX SMX (769 769) *SMX Limited:* Level 15, 19 Victoria Street West, Auckland, New Zealand *Web:* http://smxemail.com SMX | Cloud Email Hosting & Security _____________________________________________________________________________ This email has been filtered by SMX. For more info visit http://smxemail.com _____________________________________________________________________________

1 0

sql backend & upate data
by Arianna Milazzo 29 Jun '18

29 Jun '18

Hello! I use openLDAP with a sql backend. I use mySQL with InnoDB engine. Well, if I modify a data on db, I can't see the new value on openLDAP until I restart it. :O I can't restart openLDAP every time I modify something! Any solution? Thanks, Arianna

1 1

Performance problem with slapd instances.
by Saurabh Lahoti 28 Jun '18

28 Jun '18

Dear, We've started observing strange & peculiar performance behavior for slapd instances. While ldap bind response is more than 1.5 second on one server then on another it's just 0.006 seconds. Both the servers have similar OS, system resources & ldap version. OS: RHEL 6.9 LDAP: 2.4.44 Backend: bdb Could you please suggest some probable causes..? Thanks & Kind Regards, Saurabh Lahoti.

2 2

replicating memberOf attribute
by Chris 27 Jun '18

27 Jun '18

Hi, We are using a provider and three consumers that are not syncing the memberOf attributes of inetOrgPerson entries. memberof and refint modules are loaded on the provider and inetOrgPersons' memberOf attributes are correctly updated on the provider. How can I troubleshoot this? Thanks. Chris.

2 2

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical