openldap-technical

openldap-technical@openldap.org

14 participants
9795 discussions

Error message nonpresent_callback in replication
by Michael Wuensche 06 Jun '16

06 Jun '16

Hello, We recently upgraded our Openldap installation to a current version. Our installation consists of 2 servers with several databases, of which a few are replicated in mirror mode. Together with the upgrade process we reconfigured which data gets replicated and which not and modified the databases accordingly. Now, when we synchronize data between the nodes the intial replication works just fine. But replication of changes afterwards fails and we get these error messages: Jun 3 16:36:09 wam01 slapd[19027]: [ID 795510 local4.debug] do_syncrep2: rid=025 LDAP_RES_INTERMEDIATE - SYNC_ID_SET Jun 3 16:36:09 wam01 last message repeated 29 times Jun 3 16:36:09 wam01 slapd[19027]: [ID 365351 local4.debug] do_syncrep2: rid=025 LDAP_RES_SEARCH_RESULT Jun 3 16:36:09 wam01 slapd[19027]: [ID 385048 local4.debug] do_syncrep2: rid=025 cookie=rid=025,csn=20160603143607.847747Z#000000#000#000000 Jun 3 16:36:10 wam01 slapd[19027]: [ID 338579 local4.debug] nonpresent_callback: rid=025 present UUID 6c782570-d2b6-1033-84c4-3d93063e27ca, dn ou=Directories,o=lpfroot Any idea what may have gone wrong here? The suffix o=lpfroot is in a non-replicated database, the glued sub-suffix ou=Directories,o=lpfroot is in a replicated database. The databases are still bdb as before. Best regards, Michael -- Michael Wünsche Principal Consultant Identity & Access Management Tel.: +49 (6151) 868-7251 Fax: +49 (6151) 868-7264 Michael.Wuensche(a)devoteam.com Devoteam GmbH, Gutenbergstraße 10, 64331 Weiterstadt, Germany URL: www.devoteam.de ------------------------------------------------------------------------------------------------------------ Geschäftsführer: Jürgen Hatzipantelis; Sitz der Gesellschaft: Weiterstadt; Amtsgericht Darmstadt HRB 6450; Umsatzsteuer-ID Nr.: DE 172 993 071

2 1

Issues with pcache overlay purging entries
by Andrew Widdersheim 03 Jun '16

03 Jun '16

Hello, Seeing strange issues when using the pcache overlay as a caching method on some servers. More specifically, not seeing results getting purged from the cache even though the TTL has expired. I have been able to reproduce on the following: Ubuntu 16.04 running slapd 2.4.42+dfsg-2ubuntu3.1 Ubuntu 14.04 running slapd 2.4.31-1+nmu2ubuntu8 Here are the steps that I am taking in order to reproduce the issue. First, I'm running a query similar to this: ldapsearch -x -LLL -h localhost -b 'dc=foo,dc=com' '(isphoneoperator=TRUE)' uid | grep "uid:" | grep someuser When I set the "isphoneoperator" attribute to TRUE the user shows up and everything is great. If I then go and set the attribute to FALSE and restart slapd *before* the TTL of the query expires on it's own naturally then that user remains in the cache forever and the only way to correct the issue is to stop slapd, delete all the caching databases and start slapd again. There are a few things I've noticed in the logs that I don't quite understand. When starting slapd for the first time with a fresh set of cache databases the logs look like the ones below and will repeat every time the TTL expires: Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=1 Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=2 Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=3 Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=4 Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=5 Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=6 Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=7 Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=8 Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=9 However, when I restart slapd the logs change to look like this: Jun 1 20:44:55 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0] Jun 1 20:45:25 vagrant slapd[11817]: ENTRY ADDED/MERGED, CACHED ENTRIES=0 Jun 1 20:45:25 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0] Jun 1 20:45:55 vagrant slapd[11817]: ENTRY ADDED/MERGED, CACHED ENTRIES=0 Jun 1 20:45:55 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0] Jun 1 20:46:26 vagrant slapd[11817]: ENTRY ADDED/MERGED, CACHED ENTRIES=0 Jun 1 20:46:26 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0] Jun 1 20:46:56 vagrant slapd[11817]: ENTRY ADDED/MERGED, CACHED ENTRIES=0 Jun 1 20:46:56 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0] Jun 1 20:47:26 vagrant slapd[11817]: ENTRY ADDED/MERGED, CACHED ENTRIES=0 Jun 1 20:47:26 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0] I find this to be pretty odd and I'm not sure I quite understand the reason for this behavior. Also, when I run slapd in the foreground with (-d -1 for example) I notice some differences there as well. For example, with a fresh set of cache databases I'll see messages like: 57503573 ==> hdb_delete: uid=someuser,ou=users,dc=foo,dc=com However, when I restart slapd I never see these again. I have noticed that if I set pcachePersist to TRUE both my issue above where users are never purged and this logging inconsistency go away and everything works great. I guess I don't really understand what the pcachePersist parameter does. I've read the man pages for slapo-pcache(5) and I also found the following mailing list post and it is all a bit confusing to me: http://www.openldap.org/cgi-bin/wilma_hiliter/openldap-technical/201007/msg… I've looked for fixed bugs or any other posts that might indicate what is going on here but haven't turned up anything relevant that might be affecting the versions of slapd I'm running. I guess I'm wondering a few things: 1. Have a stumbled upon a bug? 2. Is this logging inconsistency normal? 3. What does pcachePersist do? 4. Is there a way to forcefully clear the cache other than deleting the database files or do you have to wait for the TTL to expire? Thanks, Andrew

1 0

Virtual list view problem
by Venish Khant 31 May '16

31 May '16

Hi all I am using cpan Net::LDAP module to access LDAP entries. I want to search LDAP entries using Net::LDAP search method. When I do search, I want some limited number of entries from search result, for this(searching) process I am using Net::LDAP::Control::VLV module. But I get error on VLV response control. Please, any one have idea about this error. * Error:* Died at vlv.pl line 50, This is my example. I changed the font style of line 50 #!/usr/bin/perl -w use Net::LDAP; use Net::LDAP::Control::VLV; use Net::LDAP::Constant qw( LDAP_CONTROL_VLVRESPONSE ); use Net::LDAP::Control::Sort; sub procentry { my ( $mesg, $entry) = @_; # Return if there is no entry to process if ( !defined($entry) ) { return; } print "dn: " . $entry->dn() . "\n"; @attrs = $entry->attributes(); foreach $attr (@attrs) { #printf("\t%s: %s\n", $attr, $entry->get_value($attr)); $attrvalue = $entry->get_value($attr,asref=>1); #print $attr.":". $entry->get_value($attr)."\n"; foreach $value(@$attrvalue) { print "$attr: $value\n"; } } $mesg->pop_entry; print "\n"; } $ldap = Net::LDAP->new( "localhost" ); # Get the first 20 entries $vlv = Net::LDAP::Control::VLV->new( before => 0, # No entries from before target entry after => 19, # 19 entries after target entry content => 0, # List size unknown offset => 1, # Target entry is the first ); my $sort = Net::LDAP::Control::Sort->new( order => 'cn' ); @args = ( base => "dc=example,dc=co,dc=in", scope => "subtree", filter => "(objectClass=inetOrgPerson)", callback => \&procentry, # Call this sub for each entry control => [ $sort, $vlv ], ); $mesg = $ldap->search( @args ); # Get VLV response control *($resp) = $mesg->control( LDAP_CONTROL_VLVRESPONSE ) or die;* $vlv->response( $resp ); # Set the control to get the last 20 entries $vlv->end; $mesg = $ldap->search( @args ); # Get VLV response control ($resp) = $mesg->control( LDAP_CONTROL_VLVRESPONSE ) or die; $vlv->response( $resp ); # Now get the previous page $vlv->scroll_page( -1 ); $mesg = $ldap->search( @args ); # Get VLV response control ($resp) = $mes # Now page with first entry starting with "B" in the middle $vlv->before(9); # Change page to show 9 before $vlv->after(10); # Change page to show 10 after $vlv->assert("B"); # assert "B" $mesg = $ldap->search( @args );g->control( LDAP_CONTROL_VLVRESPONSE ) or die; $vlv->response( $resp ); -- Venish Khant www.deeproot.co.in

4 11

Re: Documentation
by Daniel Howard 30 May '16

30 May '16

My recent experience is OpenLDAP on Ubuntu. I thought I would go with OpenLDAP's guide because they should know better, but the quick start was for older versions or something and hadn't been updated. I like documentation systems that allow for user feedback, comments, or patches via github. If you want community improvements then maybe low hanging fruit is a link in the doc template on how to interact with the documentation.... If the current docs platform has no mechanism for feedback and comments on a specific document, the source code/ build process is obscure ... Maybe consider porting even a "fork " to github and solicit community feedback (you can file issues in a familiar mechanism, and take in patches from dudes like me...) For what it's worth, the Ubuntu OpenLDAP quick start guide and only a few minor hiccups, and I would recommend that resource for anyone looking to implement or update a quickstart guide. The Faq-o-Matic ... Man a modern stack overflow site might be nice but a question in the back of my mind is whether the advice is still useful. I don't know if the FoM does this but if the answers showed dates then I could evaluate an answer that's say 2-4 years old with greater confidence than say something 12 years old ... One of the challenges of a venerable, old and mature project like OpenLDAP. I guess what I'm trying to say, from my mobile phone, is if the documentation were more "agile" in engaging the community for identifying issues and taking in corrections, we may all be happier. I would be open to volunteering to port docs... Thanks, -Danny -- Sent from my Western Electric Model 500 On Apr 29, 2016 5:01 PM, "Gavin Henry" <ghenry(a)openldap.org> wrote: Hi all, List what you love about our docs: List what you hate: List what you'd love to see: List what you can help with: Thanks. -- Kind Regards, Gavin Henry.

6 9

slapadd hang in bdb_tool_entry_close / ldap_pvt_thread_cond_wait
by Bannister, Mark 27 May '16

27 May '16

Hi, Can anyone shed light on why my slapadd process on Solaris 11 was hanging (slapd 2.4.30)? I was piping 'gunzip -c' through to slapadd (from an overnight cron job). This usually works fine. I'm guessing perhaps the gunzip process encountered an error, maybe an EPIPE was sent, but the stack trace many days later from slapadd doesn't look right. Only 2 threads, and both of them in ldap_pvt_thread_cond_wait(), but no other threads so what are they waiting for exactly? # pstack /var/tmp/cores/core.slapd.0.16189.1463492837 core '/var/tmp/cores/core.slapd.0.16189.1463492837' of 16189: /usr/sbin/slapadd -q -f /etc/openldap/slapd.conf ------------ lwp# 1 / thread# 1 --------------- 07ff7af9 lwp_park (0, 0, 0) 07ff1193 cond_wait_queue (82ed048, 82ed020, 0, 7ff16b0) + 63 07ff1728 __cond_wait (82ed048, 82ed020, f5a685c8, 7ff1771) + 89 07ff177f cond_wait (82ed048, 82ed020, f5a685f8, 7ff17b7) + 27 07ff17cc pthread_cond_wait (82ed048, 82ed020, f5a68618, 7e56137) + 24 07e5614c ldap_pvt_thread_cond_wait (82ed048, 82ed020, f5a68648, 81b7e29) + 24 081b7e7a bdb_tool_entry_close (8e9d398, ffffffff, f5a68698, 82ecc38) + 62 081a38ec slapadd (4, f5a68bbc, f5a68bbc, 80dccb1) + cb8 080dcd2c main (4, f5a68bbc, f5a68bd0, ef60f968) + ac 080dc86d _start (4, f5a68c8e, f5a68ca0, f5a68ca3, f5a68ca6, 0) + 7d ------------ lwp# 2 / thread# 2 --------------- 07ff7af9 lwp_park (0, 0, 0) 07ff1193 cond_wait_queue (8e6b0e4, 8e6b0cc, 0, 7ff16b0) + 63 07ff1728 __cond_wait (8e6b0e4, 8e6b0cc, d7dffdb8, 7ff1771) + 89 07ff177f cond_wait (8e6b0e4, 8e6b0cc, 8e6b0c8, 7ff17b7) + 27 07ff17cc pthread_cond_wait (8e6b0e4, 8e6b0cc, 0, 7e56137) + 24 07e5614c ldap_pvt_thread_cond_wait (8e6b0e4, 8e6b0cc, d7dfffc8, 7e54bfc) + 24 07e54d66 ldap_int_thread_pool_wrapper (8e6b0c8, 803a000, d7dfffe8, 7ff7769) + 242 07ff77bc _thrp_setup (ee1e0240) + 9d 07ff7aa0 _lwp_start (ee1e0240, 0, 0, 0, 0, 0) Thanks, Mark. ________________________________ NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.

2 7

Improved handling for large number of databases / Access newly opened database from another transaction
by Jürg Bircher 27 May '16

27 May '16

Hello Improved handling for large number of databases =============================================== There is a increased performance penalty the more databases are created within the same environment. I was looking for a way the improve that by keeping the simplicity of tracking databases within a list with direct access by index (MDB_dbi). mdb_dbi_open() is however not improved with the assumption that the database handle (dbi) is cached in the application. So mdb_dbi_open() should happen only once for each database during the life time of an application. One issue is that mdb_txn_begin() (for read-only transactions) calloc the sizeof(MDB_txn) + me_maxdbs * sizeof(MDB_db + 1). The plus 1 for the dbflags. However it is sufficient only to malloc that size and clear the sizeof(MDB_txn) memset(txn, 0, sizeof(MDB_txn) After that the data beyond the MDB_txn is not initialized which is ok for the moment. The next improvement happens in mdb_txn_renew0() where the dbflags are only set to DB_UNUSED (a new flag) for each database currently opened in the environment. memset(txn->mt_dbflags, DB_UNUSED, txn->mt_numdbs); The former code used to to loop through each database to calculate the dbflags. This is still done but lazily for each accessed database with the assumption that a read only transaction rarely uses all databases of the environment. The lazy initialization of the dbflag happens in the macro TXN_DBI_EXIST which is always used when a database handle (dbi) is passed to an function. The flags are updated in mdb_setup_db_info() once a database is access which is marked as unused (DB_UNUSED). static int mdb_setup_db_info(MDB_txn *txn, MDB_dbi dbi) { /* Setup db info */ uint16_t x = txn->mt_env->me_dbflags[dbi]; txn->mt_dbs[dbi].md_flags = x & PERSISTENT_FLAGS; txn->mt_dbflags[dbi] = (x & MDB_VALID) ? DB_VALID|DB_USRVALID|DB_STALE : 0; return (txn->mt_dbflags[dbi] & validity); } /** Check \b txn and \b dbi arguments to a function and initialize db info if needed */ #define TXN_DBI_EXIST(txn, dbi, validity) \ ((txn) && (dbi)<(txn)->mt_numdbs && (((txn)->mt_dbflags[dbi] & (validity)) || (((txn)->mt_dbflags[dbi] & DB_UNUSED) && mdb_setup_db_info((txn), (dbi), (validity))))) The next improvement is done in any function which needs to loop through the databases for example in mdb_cursors_close(). Again the more databases in the environment the longer the execution time. It should be best if looping only through dbflags and searching for those databases which are used (!DB_UNUSED). This could be done byte wise or more efficient in 8/4 byte steps comparing with an extended mask DB_UNUSED_LONG instead of DB_UNUSED. So we can skip 8 or 4 (32 bit) unused databases in one step (still with the assumption that a transaction rarely uses all databases of the environment). So the loop looks as follows always starting at the lower index to avoid alignment issues with ARM prior v6. #define DB_UNUSED 0x20 /**< DB not used in this txn */ #ifdef MDB_VL32 #define DB_UNUSED_LONG 0x20202020 /* DB_UNUSED long mask for fast tracking */ #else #define DB_UNUSED_LONG 0x2020202020202020 /* DB_UNUSED long mask for fast tracking */ #endif #ifdef MDB_VL32 #define MDB_WORD unsigned int #else #define MDB_WORD unsigned long long #endif MDB_dbi n = src->mt_numdbs; MDB_dbi i = 0; while (1) { unsigned int upper = i + sizeof(MDB_WORD); if (upper < n) { // skip unused if ((*(MDB_WORD *)(tdbflags + i)) == DB_UNUSED_LONG) { i = upper; continue; } } else { upper = n; } for (; i < upper; i++) { // any other filter criteria appropriate to the function .... } if (i >= n) { break; } } Access newly opened database from another transaction ======================================================= A transaction tracks newly opened databases and if the transaction is committed the newly opened databases are propagated to the list of open databases of the environment. However if the read only transaction is aborted the databases are not propagated. If database handles (MDB_dbi) are cached in the application to avoid calling mdb_dbi_open() there might be the situation of two threads running a read only transaction concurrently. First threads opens the database and commits the read-only transaction. The database is added to the list of open databases in the environment. The returned dbi is globally cached in the application. The second thread also wants to access the same database and finds the database handle in the global application cache. The database handle however is not valid as the transaction only uses a snapshot of open databases. So the second thread gets an EINVAL when using that database handle. This should not happen as the database is open and added to the environment. The following should fix this issue by updating the mt_numdbs and marking the delta with DB_UNUSED static int mdb_update_db_info(MDB_txn *txn, MDB_dbi dbi) { /* propagate newly opened db from env to current txn. Mark them as unused */ if (txn->mt_numdbs < txn->mt_env->me_numdbs) { memset(txn->mt_dbflags + txn->mt_numdbs, DB_UNUSED, txn->mt_env->me_numdbs - txn->mt_numdbs); } txn->mt_numdbs = txn->mt_env->me_numdbs; return dbi < txn->mt_numdbs; } /** Check \b txn and \b dbi arguments to a function and initialize db info if needed */ #define TXN_DBI_EXIST(txn, dbi, validity) \ ((txn) && ((dbi)<(txn)->mt_numdbs || mdb_update_db_info((txn), (dbi))) && (((txn)->mt_dbflags[dbi] & (validity)) || (((txn)->mt_dbflags[dbi] & DB_UNUSED) && mdb_setup_db_info((txn), (dbi), (validity))))) If interested let me know how to contribute. Hope it is useful! Regards Juerg

2 5

memberOf group searches very slow
by Jason Whitener 27 May '16

27 May '16

We have some groups that contain over 250,000 members. Are there any general tips for improving performance when applications/code attempts to lookup memberOf values in the ldap? For example, I have something like ou=SomeGroup,ou=Groups,o=ourOrg. ou=SomeGroup has over 250,000 memberOf: values. Attempting to modify, search, insert, etc.. into that memberOf multivalue field takes over 20 seconds per user.

2 1

problems connecting to ldaps:// under high load with ppc64 client
by Matthias Leopold 27 May '16

27 May '16

hi, i'm operating an owncloud server that connects to an IBM Tivoli Directory Server as LDAP backend. the ldap admin tells me he is seeing "null binds" from my owncloud server in his logs: 2016-05-24T14:32:56.349452+2:00 srvr_ssl_read: EIO in handshake. EWOULDBLOCK timeout. Read: -2 of 0 2016-05-24T14:32:56.350445+2:00 GLPSSL019E The SSL layer has reported an unidentified internal error, SSL extended error code:406. 2016-05-24T14:32:56.351813+2:00 GLPSRV022E Failed to initialize secure connection from client (connection ID: 61786, IP address: x.x.x.x, Port: 59921). 2016-05-24T14:32:56.357220+2:00 GLPSRV044W Client connection from x.x.x.x bound as NULL closed by server. i investigated on my server and noticed that it has problems connecting to the ldaps://ldap.example.com uri (which is the ITDS server) under high client system load, whereas connection to ldap://ldap.example.com is ok. $ ldapsearch -v -x -z 0 -H ldaps://ldap.example.com -b "ou=groups,dc=example,dc=com" -v "objectClass=posixGroup" ldap_initialize( ldaps://ldap.example.com:636/??base ) ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1) my server (RHEL 7 on a ppc64 LPAR) is using the openldap clients/libraries. the high load that is causing the problems is on _my_ server. is there any specific tuning (besides increasing RAM/CPU) i can do to optimize ldaps client queries? i'm thinking of tuning the tcp stack or something similar, but i'm not an expert on this. where can i look for debug info? i have strace and tcpdump output thx matthias

3 2

Use OpenLDAP for some users and as a proxy for another set of users?
by Siebrand Mazeland 26 May '16

26 May '16

Hi. I'm a first time poster, new to OpenLDAP, and I have identified this list as the (hopefully) best place for my question. I have an Active Directory that contains accounts and groups for employees. Besides that, there is a group of around 1000 people that also need to authenticated and authorized (based on group membership). I'm trying to assess if OpenLDAP can be used for a scenario to avoid Windows CAL license costs. Is it possible to administer and authenticate the non-employees in OpenLDAP, and proxy requests about users that are not found in OpenLDAP to an AD? The information needed by the applications using OpenLDAP would be UPN, sAMAccountName, email address and group membership of the authenticated users. If this can be accomplished with OpenLDAP, that would a) be very nice, and b) I would like you to explain this in brief here, and approach me off-list to help me accomplish this. If there's no ready-made recipe for this, and it can be done, I'm willing to publish the configuration so others can benefit from the work, too. Thanks. -- Siebrand Mazeland

2 1

ABI report for LMDB
by Ponomarenko Andrey 26 May '16

26 May '16

Hello, The LMDB library is now tracked by the ABI tracker project: http://abi-laboratory.pro/tracker/timeline/lmdb/ One can look at the recent changes in the API/ABI and analysis of backward compatibility and navigate over the history of API/ABI changes in the library. The reports are generated daily by the abi-compliance-checker and abi-tracker tools: https://github.com/lvc/abi-tracker Thank you.

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical