openldap-technical

openldap-technical@openldap.org

5 participants
9899 discussions

Re: Re: ldapdelete: Use wild card to delete multiple enetries
by Parveen Verma 09 Jun '16

09 Jun '16

Thanks for your answer. So I take it that we can't give wild card/regular expression in ldapdelete. I have just installed an OpenLDAP with LMDB only [ without bdb/hdb]. ./configure --prefix=/my/home/directory/parveen/OpenLDAP_LMDB --exec-prefix=/my/home/directory/parveen/OpenLDAP_LMDB --enable-bdb=no --enable-hdb=no So I was adding and deleting entries and it was taking time. For addition, I switch from ldapadd to slapadd [stoping slapd for a while]. For deletion, I was thinking of some wild card like things to see if timing can be improved. I was adding some 100K entries and deleting them, it was taking time, so I thought it could be faster if we give command in one go and then ldapdelete just receives the output and send to STDOUT/STDERR. To Others, Is there any plan for adding wild cards or its in my fantasy only? The GURU has well decided not to provide such things as ldapdelete is destructive and a little misunderstanding in handling wildcards/regular expression can delete the DB. Thanks anyways!! ---------------- Hi, you cannot use wildcards in deletes as per my understanding. You can, however, use wildcards in search filters. If you are on a Unix-like system, you can feed the output of a search with a bit of transformation into ldapmodify. You may want to try something like this (I ran this on Linux): ldapmodify -f <(ldapsearch -b "dc=ALCATEL,dc=FR" "(cn=Parveen30*)" | grep dn: | sed 's/$/\nchangetype:delete\n'/) Regards, tamás On Wed, Jun 8, 2016 at 5:54 PM, Parveen Verma <parveen.subs(a)gmail.com> wrote: > Hello All, > > Is there a Way to delete multiple LDAP entries by using some wildcards or > regular expression. > Like I have below entries. > dn: cn=Parveen20,dc=ALCATEL,dc=FR > dn: cn=Parveen200,dc=ALCATEL,dc=FR > dn: cn=Parveen201,dc=ALCATEL,dc=FR > dn: cn=Parveen202,dc=ALCATEL,dc=FR > dn: cn=Parveen203,dc=ALCATEL,dc=FR > dn: cn=Parveen204,dc=ALCATEL,dc=FR > dn: cn=Parveen205,dc=ALCATEL,dc=FR > dn: cn=Parveen206,dc=ALCATEL,dc=FR > dn: cn=Parveen207,dc=ALCATEL,dc=FR > dn: cn=Parveen208,dc=ALCATEL,dc=FR > dn: cn=Parveen209,dc=ALCATEL,dc=FR > dn: cn=Parveen30,dc=ALCATEL,dc=FR > dn: cn=Parveen300,dc=ALCATEL,dc=FR > dn: cn=Parveen301,dc=ALCATEL,dc=FR > dn: cn=Parveen302,dc=ALCATEL,dc=FR > dn: cn=Parveen303,dc=ALCATEL,dc=FR > dn: cn=Parveen304,dc=ALCATEL,dc=FR > dn: cn=Parveen305,dc=ALCATEL,dc=FR > dn: cn=Parveen306,dc=ALCATEL,dc=FR > dn: cn=Parveen307,dc=ALCATEL,dc=FR > dn: cn=Parveen308,dc=ALCATEL,dc=FR > dn: cn=Parveen309,dc=ALCATEL,dc=FR > > > I want to delete entries with 301,302,303.... > So how can I give this, except putting all these in a LDIF file. > > or how can below work > > cn=Parveen30*,dc=ALCATEL,dc=FR > >

1 0

Problem with ppolicy
by Il Neofita 09 Jun '16

09 Jun '16

Hi there, I have a problem with ppolicy and got stuck finding a solution. I am using openldap-2.4.40-9 Oracle 7.2, the lock seems to be ignored. This what I have configured dn: cn=module,cn=config objectClass: olcModuleList cn: module olcModuleLoad: ppolicy.la olcModulePath: /usr/lib64/openldap dn: olcOverlay=ppolicy,olcDatabase={2}hdb,cn=config objectClass: olcPPolicyConfig olcOverlay: ppolicy olcPPolicyDefault: cn=ppolicy,ou=policies,dc=ddd olcPPolicyUseLockout: TRUE olcPPolicyHashCleartext: TRUE dn: cn=ppolicy,ou=policies,dc=ddd objectClass: top objectClass: device objectClass: pwdPolicyChecker objectClass: pwdPolicy cn: ppolicy pwdAttribute: userPassword pwdAllowUserChange: TRUE pwdInHistory: 4 pwdMinLength: 8 pwdMaxFailure: 6 pwdFailureCountInterval: 1800 pwdCheckQuality: 1 pwdMustChange: TRUE pwdGraceAuthNLimit: 3 pwdMaxAge: 3600 pwdExpireWarning: 31536000 pwdLockoutDuration: 900 pwdLockout: TRUE pwdCheckModule: pwcheck.la and I tried to add the attribute pwdReset: TRUE and pwdAccountLockedTime: 20110214195250Z in a specific user, however, the user is not locked and I do not even been ask to change the password Here is what happens ldapwhoami -e ppolicy -x -D "uid=ldap001,ou=People,dc=ddd" -W -h localhost Enter LDAP Password: dn:uid=ldap001,ou=People,dc=ddd Any idea what I should check

1 0

ldapdelete: Use wild card to delete multiple enetries
by Parveen Verma 08 Jun '16

08 Jun '16

Hello All, Is there a Way to delete multiple LDAP entries by using some wildcards or regular expression. Like I have below entries. dn: cn=Parveen20,dc=ALCATEL,dc=FR dn: cn=Parveen200,dc=ALCATEL,dc=FR dn: cn=Parveen201,dc=ALCATEL,dc=FR dn: cn=Parveen202,dc=ALCATEL,dc=FR dn: cn=Parveen203,dc=ALCATEL,dc=FR dn: cn=Parveen204,dc=ALCATEL,dc=FR dn: cn=Parveen205,dc=ALCATEL,dc=FR dn: cn=Parveen206,dc=ALCATEL,dc=FR dn: cn=Parveen207,dc=ALCATEL,dc=FR dn: cn=Parveen208,dc=ALCATEL,dc=FR dn: cn=Parveen209,dc=ALCATEL,dc=FR dn: cn=Parveen30,dc=ALCATEL,dc=FR dn: cn=Parveen300,dc=ALCATEL,dc=FR dn: cn=Parveen301,dc=ALCATEL,dc=FR dn: cn=Parveen302,dc=ALCATEL,dc=FR dn: cn=Parveen303,dc=ALCATEL,dc=FR dn: cn=Parveen304,dc=ALCATEL,dc=FR dn: cn=Parveen305,dc=ALCATEL,dc=FR dn: cn=Parveen306,dc=ALCATEL,dc=FR dn: cn=Parveen307,dc=ALCATEL,dc=FR dn: cn=Parveen308,dc=ALCATEL,dc=FR dn: cn=Parveen309,dc=ALCATEL,dc=FR I want to delete entries with 301,302,303.... So how can I give this, except putting all these in a LDIF file. or how can below work cn=Parveen30*,dc=ALCATEL,dc=FR

2 1

OpenLDAP 2.4.44 and Berkley DB version
by Parveen Verma 06 Jun '16

06 Jun '16

Hello All, We are using OpenLDAP 2.4.44 in our project, but facing a dilemma over the Berkley DB version. Which version we should use or it is not required at all now as we have LMDB. Moreover Oracle has removed some links for downloading the older Berkley DB version. OS : RHEL cat /etc/redhat\-release Red Hat Enterprise Linux Server release 6.5 (Santiago) I referred the Online guide, but was not able to conclude. 4.2.4. Database Software OpenLDAP's *slapd*(8) MDB primary database backend uses the LMDB software included with the OpenLDAP source. There is no need to download any additional software to have *MDB* support. OpenLDAP's *slapd*(8) BDB and HDB deprecated database backends require Oracle Corporation <http://www.oracle.com/>'s Berkeley DB. If not available at configure time, you will not be able to build *slapd*(8) with these deprecated database backends. Your operating system may provide a supported version of Berkeley DB in the base system or as an optional software component. If not, you'll have to obtain and install it yourself. Berkeley DB is available from Oracle Corporation <http://www.oracle.com/>'s Berkeley DB download page if required. There are several versions available from Oracle Corporation <http://www.oracle.com/>. Berkeley DB version 6.0.20 and later uses a software license that is incompatible with LDAP technology and should not be used with OpenLDAP. ------------------------------ *Note: *Please see Recommended OpenLDAP Software Dependency Versions <http://www.openldap.org/doc/admin24/guide.html#Recommended OpenLDAP Software Dependency Versions> for more information.

2 4

Error message nonpresent_callback in replication
by Michael Wuensche 06 Jun '16

06 Jun '16

Hello, We recently upgraded our Openldap installation to a current version. Our installation consists of 2 servers with several databases, of which a few are replicated in mirror mode. Together with the upgrade process we reconfigured which data gets replicated and which not and modified the databases accordingly. Now, when we synchronize data between the nodes the intial replication works just fine. But replication of changes afterwards fails and we get these error messages: Jun 3 16:36:09 wam01 slapd[19027]: [ID 795510 local4.debug] do_syncrep2: rid=025 LDAP_RES_INTERMEDIATE - SYNC_ID_SET Jun 3 16:36:09 wam01 last message repeated 29 times Jun 3 16:36:09 wam01 slapd[19027]: [ID 365351 local4.debug] do_syncrep2: rid=025 LDAP_RES_SEARCH_RESULT Jun 3 16:36:09 wam01 slapd[19027]: [ID 385048 local4.debug] do_syncrep2: rid=025 cookie=rid=025,csn=20160603143607.847747Z#000000#000#000000 Jun 3 16:36:10 wam01 slapd[19027]: [ID 338579 local4.debug] nonpresent_callback: rid=025 present UUID 6c782570-d2b6-1033-84c4-3d93063e27ca, dn ou=Directories,o=lpfroot Any idea what may have gone wrong here? The suffix o=lpfroot is in a non-replicated database, the glued sub-suffix ou=Directories,o=lpfroot is in a replicated database. The databases are still bdb as before. Best regards, Michael -- Michael Wünsche Principal Consultant Identity & Access Management Tel.: +49 (6151) 868-7251 Fax: +49 (6151) 868-7264 Michael.Wuensche(a)devoteam.com Devoteam GmbH, Gutenbergstraße 10, 64331 Weiterstadt, Germany URL: www.devoteam.de ------------------------------------------------------------------------------------------------------------ Geschäftsführer: Jürgen Hatzipantelis; Sitz der Gesellschaft: Weiterstadt; Amtsgericht Darmstadt HRB 6450; Umsatzsteuer-ID Nr.: DE 172 993 071

2 1

Issues with pcache overlay purging entries
by Andrew Widdersheim 03 Jun '16

03 Jun '16

Hello, Seeing strange issues when using the pcache overlay as a caching method on some servers. More specifically, not seeing results getting purged from the cache even though the TTL has expired. I have been able to reproduce on the following: Ubuntu 16.04 running slapd 2.4.42+dfsg-2ubuntu3.1 Ubuntu 14.04 running slapd 2.4.31-1+nmu2ubuntu8 Here are the steps that I am taking in order to reproduce the issue. First, I'm running a query similar to this: ldapsearch -x -LLL -h localhost -b 'dc=foo,dc=com' '(isphoneoperator=TRUE)' uid | grep "uid:" | grep someuser When I set the "isphoneoperator" attribute to TRUE the user shows up and everything is great. If I then go and set the attribute to FALSE and restart slapd *before* the TTL of the query expires on it's own naturally then that user remains in the cache forever and the only way to correct the issue is to stop slapd, delete all the caching databases and start slapd again. There are a few things I've noticed in the logs that I don't quite understand. When starting slapd for the first time with a fresh set of cache databases the logs look like the ones below and will repeat every time the TTL expires: Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=1 Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=2 Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=3 Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=4 Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=5 Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=6 Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=7 Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=8 Jun 1 20:34:31 vagrant slapd[10042]: ENTRY ADDED/MERGED, CACHED ENTRIES=9 However, when I restart slapd the logs change to look like this: Jun 1 20:44:55 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0] Jun 1 20:45:25 vagrant slapd[11817]: ENTRY ADDED/MERGED, CACHED ENTRIES=0 Jun 1 20:45:25 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0] Jun 1 20:45:55 vagrant slapd[11817]: ENTRY ADDED/MERGED, CACHED ENTRIES=0 Jun 1 20:45:55 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0] Jun 1 20:46:26 vagrant slapd[11817]: ENTRY ADDED/MERGED, CACHED ENTRIES=0 Jun 1 20:46:26 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0] Jun 1 20:46:56 vagrant slapd[11817]: ENTRY ADDED/MERGED, CACHED ENTRIES=0 Jun 1 20:46:56 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0] Jun 1 20:47:26 vagrant slapd[11817]: ENTRY ADDED/MERGED, CACHED ENTRIES=0 Jun 1 20:47:26 vagrant slapd[11817]: message repeated 32 times: [ ENTRY ADDED/MERGED, CACHED ENTRIES=0] I find this to be pretty odd and I'm not sure I quite understand the reason for this behavior. Also, when I run slapd in the foreground with (-d -1 for example) I notice some differences there as well. For example, with a fresh set of cache databases I'll see messages like: 57503573 ==> hdb_delete: uid=someuser,ou=users,dc=foo,dc=com However, when I restart slapd I never see these again. I have noticed that if I set pcachePersist to TRUE both my issue above where users are never purged and this logging inconsistency go away and everything works great. I guess I don't really understand what the pcachePersist parameter does. I've read the man pages for slapo-pcache(5) and I also found the following mailing list post and it is all a bit confusing to me: http://www.openldap.org/cgi-bin/wilma_hiliter/openldap-technical/201007/msg… I've looked for fixed bugs or any other posts that might indicate what is going on here but haven't turned up anything relevant that might be affecting the versions of slapd I'm running. I guess I'm wondering a few things: 1. Have a stumbled upon a bug? 2. Is this logging inconsistency normal? 3. What does pcachePersist do? 4. Is there a way to forcefully clear the cache other than deleting the database files or do you have to wait for the TTL to expire? Thanks, Andrew

1 0

Virtual list view problem
by Venish Khant 31 May '16

31 May '16

Hi all I am using cpan Net::LDAP module to access LDAP entries. I want to search LDAP entries using Net::LDAP search method. When I do search, I want some limited number of entries from search result, for this(searching) process I am using Net::LDAP::Control::VLV module. But I get error on VLV response control. Please, any one have idea about this error. * Error:* Died at vlv.pl line 50, This is my example. I changed the font style of line 50 #!/usr/bin/perl -w use Net::LDAP; use Net::LDAP::Control::VLV; use Net::LDAP::Constant qw( LDAP_CONTROL_VLVRESPONSE ); use Net::LDAP::Control::Sort; sub procentry { my ( $mesg, $entry) = @_; # Return if there is no entry to process if ( !defined($entry) ) { return; } print "dn: " . $entry->dn() . "\n"; @attrs = $entry->attributes(); foreach $attr (@attrs) { #printf("\t%s: %s\n", $attr, $entry->get_value($attr)); $attrvalue = $entry->get_value($attr,asref=>1); #print $attr.":". $entry->get_value($attr)."\n"; foreach $value(@$attrvalue) { print "$attr: $value\n"; } } $mesg->pop_entry; print "\n"; } $ldap = Net::LDAP->new( "localhost" ); # Get the first 20 entries $vlv = Net::LDAP::Control::VLV->new( before => 0, # No entries from before target entry after => 19, # 19 entries after target entry content => 0, # List size unknown offset => 1, # Target entry is the first ); my $sort = Net::LDAP::Control::Sort->new( order => 'cn' ); @args = ( base => "dc=example,dc=co,dc=in", scope => "subtree", filter => "(objectClass=inetOrgPerson)", callback => \&procentry, # Call this sub for each entry control => [ $sort, $vlv ], ); $mesg = $ldap->search( @args ); # Get VLV response control *($resp) = $mesg->control( LDAP_CONTROL_VLVRESPONSE ) or die;* $vlv->response( $resp ); # Set the control to get the last 20 entries $vlv->end; $mesg = $ldap->search( @args ); # Get VLV response control ($resp) = $mesg->control( LDAP_CONTROL_VLVRESPONSE ) or die; $vlv->response( $resp ); # Now get the previous page $vlv->scroll_page( -1 ); $mesg = $ldap->search( @args ); # Get VLV response control ($resp) = $mes # Now page with first entry starting with "B" in the middle $vlv->before(9); # Change page to show 9 before $vlv->after(10); # Change page to show 10 after $vlv->assert("B"); # assert "B" $mesg = $ldap->search( @args );g->control( LDAP_CONTROL_VLVRESPONSE ) or die; $vlv->response( $resp ); -- Venish Khant www.deeproot.co.in

4 11

Re: Documentation
by Daniel Howard 30 May '16

30 May '16

My recent experience is OpenLDAP on Ubuntu. I thought I would go with OpenLDAP's guide because they should know better, but the quick start was for older versions or something and hadn't been updated. I like documentation systems that allow for user feedback, comments, or patches via github. If you want community improvements then maybe low hanging fruit is a link in the doc template on how to interact with the documentation.... If the current docs platform has no mechanism for feedback and comments on a specific document, the source code/ build process is obscure ... Maybe consider porting even a "fork " to github and solicit community feedback (you can file issues in a familiar mechanism, and take in patches from dudes like me...) For what it's worth, the Ubuntu OpenLDAP quick start guide and only a few minor hiccups, and I would recommend that resource for anyone looking to implement or update a quickstart guide. The Faq-o-Matic ... Man a modern stack overflow site might be nice but a question in the back of my mind is whether the advice is still useful. I don't know if the FoM does this but if the answers showed dates then I could evaluate an answer that's say 2-4 years old with greater confidence than say something 12 years old ... One of the challenges of a venerable, old and mature project like OpenLDAP. I guess what I'm trying to say, from my mobile phone, is if the documentation were more "agile" in engaging the community for identifying issues and taking in corrections, we may all be happier. I would be open to volunteering to port docs... Thanks, -Danny -- Sent from my Western Electric Model 500 On Apr 29, 2016 5:01 PM, "Gavin Henry" <ghenry(a)openldap.org> wrote: Hi all, List what you love about our docs: List what you hate: List what you'd love to see: List what you can help with: Thanks. -- Kind Regards, Gavin Henry.

6 9

slapadd hang in bdb_tool_entry_close / ldap_pvt_thread_cond_wait
by Bannister, Mark 27 May '16

27 May '16

Hi, Can anyone shed light on why my slapadd process on Solaris 11 was hanging (slapd 2.4.30)? I was piping 'gunzip -c' through to slapadd (from an overnight cron job). This usually works fine. I'm guessing perhaps the gunzip process encountered an error, maybe an EPIPE was sent, but the stack trace many days later from slapadd doesn't look right. Only 2 threads, and both of them in ldap_pvt_thread_cond_wait(), but no other threads so what are they waiting for exactly? # pstack /var/tmp/cores/core.slapd.0.16189.1463492837 core '/var/tmp/cores/core.slapd.0.16189.1463492837' of 16189: /usr/sbin/slapadd -q -f /etc/openldap/slapd.conf ------------ lwp# 1 / thread# 1 --------------- 07ff7af9 lwp_park (0, 0, 0) 07ff1193 cond_wait_queue (82ed048, 82ed020, 0, 7ff16b0) + 63 07ff1728 __cond_wait (82ed048, 82ed020, f5a685c8, 7ff1771) + 89 07ff177f cond_wait (82ed048, 82ed020, f5a685f8, 7ff17b7) + 27 07ff17cc pthread_cond_wait (82ed048, 82ed020, f5a68618, 7e56137) + 24 07e5614c ldap_pvt_thread_cond_wait (82ed048, 82ed020, f5a68648, 81b7e29) + 24 081b7e7a bdb_tool_entry_close (8e9d398, ffffffff, f5a68698, 82ecc38) + 62 081a38ec slapadd (4, f5a68bbc, f5a68bbc, 80dccb1) + cb8 080dcd2c main (4, f5a68bbc, f5a68bd0, ef60f968) + ac 080dc86d _start (4, f5a68c8e, f5a68ca0, f5a68ca3, f5a68ca6, 0) + 7d ------------ lwp# 2 / thread# 2 --------------- 07ff7af9 lwp_park (0, 0, 0) 07ff1193 cond_wait_queue (8e6b0e4, 8e6b0cc, 0, 7ff16b0) + 63 07ff1728 __cond_wait (8e6b0e4, 8e6b0cc, d7dffdb8, 7ff1771) + 89 07ff177f cond_wait (8e6b0e4, 8e6b0cc, 8e6b0c8, 7ff17b7) + 27 07ff17cc pthread_cond_wait (8e6b0e4, 8e6b0cc, 0, 7e56137) + 24 07e5614c ldap_pvt_thread_cond_wait (8e6b0e4, 8e6b0cc, d7dfffc8, 7e54bfc) + 24 07e54d66 ldap_int_thread_pool_wrapper (8e6b0c8, 803a000, d7dfffe8, 7ff7769) + 242 07ff77bc _thrp_setup (ee1e0240) + 9d 07ff7aa0 _lwp_start (ee1e0240, 0, 0, 0, 0, 0) Thanks, Mark. ________________________________ NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.

2 7

Improved handling for large number of databases / Access newly opened database from another transaction
by Jürg Bircher 27 May '16

27 May '16

Hello Improved handling for large number of databases =============================================== There is a increased performance penalty the more databases are created within the same environment. I was looking for a way the improve that by keeping the simplicity of tracking databases within a list with direct access by index (MDB_dbi). mdb_dbi_open() is however not improved with the assumption that the database handle (dbi) is cached in the application. So mdb_dbi_open() should happen only once for each database during the life time of an application. One issue is that mdb_txn_begin() (for read-only transactions) calloc the sizeof(MDB_txn) + me_maxdbs * sizeof(MDB_db + 1). The plus 1 for the dbflags. However it is sufficient only to malloc that size and clear the sizeof(MDB_txn) memset(txn, 0, sizeof(MDB_txn) After that the data beyond the MDB_txn is not initialized which is ok for the moment. The next improvement happens in mdb_txn_renew0() where the dbflags are only set to DB_UNUSED (a new flag) for each database currently opened in the environment. memset(txn->mt_dbflags, DB_UNUSED, txn->mt_numdbs); The former code used to to loop through each database to calculate the dbflags. This is still done but lazily for each accessed database with the assumption that a read only transaction rarely uses all databases of the environment. The lazy initialization of the dbflag happens in the macro TXN_DBI_EXIST which is always used when a database handle (dbi) is passed to an function. The flags are updated in mdb_setup_db_info() once a database is access which is marked as unused (DB_UNUSED). static int mdb_setup_db_info(MDB_txn *txn, MDB_dbi dbi) { /* Setup db info */ uint16_t x = txn->mt_env->me_dbflags[dbi]; txn->mt_dbs[dbi].md_flags = x & PERSISTENT_FLAGS; txn->mt_dbflags[dbi] = (x & MDB_VALID) ? DB_VALID|DB_USRVALID|DB_STALE : 0; return (txn->mt_dbflags[dbi] & validity); } /** Check \b txn and \b dbi arguments to a function and initialize db info if needed */ #define TXN_DBI_EXIST(txn, dbi, validity) \ ((txn) && (dbi)<(txn)->mt_numdbs && (((txn)->mt_dbflags[dbi] & (validity)) || (((txn)->mt_dbflags[dbi] & DB_UNUSED) && mdb_setup_db_info((txn), (dbi), (validity))))) The next improvement is done in any function which needs to loop through the databases for example in mdb_cursors_close(). Again the more databases in the environment the longer the execution time. It should be best if looping only through dbflags and searching for those databases which are used (!DB_UNUSED). This could be done byte wise or more efficient in 8/4 byte steps comparing with an extended mask DB_UNUSED_LONG instead of DB_UNUSED. So we can skip 8 or 4 (32 bit) unused databases in one step (still with the assumption that a transaction rarely uses all databases of the environment). So the loop looks as follows always starting at the lower index to avoid alignment issues with ARM prior v6. #define DB_UNUSED 0x20 /**< DB not used in this txn */ #ifdef MDB_VL32 #define DB_UNUSED_LONG 0x20202020 /* DB_UNUSED long mask for fast tracking */ #else #define DB_UNUSED_LONG 0x2020202020202020 /* DB_UNUSED long mask for fast tracking */ #endif #ifdef MDB_VL32 #define MDB_WORD unsigned int #else #define MDB_WORD unsigned long long #endif MDB_dbi n = src->mt_numdbs; MDB_dbi i = 0; while (1) { unsigned int upper = i + sizeof(MDB_WORD); if (upper < n) { // skip unused if ((*(MDB_WORD *)(tdbflags + i)) == DB_UNUSED_LONG) { i = upper; continue; } } else { upper = n; } for (; i < upper; i++) { // any other filter criteria appropriate to the function .... } if (i >= n) { break; } } Access newly opened database from another transaction ======================================================= A transaction tracks newly opened databases and if the transaction is committed the newly opened databases are propagated to the list of open databases of the environment. However if the read only transaction is aborted the databases are not propagated. If database handles (MDB_dbi) are cached in the application to avoid calling mdb_dbi_open() there might be the situation of two threads running a read only transaction concurrently. First threads opens the database and commits the read-only transaction. The database is added to the list of open databases in the environment. The returned dbi is globally cached in the application. The second thread also wants to access the same database and finds the database handle in the global application cache. The database handle however is not valid as the transaction only uses a snapshot of open databases. So the second thread gets an EINVAL when using that database handle. This should not happen as the database is open and added to the environment. The following should fix this issue by updating the mt_numdbs and marking the delta with DB_UNUSED static int mdb_update_db_info(MDB_txn *txn, MDB_dbi dbi) { /* propagate newly opened db from env to current txn. Mark them as unused */ if (txn->mt_numdbs < txn->mt_env->me_numdbs) { memset(txn->mt_dbflags + txn->mt_numdbs, DB_UNUSED, txn->mt_env->me_numdbs - txn->mt_numdbs); } txn->mt_numdbs = txn->mt_env->me_numdbs; return dbi < txn->mt_numdbs; } /** Check \b txn and \b dbi arguments to a function and initialize db info if needed */ #define TXN_DBI_EXIST(txn, dbi, validity) \ ((txn) && ((dbi)<(txn)->mt_numdbs || mdb_update_db_info((txn), (dbi))) && (((txn)->mt_dbflags[dbi] & (validity)) || (((txn)->mt_dbflags[dbi] & DB_UNUSED) && mdb_setup_db_info((txn), (dbi), (validity))))) If interested let me know how to contribute. Hope it is useful! Regards Juerg

2 5

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical