May 2016 - openldap-technical

by Fr3ddie

Hello to the list, I'm trying to configure the slapd-meta OpenLDAP backend on an online cn=config configuration with no luck. Slapd version is 2.4.39 (the maximum I can achieve on the target machines building from vanilla source). The documentation is clear but too concise for me so I will try to explain what I'm trying to do to see if there is anybody that can help me. Currently I have 3 slapd servers that share a common root for the DIT, i.e.: dc=loc1,dc=root dc=loc2,dc=root dc=loc3,dc=root What I would like to achieve is to obtain a fourth server that contains the previous trees, along with its own tree, i.e. a server that contains: dc=loc0,dc=root (locally hosted data) dc=loc1,dc=root (coming from the first server, chasing referrals) dc=loc2,dc=root (coming from the second server, chasing referrals) dc=loc3,dc=root (coming from the third server, chasing referrals) this way, all the clients connecting to this server will be able to retrieve data also from the other three remote servers. As far as I understood, I only need to configure the "loc0" server to access the other three servers and get the data to serve to clients. I have already configured the fourth server with its local DIT and this is the configuration: # cat 'cn=config.ldif' dn: cn=config objectClass: olcGlobal cn: config olcArgsFile: /var/run/slapd/slapd.args olcPidFile: /var/run/slapd/slapd.pid structuralObjectClass: olcGlobal creatorsName: cn=config olcServerID: 1 olcThreads: 32 olcToolThreads: 8 olcRequires: LDAPv3 olcConnMaxPendingAuth: 100 olcTLSCACertificateFile: /etc/ssl/certs/my_ca_cert.pem olcTLSCertificateFile: /etc/ssl/certs/this-host_x509_cert.pem olcTLSCertificateKeyFile: /etc/ssl/private/this-host_x509_key.key olcTLSVerifyClient: try olcTimeLimit: 600 olcLogLevel: stats2 sync [...] # cat 'cn=module{0}.ldif' dn: cn=module{0} objectClass: olcModuleList cn: module{0} olcModulePath: /usr/lib/ldap olcModuleLoad: {0}back_hdb olcModuleLoad: {1}syncprov olcModuleLoad: {2}accesslog structuralObjectClass: olcModuleList [...] Schema files are the following: cn={0}core.ldif cn={1}cosine.ldif cn={2}nis.ldif cn={3}inetorgperson.ldif cn={4}dyngroup.ldif cn={5}kerberos.ldif # cat 'olcDatabase={1}hdb.ldif' dn: olcDatabase={1}hdb objectClass: olcDatabaseConfig objectClass: olcHdbConfig olcDatabase: {1}hdb olcDbDirectory: /var/lib/ldap olcSuffix: dc=loc0,dc=root olcAccess: {0}to attrs=userPassword,shadowLastChange,krbPrincipalKey by dn="cn =admin,dc=loc0,dc=root" write by anonymous auth by self write by * none olcAccess: {1}to dn.base="" by * read olcAccess: {2}to * by dn="cn=admin,dc=loc0,dc=root" write by * read olcLastMod: TRUE olcRootDN: cn=admin,dc=loc0,dc=root olcRootPW:: xxxxxxxxxxxxxxxxxxxx olcDbCacheSize: 10000 olcDbCheckpoint: 512 10 olcDbConfig: {0}set_cachesize 0 524288000 1 olcDbConfig: {1}set_lk_max_objects 1500 olcDbConfig: {2}set_lk_max_locks 1500 olcDbConfig: {3}set_lk_max_lockers 1500 olcDbConfig: {4}set_flags DB_LOG_AUTOREMOVE olcDbIDLcacheSize: 30000 olcDbIndex: default pres,eq [...] structuralObjectClass: olcHdbConfig olcSyncrepl: {0}rid=0 provider=ldap://second-host.loc0.root bindmethod=s imple binddn="cn=admin,dc=loc0,dc=root" credentials=xxxxxx searchbase="dc=loc0,dc=root" logbase="cn=accesslog" logfilter="(&(objectClass=auditWriteObj ect)(reqResult=0))" schemachecking=on type=refreshAndPersist retry="60 +" syn cdata=accesslog starttls=yes olcMirrorMode: TRUE [...] On top of this DB I have the "syncprov" and the "accesslog" overlays configured (these are two servers in "MirrorMode", configured following the OpenLDAP admin documentation). I believe this DB is the ones containing the actual "loc0" DIT data... Then I have the accesslog DB for the replica (with the syncprov overlay on top): # cat 'olcDatabase={2}hdb.ldif' dn: olcDatabase={2}hdb objectClass: olcDatabaseConfig objectClass: olcHdbConfig olcDatabase: {2}hdb olcDbDirectory: /var/lib/ldap/accesslog olcSuffix: cn=accesslog olcRootDN: cn=admin,dc=loc0,dc=root olcDbConfig: {0}set_cachesize 0 524288000 1 olcDbConfig: {1}set_lk_max_objects 1500 olcDbConfig: {2}set_lk_max_locks 1500 olcDbConfig: {3}set_lk_max_lockers 1500 olcDbConfig: {4}set_flags DB_LOG_AUTOREMOVE olcDbIndex: default eq olcDbIndex: entryCSN,objectClass,reqEnd,reqResult,reqStart [...] On top of this environment I start loading the needed modules with this LDIF file: version: 1 dn: cn=module{0},cn=config changetype: modify add: olcModuleLoad olcModuleLoad: back_ldap - add: olcModuleLoad olcModuleLoad: back_meta - add: olcModuleLoad olcModuleLoad: rwm and it seems I'm able to load the new modules without errors into the configuration, thus I obtain: # cat 'cn=module{0}.ldif' dn: cn=module{0} structuralObjectClass: olcModuleList objectClass: olcModuleList cn: module{0} olcModulePath: /usr/lib/ldap olcModuleLoad: {0}back_hdb olcModuleLoad: {1}syncprov olcModuleLoad: {2}accesslog olcModuleLoad: {3}back_ldap olcModuleLoad: {4}back_meta olcModuleLoad: {5}rwm [...] Now I try to load the slapd-meta directives into a new database using this LDIF: version: 1 dn: olcDatabase={3}meta,cn=config objectClass: olcDatabaseConfig objectClass: olcMetaConfig olcDatabase: {3}meta olcSuffix: dc=root olcDbURI: "ldap://server-loc1.loc1.root/dc=loc1,dc=root" olcDbIdAssertBind: bindmethod=simple binddn="cn=admin,dc=loc1,dc=root" credentials=xxxxxx starttls=yes tls_reqcert=demand olcDbURI: "ldap://server-loc2.loc2.root/dc=loc2,dc=root" olcDbIdAssertBind: bindmethod=simple binddn="cn=admin,dc=loc2,dc=root" credentials=xxxxxx starttls=yes tls_reqcert=demand olcDbURI: "ldap://server-loc3.loc3.root/dc=loc3,dc=root" olcDbIdAssertBind: bindmethod=simple binddn="cn=admin,dc=loc3,dc=root" credentials=xxxxxx starttls=yes tls_reqcert=demand but I obtain an error that sticks me trying various combinations without success: # ldapadd -Y EXTERNAL -H ldapi:/// -f slapd-META-DB-CREATION.ldif SASL/EXTERNAL authentication started SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth SASL SSF: 0 adding new entry "olcDatabase={3}meta,cn=config" ldap_add: Object class violation (65) additional info: attribute 'olcDbURI' not allowed and: # tail /var/log/openldap/slapd.log Nov 9 19:47:17 server01 slapd[32392]: conn=1025 op=2 ENTRY dn="dc=loc0,dc=root" Nov 9 19:47:29 server01 slapd[32392]: conn=1052 op=2 INTERM oid=1.3.6.1.4.1.4203.1.9.1.4 Nov 9 19:49:47 server01 slapd[32392]: conn=1327 op=2 ENTRY dn="dc=loc0,dc=root" Nov 9 19:52:17 server01 slapd[32392]: conn=1628 op=2 ENTRY dn="dc=loc0,dc=root" Nov 9 19:54:46 server01 slapd[32392]: conn=1929 op=2 ENTRY dn="dc=loc0,dc=root" Nov 9 19:57:07 server01 slapd[32392]: Entry (olcDatabase={3}meta,cn=config), attribute 'olcDbURI' not allowed Into the slapd-meta documentation the "URI" directive is mentioned but the "DbURI" seems to raise a "better error", in fact if I try to modify the above LDIF file using "URI" I obtain: SASL/EXTERNAL authentication started SASL username: gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth SASL SSF: 0 adding new entry "olcDatabase={3}meta,cn=config" ldap_add: Undefined attribute type (17) additional info: olcUri: attribute type undefined Moreover, it is not stated into the slapd-meta docs that the slapd-ldap backend is needed by slapd-meta but, anyway, I think its needed because if I try to load the slapd-meta alone it raises an error (I don't remember exactly which one). At this point I'm stuck to this error and I wasn't able to find any hint on the web to solve this :( The examples I was able to find were related with the static slapd.conf configuration, I counldn't find any "full" configuration example using the cn=config. I'm wondering if I should create a "cn=root" actual DB first and then link the sub-DITs to it, or, maybe, add some other overlay... I really can't understand how it should work :( Can please anybody help me? Thank you very much

6 years, 6 months

3
9
0 / 0

Documentation Feedback

by Tom Jay

Hello, I would like to voice my frustration with the quality of the openLDAP documentation. I am compiling openLDAP from source on Debian 7, and have spent about 2-3 continuous days getting to the point where I can add an LDAP user with a UID. I have been close to giving up with the software, but need it for the LDAP functionality, and as very few viable alternatives exist, have been forced to continue with the installation. I have however almost lost confidence in the product, and am concerned that if there are any problems with it in the future, or I want to enable another feature, I will be almost helpless in getting it to work. The main problem is the Quick-Start Guide. This is anything but quick, and forces the user to consult the less-than-succinct Admin guide. The two together are inconsistent, difficult to follow and do a poor job of explaining what each feature does. The accessibility of information is less than optimal, which means that the user has to look elsewhere, consuming even more time. Unfortunately, there is not much relevant information on the Internet, forcing the user to get stuck in an almost endless loop of checking documentation, testing, reading manuals, and searching on the Internet, in order to get some kind of idea how the software works and what needs to be done to get it working. I would offer to contribute to the documentation, but due to its lack of usefulness, do not have an understanding of the basic concepts myself. The best I would be able to do is describe my experience and provide the steps that I followed to get a basic installation working. Hopefully someone can volunteer the time to test the documentation, in the same way a new user would! Tom

6 years, 8 months

8
9
0 / 0

ldap user login attempt kills slapd service

by Real, Elizabeth (392K)

Openldap gurus: Here is my setup, LDAPSERVERS: I have two ldap servers running RHEL7.2 and openldap 2.4.40. Both servers are configured with multi-master replication. Ldaps is enabled and a ppolicy applied. LDAPCLIENT: My ldap client is running RHEL7.2 as well, sssd 1.13.0, and openldap client 2.4.40. I have been troubleshooting this problem for a while and can’t figure out why everytime I try to login to an ldap client with a test user account the slapd service on only one of my ldap servers gets killed. Both getent and ldapsearch return the expected information when ran on the ldap client: ldapclient ~]# getent passwd realtest realtest:*:1004:312:Liz RealTest:/home/real:/bin/tcsh ldapclient ~]# ldapsearch -x -s sub -b 'ou=People,dc=cluster,dc=sec312' '(uid=realtest)' # extended LDIF # # LDAPv3 # base <ou=People,dc=cluster,dc=sec312> with scope subtree # filter: (uid=realtest) # requesting: ALL # # realtest, People, cluster.sec312 dn: uid=realtest,ou=People,dc=cluster,dc=sec312 gidNumber: 312 objectClass: account objectClass: top objectClass: posixAccount objectClass: shadowAccount uid: realtest loginShell: /bin/tcsh homeDirectory: /home/real cn: Liz RealTest uidNumber: 1004 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 LDAP SERVER /VAR/LOG/SECURE: serverA journal: Suppressed 19192 messages from /system.slice/slapd.service serverA journal: Suppressed 8449 messages from /system.slice/slapd.service serverA systemd: slapd.service: main process exited, code=killed, status=6/ABRT serverA systemd: Unit slapd.service entered failed state. serverA systemd: slapd.service failed. LDAP CLIENT /VAR/LOG/SECURE: ldapclient sshd[122938]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=node12.cluster.sec312 user=realtest ldapclient sshd[122938]: pam_sss(sshd:auth): received for user realtest: 7 (Authentication failure) ldapclient sshd[122938]: pam_ldap(sshd:auth): Authentication failure; user=realtest ldapclient sshd[122936]: error: PAM: Authentication failure for realtest from node12.cluster.sec312 ATTEMPT TO SSH AS TEST USER TO LDAP CLIENT: % ssh -v realtest@ldapclient OpenSSH_6.6.1, OpenSSL 1.0.1e-fips 11 Feb 2013 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 60: Applying options for * debug1: Connecting to ldapclient [] port 22. debug1: Connection established. debug1: could not open key file '/etc/ssh/ssh_host_key': Permission denied debug1: could not open key file '/etc/ssh/ssh_host_dsa_key': Permission denied debug1: could not open key file '/etc/ssh/ssh_host_ecdsa_key': Permission denied debug1: could not open key file '/etc/ssh/ssh_host_rsa_key': Permission denied debug1: could not open key file '/etc/ssh/ssh_host_ed25519_key': Permission denied debug1: could not open key file '/etc/ssh/ssh_host_dsa_key': Permission denied debug1: could not open key file '/etc/ssh/ssh_host_ecdsa_key': Permission denied debug1: could not open key file '/etc/ssh/ssh_host_rsa_key': Permission denied debug1: could not open key file '/etc/ssh/ssh_host_ed25519_key': Permission denied debug1: identity file /home/real/.ssh/id_rsa type -1 debug1: identity file /home/real/.ssh/id_rsa-cert type -1 debug1: identity file /home/real/.ssh/id_dsa type -1 debug1: identity file /home/real/.ssh/id_dsa-cert type -1 debug1: identity file /home/real/.ssh/id_ecdsa type -1 debug1: identity file /home/real/.ssh/id_ecdsa-cert type -1 debug1: identity file /home/real/.ssh/id_ed25519 type -1 debug1: identity file /home/real/.ssh/id_ed25519-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_6.6.1 debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1 debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000 debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: server->client aes128-ctr hmac-md5-etm(a)openssh.com none debug1: kex: client->server aes128-ctr hmac-md5-etm(a)openssh.com none debug1: kex: curve25519-sha256(a)libssh.org need=16 dh_need=16 debug1: kex: curve25519-sha256(a)libssh.org need=16 dh_need=16 debug1: sending SSH2_MSG_KEX_ECDH_INIT debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ECDSA 14:c5:c2:60:29:ce:99:aa:67:41:a6:6a:11:2c:ca:86 debug1: Host 'ldapclient' is known and matches the ECDSA host key. debug1: Found key in /home/real/.ssh/known_hosts:22 debug1: ssh_ecdsa_verify: signature correct debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: Roaming not allowed by server debug1: SSH2_MSG_SERVICE_REQUEST sent debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive,hostbased debug1: Next authentication method: gssapi-keyex debug1: No valid Key exchange context debug1: Next authentication method: gssapi-with-mic debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available debug1: Unspecified GSS failure. Minor code may provide more information debug1: Unspecified GSS failure. Minor code may provide more information No Kerberos credentials available debug1: Next authentication method: hostbased debug1: No more client hostkeys for hostbased authentication. debug1: Next authentication method: publickey debug1: Trying private key: /home/real/.ssh/id_rsa debug1: Trying private key: /home/real/.ssh/id_dsa debug1: Trying private key: /home/real/.ssh/id_ecdsa debug1: Trying private key: /home/real/.ssh/id_ed25519 debug1: Next authentication method: keyboard-interactive Password: debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive,host based Any help will be greatly appreciated! Thank you, Liz

6 years, 11 months

3
6
0 / 0

MMR: How do you tell that the MMR servers are in sync?

by Frank Swasey

I am a newbie with MMR. I've just set up my first test of this. I have found it to be almost impossible to determine (via the contextCSN values) if the masters (let alone all the replicas) are actually synched. How are you folks, who are already using MMR, checking/verifying that the MMR participants (and their replicas) are actually in sync? Thanks, -- Frank Swasey | http://www.uvm.edu/~fcs Sr Systems Administrator | Always remember: You are UNIQUE, University of Vermont | just like everyone else. "I am not young enough to know everything." - Oscar Wilde (1854-1900)

6 years, 11 months

7
13
0 / 0

Virtual list view problem

by Venish Khant

Hi all I am using cpan Net::LDAP module to access LDAP entries. I want to search LDAP entries using Net::LDAP search method. When I do search, I want some limited number of entries from search result, for this(searching) process I am using Net::LDAP::Control::VLV module. But I get error on VLV response control. Please, any one have idea about this error. * Error:* Died at vlv.pl line 50, This is my example. I changed the font style of line 50 #!/usr/bin/perl -w use Net::LDAP; use Net::LDAP::Control::VLV; use Net::LDAP::Constant qw( LDAP_CONTROL_VLVRESPONSE ); use Net::LDAP::Control::Sort; sub procentry { my ( $mesg, $entry) = @_; # Return if there is no entry to process if ( !defined($entry) ) { return; } print "dn: " . $entry->dn() . "\n"; @attrs = $entry->attributes(); foreach $attr (@attrs) { #printf("\t%s: %s\n", $attr, $entry->get_value($attr)); $attrvalue = $entry->get_value($attr,asref=>1); #print $attr.":". $entry->get_value($attr)."\n"; foreach $value(@$attrvalue) { print "$attr: $value\n"; } } $mesg->pop_entry; print "\n"; } $ldap = Net::LDAP->new( "localhost" ); # Get the first 20 entries $vlv = Net::LDAP::Control::VLV->new( before => 0, # No entries from before target entry after => 19, # 19 entries after target entry content => 0, # List size unknown offset => 1, # Target entry is the first ); my $sort = Net::LDAP::Control::Sort->new( order => 'cn' ); @args = ( base => "dc=example,dc=co,dc=in", scope => "subtree", filter => "(objectClass=inetOrgPerson)", callback => \&procentry, # Call this sub for each entry control => [ $sort, $vlv ], ); $mesg = $ldap->search( @args ); # Get VLV response control *($resp) = $mesg->control( LDAP_CONTROL_VLVRESPONSE ) or die;* $vlv->response( $resp ); # Set the control to get the last 20 entries $vlv->end; $mesg = $ldap->search( @args ); # Get VLV response control ($resp) = $mesg->control( LDAP_CONTROL_VLVRESPONSE ) or die; $vlv->response( $resp ); # Now get the previous page $vlv->scroll_page( -1 ); $mesg = $ldap->search( @args ); # Get VLV response control ($resp) = $mes # Now page with first entry starting with "B" in the middle $vlv->before(9); # Change page to show 9 before $vlv->after(10); # Change page to show 10 after $vlv->assert("B"); # assert "B" $mesg = $ldap->search( @args );g->control( LDAP_CONTROL_VLVRESPONSE ) or die; $vlv->response( $resp ); -- Venish Khant www.deeproot.co.in

6 years, 12 months

4
11
0 / 0

Re: Documentation

by Daniel Howard

My recent experience is OpenLDAP on Ubuntu. I thought I would go with OpenLDAP's guide because they should know better, but the quick start was for older versions or something and hadn't been updated. I like documentation systems that allow for user feedback, comments, or patches via github. If you want community improvements then maybe low hanging fruit is a link in the doc template on how to interact with the documentation.... If the current docs platform has no mechanism for feedback and comments on a specific document, the source code/ build process is obscure ... Maybe consider porting even a "fork " to github and solicit community feedback (you can file issues in a familiar mechanism, and take in patches from dudes like me...) For what it's worth, the Ubuntu OpenLDAP quick start guide and only a few minor hiccups, and I would recommend that resource for anyone looking to implement or update a quickstart guide. The Faq-o-Matic ... Man a modern stack overflow site might be nice but a question in the back of my mind is whether the advice is still useful. I don't know if the FoM does this but if the answers showed dates then I could evaluate an answer that's say 2-4 years old with greater confidence than say something 12 years old ... One of the challenges of a venerable, old and mature project like OpenLDAP. I guess what I'm trying to say, from my mobile phone, is if the documentation were more "agile" in engaging the community for identifying issues and taking in corrections, we may all be happier. I would be open to volunteering to port docs... Thanks, -Danny -- Sent from my Western Electric Model 500 On Apr 29, 2016 5:01 PM, "Gavin Henry" <ghenry(a)openldap.org> wrote: Hi all, List what you love about our docs: List what you hate: List what you'd love to see: List what you can help with: Thanks. -- Kind Regards, Gavin Henry.

7 years

6
9
0 / 0

slapadd hang in bdb_tool_entry_close / ldap_pvt_thread_cond_wait

by Bannister, Mark

Hi, Can anyone shed light on why my slapadd process on Solaris 11 was hanging (slapd 2.4.30)? I was piping 'gunzip -c' through to slapadd (from an overnight cron job). This usually works fine. I'm guessing perhaps the gunzip process encountered an error, maybe an EPIPE was sent, but the stack trace many days later from slapadd doesn't look right. Only 2 threads, and both of them in ldap_pvt_thread_cond_wait(), but no other threads so what are they waiting for exactly? # pstack /var/tmp/cores/core.slapd.0.16189.1463492837 core '/var/tmp/cores/core.slapd.0.16189.1463492837' of 16189: /usr/sbin/slapadd -q -f /etc/openldap/slapd.conf ------------ lwp# 1 / thread# 1 --------------- 07ff7af9 lwp_park (0, 0, 0) 07ff1193 cond_wait_queue (82ed048, 82ed020, 0, 7ff16b0) + 63 07ff1728 __cond_wait (82ed048, 82ed020, f5a685c8, 7ff1771) + 89 07ff177f cond_wait (82ed048, 82ed020, f5a685f8, 7ff17b7) + 27 07ff17cc pthread_cond_wait (82ed048, 82ed020, f5a68618, 7e56137) + 24 07e5614c ldap_pvt_thread_cond_wait (82ed048, 82ed020, f5a68648, 81b7e29) + 24 081b7e7a bdb_tool_entry_close (8e9d398, ffffffff, f5a68698, 82ecc38) + 62 081a38ec slapadd (4, f5a68bbc, f5a68bbc, 80dccb1) + cb8 080dcd2c main (4, f5a68bbc, f5a68bd0, ef60f968) + ac 080dc86d _start (4, f5a68c8e, f5a68ca0, f5a68ca3, f5a68ca6, 0) + 7d ------------ lwp# 2 / thread# 2 --------------- 07ff7af9 lwp_park (0, 0, 0) 07ff1193 cond_wait_queue (8e6b0e4, 8e6b0cc, 0, 7ff16b0) + 63 07ff1728 __cond_wait (8e6b0e4, 8e6b0cc, d7dffdb8, 7ff1771) + 89 07ff177f cond_wait (8e6b0e4, 8e6b0cc, 8e6b0c8, 7ff17b7) + 27 07ff17cc pthread_cond_wait (8e6b0e4, 8e6b0cc, 0, 7e56137) + 24 07e5614c ldap_pvt_thread_cond_wait (8e6b0e4, 8e6b0cc, d7dfffc8, 7e54bfc) + 24 07e54d66 ldap_int_thread_pool_wrapper (8e6b0c8, 803a000, d7dfffe8, 7ff7769) + 242 07ff77bc _thrp_setup (ee1e0240) + 9d 07ff7aa0 _lwp_start (ee1e0240, 0, 0, 0, 0, 0) Thanks, Mark. ________________________________ NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.

7 years

2
7
0 / 0

Improved handling for large number of databases / Access newly opened database from another transaction

by Jürg Bircher

Hello Improved handling for large number of databases =============================================== There is a increased performance penalty the more databases are created within the same environment. I was looking for a way the improve that by keeping the simplicity of tracking databases within a list with direct access by index (MDB_dbi). mdb_dbi_open() is however not improved with the assumption that the database handle (dbi) is cached in the application. So mdb_dbi_open() should happen only once for each database during the life time of an application. One issue is that mdb_txn_begin() (for read-only transactions) calloc the sizeof(MDB_txn) + me_maxdbs * sizeof(MDB_db + 1). The plus 1 for the dbflags. However it is sufficient only to malloc that size and clear the sizeof(MDB_txn) memset(txn, 0, sizeof(MDB_txn) After that the data beyond the MDB_txn is not initialized which is ok for the moment. The next improvement happens in mdb_txn_renew0() where the dbflags are only set to DB_UNUSED (a new flag) for each database currently opened in the environment. memset(txn->mt_dbflags, DB_UNUSED, txn->mt_numdbs); The former code used to to loop through each database to calculate the dbflags. This is still done but lazily for each accessed database with the assumption that a read only transaction rarely uses all databases of the environment. The lazy initialization of the dbflag happens in the macro TXN_DBI_EXIST which is always used when a database handle (dbi) is passed to an function. The flags are updated in mdb_setup_db_info() once a database is access which is marked as unused (DB_UNUSED). static int mdb_setup_db_info(MDB_txn *txn, MDB_dbi dbi) { /* Setup db info */ uint16_t x = txn->mt_env->me_dbflags[dbi]; txn->mt_dbs[dbi].md_flags = x & PERSISTENT_FLAGS; txn->mt_dbflags[dbi] = (x & MDB_VALID) ? DB_VALID|DB_USRVALID|DB_STALE : 0; return (txn->mt_dbflags[dbi] & validity); } /** Check \b txn and \b dbi arguments to a function and initialize db info if needed */ #define TXN_DBI_EXIST(txn, dbi, validity) \ ((txn) && (dbi)<(txn)->mt_numdbs && (((txn)->mt_dbflags[dbi] & (validity)) || (((txn)->mt_dbflags[dbi] & DB_UNUSED) && mdb_setup_db_info((txn), (dbi), (validity))))) The next improvement is done in any function which needs to loop through the databases for example in mdb_cursors_close(). Again the more databases in the environment the longer the execution time. It should be best if looping only through dbflags and searching for those databases which are used (!DB_UNUSED). This could be done byte wise or more efficient in 8/4 byte steps comparing with an extended mask DB_UNUSED_LONG instead of DB_UNUSED. So we can skip 8 or 4 (32 bit) unused databases in one step (still with the assumption that a transaction rarely uses all databases of the environment). So the loop looks as follows always starting at the lower index to avoid alignment issues with ARM prior v6. #define DB_UNUSED 0x20 /**< DB not used in this txn */ #ifdef MDB_VL32 #define DB_UNUSED_LONG 0x20202020 /* DB_UNUSED long mask for fast tracking */ #else #define DB_UNUSED_LONG 0x2020202020202020 /* DB_UNUSED long mask for fast tracking */ #endif #ifdef MDB_VL32 #define MDB_WORD unsigned int #else #define MDB_WORD unsigned long long #endif MDB_dbi n = src->mt_numdbs; MDB_dbi i = 0; while (1) { unsigned int upper = i + sizeof(MDB_WORD); if (upper < n) { // skip unused if ((*(MDB_WORD *)(tdbflags + i)) == DB_UNUSED_LONG) { i = upper; continue; } } else { upper = n; } for (; i < upper; i++) { // any other filter criteria appropriate to the function .... } if (i >= n) { break; } } Access newly opened database from another transaction ======================================================= A transaction tracks newly opened databases and if the transaction is committed the newly opened databases are propagated to the list of open databases of the environment. However if the read only transaction is aborted the databases are not propagated. If database handles (MDB_dbi) are cached in the application to avoid calling mdb_dbi_open() there might be the situation of two threads running a read only transaction concurrently. First threads opens the database and commits the read-only transaction. The database is added to the list of open databases in the environment. The returned dbi is globally cached in the application. The second thread also wants to access the same database and finds the database handle in the global application cache. The database handle however is not valid as the transaction only uses a snapshot of open databases. So the second thread gets an EINVAL when using that database handle. This should not happen as the database is open and added to the environment. The following should fix this issue by updating the mt_numdbs and marking the delta with DB_UNUSED static int mdb_update_db_info(MDB_txn *txn, MDB_dbi dbi) { /* propagate newly opened db from env to current txn. Mark them as unused */ if (txn->mt_numdbs < txn->mt_env->me_numdbs) { memset(txn->mt_dbflags + txn->mt_numdbs, DB_UNUSED, txn->mt_env->me_numdbs - txn->mt_numdbs); } txn->mt_numdbs = txn->mt_env->me_numdbs; return dbi < txn->mt_numdbs; } /** Check \b txn and \b dbi arguments to a function and initialize db info if needed */ #define TXN_DBI_EXIST(txn, dbi, validity) \ ((txn) && ((dbi)<(txn)->mt_numdbs || mdb_update_db_info((txn), (dbi))) && (((txn)->mt_dbflags[dbi] & (validity)) || (((txn)->mt_dbflags[dbi] & DB_UNUSED) && mdb_setup_db_info((txn), (dbi), (validity))))) If interested let me know how to contribute. Hope it is useful! Regards Juerg

7 years

2
5
0 / 0

memberOf group searches very slow

by Jason Whitener

We have some groups that contain over 250,000 members. Are there any general tips for improving performance when applications/code attempts to lookup memberOf values in the ldap? For example, I have something like ou=SomeGroup,ou=Groups,o=ourOrg. ou=SomeGroup has over 250,000 memberOf: values. Attempting to modify, search, insert, etc.. into that memberOf multivalue field takes over 20 seconds per user.

7 years

2
1
0 / 0

problems connecting to ldaps:// under high load with ppc64 client

by Matthias Leopold

hi, i'm operating an owncloud server that connects to an IBM Tivoli Directory Server as LDAP backend. the ldap admin tells me he is seeing "null binds" from my owncloud server in his logs: 2016-05-24T14:32:56.349452+2:00 srvr_ssl_read: EIO in handshake. EWOULDBLOCK timeout. Read: -2 of 0 2016-05-24T14:32:56.350445+2:00 GLPSSL019E The SSL layer has reported an unidentified internal error, SSL extended error code:406. 2016-05-24T14:32:56.351813+2:00 GLPSRV022E Failed to initialize secure connection from client (connection ID: 61786, IP address: x.x.x.x, Port: 59921). 2016-05-24T14:32:56.357220+2:00 GLPSRV044W Client connection from x.x.x.x bound as NULL closed by server. i investigated on my server and noticed that it has problems connecting to the ldaps://ldap.example.com uri (which is the ITDS server) under high client system load, whereas connection to ldap://ldap.example.com is ok. $ ldapsearch -v -x -z 0 -H ldaps://ldap.example.com -b "ou=groups,dc=example,dc=com" -v "objectClass=posixGroup" ldap_initialize( ldaps://ldap.example.com:636/??base ) ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1) my server (RHEL 7 on a ppc64 LPAR) is using the openldap clients/libraries. the high load that is causing the problems is on _my_ server. is there any specific tuning (besides increasing RAM/CPU) i can do to optimize ldaps client queries? i'm thinking of tuning the tcp stack or something similar, but i'm not an expert on this. where can i look for debug info? i have strace and tcpdump output thx matthias

7 years

3
2
0 / 0

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

openldap-technical May 2016