openldap-devel

openldap-devel@openldap.org

1443 discussions

Re: Playing with MMR in HEAD
by Gavin Henry 19 Nov '07

19 Nov '07

Also, With the 3 running slapd instances now, switching a drink attribute caused a crash on the con2: lt-slapd: encode.c:346: ber_put_ostring: Assertion `str != ((void *)0)' failed. Aborted I'll collect more complete debugging info as I continue. Thanks. -- Kind Regards, Gavin Henry. Managing Director. T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry(a)suretecsystems.com Open Source. Open Solutions(tm). http://www.suretecsystems.com/ <quote who="Gavin Henry"> > Hi All, > > I'm playing with test050, i.e. running it, copying con directory to con2 > and then starting the original provider and consumer up. Then starting up > con2 on ldap://:9013/ > > All fine, then I go about adding a new syncrepl on cn=config and > dc=example,dc=com pointing to ldap://:9013/ > > I'm not using LDIF, just messing with JXplorer. > > Everything worked ok, with a couple of crashes at the point I added a new > syncrepl on con2 point to itself. > > Both pro and con died with: > > lt-slapd: attr.c:481: attr_merge: Assertion `( nvals == ((void *)0) && > (*a)->a_nvals == (*a)->a_vals ) || ( nvals != ((void *)0) && ( ( > (*a)->a_vals == ((void *)0) && (*a)->a_nvals == ((void *)0) ) || ( > (*a)->a_nvals != (*a)->a_vals ) ) )' failed. > Aborted > > I guess I should pay attention more to the order in test050 and timing. > > Lastly, how do we define test050 to be a multimaster test when there are > only two instances of slapd? > > Is it because we have 2 sycnrepl on cn=config and then another two on the > bdb backends? I thought a test for this would include at least 3 running > slapd instances? > > Thanks, > > Gavin. > > -- > Kind Regards, > > Gavin Henry. > Managing Director. > > T +44 (0) 1224 279484 > M +44 (0) 7930 323266 > F +44 (0) 1224 824887 > E ghenry(a)suretecsystems.com > > Open Source. Open Solutions(tm). > > http://www.suretecsystems.com/ > > >

1 0

Playing with MMR in HEAD
by Gavin Henry 19 Nov '07

19 Nov '07

Hi All, I'm playing with test050, i.e. running it, copying con directory to con2 and then starting the original provider and consumer up. Then starting up con2 on ldap://:9013/ All fine, then I go about adding a new syncrepl on cn=config and dc=example,dc=com pointing to ldap://:9013/ I'm not using LDIF, just messing with JXplorer. Everything worked ok, with a couple of crashes at the point I added a new syncrepl on con2 point to itself. Both pro and con died with: lt-slapd: attr.c:481: attr_merge: Assertion `( nvals == ((void *)0) && (*a)->a_nvals == (*a)->a_vals ) || ( nvals != ((void *)0) && ( ( (*a)->a_vals == ((void *)0) && (*a)->a_nvals == ((void *)0) ) || ( (*a)->a_nvals != (*a)->a_vals ) ) )' failed. Aborted I guess I should pay attention more to the order in test050 and timing. Lastly, how do we define test050 to be a multimaster test when there are only two instances of slapd? Is it because we have 2 sycnrepl on cn=config and then another two on the bdb backends? I thought a test for this would include at least 3 running slapd instances? Thanks, Gavin. -- Kind Regards, Gavin Henry. Managing Director. T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry(a)suretecsystems.com Open Source. Open Solutions(tm). http://www.suretecsystems.com/

1 0

more perf numbers
by Howard Chu 16 Nov '07

16 Nov '07

Comparing OpenLDAP 2.3.39 to RE24, 2.3.39 gets about 20.3K auths/sec on a 1M entry hdb database, vs 23.3K auths/sec for RE24 on our test machine. So about a 15% gain, most of which is probably the lightweight dispatcher. (Will have to test again with back-null to see for certain.) -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 3

back-ldap & rewrite engine
by Dieter Kluenter 27 Oct '07

27 Oct '07

Hi, I am testing REL_ENG_2_4 and HEAD back-ldap and rw with the same slapd.conf, and a sasl bind. While RE_2.4 just crashes with segfault, HEAD is not rewriting the sasl ID, that is proxy ID: cn=Dieter Kluenter,ou=partner,dc=dkluenter,dc=de master ID: cn=Dieter Kluenter,ou=partner,o=avci,c=de Based on this configuration earlier HEAD versions had rewriten sasl identity properly. My searchstring for back-ldap is: ldapsearch -Y digest-md5 -U dieter -w xxxx-H ldap://localhost:9004 -b "cn=dieter kluenter,ou=partner,dc=dkluenter,dc=de" -s sub "*" ,----[ slapd.conf back-ldap ] | modulepath /opt/openldap/libexec/openldap | moduleload back_meta.la | moduleload back_ldap.la | moduleload pcache.la | moduleload rwm.la | authz-regexp uid=(.*),cn=.*,cn=auth | ldap:///dc=dkluenter,dc=de??sub?uid=$1 | | access to * by * read | database ldap | suffix dc=dkluenter,dc=de | rootdn cn=admin,dc=dkluenter,dc=de | uri ldap://localhost:389 | acl-bind | bindmethod=sasl | saslmech=digest-md5 | authcId=admanager | credentials=xxxxx | idassert-bind | bindmethod=sasl | saslmech=digest-md5 | authcId=admanager | credentials=xxxx | mode=self | overlay rwm | rwm-rewriteEngine on | rwm-suffixmassage "dc=dkluenter,dc=de" "o=avci,c=de" | overlay pcache | proxycache bdb 10000 22 50 3600 | ... `---- ,----[ back trace of RE_2.4 ] | <= ldap_dn2bv(cn=dieter kluenter,ou=partner,o=avci,c=de)=0 | <<< dnPrettyNormal: <cn=Dieter Kluenter,ou=Partner,o=avci,c=de>, <cn=dieter kluenter,ou=partner,o=avc | ,c=de> | ber_scanf fmt ({xx) ber: | Program received signal SIGSEGV, Segmentation fault. | [Switching to Thread 0xb6d2fb90 (LWP 12590)] | 0x00000000 in ?? () | | (gdb) bt | #0 0x00000000 in ?? () | #1 0xb7826a2c in ldap_back_search (op=0xb6d2eadc, rs=0xb6d2ebb0) at search.c:338 | #2 0x080d51c0 in overlay_op_walk (op=0xb6d2eadc, rs=0xb6d2ebb0, which=op_search, oi=0x8251b80, | on=0x8253970) at backover.c:652 | #3 0x080d56ee in over_op_func (op=0xb6d2eadc, rs=0xb6d2ebb0, which=op_search) at backover.c:704 | #4 0x080b92b5 in slap_sasl2dn (opx=0x82ba8a0, saslname=0xb6d2ed70, sasldn=0xb6d2ece4, flags=2) | at saslauthz.c:2005 | #5 0x080bf2e3 in slap_sasl_getdn (conn=0xb753264c, op=0x82ba8a0, id=0xb6d2ed68, user_realm=0x0, | dn=0xb6d2ed70, flags=<value optimized out>) at sasl.c:2035 | #6 0x080c0725 in slap_sasl_canonicalize (sconn=0x82b8f70, context=0xb753264c, | in=0x82ba158 "dieter", inlen=6, flags=1, user_realm=0x0, out=0x82b9c81 "", out_max=1024, | out_len=0x82b97e0) at sasl.c:624 | #7 0xb7d4f2b8 in _sasl_canon_user () from /usr/lib/libsasl2.so.2 | #8 0xb7a20576 in ?? () from /usr/lib/sasl2/libdigestmd5.so | #9 0x082b8f70 in ?? () | #10 0x082ba158 in ?? () | #11 0x00000006 in ?? () | #12 0x00000001 in ?? () | #13 0x082b97d0 in ?? () | #14 0xb7f92ee0 in ?? () from /lib/ld-linux.so.2 | #15 0xb7f36161 in ber_flush2 (sb=0x113, ber=0xb6d2edd0, freeit=-1227689916) at io.c:256 | #16 0xb7d58b07 in sasl_server_step () from /usr/lib/libsasl2.so.2 | #17 0x080c0208 in slap_sasl_bind (op=0x0, rs=0x82b8f70) at sasl.c:1654 | #18 0x08091790 in fe_op_bind (op=0x82ba8a0, rs=0xb6d2f144) at bind.c:280 | #19 0x08092191 in do_bind (op=0x82ba8a0, rs=0xb6d2f144) at bind.c:205 | ---Type <return> to continue, or q <return> to quit--- | #20 0x08074431 in connection_operation (ctx=0xb6d2f238, arg_v=0x82ba8a0) at connection.c:1145 | #21 0x08074ace in connection_read_thread (ctx=0xb6d2f238, argv=0xf) at connection.c:1271 | #22 0xb7f4a2b2 in ldap_int_thread_pool_wrapper (xpool=0x821a7b8) at tpool.c:619 | #23 0xb7d67192 in start_thread () from /lib/libpthread.so.0 | #24 0xb7b3902e in clone () from /lib/libc.so.6 | (gdb) `---- ,----[ slapd-log on master with HEAD ] | slapd[3832]: => acl_mask: access to entry "cn=Dieter Kluenter,ou=Partner,o=avci,c=de", attr "entry" requested | slapd[3832]: => acl_mask: to all values by "cn=dieter kluenter,ou=partner,dc=dkluenter,dc=de", (=0) | slapd[3832]: <= check a_dn_pat: cn=$1,ou=Partner,o=avci,c=de | slapd[3832]: <= check a_group_pat: cn=administratoren,o=avci,c=de | slapd[3832]: => bdb_entry_get: found entry: "cn=administratoren,o=avci,c=de" | slapd[3832]: <= check a_dn_pat: * | slapd[3832]: <= acl_mask: [3] applying auth(=xd) (stop) | slapd[3832]: <= acl_mask: [3] mask: auth(=xd) | [3832]: => slap_access_allowed: search access denied by auth(=xd) | slapd[3832]: => access_allowed: no more rules `---- Is there anything wrong with my slapd.conf? Or is this a bug? -Dieter -- Dieter Klünter | Systemberatung http://www.dkluenter.de GPG Key ID:8EF7B6C6

1 0

Estimated cost to develop OpenLDAP: $9,621,793
by Gavin Henry 26 Oct '07

26 Oct '07

Quite an interesting output running sloccount over OpenLDAP HEAD: SLOC Directory SLOC-by-Language (Sorted) 138537 servers ansic=138092,cpp=254,perl=152,sh=39 60773 libraries ansic=60773 28054 contrib ansic=14646,sh=8765,cpp=4472,perl=126,tcl=45 20657 tests sh=17287,ansic=3370 9518 build sh=9515,ansic=3 6025 include ansic=6025 5657 clients ansic=5657 0 CVS (none) 0 doc (none) 0 top_dir (none) Totals grouped by language (dominant language first): ansic: 228566 (84.90%) sh: 35606 (13.23%) cpp: 4726 (1.76%) perl: 278 (0.10%) tcl: 45 (0.02%) Total Physical Source Lines of Code (SLOC) = 269,221 Development Effort Estimate, Person-Years (Person-Months) = 71.23 (854.72) (Basic COCOMO model, Person-Months = 2.4 (KSLOC**1.05)) Schedule Estimate, Years (Months) = 2.71 (32.51) (Basic COCOMO model, Months = 2.5 (person-months**0.38)) Estimated Average Number of Developers (Effort/Schedule) = 26.29 Total Estimated Cost to Develop = $ 9,621,793 (average salary = $56,286/year, overhead = 2.40). SLOCCount, Copyright (C) 2001-2004 David A. Wheeler SLOCCount is Open Source Software/Free Software, licensed under the GNU GPL. SLOCCount comes with ABSOLUTELY NO WARRANTY, and you are welcome to redistribute it under certain conditions as specified by the GNU GPL license; see the documentation for details. Please credit this data as "generated using David A. Wheeler's 'SLOCCount'." :-) -- Kind Regards, Gavin Henry. Managing Director. T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry(a)suretecsystems.com Open Source. Open Solutions(tm). http://www.suretecsystems.com/

1 0

Re: ldap_int_sasl_bind() and canonical Kerberos names
by Geert Jansen 25 Oct '07

25 Oct '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Simon, >> The patch unconditionally disables hostname canonicalisation for >> the sasl client. > > I think this will break GSSAPI connections to LDAP servers that are > behind DNS round robin style load balancers. > > Assume that you have 'ldap' that is a CNAME for ldap-1 and ldap2. > The LDAP library initiates a connection to 'ldap', and DNS points > it to 'ldap-1'. Providing you ask SASL to set up a connection to > 'ldap-1', you're fine (this is what the code does at the moment). > However, if you ask the SASL library for a connection to 'ldap' > (this is what your change does, as far as I can tell), and the > library does a canonicalisation step (as most Kerberos > implementations currently do), it will get 'ldap-2' back from the > DNS. So, you end up trying to negotiate a SASL connection with > 'ldap-2', when you're actually connected to 'ldap-1'. This tends > not to work. Thanks for the explanation and indeed I see that my patch would break this use case. I have come to the conclusion that one of the two reasons why I looked into this patch (do not keep realm data on the local system) is actually a problem that has nothing to do with canonicalisation. Realm referrals work just as well with the reverse-dns based canonical name as long as it is a valid principal. The other reason (be resilient against wrong reverse-dns setups) could still be a reason to use this patch, but now that I think of it it, it probably doesn't buy you much as you need proper forward dns anyway so better have a proper reverse dns too. Therefore I withdraw my patch. Regards, Geert -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iQEVAwUBRyDy7prPkVOV4lDgAQJD9gf+KzsoMxBtCf0K0f80q6kWM+DAB64EpwCV g2Lj9JdKK0BcxcyXpnz+vBrKPwuP8RP/1dkvNiBJrgcUVc9Yo25H+UNBHkff0wUt QPZZvf9p/nxz0AQHAYrHdh94fM748y2LuMD/oVkpu+Oi8HeC5P5fo2VMpsoJ9pcg 9ee23yyuT3EyjpG3YGnApOOdAPgEqgUirvI+DibFYXo4hLrzwL5PKRmY3ggMZKa1 OrHz2qjZjvcktbs3cSU0v17tG+KLW1DtKaO80bSrbjAqb0l4rVPI+a6ixN5IEbG7 RdGKzx9jP4hgTP+Xt06e+eNFg19u0e72mrlzmH2A29C5RA2cHVsMRg== =bLmO -----END PGP SIGNATURE-----

1 0

Re: thread pools, performance
by Howard Chu 25 Oct '07

25 Oct '07

Rick Jones wrote: >> The ethernet controllers are on a hub attached by HyperTransport to a >> single processor, I don't think you can usefully distribute the >> interrupts to anything beyond that socket. > > Well, it wouldn't necessarily be goodness from the standpoint of the time to do > any PIO reads, but from the standpoint of getting packet processing spread-out > and matching with the core on which the application thread runs it might be. > All depends I guess on how busy that HT link gets I guess and whether the NIC(s) > can do the spreading in the first place. I wonder if there are any pre-packaged > linux binaries out there to measure HT link utilization? Haven't looked. Would be interesting to see. > >>> Although, if there is still 10% idle, that probably needs to go next :) >> >> Heh heh. The 80/20 rule hits this with a vengeance. That's "10%" of >> "800%" total, which means really only about 1.2% of a CPU, which is >> almost totally indistinguishable from measurement error in the oprofile >> results. This is all intuition (guesswork) now, no more obvious hot >> spots left to attack. Maybe if I'm really bored over the holidays I'll >> spend some time on it. (Not likely.) > > My degree is in AppliedMath which means I cannot do math to save my life, but if > it was 10%agepoints of 800%agepoints total, that means 10% of 8 cores or 80% of > a core doesn't it? No, there are 800 %agepoints total for all 8 cores, and 10 out of 800 are idle. That's a total of 10% of a single core, but it's actually distributed across all 8 cores. So 1.25% of each core. > If it were 10% of one core, then it would be 1.25% of 800% IIRC. Yes. > And besides, the nature of single-threading bottlenecks seems to be that when > you go to 16 cores it will be rather more than 2X what it is today :) Yes, well, I haven't got a 16-core test system handy at the moment... The changes I just checked in hit the easy stuff, reducing mutex contention by about 25% on a read-only workload with a nice corresponding boost in throughput. I haven't gotten to the multiple pools yet. The most obvious one was using per-thread lists for the Operation free lists instead of a single global free list. This assumes that threads will get work allocated relatively uniformly; I guess we can add some counts and push free Ops onto a global list if a per-thread list gets too long. At the moment there's no checking, so once a few operations have been performed Operation allocation occurs with pretty much zero mutex activity. (This is probably redundant effort when using something like Google tcmalloc, but not everyone is using that...) The other change was using per-thread slap_counters for statistics instead of just a global statistics block. In this case, the global slap_counters structure still exists as the head of the list, and its mutex is used to protect list traversals/manipulations. But actual live data is accumulated in per-thread structures chained off the global. The head mutex is locked when allocating a new per-thread structure, deallocating a structure for an exiting thread, and whenever back-monitor wants to tally up data to present to a client. In normal operation, once the configured number of threads have been created, there will be no accesses to the global at all. If cn=config is used to decrease the number of threads, then some number of threads will exit and so they'll lock the list at that time and accumulate their stats onto the head structure before destroying themselves. So again, in normal operation, if nobody is querying cn=monitor, there will be zero mutex contention due to statistics counters. Something else I've been considering is padding certain structures to sizes that are multiples of a given cache line. Not sure I want to write autoconf tests for that, may just arbitrarily choose 64 bytes and let folks override it at compile time. The Connection array is the most likely candidate here. Something like #define ALIGN 64 typedef real_struct { whatever } real_struct; typedef padded_struct { real_struct foo; char pad[(sizeof(real_struct)+ALIGN-1) & ~(ALIGN-1)]; } padded_struct; -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Re: ldap_int_sasl_bind() and canonical Kerberos names
by Quanah Gibson-Mount 24 Oct '07

24 Oct '07

--On October 24, 2007 10:42:43 PM +0100 Simon Wilkinson <simon(a)sxw.org.uk> wrote: > >> >> The patch unconditionally disables hostname canonicalisation for >> the sasl client. > > I think this will break GSSAPI connections to LDAP servers that are > behind DNS round robin style load balancers. > > Assume that you have 'ldap' that is a CNAME for ldap-1 and ldap2. The > LDAP library initiates a connection to 'ldap', and DNS points it to > 'ldap-1'. Providing you ask SASL to set up a connection to 'ldap-1', > you're fine (this is what the code does at the moment). However, if you > ask the SASL library for a connection to 'ldap' (this is what your change > does, as far as I can tell), and the library does a canonicalisation step > (as most Kerberos implementations currently do), it will get 'ldap-2' > back from the DNS. So, you end up trying to negotiate a SASL connection > with 'ldap-2', when you're actually connected to 'ldap-1'. This tends not > to work. Simon, Thanks! That'd completely destroy Stanford's setup. Ouch. --Quanah -- Quanah Gibson-Mount Principal Software Engineer Zimbra, Inc -------------------- Zimbra :: the leader in open source messaging and collaboration

2 1

servers/slapd/overlays/slapover.txt
by Gavin Henry 24 Oct '07

24 Oct '07

Anymore notes anywhere? CVS says: 1.1 Sat Mar 20 19:30:57 2004 UTC; 3 years, 7 months ago by hyc Overlay API notes. work in progress, please comment. It references docs that don't exist. Thanks. -- Kind Regards, Gavin Henry. Managing Director. T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry(a)suretecsystems.com Open Source. Open Solutions(tm). http://www.suretecsystems.com/

2 2

ldap_int_sasl_bind() and canonical Kerberos names
by Geert Jansen 24 Oct '07

24 Oct '07

Hi, at the moment, ldap_int_sasl_bind() uses ldap_host_connected_to() to get a fully qualified host name that will be used as the server fqdn with the sasl client. This fqdn is acquired by ldap_host_connected_to() using a reverse DNS lookup. The code explains why this is done: /* * do a reverse lookup on the addr to get the official hostname. * this is necessary for kerberos to work right, since the official * hostname is used as the kerberos instance. */ Using reverse DNS names has however always been problematic. The following comment is from the MIT code: (lib/krb5/os/sn2princ.c): /* XXX: This is *so* bogus. There are several cases where this won't get us the canonical name of the host, but this is what we've trained people to expect. We'll probably fix it at some point, but let's try to preserve the current behavior and only shake things up once when it comes time to fix this lossage. */ To address this issue, a draft RFC has been written (draft-ietf-krb-wg-kerberos-referrals-09) that adds server-side name canonicalisation to Kerberos and therefore removes the need to use reverse DNS for this. This draft has been implemented in MIT Kerberos 1.6. The feature is enabled by default and if you want to use it you probably want to set "rdns = false" in [libdefaults] to disable canonicalisation based on reverse DNS. Disabling these reverse DNS lookups however is not possible at the moment with the OpenLDAP client as explained above. I did a quick patch to have ldap_int_sasl_bind() use a value based on the LDAP option LDAP_OPT_HOST_NAME and that worked as expected. Would you guys be interested in a patch that allows the disabling hostname canonicalisation based on reverse DNS? The patch would need to make this behaviour optional and non-default as some real workloads may break and also it would somehow need to handle LDAP URIs with multiple hosts. Regards, Geert Jansen

2 3

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel