openldap-devel

openldap-devel@openldap.org

1416 discussions

Re: commit: ldap/tests/scripts defines.sh test050-syncrepl-multimaster
by Gavin Henry 19 Nov '07

19 Nov '07

<quote who="Aaron Richton"> > Yes, on x86_64 Fedora 8. Yeah, it's sometimes happening on i386 F8 too. Just passed now though: [ghenry@suretec tests]$ ./run test050 Cleaning up test run directory leftover from previous run. Running ./scripts/test050-syncrepl-multimaster... running defines.sh Initializing server configurations... Starting producer slapd on TCP/IP port 9011... Using ldapsearch to check that producer slapd is running... Inserting syncprov overlay on producer... Starting consumer slapd on TCP/IP port 9012... Using ldapsearch to check that consumer slapd is running... Configuring syncrepl on consumer... Starting consumer2 slapd on TCP/IP port 9013... Using ldapsearch to check that consumer2 slapd is running... Configuring syncrepl on consumer2... Adding schema and databases on producer... Using ldapadd to populate producer... Waiting 20 seconds for syncrepl to receive changes... Using ldapsearch to check that syncrepl received database changes... Using ldapsearch to check that syncrepl received database changes on consumer2... Using ldapsearch to read config from the producer... Using ldapsearch to read config from the consumer... Using ldapsearch to read config from consumer2... Filtering producer results... Filtering consumer results... Filtering consumer2 results... Comparing retrieved configs from producer and consumer... Comparing retrieved configs from producer and consumer2... Using ldapsearch to read all the entries from the producer... Using ldapsearch to read all the entries from the consumer... Using ldapsearch to read all the entries from the consumer2... Filtering producer results... Filtering consumer results... Filtering consumer2 results... Comparing retrieved entries from producer and consumer... Comparing retrieved entries from producer and consumer2... Restarting servers... Starting producer slapd on TCP/IP port 9011... Using ldapsearch to check that producer slapd is running... Waiting 5 seconds for slapd to start... Starting consumer slapd on TCP/IP port 9012... Using ldapsearch to check that consumer slapd is running... Starting consumer2 slapd on TCP/IP port 9013... Using ldapsearch to check that consumer2 slapd is running... Waiting 10 seconds for servers to resync... >>>>> Test succeeded

1 0

Re: Playing with MMR in HEAD
by Gavin Henry 19 Nov '07

19 Nov '07

<quote who="Howard Chu"> > Gavin Henry wrote: >> I am just working on a test053-syncrepl-three-multimasters for others to >> test. > > It would be sufficient to just extend test050. OK. I copied test050 to test053 and added to defines.sh. It completed, I'm just testing. Then I'll rename it to test050. ITS for this?

1 0

Re: Playing with MMR in HEAD
by Gavin Henry 19 Nov '07

19 Nov '07

So, If I start con2 up and then the pro/con, they die after a while with: lt-slapd: encode.c:346: ber_put_ostring: Assertion `str != ((void *)0)' failed. In: tests/testrun/{con,con2,pro} I'm running: ../../../servers/slapd/slapd -F slapd.d/ -h ldap://:9011/ -d 256 ../../../servers/slapd/slapd -F slapd.d/ -h ldap://:9012/ -d 256 ../../../servers/slapd/slapd -F slapd.d/ -h ldap://:9013/ -d 256 etc. Complete data: ftp://ftp.openldap.org/incoming/three_MM.tar.gz I am just working on a test053-syncrepl-three-multimasters for others to test. Gavin. -- Kind Regards, Gavin Henry. Managing Director. T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry(a)suretecsystems.com Open Source. Open Solutions(tm). http://www.suretecsystems.com/ <quote who="Gavin Henry"> > Also, > > With the 3 running slapd instances now, switching a drink attribute caused > a crash on the con2: > > lt-slapd: encode.c:346: ber_put_ostring: Assertion `str != ((void *)0)' > failed. > Aborted > > I'll collect more complete debugging info as I continue. > > Thanks. > > -- > Kind Regards, > > Gavin Henry. > Managing Director. > > T +44 (0) 1224 279484 > M +44 (0) 7930 323266 > F +44 (0) 1224 824887 > E ghenry(a)suretecsystems.com > > Open Source. Open Solutions(tm). > > http://www.suretecsystems.com/ > > <quote who="Gavin Henry"> >> Hi All, >> >> I'm playing with test050, i.e. running it, copying con directory to con2 >> and then starting the original provider and consumer up. Then starting >> up >> con2 on ldap://:9013/ >> >> All fine, then I go about adding a new syncrepl on cn=config and >> dc=example,dc=com pointing to ldap://:9013/ >> >> I'm not using LDIF, just messing with JXplorer. >> >> Everything worked ok, with a couple of crashes at the point I added a >> new >> syncrepl on con2 point to itself. >> >> Both pro and con died with: >> >> lt-slapd: attr.c:481: attr_merge: Assertion `( nvals == ((void *)0) && >> (*a)->a_nvals == (*a)->a_vals ) || ( nvals != ((void *)0) && ( ( >> (*a)->a_vals == ((void *)0) && (*a)->a_nvals == ((void *)0) ) || ( >> (*a)->a_nvals != (*a)->a_vals ) ) )' failed. >> Aborted >> >> I guess I should pay attention more to the order in test050 and timing. >> >> Lastly, how do we define test050 to be a multimaster test when there are >> only two instances of slapd? >> >> Is it because we have 2 sycnrepl on cn=config and then another two on >> the >> bdb backends? I thought a test for this would include at least 3 running >> slapd instances? >> >> Thanks, >> >> Gavin. >> >> -- >> Kind Regards, >> >> Gavin Henry. >> Managing Director. >> >> T +44 (0) 1224 279484 >> M +44 (0) 7930 323266 >> F +44 (0) 1224 824887 >> E ghenry(a)suretecsystems.com >> >> Open Source. Open Solutions(tm). >> >> http://www.suretecsystems.com/ >> >> >> > >

2 1

Re: Playing with MMR in HEAD
by Gavin Henry 19 Nov '07

19 Nov '07

Also, With the 3 running slapd instances now, switching a drink attribute caused a crash on the con2: lt-slapd: encode.c:346: ber_put_ostring: Assertion `str != ((void *)0)' failed. Aborted I'll collect more complete debugging info as I continue. Thanks. -- Kind Regards, Gavin Henry. Managing Director. T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry(a)suretecsystems.com Open Source. Open Solutions(tm). http://www.suretecsystems.com/ <quote who="Gavin Henry"> > Hi All, > > I'm playing with test050, i.e. running it, copying con directory to con2 > and then starting the original provider and consumer up. Then starting up > con2 on ldap://:9013/ > > All fine, then I go about adding a new syncrepl on cn=config and > dc=example,dc=com pointing to ldap://:9013/ > > I'm not using LDIF, just messing with JXplorer. > > Everything worked ok, with a couple of crashes at the point I added a new > syncrepl on con2 point to itself. > > Both pro and con died with: > > lt-slapd: attr.c:481: attr_merge: Assertion `( nvals == ((void *)0) && > (*a)->a_nvals == (*a)->a_vals ) || ( nvals != ((void *)0) && ( ( > (*a)->a_vals == ((void *)0) && (*a)->a_nvals == ((void *)0) ) || ( > (*a)->a_nvals != (*a)->a_vals ) ) )' failed. > Aborted > > I guess I should pay attention more to the order in test050 and timing. > > Lastly, how do we define test050 to be a multimaster test when there are > only two instances of slapd? > > Is it because we have 2 sycnrepl on cn=config and then another two on the > bdb backends? I thought a test for this would include at least 3 running > slapd instances? > > Thanks, > > Gavin. > > -- > Kind Regards, > > Gavin Henry. > Managing Director. > > T +44 (0) 1224 279484 > M +44 (0) 7930 323266 > F +44 (0) 1224 824887 > E ghenry(a)suretecsystems.com > > Open Source. Open Solutions(tm). > > http://www.suretecsystems.com/ > > >

1 0

Playing with MMR in HEAD
by Gavin Henry 19 Nov '07

19 Nov '07

Hi All, I'm playing with test050, i.e. running it, copying con directory to con2 and then starting the original provider and consumer up. Then starting up con2 on ldap://:9013/ All fine, then I go about adding a new syncrepl on cn=config and dc=example,dc=com pointing to ldap://:9013/ I'm not using LDIF, just messing with JXplorer. Everything worked ok, with a couple of crashes at the point I added a new syncrepl on con2 point to itself. Both pro and con died with: lt-slapd: attr.c:481: attr_merge: Assertion `( nvals == ((void *)0) && (*a)->a_nvals == (*a)->a_vals ) || ( nvals != ((void *)0) && ( ( (*a)->a_vals == ((void *)0) && (*a)->a_nvals == ((void *)0) ) || ( (*a)->a_nvals != (*a)->a_vals ) ) )' failed. Aborted I guess I should pay attention more to the order in test050 and timing. Lastly, how do we define test050 to be a multimaster test when there are only two instances of slapd? Is it because we have 2 sycnrepl on cn=config and then another two on the bdb backends? I thought a test for this would include at least 3 running slapd instances? Thanks, Gavin. -- Kind Regards, Gavin Henry. Managing Director. T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry(a)suretecsystems.com Open Source. Open Solutions(tm). http://www.suretecsystems.com/

1 0

more perf numbers
by Howard Chu 16 Nov '07

16 Nov '07

Comparing OpenLDAP 2.3.39 to RE24, 2.3.39 gets about 20.3K auths/sec on a 1M entry hdb database, vs 23.3K auths/sec for RE24 on our test machine. So about a 15% gain, most of which is probably the lightweight dispatcher. (Will have to test again with back-null to see for certain.) -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

2 3

back-ldap & rewrite engine
by Dieter Kluenter 27 Oct '07

27 Oct '07

Hi, I am testing REL_ENG_2_4 and HEAD back-ldap and rw with the same slapd.conf, and a sasl bind. While RE_2.4 just crashes with segfault, HEAD is not rewriting the sasl ID, that is proxy ID: cn=Dieter Kluenter,ou=partner,dc=dkluenter,dc=de master ID: cn=Dieter Kluenter,ou=partner,o=avci,c=de Based on this configuration earlier HEAD versions had rewriten sasl identity properly. My searchstring for back-ldap is: ldapsearch -Y digest-md5 -U dieter -w xxxx-H ldap://localhost:9004 -b "cn=dieter kluenter,ou=partner,dc=dkluenter,dc=de" -s sub "*" ,----[ slapd.conf back-ldap ] | modulepath /opt/openldap/libexec/openldap | moduleload back_meta.la | moduleload back_ldap.la | moduleload pcache.la | moduleload rwm.la | authz-regexp uid=(.*),cn=.*,cn=auth | ldap:///dc=dkluenter,dc=de??sub?uid=$1 | | access to * by * read | database ldap | suffix dc=dkluenter,dc=de | rootdn cn=admin,dc=dkluenter,dc=de | uri ldap://localhost:389 | acl-bind | bindmethod=sasl | saslmech=digest-md5 | authcId=admanager | credentials=xxxxx | idassert-bind | bindmethod=sasl | saslmech=digest-md5 | authcId=admanager | credentials=xxxx | mode=self | overlay rwm | rwm-rewriteEngine on | rwm-suffixmassage "dc=dkluenter,dc=de" "o=avci,c=de" | overlay pcache | proxycache bdb 10000 22 50 3600 | ... `---- ,----[ back trace of RE_2.4 ] | <= ldap_dn2bv(cn=dieter kluenter,ou=partner,o=avci,c=de)=0 | <<< dnPrettyNormal: <cn=Dieter Kluenter,ou=Partner,o=avci,c=de>, <cn=dieter kluenter,ou=partner,o=avc | ,c=de> | ber_scanf fmt ({xx) ber: | Program received signal SIGSEGV, Segmentation fault. | [Switching to Thread 0xb6d2fb90 (LWP 12590)] | 0x00000000 in ?? () | | (gdb) bt | #0 0x00000000 in ?? () | #1 0xb7826a2c in ldap_back_search (op=0xb6d2eadc, rs=0xb6d2ebb0) at search.c:338 | #2 0x080d51c0 in overlay_op_walk (op=0xb6d2eadc, rs=0xb6d2ebb0, which=op_search, oi=0x8251b80, | on=0x8253970) at backover.c:652 | #3 0x080d56ee in over_op_func (op=0xb6d2eadc, rs=0xb6d2ebb0, which=op_search) at backover.c:704 | #4 0x080b92b5 in slap_sasl2dn (opx=0x82ba8a0, saslname=0xb6d2ed70, sasldn=0xb6d2ece4, flags=2) | at saslauthz.c:2005 | #5 0x080bf2e3 in slap_sasl_getdn (conn=0xb753264c, op=0x82ba8a0, id=0xb6d2ed68, user_realm=0x0, | dn=0xb6d2ed70, flags=<value optimized out>) at sasl.c:2035 | #6 0x080c0725 in slap_sasl_canonicalize (sconn=0x82b8f70, context=0xb753264c, | in=0x82ba158 "dieter", inlen=6, flags=1, user_realm=0x0, out=0x82b9c81 "", out_max=1024, | out_len=0x82b97e0) at sasl.c:624 | #7 0xb7d4f2b8 in _sasl_canon_user () from /usr/lib/libsasl2.so.2 | #8 0xb7a20576 in ?? () from /usr/lib/sasl2/libdigestmd5.so | #9 0x082b8f70 in ?? () | #10 0x082ba158 in ?? () | #11 0x00000006 in ?? () | #12 0x00000001 in ?? () | #13 0x082b97d0 in ?? () | #14 0xb7f92ee0 in ?? () from /lib/ld-linux.so.2 | #15 0xb7f36161 in ber_flush2 (sb=0x113, ber=0xb6d2edd0, freeit=-1227689916) at io.c:256 | #16 0xb7d58b07 in sasl_server_step () from /usr/lib/libsasl2.so.2 | #17 0x080c0208 in slap_sasl_bind (op=0x0, rs=0x82b8f70) at sasl.c:1654 | #18 0x08091790 in fe_op_bind (op=0x82ba8a0, rs=0xb6d2f144) at bind.c:280 | #19 0x08092191 in do_bind (op=0x82ba8a0, rs=0xb6d2f144) at bind.c:205 | ---Type <return> to continue, or q <return> to quit--- | #20 0x08074431 in connection_operation (ctx=0xb6d2f238, arg_v=0x82ba8a0) at connection.c:1145 | #21 0x08074ace in connection_read_thread (ctx=0xb6d2f238, argv=0xf) at connection.c:1271 | #22 0xb7f4a2b2 in ldap_int_thread_pool_wrapper (xpool=0x821a7b8) at tpool.c:619 | #23 0xb7d67192 in start_thread () from /lib/libpthread.so.0 | #24 0xb7b3902e in clone () from /lib/libc.so.6 | (gdb) `---- ,----[ slapd-log on master with HEAD ] | slapd[3832]: => acl_mask: access to entry "cn=Dieter Kluenter,ou=Partner,o=avci,c=de", attr "entry" requested | slapd[3832]: => acl_mask: to all values by "cn=dieter kluenter,ou=partner,dc=dkluenter,dc=de", (=0) | slapd[3832]: <= check a_dn_pat: cn=$1,ou=Partner,o=avci,c=de | slapd[3832]: <= check a_group_pat: cn=administratoren,o=avci,c=de | slapd[3832]: => bdb_entry_get: found entry: "cn=administratoren,o=avci,c=de" | slapd[3832]: <= check a_dn_pat: * | slapd[3832]: <= acl_mask: [3] applying auth(=xd) (stop) | slapd[3832]: <= acl_mask: [3] mask: auth(=xd) | [3832]: => slap_access_allowed: search access denied by auth(=xd) | slapd[3832]: => access_allowed: no more rules `---- Is there anything wrong with my slapd.conf? Or is this a bug? -Dieter -- Dieter Klünter | Systemberatung http://www.dkluenter.de GPG Key ID:8EF7B6C6

1 0

Estimated cost to develop OpenLDAP: $9,621,793
by Gavin Henry 26 Oct '07

26 Oct '07

Quite an interesting output running sloccount over OpenLDAP HEAD: SLOC Directory SLOC-by-Language (Sorted) 138537 servers ansic=138092,cpp=254,perl=152,sh=39 60773 libraries ansic=60773 28054 contrib ansic=14646,sh=8765,cpp=4472,perl=126,tcl=45 20657 tests sh=17287,ansic=3370 9518 build sh=9515,ansic=3 6025 include ansic=6025 5657 clients ansic=5657 0 CVS (none) 0 doc (none) 0 top_dir (none) Totals grouped by language (dominant language first): ansic: 228566 (84.90%) sh: 35606 (13.23%) cpp: 4726 (1.76%) perl: 278 (0.10%) tcl: 45 (0.02%) Total Physical Source Lines of Code (SLOC) = 269,221 Development Effort Estimate, Person-Years (Person-Months) = 71.23 (854.72) (Basic COCOMO model, Person-Months = 2.4 (KSLOC**1.05)) Schedule Estimate, Years (Months) = 2.71 (32.51) (Basic COCOMO model, Months = 2.5 (person-months**0.38)) Estimated Average Number of Developers (Effort/Schedule) = 26.29 Total Estimated Cost to Develop = $ 9,621,793 (average salary = $56,286/year, overhead = 2.40). SLOCCount, Copyright (C) 2001-2004 David A. Wheeler SLOCCount is Open Source Software/Free Software, licensed under the GNU GPL. SLOCCount comes with ABSOLUTELY NO WARRANTY, and you are welcome to redistribute it under certain conditions as specified by the GNU GPL license; see the documentation for details. Please credit this data as "generated using David A. Wheeler's 'SLOCCount'." :-) -- Kind Regards, Gavin Henry. Managing Director. T +44 (0) 1224 279484 M +44 (0) 7930 323266 F +44 (0) 1224 824887 E ghenry(a)suretecsystems.com Open Source. Open Solutions(tm). http://www.suretecsystems.com/

1 0

Re: ldap_int_sasl_bind() and canonical Kerberos names
by Geert Jansen 25 Oct '07

25 Oct '07

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Simon, >> The patch unconditionally disables hostname canonicalisation for >> the sasl client. > > I think this will break GSSAPI connections to LDAP servers that are > behind DNS round robin style load balancers. > > Assume that you have 'ldap' that is a CNAME for ldap-1 and ldap2. > The LDAP library initiates a connection to 'ldap', and DNS points > it to 'ldap-1'. Providing you ask SASL to set up a connection to > 'ldap-1', you're fine (this is what the code does at the moment). > However, if you ask the SASL library for a connection to 'ldap' > (this is what your change does, as far as I can tell), and the > library does a canonicalisation step (as most Kerberos > implementations currently do), it will get 'ldap-2' back from the > DNS. So, you end up trying to negotiate a SASL connection with > 'ldap-2', when you're actually connected to 'ldap-1'. This tends > not to work. Thanks for the explanation and indeed I see that my patch would break this use case. I have come to the conclusion that one of the two reasons why I looked into this patch (do not keep realm data on the local system) is actually a problem that has nothing to do with canonicalisation. Realm referrals work just as well with the reverse-dns based canonical name as long as it is a valid principal. The other reason (be resilient against wrong reverse-dns setups) could still be a reason to use this patch, but now that I think of it it, it probably doesn't buy you much as you need proper forward dns anyway so better have a proper reverse dns too. Therefore I withdraw my patch. Regards, Geert -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.7 (GNU/Linux) Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org iQEVAwUBRyDy7prPkVOV4lDgAQJD9gf+KzsoMxBtCf0K0f80q6kWM+DAB64EpwCV g2Lj9JdKK0BcxcyXpnz+vBrKPwuP8RP/1dkvNiBJrgcUVc9Yo25H+UNBHkff0wUt QPZZvf9p/nxz0AQHAYrHdh94fM748y2LuMD/oVkpu+Oi8HeC5P5fo2VMpsoJ9pcg 9ee23yyuT3EyjpG3YGnApOOdAPgEqgUirvI+DibFYXo4hLrzwL5PKRmY3ggMZKa1 OrHz2qjZjvcktbs3cSU0v17tG+KLW1DtKaO80bSrbjAqb0l4rVPI+a6ixN5IEbG7 RdGKzx9jP4hgTP+Xt06e+eNFg19u0e72mrlzmH2A29C5RA2cHVsMRg== =bLmO -----END PGP SIGNATURE-----

1 0

Re: thread pools, performance
by Howard Chu 25 Oct '07

25 Oct '07

Rick Jones wrote: >> The ethernet controllers are on a hub attached by HyperTransport to a >> single processor, I don't think you can usefully distribute the >> interrupts to anything beyond that socket. > > Well, it wouldn't necessarily be goodness from the standpoint of the time to do > any PIO reads, but from the standpoint of getting packet processing spread-out > and matching with the core on which the application thread runs it might be. > All depends I guess on how busy that HT link gets I guess and whether the NIC(s) > can do the spreading in the first place. I wonder if there are any pre-packaged > linux binaries out there to measure HT link utilization? Haven't looked. Would be interesting to see. > >>> Although, if there is still 10% idle, that probably needs to go next :) >> >> Heh heh. The 80/20 rule hits this with a vengeance. That's "10%" of >> "800%" total, which means really only about 1.2% of a CPU, which is >> almost totally indistinguishable from measurement error in the oprofile >> results. This is all intuition (guesswork) now, no more obvious hot >> spots left to attack. Maybe if I'm really bored over the holidays I'll >> spend some time on it. (Not likely.) > > My degree is in AppliedMath which means I cannot do math to save my life, but if > it was 10%agepoints of 800%agepoints total, that means 10% of 8 cores or 80% of > a core doesn't it? No, there are 800 %agepoints total for all 8 cores, and 10 out of 800 are idle. That's a total of 10% of a single core, but it's actually distributed across all 8 cores. So 1.25% of each core. > If it were 10% of one core, then it would be 1.25% of 800% IIRC. Yes. > And besides, the nature of single-threading bottlenecks seems to be that when > you go to 16 cores it will be rather more than 2X what it is today :) Yes, well, I haven't got a 16-core test system handy at the moment... The changes I just checked in hit the easy stuff, reducing mutex contention by about 25% on a read-only workload with a nice corresponding boost in throughput. I haven't gotten to the multiple pools yet. The most obvious one was using per-thread lists for the Operation free lists instead of a single global free list. This assumes that threads will get work allocated relatively uniformly; I guess we can add some counts and push free Ops onto a global list if a per-thread list gets too long. At the moment there's no checking, so once a few operations have been performed Operation allocation occurs with pretty much zero mutex activity. (This is probably redundant effort when using something like Google tcmalloc, but not everyone is using that...) The other change was using per-thread slap_counters for statistics instead of just a global statistics block. In this case, the global slap_counters structure still exists as the head of the list, and its mutex is used to protect list traversals/manipulations. But actual live data is accumulated in per-thread structures chained off the global. The head mutex is locked when allocating a new per-thread structure, deallocating a structure for an exiting thread, and whenever back-monitor wants to tally up data to present to a client. In normal operation, once the configured number of threads have been created, there will be no accesses to the global at all. If cn=config is used to decrease the number of threads, then some number of threads will exit and so they'll lock the list at that time and accumulate their stats onto the head structure before destroying themselves. So again, in normal operation, if nobody is querying cn=monitor, there will be zero mutex contention due to statistics counters. Something else I've been considering is padding certain structures to sizes that are multiples of a given cache line. Not sure I want to write autoconf tests for that, may just arbitrarily choose 64 bytes and let folks override it at compile time. The Connection array is the most likely candidate here. Something like #define ALIGN 64 typedef real_struct { whatever } real_struct; typedef padded_struct { real_struct foo; char pad[(sizeof(real_struct)+ALIGN-1) & ~(ALIGN-1)]; } padded_struct; -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel