November 2006 - openldap-devel

by Howard Chu

Howard Chu wrote: > I've tested again with libhoard 3.5.1, and it's actually superior to > libumem for both speed and fragmentation. Here are some results against > glibc 2.3.3, libhoard, and libumem: I've retested with current HEAD, adding Google's tcmalloc to the mix. glibc hoard umem tcmalloc size time size time size time size time initial 731 741 741 744 single 1368 2:03.02 1809 1:26.67 1778 1:43.49 1453 0:58.77 single 1369 2:23.61 1827 0:25.68 1825 0:18.40 1512 0:21.74 single 1368 1:48.41 1828 0:16.52 1826 0:21.07 1529 0:22.97 single 1368 1:48.59 1829 0:16.59 1827 0:16.95 1529 0:17.07 single 1368 1:48.72 1829 0:16.53 1827 0:16.61 1529 0:17.01 single 1368 1:48.39 1829 0:20.70 1827 0:16.56 1529 0:16.99 single 1368 1:48.63 1830 0:16.56 1828 0:17.48 1529 0:17.29 single 1384 1:48.14 1829 0:16.64 1828 0:22.17 1529 0:16.94 four 1967 1:20.21 1918 0:35.96 1891 0:29.95 1606 0:42.48 four 2002 1:10.58 1919 0:30.07 1911 0:29.00 1622 0:42.38 four 2009 1:33.45 1920 0:42.06 1911 0:40.01 1628 0:40.68 four 1998 1:32.94 1920 0:35.62 1911 0:39.11 1634 0:30.41 four 1995 1:35.47 1920 0:34.20 1911 0:28.40 1634 0:40.80 four 1986 1:34.38 1920 0:28.92 1911 0:31.16 1634 0:40.42 four 1989 1:33.23 1920 0:31.48 1911 0:33.73 1634 0:33.97 four 1999 1:33.04 1920 0:33.47 1911 0:38.33 1634 0:40.91 slapd CPU 26:31.56 8:34.78 8:33.39 9:19.87 > The initial size is the size of the slapd process right after startup, > with the process totally idle. The id2entry DB is 1.3GB with about > 360,000 entries, BDB cache at 512M, entry cache at 70,000 entries, > cachefree 7000. The subsequent statistics are the size of the slapd > process after running a single search filtering on an unindexed > attribute, basically spanning the entire DB. The entries range in size > from a few K to a couple megabytes. The BDB and slapd cache configurations are the same, but the machine now has 4GB of RAM so the entire DB fits in the filesystem buffer cache. As such there is no disk I/O during these tests. > > After running the single ldapsearch 4 times, I then ran the same search > again with 4 jobs in parallel. There should of course be some process > growth for the resources for 3 additional threads (about 60MB is about > right since this is an x86_64 system). This time I ran the tests 8 times each. Basically I was looking for the slapd process size to stabilize at a constant number... > > The machine only had 2GB of RAM, and you can see that with glibc malloc > the kswapd got really busy in the 4-way run. The times might improve > slightly after I add more RAM to the box. But clearly glibc malloc is > fragmenting the heap like crazy. The current version of libhoard looks > like the winner here. There was no swap activity (or any other disk activity) this time, so the glibc numbers don't explode like they did before. But even so, the other allocators are 5-6 times faster. The Google folks claim tcmalloc is the fastest allocator they have ever seen. These tests show it is fast, but it is not the fastest. It definitely is the most space-efficient multi-threaded allocator though. It's hard to judge just by the execution times of the ldapsearch commands, so I also recorded the amount of CPU time the slapd process consumed by the end of each test. That gives a clearer picture of the performance differences for each allocator. These numbers aren't directly comparable to the ones I posted on August 30, because I was using a hacked up RE23 there and used HEAD here. Going thru these tests is a little disturbing; I would expect the single-threaded results to be 100% repeatable but they're not quite. In one run I saw glibc use up 2.1GB during the 4-way test, and never shrink back down. I later re-ran the same test (because I hadn't recorded the CPU usage the first time around) and got these numbers instead. The other thing that's disturbing is just how bad glibc's malloc really is, even in the single-threaded case which is supposed to be its ideal situation. The other thing to note is that for this run with libhoard, I doubled its SUPERBLOCK_SIZE from 64K to 128K. That's probably why it's less space-efficient here than umem, while it was more efficient in the August results. I guess I'll have to recompile that with the original size to see what difference that makes. For now I'd say my preference would be tcmalloc... -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

16 years, 11 months

2
10
0 / 0

Cancel operation in RE23?

by Michael Ströder

HI! Is the cancel operation already supported in RE23? It seems so (playing around with it with python-ldap). But supportedFeatures in root DSE does not contain 1.3.6.1.1.8. RFC 3909 states: Servers SHOULD indicate their support for this extended operation by providing 1.3.6.1.1.8 as a value of the 'supportedExtension' attribute type in their root DSE. A server MAY choose to advertise this extension only when the client is authorized to use it. Ciao, Michael.

16 years, 11 months

2
2
0 / 0

LDIF parser performance (was: write performance)

by Michael Ströder

Howard, (Cc:-ed openldap-devel(a)OpenLDAP.org in opposite to our off-line conversation). Howard Chu wrote: > I'm wondering if it's worth the effort > to rewrite the client's LDIF parser as I did for slapadd -q. As I said I cannot test on the machine where I did the original tests. But I tried to test OpenLDAP's LDIF parser with -n. $ time ldapadd -f test.ldif -n [..] real 1m30.402s user 1m29.090s sys 0m0.376s Now I have a small Python script which uses the module 'ldif' from python-ldap for reading in the LDIF file. I've implemented module 'ldif' in pure Python but off course the string module in the underlying Python standard lib is implemented in C. And the Python runtime environment does all the ugly memory management. :-) $ time python count_members.py < test.ldif [..] real 0m19.145s user 0m18.349s sys 0m0.568s I re-ran the tests twice, so test.ldif should have been in the filesystem cache. How does that sound to you? I'm not sure what different actions ldapadd -n does in comparison to my simple script. But at least count_members.py also reads the complete entries into a tuple containing the DN as string and the entry as so-called dictionary. Ciao, Michael.

17 years

3
8
0 / 0

Re: commit: ldap/clients/tools ldapmodify.c

by Howard Chu

hyc(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/clients/tools > > Modified Files: > ldapmodify.c 1.190 -> 1.191 > > Log Message: > Reject malformed LDIF - missing modops, or mismatched attributes With this patch test021 fails while adding certificates to the directory, because the provided LDIF uses e.g.: add: usercertificate usercertificate;binary:: xxxxxx Should this really be allowed? There are other ops in the LDIF that actually provide the ";binary" tag in both places so it seems that's the expected behavior. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

17 years

2
1
0 / 0

A strange problem while using variable allocated by ber_memalloc_x().

by Chen, Shi Jun SLC CT PEK

I encount a strange problem while using variable allocated by ber_memalloc_x(). For each operation request, slapd will malloc a buffer to contain the DN and NDN by using ber_memalloc_x() for fields o_req_dn & o_req_ndn in struct operation. Now I want to rewrite the DN and NDN myself, and below is the code. I always get segment fault while running it. But when I set MALLOC_CHECK_, the code does work and will not be crashed. So what's the matter of my code? -Shijun Chen- =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ ... wrap_dn(op->o_req_dn, op->o_tmpmemctx); ... wrap_dn(struct berval *dn, void *ctx) { char *tmp_buf=NULL, *psub=NULL; int len_tmp_buf=0; lem_tmp_buf = dn->bv_len + 5; dn->bv_val = LDAP_REALLOCX(dn->bv_val, len_tmp_buf, ctx); dn->bv_len = len_tmp_buf; tmp_buf = malloc(len_tmp_buf + 1); memset(tmp_buf, 0, len_tmp_buf + 1); memcpy(tmp_buf, dn->bv_val, len_tmp_buf); psub = strcasestr(tmp_buf, "dc=com"); assert( psub != NULL); sprintf(p_sub, "dc=1,%s", "dc=com"); memcpy(dn->bv_val, tmp_buf, len_tmp_buf); free(tmp_buf); return; } =+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+

17 years

2
1
0 / 0

Re: LDIF parser performance

by Howard Chu

Michael Ströder wrote: > Howard Chu wrote: >> Try this one instead, I've cleaned up most of that cruft. > > Congratulations! Any chance to see this soon in RE23? The ldapadd code path is solid. The ldapmodify code path is a bit more questionable, mostly because of the ambiguities in the RFC. Let's see what other feedback we get from testing in HEAD... > > It's *much* faster: > > real 0m1.651s > user 0m1.144s > sys 0m0.104s > > instead of: > > real 1m31.305s > user 1m29.730s > sys 0m0.360s > > Ciao, Michael. > > . > -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

17 years

1
0
0 / 0

Re: commit: ldap/servers/slapd proto-slap.h controls.c operation.c

by Howard Chu

ando(a)OpenLDAP.org wrote: > Update of /repo/OpenLDAP/pkg/ldap/servers/slapd > > Modified Files: > proto-slap.h 1.695 -> 1.696 > controls.c 1.181 -> 1.182 > operation.c 1.76 -> 1.77 > > Log Message: > destroy cached groups after identity change (ITS#4760) I think the groups should not even have been cached in the first place; lookups for auth purposes usually set op->o_nocaching. > > CVS Web URLs: > http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/ > http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/proto-slap.h > http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/controls.c > http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/operation.c > > Changes are generally available on cvs.openldap.org (and CVSweb) > within 30 minutes of being committed. > > . > -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

17 years

2
4
0 / 0

authz-policy and rootDSE

by Michael Ströder

HI! If slapd.conf contains authz-policy none should proxied authorization (2.16.840.1.113730.3.4.18) be advertised in 'supportedControl' of rootDSE? IMHO it shouldn't appear. Ciao, Michael.

17 years

2
1
0 / 0

Asyncronous connect(2) in client library?

by Pierangelo Masarati

The client library seems to allow pretty much asynchronous operations except for connect(2), and I've been requested if it's possible to remove that limitation (?). I see there is code in libraries/libldap/os-ip.c to kinda deal with asynchronous connect(2), but it's protected behind #ifdef notyet lasting since first import from UMIch's ldap-3.3 ('98); does anybody see any issue in working at that that I'm overlooking? p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.n.c. Via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ------------------------------------------ Office: +39.02.23998309 Mobile: +39.333.4963172 Email: pierangelo.masarati(a)sys-net.it ------------------------------------------

17 years, 1 month

2
1
0 / 0

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel November 2006