openldap-devel

openldap-devel@openldap.org

2 participants
1426 discussions

Re: pthread rdwr locks
by Howard Chu 10 Jan '07

10 Jan '07

Howard Chu wrote: > I just noticed that we're explicitly disabling support for native rdwr > locks in ldap_int_thread.h, and it's been that way since 2000. Any > reason not to use the native locks instead of our emulation routines? At least on Linux the native pthread locks are faster than our emulation. However, the glibc header files don't expose the phtread_rwlock_t definition by default, and getting the definition exposed involves a bit of pain. I haven't investigated on any other platforms. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

1 0

Re: BerkeleyDB performance on Linux
by Howard Chu 08 Jan '07

08 Jan '07

Rick Jones wrote: > Howard Chu wrote: >> The attached patch makes O_DIRECT work on Linux in BerkeleyDB 4.5.20. >> (You will need to manually define LINUX_NEEDS_PAGE_ALIGNMENT if you're >> using a kernel older than 2.6.) >> >> The main reason to use this patch is to conserve memory - ordinarily, >> all the I/O that BDB does to its files gets cached in the Linux >> filesystem buffer cache. This caching is redundant since BDB always >> does its own caching, and it effectively makes the BDB environment >> consume twice as much memory as it needs. Using O_DIRECT on I/Os >> disables the filesystem buffer cache for those I/Os, thus freeing up a >> sizable chunk of memory. > > Long ago and far away I was having "fun" with SPECweb99 on Linux and > found that eventually the access logs would grow and cast-out otherwise > usefully cached URLs. I then found the posix_fadvise call, which can be > used to tell the system that a certain range is no longer required: Yes, but posix_fadvise is not supported on Linux kernels before 2.6, returns ENOSYS. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

1 0

BerkeleyDB performance on Linux
by Howard Chu 07 Jan '07

07 Jan '07

The attached patch makes O_DIRECT work on Linux in BerkeleyDB 4.5.20. (You will need to manually define LINUX_NEEDS_PAGE_ALIGNMENT if you're using a kernel older than 2.6.) The main reason to use this patch is to conserve memory - ordinarily, all the I/O that BDB does to its files gets cached in the Linux filesystem buffer cache. This caching is redundant since BDB always does its own caching, and it effectively makes the BDB environment consume twice as much memory as it needs. Using O_DIRECT on I/Os disables the filesystem buffer cache for those I/Os, thus freeing up a sizable chunk of memory. The caching problem is particularly aggravated on Linux because the memory manager doesn't give program pages higher priority than cache pages. So when your system is tight on memory, the kernel will start swapping program data pages before it starts reclaiming buffer cache pages, and application performance plummets. (Possibly that indicates a kernel bug, or at least a misfeature.) Note that you must configure BerkeleyDB with --enable-o_direct to enable the support, and you must add "set_flags DB_DIRECT_DB" to your DB_CONFIG to enable it in a particular environment. With this patch, a slapd that occupies 6.8GB on a system with 8GB of RAM can run continuously without swapping, delivering a sustained 11,500 authentications per second. Without the patch, swapping starts when the process hits the 4.5GB mark (because over 3GB of buffer cache is in use), and performance drops to only *hundreds* of authentications per second. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/ --- dbinc/mp.h.orig 2006-09-07 14:31:58.000000000 -0700 +++ dbinc/mp.h 2007-01-06 19:14:56.000000000 -0800 @@ -378,6 +378,23 @@ #define BH_FREE_REUSE 0x02 #define BH_FREE_UNLOCKED 0x04 +#ifdef DIAG_MVCC +#define BH_ALIGNED +#define VM_PAGESIZE 4096 +#endif + +/* Linux O_DIRECT needs aligned buffers. 2.6 kernel allows 512 byte + * alignment, otherwise need page sized (4096). + */ +#if defined(linux) && !defined(BH_ALIGNED) +#define BH_ALIGNED +#ifdef LINUX_NEEDS_PAGE_ALIGNMENT +#define VM_PAGESIZE 4096 +#else /* Linux 2.6+ */ +#define VM_PAGESIZE 512 +#endif +#endif + /* * BH -- * Buffer header. @@ -404,7 +421,7 @@ roff_t td_off; /* MVCC: creating TXN_DETAIL offset. */ SH_CHAIN_ENTRY vc; /* MVCC: version chain. */ -#ifdef DIAG_MVCC +#ifdef BH_ALIGNED u_int16_t align_off; /* Alignment offset for diagnostics.*/ #endif @@ -465,15 +482,14 @@ (dbc->txn != NULL && F_ISSET(dbc->txn, TXN_SNAPSHOT) && \ dbc->txn->td != NULL && __memp_skip_curadj(dbc, pgno)) -#if defined(DIAG_MVCC) && defined(HAVE_MPROTECT) -#define VM_PAGESIZE 4096 -#define MVCC_BHSIZE(mfp, sz) do { \ +#ifdef BH_ALIGNED +#define BHSIZE(mfp, sz) do { \ sz += VM_PAGESIZE + sizeof(BH); \ if (mfp->stat.st_pagesize < VM_PAGESIZE) \ sz += VM_PAGESIZE - mfp->stat.st_pagesize; \ } while (0) -#define MVCC_BHALIGN(mfp, p) do { \ +#define BHALIGN(mfp, p) do { \ if (mfp != NULL) { \ BH *__bhp; \ void *__orig = (p); \ @@ -493,13 +509,19 @@ } \ } while (0) -#define MVCC_BHUNALIGN(mfp, p) do { \ +#define BHUNALIGN(mfp, p) do { \ if ((mfp) != NULL) { \ BH *bhp = (BH *)(p); \ (p) = ((u_int8_t *)bhp - bhp->align_off); \ } \ } while (0) +#else +#define BHSIZE(mfp, sz) do {} while (0) +#define BHALIGN(mfp, p) do {} while (0) +#define BHUNALIGN(mfp, p) do {} while (0) +#endif +#if defined(DIAG_MVCC) && defined(HAVE_MPROTECT) #ifdef linux #define MVCC_MPROTECT(buf, sz, mode) do { \ int __ret = mprotect((buf), (sz), (mode)); \ @@ -513,11 +535,7 @@ } \ } while (0) #endif /* linux */ - -#else /* defined(DIAG_MVCC) && defined(HAVE_MPROTECT) */ -#define MVCC_BHSIZE(mfp, sz) do {} while (0) -#define MVCC_BHALIGN(mfp, p) do {} while (0) -#define MVCC_BHUNALIGN(mfp, p) do {} while (0) +#else #define MVCC_MPROTECT(buf, size, mode) do {} while (0) #endif --- mp/mp_alloc.c.orig 2006-09-07 14:32:03.000000000 -0700 +++ mp/mp_alloc.c 2007-01-06 19:14:56.000000000 -0800 @@ -66,7 +66,7 @@ if (mfp != NULL) { len = SSZA(BH, buf) + mfp->stat.st_pagesize; /* Add space for alignment padding for MVCC diagnostics. */ - MVCC_BHSIZE(mfp, len); + BHSIZE(mfp, len); } MPOOL_REGION_LOCK(dbenv, infop); @@ -91,10 +91,10 @@ c_mp->stat.st_pages++; MPOOL_REGION_UNLOCK(dbenv, infop); /* - * For MVCC diagnostics, align the pointer so that the buffer + * If necessary, align the pointer so that the buffer * starts on a page boundary. */ - MVCC_BHALIGN(mfp, p); + BHALIGN(mfp, p); found: if (offsetp != NULL) *offsetp = R_OFFSET(infop, p); @@ -447,7 +447,7 @@ MPOOLFILE *mfp; void *buf; { - MVCC_BHUNALIGN(mfp, buf); + BHUNALIGN(mfp, buf); COMPQUIET(mfp, NULL); __db_shalloc_free(infop, buf); } --- mp/mp_fget.c.orig 2006-09-13 09:22:42.000000000 -0700 +++ mp/mp_fget.c 2007-01-06 19:14:56.000000000 -0800 @@ -708,7 +708,7 @@ * the hash bucket's priority. */ /*lint --e{668} (flexelint: bhp cannot be NULL). */ -#ifdef DIAG_MVCC +#ifdef BH_ALIGNED memset(bhp, 0, SSZ(BH, align_off)); #else memset(bhp, 0, sizeof(BH));

2 6

LDAP Sync API
by Pierangelo Masarati 06 Jan '07

06 Jan '07

It's now in HEAD; typedef's, data structure and function declarations are in include/ldap.h; the bulk of the code is in libraries/libldap/ldap_sync.c. A man page is coming, although there might be need to freeze the API first. p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.n.c. Via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ------------------------------------------ Office: +39.02.23998309 Mobile: +39.333.4963172 Email: pierangelo.masarati(a)sys-net.it ------------------------------------------

1 0

Re: better malloc strategies
by Howard Chu 05 Jan '07

05 Jan '07

Howard Chu wrote: > I've tested again with libhoard 3.5.1, and it's actually superior to > libumem for both speed and fragmentation. Here are some results against > glibc 2.3.3, libhoard, and libumem: I've retested with current HEAD, adding Google's tcmalloc to the mix. glibc hoard umem tcmalloc size time size time size time size time initial 731 741 741 744 single 1368 2:03.02 1809 1:26.67 1778 1:43.49 1453 0:58.77 single 1369 2:23.61 1827 0:25.68 1825 0:18.40 1512 0:21.74 single 1368 1:48.41 1828 0:16.52 1826 0:21.07 1529 0:22.97 single 1368 1:48.59 1829 0:16.59 1827 0:16.95 1529 0:17.07 single 1368 1:48.72 1829 0:16.53 1827 0:16.61 1529 0:17.01 single 1368 1:48.39 1829 0:20.70 1827 0:16.56 1529 0:16.99 single 1368 1:48.63 1830 0:16.56 1828 0:17.48 1529 0:17.29 single 1384 1:48.14 1829 0:16.64 1828 0:22.17 1529 0:16.94 four 1967 1:20.21 1918 0:35.96 1891 0:29.95 1606 0:42.48 four 2002 1:10.58 1919 0:30.07 1911 0:29.00 1622 0:42.38 four 2009 1:33.45 1920 0:42.06 1911 0:40.01 1628 0:40.68 four 1998 1:32.94 1920 0:35.62 1911 0:39.11 1634 0:30.41 four 1995 1:35.47 1920 0:34.20 1911 0:28.40 1634 0:40.80 four 1986 1:34.38 1920 0:28.92 1911 0:31.16 1634 0:40.42 four 1989 1:33.23 1920 0:31.48 1911 0:33.73 1634 0:33.97 four 1999 1:33.04 1920 0:33.47 1911 0:38.33 1634 0:40.91 slapd CPU 26:31.56 8:34.78 8:33.39 9:19.87 > The initial size is the size of the slapd process right after startup, > with the process totally idle. The id2entry DB is 1.3GB with about > 360,000 entries, BDB cache at 512M, entry cache at 70,000 entries, > cachefree 7000. The subsequent statistics are the size of the slapd > process after running a single search filtering on an unindexed > attribute, basically spanning the entire DB. The entries range in size > from a few K to a couple megabytes. The BDB and slapd cache configurations are the same, but the machine now has 4GB of RAM so the entire DB fits in the filesystem buffer cache. As such there is no disk I/O during these tests. > > After running the single ldapsearch 4 times, I then ran the same search > again with 4 jobs in parallel. There should of course be some process > growth for the resources for 3 additional threads (about 60MB is about > right since this is an x86_64 system). This time I ran the tests 8 times each. Basically I was looking for the slapd process size to stabilize at a constant number... > > The machine only had 2GB of RAM, and you can see that with glibc malloc > the kswapd got really busy in the 4-way run. The times might improve > slightly after I add more RAM to the box. But clearly glibc malloc is > fragmenting the heap like crazy. The current version of libhoard looks > like the winner here. There was no swap activity (or any other disk activity) this time, so the glibc numbers don't explode like they did before. But even so, the other allocators are 5-6 times faster. The Google folks claim tcmalloc is the fastest allocator they have ever seen. These tests show it is fast, but it is not the fastest. It definitely is the most space-efficient multi-threaded allocator though. It's hard to judge just by the execution times of the ldapsearch commands, so I also recorded the amount of CPU time the slapd process consumed by the end of each test. That gives a clearer picture of the performance differences for each allocator. These numbers aren't directly comparable to the ones I posted on August 30, because I was using a hacked up RE23 there and used HEAD here. Going thru these tests is a little disturbing; I would expect the single-threaded results to be 100% repeatable but they're not quite. In one run I saw glibc use up 2.1GB during the 4-way test, and never shrink back down. I later re-ran the same test (because I hadn't recorded the CPU usage the first time around) and got these numbers instead. The other thing that's disturbing is just how bad glibc's malloc really is, even in the single-threaded case which is supposed to be its ideal situation. The other thing to note is that for this run with libhoard, I doubled its SUPERBLOCK_SIZE from 64K to 128K. That's probably why it's less space-efficient here than umem, while it was more efficient in the August results. I guess I'll have to recompile that with the original size to see what difference that makes. For now I'd say my preference would be tcmalloc... -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

2 10

ldap_start_tls(), ldap_install_tls() and async
by Frank Cusack 03 Jan '07

03 Jan '07

ldap_start_tls() is async, but ldap_install_tls() appears to send stuff over the network. (I get errors on the server side about unknown protocol if my client app does not call ldap_install_tls(), so I'm just guessing that ldap_install_tls() sends network traffic.) So the problem is that ldap_install_tls() is not async and does not accept a timeout parameter. I guess that ldap_install_tls() does some crypto stuff as well, eg obtaining random data which might block. So doesn't ldap_install_tls() need to be async in order to make the start tls operation as a whole async? Or am I missing some information. thanks -frank

1 0

MSYS, Windows
by Howard Chu 02 Jan '07

02 Jan '07

Mostly off-topic, but may be of some interest... I recently noticed while running the test suite on Windows with MSYS that the server processes were not getting killed cleanly, they were just being terminated abruptly. This tended to do weird things for subsequent tests, showing complaints about not being able to delete the testrun directory for a prior test because it wasn't empty. I found that one way to work around this problem was to invoke Cygwin's "sync" command after each test. (Added into tests/scripts/all.) In some cases it would cause a test to fail that would otherwise succeed when run on its own. But the actual problem is that the MSYS DLL doesn't propagate signals to non-MSYS apps. And it turns out, I had actually noticed this back in 2003 and subsequently forgotten. http://news.gmane.org/find-root.php?group=gmane.comp.gnu.mingw.msys&article… Anyway, I've written a patch for the MSYS DLL which you can find here http://sourceforge.net/tracker/index.php?func=detail&aid=1624635&group_id=2… and I have a binary you can download here http://highlandsun.com/hyc/msys-1.0.dll.gz cksum: 2315022199 326195 msys-1.0.dll.gz It's based off a recent pull from their CVS, so roughly version 1.0.11. It also includes both of these patches http://sourceforge.net/tracker/index.php?func=detail&aid=1174106&group_id=2… If you're doing a lot of work with OpenLDAP in a Windows environment this will definitely make life more tolerable. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc OpenLDAP Core Team http://www.openldap.org/project/

1 0

Re: commit: ldap/servers/slapd/overlays accesslog.c
by Kurt D. Zeilenga 26 Dec '06

26 Dec '06

It might be more appropriate to handle this issue on the consumer than the provider. An arbitrary LDAP sync client might want this and other DSA specific attributes included in the content. That is, the provider should not assume the consumer is doing server-to-server replication. Kurt At 01:10 PM 12/23/2006, hyc(a)OpenLDAP.org wrote: >Update of /repo/OpenLDAP/pkg/ldap/servers/slapd/overlays > >Modified Files: > accesslog.c 1.68 -> 1.69 > >Log Message: >ITS#4788 don't return auditContext attr on syncrepl searches > > >CVS Web URLs: > http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/overlays/ > http://www.openldap.org/devel/cvsweb.cgi/servers/slapd/overlays/accesslog.c > >Changes are generally available on cvs.openldap.org (and CVSweb) >within 30 minutes of being committed.

4 7

Privileged connections in back-ldap/back-meta
by Pierangelo Masarati 24 Dec '06

24 Dec '06

I'm facing the problem of having too many threads operating on the same privileged connections in back-ldap/back-meta, which seems to cause poor scalability. This mainly occurs because we want all operations to be performed as a privileged identity with proxyAuthz. For this purpose I plan to move privileged connections out of the connection AVL tree into ldapinfo_t, so that we save a search in the tree, and keep multiple instances of privileged connections in a LDAP_TAILQ structure of fixed maximum length to use them round-robin. Each type of privileged connection uses a dedicated list, so in our case we basically keep two lists: one for the privileged user and one for binds, which are now serialized into one privileged connection since all subsequent operations use proxyAuthz. A prototype is being tested, which doesn't work too bad (scalability did not dramatically improve yet, so there might be other bottlenecks, but at least some repeatable transient hangs disappeared). Comments? p. Ing. Pierangelo Masarati OpenLDAP Core Team SysNet s.n.c. Via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ------------------------------------------ Office: +39.02.23998309 Mobile: +39.333.4963172 Email: pierangelo.masarati(a)sys-net.it ------------------------------------------

2 1

Re: Lifetime and Documenting of LDAP_DEPRECATED, outside ldap.h
by Kurt D. Zeilenga 20 Dec '06

20 Dec '06

At 06:35 AM 12/20/2006, Gavin Henry wrote: >> Is there a date set that LDAP_DEPRECATED will not be supported anymore? No. >> And should we (docs team) put something on Faq-O-Matic (devel section), or >> in the new devel guide (once an outline/template has been put in place) >> that presents the "Best Practice" for updating their code? Personally, I think it would be better to focus the limited documentation time/energy we have toward documenting non-deprecated code (and discussing how to write clients using non-deprecated code). But as I've noted before, (code and documentation) developers should work on what they need/want... and, if its generally useful, it will find its way into OpenLDAP Software (and/or other materials provided by the Project). That is, don't let my (or other statements) stop anyone from working on what they need/want. -- Kurt

2 1

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel