openldap-devel

openldap-devel@openldap.org

2 participants
1419 discussions

MDB slapadd potential improvements
by Howard Chu 15 Apr '12

15 Apr '12

I have a "thread" branch on ssh://ada.openldap.org/~hyc/OD/head/.git with some preliminary patches to improve multi-threaded indexing in back-mdb slapadd. So far the scale of improvement is small, but there may be ways to enhance it further from here. Currently back-mdb's multi-threaded indexing code is largely copied from back-bdb/hdb, and it is always slower than regular single-threaded slapadd. The slowdown comes from a number of reasons. 1) Thread synchronization overhead is huge. Much greater than the actual cost of index processing. 2) Index processing is awkward since MDB doesn't support multi-threaded writes (while BDB does). 1) is because we're using cond_wait/cond_broadcast to perform synchronization, and that requires every waiting thread to successfully acquire the condition mutex before progressing any further. So it is inherently serial, even using cond_broadcast. Also, we're doing this twice, once to start all of the index processing for an entry, and once at the end of index processing. The latter one is required so that we will know it's safe to dispose of the current entry and move on to the next. The new code uses a single pthread_barrier for synchronization, and it is only used at the start of index processing. Index cleanup is deferred, and all of the relevant data is double-buffered. This allows us to only need to worry about the start of processing; the processing can take as long as it needs. This restructure requires a small change in slapadd as well, to maintain two Entry pointers instead of just one, and free them in a staggered fashion. The back-bdb/hdb threaded indexer can also be restructured along these lines for a similar benefit. 2) Index results were being gathered in several malloc'd structures and then individually freed after being written to the DB. Now I'm using sl_malloc, and simply resetting the memctx between entries, thus eliminating all cost of free() operations. These changes are sufficient to bring multithreaded performance down to just on par with single-threaded slapadd. In fact there's very little to gain here when all of the DB writes are still single-threaded. One further tweak: most of our index keys are 4-byte hash values. Using the MDB_INTEGERKEY flag allows them to be compared word-at-a-time instead of byte-at-a-time, which gives a further speedup. At present this is the most significant improvement. With the original back-mdb code currently in git master, slapadd of our test LDIF (4.9GB, 6326513 entries, 31 indexed attributes) was taking 1h50m on ada. (Using -q. Without -q, 2h46m.) With no indexing, it takes only 9m11s. Even though sizewise indices don't account for much, they cost a lot in numbers of keys. 6M entries means 6M keys in the id2entry DB and 12M keys in the dn2id DB. Indexing on 31 attributes with multiple values, substrings, and other such parameters thrown in, amounts to hundreds of millions of keys, and each of these needs multiple comparisons to be inserted into their proper index. I tried a further hack on MDB_APPEND to eliminate some more of this overhead. Currently, MDB_APPEND only affects the behavior of mdb_page_split, causing the newly added key to simply be added as the first node of a new page rather than splitting the existing page in half and copying half of the keys to the new page. (This in itself is a pretty major speedup for slapadd, but obviously already factored in.) The new code also simply appends new keys to the end of the DB's last page (rather than searching for its insertion point), which again is useful for eliminating unnecessary comparisons. This is only effective when adding an entryID to an already existing index key. This brought the slapadd -q time down to 1h46m. Turning on the MDB_INTEGERKEY flag brought the slapadd -q time down to 1h32m. Adding the threading rewrite to that, the time came down to 1h29m with tool-threads set to 4. Using more threads actually slows things down, again because thread synchronization becomes too expensive. Only the index key generation is being done in the extra threads, and the reality is that index key generation is not a very significant cost in the overall workload. But anyway, some of the work done here can be ported back into back-bdb/hdb to improve things there. In particular, the use of two synchronization events per entry can be reduced to one by adding double-buffering there too. And some malloc/free overhead in the index threads can be removed by using sl_malloc in each thread. Of course we first need to add pthread_barrier detection to the configure script, as well as wrapper functions in libldap_r. (Anyone interested in handling this? And writing the compatibility code for Windows?) In the meantime, I'm considering an MDB environment option to support multiple threads in a single write TXN, as long as each thread is operating on separate databases. This would hopefully allow us to further distribute the load of indexing without adding too much new complexity to libmdb. If you have an account on ada.openldap.org I encourage you to checkout this code and think about what's suitable to merge back into master. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 1

Move DDS-specific schema out of slap_schema?
by masarati＠aero.polimi.it 12 Apr '12

12 Apr '12

Although published in an RFC, I think DDS-specific schema (entryTtl, dynamicSubtrees) could be moved outside slap_schema into slapo-dds(5) code, although this would break binary compatibility. Objections? p.

1 0

mdb meta pages
by Hallvard Breien Furuseth 12 Apr '12

12 Apr '12

Could writing a word/byte to the current meta page break someting? While I'm asking, why are metas separate pages, instead of simply a fixed 256 or so bytes apart to keep them in separate cachelines? The only reason I can think of is if a write gets garbled, the other meta page is safe - but mdb assumes correct filesystem operation anyway. This is for a "syncdelay <count>" feature to replace "dbnosync". The latter can break DB consistency after a system crash: Without fdatasync(), the OS can reorder writes, leaving meta pages to refer to trees with not written or overwritten data pages. syncdelay <count> will only sync every <count> or maybe <count>/2 commit. It'll need 4 meta pages, of which 2 may refer to unsynced data pages. mdb_env_sync() may then need to write a "synced" flag to the current meta page, or do a dummy write transaction which sets with a "synced data pages" flag in its meta page. The latter would have to wait out any existing/pending write transactions. -- Hallvard

2 1

Re: (ITS#7182) back-ldap monitoring improvements
by Ondrej Kuznik 12 Apr '12

12 Apr '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 04/04/2012 03:26 PM, hyc(a)symas.com wrote: > See ITS#7225. > > Even after the SEGV is patched, the pcache slapd fails to start in test020 > because the monitor calls are unable to register their entries. Something is > wrong in your initialization code. When the ldap database is opened before the monitor, all registration is deferred and fake OK status is returned. Then, while the subsystems are initializing, the new subsystems (that have to derive their DN from something) do not yet have access to the DN of the database monitor entry (even though it may already exist). Only after all subsystems have been opened does the deferred database registration proceed, but by then there is no way to notify anyone that the DN has been published. Thus we are left with a chicken-and-egg problem. One way out of this I currently see is providing a register_database callback that is called immediately when no deferring takes place or immediately after the entry in limbo_queue has been processed. This way once the DN information is ready, the subsystem registration can start safely. There is an alternative, adding a defferable monitor_subsys registration (and alimbo entry type) that, if monitor is not opened, takes place in the second phase (roughly at the same time as above). There are aspects in which either one is cleaner, whereas the other one can backfire/clutter the API and vice versa for others. I have not decided yet. > Also, you're using monitor_entry_stub() directly in the back-ldap code. It is > forbidden for modules to directly reference each other's symbols. If you need > this function, add it to monitor_extra_t first and call it through there. Thank you, I will. And sorry for breaking master with this ITS. - -- Ondrej Kuznik -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk99tAoACgkQ9GWxeeH+cXvgsACdGx7/7Wp1h/Ye5voKcQhHDVPa FXQAnAodz5i9e6ReWwgkeNPbPsxQhN/V =FTRg -----END PGP SIGNATURE----- This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

2 2

TLS shutdown, order of operations
by Jan Vcelak 07 Apr '12

07 Apr '12

Hello list. I was told by Mozilla NSS developers, that OpenLDAP is shutting down the TLS context incorrectly. The order of the operations is incorrect at least for MozNSS backend. As this part of the code is common for all TLS backends in OpenLDAP, I would like to be sure about the way of fixing it. We think, that ber_sockbuf_free() should be called before ldap_int_tls_destroy() in ldap_ld_free(). Currently, the order is opposite. This is the original reasoning from John Dennis: > In other words what is happening is an SSL socket is created using an NSS > context (i.e. the NSS context must exist prior to creating the SSL socket). > But when the socket is disposed of the order of operations isn't mirrored in > reverse. The socket should be disposed of first because it belongs to the > NSS context, then the NSS context should be destroyed. But openldap is > performing the opposite, it's destroying the NSS context first and then the > socket. I'm not familiar with OpenSSL and GnuTLS. And I haven't found any information about ordering the shutdown operations in theirs documentation. I just went through the OpenLDAP code for other backends and my opinion is, that this problem is related to all backends. IMO the socket should be really shut down and closed before destroying the context with opened certificates and other resources. But maybe I'm wrong. Please, can somebody confirm or invalidate this? Thanks and regards! Jan

2 1

RE24 testing call #1 (2.4.31)
by Quanah Gibson-Mount 06 Apr '12

06 Apr '12

If you know how to build OpenLDAP manually, and would like to participate in testing the next set of code for the 2.4.31 release, please do so. Generally, get the code for RE24: <http://www.openldap.org/devel/gitweb.cgi?p=openldap.git;a=snapshot;h=refs/h…> Configure & build. Execute the test suite (via make test) after it is built. Thanks! --Quanah -- Quanah Gibson-Mount Sr. Member of Technical Staff Zimbra, Inc A Division of VMware, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration

2 1

Re: (ITS#7182) back-ldap monitoring improvements
by Ondrej Kuznik 28 Mar '12

28 Mar '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/28/2012 04:26 PM, ondrej.kuznik(a)acision.com wrote: > I have prepared some patches to back-ldap (and one to back-monitor) that expose > operation and connection monitoring information from a running back-ldap > database and I would like to see this or similar functionality included in the > OpenLDAP codebase. > > The url above contains a patchset against HEAD for review. > > The following things are yet to be resolved: > - there is no monitoring of completed operations yet, only operations initiated > against the remote database(s). > - the binds performed by the ldap_back_default_rebind function are not counted > - slapo-chain has not been tested yet After playing with slapo-chain, looks like the monitoring support has been a noop and properly enabling it might take a more intrusive patch than the one included above. > - test suite integration: back-ldap looks excluded from the test suite Looking further at the test suite, enabling back-ldap testing might also be a little more effort, maybe out of scope of such an ITS. > - better connection handling (connections should have stable identifiers) So far, I have had no revelation on how to proceed on this one. Pierangelo and others, could some of you take a look at the proposed changes and comment on what else should be improved or fixed? - -- Ondrej Kuznik -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9XjowACgkQ9GWxeeH+cXvwgwCgo4SGGBhVCYLcx6wcRI3kkXDW uDkAnROsOqTPeEDVL/tDXnva1sX9yM1Y =zSS9 -----END PGP SIGNATURE----- This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

3 4

Open Position
by kurt＠OpenLDAP.org 26 Mar '12

26 Mar '12

I would like to take this time to welcome you to our hiring process and give you a brief synopsis of the position's benefits and requirements. If you are taking a career break, are on a maternity leave, recently retired or simply looking for some part-time job, this position is for you. Occupation: Flexible schedule 2 to 8 hours per day. We can guarantee a minimum 20 hrs/week occupation Salary: Starting salary is 2000 EUR per month plus commission, paid every month. Business hours: 9:00 AM to 5:00 PM, MON-FRI, 9:00 AM to 1:00 PM SAT or part time (Europe time). Region: Europe. Please note that there are no startup fees or deposits to start working for us. To request an application form, schedule your interview and receive more information about this position please reply to Terrell(a)jobdayseu.com,with your personal identification number for this position IDNO: 7830

1 0

slapmodify and back-config
by Ondrej Kuznik 19 Mar '12

19 Mar '12

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, for slapmodify to be usable on a database, that database needs to implement the bi_tool_dn2id_get and bi_tool_entry_modify functions, which is not the case of the back-config. I've put together a minimal (and maybe too naive) implementation of these for back-config and back-ldif here: ftp://ftp.openldap.org/incoming/Ondrej-Kuznik-20120319-back-config-slapmodi… Currently, there is no validation whether the modification passes back-config checks since config_modify_internal takes a modify operation, while bi_tool_entry_modify receives only the new version of the entry to be modified. Also, for slapmodify to be really useful, it would have to allow entry deletion, for which there is no tool mode callback. To implement deletes, do you think it is better that deletion of an entry with children fail or delete the entire subtree? While the former seems both easier to implement and more sensible to me, I would like your know opinion. The attached file is derived from OpenLDAP Software. All of the modifications to OpenLDAP Software represented in the following patch(es) were developed by Acision. Acision has not assigned rights and/or interest in this work to any party. I, Ondrej Kuznik am authorized by Acision, my employer, to release this work under the following terms. The attached modifications to OpenLDAP Software are subject to the following notice: Copyright 2012 Acision Redistribution and use in source and binary forms, with or without modification, are permitted only as authorized by the OpenLDAP Public License. - -- Ondrej Kuznik -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk9m6z8ACgkQ9GWxeeH+cXuHIgCfdeRlNgd+o3+hcO8a23zPBPIz D4kAniBVWmDDdD/fubrYKHpmIfUFZ9T/ =9U3w -----END PGP SIGNATURE----- This e-mail and any attachment is for authorised use by the intended recipient(s) only. It may contain proprietary material, confidential information and/or be subject to legal privilege. It should not be copied, disclosed to, retained or used by, any other party. If you are not an intended recipient then please promptly delete this e-mail and any attachment and all copies and inform the sender. Thank you for understanding.

2 1

MDB, SQLite
by Howard Chu 19 Mar '12

19 Mar '12

I'll be presenting some more details at the UKUUG LISA conference in Edinburgh in 2 days. Just a quick note here... SQLite 3.7.7.1 (unmodified) took ~22.43 seconds to insert 1000 randomly-ordered records on my laptop (with Samsung SSD). The SQLite modified to use MDB took only 1.06 seconds. This 20:1 improvement in write performance is most significant in terms of absolute efficiency. Given how pervasively SQLite is used in Android and apps like Firefox, on smartphones, tablets, and other battery-powered devices, there's a potential to significantly increase the runtimes of these devices by switching to SQLite+MDB. Read efficiency is somewhat of a lost cause, >95% of the CPU time in reads is spent in the SQL parser. Even though MDB is much more efficient at reads than stock SQLite, the difference is lost in the noise under the parser overhead. I believe, for the vast majority of situations where SQLite is embedded in applications, significant efficiency gains would be obtained by using a pure binary data interface. While a simple key-value interface may be too primitive for app convenience, an inline implementation of the X.500 data model (with indexing) may be more useful. I.e., slapd back-mdb turned into an application library. -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

1 0

Jump to page:

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

openldap-devel