On 2/16/07, Howard Chu <hyc(a)symas.com&gt; wrote: > The Admin Guide still has not been updated with all of the relevant changes, > so here are some notes on new features in the 2.4 release... I believe all of > the manpages are up to date, so you can get specifics from them. > > More complete cn=config functionality: > There is a new slapd-config(5) manpage for the cn=config backend. > > the original design called for auto-renaming of config entries when you > insert or delete entries with ordered names, but that was not implemented in > 2.3. It is now in 2.4. This means, e.g., if you have > olcDatabase={1}bdb,cn=config > olcSuffix: dc=example,dc=com > > and you want to add a new subordinate, now you can: > ldapadd olcDatabase={1}bdb,cn=config > olcSuffix: dc=foo,dc=example,dc=com > > this will insert a new BDB database in slot 1 and bump all following > databases down one, so the original BDB database will now be named > olcDatabase={2}bdb,cn=config > olcSuffix: dc=example,dc=com > > In 2.3 you were only able to add new schema elements, not delete or > modify existing elements. In 2.4 you can modify schema at will. (Except for > the hardcoded system schema, of course.) > > > More sophisticated syncrepl configurations: > the original implementation of syncrepl in OpenLDAP 2.2 was intended to > support multiple consumers within the same database, but that feature never > worked and was removed from OpenLDAP 2.3. I.e., you could only configure a > single consumer in any database. > > In 2.4 you can configure multiple consumers in a single database. The > configuration possibilities here are quite complex and numerous. You can > configure consumers over arbitrary subtrees of a database (disjoint or > overlapping). Any portion of the database may in turn be provided to other > consumers using the syncprov overlay. The syncprov overlay works with any > number of consumers over a single database or over arbitrarily many glued > databases. > > As a consequence of the work to support multiple consumer contexts, the > syncrepl system now supports full N-way multimaster replication with > entry-level conflict resolution. There are some important constraints, of > course: In order to maintain consistent results across all servers, you must > maintain tightly synchronized clocks across all participating servers (e.g., > you must use NTP on all servers). The entryCSNs used for replication now > record timestamps with microsecond resolution, instead of just seconds. The > delta-syncrepl code has not been updated to support multimaster usage yet, > that will come later in the 2.4 cycle. > > On a related note, syncrepl was explicitly disabled on cn=config in 2.3. > It is now fully supported in 2.4; you can use syncrepl to replicate an entire > server configuration from one server to arbitrarily many other servers. It's > possible to clone an entire running slapd using just a small (less than 10 > lines) seed configuration, or you can just replicate the schema subtrees, > etc. Tests 049 and 050 in the test suite provide working examples of these > capabilities. > > In 2.3 you could configure syncrepl as a full push-mode replicator by > using it in conjunction with a back-ldap pointed at the target server. But > because the back-ldap database needs to have a suffix corresponding to the > target's suffix, you could only configure one instance per slapd. > > In 2.4 you can define a database to be "hidden" which means that its > suffix is ignored when checking for name collisions, and the database will > never be used to answer requests received by the frontend. Using this hidden > database feature allows you to configure multiple databases with the same > suffix, allowing you to set up multiple back-ldap instances for pushing > replication of a single database to multiple targets. There may be other uses > for hidden databases as well (e.g., using a syncrepl consumer to maintain a > *local* mirror of a database on a separate filesystem). > > > More extensive TLS configuration control: > In 2.3, the TLS configuration in slapd was only used by the slapd > listeners. For outbound connections used by e.g. back-ldap or syncrepl their > TLS parameters came from the system's ldap.conf file. > In 2.4 all of these sessions inherit their settings from the main slapd > configuration but settings can be individually overridden on a > per-config-item basis. This is particularly helpful if you use > certificate-based authentication and need to use a different client > certificate for different destinations. > > > Various performance enhancements: > Too many to list. Some notable changes - ldapadd used to be a couple of > orders of magnitude slower than "slapadd -q". It's now at worst only about > half the speed of slapadd -q. A few weeks ago I did some comparisons of all > the 2.x OpenLDAP releases; the results are in the slides from my SCALE > presentation and you can find a copy here: > http://www.highlandsun.com/hyc/scale2007.pdf > > That compared 2.0.27, 2.1.30, 2.2.30, 2.3.33, and HEAD (as of a couple > weeks ago). Toward the latter end of the "Cached Search Performance" chart it > gets hard to see the difference because the runtimes are so small, but the > new code is about 25% faster than 2.3, which was about 20% faster than 2.2, > which was about 100% faster than 2.1, which was about 100% faster than 2.0, > in that particular search scenario. That test basically searched a 1.3GB DB > of 380836 entries (all in the slapd entry cache) in under 1 second. i.e., on > a 2.4GHz CPU with DDR400 ECC/Registered RAM we can search over 500 thousand > entries per second. The search was on an unindexed attribute using a filter > that would not match any entry, forcing slapd to examine every entry in the > DB, testing the filter for a match. > Essentially the slapd entry cache in back-bdb/back-hdb is so efficient > the search processing time is almost invisible; the runtime is limited only > by the memory bandwidth of the machine. (The search data rate corresponds to > about 3.5GB/sec; the memory bandwidth on the machine is only about 4GB/sec > due to ECC and register latency.) > > I think it goes without saying that no other Directory Server in the world is > this fast or this efficient. Couple that with the scalability, manageability, > flexibility, and just the sheer know-how behind this software, and nothing > else is even remotely comparable. Great, great, great, and great! Have you done any testing with 2.4/HEAD on bdb 4.5 or any of the newer oracle/bdb stuff? I heard they were making a lot of performance improvements for writes. (at least, that's what our local oracle fan club says)

The Admin Guide still has not been updated with all of the relevant changes, so here are some notes on new features in the 2.4 release... I believe all of the manpages are up to date, so you can get specifics from them. More complete cn=config functionality: There is a new slapd-config(5) manpage for the cn=config backend. the original design called for auto-renaming of config entries when you insert or delete entries with ordered names, but that was not implemented in 2.3. It is now in 2.4. This means, e.g., if you have olcDatabase={1}bdb,cn=config olcSuffix: dc=example,dc=com and you want to add a new subordinate, now you can: ldapadd olcDatabase={1}bdb,cn=config olcSuffix: dc=foo,dc=example,dc=com this will insert a new BDB database in slot 1 and bump all following databases down one, so the original BDB database will now be named olcDatabase={2}bdb,cn=config olcSuffix: dc=example,dc=com In 2.3 you were only able to add new schema elements, not delete or modify existing elements. In 2.4 you can modify schema at will. (Except for the hardcoded system schema, of course.) More sophisticated syncrepl configurations: the original implementation of syncrepl in OpenLDAP 2.2 was intended to support multiple consumers within the same database, but that feature never worked and was removed from OpenLDAP 2.3. I.e., you could only configure a single consumer in any database. In 2.4 you can configure multiple consumers in a single database. The configuration possibilities here are quite complex and numerous. You can configure consumers over arbitrary subtrees of a database (disjoint or overlapping). Any portion of the database may in turn be provided to other consumers using the syncprov overlay. The syncprov overlay works with any number of consumers over a single database or over arbitrarily many glued databases. As a consequence of the work to support multiple consumer contexts, the syncrepl system now supports full N-way multimaster replication with entry-level conflict resolution. There are some important constraints, of course: In order to maintain consistent results across all servers, you must maintain tightly synchronized clocks across all participating servers (e.g., you must use NTP on all servers). The entryCSNs used for replication now record timestamps with microsecond resolution, instead of just seconds. The delta-syncrepl code has not been updated to support multimaster usage yet, that will come later in the 2.4 cycle. On a related note, syncrepl was explicitly disabled on cn=config in 2.3. It is now fully supported in 2.4; you can use syncrepl to replicate an entire server configuration from one server to arbitrarily many other servers. It's possible to clone an entire running slapd using just a small (less than 10 lines) seed configuration, or you can just replicate the schema subtrees, etc. Tests 049 and 050 in the test suite provide working examples of these capabilities. In 2.3 you could configure syncrepl as a full push-mode replicator by using it in conjunction with a back-ldap pointed at the target server. But because the back-ldap database needs to have a suffix corresponding to the target's suffix, you could only configure one instance per slapd. In 2.4 you can define a database to be "hidden" which means that its suffix is ignored when checking for name collisions, and the database will never be used to answer requests received by the frontend. Using this hidden database feature allows you to configure multiple databases with the same suffix, allowing you to set up multiple back-ldap instances for pushing replication of a single database to multiple targets. There may be other uses for hidden databases as well (e.g., using a syncrepl consumer to maintain a *local* mirror of a database on a separate filesystem). More extensive TLS configuration control: In 2.3, the TLS configuration in slapd was only used by the slapd listeners. For outbound connections used by e.g. back-ldap or syncrepl their TLS parameters came from the system's ldap.conf file. In 2.4 all of these sessions inherit their settings from the main slapd configuration but settings can be individually overridden on a per-config-item basis. This is particularly helpful if you use certificate-based authentication and need to use a different client certificate for different destinations. Various performance enhancements: Too many to list. Some notable changes - ldapadd used to be a couple of orders of magnitude slower than "slapadd -q". It's now at worst only about half the speed of slapadd -q. A few weeks ago I did some comparisons of all the 2.x OpenLDAP releases; the results are in the slides from my SCALE presentation and you can find a copy here: http://www.highlandsun.com/hyc/scale2007.pdf That compared 2.0.27, 2.1.30, 2.2.30, 2.3.33, and HEAD (as of a couple weeks ago). Toward the latter end of the "Cached Search Performance" chart it gets hard to see the difference because the runtimes are so small, but the new code is about 25% faster than 2.3, which was about 20% faster than 2.2, which was about 100% faster than 2.1, which was about 100% faster than 2.0, in that particular search scenario. That test basically searched a 1.3GB DB of 380836 entries (all in the slapd entry cache) in under 1 second. i.e., on a 2.4GHz CPU with DDR400 ECC/Registered RAM we can search over 500 thousand entries per second. The search was on an unindexed attribute using a filter that would not match any entry, forcing slapd to examine every entry in the DB, testing the filter for a match. Essentially the slapd entry cache in back-bdb/back-hdb is so efficient the search processing time is almost invisible; the runtime is limited only by the memory bandwidth of the machine. (The search data rate corresponds to about 3.5GB/sec; the memory bandwidth on the machine is only about 4GB/sec due to ECC and register latency.) I think it goes without saying that no other Directory Server in the world is this fast or this efficient. Couple that with the scalability, manageability, flexibility, and just the sheer know-how behind this software, and nothing else is even remotely comparable. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc Chief Architect, OpenLDAP http://www.openldap.org/project/

More complete cn=config functionality: There is a new slapd-config(5) manpage for the cn=config backend.

> Is slapd-meta supported yet? No. That, along with slapo-rwm, will be my next task.

>>>> "hyc" == Howard Chu <hyc(a)symas.com&gt; writes:

>>>> "hyc" == Howard Chu <hyc(a)symas.com&gt; writes: