The Admin Guide still has not been updated with all of the relevant changes, so here are some notes on new features in the 2.4 release... I believe all of the manpages are up to date, so you can get specifics from them.
More complete cn=config functionality: There is a new slapd-config(5) manpage for the cn=config backend.
the original design called for auto-renaming of config entries when you insert or delete entries with ordered names, but that was not implemented in 2.3. It is now in 2.4. This means, e.g., if you have olcDatabase={1}bdb,cn=config olcSuffix: dc=example,dc=com
and you want to add a new subordinate, now you can: ldapadd olcDatabase={1}bdb,cn=config olcSuffix: dc=foo,dc=example,dc=com
this will insert a new BDB database in slot 1 and bump all following databases down one, so the original BDB database will now be named olcDatabase={2}bdb,cn=config olcSuffix: dc=example,dc=com
In 2.3 you were only able to add new schema elements, not delete or modify existing elements. In 2.4 you can modify schema at will. (Except for the hardcoded system schema, of course.)
More sophisticated syncrepl configurations: the original implementation of syncrepl in OpenLDAP 2.2 was intended to support multiple consumers within the same database, but that feature never worked and was removed from OpenLDAP 2.3. I.e., you could only configure a single consumer in any database.
In 2.4 you can configure multiple consumers in a single database. The configuration possibilities here are quite complex and numerous. You can configure consumers over arbitrary subtrees of a database (disjoint or overlapping). Any portion of the database may in turn be provided to other consumers using the syncprov overlay. The syncprov overlay works with any number of consumers over a single database or over arbitrarily many glued databases.
As a consequence of the work to support multiple consumer contexts, the syncrepl system now supports full N-way multimaster replication with entry-level conflict resolution. There are some important constraints, of course: In order to maintain consistent results across all servers, you must maintain tightly synchronized clocks across all participating servers (e.g., you must use NTP on all servers). The entryCSNs used for replication now record timestamps with microsecond resolution, instead of just seconds. The delta-syncrepl code has not been updated to support multimaster usage yet, that will come later in the 2.4 cycle.
On a related note, syncrepl was explicitly disabled on cn=config in 2.3. It is now fully supported in 2.4; you can use syncrepl to replicate an entire server configuration from one server to arbitrarily many other servers. It's possible to clone an entire running slapd using just a small (less than 10 lines) seed configuration, or you can just replicate the schema subtrees, etc. Tests 049 and 050 in the test suite provide working examples of these capabilities.
In 2.3 you could configure syncrepl as a full push-mode replicator by using it in conjunction with a back-ldap pointed at the target server. But because the back-ldap database needs to have a suffix corresponding to the target's suffix, you could only configure one instance per slapd.
In 2.4 you can define a database to be "hidden" which means that its suffix is ignored when checking for name collisions, and the database will never be used to answer requests received by the frontend. Using this hidden database feature allows you to configure multiple databases with the same suffix, allowing you to set up multiple back-ldap instances for pushing replication of a single database to multiple targets. There may be other uses for hidden databases as well (e.g., using a syncrepl consumer to maintain a *local* mirror of a database on a separate filesystem).
More extensive TLS configuration control: In 2.3, the TLS configuration in slapd was only used by the slapd listeners. For outbound connections used by e.g. back-ldap or syncrepl their TLS parameters came from the system's ldap.conf file. In 2.4 all of these sessions inherit their settings from the main slapd configuration but settings can be individually overridden on a per-config-item basis. This is particularly helpful if you use certificate-based authentication and need to use a different client certificate for different destinations.
Various performance enhancements: Too many to list. Some notable changes - ldapadd used to be a couple of orders of magnitude slower than "slapadd -q". It's now at worst only about half the speed of slapadd -q. A few weeks ago I did some comparisons of all the 2.x OpenLDAP releases; the results are in the slides from my SCALE presentation and you can find a copy here: http://www.highlandsun.com/hyc/scale2007.pdf
That compared 2.0.27, 2.1.30, 2.2.30, 2.3.33, and HEAD (as of a couple weeks ago). Toward the latter end of the "Cached Search Performance" chart it gets hard to see the difference because the runtimes are so small, but the new code is about 25% faster than 2.3, which was about 20% faster than 2.2, which was about 100% faster than 2.1, which was about 100% faster than 2.0, in that particular search scenario. That test basically searched a 1.3GB DB of 380836 entries (all in the slapd entry cache) in under 1 second. i.e., on a 2.4GHz CPU with DDR400 ECC/Registered RAM we can search over 500 thousand entries per second. The search was on an unindexed attribute using a filter that would not match any entry, forcing slapd to examine every entry in the DB, testing the filter for a match. Essentially the slapd entry cache in back-bdb/back-hdb is so efficient the search processing time is almost invisible; the runtime is limited only by the memory bandwidth of the machine. (The search data rate corresponds to about 3.5GB/sec; the memory bandwidth on the machine is only about 4GB/sec due to ECC and register latency.)
I think it goes without saying that no other Directory Server in the world is this fast or this efficient. Couple that with the scalability, manageability, flexibility, and just the sheer know-how behind this software, and nothing else is even remotely comparable.
On 2/16/07, Howard Chu hyc@symas.com wrote:
The Admin Guide still has not been updated with all of the relevant changes, so here are some notes on new features in the 2.4 release... I believe all of the manpages are up to date, so you can get specifics from them.
More complete cn=config functionality: There is a new slapd-config(5) manpage for the cn=config backend.
the original design called for auto-renaming of config entries when youinsert or delete entries with ordered names, but that was not implemented in 2.3. It is now in 2.4. This means, e.g., if you have olcDatabase={1}bdb,cn=config olcSuffix: dc=example,dc=com
and you want to add a new subordinate, now you can: ldapadd olcDatabase={1}bdb,cn=config olcSuffix: dc=foo,dc=example,dc=com
this will insert a new BDB database in slot 1 and bump all following databases down one, so the original BDB database will now be named olcDatabase={2}bdb,cn=config olcSuffix: dc=example,dc=com
In 2.3 you were only able to add new schema elements, not delete ormodify existing elements. In 2.4 you can modify schema at will. (Except for the hardcoded system schema, of course.)
More sophisticated syncrepl configurations: the original implementation of syncrepl in OpenLDAP 2.2 was intended to support multiple consumers within the same database, but that feature never worked and was removed from OpenLDAP 2.3. I.e., you could only configure a single consumer in any database.
In 2.4 you can configure multiple consumers in a single database. Theconfiguration possibilities here are quite complex and numerous. You can configure consumers over arbitrary subtrees of a database (disjoint or overlapping). Any portion of the database may in turn be provided to other consumers using the syncprov overlay. The syncprov overlay works with any number of consumers over a single database or over arbitrarily many glued databases.
As a consequence of the work to support multiple consumer contexts, thesyncrepl system now supports full N-way multimaster replication with entry-level conflict resolution. There are some important constraints, of course: In order to maintain consistent results across all servers, you must maintain tightly synchronized clocks across all participating servers (e.g., you must use NTP on all servers). The entryCSNs used for replication now record timestamps with microsecond resolution, instead of just seconds. The delta-syncrepl code has not been updated to support multimaster usage yet, that will come later in the 2.4 cycle.
On a related note, syncrepl was explicitly disabled on cn=config in 2.3.It is now fully supported in 2.4; you can use syncrepl to replicate an entire server configuration from one server to arbitrarily many other servers. It's possible to clone an entire running slapd using just a small (less than 10 lines) seed configuration, or you can just replicate the schema subtrees, etc. Tests 049 and 050 in the test suite provide working examples of these capabilities.
In 2.3 you could configure syncrepl as a full push-mode replicator byusing it in conjunction with a back-ldap pointed at the target server. But because the back-ldap database needs to have a suffix corresponding to the target's suffix, you could only configure one instance per slapd.
In 2.4 you can define a database to be "hidden" which means that itssuffix is ignored when checking for name collisions, and the database will never be used to answer requests received by the frontend. Using this hidden database feature allows you to configure multiple databases with the same suffix, allowing you to set up multiple back-ldap instances for pushing replication of a single database to multiple targets. There may be other uses for hidden databases as well (e.g., using a syncrepl consumer to maintain a *local* mirror of a database on a separate filesystem).
More extensive TLS configuration control: In 2.3, the TLS configuration in slapd was only used by the slapd listeners. For outbound connections used by e.g. back-ldap or syncrepl their TLS parameters came from the system's ldap.conf file. In 2.4 all of these sessions inherit their settings from the main slapd configuration but settings can be individually overridden on a per-config-item basis. This is particularly helpful if you use certificate-based authentication and need to use a different client certificate for different destinations.
Various performance enhancements: Too many to list. Some notable changes - ldapadd used to be a couple of orders of magnitude slower than "slapadd -q". It's now at worst only about half the speed of slapadd -q. A few weeks ago I did some comparisons of all the 2.x OpenLDAP releases; the results are in the slides from my SCALE presentation and you can find a copy here: http://www.highlandsun.com/hyc/scale2007.pdf
That compared 2.0.27, 2.1.30, 2.2.30, 2.3.33, and HEAD (as of a coupleweeks ago). Toward the latter end of the "Cached Search Performance" chart it gets hard to see the difference because the runtimes are so small, but the new code is about 25% faster than 2.3, which was about 20% faster than 2.2, which was about 100% faster than 2.1, which was about 100% faster than 2.0, in that particular search scenario. That test basically searched a 1.3GB DB of 380836 entries (all in the slapd entry cache) in under 1 second. i.e., on a 2.4GHz CPU with DDR400 ECC/Registered RAM we can search over 500 thousand entries per second. The search was on an unindexed attribute using a filter that would not match any entry, forcing slapd to examine every entry in the DB, testing the filter for a match. Essentially the slapd entry cache in back-bdb/back-hdb is so efficient the search processing time is almost invisible; the runtime is limited only by the memory bandwidth of the machine. (The search data rate corresponds to about 3.5GB/sec; the memory bandwidth on the machine is only about 4GB/sec due to ECC and register latency.)
I think it goes without saying that no other Directory Server in the world is this fast or this efficient. Couple that with the scalability, manageability, flexibility, and just the sheer know-how behind this software, and nothing else is even remotely comparable.
Great, great, great, and great!
Have you done any testing with 2.4/HEAD on bdb 4.5 or any of the newer oracle/bdb stuff? I heard they were making a lot of performance improvements for writes. (at least, that's what our local oracle fan club says)
_Matt
On 2/16/07, matthew sporleder msporleder@gmail.com wrote:
On 2/16/07, Howard Chu hyc@symas.com wrote:
The Admin Guide still has not been updated with all of the relevant changes, so here are some notes on new features in the 2.4 release... I believe all of the manpages are up to date, so you can get specifics from them.
More complete cn=config functionality: There is a new slapd-config(5) manpage for the cn=config backend.
the original design called for auto-renaming of config entries when youinsert or delete entries with ordered names, but that was not implemented in 2.3. It is now in 2.4. This means, e.g., if you have olcDatabase={1}bdb,cn=config olcSuffix: dc=example,dc=com
and you want to add a new subordinate, now you can: ldapadd olcDatabase={1}bdb,cn=config olcSuffix: dc=foo,dc=example,dc=com
this will insert a new BDB database in slot 1 and bump all following databases down one, so the original BDB database will now be named olcDatabase={2}bdb,cn=config olcSuffix: dc=example,dc=com
In 2.3 you were only able to add new schema elements, not delete ormodify existing elements. In 2.4 you can modify schema at will. (Except for the hardcoded system schema, of course.)
More sophisticated syncrepl configurations: the original implementation of syncrepl in OpenLDAP 2.2 was intended to support multiple consumers within the same database, but that feature never worked and was removed from OpenLDAP 2.3. I.e., you could only configure a single consumer in any database.
In 2.4 you can configure multiple consumers in a single database. Theconfiguration possibilities here are quite complex and numerous. You can configure consumers over arbitrary subtrees of a database (disjoint or overlapping). Any portion of the database may in turn be provided to other consumers using the syncprov overlay. The syncprov overlay works with any number of consumers over a single database or over arbitrarily many glued databases.
As a consequence of the work to support multiple consumer contexts, thesyncrepl system now supports full N-way multimaster replication with entry-level conflict resolution. There are some important constraints, of course: In order to maintain consistent results across all servers, you must maintain tightly synchronized clocks across all participating servers (e.g., you must use NTP on all servers). The entryCSNs used for replication now record timestamps with microsecond resolution, instead of just seconds. The delta-syncrepl code has not been updated to support multimaster usage yet, that will come later in the 2.4 cycle.
On a related note, syncrepl was explicitly disabled on cn=config in 2.3.It is now fully supported in 2.4; you can use syncrepl to replicate an entire server configuration from one server to arbitrarily many other servers. It's possible to clone an entire running slapd using just a small (less than 10 lines) seed configuration, or you can just replicate the schema subtrees, etc. Tests 049 and 050 in the test suite provide working examples of these capabilities.
In 2.3 you could configure syncrepl as a full push-mode replicator byusing it in conjunction with a back-ldap pointed at the target server. But because the back-ldap database needs to have a suffix corresponding to the target's suffix, you could only configure one instance per slapd.
In 2.4 you can define a database to be "hidden" which means that itssuffix is ignored when checking for name collisions, and the database will never be used to answer requests received by the frontend. Using this hidden database feature allows you to configure multiple databases with the same suffix, allowing you to set up multiple back-ldap instances for pushing replication of a single database to multiple targets. There may be other uses for hidden databases as well (e.g., using a syncrepl consumer to maintain a *local* mirror of a database on a separate filesystem).
More extensive TLS configuration control: In 2.3, the TLS configuration in slapd was only used by the slapd listeners. For outbound connections used by e.g. back-ldap or syncrepl their TLS parameters came from the system's ldap.conf file. In 2.4 all of these sessions inherit their settings from the main slapd configuration but settings can be individually overridden on a per-config-item basis. This is particularly helpful if you use certificate-based authentication and need to use a different client certificate for different destinations.
Various performance enhancements: Too many to list. Some notable changes - ldapadd used to be a couple of orders of magnitude slower than "slapadd -q". It's now at worst only about half the speed of slapadd -q. A few weeks ago I did some comparisons of all the 2.x OpenLDAP releases; the results are in the slides from my SCALE presentation and you can find a copy here: http://www.highlandsun.com/hyc/scale2007.pdf
That compared 2.0.27, 2.1.30, 2.2.30, 2.3.33, and HEAD (as of a coupleweeks ago). Toward the latter end of the "Cached Search Performance" chart it gets hard to see the difference because the runtimes are so small, but the new code is about 25% faster than 2.3, which was about 20% faster than 2.2, which was about 100% faster than 2.1, which was about 100% faster than 2.0, in that particular search scenario. That test basically searched a 1.3GB DB of 380836 entries (all in the slapd entry cache) in under 1 second. i.e., on a 2.4GHz CPU with DDR400 ECC/Registered RAM we can search over 500 thousand entries per second. The search was on an unindexed attribute using a filter that would not match any entry, forcing slapd to examine every entry in the DB, testing the filter for a match. Essentially the slapd entry cache in back-bdb/back-hdb is so efficient the search processing time is almost invisible; the runtime is limited only by the memory bandwidth of the machine. (The search data rate corresponds to about 3.5GB/sec; the memory bandwidth on the machine is only about 4GB/sec due to ECC and register latency.)
I think it goes without saying that no other Directory Server in the world is this fast or this efficient. Couple that with the scalability, manageability, flexibility, and just the sheer know-how behind this software, and nothing else is even remotely comparable.
Great, great, great, and great!
Have you done any testing with 2.4/HEAD on bdb 4.5 or any of the newer oracle/bdb stuff? I heard they were making a lot of performance improvements for writes. (at least, that's what our local oracle fan club says)
Hey, watching the slide show answers my own question. BDB 4.6 looks nice. (40% improvement over 4.2's malloc?)
matthew sporleder wrote:
Great, great, great, and great!
Have you done any testing with 2.4/HEAD on bdb 4.5 or any of the newer oracle/bdb stuff? I heard they were making a lot of performance improvements for writes. (at least, that's what our local oracle fan club says)
Yes. In fact there are significant performance improvements in BDB 4.6 which are due in part to work by Jong-Hyuk Choi analyzing the memory management behavior of the BerkeleyDB library. (In case you're not familiar with Jong's name, he wrote the initial entry cache for back-bdb, developed the syncrepl spec with Kurt, and has done a lot of other performance analysis of OpenLDAP through the years.) And in the set of slides I posted, you'll see performance comparisons for BDB 4.2, 4.5 and 4.6.
Note that BDB 4.6 has not yet been released; I'm told it's due out this summer. It is MUCH more efficient than previous releases. It will be well worth adopting when it becomes generally available. (And yes, OpenLDAP already fully supports it.)
<quote who="Howard Chu">
The Admin Guide still has not been updated with all of the relevant changes, so here are some notes on new features in the 2.4 release... I believe all of the manpages are up to date, so you can get specifics from them.
More complete cn=config functionality: There is a new slapd-config(5) manpage for the cn=config backend.
the original design called for auto-renaming of config entries whenyou insert or delete entries with ordered names, but that was not implemented in 2.3. It is now in 2.4. This means, e.g., if you have olcDatabase={1}bdb,cn=config olcSuffix: dc=example,dc=com
and you want to add a new subordinate, now you can: ldapadd olcDatabase={1}bdb,cn=config olcSuffix: dc=foo,dc=example,dc=com
this will insert a new BDB database in slot 1 and bump all following databases down one, so the original BDB database will now be named olcDatabase={2}bdb,cn=config olcSuffix: dc=example,dc=com
In 2.3 you were only able to add new schema elements, not delete ormodify existing elements. In 2.4 you can modify schema at will. (Except for the hardcoded system schema, of course.)
More sophisticated syncrepl configurations: the original implementation of syncrepl in OpenLDAP 2.2 was intended to support multiple consumers within the same database, but that feature never worked and was removed from OpenLDAP 2.3. I.e., you could only configure a single consumer in any database.
In 2.4 you can configure multiple consumers in a single database. Theconfiguration possibilities here are quite complex and numerous. You can configure consumers over arbitrary subtrees of a database (disjoint or overlapping). Any portion of the database may in turn be provided to other consumers using the syncprov overlay. The syncprov overlay works with any number of consumers over a single database or over arbitrarily many glued databases.
As a consequence of the work to support multiple consumer contexts,the syncrepl system now supports full N-way multimaster replication with entry-level conflict resolution. There are some important constraints, of course: In order to maintain consistent results across all servers, you must maintain tightly synchronized clocks across all participating servers (e.g., you must use NTP on all servers). The entryCSNs used for replication now record timestamps with microsecond resolution, instead of just seconds. The delta-syncrepl code has not been updated to support multimaster usage yet, that will come later in the 2.4 cycle.
On a related note, syncrepl was explicitly disabled on cn=config in2.3. It is now fully supported in 2.4; you can use syncrepl to replicate an entire server configuration from one server to arbitrarily many other servers. It's possible to clone an entire running slapd using just a small (less than 10 lines) seed configuration, or you can just replicate the schema subtrees, etc. Tests 049 and 050 in the test suite provide working examples of these capabilities.
In 2.3 you could configure syncrepl as a full push-mode replicator byusing it in conjunction with a back-ldap pointed at the target server. But because the back-ldap database needs to have a suffix corresponding to the target's suffix, you could only configure one instance per slapd.
In 2.4 you can define a database to be "hidden" which means that itssuffix is ignored when checking for name collisions, and the database will never be used to answer requests received by the frontend. Using this hidden database feature allows you to configure multiple databases with the same suffix, allowing you to set up multiple back-ldap instances for pushing replication of a single database to multiple targets. There may be other uses for hidden databases as well (e.g., using a syncrepl consumer to maintain a *local* mirror of a database on a separate filesystem).
More extensive TLS configuration control: In 2.3, the TLS configuration in slapd was only used by the slapd listeners. For outbound connections used by e.g. back-ldap or syncrepl their TLS parameters came from the system's ldap.conf file. In 2.4 all of these sessions inherit their settings from the main slapd configuration but settings can be individually overridden on a per-config-item basis. This is particularly helpful if you use certificate-based authentication and need to use a different client certificate for different destinations.
Various performance enhancements: Too many to list. Some notable changes - ldapadd used to be a couple of orders of magnitude slower than "slapadd -q". It's now at worst only about half the speed of slapadd -q. A few weeks ago I did some comparisons of all the 2.x OpenLDAP releases; the results are in the slides from my SCALE presentation and you can find a copy here: http://www.highlandsun.com/hyc/scale2007.pdf
That compared 2.0.27, 2.1.30, 2.2.30, 2.3.33, and HEAD (as of a coupleweeks ago). Toward the latter end of the "Cached Search Performance" chart it gets hard to see the difference because the runtimes are so small, but the new code is about 25% faster than 2.3, which was about 20% faster than 2.2, which was about 100% faster than 2.1, which was about 100% faster than 2.0, in that particular search scenario. That test basically searched a 1.3GB DB of 380836 entries (all in the slapd entry cache) in under 1 second. i.e., on a 2.4GHz CPU with DDR400 ECC/Registered RAM we can search over 500 thousand entries per second. The search was on an unindexed attribute using a filter that would not match any entry, forcing slapd to examine every entry in the DB, testing the filter for a match. Essentially the slapd entry cache in back-bdb/back-hdb is so efficient the search processing time is almost invisible; the runtime is limited only by the memory bandwidth of the machine. (The search data rate corresponds to about 3.5GB/sec; the memory bandwidth on the machine is only about 4GB/sec due to ECC and register latency.)
I think it goes without saying that no other Directory Server in the world is this fast or this efficient. Couple that with the scalability, manageability, flexibility, and just the sheer know-how behind this software, and nothing else is even remotely comparable. -- -- Howard Chu Chief Architect, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc Chief Architect, OpenLDAP http://www.openldap.org/project/
One e-mail to be framed! ;-)
You mentioned benchmarks etc. with bdb 4.6.2, being the fastest read/write ever. Any news on 4.6.x going stable?
Gavin.
On Fri, 16 Feb 2007, Howard Chu wrote:
More complete cn=config functionality: There is a new slapd-config(5) manpage for the cn=config backend.
Is slapd-meta supported yet?
--On Friday, February 16, 2007 1:53 PM +0000 Gavin Henry ghenry@suretecsystems.com wrote:
One e-mail to be framed! ;-)
You mentioned benchmarks etc. with bdb 4.6.2, being the fastest read/write ever. Any news on 4.6.x going stable?
As noted by Howard in reply to another email in this thread, BDB 4.6 is scheduled for summer release. It is currently only in testing.
--Quanah
-- Quanah Gibson-Mount Principal Software Developer ITS/Shared Application Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
Dave Horsfall wrote:
On Fri, 16 Feb 2007, Howard Chu wrote:
More complete cn=config functionality: There is a new slapd-config(5) manpage for the cn=config backend.
Is slapd-meta supported yet?
No. That, along with slapo-rwm, will be my next task.
p.
Ing. Pierangelo Masarati OpenLDAP Core Team
SysNet s.n.c. Via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it ------------------------------------------ Office: +39.02.23998309 Mobile: +39.333.4963172 Email: pierangelo.masarati@sys-net.it ------------------------------------------
On Mon, 19 Feb 2007, Pierangelo Masarati wrote:
Is slapd-meta supported yet?
No. That, along with slapo-rwm, will be my next task.
Many thanks.
Some minor bits, from browsing the Software Enhancements section of the ITS. I figured they're relatively obscure and easy to miss, so drawing attention to them here.
ITS#4554: slapindex option for attribute you can now give a list of specific attributes to (re)index on the slapindex command line. This is handy if you've just changed one or two settings. Of course, this only matters if you're manually editing slapd.conf; if you change index configurations using cn=config then the background indexer will take care of it. (This feature is described in the slapindex(8) manpage.)
ITS#4707: for LDAP application developers, a new function ldap_init_fd() has been added to allow an LDAP session to be created using a socket provided by the caller. This is useful if the application has particular constraints it needs to set on the client connection before giving control to libldap. (This function is in the ldap_open(3) manpage.)
ITS#4834: use the dn2id index for extensible filters matching against the entryDN attribute. This probably only affects a small class of search uses. One example is a search I would use in my LDAP milter for Sendmail - when looking up access rules for an email address, if an exact match for the address isn't found, the code should incrementally peel off domain components until a match is found, or until it reaches an empty domain and no matches were found. E.g., given the email address joe_bob@foo.example.com, the milter does a lookup on uid=joe_bob,dc=foo,dc=example,dc=com,<suffix> On a failure it might come back with "matchedDN: dc=example,dc=com,<suffix>" and the milter would then pick up the rules in that entry on a 2nd search attempt.
But using an extensible filter, I could get every possible rule in a single search, with a filter like this: (|(entryDN=uid=joe_bob,dc=foo,dc=example,dc=com,<suffix>) (entryDN:dnSuperiorMatch:=uid=joe_bob,dc=foo,dc=example,dc=com,<suffix>))
which would return the target entry (if it existed) and all of its parent entries.
Like I said, obscure...
"hyc" == Howard Chu hyc@symas.com writes:
hyc> The entryCSNs used for replication now record timestamps with microsecond hyc> resolution, instead of just seconds.
As slapcat'd from 2.3.39, I have CSNs in various forms:
grep CSN: users.ldapv2.2007.11.08.ldif |head entryCSN: 20071030125340Z#000000#00#000000 contextCSN: 20071108000208Z#000000#00#000000 entryCSN: 2002120818:17:53Z#0x0061#0#0000 entryCSN: 20060412175546Z#000001#00#000000 entryCSN: 20070701134407Z#000000#00#000000 entryCSN: 20060412175617Z#00000c#00#000000 entryCSN: 20070911105027Z#000002#00#000000 entryCSN: 20070908105022Z#000000#00#000000 entryCSN: 20060412175733Z#00000f#00#000000 entryCSN: 20070531105422Z#000000#00#000000
The insertion into 2.4 fails:
slapadd: schema_init.c:3486: csnNormalize: Assertion `val->bv_len == (sizeof("YYYYmmddHHMMSS.uuuuuuZ#SSSSSS#SID#ssssss")-1)' failed. Abort
My guess is that I need to warp those CSNs into the new form before I can load my data.
Is there a script I've overlooked that accomplishes that?
Thanks.
Hmmm. Looking at the code as I type this, perhaps the only evil one is the one with colons and the "0x" hex field. If I change
entryCSN: 2002120818:17:53Z#0x0061#0#0000
to
entryCSN: 20021208181753Z#000061#00#000000
will that be sufficient to let the data load? Mercifully, I only have 67 of those in my small database.
Thanks again.
"hyc" == Howard Chu hyc@symas.com writes:
hyc> The entryCSNs used for replication now record timestamps with microsecond hyc> resolution, instead of just seconds.
Would this be suitable to adjust CSNs of the form 2002120818:17:48Z#0x008d#0#0000?
*** servers/slapd/schema_init.c~ Wed Oct 17 21:35:12 2007 --- servers/slapd/schema_init.c Thu Nov 8 14:08:11 2007 *************** *** 3454,3459 **** --- 3454,3548 ---- return LDAP_SUCCESS; }
+ /* Normalize a CSN in OpenLDAP ? format */ + static int + csnNormalize_stoneage( + slap_mask_t usage, + Syntax *syntax, + MatchingRule *mr, + struct berval *val, + struct berval *normalized, + void *ctx ) + { + struct berval gt, cnt, sid, mod; + char *ptr; + int i; + + assert( SLAP_MR_IS_VALUE_OF_SYNTAX( usage ) != 0 ); + assert( !BER_BVISEMPTY( val ) ); + + gt = *val; + + ptr = ber_bvchr( >, '#' ); + if ( ptr == NULL || ptr - gt.bv_val == gt.bv_len ) { + return LDAP_INVALID_SYNTAX; + } + + gt.bv_len = ptr - gt.bv_val; + assert( gt.bv_len == STRLENOF( "YYYYmmddHH:MM:SSZ" ) ); + assert( gt.bv_val[10] == ':' ); + assert( gt.bv_val[13] == ':' ); + + cnt.bv_val = ptr + 1; + cnt.bv_len = val->bv_len - ( cnt.bv_val - val->bv_val ); + + ptr = ber_bvchr( &cnt, '#' ); + if ( ptr == NULL || ptr - val->bv_val == val->bv_len ) { + return LDAP_INVALID_SYNTAX; + } + + cnt.bv_len = ptr - cnt.bv_val; + assert( cnt.bv_len == STRLENOF( "0x0000" ) ); + assert( strncmp(cnt.bv_val, "0x", 2) == 0 ); + + sid.bv_val = ptr + 1; + sid.bv_len = val->bv_len - ( sid.bv_val - val->bv_val ); + + ptr = ber_bvchr( &sid, '#' ); + if ( ptr == NULL || ptr - val->bv_val == val->bv_len ) { + return LDAP_INVALID_SYNTAX; + } + + sid.bv_len = ptr - sid.bv_val; + assert( sid.bv_len == STRLENOF( "0" ) ); + + mod.bv_val = ptr + 1; + mod.bv_len = val->bv_len - ( mod.bv_val - val->bv_val ); + assert( mod.bv_len == STRLENOF( "0000" ) ); + + normalized->bv_len = STRLENOF( "YYYYmmddHHMMSS.uuuuuuZ#SSSSSS#SID#ssssss" ); + normalized->bv_val = ber_memalloc_x( normalized->bv_len + 1, ctx ); + + ptr = normalized->bv_val; + + ptr = lutil_strncopy( ptr, gt.bv_val, 10); + ptr = lutil_strncopy( ptr, gt.bv_val + 11, 2); + ptr = lutil_strncopy( ptr, gt.bv_val + 14, 2); + + ptr = lutil_strcopy( ptr, ".000000Z#" ); + + *ptr++ = '0'; + *ptr++ = '0'; + ptr = lutil_strncopy( ptr, cnt.bv_val + 2, cnt.bv_len - 2); + *ptr++ = '#'; + *ptr++ = '0'; + *ptr++ = '0'; + for ( i = 0; i < sid.bv_len; i++ ) { + *ptr++ = TOLOWER( sid.bv_val[ i ] ); + } + *ptr++ = '#'; + *ptr++ = '0'; + *ptr++ = '0'; + for ( i = 0; i < mod.bv_len; i++ ) { + *ptr++ = TOLOWER( mod.bv_val[ i ] ); + } + *ptr = '\0'; + + assert( ptr - normalized->bv_val == normalized->bv_len ); + + return LDAP_SUCCESS; + } + /* Normalize a CSN */ static int csnNormalize( *************** *** 3483,3488 **** --- 3572,3583 ---- return csnNormalize23( usage, syntax, mr, val, normalized, ctx ); }
+ if ( val->bv_len == STRLENOF( "YYYYmmddHH:MM:SSZ#0xSSSS#I#ssss" ) ) { + /* Openldap <= ? */ + + return csnNormalize_stoneage( usage, syntax, mr, val, normalized, ctx ); + } + assert( val->bv_len == STRLENOF( "YYYYmmddHHMMSS.uuuuuuZ#SSSSSS#SID#ssssss" ) );
ptr = ber_bvchr( val, '#' );
Allan E. Johannesen wrote:
Hmmm. Looking at the code as I type this, perhaps the only evil one is the one with colons and the "0x" hex field. If I change
entryCSN: 2002120818:17:53Z#0x0061#0#0000
This really looks evil. I think that assert needs to be relaxed into a run-time error, but I don't really see how that data could have made it into your database. The old syntax should be allowed, as far as I can remember. In any case, slapadd should tell you exactly what entry couldn't be imported, you should be able to check what the offending entryCSN looks like.
p.
Ing. Pierangelo Masarati OpenLDAP Core Team
SysNet s.r.l. via Dossi, 8 - 27100 Pavia - ITALIA http://www.sys-net.it --------------------------------------- Office: +39 02 23998309 Mobile: +39 333 4963172 Email: pierangelo.masarati@sys-net.it ---------------------------------------
openldap-software@openldap.org
-
Allan E. Johannesen -
Dave Horsfall -
Gavin Henry -
Howard Chu -
matthew sporleder -
Pierangelo Masarati -
Quanah Gibson-Mount