HI!
As far as I understood the MDB disk format changed. So the MDB files have to be re-created (either by simply removing/replicating or slapcat/slapadd). Right?
Now I'm wondering how to automate things (with ansible and puppet) in a truly idempotent way. Ideally I could determine whether existing MDB files were last maintained by OpenLDAP 2.4.x or whether they already have the 2.5 format.
Can I find out the disk format version in any way, e.g. with python-lmdb?
Ciao, Michael.
On 7/31/21 8:41 PM, A. Schulze wrote:
Am 31.07.21 um 18:05 schrieb Michael Ströder:
As far as I understood the MDB disk format changed.
I'm also start testing openldap-2.5, so could you provide a reference for that claim?
Section B.8 in [1] says:
"Due to database format changes between OpenLDAP 2.4 and OpenLDAP 2.5 it is mandatory for existing slapd-mdb(5) databases to be exported via an OpenLDAP 2.4 slapcat prior to upgrade."
Ciao, Michael.
[1] https://www.openldap.org/doc/admin25/appendix-upgrading.html#mdb%20backend
Michael Ströder michael@stroeder.com schrieb am 31.07.2021 um 18:05 in
Nachricht 60ec1d1e-a2be-95a2-c9c4-24ecd9b4f58c@stroeder.com:
HI!
As far as I understood the MDB disk format changed. So the MDB files have to be re-created (either by simply removing/replicating or slapcat/slapadd). Right?
AFAIK; Sleepycat DB could do that conversion on the fly. At least there should be some smarter conversion utility than export old DB with old binary and re-import with new binary. MHO.
Regards, Ulrich
Now I'm wondering how to automate things (with ansible and puppet) in a truly idempotent way. Ideally I could determine whether existing MDB files were last maintained by OpenLDAP 2.4.x or whether they already have the 2.5 format.
Can I find out the disk format version in any way, e.g. with python-lmdb?
Ciao, Michael.
On 8/2/21 8:06 AM, Ulrich Windl wrote:
Michael Ströder michael@stroeder.com schrieb am 31.07.2021 um 18:05 in
Nachricht 60ec1d1e-a2be-95a2-c9c4-24ecd9b4f58c@stroeder.com:
As far as I understood the MDB disk format changed. So the MDB files have to be re-created (either by simply removing/replicating or slapcat/slapadd). Right?
AFAIK; Sleepycat DB could do that conversion on the fly.
AFAICS the DB layout is application-specific. So there was also no way to let Berkeley-DB do this "on the fly".
At least there should be some smarter conversion utility than export old DB with old binary and re-import with new binary. MHO.
I have no issue with rebuilding the MDB files. I just want to know whether I have to do it or not.
Ciao, Michael.
Michael Ströder michael@stroeder.com schrieb am 02.08.2021 um 09:57 in
Nachricht 59abdf98-65a4-5bb4-fffb-f1384969772a@stroeder.com:
On 8/2/21 8:06 AM, Ulrich Windl wrote:
Michael Ströder michael@stroeder.com schrieb am 31.07.2021 um 18:05
in
Nachricht 60ec1d1e-a2be-95a2-c9c4-24ecd9b4f58c@stroeder.com:
As far as I understood the MDB disk format changed. So the MDB files have to be re-created (either by simply removing/replicating or slapcat/slapadd). Right?
AFAIK; Sleepycat DB could do that conversion on the fly.
AFAICS the DB layout is application-specific. So there was also no way to let Berkeley-DB do this "on the fly".
Of course Berkely DB cannot upgrade the MDB database; I was talking about the layout versions of Berkeley DB.
At least there should be some smarter conversion utility than export old
DB
with old binary and re-import with new binary. MHO.
I have no issue with rebuilding the MDB files. I just want to know whether I have to do it or not.
Ciao, Michael.
On 8/2/21 11:00 AM, Ulrich Windl wrote:
Michael Ströder michael@stroeder.com schrieb am 02.08.2021 um 09:57 in
Nachricht 59abdf98-65a4-5bb4-fffb-f1384969772a@stroeder.com:
On 8/2/21 8:06 AM, Ulrich Windl wrote:
Michael Ströder michael@stroeder.com schrieb am 31.07.2021 um 18:05
in
Nachricht 60ec1d1e-a2be-95a2-c9c4-24ecd9b4f58c@stroeder.com:
As far as I understood the MDB disk format changed. So the MDB files have to be re-created (either by simply removing/replicating or slapcat/slapadd). Right?
AFAIK; Sleepycat DB could do that conversion on the fly.
AFAICS the DB layout is application-specific. So there was also no way to let Berkeley-DB do this "on the fly".
Of course Berkely DB cannot upgrade the MDB database; I was talking about the layout versions of Berkeley DB.
I did not misunderstand you. I meant that Berkeley-DB could not do automagic upgrades of its files in case the application-specific format changes.
IIRC in former times with back-bdb and back-hdb there was also sometimes the need to reload BDB files because the slapd-specific layout changed.
Ciao, Michael.
Michael Ströder michael@stroeder.com schrieb am 02.08.2021 um 11:22 in
Nachricht 41eb0de3-d648-6ecb-6a2d-e03decb6bf03@stroeder.com:
On 8/2/21 11:00 AM, Ulrich Windl wrote:
Michael Ströder michael@stroeder.com schrieb am 02.08.2021 um 09:57
in
Nachricht 59abdf98-65a4-5bb4-fffb-f1384969772a@stroeder.com:
On 8/2/21 8:06 AM, Ulrich Windl wrote:
> Michael Ströder michael@stroeder.com schrieb am 31.07.2021 um 18:05
in
Nachricht 60ec1d1e-a2be-95a2-c9c4-24ecd9b4f58c@stroeder.com:
As far as I understood the MDB disk format changed. So the MDB files have to be re-created (either by simply removing/replicating or slapcat/slapadd). Right?
AFAIK; Sleepycat DB could do that conversion on the fly.
AFAICS the DB layout is application-specific. So there was also no way to let Berkeley-DB do this "on the fly".
Of course Berkely DB cannot upgrade the MDB database; I was talking about the layout versions of Berkeley DB.
I did not misunderstand you. I meant that Berkeley-DB could not do automagic upgrades of its files in case the application-specific format changes.
IIRC in former times with back-bdb and back-hdb there was also sometimes the need to reload BDB files because the slapd-specific layout changed.
Admittedly I never had or needed those. I only noticed changes of hash functions, and those did not need manual intervention.
Ciao, Michael.
--On Monday, August 2, 2021 12:25 PM +0200 Ulrich Windl Ulrich.Windl@rz.uni-regensburg.de wrote:
Of course Berkely DB cannot upgrade the MDB database; I was talking about the layout versions of Berkeley DB.
I did not misunderstand you. I meant that Berkeley-DB could not do automagic upgrades of its files in case the application-specific format changes.
IIRC in former times with back-bdb and back-hdb there was also sometimes the need to reload BDB files because the slapd-specific layout changed.
Admittedly I never had or needed those. I only noticed changes of hash functions, and those did not need manual intervention.
Don't confuse the underlying database software (LMDB) with the databse backend (slapd-mdb).
The slapd-mdb database backend had a format change that requires a reload. This is, as Michael already noted, no different than when the slapd-bdb or slapd-hdb backends (which made use of BerkeleyDB) had format changes. Those type of format changes always require a reload regardless of the underlying database software being used.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
Hi all, The 2.4 to 2.5 migration clearly explains the requirement and the rational behind needing to do a slapcat with 2.4 binaries followed by slapadd with 2.5 binaries:
https://www.openldap.org/doc/admin25/appendix-upgrading.html
This can all be automated with the use of Ansible or Puppet (or a long list of other automation tools), however all would require a bit of development work. Seems like Ansible would be a better fit for the upgrade given it's a 1 time change likely spanning multiple nodes in most environments (i.e. Puppet wouldn't be a great fit for the data upgrade bits IMO).
Best, Aaron
-----Original Message----- From: Ulrich Windl Ulrich.Windl@rz.uni-regensburg.de Sent: Monday, August 2, 2021 2:06 AM To: openldap-technical@openldap.org; michael@stroeder.com Subject: Antw: [EXT] migrate from 2.4 to 2.5, determine existing MDB format
Warning: This email is from outside the company. Be careful clicking links or attachments.
Michael Ströder michael@stroeder.com schrieb am 31.07.2021 um 18:05 in
Nachricht 60ec1d1e-a2be-95a2-c9c4-24ecd9b4f58c@stroeder.com:
HI!
As far as I understood the MDB disk format changed. So the MDB files have to be re-created (either by simply removing/replicating or slapcat/slapadd). Right?
AFAIK; Sleepycat DB could do that conversion on the fly. At least there should be some smarter conversion utility than export old DB with old binary and re-import with new binary. MHO.
Regards, Ulrich
Now I'm wondering how to automate things (with ansible and puppet) in a truly idempotent way. Ideally I could determine whether existing MDB files were last maintained by OpenLDAP 2.4.x or whether they already have the 2.5 format.
Can I find out the disk format version in any way, e.g. with python-lmdb?
Ciao, Michael.
---------------------------------------------------------------------- The information contained in this message may be privileged, confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify your representative immediately and delete this message from your computer. Thank you.
--On Saturday, July 31, 2021 7:05 PM +0200 Michael Ströder michael@stroeder.com wrote:
Can I find out the disk format version in any way, e.g. with python-lmdb?
The id2v DB only exists in OpenLDAP 2.5 databases. However, stay tuned...
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
On 8/6/21 11:01 PM, Quanah Gibson-Mount wrote:
--On Saturday, July 31, 2021 7:05 PM +0200 Michael Ströder michael@stroeder.com wrote:
Can I find out the disk format version in any way, e.g. with python-lmdb?
The id2v DB only exists in OpenLDAP 2.5 databases. However, stay tuned...
Ah, this is very helpful:
mdb_stat -s id2v <db-path>
Ciao, Michael.
Michael Ströder wrote:
On 8/6/21 11:01 PM, Quanah Gibson-Mount wrote:
--On Saturday, July 31, 2021 7:05 PM +0200 Michael Ströder michael@stroeder.com wrote:
Can I find out the disk format version in any way, e.g. with python-lmdb?
The id2v DB only exists in OpenLDAP 2.5 databases. However, stay tuned...
Ah, this is very helpful:
mdb_stat -s id2v <db-path>
Just to be clear, the current upgrade doc is a bit paranoid. A 2.4 DB is forward compatible with 2.5. But 2.5 allows you to configure new DB parameters that would make it incompatible with 2.4.
--On Friday, August 6, 2021 11:49 PM +0100 Howard Chu hyc@symas.com wrote:
Michael Ströder wrote:
On 8/6/21 11:01 PM, Quanah Gibson-Mount wrote:
--On Saturday, July 31, 2021 7:05 PM +0200 Michael Ströder michael@stroeder.com wrote:
Can I find out the disk format version in any way, e.g. with python-lmdb?
The id2v DB only exists in OpenLDAP 2.5 databases. However, stay tuned...
Ah, this is very helpful:
mdb_stat -s id2v <db-path>
Just to be clear, the current upgrade doc is a bit paranoid. A 2.4 DB is forward compatible with 2.5. But 2.5 allows you to configure new DB parameters that would make it incompatible with 2.4.
Or another way to put it:
Unless sortvals, multival, or idlexp is being newly configured as a configuration change during the upgrade process, there is no requirement to reload a back-mdb database on reload with OpenLDAP 2.5.6 and prior. However it will not be possible to export this database via slapcat until after an OpenLDAP 2.5 slapd has been executed with the database in place. With OpenLDAP 2.5.7 and later it is possible to export a 2.4 database with slapcat in all circumstances.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
On 8/7/21 12:02 AM, Quanah Gibson-Mount wrote:
--On Friday, August 6, 2021 11:49 PM +0100 Howard Chu hyc@symas.com wrote:
Just to be clear, the current upgrade doc is a bit paranoid. A 2.4 DB is forward compatible with 2.5. But 2.5 allows you to configure new DB parameters that would make it incompatible with 2.4.
Or another way to put it:
Unless sortvals, multival, or idlexp is being newly configured as a configuration change during the upgrade process, there is no requirement to reload a back-mdb database on reload with OpenLDAP 2.5.6 and prior.
Ah, this is good to know.
Thank you both for clarification.
However it will not be possible to export this database via slapcat until after an OpenLDAP 2.5 slapd has been executed with the database in place.
Ok, noted. Easy with Æ-DIR because slapd will be restarted after upgrade during ansible play.
With OpenLDAP 2.5.7 and later it is possible to export a 2.4 database with slapcat in all circumstances.
This will be very helpful because downstream packagers won't have to provide two different versions of slapcat (as currently discussed on opensuse-factory list).
Ciao, Michael.
On 8/7/21 9:58 AM, Michael Ströder wrote:
On 8/7/21 12:02 AM, Quanah Gibson-Mount wrote:
With OpenLDAP 2.5.7 and later it is possible to export a 2.4 database with slapcat in all circumstances.
This will be very helpful because downstream packagers won't have to provide two different versions of slapcat (as currently discussed on opensuse-factory list).
Is it sufficient to use commit d00efea7d19ad339f244f1325883031830125876 as back-port patch?
Ciao, Michael.
Michael Ströder wrote:
On 8/7/21 9:58 AM, Michael Ströder wrote:
On 8/7/21 12:02 AM, Quanah Gibson-Mount wrote:
With OpenLDAP 2.5.7 and later it is possible to export a 2.4 database with slapcat in all circumstances.
This will be very helpful because downstream packagers won't have to provide two different versions of slapcat (as currently discussed on opensuse-factory list).
Is it sufficient to use commit d00efea7d19ad339f244f1325883031830125876 as back-port patch?
backport into 2.5? Yes. But why bother, since 2.5.7 will be available?
Ciao, Michael.
On 8/7/21 1:34 PM, Howard Chu wrote:
Michael Ströder wrote:
On 8/7/21 9:58 AM, Michael Ströder wrote:
On 8/7/21 12:02 AM, Quanah Gibson-Mount wrote:
With OpenLDAP 2.5.7 and later it is possible to export a 2.4 database with slapcat in all circumstances.
This will be very helpful because downstream packagers won't have to provide two different versions of slapcat (as currently discussed on opensuse-factory list).
Is it sufficient to use commit d00efea7d19ad339f244f1325883031830125876 as back-port patch?
backport into 2.5? Yes.
Yes, of course into 2.5.
But why bother, since 2.5.7 will be available?
Well, I didn't want to bother you again with my usual question when the next release will be available. :-)
Ciao, Michael.
--On Saturday, August 7, 2021 3:48 PM +0200 Michael Ströder michael@stroeder.com wrote:
On 8/7/21 1:34 PM, Howard Chu wrote:
Michael Ströder wrote:
On 8/7/21 9:58 AM, Michael Ströder wrote:
On 8/7/21 12:02 AM, Quanah Gibson-Mount wrote:
With OpenLDAP 2.5.7 and later it is possible to export a 2.4 database with slapcat in all circumstances.
This will be very helpful because downstream packagers won't have to provide two different versions of slapcat (as currently discussed on opensuse-factory list).
Is it sufficient to use commit d00efea7d19ad339f244f1325883031830125876 as back-port patch?
backport into 2.5? Yes.
Yes, of course into 2.5.
I would also backport:
commit dffa47ed752fd840e7a7cc0d1b95e4474f2de95b Author: Howard Chu hyc@openldap.org Date: Fri Aug 6 22:13:47 2021 +0100
ITS#9611 bconfig: canonicalize structuralObjectclass
until 2.5.7 is available, as that is also for 2.4 to 2.5 upgrade compatibility. It affects the dyngroup, dynlist, and memberof overlays.
--Quanah
--
Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: http://www.symas.com
openldap-technical@openldap.org
-
A. Schulze -
Bliss, Aaron -
Howard Chu -
Michael Ströder -
Quanah Gibson-Mount -
Ulrich Windl