Ulrich Windl wrote:
Howard Chu hyc@symas.com schrieb am 08.09.2022 um 01:34 in Nachricht
4bc80e7e-17f9-b385-6b11-2aab806edc43@symas.com:
Steffen Michels wrote:
Hi,
We are considering using the mdb.master3 branch of LMDB, but it is not clear
to me whether the data.mdb format will remain stable. Is there a chance that
another migration of all databases will be required in the future when
switching now?
Yes. It is still unreleased because additional changes to the freelist format are planned, and possibly a few other changes.
In any case, mdb_dump/mdb_load will always work for migrating to a new version.
I think at some point an inplace upgrade would be the way to go. On Linux filesystems, you could even upgrade yout ext3 to btrfs ;-)
That will never happen here. Supporting an in-place upgrade requires the library to support old and new formats simultaneously, which is a waste of RAM. Anything that pushes the object code size of the hot path over 64KB is a non-starter.
Regards, Ulrich
regards,
Steffen
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Den 08.09.2022 kl. 17.30 skrev Howard Chu:
Ulrich Windl wrote:
Howard Chu hyc@symas.com schrieb am 08.09.2022 um 01:34 in Nachricht
4bc80e7e-17f9-b385-6b11-2aab806edc43@symas.com:
Steffen Michels wrote:
Hi,
We are considering using the mdb.master3 branch of LMDB, but it is not clear
to me whether the data.mdb format will remain stable. Is there a chance that
another migration of all databases will be required in the future when
switching now?
Yes. It is still unreleased because additional changes to the freelist format are planned, and possibly a few other changes.
In any case, mdb_dump/mdb_load will always work for migrating to a new version.
I think at some point an inplace upgrade would be the way to go. On Linux filesystems, you could even upgrade yout ext3 to btrfs ;-)
That will never happen here. Supporting an in-place upgrade requires the library to support old and new formats simultaneously, which is a waste of RAM. Anything that pushes the object code size of the hot path over 64KB is a non-starter.
Well - if the old version is kept in it's own memory region the new version will still fit within it's own 64KB region.
If the library should not contain both how would you image an application managing the upgrade by itself. I mean, the symbols are names the same etc. so it is difficult to link with both the old and new version.
Søren Holm wrote:
Den 08.09.2022 kl. 17.30 skrev Howard Chu:
Ulrich Windl wrote:
Howard Chu hyc@symas.com schrieb am 08.09.2022 um 01:34 in Nachricht
4bc80e7e-17f9-b385-6b11-2aab806edc43@symas.com:
Steffen Michels wrote:
Hi,
We are considering using the mdb.master3 branch of LMDB, but it is not clear
to me whether the data.mdb format will remain stable. Is there a chance that
another migration of all databases will be required in the future when
switching now?
Yes. It is still unreleased because additional changes to the freelist format are planned, and possibly a few other changes.
In any case, mdb_dump/mdb_load will always work for migrating to a new version.
I think at some point an inplace upgrade would be the way to go. On Linux filesystems, you could even upgrade yout ext3 to btrfs ;-)
That will never happen here. Supporting an in-place upgrade requires the library to support old and new formats simultaneously, which is a waste of RAM. Anything that pushes the object code size of the hot path over 64KB is a non-starter.
Well - if the old version is kept in it's own memory region the new version will still fit within it's own 64KB region.
If the library should not contain both how would you image an application managing the upgrade by itself. I mean, the symbols are names the same etc. so it is difficult to link with both the old and new version.
A database upgrade is an administrative change, thus handled by the administrative tools mdb_dump/mdb_load. There's no need to handle it in an application's main runtime code.
Den 08.09.2022 kl. 18.20 skrev Howard Chu:
Søren Holm wrote:
Den 08.09.2022 kl. 17.30 skrev Howard Chu:
Ulrich Windl wrote:
> Howard Chu hyc@symas.com schrieb am 08.09.2022 um 01:34 in Nachricht
4bc80e7e-17f9-b385-6b11-2aab806edc43@symas.com:
Steffen Michels wrote:
Hi,
We are considering using the mdb.master3 branch of LMDB, but it is not clear
to me whether the data.mdb format will remain stable. Is there a chance that
another migration of all databases will be required in the future when
switching now?
Yes. It is still unreleased because additional changes to the freelist format are planned, and possibly a few other changes.
In any case, mdb_dump/mdb_load will always work for migrating to a new version.
I think at some point an inplace upgrade would be the way to go. On Linux filesystems, you could even upgrade yout ext3 to btrfs ;-)
That will never happen here. Supporting an in-place upgrade requires the library to support old and new formats simultaneously, which is a waste of RAM. Anything that pushes the object code size of the hot path over 64KB is a non-starter.
Well - if the old version is kept in it's own memory region the new version will still fit within it's own 64KB region.
If the library should not contain both how would you image an application managing the upgrade by itself. I mean, the symbols are names the same etc. so it is difficult to link with both the old and new version.
A database upgrade is an administrative change, thus handled by the administrative tools mdb_dump/mdb_load. There's no need to handle it in an application's main runtime code.
Are you sure about that? Let's say that fx. Firefox of SQlite (v4 will be on top of LMDB afaik) starts using LMDB for internal storage. Asking the user to run the administrative tools is a bit much. A software controlled migration path is crucial.
-- Søren Holm
Søren Holm sgh@sgh.dk schrieb am 08.09.2022 um 20:23 in Nachricht
f094cb45-3c00-785c-ed4d-3c7e78e2cf91@sgh.dk:
Den 08.09.2022 kl. 18.20 skrev Howard Chu:
Søren Holm wrote:
Den 08.09.2022 kl. 17.30 skrev Howard Chu:
Ulrich Windl wrote:
>> Howard Chu hyc@symas.com schrieb am 08.09.2022 um 01:34 in
Nachricht
4bc80e7e-17f9-b385-6b11-2aab806edc43@symas.com:
Steffen Michels wrote: > Hi, > > We are considering using the mdb.master3 branch of LMDB, but it is not
clear
to me whether the data.mdb format will remain stable. Is there a chance
that
> another migration of all databases will be required in the future
when
switching now?
Yes. It is still unreleased because additional changes to the freelist format are planned, and possibly a few other changes.
In any case, mdb_dump/mdb_load will always work for migrating to a new version.
I think at some point an inplace upgrade would be the way to go. On Linux filesystems, you could even upgrade yout ext3 to btrfs ;-)
That will never happen here. Supporting an in-place upgrade requires the
library to support
old and new formats simultaneously, which is a waste of RAM. Anything
that
pushes the object
code size of the hot path over 64KB is a non-starter.
Well - if the old version is kept in it's own memory region the new
version
will still fit within it's own 64KB region.
If the library should not contain both how would you image an application
managing the upgrade by itself. I mean, the symbols are names the same etc.
so it is
difficult to link with both the old and new version.
A database upgrade is an administrative change, thus handled by the
administrative tools mdb_dump/mdb_load.
There's no need to handle it in an application's main runtime code.
Are you sure about that? Let's say that fx. Firefox of SQlite (v4 will be on top of LMDB afaik) starts using LMDB for internal storage. Asking the user to run the administrative tools is a bit much. A software controlled migration path is crucial.
Also I think the size of SQlite database as use in Firefox would qualify as "toy database" (a few MB at most, if at all). So I'm surprised that Firefox switches the "database". Real databases have diffetent requirements when extended downtimes are not always an option. Firefox is not a good example here IMHO.
Regards, Ulrich
-- Søren Holm
Den 09.09.2022 kl. 08.09 skrev Ulrich Windl:
Søren Holm sgh@sgh.dk schrieb am 08.09.2022 um 20:23 in Nachricht
f094cb45-3c00-785c-ed4d-3c7e78e2cf91@sgh.dk:
Den 08.09.2022 kl. 18.20 skrev Howard Chu:
Søren Holm wrote:
Den 08.09.2022 kl. 17.30 skrev Howard Chu:
Ulrich Windl wrote:
>>> Howard Chu hyc@symas.com schrieb am 08.09.2022 um 01:34 in
Nachricht
4bc80e7e-17f9-b385-6b11-2aab806edc43@symas.com: > Steffen Michels wrote: >> Hi, >> >> We are considering using the mdb.master3 branch of LMDB, but it is not
clear
> to me whether the data.mdb format will remain stable. Is there a chance
that
>> another migration of all databases will be required in the future
when
> switching now? > > Yes. It is still unreleased because additional changes to the freelist > format are planned, > and possibly a few other changes. > > In any case, mdb_dump/mdb_load will always work for migrating to a new > version.
I think at some point an inplace upgrade would be the way to go. On Linux filesystems, you could even upgrade yout ext3 to btrfs ;-)
That will never happen here. Supporting an in-place upgrade requires the
library to support
old and new formats simultaneously, which is a waste of RAM. Anything
that
pushes the object
code size of the hot path over 64KB is a non-starter.
Well - if the old version is kept in it's own memory region the new
version
will still fit within it's own 64KB region.
If the library should not contain both how would you image an application
managing the upgrade by itself. I mean, the symbols are names the same etc.
so it is
difficult to link with both the old and new version.
A database upgrade is an administrative change, thus handled by the
administrative tools mdb_dump/mdb_load.
There's no need to handle it in an application's main runtime code.
Are you sure about that? Let's say that fx. Firefox of SQlite (v4 will be on top of LMDB afaik) starts using LMDB for internal storage. Asking the user to run the administrative tools is a bit much. A software controlled migration path is crucial.
Also I think the size of SQlite database as use in Firefox would qualify as "toy database" (a few MB at most, if at all). So I'm surprised that Firefox switches the "database". Real databases have diffetent requirements when extended downtimes are not always an option. Firefox is not a good example here IMHO.
It was purely a hypothetical example - firefox does not change db as far as I know. The main point is that whatever an end-user run on his/her pc,phone etc. should handle migrations by itself in software. Users are not database administrators. I'm sure that when SQlite4 is done there will be migration-code in there to enable software developers to handle the migration without the user noticing. How are they sopposed to migrate otherwise?
Regards, Ulrich
-- Søren Holm
On 09/09/2022 08:41, Søren Holm wrote:
Den 09.09.2022 kl. 08.09 skrev Ulrich Windl:
Søren Holm sgh@sgh.dk schrieb am 08.09.2022 um 20:23 in Nachricht
f094cb45-3c00-785c-ed4d-3c7e78e2cf91@sgh.dk:
Den 08.09.2022 kl. 18.20 skrev Howard Chu:
Søren Holm wrote:
Den 08.09.2022 kl. 17.30 skrev Howard Chu:
Ulrich Windl wrote: >>>> Howard Chu hyc@symas.com schrieb am 08.09.2022 um 01:34 in
Nachricht
> 4bc80e7e-17f9-b385-6b11-2aab806edc43@symas.com: >> Steffen Michels wrote: >>> Hi, >>> >>> We are considering using the mdb.master3 branch of LMDB, but >>> it is not
clear
>> to me whether the data.mdb format will remain stable. Is there >> a chance
that
>>> another migration of all databases will be required in the future
when
>> switching now? >> >> Yes. It is still unreleased because additional changes to the >> freelist >> format are planned, >> and possibly a few other changes. >> >> In any case, mdb_dump/mdb_load will always work for migrating >> to a new >> version. > > I think at some point an inplace upgrade would be the way to go. > On Linux filesystems, you could even upgrade yout ext3 to btrfs ;-)
That will never happen here. Supporting an in-place upgrade requires the
library to support
old and new formats simultaneously, which is a waste of RAM. Anything
that
pushes the object
code size of the hot path over 64KB is a non-starter.
Well - if the old version is kept in it's own memory region the new
version
will still fit within it's own 64KB region.
If the library should not contain both how would you image an application
managing the upgrade by itself. I mean, the symbols are names the same etc.
so it is
difficult to link with both the old and new version.
A database upgrade is an administrative change, thus handled by the
administrative tools mdb_dump/mdb_load.
There's no need to handle it in an application's main runtime code.
Are you sure about that? Let's say that fx. Firefox of SQlite (v4 will be on top of LMDB afaik) starts using LMDB for internal storage. Asking the user to run the administrative tools is a bit much. A software controlled migration path is crucial.
Also I think the size of SQlite database as use in Firefox would qualify as "toy database" (a few MB at most, if at all). So I'm surprised that Firefox switches the "database". Real databases have diffetent requirements when extended downtimes are not always an option. Firefox is not a good example here IMHO.
It was purely a hypothetical example - firefox does not change db as far as I know. The main point is that whatever an end-user run on his/her pc,phone etc. should handle migrations by itself in software. Users are not database administrators. I'm sure that when SQlite4 is done there will be migration-code in there to enable software developers to handle the migration without the user noticing. How are they sopposed to migrate otherwise?
Thanks for the answer. For us it is not a problem to bundle mdb_dump/mdb_load with our software and run it automatically when deploying a new version. However, as we have large databases (several terabytes), this will cause some downtime, which we have to justify towards our customers. So we don't want to do such migration too often.
I understand that a "live" migration, so keeping the database usable while migrating, would make LMDB too complex and the code too large. So we rather wait until LMDB 1.0 is stable.
Any indication when the final version of LMDB 1.0 will be released?
Regards, Ulrich
-- Søren Holm
Howard Chu hyc@symas.com schrieb am 08.09.2022 um 17:30 in Nachricht
f331a21a-7062-de6d-73b6-60a925de7cd8@symas.com:
Ulrich Windl wrote:
Howard Chu hyc@symas.com schrieb am 08.09.2022 um 01:34 in Nachricht
4bc80e7e-17f9-b385-6b11-2aab806edc43@symas.com:
Steffen Michels wrote:
Hi,
We are considering using the mdb.master3 branch of LMDB, but it is not clear
to me whether the data.mdb format will remain stable. Is there a chance that
another migration of all databases will be required in the future when
switching now?
Yes. It is still unreleased because additional changes to the freelist format are planned, and possibly a few other changes.
In any case, mdb_dump/mdb_load will always work for migrating to a new version.
I think at some point an inplace upgrade would be the way to go. On Linux filesystems, you could even upgrade yout ext3 to btrfs ;-)
That will never happen here. Supporting an in-place upgrade requires the library to support old and new formats simultaneously, which is a waste of RAM. Anything that pushes the object code size of the hot path over 64KB is a non-starter.
First, the so much criticized bdb could do that also, and second I don't think that a one-time conversion would be part of the code "hot path".
Regards, Ulrich
regards,
Steffen
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
openldap-technical@openldap.org
-
Howard Chu -
Steffen Michels -
Søren Holm -
Ulrich Windl