https://bugs.openldap.org/show_bug.cgi?id=10402
Issue ID: 10402 Summary: Feature request: parameter for mdb_env_copy to exclude databases Product: LMDB Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Keywords: needs_review Severity: normal Priority: --- Component: liblmdb Assignee: bugs@openldap.org Reporter: jeffro256@tutanota.com Target Milestone: ---
## Desired behavior
New overload or API modification to `mdb_env_copy{...}()` which allows passing a list of names of databases to exclude from the environment copy.
## Example use-case
The Monero blockchain database [1] has the option to derived a "pruned" database for space-saving purposes. This removes some ~60% of data which some users may find non-essential. Currently, the pruning code [2] copies each non-pruned table manually, specifying key ordering functions, DB flags, etc. It then drops some entries from the relevant "prunable tables". This, however, adds technical maintenance debt when databases are added/updated. A preferable alternative would be to write high-level modification code once and use an overload of `mdb_env_copy` which excludes copying database that we know in advance we don't want to copy. Then our pruning utility would work agnostic to database changes.
## Why
Adding this functionality ourselves would involve re-writing large portions of `mdb_env_copyfd{0,1}` and `mdb_env_cwalk`, which requires either A) vendoring LMDB, or B) possibly breaking in future updates.
You may understandably be of the opinion that the maintenance burden is an "us problem", and not deem pursuing this feature request worth it, but hopefully you seem the value in this utility. Thanks for y'alls hard work on LMDB.
## Links
[1] https://github.com/monero-project/monero/blob/d32b5bfe18e2f5b979fa8dc3a8966c... [2] https://github.com/monero-project/monero/blob/master/src/blockchain_utilitie...
https://bugs.openldap.org/show_bug.cgi?id=10402
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Target Milestone|--- |1.0.0 Keywords|needs_review |
https://bugs.openldap.org/show_bug.cgi?id=10402
--- Comment #1 from Howard Chu hyc@openldap.org --- (In reply to jeffro256 from comment #0)
## Desired behavior
New overload or API modification to `mdb_env_copy{...}()` which allows passing a list of names of databases to exclude from the environment copy.
Why wouldn't you just mdb_drop() the list of DBs you want to exclude?
https://bugs.openldap.org/show_bug.cgi?id=10402
--- Comment #2 from jeffro256@tutanota.com --- (In reply to Howard Chu from comment #1)
(In reply to jeffro256 from comment #0)
## Desired behavior
New overload or API modification to `mdb_env_copy{...}()` which allows passing a list of names of databases to exclude from the environment copy.
Why wouldn't you just mdb_drop() the list of DBs you want to exclude?
Because that doesn't decrease the physical size of the environment, right? It just adds the pages to the free list.
https://bugs.openldap.org/show_bug.cgi?id=10402
--- Comment #3 from Howard Chu hyc@openldap.org --- (In reply to jeffro256 from comment #2)
(In reply to Howard Chu from comment #1)
(In reply to jeffro256 from comment #0)
## Desired behavior
New overload or API modification to `mdb_env_copy{...}()` which allows passing a list of names of databases to exclude from the environment copy.
Why wouldn't you just mdb_drop() the list of DBs you want to exclude?
Because that doesn't decrease the physical size of the environment, right? It just adds the pages to the free list.
And those free pages will then be reused as new data gets written. So what's the problem?
https://bugs.openldap.org/show_bug.cgi?id=10402
Howard Chu hyc@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |SUSPENDED Status|UNCONFIRMED |RESOLVED
--- Comment #4 from Howard Chu hyc@openldap.org --- No compelling use case
https://bugs.openldap.org/show_bug.cgi?id=10402
Quanah Gibson-Mount quanah@openldap.org changed:
What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |VERIFIED