Re: LMDB, virtual machines, and copying mdb files

Sela M. wrote:

Thanks! This is very helpful.

To avoid issues with overwriting the file, would it work if I write a command-line utility that gets exclusive access to the db with mdb_env_excl_lock and then overwrite/move/rename the main mdb file before releasing the lock?

No.

The file is mmap'd, moving or renaming won't change what the map uses. The reader process must close the env on the old file and reopen on the new file. That is the only sequence that will work.

Sela

On Thu, Sep 15, 2022 at 10:00 AM Howard Chu <hyc@symas.com mailto:hyc@symas.com> wrote:

selama@gmail.com <mailto:selama@gmail.com> wrote:
> I am working on developing a new document-oriented (XML+JSON) database, using LMDB as an engine, and I have two questions.
>
> 1. So far, it have been working really smoothly for me. But my one customer so far for the DB is really concerned about running LMDB in a virtual
environment such as Docker, when performing reads and writes. Especially when mounting volumes. Their concern is because of the following caveat:
> "Do not use LMDB databases on remote filesystems, even between processes on the same host. This breaks flock() on some OSes, possibly memory map sync, and
certainly sync between programs on different hosts"
> I think it shouldn't be an issue with Docker, but I want to be certain.

That caveat applies to actual remote filesystems like NFS or SMB.

If you're just using a local filesystem as persistent storage it should be fine. But, LMDB records process IDs in its reader table,
and containers all number their processes starting from 1. So if you have multiple containers accessing the same DB at once, you're
likely to get PID collisions and break reader consistency.

> 2. We have one server that updates the database, and another server with a read-only copy of the same database. Our plan was to simply copy the mdb files
from the update machine to the read only machine, but we noticed that if we copy the file immediately after writing, the copy may end up being corrupted. My
solution was to suspend all writing and wait few minutes before writing, to make sure everything back from memory, and I'm also using the "sync" command
(not sure if it does anything here). It seem to be working, but I wonder if there is a more robust way of doing that? And also, is it safe to overwrite to
the read-only server while it performs read transactions to the current file (or maybe rename it and copy to a new file with the same name)?
>
That's what the mdb_copy command is for...

And no, it is not safe to overwrite the file while any process is reading it.