Thanks a lot! My proof-of-concept code works OK.
I do not understand all subtle details of mmap reliability, could you
please help with these two:
If I write data to a pointer to an opaque blob as discussed above, and my
process crashes before mdb_env_sync, but OS doesn't crash - will that data
be secure in the mmap file?
Also, am I correct that mdb_env_sync synchronizes all dirty pages in the
mmap file as seen by a file system, regardless how they were modified -
either via LMDB API or via a direct pointer writes?
As for "you could at least set a callback to notify you that a block has
moved" - if that is implemented, it would be nice to have a notification
*before* a block is moved (with old and new address, so that right after
the callback it is OK to use the new address), otherwise this non-intended
but convenient use of LMDB won't work anymore.
On Sat, Oct 3, 2015 at 1:27 AM, Howard Chu <hyc(a)symas.com> wrote:
Howard Chu wrote:
> Victor Baybekov wrote:
>> Thank you! I understand this copy-on-write behavior, but am interested
>> if I
>> could control it a little. What if I use records that are always much
>> than a single page, e.g. 100 kb with 4kb pages, and make sure that a
>> record is
>> never updated (via LMDB means) during a lifetime of an environment, - is
>> any scenario that the location of such a big record could be changed
>> during a
>> lifetime of an environment, without updating the record?
> At this point in time, no, if you don't update a large record there is no
> reason that it will move. That is not to say that this won't change in the
> future. The documentation tells you what promises we are willing to make.
> Relying on any non-documented behavior is your own responsibility.
Note that the relocation functions in LMDB are intended to accommodate
blocks being moved around. The actual guts of that API haven't been
implemented, but probably in 1.x we'll flesh them out. Given that support,
you could at least set a callback to notify you that a block has moved. But
currently, overflow pages don't move if they're not modified.
>> On Fri, Oct 2, 2015 at 4:38 PM, Howard Chu <hyc(a)symas.com
>> <mailto:firstname.lastname@example.org>> wrote:
>> Victor Baybekov wrote:
>> Docs for MDB_RESERVE say that a returned pointer to the reserved
>> space is
>> valid "before the next update operation or the transaction
>> ends." Docs
>> for MDB_WRITEMAP say that it "writes directly to the mmap
>> instead of
>> malloc for pages." Does combining the two options return a
>> directly to
>> a place in a mmap
>> so that this pointer could be used after a transaction ends
>> or after the next update?
>> Longer answer: maybe.
>> Full answer: LMDB is copy-on-write. If you update another record on
>> same page, in a later transaction, the contents of that page will be
>> copied to a new page and the original page will go onto the
>> freelist. In
>> that case, the pointer you got must not be used again.
>> If you don't directly update that page and cause it to be copied,
>> then you
>> might get lucky and be able to use the pointer for a while. It all
>> on what other modifications you do and how they affect that node or
>> neighboring nodes.
>> I have a use case where I want to somewhat abuse LMDB safety for
>> If I could get a pointer to a place inside a mmap I could work
>> LMDB value
>> as opaque blob or as a region inside the single big mmap. This
>> be more
>> convenient than creating and opening hundreds of temporary memory
>> mapped files
>> and keeping open handles to them. For example, Aeron terms could
>> like this: a stream id per an LMDB db and a term id for a key in
>> -- Howard Chu
>> CTO, Symas Corp. http://www.symas.com
>> Director, Highland Sun http://highlandsun.com/hyc/
>> Chief Architect, OpenLDAP http://www.openldap.org/project/
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/