Re: question about use of buffers in lmdb
Luke Kenneth Casson Leighton wrote:
howard, et al, hi,
i notice that LMDB stores data in an "append only" manner (from the
Q&A). i was thinking of recommending that in python the buffers be
handed around with *direct* and persistent pointers to the data within
the LMDB shm/memory-map.
the reason that i believe this will be successful is because the
circumstances are rather unusual: once data is entered (added) it is
*never* overwritten, updated, changed, or modified. there simply is
no need [in this application]. so there will only ever be insertion
(once and only once) or controlled and monitored deletion (by age of
record).
under these circumstances is it perfectly reasonable to pass around
the pointers to the buffers returned from value look-ups
*indefinitely*? if so that would be awesome.
Any time a new value is inserted into an existing page, that page's address
will change due to COW. Any time an old page gets sufficiently old that no
other txn points to it, it becomes eligible for reuse - thus completely
unrelated data could get onto it. So unless you're keeping very close tabs on
your application's key+data sizes and know how full a page got, you can't rely
on particular addresses staying valid.
l.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/