howard, et al, hi,
i notice that LMDB stores data in an "append only" manner (from the Q&A). i was thinking of recommending that in python the buffers be handed around with *direct* and persistent pointers to the data within the LMDB shm/memory-map.
the reason that i believe this will be successful is because the circumstances are rather unusual: once data is entered (added) it is *never* overwritten, updated, changed, or modified. there simply is no need [in this application]. so there will only ever be insertion (once and only once) or controlled and monitored deletion (by age of record).
under these circumstances is it perfectly reasonable to pass around the pointers to the buffers returned from value look-ups *indefinitely*? if so that would be awesome.
l.
Luke Kenneth Casson Leighton wrote:
howard, et al, hi,
i notice that LMDB stores data in an "append only" manner (from the Q&A). i was thinking of recommending that in python the buffers be handed around with *direct* and persistent pointers to the data within the LMDB shm/memory-map.
the reason that i believe this will be successful is because the circumstances are rather unusual: once data is entered (added) it is *never* overwritten, updated, changed, or modified. there simply is no need [in this application]. so there will only ever be insertion (once and only once) or controlled and monitored deletion (by age of record).
under these circumstances is it perfectly reasonable to pass around the pointers to the buffers returned from value look-ups *indefinitely*? if so that would be awesome.
Any time a new value is inserted into an existing page, that page's address will change due to COW. Any time an old page gets sufficiently old that no other txn points to it, it becomes eligible for reuse - thus completely unrelated data could get onto it. So unless you're keeping very close tabs on your application's key+data sizes and know how full a page got, you can't rely on particular addresses staying valid.
l.
On Fri, Oct 17, 2014 at 6:54 PM, Howard Chu hyc@symas.com wrote:
Luke Kenneth Casson Leighton wrote:
howard, et al, hi,
i notice that LMDB stores data in an "append only" manner (from the Q&A). i was thinking of recommending that in python the buffers be handed around with *direct* and persistent pointers to the data within the LMDB shm/memory-map.
the reason that i believe this will be successful is because the circumstances are rather unusual: once data is entered (added) it is *never* overwritten, updated, changed, or modified. there simply is no need [in this application]. so there will only ever be insertion (once and only once) or controlled and monitored deletion (by age of record).
under these circumstances is it perfectly reasonable to pass around the pointers to the buffers returned from value look-ups *indefinitely*? if so that would be awesome.
Any time a new value is inserted into an existing page, that page's address will change due to COW. Any time an old page gets sufficiently old that no other txn points to it, it becomes eligible for reuse - thus completely unrelated data could get onto it. So unless you're keeping very close tabs on your application's key+data sizes and know how full a page got, you can't rely on particular addresses staying valid.
ok. that makes sense. a pity, though, because it would be reaally cool to refer to data by buffer address forever. COW means it's beyond tiblmdb's control, but hey, i tried :)
l.
-
Howard Chu -
Luke Kenneth Casson Leighton