Russell Haley wrote:
Hello,
I wasn't fully subscribed when I sent this so I'll send it again.
Any input or reference hints would be great. I love reading manuals. :)
Russ
---------- Forwarded message ---------- From: Russell Haley russ.haley@gmail.com Date: Tue, Sep 20, 2016 at 4:52 PM Subject: Using lmdb as a pure in memory store To: openldap-technical@openldap.org
Hi there,
We are currently evaluating in memory key value stores for ~100,000 - 200,000 records in an embedded system. I suggested lmdb but it is being discounted for some reasons I thought I'd validate:
- It is currently thought that a on disk file is REQUIRED for the
system. Does MDB_NOSYNC turn off the disk caching? Can it be run as a pure in-memory database? Could the file not just be mapped to a memdisk?
A file is required. Whether the file is on-disk or not is irrelevant. It can of course be on a RAMdisk (or, in Linux, a tmpfs), which would then make it pure in-memory.
MDB_NOSYNC turns off syncing, that's why it is named that.
- Because these values can come very fast, that the use of a lock
file would cause delay and too much wear on the nand based disk (SSD).
No. That's not how the lock file is used.
I see a no locking option ( MDB_NOLOCK) that would stop a lock file being written.
This option should only be used if you're implementing your own lock manager. Hint - no matter what approach you use, any other lock manager you use will be slower than LMDB's.
Again, another option would be mapping the lock file to a memdisk to handle that?
Yes.
Thanks so much for your response and thanks for the link. I've been pouring over that documentation off and on.
The developer currently doing the prototyping has said he ran into an issue where the use of a very large key value caused the system to slow down (he didn't give specifics and I didn't have time to ask). He said it was a sparsely populated record set of about 100,00 rows but he specifically used a large value for the key as we hope to support UInt64 for keys (I don't know why, apparently we're indexing sub-atomic particles?). I saw a setting for MDB_MAXKEYSIZE which was 0 for computed, but I don't know what that means on an ARM (v6 - 32 bit) system. Would you have any idea what would cause the performance hit he was referring too?
Sorry if my questions are weird, I'm trying to learn C and embedded development through osmosis.
Anyway, on a personal note, I was thoroughly happy to get lmdb working in Lua through
https://github.com/shmul/lightningmdb
and can't wait to try his Mule Round-Robin Database tool
Awesome stuff.
Russ
On Thu, Sep 22, 2016 at 4:25 PM, Howard Chu hyc@symas.com wrote:
Russell Haley wrote:
Hello,
I wasn't fully subscribed when I sent this so I'll send it again.
Any input or reference hints would be great. I love reading manuals. :)
Russ
---------- Forwarded message ---------- From: Russell Haley russ.haley@gmail.com Date: Tue, Sep 20, 2016 at 4:52 PM Subject: Using lmdb as a pure in memory store To: openldap-technical@openldap.org
Hi there,
We are currently evaluating in memory key value stores for ~100,000 - 200,000 records in an embedded system. I suggested lmdb but it is being discounted for some reasons I thought I'd validate:
- It is currently thought that a on disk file is REQUIRED for the
system. Does MDB_NOSYNC turn off the disk caching? Can it be run as a pure in-memory database? Could the file not just be mapped to a memdisk?
A file is required. Whether the file is on-disk or not is irrelevant. It can of course be on a RAMdisk (or, in Linux, a tmpfs), which would then make it pure in-memory.
MDB_NOSYNC turns off syncing, that's why it is named that.
- Because these values can come very fast, that the use of a lock
file would cause delay and too much wear on the nand based disk (SSD).
No. That's not how the lock file is used.
I see a no locking option ( MDB_NOLOCK) that would stop a lock file being written.
This option should only be used if you're implementing your own lock manager. Hint - no matter what approach you use, any other lock manager you use will be slower than LMDB's.
Again, another option would be mapping the lock file to a memdisk to handle that?
Yes.
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
openldap-technical@openldap.org
-
Howard Chu -
Russell Haley