Harry B wrote:
I am planning to use LMDB to create a resonably large database, few TBs, >
500mil keys, on a Fusion IO flash storage. Memory to storage ratio of the
available hardware is about 1:10
Assuming the caching of "5 to 10%" of most-frequently-accessed data is good
enough for my use-case, is this a valid/legitimate use of LMDB ? Or am I using
the wrong tool for the job?
The information you've provided isn't relevant to answering the question. The
most important question is, what is your read/write ratio? LMDB is designed
for read-heavy workloads. We've gotten good performance on it even with a 1:50
memory to storage ratio, but that's with an 80:20 read:write ratio.
My other choices are RocksDB (haven't looked at it) or Postgres
limited subset of features), the latter mainly because we already use it
across the company.
My tests shows that RocksDB uses about 3x as much memory as LMDB for a given
workload. Postgres is (at least) an order of magnitude larger/slower. If these
are your only two alternatives, I'd guess that LMDB will be the best choice.
But asking questions like this is pointless. The only way to get a meaningful
answer is to prototype your workload and actually profile it for each of your
potential solutions. Otherwise you're just guessing.
Any advice is appreciated.
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/