Harry B wrote:

Hello,

I am planning to use LMDB to create a resonably large database, few TBs, > 500mil keys, on a Fusion IO flash storage. Memory to storage ratio of the available hardware is about 1:10

Assuming the caching of "5 to 10%" of most-frequently-accessed data is good enough for my use-case, is this a valid/legitimate use of LMDB ? Or am I using the wrong tool for the job?

The information you've provided isn't relevant to answering the question. The most important question is, what is your read/write ratio? LMDB is designed for read-heavy workloads. We've gotten good performance on it even with a 1:50 memory to storage ratio, but that's with an 80:20 read:write ratio.

My other choices are RocksDB (haven't looked at it) or Postgres (using a limited subset of features), the latter mainly because we already use it across the company.

My tests shows that RocksDB uses about 3x as much memory as LMDB for a given workload. Postgres is (at least) an order of magnitude larger/slower. If these are your only two alternatives, I'd guess that LMDB will be the best choice.

But asking questions like this is pointless. The only way to get a meaningful answer is to prototype your workload and actually profile it for each of your potential solutions. Otherwise you're just guessing.

Any advice is appreciated.

Thanks

Harry