2:25 p.m.
Hello,
I am planning to use LMDB to create a resonably large database, few TBs, >
500mil keys, on a Fusion IO flash storage. Memory to storage ratio of the
available hardware is about 1:10
Assuming the caching of "5 to 10%" of most-frequently-accessed data is good
enough for my use-case, is this a valid/legitimate use of LMDB ? Or am I
using the wrong tool for the job?
My other choices are RocksDB (haven't looked at it) or Postgres (using a
limited subset of features), the latter mainly because we already use it
across the company.
Any advice is appreciated.
Thanks
--
Harry
4:52 p.m.
Harry B wrote:
Hello,
I am planning to use LMDB to create a resonably large database, few TBs, >
500mil keys, on a Fusion IO flash storage. Memory to storage ratio of the
available hardware is about 1:10
Assuming the caching of "5 to 10%" of most-frequently-accessed data is good
enough for my use-case, is this a valid/legitimate use of LMDB ? Or am I using
the wrong tool for the job?
The information you've provided isn't relevant to answering the question. The
most important question is, what is your read/write ratio? LMDB is designed
for read-heavy workloads. We've gotten good performance on it even with a 1:50
memory to storage ratio, but that's with an 80:20 read:write ratio.
My other choices are RocksDB (haven't looked at it) or Postgres
(using a
limited subset of features), the latter mainly because we already use it
across the company.
My tests shows that RocksDB uses about 3x as much memory as LMDB for a given
workload. Postgres is (at least) an order of magnitude larger/slower. If these
are your only two alternatives, I'd guess that LMDB will be the best choice.
But asking questions like this is pointless. The only way to get a meaningful
answer is to prototype your workload and actually profile it for each of your
potential solutions. Otherwise you're just guessing.
Any advice is appreciated.
Thanks
--
Harry
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
4:05 a.m.
Harry B wrote:
Hello,
I am planning to use LMDB to create a resonably large database, few
TBs, >
500mil keys, on a Fusion IO flash storage. Memory to storage ratio
of the
available hardware is about 1:10
Assuming the caching of "5 to 10%" of
most-frequently-accessed data
is good
enough for my use-case, is this a valid/legitimate use of LMDB ? Or
am I using
the wrong tool for the job?
Benchmarks of various DB engines on Fusion IO storage, with DBs 5x
larger than RAM, available on http://symas.com/mdb/ondisk/
The answer also depends on the size of your data records.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
3186
days inactive
3303
days old
openldap-technical@openldap.org
2 comments
2 participants
participants (2)
-
Harry B -
Howard Chu