Well... maybe am I a too "old school": I do appreciate lmdb a lot for its clean and simple API, but a parsimonious side of myself is kind of "shocked" demanding 1 TB upfront. After all, my application might not be the only one running on such a system, and there are many technologies using memory mapped files (like Lucene) that might need this addressing space as well. So I have the impression that we might agree to disagree on this point. In any case, thanks again for your support and answers! Very appreciated.
With best regards,
Bruno.
-----Message d'origine----- De : Howard Chu [mailto:hyc@symas.com] Envoyé : jeudi 14 janvier 2016 10:25 À : Bruno Freudensprung bruno.freudensprung@temis.com; openldap-technical@openldap.org Objet : Re: LMBD questions (stat & copy)
Bruno Freudensprung wrote:
Hmmm... doing it this way has several drawbacks (if I am understanding the docs correctly):
- the mutex of the write transactions prevents from writing
simultaneously in my dbis (I am inserting big graphs and sometimes it takes from seconds to minutes to do so, plus those graphs are really independent and modified by independent users)
- the doc of mdb_env_set_maxdbs mentions a cost when there are more
than a "moderate number" of dbis
- and, in the end, this adds another "maximum number of" to the
solution
However don't get me wrong: lmdb has the best seek/get performances I have seen so far... this is why I really would like to use it (iterating over graph edges is so fast with lmdb and its zero copy API design). To me it is a the perfect fit. The last thing I really would like to do is being able to detect when I am getting close to the map size (thus my question on the mt_next_pgno field).
Even so, on a current x86-64 server you have 48 bits of virtual address space - 256 terabytes worth. Even with hundreds of DBs open you can give them a terabyte each and not have to worry about it.
Best regards,
Bruno.
-----Message d'origine----- De : Howard Chu [mailto:hyc@symas.com] Envoyé : mercredi 13 janvier 2016 23:35 À : Bruno Freudensprung bruno.freudensprung@temis.com; openldap-technical@openldap.org Objet : Re: LMBD questions (stat & copy)
Bruno Freudensprung wrote:
Hi Howard,
Thank you very much for your patient answers, and my apologies for my late reply. The "reserved space" was indeed occupied by the FREE_DBI, thanks again for pointing this out for me. Since I will have hundreds of envs (of unpredictable size) in my application,
Sounds like a misuse. For hundreds of tables you should be using individual named DBs within a single env. The point is to use a single env and not have to worry about hundreds of configuration details.
I am a bit reluctant in allocating too much space for all of them, and that's why I was trying to find a way emit a warning when room is about to lack. In this regard I think I have read something very promising in the docs: the mt_next_pgno field of the txn. It looks perfect. Do you think it would be safe to expose it in a read-only way? In any case I will follow you advice and try to find a safe compromise for the map size.