Hi Howard,

Thank you very much for your patient answers, and my apologies for my late reply. The "reserved space" was indeed occupied by the FREE_DBI, thanks again for pointing this out for me. Since I will have hundreds of envs (of unpredictable size) in my application, I am a bit reluctant in allocating too much space for all of them, and that's why I was trying to find a way emit a warning when room is about to lack. In this regard I think I have read something very promising in the docs: the mt_next_pgno field of the txn. It looks perfect. Do you think it would be safe to expose it in a read-only way? In any case I will follow you advice and try to find a safe compromise for the map size.

With best regards,

Bruno. ________________________________________ De : openldap-technical [openldap-technical-bounces@openldap.org] de la part de Howard Chu [hyc@symas.com] Envoyé : jeudi 7 janvier 2016 22:25 À : Bruno Freudensprung; openldap-technical@openldap.org Objet : Re: LMBD questions (stat & copy)

Bruno Freudensprung wrote:

Hi,

I am still experimenting on this area and noticed that I can actually put more data in LMDB by commiting after each put (for the sake of the test) than by doing one commit in the middle of the data set. In the end, in terms of amount of data inserted into the database, the big picture is:

1 commit at the end > 1 commit after each put > 1 commit in the middle of

the data set + 1 commit at the end

And mdb_stat can't really help anticipating MDB_MAP_FULL (maybe that wasn't its purpose).

The docs tell you to configure a large mapsize and go about your business. Doing anything else is a waste of time.

Anyway, if anybody had a hint about the maximum "reserved pages" (for internal purpose) that I could subtract from the map size, I would be very grateful to hear about it.

Your sample code gives useless results because you're only counting pages in the Main DB. As plainly stated in the LMDB documentation, there is also a Free DB which is used to track free pages, and it obviously consumes pages itself in order to store the free page info.

With best regards,

Bruno (lmdb enthousiast).

---

*De :* openldap-technical [openldap-technical-bounces@openldap.org] de la part de Bruno Freudensprung [bruno.freudensprung@temis.com] *Envoyé :* jeudi 31 décembre 2015 10:02 *À :* openldap-technical@openldap.org *Objet :* LMBD questions (stat & copy)

Hi,

I am trying to detect MDB_MAP_FULL based on mdb_stat(txn, dbi, &stat) results and I am wondering if I am on the right track or doing things correctly. I have a small and simple test program (http://pastebin.com/SPCYgWMC) exhibiting the following behavior (consistent between Windows 7 64-bit and Ubuntu 15.10 64-bit - latest version of lmdb cloned and recompiled on Linux):

• Inserting 20990 key value pairs in a database is OK when using one transaction. Just before commiting the write transaction, mdb_stat(txn, dbi, &stat) shows that the total number of pages (253) is very close the the number of pages of the map (256). Indeed, adding another entry leads to MDB_MAP_FULL. To put it shortly, this scenario looks OK (I am assuming there might be 3 "internal-purpose" pages not shown by mdb_stat) and can be tested by setting the stat_test variable to 0 at line 20 of the test program.
• Inserting same 20990 key value pairs in a database fails (MDB_MAP_FULL encountered at 19357) when using two consecutive write transactions. Just before MDB_MAP_FULL, mdb_stat(txn, dbi, &stat) shows that the total number of pages (232) is "quite far" from the the number of pages of the map (256), leading my MDB_MAP_FULL detection heuristic to fail. It means that only 92% of the total amount of data can be sucessfully added to the database. Other scenarii (with same key/value lengths though) stop at 85% only. This can be tested by setting the stat_test variable to 1 (line 20).

By "total number of pages" I mean ms_branch_pages + ms_leaf_pages + ms_overflow_pages.

Do you think I am on a wrong track? (maybe mdb_stat does not show 24 "internal-purpose" pages in the second case? or I am forgetting something?).

One last remark: it seems that mdb_env_copy(db, copy_dir) is possible during a write transaction on Windows (like stated in the documentation), but not on Linux (the test program seems to be stopped on a mutex at line 62). This can be tested by changing the copy_test variable to 1 at line 19. Is it expected given my program?

Best regards,

Bruno.

-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/

Hmmm... doing it this way has several drawbacks (if I am understanding the docs correctly): - the mutex of the write transactions prevents from writing simultaneously in my dbis (I am inserting big graphs and sometimes it takes from seconds to minutes to do so, plus those graphs are really independent and modified by independent users) - the doc of mdb_env_set_maxdbs mentions a cost when there are more than a "moderate number" of dbis - and, in the end, this adds another "maximum number of" to the solution

However don't get me wrong: lmdb has the best seek/get performances I have seen so far... this is why I really would like to use it (iterating over graph edges is so fast with lmdb and its zero copy API design). To me it is a the perfect fit. The last thing I really would like to do is being able to detect when I am getting close to the map size (thus my question on the mt_next_pgno field).

Best regards,

Bruno.

-----Message d'origine----- De : Howard Chu [mailto:hyc@symas.com] Envoyé : mercredi 13 janvier 2016 23:35 À : Bruno Freudensprung bruno.freudensprung@temis.com; openldap-technical@openldap.org Objet : Re: LMBD questions (stat & copy)

Bruno Freudensprung wrote:

Hi Howard,

Thank you very much for your patient answers, and my apologies for my late reply. The "reserved space" was indeed occupied by the FREE_DBI, thanks again for pointing this out for me. Since I will have hundreds of envs (of unpredictable size) in my application,

Sounds like a misuse. For hundreds of tables you should be using individual named DBs within a single env. The point is to use a single env and not have to worry about hundreds of configuration details.

I am a bit reluctant in allocating too much space for all of them, and that's why I was trying to find a way emit a warning when room is about to lack. In this regard I think I have read something very promising in the docs: the mt_next_pgno field of the txn. It looks perfect. Do you think it would be safe to expose it in a read-only way? In any case I will follow you advice and try to find a safe compromise for the map size.

Well... maybe am I a too "old school": I do appreciate lmdb a lot for its clean and simple API, but a parsimonious side of myself is kind of "shocked" demanding 1 TB upfront. After all, my application might not be the only one running on such a system, and there are many technologies using memory mapped files (like Lucene) that might need this addressing space as well. So I have the impression that we might agree to disagree on this point. In any case, thanks again for your support and answers! Very appreciated.

With best regards,

Bruno.

-----Message d'origine----- De : Howard Chu [mailto:hyc@symas.com] Envoyé : jeudi 14 janvier 2016 10:25 À : Bruno Freudensprung bruno.freudensprung@temis.com; openldap-technical@openldap.org Objet : Re: LMBD questions (stat & copy)

Bruno Freudensprung wrote:

Hmmm... doing it this way has several drawbacks (if I am understanding the docs correctly):

• the mutex of the write transactions prevents from writing

simultaneously in my dbis (I am inserting big graphs and sometimes it takes from seconds to minutes to do so, plus those graphs are really independent and modified by independent users)

• the doc of mdb_env_set_maxdbs mentions a cost when there are more

than a "moderate number" of dbis

• and, in the end, this adds another "maximum number of" to the

solution

However don't get me wrong: lmdb has the best seek/get performances I have seen so far... this is why I really would like to use it (iterating over graph edges is so fast with lmdb and its zero copy API design). To me it is a the perfect fit. The last thing I really would like to do is being able to detect when I am getting close to the map size (thus my question on the mt_next_pgno field).

Even so, on a current x86-64 server you have 48 bits of virtual address space - 256 terabytes worth. Even with hundreds of DBs open you can give them a terabyte each and not have to worry about it.

Best regards,

Bruno.

-----Message d'origine----- De : Howard Chu [mailto:hyc@symas.com] Envoyé : mercredi 13 janvier 2016 23:35 À : Bruno Freudensprung bruno.freudensprung@temis.com; openldap-technical@openldap.org Objet : Re: LMBD questions (stat & copy)

Bruno Freudensprung wrote:

...

Hi Howard,

Thank you very much for your patient answers, and my apologies for my late reply. The "reserved space" was indeed occupied by the FREE_DBI, thanks again for pointing this out for me. Since I will have hundreds of envs (of unpredictable size) in my application,

Sounds like a misuse. For hundreds of tables you should be using individual named DBs within a single env. The point is to use a single env and not have to worry about hundreds of configuration details.

...

I am a bit reluctant in allocating too much space for all of them, and that's why I was trying to find a way emit a warning when room is about to lack. In this regard I think I have read something very promising in the docs: the mt_next_pgno field of the txn. It looks perfect. Do you think it would be safe to expose it in a read-only way? In any case I will follow you advice and try to find a safe compromise for the map size.