Hi,
Some background; a Windows user of my open source ETL software complained about the huge files. Under Linux I don't have any issues, but I wanted to resolve this issue by implementing automatic growth via a pattern: MapFullError, set_mapsize, apply writing batch again.
This week I entered a bug[1] which was stated as not a bug in LMDB. Today I ended up in build/lib/mdb.c:5966: Assertion 'IS_LEAF(mp)' failed in mdb_cursor_next() which also seem I am not the only one hitting, and several fixes have been introduced.
The issue does not occur every time when the resize takes place, but can be consistently reproduced given the same input. The error will then occur at the same place. When my initial size is big enough, not to resize there is no issue.
In short I am using py-lmdb to access a read only env, and in a second thread a write function is launched which reads out a queue. Upon the MapFullError I am linearly increasing the mapsize, and continue with the failed batch. I have four dupSort tables, the rest is ordinary.
I was pointed in the bug at the documentation. One sentence struck me:
"It may be called at later times if no transactions are active in this process."
My assumption was that if I would have two independent env's I could do whatever I wanted in the first, and the second could do its resize. Does the documentation suggest that the full application must finish all its transactions independent of env?
[1] https://bugs.openldap.org/show_bug.cgi?id=10316
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512
Stefan de Konink wrote:
Hi,
Some background; a Windows user of my open source ETL software complained about the huge files. Under Linux I don't have any issues, but I wanted to resolve this issue by implementing automatic growth via a pattern: MapFullError, set_mapsize, apply writing batch again.
This week I entered a bug[1] which was stated as not a bug in LMDB. Today I ended up in build/lib/mdb.c:5966: Assertion 'IS_LEAF(mp)' failed in mdb_cursor_next() which also seem I am not the only one hitting, and several fixes have been introduced.
The issue does not occur every time when the resize takes place, but can be consistently reproduced given the same input. The error will then occur at the same place. When my initial size is big enough, not to resize there is no issue.
In short I am using py-lmdb to access a read only env, and in a second thread a write function is launched which reads out a queue. Upon the MapFullError I am linearly increasing the mapsize, and continue with the failed batch. I have four dupSort tables, the rest is ordinary.
I was pointed in the bug at the documentation. One sentence struck me:
"It may be called at later times if no transactions are active in this process."
My assumption was that if I would have two independent env's I could do whatever I wanted in the first, and the second could do its resize. Does the documentation suggest that the full application must finish all its transactions independent of env?
Sounds like you're misusing the API still. There is no such thing as 2 independent envs in the same process.
Do not have open an LMDB database twice in the same process at the same time.
- -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
openldap-technical@openldap.org
-
Howard Chu -
Stefan de Konink