Hello,
I've created a program that writes data to an lmdb database using python bindings. I've written about 300gb worth of data but when I was re-reading the database by running my script again, it crashes immediately. Did a backtrace in gdb and this is what I got:
Really have no clue what's going on here. I suspect that the server has rebooted (or was rebooted by someone while I was running the script but I thought lmdb is crash-proof.
Do you have any idea how to best troubleshoot this? The database is worth 300gb+ so it's not easy to share it, not to mention it contains sensitive information.
Starting program: /usr/bin/python3 read_events.py -d jul012022-23/ [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". build/lib/mdb.c:3274: Assertion 'len >= 0 && id <= env->me_pglast' failed in mdb_freelist_save()
Program received signal SIGABRT, Aborted. __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737350283264) at ./nptl/pthread_kill.c:44 44 ./nptl/pthread_kill.c: No such file or directory. (gdb) backtrace #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737350283264) at ./nptl/pthread_kill.c:44 #1 __pthread_kill_internal (signo=6, threadid=140737350283264) at ./nptl/pthread_kill.c:78 #2 __GI___pthread_kill (threadid=140737350283264, signo=signo@entry=6) at ./nptl/pthread_kill.c:89 #3 0x00007ffff7c96476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26 #4 0x00007ffff7c7c7f3 in __GI_abort () at ./stdlib/abort.c:79 #5 0x00007ffff62fb3e2 in mdb_assert_fail (env=0x5555565b49f0, expr_txt=expr_txt@entry=0x7ffff62fe308 "len >= 0 && id <= env->me_pglast", func=func@entry=0x7ffff62fe930 <__func__.13> "mdb_freelist_save", line=line@entry=3274, file=0x7ffff62fe010 "build/lib/mdb.c") at build/lib/mdb.c:1545 #6 0x00007ffff62f10df in mdb_freelist_save (txn=0x555556752950) at build/lib/mdb.c:3274 #7 mdb_txn_commit (txn=0x555556752950) at build/lib/mdb.c:3646 #8 0x00007ffff62f369b in txn_db_from_name (env=env@entry=0x7ffff61c8090, name=<optimized out>, flags=262144) at lmdb/cpython.c:1017 #9 0x00007ffff62f6e1d in env_open_db (self=0x7ffff61c8090, args=<optimized out>, kwds=<optimized out>) at lmdb/cpython.c:1665 #10 0x00005555556b23a9 in ?? () #11 0x0000555555699c14 in _PyEval_EvalFrameDefault () #12 0x0000555555696176 in ?? () #13 0x000055555578bc56 in PyEval_EvalCode () #14 0x00005555557b8b18 in ?? () #15 0x00005555557b196b in ?? () #16 0x00005555557b8865 in ?? () #17 0x00005555557b7d48 in _PyRun_SimpleFileObject () #18 0x00005555557b7a43 in _PyRun_AnyFileObject () #19 0x00005555557a8c3e in Py_RunMain () #20 0x000055555577ebcd in Py_BytesMain () #21 0x00007ffff7c7dd90 in __libc_start_call_main (main=main@entry=0x55555577eb90, argc=argc@entry=14, argv=argv@entry=0x7fffffffe378) at ../sysdeps/nptl/libc_start_call_main.h:58 #22 0x00007ffff7c7de40 in __libc_start_main_impl (main=0x55555577eb90, argc=14, argv=0x7fffffffe378, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffe368) at ../csu/libc-start.c:392 #23 0x000055555577eac5 in _start () (gdb) client_loop: send disconnect: Broken pipe
For what it's worth, this is how I've opened the database environment:
dbenv = lmdb.open(db_dir, map_size=1099511627776, max_dbs=11, readahead=False)