Assuming I have an LMDB database: to traverse all data in a table I open a
cursor and go over each element in a loop using MDB_NEXT or MDB_NEXT_DUP
: :mdb_cursor_get (cur, &_entry.first, &_entry.second, MDB_NEXT_DUP);
How can one traverse the data in an LMDB table using multiple threads for a
faster overall traversal?
I am assuming a read only transaction and I would like each thread to work
on distinct entries; Like if there are 100 [key, value] and I use 4
threads, thread zero will work on first 25 elements, thread one will work
on the next 25 and so on. 25 doesn't have to be exactly 25 but ideally the
partitioned should be balanced.
A simple solution I designed is to read all keys into a vector first and
then subsequently parallelize with each thread working on a subset of key
from this vector. Each thread will traverse a subset of keys, and for each
key we find the corresponding [key, value] and work on the value.
Is there an alternative ?
Show replies by date