Assuming I have an LMDB database: to traverse all data in a table I open a cursor and go over each element in a loop using MDB_NEXT or MDB_NEXT_DUP
: :mdb_cursor_get (cur, &_entry.first, &_entry.second, MDB_NEXT_DUP);
How can one traverse the data in an LMDB table using multiple threads for a faster overall traversal?
I am assuming a read only transaction and I would like each thread to work on distinct entries; Like if there are 100 [key, value] and I use 4 threads, thread zero will work on first 25 elements, thread one will work on the next 25 and so on. 25 doesn't have to be exactly 25 but ideally the partitioned should be balanced.
A simple solution I designed is to read all keys into a vector first and then subsequently parallelize with each thread working on a subset of key from this vector. Each thread will traverse a subset of keys, and for each key we find the corresponding [key, value] and work on the value.
Is there an alternative ?
Thx,
Gabriel