Hi all,
I am a new user to LMDB. Please forgive me for my ignorance or mistakes you
might encounter.
However, I have spent quite some time trying to resolve my issues but
without a success.
That's why I am writing to you asking for your help to shade some light.
To the point:
1) I am having a database where there is only one writer and a few
readers(e.g. 4 readers).
2) The database-writer lives on different thread than the
database-reader(s).
3) The writer-thread keeps on running/writing all the time to the database.
4) At random intervals readers from different threads will request to
extract data from the database.
The problem that occurs is the following: If the writer is active( during
writing), If I try to fetch any data
from the db using the reader, it fails, with error MDB_NOTFOUND. As a
remark here, I make sure that the
reader-thread is launched AFTER the writer thread to ensure that the data
exist in the db.
In order to simulate this kind of behaviour(write-read) over different
threads, I have written a small test application,
that tries to depict my case/usecase.
In main() I simply launch two threads/functions, the writer-thread and
after some time, the reader thread.
The common enviroment is constructed() in main and passed as parameter to
the auxilliary functions.
If the reader starts reading while writing is not finished, then the
application crashes or returns MDB_NOTFOUND.
This contradicts with the capabilities of the library, that is, parallel
write/read from different threads.
If the writer has finished, then the reader works properly.
Some general questions.
i) How many times do i need to open the database? Currently I open the
(unique) database twice: one in the writer and one in the reader.
The doc mentions that it has to be opened only one time. Even in that case,
my app does not work.
ii) What is the practical usage of mdb_txn_reset() and mdb_txn_renew()?
Does it need to be used in my case?
iii) Since I am only ADDING data in the db and only READING them, what kind
of optimizations can be applied?
iv) Regarding for-loops, what is a proper way of using the trasnactions and
commit/abort/reset/etc of transactions?
Please any helpfull information is highly apprecited or if someone can
add some examples with different threads that would be awesome.
#include <iostream>
#include <thread>
#include <assert.h>
#include <string>
#include "lmdb.h"
#include <unistd.h>
using namespace std;
void writeLMDB( MDB_env * env, size_t nrElems )
{
cout << "BEGIN WRITING." << endl;
MDB_txn *txn;
MDB_dbi dbi;
MDB_stat status;
// create transaction
assert( MDB_SUCCESS == mdb_txn_begin( env, NULL, 0, &txn ) );
// open db
assert( MDB_SUCCESS == mdb_open( txn, NULL, MDB_DUPSORT , &dbi ) );
// this commit-transaction is needed since we are creating the db for
the first time
assert( MDB_SUCCESS == mdb_txn_commit( txn ) );
MDB_val key, data;
// begin (again a new transaction)
assert( MDB_SUCCESS == mdb_txn_begin( env, NULL, 0, &txn ) );
// keep on writing elements in the db
for( size_t i = 0; i < nrElems ; ++i )
{
key.mv_size = sizeof( size_t );
key.mv_data = &i;
string value = "test";
data.mv_size = value.size();
data.mv_data = (void*)value.data();
cout << "WRITER-KEY:" << *(size_t*)key.mv_data << " VALUE:" ;
cout.write( (char*)data.mv_data, data.mv_size ); cout << endl;
assert( MDB_SUCCESS == mdb_put( txn, dbi, &key, &data, 0 ) );
}
assert( MDB_SUCCESS == mdb_txn_commit( txn ) );
assert( MDB_SUCCESS == mdb_env_stat( env, &status ) );
mdb_close( env, dbi );
cout << "FINISHED WRITING." << endl;
}
void readLMDB( MDB_env * _env, size_t start, size_t end )
{
cout << "BEGIN READING.." << endl;
MDB_dbi dbi;
MDB_txn *txn = NULL;
// lmdb operations
assert( MDB_SUCCESS == mdb_txn_begin( _env, NULL, MDB_RDONLY, &txn ) );
assert( MDB_SUCCESS == mdb_open( txn, NULL, 0, &dbi ) );
// key+data
MDB_val key, data;
for ( size_t i = start; i <= end ; i++ )
{
key.mv_size = sizeof( size_t );
key.mv_data = &i;
int res = mdb_get( txn, dbi , &key, &data );
//cout << "READER-KEY:" << *(size_t*)key.mv_data << " VALUE:" ;
cout.write( (char*)data.mv_data, data.mv_size ); cout << endl;
if( res != 0 ) cout << "error fetching data" << endl;
}
// close db
mdb_close( _env, dbi );
cout << "FINISHED READING.." << endl;
}
int main(int argc, char *argv[] )
{
// db location(folder must exist)
string dbLocation = "/work/vasilis/databases/lmdbtest/";
size_t maxdbSize = 10485760;
system( "rm -rf /work/vasilis/databases/lmdbtest/*" );
//create enviroment:ONE enviroment only for one process
MDB_env * _env;
assert( MDB_SUCCESS == ( mdb_env_create( &_env ) ) );
//set the size of the database
assert( MDB_SUCCESS == mdb_env_set_mapsize( _env, maxdbSize ) );
// open the enviroment: ready to be used by everyone else
assert( MDB_SUCCESS == mdb_env_open( _env, dbLocation.c_str(), 0 , 0664
) );
// writer-thread: write 1000 elements in the database
thread writethread( writeLMDB, _env, 1000 );
usleep( 1000 * 100 );
// read from - to
thread readthread( readLMDB, _env, 0, 20 );
// wait for threads to finish
writethread.join();
readthread.join();
cout << "THREADS COMPLETED " << endl;
// close the enviroment
mdb_env_close( _env );
return 0;
}
regards,
Vasilis