Hi all,
First, I'm having trouble finding resources to answer a question like this
myself, so please forgive me if I've missed something.
I'm considering using LMDB (versus LevelDB) for a project I'm working on
where I'll be receiving a high volume (hundreds per second) of high
priority requests (over HTTP) and issuing multiple (<10) database queries
per request.
I'll also have a separate process receiving updates for the data and
writing to the database. This will happen often (several times a minute,
perhaps), but the priority is much lower than the read requests.
LMDB appealed to me because of the read performance and that I could have
one processing reading data from LMDB and another process writing data
updates to LMDB.
For proof of concept, I hacked up the following (I'll use pseudocode since
I used the Go bindings for my actual programs, and hopefully my question is
sufficiently abstract not to matter):
Process 1, the writer, simply writes a random integer (from 0 to 1000) to a
defined set of keys:
env = NewEnv()
env.Open("/tmp/foo", 0, 0664)
txn = env.BeginTxn(nil, 0)
dbi = txn.DBIOpen(nil, 0)
txn.Commit()
txn = env.BeginTxn(nil, 0)
n_entries = 5
for i = 0; i < n_entries; i++ {
key = sprintf("Key-%d", i)
val = sprintf("Val-%d", rand.Int(1000))
txn.Put(dbi, key, val, 0)
}
txn.Commit()
env.DBIClose(dbi)
env.Close()
Process 2, the reader, simply loops forever and does random access reads on
the data from process 1 (I won't benefit from a cursor for my actual
problem), and prints out that data occasionally:
env = NewEnv()
env.Open("/tmp/foo", 0, 0664)
while {
txn = BeginTxn(nil, 0)
dbi = txn.DBIOpen(nil, 0)
txn.Commit()
for i = 0; i < n_entries; i++ {
key = sprintf("Key-%d", i)
val = txn.Get(dbi, key)
print("%s: %s", key, value)
}
env.DBIClose(dbi)
sleep(5)
}
So my high level question is: What am I doing wrong? This seems to work
OK, but a lot of it was guesswork, so I'm sure I'm doing some silly things.
For example, first I put the BeginTxn() and DBIOpen() calls in process 2
outside of the while loop, but when I did that, I never saw the updates
values upon running process 1 simultaneously. In my real-world
application, it seems like adding these calls to every request (to be sure
the data being read is up-to-date) could be an unnecessary performance
penalty.
I was suspect there are flags that I can/should be using, but I'm not sure.
Thanks for any input.
Brian