Hallvard Breien Furuseth wrote:
On 22. mai 2017 14:00, Howard Chu wrote:
Muhammed Muneer wrote:
Hallvard wrote
"With threads 1 and 2 coexisting? When thread 2 called mdb_dbi_open(), thread 1's prospect of using mdb_dbi_open() at all was lost."
Yeah with both coexisting. Thats what I thought.
Sorry, I should have said with _transactions_ #1 and #2 coexisting, _transaction_ #1's prospect of using mdb_dbi_open() at all was lost. Transaction #3 in thread #1 can use it if it stared after txn#2 ended.
Anyway, just forget about being clever with DBIs. LMDB does not support DBI cleverness, that's one of its trade-offs for speed and simplicity.
Exactly.
If you want to use dbi_open in multiple threads then put it in your own wrapper function, protected by a mutex.
Wait, what? mdb_dbi_open() isn't coded to handle concurrent txns calling it, regardless of any mutexes the caller has. In particular, it does not use nor update the environment's numdbs, so the two transactions could end up creating the same DBI for different DBs.
Yes. This is only safe if you also close the dbi in the same txn that opened it. Which you are probably doing, if you are opening and closing on the fly.
Naturally you will also have to wrap dbi_close the same way.
If we did support this, txn_commit() of the txn which used dbi_open() would also need the mutex.
Sorry for the late reply, I was on vacation.
Klaus wrote: "I am still unsure what you are trying to achieve. If you are in a read transaction and discover that your database does not exist, what can you do anyway? You cannot create the database at this point, since it is a write operation."
When I discover a dbi does not exist in a read transaction, I can assume it to be the same as a dbi which is empty and create it when and only when there is a write-request into that dbi later.
Howard wrote: "If you want to use dbi_open in multiple threads then put it in your own wrapper function, protected by a mutex."
This defeats the whole purpose of parallel reads.
Hallvard wrote: "Anyway, just forget about being clever with DBIs. LMDB does not support DBI cleverness"
There is nothing clever about opening a dbi on the fly. Its a normal requirement especially when the database handles multiple clients.
Imagine a server database listening in on clients. The server issues a write request from Client A and dbi_open ABC and is about to write data. At the same time in another thread, it issues a read request from Client B to the same dbi ABC but is unable to dbi_open ABC.
Isn't this scenario one of the basic requirements of a database listening in on clients.
On Mon, May 22, 2017 at 6:19 PM, Howard Chu hyc@symas.com wrote:
Hallvard Breien Furuseth wrote:
On 22. mai 2017 14:00, Howard Chu wrote:
Muhammed Muneer wrote:
Hallvard wrote
"With threads 1 and 2 coexisting? When thread 2 called mdb_dbi_open(),
thread 1's prospect of using mdb_dbi_open() at all was lost."
Yeah with both coexisting. Thats what I thought.
Sorry, I should have said with _transactions_ #1 and #2 coexisting, _transaction_ #1's prospect of using mdb_dbi_open() at all was lost. Transaction #3 in thread #1 can use it if it stared after txn#2 ended.
Anyway, just forget about being clever with DBIs. LMDB does not support DBI cleverness, that's one of its trade-offs for speed and simplicity.
Exactly.
If you want to use dbi_open in multiple threads then put it in your own
wrapper function, protected by a mutex.
Wait, what? mdb_dbi_open() isn't coded to handle concurrent txns calling it, regardless of any mutexes the caller has. In particular, it does not use nor update the environment's numdbs, so the two transactions could end up creating the same DBI for different DBs.
Yes. This is only safe if you also close the dbi in the same txn that opened it. Which you are probably doing, if you are opening and closing on the fly.
Naturally you will also have to wrap dbi_close the same way.
If we did support this, txn_commit() of the txn which used dbi_open() would also need the mutex.
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
On 6/19/17 10:31 AM, Muhammed Muneer wrote:
Sorry for the late reply, I was on vacation.
Klaus wrote: "I am still unsure what you are trying to achieve. If you are in a read transaction and discover that your database does not exist, what can you do anyway? You cannot create the database at this point, since it is a write operation."
When I discover a dbi does not exist in a read transaction, I can assume it to be the same as a dbi which is empty and create it when and only when there is a write-request into that dbi later.
To my understanding, the API does not prevent you from doing so. As long as you do not specify the MDB_CREATE flag, you can call mdb_dbi_open and will get informed whether the database exists or not (MDB_NOTFOUND).
Klaus
I know. But that is not my problem.
On Mon, Jun 19, 2017 at 2:31 PM, Klaus Malorny Klaus.Malorny@knipp.de wrote:
On 6/19/17 10:31 AM, Muhammed Muneer wrote:
Sorry for the late reply, I was on vacation.
Klaus wrote: "I am still unsure what you are trying to achieve. If you are in a read transaction and discover that your database does not exist, what can you do anyway? You cannot create the database at this point, since it is a write operation."
When I discover a dbi does not exist in a read transaction, I can assume it to be the same as a dbi which is empty and create it when and only when there is a write-request into that dbi later.
To my understanding, the API does not prevent you from doing so. As long as you do not specify the MDB_CREATE flag, you can call mdb_dbi_open and will get informed whether the database exists or not (MDB_NOTFOUND).
Klaus
Sorry for the late reply.
I finally went with the solution Klaus Malorny stated. Storing all the data in in one big database named "main" and prep-ending a 4 byte int (database id or collection id) to the key. These collection ids are stored in a seperate database named "collections". And the query handler has been modified to take care of the "range" queries that corresponds to a logical database.
Thanks for all the response and suggestions.
On Mon, Jun 19, 2017 at 6:52 PM, Muhammed Muneer elendilm@gmail.com wrote:
I know. But that is not my problem.
On Mon, Jun 19, 2017 at 2:31 PM, Klaus Malorny Klaus.Malorny@knipp.de wrote:
On 6/19/17 10:31 AM, Muhammed Muneer wrote:
Sorry for the late reply, I was on vacation.
Klaus wrote: "I am still unsure what you are trying to achieve. If you are in a read transaction and discover that your database does not exist, what can you do anyway? You cannot create the database at this point, since it is a write operation."
When I discover a dbi does not exist in a read transaction, I can assume it to be the same as a dbi which is empty and create it when and only when there is a write-request into that dbi later.
To my understanding, the API does not prevent you from doing so. As long as you do not specify the MDB_CREATE flag, you can call mdb_dbi_open and will get informed whether the database exists or not (MDB_NOTFOUND).
Klaus
openldap-technical@openldap.org
-
Howard Chu -
Klaus Malorny -
Muhammed Muneer