5:49 a.m.
Hallvard Breien Furuseth wrote:
On 22. mai 2017 14:00, Howard Chu wrote:
> Muhammed Muneer wrote:
>> Hallvard wrote
>
>> "With threads 1 and 2 coexisting? When thread 2 called mdb_dbi_open(),
>> thread 1's prospect of using mdb_dbi_open() at all was lost."
>>
>> Yeah with both coexisting. Thats what I thought.
Sorry, I should have said with _transactions_ #1 and #2 coexisting,
_transaction_ #1's prospect of using mdb_dbi_open() at all was lost.
Transaction #3 in thread #1 can use it if it stared after txn#2 ended.
Anyway, just forget about being clever with DBIs. LMDB does not support
DBI cleverness, that's one of its trade-offs for speed and simplicity.
Exactly.
> If you want to use dbi_open in multiple threads then put it in
your own
> wrapper function, protected by a mutex.
Wait, what? mdb_dbi_open() isn't coded to handle concurrent txns
calling it, regardless of any mutexes the caller has. In particular,
it does not use nor update the environment's numdbs, so the two
transactions could end up creating the same DBI for different DBs.
Yes. This is only safe if you also close the dbi in the same txn that opened
it. Which you are probably doing, if you are opening and closing on the fly.
> Naturally you will also have to wrap dbi_close the same way.
If we did support this, txn_commit() of the txn which used dbi_open()
would also need the mutex.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
1:31 a.m.
New subject: mdb_dbi_open and threads
Sorry for the late reply, I was on vacation.
Klaus wrote:
"I am still unsure what you are trying to achieve. If you are in a read
transaction and discover that your database does not exist, what can you do
anyway? You cannot create the database at this point, since it is a write
operation."
When I discover a dbi does not exist in a read transaction, I can assume it
to be the same as a dbi which is empty and create it when and only when
there is a write-request into that dbi later.
Howard wrote:
"If you want to use dbi_open in multiple threads then put it in your own
wrapper function, protected by a mutex."
This defeats the whole purpose of parallel reads.
Hallvard wrote:
"Anyway, just forget about being clever with DBIs. LMDB does not support
DBI cleverness"
There is nothing clever about opening a dbi on the fly. Its a normal
requirement especially when the database handles multiple clients.
Imagine a server database listening in on clients.
The server issues a write request from Client A and dbi_open ABC and is
about to write data.
At the same time in another thread, it issues a read request from Client B
to the same dbi ABC but is
unable to dbi_open ABC.
Isn't this scenario one of the basic requirements of a database listening
in on clients.
On Mon, May 22, 2017 at 6:19 PM, Howard Chu <hyc(a)symas.com> wrote:
Hallvard Breien Furuseth wrote:
> On 22. mai 2017 14:00, Howard Chu wrote:
>
>> Muhammed Muneer wrote:
>>
>>> Hallvard wrote
>>>
>>
>> "With threads 1 and 2 coexisting? When thread 2 called mdb_dbi_open(),
>>> thread 1's prospect of using mdb_dbi_open() at all was lost."
>>>
>>> Yeah with both coexisting. Thats what I thought.
>>>
>>
> Sorry, I should have said with _transactions_ #1 and #2 coexisting,
> _transaction_ #1's prospect of using mdb_dbi_open() at all was lost.
> Transaction #3 in thread #1 can use it if it stared after txn#2 ended.
>
> Anyway, just forget about being clever with DBIs. LMDB does not support
> DBI cleverness, that's one of its trade-offs for speed and simplicity.
>
Exactly.
If you want to use dbi_open in multiple threads then put it in your own
>> wrapper function, protected by a mutex.
>>
>
> Wait, what? mdb_dbi_open() isn't coded to handle concurrent txns
> calling it, regardless of any mutexes the caller has. In particular,
> it does not use nor update the environment's numdbs, so the two
> transactions could end up creating the same DBI for different DBs.
>
Yes. This is only safe if you also close the dbi in the same txn that
opened it. Which you are probably doing, if you are opening and closing on
the fly.
>
> Naturally you will also have to wrap dbi_close the same way.
>>
>
> If we did support this, txn_commit() of the txn which used dbi_open()
> would also need the mutex.
>
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
2:01 a.m.
New subject: mdb_dbi_open and threads
On 6/19/17 10:31 AM, Muhammed Muneer wrote:
Sorry for the late reply, I was on vacation.
Klaus wrote:
"I am still unsure what you are trying to achieve. If you are in a read
transaction and discover that your database does not exist, what can you do
anyway? You cannot create the database at this point, since it is a write
operation."
When I discover a dbi does not exist in a read transaction, I can assume it to
be the same as a dbi which is empty and create it when and only when there is a
write-request into that dbi later.
To my understanding, the API does not prevent you from doing so. As long as you
do not specify the MDB_CREATE flag, you can call mdb_dbi_open and will get
informed whether the database exists or not (MDB_NOTFOUND).
Klaus
7:52 a.m.
New subject: mdb_dbi_open and threads
I know. But that is not my problem.
On Mon, Jun 19, 2017 at 2:31 PM, Klaus Malorny <Klaus.Malorny(a)knipp.de>
wrote:
On 6/19/17 10:31 AM, Muhammed Muneer wrote:
> Sorry for the late reply, I was on vacation.
>
> Klaus wrote:
> "I am still unsure what you are trying to achieve. If you are in a read
> transaction and discover that your database does not exist, what can you do
> anyway? You cannot create the database at this point, since it is a write
> operation."
>
> When I discover a dbi does not exist in a read transaction, I can assume
> it to be the same as a dbi which is empty and create it when and only when
> there is a write-request into that dbi later.
>
>
>
To my understanding, the API does not prevent you from doing so. As long
as you do not specify the MDB_CREATE flag, you can call mdb_dbi_open and
will get informed whether the database exists or not (MDB_NOTFOUND).
Klaus
8:31 p.m.
New subject: mdb_dbi_open and threads
Sorry for the late reply.
I finally went with the solution Klaus Malorny stated. Storing all the data
in in one big database named "main" and prep-ending a 4 byte int (database
id or collection id) to the key.
These collection ids are stored in a seperate database named "collections".
And the query handler has been modified to take care of the "range" queries
that corresponds to a logical database.
Thanks for all the response and suggestions.
On Mon, Jun 19, 2017 at 6:52 PM, Muhammed Muneer <elendilm(a)gmail.com> wrote:
I know. But that is not my problem.
On Mon, Jun 19, 2017 at 2:31 PM, Klaus Malorny <Klaus.Malorny(a)knipp.de>
wrote:
> On 6/19/17 10:31 AM, Muhammed Muneer wrote:
>
>> Sorry for the late reply, I was on vacation.
>>
>> Klaus wrote:
>> "I am still unsure what you are trying to achieve. If you are in a read
>> transaction and discover that your database does not exist, what can you do
>> anyway? You cannot create the database at this point, since it is a write
>> operation."
>>
>> When I discover a dbi does not exist in a read transaction, I can assume
>> it to be the same as a dbi which is empty and create it when and only when
>> there is a write-request into that dbi later.
>>
>>
>>
> To my understanding, the API does not prevent you from doing so. As long
> as you do not specify the MDB_CREATE flag, you can call mdb_dbi_open and
> will get informed whether the database exists or not (MDB_NOTFOUND).
>
> Klaus
>
>
2135
days inactive
2209
days old
openldap-technical@openldap.org
4 comments
3 participants
participants (3)
-
Howard Chu -
Klaus Malorny -
Muhammed Muneer