Hi,
I'm interested in understanding these functions better. These are not criticisms.. since these functions are called rarely, I have no problem synchronizing them. I ask only for curiosity's sake.
* mdb_env_close: "Only a single thread may call this function." Why can't different environments (separate files/dirs on disk) be closed from different threads?
* mdb_dbi_close: "Handles should only be closed by a single thread." Why can't at least databases on different environments (separate files/dirs on disk) be closed from different threads?
* mdb_dbi_open: "This function must not be called from multiple concurrent transactions in the same process." Why can't this function be called from different threads to open at least databases on different environments (separate files/dirs on disk)?
What are the reasons for these limitations, under the hood? Could these functions have been designed in a different way, to allow for more multithreading (again, I'm not asking for this, I'm just curious).
- Sam
Sam Dave wrote:
Hi,
I'm interested in understanding these functions better. These are not criticisms.. since these functions are called rarely, I have no problem synchronizing them. I ask only for curiosity's sake.
Everything in LMDB is defined within the context of a single environment. That should already be clear from the documentation.
The constraints you reference below only apply within a single environment. Separate environments are completely independent of each other.
mdb_env_close: "Only a single thread may call this function." Why can't different environments (separate files/dirs on disk) be closed from different threads?
mdb_dbi_close: "Handles should only be closed by a single thread." Why can't at least databases on different environments (separate files/dirs on disk) be
closed from different threads?
- mdb_dbi_open: "This function must not be called from multiple concurrent transactions in the same process." Why can't this function be called from different
threads to open at least databases on different environments (separate files/dirs on disk)?
What are the reasons for these limitations, under the hood? Could these functions have been designed in a different way, to allow for more multithreading (again, I'm not asking for this, I'm just curious).
- Sam
Thanks, knowing that (complete independence of environments) makes things easier.
Some follow-ups for clarification,
* Why did "Only a single thread may call this function" have to be mentioned at all for mdb_env_close? It can afaik only be called at most once per environment anyway. (It seems implied by "The environment handle will be freed and must not be used again after this call", and sort of too from the caveats about opening an environment on disk only once).
* About mdb_dbi_close, to "Handles should only be closed by a single thread", would it be worth it to add something like "or mdb_dbi_close should be synchronized, if called from multiple threads" ?
- Sam
Jul 1, 2023, 17:32 by hyc@symas.com:
Sam Dave wrote:
Hi,
I'm interested in understanding these functions better. These are not criticisms.. since these functions are called rarely, I have no problem synchronizing them. I ask only for curiosity's sake.
Everything in LMDB is defined within the context of a single environment. That should already be clear from the documentation.
The constraints you reference below only apply within a single environment. Separate environments are completely independent of each other.
mdb_env_close: "Only a single thread may call this function." Why can't different environments (separate files/dirs on disk) be closed from different threads?
mdb_dbi_close: "Handles should only be closed by a single thread." Why can't at least databases on different environments (separate files/dirs on disk) be
closed from different threads?
- mdb_dbi_open: "This function must not be called from multiple concurrent transactions in the same process." Why can't this function be called from different
threads to open at least databases on different environments (separate files/dirs on disk)?
What are the reasons for these limitations, under the hood? Could these functions have been designed in a different way, to allow for more multithreading (again, I'm not asking for this, I'm just curious).
- Sam
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
I'll rephrase.
For mdb_env_close: "Only a single thread may call this function". Is the meaning same as "For each environment, this function can be called at most once (from any thread)."?
For mdb_dbi_close: "This call is not mutex protected. Handles should only be closed by a single thread". Is the meaning same as "For each handle, this function can be called at most once (from any thread). If you want to close different handles on the same environment from different threads, you need to synchronize their use."?
If someone can confirm my understanding (or correct if wrong), I'd appreciate it.
Jul 2, 2023, 09:46 by samdave@tuta.io:
Thanks, knowing that (complete independence of environments) makes things easier.
Some follow-ups for clarification,
Why did "Only a single thread may call this function" have to be mentioned at all for mdb_env_close? It can afaik only be called at most once per environment anyway. (It seems implied by "The environment handle will be freed and must not be used again after this call", and sort of too from the caveats about opening an environment on disk only once).
About mdb_dbi_close, to "Handles should only be closed by a single thread", would it be worth it to add something like "or mdb_dbi_close should be synchronized, if called from multiple threads" ?
- Sam
Jul 1, 2023, 17:32 by hyc@symas.com:
Sam Dave wrote:
Hi,
I'm interested in understanding these functions better. These are not criticisms.. since these functions are called rarely, I have no problem synchronizing them. I ask only for curiosity's sake.
Everything in LMDB is defined within the context of a single environment. That should already be clear from the documentation.
The constraints you reference below only apply within a single environment. Separate environments are completely independent of each other.
mdb_env_close: "Only a single thread may call this function." Why can't different environments (separate files/dirs on disk) be closed from different threads?
mdb_dbi_close: "Handles should only be closed by a single thread." Why can't at least databases on different environments (separate files/dirs on disk) be
closed from different threads?
- mdb_dbi_open: "This function must not be called from multiple concurrent transactions in the same process." Why can't this function be called from different
threads to open at least databases on different environments (separate files/dirs on disk)?
What are the reasons for these limitations, under the hood? Could these functions have been designed in a different way, to allow for more multithreading (again, I'm not asking for this, I'm just curious).
- Sam
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Does anybody know? (Sorry for the bump)
If I've asked something too stupid/newbie-ish, anyone welcome to let me know.
Jul 5, 2023, 17:43 by samdave@tuta.io:
I'll rephrase.
For mdb_env_close: "Only a single thread may call this function". Is the meaning same as "For each environment, this function can be called at most once (from any thread)."?
For mdb_dbi_close: "> This call is not mutex protected. > Handles should only be closed by a single thread". Is the meaning same as "For each handle, this function can be called at most once (from any thread). If you want to close different handles on the same environment from different threads, you need to synchronize their use."?
If someone can confirm my understanding (or correct if wrong), I'd appreciate it.
Jul 2, 2023, 09:46 by samdave@tuta.io:
Thanks, knowing that (complete independence of environments) makes things easier.
Some follow-ups for clarification,
Why did "Only a single thread may call this function" have to be mentioned at all for mdb_env_close? It can afaik only be called at most once per environment anyway. (It seems implied by "The environment handle will be freed and must not be used again after this call", and sort of too from the caveats about opening an environment on disk only once).
About mdb_dbi_close, to "Handles should only be closed by a single thread", would it be worth it to add something like "or mdb_dbi_close should be synchronized, if called from multiple threads" ?
- Sam
Jul 1, 2023, 17:32 by hyc@symas.com:
Sam Dave wrote:
Hi,
I'm interested in understanding these functions better. These are not criticisms.. since these functions are called rarely, I have no problem synchronizing them. I ask only for curiosity's sake.
Everything in LMDB is defined within the context of a single environment. That should already be clear from the documentation.
The constraints you reference below only apply within a single environment. Separate environments are completely independent of each other.
mdb_env_close: "Only a single thread may call this function." Why can't different environments (separate files/dirs on disk) be closed from different threads?
mdb_dbi_close: "Handles should only be closed by a single thread." Why can't at least databases on different environments (separate files/dirs on disk) be
closed from different threads?
- mdb_dbi_open: "This function must not be called from multiple concurrent transactions in the same process." Why can't this function be called from different
threads to open at least databases on different environments (separate files/dirs on disk)?
What are the reasons for these limitations, under the hood? Could these functions have been designed in a different way, to allow for more multithreading (again, I'm not asking for this, I'm just curious).
- Sam
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
openldap-technical@openldap.org
-
Howard Chu -
Sam Dave