Hello,
I successfully ran a multi-threaded application using LMDB 0.9.10-1 on a 64-bit Ubuntu 14.04 machine with 4x10-core Xeon E5-2660 v3 CPUs and 132 GB RAM. I then attempted to run the same application on a CentOS 6.7 box with 2x4-core i7-4790K CPUs and 32 GB RAM, where I tried two versions of LMDB: first EPEL RPM package version 0.9.18-1, and then the latest version 0.9.70 which I downloaded and compiled from GitHub sources on 4/27/2016. On the CentOS system with fewer cores, my application started to hang. I have traced the problem to a periodic commit, after which a new transaction must be created. I commit periodically to avoid a long delay when the application shuts down, at which point a 2 GB database is flushed to disk.
I have a single write transaction, which is protected by a mutex when calling mdb_txn_commit(), mdb_txn_begin() and mdb_put(). The threads are created by TBB using parallel_for, and each thread has access to the single MDB_dbi and associated MDB_txn for writing. When a counter hits a certain value, we lock a mutex and call mdb_txn_commit(), and then we create a new MDB_txn to replace the invalidated one. Other threads try to acquire the same mutex before attempting to call mdb_put(). After a new write transaction is created, the next call the mdb_put() works fine on my 40-core Ubuntu box and the database is created successfully, but the application hangs on my 8-core CentOS box. My only guess is that I am encountering a race condition that is somehow avoided on the faster machine.
Is the above valid LMDB usage? I have tried to find information about multi-threaded writing, and I have not found any definitive answers. I would be happy to share the relevant code, but I need to first strip out a great deal of project-specific ancillary code, which will take some time.
Thanks,
Alex
Yahoo! wrote:
Hello,
I successfully ran a multi-threaded application using LMDB 0.9.10-1 on a
64-bit Ubuntu 14.04 machine with 4x10-core Xeon E5-2660 v3 CPUs and 132 GB RAM. I then attempted to run the same application on a CentOS 6.7 box with 2x4-core i7-4790K CPUs and 32 GB RAM, where I tried two versions of LMDB: first EPEL RPM package version 0.9.18-1, and then the latest version 0.9.70 which I downloaded and compiled from GitHub sources on 4/27/2016. On the CentOS system with fewer cores, my application started to hang. I have traced the problem to a periodic commit, after which a new transaction must be created. I commit periodically to avoid a long delay when the application shuts down, at which point a 2 GB database is flushed to disk.
I have a single write transaction, which is protected by a mutex when
calling mdb_txn_commit(), mdb_txn_begin() and mdb_put(). The threads are created by TBB using parallel_for, and each thread has access to the single MDB_dbi and associated MDB_txn for writing. When a counter hits a certain value, we lock a mutex and call mdb_txn_commit(), and then we create a new MDB_txn to replace the invalidated one. Other threads try to acquire the same mutex before attempting to call mdb_put(). After a new write transaction is created, the next call the mdb_put() works fine on my 40-core Ubuntu box and the database is created successfully, but the application hangs on my 8-core CentOS box. My only guess is that I am encountering a race condition that is somehow avoided on the faster machine.
Is the above valid LMDB usage?
No.
I have tried to find information about
multi-threaded writing, and I have not found any definitive answers.
http://lmdb.tech/doc/group__mdb.html#gad7ea55da06b77513609efebd44b26920
I don't know where you've been looking, but it's plainly stated in the docs.
I would
be happy to share the relevant code, but I need to first strip out a great deal of project-specific ancillary code, which will take some time.
Thanks,
Alex
Thank you for your reply. The documentation is not clear. It states:
"A transaction and its cursors must only be used by a single thread, and a thread may only have a single transaction at a time."
When you protect the transaction with a mutex, only one thread uses it at a time, and only a single write transaction exists for the entire process.
Thanks,
Alex
On Jun 8, 2016, at 2:45 PM, Howard Chu hyc@symas.com wrote:
Yahoo! wrote:
Hello,
I successfully ran a multi-threaded application using LMDB 0.9.10-1 on a
64-bit Ubuntu 14.04 machine with 4x10-core Xeon E5-2660 v3 CPUs and 132 GB RAM. I then attempted to run the same application on a CentOS 6.7 box with 2x4-core i7-4790K CPUs and 32 GB RAM, where I tried two versions of LMDB: first EPEL RPM package version 0.9.18-1, and then the latest version 0.9.70 which I downloaded and compiled from GitHub sources on 4/27/2016. On the CentOS system with fewer cores, my application started to hang. I have traced the problem to a periodic commit, after which a new transaction must be created. I commit periodically to avoid a long delay when the application shuts down, at which point a 2 GB database is flushed to disk.
I have a single write transaction, which is protected by a mutex when
calling mdb_txn_commit(), mdb_txn_begin() and mdb_put(). The threads are created by TBB using parallel_for, and each thread has access to the single MDB_dbi and associated MDB_txn for writing. When a counter hits a certain value, we lock a mutex and call mdb_txn_commit(), and then we create a new MDB_txn to replace the invalidated one. Other threads try to acquire the same mutex before attempting to call mdb_put(). After a new write transaction is created, the next call the mdb_put() works fine on my 40-core Ubuntu box and the database is created successfully, but the application hangs on my 8-core CentOS box. My only guess is that I am encountering a race condition that is somehow avoided on the faster machine.
Is the above valid LMDB usage?
No.
I have tried to find information about
multi-threaded writing, and I have not found any definitive answers.
http://lmdb.tech/doc/group__mdb.html#gad7ea55da06b77513609efebd44b26920
I don't know where you've been looking, but it's plainly stated in the docs.
I would
be happy to share the relevant code, but I need to first strip out a great deal of project-specific ancillary code, which will take some time.
Thanks,
Alex
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Eskandar Ensafi wrote:
Thank you for your reply. The documentation is not clear. It states:
"A transaction and its cursors must only be used by a single thread, and a
thread may only have a single transaction at a time."
When you protect the transaction with a mutex, only one thread uses it at a
time, and only a single write transaction exists for the entire process.
No. The documentation states EXACTLY: "a transaction must only be used by a single thread."
It does *NOT* say "one thread at a time."
Read more carefully.
Thanks,
Alex
On Jun 8, 2016, at 2:45 PM, Howard Chu hyc@symas.com wrote:
Yahoo! wrote:
Hello,
I successfully ran a multi-threaded application using LMDB 0.9.10-1 on a
64-bit Ubuntu 14.04 machine with 4x10-core Xeon E5-2660 v3 CPUs and 132 GB RAM. I then attempted to run the same application on a CentOS 6.7 box with 2x4-core i7-4790K CPUs and 32 GB RAM, where I tried two versions of LMDB: first EPEL RPM package version 0.9.18-1, and then the latest version 0.9.70 which I downloaded and compiled from GitHub sources on 4/27/2016. On the CentOS system with fewer cores, my application started to hang. I have traced the problem to a periodic commit, after which a new transaction must be created. I commit periodically to avoid a long delay when the application shuts down, at which point a 2 GB database is flushed to disk.
I have a single write transaction, which is protected by a mutex when
calling mdb_txn_commit(), mdb_txn_begin() and mdb_put(). The threads are created by TBB using parallel_for, and each thread has access to the single MDB_dbi and associated MDB_txn for writing. When a counter hits a certain value, we lock a mutex and call mdb_txn_commit(), and then we create a new MDB_txn to replace the invalidated one. Other threads try to acquire the same mutex before attempting to call mdb_put(). After a new write transaction is created, the next call the mdb_put() works fine on my 40-core Ubuntu box and the database is created successfully, but the application hangs on my 8-core CentOS box. My only guess is that I am encountering a race condition that is somehow avoided on the faster machine.
Is the above valid LMDB usage?
No.
I have tried to find information about
multi-threaded writing, and I have not found any definitive answers.
http://lmdb.tech/doc/group__mdb.html#gad7ea55da06b77513609efebd44b26920
I don't know where you've been looking, but it's plainly stated in the docs.
I would
be happy to share the relevant code, but I need to first strip out a great deal of project-specific ancillary code, which will take some time.
Thanks,
Alex
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
So is the correct usage to have a single thread create the transaction, and the same thread that created it waits for some kind of event to occur indicating the data has arrived, and then it adds data to the database and waits for the next event? Basically, it seems that only one thread can be in charge of putting and committing.
On Jun 8, 2016, at 4:36 PM, Howard Chu hyc@symas.com wrote:
Eskandar Ensafi wrote:
Thank you for your reply. The documentation is not clear. It states:
"A transaction and its cursors must only be used by a single thread, and a
thread may only have a single transaction at a time."
When you protect the transaction with a mutex, only one thread uses it at a
time, and only a single write transaction exists for the entire process.
No. The documentation states EXACTLY: "a transaction must only be used by a single thread."
It does *NOT* say "one thread at a time."
Read more carefully.
Thanks,
Alex
On Jun 8, 2016, at 2:45 PM, Howard Chu hyc@symas.com wrote:
Yahoo! wrote:
Hello,
I successfully ran a multi-threaded application using LMDB 0.9.10-1 on a
64-bit Ubuntu 14.04 machine with 4x10-core Xeon E5-2660 v3 CPUs and 132 GB RAM. I then attempted to run the same application on a CentOS 6.7 box with 2x4-core i7-4790K CPUs and 32 GB RAM, where I tried two versions of LMDB: first EPEL RPM package version 0.9.18-1, and then the latest version 0.9.70 which I downloaded and compiled from GitHub sources on 4/27/2016. On the CentOS system with fewer cores, my application started to hang. I have traced the problem to a periodic commit, after which a new transaction must be created. I commit periodically to avoid a long delay when the application shuts down, at which point a 2 GB database is flushed to disk.
I have a single write transaction, which is protected by a mutex when
calling mdb_txn_commit(), mdb_txn_begin() and mdb_put(). The threads are created by TBB using parallel_for, and each thread has access to the single MDB_dbi and associated MDB_txn for writing. When a counter hits a certain value, we lock a mutex and call mdb_txn_commit(), and then we create a new MDB_txn to replace the invalidated one. Other threads try to acquire the same mutex before attempting to call mdb_put(). After a new write transaction is created, the next call the mdb_put() works fine on my 40-core Ubuntu box and the database is created successfully, but the application hangs on my 8-core CentOS box. My only guess is that I am encountering a race condition that is somehow avoided on the faster machine.
Is the above valid LMDB usage?
No.
I have tried to find information about
multi-threaded writing, and I have not found any definitive answers.
http://lmdb.tech/doc/group__mdb.html#gad7ea55da06b77513609efebd44b26920
I don't know where you've been looking, but it's plainly stated in the docs.
I would
be happy to share the relevant code, but I need to first strip out a great deal of project-specific ancillary code, which will take some time.
Thanks,
Alex
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
On 09/06/16 02:46, Eskandar Ensafi wrote:
So is the correct usage to have a single thread create the transaction, and the same thread that created it waits for some kind of event to occur indicating the data has arrived, and then it adds data to the database and waits for the next event?
Swap the order to avoid long-lived transactions, like Caveats in lmdb.h says. I.e. wait for some work to do, start a transaction, write whatever to the database, commit.
Basically, it seems that only one thread can be in charge of putting and committing.
Well, only one thread *at a time* since there only can be one write-transaction at a time. If a write-transaction exists, mdb_txn_begin(new write-transaction) in another thread/process will wait for the existing write-transaction to commit/abort.
openldap-technical@openldap.org
-
Eskandar Ensafi -
Hallvard Breien Furuseth -
Howard Chu -
Yahoo!