Hi,
LMDB seems to be quite the right database I am looking for in order to speed up my application. However, I still need to figure some things out. While I could get many answers to my questions by skimming through the archive, the following things are still unclear to me, and it would be quite nice if someone could clarify them:
- in the documentation, I saw two limits: the key is limited to 511 bytes (by default) and the data is limited to 16777215 bytes. Is this correct? For larger data sizes, is the data still maintained in one contiguous block, so that the application can still access it directly without any need of copying within LMDB? Or it is better to split larger data into multiple key/value pairs?
- I discovered the return code "MDB_TXN_FULL". By what means are transactions limited? I might consider to use few, but quite large transactions dealing with hundreds of thousands key/value pairs to be created, updated or deleted in some circumstances.
- I saw the discussion about the predefined database size, and that it is recommended to use a never-to-be-reached size and to rely on the sparse file capabilities of the underlying OS. If I would not want to go this way: Is it correct that if I encounter an "MDB_MAP_FULL" return code, it would be sufficient to abort the current transaction, resize the database using mdp_env_set_mapsize and repeat the transaction?
Thanks in advance and regards,
Klaus
Klaus Malorny wrote:
Hi,
LMDB seems to be quite the right database I am looking for in order to speed up my application. However, I still need to figure some things out. While I could get many answers to my questions by skimming through the archive, the following things are still unclear to me, and it would be quite nice if someone could clarify them:
- in the documentation, I saw two limits: the key is limited to 511 bytes (by default) and the data is limited to 16777215 bytes. Is this correct?
No, the data size is limited to 4294967295 bytes.
For larger data sizes, is the data still maintained in one contiguous block, so that the application can still access it directly without any need of copying within LMDB?
Yes.
Or it is better to split larger data into multiple key/value pairs?
Depends on your environment. It is a requirement that a single value actually fits in the system's RAM.
- I discovered the return code "MDB_TXN_FULL". By what means are transactions limited? I might consider to use few, but quite large transactions dealing with hundreds of thousands key/value pairs to be created, updated or deleted in some circumstances.
The limit has mostly been removed, although large transactions will probably have a performance penalty.
- I saw the discussion about the predefined database size, and that it is recommended to use a never-to-be-reached size and to rely on the sparse file capabilities of the underlying OS. If I would not want to go this way: Is it correct that if I encounter an "MDB_MAP_FULL" return code, it would be sufficient to abort the current transaction, resize the database using mdp_env_set_mapsize and repeat the transaction?
Yes, as long as you follow the other restrictions on changing the size.
Thanks in advance and regards,
Klaus
Hi Howard,
thanks a lot for your answers.
No, the data size is limited to 4294967295 bytes.
Oh, I missed two "f"s in the constant ;-)
- I discovered the return code "MDB_TXN_FULL". By what means are transactions limited? I might consider to use few, but quite large transactions dealing with hundreds of thousands key/value pairs to be created, updated or deleted in some circumstances.
The limit has mostly been removed, although large transactions will probably have a performance penalty.
Sure, I don't expect that to happen in a second anyway. But if the cost of a key/value op is still less linear, it would be fine to me, as the situation would occur rather seldom.
- I saw the discussion about the predefined database size, and that it is recommended to use a never-to-be-reached size and to rely on the sparse file capabilities of the underlying OS. If I would not want to go this way: Is it correct that if I encounter an "MDB_MAP_FULL" return code, it would be sufficient to abort the current transaction, resize the database using mdp_env_set_mapsize and repeat the transaction?
Yes, as long as you follow the other restrictions on changing the size.
You mean that a separate (reader) process would have to react on the changed file size also (MDB_MAP_RESIZED result)? Or are there any further things to observe that I missed?
Regards,
Klaus
Klaus Malorny wrote:
- I saw the discussion about the predefined database size, and that it is recommended to use a never-to-be-reached size and to rely on the sparse file capabilities of the underlying OS. If I would not want to go this way: Is it correct that if I encounter an "MDB_MAP_FULL" return code, it would be sufficient to abort the current transaction, resize the database using mdp_env_set_mapsize and repeat the transaction?
Yes, as long as you follow the other restrictions on changing the size.
You mean that a separate (reader) process would have to react on the changed file size also (MDB_MAP_RESIZED result)? Or are there any further things to observe that I missed?
mdb_env_set_mapsize:
* It may be called at later times if no transactions are active in * this process. Note that the library does not check for this condition, * the caller must ensure it explicitly.
openldap-technical@openldap.org
-
Howard Chu -
Klaus Malorny