Hello,
following up to my previous question regarding file size, I would like to ask about the recommended approach:
1. set a low file size initially, watch for MDB_MAP_FULL return codes and increase the file/map size stepwise as needed 2. choose a large file size and reduce the size by doing a mdb_copy when the application shuts down
or maybe a combination of the two? How does SQLightning handle this (2. doesn't sound practical)?
thanks, Christian
Christian Sell wrote:
Hello, following up to my previous question regarding file size, I would like to ask about the recommended approach:
You're being stupid.
The documentation already clearly states: http://symas.com/mdb/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5
The size of the memory map is also the maximum size of the database. The value should be chosen as large as possible, to accommodate future growth of the database.
- set a low file size initially, watch for MDB_MAP_FULL return codes and
increase the file/map size stepwise as needed
No. That would be stupid.
- choose a large file size and reduce the size by doing a mdb_copy when the
application shuts down
No. That would be stupid.
or maybe a combination of the two? How does SQLightning handle this (2. doesn't sound practical)?
--On Wednesday, November 11, 2015 9:51 PM +0000 Howard Chu hyc@symas.com wrote:
Christian Sell wrote:
Hello, following up to my previous question regarding file size, I would like to ask about the recommended approach:
Or to put it a little more politely:
The max size really does not matter, since it is not actually in use or allocated until it is used. I set my default maxsize to 80GB, and then scale it back based on the amount of freespace on the disk at the time of creation. Generally, you do not want the maxsize to be > disk size, since you really wouldn't want the DB to grow past the end of the disk. ;)
--Quanah
--
Quanah Gibson-Mount Platform Architect Zimbra, Inc. -------------------- Zimbra :: the leader in open source messaging and collaboration
You're being stupid.
let's say I am stubborn. As I mentioned, I have the requirement that the user be able to copy the database file away on their USB stick (or send it via email, for that matter). With SQLite, that is no problem. But with LMDB under Windows the user is looking at a huge file, which will be unacceptable in all likelihood. I also must allow the user to copy using file system tools, and not some custom utility.
So I take it that LMDB is simply not designed or suitable for such a scenario.
Christian Sell wrote:
You're being stupid.
let's say I am stubborn. As I mentioned, I have the requirement that the user be able to copy the database file away on their USB stick (or send it via email, for that matter). With SQLite, that is no problem. But with LMDB under Windows the user is looking at a huge file, which will be unacceptable in all likelihood. I also must allow the user to copy using file system tools, and not some custom utility.
So I take it that LMDB is simply not designed or suitable for such a scenario.
Windows and its lack of reasonable sparse file support makes it not a suitable OS. (Windows supports sparse files, but not by default, and there is a 30% performance degradation when using that support in LMDB.)
If you're stuck with Windows and these other requirements then I recommend you use something other than LMDB, it is not appropriate for your use case.
Windows and its lack of reasonable sparse file support makes it not a suitable OS. (Windows supports sparse files, but not by default, and there is a 30% performance degradation when using that support in LMDB.)
I wouldn't mind a 30% degradation, really. Do you see a possibility of supporting sparse files under Windows? Is there more to it than adding a FSCTL_SET_SPARSE flag somewhere?
thanks, Christian
On 11.11.2015 23:51, Christian Sell wrote:
You're being stupid.
let's say I am stubborn. As I mentioned, I have the requirement that the user be able to copy the database file away on their USB stick (or send it via email, for that matter). With SQLite, that is no problem. But with LMDB under Windows the user is looking at a huge file, which will be unacceptable in all likelihood. I also must allow the user to copy using file system tools, and not some custom utility.
So I take it that LMDB is simply not designed or suitable for such a scenario.
Maybe you should consider to write your own exporter/importer. It shouldn't be too hard to walk through all key/value pairs and write out the data (prepended by their respective lengths), plus some metadata at the beginning to recognize the format on import and to know the number of records (if there is no special end-of-file marker).
Just my two cents.
Klaus
On 16.11.2015 11:04, Klaus Malorny wrote:
On 11.11.2015 23:51, Christian Sell wrote:
You're being stupid.
let's say I am stubborn. As I mentioned, I have the requirement that the user be able to copy the database file away on their USB stick (or send it via email, for that matter). With SQLite, that is no problem. But with LMDB under Windows the user is looking at a huge file, which will be unacceptable in all likelihood. I also must allow the user to copy using file system tools, and not some custom utility.
So I take it that LMDB is simply not designed or suitable for such a scenario.
Maybe you should consider to write your own exporter/importer. It shouldn't be too hard to walk through all key/value pairs and write out the data (prepended by their respective lengths), plus some metadata at the beginning to recognize the format on import and to know the number of records (if there is no special end-of-file marker).
Just my two cents.
Klaus
Hmm, I missed the requirement "I also must allow the user to copy using file system tools, and not some custom utility." However, I think this is not a good idea for any database since there is no guarantee that you get a valid state of the database without proper synchronization with the running process(es). Even if you are using the database with an application that does not run permanently (like a GUI app), I don't know whether users are smart enough to stop the application before performing the copy.
Regards,
Klaus
well, beyond the requirement about copyability of the file, there is also the requirement that the app coexist well with other processes that are using the disk and not grab all disk space while running. And then, there is the requirement that the database should still be able to grow up to the limit of the available resources if need be.
This is a desktop app, and the database can be looked upon as a "project file". If user copies it in a inconsistent state, it's their problem.
I have decided to implement a database growth algorithm. There is an initial map size and a threshhold. Whenever a write transaction is started, I check that enough space is available and enlarge the map if needed. The app may also also request more space for a transaction if known beforehand. If the transaction hits the MAP_FULL anyway, bad luck.
regards + thanks, Chris
Klaus Malorny Klaus.Malorny@knipp.de hat am 16. November 2015 um 11:04 geschrieben:
On 11.11.2015 23:51, Christian Sell wrote:
You're being stupid.
let's say I am stubborn. As I mentioned, I have the requirement that the user be able to copy the database file away on their USB stick (or send it via email, for that matter). With SQLite, that is no problem. But with LMDB under Windows the user is looking at a huge file, which will be unacceptable in all likelihood. I also must allow the user to copy using file system tools, and not some custom utility.
So I take it that LMDB is simply not designed or suitable for such a scenario.
Maybe you should consider to write your own exporter/importer. It shouldn't be too hard to walk through all key/value pairs and write out the data (prepended by their respective lengths), plus some metadata at the beginning to recognize the format on import and to know the number of records (if there is no special end-of-file marker).
Just my two cents.
Klaus
Hello,
at the risk of being called stupid again: I am still labouring with said issue. Before giving up my invest in LMBD, I'd like to try another route to meet my requirements. Since the main issue is the file size under Windows, my idea is to resize the map to the actual data size upon application shutdown. My problem then would be to determine the actual datasize. Can somebody tell me how that can be achieved?
thanks, Christian
Howard Chu hyc@symas.com hat am 11. November 2015 um 22:51 geschrieben:
Christian Sell wrote:
Hello, following up to my previous question regarding file size, I would like to ask about the recommended approach:
You're being stupid.
The documentation already clearly states: http://symas.com/mdb/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5
The size of the memory map is also the maximum size of the database. The value should be chosen as large as possible, to accommodate future growth of the database.
- set a low file size initially, watch for MDB_MAP_FULL return codes and
increase the file/map size stepwise as needed
No. That would be stupid.
- choose a large file size and reduce the size by doing a mdb_copy when the
application shuts down
No. That would be stupid.
or maybe a combination of the two? How does SQLightning handle this (2. doesn't sound practical)?
-- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/
Christian Sell
GS Vitec GmbH Im Ziegelhaus 6-8 D-63571 Gelnhausen
mail: christian@gsvitec.com mobil: +49 (0) 173 5384289
Tel: +49 (0) 6051 601.26-90 Fax: +49 (0) 6051 601.26-91
Christian Sell wrote:
Hello, at the risk of being called stupid again: I am still labouring with said issue. Before giving up my invest in LMBD, I'd like to try another route to meet my requirements. Since the main issue is the file size under Windows, my idea is to resize the map to the actual data size upon application shutdown. My problem then would be to determine the actual datasize. Can somebody tell me how that can be achieved?
What is your actual requirement?
Why are you setting a large mapsize in the first place, if you only want a smaller size at application shutdown? How much space are you *really* prepared to allow your application to use? Why aren't you configuring with that specific mapsize?
What are you going to do if/when your application actually needs to write beyond a particular size limit?
This is ultimately the question you must answer - how much space will you actually allow your app to use? Once you have that figure, configure it once and forget about it. It is *STUPID* to ask the OS to allocate space for you and then ask it to deallocate space repeatedly. You wind up fragmenting the disk and killing performance. Either you have enough disk space for the app's needs, or you don't. If you don't have enough space, you need to reconfigure your system environment. If you have enough space, allocate it and be done with it.
thanks, Christian
Howard Chu hyc@symas.com hat am 11. November 2015 um 22:51 geschrieben:
Christian Sell wrote:
Hello, following up to my previous question regarding file size, I would like to ask about the recommended approach:
You're being stupid.
The documentation already clearly states: http://symas.com/mdb/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5
The size of the memory map is also the maximum size of the database. The value should be chosen as large as possible, to accommodate future growth of the database.
- set a low file size initially, watch for MDB_MAP_FULL return codes and
increase the file/map size stepwise as needed
No. That would be stupid.
- choose a large file size and reduce the size by doing a mdb_copy when the
application shuts down
No. That would be stupid.
or maybe a combination of the two? How does SQLightning handle this (2. doesn't sound practical)?
Hello Howard,
What is your actual requirement?
As stated before: I need the ability for the database to grow undeterminably (up to the available disk space - but I don’t know the available disk space, because there are other applications using the disk), while maintaining the smallest possible file size (because the user should be able to carry the file away as easily as possible)
Why are you setting a large mapsize in the first place,
Because I cannot determine how large the file might get (as described above)
if you only want a smaller size at application shutdown?
I want the SMALLEST POSSIBLE SIZE
How much space are you *really* prepared to allow your application to use?
Again: I don’t know beforehand
What are you going to do if/when your application actually needs to write beyond a particular size limit?
If the available disk space has been exhausted (which may happen also due to other application's activities), I will let the user handle it. Nothing else I can do.
This is a desktop application that must play well with other applications. There is nothing unreasonable with my (or my user's) requirements
Thanks, Christian
openldap-technical@openldap.org
-
Christian Sell -
Howard Chu -
Klaus Malorny -
Quanah Gibson-Mount