10:20 a.m.
Hello,
following up to my previous question regarding file size, I would like to ask
about the recommended approach:
1. set a low file size initially, watch for MDB_MAP_FULL return codes and
increase the file/map size stepwise as needed
2. choose a large file size and reduce the size by doing a mdb_copy when the
application shuts down
or maybe a combination of the two? How does SQLightning handle this (2. doesn't
sound practical)?
thanks,
Christian
1:51 p.m.
Christian Sell wrote:
Hello,
following up to my previous question regarding file size, I would like to ask
about the recommended approach:
You're being stupid.
The documentation already clearly states:
http://symas.com/mdb/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5
The size of the memory map is also the maximum size of the database.
The value should be chosen as large as possible, to accommodate future
growth of the database.
1. set a low file size initially, watch for MDB_MAP_FULL return codes
and
increase the file/map size stepwise as needed
No. That would be stupid.
2. choose a large file size and reduce the size by doing a mdb_copy
when the
application shuts down
No. That would be stupid.
or maybe a combination of the two? How does SQLightning handle this
(2.
doesn't sound practical)?
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
2:04 p.m.
--On Wednesday, November 11, 2015 9:51 PM +0000 Howard Chu <hyc(a)symas.com>
wrote:
Christian Sell wrote:
> Hello,
> following up to my previous question regarding file size, I would like
> to ask about the recommended approach:
Or to put it a little more politely:
The max size really does not matter, since it is not actually in use or
allocated until it is used. I set my default maxsize to 80GB, and then
scale it back based on the amount of freespace on the disk at the time of
creation. Generally, you do not want the maxsize to be > disk size, since
you really wouldn't want the DB to grow past the end of the disk. ;)
--Quanah
--
Quanah Gibson-Mount
Platform Architect
Zimbra, Inc.
--------------------
Zimbra :: the leader in open source messaging and collaboration
2:51 p.m.
You're being stupid.
let's say I am stubborn. As I mentioned, I have the requirement that the user be
able to copy the database file away on their USB stick (or send it via email,
for that matter). With SQLite, that is no problem. But with LMDB under Windows
the user is looking at a huge file, which will be unacceptable in all
likelihood. I also must allow the user to copy using file system tools, and not
some custom utility.
So I take it that LMDB is simply not designed or suitable for such a scenario.
5:48 a.m.
Christian Sell wrote:
> You're being stupid.
let's say I am stubborn. As I mentioned, I have the requirement that the user be
able to copy the database file away on their USB stick (or send it via email,
for that matter). With SQLite, that is no problem. But with LMDB under Windows
the user is looking at a huge file, which will be unacceptable in all
likelihood. I also must allow the user to copy using file system tools, and not
some custom utility.
So I take it that LMDB is simply not designed or suitable for such a scenario.
Windows and its lack of reasonable sparse file support makes it not a suitable
OS. (Windows supports sparse files, but not by default, and there is a 30%
performance degradation when using that support in LMDB.)
If you're stuck with Windows and these other requirements then I recommend you
use something other than LMDB, it is not appropriate for your use case.
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
7:13 a.m.
Windows and its lack of reasonable sparse file support makes it not a
suitable
OS. (Windows supports sparse files, but not by default, and there is a 30%
performance degradation when using that support in LMDB.)
I wouldn't mind a 30% degradation, really. Do you see a possibility of
supporting sparse files under Windows? Is there more to it than adding a
FSCTL_SET_SPARSE flag somewhere?
thanks,
Christian
2:04 a.m.
On 11.11.2015 23:51, Christian Sell wrote:
> You're being stupid.
let's say I am stubborn. As I mentioned, I have the requirement that the user be
able to copy the database file away on their USB stick (or send it via email,
for that matter). With SQLite, that is no problem. But with LMDB under Windows
the user is looking at a huge file, which will be unacceptable in all
likelihood. I also must allow the user to copy using file system tools, and not
some custom utility.
So I take it that LMDB is simply not designed or suitable for such a scenario.
Maybe you should consider to write your own exporter/importer. It shouldn't be
too hard to walk through all key/value pairs and write out the data (prepended
by their respective lengths), plus some metadata at the beginning to recognize
the format on import and to know the number of records (if there is no special
end-of-file marker).
Just my two cents.
Klaus
2:11 a.m.
On 16.11.2015 11:04, Klaus Malorny wrote:
On 11.11.2015 23:51, Christian Sell wrote:
>> You're being stupid.
>
> let's say I am stubborn. As I mentioned, I have the requirement that the user be
> able to copy the database file away on their USB stick (or send it via email,
> for that matter). With SQLite, that is no problem. But with LMDB under Windows
> the user is looking at a huge file, which will be unacceptable in all
> likelihood. I also must allow the user to copy using file system tools, and not
> some custom utility.
>
> So I take it that LMDB is simply not designed or suitable for such a scenario.
>
Maybe you should consider to write your own exporter/importer. It shouldn't be
too hard to walk through all key/value pairs and write out the data (prepended
by their respective lengths), plus some metadata at the beginning to recognize
the format on import and to know the number of records (if there is no special
end-of-file marker).
Just my two cents.
Klaus
Hmm, I missed the requirement "I also must allow the user to copy using file
system tools, and not some custom utility." However, I think this is not a good
idea for any database since there is no guarantee that you get a valid state of
the database without proper synchronization with the running process(es). Even
if you are using the database with an application that does not run permanently
(like a GUI app), I don't know whether users are smart enough to stop the
application before performing the copy.
Regards,
Klaus
3:01 a.m.
well, beyond the requirement about copyability of the file, there is also the
requirement that the app coexist well with other processes that are using the
disk and not grab all disk space while running. And then, there is the
requirement that the database should still be able to grow up to the limit of
the available resources if need be.
This is a desktop app, and the database can be looked upon as a "project file".
If user copies it in a inconsistent state, it's their problem.
I have decided to implement a database growth algorithm. There is an initial map
size and a threshhold. Whenever a write transaction is started, I check that
enough space is available and enlarge the map if needed. The app may also also
request more space for a transaction if known beforehand. If the transaction
hits the MAP_FULL anyway, bad luck.
regards + thanks,
Chris
Klaus Malorny <Klaus.Malorny(a)knipp.de> hat am 16. November 2015
um 11:04
geschrieben:
On 11.11.2015 23:51, Christian Sell wrote:
>> You're being stupid.
>
> let's say I am stubborn. As I mentioned, I have the requirement that the
> user be
> able to copy the database file away on their USB stick (or send it via
> email,
> for that matter). With SQLite, that is no problem. But with LMDB under
> Windows
> the user is looking at a huge file, which will be unacceptable in all
> likelihood. I also must allow the user to copy using file system tools, and
> not
> some custom utility.
>
> So I take it that LMDB is simply not designed or suitable for such a
> scenario.
>
Maybe you should consider to write your own exporter/importer. It shouldn't be
too hard to walk through all key/value pairs and write out the data (prepended
by their respective lengths), plus some metadata at the beginning to recognize
the format on import and to know the number of records (if there is no special
end-of-file marker).
Just my two cents.
Klaus
4:27 a.m.
Hello,
at the risk of being called stupid again: I am still labouring with said issue.
Before giving up my invest in LMBD, I'd like to try another route to meet my
requirements. Since the main issue is the file size under Windows, my idea is to
resize the map to the actual data size upon application shutdown. My problem
then would be to determine the actual datasize. Can somebody tell me how that
can be achieved?
thanks,
Christian
Howard Chu <hyc(a)symas.com> hat am 11. November 2015 um 22:51
geschrieben:
Christian Sell wrote:
> Hello,
> following up to my previous question regarding file size, I would like to
> ask
> about the recommended approach:
You're being stupid.
The documentation already clearly states:
http://symas.com/mdb/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5
The size of the memory map is also the maximum size of the database.
The value should be chosen as large as possible, to accommodate future
growth of the database.
> 1. set a low file size initially, watch for MDB_MAP_FULL return codes and
> increase the file/map size stepwise as needed
No. That would be stupid.
> 2. choose a large file size and reduce the size by doing a mdb_copy when the
> application shuts down
No. That would be stupid.
> or maybe a combination of the two? How does SQLightning handle this (2.
> doesn't sound practical)?
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
Christian Sell
GS Vitec GmbH
Im Ziegelhaus 6-8
D-63571 Gelnhausen
mail: christian(a)gsvitec.com
mobil: +49 (0) 173 5384289
Tel: +49 (0) 6051 601.26-90
Fax: +49 (0) 6051 601.26-91
5:31 a.m.
Christian Sell wrote:
Hello,
at the risk of being called stupid again: I am still labouring with said
issue. Before giving up my invest in LMBD, I'd like to try another route to
meet my requirements. Since the main issue is the file size under Windows, my
idea is to resize the map to the actual data size upon application shutdown.
My problem then would be to determine the actual datasize. Can somebody tell
me how that can be achieved?
What is your actual requirement?
Why are you setting a large mapsize in the first place, if you only want a
smaller size at application shutdown? How much space are you *really* prepared
to allow your application to use? Why aren't you configuring with that
specific mapsize?
What are you going to do if/when your application actually needs to write
beyond a particular size limit?
This is ultimately the question you must answer - how much space will you
actually allow your app to use? Once you have that figure, configure it once
and forget about it. It is *STUPID* to ask the OS to allocate space for you
and then ask it to deallocate space repeatedly. You wind up fragmenting the
disk and killing performance. Either you have enough disk space for the app's
needs, or you don't. If you don't have enough space, you need to reconfigure
your system environment. If you have enough space, allocate it and be done
with it.
thanks,
Christian
> Howard Chu <hyc(a)symas.com> hat am 11. November 2015 um 22:51 geschrieben:
>
>
> Christian Sell wrote:
> > Hello,
> > following up to my previous question regarding file size, I would like to ask
> > about the recommended approach:
>
> You're being stupid.
>
> The documentation already clearly states:
> http://symas.com/mdb/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5
>
> The size of the memory map is also the maximum size of the database.
> The value should be chosen as large as possible, to accommodate future
> growth of the database.
>
> > 1. set a low file size initially, watch for MDB_MAP_FULL return codes and
> > increase the file/map size stepwise as needed
>
> No. That would be stupid.
>
> > 2. choose a large file size and reduce the size by doing a mdb_copy when the
> > application shuts down
>
> No. That would be stupid.
>
> > or maybe a combination of the two? How does SQLightning handle this (2.
> > doesn't sound practical)?
--
-- Howard Chu
CTO, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc/
Chief Architect, OpenLDAP http://www.openldap.org/project/
6:33 a.m.
New subject: AW: LMDB file size, again
Hello Howard,
What is your actual requirement?
As stated before: I need the ability for the database to grow undeterminably (up to the
available disk space - but I don’t know the available disk space, because there are other
applications using the disk), while maintaining the smallest possible file size (because
the user should be able to carry the file away as easily as possible)
Why are you setting a large mapsize in the first place,
Because I cannot determine how large the file might get (as described above)
if you only want a smaller size at application shutdown?
I
want the SMALLEST POSSIBLE SIZE
How much space are you *really* prepared to allow your application to
use?
Again: I don’t know beforehand
What are you going to do if/when your application actually needs to
write beyond a particular size limit?
If the available disk space has been exhausted
(which may happen also due to other application's activities), I will let the user
handle it. Nothing else I can do.
This is a desktop application that must play well with other applications. There is
nothing unreasonable with my (or my user's) requirements
Thanks,
Christian
2753
days inactive
2758
days old
openldap-technical@openldap.org
11 comments
4 participants
participants (4)
-
Christian Sell -
Howard Chu -
Klaus Malorny -
Quanah Gibson-Mount