mark diener wrote:
I came across a python write up about lmdb:
http://lmdb.readthedocs.io/en/release/#environment-class
map_size:
Maximum size database may grow to; used to size the memory mapping. If database grows larger than map_size, an exception will be raised and the user must close and reopen Environment. On 64-bit there is no penalty for making this huge (say 1TB). Must be <2GB on 32-bit.
Well, it will be a penalty on MacOSX, it will create a 1TB file on disk upon initialization!
If you have a 1TB or larger filesystem, then there's obviously no problem. If the filesystem is smaller than that, obviously you can't choose a size bigger than the filesystem has space for.
What am I missing here?
Most POSIX filesystems don't preallocate space unless you explicitly request it. I.e., setting the size of a file (using ftruncate()) merely stores that number in an inode. You have to do an explicit fallocate() to make the filesystem allocate all of the space.
Since you're on MacOSX, things are different. The default filesystem (HFS or HFS+) doesn't support sparse files, so when you set the size of a file to a particular number, the filesystem allocates the space at that moment.
If you want to get the behavior you're looking for, you need to switch to UFS, which supports sparse files like most other POSIX filesystems.
Meanwhile - even if you're using HFS, you should still just set the map size to the largest value you ever expect to use. Preallocating the space will give better runtime performance, and will allow the FS to optimize the layout of the allocated blocks.
Marco
On Wed, Apr 11, 2018 at 10:27 AM, mark diener rpzrpzrpz@gmail.com wrote:
Anybody have any idea on how to have the database file grow until you get to the mapsize instead of pre-allocating the entire mapsize (4GB) upon initialization?
mdb_env_set_mapsize() ;
I don't mind having an upper limit on the map size, but I think it would be helpful to have it grow to the upper limit, not preallocate the space on storage.
Your thought here is invalid.
There are various areas in the documentation where it talks about having the database not grow without bounds.
Comments?
Marco