Hallvard Breien Furuseth wrote:
Howard Chu writes:
Fixing this will either require adding a bunch of ugly code, or changing the on-disk format again. Opinions? (...) Adjusting this would require only a few minor tweaks to the code, but would require a full dump/reload of existing databases.
Then I hope for the format change. But can this be macroized without said ugliness, so existing users can compile with e.g. -DMDB_DATA_VERSION=1 until they're prepared to dump/reload?
Maybe, will need to look into where the changes lie.
Still, 64K pages sounds like it can get quite wasteful. I can clean up my patch to support datafile pagesize != OS pagesize. That would be a user option, not the default: Smaller datafile pagesize means more I/O for writes. (The OS operates on pages, so to write part of an OS page it must first read the full page.)
We may still want that, yeah. Kind of wondering if 64K is a real maximum on pageable memory blocks. CPUs support larger pages but in current OSs they're not actually pageable yet; reading thru linux-kernel posts seems to imply that making huge pages pageable is actually a future goal.
It seems impractical given HDD storage, but with the advent of nonvolatile RAMs it may actually make sense down the road. In which case we'll just have to make indx_t be an int anyway.
A problem here is that liblmdb lacks some checks for data sizes. We can add those, but older liblmdb can break if it commits to a datafile written by newer liblmdb and non-default sizes. IIRC it gets a bit hairy to support this for ldmb data format 2 but support V1 sizes safely, so I'm either talking V2 only or just telling users to not do this when also using an older liblmdb.
For simplicity we should probably just make a hard break with the V2 format. liblmdb built for V1 format won't be able to touch a V2 file.