> I just had a look at how BDB handled this. As you can see they used a > TO_TSTRING macro to convert incoming pathnames from UTF8 to UTF16. > > https://gitorious.org/berkeleydb/berkeleydb/source/347d239a1e44ed4f773ae9... > > https://gitorious.org/berkeleydb/berkeleydb/source/347d239a1e44ed4f773ae9... > > (And a FROM_TSTRING for the reverse, as well.) (Mea culpa, I accidentally hit "reply" instead of "reply all". Sorry. Now reposting to the mailing list.) Since we only need to do this on Windows, we could use MultiByteToWideChar with CP_UTF8. (That's what TO_TSTRING does, too.) I do not think we would ever need to do any such conversion on UNIX.

https://msdn.microsoft.com/en-us/library/windows/desktop/dd319072%28v=vs.... I'm not sure if we can just copy-paste BDB's code. Probably not, that would lead to licensing issues, wouldn't it?

I suggest we wait to deal with DB names until we also have a way to deal with filenames. And this time test that it works is practice:-) Hopefully users and programmers will only need one method of handling non-ASCII LMDB names on Windows, not two. I'd be nice if 'mdb_stat filename -s dbname' would Just Work, as would reading DB names and filenames from an config file. Yet OS-aware and OS-specific config files can look rather different. Maybe LMDB must handle DB names more flexibly than filenames, or maybe we'll end up recommending that "portable" DB names must be UTF-8. And add a "flag convert UTF8<->WCHAR if this is Windows".

Hallvard Breien Furuseth wrote: > I suggest we wait to deal with DB names until we also have a way to > deal with filenames. And this time test that it works is practice:-) > Hopefully users and programmers will only need one method of handling > non-ASCII LMDB names on Windows, not two. > > I'd be nice if 'mdb_stat filename -s dbname' would Just Work, as would > reading DB names and filenames from an config file. Yet OS-aware and > OS-specific config files can look rather different. Maybe LMDB must > handle DB names more flexibly than filenames, or maybe we'll end up > recommending that "portable" DB names must be UTF-8. And add a "flag > convert UTF8<->WCHAR if this is Windows". > DB names are purely internal to LMDB, so they bear no relation to OS filenames and none of this discussion matters to them.

A path is always a Unicode string, while a DB name can be an arbitrary binary blob. So I don't think that we can treat them the same way.

Okay. What do you suggest?

> I suggest we wait to deal with DB names until we also have a way to > deal with filenames. And this time test that it works is practice. > > And then I also suggest to try to make this mess simple to deal > with for programmers and or users. I guess I should have separated > that from the rest more clearly. I can write a patch which does the UTF-8 to UTF-16 conversion on Windows for file paths, but I would hate to restrict db names to UTF-8 text only (or for that matter, any text only). However, not supporting non-UTF-8 db names in mdb_dump and mdb_load sounds like a reasonable compromise to me.

> A path is always a Unicode string, while a DB name can be an arbitrary > binary blob. On many POSIX platforms, a path is a blob which does not contain '\000'. These systems do not enforce Unicode encoding at all.

>> > A path is always a Unicode string, while a DB name can be an arbitrary >> > binary blob. >> >> On many POSIX platforms, a path is a blob which does not contain >> '\000'. These systems do not enforce Unicode encoding at all. > > My mistake. I was unaware. > On those platforms, how do you type a path name into a terminal? There are some files which are not directly nameable. Many programs support special sequences such as “Ctrl+V 3 7 7” to enter arbitrary bytes, but that's not universal. Depending on the actual implementation of the terminal, cut-and-paste of funny file names can work, too. Older programs have trouble accessing such files even if the user chooses them in a file selection dialog, but current version are supposed to have been fixed (including OpenJDK, which took a ridiculously long time). Beyond that, it's not much different from dealing with file names in an unfamiliar script.