Re: Building for windows. Again.

Kristoffer Sjögren wrote: > > Just to let you know. I merged the fix manually onto LMDB_0.9.16 and it > works. Great. But as we announced just last week http://www.openldap.org/lists/openldap-technical/201511/msg00092.html LMDB 0.9.17 is in preparation for release now. Probably will be out in a few days. > > > https://github.com/deephacks/lmdbjni/commit/54a6ed3f1e0896a29e4bd10594bf9... > > > On Fri, Nov 20, 2015 at 10:34 AM, Kristoffer Sjögren <stoffe(a)gmail.com&gt; > wrote: >> >> The last build I know worked for lmdbjni on windows used LMDB_0.9.14. >> >> I was going to try a patch release on LMDB_0.9.16 but the fix doesn't >> merge cleanly. >> >> On Fri, Nov 20, 2015 at 10:26 AM, Howard Chu <hyc(a)symas.com&gt; wrote: >>> >>> Kristoffer Sjögren wrote: >>>> >>>> >>>> Yep, the commit fixes the problem. Thanks Howard! >>> >>> >>> >>> The only question now - what is "modified" in the Java UTF-8 encoding? >>> >>>> >>>> On Fri, Nov 20, 2015 at 2:14 AM, Howard Chu <hyc(a)symas.com&gt; wrote: >>>>> >>>>> >>>>> Kristoffer Sjögren wrote: >>>>>> >>>>>> >>>>>> >>>>>> Changing CharacterSet in the build file doesn't affect the garbled >>>>>> path. >>>>>> >>>>>> I tried to use explicit GetStringChars from the JNI code but with >>>>>> same >>>>>> result. >>>>>> >>>>>> Any pointers? I'm running short of ideas.. >>>>> >>>>> >>>>> >>>>> >>>>> Looks like we need to use UTF-8, as in ITS#7992. I've just merged that >>>>> patch >>>>> into mdb.master, see if that works better. You should use >>>>> GetStringUTFChars. >>>>> >>>>>> >>>>>> On Thu, Nov 19, 2015 at 11:55 PM, Kristoffer Sjögren >>>>>> <stoffe(a)gmail.com&gt; >>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> Oh, but the build file [2] have a CharacterSet element set to >>>>>>> Unicode. >>>>>>> Let me see if I can change this. >>>>>>> >>>>>>> [2] >>>>>>> >>>>>>> >>>>>>> https://github.com/deephacks/lmdbjni/blob/master/lmdbjni/src/main/native-... >>>>>>> >>>>>>> On Thu, Nov 19, 2015 at 11:45 PM, Kristoffer Sjögren >>>>>>> <stoffe(a)gmail.com&gt; >>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> I can see the problem now from a local test. >>>>>>>> >>>>>>>> Hmm i'm a bit confused. The JNI code uses GetStringUTFChars calls >>>>>>>> [1] >>>>>>>> for all char * arguments going from Java to C, which is an array of >>>>>>>> bytes representing the string in modified UTF-8 encoding. Is this >>>>>>>> OK? >>>>>>>> The other option is to use GetStringChars which is unicode... >>>>>>>> >>>>>>>> [1] >>>>>>>> >>>>>>>> >>>>>>>> http://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/functions.... >>>>>>>> >>>>>>>> On Thu, Nov 19, 2015 at 10:32 PM, Howard Chu <hyc(a)symas.com&gt; wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Kristoffer Sjögren wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> The py-lmdb project have a comment [1] about reusing Python.h >>>>>>>>>> headers >>>>>>>>>> which seems to work for them. >>>>>>>>>> >>>>>>>>>> # Microsoft Visual Studio 9 ships with neither inttypes.h, >>>>>>>>>> stdint.h, >>>>>>>>>> or a >>>>>>>>>> sane >>>>>>>>>> # definition for ssize_t, so here we add lib/win32 to the search >>>>>>>>>> path, >>>>>>>>>> which >>>>>>>>>> # contains emulation header files provided by a third party. We >>>>>>>>>> force-include >>>>>>>>>> # Python.h everywhere since it has a portable definition of >>>>>>>>>> ssize_t, >>>>>>>>>> which >>>>>>>>>> # inttypes.h and stdint.h lack, and to avoid having to modify the >>>>>>>>>> LMDB >>>>>>>>>> source >>>>>>>>>> # code. Advapi32 is needed for LMDB's use of Windows security >>>>>>>>>> APIs. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Note in ITS#8067 we added an ssize_t typedef for MSVC. Also in >>>>>>>>> #8069 >>>>>>>>> we >>>>>>>>> changed to explicitly use the ANSI APIs, so if you're using >>>>>>>>> Unicode >>>>>>>>> file >>>>>>>>> paths that will certainly be garbled now. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> [1] https://github.com/dw/py-lmdb/blob/master/setup.py >>>>>>>>>> >>>>>>>>>> On Thu, Nov 19, 2015 at 10:13 PM, Kristoffer Sjögren >>>>>>>>>> <stoffe(a)gmail.com&gt; >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> The actual build command: >>>>>>>>>>> >>>>>>>>>>> $ cmd.exe /X /C "vcbuild /platform:x64 vs2008.vcproj release" >>>>>>>>>>> >>>>>>>>>>> On Thu, Nov 19, 2015 at 10:11 PM, Kristoffer Sjögren >>>>>>>>>>> <stoffe(a)gmail.com&gt; >>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> That's the thing, the build doesn't complain about anything >>>>>>>>>>>> missing. >>>>>>>>>>>> But the binary seems broken because tests fail randomly. >>>>>>>>>>>> >>>>>>>>>>>> A user tried the generated binary on his machine and it worked >>>>>>>>>>>> but >>>>>>>>>>>> also >>>>>>>>>>>> said... >>>>>>>>>>>> >>>>>>>>>>>> "Hm, almost, the database file is no longer put to the >>>>>>>>>>>> requested >>>>>>>>>>>> directory, and its name is totally garbled. It finds it >>>>>>>>>>>> consistently >>>>>>>>>>>> though, and the contents are correct." >>>>>>>>>>>> >>>>>>>>>>>> https://github.com/deephacks/lmdbjni/issues/48 >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Nov 19, 2015 at 9:57 PM, Howard Chu <hyc(a)symas.com&gt; >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Kristoffer Sjögren wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm trying to build LMDB with Java/JNI bindings with Visual >>>>>>>>>>>>>> C++ >>>>>>>>>>>>>> Project Builder 9.00.30729 (vcbuild). >>>>>>>>>>>>>> >>>>>>>>>>>>>> Unfortunately, vcbuild don't ship with inttypes.h, stdint.h, >>>>>>>>>>>>>> or >>>>>>>>>>>>>> a >>>>>>>>>>>>>> sane >>>>>>>>>>>>>> ssize_t. So I searched around and found a few candidates [1] >>>>>>>>>>>>>> of >>>>>>>>>>>>>> inttypes.h and stdint.h that seems to be working for py-lmdb. >>>>>>>>>>>>>> >>>>>>>>>>>>>> However, unistd.h seems broken for windows - and the python >>>>>>>>>>>>>> guys >>>>>>>>>>>>>> use >>>>>>>>>>>>>> python.h instead. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Any ideas where I might find a good unistd.h? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> I see you have one already - what is wrong with it, what >>>>>>>>>>>>> definitions >>>>>>>>>>>>> are >>>>>>>>>>>>> missing? >>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>> -Kristoffer >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://github.com/deephacks/lmdbjni/tree/master/lmdbjni-win64/headers >>>>>>>>>>>>>> >>> -- -- Howard Chu CTO, Symas Corp. http://www.symas.com Director, Highland Sun http://highlandsun.com/hyc/ Chief Architect, OpenLDAP http://www.openldap.org/project/