Building on Windows #119
Comments
|
Can you compile other things using Python.h? Did you try taking out the spaces after |
|
Hi, I've been trying this for a few days. here is how far I've got:
|
|
What are the errors? |
|
the error is useless. you will know it when you read lock.go: func (lock *Lock) dec() bool, it's basically the BaseObject can't be converted to Long C:\Users\hb19623\Gocode\src\lime\3rdparty\libs\gopy>go test
# lime/3rdparty/libs/gopy/lib
lib\utils.c:254:5: warning: initialization makes integer from pointer without a
cast [enabled by default]
"GoObjMember", /*tp_name*/
^
lib\utils.c:254:5: warning: (near initialization for 'goObjMemberType.tp_basicsi
ze') [enabled by default]
lib\utils.c:272:5: warning: initialization makes pointer from integer without a
cast [enabled by default]
Py_TPFLAGS_DEFAULT, /*tp_flags*/
^
lib\utils.c:272:5: warning: (near initialization for 'goObjMemberType.tp_doc') [
enabled by default]
lib\utils.c:273:5: warning: initialization from incompatible pointer type [enabl
ed by default]
"GoObjMember objects", /* tp_doc */
^
lib\utils.c:273:5: warning: (near initialization for 'goObjMemberType.tp_travers
e') [enabled by default]
lib\utils.c:282:5: warning: initialization from incompatible pointer type [enabl
ed by default]
objmemb_getset, /* tp_getset */
^
lib\utils.c:282:5: warning: (near initialization for 'goObjMemberType.tp_base')
[enabled by default]
lib\utils.c:285:5: warning: initialization from incompatible pointer type [enabl
ed by default]
(descrgetfunc)objmemb_get, /* tp_descr_get */
^
lib\utils.c:285:5: warning: (near initialization for 'goObjMemberType.tp_descr_s
et') [enabled by default]
lib\utils.c:286:5: warning: initialization makes integer from pointer without a
cast [enabled by default]
(descrsetfunc)objmemb_set, /* tp_descr_set */
^
lib\utils.c:286:5: warning: (near initialization for 'goObjMemberType.tp_dictoff
set') [enabled by default]
lib\utils.c:335:5: warning: initialization makes integer from pointer without a
cast [enabled by default]
"GoNatMember", /*tp_name*/
^
lib\utils.c:335:5: warning: (near initialization for 'goNatMemberType.tp_basicsi
ze') [enabled by default]
lib\utils.c:353:5: warning: initialization makes pointer from integer without a
cast [enabled by default]
Py_TPFLAGS_DEFAULT, /*tp_flags*/
^
lib\utils.c:353:5: warning: (near initialization for 'goNatMemberType.tp_doc') [
enabled by default]
lib\utils.c:354:5: warning: initialization from incompatible pointer type [enabl
ed by default]
"GoNatMember objects", /* tp_doc */
^
lib\utils.c:354:5: warning: (near initialization for 'goNatMemberType.tp_travers
e') [enabled by default]
lib\utils.c:363:5: warning: initialization from incompatible pointer type [enabl
ed by default]
natmemb_getset, /* tp_getset */
^
lib\utils.c:363:5: warning: (near initialization for 'goNatMemberType.tp_base')
[enabled by default]
lib\utils.c:366:5: warning: initialization from incompatible pointer type [enabl
ed by default]
(descrgetfunc)natmemb_get, /* tp_descr_get */
^
lib\utils.c:366:5: warning: (near initialization for 'goNatMemberType.tp_descr_s
et') [enabled by default]
lib\utils.c:367:5: warning: initialization makes integer from pointer without a
cast [enabled by default]
(descrsetfunc)natmemb_set, /* tp_descr_set */
^
lib\utils.c:367:5: warning: (near initialization for 'goNatMemberType.tp_dictoff
set') [enabled by default]
tick
tick
--- FAIL: TestLock (0.07 seconds)
panic: (*py.BaseObject) (0x51aa20,0x1e279650) [recovered]
panic: (*py.BaseObject) (0x51aa20,0x1e279650)
goroutine 7 [running]:
runtime.panic(0x51aa20, 0x1e279650)
C:/Go/src/pkg/runtime/panic.c:266 +0xa6
testing.func┬À005()
C:/Go/src/pkg/testing/testing.go:383 +0xd9
runtime.panic(0x51aa20, 0x1e279650)
C:/Go/src/pkg/runtime/panic.c:248 +0xe8
lime/3rdparty/libs/gopy/lib.(*Lock).dec(0x2e4a02e8, 0x0)
C:/Users/hb19623/Gocode/src/lime/3rdparty/libs/gopy/lib/lock.go:203 +0x1
64
lime/3rdparty/libs/gopy/lib.(*Lock).Unlock(0x2e4a02e8)
C:/Users/hb19623/Gocode/src/lime/3rdparty/libs/gopy/lib/lock.go:227 +0x5
5
lime/3rdparty/libs/gopy.TestLock(0x2e4a9240)
C:/Users/hb19623/Gocode/src/lime/3rdparty/libs/gopy/lock_test.go:30 +0x3
6
testing.tRunner(0x2e4a9240, 0x6baa4c)
C:/Go/src/pkg/testing/testing.go:389 +0x87
created by testing.RunTests
C:/Go/src/pkg/testing/testing.go:469 +0x6d2
goroutine 1 [chan receive]:
testing.RunTests(0x562994, 0x6baa40, 0x6, 0x6, 0x1)
C:/Go/src/pkg/testing/testing.go:470 +0x6ed
testing.Main(0x562994, 0x6baa40, 0x6, 0x6, 0x6bea20, ...)
C:/Go/src/pkg/testing/testing.go:401 +0x6a
main.main()
lime/3rdparty/libs/gopy/_test/_testmain.go:57 +0x82
goroutine 3 [syscall]:
runtime.goexit()
C:/Go/src/pkg/runtime/proc.c:1396
exit status 2
FAIL lime/3rdparty/libs/gopy 3.676s
|
|
Your gopy is outdated, please sync. |
|
I followed jackielii's instuctions. Thanks! I got further at least. go test in gopy still failing. Here is the trace: |
|
I don't have a Windows machine, so I'm not sure how much help you can be, but if you figure anything out we'd love to hear how you got it to work. |
|
I suspect my latest error (linker unable to find definition for _PyGC_generation0) is caused by memory.go using a reference to _PyObject_GC_TRACK which is the macro version of PyObject_GC_Track . This requires the build system to be able to access gcmodule.c to find the definition of _PyGC_generation0. Not sure why this works on linux as I don't have a linux machine handy. memory.go is in some sense an extension module and in the extension documentation it says that the macros should not be used for extension modules. I am at work but will see if replacing the macro with the function will fix it when I get home. |
|
@l1n3n01z Yes, you're right, I've no idea how this works in Linux, but I inserted a line: just after and it compiles, but I don't know if it's the right thing to do. After done that and go test. You'll get that error that trace back to lock.go:193 @quarnster thanks. but I have no luck after updating gopy to c880d66, still the same error. I'll keep investigating, and maybe come back with a pull request:) |
|
Did the "tick" prints go away at least? If not it's not updated as that test was removed. There was a fix in the lock test so that PyFinalize isn't called until python has actually finished which might have been a cause of the issue you are seeing. But if you are updated and you are still seeing it, I don't know what's broken |
|
Yes, the tick was gone. About this error, I printed the python type and see it's "int" and with value of "1", but in go arena it's a *py.BaseObject, not a *py.Long. Any idea how this happened? |
|
In memory.go I changed line 18 to |
|
(tl;dr: &C.PyLong_Type != &PyLong_Type in utils.c) Further investigation reveals: calls calls calls which has this switch statement However! C.getBasePyType(obj) is never matched in the switch statement so newObject returns newBaseObject(obj) which causes the returned object to be of the wrong type as @jackielii observed. More curiously, prints 0x1e23e158 which is the same value as internally in utils.c prints 0x6bf784 I'm not sure what conclusions to draw from this :) I'll keep looking. |
|
EDIT: sorry, my test file was wrong, please ignore this. I get same error as the original one. @l1n3n01z I went through exactly the same stuff as you noted here. I wrote a test file called long_test.go in package py
import (
"fmt"
"testing"
)
func TestLong(t *testing.T) {
l := NewLong(10)
d, err := NewDict()
if err != nil {
t.Fatal(err)
}
if err = d.SetItemString("gopy.count", l); err != nil {
t.Fatal(err)
}
if l2, err := d.GetItemString("gopy.count"); err != nil {
t.Fatal(err)
} else {
if l3, ok := l2.(*Long); !ok {
t.Fail()
} else {
fmt.Println(l3.Int64())
}
}
}It actually passes. So the problem might lie in dict := newDict(C.PyThreadState_GetDict()) in lock.go:159, :169, :188 |
|
I also created a test, but in This fails in exactly the same manner as expected, i.e. the object is changed into a base object. |
|
I'm a bit confused. I was going back and forth, and now I seem to be unable to run your test singly. How are you running your test without running into a failure first? |
|
If I add to object.c then TestLong2 passes. and also why this works in Linux. (My utterly uneducated guess is that the whole of python is being compiled at the same time or something, rather than just being linked in) I must admit that I am a bit concerned that there may be a plethora of similar bugs waiting for us. |
|
What's the output of this: package main
/*
#cgo CFLAGS: -I/usr/local/include/python3.3m -I/usr/local/include/python3.3m -Wno-unused-result -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
#cgo LDFLAGS: -L/usr/local/lib -ldl -framework CoreFoundation -lpython3.3m -framework CoreFoundation
#include <Python.h>
PyTypeObject* getLongObj() { return &PyLong_Type; }
*/
import "C"
import "fmt"
func main() {
C.Py_InitializeEx(0)
defer C.Py_Finalize()
a := &C.PyLong_Type
b := C.getLongObj()
fmt.Printf("a: %+v\nb: %+v\nequal: %v\n", a, b, a == b)
}As a reference, on my mac it's: |
|
@quarnster the result of that program on my machine is: $ go run gopytest.go
a: &{ob_base:{ob_base:{ob_refcnt:505667928 ob_type:0x1e125860} ob_size:504525360} tp_name: tp_basicsize:1986950281 tp_itemsize:0 tp_dealloc:0x748bdf14 tp_print:0x748bdfc8 tp_getattr:0x748be124 tp_setattr: tp_reserved:0x74059ef7 tp_repr: tp_as_number:0x7711f8bc tp_as_sequence: tp_as_mapping:0x74d91826 tp_hash:0x74d9183e tp_call:0x74dad35b tp_str:0x74d913e0 tp_getattro:0x74d9322c tp_setattro:0x74d93475 tp_as_buffer:0x74e14c54 tp_flags:1960384598 tp_doc:0x74d979b0 tp_traverse:0x74d9516b tp_clear:0x74d95183 tp_richcompare:0x74d91222 tp_weaklistoffset:1960399187 tp_iter:0x74d9496a tp_iternext:0x74d934a9 tp_methods:0x74db796c tp_members:0x74d948cb tp_getset:0x74d94977 tp_base:0x74d9438f tp_dict:0x74d989a9 tp_descr_get:0x74d91695 tp_descr_set:0x74d9325b tp_dictoffset:1960557255 tp_init:0x74db7d16 tp_alloc:0x74d91136 tp_new:0x74d91282 tp_free:0x74d9165c tp_is_gc:0x74d92cdc tp_bases: tp_mro:0x74c82900 tp_cache:0x74c48e6b tp_subclasses:0x74be9894 tp_weaklist:0x74bf76ac tp_del:0x74be9cee tp_version_tag:1958802771}
b: &{ob_base:{ob_base:{ob_refcnt:67 ob_type:0x1e241408} ob_size:0} tp_name:0x1e1e81f8 tp_basicsize:12 tp_itemsize:2 tp_dealloc:0x1e0a5ae0 tp_print: tp_getattr: tp_setattr: tp_reserved: tp_repr:0x1e0a4960 tp_as_number:0x1e23e0d0 tp_as_sequence: tp_as_mapping: tp_hash:0x1e0a5c30 tp_call: tp_str:0x1e0a4960 tp_getattro:0x1e0aeb00 tp_setattro:0x1e0aed00 tp_as_buffer: tp_flags:17568768 tp_doc:0x1e23de70 tp_traverse: tp_clear: tp_richcompare:0x1e0a5b70 tp_weaklistoffset:0 tp_iter: tp_iternext: tp_methods:0x1e23dd48 tp_members: tp_getset:0x1e23de08 tp_base:0x1e241708 tp_dict:0x29541c0 tp_descr_get: tp_descr_set: tp_dictoffset:0 tp_init:0x1e0b95b0 tp_alloc:0x1e0b80b0 tp_new:0x1e0a73a0 tp_free:0x1e0b0130 tp_is_gc: tp_bases:0x29514b0 tp_mro:0x2954620 tp_cache: tp_subclasses:0x2954080 tp_weaklist:0x29568a0 tp_del: tp_version_tag:179}
equal: false
|
|
"a" looks completely messed up. I'd search for an existing or open up a new bug |
|
Could be issue 4339 or a variant thereof. Does your python end up as a dll or a static library? If you change it to the other form, does that make any difference? |
|
@quarnster I didn't compile python with mingw from scratch, I downloaded binary installer for windows 32bit. and use pexport and dlltool to create from python33.dll to libpython33.a see the comment: #119 (comment) |
|
I concur, It seems to be the same bug. I am linking against python as a dll and PyLong_Type is exported in a similar manner as in issue 4339. I've been searching for something like this using variants/combinations of the search words "golang", "static linking", "golang dll", "static variable", "cgo" but not found anything. Stupidly I didn't search the golang bug database directly. Question is, is the way forward to access the static variables as suggested in 4339, using short inline wrapper functions (this worked for me in my test) and change a lot of code in the project, or do we link statically against Python? It seems in any case that Python should be internal to the project, as it needs the special compilation of signal handling. |
|
Actually it seems that building Python on Win using MingW is quite unorthodox and apparently is a major hassle (impossible according to some, though others have managed to get at least a Python, though without some modules building properly). It will also result in incompatibility with binary extensions. Most people on Win use binary extensions as building Python stuff (language itself and/or extensions) in general, using MSVC, is a huge effort and requires downloading tons of hard to find Microsoft stuff. |
|
FWIW, IIRC there's something odd going on with mingw32/64 as I've stumbled upon issues using a mingw created dll from python in the past. Don't know if it's possible to use standard ms' cl together with go, but that might be another path.
Actually, they probably just require the commandline compiler not the full MSVC. The commandline compiler tools are in the Windows 7 sdk.
IMO not this one because it adds complexity that shouldn't be needed and if any other dll is used in the future similar problems wouldn't be solved by this work around.
Is it known for sure that this fixes this issue?
Ideally building Python should be part of the build system discussed in #107. |
|
BTW, does windows even have sigaltstack? It's possible a standard non-modified python would work just fine on windows, except for the 4339 of course. |
Not entirely sure if this is such a big hurdle. It seems to me to me that there are relatively few of those static variables lying around anyway, and to link against a dynamic library through a static export library does seem to be the accepted way of doing things in Windows. Any other dlls which we wanted to build against through cgo would have to use the same workaround, yes, but that would be inevitable anyway. Using these wrappers wouldn't really affect anyone else, except it would make the code slightly more indirect in these few cases, and would have some minor effects on speed. As the object thunking is done quite a bit, we could also invest a bit into just having constants for the base types and just returning a constant for the switch statement. I'm happy to do this work. On the other hand: The solution of compiling python into the project seems to be a much bigger hurdle in Windows, requiring either of two very difficult things, getting golang to use the msvc ecology or python to compile with mingw. On whether python on Windows has sigalstack, I am unsure (not entirely sure what a sigalstack is, TBH, ehemm..), but the test for checking for it results in a negative with the default python (from here http://stackoverflow.com/questions/19546418/how-to-compile-python-3-without-sigaltstack-enabled ). So it should be possible to use an off-the-rack Python. |
|
@jackielii For future reference it would be useful if you could paste in the complete gopytest.go file you ran, as I am sure you changed some of the cgo flags before running it from what they were in the original file from @quarnster . Also we should clean up this discussion and put it up on the wiki ASAP for other utterly lost Windows devs 😃 |
|
Copied @jackielii first post to https://github.com/limetext/lime/wiki/Building-on-Windows . Cleaned it up a little bit. We should then refine it :-) |
@l1n3n01z, ok go for it. And thanks for kicking off the wiki page! |
|
I haven't had time to setup all the things but I will do in the next week (time permitting). I would seriously take a look at checking out the following repo for Windows users. https://github.com/lukesampson/scoop/ Currently included packages: Check out some of the others also https://github.com/lukesampson/scoop/tree/master/bucket There are no packages for Oniguruma or qt5, I haven't tested any of this yet but I am quite sure (and hoping) that this will make much of what is above in this ticket not required. I am also in the 'I know nothing boat' regarding HAVE_SIGALTSTACK so that could be an issue if IT IS required. |
|
Great idea. I just heard about scoop also just this week. It would definitely simplify all the installation, and it would also help with isolating all the stuff in a single directory. Am I right that we could even write our own packet definition and it would just take care of all the preliminaries? So going forward, we could use this to simplify the installation and the cgo windows flags to isolate the windows specific things in the build. I also think that qt5 should be made completely optional, so that you if want it you specify "qt" in your build tags. I suspect that qt5 is just there as a proof of concept and that the idea is that in the future you could use pretty much any GUI kit that you are willing to write a front end for. |
|
I'd prefer to just use cmake which can be configured to download, patch and build all dependencies as needed and can generate project files for your favorite build system whether that is pure makefiles, msvc, eclipse, xcode or whatever. Yes the qt5 frontend is completely broken (as opposed to the termbox frontend which is just partially broken ;)). QML looks like a much better approach imo if people want to use qt5. People might also want to look into creating a skia frontend, which is what ST3 is using to render with, created issue #129 for that. The graphics for ST3 themes is pretty much layered 9-patch ui elements IIRC, so shouldn't be too hard to code up. In the history of the gh-pages branch there's a css version and |
|
On cmake: I'm not familiar with cmake, and was not thinking of using scoop as a build system per se. Only as a helper for installing the needed software. Is cmake a good fit for that? I found this discussion http://stackoverflow.com/questions/8153519/how-to-automatically-download-c-dependencies-in-a-cross-platform-way-cmake which looks like it is relevant. |
Yes, one of my other repos downloads source code for compilers, cross compiles the compilers, then downloads, patches, builds and installs all dependencies to be able to cross compile XBMC for the Boxeebox. |
|
Excellent :) Anyway, back to the topic at hand. @jackielii I have updated my lime fork https://github.com/l1n3n01z/lime with the fixes I made to object.go, using constant numbers as type identifiers. It fixes all but one test for me. Can you please test on your machine? @quarnster I have not tested on linux or mac, and not sure what our integration model should be. I can probably set up a VM for linux, but not OsX The last test which is failing for me ends in a segfault or something of the sort. It should be possible to run all tests successfully by commenting out module_test.go:112 It seems that this codepath invokes a call through libffi which fails spectacularly. No idea why. I'm not a hundred percent sure, but I think I have found the line that fails through judicious printing and exception throwing. Of course this being c it is quite possible that the actual error is somewhere else. The line in question is utils.c:132 We should create a minimum repro for this ffi_call failure in a test. I should be able to do it in the weekend but it's up for grabs :) |
|
@l1n3n01z Hi, I cloned your fork and ran the tests, everything is passing except When run just Great work, Thanks! |
|
I think I have solved the last crash. libffi is interfacing with Python which - because it is compiled with msvc - uses msvc declspec. So it needs to be configured to use the correct declspec. Can you try the following, which fixes the crash for me?: In utils.c:127, change to It should fix the last test. The actual fix will require a bit more finesse not to break compatibility for other architectures and also needs to do the change everywhere where libffi is used. It also seems to me that gopy requires a bit more unit tests, since the PyParse_Tuple codepath is not even touched. I hope to get a patch out soon. |
We can make the cmake build system select the correct one (#107). I'll work on it this weekend.
Yes, the original gopy didn't have any unit tests at all and the ones that are there now were all created by myself to figure out where things were broken IIRC. Please do add more tests as you see fit both for gopy and lime. |
|
At another computer and the fix is not working there, I may have been too hasty. We'll see :) |
|
@l1n3n01z yes, It's not working on my system either. |
|
I was returning a value from a shortcut in the test that passed 😊 In other news. libffi seems to be borked. I'm using it internally for some testing and it plain doesn't work when passing any arguments at all. In utils.c I added the following functions First I checked some things. I replaced doBuildValue with the following as a sanity check: This works and the test passes. When I replace it with this, the test also passes and prints out as expected "test void" If I do the same thing, but making ffi call HOWEVER! If I do this: libffi thrashes the stack and everything fails horribly. I have tested a number of different ffi types and calling conventions. It always dies horribly. Not sure what to do TBH. Talk to the libffi guys, I guess. |
|
Another thing. Why is libffi not being called with Why does this code actually work in linux/OsX? |
My guess would be the same c compiler being used for both python and libffi. I didn't write the original code so don't know it too well, but I think libffi is only used for the variable argument functions. Getting gopy working aside, are these problematic code paths important for lime specifically? The easy solution might be to just remove them. |
|
There is a "clone" of gopy that eliminated libffi, I don't know if we can use it? (I didn't look into the code myself) |
|
@quarnster I'm not sure we can remove the codepaths, I suspect this is heavily used. I can look into it. It might actually be good to know what functionality is required of gopy and why. I was under the impression that it would be used for all of the python code. @jackielii That's quite probable, it looks like ffi is only used to call two python API functions, and the only reason it is used is because these take variadic arguments. There might be easier ways to get the same functionality using varargs. I will look into that. |
|
silly me, forgot to paste the url!!! |
|
It doesn't look like the ffi path is used by lime. I'll remove it in an upcoming commit |
|
Excellent news! Looking forward to it. |
|
I've made lots of changes all over the place. There are still odd things going on with python, but if |
|
I've been busy with other things, but I'll try cmake from scratch over the weekend. Thanks for all the work on it @quarnster . Did you have the opportunity to try it out on Win? @jackielii Did you ever get linking errors for onigiruma when building the editor? Onigiruma is supposed to be linked statically I think, at least the mingw build creates only static libs, but it seems cgo is not entirely happy about that. |
|
Yes, my previous comment was with Windows 7.1. FWIW I removed my windows Go installation and just let the build system download Go in case that makes any difference regarding the linking errors you're seeing. |
|
BTW, the static lib is a workaround for issue 4339 which only shows up for dlls. If it's possible to compile python 3 as a static lib on windows too, that would be ideal but is not something I've looked into. |
|
This might be of interest: https://github.com/bendmorris/static-python/tree/3.3 |
|
I think #228 has superseeded this one. Please re-open if I'm mistaken. |
I am trying to build on Windows 64 and seem to be way out of my depth.
I have followed the build guide in the readme and I'm stuck on building gopy.
I have setup Mingw-w64, which seemed to be the most sensible way to get off the ground.
I have compiled libffi using it.
I have changed cgo.go to the following:
When running
[..]\libs\gopy> go test
I get the following output:
Which seems to suggest the I am doing something wrong in cgo.go.
Has anyone been able to build on Windows yet? Any pointers?
The text was updated successfully, but these errors were encountered: