Can't run on PyPy on ARM

[depau]

What did you do?

$ pypy -m pip install pillow
Collecting pillow
  Downloading Pillow-4.3.0.tar.gz (13.9MB)
    100% |████████████████████████████████| 13.9MB 11kB/s 
 Collecting olefile (from pillow)
  Downloading olefile-0.44.zip (74kB)
    100% |████████████████████████████████| 81kB 498kB/s 
Installing collected packages: olefile, pillow
  Running setup.py install for olefile ... done
  Running setup.py install for pillow ... done
Successfully installed olefile-0.44 pillow-4.3.0

$ pypy
Python 2.7.12 (5.6.0+dfsg-4, Nov 20 2016, 10:56:52)
[PyPy 5.6.0 with GCC 6.2.0 20161103] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>> from PIL import Image

What did you expect to happen?

I expected it to import Image.

What actually happened?

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/pypy2.7/dist-packages/PIL/Image.py", line 56, in <module>
    from . import _imaging as core
ImportError: function _cffi_pypyinit__imaging or init_imaging not found in library /usr/local/lib/pypy2.7/dist-packages/PIL/_imaging.pypy-41-arm-linux-gnueabihf.so

What versions of Pillow and Python are you using?

• Pillow 4.3.0
• PyPy 5.6.0 with GCC 6.2.0 20161103
• Debian Stretch on a Banana Pi M2 Berry
• Linux Banana 3.10.107-BPI-M2U-Kernel Merge pull request from @shuge #1 SMP Thu Aug 24 08:01:24 CST 2017 armv7l GNU/Linux
• CPU reports itself as an ARMv7 Processor rev 5 (v7l)
• CPU flags: swp half thumb fastmult vfp edsp neon vfpv3 tls vfpv4 idiva idivt vfpd32

[wiredfool]

That's not a combination that I've ever heard of anyone trying. We could probably check on a raspberry pi, but that's about as close to that hardware as I've got access to.

[depau]

Raspberry Pi should have similar hardware. I was trying to run OctoPrint on PyPy as it's much faster (you can really notice the difference), but there's a plugin (the one that I referenced) that depends on Pillow.

[wiredfool]

I've tried this on a RaspberryPi2, running Raspbian Stretch, and I can't replicate this issue.

With system pypy, (and pypy-dev) I can build from source on current master (which includes a tweak for fixing compilation on Debian. Tests mostly pass. I can also install from pip install pillow (version 4.3.0), tests again mostly pass. The test to fail are mainly memory leaks that are very sensitive, and prone to false positives.

I was unable to get the latest pypy 5.9.0 to run at all -- it failed with a library error.

[depau]

I don't know, I just reinstalled it again in a new virtualenv and I still get

>>>> from PIL import Image
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/depaulicious/venv/site-packages/PIL/Image.py", line 56, in <module>
    from . import _imaging as core
ImportError: function _cffi_pypyinit__imaging or init_imaging not found in library /home/depaulicious/venv/site-packages/PIL/_imaging.pypy-41-arm-linux-gnueabihf.so

As a temporary workaround, if it works on your setup, you should be able to build and upload an armhf wheel to PyPI, but that's just a workaround.

[depau]

Any updates on this? What options do I need to pass to get the most verbose build output so I can send it here?

[depau]

By the way, I tried to build it again with -vf --debug, build was successful but it still doesn't import.

https://hastebin.com/jisajetiba.hs

>>>> from PIL import Image
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "PIL/Image.py", line 56, in <module>
    from . import _imaging as core
ImportError: function _cffi_pypyinit__imaging or init_imaging not found in library PIL/_imaging.pypy-41-arm-linux-gnueabihf.so

I'll try to install all optional libraries and build it again.

Update: it didn't help. I checked _imaging.c source and noticed the init_imaging function at the bottom of the file. I tried making that preprocessor if always evaluate false to make sure it was present in the output but it didn't help.

[depau]

I tried to install it again after a few months and nothing has changed. I'll try to ask PyPy devs and see if they know why this happens.

EDIT:
PyPy issue: https://bitbucket.org/pypy/pypy/issues/2762

[mattip]

PyPy 5.6 is quite an old version of PyPy, could you try with a newer version? We have worked quite a bit on the compatibility layer needed to support c-extension modules over the last 15 months, so you may find that PyPy 5.10 just works, as it does for me on an Odroid board. You simply download the EL or HF (HF is modern, but check this page to find out if your hardware needs EL), untar the file into a directory, and then call pypy-c -mensurepip to install pip and setuptools, then pypy -mpip install pillow

You seem to be using hf, maybe you should be using el with that hardware?

You could also use nm to find out whether that function is exported: nm library PIL/_imaging.pypy-41-arm-linux-gnueabihf.so | grep init_imaging

@wiredfool - if 5.10 doesn't work for you, could you file an issue (assuming you haven't)?

[depau]

@mattip nm _imaging.pypy-41-arm-linux-gnueabihf.so | grep init_imaging outputs

0000d0f8 T init_imaging

so it is exported. I'll update PyPy and let you know.

[depau]

I downloaded PyPy from their website. Now I have this issue:

Python 2.7.13 (0e7ea4fe15e8, Dec 22 2017, 08:46:38)
[PyPy 5.10.0 with GCC 4.7.2 20120731 (prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>> from PIL import Image
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/pypy2-v5.10.0-linux-armhf-raspbian/site-packages/PIL/Image.py", line 58, in <module>
    from . import _imaging as core
ImportError: unable to load extension module '/opt/pypy2-v5.10.0-linux-armhf-raspbian/site-packages/PIL/_imaging.pypy-41.so': /usr/lib/arm-linux-gnueabihf/libc.so: invalid ELF header

I think that's because I downloaded the Raspbian image and this is not Raspbian (it's Debian stretch). I'll try to build it from source.

BTW cat /usr/lib/arm-linux-gnueabihf/libc.so:

/* GNU ld script
   Use the shared library, but some functions are only in
   the static library, so try that secondarily.  */
OUTPUT_FORMAT(elf32-littlearm)
GROUP ( /lib/arm-linux-gnueabihf/libc.so.6 /usr/lib/arm-linux-gnueabihf/libc_nonshared.a  AS_NEEDED ( /lib/arm-linux-gnueabihf/ld-linux-armhf.so.3 ) )

[mattip]

it seems something is not right with your build tools. Can you build and run any gcc project with a shared library?

[depau]

I fried my Banana Pi a few days ago so I can't test it any more 🤦‍
I will get a new SBC soon and test it again.

[wiredfool]

Unfortunately, I'm between pis as well.

[aclark4life]

@depau @wiredfool So we need a raspberry PI to test this?

[wiredfool]

We never did get it to replicate on a RaspberryPI.

[depau]

@aclark4life I think so. Maybe you can replicate it with qemu.

[aclark4life]

I'm sure we could get someone to donate a Raspberry Pi, we just need a volunteer to fix it. @wiredfool If a Raspberry Pi shows up at your door, do you want to fix this one still?

[wiredfool]

I’ve got pis, I just need to retrieve them from teenager.

[radarhere]

The BitBucket link no longer works, but it can be seen at https://web.archive.org/web/20200801201818/https://bitbucket.org/pypy/pypy/issues/2762

Also, Pillow no longer supports Python 2.7, and Debian 9 has also stopped receiving security updates.

[mattip]

The correct link is now https://foss.heptapod.net/pypy/pypy/-/issues/2762

[radarhere]

Note that the BitBucket issue closed, with the comment that it "seems to be a specific system failure"

[radarhere]

Given that we were unable to replicate this, and the last report from the OP is they no longer have an environment to replicate this, closing.