PNG max dimensions

[jamesra]

Pillow is raising an exception when attempting to write large PNG's with dimensions larger than a signed 16-bit integer. Previously I believed that the maximum dimension was 65536x65536 but the spec states the dimension is stored in 4 bytes. http://www.libpng.org/pub/png/spec/iso/index-object.html#11IHDR I am requesting support for 65K.

Oddly Pillow is able to write images with both dimensions larger than 2 << 31. Where the limit for crashing lives isn't clear to me yet.

This is running on Win7 x64 using Pillow 2.2.1 and Python 2.7.6 x64. Pillow was downloaded from Gohlke's site. Scipy is 13.1 and numpy is 1.8

I've done some debugging and the crash appears to occur in Image.load. Referencing the Images "im" member results in a "SystemError: error return without exception set" This is where I got stuck.

I've uploaded code that reproduces the behavior to https://github.com/jamesra/random/blob/master/pillow_png_size.py

I try to avoid PNG's of this size but I have ported some code from ITK (C++) to Scipy/Pillow and the lower limit creates a backward compatibility issue for me.

If this is fixable within the python code with a couple hours of time I'm willing to attempt it. Eclipse wasn't letting me dig past the self.im line that throws the exception.

[wiredfool]

Yeah, well, that's big. The OOM killer gets me when numpy is allocating the second bunch of memory. (at least, on my dev vm. I can try this on something bigger)

I think that the fromarray is a memory copy, I'm pretty sure that the direct Image.new is an equivalent call without the dependencies. A more concise test would be something like:

from PIL import Image
ydim = 32769
xdim = 48000

def test(xdim,ydim):
   im = Image.new('L',(xdim,ydim),(0))
   im.save('test.png')

test(xdim,ydim)
test(xdim,xdim)

[wiredfool]

Actually, I think this is more fundamental:

Python 2.7.3 (default, Apr 20 2012, 22:39:59) 
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from PIL import Image
>>> im = Image.new('L', (48000,48000), (0))
>>> im
<PIL.Image.Image image mode=L size=48000x48000 at 0x7F9B794AF4D0>
>>> im.im
<ImagingCore object at 0x7f9b7958f090>
>>> im.im and True
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
SystemError: error return without exception set
>>> im = Image.new('L', (100,100),(0))
>>> im and True
True

It looks like there's an issue somewhere with images that are greater than 2 GB.

[jamesra]

Cool, thanks for taking a look. Maybe there is hope someone is using an signed int32 in a x64 build?

[wiredfool]

Does look that way, doesn't it.

I think this is it: https://github.com/python-imaging/Pillow/blob/master/_imaging.c#L3076

[wiredfool]

So, FWIW: SystemError: error return without exception set is an error from the C extension level where something is returning an error condition but not setting an exception. It's not going to be something that is going to be traceable in the python side of things.

In this case, it's because of a function returning a Py_ssize_t (a signed 64 bit integer) by calling return xsize * ysize, which were both signed 32bit ints. If that was > 2gpix, then it overflowed into negative values. That was then cast to a Py_ssize_t as a simple widening, preserving the sign. Casting before returning prevents the overflow.

Try that patch and see what it does for you.

[jamesra]

Thanks for chasing that down. I haven’t debugged Python extensions before. Out of curiosity with that fix is there any limit to PNG dimensions?

Anything I can do to help?

[wiredfool]

Test it. I checked the basics this morning and it was able to save a 64k-1x64k-1 png. It took 4 gigs of memory to do it, but it worked.

As I noted in the pull request, the quantization routines are limited to int32 pixels, and it's a little more complicated to fix than just a few casts. (and, I'd have to touch up the testing, and to do that, understand what the quantization is really doing).

[jamesra]

I’m trying to test the fix, but I am having trouble with building. Setup.py build claims I have ZLIB support. Selftest.py claims the support is missing. The log is below. Any ideas?

Thanks.

c:\Src\Git\Pillow>python setup.py build
running build
running build_py
running egg_info
writing Pillow.egg-info\PKG-INFO
writing top-level names to Pillow.egg-info\top_level.txt
writing dependency_links to Pillow.egg-info\dependency_links.txt
reading manifest file 'Pillow.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'Pillow.egg-info\SOURCES.txt'

running build_ext

PIL SETUP SUMMARY

version Pillow 2.3.0
platform win32 2.7.6 (default, Nov 10 2013, 19:24:24)

[MSC v.1500 64 bit (AMD64)]

*** TKINTER support not available
(Tcl/Tk 8.5 libraries needed)
--- JPEG support available
--- ZLIB (PNG/ZIP) support available
*** TIFF G3/G4 (experimental) support not available
*** FREETYPE2 support not available
*** LITTLECMS2 support not available
*** WEBP support not available

*** WEBPMUX support not available

To add a missing option, make sure you have the required
library, and set the corresponding ROOT variable in the
setup.py script.

To check the build, run the selftest.py script.

running build_scripts

c:\Src\Git\Pillow>python selftest.py

Pillow 2.3.0 TEST SUMMARY

Python modules loaded from c:\Src\Git\Pillow\PIL

Binary modules loaded from c:\Src\Git\Pillow\PIL

--- PIL CORE support ok
*** TKINTER support not installed
*** JPEG support not installed
*** ZLIB (PNG/ZIP) support not installed
*** G4 TIFF support not installed
*** FREETYPE2 support not installed
*** LITTLECMS2 support not installed

*** WEBP support not installed

Running selftest:

---

File "c:\Src\Git\Pillow\selftest.py", line 57, in selftest.testimage
Failed example:
try:
_info(Image.open(os.path.join(ROOT, "Images/lena.jpg")))
except IOError as v:
print(v)
Expected:
('JPEG', 'RGB', (128, 128))
Got:
decoder jpeg not available

---

1 items had failures:
1 of 57 in selftest.testimage
_Test Failed_ 1 failures.
*** 1 tests of 57 failed.

[jamesra]

Nevermind. I think what is happening is I have a failure at link time, but the setup package found zlib and jpeg files so the summary is claiming they are there.

[jamesra]

I’ve got a build with the fix running. Using my original test I no longer get the same error However there is an access violation occurring sometime after the file is created and some data is written to it. I’m building a debug version of pillow so I can provide more detailed information than an address but I need to find/build python_d first.

I did not use the simplified “Allocate an image and save it”. I am using the fromarray using a numpy array. It is entirely possible I have strangeness in my build somewhere. I don’t have JPG support to run the selftest.py yet.

I’ll let you know once I have a cleaner story. It may be Monday before I can get back to working on code.

Thanks

[wiredfool]

I built a debug version of python windows once, and... It did work, but it wasn't a pleasant experience. These days, I'm using Ubuntu, where I can just install the package.

(Though, I read something on integrated python/C debugging in the latest visual studio, which would be nice.)

[jamesra]

Yea… I gave up for the night. I can actually build Python_d in Windows with the included solution. I got Pillow built with self.debug=True. setup tools has PDB’s turned off for visual studio for some reason so I got those generated. However when I do a debug pillow build and install it I cannot from PIL import Image.

If I run python from the pillow directory I can import Image correctly but I think that is just an illusion. When I put the process in visual studio and crash the images don’t match so I cannot load the pdb file.

I think it may be because _image is now _image_d?

Anyway, if any of that seems familiar let me know. Otherwise perhaps I should try and install the debug build on a Mac I have. That is a whole different learning curve for me though.

[jamesra]

Sorry for the delay in getting back to you. I was taking an involuntary vacation due to weather. I’ve repeated the test of saving large PNG’s using your fix on my Mac. The stack for the next crash is below. My dev experience on the mac is limited. I can try for a debug build if it would be helpful and you are interested in chasing this further.

There are some warnings thrown by the MSVC compiler regarding 64-bit portability.

Process: Python [25893]
Path: /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
Identifier: Python
Version: 2.7.6 (2.7.6)
Code Type: X86-64 (Native)
Parent Process: bash [25812]
Responsible: Terminal [10247]
User ID: 501

Date/Time: 2013-12-12 21:39:32.082 -0800
OS Version: Mac OS X 10.9 (13A603)
Report Version: 11
Anonymous UUID: 11E0853F-E5F3-7C9B-56D7-858EA85B8BF4

Sleep/Wake UUID: 1EB2C86F-581A-4A95-BF3A-D4723BCB6EF4

Crashed Thread: 0 Dispatch queue: com.apple.main-thread

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x00000000e9905e00

VM Regions Near 0xe9905e00:
-->
__TEXT 000000010ab91000-000000010ab93000 [ 8K] r-x/rwx SM=COW /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 libsystem_platform.dylib 0x00007fff902ebbe0 _platform_memmove$VARIANT$Unknown + 224
1 _imaging.so 0x000000010ba26f4c copy1 + 12 (Pack.c:414)
2 _imaging.so 0x000000010ba3311e ImagingZipEncode + 494 (ZipEncode.c:163)
3 _imaging.so 0x000000010ba13adf _encode + 111 (encode.c:115)
4 org.python.python 0x000000010ac37010 PyEval_EvalFrameEx + 7712
5 org.python.python 0x000000010ac35076 PyEval_EvalCodeEx + 1734
6 org.python.python 0x000000010ac3bf36 fast_function + 294
7 org.python.python 0x000000010ac3828b PyEval_EvalFrameEx + 12443
8 org.python.python 0x000000010ac35076 PyEval_EvalCodeEx + 1734
9 org.python.python 0x000000010ac3bf36 fast_function + 294
10 org.python.python 0x000000010ac3828b PyEval_EvalFrameEx + 12443
11 org.python.python 0x000000010ac35076 PyEval_EvalCodeEx + 1734
12 org.python.python 0x000000010ac3bf36 fast_function + 294
13 org.python.python 0x000000010ac3828b PyEval_EvalFrameEx + 12443
14 org.python.python 0x000000010ac3bed2 fast_function + 194
15 org.python.python 0x000000010ac3828b PyEval_EvalFrameEx + 12443
16 org.python.python 0x000000010ac35076 PyEval_EvalCodeEx + 1734
17 org.python.python 0x000000010ac349a6 PyEval_EvalCode + 54
18 org.python.python 0x000000010ac5c611 PyRun_FileExFlags + 161
19 org.python.python 0x000000010ac5c15e PyRun_SimpleFileExFlags + 718
20 org.python.python 0x000000010ac70002 Py_Main + 3314
21 libdyld.dylib 0x00007fff897fc5fd start + 1

Thread 0 crashed with X86 Thread State (64-bit):
rax: 0x00007fe7b19eb201 rbx: 0x0000000000000001 rcx: 0x000000016971f000 rdx: 0x000000000000bb80
rdi: 0x00007fe7b19eb201 rsi: 0x00000000e9905e00 rbp: 0x00007fff5506dd90 rsp: 0x00007fff5506dd90
r8: 0x0000000000000436 r9: 0x0000000000007fff r10: 0x000000000000bcc1 r11: 0x00007fe6c80e5401
r12: 0x000000010b688638 r13: 0x000000010ae2d024 r14: 0x000000010b9ce908 r15: 0x00007fe7b0f46150
rip: 0x00007fff902ebbe0 rfl: 0x0000000000010283 cr2: 0x00000000e9905e00

Logical CPU: 2
Error Code: 0x00000004
Trap Number: 14

Binary Images:
0x10ab91000 - 0x10ab92fff +org.python.python (2.7.6 - 2.7.6) <9EAA7EA7-0461-3548-A192-07D38564031C> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
0x10ab97000 - 0x10acadff7 +org.python.python (2.7.6, [c] 2004-2013 Python Software Foundation. - 2.7.6) <980B2D89-E75B-3026-B8F3-CFFA635C4792> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python
0x10af1d000 - 0x10af21fff +math.so (0) <7402006C-C00D-393A-9A6E-7240B2EFAAC4> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/math.so
0x10af28000 - 0x10b012fff +multiarray.so (0) <030DF069-55B2-31B4-A242-75A0E20C188E> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/multiarray.so
0x10b0b8000 - 0x10b0c4fff +datetime.so (0) <14C3C837-ACA0-3F99-9BF8-368E32EE6E24> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/datetime.so
0x10b110000 - 0x10b155ff7 +umath.so (0) <4840927E-47BA-3354-AB5B-22BF4FCD4E1B> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/umath.so
0x10b1c1000 - 0x10b1c4ff7 +_collections.so (0) /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_collections.so
0x10b1ca000 - 0x10b1cdfff +operator.so (0) <95D19C2D-417E-3A67-A1BB-97F6234BD4A4> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/operator.so
0x10b1d4000 - 0x10b1d9ff7 +itertools.so (0) /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/itertools.so
0x10b1e4000 - 0x10b1e5fff +_heapq.so (0) /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_heapq.so
0x10b1e9000 - 0x10b1f6fff +cPickle.so (0) <2CD483B1-CE6C-3BE8-BD5F-B0A20F477DBD> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cPickle.so
0x10b1fe000 - 0x10b1fffff +cStringIO.so (0) /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/cStringIO.so
0x10b204000 - 0x10b208fff +_dotblas.so (0) <8F9FAB93-80F6-32F9-A406-89A6A06E8B19> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/_dotblas.so
0x10b20c000 - 0x10b20dff7 +_functools.so (0) /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_functools.so
0x10b251000 - 0x10b26dff7 +scalarmath.so (0) <36D6FDF0-0B2A-316C-A655-5B8021B957F5> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/scalarmath.so
0x10b37f000 - 0x10b381fff +time.so (0) <7B313687-32FF-34C2-ACD8-C7A1790A3343> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/time.so
0x10b3c7000 - 0x10b3cbff7 +_compiled_base.so (0) <72B98C02-BC6F-32F2-B2FF-A875DF946BC0> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/lib/_compiled_base.so
0x10b40f000 - 0x10b413fff +lapack_lite.so (0) /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/linalg/lapack_lite.so
0x10b417000 - 0x10b424ff7 +_umath_linalg.so (0) /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/linalg/_umath_linalg.so
0x10b430000 - 0x10b431fff +grp.so (0) /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/grp.so
0x10b434000 - 0x10b434fff +future_builtins.so (0) <84110E0D-0171-3A3F-B660-70CE6B14442A> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/future_builtins.so
0x10b477000 - 0x10b47ffff +fftpack_lite.so (0) /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/fft/fftpack_lite.so
0x10b483000 - 0x10b486fff +strop.so (0) /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/strop.so
0x10b58b000 - 0x10b5c2ff7 +mtrand.so (0) /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/random/mtrand.so
0x10b610000 - 0x10b620fff +_ctypes.so (0) <2B4B6460-0913-3E02-AB47-B705EDA9D98B> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_ctypes.so
0x10b631000 - 0x10b634ff7 +_struct.so (0) <82587028-EEA5-3914-A170-AE64F1DE5B17> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_struct.so
0x10b63c000 - 0x10b63ffff +binascii.so (0) <348F9879-1F59-35F6-9432-C30EB2FF50B9> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/binascii.so
0x10b643000 - 0x10b652ff7 +libz.1.dylib (0) /opt/local/lib/libz.1.dylib
0x10b656000 - 0x10b666fff +_io.so (0) <9CE31053-F63E-358B-9A65-C609C04AB8EB> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_io.so
0x10b6bd000 - 0x10b6bfff7 +_hashlib.so (0) <6125E492-1549-3D59-9779-E1BE90A89089> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_hashlib.so
0x10b6c3000 - 0x10b704ff7 +libssl.1.0.0.dylib (0) <7F04BE5C-2F7A-3065-B96D-25DD4EE07BCA> /opt/local/lib/libssl.1.0.0.dylib
0x10b71e000 - 0x10b830fef +libcrypto.1.0.0.dylib (0) /opt/local/lib/libcrypto.1.0.0.dylib
0x10b8a1000 - 0x10b8a2ff7 +_random.so (0) /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_random.so
0x10b8a6000 - 0x10b8a7fff +fcntl.so (0) <0FDB39F1-E6D6-3771-9FEE-E140DDA8DD49> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/fcntl.so
0x10b8aa000 - 0x10b8adff7 +zlib.so (0) <14A900B4-774C-321D-9446-88B3A1A0BBC7> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/zlib.so
0x10b932000 - 0x10b93efff +parser.so (0) <2DA0D5CF-EC8E-3F25-AB26-E813D5D296A1> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/parser.so
0x10b943000 - 0x10b969ff7 +pyexpat.so (0) /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/pyexpat.so
0x10ba0a000 - 0x10ba38ff7 +_imaging.so (0) <73083548-6E32-3E4A-A535-7CC2B958415D> /Users/USER/*/_imaging.so
0x10ba59000 - 0x10ba86ff7 +libjpeg.9.dylib (0) <64FFA3DF-D255-33B5-9D22-61CD36314C53> /opt/local/lib/libjpeg.9.dylib
0x10ba8d000 - 0x10bae5ff7 +libtiff.5.dylib (0) <76780903-EAC1-3150-890C-CC2A050CF566> /opt/local/lib/libtiff.5.dylib
0x10baf3000 - 0x10bb0dff7 +liblzma.5.dylib (0) /opt/local/lib/liblzma.5.dylib
0x16977d000 - 0x169781fff +array.so (0) <4929B153-16EF-32EB-B6AA-0778F694A864> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/array.so
0x169789000 - 0x16978bfff +_locale.so (0) /opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so
0x16978f000 - 0x169797fff +libintl.8.dylib (0) <569D1B6B-5EBD-3842-99E7-479B20BF4D1B> /opt/local/lib/libintl.8.dylib
0x16979c000 - 0x16988fff7 +libiconv.2.dylib (0) <103D57C8-BE15-3BF8-BDBE-53EDBDD0B6A4> /opt/local/lib/libiconv.2.dylib
0x7fff6b763000 - 0x7fff6b796817 dyld (239.3) /usr/lib/dyld
0x7fff87727000 - 0x7fff8772eff7 libsystem_pthread.dylib (53.1.4) /usr/lib/system/libsystem_pthread.dylib
0x7fff87779000 - 0x7fff8777aff7 libsystem_sandbox.dylib (278.10) /usr/lib/system/libsystem_sandbox.dylib
0x7fff87ac4000 - 0x7fff87ac4fff com.apple.Accelerate.vecLib (3.9 - vecLib 3.9) /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
0x7fff87c49000 - 0x7fff87c50ff7 liblaunch.dylib (842.1.4) /usr/lib/system/liblaunch.dylib
0x7fff8944c000 - 0x7fff894fcff7 libvMisc.dylib (423.32) <049C0735-1808-39B9-943F-76CB8021744F> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib
0x7fff89639000 - 0x7fff89639fff com.apple.Accelerate (1.9 - Accelerate 1.9) <509BB27A-AE62-366D-86D8-0B06D217CF56> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
0x7fff897f9000 - 0x7fff897fcff7 libdyld.dylib (239.3) <62F4D752-4089-31A8-8B73-B95A68893B3C> /usr/lib/system/libdyld.dylib
0x7fff89f76000 - 0x7fff89fa5fd2 libsystem_m.dylib (3047.16) /usr/lib/system/libsystem_m.dylib
0x7fff89fa6000 - 0x7fff89fa7fff libunc.dylib (28) <62682455-1862-36FE-8A04-7A6B91256438> /usr/lib/system/libunc.dylib
0x7fff8a6f8000 - 0x7fff8a71fff7 libsystem_network.dylib (241.3) <8B1E1F1D-A5CC-3BAE-8B1E-ABC84337A364> /usr/lib/system/libsystem_network.dylib
0x7fff8aa0f000 - 0x7fff8aa11ff3 libsystem_configuration.dylib (596.12) /usr/lib/system/libsystem_configuration.dylib
0x7fff8aa29000 - 0x7fff8aa77fff libcorecrypto.dylib (161.1) /usr/lib/system/libcorecrypto.dylib
0x7fff8ab39000 - 0x7fff8ab40ff3 libcopyfile.dylib (103) <5A881779-D0D6-3029-B371-E3021C2DDA5E> /usr/lib/system/libcopyfile.dylib
0x7fff8ac9d000 - 0x7fff8aca2ff7 libunwind.dylib (35.3) <78DCC358-2FC1-302E-B395-0155B47CB547> /usr/lib/system/libunwind.dylib
0x7fff8aca3000 - 0x7fff8aca5ff7 libquarantine.dylib (71) <7A1A2BCB-C03D-3A25-BFA4-3E569B2D2C38> /usr/lib/system/libquarantine.dylib
0x7fff8ad24000 - 0x7fff8ad28ff7 libcache.dylib (62) /usr/lib/system/libcache.dylib
0x7fff8ad29000 - 0x7fff8ad2efff libmacho.dylib (845) <1D2910DF-C036-3A82-A3FD-44FF73B5FF9B> /usr/lib/system/libmacho.dylib
0x7fff8b075000 - 0x7fff8b07ffff libcommonCrypto.dylib (60049) <8C4F0CA0-389C-3EDC-B155-E62DD2187E1D> /usr/lib/system/libcommonCrypto.dylib
0x7fff8b571000 - 0x7fff8b952ffe libLAPACK.dylib (1094.5) <7E7A9B8D-1638-3914-BAE0-663B69865986> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
0x7fff8b953000 - 0x7fff8b95cff3 libsystem_notify.dylib (121) <52571EC3-6894-37E4-946E-064B021ED44E> /usr/lib/system/libsystem_notify.dylib
0x7fff8d1c9000 - 0x7fff8d1d0fff libcompiler_rt.dylib (35) <4CD916B2-1B17-362A-B403-EF24A1DAC141> /usr/lib/system/libcompiler_rt.dylib
0x7fff8d825000 - 0x7fff8d826ff7 libSystem.B.dylib (1197.1.1) /usr/lib/libSystem.B.dylib
0x7fff8dc5d000 - 0x7fff8dc5dff7 libkeymgr.dylib (28) <3AA8D85D-CF00-3BD3-A5A0-E28E1A32A6D8> /usr/lib/system/libkeymgr.dylib
0x7fff8dc9f000 - 0x7fff8de0dff7 libBLAS.dylib (1094.5) /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
0x7fff8e035000 - 0x7fff8e046ff7 libsystem_asl.dylib (217.1.4) <655FB343-52CF-3E2F-B14D-BEBF5AAEF94D> /usr/lib/system/libsystem_asl.dylib
0x7fff8e98b000 - 0x7fff8e98cff7 libsystem_blocks.dylib (63) /usr/lib/system/libsystem_blocks.dylib
0x7fff8f57d000 - 0x7fff8f597fff libdispatch.dylib (339.1.9) <46878A5B-4248-3057-962C-6D4A235EEF31> /usr/lib/system/libdispatch.dylib
0x7fff8fecb000 - 0x7fff8fee7ff7 libsystem_kernel.dylib (2422.1.72) /usr/lib/system/libsystem_kernel.dylib
0x7fff8feec000 - 0x7fff8fef4fff libsystem_dnssd.dylib (522.1.11) <270DCF6C-502D-389A-AA9F-DE4624A36FF7> /usr/lib/system/libsystem_dnssd.dylib
0x7fff90141000 - 0x7fff90152ff7 libz.1.dylib (53) <42E0C8C6-CA38-3CA4-8619-D24ED5DD492E> /usr/lib/libz.1.dylib
0x7fff902e8000 - 0x7fff902eeff7 libsystem_platform.dylib (24.1.4) <331BA4A5-55CE-3B95-99EB-44E0C89D7FB8> /usr/lib/system/libsystem_platform.dylib
0x7fff90342000 - 0x7fff903cbff7 libsystem_c.dylib (997.1.1) <61833FAA-7281-3FF9-937F-686B6F20427C> /usr/lib/system/libsystem_c.dylib
0x7fff903cc000 - 0x7fff9040eff7 libauto.dylib (185.5) /usr/lib/libauto.dylib
0x7fff90454000 - 0x7fff90601f27 libobjc.A.dylib (551.1) /usr/lib/libobjc.A.dylib
0x7fff9096e000 - 0x7fff90972fff libsystem_stats.dylib (93.1.26) /usr/lib/system/libsystem_stats.dylib
0x7fff91923000 - 0x7fff91924ff7 libDiagnosticMessagesClient.dylib (100) <4CDB0F7B-C0AF-3424-BC39-495696F0DB1E> /usr/lib/libDiagnosticMessagesClient.dylib
0x7fff91968000 - 0x7fff91a33fff libvDSP.dylib (423.32) <3BF732BE-DDE0-38EB-8C54-E4E3C64F77A7> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvDSP.dylib
0x7fff91bbb000 - 0x7fff91be4ff7 libc++abi.dylib (48) <8C16158F-CBF8-3BD7-BEF4-022704B2A326> /usr/lib/libc++abi.dylib
0x7fff91cf2000 - 0x7fff91d0dff7 libsystem_malloc.dylib (23.1.10) /usr/lib/system/libsystem_malloc.dylib
0x7fff91e31000 - 0x7fff91e58ffb libsystem_info.dylib (449.1.3) <7D41A156-D285-3849-A2C3-C04ADE797D98> /usr/lib/system/libsystem_info.dylib
0x7fff92205000 - 0x7fff924d9fc7 com.apple.vImage (7.0 - 7.0) /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/vImage
0x7fff92655000 - 0x7fff9283aff7 com.apple.CoreFoundation (6.9 - 855.11) /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
0x7fff93856000 - 0x7fff93a0eff3 libicucore.A.dylib (511.25) <3ED7B656-416E-3071-AEC8-E85C90232F78> /usr/lib/libicucore.A.dylib
0x7fff949db000 - 0x7fff949dcffb libremovefile.dylib (33) <3543F917-928E-3DB2-A2F4-7AB73B4970EF> /usr/lib/system/libremovefile.dylib
0x7fff94b17000 - 0x7fff94b69fff libc++.1.dylib (120) <4F68DFC5-2077-39A8-A449-CAC5FDEE7BDE> /usr/lib/libc++.1.dylib
0x7fff94eb6000 - 0x7fff94edafff libxpc.dylib (300.1.17) <4554927A-9467-365C-91F1-5A116989DD7F> /usr/lib/system/libxpc.dylib

External Modification Summary:
Calls made by other processes targeting this process:
task_for_pid: 7
thread_create: 0
thread_set_state: 0
Calls made by this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by all processes on this machine:
task_for_pid: 142272
thread_create: 0
thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=92.9M resident=18.0M(19%) swapped_out_or_unallocated=74.8M(81%)
Writable regions: Total=2.2G written=2.0G(92%) resident=1.9G(89%) swapped_out=69.6M(3%) unallocated=253.0M(11%)

REGION TYPE VIRTUAL
=========== =======
Kernel Alloc Once 4K
MALLOC 2.2G
MALLOC (admin) 16K
MALLOC_LARGE (reserved) 21.3M reserved VM address space (unallocated)
STACK GUARD 56.0M
Stack 8192K
VM_ALLOCATE 16K
__DATA 2256K
__LINKEDIT 67.9M
__TEXT 25.0M
__UNICODE 544K
shared memory 4K
=========== =======
TOTAL 2.3G
TOTAL, minus reserved VM space 2.3G

Model: MacBookAir6,2, BootROM MBA61.0099.B04, 2 processors, Intel Core i7, 1.7 GHz, 8 GB, SMC 2.13f7
Graphics: Intel HD Graphics 5000, Intel HD Graphics 5000, Built-In, 1024 MB
Memory Module: BANK 0/DIMM0, 4 GB, DDR3, 1600 MHz, 0x02FE, 0x000000000000000000000000000000000000
Memory Module: BANK 1/DIMM0, 4 GB, DDR3, 1600 MHz, 0x02FE, 0x000000000000000000000000000000000000
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0x117), Broadcom BCM43xx 1.0 (6.30.223.154.45)
Bluetooth: Version 4.2.0f6 12982, 3 services, 15 devices, 1 incoming serial ports
Network Service: Wi-Fi, AirPort, en0
Serial ATA Device: APPLE SSD SM0256F, 251 GB
USB Device: Internal Memory Card Reader
USB Device: BRCM20702 Hub
USB Device: Bluetooth USB Host Controller
Thunderbolt Bus: MacBook Air, Apple Inc., 23.6

[wiredfool]

What's the python code that triggers this?

[jamesra]

This file:
https://github.com/jamesra/random/blob/master/pillow_png_size.py

On Dec 12, 2013, at 10:27 PM, wiredfool <notifications@github.commailto:notifications@github.com> wrote:

What's the python code that triggers this?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/436#issuecomment-30489572.

[wiredfool]

Ok, so I get an error on ubuntu as well in the same spot:

(vpy27-dbg)erics@phantom-ssd:~/src/png$ gdb python
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /home/erics/vpy27-dbg/bin/python...done.
(gdb) r pillow_png_size.py
Starting program: /home/erics/vpy27-dbg/bin/python pillow_png_size.py
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Saving this big png works for me
All done!
Saving a slightly larger png does not work for me

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff69f2789 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) frame
#0  0x00007ffff69f2789 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) up
#1  0x00007ffff42b57c9 in copy1 (out=0x14cb461 "", 
    in=0x7ffe8ca50e10 <Address 0x7ffe8ca50e10 out of bounds>, pixels=48000)
    at libImaging/Pack.c:413
413     memcpy(out, in, pixels);

(gdb) up
#2  0x00007ffff42c814f in ImagingZipEncode (im=0xb0dc40, state=0x12fd4c8, 
    buf=0x14d7034 'U' <repeats 200 times>..., bytes=192000) at libImaging/ZipEncode.c:158
158         state->shuffle(state->buffer+1,
(gdb) l
153             break;
154 
155         }
156 
157         /* Stuff image data into the compressor */
158         state->shuffle(state->buffer+1,
159                    (UINT8*) im->image[state->y + state->yoff] +
160                    state->xoff * im->pixelsize,
161                    state->xsize);
162 
(gdb) p state->y
$3 = 44740
(gdb) p state->yoff
$4 = 0
(gdb) p state->xoff
$5 = 0
(gdb) p state->xsize
$6 = 48000

(gdb) p im->image[0][0]
$16 = 0 '\000'
(gdb) p im->image[1][0]
$17 = 0 '\000'
(gdb) p im->image[2][0]
$18 = 0 '\000'
(gdb) p im->image[3][0]
$19 = 0 '\000'
(gdb) p im->image[state-y-1][0]
No symbol "y" in current context.
(gdb) p im->image[state->y-1][0]
$20 = 0 '\000'
(gdb) p im->image[state->y][0]
Cannot access memory at address 0x7ffe8ca50e10
(gdb) p im->image8[0]
$21 = (unsigned char *) 0x7fff0ca48010 ""
(gdb) p im->image8[0][0]
$22 = 0 '\000'
(gdb) p im->image8[0][state->y*state->xsize]
Cannot access memory at address 0x7ffe8ca50e10
(gdb) p im->image8[0][(Py_ssize_t)state->y*state->xsize]
$23 = 0 '\000'

Ok, that last one is a smoking gun. Pretty sure that the error is in storage.c, where all the pointers to the rows are setup. Or not. I'll pick it up in the morning.

[wiredfool]

s/morning/evening/;

It was in map.c, not storage.c. Give this a whirl and see what you get.

[jamesra]

Looks like the fixed worked. Thanks!

The test ran successfully. The 48,000 x 48,000 pixel image was able to open with Mac Preview app.