Unsupported JPEG stops the Gallery App from showing pictures

[dtrumpfheller]

Bug report

Unsupported JPEG stops the Gallery App in the grid view from showing pictures after the unsupported one.

Steps to reproduce

1. Add multiple JPEG pictures
2. Add unsupported JPEG picture. In my case the unsupported picture is a panorama took with a Samsung Galaxy S7
3. Add multiple JPEG pictures

Expected behaviour

Show all pictures. If some pictures are not supported then just skip them.

Actual behaviour

Only pictures till the unsupported one are shown. The spinner which indicates that the unsupported JPEG is being processes stays forever.

Server configuration

Operating system:
Debian Jessie

Web server:
nginx

Database:
MariaDB

PHP version:
5.7 (from Debian Stretch)

cloud server: Nextcloud or ownCloud
Nextcloud

cloud version: (see admin page or version.php)
11.0.1

Updated from an older installation or fresh install:
Updated

List of activated apps:
Gallery App, ...

The content of config/config.php:

Are you using external storage, if yes which one: local folder, smb share, sftp, etc.
no

Are you using encryption: yes/no
no

Are you using custom gallery.cnf config files: yes/no
no

Web server error log

none

Cloud log

imagecreatefromjpeg(): gd-jpeg: JPEG library reports unrecoverable error: Unsupported JPEG process: SOF type 0xc8 at /var/www/nextcloud/lib/private/legacy/image.php#538
imagecreatefromjpeg(): gd-jpeg: JPEG library reports unrecoverable error: Unsupported marker type 0xf3 at /var/www/nextcloud/lib/private/legacy/image.php#538

Client configuration

Browser:
Firefox

Operating system:
Windows 7

Browser log

None

[oparoz]

This is an issue with the code in server, but I don't think anybody can do anything about it.
I believe it has been fixed in the GD library itself, so keep updating your server and close the issue once this has been resolved for you.

[dtrumpfheller]

Hello,

there are two problems.
Problem 1)
The generation of the preview fails. It sounds like it is done in a different lib, so it can't be solved here. So fine so good.

Problem 2)
An exception during preview generation leads to a not-working gallery app. In my eyes this is the result of a missing or wrong exception handling within the gallery app. The app should catch the exception and proceed with the next image.

Greetings

[oparoz]

The app should catch the exception and proceed with the next image.

That's what we do, but when PHP generates a fatal error, everything stops.
I haven't found a solution for that problem.

[linuxrrze]

I did some research on this topic, and I found, that the wrong error handling seems to occur on a lower level:

When trying to load / convert images that cause the above error with other tools like "gimp" or "jpegtran" I get very similar errors.

So I guess the error is located somewhere deep in a basic (jpeg?) library ...

In my case the error is often caused by panorama photos taken by my smartphone.

My environment:
Running Ubuntu 16.04 (x86_64), Nextcloud 11.0.2

[godfuture]

This issue happens for me as well.

I try to generate all thumbnails
sudo -u www-data php /var/www/nextcloud/occ preview:generate-all

and the job always fails at the same folder (or lets say at the same photos) being processed.

gd-jpeg: JPEG library reports unrecoverable error: Unsupported marker type 0x8c in /var/www/nextcloud/lib/private/legacy/image.php on line 538

Are there any news?

[linuxrrze]

Hi I just did a little more research on this topic, and to me it looks like
the following commit to the libjpeg-turbo library from Jun 28, 2017 could
point to the root of this problem:

libjpeg-turbo/libjpeg-turbo@b0971e4

Unfortunately it also mentions that the problem could be hard to fix without API changes.

Note:
I'm using Ubuntu 16.04 and the php7.0-gd package is using libjpeg-turbo8 as JPEG library:

# dpkg -L php7.0-gd
...
/usr/lib/php/20151012
/usr/lib/php/20151012/gd.so
...

# ldd  /usr/lib/php/20151012/gd.so | grep jpeg
	libjpeg.so.8 => /usr/lib/x86_64-linux-gnu/libjpeg.so.8 (0x00007f59d43ce000)

[linuxrrze]

It was mentioned above that this bug was fixed in the gd library, however
I could only find 2 issues and a proposed fix (not included in latest stable 2.2.4):

libgd/libgd#380
libgd/libgd#338

Fix (proposed):

libgd/libgd#346

I just build 2.2.4 Ubuntu packages with this fix included. Currently running

./occ preview:pre-generate

I'll report here about the results once it's finished.

[linuxrrze]

The above fix to libgd does not solve this problem:

Replacing libgd with version 2.2.4 (and fix) on Ubuntu 16.04 still fails on certain images (Galaxy S7 panorama).

If there's some other patch to try, please let me/us know.

[linuxrrze]

For panorama pictures created using a Galaxy S7 the file format can be fixed (in many cases) using this script:

https://gist.github.com/bcyrill/e59fda6c7ffe23c7c4b08a990804b269

Make sure to read the comments and apply the changes mentioned there (otherwise this will not work for new panorama fotos as the format changed during one of the recent android updates)

And make sure to have a backup of your files before trying!

[pren0x]

For panorama pictures created using a Galaxy S7 the file format can be fixed (in many cases) using this script:
https://gist.github.com/bcyrill/e59fda6c7ffe23c7c4b08a990804b269
Make sure to read the comments and apply the changes mentioned there (otherwise this will not work for
new panorama fotos as the format changed during one of the recent android updates)
And make sure to have a backup of your files before trying!

Thanks! It works for me. It's not perfect, but I can use gallery app.

[godfuture]

In this thread #1732 they talk about other libraries like ImageMagick. As it is insecure, GraphicsMagick was proposed. Maybe GraphicsMagick has less of these issues?

Can someone try to use GraphicsMagick to generate a preview of the same problematic image? Or is this not really easy to test?

[srkunze]

Same problem here. What can we do here?

I incur the problem when issuing nextcloud.occ preview:generate-all: 'Unsupported marker type 0xfb'
Unfortunately, it doesn't tell what image seems to be corrupted.

[srkunze]

Would it help to wrap the conversion into its own subprocess/thread?

[BeehiveSystems]

I am having this issue as well. It seems that it is indeed the Galaxy S7 Edge panorama photos causing the issue.

[godfuture]

I would like to create all previews with cmdline but can't, because my Samsung S4 Mini photos make the process crash:
sudo -u www-data php /var/www/nextcloud/occ preview:generate-all

What is the status on this? The gallery app is for no use, if I have to wait for the previews all the time...

[j-ed]

The workaround given by @linuxrrze still works but it might be necessary to slightly modify the script so that it also supports newer versions of the panorama images as described here, This can be done by using the version string "105" instead of "103":
https://gist.github.com/bcyrill/e59fda6c7ffe23c7c4b08a990804b269#gistcomment-1860291