Warning on preview generation with object storage as primary storage

[schiessle]

On master (and stable12 once this PR is merged #7186) I get a error during preview generation every time a new file gets uploaded. Because I can't reproduce it with fakes3 I recommend to use try it with Amazon S3.

My Setup:

1. Have a empty mariadb as database
2. make sure that the s3 bucket doesn't exists yet at Amazon S3
3. pre-configure the config.php so that we use the object storage right from the beginning:

<?php
$CONFIG = array (
  'installed' => false,
  'cache_path' => '/home/schiesbn/repos/nextcloud/server/tmp',
  'objectstore' => 
  array (
    'class' => 'OC\\Files\\ObjectStore\\S3',
    'arguments' => 
    array (
      'bucket' => 'nextcloud-schiesbn2',
      'autocreate' => false,
      'key' => 'my-s3-key',
      'secret' => 'my-s3-secret',
      'use_path_style' => true,
    ),
  ),
);

4. Then I complete the setup in the web interface.

5. Once the setup is completed and the file list is loaded for the first time I get this error messages, for each skeleton file (I used the skeleton files from here: https://github.com/nextcloud/example-files)

{"reqId":"fhpm3fMo5ZC1U8UPiXQw","level":3,"time":"2017-11-09T15:52:47+00:00","remoteAddr":"::1","user":"admin","app":"PHP","method":"POST","url":"\/master\/index.php","message":"fopen(https:\/\/s3.eu-west-1.amazonaws.com\/nextcloud-schiesbn2\/urn%3Aoid%3A5): failed to open stream: HTTP request failed! HTTP\/1.1 404 Not Found\r\n at \/home\/schiesbn\/repos\/nextcloud\/server\/master\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php#68","userAgent":"Mozilla\/5.0 (X11; Linux x86_64; rv:57.0) Gecko\/20100101 Firefox\/57.0","version":"13.0.0.5"}
{"reqId":"fhpm3fMo5ZC1U8UPiXQw","level":3,"time":"2017-11-09T15:53:55+00:00","remoteAddr":"::1","user":"admin","app":"PHP","method":"POST","url":"\/master\/index.php","message":"fopen(https:\/\/s3.eu-west-1.amazonaws.com\/nextcloud-schiesbn2\/urn%3Aoid%3A11): failed to open stream: HTTP request failed! HTTP\/1.1 404 Not Found\r\n at \/home\/schiesbn\/repos\/nextcloud\/server\/master\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php#68","userAgent":"Mozilla\/5.0 (X11; Linux x86_64; rv:57.0) Gecko\/20100101 Firefox\/57.0","version":"13.0.0.5"}
{"reqId":"fhpm3fMo5ZC1U8UPiXQw","level":3,"time":"2017-11-09T15:53:58+00:00","remoteAddr":"::1","user":"admin","app":"PHP","method":"POST","url":"\/master\/index.php","message":"fopen(https:\/\/s3.eu-west-1.amazonaws.com\/nextcloud-schiesbn2\/urn%3Aoid%3A13): failed to open stream: HTTP request failed! HTTP\/1.1 404 Not Found\r\n at \/home\/schiesbn\/repos\/nextcloud\/server\/master\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php#68","userAgent":"Mozilla\/5.0 (X11; Linux x86_64; rv:57.0) Gecko\/20100101 Firefox\/57.0","version":"13.0.0.5"}
{"reqId":"fhpm3fMo5ZC1U8UPiXQw","level":3,"time":"2017-11-09T15:54:00+00:00","remoteAddr":"::1","user":"admin","app":"PHP","method":"POST","url":"\/master\/index.php","message":"fopen(https:\/\/s3.eu-west-1.amazonaws.com\/nextcloud-schiesbn2\/urn%3Aoid%3A15): failed to open stream: HTTP request failed! HTTP\/1.1 404 Not Found\r\n at \/home\/schiesbn\/repos\/nextcloud\/server\/master\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php#68","userAgent":"Mozilla\/5.0 (X11; Linux x86_64; rv:57.0) Gecko\/20100101 Firefox\/57.0","version":"13.0.0.5"}
{"reqId":"fhpm3fMo5ZC1U8UPiXQw","level":3,"time":"2017-11-09T15:54:02+00:00","remoteAddr":"::1","user":"admin","app":"PHP","method":"POST","url":"\/master\/index.php","message":"fopen(https:\/\/s3.eu-west-1.amazonaws.com\/nextcloud-schiesbn2\/urn%3Aoid%3A18): failed to open stream: HTTP request failed! HTTP\/1.1 404 Not Found\r\n at \/home\/schiesbn\/repos\/nextcloud\/server\/master\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php#68","userAgent":"Mozilla\/5.0 (X11; Linux x86_64; rv:57.0) Gecko\/20100101 Firefox\/57.0","version":"13.0.0.5"}
{"reqId":"fhpm3fMo5ZC1U8UPiXQw","level":3,"time":"2017-11-09T15:54:04+00:00","remoteAddr":"::1","user":"admin","app":"PHP","method":"POST","url":"\/master\/index.php","message":"fopen(https:\/\/s3.eu-west-1.amazonaws.com\/nextcloud-schiesbn2\/urn%3Aoid%3A21): failed to open stream: HTTP request failed! HTTP\/1.1 404 Not Found\r\n at \/home\/schiesbn\/repos\/nextcloud\/server\/master\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php#68","userAgent":"Mozilla\/5.0 (X11; Linux x86_64; rv:57.0) Gecko\/20100101 Firefox\/57.0","version":"13.0.0.5"}
{"reqId":"fhpm3fMo5ZC1U8UPiXQw","level":3,"time":"2017-11-09T15:54:06+00:00","remoteAddr":"::1","user":"admin","app":"PHP","method":"POST","url":"\/master\/index.php","message":"fopen(https:\/\/s3.eu-west-1.amazonaws.com\/nextcloud-schiesbn2\/urn%3Aoid%3A23): failed to open stream: HTTP request failed! HTTP\/1.1 404 Not Found\r\n at \/home\/schiesbn\/repos\/nextcloud\/server\/master\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php#68","userAgent":"Mozilla\/5.0 (X11; Linux x86_64; rv:57.0) Gecko\/20100101 Firefox\/57.0","version":"13.0.0.5"}
{"reqId":"PcD03Sw9N7f8sDHfRPWN","level":3,"time":"2017-11-09T15:54:10+00:00","remoteAddr":"::1","user":"admin","app":"PHP","method":"GET","url":"\/master\/index.php\/apps\/files\/","message":"fopen(https:\/\/s3.eu-west-1.amazonaws.com\/nextcloud-schiesbn2\/urn%3Aoid%3A29): failed to open stream: HTTP request failed! HTTP\/1.1 404 Not Found\r\n at \/home\/schiesbn\/repos\/nextcloud\/server\/master\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php#68","userAgent":"Mozilla\/5.0 (X11; Linux x86_64; rv:57.0) Gecko\/20100101 Firefox\/57.0","version":"13.0.0.5"}
...

6. Every time I upload a new file for which a preview can be generated, let's say a *.jpg, I get again this warning:

{"reqId":"Brpu81L1TzBlXBCEfZR5","level":3,"time":"2017-11-17T10:31:08+00:00","remoteAddr":"::1","user":"admin","app":"PHP","method":"GET","url":"\/master\/index.php\/core\/preview.png?file=%2FIMG_0053.JPG&c=5a0eba67b6c92&x=32&y=32&forceIcon=0","message":"fopen(https:\/\/s3.eu-west-1.amazonaws.com\/nextcloud-schiesbn2\/urn%3Aoid%3A111): failed to open stream: HTTP request failed! HTTP\/1.1 404 Not Found\r\n at \/home\/schiesbn\/repos\/nextcloud\/server\/master\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php#66","userAgent":"Mozilla\/5.0 (X11; Linux x86_64; rv:57.0) Gecko\/20100101 Firefox\/57.0","version":"12.0.3.3"}
{"reqId":"Brpu81L1TzBlXBCEfZR5","level":3,"time":"2017-11-17T10:31:10+00:00","remoteAddr":"::1","user":"admin","app":"PHP","method":"GET","url":"\/master\/index.php\/core\/preview.png?file=%2FIMG_0053.JPG&c=5a0eba67b6c92&x=32&y=32&forceIcon=0","message":"fopen(https:\/\/s3.eu-west-1.amazonaws.com\/nextcloud-schiesbn2\/urn%3Aoid%3A112): failed to open stream: HTTP request failed! HTTP\/1.1 404 Not Found\r\n at \/home\/schiesbn\/repos\/nextcloud\/server\/master\/lib\/private\/Files\/ObjectStore\/S3ObjectTrait.php#66","userAgent":"Mozilla\/5.0 (X11; Linux x86_64; rv:57.0) Gecko\/20100101 Firefox\/57.0","version":"12.0.3.3"}

During debugging I ended up at this point: https://github.com/nextcloud/server/blob/master/lib/private/Files/ObjectStore/ObjectStoreStorage.php#L301. As you can see, we check if the file exists before we call fopen(). But still, fopen will fail with the warnings shown above and return false.

Another important point, this also happens the first time I navigate to a folder with pictures which already exists for some time, like the one from the example files. file_exists and fopen is not performed on the actual file but on the preview file, e.g. appdata_oc4uy84fig1f/preview/5/1100-734-max.png which of course doesn't exists the first time we create a preview. So the question is, why does file_exists return true at this point?

[schiessle]

cc @icewind1991 here the issue, that we can investigate further.

[schiessle]

I think I finally found it... Sometimes it is really useful to write down all the details again 😃

PR is on the way...