Adjust Testsuite to async uploads

[kobergj]

Async uploads are implemented on the shiny new experimental branch cs3org/reva#3029
The idea is that the server returns immediately after he has received all bytes, then handle postprocessing asynchronly.
With virusscanning around the corner and potentially more features that make use of async postprocessing mechanism it would be nice to have all that properly tested.

Unfortunately the testsuite in its current state cannot handle async uploads. There a mainly two reasons for it

425 status code while processing

During postprocessing of a file the server will send status code 425 on propfinds or get requests. This is because the file is not ready yet and can therefore not be downloaded until processing is finished. The server does send metadata information already with the 425 status code, so for most tests it might be sufficient to just ignore the status code of the response and continue checking file metadata. However tests that actually download a file and check its content need to wait until the processing is finished and the server returns a 200 status code again.

Last Updated time

Some tests check for the Modified time of a file. They do expect it to match a specific timestamp. However, if postprocessing is finished, the file will get a new timestamp. This is because the file is now processed and ready to download, so the etags need to be updated. These checks probably need to be omitted, or one could just check that the actual timestamp is after the original expected one.

These are the two main issues which (I hope) will allow most tests to run async. There might be more adjustments needed. But I can't say for sure because of too many failing tests.

[kobergj]

cc @micbar @individual-it

[phil-davis]

Thoughts:

• for downloads we can just embed an extra loop when running on oCIS or reva. If 425 is returned then the test code can wait 1 second and try again. And we put some limit on the loop so that it stops after "a while" (however long we think is the maximum time that the server should to take realistically to "process" an uploaded file)
• modified time - if the upload sends an mtime along with the upload, then the server should respect that. If no mtime is specified then the server is free to assign an mtime. The test code can be less fussy about the mtime. I thought that we already had some flexibility for this already, but I need to check. Because the test-runner and the system-under-test can run in different systems anyway, which might have their system time slightly different, and the mtime can anyway be 1-second later than what the test-runner "expects" because the time might be "12:30:51.98" when the test-runner starts an upload, and by the time the system-under-test receives the upload it is "12:30:52.01" - I will have a look for where we already put this flexibility.

[phil-davis]

@kobergj see core PR owncloud/core#40180 and test demonstration PR cs3org/reva#3034

You could make a PR in a branch off your experimental branch in reva, and adjust .drone.env like I did in cs3org/reva#3034 - and see if this helps the download-related steps to pass.

The core code changes in owncloud/core#40180 will allow the test suite to retry any request that gets an HTTP 425 status response. Maybe that could happen for things like upload-overwrite (if a test uploads a file then quickly uploads a next version, the original upload could be still in post-processing). For any request, it will be retried up to 10 times at 1-second intervals if it gets 425.

[kobergj]

Sorry, this ticket got completely lost for me somehow 😞

I have now created a PR that tests with async uploads. I can change the core commit id if needed: #5251

To activate async uploads for local testing one only needs to set the STORAGE_USERS_OCIS_ASYNC_UPLOADS envvar to true. When trying to test edge cases one can additionally set DELAY_POSTPROCESSING envvar to e.g. 10s to artificially delay postprocessing on the server

[individual-it]

ToDo:

• run all tests with enabled async upload and check what is failing and why. (it could be a bug or it could be that tests need to be adjusted to wait) ([full-ci][tests-only] Enable async uploads #5268)
• create new tests with DELAY_POSTPROCESSING to make sure the API really returns 425 while post-processing (@kobergj any headers that are worthwhile to test?) ([tests-only][full-ci] Add tests with explicit delay post-processing enabled #5328)

[kobergj]

No additional headers to test. Files in postprocessing should mainly behave the same as normal files. Except:

• The status code in the PROPFIND will be 425
• Trying to download the file will result in a 425

[ScharfViktor]

• The status code in the PROPFIND will be 425

Do I understand correctly that the code should change to 200 after some time? it does not change and I cannot see the uploaded file in the UI

• start ocis localy with env STORAGE_USERS_OCIS_ASYNC_UPLOADS=true
• upload file
  The uploading succeeds, but I don't have the file in the UI

- wait and reload page

Actual: no files (no matter what size). the response code remains 425

[kobergj]

Two issues here:

• web doesn't show the 425 file here because the async uploads ui is not yet merged
• You understand it correctly that file should become 200 after a while. You run plain ocis master? No additional envvars set?

[ScharfViktor]

• You understand it correctly that file should become 200 after a while. You run plain ocis master? No additional envvars set?

@kobergj I run ocis with OCIS_INSECURE and while some time code changed to 200

[ScharfViktor]

I created issue for this case #5256