Downloads fail when Transfer-Encoding = chunked

[josh4trunks]

Actual behaviour

• Download failed - Download of test.txt could not be completed

Expected behaviour

• File should be downloaded successfully

Steps to reproduce

1. Setup server environment where files will be served with Transfer-Encoding = chunked
2. Try downloading file using android app

Environment data

Nextcloud app version:
1.1.0

This bug is also present is the ownCloud android app. owncloud/android#1128
The fix seems to be discussed breifly there, though I don't see any commits referenced.

[tobiasKaminsky]

Can you help me and and explain where and what to set in the server environment to reproduce this behaviour?

[josh4trunks]

In my case chunking happens because I have Varnish in front of my nginx webserver compressing text files with gzip. I don't know how to reproduce this directly but maybe people in the owncloud bug know. possibly by enabling gzip chunked is used?

[tobiasKaminsky]

I thought it is a supported server option...
With this I doubt that we can support this.

[josh4trunks]

I believe there is a way to have chunked encoding without gzip enabled. It is part of the HTTP/1.1 specification.
But we can just wait till this is fixed in the ownCloud android client and apply the same changes.

[johnwmail]

Can you help me and and explain where and what to set in the server environment to reproduce this behaviour?

Hi, OpenBSD's httpd is enable/force chunked encoding by default.

[johnwmail]

I still have this problem, how to solve it.
Any workaround?

[josh4trunks]

In my environment, where this problem was caused by some type of incompatibility with Varnish, I was able to get this working by setting beresp.do_stream = false whenever I enable gzip for content-type = text/application.

Maybe you can check what Varnish is doing wrong/weird with do_stream

[johnwmail]

So, still no fix on client side?

[mario]

Does this happens only on 3/4g or also on wifi?

[mario]

Also, can you inspect the headers and tell me what does it says for content-size?

[johnwmail]

Yes, both 3/4g and wifi also have this problem.

I don't know how to inspect the header, this is https, tcpdump not work, I will try to Google it later

Thanks

[johnwmail]

Hello, any idea how to fix it?

[tobiasKaminsky]

@josh4trunks can you send us also credentials for a test account to android at nextcloud dot com?
Then I can have a look, but as oC also does not have a solution, I am unsure if we can do it...

[johnwmail]

@tobiasKaminsky I would also setup test account (with Transfer-Encoding = chunked) for this.
But I do not want to post login/password here, would you send email to me?

johnw.mail at gmail dot com

Thanks.

[josh4trunks]

@tobiasKaminsky I already fixed this in my production environment. See #113 (comment)

Did you still want an account on my working Nextcloud install?

[tobiasKaminsky]

Ahh, I overread that you have solved it and that is (more or less) a server/environment side problem.
So I will close this.

[mario]

Please re-open. This has to be solved client side as well.

…

On Tue, 16 May 2017 at 08:51, Tobias Kaminsky ***@***.***> wrote: Ahh, I overread that you have solved it and that is (more or less) a server/environment side problem. So I will close this. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#113 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAWskILYtOYij5qArpmULXnoDiYM18xks5r6UeDgaJpZM4JIfc7> .

[johnwmail]

@mario agree, this should be solved client side as well.
I tried talk to OpenBSD people,

http://marc.info/?l=openbsd-misc&m=149422158413997&w=2

They quoted RFC7230

"A recipient MUST be able to parse and decode the chunked transfer coding."

I would like to setup nextcloud AC with "chunked transfer coding" enabled.

Email me, if people who want/need the AC to debug/testing

johnw dot mail at gmail dot com

Thanks.

[johnwmail]

https://tools.ietf.org/html/rfc7230#section-4.1

[tobiasKaminsky]

edit: to warm here...searching for httpclient "Chunked Transfer Encoding" gives enough :-)

https://stackoverflow.com/questions/6124233/how-to-handle-read-the-response-with-transfer-encodingchunked

[tdm4]

Having the same problem here. Using nginx instead of httpd for now (on OpenBSD) with chunked_transfer_encoding off;
Not ideal, but it works. Android client breaks RFC7230 unfortunately. :(

[johnwmail]

Hi @tdm4

I written a dirty hack to disable chunked_transfer_encoding for OpenBSD httpd,
Maybe you will interesting

http://marc.info/?l=openbsd-misc&m=149422389314402&w=2

[rpcameron]

Is there any activity on this, or are the Nextcloud devs going to wait on Owncloud to fix this since the download function/code seems to be in their Android library code?

Until this issue is resolved, I cannot use either client. Yes, I see the patch for OpenBSD httpd, but I'm not going to modify my system software to fix a bug that exists in other third-party software because the third-party software is broken and doesn't properly implement their HTTP client.

[mario]

Actually our library differs from theirs in quite a lot of things... But yea, hopefully soon...

…

On Thu, 24 Aug 2017 at 15:50, rpcameron ***@***.***> wrote: Is there any activity on this, or are the Nextcloud devs going to wait on Owncloud to fix this since the download function/code seems to be in their Android library code? Until this issue is resolved, I cannot use either client. Yes, I see the patch for OpenBSD httpd, but I'm not going to modify my system software to fix a bug that exists in other third-party software because the third-party software is broken and doesn't properly implement their HTTP client. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#113 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAWslBpYsARE3dUhKRYBoZx79BaHYHuks5sbX-mgaJpZM4JIfc7> .

[rpcameron]

@mario Ah, sorry. When I was trying to sort through the code I must have followed a wrong reference when tracking source files and includes. Does this mean that Nextcloud will soon finally support this, so I no longer have field support problems from users using Android devices? Or will they still need to be redirected to some other app to access their files on their phones ...

[xstasi]

Hi there, poking in to say that I too am afflicted by this issue.

+1

[tobiasKaminsky]

Can someone provide an test account on a server where this is happening?
Please send it to android at nextcloud dot com with a reference to this issue.

[rpcameron]

@tobiasKaminsky I just sent an email with login details for a test account on an OpenBSD 6.2 server running its httpd, which only serves with "transfer encoding = chunked". Maybe this means he Android client will finally support the HTTP spec ...

[tobiasKaminsky]

@rpcameron thank you. I do my very best :-)

[tobiasKaminsky]

@xstasi thank you also for the credentials 👍

[tobiasKaminsky]

I guess I have a fix for this: nextcloud/android-library#98

We will soon have our dev version back again. This builds a nearly nightly snapshot from master. So once the fix is merged, you can give it a try and report back :-)

[rpcameron]

Just ran a quick test with the latest Dev version (20171204), and it appears to be downloaded files (at long last). Thanks to the devs for finally addressing this issue ... Hopefully a stable release will soon make its way out.

[AndyScherzinger]

nextcloud/android-library#98 did fix it and has been integrated with master and with the new dev builds now been shipped for the dev version. The next stable version doesn't have a date yet but is also progressing quite well: https://github.com/nextcloud/android/milestones/Nextcloud%20App%202.1.0

[johnwmail]

Where can we download the dev app?

[AndyScherzinger]

@johnwmail it is distributed via f-droid: https://f-droid.org/packages/com.nextcloud.android.beta/ (so best is to install the f-droid client and Search for "Nextcloud Dev").

[tobiasKaminsky]

Please note that fdroid currently does not have the latest version, it being built this day and will be available soon.

Or take a look here: https://github.com/nextcloud/android/blob/master/CONTRIBUTING.md#dev-release

[levaidaniel]

Hi there!

Is it normal that version 3.5.1 (March, 18, 2019) still doesn't download anything from OpenBSD httpd(8)? Is this mentioned fix supposed to be in this version?

[johnwmail]

Hi, they fixed a year.

[tobiasKaminsky]

Is it normal that version 3.5.1 (March, 18, 2019) still doesn't download anything from OpenBSD httpd(8)? Is this mentioned fix supposed to be in this version?

Can you create us a test account, test if the problem occurs also there and if so send the credentials to tobias at nextcloud dot com with a reference to this issue?

[levaidaniel]

Sure thing, I sent the mail containing the info.

[levaidaniel]

In the meantime, here are the logs from the application and Chrome, respectively. Maybe these shed some light into the problem:

Android app
Chrome

[levaidaniel]

Interesting thing is using the same app on Android 7.1.1 yields more or less positive results over using the same v3.5.1 app on Android 9. The former only fails sometimes, usually after waiting for a while after a successful download -- it says the file couldn't be synced so using the downloaded version (or something along these lines).
But other than that, on older Android the success rate is definitely higher. On Android 9 I was never ever able to download anything (even though using the same application version: 3.5.1).

[tobiasKaminsky]

Sure thing, I sent the mail containing the info.

I logged into testaccount and download Nextcloud Flyer.pdf and the AEG manual.
Both work fine with emulator and android 9.0.

[levaidaniel]

Thanks for trying it out!

Is there anything in the logs I pasted here that could point to an error that might cause this?

[levaidaniel]

Or better yet, can I enable something in the Android app that would log what goes wrong when I try to download something?

[tobiasKaminsky]

The log you posted did not helped, unfortunately…

You can try to provide locgat infos: https://github.com/nextcloud/android/blob/master/README.md#getting-debug-info-via-logcat

[fmikker]

Experiencing the same issue.

Do you need a additional test account?

Platform: CentOS Linux release 7.6.1810 (Core
HTTP-server: httpd24-httpd-2.4.34-7.el7.x86_64
PHP: PHP 7.0.27

Edit: Can confirm that using Nextcloud-dev from f-droid mitigates the problem.

[stale]

This request did not receive an update in the last 4 weeks. Please take a look again and update the issue with new details, otherwise the issue will be automatically closed in 2 weeks. Thank you!