[Bug]: Exception using GCS as primary storage: Rewriting objects created via Multipart Upload is not implemented yet

[thanhpd56]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• Nextcloud Server is running on 64bit capable CPU, PHP and OS.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

Exception when upload large file to a shared folder (of other user) using GCS as primary storage.
This only happens with shared folder, but can upload successfully with my own folder.

Steps to reproduce

1. Setup next cloud and using GCS as a S3 primary storage
2. User A create a folder and share this folder to user B
3. User B login + upload a file with 200 MB size
4. Server return this errror

Error executing "CopyObject" on "https://honganh-production-nextcloud.storage.googleapis.com/urn%3Aoid%3A1664"; AWS HTTP error: Client error: PUT https://honganh-production-nextcloud.storage.googleapis.com/urn%3Aoid%3A1664 resulted in a 400 Bad Request response:

InvalidArgumentInvalid argument.

Re (truncated...)

InvalidArgument (client): Invalid argument. - InvalidArgumentInvalid argument.

Rewriting objects created via Multipart Upload is not implemented yet. As a workaround, you can use compose to overwrite the object (by specifying honganh-production-nextcloud/urn:oid:1663 as both the source and output of compose) prior to rewrite.

Expected behavior

Upload file succes

Installation method

Community Docker image

Nextcloud Server version

26.0.1

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.0

Web server

Other

Database engine version

None

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

None

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

No response

List of activated Apps

Default apps

Nextcloud Signing status

No response

Nextcloud Logs

No response

Additional info

I think it's relead to this consideration https://cloud.google.com/storage/docs/multipart-uploads#considerations.
An object uploaded using this method cannot be copied or rewritten, unless you compose the object prior to the copy or rewrite.

[szaimen]

This sound slike a feature request to me

[joshtrichards]

Hi @thanhpd56:

I can't reproduce this issue on AWS S3. I'd like to take a closer look at at it on GCS's S3 compatibility mode.

Can you provide your NC configuration as requested in the submission form? (without your passwords/keys obviously)

Also, can you provide the entire stack trace associated with the log entry rather than just a snippet? Easiest way to grab this would be to go to Admin->Logging and move your cursor over to the right of the entry in question then selection Copy raw.

When including a stack trace/configs/etc, make sure to indicate in the GitHub editor it is:

preformatted / code

...so that it doesn't get reformatted or we'll be unable to review it. Thanks.

[joshtrichards]

A brief follow-up. I believe you were on the right track about the culprit:

https://cloud.google.com/storage/docs/release-notes#October_12_2021
https://cloud.google.com/storage/docs/multipart-uploads#considerations

Specifically, even though GCS has an S3 "compatibility" layer, it's not actually entirely up to spec with AWS S3 (the compose stuff is GCS specific and, unfortunately, kind of important).

In some situations, even when that error you're seeing appears, operations still work (because of some fallback behavior that helps in some cases in NC I think). But in other cases it's a true hard error. For example, I can trigger the error doing a copy while utilizing GCS, but the file does ultimately copy. But I can also trigger the error doing a move and the file never moves. (For the record, this testing was with External Storage not Primary Storage so there are some differences).

NC utilizes the official aws-sdk-php for S3 access so I'm not really sure we're going to be able to do much about this directly.

Google has evolved their S3 support over time. As a customer it might be worth lodging a support case with them to see when they intend to fix this issue...

Other S3 compatible providers like Backblaze B2 and off-the-shelf solutions like MinIO do not have this problem.

[joshtrichards]

I believe this is the tracker on the Google side for this: https://issuetracker.google.com/issues/229020040

Go vote that you're impacted by it.

Bottom line: This is not an NC issue, but a GCS S3 compatibility issue.

[joshtrichards]

This looks finally fixed on Google's end:

https://cloud.google.com/storage/docs/release-notes#June_23_2023

Objects created using XML API multipart uploads can now be copied and rewritten normally. Previously, you had to perform an object composition on such objects before the output could be copied or rewritten.

[thanhpd56]

This looks finally fixed on Google's end:

https://cloud.google.com/storage/docs/release-notes#June_23_2023

Objects created using XML API multipart uploads can now be copied and rewritten normally. Previously, you had to perform an object composition on such objects before the output could be copied or rewritten.

Thanks for your info. Infact I switched to S3 as main storage, and using Terraform for multi cloud management. However, multiple billings still is the problem. :v