Large Object support for OpenStack Swift #18955
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Until now, you wouldn't be able to create objects larger that 5GB. It's somewhat related with pull #18883 Signed-off-by: Adrian Brzezinski <adrian.brzezinski@eo.pl>
icewind1991
approved these changes
Apr 27, 2020
|
ping @rullzer can this still get in 19? |
rullzer
approved these changes
May 6, 2020
rullzer
added
the
4. to release
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Until now, you wouldn't be able to create objects larger that 5GB.
It's worth noting that it doesn't improve efficiency. Here is how it work for default configuration:
Client divide file to chunks and upload each of those chunks as separate file
All chunks are pushed to object storage as they are received by Nextcloud
Client send MOVE request which assembly file and store it. In case of Swift (and probably other object storages too), it download all chunks, assembly it locally into one file, and then uploads again to Swift
You can improve efficiency by setting
cache_pathto for e.g./tmp/nextcloud_cache, then file chunks wont be uploaded to Swift, but you will still have to wait when the assembled file is being uploaded to Swift. For large file it can lead to timeout on the client side.Setting
cache_pathmay not be appropriate for multi-node installation, in that case you probably need to have sticky-session or place cache on shared storage.Edit: I see that issue has been already reported for e.g.: #6105, #7070