Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use unique combination of hostname/bucket/key for external storages #29484

Merged
merged 2 commits into from
Nov 10, 2021

Conversation

juliusknorr
Copy link
Member

Avoid conflicts with the storage id of external s3 storages when the bucket name is the same for different mountpoints.

@juliusknorr juliusknorr force-pushed the bugfix/noid/s3-external-conflict branch from a992104 to 13c7b05 Compare October 29, 2021 11:49
Copy link
Member

@icewind1991 icewind1991 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately this will need some kind of migration script.

Changing the storage id of an existing external storage will cause all fileids for the storage to change with all the effects that has

@juliusknorr
Copy link
Member Author

juliusknorr commented Oct 29, 2021

Right, so my attempt for that would be to just iterate over all configured external mounts and if we match the old pattern for an entry in oc_storages update that with the new format. Doing it during storage setup sounds a bit too risky to me and we can actually do this during upgrade since s3 only has key/secret auth and does not require any user session.

Signed-off-by: Julius Härtl <jus@bitgrid.net>
@juliusknorr
Copy link
Member Author

Added a migration step to rename the storage ids for all external storages using the "amazons3" backend, please have another look

@AndyScherzinger
Copy link
Member

@icewind1991 can you re-review with latest changes from @juliushaertl so we can provide it to the customer :) Thanks ❤️

Copy link
Member

@CarlSchwan CarlSchwan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After enabling the app and applying this patch I get

An exception occurred while executing a query: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'nextcloud.oc_external_mounts' doesn't exist

@CarlSchwan
Copy link
Member

After enabling the app and applying this patch I get

An exception occurred while executing a query: SQLSTATE[42S02]: Base table or view not found: 1146 Table 'nextcloud.oc_external_mounts' doesn't exist

Found a fix, if apps/files_external/lib/Migration/Version1011Date20200630192246.php::changeSchema return null. The schema will be written immediately to the database, if it returns $schema, it will only write it in the next MigrationStep that returns null (or the last one).

@CarlSchwan CarlSchwan force-pushed the bugfix/noid/s3-external-conflict branch 3 times, most recently from f2fa7ad to ad08803 Compare November 8, 2021 20:50
Copy link
Member

@CarlSchwan CarlSchwan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested under this two scenarios and the migration worked now:

  1. Setup Nextcloud with master branch => setup external storage with s3 => switch to this branch => migrations work
  2. Setup Nextcloud with this branch, activating the files_external app works now and I can add an external storage

Signed-off-by: Julius Härtl <jus@bitgrid.net>
@CarlSchwan CarlSchwan force-pushed the bugfix/noid/s3-external-conflict branch from ad08803 to b7d0075 Compare November 9, 2021 13:34
Copy link
Member

@PVince81 PVince81 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 please see comment about some thing to test

@juliusknorr
Copy link
Member Author

Also gave that another test run 👍

Thanks for helping out here @CarlSchwan

Copy link
Member

@icewind1991 icewind1991 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did some basic testing for the migration

@CarlSchwan CarlSchwan merged commit 344a889 into master Nov 10, 2021
@CarlSchwan CarlSchwan deleted the bugfix/noid/s3-external-conflict branch November 10, 2021 14:56
@CarlSchwan
Copy link
Member

/backport to stable22

@CarlSchwan
Copy link
Member

/backport to stable21

@CarlSchwan
Copy link
Member

/backport to stable20

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3. to review Waiting for reviews bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants