Fix directory detection for s3

[kesselb]

Fix #6954
Fix #9808

Test:

1. Add some directories with files to s3 storage (without nextcloud)
2. Open external storage in nextcloud -> files are not there
3. Run occ files:scan -> files are there (after this patch)

Tried to add some test cases for this but i don't understand how the s3 testing works 😞

[wanchic]

I can confirm this fixed my S3 sync issue for NC 15.0.7.
I was not able to see files that I synced from one S3 bucket to another S3 bucket.

[MorrisJobke]

/backport to stable16

[MorrisJobke]

/backport to stable15

[arno01]

Just tested this PR against my Nextcloud 15.0.8 running in the docker.
Nextcloud can finally see the directories in S3 (Minio)!
Much thanks @kesselb ! 👍

Upd: Whilst this PR makes Nextcloud work with S3 (minio) directories, I think it still needs to be polished a bit (see my next comment).

[arno01]

For some reason, though, I am getting these tooltip messages when creating a subdirectory under any top-level directory, e.g. /minio/top-level-dir/sub-dir:

Could not create file ""
Could not create folder "sub-dir"

Despite of that tooltip message, the directory gets created fine.

Upd: and the size for all directories is "Pending", files are ok (files do get their size there).
Upd2: the Could not create message does not always appear.. sometimes it does, sometimes it does not.. looks like Nextcloud time outs faster than it checks (stat) the newly created directory is there?

[eigood]

My situation: local nextcloud+minio, all auto-configured to run isolated in a docker-compose. A short shell script configures everything at start-of-day. My test procedure is to do 'docker-compose down -v', then './first-run.sh', which brings everything back up.

I then create a folder in nextcloud, for the minio external storage, and I can see that the folder does exist in the /data directory in minio(from the shell). I can not create a file, via nextcloud, I get a tooltip error.

If I then create file by hand, using the shell, and initiate a occ files:scan, the folder disappears. I checked the filecache table in postgresql. If I delete the manually created file, do another scan, the folder comes back.

I then placed the modified file from this PR, into my nextcloud.

Now, the occ file:scan does see the folder, when a file has been manually created in /data in minio.

However, I still can't create files(in nextcloud, I select "New File", and then type "AnyName.txt". I get a tooltip error upon creation.

So, this PR still needs work to fix the full minio+nextcloud integration.

[kesselb]

Mind to create a new issue for your problem? Please include the error message and logs.

[rullzer]

@icewind1991 mind to have a look?

[rullzer]

Master is no 18 development. If this needs to go into 17 please follow the normal backport procedures after merging.

[kesselb]

/backport to stable17

[alexander-rieder]

when will this be backported to v16?

[backportbot-nextcloud]

backport to stable16 in #17271

[backportbot-nextcloud]

backport to stable15 in #17272

[backportbot-nextcloud]

backport to stable17 in #17273

[crypdick]

@kesselb I upgraded my snap to version: 17.0.0.9 and successfully ran sudo nextcloud.occ files:scan --all -v. However, I didn't see the script find the missing S3 folders. When I "create" a new folder inside the external store in Nextcloud, it finally finds them. I don't want to do that for hundreds of folders...

[rullzer]

That is because it is in 17.0.1 only