Minio as s3 external storage folder (directory) creation is not working properly or at all #17987

Cinerar · 2019-11-18T15:15:31Z

Steps to reproduce

Spin up local instanse of nextcloud (17.0.1 in my case)
Spin up local instance of minio or instance on another pc in network
docker run -p 9000:9000 --name minio1 -e "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE" -e "MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" minio/minio server /data
Connect minio as s3 external storage
Try to create folder on external storage (tooltip that folder is not created pops up)

Expected behaviour

No tooltip and folder created.

Actual behaviour

Somehow when i do this on same machine where nextcloud installed tooltip pops up, but folder is actually created.
IF i spin up minio on another machine in network tooltip exists and folder is not created.

Server configuration

Operating system:
alpine inside docker on windows 10

Web server:
official docker-compose mariadb-cron-redis

Nextcloud version: (see Nextcloud admin page)
17.0.1

Updated from an older Nextcloud/ownCloud or fresh install:
fresh
Where did you install Nextcloud from:
docker hub
Signing status:

Signing status

``` No errors have been found. ```

List of activated apps:

App list

1.8.0 official external storage support

Are you using external storage, if yes which one:
Minio S3
Are you using encryption:
No

Client configuration

Browser:
Chrome 78.0.3904.97
Operating system:
windows 10

Logs

Web server error log

app_1    | 172.24.0.6 -  18/Nov/2019:15:09:09 +0000 "MKCOL /remote.php" 201
web_1    | 172.24.0.1 - - [18/Nov/2019:15:09:09 +0000] "MKCOL /remote.php/dav/files/visualrecord/minio/test12345 HTTP/1.1" 201 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like 
Gecko) Chrome/78.0.3904.97 Safari/537.36" "-"
app_1    | 172.24.0.6 -  18/Nov/2019:15:09:09 +0000 "PROPFIND /remote.php" 404
web_1    | 172.24.0.1 - - [18/Nov/2019:15:09:09 +0000] "PROPFIND /remote.php/dav/files/visualrecord/minio/test12345 HTTP/1.1" 404 212 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, 
like Gecko) Chrome/78.0.3904.97 Safari/537.36" "-"
app_1    | 172.24.0.6 -  18/Nov/2019:15:09:09 +0000 "GET /ocs/v2.php" 200
web_1    | 172.24.0.1 - - [18/Nov/2019:15:09:09 +0000] "GET /ocs/v2.php/apps/notifications/api/v2/notifications HTTP/1.1" 200 1594 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36" "-"

Nextcloud log (data/nextcloud.log)

Nextcloud on another pc log

[no app in context] Error: OCP\Files\NotFoundException: /minio/anotherpctest not found while trying to get owner at <<closure>>

 0. /var/www/html/apps/activity/lib/FilesHooks.php line 638
    OC\Files\View->getOwner("/minio/anotherpctest")
 1. /var/www/html/apps/activity/lib/FilesHooks.php line 190
    OCA\Activity\FilesHooks->getSourcePathAndOwner("/minio/anotherpctest")
 2. /var/www/html/apps/activity/lib/FilesHooks.php line 146
    OCA\Activity\FilesHooks->addNotificationsForFileAction("/minio/anotherpctest", "file_created", "created_self", "created_by")
 3. /var/www/html/apps/activity/lib/FilesHooksStatic.php line 47
    OCA\Activity\FilesHooks->fileCreate("/minio/anotherpctest")
 4. /var/www/html/lib/private/legacy/hook.php line 106
    OCA\Activity\FilesHooksStatic::fileCreate({run: true,path: "/minio/anotherpctest"})
 5. /var/www/html/lib/private/Files/View.php line 1273
    OC_Hook::emit("OC_Filesystem", "post_create", {run: true,path: "/minio/anotherpctest"})
 6. /var/www/html/lib/private/Files/View.php line 1201
    OC\Files\View->runHooks(["create","write"], "/minio/anotherpctest", true)
 7. /var/www/html/lib/private/Files/View.php line 270
    OC\Files\View->basicOperation("mkdir", "/minio/anotherpctest", ["create","write"])
 8. /var/www/html/apps/dav/lib/Connector/Sabre/Directory.php line 189
    OC\Files\View->mkdir("/minio/anotherpctest")
 9. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 1210
    OCA\DAV\Connector\Sabre\Directory->createDirectory("anotherpctest")
10. /var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 590
    Sabre\DAV\Server->createCollection("files/visualrecord/minio/anotherpctest", Sabre\DAV\MkCol {})
11. <<closure>>
    Sabre\DAV\CorePlugin->httpMkcol(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
12. /var/www/html/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
    undefinedundefinedcall_user_func_array([Sabre\DAV\CorePlugin {},"httpMkcol"], [Sabre\HTTP\Requ ... }])
13. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 479
    Sabre\Event\EventEmitter->emit("method:MKCOL", [Sabre\HTTP\Requ ... }])
14. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 254
    Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
15. /var/www/html/apps/dav/lib/Server.php line 317
    Sabre\DAV\Server->exec()
16. /var/www/html/apps/dav/appinfo/v2/remote.php line 35
    OCA\DAV\Server->exec()
17. /var/www/html/remote.php line 163
    undefinedundefinedrequire_once("/var/www/html/a ... p")

MKCOL /remote.php/dav/files/visualrecord/minio/anotherpctest
from 172.24.0.1 by visualrecord at 2019-11-18T15:01:40+00:00

Minio on same pc log

[no app in context] Error: OCP\Files\NotFoundException: /miniolocaltest/samepctest not found while trying to get owner at <<closure>>

 0. /var/www/html/apps/activity/lib/FilesHooks.php line 638
    OC\Files\View->getOwner("/miniolocaltest/samepctest")
 1. /var/www/html/apps/activity/lib/FilesHooks.php line 190
    OCA\Activity\FilesHooks->getSourcePathAndOwner("/miniolocaltest/samepctest")
 2. /var/www/html/apps/activity/lib/FilesHooks.php line 146
    OCA\Activity\FilesHooks->addNotificationsForFileAction("/miniolocaltest/samepctest", "file_created", "created_self", "created_by")
 3. /var/www/html/apps/activity/lib/FilesHooksStatic.php line 47
    OCA\Activity\FilesHooks->fileCreate("/miniolocaltest/samepctest")
 4. /var/www/html/lib/private/legacy/hook.php line 106
    OCA\Activity\FilesHooksStatic::fileCreate({run: true,path: ... "})
 5. /var/www/html/lib/private/Files/View.php line 1273
    OC_Hook::emit("OC_Filesystem", "post_create", {run: true,path: ... "})
 6. /var/www/html/lib/private/Files/View.php line 1201
    OC\Files\View->runHooks(["create","write"], "/miniolocaltest/samepctest", true)
 7. /var/www/html/lib/private/Files/View.php line 270
    OC\Files\View->basicOperation("mkdir", "/miniolocaltest/samepctest", ["create","write"])
 8. /var/www/html/apps/dav/lib/Connector/Sabre/Directory.php line 189
    OC\Files\View->mkdir("/miniolocaltest/samepctest")
 9. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 1210
    OCA\DAV\Connector\Sabre\Directory->createDirectory("samepctest")
10. /var/www/html/3rdparty/sabre/dav/lib/DAV/CorePlugin.php line 590
    Sabre\DAV\Server->createCollection("files/visualrec ... t", Sabre\DAV\MkCol {})
11. <<closure>>
    Sabre\DAV\CorePlugin->httpMkcol(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
12. /var/www/html/3rdparty/sabre/event/lib/EventEmitterTrait.php line 105
    undefinedundefinedcall_user_func_array([Sabre\DAV\CorePlugin {},"httpMkcol"], [Sabre\HTTP\Requ ... }])
13. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 479
    Sabre\Event\EventEmitter->emit("method:MKCOL", [Sabre\HTTP\Requ ... }])
14. /var/www/html/3rdparty/sabre/dav/lib/DAV/Server.php line 254
    Sabre\DAV\Server->invokeMethod(Sabre\HTTP\Reque ... "}, Sabre\HTTP\Response {})
15. /var/www/html/apps/dav/lib/Server.php line 317
    Sabre\DAV\Server->exec()
16. /var/www/html/apps/dav/appinfo/v2/remote.php line 35
    OCA\DAV\Server->exec()
17. /var/www/html/remote.php line 163
    undefinedundefinedrequire_once("/var/www/html/a ... p")

MKCOL /remote.php/dav/files/visualrecord/miniolocaltest/samepctest
from 172.24.0.1 by visualrecord at 2019-11-18T15:01:20+00:00

Browser log

Insert your browser log here, this could for example include:

PROPFIND /remote.php/dav/files/visualrecord/minio/anotherpctest HTTP/1.1
Host: *****:8080
Connection: keep-alive
Content-Length: 621
requesttoken: OwKW3OrixY8mrE+NypUgyCwECDGbKx2CfWoTVAdZCy8=:XHilhJmnjOJShwTPhMBV/VlFPVXcBEXgKQF6NUUgPAA=
Origin: http://******:8080
X-Requested-With: XMLHttpRequest
Content-Type: application/xml; charset=UTF-8
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36
Depth: 0
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7
Cookie: _ym_uid=1561710448197831521; _ym_d=1561710448; _ga=GA1.2.4037843.1568360611; oc8m9eda6ka0=4357258de25b3fb553997955c0974d9d; oc_sessionPassphrase=WqqDAyyn46mRbLgVmSJpnHoEJC8VIWzrwxF5IWkI9VI1Xn%2F0Kg6YWW8naOI%2FfFZikTHhEqa%2Fh10Kil7RK3qbt%2FZ1bneNltaNmMTnx0q%2B7cYQ%2BR%2BHepy2QB2zh1qL%2BJjw; nc_sameSiteCookielax=true; nc_sameSiteCookiestrict=true; ocv68uhbfnjq=44ff99c2cad18a25943f74baa154344d; oced8ucjucd8=2df2ac30ce378b8e2af0f9eebcfc8814; email=dmitriyloktionov%40gmail.com; qr_code=ekyxu; ocfm1u7q0yf5=4e6e737d8c2703e2ca4a6d100543a1b3; nc_username=visualrecord; ocgkgyxkgo87=b89da3d1284f16e0c57b1b0d300bb035; nc_token=pLQDat9RGpl3dbJLEjeo3aaGTDURTOXd; nc_session_id=b89da3d1284f16e0c57b1b0d300bb035



HTTP/1.1 404 Not Found
Server: nginx/1.17.5
Date: Mon, 18 Nov 2019 15:04:56 GMT
Content-Type: application/xml; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
X-Powered-By: PHP/7.3.11
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate
Pragma: no-cache
Content-Security-Policy: default-src 'none';
Vary: Brief,Prefer
DAV: 1, 3, extended-mkcol, access-control, calendarserver-principal-property-search, nc-calendar-search, nc-enable-birthday-calendar
Content-Encoding: gzip

Curl recieve 201, but in fact folder is not created

Curl log

curl -L -k --request MKCOL --user "****" http://*******:8080/remote.php/dav/files/visualrecord/
minio/test12345 -v
> MKCOL /remote.php/dav/files/visualrecord/minio/test12345 HTTP/1.1
> Authorization: Basic *****
> User-Agent: curl/7.40.0
> Host: ************:8080
> Accept: */*
>
< HTTP/1.1 201 Created
< Server: nginx/1.17.5
< Date: Mon, 18 Nov 2019 15:12:10 GMT
< Content-Type: text/html; charset=UTF-8
< Content-Length: 0
< Connection: keep-alive
< X-Powered-By: PHP/7.3.11
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate
< Pragma: no-cache
< Set-Cookie: oc_sessionPassphrase=GUWgOdP0wQU7U2I1DXw%2FQTGsmT3KK6AC%2BuZ%2Bq4VYw9X%2BMKP%2FARDZAUbsGhvkvbwM3RB1GZn7%2FsuVbY7KcvSwIwURKVxbaujI8dHV7pPODHfCZQgPp3p4ftIb6NoA7kNp; path=/; HttpOnly
< Set-Cookie: nc_sameSiteCookielax=true; path=/; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=lax
< Set-Cookie: nc_sameSiteCookiestrict=true; path=/; httponly;expires=Fri, 31-Dec-2100 23:59:59 GMT; SameSite=strict
< Content-Security-Policy: default-src 'none';
< Set-Cookie: ocgkgyxkgo87=31278c2602b30520192d06b12fcdc8cd; path=/; HttpOnly
< Set-Cookie: cookie_test=test; expires=Mon, 18-Nov-2019 16:12:09 GMT; Max-Age=3600
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Robots-Tag: none
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Referrer-Policy: no-referrer

The text was updated successfully, but these errors were encountered:

Cinerar · 2019-11-18T18:33:40Z

After some more experiments found out that on fresh minio installs this only happens only one time, and after page refresh folders are creating properly, but first time of folder creation it is still same error

Cinerar · 2019-12-04T10:29:19Z

#14913 (comment)
looks like this is connected

c-reiter · 2019-12-23T17:38:19Z

Are you using Minio to connect to B2? If so, the problem may be that B2 fails to create files with a trailing slash (aka folders; e.g. if you try to upload "directory/" via Minio to B2). B2 doesn't have "folders", so there can't be an empty folder. Folders are just filename prefixes for B2. If you create a folder in the B2 web interface, then it will create a file named ".bzEmpty" with a size of 0 bytes with the prefix of your folder name.

Cinerar · 2019-12-23T17:42:01Z

No, just standalone minio installation.

plinss · 2020-04-20T22:24:12Z

I'm having the same issue. Using minio for storage in file storage mode. If I create a folder via Nextcloud's web UI I get a "Could not create folder" error, however an empty folder is created within minio's file storage. Nextcloud does not see the folder however. All sorts of errors if I create a folder on my computer and attempt to let the sync client handle it.

Manually creating a file in the folder in minio's file storage lets Nextcloud see the folder and then sync completes normally.

This is most likely related to minio's handling of empty folders, which is not quite S3 compatible. A semi-recent change in minio allows empty folders, so long as minio is using regular file storage and the folder object is created with a trailing slash in it's path and a zero size, see: minio/minio#2423

I accept that Nextcloud is likely following the normal S3 behavior and that minio is not treating empty folders the same way other S3 implementations do. But hopefully something can be done to allow Nextcloud to deal with minio's quirks.

szaimen · 2021-05-31T13:03:09Z

Is this Issue still valid? If not, please close this issue. Thanks! :)

plinss · 2021-05-31T16:47:19Z

This appears to be working now with the NC 21 and Minio 2021-03-04

Cinerar added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Nov 18, 2019

solracsf added the feature: object storage label Apr 17, 2020

szaimen added the needs info label May 31, 2021

Cinerar closed this as completed May 31, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Minio as s3 external storage folder (directory) creation is not working properly or at all #17987

Minio as s3 external storage folder (directory) creation is not working properly or at all #17987

Cinerar commented Nov 18, 2019 •

edited

Loading

Cinerar commented Nov 18, 2019

Cinerar commented Dec 4, 2019

c-reiter commented Dec 23, 2019

Cinerar commented Dec 23, 2019

plinss commented Apr 20, 2020

szaimen commented May 31, 2021

plinss commented May 31, 2021

Minio as s3 external storage folder (directory) creation is not working properly or at all #17987

Minio as s3 external storage folder (directory) creation is not working properly or at all #17987

Comments

Cinerar commented Nov 18, 2019 • edited Loading

Steps to reproduce

Expected behaviour

Actual behaviour

Server configuration

Client configuration

Logs

Web server error log

Nextcloud log (data/nextcloud.log)

Browser log

Curl recieve 201, but in fact folder is not created

Cinerar commented Nov 18, 2019

Cinerar commented Dec 4, 2019

c-reiter commented Dec 23, 2019

Cinerar commented Dec 23, 2019

plinss commented Apr 20, 2020

szaimen commented May 31, 2021

plinss commented May 31, 2021

Cinerar commented Nov 18, 2019 •

edited

Loading