initial docker-compose.yml for developers

[butonic]

This PR adds a new docker-compose.yml file that allows developers to quickly spin up a development environment for ocis using both: the owncloud storage driver as well as the eos storage driver. For the latter it also contains the necessary eos containers

get started using the owncloud storage driver

1. start a container for ocis: docker-compose up -d ocis

   Note: On MacOS do not mount a local folder at the owncloud storage drivers data location. It uses a fuse based driver which does not support extended attributes.

2. initialize empty shares list: docker-compose exec ocis bash -c "echo '{}' > /var/tmp/reva/shares.json"
   • fix this, it is annoying
3. compile ocis inside the container using docker-compose exec ocis make clean build. this makes sure that all cgo bindings are there and the binary will work properly when resolving ldap users
4. run ocis server: docker-compose exec ocis ./bin/ocis server
5. in another terminal list running ocis extensions: docker-compose exec ocis ./bin/ocis list

+--------------------------+-----+
|        EXTENSION         | PID |
+--------------------------+-----+
| accounts                 | 200 |
| api                      | 238 |
| glauth                   | 205 |
...
| thumbnails               | 222 |
| web                      | 243 |
| webdav                   |  79 |
+--------------------------+-----+

6. test access @ https: https://localhost:9200/
   ocis will use the owncloud storage driver and store files in the container at /var/tmp/reva/data/<username>/files.

start eos cluster:

1. open a new terminal to keep the eos log seperate from the ocis log
2. run the necessary eos containers using docker-compose up quark-1 quark-2 quark-3 mgm-master mq-master fst
3. enable the default storage space using docker-compose exec mgm-master eos space set default on otherwise you will encounter an out of space error (code 54) when trying to upload files.
4. test if the ocis container works using docker-compose exec ocis eos whoami ... should give Virtual Identity: uid=0 (0,99,3) gid=0 (0,99,4) [authz:sss] sudo* host=ocis.testnet domain=testnet

Note: restarting eos needs

• a docker-compose down -v to delete all volumes
• AND the e folder must be deleted or eos fsts complain about existing folders

switch to eos storage driver

1. kill the home storage and data providers. we need to switch them to the eoshome driver:
   docker-compose exec ocis ./bin/ocis kill reva-storage-home
docker-compose exec ocis ./bin/ocis kill reva-storage-home-data
2. restart them with the eoshome driver and a new layout:
   docker-compose exec -e REVA_STORAGE_EOS_LAYOUT="{{substr 0 1 .Username}}/{{.Username}}" -e REVA_STORAGE_HOME_DRIVER=eoshome ocis ./bin/ocis run reva-storage-home
docker-compose exec -e REVA_STORAGE_EOS_LAYOUT="{{substr 0 1 .Username}}/{{.Username}}" -e REVA_STORAGE_HOME_DATA_DRIVER=eoshome ocis ./bin/ocis run reva-storage-home-data
3. restart the reva frontend with a new namespace (pointing to the eos storage provider) for the dav files endpoint
   docker-compose exec ocis ./bin/ocis kill reva-frontend
docker-compose exec -e DAV_FILES_NAMESPACE="/eos/" -d ocis ./bin/ocis reva-frontend
4. upload a file to einsteins home and verify the file is there using

docker-compose exec ocis eos ls -l /eos/dockertest/reva/users/e/einstein/
-rw-r--r--   1 einstein users              10 Jul  1 15:24 newfile.txt

TODO

• add make targets?
• document usage in devdocs. The above is a start...
  • all replaces in the go.mod must be added to the ocis volumes in the docker compose file as well

[update-docs]

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

[C0rby]

After submitting the login form I see an error in the log.

ERR error calling Authenticate error="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp: lookup host.docker.internal on 127.0.0.11:53: no such host\"" pkg=rhttp service=reva traceid=8493dd1e95026bc08605dac87ebbda7f

And phoenix shows the login error screen.

[butonic]

@C0rby try commenting https://github.com/owncloud/ocis/pull/349/files#diff-4e5e90c6228fd48698d074241c2ba760R46

[kulmann]

@butonic with all the steps from above, I can login. But uploading a file (.jpg) results in:

2020-07-02T07:24:37Z ERR root/go/pkg/mod/github.com/cs3org/reva@v0.1.1-0.20200629131207-04298ea1c088/internal/grpc/services/storageprovider/storageprovider.go:459 > error stating file: path:"/eos/einstein/aaa/IMG_2631.jpg"  error="user: unknown user einstein" pkg=rgrpc service=reva traceid=cc601607272855a177480d1707eaa167

in ocis logs and

mgm-master    | 200702 07:23:14 time=1593674594.615624 func=MakeResult               level=ERROR logid=static.............................. unit=mgm@mgm-master.testnet:1094 tid=00007f3eefff3700 source=ProcCommand:565                tident= sec=(null) uid=99 gid=99 name=- geo="" Unable to stat /eos/dockertest/reva/users/e/einstein; No such file or directory (errno=2)

in eos logs

However, docker-compose exec ocis eos ls -la /eos/dockertest/reva/users/e/ shows:
drwxrws--+ 1 20000 30000 0 Jul 2 07:24 einstein
🤔

[individual-it]

fixing /var/tmp/reva/shares.json in owncloud-archive/eos-stack#2

[butonic]

@felixboehm starting only ocis does not work without eos

docker-compose up ocis
Creating network "testnet" with the default driver
Creating ocis ... done
Attaching to ocis
ocis          | Waiting for EOS MGM
ocis          | Ncat: Could not resolve hostname "mgm-master.testnet": Name or service not known. QUITTING.
ocis          | Ncat: Could not resolve hostname "mgm-master.testnet": Name or service not known. QUITTING.
ocis          | Ncat: Could not resolve hostname "mgm-master.testnet": Name or service not known. QUITTING.
ocis          | Ncat: Could not resolve hostname "mgm-master.testnet": Name or service not known. QUITTING.
ocis          | Ncat: Could not resolve hostname "mgm-master.testnet": Name or service not known. QUITTING.

[C0rby]

fixing /var/tmp/reva/shares.json in owncloud-docker/eos-stack#2

Shouldn't we also fix it in the share manger?

[butonic]

@C0rby what exactly? both create a file with {} as content if it does not exist...

• the publicshare json manager
• the usershare json manager

The implementation can no doubt be improved, but AFAICT they already autocreate the file if it is missing. There might be problems when tryding to mount the file as a volume in a docker container though... would need a separate issue.

[C0rby]

I forgot to link it here. cs3org/reva#940

[butonic]

ci: no space left on device ... restarting

[felixboehm]

I have added docs to the your PR for development with eos.
https://github.com/owncloud/ocis/pull/349/files#diff-1bf07101845de6da98f1c948b82c5ce0
Can you check if that is ok? Maybe the devdocs files need some kind of header??

I tested all the steps right now with eos-stack from PR owncloud-archive/eos-stack#4, which I will publish on docker hub as soon as merged.

[felixboehm]

Hmm, still can reproduce the upload issue on eos, mentioned by @kulmann

ERR root/go/pkg/mod/github.com/cs3org/reva@v0.1.1-0.20200701152626-2f6cc60e2f66/internal/grpc/services/storageprovider/storageprovider.go:459 > error stating file: path:"/eos/einstein/IMG_0793.jpg"  error="user: unknown user einstein" pkg=rgrpc service=reva traceid=6514f73fd141e58ba62be23634282efa

maybe is the layout still not correct? Looks weird path:"/eos/einstein/IMG_0793.jpg"

[refs]

I can also reproduce @kulmann issue:

2020-07-07T16:38:15Z ERR root/go/pkg/mod/github.com/cs3org/reva@v0.1.1-0.20200701152626-2f6cc60e2f66/internal/grpc/services/storageprovider/storageprovider.go:459 > error stating file: path:"/eos/einstein/Matt Butcher, Matt Farina - Go in Practice_ Includes 70 Techniques-Manning Publications (2016).pdf"  error="user: unknown user einstein" pkg=rgrpc service=reva traceid=6ba72529e5cc8fafe8a5dceda873a492

[felixboehm]

• added md header to the devdocs file.
• fixed the commands in to start eos: missing "run" in ocis run xyz

[butonic]

I can also reproduce @kulmann issue:

2020-07-07T16:38:15Z ERR root/go/pkg/mod/github.com/cs3org/reva@v0.1.1-0.20200701152626-2f6cc60e2f66/internal/grpc/services/storageprovider/storageprovider.go:459 > error stating file: path:"/eos/einstein/Matt Butcher, Matt Farina - Go in Practice_ Includes 70 Techniques-Manning Publications (2016).pdf"  error="user: unknown user einstein" pkg=rgrpc service=reva traceid=6ba72529e5cc8fafe8a5dceda873a492

restarting the reva-users service allows it to pick up the changed ldap env. added to the docs.

[butonic]

things we noticed while testing this:

ownCloud storage:
- accessing public link does not work

eos storage
- up & download to a folder accessed via a public link work
- download of a file accessed via a public link works
  - still has a new button?

- searching for "Mar" makes the collaborators dropdown flicker, while searching for "mar" shows "Curie"

- shared with me shows an empty name
  - accepting a share seems to work, at least the accept & reject options vanish
  - but the shared folder does not appear
- does it make sense to share the "shared" folder via public link?