Skip to content

Commit

Permalink
commit 5483c4d
Browse files Browse the repository at this point in the history
Merge: 91d1d56 322be13
Author: Willy Kloucek <34452982+wkloucek@users.noreply.github.com>
Date:   Tue Feb 1 13:08:12 2022 +0100

    Merge pull request #3072 from owncloud/remove-owncloud-storage-driver

    [full-ci] remove the owncloud storage driver
  • Loading branch information
wkloucek committed Feb 1, 2022
1 parent 88f0c1b commit e63399b
Show file tree
Hide file tree
Showing 10 changed files with 10 additions and 53 deletions.
45 changes: 1 addition & 44 deletions extensions/storage/storagedrivers.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,7 +90,7 @@ The S3NG storage driver uses the same metadata layout on a POSIX storage as the

TODO add list of capabilities / tradeoffs

#### Related Kernel limits
#### Related Kernel limits
The decomposedfs currently stores CS3 grants in extended attributes. When listing extended attributes the result is currently limited to 64kB. Assuming a 20 byte uuid a grant has ~40 bytes. Which would limit the number of extended attributes to ~1630 entries or ~1600 shares. This can be extended by moving the grants from extended attributes into a dedicated file.

From [Wikipedia on Extended file attributes](https://en.wikipedia.org/wiki/Extended_file_attributes#Linux):
Expand Down Expand Up @@ -134,49 +134,6 @@ The *minimal* storage driver for a POSIX based filesystem. It literally supports

To provide the other storage aspects we plan to implement a FUSE overlay filesystem which will add the different aspects on top of local filesystems like ext4, btrfs or xfs. It should work on NFSv45 as well, although NFSv4 supports RichACLs and we will explore how to leverage them to implement sharing at a future date. The idea is to use the storages native capabilities to deliver the best user experience. But again: that means making the right tradeoffs.

### OwnCloud Storage Driver

{{< hint info >}}
**Deprecated**
The owncloud storage driver was used for testing purposes early on. For a parallel deployment focus has shifted to the owncloudsql.
{{< /hint >}}

While this storage driver implements the file tree (using redis, including id based lookup), ETag propagation, trash, versions and sharing (including expiry) using the data directory layout of ownCloud 10 it has [known limitations](https://github.com/owncloud/core/issues/28095) that cannot be fixed without changing the actual layout on disk.

To setup it up properly in a distributed fashion, the storage-home and the storage-oc need to share the same underlying FS. Their "data" counterparts also need access to the same shared FS.
For a simple docker-compose setup, you can create a volume which will be used by the "storage-storage-home", "storage-storage-home-data", "storage-storage-oc" and "storage-storage-oc-data" containers. Using the `owncloud/ocis` docker image, the volume would need to be hooked in the `/var/lib/ocis` folder inside the containers.

- tree provided by a POSIX filesystem
- file layout is mapped to the old ownCloud 10 layout
- the root of tree for a user on disk is prefixed with `/path/to/data/<username>/files/`
- efficient path by id lookup
- all files and folders get assigned a uuid in the extended attributes
- when starting the storage provider it will walk all files to populate a redis kv store for uuid to path lookup
- slow to boot trees with lots of nodes
- build in ETag propagation
- ETags are calculated based on mtime
- mtime is propagated by the storage driver
- changes bypassing ocis are not picked up until a restart of the storage provider
- no subtree accounting, same options as for local storage
- efficient rename
- TODO [update the kv store for path lookup](https://github.com/cs3org/reva/issues/985), this is an O(n) operation
- arbitrary metadata using extended attributes
- grant persistence
- using custom ACLs that are stored as extended attributes
- a grant corresponds to one extended attribute of 40-100 bytes, effectively limiting the number of shares to ~100-40
- extended attributes have varying limitations, based on the underlying filesystem
- the linux kernel imposes a limit of 255bytes per name and 64KiB per value
- ext2/3/4: total bytes for all attributes of a file is limited to 4KiB (a filesystem block)
- xfs: limit of 64KiB per value
- btrfs: total bytes used for the name, value, and implementation overhead bytes 16KiB (the default filesystem nodesize value)
- does not require OS level integration
- built in trash
- trashed files are moved to `/path/to/data/<username>/files_trashbin/`
- trashed files are appended a timestamp `.d<unixtime>`, which [breaks trashing of files that reach the filesystems specific name limit](https://github.com/owncloud/core/issues/28095)
- built in versions
- file versions are stored in `/path/to/data/<username>/files_versions/`
- file versions are appended a timestamp `.d<unixtime>`, which [breaks versioning of files that reach the filesystems specific name limit](https://github.com/owncloud/core/issues/28095)

### EOS Storage Driver

The CERN eos storage has evolved with ownCloud and natively supports id based lookup, ETag propagation, subtree size accounting, sharing, trash and versions. To use it you need to change the default configuration of the `storage storage-home` command (or have a look at the Makefile ̀ eos-start` target):
Expand Down
2 changes: 1 addition & 1 deletion grpc_apis/ocis/messages/accounts/v0/grpc.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "ocis.messages.accounts.v0"
url: /grpc_apis/ocis.messages.accounts.v0
date: 2022-02-01T12:03:16Z
date: 2022-02-01T12:09:51Z
weight: 50
geekdocRepo: https://github.com/owncloud/ocis
---
Expand Down
2 changes: 1 addition & 1 deletion grpc_apis/ocis/messages/settings/v0/grpc.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "ocis.messages.settings.v0"
url: /grpc_apis/ocis.messages.settings.v0
date: 2022-02-01T12:03:16Z
date: 2022-02-01T12:09:51Z
weight: 50
geekdocRepo: https://github.com/owncloud/ocis
---
Expand Down
2 changes: 1 addition & 1 deletion grpc_apis/ocis/messages/store/v0/grpc.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "ocis.messages.store.v0"
url: /grpc_apis/ocis.messages.store.v0
date: 2022-02-01T12:03:16Z
date: 2022-02-01T12:09:51Z
weight: 50
geekdocRepo: https://github.com/owncloud/ocis
---
Expand Down
2 changes: 1 addition & 1 deletion grpc_apis/ocis/messages/thumbnails/v0/grpc.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "ocis.messages.thumbnails.v0"
url: /grpc_apis/ocis.messages.thumbnails.v0
date: 2022-02-01T12:03:17Z
date: 2022-02-01T12:09:51Z
weight: 50
geekdocRepo: https://github.com/owncloud/ocis
---
Expand Down
2 changes: 1 addition & 1 deletion grpc_apis/ocis/services/accounts/v0/grpc.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "ocis.services.accounts.v0"
url: /grpc_apis/ocis.services.accounts.v0
date: 2022-02-01T12:03:17Z
date: 2022-02-01T12:09:51Z
weight: 50
geekdocRepo: https://github.com/owncloud/ocis
---
Expand Down
2 changes: 1 addition & 1 deletion grpc_apis/ocis/services/settings/v0/grpc.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "ocis.services.settings.v0"
url: /grpc_apis/ocis.services.settings.v0
date: 2022-02-01T12:03:17Z
date: 2022-02-01T12:09:51Z
weight: 50
geekdocRepo: https://github.com/owncloud/ocis
---
Expand Down
2 changes: 1 addition & 1 deletion grpc_apis/ocis/services/store/v0/grpc.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "ocis.services.store.v0"
url: /grpc_apis/ocis.services.store.v0
date: 2022-02-01T12:03:17Z
date: 2022-02-01T12:09:51Z
weight: 50
geekdocRepo: https://github.com/owncloud/ocis
---
Expand Down
2 changes: 1 addition & 1 deletion grpc_apis/ocis/services/thumbnails/v0/grpc.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "ocis.services.thumbnails.v0"
url: /grpc_apis/ocis.services.thumbnails.v0
date: 2022-02-01T12:03:16Z
date: 2022-02-01T12:09:51Z
weight: 50
geekdocRepo: https://github.com/owncloud/ocis
---
Expand Down
2 changes: 1 addition & 1 deletion ocis/deployment/oc10_ocis_parallel.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ geekdocFilePath: oc10_ocis_parallel.md
- ownCloud 10 is configured to synchronize users from the LDAP server
- ownCloud 10 is used to use OpenID Connect for authentication with Keycloak
- oCIS running behind Traefik as reverse proxy
- oCIS is using the ownCloud storage driver on the same files and same database as ownCloud 10
- oCIS is using the ownCloudSQL storage driver on the same files and same database as ownCloud 10
- oCIS is using Keycloak as OpenID Connect provider
- oCIS is using the LDAP server as user backend
- All requests to both oCIS and oC10 are routed through the oCIS proxy and will be routed based on an OIDC claim to one of them. Therefore admins can change on a user basis in the LDAP which backend is used.
Expand Down

0 comments on commit e63399b

Please sign in to comment.