fix idp schema, add virtual setup

[butonic]

This just adds a config for the virtual views. The local integration tests now have two storage providers mounted at /virtual with the local driver. They can be accessed via the frontent-global service.

[update-docs]

Thanks for opening this pull request! The maintainers of this repository would appreciate it if you would create a changelog item based on your changes.

[butonic]

@ishank011 @labkode what kind of tests should we run on the /virtual namespace? We want to make sure

1. the etag for /virtual changes when either /virtual/aaaaa or /virtual/zzzzz is touched.
2. the mtime taken from the youngest node in either storage provider
3. the size is the sum of both storage providers
4. what else?

@phil-davis can you recommend a set of features we can use in drone to test etag and size propagation?

cc @C0rby @aduffeck

For now I have a basic test working: https://drone.cernbox.cern.ch/cs3org/reva/3916/8/5

It sets up two storage providers with the local driver and mounts them at /virtual/[a-k] and /virtual/[l-z].
The tests then create /virtual/(a|b|c|k|l|z) and do some PROPFIND sanity checks with depth: (0|1|infinity).

• we need to set the mtime for folders ... does the localfs propagate the etag?

[butonic]

ok, I am now checking if the etage of /, /virtual and /virtual/b changes when the folder /virtual/b/bar is created.

Unfortunately, not even this basic test works, I assume becaus I am using the local driver which

1. did not do a propagation when creating a dir (fixed in this PR)
2. does not seem provide an etag for the root of the storage ... or I am not fully understanding how the stat and list across providers code works...

when doing a PROPFIND against the virtual folder I should see the etag thet corresponds to the later provider entry, like this:

diff --git a/internal/grpc/services/gateway/storageprovider.go b/internal/grpc/services/gateway/storageprovider.go
index dcfd7be3..5292ac30 100644
--- a/internal/grpc/services/gateway/storageprovider.go
+++ b/internal/grpc/services/gateway/storageprovider.go
@@ -1334,7 +1334,14 @@ func (s *svc) statAcrossProviders(ctx context.Context, req *provider.StatRequest
                }
                if resp.Info != nil {
                        info.Size += resp.Info.Size
-                       info.Mtime = utils.LaterTS(info.Mtime, resp.Info.Mtime)
+                       if utils.TSToUnixNano(resp.Info.Mtime) > utils.TSToUnixNano(info.Mtime) {
+                               info.Mtime = resp.Info.Mtime
+                               info.Etag = resp.Info.Etag
+                               info.Checksum = resp.Info.Checksum
+                       }
+                       if info.Etag == "" && info.Etag != resp.Info.Etag {
+                               info.Etag = resp.Info.Etag
+                       }
                }
        }
 
@@ -1705,7 +1712,14 @@ func (s *svc) listContainerAcrossProviders(ctx context.Context, req *provider.Li
                                }
                                // TODO(ishank011): aggregrate properties such as etag, checksum, etc.
                                p.Size += info.Size
-                               p.Mtime = utils.LaterTS(p.Mtime, info.Mtime)
+                               if utils.TSToUnixNano(info.Mtime) > utils.TSToUnixNano(p.Mtime) {
+                                       p.Mtime = info.Mtime
+                                       p.Etag = info.Etag
+                                       p.Checksum = info.Checksum
+                               }
+                               if p.Etag == "" && p.Etag != info.Etag {
+                                       p.Etag = info.Etag
+                               }
                                p.Type = provider.ResourceType_RESOURCE_TYPE_CONTAINER
                                p.MimeType = "httpd/unix-directory"
                        } else {

right @ishank011 ?

Anyway, we should add the expectations to this PRs feature. Hopefully, it can be mimicked using the localfs.

[butonic]

without the above diff I get

      WebDavPropertiesContext::theseEtagsShouldHaveChanged
      The etag '' of element '/' of user 'einstein' did not change.
      The etag '' of element '/virtual' of user 'einstein' did not change.
      The etag '"0330f2019a2524fc51bc62964a186ac4"' of element '/virtual/b' of user 'einstein' did not change.
      Failed asserting that 3 matches expected 0.

with it I can get rid of the last etag not changing:

      WebDavPropertiesContext::theseEtagsShouldHaveChanged
      The etag '' of element '/' of user 'einstein' did not change.
      The etag '' of element '/virtual' of user 'einstein' did not change.
      Failed asserting that 2 matches expected 0.

In any case, I was under the impression that / as well as /virtual should change as well, right @ishank011 @labkode ?

[individual-it]

are these tests reva specific or should they be moved/copied to ocis?

[butonic]

are these tests reva specific or should they be moved/copied to ocis?

They are reva specific. The 'virtual views' feature allows merging multiple storage providers to serve a single folder. Not something owncloud 10 can do.

[butonic]

@ishank011 I think we have a passing test! 🤞

[individual-it]

I would suggest not to use "einstein" as user in the tests but create a new user as we do everywhere else

[butonic]

I would suggest not to use "einstein" as user in the tests but create a new user as we do everywhere else

@individual-it I did not want to make an ldap server necessary as I don't want to test managing users. I would actually like to reduce this test to a point where we only register the storage providers and don't even have users ... but I would have to test further if disabling user home creation on login is enough.

[butonic]

hm ... we could change the config of the default test suite to start two storage providers for /users .... but that would make the setup even more complex ...