Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HTTP 500 response when uploading file in a specific space #6151

Closed
Tracked by #6230
individual-it opened this issue Apr 26, 2023 · 11 comments
Closed
Tracked by #6230

HTTP 500 response when uploading file in a specific space #6151

individual-it opened this issue Apr 26, 2023 · 11 comments
Assignees
Labels
Priority:p1-urgent Consider a hotfix release with only that fix Type:Bug

Comments

@individual-it
Copy link
Member

Describe the bug

When I upload a file in this specific space I get an error 500, but the file is still uploaded

Steps to reproduce

Steps to reproduce the behavior:

  1. test too much with folders having a lot of files & subfolders
  2. upload a file to that specific space

Expected behavior

file is uploaded without an issue

Actual behavior

File is uploaded, but error is shown that it could not be uploaded
https://user-images.githubusercontent.com/2425577/234560474-3e00d6c1-9052-4613-90e6-6f7d199e8143.mp4
(the video does not show how I select the file wwktm.jpg for upload)

Setup

Please describe how you started the server and provide a list of relevant environment variables or configuration files.

OCIS_LOG_PRETTY=true OCIS_DECOMPOSEDFS_METADATA_BACKEND=messagepack OCIS_URL=https://192.168.56.1:9200  ./ocis-3.0.0-rc.1-linux-amd64 server

Additional context

ocis log output:

2023-04-26T17:03:43+05:45 ERR http end="26/Apr/2023:17:03:43 +0545" host=127.0.0.1 method=PATCH pkg=rhttp proto=HTTP/1.1 service=storage-users size=69 start="26/Apr/2023:17:03:43 +0545" status=500 time_ns=31968955 traceid=00000000000000000000000000000000 uri=/data/tus/f2d1d344-2ba5-4e2d-9c8d-627065b7f2e4 url=/f2d1d344-2ba5-4e2d-9c8d-627065b7f2e4
2023-04-26T17:03:43+05:45 ERR http end="26/Apr/2023:17:03:43 +0545" host=127.0.0.1 method=PATCH pkg=rhttp proto=HTTP/1.1 service=frontend size=0 start="26/Apr/2023:17:03:43 +0545" status=500 time_ns=32617192 traceid=00000000000000000000000000000000 uri=/data/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZXZhIiwiZXhwIjoxNjgyNTk0MzIzLCJpYXQiOjE2ODI1MDc5MjMsInRhcmdldCI6Imh0dHA6Ly9sb2NhbGhvc3Q6OTE1OC9kYXRhL3R1cy9mMmQxZDM0NC0yYmE1LTRlMmQtOWM4ZC02MjcwNjViN2YyZTQifQ.uYQHx68CfIX7ve1vy616fyHG4AjPURDjBdHwm0KstGk url=/eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJyZXZhIiwiZXhwIjoxNjgyNTk0MzIzLCJpYXQiOjE2ODI1MDc5MjMsInRhcmdldCI6Imh0dHA6Ly9sb2NhbGhvc3Q6OTE1OC9kYXRhL3R1cy9mMmQxZDM0NC0yYmE1LTRlMmQtOWM4ZC02MjcwNjViN2YyZTQifQ.uYQHx68CfIX7ve1vy616fyHG4AjPURDjBdHwm0KstGk
2023/04/26 17:03:58 http: proxy error: context canceled
@ScharfViktor ScharfViktor mentioned this issue Apr 27, 2023
89 tasks
@micbar micbar added the Priority:p1-urgent Consider a hotfix release with only that fix label May 2, 2023
@micbar
Copy link
Contributor

micbar commented May 2, 2023

@dragonchaser Can you take a look?

@dragonchaser dragonchaser self-assigned this May 2, 2023
@dragonchaser
Copy link
Member

dragonchaser commented May 2, 2023

I could not reproduce that, I have uploaded parts of the linux source tree. But what I witnessed was that I am being logged out during the upload. Could that be related?

@individual-it can we have a look at this tomorrow morning? I should be around from 8 CET

@dragonchaser
Copy link
Member

Could not reproduce, I would assume that this is related to some token invalidation issue. But I would need more information. Is there some other configuration set in the env?

@dragonchaser
Copy link
Member

We (@individual-it and me) investigated, we tracked it down to two issues

  1. a broken folder in the space (@individual-it is trying to reproduce how it got broken)
  2. when uploading large folders with many files via the webui, the login token is not refreshed resulting in a logout and lots of 401 errors (this will be a seperate issue)

relevant to fixing this is #1, but ticket is blocked until breakage is reproducable.

@individual-it
Copy link
Member Author

there is a brocken folder, that I cannot delete. When I try I get

<?xml version="1.0" encoding="UTF-8"?>
<d:error xmlns:d="DAV" xmlns:s="http://sabredav.org/ns"><s:exception></s:exception><s:message>internal error: delete:strconv.ParseInt: parsing &#34;18446744073709360837&#34;: value out of range</s:message></d:error>

@individual-it
Copy link
Member Author

individual-it commented May 4, 2023

I can delete all files/folders from that space, except of that specific brocken folder.
When trying to delete files/folders I get 500 errors on some of them, with this error logs

logs
2023-05-04T13:22:21+05:45 ERR unary code=Internal end="04/May/2023:13:22:21 +0545" from=tcp://127.0.0.1:48142 pkg=rgrpc service=storage-users start="04/May/2023:13:22:21 +0545" time_ns=3422063 traceid=00000000000000000000000000000000 uri=/cs3.storage.provider.v1beta1.ProviderAPI/PurgeRecycle user-agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/112.0"
2023-05-04T13:22:21+05:45 ERR error listing grantees error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/f4/ca/8e/4a/-1abd-4615-93f3-32ae922d4fe0: no such file or directory" node="marshaling error: json: unsupported value: encountered a cycle via *node.Node" pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error reading permissions error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/f4/ca/8e/4a/-1abd-4615-93f3-32ae922d4fe0: no such file or directory" node=f4ca8e4a-1abd-4615-93f3-32ae922d4fe0 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error listing attributes error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/9d/be/5a/82/-f192-40a9-93d4-884c13ca9151: no such file or directory" node=9dbe5a82-f192-40a9-93d4-884c13ca9151 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR Internal Server Error code=500 item_path=/ key=f26f6242-9eb9-4e86-96f5-b56b96136b20 name=com.owncloud.web.ocdav reference={"path":".","resource_id":{"space_id":"0c8278e2-1126-4f9e-9e0d-b9d9dc0f1e05","storage_id":"ecabb8de-991a-426d-9a58-82fae8d6bcf4"}} request-id=7184e5e8-e017-45bf-8d47-bf587eb46524 service=ocdav status={"code":15,"message":"gateway could not call PurgeRecycle:rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference","trace":"00000000000000000000000000000000"} traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error listing grantees error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/f5/f8/b9/53/-cd01-41ea-b18f-fa63fa2f0f1e: no such file or directory" node="marshaling error: json: unsupported value: encountered a cycle via *node.Node" pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error reading permissions error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/f5/f8/b9/53/-cd01-41ea-b18f-fa63fa2f0f1e: no such file or directory" node=f5f8b953-cd01-41ea-b18f-fa63fa2f0f1e pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error listing attributes error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/09/ee/57/dd/-dcff-40f9-aaec-1560938a9454: no such file or directory" node=09ee57dd-dcff-40f9-aaec-1560938a9454 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error listing grantees error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/9d/be/5a/82/-f192-40a9-93d4-884c13ca9151: no such file or directory" node="marshaling error: json: unsupported value: encountered a cycle via *node.Node" pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error reading permissions error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/9d/be/5a/82/-f192-40a9-93d4-884c13ca9151: no such file or directory" node=9dbe5a82-f192-40a9-93d4-884c13ca9151 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
goroutine 15351 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
	runtime/debug/stack.go:16 +0x19
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x429ba58, 0xc01d52d1a0}, {0x3d9dc80, 0x5a50c60})
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/recovery/recovery.go:48 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x429ba58?, 0xc01d52d1a0?}, {0x3d9dc80?, 0x5a50c60?}, 0xc01da99c10?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x3d9dc80, 0x5a50c60})
	runtime/panic.go:884 +0x213
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Permissions).AssemblePermissions(0x3a00fd0?, {0x429ba58, 0xc01d52d4a0}, 0xc01d5285a0)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/node/permissions.go:149 +0x64c
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.Permissions.AssemblePermissions(...)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/spacepermissions.go:39
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).PurgeRecycleItem(0xc00049d0a0, {0x429ba58, 0xc01d52d4a0}, 0xc01d52d470?, {0xc01cb793e0, 0x24}, {0x3a00fd0?, 0x3a08aa0?})
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/recycle.go:349 +0x174
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).PurgeRecycle(0xc008e686c0, {0x429ba58?, 0xc01d52d4a0?}, 0xc01d243dc0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/services/storageprovider/storageprovider.go:952 +0x1af
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler.func1({0x429ba58, 0xc01d52d4a0}, {0x40cf380?, 0xc01d243dc0})
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6184 +0x7b
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x429ba58, 0xc01d52d3e0}, {0x40cf380, 0xc01d243dc0}, 0xc01d24cca0, 0xc01d24af90)
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.36.4/interceptor.go:341 +0x437
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52d3e0?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/eventsmiddleware.NewUnary.func1({0x429ba58, 0xc01d52d3b0}, {0x40cf380?, 0xc01d243dc0?}, 0x30?, 0xc01d24ccc0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/eventsmiddleware/events.go:72 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52d3b0?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/prometheus.interceptorFromConfig.func1({0x429ba58?, 0xc01d52d3b0?}, {0x40cf380?, 0xc01d243dc0?}, 0x3c73de0?, 0x3a08b01?)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/prometheus/prometheus.go:72 +0x43
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52d3b0?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func2({0x429ba58, 0xc01d52d1a0}, {0x40cf380, 0xc01d243dc0}, 0xc01d24cca0, 0xc01d24cd00)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/auth/auth.go:160 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52d1a0?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x429ba58?, 0xc01d52d1a0?}, {0x40cf380?, 0xc01d243dc0?}, 0xc01058d240?, 0x496137?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52d1a0?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x429ba58, 0xc01d52d1a0}, {0x40cf380, 0xc01d243dc0}, 0xc01d24cca0, 0xc01d24cd40)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52d1a0?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x429ba58, 0xc01d52d110}, {0x40cf380, 0xc01d243dc0}, 0x1?, 0xc01d24cd60)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/useragent/useragent.go:38 +0xf4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52d110?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x429ba58, 0xc01d52d080}, {0x40cf380, 0xc01d243dc0}, 0x5d08ea0?, 0xc01d24cd80)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/token/token.go:44 +0x16d
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52d080?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x429ba58, 0xc01d52cff0}, {0x40cf380, 0xc01d243dc0}, 0x3e64c40?, 0xc01d24cda0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52cff0?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x429ba58, 0xc01d52cff0}, {0x40cf380, 0xc01d243dc0}, 0xc0105b5a58?, 0x3d9c900?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbe
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler({0x41fb300?, 0xc008e686c0}, {0x429ba58, 0xc01d52cff0}, 0xc01cc57650, 0xc008ec6480)
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6186 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d505c20, 0xc008ec6630, 0x5a798a8, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1345 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d505c20, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1722 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.54.0/server.go:966 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.54.0/server.go:964 +0x28a
2023-05-04T13:22:21+05:45 ERR error listing grantees error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/09/ee/57/dd/-dcff-40f9-aaec-1560938a9454: no such file or directory" node="marshaling error: json: unsupported value: encountered a cycle via *node.Node" pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error reading permissions error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/09/ee/57/dd/-dcff-40f9-aaec-1560938a9454: no such file or directory" node=09ee57dd-dcff-40f9-aaec-1560938a9454 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR runtime error: invalid memory address or nil pointer dereference; stack: goroutine 15351 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x429ba58, 0xc01d52d1a0}, {0x3d9dc80, 0x5a50c60})
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/recovery/recovery.go:50 +0x5b
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x429ba58?, 0xc01d52d1a0?}, {0x3d9dc80?, 0x5a50c60?}, 0xc01da99c10?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x3d9dc80, 0x5a50c60})
	runtime/panic.go:884 +0x213
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Permissions).AssemblePermissions(0x3a00fd0?, {0x429ba58, 0xc01d52d4a0}, 0xc01d5285a0)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/node/permissions.go:149 +0x64c
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.Permissions.AssemblePermissions(...)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/spacepermissions.go:39
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).PurgeRecycleItem(0xc00049d0a0, {0x429ba58, 0xc01d52d4a0}, 0xc01d52d470?, {0xc01cb793e0, 0x24}, {0x3a00fd0?, 0x3a08aa0?})
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/recycle.go:349 +0x174
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).PurgeRecycle(0xc008e686c0, {0x429ba58?, 0xc01d52d4a0?}, 0xc01d243dc0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/services/storageprovider/storageprovider.go:952 +0x1af
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler.func1({0x429ba58, 0xc01d52d4a0}, {0x40cf380?, 0xc01d243dc0})
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6184 +0x7b
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x429ba58, 0xc01d52d3e0}, {0x40cf380, 0xc01d243dc0}, 0xc01d24cca0, 0xc01d24af90)
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.36.4/interceptor.go:341 +0x437
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52d3e0?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/eventsmiddleware.NewUnary.func1({0x429ba58, 0xc01d52d3b0}, {0x40cf380?, 0xc01d243dc0?}, 0x30?, 0xc01d24ccc0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/eventsmiddleware/events.go:72 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52d3b0?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/prometheus.interceptorFromConfig.func1({0x429ba58?, 0xc01d52d3b0?}, {0x40cf380?, 0xc01d243dc0?}, 0x3c73de0?, 0x3a08b01?)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/prometheus/prometheus.go:72 +0x43
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52d3b0?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func2({0x429ba58, 0xc01d52d1a0}, {0x40cf380, 0xc01d243dc0}, 0xc01d24cca0, 0xc01d24cd00)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/auth/auth.go:160 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52d1a0?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x429ba58?, 0xc01d52d1a0?}, {0x40cf380?, 0xc01d243dc0?}, 0xc01058d240?, 0x496137?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52d1a0?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x429ba58, 0xc01d52d1a0}, {0x40cf380, 0xc01d243dc0}, 0xc01d24cca0, 0xc01d24cd40)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52d1a0?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x429ba58, 0xc01d52d110}, {0x40cf380, 0xc01d243dc0}, 0x1?, 0xc01d24cd60)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/useragent/useragent.go:38 +0xf4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52d110?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x429ba58, 0xc01d52d080}, {0x40cf380, 0xc01d243dc0}, 0x5d08ea0?, 0xc01d24cd80)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/token/token.go:44 +0x16d
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52d080?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x429ba58, 0xc01d52cff0}, {0x40cf380, 0xc01d243dc0}, 0x3e64c40?, 0xc01d24cda0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d52cff0?}, {0x40cf380?, 0xc01d243dc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x429ba58, 0xc01d52cff0}, {0x40cf380, 0xc01d243dc0}, 0xc0105b5a58?, 0x3d9c900?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbe
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler({0x41fb300?, 0xc008e686c0}, {0x429ba58, 0xc01d52cff0}, 0xc01cc57650, 0xc008ec6480)
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6186 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d505c20, 0xc008ec6630, 0x5a798a8, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1345 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d505c20, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1722 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.54.0/server.go:966 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.54.0/server.go:964 +0x28a
 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR unary code=Internal end="04/May/2023:13:22:21 +0545" from=tcp://127.0.0.1:48142 pkg=rgrpc service=storage-users start="04/May/2023:13:22:21 +0545" time_ns=5697300 traceid=00000000000000000000000000000000 uri=/cs3.storage.provider.v1beta1.ProviderAPI/PurgeRecycle user-agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/112.0"
goroutine 15385 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
	runtime/debug/stack.go:16 +0x19
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x429ba58, 0xc01d470870}, {0x3d9dc80, 0x5a50c60})
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/recovery/recovery.go:48 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x429ba58?, 0xc01d470870?}, {0x3d9dc80?, 0x5a50c60?}, 0xc01e8d7c10?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x3d9dc80, 0x5a50c60})
	runtime/panic.go:884 +0x213
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Permissions).AssemblePermissions(0x3a00fd0?, {0x429ba58, 0xc01d470b70}, 0xc01d4559e0)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/node/permissions.go:149 +0x64c
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.Permissions.AssemblePermissions(...)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/spacepermissions.go:39
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).PurgeRecycleItem(0xc00049d0a0, {0x429ba58, 0xc01d470b70}, 0xc01d470b40?, {0xc01d1435f0, 0x24}, {0x3a00fd0?, 0x3a08aa0?})
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/recycle.go:349 +0x174
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).PurgeRecycle(0xc008e686c0, {0x429ba58?, 0xc01d470b70?}, 0xc01d41ffc0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/services/storageprovider/storageprovider.go:952 +0x1af
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler.func1({0x429ba58, 0xc01d470b70}, {0x40cf380?, 0xc01d41ffc0})
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6184 +0x7b
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x429ba58, 0xc01d470ab0}, {0x40cf380, 0xc01d41ffc0}, 0xc01d464280, 0xc01d42da10)
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.36.4/interceptor.go:341 +0x437
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d470ab0?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/eventsmiddleware.NewUnary.func1({0x429ba58, 0xc01d470a80}, {0x40cf380?, 0xc01d41ffc0?}, 0x30?, 0xc01d4642a0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/eventsmiddleware/events.go:72 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d470a80?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/prometheus.interceptorFromConfig.func1({0x429ba58?, 0xc01d470a80?}, {0x40cf380?, 0xc01d41ffc0?}, 0x3c73de0?, 0x3a08b01?)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/prometheus/prometheus.go:72 +0x43
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d470a80?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func2({0x429ba58, 0xc01d470870}, {0x40cf380, 0xc01d41ffc0}, 0xc01d464280, 0xc01d4642e0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/auth/auth.go:160 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d470870?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x429ba58?, 0xc01d470870?}, {0x40cf380?, 0xc01d41ffc0?}, 0xc0101d5240?, 0x496137?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d470870?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x429ba58, 0xc01d470870}, {0x40cf380, 0xc01d41ffc0}, 0xc01d464280, 0xc01d464320)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d470870?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x429ba58, 0xc01d4707e0}, {0x40cf380, 0xc01d41ffc0}, 0x1?, 0xc01d464340)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/useragent/useragent.go:38 +0xf4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4707e0?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x429ba58, 0xc01d470750}, {0x40cf380, 0xc01d41ffc0}, 0x5d08ea0?, 0xc01d464360)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/token/token.go:44 +0x16d
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d470750?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x429ba58, 0xc01d4706c0}, {0x40cf380, 0xc01d41ffc0}, 0x3e64c40?, 0xc01d464380)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4706c0?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x429ba58, 0xc01d4706c0}, {0x40cf380, 0xc01d41ffc0}, 0xc01ce18a58?, 0x3d9c900?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbe
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler({0x41fb300?, 0xc008e686c0}, {0x429ba58, 0xc01d4706c0}, 0xc01d46c1c0, 0xc008ec6480)
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6186 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d413e60, 0xc008ec6630, 0x5a798a8, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1345 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d413e60, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1722 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.54.0/server.go:966 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.54.0/server.go:964 +0x28a
2023-05-04T13:22:21+05:45 ERR runtime error: invalid memory address or nil pointer dereference; stack: goroutine 15385 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x429ba58, 0xc01d470870}, {0x3d9dc80, 0x5a50c60})
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/recovery/recovery.go:50 +0x5b
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x429ba58?, 0xc01d470870?}, {0x3d9dc80?, 0x5a50c60?}, 0xc01e8d7c10?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x3d9dc80, 0x5a50c60})
	runtime/panic.go:884 +0x213
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Permissions).AssemblePermissions(0x3a00fd0?, {0x429ba58, 0xc01d470b70}, 0xc01d4559e0)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/node/permissions.go:149 +0x64c
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.Permissions.AssemblePermissions(...)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/spacepermissions.go:39
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).PurgeRecycleItem(0xc00049d0a0, {0x429ba58, 0xc01d470b70}, 0xc01d470b40?, {0xc01d1435f0, 0x24}, {0x3a00fd0?, 0x3a08aa0?})
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/recycle.go:349 +0x174
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).PurgeRecycle(0xc008e686c0, {0x429ba58?, 0xc01d470b70?}, 0xc01d41ffc0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/services/storageprovider/storageprovider.go:952 +0x1af
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler.func1({0x429ba58, 0xc01d470b70}, {0x40cf380?, 0xc01d41ffc0})
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6184 +0x7b
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x429ba58, 0xc01d470ab0}, {0x40cf380, 0xc01d41ffc0}, 0xc01d464280, 0xc01d42da10)
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.36.4/interceptor.go:341 +0x437
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d470ab0?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/eventsmiddleware.NewUnary.func1({0x429ba58, 0xc01d470a80}, {0x40cf380?, 0xc01d41ffc0?}, 0x30?, 0xc01d4642a0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/eventsmiddleware/events.go:72 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d470a80?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/prometheus.interceptorFromConfig.func1({0x429ba58?, 0xc01d470a80?}, {0x40cf380?, 0xc01d41ffc0?}, 0x3c73de0?, 0x3a08b01?)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/prometheus/prometheus.go:72 +0x43
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d470a80?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func2({0x429ba58, 0xc01d470870}, {0x40cf380, 0xc01d41ffc0}, 0xc01d464280, 0xc01d4642e0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/auth/auth.go:160 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d470870?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x429ba58?, 0xc01d470870?}, {0x40cf380?, 0xc01d41ffc0?}, 0xc0101d5240?, 0x496137?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d470870?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x429ba58, 0xc01d470870}, {0x40cf380, 0xc01d41ffc0}, 0xc01d464280, 0xc01d464320)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d470870?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x429ba58, 0xc01d4707e0}, {0x40cf380, 0xc01d41ffc0}, 0x1?, 0xc01d464340)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/useragent/useragent.go:38 +0xf4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4707e0?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x429ba58, 0xc01d470750}, {0x40cf380, 0xc01d41ffc0}, 0x5d08ea0?, 0xc01d464360)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/token/token.go:44 +0x16d
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d470750?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x429ba58, 0xc01d4706c0}, {0x40cf380, 0xc01d41ffc0}, 0x3e64c40?, 0xc01d464380)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4706c0?}, {0x40cf380?, 0xc01d41ffc0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x429ba58, 0xc01d4706c0}, {0x40cf380, 0xc01d41ffc0}, 0xc01ce18a58?, 0x3d9c900?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbe
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler({0x41fb300?, 0xc008e686c0}, {0x429ba58, 0xc01d4706c0}, 0xc01d46c1c0, 0xc008ec6480)
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6186 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d413e60, 0xc008ec6630, 0x5a798a8, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1345 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d413e60, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1722 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.54.0/server.go:966 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.54.0/server.go:964 +0x28a
 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR unary code=Internal end="04/May/2023:13:22:21 +0545" from=tcp://127.0.0.1:48142 pkg=rgrpc service=storage-users start="04/May/2023:13:22:21 +0545" time_ns=5850609 traceid=00000000000000000000000000000000 uri=/cs3.storage.provider.v1beta1.ProviderAPI/PurgeRecycle user-agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/112.0"
2023-05-04T13:22:21+05:45 ERR Internal Server Error code=500 item_path=/ key=f4ca8e4a-1abd-4615-93f3-32ae922d4fe0 name=com.owncloud.web.ocdav reference={"path":".","resource_id":{"space_id":"0c8278e2-1126-4f9e-9e0d-b9d9dc0f1e05","storage_id":"ecabb8de-991a-426d-9a58-82fae8d6bcf4"}} request-id=20286b18-3ca2-4460-8363-2e9fa4e7e282 service=ocdav status={"code":15,"message":"gateway could not call PurgeRecycle:rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference","trace":"00000000000000000000000000000000"} traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR Internal Server Error code=500 item_path=/ key=f5f8b953-cd01-41ea-b18f-fa63fa2f0f1e name=com.owncloud.web.ocdav reference={"path":".","resource_id":{"space_id":"0c8278e2-1126-4f9e-9e0d-b9d9dc0f1e05","storage_id":"ecabb8de-991a-426d-9a58-82fae8d6bcf4"}} request-id=1e02f1ca-02d5-44cf-bcea-b23700e07df5 service=ocdav status={"code":15,"message":"gateway could not call PurgeRecycle:rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference","trace":"00000000000000000000000000000000"} traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error listing attributes error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/fe/d7/31/cb/-ba20-45f3-a5b1-3226941c1aea: no such file or directory" node=fed731cb-ba20-45f3-a5b1-3226941c1aea pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error listing attributes error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/fe/f5/df/76/-9b15-410a-b4ae-53028490f52c: no such file or directory" node=fef5df76-9b15-410a-b4ae-53028490f52c pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error listing grantees error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/fe/d7/31/cb/-ba20-45f3-a5b1-3226941c1aea: no such file or directory" node="marshaling error: json: unsupported value: encountered a cycle via *node.Node" pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error reading permissions error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/fe/d7/31/cb/-ba20-45f3-a5b1-3226941c1aea: no such file or directory" node=fed731cb-ba20-45f3-a5b1-3226941c1aea pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error listing attributes error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/09/ee/57/dd/-dcff-40f9-aaec-1560938a9454: no such file or directory" node=09ee57dd-dcff-40f9-aaec-1560938a9454 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error listing grantees error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/fe/f5/df/76/-9b15-410a-b4ae-53028490f52c: no such file or directory" node="marshaling error: json: unsupported value: encountered a cycle via *node.Node" pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error reading permissions error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/fe/f5/df/76/-9b15-410a-b4ae-53028490f52c: no such file or directory" node=fef5df76-9b15-410a-b4ae-53028490f52c pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error listing attributes error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/9d/be/5a/82/-f192-40a9-93d4-884c13ca9151: no such file or directory" node=9dbe5a82-f192-40a9-93d4-884c13ca9151 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error listing grantees error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/09/ee/57/dd/-dcff-40f9-aaec-1560938a9454: no such file or directory" node="marshaling error: json: unsupported value: encountered a cycle via *node.Node" pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error reading permissions error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/09/ee/57/dd/-dcff-40f9-aaec-1560938a9454: no such file or directory" node=09ee57dd-dcff-40f9-aaec-1560938a9454 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
goroutine 15401 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
	runtime/debug/stack.go:16 +0x19
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x429ba58, 0xc01d4abe30}, {0x3d9dc80, 0x5a50c60})
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/recovery/recovery.go:48 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x429ba58?, 0xc01d4abe30?}, {0x3d9dc80?, 0x5a50c60?}, 0xc01da9dc10?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x3d9dc80, 0x5a50c60})
	runtime/panic.go:884 +0x213
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Permissions).AssemblePermissions(0x3a00fd0?, {0x429ba58, 0xc01d4ba150}, 0xc01d49eb40)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/node/permissions.go:149 +0x64c
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.Permissions.AssemblePermissions(...)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/spacepermissions.go:39
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).PurgeRecycleItem(0xc00049d0a0, {0x429ba58, 0xc01d4ba150}, 0xc01d4ba120?, {0xc01d485950, 0x24}, {0x3a00fd0?, 0x3a08aa0?})
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/recycle.go:349 +0x174
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).PurgeRecycle(0xc008e686c0, {0x429ba58?, 0xc01d4ba150?}, 0xc01d4b6840)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/services/storageprovider/storageprovider.go:952 +0x1af
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler.func1({0x429ba58, 0xc01d4ba150}, {0x40cf380?, 0xc01d4b6840})
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6184 +0x7b
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x429ba58, 0xc01d4ba090}, {0x40cf380, 0xc01d4b6840}, 0xc01c2a7f40, 0xc01d4b3248)
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.36.4/interceptor.go:341 +0x437
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4ba090?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/eventsmiddleware.NewUnary.func1({0x429ba58, 0xc01d4ba060}, {0x40cf380?, 0xc01d4b6840?}, 0x30?, 0xc01c2a7f60)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/eventsmiddleware/events.go:72 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4ba060?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/prometheus.interceptorFromConfig.func1({0x429ba58?, 0xc01d4ba060?}, {0x40cf380?, 0xc01d4b6840?}, 0x3c73de0?, 0x3a08b01?)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/prometheus/prometheus.go:72 +0x43
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4ba060?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func2({0x429ba58, 0xc01d4abe30}, {0x40cf380, 0xc01d4b6840}, 0xc01c2a7f40, 0xc01c2a7fa0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/auth/auth.go:160 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4abe30?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x429ba58?, 0xc01d4abe30?}, {0x40cf380?, 0xc01d4b6840?}, 0xc015feb240?, 0x496137?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4abe30?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x429ba58, 0xc01d4abe30}, {0x40cf380, 0xc01d4b6840}, 0xc01c2a7f40, 0xc01c2a7fe0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4abe30?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x429ba58, 0xc01d4abda0}, {0x40cf380, 0xc01d4b6840}, 0x1?, 0xc01d4b8000)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/useragent/useragent.go:38 +0xf4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4abda0?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x429ba58, 0xc01d4abd10}, {0x40cf380, 0xc01d4b6840}, 0x5d08ea0?, 0xc01d4b8020)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/token/token.go:44 +0x16d
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4abd10?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x429ba58, 0xc01d4abc80}, {0x40cf380, 0xc01d4b6840}, 0x3e64c40?, 0xc01d4b8040)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4abc80?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x429ba58, 0xc01d4abc80}, {0x40cf380, 0xc01d4b6840}, 0xc01101ea58?, 0x3d9c900?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbe
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler({0x41fb300?, 0xc008e686c0}, {0x429ba58, 0xc01d4abc80}, 0xc01d4b1730, 0xc008ec6480)
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6186 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d487320, 0xc008ec6630, 0x5a798a8, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1345 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d487320, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1722 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.54.0/server.go:966 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.54.0/server.go:964 +0x28a
2023-05-04T13:22:21+05:45 ERR error listing grantees error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/9d/be/5a/82/-f192-40a9-93d4-884c13ca9151: no such file or directory" node="marshaling error: json: unsupported value: encountered a cycle via *node.Node" pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error reading permissions error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/9d/be/5a/82/-f192-40a9-93d4-884c13ca9151: no such file or directory" node=9dbe5a82-f192-40a9-93d4-884c13ca9151 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR runtime error: invalid memory address or nil pointer dereference; stack: goroutine 15401 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x429ba58, 0xc01d4abe30}, {0x3d9dc80, 0x5a50c60})
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/recovery/recovery.go:50 +0x5b
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x429ba58?, 0xc01d4abe30?}, {0x3d9dc80?, 0x5a50c60?}, 0xc01da9dc10?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x3d9dc80, 0x5a50c60})
	runtime/panic.go:884 +0x213
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Permissions).AssemblePermissions(0x3a00fd0?, {0x429ba58, 0xc01d4ba150}, 0xc01d49eb40)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/node/permissions.go:149 +0x64c
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.Permissions.AssemblePermissions(...)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/spacepermissions.go:39
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).PurgeRecycleItem(0xc00049d0a0, {0x429ba58, 0xc01d4ba150}, 0xc01d4ba120?, {0xc01d485950, 0x24}, {0x3a00fd0?, 0x3a08aa0?})
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/recycle.go:349 +0x174
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).PurgeRecycle(0xc008e686c0, {0x429ba58?, 0xc01d4ba150?}, 0xc01d4b6840)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/services/storageprovider/storageprovider.go:952 +0x1af
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler.func1({0x429ba58, 0xc01d4ba150}, {0x40cf380?, 0xc01d4b6840})
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6184 +0x7b
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x429ba58, 0xc01d4ba090}, {0x40cf380, 0xc01d4b6840}, 0xc01c2a7f40, 0xc01d4b3248)
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.36.4/interceptor.go:341 +0x437
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4ba090?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/eventsmiddleware.NewUnary.func1({0x429ba58, 0xc01d4ba060}, {0x40cf380?, 0xc01d4b6840?}, 0x30?, 0xc01c2a7f60)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/eventsmiddleware/events.go:72 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4ba060?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/prometheus.interceptorFromConfig.func1({0x429ba58?, 0xc01d4ba060?}, {0x40cf380?, 0xc01d4b6840?}, 0x3c73de0?, 0x3a08b01?)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/prometheus/prometheus.go:72 +0x43
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4ba060?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func2({0x429ba58, 0xc01d4abe30}, {0x40cf380, 0xc01d4b6840}, 0xc01c2a7f40, 0xc01c2a7fa0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/auth/auth.go:160 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4abe30?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x429ba58?, 0xc01d4abe30?}, {0x40cf380?, 0xc01d4b6840?}, 0xc015feb240?, 0x496137?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4abe30?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x429ba58, 0xc01d4abe30}, {0x40cf380, 0xc01d4b6840}, 0xc01c2a7f40, 0xc01c2a7fe0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4abe30?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x429ba58, 0xc01d4abda0}, {0x40cf380, 0xc01d4b6840}, 0x1?, 0xc01d4b8000)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/useragent/useragent.go:38 +0xf4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4abda0?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x429ba58, 0xc01d4abd10}, {0x40cf380, 0xc01d4b6840}, 0x5d08ea0?, 0xc01d4b8020)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/token/token.go:44 +0x16d
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4abd10?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x429ba58, 0xc01d4abc80}, {0x40cf380, 0xc01d4b6840}, 0x3e64c40?, 0xc01d4b8040)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4abc80?}, {0x40cf380?, 0xc01d4b6840?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x429ba58, 0xc01d4abc80}, {0x40cf380, 0xc01d4b6840}, 0xc01101ea58?, 0x3d9c900?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbe
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler({0x41fb300?, 0xc008e686c0}, {0x429ba58, 0xc01d4abc80}, 0xc01d4b1730, 0xc008ec6480)
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6186 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d487320, 0xc008ec6630, 0x5a798a8, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1345 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d487320, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1722 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.54.0/server.go:966 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.54.0/server.go:964 +0x28a
 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR unary code=Internal end="04/May/2023:13:22:21 +0545" from=tcp://127.0.0.1:48142 pkg=rgrpc service=storage-users start="04/May/2023:13:22:21 +0545" time_ns=4512948 traceid=00000000000000000000000000000000 uri=/cs3.storage.provider.v1beta1.ProviderAPI/PurgeRecycle user-agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/112.0"
goroutine 15305 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
	runtime/debug/stack.go:16 +0x19
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x429ba58, 0xc01d0eddd0}, {0x3d9dc80, 0x5a50c60})
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/recovery/recovery.go:48 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x429ba58?, 0xc01d0eddd0?}, {0x3d9dc80?, 0x5a50c60?}, 0xc01e8d7c10?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x3d9dc80, 0x5a50c60})
	runtime/panic.go:884 +0x213
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Permissions).AssemblePermissions(0x3a00fd0?, {0x429ba58, 0xc01d4ce0f0}, 0xc01d0e90e0)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/node/permissions.go:149 +0x64c
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.Permissions.AssemblePermissions(...)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/spacepermissions.go:39
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).PurgeRecycleItem(0xc00049d0a0, {0x429ba58, 0xc01d4ce0f0}, 0xc01d4ce0c0?, {0xc01d0b9800, 0x24}, {0x3a00fd0?, 0x3a08aa0?})
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/recycle.go:349 +0x174
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).PurgeRecycle(0xc008e686c0, {0x429ba58?, 0xc01d4ce0f0?}, 0xc01d0eeb80)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/services/storageprovider/storageprovider.go:952 +0x1af
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler.func1({0x429ba58, 0xc01d4ce0f0}, {0x40cf380?, 0xc01d0eeb80})
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6184 +0x7b
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x429ba58, 0xc01d4ce030}, {0x40cf380, 0xc01d0eeb80}, 0xc01d0a3080, 0xc01d0fc108)
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.36.4/interceptor.go:341 +0x437
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4ce030?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/eventsmiddleware.NewUnary.func1({0x429ba58, 0xc01d4ce000}, {0x40cf380?, 0xc01d0eeb80?}, 0x30?, 0xc01d0a30a0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/eventsmiddleware/events.go:72 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4ce000?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/prometheus.interceptorFromConfig.func1({0x429ba58?, 0xc01d4ce000?}, {0x40cf380?, 0xc01d0eeb80?}, 0x3c73de0?, 0x3a08b01?)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/prometheus/prometheus.go:72 +0x43
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4ce000?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func2({0x429ba58, 0xc01d0eddd0}, {0x40cf380, 0xc01d0eeb80}, 0xc01d0a3080, 0xc01d0a30e0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/auth/auth.go:160 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d0eddd0?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x429ba58?, 0xc01d0eddd0?}, {0x40cf380?, 0xc01d0eeb80?}, 0xc0139f1240?, 0x496137?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d0eddd0?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x429ba58, 0xc01d0eddd0}, {0x40cf380, 0xc01d0eeb80}, 0xc01d0a3080, 0xc01d0a3120)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d0eddd0?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x429ba58, 0xc01d0edd40}, {0x40cf380, 0xc01d0eeb80}, 0x1?, 0xc01d0a3140)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/useragent/useragent.go:38 +0xf4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d0edd40?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x429ba58, 0xc01d0edcb0}, {0x40cf380, 0xc01d0eeb80}, 0x5d08ea0?, 0xc01d0a3160)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/token/token.go:44 +0x16d
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d0edcb0?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x429ba58, 0xc01d0edc20}, {0x40cf380, 0xc01d0eeb80}, 0x3e64c40?, 0xc01d0a3180)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d0edc20?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x429ba58, 0xc01d0edc20}, {0x40cf380, 0xc01d0eeb80}, 0xc010425a58?, 0x3d9c900?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbe
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler({0x41fb300?, 0xc008e686c0}, {0x429ba58, 0xc01d0edc20}, 0xc01d094cb0, 0xc008ec6480)
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6186 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d0abd40, 0xc008ec6630, 0x5a798a8, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1345 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d0abd40, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1722 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.54.0/server.go:966 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.54.0/server.go:964 +0x28a
2023-05-04T13:22:21+05:45 ERR runtime error: invalid memory address or nil pointer dereference; stack: goroutine 15305 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x429ba58, 0xc01d0eddd0}, {0x3d9dc80, 0x5a50c60})
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/recovery/recovery.go:50 +0x5b
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x429ba58?, 0xc01d0eddd0?}, {0x3d9dc80?, 0x5a50c60?}, 0xc01e8d7c10?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x3d9dc80, 0x5a50c60})
	runtime/panic.go:884 +0x213
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Permissions).AssemblePermissions(0x3a00fd0?, {0x429ba58, 0xc01d4ce0f0}, 0xc01d0e90e0)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/node/permissions.go:149 +0x64c
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.Permissions.AssemblePermissions(...)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/spacepermissions.go:39
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).PurgeRecycleItem(0xc00049d0a0, {0x429ba58, 0xc01d4ce0f0}, 0xc01d4ce0c0?, {0xc01d0b9800, 0x24}, {0x3a00fd0?, 0x3a08aa0?})
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/recycle.go:349 +0x174
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).PurgeRecycle(0xc008e686c0, {0x429ba58?, 0xc01d4ce0f0?}, 0xc01d0eeb80)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/services/storageprovider/storageprovider.go:952 +0x1af
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler.func1({0x429ba58, 0xc01d4ce0f0}, {0x40cf380?, 0xc01d0eeb80})
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6184 +0x7b
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x429ba58, 0xc01d4ce030}, {0x40cf380, 0xc01d0eeb80}, 0xc01d0a3080, 0xc01d0fc108)
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.36.4/interceptor.go:341 +0x437
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4ce030?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/eventsmiddleware.NewUnary.func1({0x429ba58, 0xc01d4ce000}, {0x40cf380?, 0xc01d0eeb80?}, 0x30?, 0xc01d0a30a0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/eventsmiddleware/events.go:72 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4ce000?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/prometheus.interceptorFromConfig.func1({0x429ba58?, 0xc01d4ce000?}, {0x40cf380?, 0xc01d0eeb80?}, 0x3c73de0?, 0x3a08b01?)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/prometheus/prometheus.go:72 +0x43
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d4ce000?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func2({0x429ba58, 0xc01d0eddd0}, {0x40cf380, 0xc01d0eeb80}, 0xc01d0a3080, 0xc01d0a30e0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/auth/auth.go:160 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d0eddd0?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x429ba58?, 0xc01d0eddd0?}, {0x40cf380?, 0xc01d0eeb80?}, 0xc0139f1240?, 0x496137?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d0eddd0?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x429ba58, 0xc01d0eddd0}, {0x40cf380, 0xc01d0eeb80}, 0xc01d0a3080, 0xc01d0a3120)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d0eddd0?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x429ba58, 0xc01d0edd40}, {0x40cf380, 0xc01d0eeb80}, 0x1?, 0xc01d0a3140)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/useragent/useragent.go:38 +0xf4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d0edd40?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x429ba58, 0xc01d0edcb0}, {0x40cf380, 0xc01d0eeb80}, 0x5d08ea0?, 0xc01d0a3160)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/token/token.go:44 +0x16d
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d0edcb0?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x429ba58, 0xc01d0edc20}, {0x40cf380, 0xc01d0eeb80}, 0x3e64c40?, 0xc01d0a3180)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d0edc20?}, {0x40cf380?, 0xc01d0eeb80?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x429ba58, 0xc01d0edc20}, {0x40cf380, 0xc01d0eeb80}, 0xc010425a58?, 0x3d9c900?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbe
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler({0x41fb300?, 0xc008e686c0}, {0x429ba58, 0xc01d0edc20}, 0xc01d094cb0, 0xc008ec6480)
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6186 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d0abd40, 0xc008ec6630, 0x5a798a8, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1345 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d0abd40, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1722 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.54.0/server.go:966 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.54.0/server.go:964 +0x28a
 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR unary code=Internal end="04/May/2023:13:22:21 +0545" from=tcp://127.0.0.1:48142 pkg=rgrpc service=storage-users start="04/May/2023:13:22:21 +0545" time_ns=3476502 traceid=00000000000000000000000000000000 uri=/cs3.storage.provider.v1beta1.ProviderAPI/PurgeRecycle user-agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/112.0"
2023-05-04T13:22:21+05:45 ERR Internal Server Error code=500 item_path=/ key=fed731cb-ba20-45f3-a5b1-3226941c1aea name=com.owncloud.web.ocdav reference={"path":".","resource_id":{"space_id":"0c8278e2-1126-4f9e-9e0d-b9d9dc0f1e05","storage_id":"ecabb8de-991a-426d-9a58-82fae8d6bcf4"}} request-id=81a6adbf-ceee-442f-87f4-2d2ced79a896 service=ocdav status={"code":15,"message":"gateway could not call PurgeRecycle:rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference","trace":"00000000000000000000000000000000"} traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR Internal Server Error code=500 item_path=/ key=fef5df76-9b15-410a-b4ae-53028490f52c name=com.owncloud.web.ocdav reference={"path":".","resource_id":{"space_id":"0c8278e2-1126-4f9e-9e0d-b9d9dc0f1e05","storage_id":"ecabb8de-991a-426d-9a58-82fae8d6bcf4"}} request-id=fe1e48c3-0803-41ea-b501-c85e71d62f97 service=ocdav status={"code":15,"message":"gateway could not call PurgeRecycle:rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference","trace":"00000000000000000000000000000000"} traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error listing attributes error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/e3/b7/ab/65/-bcd3-4b0f-ae66-64642c2d76d3: no such file or directory" node=e3b7ab65-bcd3-4b0f-ae66-64642c2d76d3 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error listing grantees error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/e3/b7/ab/65/-bcd3-4b0f-ae66-64642c2d76d3: no such file or directory" node="marshaling error: json: unsupported value: encountered a cycle via *node.Node" pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error reading permissions error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/e3/b7/ab/65/-bcd3-4b0f-ae66-64642c2d76d3: no such file or directory" node=e3b7ab65-bcd3-4b0f-ae66-64642c2d76d3 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error listing attributes error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/9d/be/5a/82/-f192-40a9-93d4-884c13ca9151: no such file or directory" node=9dbe5a82-f192-40a9-93d4-884c13ca9151 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error listing grantees error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/9d/be/5a/82/-f192-40a9-93d4-884c13ca9151: no such file or directory" node="marshaling error: json: unsupported value: encountered a cycle via *node.Node" pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR error reading permissions error="stat /home/artur/.ocis/storage/users/spaces/0c/8278e2-1126-4f9e-9e0d-b9d9dc0f1e05/nodes/9d/be/5a/82/-f192-40a9-93d4-884c13ca9151: no such file or directory" node=9dbe5a82-f192-40a9-93d4-884c13ca9151 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
goroutine 15308 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
	runtime/debug/stack.go:16 +0x19
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x429ba58, 0xc01d929650}, {0x3d9dc80, 0x5a50c60})
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/recovery/recovery.go:48 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x429ba58?, 0xc01d929650?}, {0x3d9dc80?, 0x5a50c60?}, 0xc01e8d7c10?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x3d9dc80, 0x5a50c60})
	runtime/panic.go:884 +0x213
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Permissions).AssemblePermissions(0x3a00fd0?, {0x429ba58, 0xc01d929950}, 0xc01d933cb0)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/node/permissions.go:149 +0x64c
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.Permissions.AssemblePermissions(...)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/spacepermissions.go:39
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).PurgeRecycleItem(0xc00049d0a0, {0x429ba58, 0xc01d929950}, 0xc01d929920?, {0xc01d813a40, 0x24}, {0x3a00fd0?, 0x3a08aa0?})
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/recycle.go:349 +0x174
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).PurgeRecycle(0xc008e686c0, {0x429ba58?, 0xc01d929950?}, 0xc01d9368c0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/services/storageprovider/storageprovider.go:952 +0x1af
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler.func1({0x429ba58, 0xc01d929950}, {0x40cf380?, 0xc01d9368c0})
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6184 +0x7b
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x429ba58, 0xc01d929890}, {0x40cf380, 0xc01d9368c0}, 0xc01d9202e0, 0xc01d93a3a8)
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.36.4/interceptor.go:341 +0x437
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d929890?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/eventsmiddleware.NewUnary.func1({0x429ba58, 0xc01d929860}, {0x40cf380?, 0xc01d9368c0?}, 0x30?, 0xc01d920300)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/eventsmiddleware/events.go:72 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d929860?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/prometheus.interceptorFromConfig.func1({0x429ba58?, 0xc01d929860?}, {0x40cf380?, 0xc01d9368c0?}, 0x3c73de0?, 0x3a08b01?)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/prometheus/prometheus.go:72 +0x43
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d929860?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func2({0x429ba58, 0xc01d929650}, {0x40cf380, 0xc01d9368c0}, 0xc01d9202e0, 0xc01d920340)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/auth/auth.go:160 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d929650?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x429ba58?, 0xc01d929650?}, {0x40cf380?, 0xc01d9368c0?}, 0xc0139f1240?, 0x496137?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d929650?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x429ba58, 0xc01d929650}, {0x40cf380, 0xc01d9368c0}, 0xc01d9202e0, 0xc01d920380)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d929650?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x429ba58, 0xc01d9295c0}, {0x40cf380, 0xc01d9368c0}, 0x1?, 0xc01d9203a0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/useragent/useragent.go:38 +0xf4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d9295c0?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x429ba58, 0xc01d929530}, {0x40cf380, 0xc01d9368c0}, 0x5d08ea0?, 0xc01d9203c0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/token/token.go:44 +0x16d
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d929530?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x429ba58, 0xc01d9294a0}, {0x40cf380, 0xc01d9368c0}, 0x3e64c40?, 0xc01d9203e0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d9294a0?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x429ba58, 0xc01d9294a0}, {0x40cf380, 0xc01d9368c0}, 0xc010425a58?, 0x3d9c900?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbe
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler({0x41fb300?, 0xc008e686c0}, {0x429ba58, 0xc01d9294a0}, 0xc01d900230, 0xc008ec6480)
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6186 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d91d0e0, 0xc008ec6630, 0x5a798a8, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1345 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d91d0e0, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1722 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.54.0/server.go:966 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.54.0/server.go:964 +0x28a
2023-05-04T13:22:21+05:45 ERR runtime error: invalid memory address or nil pointer dereference; stack: goroutine 15308 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
github.com/cs3org/reva/v2/internal/grpc/interceptors/recovery.recoveryFunc({0x429ba58, 0xc01d929650}, {0x3d9dc80, 0x5a50c60})
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/recovery/recovery.go:50 +0x5b
github.com/grpc-ecosystem/go-grpc-middleware/recovery.recoverFrom({0x429ba58?, 0xc01d929650?}, {0x3d9dc80?, 0x5a50c60?}, 0xc01e8d7c10?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:61 +0x36
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1.1()
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:29 +0x7b
panic({0x3d9dc80, 0x5a50c60})
	runtime/panic.go:884 +0x213
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs/node.(*Permissions).AssemblePermissions(0x3a00fd0?, {0x429ba58, 0xc01d929950}, 0xc01d933cb0)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/node/permissions.go:149 +0x64c
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.Permissions.AssemblePermissions(...)
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/spacepermissions.go:39
github.com/cs3org/reva/v2/pkg/storage/utils/decomposedfs.(*Decomposedfs).PurgeRecycleItem(0xc00049d0a0, {0x429ba58, 0xc01d929950}, 0xc01d929920?, {0xc01d813a40, 0x24}, {0x3a00fd0?, 0x3a08aa0?})
	github.com/cs3org/reva/v2@v2.13.1/pkg/storage/utils/decomposedfs/recycle.go:349 +0x174
github.com/cs3org/reva/v2/internal/grpc/services/storageprovider.(*service).PurgeRecycle(0xc008e686c0, {0x429ba58?, 0xc01d929950?}, 0xc01d9368c0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/services/storageprovider/storageprovider.go:952 +0x1af
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler.func1({0x429ba58, 0xc01d929950}, {0x40cf380?, 0xc01d9368c0})
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6184 +0x7b
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1({0x429ba58, 0xc01d929890}, {0x40cf380, 0xc01d9368c0}, 0xc01d9202e0, 0xc01d93a3a8)
	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.36.4/interceptor.go:341 +0x437
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d929890?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/eventsmiddleware.NewUnary.func1({0x429ba58, 0xc01d929860}, {0x40cf380?, 0xc01d9368c0?}, 0x30?, 0xc01d920300)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/eventsmiddleware/events.go:72 +0xe8
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d929860?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/prometheus.interceptorFromConfig.func1({0x429ba58?, 0xc01d929860?}, {0x40cf380?, 0xc01d9368c0?}, 0x3c73de0?, 0x3a08b01?)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/prometheus/prometheus.go:72 +0x43
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d929860?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/auth.NewUnary.func2({0x429ba58, 0xc01d929650}, {0x40cf380, 0xc01d9368c0}, 0xc01d9202e0, 0xc01d920340)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/auth/auth.go:160 +0xa06
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d929650?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware/recovery.UnaryServerInterceptor.func1({0x429ba58?, 0xc01d929650?}, {0x40cf380?, 0xc01d9368c0?}, 0xc0139f1240?, 0x496137?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/recovery/interceptors.go:33 +0xc4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d929650?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/log.NewUnary.func1({0x429ba58, 0xc01d929650}, {0x40cf380, 0xc01d9368c0}, 0xc01d9202e0, 0xc01d920380)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/log/log.go:39 +0x9a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d929650?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/useragent.NewUnary.func1({0x429ba58, 0xc01d9295c0}, {0x40cf380, 0xc01d9368c0}, 0x1?, 0xc01d9203a0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/useragent/useragent.go:38 +0xf4
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d9295c0?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/token.NewUnary.func1({0x429ba58, 0xc01d929530}, {0x40cf380, 0xc01d9368c0}, 0x5d08ea0?, 0xc01d9203c0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/token/token.go:44 +0x16d
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d929530?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/cs3org/reva/v2/internal/grpc/interceptors/appctx.NewUnary.func1({0x429ba58, 0xc01d9294a0}, {0x40cf380, 0xc01d9368c0}, 0x3e64c40?, 0xc01d9203e0)
	github.com/cs3org/reva/v2@v2.13.1/internal/grpc/interceptors/appctx/appctx.go:51 +0x73c
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1.1.1({0x429ba58?, 0xc01d9294a0?}, {0x40cf380?, 0xc01d9368c0?})
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:25 +0x3a
github.com/grpc-ecosystem/go-grpc-middleware.ChainUnaryServer.func1({0x429ba58, 0xc01d9294a0}, {0x40cf380, 0xc01d9368c0}, 0xc010425a58?, 0x3d9c900?)
	github.com/grpc-ecosystem/go-grpc-middleware@v1.3.0/chain.go:34 +0xbe
github.com/cs3org/go-cs3apis/cs3/storage/provider/v1beta1._ProviderAPI_PurgeRecycle_Handler({0x41fb300?, 0xc008e686c0}, {0x429ba58, 0xc01d9294a0}, 0xc01d900230, 0xc008ec6480)
	github.com/cs3org/go-cs3apis@v0.0.0-20221012090518-ef2996678965/cs3/storage/provider/v1beta1/provider_api.pb.go:6186 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d91d0e0, 0xc008ec6630, 0x5a798a8, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1345 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc001846d20, {0x42ae680, 0xc01c041520}, 0xc01d91d0e0, 0x0)
	google.golang.org/grpc@v1.54.0/server.go:1722 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.2()
	google.golang.org/grpc@v1.54.0/server.go:966 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
	google.golang.org/grpc@v1.54.0/server.go:964 +0x28a
 pkg=rgrpc service=storage-users traceid=00000000000000000000000000000000
2023-05-04T13:22:21+05:45 ERR unary code=Internal end="04/May/2023:13:22:21 +0545" from=tcp://127.0.0.1:48142 pkg=rgrpc service=storage-users start="04/May/2023:13:22:21 +0545" time_ns=5074409 traceid=00000000000000000000000000000000 uri=/cs3.storage.provider.v1beta1.ProviderAPI/PurgeRecycle user-agent="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/112.0"
2023-05-04T13:22:21+05:45 ERR Internal Server Error code=500 item_path=/ key=e3b7ab65-bcd3-4b0f-ae66-64642c2d76d3 name=com.owncloud.web.ocdav reference={"path":".","resource_id":{"space_id":"0c8278e2-1126-4f9e-9e0d-b9d9dc0f1e05","storage_id":"ecabb8de-991a-426d-9a58-82fae8d6bcf4"}} request-id=f2574629-3180-42ff-a8e5-0705d7acef6f service=ocdav status={"code":15,"message":"gateway could not call PurgeRecycle:rpc error: code = Internal desc = runtime error: invalid memory address or nil pointer dereference","trace":"00000000000000000000000000000000"} traceid=00000000000000000000000000000000

@micbar
Copy link
Contributor

micbar commented May 4, 2023

@individual-it

The nil pointer was fixed in cs3org/reva#3841

@dragonchaser
Copy link
Member

@individual-it can you please update your ocis and see if the error persists?

@individual-it
Copy link
Member Author

individual-it commented May 4, 2023

@dragonchaser here how to reproduce the main issue:

  1. create a new space
  2. upload a folder with a lot of files and subfolder on many levels via webUI (you might run into logout when uploading a big folder with a lot of subfolders and files web#8977 but it does not matter if you do or not)
  3. connect the space to the desktop client and sync it (tried with windows client 4.0 rc2)
  4. delete the uploaded folder with the desktop client
  • some content cannot be deleted
  • client shows errors
    image
  1. with the webUI try to upload a file into that space

desktop client log file:
ownCloud.zip

@individual-it
Copy link
Member Author

The issue with reproducing it, is that it does not happen always, but if it does the client is not able to delete some items (maybe depending on the load of the server). Trying to delete one of the broken items results in an error mentioned in #6151 (comment)

The value that the code tries to parse is bigger than 2^64, e.g.

<?xml version="1.0" encoding="UTF-8"?>
<d:error
	xmlns:d="DAV"
	xmlns:s="http://sabredav.org/ns">
	<s:exception></s:exception>
	<s:message>internal error: delete:strconv.ParseInt: parsing &#34;18446744073709546744&#34;: value out of range</s:message>
</d:error>

This value can be found in a messagepack file in the key user.ocis.treesize

here the decoded messagepack file

{
  "user.ocis.name": "c2ltdWxhdG9y",
  "user.ocis.parentid": "NzA1MTNhNDUtZmJlMy00MWMzLWFkNzYtOTA0MTdhZmE0NjY4",
  "user.ocis.propagation": "MQ==",
  "user.ocis.tmp.etag": "",
  "user.ocis.tmtime": "MjAyMy0wNS0wNVQwOToyMDo1Mi40OTA3NVo=",
  "user.ocis.trash.origin": "L2Nvbm5lY3RpcS1zZGstbGluLTQuMS43LTIwMjItMTEtMjEtNTYyYjhhMTk1L3NoYXJlL3NpbXVsYXRvcg==",
  "user.ocis.treesize": "MTg0NDY3NDQwNzM3MDk1NDc4NzA=",
  "user.ocis.type": "Mg=="
}

https://jankaritech.ocloud.de/index.php/s/KvQD8uDa6Ir9l3o

@individual-it
Copy link
Member Author

fixed by cs3org/reva#3845

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Priority:p1-urgent Consider a hotfix release with only that fix Type:Bug
Projects
Archived in project
Development

No branches or pull requests

3 participants