Mint Zarr assets at start of upload and sync Zarr contents

[jwodder]

Closes #884.
Closes #906.

[codecov]

Codecov Report

Merging #907 (f1b372f) into master (0e6c5ef) will increase coverage by 0.58%.
The diff coverage is 96.66%.

@@            Coverage Diff             @@
##           master     #907      +/-   ##
==========================================
+ Coverage   86.79%   87.38%   +0.58%     
==========================================
  Files          61       61              
  Lines        7424     7453      +29     
==========================================
+ Hits         6444     6513      +69     
+ Misses        980      940      -40     

Flag	Coverage Δ
unittests	87.38% <96.66%> (+0.58%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
dandi/dandiapi.py	88.90% <93.10%> (-0.80%)	⬇️
dandi/files.py	82.53% <96.42%> (+2.70%)	⬆️
dandi/consts.py	100.00% <100.00%> (ø)
dandi/tests/test_upload.py	100.00% <100.00%> (ø)
dandi/tests/skip.py	89.06% <0.00%> (-0.17%)	⬇️
dandi/support/pyout.py	90.00% <0.00%> (-0.17%)	⬇️
dandi/organize.py	82.19% <0.00%> (-0.05%)	⬇️
dandi/cli/tests/test_ls.py	100.00% <0.00%> (ø)
dandi/cli/tests/test_download.py	100.00% <0.00%> (ø)
... and 11 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e6c5ef...f1b372f. Read the comment docs.

[yarikoptic]

looks good. seems in need of test tune up to trigger "delete" logic coverage

[jwodder]

While testing, I hit a 409 error when trying to upload a file to a path that was previously a directory, even after deleting all of the directory's files:

<Error><Code>XMinioObjectExistsAsDirectory</Code><Message>Object name already exists as a directory.</Message><Key>zarr/b1d62558-c3e9-45c8-a395-0a92a8f61c46/changed-type</Key><BucketName>dandi-dandisets</BucketName><Resource>/dandi-dandisets/zarr/b1d62558-c3e9-45c8-a395-0a92a8f61c46/changed-type</Resource><RequestId>16D2C651D13510B0</RequestId><HostId>0e495f50-1b7d-4ffb-bb32-90394f235a56</HostId></Error>

@dchiquito What is the proper way to delete a directory inside a Zarr on the server?

EDIT: Never mind, my mistake, though I'd still be interested in if there's a better way to delete a directory than deleting all its contents individually.

[satra]

I'd still be interested in if there's a better way to delete a directory than deleting all its contents individually.

the only thing i can think of is a dandi api request that launches a task that internally lists all objects with a given prefix, and then deletes all of them. this can take a while though for zarr trees and we have to be careful on what that prefix can be for a request.

[yarikoptic]

what is the "life cycle" of this ._digest? :

• I do not see it just being a lazy one (not assigned below in this function)
• digests of zarr could change on remote/server since zarr files are mutable, so aren't we doomed to query it every time .digest is requested?

[yarikoptic]

oh, the latter is a wrong question since we are talking here about RemoteZarrEntry not an entire Zarr. But still not certain about what should set ._digest?

[jwodder]

_digest is set by _get_subpath() when called by iterdir(), in which case the code is going through a Zarr listing with checksums, so storing the entry's checksum saves a potential HTTP request later. _digest can be unset by the cache_clear() method.

[yarikoptic]

so is it the get_listing below which would set it? this is all makes it not obvious, and worth at least a comment here.

[jwodder]

Comment added.

[yarikoptic]

@jwodder - please resolve freshly came up conflict and let's merge to carry forward

[jwodder]

@yarikoptic Conflict resolved.

[yarikoptic]

ok, still green, let's proceed. thanks @jwodder !