DJANGO_API: allow for upload of 0-length file

[yarikoptic]

initially detected/reported in dandi/dandi-cli#479 (comment)

Error 400 while sending POST request to http://localhost:8000/api/uploads/initialize/: {"contentSize":["Ensure this value is greater than or equal to 1."]}

although it is an odd use case and would exist solely in a singular blob instance due to deduplication, api backend should allow for an upload of such a file which I expect us to encounter whenever we allow uploads of non-nwb files. As such it is not a priority but should be done/fixed.

[dchiquito]

This will require adjusting the part size algorithm first. Currently it does not yield any parts when the requested size is 0. It should instead yield one part of size 0.

[yarikoptic]

sounds like a way to go forward and then (for #160) to update dandi-cli implementation since ATM we produce

$> dandi digest /tmp/zero
/tmp/zero: d41d8cd98f00b204e9800998ecf8427e-0

thus reflecting the fact that there is no parts (-0), so may be worth checking an alternative -- if there is no parts may be complete url be obtained and posted to with no parts?

[yarikoptic]

the issue persist (although not hit for any real use case yet):

2021-04-15T09:24:08-0400 [ERROR   ] dandi 19632:140032822732544 Error 400 while sending POST request to https://api.dandiarchive.org/api/uploads/initialize/: {"contentSize":["Ensure this value is greater than or equal to 1."]}

and I thought that we already use dandi-cli's DandiETag in the dandi-api , so this might be some additional check on file size (since it is about contentSize)? we can adjust DandiETag to return 1 part upload, but would that automagically resolve this issue on dandi-api side @dchiquito ?

[dchiquito]

Looks like the serializers are still setting the min_value of the part count to 1.

I just tried to set them to 0, but Minio is giving me this error when attempting to complete an upload with no parts in it:

The XML you provided was not well-formed or did not validate against our published schema.

The request body to complete the multipart upload looked like this:

<?xml version="1.0" encoding="UTF-8"?>
<CompleteMultipartUpload xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
</CopmleteMultipartUpload>

That looks fine to me, so I assume that the object store requires at least one part for a multipart upload.

If you really need an upload of size 0, we would have to try tweaking it so that it returns a part of size 0. It's possible that might also fail, in which case we would have to resort to a normal, non-multipart upload just for this one edge case.

[yarikoptic]

I don't mind us to returning 1-part of 0 size. But could you check first, even if manually, that minio/aws would allow for such upload?

[yarikoptic]

of cause we could easily provide a branch with dandi-cli which does that (without merging /releasing) if that would make it easier to test

[waxlamp]

This seems like a lot of work to support an edge case that will maybe never arise in practice. I'd like to either close the issue or de-emphasize it until it crops up as a hard requirement from, e.g., a real user.

Let me know your thoughts, @yarikoptic. In the meantime, I will unassign @dchiquito from working on it.

[waxlamp]

Closing this until it's needed by an actual use case.

[yarikoptic]

me, as a non-real use case again ran into this use-case... arrived to this issue only thanks due to search for Ensure this value is greater than or equal to since details escaped me. I think we should strive for "Equal opportunity" and I demand to be considered "real" ;) anyways -- will see if we could somehow at least give user a better feedback instead of just "400" error