The Grand NoCopy Bug

[DonaldTsang]

Main issue: #4558 ("nocopy" option for "add" method doesn't work in a HTTP API call)
Secondary issue: #4224 (nocopy option doesnt work without moving files (defeating part of the purpose))
Third issue: #5815 (Recursive add of large directory fails at 100% (with nocopy and fscache))
Fourth issue: #4318 (nocopy adds files with different hashes than without nocopy)

Software that uses NoCopy (that I am aware of):

• https://github.com/hydrusnetwork/hydrus (File Manager)
• https://github.com/Siderus/Orion (Desktop GUI client)
• https://github.com/RTradeLtd/Temporal (Cloud Storage)

Software that could use NoCopy:

• https://github.com/Kycklingar/PBooru (Image Gallery)
• https://github.com/smugdev/smugboard (Forum)
• https://github.com/daijiale/ipfs-md-wiki (Wiki)

An odd solution referenced by a random forum:

you need to pass the Abspath header with the full file path, in the http multipart headers
AND you need to have the mime type set as
application/octet-stream or whatever
for other more specific mime types it wouldnt work even with the Abspath header
[application] does the right thing and passes the real mime type
which caused the problems
yea it was application/octet-stream thats the one that works
none of this is documented anywhere of course
so even though they provide the nocopy option, they dont tell you how to use it
and that guy that opened the original github issue which you linked to had run into this same problem

Now all we need to do is to reproduce the bug

[eingenito]

@DonaldTsang - Thanks for creating this meta issue. Upon further reading I'm not sure these issues are actually closely related.

#4558 is a documentation (?) issue I guess and I think @magik6k explains why it exists and how to get the desired behavior here: #4558 (comment)

#4224 is a feature request of sorts - although not super clear. There is a potential solution using symlinks, but it doesn't suit everyone's use case.

#4318 is also I guess a documentation issue? The reason for this behavior is summarized here: #4318 (comment) and is discussed more in the issue.

#5184 looks like a bug straight up.

What is the problem that you are running into specifically? Is it #5184? The others are behaviors that are unlikely to change, although we could document them better. I'm tempted to close them and open up a separate issue that just covers documentation of that behavior.

[DonaldTsang]

@eingenito the main issues are #4224 and #4558 in my case for Hydrus, which I hope there should be more documentation and research on the matter.

[radfish]

#5987 : server does not report error if path is invalid

#5988 : add --nocopy does not always add files to filestore. It doesn't add to filestore if blocks exist in blockstore. This leaves the user with duplicated data: in original filesystem files AND in blockstore, despite having used --nocopy.

[Stebalien]

I've created an issue to specifically track what needs to land for this feature to stabilize: #7161.