Update to use api.dandiarchive.org instead of publish.dandiarchive.org

[jwodder]

publish.dandiarchive.org was retired at some point recently, and now api.dandiarchive.org should be used instead. This change caused several tests to fail; all of them are in dandi/tests/test_download.py and involve the URL https://dandiarchive.org/dandiset/000027/0.200721.2222. For now, I have marked the failing tests with pytest.mark.xfail(reason="publish.dandiarchive.org is gone") so that they will not cause the test suite to fail.

[yarikoptic]

rright. Also looking at https://gui.dandiarchive.org/#/dandiset/000027/draft -- that released version is "gone" as well. So it is the only way (but not sure if entire test need to be marked as xfail if it tests other urls as well -- we could as well just comment out those URLs tests if they are part of a bigger test)

[jwodder]

@yarikoptic Because these tests try to follow a redirect that 404's, the increased number of retries introduced in #762 means that the test suite now takes over two hours to run. Our options include:

• Publish 000027 or another Dandiset so that we have a published version URL to retrieve for these tests
• Change the xfail to skip
• Delete the tests entirely

[yarikoptic]

Thanks for unburing this issue ;) I have now fixed up dandiset metadata, released 000027 (https://gui.dandiarchive.org/#/dandiset/000027/0.210831.2033) and filed some UX issues for dandiarchive ;)

Also doing some code archaeology (would be nice to have commit messages which have more relevant information): 6d87780 of #734 to fix #727 (retry redirect if 400). So, I think we did mistake there by checking for 404 instead of 400. For 404 we should just spit it out -- "gone is gone".