automate switching to -i dandi-api via dandiarchive.org/server-info information

[yarikoptic]

ATM that one reports

{
"version": "1.0.0",
"cli-minimal-version": "0.6.0",
"cli-bad-versions": [],
"services": {
"girder": {
"url": "https://girder.dandiarchive.org"
},
"webui": {
"url": "https://gui.dandiarchive.org"
},
"api": {
"url": "https://publish.dandiarchive.org/api"
},
"jupyterhub": {
"url": "https://hub.dandiarchive.org"
}
}
}

I think we should

• change redirector it to remove entry for "api" since that service is long gone (and boost version to e.g. 1.1.0). that was a bad idea, replaced with
  • make "api" empty (null == None)
• double check that dandi-cli would refuse to interact with the archive if its version is below minimal one specified in that server-info
• adjust dandi-cli so if there is a url for "api", we would use dandi-api instance by default and not the girder one. (alternative - do that if there is no "girder")
• release dandi-cli 0.14.0

Upon migration

• in redirector - adjust minimal version of the client to become 0.14.0 (assuming that is the one which it would be with default switch) and add "url" for the "api" and remove "girder", so released client would automagically switch to use dandi-api instance

any flaw in above plan/logic @satra ?

[jwodder]

@yarikoptic

• double check that dandi-cli would refuse to interact with the archive if its version is below minimal one specified in that server-info
• adjust dandi-cli so if there is a url for "api", we would use dandi-api instance by default and not the girder one. (alternative - do that if there is no "girder")

These steps are or would be performed by get_instance(), which is only used by upload(), delete(), register(), and when parse_dandi_url() receives a *dandiarchive-org.netlify.app or dandi://* URL. The only other commands that can interact with an archive are ls and download, which don't do version checks and don't care where their URL arguments point (though download can take an explicit -i option to set the instance from which to download a locally-mirrored Dandiset; as this behavior only occurs when an instance name is explicitly given, there is no default instance to change for download).

[yarikoptic]

Well, ls and download could do a version check since they are to interact with the dandi-api and issue at least a WARNING that they would proceed but server is known to require newer dandi-cli.

though download can take an explicit -i option to set the instance from which to download a locally-mirrored Dandiset; as this behavior only occurs when an instance name is explicitly given, there is no default instance to change for download.

rright... but in dandiarchive.py we setup regexes to decide between instances:

        # New DANDI API, ATM can be reached only via enable('DJANGO_API')  in
        # browser console
        # https://gui.dandiarchive.org/#/dandiset/000001/0.201104.2302/files
        # TODO: upload something to any dandiset to see what happens when there
        # are files and adjust for how path is provided (if not ?location=)
        (
            re.compile(
                r"(?P<server>(?P<protocol>https?)://"
                r"(?P<hostname>gui-beta-dandiarchive-org\.netlify\.app)/)"
                rf"#/(?P<asset_type>dandiset)/{dandiset_id_grp}"
                r"(/(?P<version>[.0-9]{5,}|draft))?"
                rf"(/files(\?location=(?P<location>.*)?)?)?"
            ),
            {"server_type": "api"},
            "https://gui-beta-dandiarchive-org.netflif.app/#/dandiset"
            "/<dandiset id>[/<version>][/files[?location=<path>]]",
        ),
        # PRs are also on netlify - so above takes precedence. TODO: make more
        # specific?
        (
            re.compile(r"https?://[^/]*dandiarchive-org\.netlify\.app/.*"),
            {"map_instance": "dandi"},
            "https://*dandiarchive-org.netflify.app/...",
        ),
...
        # But for drafts files navigator it is a bit different beast and there
        # could be no versions, only draft
        # https://deploy-preview-341--gui-dandiarchive-org.netlify.app/#/dandiset/000027/draft/files?_id=5f176583f63d62e1dbd06943&_modelType=folder
        (
            re.compile(
                rf"{server_grp}#/(?P<asset_type>dandiset)/{dandiset_id_grp}"
                r"(/(?P<version>draft))?"
                rf"(/files(\?_id={id_grp}(&_modelType=folder)?)?)?"
            ),
            {"server_type": "girder"},
            "https://<server>[/api]#/dandiset/<dandiset id>[/draft]"
            "[/files[?_id=<id>[&_modelType=folder]]]",
        ),

so it would be only for gui-beta-dandiarchive-org.netlify.app instance when we explicitly instruct to handle it as dandi-api.

And URLs for both girder based and new dandi-api based instances are identical (e.g. https://gui.dandiarchive.org/#/dandiset/000003 and https://gui-beta-dandiarchive-org.netlify.app/#/dandiset/000003) now. So we can't automagically tell based on the full dandiset url alone either it is a girder or dandi-api instance :-/ That is why I wondered how we could automate such a switch. I thought to (ab)use redirector for this purpose as to consult on which type of instance it should be: if there is a redirector for the instance (dandi), we can consult it and disregard hard-coded server addresses.

It indeed seems to just over-complicate situation though and we might just adjust regexes and the records in known_instances whenever migration happens for download to correctly handle download of entire dandisets (folders might be ok since URLs differ).

WDYT?

[jwodder]

@yarikoptic

Well, ls and download could do a version check since they are to interact with the dandi-api and issue at least a WARNING that they would proceed but server is known to require newer dandi-cli.

ls and download only interact with the API via explicit URLs. If someone uses a Girder URL, it'll fail once Girder is no longer available, and if they use a dandi-api URL, it'll succeed, so I don't see the need for those two commands to perform version checks, as whether they work is not dependent on having the most recent version of the client.

[yarikoptic]

what will happen in my example above whenever https://gui-beta-dandiarchive-org.netlify.app/#/dandiset/000003 would become https://gui.dandiarchive.org/#/dandiset/000003 and require interaction with dandi-api to get to that dandiset instead of girder? I just want to make sure that for that instance record dandi, we would consult the redirector to get the information that for gui.dandiarchive.org/ we then need to talk to dandi-api and not girder (as hardcoded ATM within dandi instance record).

[jwodder]

@yarikoptic I've updated my PR to use get_instance() to resolve GUI URLs, so https://gui.dandiarchive.org/#/dandiset/000003 will be mapped to the Django API once the redirector starts pointing to it.