ENH: adding sphinx documentation for the dandi-cli

.gitignore

@@ -10,7 +10,7 @@
 __pycache__/
 build/
 dist/
-docs/source/generated/
+docs/**/generated/
 pip-wheel-metadata/
 sandbox/
 venv/

docs/source/cmdline/delete.rst

@@ -0,0 +1,16 @@
+:program:`dandi delete`
+=======================
+
+::
+
+    dandi [<global options>] delete [<options>] [<paths> ...]
+
+Delete dandisets and assets from the server.
+
+PATH could be a local path or a URL to an asset, directory, or an entire
+dandiset.
+
+Options
+-------
+
+.. option:: --skip-missing

docs/source/cmdline/digest.rst

@@ -0,0 +1,15 @@
+:program:`dandi digest`
+=======================
+
+::
+
+    dandi [<global options>] digest [<options>] [<path> ...]
+
+Calculate file digests
+
+Options
+-------
+
+.. option:: -d, --digest [dandi-etag|md5|sha1|sha256|sha512]
+
+    Digest algorithm to use  [default: dandi-etag]

docs/source/cmdline/download.rst

@@ -0,0 +1,36 @@
+:program:`dandi download`
+=========================
+
+::
+
+    dandi [<global options>] download [<options>] [<url> ...]
+
+Options
+-------
+
+.. option:: -o, --output-dir <dir>
+
+    Directory where to download to (directory must exist).  Files will be
+    downloaded with paths relative to that directory.
+
+.. option:: -e, --existing [error|skip|overwrite|overwrite-different|refresh]
+
+    What to do if a file found existing locally. 'refresh': verify
+    that according to the size and mtime, it is the same file, if not -
+    download and overwrite.
+
+.. option:: -f, --format [pyout|debug]
+
+    Choose the format/frontend for output.
+
+.. option:: -J, --jobs INT
+
+    Number of parallel download jobs.
+
+.. option:: --download [dandiset.yaml,assets,all]
+
+    Comma-separated list of elements to download
+
+.. option:: --sync
+
+    Delete local assets that do not exist on the server

docs/source/cmdline/index.rst

@@ -0,0 +1,8 @@
+**********************
+Command-Line Interface
+**********************
+
+.. toctree::
+    :glob:
+
+    *

docs/source/cmdline/ls.rst

@@ -0,0 +1,50 @@
+:program:`dandi ls`
+===================
+
+::
+
+    dandi [<global options>] ls [<options>] [<path|url> ...]
+
+List .nwb files and dandisets metadata.
+
+Patterns for known setups:
+
+- ``DANDI:<dandiset id>``
+- ``https://dandiarchive.org/...``
+- ``https://identifiers.org/DANDI:<dandiset id>``
+- ``https://<server>[/api]/[#/]dandiset/<dandiset id>[/<version>][/files[?location=<path>]]``
+- ``https://*dandiarchive-org.netflify.app/...``
+- ``https://<server>[/api]/dandisets/<dandiset id>[/versions[/<version>]]``
+- ``https://<server>[/api]/dandisets/<dandiset id>/versions/<version>/assets/<asset id>[/download]``
+- ``https://<server>[/api]/dandisets/<dandiset id>/versions/<version>/assets/?path=<path>``
+- ``dandi://<instance name>/<dandiset id>[@<version>][/<path>]``
+- ``https://<server>/...``
+
+
+Options
+-------
+
+.. option:: -F, --fields <fields>
+
+    Comma-separated list of fields to display.  An empty value to trigger a
+    list of available fields to be printed out
+
+.. option:: -f, --format [auto|pyout|json|json_pp|json_lines|yaml]
+
+    Choose the format/frontend for output. If 'auto' (default), 'pyout' will be
+    used in case of multiple files, and 'yaml' for a single file.
+
+.. option:: -r, --recursive
+
+    Recurse into content of dandisets/directories. Only .nwb files will be
+    considered.
+
+.. option:: -J, --jobs <int>
+
+    Number of parallel download jobs.
+
+.. option:: --metadata [api|all|assets]
+
+.. option:: --schema <version>
+
+    Convert metadata to new schema version

docs/source/cmdline/organize.rst

@@ -0,0 +1,58 @@
+:program:`dandi organize`
+=========================
+
+::
+
+    dandi [<global options>] organize [<options>] [<path> ...]
+
+(Re)organize files according to the metadata.
+
+The purpose of this command is to take advantage of metadata contained in the
+.nwb files to provide datasets with consistently named files, so their naming
+reflects data they contain.
+
+.nwb files are organized into a hierarchy of subfolders one per each "subject",
+e.g. sub-0001 if .nwb file had contained a Subject group with subject_id=0001.
+Each file in a subject-specific subfolder follows the convention::
+
+    sub-<subject_id>[_key-<value>][_mod1+mod2+...].nwb
+
+where following keys are considered if present in the data::
+
+    ses -- session_id
+    tis -- tissue_sample_id
+    slice -- slice_id
+    cell -- cell_id
+
+and ``modX`` are "modalities" as identified based on detected neural data types
+(such as "ecephys", "icephys") per extensions found in nwb-schema definitions:
+https://github.com/NeurodataWithoutBorders/nwb-schema/tree/dev/core
+
+In addition an "obj" key with a value corresponding to crc32 checksum of
+"object_id" is added if aforementioned keys and the list of modalities are
+not sufficient to disambiguate different files.
+
+You can visit https://dandiarchive.org for a growing collection of
+(re)organized dandisets.
+
+Options
+-------
+
+.. options:: -d, --dandiset-path <dir>
+
+    A top directory (local) of the dandiset to organize files under.  If not
+    specified, dandiset current directory is under is assumed.  For 'simulate'
+    mode target dandiset/directory must not exist.
+
+.. option:: --invalid [fail|warn]
+
+    What to do if files without sufficient metadata are encountered.
+
+.. option:: -f, --files-mode [dry|simulate|copy|move|hardlink|symlink|auto]
+
+    If 'dry' - no action is performed, suggested renames are printed.  If
+    'simulate' - hierarchy of empty files at --local-top-path is created.  Note
+    that previous layout should be removed prior this operation.  If 'auto'
+    (default) - whichever of symlink, hardlink, copy is allowed by system.  The
+    other modes (copy, move, symlink, hardlink) define how data files should be
+    made available.

docs/source/cmdline/shell-completion.rst

@@ -0,0 +1,24 @@
+:program:`dandi shell-completion`
+=================================
+
+::
+
+    dandi [<global options>] shell-completion [<options>]
+
+Emit shell script for enabling command completion.
+
+The output of this command should be "sourced" by bash or zsh to enable command
+completion.
+
+Example::
+
+    $ source <(dandi shell-completion)
+    $ dandi --<PRESS TAB to display available option>
+
+Options
+-------
+
+.. option:: -s, --shell [bash|zsh|fish|auto]
+
+    The shell for which to generate completion code; `auto` (default) attempts
+    autodetection

docs/source/cmdline/upload.rst

@@ -0,0 +1,42 @@
+:program:`dandi upload`
+=======================
+
+::
+
+    dandi [<global options>] upload [<options>] [<path> ...]
+
+Upload dandiset (files) to DANDI archive.
+
+Target dandiset to upload to must already be registered in the archive and
+locally :file:`dandiset.yaml` should exist in :option:`--dandiset-path`.
+
+Local dandiset should pass validation.  For that it should be first organized
+using ``dandi organize`` command.
+
+By default all files in the dandiset (not following directories starting with a
+period) will be considered for the upload.  You can point to specific files you
+would like to validate and have uploaded.
+
+Options
+-------
+
+.. option:: -e, --existing [error|skip|force|overwrite|refresh]
+
+    What to do if a file found existing on the server. 'skip' would skip the
+    file, 'force' - force reupload, 'overwrite' - force upload if either size
+    or modification time differs; 'refresh' (default) - upload only if local
+    modification time is ahead of the remote.
+
+.. option:: -J, --jobs N[:M]
+
+    Number of files to upload in parallel and, optionally, number of upload
+    threads per file
+
+.. option:: --sync
+
+    Delete assets on the server that do not exist locally
+
+.. option:: --validation [require|skip|ignore]
+
+    Data must pass validation before the upload.  Use of this option is highly
+    discouraged.

docs/source/cmdline/validate.rst

@@ -0,0 +1,10 @@
+:program:`dandi validate`
+=========================
+
+::
+
+    dandi [<global options>] validate [<path> ...]
+
+Validate files for NWB (and DANDI) compliance.
+
+Exits with non-0 exit code if any file is not compliant.

docs/source/index.rst

@@ -13,8 +13,8 @@ Archive <https://dandiarchive.org>`_.
    :maxdepth: 2
    :caption: Contents:
 
-   cmdline
-   modref
+   cmdline/index
+   modref/index
 
 
 Indices and tables

docs/source/modref/consts.rst

@@ -0,0 +1,4 @@
+``dandi.consts``
+================
+
+.. automodule:: dandi.consts

docs/source/modref.rst → docs/source/modref/index.rst

@@ -32,8 +32,7 @@ Mid-level user interfaces
 Support functionality
 =====================
 
-.. autosummary::
-   :toctree: generated
+.. toctree::
 
    consts
    utils

docs/source/modref/support.digests.rst

@@ -0,0 +1,4 @@
+``dandi.support.digests``
+=========================
+
+.. automodule:: dandi.support.digests

docs/source/modref/utils.rst

@@ -0,0 +1,4 @@
+``dandi.utils``
+===============
+
+.. automodule:: dandi.utils