upload (dandi-api): we keep replacing an asset which is already in the archive

[yarikoptic]

drogon:/mnt/backup/dandi/dandisets/000008
$> dandi --version
0.10.0+156.gea86eee.dirty
(dandi-devel) (git)/mnt/backup/dandi/dandisets/000008:[master]
#179 !451 [0].....................................:Tue Feb 16 13:08:27:.
drogon:/mnt/backup/dandi/dandisets/000008
$> DANDI_DEVEL=1 dandi --pdb upload -i dandi-api --validation ignore sub-mouse-AAYYT/
2021-02-16 13:08:33,439 [    INFO] Found 3 files to consider
PATH                                                                                                            SIZE       ERRORS     UPLOAD STATUS                MESSAGE
sub-mouse-AAYYT/sub-mouse-AAYYT_ses-20180420-sample-2_slice-20180420-slice-2_cell-20180420-sample-2_icephys.nwb 9.3 MB        4              extracting metadata          
sub-mouse-AAYYT/sub-mouse-AAYYT_ses-20180420-sample-3_slice-20180420-slice-3_cell-20180420-sample-3_icephys.nwb 9.3 MB        4              extracting metadata          
sub-mouse-AAYYT/sub-mouse-AAYYT_ses-20180420-sample-4_slice-20180420-slice-4_cell-20180420-sample-4_icephys.nwb 9.5 MB                       digesting                    
sub-mouse-AAYYT/sub-mouse-AAYYT_ses-20180420-sample-2_slice-20180420-slice-2_cell-20180420-sample-2_icephys.nwb 9.3 MB        4              done                         
sub-mouse-AAYYT/sub-mouse-AAYYT_ses-20180420-sample-3_slice-20180420-slice-3_cell-20180420-sample-3_icephys.nwb 9.3 MB        4              done                         
sub-mouse-AAYYT/sub-mouse-AAYYT_ses-20180420-sample-4_slice-20180420-slice-4_cell-20180420-sample-4_icephys.nwb 9.5 MB        4              done                         
Summary:                                                                                                        28.1 MB 3 with errors        3 done                       
2021-02-16 13:08:36,268 [    INFO] Logs saved in /home/yoh/.cache/dandi-cli/log/20210216180831Z-2938.log
(dandi-devel) (git)/mnt/backup/dandi/dandisets/000008:[master]
#180 !452 [0].....................................:Tue Feb 16 13:08:36:.
drogon:/mnt/backup/dandi/dandisets/000008
$> DANDI_DEVEL=1 dandi --pdb upload -i dandi-api --validation ignore sub-mouse-AAYYT/
2021-02-16 13:08:39,827 [    INFO] Found 3 files to consider
PATH                                                                                                            SIZE       ERRORS     UPLOAD STATUS                MESSAGE
sub-mouse-AAYYT/sub-mouse-AAYYT_ses-20180420-sample-2_slice-20180420-slice-2_cell-20180420-sample-2_icephys.nwb 9.3 MB        4              extracting metadata          
sub-mouse-AAYYT/sub-mouse-AAYYT_ses-20180420-sample-3_slice-20180420-slice-3_cell-20180420-sample-3_icephys.nwb 9.3 MB        4              extracting metadata          
sub-mouse-AAYYT/sub-mouse-AAYYT_ses-20180420-sample-4_slice-20180420-slice-4_cell-20180420-sample-4_icephys.nwb 9.5 MB                       digesting                    
sub-mouse-AAYYT/sub-mouse-AAYYT_ses-20180420-sample-2_slice-20180420-slice-2_cell-20180420-sample-2_icephys.nwb 9.3 MB        4              done                         
sub-mouse-AAYYT/sub-mouse-AAYYT_ses-20180420-sample-3_slice-20180420-slice-3_cell-20180420-sample-3_icephys.nwb 9.3 MB        4              done                         
sub-mouse-AAYYT/sub-mouse-AAYYT_ses-20180420-sample-4_slice-20180420-slice-4_cell-20180420-sample-4_icephys.nwb 9.5 MB        4              done                         
Summary:                                                                                                        28.1 MB 3 with errors        3 done                       
2021-02-16 13:08:42,146 [    INFO] Logs saved in /home/yoh/.cache/dandi-cli/log/20210216180838Z-2959.log
(dandi-devel) (git)/mnt/backup/dandi/dandisets/000008:[master]
#181 !453 [0].....................................:Tue Feb 16 13:08:42:.
drogon:/mnt/backup/dandi/dandisets/000008
$> cat /home/yoh/.cache/dandi-cli/log/20210216180838Z-2959.log
2021-02-16T13:08:38-0500 [INFO    ] dandi [META] sys.argv = ['/home/yoh/miniconda3/bin/dandi', '--pdb', 'upload', '-i', 'dandi-api', '--validation', 'ignore', 'sub-mouse-AAYYT/']
2021-02-16T13:08:38-0500 [INFO    ] dandi [META] os.getcwd() = /mnt/backup/dandi/dandisets/000008
2021-02-16T13:08:39-0500 [INFO    ] dandi Found 3 files to consider
2021-02-16T13:08:40-0500 [INFO    ] dandi Replacing asset 96867e3d-456d-491a-8469-fdbf03e06e24
2021-02-16T13:08:40-0500 [INFO    ] dandi Replacing asset 2a1c9561-5d6f-4630-8a15-c8304e50dd6a
2021-02-16T13:08:40-0500 [INFO    ] dandi Replacing asset 0dcda3ae-90e3-4603-a01d-3819a2b2377e
2021-02-16T13:08:42-0500 [INFO    ] dandi Logs saved in /home/yoh/.cache/dandi-cli/log/20210216180838Z-2959.log

It should have not replaced asset

• because I have no --existing overwrite or alike
• because it didn't change

Also please move "Replacing asset" to DEBUG level (#358 for a more general solution yet TODO)

[jwodder]

@yarikoptic I believe this is because the default value for the --existing option is "refresh", which is not implemented for the new API; currently, setting existing='refresh' for the new API is the same as setting existing='force'.

[yarikoptic]

oh, that makes sense... and indeed #367 well -- so let's make refresh on new dandi-api behave as "overwrite" which would upload only if changes are detected (without accounting for "fresher" or not)