Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some schema updates #286

Merged
merged 154 commits into from
Dec 1, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
154 commits
Select commit Hold shift + click to select a range
026cb45
Add dandi-publish to local_docker_compose fixture
jwodder Sep 3, 2020
2022f29
Get django image from Docker Hub
jwodder Sep 4, 2020
93f2222
0.6.3 CHANGELOG
yarikoptic Sep 4, 2020
1434966
Merge branch 'master' of https://github.com/dandi/dandi-cli
yarikoptic Sep 4, 2020
74f1410
Abandon 0.6.3 (will remove from pypi and git tag)
yarikoptic Sep 4, 2020
80e253b
RF: move+RF code for navigating dandi archive url from download into …
yarikoptic Sep 2, 2020
0abc439
RF: move conditioning of dandiset metadata into dandiapi
yarikoptic Sep 4, 2020
d091704
Merge pull request #232 from dandi/gh-164b
yarikoptic Sep 8, 2020
0b11830
ENH: find_files exclude_dotdirs option (True by default)
yarikoptic Sep 9, 2020
1d137be
ENH: ls - be able to list dandiarchive urls along with paths
yarikoptic Sep 9, 2020
a03906c
ENH: ls - json formatter -- sort_keys=True
yarikoptic Sep 9, 2020
952fc6c
RF: place girder id for file into a "girder.id"; bring sha256 into th…
yarikoptic Sep 9, 2020
bafe180
Update dandi/cli/cmd_ls.py
yarikoptic Sep 10, 2020
8bc847f
Fix spelling
jwodder Sep 10, 2020
419e8de
Merge pull request #233 from dandi/rf-download
jwodder Sep 10, 2020
fc6bd96
Make YAMLFormatter sort keys
jwodder Sep 10, 2020
94ba354
Configure YAMLFormatter flow style
jwodder Sep 14, 2020
9e29cdf
Merge pull request #237 from dandi/gh-235
yarikoptic Sep 14, 2020
3c55425
RF: do not upload dandiset.yaml
yarikoptic Sep 15, 2020
3444ae3
RF+BF: centralize "deduction" of dandiset identifier from the metadata
yarikoptic Sep 15, 2020
77eb4b5
BF: we need to take metadata.dandiset into metadata
yarikoptic Sep 15, 2020
16617d3
RF: TEMP - skip validation of dandiset.yaml + report status validated…
yarikoptic Sep 15, 2020
62e4fe7
ENH: provide original exception message while preparing LockingError …
yarikoptic Sep 15, 2020
51bab5d
ENH: more logging
yarikoptic Sep 15, 2020
8c33ae0
Failing download + upload test
jwodder Sep 15, 2020
4b64f95
RF: register - just log identifier and URL, no need for full printout
yarikoptic Sep 15, 2020
683e1e5
ENH: test_upload_external_download - do call register
yarikoptic Sep 15, 2020
7aa6877
ENH: return back info msg about no dandiset being provided/detected
yarikoptic Sep 16, 2020
928b45b
Merge pull request #239 from dandi/enh-lock2
yarikoptic Sep 16, 2020
5f7d49d
Script for "instantiating" Dandisets from asset store
jwodder Sep 18, 2020
db93509
Populate dandiset.yaml
jwodder Sep 18, 2020
bf5abcf
register command: Always print Dandiset identifier
jwodder Sep 21, 2020
2af6eeb
Merge pull request #242 from dandi/bf-dandiset-meta-assumptions
yarikoptic Sep 21, 2020
05dff2d
Merge pull request #244 from dandi/gh-243
yarikoptic Sep 21, 2020
77014a2
Download files to temporary directory containing checksum, usable for…
jwodder Sep 25, 2020
4bdac70
Resume at dldir.offset
jwodder Sep 25, 2020
42900b9
Include filepath in lock failure error
jwodder Sep 28, 2020
cddb439
Change .download extension to .dandidownload
jwodder Sep 28, 2020
580319d
Edit DownloadDirectory.__exit__
jwodder Sep 28, 2020
49e4575
Unlink writefile before opening with "wb"
jwodder Sep 28, 2020
13d375f
Rename DownloadDirectory files
jwodder Sep 29, 2020
e0561fa
Test DownloadDirectory
jwodder Sep 29, 2020
39a2518
Add debug messages to DownloadDirectory
jwodder Sep 29, 2020
a9a7e87
Handle request resumption with Girder
jwodder Sep 29, 2020
46237b0
Merge pull request #247 from dandi/gh-198
yarikoptic Oct 2, 2020
91360da
Initial draft of script for instantiating Datalad datasets from Dandi…
jwodder Oct 5, 2020
24aabd9
Add logging and custom commit date
jwodder Oct 5, 2020
6736a32
Adjust logging config
jwodder Oct 5, 2020
bec40a7
Fix logging config
jwodder Oct 5, 2020
8bd5229
Typo
jwodder Oct 5, 2020
55ed9c7
Logging and a fix
jwodder Oct 5, 2020
90fff5f
Include versionId in asset URLs
jwodder Oct 6, 2020
bd37895
Fix deletion of deleted assets
jwodder Oct 6, 2020
5a0c5fe
Publishing to GitHub
jwodder Oct 6, 2020
22b5f96
Typo
jwodder Oct 6, 2020
0c3935f
Not all assets have .attrs.mtime
jwodder Oct 6, 2020
d579f96
Die if this fails
jwodder Oct 6, 2020
d80bb29
Redo GitHub credentials handling
jwodder Oct 7, 2020
64fa59e
Handle any possible file layout in dandisets
jwodder Oct 7, 2020
c743d63
Include number of files added/updated/deleted in commit message
jwodder Oct 7, 2020
1f8ae15
Batch add_url_to_file
jwodder Oct 8, 2020
eff38ef
Better way to set Datalad backend
jwodder Oct 8, 2020
8ab6efb
Support specifying the Dandisets to run on
jwodder Oct 8, 2020
a523c81
Copy logs to file in .dandi/logs
jwodder Oct 8, 2020
179f490
Don't trust mtimes
jwodder Oct 8, 2020
b5f9ae0
Improve logging
jwodder Oct 8, 2020
d87f7d2
Fix
jwodder Oct 8, 2020
2cd500d
Fix
jwodder Oct 8, 2020
bd47a37
Only add URLs to files under annex
jwodder Oct 9, 2020
5d5bbcd
ENH: backups2datalad - add python shebang and make executable
yarikoptic Oct 9, 2020
37cd479
ENH: add "tools" install extras with boto3
yarikoptic Oct 9, 2020
b28c37a
Die if girder-assetstore not in assetstore path
jwodder Oct 9, 2020
8c03daf
Add --re-filter option
jwodder Oct 9, 2020
a297195
BF: pass individual path not a list of paths for is_under_annex
yarikoptic Oct 9, 2020
0e6588e
ENH: use batch mode for is_under_annex
yarikoptic Oct 9, 2020
62c19bf
Use Digester to calculate hashes of non-annexed files
jwodder Oct 9, 2020
c06a87c
Record list of assets
jwodder Oct 9, 2020
65e3de4
Fix
jwodder Oct 9, 2020
9182877
Merge pull request #251 from dandi/gh-250
yarikoptic Oct 9, 2020
81cee87
Store complete asset metadata
jwodder Oct 12, 2020
91a2616
Use environment file instead of set-env
jwodder Oct 14, 2020
4ceeace
Quote environment file var
jwodder Oct 14, 2020
49b0234
Merge pull request #254 from dandi/use-env-files
yarikoptic Oct 15, 2020
d9428a2
Merge pull request #253 from dandi/gh-250b
yarikoptic Oct 20, 2020
8f617c8
ENH: establish backup (TODO: made into an option)
Oct 21, 2020
6959990
Update for datalad/datalad#5065
jwodder Oct 21, 2020
b22af35
Add --backup-remote option
jwodder Oct 21, 2020
2ef0cf4
Add --jobs option
jwodder Oct 21, 2020
6fe0beb
Only save logs if there were changes to the repository
jwodder Oct 21, 2020
8c559e3
Error if repo is dirty
jwodder Oct 21, 2020
5f53f10
Save log if an exception occurs
jwodder Oct 21, 2020
671e459
Merge pull request #255 from dandi/enh-backup
yarikoptic Oct 21, 2020
65b870b
Added TODOs to backups2datalad.py to be addressed before it could be …
Oct 22, 2020
1d3fef7
RF: moved backups2datalad.py to https://github.com/dandi/dandisets
Oct 22, 2020
52d8cab
Update CHANGELOG for v0.7.0
jwodder Nov 4, 2020
bffa737
Merge pull request #258 from dandi/chlog-0.7.0
jwodder Nov 4, 2020
2cee06e
Restrict h5py to pre-3.0 in order to not break tests
jwodder Nov 4, 2020
b6635a9
Mark publish.dandiarchive.org tests as xfailing
jwodder Nov 4, 2020
41d8b05
Mention h5py version restriction in CHANGELOG
jwodder Nov 4, 2020
f424a31
Merge pull request #260 from dandi/fixes
jwodder Nov 4, 2020
c14578a
Initial auto setup
jwodder Oct 21, 2020
a0453be
Add PyPI publication to release workflow
jwodder Nov 2, 2020
db33c0d
Update DEVELOPMENT.md
jwodder Nov 2, 2020
e9b08aa
Adjust CHANGELOG formatting
jwodder Nov 4, 2020
3e93250
Use oldest file when race condition causes multiple files per item
jwodder Nov 5, 2020
da60edf
Merge pull request #265 from dandi/gh-264
jwodder Nov 5, 2020
7c8f351
Merge pull request #257 from dandi/gh-252
jwodder Nov 5, 2020
6c870e5
Remove unused link from CHANGELOG.md
jwodder Nov 5, 2020
532d5da
Merge pull request #266 from dandi/minor
jwodder Nov 5, 2020
a2b6789
Update CHANGELOG.md [skip ci]
Nov 5, 2020
634135c
Make keyring lookup more flexible
jwodder Nov 6, 2020
da37ec3
Add comment to pycryptodomex dependency
jwodder Nov 6, 2020
690644d
Move askyesno() to its own module and give it a default option
jwodder Nov 6, 2020
7edc9b8
Basic test of getting an API key via input()
jwodder Nov 6, 2020
a6fadc8
Test keyring_lookup() when backend is set via env var
jwodder Nov 6, 2020
6b4abf9
Test asking for an API key twice via input()
jwodder Nov 6, 2020
323acbe
Test keyring_lookup() when backend is set via config file
jwodder Nov 6, 2020
043c758
Test that keyring backends specified with env vars take precedence ov…
jwodder Nov 6, 2020
0dc06fc
Test askyesno()
jwodder Nov 6, 2020
be79fc4
More keyring_lookup() tests
jwodder Nov 6, 2020
c74b87c
Test the rest of keyring_lookup()
jwodder Nov 6, 2020
7e83e96
Merge pull request #267 from dandi/gh-263
yarikoptic Nov 9, 2020
dfe97d3
Copy files with `cp --reflink=auto` where supported
jwodder Nov 9, 2020
bcbfd85
Merge pull request #269 from dandi/gh-268
yarikoptic Nov 9, 2020
bfec91c
Include item path in "Multiple files found for item" message
jwodder Nov 10, 2020
3e5d465
Add healthchecks for the Postgres and minio Docker containers
jwodder Nov 10, 2020
1cad37d
Merge pull request #272 from dandi/docker-healthchecks
yarikoptic Nov 12, 2020
43be589
Merge pull request #271 from dandi/gh-270
yarikoptic Nov 12, 2020
69cfc1e
Support h5py 3.0
jwodder Nov 19, 2020
ad200a8
[#275] Ignore numpy API warning
jwodder Nov 19, 2020
a50f6d7
Workflow for publishing model schemata to dandi/schema
jwodder Nov 19, 2020
87e6e2a
Add comment to numpy API warning ignore setting
jwodder Nov 19, 2020
26c2e33
Adjust workflow path triggers
jwodder Nov 19, 2020
32d49e7
Tweak pubschemata commit message
jwodder Nov 19, 2020
343ee3b
Merge pull request #275 from dandi/gh-259
yarikoptic Nov 19, 2020
777e5af
Update CHANGELOG.md [skip ci]
Nov 19, 2020
73fc6fb
DOC: strip away duplicate with the handbook information
yarikoptic Nov 21, 2020
97292b7
Merge pull request #279 from dandi/Doc-todo
yarikoptic Nov 23, 2020
39c62bc
Merge pull request #276 from dandi/gh-273
yarikoptic Nov 23, 2020
142e21e
BF: add h5py.__version__ into the list of tokens for caching
yarikoptic Nov 25, 2020
16ff607
ENH: basic duecredit support
yarikoptic Nov 25, 2020
194ce14
Merge pull request #285 from dandi/enh-duecredit
yarikoptic Nov 25, 2020
f110465
Merge remote-tracking branch 'upstream/master' into nwb2asset
satra Nov 30, 2020
0155c3f
updates to schema
satra Nov 30, 2020
ea3f354
updated just models
satra Nov 30, 2020
0b92768
Merge pull request #287 from satra/model-updates
yarikoptic Nov 30, 2020
bb1a8f3
Fix publish-schemata workflow
jwodder Dec 1, 2020
55b7dd1
change from disease to disorder
satra Dec 1, 2020
8a54472
Merge pull request #290 from dandi/gh-289
satra Dec 1, 2020
d6ec80d
Merge pull request #291 from satra/model-updates
satra Dec 1, 2020
1974553
Merge pull request #284 from dandi/bf-h5py-ver-token
yarikoptic Dec 1, 2020
9d0d2bd
Update CHANGELOG.md [skip ci]
Dec 1, 2020
bbee349
Merge remote-tracking branch 'upstream/master' into nwb2asset
satra Dec 1, 2020
7be857f
tst: some updates based on model
satra Dec 1, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .autorc
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{
"onlyPublishWithReleaseLabel": true,
"baseBranch": "master",
"author": "auto <auto@nil>",
"noVersionPrefix": true,
"plugins": ["git-tag"]
}
51 changes: 51 additions & 0 deletions .github/workflows/publish-schemata.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
name: Publish model schemata

on:
# ATM, this is the closest trigger to a PR merging
push:
branches:
- master
paths:
- dandi/model*.py
- tools/pubschemata.py

jobs:
publish:
runs-on: ubuntu-latest
steps:
- name: Checkout this repository
uses: actions/checkout@v2
with:
fetch-depth: 0 # Need history for `git describe`
path: dandi

- name: Set up Python
uses: actions/setup-python@v1
with:
python-version: '3.8'

- name: Install dandi
run: python -m pip install .
working-directory: dandi

- name: Checkout dandi/schema
uses: actions/checkout@v2
with:
repository: dandi/schema
path: schema
token: ${{ secrets.DANDI_GITHUB_TOKEN }}

- name: Generate model schemata
run: python dandi/tools/pubschemata.py schema/releases

- name: Commit changes
run: |
git add releases
if ! git diff --quiet --cached
then git commit -m "Publish model schemata as of dandi-cli $(git -C ../dandi describe --tags)"
git push
else echo "No changes to commit"
fi
working-directory: schema

# vim:set sts=2:
78 changes: 78 additions & 0 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
name: Auto-release on PR merge

on:
# ATM, this is the closest trigger to a PR merging
push:
branches:
- master

jobs:
auto-release:
runs-on: ubuntu-latest
if: "!contains(github.event.head_commit.message, 'ci skip') && !contains(github.event.head_commit.message, 'skip ci')"
outputs:
auto-version: ${{ steps.auto-version.outputs.version }}
steps:
- name: Checkout source
uses: actions/checkout@v2
with:
fetch-depth: 0

- name: Download latest auto
run: |
auto_download_url="$(curl -fsSL https://github.com/gitapi/repos/intuit/auto/releases/latest | jq -r '.assets[] | select(.name == "auto-linux.gz") | .browser_download_url')"
wget -O- "$auto_download_url" | gunzip > ~/auto
chmod a+x ~/auto

- name: Check whether a release is due
id: auto-version
run: |
version="$(~/auto version)"
echo "::set-output name=version::$version"
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

- name: Create release
run: ~/auto shipit
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

pypi:
runs-on: ubuntu-latest
needs: auto-release
if: "needs.auto-release.outputs.auto-version != ''"
steps:
# By default, actions/checkout will checkout the commit that that was
# pushed to master and triggered the workflow, but this does not include
# the commit & tag created by `auto`. In order to get that, we need to
# look up the tag for the latest release.
- name: Get tag of latest release
id: latest-release
run: |
latest_tag="$(curl -fsSL https://github.com/gitapi/repos/$GITHUB_REPOSITORY/releases/latest | jq -r .tag_name)"
echo "::set-output name=tag::$latest_tag"

- name: Checkout source
uses: actions/checkout@v2
with:
fetch-depth: 0
ref: ${{ steps.latest-release.outputs.tag }}

- name: Set up Python
uses: actions/setup-python@v1
with:
python-version: 3.6

- name: Install build & twine
run: python -m pip install build twine

- name: Build
run: python -m build

- name: Upload
run: twine upload dist/*
env:
TWINE_USERNAME: __token__
TWINE_PASSWORD: ${{ secrets.PYPI_TOKEN }}

# vim:set sts=2:
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
python -m pip install --upgrade pip
pip install ".[test]"
- name: Set environment variable when using Python 3.6 to cover more code
run: echo "::set-env name=DANDI_LOG_GIRDER::1"
run: echo DANDI_LOG_GIRDER=1 >> "$GITHUB_ENV"
if: matrix.python == '3.6'
- name: ${{ matrix.module }} tests
run: |
Expand Down
Loading