adding to_datacite method #596

djarecka · 2021-04-26T17:05:49Z

this is a continuation of #516 (I had issues with using the secrets, so testing this from the dandi branch)

Adding a new method to_dandicite that creates a datacite dictionary from the metadata object

…datacite approved list); adding Sponsors to fundingReferences; adding automatic fix for relatedIdentifiers types

Datacite tmp

codecov · 2021-04-26T17:10:00Z

Codecov Report

Merging #596 (2ae1c02) into master (36141db) will increase coverage by 0.31%.
The diff coverage is 96.57%.

@@            Coverage Diff             @@
##           master     #596      +/-   ##
==========================================
+ Coverage   84.08%   84.40%   +0.31%     
==========================================
  Files          59       59              
  Lines        5644     5790     +146     
==========================================
+ Hits         4746     4887     +141     
- Misses        898      903       +5

Flag	Coverage Δ
unittests	`84.40% <96.57%> (+0.31%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
dandi/models.py	`88.79% <93.33%> (+0.83%)`	⬆️
dandi/tests/test_models.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 36141db...2ae1c02. Read the comment docs.

djarecka · 2021-04-27T04:20:16Z

@satra - I'm testing metadata that I took from dandisets 04 and 08. I could modify dataset 27 and add to the test, but can also keep adding various metadata to test_dantimeta_datacite

satra · 2021-04-27T11:06:27Z

for the moment you can use a local test, but we should also add a test which downloads from the API, augments it with the missing pieces (because draft metadata may not have doi, url, etc.,.) and then runs through your datacite test using the test server.

dandi/models.py

satra · 2021-04-27T11:15:05Z

dandi/tests/test_models.py

+
+    # checking if i'm able to get the url
+    rg = requests.get(
+        url=f"https://api.test.datacite.org/dois/{doi.replace('/','%2F')}/activities"


why activities and not compare to the posted metadata?

also we shouldn't have to do the doi.replace

indeed, the replacement is not needed. But the form with "activities" I took from datacite examples and it doesn't work without

dandi/tests/test_models.py

dandi/models.py

dandi/tests/test_models.py

yarikoptic · 2021-05-17T13:30:21Z

Please merge master or rebase -- "Test populate_dandiset_yaml.py / test (push) " should be gone now.

yarikoptic · 2021-05-17T15:12:22Z

didn't look through all details (we should put con/tinuous to work here, filed #639) but those failing which are saw are of the

dandi/tests/test_models.py:232: in test_datacite
    newmeta = PublishedDandisetMeta(**newmeta_js)
pydantic/main.py:406: in pydantic.main.BaseModel.__init__
    ???
E   pydantic.error_wrappers.ValidationError: 1 validation error for PublishedDandisetMeta
E   assetsSummary
E     field required (type=value_error.missing)

nature

djarecka · 2021-05-17T19:55:54Z

the biggest problem is that assetsSummary was added to PublishedDandisetMeta. Do you have any function to create it?

satra · 2021-05-17T20:23:53Z

that would be a good function to right. this function should iterate over all assets in a dandiset to create a summary of unique values for those fields.

however, this is also what yarik was going to add to the asset. for example approach/technique are not being populated at the asset level right now. we can relax some of the components to optional, but i think species, approach, technique should really be there.

djarecka · 2021-05-17T21:27:51Z

i'm a bit confused since it looks like a combination of information that should be provided by the author and info that could be calculated automatically...

satra · 2021-05-17T21:31:11Z

all of that info is calculated from asset metadata, hence the name. the authors would provide this when creating nwb files (and they generally do). we have to do a bit more magic for bids datasets, so those would likely not be publishable till we have figured out some pieces.

djarecka · 2021-05-17T21:51:49Z

ok, so for testing, I guess it's ok for me to use some random entry

satra · 2021-05-17T21:53:15Z

I guess it's ok for me to use some random entry

yes

…tests/data/metadata

djarecka · 2021-05-17T23:50:17Z

for now, I just saved meta from 000004 and 00008 in tests/data/metadata for testing. I could work on dandiset 000027 and additional dandisets later. Today I was not really able to use https://gui-staging.dandiarchive.org/ (doesn't load) and I still can't edit 000027

…t in parallel

djarecka · 2021-05-18T00:15:36Z

not sure what to do with Test model schemata / publish test

yarikoptic · 2021-05-19T18:02:17Z

well, if you look at that fail run fail you will see

diff --git a/releases/0.3.1/published-dandiset.json b/releases/0.3.1/published-dandiset.json
index 14b909b..3f03ea5 100644
--- a/releases/0.3.1/published-dandiset.json
+++ b/releases/0.3.1/published-dandiset.json
@@ -270,7 +270,8 @@
     "contributor",
     "license",
     "assetsSummary",
-    "version"
+    "version",
+    "doi"
   ],
   "definitions": {
     "RoleType": {
Error:  Existing schema files modified instead of creating a new version
Error: Process completed with exit code 1.

so this PR modifies schema by adding doi (I am just surprised we did not have it for published-dandiset yet). I guess it is intended, so let's just boost schema version to 0.3.2 and merge. Or @satra -- you want to mutate 0.3.1? change is minor but not sure if would not cause some ripples in dandi-api / webui.

satra · 2021-05-19T18:08:05Z

@yarikoptic - the doi was already there in the model, and should also be non-optional. this mutates it by removing a default value, which i think has an impact on whether or not it got exported to JSON schema. so this was a bug in the schema generation which this PR fixes.

satra · 2021-05-19T18:09:27Z

technically this should not cause any issues in api/ui since we have had discussions about doi, but pinging @dchiquito. would be fine with me to mutate.

dandi/models.py

yarikoptic · 2021-05-19T18:14:47Z

ok, left minor comments but could be handled post-merge/migration to dedicated dandischema.
So I guess we could just wait for @dchiquito to give blessing to mutate or could boost the .patch and proceed with the merge.

yarikoptic · 2021-05-19T18:41:34Z

Ok, @dchiquito gave blessing in slack, I fixed lint issue I introduced, let's proceed. Anything remaining should be done against dandischema

djarecka added 17 commits March 27, 2021 15:53

adding to_cite

1056afb

using the first contributor as a creator

f3adbd8

fixing contributors and creators

a2d1595

updating to_datacite

1d0d38a

removing notes that were added by mistake

dd291ea

removing doi, url creation - this should be already in the metadata

6c2c358

merging

8d5d0c3

fixing a few items; allowing creators to be contributors

16e76ac

adding dandi identifier to relatedIdentifiers

6930295

updates to to_datacite: fixing contribution type (choosing only from …

7abd530

…datacite approved list); adding Sponsors to fundingReferences; adding automatic fix for relatedIdentifiers types

adding doi to PublishedDandisetMeta

11d7c73

using meta as an argument in to_datacite

c13ca95

adding tests for creating datacite for 04 and 08 datasets

e68c83d

linting

3dace98

using DATACITE_DEV_PASSWORD secrets to upload doi to datacite

872fc0f

resolving conflicts

8a99f60

Merge pull request #595 from djarecka/datacite_tmp

b37cc7b

Datacite tmp

djarecka mentioned this pull request Apr 26, 2021

[wip] adding to_datacite #516

Closed

djarecka marked this pull request as draft April 26, 2021 17:08

djarecka added 6 commits April 26, 2021 15:17

updating datacite tests, and data files with metadata

d406d60

fix of buggy conflicts resolve

f642015

Merge branch 'datacite_tmp' into datacite_upstr_tmp

18d6bee

rearranging tests

852d3b9

adding additional tests

21d99af

adding random int as dandi_it to avoid posting the same doi in parallel

61944a2

satra reviewed Apr 27, 2021

View reviewed changes

dandi/models.py Outdated Show resolved Hide resolved

satra reviewed Apr 27, 2021

View reviewed changes

yarikoptic reviewed May 12, 2021

View reviewed changes

dandi/tests/test_models.py Outdated Show resolved Hide resolved

yarikoptic requested a review from jwodder May 12, 2021 21:00

yarikoptic added the patch Increment the patch version when merged label May 12, 2021

jwodder requested changes May 13, 2021

View reviewed changes

small updates after the review

955054f

yarikoptic mentioned this pull request May 17, 2021

Move metadata functionality to a dedicated dandischema #638

Closed

Merge branch 'master' into datacite_upstr_tmp

5b7b019

djarecka added 2 commits May 17, 2021 19:22

fixing tests: adding assetsSummary and updating dandi_id format

f462b93

changing test_datacite so it uses exemplary metadata written in json …

cb56b8f

…tests/data/metadata

fixing tests - adding some random number to doi to be able to run tes…

c9818e0

…t in parallel

yarikoptic reviewed May 19, 2021

View reviewed changes

dandi/models.py Outdated Show resolved Hide resolved

Make exception more specific and informative

2ae1c02

yarikoptic force-pushed the datacite_upstr_tmp branch from 7ea408e to 2ae1c02 Compare May 19, 2021 18:20

yarikoptic merged commit 8f29c1b into master May 19, 2021

yarikoptic deleted the datacite_upstr_tmp branch May 19, 2021 18:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adding to_datacite method #596

adding to_datacite method #596

djarecka commented Apr 26, 2021

codecov bot commented Apr 26, 2021 •

edited

Loading

djarecka commented Apr 27, 2021

satra commented Apr 27, 2021

satra Apr 27, 2021

satra Apr 27, 2021

djarecka Apr 27, 2021

yarikoptic commented May 17, 2021

yarikoptic commented May 17, 2021

djarecka commented May 17, 2021

satra commented May 17, 2021

djarecka commented May 17, 2021

satra commented May 17, 2021

djarecka commented May 17, 2021

satra commented May 17, 2021

djarecka commented May 17, 2021

djarecka commented May 18, 2021

yarikoptic commented May 19, 2021

satra commented May 19, 2021

satra commented May 19, 2021

yarikoptic commented May 19, 2021

yarikoptic commented May 19, 2021

adding to_datacite method #596

adding to_datacite method #596

Conversation

djarecka commented Apr 26, 2021

codecov bot commented Apr 26, 2021 • edited Loading

Codecov Report

djarecka commented Apr 27, 2021

satra commented Apr 27, 2021

satra Apr 27, 2021

Choose a reason for hiding this comment

satra Apr 27, 2021

Choose a reason for hiding this comment

djarecka Apr 27, 2021

Choose a reason for hiding this comment

yarikoptic commented May 17, 2021

yarikoptic commented May 17, 2021

djarecka commented May 17, 2021

satra commented May 17, 2021

djarecka commented May 17, 2021

satra commented May 17, 2021

djarecka commented May 17, 2021

satra commented May 17, 2021

djarecka commented May 17, 2021

djarecka commented May 18, 2021

yarikoptic commented May 19, 2021

satra commented May 19, 2021

satra commented May 19, 2021

yarikoptic commented May 19, 2021

yarikoptic commented May 19, 2021

codecov bot commented Apr 26, 2021 •

edited

Loading