Fetcher tests failing

[tobiasdiez]

Some of the fetcher tests are failing (ISBN, Inspire, ...)
https://github.com/JabRef/jabref/actions?query=workflow%3A%22Fetcher+Tests%22

[Siedlerchr]

• Grobid is failing due to Connection refused
• IEEE is failing due to API key probably
• MrDLibFetcherTest Gateway Timeout
• AstrophysicsDataSystemTest (needs ne API ?=>https://github.com/adsabs/adsabs-dev-api)

[koppor]

API keys: Howto for getting a sandbox key for developers

ADs-Fetcher: This is the PR switchting to the new API, isn't it? --> #5501

[Siedlerchr]

Just got mail that the INSPIRE has now an official API:
https://github.com/inspirehep/rest-api-doc

[koppor]

https://github.com/JabRef/jabref/runs/794938868

[Siedlerchr]

Springerlink is giving m,e 401 locally when I debug

[koppor]

@DominikVoigt Can you confirm for SpringerLink? 😅

[DominikVoigt]

If I substitute the API key that is hardcoded into the SpringerLink fetcher with my own API key all tests pass.

[koppor]

Springer mailed us on 2020-04-29:

Dear User,

Thank you for using Springer Nature APIs! While we are most pleased at your interest, we see traffic that is very high and well over the limit of 300 calls/min and 5000/day. The limit is 300 hits/min and 5000/day. Can you please adjust your code to stay within these limits? Please advise/confirm so we may restore service.

Sincerely,
SpringerNature API Team

Internal stats show "On November 28, your application called the API more than 13,000 times."

[koppor]

State as of 2020-08-02

Source: https://github.com/JabRef/jabref/runs/937189942

• Grobid: java.net.ConnectException: Connection timed out
• IEEE: Results differ
• CollectionOfComputerScienceBibliographiesFetcher: Assertion error
• LibraryOfCongressTest: Results differ
• CiteSeerTest: no answer
• DoiFetcherTest: Results differ
• AstrophysicsDataSystemTest: Results differ
• Springer: @DisabledOnCIServer("Disable on CI Server to not hit the API call limit")

[Siedlerchr]

We should add the Springer key to the build secrets as well

[koppor]

I will aquire a sepearate key for Springer and enable Springer tests in a separate PR.

[koppor]

Rate limit of IEEE

• 10 calls / second
• 200 calls / day

Source: Key agreement

[Siedlerchr]

We should execute the fetcher tests maybe once a month on CI and not on every PR

[koppor]

Since a few months, we DO NOT execute the fetcher tests on each PR.

We currently the fetcher tests, when

• something in the fetcher code changed
• each wednesday

See https://github.com/JabRef/jabref/blob/master/.github/workflows/tests-fetchers.yml

[koppor]

The discussion on this move can be found at #5807.

[calixtus]

• IEEE returns a bad URL: https:/ieeeexplore… (Note: only one slash). Problem seems to be serverside
• LibraryOfCongress returns bad year ( year = {c201} instead of year = {2011} )
• The JabRef BibEntry builder seems to automatically add an url-field, which is not included in the returned result from the server. Will do some research on this.
• ADS seems to be identical on first sight. Not sure why failing, maybe something I overlooked... Will also do some research on this.

[Siedlerchr]

citesser => journaltitle instead of journal

[koppor]

@DominikVoigt added a conversion layer at #6687. Thus, it should not matter whether journaltitle (BibLaTeX) or journal (BibTeX) is returned. -- the layer converts to the format required by the target database

[koppor]

@Siedlerchr Thank you for the hint --> @DominikVoigt's layer doesn't help here as the fetcher tests directly check the fetcher result - and not the reformatting by the intermediate layer.

[koppor]

Refs #3854 (as background information)

[koppor]

• Grobid runs in a time out. @LinusDietz can you check please?
• org.jabref.logic.importer.fetcher.ArXivTest has some differences, need to check later
• IEEE and Google (currently) do not run on the CI

[Siedlerchr]

Currently there seems to be a latex/unicode conversion issue in the INPIRE test

 org.opentest4j.AssertionFailedError: expected: <[@mastersthesis{Diez:2014ppa,
    archiveprefix = {arXiv},
    author = {Diez, Tobias},
    eprint = {1405.2249},
    primaryclass = {math-ph},
    school = {Leipzig U.},
    title = {Slice theorem for Fr'echet group actions and covariant symplectic field theory},
    year = {2013},
    _jabref_shared = {sharedId: -1, version: 1}
  }]> but was: <[@mastersthesis{Diez:2014ppa,
    archiveprefix = {arXiv},
    author = {Diez, Tobias},
    eprint = {1405.2249},
    primaryclass = {math-ph},
    school = {Leipzig U.},
    title = {Slice theorem for Frtextbackslash{}'echet group actions and covariant symplectic field theory},
    year = {2013},
    _jabref_shared = {sharedId: -1, version: 1}
  }]>
      at org.jabref.logic.importer.fetcher.INSPIREFetcherTest.searchByQueryFindsEntry(INSPIREFetcherTest.java:45)

[koppor]

• Jstor: Result differs
• CiteSeer: Result differs
• TitleFetcher: Change format
• Disable CompositeFetcherTest on CI
• Grobid: Server down

---

• Google Scholar. Refs Fix Google Scholar fetcher for downloading a single entry #7075
• Jstor: 403?
• IEEE: 403 --> rate limit

---

• ACM Fetcher --> disable or rewrite (https://github.com/NatashaDudi/jabref/blob/develop/src/main/java/org/jabref/logic/importer/fetcher/ACMPortalFetcher.java)

[koppor]

We disabled ACM, Google Scholar and JSTOR at #7229. We have different follow-up tickets and PRs. Thus, I can close this issue.