Fixes miss-parsed names in AutomaticPersonsGroup

[k3KAW8Pnf7mkmdSMPHz27]

Fixes #5833.

This fixes the parsing and matching of authors' last names. It is much slower than the previous solution, and uses more RAM, but I think that is hard/impossible to avoid. However, it is possible that the caching solution can be made smarter, but I leave the decision if it should, to the reviewers 😁

An alternative solution is moving the caching to AuthorList, it will make it easier to modify/improve at a later point in time. It will make this PR take quite a bit of more time as I'd like to make AuthorList extend List in that case. See #6552 (comment) for a much longer discussion on that topic.
Given that the only thing that should be cached are the last names (and always latex-free), perhaps it has more in common with the autocomplete functionality. I have not looked into that.

• Change in CHANGELOG.md described (if applicable)
• Tests created for changes (if applicable)
• Manually tested changed features in running JabRef (always required)
• Screenshots added in PR description (for UI changes)
• Checked documentation: Is the information available and up to date? If not created an issue at https://github.com/JabRef/user-documentation/issues or, even better, submitted a pull request to the documentation repository.

[tobiasdiez]

So, what was the problem in the end? It looks like the code is more or less equivalent to the one before, except that now the author parsing and latex2unicode conversion are in a different order (which does make sense!). Is this the only change?

Concerning the caching, I think this is not needed. The parsing of a string to an authorlist is already cached, and getting the last names with latex converted is also cached:

jabref/src/main/java/org/jabref/model/entry/AuthorList.java

Line 377 in afdb194

public String getAsLastNamesLatexFree(boolean oxfordComma) {

However, this code probably needs to be changed a bit to be usable for our purposes. I would suggest to add a getLastNameLatexFree in Author, which caches it's result. This method then can be used in the groups parsing, as well as in AuthorList.getAsLastNamesLatexFree (there is no need to have an additional cache there again).

[k3KAW8Pnf7mkmdSMPHz27]

So, what was the problem in the end? It looks like the code is more or less equivalent to the one before, except that now the author parsing and latex2unicode conversion are in a different order (which does make sense!). Is this the only change?

The other change is that the previous solution relied on the existing WordKeywordGroup SearchStrategies which did not parse authors. Hence, "Turing, Allan" ended up attempting to match the group "Turing" with the last name "Turing," (note the ,). There might be other differences as well but my "contribution" is to use the same code that parses the names for the groups to check if an author matches.
Note that among the test cases it is only createSubgroupsContainingCommaSeparatedLastNames that fails without these changes.

Now that I am looking at it again, I'll see if I can extend KeywordGroup instead. It should make the changes more readable.

Concerning the caching, I think this is not needed. The parsing of a string to an authorlist is already cached, and getting the last names with latex converted is also cached:

jabref/src/main/java/org/jabref/model/entry/AuthorList.java

Line 377 in afdb194

public String getAsLastNamesLatexFree(boolean oxfordComma) {

However, this code probably needs to be changed a bit to be usable for our purposes. I would suggest to add a getLastNameLatexFree in Author, which caches it's result. This method then can be used in the groups parsing, as well as in AuthorList.getAsLastNamesLatexFree (there is no need to have an additional cache there again).

In some cases the getAsLastNamesLatexFree contain "and" between the two last authors. Given that a latex-free string is allowed to contain "and" for other reasons (e.g., {Barnes and Noble}) I'd rather avoid using a plain String.
I don't have an opinion regarding adding an Author#getLastNameLatexFree method. Every time I look at this issue I end up thinking "this should be fixed in BibEntry#getResolvedFieldOrAliasLatexFree" in which case the AUTHOR_CACHE would cache latexfree AuthorLists, but in my opinion, it is a bit of a nightmare to implement.
Should I add the Author#getLastNameLatexFree method? At the very least it would allow

.map(Author::getLastNameLatexFree)

and removing the caching for the AuthorList method.

[k3KAW8Pnf7mkmdSMPHz27]

I just feel that if the caching for AuthorList.getAsLastNamesLatexFree is removed, then all the AuthorList.getAs...LatexFree should be removed. I'd guess the stream...Collectors.joining is fast enough for that.

[k3KAW8Pnf7mkmdSMPHz27]

Just a note... It seems that none of the groups use BibDatabase which means they all use entry.getField instead of entry.getResolvedFieldOrAlias.
I'd argue it would be out of scope for this issue, but it could lead to problems (perhaps it is unlikely to be used for the author/editor fields).

[k3KAW8Pnf7mkmdSMPHz27]

I just realized that AuthorList.getAsLastNamesLatexFree uses getLastOnly, not getLast. So I can't reuse the last name cache easily in AuthorList. I think that for the AutomaticPersonGroup it makes sense that the subgroup is named Neumann and not von Neumann.

[tobiasdiez]

I just feel that if the caching for AuthorList.getAsLastNamesLatexFree is removed, then all the AuthorList.getAs...LatexFree should be removed. I'd guess the stream...Collectors.joining is fast enough for that.

I completely agree! The Author class should be responsible to calculate the latex-free version of names etc, and cache its result. Maybe, the most readable version is a method Author latexfree() which returns the complete latex2unicode converted representation of the author, so you can write author.latexfree().getLast(). That's something of value even beyond this ticket!
Would also make it pretty easy to cache in a flexible way by having only Author latexfree as a cached variable. It's also pretty extensible you get automatic caching for methods like getLastOnly.

Every time I look at this issue I end up thinking "this should be fixed in BibEntry#getResolvedFieldOrAliasLatexFree"

That would be indeed a good solution, but is sadly not possible. {Barnes and Noble} and Author is sadly different to Barnes and Noble and Author. The author specification of bibtex is just a mess, and relies on so many assumptions. The best would be if people would only rely on the explicit name format of biber/biblatex: AUTHOR = {given=Arnar, family=Vigfusson}, but that's a very long way to go.

[k3KAW8Pnf7mkmdSMPHz27]

I just feel that if the caching for AuthorList.getAsLastNamesLatexFree is removed, then all the AuthorList.getAs...LatexFree should be removed. I'd guess the stream...Collectors.joining is fast enough for that.

I completely agree! The Author class should be responsible to calculate the latex-free version of names etc, and cache its result. Maybe, the most readable version is a method Author latexfree() which returns the complete latex2unicode converted representation of the author, so you can write author.latexfree().getLast(). That's something of value even beyond this ticket!
Would also make it pretty easy to cache in a flexible way by having only Author latexfree as a cached variable. It's also pretty extensible you get automatic caching for methods like getLastOnly.

Perhaps this is the good way out of this... I just have this itch to kill the Author class and make it a Map, preferably an EnumMap (thought that might not be possible)... I might be tunnel-visioning on that 😁
I'd argue it is out of scope of this issue. Rewriting the AuthorList#getAs... would be (one) of the benefits but it would be quite time consuming. Additionally, if someone wants to track down possible performance bottlenecks, it would be better in its own PR. You think opening up a new PR with Author.latexFree (Author.resolveLatex?) is the way to go (rather than continuing it here)?

Every time I look at this issue I end up thinking "this should be fixed in BibEntry#getResolvedFieldOrAliasLatexFree"

That would be indeed a good solution, but is sadly not possible. {Barnes and Noble} and Author is sadly different to Barnes and Noble and Author. The author specification of bibtex is just a mess, and relies on so many assumptions. The best would be if people would only rely on the explicit name format of biber/biblatex: AUTHOR = {given=Arnar, family=Vigfusson}, but that's a very long way to go.

In my opinion, that solution is possible. However, it would require changes to the author parser so that delimiters can be reconstructed. The surrounding braces are removed by Author, internal braces might be a bigger challenge. It would also tie in to the discussion about the mess outside the bibtex/biber/biblatex definitions of authors, if #2205 is still an issue (I have been eyeing it but not found the time to look at the details).

And thank you for multiple code review on new year as well! 🎆

[tobiasdiez]

The current changes are fine with me. The author caching of latexfree things can be done in a follow-up PR.

[k3KAW8Pnf7mkmdSMPHz27]

k3KAW8Pnf7mkmdSMPHz27 dismissed tobiasdiez’s stale review via edd24bd

Oups. Sorry, I did not know that was a thing. I just wanted to play around with what went wrong with the equality and hashCode (barring my thinking 🤡 ), since it should not hurt to have more "standard" equality and hashCode methods.

[tobiasdiez]

I've also only recently noticed this. I guess it's coming from the recent changes that PRs need to be reviewed before they are merged (that was always a standing requirement, but now github is configured for it is well). @koppor

[koppor]

k3KAW8Pnf7mkmdSMPHz27 dismissed tobiasdiez’s stale review via edd24bd

This is some automatic thing. Any change will dismiss a review. We want to be sure that approvals are "done right". The final approver can check whether the changes from the dismissed approval to the current one are OK.

Thus, no worries! 🎉 - And thank you for continuing on working on this.