Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reorder AUC and Python version 3.7 #90

Closed
wants to merge 3 commits into from
Closed

Reorder AUC and Python version 3.7 #90

wants to merge 3 commits into from

Conversation

alexriedel1
Copy link
Contributor

@alexriedel1 alexriedel1 commented Jan 28, 2022

return auc(fpr, tpr)

throws a ValueError: The 'x' tensor is neither increasing or decreasing. Try setting the reorder argument to 'True'.
It happens for large datasets I guess with only little variance in some validation datapoints?

I set reorder=True to resolve the problem.

Also as the development branch is compatible to Python 3.7, it should also be possible to install the package und 3.7
#72

@samet-akcay
Copy link
Contributor

Hi @alexriedel1, thank you for your contribution to the repo! I've just triggered the Pre-merge checks, which is currently running.

@djdameln
Copy link
Contributor

djdameln commented Jan 28, 2022

@alexriedel1 Thanks for your PR! I agree that ideally the AUC should be computed with reorder=True. However, calling auc with this parameter may currently lead to an underestimation of the model's performance. The reason for this is that pytorch does not use stable sorting (see https://stackoverflow.com/questions/60366033/torch-sort-and-argsort-sorting-randomly-in-case-of-same-element and pytorch/pytorch#28871). This means that equal elements in the fpr array will lose their original ordering, which can lead to unpredictable behavior in the AUC computation.

It seems that stable sorting has been added in pytorch 1.9.0 as an option to sort, but for technical reasons we are currently bound to 1.8.1. Besides, even in torch 1.9.0, stable sorting is disabled by default, so updating pytorch would not solve the problem by itself.

As a temporary solution we could apply the sorting ourselves, using a stable sorting algorithm, inside the compute method (before computing the AUC).

@djdameln
Copy link
Contributor

@alexriedel1 I created a new PR that should address your problem, but maintain consistent results when the fpr is already ordered correctly. Will be closing this one.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants