Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement precision and recall metrics for classification evaluation #49671

Merged
merged 13 commits into from
Dec 19, 2019

Conversation

przemekwitek
Copy link
Contributor

@przemekwitek przemekwitek commented Nov 28, 2019

This PR implements precision and recall metrics for classification evaluation.

Additionally, it:

  • registers evaluation metrics in XContent and Writeable registries under qualified names in order to avoid name clashes
  • changes the interface of EvaluationMetric interface by allowing pipeline aggregations to be requested by aggs method
  • moves actualIsTrueQuery method from interface to implementation as this method is implementation-specific

Relates #48759

@cbuescher cbuescher added the :ml Machine learning label Nov 28, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/ml-core (:ml)

@przemekwitek przemekwitek force-pushed the precision_and_recall branch 12 times, most recently from 91d83e0 to 3edeb6b Compare December 3, 2019 14:45
@przemekwitek przemekwitek removed the WIP label Dec 3, 2019
@przemekwitek przemekwitek marked this pull request as ready for review December 3, 2019 15:00
@przemekwitek
Copy link
Contributor Author

run elasticsearch-ci/1
run elasticsearch-ci/packaging-sample-matrix

Copy link
Contributor

@dimitris-athanasiou dimitris-athanasiou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving a few comments. I'll revisit when recall has been adjusted.

Copy link
Contributor

@dimitris-athanasiou dimitris-athanasiou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some more comments as I go along the way.

return Tuple.tuple(
List.of(
AggregationBuilders.terms(ACTUAL_CLASSES_NAMES_AGG_NAME)
.field(actualField)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we also need a size parameter in all these like in the multiclass confusion matrix?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just added size parameter to Precision and Recall metrics.
Also, I've added other_class_count parameter to Precision.Result and Recall.Result so that the user can tell if the result is complete.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've just added size parameter to Precision and Recall metrics.
Also, I've added other_class_count parameter to Precision.Result and Recall.Result so that the user can tell if the result is complete.

I reverted this change and implemented max cardinality enforcement as discussed. PTAL

String className = bucket.getKeyAsString();
NumericMetricsAggregation.SingleValue precisionAgg = bucket.getAggregations().get(PER_PREDICTED_CLASS_PRECISION_AGG_NAME);
double precision = precisionAgg.value();
if (Double.isFinite(precision)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we be checking this? If for some reason precision is not finite, we'll end up reporting zero instead of NaN or infinity.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will not report zero but rather we will not report this particular class.
My reasoning behind this condition is that if we have an actual class that is never predicted (e.g: "cat"), precision for "cats" cannot be calculated so there is no point in reporting a precision entry with NaN value.

@przemekwitek przemekwitek force-pushed the precision_and_recall branch 2 times, most recently from 6c6406d to 6f254d8 Compare December 18, 2019 13:30
@przemekwitek
Copy link
Contributor Author

run elasticsearch-ci/default-distro
run elasticsearch-ci/bwc

@przemekwitek
Copy link
Contributor Author

run elasticsearch-ci/bwc

Copy link
Contributor

@dimitris-athanasiou dimitris-athanasiou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants