Create transparency with respect to the metric’s data and model applicability

[annahedstroem]

Addressed issues:

• Create transparency with respect to the metric’s individual data and model applicability #280
• Normalisation per category #202
• Renaming from last_results to evaluation_scores and all_results to all_evaluation_scores
• Normalisation per category #202

Minimum acceptance criteria

• Specify what is necessary for the PR to be merged with the main branch.
• @mentions of the person that is apt to review these changes e.g., @annahedstroem

[aaarrti]

Addressed issues:

• Create transparency with respect to the metric’s individual data and model applicability #280
• Normalisation per category #202

Minimum acceptance criteria

• Specify what is necessary for the PR to be merged with the main branch.
• @mentions of the person that is apt to review these changes e.g., @annahedstroem

Just to make sure we are on the same page, the goals of this task are:

• make it easier for users to understand if their data and metric of choice are compatible
• systematize available metrics, pre-processing features, supported model, etc.

Am I right?

[annahedstroem]

Addressed issues:

• Create transparency with respect to the metric’s individual data and model applicability #280
• Normalisation per category #202

Minimum acceptance criteria

• Specify what is necessary for the PR to be merged with the main branch.
• @mentions of the person that is apt to review these changes e.g., @annahedstroem

Just to make sure we are on the same page, the goals of this task are:

• make it easier for users to understand if their data and metric of choice are compatible
• systematize available metrics, pre-processing features, supported model, etc.

Am I right?

Yes, this is a starting point for this effort. Then we can complement this PR with a more in-depth ModelType and DataType checker, as the need arise.

[aaarrti]

This one can safely be implemented in the base class,

@property
def model_applicability(self) -> Set[ModelType]:
     return {ModelType.Torch, ModelType.TensorFlow}

Since all metrics support both, afaik.

[annahedstroem]

Sure, can be done.

[codecov-commenter]

Codecov Report

Merging #279 (5a9ee4b) into main (72af154) will increase coverage by 0.49%.
The diff coverage is 96.72%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##             main     #279      +/-   ##
==========================================
+ Coverage   92.87%   93.36%   +0.49%     
==========================================
  Files          60       62       +2     
  Lines        3200     3468     +268     
==========================================
+ Hits         2972     3238     +266     
- Misses        228      230       +2     

Impacted Files	Coverage Δ
quantus/functions/loss_func.py	83.33% <0.00%> (ø)
quantus/functions/normalise_func.py	74.07% <ø> (ø)
...ics/randomisation/model_parameter_randomisation.py	82.89% <64.70%> (+1.46%)	⬆️
quantus/metrics/faithfulness/irof.py	98.33% <87.50%> (+0.18%)	⬆️
quantus/metrics/base_perturbed.py	93.33% <93.33%> (ø)
quantus/helpers/enums.py	100.00% <100.00%> (ø)
quantus/metrics/__init__.py	100.00% <100.00%> (ø)
quantus/metrics/axiomatic/completeness.py	95.12% <100.00%> (+0.83%)	⬆️
quantus/metrics/axiomatic/input_invariance.py	100.00% <100.00%> (ø)
quantus/metrics/axiomatic/non_sensitivity.py	100.00% <100.00%> (ø)
... and 31 more