MemoryEvaluator.md

TFSimilarity.callbacks.MemoryEvaluator

In memory index performance evaluation and classification.

Inherits From: Evaluator, ABC

Methods

calibrate

View source

calibrate(
    target_labels: Sequence[int],
    lookups: Sequence[Sequence[Lookup]],
    thresholds_targets: MutableMapping[str, float],
    calibration_metric: <a href="../../TFSimilarity/callbacks/ClassificationMetric.md">TFSimilarity.callbacks.ClassificationMetric```
</a>,
    matcher: Union[str, <a href="../../TFSimilarity/callbacks/ClassificationMatch.md">TFSimilarity.callbacks.ClassificationMatch```
</a>],
    extra_metrics: Sequence[<a href="../../TFSimilarity/callbacks/ClassificationMetric.md">TFSimilarity.callbacks.ClassificationMetric```
</a>] = [],
    distance_rounding: int = 8,
    metric_rounding: int = 6,
    verbose: int = 1
) -> <a href="../../TFSimilarity/indexer/CalibrationResults.md">TFSimilarity.indexer.CalibrationResults```
</a>

Computes the distances thresholds that the classification must match to meet a fixed target.

Args
target_labels	Sequence of expected labels for the lookups.
lookup	Sequence of lookup results as produced by the Index.batch_lookup() method.
thresholds_targets	classification metrics thresholds that are targeted. The function will find the closed distance value.
calibration_metric	Classification metric used for calibration.
matcher	'match_nearest', 'match_majority_vote' or ClassificationMatch object. Defines the classification matching, e.g., match_nearest will count a True Positive if the query_label is equal to the label of the nearest neighbor and the distance is less than or equal to the distance threshold.
extra_metrics	Additional classification metrics that should be computed and reported as part of the calibration. Defaults to [].
distance_rounding	How many digit to consider to decide if the distance changed. Defaults to 8.
metric_rounding	How many digit to consider to decide if the metric changed. Defaults to 6.
verbose	Be verbose. Defaults to 1.

Returns
CalibrationResults containing the thresholds and cutpoints Dicts.

Raises
ValueError	lookupds must not be empty.

evaluate_classification

View source

evaluate_classification(
    query_labels: <a href="../../TFSimilarity/callbacks/IntTensor.md">TFSimilarity.callbacks.IntTensor```
</a>,
    lookup_labels: <a href="../../TFSimilarity/callbacks/IntTensor.md">TFSimilarity.callbacks.IntTensor```
</a>,
    lookup_distances: <a href="../../TFSimilarity/callbacks/FloatTensor.md">TFSimilarity.callbacks.FloatTensor```
</a>,
    distance_thresholds: <a href="../../TFSimilarity/callbacks/FloatTensor.md">TFSimilarity.callbacks.FloatTensor```
</a>,
    metrics: Sequence[<a href="../../TFSimilarity/callbacks/ClassificationMetric.md">TFSimilarity.callbacks.ClassificationMetric```
</a>],
    matcher: Union[str, <a href="../../TFSimilarity/callbacks/ClassificationMatch.md">TFSimilarity.callbacks.ClassificationMatch```
</a>],
    distance_rounding: int = 8,
    verbose: int = 1
) -> Dict[str, np.ndarray]

Evaluate the classification performance.

Compute the classification metrics given a set of queries, lookups, and distance thresholds.

Args
query_labels	Sequence of expected labels for the lookups.
lookup_labels	A 2D tensor where the jth row is the labels associated with the set of k neighbors for the jth query.
lookup_distances	A 2D tensor where the jth row is the distances between the jth query and the set of k neighbors.
distance_thresholds	A 1D tensor denoting the distances points at which we compute the metrics.
metrics	The set of classification metrics.
matcher	'match_nearest', 'match_majority_vote' or ClassificationMatch object. Defines the classification matching, e.g., match_nearest will count a True Positive if the query_label is equal to the label of the nearest neighbor and the distance is less than or equal to the distance threshold.
distance_rounding	How many digit to consider to decide if the distance changed. Defaults to 8.
verbose	Be verbose. Defaults to 1.

Returns
A Mapping from metric name to the list of values computed for each distance threshold.

evaluate_retrieval

View source

evaluate_retrieval(
    target_labels: Sequence[int],
    lookups: Sequence[Sequence[Lookup]],
    retrieval_metrics: Sequence[<a href="../../TFSimilarity/indexer/RetrievalMetric.md">TFSimilarity.indexer.RetrievalMetric```
</a>],
    distance_rounding: int = 8
) -> Dict[str, np.ndarray]

Evaluates lookup performances against a supplied set of metrics

Args
target_labels	Sequence of the expected labels to match.
lookups	Sequence of lookup results as produced by the Index().batch_lookup() method.
retrieval_metrics	Sequence of RetrievalMetric() to evaluate lookup matches against.
distance_rounding	How many digit to consider to decide if the distance changed. Defaults to 8.

Returns
Dictionary of metric results where keys are the metric names and values are the metrics values.