Release 0.21.0

Release 0.21.0

Major Features and Improvements

• Added tfma.metrics.MinLabelPosition and tfma.metrics.QueryStatistics for
  use with V2 metrics API.
• Added tfma.metrics.CoefficientOfDiscrimination and
  tfma.metrics.RelativeCoefficientOfDiscrimination for use with V2 metrics
  API.
• Added support for using tf.keras.metrics.* metrics with V2 metrics API.
• Added support for default V2 MetricSpecs and creating specs from
  tf.kera.metrics.* and tfma.metrics.* metric classes.
• Added new MetricsAndPlotsEvaluator based on V2 infrastructure. Note this
  evaluator also supports query-based metrics.
• Add support for micro_average, macro_average, and weighted_macro_average
  metrics.
• Added support for running V2 extractors and evaluators. V2 extractors will
  be used whenever the default_eval_saved_model is created using a non-eval
  tag (e.g. tf.saved_model.SERVING). The V2 evaluator will be used whenever
  a tfma.EvalConfig is used containing metrics_specs.
• Added support for tfma.metrics.SquaredPearsonCorrelation for use with V2
  metrics API.
• Improved support for TPU autoscaling and handling batch_size related
  scaling.
• Added support for tfma.metrics.Specificity, tfma.metrics.FallOut, and
  tfma.metrics.MissRate for use with V2 metrics API. Renamed AUCPlot to
  ConfusionMatrixPlot, MultiClassConfusionMatrixAtThresholds to
  MultiClassConfusionMatrixPlot and MultiLabelConfusionMatrixAtThresholds
  to MultiLabelConfusionMatrixPlot.
• Added Jupyter support to Fairness Indicators. Currently does not support WIT
  integration.
• Added fairness indicators metrics
  tfma.addons.fairness.metrics.FairnessIndicators.
• Updated documentation for new metrics infrastructure and newly supported
  models (keras, etc).

Bug fixes and other changes

• Fixed error in tfma-multi-class-confusion-matrix-at-thresholds with
  default classNames value.
• Fairness Indicators
  • Compute ratio metrics with safe division.
  • Remove "post_export_metrics" from metric names.
  • Move threshold dropdown selector to a metric-by-metric basis, allowing
    different metrics to be inspected with different thresholds. Don't show
    thresholds for metrics that do not support them.
  • Slices are now displayed in alphabetic order.
  • Adding an option to "Select all" metrics in UI.

Breaking changes

• Updated proto config to remove input/output data specs in favor of passing
  them directly to the run_eval.

Deprecations

Release 0.15.4

Release 0.15.4

Major Features and Improvements

Bug fixes and other changes

• Fixed the bug that Fairness Indicator will skip metrics with NaN value.

Breaking changes

Deprecations

Release 0.15.3

Release 0.15.3

Major Features and Improvements

Bug fixes and other changes

• Updated vulcanized_tfma.js with UI changes in addons/fairness_indicators.

Breaking changes

Deprecations

Release 0.15.2

Release 0.15.2

Major Features and Improvements

Bug fixes and other changes

• Updated to use tf.io.gfile for reading config files (fixes issue with
  reading from GCS/HDFS in 0.15.0 and 0.15.1 releases).

Breaking changes

Deprecations

Release 0.15.1

Release 0.15.1

Major Features and Improvements

• Added support for defaulting to using class IDs when classes are not present
  in outputs for multi-class metrics (for use in keras model_to_estimator).
• Added example count metrics (tfma.metrics.ExampleCount and
  tfma.metrics.WeightedExampleCount) for use with V2 metrics API.
• Added calibration metrics (tfma.metrics.MeanLabel,
  tfma.metrics.MeanPrediction, and tfma.metrics.Calibration) for use with
  V2 metrics API.
• Added tfma.metrics.ConfusionMatrixAtThresholds for use with V2 metrics
  API.
• Added tfma.metrics.CalibrationPlot and tfma.metrics.AUCPlot for use with
  V2 metrics API.
• Added multi_class / multi_label plots (
  tfma.metrics.MultiClassConfusionMatrixAtThresholds,
  tfma.metrics.MultiLabelConfusionMatrixAtThresholds) for use with V2
  metrics API.
• Added tfma.metrics.NDCG metric for use with V2 metrics API.

Bug fixes and other changes

• Depends on tensorflow>=1.15,<3.0.
  • Starting from 1.15, package tensorflow comes with GPU support. Users
    won't need to choose between tensorflow and tensorflow-gpu.
  • Caveat: tensorflow 2.0.0 is an exception and does not have GPU
    support. If tensorflow-gpu 2.0.0 is installed before installing
    tensorflow_model_analysis, it will be replaced with tensorflow
    2.0.0. Re-install tensorflow-gpu 2.0.0 if needed.

Breaking changes

Deprecations

Release 0.15.0

Release 0.15.0

Major Features and Improvements

• Added V2 of PredictExtractor that uses TF 2.0 signature APIs and supports
  keras models (note: keras model evaluation not fully supported yet).
• tfma.run_model_analysis, tfma.default_extractors,
  tfma.default_evaluators, and tfma.default_writers now allow settings to
  be passed as an EvalConfig.
• tfma.run_model_analysis, tfma.default_extractors,
  tfma.default_evaluators, and tfma.default_writers now allow multiple
  models to be passed (note: multi-model support not fully implemented yet).
• Added InputExtractor for extracting labels, features, and example weights
  from tf.Examples.
• Added Fairness Indicator as an addon.

Bug fixes and other changes

• Enabled TF 2.0 support using compat.v1.
• Added support for slicing on native dicts of features in addition to FPL
  types.
• For multi-output and / or multi-class models, please provide output_name and
  / or class_id to tfma.view.render_plot.
• Replaced dependency on tensorflow-transform with tfx-bsl. If running
  with latest master, tfx-bsl must also be latest master.
• Depends on tfx-bsl>=0.15,<0.16.
• Slicing now supports conversion between int/floats and strings.
• Depends on apache-beam[gcp]>=2.16,<3.
• Depends on six==1.12.

Breaking changes

• tfma.EvalResult.slicing_metrics now contains nested dictionaries of output,
  class id and then metric names.
• Update config serialization to use JSON instead of pickling and reformat
  config to include input_data_specs, model_specs, output_data_specs, and
  metrics_specs.
• Requires pre-installed TensorFlow >=1.15,<3.

Deprecations

Release 0.14.0

Release 0.14.0

Major Features and Improvements

• Added documentation on architecture.
• Added an adapt_to_remove_metrics function to tfma.exporter which can be
  used to remove metrics incompatible with TFMA (e.g. py_func or streaming
  metrics) before exporting the TFMA EvalSavedModel.
• Added support for passing sparse int64 tensors to precision/recall@k.
• Added support for binarization of multiclass metrics that use labels of the
  from (N) in addition to (N, 1).
• Added support for using iterators with EvalInputReceiver.
• Improved performance of confidence interval computations by modifying the
  pipeline shape.
• Added QueryBasedMetricsEvaluator which supports computing query-based
  metrics (e.g. normalized discounted cumulative gain).
• Added support for merging metrics produced by different evaluators.
• Added support for blacklisting specified features from fetches.
• Added functionality to the FeatureExtractor to specify the features dict as
  a possible destination.
• Added support for label vocabularies for binary and multi-class estimators
  that support the new ALL_CLASSES prediction output.
• Move example parsing in aggregation into the graph for performance
  improvements in both standard and model_agnostic evaluation modes.

Bug fixes and other changes

• Upgraded codebase for TF 2.0 compatibility.
• Make metrics-related operations thread-safe by wrapping them with locks.
  This eliminates race conditions that were previously possible in
  multi-threaded runners which could result in incorrect metric values.
• More flexible FanoutSlices.
• Limit the number of sampling buckets to 20.
• Improved performance in Confidence Interval computation.
• Refactored poisson bootstrap code to be re-usable in other evaluators.
• Refactored k-anonymity code to be re-usable in other evaluators.
• Fixed slicer feature string value handling in Python3.
• Added support for example weight keys for multi-output models.
• Added option to set the desired batch size when calling run_model_analysis.
• Changed TFRecord compression type from UNCOMPRESSED to AUTO.
• Depends on apache-beam[gcp]>=2.14,<3.
• Depends on numpy>=1.16,<2.
• Depends on protobuf>=3.7,<4.
• Depends on scipy==1.1.0.

Breaking changes

• Removed uses of deprecated tf.contrib packages (where possible).
• tfma.default_writers now requires the eval_saved_model to be passed as
  an argument.
• Requires pre-installed TensorFlow >=1.14,<2.

Deprecations

Release 0.13.2

This release has the exact same source code as 0.13.1. It was created because 0.13.1 was released to PyPI with the wrong version of the code for the Python 2 version.

Release 0.13.1

Major Features and Improvements

• Added support for squared pearson correlation (R squared) post export
  metric.

Bug fixes and other changes

• Cast / convert labels for precision / recall at K so that they work even if
  the label and the classes Tensors have different types, as long as the types
  are compatible.
• Post export metrics will now also search for prediction keys prefixed by
  metric_tag if it is specified.
• Added support for precision/recall @ k using canned estimators provided
  label vocab not used.
• Preserve unicode type of slice keys when serialising to and deserialising
  from disk, instead of always converting them to bytes.
• Use __slots__ in accumulators.

Breaking changes

• Expose Python 3 types in the code (this will break Python 2 compatibility)

Release 0.13.0

Release 0.13.0

Major Features and Improvements

• Python 3.5 is supported.

Bug fixes and other changes

• Added support for fetching additional tensors at prediction time besides
  features, predictions, and labels (predict now returns FetchedTensorValues
  type).
• Removed internal usages of encoding.NODE_SUFFIX indirection within dicts
  in the eval_saved_model module (encoding.NODE_SUFFIX is still used in
  FeaturesPredictionLabels)
• Predictions are now returned as tensors (vs dicts) when "predictions" is the
  only output (this is consistent with how features and labels work).
• Depends on apache-beam[gcp]>=2.11,<3.
• Depends on protobuf>=3.7,<4.
• Depends on scipy==1.1.0.

Breaking changes

• Post export metrics for precision_recall_at_k were split into separate
  fuctions: precision_at_k and recall_at_k.
• Requires pre-installed TensorFlow >=1.13,<2.

Deprecations