TensorFlow Model Analysis 0.37.0

Major Features and Improvements

• N/A

Bug fixes and other Changes

• Fixed issue with aggregation type not being set properly in keys associated
  with confusion matrix metrics.
• Enabled the sql_slice_key extractor when evaluating a model.
• Depends on numpy>=1.16,<2.
• Depends on absl-py>=0.9,<2.0.0.
• Depends on
  tensorflow>=1.15.5,!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,<3.
• Depends on tfx-bsl>=1.6.0,<1.7.0.
• Depends on tensorflow-metadata>=1.6.0,<1.7.0.
• Depends on apache-beam[gcp]>=2.35,<3.

Breaking Changes

• N/A

Deprecations

• N/A

TensorFlow Model Analysis 0.36.0

Major Features and Improvements

• Replaced keras metrics with TFMA implementations. To use a keras metric in a
  tfma.MetricConfig you must now specify a module (i.e. tf.keras.metrics).
• Added FixedSizeSample metric which can be used to extract a random,
  per-slice, fixed-sized sample of values for a user-configured feature key.

Bug fixes and other Changes

• Updated QueryStatistics to support weighted examples.
• Depends on apache-beam[gcp]>=2.34,<3.
• Depends on
  tensorflow>=1.15.2,!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,<3.
• Depends on tfx-bsl>=1.5.0,<1.6.0.
• Depends on tensorflow-metadata>=1.5.0,<1.6.0.

Breaking Changes

• Removes register_metric from public API, as it is not intended to be public
  facing. To use a custom metric, provide the module name in which the
  metric is defined in the MetricConfig message, instead.

Deprecations

• N/A

TensorFlow Model Analysis 0.35.0

Major Features and Improvements

• Added support for specifying weighted vs unweighted metrics. The setting is
  available in the tfma.MetricsSpec( example_weights=tfma.ExampleWeightOptions(weighted=True, unweighted=True)).
  If no options are provided then TFMA will default to weighted provided the
  associated tfma.ModelSpec has an example weight key configured, otherwise
  unweighted will be used.

Bug fixes and other Changes

• Added support for example_weights that are arrays.

• Reads baseUrl in JupyterLab to support TFMA rendering:
  #112

• Fixing couple of issues with CIDerivedMetricComputation:

  • no CI derived metric, deriving from private metrics such as
    binary_confusion_matrices, was being computed
  • convert_slice_metrics_to_proto method didn't have support for bounded
    values metrics.

• Depends on tfx-bsl>=1.4.0,<1.5.0.

• Depends on tensorflow-metadata>=1.4.0,<1.5.0.

• Depends on apache-beam[gcp]>=2.33,<3.

Breaking Changes

• Confidence intervals for scalar metrics are no longer stored in the
  MetricValue.bounded_value. Instead, the confidence interval for a metric
  can be found under MetricKeysAndValues.confidence_interval.
• MetricKeys now require specifying whether they are weighted (
  tfma.metrics.MetricKey(..., example_weighted=True)) or unweighted (the
  default). If the weighting is unknown then example_weighted will be None.
  Any metric computed outside of a tfma.metrics.MetricConfig setting (i.e.
  metrics loaded from a saved model) will have the weighting set to None.
• ExampleCount is now weighted based on tfma.MetricsSpec.example_weights
  settings. WeightedExampleCount has been deprecated (use ExampleCount
  instead). To get unweighted example counts (i.e. the previous implementation
  of ExampleCount), ExampleCount must now be added to a MetricsSpec
  where example_weights.unweighted is true. To get a weighted example count
  (i.e. what was previously WeightedExampleCount), ExampleCount must now
  be added to a MetricsSpec where example_weights.weighted is true.

Deprecations

• Deprecated python3.6 support.

TensorFlow Model Analysis 0.34.1

Major Features and Improvements

• N/A

Bug fixes and other Changes

• Correctly skips non-numeric numpy array type metrics for confidence interval
  computations.
• Depends on apache-beam[gcp]>=2.32,<3.
• Depends on tfx-bsl>=1.3.0,<1.4.0.

Breaking Changes

• In preparation for TFMA 1.0, the following imports have been moved (note
  that other modules were also moved, but TFMA only supports types that are
  explicitly declared inside of __init__.py files):
  • tfma.CombineFnWithModels -> tfma.utils.CombineFnWithModels
  • tfma.DoFnWithModels -> tfma.utils.DoFnWithModels
  • tfma.get_baseline_model_spec -> tfma.utils.get_baseline_model_spec
  • tfma.get_model_type -> tfma.utils.get_model_type
  • tfma.get_model_spec -> tfma.utils.get_model_spec
  • tfma.get_non_baseline_model_specs ->
    tfma.utils.get_non_baseline_model_specs
  • tfma.verify_eval_config -> tfma.utils.verify_eval_config
  • tfma.update_eval_config_with_defaults ->
    tfma.utils.update_eval_config_with_defaults
  • tfma.verify_and_update_eval_shared_models ->
    tfma.utils.verify_and_update_eval_shared_models
  • tfma.create_keys_key -> tfma.utils.create_keys_key
  • tfma.create_values_key -> tfma.utils.create_values_key
  • tfma.compound_key -> tfma.utils.compound_key
  • tfma.unique_key -> tfma.utils.unique_key

Deprecations

• N/A

TensorFlow Model Analysis 0.34.0

Major Features and Improvements

• Added SparseTensorValue and RaggedTensorValue types for storing
  in-memory versions of sparse and ragged tensor values used in extracts.
  Tensor values used for features, etc should now be based on either
  np.ndarray, SparseTensorValue, or RaggedTensorValue and not
  tf.compat.v1 value types.
• Add CIDerivedMetricComputation metric type.

Bug fixes and other Changes

• Fixes bug when computing confidence intervals for
  binary_confusion_metrics.ConfusionMatrixAtThresholds (or any other
  structured metric).
• Fixed bug where example_count post_export_metric is added even if
  include_default_metrics is False.
• Depends on apache-beam[gcp]>=2.31,<2.32.
• Depends on
  tensorflow>=1.15.2,!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,<3.
• Depends on tfx-bsl>=1.3.1,<1.4.0.

Breaking Changes

• N/A

Deprecations

• N/A

TensorFlow Model Analysis 0.33.0

Major Features and Improvements

• Provided functionality for slice_keys_sql config. It's not available under
  Windows.

Bug fixes and other Changes

• Improve rendering of HTML stubs for TFMA and Fairness Indicators UI.
• Update README for JupyterLab 3
• Provide implementation of ExactMatch metric.
• Jackknife CI method now works with cross-slice metrics.
• Depends on apache-beam[gcp]>=2.31,<3.
• Depends on tensorflow-metadata>=1.2.0,<1.3.0.
• Depends on tfx-bsl>=1.2.0,<1.3.0.

Breaking Changes

• The binary_confusion_matrices metric formerly returned confusion matrix
  counts (i.e number of {true,false} {positives,negatives}) and optionally a
  set of representative examples in a single object. Now, this metric class
  generates two separate metrics values when examples are configured: one
  containing just the counts, and the other just examples. This should only
  affect users who created a custom derived metric that used
  binary_confusion_matrices metric as an input.

Deprecations

• N/A

TensorFlow Model Analysis 0.32.1

Major Features and Improvements

• N/A

Bug fixes and other Changes

• Depends on google-cloud-bigquery>=1.28.0,<2.21.
• Depends on tfx-bsl>=1.1.1,<1.2.0.

Breaking Changes

• N/A

Deprecations

• N/A

TensorFlow Model Analysis 0.32.0

Major Features and Improvements

• N/A

Bug fixes and other Changes

• Depends on protobuf>=3.13,<4.
• Depends on tensorflow-metadata>=1.1.0,<1.2.0.
• Depends on tfx-bsl>=1.1.0,<1.2.0.

Breaking Changes

• N/A

Deprecations

• N/A

TensorFlow Model Analysis 0.31.0

Major Features and Improvements

• N/A

Bug fixes and other Changes

• Depends on apache-beam[gcp]>=2.29,<3.
• Depends on tensorflow>=1.15.2,!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,<3.
• Depends on tensorflowjs>=3.6.0,<4.
• Depends on tensorflow-metadata>=1.0.0,<1.1.0.
• Depends on tfx-bsl>=1.0.0,<1.1.0.

Breaking Changes

• N/A

Deprecations

• N/A

TensorFlow Model Analysis 0.26.1

Major Features and Improvements

• N/A

Bug fixes and other changes

• Fix support for exporting the UI from a notebook to a standalone HTML page.
• Depends on apache-beam[gcp]>=2.25,!=2.26,<2.29.
• Depends on numpy>=1.16,<1.20.

Breaking changes

• N/A

Deprecations

• N/A