Add Metrics to Metadata

[blueoceancruiser]

I'm still reviewing the package, so my apologies if I missed it, but I don't see where any scoring metrics are retained in the metadata. Perhaps this is better stored in a database, but it would be nice to have scores in the metadata across the experiments.

[nlathia]

@blueoceancruiser Thanks for raising this! I can see it's not very clear in the documentation, so I'll get this updated.

To answer your question: the next release of modelstore will allow you to upload a dictionary of extra meta data with any model, using the extra_metadata= kwarg. This is more generic than scoring metrics, but can be used to track metrics as well.

Here's the issue that lead to this feature being developed:

• Feature: Allow adding additional information to metadata on model upload #184

In the model upload code, that kwarg is used to set an additional field in the meta data Summary data class. That dictionary will then be returned to you if you use the get_model_info() function.

You can use this to add scoring metrics to the meta data -- for example something like this:

# Train a model & compute its metrics
model = train()
precision, recall = score_model(model, X_test, y_test)

# Create a dictionary with the scoring metrics
metrics = {"precision": precision, "recall": recall}

# Upload the model with the extra meta data
meta_data = model_store.upload("my-domain", model, extra_metadata=metrics)

print(meta_data)

Let me know if this helps 😄 . If you can share a bit more about what you hoped to see / what your use case is, I can factor that feedback into future development plans.

[raulmarindev]

Hello! We are starting to use modelstore, and we like a lot what it offers. Currently, we are missing this specific feature. Any news regarding when this will be available?

[nlathia]

@raulmarindev 👋🏽 hi! This feature is out as ofmodelstore>=0.0.75

https://github.com/operatorai/modelstore/blob/main/CHANGELOG.md#modelstore-0075-september-2022

Let me know how well it works for you & please feel free to open an issue or reach out in the MLOps Community slack (the #oss-modelstore channel) if you find any issues.

[raulmarindev]

That's amazing! Thanks