Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH Refactor Model Card #37

Merged
merged 64 commits into from
Jul 27, 2022
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
b0049bd
refactored card class
merveenoyan Jul 18, 2022
1d34c6f
updated example
merveenoyan Jul 18, 2022
e8d27e4
add function and removed adding sections in save
merveenoyan Jul 19, 2022
c9414b4
added plots & more
merveenoyan Jul 20, 2022
6000294
addressed comments
merveenoyan Jul 20, 2022
c9331ae
Update skops/card/_model_card.py
merveenoyan Jul 20, 2022
4901ee0
added docs and fixed tests
merveenoyan Jul 20, 2022
1ba069a
fix test
merveenoyan Jul 20, 2022
7bd29b1
added temporary plot
merveenoyan Jul 21, 2022
8236b6c
make html for docs
merveenoyan Jul 21, 2022
ac466e3
Update examples/plot_model_card.py
merveenoyan Jul 22, 2022
985a971
Update skops/card/_model_card.py
merveenoyan Jul 22, 2022
8b852c3
Update skops/card/_model_card.py
merveenoyan Jul 22, 2022
df7ad96
Update skops/card/_model_card.py
merveenoyan Jul 22, 2022
456acbd
Update examples/plot_model_card.py
merveenoyan Jul 22, 2022
73dfed6
Update skops/card/tests/test_card.py
merveenoyan Jul 22, 2022
3eafa1e
Update examples/plot_model_card.py
merveenoyan Jul 22, 2022
ed17c0e
Update examples/plot_model_card.py
merveenoyan Jul 22, 2022
449c6ca
Update examples/plot_model_card.py
merveenoyan Jul 22, 2022
a93816d
Update skops/card/_model_card.py
merveenoyan Jul 22, 2022
8fddf62
Update skops/card/_model_card.py
merveenoyan Jul 22, 2022
2428a10
Update skops/card/_model_card.py
merveenoyan Jul 22, 2022
2bd553e
Update skops/card/_model_card.py
merveenoyan Jul 22, 2022
c384f59
addressed comments + cleanup
merveenoyan Jul 22, 2022
6514e6f
added documentation for card class
merveenoyan Jul 22, 2022
522d5f4
Update examples/plot_model_card.py
merveenoyan Jul 25, 2022
04da3b6
Update skops/card/_model_card.py
merveenoyan Jul 25, 2022
7479cc8
Update skops/card/_model_card.py
merveenoyan Jul 25, 2022
a8789b5
Update skops/card/_model_card.py
merveenoyan Jul 25, 2022
1aa996e
Update examples/plot_model_card.py
merveenoyan Jul 25, 2022
611c7ca
replaced Path, added fixture, misc nits
merveenoyan Jul 25, 2022
3a0f9a9
replaced Path, added fixture, misc nits
merveenoyan Jul 25, 2022
b921994
removed unnecessary assert
merveenoyan Jul 25, 2022
c312a1f
added plotting option for diagram and associated test
merveenoyan Jul 25, 2022
b7f77d9
make html
merveenoyan Jul 25, 2022
f379b83
fix docs
merveenoyan Jul 25, 2022
535e698
fix docstring user guide
merveenoyan Jul 25, 2022
e9908f3
fix docstring
merveenoyan Jul 25, 2022
a2aaaa7
fix docstring
merveenoyan Jul 25, 2022
5f4b0cc
fix docstring
merveenoyan Jul 25, 2022
27051f4
fix docstring
merveenoyan Jul 25, 2022
d6ded6b
fix docstring
merveenoyan Jul 25, 2022
129d710
fix docstring
merveenoyan Jul 25, 2022
3a0ceec
fix docstring
merveenoyan Jul 25, 2022
0403c0f
fix docstring
merveenoyan Jul 25, 2022
725ba0f
fix expected nothing error
merveenoyan Jul 25, 2022
96453b0
added ellipsis
merveenoyan Jul 25, 2022
1b9caea
added ellipsis
merveenoyan Jul 25, 2022
c9d7c3d
putting ellipsis inside object
merveenoyan Jul 25, 2022
16693fa
putting ellipsis inside object works lol
merveenoyan Jul 25, 2022
e0691ff
putting ellipsis inside object
merveenoyan Jul 25, 2022
9e338b5
swapped namedtemporaryfile with tempdir for windows tests
merveenoyan Jul 25, 2022
f15b7d4
removed ellipsis at every line
merveenoyan Jul 26, 2022
68fd0e9
brought back ellipsis
merveenoyan Jul 26, 2022
2f2b05f
revert whitespaces for ellipsis
merveenoyan Jul 26, 2022
cb6379a
added model card generation to fixtures and added tests to metadata
merveenoyan Jul 26, 2022
57ab801
added model card generation to fixtures and added tests to metadata
merveenoyan Jul 26, 2022
bfc9f4d
misc nits
merveenoyan Jul 26, 2022
adbb9e3
Update examples/plot_model_card.py
merveenoyan Jul 26, 2022
e01143d
removed extra parantheses
merveenoyan Jul 26, 2022
3407014
Merge branch 'model_card_class' of github.com:merveenoyan/skops into …
merveenoyan Jul 26, 2022
8eaf98b
addressed comments
merveenoyan Jul 27, 2022
37858c1
Update skops/card/_model_card.py
merveenoyan Jul 27, 2022
ce57f6d
Update skops/card/_model_card.py
merveenoyan Jul 27, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 20 additions & 22 deletions examples/plot_model_card.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,12 +64,6 @@
# Then, we pass information other than metadata in kwargs.
# We'll initialize a local repository and save the card with the model in it.

limitations = "This model is not ready to be used in production."
model_description = (
"This is a HistGradientBoostingClassifier model trained on breast cancer dataset."
" It's trained with Halving Grid Search Cross Validation, with parameter grids on"
" max_leaf_nodes and max_depth."
)
license = "mit"

card_data = CardData(
Expand All @@ -79,21 +73,9 @@
metrics=["acc"],
)

model_card_authors = "skops_user"
get_started_code = (
"import pickle\nwith open(dtc_pkl_filename, 'rb') as file:\nclf = pickle.load(file)"
)
citation = "bibtex\n@inproceedings{...,year={2020}}"

model_card = card.create_model_card(
model,
card_data=card_data,
limitations=limitations,
model_description=model_description,
citation_bibtex=citation,
model_card_authors=model_card_authors,
get_started_code=get_started_code,
)

model_card = card.Card(model)


_, pkl_name = mkstemp(prefix="skops-", suffix=".pkl")

Expand All @@ -105,4 +87,20 @@
model=pkl_name, requirements=[f"scikit-learn={sklearn.__version__}"], dst=local_repo
merveenoyan marked this conversation as resolved.
Show resolved Hide resolved
adrinjalali marked this conversation as resolved.
Show resolved Hide resolved
)

model_card.save(os.path.join(f"{local_repo}", "README.md"))
limitations = "This model is not ready to be used in production."
model_description = (
"This is a HistGradientBoostingClassifier model trained on breast cancer dataset."
" It's trained with Halving Grid Search Cross Validation, with parameter grids on"
" max_leaf_nodes and max_depth."
)
model_card_authors = "skops_user"
get_started_code = (
"import pickle\nwith open(dtc_pkl_filename, 'rb') as file:\nclf = pickle.load(file)"
)

model_card.add("citation", "bibtex\n@inproceedings{...,year={2020}}")
model_card.add("get_started_code", get_started_code).add(
"model_card_authors", model_card_authors
)
model_card.add("limitations", limitations).add("model_description", model_description)
model_card.save(os.path.join(f"{local_repo}", "README.md"), card_data=card_data)
114 changes: 62 additions & 52 deletions skops/card/_model_card.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,55 +7,65 @@
import skops


def _extract_estimator_config(model):
"""Extracts estimator configuration and renders them into a vertical table.

Parameters
----------
model (estimator): scikit-learn pipeline or model.

Returns
-------
str:
Markdown table of hyperparameters.
"""
hyperparameter_dict = model.get_params(deep=True)
table = "| Hyperparameters | Value |\n| :-- | :-- |\n"
for hyperparameter, value in hyperparameter_dict.items():
table += f"| {hyperparameter} | {value} |\n"
return table


def create_model_card(
model,
card_data,
**card_kwargs,
):
"""Creates a model card for the model and saves it to the target directory.

Parameters:
----------
model: estimator
scikit-learn compatible estimator.
card_data: CardData
CardData object.
card_kwargs:
Card kwargs are information you can pass to fill in the sections of the
card template, e.g. model_description, citation_bibtex, get_started_code.
"""
ROOT = skops.__path__
model_plot = re.sub(r"\n\s+", "", str(estimator_html_repr(model)))
hyperparameter_table = _extract_estimator_config(model)
card_data.library_name = "sklearn"
template_path = card_kwargs.get("template_path")
if template_path is None:
template_path = os.path.join(f"{ROOT[0]}", "card", "default_template.md")
card_kwargs["template_path"] = template_path
card = ModelCard.from_template(
card_data=card_data,
hyperparameter_table=hyperparameter_table,
model_plot=model_plot,
**card_kwargs,
)

return card
class Card:
adrinjalali marked this conversation as resolved.
Show resolved Hide resolved
def __init__(self, model):
merveenoyan marked this conversation as resolved.
Show resolved Hide resolved
self.model = model
self.hyperparameters_table = self.extract_estimator_config(self.model)
self.model_plot = re.sub(r"\n\s+", "", str(estimator_html_repr(model)))
merveenoyan marked this conversation as resolved.
Show resolved Hide resolved
merveenoyan marked this conversation as resolved.
Show resolved Hide resolved
self.template_sections = {}

def add(self, section, value):
"""Takes values to fill model card template.
merveenoyan marked this conversation as resolved.
Show resolved Hide resolved
Parameters:
----------
section: str
Section in the template.
value: str
Value to fill in the section.
"""
merveenoyan marked this conversation as resolved.
Show resolved Hide resolved
self.template_sections[section] = value
return self
merveenoyan marked this conversation as resolved.
Show resolved Hide resolved
adrinjalali marked this conversation as resolved.
Show resolved Hide resolved

def add_inspection(self):
pass

def save(self, path, card_data):
"""Fills model card template, renders and saves it as markdown file to the target directory.
merveenoyan marked this conversation as resolved.
Show resolved Hide resolved

Parameters:
----------
card_data: CardData
CardData object.
"""
ROOT = skops.__path__
merveenoyan marked this conversation as resolved.
Show resolved Hide resolved
card_data.library_name = "sklearn"
template_path = self.template_sections.get("template_path")
if template_path is None:
template_path = os.path.join(f"{ROOT[0]}", "card", "default_template.md")
self.template_sections["template_path"] = template_path
card = ModelCard.from_template(
card_data=card_data,
hyperparameter_table=self.hyperparameter_table,
model_plot=self.model_plot,
**self.template_sections,
)
card.save(path)

def extract_estimator_config(model):
merveenoyan marked this conversation as resolved.
Show resolved Hide resolved
"""Extracts estimator configuration and renders them into a vertical table.
merveenoyan marked this conversation as resolved.
Show resolved Hide resolved

Parameters
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The method takes no arguments, so please remove.

----------
model (estimator): scikit-learn pipeline or model.

Returns
-------
str:
Markdown table of hyperparameters.
"""

hyperparameter_dict = model.get_params(deep=True)
table = "| Hyperparameters | Value |\n| :-- | :-- |\n"
for hyperparameter, value in hyperparameter_dict.items():
table += f"| {hyperparameter} | {value} |\n"
return table