Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a method to model cards to generate an overview of sections #302

Closed
BenjaminBossan opened this issue Feb 17, 2023 · 6 comments · Fixed by #305
Closed

Add a method to model cards to generate an overview of sections #302

BenjaminBossan opened this issue Feb 17, 2023 · 6 comments · Fixed by #305

Comments

@BenjaminBossan
Copy link
Collaborator

This would be nice to have.

Right now, when working with model cards, it's not so easy to get a good overview of what sections currently do or don't exist. The repr of the object gives some indication, but besides sections, it also displays the contents, which is distracting. It also truncates long strings, which can obscure the section names.

Having a nice overview of the section names, especially when it comes to nested subsections, would be really helpful. Especially when I want to add a subsection to an existing subsection, since I need to know the full name to the subsection, it requires some digging right now.

For now, I'm using something like this:

from skops.card._model_card import Section

def iterate_key_section_content(
    data: dict[str, Section],
    parent_section: str = "",
    parent_keys: list[str] | None = None,
    level: int = 0,
):
    parent_keys = parent_keys or []

    for key, val in data.items():
        if not getattr(val, "visible", True):
            continue

        if parent_section:
            title = "/".join((parent_section, val.title))
        else:
            title = val.title

        yield title, level

        if val.subsections:
            yield from iterate_key_section_content(
                val.subsections,
                parent_section=title,
                parent_keys=parent_keys + [key],
                level=level + 1,
            )

With that function, I can do something like this:

>>> for title, level in iterate_key_section_content(model_card._data):
...     print(title)
Model description
Model description/Intended uses & limitations
Model description/Training Procedure
Model description/Training Procedure/Hyperparameters
Model description/Training Procedure/Model Plot
Model description/Evaluation Results
How to Get Started with the Model
Model Card Authors
Model Card Contact
Citation
@lazarust
Copy link
Contributor

@BenjaminBossan I can take this if it's not already taken!

@BenjaminBossan
Copy link
Collaborator Author

@lazarust Yes, feel free to give it a shot. Use this thread to discuss designs.

@lazarust
Copy link
Contributor

@BenjaminBossan Here are some of my ideas. I'm thinking this should live in skops/utils/modelutils.py. For printing the structure I'm thinking of something like

Model description
 |----Intended uses & limitations
 |----Training Procedure
 |          |----Hyperparameters
 |          |----Model Plot
 |----Evaluation Results
How to Get Started with the Model
Model Card Authors
Model Card Contact
Citation

I'm still thinking through how to dynamically generate this, but what're your thoughts?

@BenjaminBossan
Copy link
Collaborator Author

I was thinking about having something like the snippet I posted as a (probably) private method on the Card class. Maybe we could then add a create_toc method or something that will return a table of contents str in markdown style. For some users, that could be a nice addition for the model card, but on GH and Hugging Face, the TOC is created automatically, so I'm not sure.

We could also add a function that gives a more visually pleasing output based on the private method, like you showed above. I think we can just put it into skops.card, no need for a separate module for now. For the visual style, we could use unicode characters like and .

I'm still thinking through how to dynamically generate this

We're probably not the first Python lib that tries to visualize a tree in this style. Perhaps there is some code out there that could be used for inspiration.

@lazarust
Copy link
Contributor

@BenjaminBossan After thinking/working on this a little, I wonder if adding the separate function to create the visually appealing version is worth it since users could just use print(card.create_toc() to see:

- [Model description](#Model description)
    - [Intended uses & limitations](#Intended uses & limitations)
    - [Training Procedure](#Training Procedure)
        - [Hyperparameters](#Hyperparameters)
        - [Model Plot](#Model Plot)
    - [Evaluation Results](#Evaluation Results)
- [How to Get Started with the Model](#How to Get Started with the Model)
- [Model Card Authors](#Model Card Authors)
- [Model Card Contact](#Model Card Contact)
- [Citation](#Citation)

To me, that seems pretty readable in the terminal, what're your thoughts? Would adding the tree structure be useful?

@BenjaminBossan
Copy link
Collaborator Author

@lazarust I'll reply on the actual PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants