Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FEAT: Refactored SelfAskGptClassifier into SelfAskScore class and added Likert scale scoring #154

Merged
merged 33 commits into from
Apr 25, 2024

Conversation

blakebullwinkel
Copy link
Contributor

Description

This PR refactors SelfAskGptClassifier and adds SelfAskGptLikertScorer, which scores responses on a Likert scale. Both of these scorers inherit from a generalized SelfAskScore class, which takes both a system prompt and a classifier as input. This also enables the creation of custom scorers that require new system prompts. Kudos to @dlmgary for writing the original implementation of SelfAskScore!

In addition, this PR includes many new classifiers to score different types of content in score/content_classifiers.

Tests and Documentation

This PR updates the scoring.ipynb notebook and provides an example of how to use SelfAskGptClassifier, SelfAskGptLikertScorer, as well as the base SelfAskScore class. I made change to scoring.py and ran JupyText to format the demo notebook.

Currently, no tests exist for scorers, and this PR does not add any (yet).

Copy link
Contributor

@romanlutz romanlutz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to add tests here? Personally, I'm fine with adding a story for that, too, since you worked on a part of code that didn't have tests.

pyrit/score/gpt_classifier.py Outdated Show resolved Hide resolved
pyrit/models/models.py Outdated Show resolved Hide resolved
doc/code/scoring.py Outdated Show resolved Hide resolved
pyrit/score/__init__.py Outdated Show resolved Hide resolved
Copy link
Contributor

@nina-msft nina-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops apologies - seems like I was reviewing on an older commit but I think most of my comments still stand. Lmk if you want to talk about anything :)

Copy link
Contributor

@nina-msft nina-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me - there are some nits that I noticed on spelling etc. that I can handle in my next PR that'll come out re: this. Great work!

@romanlutz romanlutz merged commit 822e7f6 into Azure:main Apr 25, 2024
3 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants