-
Notifications
You must be signed in to change notification settings - Fork 281
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FEAT: Refactored SelfAskGptClassifier into SelfAskScore class and added Likert scale scoring #154
Conversation
… of a generalized SelfAskScore class, which takes a system prompt and a content classifier
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we want to add tests here? Personally, I'm fine with adding a story for that, too, since you worked on a part of code that didn't have tests.
…ry to avoid circular import error
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Whoops apologies - seems like I was reviewing on an older commit but I think most of my comments still stand. Lmk if you want to talk about anything :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me - there are some nits that I noticed on spelling etc. that I can handle in my next PR that'll come out re: this. Great work!
Description
This PR refactors
SelfAskGptClassifier
and addsSelfAskGptLikertScorer
, which scores responses on a Likert scale. Both of these scorers inherit from a generalizedSelfAskScore
class, which takes both a system prompt and a classifier as input. This also enables the creation of custom scorers that require new system prompts. Kudos to @dlmgary for writing the original implementation ofSelfAskScore
!In addition, this PR includes many new classifiers to score different types of content in
score/content_classifiers
.Tests and Documentation
This PR updates the
scoring.ipynb
notebook and provides an example of how to useSelfAskGptClassifier
,SelfAskGptLikertScorer
, as well as the baseSelfAskScore
class. I made change toscoring.py
and ran JupyText to format the demo notebook.Currently, no tests exist for scorers, and this PR does not add any (yet).