-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature/track raw score #63
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What if we streamline the whole process into an abstraction where:
get_rewards
return aBaseRewardEvent
,BaseRewardModel.apply
returns aBaseRewardEvent
BaseRewardEvent
has the logic to convert it itself into a dictionary, from which we can spread its properties later in theforward
call and in the event schema- Complement the event schema with the expected new fields
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM (BTW I set the PR as ready for review by accident, apologies for that)
One thing missing though:
- Fixing branch conflicts
with torch.no_grad(): | ||
inputs = self.tokenizer(prompt, completion, return_tensors="pt").to( | ||
self.device | ||
) | ||
return float(self.model(**inputs).logits[0].cpu().detach()) | ||
reward_event.reward = float(self.model(**inputs).logits[0].cpu().detach()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shouldn't this be reward_event.score
since its float? where is reward_event.reward:Bool
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rlhf is a reward model (not filter model), so there is no bool reward, and the score = reward
Co-authored-by: Steffen Cruz <steffenjcruz@gmail.com>
Co-authored-by: Steffen Cruz <steffenjcruz@gmail.com>
Co-authored-by: Steffen Cruz <steffenjcruz@gmail.com>
Co-authored-by: Steffen Cruz <steffenjcruz@gmail.com>
Co-authored-by: Steffen Cruz <steffenjcruz@gmail.com>
Co-authored-by: Steffen Cruz <steffenjcruz@gmail.com>
Co-authored-by: Steffen Cruz <steffenjcruz@gmail.com>
Co-authored-by: Steffen Cruz <steffenjcruz@gmail.com>
…or/text-prompting into feature/track_raw_score
No description provided.