-
Notifications
You must be signed in to change notification settings - Fork 14
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Openvalidators #2
Conversation
Eugene-hu
commented
Sep 6, 2023
•
edited
Loading
edited
- adds openvalidators to the repo
- The requirements of the validator are downgraded to allow for easier testing
- A lot of the reward models are being turned off for the same reasons
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! Left a couple comments we can discuss for later, let's merge it!
from prompting.validators.reward import RewardModelType | ||
|
||
|
||
@dataclass |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is quite nice to have for events. Stealing this for the miner side 👍
from .config import RewardModelType | ||
from .reward import BaseRewardModel | ||
|
||
blacklist = ["That is an excellent question."] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Assuming this list will grow over time, should we consider moving to a text/csv file or some other solution for loading at startup?
(We don't need to do anything now, curious what your thoughts are moving forward)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thats a good point, This was actually one of the earlier reward models. I think static reward text blocking would be too slow for the network, and we should probably remove it. The main use of the blacklist model is now to prevent repeated responses in the same chain of thought prompting.
https://github.com/opentensor/subnet-1/pull/2/files/d40d3859c3d5bb620b8d91d7af9c466fba244542#diff-79ab634153b24d60d2321b7cd4f751475a4f72b306219290eb76a6f218712b23R213
prompting/validators/utils.py
Outdated
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could this go into a repo-wide utils folder/file?
Not sure where it makes most sense for this to live if we're using wandb for both miners and validators, albeit separate instances. (This is great as-is, no action needed)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are a few functions here that are specific for the validators ie sync_metagraph
. Are there any functions besides the wandb ones that might be useful for the miners as well? If so, I think moving those functions to shared utils would be great