Openvalidators #2
Conversation
Eugene-hu
commented
Sep 6, 2023
•
Eugene-hu
commented
- adds openvalidators to the repo
- The requirements of the validator are downgraded to allow for easier testing
- A lot of the reward models are being turned off for the same reasons
| from prompting.validators.reward import RewardModelType | ||
|
|
||
|
|
||
| @dataclass |
There was a problem hiding this comment.
This is quite nice to have for events. Stealing this for the miner side 👍
| from .config import RewardModelType | ||
| from .reward import BaseRewardModel | ||
|
|
||
| blacklist = ["That is an excellent question."] |
There was a problem hiding this comment.
Assuming this list will grow over time, should we consider moving to a text/csv file or some other solution for loading at startup?
(We don't need to do anything now, curious what your thoughts are moving forward)
There was a problem hiding this comment.
Thats a good point, This was actually one of the earlier reward models. I think static reward text blocking would be too slow for the network, and we should probably remove it. The main use of the blacklist model is now to prevent repeated responses in the same chain of thought prompting.
https://github.com/opentensor/subnet-1/pull/2/files/d40d3859c3d5bb620b8d91d7af9c466fba244542#diff-79ab634153b24d60d2321b7cd4f751475a4f72b306219290eb76a6f218712b23R213
prompting/validators/utils.py
Outdated
There was a problem hiding this comment.
Could this go into a repo-wide utils folder/file?
Not sure where it makes most sense for this to live if we're using wandb for both miners and validators, albeit separate instances. (This is great as-is, no action needed)
There was a problem hiding this comment.
There are a few functions here that are specific for the validators ie sync_metagraph. Are there any functions besides the wandb ones that might be useful for the miners as well? If so, I think moving those functions to shared utils would be great
