Increment max penalty + complement task with layout criteria #86
Conversation
p-ferreira
requested review from
steffencruz,
isabella618033,
Eugene-hu,
bkb2135 and
schampoux
| else: | ||
| layout_criteria = MatchLayoutCriteria( | ||
| layout_type=LayoutMatchTypeEnum.NUMBERED_LIST, | ||
| penalty=0.5, |
There was a problem hiding this comment.
it will get the default text Your response should be ordered in format of numbered list. The bullet point criteria would list like "unordered list", that's why I override it to be "bullet points"
| else: | ||
| layout_criteria = MatchLayoutCriteria( | ||
| layout_type=LayoutMatchTypeEnum.NUMBERED_LIST, | ||
| penalty=0.5, |
There was a problem hiding this comment.
| penalty=0.5, | |
| penalty=0.5, | |
| text="Your response should be ordered in format of numbered points.", |
There was a problem hiding this comment.
It will get it by default Your response should be ordered in format of numbered list
|
Extra comment: what are the existing layout criteria used by qg and qa?
|
qg:
qa:
|
|
Do we specify the number of list items? |
|
and why not update the qa criteria to sometimes allow lists? this is actually reasonable behaviour at this point |
no, it's a binary verification, it was too last minute to bring this sort of complexity to the release imho |
because most of the miners already return in list layout already since they are leveraging the qg+qa / key takeaways exploits, so forcing it to be simple punishes those type of exploits |
|
good stuff. thanks! |
Note: The tasks of qg and qa already follow a defined layout, so the expansion was performed on the summarization task