Expose stop_sequence at command line

[lewtun]

Models like llama-3 use a chat template where the expected stop sequence is <|eot_id|> instead of the common EOS token used in other models. This means that generative benchmarks like ifeval continue generating past the EOS token and give incorrect results.

One way to handle this would be to either include this special token as a default in the generative benchmarks or alternatively expose --stop_sequence as an argument in the main script that users can control.

[clefourrier]

Hi @lewtun , just to clarify, is <|eot_id|> a different stop token for the chat template specifically?
And is the model using <|end_of_text|> for the rest?

Other question, do you think fixing #16 would be enough to solve this too?