[Feature] *_batch_generate* function, add the MultiTokenEOSCriteria

[jiangjin1999]

jiangjin1999: In _batch_generate function, add the MultiTokenEOSCriteria feature to speed up inference.

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

During the model generation phase, we sometimes need to set certain stop words to prompt the model to cease generation. The conventional Huggingface code supports token-based termination only, whereas OpenCompass has introduced support for multi-token termination through the MultiTokenEOSCriteria class. Notably, OpenCompass previously only permitted the use of MultiTokenEOSCriteria within the _single_generate context (with no-batch-padding implying _single_generate).

Modification

• Modification 1: Our enhancements have enabled the use of MultiTokenEOSCriteria within the _batch_generate function.
• Modification 2: Some open-source models, such as "qwen," have unique end-of-sequence (EOS) tokens. We have also implemented a simple filtering mechanism for these models.

Advantages:

• Our modifications facilitate the use of multi-token termination during batch inference, which significantly boosts inference efficiency.
• The continuation of text from the base model incurs additional resource consumption, and our changes have addressed this issue to a certain extent.
• Comparison of Inference Efficiency: We conduct experiments to validate. Experimental setup: for llama2-13b-base & 2-shots. In the past: the DROP dataset took 1:33:01 to process; Currently: the DROP dataset takes 35:35 to process. And there is no difference between the evaluation results of the two.

Limitations:

• The application of MultiTokenEOSCriteria during batch inference will halt generation based on the longest case within the current batch. This approach does not entirely eliminate the extra resource consumption caused by text continuation.

BC-breaking (Optional)

No

Use cases (Optional)

Previously, if stopping criteria were set in the config, it was necessary to append the --no-batch-padding flag when running the run.py script. Now, this step is no longer required, and you can execute the script directly without adding that flag.

Checklist

Before PR:

• Pre-commit or other linting tools are used to fix the potential lint issues.
• Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
• The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
• The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

• If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
• CLA has been signed and all committers have signed the CLA in this PR.

[Leymore]

LGTM