This is the repo for paper "Auto Arena of LLMs: Automating LLM Evaluations with Agent Peer-battles and Committee Discussions".

Project Website | Paper | Leaderboard

How to use the repository

Prepare the environment:

1. Set up the environment using: conda env create -f env.yml
2. Activate the environment with: conda activate LLM_Eval
3. Make sure you have the environment variables listed in APIs

Before including any participants, make sure:

1. The participant's calling function is written in the "generate_response" function inside "utils/api_utils.py".
2. The participant's MMLU score is included in "data/MMLU.csv". If the participant doesn't have an MMLU score, fill in "-1" and it will be initially paired with the median candidate.

To run a tournament, first change the "player_names" variable in the python files to adjust the tournament members. Example commands:

1. English: python run_tournament.py --tournament_dir data/main_tour_40
2. Chinese: python run_tournament.py --tournament_dir data/main_tour_40_zh --language zh

To add a new participant to a finished tournament, here is an example command:

1. python run_tournament_add_participant.py --tournament_dir data/main_tour_40 --add_participant SenseChat_5

To run debates between a pair of selected models, here is an example command:

1. python run_LD_pair.py --model_a Qwen/Qwen1.5-72B-Chat --model_b claude-3-haiku-20240307

We release 2 questions per category as demos for each debate on the website. The full results are not pushed to the website. An analysis including all figures in the papers was run in notebook result_analysis.ipynb.