Fix #320. Reuse the asyncio loop between sync calls #336
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…onents might cache it).
ProKil
added a commit
to sotopia-lab/sotopia
that referenced
this pull request
May 27, 2024
ProKil
added a commit
to sotopia-lab/sotopia
that referenced
this pull request
Jun 17, 2024
* add benchmark social agents * add benchmark agents * Add sotopia_benchmark cli api * fix pre-commit * add evaluator model argument * finish benchmarking * benchmark done * chore: Fix formatting issue in redis_stats.ipynb and cli.py * switch back to LLM_Name * merge main * add together ai * fix naming error * roll back to llama2 * chore: Update langchain-together dependency to version 0.1.2 * use chatopenai for together models * add logging * fix pre-commit * add more logging options * probably fix the event loop closed error: following NVIDIA/NeMo-Guardrails#336 * modify cli; fix model position bug * chore: Update benchmark tag to "benchmark_{model}_final" * Refactor _iterate_all_env_agent_combo_not_in_db function * chore: Update python version to 3.11.2 * change to dict comparison * ignore jsonl * ✨ finish benchmarking script * chore: Refactor server.py and redis_stats.ipynb * add type ignore * push for the eval * Refactor run_async_benchmark_in_batch function * Refactor run_async_benchmark_in_batch function * add doc * precommit fix * pre-commit * refactor * update w feedback * pre commit * chore: Update authors in pyproject.toml and fetch benchmark_agents.json from Hugging Face API * hotfix * chore: Remove unnecessary type hint in benchmark/cli.py --------- Co-authored-by: Hao <prokilchu@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Other components might cache it, like
openaiand it would break things.