Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Yi-Large Preview to AlpacaEval #304

Merged
merged 4 commits into from
May 8, 2024
Merged

Conversation

HyperdriveHustle
Copy link
Contributor

Adding the result of yi-large-preview model (The model will soon be launched on the 01.ai API.).

By the way, our prompt template should be:

<|startoftext|><|im_start|>user
{instruction}<|im_end|>

But when I configure requires_chatml: True in configs.yaml and <|startoftext|> in prompt template at the same time, the code will raise an exception.
And it seems that the prompt template does not take effect during the process of calling the API to obtain the model output.
Do I need to make modifications?

@YannDubs
Copy link
Collaborator

YannDubs commented May 8, 2024

Impressive submission @HyperdriveHustle!

concerning the prompt format can you explain more? By using requires_chatml the prompt

<|im_start|>user
{instruction}<|im_end|>

will be converted to (using prompt_to_chatml) :

[{'content': instruction, 'role': 'user'}]

when calling openai completion API. Where do you want the <|startoftext|> to be? In other words, what the list of dict that you want to call OpenAI's client with?

I will merge the current submission but am happy to change the prompt_to_chatml to change the prompt

@YannDubs YannDubs merged commit f693f1f into tatsu-lab:main May 8, 2024
2 checks passed
@HyperdriveHustle
Copy link
Contributor Author

HyperdriveHustle commented May 9, 2024

I see, I misunderstood before. There's no issue with the current prompt template!
Thanks for clarifying @YannDubs ~

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants