Upgrade hf chat

[j316chuck]

Description

Upgrade hf chat to use the new HF chat template format: https://huggingface.co/docs/transformers/main/en/chat_templating#how-do-i-use-chat-templates

Test

python3 hf_chat.py --max_new_tokens 10 -n $hf_model_path

Loading HF tokenizer...
/llm-foundry/scripts/inference/hf_chat.py:350: UserWarning: pad_token_id is not set for the tokenizer. Using eos_token_id as pad_token_id.
  warnings.warn(
NOT using autocast...
Warming up...
Assistant:

 kel_ax_REPORT hioverse Fireplace seasoning(infoBegan agli eb

Took 4.54 seconds
Starting conversation...
Enter your message below.
- Hit return twice to send input to the model
- Type 'clear' to restart the conversation
- Type 'history' to see the conversation
- Type 'quit' to end
- Type 'system' to change the system prompt

User:
Assistant:
hi

Neither.YearwpinguResidentsdependent elabor'} experi ApiResponse

Took 3.95 seconds
User:
Assistant:
 Giuldating_addr@example AirportVarInsn paramount turningі��

Took 3.97 seconds
User:
history

history: [{ 'role': system, 'content': You are a friendly chatbot who aims to be helpful and honest. }, { 'role': user, 'content':  }, { 'role': assistant, 'content': Neither.YearwpinguResidentsdependent elabor'} experi ApiResponse }, { 'role': user, 'content': hihi }, { 'role': assistant, 'content':  Giuldating_addr@example AirportVarInsn paramount turningі�� }]
User:
history_fmt

history: <|im_start|>system
You are a friendly chatbot who aims to be helpful and honest.<|im_end|>
<|im_start|>user
<|im_end|>
<|im_start|>assistant
Neither.YearwpinguResidentsdependent elabor'} experi ApiResponse<|im_end|>
<|im_start|>user
hihi<|im_end|>
<|im_start|>assistant
 Giuldating_addr@example AirportVarInsn paramount turningі��<|im_end|>
<|im_start|>assistant

User:
clear

User:
history_fmt

history: 
User:
history

history: [{ 'role': system, 'content': You are a friendly chatbot who aims to be helpful and honest. }]
User:
history_fmt

history: 
User:
^CTraceback (most recent call last):
  File "/llm-foundry/scripts/inference/hf_chat.py", line 395, in <module>
    main(parse_args())
  File "/llm-foundry/scripts/inference/hf_chat.py", line 391, in main
    conversation()
  File "/llm-foundry/scripts/inference/hf_chat.py", line 150, in __call__
    line = input()
KeyboardInterrupt
^C^C^C
root@6b4af5be-241a-4961-8201-659d65a1e591-0:/llm-foundry/scripts/inference# ^C
root@6b4af5be-241a-4961-8201-659d65a1e591-0:/llm-foundry/scripts/inference# ^C
root@6b4af5be-241a-4961-8201-659d65a1e591-0:/llm-foundry/scripts/inference# ^C
root@6b4af5be-241a-4961-8201-659d65a1e591-0:/llm-foundry/scripts/inference# python3 hf_chat.py --max_new_tokens 10 -n databricks/test-dbrx-tiny
Using device=None and device_map='auto'
Using model_dtype=torch.float32
Loading HF Config...
Loading HF model with dtype=torch.float32...
n_params=931283968

Loading HF tokenizer...
/llm-foundry/scripts/inference/hf_chat.py:350: UserWarning: pad_token_id is not set for the tokenizer. Using eos_token_id as pad_token_id.
  warnings.warn(
NOT using autocast...
Warming up...
Assistant:
 Damn mín JR cylindrical LiquidMMShows epile bas Rec

Took 4.54 seconds
Starting conversation...
Enter your message below.
- Hit return twice to send input to the model
- Type 'clear' to restart the conversation
- Type 'history' to see the conversation
- Type 'history_fmt' to see the conversation
- Type 'quit' to end
- Type 'system' to change the system prompt

User:
hello hello

Assistant:
.getStatusCode(states Д primeira Rug WELLamdatreadoptcentration

Took 3.96 seconds
User:
history

history: [{ 'role': system, 'content': You are a friendly chatbot who aims to be helpful and honest. }, { 'role': user, 'content': hello hello }, { 'role': assistant, 'content': .getStatusCode(states Д primeira Rug WELLamdatreadoptcentration }]
User:
history_fmt

history: <|im_start|>system
You are a friendly chatbot who aims to be helpful and honest.<|im_end|>
<|im_start|>user
hello hello<|im_end|>
<|im_start|>assistant
.getStatusCode(states Д primeira Rug WELLamdatreadoptcentration<|im_end|>
<|im_start|>assistant

User:
system 

Assistant:
 sharpen_CONFIGMockgetString.mockitoumbo burgeất smuggling weekdays

Took 3.98 seconds
User:
system

Enter a new system prompt:
hi
User:
history

history: [{ 'role': system, 'content': hi }, { 'role': user, 'content': hello hello }, { 'role': assistant, 'content': .getStatusCode(states Д primeira Rug WELLamdatreadoptcentration }, { 'role': user, 'content': system  }, { 'role': assistant, 'content':  sharpen_CONFIGMockgetString.mockitoumbo burgeất smuggling weekdays }]
User:
clear

User:
history

history: [{ 'role': system, 'content': hi }]
User:
history_fmt

history: 
User:
hi

Assistant:
histNeither oryCOUNTY prefer('=' tm automation elseif MongoClient serialize Implementation

Took 3.91 seconds
User:


history: [{ 'role': system, 'content': hi }, { 'role': user, 'content': hi }, { 'role': assistant, 'content': Neither COUNTY prefer('=' tm automation elseif MongoClient serialize Implementation }]
User:

[samhavens]

I believe there are some args we can remove from the script but otherwise lgtm

[samhavens]

LGTM!