Token counts are incorrect when request includes function messages

[kylewlacy]

When using the relatively recent "Functions" feature of the ChatGPT API, it seems like tiktoken-rs underestimates the total number of tokens in the request. Here's a minimal example request:

{
  "messages": [
    {
      "role": "system",
      "content": "You are a friendly chatbot.\n"
    },
    {
      "role": "assistant",
      "content": "Hello, I am a friendly chatbot!\n"
    },
    {
      "role": "user",
      "content": "What is the weather in New York?"
    },
    {
      "content": "",
      "function_call": {
        "arguments": "{\n  \"city\": \"New York\"\n}",
        "name": "get_weather"
      },
      "role": "assistant"
    },
    {
      "role": "function",
      "name": "get_weather",
      "content": "{\"temperature\": 72, \"conditions\": \"partly_cloudy\"}"
    }
  ],
  "model": "gpt-4-0613",
  "temperature": 0,
  "stream": false
}

I get this response from OpenAI:

{
    // ...
    "usage": {
        "prompt_tokens": 78,
        "completion_tokens": 19,
        "total_tokens": 97
    }
}

...indicating the request consumed 78 tokens for the prompt. However, tiktoken_rs::num_tokens_from_messages returns a value of 66 tokens.

[zurawiki]

Good point @kylewlacy. Do you know if OpenAI has any documentation on how tokens are counted?

Their cookbook documentation at https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken doesn't seem to mention functions

I suspect part of the issue is that the "function_call" property is not considered when calculating the length of a message.

See the code below making use of role and content while function_call is unused.

tiktoken-rs/tiktoken-rs/src/api.rs

Lines 117 to 129 in 8b5dc54

	for message in messages {
	num_tokens += tokens_per_message;
	num_tokens += bpe
	.encode_with_special_tokens(&message.role.to_string())
	.len() as i32;
	num_tokens += bpe
	.encode_with_special_tokens(&message.content.clone().unwrap_or_default())
	.len() as i32;
	if let Some(name) = &message.name {
	num_tokens += bpe.encode_with_special_tokens(name).len() as i32;
	num_tokens += tokens_per_name;
	}
	}