This project implements token calculation for OpenAI's gpt-4 and gpt-3.5-turbo model, specifically using `cl100k_base` encoding.
-
Updated
Aug 5, 2024 - C#
This project implements token calculation for OpenAI's gpt-4 and gpt-3.5-turbo model, specifically using `cl100k_base` encoding.
Add a description, image, and links to the tiktoken topic page so that developers can more easily learn about it.
To associate your repository with the tiktoken topic, visit your repo's landing page and select "manage topics."