Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion
-
Updated
Aug 5, 2024 - Python
Specify a github or local repo, github pull request, arXiv or Sci-Hub paper, Youtube transcript or documentation URL on the web and scrape into a text file and clipboard for easier LLM ingestion
The ChatGPT Long Term Memory package is a powerful tool designed to empower your projects with the ability to handle a large number of simultaneous users and external sources.
Predict the exact openai token usage of functions
A custom GPT based on [Zero To Hero](https://karpathy.ai/zero-to-hero.html) utilizing tiktoken with the intent to augment AI Transformer-model education and reverse engineer GPT models from scratch.
Count tokens in a text file.
Simple wrapper around tiktoken to use it in your favorite language.
Logseq Markdown Tokenizer is a Python application that tokenizes and estimates prices for one to many markdown files.
Tiktoken-counter as standalone API
В рамках хакатона был создан телеграм бот для консультации с клиентами с сфере обслуживания магазина "Ашан"
console based game based on a llm
This project harnesses the power of Tiktoken and the OpenAI API to create a Python Streamlit web application with a primary focus on token counting and price estimation.
An Industry Standard Tokenizer, purposed for large-scale language models like OpenAI's GPT Series.
Web crawling GUI app for non-govermental events, easy integration with custom proxy, user agents, ApenAI Api and Redis Stack. Enjoy!
Add a description, image, and links to the tiktoken topic page so that developers can more easily learn about it.
To associate your repository with the tiktoken topic, visit your repo's landing page and select "manage topics."