New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Introduce LM cache. #11

Closed

copybara-service wants to merge 0 commits into main from test_567398279

copybara-service bot commented Sep 21, 2023

Introduce LM cache.

Add interface lf.LMCache
Implement lf.llms.cache.InMemory.
Support custom key function and ttl.

NOTE: cache key is determined by (model_id, sampling_options, prompt).

Usage:

lm = lf.llms.Gpt35(cache=lf.llms.cache.InMemory())
print(lf.LangFunc('Intro to the U.S.A', lm=lm))

copybara-service bot force-pushed the test_567398279 branch from 69da052 to 32b6c5e Compare

September 21, 2023 20:54

copybara-service bot closed this

copybara-service bot force-pushed the test_567398279 branch from 32b6c5e to 8e2d7a4 Compare

September 21, 2023 20:55

copybara-service bot deleted the test_567398279 branch

September 21, 2023 20:55

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet