-
Notifications
You must be signed in to change notification settings - Fork 399
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
12 changed files
with
449 additions
and
5 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,3 +3,4 @@ __pycache__ | |
*_version.py | ||
docs/build | ||
.coverage | ||
.idea/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
# Llama.cpp | ||
|
||
!!! Installation | ||
|
||
You need to install the `llama-cpp-python` library to be able to use these models in Outlines. | ||
|
||
Outlines provides an integration with [Llama.cpp](https://github.com/ggerganov/llama.cpp) using the [llama-cpp-python library](https://github.com/abetlen/llama-cpp-python). Llamacpp allows to run quantized models on machines with limited compute. | ||
|
||
Assuming [Phi2's weights](https://huggingface.co/TheBloke/phi-2-GGUF) are in the current directory: | ||
|
||
```python | ||
from outlines import models, generate | ||
|
||
model = models.llamacpp("./phi-2.Q4_K_M.gguf", device="cpu") | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
from enum import Enum | ||
|
||
import torch | ||
from pydantic import BaseModel, constr | ||
|
||
import outlines | ||
|
||
|
||
class Weapon(str, Enum): | ||
sword = "sword" | ||
axe = "axe" | ||
mace = "mace" | ||
spear = "spear" | ||
bow = "bow" | ||
crossbow = "crossbow" | ||
|
||
|
||
class Armor(str, Enum): | ||
leather = "leather" | ||
chainmail = "chainmail" | ||
plate = "plate" | ||
|
||
|
||
class Character(BaseModel): | ||
name: constr(max_length=10) | ||
age: int | ||
armor: Armor | ||
weapon: Weapon | ||
strength: int | ||
|
||
|
||
if __name__ == "__main__": | ||
# Download model from https://huggingface.co/TheBloke/phi-2-GGUF | ||
model = outlines.models.llamacpp("./phi-2.Q3_K_M.gguf", device="cpu") | ||
|
||
# Construct guided sequence generator | ||
generator = outlines.generate.json(model, Character, max_tokens=512) | ||
|
||
# Draw a sample | ||
rng = torch.Generator(device="cpu") | ||
rng.manual_seed(789005) | ||
|
||
prompt = "Instruct: You are a leading role play gamer. You have seen thousands of different characters and their attributes.\nPlease return a JSON object with common attributes of an RPG character. Give me a character description\nOutput:" | ||
|
||
sequence = generator(prompt, rng=rng) | ||
print(sequence) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -11,6 +11,7 @@ | |
"clear_cache", | ||
"disable_cache", | ||
"get_cache", | ||
"Function", | ||
"prompt", | ||
"vectorize", | ||
] |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.