-
Notifications
You must be signed in to change notification settings - Fork 399
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use PyTorch instead of NumPy #167
Conversation
687c7d4
to
6f253bc
Compare
In general there are many places where tensors are not moved to the right device like I just tested this branch vs main branch for CPU: # model_name = "gpt2"
model_name = "togethercomputer/RedPajama-INCITE-Instruct-3B-v1"
prompt = "This is a prompt"
def test_time_outlines():
import time
import outlines.models as models
from outlines.text import continuation
now = time.time()
model = models.transformers(model_name, 'cpu')
sequence = continuation(model, max_tokens=100)(prompt, samples=10)
print(f"Outlines: {time.time()-now:.2f}")
def test_time_hf():
import time
from transformers import AutoModelForCausalLM, AutoTokenizer
now = time.time()
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
inputs = tokenizer([prompt], return_tensors="pt")
out = model.generate(**inputs, do_sample=True, min_new_tokens=100, max_new_tokens=100, num_return_sequences=10, use_cache=False)
print(out.shape)
sequence = tokenizer.batch_decode(out)
print(f"HuggingFace: {time.time()-now:.2f}")
HuggingFace: 100s There is a slight advantage in only using torch Tested on |
Can we get some high-level (line)profiling measures instead? That would be a much more direct approach. |
What does line profiling mean? Do you mean to inspect inside the |
Something like the output of The standard Python profiler might also be a good next step, since we really just need to get an idea of exactly where to focus. I'm assuming that such profiler output will help tell us (implicitly) exactly where/when those missed/poorly managed transfers are happening, as well as anything else we might not have guessed (e.g. we're measuring one-time cost operations that aren't priorities). |
Also if you could seed the generation (so we can reproduce it) and normalise by the length of the longest sequence it would be great. |
import torch
import numpy
import random
SEED = 42
def set_seed(seed: int):
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
numpy.random.seed(seed)
random.seed(seed)
model_name = "gpt2"
# model_name = "togethercomputer/RedPajama-INCITE-Instruct-3B-v1"
prompt = "This is a prompt"
def test_time_outlines():
set_seed(SEED)
import time
import outlines.models as models
from outlines.text import continuation
now = time.time()
model = models.transformers(model_name, 'cpu')
sequence = continuation(model, max_tokens=100)(prompt, samples=10)
print(sequence[0])
print(f"Outlines: {time.time()-now:.2f}")
def test_time_hf():
set_seed(SEED)
import time
from transformers import AutoModelForCausalLM, AutoTokenizer
now = time.time()
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
inputs = tokenizer([prompt], return_tensors="pt")
out = model.generate(**inputs, do_sample=True, min_new_tokens=100, max_new_tokens=100, num_return_sequences=10, use_cache=False)
sequence = tokenizer.batch_decode(out)
print(sequence[0])
print(f"HuggingFace: {time.time()-now:.2f}") Seeding reproduces results for HF but not for Outlines. I suspect this is because outlines instantiates All sequences also produce exactly num_tokens because of setting |
Not fully sure how you want me to do the line_profiling, maybe you can share a MWE? Here is something: Code: from line_profiler import LineProfiler
from outlines.text.generate.sequence import Sequence
profiler = LineProfiler()
profiler.add_function(test_time_outlines)
profiler.add_function(Sequence.step)
profiler.add_function(Sequence.expand_attention_mask)
profiler.add_function(Sequence.__call__)
profiler.run('test_time_outlines()')
profiler.print_stats() Output:
|
That's perfect, thanks! It looks like 99% of the time is spent in In the meantime I'm going through the code line by line making sure that all tensors are on the same device. |
I re-ran the import outlines.models as models
import outlines.text as text
from line_profiler import LineProfiler
from outlines.text.generate.sequence import Sequence
from outlines.text.generate.continuation import Continuation
model_name = "togethercomputer/RedPajama-INCITE-Instruct-3B-v1"
prompt = "This is a prompt"
def outlines_run():
model = models.transformers(model_name, device="cuda")
a = text.generate.continuation(model, max_tokens=100)(prompt)
profiler = LineProfiler()
profiler.add_function(outlines_run)
profiler.add_function(Sequence.step.__wrapped__)
profiler.add_function(Sequence.expand_attention_mask.__wrapped__)
profiler.add_function(Sequence.update_token_ids.__wrapped__)
profiler.add_function(Sequence.__call__.__wrapped__)
profiler.add_function(Continuation.is_finished)
profiler.run('outlines_run()')
profiler.print_stats() And the results are similar to what you obtained: Full output
The interesting lines are below, where we see that
And most of the time is spent on this line:
which tells us that the boolean indexing that we currently use is very inefficient, and likely explains the difference that we see (at least on GPU) with The difference with A first approach is to simplify the code completely to remove boolean indexing and do something equivalent to |
For some reason I added the following lines in model_inputs.pop("past_key_values")
model_inputs["input_ids"] = input_ids On average over 10 runs I find the following runtime for 10 sampled sequences on my Quadro RTX5000:
So it's a wash. SImilarly for a single sampled sequence, both libraries take 27s on average over 10 runs. |
46d9c89
to
3a3eae7
Compare
This is good for review again. |
Closes #164. Closes #165.