Skip to content

Commit

Permalink
Update generator.py
Browse files Browse the repository at this point in the history
Addressed the issue reported in #656 with the solution proposed by @rlouf . 

I am really not sure whether this is the best solution.
  • Loading branch information
raphaelchinchilla authored and rlouf committed Feb 14, 2024
1 parent b89e8df commit c548a8a
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion outlines/generate/generator.py
Original file line number Diff line number Diff line change
Expand Up @@ -265,7 +265,7 @@ def reorder_kv_cache(
for cache_item in kv_cache:
new_cache_item: Tuple = tuple()
for layer in cache_item:
layer = torch.index_select(layer, 0, ancestors)
layer = torch.index_select(layer, 0, ancestors.to(layer.device))
new_cache_item += (layer,)
new_kv_cache += (new_cache_item,)

Expand Down

0 comments on commit c548a8a

Please sign in to comment.