Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TextGeneration] Fix initialization; don't try v1 init for text gen #1571

Merged
merged 3 commits into from
Jan 31, 2024
Merged

[TextGeneration] Fix initialization; don't try v1 init for text gen #1571

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8fad569
only check capacity condition durin prefill; already have check in ge…
dsikka Jan 31, 2024
23ab068
Merge branch 'main' into fix_initi
dsikka Jan 31, 2024
5579316
dont try v1 if running text gen; just raise error
dsikka Jan 31, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion src/deepsparse/pipeline.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
SchedulerGroup,
)
from deepsparse.subgraph_execute import SubGraphExecutor
from deepsparse.tasks import SupportedTasks
from deepsparse.utils import InferenceState, PipelineState
from deepsparse.utils.subgraph import SubGraph
from deepsparse.utils.time import TIMER_KEY, InferenceStages, TimerManager
Expand Down Expand Up @@ -139,7 +140,10 @@ def create(cls, task: str, **kwargs) -> "Pipeline":
"Pipeline was not created for the given task. The "
"provided task should be registered using the OperatorRegistry"
)
except Exception:
except Exception as e:
if SupportedTasks.is_text_generation(task):
raise e

_LOGGER.warning(f"Could not create v2 '{task}' pipeline, trying legacy")
from deepsparse.legacy import Pipeline

Expand Down
5 changes: 4 additions & 1 deletion src/deepsparse/transformers/pipelines/text_generation/autoregressive_preprocess_operator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,10 @@ def can_operate(self, inp: Any) -> bool:
if inp.get("in_generation"):
return True

if kv_cache.total_num_processed_tokens >= kv_cache.capacity:
if (
kv_cache.total_num_processed_tokens >= kv_cache.capacity
and inp.get("in_generation") is None
):
raise RuntimeError(
"Not enough kv_cache capacity to run generation. Please use a larger "
"sequence_length or a shorter prompt"
Expand Down
Loading