You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I used the GitHub search to find a similar issue and didn't find it.
I searched the Prefect documentation for this issue.
I checked that this issue is related to Prefect and not one of its dependencies.
Bug summary
In the following flow, I'd expect the tasks to fail once, but then the caller top_task should retry, and everything should be successful on the second try. However, the overall flow state is marked as Failed despite the tasks eventually succeeding.
Reproduction
fromprefectimportflow, taskfailed=False@taskdefnested_flaky_task():
# This task will fail the first time it is run, but will succeed if called a second timeglobalfailedifnotfailed:
failed=TrueraiseValueError("Forced task failure")
@task(retries=1,)deftop_task():
nested_flaky_task()
@flowdefnested_task_flow():
top_task()
if__name__=="__main__":
nested_task_flow()
Error
15:47:30.527|INFO|Taskrun'top_task-0'-Receivednon-finalstate'AwaitingRetry'whenproposingfinalstate'Failed'andwillattempttorunagain...
15:47:30.572|INFO|Taskrun'top_task-0'-Createdtaskrun'nested_flaky_task-1'fortask'nested_flaky_task'15:47:30.573|INFO|Taskrun'top_task-0'-Executing'nested_flaky_task-1'immediately...
15:47:30.613|INFO|Taskrun'nested_flaky_task-1'-FinishedinstateCompleted()
15:47:30.627|INFO|Taskrun'top_task-0'-FinishedinstateCompleted()
15:47:30.645|ERROR|Flowrun'berserk-loon'-FinishedinstateFailed('1/3 states failed.')
Versions (prefect version output)
Version: 2.19.6
API version: 0.8.4
Python version: 3.11.6
Git commit: 9d938fe7
Built: Mon, Jun 24, 2024 10:23 AM
OS/Arch: darwin/arm64
Profile: default
Server type: ephemeral
Server:
Database: sqlite
SQLite version: 3.43.2
Additional context
Workaround here is pretty straightforward - either add retries to the flaky task, or remove the task decorator. So it isn't a huge blocker to being able to build effective flows. But, it's confusing and hard to retroactively figure out what happened when this happens in the context of a big flow - behavior more in line with expectations could prevent some debugging headaches.
The text was updated successfully, but these errors were encountered:
First check
Bug summary
In the following flow, I'd expect the tasks to fail once, but then the caller
top_task
should retry, and everything should be successful on the second try. However, the overall flow state is marked as Failed despite the tasks eventually succeeding.Reproduction
Error
Versions (
prefect version
output)Additional context
Workaround here is pretty straightforward - either add retries to the flaky task, or remove the task decorator. So it isn't a huge blocker to being able to build effective flows. But, it's confusing and hard to retroactively figure out what happened when this happens in the context of a big flow - behavior more in line with expectations could prevent some debugging headaches.
The text was updated successfully, but these errors were encountered: