Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallelism with graceful failure #1010

Merged
merged 4 commits into from
Jul 9, 2024
Merged

Parallelism with graceful failure #1010

merged 4 commits into from
Jul 9, 2024

Conversation

elijahbenizzy
Copy link
Collaborator

Changes

How I tested this

Notes

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future TODOs are captured in comments
  • Project documentation has been updated if adding/changing functionality.

@elijahbenizzy elijahbenizzy mentioned this pull request Jul 3, 2024
Closed
@elijahbenizzy elijahbenizzy changed the title Implementation help for 1009 Parallelism with graceful failure Jul 9, 2024
elijahbenizzy and others added 4 commits July 8, 2024 21:50
@elijahbenizzy
Sets up is_expander and is_collect
d9e8ef1 
This preps us for handling failure through our GracefulErrorAdapter
better as we can react to the node type.
@JamesArruda @elijahbenizzy
Mega-commit for James Arruda's fixes on #1017
907b690 
Initial issue description as a test.t

Modifying GracefulErrorAdpater for Parallelizable blocks.

Added docs to adapter. Updated for simpler passthrough.

Added sentinel injection feature and test. Updated gitignore for vscode settings folder.

Fix for tag error

Updating adapter test for proper tags

Fixing types for 3.8

Type fixing for 3.8

Fixing sentinel equality testing to avoid incomparible types.

Adding example of paralellism and GracefulErrorAdapter

Added decorator for sentinel acceptance as input. Updated tests and example. Added DAG image to example. Typo.

Simplifying node callable retrieval.

Adding docstring clarification on try_all_parallel.

Parametrizing tests. Moving test module to resources.

Added docs to decorator. Changed node tag key.
@elijahbenizzy
Does inline import to avoid circular references
fd55eb2 
There's an issue that we have not solved yet -- we'll need to audit
circular references. For now an inline import is OK, but see
PR + comment here for more details
#1017. We'll need to fix
this later.
@elijahbenizzy
Adds accept_error_sentinel decorator to docs
ac98120
@elijahbenizzy elijahbenizzy marked this pull request as ready for review July 9, 2024 04:57
@elijahbenizzy elijahbenizzy merged commit 5bf8bac into main Jul 9, 2024
22 of 24 checks passed
@elijahbenizzy elijahbenizzy deleted the graceful-failure-parallelism branch July 9, 2024 04:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@elijahbenizzy @JamesArruda