-
Notifications
You must be signed in to change notification settings - Fork 109
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Enables inputs to be also outputs; Fixes #936 #947
Conversation
This enables one to pass in inputs and request them as outputs independent of the graph. The use case here is that you want to join some data at the end that is extra and not in the DAG. E.g. extra pandas data. Adds test cases to check for different angles.
7af8167
to
b442d36
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this is a little too deep -- we can do this at the driver level only, no?
367bd60
to
ed0a9c6
Compare
ed0a9c6
to
12bf9ac
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's ship it. Still think it should be at the driver layer but this is OK for now.
This enables one to pass in inputs and request them as outputs independent of the graph.
The use case here is that you want to join some data at the end that is extra and not in the DAG. E.g. extra pandas data.
I skip the visualization parts because this is an odd case.
This was easier than expected because the new executor already uses inputs as outputs... So really only had to modify the old code path.
Changes
How I tested this
Notes
Checklist