Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable outputs to access all inputs #936

Closed
skrawcz opened this issue Jun 4, 2024 · 2 comments · Fixed by #947
Closed

Enable outputs to access all inputs #936

skrawcz opened this issue Jun 4, 2024 · 2 comments · Fixed by #947
Labels
triage label for issues that need to be triaged.

Comments

@skrawcz
Copy link
Collaborator

skrawcz commented Jun 4, 2024

Is your feature request related to a problem? Please describe.
You cannot currently request inputs that are not referenced in the DAG.

Describe the solution you'd like
Allow one to pass in arbitrary inputs not used in the DAG and then be able to request them as outputs.

To recreate

# functions.py - declare and link your transformations as functions....
import pandas as pd

def a(input: pd.Series) -> pd.Series:
    return input % 7

def b(a: pd.Series) -> pd.Series:
    return a * 2

def c(a: pd.Series, b: pd.Series) -> pd.Series:
    return a * 3 + b * 2

def d(c: pd.Series) -> pd.Series:
    return c ** 3
# And run them!
import functions
from hamilton import driver
dr = driver.Driver({}, functions)
result = dr.execute(
   ['c','d', 'foo'],  # want to request foo -- but it's not referenced anywhere
   inputs={'input': pd.Series([1, 2, 3, 4, 5]), 'foo': 1}
)
print(result)
dr.display_all_functions(
   "graph.dot", orient="TB", show_legend=False)
Traceback (most recent call last):
ValueError: Unknown nodes [foo] requested. Check for typos?

Describe alternatives you've considered
You write a dummy function to contain it as input.

@skrawcz skrawcz added the triage label for issues that need to be triaged. label Jun 4, 2024
@skrawcz skrawcz changed the title Enable outputs to access inputs Enable outputs to access all inputs Jun 4, 2024
@zilto
Copy link
Collaborator

zilto commented Jun 4, 2024

Should the values appear in the viz? Or that isn't part of the requirements?

@elijahbenizzy
Copy link
Collaborator

If we want to support this we can just add this to the final result computation (E.G. if it's included in the output we just wire it from the input. Seems decidedly reasonable.

That said, what's the use-case? This should work if these are inputs are referenced, but why not just join later if they're not? Or is it to leverage the result builder?

skrawcz added a commit that referenced this issue Jun 12, 2024
This enables one to pass in inputs and request them as outputs
independent of the graph.

The use case here is that you want to join some data at the end
that is extra and not in the DAG. E.g. extra pandas data.
skrawcz added a commit that referenced this issue Jun 12, 2024
This enables one to pass in inputs and request them as outputs
independent of the graph.

The use case here is that you want to join some data at the end
that is extra and not in the DAG. E.g. extra pandas data.

Adds test cases to check for different angles.
@skrawcz skrawcz linked a pull request Jun 12, 2024 that will close this issue
7 tasks
skrawcz added a commit that referenced this issue Jun 12, 2024
* Enables inputs to be also outputs; Fixes #936

This enables one to pass in inputs and request them as outputs
independent of the graph.

The use case here is that you want to join some data at the end
that is extra and not in the DAG. E.g. extra pandas data.

Adds test cases to check for different angles.


* Adds two extra checks

* PR review feedback
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
triage label for issues that need to be triaged.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants