Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

R as an available Execution Environment #2624

Open
JohnMav opened this issue Jun 19, 2020 · 1 comment
Open

R as an available Execution Environment #2624

JohnMav opened this issue Jun 19, 2020 · 1 comment
Labels
area: integrations Related to general integrations, including requests for a new integration type: feature-request type: project Big Projects

Comments

@JohnMav
Copy link

JohnMav commented Jun 19, 2020

Since 0.8.0 the separation of Dagster Host and User process has allowed creating workspace environments with different versions of Python. For teams that have users that work primarily in R driven environments it would be amazing to be able to designate a workspace environment as a specific R version and be able to run dagster pipelines within that environment.

issue #1585 raises the notion of supporting polyglot notebooks but would love for this to go a step further and support tools like RMarkdown/Shiny apps in a similar fashion.

@natekupp natekupp added this to the 0.10.0 (planned) milestone Aug 4, 2020
@mgasner mgasner added the type: project Big Projects label Aug 6, 2020
@catherinewu catherinewu added the area: integrations Related to general integrations, including requests for a new integration label Mar 3, 2021
@sryza sryza removed this from the 0.10.0 milestone Apr 1, 2021
@nlarusstone
Copy link

nlarusstone commented Apr 13, 2022

I want to add a big +1 here and add a bit more context about why this is something that would be extremely useful. Many organizations (mine included) have a combination of R and Python scripts used in their pipelines. In particular, R is really powerful for statistical modeling and has a huge number of packages that don't exist in Python. It's not a great language for writing production code, but it's hard to move entirely off of it into Python.

I love a lot of the motivations behind Dagster (typing, testing, etc.) and would like to bring that to as much of our pipelines as possible. I understand it's probably hard to do that, so even just having a simple R operator that passes dataframes between R and Python would be incredibly powerful. It seems like the current workaround involves running a shell script or Docker container to pass data via files -- which works, but reduces a lot of the power of Dagster.

I've only just started using Dagster, so I'm not familiar enough to know how easy/hard this is or where I would even begin to contribute a feature like this, but wanted to make sure this issue doesn't die!

I've collected a few mentions from Slack where other people have mentioned this as something they would like to do:
(Original post that spawned this issue): https://dagster.slack.com/archives/CCCR6P2UR/p1592431695124700
https://dagster.slack.com/archives/CCCR6P2UR/p1614100503079800
https://dagster.slack.com/archives/C01U954MEER/p1638575143351700?thread_ts=1638550558.333200&cid=C01U954MEER
https://dagster.slack.com/archives/C014N0PK37E/p1641375596083800

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: integrations Related to general integrations, including requests for a new integration type: feature-request type: project Big Projects
Projects
None yet
Development

No branches or pull requests

8 participants