Executing R scripts from Dagster #18972
-
I'm currently exploring Dagster for data orchestration and find myself in need of some community wisdom. My primary goal is to execute R scripts within the Dagster environment, a task I previously accomplished using Airflow's BashOperator and DockerOperator. In Airflow, these operators provided a straightforward way to run R scripts either directly through shell commands or within Docker containers. I'm looking for Dagster's equivalents to these operators. Specifically, I'm interested in:
from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from datetime import datetime
default_args = {
'owner': 'airflow',
'start_date': datetime(2023, 1, 1),
# other args...
}
dag = DAG('r_script_bash_operator', default_args=default_args, schedule_interval='@daily')
run_r_script = BashOperator(
task_id='run_r_script',
bash_command='Rscript /path/to/your_script.R ',
dag=dag,
)
run_r_script
from airflow import DAG
from airflow.operators.docker_operator import DockerOperator
from datetime import datetime
default_args = {
'owner': 'airflow',
'start_date': datetime(2023, 1, 1),
# other args...
}
dag = DAG('r_script_docker_operator', default_args=default_args, schedule_interval='@daily')
run_r_script_in_docker = DockerOperator(
task_id='run_r_script_in_docker',
image='r-base', # Make sure this image has R installed
command='Rscript /path/to/your_script.R',
docker_url='unix://var/run/docker.sock', # Update as needed
network_mode='bridge',
dag=dag,
)
run_r_script_in_docker I would greatly appreciate any examples, tips, or pointers to relevant documentation. Insights on capturing logging, metadata, and results while executing these scripts would also be incredibly helpful. Thank you in advance for your guidance and support. This is related to #2624. |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 13 replies
-
Hi @philiporlando and fellow R-user. You have a few options, but I think the simplest would be to use the dagster-docker op. From the docs: from dagster_docker import docker_container_op
from dagster import job
first_op = docker_container_op.configured(
{
"image": "busybox",
"command": ["echo HELLO"],
},
name="first_op",
)
second_op = docker_container_op.configured(
{
"image": "busybox",
"command": ["echo GOODBYE"],
},
name="second_op",
)
@job
def full_job():
second_op(first_op()) For more complex integrations, you might find Dagster Pipes interesting as well: https://docs.dagster.io/guides/dagster-pipes Let me know if this is helpful or if you have other questions. |
Beta Was this translation helpful? Give feedback.
Hi @philiporlando and fellow R-user.
You have a few options, but I think the simplest would be to use the dagster-docker op.
From the docs:
For more complex integrations, you might find Dagster Pipes interesting as well: https://docs.dagster.io/guides/dagster-pipes
Let me know if this is…