Skip to content

πŸ¦€ Rust implementation of Celery for producing and consuming background tasks

License

Notifications You must be signed in to change notification settings

gistia/rusty-celery

Β 
Β 

Repository files navigation




A Rust implementation of Celery for producing and consuming asynchronous tasks with a distributed message queue.


Build License Crates Docs Help wanted


We welcome contributions from everyone regardless of your experience level with Rust. For complete beginners, see HACKING_QUICKSTART.md.

If you already know the basics of Rust but are new to Celery, check out the Rusty Celery Book or the original Python Celery Project.

Quick start

Define tasks by decorating functions with the task attribute.

use celery::prelude::*;

#[celery::task]
fn add(x: i32, y: i32) -> TaskResult<i32> {
    Ok(x + y)
}

Create an app with the app macro and register your tasks with it:

let my_app = celery::app!(
    broker = AMQPBroker { std::env::var("AMQP_ADDR").unwrap() },
    tasks = [add],
    task_routes = [
        "*" => "celery",
    ],
).await?;

Then send tasks to a queue with

my_app.send_task(add::new(1, 2)).await?;

And consume tasks as a worker from a queue with

my_app.consume().await?;

Examples

The examples/ directory contains:

Prerequisites

If you already have an AMQP broker running you can set the environment variable AMQP_ADDR to your broker's URL (e.g., amqp://localhost:5672//, where the second slash at the end is the name of the default vhost). Otherwise simply run the helper script:

./scripts/brokers/amqp.sh

This will download and run the official RabbitMQ image (RabbitMQ is a popular AMQP broker).

Run the examples

Run Rust Celery app

You can consume tasks with:

cargo run --example celery_app consume

And you can produce tasks with:

cargo run --example celery_app produce [task_name]

Current supported tasks for this example are: add, buggy_task, long_running_task and bound_task

Run Python Celery app

Similarly, you can consume or produce tasks from Python by running

python examples/celery_app.py consume [task_name]

or

python examples/celery_app.py produce

You'll need to have Python 3 installed, along with the requirements listed in the requirements.txt file. You'll also have to provide a task name. This example implements 4 tasks: add, buggy_task, long_running_task and bound_task

Run Rust Beat app

You can start the Rust beat with:

cargo run --example beat_app

And then you can consume tasks from Rust or Python as explained above.

Road map and current state

βœ… = Supported and mostly stable, although there may be a few incomplete features.
⚠️ = Partially implemented and under active development.
πŸ”΄ = Not supported yet but on-deck to be implemented soon.

Core

Status Tracking
Protocol ⚠️
Producers βœ…
Consumers βœ…
Brokers βœ…
Beat βœ…
Backends πŸ”΄
Baskets πŸ”΄

Brokers

Status Tracking
AMQP βœ…
Redis βœ…

Backends

Status Tracking
RPC πŸ”΄
Redis πŸ”΄

About

πŸ¦€ Rust implementation of Celery for producing and consuming background tasks

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Rust 99.0%
  • Other 1.0%