-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add README for installation and usage
- Loading branch information
1 parent
c1ecfbb
commit 311d71e
Showing
1 changed file
with
60 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1 +1,60 @@ | ||
# Airflow Wherobots Provider | ||
# Airflow Wherobots Provider | ||
|
||
Airflow Provider that integrates [wherobots cloud](https://wherobots.com/)'s computation features into your workflows. | ||
|
||
## Installation | ||
|
||
If you use [Poetry](https://python-poetry.org) in your project, add the | ||
dependency with `poetry add`: | ||
|
||
``` | ||
$ poetry add git+https://github.com/wherobots/airflow-providers-wherobots | ||
``` | ||
|
||
Otherwise, just `pip install` it: | ||
|
||
``` | ||
$ pip install git+https://github.com/wherobots/airflow-providers-wherobots | ||
``` | ||
|
||
## Usage | ||
|
||
### Create Connection | ||
Create a connection first, the default wherobots connection name is `wherobots_default`. | ||
For any other name, please pass in the `wherobots_conn_id` parameter when initialize Wherobots Operators. | ||
Fill in the following fields in connection detail page: | ||
1. Fill the wherobots API Service endpoint into the `Host` field | ||
2. Fill the personal API Token into the `Password` field | ||
|
||
### Execute SQL query | ||
You can use the `WherobotsSqlOperator` to run sql queries on the Wherobots cloud. | ||
You can build your ETL jobs on Wherobots Catalogs. | ||
Check this [guidance](https://docs.wherobots.services/1.2.2/tutorials/sedonadb/vector-data/vector-load/) to learn how to **read data, transformation and write results by pure SQL in Wherobots Cloud**. | ||
Below is an example dag that executes SQL query on Wherobots Cloud | ||
```python | ||
import datetime | ||
|
||
from airflow import DAG | ||
from airflow_providers_wherobots.operators.sql import WherobotsSqlOperator | ||
|
||
|
||
with DAG( | ||
dag_id="example_wherobots_sql_dag", | ||
start_date=datetime.datetime(2024, 1, 1), | ||
schedule="@hourly", | ||
catchup=False | ||
): | ||
# operator = PythonOperator(task_id="print_the_context", python_callable=print_context) | ||
operator = WherobotsSqlOperator( | ||
task_id="execute_query", | ||
sql=f""" | ||
INSERT INTO | ||
wherobots.test_db.example_geom | ||
SELECT | ||
id, geometry, confidence, geohash | ||
FROM | ||
wherobots_open_data.overture.places_place | ||
""", | ||
return_last=False, | ||
) | ||
``` |