-
-
Notifications
You must be signed in to change notification settings - Fork 35
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update supported versions matrix #137
Merged
tomasfarias
merged 13 commits into
tomasfarias:master
from
millin:update_suppoted_versions
Jul 5, 2024
Merged
Changes from all commits
Commits
Show all changes
13 commits
Select commit
Hold shift + click to select a range
a777d8e
Update ci
millin bb18e72
jaffle_shop renamed to jaffle-shop-classic
millin 6ef3c97
Update packages
millin b2e46d3
dbt >=1.7 version compatibility
millin cce504a
Lint using Black
millin 010acb7
Fix deprecation warning
millin c419149
Update code compatibility: dbt>=1.7.5,<1.9.0
millin bd1985b
Delete connections
millin a11769d
Update pyproject
millin 0cab907
Update readme
millin f569c5b
Lint
millin 7749bd0
Fix tests
millin 43c1f47
Fix coverage
millin File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -22,32 +22,30 @@ jobs: | |
fail-fast: false | ||
matrix: | ||
python-version: | ||
- '3.12' | ||
- '3.11' | ||
- '3.10' | ||
- '3.9' | ||
- '3.8' | ||
airflow-version: | ||
- '2.7.2' | ||
- '2.6.3' | ||
- '2.5.3' | ||
- '2.4.3' | ||
- '2.9.2' | ||
- '2.8.4' | ||
- '2.7.3' | ||
dbt-version: | ||
- 1.8 | ||
- 1.7 | ||
Comment on lines
34
to
36
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 2 latest dbt versions is a fair policy 👍 |
||
- 1.6 | ||
- 1.5 | ||
- 1.4 | ||
exclude: | ||
# Incompatible combinations | ||
- python-version: 3.11 | ||
airflow-version: '2.4.3' | ||
- python-version: 3.12 | ||
airflow-version: '2.8.4' | ||
|
||
- python-version: 3.11 | ||
airflow-version: '2.5.3' | ||
- python-version: 3.12 | ||
airflow-version: '2.7.3' | ||
|
||
runs-on: ubuntu-latest | ||
steps: | ||
- name: Harden Runner | ||
uses: step-security/harden-runner@v2.6.1 | ||
uses: step-security/harden-runner@v2.8.1 | ||
with: | ||
egress-policy: block | ||
allowed-endpoints: > | ||
|
@@ -73,34 +71,35 @@ jobs: | |
sudo apt-get update | ||
sudo apt-get install --yes --no-install-recommends postgresql | ||
|
||
- uses: actions/checkout@v4.1.1 | ||
- uses: actions/checkout@v4.1.7 | ||
- name: Set up Python ${{ matrix.python-version }} | ||
uses: actions/setup-python@v4.7.1 | ||
uses: actions/setup-python@v5.1.0 | ||
with: | ||
python-version: ${{ matrix.python-version }} | ||
|
||
- name: Install Poetry | ||
uses: abatilo/actions-poetry@v2.3.0 | ||
with: | ||
poetry-version: 1.7.0 | ||
poetry-version: 1.8.3 | ||
|
||
- name: Install airflow-dbt-python with Poetry | ||
run: poetry install -E postgres --with dev | ||
|
||
- name: Install Airflow with constraints | ||
- name: Install Airflow & dbt | ||
run: | | ||
wget https://github.com/raw/apache/airflow/constraints-${{ matrix.airflow-version }}/constraints-${{ matrix.python-version }}.txt -O constraints.txt | ||
poetry run pip install apache-airflow==${{ matrix.airflow-version }} apache-airflow-providers-amazon apache-airflow-providers-ssh -c constraints.txt | ||
poetry run pip install "dbt-core~=${{ matrix.dbt-version }}.0" "dbt-postgres~=${{ matrix.dbt-version }}.0" | ||
poetry run airflow db init | ||
poetry env use ${{ matrix.python-version }} | ||
poetry add "apache-airflow==${{ matrix.airflow-version }}" \ | ||
"dbt-core~=${{ matrix.dbt-version }}.0" \ | ||
"dbt-postgres~=${{ matrix.dbt-version }}.0" \ | ||
--python ${{ matrix.python-version }} | ||
poetry install -E postgres --with dev | ||
poetry run airflow db migrate | ||
poetry run airflow connections create-default-connections | ||
|
||
- name: Linting with ruff | ||
run: poetry run ruff . | ||
run: poetry run ruff check . | ||
|
||
- name: Static type checking with mypy | ||
# We only run mypy on the latest supported versions of Airflow & dbt, | ||
# so it is currently impossible to write conditions for that depend on package versions. | ||
if: matrix.airflow-version == '2.7.2' && matrix.dbt-version == '1.7' | ||
if: matrix.python-version == '3.12' && matrix.airflow-version == '2.9.2' && matrix.dbt-version == '1.8' | ||
run: poetry run mypy . | ||
|
||
- name: Code formatting with black | ||
|
@@ -131,7 +130,7 @@ jobs: | |
|
||
steps: | ||
- name: Harden Runner | ||
uses: step-security/harden-runner@v2.6.0 | ||
uses: step-security/harden-runner@v2.8.1 | ||
with: | ||
egress-policy: block | ||
allowed-endpoints: > | ||
|
@@ -140,15 +139,15 @@ jobs: | |
api.github.com:443 | ||
pypi.org:443 | ||
|
||
- uses: actions/checkout@v4.1.1 | ||
- uses: actions/setup-python@v4.7.1 | ||
- uses: actions/checkout@v4.1.7 | ||
- uses: actions/setup-python@v5.1.0 | ||
with: | ||
python-version: '3.11' | ||
python-version: '3.12' | ||
|
||
- name: Install Poetry | ||
uses: abatilo/actions-poetry@v2.3.0 | ||
with: | ||
poetry-version: 1.7.0 | ||
poetry-version: 1.8.3 | ||
|
||
- name: Install airflow-dbt-python with Poetry | ||
run: poetry install --with dev -E airflow-providers | ||
|
@@ -179,7 +178,7 @@ jobs: | |
path: htmlcov | ||
|
||
- name: "Make coverage badge" | ||
uses: schneegans/dynamic-badges-action@v1.6.0 | ||
uses: schneegans/dynamic-badges-action@v1.7.0 | ||
if: github.event_name != 'pull_request' | ||
with: | ||
auth: ${{ secrets.GIST_TOKEN }} | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -15,10 +15,10 @@ Read the [documentation](https://airflow-dbt-python.readthedocs.io) for examples | |
## Requirements | ||
|
||
Before using *airflow-dbt-python*, ensure you meet the following requirements: | ||
* A *dbt* project using [dbt-core](https://pypi.org/project/dbt-core/) version 1.4.0 or later. | ||
* An Airflow environment using version 2.2 or later. | ||
* A *dbt* project using [dbt-core](https://pypi.org/project/dbt-core/) version 1.7.5 or later. | ||
* An Airflow environment using version 2.7 or later. | ||
|
||
* If using any managed service, like AWS MWAA, ensure your environment is created with a supported version of Airflow. | ||
* If using any managed service, like AWS MWAA or GCP Cloud Composer 2/3, ensure your environment is created with a supported version of Airflow. | ||
* If self-hosting, Airflow installation instructions can be found in their [official documentation](https://airflow.apache.org/docs/apache-airflow/stable/installation/index.html). | ||
|
||
* Running Python 3.8 or later in your Airflow environment. | ||
|
@@ -29,7 +29,7 @@ Before using *airflow-dbt-python*, ensure you meet the following requirements: | |
|
||
> **Note** | ||
> | ||
> Older versions of Airflow and *dbt* may work with *airflow-dbt-python*, although we cannot guarantee this. Our testing pipeline runs the latest *dbt-core* with the latest Airflow release, and the latest version supported by [AWS MWAA](https://aws.amazon.com/managed-workflows-for-apache-airflow/). | ||
> Older versions of Airflow and *dbt* may work with *airflow-dbt-python*, although we cannot guarantee this. Our testing pipeline runs the latest *dbt-core* with the latest Airflow release, and the latest version supported by [AWS MWAA](https://aws.amazon.com/managed-workflows-for-apache-airflow/) and [GCP Cloud Composer 2/3](https://aws.amazon.com/managed-workflows-for-apache-airflow/). | ||
|
||
## From PyPI | ||
|
||
|
@@ -66,6 +66,12 @@ Add *airflow-dbt-python* to your `requirements.txt` file and edit your Airflow e | |
|
||
Read the [documentation](https://airflow-dbt-python.readthedocs.io/en/latest/getting_started.html#installing-in-mwaa) for more a more detailed AWS MWAA installation breakdown. | ||
|
||
## In GCP Cloud Composer | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Thanks! Great updates! |
||
|
||
Add *airflow-dbt-python* to your PyPI packages list. | ||
|
||
Read the [documentation](https://cloud.google.com/composer/docs/composer-2/install-python-dependencies#install-pypi) for more a more detailed GCP Cloud Composer 2 installation breakdown. | ||
|
||
## In other managed services | ||
|
||
*airflow-dbt-python* should be compatible with most or all Airflow managed services. Consult the documentation specific to your provider. | ||
|
@@ -119,7 +125,7 @@ See an example DAG [here](examples/airflow_connection_target_dag.py). | |
|
||
Although [`dbt`](https://docs.getdbt.com/) is meant to be installed and used as a CLI, we may not have control of the environment where Airflow is running, disallowing us the option of using *dbt* as a CLI. | ||
|
||
This is exactly what happens when using [Amazon's Managed Workflows for Apache Airflow](https://aws.amazon.com/managed-workflows-for-apache-airflow/) or MWAA: although a list of Python requirements can be passed, the CLI cannot be found in the worker's PATH. | ||
This is exactly what happens when using [Amazon's Managed Workflows for Apache Airflow](https://aws.amazon.com/managed-workflows-for-apache-airflow/) (aka MWAA): although a list of Python requirements can be passed, the CLI cannot be found in the worker's PATH. | ||
|
||
There is a workaround which involves using Airflow's `BashOperator` and running Python from the command line: | ||
|
||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,2 +1,3 @@ | ||
"""Provides an Airflow operator and hooks to run all or most dbtcommands.""" | ||
|
||
from .__version__ import __author__, __copyright__, __title__, __version__ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,5 +1,6 @@ | ||
"""The module's version information.""" | ||
|
||
__author__ = "Tomás Farías Santana" | ||
__copyright__ = "Copyright 2021 Tomás Farías Santana" | ||
__title__ = "airflow-dbt-python" | ||
__version__ = "2.0.1" | ||
__version__ = "2.1.0" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'll probably want a policy for Airflow version support.
For Python versions it's easy: We aim to support whatever dbt supports, which currently has only a >=3.8. restriction.
For Airflow, I'd say we definitely want to test against latest official minor version (2.9.x), and ideally also latest MWAA-supported version (2.8.1). Astronomer supports new Airflow versions pretty much as soon as they come out, so we are covered there already.
Any other hosted service we should consider prioritizing? If not, I'd consider just supporting 2.9 and 2.8.1 here. There is a tradeoff between as much support as possible and CI times and dependency resolution. So, I'd try to keep the number of "tested" supported versions as small as possible, to avoid the test matrix exploding too much.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would like to support Airflow 2.7.x as it is still in use by Cloud Composer
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you mind mentioning in the docs?
We mention testing against latest MWAA version in https://github.com/tomasfarias/airflow-dbt-python/blob/master/docs/development.rst?plain=1#L40, so we could extend that with a few extra words: "... latest version available in AWS MWAA and in Cloud Composer" + a link to the Cloud composer docs.