Adds support for running integration tests against Airflow #1204
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
saurav-c
added
the
skip_integration_test
kenxu95
reviewed
Apr 13, 2023
saurav-c
force-pushed
the
eng-2116-get-sdk-integration-tests-working-with-airflow
branch
from
506e4bc
to
7ecd07a
Compare
kenxu95
approved these changes
Apr 28, 2023
| @@ -211,6 +234,12 @@ def delete_all_flows(client: aqueduct.Client) -> None: | |||
| else: | |||
| print("Successfully deleted workflow %s" % flow_name) | |||
|
|
|||
| # Try deleting Airflow DAG file if it exists | |||
| # TODO - Only do this for Airflow workflows | |||
There was a problem hiding this comment.
Should we make a task for this TODO?
saurav-c
deleted the
eng-2116-get-sdk-integration-tests-working-with-airflow
branch
agiron123
pushed a commit
that referenced
this pull request
May 9, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Describe your changes and why you are making these changes
This PR adds initial support for running integration tests against Airflow. It has a setup script for starting a local Airflow instance. It also adds logic for manually copying the Airflow DAG file to the appropriate folder.
There is still follow on work that needs to be done before these tests can be run on Github actions:
Related issue number (if any)
ENG 2116
Tests
Ran the basic flow_test locally and uncovered the syncing bug.
Loom demo (if any)
Checklist before requesting a review
python3 scripts/run_linters.py -hfor usage).run_integration_test: Runs integration testsskip_integration_test: Skips integration tests (Should be used when changes are ONLY documentation/UI)