Update supported versions matrix

[millin]

fixes #134
fixes #135

[tomasfarias]

We'll probably want a policy for Airflow version support.

For Python versions it's easy: We aim to support whatever dbt supports, which currently has only a >=3.8. restriction.

For Airflow, I'd say we definitely want to test against latest official minor version (2.9.x), and ideally also latest MWAA-supported version (2.8.1). Astronomer supports new Airflow versions pretty much as soon as they come out, so we are covered there already.

Any other hosted service we should consider prioritizing? If not, I'd consider just supporting 2.9 and 2.8.1 here. There is a tradeoff between as much support as possible and CI times and dependency resolution. So, I'd try to keep the number of "tested" supported versions as small as possible, to avoid the test matrix exploding too much.

[millin]

I would like to support Airflow 2.7.x as it is still in use by Cloud Composer

[tomasfarias]

Do you mind mentioning in the docs?

We mention testing against latest MWAA version in https://github.com/tomasfarias/airflow-dbt-python/blob/master/docs/development.rst?plain=1#L40, so we could extend that with a few extra words: "... latest version available in AWS MWAA and in Cloud Composer" + a link to the Cloud composer docs.

[tomasfarias]

2 latest dbt versions is a fair policy 👍

[tomasfarias]

Tried testing locally, but it seems like some new flags need to be supported starting with dbt 1.7, and user_config parameter has been removed from Profile. I managed to get some tests to pass by adding the following flags in utils/config.py:

        flags_dict["PARTIAL_PARSE_FILE_DIFF"] = self.partial_parse_file_diff
        flags_dict["USE_EXPERIMENTAL_PARSER"] = self.use_experimental_parser
        flags_dict["CACHE_SELECTED_ONLY"] = self.cache_selected_only
        flags_dict["PARTIAL_PARSE_FILE_PATH"] = self.partial_parse_file_path

(And also the corresponding class attributes)

As well as dropping user_config=user_config when calling from_raw_profile_info.

Dag tests (tests/dags) are passing, but other tests seem to fail to select any models to run. Something seems to have changed with latter versions of dbt but I haven't figured out what 🤔

[millin]

Sorry, I created this pr prematurely.
I've already fixed most of the bugs locally, but haven't pushed yet. I'll try to finish on Monday.

[tomasfarias]

Sorry, I created this pr prematurely.
I've already fixed most of the bugs locally, but haven't pushed yet. I'll try to finish on Monday.

No problem! Happy to review next week and get this released.

[tomasfarias]

Thanks! Great updates!

[tomasfarias]

Will release shortly. Thanks for the great work @millin