Number of queries to Airflow database in "DAG File Processing Stats"

[MaksYermak]

Description

This new feature will add a new column to the "DAG File Processing Stats" of DAG processor logs. This column will store information about the number of queries to the Airflow database per DAG.

Use case/motivation

This new column may be convenient to have it in case of debugging issues related to high load on Airflow database, e.g. typical scenario is when DAG file(s) have a lot of queries to database done on the top level of code and those are executed each time during parsing of these DAG files. One common example is excessive usage of "Variables.get" as top-level statements in DAG files.

Having information about "number of queries to Airflow database" per DAG file may help a lot during debugging issues related to high load on database or issues related to long parsing of the DAG files.

Related issues

Thread with discussion in the Airflow community: https://lists.apache.org/thread/9j6q2lq521rt5zx46l2dvow2c85sgqwb

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[raphaelauv]

Good idea !

I would also love a global airflow option DAG_PARSING_METADATABASE_CALL_FORBIDEN

that could be set to True , so parsing of dags with variable / connection ... at top level fail. And promote usage of Dynamic Task mapping in the error log

[troxil]

@raphaelauv Wouldn't such a feature conflict with the theory of the Secrets Cache that was recently released too?

[raphaelauv]

AIRFLOW__SECRETS__USE_CACHE is by default at false

and yes this feature will no more be usefull when DAG_PARSING_METADATABASE_CALL_FORBIDEN is set to True

[VladaZakharova]

@eladkal @potiuk @kaxil
Let's continue here our discussion about adding more logs in our output
@raphaelauv Thank you for your ideas, but I think this can be not directly related to this exact issue with adding exact logs to output. Maybe it will be more efficient to create a new issue for this? WDYT?

[kaxil]

@raphaelauv Thank you for your ideas, but I think this can be not directly related to this exact issue with adding exact logs to output. Maybe it will be more efficient to create a new issue for this? WDYT?

Yeah, let's keep it separate

[MaksYermak]

In my PR for this feature I have added a new column to the table with processing results in the log file. I have seen in the discussion thread some ideas about adding this information to the UI or DB, but I didn't notice a strong consensus about it and I decided not to make any UI or DB changes in this PR. I will be ready to create a new PR in future when we have a clear vision where on UI and DB are better to have this data.