-
Notifications
You must be signed in to change notification settings - Fork 14.1k
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Airflow log cannot be displayed on logs page #40431
Comments
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval. |
Can you upgrade to Airflow 2.9.2 and see if you have the same problem ? Look at the changelog - there were a related issue solved there. Not sure if the same but the easiest way to check it is if you actually upgrade (which you should do anyway). |
Update:
This is causing the log issue. If I remove from common.constants import part or just use this instead:
The log is good. Looks like using a constants file is causing airflow trouble when initiating a dag. |
Update: |
Do you have imports in the |
No, its just a constant file, look like this:
|
Can you look and 'modules management' (search for it in our docs) and follow the best practices there ? The - most likely - problem (and explained in the best practices) is that you are using a 'common' name as top package import. This is a Python 'property' that when importing a module, Python will import the first found module on the PYTHONPATH and since there are various places it can come from, likely reason is that you have 'common' somewhere there that is different than what you want to import. Simply when importing something don't start the import with any name that is likely to be used somewhere else. Our recommendation is to put all packages of yours in uniquely named package (say your organisation name - say 'myorg'. And always import your code as 'feom myorg.something' - this effectively namespaces your import and avoids any kinds of conflicts with similar names. |
I will definitely do that, thank you! |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Apache Airflow version
Other Airflow 2 version (please specify below)
If "Other Airflow 2 version" selected, which one?
2.9.1
What happened?
When trying to check logs in airflow page, the log keeps loading and cannot display.
Logs are configured to push to elasticsearch. Click "View Logs in Elasticsearch", logs can be seen on ES
Click "See More" button, the page will keep loading and never stop. Same log content will be append to the end of page again and again
We are running airflow on AWS EKS nodes, EKS version is 1.28
The docker image is built by ourself
airflow config map:
[logging]
logging_level = DEBUG
base_log_folder = /opt/airflow/logs
log_format = [%%(asctime)s] [%%(levelname)s] %%(filename)s:%%(lineno)d - %%(message)s
simple_log_format = [%%(asctime)s] [%%(levelname)s] - %%(message)s
colored_log_format = [%%(blue)s%%(asctime)s%%(reset)s] [%%(log_color)s%%(levelname)s%%(reset)s] %%(blue)s%%(filename)s:%%(reset)s%%(lineno)d - %%(log_color)s%%(message)s%%(reset)s
log_formatter_class = log_config.CustomSecretsMasker
file_task_handler_new_folder_permissions = 0o777
remote_logging = True
[elasticsearch]
host = https://xxx:xxx@xxx.xxx.xxx:xxxx
write_stdout = True
json_format = True
offset_field = log.offset
frontend = https://xxx.xxx.xxx/kibana/app/kibana#/discover?_a=(columns:!(message),query:(language:kuery,query:'log_id:"{log_id}"'),sort:!(offset,asc))&_g=(refreshInterval:(pause:!t,value:0),time:(from:now-1y,mode:quick,to:now))
end_of_log_mark = end_of_log
log_id_template = {dag_id}-{task_id}-{run_id}-{map_index}-{try_number}
[elasticsearch_configs]
use_ssl = True
verify_certs = True
ca_certs = /ssl/certs/generated-cacert.pem
For some dags the log does not work, but for some dags the log is good. Comparing the difference, I see the good logs all have an additional line in ES
When I check airflow 2.9.1 source code, this is the line that airflow use to tell where is the end of a task's log.
All the dags are using same airflow config running in same airflow pod. The dag can all run successfully, only have log issues.
I also try to use monkey patch to print more log. Monkey patch code is below:
Doing further code debug, I see the normal dags have this line in airflow-worker log
The dags with log issues ended here
Which means the dags with log issues somehow ended on this line
Could someone assist here? Really need to know what caused this log issue.
What you think should happen instead?
Logs for all dags should display normally
How to reproduce
Use airflow 2.9.1, try to run dags
Normal dag
Logging issue dag:
Try to run these 2 dags on airflow2.9.1
Operating System
Debian GNU/Linux 12 (bookworm)
Versions of Apache Airflow Providers
No response
Deployment
Amazon (AWS) MWAA
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: