Skip to content

Do not flood worker logs with secrets backend loading logs #50581

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
May 14, 2025
Merged

Do not flood worker logs with secrets backend loading logs #50581

merged 1 commit into from
May 14, 2025

Conversation

amoghrajesh
Copy link
Contributor

closes: #50517

The info logs for loading of secrets backend was present in ensure_secrets_backend_loaded which is called every time Variable.get or Connection.get is called. This lead to flooding of worker logs with these logs finding it hard to locate the important ones.

Instead, since this is a debugging log, moving it up the ladder to every time a worker is fired up makes sense to get just enough information about the secrets backned loaded.

Example DAG this was tested with:

Logs in the worker:

2025-05-14 07:20:47.936410 [info     ] Task execute_workload[d8eaf207-2106-44ee-8100-16e5fb915f7c] received [celery.worker.strategy]
2025-05-14 07:20:48.045088 [info     ] [d8eaf207-2106-44ee-8100-16e5fb915f7c] Executing workload in Celery: token='eyJ***' ti=TaskInstance(id=UUID('0196cdaa-d590-75e3-a1ef-b13ca77a67bb'), task_id='test_task', dag_id='variable_flooding', run_id='manual__2025-05-14T07:20:47.238678+00:00', try_number=1, map_index=-1, pool_slots=1, queue='default', priority_weight=1, executor_config=None, parent_context_carrier={}, context_carrier={}, queued_dttm=None) dag_rel_path=PurePosixPath('dags/flooding.py') bundle_info=BundleInfo(name='dags-folder', version=None) log_path='dag_id=variable_flooding/run_id=manual__2025-05-14T07:20:47.238678+00:00/task_id=test_task/attempt=1.log' type='ExecuteTask' [airflow.providers.celery.executors.celery_executor_utils]
2025-05-14 07:20:48.061952 [info     ] Secrets backends loaded for worker [supervisor] backend_classes=['MetastoreBackend', 'EnvironmentVariablesBackend'] count=2
2025-05-14 07:20:48.521795 [warning  ] Server error                   [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "Variable with key 'TIME_ZONE' not found"}}
2025-05-14 07:20:48.521948 [error    ] Variable not found             [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "Variable with key 'TIME_ZONE' not found"}} key=TIME_ZONE status_code=404
2025-05-14 07:20:48.544115 [warning  ] Server error                   [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "Variable with key 'var_0' not found"}}
2025-05-14 07:20:48.544230 [error    ] Variable not found             [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "Variable with key 'var_0' not found"}} key=var_0 status_code=404
2025-05-14 07:20:48.547425 [warning  ] Server error                   [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "Variable with key 'var_1' not found"}}
2025-05-14 07:20:48.547524 [error    ] Variable not found             [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "Variable with key 'var_1' not found"}} key=var_1 status_code=404
2025-05-14 07:20:48.551086 [warning  ] Server error                   [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "Variable with key 'var_2' not found"}}
2025-05-14 07:20:48.551248 [error    ] Variable not found             [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "Variable with key 'var_2' not found"}} key=var_2 status_code=404
2025-05-14 07:20:48.554392 [warning  ] Server error                   [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "Variable with key 'var_3' not found"}}
2025-05-14 07:20:48.554483 [error    ] Variable not found             [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "Variable with key 'var_3' not found"}} key=var_3 status_code=404
2025-05-14 07:20:48.557760 [warning  ] Server error                   [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "Variable with key 'var_4' not found"}}
2025-05-14 07:20:48.557871 [error    ] Variable not found             [airflow.sdk.api.client] detail={'detail': {'reason': 'not_found', 'message': "Variable with key 'var_4' not found"}} key=var_4 status_code=404

Compared to earlier:

[2025-05-09, 15:03:48] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:03:48] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:05:00] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:05:00] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:05:41] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:05:41] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:06:21] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:06:21] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:07:03] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:07:03] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:07:55] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:07:55] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:07:55] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:07:55] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:07:55] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:07:55] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:08:06] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:08:06] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:09:15] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:09:15] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:10:11] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:10:11] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:10:20] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:10:20] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:10:33] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"
[2025-05-09, 15:10:33] INFO - Secrets backends loaded for worker: count=1: backend_classes=["EnvironmentVariablesBackend"]: source="supervisor"

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@amoghrajesh amoghrajesh requested a review from tirkarthi May 14, 2025 07:26
@amoghrajesh amoghrajesh requested review from ashb and kaxil as code owners May 14, 2025 07:26
@amoghrajesh amoghrajesh added backport-to-v3-0-test Mark PR with this label to backport to v3-0-test branch area:task-sdk area:logging and removed area:task-sdk labels May 14, 2025
@amoghrajesh amoghrajesh added this to the Airflow 3.0.2 milestone May 14, 2025
@kaxil kaxil merged commit e46508b into apache:main May 14, 2025
74 checks passed
@kaxil kaxil deleted the do-not-flood-logs branch May 14, 2025 10:43
github-actions bot pushed a commit that referenced this pull request May 14, 2025
@amoghrajesh
[v3-0-test] Do not flood worker logs with secrets backend loading logs (
8bfe3bc 
#50581)

(cherry picked from commit e46508b)

Co-authored-by: Amogh Desai <[email protected]>
closes: #50517
@github-actions GitHub Actions
Copy link

Backport successfully created: v3-0-test

Status Branch Result
✅ v3-0-test PR Link

kaxil pushed a commit that referenced this pull request May 20, 2025
@amoghrajesh @kaxil
[v3-0-test] Do not flood worker logs with secrets backend loading logs (
aef4350 
#50581)

(cherry picked from commit e46508b)

Co-authored-by: Amogh Desai <[email protected]>
closes: #50517
potiuk pushed a commit that referenced this pull request May 21, 2025
@github-actions @amoghrajesh
[v3-0-test] Do not flood worker logs with secrets backend loading logs (
5f81faf 
#50581) (#50596)

(cherry picked from commit e46508b)


closes: #50517

Co-authored-by: Amogh Desai <[email protected]>
kaxil pushed a commit that referenced this pull request Jun 3, 2025
@github-actions @amoghrajesh @kaxil
[v3-0-test] Do not flood worker logs with secrets backend loading logs (
d77ed10 
#50581) (#50596)

(cherry picked from commit e46508b)


closes: #50517

Co-authored-by: Amogh Desai <[email protected]>
sanederchik pushed a commit to sanederchik/airflow that referenced this pull request Jun 7, 2025
@amoghrajesh @sanederchik
Do not flood worker logs with secrets backend loading logs (apache#50581
36370ac 
)

closes: apache#50517
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kaxil kaxil kaxil approved these changes

@tirkarthi tirkarthi Awaiting requested review from tirkarthi

@ashb ashb Awaiting requested review from ashb ashb is a code owner

Assignees
No one assigned
Labels
area:logging area:task-sdk backport-to-v3-0-test Mark PR with this label to backport to v3-0-test branch
Projects
None yet
Milestone
Airflow 3.0.2
Development

Successfully merging this pull request may close these issues.

CeleryExecutor tasks flooded by repeated ‘Secrets backends loaded for worker’ INFO logs
2 participants
@amoghrajesh @kaxil