Propagate resolved task log level to language SDK runtimes#68712
Open
jason810496 wants to merge 2 commits into
Open
Propagate resolved task log level to language SDK runtimes#68712jason810496 wants to merge 2 commits into
jason810496 wants to merge 2 commits into
Conversation
jason810496
added
AIP-108: Coordinator
github-project-automation
Bot
moved this to Backlog
in AIP-108: Java Task SDK and the Language Coordinator Layer
ashb
reviewed
Jun 18, 2026
Member
|
We should also mention this in sdk contribution doc. Each sdk needs to handle this on their own. |
A language-SDK runtime (Java, Go) launched by a coordinator cannot read Airflow's configuration, so it could only guess the task log level from inherited environment variables. The subprocess starts before the StartupDetails handshake arrives, so carrying the level in that message would reach it too late. Passing the supervisor-resolved level in the environment at launch makes it available before the runtime configures logging.
The root log level alone does not reflect a user's per-logger overrides (`[logging] namespace_levels`), so a runtime would still emit logs the worker is configured to suppress. Passing the levels alongside the root level keeps the runtime's filtering consistent with the rest of Airflow. Each language SDK reads and applies these on its own, so the contract is documented for SDK authors.
jason810496
force-pushed
the
task-sdk/propagate-log-level-in-startup-details
branch
from
5c2a5d3 to
b5e450b
Compare
jason810496
commented
Jun 19, 2026
jason810496
left a comment
jason810496
left a comment
Member
Author
There was a problem hiding this comment.
Thanks Ash and TP for the review.
Addressed both comments in b5e450b.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Pointed out in above related link, we should propagate the
logging_levelinfo from Supervisor (Task-SDK world) to Lang-SDK runtime to respect the Airflow sideconf. So that no matter user configure the[logging/logging_level]usingairflow.cfgor secret backend, the Lang-SDK runtime should retrieve the same conf.How
Here're the approach we came up with:
AIRFLOW__LOGGING__LOGGING_LEVELenv, each Lang-SDK will consume byos.environthemself.java -classpath ... --comms=... --logs=... --logging-level=...CLI. Cons: We will introduce a new CLI flag for all the furtherconfpropagation. (There isn't--commsor--logsequivalent Airflow env, so having the existing--comms, --logsas-is is a valid direction)StartUpDetails.logging_level. Cons: Caught by @uranusjr that the Task subprocess had already started producing logs (e.g. logging for connecting to the coordinator) before the Task subprocess got theStartUpDetailsmsgpack, which causes the wrong logging level at the Task subprocess startup time.Was generative AI tooling used to co-author this PR?