Fix Java-SDK logging level#68696
Conversation
| const val LEVEL_PROPERTY = "airflow.logging.level" | ||
| const val LEVEL_ENV = "AIRFLOW__LOGGING__LOGGING_LEVEL" |
There was a problem hiding this comment.
Actually, I felt we should provide the log_level in StartupDetails.
I prefer to respect the Airflow side logging level, which means user might setup custom secret backend for the conf. So we need to send the [logging/log_level] info from supervisor.
There was a problem hiding this comment.
I agree, let’s do this. The Python implementation should probably also be changed to respect this log_level value (and fall back to various configurations).
Since this requires Python side changes, I would say for 3.3 let’s just default to INFO in SDKs and add support for Airflow configuration later.
There was a problem hiding this comment.
Sure, then I will go with this direction. This should solve Phani's review comment as well.
The logs socket was wrapped in use{}, which closed it as soon as
LogSender.configure() returned. Logs produced later while the task ran
then hit a closed channel, were buffered, and were lost at process
exit, so the UI showed no application logs for Java tasks. Keep the
socket open until the comm job completes so they are flushed.
|
|
||
| enum class Level { ERROR, DEBUG, } | ||
| // wireName is the level string the Airflow supervisor understands (structlog's | ||
| // NAME_TO_LEVEL). It has no TRACE, so TRACE maps to "debug"; a level the |
There was a problem hiding this comment.
I think TRACE should be mapped to NOTSET (0).
There was a problem hiding this comment.
Actually, why do we need to reinvent level? This enum can simply map 1:1 to Python levels, and we just resolve SLF4J levels to Python levels directly.
| const val LEVEL_PROPERTY = "airflow.logging.level" | ||
| const val LEVEL_ENV = "AIRFLOW__LOGGING__LOGGING_LEVEL" | ||
|
|
||
| fun threshold(): Level = parse(configuredLevel()) ?: Level.INFO |
There was a problem hiding this comment.
Can you test this scenario with this PR, I think it breaks.
In airflow.cfg
[logging]
logging_level = WARNING
Since it wont be in os.environ: no AIRFLOW__LOGGING__LOGGING_LEVEL set . Hence Java resolves to INFO.
Overall Result: Java WARNING logs will be silently dropped. A Python task in the same deployment with the same cfg would show WARNING logs.
I think it is identical config, divergent behavior by task language .
|
Moved to #68725 |
Why
A Java task's SLF4J logs (e.g.
logger.info(...)) fell through to stderr, which the supervisor tags asERROR, so every application log showed up asERRORin the UI regardless of its real level.Java-Task code:
What Airflow UI got:

What
AirflowSlf4jServiceProvider) registered viaMETA-INF/services, soLoggerFactory.getLogger(...)routes through the Airflow logs socket carrying each message's real level instead of writing to stderr.NAME_TO_LEVEL) understands;TRACEmaps todebugsince there is no TRACE level on the Python side.Was generative AI tooling used to co-author this PR?