[runtime] Support per-event-type configurable log levels for event log by weiqingy · Pull Request #609 · apache/flink-agents

weiqingy · 2026-04-07T06:53:31Z

Linked issue: #541 , #552

Purpose of change

Add per-event-type configurable log levels (OFF, STANDARD, VERBOSE) to the event log system, as designed in #552.

Key changes:

EventLogLevel enum (OFF, STANDARD, VERBOSE) with hierarchical config key inheritance (event-log.type.<EVENT_TYPE>.level)
At STANDARD level, large content fields are truncated using configurable per-field thresholds (max-string-length, max-array-elements, max-depth) with JSON-parseable wrapper objects (truncatedString, truncatedList, truncatedObject)
Top-level logLevel and eventType fields added to JSON log record schema for downstream filtering
EventFilter removed — fully subsumed by the log level system
eventLogTruncatedEvents counter metric for observability
Python config options and E2E tests

Config options:

Key	Default	Description
`event-log.level`	`STANDARD`	Root default log level
`event-log.type.<EVENT_TYPE>.level`	(inherits)	Per-type override
`event-log.standard.max-string-length`	`2000`	Max string chars at STANDARD
`event-log.standard.max-array-elements`	`20`	Max array elements at STANDARD
`event-log.standard.max-depth`	`5`	Max nesting depth at STANDARD

Tests

EventLogLevelResolverTest — 11 tests: exact match, hierarchy inheritance, root/built-in defaults, case insensitivity, edge cases
JsonTruncatorTest — 8 tests: string/array/depth truncation, wrappers, protected fields, composition, disabled thresholds
FileEventLoggerTest — 7 new tests: STANDARD truncation, VERBOSE preservation, OFF suppression, per-type override, hierarchical inheritance, JSON schema fields, backward-compatible deserialization
Python E2E tests — 3 new: VERBOSE, OFF, STANDARD truncation

API

New public API: EventLogLevel enum in api module
Removed public API: EventFilter interface (0.x pre-release, no known users)
Modified: EventLoggerConfig — removed eventFilter field/builder method

Documentation

doc-needed

- Create EventLogLevel enum (OFF, STANDARD, VERBOSE) with case-insensitive fromString() parser in api module - Add 4 new config options to AgentConfigOptions: event-log.level, event-log.standard.max-string-length, event-log.standard.max-array-elements, event-log.standard.max-depth - Delete EventFilter.java — subsumed by the log level system - Remove eventFilter field and references from EventLoggerConfig and FileEventLogger - Remove 6 EventFilter-related tests from FileEventLoggerTest Part of apache#541: per-event-type configurable log levels.

- EventLogLevelResolver: resolves effective EventLogLevel for event type strings using hierarchical config key inheritance (exact match → parent package walk-up → root default → built-in STANDARD). Caches results in ConcurrentHashMap. Takes Map<String, Object> to avoid runtime→plan module dependency. - JsonTruncator: truncates Jackson JsonNode trees per configurable thresholds (max-string-length, max-array-elements, max-depth). Produces JSON-parseable wrapper objects (truncatedString, truncatedList, truncatedObject). Protected fields (eventType, id, attributes) at top level are never truncated. - Comprehensive unit tests for both classes covering exact match, hierarchy inheritance, edge cases, all truncation types, composition, and disabled thresholds. Part of apache#541: per-event-type configurable log levels.

…operator - EventLogRecord: add logLevel, eventType, truncator, and truncatedEventsCounter fields. Old 2-arg constructor defaults to VERBOSE for backward compatibility. - EventLogRecordJsonSerializer: write logLevel and eventType as top-level JSON fields. Apply JsonTruncator when level is STANDARD, increment truncatedEventsCounter when truncation occurs. - EventLogRecordJsonDeserializer: read optional logLevel field, default to VERBOSE for old records (backward compat). - FileEventLogger: resolve log level per event via EventLogLevelResolver, skip OFF events, initialize truncator from agent config. Accept truncation counter via setter. - ActionExecutionOperator: pass full AgentConfiguration.getConfData() to logger via agentConfig property. Wire truncation counter from BuiltInMetrics to FileEventLogger after open(). - BuiltInMetrics: add eventLogTruncatedEvents counter with getter. - FileEventLoggerTest: add 7 new tests for STANDARD truncation, VERBOSE preservation, OFF suppression, per-type override, hierarchical inheritance, new JSON fields, backward deserialization. Part of apache#541: per-event-type configurable log levels.

- Add EVENT_LOG_LEVEL, EVENT_LOG_MAX_STRING_LENGTH, EVENT_LOG_MAX_ARRAY_ELEMENTS, EVENT_LOG_MAX_DEPTH config options to Python AgentConfigOptions (keys and defaults match Java) - Update existing test_python_event_logging to verify new logLevel and top-level eventType fields in JSON output - Add test_event_log_verbose_level: verifies VERBOSE produces full content without truncation - Add test_event_log_off_level: verifies OFF suppresses all event logging - Add test_event_log_standard_truncation: verifies STANDARD with low max-string-length threshold produces truncatedString wrappers - Extract _run_event_logging_pipeline() and _read_log_records() helpers Part of apache#541: per-event-type configurable log levels.

- FileEventLogger: use PythonEvent.getEventType() (Python module path) for level resolution instead of EventContext.getEventType() (Java wrapper class name). Without this, per-type Python config keys like event-log.type.flink_agents.api.events.event.OutputEvent.level would never match. - EventLogRecordJsonSerializer: use Python module path for top-level eventType field for PythonEvents, consistent with the eventType inside the event object. - BuiltInMetrics: remove dead markEventTruncated() method — counter is incremented via serializer path directly. Part of apache#541: per-event-type configurable log levels.

weiqingy · 2026-04-07T06:55:22Z

Hey @wenjin272 @xintongsong could you please take a look at the PR? Feedback is welcome. Thanks!

wenjin272

Hi, @weiqingy, thanks for your work. I left some minor comments.

wenjin272 · 2026-04-10T03:04:29Z

runtime/src/main/java/org/apache/flink/agents/runtime/eventlog/EventLogLevelResolver.java

+        Map<String, Object> data = confData != null ? confData : Collections.emptyMap();
+
+        // Parse root default
+        Object rootValue = data.get(ROOT_LEVEL_KEY);


Shall we just use EVENT_LOG_LEVEL.getKey() here?

wenjin272 · 2026-04-10T03:43:52Z

...ime/src/main/java/org/apache/flink/agents/runtime/eventlog/EventLogRecordJsonSerializer.java

+            if (pythonType != null) {
+                topLevelEventType = pythonType;
+            }
+        }


The logic for extracting the python event type is the same as in FileEventLogger. We can extract a common util function.

wenjin272 · 2026-04-10T03:52:17Z

runtime/src/main/java/org/apache/flink/agents/runtime/eventlog/EventLogRecord.java

+    private final EventLogLevel logLevel;
+    private final String eventType;
+    private final transient JsonTruncator truncator;
+    private final transient Counter truncatedEventsCounter;


Maybe we can mark truncator and truncatedEventsCounter as @Nullable.
Additionally, these two members are actually required variables for the serialization of EventLogRecord, rather than part of its content. During deserialization, we have to set them to null. I wonder if they could be removed from within EventLogRecord itself.

wenjin272 · 2026-04-10T06:41:12Z

api/src/main/java/org/apache/flink/agents/api/configuration/AgentConfigOptions.java

+     * values are "OFF", "STANDARD", and "VERBOSE". Defaults to "STANDARD".
+     */
+    public static final ConfigOption<String> EVENT_LOG_LEVEL =
+            new ConfigOption<>("event-log.level", String.class, "STANDARD");


Maybe we can just use enum here

public static final ConfigOption<EventLogLevel> EVENT_LOG_LEVEL = new ConfigOption<>("event-log.level", EventLogLevel.class, EventLogLevel.STANDARD);

wenjin272 · 2026-04-10T06:42:11Z

runtime/src/main/java/org/apache/flink/agents/runtime/eventlog/EventLogLevelResolver.java

+        this.rootDefault =
+                rootValue != null
+                        ? EventLogLevel.fromString(rootValue.toString())
+                        : BUILT_IN_DEFAULT;


Shall we just use EVENT_LOG_LEVEL.getDefaultValue() here?

wenjin272 · 2026-04-10T06:50:00Z

runtime/src/main/java/org/apache/flink/agents/runtime/operator/ActionExecutionOperator.java

            return;
        }
        eventLogger.open(new EventLoggerOpenParams(runtimeContext));
+        if (eventLogger instanceof FileEventLogger) {


Why is it necessary to specifically check the type of eventLogger here?

weiqingy added 5 commits April 6, 2026 22:15

github-actions bot added doc-needed Your PR changes impact docs. fixVersion/0.3.0 The feature or bug should be implemented/fixed in the 0.3.0 version. priority/major Default priority of the PR or issue. labels Apr 7, 2026

weiqingy mentioned this pull request Apr 7, 2026

[FLINK-AGENTS-541][api][runtime] Support per-event-type configurable log levels for event log #542

Closed

3 tasks

weiqingy added 2 commits April 7, 2026 10:50

Retrigger CI

c3bbc3e

Retrigger CI

48ed7db

wenjin272 reviewed Apr 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[runtime] Support per-event-type configurable log levels for event log#609

[runtime] Support per-event-type configurable log levels for event log#609
weiqingy wants to merge 7 commits intoapache:mainfrom
weiqingy:541-impl

weiqingy commented Apr 7, 2026

Uh oh!

weiqingy commented Apr 7, 2026

Uh oh!

wenjin272 left a comment

Uh oh!

wenjin272 Apr 10, 2026

Uh oh!

wenjin272 Apr 10, 2026

Uh oh!

wenjin272 Apr 10, 2026

Uh oh!

wenjin272 Apr 10, 2026

Uh oh!

wenjin272 Apr 10, 2026

Uh oh!

wenjin272 Apr 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

weiqingy commented Apr 7, 2026

Purpose of change

Tests

API

Documentation

Uh oh!

weiqingy commented Apr 7, 2026

Uh oh!

wenjin272 left a comment

Choose a reason for hiding this comment

Uh oh!

wenjin272 Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

wenjin272 Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

wenjin272 Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

wenjin272 Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

wenjin272 Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

wenjin272 Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants