Search before asking
Motivation
Currently, we store the task log and task output in the same file.
The problem is that the task log is generated by DolphinScheduler, the end users don't actually understand the underlying logic. They are more care about the output of their tasks.
Therefore, they need to locate their task outputs within the vast array of task instances.
Additionally, since we often use separate threads to collect task outputs, this can also lead to issues where task outputs and task log entries become interspersed and out of order.
This DSIP aims to separate the task log and task output and use an extra file to store the task output.
Design Detail
todo:
Compatibility, Deprecation, and Migration Plan
todo:
Test Plan
todo:
Code of Conduct
Search before asking
Motivation
Currently, we store the task log and task output in the same file.
The problem is that the task log is generated by
DolphinScheduler, the end users don't actually understand the underlying logic. They are more care about the output of their tasks.Therefore, they need to locate their task outputs within the vast array of task instances.
Additionally, since we often use separate threads to collect task outputs, this can also lead to issues where task outputs and task log entries become interspersed and out of order.
This DSIP aims to separate the task log and task output and use an extra file to store the task output.
Design Detail
todo:
Compatibility, Deprecation, and Migration Plan
todo:
Test Plan
todo:
Code of Conduct