Preemption error message should dominate

[robertdavidsmith]

Hi,

In same cases, a job is preempted (so ends up as state preempted as visible in lookout). However the preemption reason isn't visible in the RESULTS tab in lookout as instead another error is there.

• If a running job is preempted but the job writes to terminationMessagePath, the termination message is displayed in lookout, and the preemption reason is lost. (We have seen examples where jobs catch SIGTERM but then shut down uncleanly and so write a termination message as they are shutting down).
• If a pending job is preempted, the Kubernetes events are displayed as the error message, and again the preemption reason is lost.

Please can you ensure the preemption reason is not lost in these cases. Either append to the existing error message or just display the preemption reason.

Thanks,

Rob

[robertdavidsmith]

Hi @williamvega

We discussed the design some more, we decided to add a jsonb column to the lookout job runs table, called something like scheduler_termination_reason. For preempted/cancelled jobs, this will contain structured information as to why the job was killed. @masipauskas is going to post example json shortly, and also a sample addtion to JobPreemptedEvent.

[masipauskas]

I think we should, basically add ReasonArgs dict[str, str] to JobPreemptedEvent. Then serialize new jsonb column as follows:

JobPreemptedEvent(ReasonArgs: dict[str, str]) // new property for now nil
{
    "reason": JobPreemptedEvent.Reason
    "args": {
        "preemptingRunId": JobPreemptedEvent.PreemptiveRunId (if not null/empty)
        ... JobPreemptedEvent.ReasonArgs
    }
}

JobCancelledEvent
{
    "reason": JobCancelledEvent.Reason
    "args": {
        "requestor": JobCancelledEvent.Requestor (if not null/empty)
    }
}

This enables us to do analytics in relatively simple way, as well as attaching more structured context to both Preemption/Cancellation events.