Support extra field for Coordinator

[jason810496]

Why

While verifying KuberntesExecutor setup for Multi-Lang feature, I found that we need a new config mapping to represent coordinator -> pod_template_file. For example, KubernetesExecutor needs image with JVM runtime for Java-Task.

How

Here're several directions that I went through:

1. (Current choice) Generic extra object on the [sdk] coordinators entry. Regarding the extra field naming, I also considered executor, deployment, etc. But I prefer as extra to make it more extendable as user might introduce other optional fields in extra for non-deployment purpose (e.g. referencing conn_id).

{
  "jdk-17": {
    "classpath": "airflow.sdk.coordinators.java.JavaCoordinator",
    "kwargs": {"java_executable": "/usr/lib/jvm/java-17-openjdk/bin/java", "jvm_args": ["-Xmx1024m"]},
    "extra": {"pod_template_file": "/opt/airflow/pod_templates/java.yaml"}
  }
}

2. An optional pod_template_file field at first level -- Cons: coupling the optional / provider-level field at first level, needs coordinator code change for every new field.

{
  "jdk-17": {
    "classpath": "airflow.sdk.coordinators.java.JavaCoordinator",
    "kwargs": {"java_executable": "/usr/lib/jvm/java-17-openjdk/bin/java", "jvm_args": ["-Xmx1024m"]},
    "pod_template_file": "/opt/airflow/pod_templates/java.yaml"
  }
}

3. An optional pod_template_file in kwargs level -- Same cons as point 2.

{
  "jdk-17": {
    "classpath": "airflow.sdk.coordinators.java.JavaCoordinator",
    "kwargs": {
      "java_executable": "/usr/lib/jvm/java-17-openjdk/bin/java",
      "jvm_args": ["-Xmx1024m"],
      "pod_template_file": "/opt/airflow/pod_templates/java.yaml"
    },
  }
}

4. A fully decouple config setup -- AIRFLOW__KUBERNETES__COORDINATOR_TO_POD_TEMPLATE_FILE

AIRFLOW__SDK__COORDINATORS='{"jdk-17": {"classpath": "airflow.sdk.coordinators.java.JavaCoordinator", "kwargs": {"jars_root": ["/files/java-bundles"]}}}'
AIRFLOW__SDK__QUEUE_TO_COORDINATOR='{"java": "jdk-17"}'
AIRFLOW__KUBERNETES__COORDINATOR_TO_POD_TEMPLATE_FILE='{"jdk-17": "/opt/airflow/pod_templates/java.yaml"}'

The cons is quite obvious, user have to understand the queue -> coordinator -> pod_template relationship to setup the Multi-Lang with KubernetesExecutor properly. IMO, it's a too complicated, and it's fine to store the optional metadata in core level config but parsing the optional info at provider level.

What

• Add an extra field to _CoordinatorSpec (kept separate from kwargs) and CoordinatorManager.extra_for_queue.
  • The CoordinatorManager.extra_for_queue method will not instantiate the coordinator instance.

Next

• Decide pod_template and image based on Coordinator for lang-SDK tasks on KubernetesExecutor #68713

---

Was generative AI tooling used to co-author this PR?

• Yes, with help of Claude Code (Opus 4.8) following the guidelines