Skip to content

OpsRequest progress tracking never converges when FlatInstanceOrdinal is enabled #10704

Description

@shanshanying

What happened

When a component has flatInstanceOrdinal: true and contains named instance templates, OpsRequest (HorizontalScaling, Start, Stop, VerticalScaling, VolumeExpansion, RebuildInstance) cannot converge — the progress tracking stays in a pending state indefinitely.

Root Cause

The OpsRequest progress tracking code uses deprecated pod name generation functions that always produce non-flat names, while the InstanceSet controller uses the flat name builder when FlatInstanceOrdinal is enabled. The name mismatch means the progress tracker can never find the pods it expects.

Flat naming (used by InstanceSet controller via instancetemplate.PodNameBuilder / flatNameBuilder):

All ordinals are globally unique across templates. For a component with default template replicas=2 and a named template "big" replicas=1:

mycluster-mysql-0   (default template)
mycluster-mysql-1   (default template)
mycluster-mysql-2   (named template "big")

Non-flat naming (used by OpsRequest via deprecated instanceset.GenerateAllInstanceNames):

mycluster-mysql-0      (default template)
mycluster-mysql-1      (default template)
mycluster-mysql-big-0  (named template "big")  ← does not match!

The progress tracker expects pod mycluster-mysql-big-0 but the actual pod is mycluster-mysql-2. It never matches, so the progress stays Pending forever.

Key Code Paths

File Function Issue
pkg/operations/ops_runtime.go:548 generateAllPodNamesToSet Deprecated comment already exists, uses instanceset.GenerateAllInstanceNames
pkg/operations/ops_runtime.go:566 generateAllPodNames Always calls non-flat name generation
pkg/operations/ops_runtime.go:185 GenerateTemplateInstanceNames Always calls non-flat name generation
pkg/controller/instanceset/instance_util.go:196 GenerateAllInstanceNames Deprecated, no flat awareness
pkg/controller/instancetemplate/name_builder.go:34 NewPodNameBuilder Correctly dispatches flat vs default — but OpsRuntime never uses this

Affected OpsRequest Types

Ops Type Affected? Impact
HorizontalScaling getCreateAndDeletePodSet, filterHorizontalScalingSpec, getToOnlineInsCountMap, restoreDataFromBackup
Start Computes createdPodSet for progress tracking
Stop Computes deletedPodSet for progress tracking
VerticalScaling Computes updatedPodSet via GenerateTemplateInstanceNames
VolumeExpansion Uses GenerateTemplateInstanceNames
RebuildInstance Uses both GenerateTemplateInstanceNames and GenerateInstanceNameSet
Restart Uses actual pod names from workload
Reconfiguring Does not depend on pod name generation

Steps to Reproduce

  1. Create a cluster with a component that has flatInstanceOrdinal: true and named instance templates
  2. Issue a HorizontalScaling OpsRequest to scale out
  3. Observe that the OpsRequest never completes — progress stays in processing state indefinitely
  4. The actual pods are created correctly (with flat names) but the ops progress tracking cannot find them because it looks for non-flat pod names

Environment

  • KubeBlocks version: main branch

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions