coordinator: avoid cdc hang on recvMessages (#4441)

[ti-chi-bot]

This is an automated cherry-pick of #4441

What problem does this PR solve?

Issue Number: close #4440

What is changed and how it works?

Coordinator Message Handling: Modified the recvMessages function in the coordinator to prevent potential hangs. The change involves updating the select statement to directly attempt sending messages to the eventCh rather than using a default case, ensuring proper synchronization and avoiding blocking.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Please refer to [Release Notes Language Style Guide](https://pingcap.github.io/tidb-dev-guide/contribute-to-tidb/release-notes-style-guide.html) to write a quality release note.

If you don't think this PR needs a release note then fill it with `None`.

Summary by CodeRabbit

• Bug Fixes
  • Improved message handling reliability by ensuring proper synchronization when processing incoming messages.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 0a34301b-54e2-4d5b-90a7-d3da48c87780

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Code Review

This pull request changes the message sending behavior in coordinator.go from non-blocking to blocking by removing the default case in the select statement. Feedback highlights a potential deadlock during shutdown or error handling, where a sender could block indefinitely on c.eventCh.In() while Stop() waits for the wait group. To resolve this, the reviewer suggests introducing a stopCh to signal cancellation to blocked senders.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

Potential Deadlock during Shutdown / Error Handling

There is a potential deadlock scenario when a goroutine in the errgroup fails or during coordinator shutdown:

1. Goroutine Failure: If any goroutine in the errgroup (e.g., run, gcCleaner.Run) fails, the errgroup context is canceled, causing runHandleEvent to exit.
2. No Reader: Once runHandleEvent exits, nothing is reading from c.eventCh.Out().
3. Blocked Sender: If a new message arrives before Stop() is called, recvMessages passes the c.closed check and blocks on c.eventCh.In() <- ....
4. Deadlock in Stop(): When Stop() is eventually called, it executes c.msgGuardWaitGroup.Wait() before canceling the context or closing the channel. Since recvMessages is blocked on the channel send, it never calls Done(), causing Stop() to hang indefinitely.

Recommended Solution

Introduce a stopCh chan struct{} to signal cancellation to blocked senders:

1. Add stopCh chan struct{} to the coordinator struct.
2. Initialize it in New: stopCh: make(chan struct{}).
3. Close it in Stop() before waiting on the wait group:

   close(c.stopCh)
   c.msgGuardWaitGroup.Wait()

4. Select on c.stopCh in recvMessages as shown in the suggestion.

Suggested change

	select {
	case <-ctx.Done():
	return context.Cause(ctx)
	default:
	c.eventCh.In() <- &Event{message: msg}
	case c.eventCh.In() <- &Event{message: msg}:
	}
	select {
	case <-ctx.Done():
	return context.Cause(ctx)
	case <-c.stopCh:
	return errors.New("coordinator stopped")
	case c.eventCh.In() <- &Event{message: msg}:
	}

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: wk989898

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [wk989898]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment