Skip to content

Add version annotation to APIService to prevent race conditions during rolling upgrades#1813

Open
mdzhigarov wants to merge 1 commit intocarvel-dev:developfrom
mdzhigarov:fix-apiservice-version-annotation
Open

Add version annotation to APIService to prevent race conditions during rolling upgrades#1813
mdzhigarov wants to merge 1 commit intocarvel-dev:developfrom
mdzhigarov:fix-apiservice-version-annotation

Conversation

@mdzhigarov
Copy link
Copy Markdown
Contributor

Summary

This PR adds a version annotation to the APIService resource to prevent race conditions during kapp-controller rolling upgrades.

Problem

During kapp-controller rolling upgrades via kapp deploy <KC-new-release.yml>, a race condition occurs where:

  1. kapp doesn't detect any changes in the APIService resource (since it remains static)
  2. kapp reports deployment success before the new APIService webhook is fully operational
  3. This can lead to failed operations during the upgrade window

Solution

Add a version annotation (kapp-controller.carvel.dev/version) to the APIService that:

  • Gets updated with each release version (following the same pattern as the Deployment resource)
  • Ensures kapp detects changes in the APIService during upgrades
  • Forces kapp to properly wait for APIService readiness before reporting success

Changes

  • config/config/agg-api.yml: Add kapp-controller.carvel.dev/version: v0.0.0 annotation to APIService metadata
  • hack/build-release.sh: Add sed replacement to update version placeholder with actual release version during build

Implementation Details

This follows the exact same pattern already used for the Deployment resource:

  1. Static annotation with v0.0.0 placeholder in source files
  2. Build script replaces placeholder with actual version during release process
  3. Final release.yml contains the real version annotation

Test Plan

  • Verified annotation is correctly added to APIService
  • Tested build script properly replaces version placeholder
  • Confirmed ytt processing works without errors
  • Validated APIService structure remains valid

Impact

  • Positive: Eliminates race condition during kapp-controller upgrades
  • Risk: Minimal - only adds metadata annotation, no functional changes to APIService behavior
  • Compatibility: Fully backward compatible

Fixes race condition during kapp-controller rolling upgrades where APIService readiness validation was bypassed.

Made with Cursor

@carvel-bot carvel-bot added this to Carvel Apr 9, 2026
@mdzhigarov mdzhigarov force-pushed the fix-apiservice-version-annotation branch 2 times, most recently from eb9e40c to 66d8ad5 Compare April 9, 2026 06:50
…g rolling upgrades

During kapp-controller rolling upgrades via `kapp deploy`, a race condition can occur
where kapp reports success even though the APIService webhook is not ready yet. This
happens because kapp doesn't detect any changes in the APIService resource and
considers the deployment complete before the new APIService is fully operational.

This change adds a version annotation to the APIService that gets updated with each
release, ensuring kapp detects changes and properly waits for the APIService to be
ready during rolling upgrades.

Changes:
- Add kapp-controller.carvel.dev/version annotation to APIService metadata
- Update build script to replace version placeholder with actual release version
- Follow same pattern as existing Deployment resource versioning

Fixes race condition during kapp-controller upgrades where APIService readiness
is not properly validated.

Signed-off-by: Marin Dzhigarov <m.dzhigarov@gmail.com>
Made-with: Cursor
@mdzhigarov mdzhigarov force-pushed the fix-apiservice-version-annotation branch from 66d8ad5 to 020a89e Compare April 9, 2026 06:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants