Skip to content

ci: add retries to e2e install/setup steps to reduce canary flakiness#14876

Open
bobbor wants to merge 4 commits into
mainfrom
ci/fix/canary-e2e-install-retries
Open

ci: add retries to e2e install/setup steps to reduce canary flakiness#14876
bobbor wants to merge 4 commits into
mainfrom
ci/fix/canary-e2e-install-retries

Conversation

@bobbor

@bobbor bobbor commented Jul 14, 2026

Copy link
Copy Markdown
Member

Description of changes

The daily on-schedule-canary-test.yml canaries have been failing intermittently (4 of the last 15 runs, each a different suite). All but one failure were transient network errors during dependency install/setup steps — the retry_count: 3 input only wraps test execution, while install/setup steps had no retry:

Run Failed suite Cause
2026-07-13 integ_react_storage plain run: yarn — better-sqlite3 prebuild download timeout, then getaddrinfo EAI_AGAIN nodejs.org during node-gyp fallback
2026-07-09 integ_react_graphql_api, integ_react_datastore_selective_sync Failed to download binary (statusCode 504): Gateway Time-out
2026-07-02 integ_rollup_datastore_basic_crud git failure (exit code 128) during samples-staging setup

Changes:

  • .github/workflows/callable-canary-e2e-tests.yml: the Install step now uses the repo's existing scripts/retry-yarn-script.sh -s install -n 3 (same pattern as callable-e2e-test-detox.yml) instead of a bare yarn.
  • .github/actions/setup-samples-staging/action.yml: the staging-branch selection is wrapped in a 3-attempt retry loop that distinguishes a genuinely missing branch (git ls-remote --exit-code → 2, falls back to default branch, no retry) from transient git/network failures (e.g. exit 128, retried). Emits a ::warning:: annotation if retries are exhausted.

The test-execution retry logic consuming retry_count is unchanged.

Validation

  • Both YAML files parse cleanly (yaml.safe_load).
  • The new branch-selection logic was functionally tested under bash -e -o pipefail with a mocked git covering: branch exists (checkout), branch missing (rc=2 → default branch), transient rc=128 (retries then falls back), fetch failure after successful ls-remote (retries).
  • CI-only change — no changeset needed.

Out of scope

  • The 2026-06-30 failure (integ_react_datastore_multi_auth_three_plus_rules, Cypress assertion 'Current User: test01' never rendered) is a genuine test-level flake not addressed by install retries.
  • callable-canary-sampleapp-tests.yml still has unretried npx create-react-app/npm install steps — candidate for a follow-up.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

@bobbor
bobbor requested a review from a team as a code owner July 14, 2026 16:30
@changeset-bot

changeset-bot Bot commented Jul 14, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: 95e1b59

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants