Commit 52d6746
ci: a repo-wide test pipeline, and the one check that gates merge on it (#772)
* ci: a repo-wide test pipeline, and the suites Camoufox was missing
Nothing checked a pull request before this. `build.yml` runs on tags and takes
about forty minutes, and `lint.yml` ran a single static script, so a change
could reach main having had no browser suite run against it at all.
This adds one pipeline, driven identically from a pull request, a push to main,
and -- through `workflow_call` -- any caller that needs to test a specific
browser version, so there is exactly one definition of "the tests pass".
resolve ──┬─ static ────────── tribal rules, skiplist, self-tests
├─ pythonlib ─────── the package's own tests
└─ build ──┬─ playwright upstream × 6 shards (conformance)
├─ playwright vendored (regression)
├─ native ───────────── leaks, contexts
├─ patch guards ─────── one per spoofing patch
├─ build-tester ─────── 8 fingerprint profiles
└─ sundial ──────────── stealth grade (off, see below)
│
summary ──► one comment on the PR
Two Playwright suites, because they answer different questions. `tests/` is a
frozen ~v1.55-era fork carrying roughly 1800 lines of Camoufox adaptations, so
every test in it has a known prior outcome: that is the regression check. The
upstream suite is fetched fresh at the tag `ci/versions.py` resolves and runs
unmodified, which is the conformance check -- `ci/pw_camoufox_plugin.py` adapts
the environment around it rather than editing it, hooking BrowserType at the
_impl layer so upstream can refactor its fixtures freely.
`native-tests/` covers what neither can ask about: that resource cost does not
scale with launch count (the shape an FD or socket leak actually has), that two
contexts in one browser get different fingerprints while two pages in one
context get the same one (get this wrong and per-context injection silently
degrades to process-global, which passes every single-context test there is),
and that decisions already made stay made -- `ci/tribal-rules.yml` lists them
with the issue or PR that settled each.
Cost is tiered so a two-second lint failure never reaches a build, and a
driver-only pull request never builds at all: it fetches the published release
and tests against the build users are actually running, a minute instead of
seventy. Merges gate on one required check, `All tests passed`, so the
branch-protection list does not need editing every time a suite is added or
resharded; `ci/branch-protection.json` holds the settings so they are reviewable
rather than lore.
**The stealth check ships disabled** (`ci/sundial.yml: enabled: false`). It
drives a private detection suite, and the deployment it talks to predates that
suite's score mode; an older one ignores `?score=1` and posts the entire report
-- every vector's id, name, brief, source and value -- to whatever collector
asked. Receiving that on a public runner and discarding it afterwards is not the
same guarantee as never being sent it, so while the flag is false the job is not
scheduled, no credential enters a runner, and `run_sundial.py` refuses a hand-run
too. When it is enabled, `redact()` publishes a grade and counts against a
runtime whitelist and refuses anything that is not already aggregated.
Also included: the fixes these suites exposed on a clean runner -- build-tester
hashing canvas pixels rather than a prefix of the data URL, the virtdisplay
cleanup when Xvfb has already died, a juggler sandbox released on frame destroy
rather than only on navigation, and the pythonlib geometry and version-floor
corrections. `lint.yml` is removed because the static job absorbed its one check.
Verified locally: ci/tests 68 passed, tribal rules 24 passed, pythonlib 209
passed, input-dispatch clean, `ci.versions` resolves 152.0.4/beta.31 against
playwright v1.61.0, and `ci.summarize` folds a run to "all suites passed".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* ci: make result files survive the trip from job to summary
The first full run failed, and the summary could not say why: five suites came
back "required, but produced no result", including two whose jobs had passed.
Three separate plumbing bugs, none of them in a test.
**Hidden files.** `actions/upload-artifact@v4` excludes dotfiles unless told
otherwise, and every result we write lives under `.ci-work`. The jobs whose
`path:` was a list containing a glob uploaded nothing at all -- the Playwright
suites and the leak suite each wrote their evidence and then had it silently
dropped:
evidence -> .../.ci-work/results/playwright_vendored.json (fail, 1203 tests)
##[warning]No files were found with the provided path: .ci-work/results/
.ci-work/junit-*.xml. No artifacts will be uploaded.
**Common root.** Where a list did upload, the second entry moved
upload-artifact's common root from `.ci-work/results/` up to `.ci-work/`, so the
JSON arrived at `results/build_tester.json` instead of the artifact root. The
summary merges every `results-*` into one directory and `load_all()` globs a
single level, so the file was there and invisible. build_tester passed and was
reported missing.
Every `results-*` artifact now uploads exactly `.ci-work/results/`, with
diagnostics (junit XML, the build-tester graded tree) split into their own
`diagnostics-*` artifacts that the summary's `results-*` pattern ignores.
`include-hidden-files: true` everywhere that touches `.ci-work`.
**A required name nothing writes.** `static` was in the required list, but it is
a job, not a suite -- no runner writes a result by that name, so summarize
reported it missing on every run including a wholly green one. The suites that
job runs are the pipeline self-tests, which write no result, and native_rules,
which is required by name. The job is already covered: the gate fails on any job
that is not success.
Three guards, each verified by reintroducing the bug it catches:
- results-* artifacts upload exactly one path, so nothing nests
- anything touching .ci-work sets include-hidden-files
- every required name is one some runner can actually write
This changes no test. The real failures the first run found -- 6 in the vendored
suite, plus upstream shards 1 and 5 and the leak suite -- were masked by the
above and should now be reported rather than swallowed.
ci/tests 71 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* test: two failures that were the tests' fault, not the browser's
**The leak check waited on the wrong set of processes.**
`test_a_single_launch_leaves_nothing` failed with Gecko's GPU probe still alive:
1 process(es) this test started are still alive: glxtest(2887, now ppid=1)
`settle()` polled `children(recursive=True)`, but `survivors()` judges the
sampled PID set -- deliberately, so that a process reparented to init cannot
hide a leak. Those two sets differ exactly when a process outlives its parent:
it stops being our child, `settle()` sees nothing left and returns at once, and
anything still winding down is reported as leaked. `glxtest` does this on every
launch; it is spawned by Gecko, its parent exits first, and it needs a moment.
So settle on the set the assertion actually uses. This is a grace period, not an
exemption -- a process that is still there when the timeout expires fails the
test exactly as before, and no name is special-cased.
**Playwright renamed a protocol method the tracing tests spelled out.**
`Page.waitForEventInfo` is `Page.__waitInfo__` in newer versions, so two tracing
assertions failed on a name, not on behaviour. The suite is pinned to a range
(`playwright<1.63`), not a version, so hard-coding either spelling is wrong.
Normalised in `get_trace_actions()`, next to the comment about the last time
Playwright moved this data -- the tests care which actions ran and in what
order, not what Playwright calls them this month.
Neither of these was Camoufox misbehaving.
Still failing, and genuinely about the browser or by design -- triaged next:
navigation popup load state, locator handler visibility, clock pause off by 1ms,
websocket close reason, and the three upstream ones (request headers, worker
locale, screencast viewport) which all look like deliberate spoofing divergence
and probably belong in the skiplist with a stated reason.
ci/tests 71 passed, tribal rules 24 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* fix: the failures the new pipeline found, and the flake that hid them
Eleven gates were red on PR #9. Each one is now either a fixed defect or an
entry that says why the test cannot apply here -- nothing is silenced.
One real browser bug, found by the conformance suite:
The compositor-backed screencast added in fa8a935 reported the *scaled frame*
size as the viewport. Playwright's Firefox delegate maps deviceWidth/Height
straight onto the client-visible viewportWidth/viewportHeight, and every other
backend fills them from the page's viewport -- the native path right below it
sends pageWidth/pageHeight, clamped to the viewport and never scaled. So a
client asking for a 500x400 frame of a 1000x400 page was told the viewport was
500x200, and asking for a frame *larger* than the page reported a viewport
larger than the page. Confirmed against the built binary, fixed, and confirmed
again: 11 screencast and video tests pass, where the requested size no longer
moves the reported viewport at all.
Four stale expectations in the vendored suite. All four pass in upstream's
v1.61 suite against this same binary, which is what identified them as the
suite's problem rather than the browser's:
websocket Firefox no longer collapses a refused handshake to
CLOSE_ABNORMAL; it reports the HTTP status, like the other
engines. The handler also set a settled future twice, which
surfaced as a suite ERROR rather than a failure.
navigation Firefox now reports a window.open('') popup as "complete". The
old assertion also mis-parsed -- the conditional bound to the
whole assert, so the non-Firefox arm compared nothing.
locator expect().to_be_visible() kept re-arming the locator handler the
click was still waiting to see finish, so the check meant to
observe the interstitial kept it alive. One-shot is_visible().
page_clock resume() before reading the clock added the real second spent in
wait_for_timeout back, landing at 1001 -- one millisecond out,
every time.
Two upstream tests that encode a stock-Firefox quirk this fork does not
reproduce, now in the skiplist with the reason:
A worker inheriting the context locale -- upstream expects en-US from a
ru-RU context, citing playwright#38919, because stock Firefox applies the
locale to the page and not to its workers. Camoufox sets it below that layer,
so matching upstream would mean reintroducing a main-thread/worker
disagreement that anything looking in both places gets for free.
"Firefox" in the User-Agent -- the bare binary advertises its own build
token; the Python package replaces it when it injects a fingerprint. The
vendored suite already asserts what this layer can promise.
And three pieces of the harness that were reporting badly:
A profile that asked for llvmpipe is no longer graded as headless. camoufox's
own preset pool ships "llvmpipe, or similar", so when that preset is drawn,
reporting it is the WebGL spoof working -- and grading it a failure made this
gate fail at random depending on which presets the run happened to draw. The
check still fails on a software renderer the profile did not ask for, which
is the case it exists for.
junit ids put a test's class in the path (test_page_clock/TestWhileRunning.py
::test_should_pause), naming a directory that does not exist -- so the id
could not be fed back to pytest and no skiplist entry could match it.
A `test:` skiplist entry was compared for exact equality against a node id
that always ends in [firefox], so every such entry was a silent no-op: the
test went on running and failing while the list read as handled.
Finally, `mach bootstrap` pulls toolchains from Taskcluster, and a connection
reset there failed the whole pull request (run 34673115086). ci.run_prepare
retries the two steps that download things, and only when the failure reads as
transient -- a failed patch hunk still fails on the first try, because retrying
a broken tree only spends a runner to reach the same answer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* docs(ci): say what the stealth gate actually guarantees now
The README described a `ci` role that is not merged into sundial's master and
so is not deployed, and a kill switch that had since been flipped. Both claims
now match what is running.
The substantive change is separating the two halves of the guarantee, because
only one of them is enforced by the server: `guest` is refused the
private-vector bundle by sundial's middleware, and this repository refuses to
process anything that is not a score payload. What is still missing is a server
that refuses the *request* -- which is what the `ci` role adds, and why the
upgrade path is worth keeping written down rather than implied.
Also documents ci.run_prepare, since "the build retries" is the kind of thing
that needs its limits stated: the network, and nothing else.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* fix(ci): authenticate to sundial with either shape of credential
`SUNDIAL_AUTOMATION_KEY` has been two different things over this repository's
life -- an automation key for `/automated?key=`, which is what sundial's
`make pages-automation-keys` mints and what resolves to the `guest` role, and a
password for the login form. They are indistinguishable by inspection, and the
gate only knew how to use it as a password, so a repository holding a key would
have failed to log in with no hint as to why.
Try the key route first, since it needs no username and so nothing has to be
kept in step with whatever `GUEST_USER` was set to, and fall back to the form.
When the secret really is a password the only cost is one extra request that
401s. If both routes fail the error names both attempts and says what the secret
is supposed to be.
Also sends browser headers on every request rather than `User-Agent:
camoufox-harness`. Cloudflare sits in front of this host and refuses a document
request from a non-browser agent before it reaches sundial at all, which
produces a 403 that looks like a permissions problem and is not one.
test_a_disabled_gate_makes_no_request now makes every route out fatal --
authenticate() and both login functions, not just the one main() used to call --
so the kill switch cannot be bypassed through a path the test does not watch.
Mutation-checked: making the gate ignore `enabled:` fails it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* fix(ci): never let run_prepare invent a success
`--attempts 0` would have fallen straight past the loop bound and returned 0
without running the step at all, which is the one answer it must not invent.
Clamps to one attempt and makes falling out of the loop an assertion rather
than a bare success.
Also corrects a direction in the screencast comment: the native path it
contrasts with sits above _startSnapshotScreencast, not below it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* fix(ci): a sundial category nobody has ruled on now fails the gate
ci/sundial.yml partitions sundial's taxonomy: its nine SECTIONS labels
(Identity, Security, JS Engine, Graphics, Display, Locale, Audio, CPU, Network)
are exactly the six gated plus the three ungated, with nothing left over. That
is now asserted, because it is the property the pass rate depends on and
nothing was checking it.
If sundial grows a tenth section, every check in it used to fold into
"out of scope" -- measured, never gated, and indistinguishable in the summary
from a category someone had deliberately decided not to gate. That answers
"does Camoufox claim this?" by default, in the only direction that never fails
a build: a stealth blind spot that reads as a clean run.
The count is now carried separately and fails the gate, with a note saying to
put the new section in one list or the other. Only ever a count -- which
category, like which vector, does not leave redact().
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* fix(ci): stream the prepare log, and make its timeout real
Two problems with capturing the output of `make dir` / `make mozbootstrap`
instead of streaming it. The visible one: the step printed nothing for minutes
while aria2c pulled a 500MB tarball, which looks exactly like a hung job. The
one that mattered: draining the pipe on the calling thread blocks in readline
until EOF and only *then* reaches proc.wait(timeout=...), so a step that wedged
without printing anything -- a stalled download, precisely the failure this
module exists for -- would never have been timed out at all.
The reader now runs on its own thread, so output appears as it arrives and the
timeout covers a silent hang. Both are asserted against real processes:
stdout and stderr both survive into the text the transient-classifier reads,
and `sleep 60` under a 2s timeout dies in 2s with exit 124 rather than
reporting success.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* docs(ci): two comments that said less than they meant to
The skiplist header claimed to quote failure text it only described, and the
sundial header comment had lost the word that made it a sentence.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* docs(ci): the stealth job comment described the role we do not have yet
It claimed the credential is for a role sundial refuses to serve a report to.
That is the `ci` role, which is not deployed. Says what is actually true of
`guest` instead: the request asks for counts, the role cannot load the private
vectors, and the gate refuses a payload that is not a score.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* test(ci): name what this section actually asserts
The heading said "the score-only `ci` role" and the docstring said the gate
runs as a role sundial refuses to serve a report to. Neither is true yet --
that role is not deployed. What the tests actually pin is that score mode is a
requirement rather than a preference: a full report is refused whoever asked
for it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* fix(ci): the sundial credential must not ride out on an error message
Whatever reaches result.note() is written to the results artifact and posted as
a pull request comment. An exception raised inside urllib can carry the URL that
produced it -- and for the token route that URL *is* the credential,
percent-encoded in the query string. A 401 from a stale key would have published
the key.
Every string built from an exception now goes through scrub() first, which
removes both the raw secret and its percent-encoded form. The message still says
which routes were tried and what the secret is supposed to be, so a real
misconfiguration is still diagnosable from the log alone.
Mutation-checked: with the replacement removed, both new tests fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* docs(ci): say which role the private-vector gate actually blocks
"serves that bundle to nobody else" read as though sundial withheld the private
vectors from everyone. It withholds them from `guest` specifically -- admin and
private still get them -- which is the whole reason CI authenticates as the
least privileged account rather than whichever one was to hand. Verified against
the deployed middleware: guests get a 200 with a no-op body, not a 404.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* docs(ci): the Cloudflare 403 is narrower than the comment claimed
It said any non-browser User-Agent is refused. In fact `POST /__auth/login`
worked for months with `User-Agent: camoufox-harness` -- the block tracks
document-shaped requests, which is what `/automated?key=` is. Browser headers
everywhere are still right, but as "cheaper than remembering which hop is
which", not as a fix for an outage that was never happening.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* fix(ci): verify the sundial role instead of assuming it
The CI log for the first real run says `sundial auth OK (automation key)` --
the stored credential is an automation key, not a form password. Which matters
more than it sounds: sundial's `/automated?key=` resolves to the **private**
role when handed the private key and to `guest` when handed the guest one, and
the two are indistinguishable by looking at them. `private` is served the
private vectors.
So the guarantee this gate has been documenting -- "CI runs as a role sundial
withholds the vectors from" -- was resting on whoever set the secret having
picked the right key. redact() does not help here: it governs what this
repository *publishes*, not what the browser is *given*, and the thing being
prevented is a public runner holding the vectors at all.
The gate now reads sundial's own /__auth/me and refuses to open the browser
unless the session is a role the vectors are withheld from. Not knowing the role
counts as not safe: an absent or unreadable endpoint fails closed, because the
alternative is loading them on the assumption that the credential was right.
Probed live: a bogus cookie yields no role and the check refuses, as it should.
The positive case is what the next run confirms -- and if that credential turns
out to be the private key, this goes red, which is the correct and useful
outcome.
Also corrects the Cloudflare comment, which I had just rewritten on a false
premise. The form-login route was never exercised in CI (the gate shipped
disabled), so "POST /__auth/login worked for months with camoufox-harness" was
unfounded. What is measured: /automated?key= answers 401 with a browser
User-Agent and 403 with urllib's default -- and since that is the route the real
credential uses, the browser headers were load-bearing, not cosmetic.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* fix(ci): record the sundial role in the evidence, not just in the log
`result.metrics["sundial_role"]` was set before the scan and then thrown away:
redact()'s output replaces result.metrics wholesale a few lines later. The check
itself was unaffected -- it ran, and it fails closed -- but the saved artifact
did not say which role had been confirmed, which leaves "the vectors were never
served to this session" unverifiable after the fact. That is most of the reason
to record it.
Caught by reading the artifact the live run actually published, not by the 105
tests, none of which exercised gate() end to end. There is now one that does,
and it fails when the re-assignment is removed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* test: delete the vendored Playwright fork, keep only what is ours
`tests/` held a fork of a ~v1.55-era playwright-python suite, run in CI as
`playwright_vendored` alongside upstream's own suite. Measuring the two against
the same binary says the fork was strictly the weaker of them:
* 73 of the 74 tests it skipped as "Not supported by Camoufox" PASS in
upstream's copy. Those skips predate main-world execution and were never
revisited, so the fork was asserting the browser is worse than it is.
* Of its passing tests, eight had no upstream counterpart. Six were genuinely
Camoufox-specific; the other two were tests upstream had since renamed.
* Everything else was upstream code, one generation stale, run twice.
So the fork goes, and the six tests that were doing real work become
`tests/camoufox/` -- three modules that `ci/suite.py` overlays into the fetched
upstream checkout, where they run against its conftest and its server at
whatever tag was resolved. Nothing frozen, so nothing to go stale.
Two of them take over from skiplist entries that had been pointing at the fork:
* the worker locale. Upstream expects a worker NOT to inherit the context
locale (playwright#38919); Camoufox sets it below that layer, so its workers
agree with the main thread. A page/worker disagreement is a free signal, so
the replacement asserts they match rather than hardcoding one string.
* the User-Agent. The bare binary advertises `Camoufox/<version>`, not
`Firefox/<version>`, and only the Python package rewrites it. The
replacement asserts a well-formed Gecko token and, more usefully, that the
wire and the DOM agree on it.
A skiplist entry that hands its job to another test now says so in a
`replaced-by:` field, and `ci/summarize.py` fails the run if that file does not
exist -- otherwise a rename quietly turns "covered elsewhere" into "not covered".
Also here, because deleting the fork exposed them:
* `tests/local-requirements.txt` was the only thing holding the suite below
pythonlib's `playwright = "<1.63"` ceiling, and it is gone. `ci/versions.py`
now applies that ceiling directly, so a future Firefox bump cannot silently
resolve to a client the shipped package refuses to install.
* the pytest header named `<plugin dir>/skiplist.yml` whatever it had actually
read. Since ci/suite.py copies the plugin into the checkout, that was a path
with no file at the end of it -- misleading precisely when someone is chasing
down a skip. It now reports the file it loaded.
* the summary line printed "Camoufox 152.0.4 ... against Playwright v1.61.0,
which targets Firefox 151.0", which reads like a misconfiguration. Playwright
trails Firefox and skips generations -- it pinned 151 then 153, never 152 --
so it now says which rule picked the tag and that the browser is Firefox 152.
`make tests` runs the one suite. Verified: the three overlaid modules pass
against 152.0.4-beta.31 (8/8), the overlay refuses to shadow an upstream module
and is idempotent across a reused checkout, and both new guards were
mutation-checked. ci/tests: 112 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* feat(ci): a local way to ask where the stealth failures are
"Which five sundial checks failed?" has no answer in CI, and the reason is worth
writing down rather than rediscovering: score mode's payload is buckets keyed
"<Category>|<class>" holding two integers each. It carries no check names and no
ids, so a failing check's identity is not something the gate discards -- it is
something sundial never sends. No artifact, log or sealed report can recover it.
What the payload does know is which category the failures are in, and the gate
was throwing that away too. `--explain` prints a per-category breakdown, and:
* it prints, never records -- `result` is untouched, so the artifact still
carries only `_PUBLISHABLE`;
* it is refused outright under GITHUB_ACTIONS, before anything is sent. A
category table is not a vector, but "Graphics 3/17" is the most useful
single fact an adversary could take from a public log, which is precisely
why redact() does not publish one;
* it says in its own output that names need --allow-full-report and a role
sundial serves full reports to, so a reader does not mistake the category
view for the whole answer;
* an unclassified category is flagged there too, on the same rule the gate
uses -- a new sundial section must not default to ignored.
Also corrects ci/README.md, which claimed "identities in the results file are
HMACs". They are not, and have not been: redact() ships no per-check rows at
all, deliberately, because a map of HMACs still says how many distinct checks
fail and lets a reader follow one across releases. The README was describing a
weaker guarantee than the code actually makes.
The CI refusal is mutation-checked. ci/tests: 116 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* fix(ci): 193 of the 202 tests the skiplist skipped actually pass
The skiplist was wrong, and wrong in the way that matters: it claimed the
browser could not pass tests it passes, and the stated reasons made that look
checked.
Where it came from: the first version of ci/skiplist.yml took all nine
`tests/async/*.disabled` modules from the vendored suite -- a 1:1 match, nothing
independently derived -- and gave each a plausible justification without running
any of them. That is the exact thing the file's own header calls "a regression
wearing a disguise", committed in the commit that wrote the header.
Ran every entry with the skiplist disabled, against 152.0.4-beta.31:
test_element_handle.py 59 skipped 0 fail
test_popup.py 26 skipped 0 fail
test_dispatch_event.py 12 skipped 0 fail
test_check.py 9 skipped 0 fail
test_launcher.py 9 skipped 0 fail
test_focus.py 6 skipped 0 fail
test_fill.py 3 skipped 0 fail
test_click.py 71 skipped 2 fail
client_certificates 5 skipped 5 fail
(two individual tests) 2 skipped 2 fail
Seven of nine modules failed nothing at all. 193 of 202 passed. The suite was
reporting 1339 passing while silently excluding 193 more that also pass -- and
nothing would have caught those 193 regressing.
Some reasons were not just over-broad but wrong. test_popup.py was skipped for
"Camoufox resolves the User-Agent from its fingerprint config, so an arbitrary
override does not and must not stick"; the two UA tests in it pass, because the
bare binary under plain Playwright does honour `user_agent=` -- the fingerprint
resolution is in the pythonlib wrapper, which this suite does not use. The
neighbouring test_request_headers_should_work entry gets that distinction right,
four entries earlier. test_dispatch_event.py was skipped for "Camoufox only
emits trusted events"; that is true and those tests never assert isTrusted.
So: 13 entries down to 5. test_click.py narrows from the module to the two tests
that fail -- Playwright's stable-position wait polls the bounding box then
dispatches instantly, and the humanized path spends real time travelling, so an
animating button is clicked mid-flight (offset 100, expected 300). The
client-certificate module stays whole: all five fail at the TLS layer, which is
support that genuinely is not compiled in. The `[chromium]` and `[webkit]`
patterns are deleted -- the runner pins `--browser firefox`, so they matched 0
tests and only made the list look more considered than it was.
And the guard, because a reason is an assertion about the browser and nothing
was checking it: ci/run_skiplist_audit.py runs every entry with the skiplist
disabled and FAILS THE BUILD if a skipped test passes. It is cheap exactly
because a correct skiplist is short -- 9 tests, 8 seconds -- so it runs on every
PR in tier 3a and is a required gate. Mutation-checked both directions: exit 1
naming the newly-passing tests when a stale entry is re-added, exit 0 now.
ci/tests: 118 passed, including that every shipped entry is actually auditable
(a `pattern` entry cannot be, and now fails that test rather than riding along
unverified).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* feat(ci): run the sync suite too, and say what is left out
The Playwright gate ran `tests/async/` and nothing else. Upstream v1.61 collects
2306 tests; that was 1584 of them. The other 722 -- `tests/sync/` (715),
`tests/common/` (4), `test_reference_count_async.py` (2), `test_installation.py`
(1) -- never ran, and unlike a skiplist entry there was nothing anywhere saying
so. No reason was recorded because there was no decision: the vendored fork in
tests/ carried `async/` and `async_imp/` and no sync suite, and this runner was
pointed at the same shape without checking what upstream had. Same inherited
shape as the stale skiplist, one level up.
TARGETS now names `tests/async/` and `tests/sync/`. The sync API is a greenlet
wrapper over the same Juggler traffic, so much of it duplicates async at the
protocol level; it is here because pythonlib ships a sync API users actually
drive, and the wrapper has its own timeout and reentrancy behaviour the async
tests cannot reach.
ISOLATED_TARGETS runs `tests/common/` and `test_reference_count_async.py` in a
second pytest process, on the first shard only. They cannot share a process with
the others: each calls sync_playwright()/async_playwright() inside the test body,
which cannot start while the session fixtures hold a loop. Together all six fail
with "Cannot run the event loop while another loop is running"; alone all six
pass. They are worth the extra invocation rather than dropping, because
ProtocolCallback objects accumulate when the browser never replies to a protocol
message -- and this fork patches Juggler heavily, so that leak can be ours.
EXCLUDED holds the one real exclusion with its reason: test_installation.py
pip-installs playwright to check packaging, which exercises Playwright's release
process and not this browser. A self-test requires every exclusion to carry a
reason.
And unclaimed() closes the level above the skiplist: if upstream adds a test path
that is in neither set, the run errors instead of quietly getting narrower. It
ignores assets/ and golden-*/ fixtures, and a self-test proves it catches a new
tests/integration/.
Verified against the CI-built binary: 2225 passed, 0 failed, 66 skipped, 2291
collected. ci/tests: 122 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* fix(ci): client certs are only unsupported where the BROWSER does the handshake
The skiplist audit failed its first CI run, on an entry written one commit
earlier. It was right to.
`tests/async/test_browsercontext_client_certificates.py` was skipped as a whole
module, reasoned as "client-certificate support is not built into the Camoufox
binary". On the GitHub runner two of its five tests pass:
test_should_throw_with_untrusted_client_certs PASSES
test_should_work_with_global_api_request_context PASSES
test_should_work_with_new_context fails
test_should_work_with_new_context_passing_as_content fails
test_should_work_with_new_persistent_context fails
The split is not noise. Playwright offers client certificates two ways:
playwright.request.new_context(client_certificates=...) the Node driver does
the TLS handshake itself -- works, and those are the two that pass.
browser.new_context(client_certificates=...) the BROWSER does the
handshake -- and this build has nothing to do it with.
So the reason was over-broad rather than wrong, and a module entry was exactly
the shape that hid the distinction. Now six per-test entries, async and sync,
naming the browser-side handshake specifically.
Worth recording how the bad entry got written, because the mechanism matters
more than the entry: locally all five fail, because this machine's Node/OpenSSL
rejects the fixture server outright ("wrong version number"). That looked like
uniform absence of support and it was not. A local run is a hypothesis; CI is
the authority for what fails. The audit is what turned that from an opinion into
a build failure, one run after the mistake.
The sync entries assume the same split from identical test names; the audit will
confirm or correct that on the next run rather than my asserting it from here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* fix(ci): async and sync cannot share a pytest process
Adding tests/sync/ to the same pytest run as tests/async/ produced 50 tests that
passed only on retry. 47 were fixture-setup errors in two fetch modules:
RuntimeError: Runner.run() cannot be called from a running event loop
Upstream's sync suite is a greenlet wrapper; its async suite runs under
pytest-asyncio. In one process, whichever runs second breaks the other's loop.
Measured, against the CI-built binary:
tests/async alone 1526 passed, 1 timing flake
the two fetch modules alone 102 passed
async fetch + one sync module 14 failed
sync module first, then async fetch 46 errors
So sharding was not hiding anything -- async alone is clean -- and the fault is
the mixing, not the size of the run.
What makes this worth a structural fix rather than a retry budget: the damage
lands in async FIXTURE SETUP, so it presents as "the fetch tests are flaky" --
a browser-shaped symptom for a harness-shaped cause -- and the retry pass then
makes it vanish. Left alone, CI goes green with 50 silent retries, and the
natural response to any that stuck would be a skiplist entry recording a browser
failure that does not exist. That is the failure this branch has spent its last
several commits removing.
GROUPS now names three sets, each run in its own process: tests/async/,
tests/sync/, and (tests/common/ + test_reference_count_async.py). The first two
shard; the third is six tests and does not, because splitting it hands some
shard an empty selection and pytest exits 5 for that. The pairing in the third
group is measured, not assumed -- those two run together cleanly (6 passed).
Retries stay inside their own group, for the reason the groups exist.
Verified: 2225 passed, 4 failed, 66 skipped, 2295 collected, and 1 retry-passer
instead of 50. All four failures are the two client-cert tests that go through
playwright.request.new_context(), which fail only on this machine -- its
Node/OpenSSL rejects the fixture server ("wrong version number"). They pass on
the runner, which is why they are not skiplisted; the audit is what will hold
that claim honest. Expect 2229/0 in CI.
ci/tests: 124 passed, three of them pinning this shape -- async and sync in
different groups, no target in two groups, and the small group unsharded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* fix(ci): refuse to test a Firefox the branch does not pin
Suite selection follows the browser version correctly: ci/versions.py resolves
Firefox 155 to Playwright v1.62.0 and 152 to v1.61.0, and the workflow threads
`browser_version` from workflow_call/dispatch through resolve into the Playwright
job, which fetches that tag. That half works.
The other half does not. Nothing else honours the input:
ci.run_build / make read upstream.sh
fetch-browser `python -m camoufox fetch`, whatever is current
ccache key uses the resolved version (cosmetic)
So `browser_version: 155.0` on a branch pinning 152.0.4 compiles 152 and judges
it against the suite chosen for 155. It passes, it means nothing, and no output
anywhere says the browser and the suite are describing different releases. For a
gate whose entire job is to make an upgrade provable, that is the worst
available outcome.
The legitimate flow never had this in it: an upgrade to a new Firefox is a
branch that edits upstream.sh -- that is what an upgrade is -- and resolution
then reads it by default, so the build, the suite and the input cannot disagree.
The mismatch only arises from a dispatch that asks for a version the branch does
not pin.
`--check-upstream` refuses that combination with a message saying what would
have happened and what to do instead (bump upstream.sh). The workflow passes it,
and a self-test asserts the workflow passes it, because a guard nothing invokes
is decoration. Mutation-checked.
Not fixed here, and deliberately: making the build honour an arbitrary version
would mean synthesising an upstream.sh -- version plus release tag plus
closedsrc_rev -- for a release that may not exist. Refusing the contradiction is
the honest amount of machinery for it.
ci/tests: 126 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* fix(ci): the fetched browser must be the generation the suite was chosen for
A driver-only pull request does not build. It downloads the current release,
which is the right browser to judge a driver change against -- it is what users
run. But the Playwright suite is chosen from upstream.sh, and those two agree
only until an upgrade window opens.
The sequence that breaks: the version agent bumps upstream.sh to Firefox 155 and
that merges. No build of 155 is published yet. The next driver-only PR resolves
its suite for 155, fetches the published 152, and tests the old browser against
the new suite. It passes, and says nothing.
Today they happen to agree exactly -- upstream.sh pins 152.0.4-beta.31 and the
latest release is v152.0.4-beta.31 -- which is timing, not a guarantee, and
precisely the kind of coincidence that hides this until the upgrade it is
supposed to protect.
So the fetch step now reports which build it installed and refuses a mismatch.
Beta drift inside a generation is fine and expected: beta.30 against beta.31
does not change which Playwright tag is right, and demanding an exact match
would fail every run between a bump and a release. A generation apart is not
fine, and that is what is checked.
Fails closed on input it cannot read, rather than passing by accident.
Mutation-checked. A self-test asserts the workflow actually invokes it.
ci/tests: 128 passed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* fix(ci): a driver-only pull request cannot satisfy a required build
The summary was told to require the `build` suite on every run:
required="pythonlib native_rules"
if [ "$browser_changed" != "skip" ]; then
required="$required build patch_guards ..."
fi
`browser_changed` is only ever `true` or `false` -- nothing emits `skip` -- so
the branch is always taken and `build` is always required. But the build job is
deliberately skipped whenever the browser was fetched rather than compiled, and
only that job writes a `build` result. summarize then reports
`build` is required but produced no result file. A suite that did not run
has not passed.
which is the correct rule applied to a suite that was never supposed to run,
and the gate goes red.
That blocked every pull request touching only pythonlib/, ci/, tests/ or the
documentation -- most of them, and precisely the cheap path ci/README.md
advertises as "driver-only pull requests never build". It was never seen
because this branch edits the Makefile and additions/, so its own runs always
took the build path.
The browser suites stay required either way: fetched or built, the browser is
there and they run against it. `build` is the only one that follows.
`test_build_is_required_only_when_the_browser_was_built` extracts the workflow's
own `required=` assembly and runs it under bash for both values, rather than
pattern-matching the shell -- the bug was a comparison that read as deliberate,
and only running it says what it does. Mutation-checked: restoring `!= "skip"`
fails it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* fix(ci): sundial being down is a skip, not a verdict on the browser
The stealth check drives a service on another host. Any failure reaching it --
DNS, a refused connection, the edge answering 502 for a minute -- came back as
ERROR, failed the job, and failed the merge gate. So an outage over there
blocked every pull request in this repository, including ones that had nothing
to do with stealth, and the only fix available to a contributor was to wait.
That is not a statement about the browser. When sundial cannot be reached the
browser was not measured at all, so neither a pass nor a failure is true. It is
now recorded as SKIP with the reason attached, the job exits 0, and
`ci/summarize.py --allow-skip sundial` tolerates it: shown on the summary table
with its own icon and its reason, and not a merge block.
The line is drawn at whether sundial answered:
down no HTTP reply at all (URLError, timeout, connection reset), or a
5xx, or a 429 -- the origin is broken or is refusing everyone.
answered everything else. 401 is a bad credential, 403 is the edge refusing
a non-browser request, and both are this repository's problem to
fix. Skipping past those would turn a misconfigured stealth gate
into a permanently green one.
And an answer stays an answer further in: a role sundial would serve the
private vectors to, a full report where a score was requested, a pass rate under
the floor -- all still fail, as before.
Both auth routes are tried, and the key route 401s whenever the stored secret is
a password rather than an automation key. One real reply is enough to know
sundial is up, so `authenticate()` reports an outage only when neither route got
an answer.
The skip note goes through `scrub()` like every other published string: the
token route puts the credential in the URL, and a URLError carries the URL that
raised it.
`--allow-skip` is per suite and nothing else is on the list. Every other suite
runs on the runner; none of them has this excuse.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* feat(ci): run the suite isolated first, and count what needs the main world
The conformance suite forced `disableWorldIsolation` for every run. That made
upstream's tests pass -- they assert upstream semantics, reading globals their
own page scripts defined -- but it measured a mode nobody ships, and it produced
no number at all for what isolation costs. Isolation is the reason this fork
exists; running 2229 tests against it turned off says less than it looks like.
Each group is now run up to three times:
1. isolated -- the configuration users get. A test reading a page-defined
global fails here by design.
2. the same failures again, still isolated. A pass is a flake, and a flake
must not be counted as a world difference.
3. what is still failing, with isolation off.
A test that passes in 3 is a **main-world fallback**: it counts as a pass -- the
browser does honour the contract -- and its identity is recorded in
`metrics.main_world_fallbacks`, with the count on the summary table. That count
is the isolated-world conformance gap. It is invisible in the pass/fail totals
by construction, which is exactly why it has to be printed: a jump in it means
the isolation boundary moved, and nothing else in this pipeline would say so.
A test failing in *both* worlds is a plain failure, as before.
`CI_WORLD` selects the world and defaults to isolated, so a plain
`pytest -p pw_camoufox_plugin` by hand measures the browser as it ships.
`_apply_world()` *clears* a stale `disableWorldIsolation` as well as setting it:
the passes are separate processes inheriting one job environment, and a
leftover flag would make the isolated pass quietly measure the main world --
which would silently zero the very number this is for.
Two things this depends on, fixed here:
Each group gets its own pytest cache. All three run in one checkout, and
pytest only drops a `lastfailed` entry when that test is collected again and
passes -- so with the shared cache the async group's rerun was selecting from
a set the sync group had also written into. Depending on which groups had
failed that meant re-running the whole group or selecting nothing at all.
Per-group, `--last-failed` means what it says, and passes 2 and 3 can use it
to name exactly the right tests.
`ci/run_skiplist_audit.py` pins the MAIN world. The suite now counts a test
needing the main world as a fallback rather than a failure, so a skiplist
entry has to claim the test cannot pass in either world -- auditing under
isolation would let an entry justify itself with a failure the suite would
never have counted, which is the same class of untrue-but-plausible reason
the audit exists to catch.
Shard merging sums the fallback counts and unions the identity lists; taking
the first shard's, as the generic metric merge did, would report a sixth of the
number. Mutation-checked, along with the stale-flag clear.
The first CI run on this is what establishes the real fallback count. The
plugin's own note put it at roughly 37; that was measured a while ago and is not
a promise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* docs: describe the pipeline that exists, not the one that is coming
CONTRIBUTING.md and the pull request template still described the process this
branch replaced: run both suites by hand, screenshot the output, paste it in.
CLAUDE.md was already pointing at CONTRIBUTING.md for "the full pipeline" that
CONTRIBUTING.md did not mention. Since the premise of this work is that the rule
in CONTRIBUTING.md was never enforced, leaving it unchanged left the rule
describing the wrong thing.
Both now say what CI does and what the one required check is. The template no
longer asks for a screenshot: nothing checked that the browser in one was built
from the branch under review, which is the whole reason the pipeline exists, and
CI leaves its own report as a comment. The local commands stay, because running
build-tester by hand while working on a spoofing patch is still the fastest way
to find out whether it did what you meant.
Several comments described this repository as containing an auto-update harness
that is not in it: `ci/results.py` credited `verify.py` and "the repair agent"
with computing the verdict that `ci/summarize.py` computes, and `ci/sundial.yml`
and `ci/build-tester.yml` sited themselves relative to a `harness/policy.yml`
nobody can open. Each now names what actually decides here and marks the harness
as the out-of-repository caller it is. `run_sundial.py waive` says outright that
the file it prints a stanza for is not in this repository, which is worth
knowing before going to look for it.
Also: PEP 8 blank lines around `resolve_verstr()`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* perf(ci): drop the same-world retry -- 7m50s a shard, nothing recovered
The first real run of the isolated-first suite took 21m13s a shard against
4m34s before. Most of that was a pass that could not have worked. Shard 4, both
sharded groups:
isolated (full) 335s + 331s 35 and 11 failures
isolated retry 205s + 265s 0 recovered
main world 19s + 12s 46 recovered
The retry was there so that a flake could not be mistaken for a world
difference. It cannot do that job:
* These failures are deterministic. A test reading a global its own page
script defined does not intermittently stop seeing it.
* Failing that way is slow. The read returns undefined rather than throwing,
so the test sits on a Playwright timeout -- which is why re-running 46
known failures cost nearly eight minutes while proving them in the other
world cost thirty-one seconds.
* Upstream already reruns. The `105 rerun` on that first line is every one of
those 35 failures having been retried three times by the suite's own
pytest-rerunfailures before the run reported them. A flake does not survive
that, so there was nothing left for a fourth and fifth attempt to find.
So: isolated, then the main world, then -- only for what failed in BOTH -- one
retry. That last set is normally empty, so the retry is free on a healthy run
and still answers the one open question on an unhealthy one: a test no world
satisfies is either broken or flaky. It runs in the main world, where a pass
means "not reproducible" rather than "needed isolation off", which is already
known by then.
A flake surviving upstream's three reruns and then passing in the main world
would now be counted as a fallback rather than as a flake. That is the trade,
and it is worth it: the count is reported, not gated, and no verdict moves.
Also adds the guard that was missing between the phases. `--last-failed` with
nothing previously failed does not select nothing -- pytest declines to filter,
and runs the whole group. Unguarded, a group that passed cleanly under
isolation would have been re-run end to end in the main world, silently
replacing the result it was meant to refine. Both rerun passes are now behind a
non-empty check, and a self-test asserts it of every `--last-failed` in the
loop. Mutation-checked, as is the single-isolated-pass rule -- "retry it in the
same world first, just to be safe" reads as obviously correct and costs eight
minutes a shard.
Expected shard time is now ~11 min: the isolated pass, which is the
measurement, plus half a minute to resolve it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* perf(ci): bound the isolated pass, which is where the timeouts come from
Some isolated failures do not fail. They hang for the full 180s per-test
timeout and are then rerun three more times, so one test can cost twelve
minutes. The main-world baseline had zero timeouts across all six shards, so
this arrived with isolation.
Reproduced against a build, and the cause is not the harness:
page script calls window.exposedFn() isolated -> HANG main -> resolves
evaluate() calls window.exposedFn() isolated -> resolves main -> resolves
`expose_function` installs its binding on the isolated world's global. Page
script calling `window.fn()` looks at the page's own window, does not find it,
and the call never reaches Python -- so a test awaiting the future that call was
meant to resolve waits forever, because that await has no Playwright timeout
behind it. evaluate() works because it runs in the same world as the binding.
That is isolation doing exactly what it is for. A page that can reach an
automation binding can detect it, which is the reason this fork exists. These
tests assert a behaviour Camoufox deliberately does not have, and they cannot be
fixed -- only recognised, which pass 2 does in about half a second each.
What can be fixed is the price of recognising them. Pass 1 is a classifier: its
only question is whether a test passes as Camoufox ships, and a test that hangs
has already answered it. So pass 1, and only pass 1, is bounded:
per-test timeout 90s. The slowest test in the entire main-world baseline was
30.4s of 2295; two exceeded 30s and none exceeded 45s. A Playwright action
times out at 30s. Three times the slowest honest thing that happens, and half
the previous bound.
upstream's reruns off, via CI="". tests/conftest.py sets `reruns = 3`
whenever $CI is set, and that is the only thing it reads $CI for. It is
insurance that almost never pays out -- 2 reruns across all 2295 baseline
tests -- and under isolation it turned every deterministic world difference
into four attempts: 138 reruns in a single shard's isolated pass, recovering
nothing. Note that `--reruns 0` as an argument would not work; conftest
overwrites config.option.reruns in pytest_configure, so the environment is the
only lever that holds.
A hang now costs one 90s wait instead of up to 720s. A flake missed by not
rerunning is not lost: it fails pass 1, passes pass 2, and is counted as a
fallback -- noise in a reported metric, not a change in any verdict.
Passes 2 and 3 keep upstream's conditions untouched. They are the ones deciding
what an answer means, and they run against a handful of tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* perf(ci): stop rebuilding a browser that has not changed
Every push to this branch recompiled Firefox, 24 minutes at a time, including
the three in a row that changed nothing but ci/ and documentation.
`browser_changed` is computed against the pull request's BASE, not against the
push, so a branch that touched patches/ or additions/ even once keeps rebuilding
forever. That part is correct and should stay: the published release does not
contain this branch's browser changes, so testing against it would test a
different browser than the one under review. Using the release is not the
alternative.
The alternative is not building the same thing twice. The build job now asks a
narrower question first -- not "does this branch change the browser" but "has
the browser changed since the last one we built" -- and answers it with a cache
keyed on a hash of every input that can alter the binary: the same path list
browser_changed greps for, plus this workflow, which pins the toolchain the
build runs on. On a hit the 634 MB dist is restored and every build step is
skipped, which is the difference between 24 minutes and about one.
Two things that would otherwise have made this quietly wrong:
A hit still has to report a `build` result. It is a required suite whenever
the browser was built rather than fetched, and a required suite that produced
no result is -- correctly -- a failure. Same trap as requiring `build` on a
driver-only pull request, reached from the other side. The hit path writes one
recording that the browser was restored and under which key, so "this run
compiled nothing" is a fact in the evidence rather than an absence in it.
No restore-keys. Everywhere else in this workflow a prefix match is right; a
partly warm ccache is still warm. Here it would hand the test jobs a browser
built from different sources while every suite reported on it looking
perfectly healthy.
The self-tests hold the two lists together: if a path is ever added to the
browser_changed grep without being added to the cache key, a change there would
neither force a build nor invalidate the cache, and the run would silently test
a browser that predates it. Mutation-checked, along with the guards on each
expensive step and the absence of restore-keys.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* perf(ci): a Juggler JavaScript change does not need libxul relinked
Juggler is mostly JavaScript, and JavaScript does not need a compiler. Measured
on a real build, ccache reported a 98.63% hit rate (4167/4225), so almost none
of those 24 minutes was compiling C++ -- it was Rust, linking libxul, and
packaging, none of which a .js file affects.
And there is no omni.ja to rebuild either. CI archives the UNPACKAGED dist/bin;
omni.ja only exists in dist/camoufox/, which `mach package` produces and nothing
here uses. In dist/bin, Juggler is loose files under chrome/juggler/ -- symlinks
into the source tree, dereferenced into the artifact by `tar -ch`. Delivering
new JavaScript is a copy.
So the build cache is now keyed on a hash of the COMPILED inputs only. If that
hash matches, the compiled half is identical by construction and this branch's
resources are laid over the restored browser. The hash is the classification:
there is no "did only JavaScript change?" diff, because a diff answers the wrong
question -- it compares against the pull request's base, while what matters is
whether the cached browser has the same native sources. Same hash, same binary,
whatever the diff says.
Two traps, both closed and both mutation-tested, because either one silently
serves a browser that is not the one under review while every suite reports
green:
additions/juggler/ is not all JavaScript. It also holds the screencast encoder
and the remote-debugging pipe -- 5 .cpp, 5 .h, 2 .idl, 3 components.conf, 4
moz.build -- compiled into libxul. Only the files jar.mn lists are treated as
resources; everything else, including any extension nobody has considered yet,
is native and forces a build. jar.mn is itself native, so a resource removed
from it cannot leave a stale copy behind.
The mapping is per-file, not a prefix. jar.mn maps TargetRegistry.js to
content/TargetRegistry.js (a level added), content/FrameTree.js to
content/content/FrameTree.js (preserved), and content/JugglerFrameChild.sys.mjs
to content/JugglerFrameChild.sys.mjs (dropped). Two files in one source
directory land at different depths. A prefix rule writes one of them to the
wrong path and leaves the old copy in place.
Verified against a real build rather than reasoned about: all 22 jar.mn entries
resolve to files that exist in dist/bin, and overlaying this branch onto a dist
built before it turns FrameTree.js from 0 occurrences of nukeSandbox to 1, with
all 22 resources byte-identical to the branch afterwards.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* docs(ci): name the mechanism that actually hangs, not the one I found first
The comment blamed expose_function. That mechanism is real -- verified against a
build -- but no upstream test has that shape, so it is not what hangs in CI.
Running the full async suite isolated against a build names them: four tests,
always the same four, all in tests/async/test_route_web_socket.py. 225 of 1509
fail under isolation; four of those hang.
The general shape, which is worth stating because it will recur: a Playwright
feature implemented by installing something on the page's global lands in the
isolated world instead, so anything the PAGE originates never reaches the
automation. route_web_socket replaces window.WebSocket from an init script;
isolated, that replacement is in the sandbox and a socket the page's own script
opens is never intercepted. expose_function puts its binding on the sandbox
global, so page script calling window.fn() finds nothing -- called from
evaluate() it works, which is why it does not hang.
They hang rather than fail because the waits involved have no Playwright timeout
behind them: a Twisted future from the test server, an asyncio future a binding
was meant to resolve. Everything else isolation breaks fails at Playwright's 30s.
One thing to flag beyond the test suite: the route_web_socket half is not a test
artifact. Measured with a page whose own script opens a socket -- what a real
site does -- the handler fires in the main world and never fires isolated. A
user calling page.route_web_socket() against a real site gets no interception
and no error. It is not fixable here, because the feature works by replacing a
page global and that is exactly what an isolated world exists to prevent a page
from seeing, but it deserves an issue of its own rather than a comment in a CI
runner.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K1UY3f8gm2jA1J23C3ew9s
* fix(ci): the isolated pass cannot bound a hang, so stop asking it to
Four shards ran for two hours each and were killed by timeout-minutes, three
runs in a row. The cause is not the duration of anything.
Measured on run 34799668707, with ISOLATED_TIME…1 parent 571e416 commit 52d6746
268 files changed
Lines changed: 12501 additions & 28289 deletions
File tree
- .github
- actions/prepare-browser
- workflows
- additions/juggler
- content
- build-tester
- scripts
- src/lib/checks
- ci
- tests
- native-tests
- pythonlib
- camoufox
- tests
- tests
- assets
- client-certificates
- client
- self-signed
- trusted
- server
- digits
- es6
- frames
- input
- popup
- react
- serviceworkers
- empty
- fetchdummy
- fetch
- simple-extension
- worker
- async_imp
- async
- camoufox
- golden-firefox
- patches
- assets
- testserver
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
18 | | - | |
19 | | - | |
20 | | - | |
21 | | - | |
22 | | - | |
23 | | - | |
24 | | - | |
25 | | - | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
30 | 29 | | |
31 | 30 | | |
32 | 31 | | |
33 | 32 | | |
34 | 33 | | |
35 | 34 | | |
36 | | - | |
37 | | - | |
| 35 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
This file was deleted.
0 commit comments