Watch a pull request, get a verdict on every push.
You watch a pull request once. A planning run reads the diff and assembles a suite: new smoke and regression tests for the change plus the existing catalog tests it touches. Every push re-runs that suite against an environment built from the exact head commit, and the result lands where reviewers already look: one label and one comment that updates in place.
QA Web Agent · run for a3f9c1e
5 passed · 1 failed · 0 blocked · 6 tests · 14m 02s
| Test | Result | Time |
|---|---|---|
| pr197-smoke-public-site-home-and-navigation | passed | 1m 48s |
| pr197-regression-public-site-canonical-and-seo-surface | failed | 1m 41s |
| pr197-regression-dashboard-handoff-callbacks-and-legacy-links | passed | 1m 13s |
| pr109-smoke-dashboard-load-and-login | passed | 0m 51s |
| pr173-smoke-mobile-sidebar-drawer | passed | 2m 07s |
| pr184-regression-navigation-and-invalid-deep-route | passed | 1m 36s |
Failed: pr197-regression-public-site-canonical-and-seo-surface
Step 6: navigating to/appreturned ERR_CONNECTION_REFUSED;/app/loads. The redirect's Location header carried an absolutehttp://127.0.0.1/app/without the port. Confirmed on retry.
2 screenshots · session video · transcript
AI-generated content may be incorrect.
This is a real comment from this site's own pull request.
The failed test above found a real bug: nginx was building an absolute redirect that lost the port behind the ingress. It was fixed in the next commit and the same comment updated to six passes. The comment carries a marker block the changelog later reads, so the release note for this change can cite the PR and pull a screenshot from a passing test.
Failures expand; passes do not. A project can set the comment to failures-only, or to a single aggregate comment for the life of the PR.
One label at a time.
The tracked state and the label on GitHub move together. Branch protection that requires QA-WA-Passing turns the label into a gate today.
| State | Label | Meaning |
|---|---|---|
| planning | QA-WA-Testing | a planning run is reading the diff and choosing the suite |
| testing | QA-WA-Testing | the suite is running against the head commit |
| passing | QA-WA-Passing | every planned test passed on the latest push |
| failing | QA-WA-Failing | at least one planned test failed; the comment names it |
| needs-attention | QA-WA-NeedsAttention | a run was blocked or inconclusive; a person should look |
| merged | none | the PR merged; labels are stripped, the comment stays |
The plan is a real decision
The planner reads the diff, the catalog and your project's plan instructions. It can borrow existing tests, author new ones, and skip tests that were already broken before the PR existed (those show under Suite health as muted coverage, with a chip to force one back in). A person can re-plan or retest at any time.
Pinned to the head
Each push runs against an environment cloned at the head SHA, so a squash-merged or force-pushed branch still reproduces. Pushes to an already-watched PR re-test automatically; opening a PR only starts a plan if the project has auto-watch on.
Not everything is browser-testable
PRs and issues are classified as UI-surface or not, on request or automatically. A backend-only change gets QA-WA-Non-UI and an explanatory comment instead of a wasted run; a person can watch it anyway.
Affected pages, optionally
The plan can add an exploration of the pages the diff touched. Those results appear on the PR's panel and on the Explore screen, and they never decide the PR's verdict.
Where it stops today
- The verdict is a label and a comment, not a GitHub check run. It can gate a merge only through a required-label rule.
- Detection is polling-based (a sweep every few minutes), not webhook-driven; a push shows up as testing within minutes, not seconds.
- A PR whose branch lives only on the platform's internal git host (a graduated agent change before it is pushed) cannot boot a suite until the real branch exists.
- Retrospectively watching PRs merged before the project existed works, but their environments boot from the merge commit, not from a branch you can push to.
Questions people ask
Can a verdict block a merge?
Today the verdict is a label plus an updating comment, and branch protection can require the QA-WA-Passing label. A native check run is on the roadmap.