ci: gate web browser snapshots
This commit is contained in:
imccyu
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md
|
||||
2026-07-24-web-gui-browser-e2e-lane.md: ce59dcce270d548c91e3719eee8e9c83aea0c154
|
||||
2026-07-24-web-gui-browser-e2e-lane.zh.md: bad3dd15ed7b98cc17340666a6c1094d0de057b1
|
||||
2026-07-24-web-gui-browser-e2e-lane.md: 74522e070e9a26e61db02ea4f71c2b205f222037
|
||||
2026-07-24-web-gui-browser-e2e-lane.zh.md: 4ab1e70b3fedb1eae88be457aa0d533c1d193bb1
|
||||
|
||||
@@ -46,7 +46,7 @@ The lane covers three behavior families. Live-turn scenarios pin ordinary tool e
|
||||
|
||||
### CI stance
|
||||
|
||||
The lane ships gate-exempt inside `pnpm run test:web`, exactly as that config's header records. Adding chromium to CI would reverse the "no browser infrastructure in CI" premise in the [GUI testing note](../process/2026-07-20-gui-testing-system.md) and therefore requires its own Agent Note cross-linked from there, staged as a non-required job first with measured promotion criteria (consecutive green runs, wall time, zero-retry flake budget, runner browser-cache strategy). `TODO(ci-browser)` marks the seam. Scenarios are POSIX-oriented (the lane is not in the Windows matrix).
|
||||
The lane is a required compare-only gate for Linux pull requests under the [browser snapshot CI decision](2026-07-30-web-browser-snapshot-ci-gate.md). The static job publishes `apps/web/dist` with the package build artifacts; the consumer job installs the lockfile-selected Chromium, restores its OS-and-lockfile-keyed cache, and runs the lane with `DSH_SNAPSHOT=replay`. CI never records or refreshes goldens. The default-branch Linux serial jobs run the same comparison and seed the hosted browser cache. Scenarios remain POSIX-oriented and stay outside the Windows and macOS matrices.
|
||||
|
||||
## Prior art
|
||||
|
||||
@@ -76,12 +76,11 @@ Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot
|
||||
|
||||
## Testing
|
||||
|
||||
`pnpm run test:web` runs the lane keylessly. `DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/<spec>` records a prompting scenario against the live model, and `DSH_SNAPSHOT=refresh` rewrites aria goldens keylessly. `dsh-llm-replay` unit coverage pins pacing, cancellation, consumption diagnostics, sidecar validation, indexed replacement, and the single append position.
|
||||
`pnpm run test:web` builds and runs the lane keylessly; `test:web:built` runs it against existing build artifacts. `DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/<spec>` records a prompting scenario against the live model, and `DSH_SNAPSHOT=refresh pnpm run test:web` rewrites aria goldens keylessly. CI explicitly selects replay mode. `dsh-llm-replay` unit coverage pins pacing, cancellation, consumption diagnostics, sidecar validation, indexed replacement, and the single append position.
|
||||
|
||||
## Deferred
|
||||
|
||||
- **Web header-class pin**: web fixtures tokenize `{{system}}`/`{{tools}}` everywhere and no scenario pins the web composition's prompt/tool schemas (`TODO(web-header-pin)` — the scaffold `recordFixture` JSDoc marks it). Following the TUI scrub-everywhere precedent; revisit when the web assembly's header diverges from the repl composition it mirrors.
|
||||
- **CI browser provisioning**: reversal of the no-browser-in-CI ruling, staged criteria above (`TODO(ci-browser)`).
|
||||
- **Follow-up-prompt-after-resume scenario**: the history/live stitch path over the real wire; add as its own scenario when that code changes or regresses.
|
||||
- **Web error surface**: the client consumes no `agent/error` frames and a pre-chunk failure freezes no partial, so a non-retryable provider failure renders no error copy — the user sees the send simply stop. The AUTH scenario pins the current contract (no crash, composer recovers, turn logged `error`) and `FIXME(web-error-surface)` marks where visible error text gets asserted once the UI grows an error rendering.
|
||||
- **Composer steering gesture**: the input locks while running (stop-or-wait), so the steering scenario steers over the wire from the page; `TODO(web-steer-composer)` upgrades the drive step to a real composer gesture when the product grows one.
|
||||
@@ -89,4 +88,4 @@ Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot
|
||||
|
||||
## Consequences
|
||||
|
||||
The web surface gains its record-once/replay-forever tier: the real chromium → SSE → apiproxy → loop → tools → persistence chain runs keylessly in ~10-30s, deterministic across repeat runs, with fixtures owned and re-recordable by the lane itself. Costs accepted: every intentional conversation-UI change ends with a keyless `DSH_SNAPSHOT=refresh` (golden churn is reviewed diff, anchors keep semantic green); the aria format is Playwright-owned — the one committed snapshot format the repo does not control — so playwright version bumps must be deliberate bump-and-refresh commits (the dependency floats `^1.49.0` in `apps/web/package.json`; pin exactly if churn bites); replay's first-call-order binding constrains scenarios to one prompting session each, with the consumption assertion as the tripwire; `compact-basic` shares the session's replay cursor and stays inert only under the published 128k catalog window; and the lane guards regressions only where it runs (locally, `test:web`) until the CI reversal is separately decided.
|
||||
The web surface gains its record-once/replay-forever tier: the real chromium → SSE → apiproxy → loop → tools → persistence chain runs keylessly in ~10-30s, deterministic across repeat runs, with fixtures owned and re-recordable by the lane itself. Costs accepted: every intentional conversation-UI change ends with a keyless `DSH_SNAPSHOT=refresh` (golden churn is reviewed diff, anchors keep semantic green); the aria format is Playwright-owned — the one committed snapshot format the repo does not control — so playwright version bumps must be deliberate bump-and-refresh commits (the dependency floats `^1.49.0` in `apps/web/package.json`; pin exactly if churn bites); replay's first-call-order binding constrains scenarios to one prompting session each, with the consumption assertion as the tripwire; `compact-basic` shares the session's replay cursor and stays inert only under the published 128k catalog window; and required Linux CI pays for Chromium provisioning and the serial browser run so the PR that changes the assembled UI owns its expected-output diff.
|
||||
|
||||
@@ -46,7 +46,7 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu
|
||||
|
||||
### CI 立场
|
||||
|
||||
车道随 `pnpm run test:web` 交付、豁免门禁，与该配置头部注释所记一致。往 CI 加 chromium 会推翻 [GUI 测试笔记](../process/2026-07-20-gui-testing-system.md)中「CI 无浏览器基础设施」的前提，因此需要自己的 Agent Note 并从那里交叉链接，分阶段推进：先作为非必需任务，再以量化标准晋升（连续绿色运行次数、耗时、零重试的抖动预算、runner 浏览器缓存策略）。`TODO(ci-browser)` 标记该接缝。场景目前面向 POSIX（车道不在 Windows 矩阵中）。
|
||||
根据[浏览器快照 CI 决策](2026-07-30-web-browser-snapshot-ci-gate.md)，该车道是 Linux 拉取请求必需的只比较门禁。static 任务会把 `apps/web/dist` 与包构建产物一同发布；消费方任务安装锁文件选定的 Chromium，恢复以操作系统和锁文件为键的缓存，并用 `DSH_SNAPSHOT=replay` 运行该车道。CI 从不录制或刷新预期输出。默认分支上的 Linux 串行任务执行同样的比较，并填充托管浏览器缓存。场景仍面向 POSIX，并继续置于 Windows 和 macOS 矩阵之外。
|
||||
|
||||
## 业界先例
|
||||
|
||||
@@ -76,12 +76,11 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu
|
||||
|
||||
## Testing
|
||||
|
||||
`pnpm run test:web` 无密钥运行该车道。`DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/<spec>` 对真实模型录制一个发起提示的场景，`DSH_SNAPSHOT=refresh` 则无密钥重写 aria 预期输出。`dsh-llm-replay` 单元覆盖率钉住节奏控制、取消、消费诊断、sidecar 校验、按索引替换与唯一的追加位置。
|
||||
`pnpm run test:web` 构建并无密钥运行该车道；`test:web:built` 基于现有构建产物运行。`DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/<spec>` 对真实模型录制一个发起提示的场景，`DSH_SNAPSHOT=refresh pnpm run test:web` 则无密钥重写 aria 预期输出。CI 显式选择回放模式。`dsh-llm-replay` 单元覆盖率钉住节奏控制、取消、消费诊断、sidecar 校验、按索引替换与唯一的追加位置。
|
||||
|
||||
## 暂缓
|
||||
|
||||
- **Web 头类别钉住**：web fixture 处处 token 化 `{{system}}`/`{{tools}}`，没有场景钉住 web 组合的提示词/工具 schema（`TODO(web-header-pin)`——scaffold 的 `recordFixture` JSDoc 有标记）。沿用 TUI 处处脱敏先例；当 web 组装的请求头与其镜像的 repl 组合进一步分叉时重审。
|
||||
- **CI 浏览器供给**：推翻 CI 无浏览器裁定，分阶段标准见上（`TODO(ci-browser)`）。
|
||||
- **恢复后追问场景**：真实 wire 上的历史/实时缝合路径；当该代码变更或回归时作为独立场景补充。
|
||||
- **Web 错误表面**：客户端不消费任何 `agent/error` 帧，分片前的失败也没有可冻结的部分输出，因此不可重试的提供方失败不渲染任何错误文案——用户看到的只是发送就此停住。AUTH 场景钉住当前契约（不崩溃、输入框恢复可用、轮次记录为 `error`），`FIXME(web-error-surface)` 标记了待 UI 长出错误渲染后断言可见错误文本的位置。
|
||||
- **输入框 steering 手势**：输入在运行期间锁定（只能停止或等待），因此 steering 场景从页面走 wire 做 steer；`TODO(web-steer-composer)` 待产品长出真实的输入框手势后，把驱动步骤升级为该手势。
|
||||
@@ -89,4 +88,4 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu
|
||||
|
||||
## 后果
|
||||
|
||||
Web 表面获得了录制一次/永久回放的层级：真实 chromium → SSE → apiproxy → 循环 → 工具 → 持久化的链路以约 10-30 秒无密钥运行，重复运行结果确定，fixture 由车道自身持有并可重录。接受的成本：每次有意的会话 UI 变更都以一次无密钥 `DSH_SNAPSHOT=refresh` 收尾（预期输出变动是受评审的 diff，锚断言保住语义绿色）；aria 格式归 Playwright 所有——仓库唯一不受自己控制的提交快照格式——因此 playwright 版本升级必须是刻意的升级加刷新提交（依赖在 `apps/web/package.json` 中浮动为 `^1.49.0`；若变动伤人则改为精确锁定）；回放的首次调用顺序绑定把每个场景限制为至多一个发起提示的会话，消费断言是绊线；`compact-basic` 与会话共享回放游标，仅在发布的 128k 目录窗口下保持闲置；在 CI 反转被单独决策之前，车道只在其运行之处（本地，`test:web`）把守回归。
|
||||
Web 表面获得了录制一次/永久回放的层级：真实 chromium → SSE → apiproxy → 循环 → 工具 → 持久化的链路以约 10-30 秒无密钥运行，重复运行结果确定，fixture 由车道自身持有并可重录。接受的成本：每次有意的会话 UI 变更都以一次无密钥 `DSH_SNAPSHOT=refresh` 收尾（预期输出变动是受评审的 diff，锚断言保住语义绿色）；aria 格式归 Playwright 所有——仓库唯一不受自己控制的提交快照格式——因此 playwright 版本升级必须是刻意的升级加刷新提交（依赖在 `apps/web/package.json` 中浮动为 `^1.49.0`；若变动伤人则改为精确锁定）；回放的首次调用顺序绑定把每个场景限制为至多一个发起提示的会话，消费断言是绊线；`compact-basic` 与会话共享回放游标，仅在发布的 128k 目录窗口下保持闲置；必需的 Linux CI 承担 Chromium 供给与串行浏览器运行的成本，使改动组装后 UI 的 PR（Pull Request）持有相应的预期输出 diff。
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-07-30-web-browser-snapshot-ci-gate.md
|
||||
2026-07-30-web-browser-snapshot-ci-gate.md: 71d6bcdbfad11032ea6f8e7d570780f780cfe071
|
||||
2026-07-30-web-browser-snapshot-ci-gate.zh.md: bb28b42c5d2684d29e7e18646d9756b40557eb11
|
||||
@@ -0,0 +1,33 @@
|
||||
# Agent Note: Required CI gate for web browser expected outputs
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-07-30-web-browser-snapshot-ci-gate.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The [keyless web browser e2e lane](2026-07-24-web-gui-browser-e2e-lane.md) runs only under the local `pnpm run test:web` command, and PR CI does not compare `apps/web/tests/snapshots/**/*.expected.md`. A PR that changes user-visible web output can therefore remain green when its expected outputs are not refreshed; when any later branch explicitly runs `DSH_SNAPSHOT=refresh`, it backfills the earlier change and produces a diff unrelated to that branch. Ordinary local runs already default to read-only replay, so the gap is mandatory enforcement at the PR level, not a ban on writes in refresh mode.
|
||||
|
||||
## Decision
|
||||
|
||||
For Linux PRs, the `node 24 / snapshots and artifacts` job must run the full web browser replay/compare suite. `scripts/run-gates.ts` registers `test:web:built` as a `ci-consumers` gate and explicitly injects `DSH_SNAPSHOT=replay`; CI never runs in `record` or `refresh` mode, so when the committed goldens disagree with the currently assembled application, the tests fail directly instead of silently rewriting them on the runner and then passing.
|
||||
|
||||
The static CI job already builds all publishable artifacts; it puts `apps/web/dist` and the package `lib/` directories in the built-tree artifact, which the consumer job reuses without rebuilding the entire repository. The consumer job installs Chromium and its system dependencies at the Playwright version in the lockfile, and caches the browser by operating system and the `pnpm-lock.yaml` hash. The default branch's serial Linux job runs the same compare command and populates the default-branch cache, which subsequent PRs can restore directly.
|
||||
|
||||
Local `pnpm run test:web` continues to build first and then run the full browser suite; `test:web:built` is the entry point for existing build artifacts. Developers explicitly run `DSH_SNAPSHOT=refresh pnpm run test:web` only after confirming that user-visible output changed intentionally, review every expected-output diff, and then verify again in replay mode that no files are written.
|
||||
|
||||
The gate remains Linux-only: these scenarios target POSIX, and the Windows and macOS serial reference jobs do not run them again. A PR's `all checks passed` verdict already depends on the consumer job, so a browser compare failure blocks the merge without requiring a new branch-protection check name.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Continue requiring only local runs.** Rejected: execution depends on developer memory, which is precisely why stale goldens drift across PRs, and cannot guarantee that the PR introducing a behavior change carries its own expected-output diff.
|
||||
|
||||
**Run CI in `refresh` mode and then check the working tree.** Rejected: checking after writing turns the assertion mechanism into a generator; if the working-tree check is wired incorrectly, it can turn a regression into a passing expected-output update. Replay compares the existing goldens directly and has a smaller failure surface.
|
||||
|
||||
**Create a standalone browser job and rebuild the entire repository.** Rejected: it would duplicate dependency installation and the publishable build. The existing Linux consumer job already consumes the same built-tree artifact and is part of the unified required verdict.
|
||||
|
||||
**Replace real Chromium with jsdom snapshots.** Rejected: jsdom does not cover the browser, HTTP/SSE carriage, or the composition of real client plugin bundles. It remains useful for fast lower-layer feedback, but cannot replace the assembled browser chain.
|
||||
|
||||
## Consequences
|
||||
|
||||
Before merge, every PR proves that the current web assembly matches all committed browser expected outputs, turning a missed refresh from an “unrelated change in a later PR” into a failure in the PR that introduced it. The cost is Chromium provisioning and one serial pass through the browser scenarios in Linux CI; built-artifact reuse and the default-branch browser cache avoid duplicate builds and routine downloads. The gate still makes no claim of cross-platform browser consistency, and if a Playwright/Chromium upgrade changes the ARIA format, the upgrade PR must explicitly refresh the expected outputs and review the churn.
|
||||
@@ -0,0 +1,33 @@
|
||||
# Agent Note: Web 浏览器预期输出的必需 CI 门禁
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-07-30-web-browser-snapshot-ci-gate.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
[无密钥 Web 浏览器 e2e 车道](2026-07-24-web-gui-browser-e2e-lane.md)只由本地 `pnpm run test:web` 运行，PR CI 不比较 `apps/web/tests/snapshots/**/*.expected.md`。因此，改变用户可见 Web 输出的 PR 可以在漏刷预期输出时保持绿色；后来任意分支显式运行 `DSH_SNAPSHOT=refresh`，都会替前序变更补账并产生与本分支无关的 diff。普通本地运行已经默认使用只读 replay，缺口是 PR 级的强制执行，而不是禁止 refresh 写入。
|
||||
|
||||
## 决策
|
||||
|
||||
Linux PR 的 `node 24 / snapshots and artifacts` 必须运行完整 Web 浏览器 replay/compare。`scripts/run-gates.ts` 把 `test:web:built` 作为 `ci-consumers` 的一个 gate，并显式注入 `DSH_SNAPSHOT=replay`；CI 永不以 `record` 或 `refresh` 模式运行，因此提交的 golden 与当前组装应用不一致时测试直接失败，不会在 runner 内静默改写后通过。
|
||||
|
||||
静态 CI job 已经构建全部发布产物；它把 `apps/web/dist` 连同 package `lib/` 放进 built-tree artifact，consumer job 复用该 artifact 而不重复全仓构建。consumer job 按 lockfile 中的 Playwright 版本安装 Chromium 及系统依赖，并以操作系统和 `pnpm-lock.yaml` 哈希缓存浏览器。默认分支的 Linux 串行 job 运行同一 compare 命令并产出默认分支缓存，后续 PR 可直接恢复。
|
||||
|
||||
本地 `pnpm run test:web` 仍先构建再运行浏览器全集；`test:web:built` 是已有构建产物的执行入口。开发者只在确认用户可见输出有意变化后显式运行 `DSH_SNAPSHOT=refresh pnpm run test:web`，评审每一处 expected diff，再以 replay 模式复验不再写文件。
|
||||
|
||||
门禁保持 Linux-only：这些场景面向 POSIX，Windows 与 macOS 的串行参考 job 不重复运行。PR 的 `all checks passed` 已依赖 consumer job，因此浏览器 compare 失败会阻止合并，无需新增 branch-protection check 名称。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
**继续只要求本地运行。** 已否决：执行依赖开发者记忆，正是旧 golden 跨 PR 漂移的原因，不能保证产生行为变化的 PR 自己携带 expected diff。
|
||||
|
||||
**让 CI 以 `refresh` 模式运行后检查工作树。** 已否决：写后比较把断言机制变成生成器，若工作树检查接线失效就会把回归更新成绿色；replay 直接比较已有 golden，失败面更小。
|
||||
|
||||
**新建独立 browser job 并重新构建全仓。** 已否决：它会重复依赖安装和发布构建。现有 Linux consumer job 已消费同一 built-tree artifact，并已被统一的 required verdict 聚合。
|
||||
|
||||
**用 jsdom 快照代替真实 Chromium。** 已否决：jsdom 不覆盖浏览器、HTTP/SSE 承载及真实 client plugin bundle 组合；它保留为快速的下层反馈，不能替代 assembled browser chain。
|
||||
|
||||
## 后果
|
||||
|
||||
每个 PR 都在合并前证明当前 Web 组装与所有已提交的浏览器 expected 一致，漏刷从“后续 PR 的无关变化”变成引入 PR 自己的失败。成本是 Linux CI 增加 Chromium 供给和一轮串行浏览器场景；built artifact 复用与默认分支浏览器缓存避免重复构建和常态下载。门禁仍不声称跨平台浏览器一致性，Playwright/Chromium 升级若改变 aria 格式，升级 PR 必须显式 refresh 并评审 churn。
|
||||
Reference in New Issue
Block a user