test(web): expand long-chat regression and performance coverage
This commit is contained in:
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md
|
||||
2026-07-24-web-gui-browser-e2e-lane.md: e9559aa1f6fba89a99d91e79026552511b337793
|
||||
2026-07-24-web-gui-browser-e2e-lane.zh.md: 5f2047125e38f6e1eecf4b8b81d6f5da16686afa
|
||||
2026-07-24-web-gui-browser-e2e-lane.md: f8519a9622d2f7216226a695db95dbebdbf24ea1
|
||||
2026-07-24-web-gui-browser-e2e-lane.zh.md: 294f3e840e0242d9a0d9c53ac510d44d3b0d100f
|
||||
|
||||
@@ -24,7 +24,7 @@ Keyless model displacement is the disabled adapter row plus `installLlmReplay` f
|
||||
|
||||
### Determinism rules
|
||||
|
||||
The barrier stack for replay-mode browser assertions is, in order: (1) host-side `await agent.whenIdle()` under a timeout, keyed off the in-process `turn/end` — the idle flip follows the persistence flush, so one await covers turn completion and durability; (2) browser settled poll (streaming detached, final text visible). Record-mode log harvest runs after `whenIdle()` and before scaffold disposal while the live session remains available. An in-process `turn/end` listener alone is a wrong barrier (it fires before the SSE frame reaches the browser and before the fsync); file polling is banned (slow on NFS, superseded by `whenIdle`); `networkidle` is banned outright (never resolves while an SSE stream is open).
|
||||
The barrier stack for replay-mode browser assertions is, in order: (1) host-side `await agent.whenIdle()` under a timeout, keyed off the in-process `turn/end` — the idle flip follows the persistence flush, so one await covers turn completion and durability; (2) browser settled poll (streaming detached, final text visible). Record-mode log harvest runs after `whenIdle()` and before scaffold disposal while the live session remains available. An in-process `turn/end` listener alone is a wrong barrier (it fires before the SSE frame reaches the browser and before the fsync); polling persistence files as a turn-completion or durability barrier is banned (slow on NFS, superseded by `whenIdle`), while a tool-controlled temp readiness marker may be polled only as an interaction gate before that completion barrier; `networkidle` is banned outright (never resolves while an SSE stream is open). Navigation assertions arm both initial `session.list` and `workspace.list` responses before page load, then wait for the seeded DOM projection; the mounted shell alone is not readiness because late bootstrap can replace controlled state.
|
||||
|
||||
No single-shot transient-DOM assertions: every hop from replay yield to React commit can coalesce chunks, so sampling `[data-streaming]` is a race by construction. Streaming incrementality is asserted from the persisted `assistant/chunk` events (model-visible ⟺ logged makes the log the authoritative proof). `dsh-llm-replay`'s opt-in `paceMs` (default absent = burst) is a realism knob so the browser observes genuinely incremental SSE; correctness never leans on it, and abort during a pace wait cancels promptly.
|
||||
|
||||
@@ -42,13 +42,13 @@ The typecheck plane split is structural: the host scaffold, its support module,
|
||||
|
||||
### Coverage contract
|
||||
|
||||
The lane covers three behavior families. Live-turn scenarios pin ordinary tool execution, cancellation, non-retryable failure, transient retry, resident questions, and mid-turn steering; synchronization uses durable events, `whenIdle()`, or an explicit replay marker rather than delays. Cold-history scenarios seed through the real persistence API and cover history rendering, sidebar search, trajectory and waterfall views, and tool details without model calls. Browser-lifecycle scenarios cover first-send workspace materialization, reload recovery, layout reset, theme and locale preferences, and workspace create/rename/view operations. Each family asserts the browser surface and the authoritative host state; a stray model call or under-consumed fixture fails teardown.
|
||||
The lane covers three behavior families. Live-turn scenarios pin ordinary tool execution, cancellation, non-retryable failure, transient retry, resident questions, and mid-turn steering; synchronization uses durable events, `whenIdle()`, or an explicit replay marker rather than delays. Cold-history scenarios seed through the real persistence API and cover history rendering, sidebar search, trajectory and waterfall views, and tool details without model calls. Browser-lifecycle scenarios cover first-send workspace materialization, reload recovery, layout reset, theme and locale preferences, and workspace create/rename/view operations. Each family asserts the browser surface and the authoritative host state; a stray model call or under-consumed fixture fails teardown. The required lane additionally carries an 88-turn synthetic Chat scroll contract mixing wrapped Markdown, fenced code, and paired bash calls/results. Real wheel, composer, tool, tab, session, and viewport interactions assert a named settled row's top relative to the transcript scrollport and distance from the true bottom across concurrent history prepend plus paced streaming, pinned/away streaming, tool-disclosure offscreen cycles, expanded-history view/session remount, width reflow, immediate pinned remount, composer resize, and textarea wheel chaining; it deliberately pins neither DOM cardinality nor absolute `scrollTop`, so the same contract can qualify a virtualized implementation. A separate interaction contract over the same fixture pins heterogeneous-row order, independent adjacent tool disclosure, exact user-message clipboard content, a turn-bounded message fork, source/child isolation, and a real follow-up turn in the child; wheel input only navigates to semantic targets and carries no geometry expectation. A short live-history contract starts from a blank workspace and drives consecutive composer turns, including real bash call/result rounds and a paced long final response, pinning one session identity, exact per-turn event ownership, browser echo uniqueness, and composer recovery without timing thresholds.
|
||||
|
||||
### CI stance
|
||||
|
||||
The lane is a required compare-only gate for Linux pull requests under the [browser snapshot CI decision](2026-07-30-web-browser-snapshot-ci-gate.md). The `node 24 / snapshots and artifacts` consumer job owns the [single Linux build](../process/2026-07-30-independent-ci-consumer-build.md), installs the lockfile-selected Chromium, restores its OS-and-lockfile-keyed cache, and runs the lane with `DSH_SNAPSHOT=replay`. This is an intentional plane split: the host and specs use the [tsx source-launch contract](../architecture/2026-07-29-dsh-source-launch-tsx-esm.md), while the browser consumes `apps/web/dist` and package `lib/client.js` artifacts, so the gate depends on `built-package-invariants` for those client artifacts. The hosted and self-hosted default-branch Linux serial jobs run the same gate; the hosted job produces the browser cache consumed by pull requests, while the persistent self-hosted pool needs no hosted cache. CI never records or refreshes goldens. Scenarios remain POSIX-oriented and stay outside the Windows and macOS matrices.
|
||||
|
||||
High-cardinality performance diagnostics use the separate opt-in `apps/web/tests/**/*.perf.ts` inventory selected only by `vitest.web.perf.config.ts`. The isolated `complex-history.perf.ts` cases reuse the real scaffold: the workspace case seeds 1,000 compact sessions plus one 500-turn history containing 500 tool calls, exhausts and remounts that history in Chat, and reports Chromium main-thread, DOM, listener, heap, paging, search, and Trajectory measurements. Two continuation cases seed the same long history but compare the default 24-turn Chat window with all 500 turns expanded before each continues eight identical turns through the real composer, agent loop, SSE wire, tools, and persistence; two turns execute a real `bash` call and assert its durable result, while the final turn fills an 8,232-character mixed-language prompt and replays 120 paced text deltas. A separate soak case starts from a blank session, drives 100 consecutive real composer turns with a `bash` call and result every tenth turn, forces GC every ten turns, and reports ten-turn latency windows plus retained browser state. Per-turn diagnostics cover composer fill, click-to-user-echo, click-to-first-chunk, completion, browser mutations, persisted chunks, and tool events; the synthetic replay model has enough context capacity to keep fixture cardinality stable instead of consuming scripted calls through compaction. Structural assertions pin the intended load, stream, and tool shapes, but timing remains threshold-free because machine speed is not a correctness contract. The required `vitest.web.config.ts` inventory remains limited to `*.e2e.ts` and `*.snapshot.ts`, so neither `test:web:built` nor its CI gate collects performance cases.
|
||||
High-cardinality performance diagnostics use the separate opt-in `apps/web/tests/**/*.perf.ts` inventory selected only by `vitest.web.perf.config.ts`. The isolated `complex-history.perf.ts` cases reuse the real scaffold: the workspace case seeds 1,000 compact sessions plus one 500-turn history containing 500 tool calls, exhausts and remounts that history in Chat, and reports Chromium main-thread, DOM, listener, heap, paging, search, and Trajectory measurements. Two continuation cases seed the same long history but compare the default 24-turn Chat window with all 500 turns expanded before each continues eight identical turns through the real composer, agent loop, SSE wire, tools, and persistence; two turns execute a real `bash` call and assert its durable result, while the final turn fills an 8,232-character mixed-language prompt and replays 120 paced text deltas. A separate soak case starts from a blank session, drives 100 consecutive real composer turns with a `bash` call and result every tenth turn, forces GC every ten turns, and reports ten-turn latency windows plus retained browser state. It then submits a 101st text-only turn with a trusted browser click and measures browser-clock send-to-transcript-DOM and send-to-post-paint latency, excluding the composer's draft mirrors, separately from full-turn completion. Per-turn diagnostics cover composer fill, click-to-user-echo, click-to-first-chunk, completion, browser mutations, persisted chunks, and tool events; the synthetic replay model has enough context capacity to keep fixture cardinality stable instead of consuming scripted calls through compaction. Structural assertions pin the intended load, stream, and tool shapes, but timing remains threshold-free because machine speed is not a correctness contract. The required `vitest.web.config.ts` inventory remains limited to `*.e2e.ts` and `*.snapshot.ts`, so neither `test:web:built` nor its CI gate collects performance cases.
|
||||
|
||||
## Prior art
|
||||
|
||||
@@ -88,6 +88,7 @@ Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot
|
||||
- **Follow-up-prompt-after-resume scenario**: the history/live stitch path over the real wire; add as its own scenario when that code changes or regresses.
|
||||
- **Composer steering gesture**: the input locks while running (stop-or-wait), so the steering scenario steers over the wire from the page; `TODO(web-steer-composer)` upgrades the drive step to a real composer gesture when the product grows one.
|
||||
- **Drag session reorder**: `workspace.insertSessionBefore` has no browser scenario; it needs two sessions materialized in one workspace plus synthesized HTML5 drag events. Add it when that surface changes or regresses. The inert session Rename/Fork/Delete and workspace Delete menu rows get scenarios when they gain behavior.
|
||||
- **Long-history Chat-to-Trajectory Inspect**: the independent inspection source exhausts history after the view opens, while the selected record is addressed by a derived table index that can move as older pages prepend. Short-history Inspect remains covered; the long-history interaction contract excludes this handoff until selection has a stable semantic identity.
|
||||
|
||||
## Consequences
|
||||
|
||||
|
||||
@@ -24,7 +24,7 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu
|
||||
|
||||
### 确定性规则
|
||||
|
||||
回放模式下浏览器断言的屏障栈,按序:(1)host 侧 `await agent.whenIdle()` 加超时,以进程内 `turn/end` 为锚——空闲翻转发生在持久化落盘之后,一次等待同时覆盖轮次完成与持久性;(2)浏览器安定轮询(流式输出节点已卸载、最终文本可见)。录制模式下,日志采收在 `whenIdle()` 之后、scaffold 释放之前进行,此时运行中的会话仍然可用。单独监听进程内 `turn/end` 是错误屏障(它先于 SSE 帧到达浏览器、先于 fsync 触发);文件轮询被禁止(NFS 上慢,且被 `whenIdle` 取代);`networkidle` 被彻底禁止(SSE 流保持打开时它永不解析)。
|
||||
回放模式下浏览器断言的屏障栈,按序:(1)host 侧 `await agent.whenIdle()` 加超时,以进程内 `turn/end` 为锚——空闲翻转发生在持久化落盘之后,一次等待同时覆盖轮次完成与持久性;(2)浏览器安定轮询(流式输出节点已卸载、最终文本可见)。录制模式下,日志采收在 `whenIdle()` 之后、scaffold 释放之前进行,此时运行中的会话仍然可用。单独监听进程内 `turn/end` 是错误屏障(它先于 SSE 帧到达浏览器、先于 fsync 触发);禁止轮询持久化文件来充当轮次完成或持久性屏障(NFS 上慢,且被 `whenIdle` 取代),但工具控制的临时就绪标记可以仅作为该完成屏障之前的交互门控进行轮询;`networkidle` 被彻底禁止(SSE 流保持打开时它永不解析)。导航断言会在页面加载前同时监听 `session.list` 和 `workspace.list` 的初始响应,随后等待播种数据投影到 DOM;仅凭 shell 已挂载不能判定就绪,因为较晚完成的 bootstrap 可能替换受控状态。
|
||||
|
||||
不做单次瞬态 DOM 断言:从回放产出到 React 提交的每一跳都可能合并分片,采样 `[data-streaming]` 天然就是竞态。流式输出的增量性由持久化的 `assistant/chunk` 事件断言(模型可见 ⟺ 已记录,使日志成为权威证据)。`dsh-llm-replay` 的可选 `paceMs`(默认缺省 = 突发)只是让浏览器观察到真正增量 SSE 的真实感旋钮;正确性绝不依赖它,且节奏等待期间中止会即时取消。
|
||||
|
||||
@@ -42,13 +42,13 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu
|
||||
|
||||
### 覆盖契约
|
||||
|
||||
该车道覆盖三类行为。实时轮次场景钉住普通工具执行、取消、不可重试失败、瞬态重试、常驻提问与轮次中途 steering;同步依赖持久事件、`whenIdle()` 或显式回放标记,而不使用延时。冷历史场景通过真实持久化 API 播种,在不调用模型的情况下覆盖历史渲染、侧栏搜索、Trajectory 与 Waterfall 视图及工具详情。浏览器生命周期场景覆盖首次发送时物化工作区、重新加载恢复、布局重置、主题与语言偏好,以及工作区的创建、重命名和视图操作。每类场景都断言浏览器表面和权威的 host 状态;离群的模型调用或未耗尽的 fixture 会使拆卸失败。
|
||||
该车道覆盖三类行为。实时轮次场景钉住普通工具执行、取消、不可重试失败、瞬态重试、常驻提问与轮次中途 steering;同步依赖持久事件、`whenIdle()` 或显式回放标记,而不使用延时。冷历史场景通过真实持久化 API 播种,在不调用模型的情况下覆盖历史渲染、侧栏搜索、Trajectory 与 Waterfall 视图及工具详情。浏览器生命周期场景覆盖首次发送时物化工作区、重新加载恢复、布局重置、主题与语言偏好,以及工作区的创建、重命名和视图操作。每类场景都断言浏览器表面和权威的 host 状态;离群的模型调用或未耗尽的 fixture 会使拆卸失败。必需车道还包含一份合成的 88 轮 Chat 滚动契约,其中混合了换行 Markdown、围栏代码以及成对的 bash 调用/结果。真实 wheel、输入框、工具、tab、会话与 viewport 交互会在并发历史前插加带节奏流式输出、贴底/离底流式输出、工具 disclosure 离屏循环、扩展历史后的视图/会话重新挂载、宽度重排、贴底后立即重新挂载、输入框尺寸变化以及 textarea wheel 链场景中,断言一个具名已结算行相对 transcript scrollport 的顶部位置和到真实底部的距离;它刻意不钉 DOM 基数或绝对 `scrollTop`,因此同一契约可以验收虚拟化实现。另一份基于同一 fixture 的交互契约钉住异构行顺序、相邻工具 disclosure 的独立状态、用户消息剪贴板内容的精确值、以轮次为边界的消息 fork、源会话/子会话隔离,以及子会话中的一次真实追问轮次;wheel 输入只用于导航到语义目标,不承载几何预期。一份简短的实时历史契约从空白工作区开始,连续驱动输入框轮次,其中包括真实的 bash 调用/结果轮次和一段带节奏的长篇最终响应;它钉住单一会话身份、每轮事件的精确归属、浏览器回显唯一性与输入框恢复,不设置时间阈值。
|
||||
|
||||
### CI 立场
|
||||
|
||||
根据[浏览器快照 CI 决策](2026-07-30-web-browser-snapshot-ci-gate.md),该车道是 Linux 拉取请求必需的只比较门禁。`node 24 / snapshots and artifacts` 消费方任务在[消费方独立构建](../process/2026-07-30-independent-ci-consumer-build.md)中负责唯一一次 Linux 构建,安装锁文件选定的 Chromium,恢复以操作系统和锁文件为键的缓存,并用 `DSH_SNAPSHOT=replay` 运行该车道。这是有意的平面切分:host 与 spec 使用 [tsx 源码启动契约](../architecture/2026-07-29-dsh-source-launch-tsx-esm.md),浏览器则消费 `apps/web/dist` 和包的 `lib/client.js` 产物,因此门禁依赖 `built-package-invariants` 提供这些客户端产物。托管和自托管的默认分支 Linux 串行任务运行同一门禁;托管任务生成供 PR 消费的浏览器缓存,持久化自托管池则不需要托管侧缓存。CI 从不录制或刷新预期输出。场景仍面向 POSIX,并继续置于 Windows 和 macOS 矩阵之外。
|
||||
|
||||
高基数性能诊断使用单独按需启用的 `apps/web/tests/**/*.perf.ts` 清单,并且只由 `vitest.web.perf.config.ts` 选中。`complex-history.perf.ts` 的隔离用例复用真实 scaffold:工作区用例播种 1,000 个紧凑会话以及一份包含 500 次工具调用的 500 轮次历史,在 Chat 中穷尽并重新挂载该历史,并报告 Chromium 主线程、DOM、监听器、堆内存、分页、搜索和 Trajectory 测量结果。两个续聊用例播种同一份长历史,但比较默认的 24 轮次 Chat 窗口与展开全部 500 轮次的状态,然后各自通过真实输入框、agent loop、SSE wire、工具和持久化继续进行 8 个相同轮次;其中两轮执行真实 `bash` 调用并断言其持久化结果,最后一轮则填入一条包含 8,232 个字符的混合语言提示词,并回放 120 个带节奏的文本增量。一个单独的 soak 用例从空白会话开始,通过真实输入框连续驱动 100 轮,每第 10 轮执行一次 `bash` 调用并产生结果,每 10 轮强制执行一次 GC,并报告每 10 轮的延迟窗口及保留的浏览器状态。逐轮诊断涵盖输入框填入、点击到用户消息回显、点击到首个分片、完成、浏览器变更、持久化分片和工具事件;合成回放模型拥有足够的上下文容量,可使 fixture 基数保持稳定,而不会因压缩(compaction)消耗脚本化调用。结构性断言钉住预期的负载、流和工具形状,但时间仍不设阈值,因为机器速度不属于正确性契约。必需的 `vitest.web.config.ts` 清单仍仅限 `*.e2e.ts` 和 `*.snapshot.ts`,因此 `test:web:built` 及其 CI 门禁都不会收集性能用例。
|
||||
高基数性能诊断使用单独按需启用的 `apps/web/tests/**/*.perf.ts` 清单,并且只由 `vitest.web.perf.config.ts` 选中。`complex-history.perf.ts` 的隔离用例复用真实 scaffold:工作区用例播种 1,000 个紧凑会话以及一份包含 500 次工具调用的 500 轮次历史,在 Chat 中穷尽并重新挂载该历史,并报告 Chromium 主线程、DOM、监听器、堆内存、分页、搜索和 Trajectory 测量结果。两个续聊用例播种同一份长历史,但比较默认的 24 轮次 Chat 窗口与展开全部 500 轮次的状态,然后各自通过真实输入框、agent loop、SSE wire、工具和持久化继续进行 8 个相同轮次;其中两轮执行真实 `bash` 调用并断言其持久化结果,最后一轮则填入一条包含 8,232 个字符的混合语言提示词,并回放 120 个带节奏的文本增量。一个单独的 soak 用例从空白会话开始,通过真实输入框连续驱动 100 轮,每第 10 轮执行一次 `bash` 调用并产生结果,每 10 轮强制执行一次 GC,并报告每 10 轮的延迟窗口及保留的浏览器状态。随后它通过受信任的浏览器点击提交第 101 个纯文本轮次,并使用浏览器时钟分别测量发送到 transcript DOM 和发送到绘制后的延迟,排除输入框的草稿镜像,并与完整轮次完成时间分开。逐轮诊断涵盖输入框填入、点击到用户消息回显、点击到首个分片、完成、浏览器变更、持久化分片和工具事件;合成回放模型拥有足够的上下文容量,可使 fixture 基数保持稳定,而不会因压缩(compaction)消耗脚本化调用。结构性断言钉住预期的负载、流和工具形状,但时间仍不设阈值,因为机器速度不属于正确性契约。必需的 `vitest.web.config.ts` 清单仍仅限 `*.e2e.ts` 和 `*.snapshot.ts`,因此 `test:web:built` 及其 CI 门禁都不会收集性能用例。
|
||||
|
||||
## 业界先例
|
||||
|
||||
@@ -88,6 +88,7 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu
|
||||
- **恢复后追问场景**:真实 wire 上的历史/实时缝合路径;当该代码变更或回归时作为独立场景补充。
|
||||
- **输入框 steering 手势**:输入在运行期间锁定(只能停止或等待),因此 steering 场景从页面走 wire 做 steer;`TODO(web-steer-composer)` 待产品长出真实的输入框手势后,把驱动步骤升级为该手势。
|
||||
- **拖拽会话重排**:`workspace.insertSessionBefore` 尚无浏览器场景;它需要在同一个工作区里物化两个会话,并合成 HTML5 拖拽事件。当该表面变更或回归时再补充。无行为的会话 Rename/Fork/Delete 和工作区 Delete 菜单行待获得行为后再补充场景。
|
||||
- **长历史 Chat 到 Trajectory 的 Inspect**:独立的检查数据源会在视图打开后穷尽历史,而所选记录由一个派生的表格索引定位;随着较早页面前插,该索引可能移动。短历史 Inspect 仍有覆盖;在选中项具有稳定的语义身份之前,长历史交互契约不包含这项交接。
|
||||
|
||||
## 后果
|
||||
|
||||
|
||||
Reference in New Issue
Block a user