test(web): expand long-chat regression and performance coverage

This commit is contained in:
kingwl
2026-08-02 13:16:38 +08:00
committed by imccyu
parent 6514a59d5d
commit bf6cb38fbc
11 changed files with 1866 additions and 25 deletions

View File

@@ -24,7 +24,7 @@ Keyless model displacement is the disabled adapter row plus `installLlmReplay` f
### Determinism rules
The barrier stack for replay-mode browser assertions is, in order: (1) host-side `await agent.whenIdle()` under a timeout, keyed off the in-process `turn/end` — the idle flip follows the persistence flush, so one await covers turn completion and durability; (2) browser settled poll (streaming detached, final text visible). Record-mode log harvest runs after `whenIdle()` and before scaffold disposal while the live session remains available. An in-process `turn/end` listener alone is a wrong barrier (it fires before the SSE frame reaches the browser and before the fsync); file polling is banned (slow on NFS, superseded by `whenIdle`); `networkidle` is banned outright (never resolves while an SSE stream is open).
The barrier stack for replay-mode browser assertions is, in order: (1) host-side `await agent.whenIdle()` under a timeout, keyed off the in-process `turn/end` — the idle flip follows the persistence flush, so one await covers turn completion and durability; (2) browser settled poll (streaming detached, final text visible). Record-mode log harvest runs after `whenIdle()` and before scaffold disposal while the live session remains available. An in-process `turn/end` listener alone is a wrong barrier (it fires before the SSE frame reaches the browser and before the fsync); polling persistence files as a turn-completion or durability barrier is banned (slow on NFS, superseded by `whenIdle`), while a tool-controlled temp readiness marker may be polled only as an interaction gate before that completion barrier; `networkidle` is banned outright (never resolves while an SSE stream is open). Navigation assertions arm both initial `session.list` and `workspace.list` responses before page load, then wait for the seeded DOM projection; the mounted shell alone is not readiness because late bootstrap can replace controlled state.
No single-shot transient-DOM assertions: every hop from replay yield to React commit can coalesce chunks, so sampling `[data-streaming]` is a race by construction. Streaming incrementality is asserted from the persisted `assistant/chunk` events (model-visible ⟺ logged makes the log the authoritative proof). `dsh-llm-replay`'s opt-in `paceMs` (default absent = burst) is a realism knob so the browser observes genuinely incremental SSE; correctness never leans on it, and abort during a pace wait cancels promptly.
@@ -42,13 +42,13 @@ The typecheck plane split is structural: the host scaffold, its support module,
### Coverage contract
The lane covers three behavior families. Live-turn scenarios pin ordinary tool execution, cancellation, non-retryable failure, transient retry, resident questions, and mid-turn steering; synchronization uses durable events, `whenIdle()`, or an explicit replay marker rather than delays. Cold-history scenarios seed through the real persistence API and cover history rendering, sidebar search, trajectory and waterfall views, and tool details without model calls. Browser-lifecycle scenarios cover first-send workspace materialization, reload recovery, layout reset, theme and locale preferences, and workspace create/rename/view operations. Each family asserts the browser surface and the authoritative host state; a stray model call or under-consumed fixture fails teardown.
The lane covers three behavior families. Live-turn scenarios pin ordinary tool execution, cancellation, non-retryable failure, transient retry, resident questions, and mid-turn steering; synchronization uses durable events, `whenIdle()`, or an explicit replay marker rather than delays. Cold-history scenarios seed through the real persistence API and cover history rendering, sidebar search, trajectory and waterfall views, and tool details without model calls. Browser-lifecycle scenarios cover first-send workspace materialization, reload recovery, layout reset, theme and locale preferences, and workspace create/rename/view operations. Each family asserts the browser surface and the authoritative host state; a stray model call or under-consumed fixture fails teardown. The required lane additionally carries an 88-turn synthetic Chat scroll contract mixing wrapped Markdown, fenced code, and paired bash calls/results. Real wheel, composer, tool, tab, session, and viewport interactions assert a named settled row's top relative to the transcript scrollport and distance from the true bottom across concurrent history prepend plus paced streaming, pinned/away streaming, tool-disclosure offscreen cycles, expanded-history view/session remount, width reflow, immediate pinned remount, composer resize, and textarea wheel chaining; it deliberately pins neither DOM cardinality nor absolute `scrollTop`, so the same contract can qualify a virtualized implementation. A separate interaction contract over the same fixture pins heterogeneous-row order, independent adjacent tool disclosure, exact user-message clipboard content, a turn-bounded message fork, source/child isolation, and a real follow-up turn in the child; wheel input only navigates to semantic targets and carries no geometry expectation. A short live-history contract starts from a blank workspace and drives consecutive composer turns, including real bash call/result rounds and a paced long final response, pinning one session identity, exact per-turn event ownership, browser echo uniqueness, and composer recovery without timing thresholds.
### CI stance
The lane is a required compare-only gate for Linux pull requests under the [browser snapshot CI decision](2026-07-30-web-browser-snapshot-ci-gate.md). The `node 24 / snapshots and artifacts` consumer job owns the [single Linux build](../process/2026-07-30-independent-ci-consumer-build.md), installs the lockfile-selected Chromium, restores its OS-and-lockfile-keyed cache, and runs the lane with `DSH_SNAPSHOT=replay`. This is an intentional plane split: the host and specs use the [tsx source-launch contract](../architecture/2026-07-29-dsh-source-launch-tsx-esm.md), while the browser consumes `apps/web/dist` and package `lib/client.js` artifacts, so the gate depends on `built-package-invariants` for those client artifacts. The hosted and self-hosted default-branch Linux serial jobs run the same gate; the hosted job produces the browser cache consumed by pull requests, while the persistent self-hosted pool needs no hosted cache. CI never records or refreshes goldens. Scenarios remain POSIX-oriented and stay outside the Windows and macOS matrices.
High-cardinality performance diagnostics use the separate opt-in `apps/web/tests/**/*.perf.ts` inventory selected only by `vitest.web.perf.config.ts`. The isolated `complex-history.perf.ts` cases reuse the real scaffold: the workspace case seeds 1,000 compact sessions plus one 500-turn history containing 500 tool calls, exhausts and remounts that history in Chat, and reports Chromium main-thread, DOM, listener, heap, paging, search, and Trajectory measurements. Two continuation cases seed the same long history but compare the default 24-turn Chat window with all 500 turns expanded before each continues eight identical turns through the real composer, agent loop, SSE wire, tools, and persistence; two turns execute a real `bash` call and assert its durable result, while the final turn fills an 8,232-character mixed-language prompt and replays 120 paced text deltas. A separate soak case starts from a blank session, drives 100 consecutive real composer turns with a `bash` call and result every tenth turn, forces GC every ten turns, and reports ten-turn latency windows plus retained browser state. Per-turn diagnostics cover composer fill, click-to-user-echo, click-to-first-chunk, completion, browser mutations, persisted chunks, and tool events; the synthetic replay model has enough context capacity to keep fixture cardinality stable instead of consuming scripted calls through compaction. Structural assertions pin the intended load, stream, and tool shapes, but timing remains threshold-free because machine speed is not a correctness contract. The required `vitest.web.config.ts` inventory remains limited to `*.e2e.ts` and `*.snapshot.ts`, so neither `test:web:built` nor its CI gate collects performance cases.
High-cardinality performance diagnostics use the separate opt-in `apps/web/tests/**/*.perf.ts` inventory selected only by `vitest.web.perf.config.ts`. The isolated `complex-history.perf.ts` cases reuse the real scaffold: the workspace case seeds 1,000 compact sessions plus one 500-turn history containing 500 tool calls, exhausts and remounts that history in Chat, and reports Chromium main-thread, DOM, listener, heap, paging, search, and Trajectory measurements. Two continuation cases seed the same long history but compare the default 24-turn Chat window with all 500 turns expanded before each continues eight identical turns through the real composer, agent loop, SSE wire, tools, and persistence; two turns execute a real `bash` call and assert its durable result, while the final turn fills an 8,232-character mixed-language prompt and replays 120 paced text deltas. A separate soak case starts from a blank session, drives 100 consecutive real composer turns with a `bash` call and result every tenth turn, forces GC every ten turns, and reports ten-turn latency windows plus retained browser state. It then submits a 101st text-only turn with a trusted browser click and measures browser-clock send-to-transcript-DOM and send-to-post-paint latency, excluding the composer's draft mirrors, separately from full-turn completion. Per-turn diagnostics cover composer fill, click-to-user-echo, click-to-first-chunk, completion, browser mutations, persisted chunks, and tool events; the synthetic replay model has enough context capacity to keep fixture cardinality stable instead of consuming scripted calls through compaction. Structural assertions pin the intended load, stream, and tool shapes, but timing remains threshold-free because machine speed is not a correctness contract. The required `vitest.web.config.ts` inventory remains limited to `*.e2e.ts` and `*.snapshot.ts`, so neither `test:web:built` nor its CI gate collects performance cases.
## Prior art
@@ -88,6 +88,7 @@ Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot
- **Follow-up-prompt-after-resume scenario**: the history/live stitch path over the real wire; add as its own scenario when that code changes or regresses.
- **Composer steering gesture**: the input locks while running (stop-or-wait), so the steering scenario steers over the wire from the page; `TODO(web-steer-composer)` upgrades the drive step to a real composer gesture when the product grows one.
- **Drag session reorder**: `workspace.insertSessionBefore` has no browser scenario; it needs two sessions materialized in one workspace plus synthesized HTML5 drag events. Add it when that surface changes or regresses. The inert session Rename/Fork/Delete and workspace Delete menu rows get scenarios when they gain behavior.
- **Long-history Chat-to-Trajectory Inspect**: the independent inspection source exhausts history after the view opens, while the selected record is addressed by a derived table index that can move as older pages prepend. Short-history Inspect remains covered; the long-history interaction contract excludes this handoff until selection has a stable semantic identity.
## Consequences