test(tui): replay recorded sessions through real tools

This commit is contained in:
Tianyi Cui
2026-07-19 01:31:55 +08:00
parent bea6a74ea0
commit e68cef1be1
32 changed files with 1874 additions and 488 deletions

View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write
2026-07-18-tui-terminal-state-snapshots.md: 225efe8de95973ecae3f9bf73308e7ae879dddbb
2026-07-18-tui-terminal-state-snapshots.zh.md: b7399a31b0abf41905f495a1ea3b4ecd0c858526
2026-07-18-tui-terminal-state-snapshots.md: 280c2b4faec3bd5e14a24a5df7bc901ad31718cd
2026-07-18-tui-terminal-state-snapshots.zh.md: d1c609c6ce511bba1f498b72444af662b4444578

View File

@@ -8,50 +8,63 @@ English | [中文](2026-07-18-tui-terminal-state-snapshots.zh.md)
The TUI is a stateful renderer. Its user-visible result depends on ANSI parsing, differential frames, wrapping, scrollback, viewport position, terminal width, focus, cursor state, and each tool's presentation intent. Unit tests that collect `Terminal.write()` fragments can prove event handling, but they cannot prove the final screen a terminal displays. The same screen may also be emitted through different write fragments, so pinning those fragments creates false regressions.
Component-line snapshots stop before ANSI reaches a terminal and miss cursor movement, clearing, styling, overlay composition, and reflow. Raster screenshots include font and platform rendering noise that is unrelated to the TUI contract. The TUI therefore needs a deterministic, reviewable representation of terminal state plus a smaller test at the real process and PTY boundary.
Component-line snapshots stop before ANSI reaches a terminal and miss cursor movement, clearing, styling, overlay composition, and reflow. Raster screenshots include font and platform rendering noise that is unrelated to the TUI contract. A completed flow built by directly appending plausible session events has another blind spot: it proves the renderer accepts those shapes, not that the production agent loop and tool implementations produce them.
The TUI therefore needs a deterministic, reviewable representation of terminal state, recorded model journeys that execute the real downstream stack, and a smaller test at the real process and PTY boundary.
## Decision
TUI coverage has three complementary layers:
TUI coverage has four complementary layers:
1. `tui.spec.ts` tests event mapping, input routing, disposal, and error behavior directly.
2. `tui.snapshot.ts` mounts the production TUI against a headless terminal emulator and compares semantic terminal-state goldens.
3. `tui-keyless-smoke.e2e.ts` boots the real Loader composition in a PTY, drives a complete scripted conversation through streaming and `ask_user_question`, exits through `/exit`, and verifies terminal teardown. The production coding-agent configuration also retains its banner/exit and startup-failure PTY cases.
1. `packages/ui/tui/tests/tui.spec.ts` tests event mapping, input routing, disposal, and error behavior directly.
2. `packages/ui/tui/tests/tui.snapshot.ts` mounts the production TUI against a headless terminal emulator for transient states that a completed session log cannot retain: in-flight streaming, pending tool calls, overlays, expansion, compaction reflow, errors, and shutdown.
3. `examples/tui-agent/tests/tui.snapshot.ts` replays committed JSONL session logs through the production agent loop and real tools, then compares the resulting semantic terminal state.
4. `examples/tui-agent/tests/tui-keyless-smoke.e2e.ts` boots the real Loader composition in a PTY, drives a scripted conversation through streaming and `ask_user_question`, and verifies startup, input, exit, failure reporting, and terminal restoration.
The package-local `HeadlessTerminal` implements the same pi-tui `Terminal` interface as the process terminal and feeds every ANSI write into the pinned `@xterm/headless` parser. A snapshot waits for pi-tui's synchronized-output end marker before reading state. This makes a checkpoint represent a completed frame rather than a timer-dependent write prefix.
The runnable TUI has its own `examples/tui-agent` leaf beside the readline `coding-agent` and `acp-agent` leaves. It reuses the coding agent's backend and tool composition through an asserted include patch while fixing the shared terminal app to `ui.mode: tui`; TUI snapshots and PTY tests live with that leaf.
Each golden projects terminal state into text: dimensions, active-buffer and viewport coordinates, lifecycle and cursor state, rows, wrap markers, and non-default style ranges. Scroll-heavy cards capture the used buffer; overlays capture the visible viewport. Text and style remain separate so a reviewer can distinguish content changes from presentation changes without decoding ANSI bytes.
### Recorded-session replay
Every checkpoint also enforces theme independence across the complete terminal state: no RGB colors, no palette entries beyond ANSI 015, and no explicit background colors. Reverse video remains valid for selection because it uses terminal defaults. The suite owns a closed checkpoint list: its type rejects undeclared names, and its inventory checks reject missing checkpoints and orphaned `.golden.txt` files.
Each example-level scenario directory owns `session.jsonl`, optional child logs `session.<n>.jsonl`, and `terminal.golden.txt`. The primary log supplies user-authored `user/message` prompts and the recorded `assistant/chunk` sequence. `dsh-llm-replay` derives one model-call script per session, binds child logs to fresh child sessions, and is the only mocked boundary. The agent loop, bash and filesystem implementations, Code Mode worker, subagent provider, workflow worker, Cordis tools, presenters, and TUI are production implementations.
The suite rejects a journey when its tool-call sequence differs, an expected event count is missing, a tool result is an error, a turn ends in error, a workflow lifecycle is incomplete, or the live child-session count differs from the fixture set. These assertions prevent an attractive terminal golden from hiding a failed or bypassed production path.
The live-model fixtures use `DSH_SNAPSHOT=record`; record mode rewrites their primary and child JSONL logs and terminal goldens. The deterministic Cordis toolchain keeps an authored complete JSONL script because reliably coercing a live model through five exact tool boundaries and two children is not a stable recording contract. `DSH_SNAPSHOT=refresh` replays every committed script keylessly and rewrites only derived terminal goldens. Plain replay compares without writing, and unknown mode values fail loud.
### Semantic terminal projection
The package-local `HeadlessTerminal` implements the same pi-tui `Terminal` interface as the process terminal and feeds every ANSI write into the pinned `@xterm/headless` parser. Snapshot code waits for synchronized frames to quiesce before reading state, so a checkpoint represents a completed screen rather than a timer-dependent write prefix.
Each golden projects dimensions, active-buffer and viewport coordinates, lifecycle and cursor state, rows, wrap markers, and non-default style ranges into text. Scroll-heavy cards capture the used buffer; overlays capture the visible viewport. Text and style remain separate so a reviewer can distinguish content changes from presentation changes without decoding ANSI bytes.
Every checkpoint enforces theme independence across the complete terminal state: no RGB colors, no palette entries beyond ANSI 015, and no explicit background colors. Reverse video remains valid for selection because it uses terminal defaults. Both suites own closed inventories that reject missing scenarios, missing checkpoints, and orphaned golden files.
### Required scenario matrix
| Area | Representative checkpoints | Contract pinned |
| Layer | Scenario | Contract pinned |
|---|---|---|
| Conversation | replay, streaming, completion | Resumed Markdown and reasoning, live deltas, plans, token usage, and max-token completion |
| Code Mode | pending and completed `run_code` | The production Code Mode registry and presenter, source program, captured logs, and result |
| Dynamic workflows | pending and completed `workflow` | The production workflow presenter, metadata, phases, parallel agents, script, and structured result |
| Cordis tools | pending and completed inspect/mount/unmount | The production `cordis_inspect`, `cordis_mount`, and `cordis_unmount` presenters and lifecycle results |
| Advanced tool cards | collapsed and expanded | Terminal, diff, generic, subagent, background-task, and skill card shapes plus output truncation |
| Interaction | question and validation | Constrained multi-select overlay composition, focus, scrolling, selection, and validation errors |
| Surface and layout | before compaction, narrow replacement, wide replacement | Surface replacement removes retired content; resize reflows the surviving surface without resurrection |
| Failure and shutdown | errors/help and disposed terminal | Help and unknown commands, live/turn error de-duplication, interruption, cursor restoration, and terminal stop |
The explicitly model-facing advanced cases use the real `ToolRegistry` configuration and the production Code Mode, workflow, and Cordis tool presenters. Synthetic presenter fixtures are limited to the generic card-shape matrix, where the TUI's input contract is the presenter view itself. Session events remain the driver so replay, streaming, result arrival, surface replacement, and lifecycle ordering exercise the same projection path as production.
The TUI suite is included by `vitest.snapshot.config.ts`, so `pnpm run test:snapshot` compares it keylessly. `pnpm run test:snapshot:refresh` rewrites its derived terminal goldens without contacting a model; `test:snapshot:record` remains meaningful for suites whose transcript source requires recording. Both refresh paths still compare the resulting files in the same run.
| Recorded journey | Multi-turn conversation | Recorded reasoning/text chunks, two input turns, retained history, token totals, and idle editor state |
| Recorded journey | Todo plan | Real `todo_write` execution, result card, and persistent plan rendering |
| Recorded journey | Bash terminal card | Real local executor output, description, exit status, and completed terminal card |
| Recorded journey | Parallel filesystem reads | Two calls from one assistant message, real file contents, ordering, and separate completed cards |
| Recorded journey | Code Mode | Real `run_code` worker execution, two `tool/code-dispatch` events, captured program output, and completed card |
| Recorded journey | Dynamic workflow | Real workflow worker, phase lifecycle, replayed child session, structured return value, and completed card |
| Recorded journey | Cordis dynamic toolchain | Real mount, Code Mode inspect, direct subagent, workflow child, unmount, and all production presenters |
| Transient state | Streaming and pending advanced calls | In-flight reasoning/text plus pending Code Mode, workflow, and Cordis cards that disappear from completed logs |
| Transient state | Cards, interaction, layout, failure, and shutdown | Collapsed/expanded card families, question validation, compaction replacement, resize reflow, help/errors, cursor restoration, and terminal stop |
## Alternatives considered
- **Snapshot raw terminal writes** — rejected because differential rendering may change write boundaries without changing the screen, while cursor and clear sequences are unreadable in review.
- **Snapshot component render lines before terminal output** — rejected because it does not test ANSI parsing, cursor movement, overlays, viewport behavior, or the interaction between independent components in one frame.
- **Snapshot component render lines before terminal output** — rejected because it does not test ANSI parsing, cursor movement, overlays, viewport behavior, or independent components in one frame.
- **Build every completed flow by appending session events** — rejected because a hand-authored event sequence can drift from the agent loop, tool execution, child-session binding, or worker behavior while its presentation test stays green. Direct event construction remains limited to transient renderer states.
- **Reuse ACP stdout goldens as the TUI oracle** — rejected because a recorded model journey is transport-neutral but its presentation is not. TUI scenarios own terminal goldens while using the same JSONL replay vocabulary.
- **Commit raster screenshots** — rejected because fonts, glyph metrics, antialiasing, and host terminal themes make them platform-sensitive and make semantic style changes difficult to review.
- **Use only PTY end-to-end tests** — rejected because raw PTY output is a stream of historical drawing operations, not queryable final state. PTY tests retain the real Loader/input/teardown boundary, while the emulator owns broad state coverage.
- **Copy pi-tui's unpublished virtual-terminal test helper** — rejected because the installed package does not export that helper. A small adapter around the public `@xterm/headless` API keeps the dependency explicit and the projection owned by this package.
## Consequences
- TUI visual regressions produce readable cell-and-style diffs, and the required matrix makes advanced features first-class rather than incidental coverage.
- The test dependency is pinned to the xterm version used by pi-tui. The adapter uses xterm's proposed buffer API, so an xterm upgrade requires rerunning and reviewing the semantic projection.
- The emulator models ANSI terminal state but cannot prove behavior unique to every terminal implementation. The real PTY conversation covers process selection, keyboard input, user interaction, and teardown without duplicating the full matrix.
- Goldens deliberately encode wrapping and viewport behavior at fixed sizes. Intentional layout changes update them through the keyless refresh command and receive ordinary snapshot review.
- Completed advanced snapshots now fail when the real Code Mode, workflow, subagent, filesystem, bash, or Cordis path breaks, rather than accepting a fabricated result event.
- TUI visual regressions produce readable cell-and-style diffs, while JSONL fixtures retain the exact model chunks that made the production path execute.
- The emulator uses xterm's proposed buffer API. An xterm upgrade requires rerunning and reviewing the semantic projection; terminal-specific behavior still needs the PTY smoke.
- Goldens deliberately encode wrapping and viewport behavior at fixed sizes. Intentional layout changes use keyless refresh, while model-journey changes use record mode and review both JSONL and terminal diffs.

View File

@@ -8,50 +8,63 @@ Status: implemented
TUI 是有状态的渲染器。用户最终看到的结果取决于 ANSI 解析、差分帧、换行、回滚缓冲、视口位置、终端宽度、焦点、光标状态,以及各工具的呈现意图。收集 `Terminal.write()` 片段的单元测试可以验证事件处理,却无法验证终端最终显示的画面。同一画面也可能由不同的写入片段产生,因此固定这些片段会制造误报。
组件行快照止于 ANSI 进入终端之前,无法覆盖光标移动、清屏、样式、浮层组合和重排。栅格截图会带入与 TUI 契约无关的字体和平台渲染噪声。因此TUI 既需要一种确定、便于评审的终端状态表示,也需要一项范围更小、覆盖真实进程与 PTY 边界的测试
组件行快照止于 ANSI 进入终端之前,无法覆盖光标移动、清屏、样式、浮层组合和重排。栅格截图会带入与 TUI 契约无关的字体和平台渲染噪声。直接追加看似合理的会话事件来构造完整流程还存在另一处盲区:这种测试只能证明渲染器接受这些数据形态,无法证明生产环境的 agent loop智能体循环和工具实现会生成这些事件
因此TUI 既需要确定、便于评审的终端状态表示,也需要通过已录制模型流程执行真实下游组件,并保留一项范围更小、覆盖真实进程与 PTY 边界的测试。
## 决策
TUI 覆盖分为个互补层次:
TUI 覆盖分为个互补层次:
1. `tui.spec.ts` 直接测试事件映射、输入路由、资源释放和错误行为。
2. `tui.snapshot.ts` 将生产 TUI 挂载到无界面终端模拟器,并比较语义终端状态金标
3. `tui-keyless-smoke.e2e.ts` 在 PTY 中启动真实 Loader 组合,驱动一段完整的脚本化会话,使其依次经过流式输出和 `ask_user_question`,再通过 `/exit` 退出并验证终端清理。生产 coding-agent 配置还保留欢迎信息与退出,以及启动失败两类 PTY 场景
1. `packages/ui/tui/tests/tui.spec.ts` 直接测试事件映射、输入路由、资源释放和错误行为。
2. `packages/ui/tui/tests/tui.snapshot.ts` 将生产 TUI 挂载到无界面终端模拟器,覆盖完整会话日志无法保留的瞬态:进行中的流式输出、待完成工具调用、浮层、展开状态、压缩重排、错误和关闭过程
3. `examples/tui-agent/tests/tui.snapshot.ts` 通过生产 agent loop 和真实工具回放已提交的 JSONL 会话日志,再比较生成的语义终端状态
4. `examples/tui-agent/tests/tui-keyless-smoke.e2e.ts` 在 PTY 中启动真实 Loader 组合,驱动一段经过流式输出和 `ask_user_question` 的脚本化会话,并验证启动、输入、退出、失败报告和终端恢复。
包内的 `HeadlessTerminal` 实现与进程终端相同的 pi-tui `Terminal` 接口,并把每次 ANSI 写入交给固定版本的 `@xterm/headless` 解析器。快照会等待 pi-tui 的同步输出结束标记,再读取状态。因此,每个检查点表示已经完成的帧,而不是依赖计时的写入前缀
可运行 TUI 在 `examples/tui-agent` 中拥有独立叶节点,与 readline `coding-agent``acp-agent` 叶节点并列。它通过带断言的 include patch 复用 coding agent 的后端与工具组合,只把共享终端应用固定为 `ui.mode: tui`TUI 快照和 PTY 测试也归属这个叶节点
每份金标把终端状态投影为文本:尺寸、活动缓冲区和视口坐标、生命周期与光标状态、各行、换行标记,以及非默认样式区间。滚动内容较多的卡片捕获已使用缓冲区;浮层捕获可见视口。文本和样式相互分离,评审人无需解码 ANSI 字节即可区分内容变化与呈现变化。
### 已录制会话回放
每个检查点还会对完整终端状态强制执行主题无关性:禁止 RGB 颜色、禁止 ANSI 015 以外的调色板项,也禁止显式背景色。选择行使用终端默认色进行反显,因此仍然有效。测试套件拥有封闭的检查点清单:类型会拒绝未声明的名称,清单检查会拒绝缺失的检查点和遗留的 `.golden.txt` 文件
每个示例级场景目录都包含 `session.jsonl`、可选的子会话日志 `session.<n>.jsonl`,以及 `terminal.golden.txt`。主日志提供用户来源的 `user/message` 提示词和已录制的 `assistant/chunk` 序列。`dsh-llm-replay` 为每个会话派生一份模型调用脚本,并将子日志绑定到新建的子会话;这是测试中唯一的 mock 边界。agent loop、bash 与文件系统实现、Code Mode worker、subagent 提供方、工作流 worker、Cordis 工具、呈现器和 TUI 都使用生产实现
如果工具调用顺序不符、预期事件数量不足、工具结果报错、轮次以错误结束、工作流生命周期不完整,或者实时子会话数量与 fixture测试前置数据集合不一致测试都会失败。即使终端金标表面正确这些断言也能阻止失败或被绕过的生产路径混入结果。
真实模型 fixture 通过 `DSH_SNAPSHOT=record` 更新;录制模式会重写其主会话与子会话 JSONL 日志以及终端金标。确定性的 Cordis 工具链保留一份人工编写的完整 JSONL 脚本,因为要求真实模型稳定经过五个指定工具边界和两个子会话并不是可靠的录制契约。`DSH_SNAPSHOT=refresh` 会无密钥回放所有已提交脚本,并且只重写派生的终端金标。普通回放只比较而不写入,未知模式值会快速失败。
### 语义终端投影
包内的 `HeadlessTerminal` 实现与进程终端相同的 pi-tui `Terminal` 接口,并把每次 ANSI 写入交给固定版本的 `@xterm/headless` 解析器。读取状态前,快照代码会等待同步帧稳定,因此每个检查点表示已经完成的画面,而不是依赖计时的写入前缀。
每份金标把终端尺寸、活动缓冲区和视口坐标、生命周期与光标状态、各行、换行标记以及非默认样式区间投影为文本。滚动内容较多的卡片捕获已使用缓冲区;浮层捕获可见视口。文本和样式相互分离,评审人无需解码 ANSI 字节即可区分内容变化与呈现变化。
每个检查点还会对完整终端状态强制执行主题无关性:禁止 RGB 颜色、禁止 ANSI 015 以外的调色板项,也禁止显式背景色。选择行使用终端默认色进行反显,因此仍然有效。两套测试都拥有封闭清单,会拒绝缺失的场景、缺失的检查点和遗留金标文件。
### 必需场景矩阵
| 范围 | 代表性检查点 | 固定的契约 |
| 层次 | 场景 | 固定的契约 |
|---|---|---|
| 会话 | 回放、流式输出、完成 | 恢复后的 Markdown 与推理、实时增量、计划、token 用量,以及达到 token 上限时的完成状态 |
| Code Mode | `run_code` 待完成与已完成 | 生产 Code Mode 注册表与呈现器、源程序、捕获日志和结果 |
| 动态工作流 | `workflow` 待完成与已完成 | 生产工作流呈现器、元数据、阶段、并行 agent、脚本和结构化结果 |
| Cordis 工具 | inspect/mount/unmount 待完成与已完成 | 生产 `cordis_inspect``cordis_mount``cordis_unmount` 呈现器及其生命周期结果 |
| 高级工具卡片 | 折叠与展开 | 终端、diff、通用、subagent、后台任务和 skill 卡片形态,以及输出截断 |
| 交互 | 问题与校验 | 受限多选浮层的组合、焦点、滚动、选择和校验错误 |
| 表层与布局 | 压缩前、窄幅替换、宽幅替换 | 表层替换会移除退役内容;调整尺寸只会重排保留的表层,不会让旧内容重新出现 |
| 失败与关闭 | 错误与帮助、终端已释放 | 帮助与未知命令、实时错误和轮次错误去重、中断、光标恢复及终端停止 |
面向模型的高级场景明确使用真实 `ToolRegistry` 配置,以及生产 Code Mode、工作流和 Cordis 工具呈现器。只有通用卡片形态矩阵使用合成呈现器 fixture在这里呈现器视图本身就是 TUI 的输入契约。测试仍由会话事件驱动,因此回放、流式输出、结果到达、表层替换和生命周期顺序都会经过与生产环境相同的投影路径。
`vitest.snapshot.config.ts` 会包含 TUI 测试套件,因此 `pnpm run test:snapshot` 可以无密钥比较快照。`pnpm run test:snapshot:refresh` 会重写从终端状态派生的金标,而不会联系模型;对于 transcript文本记录来源需要录制的测试套件`test:snapshot:record` 仍有其原有含义。两条刷新路径都会在同一次运行中继续比较生成后的文件。
| 已录制流程 | 多轮会话 | 已录制的推理与文本分片、两轮输入、保留历史、token 总量和空闲编辑器状态 |
| 已录制流程 | Todo 计划 | 真实 `todo_write` 执行、结果卡片和持久计划渲染 |
| 已录制流程 | Bash 终端卡片 | 真实本地执行器输出、说明、退出状态和已完成终端卡片 |
| 已录制流程 | 并行文件读取 | 同一条 assistant 消息中的两次调用、真实文件内容、顺序和两个独立完成卡片 |
| 已录制流程 | Code Mode | 真实 `run_code` worker 执行、两条 `tool/code-dispatch` 事件、捕获的程序输出和已完成卡片 |
| 已录制流程 | 动态工作流 | 真实工作流 worker、阶段生命周期、回放的子会话、结构化返回值和已完成卡片 |
| 已录制流程 | Cordis 动态工具链 | 真实挂载、Code Mode 检查、直接 subagent、工作流子会话、卸载和全部生产呈现器 |
| 瞬态 | 流式输出与待完成高级调用 | 进行中的推理和文本,以及完整日志中不会保留的待完成 Code Mode、工作流和 Cordis 卡片 |
| 瞬态 | 卡片、交互、布局、失败和关闭 | 折叠与展开的卡片族、问题校验、压缩替换、尺寸重排、帮助与错误、光标恢复和终端停止 |
## 曾考虑的替代方案
- **快照原始终端写入**:不予采纳,因为差分渲染可能在画面不变时改变写入边界,而且光标与清屏序列难以评审。
- **快照进入终端输出之前的组件渲染行**:不予采纳,因为它无法测试 ANSI 解析、光标移动、浮层、视口行为,也无法测试独立组件在同一帧中的相互作用。
- **通过追加会话事件构造所有完整流程**:不予采纳,因为人工编写的事件序列可能与 agent loop、工具执行、子会话绑定或 worker 行为发生偏差,但呈现测试仍然保持绿色。直接构造事件只用于渲染器瞬态。
- **复用 ACP stdout 金标作为 TUI 判定依据**不予采纳因为已录制模型流程与传输方式无关其呈现方式却并非如此。TUI 场景使用同一套 JSONL 回放词汇,但拥有独立的终端金标。
- **提交栅格截图**:不予采纳,因为字体、字形度量、抗锯齿和宿主终端主题会使结果依赖平台,也会增加语义样式变更的评审难度。
- **只使用 PTY 端到端测试**:不予采纳,因为原始 PTY 输出是一系列历史绘制操作而不是可查询的最终状态。PTY 测试保留真实 Loader、输入与清理边界模拟器负责广泛的状态覆盖。
- **复制 pi-tui 未发布的虚拟终端测试 helper**:不予采纳,因为已安装的包并未导出该 helper。围绕公开 `@xterm/headless` API 编写小型适配器,可以显式声明依赖,并让本包拥有状态投影。
## 后果
- TUI 视觉回归会产生便于阅读的单元格和样式 diff必需场景矩阵也让高级功能成为一等测试对象而不是偶然覆盖
- 测试依赖固定到 pi-tui 使用的 xterm 版本。适配器使用 xterm 的拟议缓冲区 API因此升级 xterm 时必须重新运行并评审语义投影
- 模拟器可以建模 ANSI 终端状态,但无法证明每种终端实现独有的行为。真实 PTY 会话覆盖进程选择、键盘输入、用户交互和清理,无需复制完整矩阵
- 金标有意固定指定尺寸下的换行与视口行为。布局的预期变更通过无密钥刷新命令更新,并接受常规快照评审
- 当真实 Code Mode、工作流、subagent、文件系统、bash 或 Cordis 路径损坏时,已完成高级快照会失败,不会继续接受伪造的结果事件
- TUI 视觉回归会产生便于阅读的单元格和样式 diff而 JSONL fixture 会保留触发生产路径的确切模型分片
- 模拟器使用 xterm 的拟议缓冲区 API。升级 xterm 时必须重新运行并评审语义投影;终端特有行为仍需由 PTY 冒烟测试覆盖
- 金标有意固定指定尺寸下的换行与视口行为。预期布局变更使用无密钥刷新;模型流程变更使用录制模式,并同时评审 JSONL 与终端 diff

View File

@@ -7,7 +7,7 @@ How this repo tests, tier by tier, and the rules that keep a green suite meaning
- **Unit** (`pnpm run test`): vitest over `packages|examples/*/tests/**/*.spec.ts`, colocated with what they test. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Prefer edge cases, error paths, event ordering, concurrency races, and permanent contract regressions (see `packages/core/agent-loop/tests/contract-regressions.spec.ts`).
- **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped.
- **Real-API e2e** (`pnpm run test:e2e`): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (`EXA_API_KEY`, `PERPLEXITY_API_KEY`, …); each suite self-skips without its key so keyless CI stays green ([real-API e2e RFC](rfc/implemented/testing/2026-06-19-real-api-e2e-ci.md)).
- **Snapshot** (`pnpm run test:snapshot`): transport-specific keyless goldens cover external presentation. ACP suites boot the real example subprocess, replay a recorded session, and diff normalized stdout plus the re-persisted log ([ACP snapshot RFC](rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md)); the TUI suite parses real ANSI output into semantic terminal-state goldens and retains a real PTY conversation at the process boundary ([TUI snapshot RFC](rfc/implemented/testing/2026-07-18-tui-terminal-state-snapshots.md)). Use `pnpm run test:snapshot:record` when a model transcript must change and `pnpm run test:snapshot:refresh` when the committed transcript or scripted events remain correct; review every golden diff. System-prompt/tool-schema content is pinned by ONE ACP scenario (`text-turn`) and tokenized in every other fixture, so a prompt or schema edit churns one committed line ([pinned-header RFC](rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **Snapshot** (`pnpm run test:snapshot`): transport-specific keyless goldens cover external presentation. ACP suites boot the real example subprocess, replay a recorded session, and diff normalized stdout plus the re-persisted log ([ACP snapshot RFC](rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md)). TUI completed journeys replay recorded primary/child JSONL through the real agent loop and tools before projecting ANSI into semantic terminal-state goldens; package-local snapshots retain transient renderer states, and a real PTY conversation covers the process boundary ([TUI snapshot RFC](rfc/implemented/testing/2026-07-18-tui-terminal-state-snapshots.md)). Use `pnpm run test:snapshot:record` when a model transcript must change and `pnpm run test:snapshot:refresh` when committed replay input remains correct; review every JSONL and golden diff. System-prompt/tool-schema content is pinned by ONE ACP scenario (`text-turn`) and tokenized in every other fixture, so a prompt or schema edit churns one committed line ([pinned-header RFC](rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
## The with-key policy: inference is cheap here
@@ -35,4 +35,4 @@ An e2e assertion re-runs the command or re-reads the file externally; a keyword
## When a snapshot test is required
Any change affecting an editor-facing transcript or end-to-end agent UX adds or updates a scenario in the owning snapshot suite, or states in the PR why none applies. ACP surfaces use `examples/<name>/tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/support/acp-snapshot/README.md) suite factory (`examples/acp-agent` is primary); interactive-terminal presentation uses the semantic TUI matrix, with a PTY case when input, Loader selection, or terminal teardown changes. New capability seams, lifecycle shapes, or transcript surfaces name their coverage at every tier at plan time and verify the harness can express it — a harness gap is scheduled work, not a mid-build surprise.
Any change affecting an editor-facing transcript or end-to-end agent UX adds or updates a scenario in the owning snapshot suite, or states in the PR why none applies. ACP surfaces use `examples/<name>/tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/support/acp-snapshot/README.md) suite factory (`examples/acp-agent` is primary). Completed interactive-terminal journeys use JSONL-driven scenarios under `examples/tui-agent/tests/snapshots/`; transient presentation uses the package-local semantic matrix, with a PTY case when input, Loader selection, or terminal teardown changes. New capability seams, lifecycle shapes, or transcript surfaces name their coverage at every tier at plan time and verify the harness can express it — a harness gap is scheduled work, not a mid-build surprise.