Merge remote-tracking branch 'refs/remotes/origin/master' into worktree/retarget-pr961-20260808
# Conflicts: # .agents/notes/implemented/bug-fix/2026-07-27-stable-snapshot-refresh-volatiles.i18n.yaml # .agents/notes/implemented/bug-fix/2026-07-27-stable-snapshot-refresh-volatiles.zh.md # apps/web/tests/scaffold.ts # examples/tui-agent/tests/tui.snapshot.ts # packages/support/acp-snapshot/README.i18n.yaml # packages/support/acp-snapshot/README.md # packages/support/acp-snapshot/README.zh.md
This commit is contained in:
Tianyi Cui
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/support/README.md
|
||||
README.md: b9550fd54feb36448227faae8485fe8b6dbf4fb0
|
||||
README.zh.md: 8eec9b96f621724e1ddc9a70abed069f11cb7fdc
|
||||
README.md: 15cb82d3d76a1241af8b40e1c8536292618409a9
|
||||
README.zh.md: ff8595b47f1a1220912dc77a072463adf74f7bf9
|
||||
|
||||
@@ -1,16 +1,16 @@
|
||||
# support/ — dev/test/example infrastructure
|
||||
# support/ — development and test infrastructure
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
Packages that exist to serve development, testing, and the examples rather than to ship as product API. They are real workspace packages (typed, tested, under the coverage gate), but they carry **lower compatibility expectations**: they may change or be removed when the development need behind them does, without the deprecation care a product package would warrant.
|
||||
These packages support repository development, tests, and examples rather than product APIs. Their compatibility follows the development need they serve.
|
||||
|
||||
| Package | Role | ctx key |
|
||||
|---|---|---|
|
||||
| `acp-snapshot/` | ACP test kit: shared subprocess/client launcher + snapshot harness, normalizers, and suite factory | (library — imported by ACP e2e and `*.snapshot.ts` suites) |
|
||||
| `agent-loop-testkit/` | Shared prerequisite mounting for tests that exercise the concrete agent loop | (library — imported by AgentLoop integration tests) |
|
||||
| `invariants/` | Runtime event-contract assertions for development diagnostics | (listens on `session/*`, `agent/*`) |
|
||||
| `loader-smoke/` | Shared real-Loader subprocess harness for keyless example smokes | (library — imported by example e2e suites) |
|
||||
| `llm-mock-server/` | Scriptable OpenAI-compatible HTTP/SSE fault server + CLI for LLM recovery tests | (standalone server and test library) |
|
||||
| `llm-replay/` | Record/replay adapter: short-circuits `llm/stream` from a recorded session JSONL (keyless snapshot tests) | (listens on `llm/stream`) |
|
||||
| Package | Role |
|
||||
|---|---|
|
||||
| [`acp-snapshot/`](acp-snapshot/README.md) | Provides the ACP snapshot-test toolkit |
|
||||
| [`agent-loop-testkit/`](agent-loop-testkit/README.md) | Mounts shared prerequisites for AgentLoop tests |
|
||||
| [`invariants/`](invariants/README.md) | Runs development-time runtime-contract assertions |
|
||||
| [`loader-smoke/`](loader-smoke/README.md) | Launches Loader-composed applications for smoke tests |
|
||||
| [`llm-mock-server/`](llm-mock-server/README.md) | Provides a deterministic OpenAI-compatible fault server |
|
||||
| [`llm-replay/`](llm-replay/README.md) | Replays recorded model responses for keyless tests and demos |
|
||||
|
||||
`invariants` is development support but has no environment guard: it runs wherever registered, and the default `dsh-agent-spine-demo` bundle mounts it unconditionally. `agent-loop-testkit` centralizes the mandatory service spine for hand-built AgentLoop tests without owning their loop or scenario. `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate, while `llm-mock-server` drives real provider adapters through deterministic HTTP/SSE faults. `acp-snapshot` carries the ACP subprocess/client boundary plus the snapshot harness, normalizers, and suite machinery, while `loader-smoke` owns the parallel real-Loader launch boundary used by keyless example e2e suites. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.
|
||||
A package moves out of `support/` when it gains a product contract and product consumers.
|
||||
|
||||
@@ -1,16 +1,16 @@
|
||||
# support/：开发／测试／示例基础设施
|
||||
# support/：开发和测试基础设施
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
这些包（package）用于开发、测试和示例，而非作为产品 API 发布。它们是实际的工作区包（具备类型、经过测试，并受覆盖率门禁约束），但具有**较低的兼容性预期**：当其背后的开发需求变化时，它们可以改变或被移除，无需像产品包那样谨慎执行弃用流程。
|
||||
这些包为仓库开发、测试和示例提供支持，而不是产品 API。其兼容性取决于所服务的开发需求。
|
||||
|
||||
| 包 | 职责 | ctx 键 |
|
||||
|---|---|---|
|
||||
| `acp-snapshot/` | ACP（Agent Client Protocol）测试工具包：共享子进程/客户端启动器、快照 harness、规范化器和套件工厂 | （库：由 ACP e2e 和 `*.snapshot.ts` 套件导入） |
|
||||
| `agent-loop-testkit/` | 为验证具体 agent loop（智能体循环）的测试挂载共享先决条件 | （库：由 AgentLoop 集成测试导入） |
|
||||
| `invariants/` | 用于开发诊断的运行时事件契约断言 | （监听 `session/*`、`agent/*`） |
|
||||
| `loader-smoke/` | 共享的真实 Loader 子进程 harness，用于无密钥示例冒烟测试 | （库：由示例 e2e 套件导入） |
|
||||
| `llm-mock-server/` | 可编程的 OpenAI 兼容 HTTP/SSE（Server-Sent Events）故障服务器与 CLI（命令行界面），用于 LLM（大语言模型）恢复测试 | （独立服务器和测试库） |
|
||||
| `llm-replay/` | 录制/回放适配器：通过已记录的会话 JSONL 对 `llm/stream` 进行短路处理（无密钥快照测试） | （监听 `llm/stream`） |
|
||||
| 包 | 职责 |
|
||||
|---|---|
|
||||
| [`acp-snapshot/`](acp-snapshot/README.md) | 提供 ACP（Agent Client Protocol）快照测试工具包 |
|
||||
| [`agent-loop-testkit/`](agent-loop-testkit/README.md) | 为 AgentLoop 测试挂载共享先决条件 |
|
||||
| [`invariants/`](invariants/README.md) | 运行开发期运行时契约断言 |
|
||||
| [`loader-smoke/`](loader-smoke/README.md) | 启动由 Loader 组合的应用以执行冒烟测试 |
|
||||
| [`llm-mock-server/`](llm-mock-server/README.md) | 提供确定性的 OpenAI 兼容故障服务器 |
|
||||
| [`llm-replay/`](llm-replay/README.md) | 为无密钥测试和演示回放已记录的模型响应 |
|
||||
|
||||
`invariants` 是开发支持，但没有环境条件限制：无论在何处注册，它都会运行；默认 `dsh-agent-spine-demo` bundle 无条件挂载它。`agent-loop-testkit` 为手工构建的 AgentLoop 测试集中管理必需服务主干，而不负责其 agent loop 或场景。`llm-replay` 支撑演示和受逐文件覆盖率门禁约束的快照测试层，`llm-mock-server` 则通过确定性 HTTP/SSE 故障驱动真实提供方适配器。`acp-snapshot` 包含 ACP 子进程/客户端边界以及快照 harness、规范化器和套件机制，`loader-smoke` 负责无密钥示例 e2e 套件使用的并列真实 Loader 启动边界。只有当某个包获得文档记载的产品消费方时，它才会从 `support/` 转入产品分组。
|
||||
当一个包获得产品契约和产品消费方时，它会移出 `support/`。
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/support/acp-snapshot/README.md
|
||||
README.md: c5a1a07b9f85e1a91c52fe102be17e4172be112d
|
||||
README.zh.md: 2454df9b3d8e67b4728d6582279ba21798d2ba9e
|
||||
README.md: 9d142dc964e60f9508b6c137525eb916cdbd969f
|
||||
README.zh.md: 2e88e7c5bb8acb0cd99a35b5fe0fbfc15d6101e3
|
||||
|
||||
@@ -8,8 +8,8 @@ Four layers, importable separately:
|
||||
|
||||
- **`launchAcpTestAgent` (launcher)** — boots a source agent under tsx or a built `lib` agent under plain Node from a supplied cwd, connects the SDK client over a raw-byte stdout tee, collects session updates and stderr, surfaces asynchronous spawn failures through startup, fails closed on unhandled permission requests, and owns graceful or signalled shutdown. Shutdown waits for process exit, inherited stdio closure, and ACP parser exhaustion before resolving or propagating a child error, so captures are complete and callers can remove owned paths after either outcome. When Windows accepts forced termination but publishes its exit marker asynchronously, shutdown gives that marker a bounded grace before treating fallback refusal as a second failure. Snapshot and ordinary e2e suites share this process boundary; a test supplies only agent paths, cwd, environment overrides, and any permission policy.
|
||||
- **`runScenario` (harness)** — drives ACP JSON-RPC stdio from a deterministic `input.json` script through the launcher, tees raw stdout for the expected-output and purity checks, and harvests every persisted raw JSONL session log (parent and subagent children, primary-first) after graceful stdin EOF. `AgentUnderTest` supplies absolute `binScript`, optional `libBinScript`, `configPath`, and `tsconfigPath` paths because the subprocess cwd is outside the repo; `workspaceParent` may move the generated child cwd from the platform temp directory when that grant is itself under test. Startup failures preserve captured agent stderr in the rejected diagnostic.
|
||||
- **Normalizers** — pure functions turning captured surfaces into stable text or portable fixtures: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs and every native/JavaScript filesystem spelling of the generated cwd → tokens, longest-first; cwd-rooted separators selected as canonical `/` or host-native; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept, the same cwd-path policy), `tokenizeSessionFixtureCwd` (the generated workspace and its filesystem aliases → `{{cwd}}`, authored temp paths unchanged), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header Agent Note](../../../.agents/notes/archived/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)), and `stabilizeFixtureMessageIds` (committed UUIDs carried into unchanged, unambiguous messages across any recorder's fixture-ready parent/child logs).
|
||||
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario expected-output and re-persisted-log comparisons, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, a tokenized pin per header class composed with independently shared `system-prompt.expected.md` and `tool-schemas.expected.json` sidecars, and a live uniformity guard. Its fixture guards reject orphan scenario dirs, missing files, multiple pins for one class, duplicate sidecar content, unscrubbed JSONL headers, and malformed pinning headers. Before record or refresh writes fixtures, an unchanged complete message retains its committed UUID when its identity-free value resolves to exactly one fresh ID and one existing ID across the scenario's parent/child logs; new, changed, and ambiguous messages keep fresh UUIDs. Refresh evaluates fresh leaves with the harvested run's ids, cwd, and every cwd alias, then reuses normalized-equivalent leaves only when the complete logical-record layout aligns and volatile string replacements form a bijection; ambiguous logs keep fresh strings, and fresh semantic values remain authoritative. It also expands packed timing envelopes before aligning event times, so switching between packed and unpacked layouts cannot shift later records. A newly inserted `session/title` receives its preceding event's time so feature-driven insertions do not churn the remainder of a fixture. Each scenario directory's `session.jsonl` plus contiguous `session.<n>.jsonl` siblings are the ordered primary/child inventory; the scenario table does not duplicate their count. Must be called at vitest collection time.
|
||||
- **Normalizers** — pure functions turning captured surfaces into stable text or portable fixtures: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs and every native/JavaScript filesystem spelling of the generated cwd → tokens, longest-first; cwd-rooted separators selected as canonical `/` or host-native; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept, the same cwd-path policy), `tokenizeSessionFixtureCwd` (the generated workspace and its filesystem aliases → one canonical `{{cwd}}`, including an already-tokenized macOS `/private` alias; authored temp paths unchanged), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header Agent Note](../../../.agents/notes/archived/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)), and `stabilizeFixtureMessageIds` (committed UUIDs carried into unchanged, unambiguous messages across any recorder's fixture-ready parent/child logs).
|
||||
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario expected-output and re-persisted-log comparisons, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, a tokenized pin per header class composed with independently shared `system-prompt.expected.md` and `tool-schemas.expected.json` sidecars, and a live uniformity guard. Its fixture guards reject orphan scenario dirs, missing files, multiple pins for one class, duplicate sidecar content, noncanonical macOS-prefixed cwd tokens, unscrubbed JSONL headers, and malformed pinning headers. Before record or refresh writes fixtures, an unchanged complete message retains its committed UUID when its identity-free value resolves to exactly one fresh ID and one existing ID across the scenario's parent/child logs; new, changed, and ambiguous messages keep fresh UUIDs. Refresh evaluates fresh leaves with the harvested run's ids, cwd, and every cwd alias, then reuses normalized-equivalent leaves only when the complete logical-record layout aligns and volatile string replacements form a bijection; ambiguous logs keep fresh strings, and fresh semantic values remain authoritative. It also expands packed timing envelopes before aligning event times, so switching between packed and unpacked layouts cannot shift later records. A newly inserted `session/title` receives its preceding event's time so feature-driven insertions do not churn the remainder of a fixture. Each scenario directory's `session.jsonl` plus contiguous `session.<n>.jsonl` siblings are the ordered primary/child inventory; the scenario table does not duplicate their count. Must be called at vitest collection time.
|
||||
|
||||
Committed session fixtures use canonical packed rows. An in-flight branch that merges this contract runs the [temporary repository migrator](../../../scripts/migrate-packed-session-fixtures.ts) with `pnpm run migrate:packed-session-fixtures`; its [removal proposal](../../../.agents/notes/proposed/process/2026-07-26-remove-packed-session-fixture-migrator.md) owns deletion after affected branches converge.
|
||||
|
||||
@@ -55,11 +55,11 @@ A scenario booting a differently-composed tree sets its own `configPath` (an ove
|
||||
|
||||
A pin owns its generated `system-prompt.expected.md` or `tool-schemas.expected.json` by default; `systemPromptSource` and `toolSchemasSource` name another pin when the complete corresponding sequence is identical, so each distinct version is committed once. The pin's `session.jsonl` stores `"system":"{{system}}","tools":"{{tools}}"` while retaining config, reason, and any model-visible prefix. A pin with legitimate mid-run header changes declares `expectedHeaderChanges`; a shared source must declare the same count, and record/refresh rejects claimants that generate different bytes.
|
||||
|
||||
Every scenario compares `stdout.expected.jsonl` with cwd-rooted separators canonicalized to `/`. On Windows, `pinsNativeWindowsStdout` additionally compares the complete `stdout.expected.windows.jsonl` after the shared expected output and requires that sidecar exactly when enabled. A scenario requiring a non-Windows host declares `posixOnly`, which skips its run test on Windows while the fixture guards keep covering its committed files everywhere; examples include POSIX process semantics (e.g. cancelling a live bash call kills a detached process group) and generated paths Windows cannot represent.
|
||||
Every scenario compares `stdout.expected.jsonl` with cwd-rooted separators canonicalized to `/`. On Windows, `pinsNativeWindowsStdout` additionally compares the complete `stdout.expected.windows.jsonl` after the shared expected output and requires that sidecar exactly when enabled. A scenario requiring a non-Windows host declares `posixOnly`, which skips its run test on Windows while the fixture guards keep covering its committed files everywhere; examples include POSIX process semantics (e.g. cancelling a live bash call kills a detached process group) and generated paths Windows cannot represent. A scenario whose composition needs a usable `pwsh` declares `pwshOnly`; the caller-supplied `hasPwsh` probe (the shipped acp-agent suite follows the executor's own resolution, so Program Files installs count) skips the run test when no usable `pwsh` resolves while the fixture guards keep covering its committed files everywhere.
|
||||
|
||||
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config Agent Note](../../../.agents/notes/archived/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log expected outputs, and owned prompt and tool-schema sidecars from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md).
|
||||
|
||||
Constraints: `suite.ts` and `harness.ts` import vitest (the harness polls its durable-boundary waits through `vi.waitFor`), so the package entry is importable only inside a vitest run (the launcher and normalizers have no such dependency but ship from the same entry). The launcher and suite factory are ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection` — while the normalizers are transport-neutral session-log/text helpers also consumed by the JSON-RPC, TUI, and web snapshot recorders. Input scripts cover initialization, fresh-session creation, text prompting, cancellation, expected RPC failures, and durable turn-boundary waits. Permission round-trips are a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) mapped to the agent-issued `optionId`; an absent or exhausted queue answers `cancelled`, and an unoffered kind rejects the run.
|
||||
Constraints: `suite.ts` and `harness.ts` import vitest (the harness polls its durable-boundary waits through `vi.waitFor`), so the package entry is importable only inside a vitest run (the launcher and normalizers have no such dependency but ship from the same entry). The launcher and suite factory are ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection` — while the normalizers are transport-neutral session-log/text helpers also consumed by the JSON-RPC and Web snapshot recorders. Input scripts cover initialization, fresh-session creation, text prompting, cancellation, expected RPC failures, and durable turn-boundary waits. Permission round-trips are a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) mapped to the agent-issued `optionId`; an absent or exhausted queue answers `cancelled`, and an unoffered kind rejects the run.
|
||||
|
||||
## Model Experience
|
||||
|
||||
|
||||
@@ -8,8 +8,8 @@ ACP 快照套件工具包：无密钥快照层（`pnpm run test:snapshot`，见[
|
||||
|
||||
- **`launchAcpTestAgent`（启动器）**：从指定 cwd 在 tsx 下启动源 agent，或在普通 Node 下启动已构建 `lib` agent；通过原始字节 stdout tee 连接 SDK 客户端，收集会话更新和 stderr，在启动过程中公开异步 spawn 失败，对未处理权限请求快速失败，并负责优雅或带信号关闭。关闭会等待进程退出、继承 stdio 关闭和 ACP parser 耗尽，然后才解析或传播子级错误，使捕获内容完整，且调用方可在任一结果后移除自有路径。当 Windows 接受强制终止但异步发布退出标记时，关闭会给该标记有界宽限，然后才将回退拒绝视为第二次失败。快照和普通 e2e 套件共享该进程边界；测试只需提供 agent 路径、cwd、环境覆盖和任何权限策略。
|
||||
- **`runScenario`（harness）**：通过启动器从确定性 `input.json` 脚本驱动 ACP JSON-RPC stdio，将原始 stdout tee 给预期输出和纯度检查，并在优雅 stdin EOF 后收集每个持久化原始 JSONL 会话日志（父级和 subagent 子级，主级优先）。`AgentUnderTest` 提供绝对 `binScript`、可选 `libBinScript`、`configPath` 和 `tsconfigPath` 路径，因为子进程 cwd 位于仓库外。当生成子级 cwd 自身位于待测授权中时，`workspaceParent` 可以将它从平台临时目录移出。启动失败会在拒绝诊断中保留已捕获 agent stderr。
|
||||
- **规范化器**：将已捕获接口转换为稳定文本或可移植 fixture 的纯函数：`normalizeStdout`（JSON-RPC id → 首次出现序列；UUID 以及生成 cwd 的每个原生/JavaScript 文件系统写法 → token，按最长优先；根据 cwd 的分隔符选择规范 `/` 或宿主原生形式；同时作为 stdout 纯度检查）、`normalizeSessionLog`（时间归零、保留 `seq`、使用同一 cwd 路径策略）、`tokenizeSessionFixtureCwd`（生成的 workspace 及其文件系统别名 → `{{cwd}}`，手工编写的临时路径保持不变）、`scrubSystemPrompts`（提示词文本 → `{{system}}`）、`scrubToolSchemas`（schema bulk → `{{tools}}`）、`scrubRequestHeaders`（每个 pin 之外的所有 header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}`，保留结构；见[header 固定 Agent Note](../../../.agents/notes/archived/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)）和 `stabilizeFixtureMessageIds`（针对任意录制器已准备写入 fixture 的父级/子级日志，将已提交 UUID 带入未变化且无歧义的消息）。
|
||||
- **`defineAcpSnapshotSuite`（工厂）**：为场景表注册完整 describe/it 树：每场景预期输出与重新持久化日志比较、录制/刷新 fixture 回写、拒绝结构化 `UNKNOWN_TOOL` 结果、每个 header 类别一个 token 化 pin（由可独立共享的 `system-prompt.expected.md` 和 `tool-schemas.expected.json` sidecar 组合而成），以及实时一致性保护。其 fixture 保护会拒绝遗留场景目录、缺失文件、一个类别包含多个 pin、重复的 sidecar 内容、未擦除的 JSONL header，以及格式错误的 pin header。在录制或刷新写入 fixture 前，如果一条未变化的完整消息去除身份后的值在场景的父级/子级日志中恰好对应一个本次生成的 ID 和一个现有 ID，它就会保留已提交的 UUID；新增、发生变化和有歧义的消息则保留本次生成的 UUID。刷新会使用收集所得本次运行的 id、cwd 及全部 cwd 别名评估本次生成的叶值；只有完整逻辑记录布局对齐且易变字符串替换形成双射时，才会复用规范化后等价的叶值；有歧义的日志保留本次生成的字符串，而本次生成的语义值仍为权威数据。它还会在对齐事件时间前展开打包时序 envelope，因此切换打包/非打包布局无法移动后续记录。新插入的 `session/title` 使用前一个事件的时间，因此功能驱动的插入不会扰动 fixture 余下部分。每个场景目录的 `session.jsonl` 和连续 `session.<n>.jsonl` 同级文件是有序主级/子级清单；场景表不重复其数量。必须在 vitest 收集时调用。
|
||||
- **规范化器**：将已捕获接口转换为稳定文本或可移植 fixture 的纯函数：`normalizeStdout`（JSON-RPC id → 首次出现序列；UUID 以及生成 cwd 的每个原生/JavaScript 文件系统写法 → token，按最长优先；根据 cwd 的分隔符选择规范 `/` 或宿主原生形式；同时作为 stdout 纯度检查）、`normalizeSessionLog`（时间归零、保留 `seq`、使用同一 cwd 路径策略）、`tokenizeSessionFixtureCwd`（生成的 workspace 及其文件系统别名，包括已 token 化的 macOS `/private` 别名 → 单一规范 `{{cwd}}`；手工编写的临时路径保持不变）、`scrubSystemPrompts`（提示词文本 → `{{system}}`）、`scrubToolSchemas`（schema bulk → `{{tools}}`）、`scrubRequestHeaders`（每个 pin 之外的所有 header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}`，保留结构；见[header 固定 Agent Note](../../../.agents/notes/archived/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)）和 `stabilizeFixtureMessageIds`（针对任意录制器已准备写入 fixture 的父级/子级日志，将已提交 UUID 带入未变化且无歧义的消息）。
|
||||
- **`defineAcpSnapshotSuite`（工厂）**：为场景表注册完整 describe/it 树：每场景预期输出与重新持久化日志比较、录制/刷新 fixture 回写、拒绝结构化 `UNKNOWN_TOOL` 结果、每个 header 类别一个 token 化 pin（由可独立共享的 `system-prompt.expected.md` 和 `tool-schemas.expected.json` sidecar 组合而成），以及实时一致性保护。其 fixture 保护会拒绝遗留场景目录、缺失文件、一个类别包含多个 pin、重复的 sidecar 内容、带非规范 macOS 前缀的 cwd token、未擦除的 JSONL header，以及格式错误的 pin header。在录制或刷新写入 fixture 前，如果一条未变化的完整消息去除身份后的值在场景的父级/子级日志中恰好对应一个本次生成的 ID 和一个现有 ID，它就会保留已提交的 UUID；新增、发生变化和有歧义的消息则保留本次生成的 UUID。刷新会使用收集所得本次运行的 id、cwd 及全部 cwd 别名评估本次生成的叶值；只有完整逻辑记录布局对齐且易变字符串替换形成双射时，才会复用规范化后等价的叶值；有歧义的日志保留本次生成的字符串，而本次生成的语义值仍为权威数据。它还会在对齐事件时间前展开打包时序 envelope，因此切换打包/非打包布局无法移动后续记录。新插入的 `session/title` 使用前一个事件的时间，因此功能驱动的插入不会扰动 fixture 余下部分。每个场景目录的 `session.jsonl` 和连续 `session.<n>.jsonl` 同级文件是有序主级/子级清单；场景表不重复其数量。必须在 vitest 收集时调用。
|
||||
|
||||
签入仓库的会话 fixture 使用规范打包行。合并此契约的在途分支通过 `pnpm run migrate:packed-session-fixtures` 运行[临时仓库迁移器](../../../scripts/migrate-packed-session-fixtures.ts)；待受影响分支收敛后，由其[移除提案](../../../.agents/notes/proposed/process/2026-07-26-remove-packed-session-fixture-migrator.md)负责删除该迁移器。
|
||||
|
||||
@@ -55,11 +55,11 @@ defineAcpSnapshotSuite({
|
||||
|
||||
每个 pin 默认拥有其生成的 `system-prompt.expected.md` 或 `tool-schemas.expected.json`；当完整的对应序列相同时，`systemPromptSource` 和 `toolSchemasSource` 指定另一个 pin 作为来源，因此每个不同版本只提交一次。该 pin 的 `session.jsonl` 存储 `"system":"{{system}}","tools":"{{tools}}"`，同时保留配置、原因和任何模型可见前缀。具有合法运行中 header 变更的 pin 声明 `expectedHeaderChanges`；共享来源必须声明相同的 header 变更数量，录制/刷新会拒绝生成不同字节的共享引用方。
|
||||
|
||||
每个场景都比较 `stdout.expected.jsonl`，其中以 cwd 为根的分隔符规范化为 `/`。在 Windows 上，`pinsNativeWindowsStdout` 还会在共享预期输出之后比较完整 `stdout.expected.windows.jsonl`，并在启用时精确要求该 sidecar。需要非 Windows 主机的场景声明 `posixOnly`，在 Windows 上跳过运行测试，但 fixture 保护仍在所有平台覆盖其已提交文件；示例包括 POSIX 进程语义（例如取消实时 bash 调用会终止脱离进程组）和 Windows 无法表示的生成路径。
|
||||
每个场景都比较 `stdout.expected.jsonl`，其中以 cwd 为根的分隔符规范化为 `/`。在 Windows 上，`pinsNativeWindowsStdout` 还会在共享预期输出之后比较完整 `stdout.expected.windows.jsonl`，并在启用时精确要求该 sidecar。需要非 Windows 主机的场景声明 `posixOnly`，在 Windows 上跳过运行测试，但 fixture 保护仍在所有平台覆盖其已提交文件；示例包括 POSIX 进程语义（例如取消实时 bash 调用会终止脱离进程组）和 Windows 无法表示的生成路径。组合需要可用 `pwsh` 的场景声明 `pwshOnly`；调用方提供的 `hasPwsh` 探测（随附的 acp-agent 套件遵循执行器自身的解析，因此 Program Files 安装也计入）在解析不到可用 `pwsh` 时跳过运行测试，而 fixture 保护仍处处覆盖其已提交文件。
|
||||
|
||||
示例还发布 `cordis.snapshot.yml` 回放 overlay，位于 `cordis.yml` 旁边（bin 在 `DSH_SNAPSHOT=replay` 下交换它们，见[单源回放配置 Agent Note](../../../.agents/notes/archived/testing/2026-07-04-single-source-acp-replay-config.md)）；回放 fixture 由 [`dsh-llm-replay`](../llm-replay/README.md) 提供，该包通过对子级设置的 `DSH_SNAPSHOT_*` env var 指向它。`pnpm run test:snapshot:record` 调用实时 LLM，并重写已记录场景的模型 fixture；`pnpm run test:snapshot:refresh` 保持无密钥，运行回放 overlay，并从已提交模型脚本重写 stdout、可比较会话日志预期输出，以及各 pin 自有的提示词与工具 schema sidecar。Fixture 角色、录制/回放/刷新语义和场景表字段记录在 `Scenario` 以及[快照 Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md) 中。
|
||||
|
||||
约束：`suite.ts` 与 `harness.ts` 导入 vitest（harness 通过 `vi.waitFor` 轮询其持久边界等待），因此包入口只能在 vitest 运行中导入（启动器和规范化器没有此依赖，但从同一入口发布）。启动器和套件工厂按设计专用于 ACP，启动器使用 SDK 的 `ClientSideConnection`；规范化器是与传输无关的会话日志/文本辅助工具，还由 JSON-RPC、TUI 和 web 快照录制器消费。输入脚本覆盖初始化、新建会话、文本提示、取消、预期 RPC 失败和持久轮次边界等待。权限往返是选项类别选择（`allow_once`、`reject_once`等）的 FIFO 队列，映射到 agent 发出的 `optionId`；缺少或耗尽的队列回答 `cancelled`，未提供类别会拒绝运行。
|
||||
约束：`suite.ts` 与 `harness.ts` 导入 vitest（harness 通过 `vi.waitFor` 轮询其持久边界等待），因此包入口只能在 vitest 运行中导入（启动器和规范化器没有此依赖，但从同一入口发布）。启动器和套件工厂按设计专用于 ACP，启动器使用 SDK 的 `ClientSideConnection`；规范化器是与传输无关的会话日志/文本辅助工具，还由 JSON-RPC 和 Web 快照录制器消费。输入脚本覆盖初始化、新建会话、文本提示、取消、预期 RPC 失败和持久轮次边界等待。权限往返是选项类别选择（`allow_once`、`reject_once` 等）的 FIFO 队列，映射到 agent 发出的 `optionId`；缺少或耗尽的队列回答 `cancelled`，未提供类别会拒绝运行。
|
||||
|
||||
## 模型体验
|
||||
|
||||
|
||||
@@ -21,9 +21,7 @@
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"lib/types/**/*.d.ts",
|
||||
"lib/types/**/*.d.ts.map",
|
||||
"src"
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
|
||||
@@ -43,14 +43,24 @@ const WAIT_POLL_INTERVAL_MS = 10
|
||||
* reference, since a committed file cannot know the id in advance.
|
||||
*
|
||||
* `promptAndCancel` starts a prompt without awaiting completion, waits for a
|
||||
* readiness condition, then cancels and awaits completion. `waitForFile`
|
||||
* observes a cwd-relative marker; the default observes the durable turn start.
|
||||
* readiness condition, then cancels and awaits completion. Its optional
|
||||
* `waitForFile` observes a cwd-relative marker; otherwise it waits for the
|
||||
* durable turn start. The standalone `waitForFile` holds the next script step
|
||||
* behind the same marker.
|
||||
* `promptAndWaitForAgentMessage` arms an exact text-chunk waiter before sending
|
||||
* the prompt, then keeps the application live until that later update arrives.
|
||||
* `waitForTurnStart` waits for an open durable turn, optionally at or beyond a
|
||||
* specified turn number. `waitForTurnEnd` holds the subprocess open until the
|
||||
* selected session's latest complete raw-JSONL turn boundary is `turn/end`.
|
||||
* `waitForGoalPhase` waits for the latest durable goal snapshot to reach one phase.
|
||||
* `waitForInboxMessage` waits for inserted inbox text containing a scenario marker.
|
||||
* `waitForSubagentTurnEnd` waits until one background child has persisted a
|
||||
* closed model-work turn after its own descriptor; child progress has no ACP
|
||||
* update to wait on.
|
||||
* `waitForTitleAfterTurnEnd` additionally waits for a later durable title.
|
||||
* `waitForEventAfterTurnEnd` waits until a complete record of the given event
|
||||
* type follows the latest closed turn — for scenarios whose asserted state
|
||||
* (e.g. a goal pause) is appended only after cancellation reaches idle.
|
||||
* A standalone `cancel` may also wait for a cwd-relative readiness marker.
|
||||
* All wait timeouts default to 10s.
|
||||
*/
|
||||
@@ -66,9 +76,14 @@ export type InputStep =
|
||||
text: string
|
||||
waitForFile?: { path: string; timeoutMs?: number }
|
||||
}
|
||||
| { op: 'waitForFile'; path: string; timeoutMs?: number }
|
||||
| { op: 'waitForTurnStart'; minimumTurn?: number; timeoutMs?: number }
|
||||
| { op: 'waitForTurnEnd'; timeoutMs?: number }
|
||||
| { op: 'waitForSubagentTurnEnd'; child?: number; minimumTurn?: number; timeoutMs?: number }
|
||||
| { op: 'waitForGoalPhase'; phase: 'active' | 'paused' | 'blocked' | 'complete'; timeoutMs?: number }
|
||||
| { op: 'waitForInboxMessage'; text: string; timeoutMs?: number }
|
||||
| { op: 'waitForTitleAfterTurnEnd'; timeoutMs?: number }
|
||||
| { op: 'waitForEventAfterTurnEnd'; type: string; timeoutMs?: number }
|
||||
| { op: 'cancel'; waitForFile?: { path: string; timeoutMs?: number } }
|
||||
|
||||
/** A scenario's `input.json`: an ordered list of input steps. */
|
||||
@@ -290,7 +305,11 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
|
||||
(id) => { sessionId = id },
|
||||
(id, timeoutMs, minimumTurn) => waitForPersistedTurnStart(sessionsRoot, id, timeoutMs, minimumTurn),
|
||||
(id, timeoutMs) => waitForPersistedTurnEnd(sessionsRoot, id, timeoutMs),
|
||||
(child, timeoutMs, minimumTurn) => waitForPersistedChildTurnEnd(sessionsRoot, child, timeoutMs, minimumTurn),
|
||||
(id, phase, timeoutMs) => waitForPersistedGoalPhase(sessionsRoot, id, phase, timeoutMs),
|
||||
(id, text, timeoutMs) => waitForPersistedInboxMessage(sessionsRoot, id, text, timeoutMs),
|
||||
(id, timeoutMs) => waitForPersistedTitleAfterTurnEnd(sessionsRoot, id, timeoutMs),
|
||||
(id, type, timeoutMs) => waitForPersistedEventAfterTurnEnd(sessionsRoot, id, type, timeoutMs),
|
||||
)
|
||||
// A permission exchange happens while a step's request is in flight, so
|
||||
// by the time the step settles any script bug it exposed is captured —
|
||||
@@ -364,7 +383,11 @@ async function runStep(
|
||||
setSessionId: (id: string) => void,
|
||||
waitForTurnStart: (sessionId: string, timeoutMs?: number, minimumTurn?: number) => Promise<void>,
|
||||
waitForTurnEnd: (sessionId: string, timeoutMs?: number) => Promise<void>,
|
||||
waitForChildTurnEnd: (child: number, timeoutMs?: number, minimumTurn?: number) => Promise<void>,
|
||||
waitForGoalPhase: (sessionId: string, phase: string, timeoutMs?: number) => Promise<void>,
|
||||
waitForInboxMessage: (sessionId: string, text: string, timeoutMs?: number) => Promise<void>,
|
||||
waitForTitleAfterTurnEnd: (sessionId: string, timeoutMs?: number) => Promise<void>,
|
||||
waitForEventAfterTurnEnd: (sessionId: string, type: string, timeoutMs?: number) => Promise<void>,
|
||||
): Promise<void> {
|
||||
switch (step.op) {
|
||||
case 'initialize':
|
||||
@@ -436,18 +459,42 @@ async function runStep(
|
||||
await promptDone
|
||||
return
|
||||
}
|
||||
case 'waitForFile':
|
||||
await waitForWorkspaceFile(cwd, step.path, step.timeoutMs)
|
||||
return
|
||||
case 'waitForTurnEnd': {
|
||||
const sessionId = getSessionId()
|
||||
if (sessionId === undefined) throw new Error('snapshot-harness: waitForTurnEnd before newSession')
|
||||
await waitForTurnEnd(sessionId, step.timeoutMs)
|
||||
return
|
||||
}
|
||||
case 'waitForSubagentTurnEnd':
|
||||
await waitForChildTurnEnd(step.child ?? 1, step.timeoutMs, step.minimumTurn)
|
||||
return
|
||||
case 'waitForGoalPhase': {
|
||||
const sessionId = getSessionId()
|
||||
if (sessionId === undefined) throw new Error('snapshot-harness: waitForGoalPhase before newSession')
|
||||
await waitForGoalPhase(sessionId, step.phase, step.timeoutMs)
|
||||
return
|
||||
}
|
||||
case 'waitForInboxMessage': {
|
||||
const sessionId = getSessionId()
|
||||
if (sessionId === undefined) throw new Error('snapshot-harness: waitForInboxMessage before newSession')
|
||||
await waitForInboxMessage(sessionId, step.text, step.timeoutMs)
|
||||
return
|
||||
}
|
||||
case 'waitForTitleAfterTurnEnd': {
|
||||
const sessionId = getSessionId()
|
||||
if (sessionId === undefined) throw new Error('snapshot-harness: waitForTitleAfterTurnEnd before newSession')
|
||||
await waitForTitleAfterTurnEnd(sessionId, step.timeoutMs)
|
||||
return
|
||||
}
|
||||
case 'waitForEventAfterTurnEnd': {
|
||||
const sessionId = getSessionId()
|
||||
if (sessionId === undefined) throw new Error('snapshot-harness: waitForEventAfterTurnEnd before newSession')
|
||||
await waitForEventAfterTurnEnd(sessionId, step.type, step.timeoutMs)
|
||||
return
|
||||
}
|
||||
case 'waitForTurnStart': {
|
||||
const sessionId = getSessionId()
|
||||
if (sessionId === undefined) throw new Error('snapshot-harness: waitForTurnStart before newSession')
|
||||
@@ -515,6 +562,95 @@ async function waitForPersistedTurnEnd(
|
||||
}, { interval: WAIT_POLL_INTERVAL_MS, timeout: timeoutMs })
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait until the Nth harvested child Session closes a model work turn.
|
||||
*
|
||||
* Harvest order matches `session.1.jsonl`, `session.2.jsonl`, and so on. A
|
||||
* continuable child appends its descriptor after any inherited history and
|
||||
* before accepting its first prompt, so only a later request header proves its
|
||||
* own model work reached a closed turn.
|
||||
*/
|
||||
async function waitForPersistedChildTurnEnd(
|
||||
root: string,
|
||||
child: number,
|
||||
timeoutMs = DEFAULT_WAIT_TIMEOUT_MS,
|
||||
minimumTurn = 1,
|
||||
): Promise<void> {
|
||||
await vi.waitFor(async () => {
|
||||
const log = (await harvestSessionLogs(root))[child]
|
||||
if (log === undefined || !latestTurnIsClosed(log.content)
|
||||
|| !hasRequestHeaderAfterDescriptor(log.content)
|
||||
|| !hasClosedTurn(log.content, minimumTurn)) {
|
||||
throw new Error(
|
||||
`snapshot-harness: subagent child #${child} did not persist closed turn ${minimumTurn} within ${timeoutMs}ms`,
|
||||
)
|
||||
}
|
||||
}, { interval: WAIT_POLL_INTERVAL_MS, timeout: timeoutMs })
|
||||
}
|
||||
|
||||
/** Whether a raw session log contains the requested closed turn. */
|
||||
function hasClosedTurn(content: string, turn: number): boolean {
|
||||
return content.split('\n').filter(Boolean).some((line) => {
|
||||
const event = JSON.parse(line) as { type?: unknown; data?: { turn?: unknown } }
|
||||
return event.type === 'turn/end' && event.data?.turn === turn
|
||||
})
|
||||
}
|
||||
|
||||
/** Wait until the latest durable goal snapshot reaches one phase. */
|
||||
async function waitForPersistedGoalPhase(
|
||||
root: string,
|
||||
sessionId: string,
|
||||
phase: string,
|
||||
timeoutMs = DEFAULT_WAIT_TIMEOUT_MS,
|
||||
): Promise<void> {
|
||||
await vi.waitFor(async () => {
|
||||
const content = (await harvestSessionLogs(root)).find(log => log.id === sessionId)?.content
|
||||
const matched = content?.split('\n').filter(Boolean).some((line) => {
|
||||
const event = JSON.parse(line) as { type?: unknown; data?: { goal?: { phase?: unknown } } }
|
||||
return event.type === 'goal/change' && event.data?.goal?.phase === phase
|
||||
}) ?? false
|
||||
if (!matched) {
|
||||
throw new Error(`snapshot-harness: session "${sessionId}" did not persist goal phase "${phase}" within ${timeoutMs}ms`)
|
||||
}
|
||||
}, { interval: WAIT_POLL_INTERVAL_MS, timeout: timeoutMs })
|
||||
}
|
||||
|
||||
/** Wait until an inserted inbox message contains scenario-owned text. */
|
||||
async function waitForPersistedInboxMessage(
|
||||
root: string,
|
||||
sessionId: string,
|
||||
text: string,
|
||||
timeoutMs = DEFAULT_WAIT_TIMEOUT_MS,
|
||||
): Promise<void> {
|
||||
await vi.waitFor(async () => {
|
||||
const log = (await harvestSessionLogs(root)).find(candidate => candidate.id === sessionId)
|
||||
const matched = log?.content.split('\n').some((line) => {
|
||||
if (line.length === 0) return false
|
||||
const record = JSON.parse(line) as {
|
||||
type?: unknown
|
||||
data?: { inserted?: Array<{ content?: Array<{ type?: unknown; text?: unknown }> }> }
|
||||
}
|
||||
return record.type === 'agent/inbox/spliced' && record.data?.inserted?.some(message =>
|
||||
message.content?.some(block => block.type === 'text'
|
||||
&& typeof block.text === 'string' && block.text.includes(text))) === true
|
||||
}) ?? false
|
||||
if (!matched) {
|
||||
throw new Error(`snapshot-harness: session "${sessionId}" did not persist expected inbox message within ${timeoutMs}ms`)
|
||||
}
|
||||
}, { interval: WAIT_POLL_INTERVAL_MS, timeout: timeoutMs })
|
||||
}
|
||||
|
||||
/** Whether a child log contains model work after its own descriptor event. */
|
||||
function hasRequestHeaderAfterDescriptor(content: string): boolean {
|
||||
const events = content.slice(0, content.lastIndexOf('\n') + 1)
|
||||
.split('\n')
|
||||
.filter(line => line.length > 0)
|
||||
.map(line => JSON.parse(line) as { type?: unknown })
|
||||
const descriptor = events.findLastIndex(event => event.type === 'subagent/descriptor')
|
||||
return descriptor >= 0
|
||||
&& events.slice(descriptor + 1).some(event => event.type === 'request/header')
|
||||
}
|
||||
|
||||
/** Wait until a complete provider or fallback title record follows the latest closed turn. */
|
||||
async function waitForPersistedTitleAfterTurnEnd(
|
||||
root: string,
|
||||
@@ -529,6 +665,21 @@ async function waitForPersistedTitleAfterTurnEnd(
|
||||
}, { interval: WAIT_POLL_INTERVAL_MS, timeout: timeoutMs })
|
||||
}
|
||||
|
||||
/** Wait until a complete record of `type` follows the latest closed turn. */
|
||||
async function waitForPersistedEventAfterTurnEnd(
|
||||
root: string,
|
||||
sessionId: string,
|
||||
type: string,
|
||||
timeoutMs = DEFAULT_WAIT_TIMEOUT_MS,
|
||||
): Promise<void> {
|
||||
await vi.waitFor(async () => {
|
||||
const log = (await harvestSessionLogs(root)).find(candidate => candidate.id === sessionId)
|
||||
if (log === undefined || !latestEventFollowsTurnEnd(log.content, type)) {
|
||||
throw new Error(`snapshot-harness: session "${sessionId}" did not persist ${type} after turn/end within ${timeoutMs}ms`)
|
||||
}
|
||||
}, { interval: WAIT_POLL_INTERVAL_MS, timeout: timeoutMs })
|
||||
}
|
||||
|
||||
/** Wait for a cwd-relative marker proving an external action reached readiness. */
|
||||
async function waitForWorkspaceFile(
|
||||
cwd: string,
|
||||
@@ -557,6 +708,13 @@ function latestTitleFollowsTurnEnd(content: string): boolean {
|
||||
return turnEnd >= 0 && complete.lastIndexOf('\n{"type":"session/title",') > turnEnd
|
||||
}
|
||||
|
||||
/** Return whether a complete record of `type` occurs after the last complete turn end. */
|
||||
function latestEventFollowsTurnEnd(content: string, type: string): boolean {
|
||||
const complete = content.slice(0, content.lastIndexOf('\n') + 1)
|
||||
const turnEnd = complete.lastIndexOf('\n{"type":"turn/end",')
|
||||
return turnEnd >= 0 && complete.lastIndexOf(`\n{"type":"${type}",`) > turnEnd
|
||||
}
|
||||
|
||||
/** Return the latest open turn number, validating the persisted boundary record. */
|
||||
function latestOpenTurn(content: string): number | undefined {
|
||||
const complete = content.slice(0, content.lastIndexOf('\n') + 1)
|
||||
|
||||
@@ -197,7 +197,7 @@ function tokenizeFixtureString(value: string, ctx: NormalizeContext, basename: s
|
||||
+ String.raw`(?=$|[\\/\s<>'"()\[\]{},;:!?=])`,
|
||||
'g',
|
||||
)
|
||||
return exact.replace(absoluteCwd, CWD)
|
||||
return exact.replace(absoluteCwd, CWD).split(`/private${CWD}`).join(CWD)
|
||||
}
|
||||
|
||||
/** Recursively replace generated-cwd spellings while preserving every other JSON value. */
|
||||
|
||||
@@ -40,6 +40,11 @@ const SYSTEM_PROMPT_SNAPSHOT = 'system-prompt.expected.md'
|
||||
/** The structured tool-schema snapshot beside its owning header pin. */
|
||||
const TOOL_SCHEMAS_SNAPSHOT = 'tool-schemas.expected.json'
|
||||
|
||||
/** Return the dedicated tool-schema sidecar for one child fixture index. */
|
||||
function childToolSchemasSnapshot(index: number): string {
|
||||
return `tool-schemas.${index}.expected.json`
|
||||
}
|
||||
|
||||
/** The optional full Windows-native stdout transcript. */
|
||||
const WINDOWS_STDOUT_SNAPSHOT = 'stdout.expected.windows.jsonl'
|
||||
|
||||
@@ -103,6 +108,13 @@ export interface Scenario {
|
||||
* declare the same {@link expectedHeaderChanges}; meaningless off a pin.
|
||||
*/
|
||||
toolSchemasSource?: string
|
||||
/**
|
||||
* Child fixture indices whose own schema sequence is pinned separately,
|
||||
* where `1` names `session.1.jsonl` and
|
||||
* `tool-schemas.1.expected.json`. The class pin still owns every other
|
||||
* request-header field.
|
||||
*/
|
||||
pinsChildToolSchemas?: readonly number[]
|
||||
/**
|
||||
* How many changed `request/header` snapshots this PINNING scenario's primary
|
||||
* fixture legitimately carries (default 0). Their full prompt text is kept in
|
||||
@@ -152,25 +164,37 @@ export interface Scenario {
|
||||
* test is skipped on Windows; its fixtures stay guarded on every platform.
|
||||
*/
|
||||
posixOnly?: boolean
|
||||
/**
|
||||
* Whether the scenario boots a composition that needs a usable `pwsh`
|
||||
* (the pwsh-tool-turn scenario). The run test is skipped when the suite's
|
||||
* {@link SnapshotSuiteOptions.hasPwsh} probe is false; fixtures stay guarded
|
||||
* on every platform.
|
||||
*/
|
||||
pwshOnly?: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a scenario's run test is skipped for this mode and host: record mode
|
||||
* skips authored (non-`recorded`) scenarios, and {@link Scenario.posixOnly}
|
||||
* scenarios skip on Windows.
|
||||
* skips authored (non-`recorded`) scenarios, {@link Scenario.posixOnly}
|
||||
* scenarios skip on Windows, and {@link Scenario.pwshOnly} scenarios skip
|
||||
* when the caller's `hasPwsh` probe is false.
|
||||
*
|
||||
* @param scenario The scenario whose run test is being registered.
|
||||
* @param recording Whether the suite runs in record mode.
|
||||
* @param platform The running Node platform, injectable for unit coverage.
|
||||
* @param hasPwsh The caller's pwsh-availability probe; `pwshOnly` scenarios
|
||||
* skip unless it is true.
|
||||
* @returns True when the scenario's run test must not execute.
|
||||
*/
|
||||
export function scenarioSkipped(
|
||||
scenario: Scenario,
|
||||
recording: boolean,
|
||||
platform: NodeJS.Platform = process.platform,
|
||||
hasPwsh?: boolean,
|
||||
): boolean {
|
||||
if (recording && !scenario.recorded) return true
|
||||
return scenario.posixOnly === true && platform === 'win32'
|
||||
if (scenario.posixOnly === true && platform === 'win32') return true
|
||||
return scenario.pwshOnly === true && hasPwsh !== true
|
||||
}
|
||||
|
||||
/** One stdout expected output selected for a platform run. */
|
||||
@@ -211,6 +235,11 @@ export interface SnapshotSuiteOptions {
|
||||
* from `$DSH_SNAPSHOT` — env reading stays outside this library.
|
||||
*/
|
||||
mode: 'replay' | 'record' | 'refresh'
|
||||
/**
|
||||
* Whether a real `pwsh` executable is available on this host (the probe the
|
||||
* caller owns; `pwshOnly` scenarios skip when this is not true).
|
||||
*/
|
||||
hasPwsh?: boolean
|
||||
}
|
||||
|
||||
/** One scenario's generated claim on a shared snapshot file. */
|
||||
@@ -1052,8 +1081,9 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
scenarioSuite('snapshot scenarios', () => {
|
||||
for (const scenario of scenarios) {
|
||||
// In RECORD mode, only re-run the `recorded` (live-API) scenarios; the `authored` ones
|
||||
// (sidecar-driven errors/cancel) are never re-recorded. `posixOnly` scenarios skip on Windows.
|
||||
it.skipIf(scenarioSkipped(scenario, RECORDING))(`snapshot: ${scenario.name} matches the expected outputs`, async ({ expect }) => {
|
||||
// (sidecar-driven errors/cancel) are never re-recorded. `posixOnly` scenarios skip on Windows;
|
||||
// `pwshOnly` scenarios skip when the caller's `hasPwsh` probe is false.
|
||||
it.skipIf(scenarioSkipped(scenario, RECORDING, process.platform, options.hasPwsh))(`snapshot: ${scenario.name} matches the expected outputs`, async ({ expect }) => {
|
||||
const dir = join(snapshotsDir, scenario.name)
|
||||
const input = JSON.parse(await readFile(join(dir, 'input.json'), 'utf8')) as InputScript
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
@@ -1099,6 +1129,8 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
cwdAliases: result.cwdAliases,
|
||||
}
|
||||
|
||||
const childSchemaPins = new Set(scenario.pinsChildToolSchemas ?? [])
|
||||
|
||||
// Record writes live model fixtures; keyless refresh writes every comparable replayed
|
||||
// fixture. Pinning JSONL keeps prefixes but moves prompts and schemas into sidecars.
|
||||
const scrub = scenario.pinsHeader === true
|
||||
@@ -1177,6 +1209,18 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
claimSharedSnapshot(schemaClaims, schemaPath, scenario.name, toolSchemasSnapshot)
|
||||
await writeFile(schemaPath, toolSchemasSnapshot)
|
||||
}
|
||||
for (const index of childSchemaPins) {
|
||||
const log = result.sessionLogs[index]
|
||||
expect(log, `${mode}: no child session log at index ${index} to snapshot schemas from`)
|
||||
.toBeDefined()
|
||||
const schemaSets = normalizedToolSchemas((log as HarvestedLog).content, ctx)
|
||||
expect(schemaSets.length, `${mode}: child ${index} produced no tool schemas to snapshot`)
|
||||
.toBeGreaterThan(0)
|
||||
await writeFile(join(dir, childToolSchemasSnapshot(index)), formatToolSchemasSnapshot(
|
||||
schemaSets[0] as unknown[],
|
||||
schemaSets.slice(1),
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
for (const expected of stdoutExpectedVariants(scenario)) {
|
||||
@@ -1230,7 +1274,14 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
header,
|
||||
pinnedSchemaSets[index] as unknown[],
|
||||
))
|
||||
const childPinnedSchemas = new Map<number, unknown[][]>()
|
||||
for (const index of childSchemaPins) {
|
||||
const sidecar = await readFile(join(dir, childToolSchemasSnapshot(index)), 'utf8')
|
||||
const parsed = parseToolSchemasSnapshot(sidecar)
|
||||
childPinnedSchemas.set(index, [parsed.initial, ...parsed.changes])
|
||||
}
|
||||
for (const [logIndex, log] of result.sessionLogs.entries()) {
|
||||
const childSchemas = childPinnedSchemas.get(logIndex)
|
||||
const expectedChanges = scenario.pinsHeader === true && logIndex === 0
|
||||
? scenario.expectedHeaderChanges ?? 0
|
||||
: 0
|
||||
@@ -1243,8 +1294,15 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
.toBe(headers.length)
|
||||
expect(schemaSets.length, `session ${log.id}: every request/header must carry an array-valued tools field`)
|
||||
.toBe(headers.length)
|
||||
if (childSchemas !== undefined) {
|
||||
expect(childSchemas.length, `session ${log.id}: ${childToolSchemasSnapshot(logIndex)} has an unexpected tool-schema count`)
|
||||
.toBe(schemaSets.length)
|
||||
}
|
||||
for (const [k, header] of headers.entries()) {
|
||||
const expected = expectedChanges > 0 ? pinnedHeaders[k] : pinnedHeaders[0]
|
||||
const classPin = expectedChanges > 0 ? pinnedHeaders[k] : pinnedHeaders[0]
|
||||
const expected = childSchemas === undefined
|
||||
? classPin
|
||||
: { ...classPin as Record<string, unknown>, tools: childSchemas[k] }
|
||||
expect(header, `session ${log.id}: request/header #${k + 1} diverged from the pinned (${pinningScenario.name}) header`)
|
||||
.toEqual(expected)
|
||||
if (expectedChanges === 0) {
|
||||
@@ -1282,8 +1340,16 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
|
||||
it('every registered scenario has its required fixture files', async () => {
|
||||
// Every scenario needs input, stdout, a primary session fixture, and matching optional sidecars.
|
||||
for (const { name, overridden, pinsNativeWindowsStdout } of scenarios) {
|
||||
for (const { name, overridden, pinsNativeWindowsStdout, pinsChildToolSchemas } of scenarios) {
|
||||
const dir = join(snapshotsDir, name)
|
||||
const declaredChildPins = new Set(pinsChildToolSchemas ?? [])
|
||||
const childSidecars = (await readdir(dir, { withFileTypes: true }))
|
||||
.filter(entry => entry.isFile())
|
||||
.map(entry => /^tool-schemas\.([1-9]\d*)\.expected\.json$/.exec(entry.name))
|
||||
.filter((match): match is RegExpExecArray => match !== null)
|
||||
.map(match => Number(match[1]))
|
||||
expect(new Set(childSidecars), `${name}: child tool-schema sidecars must match \`pinsChildToolSchemas\``)
|
||||
.toEqual(declaredChildPins)
|
||||
expect(existsSync(join(dir, 'input.json')), `${name}/input.json`).toBe(true)
|
||||
expect(existsSync(join(dir, 'stdout.expected.jsonl')), `${name}/stdout.expected.jsonl`).toBe(true)
|
||||
expect(
|
||||
@@ -1366,9 +1432,30 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
assertUniqueSnapshotContents('tool-schema', schemas)
|
||||
})
|
||||
|
||||
it('every committed JSONL has valid tool results and canonical header storage', async () => {
|
||||
it('every declared child tool-schema sidecar is canonical and names a real child', async () => {
|
||||
for (const scenario of scenarios) {
|
||||
const pins = scenario.pinsChildToolSchemas ?? []
|
||||
if (pins.length === 0) continue
|
||||
const dir = join(snapshotsDir, scenario.name)
|
||||
const files = await sessionFixtures(dir)
|
||||
for (const index of pins) {
|
||||
expect(files[index], `${scenario.name}: child schema pin ${index} must name an existing session.<n>.jsonl fixture`)
|
||||
.toBeDefined()
|
||||
const file = childToolSchemasSnapshot(index)
|
||||
const sidecar = await readFile(join(dir, file), 'utf8')
|
||||
const parsed = parseToolSchemasSnapshot(sidecar)
|
||||
expect(sidecar, `${scenario.name}/${file} must use canonical JSON formatting`)
|
||||
.toBe(formatToolSchemasSnapshot(parsed.initial, parsed.changes))
|
||||
expect(parsed.initial.length, `${scenario.name}/${file} must pin at least one schema`)
|
||||
.toBeGreaterThan(0)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
it('every committed JSONL has valid tool results and canonical fixture storage', async () => {
|
||||
// Prompts and schemas always leave JSONL. Header pins retain prefixes;
|
||||
// every other fixture tokenizes those too. Fixed-point checks make both
|
||||
// every other fixture tokenizes those too. Portable cwd tokens never
|
||||
// retain a platform realpath prefix. Fixed-point checks make these
|
||||
// storage rules fail loud.
|
||||
for (const scenario of scenarios) {
|
||||
const dir = join(snapshotsDir, scenario.name)
|
||||
@@ -1377,6 +1464,8 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
const fixture = await readFile(join(dir, file), 'utf8')
|
||||
expect(unknownToolCallIds(fixture), `${scenario.name}/${file} contains UNKNOWN_TOOL`)
|
||||
.toEqual([])
|
||||
expect(fixture, `${scenario.name}/${file} carries a non-canonical macOS cwd token`)
|
||||
.not.toContain('/private{{cwd}}')
|
||||
expect(scrubSystemPrompts(fixture), `${scenario.name}/${file} carries an unscrubbed system prompt`)
|
||||
.toEqual(fixture)
|
||||
expect(scrubToolSchemas(fixture), `${scenario.name}/${file} carries unscrubbed tool schemas`)
|
||||
|
||||
@@ -9,7 +9,7 @@
|
||||
]},
|
||||
{ "file": "b/child/session.jsonl", "lines": [
|
||||
{ "type": "session", "id": "eeeeeeee-1111-4222-8333-444444444444", "createdAt": 300, "cwd": "{{CWD}}", "parentSession": "{{SID}}", "delegationDepth": 1 },
|
||||
{ "type": "request/header", "seq": 0, "time": 6, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
|
||||
{ "type": "request/header", "seq": 0, "time": 6, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "child-only", "description": "Child D", "parameters": { "type": "object" } }] }, "reason": "initial" } }
|
||||
]}
|
||||
]
|
||||
}
|
||||
|
||||
12
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/tool-schemas.1.expected.json
vendored
Normal file
12
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/tool-schemas.1.expected.json
vendored
Normal file
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"initial": [
|
||||
{
|
||||
"name": "child-only",
|
||||
"description": "Child D",
|
||||
"parameters": {
|
||||
"type": "object"
|
||||
}
|
||||
}
|
||||
],
|
||||
"changes": []
|
||||
}
|
||||
@@ -584,6 +584,55 @@ describe('runScenario', () => {
|
||||
expect(result.sessionLogs[0]?.content).toContain('"type":"turn/end"')
|
||||
})
|
||||
|
||||
it('waitForInboxMessage holds the app through a matching durable insertion', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({
|
||||
prompt: 'hang-until-cancel',
|
||||
persistLogsOnCancel: true,
|
||||
logs: [{
|
||||
file: 'project/main/session.jsonl',
|
||||
lines: [
|
||||
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
|
||||
{
|
||||
type: 'agent/inbox/spliced',
|
||||
seq: 0,
|
||||
time: 2,
|
||||
data: {
|
||||
target: 'next-turn',
|
||||
start: 0,
|
||||
inserted: [{ role: 'user', content: [{ type: 'text', text: 'durable marker' }] }],
|
||||
},
|
||||
},
|
||||
],
|
||||
}],
|
||||
})
|
||||
const result = await runScenario(
|
||||
{ steps: [...boot, { op: 'promptAndCancel', text: 'hang' }, { op: 'waitForInboxMessage', text: 'marker' }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)
|
||||
expect(result.sessionLogs[0]?.content).toContain('durable marker')
|
||||
})
|
||||
|
||||
it('waitForInboxMessage times out when the session log or matching insertion is absent', { timeout: 20_000 }, async () => {
|
||||
const absent = await scenario({ prompt: 'hang-until-cancel', persistLogsOnCancel: true })
|
||||
await expect(runScenario(
|
||||
{ steps: [...boot, { op: 'promptAndCancel', text: 'hang' }, { op: 'waitForInboxMessage', text: 'missing', timeoutMs: 20 }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile: absent.fixtureFile },
|
||||
)).rejects.toThrow(/did not persist expected inbox message within 20ms/)
|
||||
|
||||
const unmatched = await scenario({
|
||||
prompt: 'hang-until-cancel',
|
||||
persistLogsOnCancel: true,
|
||||
logs: [{
|
||||
file: 'project/main/session.jsonl',
|
||||
lines: [{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 }],
|
||||
}],
|
||||
})
|
||||
await expect(runScenario(
|
||||
{ steps: [...boot, { op: 'promptAndCancel', text: 'hang' }, { op: 'waitForInboxMessage', text: 'missing', timeoutMs: 20 }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile: unmatched.fixtureFile },
|
||||
)).rejects.toThrow(/did not persist expected inbox message within 20ms/)
|
||||
})
|
||||
|
||||
it('waitForTitleAfterTurnEnd holds the app through a standalone durable title', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({
|
||||
prompt: 'hang-until-cancel',
|
||||
@@ -742,6 +791,125 @@ describe('runScenario', () => {
|
||||
)).rejects.toThrow(/did not persist turn\/end within 20ms/)
|
||||
})
|
||||
|
||||
it('waitForGoalPhase requires the requested durable goal phase', { timeout: 20_000 }, async () => {
|
||||
const reached = await scenario({
|
||||
prompt: 'hang-until-cancel',
|
||||
persistLogsOnCancel: true,
|
||||
logs: [{
|
||||
file: 'project/main/session.jsonl',
|
||||
lines: [
|
||||
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
|
||||
{ type: 'goal/change', seq: 1, time: 2, data: {} },
|
||||
{ type: 'goal/change', seq: 2, time: 3, data: { goal: { phase: 'active' } } },
|
||||
],
|
||||
}],
|
||||
})
|
||||
const result = await runScenario(
|
||||
{
|
||||
steps: [
|
||||
...boot,
|
||||
{ op: 'promptAndCancel', text: 'hang' },
|
||||
{ op: 'waitForGoalPhase', phase: 'active' },
|
||||
],
|
||||
},
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile: reached.fixtureFile },
|
||||
)
|
||||
expect(result.sessionLogs[0]?.content).toContain('"phase":"active"')
|
||||
|
||||
const missing = await scenario({})
|
||||
await expect(runScenario(
|
||||
{ steps: [...boot, { op: 'waitForGoalPhase', phase: 'blocked', timeoutMs: 20 }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile: missing.fixtureFile },
|
||||
)).rejects.toThrow(/did not persist goal phase "blocked" within 20ms/)
|
||||
})
|
||||
|
||||
it('waitForSubagentTurnEnd requires a closed child work turn', { timeout: 20_000 }, async () => {
|
||||
const closed = await scenario({
|
||||
prompt: 'hang-until-cancel',
|
||||
persistLogsOnCancel: true,
|
||||
logs: [
|
||||
{
|
||||
file: 'project/main/session.jsonl',
|
||||
lines: [
|
||||
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
|
||||
{ type: 'turn/end', seq: 1, time: 2, data: { turn: 1, reason: { kind: 'aborted' } } },
|
||||
],
|
||||
},
|
||||
{
|
||||
file: 'project/child/session.jsonl',
|
||||
lines: [
|
||||
{ type: 'session', version: 0, id: 'child-1', createdAt: 2, parentSession: '{{SID}}', delegationDepth: 1 },
|
||||
{ type: 'subagent/descriptor', seq: 0, time: 1, data: {} },
|
||||
{ type: 'turn/start', seq: 1, time: 2, data: { turn: 1 } },
|
||||
{ type: 'request/header', seq: 2, time: 3, data: { header: {}, reason: 'initial' } },
|
||||
{ type: 'turn/end', seq: 3, time: 4, data: { turn: 1, reason: { kind: 'completed' } } },
|
||||
],
|
||||
},
|
||||
],
|
||||
})
|
||||
const result = await runScenario(
|
||||
{
|
||||
steps: [
|
||||
...boot,
|
||||
{ op: 'promptAndCancel', text: 'hang' },
|
||||
{ op: 'waitForSubagentTurnEnd' },
|
||||
],
|
||||
},
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile: closed.fixtureFile },
|
||||
)
|
||||
expect(result.sessionLogs[1]?.parentSession).toBe(result.sessionId)
|
||||
await expect(runScenario(
|
||||
{
|
||||
steps: [
|
||||
...boot,
|
||||
{ op: 'promptAndCancel', text: 'hang' },
|
||||
{ op: 'waitForSubagentTurnEnd', minimumTurn: 2, timeoutMs: 20 },
|
||||
],
|
||||
},
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile: closed.fixtureFile },
|
||||
)).rejects.toThrow(/subagent child #1 did not persist closed turn 2 within 20ms/)
|
||||
|
||||
const seedOnly = await scenario({
|
||||
prompt: 'hang-until-cancel',
|
||||
persistLogsOnCancel: true,
|
||||
logs: [
|
||||
{
|
||||
file: 'project/main/session.jsonl',
|
||||
lines: [
|
||||
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
|
||||
{ type: 'turn/end', seq: 1, time: 2, data: { turn: 1, reason: { kind: 'aborted' } } },
|
||||
],
|
||||
},
|
||||
{
|
||||
file: 'project/child/session.jsonl',
|
||||
lines: [
|
||||
{ type: 'session', version: 0, id: 'child-1', createdAt: 2, parentSession: '{{SID}}', delegationDepth: 1 },
|
||||
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1, trigger: { kind: 'message' } } },
|
||||
{ type: 'request/header', seq: 1, time: 2, data: { header: {}, reason: 'initial' } },
|
||||
{ type: 'turn/end', seq: 2, time: 3, data: { turn: 1, reason: { kind: 'completed' } } },
|
||||
{ type: 'subagent/descriptor', seq: 3, time: 4, data: {} },
|
||||
],
|
||||
},
|
||||
],
|
||||
})
|
||||
await expect(runScenario(
|
||||
{
|
||||
steps: [
|
||||
...boot,
|
||||
{ op: 'promptAndCancel', text: 'hang' },
|
||||
{ op: 'waitForSubagentTurnEnd', timeoutMs: 20 },
|
||||
],
|
||||
},
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile: seedOnly.fixtureFile },
|
||||
)).rejects.toThrow(/subagent child #1 did not persist closed turn 1 within 20ms/)
|
||||
|
||||
const missing = await scenario({})
|
||||
await expect(runScenario(
|
||||
{ steps: [...boot, { op: 'waitForSubagentTurnEnd', child: 2, timeoutMs: 20 }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile: missing.fixtureFile },
|
||||
)).rejects.toThrow(/subagent child #2 did not persist closed turn 1 within 20ms/)
|
||||
})
|
||||
|
||||
it('waitForTitleAfterTurnEnd times out when the title precedes the boundary', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({
|
||||
prompt: 'hang-until-cancel',
|
||||
@@ -767,6 +935,55 @@ describe('runScenario', () => {
|
||||
)).rejects.toThrow(/did not persist session\/title after turn\/end within 20ms/)
|
||||
})
|
||||
|
||||
it('waitForEventAfterTurnEnd holds the app for a typed post-boundary record and times out otherwise', { timeout: 20_000 }, async () => {
|
||||
const late = await scenario({
|
||||
prompt: 'hang-until-cancel',
|
||||
persistLogsOnCancel: true,
|
||||
logs: [{
|
||||
file: 'project/main/session.jsonl',
|
||||
lines: [
|
||||
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
|
||||
{ type: 'turn/end', seq: 1, time: 2, data: { turn: 1, reason: { kind: 'aborted' } } },
|
||||
{ type: 'user/message', seq: 2, time: 3, data: { content: [{ type: 'text', text: 'late goal state' }], source: { kind: 'user' } } },
|
||||
],
|
||||
}],
|
||||
})
|
||||
const result = await runScenario(
|
||||
{
|
||||
steps: [
|
||||
...boot,
|
||||
{ op: 'promptAndCancel', text: 'hang' },
|
||||
{ op: 'waitForEventAfterTurnEnd', type: 'user/message' },
|
||||
],
|
||||
},
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile: late.fixtureFile },
|
||||
)
|
||||
expect(result.sessionLogs[0]?.content).toMatch(/"turn\/end"[\s\S]*"user\/message"/)
|
||||
|
||||
const early = await scenario({
|
||||
prompt: 'hang-until-cancel',
|
||||
persistLogsOnCancel: true,
|
||||
logs: [{
|
||||
file: 'project/main/session.jsonl',
|
||||
lines: [
|
||||
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
|
||||
{ type: 'user/message', seq: 1, time: 1, data: { content: [{ type: 'text', text: 'early' }], source: { kind: 'user' } } },
|
||||
{ type: 'turn/end', seq: 2, time: 2, data: { turn: 1, reason: { kind: 'aborted' } } },
|
||||
],
|
||||
}],
|
||||
})
|
||||
await expect(runScenario(
|
||||
{
|
||||
steps: [
|
||||
...boot,
|
||||
{ op: 'promptAndCancel', text: 'hang' },
|
||||
{ op: 'waitForEventAfterTurnEnd', type: 'user/message', timeoutMs: 20 },
|
||||
],
|
||||
},
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile: early.fixtureFile },
|
||||
)).rejects.toThrow(/did not persist user\/message after turn\/end within 20ms/)
|
||||
})
|
||||
|
||||
it('promptExpectError swallows a model-error response as the expected outcome', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({ prompt: 'error' })
|
||||
const result = await runScenario(
|
||||
@@ -865,6 +1082,19 @@ describe('runScenario', () => {
|
||||
expect(result.sessionId).toBeDefined()
|
||||
})
|
||||
|
||||
it('waitForFile holds the next input step behind cwd-relative readiness', { timeout: 20_000 }, async () => {
|
||||
const { dir, fixtureFile } = await scenario({})
|
||||
const workspaceDir = join(dir, 'workspace')
|
||||
const { mkdir } = await import('node:fs/promises')
|
||||
await mkdir(workspaceDir, { recursive: true })
|
||||
await writeFile(join(workspaceDir, 'ready'), '')
|
||||
const result = await runScenario(
|
||||
{ steps: [...boot, { op: 'waitForFile', path: 'ready' }, { op: 'cancel' }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile, workspaceDir },
|
||||
)
|
||||
expect(result.sessionId).toBeDefined()
|
||||
})
|
||||
|
||||
it.each([
|
||||
[{ op: 'prompt', text: 'x' }, /prompt before newSession/],
|
||||
[{ op: 'promptAndWaitForAgentMessage', text: 'x', waitForText: 'later' }, /promptAndWaitForAgentMessage before newSession/],
|
||||
@@ -872,7 +1102,10 @@ describe('runScenario', () => {
|
||||
[{ op: 'promptAndCancel', text: 'x' }, /promptAndCancel before newSession/],
|
||||
[{ op: 'waitForTurnStart' }, /waitForTurnStart before newSession/],
|
||||
[{ op: 'waitForTurnEnd' }, /waitForTurnEnd before newSession/],
|
||||
[{ op: 'waitForGoalPhase', phase: 'active' }, /waitForGoalPhase before newSession/],
|
||||
[{ op: 'waitForInboxMessage', text: 'marker' }, /waitForInboxMessage before newSession/],
|
||||
[{ op: 'waitForTitleAfterTurnEnd' }, /waitForTitleAfterTurnEnd before newSession/],
|
||||
[{ op: 'waitForEventAfterTurnEnd', type: 'user/message' }, /waitForEventAfterTurnEnd before newSession/],
|
||||
[{ op: 'cancel' }, /cancel before newSession/],
|
||||
] as [InputStep, RegExp][])('rejects %j before newSession', { timeout: 20_000 }, async (step, message) => {
|
||||
const { fixtureFile } = await scenario({})
|
||||
|
||||
@@ -470,6 +470,24 @@ describe('tokenizeSessionFixtureCwd', () => {
|
||||
expect(tokenizeSessionFixtureCwd(out)).toBe(out)
|
||||
})
|
||||
|
||||
it('collapses a residual macOS realpath prefix around an existing cwd token', () => {
|
||||
const raw = [
|
||||
JSON.stringify({ type: 'session', id: 's', createdAt: 1, cwd: '{{cwd}}' }),
|
||||
JSON.stringify({
|
||||
type: 'tool/result',
|
||||
seq: 1,
|
||||
time: 2,
|
||||
data: { content: [{ type: 'text', text: 'wrote /private{{cwd}}/proof.txt' }] },
|
||||
}),
|
||||
'',
|
||||
].join('\n')
|
||||
|
||||
const out = tokenizeSessionFixtureCwd(raw)
|
||||
expect(out).toContain('wrote {{cwd}}/proof.txt')
|
||||
expect(out).not.toContain('/private{{cwd}}')
|
||||
expect(tokenizeSessionFixtureCwd(out)).toBe(out)
|
||||
})
|
||||
|
||||
it('rejects a log without a session cwd', () => {
|
||||
expect(() => tokenizeSessionFixtureCwd('')).toThrow(
|
||||
'acp-snapshot: cannot tokenize a cwd without a basename',
|
||||
|
||||
@@ -83,6 +83,7 @@ const REPLAY_SCENARIOS: Scenario[] = [
|
||||
env: { DSH_PERMISSION_MODE: 'never' },
|
||||
configPath: AGENT.configPath,
|
||||
workspaceParent: tmpdir(),
|
||||
pinsChildToolSchemas: [1],
|
||||
prepareWorkspace: (cwd) => {
|
||||
writeFileSync(join(cwd, 'seed.txt'), 'prepared at runtime')
|
||||
},
|
||||
@@ -94,7 +95,7 @@ const REPLAY_SCENARIOS: Scenario[] = [
|
||||
|
||||
const RECORD_SCENARIOS: Scenario[] = [
|
||||
{ name: 'rec-pin', hasModelTurn: true, recorded: true, pinsHeader: true },
|
||||
{ name: 'rec-child', hasModelTurn: true, recorded: true },
|
||||
{ name: 'rec-child', hasModelTurn: true, recorded: true, pinsChildToolSchemas: [1] },
|
||||
// recorded:false in record mode → registered but skipped (never re-recorded).
|
||||
{ name: 'rec-skip', hasModelTurn: true, recorded: false, overridden: true },
|
||||
]
|
||||
@@ -123,6 +124,7 @@ function staleRefreshFixtures(dir: string): void {
|
||||
writeFileSync(join(dir, 'plain-turn', 'stdout.expected.jsonl'), 'stale stdout\n')
|
||||
writeFileSync(join(dir, 'pin-turn', 'system-prompt.expected.md'), 'STALE PROMPT\n')
|
||||
writeFileSync(join(dir, 'pin-turn', 'tool-schemas.expected.json'), '{"initial":[{"name":"stale"}],"changes":[]}\n')
|
||||
writeFileSync(join(dir, 'plain-turn', 'tool-schemas.1.expected.json'), '{"initial":[{"name":"stale-child"}],"changes":[]}\n')
|
||||
|
||||
const plainBehaviorFile = join(dir, 'plain-turn', 'behavior.json')
|
||||
const plainBehavior = JSON.parse(readFileSync(plainBehaviorFile, 'utf8')) as Record<string, unknown>
|
||||
@@ -185,6 +187,9 @@ describe('defineAcpSnapshotSuite: refresh write-back', () => {
|
||||
const schemas = readFileSync(join(refreshDir, 'pin-turn', 'tool-schemas.expected.json'), 'utf8')
|
||||
expect(schemas).toContain('"description": "D1"')
|
||||
expect(schemas).not.toContain('"name":"stale"')
|
||||
const childSchemas = readFileSync(join(refreshDir, 'plain-turn', 'tool-schemas.1.expected.json'), 'utf8')
|
||||
expect(childSchemas).toContain('"name": "child-only"')
|
||||
expect(childSchemas).not.toContain('stale-child')
|
||||
|
||||
const pinSession = readFileSync(join(refreshDir, 'pin-turn', 'session.jsonl'), 'utf8')
|
||||
expect(pinSession).toContain('"cwd":"{{cwd}}"')
|
||||
@@ -197,6 +202,8 @@ describe('defineAcpSnapshotSuite: record inventory write-back', () => {
|
||||
expect(fixture).toContain('"type":"session"')
|
||||
expect(fixture).toContain('"cwd":"{{cwd}}"')
|
||||
expect(() => readFileSync(join(recordDir, 'rec-child', 'session.2.jsonl'), 'utf8')).toThrow()
|
||||
expect(readFileSync(join(recordDir, 'rec-child', 'tool-schemas.1.expected.json'), 'utf8'))
|
||||
.toContain('"name": "t1"')
|
||||
})
|
||||
|
||||
it('retains an unchanged message id across the recorded parent and child fixtures', () => {
|
||||
@@ -469,6 +476,7 @@ describe('stdoutExpectedVariants', () => {
|
||||
describe('scenarioSkipped', () => {
|
||||
const authored: Scenario = { name: 'authored', hasModelTurn: true, recorded: false }
|
||||
const posix: Scenario = { name: 'posix-cancel', hasModelTurn: true, recorded: false, posixOnly: true }
|
||||
const pwsh: Scenario = { name: 'pwsh-tool', hasModelTurn: true, recorded: false, pwshOnly: true }
|
||||
|
||||
it('skips authored scenarios only while recording', () => {
|
||||
expect(scenarioSkipped(authored, true, 'linux')).toBe(true)
|
||||
@@ -481,6 +489,13 @@ describe('scenarioSkipped', () => {
|
||||
expect(scenarioSkipped(posix, false, 'darwin')).toBe(false)
|
||||
expect(scenarioSkipped(authored, false, 'win32')).toBe(false)
|
||||
})
|
||||
|
||||
it('skips pwshOnly scenarios when the host lacks pwsh, and runs them otherwise', () => {
|
||||
expect(scenarioSkipped(pwsh, false, 'linux', false)).toBe(true)
|
||||
expect(scenarioSkipped(pwsh, false, 'win32', true)).toBe(false)
|
||||
expect(scenarioSkipped(pwsh, false, 'linux', true)).toBe(false)
|
||||
expect(scenarioSkipped(authored, false, 'linux', false)).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('fixtureContext', () => {
|
||||
|
||||
@@ -21,9 +21,7 @@
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"lib/types/**/*.d.ts",
|
||||
"lib/types/**/*.d.ts.map",
|
||||
"src"
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
|
||||
@@ -3,4 +3,4 @@
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/support/invariants/README.md
|
||||
README.md: 203dbd5ad09f5b1378061fbf9adcff885889eae2
|
||||
README.zh.md: 841e7a32f8e2605c25aadef35862a0223046933f
|
||||
README.zh.md: fb9d15f3d0cd77f8002c6ff1a909e09e8c6c61fa
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
用于包（package）自有运行时不变量检查的可配置注册表服务。根插件注册 `ctx.invariants`；它不包含产品检查或产品包导入。每个工作区包都发布一个 `./invariant` 配套入口，用于注册其精确 NPM 包名。
|
||||
用于包自有运行时不变量检查的可配置注册表服务。根插件注册 `ctx.invariants`；它不包含产品检查或产品包导入。每个工作区包都发布一个 `./invariant` 配套入口，用于注册其精确 NPM 包名。
|
||||
|
||||
## 服务：`InvariantService`（`ctx.invariants`）
|
||||
|
||||
|
||||
@@ -21,9 +21,7 @@
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"lib/types/**/*.d.ts",
|
||||
"lib/types/**/*.d.ts.map",
|
||||
"src"
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
|
||||
@@ -3,4 +3,4 @@
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/support/llm-mock-server/README.md
|
||||
README.md: a535c086bf688ad48b1a3bb19c7b81da21cbad92
|
||||
README.zh.md: e013cc47399da7fdde42c10dfe086ffab9105785
|
||||
README.zh.md: d814d8815b38bb34bd0d871d552e6f3ec75e042a
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
可编脚本的 OpenAI 兼容 HTTP/SSE 服务器，用于在无提供方密钥的情况下测试真实 LLM 适配器、agent loop 和恢复策略。它接受 `POST /chat/completions` 和 `POST /v1/chat/completions`；每个已接受请求按到达顺序消费一个已配置行为。无效 method、path、bearer token 和 JSON 不消费脚本。
|
||||
可编脚本的 OpenAI 兼容 HTTP／SSE（Server-Sent Events）服务器，用于在无提供方密钥的情况下测试真实 LLM（大语言模型）适配器、agent loop（智能体循环）和恢复策略。它接受 `POST /chat/completions` 和 `POST /v1/chat/completions`；每个已接受请求按到达顺序消费一个已配置行为。无效的请求方法、路径、Bearer token 和 JSON 不会消费脚本条目。
|
||||
|
||||
库入口导出 `startMockLlmServer(options)`、行为和 telemetry 类型、默认随机压力权重、可接受的 Node timer 边界，以及带有绑定 `baseURL`、已生成或已配置 `randomSeed`、已捕获请求和幂等 `close()` 的运行句柄。关闭会强制终止停滞连接。
|
||||
库入口导出 `startMockLlmServer(options)`、行为类型和遥测（telemetry）类型、默认随机压力权重、Node 定时器允许的上限，以及带有绑定 `baseURL`、自动生成或显式配置 `randomSeed`、已捕获请求和幂等 `close()` 的运行句柄。关闭会强制终止停滞连接。
|
||||
|
||||
## 独立使用
|
||||
|
||||
@@ -26,7 +26,7 @@ DEEPSEEK_API_KEY=mock-key \
|
||||
pnpm run demo:headless "test provider recovery"
|
||||
```
|
||||
|
||||
仓库脚本将 JSONL 写入 stdout：`ready` 记录携带 `/v1` base URL 和随机种子，后续请求/结果记录同时命名脚本行为和实际选中的具体行为。这个私有支持包（package）不公开可安装的二进制命令。
|
||||
仓库脚本将 JSONL 写入 stdout：`ready` 记录携带以 `/v1` 结尾的基础 URL 和随机种子，后续请求/结果记录同时命名脚本行为和实际选中的具体行为。这个私有支持包不公开可安装的二进制命令。
|
||||
|
||||
## 行为脚本
|
||||
|
||||
@@ -34,23 +34,23 @@ pnpm run demo:headless "test provider recovery"
|
||||
|
||||
| 行为 | 协议结果 |
|
||||
|---|---|
|
||||
| `connection_reset` | 在 HTTP header 前销毁 socket |
|
||||
| `stream_disconnect` | 发送 SSE header，然后在第一个事件前 reset |
|
||||
| `partial_disconnect` | 发送文本 delta，然后 reset socket |
|
||||
| `stall` | 发送 SSE header，并保持空闲，直到客户端/服务器取消 |
|
||||
| `connection_reset` | 在发送 HTTP 标头前销毁 socket |
|
||||
| `stream_disconnect` | 发送 SSE 标头，然后在第一个事件前重置连接 |
|
||||
| `partial_disconnect` | 发送文本增量，然后重置 socket |
|
||||
| `stall` | 发送 SSE header，并保持空闲，直到客户端／服务器取消 |
|
||||
| `empty` | 发送有效的无内容 stop 和 `[DONE]` |
|
||||
| `empty_body` / `stream_eof` / `partial_eof` | 正常结束，但缺少必需的 `[DONE]` 边界 |
|
||||
| `malformed_json` / `malformed_event` | 发送无效 SSE JSON 或无效提供方分片形态 |
|
||||
| `rate_limit` / `server_error` / `service_unavailable` | 返回面向重试的 429/500/503 JSON 错误 |
|
||||
| `auth_error` / `invalid_request` / `context_overflow` / `quota_exceeded` | 返回终止性或单独恢复的提供方错误 |
|
||||
| `auth_error` / `invalid_request` / `context_overflow` / `quota_exceeded` | 返回终止性错误或需要单独恢复的提供方错误 |
|
||||
| `success` / `slow_success` / `reasoning_success` | 流式发送完整文本响应，可选延迟或先发送 reasoning |
|
||||
| `tool_call_success` / `max_tokens` | 以工具调用或 `length` 结束原因完成 |
|
||||
| `wrong_content_type` | 在 `application/json` 下发送有效 SSE 正文 |
|
||||
| `random` | 从加权播种随机性中选择具体请求行为 |
|
||||
| `tool_call_success` / `max_tokens` | 以工具调用或结束原因 `length` 完成 |
|
||||
| `wrong_content_type` | 以 `application/json` 内容类型发送有效 SSE 正文 |
|
||||
| `random` | 按带权重的种子随机选择具体请求行为 |
|
||||
|
||||
`connection_refused` 只能在 CLI 中使用，且必须是第一个条目。它会延迟绑定调用方指定的非零端口，因此 `--listen-delay-ms` 期间的请求会收到真实 TCP 拒绝；其余条目在 listener 启动后开始。
|
||||
|
||||
## 随机 mode
|
||||
## 随机模式
|
||||
|
||||
使用重复 `random` 条目执行开放式混合运行：
|
||||
|
||||
@@ -63,24 +63,24 @@ pnpm run mock:llm -- \
|
||||
--random-weights 'success=60,slow_success=10,connection_reset=5,stream_disconnect=5,partial_disconnect=10,empty=5,server_error=5'
|
||||
```
|
||||
|
||||
省略 `--seed` 会生成种子，并在 `ready` 记录中打印。`--random-weights` 接受非负的相对 `behavior=weight` 条目，并要求至少一个正权重具体行为。导出默认值是一个成功占主导的压力分布，包含 reset、disconnect、部分输出、空完成、stall、429/5xx、干净截断和格式错误 JSON；它用于施加测试压力，而非估计生产事故频率。`connection_refused` 被排除，因为已绑定的请求处理器无法产生真实拒绝。
|
||||
省略 `--seed` 会生成种子，并在 `ready` 记录中打印。`--random-weights` 接受非负的相对 `behavior=weight` 条目，并要求至少一个正权重具体行为。导出默认值是一个成功占主导的压力分布，包含 reset、disconnect、部分输出、空完成、stall、429/5xx、干净截断和格式错误的 JSON；它用于施加测试压力，而非估计生产事故频率。`connection_refused` 被排除，因为已绑定的请求处理器无法产生真实拒绝。
|
||||
|
||||
随机权重包含 `stall` 时，为待测客户端配置较短的流空闲超时，使场景及时结束。
|
||||
|
||||
## 时序与内容控制
|
||||
|
||||
CLI 公开 `--success-text`、`--partial-text`、`--reasoning-text`、`--chunk-size`、`--chunk-delay-ms`、`--disconnect-delay-ms`、`--retry-after-ms`、`--request-id`、`--tool-name` 和 `--tool-arguments`。毫秒延迟是 Node timer 范围内的有界整数；`retryAfterMs` 还必须为正数。库接受相同的 camel-case 选项。可选的精确 `apiKey` 验证 `Authorization: Bearer <token>`；省略时接受任何 token。
|
||||
CLI 公开 `--success-text`、`--partial-text`、`--reasoning-text`、`--chunk-size`、`--chunk-delay-ms`、`--disconnect-delay-ms`、`--retry-after-ms`、`--request-id`、`--tool-name` 和 `--tool-arguments`。毫秒延迟是 Node timer 范围内的有界整数；`retryAfterMs` 还必须为正数。库接受相同的 camel-case 选项。可选的 `apiKey` 会精确验证 `Authorization: Bearer <token>`；省略时接受任何 token。
|
||||
|
||||
## 模型体验
|
||||
|
||||
无。该测试服务器替代提供方协议行为，而不调用真实模型。
|
||||
|
||||
#### KV 缓存影响
|
||||
#### KV Cache 影响
|
||||
|
||||
无；请求在本地终止，绝不会到达提供方缓存。
|
||||
|
||||
## 已知限制与待完成工作
|
||||
## 已知限制与暂缓事项
|
||||
|
||||
- **随机权重建模测试压力，而非生产事故频率**：需要环境专用分布的调用方必须提供已测量权重，并记录发出的种子。
|
||||
- **请求脚本按到达顺序执行**：并发调用方共享一个游标，因此确定性的每会话故障分配需要独立服务器实例。
|
||||
- **真实连接拒绝是 listener 生命周期阶段**：CLI 延迟必须与客户端尝试重叠；请求级随机选择只能 reset 已接受连接。
|
||||
- **真实连接拒绝发生在监听器生命周期阶段**：CLI 延迟必须与客户端尝试重叠；请求级随机选择只能重置已接受的连接。
|
||||
|
||||
@@ -21,9 +21,7 @@
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"lib/types/**/*.d.ts",
|
||||
"lib/types/**/*.d.ts.map",
|
||||
"src"
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/support/llm-replay/README.md
|
||||
README.md: 6c934e01a5f13b94724d0e5435524d9d463adb2f
|
||||
README.zh.md: d13d84ad95a6b63177ee0e2aa83965be19fdeeb8
|
||||
README.md: ee062d0c2804905f33f1ff476d12bb6dd57666e5
|
||||
README.zh.md: ab3420d9500a6ca77f04a2ad96095f8883aeb874
|
||||
|
||||
@@ -4,7 +4,7 @@ English | [中文](README.zh.md)
|
||||
|
||||
A replay LLM plugin for keyless snapshot tests. It yields model streams reconstructed from a recorded **session JSONL** fixture, so a test can boot the real agent against a fixed model transcript with no API key. With `providers` configured it registers a replay-only adapter whose catalog is available to scenarios that exercise model discovery; without `providers` it installs the catch-all `llm/stream` waterfall used by tests that do not need discovery.
|
||||
|
||||
Its consumers are the ACP, headless `stream-json`, and TUI snapshot suites plus the web browser e2e lane. Loader-driven suites mount this plugin in place of a real LLM adapter; the web lane installs it directly to retain the teardown consumption handle. Keeping derivation and replay here places that logic under the per-file 100% coverage gate on `packages/*/src`.
|
||||
Its consumers are the ACP and headless `stream-json` snapshot suites plus the Web browser e2e lane. Loader-driven suites mount this plugin in place of a real LLM adapter; the Web lane installs it directly to retain the teardown consumption handle.
|
||||
|
||||
## How the fixture works
|
||||
|
||||
@@ -12,6 +12,8 @@ The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assi
|
||||
|
||||
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`) that either replaces the derived script (a bare `ReplayEntry[]`) or augments it (`{ patches: [{ at, entry }] }`: keep every JSONL-derived call and swap the named 0-based call indexes; `at` equal to the derived length appends the retry attempt after an injected transient throw). Patch indexes must be unique. The override document, each patch and entry, and every chunk discriminant are validated when the file loads. A `hang` entry may name `readyFile`; replay writes that empty marker after its prefix chunks reach the loop and before it waits for cancellation, so an external driver can cancel deterministically without observing a presentation update.
|
||||
|
||||
A scripted string may embed `{{fromRequest:<regex>}}` to fill a value no static sidecar can know — for example a randomly minted goal id the model must echo back into `update_goal`. At stream time every placeholder resolves against the live request: the corpus is every string leaf of the request messages joined by newlines, the pattern's LAST corpus match wins, and its first capture group (or the whole match without one) substitutes in place. A pattern that matches nothing, an invalid pattern, and an unterminated placeholder each fail loud. The last two braces of a consecutive `}` run terminate the placeholder, so a pattern may end with a brace quantifier (`[0-9a-f]{4}`) but cannot contain `}}` followed by further pattern content. Resolution applies to every scripted entry, including ones derived from the recorded JSONL — a recorded fixture whose text legitimately contains the literal marker must be expressed through a sidecar without it.
|
||||
|
||||
## Nested agents: per-session keying
|
||||
|
||||
A scenario where a parent agent delegates to in-process subagents records more than one log: the parent (`session.jsonl`) plus one per child (`session.1.jsonl`, …). Each agent runs as its own `Session` on the same context, so replay must serve each one its own script.
|
||||
@@ -33,7 +35,7 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
|
||||
name: '@deepseek-ai/dsh-llm-replay'
|
||||
config:
|
||||
providers:
|
||||
- id: deepseek
|
||||
- id: deepseek-official
|
||||
name: DeepSeek
|
||||
retryPolicy:
|
||||
mode: normal
|
||||
@@ -55,7 +57,7 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
|
||||
- `installLlmReplay(ctx, config)` — install the configured replay adapter or catch-all `llm/stream` listener; returns a `ReplayHandle` (`dispose()` for HMR safety plus `assertConsumed()`, the teardown check that every recorded script bound to a live session and every bound cursor drained — turning a scenario that silently drove fewer model calls than recorded into a crisp diagnostic). Use this in tests to drive replay without the Loader or env vars.
|
||||
- `loadSessionScripts(config)` — resolve the ordered `SessionScript[]` (primary + children) for a scenario, ready to bind to live sessions in first-call order.
|
||||
- `loadReplayScript(config)` — resolve the `ReplayEntry[]` for the primary session only (validated sidecar replacement/patches if present, else derived from the JSONL; fail-loud if the fixture is missing).
|
||||
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)` — the pure helpers that turn a recorded session log into a script and read its header `id`/`createdAt`. A derived group must end in a `finish` chunk; a group without one is the fingerprint of a thrown `stream()` and must instead be expressed via an override sidecar.
|
||||
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)` / `resolveScriptedEntry(entry, messages)` — the pure helpers that turn a recorded session log into a script, read its header `id`/`createdAt`, and resolve `{{fromRequest:...}}` placeholders against one live request. A derived group must end in a `finish` chunk; a group without one is the fingerprint of a thrown `stream()` and must instead be expressed via an override sidecar.
|
||||
- Types `ReplayEntry` / `ReplayOverrideDoc` / `ReplayOverridePatch` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`.
|
||||
|
||||
## Plugin export shape
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
用于无密钥快照测试的大语言模型（LLM）回放插件。它根据已记录的**会话 JSONL** fixture（测试前置数据）重建模型流，使测试无需 API 密钥即可针对固定的模型 transcript（文本记录）启动真实 agent（智能体）。配置 `providers` 后，它会注册仅用于回放的适配器，其模型目录可供测试模型发现功能的场景使用；未配置 `providers` 时，它会安装无需模型发现功能的测试所用 catch-all `llm/stream` waterfall（瀑布式事件）。
|
||||
用于无密钥快照测试的 LLM（大语言模型）回放插件。它根据已记录的**会话 JSONL** fixture（测试前置数据）重建模型流，使测试无需 API 密钥即可针对固定的模型 transcript（文本记录）启动真实 agent（智能体）。配置 `providers` 后，它会注册仅用于回放的适配器，其模型目录可供测试模型发现功能的场景使用；未配置 `providers` 时，它会安装无需模型发现功能的测试所用 catch-all `llm/stream` waterfall（瀑布式事件）。
|
||||
|
||||
其消费方包括 ACP（Agent Client Protocol）、headless `stream-json` 和 TUI 快照套件，以及 Web 浏览器 e2e 流水线。Loader 驱动的套件使用此插件替代真实 LLM 适配器；Web 流水线直接安装它，以保留清理阶段的消费检查句柄。将派生和回放逻辑放在此处，可使其受 `packages/*/src` 的逐文件 100% 覆盖率门禁约束。
|
||||
其消费方包括 ACP（Agent Client Protocol）与 headless `stream-json` 快照套件，以及 Web 浏览器 e2e 流水线。Loader 驱动的套件使用此插件替代真实 LLM 适配器；Web 流水线直接安装它，以保留清理阶段的消费检查句柄。
|
||||
|
||||
## fixture 的工作方式
|
||||
|
||||
@@ -12,18 +12,20 @@ fixture 就是持久化的会话日志（`<scenario>/session.jsonl`）。其 `as
|
||||
|
||||
有两种失败模式无法仅根据 `assistant/chunk` 重建：在产生任何分片前直接抛出异常（例如 HTTP 401，此时日志只有 `turn/end {error}` 而没有分片），以及取消或挂起（差异在时序，而非分片内容）。需要这些行为的场景可提供伴随文件（`<scenario>/replay.override.json`）：它可以替换派生脚本（裸 `ReplayEntry[]`），也可以增补派生脚本（`{ patches: [{ at, entry }] }`：保留所有从 JSONL 派生的调用，只替换指定的从 0 开始计数的调用索引；当 `at` 等于派生长度时，则在注入瞬态异常后的重试位置追加一次调用）。补丁索引不得重复。文件加载时会校验覆写文档、每个补丁和条目，以及每个分片的判别标签。`hang` 条目可以指定 `readyFile`；当前缀分片到达循环后、开始等待取消前，回放会写入这个空标记，使外部驱动程序无需观察展示层更新即可确定性地取消。
|
||||
|
||||
脚本字符串可以内嵌 `{{fromRequest:<regex>}}`，用来填入静态伴随文件不可能预知的值——例如模型必须原样回填到 `update_goal` 的随机生成 goal id。回放时每个占位符针对实时请求解析：语料是请求消息的所有字符串叶子按换行拼接的结果，取该模式在语料中的最后一次匹配，用其第一个捕获组（无捕获组时用整个匹配）原位替换。模式匹配不到内容、模式非法、占位符未闭合都会明确报错。连续右花括号串的最后两个花括号才是占位符结束符，因此模式可以以花括号量词收尾（如 `[0-9a-f]{4}`），但不能在 `}}` 之后还有后续模式内容。解析作用于所有脚本条目，包括从已记录 JSONL 派生的条目——若录制文本本身合法地含有该字面量标记，需改用不含标记的伴随文件表达。
|
||||
|
||||
## 嵌套 agent：每会话键控
|
||||
|
||||
父 agent 委托给进程内 subagent（子 agent）的场景会记录多个日志：父会话使用 `session.jsonl`，每个子会话各使用一个日志（`session.1.jsonl` 等）。每个 agent 都在同一上下文中作为独立的 `Session` 运行，因此回放必须为每个 agent 提供各自的脚本。
|
||||
父 agent 委托给进程内 subagent 的场景会记录多个日志：父会话使用 `session.jsonl`，每个子会话各使用一个日志（`session.1.jsonl` 等）。每个 agent 都在同一上下文中作为独立的 `Session` 运行，因此回放必须为每个 agent 提供各自的脚本。
|
||||
|
||||
回放根据发起调用的会话 id 为每次调用建立键（`GenerateOptions.sessionId` 由 agent loop 写入）。实时会话 id 每次运行时都会重新随机生成，绝不会等于记录中的 id，因此实时会话按**首次调用顺序**绑定到已记录脚本：脚本按 header 中的 `createdAt` 排序（父会话在前，因为它必须先开始流式输出才能委托）；第一个发起调用的实时会话取得第一个脚本，下一个新会话取得下一个脚本，以此类推。此后每个会话分别推进自己的游标。没有 `sessionId` 的调用视为一个绑定主脚本的匿名会话，因此单会话场景的行为与以前完全相同。不同实时会话的数量超过已记录脚本数时会明确报错。
|
||||
回放根据发起调用的会话 id 为每次调用建立键（`GenerateOptions.sessionId` 由 agent loop（智能体循环）写入）。实时会话 id 每次运行时都会重新随机生成，绝不会等于记录中的 id，因此实时会话按**首次调用顺序**绑定到已记录脚本：脚本按 header 中的 `createdAt` 排序（父会话在前，因为它必须先开始流式输出才能委托）；第一个发起调用的实时会话取得第一个脚本，下一个新会话取得下一个脚本，以此类推。此后每个会话分别推进自己的游标。没有 `sessionId` 的调用视为一个绑定主脚本的匿名会话，因此单会话场景的行为与以前完全相同。不同实时会话的数量超过已记录脚本数时会明确报错。
|
||||
|
||||
## 配置
|
||||
|
||||
| 键 | 类型 | 默认值 | 说明 |
|
||||
|---|---|---|---|
|
||||
| `file` | string | `$DSH_SNAPSHOT_FILE` | 主（父）`session.jsonl` fixture 的路径。必需（配置或 env）。 |
|
||||
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | 主会话的可选 `ReplayOverrideDoc` sidecar：裸 `ReplayEntry[]` 替换其派生脚本，`{ patches }` 则按调用索引增补该脚本。 |
|
||||
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | 主会话的可选 `ReplayOverrideDoc` 伴随文件：裸 `ReplayEntry[]` 替换其派生脚本，`{ patches }` 则按调用索引增补该脚本。 |
|
||||
| `childFiles` | string[] | `$DSH_SNAPSHOT_CHILD_FILES`（以路径分隔符分隔） | 嵌套场景中已记录的 subagent 子会话日志；单会话场景为空。 |
|
||||
| `providers` | `ReplayProviderConfig[]` | 无 | 可选的仅回放提供方和模型目录。每个提供方可以设置 `retryPolicy`，每个模型可以发布 `contextWindow`；已配置路由通过回放适配器分派，绝不执行提供方 I/O。 |
|
||||
| `paceMs` | number | 无（突发） | 可选的每分片毫秒延迟，使下游传输（例如真实浏览器观察到的 Web SSE（Server-Sent Events）多路复用器）看到真正的增量传递。它只是仿真开关，测试不得依赖它保证正确性。值必须是非负整数；pace 等待期间中止会迅速取消流。 |
|
||||
@@ -33,7 +35,7 @@ fixture 就是持久化的会话日志（`<scenario>/session.jsonl`）。其 `as
|
||||
name: '@deepseek-ai/dsh-llm-replay'
|
||||
config:
|
||||
providers:
|
||||
- id: deepseek
|
||||
- id: deepseek-official
|
||||
name: DeepSeek
|
||||
retryPolicy:
|
||||
mode: normal
|
||||
@@ -53,9 +55,9 @@ fixture 就是持久化的会话日志（`<scenario>/session.jsonl`）。其 `as
|
||||
## 导出项
|
||||
|
||||
- `installLlmReplay(ctx, config)`：安装已配置回放适配器或 catch-all `llm/stream` 监听器；返回 `ReplayHandle`（包含用于保证 HMR（热模块替换）安全的 `dispose()`，以及清理阶段执行的 `assertConsumed()` 检查；后者确保每个已记录脚本都绑定到实时会话，且每个已绑定游标都已耗尽，从而将场景静默驱动的模型调用少于记录数转换为明确诊断）。在测试中使用它，可以不通过 Loader 或 env var 驱动回放。
|
||||
- `loadSessionScripts(config)`：解析场景的有序的 `SessionScript[]`（主会话 + 子会话），准备按首次调用顺序绑定到实时会话。
|
||||
- `loadSessionScripts(config)`：解析场景中有序的 `SessionScript[]`（主会话 + 子会话），准备按首次调用顺序绑定到实时会话。
|
||||
- `loadReplayScript(config)`：只解析主会话的 `ReplayEntry[]`（如果伴随文件存在，则使用经校验的替换或补丁；否则从 JSONL 派生；fixture 缺失时明确报错）。
|
||||
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)`：将已记录会话日志转换为脚本并读取其 header `id`/`createdAt` 的纯辅助工具。派生分组必须以 `finish` 分片结束；没有该分片的分组是 `stream()` 抛出异常的指纹，必须改用 override sidecar 表达。
|
||||
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)` / `resolveScriptedEntry(entry, messages)`：将已记录会话日志转换为脚本、读取其 header `id`/`createdAt`、并针对单次实时请求解析 `{{fromRequest:...}}` 占位符的纯辅助工具。派生分组必须以 `finish` 分片结束；没有该分片的分组是 `stream()` 抛出异常的指纹，必须改用 override 伴随文件表达。
|
||||
- 类型 `ReplayEntry` / `ReplayOverrideDoc` / `ReplayOverridePatch` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`。
|
||||
|
||||
## 插件导出形态
|
||||
@@ -68,7 +70,7 @@ fixture 就是持久化的会话日志（`<scenario>/session.jsonl`）。其 `as
|
||||
|
||||
#### KV Cache 影响
|
||||
|
||||
无；该包（package）既不组装也不发送提供方请求。
|
||||
无；该包既不组装也不发送提供方请求。
|
||||
|
||||
## 已知限制与暂缓事项
|
||||
|
||||
|
||||
@@ -21,9 +21,7 @@
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"lib/types/**/*.d.ts",
|
||||
"lib/types/**/*.d.ts.map",
|
||||
"src"
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
|
||||
@@ -173,9 +173,10 @@ export function parseSessionHeader(text: string): { id: string; createdAt: numbe
|
||||
/**
|
||||
* Reconstruct the per-`stream()` replay script from a recorded session log.
|
||||
*
|
||||
* Groups `assistant/chunk` events by turn and step. Every group must end in a
|
||||
* `finish`; a missing terminator means the live stream threw, so derivation
|
||||
* rejects and the scenario must provide an explicit override.
|
||||
* Splits `assistant/chunk` events at every `finish`, using turn and step changes
|
||||
* to detect an unterminated prior call. A missing terminator means the live
|
||||
* stream threw, so derivation rejects and the scenario must provide an explicit
|
||||
* override. Multiple calls may share one turn and step when the loop retries.
|
||||
* @param events - the recorded session's events; only `assistant/chunk` is consulted.
|
||||
* @returns one `chunks` entry per recorded model call, in call order.
|
||||
*/
|
||||
@@ -197,14 +198,16 @@ export function deriveReplayScript(events: SessionEvent[]): ReplayEntry[] {
|
||||
if (event.type !== 'assistant/chunk') continue
|
||||
const { turn, step, chunk } = event.data
|
||||
const key = `${turn}/${step}`
|
||||
if (key !== currentKey) {
|
||||
// A new (turn, step) — i.e. a new stream() call. Close the previous one
|
||||
// (skip the initial empty buffer before any chunk has been seen).
|
||||
if (current.length > 0 && key !== currentKey) {
|
||||
close(currentKey, current)
|
||||
currentKey = key
|
||||
}
|
||||
if (current.length === 0) currentKey = key
|
||||
current.push(chunk)
|
||||
if (chunk.type === 'finish') {
|
||||
close(currentKey, current)
|
||||
currentKey = undefined
|
||||
current = []
|
||||
}
|
||||
current.push(chunk)
|
||||
}
|
||||
close(currentKey, current)
|
||||
return script
|
||||
@@ -241,6 +244,96 @@ const REPLAY_CHUNK_TYPES = new Set<StreamChunk['type']>([
|
||||
'finish',
|
||||
])
|
||||
|
||||
const FROM_REQUEST_OPEN = '{{fromRequest:'
|
||||
const FROM_REQUEST_CLOSE = '}}'
|
||||
|
||||
/** Collect every string leaf of one JSON-shaped value, in traversal order. */
|
||||
function collectStrings(value: unknown, out: string[]): void {
|
||||
if (typeof value === 'string') {
|
||||
out.push(value)
|
||||
return
|
||||
}
|
||||
if (Array.isArray(value)) {
|
||||
for (const item of value) collectStrings(item, out)
|
||||
return
|
||||
}
|
||||
if (value !== null && typeof value === 'object') {
|
||||
for (const item of Object.values(value)) collectStrings(item, out)
|
||||
}
|
||||
}
|
||||
|
||||
/** Resolve one placeholder pattern against the request corpus; the LAST match wins. */
|
||||
function resolveFromRequest(pattern: string, corpus: string): string {
|
||||
let regex: RegExp
|
||||
try {
|
||||
regex = new RegExp(pattern, 'g')
|
||||
} catch (error) {
|
||||
// RegExp construction only throws SyntaxError; String() carries its message.
|
||||
throw new Error(`llm-replay: fromRequest has an invalid pattern ${JSON.stringify(pattern)}: ${String(error)}`)
|
||||
}
|
||||
let last: RegExpExecArray | undefined
|
||||
for (const match of corpus.matchAll(regex)) last = match
|
||||
if (last === undefined) {
|
||||
throw new Error(`llm-replay: fromRequest pattern ${JSON.stringify(pattern)} matched nothing in the request`)
|
||||
}
|
||||
return last[1] ?? last[0]
|
||||
}
|
||||
|
||||
/** Replace every `{{fromRequest:<pattern>}}` occurrence in one scripted string. */
|
||||
function substituteString(text: string, corpus: string): string {
|
||||
let result = ''
|
||||
let cursor = 0
|
||||
while (true) {
|
||||
const open = text.indexOf(FROM_REQUEST_OPEN, cursor)
|
||||
if (open === -1) return result + text.slice(cursor)
|
||||
let close = text.indexOf(FROM_REQUEST_CLOSE, open + FROM_REQUEST_OPEN.length)
|
||||
if (close === -1) {
|
||||
throw new Error(`llm-replay: fromRequest placeholder is unterminated in ${JSON.stringify(text)}`)
|
||||
}
|
||||
// The last two braces of a consecutive `}` run terminate the placeholder,
|
||||
// so a pattern may end with a brace quantifier like `[0-9a-f]{4}`.
|
||||
while (text[close + FROM_REQUEST_CLOSE.length] === '}') close += 1
|
||||
const pattern = text.slice(open + FROM_REQUEST_OPEN.length, close)
|
||||
result += text.slice(cursor, open) + resolveFromRequest(pattern, corpus)
|
||||
cursor = close + FROM_REQUEST_CLOSE.length
|
||||
}
|
||||
}
|
||||
|
||||
/** Deep-copy one JSON-shaped value with scripted placeholders resolved. */
|
||||
function substituteValue(value: unknown, corpus: string): unknown {
|
||||
if (typeof value === 'string') {
|
||||
return value.includes(FROM_REQUEST_OPEN) ? substituteString(value, corpus) : value
|
||||
}
|
||||
if (Array.isArray(value)) return value.map(item => substituteValue(item, corpus))
|
||||
if (value !== null && typeof value === 'object') {
|
||||
return Object.fromEntries(Object.entries(value).map(([key, item]) => [key, substituteValue(item, corpus)]))
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve every `{{fromRequest:<regex>}}` placeholder in one scripted entry
|
||||
* against the live request. The corpus is every string leaf of the request
|
||||
* messages joined by newlines; the pattern's LAST corpus match wins and its
|
||||
* first capture group (or, without one, the whole match) substitutes in place.
|
||||
* Scenario sidecars use this to script arguments no static file can know,
|
||||
* such as a randomly minted goal id the model must echo back. A pattern that
|
||||
* matches nothing, an invalid pattern, and an unterminated placeholder each
|
||||
* fail loud. The last two braces of a consecutive `}` run terminate the
|
||||
* placeholder, so a pattern may end with a brace quantifier but cannot
|
||||
* contain `}}` followed by further pattern content. Derived entries pass
|
||||
* through the same resolution as sidecar entries.
|
||||
* @param entry - the scripted entry about to replay.
|
||||
* @param messages - the live request messages searched by the placeholders.
|
||||
* @returns the entry itself when no placeholder appears, else a resolved deep copy.
|
||||
*/
|
||||
export function resolveScriptedEntry(entry: ReplayEntry, messages: GenerateOptions['messages']): ReplayEntry {
|
||||
if (!JSON.stringify(entry).includes(FROM_REQUEST_OPEN)) return entry
|
||||
const leaves: string[] = []
|
||||
collectStrings(messages, leaves)
|
||||
return substituteValue(entry, leaves.join('\n')) as ReplayEntry
|
||||
}
|
||||
|
||||
function isRecord(value: unknown): value is Record<string, unknown> {
|
||||
return typeof value === 'object' && value !== null && !Array.isArray(value)
|
||||
}
|
||||
@@ -583,7 +676,7 @@ export function installLlmReplay(ctx: Context, config: ReplayConfig): ReplayHand
|
||||
+ `but its script has only ${boundState.entries.length}; re-record the scenario`,
|
||||
)
|
||||
}
|
||||
yield* replayEntry(entry, options.signal, paceMs)
|
||||
yield* replayEntry(resolveScriptedEntry(entry, options.messages), options.signal, paceMs)
|
||||
})()
|
||||
}
|
||||
const providers = config.providers ?? []
|
||||
|
||||
@@ -4,7 +4,7 @@ import { join } from 'node:path'
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import LlmService, { GenerateOptions, LlmAdapter, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import LlmService, { CallId, createUserMessage, GenerateOptions, LlmAdapter, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import {
|
||||
type ReplayEntry,
|
||||
type SessionScript,
|
||||
@@ -17,6 +17,7 @@ import {
|
||||
name,
|
||||
parseSessionHeader,
|
||||
parseSessionLog,
|
||||
resolveScriptedEntry,
|
||||
} from '../src/index.ts'
|
||||
|
||||
/**
|
||||
@@ -106,11 +107,27 @@ describe('parseSessionLog', () => {
|
||||
})
|
||||
|
||||
describe('deriveReplayScript', () => {
|
||||
it('groups assistant/chunk by (turn, step) into one entry per stream() call', () => {
|
||||
it('groups one finished assistant/chunk stream into one replay entry', () => {
|
||||
const events: SessionEvent[] = TEXT_CHUNKS.map((c, i) => chunkEvent(i + 1, 1, 1, c))
|
||||
expect(deriveReplayScript(events)).toEqual([{ kind: 'chunks', chunks: TEXT_CHUNKS }])
|
||||
})
|
||||
|
||||
it('separates retry calls that share one turn and step at their finish chunks', () => {
|
||||
const failed: StreamChunk[] = [
|
||||
{ type: 'usage', usage: { inputTokens: 0, outputTokens: 0 } },
|
||||
{ type: 'finish', reason: { kind: 'error', failure: { message: 'empty', code: 'EMPTY_RESPONSE' } } },
|
||||
]
|
||||
let seq = 1
|
||||
const events: SessionEvent[] = [
|
||||
...failed.map(chunk => chunkEvent(seq++, 1, 1, chunk)),
|
||||
...TEXT_CHUNKS.map(chunk => chunkEvent(seq++, 1, 1, chunk)),
|
||||
]
|
||||
expect(deriveReplayScript(events)).toEqual([
|
||||
{ kind: 'chunks', chunks: failed },
|
||||
{ kind: 'chunks', chunks: TEXT_CHUNKS },
|
||||
])
|
||||
})
|
||||
|
||||
it('produces one entry per distinct (turn, step), in log order', () => {
|
||||
const callA = TEXT_CHUNKS
|
||||
const callB: StreamChunk[] = [
|
||||
@@ -141,7 +158,7 @@ describe('deriveReplayScript', () => {
|
||||
it('ignores non-assistant/chunk events', () => {
|
||||
let seq = 1
|
||||
const events: SessionEvent[] = [
|
||||
{ type: 'turn/start', seq: seq++, time: 0, data: { turn: 1, trigger: { kind: 'injection', source: { kind: 'user' } } } },
|
||||
{ type: 'turn/start', seq: seq++, time: 0, data: { turn: 1 } },
|
||||
...TEXT_CHUNKS.map(c => chunkEvent(seq++, 1, 1, c)),
|
||||
{ type: 'turn/end', seq: seq++, time: 0, data: { turn: 1, reason: { kind: 'completed' } } },
|
||||
]
|
||||
@@ -166,7 +183,7 @@ describe('deriveReplayScript', () => {
|
||||
const events: SessionEvent[] = [
|
||||
chunkEvent(1, 1, 1, { type: 'block-start', index: 0, blockType: 'text' }),
|
||||
chunkEvent(2, 1, 1, { type: 'text-delta', index: 0, text: 'par' }),
|
||||
{ type: 'turn/end', seq: 3, time: 0, data: { turn: 1, reason: { kind: 'error', step: 1, message: 'x' } } },
|
||||
{ type: 'turn/end', seq: 3, time: 0, data: { turn: 1, reason: { kind: 'error', error: { message: 'x', code: 'UNKNOWN' } } } },
|
||||
]
|
||||
expect(() => deriveReplayScript(events)).toThrow(/without a finish chunk.*replay\.override\.json/s)
|
||||
})
|
||||
@@ -177,6 +194,14 @@ describe('deriveReplayScript', () => {
|
||||
]
|
||||
expect(() => deriveReplayScript(events)).toThrow(/2\/3/)
|
||||
})
|
||||
|
||||
it('rejects an unfinished call before consuming chunks from a new step', () => {
|
||||
const events: SessionEvent[] = [
|
||||
chunkEvent(1, 1, 1, { type: 'block-start', index: 0, blockType: 'text' }),
|
||||
chunkEvent(2, 1, 2, { type: 'finish', reason: { kind: 'stop' } }),
|
||||
]
|
||||
expect(() => deriveReplayScript(events)).toThrow(/model call 1\/1 ended without a finish chunk/)
|
||||
})
|
||||
})
|
||||
|
||||
describe('loadReplayScript', () => {
|
||||
@@ -310,6 +335,80 @@ describe('installLlmReplay (through the real LlmService)', () => {
|
||||
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
})
|
||||
|
||||
describe('{{fromRequest:...}} substitution', () => {
|
||||
const requestMessages = [createUserMessage({
|
||||
content: [{ type: 'text' as const, text: 'stale {"goal":{"id":"goal-old"}} then {"goal":{"id":"goal-42ab"}}' }],
|
||||
source: { kind: 'user' as const },
|
||||
})]
|
||||
|
||||
function scriptedCall(argumentsDelta: string): StreamChunk[] {
|
||||
return [
|
||||
{ type: 'block-start', index: 0, blockType: 'tool-call' },
|
||||
{ type: 'tool-call-delta', index: 0, id: CallId('c1'), name: 'update_goal', argumentsDelta },
|
||||
{ type: 'block-end', index: 0, block: { type: 'tool-call', id: CallId('c1'), name: 'update_goal', arguments: argumentsDelta } },
|
||||
{ type: 'finish', reason: { kind: 'tool-calls' } },
|
||||
]
|
||||
}
|
||||
|
||||
async function streamScripted(argumentsDelta: string): Promise<StreamChunk[]> {
|
||||
writeLog(TEXT_CHUNKS)
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
writeFileSync(overrideFile, JSON.stringify([{ kind: 'chunks', chunks: scriptedCall(argumentsDelta) }]), 'utf8')
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
installLlmReplay(ctx, { file, overrideFile })
|
||||
return drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: requestMessages }))
|
||||
}
|
||||
|
||||
it('resolves the capture group from the LAST request match in every scripted string field', async () => {
|
||||
const streamed = await streamScripted('{"goal_id":"{{fromRequest:"id":"(goal-[^"]+)"}}","revision":1}')
|
||||
const delta = streamed.find(chunk => chunk.type === 'tool-call-delta')
|
||||
expect(delta).toMatchObject({ argumentsDelta: '{"goal_id":"goal-42ab","revision":1}' })
|
||||
const end = streamed.find(chunk => chunk.type === 'block-end')
|
||||
expect(end).toMatchObject({ block: { arguments: '{"goal_id":"goal-42ab","revision":1}' } })
|
||||
})
|
||||
|
||||
it('substitutes the whole match when the pattern has no capture group', async () => {
|
||||
const streamed = await streamScripted('{"goal_id":"{{fromRequest:goal-[0-9a-z]+}}"}')
|
||||
const delta = streamed.find(chunk => chunk.type === 'tool-call-delta')
|
||||
expect(delta).toMatchObject({ argumentsDelta: '{"goal_id":"goal-42ab"}' })
|
||||
})
|
||||
|
||||
it('keeps a trailing brace quantifier inside the pattern (terminator is the run tail)', async () => {
|
||||
const streamed = await streamScripted('{"goal_id":"{{fromRequest:goal-[0-9a-z]{4}}}"}')
|
||||
const delta = streamed.find(chunk => chunk.type === 'tool-call-delta')
|
||||
expect(delta).toMatchObject({ argumentsDelta: '{"goal_id":"goal-42ab"}' })
|
||||
})
|
||||
|
||||
it('fails loud when a placeholder matches nothing in the request', async () => {
|
||||
await expect(streamScripted('{"goal_id":"{{fromRequest:task-[0-9]+}}"}'))
|
||||
.rejects.toThrow(/fromRequest.*matched nothing/)
|
||||
})
|
||||
|
||||
it('fails loud on an invalid placeholder pattern', async () => {
|
||||
await expect(streamScripted('{"goal_id":"{{fromRequest:(goal-}}"}'))
|
||||
.rejects.toThrow(/fromRequest.*invalid pattern/)
|
||||
})
|
||||
|
||||
it('fails loud on an unterminated placeholder', () => {
|
||||
const entry: ReplayEntry = { kind: 'chunks', chunks: scriptedCall('{"goal_id":"{{fromRequest:goal-1"}') }
|
||||
expect(() => resolveScriptedEntry(entry, requestMessages)).toThrow(/fromRequest placeholder is unterminated/)
|
||||
})
|
||||
|
||||
it('returns the exact same entry when no placeholder appears', () => {
|
||||
const entry: ReplayEntry = { kind: 'chunks', chunks: TEXT_CHUNKS }
|
||||
expect(resolveScriptedEntry(entry, requestMessages)).toBe(entry)
|
||||
})
|
||||
|
||||
it('skips non-string request leaves when building the corpus', () => {
|
||||
const messages = requestMessages.map(message => ({ ...message, seq: 7 })) as unknown as GenerateOptions['messages']
|
||||
const entry: ReplayEntry = { kind: 'chunks', chunks: scriptedCall('{"goal_id":"{{fromRequest:goal-42[a-z]+}}"}') }
|
||||
const resolved = resolveScriptedEntry(entry, messages)
|
||||
if (resolved.kind !== 'chunks') throw new Error('expected chunks entry')
|
||||
expect(resolved.chunks[1]).toMatchObject({ argumentsDelta: '{"goal_id":"goal-42ab"}' })
|
||||
})
|
||||
})
|
||||
|
||||
it('registers a replay-only provider catalog when configured', async () => {
|
||||
writeLog(TEXT_CHUNKS)
|
||||
const ctx = new Context()
|
||||
|
||||
@@ -21,9 +21,7 @@
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"lib/types/**/*.d.ts",
|
||||
"lib/types/**/*.d.ts.map",
|
||||
"src"
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
|
||||
@@ -141,6 +141,13 @@ export interface LoaderSmokeOptions {
|
||||
readonly prepare?: (cwd: string) => Promise<void> | void
|
||||
/** Optional world-state assertion run in the isolated cwd before cleanup. */
|
||||
readonly inspect?: (cwd: string) => Promise<void> | void
|
||||
/**
|
||||
* Exact process exit code this smoke expects; defaults to `0`. Scenarios
|
||||
* pinning a designed failure surface (a one-shot turn ending in an error
|
||||
* result) declare its nonzero exit here, and a run that exits any other
|
||||
* way — including succeeding — still fails the smoke.
|
||||
*/
|
||||
readonly expectedExitCode?: number
|
||||
}
|
||||
|
||||
/** Captured output from a Loader smoke that exited successfully. */
|
||||
@@ -187,8 +194,9 @@ export async function runLoaderSmoke(options: LoaderSmokeOptions): Promise<Loade
|
||||
if (result.timedOut) {
|
||||
throw new Error(`${options.label} did not exit within ${processTimeoutMs / 1_000}s. stdout:\n${result.stdout}\nstderr:\n${result.stderr}`)
|
||||
}
|
||||
if (result.failed) {
|
||||
throw new Error(`${options.label} exited ${String(result.exitCode)}. stdout:\n${result.stdout}\nstderr:\n${result.stderr}`)
|
||||
const expectedExitCode = options.expectedExitCode ?? 0
|
||||
if (result.exitCode !== expectedExitCode) {
|
||||
throw new Error(`${options.label} exited ${String(result.exitCode)} (expected ${expectedExitCode}). stdout:\n${result.stdout}\nstderr:\n${result.stderr}`)
|
||||
}
|
||||
await options.inspect?.(cwd)
|
||||
return { stdout: result.stdout, stderr: result.stderr }
|
||||
|
||||
@@ -8,7 +8,8 @@ import { LOADER_SMOKE_TEST_TIMEOUT_MS, runLoaderSmoke } from '@deepseek-ai/dsh-l
|
||||
const configPath = '/tmp/fixture.cordis.yml'
|
||||
const tsconfigPath = fileURLToPath(new URL('../../../../tsconfig.json', import.meta.url))
|
||||
const fixture = (name: string): string => fileURLToPath(new URL(`./fixtures/${name}.ts`, import.meta.url))
|
||||
const canonicalTempPath = (path: string): string => path.replace(/^\/private(?=\/var\/)/, '')
|
||||
// macOS realpaths temp dirs into /private; TMPDIR may live under /var or /tmp.
|
||||
const canonicalTempPath = (path: string): string => path.replace(/^\/private(?=\/(?:var|tmp)\/)/, '')
|
||||
|
||||
describe('runLoaderSmoke', () => {
|
||||
it('isolates the process, closes stdin, captures output, and removes the cwd', async () => {
|
||||
@@ -74,7 +75,32 @@ describe('runLoaderSmoke', () => {
|
||||
libBinScript: fixture('fail'),
|
||||
configPath,
|
||||
tsconfigPath,
|
||||
})).rejects.toThrow('failure fixture exited 7. stdout:\n\nstderr:\nfixture failed')
|
||||
})).rejects.toThrow('failure fixture exited 7 (expected 0). stdout:\n\nstderr:\nfixture failed')
|
||||
})
|
||||
|
||||
it('accepts a declared expected failure exit and rejects any other outcome', async () => {
|
||||
// A scenario pinning a designed failure surface declares its exit code…
|
||||
const declared = await runLoaderSmoke({
|
||||
label: 'declared failure fixture',
|
||||
tempDirPrefix: 'loader-smoke-declared-fail-',
|
||||
binScript: fixture('fail'),
|
||||
libBinScript: fixture('fail'),
|
||||
configPath,
|
||||
tsconfigPath,
|
||||
expectedExitCode: 7,
|
||||
})
|
||||
expect(declared.stderr).toBe('fixture failed\n')
|
||||
|
||||
// …and a run that succeeds instead still fails the smoke.
|
||||
await expect(runLoaderSmoke({
|
||||
label: 'unexpectedly clean fixture',
|
||||
tempDirPrefix: 'loader-smoke-clean-',
|
||||
binScript: fixture('success'),
|
||||
libBinScript: fixture('success'),
|
||||
configPath,
|
||||
tsconfigPath,
|
||||
expectedExitCode: 7,
|
||||
})).rejects.toThrow(/exited 0 \(expected 7\)/)
|
||||
})
|
||||
|
||||
it('kills a process at its deadline and reports captured output', async () => {
|
||||
|
||||
Reference in New Issue
Block a user