Merge remote-tracking branch 'origin/master' into nih-imp-execa

# Conflicts:
#	.agents/notes/rejected/simplification/2026-07-26-dependency-swaps-rejected-by-nih-audit.i18n.yaml
#	apps/web/tests/scaffold.ts
#	packages/session-persistence/session-checkpoint-policy/tests/crash-recovery.e2e.ts
#	packages/support/acp-snapshot/README.i18n.yaml
This commit is contained in:
Tianyi Cui
2026-07-27 06:31:17 +08:00
665 changed files with 11055 additions and 7377 deletions

View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write
README.md: 8babb67c30aed87ace4cfff81b2494a03a5b0335
README.zh.md: 3f43627740054e200e928ffe527f827c710599b0
README.md: d22e6e2d95a1ed930a7f4876daf4b06e2f761f7a
README.zh.md: 514b7ebfe02cb34ed559633a0fd82cf6194fa4b3

View File

@@ -8,9 +8,11 @@ Four layers, importable separately:
- **`launchAcpTestAgent` (launcher)** — boots a source agent under tsx or a built `lib` agent under plain Node from a supplied cwd, connects the SDK client over a raw-byte stdout tee, collects session updates and stderr, surfaces asynchronous spawn failures through startup, fails closed on unhandled permission requests, and owns graceful or signalled shutdown. Shutdown waits for process exit, inherited stdio closure, and ACP parser exhaustion before resolving or propagating a child error, so captures are complete and callers can remove owned paths after either outcome. When Windows accepts forced termination but publishes its exit marker asynchronously, shutdown gives that marker a bounded grace before treating fallback refusal as a second failure. Snapshot and ordinary e2e suites share this process boundary; a test supplies only agent paths, cwd, environment overrides, and any permission policy.
- **`runScenario` (harness)** — drives ACP JSON-RPC stdio from a deterministic `input.json` script through the launcher, tees raw stdout for the expected-output and purity checks, and harvests every persisted raw JSONL session log (parent and subagent children, primary-first) after graceful stdin EOF. `AgentUnderTest` supplies absolute `binScript`, optional `libBinScript`, `configPath`, and `tsconfigPath` paths because the subprocess cwd is outside the repo; `workspaceParent` may move the generated child cwd from the platform temp directory when that grant is itself under test. Startup failures preserve captured agent stderr in the rejected diagnostic.
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs and every native/JavaScript filesystem spelling of the generated cwd → tokens, longest-first; cwd-rooted separators selected as canonical `/` or host-native; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept, the same cwd-path policy), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header Agent Note](../../../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs and every native/JavaScript filesystem spelling of the generated cwd → tokens, longest-first; cwd-rooted separators selected as canonical `/` or host-native; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept, the same cwd-path policy), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header Agent Note](../../../.agents/notes/archived/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario expected-output and re-persisted-log comparisons, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, the per-header-class pin (`system-prompt.expected.md` plus `tool-schemas.expected.json`) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt/schema-scrubbed, non-pinning fixtures fully header-scrubbed). Refresh expands packed timing envelopes before aligning existing volatile event times, so switching between packed and unpacked layouts cannot shift later records; fresh chunk-fragment arrays remain authoritative. A newly inserted `session/title` receives its preceding event's time so feature-driven insertions do not churn the remainder of a fixture. Each scenario directory's `session.jsonl` plus contiguous `session.<n>.jsonl` siblings are the ordered primary/child inventory; the scenario table does not duplicate their count. Must be called at vitest collection time.
Committed session fixtures use canonical packed rows. An in-flight branch that merges this contract runs the [temporary repository migrator](../../../scripts/migrate-packed-session-fixtures.ts) with `pnpm run migrate:packed-session-fixtures`; its [removal proposal](../../../.agents/notes/proposed/process/2026-07-26-remove-packed-session-fixture-migrator.md) owns deletion after affected branches converge.
A consuming `*.snapshot.ts` is the scenario table plus one factory call:
```ts
@@ -53,7 +55,7 @@ A scenario booting a differently-composed tree sets its own `configPath` (an ove
Every scenario compares `stdout.expected.jsonl` with cwd-rooted separators canonicalized to `/`. On Windows, `pinsNativeWindowsStdout` additionally compares the complete `stdout.expected.windows.jsonl` after the shared expected output and requires that sidecar exactly when enabled. A scenario whose driven behavior needs POSIX process semantics (e.g. cancelling a live bash call kills a detached process group) declares `posixOnly`, which skips its run test on Windows while the fixture guards keep covering its committed files everywhere.
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config Agent Note](../../../.agents/notes/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log expected outputs, and each pin's prompt and tool-schema sidecars from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md).
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config Agent Note](../../../.agents/notes/archived/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log expected outputs, and each pin's prompt and tool-schema sidecars from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md).
Constraints: `suite.ts` and `harness.ts` import vitest (the harness polls its durable-boundary waits through `vi.waitFor`), so the package entry is importable only inside a vitest run (the launcher and normalizers have no such dependency but ship from the same entry). The launcher and suite factory are ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection` — while the normalizers are transport-neutral session-log/text helpers also consumed by the TUI snapshot suite and the web browser e2e lane. Input scripts cover initialization, fresh-session creation, text prompting, cancellation, expected RPC failures, and durable turn-boundary waits. Permission round-trips are a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) mapped to the agent-issued `optionId`; an absent or exhausted queue answers `cancelled`, and an unoffered kind rejects the run.

View File

@@ -8,9 +8,11 @@ ACP 快照套件工具包:无密钥快照层(`pnpm run test:snapshot`,见[
- **`launchAcpTestAgent`(启动器)**:从指定 cwd 在 tsx 下启动源 agent或在普通 Node 下启动已构建 `lib` agent通过原始字节 stdout tee 连接 SDK 客户端,收集会话更新和 stderr在启动过程中公开异步 spawn 失败,对未处理权限请求快速失败,并负责优雅或带信号关闭。关闭会等待进程退出、继承 stdio 关闭和 ACP parser 耗尽,然后才解析或传播子级错误,使捕获内容完整,且调用方可在任一结果后移除自有路径。当 Windows 接受强制终止但异步发布退出标记时,关闭会给该标记有界宽限,然后才将回退拒绝视为第二次失败。快照和普通 e2e 套件共享该进程边界;测试只需提供 agent 路径、cwd、环境覆盖和任何权限策略。
- **`runScenario`harness**:通过启动器从确定性 `input.json` 脚本驱动 ACP JSON-RPC stdio将原始 stdout tee 给预期输出和纯度检查,并在优雅 stdin EOF 后收集每个持久化原始 JSONL 会话日志(父级和 subagent 子级,主级优先)。`AgentUnderTest` 提供绝对 `binScript`、可选 `libBinScript``configPath``tsconfigPath` 路径,因为子进程 cwd 位于仓库外。当生成子级 cwd 自身位于待测授权中时,`workspaceParent` 可以将它从平台临时目录移出。启动失败会在拒绝诊断中保留已捕获 agent stderr。
- **规范化器**:将两个已捕获接口转换为稳定文本的纯函数:`normalizeStdout`JSON-RPC id → 首次出现序列UUID 以及生成 cwd 的每个原生/JavaScript 文件系统写法 → token按最长优先根据 cwd 的分隔符选择规范 `/` 或宿主原生形式;同时作为 stdout 纯度检查)、`normalizeSessionLog`(时间归零、保留 `seq`、使用同一 cwd 路径策略)、`scrubSystemPrompts`(提示词文本 → `{{system}}`)、`scrubToolSchemas`schema bulk → `{{tools}}`)和 `scrubRequestHeaders`(每个 pin 之外的所有 header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}`,保留结构;见[header 固定 Agent Note](../../../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md))。
- **规范化器**:将两个已捕获接口转换为稳定文本的纯函数:`normalizeStdout`JSON-RPC id → 首次出现序列UUID 以及生成 cwd 的每个原生/JavaScript 文件系统写法 → token按最长优先根据 cwd 的分隔符选择规范 `/` 或宿主原生形式;同时作为 stdout 纯度检查)、`normalizeSessionLog`(时间归零、保留 `seq`、使用同一 cwd 路径策略)、`scrubSystemPrompts`(提示词文本 → `{{system}}`)、`scrubToolSchemas`schema bulk → `{{tools}}`)和 `scrubRequestHeaders`(每个 pin 之外的所有 header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}`,保留结构;见[header 固定 Agent Note](../../../.agents/notes/archived/testing/2026-07-06-pin-request-header-content-in-one-scenario.md))。
- **`defineAcpSnapshotSuite`(工厂)**:为场景表注册完整 describe/it 树:每场景预期输出与重新持久化日志比较、录制/刷新 fixture 回写、拒绝结构化 `UNKNOWN_TOOL` 结果、每 header 类别 pin`system-prompt.expected.md``tool-schemas.expected.json`)及其实时一致性保护,以及 fixture 保护块(无遗留场景目录、必需文件存在、每类别恰好一个 pin、每个 JSONL 的提示词/schema 已擦除、非 pin fixture 的 header 已完全擦除)。刷新会在对齐现有可变事件时间前展开打包时序 envelope因此切换打包/非打包布局无法移动后续记录;新分片碎片数组仍为权威数据。新插入的 `session/title` 使用前一个事件的时间,因此功能驱动的插入不会扰动 fixture 余下部分。每个场景目录的 `session.jsonl` 和连续 `session.<n>.jsonl` 同级文件是有序主级/子级清单;场景表不重复其数量。必须在 vitest 收集时调用。
签入仓库的会话 fixture 使用规范打包行。合并此契约的在途分支通过 `pnpm run migrate:packed-session-fixtures` 运行[临时仓库迁移器](../../../scripts/migrate-packed-session-fixtures.ts);待受影响分支收敛后,由其[移除提案](../../../.agents/notes/proposed/process/2026-07-26-remove-packed-session-fixture-migrator.md)负责删除该迁移器。
消费方 `*.snapshot.ts` 就是场景表加一次工厂调用:
```ts
@@ -53,7 +55,7 @@ defineAcpSnapshotSuite({
每个场景都比较 `stdout.expected.jsonl`,其中以 cwd 为根的分隔符规范化为 `/`。在 Windows 上,`pinsNativeWindowsStdout` 还会在共享预期输出之后比较完整 `stdout.expected.windows.jsonl`,并在启用时精确要求该 sidecar。驱动行为需要 POSIX 进程语义的场景(例如取消实时 bash 调用会终止脱离进程组)声明 `posixOnly`,在 Windows 上跳过运行测试,但 fixture 保护仍在所有平台覆盖其已提交文件。
示例还发布 `cordis.snapshot.yml` 回放 overlay位于 `cordis.yml` 旁边bin 在 `DSH_SNAPSHOT=replay` 下交换它们,见[单源回放配置 Agent Note](../../../.agents/notes/implemented/testing/2026-07-04-single-source-acp-replay-config.md));回放 fixture 由 [`dsh-llm-replay`](../llm-replay/README.md) 提供,该包通过对子级设置的 `DSH_SNAPSHOT_*` env var 指向它。`pnpm run test:snapshot:record` 调用实时 LLM并重写已记录场景的模型 fixture`pnpm run test:snapshot:refresh` 保持无密钥,运行回放 overlay并从已提交模型脚本重写 stdout、可比较会话日志预期输出以及每个 pin 的提示词与工具 schema sidecar。Fixture 角色、录制/回放/刷新语义和场景表字段记录在 `Scenario` 以及[快照 Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md) 中。
示例还发布 `cordis.snapshot.yml` 回放 overlay位于 `cordis.yml` 旁边bin 在 `DSH_SNAPSHOT=replay` 下交换它们,见[单源回放配置 Agent Note](../../../.agents/notes/archived/testing/2026-07-04-single-source-acp-replay-config.md));回放 fixture 由 [`dsh-llm-replay`](../llm-replay/README.md) 提供,该包通过对子级设置的 `DSH_SNAPSHOT_*` env var 指向它。`pnpm run test:snapshot:record` 调用实时 LLM并重写已记录场景的模型 fixture`pnpm run test:snapshot:refresh` 保持无密钥,运行回放 overlay并从已提交模型脚本重写 stdout、可比较会话日志预期输出以及每个 pin 的提示词与工具 schema sidecar。Fixture 角色、录制/回放/刷新语义和场景表字段记录在 `Scenario` 以及[快照 Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md) 中。
约束:`suite.ts``harness.ts` 导入 vitestharness 通过 `vi.waitFor` 轮询其持久边界等待),因此包入口只能在 vitest 运行中导入(启动器和规范化器没有此依赖,但从同一入口发布)。启动器和套件工厂按设计专用于 ACP启动器使用 SDK 的 `ClientSideConnection`;规范化器是与传输无关的会话日志/文本辅助工具,还由 TUI 快照套件和 web 浏览器 e2e lane 消费。输入脚本覆盖初始化、新建会话、文本提示、取消、预期 RPC 失败和持久轮次边界等待。权限往返是选项类别选择(`allow_once``reject_once`等)的 FIFO 队列,映射到 agent 发出的 `optionId`;缺少或耗尽的队列回答 `cancelled`,未提供类别会拒绝运行。

View File

@@ -71,12 +71,13 @@ export interface Scenario {
recorded: boolean
/**
* Whether replay is driven by a hand-written `replay.override.json` sidecar
* (a `ReplayEntry[]` that REPLACES the script derived from `session.jsonl`)
* — the throw/hang cases chunks cannot express. The fixture guard requires
* the sidecar exactly when this is set: the harness forwards the file purely
* on existence, so an unregistered stray sidecar would silently replace the
* derived script — the guard fails loud on either mismatch. Defaults to
* false (replay derives from the fixture's `assistant/chunk` events).
* (a `ReplayOverrideDoc` that replaces or patches the script derived from
* `session.jsonl`) — the throw/hang cases chunks cannot express. The fixture
* guard requires the sidecar exactly when this is set: the harness forwards
* the file purely on existence, so an unregistered stray sidecar would
* silently alter the derived script. The guard fails loud on either
* mismatch. Defaults to false (replay derives from the fixture's
* `assistant/chunk` events).
*/
overridden?: boolean
/**

View File

@@ -1,5 +1,5 @@
import { request } from 'node:http'
import { afterEach, describe, expect, it, vi } from 'vitest'
import { afterEach, describe, expect, it } from 'vitest'
import type { MockLlmBehavior, MockLlmServer, MockLlmServerEvent } from '../src/index.ts'
import { startMockLlmServer } from '../src/index.ts'
@@ -169,21 +169,23 @@ describe('mock LLM server wire behaviors', () => {
['partial_disconnect', 100] as const,
])('records a client that closes during %s', async (behavior, delayMs) => {
const events: MockLlmServerEvent[] = []
const result = Promise.withResolvers<Extract<MockLlmServerEvent, { type: 'result' }>>()
const server = await start([behavior], {
chunkDelayMs: delayMs,
disconnectDelayMs: delayMs,
chunkSize: 1,
onEvent: (event) => { events.push(event) },
onEvent: (event) => {
events.push(event)
if (event.type === 'result') result.resolve(event)
},
})
const controller = new AbortController()
const response = await chat(server, { signal: controller.signal })
controller.abort()
await expect(response.text()).rejects.toThrow()
// The server observes the socket close asynchronously; a fixed sleep
// raced slow runners, so poll until the outcome lands.
await vi.waitFor(() => {
expect(server.requests[0]).toMatchObject({ behavior, outcome: 'client_closed' })
})
await result.promise
expect(server.requests[0]).toMatchObject({ behavior, outcome: 'client_closed' })
expect(events.filter(event => event.type === 'result')).toEqual([
expect.objectContaining({ behavior, outcome: 'client_closed' }),
])

View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write
README.md: 901a3b7b4312fffd93e6d375c378e39064318260
README.zh.md: b9a8068d329e28933c934e7ad352ac65641f3d23
README.md: ce0758641f3d49a54b29415ed449e43043840f9a
README.zh.md: 47a2b9aa211b44c4e476a1adf5a9a72d927cd0ed

View File

@@ -10,7 +10,7 @@ Its consumers are the ACP, headless `stream-json`, and TUI snapshot suites plus
The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`", done by the snapshot harness — this plugin does not record. A fixture may carry its `request/header` content tokenized to `{{system}}`/`{{tools}}` (the harness pins that content in one scenario and scrubs the rest); replay is indifferent — derivation reads only `assistant/chunk` events and the line-0 session header.
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`: a `ReplayEntry[]`) that REPLACES the derived script. A `hang` entry may name `readyFile`; replay writes that empty marker after its prefix chunks reach the loop and before it waits for cancellation, so an external driver can cancel deterministically without observing a presentation update.
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`) that either replaces the derived script (a bare `ReplayEntry[]`) or augments it (`{ patches: [{ at, entry }] }`: keep every JSONL-derived call and swap the named 0-based call indexes; `at` equal to the derived length appends the retry attempt after an injected transient throw). Patch indexes must be unique. The override document, each patch and entry, and every chunk discriminant are validated when the file loads. A `hang` entry may name `readyFile`; replay writes that empty marker after its prefix chunks reach the loop and before it waits for cancellation, so an external driver can cancel deterministically without observing a presentation update.
## Nested agents: per-session keying
@@ -23,7 +23,7 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
| Key | Type | Default | Notes |
|---|---|---|---|
| `file` | string | `$DSH_SNAPSHOT_FILE` | Path to the primary (parent) `session.jsonl` fixture. Required (config or env). |
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | Optional path to a `ReplayEntry[]` sidecar that replaces the PRIMARY session's derived script. |
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | Optional `ReplayOverrideDoc` sidecar for the primary session: a bare `ReplayEntry[]` replaces its derived script, while `{ patches }` augments it by call index. |
| `childFiles` | string[] | `$DSH_SNAPSHOT_CHILD_FILES` (path-delimited) | Recorded subagent child-session logs for a nested scenario; empty for a single-session scenario. |
| `providers` | `ReplayProviderConfig[]` | — | Optional replay-only provider and model catalog. Each model may publish `contextWindow`; configured routes dispatch through the replay adapter and never perform provider I/O. |
| `paceMs` | number | — (burst) | Optional per-chunk delay in ms so downstream transports (e.g. the web SSE mux observed by a real browser) see genuinely incremental delivery. A realism knob only — tests must not depend on it for correctness. Non-negative integer; abort during a pace wait cancels the stream promptly. |
@@ -48,9 +48,9 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
- `installLlmReplay(ctx, config)` — install the configured replay adapter or catch-all `llm/stream` listener; returns a `ReplayHandle` (`dispose()` for HMR safety plus `assertConsumed()`, the teardown check that every recorded script bound to a live session and every bound cursor drained — turning a scenario that silently drove fewer model calls than recorded into a crisp diagnostic). Use this in tests to drive replay without the Loader or env vars.
- `loadSessionScripts(config)` — resolve the ordered `SessionScript[]` (primary + children) for a scenario, ready to bind to live sessions in first-call order.
- `loadReplayScript(config)` — resolve the `ReplayEntry[]` for the PRIMARY session only (sidecar override if present, else derived from the JSONL; fail-loud if the fixture is missing).
- `loadReplayScript(config)` — resolve the `ReplayEntry[]` for the primary session only (validated sidecar replacement/patches if present, else derived from the JSONL; fail-loud if the fixture is missing).
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)` — the pure helpers that turn a recorded session log into a script and read its header `id`/`createdAt`. A derived group must end in a `finish` chunk; a group without one is the fingerprint of a thrown `stream()` and must instead be expressed via an override sidecar.
- Types `ReplayEntry` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`.
- Types `ReplayEntry` / `ReplayOverrideDoc` / `ReplayOverridePatch` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`.
## Plugin export shape
@@ -67,4 +67,4 @@ None; this package neither assembles nor sends a provider request.
## Known Limitations and Deferred Work
- **First-call-order script binding assumes sequential delegation** — a cut that runs sibling subagents concurrently (or a compaction summarize call landing mid-run) would bind live sessions to recorded scripts non-deterministically; a stronger keying is deferred until such a scenario exists (`XXX(concurrent-subagents)`).
- **Only chunk-producing calls are derivable** — a pure pre-chunk throw or a cancel/hang scenario needs the `replay.override.json` sidecar; the override replaces the PRIMARY session's script only.
- **Only chunk-producing calls are derivable** — a pure pre-chunk throw or a cancel/hang scenario needs the `replay.override.json` sidecar. Replacement and patch forms affect only the primary session; child scripts still derive from their logs.

View File

@@ -10,7 +10,7 @@
Fixture 就是持久化会话日志(`<scenario>/session.jsonl`)。其 `assistant/chunk` 事件携带每个 `StreamChunk`,因此按 `(turn, step)` 对其分组可重建每次 `stream()` 调用的分片序列(每个 loop 步骤一次模型调用)。因此,录制操作是「运行一次真实 agent 并收集 `.jsonl`」,由快照 harness 完成该插件不执行录制。Fixture 的 `request/header` 内容可能被 token 化为 `{{system}}`/`{{tools}}`harness 在一个场景中固定该内容,并擦除其余场景);回放对此并不关心,因为派生只读取 `assistant/chunk` 事件和第 0 行会话 header。
有两种失败 mode 无法仅从 `assistant/chunk` 重建:在任何分片前纯抛出(例如 HTTP 401日志只包含 `turn/end {error}` 而没有分片),以及 cancel/hang是时序而非分片内容。需要这些的场景提供可选 sidecar`<scenario>/replay.override.json`:一个 `ReplayEntry[]`以替换派生脚本`hang` 条目可以指定 `readyFile`;在其前缀分片到达 loop 后、等待取消前,回放会写入该空标记,使外部驱动器可以在不观察展示更新的情况下确定性取消。
有两种失败 mode 无法仅从 `assistant/chunk` 重建:在任何分片前纯抛出(例如 HTTP 401日志只包含 `turn/end {error}` 而没有分片),以及 cancel/hang是时序而非分片内容。需要这些的场景提供可选 sidecar`<scenario>/replay.override.json`),它要么替换派生脚本(裸 `ReplayEntry[]`要么增补派生脚本(`{ patches: [{ at, entry }] }`:保留全部由 JSONL 派生的调用,仅在点名的调用索引处换入,索引从 0 计;`at` 等于派生长度时为追加正是注入的瞬态抛出之后那次重试尝试所占的槽位。Patch 索引必须互不重复。覆写文档、每个 patch 与每个条目,以及每个分片的判别字段都会在文件加载时接受校验`hang` 条目可以指定 `readyFile`;在其前缀分片到达 loop 后、等待取消前,回放会写入该空标记,使外部驱动器可以在不观察展示更新的情况下确定性取消。
## 嵌套 agent每会话键控
@@ -23,7 +23,7 @@ Fixture 就是持久化会话日志(`<scenario>/session.jsonl`)。其 `assis
| 键 | 类型 | 默认值 | 说明 |
|---|---|---|---|
| `file` | string | `$DSH_SNAPSHOT_FILE` | 主(父)`session.jsonl` fixture 的路径。必需(配置或 env。 |
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | 替换主会话派生脚本的 `ReplayEntry[]` sidecar 可选路径。 |
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | 主会话的可选 `ReplayOverrideDoc` sidecar`ReplayEntry[]` 替换其派生脚本,`{ patches }` 则按调用索引增补该脚本。 |
| `childFiles` | string[] | `$DSH_SNAPSHOT_CHILD_FILES` (path-delimited) | 嵌套场景中已记录的 subagent 子会话日志;单会话场景为空。 |
| `providers` | `ReplayProviderConfig[]` | 无 | 可选的仅回放提供方和模型目录。每个模型可以发布 `contextWindow`;已配置路由通过回放适配器分派,绝不执行提供方 I/O。 |
| `paceMs` | number | 无(突发) | 可选的每分片毫秒延迟,使下游传输(例如真实浏览器观察的 web SSE mux看到真正的增量传递。它只是仿真开关测试不得依赖它保证正确性。值必须是非负整数pace 等待期间中止会迅速取消流。 |
@@ -48,9 +48,9 @@ Fixture 就是持久化会话日志(`<scenario>/session.jsonl`)。其 `assis
- `installLlmReplay(ctx, config)`:安装已配置回放适配器或 catch-all `llm/stream` 监听器;返回 `ReplayHandle`(包含用于 HMR 安全的 `dispose()`,以及 `assertConsumed()` 拆卸检查;后者确保每个已记录脚本都绑定到实时会话,且每个已绑定游标都已耗尽,从而将场景静默驱动的模型调用少于记录数转换为明确诊断)。在测试中使用它,可以不通过 Loader 或 env var 驱动回放。
- `loadSessionScripts(config)`:解析场景的有序 `SessionScript[]` (主级 + 子级),准备按首次调用顺序绑定到实时会话。
- `loadReplayScript(config)`:只解析主会话的 `ReplayEntry[]` (如果存在则使用 sidecar override,否则从 JSONL 派生fixture 缺失时快速失败)。
- `loadReplayScript(config)`:只解析主会话的 `ReplayEntry[]` (如果存在则使用经校验的 sidecar 替换或 patch,否则从 JSONL 派生fixture 缺失时快速失败)。
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)`:将已记录会话日志转换为脚本并读取其 header `id`/`createdAt` 的纯辅助工具。派生分组必须以 `finish` 分片结束;没有该分片的分组是已抛出 `stream()` 的指纹,必须改用 override sidecar 表达。
- 类型 `ReplayEntry` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`
- 类型 `ReplayEntry` / `ReplayOverrideDoc` / `ReplayOverridePatch` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`
## 插件导出形态
@@ -67,4 +67,4 @@ Fixture 就是持久化会话日志(`<scenario>/session.jsonl`)。其 `assis
## 已知限制与待完成工作
- **首次调用顺序脚本绑定假设串行委托**:并发运行同级 subagent 的 cut或运行中落地的压缩摘要调用会非确定性地将实时会话绑定到已记录脚本在这种场景出现前暂不实现更强的键控`XXX(concurrent-subagents)`)。
- **只有生产分片的调用可派生**:纯分片前抛出或 cancel/hang 场景需要 `replay.override.json` sidecaroverride 只替换主会话的脚本
- **只有生产分片的调用可派生**:纯分片前抛出或 cancel/hang 场景需要 `replay.override.json` sidecar。替换和 patch 两种形式都只影响主会话;子会话脚本仍从各自日志派生

View File

@@ -59,10 +59,11 @@ export interface ReplayConfig {
*/
file: string
/**
* Optional `ReplayEntry[]` sidecar that REPLACES the derived script for the
* PRIMARY session. Used by the two single-session scenarios not expressible as
* `assistant/chunk` (pure throw-before-chunk, cancel/hang). Absent for normal
* and nested scenarios.
* Optional sidecar for the PRIMARY session: a bare `ReplayEntry[]` replaces
* the derived script; `{ patches }` keeps it and swaps the named call
* indexes ({@link ReplayOverrideDoc}). Used by single-session scenarios not
* expressible as `assistant/chunk` (throw-before-chunk, cancel/hang,
* injected transient failures). Absent for normal and nested scenarios.
*/
overrideFile?: string
/**
@@ -200,26 +201,157 @@ export function deriveReplayScript(events: SessionEvent[]): ReplayEntry[] {
}
/**
* Build the replay script for the PRIMARY session: the sidecar override if
* present, otherwise the script derived from the recorded session JSONL.
* Fail-loud if the JSONL fixture is missing (the scenario was never recorded) —
* never silently returns an empty script, so a coverage hole can't masquerade
* as a passing replay.
* One positional patch in an augmentation sidecar: replaces the derived
* entry at call index `at` (0-based) with `entry`, or appends when `at`
* equals the derived length (an extra recorded-after-the-fact call, e.g. the
* retry attempt following an injected transient throw).
*/
export interface ReplayOverridePatch {
/** 0-based call index into the derived script; == length appends. */
at: number
/** The replacement (or appended) entry at that call position. */
entry: ReplayEntry
}
/**
* Override sidecar document: either a whole-script replacement (a
* bare `ReplayEntry[]`) or the augmentation form `{ patches }`, which keeps
* the JSONL-derived script and swaps only the named call indexes — the shape
* for "turn N errors, everything else replays as recorded".
*/
export type ReplayOverrideDoc = ReplayEntry[] | { patches: ReplayOverridePatch[] }
const REPLAY_CHUNK_TYPES = new Set<StreamChunk['type']>([
'block-start',
'text-delta',
'reasoning-delta',
'tool-call-delta',
'block-end',
'usage',
'finish',
])
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === 'object' && value !== null && !Array.isArray(value)
}
function hasExactKeys(value: Record<string, unknown>, keys: readonly string[]): boolean {
return Object.keys(value).length === keys.length && keys.every(key => Object.hasOwn(value, key))
}
function invalidOverride(file: string, location: string, detail: string): never {
throw new Error(`llm-replay: invalid override ${file}: ${location} ${detail}`)
}
function readChunks(value: unknown, file: string, location: string): StreamChunk[] {
if (!Array.isArray(value)) invalidOverride(file, location, 'chunks must be an array')
for (const [index, chunk] of value.entries()) {
if (!isRecord(chunk)
|| typeof chunk['type'] !== 'string'
|| !REPLAY_CHUNK_TYPES.has(chunk['type'] as StreamChunk['type'])) {
invalidOverride(file, `${location}.chunks[${index}]`, 'must have a known StreamChunk type')
}
}
return value as StreamChunk[]
}
function readReplayEntry(value: unknown, file: string, location: string): ReplayEntry {
if (!isRecord(value)) invalidOverride(file, location, 'must be an object')
switch (value['kind']) {
case 'chunks': {
if (!hasExactKeys(value, ['kind', 'chunks'])) invalidOverride(file, location, 'has invalid chunks-entry fields')
return { kind: 'chunks', chunks: readChunks(value['chunks'], file, location) }
}
case 'throw': {
if (!hasExactKeys(value, ['kind', 'chunks', 'message', 'code'])) {
invalidOverride(file, location, 'has invalid throw-entry fields')
}
if (typeof value['message'] !== 'string' || value['message'].length === 0) {
invalidOverride(file, location, 'message must be a non-empty string')
}
if (typeof value['code'] !== 'string' || value['code'].length === 0) {
invalidOverride(file, location, 'code must be a non-empty string')
}
return {
kind: 'throw',
chunks: readChunks(value['chunks'], file, location),
message: value['message'],
code: value['code'],
}
}
case 'hang': {
const readyFile = value['readyFile']
const keys = readyFile === undefined ? ['kind'] : ['kind', 'readyFile']
if (!hasExactKeys(value, keys)) invalidOverride(file, location, 'has invalid hang-entry fields')
if (readyFile !== undefined && (typeof readyFile !== 'string' || readyFile.length === 0)) {
invalidOverride(file, location, 'readyFile must be a non-empty string')
}
return { kind: 'hang', ...(readyFile === undefined ? {} : { readyFile }) }
}
default:
return invalidOverride(file, location, `has unknown kind ${JSON.stringify(value['kind'])}`)
}
}
function readOverrideDoc(value: unknown, file: string): ReplayOverrideDoc {
if (Array.isArray(value)) return value.map((entry, index) => readReplayEntry(entry, file, `entry ${index}`))
if (!isRecord(value) || !hasExactKeys(value, ['patches']) || !Array.isArray(value['patches'])) {
return invalidOverride(file, 'document', 'must be a ReplayEntry[] or { patches: [...] }')
}
return {
patches: value['patches'].map((value, index): ReplayOverridePatch => {
const location = `patch ${index}`
if (!isRecord(value) || !hasExactKeys(value, ['at', 'entry'])) {
return invalidOverride(file, location, 'must contain exactly at and entry')
}
const at = value['at']
if (typeof at !== 'number' || !Number.isSafeInteger(at) || at < 0) {
return invalidOverride(file, location, 'at must be a non-negative safe integer')
}
return { at, entry: readReplayEntry(value['entry'], file, `${location}.entry`) }
}),
}
}
/**
* Load the PRIMARY session's replay script: the sidecar override when present
* (whole-script replacement or `{ patches }` augmentation over the derived
* script), else the script derived from the session JSONL (fail-loud when the
* fixture is missing).
* @param config - the fixture paths; only `file` and `overrideFile` are consulted.
* @returns the primary session's replay entries.
* @returns the resolved primary-session script.
*/
export function loadReplayScript(config: ReplayConfig): ReplayEntry[] {
if (config.overrideFile !== undefined && existsSync(config.overrideFile)) {
const parsed: unknown = JSON.parse(readFileSync(config.overrideFile, 'utf8'))
if (!Array.isArray(parsed)) {
throw new Error(`llm-replay: override is not a JSON array: ${config.overrideFile}`)
const doc = readOverrideDoc(JSON.parse(readFileSync(config.overrideFile, 'utf8')) as unknown, config.overrideFile)
if (Array.isArray(doc)) return doc
const script = deriveScriptFromFile(config.file)
const derivedLength = script.length
const seenIndexes = new Set<number>()
for (const patch of doc.patches) {
if (patch.at > derivedLength) {
throw new Error(
`llm-replay: override patch index ${String(patch.at)} out of range `
+ `(derived script has ${derivedLength} call(s); == length appends): ${config.overrideFile}`,
)
}
if (seenIndexes.has(patch.at)) {
throw new Error(`llm-replay: duplicate override patch index ${patch.at}: ${config.overrideFile}`)
}
seenIndexes.add(patch.at)
script[patch.at] = patch.entry
}
return parsed as ReplayEntry[]
return script
}
if (!existsSync(config.file)) {
throw new Error(`llm-replay: fixture not found: ${config.file} — run \`pnpm run test:snapshot:record\` first`)
return deriveScriptFromFile(config.file)
}
/** Derive the primary script from the session JSONL, failing loud on a missing fixture. */
function deriveScriptFromFile(file: string): ReplayEntry[] {
if (!existsSync(file)) {
throw new Error(`llm-replay: fixture not found: ${file} — run \`pnpm run test:snapshot:record\` first`)
}
return deriveReplayScript(parseSessionLog(readFileSync(config.file, 'utf8')))
return deriveReplayScript(parseSessionLog(readFileSync(file, 'utf8')))
}
/**
@@ -359,9 +491,8 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined,
})
/* v8 ignore next -- unreachable: the hang promise only ever rejects (on abort), never resolves; control never reaches here */
return
/* v8 ignore next -- sidecar entries are validated before they reach the closed local union. */
default:
// Closed local union: an unknown kind means malformed (hand-edited or
// drifted) sidecar data — fail loud with a runtime diagnostic.
return assertNever(entry, 'llm-replay replay entry')
}
}

View File

@@ -203,11 +203,91 @@ describe('loadReplayScript', () => {
expect(() => loadReplayScript({ file: join(dir, 'absent.jsonl') })).toThrow(/fixture not found/)
})
it('throws when the override is not a JSON array', () => {
it('rejects an override document that is neither supported form', () => {
writeFileSync(file, sessionJsonl([]), 'utf8')
const overrideFile = join(dir, 'replay.override.json')
writeFileSync(overrideFile, '{"not":"array"}', 'utf8')
expect(() => loadReplayScript({ file, overrideFile })).toThrow(/not a JSON array/)
expect(() => loadReplayScript({ file, overrideFile })).toThrow(/document must be a ReplayEntry\[\] or \{ patches/)
})
it('patches form: swaps the named call index and keeps derived siblings', () => {
const callB: StreamChunk[] = [
{ type: 'block-start', index: 0, blockType: 'text' },
{ type: 'text-delta', index: 0, text: 'two' },
{ type: 'finish', reason: { kind: 'stop' } },
]
let seq = 1
writeFileSync(file, sessionJsonl([
...TEXT_CHUNKS.map(c => chunkEvent(seq++, 1, 1, c)),
...callB.map(c => chunkEvent(seq++, 1, 2, c)),
]), 'utf8')
const overrideFile = join(dir, 'replay.override.json')
writeFileSync(overrideFile, JSON.stringify({
patches: [{ at: 0, entry: { kind: 'throw', chunks: [], message: 'transient', code: 'SERVER' } }],
}), 'utf8')
expect(loadReplayScript({ file, overrideFile })).toEqual([
{ kind: 'throw', chunks: [], message: 'transient', code: 'SERVER' },
{ kind: 'chunks', chunks: callB },
])
})
it('patches form: at == derived length appends (the retry-attempt slot)', () => {
writeFileSync(file, sessionJsonl(TEXT_CHUNKS.map((c, i) => chunkEvent(i + 1, 1, 1, c))), 'utf8')
const overrideFile = join(dir, 'replay.override.json')
writeFileSync(overrideFile, JSON.stringify({
patches: [
{ at: 0, entry: { kind: 'throw', chunks: [], message: '429', code: 'RATE_LIMIT' } },
{ at: 1, entry: { kind: 'chunks', chunks: TEXT_CHUNKS } },
],
}), 'utf8')
expect(loadReplayScript({ file, overrideFile })).toEqual([
{ kind: 'throw', chunks: [], message: '429', code: 'RATE_LIMIT' },
{ kind: 'chunks', chunks: TEXT_CHUNKS },
])
})
it('patches form: an out-of-range index fails loud with the derived length', () => {
writeFileSync(file, sessionJsonl(TEXT_CHUNKS.map((c, i) => chunkEvent(i + 1, 1, 1, c))), 'utf8')
const overrideFile = join(dir, 'replay.override.json')
writeFileSync(overrideFile, JSON.stringify({ patches: [{ at: 2, entry: { kind: 'hang' } }] }), 'utf8')
expect(() => loadReplayScript({ file, overrideFile })).toThrow(/patch index 2 out of range.*1 call/s)
})
it('validates patch and entry shapes at the file boundary', () => {
writeFileSync(file, sessionJsonl([]), 'utf8')
const overrideFile = join(dir, 'replay.override.json')
const invalid: Array<{ doc: unknown; message: RegExp }> = [
{ doc: null, message: /document must be/ },
{ doc: { patches: [null] }, message: /patch 0 must contain exactly at and entry/ },
{ doc: { patches: [{ at: -1, entry: { kind: 'hang' } }] }, message: /at must be a non-negative safe integer/ },
{ doc: { patches: [{ at: 1.5, entry: { kind: 'hang' } }] }, message: /at must be a non-negative safe integer/ },
{ doc: [42], message: /entry 0 must be an object/ },
{ doc: [{ kind: 'chunks', chunks: 'nope' }], message: /chunks must be an array/ },
{ doc: [{ kind: 'chunks', chunks: [], extra: true }], message: /invalid chunks-entry fields/ },
{ doc: [{ kind: 'chunks', chunks: [{ type: 'bogus' }] }], message: /known StreamChunk type/ },
{ doc: [{ kind: 'throw', chunks: [], message: 'nope', code: 'AUTH', extra: true }], message: /invalid throw-entry fields/ },
{ doc: [{ kind: 'throw', chunks: [], message: '', code: 'AUTH' }], message: /message must be a non-empty string/ },
{ doc: [{ kind: 'throw', chunks: [], message: 'nope', code: '' }], message: /code must be a non-empty string/ },
{ doc: [{ kind: 'hang', extra: true }], message: /invalid hang-entry fields/ },
{ doc: [{ kind: 'hang', readyFile: 1 }], message: /readyFile must be a non-empty string/ },
{ doc: [{ kind: 'bogus' }], message: /unknown kind/ },
]
for (const { doc, message } of invalid) {
writeFileSync(overrideFile, JSON.stringify(doc), 'utf8')
expect(() => loadReplayScript({ file, overrideFile })).toThrow(message)
}
})
it('rejects duplicate patch indexes instead of silently taking the last one', () => {
writeFileSync(file, sessionJsonl(TEXT_CHUNKS.map((c, i) => chunkEvent(i + 1, 1, 1, c))), 'utf8')
const overrideFile = join(dir, 'replay.override.json')
writeFileSync(overrideFile, JSON.stringify({
patches: [
{ at: 0, entry: { kind: 'hang' } },
{ at: 0, entry: { kind: 'throw', chunks: [], message: 'busy', code: 'SERVER' } },
],
}), 'utf8')
expect(() => loadReplayScript({ file, overrideFile })).toThrow(/duplicate override patch index 0/)
})
})
@@ -364,16 +444,14 @@ describe('installLlmReplay (through the real LlmService)', () => {
.toEqual([{ type: 'finish', reason: { kind: 'stop' } }])
})
it('throws on a malformed sidecar entry kind (the assertNever guard)', async () => {
it('rejects a malformed sidecar entry kind before installing replay', async () => {
writeFileSync(file, sessionJsonl([]), 'utf8')
const overrideFile = join(dir, 'replay.override.json')
// A kind the union does not know — hand-edited/drifted sidecar data.
writeFileSync(overrideFile, JSON.stringify([{ kind: 'bogus' }]), 'utf8')
const ctx = new Context()
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file, overrideFile })
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })))
.rejects.toThrow(/llm-replay replay entry/)
expect(() => installLlmReplay(ctx, { file, overrideFile })).toThrow(/unknown kind/)
})
it('rejects a hang entry when the signal fires DURING the wait (abort listener path)', async () => {