Merge remote-tracking branch 'origin/master' into worktree/llm-reasoning-effort
# Conflicts: # .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.i18n.yaml # docs/config-catalog.md # docs/core-data-structures/session.i18n.yaml # scripts/type-equiv.manifest.json
This commit is contained in:
Yichen Jiang
6
packages/support/README.i18n.yaml
Normal file
6
packages/support/README.i18n.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
README.md: b9550fd54feb36448227faae8485fe8b6dbf4fb0
|
||||
README.zh.md: dc017c5f6544e29533f14600deb5b9261e351536
|
||||
@@ -1,5 +1,7 @@
|
||||
# support/ — dev/test/example infrastructure
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
Packages that exist to serve development, testing, and the examples rather than to ship as product API. They are real workspace packages (typed, tested, under the coverage gate), but they carry **lower compatibility expectations**: they may change or be removed when the development need behind them does, without the deprecation care a product package would warrant.
|
||||
|
||||
| Package | Role | ctx key |
|
||||
|
||||
16
packages/support/README.zh.md
Normal file
16
packages/support/README.zh.md
Normal file
@@ -0,0 +1,16 @@
|
||||
# support/：开发／测试／示例基础设施
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
这些包用于开发、测试和示例，而非作为产品 API 发布。它们是真实工作区包（有类型、经过测试、受覆盖率门禁约束），但具有**较低的兼容性预期**：当其背后的开发需求变化时，它们可以改变或被移除，无需像产品包那样谨慎执行弃用流程。
|
||||
|
||||
| 包 | 职责 | ctx 键 |
|
||||
|---|---|---|
|
||||
| `acp-snapshot/` | ACP 测试工具包：共享子进程/客户端启动器、快照 harness、规范化器和套件工厂 | （库：由 ACP e2e 和 `*.snapshot.ts` 套件导入） |
|
||||
| `agent-loop-testkit/` | 为测试具体 agent loop 的测试挂载共享先决条件 | （库：由 AgentLoop 集成测试导入） |
|
||||
| `invariants/` | 用于开发诊断的运行时事件契约断言 | （监听 `session/*`、`agent/*`） |
|
||||
| `loader-smoke/` | 共享的真实 Loader 子进程 harness，用于无密钥示例冒烟测试 | （库：由示例 e2e 套件导入） |
|
||||
| `llm-mock-server/` | 可编脚本的 OpenAI 兼容 HTTP/SSE 故障服务器与 CLI，用于 LLM 恢复测试 | （独立服务器和测试库） |
|
||||
| `llm-replay/` | 录制/回放适配器：从已记录的会话 JSONL 短路 `llm/stream`（无密钥快照测试） | （监听 `llm/stream`） |
|
||||
|
||||
`invariants` 是开发支持，但没有环境保护：无论在何处注册，它都会运行；默认 `dsh-agent-spine-demo` bundle 无条件挂载它。`agent-loop-testkit` 为手工构建的 AgentLoop 测试集中管理必需服务主干，而不负责其 loop 或场景。`llm-replay` 支撑演示和受每文件覆盖率门禁约束的快照测试层，`llm-mock-server` 则通过确定性 HTTP/SSE 故障驱动真实提供方适配器。`acp-snapshot` 包含 ACP 子进程/客户端边界以及快照 harness、规范化器和套件机制，`loader-smoke` 负责无密钥示例 e2e 套件使用的并列真实 Loader 启动边界。只有当某个包获得已记录的产品消费方时，它才会从 `support/` 升级到产品分组。
|
||||
6
packages/support/acp-snapshot/README.i18n.yaml
Normal file
6
packages/support/acp-snapshot/README.i18n.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
README.md: d35872e5bb06be88dc5999bfa1800083b2fbbf3c
|
||||
README.zh.md: a706b6db5408538c578cb2a1cfc3aa99804a930d
|
||||
@@ -1,14 +1,18 @@
|
||||
# `@deepseek-ai/dsh-acp-snapshot`
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
The ACP snapshot suite kit: the shared machinery behind the keyless snapshot tier (`pnpm run test:snapshot`, [testing policy](../../../docs/testing.md)). An example gets a full snapshot suite from a scenario table plus a fixtures directory; every compare/guard mechanic lives here, under the per-file coverage gate, instead of being copied per example.
|
||||
|
||||
Four layers, importable separately:
|
||||
|
||||
- **`launchAcpTestAgent` (launcher)** — boots a source agent under tsx or a built `lib` agent under plain Node from a supplied cwd, connects the SDK client over a raw-byte stdout tee, collects session updates and stderr, surfaces asynchronous spawn failures through startup, fails closed on unhandled permission requests, and owns graceful or signalled shutdown. Shutdown waits for process exit, inherited stdio closure, and ACP parser exhaustion before resolving or propagating a child error, so captures are complete and callers can remove owned paths after either outcome. When Windows accepts forced termination but publishes its exit marker asynchronously, shutdown gives that marker a bounded grace before treating fallback refusal as a second failure. Snapshot and ordinary e2e suites share this process boundary; a test supplies only agent paths, cwd, environment overrides, and any permission policy.
|
||||
- **`runScenario` (harness)** — drives ACP JSON-RPC stdio from a deterministic `input.json` script through the launcher, tees raw stdout for the expected-output and purity checks, and harvests every persisted raw JSONL session log (parent and subagent children, primary-first) after graceful stdin EOF. `AgentUnderTest` supplies absolute `binScript`, optional `libBinScript`, `configPath`, and `tsconfigPath` paths because the subprocess cwd is outside the repo; `workspaceParent` may move the generated child cwd from the platform temp directory when that grant is itself under test. Startup failures preserve captured agent stderr in the rejected diagnostic.
|
||||
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs and every native/JavaScript filesystem spelling of the generated cwd → tokens, longest-first; cwd-rooted separators selected as canonical `/` or host-native; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept, the same cwd-path policy), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header Agent Note](../../../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
|
||||
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs and every native/JavaScript filesystem spelling of the generated cwd → tokens, longest-first; cwd-rooted separators selected as canonical `/` or host-native; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept, the same cwd-path policy), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header Agent Note](../../../.agents/notes/archived/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
|
||||
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario expected-output and re-persisted-log comparisons, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, the per-header-class pin (`system-prompt.expected.md` plus `tool-schemas.expected.json`) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt/schema-scrubbed, non-pinning fixtures fully header-scrubbed). Refresh expands packed timing envelopes before aligning existing volatile event times, so switching between packed and unpacked layouts cannot shift later records; fresh chunk-fragment arrays remain authoritative. A newly inserted `session/title` receives its preceding event's time so feature-driven insertions do not churn the remainder of a fixture. Each scenario directory's `session.jsonl` plus contiguous `session.<n>.jsonl` siblings are the ordered primary/child inventory; the scenario table does not duplicate their count. Must be called at vitest collection time.
|
||||
|
||||
Committed session fixtures use canonical packed rows. An in-flight branch that merges this contract runs the [temporary repository migrator](../../../scripts/migrate-packed-session-fixtures.ts) with `pnpm run migrate:packed-session-fixtures`; its [removal proposal](../../../.agents/notes/proposed/process/2026-07-26-remove-packed-session-fixture-migrator.md) owns deletion after affected branches converge.
|
||||
|
||||
A consuming `*.snapshot.ts` is the scenario table plus one factory call:
|
||||
|
||||
```ts
|
||||
@@ -51,7 +55,7 @@ A scenario booting a differently-composed tree sets its own `configPath` (an ove
|
||||
|
||||
Every scenario compares `stdout.expected.jsonl` with cwd-rooted separators canonicalized to `/`. On Windows, `pinsNativeWindowsStdout` additionally compares the complete `stdout.expected.windows.jsonl` after the shared expected output and requires that sidecar exactly when enabled. A scenario whose driven behavior needs POSIX process semantics (e.g. cancelling a live bash call kills a detached process group) declares `posixOnly`, which skips its run test on Windows while the fixture guards keep covering its committed files everywhere.
|
||||
|
||||
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config Agent Note](../../../.agents/notes/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log expected outputs, and each pin's prompt and tool-schema sidecars from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md).
|
||||
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config Agent Note](../../../.agents/notes/archived/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log expected outputs, and each pin's prompt and tool-schema sidecars from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md).
|
||||
|
||||
Constraints: `suite.ts` imports vitest, so the package entry is importable only inside a vitest run (the launcher, harness, and normalizers have no such dependency but ship from the same entry). The launcher and suite factory are ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection` — while the normalizers are transport-neutral session-log/text helpers also consumed by the TUI snapshot suite and the web browser e2e lane. Input scripts cover initialization, fresh-session creation, text prompting, cancellation, expected RPC failures, and durable turn-boundary waits. Permission round-trips are a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) mapped to the agent-issued `optionId`; an absent or exhausted queue answers `cancelled`, and an unoffered kind rejects the run.
|
||||
|
||||
|
||||
74
packages/support/acp-snapshot/README.zh.md
Normal file
74
packages/support/acp-snapshot/README.zh.md
Normal file
@@ -0,0 +1,74 @@
|
||||
# `@deepseek-ai/dsh-acp-snapshot`
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
ACP 快照套件工具包：无密钥快照层（`pnpm run test:snapshot`，见[测试策略](../../../docs/testing.md)）背后的共享机制。示例只需场景表和 fixture 目录就能获得完整快照套件；每项比较/保护机制都位于此处，受每文件覆盖率门禁约束，而不是在每个示例中复制。
|
||||
|
||||
四层可单独导入：
|
||||
|
||||
- **`launchAcpTestAgent`（启动器）**：从指定 cwd 在 tsx 下启动源 agent，或在普通 Node 下启动已构建 `lib` agent；通过原始字节 stdout tee 连接 SDK 客户端，收集会话更新和 stderr，在启动过程中公开异步 spawn 失败，对未处理权限请求快速失败，并负责优雅或带信号关闭。关闭会等待进程退出、继承 stdio 关闭和 ACP parser 耗尽，然后才解析或传播子级错误，使捕获内容完整，且调用方可在任一结果后移除自有路径。当 Windows 接受强制终止但异步发布退出标记时，关闭会给该标记有界宽限，然后才将回退拒绝视为第二次失败。快照和普通 e2e 套件共享该进程边界；测试只需提供 agent 路径、cwd、环境覆盖和任何权限策略。
|
||||
- **`runScenario`（harness）**：通过启动器从确定性 `input.json` 脚本驱动 ACP JSON-RPC stdio，将原始 stdout tee 给预期输出和纯度检查，并在优雅 stdin EOF 后收集每个持久化原始 JSONL 会话日志（父级和 subagent 子级，主级优先）。`AgentUnderTest` 提供绝对 `binScript`、可选 `libBinScript`、`configPath` 和 `tsconfigPath` 路径，因为子进程 cwd 位于仓库外。当生成子级 cwd 自身位于待测授权中时，`workspaceParent` 可以将它从平台临时目录移出。启动失败会在拒绝诊断中保留已捕获 agent stderr。
|
||||
- **规范化器**：将两个已捕获接口转换为稳定文本的纯函数：`normalizeStdout`（JSON-RPC id → 首次出现序列；UUID 以及生成 cwd 的每个原生/JavaScript 文件系统写法 → token，按最长优先；根据 cwd 的分隔符选择规范 `/` 或宿主原生形式；同时作为 stdout 纯度检查）、`normalizeSessionLog`（时间归零、保留 `seq`、使用同一 cwd 路径策略）、`scrubSystemPrompts`（提示词文本 → `{{system}}`）、`scrubToolSchemas`（schema bulk → `{{tools}}`）和 `scrubRequestHeaders`（每个 pin 之外的所有 header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}`，保留结构；见[header 固定 Agent Note](../../../.agents/notes/archived/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)）。
|
||||
- **`defineAcpSnapshotSuite`（工厂）**：为场景表注册完整 describe/it 树：每场景预期输出与重新持久化日志比较、录制/刷新 fixture 回写、拒绝结构化 `UNKNOWN_TOOL` 结果、每 header 类别 pin（`system-prompt.expected.md` 加 `tool-schemas.expected.json`）及其实时一致性保护，以及 fixture 保护块（无遗留场景目录、必需文件存在、每类别恰好一个 pin、每个 JSONL 的提示词/schema 已擦除、非 pin fixture 的 header 已完全擦除）。刷新会在对齐现有可变事件时间前展开打包时序 envelope，因此切换打包/非打包布局无法移动后续记录；新分片碎片数组仍为权威数据。新插入的 `session/title` 使用前一个事件的时间，因此功能驱动的插入不会扰动 fixture 余下部分。每个场景目录的 `session.jsonl` 和连续 `session.<n>.jsonl` 同级文件是有序主级/子级清单；场景表不重复其数量。必须在 vitest 收集时调用。
|
||||
|
||||
签入仓库的会话 fixture 使用规范打包行。合并此契约的在途分支通过 `pnpm run migrate:packed-session-fixtures` 运行[临时仓库迁移器](../../../scripts/migrate-packed-session-fixtures.ts)；待受影响分支收敛后，由其[移除提案](../../../.agents/notes/proposed/process/2026-07-26-remove-packed-session-fixture-migrator.md)负责删除该迁移器。
|
||||
|
||||
消费方 `*.snapshot.ts` 就是场景表加一次工厂调用：
|
||||
|
||||
```ts
|
||||
import { dirname, join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import {
|
||||
defineAcpSnapshotSuite,
|
||||
type Scenario,
|
||||
type SnapshotSuiteOptions,
|
||||
} from '@deepseek-ai/dsh-acp-snapshot'
|
||||
|
||||
function snapshotMode(value: string | undefined): SnapshotSuiteOptions['mode'] {
|
||||
switch (value) {
|
||||
case undefined:
|
||||
case '':
|
||||
case 'replay': return 'replay'
|
||||
case 'record': return 'record'
|
||||
case 'refresh': return 'refresh'
|
||||
default: throw new Error(`unknown DSH_SNAPSHOT mode: ${value}`)
|
||||
}
|
||||
}
|
||||
|
||||
const SCENARIOS: Scenario[] = [
|
||||
{ name: 'text-turn', hasModelTurn: true, recorded: true, pinsHeader: true },
|
||||
]
|
||||
|
||||
defineAcpSnapshotSuite({
|
||||
agent: { // absolute paths, resolved from the suite's own location
|
||||
binScript: fileURLToPath(new URL('../../../packages/examples/acp-demo/src/bin.ts', import.meta.url)),
|
||||
configPath: fileURLToPath(new URL('../cordis.yml', import.meta.url)),
|
||||
tsconfigPath: fileURLToPath(new URL('../../../tsconfig.json', import.meta.url)),
|
||||
},
|
||||
snapshotsDir: join(dirname(fileURLToPath(import.meta.url)), 'snapshots'),
|
||||
scenarios: SCENARIOS, // exactly one entry per header class sets pinsHeader
|
||||
mode: snapshotMode(process.env.DSH_SNAPSHOT),
|
||||
})
|
||||
```
|
||||
|
||||
启动不同组合树的场景会设置自己的 `configPath`（一个 basename 仍以 `cordis.yml` 结尾的 overlay，使 bin 的回放交换可找到同级 `*cordis.snapshot.yml`）；当该组合改变请求 header 时，还会设置自己的 `headerClass` 和 pin 场景，acp-agent 示例的 Code Mode 与文件系统场景是模板。当临时目录授权自身待测时，`workspaceParent` 将生成 cwd 移出平台临时区域；harness 仍只拥有并移除生成的子级。每个 pin 目录将规范化的完整提示词序列存入生成的 `system-prompt.expected.md`，将对应完整工具 schema 序列存入生成的 `tool-schemas.expected.json`；`session.jsonl` 存储 `"system":"{{system}}","tools":"{{tools}}"`，同时保留配置、原因和任何模型可见前缀。具有合法运行中 header 变更的 pin 声明 `expectedHeaderChanges`，用于固定两个 sidecar 序列的长度。
|
||||
|
||||
每个场景都比较 `stdout.expected.jsonl`，其中以 cwd 为根的分隔符规范化为 `/`。在 Windows 上，`pinsNativeWindowsStdout` 还会在共享预期输出之后比较完整 `stdout.expected.windows.jsonl`，并在启用时精确要求该 sidecar。驱动行为需要 POSIX 进程语义的场景（例如取消实时 bash 调用会终止脱离进程组）声明 `posixOnly`，在 Windows 上跳过运行测试，但 fixture 保护仍在所有平台覆盖其已提交文件。
|
||||
|
||||
示例还发布 `cordis.snapshot.yml` 回放 overlay，位于 `cordis.yml` 旁边（bin 在 `DSH_SNAPSHOT=replay` 下交换它们，见[单源回放配置 Agent Note](../../../.agents/notes/archived/testing/2026-07-04-single-source-acp-replay-config.md)）；回放 fixture 由 [`dsh-llm-replay`](../llm-replay/README.md) 提供，该包通过对子级设置的 `DSH_SNAPSHOT_*` env var 指向它。`pnpm run test:snapshot:record` 调用实时 LLM，并重写已记录场景的模型 fixture；`pnpm run test:snapshot:refresh` 保持无密钥，运行回放 overlay，并从已提交模型脚本重写 stdout、可比较会话日志预期输出，以及每个 pin 的提示词与工具 schema sidecar。Fixture 角色、录制/回放/刷新语义和场景表字段记录在 `Scenario` 以及[快照 Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md) 中。
|
||||
|
||||
约束：`suite.ts` 导入 vitest，因此包入口只能在 vitest 运行中导入（启动器、harness 和规范化器没有此依赖，但从同一入口发布）。启动器和套件工厂按设计专用于 ACP，启动器使用 SDK 的 `ClientSideConnection`；规范化器是与传输无关的会话日志/文本辅助工具，还由 TUI 快照套件和 web 浏览器 e2e lane 消费。输入脚本覆盖初始化、新建会话、文本提示、取消、预期 RPC 失败和持久轮次边界等待。权限往返是选项类别选择（`allow_once`、`reject_once`等）的 FIFO 队列，映射到 agent 发出的 `optionId`；缺少或耗尽的队列回答 `cancelled`，未提供类别会拒绝运行。
|
||||
|
||||
## 模型体验
|
||||
|
||||
无。该测试专用 harness 记录、规范化并比较 ACP transcript，不会改变 agent 组装的模型请求。
|
||||
|
||||
#### KV 缓存影响
|
||||
|
||||
无；该包既不组装也不发送提供方请求。
|
||||
|
||||
## 已知限制与待完成工作
|
||||
|
||||
- **会话收集需要原始 JSONL mode**：`runScenario` 收集持久化 `.jsonl` 日志，因此快照配置使用 `persistenceCompression: 'none'`；压缩 JSONL 和 SQLite 组合没有快照收集路径。
|
||||
- **构建 mode 需要当前产物**：先运行 `pnpm run build`，再选择 `DSH_EXAMPLE_MODE=lib`；源 mode 仍是零构建路径。
|
||||
- **后端覆盖仍使用 ACP 驱动器**：保留场景为何使用该传输，见[仅自动化 ACP 决策](../../../.agents/notes/implemented/simplification/2026-07-23-acp-automation-only-protocol.md#snapshot-boundary)。
|
||||
@@ -71,12 +71,13 @@ export interface Scenario {
|
||||
recorded: boolean
|
||||
/**
|
||||
* Whether replay is driven by a hand-written `replay.override.json` sidecar
|
||||
* (a `ReplayEntry[]` that REPLACES the script derived from `session.jsonl`)
|
||||
* — the throw/hang cases chunks cannot express. The fixture guard requires
|
||||
* the sidecar exactly when this is set: the harness forwards the file purely
|
||||
* on existence, so an unregistered stray sidecar would silently replace the
|
||||
* derived script — the guard fails loud on either mismatch. Defaults to
|
||||
* false (replay derives from the fixture's `assistant/chunk` events).
|
||||
* (a `ReplayOverrideDoc` that replaces or patches the script derived from
|
||||
* `session.jsonl`) — the throw/hang cases chunks cannot express. The fixture
|
||||
* guard requires the sidecar exactly when this is set: the harness forwards
|
||||
* the file purely on existence, so an unregistered stray sidecar would
|
||||
* silently alter the derived script. The guard fails loud on either
|
||||
* mismatch. Defaults to false (replay derives from the fixture's
|
||||
* `assistant/chunk` events).
|
||||
*/
|
||||
overridden?: boolean
|
||||
/**
|
||||
|
||||
6
packages/support/agent-loop-testkit/README.i18n.yaml
Normal file
6
packages/support/agent-loop-testkit/README.i18n.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
README.md: 18c46069d3cfd402c83b5ecab68458667738163b
|
||||
README.zh.md: 93af58a279558007c252ef0734cc95b2ad79a5a5
|
||||
@@ -1,5 +1,7 @@
|
||||
# `@deepseek-ai/dsh-agent-loop-testkit`
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
Shared prerequisite mounting for tests that exercise the concrete `AgentLoop`. `mountAgentLoopTestDependencies(ctx, options?)` installs the LLM, session, system-prompt, tool, and agent services in dependency order, then returns before the loop is mounted.
|
||||
|
||||
The caller registers adapters and optional plugins, mounts `AgentLoop` with the configuration under test, and disposes its own Context. System-prompt and tool-registry configuration can be forwarded through `options`; the helper does not provide test defaults beyond those owned by the services. A plugin-load failure rejects the helper call, while services activated earlier in the sequence remain owned by the caller's Context.
|
||||
|
||||
33
packages/support/agent-loop-testkit/README.zh.md
Normal file
33
packages/support/agent-loop-testkit/README.zh.md
Normal file
@@ -0,0 +1,33 @@
|
||||
# `@deepseek-ai/dsh-agent-loop-testkit`
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
为测试具体 `AgentLoop` 的测试挂载共享先决条件。`mountAgentLoopTestDependencies(ctx, options?)` 按依赖顺序安装 LLM、会话、系统提示词、工具和 agent 服务，然后在 loop 挂载前返回。
|
||||
|
||||
调用方注册适配器和可选插件，使用待测配置挂载 `AgentLoop`，并 dispose 自己的 Context。系统提示词和工具注册表配置可通过 `options` 转发；辅助工具不提供超出服务自有默认值的测试默认值。插件加载失败会拒绝辅助工具调用，而顺序中较早激活的服务仍归调用方的 Context 所有。
|
||||
|
||||
```ts
|
||||
import { Context } from 'cordis'
|
||||
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
|
||||
import { mountAgentLoopTestDependencies } from '@deepseek-ai/dsh-agent-loop-testkit'
|
||||
|
||||
const ctx = new Context()
|
||||
|
||||
await mountAgentLoopTestDependencies(ctx)
|
||||
// Register the test adapter and any optional plugins here.
|
||||
await ctx.plugin(AgentLoop, { agents: [] })
|
||||
```
|
||||
|
||||
针对注入失败、部分拓扑、服务加载顺序或服务拆卸的测试会直接挂载其依赖，而不使用此辅助工具。
|
||||
|
||||
## 模型体验
|
||||
|
||||
无。该测试专用组合辅助工具既不驱动也不修改模型请求。
|
||||
|
||||
#### KV 缓存影响
|
||||
|
||||
无；该包既不组装也不发送提供方请求。
|
||||
|
||||
## 已知限制与待完成工作
|
||||
|
||||
- **只共享必需的先决主干**：适配器、可选插件、`AgentLoop`、agent 和 Context 拆卸仍由调用方负责，以使场景专用顺序保持可见。
|
||||
6
packages/support/invariants/README.i18n.yaml
Normal file
6
packages/support/invariants/README.i18n.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
README.md: 203dbd5ad09f5b1378061fbf9adcff885889eae2
|
||||
README.zh.md: e101a30046c6b1ac18870c268ea6a2c960cb5d90
|
||||
@@ -1,5 +1,7 @@
|
||||
# dsh-invariants
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
Configurable registry service for package-owned runtime invariant checks. The root plugin registers `ctx.invariants`; it contains no product checks or product-package imports. Every workspace package publishes a `./invariant` companion that registers its exact npm package name.
|
||||
|
||||
## Service: `InvariantService` (`ctx.invariants`)
|
||||
|
||||
85
packages/support/invariants/README.zh.md
Normal file
85
packages/support/invariants/README.zh.md
Normal file
@@ -0,0 +1,85 @@
|
||||
# dsh-invariants
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
用于包自有运行时不变量检查的可配置注册表服务。根插件注册 `ctx.invariants`；它不包含产品检查或产品包导入。每个工作区包都发布一个 `./invariant` 配套入口，用于注册其精确 NPM 包名。
|
||||
|
||||
## 服务：`InvariantService`（`ctx.invariants`）
|
||||
|
||||
```ts
|
||||
interface Config {
|
||||
enabled?: boolean
|
||||
package_allowlist?: string[]
|
||||
package_blocklist?: string[]
|
||||
}
|
||||
```
|
||||
|
||||
默认值为 `enabled: true`、`package_allowlist: []` 和 `package_blocklist: []`。只有在服务启用、allowlist 为空或至少一个 allowlist pattern 匹配完整 NPM 名称，且没有 blocklist pattern 匹配时，包才被选中。因此，blocklist 匹配优先于 allowlist 匹配。
|
||||
|
||||
每个条目都是区分大小写的 JavaScript 正则表达式源，使用 `new RegExp(pattern)` 编译。除非源提供 `^` 和 `$`，否则匹配不锚定；不解析 `/pattern/flags` 语法。同一列表中的空白、带前后空白、无效或重复条目会使服务启动失败。有效 pattern 可以不匹配任何当前已加载包，以使后续加载和 HMR 保持确定性。
|
||||
|
||||
`ctx.invariants.register(packageName, installer)` 为完整 NPM 包名保留一个活动注册，即使过滤器使其 installer 保持非活动，并返回 disposer。已启用贡献在专用子 Cordis fiber 中运行。Installer 可以通过 `installer.inject` 声明所需服务接口，并收到 `fail(message)`；后者抛出绑定到注册包的 `InvariantError`。在注册成功前，系统会等待同步或异步 installer 完成；失败会 dispose 子级，并原子释放归属。
|
||||
|
||||
服务拥有每个注册 fiber，返回的 disposer 同时属于配套 fiber。卸载任一侧都会移除监听器、跟踪状态和保留。因此，配套入口可以重新加载并注册同一包名，而不保留旧状态。由会话支撑的配套入口从持久事件重建 baseline；仅实时配套入口观察重新加载后开始的操作。
|
||||
|
||||
`InvariantError` 扩展 `Error`，携带稳定 `code: 'INVARIANT'`，并公开所属 `packageName`，而不向服务添加产品依赖。
|
||||
|
||||
在每个组合中，Session 自身负责不可变且接口有效的日志存储：它对每个候选项制作一份无损 JSON 快照，验证完整来源和位置替换，将 `tool/result` 替换限制为一个当前结果的 `content`，深度冻结已接受记录，并通过不可变数组快照公开日志。`dsh-session` 不变量配套入口检查 Session 不负责的其余跨记录规则。
|
||||
|
||||
## 包配套入口
|
||||
|
||||
发布和注册覆盖全部包；运行时断言刻意不使用合成内容。只有当包拥有可观察事件关系或相关可变数据关系时，配套入口才安装检查。确认必需方法、插件名称、注入、effect 或固定纯函数结果属于类型、加载或单元测试关注点，而非运行时不变量。
|
||||
|
||||
如果不存在合理的运行时关系，配套入口使用空 installer，并以包专用的前置 `No runtime invariant:` 注释说明原因。纯工具、行为已通过 seam 观察的薄实现、仅组合包、二进制程序、契约需要崩溃/往返测试的持久化适配器和测试支持包通常属于此类。当 owner 获得可变状态或事件协议时，必须重新审视该说明。
|
||||
|
||||
当前可执行配套入口保护以下关系：
|
||||
|
||||
| 配套入口 | 检查 |
|
||||
|---|---|
|
||||
| `dsh-session`, `dsh-agent`, `dsh-scope`, `dsh-agent-loop` | 会话包含关系和调用/结果跟踪、agent 状态转换、inbox FIFO 守恒、作用域 subject 和模型请求重建。 |
|
||||
| `dsh-llm`, `dsh-llm-retry`, `dsh-tools`, `dsh-system-prompt` | 流语法、持久重试位置和边界、工具流水线阶段与冻结结果，以及权威提示词组装数据。 |
|
||||
| `dsh-compact`, `dsh-hook-protocol`, `dsh-sandbox-policy` | 持久压缩与钩子配对、压缩元数据和沙箱 mode 词汇。 |
|
||||
| `dsh-fs`, `dsh-subagent`, `dsh-workflow` | 文件系统事件身份、提供方/子级配对和工作流/agent 生命周期身份。 |
|
||||
| `dsh-goal`, `dsh-goal-session` | 持久 goal 来源/内容一致性、修订和生命周期转换、时间戳、顺序接纳 Round 和重建的继续提示词。 |
|
||||
| `dsh-permission`, `dsh-user-approval` | 活动 preset 引用和审批询问/决定审计配对。 |
|
||||
| `dsh-tasks`, `dsh-tool-todo` | 任务快照生命周期/归属字段和持久整表 todo 结构。 |
|
||||
| `dsh-time-context` | 持久时钟读数与会话开放轮次、下一个步骤前位置和已用 baseline 一致；渲染时间可解析，且不晚于其事件。 |
|
||||
|
||||
每个 owner 的根入口仍与诊断独立。单独加载服务不会安装产品检查；在没有服务时加载配套入口，会等待其声明的 `invariants` 注入。
|
||||
|
||||
`pnpm run verify-package-invariants` 发现全部工作区包。它拒绝生成标记、未说明的空 installer、省略或忽略 reporter 的非空 installer、错误注册名称，以及不完整的导出、发布、依赖、TypeScript 引用或 bundle 接线。该源规则是最低归属检查；聚焦测试证明每个可执行配套入口的语义。
|
||||
|
||||
## 组合
|
||||
|
||||
```ts
|
||||
import type { Context } from 'cordis'
|
||||
import InvariantService from '@deepseek-ai/dsh-invariants'
|
||||
import * as SessionInvariant from '@deepseek-ai/dsh-session/invariant'
|
||||
|
||||
declare const ctx: Context
|
||||
|
||||
ctx.plugin(InvariantService, {
|
||||
enabled: true,
|
||||
package_allowlist: ['^@deepseek-ai/dsh-'],
|
||||
package_blocklist: ['^@deepseek-ai/dsh-agent-loop$'],
|
||||
})
|
||||
ctx.plugin(SessionInvariant)
|
||||
```
|
||||
|
||||
标准 agent 主干挂载服务和 4 个核心有状态配套入口。自定义组合为希望检查其契约的其他已加载包显式添加配套入口；过滤器可以在不改变包入口的情况下禁用或选择注册。
|
||||
|
||||
每个普通 Vitest 拓扑都挂载显式启用的服务和当前测试包的配套入口。聚焦套件覆盖可执行配套入口的有效和无效观察，一个穷尽拓扑则挂载全部配套入口，以证明注册和 dispose 接线。
|
||||
|
||||
## 模型体验
|
||||
|
||||
无。服务和配套入口观察运行时事件和可变快照，不会更改提示词、消息、schema、流或工具结果。
|
||||
|
||||
#### KV 缓存影响
|
||||
|
||||
无；不变量检查不组装或发送提供方请求。
|
||||
|
||||
## 已知限制与待完成工作
|
||||
|
||||
- 请求重建覆盖 loop 在冻结前显式标记的请求；直接一次性 LLM 调用即使由调用方冻结或附加会话 id，仍不在该标记契约内。
|
||||
- 仅实时生命周期配套入口无法重建自身重新加载前开始的操作。标准组合和测试组合会在相应操作开始前挂载它们。
|
||||
- 正则表达式过滤器在服务生命周期内固定；更改它们需要执行普通 Cordis 插件重新加载。
|
||||
6
packages/support/llm-mock-server/README.i18n.yaml
Normal file
6
packages/support/llm-mock-server/README.i18n.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
README.md: 77a5c5e35fe0b4b1c92968eecea85b6059c889fc
|
||||
README.zh.md: bf84a1c5f5428e845a6917a82287733d82405144
|
||||
@@ -1,5 +1,7 @@
|
||||
# `@deepseek-ai/dsh-llm-mock-server`
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
A scriptable OpenAI-compatible HTTP/SSE server for exercising real LLM adapters, the agent loop, and recovery policy without a provider key. It accepts `POST /chat/completions` and `POST /v1/chat/completions`; each accepted request consumes one configured behavior in arrival order. Invalid methods, paths, bearer tokens, and JSON do not consume the script.
|
||||
|
||||
The library entry exports `startMockLlmServer(options)`, behavior and telemetry types, the default random stress weights, the accepted Node timer bound, and a running handle with the bound `baseURL`, generated or configured `randomSeed`, captured requests, and idempotent `close()`. Closing force-terminates stalled connections.
|
||||
|
||||
86
packages/support/llm-mock-server/README.zh.md
Normal file
86
packages/support/llm-mock-server/README.zh.md
Normal file
@@ -0,0 +1,86 @@
|
||||
# `@deepseek-ai/dsh-llm-mock-server`
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
可编脚本的 OpenAI 兼容 HTTP/SSE 服务器，用于在无提供方密钥的情况下测试真实 LLM 适配器、agent loop 和恢复策略。它接受 `POST /chat/completions` 和 `POST /v1/chat/completions`；每个已接受请求按到达顺序消费一个已配置行为。无效 method、path、bearer token 和 JSON 不消费脚本。
|
||||
|
||||
库入口导出 `startMockLlmServer(options)`、行为和 telemetry 类型、默认随机压力权重、可接受的 Node timer 边界，以及带有绑定 `baseURL`、已生成或已配置 `randomSeed`、已捕获请求和幂等 `close()` 的运行句柄。关闭会强制终止停滞连接。
|
||||
|
||||
## 独立使用
|
||||
|
||||
从本仓库运行源入口：
|
||||
|
||||
```sh
|
||||
pnpm run mock:llm -- \
|
||||
--port 8000 \
|
||||
--api-key mock-key \
|
||||
--sequence partial_disconnect,success \
|
||||
--partial-text "discard this half"
|
||||
```
|
||||
|
||||
将发布的 DeepSeek 适配器指向服务器；它会将 `/chat/completions` 追加到已配置 base：
|
||||
|
||||
```sh
|
||||
DEEPSEEK_BASE_URL=http://127.0.0.1:8000/v1 \
|
||||
DEEPSEEK_API_KEY=mock-key \
|
||||
pnpm run demo:headless "test provider recovery"
|
||||
```
|
||||
|
||||
构建包还公开 `dsh-llm-mock-server`。Stdout 是 JSONL：`ready` 记录携带 `/v1` base URL 和随机种子，后续请求/结果记录同时命名脚本行为和实际选中的具体行为。
|
||||
|
||||
## 行为脚本
|
||||
|
||||
`--sequence` 是逗号分隔的 FIFO。耗尽时返回结构化 HTTP 500；`--repeat-last` 显式重用最后一项。
|
||||
|
||||
| 行为 | 协议结果 |
|
||||
|---|---|
|
||||
| `connection_reset` | 在 HTTP header 前销毁 socket |
|
||||
| `stream_disconnect` | 发送 SSE header，然后在第一个事件前 reset |
|
||||
| `partial_disconnect` | 发送文本 delta，然后 reset socket |
|
||||
| `stall` | 发送 SSE header，并保持空闲，直到客户端/服务器取消 |
|
||||
| `empty` | 发送有效的无内容 stop 和 `[DONE]` |
|
||||
| `empty_body` / `stream_eof` / `partial_eof` | 正常结束，但缺少必需的 `[DONE]` 边界 |
|
||||
| `malformed_json` / `malformed_event` | 发送无效 SSE JSON 或无效提供方分片形态 |
|
||||
| `rate_limit` / `server_error` / `service_unavailable` | 返回面向重试的 429/500/503 JSON 错误 |
|
||||
| `auth_error` / `invalid_request` / `context_overflow` / `quota_exceeded` | 返回终止性或单独恢复的提供方错误 |
|
||||
| `success` / `slow_success` / `reasoning_success` | 流式发送完整文本响应，可选延迟或先发送 reasoning |
|
||||
| `tool_call_success` / `max_tokens` | 以工具调用或 `length` 结束原因完成 |
|
||||
| `wrong_content_type` | 在 `application/json` 下发送有效 SSE 正文 |
|
||||
| `random` | 从加权播种随机性中选择具体请求行为 |
|
||||
|
||||
`connection_refused` 只能在 CLI 中使用，且必须是第一个条目。它会延迟绑定调用方指定的非零端口，因此 `--listen-delay-ms` 期间的请求会收到真实 TCP 拒绝；其余条目在 listener 启动后开始。
|
||||
|
||||
## 随机 mode
|
||||
|
||||
使用重复 `random` 条目执行开放式混合运行：
|
||||
|
||||
```sh
|
||||
pnpm run mock:llm -- \
|
||||
--port 8000 \
|
||||
--sequence random \
|
||||
--repeat-last \
|
||||
--seed 42 \
|
||||
--random-weights 'success=60,slow_success=10,connection_reset=5,stream_disconnect=5,partial_disconnect=10,empty=5,server_error=5'
|
||||
```
|
||||
|
||||
省略 `--seed` 会生成种子，并在 `ready` 记录中打印。`--random-weights` 接受非负的相对 `behavior=weight` 条目，并要求至少一个正权重具体行为。导出默认值是一个成功占主导的压力分布，包含 reset、disconnect、部分输出、空完成、stall、429/5xx、干净截断和格式错误 JSON；它用于施加测试压力，而非估计生产事故频率。`connection_refused` 被排除，因为已绑定的请求处理器无法产生真实拒绝。
|
||||
|
||||
随机权重包含 `stall` 时，为待测客户端配置较短的流空闲超时，使场景及时结束。
|
||||
|
||||
## 时序与内容控制
|
||||
|
||||
CLI 公开 `--success-text`、`--partial-text`、`--reasoning-text`、`--chunk-size`、`--chunk-delay-ms`、`--disconnect-delay-ms`、`--retry-after-ms`、`--request-id`、`--tool-name` 和 `--tool-arguments`。毫秒延迟是 Node timer 范围内的有界整数；`retryAfterMs` 还必须为正数。库接受相同的 camel-case 选项。可选的精确 `apiKey` 验证 `Authorization: Bearer <token>`；省略时接受任何 token。
|
||||
|
||||
## 模型体验
|
||||
|
||||
无。该测试服务器替代提供方协议行为，而不调用真实模型。
|
||||
|
||||
#### KV 缓存影响
|
||||
|
||||
无；请求在本地终止，绝不会到达提供方缓存。
|
||||
|
||||
## 已知限制与待完成工作
|
||||
|
||||
- **随机权重建模测试压力，而非生产事故频率**：需要环境专用分布的调用方必须提供已测量权重，并记录发出的种子。
|
||||
- **请求脚本按到达顺序执行**：并发调用方共享一个游标，因此确定性的每会话故障分配需要独立服务器实例。
|
||||
- **真实连接拒绝是 listener 生命周期阶段**：CLI 延迟必须与客户端尝试重叠；请求级随机选择只能 reset 已接受连接。
|
||||
@@ -1,5 +1,5 @@
|
||||
import { request } from 'node:http'
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import type { MockLlmBehavior, MockLlmServer, MockLlmServerEvent } from '../src/index.ts'
|
||||
import { startMockLlmServer } from '../src/index.ts'
|
||||
|
||||
@@ -169,21 +169,23 @@ describe('mock LLM server wire behaviors', () => {
|
||||
['partial_disconnect', 100] as const,
|
||||
])('records a client that closes during %s', async (behavior, delayMs) => {
|
||||
const events: MockLlmServerEvent[] = []
|
||||
const result = Promise.withResolvers<Extract<MockLlmServerEvent, { type: 'result' }>>()
|
||||
const server = await start([behavior], {
|
||||
chunkDelayMs: delayMs,
|
||||
disconnectDelayMs: delayMs,
|
||||
chunkSize: 1,
|
||||
onEvent: (event) => { events.push(event) },
|
||||
onEvent: (event) => {
|
||||
events.push(event)
|
||||
if (event.type === 'result') result.resolve(event)
|
||||
},
|
||||
})
|
||||
const controller = new AbortController()
|
||||
const response = await chat(server, { signal: controller.signal })
|
||||
controller.abort()
|
||||
await expect(response.text()).rejects.toThrow()
|
||||
// The server observes the socket close asynchronously; a fixed sleep
|
||||
// raced slow runners, so poll until the outcome lands.
|
||||
await vi.waitFor(() => {
|
||||
expect(server.requests[0]).toMatchObject({ behavior, outcome: 'client_closed' })
|
||||
})
|
||||
await result.promise
|
||||
|
||||
expect(server.requests[0]).toMatchObject({ behavior, outcome: 'client_closed' })
|
||||
expect(events.filter(event => event.type === 'result')).toEqual([
|
||||
expect.objectContaining({ behavior, outcome: 'client_closed' }),
|
||||
])
|
||||
|
||||
6
packages/support/llm-replay/README.i18n.yaml
Normal file
6
packages/support/llm-replay/README.i18n.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
README.md: ce0758641f3d49a54b29415ed449e43043840f9a
|
||||
README.zh.md: 47a2b9aa211b44c4e476a1adf5a9a72d927cd0ed
|
||||
@@ -1,5 +1,7 @@
|
||||
# @deepseek-ai/dsh-llm-replay
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
A replay LLM plugin for keyless snapshot tests. It yields model streams reconstructed from a recorded **session JSONL** fixture, so a test can boot the real agent against a fixed model transcript with no API key. With `providers` configured it registers a replay-only adapter whose catalog is available to scenarios that exercise model discovery; without `providers` it installs the catch-all `llm/stream` waterfall used by tests that do not need discovery.
|
||||
|
||||
Its consumers are the ACP, headless `stream-json`, and TUI snapshot suites plus the web browser e2e lane. Loader-driven suites mount this plugin in place of a real LLM adapter; the web lane installs it directly to retain the teardown consumption handle. Keeping derivation and replay here places that logic under the per-file 100% coverage gate on `packages/*/src`.
|
||||
@@ -8,7 +10,7 @@ Its consumers are the ACP, headless `stream-json`, and TUI snapshot suites plus
|
||||
|
||||
The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`", done by the snapshot harness — this plugin does not record. A fixture may carry its `request/header` content tokenized to `{{system}}`/`{{tools}}` (the harness pins that content in one scenario and scrubs the rest); replay is indifferent — derivation reads only `assistant/chunk` events and the line-0 session header.
|
||||
|
||||
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`: a `ReplayEntry[]`) that REPLACES the derived script. A `hang` entry may name `readyFile`; replay writes that empty marker after its prefix chunks reach the loop and before it waits for cancellation, so an external driver can cancel deterministically without observing a presentation update.
|
||||
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`) that either replaces the derived script (a bare `ReplayEntry[]`) or augments it (`{ patches: [{ at, entry }] }`: keep every JSONL-derived call and swap the named 0-based call indexes; `at` equal to the derived length appends the retry attempt after an injected transient throw). Patch indexes must be unique. The override document, each patch and entry, and every chunk discriminant are validated when the file loads. A `hang` entry may name `readyFile`; replay writes that empty marker after its prefix chunks reach the loop and before it waits for cancellation, so an external driver can cancel deterministically without observing a presentation update.
|
||||
|
||||
## Nested agents: per-session keying
|
||||
|
||||
@@ -21,7 +23,7 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
|
||||
| Key | Type | Default | Notes |
|
||||
|---|---|---|---|
|
||||
| `file` | string | `$DSH_SNAPSHOT_FILE` | Path to the primary (parent) `session.jsonl` fixture. Required (config or env). |
|
||||
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | Optional path to a `ReplayEntry[]` sidecar that replaces the PRIMARY session's derived script. |
|
||||
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | Optional `ReplayOverrideDoc` sidecar for the primary session: a bare `ReplayEntry[]` replaces its derived script, while `{ patches }` augments it by call index. |
|
||||
| `childFiles` | string[] | `$DSH_SNAPSHOT_CHILD_FILES` (path-delimited) | Recorded subagent child-session logs for a nested scenario; empty for a single-session scenario. |
|
||||
| `providers` | `ReplayProviderConfig[]` | — | Optional replay-only provider and model catalog. Each model may publish `contextWindow`; configured routes dispatch through the replay adapter and never perform provider I/O. |
|
||||
| `paceMs` | number | — (burst) | Optional per-chunk delay in ms so downstream transports (e.g. the web SSE mux observed by a real browser) see genuinely incremental delivery. A realism knob only — tests must not depend on it for correctness. Non-negative integer; abort during a pace wait cancels the stream promptly. |
|
||||
@@ -46,9 +48,9 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
|
||||
|
||||
- `installLlmReplay(ctx, config)` — install the configured replay adapter or catch-all `llm/stream` listener; returns a `ReplayHandle` (`dispose()` for HMR safety plus `assertConsumed()`, the teardown check that every recorded script bound to a live session and every bound cursor drained — turning a scenario that silently drove fewer model calls than recorded into a crisp diagnostic). Use this in tests to drive replay without the Loader or env vars.
|
||||
- `loadSessionScripts(config)` — resolve the ordered `SessionScript[]` (primary + children) for a scenario, ready to bind to live sessions in first-call order.
|
||||
- `loadReplayScript(config)` — resolve the `ReplayEntry[]` for the PRIMARY session only (sidecar override if present, else derived from the JSONL; fail-loud if the fixture is missing).
|
||||
- `loadReplayScript(config)` — resolve the `ReplayEntry[]` for the primary session only (validated sidecar replacement/patches if present, else derived from the JSONL; fail-loud if the fixture is missing).
|
||||
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)` — the pure helpers that turn a recorded session log into a script and read its header `id`/`createdAt`. A derived group must end in a `finish` chunk; a group without one is the fingerprint of a thrown `stream()` and must instead be expressed via an override sidecar.
|
||||
- Types `ReplayEntry` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`.
|
||||
- Types `ReplayEntry` / `ReplayOverrideDoc` / `ReplayOverridePatch` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`.
|
||||
|
||||
## Plugin export shape
|
||||
|
||||
@@ -65,4 +67,4 @@ None; this package neither assembles nor sends a provider request.
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **First-call-order script binding assumes sequential delegation** — a cut that runs sibling subagents concurrently (or a compaction summarize call landing mid-run) would bind live sessions to recorded scripts non-deterministically; a stronger keying is deferred until such a scenario exists (`XXX(concurrent-subagents)`).
|
||||
- **Only chunk-producing calls are derivable** — a pure pre-chunk throw or a cancel/hang scenario needs the `replay.override.json` sidecar; the override replaces the PRIMARY session's script only.
|
||||
- **Only chunk-producing calls are derivable** — a pure pre-chunk throw or a cancel/hang scenario needs the `replay.override.json` sidecar. Replacement and patch forms affect only the primary session; child scripts still derive from their logs.
|
||||
|
||||
70
packages/support/llm-replay/README.zh.md
Normal file
70
packages/support/llm-replay/README.zh.md
Normal file
@@ -0,0 +1,70 @@
|
||||
# @deepseek-ai/dsh-llm-replay
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
用于无密钥快照测试的回放 LLM 插件。它从已记录的**会话 JSONL** fixture 重建模型流，使测试可以在无 API 密钥的情况下使用固定模型 transcript 启动真实 agent。配置 `providers` 后，它会注册仅回放适配器，其目录可供测试模型发现的场景使用；没有 `providers` 时，它会安装不需要发现的测试所用 catch-all `llm/stream` waterfall。
|
||||
|
||||
其消费方是 ACP、headless `stream-json` 和 TUI 快照套件，以及 web 浏览器 e2e lane。Loader 驱动套件使用此插件替换真实 LLM 适配器；web lane 直接安装它，以保留拆卸消费句柄。将派生和回放逻辑保留在此处，可使其受 `packages/*/src` 上每文件 100% 覆盖率门禁约束。
|
||||
|
||||
## Fixture 的工作方式
|
||||
|
||||
Fixture 就是持久化会话日志（`<scenario>/session.jsonl`）。其 `assistant/chunk` 事件携带每个 `StreamChunk`，因此按 `(turn, step)` 对其分组可重建每次 `stream()` 调用的分片序列（每个 loop 步骤一次模型调用）。因此，录制操作是「运行一次真实 agent 并收集 `.jsonl`」，由快照 harness 完成；该插件不执行录制。Fixture 的 `request/header` 内容可能被 token 化为 `{{system}}`/`{{tools}}`（harness 在一个场景中固定该内容，并擦除其余场景）；回放对此并不关心，因为派生只读取 `assistant/chunk` 事件和第 0 行会话 header。
|
||||
|
||||
有两种失败 mode 无法仅从 `assistant/chunk` 重建：在任何分片前纯抛出（例如 HTTP 401，日志只包含 `turn/end {error}` 而没有分片），以及 cancel/hang（是时序，而非分片内容）。需要这些的场景提供可选 sidecar（`<scenario>/replay.override.json`），它要么替换派生脚本（裸 `ReplayEntry[]`），要么增补派生脚本（`{ patches: [{ at, entry }] }`：保留全部由 JSONL 派生的调用，仅在点名的调用索引处换入，索引从 0 计；`at` 等于派生长度时为追加，正是注入的瞬态抛出之后那次重试尝试所占的槽位）。Patch 索引必须互不重复。覆写文档、每个 patch 与每个条目，以及每个分片的判别字段都会在文件加载时接受校验。`hang` 条目可以指定 `readyFile`；在其前缀分片到达 loop 后、等待取消前，回放会写入该空标记，使外部驱动器可以在不观察展示更新的情况下确定性取消。
|
||||
|
||||
## 嵌套 agent：每会话键控
|
||||
|
||||
父 agent 委托给进程内 subagent 的场景会记录多个日志：父级（`session.jsonl`）和每个子级各一个（`session.1.jsonl`等）。每个 agent 在同一上下文中作为自己的 `Session` 运行，因此回放必须为每个 agent 提供自己的脚本。
|
||||
|
||||
回放按调用会话 id 为每次调用建键（由 agent loop 标记的 `GenerateOptions.sessionId`）。实时会话 id 在每次运行中都是新的随机值，绝不等于已记录值，因此实时会话通过**首次调用顺序** 绑定到已记录脚本：脚本按 header `createdAt` 排序（父级在前，因为它必须先进行流式输出才能委托）；第一个进行任何调用的实时会话领取第一个脚本，下一个新会话领取下一个，以此类推。然后，每个会话推进自己的游标。没有 `sessionId` 的调用是绑定到主脚本的单一匿名会话，因此单会话场景与以前完全相同。实时会话数超过已记录脚本数时快速失败。
|
||||
|
||||
## 配置
|
||||
|
||||
| 键 | 类型 | 默认值 | 说明 |
|
||||
|---|---|---|---|
|
||||
| `file` | string | `$DSH_SNAPSHOT_FILE` | 主（父）`session.jsonl` fixture 的路径。必需（配置或 env）。 |
|
||||
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | 主会话的可选 `ReplayOverrideDoc` sidecar：裸 `ReplayEntry[]` 替换其派生脚本，`{ patches }` 则按调用索引增补该脚本。 |
|
||||
| `childFiles` | string[] | `$DSH_SNAPSHOT_CHILD_FILES` (path-delimited) | 嵌套场景中已记录的 subagent 子会话日志；单会话场景为空。 |
|
||||
| `providers` | `ReplayProviderConfig[]` | 无 | 可选的仅回放提供方和模型目录。每个模型可以发布 `contextWindow`；已配置路由通过回放适配器分派，绝不执行提供方 I/O。 |
|
||||
| `paceMs` | number | 无（突发） | 可选的每分片毫秒延迟，使下游传输（例如真实浏览器观察的 web SSE mux）看到真正的增量传递。它只是仿真开关，测试不得依赖它保证正确性。值必须是非负整数；pace 等待期间中止会迅速取消流。 |
|
||||
|
||||
```yaml
|
||||
- id: llm-replay
|
||||
name: '@deepseek-ai/dsh-llm-replay'
|
||||
config:
|
||||
providers:
|
||||
- id: deepseek
|
||||
name: DeepSeek
|
||||
models:
|
||||
- id: deepseek-v4-flash
|
||||
contextWindow: 128000
|
||||
- id: deepseek-v4-pro
|
||||
# file/overrideFile/childFiles default to $DSH_SNAPSHOT_FILE /
|
||||
# $DSH_SNAPSHOT_OVERRIDE / $DSH_SNAPSHOT_CHILD_FILES, set by the snapshot
|
||||
# harness per scenario.
|
||||
```
|
||||
|
||||
## 导出项
|
||||
|
||||
- `installLlmReplay(ctx, config)`：安装已配置回放适配器或 catch-all `llm/stream` 监听器；返回 `ReplayHandle`（包含用于 HMR 安全的 `dispose()`，以及 `assertConsumed()` 拆卸检查；后者确保每个已记录脚本都绑定到实时会话，且每个已绑定游标都已耗尽，从而将场景静默驱动的模型调用少于记录数转换为明确诊断）。在测试中使用它，可以不通过 Loader 或 env var 驱动回放。
|
||||
- `loadSessionScripts(config)`：解析场景的有序 `SessionScript[]` （主级 + 子级），准备按首次调用顺序绑定到实时会话。
|
||||
- `loadReplayScript(config)`：只解析主会话的 `ReplayEntry[]` （如果存在则使用经校验的 sidecar 替换或 patch，否则从 JSONL 派生；fixture 缺失时快速失败）。
|
||||
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)`：将已记录会话日志转换为脚本并读取其 header `id`/`createdAt` 的纯辅助工具。派生分组必须以 `finish` 分片结束；没有该分片的分组是已抛出 `stream()` 的指纹，必须改用 override sidecar 表达。
|
||||
- 类型 `ReplayEntry` / `ReplayOverrideDoc` / `ReplayOverridePatch` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`。
|
||||
|
||||
## 插件导出形态
|
||||
|
||||
命名导出 `name` / `inject` / `Config` / `apply`，且**没有默认导出**：Cordis Loader 的 `unwrapExports` 执行 `exports.default ?? exports`，因此意外的默认导出会将模块折叠为纯函数，并丢弃 `inject` 命名空间（见 [docs/postmortem/0001](../../../docs/postmortem/0001-acp-default-export-drops-inject.md)）。
|
||||
|
||||
## 模型体验
|
||||
|
||||
无。该无密钥测试适配器不向提供方模型发送请求，只将已记录 assistant 分片回放到测试 loop 中。
|
||||
|
||||
#### KV 缓存影响
|
||||
|
||||
无；该包既不组装也不发送提供方请求。
|
||||
|
||||
## 已知限制与待完成工作
|
||||
|
||||
- **首次调用顺序脚本绑定假设串行委托**：并发运行同级 subagent 的 cut（或运行中落地的压缩摘要调用）会非确定性地将实时会话绑定到已记录脚本；在这种场景出现前暂不实现更强的键控（`XXX(concurrent-subagents)`）。
|
||||
- **只有生产分片的调用可派生**：纯分片前抛出或 cancel/hang 场景需要 `replay.override.json` sidecar。替换和 patch 两种形式都只影响主会话；子会话脚本仍从各自日志派生。
|
||||
@@ -59,10 +59,11 @@ export interface ReplayConfig {
|
||||
*/
|
||||
file: string
|
||||
/**
|
||||
* Optional `ReplayEntry[]` sidecar that REPLACES the derived script for the
|
||||
* PRIMARY session. Used by the two single-session scenarios not expressible as
|
||||
* `assistant/chunk` (pure throw-before-chunk, cancel/hang). Absent for normal
|
||||
* and nested scenarios.
|
||||
* Optional sidecar for the PRIMARY session: a bare `ReplayEntry[]` replaces
|
||||
* the derived script; `{ patches }` keeps it and swaps the named call
|
||||
* indexes ({@link ReplayOverrideDoc}). Used by single-session scenarios not
|
||||
* expressible as `assistant/chunk` (throw-before-chunk, cancel/hang,
|
||||
* injected transient failures). Absent for normal and nested scenarios.
|
||||
*/
|
||||
overrideFile?: string
|
||||
/**
|
||||
@@ -200,26 +201,157 @@ export function deriveReplayScript(events: SessionEvent[]): ReplayEntry[] {
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the replay script for the PRIMARY session: the sidecar override if
|
||||
* present, otherwise the script derived from the recorded session JSONL.
|
||||
* Fail-loud if the JSONL fixture is missing (the scenario was never recorded) —
|
||||
* never silently returns an empty script, so a coverage hole can't masquerade
|
||||
* as a passing replay.
|
||||
* One positional patch in an augmentation sidecar: replaces the derived
|
||||
* entry at call index `at` (0-based) with `entry`, or appends when `at`
|
||||
* equals the derived length (an extra recorded-after-the-fact call, e.g. the
|
||||
* retry attempt following an injected transient throw).
|
||||
*/
|
||||
export interface ReplayOverridePatch {
|
||||
/** 0-based call index into the derived script; == length appends. */
|
||||
at: number
|
||||
/** The replacement (or appended) entry at that call position. */
|
||||
entry: ReplayEntry
|
||||
}
|
||||
|
||||
/**
|
||||
* Override sidecar document: either a whole-script replacement (a
|
||||
* bare `ReplayEntry[]`) or the augmentation form `{ patches }`, which keeps
|
||||
* the JSONL-derived script and swaps only the named call indexes — the shape
|
||||
* for "turn N errors, everything else replays as recorded".
|
||||
*/
|
||||
export type ReplayOverrideDoc = ReplayEntry[] | { patches: ReplayOverridePatch[] }
|
||||
|
||||
const REPLAY_CHUNK_TYPES = new Set<StreamChunk['type']>([
|
||||
'block-start',
|
||||
'text-delta',
|
||||
'reasoning-delta',
|
||||
'tool-call-delta',
|
||||
'block-end',
|
||||
'usage',
|
||||
'finish',
|
||||
])
|
||||
|
||||
function isRecord(value: unknown): value is Record<string, unknown> {
|
||||
return typeof value === 'object' && value !== null && !Array.isArray(value)
|
||||
}
|
||||
|
||||
function hasExactKeys(value: Record<string, unknown>, keys: readonly string[]): boolean {
|
||||
return Object.keys(value).length === keys.length && keys.every(key => Object.hasOwn(value, key))
|
||||
}
|
||||
|
||||
function invalidOverride(file: string, location: string, detail: string): never {
|
||||
throw new Error(`llm-replay: invalid override ${file}: ${location} ${detail}`)
|
||||
}
|
||||
|
||||
function readChunks(value: unknown, file: string, location: string): StreamChunk[] {
|
||||
if (!Array.isArray(value)) invalidOverride(file, location, 'chunks must be an array')
|
||||
for (const [index, chunk] of value.entries()) {
|
||||
if (!isRecord(chunk)
|
||||
|| typeof chunk['type'] !== 'string'
|
||||
|| !REPLAY_CHUNK_TYPES.has(chunk['type'] as StreamChunk['type'])) {
|
||||
invalidOverride(file, `${location}.chunks[${index}]`, 'must have a known StreamChunk type')
|
||||
}
|
||||
}
|
||||
return value as StreamChunk[]
|
||||
}
|
||||
|
||||
function readReplayEntry(value: unknown, file: string, location: string): ReplayEntry {
|
||||
if (!isRecord(value)) invalidOverride(file, location, 'must be an object')
|
||||
switch (value['kind']) {
|
||||
case 'chunks': {
|
||||
if (!hasExactKeys(value, ['kind', 'chunks'])) invalidOverride(file, location, 'has invalid chunks-entry fields')
|
||||
return { kind: 'chunks', chunks: readChunks(value['chunks'], file, location) }
|
||||
}
|
||||
case 'throw': {
|
||||
if (!hasExactKeys(value, ['kind', 'chunks', 'message', 'code'])) {
|
||||
invalidOverride(file, location, 'has invalid throw-entry fields')
|
||||
}
|
||||
if (typeof value['message'] !== 'string' || value['message'].length === 0) {
|
||||
invalidOverride(file, location, 'message must be a non-empty string')
|
||||
}
|
||||
if (typeof value['code'] !== 'string' || value['code'].length === 0) {
|
||||
invalidOverride(file, location, 'code must be a non-empty string')
|
||||
}
|
||||
return {
|
||||
kind: 'throw',
|
||||
chunks: readChunks(value['chunks'], file, location),
|
||||
message: value['message'],
|
||||
code: value['code'],
|
||||
}
|
||||
}
|
||||
case 'hang': {
|
||||
const readyFile = value['readyFile']
|
||||
const keys = readyFile === undefined ? ['kind'] : ['kind', 'readyFile']
|
||||
if (!hasExactKeys(value, keys)) invalidOverride(file, location, 'has invalid hang-entry fields')
|
||||
if (readyFile !== undefined && (typeof readyFile !== 'string' || readyFile.length === 0)) {
|
||||
invalidOverride(file, location, 'readyFile must be a non-empty string')
|
||||
}
|
||||
return { kind: 'hang', ...(readyFile === undefined ? {} : { readyFile }) }
|
||||
}
|
||||
default:
|
||||
return invalidOverride(file, location, `has unknown kind ${JSON.stringify(value['kind'])}`)
|
||||
}
|
||||
}
|
||||
|
||||
function readOverrideDoc(value: unknown, file: string): ReplayOverrideDoc {
|
||||
if (Array.isArray(value)) return value.map((entry, index) => readReplayEntry(entry, file, `entry ${index}`))
|
||||
if (!isRecord(value) || !hasExactKeys(value, ['patches']) || !Array.isArray(value['patches'])) {
|
||||
return invalidOverride(file, 'document', 'must be a ReplayEntry[] or { patches: [...] }')
|
||||
}
|
||||
return {
|
||||
patches: value['patches'].map((value, index): ReplayOverridePatch => {
|
||||
const location = `patch ${index}`
|
||||
if (!isRecord(value) || !hasExactKeys(value, ['at', 'entry'])) {
|
||||
return invalidOverride(file, location, 'must contain exactly at and entry')
|
||||
}
|
||||
const at = value['at']
|
||||
if (typeof at !== 'number' || !Number.isSafeInteger(at) || at < 0) {
|
||||
return invalidOverride(file, location, 'at must be a non-negative safe integer')
|
||||
}
|
||||
return { at, entry: readReplayEntry(value['entry'], file, `${location}.entry`) }
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Load the PRIMARY session's replay script: the sidecar override when present
|
||||
* (whole-script replacement or `{ patches }` augmentation over the derived
|
||||
* script), else the script derived from the session JSONL (fail-loud when the
|
||||
* fixture is missing).
|
||||
* @param config - the fixture paths; only `file` and `overrideFile` are consulted.
|
||||
* @returns the primary session's replay entries.
|
||||
* @returns the resolved primary-session script.
|
||||
*/
|
||||
export function loadReplayScript(config: ReplayConfig): ReplayEntry[] {
|
||||
if (config.overrideFile !== undefined && existsSync(config.overrideFile)) {
|
||||
const parsed: unknown = JSON.parse(readFileSync(config.overrideFile, 'utf8'))
|
||||
if (!Array.isArray(parsed)) {
|
||||
throw new Error(`llm-replay: override is not a JSON array: ${config.overrideFile}`)
|
||||
const doc = readOverrideDoc(JSON.parse(readFileSync(config.overrideFile, 'utf8')) as unknown, config.overrideFile)
|
||||
if (Array.isArray(doc)) return doc
|
||||
const script = deriveScriptFromFile(config.file)
|
||||
const derivedLength = script.length
|
||||
const seenIndexes = new Set<number>()
|
||||
for (const patch of doc.patches) {
|
||||
if (patch.at > derivedLength) {
|
||||
throw new Error(
|
||||
`llm-replay: override patch index ${String(patch.at)} out of range `
|
||||
+ `(derived script has ${derivedLength} call(s); == length appends): ${config.overrideFile}`,
|
||||
)
|
||||
}
|
||||
if (seenIndexes.has(patch.at)) {
|
||||
throw new Error(`llm-replay: duplicate override patch index ${patch.at}: ${config.overrideFile}`)
|
||||
}
|
||||
seenIndexes.add(patch.at)
|
||||
script[patch.at] = patch.entry
|
||||
}
|
||||
return parsed as ReplayEntry[]
|
||||
return script
|
||||
}
|
||||
if (!existsSync(config.file)) {
|
||||
throw new Error(`llm-replay: fixture not found: ${config.file} — run \`pnpm run test:snapshot:record\` first`)
|
||||
return deriveScriptFromFile(config.file)
|
||||
}
|
||||
|
||||
/** Derive the primary script from the session JSONL, failing loud on a missing fixture. */
|
||||
function deriveScriptFromFile(file: string): ReplayEntry[] {
|
||||
if (!existsSync(file)) {
|
||||
throw new Error(`llm-replay: fixture not found: ${file} — run \`pnpm run test:snapshot:record\` first`)
|
||||
}
|
||||
return deriveReplayScript(parseSessionLog(readFileSync(config.file, 'utf8')))
|
||||
return deriveReplayScript(parseSessionLog(readFileSync(file, 'utf8')))
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -367,9 +499,8 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined,
|
||||
})
|
||||
/* v8 ignore next -- unreachable: the hang promise only ever rejects (on abort), never resolves; control never reaches here */
|
||||
return
|
||||
/* v8 ignore next -- sidecar entries are validated before they reach the closed local union. */
|
||||
default:
|
||||
// Closed local union: an unknown kind means malformed (hand-edited or
|
||||
// drifted) sidecar data — fail loud with a runtime diagnostic.
|
||||
return assertNever(entry, 'llm-replay replay entry')
|
||||
}
|
||||
}
|
||||
|
||||
@@ -203,11 +203,91 @@ describe('loadReplayScript', () => {
|
||||
expect(() => loadReplayScript({ file: join(dir, 'absent.jsonl') })).toThrow(/fixture not found/)
|
||||
})
|
||||
|
||||
it('throws when the override is not a JSON array', () => {
|
||||
it('rejects an override document that is neither supported form', () => {
|
||||
writeFileSync(file, sessionJsonl([]), 'utf8')
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
writeFileSync(overrideFile, '{"not":"array"}', 'utf8')
|
||||
expect(() => loadReplayScript({ file, overrideFile })).toThrow(/not a JSON array/)
|
||||
expect(() => loadReplayScript({ file, overrideFile })).toThrow(/document must be a ReplayEntry\[\] or \{ patches/)
|
||||
})
|
||||
|
||||
it('patches form: swaps the named call index and keeps derived siblings', () => {
|
||||
const callB: StreamChunk[] = [
|
||||
{ type: 'block-start', index: 0, blockType: 'text' },
|
||||
{ type: 'text-delta', index: 0, text: 'two' },
|
||||
{ type: 'finish', reason: { kind: 'stop' } },
|
||||
]
|
||||
let seq = 1
|
||||
writeFileSync(file, sessionJsonl([
|
||||
...TEXT_CHUNKS.map(c => chunkEvent(seq++, 1, 1, c)),
|
||||
...callB.map(c => chunkEvent(seq++, 1, 2, c)),
|
||||
]), 'utf8')
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
writeFileSync(overrideFile, JSON.stringify({
|
||||
patches: [{ at: 0, entry: { kind: 'throw', chunks: [], message: 'transient', code: 'SERVER' } }],
|
||||
}), 'utf8')
|
||||
expect(loadReplayScript({ file, overrideFile })).toEqual([
|
||||
{ kind: 'throw', chunks: [], message: 'transient', code: 'SERVER' },
|
||||
{ kind: 'chunks', chunks: callB },
|
||||
])
|
||||
})
|
||||
|
||||
it('patches form: at == derived length appends (the retry-attempt slot)', () => {
|
||||
writeFileSync(file, sessionJsonl(TEXT_CHUNKS.map((c, i) => chunkEvent(i + 1, 1, 1, c))), 'utf8')
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
writeFileSync(overrideFile, JSON.stringify({
|
||||
patches: [
|
||||
{ at: 0, entry: { kind: 'throw', chunks: [], message: '429', code: 'RATE_LIMIT' } },
|
||||
{ at: 1, entry: { kind: 'chunks', chunks: TEXT_CHUNKS } },
|
||||
],
|
||||
}), 'utf8')
|
||||
expect(loadReplayScript({ file, overrideFile })).toEqual([
|
||||
{ kind: 'throw', chunks: [], message: '429', code: 'RATE_LIMIT' },
|
||||
{ kind: 'chunks', chunks: TEXT_CHUNKS },
|
||||
])
|
||||
})
|
||||
|
||||
it('patches form: an out-of-range index fails loud with the derived length', () => {
|
||||
writeFileSync(file, sessionJsonl(TEXT_CHUNKS.map((c, i) => chunkEvent(i + 1, 1, 1, c))), 'utf8')
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
writeFileSync(overrideFile, JSON.stringify({ patches: [{ at: 2, entry: { kind: 'hang' } }] }), 'utf8')
|
||||
expect(() => loadReplayScript({ file, overrideFile })).toThrow(/patch index 2 out of range.*1 call/s)
|
||||
})
|
||||
|
||||
it('validates patch and entry shapes at the file boundary', () => {
|
||||
writeFileSync(file, sessionJsonl([]), 'utf8')
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
const invalid: Array<{ doc: unknown; message: RegExp }> = [
|
||||
{ doc: null, message: /document must be/ },
|
||||
{ doc: { patches: [null] }, message: /patch 0 must contain exactly at and entry/ },
|
||||
{ doc: { patches: [{ at: -1, entry: { kind: 'hang' } }] }, message: /at must be a non-negative safe integer/ },
|
||||
{ doc: { patches: [{ at: 1.5, entry: { kind: 'hang' } }] }, message: /at must be a non-negative safe integer/ },
|
||||
{ doc: [42], message: /entry 0 must be an object/ },
|
||||
{ doc: [{ kind: 'chunks', chunks: 'nope' }], message: /chunks must be an array/ },
|
||||
{ doc: [{ kind: 'chunks', chunks: [], extra: true }], message: /invalid chunks-entry fields/ },
|
||||
{ doc: [{ kind: 'chunks', chunks: [{ type: 'bogus' }] }], message: /known StreamChunk type/ },
|
||||
{ doc: [{ kind: 'throw', chunks: [], message: 'nope', code: 'AUTH', extra: true }], message: /invalid throw-entry fields/ },
|
||||
{ doc: [{ kind: 'throw', chunks: [], message: '', code: 'AUTH' }], message: /message must be a non-empty string/ },
|
||||
{ doc: [{ kind: 'throw', chunks: [], message: 'nope', code: '' }], message: /code must be a non-empty string/ },
|
||||
{ doc: [{ kind: 'hang', extra: true }], message: /invalid hang-entry fields/ },
|
||||
{ doc: [{ kind: 'hang', readyFile: 1 }], message: /readyFile must be a non-empty string/ },
|
||||
{ doc: [{ kind: 'bogus' }], message: /unknown kind/ },
|
||||
]
|
||||
for (const { doc, message } of invalid) {
|
||||
writeFileSync(overrideFile, JSON.stringify(doc), 'utf8')
|
||||
expect(() => loadReplayScript({ file, overrideFile })).toThrow(message)
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects duplicate patch indexes instead of silently taking the last one', () => {
|
||||
writeFileSync(file, sessionJsonl(TEXT_CHUNKS.map((c, i) => chunkEvent(i + 1, 1, 1, c))), 'utf8')
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
writeFileSync(overrideFile, JSON.stringify({
|
||||
patches: [
|
||||
{ at: 0, entry: { kind: 'hang' } },
|
||||
{ at: 0, entry: { kind: 'throw', chunks: [], message: 'busy', code: 'SERVER' } },
|
||||
],
|
||||
}), 'utf8')
|
||||
expect(() => loadReplayScript({ file, overrideFile })).toThrow(/duplicate override patch index 0/)
|
||||
})
|
||||
})
|
||||
|
||||
@@ -366,16 +446,14 @@ describe('installLlmReplay (through the real LlmService)', () => {
|
||||
.toEqual([{ type: 'finish', reason: { kind: 'stop' } }])
|
||||
})
|
||||
|
||||
it('throws on a malformed sidecar entry kind (the assertNever guard)', async () => {
|
||||
it('rejects a malformed sidecar entry kind before installing replay', async () => {
|
||||
writeFileSync(file, sessionJsonl([]), 'utf8')
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
// A kind the union does not know — hand-edited/drifted sidecar data.
|
||||
writeFileSync(overrideFile, JSON.stringify([{ kind: 'bogus' }]), 'utf8')
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
installLlmReplay(ctx, { file, overrideFile })
|
||||
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })))
|
||||
.rejects.toThrow(/llm-replay replay entry/)
|
||||
expect(() => installLlmReplay(ctx, { file, overrideFile })).toThrow(/unknown kind/)
|
||||
})
|
||||
|
||||
it('rejects a hang entry when the signal fires DURING the wait (abort listener path)', async () => {
|
||||
|
||||
6
packages/support/loader-smoke/README.i18n.yaml
Normal file
6
packages/support/loader-smoke/README.i18n.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
README.md: 8e53550608037a3c9a272db825933b7224ab24db
|
||||
README.zh.md: 5310429ab59cf3cd04ac024746f5ed557e003637
|
||||
@@ -1,5 +1,7 @@
|
||||
# `@deepseek-ai/dsh-loader-smoke`
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
Shared subprocess harness for tests that boot an app and `cordis.yml` through the Cordis Loader. `resolveExampleLaunch` selects local `src` mode (tsx and root tsconfig paths) or CI `lib` mode (plain Node and package exports) from an explicit mode or `DSH_EXAMPLE_MODE`.
|
||||
|
||||
`runLoaderSmoke` accepts bin and config paths, optional complete bin arguments, environment overrides, stdin, pre-run setup, and pre-cleanup inspection. It owns the isolated cwd, DSH homes, diagnostics, deadline, termination, EOF, and cleanup; it returns both streams after a zero exit and rejects with both streams on failure.
|
||||
|
||||
23
packages/support/loader-smoke/README.zh.md
Normal file
23
packages/support/loader-smoke/README.zh.md
Normal file
@@ -0,0 +1,23 @@
|
||||
# `@deepseek-ai/dsh-loader-smoke`
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
用于测试通过 Cordis Loader 启动应用和 `cordis.yml` 的共享子进程 harness。`resolveExampleLaunch` 选择本地 `src` mode（tsx 和根 tsconfig 路径）或 CI `lib` mode（普通 Node 和包导出）；选择依据为显式 mode 或 `DSH_EXAMPLE_MODE`。
|
||||
|
||||
`runLoaderSmoke` 接受 bin 和配置路径、可选的完整 bin 参数、环境覆盖、stdin、运行前设置和清理前检查。它负责隔离 cwd、DSH 主目录、诊断、deadline、终止、EOF 和清理；在零退出后返回两个流，失败时拒绝并携带两个流。
|
||||
|
||||
这是支持层测试基础设施，而非产品 API。
|
||||
|
||||
## 模型体验
|
||||
|
||||
无。该测试专用 harness 启动示例进程并检查它们的流，不会改变已组装模型请求。
|
||||
|
||||
#### KV 缓存影响
|
||||
|
||||
无；该包既不组装也不发送提供方请求。
|
||||
|
||||
## 已知限制与待完成工作
|
||||
|
||||
- **构建 mode 需要事先构建**：配置还必须能够通过 `examples/node_modules` 向上解析每个命名包。
|
||||
- **捕获的 stdout 和 stderr 无界**：失控子进程可以消耗内存，直到 deadline 将其终止。
|
||||
- **超时只终止直接子进程**：故障 fixture 生成的进程树可以比冒烟测试存活更久，需要外部清理。
|
||||
Reference in New Issue
Block a user