Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into codex/ask-user-question

Browse Source 
# Conflicts:
#	docs/module-graph.md
This commit is contained in:
Yichen Jiang
2026-07-08 22:22:25 +08:00
parent d3f86996f7 970971fd53
commit b31399bdc4
99 changed files with 3332 additions and 484 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
packages/README.md
View File

@@ -11,11 +11,13 @@ Packages are grouped by modular role at `packages/<group>/<pkg>/`. The group dir
| [`core/`](core/README.md) | Product API spine: session, system-prompt, tools, agent, and the concrete loop | Product — stable surface |
| [`llm/`](llm/README.md) | LLM capability family: the abstract service + provider adapters | Product — stable surface |
| [`bash/`](bash/README.md) | Bash capability family: the executor seam, a local impl, and the model-facing tool | Product — stable surface |
| [`code-runtime/`](code-runtime/README.md) | Code-execution capability family: the abstract runtime seam for model-written programs | Product — stable surface |
| [`fs/`](fs/README.md) | Filesystem capability family: the abstract seam, a local impl, and the model-facing file tools | Product — stable surface |
| [`compact/`](compact/README.md) | Compaction capability family: the abstract seam + a basic backend (tool deferred) | Product — stable surface |
| [`subagent/`](subagent/README.md) | Subagent capability family: the provider-registry seam and the model-facing delegation tool | Product — stable surface |
| [`web/`](web/README.md) | Web capability family: the abstract seam, search/fetch provider impls, and the model-facing web tools | Product — stable surface |
| [`todo/`](todo/README.md) | Todo/planning family: the model-facing `todo_write` tool (whole-list task tracking on the session log) | Product — stable surface |
| [`todo/`](todo/README.md) | Todo/planning family: the model-facing `todo_write` tool | Product — stable surface |
| [`guard/`](guard/README.md) | Loop-hygiene guards: advisory repeat-call reminders | Product — stable surface |
| [`hooks/`](hooks/README.md) | Hook bridges + the shared Claude Code / Codex wire-protocol library | Product — stable surface |
| [`session-persistence/`](session-persistence/README.md) | Persistence capability family: the seam + JSONL/SQLite backends | Product — stable surface |
| [`ui/`](ui/README.md) | Editor/client integration surfaces (the ACP bridge) + the app packages | Product — stable surface |

9
packages/code-runtime/README.md Normal file
View File

@@ -0,0 +1,9 @@
# code-runtime/ — code-execution capability family
The code-execution capability seam (see [capability seams](../../docs/rfc/implemented/architecture/2026-06-13-capability-seams.md)): an abstract runtime interface for executing one model-written program against host-provided async bindings, capturing what it printed and returned. The consumer is the tool registry's Code Mode, and the first implementation (a Node worker-thread backend) is specified alongside it in the [Code Mode RFC](../../docs/rfc/proposed/feature/2026-06-15-code-mode.md). **Product** packages.
| Package | Role | ctx key |
|---|---|---|
| `code-runtime/` | Abstract code-execution seam (interface + vocabulary) | `ctx.codeRuntime` |
The interface lives at `code-runtime/code-runtime/`. Backends differ by execution substrate (worker thread, process, container) and by source language — both readonly descriptors on the service — and register `ctx.codeRuntime` without touching the interface or its consumer; that split is what makes a hardened backend a drop-in later.

19
packages/code-runtime/code-runtime/README.md Normal file
View File

@@ -0,0 +1,19 @@
# @deepseek-ai/dsh-code-runtime
The **code-execution seam**: an abstract `CodeRuntime` service (`ctx.codeRuntime`) defining WHAT a code runtime does — run one model-written program against a set of host-provided async bindings and report `{ value, logs, error? }` — without saying HOW.
This package is the interface third of the capability (the bash trio is the template — see [capability seams](../../../docs/rfc/implemented/architecture/2026-06-13-capability-seams.md)): implementations subclass `CodeRuntime` and register the service; the consumer is the tool registry's Code Mode, which generates the model-facing SDK and bridges tool dispatch — both specified in the [Code Mode RFC](../../../docs/rfc/proposed/feature/2026-06-15-code-mode.md), whose first implementation is a Node worker-thread backend. The runtime knows nothing about tools or sessions: it is handed named async functions and a program string, and everything tool-shaped stays with the consumer.
## Service API (`ctx.codeRuntime`)
| Member | Semantics |
|---|---|
| `run(request)` | Execute one program against the request's bindings. **Resolves with an error FIELD for every program outcome** — parse/transform failure, thrown exception, budget expiry, abort, substrate death (`CodeRunFailure`'s orthogonal `kind` taxonomy); it rejects only for caller misuse of the seam itself (e.g. a run submitted after disposal). The program runs as the body of an async function: top-level `await`/`return` work, and the completion value becomes `result.value` when it survives the serialization boundary. |
| `language` | Readonly descriptor: the source language `run` expects (`'typescript'` is the well-known value). Informational, not gating — a consumer that generates language-specific presentation switches on it and fails loud on a language it cannot present. |
| `isolation` | Readonly descriptor: the execution substrate (`'worker-thread'`, `'process'`, `'container'`). A label for deployments and diagnostics, **not a security claim**. |
Semantics every implementation must honor (contract details in the class JSDoc): binding calls bridge to the caller's functions verbatim with structured-cloneable arguments/resolutions; the program is treated as a hostile peer (arbitrary binding names are own properties, malformed traffic never crashes the host); no state survives between runs; disposal terminates in-flight runs AND awaits their exit before completing.
## Vocabulary
`CodeRunRequest` (`program`, `bindings`, `signal?`) carries everything the runtime acts on — defaulting (time budgets, output caps) is the implementation's validated config, never a hidden `??` inside `run()`. `bindings` is a list of `CodeBindingNamespace`s (`global` + `functions`), each exposed to the program as one global object of async callables. `CodeRunResult` reports the completion `value?`, the ordered `logs` (`CodeLogEntry`: `console`/`stdout`/`stderr` source, console `level`, capped text), and the `error?` (`CodeRunFailure`: `kind` + model-feedable `message`). See `src/types.ts` for the full contracts.

30
packages/code-runtime/code-runtime/package.json Normal file
View File

@@ -0,0 +1,30 @@
{
"name": "@deepseek-ai/dsh-code-runtime",
"description": "Abstract code-execution seam (ctx.codeRuntime) for the DeepSeek Harness",
"version": "0.0.1",
"private": true,
"type": "module",
"main": "lib/index.js",
"types": "lib/types/index.d.ts",
"exports": {
".": {
"types": "./lib/types/index.d.ts",
"default": "./lib/index.js"
},
"./src/*": "./src/*",
"./package.json": "./package.json"
},
"files": [
"lib/index.js",
"lib/types/**/*.d.ts",
"lib/types/**/*.d.ts.map",
"src"
],
"license": "BSD-3-Clause",
"peerDependencies": {
"cordis": "^4.0.0-rc.6"
},
"devDependencies": {
"cordis": "^4.0.0-rc.6"
}
}

93
packages/code-runtime/code-runtime/src/index.ts Normal file
View File

@@ -0,0 +1,93 @@
/**
* The code-execution seam (`ctx.codeRuntime`): an abstract service defining
* WHAT a code runtime does — run one model-written program against a set of
* host-provided async bindings and report `{ value, logs, error? }` — without
* saying HOW. Implementations subclass {@link CodeRuntime} and register
* themselves as the `codeRuntime` service; backends may differ by execution
* substrate (worker thread, separate process, container) and by source
* language, both declared as readonly descriptors. The design and its
* consumer (the tool registry's Code Mode) are specified in the Code Mode RFC
* (docs/rfc/proposed/feature/2026-06-15-code-mode.md).
*
* The split mirrors the bash seam (`BashExecutor`): the runtime knows nothing
* about tools or sessions — it is handed named async functions and a program,
* and everything tool-shaped stays with the consumer.
*
* @module @deepseek-ai/dsh-code-runtime
*/
import { Context, Service } from 'cordis'
import type { CodeRunRequest, CodeRunResult } from './types.ts'
export type {
CodeBindingFunction,
CodeBindingNamespace,
CodeLogEntry,
CodeRunFailure,
CodeRunRequest,
CodeRunResult,
} from './types.ts'
declare module 'cordis' {
interface Context {
codeRuntime: CodeRuntime
}
}
/**
* Abstract code-execution service. Subclass, implement {@link run} and the
* two descriptors, and load the subclass as a plugin — it registers as
* `ctx.codeRuntime` (one implementation per context; loading a second throws,
* cordis' standard duplicate-service behavior).
*
* Semantics every implementation must honor:
* - {@link run} resolves with an error FIELD for every program outcome —
* parse/transform failures, thrown exceptions, budget expiry, abort,
* substrate death ({@link CodeRunFailure}'s taxonomy). It REJECTS only for
* caller misuse of the seam itself (e.g. a run submitted after disposal).
* - Binding calls bridge to the caller's {@link CodeBindingFunction}s
* verbatim; arguments and resolutions must be structured-cloneable, and the
* runtime treats the program as a hostile peer (arbitrary binding names are
* own properties, malformed traffic is rejected or ignored, never crashes
* the host).
* - Runs are isolated from each other: no state survives from one run to the
* next through the runtime.
* - Disposal reaches quiescence: in-flight runs are terminated AND awaited
* before the service's own teardown completes (no orphan substrate survives
* `fiber.dispose()`).
*/
export abstract class CodeRuntime extends Service {
/**
* The source language {@link run} expects `program` to be written in, as a
* lowercase identifier. Informational, not gating — a consumer that
* generates language-specific presentation (typed SDK stubs, usage
* instructions) switches on it and fails loud on a language it cannot
* present. Well-known value: `'typescript'`.
*/
abstract readonly language: string
/**
* The execution substrate, as a lowercase identifier. Informational, not
* gating — a descriptor so deployments and diagnostics can tell backends
* apart, not a security claim. Well-known values: `'worker-thread'`,
* `'process'`, `'container'`.
*/
abstract readonly isolation: string
constructor(ctx: Context) {
super(ctx, 'codeRuntime')
}
/**
* Execute one program against the request's bindings and capture what it
* emitted. See the class doc for the resolution contract (error is a result
* field; rejection means seam misuse only).
* @param request - the program, its bindings, and the abort signal; the
* request carries everything the runtime acts on, with no hidden defaults.
* @returns the run's outcome: completion value (when transferable), the
* ordered log capture, and the failure (if any).
*/
abstract run(request: CodeRunRequest): Promise<CodeRunResult>
}
export default CodeRuntime

105
packages/code-runtime/code-runtime/src/types.ts Normal file
View File

@@ -0,0 +1,105 @@
/**
* Vocabulary types for the code-execution seam: what a caller hands a
* {@link ../index.ts | CodeRuntime} and what it gets back. Pure types — no
* runtime code lives here.
*
* @module @deepseek-ai/dsh-code-runtime/src/types
*/
/**
* One host-side function exposed to the program as an async callable. The
* runtime bridges calls to it (possibly across a serialization boundary), so
* `args` and the resolution value MUST be structured-cloneable; a runtime
* rejects a non-cloneable value with a descriptive error rather than
* corrupting the run. A rejection of this function surfaces inside the
* program as a rejection of the corresponding call.
*/
export type CodeBindingFunction = (args: unknown) => Promise<unknown>
/**
* A named group of {@link CodeBindingFunction}s the runtime exposes to the
* program as one global object (e.g. `tools`). Function names are arbitrary
* strings — a runtime must treat names like `__proto__` or `constructor` as
* ordinary own properties (null-prototype construction), never as prototype
* collisions.
*/
export interface CodeBindingNamespace {
/** The global identifier the program sees (must be a valid JS identifier). */
global: string
/** The callable members, keyed by the exact name the program calls. */
functions: Record<string, CodeBindingFunction>
}
/**
* One run: the program source plus everything the runtime acts on. Per the
* explicit-over-implicit convention, defaulting (time budgets, output caps)
* is the implementation's validated config — a request carries no optional
* tuning knobs for a hidden `??` to fill in.
*/
export interface CodeRunRequest {
/**
* The program source, in the runtime's {@link ../index.ts | language}. It
* runs as the body of an async function: top-level `await` and `return`
* are available, and the completion value becomes
* {@link CodeRunResult.value}.
*/
program: string
/** Host functions exposed to the program, one global object per namespace. */
bindings: CodeBindingNamespace[]
/**
* Abort the run: the runtime stops the program (hard, even mid-loop) and
* resolves with a {@link CodeRunFailure} of kind `'abort'`. In-flight
* binding calls are the CALLER's to settle — the runtime only stops asking.
*/
signal?: AbortSignal
}
/**
* One captured output entry, in emission order. `source` says which channel
* produced it: the program's `console` (shimmed by the runtime), or a stray
* write to the underlying stdout/stderr streams.
*/
export interface CodeLogEntry {
/** Which channel produced the text. */
source: 'console' | 'stdout' | 'stderr'
/** The console method used; present only when `source` is `'console'`. */
level?: 'log' | 'info' | 'warn' | 'error' | 'debug'
/** The captured text (possibly truncated by the implementation's caps, marked in-band). */
text: string
}
/**
* Why a run failed. The kinds are orthogonal outcomes reported independently
* (per docs/defensive-patterns.md): a budget expiry is not an exception, an
* abort is not a timeout, and a substrate death is neither.
*
* - `'exception'` — the program threw or failed to parse/transform.
* - `'timeout'` — an implementation-owned budget expired; the message says which.
* - `'abort'` — {@link CodeRunRequest.signal} fired.
* - `'worker-exit'` — the execution substrate died without settling (e.g. OOM).
*/
export interface CodeRunFailure {
/** The failure class (see the interface doc for each kind's meaning). */
kind: 'exception' | 'timeout' | 'abort' | 'worker-exit'
/** Human-readable detail, suitable for feeding back to a model to self-correct. */
message: string
}
/**
* The outcome of one run. An error is a FIELD on a resolved result, never a
* rejection of `run()` — reporting a failed program is the caller's job, not
* an exception path.
*/
export interface CodeRunResult {
/**
* The program's completion value (its top-level `return`), when it ran to
* completion and the value survived the runtime's serialization boundary;
* a non-transferable value is replaced by a string rendering, and a failed
* or value-less run leaves this absent.
*/
value?: unknown
/** Everything the program emitted, in order (capped by the implementation). */
logs: CodeLogEntry[]
/** Present iff the run failed; see {@link CodeRunFailure} for the taxonomy. */
error?: CodeRunFailure
}

87
packages/code-runtime/code-runtime/tests/service.spec.ts Normal file
View File

@@ -0,0 +1,87 @@
import { describe, expect, it } from 'vitest'
import { Context } from 'cordis'
import { CodeRuntime } from '@deepseek-ai/dsh-code-runtime'
import type { CodeRunRequest, CodeRunResult } from '@deepseek-ai/dsh-code-runtime'
/**
* Minimal concrete runtime: records requests, "executes" by invoking every
* binding once in declaration order, and lets tests script the outcome. The
* seam package ships no implementation, so the contract is exercised through
* the smallest subclass that honors it.
*/
class StubRuntime extends CodeRuntime {
readonly language = 'typescript'
readonly isolation = 'in-process-stub'
requests: CodeRunRequest[] = []
nextResult: CodeRunResult = { logs: [] }
async run(request: CodeRunRequest): Promise<CodeRunResult> {
this.requests.push(request)
if (request.signal?.aborted) {
return { logs: [], error: { kind: 'abort', message: String(request.signal.reason) } }
}
for (const namespace of request.bindings) {
for (const fn of Object.values(namespace.functions)) {
await fn({ from: 'stub' })
}
}
return this.nextResult
}
}
async function setup() {
const ctx = new Context()
await ctx.plugin(StubRuntime)
const runtime = ctx.codeRuntime as StubRuntime
return { ctx, runtime }
}
describe('CodeRuntime service seam', () => {
it('registers as ctx.codeRuntime and serves the abstract API', async () => {
const { runtime } = await setup()
expect(runtime.language).toBe('typescript')
expect(runtime.isolation).toBe('in-process-stub')
const calls: unknown[] = []
const result = await runtime.run({
program: 'return 1',
bindings: [{ global: 'tools', functions: { probe: async args => void calls.push(args) } }],
})
expect(result).toEqual({ logs: [] })
expect(calls).toEqual([{ from: 'stub' }])
expect(runtime.requests).toHaveLength(1)
})
it('reports a failed run as an error field on a resolved result, never a rejection', async () => {
const { runtime } = await setup()
runtime.nextResult = {
logs: [{ source: 'console', level: 'error', text: 'boom' }],
error: { kind: 'exception', message: 'boom' },
}
const result = await runtime.run({ program: 'throw new Error("boom")', bindings: [] })
expect(result.error).toEqual({ kind: 'exception', message: 'boom' })
expect(result.value).toBeUndefined()
})
it('reports a pre-aborted signal as an abort failure', async () => {
const { runtime } = await setup()
const controller = new AbortController()
controller.abort('cancelled')
const result = await runtime.run({ program: 'return 1', bindings: [], signal: controller.signal })
expect(result.error).toEqual({ kind: 'abort', message: 'cancelled' })
})
it('is removed from the context when the providing fiber disposes (HMR safety)', async () => {
const ctx = new Context()
const fiber = await ctx.plugin(StubRuntime)
expect(ctx.get('codeRuntime')).toBeInstanceOf(StubRuntime)
await fiber.dispose()
expect(ctx.get('codeRuntime')).toBeUndefined()
})
it('rejects a second implementation in the same context (duplicate service)', async () => {
const { ctx } = await setup()
await expect(ctx.plugin(StubRuntime)).rejects.toThrow(/registered/)
})
})

18
packages/code-runtime/code-runtime/tsconfig.json Normal file
View File

@@ -0,0 +1,18 @@
{
"extends": "../../../tsconfig.base.json",
"compilerOptions": {
"rootDir": "src",
"outDir": "lib/types"
},
"include": [
"src"
],
"references": [
{
"path": "../../../vendor/cosmokit"
},
{
"path": "../../../vendor/cordis"
}
]
}

9
packages/guard/README.md Normal file
View File

@@ -0,0 +1,9 @@
# guard/ — loop-hygiene guard family
Behavioral guard plugins that watch the agent loop for unproductive patterns and nudge the model back on course. A single **product** package — there is no interface/implementation seam here, because a guard is a self-contained consumer of existing core seams (`tools/post-execute`, `agent/prompt-submit`, `agent/status`), not a swappable capability.
| Package | Role | ctx key |
|---|---|---|
| `repeat-tool-guard/` | Advisory reminders when an agent loops on identical tool calls | (listens on `ctx.tools`' waterfalls) |
Reminders travel as `additionalContext` on the `tools/post-execute` decision; the agent loop appends them as logged `context/message` events after the step's tool results (see [the tools package](../core/tools)), so everything a guard says to the model is reconstructable from the session log.

37
packages/guard/repeat-tool-guard/README.md Normal file
View File

@@ -0,0 +1,37 @@
# @deepseek-ai/dsh-repeat-tool-guard
An advisory loop-breaker, not a model-facing tool: it never appears in the tool list, never vetoes or rewrites a call, and adds exactly one behavior — it watches each agent's stream of tool calls, counts runs of consecutive calls to the same tool with identical canonicalized arguments, and at configured run lengths injects an escalating advisory reminder telling the model to stop repeating itself, re-read the last result, and either change approach or conclude. The decision (retry differently, gather more evidence, or finish) stays entirely with the model: a legitimately repeated call is delayed by nothing and blocked by nothing. Decision record: [the repeat-tool-guard RFC](../../../docs/rfc/implemented/feature/2026-07-08-repeat-tool-guard.md).
## Config
```yaml
- id: repeat-tool-guard
name: '@deepseek-ai/dsh-repeat-tool-guard'
config:
thresholds: [3, 5, 8] # default; consecutive counts that trigger a reminder
include: [] # tool-name patterns to track; empty ⇒ all tools
exclude: [todo_write] # tool-name patterns transparent to the chain
argumentsPreviewChars: 500 # default; cap on arguments quoted in the detailed reminder
```
`thresholds` fails loud at plugin load: an empty list, a non-integer, a value below 2, or a duplicate throws, never a silent fall-back to defaults; `argumentsPreviewChars` equally rejects anything but an integer >= 1. The list is normalized to ascending order; the FIRST threshold delivers a short generic nudge, every later threshold delivers the detailed form naming the tool, the run length, and the canonical arguments — head-truncated at `argumentsPreviewChars` with an omitted-count marker, so a looping `write`/`edit` payload cannot ride into the next request unbounded (the chain key always compares the FULL canonical string; the cap bounds the reminder, never the detection).
`include`/`exclude` entries support `*` wildcards and are predicates over whatever tools exist at call time, not references to registry entries — a pattern matching no currently registered tool is NOT an error (`exclude: [mcp_*]` stays valid in a deployment that loads no MCP tools), unlike `toolOrder`'s referent check.
## Chain semantics
The chain key is `(tool name, canonical arguments)` — canonicalization is a deep key-sort plus `JSON.stringify`, so argument objects differing only in property order count as identical. A call identical to the previous tracked call increments the agent's consecutive counter; a different tracked call resets it to 1.
- **Untracked calls are transparent to the chain.** A call excluded by `include`/`exclude` neither increments nor resets the counter, so `grep X → todo_write → grep X` still counts as two consecutive `grep X` when `todo_write` is excluded. This is what makes exclusion useful: bookkeeping tools interleaved into a loop must not launder it.
- **Denied calls count.** Detection sits on `tools/post-execute`, which also runs for calls a `tools/pre-execute` listener denied — a model hammering a denied call is exactly the loop worth breaking.
- **Calls without an agent are ignored.** A direct `ctx.tools.execute()` caller has no model to remind and no `AgentId` to key on.
- **Per-agent keying.** The tool registry is context-level and subagents interleave through the same waterfall, so chains are keyed by `AgentId`; one agent's repetition never trips another's reminder. A user prompt (`agent/prompt-submit`) resets the submitting agent's chain; agent disposal drops its state.
- **In-memory only.** A session resumed from persistence starts with a fresh chain — the guard is a heuristic nudge, not a logged invariant, later reminders are the accepted cost.
## Reminder delivery
Reminders ride the post-execute decision's `additionalContext` (source `{kind: 'plugin', plugin: 'repeat-tool-guard'}`), never a `content` replacement: the `tool/result` event stays the tool's own output for audit. The loop buffers the context and appends it as a `context/message` after the step's tool results, which the session renders as the tagged synthetic-user envelope — so the reminder is model-visible, source-attributed, and reconstructable from the session log with no new session event. The guard always delegates via `next()` and folds its reminder onto the downstream decision (both variants — a blocked call still gets the nudge); when a downstream listener attached its own `additionalContext`, the fold concatenates content and carries the guard's `source` (a `HookContext` holds one `MessageSource`; `source.kind` is what framing depends on).
## Testing
Unit suites drive a real agent loop against a mock adapter (no network) and cover the chain semantics above to per-file 100%. The snapshot tier owns the transcript surface: a scripted-replay scenario repeats a call five times and pins both reminder tiers (gentle at 3, detailed at 5) as `context/message`s in the ACP transcript.

41
packages/guard/repeat-tool-guard/package.json Normal file
View File

@@ -0,0 +1,41 @@
{
"name": "@deepseek-ai/dsh-repeat-tool-guard",
"description": "Repeat-tool-call guard plugin: advisory reminders when an agent loops on identical tool calls",
"version": "0.0.1",
"private": true,
"type": "module",
"main": "lib/index.js",
"types": "lib/types/index.d.ts",
"exports": {
".": {
"types": "./lib/types/index.d.ts",
"default": "./lib/index.js"
},
"./src/*": "./src/*",
"./package.json": "./package.json"
},
"files": [
"lib/index.js",
"lib/types/**/*.d.ts",
"lib/types/**/*.d.ts.map",
"src"
],
"license": "BSD-3-Clause",
"dependencies": {
"schemastery": "^3.18.0"
},
"peerDependencies": {
"@deepseek-ai/dsh-agent": "^0.0.1",
"@deepseek-ai/dsh-tools": "^0.0.1",
"cordis": "^4.0.0-rc.6"
},
"devDependencies": {
"@deepseek-ai/dsh-agent": "workspace:^",
"@deepseek-ai/dsh-agent-loop": "workspace:^",
"@deepseek-ai/dsh-llm": "workspace:^",
"@deepseek-ai/dsh-session": "workspace:^",
"@deepseek-ai/dsh-system-prompt": "workspace:^",
"@deepseek-ai/dsh-tools": "workspace:^",
"cordis": "^4.0.0-rc.6"
}
}

268
packages/guard/repeat-tool-guard/src/index.ts Normal file
View File

@@ -0,0 +1,268 @@
/**
* Repeat-tool-call guard: advisory loop-breaker for agents stuck re-issuing
* the same tool call with identical arguments.
*
* Not a model-facing tool — it registers no tool, never vetoes or rewrites a
* call, and adds exactly one behavior: watch each agent's stream of tool calls
* through the `tools/post-execute` waterfall, count runs of consecutive calls
* to the same tool with identical canonicalized arguments, and at configured
* run lengths fold an escalating advisory reminder onto the decision's
* `additionalContext`. The loop appends that context as a logged
* `context/message` after the step's tool results, so the reminder is
* model-visible, source-attributed, and reconstructable from the session log
* with no new session event. Decision record:
* docs/rfc/implemented/feature/2026-07-08-repeat-tool-guard.md.
*
* ```yaml
* - id: repeat-tool-guard
* name: '@deepseek-ai/dsh-repeat-tool-guard'
* config:
* thresholds: [3, 5, 8] # consecutive counts that trigger a reminder
* include: [] # tool-name patterns to track; empty = all tools
* exclude: [todo_write] # tool-name patterns transparent to the chain
* ```
*
* Chain state is keyed per {@link AgentId} — the tool registry is a
* context-level singleton whose waterfalls interleave every agent's calls, so
* a shared counter would let one agent's repetition trip another's reminder.
* State is in-memory only: a session resumed from persistence starts with a
* fresh chain (the guard is a heuristic nudge, not a logged invariant).
*
* Plugin export shape: named exports, NO default. The cordis Loader's
* `unwrapExports` does `exports.default ?? exports`, so a stray default would
* collapse the module to the bare `apply` (see docs/postmortem/0001).
*
* @module @deepseek-ai/dsh-repeat-tool-guard
*/
import type { Context } from 'cordis'
import z from 'schemastery'
import type { AgentId, HookContext, PromptDecision } from '@deepseek-ai/dsh-agent'
import type { MessageSource } from '@deepseek-ai/dsh-llm'
import type { PostToolDecision, ToolExecution } from '@deepseek-ai/dsh-tools'
export const name = 'repeat-tool-guard'
/**
* Plugin config, validated by the same-named schemastery schema plus the
* load-time checks in `apply` (misconfiguration fails loud: an empty
* `thresholds` list, a non-integer, a value below 2, or a duplicate throws at
* plugin load, never a silent fall-back). `include`/`exclude` entries are
* `*`-wildcard predicates over tool names at call time, not references to
* registry entries — a pattern matching no currently registered tool is valid
* (`exclude: [mcp_*]` must stay legal in a deployment that loads no MCP tools).
*/
export interface Config {
/** Consecutive-repeat counts that trigger a reminder (default `[3, 5, 8]`). */
thresholds?: number[]
/** Tool-name patterns to track; empty means every tool is tracked. */
include?: string[]
/** Tool-name patterns transparent to the chain (neither count nor reset). */
exclude?: string[]
/**
* Maximum characters of canonical arguments quoted in the DETAILED reminder
* (default 500). Large payloads (a `write` body, a long command) would
* otherwise ride into the next request unbounded — precisely in a loop
* scenario; the cap bounds the reminder, never the detection (the chain key
* always compares the FULL canonical string).
*/
argumentsPreviewChars?: number
}
export const Config: z<Config> = z.object({
thresholds: z.array(z.number()).default([3, 5, 8]),
include: z.array(z.string()).default([]),
exclude: z.array(z.string()).default([]),
argumentsPreviewChars: z.number().default(500),
})
/**
* The `{kind:'plugin'}` source stamped on every reminder this guard injects —
* the label is load-bearing (an unlabeled context would render as a user
* prompt in derived history).
*/
const PLUGIN_SOURCE: MessageSource = { kind: 'plugin', plugin: 'repeat-tool-guard' }
/**
* The gentle first-threshold reminder. Keyed to `thresholds[0]`, not a literal
* count, so a custom first threshold keeps the gentle-then-detailed escalation.
*/
const GENTLE_REMINDER =
'You are repeating the exact same tool call with identical arguments. '
+ 'Carefully analyze the previous result before calling again: if the task is '
+ 'not complete, try a different approach or different arguments instead of '
+ 'repeating the call.'
/** The detailed later-threshold reminder naming the tool, the run length, and the canonical arguments. */
function detailedReminder(toolName: string, count: number, canonicalArguments: string): string {
return 'Repeated tool call detected:\n'
+ `- tool: ${toolName}\n`
+ `- consecutive_calls: ${count}\n`
+ `- arguments: ${canonicalArguments}\n`
+ 'The repeated calls are not making progress. Do not call this tool with '
+ 'these exact arguments again. Inspect the latest result and choose a '
+ 'different action, different arguments, or finish the task if enough '
+ 'evidence has been gathered.'
}
/**
* Deep key-sort of a parsed-JSON value so two argument objects that differ
* only in property order canonicalize identically. Arguments reach the guard
* as the loop's `JSON.parse` output (or its raw-string fallback for malformed
* argument JSON), so JSON's value domain is the whole input domain — no
* bigint, cycle, or `undefined` handling exists because no input path can
* produce them.
*/
function sortJsonValue(value: unknown): unknown {
if (Array.isArray(value)) return value.map(sortJsonValue)
if (value !== null && typeof value === 'object') {
const record = value as Record<string, unknown>
const sorted: Record<string, unknown> = {}
for (const key of Object.keys(record).sort()) {
sorted[key] = sortJsonValue(record[key])
}
return sorted
}
return value
}
/** Canonical string form of a call's arguments: deep key-sort, then stringify. */
function canonicalize(argumentsValue: unknown): string {
return JSON.stringify(sortJsonValue(argumentsValue))
}
/** Compile one `*`-wildcard pattern to an anchored RegExp (every other regex metacharacter is matched literally). */
function wildcardToRegExp(pattern: string): RegExp {
const escaped = pattern.replace(/[|\\{}()[\]^$+?.]/g, String.raw`\$&`)
return new RegExp(`^${escaped.replaceAll('*', '.*')}$`)
}
/**
* Head-truncate the canonical arguments for quoting in the detailed reminder,
* marking how much was omitted. Bounds only the model-visible text — the
* chain key always uses the full canonical string.
*/
function previewArguments(canonical: string, cap: number): string {
if (canonical.length <= cap) return canonical
return `${canonical.slice(0, cap)}… (+${canonical.length - cap} more chars)`
}
/**
* Validate `thresholds` per the fail-loud contract and return them sorted
* ascending (the escalation rule reads `thresholds[0]` as the gentle tier, so
* order is normalized here, once).
*/
function validateThresholds(values: number[]): number[] {
if (values.length === 0) {
throw new Error('repeat-tool-guard: `thresholds` must not be empty')
}
for (const value of values) {
if (!Number.isInteger(value) || value < 2) {
throw new Error(`repeat-tool-guard: invalid threshold ${value} — every threshold must be an integer >= 2`)
}
}
if (new Set(values).size !== values.length) {
throw new Error('repeat-tool-guard: `thresholds` must not contain duplicates')
}
return [...values].sort((a, b) => a - b)
}
/**
* Concatenate the guard's reminder context with a downstream listener's
* optional one so folding drops neither. The merged block carries the guard's
* `source` — a `HookContext` holds one `MessageSource` and the seam cannot
* represent mixed provenance; the rendered `context/message` only
* distinguishes by `source.kind`, so a downstream plugin's text is still
* correctly framed as plugin context.
*/
function concatContext(ours: HookContext, theirs: HookContext | undefined): HookContext {
if (!theirs) return ours
return { content: [...ours.content, ...theirs.content], source: ours.source }
}
/** One agent's consecutive-repeat chain: the last tracked call's identity key and its run length. */
interface Chain {
key: string
count: number
}
/**
* Install the guard's listeners.
* @param ctx - plugin context; listeners are scoped to it and disposed with it.
* @param config - validated {@link Config}; `thresholds` is re-checked fail-loud here.
*/
export function apply(ctx: Context, config: Config): void {
// schemastery's .default() guarantees the fields are set after validation.
const thresholds = validateThresholds(config.thresholds as number[])
const thresholdSet = new Set(thresholds)
const includePatterns = (config.include as string[]).map(wildcardToRegExp)
const excludePatterns = (config.exclude as string[]).map(wildcardToRegExp)
const argumentsPreviewChars = config.argumentsPreviewChars as number
if (!Number.isInteger(argumentsPreviewChars) || argumentsPreviewChars < 1) {
throw new Error(`repeat-tool-guard: invalid argumentsPreviewChars ${argumentsPreviewChars} — must be an integer >= 1`)
}
const chains = new Map<AgentId, Chain>()
/** Whether a tool participates in the chain (untracked calls are transparent: they neither count nor reset). */
function tracked(toolName: string): boolean {
if (includePatterns.length > 0 && !includePatterns.some(pattern => pattern.test(toolName))) return false
return !excludePatterns.some(pattern => pattern.test(toolName))
}
/**
* Advance the calling agent's chain for one attempt and return the reminder
* to deliver, if this attempt's run length hits a configured threshold.
* Counting happens here — in post-execute — because denied calls also flow
* through this waterfall (`ToolRegistry.execute` routes a deny through the
* same pipeline), and a model hammering a denied call is exactly the loop
* worth breaking.
*/
function observe(exec: ToolExecution): HookContext | undefined {
// A direct `ctx.tools.execute()` caller has no model to remind and no id
// to key on; only agent-loop calls participate.
if (!exec.agent) return undefined
if (!tracked(exec.name)) return undefined
const canonical = canonicalize(exec.arguments)
const key = JSON.stringify([exec.name, canonical])
const chain = chains.get(exec.agent.id)
const count = chain !== undefined && chain.key === key ? chain.count + 1 : 1
chains.set(exec.agent.id, { key, count })
if (!thresholdSet.has(count)) return undefined
const text = count === thresholds[0]
? GENTLE_REMINDER
: detailedReminder(exec.name, count, previewArguments(canonical, argumentsPreviewChars))
return { content: [{ type: 'text', text }], source: PLUGIN_SOURCE }
}
// Observe-and-enrich, never veto: count first (state advances regardless of
// the downstream outcome), DELEGATE so a later listener can still block or
// replace, then fold the reminder onto whatever came back — additionalContext
// rides both decision variants, so a blocked call still gets the nudge.
ctx.on('tools/post-execute', async (exec, _result, next): Promise<PostToolDecision> => {
const reminder = observe(exec)
const downstream = await next()
if (!reminder) return downstream
if (downstream.kind === 'block') {
return { kind: 'block', feedback: downstream.feedback, additionalContext: concatContext(reminder, downstream.additionalContext) }
}
return {
kind: 'accept',
...downstream.content !== undefined ? { content: downstream.content } : {},
additionalContext: concatContext(reminder, downstream.additionalContext),
}
})
// A user interjection changes the context; repetition across it is not a
// loop. Pure reset hook: always delegates (attaching nothing, vetoing
// nothing).
ctx.on('agent/prompt-submit', (agent, _content, _source, next): Promise<PromptDecision> => {
chains.delete(agent.id)
return next()
})
// Drop state when an agent goes away, bounding the map over harness lifetime.
ctx.on('agent/status', (agent, status) => {
if (status === 'disposed') chains.delete(agent.id)
})
}

401
packages/guard/repeat-tool-guard/tests/repeat-tool-guard.spec.ts Normal file
View File

@@ -0,0 +1,401 @@
import { describe, expect, it } from 'vitest'
import { Context } from 'cordis'
import LlmService, { CallId } from '@deepseek-ai/dsh-llm'
import SessionStore, { type SessionEvent } from '@deepseek-ai/dsh-session'
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
import ToolRegistry, { defineTool } from '@deepseek-ai/dsh-tools'
import AgentRegistry, { AgentId } from '@deepseek-ai/dsh-agent'
import AgentLoop, { type ReactLoopAgent } from '@deepseek-ai/dsh-agent-loop'
import * as RepeatToolGuard from '@deepseek-ai/dsh-repeat-tool-guard'
import type { Config } from '@deepseek-ai/dsh-repeat-tool-guard'
import { MockAdapter, textResponse, toolCallResponse } from '../../../core/agent-loop/tests/mock-adapter.ts'
/**
* Behavior suite for the repeat-tool-call guard: chain semantics (identical /
* different-tracked / untracked-transparent / per-agent / resets), threshold
* escalation incl. the `thresholds[0]` gentle-text rule, canonicalization,
* fold-onto-downstream-decision, and fail-loud config validation — all driven
* through a real agent loop against a scripted mock adapter (no network).
*/
/** Boot the core spine + the guard; the caller registers adapters and extra listeners. */
async function harness(config: Config = {}): Promise<Context> {
const ctx = new Context()
await ctx.plugin(LlmService)
await ctx.plugin(SessionStore)
await ctx.plugin(SystemPrompt)
await ctx.plugin(ToolRegistry)
await ctx.plugin(AgentRegistry)
await ctx.plugin(AgentLoop, { agents: [] })
await ctx.plugin(RepeatToolGuard, config)
ctx.tools.register(defineTool({ name: 'probe', description: 'p', parameters: {}, async execute() { return [{ type: 'text', text: 'ok' }] } }))
ctx.tools.register(defineTool({ name: 'other', description: 'o', parameters: {}, async execute() { return [{ type: 'text', text: 'ok' }] } }))
return ctx
}
function waitForIdle(ctx: Context, agent: ReactLoopAgent): Promise<void> {
return new Promise((resolve) => { const d = ctx.on('agent/status', (s, st) => { if (s === agent && st === 'idle') { d(); resolve() } }) })
}
/** Every `context/message` in the agent's log, flattened to joined text + source for terse assertions. */
function reminders(agent: ReactLoopAgent): { text: string; source: unknown }[] {
return [...agent.session.events]
.filter((e): e is SessionEvent<'context/message'> => e.type === 'context/message')
.map(e => ({
text: e.data.content.map(block => block.type === 'text' ? block.text : '').join('|'),
source: e.data.source,
}))
}
const GUARD_SOURCE = { kind: 'plugin', plugin: 'repeat-tool-guard' }
describe('threshold escalation', () => {
it('reminds gently at the first default threshold (3) and in detail at the second (5)', async () => {
const ctx = await harness()
const adapter = new MockAdapter([
...Array.from({ length: 5 }, (_, i) => toolCallResponse(`c${i}`, 'probe', { q: 'same' })),
textResponse('done'),
])
ctx.llm.registerAdapter(['mock'], adapter)
const agent = ctx.agentLoop.create(AgentId('a1'), { model: 'mock' })
agent.send([{ type: 'text', text: 'go' }])
await waitForIdle(ctx, agent)
const found = reminders(agent)
expect(found).toHaveLength(2)
expect(found[0]!.text).toContain('repeating the exact same tool call')
expect(found[0]!.source).toEqual(GUARD_SOURCE)
expect(found[1]!.text).toContain('consecutive_calls: 5')
expect(found[1]!.text).toContain('- tool: probe')
expect(found[1]!.text).toContain('{"q":"same"}')
expect(found[1]!.source).toEqual(GUARD_SOURCE)
})
it('keys the gentle text to thresholds[0], not the literal 3', async () => {
const ctx = await harness({ thresholds: [4, 2] }) // unsorted on purpose: normalized ascending
const adapter = new MockAdapter([
...Array.from({ length: 4 }, (_, i) => toolCallResponse(`c${i}`, 'probe', {})),
textResponse('done'),
])
ctx.llm.registerAdapter(['mock'], adapter)
const agent = ctx.agentLoop.create(AgentId('a1'), { model: 'mock' })
agent.send([{ type: 'text', text: 'go' }])
await waitForIdle(ctx, agent)
const found = reminders(agent)
expect(found).toHaveLength(2)
expect(found[0]!.text).toContain('repeating the exact same tool call') // gentle at 2
expect(found[1]!.text).toContain('consecutive_calls: 4') // detailed at 4
})
})
describe('chain semantics', () => {
it('caps the detailed reminder arguments at argumentsPreviewChars (detection still keys on the full string)', async () => {
const ctx = await harness({ thresholds: [2, 3], argumentsPreviewChars: 24 })
const bigPayload = 'x'.repeat(400)
const adapter = new MockAdapter([
toolCallResponse('c1', 'probe', { body: bigPayload }),
toolCallResponse('c2', 'probe', { body: bigPayload }),
toolCallResponse('c3', 'probe', { body: bigPayload }),
textResponse('done'),
])
ctx.llm.registerAdapter(['mock'], adapter)
const agent = ctx.agentLoop.create(AgentId('a1'), { model: 'mock' })
agent.send([{ type: 'text', text: 'go' }])
await waitForIdle(ctx, agent)
const found = reminders(agent)
expect(found).toHaveLength(2) // gentle at 2, detailed at 3 — full-key matching survived the cap
const detailed = found[1]!.text
expect(detailed).toContain('- arguments: {"body":"xxxxxxxxxxxxxx') // 24-char head
expect(detailed).toContain('… (+387 more chars)')
expect(detailed).not.toContain(bigPayload)
})
it('a different tracked call resets the chain', async () => {
const ctx = await harness()
const adapter = new MockAdapter([
toolCallResponse('c1', 'probe', { q: 1 }),
toolCallResponse('c2', 'probe', { q: 1 }),
toolCallResponse('c3', 'other', {}), // tracked, different → reset
toolCallResponse('c4', 'probe', { q: 1 }),
toolCallResponse('c5', 'probe', { q: 1 }),
toolCallResponse('c6', 'probe', { q: 1 }), // 3rd consecutive AFTER the reset
textResponse('done'),
])
ctx.llm.registerAdapter(['mock'], adapter)
const agent = ctx.agentLoop.create(AgentId('a1'), { model: 'mock' })
agent.send([{ type: 'text', text: 'go' }])
await waitForIdle(ctx, agent)
expect(reminders(agent)).toHaveLength(1)
})
it('excluded calls are transparent: they neither count nor reset', async () => {
const ctx = await harness({ exclude: ['other'] })
const adapter = new MockAdapter([
toolCallResponse('c1', 'probe', { q: 1 }),
toolCallResponse('c2', 'other', {}), // excluded → invisible to the chain
toolCallResponse('c3', 'probe', { q: 1 }),
toolCallResponse('c4', 'other', {}),
toolCallResponse('c5', 'probe', { q: 1 }), // 3rd consecutive probe
textResponse('done'),
])
ctx.llm.registerAdapter(['mock'], adapter)
const agent = ctx.agentLoop.create(AgentId('a1'), { model: 'mock' })
agent.send([{ type: 'text', text: 'go' }])
await waitForIdle(ctx, agent)
const found = reminders(agent)
expect(found).toHaveLength(1)
expect(found[0]!.text).toContain('repeating the exact same tool call')
})
it('include patterns track only matching tools (wildcard star)', async () => {
const ctx = await harness({ include: ['pro*'] })
const adapter = new MockAdapter([
toolCallResponse('c1', 'other', {}),
toolCallResponse('c2', 'other', {}),
toolCallResponse('c3', 'other', {}), // 3 identical, but untracked
toolCallResponse('c4', 'probe', {}),
toolCallResponse('c5', 'probe', {}),
toolCallResponse('c6', 'probe', {}), // 3 identical, tracked
textResponse('done'),
])
ctx.llm.registerAdapter(['mock'], adapter)
const agent = ctx.agentLoop.create(AgentId('a1'), { model: 'mock' })
agent.send([{ type: 'text', text: 'go' }])
await waitForIdle(ctx, agent)
const found = reminders(agent)
expect(found).toHaveLength(1)
expect(found[0]!.text).toContain('repeating the exact same tool call')
})
it('escapes regex metacharacters in patterns (a dot matches only a literal dot)', async () => {
const ctx = await harness({ exclude: ['pr.be'] }) // would match 'probe' as a regex; must not as a wildcard
const adapter = new MockAdapter([
...Array.from({ length: 3 }, (_, i) => toolCallResponse(`c${i}`, 'probe', {})),
textResponse('done'),
])
ctx.llm.registerAdapter(['mock'], adapter)
const agent = ctx.agentLoop.create(AgentId('a1'), { model: 'mock' })
agent.send([{ type: 'text', text: 'go' }])
await waitForIdle(ctx, agent)
expect(reminders(agent)).toHaveLength(1) // probe was NOT excluded
})
it('canonicalization ignores property order, deeply', async () => {
const ctx = await harness()
const adapter = new MockAdapter([
toolCallResponse('c1', 'probe', { a: 1, nested: { x: [1, 2], y: null } }),
toolCallResponse('c2', 'probe', { nested: { y: null, x: [1, 2] }, a: 1 }),
toolCallResponse('c3', 'probe', { a: 1, nested: { x: [1, 2], y: null } }),
textResponse('done'),
])
ctx.llm.registerAdapter(['mock'], adapter)
const agent = ctx.agentLoop.create(AgentId('a1'), { model: 'mock' })
agent.send([{ type: 'text', text: 'go' }])
await waitForIdle(ctx, agent)
expect(reminders(agent)).toHaveLength(1) // all three canonicalize identically
})
it('keys chains per agent: one agent repeating never trips another', async () => {
const ctx = await harness()
ctx.llm.registerAdapter(['mock-a'], new MockAdapter([
toolCallResponse('a1', 'probe', { q: 1 }),
toolCallResponse('a2', 'probe', { q: 1 }),
textResponse('done'),
]))
ctx.llm.registerAdapter(['mock-b'], new MockAdapter([
toolCallResponse('b1', 'probe', { q: 1 }),
toolCallResponse('b2', 'probe', { q: 1 }),
toolCallResponse('b3', 'probe', { q: 1 }),
textResponse('done'),
]))
const agentA = ctx.agentLoop.create(AgentId('a'), { model: 'mock-a' })
const agentB = ctx.agentLoop.create(AgentId('b'), { model: 'mock-b' })
agentA.send([{ type: 'text', text: 'go' }])
agentB.send([{ type: 'text', text: 'go' }])
await Promise.all([waitForIdle(ctx, agentA), waitForIdle(ctx, agentB)])
expect(reminders(agentA)).toHaveLength(0) // 2 repeats < 3, despite B's 3 in the same registry
expect(reminders(agentB)).toHaveLength(1)
})
it('a new user prompt resets the chain', async () => {
const ctx = await harness()
const adapter = new MockAdapter([
toolCallResponse('c1', 'probe', { q: 1 }),
toolCallResponse('c2', 'probe', { q: 1 }),
textResponse('turn one done'),
toolCallResponse('c3', 'probe', { q: 1 }), // without the reset this would be the 3rd
textResponse('turn two done'),
])
ctx.llm.registerAdapter(['mock'], adapter)
const agent = ctx.agentLoop.create(AgentId('a1'), { model: 'mock' })
agent.send([{ type: 'text', text: 'go' }])
await waitForIdle(ctx, agent)
agent.send([{ type: 'text', text: 'again' }])
await waitForIdle(ctx, agent)
expect(reminders(agent)).toHaveLength(0)
})
it('drops an agent chain on disposal', async () => {
const ctx = await harness({ thresholds: [2] })
ctx.llm.registerAdapter(['mock'], new MockAdapter([
toolCallResponse('c1', 'probe', { q: 1 }),
textResponse('done'),
toolCallResponse('c2', 'probe', { q: 1 }), // same id, fresh agent: count 1, not 2
textResponse('done'),
]))
// Loop agents are torn down by disposing the scope that created them
// (the loop.spec pattern): a child plugin fiber owns `first`.
let first!: ReactLoopAgent
const fiber = await ctx.plugin(Object.assign((inner: Context) => {
first = inner.agentLoop.create(AgentId('reused'), { model: 'mock' })
}, { inject: ['agentLoop'] }))
first.send([{ type: 'text', text: 'go' }])
await waitForIdle(ctx, first)
await fiber.dispose()
await first.done
const second = ctx.agentLoop.create(AgentId('reused'), { model: 'mock' })
second.send([{ type: 'text', text: 'go' }])
await waitForIdle(ctx, second)
expect(reminders(second)).toHaveLength(0)
})
it('counts denied calls: hammering a denied tool still draws the reminder', async () => {
const ctx = await harness({ thresholds: [2] })
ctx.on('tools/pre-execute', async () => ({ kind: 'deny' as const, reason: 'sealed' }))
const adapter = new MockAdapter([
toolCallResponse('c1', 'probe', { q: 1 }),
toolCallResponse('c2', 'probe', { q: 1 }),
textResponse('done'),
])
ctx.llm.registerAdapter(['mock'], adapter)
const agent = ctx.agentLoop.create(AgentId('a1'), { model: 'mock' })
agent.send([{ type: 'text', text: 'go' }])
await waitForIdle(ctx, agent)
expect(reminders(agent)).toHaveLength(1)
})
it('ignores direct executes with no agent (they neither crash nor advance any chain)', async () => {
const ctx = await harness({ thresholds: [2] })
const direct = await ctx.tools.execute({ callId: CallId('d1'), name: 'probe', arguments: { q: 1 } })
expect(direct.isError).toBe(false)
ctx.llm.registerAdapter(['mock'], new MockAdapter([
toolCallResponse('c1', 'probe', { q: 1 }), // if the direct call had counted, this would be #2
textResponse('done'),
]))
const agent = ctx.agentLoop.create(AgentId('a1'), { model: 'mock' })
agent.send([{ type: 'text', text: 'go' }])
await waitForIdle(ctx, agent)
expect(reminders(agent)).toHaveLength(0)
})
})
describe('fold onto the downstream decision', () => {
it('folds the reminder onto a downstream block and keeps its feedback', async () => {
const ctx = await harness({ thresholds: [2] })
ctx.on('tools/post-execute', async () => ({
kind: 'block' as const,
feedback: [{ type: 'text' as const, text: 'nope' }],
additionalContext: { content: [{ type: 'text' as const, text: 'downstream-ctx' }], source: { kind: 'plugin' as const, plugin: 'test' } },
}))
const adapter = new MockAdapter([
toolCallResponse('c1', 'probe', { q: 1 }),
toolCallResponse('c2', 'probe', { q: 1 }),
textResponse('done'),
])
ctx.llm.registerAdapter(['mock'], adapter)
const agent = ctx.agentLoop.create(AgentId('a1'), { model: 'mock' })
agent.send([{ type: 'text', text: 'go' }])
await waitForIdle(ctx, agent)
const found = reminders(agent)
expect(found).toHaveLength(2)
// Call 1: below threshold — the downstream context passes through untouched.
expect(found[0]!.text).toBe('downstream-ctx')
expect(found[0]!.source).toEqual({ kind: 'plugin', plugin: 'test' })
// Call 2: reminder folded in front, single merged context, the guard's source.
expect(found[1]!.text).toContain('repeating the exact same tool call')
expect(found[1]!.text).toContain('|downstream-ctx')
expect(found[1]!.source).toEqual(GUARD_SOURCE)
// The block's feedback reached the tool result unchanged.
const results = [...agent.session.events].filter((e): e is SessionEvent<'tool/result'> => e.type === 'tool/result')
expect(results.every(r => r.data.isError)).toBe(true)
expect(results[1]!.data.content).toEqual([{ type: 'text', text: 'nope' }])
})
it('preserves a downstream accept content replacement while folding', async () => {
const ctx = await harness({ thresholds: [2] })
ctx.on('tools/post-execute', async () => ({
kind: 'accept' as const,
content: [{ type: 'text' as const, text: 'replaced' }],
}))
const adapter = new MockAdapter([
toolCallResponse('c1', 'probe', { q: 1 }),
toolCallResponse('c2', 'probe', { q: 1 }),
textResponse('done'),
])
ctx.llm.registerAdapter(['mock'], adapter)
const agent = ctx.agentLoop.create(AgentId('a1'), { model: 'mock' })
agent.send([{ type: 'text', text: 'go' }])
await waitForIdle(ctx, agent)
const found = reminders(agent)
expect(found).toHaveLength(1)
expect(found[0]!.text).toContain('repeating the exact same tool call')
const results = [...agent.session.events].filter((e): e is SessionEvent<'tool/result'> => e.type === 'tool/result')
expect(results[1]!.data.content).toEqual([{ type: 'text', text: 'replaced' }])
})
})
describe('config validation fails loud', () => {
async function spine(): Promise<Context> {
const ctx = new Context()
await ctx.plugin(LlmService)
await ctx.plugin(SessionStore)
await ctx.plugin(SystemPrompt)
await ctx.plugin(ToolRegistry)
await ctx.plugin(AgentRegistry)
await ctx.plugin(AgentLoop, { agents: [] })
return ctx
}
it('rejects an empty thresholds list', async () => {
const ctx = await spine()
await expect(ctx.plugin(RepeatToolGuard, { thresholds: [] })).rejects.toThrow(/must not be empty/)
})
it('rejects a threshold below 2', async () => {
const ctx = await spine()
await expect(ctx.plugin(RepeatToolGuard, { thresholds: [1, 3] })).rejects.toThrow(/integer >= 2/)
})
it('rejects a non-integer threshold', async () => {
const ctx = await spine()
await expect(ctx.plugin(RepeatToolGuard, { thresholds: [2.5] })).rejects.toThrow(/integer >= 2/)
})
it('rejects duplicate thresholds', async () => {
const ctx = await spine()
await expect(ctx.plugin(RepeatToolGuard, { thresholds: [3, 3] })).rejects.toThrow(/duplicates/)
})
it('rejects a non-positive or fractional argumentsPreviewChars', async () => {
const ctx = await spine()
await expect(ctx.plugin(RepeatToolGuard, { argumentsPreviewChars: 0 })).rejects.toThrow(/argumentsPreviewChars/)
const ctx2 = await spine()
await expect(ctx2.plugin(RepeatToolGuard, { argumentsPreviewChars: 12.5 })).rejects.toThrow(/argumentsPreviewChars/)
})
})

30
packages/guard/repeat-tool-guard/tsconfig.json Normal file
View File

@@ -0,0 +1,30 @@
{
"extends": "../../../tsconfig.base.json",
"compilerOptions": {
"rootDir": "src",
"outDir": "lib/types"
},
"include": [
"src"
],
"references": [
{
"path": "../../../vendor/cosmokit"
},
{
"path": "../../../vendor/cordis"
},
{
"path": "../../../vendor/schemastery"
},
{
"path": "../../core/tools"
},
{
"path": "../../core/agent"
},
{
"path": "../../llm/llm"
}
]
}

3
packages/support/README.md
View File

@@ -4,8 +4,9 @@ Packages that exist to serve development, testing, and the examples rather than
| Package | Role | ctx key |
|---|---|---|
| `acp-snapshot/` | ACP snapshot suite kit: subprocess scenario harness + golden normalizers + the `defineAcpSnapshotSuite` factory | (library — imported by example `*.snapshot.ts` suites) |
| `invariants/` | Dev-mode event-contract invariants + session-log freeze | (listens on `session/*`, `agent/*`) |
| `llm-replay/` | Record/replay adapter: short-circuits `llm/stream` from a recorded session JSONL (keyless snapshot tests) | (listens on `llm/stream`) |
| `subagent-mock/` | Scripted `SubagentProvider` for deterministic seam/tool tests | (registers on `ctx.subagents`) |
`invariants` runs only in dev mode (contract checks, not runtime behavior). `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate. `subagent-mock` exercises the real `ctx.subagents` load path without a model or child agent. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.
`invariants` runs only in dev mode (contract checks, not runtime behavior). `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate. `acp-snapshot` carries the snapshot tier's harness/normalizer/suite machinery so every example's suite is a scenario table over one shared, gate-covered implementation. `subagent-mock` exercises the real `ctx.subagents` load path without a model or child agent. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.

36
packages/support/acp-snapshot/README.md Normal file
View File

@@ -0,0 +1,36 @@
# `@deepseek-ai/dsh-acp-snapshot`
The ACP snapshot suite kit: the shared machinery behind the keyless snapshot tier (`pnpm run test:snapshot`, [testing policy](../../../docs/testing.md)). An example gets a full snapshot suite from a scenario table plus a fixtures directory; every compare/guard mechanic lives here, under the per-file coverage gate, instead of being copied per example.
Three layers, importable separately:
- **`runScenario` (harness)** — boots the real agent bin as a subprocess via tsx (unbuilt, Loader path), drives it over ACP JSON-RPC stdio from a deterministic `input.json` script, tees raw stdout for the golden + purity check, and harvests every persisted session JSONL (parent + subagent children, primary-first) after a graceful stdin-EOF shutdown. Parameterized by `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath` — absolute paths; the subprocess cwd is a temp dir outside the repo).
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept), and the composable `scrubRequestHeaders` (header bulk → `{{system}}`/`{{tools}}`, structure kept — [pinned-header RFC](../../../docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record-mode fixture write-back, the per-suite header pin with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin, non-pinning fixtures header-scrubbed). Must be called at vitest collection time.
A consuming `*.snapshot.ts` is the scenario table plus one factory call:
```ts
import { dirname, join } from 'node:path'
import { fileURLToPath } from 'node:url'
import { defineAcpSnapshotSuite, type Scenario } from '@deepseek-ai/dsh-acp-snapshot'
const SCENARIOS: Scenario[] = [
{ name: 'text-turn', hasModelTurn: true, recorded: true, pinsHeader: true },
]
defineAcpSnapshotSuite({
agent: { // absolute paths, resolved from the suite's own location
binScript: fileURLToPath(new URL('../../../packages/ui/acp-agent/src/bin.ts', import.meta.url)),
configPath: fileURLToPath(new URL('../cordis.yml', import.meta.url)),
tsconfigPath: fileURLToPath(new URL('../../../tsconfig.json', import.meta.url)),
},
snapshotsDir: join(dirname(fileURLToPath(import.meta.url)), 'snapshots'),
scenarios: SCENARIOS, // exactly one entry sets pinsHeader
mode: process.env.DSH_SNAPSHOT === 'record' ? 'record' : 'replay',
})
```
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config RFC](../../../docs/rfc/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. Fixture roles, record/replay semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
Constraints: `suite.ts` imports vitest, so the package is importable only inside a vitest run (the harness and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the harness speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug).

35
packages/support/acp-snapshot/package.json Normal file
View File

@@ -0,0 +1,35 @@
{
"name": "@deepseek-ai/dsh-acp-snapshot",
"description": "ACP snapshot suite kit: real-subprocess scenario harness, golden normalizers, and the suite factory behind the keyless snapshot tier",
"version": "0.0.1",
"private": true,
"type": "module",
"main": "lib/index.js",
"types": "lib/types/index.d.ts",
"exports": {
".": {
"types": "./lib/types/index.d.ts",
"default": "./lib/index.js"
},
"./src/*": "./src/*",
"./package.json": "./package.json"
},
"files": [
"lib/index.js",
"lib/types/**/*.d.ts",
"lib/types/**/*.d.ts.map",
"src"
],
"license": "BSD-3-Clause",
"dependencies": {
"@agentclientprotocol/sdk": "0.25.1",
"tsx": "^4.22.4",
"vitest": "^4.1.8"
},
"peerDependencies": {
"cordis": "^4.0.0-rc.6"
},
"devDependencies": {
"cordis": "^4.0.0-rc.6"
}
}

469
packages/support/acp-snapshot/src/harness.ts Normal file
View File

@@ -0,0 +1,469 @@
/**
* Shared subprocess harness for ACP snapshot suites. A library module driven by
* the suite factory in ./suite.ts (and directly by harness-level specs); each
* example's `*.snapshot.ts` names its own agent-under-test paths.
*
* It boots the REAL agent bin subprocess via the cordis Loader (so the
* export-shape bug class stays guarded — see docs/postmortem/0001), drives it
* over real ACP JSON-RPC stdio with a deterministic input script, tees raw
* stdout (for the golden + a purity check) into an SDK `ClientSideConnection`,
* and — in record mode — harvests the persisted session JSONL after a graceful
* shutdown flush. The pure normalizers in ./normalize.ts turn the captured
* stdout frames and the session-log events into stable, snapshot-able text.
*
* See docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
*
* @module @deepseek-ai/dsh-acp-snapshot/harness
*/
import { spawn, type ChildProcessWithoutNullStreams } from 'node:child_process'
import { cp, mkdtemp, readFile, readdir, rm } from 'node:fs/promises'
import { existsSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join, delimiter } from 'node:path'
import { fileURLToPath } from 'node:url'
import { Readable, Writable } from 'node:stream'
import {
ClientSideConnection,
ndJsonStream,
PROTOCOL_VERSION,
type Agent as AcpAgent,
type Client,
type RequestPermissionRequest,
type RequestPermissionResponse,
type SessionNotification,
} from '@agentclientprotocol/sdk'
// Resolve tsx's ESM loader to an ABSOLUTE path once: the child runs with its
// cwd in a temp dir OUTSIDE the repo, where a bare `--import tsx` would not
// resolve from node_modules. import.meta.resolve gives this package's tsx
// regardless of the child cwd.
const tsxLoader = fileURLToPath(import.meta.resolve('tsx'))
/**
* The agent composition a scenario runs against: which bin to boot and which
* leaf config it loads. All paths are ABSOLUTE — the subprocess cwd is a temp
* dir outside the repo, so relative resolution would miss; a suite resolves
* them from its own `import.meta.url`.
*/
export interface AgentUnderTest {
/** The agent bin entry (e.g. `packages/ui/acp-agent/src/bin.ts`), run unbuilt via tsx. */
binScript: string
/**
* The example's live `cordis.yml`. Under `DSH_SNAPSHOT=replay` the bin swaps
* it for the sibling `cordis.snapshot.yml` (the keyless replay overlay), so
* one path serves both modes.
*/
configPath: string
/**
* The repo-root tsconfig whose `paths` map resolves the unbuilt workspace
* imports. Passed to the child as `TSX_TSCONFIG_PATH`: tsx finds a tsconfig
* by searching UP from the child's cwd — a temp dir outside the repo — so
* without the explicit pin the dsh-* imports fail before the bin writes a
* byte.
*/
tsconfigPath: string
}
/**
* One step of a scenario's deterministic input script (`input.json`). The
* harness interprets these in order. `newSession` captures the server-issued
* (random) session id into a `{{sessionId}}` variable that later steps
* reference, since a committed file cannot know the id in advance.
*
* `promptAndCancel` sends a prompt WITHOUT awaiting its response, waits until
* the client observes the first streamed `agent_message_chunk` (so the emitted
* frames deterministically precede the cancellation), then cancels the turn —
* the only way to exercise a cancel deterministically (a plain `prompt` step
* awaits the response, which a cancel/hang scenario would block on forever).
*/
export type InputStep =
| { op: 'initialize'; terminalOutput?: boolean }
| { op: 'newSession' }
| { op: 'newSessionExpectError'; additionalDirectories?: string[] }
| { op: 'prompt'; text: string }
| { op: 'promptExpectError'; text: string }
| { op: 'promptAndCancel'; text: string }
| { op: 'cancel' }
/** A scenario's `input.json`: an ordered list of input steps. */
export interface InputScript {
steps: InputStep[]
/**
* Ordered answers for the agent's `session/request_permission` round-trips,
* consumed FIFO — the Nth request gets the Nth answer. Each answer selects
* by option KIND: option ids are agent-issued randoms a committed script
* cannot know, while kinds are the ACP-stable vocabulary, so the client maps
* kind → the offered `optionId` at answer time. A request beyond the queue
* (or with no queue at all) is answered `cancelled` — the stub behavior a
* scenario without approvals relies on. A scripted kind the request does
* not offer REJECTS the run: the scenario scripted an impossible click,
* and {@link runScenario} throws once the in-flight step settles (the
* agent itself just sees `cancelled`, so it cannot absorb the bug).
*/
permissionAnswers?: PermissionAnswer[]
}
/** One scripted answer to a permission request: which offered option kind to select. */
export interface PermissionAnswer {
/** The `PermissionOption.kind` to select (`allow_once`, `reject_always`, …). */
kind: 'allow_once' | 'allow_always' | 'reject_once' | 'reject_always'
}
/** One harvested session log plus the identifying facts off its header line. */
export interface HarvestedLog {
/** The recorded session id (header `id`). */
id: string
/** Session creation time (header `createdAt`) — the child-ordering key. */
createdAt: number
/** The parent session id, if this log is a subagent child (header `parentSession`). */
parentSession?: string
/** The full `.jsonl` file content. */
content: string
}
/** The result of running a scenario: raw stdout + the harvested session log(s). */
export interface RunResult {
/** Raw stdout bytes (decoded utf8), every newline-delimited JSON-RPC frame. */
rawStdout: string
/** stderr (for diagnostics on failure). */
stderr: string
/** The session id the server issued (undefined if no session was created). */
sessionId?: string
/** The temp cwd the session ran in (the bash workspace). */
cwd: string
/**
* Every persisted session log harvested after the run, ordered primary-first:
* the top-level (parent) session — the one with no `parentSession` — then each
* subagent child by ascending `createdAt`. A single-session scenario harvests
* exactly one; a nested-agent scenario harvests the parent plus one per child.
*/
sessionLogs: HarvestedLog[]
}
/** How to run one scenario: the agent to boot, the mode, and the fixture wiring. */
export interface RunOptions {
/** The agent composition to boot. */
agent: AgentUnderTest
/** `replay` (default, keyless) or `record` (real API, harvests the log). */
mode: 'replay' | 'record'
/** The recorded session JSONL fixture path (replay reads it; record writes near it). */
fixtureFile: string
/** Optional sidecar override path (replay). */
overrideFile?: string
/**
* Recorded SUBAGENT child-session fixture paths (replay). A nested-agent
* scenario ships one per child (`session.1.jsonl`, …); the harness forwards
* them to `dsh-llm-replay` via `$DSH_SNAPSHOT_CHILD_FILES` so each child
* session replays from its own recorded script. Empty for single-session
* scenarios. Ignored in record mode (children are harvested, not replayed).
*/
childFiles?: string[]
/**
* Optional `<scenario>/workspace/` directory whose contents are copied into
* the temp cwd BEFORE the run — the standard way to seed files the agent
* operates on (a file to read, edit, or grep). Absent for scenarios that
* start from an empty workspace.
*/
workspaceDir?: string
}
/**
* Run a scenario end-to-end against a freshly-spawned subprocess. Owns the
* child and its temp dirs; always tears them down. Returns the captured stdout
* and (record mode) the harvested session-log path.
*
* @param input The scenario's input script (steps + optional permission answers).
* @param opts The agent to boot, the mode, and the fixture wiring.
* @returns The captured stdout/stderr, session id, temp cwd, and harvested logs.
*/
export async function runScenario(input: InputScript, opts: RunOptions): Promise<RunResult> {
const cwd = await mkdtemp(join(tmpdir(), 'acp-snap-cwd-'))
const sessionsRoot = await mkdtemp(join(tmpdir(), 'acp-snap-sessions-'))
// Everything past the temp-dir creation runs under a try/finally that always
// removes both dirs — so a failure in workspace seeding, spawn, or any step
// never leaks them (the "e2e tests own their resources" rule).
let child: ChildProcessWithoutNullStreams | undefined
let sessionId: string | undefined
let sessionLogs: HarvestedLog[] = []
const rawBuffers: Buffer[] = []
const stderrChunks: string[] = []
try {
// Seed the workspace if the scenario ships one (a file the agent reads/edits).
// Copied into the temp cwd so the agent's bash tools see it; the goldens
// normalize the cwd, so the seeded paths stay stable across runs.
if (opts.workspaceDir !== undefined && existsSync(opts.workspaceDir)) {
await cp(opts.workspaceDir, cwd, { recursive: true })
}
const env: NodeJS.ProcessEnv = {
...process.env,
TSX_TSCONFIG_PATH: opts.agent.tsconfigPath,
DSH_SNAPSHOT: opts.mode,
DSH_SNAPSHOT_FILE: opts.fixtureFile,
DSH_SNAPSHOT_SESSIONS_ROOT: sessionsRoot,
...opts.overrideFile !== undefined ? { DSH_SNAPSHOT_OVERRIDE: opts.overrideFile } : {},
...opts.childFiles !== undefined && opts.childFiles.length > 0
? { DSH_SNAPSHOT_CHILD_FILES: opts.childFiles.join(delimiter) }
: {},
}
child = spawn(
process.execPath,
['--import', tsxLoader, opts.agent.binScript, opts.agent.configPath],
{ cwd, env, stdio: ['pipe', 'pipe', 'pipe'] },
)
child.stderr.setEncoding('utf8')
child.stderr.on('data', (c: string) => stderrChunks.push(c))
// Tee raw stdout: accumulate the bytes for the golden + purity check, and ALSO
// feed the same bytes to the SDK client through a passthrough. Buffer the raw
// bytes (not per-chunk utf8 strings) and decode once at the end, so a
// multibyte sequence split across two 'data' events can't corrupt the golden.
const passthrough = new Readable({ read() {} })
child.stdout.on('data', (buf: Buffer) => {
rawBuffers.push(buf)
passthrough.push(buf)
})
child.stdout.on('end', () => passthrough.push(null))
const stream = ndJsonStream(
Writable.toWeb(child.stdin) as WritableStream<Uint8Array>,
Readable.toWeb(passthrough) as ReadableStream<Uint8Array>,
)
// Watcher so a step can block until the client OBSERVES a particular
// session/update — used by promptAndCancel to pin frame order (send cancel
// only after the streamed agent_message_chunk has arrived, so those frames
// deterministically precede the cancelled prompt response).
const updateWaiters: { match: (u: SessionNotification['update']) => boolean; resolve: () => void }[] = []
const waitForUpdate = (match: (u: SessionNotification['update']) => boolean): Promise<void> =>
new Promise<void>(resolve => updateWaiters.push({ match, resolve }))
// Permission answers are consumed FIFO across the whole run; exhaustion
// falls back to `cancelled` so approval-free scenarios keep the plain stub.
const permissionQueue = [...input.permissionAnswers ?? []]
// A scenario bug detected inside a client callback (a scripted permission
// kind the agent never offered). It cannot fail the run from in there: a
// callback throw only becomes a JSON-RPC error RESPONSE to the agent, and
// a tolerant agent treats that as a denial and carries on — the run (or
// worse, a record) would absorb the impossible click silently. So the
// callback answers `cancelled` (a well-defined path for the agent),
// captures the error here, and the step loop fails the run on it.
let scriptError: Error | undefined
const makeClient = (_agent: AcpAgent): Client => ({
sessionUpdate(params: SessionNotification): Promise<void> {
for (let i = updateWaiters.length - 1; i >= 0; i--) {
const waiter = updateWaiters[i]
// The index is always in-bounds (i only decreases; splice removes at
// i, so lower entries stay valid); the guard satisfies
// noUncheckedIndexedAccess.
/* v8 ignore next 1 -- unreachable in-bounds guard, see above */
if (waiter === undefined) continue
if (waiter.match(params.update)) {
updateWaiters.splice(i, 1)
waiter.resolve()
}
}
return Promise.resolve()
},
requestPermission(params: RequestPermissionRequest): Promise<RequestPermissionResponse> {
const answer = permissionQueue.shift()
if (answer === undefined) return Promise.resolve({ outcome: { outcome: 'cancelled' } })
const option = params.options.find(o => o.kind === answer.kind)
if (option === undefined) {
// The scenario scripted a click the agent never offered — a scenario
// bug. Captured (last one wins; same bug class either way) and
// answered `cancelled`; the step loop rejects the run on it.
scriptError = new Error(
`snapshot-harness: scripted permission answer ${answer.kind} not among `
+ `the offered options [${params.options.map(o => o.kind).join(', ')}]`,
)
return Promise.resolve({ outcome: { outcome: 'cancelled' } })
}
return Promise.resolve({ outcome: { outcome: 'selected', optionId: option.optionId } })
},
})
const client = new ClientSideConnection(makeClient, stream)
for (const step of input.steps) {
await runStep(client, step, cwd, waitForUpdate, () => sessionId, (id) => { sessionId = id })
// A permission exchange happens while a step's request is in flight, so
// by the time the step settles any script bug it exposed is captured —
// fail the run HERE, as a harness error, rather than hoping the agent's
// reaction to the answer perturbs the transcript.
if (scriptError !== undefined) throw scriptError
}
// Done driving: close stdin so the server disposes gracefully (flushing
// persistence) and exits. Then await exit so the harvested log is complete.
child.stdin.end()
await waitForExit(child)
// Harvest EVERY persisted log (parent + any subagent children) while the
// temp dirs still exist, ordered primary-first.
sessionLogs = await harvestSessionLogs(sessionsRoot)
} finally {
// Failure-safe teardown: kill a still-running child and drop the temp dirs
// even if seeding/spawn/a step/harvest threw, so a flaky run never leaks a
// process or dir. `child` is undefined only if spawn itself threw.
if (child !== undefined && child.exitCode === null && child.signalCode === null) {
child.kill('SIGKILL')
await waitForExit(child)
}
await rm(cwd, { recursive: true, force: true })
await rm(sessionsRoot, { recursive: true, force: true })
}
return {
rawStdout: Buffer.concat(rawBuffers).toString('utf8'),
stderr: stderrChunks.join(''),
cwd,
...sessionId !== undefined ? { sessionId } : {},
sessionLogs,
}
}
/** Drive one input step over the client connection. */
async function runStep(
client: ClientSideConnection,
step: InputStep,
cwd: string,
waitForUpdate: (match: (u: SessionNotification['update']) => boolean) => Promise<void>,
getSessionId: () => string | undefined,
setSessionId: (id: string) => void,
): Promise<void> {
switch (step.op) {
case 'initialize':
await client.initialize({
protocolVersion: PROTOCOL_VERSION,
clientCapabilities: step.terminalOutput === true ? { _meta: { terminal_output: true } } : {},
})
return
case 'newSession': {
const { sessionId } = await client.newSession({ cwd, mcpServers: [] })
setSessionId(sessionId)
return
}
case 'newSessionExpectError': {
// The bridge rejects a session/new that widens the workspace scope
// (non-empty additionalDirectories / mcpServers — unimplemented). The SDK
// surfaces that as a rejected RPC; swallow it so the run completes and the
// error frame is captured in the transcript.
await client.newSession({
cwd,
mcpServers: [],
...step.additionalDirectories !== undefined ? { additionalDirectories: step.additionalDirectories } : {},
}).then(
() => { throw new Error('snapshot-harness: expected session/new to be rejected but it succeeded') },
() => { /* expected: the bridge rejected the unsupported workspace scope */ },
)
return
}
case 'prompt': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: prompt before newSession')
await client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
return
}
case 'promptExpectError': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: promptExpectError before newSession')
// The model fails this turn (a recorded provider error), so the bridge
// answers the prompt with a JSON-RPC error and the SDK rejects. That
// rejection IS the expected editor experience — swallow it so the run
// completes and the stdout transcript (the error frame) is captured.
await client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
.then(() => { throw new Error('snapshot-harness: expected the prompt to fail but it succeeded') },
() => { /* expected: the turn failed and the bridge returned an error */ })
return
}
case 'promptAndCancel': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: promptAndCancel before newSession')
// Dispatch the prompt WITHOUT awaiting (a hang fixture never resolves on
// its own). To pin frame order deterministically, wait until the client
// has OBSERVED the hang's streamed agent_message_chunk before cancelling —
// so those update frames always precede the cancelled prompt response in
// the transcript (without this, the late chunk and the response race).
// Then cancel and await the prompt, which the bridge settles as
// `cancelled` once the abort propagates.
const promptDone = client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
await waitForUpdate(u => u.sessionUpdate === 'agent_message_chunk')
await client.cancel({ sessionId })
await promptDone
return
}
case 'cancel': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: cancel before newSession')
await client.cancel({ sessionId })
return
}
default:
throw new Error(`snapshot-harness: unknown input op ${JSON.stringify(step)}`)
}
}
/** Resolve once the child process exits (any code/signal). */
function waitForExit(child: ChildProcessWithoutNullStreams): Promise<void> {
// Race guard: both call sites run within one synchronous frame of
// stdin.end()/kill(), so the exit event cannot have been delivered yet;
// kept for any future caller that awaits in between.
/* v8 ignore next 1 -- unreachable race guard, see above */
if (child.exitCode !== null || child.signalCode !== null) return Promise.resolve()
return new Promise<void>(resolve => child.once('exit', () => { resolve() }))
}
/**
* Harvest EVERY persisted `.jsonl` session log under a sessions root, parse each
* header line, and return them ordered primary-first: the top-level session (no
* `parentSession`) leads, then each subagent child by ascending `createdAt`.
*
* The JSONL backend lays sessions out as `<root>/<cwd-bucket>/<encoded-id>.jsonl`
* (one bucket per cwd), so a parent and its same-cwd in-process child land in
* the SAME bucket — collecting all files across all buckets catches both (a
* first-match short-circuit would silently drop the child). Returns `[]` if no
* log was produced (a no-session scenario).
*/
async function harvestSessionLogs(root: string): Promise<HarvestedLog[]> {
let cwdDirs: string[]
try {
cwdDirs = await readdir(root)
} catch {
return []
}
const logs: HarvestedLog[] = []
for (const dir of cwdDirs) {
const sub = join(root, dir)
let files: string[]
try {
files = await readdir(sub)
} catch {
continue
}
for (const f of files) {
if (!f.endsWith('.jsonl')) continue
const content = await readFile(join(sub, f), 'utf8')
const firstLine = content.split('\n').find(line => line.trim().length > 0) ?? '{}'
const header = JSON.parse(firstLine) as { id?: unknown; createdAt?: unknown; parentSession?: unknown }
logs.push({
id: typeof header.id === 'string' ? header.id : '',
createdAt: typeof header.createdAt === 'number' ? header.createdAt : 0,
...typeof header.parentSession === 'string' ? { parentSession: header.parentSession } : {},
content,
})
}
}
// Primary (no parentSession) first, then children by ascending createdAt. A
// scenario has exactly one top-level session. In the synchronous cut sibling
// children are created strictly sequentially, so their createdAt values are
// strictly ordered; the recordedId tiebreak only keeps a degenerate
// same-millisecond collision (unreachable here) deterministic. This harvest
// order must match the replay load order in dsh-llm-replay's loadSessionScripts
// so session.<n>.jsonl maps to the same child on record and replay — replay
// re-sorts childFiles by the same key, so the two stay consistent.
logs.sort((a, b) => {
const ap = a.parentSession === undefined ? 0 : 1
const bp = b.parentSession === undefined ? 0 : 1
return ap - bp || a.createdAt - b.createdAt || a.id.localeCompare(b.id)
})
return logs
}

38
packages/support/acp-snapshot/src/index.ts Normal file
View File

@@ -0,0 +1,38 @@
/**
* ACP snapshot suite kit — the shared machinery behind the keyless snapshot
* tier (`pnpm run test:snapshot`). Three layers, composable per example:
* the subprocess scenario harness ({@link runScenario}), the pure golden
* normalizers ({@link normalizeStdout} / {@link normalizeSessionLog} /
* {@link scrubRequestHeaders}), and the suite factory
* ({@link defineAcpSnapshotSuite}) that registers a scenario table as a full
* describe/it tree. An example's `*.snapshot.ts` supplies only its
* {@link AgentUnderTest} paths, its snapshots directory, and its
* {@link Scenario} table.
*
* NOTE: ./suite.ts imports vitest, so this package is importable only inside a
* vitest run — a support-tier constraint stated in the README.
*
* @module @deepseek-ai/dsh-acp-snapshot
*/
export {
runScenario,
type AgentUnderTest,
type HarvestedLog,
type InputScript,
type InputStep,
type PermissionAnswer,
type RunOptions,
type RunResult,
} from './harness.ts'
export {
normalizeSessionLog,
normalizeStdout,
scrubRequestHeaders,
type NormalizeContext,
} from './normalize.ts'
export {
defineAcpSnapshotSuite,
type Scenario,
type SnapshotSuiteOptions,
} from './suite.ts'

199
packages/support/acp-snapshot/src/normalize.ts Normal file
View File

@@ -0,0 +1,199 @@
/**
* Pure normalizers for the ACP snapshot goldens. They replace the
* non-deterministic values in the two captured surfaces — the stdout JSON-RPC
* transcript and the persisted session JSONL — with stable tokens, so a golden
* compare reflects behavior, not run-to-run noise. Kept dependency-free and
* side-effect-free so they unit-test trivially.
*
* Scrubbed: `randomUUID()` session ids → `{{sessionId}}`; the temp `mkdtemp`
* cwd → `{{cwd}}` (it appears in terminal-card `_meta` and the log header);
* JSON-RPC request `id` → a stable per-transcript sequence; the log's per-event
* `time` (epoch ms) and header `createdAt` → 0; a `hook/result` event's
* `durationMs` (wall-clock hook runtime) → 0. NOT scrubbed: the log's `seq`
* (deterministic — `seq = log.length`, part of the event-log contract).
*
* A separate, composable normalizer — {@link scrubRequestHeaders} — replaces
* the bulky request-header CONTENT (the composed system prompt and the tool
* schema list) with `{{system}}`/`{{tools}}` tokens. It is deliberately NOT
* folded into {@link normalizeSessionLog}: each suite's one header-pinning
* scenario compares that content verbatim, every other scenario composes the
* scrub in (the `pinsHeader` flag on the scenario table, consumed by the suite
* factory in ./suite.ts; see the pinned-header RFC,
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
*
* See docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
*
* @module @deepseek-ai/dsh-acp-snapshot/normalize
*/
const SESSION_ID = '{{sessionId}}'
const CWD = '{{cwd}}'
const SYSTEM = '{{system}}'
const TOOLS = '{{tools}}'
/** A UUID v4 string, the shape `randomUUID()` produces for session ids. */
const UUID_RE = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi
/** Inputs the normalizers need to recognize a run's volatile values. */
export interface NormalizeContext {
/** The session id(s) the run issued — replaced with `{{sessionId}}`. */
sessionIds: string[]
/** The temp cwd the run used — replaced with `{{cwd}}`. */
cwd: string
}
/** Replace cwd, session ids, and any stray UUID with stable tokens in a string. */
function scrubString(value: string, ctx: NormalizeContext): string {
let out = value
// cwd first (longest, most specific), then explicit session ids, then any
// residual UUID (covers ids that appear in places we didn't enumerate).
out = out.split(ctx.cwd).join(CWD)
for (const id of ctx.sessionIds) out = out.split(id).join(SESSION_ID)
out = out.replace(UUID_RE, SESSION_ID)
return out
}
/** Recursively scrub a parsed JSON value (strings replaced; structure kept). */
function scrubValue(value: unknown, ctx: NormalizeContext): unknown {
if (typeof value === 'string') return scrubString(value, ctx)
if (Array.isArray(value)) return value.map(v => scrubValue(v, ctx))
if (value !== null && typeof value === 'object') {
const out: Record<string, unknown> = {}
for (const [k, v] of Object.entries(value)) out[k] = scrubValue(v, ctx)
return out
}
return value
}
/**
* Normalize a raw stdout transcript (newline-delimited JSON-RPC frames) into a
* stable golden in the SAME shape as the wire: one compact JSON frame per line
* (NDJSON), with the JSON-RPC `id` rewritten to a per-transcript sequence
* (1, 2, 3, …) and all volatile strings scrubbed. Throws if any non-empty line
* is not valid JSON — that doubles as the stdout-purity check (no logger leaked
* onto the protocol).
*
* @param rawStdout The captured stdout bytes, decoded utf8.
* @param ctx The run's volatile values to scrub.
* @returns The normalized NDJSON transcript, one frame per line.
*/
export function normalizeStdout(rawStdout: string, ctx: NormalizeContext): string {
const lines = rawStdout.split('\n').filter(line => line.trim().length > 0)
// Map each distinct JSON-RPC id (request/response correlate by id) to a stable
// sequence number, in first-seen order, so id churn doesn't perturb the golden.
const idSeq = new Map<string, number>()
const stableId = (id: unknown): number => {
const key = JSON.stringify(id)
let n = idSeq.get(key)
if (n === undefined) { n = idSeq.size + 1; idSeq.set(key, n) }
return n
}
const frames = lines.map((line) => {
const frame = JSON.parse(line) as Record<string, unknown>
if ('id' in frame && frame.id !== undefined && frame.id !== null) {
frame.id = stableId(frame.id)
}
return scrubValue(frame, ctx) as Record<string, unknown>
})
return frames.map(f => JSON.stringify(f)).join('\n') + '\n'
}
/**
* Normalize a session JSONL log into a stable golden: the header line's
* volatile fields (`createdAt`, `id`, `cwd`) and every event's `time` are
* zeroed/scrubbed, all volatile strings scrubbed, and `seq` is LEFT INTACT
* (deterministic by contract). Output is JSONL in the same shape as the input —
* one compact record per line.
*
* @param rawLog The raw session `.jsonl` content.
* @param ctx The run's volatile values to scrub.
* @returns The normalized JSONL log, one record per line.
*/
export function normalizeSessionLog(rawLog: string, ctx: NormalizeContext): string {
const lines = rawLog.split('\n').filter(line => line.trim().length > 0)
const records = lines.map((line) => {
const record = JSON.parse(line) as Record<string, unknown>
// Header line: { type: 'session', createdAt, id, cwd, … }.
if (record.type === 'session') {
if ('createdAt' in record) record.createdAt = 0
} else if ('time' in record) {
// Event line: zero the epoch-ms timestamp; keep seq (deterministic).
record.time = 0
// A hook/result carries the hook's wall-clock runtime (`data.durationMs`),
// which is run-to-run noise like `time` — zero it so the golden reflects
// the hook's decision/exit, not how long the shell took.
if (record.type === 'hook/result' && record.data !== null && typeof record.data === 'object') {
const data = record.data as Record<string, unknown>
if ('durationMs' in data) data.durationMs = 0
}
}
return scrubValue(record, ctx) as Record<string, unknown>
})
return records.map(r => JSON.stringify(r)).join('\n') + '\n'
}
/**
* Replace request-header CONTENT in a session JSONL with stable tokens,
* keeping its structure: a `request/header` event's `data.header.system` →
* `{{system}}` and `data.header.tools` → `{{tools}}`; a
* `request/header-delta` event keeps every structural fact — the system
* delta's `keepStart`/`keepEnd` line positions and inserted-line COUNT (one
* `{{system}}` token per inserted line), the tools delta's
* added/removed/changed tool NAMES — and tokenizes only the bulk (prompt
* text; each added/changed schema's fields other than `name` → `{{tools}}`),
* so two different deltas still compare different.
* Absent fields stay absent — WHETHER a header carried a system prompt or
* tools is behavior and stays visible; `config` and `reason` are small and
* stable, so they stay verbatim (a model swap churns every fixture by design
* — it invalidates the recorded responses; a prompt/schema edit churns none —
* replay never reads this content, see dsh-llm-replay).
*
* Only lines with something to scrub are re-serialized; every other line
* passes through byte-for-byte, so the transform is idempotent and applying
* it to an already-scrubbed fixture is a no-op — the on-disk-fixtures guard
* in ./suite.ts relies on exactly that.
*
* @param rawLog The raw session `.jsonl` content.
* @returns The JSONL with header content tokenized, other lines byte-identical.
*/
export function scrubRequestHeaders(rawLog: string): string {
const lines = rawLog.split('\n')
const out = lines.map((line) => {
if (line.trim().length === 0) return line
const record = JSON.parse(line) as Record<string, unknown>
const data = record.data as Record<string, unknown> | null | undefined
if (data === null || typeof data !== 'object') return line
if (record.type === 'request/header') {
const header = data.header as Record<string, unknown> | null | undefined
if (header === null || typeof header !== 'object') return line
if (!('system' in header) && !('tools' in header)) return line
if ('system' in header) header.system = SYSTEM
if ('tools' in header) header.tools = TOOLS
return JSON.stringify(record)
}
if (record.type === 'request/header-delta') {
let touched = false
const system = data.system as Record<string, unknown> | null | undefined
if (system !== null && typeof system === 'object' && Array.isArray(system.insert)) {
system.insert = system.insert.map(() => SYSTEM)
touched = true
}
const tools = data.tools as Record<string, unknown> | null | undefined
if (tools !== null && typeof tools === 'object') {
if (Array.isArray(tools.added)) { tools.added = tools.added.map(scrubToolSchema); touched = true }
if (Array.isArray(tools.changed)) { tools.changed = tools.changed.map(scrubToolSchema); touched = true }
}
return touched ? JSON.stringify(record) : line
}
return line
})
return out.join('\n')
}
/** Tokenize one tool schema's bulk (description, parameters, anything else), keeping its identifying `name`. */
function scrubToolSchema(tool: unknown): unknown {
if (tool === null || typeof tool !== 'object' || Array.isArray(tool)) return tool
const out: Record<string, unknown> = {}
for (const [k, v] of Object.entries(tool)) out[k] = k === 'name' ? v : TOOLS
return out
}

384
packages/support/acp-snapshot/src/suite.ts Normal file
View File

@@ -0,0 +1,384 @@
/**
* The ACP snapshot suite factory (REPLAY by default, keyless). A suite is a
* scenario table plus a snapshots directory: each scenario under
* `<snapshotsDir>/<name>/` ships an `input.json` (the client stdin script) and
* a `session.jsonl` fixture; replay boots the real agent subprocess
* (./harness.ts), drives it, and diffs the normalized stdout transcript
* against the committed `stdout.golden.jsonl`. For model scenarios it ALSO
* checks the re-persisted session log — against the `session.jsonl` fixture
* itself, not a separate golden: the fixture doubles as the replay source
* (recorded scenarios) and the expected produced log (both sides normalized
* before comparing).
*
* Request-header content (the composed system prompt + tool schemas riding on
* `request/header` events) is pinned by exactly ONE scenario per suite — the
* one with `pinsHeader` — and scrubbed to `{{system}}`/`{{tools}}` tokens in
* every other fixture and compare, so a prompt or tool-schema edit churns one
* committed line instead of every fixture. A per-run uniformity guard keeps
* the single pin sound: every live header must equal the pinned one, and no
* header-delta may appear outside the pinning scenario (see the
* pinned-header RFC,
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
*
* `pnpm run test:snapshot:record` (DSH_SNAPSHOT=record + -u) re-records the
* `session.jsonl` fixtures against the real API and refreshes the stdout golden
* in one pass; the caller resolves that env into {@link SnapshotSuiteOptions}
* (env reading stays at the suite edge, not in this library).
*
* @module @deepseek-ai/dsh-acp-snapshot/suite
*/
import { readFile, readdir, writeFile } from 'node:fs/promises'
import { existsSync } from 'node:fs'
import { join } from 'node:path'
import { describe, expect, it } from 'vitest'
import { type AgentUnderTest, type HarvestedLog, type InputScript, runScenario } from './harness.ts'
import { type NormalizeContext, normalizeSessionLog, normalizeStdout, scrubRequestHeaders } from './normalize.ts'
/** A snapshot scenario and how its fixtures are produced. */
export interface Scenario {
name: string
/** Whether the scenario drives at least one model turn (so a JSONL golden applies). */
hasModelTurn: boolean
/**
* Whether the run persists a comparable session log to diff against the
* `session.jsonl` fixture. Defaults to {@link hasModelTurn} (a model turn
* always produces a log worth comparing). Set it independently for a scenario
* that produces a non-trivial log WITHOUT a model turn — e.g. a prompt blocked
* by a `UserPromptSubmit` hook, which opens a `rejected` turn carrying `hook/*`
* events but never calls the model.
*/
comparesLog?: boolean
/**
* Whether `test:snapshot:record` regenerates this scenario's `session.jsonl`
* from the LIVE API. `recorded` scenarios are model-driven and reproducible;
* `authored` scenarios (fixtures hand-written or hand-harvested — e.g. a
* provider error or a cancel the live API can't be coaxed into
* deterministically, a deterministic hook scenario, or a scripted repetition
* a live model won't reproduce) are NEVER re-recorded.
*/
recorded: boolean
/**
* Whether replay is driven by a hand-written `replay.override.json` sidecar
* (a `ReplayEntry[]` that REPLACES the script derived from `session.jsonl`)
* — the throw/hang cases chunks cannot express. The fixture guard requires
* the sidecar exactly when this is set: the harness forwards the file purely
* on existence, so an unregistered stray sidecar would silently replace the
* derived script — the guard fails loud on either mismatch. Defaults to
* false (replay derives from the fixture's `assistant/chunk` events).
*/
overridden?: boolean
/**
* How many SUBAGENT child sessions this scenario records beyond the top-level
* one (0 for a single-session scenario). Each child rides in a sibling fixture
* `session.<n>.jsonl` (1-based); replay forwards them to `dsh-llm-replay` so
* each child session replays from its own script, and record mode writes the
* harvested child logs back to those files. Defaults to 0.
*/
childSessions?: number
/**
* Whether THIS scenario's fixtures keep the full request-header content (the
* composed system prompt and tool schema list on `request/header` /
* `request/header-delta` events) and compare it verbatim. Exactly one
* scenario per suite pins it; every other scenario stores and compares that
* content as `{{system}}`/`{{tools}}` tokens ({@link scrubRequestHeaders}),
* so a system prompt or tool-schema change shows up as ONE committed-fixture
* diff, not one per scenario. One pin suffices because header composition is
* suite-uniform (parent, spawn child, and fork child all compose the same
* prompt-modulo-cwd and the same tools) — and that premise is ASSERTED, not
* assumed: every non-pinning run's live headers must equal the pinned
* fixture's (normalized), so a session-dependent header (say, a restricted
* subagent toolset) fails loud until it gets its own pinning scenario.
* Defaults to false.
*/
pinsHeader?: boolean
}
/** One suite's inputs: the agent to boot, where its fixtures live, and its scenario table. */
export interface SnapshotSuiteOptions {
/** The agent composition every scenario boots. */
agent: AgentUnderTest
/** Absolute path of the suite's `snapshots/` directory (one subdir per scenario). */
snapshotsDir: string
/** The scenario table; exactly one entry must set `pinsHeader`. */
scenarios: Scenario[]
/**
* `replay` (keyless, the default tier) or `record` (live API; re-records the
* `recorded` scenarios' fixtures and refreshes the vitest goldens under
* `--update`). The caller derives this from `$DSH_SNAPSHOT` — env reading
* stays outside this library.
*/
mode: 'replay' | 'record'
}
/**
* The sibling child-fixture paths for a scenario (`session.1.jsonl` …).
*
* @param dir The scenario's snapshots directory (`<snapshotsDir>/<name>`).
* @param childSessions How many subagent child sessions the scenario records.
* @returns One path per child, 1-based, in fixture order.
*/
export function childFixturePaths(dir: string, childSessions: number): string[] {
return Array.from({ length: childSessions }, (_, i) => join(dir, `session.${i + 1}.jsonl`))
}
/**
* Derive the {@link NormalizeContext} for a `session.jsonl` fixture from its own
* header line (`{ type: 'session', id, cwd }`). A committed fixture carries the
* session id and cwd of the run that harvested it — different from the live
* replay run — so normalizing it against the live run's ctx would leave those
* recorded values unscrubbed. Reading them from the header scrubs the fixture's
* own id/cwd to the same `{{sessionId}}`/`{{cwd}}` tokens the replay output gets.
* An authored fixture whose header is already normalized (`id:'{{sessionId}}'`,
* `cwd:'{{cwd}}'`) yields those tokens as the volatile values, so scrubbing them
* is an idempotent no-op. A header with no `cwd` falls back to a sentinel that
* cannot occur in a log (NOT `''`, which `String.split` would match on every
* character boundary and corrupt the output).
*
* @param fixture The committed `session.jsonl` content.
* @returns The fixture's own volatile values, ready for {@link normalizeSessionLog}.
*/
export function fixtureContext(fixture: string): NormalizeContext {
const firstLine = fixture.split('\n').find(line => line.trim().length > 0) ?? '{}'
const header = JSON.parse(firstLine) as { id?: unknown; cwd?: unknown }
return {
sessionIds: typeof header.id === 'string' ? [header.id] : [],
cwd: typeof header.cwd === 'string' ? header.cwd : '\0no-cwd\0',
}
}
/**
* The `data.header` payload of every `request/header` event in a session
* JSONL, in log order, with the log's volatile values scrubbed first
* ({@link normalizeSessionLog}) so headers harvested from different runs —
* each embedding its own temp cwd in the composed prompt — compare on equal
* footing.
*
* @param rawLog The session `.jsonl` content to extract headers from.
* @param ctx The volatile values of the run that produced it.
* @returns The normalized `data.header` payloads, in log order.
*/
export function normalizedHeaders(rawLog: string, ctx: NormalizeContext): unknown[] {
return normalizeSessionLog(rawLog, ctx)
.split('\n')
.filter(line => line.trim().length > 0)
.map(line => JSON.parse(line) as { type?: unknown; data?: { header?: unknown } })
.filter(record => record.type === 'request/header')
.map(record => record.data?.header)
}
/**
* Count the `request/header-delta` events in a session JSONL.
*
* @param rawLog The session `.jsonl` content.
* @returns How many `request/header-delta` events the log carries.
*/
export function headerDeltaCount(rawLog: string): number {
return rawLog.split('\n')
.filter(line => line.trim().length > 0)
.filter(line => (JSON.parse(line) as { type?: unknown }).type === 'request/header-delta')
.length
}
/**
* Register the suite: one `describe` per scenario (the golden/log compares and
* the header-uniformity guard) plus the fixture guard block (no orphan
* scenario dirs, required files present, exactly one pin, non-pinning fixtures
* header-scrubbed). Must run at vitest collection time — it calls
* `describe`/`it`. Throws immediately if no scenario pins the header (the
* uniformity guard would have nothing to compare against).
*
* @param options The agent, snapshots directory, scenario table, and mode.
*/
export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
const { agent, snapshotsDir, scenarios, mode } = options
const RECORDING = mode === 'record'
/** The suite's single header-pinning scenario. Guarded here (and by a meta-test) so the pin cannot silently vanish. */
const pinningScenario = scenarios.find(s => s.pinsHeader === true)
if (pinningScenario === undefined) throw new Error('acp-snapshot: no scenario pins the request-header content')
for (const scenario of scenarios) {
describe(`snapshot: ${scenario.name}`, () => {
// In RECORD mode, only re-run the `recorded` (live-API) scenarios; the
// `authored` ones (sidecar-driven errors/cancel) are never re-recorded.
it.skipIf(RECORDING && !scenario.recorded)('matches the goldens', async () => {
const dir = join(snapshotsDir, scenario.name)
const input = JSON.parse(await readFile(join(dir, 'input.json'), 'utf8')) as InputScript
const overrideFile = join(dir, 'replay.override.json')
const workspaceDir = join(dir, 'workspace')
const childSessions = scenario.childSessions ?? 0
const result = await runScenario(input, {
agent,
mode,
fixtureFile: join(dir, 'session.jsonl'),
...existsSync(overrideFile) ? { overrideFile } : {},
// In REPLAY, forward the recorded child fixtures so each subagent session
// replays from its own script. In RECORD they are harvested, not read.
...!RECORDING && childSessions > 0 ? { childFiles: childFixturePaths(dir, childSessions) } : {},
...existsSync(workspaceDir) ? { workspaceDir } : {},
})
// Scrub every volatile id the run produced: the ACP server-issued session
// id plus every harvested log's recorded id (a subagent child id never
// surfaces over ACP, but it appears in the child's own log header). The
// normalizer's UUID catch-all covers any we don't enumerate.
const ctx: NormalizeContext = {
sessionIds: [
...result.sessionId !== undefined ? [result.sessionId] : [],
...result.sessionLogs.map(l => l.id),
],
cwd: result.cwd,
}
// RECORD mode (recorded model scenarios only): persist the freshly-harvested
// logs back to their fixtures — the primary to session.jsonl, each child to
// session.<n>.jsonl in harvest order. `--update` refreshes the Vitest
// goldens but NOT these fixtures, so write them here. A non-pinning
// scenario's fixtures are written header-scrubbed, so a re-record can
// never smuggle the full prompt/schema content back into every fixture.
const scrub = scenario.pinsHeader === true
? (log: string): string => log
: scrubRequestHeaders
if (RECORDING && scenario.recorded && scenario.hasModelTurn) {
expect(result.sessionLogs.length, 'record produced no session log to harvest').toBeGreaterThan(0)
expect(result.sessionLogs.length, `expected ${childSessions + 1} session logs (parent + children)`)
.toBe(childSessions + 1)
await writeFile(join(dir, 'session.jsonl'), scrub((result.sessionLogs[0] as HarvestedLog).content))
for (let i = 1; i < result.sessionLogs.length; i++) {
await writeFile(join(dir, `session.${i}.jsonl`), scrub((result.sessionLogs[i] as HarvestedLog).content))
}
}
await expect(normalizeStdout(result.rawStdout, ctx))
.toMatchFileSnapshot(join(dir, 'stdout.golden.jsonl'))
// A model turn always produces a log worth comparing; a hook scenario can
// produce one without a model turn (a `rejected` turn carrying `hook/*`).
const comparesLog = scenario.comparesLog ?? scenario.hasModelTurn
if (comparesLog) {
// The harvested logs (primary-first) must match their committed fixtures
// 1:1. Each side passes through normalizeSessionLog, scrubbed against ITS
// OWN volatile values — the live run's via `ctx`, the committed fixture's
// via its own header (a committed file cannot share the live run's ids).
// Unless this scenario pins the header, both sides ALSO pass through
// scrubRequestHeaders: the live log carries the real prompt/schemas, the
// fixture carries the `{{system}}`/`{{tools}}` tokens, and the scrub is
// idempotent — so the compare checks the header's presence, position,
// reason, and config, but not its bulk content (pinned once, in the
// `pinsHeader` scenario).
expect(result.sessionLogs.length, 'this scenario must persist a session log').toBe(childSessions + 1)
const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)]
for (let i = 0; i < fixtureFiles.length; i++) {
const harvested = scrub((result.sessionLogs[i] as HarvestedLog).content)
const fixture = scrub(await readFile(join(dir, fixtureFiles[i] as string), 'utf8'))
expect(normalizeSessionLog(harvested, ctx), `${fixtureFiles[i]} mismatch`)
.toEqual(normalizeSessionLog(fixture, fixtureContext(fixture)))
}
}
// Header-uniformity guard: the single pin is sound only while every
// session in the suite composes the SAME header and keeps it for the
// whole run. Assert both halves live. (1) Every request/header the run
// produced (parent, spawn child, fork child, initial or resume) must
// equal the pinned fixture's header after each side is normalized
// against its own volatile values. (2) No request/header-delta may
// appear at all — a mid-run header change diverges from the pin by
// construction, and its content would be invisible under the scrub. If
// either fails, either the header changed (update the pin: re-record or
// hand-edit the pinning scenario's fixture) or composition became
// session-dependent by design (give the divergent shape its own
// pinning scenario).
if (scenario.pinsHeader !== true) {
const pinnedFixture = await readFile(join(snapshotsDir, pinningScenario.name, 'session.jsonl'), 'utf8')
const pinned = normalizedHeaders(pinnedFixture, fixtureContext(pinnedFixture))
expect(pinned.length, `the pinning fixture (${pinningScenario.name}) must carry exactly one request/header`)
.toBe(1)
for (const log of result.sessionLogs) {
expect(headerDeltaCount(log.content), `session ${log.id}: a request/header-delta in a non-pinning scenario`)
.toBe(0)
const headers = normalizedHeaders(log.content, ctx)
for (const [k, header] of headers.entries()) {
expect(header, `session ${log.id}: request/header #${k + 1} diverged from the pinned (${pinningScenario.name}) header`)
.toEqual(pinned[0])
}
}
}
})
})
}
describe('snapshot fixtures', () => {
it('every scenario directory is registered (no orphans)', async () => {
// toMatchFileSnapshot does not prune orphaned golden/fixture files, so a
// renamed/removed scenario could leave a stale dir that nothing exercises.
// Fail loud on any snapshots/<dir> not present in the scenario table.
const entries = await readdir(snapshotsDir, { withFileTypes: true })
const onDisk = entries.filter(e => e.isDirectory()).map(e => e.name).sort()
const registered = scenarios.map(s => s.name).sort()
expect(onDisk).toEqual(registered)
})
it('every registered scenario has its required fixture files', () => {
// Every scenario has an input script and an stdout golden. EVERY scenario
// also needs `session.jsonl`: the suite boots `llm-replay` with that path
// as the replay source for ALL scenarios (the factory passes
// `fixtureFile: <dir>/session.jsonl` unconditionally), and `loadReplayScript`
// throws "fixture not found" when it is absent and no override replaces it.
// A no-model scenario ships a header-only `session.jsonl` (it derives to an
// empty script — no model call is made); a model scenario's fixture also
// doubles as the expected-log artifact the run is diffed against. The
// `replay.override.json` sidecar is matched BOTH ways against the table's
// `overridden` flag: required when set, forbidden when not — the harness
// forwards the file purely on existence, so an unregistered stray sidecar
// would silently replace the derived script.
for (const { name, overridden, childSessions } of scenarios) {
const dir = join(snapshotsDir, name)
expect(existsSync(join(dir, 'input.json')), `${name}/input.json`).toBe(true)
expect(existsSync(join(dir, 'stdout.golden.jsonl')), `${name}/stdout.golden.jsonl`).toBe(true)
expect(existsSync(join(dir, 'session.jsonl')), `${name}/session.jsonl`).toBe(true)
expect(existsSync(join(dir, 'replay.override.json')), `${name}/replay.override.json presence must match \`overridden\``)
.toBe(overridden === true)
// A nested-agent scenario ships one child fixture per recorded subagent
// session (`session.1.jsonl` …), the replay source for that child session.
for (const childFixture of childFixturePaths(dir, childSessions ?? 0)) {
expect(existsSync(childFixture), childFixture).toBe(true)
}
}
})
it('exactly one scenario pins the request-header content', () => {
// Zero pins would drop the prompt/schema surface from the suite entirely;
// two would split it. One pin per suite is the design (pinned-header RFC);
// WHICH scenario pins is the scenario table's reviewable choice.
expect(scenarios.filter(s => s.pinsHeader === true).map(s => s.name)).toEqual([pinningScenario.name])
})
it('committed fixtures carry request-header content ONLY in the pinning scenario', async () => {
// The whole point of the pin: a system-prompt or tool-schema change must
// churn exactly one committed line. A non-pinning fixture that carries the
// full header (a hand-recorded file, or a header line hand-edited out of
// its canonical JSON form) silently reopens the suite-wide churn, so fail
// loud here: every non-pinning session*.jsonl must be a fixed point of
// scrubRequestHeaders (apply the scrub to fix a violation), and the
// pinning scenario's fixtures must NOT be (their content IS the pin).
for (const scenario of scenarios) {
const dir = join(snapshotsDir, scenario.name)
const files = [
'session.jsonl',
...Array.from({ length: scenario.childSessions ?? 0 }, (_, i) => `session.${i + 1}.jsonl`),
]
for (const file of files) {
const fixture = await readFile(join(dir, file), 'utf8')
if (scenario.pinsHeader === true) {
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} must PIN the full header content`)
.not.toEqual(fixture)
} else {
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} carries unscrubbed header content`)
.toEqual(fixture)
}
}
}
})
})
}

232
packages/support/acp-snapshot/tests/fixtures/fake-acp-agent.ts vendored Normal file
View File

@@ -0,0 +1,232 @@
/**
* Scripted fake ACP agent bin for `dsh-acp-snapshot`'s unit specs. Speaks
* newline-delimited JSON-RPC on stdio like the real `dsh-acp-agent` bin, but
* every behavior — how prompts settle, whether session/new rejects, which
* session logs get persisted, what filesystem noise to leave — comes from a
* `behavior.json` sitting NEXT to the `$DSH_SNAPSHOT_FILE` fixture, so a spec
* scripts a whole subprocess run from data. The specs launch it through the
* REAL `runScenario` spawn path (tsx loader, temp cwd, env plumbing), so the
* harness plumbing is exercised for real; only the agent behind the protocol
* is scripted.
*
* The specs (not the golden tier) own this bin: it asserts nothing, echoes
* observable facts into `session/update` text chunks (env probe, permission
* outcome, seeded-workspace listing) for the spec to read off `rawStdout`, and
* exits 0 on stdin EOF after writing the scripted logs — mirroring the real
* bin's dispose-flush-exit shape.
*/
import { mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
import { readdirSync } from 'node:fs'
import { dirname, join } from 'node:path'
import { randomUUID } from 'node:crypto'
import { createInterface } from 'node:readline'
/** One scripted session log: a file path under the sessions root plus its JSONL lines. */
interface ScriptedLog {
/** Path relative to `$DSH_SNAPSHOT_SESSIONS_ROOT`, e.g. `bucket/a.jsonl` (an empty dir segment is invalid). */
file: string
/**
* The JSONL records. String templates `{{CWD}}` and `{{SID}}` are replaced
* with the run's real cwd and the ACP session id this bin issued, so a
* written log carries genuine volatile values for the normalizers to scrub.
*/
lines: unknown[]
}
/** The whole scripted behavior for one run. Every field defaults to the least surprising choice. */
interface Behavior {
/** Reject every `session/new` (exercises the expect-error step without extra dirs). */
rejectNewSession?: boolean
/** Reject `session/new` only when `additionalDirectories` is non-empty (the real bridge's rule). */
rejectExtraDirs?: boolean
/** How `session/prompt` settles: a clean response, a JSON-RPC error, or a hang until `session/cancel`. */
prompt?: 'respond' | 'error' | 'hang-until-cancel'
/** Before responding to a prompt, send a `session/request_permission` request and echo its outcome as a chunk. */
permissionProbe?: boolean
/** Echo the `DSH_SNAPSHOT_*` env the harness set as a chunk (spec-side env-plumbing assertions). */
echoEnv?: boolean
/** Echo the sorted cwd listing as a chunk (spec-side workspace-seeding assertions). */
echoWorkspace?: boolean
/** Write a line to stderr on boot (spec-side stderr-capture assertions). */
stderrNote?: string
/** Session logs to persist on stdin EOF. */
logs?: ScriptedLog[]
/** Leave a stray FILE directly under the sessions root (harvest must skip it). */
strayRootFile?: boolean
/** Leave a stray non-`.jsonl` file inside a bucket (harvest must skip it). */
strayBucketFile?: boolean
/** Delete the sessions root entirely (harvest must yield no logs). */
deleteSessionsRoot?: boolean
}
const sessionsRoot = process.env.DSH_SNAPSHOT_SESSIONS_ROOT ?? ''
const fixtureFile = process.env.DSH_SNAPSHOT_FILE ?? ''
const behavior: Behavior = fixtureFile === ''
? {}
: JSON.parse(readFileSync(join(dirname(fixtureFile), 'behavior.json'), 'utf8')) as Behavior
if (behavior.stderrNote !== undefined) process.stderr.write(`${behavior.stderrNote}\n`)
let nextOutboundId = 1000
let sessionId = ''
/**
* The cwd the client passed to `session/new` — used verbatim for `{{CWD}}`
* substitution, mirroring the real bin (whose persisted header carries the
* session cwd as given, NOT `process.cwd()`, which the OS realpaths — on
* macOS `/var/folders/…` vs `/private/var/folders/…`).
*/
let sessionCwd = ''
/** The parked prompt request id while `hang-until-cancel` waits for the cancel notification. */
let parkedPromptId: number | string | null = null
/** Resolvers for permission-probe responses, keyed by outbound request id. */
const pendingPermission = new Map<number, (outcome: unknown) => void>()
function send(frame: Record<string, unknown>): void {
process.stdout.write(`${JSON.stringify({ jsonrpc: '2.0', ...frame })}\n`)
}
function respond(id: number | string, result: unknown): void {
send({ id, result })
}
function respondError(id: number | string, message: string): void {
send({ id, error: { code: -32603, message } })
}
function chunk(text: string): void {
send({
method: 'session/update',
params: { sessionId, update: { sessionUpdate: 'agent_message_chunk', content: { type: 'text', text } } },
})
}
/** Substitute the `{{CWD}}`/`{{SID}}` templates through a scripted log record. */
function instantiate(value: unknown): unknown {
if (typeof value === 'string') return value.split('{{CWD}}').join(sessionCwd).split('{{SID}}').join(sessionId)
if (Array.isArray(value)) return value.map(instantiate)
if (value !== null && typeof value === 'object') {
const out: Record<string, unknown> = {}
for (const [k, v] of Object.entries(value)) out[k] = instantiate(v)
return out
}
return value
}
async function handlePrompt(id: number | string): Promise<void> {
if ((behavior.prompt ?? 'respond') === 'hang-until-cancel') {
// A thought chunk BEFORE any message chunk: a promptAndCancel waiter
// watches for agent_message_chunk, so this exercises its non-matching
// update path while the waiter is armed.
send({
method: 'session/update',
params: { sessionId, update: { sessionUpdate: 'agent_thought_chunk', content: { type: 'text', text: 'mulling' } } },
})
}
chunk('thinking about it')
if (behavior.echoEnv === true) {
chunk(`env:${JSON.stringify({
mode: process.env.DSH_SNAPSHOT,
override: process.env.DSH_SNAPSHOT_OVERRIDE ?? null,
childFiles: process.env.DSH_SNAPSHOT_CHILD_FILES ?? null,
})}`)
}
if (behavior.echoWorkspace === true) {
chunk(`workspace:${readdirSync(process.cwd()).sort().join(',')}`)
}
if (behavior.permissionProbe === true) {
const requestId = nextOutboundId++
const outcome = await new Promise<unknown>((resolve) => {
pendingPermission.set(requestId, resolve)
send({
id: requestId,
method: 'session/request_permission',
params: {
sessionId,
toolCall: { toolCallId: 'call_fake_1', title: 'fake tool', kind: 'execute', status: 'pending' },
options: [
{ optionId: 'opt-allow', name: 'Allow once', kind: 'allow_once' },
{ optionId: 'opt-reject', name: 'Reject once', kind: 'reject_once' },
],
},
})
})
chunk(`permission:${JSON.stringify(outcome)}`)
}
switch (behavior.prompt ?? 'respond') {
case 'respond':
respond(id, { stopReason: 'end_turn' })
return
case 'error':
respondError(id, 'model exploded')
return
case 'hang-until-cancel':
parkedPromptId = id
return
}
}
function handleFrame(frame: Record<string, unknown>): void {
const id = frame.id as number | string | undefined
const method = frame.method as string | undefined
const params = (frame.params ?? {}) as Record<string, unknown>
// A response to one of OUR outbound requests (the permission probe).
if (method === undefined && id !== undefined && typeof id === 'number' && pendingPermission.has(id)) {
const resolve = pendingPermission.get(id) as (outcome: unknown) => void
pendingPermission.delete(id)
resolve((frame.result as { outcome?: unknown } | undefined)?.outcome ?? null)
return
}
switch (method) {
case 'initialize':
respond(id as number | string, { protocolVersion: 1, agentCapabilities: { loadSession: false } })
return
case 'session/new': {
const extra = params.additionalDirectories as unknown[] | undefined
if (behavior.rejectNewSession === true || (behavior.rejectExtraDirs === true && extra !== undefined && extra.length > 0)) {
respondError(id as number | string, 'unsupported workspace scope')
return
}
sessionId = randomUUID()
sessionCwd = typeof params.cwd === 'string' ? params.cwd : process.cwd()
respond(id as number | string, { sessionId })
return
}
case 'session/prompt':
void handlePrompt(id as number | string)
return
case 'session/cancel':
if (parkedPromptId !== null) {
const parked = parkedPromptId
parkedPromptId = null
respond(parked, { stopReason: 'cancelled' })
}
return
default:
// Unknown method: a notification is ignored; a request gets an error so
// the SDK never waits forever on a frame this fake doesn't model.
if (id !== undefined) respondError(id, `unhandled method ${String(method)}`)
}
}
function flushLogsAndExit(): void {
for (const log of behavior.logs ?? []) {
const target = join(sessionsRoot, log.file)
mkdirSync(dirname(target), { recursive: true })
writeFileSync(target, log.lines.map(l => JSON.stringify(instantiate(l))).join('\n') + '\n')
}
if (behavior.strayRootFile === true) writeFileSync(join(sessionsRoot, 'stray.txt'), 'not a bucket\n')
if (behavior.strayBucketFile === true) {
mkdirSync(join(sessionsRoot, 'bucket-noise'), { recursive: true })
writeFileSync(join(sessionsRoot, 'bucket-noise', 'notes.txt'), 'not a session log\n')
}
if (behavior.deleteSessionsRoot === true) rmSync(sessionsRoot, { recursive: true, force: true })
process.exit(0)
}
const rl = createInterface({ input: process.stdin })
rl.on('line', (line) => {
if (line.trim().length === 0) return
handleFrame(JSON.parse(line) as Record<string, unknown>)
})
rl.on('close', () => { flushLogsAndExit() })

13
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-child/behavior.json vendored Normal file
View File

@@ -0,0 +1,13 @@
{
"prompt": "respond",
"logs": [
{ "file": "b/parent.jsonl", "lines": [
{ "type": "session", "id": "{{SID}}", "createdAt": 700, "cwd": "{{CWD}}" },
{ "type": "request/header", "seq": 0, "time": 3, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
]},
{ "file": "b/child.jsonl", "lines": [
{ "type": "session", "id": "abababab-cdcd-4efe-8ada-badabadabada", "createdAt": 800, "cwd": "{{CWD}}", "parentSession": "{{SID}}" },
{ "type": "request/header", "seq": 0, "time": 2, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
]}
]
}

1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-child/input.json vendored Normal file
View File

@@ -0,0 +1 @@
{ "steps": [{ "op": "initialize" }, { "op": "newSession" }, { "op": "prompt", "text": "rec child" }] }

2
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-child/session.1.jsonl vendored Normal file
View File

@@ -0,0 +1,2 @@
{"type":"session","id":"abababab-cdcd-4efe-8ada-badabadabada","createdAt":800,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-KBQJbW","parentSession":"f6fa7fcf-dd9c-4b39-8815-b25ddcebfd88"}
{"type":"request/header","seq":0,"time":2,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}

2
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-child/session.jsonl vendored Normal file
View File

@@ -0,0 +1,2 @@
{"type":"session","id":"f6fa7fcf-dd9c-4b39-8815-b25ddcebfd88","createdAt":700,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-KBQJbW"}
{"type":"request/header","seq":0,"time":3,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}

4
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-child/stdout.golden.jsonl vendored Normal file
View File

@@ -0,0 +1,4 @@
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"loadSession":false}}}
{"jsonrpc":"2.0","id":2,"result":{"sessionId":"{{sessionId}}"}}
{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"{{sessionId}}","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"thinking about it"}}}}
{"jsonrpc":"2.0","id":3,"result":{"stopReason":"end_turn"}}

10
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/behavior.json vendored Normal file
View File

@@ -0,0 +1,10 @@
{
"prompt": "respond",
"logs": [{
"file": "b/main.jsonl",
"lines": [
{ "type": "session", "id": "{{SID}}", "createdAt": 600, "cwd": "{{CWD}}" },
{ "type": "request/header", "seq": 0, "time": 4, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
]
}]
}

1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/input.json vendored Normal file
View File

@@ -0,0 +1 @@
{ "steps": [{ "op": "initialize" }, { "op": "newSession" }, { "op": "prompt", "text": "rec pin" }] }

2
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/session.jsonl vendored Normal file
View File

@@ -0,0 +1,2 @@
{"type":"session","id":"ccdc749f-56f3-4267-9750-598b5c60b7b2","createdAt":600,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-nOQ4Gy"}
{"type":"request/header","seq":0,"time":4,"data":{"header":{"config":{"model":"fake"},"system":"SYS PROMPT","tools":[{"name":"t1","description":"D1","parameters":{"type":"object"}}]},"reason":"initial"}}

4
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/stdout.golden.jsonl vendored Normal file
View File

@@ -0,0 +1,4 @@
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"loadSession":false}}}
{"jsonrpc":"2.0","id":2,"result":{"sessionId":"{{sessionId}}"}}
{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"{{sessionId}}","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"thinking about it"}}}}
{"jsonrpc":"2.0","id":3,"result":{"stopReason":"end_turn"}}

1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-skip/behavior.json vendored Normal file
View File

@@ -0,0 +1 @@
{}

1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-skip/input.json vendored Normal file
View File

@@ -0,0 +1 @@
{ "steps": [{ "op": "initialize" }] }

1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-skip/replay.override.json vendored Normal file
View File

@@ -0,0 +1 @@
[{ "kind": "hang" }]

1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-skip/session.jsonl vendored Normal file
View File

@@ -0,0 +1 @@
{"type":"session","id":"{{sessionId}}","createdAt":0,"cwd":"{{cwd}}"}

1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-skip/stdout.golden.jsonl vendored Normal file
View File

@@ -0,0 +1 @@
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"loadSession":false}}}

10
packages/support/acp-snapshot/tests/fixtures/suite/authored-error/behavior.json vendored Normal file
View File

@@ -0,0 +1,10 @@
{
"prompt": "error",
"logs": [{
"file": "b/main.jsonl",
"lines": [
{ "type": "session", "id": "{{SID}}", "createdAt": 500, "cwd": "{{CWD}}" },
{ "type": "turn/end", "seq": 1, "time": 9, "data": { "error": "model exploded" } }
]
}]
}

1
packages/support/acp-snapshot/tests/fixtures/suite/authored-error/input.json vendored Normal file
View File

@@ -0,0 +1 @@
{ "steps": [{ "op": "initialize" }, { "op": "newSession" }, { "op": "promptExpectError", "text": "boom" }] }

1
packages/support/acp-snapshot/tests/fixtures/suite/authored-error/replay.override.json vendored Normal file
View File

@@ -0,0 +1 @@
[{ "kind": "throw", "chunks": [], "message": "model exploded", "code": "PROVIDER" }]

2
packages/support/acp-snapshot/tests/fixtures/suite/authored-error/session.jsonl vendored Normal file
View File

@@ -0,0 +1,2 @@
{"type":"session","id":"44444444-3333-4222-8111-000000000000","createdAt":17,"cwd":"/rec/authored-cwd"}
{"type":"turn/end","seq":1,"time":17,"data":{"error":"model exploded"}}

4
packages/support/acp-snapshot/tests/fixtures/suite/authored-error/stdout.golden.jsonl vendored Normal file
View File

@@ -0,0 +1,4 @@
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"loadSession":false}}}
{"jsonrpc":"2.0","id":2,"result":{"sessionId":"{{sessionId}}"}}
{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"{{sessionId}}","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"thinking about it"}}}}
{"jsonrpc":"2.0","id":3,"error":{"code":-32603,"message":"model exploded"}}

10
packages/support/acp-snapshot/tests/fixtures/suite/blocked-log/behavior.json vendored Normal file
View File

@@ -0,0 +1,10 @@
{
"prompt": "error",
"logs": [{
"file": "b/main.jsonl",
"lines": [
{ "type": "session", "id": "{{SID}}", "createdAt": 400, "cwd": "{{CWD}}" },
{ "type": "hook/result", "seq": 1, "time": 8, "data": { "decision": "block", "durationMs": 37 } }
]
}]
}

1
packages/support/acp-snapshot/tests/fixtures/suite/blocked-log/input.json vendored Normal file
View File

@@ -0,0 +1 @@
{ "steps": [{ "op": "initialize" }, { "op": "newSession" }, { "op": "promptExpectError", "text": "blocked" }] }

2
packages/support/acp-snapshot/tests/fixtures/suite/blocked-log/session.jsonl vendored Normal file
View File

@@ -0,0 +1,2 @@
{"type":"session","id":"99999999-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/blocked-cwd"}
{"type":"hook/result","seq":1,"time":13,"data":{"decision":"block","durationMs":99}}

4
packages/support/acp-snapshot/tests/fixtures/suite/blocked-log/stdout.golden.jsonl vendored Normal file
View File

@@ -0,0 +1,4 @@
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"loadSession":false}}}
{"jsonrpc":"2.0","id":2,"result":{"sessionId":"{{sessionId}}"}}
{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"{{sessionId}}","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"thinking about it"}}}}
{"jsonrpc":"2.0","id":3,"error":{"code":-32603,"message":"model exploded"}}

1
packages/support/acp-snapshot/tests/fixtures/suite/no-model/behavior.json vendored Normal file
View File

@@ -0,0 +1 @@
{}

1
packages/support/acp-snapshot/tests/fixtures/suite/no-model/input.json vendored Normal file
View File

@@ -0,0 +1 @@
{ "steps": [{ "op": "initialize" }] }

1
packages/support/acp-snapshot/tests/fixtures/suite/no-model/session.jsonl vendored Normal file
View File

@@ -0,0 +1 @@
{"type":"session","id":"{{sessionId}}","createdAt":0,"cwd":"{{cwd}}"}

1
packages/support/acp-snapshot/tests/fixtures/suite/no-model/stdout.golden.jsonl vendored Normal file
View File

@@ -0,0 +1 @@
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"loadSession":false}}}

11
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/behavior.json vendored Normal file
View File

@@ -0,0 +1,11 @@
{
"prompt": "respond",
"logs": [{
"file": "b/main.jsonl",
"lines": [
{ "type": "session", "id": "{{SID}}", "createdAt": 100, "cwd": "{{CWD}}" },
{ "type": "request/header", "seq": 0, "time": 100, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } },
{ "type": "turn/start", "seq": 1, "time": 100, "data": { "turn": 1 } }
]
}]
}

1
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/input.json vendored Normal file
View File

@@ -0,0 +1 @@
{ "steps": [{ "op": "initialize" }, { "op": "newSession" }, { "op": "prompt", "text": "pin" }] }

3
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/session.jsonl vendored Normal file
View File

@@ -0,0 +1,3 @@
{"type":"session","id":"12121212-3434-4545-8686-787878787878","createdAt":7,"cwd":"/rec/pin-cwd"}
{"type":"request/header","seq":0,"time":7,"data":{"header":{"config":{"model":"fake"},"system":"SYS PROMPT","tools":[{"name":"t1","description":"D1","parameters":{"type":"object"}}]},"reason":"initial"}}
{"type":"turn/start","seq":1,"time":7,"data":{"turn":1}}

4
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/stdout.golden.jsonl vendored Normal file
View File

@@ -0,0 +1,4 @@
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"loadSession":false}}}
{"jsonrpc":"2.0","id":2,"result":{"sessionId":"{{sessionId}}"}}
{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"{{sessionId}}","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"thinking about it"}}}}
{"jsonrpc":"2.0","id":3,"result":{"stopReason":"end_turn"}}

15
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/behavior.json vendored Normal file
View File

@@ -0,0 +1,15 @@
{
"prompt": "respond",
"echoWorkspace": true,
"logs": [
{ "file": "b/parent.jsonl", "lines": [
{ "type": "session", "id": "{{SID}}", "createdAt": 200, "cwd": "{{CWD}}" },
{ "type": "request/header", "seq": 0, "time": 5, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } },
{ "type": "assistant/chunk", "seq": 1, "time": 5, "data": { "turn": 1, "step": 1, "chunk": { "type": "text-delta", "index": 0, "text": "hi" } } }
]},
{ "file": "b/child.jsonl", "lines": [
{ "type": "session", "id": "eeeeeeee-1111-4222-8333-444444444444", "createdAt": 300, "cwd": "{{CWD}}", "parentSession": "{{SID}}" },
{ "type": "request/header", "seq": 0, "time": 6, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
]}
]
}

1
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/input.json vendored Normal file
View File

@@ -0,0 +1 @@
{ "steps": [{ "op": "initialize" }, { "op": "newSession" }, { "op": "prompt", "text": "plain" }] }

2
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/session.1.jsonl vendored Normal file
View File

@@ -0,0 +1,2 @@
{"type":"session","id":"eeeeeeee-1111-4222-8333-444444444444","createdAt":12,"cwd":"/rec/plain-cwd","parentSession":"56565656-7878-4989-8a9a-9b9b9b9b9b9b"}
{"type":"request/header","seq":0,"time":12,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}

3
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/session.jsonl vendored Normal file
View File

@@ -0,0 +1,3 @@
{"type":"session","id":"56565656-7878-4989-8a9a-9b9b9b9b9b9b","createdAt":11,"cwd":"/rec/plain-cwd"}
{"type":"request/header","seq":0,"time":11,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
{"type":"assistant/chunk","seq":1,"time":11,"data":{"turn":1,"step":1,"chunk":{"type":"text-delta","index":0,"text":"hi"}}}

5
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/stdout.golden.jsonl vendored Normal file
View File

@@ -0,0 +1,5 @@
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"loadSession":false}}}
{"jsonrpc":"2.0","id":2,"result":{"sessionId":"{{sessionId}}"}}
{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"{{sessionId}}","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"thinking about it"}}}}
{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"{{sessionId}}","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"workspace:seed.txt"}}}}
{"jsonrpc":"2.0","id":3,"result":{"stopReason":"end_turn"}}

1
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/workspace/seed.txt vendored Normal file
View File

@@ -0,0 +1 @@
seeded

272
packages/support/acp-snapshot/tests/harness.spec.ts Normal file
View File

@@ -0,0 +1,272 @@
import { mkdtemp, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { delimiter, join } from 'node:path'
import { fileURLToPath } from 'node:url'
import { afterAll, describe, expect, it } from 'vitest'
import { runScenario, type AgentUnderTest, type InputStep } from '../src/harness.ts'
/**
* Unit tests for the subprocess harness, driven through the REAL spawn path
* (tsx loader, temp cwd, env plumbing) against the scripted fake ACP bin in
* ./fixtures/fake-acp-agent.ts. Each case writes a `behavior.json` next to a
* throwaway fixture path; the fake bin echoes observable facts (env, seeded
* workspace, permission outcomes) into `agent_message_chunk` text, so the
* assertions read plain `rawStdout`.
*/
const AGENT: AgentUnderTest = {
binScript: fileURLToPath(new URL('./fixtures/fake-acp-agent.ts', import.meta.url)),
// The fake bin ignores its config argv; any real path documents the shape.
configPath: fileURLToPath(new URL('./fixtures/fake-acp-agent.ts', import.meta.url)),
tsconfigPath: fileURLToPath(new URL('../../../../tsconfig.json', import.meta.url)),
}
/** Temp scenario dirs to drop after the suite. */
const tempDirs: string[] = []
afterAll(async () => {
for (const dir of tempDirs) await rm(dir, { recursive: true, force: true })
})
/** Write a behavior.json into a fresh temp dir; return the sibling fixture path the harness points the bin at. */
async function scenario(behavior: object): Promise<{ dir: string; fixtureFile: string }> {
const dir = await mkdtemp(join(tmpdir(), 'acp-snap-spec-'))
tempDirs.push(dir)
await writeFile(join(dir, 'behavior.json'), JSON.stringify(behavior))
return { dir, fixtureFile: join(dir, 'session.jsonl') }
}
const boot: InputStep[] = [{ op: 'initialize' }, { op: 'newSession' }]
describe('runScenario', () => {
it('drives a full turn: initialize (terminal caps), session, prompt, permission stub, harvest', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({
permissionProbe: true,
logs: [{
file: 'bucket/main.jsonl',
lines: [
{ type: 'session', id: '{{SID}}', createdAt: 42, cwd: '{{CWD}}' },
{ type: 'turn/start', seq: 1, time: 9, data: { turn: 1 } },
],
}],
})
const result = await runScenario(
{ steps: [{ op: 'initialize', terminalOutput: true }, { op: 'newSession' }, { op: 'prompt', text: 'go' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.sessionId).toBeDefined()
// The harness's client answers a permission request with `cancelled`; the
// fake bin echoes the outcome it received back as a chunk.
expect(result.rawStdout).toContain('permission:{\\"outcome\\":\\"cancelled\\"}')
expect(result.sessionLogs).toHaveLength(1)
expect(result.sessionLogs[0]?.id).toBe(result.sessionId)
expect(result.sessionLogs[0]?.createdAt).toBe(42)
expect(result.sessionLogs[0]?.content).toContain('turn/start')
// The harvested log embeds the run's REAL temp cwd (template-substituted).
expect(result.sessionLogs[0]?.content).toContain(result.cwd)
})
it('forwards override/child fixture paths into the child env and captures stderr', { timeout: 20_000 }, async () => {
const { dir, fixtureFile } = await scenario({ echoEnv: true, stderrNote: 'fake bin booted' })
const childFiles = [join(dir, 'session.1.jsonl'), join(dir, 'session.2.jsonl')]
const result = await runScenario(
{ steps: [...boot, { op: 'prompt', text: 'env?' }] },
{
agent: AGENT,
mode: 'replay',
fixtureFile,
overrideFile: join(dir, 'replay.override.json'),
childFiles,
// A workspaceDir that does not exist is skipped, not an error.
workspaceDir: join(dir, 'no-such-workspace'),
},
)
expect(result.stderr).toContain('fake bin booted')
expect(result.rawStdout).toContain('replay.override.json')
// Child paths ride one env var, joined with the platform delimiter.
expect(result.rawStdout).toContain(JSON.stringify(childFiles.join(delimiter)).slice(1, -1))
})
it('seeds the workspace dir into the temp cwd before the run', { timeout: 20_000 }, async () => {
const { dir, fixtureFile } = await scenario({ echoWorkspace: true })
const workspaceDir = join(dir, 'workspace')
await writeFile(join(dir, 'behavior.json'), JSON.stringify({ echoWorkspace: true }))
const { mkdir } = await import('node:fs/promises')
await mkdir(workspaceDir, { recursive: true })
await writeFile(join(workspaceDir, 'seeded.txt'), 'hello')
const result = await runScenario(
{ steps: [...boot, { op: 'prompt', text: 'ls' }] },
{ agent: AGENT, mode: 'replay', fixtureFile, workspaceDir },
)
expect(result.rawStdout).toContain('workspace:seeded.txt')
})
it('promptAndCancel waits for the streamed chunk, cancels, and settles the prompt', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ prompt: 'hang-until-cancel' })
const result = await runScenario(
{ steps: [...boot, { op: 'promptAndCancel', text: 'hang' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.rawStdout).toContain('"stopReason":"cancelled"')
// The streamed chunk deterministically precedes the cancelled response.
expect(result.rawStdout.indexOf('thinking about it')).toBeLessThan(result.rawStdout.indexOf('cancelled'))
})
it('promptExpectError swallows a model-error response as the expected outcome', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ prompt: 'error' })
const result = await runScenario(
{ steps: [...boot, { op: 'promptExpectError', text: 'boom' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.rawStdout).toContain('model exploded')
})
it('promptExpectError throws when the prompt unexpectedly succeeds (and teardown kills the live child)', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ prompt: 'respond' })
await expect(runScenario(
{ steps: [...boot, { op: 'promptExpectError', text: 'fine' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/expected the prompt to fail/)
})
it('newSessionExpectError swallows the rejection, with and without extra dirs', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ rejectExtraDirs: true })
const result = await runScenario(
{ steps: [{ op: 'initialize' }, { op: 'newSessionExpectError', additionalDirectories: ['/elsewhere'] }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
// No session was created, so no id and no logs.
expect(result.sessionId).toBeUndefined()
expect(result.sessionLogs).toHaveLength(0)
const rejectAll = await scenario({ rejectNewSession: true })
const second = await runScenario(
{ steps: [{ op: 'initialize' }, { op: 'newSessionExpectError' }] },
{ agent: AGENT, mode: 'replay', fixtureFile: rejectAll.fixtureFile },
)
expect(second.rawStdout).toContain('unsupported workspace scope')
})
it('newSessionExpectError throws when session/new unexpectedly succeeds', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({})
await expect(runScenario(
{ steps: [{ op: 'initialize' }, { op: 'newSessionExpectError' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/expected session\/new to be rejected/)
})
it('a plain cancel step is forwarded (and ignored by an idle agent)', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({})
const result = await runScenario(
{ steps: [...boot, { op: 'cancel' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.sessionId).toBeDefined()
})
it.each([
[{ op: 'prompt', text: 'x' }, /prompt before newSession/],
[{ op: 'promptExpectError', text: 'x' }, /promptExpectError before newSession/],
[{ op: 'promptAndCancel', text: 'x' }, /promptAndCancel before newSession/],
[{ op: 'cancel' }, /cancel before newSession/],
] as [InputStep, RegExp][])('rejects %j before newSession', { timeout: 20_000 }, async (step, message) => {
const { fixtureFile } = await scenario({})
await expect(runScenario(
{ steps: [{ op: 'initialize' }, step] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(message)
})
it('rejects an unknown input op', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({})
const bogus = { op: 'reticulate' } as unknown as InputStep
await expect(runScenario(
{ steps: [bogus] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/unknown input op/)
})
it('harvests all logs primary-first, children by createdAt then id, skipping filesystem noise', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({
strayRootFile: true,
strayBucketFile: true,
logs: [
// File names chosen so readdir feeds the sort children-first AND
// parent-in-the-middle: the comparator then sees a parent on both
// sides of a pair, plus the same-createdAt (localeCompare) tiebreak.
{ file: 'b1/aa-child-c.jsonl', lines: [{ type: 'session', id: 'cccccccc-0000-4000-8000-000000000000', createdAt: 500, parentSession: '{{SID}}' }] },
{ file: 'b1/bb-parent.jsonl', lines: [{ type: 'session', id: '{{SID}}', createdAt: 900 }] },
{ file: 'b1/cc-child-a.jsonl', lines: [{ type: 'session', id: 'aaaaaaaa-0000-4000-8000-000000000000', createdAt: 500, parentSession: '{{SID}}' }] },
// Missing id/createdAt fall back to ''/0; earliest child by createdAt.
{ file: 'b2/orphan-fields.jsonl', lines: [{ type: 'session', parentSession: '{{SID}}' }] },
],
})
const result = await runScenario(
{ steps: [...boot, { op: 'prompt', text: 'go' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.sessionLogs.map(l => [l.id, l.createdAt])).toEqual([
[result.sessionId, 900],
['', 0],
['aaaaaaaa-0000-4000-8000-000000000000', 500],
['cccccccc-0000-4000-8000-000000000000', 500],
])
expect(result.sessionLogs[1]?.parentSession).toBe(result.sessionId)
})
it('treats an empty log file as a header-less primary with default fields', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ logs: [{ file: 'b/empty.jsonl', lines: [] }] })
const result = await runScenario(
{ steps: boot },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.sessionLogs.map(l => [l.id, l.createdAt, l.parentSession])).toEqual([['', 0, undefined]])
})
it('yields no logs when the sessions root vanished', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ deleteSessionsRoot: true })
const result = await runScenario(
{ steps: boot },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.sessionLogs).toHaveLength(0)
})
it('answers permission requests from the scripted queue by option kind, falling back to cancelled', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ permissionProbe: true })
// Two prompts → two permission round-trips; one scripted answer, so the
// second request exercises the exhausted-queue fallback.
const result = await runScenario(
{
steps: [...boot, { op: 'prompt', text: 'one' }, { op: 'prompt', text: 'two' }],
permissionAnswers: [{ kind: 'allow_once' }],
},
{ agent: AGENT, mode: 'replay', fixtureFile },
)
const first = result.rawStdout.indexOf('permission:{\\"outcome\\":\\"selected\\",\\"optionId\\":\\"opt-allow\\"}')
const second = result.rawStdout.indexOf('permission:{\\"outcome\\":\\"cancelled\\"}')
expect(first).toBeGreaterThanOrEqual(0)
expect(second).toBeGreaterThan(first)
})
it('selects a non-first offered option by kind', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ permissionProbe: true })
const result = await runScenario(
{ steps: [...boot, { op: 'prompt', text: 'deny it' }], permissionAnswers: [{ kind: 'reject_once' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.rawStdout).toContain('permission:{\\"outcome\\":\\"selected\\",\\"optionId\\":\\"opt-reject\\"}')
})
it('rejects the run on a scripted permission kind the agent never offered', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ permissionProbe: true })
// The fake bin offers allow_once/reject_once; scripting allow_always is a
// scenario bug. The agent is answered `cancelled` (it must not be able to
// absorb the bug as an error-means-denial), and the RUN fails: a callback
// throw would only reach the agent as a JSON-RPC error response, letting
// a tolerant agent carry on and the scenario pass — or record.
await expect(runScenario(
{ steps: [...boot, { op: 'prompt', text: 'impossible click' }], permissionAnswers: [{ kind: 'allow_always' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/allow_always not among the offered options \[allow_once, reject_once\]/)
})
})

230
packages/support/acp-snapshot/tests/normalize.spec.ts Normal file
View File

@@ -0,0 +1,230 @@
import { describe, expect, it } from 'vitest'
import { type NormalizeContext, normalizeSessionLog, normalizeStdout, scrubRequestHeaders } from '../src/normalize.ts'
/**
* Unit tests for the pure snapshot normalizers. Live as a *.spec.ts (runs in
* the default unit gate) and import the normalizers directly.
*/
const ctx: NormalizeContext = {
sessionIds: ['11111111-2222-3333-4444-555555555555'],
cwd: '/tmp/acp-snap-cwd-abc123',
}
describe('normalizeStdout', () => {
it('rewrites JSON-RPC ids to a stable first-seen sequence', () => {
const raw = [
JSON.stringify({ jsonrpc: '2.0', id: 42, method: 'initialize' }),
JSON.stringify({ jsonrpc: '2.0', id: 42, result: {} }),
JSON.stringify({ jsonrpc: '2.0', id: 99, method: 'session/new' }),
].join('\n')
const out = normalizeStdout(raw, ctx)
expect(out).toContain('"id":1')
expect(out).toContain('"id":2')
expect(out).not.toContain('42')
expect(out).not.toContain('99')
})
it('scrubs the cwd and session id anywhere they appear', () => {
const raw = JSON.stringify({
jsonrpc: '2.0', method: 'session/update',
params: { sessionId: ctx.sessionIds[0], cwd: ctx.cwd, note: `at ${ctx.cwd}/x` },
})
const out = normalizeStdout(raw, ctx)
expect(out).toContain('{{sessionId}}')
expect(out).toContain('{{cwd}}')
expect(out).not.toContain(ctx.cwd)
expect(out).not.toContain(ctx.sessionIds[0] as string)
})
it('scrubs a stray UUID not in the known list', () => {
const raw = JSON.stringify({ jsonrpc: '2.0', method: 'x', params: { id: 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee' } })
expect(normalizeStdout(raw, ctx)).toContain('{{sessionId}}')
})
it('leaves notification frames without an id untouched in id-space', () => {
const raw = JSON.stringify({ jsonrpc: '2.0', method: 'session/update', params: {} })
const out = normalizeStdout(raw, ctx)
expect(out).not.toContain('"id"')
})
it('throws on a non-JSON stdout line (the purity check)', () => {
const raw = `${JSON.stringify({ jsonrpc: '2.0', id: 1 })}\noops a log leaked\n`
expect(() => normalizeStdout(raw, ctx)).toThrow()
})
it('ignores blank lines', () => {
const raw = `\n${JSON.stringify({ jsonrpc: '2.0', id: 1, method: 'm' })}\n\n`
expect(() => normalizeStdout(raw, ctx)).not.toThrow()
})
})
describe('normalizeSessionLog', () => {
const header = (over: object) => JSON.stringify({ type: 'session', version: 0, id: 's', createdAt: 123, ...over })
const event = (over: object) => JSON.stringify({ type: 'turn/start', seq: 1, time: 999, data: { turn: 1 }, ...over })
it('zeroes the header createdAt', () => {
const out = normalizeSessionLog(`${header({})}\n`, ctx)
expect(out).toContain('"createdAt":0')
expect(out).not.toContain('123')
})
it('zeroes each event time but keeps seq', () => {
const out = normalizeSessionLog(`${header({})}\n${event({ seq: 7, time: 999 })}\n`, ctx)
expect(out).toContain('"time":0')
expect(out).toContain('"seq":7') // seq is deterministic — NOT scrubbed
expect(out).not.toContain('999')
})
it('scrubs cwd and session id deep inside event data', () => {
const ev = JSON.stringify({
type: 'tool/result', seq: 2, time: 5,
data: { content: [{ type: 'text', text: `wrote ${ctx.cwd}/proof.txt` }] },
})
const out = normalizeSessionLog(`${header({ cwd: ctx.cwd })}\n${ev}\n`, ctx)
expect(out).toContain('{{cwd}}')
expect(out).not.toContain(ctx.cwd)
})
it('scrubs the session id in the header', () => {
const out = normalizeSessionLog(`${header({ id: ctx.sessionIds[0] })}\n`, ctx)
expect(out).toContain('{{sessionId}}')
})
it('zeroes a hook/result durationMs (run-to-run noise) but keeps its decision', () => {
const ev = JSON.stringify({
type: 'hook/result', seq: 2, time: 5,
data: { turn: 1, point: 'UserPromptSubmit', handlerId: 'h', decision: 'block', exitCode: 2, durationMs: 37 },
})
const out = normalizeSessionLog(`${header({})}\n${ev}\n`, ctx)
expect(out).toContain('"durationMs":0')
expect(out).not.toContain('37')
expect(out).toContain('"decision":"block"') // the decision is the behavior — kept
})
it('leaves a non-hook event durationMs untouched (only hook/result is scrubbed)', () => {
const ev = JSON.stringify({ type: 'tool/result', seq: 2, time: 5, data: { durationMs: 88 } })
const out = normalizeSessionLog(`${header({})}\n${ev}\n`, ctx)
expect(out).toContain('"durationMs":88')
})
it('tolerates records missing the volatile fields it would zero', () => {
const bareHeader = JSON.stringify({ type: 'session', id: 's' })
const timeless = JSON.stringify({ type: 'note', seq: 1 })
const bareHook = JSON.stringify({ type: 'hook/result', seq: 2, time: 5, data: { decision: 'allow' } })
const nullDataHook = JSON.stringify({ type: 'hook/result', seq: 3, time: 6, data: null })
const out = normalizeSessionLog(`${bareHeader}\n${timeless}\n${bareHook}\n${nullDataHook}\n`, ctx)
expect(out).toContain('"type":"note","seq":1')
expect(out).toContain('"decision":"allow"')
expect(out).not.toContain('durationMs')
})
})
describe('scrubRequestHeaders', () => {
const headerLine = JSON.stringify({ type: 'session', version: 0, id: 's', createdAt: 1, cwd: '/w' })
const headerEvent = (header: object) =>
JSON.stringify({ type: 'request/header', seq: 3, time: 9, data: { header, reason: 'initial' } })
it('replaces header system and tools with tokens, keeping config and reason', () => {
const ev = headerEvent({
config: { model: 'm' },
system: 'You are an agent.\nBe brief.',
tools: [{ name: 'read', description: 'Read a file.', parameters: { type: 'object' } }],
})
const out = scrubRequestHeaders(`${headerLine}\n${ev}\n`)
expect(out).toContain('"system":"{{system}}"')
expect(out).toContain('"tools":"{{tools}}"')
expect(out).toContain('"config":{"model":"m"}')
expect(out).toContain('"reason":"initial"')
expect(out).not.toContain('You are an agent')
expect(out).not.toContain('Read a file')
})
it('keeps an absent system/tools absent (presence is behavior)', () => {
const out = scrubRequestHeaders(`${headerLine}\n${headerEvent({ config: { model: 'm' } })}\n`)
expect(out).not.toContain('{{system}}')
expect(out).not.toContain('{{tools}}')
})
it('scrubs a header carrying only one of system/tools, leaving the other absent', () => {
const systemOnly = scrubRequestHeaders(`${headerLine}\n${headerEvent({ system: 'secret prompt' })}\n`)
expect(systemOnly).toContain('"system":"{{system}}"')
expect(systemOnly).not.toContain('{{tools}}')
const toolsOnly = scrubRequestHeaders(`${headerLine}\n${headerEvent({ tools: [{ name: 't' }] })}\n`)
expect(toolsOnly).toContain('"tools":"{{tools}}"')
expect(toolsOnly).not.toContain('{{system}}')
})
it('leaves a delta with no scrubbable payload byte-identical (config-only, or non-array shapes)', () => {
const configOnly = JSON.stringify({ type: 'request/header-delta', seq: 8, time: 9, data: { config: { model: 'm2' } } })
const oddShapes = JSON.stringify({ type: 'request/header-delta', seq: 9, time: 9, data: { system: { insert: 'not-an-array' }, tools: null } })
const headerless = JSON.stringify({ type: 'request/header', seq: 10, time: 9, data: { reason: 'initial' } })
const nullData = JSON.stringify({ type: 'request/header', seq: 11, time: 9, data: null })
const raw = `${headerLine}\n${configOnly}\n${oddShapes}\n${headerless}\n${nullData}\n`
expect(scrubRequestHeaders(raw)).toBe(raw)
})
it('scrubs a one-sided tools delta and passes non-object schema entries through', () => {
const addedOnly = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: { tools: { added: [null, 'weird', { name: 'x', description: 'D' }] } },
})
const out = scrubRequestHeaders(`${headerLine}\n${addedOnly}\n`)
// Non-object entries survive untouched; the object entry keeps only name.
expect(out).toContain('"added":[null,"weird",{"name":"x","description":"{{tools}}"}]')
const changedOnly = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: { tools: { changed: [{ name: 'y', parameters: {} }] } },
})
expect(scrubRequestHeaders(`${headerLine}\n${changedOnly}\n`))
.toContain('"changed":[{"name":"y","parameters":"{{tools}}"}]')
})
it('scrubs a header-delta system payload but keeps its line positions and arity', () => {
const delta = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: { system: { keepStart: 1, keepEnd: 4, insert: ['leaked prompt line', 'second line'] }, config: { model: 'm2' } },
})
const out = scrubRequestHeaders(`${headerLine}\n${delta}\n`)
// One token PER inserted line: the edit's position AND extent survive.
expect(out).toContain('"insert":["{{system}}","{{system}}"]')
expect(out).toContain('"keepStart":1')
expect(out).toContain('"keepEnd":4')
expect(out).toContain('"config":{"model":"m2"}')
expect(out).not.toContain('leaked prompt line')
expect(out).not.toContain('{{tools}}') // no tools delta → none invented
})
it('scrubs a header-delta tools payload but keeps the added/removed/changed names', () => {
const delta = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: {
tools: {
added: [{ name: 'grep', description: 'Search files.', parameters: { type: 'object' } }],
removed: ['bash_kill'],
changed: [{ name: 'read', description: 'Read v2.', parameters: { type: 'object' } }],
},
},
})
const out = scrubRequestHeaders(`${headerLine}\n${delta}\n`)
// WHICH tools changed is behavior and survives; their bulk does not.
expect(out).toContain('"added":[{"name":"grep","description":"{{tools}}","parameters":"{{tools}}"}]')
expect(out).toContain('"removed":["bash_kill"]')
expect(out).toContain('"changed":[{"name":"read","description":"{{tools}}","parameters":"{{tools}}"}]')
expect(out).not.toContain('Search files')
expect(out).not.toContain('Read v2')
})
it('passes every other line through byte-for-byte and is idempotent', () => {
const other = JSON.stringify({ type: 'assistant/chunk', seq: 4, time: 9, data: { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'hi' } } })
const delta = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: { system: { keepStart: 0, keepEnd: 0, insert: ['x'] }, tools: { added: [{ name: 't', description: 'd', parameters: {} }], removed: [], changed: [] } },
})
const raw = `${headerLine}\n${headerEvent({ config: { model: 'm' }, system: 's', tools: [] })}\n${delta}\n${other}\n`
const once = scrubRequestHeaders(raw)
expect(once.split('\n')[0]).toBe(headerLine)
expect(once.split('\n')[3]).toBe(other)
expect(scrubRequestHeaders(once)).toBe(once)
})
})

145
packages/support/acp-snapshot/tests/suite.spec.ts Normal file
View File

@@ -0,0 +1,145 @@
import { cpSync, mkdtempSync } from 'node:fs'
import { rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { fileURLToPath } from 'node:url'
import { afterAll, describe, expect, it } from 'vitest'
import { defineAcpSnapshotSuite, type Scenario } from '../src/index.ts'
import { childFixturePaths, fixtureContext, headerDeltaCount, normalizedHeaders } from '../src/suite.ts'
/**
* Unit tests for the suite factory, by running it: two synthetic suites over
* the scripted fake ACP bin (./fixtures/fake-acp-agent.ts) register REAL
* describe/it trees at collection time, so every factory path — golden and log
* compares, the per-suite header pin and its uniformity guard, record-mode
* fixture write-back, skip semantics, and the fixture guard block — executes
* as an ordinary green test. The pure helpers get direct cases below.
*
* The replay suite runs against the committed fixtures in ./fixtures/suite.
* The record suite runs against a TEMP COPY of ./fixtures/record-suite
* (record mode writes session fixtures back into its snapshots dir; a run must
* never touch the committed tree). To re-bootstrap the record tree's goldens
* after changing the fake bin's output, run this spec once with
* `ACP_SNAPSHOT_SPEC_BOOTSTRAP=1` (points the record suite at the committed
* tree so vitest creates/updates the goldens and the write-back lands there),
* then commit the result.
*/
const AGENT = {
binScript: fileURLToPath(new URL('./fixtures/fake-acp-agent.ts', import.meta.url)),
configPath: fileURLToPath(new URL('./fixtures/fake-acp-agent.ts', import.meta.url)),
tsconfigPath: fileURLToPath(new URL('../../../../tsconfig.json', import.meta.url)),
}
const REPLAY_DIR = fileURLToPath(new URL('./fixtures/suite', import.meta.url))
const RECORD_SRC = fileURLToPath(new URL('./fixtures/record-suite', import.meta.url))
const REPLAY_SCENARIOS: Scenario[] = [
{ name: 'pin-turn', hasModelTurn: true, recorded: true, pinsHeader: true },
{ name: 'plain-turn', hasModelTurn: true, recorded: true, childSessions: 1 },
{ name: 'no-model', hasModelTurn: false, recorded: false },
{ name: 'blocked-log', hasModelTurn: false, comparesLog: true, recorded: false },
{ name: 'authored-error', hasModelTurn: true, recorded: false, overridden: true },
]
const RECORD_SCENARIOS: Scenario[] = [
{ name: 'rec-pin', hasModelTurn: true, recorded: true, pinsHeader: true },
{ name: 'rec-child', hasModelTurn: true, recorded: true, childSessions: 1 },
// recorded:false in record mode → registered but skipped (never re-recorded).
{ name: 'rec-skip', hasModelTurn: true, recorded: false, overridden: true },
]
// Record mode mutates its snapshots dir, so run it on a throwaway copy —
// except under the documented bootstrap knob, which regenerates the committed
// fixtures/goldens in place.
const BOOTSTRAP = process.env.ACP_SNAPSHOT_SPEC_BOOTSTRAP === '1'
const recordDir = BOOTSTRAP ? RECORD_SRC : mkdtempSync(join(tmpdir(), 'acp-snap-record-suite-'))
if (!BOOTSTRAP) cpSync(RECORD_SRC, recordDir, { recursive: true })
afterAll(async () => {
if (!BOOTSTRAP) await rm(recordDir, { recursive: true, force: true })
})
describe('defineAcpSnapshotSuite: replay mode', () => {
defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: REPLAY_DIR, scenarios: REPLAY_SCENARIOS, mode: 'replay' })
})
// The record suite's tests run in registration order: rec-pin re-records the
// pinned fixture FIRST, so rec-child's uniformity guard reads the fresh pin.
describe('defineAcpSnapshotSuite: record mode', () => {
defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: recordDir, scenarios: RECORD_SCENARIOS, mode: 'record' })
})
describe('defineAcpSnapshotSuite: registration contract', () => {
it('throws when no scenario pins the request-header content', () => {
expect(() => {
defineAcpSnapshotSuite({
agent: AGENT,
snapshotsDir: REPLAY_DIR,
scenarios: [{ name: 'pinless', hasModelTurn: true, recorded: true }],
mode: 'replay',
})
}).toThrow(/no scenario pins/)
})
})
describe('childFixturePaths', () => {
it('yields one sibling path per child, 1-based', () => {
expect(childFixturePaths('/snap/s', 2)).toEqual(['/snap/s/session.1.jsonl', '/snap/s/session.2.jsonl'])
})
it('yields nothing for a single-session scenario', () => {
expect(childFixturePaths('/snap/s', 0)).toEqual([])
})
})
describe('fixtureContext', () => {
it('reads the fixture header id and cwd', () => {
const ctx = fixtureContext('{"type":"session","id":"abc","cwd":"/rec"}\n{"type":"turn/start"}\n')
expect(ctx).toEqual({ sessionIds: ['abc'], cwd: '/rec' })
})
it('yields no session ids for a header without a string id', () => {
expect(fixtureContext('{"type":"session","cwd":"/rec"}\n').sessionIds).toEqual([])
})
it('falls back to an impossible sentinel cwd (never the empty string)', () => {
const ctx = fixtureContext('{"type":"session","id":"abc"}\n')
expect(ctx.cwd).toBe('\0no-cwd\0')
expect(ctx.cwd).not.toBe('')
})
it('treats an empty fixture as an empty header', () => {
expect(fixtureContext('')).toEqual({ sessionIds: [], cwd: '\0no-cwd\0' })
})
})
describe('normalizedHeaders', () => {
const header = (system: string): string => JSON.stringify({
type: 'request/header', seq: 0, time: 9, data: { header: { config: { model: 'm' }, system }, reason: 'initial' },
})
it('extracts every request/header payload in log order, normalized', () => {
const id = '11111111-2222-4333-8444-555555555555'
const log = `${JSON.stringify({ type: 'session', id, createdAt: 5, cwd: '/w' })}\n${header('one')}\n`
+ `${JSON.stringify({ type: 'turn/start', seq: 1, time: 9, data: { turn: 1 } })}\n${header('two')}\n`
const headers = normalizedHeaders(log, { sessionIds: [id], cwd: '/w' })
expect(headers).toEqual([
{ config: { model: 'm' }, system: 'one' },
{ config: { model: 'm' }, system: 'two' },
])
})
it('yields nothing for a log without header events', () => {
const log = `${JSON.stringify({ type: 'session', id: 'a', createdAt: 5 })}\n`
expect(normalizedHeaders(log, { sessionIds: [], cwd: '/w' })).toEqual([])
})
})
describe('headerDeltaCount', () => {
it('counts request/header-delta events, ignoring blanks and other lines', () => {
const delta = JSON.stringify({ type: 'request/header-delta', seq: 2, time: 9, data: {} })
const other = JSON.stringify({ type: 'request/header', seq: 0, time: 9, data: {} })
expect(headerDeltaCount(`${other}\n\n${delta}\n${delta}\n`)).toBe(2)
expect(headerDeltaCount(`${other}\n`)).toBe(0)
})
})

11
packages/support/acp-snapshot/tsconfig.json Normal file
View File

@@ -0,0 +1,11 @@
{
"extends": "../../../tsconfig.base.json",
"compilerOptions": {
"rootDir": "src",
"outDir": "lib/types"
},
"include": [
"src"
],
"references": []
}