Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge origin/master into codex/truncated-design

Browse Source
This commit is contained in:
Dudu-0223
2026-07-17 18:21:54 +08:00
parent 8ab39267f3 44e126e306
commit a3f62478e1
1057 changed files with 42961 additions and 18526 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
packages/support/README.md
View File

@@ -5,8 +5,9 @@ Packages that exist to serve development, testing, and the examples rather than
| Package | Role | ctx key |
|---|---|---|
| `acp-snapshot/` | ACP snapshot suite kit: subprocess scenario harness + golden normalizers + the `defineAcpSnapshotSuite` factory | (library — imported by example `*.snapshot.ts` suites) |
| `invariants/` | Dev-mode event-contract assertions | (listens on `session/*`, `agent/*`) |
| `invariants/` | Runtime event-contract assertions for development diagnostics | (listens on `session/*`, `agent/*`) |
| `loader-smoke/` | Shared real-Loader subprocess harness for keyless example smokes | (library — imported by example e2e suites) |
| `llm-replay/` | Record/replay adapter: short-circuits `llm/stream` from a recorded session JSONL (keyless snapshot tests) | (listens on `llm/stream`) |
| `subagent-mock/` | Scripted `SubagentProvider` for deterministic seam/tool tests | (registers on `ctx.subagents`) |
`invariants` runs only in dev mode (contract checks, not runtime behavior). `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate. `acp-snapshot` carries the snapshot tier's harness/normalizer/suite machinery so every example's suite is a scenario table over one shared, gate-covered implementation. `subagent-mock` exercises the real `ctx.subagents` load path without a model or child agent. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.
`invariants` is development support but has no environment guard: it runs wherever registered, and the default `dsh-agent-spine-demo` bundle mounts it unconditionally. `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate. `acp-snapshot` carries the snapshot tier's harness/normalizer/suite machinery, while `loader-smoke` owns the parallel stdio/Loader process boundary used by keyless example e2e suites. `subagent-mock` exercises the real `ctx.subagents` load path without a model or child agent. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.

21
packages/support/acp-snapshot/README.md
View File

@@ -5,8 +5,8 @@ The ACP snapshot suite kit: the shared machinery behind the keyless snapshot tie
Three layers, importable separately:
- **`runScenario` (harness)** — boots the real agent bin as a subprocess via tsx (unbuilt, Loader path), drives it over ACP JSON-RPC stdio from a deterministic `input.json` script, tees raw stdout for the golden + purity check, and harvests every persisted session JSONL (parent + subagent children, primary-first) after a graceful stdin-EOF shutdown. Parameterized by `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath` — absolute paths; the subprocess cwd is a temp dir outside the repo).
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept), `scrubSystemPrompts` (prompt text → `{{system}}` in every JSONL), and `scrubRequestHeaders` (the remaining header bulk → `{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header RFC](../../../docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record/refresh fixture write-back, the per-header-class pin (`system-prompt.golden.md` plus the JSONL's full tool schemas) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt-scrubbed, non-pinning fixtures fully header-scrubbed). Must be called at vitest collection time.
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header RFC](../../../docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, the per-header-class pin (`system-prompt.golden.md` plus `tool-schemas.golden.json`) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt/schema-scrubbed, non-pinning fixtures fully header-scrubbed). Must be called at vitest collection time.
A consuming `*.snapshot.ts` is the scenario table plus one factory call:
@@ -21,7 +21,7 @@ const SCENARIOS: Scenario[] = [
defineAcpSnapshotSuite({
agent: { // absolute paths, resolved from the suite's own location
binScript: fileURLToPath(new URL('../../../packages/ui/acp-agent/src/bin.ts', import.meta.url)),
binScript: fileURLToPath(new URL('../../../packages/examples/acp-demo/src/bin.ts', import.meta.url)),
configPath: fileURLToPath(new URL('../cordis.yml', import.meta.url)),
tsconfigPath: fileURLToPath(new URL('../../../tsconfig.json', import.meta.url)),
},
@@ -35,8 +35,17 @@ defineAcpSnapshotSuite({
})
```
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode scenarios are the template. Each pinning directory's generated `system-prompt.golden.md` is the reviewable snapshot of the normalized composed prompt; `session.jsonl` stores `"system":"{{system}}"` while retaining the complete tool list.
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode and filesystem scenarios are templates. Each pinning directory stores the normalized composed prompt in generated `system-prompt.golden.md` and the initial schemas plus schema deltas in generated `tool-schemas.golden.json`; `session.jsonl` stores `"system":"{{system}}","tools":"{{tools}}"` while retaining config, reason, and any model-visible prefix.
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config RFC](../../../docs/rfc/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log goldens, and each pin's Markdown prompt snapshot from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
Examples use a `cordis.snapshot.yml` overlay with [`dsh-llm-replay`](../llm-replay/README.md). Recording calls the live model and updates model fixtures; keyless refresh replays those fixtures and updates derived stdout, session-log, prompt, and tool-schema snapshots. See the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
Constraints: `suite.ts` imports vitest, so the package is importable only inside a vitest run (the harness and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the harness speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug). Session config options are scriptable too: the `setConfigOption` step switches a knob over `session/set_config_option`, and `setConfigOptionExpectError` asserts the bridge rejects an unknown id or out-of-vocabulary value (the error frame stays in the transcript).
`suite.ts` imports Vitest, so use this package only inside a Vitest run. The ACP-specific script queues permission answers by stable option kind and maps them to current option ids; a missing answer cancels, while an unavailable kind fails the scenario after cancelling the agent request. It can also set session config options or assert that unknown ids and values are rejected in the transcript.
## Model Experience
None, as this test-only harness records, normalizes, and compares ACP transcripts without changing the agent's assembled model request.
## Known Limitations and Deferred Work
- **Session harvest is JSONL-only** — `runScenario` collects persisted `.jsonl` logs, so an example composed over the SQLite persistence backend has no snapshot path.
- **The subprocess boots the unbuilt tsx/Loader path only** — the built-bin artifact is guarded by the separate `built-bin` e2e smokes, never by this tier.

4
packages/support/acp-snapshot/package.json
View File

@@ -27,9 +27,9 @@
"vitest": "^4.1.8"
},
"peerDependencies": {
"cordis": "^4.0.0-rc.6"
"cordis": "^4.0.0-rc.7"
},
"devDependencies": {
"cordis": "^4.0.0-rc.6"
"cordis": "^4.0.0-rc.7"
}
}

95
packages/support/acp-snapshot/src/harness.ts
View File

@@ -1,18 +1,8 @@
/**
* Shared subprocess harness for ACP snapshot suites. A library module driven by
* the suite factory in ./suite.ts (and directly by harness-level specs); each
* example's `*.snapshot.ts` names its own agent-under-test paths.
*
* It boots the REAL agent bin subprocess via the cordis Loader (so the
* export-shape bug class stays guarded — see docs/postmortem/0001), drives it
* over real ACP JSON-RPC stdio with a deterministic input script, tees raw
* stdout (for the golden + a purity check) into an SDK `ClientSideConnection`,
* and — in record mode — harvests the persisted session JSONL after a graceful
* shutdown flush. The pure normalizers in ./normalize.ts turn the captured
* stdout frames and the session-log events into stable, snapshot-able text.
*
* See docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
*
* Shared ACP snapshot subprocess harness. It boots the real agent bin through the Cordis
* loader, drives deterministic ACP JSON-RPC over stdio, captures protocol-pure stdout, and
* harvests persisted session logs after graceful shutdown. Normalization stays in
* `normalize.ts`; suite registration stays in `suite.ts`.
* @module @deepseek-ai/dsh-acp-snapshot/harness
*/
@@ -47,7 +37,7 @@ const tsxLoader = fileURLToPath(import.meta.resolve('tsx'))
* them from its own `import.meta.url`.
*/
export interface AgentUnderTest {
/** The agent bin entry (e.g. `packages/ui/acp-agent/src/bin.ts`), run unbuilt via tsx. */
/** The agent bin entry (e.g. `packages/examples/acp-demo/src/bin.ts`), run unbuilt via tsx. */
binScript: string
/**
* The example's live `cordis.yml`. Under `DSH_SNAPSHOT=replay` the bin swaps
@@ -66,16 +56,10 @@ export interface AgentUnderTest {
}
/**
* One step of a scenario's deterministic input script (`input.json`). The
* harness interprets these in order. `newSession` captures the server-issued
* (random) session id into a `{{sessionId}}` variable that later steps
* reference, since a committed file cannot know the id in advance.
*
* `promptAndCancel` sends a prompt WITHOUT awaiting its response, waits until
* the client observes the first streamed `agent_message_chunk` (so the emitted
* frames deterministically precede the cancellation), then cancels the turn —
* the only way to exercise a cancel deterministically (a plain `prompt` step
* awaits the response, which a cancel/hang scenario would block on forever).
* One step of a scenario's deterministic input script (`input.json`). The harness interprets
* these in order. `newSession` captures the server-issued (random) session id into a
* `{{sessionId}}` variable that later steps reference. `promptAndCancel` sends without awaiting,
* waits for the first streamed message, then cancels, making transcript order deterministic.
*/
export type InputStep =
| { op: 'initialize'; terminalOutput?: boolean }
@@ -92,16 +76,9 @@ export type InputStep =
export interface InputScript {
steps: InputStep[]
/**
* Ordered answers for the agent's `session/request_permission` round-trips,
* consumed FIFO — the Nth request gets the Nth answer. Each answer selects
* by option KIND: option ids are agent-issued randoms a committed script
* cannot know, while kinds are the ACP-stable vocabulary, so the client maps
* kind → the offered `optionId` at answer time. A request beyond the queue
* (or with no queue at all) is answered `cancelled` — the stub behavior a
* scenario without approvals relies on. A scripted kind the request does
* not offer REJECTS the run: the scenario scripted an impossible click,
* and {@link runScenario} throws once the in-flight step settles (the
* agent itself just sees `cancelled`, so it cannot absorb the bug).
* FIFO permission answers selected by stable option kind; the harness maps each kind to the
* agent-issued option id. Exhaustion cancels, while a kind the agent did not offer fails the
* scenario.
*/
permissionAnswers?: PermissionAnswer[]
}
@@ -204,8 +181,6 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
const stderrChunks: string[] = []
try {
// Seed the workspace if the scenario ships one (a file the agent reads/edits).
// Copied into the temp cwd so the agent's bash tools see it; the goldens
// normalize the cwd, so the seeded paths stay stable across runs.
if (opts.workspaceDir !== undefined && existsSync(opts.workspaceDir)) {
await cp(opts.workspaceDir, cwd, { recursive: true })
}
@@ -233,10 +208,8 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
child.stderr.setEncoding('utf8')
child.stderr.on('data', (c: string) => stderrChunks.push(c))
// Tee raw stdout: accumulate the bytes for the golden + purity check, and ALSO
// feed the same bytes to the SDK client through a passthrough. Buffer the raw
// bytes (not per-chunk utf8 strings) and decode once at the end, so a
// multibyte sequence split across two 'data' events can't corrupt the golden.
// Tee the same raw bytes to the golden and SDK client. Decode once at the end so a UTF-8
// sequence split across stream chunks cannot corrupt the transcript.
const passthrough = new Readable({ read() {} })
child.stdout.on('data', (buf: Buffer) => {
rawBuffers.push(buf)
@@ -259,13 +232,8 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
// Permission answers are consumed FIFO across the whole run; exhaustion
// falls back to `cancelled` so approval-free scenarios keep the plain stub.
const permissionQueue = [...input.permissionAnswers ?? []]
// A scenario bug detected inside a client callback (a scripted permission
// kind the agent never offered). It cannot fail the run from in there: a
// callback throw only becomes a JSON-RPC error RESPONSE to the agent, and
// a tolerant agent treats that as a denial and carries on — the run (or
// worse, a record) would absorb the impossible click silently. So the
// callback answers `cancelled` (a well-defined path for the agent),
// captures the error here, and the step loop fails the run on it.
// A callback throw would become only an RPC error the agent could absorb. Record an
// impossible permission choice here, answer cancelled, and fail the outer scenario.
let scriptError: Error | undefined
const makeClient = (_agent: AcpAgent): Client => ({
sessionUpdate(params: SessionNotification): Promise<void> {
@@ -361,10 +329,8 @@ async function runStep(
return
}
case 'newSessionExpectError': {
// The bridge rejects a session/new that widens the workspace scope
// (non-empty additionalDirectories / mcpServers — unimplemented). The SDK
// surfaces that as a rejected RPC; swallow it so the run completes and the
// error frame is captured in the transcript.
// The bridge rejects a session/new that widens the workspace scope (non-empty
// additionalDirectories / mcpServers — unimplemented).
await client.newSession({
cwd,
mcpServers: [],
@@ -384,10 +350,8 @@ async function runStep(
case 'promptExpectError': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: promptExpectError before newSession')
// The model fails this turn (a recorded provider error), so the bridge
// answers the prompt with a JSON-RPC error and the SDK rejects. That
// rejection IS the expected editor experience — swallow it so the run
// completes and the stdout transcript (the error frame) is captured.
// The model fails this turn (a recorded provider error), so the bridge answers the prompt
// with a JSON-RPC error and the SDK rejects.
await client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
.then(() => { throw new Error('snapshot-harness: expected the prompt to fail but it succeeded') },
() => { /* expected: the turn failed and the bridge returned an error */ })
@@ -396,13 +360,8 @@ async function runStep(
case 'promptAndCancel': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: promptAndCancel before newSession')
// Dispatch the prompt WITHOUT awaiting (a hang fixture never resolves on
// its own). To pin frame order deterministically, wait until the client
// has OBSERVED the hang's streamed agent_message_chunk before cancelling —
// so those update frames always precede the cancelled prompt response in
// the transcript (without this, the late chunk and the response race).
// Then cancel and await the prompt, which the bridge settles as
// `cancelled` once the abort propagates.
// A hang fixture never resolves alone. Wait for its streamed chunk before cancellation
// so updates deterministically precede the cancelled prompt response.
const promptDone = client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
await waitForUpdate(u => u.sessionUpdate === 'agent_message_chunk')
await client.cancel({ sessionId })
@@ -488,14 +447,8 @@ async function harvestSessionLogs(root: string): Promise<HarvestedLog[]> {
})
}
}
// Primary (no parentSession) first, then children by ascending createdAt. A
// scenario has exactly one top-level session. In the synchronous cut sibling
// children are created strictly sequentially, so their createdAt values are
// strictly ordered; the recordedId tiebreak only keeps a degenerate
// same-millisecond collision (unreachable here) deterministic. This harvest
// order must match the replay load order in dsh-llm-replay's loadSessionScripts
// so session.<n>.jsonl maps to the same child on record and replay — replay
// re-sorts childFiles by the same key, so the two stay consistent.
// Match replay fixture assignment: primary first, then children by creation time, with id as
// a deterministic collision tiebreaker.
logs.sort((a, b) => {
const ap = a.parentSession === undefined ? 0 : 1
const bp = b.parentSession === undefined ? 0 : 1

17
packages/support/acp-snapshot/src/index.ts
View File

@@ -1,17 +1,7 @@
/**
* ACP snapshot suite kit — the shared machinery behind the keyless snapshot
* tier (`pnpm run test:snapshot`). Three layers, composable per example:
* the subprocess scenario harness ({@link runScenario}), the pure golden
* normalizers ({@link normalizeStdout} / {@link normalizeSessionLog} /
* {@link scrubRequestHeaders} / {@link scrubSystemPrompts}), and the suite factory
* ({@link defineAcpSnapshotSuite}) that registers a scenario table as a full
* describe/it tree. An example's `*.snapshot.ts` supplies only its
* {@link AgentUnderTest} paths, its snapshots directory, and its
* {@link Scenario} table.
*
* NOTE: ./suite.ts imports vitest, so this package is importable only inside a
* vitest run — a support-tier constraint stated in the README.
*
* ACP snapshot suite kit: subprocess scenario harness, pure golden normalizers, and the Vitest
* suite factory behind `pnpm run test:snapshot`. Because this entry exports `suite.ts`, importing
* it requires a Vitest run.
* @module @deepseek-ai/dsh-acp-snapshot
*/
@@ -30,6 +20,7 @@ export {
normalizeStdout,
scrubRequestHeaders,
scrubSystemPrompts,
scrubToolSchemas,
type NormalizeContext,
} from './normalize.ts'
export {

80
packages/support/acp-snapshot/src/normalize.ts
View File

@@ -1,29 +1,8 @@
/**
* Pure normalizers for the ACP snapshot goldens. They replace the
* non-deterministic values in the two captured surfaces — the stdout JSON-RPC
* transcript and the persisted session JSONL — with stable tokens, so a golden
* compare reflects behavior, not run-to-run noise. Kept dependency-free and
* side-effect-free so they unit-test trivially.
*
* Scrubbed: `randomUUID()` session ids → `{{sessionId}}`; the temp `mkdtemp`
* cwd → `{{cwd}}` (it appears in terminal-card `_meta` and the log header);
* JSON-RPC request `id` → a stable per-transcript sequence; the log's per-event
* `time` (epoch ms) and header `createdAt` → 0; a `hook/result` event's
* `durationMs` (wall-clock hook runtime) → 0. NOT scrubbed: the log's `seq`
* (deterministic — `seq = log.length`, part of the event-log contract).
*
* Separate, composable normalizers keep bulky request-header content out of
* session fixtures. {@link scrubSystemPrompts} replaces the composed system
* prompt in EVERY fixture; {@link scrubRequestHeaders} additionally replaces
* tool schemas and the session prefix outside each suite's header-pinning
* scenario. They are deliberately NOT folded into
* {@link normalizeSessionLog}: the suite factory composes the right scrub for
* each scenario and snapshots the pin's actual prompt as Markdown (see the
* pinned-header RFC,
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
*
* See docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
*
* Pure ACP transcript and session-log normalizers. They scrub session ids, temp cwd, RPC ids,
* timestamps, and hook duration while preserving deterministic event sequence numbers.
* Request-header scrubbers stay composable so one scenario per header class can pin prompt and
* tool-schema sidecars while retaining any model-visible prefix in the session log.
* @module @deepseek-ai/dsh-acp-snapshot/normalize
*/
@@ -81,12 +60,10 @@ function scrubValue(value: unknown, ctx: NormalizeContext): unknown {
}
/**
* Normalize a raw stdout transcript (newline-delimited JSON-RPC frames) into a
* stable golden in the SAME shape as the wire: one compact JSON frame per line
* (NDJSON), with the JSON-RPC `id` rewritten to a per-transcript sequence
* (1, 2, 3, …) and all volatile strings scrubbed. Throws if any non-empty line
* is not valid JSON — that doubles as the stdout-purity check (no logger leaked
* onto the protocol).
* Normalize a raw stdout transcript (newline-delimited JSON-RPC frames) into a stable golden
* in the same shape as the wire: one compact JSON frame per line (NDJSON), with the JSON-RPC
* `id` rewritten to a per-transcript sequence (1, 2, 3, …) and all volatile strings scrubbed.
* Invalid JSON throws, doubling as a protocol-stdout purity check.
*
* @param rawStdout The captured stdout bytes, decoded utf8.
* @param ctx The run's volatile values to scrub.
@@ -159,7 +136,21 @@ export function normalizeSessionLog(rawLog: string, ctx: NormalizeContext): stri
* @returns The JSONL with system-prompt content tokenized.
*/
export function scrubSystemPrompts(rawLog: string): string {
return scrubHeaderContent(rawLog, false)
return scrubHeaderContent(rawLog, { system: true })
}
/**
* Replace tool schemas in request headers and header deltas with `{{tools}}`
* tokens while retaining field presence, tool names, and delta structure.
* System prompts and session-prefix messages stay verbatim so pinning fixtures
* can move only schema bulk into their dedicated JSON sidecar. Lines without a
* tool payload pass through byte-for-byte; the transform is idempotent.
*
* @param rawLog The raw session `.jsonl` content.
* @returns The JSONL with tool-schema content tokenized.
*/
export function scrubToolSchemas(rawLog: string): string {
return scrubHeaderContent(rawLog, { tools: true })
}
/**
@@ -174,11 +165,18 @@ export function scrubSystemPrompts(rawLog: string): string {
* @returns The JSONL with all header bulk tokenized, other lines byte-identical.
*/
export function scrubRequestHeaders(rawLog: string): string {
return scrubHeaderContent(rawLog, true)
return scrubHeaderContent(rawLog, { system: true, tools: true, prefix: true })
}
/** Transform header content, optionally including tool schemas and the session prefix. */
function scrubHeaderContent(rawLog: string, scrubToolsAndPrefix: boolean): string {
/** Which independent request-header payloads a scrubber replaces. */
interface HeaderScrubOptions {
system?: boolean
tools?: boolean
prefix?: boolean
}
/** Transform the selected request-header payloads. */
function scrubHeaderContent(rawLog: string, options: HeaderScrubOptions): string {
const lines = rawLog.split('\n')
const out = lines.map((line) => {
if (line.trim().length === 0) return line
@@ -189,9 +187,9 @@ function scrubHeaderContent(rawLog: string, scrubToolsAndPrefix: boolean): strin
const header = data.header as Record<string, unknown> | null | undefined
if (header === null || typeof header !== 'object') return line
let touched = false
if ('system' in header) { header.system = SYSTEM; touched = true }
if (scrubToolsAndPrefix && 'tools' in header) { header.tools = TOOLS; touched = true }
if (scrubToolsAndPrefix && Array.isArray(header.messagePrefix)) {
if (options.system === true && 'system' in header) { header.system = SYSTEM; touched = true }
if (options.tools === true && 'tools' in header) { header.tools = TOOLS; touched = true }
if (options.prefix === true && Array.isArray(header.messagePrefix)) {
header.messagePrefix = header.messagePrefix.map(() => MESSAGE_PREFIX)
touched = true
}
@@ -200,16 +198,16 @@ function scrubHeaderContent(rawLog: string, scrubToolsAndPrefix: boolean): strin
if (record.type === 'request/header-delta') {
let touched = false
const system = data.system as Record<string, unknown> | null | undefined
if (system !== null && typeof system === 'object' && Array.isArray(system.insert)) {
if (options.system === true && system !== null && typeof system === 'object' && Array.isArray(system.insert)) {
system.insert = system.insert.map(() => SYSTEM)
touched = true
}
const tools = data.tools as Record<string, unknown> | null | undefined
if (scrubToolsAndPrefix && tools !== null && typeof tools === 'object') {
if (options.tools === true && tools !== null && typeof tools === 'object') {
if (Array.isArray(tools.added)) { tools.added = tools.added.map(scrubToolSchema); touched = true }
if (Array.isArray(tools.changed)) { tools.changed = tools.changed.map(scrubToolSchema); touched = true }
}
if (scrubToolsAndPrefix && Array.isArray(data.messagePrefix)) {
if (options.prefix === true && Array.isArray(data.messagePrefix)) {
data.messagePrefix = data.messagePrefix.map(() => MESSAGE_PREFIX)
touched = true
}

312
packages/support/acp-snapshot/src/suite.ts
View File

@@ -1,33 +1,12 @@
/**
* The ACP snapshot suite factory (REPLAY by default, keyless). A suite is a
* scenario table plus a snapshots directory: each scenario under
* `<snapshotsDir>/<name>/` ships an `input.json` (the client stdin script) and
* a `session.jsonl` fixture; replay boots the real agent subprocess
* (./harness.ts), drives it, and diffs the normalized stdout transcript
* against the committed `stdout.golden.jsonl`. For model scenarios it ALSO
* checks the re-persisted session log — against the `session.jsonl` fixture
* itself, not a separate golden: the fixture doubles as the replay source
* (recorded scenarios) and the expected produced log (both sides normalized
* before comparing).
*
* Request-header content is pinned by exactly ONE scenario per HEADER CLASS —
* scenarios that boot the same config compose the same header. Every JSONL
* fixture scrubs the system prompt to `{{system}}`; each class's pinning
* scenario stores the readable prompt in `system-prompt.golden.md` and keeps its full
* tool schemas in `session.jsonl`, while every other fixture also scrubs tools
* to `{{tools}}`. A per-run uniformity guard compares both artifacts against
* every live header and forbids unrepresented header deltas (see the
* pinned-header RFC,
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
*
* `pnpm run test:snapshot:record` (DSH_SNAPSHOT=record + -u) re-records the
* `session.jsonl` fixtures against the real API and refreshes the stdout golden
* in one pass. `pnpm run test:snapshot:refresh` (DSH_SNAPSHOT=refresh) instead
* replays the committed model scripts keylessly and writes the current stdout
* + persisted-log goldens back without calling a live LLM. The caller resolves
* that env into {@link SnapshotSuiteOptions} (env reading stays at the suite
* edge, not in this library).
* Keyless-by-default ACP snapshot suite factory. Each scenario drives the real subprocess and
* compares normalized stdout; comparable session fixtures are both replay input and expected
* output. Record mode refreshes reproducible model scenarios from the live API, while refresh
* mode replays committed scripts and rewrites derived artifacts without a key.
*
* Exactly one scenario per header-composition class pins the system prompt and tool schemas in
* dedicated sidecars. Every live header is checked against that pin, so session-dependent
* composition must declare a separate class instead of escaping coverage.
* @module @deepseek-ai/dsh-acp-snapshot/suite
*/
@@ -42,11 +21,18 @@ import {
normalizeStdout,
scrubRequestHeaders,
scrubSystemPrompts,
scrubToolSchemas,
} from './normalize.ts'
/** The readable system-prompt snapshot beside each header-pinning fixture. */
const SYSTEM_PROMPT_SNAPSHOT = 'system-prompt.golden.md'
/** The structured tool-schema snapshot beside each header-pinning fixture. */
const TOOL_SCHEMAS_SNAPSHOT = 'tool-schemas.golden.json'
/** Stable session-log token standing in for the sidecar's initial schemas. */
const TOOLS_TOKEN = '{{tools}}'
/** A snapshot scenario and how its fixtures are produced. */
export interface Scenario {
name: string
@@ -89,20 +75,8 @@ export interface Scenario {
*/
childSessions?: number
/**
* Whether THIS scenario pins its header class's model-facing request-header
* content. Its actual composed prompt is maintained as a readable
* `system-prompt.golden.md`; its JSONL keeps full tool schemas but stores the prompt
* as `{{system}}`. Every other scenario of the class stores tools as
* `{{tools}}` too ({@link scrubRequestHeaders}). A prompt or tool-schema
* change therefore shows up in one focused artifact per class, not every
* session fixture. One pin per class suffices because
* header composition is class-uniform (parent, spawn child, and fork child
* all compose the same prompt-modulo-cwd and the same tools) — and that
* premise is ASSERTED, not assumed: every non-pinning run's live headers
* must equal its class's pinned fixture's (normalized), so a
* session-dependent header (say, a restricted subagent toolset) fails loud
* until it gets its own pinning scenario.
* Defaults to false.
* Whether this scenario is its header class's sole request-header pin. Dedicated sidecars own
* the prompt and tool schemas, while every classmate is checked for equality.
*/
pinsHeader?: boolean
/**
@@ -164,17 +138,9 @@ export function childFixturePaths(dir: string, childSessions: number): string[]
}
/**
* Derive the {@link NormalizeContext} for a `session.jsonl` fixture from its own
* header line (`{ type: 'session', id, cwd }`). A committed fixture carries the
* session id and cwd of the run that harvested it — different from the live
* replay run — so normalizing it against the live run's ctx would leave those
* recorded values unscrubbed. Reading them from the header scrubs the fixture's
* own id/cwd to the same `{{sessionId}}`/`{{cwd}}` tokens the replay output gets.
* An authored fixture whose header is already normalized (`id:'{{sessionId}}'`,
* `cwd:'{{cwd}}'`) yields those tokens as the volatile values, so scrubbing them
* is an idempotent no-op. A header with no `cwd` falls back to a sentinel that
* cannot occur in a log (NOT `''`, which `String.split` would match on every
* character boundary and corrupt the output).
* Derive normalization values from a fixture's own session header. Recorded ids and cwd differ
* from the live replay run; the non-empty sentinel for missing cwd avoids accidental empty-
* string replacement.
*
* @param fixture The committed `session.jsonl` content.
* @returns The fixture's own volatile values, ready for {@link normalizeSessionLog}.
@@ -225,6 +191,98 @@ export function normalizedSystemPrompts(rawLog: string, ctx: NormalizeContext):
})
}
/**
* The normalized tool-schema arrays carried by request headers in a session
* JSONL, in log order. Headers without an array-valued tools field are omitted
* so callers can assert one schema set per header explicitly.
*
* @param rawLog The session `.jsonl` content to inspect.
* @param ctx The volatile values of the run that produced it.
* @returns The normalized initial tool-schema arrays, in header order.
*/
export function normalizedToolSchemas(rawLog: string, ctx: NormalizeContext): unknown[][] {
return normalizedHeaders(rawLog, ctx).flatMap((header) => {
if (header === null || typeof header !== 'object') return []
const tools = (header as { tools?: unknown }).tools
return Array.isArray(tools) ? [tools] : []
})
}
/**
* Extract normalized tool-schema edits from request-header deltas in log order.
* Deltas without an object-valued tools edit are omitted; their remaining
* structure stays pinned in the session JSONL.
*
* @param rawLog The session `.jsonl` content to inspect.
* @param ctx The volatile values of the run that produced it.
* @returns The normalized tool-schema edits, in event order.
*/
export function normalizedToolSchemaDeltas(rawLog: string, ctx: NormalizeContext): unknown[] {
return normalizeSessionLog(rawLog, ctx)
.split('\n')
.filter(line => line.trim().length > 0)
.map(line => JSON.parse(line) as { type?: unknown; data?: { tools?: unknown } })
.filter(record => record.type === 'request/header-delta')
.flatMap((record) => {
const tools = record.data?.tools
return tools !== null && typeof tools === 'object' && !Array.isArray(tools) ? [tools] : []
})
}
/** The structured contents of a tool-schema sidecar. */
export interface ToolSchemasSnapshot {
/** The complete tool schemas from the pinned request header. */
initial: unknown[]
/** Complete tool-schema edits from subsequent request-header deltas. */
deltas: unknown[]
}
/**
* Render tool schemas and later schema edits as canonical, readable JSON.
*
* @param initial The pinned request header's complete tool schemas.
* @param deltas Complete tool-schema edits from request-header deltas.
* @returns A pretty-printed JSON snapshot ending in one newline.
*/
export function formatToolSchemasSnapshot(initial: readonly unknown[], deltas: readonly unknown[] = []): string {
return `${JSON.stringify({ initial, deltas }, null, 2)}\n`
}
/**
* Parse and validate the stable top-level shape of a tool-schema sidecar.
*
* @param snapshot The JSON sidecar text.
* @returns Its initial schemas and schema deltas.
*/
export function parseToolSchemasSnapshot(snapshot: string): ToolSchemasSnapshot {
const parsed = JSON.parse(snapshot) as unknown
if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) {
throw new Error('acp-snapshot: tool-schema snapshot must be an object')
}
const { initial, deltas } = parsed as { initial?: unknown; deltas?: unknown }
if (!Array.isArray(initial) || !Array.isArray(deltas)) {
throw new Error('acp-snapshot: tool-schema snapshot must carry array-valued initial and deltas fields')
}
return { initial, deltas }
}
/**
* Restore a sidecar's initial schemas into a tokenized pinned header.
*
* @param header The parsed request header carrying `tools: "{{tools}}"`.
* @param snapshot The parsed tool-schema sidecar.
* @returns A copy of the header with its complete initial schemas restored.
*/
export function restorePinnedToolSchemas(header: unknown, snapshot: ToolSchemasSnapshot): unknown {
if (header === null || typeof header !== 'object' || Array.isArray(header)) {
throw new Error('acp-snapshot: pinned request header must be an object')
}
if ((header as { tools?: unknown }).tools !== TOOLS_TOKEN) {
throw new Error(`acp-snapshot: pinned request header tools must equal ${TOOLS_TOKEN}`)
}
return { ...header, tools: snapshot.initial }
}
/** One normalized system-prompt edit carried by a `request/header-delta`. */
export interface SystemPromptDeltaSnapshot {
/** How many leading lines remain from the prior prompt. */
@@ -315,6 +373,27 @@ function parseJsonlRecords(text: string): Record<string, unknown>[] {
.map(line => JSON.parse(line) as Record<string, unknown>)
}
/**
* Find tool calls whose structured result reports `UNKNOWN_TOOL`.
*
* Snapshot refresh must not turn a missing registration into accepted behavior;
* intentional unknown-tool behavior belongs in a focused unit or e2e test.
*
* @param rawLog The session JSONL to inspect.
* @returns The failing call ids in log order, using a diagnostic placeholder when absent.
*/
export function unknownToolCallIds(rawLog: string): string[] {
return parseJsonlRecords(rawLog).flatMap((record) => {
if (record.type !== 'tool/result') return []
const data = record.data
if (data === null || typeof data !== 'object') return []
const { callId, error } = data as { callId?: unknown; error?: unknown }
if (error === null || typeof error !== 'object') return []
if ((error as { code?: unknown }).code !== 'UNKNOWN_TOOL') return []
return [typeof callId === 'string' ? callId : '<missing callId>']
})
}
/**
* Build the cross-log id/cwd replacements used by refresh write-back.
*
@@ -419,10 +498,8 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
for (const scenario of scenarios) {
describe(`snapshot: ${scenario.name}`, () => {
// In RECORD mode, only re-run the `recorded` (live-API) scenarios; the
// `authored` ones (sidecar-driven errors/cancel) are never re-recorded.
// REFRESH mode is replay-backed and deterministic, so it runs every
// scenario and rewrites the comparable fixtures from that replay run.
// In RECORD mode, only re-run the `recorded` (live-API) scenarios; the `authored` ones
// (sidecar-driven errors/cancel) are never re-recorded.
it.skipIf(RECORDING && !scenario.recorded)('matches the goldens', async () => {
const dir = join(snapshotsDir, scenario.name)
const input = JSON.parse(await readFile(join(dir, 'input.json'), 'utf8')) as InputScript
@@ -444,10 +521,14 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
...scenario.configPath !== undefined ? { configPath: scenario.configPath } : {},
})
// Scrub every volatile id the run produced: the ACP server-issued session
// id plus every harvested log's recorded id (a subagent child id never
// surfaces over ACP, but it appears in the child's own log header). The
// normalizer's UUID catch-all covers any we don't enumerate.
for (const log of result.sessionLogs) {
expect(unknownToolCallIds(log.content), `session ${log.id}: snapshot scenarios must not accept UNKNOWN_TOOL`)
.toEqual([])
}
// Scrub every volatile id the run produced: the ACP server-issued session id plus every
// harvested log's recorded id (a subagent child id never surfaces over ACP, but it
// appears in the child's own log header).
const ctx: NormalizeContext = {
sessionIds: [
...result.sessionId !== undefined ? [result.sessionId] : [],
@@ -456,17 +537,10 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
cwd: result.cwd,
}
// RECORD mode (recorded model scenarios only): persist the freshly-harvested
// live logs back to their fixtures. REFRESH mode does the same from a
// keyless replay run for every comparable log, including authored
// scenarios that live record deliberately skips. The primary goes to
// session.jsonl, each child to session.<n>.jsonl in harvest order. A
// Every fixture is written with its system prompt scrubbed. A pinning
// scenario keeps the remaining header content (notably tool schemas);
// every other scenario scrubs that bulk too. Record/refresh therefore
// cannot smuggle prompt text back into JSONL or duplicate schemas.
// Record writes live model fixtures; keyless refresh writes every comparable replayed
// fixture. Pinning JSONL keeps prefixes but moves prompts and schemas into sidecars.
const scrub = scenario.pinsHeader === true
? scrubSystemPrompts
? (log: string): string => scrubToolSchemas(scrubSystemPrompts(log))
: scrubRequestHeaders
const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)]
const existingFixtures = REFRESHING
@@ -503,6 +577,18 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
normalizedSystemPromptDeltas(primary.content, ctx),
)
await writeFile(join(dir, SYSTEM_PROMPT_SNAPSHOT), snapshot)
const schemaSets = result.sessionLogs.flatMap(log => normalizedToolSchemas(log.content, ctx))
expect(schemaSets.length, `${mode} produced no tool schemas to snapshot`).toBeGreaterThan(0)
const initialSchemaSnapshot = formatToolSchemasSnapshot(schemaSets[0] as unknown[])
for (const schemas of schemaSets) {
expect(formatToolSchemasSnapshot(schemas), 'the pinning run produced divergent tool schemas')
.toEqual(initialSchemaSnapshot)
}
await writeFile(join(dir, TOOL_SCHEMAS_SNAPSHOT), formatToolSchemasSnapshot(
schemaSets[0] as unknown[],
normalizedToolSchemaDeltas(primary.content, ctx),
))
}
}
@@ -515,14 +601,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
// A model turn always produces a log worth comparing; a hook scenario can
// produce one without a model turn (a `rejected` turn carrying `hook/*`).
if (comparesLog) {
// The harvested logs (primary-first) must match their committed fixtures
// 1:1. Each side passes through normalizeSessionLog, scrubbed against ITS
// OWN volatile values — the live run's via `ctx`, the committed fixture's
// via its own header (a committed file cannot share the live run's ids).
// Both sides pass through the scenario's idempotent scrub: every live
// prompt becomes the fixture's `{{system}}`; non-pinning scenarios
// additionally tokenize tools/prefix. The dedicated header guard below
// compares those omitted values against their class's pin artifacts.
// The harvested logs (primary-first) must match their committed fixtures 1:1.
expect(result.sessionLogs.length, 'this scenario must persist a session log').toBe(childSessions + 1)
for (let i = 0; i < fixtureFiles.length; i++) {
const harvested = scrub((result.sessionLogs[i] as HarvestedLog).content)
@@ -532,11 +611,8 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
}
}
// Header-uniformity guard: every live header in a class must equal the
// class pin split across its JSONL header (system token + real tools)
// and readable Markdown prompt. A pinning scenario may carry its
// declared header deltas; their prompt edits live in the Markdown
// golden while JSONL retains the tokenized edit structure.
// Header-uniformity guard: every live header in a class must equal the class pin split
// across tokenized JSONL plus readable prompt and structured schema sidecars.
/* v8 ignore next -- construction guarantees the pin exists; a miss would fail the one-header assertion loudly. */
const pinningScenario = pinningByClass.get(classOf(scenario)) ?? scenario
const pinningDir = join(snapshotsDir, pinningScenario.name)
@@ -544,8 +620,11 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
const pinned = normalizedHeaders(pinnedFixture, fixtureContext(pinnedFixture))
const promptSnapshot = await readFile(join(pinningDir, SYSTEM_PROMPT_SNAPSHOT), 'utf8')
const initialPromptSnapshot = initialSystemPromptSnapshot(promptSnapshot)
const toolSchemasSnapshot = await readFile(join(pinningDir, TOOL_SCHEMAS_SNAPSHOT), 'utf8')
const toolSchemas = parseToolSchemasSnapshot(toolSchemasSnapshot)
expect(pinned.length, `the pinning fixture (${pinningScenario.name}) must carry exactly one request/header`)
.toBe(1)
const pinnedHeader = restorePinnedToolSchemas(pinned[0], toolSchemas)
for (const [logIndex, log] of result.sessionLogs.entries()) {
const expectedDeltas = scenario.pinsHeader === true && logIndex === 0
? scenario.expectedHeaderDeltas ?? 0
@@ -554,11 +633,14 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
.toBe(expectedDeltas)
const headers = normalizedHeaders(scrubSystemPrompts(log.content), ctx)
const prompts = normalizedSystemPrompts(log.content, ctx)
const schemaSets = normalizedToolSchemas(log.content, ctx)
expect(prompts.length, `session ${log.id}: every request/header must carry a string system prompt`)
.toBe(headers.length)
expect(schemaSets.length, `session ${log.id}: every request/header must carry an array-valued tools field`)
.toBe(headers.length)
for (const [k, header] of headers.entries()) {
expect(header, `session ${log.id}: request/header #${k + 1} diverged from the pinned (${pinningScenario.name}) header`)
.toEqual(pinned[0])
.toEqual(pinnedHeader)
expect(formatSystemPromptSnapshot(prompts[k] as string), `session ${log.id}: initial system prompt #${k + 1} diverged from ${pinningScenario.name}/${SYSTEM_PROMPT_SNAPSHOT}`)
.toEqual(initialPromptSnapshot)
}
@@ -568,6 +650,11 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
normalizedSystemPromptDeltas(log.content, ctx),
), `session ${log.id}: system-prompt deltas diverged from ${pinningScenario.name}/${SYSTEM_PROMPT_SNAPSHOT}`)
.toEqual(promptSnapshot)
expect(formatToolSchemasSnapshot(
schemaSets[0] as unknown[],
normalizedToolSchemaDeltas(log.content, ctx),
), `session ${log.id}: tool-schema deltas diverged from ${pinningScenario.name}/${TOOL_SCHEMAS_SNAPSHOT}`)
.toEqual(toolSchemasSnapshot)
}
}
})
@@ -586,18 +673,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
})
it('every registered scenario has its required fixture files', () => {
// Every scenario has an input script and an stdout golden. EVERY scenario
// also needs `session.jsonl`: the suite boots `llm-replay` with that path
// as the replay source for ALL scenarios (the factory passes
// `fixtureFile: <dir>/session.jsonl` unconditionally), and `loadReplayScript`
// throws "fixture not found" when it is absent and no override replaces it.
// A no-model scenario ships a header-only `session.jsonl` (it derives to an
// empty script — no model call is made); a model scenario's fixture also
// doubles as the expected-log artifact the run is diffed against. The
// `replay.override.json` sidecar is matched BOTH ways against the table's
// `overridden` flag: required when set, forbidden when not — the harness
// forwards the file purely on existence, so an unregistered stray sidecar
// would silently replace the derived script.
// Every scenario has an input script and an stdout golden.
for (const { name, overridden, childSessions, pinsHeader } of scenarios) {
const dir = join(snapshotsDir, name)
expect(existsSync(join(dir, 'input.json')), `${name}/input.json`).toBe(true)
@@ -607,6 +683,8 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
.toBe(overridden === true)
expect(existsSync(join(dir, SYSTEM_PROMPT_SNAPSHOT)), `${name}/${SYSTEM_PROMPT_SNAPSHOT} presence must match \`pinsHeader\``)
.toBe(pinsHeader === true)
expect(existsSync(join(dir, TOOL_SCHEMAS_SNAPSHOT)), `${name}/${TOOL_SCHEMAS_SNAPSHOT} presence must match \`pinsHeader\``)
.toBe(pinsHeader === true)
// A nested-agent scenario ships one child fixture per recorded subagent
// session (`session.1.jsonl` …), the replay source for that child session.
for (const childFixture of childFixturePaths(dir, childSessions ?? 0)) {
@@ -616,10 +694,8 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
})
it('exactly one scenario pins the request-header content of each header class', () => {
// Zero pins would drop a class's prompt/schema surface from the suite
// entirely; two would split it. One pin per class is the design
// (pinned-header RFC); WHICH scenario pins is the scenario table's
// reviewable choice.
// Zero pins would drop a class's prompt/schema surface from the suite entirely; two would
// split it.
const pins = new Map<string, string[]>()
for (const scenario of scenarios.filter(s => s.pinsHeader === true)) {
const cls = classOf(scenario)
@@ -632,29 +708,32 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
}
})
it('every pinning fixture carries one request/header, one readable prompt, and its declared deltas', async () => {
// The live uniformity guard runs only in NON-pinning scenarios, so a
// class made of just its pinning scenario would otherwise accept a
// re-recorded pin with several headers or an undeclared mid-run
// header-delta — shapes the pin design cannot represent. Assert the
// committed pins directly; a scenario whose arc legitimately rewrites
// a prompt section declares the exact count via expectedHeaderDeltas.
it('every pinning fixture carries one tokenized request/header, two sidecars, and its declared deltas', async () => {
// The live uniformity guard runs only in NON-pinning scenarios, so a class made of just
// its pinning scenario would otherwise accept a re-recorded pin with several headers or
// an undeclared mid-run header-delta — shapes the pin design cannot represent.
for (const scenario of pinningByClass.values()) {
const fixture = await readFile(join(snapshotsDir, scenario.name, 'session.jsonl'), 'utf8')
const headers = normalizedHeaders(fixture, fixtureContext(fixture))
const promptSnapshot = await readFile(join(snapshotsDir, scenario.name, SYSTEM_PROMPT_SNAPSHOT), 'utf8')
const toolSchemasSnapshot = await readFile(join(snapshotsDir, scenario.name, TOOL_SCHEMAS_SNAPSHOT), 'utf8')
const toolSchemas = parseToolSchemasSnapshot(toolSchemasSnapshot)
expect(headers.length, `${scenario.name}: a pinning fixture must carry exactly one request/header`).toBe(1)
expect(() => restorePinnedToolSchemas(headers[0], toolSchemas), `${scenario.name}: tools must use the sidecar token`)
.not.toThrow()
expect(promptSnapshot.length, `${scenario.name}/${SYSTEM_PROMPT_SNAPSHOT} must not be empty`).toBeGreaterThan(0)
expect(promptSnapshot.endsWith('\n'), `${scenario.name}/${SYSTEM_PROMPT_SNAPSHOT} must end in a newline`).toBe(true)
expect(toolSchemasSnapshot, `${scenario.name}/${TOOL_SCHEMAS_SNAPSHOT} must use canonical JSON formatting`)
.toBe(formatToolSchemasSnapshot(toolSchemas.initial, toolSchemas.deltas))
expect(headerDeltaCount(fixture), `${scenario.name}: a pinning fixture must carry exactly its declared request/header-deltas`)
.toBe(scenario.expectedHeaderDeltas ?? 0)
}
})
it('every committed JSONL omits system prompts and only pinning fixtures keep other header bulk', async () => {
// System prompts always live in the readable Markdown artifact. Header
// pins keep tool schemas/prefixes in JSONL; every other fixture tokenizes
// all header bulk. Fixed-point checks make both storage rules fail loud.
it('every committed JSONL has valid tool results and canonical header storage', async () => {
// Prompts and schemas always leave JSONL. Header pins retain prefixes;
// every other fixture tokenizes those too. Fixed-point checks make both
// storage rules fail loud.
for (const scenario of scenarios) {
const dir = join(snapshotsDir, scenario.name)
const files = [
@@ -663,12 +742,13 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
]
for (const file of files) {
const fixture = await readFile(join(dir, file), 'utf8')
expect(unknownToolCallIds(fixture), `${scenario.name}/${file} contains UNKNOWN_TOOL`)
.toEqual([])
expect(scrubSystemPrompts(fixture), `${scenario.name}/${file} carries an unscrubbed system prompt`)
.toEqual(fixture)
if (scenario.pinsHeader === true) {
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} must pin the non-system header content`)
.not.toEqual(fixture)
} else {
expect(scrubToolSchemas(fixture), `${scenario.name}/${file} carries unscrubbed tool schemas`)
.toEqual(fixture)
if (scenario.pinsHeader !== true) {
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} carries unscrubbed header content`)
.toEqual(fixture)
}

18
packages/support/acp-snapshot/tests/fixtures/fake-acp-agent.ts vendored
View File

@@ -1,19 +1,7 @@
/**
* Scripted fake ACP agent bin for `dsh-acp-snapshot`'s unit specs. Speaks
* newline-delimited JSON-RPC on stdio like the real `dsh-acp-agent` bin, but
* every behavior — how prompts settle, whether session/new rejects, which
* session logs get persisted, what filesystem noise to leave — comes from a
* `behavior.json` sitting NEXT to the `$DSH_SNAPSHOT_FILE` fixture, so a spec
* scripts a whole subprocess run from data. The specs launch it through the
* REAL `runScenario` spawn path (tsx loader, temp cwd, env plumbing), so the
* harness plumbing is exercised for real; only the agent behind the protocol
* is scripted.
*
* The specs (not the golden tier) own this bin: it asserts nothing, echoes
* observable facts into `session/update` text chunks (env probe, permission
* outcome, seeded-workspace listing) for the spec to read off `rawStdout`, and
* exits 0 on stdin EOF after writing the scripted logs — mirroring the real
* bin's dispose-flush-exit shape.
* Scripted ACP agent for snapshot-kit tests. A fixture-adjacent `behavior.json` controls the
* subprocess reached through the real harness path; the bin reports observations over ACP and
* writes scripted logs before exiting on stdin EOF.
*/
import { mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'

2
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/session.jsonl vendored
View File

@@ -1,2 +1,2 @@
{"type":"session","id":"ccdc749f-56f3-4267-9750-598b5c60b7b2","createdAt":600,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-nOQ4Gy"}
{"type":"request/header","seq":0,"time":4,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":[{"name":"t1","description":"D1","parameters":{"type":"object"}}]},"reason":"initial"}}
{"type":"request/header","seq":0,"time":4,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}

12
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/tool-schemas.golden.json vendored Normal file
View File

@@ -0,0 +1,12 @@
{
"initial": [
{
"name": "t1",
"description": "D1",
"parameters": {
"type": "object"
}
}
],
"deltas": []
}

2
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/session.jsonl vendored
View File

@@ -1,4 +1,4 @@
{"type":"session","id":"12121212-3434-4545-8686-787878787878","createdAt":7,"cwd":"/rec/pin-cwd"}
{"type":"request/header","seq":0,"time":7,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":[{"name":"t1","description":"D1","parameters":{"type":"object"}}]},"reason":"initial"}}
{"type":"request/header","seq":0,"time":7,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
{"type":"request/header-delta","seq":1,"time":7,"data":{"system":{"keepStart":1,"keepEnd":0,"insert":["{{system}}"]}}}
{"type":"turn/start","seq":2,"time":7,"data":{"turn":1}}

12
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/tool-schemas.golden.json vendored Normal file
View File

@@ -0,0 +1,12 @@
{
"initial": [
{
"name": "t1",
"description": "D1",
"parameters": {
"type": "object"
}
}
],
"deltas": []
}

7
packages/support/acp-snapshot/tests/harness.spec.ts
View File

@@ -308,11 +308,8 @@ describe('runScenario', () => {
it('rejects the run on a scripted permission kind the agent never offered', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ permissionProbe: true })
// The fake bin offers allow_once/reject_once; scripting allow_always is a
// scenario bug. The agent is answered `cancelled` (it must not be able to
// absorb the bug as an error-means-denial), and the RUN fails: a callback
// throw would only reach the agent as a JSON-RPC error response, letting
// a tolerant agent carry on and the scenario pass — or record.
// The fake offers only allow_once/reject_once. The harness must reject an impossible click,
// not merely send an RPC error that a tolerant agent could absorb.
await expect(runScenario(
{ steps: [...boot, { op: 'prompt', text: 'impossible click' }], permissionAnswers: [{ kind: 'allow_always' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },

43
packages/support/acp-snapshot/tests/normalize.spec.ts
View File

@@ -5,6 +5,7 @@ import {
normalizeStdout,
scrubRequestHeaders,
scrubSystemPrompts,
scrubToolSchemas,
} from '../src/normalize.ts'
/**
@@ -352,3 +353,45 @@ describe('scrubSystemPrompts', () => {
expect(scrubSystemPrompts(out)).toBe(out)
})
})
describe('scrubToolSchemas', () => {
it('scrubs only tool-schema payloads while keeping prompts and prefixes verbatim', () => {
const header = JSON.stringify({
type: 'request/header', seq: 1, time: 2,
data: {
header: {
system: 'full prompt',
tools: [{ name: 'read', description: 'full schema', parameters: { type: 'object' } }],
messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'full prefix' }] }],
},
reason: 'initial',
},
})
const delta = JSON.stringify({
type: 'request/header-delta', seq: 2, time: 3,
data: {
system: { keepStart: 1, keepEnd: 2, insert: ['new prompt line'] },
tools: { added: [{ name: 'grep', description: 'new schema' }], changed: [{ name: 'read', description: 'changed schema' }] },
messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'changed prefix' }] }],
},
})
const systemOnly = JSON.stringify({
type: 'request/header', seq: 3, time: 4,
data: { header: { system: 'prompt only' }, reason: 'resume' },
})
const out = scrubToolSchemas(`${header}\n${delta}\n${systemOnly}\n`)
expect(out).toContain('"tools":"{{tools}}"')
expect(out).toContain('"added":[{"name":"grep","description":"{{tools}}"}]')
expect(out).toContain('"changed":[{"name":"read","description":"{{tools}}"}]')
expect(out).not.toContain('full schema')
expect(out).not.toContain('new schema')
expect(out).not.toContain('changed schema')
expect(out).toContain('full prompt')
expect(out).toContain('new prompt line')
expect(out).toContain('full prefix')
expect(out).toContain('changed prefix')
expect(out.split('\n')[2]).toBe(systemOnly)
expect(scrubToolSchemas(out)).toBe(out)
})
})

126
packages/support/acp-snapshot/tests/suite.spec.ts
View File

@@ -9,30 +9,29 @@ import {
childFixturePaths,
fixtureContext,
formatSystemPromptSnapshot,
formatToolSchemasSnapshot,
headerDeltaCount,
normalizedHeaders,
normalizedSystemPromptDeltas,
normalizedSystemPrompts,
normalizedToolSchemaDeltas,
normalizedToolSchemas,
parseToolSchemasSnapshot,
refreshFixtureReplacements,
restorePinnedToolSchemas,
stabilizeRefreshLog,
unknownToolCallIds,
} from '../src/suite.ts'
/**
* Unit tests for the suite factory, by running it: two synthetic suites over
* the scripted fake ACP bin (./fixtures/fake-acp-agent.ts) register REAL
* describe/it trees at collection time, so every factory path — golden and log
* compares, the per-suite header pin and its uniformity guard, record-mode
* fixture write-back, skip semantics, and the fixture guard block — executes
* as an ordinary green test. The pure helpers get direct cases below.
* Unit tests for the suite factory, by running it: two synthetic suites over the scripted fake
* ACP bin (./fixtures/fake-acp-agent.ts) register real describe/it trees at collection time,
* so every factory path — golden and log compares, the per-suite header pin and its uniformity
* guard, record-mode fixture write-back, skip semantics, and the fixture guard block —
* executes as an ordinary green test.
*
* The replay suite runs against the committed fixtures in ./fixtures/suite.
* The record suite runs against a TEMP COPY of ./fixtures/record-suite
* (record mode writes session fixtures back into its snapshots dir; a run must
* never touch the committed tree). To re-bootstrap the record tree's goldens
* after changing the fake bin's output, run this spec once with
* `ACP_SNAPSHOT_SPEC_BOOTSTRAP=1` (points the record suite at the committed
* tree so vitest creates/updates the goldens and the write-back lands there),
* then commit the result.
* Record tests use a temp copy. To intentionally rebuild their committed fixtures, run this
* spec once with `ACP_SNAPSHOT_SPEC_BOOTSTRAP=1`, then review and commit the resulting tree.
*/
const AGENT = {
@@ -44,12 +43,7 @@ const AGENT = {
const REPLAY_DIR = fileURLToPath(new URL('./fixtures/suite', import.meta.url))
const RECORD_SRC = fileURLToPath(new URL('./fixtures/record-suite', import.meta.url))
// The replay suite doubles as the header-CLASS coverage: every scenario names
// the same explicit class (the record suite exercises the 'default' fallback),
// and plain-turn boots through a per-scenario configPath override (the same
// dummy path the agent default carries — the plumbing, not the composition,
// is what this suite can exercise; the real overlay boot is the acp-agent
// example's code-mode scenarios).
// Replay pins explicit header classes; recording covers the default fallback.
const REPLAY_SCENARIOS: Scenario[] = [
{ name: 'pin-turn', hasModelTurn: true, recorded: true, pinsHeader: true, expectedHeaderDeltas: 1, headerClass: 'main' },
{ name: 'plain-turn', hasModelTurn: true, recorded: true, childSessions: 1, headerClass: 'main', configPath: AGENT.configPath },
@@ -82,6 +76,7 @@ afterAll(async () => {
function staleRefreshFixtures(dir: string): void {
writeFileSync(join(dir, 'plain-turn', 'stdout.golden.jsonl'), 'stale stdout\n')
writeFileSync(join(dir, 'pin-turn', 'system-prompt.golden.md'), 'STALE PROMPT\n')
writeFileSync(join(dir, 'pin-turn', 'tool-schemas.golden.json'), '{"initial":[{"name":"stale"}],"deltas":[]}\n')
const plainBehaviorFile = join(dir, 'plain-turn', 'behavior.json')
const plainBehavior = JSON.parse(readFileSync(plainBehaviorFile, 'utf8')) as Record<string, unknown>
@@ -137,6 +132,9 @@ describe('defineAcpSnapshotSuite: refresh write-back', () => {
'NEW PROMPT LINE',
'',
].join('\n'))
const schemas = readFileSync(join(refreshDir, 'pin-turn', 'tool-schemas.golden.json'), 'utf8')
expect(schemas).toContain('"description": "D1"')
expect(schemas).not.toContain('"name":"stale"')
})
})
@@ -247,6 +245,40 @@ describe('normalizedSystemPrompts', () => {
})
})
describe('normalizedToolSchemas', () => {
it('extracts normalized schema arrays and omits absent or non-array fields', () => {
const log = [
'{"type":"session","id":"a","createdAt":5,"cwd":"/w"}',
'{"type":"request/header","seq":0,"time":9,"data":{"header":{"tools":[{"name":"read","description":"work in /w"}]}}}',
'{"type":"request/header","seq":1,"time":9,"data":{"header":{}}}',
'{"type":"request/header","seq":2,"time":9,"data":{"header":{"tools":null}}}',
'{"type":"request/header","seq":3,"time":9,"data":{"header":null}}',
'{"type":"request/header","seq":4,"time":9,"data":{"header":"invalid"}}',
'',
].join('\n')
expect(normalizedToolSchemas(log, { sessionIds: [], cwd: '/w' })).toEqual([
[{ name: 'read', description: 'work in {{cwd}}' }],
])
})
})
describe('normalizedToolSchemaDeltas', () => {
it('extracts and normalizes object-valued schema edits', () => {
const log = [
'{"type":"request/header-delta","data":{"tools":{"added":[{"name":"read","description":"work in /w"}]}}}',
'{"type":"request/header-delta","data":{"tools":null}}',
'{"type":"request/header-delta","data":{"tools":"invalid"}}',
'{"type":"request/header-delta","data":{"tools":[]}}',
'{"type":"request/header-delta","data":{"system":{"insert":[]}}}',
'{"type":"request/header","data":{"tools":{"added":[]}}}',
'',
].join('\n')
expect(normalizedToolSchemaDeltas(log, { sessionIds: [], cwd: '/w' })).toEqual([
{ added: [{ name: 'read', description: 'work in {{cwd}}' }] },
])
})
})
describe('normalizedSystemPromptDeltas', () => {
it('extracts and normalizes well-formed system edits', () => {
const log = [
@@ -281,6 +313,39 @@ describe('formatSystemPromptSnapshot', () => {
})
})
describe('tool-schema snapshots', () => {
const snapshot = {
initial: [{ name: 'read', description: 'Read a file.' }],
deltas: [{ added: [{ name: 'grep', description: 'Search files.' }] }],
}
it('formats and parses canonical structured JSON', () => {
const formatted = formatToolSchemasSnapshot(snapshot.initial, snapshot.deltas)
expect(formatted).toBe(`${JSON.stringify(snapshot, null, 2)}\n`)
expect(parseToolSchemasSnapshot(formatted)).toEqual(snapshot)
})
it('rejects invalid top-level and field shapes', () => {
expect(() => parseToolSchemasSnapshot('null')).toThrow(/must be an object/)
expect(() => parseToolSchemasSnapshot('"invalid"')).toThrow(/must be an object/)
expect(() => parseToolSchemasSnapshot('[]')).toThrow(/must be an object/)
expect(() => parseToolSchemasSnapshot('{"initial":{},"deltas":[]}')).toThrow(/array-valued/)
expect(() => parseToolSchemasSnapshot('{"initial":[],"deltas":{}}')).toThrow(/array-valued/)
})
it('restores initial schemas into the pinned header token', () => {
expect(restorePinnedToolSchemas({ system: '{{system}}', tools: '{{tools}}' }, snapshot))
.toEqual({ system: '{{system}}', tools: snapshot.initial })
})
it('rejects invalid headers and a missing tool token', () => {
expect(() => restorePinnedToolSchemas(null, snapshot)).toThrow(/must be an object/)
expect(() => restorePinnedToolSchemas('invalid', snapshot)).toThrow(/must be an object/)
expect(() => restorePinnedToolSchemas([], snapshot)).toThrow(/must be an object/)
expect(() => restorePinnedToolSchemas({ tools: [] }, snapshot)).toThrow(/must equal/)
})
})
describe('headerDeltaCount', () => {
it('counts request/header-delta events, ignoring blanks and other lines', () => {
const delta = JSON.stringify({ type: 'request/header-delta', seq: 2, time: 9, data: {} })
@@ -290,6 +355,27 @@ describe('headerDeltaCount', () => {
})
})
describe('unknownToolCallIds', () => {
it('returns structured UNKNOWN_TOOL call ids and ignores other results', () => {
const log = [
'{"type":"tool/result","data":{"callId":"missing","error":{"code":"UNKNOWN_TOOL"}}}',
'{"type":"tool/result","data":{"callId":"failed","error":{"code":"EXECUTION_FAILED"}}}',
'{"type":"tool/result","data":null}',
'{"type":"tool/result","data":"invalid"}',
'{"type":"tool/result","data":{"error":null}}',
'{"type":"tool/result","data":{"error":"invalid"}}',
'{"type":"assistant/message","data":{"error":{"code":"UNKNOWN_TOOL"}}}',
'{"type":"tool/result","data":{"error":{"code":"UNKNOWN_TOOL"}}}',
'',
].join('\n')
expect(unknownToolCallIds(log)).toEqual(['missing', '<missing callId>'])
})
it('returns no failures for ordinary tool results', () => {
expect(unknownToolCallIds('{"type":"tool/result","data":{"callId":"ok"}}\n')).toEqual([])
})
})
describe('refreshFixtureReplacements', () => {
it('maps fresh ids and cwd values to the existing fixture values, skipping non-replacements', () => {
const log = (content: string): HarvestedLog => ({ id: 'diagnostic', createdAt: 1, content })

17
packages/support/invariants/README.md
View File

@@ -1,8 +1,8 @@
# dsh-invariants
Dev-mode event-contract assertions. This pure-listener plugin checks relationships among session events, agent states, scoped dispatches, and model requests at runtime; it does not own or change product behavior.
Runtime event-contract assertions intended for development diagnostics. This pure-listener plugin checks relationships among session events, agent states, scoped dispatches, and model requests; it does not own or change product behavior.
**Off in production.** Enable it in tests and the demos, where a contract violation should fail loudly. It costs nothing when not registered, and doubles as executable documentation of the event taxonomy — the assertions *are* the contract.
The plugin has no environment guard: it is active wherever it is registered. The default [`dsh-agent-spine-demo`](../../examples/agent-spine-demo/README.md) bundle mounts it unconditionally; a custom composition can omit it when the runtime cost is undesirable. It doubles as executable documentation of the event taxonomy — the assertions *are* the contract.
Session itself owns immutable log storage in every composition: it takes one lossless JSON snapshot of each accepted event, deep-freezes that record, and exposes the log through immutable array snapshots. The invariants plugin checks the cross-record and cross-seam rules that storage immutability cannot express.
@@ -39,14 +39,23 @@ Agent status (per agent):
Model requests (on `llm/stream`):
- **a loop-built request is exactly what the log reconstructs** — a frozen request with a live `sessionId` (the loop-built marker; hand-built one-shots like compaction's summarize are unfrozen and skipped) must carry frozen `messages` deep-equal to the derivation over the log prefix strictly before the in-flight step's `step/start` (rebuilt through a FRESH `Session`, so the live cache cannot vouch for itself — and boundary-correct: content logged after `step/start` legitimately belongs to the next request), and every non-content field must equal the fold of the log's `request/header*` events (see [the reconstructability RFC](../../../docs/rfc/implemented/architecture/2026-07-05-reconstructable-requests.md)). Registered with `prepend: true` so a short-circuiting `llm/stream` listener (the replay adapter) cannot silence it; prepend orders it against append-registered listeners only — correctness rests on the seq-bounded rebuild, never listener timing.
- **a loop-built request is exactly what the log reconstructs** — a frozen request with a live `sessionId` is rebuilt through a fresh `Session` from the prefix before its in-flight `step/start`; later content belongs to the next request, and hand-built unfrozen one-shots are excluded. Frozen messages must match that derivation, while every other field matches folded `request/header*` events. The prepended check runs before ordinary short-circuiting stream listeners, but correctness comes from the sequence boundary rather than listener timing. See the [reconstructability RFC](../../../docs/rfc/implemented/architecture/2026-07-05-reconstructable-requests.md).
On any violation it throws `InvariantError` (`code: 'INVARIANT'`).
## Why runtime assertions remain useful
Session enforces the per-record storage boundary at runtime, where a cast cannot bypass it. Pervasive `DeepReadonly<SessionEvent>` types would add noise across consumers without expressing relationships such as turn/step nesting, subject-correct scoped dispatch, or equality between a request and its log reconstruction. This plugin checks those relationships in development while `dsh-session` keeps history immutable in every composition. See [source-owned session immutability and dev-mode invariants](../../../docs/rfc/implemented/architecture/2026-06-11-dev-invariants-over-deep-readonly.md).
Session enforces the per-record storage boundary at runtime, where a cast cannot bypass it. Pervasive `DeepReadonly<SessionEvent>` types would add noise across consumers without expressing relationships such as turn/step nesting, subject-correct scoped dispatch, or equality between a request and its log reconstruction. This plugin checks those relationships wherever it is mounted while `dsh-session` keeps history immutable in every composition. See [source-owned session immutability and dev-mode invariants](../../../docs/rfc/implemented/architecture/2026-06-11-dev-invariants-over-deep-readonly.md).
## Seeded sessions
A seeded or forked session arrives with events already in its log because construction does not emit `session/event` for each seed record. `Session` validates, snapshots, and freezes every seed record before accepting it; on `session/created`, this plugin replays the accepted log only to rebuild and check its relational trace state.
## Model Experience
None, as this observer only validates events and frozen requests and never rewrites prompts, schemas, messages, or streams.
## Known Limitations and Deferred Work
- **The request-reconstructability assertion covers loop-built requests only** — hand-built one-shots (e.g. compaction's summarize call) carry no live `sessionId` marker and are skipped.
- **Merge-extended event families get no family-specific assertions** — `compact/*` lock pairing and `hook/*` invoked/result pairing are not checked here; only the core turn/step/chunk/tool-result contract is.

10
packages/support/invariants/package.json
View File

@@ -1,6 +1,6 @@
{
"name": "@deepseek-ai/dsh-invariants",
"description": "Dev-mode event-contract assertions for the DeepSeek Harness",
"description": "Runtime event-contract assertions for DeepSeek Harness development diagnostics",
"version": "0.0.1",
"private": true,
"type": "module",
@@ -26,13 +26,17 @@
"@deepseek-ai/dsh-llm": "^0.0.1",
"@deepseek-ai/dsh-scope": "^0.0.1",
"@deepseek-ai/dsh-session": "^0.0.1",
"cordis": "^4.0.0-rc.6"
"cordis": "^4.0.0-rc.7"
},
"devDependencies": {
"@deepseek-ai/dsh-agent": "workspace:^",
"@deepseek-ai/dsh-llm": "workspace:^",
"@deepseek-ai/dsh-scope": "workspace:^",
"@deepseek-ai/dsh-session": "workspace:^",
"cordis": "^4.0.0-rc.6"
"@deepseek-ai/dsh-subagent": "workspace:^",
"@deepseek-ai/dsh-system-prompt": "workspace:^",
"@deepseek-ai/dsh-tools": "workspace:^",
"@deepseek-ai/dsh-user-approval": "workspace:^",
"cordis": "^4.0.0-rc.7"
}
}

78
packages/support/invariants/src/index.ts
View File

@@ -1,19 +1,9 @@
/**
* Dev-mode invariants: a pure-listener plugin that asserts relationships in
* the harness event contract at runtime.
*
* Everything is a plugin — this is just listeners on `session/created`,
* `session/event`, `agent/status`, and the scoped dispatch and request seams.
* It is **off in production**: enable it in tests and demos, where a contract
* violation should be a loud failure rather than a subtle one. It doubles as
* executable documentation of the event taxonomy: the assertions below are
* the contract.
*
* Session owns immutable log storage: it snapshots and deep-freezes every
* accepted event at the source. This plugin checks relationships that one
* event's types and immutability cannot express, including turn/step nesting,
* scoped dispatch, status transitions, and request reconstructability.
*
* Runtime listeners that fail loudly when cross-event contracts are broken:
* turn and step nesting, scoped dispatch, status transitions, and request
* reconstruction. The plugin has no environment guard and is active wherever
* mounted, including the default `dsh-agent-spine-demo` bundle; custom compositions
* may omit it. Sessions still own event snapshots and freezing.
* @module @deepseek-ai/dsh-invariants
*/
@@ -24,6 +14,7 @@ import type { CallId, GenerateOptions } from '@deepseek-ai/dsh-llm'
import type { Agent, AgentStatus } from '@deepseek-ai/dsh-agent'
import { Session, SessionId, foldRequestHeader } from '@deepseek-ai/dsh-session'
import type { SessionEvent, SurfaceEventType } from '@deepseek-ai/dsh-session'
import { scopedSubjectResolverFor } from './scoped-events.generated.ts'
export const name = 'invariants'
export const inject = ['sessions']
@@ -85,17 +76,6 @@ interface SessionTraceTransition {
seq: number
}
/** Event payload prefix for scoped seams whose first argument names its agent. */
interface AgentSubject {
agent: Agent
}
/** Structural subject fields used without coupling this dev plugin to owning services. */
interface ScopedSubjectFields {
agent?: Agent
scope?: object
}
/** Assert that a step-scoped event names the currently open turn and step. */
function requireOpenStep(trace: SessionTrace, kind: string, turn: number, step: number): void {
if (trace.openTurn !== turn || trace.openStep !== step) {
@@ -328,23 +308,19 @@ function replayEvent(trace: SessionTrace, event: SessionEvent): void {
applyTransition(trace, validateEvent(trace, event))
}
/** Legal agent status transitions (the only state machine the loop guarantees). */
/** Allow an initial observation, idle/running transitions, and terminal disposal; reject repeats and leaving disposed. */
function checkTransition(from: AgentStatus | undefined, to: AgentStatus): void {
// First observation: any status is a valid starting point.
if (from === undefined) return
// A no-op transition is illegal — setStatus dedups, so we never see it.
if (from === to) {
throw new InvariantError(`agent/status repeated ${to} (no-op transition)`)
}
// Leaving `disposed` is illegal — disposal is terminal.
if (from === 'disposed') {
throw new InvariantError(`agent/status left terminal state disposed → ${to}`)
}
// idle↔running and (idle|running)→disposed are all legal; nothing else exists.
}
/**
* Register the dev-mode invariants. Contributions are effect-scoped, so
* Register the runtime invariants. Contributions are effect-scoped, so
* disposing the plugin fiber removes all listeners (HMR-safe). On (re-)apply
* the trace state is rebuilt by replaying each existing session's log, so a
* hot reload mid-turn does not falsely reject the next event.
@@ -424,40 +400,12 @@ export function apply(ctx: Context): void {
// (agent-scoped listeners over-hear foreign agents), and a mis-keyed one
// delivers to the wrong agent's listeners. `internal/dispatch` fires
// synchronously before listener delivery, so a violation throws at the
// dispatching call site. The table maps each family to how its subject is
// read from the event arguments; `null` = the subject is not recoverable
// from the arguments (session events key by the OWNING agent; subagent
// lifecycle events key by the delegating parent), so only carrier
// PRESENCE is asserted there.
const scopedSubject: Record<string, ((args: unknown[]) => unknown) | null> = {
'agent/created': args => args[0],
'agent/disposed': args => args[0],
'agent/status': args => args[0],
'agent/queued': args => args[0],
'agent/session-start': args => args[0],
'agent/pre-step': args => args[0],
'agent/prompt-submit': args => args[0],
'agent/request': args => args[0],
'agent/session-prefix': args => args[0],
'agent/step-result': args => args[0],
'agent/turn-continuation': args => args[0],
'agent/turn-stop': args => args[0],
'agent/error': args => args[0],
'approval/request': args => (args[0] as AgentSubject).agent,
'tools/pre-execute': args => (args[0] as ScopedSubjectFields).agent,
'tools/execute': args => (args[0] as ScopedSubjectFields).agent,
'tools/post-execute': args => (args[0] as ScopedSubjectFields).agent,
'tools/result': args => (args[0] as ScopedSubjectFields).agent,
'system-prompt/assemble': args => (args[1] as ScopedSubjectFields).scope,
'session/created': null,
'session/disposed': null,
'session/event': null,
'session/flush': null,
'subagent/start': null,
'subagent/end': null,
}
// dispatching call site. The generated table maps each family to the unique
// payload path whose Program type matches the real scopeTarget routing key;
// `null` means the key is external to the payload, so only carrier presence
// can be asserted.
ctx.on('internal/dispatch', (_mode, name, args, thisArg) => {
const subjectOf = scopedSubject[name]
const subjectOf = scopedSubjectResolverFor(name)
if (subjectOf === undefined) return
if (!isScopeCarrier(thisArg)) {
throw new InvariantError(

69
packages/support/invariants/src/scoped-events.generated.ts Normal file
View File

@@ -0,0 +1,69 @@
/**
* Generated scoped-event routing-subject resolvers for dsh-invariants.
* Do not edit by hand; run `pnpm run gen-scoped-events`.
*
* @module @deepseek-ai/dsh-invariants/scoped-events.generated
*/
import type { Events } from 'cordis'
import type { Scoped } from '@deepseek-ai/dsh-scope'
import type {} from '@deepseek-ai/dsh-agent'
import type {} from '@deepseek-ai/dsh-session'
import type {} from '@deepseek-ai/dsh-subagent'
import type {} from '@deepseek-ai/dsh-system-prompt'
import type {} from '@deepseek-ai/dsh-tools'
import type {} from '@deepseek-ai/dsh-user-approval'
type ScopedEventName = {
[K in keyof Events]: ThisParameterType<Events[K]> extends Scoped<object> ? K : never
}[keyof Events]
type ScopedSubjectResolver = (args: readonly unknown[]) => unknown
function adapt<K extends ScopedEventName>(
resolver: (args: Parameters<Events[K]>) => unknown,
): ScopedSubjectResolver {
return args => resolver(args as Parameters<Events[K]>)
}
const scopedSubjectResolvers = Object.freeze({
'agent/created': adapt<'agent/created'>(args => args[0]),
'agent/disposed': adapt<'agent/disposed'>(args => args[0]),
'agent/error': adapt<'agent/error'>(args => args[0]),
'agent/pre-step': adapt<'agent/pre-step'>(args => args[0]),
'agent/prompt-submit': adapt<'agent/prompt-submit'>(args => args[0]),
'agent/queued': adapt<'agent/queued'>(args => args[0]),
'agent/request': adapt<'agent/request'>(args => args[0]),
'agent/session-prefix': adapt<'agent/session-prefix'>(args => args[0]),
'agent/session-start': adapt<'agent/session-start'>(args => args[0]),
'agent/status': adapt<'agent/status'>(args => args[0]),
'agent/step-result': adapt<'agent/step-result'>(args => args[0]),
'agent/turn-continuation': adapt<'agent/turn-continuation'>(args => args[0]),
'agent/turn-stop': adapt<'agent/turn-stop'>(args => args[0]),
'approval/request': adapt<'approval/request'>(args => args[0].agent),
'session/created': null,
'session/disposed': null,
'session/event': null,
'session/flush': null,
'subagent/end': null,
'subagent/start': null,
'system-prompt/assemble': adapt<'system-prompt/assemble'>(args => args[1].scope),
'tools/execute': adapt<'tools/execute'>(args => args[0].agent),
'tools/post-execute': adapt<'tools/post-execute'>(args => args[0].agent),
'tools/pre-execute': adapt<'tools/pre-execute'>(args => args[0].agent),
'tools/result': adapt<'tools/result'>(args => args[0].agent),
} as const satisfies Readonly<Record<ScopedEventName, ScopedSubjectResolver | null>>)
const scopedSubjectResolverIndex: Readonly<Record<string, ScopedSubjectResolver | null>> = scopedSubjectResolvers
/**
* Resolve the routing key named by one scoped event payload. A null
* resolver means the payload cannot expose its external routing key, so the
* invariant checks carrier presence only.
* @param event - runtime Cordis event name.
* @returns the generated subject resolver, null for presence-only,
* or undefined when the event is not scope-filtered.
*/
export function scopedSubjectResolverFor(event: string): ScopedSubjectResolver | null | undefined {
return scopedSubjectResolverIndex[event]
}

32
packages/support/invariants/tests/invariants.spec.ts
View File

@@ -325,19 +325,17 @@ describe('HMR state rebuild', () => {
it('rebuilds trace state for a session that exists at (re-)apply time', async () => {
const ctx = new Context()
await ctx.plugin(SessionStore)
// First registration, mid-turn: a turn is open when the plugin reloads.
const first = await ctx.plugin(Invariants)
const session = ctx.sessions.create()
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('step/start', { turn: 1, step: 1 })
await first.dispose()
// Re-apply (HMR): the fresh fiber must replay the existing log so the open
// step is known — the next chunk must NOT be a false positive.
// Re-apply mid-step: the new fiber must reconstruct the open boundaries from the log.
await ctx.plugin(Invariants)
expect(() => session.append('assistant/chunk', { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'h' } }))
.not.toThrow()
// And a genuine violation is still caught after the rebuild.
// Rebuild must not disable later violations.
expect(() => session.append('turn/start', { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } }))
.toThrow(/turn 1 is still open/)
})
@@ -541,25 +539,17 @@ describe('surface invariants', () => {
})
it('rejects sourceEventSeqs referencing unknown seq (gap in event log)', async () => {
// The unknown-seq check fires when a ref passes the "earlier" test but is
// not in knownSeqs — only possible with a gap in seqs. We create a gap by
// directly manipulating the private log array to skip a seq.
// Create an impossible-through-public-API gap so seq 2 is earlier but unknown.
const { ctx } = await setup()
const session = ctx.sessions.create()
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('step/start', { turn: 1, step: 1 })
// Push a fake event at seq 3 into the internal log, creating a gap at seq 2.
// The invariants plugin replays session.events on every append, so it sees
// this gap during trace reconstruction.
;(session as unknown as { log: unknown[] }).log.push({
type: 'assistant/chunk',
seq: 3,
time: Date.now(),
data: { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'x' } },
})
// Now the log has seqs 0, 1, 3 (gap at 2). Append at what session believes
// is seq 3 (log.length). Reference seq 2: passes earlier (2 < 3) but not
// in knownSeqs ({0, 1, 3} — gap at 2).
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [2] })
}).toThrow(/unknown seq 2/)
@@ -803,12 +793,8 @@ describe('request-reconstruction cross-check (llm/stream)', () => {
describe('request cross-check ordering (prepend)', () => {
it('runs ahead of a short-circuiting llm/stream listener registered before it', async () => {
// The replay adapter returns its chunks WITHOUT calling next(), which
// would silence a later-registered check — snapshot compositions load
// replay before the app bundle that loads invariants. The check prepends,
// so it fires ahead of append-registered listeners regardless of load
// order. (Prepend orders it against APPENDED listeners only; correctness
// rests on the seq-bounded rebuild, not on listener timing.)
// Replay short-circuits without next(), so the check prepends ahead of ordinary listeners;
// correctness still comes from its sequence-bounded rebuild, not listener timing.
const ctx = new Context()
await ctx.plugin(SessionStore)
ctx.on('llm/stream', () => (async function* () {})() as never) // short-circuits, no next()
@@ -848,7 +834,7 @@ describe('scoped-dispatch invariants', () => {
it('accepts a matching carrier and rejects a mismatched one for EVERY agent-subject event', async () => {
const ctx = await scopedCtx()
// Real Session objects: the session-start tracker WeakSet-keys them.
// Real Session objects keep the synthetic Agent handles structurally valid.
const agent = { id: 'a1', session: new Session(SessionId('a1-s')) } as unknown as Agent
const other = { id: 'a2', session: new Session(SessionId('a2-s')) } as unknown as Agent
// One dispatch per table row keeps every subject extractor covered: the
@@ -869,9 +855,9 @@ describe('scoped-dispatch invariants', () => {
['agent/error', [agent, 1, 0, new Error('x')]],
['approval/request', [{ agent, toolName: 'echo' }, () => Promise.resolve('unavailable')]],
['tools/pre-execute', [{ callId: 'c', name: 't', arguments: {}, agent }, () => Promise.resolve({ kind: 'allow' })]],
['tools/execute', [{ callId: 'c', name: 't', arguments: {}, agent }, () => Promise.resolve({ callId: 'c', content: [], isError: false })]],
['tools/post-execute', [{ callId: 'c', name: 't', arguments: {}, agent }, { callId: 'c', content: [], isError: false }, () => Promise.resolve({ kind: 'accept' })]],
['tools/result', [{ callId: 'c', name: 't', arguments: {}, agent }, { callId: 'c', content: [], isError: false }]],
['tools/execute', [{ callId: 'c', name: 't', arguments: {}, agent }, () => Promise.resolve({ content: [], isError: false })]],
['tools/post-execute', [{ callId: 'c', name: 't', arguments: {}, agent }, { content: [], isError: false }, () => Promise.resolve({ kind: 'accept' })]],
['tools/result', [{ callId: 'c', name: 't', arguments: {}, agent }, { content: [], isError: false }]],
]
for (const [event, args] of rows) {
const subject = agent

12
packages/support/invariants/tsconfig.json
View File

@@ -25,6 +25,18 @@
},
{
"path": "../../core/scope"
},
{
"path": "../../core/system-prompt"
},
{
"path": "../../ui/user-approval"
},
{
"path": "../../core/tools"
},
{
"path": "../../subagent/subagent"
}
]
}

9
packages/support/llm-replay/README.md
View File

@@ -43,3 +43,12 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
## Plugin export shape
Named `name` / `inject` / `Config` / `apply`, with **no default export**: the cordis Loader's `unwrapExports` does `exports.default ?? exports`, so a stray default would collapse the module to the bare function and drop the `inject` namespace (see [docs/postmortem/0001](../../../docs/postmortem/0001-acp-default-export-drops-inject.md)).
## Model Experience
None, as this keyless test adapter sends no request to a provider model; it only replays recorded assistant chunks into the test loop.
## Known Limitations and Deferred Work
- **First-call-order script binding assumes sequential delegation** — a cut that runs sibling subagents concurrently (or a compaction summarize call landing mid-run) would bind live sessions to recorded scripts non-deterministically; a stronger keying is deferred until such a scenario exists (`XXX(concurrent-subagents)`).
- **Only chunk-producing calls are derivable** — a pure pre-chunk throw or a cancel/hang scenario needs the `replay.override.json` sidecar; the override replaces the PRIMARY session's script only.

4
packages/support/llm-replay/package.json
View File

@@ -24,11 +24,11 @@
"peerDependencies": {
"@deepseek-ai/dsh-llm": "^0.0.1",
"@deepseek-ai/dsh-session": "^0.0.1",
"cordis": "^4.0.0-rc.6"
"cordis": "^4.0.0-rc.7"
},
"devDependencies": {
"@deepseek-ai/dsh-llm": "workspace:^",
"@deepseek-ai/dsh-session": "workspace:^",
"cordis": "^4.0.0-rc.6"
"cordis": "^4.0.0-rc.7"
}
}

173
packages/support/llm-replay/src/index.ts
View File

@@ -1,50 +1,8 @@
/**
* Replay LLM plugin for snapshot tests.
*
* Installs a single `llm/stream` waterfall listener that short-circuits the
* waterfall (never calls `next()`) and yields model streams reconstructed from
* a recorded **session JSONL** fixture — so a snapshot test can boot the real
* agent against a fixed model transcript with no API key. See
* docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
*
* The fixture IS the persisted session log (`<scenario>/session.jsonl`): its
* `assistant/chunk` events carry every {@link StreamChunk}, so grouping them by
* `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model
* call per loop step — see packages/core/agent-loop/src/loop.ts). Recording is
* therefore "run the real agent once and harvest the `.jsonl`", done by the
* snapshot harness — this plugin does not record. A fixture may carry its
* `request/header` content tokenized to `{{system}}`/`{{tools}}` (the harness
* pins that content in one scenario and scrubs the rest); replay is
* indifferent — derivation reads ONLY `assistant/chunk` events and the line-0
* session header.
*
* A NESTED-agent scenario records more than one log: the parent plus one per
* in-process subagent (each subagent runs as its own {@link Session} on the same
* context). Replay loads them all ({@link loadSessionScripts}), derives a script
* per recorded session, and keys each live call by its calling session id
* (`GenerateOptions.sessionId`, stamped by the loop). Live session ids are fresh
* random values, so a live session binds to a recorded script by FIRST-CALL
* order (parent first — it streams before it delegates); see
* {@link installLlmReplay}.
*
* Two failure modes are NOT reconstructable from `assistant/chunk` alone — a
* pure throw before any chunk (e.g. an HTTP 401: the log holds only a
* `turn/end {error}`, no chunks) and a cancel/hang (timing, not chunk content).
* A scenario that needs those supplies an optional sidecar
* (`<scenario>/replay.override.json`: a `ReplayEntry[]`) that REPLACES the
* derived script.
*
* It lives in its own package (not under `examples/`) so its derive/parse/
* replay logic falls under the per-file 100% coverage gate on package `src`
* trees — its tests previously lived under `examples/`, which the gate does
* not measure, leaving these branches (clean chunks / mid-stream throw / hang)
* unguarded. Its consumer is the ACP snapshot harness in `examples/acp-agent`,
* which loads it (via `cordis.snapshot.yml`) in place of a real LLM adapter.
*
* Plugin export shape: named `name`/`inject`/`Config`/`apply`, NO default
* export (the cordis Loader's `unwrapExports` does `exports.default ?? exports`,
* so a stray default would drop the namespace — see docs/postmortem/0001).
*
* Keyless snapshot-test LLM replay. It derives one model-call script per
* recorded session from `assistant/chunk` events and binds fresh live sessions
* to parent/child scripts by first-call order. Throw and hang cases require an
* explicit override because a session log cannot reconstruct them alone.
* @module @deepseek-ai/dsh-llm-replay
*/
@@ -56,21 +14,9 @@ import type { GenerateOptions, StreamChunk } from '@deepseek-ai/dsh-llm'
import { LlmError, assertNever } from '@deepseek-ai/dsh-llm'
/**
* One recorded model call. A discriminated union (not a bare `StreamChunk[]`)
* so it can faithfully replay BOTH branches of the documented LLM failure
* contract — an adapter may THROW from `stream()` or end with a `finish` error
* chunk — plus a `hang` marker for cancellation scenarios (mirrors the
* `MockAdapter` `hang` support in packages/core/agent-loop/tests).
*
* A `throw` entry carries any `chunks` the adapter emitted BEFORE it threw, so
* a mid-stream transport failure (partial output then `STREAM_CLOSED`) replays
* the partial chunks first and only then throws — exactly what the agent loop
* saw live (it may already have emitted partial assistant chunks).
*
* The normal/finish-terminated cases are DERIVED from the session JSONL
* ({@link deriveReplayScript}); only the throw and hang cases need a
* hand-authored sidecar entry (a thrown stream leaves no terminal `finish` in
* the log, so it cannot be derived as `chunks`).
* One recorded model call. `throw` may replay prefix chunks before failing;
* `hang` models cancellation. Only ordinary chunk entries derive from JSONL;
* the other variants come from an override sidecar.
*/
export type ReplayEntry =
| { kind: 'chunks'; chunks: StreamChunk[] }
@@ -102,13 +48,8 @@ export interface ReplayConfig {
}
/**
* One recorded session's replay script: the per-call entries plus the header
* facts needed to ORDER and key it. Live session ids are freshly random at
* replay time and never equal the recorded `id`, so the recorded id is only a
* diagnostic; `createdAt` is the load-bearing field — scripts are ordered by it
* (a parent is created before its children) and each newly-seen live session is
* bound to the next script in that order (= first-call order in the synchronous
* nested cut, where the parent streams before it delegates).
* Recorded calls plus header facts used to order parent and child scripts.
* Recorded ids are diagnostic; fresh live ids bind by ordered first use.
*/
export interface SessionScript {
/** The recorded session id (diagnostics only — the live id differs). */
@@ -134,9 +75,7 @@ export interface SessionScript {
export function parseSessionLog(text: string): SessionEvent[] {
const lines = text.split('\n').filter(line => line.trim().length > 0)
const events: SessionEvent[] = []
// Skip line 0 (the header). A reader distinguishes it by its `type:'session'`
// tag; we simply drop the first line, which the JSONL backend guarantees is
// the header.
// The JSONL backend guarantees line 0 is the session header.
for (let i = 1; i < lines.length; i++) {
const parsed: unknown = JSON.parse(lines[i] as string)
events.push(parsed as SessionEvent)
@@ -145,14 +84,8 @@ export function parseSessionLog(text: string): SessionEvent[] {
}
/**
* Read the identifying facts off a session log's header line (line 0): the
* recorded session `id` (diagnostics), `createdAt` (the deterministic ordering
* key that binds a recorded script to a live session — see
* {@link SessionScript}), and `seedLength` (the seed boundary — how many leading
* events were INHERITED via a fork seed rather than produced by this session's
* own model calls; absent ⇒ 0). A header missing a field falls back to a stable
* default (`''` / `0` / `0`) rather than throwing: a no-model fixture is
* header-only and still orders fine as the single (primary) script.
* Read replay identity, ordering, and fork-seed facts from the JSONL header.
*
* @param text - the raw `.jsonl` file contents (only the header line is read).
* @returns the header's `id`, `createdAt`, and `seedLength`, defaulted when absent.
*/
@@ -169,21 +102,9 @@ export function parseSessionHeader(text: string): { id: string; createdAt: numbe
/**
* Reconstruct the per-`stream()` replay script from a recorded session log.
*
* The agent loop makes exactly one `ctx.llm.stream()` call per step and appends
* every chunk as an `assistant/chunk` event tagged with the current
* `(turn, step)`. Grouping those events by `(turn, step)` in log order
* therefore yields one `{kind:'chunks'}` entry per model call, in call order.
*
* A group is only valid if it ends in a `finish` chunk — the adapter contract
* guarantees a successful (or finish-error) stream terminates with `finish`,
* and the loop relies on it. A group WITHOUT a terminal `finish` is the
* fingerprint of a *thrown* `stream()` (the loop recorded the prefix chunks,
* then an `error`/`turn/end`, but no `finish`): such a stream cannot be
* faithfully replayed as `{kind:'chunks'}` (that would look like a clean stop),
* so deriving it is an error — the scenario must supply a `replay.override.json`
* sidecar with an explicit `throw` (or `hang`) entry instead. {@link
* deriveReplayScript} throws, naming the offending `(turn, step)`, so a missing
* override fails loud rather than silently replaying a thrown call as success.
* Groups `assistant/chunk` events by turn and step. Every group must end in a
* `finish`; a missing terminator means the live stream threw, so derivation
* rejects and the scenario must provide an explicit override.
* @param events - the recorded session's events; only `assistant/chunk` is consulted.
* @returns one `chunks` entry per recorded model call, in call order.
*/
@@ -242,17 +163,9 @@ export function loadReplayScript(config: ReplayConfig): ReplayEntry[] {
}
/**
* Load every recorded session's script for a scenario, ordered by `createdAt`
* (earliest first), ready to bind to live sessions in first-call order.
* Load the primary and child scripts in bind order. Child derivation begins at
* `seedLength` so inherited parent chunks are never replayed as child calls.
*
* The PRIMARY session (`config.file`, with its optional `overrideFile`) is the
* parent; each `config.childFiles` entry is a recorded subagent session. A
* single-session scenario has no `childFiles`, so this returns one script and
* behaves exactly like the old single-cursor replay. The primary always sorts
* first when ties occur (a sub-millisecond parent/child `createdAt` collision):
* the parent issues the FIRST model call (it must stream before it can delegate
* in the synchronous nested cut), so binding it to the first live session is
* correct regardless of a timestamp tie.
* @param config - the fixture paths: the primary log plus any recorded child logs.
* @returns the primary script first, then the child scripts in bind order.
*/
@@ -274,12 +187,8 @@ export function loadSessionScripts(config: ReplayConfig): SessionScript[] {
}
const text = readFileSync(childFile, 'utf8')
const header = parseSessionHeader(text)
// Derive the child's script from its OWN events only — events AT OR AFTER
// the seed boundary. A FORK child's log begins with the seeded parent prefix
// (the parent's events, including its `assistant/chunk`s); replaying those as
// the child's model calls would feed the child the PARENT's recorded
// responses. `seedLength` is 0 for a fresh (spawn) child, so this is a no-op
// there.
// Derive the child's script from its own events only — events AT OR after the seed
// boundary.
const ownEvents = parseSessionLog(text).slice(header.seedLength)
children.push({
recordedId: header.id,
@@ -288,20 +197,8 @@ export function loadSessionScripts(config: ReplayConfig): SessionScript[] {
primary: false,
})
}
// The primary (parent) always binds first — it issues the first model call,
// because it must run a turn before it can delegate. Children follow in
// createdAt order. In the current synchronous cut sibling children are created
// STRICTLY SEQUENTIALLY — the subagent tool awaits one child's result and
// disposes it before the parent's next tool call can start the next — so their
// createdAt values are strictly ordered and match first-call order exactly.
// The recordedId tiebreak only makes a degenerate same-millisecond collision
// (unreachable in this cut) deterministic; it does NOT recover first-call
// order, so it is arbitrary if such a tie ever occurs.
// XXX(concurrent-subagents): a future cut that runs siblings concurrently or
// backgrounded could create two children in the same millisecond, where this
// createdAt+id order may diverge from first-call order. That cut must thread a
// real first-call ordinal (the order live sessions first stream) instead of
// leaning on createdAt — see the per-session-replay RFC.
// Synchronous children start in creation order; the id only stabilizes timestamp ties.
// XXX(concurrent-subagents): concurrent children need an explicit first-call ordinal.
children.sort((a, b) => a.createdAt - b.createdAt || a.recordedId.localeCompare(b.recordedId))
return [primary, ...children]
}
@@ -344,31 +241,11 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined)
}
/**
* Install the replay `llm/stream` listener on `ctx`. Returns the listener
* disposer (so a fiber dispose removes it — HMR safety). Exported separately
* from {@link apply} so unit tests can drive it without the Loader or env vars.
* Install per-session positional replay. A newly seen live session takes the
* next ordered recorded script, then advances its own cursor synchronously at
* invocation time; calls without `sessionId` share one anonymous session.
* Returns the effect disposer for HMR-safe removal.
*
* Replay is PER-SESSION POSITIONAL: each recorded session has its own script
* (parent + any subagent children, loaded by {@link loadSessionScripts} ordered
* by `createdAt`), and the Nth `stream()` call FROM A GIVEN SESSION serves that
* session's Nth entry. The calling session is read off `options.sessionId` (the
* agent loop stamps it from `agent.session.id`).
*
* Live session ids are freshly random and never equal the recorded ones, so a
* live session binds to a recorded script by FIRST-CALL ORDER: the first live
* session to make any call takes the first ordered script (the parent — earliest
* `createdAt`, and the first to stream because it must run before it delegates),
* the next new live session takes the next script, and so on. This keys by WHO
* calls rather than global call order, so it stays correct even if subagents
* ever run concurrently/backgrounded (a global cursor would interleave them).
*
* A call with no `sessionId` (a direct unit-test `ctx.llm.stream` that omits it)
* is treated as one anonymous session — it binds to the first script, so the
* single-session path behaves exactly as the old global cursor did.
*
* Each per-session cursor advances synchronously at listener-invocation time
* (not lazily inside the generator) so call ORDER within a session, not
* iteration order, fixes the mapping.
* @param ctx - the context whose `llm/stream` waterfall the listener short-circuits.
* @param config - the resolved fixture paths (env-var defaulting is `apply`'s job).
* @returns the `ctx.on` disposer that removes the listener.

14
packages/support/llm-replay/tests/llm-replay.spec.ts
View File

@@ -427,11 +427,8 @@ describe('loadSessionScripts', () => {
})
it('derives a FORK child script from its OWN events only (skips the seeded parent prefix)', () => {
// A fork child's log begins with the seeded parent prefix — the parent's
// events, INCLUDING its assistant/chunk events. Deriving the child script
// from the whole log would replay the PARENT's recorded responses as the
// child's model calls. With seedLength recorded, the child script must
// contain only the child's OWN chunks (those after the boundary).
// A fork log includes the parent's assistant chunks before `seedLength`. Deriving from the
// whole log would replay parent responses as child calls, so only child-owned chunks qualify.
const parentChunk: StreamChunk = { type: 'text-delta', index: 0, text: 'PARENT-RESPONSE' }
const childChunks: StreamChunk[] = [{ type: 'text-delta', index: 0, text: 'CHILD-RESPONSE' }, { type: 'finish', reason: { kind: 'stop' } }]
const f = writeSession('session.jsonl', { id: 'parent', createdAt: 100 }, [TEXT_CHUNKS])
@@ -488,11 +485,8 @@ describe('loadSessionScripts', () => {
})
it('keeps the primary first even when a child sorts BEFORE it in input order', () => {
// The primary is appended first internally but the child has an EARLIER
// createdAt — the primary must still win on the tie-break against a
// later-but-equal child, and lose only to a genuinely earlier child via
// createdAt (here the child is earlier, so order is child-then-primary only
// if createdAt strictly less; equal createdAt keeps primary first).
// The primary is appended first internally. A strictly earlier child sorts before it, while
// equal creation times preserve primary-first order regardless of input order.
const f = writeSession('session.jsonl', { id: 'parent', createdAt: 100 }, [TEXT_CHUNKS])
const earlier = writeSession('session.1.jsonl', { id: 'early', createdAt: 100 }, [TEXT_CHUNKS])
const scripts = loadSessionScripts({ file: f, childFiles: [earlier] })

17
packages/support/loader-smoke/README.md Normal file
View File

@@ -0,0 +1,17 @@
# `@deepseek-ai/dsh-loader-smoke`
Shared subprocess harness for keyless example smokes that boot the real stdio-agent bin and a real `cordis.yml` through the Cordis Loader. A test supplies absolute bin/config/tsconfig paths, optional environment overrides, and stdin lines; `runLoaderSmoke` owns the isolated cwd, DSH homes, tsx path resolution, 30-second process deadline, captured diagnostics, forced kill, EOF, and cleanup.
Successful runs return stdout and stderr only after a zero exit. Non-zero exits and deadlines reject with both captured streams. `LOADER_SMOKE_TEST_TIMEOUT_MS` leaves Vitest enough room for the process-owned diagnostic timeout to fire first.
This is support-tier test infrastructure, not product API. The consumers are the Loader-path smokes under `examples/{echo-agent,coding-agent,cordis-agent}`.
## Model Experience
None, as this test-only harness boots example processes and inspects their streams without changing an assembled model request.
## Known Limitations and Deferred Work
- **Only the unbuilt tsx/Loader path is exercised** — built-bin artifacts remain the responsibility of their separate e2e smokes.
- **Captured stdout and stderr are unbounded** — a runaway child can consume memory until the deadline kills it.
- **Timeout kills only the direct child** — a process tree spawned by a faulty fixture can outlive the smoke and needs external cleanup.

33
packages/support/loader-smoke/package.json Normal file
View File

@@ -0,0 +1,33 @@
{
"name": "@deepseek-ai/dsh-loader-smoke",
"description": "Shared subprocess harness for keyless real-Loader example smoke tests",
"version": "0.0.1",
"private": true,
"type": "module",
"main": "lib/index.js",
"types": "lib/types/index.d.ts",
"exports": {
".": {
"types": "./lib/types/index.d.ts",
"default": "./lib/index.js"
},
"./src/*": "./src/*",
"./package.json": "./package.json"
},
"files": [
"lib/index.js",
"lib/types/**/*.d.ts",
"lib/types/**/*.d.ts.map",
"src"
],
"license": "BSD-3-Clause",
"dependencies": {
"tsx": "^4.22.4"
},
"peerDependencies": {
"cordis": "^4.0.0-rc.6"
},
"devDependencies": {
"cordis": "^4.0.0-rc.6"
}
}

117
packages/support/loader-smoke/src/index.ts Normal file
View File

@@ -0,0 +1,117 @@
/**
* Shared subprocess harness for keyless example smokes that boot a real
* `cordis.yml` through the stdio-agent bin and Cordis Loader.
*
* @module @deepseek-ai/dsh-loader-smoke
*/
import { spawn } from 'node:child_process'
import { mkdtemp, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { fileURLToPath } from 'node:url'
const DEFAULT_PROCESS_TIMEOUT_MS = 30_000
const TSX_LOADER = fileURLToPath(import.meta.resolve('tsx'))
/** Vitest deadline that leaves room for the subprocess-owned 30-second diagnostic timeout. */
export const LOADER_SMOKE_TEST_TIMEOUT_MS = DEFAULT_PROCESS_TIMEOUT_MS + 15_000
/** Inputs that vary between real-Loader example smokes. */
export interface LoaderSmokeOptions {
/** Human-readable example name used in failure diagnostics. */
readonly label: string
/** Prefix for the isolated temporary process cwd. */
readonly tempDirPrefix: string
/** Absolute stdio-agent bin path. */
readonly binScript: string
/** Absolute real Loader config path. */
readonly configPath: string
/** Absolute repo tsconfig path used for unbuilt workspace-package resolution. */
readonly tsconfigPath: string
/** Environment overrides layered over the parent and isolated DSH homes. */
readonly env?: Readonly<NodeJS.ProcessEnv>
/** Lines written to stdin before EOF; omitted means immediate EOF. */
readonly stdinLines?: readonly string[]
/** Process deadline override for harness tests. */
readonly processTimeoutMs?: number
}
/** Captured output from a Loader smoke that exited successfully. */
export interface LoaderSmokeResult {
/** Complete stdout after clean exit. */
readonly stdout: string
/** Complete stderr after clean exit. */
readonly stderr: string
}
/**
* Boot one real Loader tree from an isolated cwd, write the requested stdin
* script, close stdin, and await a clean exit. The helper owns process kill and
* temp-directory cleanup on every outcome.
* @param options - example paths, environment, stdin, and diagnostic identity.
* @returns captured stdout and stderr after a zero exit.
*/
export async function runLoaderSmoke(options: LoaderSmokeOptions): Promise<LoaderSmokeResult> {
const cwd = await mkdtemp(join(tmpdir(), options.tempDirPrefix))
const processTimeoutMs = options.processTimeoutMs ?? DEFAULT_PROCESS_TIMEOUT_MS
try {
return await new Promise((resolve, reject) => {
const child = spawn(
process.execPath,
['--expose-internals', '--import', TSX_LOADER, options.binScript, options.configPath],
{
cwd,
env: {
...process.env,
DSH_HOME: join(cwd, '.dsh'),
DSH_AGENTS_HOME: join(cwd, '.agents'),
...options.env,
TSX_TSCONFIG_PATH: options.tsconfigPath,
},
stdio: ['pipe', 'pipe', 'pipe'],
},
)
let stdout = ''
let stderr = ''
let deferredFailure: Error | undefined
child.stdout.setEncoding('utf8')
child.stdout.on('data', (chunk: string) => { stdout += chunk })
child.stderr.setEncoding('utf8')
child.stderr.on('data', (chunk: string) => { stderr += chunk })
const timer = setTimeout(() => {
deferredFailure = new Error(`${options.label} did not exit within ${processTimeoutMs / 1_000}s. stdout:\n${stdout}\nstderr:\n${stderr}`)
child.kill('SIGKILL')
}, processTimeoutMs)
child.once('exit', (code) => {
clearTimeout(timer)
if (deferredFailure !== undefined) {
reject(deferredFailure)
} else if (code === 0) {
resolve({ stdout, stderr })
} else {
reject(new Error(`${options.label} exited ${String(code)}. stdout:\n${stdout}\nstderr:\n${stderr}`))
}
})
// process.execPath and a just-created pipe make these OS-error paths
// impractical to induce without replacing the boundary under test.
/* v8 ignore start */
child.once('error', (error) => {
clearTimeout(timer)
reject(new Error(`${options.label} failed to start: ${error.message}`))
})
child.stdin.once('error', (error) => {
deferredFailure ??= new Error(`${options.label} stdin failed: ${error.message}`)
child.kill('SIGKILL')
})
/* v8 ignore stop */
child.stdin.end((options.stdinLines ?? []).map(line => `${line}\n`).join(''))
})
} finally {
await rm(cwd, { recursive: true, force: true })
}
}

4
packages/support/loader-smoke/tests/fixtures/fail.ts vendored Normal file
View File

@@ -0,0 +1,4 @@
/** Non-zero subprocess fixture for the Loader-smoke harness. */
console.error('fixture failed')
process.exitCode = 7

4
packages/support/loader-smoke/tests/fixtures/hang.ts vendored Normal file
View File

@@ -0,0 +1,4 @@
/** Deadline subprocess fixture for the Loader-smoke harness. */
console.log('fixture hanging')
setInterval(() => {}, 1_000)

16
packages/support/loader-smoke/tests/fixtures/success.ts vendored Normal file
View File

@@ -0,0 +1,16 @@
/** Successful subprocess fixture for the Loader-smoke harness. */
let input = ''
process.stdin.setEncoding('utf8')
process.stdin.on('data', (chunk: string) => { input += chunk })
process.stdin.on('end', () => {
console.log(JSON.stringify({
configPath: process.argv[2],
cwd: process.cwd(),
dshHome: process.env.DSH_HOME,
agentsHome: process.env.DSH_AGENTS_HOME,
marker: process.env.LOADER_SMOKE_MARKER,
input,
}))
console.error('fixture stderr')
})

61
packages/support/loader-smoke/tests/loader-smoke.spec.ts Normal file
View File

@@ -0,0 +1,61 @@
import { existsSync } from 'node:fs'
import { fileURLToPath } from 'node:url'
import { describe, expect, it } from 'vitest'
import { LOADER_SMOKE_TEST_TIMEOUT_MS, runLoaderSmoke } from '@deepseek-ai/dsh-loader-smoke'
const configPath = '/tmp/fixture.cordis.yml'
const tsconfigPath = fileURLToPath(new URL('../../../../tsconfig.json', import.meta.url))
const fixture = (name: string): string => fileURLToPath(new URL(`./fixtures/${name}.ts`, import.meta.url))
const canonicalTempPath = (path: string): string => path.replace(/^\/private(?=\/var\/)/, '')
describe('runLoaderSmoke', () => {
it('isolates the process, writes stdin, captures output, and removes the cwd', async () => {
const result = await runLoaderSmoke({
label: 'success fixture',
tempDirPrefix: 'loader-smoke-success-',
binScript: fixture('success'),
configPath,
tsconfigPath,
env: { LOADER_SMOKE_MARKER: 'present' },
stdinLines: ['one', 'two'],
})
const output = JSON.parse(result.stdout) as {
configPath: string
cwd: string
dshHome: string
agentsHome: string
marker: string
input: string
}
expect(output).toMatchObject({
configPath,
marker: 'present',
input: 'one\ntwo\n',
})
expect(canonicalTempPath(output.dshHome)).toBe(`${canonicalTempPath(output.cwd)}/.dsh`)
expect(canonicalTempPath(output.agentsHome)).toBe(`${canonicalTempPath(output.cwd)}/.agents`)
expect(result.stderr).toContain('fixture stderr')
expect(existsSync(output.cwd)).toBe(false)
}, LOADER_SMOKE_TEST_TIMEOUT_MS)
it('rejects a non-zero exit with captured diagnostics', async () => {
await expect(runLoaderSmoke({
label: 'failure fixture',
tempDirPrefix: 'loader-smoke-fail-',
binScript: fixture('fail'),
configPath,
tsconfigPath,
})).rejects.toThrow('failure fixture exited 7. stdout:\n\nstderr:\nfixture failed')
})
it('kills a process at its deadline and reports captured output', async () => {
await expect(runLoaderSmoke({
label: 'hanging fixture',
tempDirPrefix: 'loader-smoke-hang-',
binScript: fixture('hang'),
configPath,
tsconfigPath,
processTimeoutMs: 100,
})).rejects.toThrow('hanging fixture did not exit within 0.1s.')
})
})

11
packages/support/loader-smoke/tsconfig.json Normal file
View File

@@ -0,0 +1,11 @@
{
"extends": "../../../tsconfig.base.json",
"compilerOptions": {
"rootDir": "src",
"outDir": "lib/types"
},
"include": [
"src"
],
"references": []
}

9
packages/support/subagent-mock/README.md
View File

@@ -18,3 +18,12 @@ Load it as a plugin (functional shape: `name`/`inject`/`Config`/`apply`, no defa
| `structured` | `{ reply }` | Structured value surfaced when a request carries an `outputSchema` and the capability is on. |
Aborting the required request signal or disposing before `result` settles flips the stop reason to `aborted`, so both holder-facing cancellation paths are observable.
## Model Experience
Indirectly, through `dsh-tool-subagent`, which renders this test provider's configured reply or stop-reason error into the parent test history.
## Known Limitations and Deferred Work
- **Scripted provider only** — it does not run a model, create a child agent, or exercise real prompt/tool-loop behavior.
- **One synthetic outcome per run** — it models no multi-turn, streaming, steering, resume, or subprocess transport behavior.

6
packages/support/subagent-mock/package.json
View File

@@ -25,7 +25,7 @@
"@deepseek-ai/dsh-agent": "^0.0.1",
"@deepseek-ai/dsh-llm": "^0.0.1",
"@deepseek-ai/dsh-subagent": "^0.0.1",
"cordis": "^4.0.0-rc.6"
"cordis": "^4.0.0-rc.7"
},
"dependencies": {
"schemastery": "^3.18.0"
@@ -34,7 +34,7 @@
"@deepseek-ai/dsh-agent": "workspace:^",
"@deepseek-ai/dsh-llm": "workspace:^",
"@deepseek-ai/dsh-subagent": "workspace:^",
"@cordisjs/plugin-loader": "^1.0.0-rc.4",
"cordis": "^4.0.0-rc.6"
"@cordisjs/plugin-loader": "^1.0.0-rc.5",
"cordis": "^4.0.0-rc.7"
}
}

19
packages/support/subagent-mock/src/index.ts
View File

@@ -1,13 +1,7 @@
/**
* A scripted {@link SubagentProvider} for testing the subagent seam WITHOUT a
* model or a real child agent. Mirrors `@deepseek-ai/dsh-llm-replay`: it lets a
* test drive the service and the model-facing tool through the REAL cordis
* Loader / export path, exercising registration, capability validation, the
* run lifecycle, and the structured-output branch deterministically.
*
* Plugin export shape: named `name`/`inject`/`Config`/`apply`, NO default —
* a functional plugin (it only registers a provider; it is never injected).
*
* Scripted, model-free subagent provider for deterministic coverage of registration,
* capability checks, lifecycle, the model-facing tool, and structured results through the real
* loader path. It is a named-export functional plugin; no default export.
* @module @deepseek-ai/dsh-subagent-mock
*/
@@ -28,12 +22,7 @@ const STOP_REASONS = ['completed', 'aborted', 'error', 'max-tokens', 'refusal']
const DEFAULT_CAPS: SubagentCapabilities = { outputSchema: true, depthLimit: true, toolFilter: true, persona: true }
/**
* A scripted provider: every {@link start} returns a ready run whose `result`
* resolves on the next task with the configured reply (and a structured value
* when the request asked for one and the capability is on). The required
* signal and `dispose()` both flip an unsettled result to `aborted`.
*/
/** Scripted provider whose configured result aborts if disposed or signalled first. */
class MockSubagentProvider implements SubagentProvider {
readonly capabilities: SubagentCapabilities
readonly inheritsParentContext: boolean

5
packages/support/subagent-mock/tests/subagent-mock.spec.ts
View File

@@ -105,10 +105,7 @@ describe('dsh-subagent-mock', () => {
})
it('has the namespace-plugin export shape (no stray default) so the Loader keeps name/inject/Config/apply', () => {
// Postmortem 0001 guard: this plugin HAS `inject = ['subagents']`, so a stray
// `export default apply` would collapse the module via `unwrapExports`
// (`exports.default ?? exports`), DROP `inject`, and crash at load with
// "cannot get property … without inject". Guard the shape directly.
// A default export would make Loader unwrap only that value and drop `inject`.
expect('default' in mock).toBe(false)
expect(mock.name).toBe('subagent-mock')
expect(mock.inject).toEqual(['subagents'])