Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into codex/truncated-design

Browse Source 
# Conflicts:
#	docs/capability-seams.md
#	docs/config-catalog.md
#	docs/cordis-catalog/services.md
#	docs/core-data-structures/core.md
#	docs/event-producer-consumer.md
#	docs/module-graph.md
#	docs/rfc/INDEX.md
#	docs/tool-catalog.md
#	examples/acp-agent/README.md
#	packages/README.md
#	packages/bash/bash/README.md
#	packages/core/tools/tests/gen-tool-catalog.spec.ts
#	packages/support/acp-snapshot/src/harness.ts
#	pnpm-lock.yaml
#	scripts/gen-doc-graphs.ts
#	scripts/gen-tool-catalog.ts
#	scripts/type-equiv.manifest.json
This commit is contained in:
Dudu-0223
2026-07-13 09:49:46 +08:00
parent 59025dde14 4cda9dd03d
commit 43535aab42
424 changed files with 35979 additions and 6313 deletions
Download Patch File Download Diff File Expand all files Collapse all files

18
packages/support/acp-snapshot/README.md
View File

@@ -5,8 +5,8 @@ The ACP snapshot suite kit: the shared machinery behind the keyless snapshot tie
Three layers, importable separately:
- **`runScenario` (harness)** — boots the real agent bin as a subprocess via tsx (unbuilt, Loader path), drives it over ACP JSON-RPC stdio from a deterministic `input.json` script, tees raw stdout for the golden + purity check, and harvests every persisted session JSONL (parent + subagent children, primary-first) after a graceful stdin-EOF shutdown. Parameterized by `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath` — absolute paths; the subprocess cwd is a temp dir outside the repo).
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept), and the composable `scrubRequestHeaders` (header bulk → `{{system}}`/`{{tools}}`, structure kept — [pinned-header RFC](../../../docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record-mode fixture write-back, the per-suite header pin with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin, non-pinning fixtures header-scrubbed). Must be called at vitest collection time.
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept), `scrubSystemPrompts` (prompt text → `{{system}}` in every JSONL), and `scrubRequestHeaders` (the remaining header bulk → `{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header RFC](../../../docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record/refresh fixture write-back, the per-header-class pin (`system-prompt.golden.md` plus the JSONL's full tool schemas) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt-scrubbed, non-pinning fixtures fully header-scrubbed). Must be called at vitest collection time.
A consuming `*.snapshot.ts` is the scenario table plus one factory call:
@@ -26,11 +26,17 @@ defineAcpSnapshotSuite({
tsconfigPath: fileURLToPath(new URL('../../../tsconfig.json', import.meta.url)),
},
snapshotsDir: join(dirname(fileURLToPath(import.meta.url)), 'snapshots'),
scenarios: SCENARIOS, // exactly one entry sets pinsHeader
mode: process.env.DSH_SNAPSHOT === 'record' ? 'record' : 'replay',
scenarios: SCENARIOS, // exactly one entry per header class sets pinsHeader
mode: process.env.DSH_SNAPSHOT === 'record'
? 'record'
: process.env.DSH_SNAPSHOT === 'refresh'
? 'refresh'
: 'replay',
})
```
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config RFC](../../../docs/rfc/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. Fixture roles, record/replay semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode scenarios are the template. Each pinning directory's generated `system-prompt.golden.md` is the reviewable snapshot of the normalized composed prompt; `session.jsonl` stores `"system":"{{system}}"` while retaining the complete tool list.
Constraints: `suite.ts` imports vitest, so the package is importable only inside a vitest run (the harness and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the harness speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug).
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config RFC](../../../docs/rfc/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log goldens, and each pin's Markdown prompt snapshot from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
Constraints: `suite.ts` imports vitest, so the package is importable only inside a vitest run (the harness and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the harness speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug). Session config options are scriptable too: the `setConfigOption` step switches a knob over `session/set_config_option`, and `setConfigOptionExpectError` asserts the bridge rejects an unknown id or out-of-vocabulary value (the error frame stays in the transcript).

33
packages/support/acp-snapshot/src/harness.ts
View File

@@ -85,6 +85,8 @@ export type InputStep =
| { op: 'promptExpectError'; text: string }
| { op: 'promptAndCancel'; text: string }
| { op: 'cancel' }
| { op: 'setConfigOption'; configId: string; value: string }
| { op: 'setConfigOptionExpectError'; configId: string; value: string }
/** A scenario's `input.json`: an ordered list of input steps. */
export interface InputScript {
@@ -166,6 +168,15 @@ export interface RunOptions {
* start from an empty workspace.
*/
workspaceDir?: string
/**
* Alternate LIVE config path for the boot (absolute), overriding
* {@link AgentUnderTest.configPath} for this run. A scenario needing a
* differently-composed tree (the Code Mode scenarios) ships an overlay
* whose basename still ends in `cordis.yml`, so the bin's replay swap
* resolves the sibling `*cordis.snapshot.yml` the same way it does for
* the default.
*/
configPath?: string
}
/**
@@ -205,6 +216,8 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
DSH_SNAPSHOT_FILE: opts.fixtureFile,
DSH_SNAPSHOT_SESSIONS_ROOT: sessionsRoot,
DSH_SNAPSHOT_SPILL_ROOT: spillRoot,
DSH_HOME: join(cwd, '.dsh'),
DSH_AGENTS_HOME: join(cwd, '.agents'),
...opts.overrideFile !== undefined ? { DSH_SNAPSHOT_OVERRIDE: opts.overrideFile } : {},
...opts.childFiles !== undefined && opts.childFiles.length > 0
? { DSH_SNAPSHOT_CHILD_FILES: opts.childFiles.join(delimiter) }
@@ -213,7 +226,7 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
child = spawn(
process.execPath,
['--import', tsxLoader, opts.agent.binScript, opts.agent.configPath],
['--import', tsxLoader, opts.agent.binScript, opts.configPath ?? opts.agent.configPath],
{ cwd, env, stdio: ['pipe', 'pipe', 'pipe'] },
)
@@ -402,6 +415,24 @@ async function runStep(
await client.cancel({ sessionId })
return
}
case 'setConfigOption': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: setConfigOption before newSession')
await client.setSessionConfigOption({ sessionId, configId: step.configId, value: step.value })
return
}
case 'setConfigOptionExpectError': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: setConfigOptionExpectError before newSession')
// The bridge rejects an unknown id / out-of-vocabulary value; the SDK
// surfaces that as a rejected RPC — swallow it so the run completes and
// the error frame is captured in the transcript.
await client.setSessionConfigOption({ sessionId, configId: step.configId, value: step.value }).then(
() => { throw new Error('snapshot-harness: expected set_config_option to be rejected but it succeeded') },
() => { /* expected: the bridge rejected the id or value */ },
)
return
}
default:
throw new Error(`snapshot-harness: unknown input op ${JSON.stringify(step)}`)
}

3
packages/support/acp-snapshot/src/index.ts
View File

@@ -3,7 +3,7 @@
* tier (`pnpm run test:snapshot`). Three layers, composable per example:
* the subprocess scenario harness ({@link runScenario}), the pure golden
* normalizers ({@link normalizeStdout} / {@link normalizeSessionLog} /
* {@link scrubRequestHeaders}), and the suite factory
* {@link scrubRequestHeaders} / {@link scrubSystemPrompts}), and the suite factory
* ({@link defineAcpSnapshotSuite}) that registers a scenario table as a full
* describe/it tree. An example's `*.snapshot.ts` supplies only its
* {@link AgentUnderTest} paths, its snapshots directory, and its
@@ -29,6 +29,7 @@ export {
normalizeSessionLog,
normalizeStdout,
scrubRequestHeaders,
scrubSystemPrompts,
type NormalizeContext,
} from './normalize.ts'
export {

81
packages/support/acp-snapshot/src/normalize.ts
View File

@@ -12,13 +12,14 @@
* `durationMs` (wall-clock hook runtime) → 0. NOT scrubbed: the log's `seq`
* (deterministic — `seq = log.length`, part of the event-log contract).
*
* A separate, composable normalizer — {@link scrubRequestHeaders} — replaces
* the bulky request-header CONTENT (the composed system prompt and the tool
* schema list) with `{{system}}`/`{{tools}}` tokens. It is deliberately NOT
* folded into {@link normalizeSessionLog}: each suite's one header-pinning
* scenario compares that content verbatim, every other scenario composes the
* scrub in (the `pinsHeader` flag on the scenario table, consumed by the suite
* factory in ./suite.ts; see the pinned-header RFC,
* Separate, composable normalizers keep bulky request-header content out of
* session fixtures. {@link scrubSystemPrompts} replaces the composed system
* prompt in EVERY fixture; {@link scrubRequestHeaders} additionally replaces
* tool schemas and the session prefix outside each suite's header-pinning
* scenario. They are deliberately NOT folded into
* {@link normalizeSessionLog}: the suite factory composes the right scrub for
* each scenario and snapshots the pin's actual prompt as Markdown (see the
* pinned-header RFC,
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
*
* See docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
@@ -30,6 +31,7 @@ const SESSION_ID = '{{sessionId}}'
const CWD = '{{cwd}}'
const SYSTEM = '{{system}}'
const TOOLS = '{{tools}}'
const MESSAGE_PREFIX = '{{messagePrefix}}'
/** A UUID v4 string, the shape `randomUUID()` produces for session ids. */
const UUID_RE = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi
@@ -146,30 +148,37 @@ export function normalizeSessionLog(rawLog: string, ctx: NormalizeContext): stri
}
/**
* Replace request-header CONTENT in a session JSONL with stable tokens,
* keeping its structure: a `request/header` event's `data.header.system` →
* `{{system}}` and `data.header.tools` → `{{tools}}`; a
* `request/header-delta` event keeps every structural fact — the system
* delta's `keepStart`/`keepEnd` line positions and inserted-line COUNT (one
* `{{system}}` token per inserted line), the tools delta's
* added/removed/changed tool NAMES — and tokenizes only the bulk (prompt
* text; each added/changed schema's fields other than `name` → `{{tools}}`),
* so two different deltas still compare different.
* Absent fields stay absent — WHETHER a header carried a system prompt or
* tools is behavior and stays visible; `config` and `reason` are small and
* stable, so they stay verbatim (a model swap churns every fixture by design
* — it invalidates the recorded responses; a prompt/schema edit churns none —
* replay never reads this content, see dsh-llm-replay).
*
* Only lines with something to scrub are re-serialized; every other line
* passes through byte-for-byte, so the transform is idempotent and applying
* it to an already-scrubbed fixture is a no-op — the on-disk-fixtures guard
* in ./suite.ts relies on exactly that.
* Replace system-prompt content in request headers and header deltas with
* `{{system}}` tokens while retaining field presence and delta structure.
* Other header content stays verbatim, so a header-pinning fixture can keep
* its complete tool schemas while every JSONL fixture omits the prompt text.
* Lines without a system payload pass through byte-for-byte; the transform is
* idempotent.
*
* @param rawLog The raw session `.jsonl` content.
* @returns The JSONL with header content tokenized, other lines byte-identical.
* @returns The JSONL with system-prompt content tokenized.
*/
export function scrubSystemPrompts(rawLog: string): string {
return scrubHeaderContent(rawLog, false)
}
/**
* Replace all bulky request-header content in a session JSONL with stable
* tokens. This includes the system-prompt fields handled by
* {@link scrubSystemPrompts}, tool schemas, and session-prefix messages. It
* keeps system-delta line positions and arity, tool-delta names, prefix
* message counts, field presence, config, and reason. Lines without content
* to scrub pass through byte-for-byte, and the transform is idempotent.
*
* @param rawLog The raw session `.jsonl` content.
* @returns The JSONL with all header bulk tokenized, other lines byte-identical.
*/
export function scrubRequestHeaders(rawLog: string): string {
return scrubHeaderContent(rawLog, true)
}
/** Transform header content, optionally including tool schemas and the session prefix. */
function scrubHeaderContent(rawLog: string, scrubToolsAndPrefix: boolean): string {
const lines = rawLog.split('\n')
const out = lines.map((line) => {
if (line.trim().length === 0) return line
@@ -179,10 +188,14 @@ export function scrubRequestHeaders(rawLog: string): string {
if (record.type === 'request/header') {
const header = data.header as Record<string, unknown> | null | undefined
if (header === null || typeof header !== 'object') return line
if (!('system' in header) && !('tools' in header)) return line
if ('system' in header) header.system = SYSTEM
if ('tools' in header) header.tools = TOOLS
return JSON.stringify(record)
let touched = false
if ('system' in header) { header.system = SYSTEM; touched = true }
if (scrubToolsAndPrefix && 'tools' in header) { header.tools = TOOLS; touched = true }
if (scrubToolsAndPrefix && Array.isArray(header.messagePrefix)) {
header.messagePrefix = header.messagePrefix.map(() => MESSAGE_PREFIX)
touched = true
}
return touched ? JSON.stringify(record) : line
}
if (record.type === 'request/header-delta') {
let touched = false
@@ -192,10 +205,14 @@ export function scrubRequestHeaders(rawLog: string): string {
touched = true
}
const tools = data.tools as Record<string, unknown> | null | undefined
if (tools !== null && typeof tools === 'object') {
if (scrubToolsAndPrefix && tools !== null && typeof tools === 'object') {
if (Array.isArray(tools.added)) { tools.added = tools.added.map(scrubToolSchema); touched = true }
if (Array.isArray(tools.changed)) { tools.changed = tools.changed.map(scrubToolSchema); touched = true }
}
if (scrubToolsAndPrefix && Array.isArray(data.messagePrefix)) {
data.messagePrefix = data.messagePrefix.map(() => MESSAGE_PREFIX)
touched = true
}
return touched ? JSON.stringify(record) : line
}
return line

487
packages/support/acp-snapshot/src/suite.ts
View File

@@ -10,20 +10,23 @@
* (recorded scenarios) and the expected produced log (both sides normalized
* before comparing).
*
* Request-header content (the composed system prompt + tool schemas riding on
* `request/header` events) is pinned by exactly ONE scenario per suite — the
* one with `pinsHeader` — and scrubbed to `{{system}}`/`{{tools}}` tokens in
* every other fixture and compare, so a prompt or tool-schema edit churns one
* committed line instead of every fixture. A per-run uniformity guard keeps
* the single pin sound: every live header must equal the pinned one, and no
* header-delta may appear outside the pinning scenario (see the
* Request-header content is pinned by exactly ONE scenario per HEADER CLASS —
* scenarios that boot the same config compose the same header. Every JSONL
* fixture scrubs the system prompt to `{{system}}`; each class's pinning
* scenario stores the readable prompt in `system-prompt.golden.md` and keeps its full
* tool schemas in `session.jsonl`, while every other fixture also scrubs tools
* to `{{tools}}`. A per-run uniformity guard compares both artifacts against
* every live header and forbids unrepresented header deltas (see the
* pinned-header RFC,
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
*
* `pnpm run test:snapshot:record` (DSH_SNAPSHOT=record + -u) re-records the
* `session.jsonl` fixtures against the real API and refreshes the stdout golden
* in one pass; the caller resolves that env into {@link SnapshotSuiteOptions}
* (env reading stays at the suite edge, not in this library).
* in one pass. `pnpm run test:snapshot:refresh` (DSH_SNAPSHOT=refresh) instead
* replays the committed model scripts keylessly and writes the current stdout
* + persisted-log goldens back without calling a live LLM. The caller resolves
* that env into {@link SnapshotSuiteOptions} (env reading stays at the suite
* edge, not in this library).
*
* @module @deepseek-ai/dsh-acp-snapshot/suite
*/
@@ -33,7 +36,16 @@ import { existsSync } from 'node:fs'
import { join } from 'node:path'
import { describe, expect, it } from 'vitest'
import { type AgentUnderTest, type HarvestedLog, type InputScript, runScenario } from './harness.ts'
import { type NormalizeContext, normalizeSessionLog, normalizeStdout, scrubRequestHeaders } from './normalize.ts'
import {
type NormalizeContext,
normalizeSessionLog,
normalizeStdout,
scrubRequestHeaders,
scrubSystemPrompts,
} from './normalize.ts'
/** The readable system-prompt snapshot beside each header-pinning fixture. */
const SYSTEM_PROMPT_SNAPSHOT = 'system-prompt.golden.md'
/** A snapshot scenario and how its fixtures are produced. */
export interface Scenario {
@@ -77,21 +89,49 @@ export interface Scenario {
*/
childSessions?: number
/**
* Whether THIS scenario's fixtures keep the full request-header content (the
* composed system prompt and tool schema list on `request/header` /
* `request/header-delta` events) and compare it verbatim. Exactly one
* scenario per suite pins it; every other scenario stores and compares that
* content as `{{system}}`/`{{tools}}` tokens ({@link scrubRequestHeaders}),
* so a system prompt or tool-schema change shows up as ONE committed-fixture
* diff, not one per scenario. One pin suffices because header composition is
* suite-uniform (parent, spawn child, and fork child all compose the same
* prompt-modulo-cwd and the same tools) — and that premise is ASSERTED, not
* assumed: every non-pinning run's live headers must equal the pinned
* fixture's (normalized), so a session-dependent header (say, a restricted
* subagent toolset) fails loud until it gets its own pinning scenario.
* Whether THIS scenario pins its header class's model-facing request-header
* content. Its actual composed prompt is maintained as a readable
* `system-prompt.golden.md`; its JSONL keeps full tool schemas but stores the prompt
* as `{{system}}`. Every other scenario of the class stores tools as
* `{{tools}}` too ({@link scrubRequestHeaders}). A prompt or tool-schema
* change therefore shows up in one focused artifact per class, not every
* session fixture. One pin per class suffices because
* header composition is class-uniform (parent, spawn child, and fork child
* all compose the same prompt-modulo-cwd and the same tools) — and that
* premise is ASSERTED, not assumed: every non-pinning run's live headers
* must equal its class's pinned fixture's (normalized), so a
* session-dependent header (say, a restricted subagent toolset) fails loud
* until it gets its own pinning scenario.
* Defaults to false.
*/
pinsHeader?: boolean
/**
* How many `request/header-delta` events this PINNING scenario's fixture
* legitimately carries (default 0). A recorded mid-run header change — a
* config-option switch rewriting a prompt section — is part of the pinned
* surface, with readable prompt text in Markdown; any OTHER count
* still fails, so fixture rot stays caught. Meaningless off the pin (the
* live uniformity guard keeps non-pinning scenarios delta-free).
*/
expectedHeaderDeltas?: number
/**
* Which header-composition class this scenario belongs to. Scenarios that
* boot the same config compose the same header; each class has exactly one
* {@link pinsHeader} scenario, and the uniformity guard compares every
* other member against ITS class's pin. Defaults to `'default'`; a
* scenario booting an alternate config ({@link configPath}) whose tool
* list or prompt sections differ by construction carries its own class.
*/
headerClass?: string
/**
* Alternate LIVE config path (absolute) this scenario boots instead of
* {@link AgentUnderTest.configPath} — an overlay composing a different
* tree (its basename must still end in `cordis.yml` so the bin's replay
* swap finds the sibling `*cordis.snapshot.yml`). A scenario whose
* overlay changes the composed header also needs its own
* {@link headerClass}.
*/
configPath?: string
}
/** One suite's inputs: the agent to boot, where its fixtures live, and its scenario table. */
@@ -100,15 +140,16 @@ export interface SnapshotSuiteOptions {
agent: AgentUnderTest
/** Absolute path of the suite's `snapshots/` directory (one subdir per scenario). */
snapshotsDir: string
/** The scenario table; exactly one entry must set `pinsHeader`. */
/** The scenario table; exactly one entry per header class must set `pinsHeader`. */
scenarios: Scenario[]
/**
* `replay` (keyless, the default tier) or `record` (live API; re-records the
* `recorded` scenarios' fixtures and refreshes the vitest goldens under
* `--update`). The caller derives this from `$DSH_SNAPSHOT` — env reading
* stays outside this library.
* `replay` (keyless, the default tier), `record` (live API; re-records the
* `recorded` scenarios' fixtures and refreshes the Vitest goldens under
* `--update`), or `refresh` (keyless replay that rewrites stdout goldens and
* comparable session fixtures from the replay run). The caller derives this
* from `$DSH_SNAPSHOT` — env reading stays outside this library.
*/
mode: 'replay' | 'record'
mode: 'replay' | 'record' | 'refresh'
}
/**
@@ -167,6 +208,86 @@ export function normalizedHeaders(rawLog: string, ctx: NormalizeContext): unknow
.map(record => record.data?.header)
}
/**
* The normalized string-valued system prompts carried by request headers in a
* session JSONL, in log order. Headers without a string prompt are omitted so
* callers can assert one prompt per header explicitly.
*
* @param rawLog The session `.jsonl` content to inspect.
* @param ctx The volatile values of the run that produced it.
* @returns The normalized system prompts, in header order.
*/
export function normalizedSystemPrompts(rawLog: string, ctx: NormalizeContext): string[] {
return normalizedHeaders(rawLog, ctx).flatMap((header) => {
if (header === null || typeof header !== 'object') return []
const system = (header as { system?: unknown }).system
return typeof system === 'string' ? [system] : []
})
}
/** One normalized system-prompt edit carried by a `request/header-delta`. */
export interface SystemPromptDeltaSnapshot {
/** How many leading lines remain from the prior prompt. */
keepStart: number
/** How many trailing lines remain from the prior prompt. */
keepEnd: number
/** The normalized replacement lines inserted between the retained ranges. */
insert: string[]
}
/**
* Extract normalized system-prompt edits from request-header deltas in log
* order. Deltas without a well-formed system edit are omitted; their non-prompt
* structure remains pinned in JSONL.
*
* @param rawLog The session `.jsonl` content to inspect.
* @param ctx The volatile values of the run that produced it.
* @returns The normalized system-prompt edits, in event order.
*/
export function normalizedSystemPromptDeltas(rawLog: string, ctx: NormalizeContext): SystemPromptDeltaSnapshot[] {
return normalizeSessionLog(rawLog, ctx)
.split('\n')
.filter(line => line.trim().length > 0)
.map(line => JSON.parse(line) as { type?: unknown; data?: { system?: unknown } })
.filter(record => record.type === 'request/header-delta')
.flatMap((record) => {
const system = record.data?.system
if (system === null || typeof system !== 'object') return []
const { keepStart, keepEnd, insert } = system as { keepStart?: unknown; keepEnd?: unknown; insert?: unknown }
if (typeof keepStart !== 'number' || typeof keepEnd !== 'number' || !Array.isArray(insert)) return []
if (!insert.every(line => typeof line === 'string')) return []
return [{ keepStart, keepEnd, insert: insert }]
})
}
/**
* Render a normalized prompt as a repository-friendly Markdown snapshot.
* Prompt text is unchanged except that a missing terminal newline is added so
* the committed file follows the repository newline contract.
*
* @param prompt The normalized system prompt.
* @param deltas Normalized prompt edits to append as readable sections.
* @returns Markdown snapshot text ending in a newline.
*/
export function formatSystemPromptSnapshot(
prompt: string,
deltas: readonly SystemPromptDeltaSnapshot[] = [],
): string {
let snapshot = prompt.endsWith('\n') ? prompt : `${prompt}\n`
for (const [index, delta] of deltas.entries()) {
snapshot += `\n<!-- request/header-delta ${index + 1}: keepStart=${delta.keepStart}, keepEnd=${delta.keepEnd} -->\n\n`
const insert = delta.insert.join('\n')
snapshot += insert.endsWith('\n') ? insert : `${insert}\n`
}
return snapshot
}
/** Return the initial-prompt portion of a possibly delta-bearing snapshot. */
function initialSystemPromptSnapshot(snapshot: string): string {
const marker = snapshot.indexOf('\n<!-- request/header-delta ')
return marker < 0 ? snapshot : snapshot.slice(0, marker)
}
/**
* Count the `request/header-delta` events in a session JSONL.
*
@@ -180,43 +301,147 @@ export function headerDeltaCount(rawLog: string): number {
.length
}
/** A literal string replacement used to carry an existing fixture's volatile value into a refreshed log. */
export interface FixtureReplacement {
/** The fresh replay-run value to replace. */
from: string
/** The existing fixture value to keep. */
to: string
}
function parseJsonlRecords(text: string): Record<string, unknown>[] {
return text.split('\n')
.filter(line => line.trim().length > 0)
.map(line => JSON.parse(line) as Record<string, unknown>)
}
/**
* Build the cross-log id/cwd replacements used by refresh write-back.
*
* @param logs The freshly harvested logs, in fixture order.
* @param fixtures The existing fixture contents, in matching order.
* @returns Literal replacements from fresh volatile values to the fixture's old values.
*/
export function refreshFixtureReplacements(logs: HarvestedLog[], fixtures: string[]): FixtureReplacement[] {
const replacements: FixtureReplacement[] = []
for (let i = 0; i < logs.length; i++) {
const fresh = parseJsonlRecords((logs[i] as HarvestedLog).content)[0]
const existing = parseJsonlRecords(fixtures[i] ?? '')[0]
for (const field of ['id', 'cwd'] as const) {
const from = fresh?.[field]
const to = existing?.[field]
if (typeof from === 'string' && typeof to === 'string' && from.length > 0 && from !== to) {
replacements.push({ from, to })
}
}
}
return replacements
}
function preserveFixtureVolatiles(record: Record<string, unknown>, existing: Record<string, unknown> | undefined): void {
if (existing === undefined || existing.type !== record.type) return
if (record.type === 'session') {
for (const field of ['id', 'createdAt', 'cwd', 'parentSession'] as const) {
if (field in record && field in existing) record[field] = existing[field]
}
return
}
if ('time' in record && 'time' in existing) record.time = existing.time
if (record.type !== 'hook/result') return
const data = record.data
const existingData = existing.data
if (
data !== null && typeof data === 'object'
&& existingData !== null && typeof existingData === 'object'
&& 'durationMs' in data && 'durationMs' in existingData
) {
(data as Record<string, unknown>).durationMs = (existingData as Record<string, unknown>).durationMs
}
}
/**
* Rewrite a fresh replay-produced log so repeated refreshes do not churn
* volatile fixture fields. Meaningful event payloads come from `fresh`; the
* existing fixture lends session ids, cwd, creation times, event times, and
* hook durations where the record shape still matches.
*
* @param fresh The newly harvested session JSONL.
* @param existing The committed fixture JSONL being refreshed.
* @param replacements Cross-log literal replacements from {@link refreshFixtureReplacements}.
* @returns The stabilized JSONL content to write back.
*/
export function stabilizeRefreshLog(fresh: string, existing: string, replacements: FixtureReplacement[]): string {
let stable = fresh
for (const { from, to } of replacements) stable = stable.split(from).join(to)
const existingRecords = parseJsonlRecords(existing)
const records = parseJsonlRecords(stable)
for (let i = 0; i < records.length; i++) {
preserveFixtureVolatiles(records[i] as Record<string, unknown>, existingRecords[i])
}
return records.map(record => JSON.stringify(record)).join('\n') + '\n'
}
/**
* Register the suite: one `describe` per scenario (the golden/log compares and
* the header-uniformity guard) plus the fixture guard block (no orphan
* scenario dirs, required files present, exactly one pin, non-pinning fixtures
* header-scrubbed). Must run at vitest collection time — it calls
* `describe`/`it`. Throws immediately if no scenario pins the header (the
* uniformity guard would have nothing to compare against).
* scenario dirs, required files present, exactly one pin per header class,
* pinning fixtures well-formed, every JSONL prompt-scrubbed, non-pinning
* fixtures fully header-scrubbed). Must
* run at vitest collection time — it calls `describe`/`it`. Throws
* immediately if any header class lacks a pinning scenario or carries two
* (the uniformity guard needs exactly one comparison anchor per class).
*
* @param options The agent, snapshots directory, scenario table, and mode.
*/
export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
const { agent, snapshotsDir, scenarios, mode } = options
const RECORDING = mode === 'record'
const REFRESHING = mode === 'refresh'
const childMode: 'replay' | 'record' = RECORDING ? 'record' : 'replay'
/** The suite's single header-pinning scenario. Guarded here (and by a meta-test) so the pin cannot silently vanish. */
const pinningScenario = scenarios.find(s => s.pinsHeader === true)
if (pinningScenario === undefined) throw new Error('acp-snapshot: no scenario pins the request-header content')
/** The class a scenario's header composition belongs to (see {@link Scenario.headerClass}). */
const classOf = (scenario: Scenario): string => scenario.headerClass ?? 'default'
/** Each header class's single pinning scenario. Guarded here (and by meta-tests) so a pin cannot silently vanish or split. */
const pinningByClass = new Map<string, Scenario>()
for (const scenario of scenarios) {
if (scenario.pinsHeader !== true) continue
const cls = classOf(scenario)
const existing = pinningByClass.get(cls)
if (existing) throw new Error(`acp-snapshot: header class "${cls}" pinned by both ${existing.name} and ${scenario.name}`)
pinningByClass.set(cls, scenario)
}
for (const scenario of scenarios) {
if (!pinningByClass.has(classOf(scenario))) {
throw new Error(`acp-snapshot: no scenario pins the request-header content of class "${classOf(scenario)}" (needed by ${scenario.name})`)
}
}
for (const scenario of scenarios) {
describe(`snapshot: ${scenario.name}`, () => {
// In RECORD mode, only re-run the `recorded` (live-API) scenarios; the
// `authored` ones (sidecar-driven errors/cancel) are never re-recorded.
// REFRESH mode is replay-backed and deterministic, so it runs every
// scenario and rewrites the comparable fixtures from that replay run.
it.skipIf(RECORDING && !scenario.recorded)('matches the goldens', async () => {
const dir = join(snapshotsDir, scenario.name)
const input = JSON.parse(await readFile(join(dir, 'input.json'), 'utf8')) as InputScript
const overrideFile = join(dir, 'replay.override.json')
const workspaceDir = join(dir, 'workspace')
const childSessions = scenario.childSessions ?? 0
const comparesLog = scenario.comparesLog ?? scenario.hasModelTurn
const result = await runScenario(input, {
agent,
mode,
mode: childMode,
fixtureFile: join(dir, 'session.jsonl'),
...existsSync(overrideFile) ? { overrideFile } : {},
// In REPLAY, forward the recorded child fixtures so each subagent session
// replays from its own script. In RECORD they are harvested, not read.
...!RECORDING && childSessions > 0 ? { childFiles: childFixturePaths(dir, childSessions) } : {},
...existsSync(workspaceDir) ? { workspaceDir } : {},
// A scenario booting an overlay tree passes its own live config; the
// bin's replay swap derives the sibling `*cordis.snapshot.yml` from it.
...scenario.configPath !== undefined ? { configPath: scenario.configPath } : {},
})
// Scrub every volatile id the run produced: the ACP server-issued session
@@ -232,43 +457,73 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
}
// RECORD mode (recorded model scenarios only): persist the freshly-harvested
// logs back to their fixtures — the primary to session.jsonl, each child to
// session.<n>.jsonl in harvest order. `--update` refreshes the Vitest
// goldens but NOT these fixtures, so write them here. A non-pinning
// scenario's fixtures are written header-scrubbed, so a re-record can
// never smuggle the full prompt/schema content back into every fixture.
// live logs back to their fixtures. REFRESH mode does the same from a
// keyless replay run for every comparable log, including authored
// scenarios that live record deliberately skips. The primary goes to
// session.jsonl, each child to session.<n>.jsonl in harvest order. A
// Every fixture is written with its system prompt scrubbed. A pinning
// scenario keeps the remaining header content (notably tool schemas);
// every other scenario scrubs that bulk too. Record/refresh therefore
// cannot smuggle prompt text back into JSONL or duplicate schemas.
const scrub = scenario.pinsHeader === true
? (log: string): string => log
? scrubSystemPrompts
: scrubRequestHeaders
if (RECORDING && scenario.recorded && scenario.hasModelTurn) {
expect(result.sessionLogs.length, 'record produced no session log to harvest').toBeGreaterThan(0)
const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)]
const existingFixtures = REFRESHING
? await Promise.all(fixtureFiles.map(file => readFile(join(dir, file), 'utf8')))
: []
const replacements = REFRESHING ? refreshFixtureReplacements(result.sessionLogs, existingFixtures) : []
const writesSessionFixtures = (RECORDING && scenario.recorded && scenario.hasModelTurn)
|| (REFRESHING && comparesLog)
if (writesSessionFixtures) {
expect(result.sessionLogs.length, `${mode} produced no session log to harvest`).toBeGreaterThan(0)
expect(result.sessionLogs.length, `expected ${childSessions + 1} session logs (parent + children)`)
.toBe(childSessions + 1)
await writeFile(join(dir, 'session.jsonl'), scrub((result.sessionLogs[0] as HarvestedLog).content))
const primary = (result.sessionLogs[0] as HarvestedLog).content
await writeFile(join(dir, 'session.jsonl'), scrub(
REFRESHING ? stabilizeRefreshLog(primary, existingFixtures[0] as string, replacements) : primary,
))
for (let i = 1; i < result.sessionLogs.length; i++) {
await writeFile(join(dir, `session.${i}.jsonl`), scrub((result.sessionLogs[i] as HarvestedLog).content))
const child = (result.sessionLogs[i] as HarvestedLog).content
await writeFile(join(dir, `session.${i}.jsonl`), scrub(
REFRESHING ? stabilizeRefreshLog(child, existingFixtures[i] as string, replacements) : child,
))
}
if (scenario.pinsHeader === true) {
const prompts = result.sessionLogs.flatMap(log => normalizedSystemPrompts(log.content, ctx))
expect(prompts.length, `${mode} produced no system prompt to snapshot`).toBeGreaterThan(0)
const initialSnapshot = formatSystemPromptSnapshot(prompts[0] as string)
for (const prompt of prompts) {
expect(formatSystemPromptSnapshot(prompt), 'the pinning run produced divergent system prompts')
.toEqual(initialSnapshot)
}
const primary = result.sessionLogs[0] as HarvestedLog
const snapshot = formatSystemPromptSnapshot(
prompts[0] as string,
normalizedSystemPromptDeltas(primary.content, ctx),
)
await writeFile(join(dir, SYSTEM_PROMPT_SNAPSHOT), snapshot)
}
}
await expect(normalizeStdout(result.rawStdout, ctx))
.toMatchFileSnapshot(join(dir, 'stdout.golden.jsonl'))
const stdout = normalizeStdout(result.rawStdout, ctx)
if (REFRESHING) {
await writeFile(join(dir, 'stdout.golden.jsonl'), stdout)
}
await expect(stdout).toMatchFileSnapshot(join(dir, 'stdout.golden.jsonl'))
// A model turn always produces a log worth comparing; a hook scenario can
// produce one without a model turn (a `rejected` turn carrying `hook/*`).
const comparesLog = scenario.comparesLog ?? scenario.hasModelTurn
if (comparesLog) {
// The harvested logs (primary-first) must match their committed fixtures
// 1:1. Each side passes through normalizeSessionLog, scrubbed against ITS
// OWN volatile values — the live run's via `ctx`, the committed fixture's
// via its own header (a committed file cannot share the live run's ids).
// Unless this scenario pins the header, both sides ALSO pass through
// scrubRequestHeaders: the live log carries the real prompt/schemas, the
// fixture carries the `{{system}}`/`{{tools}}` tokens, and the scrub is
// idempotent — so the compare checks the header's presence, position,
// reason, and config, but not its bulk content (pinned once, in the
// `pinsHeader` scenario).
// Both sides pass through the scenario's idempotent scrub: every live
// prompt becomes the fixture's `{{system}}`; non-pinning scenarios
// additionally tokenize tools/prefix. The dedicated header guard below
// compares those omitted values against their class's pin artifacts.
expect(result.sessionLogs.length, 'this scenario must persist a session log').toBe(childSessions + 1)
const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)]
for (let i = 0; i < fixtureFiles.length; i++) {
const harvested = scrub((result.sessionLogs[i] as HarvestedLog).content)
const fixture = scrub(await readFile(join(dir, fixtureFiles[i] as string), 'utf8'))
@@ -277,31 +532,42 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
}
}
// Header-uniformity guard: the single pin is sound only while every
// session in the suite composes the SAME header and keeps it for the
// whole run. Assert both halves live. (1) Every request/header the run
// produced (parent, spawn child, fork child, initial or resume) must
// equal the pinned fixture's header after each side is normalized
// against its own volatile values. (2) No request/header-delta may
// appear at all — a mid-run header change diverges from the pin by
// construction, and its content would be invisible under the scrub. If
// either fails, either the header changed (update the pin: re-record or
// hand-edit the pinning scenario's fixture) or composition became
// session-dependent by design (give the divergent shape its own
// pinning scenario).
if (scenario.pinsHeader !== true) {
const pinnedFixture = await readFile(join(snapshotsDir, pinningScenario.name, 'session.jsonl'), 'utf8')
const pinned = normalizedHeaders(pinnedFixture, fixtureContext(pinnedFixture))
expect(pinned.length, `the pinning fixture (${pinningScenario.name}) must carry exactly one request/header`)
.toBe(1)
for (const log of result.sessionLogs) {
expect(headerDeltaCount(log.content), `session ${log.id}: a request/header-delta in a non-pinning scenario`)
.toBe(0)
const headers = normalizedHeaders(log.content, ctx)
for (const [k, header] of headers.entries()) {
expect(header, `session ${log.id}: request/header #${k + 1} diverged from the pinned (${pinningScenario.name}) header`)
.toEqual(pinned[0])
}
// Header-uniformity guard: every live header in a class must equal the
// class pin split across its JSONL header (system token + real tools)
// and readable Markdown prompt. A pinning scenario may carry its
// declared header deltas; their prompt edits live in the Markdown
// golden while JSONL retains the tokenized edit structure.
/* v8 ignore next -- construction guarantees the pin exists; a miss would fail the one-header assertion loudly. */
const pinningScenario = pinningByClass.get(classOf(scenario)) ?? scenario
const pinningDir = join(snapshotsDir, pinningScenario.name)
const pinnedFixture = await readFile(join(pinningDir, 'session.jsonl'), 'utf8')
const pinned = normalizedHeaders(pinnedFixture, fixtureContext(pinnedFixture))
const promptSnapshot = await readFile(join(pinningDir, SYSTEM_PROMPT_SNAPSHOT), 'utf8')
const initialPromptSnapshot = initialSystemPromptSnapshot(promptSnapshot)
expect(pinned.length, `the pinning fixture (${pinningScenario.name}) must carry exactly one request/header`)
.toBe(1)
for (const [logIndex, log] of result.sessionLogs.entries()) {
const expectedDeltas = scenario.pinsHeader === true && logIndex === 0
? scenario.expectedHeaderDeltas ?? 0
: 0
expect(headerDeltaCount(log.content), `session ${log.id}: request/header-delta count`)
.toBe(expectedDeltas)
const headers = normalizedHeaders(scrubSystemPrompts(log.content), ctx)
const prompts = normalizedSystemPrompts(log.content, ctx)
expect(prompts.length, `session ${log.id}: every request/header must carry a string system prompt`)
.toBe(headers.length)
for (const [k, header] of headers.entries()) {
expect(header, `session ${log.id}: request/header #${k + 1} diverged from the pinned (${pinningScenario.name}) header`)
.toEqual(pinned[0])
expect(formatSystemPromptSnapshot(prompts[k] as string), `session ${log.id}: initial system prompt #${k + 1} diverged from ${pinningScenario.name}/${SYSTEM_PROMPT_SNAPSHOT}`)
.toEqual(initialPromptSnapshot)
}
if (scenario.pinsHeader === true && logIndex === 0) {
expect(formatSystemPromptSnapshot(
prompts[0] as string,
normalizedSystemPromptDeltas(log.content, ctx),
), `session ${log.id}: system-prompt deltas diverged from ${pinningScenario.name}/${SYSTEM_PROMPT_SNAPSHOT}`)
.toEqual(promptSnapshot)
}
}
})
@@ -332,13 +598,15 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
// `overridden` flag: required when set, forbidden when not — the harness
// forwards the file purely on existence, so an unregistered stray sidecar
// would silently replace the derived script.
for (const { name, overridden, childSessions } of scenarios) {
for (const { name, overridden, childSessions, pinsHeader } of scenarios) {
const dir = join(snapshotsDir, name)
expect(existsSync(join(dir, 'input.json')), `${name}/input.json`).toBe(true)
expect(existsSync(join(dir, 'stdout.golden.jsonl')), `${name}/stdout.golden.jsonl`).toBe(true)
expect(existsSync(join(dir, 'session.jsonl')), `${name}/session.jsonl`).toBe(true)
expect(existsSync(join(dir, 'replay.override.json')), `${name}/replay.override.json presence must match \`overridden\``)
.toBe(overridden === true)
expect(existsSync(join(dir, SYSTEM_PROMPT_SNAPSHOT)), `${name}/${SYSTEM_PROMPT_SNAPSHOT} presence must match \`pinsHeader\``)
.toBe(pinsHeader === true)
// A nested-agent scenario ships one child fixture per recorded subagent
// session (`session.1.jsonl` …), the replay source for that child session.
for (const childFixture of childFixturePaths(dir, childSessions ?? 0)) {
@@ -347,21 +615,46 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
}
})
it('exactly one scenario pins the request-header content', () => {
// Zero pins would drop the prompt/schema surface from the suite entirely;
// two would split it. One pin per suite is the design (pinned-header RFC);
// WHICH scenario pins is the scenario table's reviewable choice.
expect(scenarios.filter(s => s.pinsHeader === true).map(s => s.name)).toEqual([pinningScenario.name])
it('exactly one scenario pins the request-header content of each header class', () => {
// Zero pins would drop a class's prompt/schema surface from the suite
// entirely; two would split it. One pin per class is the design
// (pinned-header RFC); WHICH scenario pins is the scenario table's
// reviewable choice.
const pins = new Map<string, string[]>()
for (const scenario of scenarios.filter(s => s.pinsHeader === true)) {
const cls = classOf(scenario)
pins.set(cls, [...pins.get(cls) ?? [], scenario.name])
}
expect(Object.fromEntries([...pins].map(([cls, names]) => [cls, names.length]))).toEqual(
Object.fromEntries([...pinningByClass.keys()].map(cls => [cls, 1])))
for (const scenario of scenarios) {
expect(pinningByClass.has(classOf(scenario)), `class "${classOf(scenario)}" (scenario ${scenario.name}) has a pin`).toBe(true)
}
})
it('committed fixtures carry request-header content ONLY in the pinning scenario', async () => {
// The whole point of the pin: a system-prompt or tool-schema change must
// churn exactly one committed line. A non-pinning fixture that carries the
// full header (a hand-recorded file, or a header line hand-edited out of
// its canonical JSON form) silently reopens the suite-wide churn, so fail
// loud here: every non-pinning session*.jsonl must be a fixed point of
// scrubRequestHeaders (apply the scrub to fix a violation), and the
// pinning scenario's fixtures must NOT be (their content IS the pin).
it('every pinning fixture carries one request/header, one readable prompt, and its declared deltas', async () => {
// The live uniformity guard runs only in NON-pinning scenarios, so a
// class made of just its pinning scenario would otherwise accept a
// re-recorded pin with several headers or an undeclared mid-run
// header-delta — shapes the pin design cannot represent. Assert the
// committed pins directly; a scenario whose arc legitimately rewrites
// a prompt section declares the exact count via expectedHeaderDeltas.
for (const scenario of pinningByClass.values()) {
const fixture = await readFile(join(snapshotsDir, scenario.name, 'session.jsonl'), 'utf8')
const headers = normalizedHeaders(fixture, fixtureContext(fixture))
const promptSnapshot = await readFile(join(snapshotsDir, scenario.name, SYSTEM_PROMPT_SNAPSHOT), 'utf8')
expect(headers.length, `${scenario.name}: a pinning fixture must carry exactly one request/header`).toBe(1)
expect(promptSnapshot.length, `${scenario.name}/${SYSTEM_PROMPT_SNAPSHOT} must not be empty`).toBeGreaterThan(0)
expect(promptSnapshot.endsWith('\n'), `${scenario.name}/${SYSTEM_PROMPT_SNAPSHOT} must end in a newline`).toBe(true)
expect(headerDeltaCount(fixture), `${scenario.name}: a pinning fixture must carry exactly its declared request/header-deltas`)
.toBe(scenario.expectedHeaderDeltas ?? 0)
}
})
it('every committed JSONL omits system prompts and only pinning fixtures keep other header bulk', async () => {
// System prompts always live in the readable Markdown artifact. Header
// pins keep tool schemas/prefixes in JSONL; every other fixture tokenizes
// all header bulk. Fixed-point checks make both storage rules fail loud.
for (const scenario of scenarios) {
const dir = join(snapshotsDir, scenario.name)
const files = [
@@ -370,8 +663,10 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
]
for (const file of files) {
const fixture = await readFile(join(dir, file), 'utf8')
expect(scrubSystemPrompts(fixture), `${scenario.name}/${file} carries an unscrubbed system prompt`)
.toEqual(fixture)
if (scenario.pinsHeader === true) {
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} must PIN the full header content`)
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} must pin the non-system header content`)
.not.toEqual(fixture)
} else {
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} carries unscrubbed header content`)

35
packages/support/acp-snapshot/tests/fixtures/fake-acp-agent.ts vendored
View File

@@ -58,6 +58,13 @@ interface Behavior {
strayBucketFile?: boolean
/** Delete the sessions root entirely (harvest must yield no logs). */
deleteSessionsRoot?: boolean
/**
* Vocabulary for `session/set_config_option`: allowed values per config id.
* A set naming an unknown id or an out-of-vocabulary value rejects (the
* real bridge's rule); a valid set answers with the complete refreshed
* option state, `currentValue` updated. Absent: every set rejects.
*/
configOptions?: Record<string, string[]>
}
const sessionsRoot = process.env.DSH_SNAPSHOT_SESSIONS_ROOT ?? ''
@@ -81,6 +88,8 @@ let sessionCwd = ''
let parkedPromptId: number | string | null = null
/** Resolvers for permission-probe responses, keyed by outbound request id. */
const pendingPermission = new Map<number, (outcome: unknown) => void>()
/** Per-run `session/set_config_option` state: config id → current value (first vocabulary entry until set). */
const currentConfig: Record<string, string> = {}
function send(frame: Record<string, unknown>): void {
process.stdout.write(`${JSON.stringify({ jsonrpc: '2.0', ...frame })}\n`)
@@ -195,6 +204,32 @@ function handleFrame(frame: Record<string, unknown>): void {
case 'session/prompt':
void handlePrompt(id as number | string)
return
case 'session/set_config_option': {
const vocabulary = behavior.configOptions
const configId = params.configId as string
const value = params.value as string
const values = vocabulary?.[configId]
if (values === undefined) {
respondError(id as number | string, `unknown config option ${configId}`)
return
}
if (!values.includes(value)) {
respondError(id as number | string, `unknown ${configId} value ${value}`)
return
}
currentConfig[configId] = value
// The real bridge's contract: every set answers with the COMPLETE
// refreshed option state, not just the changed entry.
respond(id as number | string, {
configOptions: Object.entries(vocabulary as Record<string, string[]>).map(([cid, vs]) => ({
id: cid,
type: 'select',
currentValue: currentConfig[cid] ?? vs[0],
options: vs.map(v => ({ value: v, name: v })),
})),
})
return
}
case 'session/cancel':
if (parkedPromptId !== null) {
const parked = parkedPromptId

2
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/session.jsonl vendored
View File

@@ -1,2 +1,2 @@
{"type":"session","id":"ccdc749f-56f3-4267-9750-598b5c60b7b2","createdAt":600,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-nOQ4Gy"}
{"type":"request/header","seq":0,"time":4,"data":{"header":{"config":{"model":"fake"},"system":"SYS PROMPT","tools":[{"name":"t1","description":"D1","parameters":{"type":"object"}}]},"reason":"initial"}}
{"type":"request/header","seq":0,"time":4,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":[{"name":"t1","description":"D1","parameters":{"type":"object"}}]},"reason":"initial"}}

1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/system-prompt.golden.md vendored Normal file
View File

@@ -0,0 +1 @@
SYS PROMPT

3
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/behavior.json vendored
View File

@@ -5,7 +5,8 @@
"lines": [
{ "type": "session", "id": "{{SID}}", "createdAt": 100, "cwd": "{{CWD}}" },
{ "type": "request/header", "seq": 0, "time": 100, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } },
{ "type": "turn/start", "seq": 1, "time": 100, "data": { "turn": 1 } }
{ "type": "request/header-delta", "seq": 1, "time": 100, "data": { "system": { "keepStart": 1, "keepEnd": 0, "insert": ["NEW PROMPT LINE"] } } },
{ "type": "turn/start", "seq": 2, "time": 100, "data": { "turn": 1 } }
]
}]
}

5
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/session.jsonl vendored
View File

@@ -1,3 +1,4 @@
{"type":"session","id":"12121212-3434-4545-8686-787878787878","createdAt":7,"cwd":"/rec/pin-cwd"}
{"type":"request/header","seq":0,"time":7,"data":{"header":{"config":{"model":"fake"},"system":"SYS PROMPT","tools":[{"name":"t1","description":"D1","parameters":{"type":"object"}}]},"reason":"initial"}}
{"type":"turn/start","seq":1,"time":7,"data":{"turn":1}}
{"type":"request/header","seq":0,"time":7,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":[{"name":"t1","description":"D1","parameters":{"type":"object"}}]},"reason":"initial"}}
{"type":"request/header-delta","seq":1,"time":7,"data":{"system":{"keepStart":1,"keepEnd":0,"insert":["{{system}}"]}}}
{"type":"turn/start","seq":2,"time":7,"data":{"turn":1}}

5
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/system-prompt.golden.md vendored Normal file
View File

@@ -0,0 +1,5 @@
SYS PROMPT
<!-- request/header-delta 1: keepStart=1, keepEnd=0 -->
NEW PROMPT LINE

49
packages/support/acp-snapshot/tests/harness.spec.ts
View File

@@ -168,6 +168,8 @@ describe('runScenario', () => {
[{ op: 'promptExpectError', text: 'x' }, /promptExpectError before newSession/],
[{ op: 'promptAndCancel', text: 'x' }, /promptAndCancel before newSession/],
[{ op: 'cancel' }, /cancel before newSession/],
[{ op: 'setConfigOption', configId: 'sandbox-mode', value: 'read-only' }, /setConfigOption before newSession/],
[{ op: 'setConfigOptionExpectError', configId: 'sandbox-mode', value: 'yolo' }, /setConfigOptionExpectError before newSession/],
] as [InputStep, RegExp][])('rejects %j before newSession', { timeout: 20_000 }, async (step, message) => {
const { fixtureFile } = await scenario({})
await expect(runScenario(
@@ -176,6 +178,53 @@ describe('runScenario', () => {
)).rejects.toThrow(message)
})
it('setConfigOption switches a value and receives the complete refreshed option state', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({
configOptions: { 'sandbox-mode': ['read-only', 'workspace-write'], 'approval-policy': ['ask', 'never'] },
})
const result = await runScenario(
{
steps: [...boot,
{ op: 'setConfigOption', configId: 'sandbox-mode', value: 'workspace-write' },
{ op: 'setConfigOption', configId: 'approval-policy', value: 'never' }],
},
{ agent: AGENT, mode: 'replay', fixtureFile },
)
// Every set answers with the FULL state: the second response carries the
// first switch's value too — the complete-refreshed-state contract.
const frames = result.rawStdout.trim().split('\n').map(line => JSON.parse(line) as { result?: { configOptions?: { id: string; currentValue: string }[] } })
const states = frames
.map(f => f.result?.configOptions)
.filter(options => options !== undefined)
.map(options => Object.fromEntries((options as { id: string; currentValue: string }[]).map(o => [o.id, o.currentValue])))
expect(states).toEqual([
{ 'sandbox-mode': 'workspace-write', 'approval-policy': 'ask' },
{ 'sandbox-mode': 'workspace-write', 'approval-policy': 'never' },
])
})
it('setConfigOptionExpectError swallows the rejection for unknown ids and out-of-vocabulary values', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ configOptions: { 'sandbox-mode': ['read-only'] } })
const result = await runScenario(
{
steps: [...boot,
{ op: 'setConfigOptionExpectError', configId: 'sandbox-mode', value: 'yolo' },
{ op: 'setConfigOptionExpectError', configId: 'reasoning-effort', value: 'max' }],
},
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.rawStdout).toContain('unknown sandbox-mode value yolo')
expect(result.rawStdout).toContain('unknown config option reasoning-effort')
})
it('setConfigOptionExpectError throws when the set unexpectedly succeeds', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ configOptions: { 'sandbox-mode': ['read-only'] } })
await expect(runScenario(
{ steps: [...boot, { op: 'setConfigOptionExpectError', configId: 'sandbox-mode', value: 'read-only' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/expected set_config_option to be rejected/)
})
it('rejects an unknown input op', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({})
const bogus = { op: 'reticulate' } as unknown as InputStep

80
packages/support/acp-snapshot/tests/normalize.spec.ts
View File

@@ -1,5 +1,11 @@
import { describe, expect, it } from 'vitest'
import { type NormalizeContext, normalizeSessionLog, normalizeStdout, scrubRequestHeaders } from '../src/normalize.ts'
import {
type NormalizeContext,
normalizeSessionLog,
normalizeStdout,
scrubRequestHeaders,
scrubSystemPrompts,
} from '../src/normalize.ts'
/**
* Unit tests for the pure snapshot normalizers. Live as a *.spec.ts (runs in
@@ -201,6 +207,38 @@ describe('scrubRequestHeaders', () => {
expect(toolsOnly).not.toContain('{{system}}')
})
it('scrubs the header session prefix to one token per message, keeping the count', () => {
const ev = headerEvent({
config: { model: 'm' },
messagePrefix: [
{ role: 'user', content: [{ type: 'text', text: 'workspace AGENTS digest' }] },
{ role: 'user', content: [{ type: 'text', text: 'skills catalog' }] },
],
})
const out = scrubRequestHeaders(`${headerLine}\n${ev}\n`)
expect(out).toContain('"messagePrefix":["{{messagePrefix}}","{{messagePrefix}}"]')
expect(out).not.toContain('AGENTS digest')
expect(out).not.toContain('skills catalog')
// Absence stays absent — a prefix-less header gains no token…
expect(scrubRequestHeaders(`${headerLine}\n${headerEvent({ system: 's' })}\n`)).not.toContain('{{messagePrefix}}')
// …and a non-array shape passes through untouched.
const odd = JSON.stringify({ type: 'request/header', seq: 4, time: 9, data: { header: { config: { model: 'm' }, messagePrefix: 'weird' }, reason: 'initial' } })
expect(scrubRequestHeaders(`${headerLine}\n${odd}\n`)).toContain('"messagePrefix":"weird"')
})
it('scrubs a header-delta prefix replacement to one token per message', () => {
const delta = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: { messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'leaked opener' }] }] },
})
const out = scrubRequestHeaders(`${headerLine}\n${delta}\n`)
expect(out).toContain('"messagePrefix":["{{messagePrefix}}"]')
expect(out).not.toContain('leaked opener')
// The empty-array transition-to-absence stays a structural fact.
const toNone = JSON.stringify({ type: 'request/header-delta', seq: 9, time: 9, data: { messagePrefix: [] } })
expect(scrubRequestHeaders(`${headerLine}\n${toNone}\n`)).toContain('"messagePrefix":[]')
})
it('leaves a delta with no scrubbable payload byte-identical (config-only, or non-array shapes)', () => {
const configOnly = JSON.stringify({ type: 'request/header-delta', seq: 8, time: 9, data: { config: { model: 'm2' } } })
const oddShapes = JSON.stringify({ type: 'request/header-delta', seq: 9, time: 9, data: { system: { insert: 'not-an-array' }, tools: null } })
@@ -274,3 +312,43 @@ describe('scrubRequestHeaders', () => {
expect(scrubRequestHeaders(once)).toBe(once)
})
})
describe('scrubSystemPrompts', () => {
it('scrubs only system prompt payloads while keeping tools and prefixes verbatim', () => {
const header = JSON.stringify({
type: 'request/header', seq: 1, time: 2,
data: {
header: {
system: 'full prompt',
tools: [{ name: 'read', description: 'full schema' }],
messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'full prefix' }] }],
},
reason: 'initial',
},
})
const delta = JSON.stringify({
type: 'request/header-delta', seq: 2, time: 3,
data: {
system: { keepStart: 1, keepEnd: 2, insert: ['new prompt line'] },
tools: { changed: [{ name: 'read', description: 'changed schema' }] },
messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'changed prefix' }] }],
},
})
const toolsOnly = JSON.stringify({
type: 'request/header', seq: 3, time: 4,
data: { header: { tools: [{ name: 'read', description: 'schema only' }] }, reason: 'resume' },
})
const out = scrubSystemPrompts(`${header}\n${delta}\n${toolsOnly}\n`)
expect(out).toContain('"system":"{{system}}"')
expect(out).toContain('"insert":["{{system}}"]')
expect(out).not.toContain('full prompt')
expect(out).not.toContain('new prompt line')
expect(out).toContain('full schema')
expect(out).toContain('full prefix')
expect(out).toContain('changed schema')
expect(out).toContain('changed prefix')
expect(out.split('\n')[2]).toBe(toolsOnly)
expect(scrubSystemPrompts(out)).toBe(out)
})
})

225
packages/support/acp-snapshot/tests/suite.spec.ts
View File

@@ -1,11 +1,21 @@
import { cpSync, mkdtempSync } from 'node:fs'
import { cpSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs'
import { rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { fileURLToPath } from 'node:url'
import { afterAll, describe, expect, it } from 'vitest'
import { defineAcpSnapshotSuite, type Scenario } from '../src/index.ts'
import { childFixturePaths, fixtureContext, headerDeltaCount, normalizedHeaders } from '../src/suite.ts'
import { defineAcpSnapshotSuite, type HarvestedLog, type Scenario } from '../src/index.ts'
import {
childFixturePaths,
fixtureContext,
formatSystemPromptSnapshot,
headerDeltaCount,
normalizedHeaders,
normalizedSystemPromptDeltas,
normalizedSystemPrompts,
refreshFixtureReplacements,
stabilizeRefreshLog,
} from '../src/suite.ts'
/**
* Unit tests for the suite factory, by running it: two synthetic suites over
@@ -34,12 +44,18 @@ const AGENT = {
const REPLAY_DIR = fileURLToPath(new URL('./fixtures/suite', import.meta.url))
const RECORD_SRC = fileURLToPath(new URL('./fixtures/record-suite', import.meta.url))
// The replay suite doubles as the header-CLASS coverage: every scenario names
// the same explicit class (the record suite exercises the 'default' fallback),
// and plain-turn boots through a per-scenario configPath override (the same
// dummy path the agent default carries — the plumbing, not the composition,
// is what this suite can exercise; the real overlay boot is the acp-agent
// example's code-mode scenarios).
const REPLAY_SCENARIOS: Scenario[] = [
{ name: 'pin-turn', hasModelTurn: true, recorded: true, pinsHeader: true },
{ name: 'plain-turn', hasModelTurn: true, recorded: true, childSessions: 1 },
{ name: 'no-model', hasModelTurn: false, recorded: false },
{ name: 'blocked-log', hasModelTurn: false, comparesLog: true, recorded: false },
{ name: 'authored-error', hasModelTurn: true, recorded: false, overridden: true },
{ name: 'pin-turn', hasModelTurn: true, recorded: true, pinsHeader: true, expectedHeaderDeltas: 1, headerClass: 'main' },
{ name: 'plain-turn', hasModelTurn: true, recorded: true, childSessions: 1, headerClass: 'main', configPath: AGENT.configPath },
{ name: 'no-model', hasModelTurn: false, recorded: false, headerClass: 'main' },
{ name: 'blocked-log', hasModelTurn: false, comparesLog: true, recorded: false, headerClass: 'main' },
{ name: 'authored-error', hasModelTurn: true, recorded: false, overridden: true, headerClass: 'main' },
]
const RECORD_SCENARIOS: Scenario[] = [
@@ -49,16 +65,41 @@ const RECORD_SCENARIOS: Scenario[] = [
{ name: 'rec-skip', hasModelTurn: true, recorded: false, overridden: true },
]
// Record mode mutates its snapshots dir, so run it on a throwaway copy —
// except under the documented bootstrap knob, which regenerates the committed
// fixtures/goldens in place.
// Record/refresh modes mutate their snapshots dir, so run them on throwaway
// copies — except record's documented bootstrap knob, which regenerates the
// committed record fixtures/goldens in place.
const BOOTSTRAP = process.env.ACP_SNAPSHOT_SPEC_BOOTSTRAP === '1'
const recordDir = BOOTSTRAP ? RECORD_SRC : mkdtempSync(join(tmpdir(), 'acp-snap-record-suite-'))
if (!BOOTSTRAP) cpSync(RECORD_SRC, recordDir, { recursive: true })
const refreshDir = mkdtempSync(join(tmpdir(), 'acp-snap-refresh-suite-'))
cpSync(REPLAY_DIR, refreshDir, { recursive: true })
staleRefreshFixtures(refreshDir)
afterAll(async () => {
if (!BOOTSTRAP) await rm(recordDir, { recursive: true, force: true })
await rm(refreshDir, { recursive: true, force: true })
})
function staleRefreshFixtures(dir: string): void {
writeFileSync(join(dir, 'plain-turn', 'stdout.golden.jsonl'), 'stale stdout\n')
writeFileSync(join(dir, 'pin-turn', 'system-prompt.golden.md'), 'STALE PROMPT\n')
const plainBehaviorFile = join(dir, 'plain-turn', 'behavior.json')
const plainBehavior = JSON.parse(readFileSync(plainBehaviorFile, 'utf8')) as Record<string, unknown>
plainBehavior.echoEnv = true
writeFileSync(plainBehaviorFile, `${JSON.stringify(plainBehavior, null, 2)}\n`)
writeFileSync(join(dir, 'blocked-log', 'session.jsonl'), [
'{"type":"session","id":"99999999-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/blocked-cwd"}',
'{"type":"hook/result","seq":1,"time":13,"data":{"decision":"stale","durationMs":99}}',
'',
].join('\n'))
writeFileSync(join(dir, 'authored-error', 'session.jsonl'), [
'{"type":"session","id":"77777777-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/error-cwd"}',
'{"type":"turn/end","seq":1,"time":9,"data":{"error":"stale"}}',
'',
].join('\n'))
}
describe('defineAcpSnapshotSuite: replay mode', () => {
defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: REPLAY_DIR, scenarios: REPLAY_SCENARIOS, mode: 'replay' })
})
@@ -69,8 +110,38 @@ describe('defineAcpSnapshotSuite: record mode', () => {
defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: recordDir, scenarios: RECORD_SCENARIOS, mode: 'record' })
})
describe('defineAcpSnapshotSuite: refresh mode', () => {
defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: refreshDir, scenarios: REPLAY_SCENARIOS, mode: 'refresh' })
})
describe('defineAcpSnapshotSuite: refresh write-back', () => {
it('rewrites stdout and comparable logs from a replay-mode child run', () => {
const stdout = readFileSync(join(refreshDir, 'plain-turn', 'stdout.golden.jsonl'), 'utf8')
expect(stdout).not.toContain('stale stdout')
expect(stdout).toContain('env:{\\"mode\\":\\"replay\\"')
expect(stdout).not.toContain('\\"mode\\":\\"refresh\\"')
const blocked = readFileSync(join(refreshDir, 'blocked-log', 'session.jsonl'), 'utf8')
expect(blocked).toContain('"decision":"block"')
expect(blocked).not.toContain('"decision":"stale"')
const authored = readFileSync(join(refreshDir, 'authored-error', 'session.jsonl'), 'utf8')
expect(authored).toContain('"error":"model exploded"')
expect(authored).not.toContain('"error":"stale"')
expect(readFileSync(join(refreshDir, 'pin-turn', 'system-prompt.golden.md'), 'utf8')).toBe([
'SYS PROMPT',
'',
'<!-- request/header-delta 1: keepStart=1, keepEnd=0 -->',
'',
'NEW PROMPT LINE',
'',
].join('\n'))
})
})
describe('defineAcpSnapshotSuite: registration contract', () => {
it('throws when no scenario pins the request-header content', () => {
it("throws when a scenario's header class has no pinning scenario", () => {
expect(() => {
defineAcpSnapshotSuite({
agent: AGENT,
@@ -78,7 +149,33 @@ describe('defineAcpSnapshotSuite: registration contract', () => {
scenarios: [{ name: 'pinless', hasModelTurn: true, recorded: true }],
mode: 'replay',
})
}).toThrow(/no scenario pins/)
}).toThrow(/no scenario pins the request-header content of class "default"/)
// A pinned class does not cover a DIFFERENT class's members.
expect(() => {
defineAcpSnapshotSuite({
agent: AGENT,
snapshotsDir: REPLAY_DIR,
scenarios: [
{ name: 'pinned', hasModelTurn: true, recorded: true, pinsHeader: true },
{ name: 'classless-orphan', hasModelTurn: true, recorded: true, headerClass: 'other' },
],
mode: 'replay',
})
}).toThrow(/class "other" \(needed by classless-orphan\)/)
})
it('throws when two scenarios pin the same header class', () => {
expect(() => {
defineAcpSnapshotSuite({
agent: AGENT,
snapshotsDir: REPLAY_DIR,
scenarios: [
{ name: 'first-pin', hasModelTurn: true, recorded: true, pinsHeader: true },
{ name: 'second-pin', hasModelTurn: true, recorded: true, pinsHeader: true },
],
mode: 'replay',
})
}).toThrow(/header class "default" pinned by both first-pin and second-pin/)
})
})
@@ -135,6 +232,55 @@ describe('normalizedHeaders', () => {
})
})
describe('normalizedSystemPrompts', () => {
it('extracts normalized string prompts and omits absent or non-string fields', () => {
const log = [
'{"type":"session","id":"a","createdAt":5,"cwd":"/w"}',
'{"type":"request/header","seq":0,"time":9,"data":{"header":{"system":"work in /w"}}}',
'{"type":"request/header","seq":1,"time":9,"data":{"header":{}}}',
'{"type":"request/header","seq":2,"time":9,"data":{"header":{"system":null}}}',
'{"type":"request/header","seq":3,"time":9,"data":{"header":null}}',
'{"type":"request/header","seq":4,"time":9,"data":{"header":"invalid"}}',
'',
].join('\n')
expect(normalizedSystemPrompts(log, { sessionIds: [], cwd: '/w' })).toEqual(['work in {{cwd}}'])
})
})
describe('normalizedSystemPromptDeltas', () => {
it('extracts and normalizes well-formed system edits', () => {
const log = [
'{"type":"request/header-delta","data":{"system":{"keepStart":1,"keepEnd":0,"insert":["work in /w"]}}}',
'{"type":"request/header-delta","data":{"tools":{"replace":[]}}}',
'{"type":"request/header-delta","data":{"system":{"keepStart":"1","keepEnd":0,"insert":[]}}}',
'{"type":"request/header-delta","data":{"system":{"keepStart":1,"keepEnd":0,"insert":[null]}}}',
'',
].join('\n')
expect(normalizedSystemPromptDeltas(log, { sessionIds: [], cwd: '/w' })).toEqual([
{ keepStart: 1, keepEnd: 0, insert: ['work in {{cwd}}'] },
])
})
})
describe('formatSystemPromptSnapshot', () => {
it('adds a missing terminal newline without changing an existing one', () => {
expect(formatSystemPromptSnapshot('prompt')).toBe('prompt\n')
expect(formatSystemPromptSnapshot('prompt\n')).toBe('prompt\n')
})
it('renders readable system-prompt delta sections', () => {
expect(formatSystemPromptSnapshot('prompt', [
{ keepStart: 1, keepEnd: 0, insert: ['new', 'lines'] },
])).toBe('prompt\n\n<!-- request/header-delta 1: keepStart=1, keepEnd=0 -->\n\nnew\nlines\n')
})
it('does not double the newline of a delta insert with a trailing blank line', () => {
expect(formatSystemPromptSnapshot('prompt\n', [
{ keepStart: 2, keepEnd: 1, insert: ['tail', ''] },
])).toBe('prompt\n\n<!-- request/header-delta 1: keepStart=2, keepEnd=1 -->\n\ntail\n')
})
})
describe('headerDeltaCount', () => {
it('counts request/header-delta events, ignoring blanks and other lines', () => {
const delta = JSON.stringify({ type: 'request/header-delta', seq: 2, time: 9, data: {} })
@@ -143,3 +289,56 @@ describe('headerDeltaCount', () => {
expect(headerDeltaCount(`${other}\n`)).toBe(0)
})
})
describe('refreshFixtureReplacements', () => {
it('maps fresh ids and cwd values to the existing fixture values, skipping non-replacements', () => {
const log = (content: string): HarvestedLog => ({ id: 'diagnostic', createdAt: 1, content })
const logs = [
log('{"type":"session","id":"","cwd":"/same"}\n'),
log('{"type":"session","id":"new-parent","cwd":"/new"}\n'),
log('{"type":"session","id":"new-child","cwd":"/new"}\n'),
]
const fixtures = [
'{"type":"session","id":"","cwd":"/same"}\n',
'{"type":"session","id":"old-parent","cwd":"/old"}\n',
]
expect(refreshFixtureReplacements(logs, fixtures)).toEqual([
{ from: 'new-parent', to: 'old-parent' },
{ from: '/new', to: '/old' },
])
})
})
describe('stabilizeRefreshLog', () => {
it('keeps volatile fixture fields while preserving fresh meaningful payloads', () => {
const fresh = [
'{"type":"session","id":"new-child","createdAt":200,"cwd":"/new","parentSession":"new-parent","seedLength":1}',
'{"type":"hook/result","seq":1,"time":22,"data":{"decision":"block","durationMs":37}}',
'{"type":"turn/end","seq":2,"time":33,"data":{"error":"fresh error"}}',
'{"type":"tool/result","seq":3,"time":44,"data":{"text":"new-parent in /new"}}',
'{"type":"hook/result","seq":4,"time":55,"data":{"decision":"allow","durationMs":5}}',
'',
].join('\n')
const existing = [
'{"type":"session","id":"old-child","createdAt":100,"cwd":"/old","parentSession":"old-parent","seedLength":5}',
'{"type":"hook/result","seq":1,"time":11,"data":{"decision":"stale","durationMs":99}}',
'{"type":"turn/end","seq":2,"data":{"error":"stale"}}',
'{"type":"assistant/message","seq":3,"time":12,"data":{"text":"different type"}}',
'{"type":"hook/result","seq":4,"time":13,"data":{"decision":"stale"}}',
'',
].join('\n')
expect(stabilizeRefreshLog(fresh, existing, [
{ from: 'new-parent', to: 'old-parent' },
{ from: 'new-child', to: 'old-child' },
{ from: '/new', to: '/old' },
])).toBe([
'{"type":"session","id":"old-child","createdAt":100,"cwd":"/old","parentSession":"old-parent","seedLength":1}',
'{"type":"hook/result","seq":1,"time":11,"data":{"decision":"block","durationMs":99}}',
'{"type":"turn/end","seq":2,"time":33,"data":{"error":"fresh error"}}',
'{"type":"tool/result","seq":3,"time":44,"data":{"text":"old-parent in /old"}}',
'{"type":"hook/result","seq":4,"time":13,"data":{"decision":"allow","durationMs":5}}',
'',
].join('\n'))
})
})

27
packages/support/invariants/src/index.ts
View File

@@ -367,8 +367,11 @@ export function apply(ctx: Context, config: Config = {}): void {
// hand-built one-shot (compaction summarize) is unfrozen and skipped — must
// be EXACTLY what the session log reconstructs:
//
// - messages: the derivation over the log prefix strictly before the
// in-flight step's `step/start` (the reconstruction boundary). Compared
// - messages: the folded header's session prefix (messagePrefix — the
// `agent/session-prefix` product, logged on the header because no
// session event carries it) followed by the
// derivation over the log prefix strictly before the in-flight step's
// `step/start` (the reconstruction boundary). The derivation is compared
// against a FRESH Session built over that prefix — the same projection
// code with zero shared state, so the live cache under test cannot vouch
// for itself. Boundary-correct by construction: content appended after
@@ -408,18 +411,22 @@ export function apply(ctx: Context, config: Config = {}): void {
if (boundary === -1) {
throw new InvariantError('a loop-built request with no step/start in its session log')
}
const rebuilt = new Session(SessionId(`${String(session.id)}-invariant-rebuild`), structuredClone(events.slice(0, boundary)))
// JSON equality is sound here: both sides are structuredClones produced by
// the same projection code path, so key insertion order matches when the
// values do.
if (JSON.stringify(options.messages) !== JSON.stringify(rebuilt.deriveMessages())) {
throw new InvariantError(`llm request for session "${String(session.id)}" diverges from the boundary derivation (log-reconstruction desync)`)
}
const header = foldRequestHeader(events)
if (header === undefined) {
throw new InvariantError('a loop-built request with no request/header event in its session log')
}
const rebuilt = new Session(SessionId(`${String(session.id)}-invariant-rebuild`), structuredClone(events.slice(0, boundary)))
// The reconstruction equation: the folded header's session prefix, then
// the boundary derivation — the loop
// logs the header event BEFORE dispatch, so the fold already covers this
// request's prefix. JSON equality is sound here: both sides are
// structuredClones produced by the same projection/build code path, so key
// insertion order matches when the values do.
const expected = [...header.messagePrefix ?? [], ...rebuilt.deriveMessages()]
if (JSON.stringify(options.messages) !== JSON.stringify(expected)) {
throw new InvariantError(`llm request for session "${String(session.id)}" diverges from the boundary derivation (log-reconstruction desync)`)
}
const headerMatches = options.model === header.config.model
&& options.system === header.system
&& options.temperature === header.config.temperature

15
packages/support/invariants/tests/invariants.spec.ts
View File

@@ -707,6 +707,21 @@ describe('request-reconstruction cross-check (llm/stream)', () => {
expect(() => { dispatch(ctx, options) }).not.toThrow()
})
it('expects the folded header\'s session prefix ahead of the derivation (prefix + derived)', async () => {
const { ctx, session, boundary } = await requestSetup()
const prefix = { role: 'user' as const, content: [{ type: 'text' as const, text: '<system-reminder>catalog</system-reminder>' }] }
session.append('request/header-delta', { messagePrefix: [prefix] })
// The prefixed request matches the fold…
const prefixed = Object.freeze({ model: 'm', messages: Object.freeze([prefix, ...boundary]), sessionId: session.id })
expect(() => { dispatch(ctx, prefixed) }).not.toThrow()
// …a request that DROPPED the logged prefix diverges…
const bare = Object.freeze({ model: 'm', messages: Object.freeze([...boundary]), sessionId: session.id })
expect(() => { dispatch(ctx, bare) }).toThrow(/diverges from the boundary derivation/)
// …and so does one that misplaced it (prefix sent after the history).
const misplaced = Object.freeze({ model: 'm', messages: Object.freeze([...boundary, prefix]), sessionId: session.id })
expect(() => { dispatch(ctx, misplaced) }).toThrow(/diverges from the boundary derivation/)
})
it('rejects a frozen request whose messages diverge from the boundary derivation', async () => {
const { ctx, session, boundary } = await requestSetup()
const messages = [...boundary, { role: 'user', content: [{ type: 'text', text: 'phantom' }] }]