test(snapshot): split pinned schemas into sidecars

This commit is contained in:
Tianyi Cui
2026-07-14 23:11:21 +08:00
parent 28cad74b7c
commit 8ffcef7db9
28 changed files with 1995 additions and 50 deletions

View File

@@ -5,8 +5,8 @@ The ACP snapshot suite kit: the shared machinery behind the keyless snapshot tie
Three layers, importable separately:
- **`runScenario` (harness)** — boots the real agent bin as a subprocess via tsx (unbuilt, Loader path), drives it over ACP JSON-RPC stdio from a deterministic `input.json` script, tees raw stdout for the golden + purity check, and harvests every persisted session JSONL (parent + subagent children, primary-first) after a graceful stdin-EOF shutdown. Parameterized by `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath` — absolute paths; the subprocess cwd is a temp dir outside the repo).
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept), `scrubSystemPrompts` (prompt text → `{{system}}` in every JSONL), and `scrubRequestHeaders` (the remaining header bulk → `{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header RFC](../../../docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, the per-header-class pin (`system-prompt.golden.md` plus the JSONL's full tool schemas) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt-scrubbed, non-pinning fixtures fully header-scrubbed). Must be called at vitest collection time.
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header RFC](../../../docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, the per-header-class pin (`system-prompt.golden.md` plus `tool-schemas.golden.json`) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt/schema-scrubbed, non-pinning fixtures fully header-scrubbed). Must be called at vitest collection time.
A consuming `*.snapshot.ts` is the scenario table plus one factory call:
@@ -35,9 +35,9 @@ defineAcpSnapshotSuite({
})
```
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode and filesystem scenarios are templates. Each pinning directory's generated `system-prompt.golden.md` is the reviewable snapshot of the normalized composed prompt; `session.jsonl` stores `"system":"{{system}}"` while retaining the complete tool list.
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode and filesystem scenarios are templates. Each pinning directory stores the normalized composed prompt in generated `system-prompt.golden.md` and the initial schemas plus schema deltas in generated `tool-schemas.golden.json`; `session.jsonl` stores `"system":"{{system}}","tools":"{{tools}}"` while retaining config, reason, and any model-visible prefix.
Examples use a `cordis.snapshot.yml` overlay with [`dsh-llm-replay`](../llm-replay/README.md). Recording calls the live model and updates model fixtures; keyless refresh replays those fixtures and updates derived stdout, session-log, and prompt snapshots. See the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
Examples use a `cordis.snapshot.yml` overlay with [`dsh-llm-replay`](../llm-replay/README.md). Recording calls the live model and updates model fixtures; keyless refresh replays those fixtures and updates derived stdout, session-log, prompt, and tool-schema snapshots. See the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
`suite.ts` imports Vitest, so use this package only inside a Vitest run. The ACP-specific script queues permission answers by stable option kind and maps them to current option ids; a missing answer cancels, while an unavailable kind fails the scenario after cancelling the agent request. It can also set session config options or assert that unknown ids and values are rejected in the transcript.

View File

@@ -20,6 +20,7 @@ export {
normalizeStdout,
scrubRequestHeaders,
scrubSystemPrompts,
scrubToolSchemas,
type NormalizeContext,
} from './normalize.ts'
export {

View File

@@ -1,8 +1,8 @@
/**
* Pure ACP transcript and session-log normalizers. They scrub session ids, temp cwd, RPC ids,
* timestamps, and hook duration while preserving deterministic event sequence numbers.
* Request-header scrubbers stay separate so one scenario per header class can pin tools and a
* readable prompt while other fixtures omit duplicated header bulk.
* Request-header scrubbers stay composable so one scenario per header class can pin prompt and
* tool-schema sidecars while retaining any model-visible prefix in the session log.
* @module @deepseek-ai/dsh-acp-snapshot/normalize
*/
@@ -123,7 +123,21 @@ export function normalizeSessionLog(rawLog: string, ctx: NormalizeContext): stri
* @returns The JSONL with system-prompt content tokenized.
*/
export function scrubSystemPrompts(rawLog: string): string {
return scrubHeaderContent(rawLog, false)
return scrubHeaderContent(rawLog, { system: true })
}
/**
* Replace tool schemas in request headers and header deltas with `{{tools}}`
* tokens while retaining field presence, tool names, and delta structure.
* System prompts and session-prefix messages stay verbatim so pinning fixtures
* can move only schema bulk into their dedicated JSON sidecar. Lines without a
* tool payload pass through byte-for-byte; the transform is idempotent.
*
* @param rawLog The raw session `.jsonl` content.
* @returns The JSONL with tool-schema content tokenized.
*/
export function scrubToolSchemas(rawLog: string): string {
return scrubHeaderContent(rawLog, { tools: true })
}
/**
@@ -138,11 +152,18 @@ export function scrubSystemPrompts(rawLog: string): string {
* @returns The JSONL with all header bulk tokenized, other lines byte-identical.
*/
export function scrubRequestHeaders(rawLog: string): string {
return scrubHeaderContent(rawLog, true)
return scrubHeaderContent(rawLog, { system: true, tools: true, prefix: true })
}
/** Transform header content, optionally including tool schemas and the session prefix. */
function scrubHeaderContent(rawLog: string, scrubToolsAndPrefix: boolean): string {
/** Which independent request-header payloads a scrubber replaces. */
interface HeaderScrubOptions {
system?: boolean
tools?: boolean
prefix?: boolean
}
/** Transform the selected request-header payloads. */
function scrubHeaderContent(rawLog: string, options: HeaderScrubOptions): string {
const lines = rawLog.split('\n')
const out = lines.map((line) => {
if (line.trim().length === 0) return line
@@ -153,9 +174,9 @@ function scrubHeaderContent(rawLog: string, scrubToolsAndPrefix: boolean): strin
const header = data.header as Record<string, unknown> | null | undefined
if (header === null || typeof header !== 'object') return line
let touched = false
if ('system' in header) { header.system = SYSTEM; touched = true }
if (scrubToolsAndPrefix && 'tools' in header) { header.tools = TOOLS; touched = true }
if (scrubToolsAndPrefix && Array.isArray(header.messagePrefix)) {
if (options.system === true && 'system' in header) { header.system = SYSTEM; touched = true }
if (options.tools === true && 'tools' in header) { header.tools = TOOLS; touched = true }
if (options.prefix === true && Array.isArray(header.messagePrefix)) {
header.messagePrefix = header.messagePrefix.map(() => MESSAGE_PREFIX)
touched = true
}
@@ -164,16 +185,16 @@ function scrubHeaderContent(rawLog: string, scrubToolsAndPrefix: boolean): strin
if (record.type === 'request/header-delta') {
let touched = false
const system = data.system as Record<string, unknown> | null | undefined
if (system !== null && typeof system === 'object' && Array.isArray(system.insert)) {
if (options.system === true && system !== null && typeof system === 'object' && Array.isArray(system.insert)) {
system.insert = system.insert.map(() => SYSTEM)
touched = true
}
const tools = data.tools as Record<string, unknown> | null | undefined
if (scrubToolsAndPrefix && tools !== null && typeof tools === 'object') {
if (options.tools === true && tools !== null && typeof tools === 'object') {
if (Array.isArray(tools.added)) { tools.added = tools.added.map(scrubToolSchema); touched = true }
if (Array.isArray(tools.changed)) { tools.changed = tools.changed.map(scrubToolSchema); touched = true }
}
if (scrubToolsAndPrefix && Array.isArray(data.messagePrefix)) {
if (options.prefix === true && Array.isArray(data.messagePrefix)) {
data.messagePrefix = data.messagePrefix.map(() => MESSAGE_PREFIX)
touched = true
}

View File

@@ -4,8 +4,8 @@
* output. Record mode refreshes reproducible model scenarios from the live API, while refresh
* mode replays committed scripts and rewrites derived artifacts without a key.
*
* Exactly one scenario per header-composition class pins tool schemas in JSONL and the system
* prompt in Markdown. Every live header is checked against that pin, so session-dependent
* Exactly one scenario per header-composition class pins the system prompt and tool schemas in
* dedicated sidecars. Every live header is checked against that pin, so session-dependent
* composition must declare a separate class instead of escaping coverage.
* @module @deepseek-ai/dsh-acp-snapshot/suite
*/
@@ -21,11 +21,18 @@ import {
normalizeStdout,
scrubRequestHeaders,
scrubSystemPrompts,
scrubToolSchemas,
} from './normalize.ts'
/** The readable system-prompt snapshot beside each header-pinning fixture. */
const SYSTEM_PROMPT_SNAPSHOT = 'system-prompt.golden.md'
/** The structured tool-schema snapshot beside each header-pinning fixture. */
const TOOL_SCHEMAS_SNAPSHOT = 'tool-schemas.golden.json'
/** Stable session-log token standing in for the sidecar's initial schemas. */
const TOOLS_TOKEN = '{{tools}}'
/** A snapshot scenario and how its fixtures are produced. */
export interface Scenario {
name: string
@@ -68,8 +75,8 @@ export interface Scenario {
*/
childSessions?: number
/**
* Whether this scenario is its header class's sole request-header pin. Its Markdown file owns
* the prompt, its JSONL keeps tool schemas, and every classmate is checked for equality.
* Whether this scenario is its header class's sole request-header pin. Dedicated sidecars own
* the prompt and tool schemas, while every classmate is checked for equality.
*/
pinsHeader?: boolean
/**
@@ -184,6 +191,98 @@ export function normalizedSystemPrompts(rawLog: string, ctx: NormalizeContext):
})
}
/**
* The normalized tool-schema arrays carried by request headers in a session
* JSONL, in log order. Headers without an array-valued tools field are omitted
* so callers can assert one schema set per header explicitly.
*
* @param rawLog The session `.jsonl` content to inspect.
* @param ctx The volatile values of the run that produced it.
* @returns The normalized initial tool-schema arrays, in header order.
*/
export function normalizedToolSchemas(rawLog: string, ctx: NormalizeContext): unknown[][] {
return normalizedHeaders(rawLog, ctx).flatMap((header) => {
if (header === null || typeof header !== 'object') return []
const tools = (header as { tools?: unknown }).tools
return Array.isArray(tools) ? [tools] : []
})
}
/**
* Extract normalized tool-schema edits from request-header deltas in log order.
* Deltas without an object-valued tools edit are omitted; their remaining
* structure stays pinned in the session JSONL.
*
* @param rawLog The session `.jsonl` content to inspect.
* @param ctx The volatile values of the run that produced it.
* @returns The normalized tool-schema edits, in event order.
*/
export function normalizedToolSchemaDeltas(rawLog: string, ctx: NormalizeContext): unknown[] {
return normalizeSessionLog(rawLog, ctx)
.split('\n')
.filter(line => line.trim().length > 0)
.map(line => JSON.parse(line) as { type?: unknown; data?: { tools?: unknown } })
.filter(record => record.type === 'request/header-delta')
.flatMap((record) => {
const tools = record.data?.tools
return tools !== null && typeof tools === 'object' && !Array.isArray(tools) ? [tools] : []
})
}
/** The structured contents of a tool-schema sidecar. */
export interface ToolSchemasSnapshot {
/** The complete tool schemas from the pinned request header. */
initial: unknown[]
/** Complete tool-schema edits from subsequent request-header deltas. */
deltas: unknown[]
}
/**
* Render tool schemas and later schema edits as canonical, readable JSON.
*
* @param initial The pinned request header's complete tool schemas.
* @param deltas Complete tool-schema edits from request-header deltas.
* @returns A pretty-printed JSON snapshot ending in one newline.
*/
export function formatToolSchemasSnapshot(initial: readonly unknown[], deltas: readonly unknown[] = []): string {
return `${JSON.stringify({ initial, deltas }, null, 2)}\n`
}
/**
* Parse and validate the stable top-level shape of a tool-schema sidecar.
*
* @param snapshot The JSON sidecar text.
* @returns Its initial schemas and schema deltas.
*/
export function parseToolSchemasSnapshot(snapshot: string): ToolSchemasSnapshot {
const parsed = JSON.parse(snapshot) as unknown
if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) {
throw new Error('acp-snapshot: tool-schema snapshot must be an object')
}
const { initial, deltas } = parsed as { initial?: unknown; deltas?: unknown }
if (!Array.isArray(initial) || !Array.isArray(deltas)) {
throw new Error('acp-snapshot: tool-schema snapshot must carry array-valued initial and deltas fields')
}
return { initial, deltas }
}
/**
* Restore a sidecar's initial schemas into a tokenized pinned header.
*
* @param header The parsed request header carrying `tools: "{{tools}}"`.
* @param snapshot The parsed tool-schema sidecar.
* @returns A copy of the header with its complete initial schemas restored.
*/
export function restorePinnedToolSchemas(header: unknown, snapshot: ToolSchemasSnapshot): unknown {
if (header === null || typeof header !== 'object' || Array.isArray(header)) {
throw new Error('acp-snapshot: pinned request header must be an object')
}
if ((header as { tools?: unknown }).tools !== TOOLS_TOKEN) {
throw new Error(`acp-snapshot: pinned request header tools must equal ${TOOLS_TOKEN}`)
}
return { ...header, tools: snapshot.initial }
}
/** One normalized system-prompt edit carried by a `request/header-delta`. */
export interface SystemPromptDeltaSnapshot {
/** How many leading lines remain from the prior prompt. */
@@ -439,9 +538,9 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
}
// Record writes live model fixtures; keyless refresh writes every comparable replayed
// fixture. Pins keep tools but all JSONL files scrub prompt text.
// fixture. Pinning JSONL keeps prefixes but moves prompts and schemas into sidecars.
const scrub = scenario.pinsHeader === true
? scrubSystemPrompts
? (log: string): string => scrubToolSchemas(scrubSystemPrompts(log))
: scrubRequestHeaders
const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)]
const existingFixtures = REFRESHING
@@ -478,6 +577,18 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
normalizedSystemPromptDeltas(primary.content, ctx),
)
await writeFile(join(dir, SYSTEM_PROMPT_SNAPSHOT), snapshot)
const schemaSets = result.sessionLogs.flatMap(log => normalizedToolSchemas(log.content, ctx))
expect(schemaSets.length, `${mode} produced no tool schemas to snapshot`).toBeGreaterThan(0)
const initialSchemaSnapshot = formatToolSchemasSnapshot(schemaSets[0] as unknown[])
for (const schemas of schemaSets) {
expect(formatToolSchemasSnapshot(schemas), 'the pinning run produced divergent tool schemas')
.toEqual(initialSchemaSnapshot)
}
await writeFile(join(dir, TOOL_SCHEMAS_SNAPSHOT), formatToolSchemasSnapshot(
schemaSets[0] as unknown[],
normalizedToolSchemaDeltas(primary.content, ctx),
))
}
}
@@ -501,7 +612,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
}
// Header-uniformity guard: every live header in a class must equal the class pin split
// across its JSONL header (system token + real tools) and readable Markdown prompt.
// across tokenized JSONL plus readable prompt and structured schema sidecars.
/* v8 ignore next -- construction guarantees the pin exists; a miss would fail the one-header assertion loudly. */
const pinningScenario = pinningByClass.get(classOf(scenario)) ?? scenario
const pinningDir = join(snapshotsDir, pinningScenario.name)
@@ -509,8 +620,11 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
const pinned = normalizedHeaders(pinnedFixture, fixtureContext(pinnedFixture))
const promptSnapshot = await readFile(join(pinningDir, SYSTEM_PROMPT_SNAPSHOT), 'utf8')
const initialPromptSnapshot = initialSystemPromptSnapshot(promptSnapshot)
const toolSchemasSnapshot = await readFile(join(pinningDir, TOOL_SCHEMAS_SNAPSHOT), 'utf8')
const toolSchemas = parseToolSchemasSnapshot(toolSchemasSnapshot)
expect(pinned.length, `the pinning fixture (${pinningScenario.name}) must carry exactly one request/header`)
.toBe(1)
const pinnedHeader = restorePinnedToolSchemas(pinned[0], toolSchemas)
for (const [logIndex, log] of result.sessionLogs.entries()) {
const expectedDeltas = scenario.pinsHeader === true && logIndex === 0
? scenario.expectedHeaderDeltas ?? 0
@@ -519,11 +633,14 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
.toBe(expectedDeltas)
const headers = normalizedHeaders(scrubSystemPrompts(log.content), ctx)
const prompts = normalizedSystemPrompts(log.content, ctx)
const schemaSets = normalizedToolSchemas(log.content, ctx)
expect(prompts.length, `session ${log.id}: every request/header must carry a string system prompt`)
.toBe(headers.length)
expect(schemaSets.length, `session ${log.id}: every request/header must carry an array-valued tools field`)
.toBe(headers.length)
for (const [k, header] of headers.entries()) {
expect(header, `session ${log.id}: request/header #${k + 1} diverged from the pinned (${pinningScenario.name}) header`)
.toEqual(pinned[0])
.toEqual(pinnedHeader)
expect(formatSystemPromptSnapshot(prompts[k] as string), `session ${log.id}: initial system prompt #${k + 1} diverged from ${pinningScenario.name}/${SYSTEM_PROMPT_SNAPSHOT}`)
.toEqual(initialPromptSnapshot)
}
@@ -533,6 +650,11 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
normalizedSystemPromptDeltas(log.content, ctx),
), `session ${log.id}: system-prompt deltas diverged from ${pinningScenario.name}/${SYSTEM_PROMPT_SNAPSHOT}`)
.toEqual(promptSnapshot)
expect(formatToolSchemasSnapshot(
schemaSets[0] as unknown[],
normalizedToolSchemaDeltas(log.content, ctx),
), `session ${log.id}: tool-schema deltas diverged from ${pinningScenario.name}/${TOOL_SCHEMAS_SNAPSHOT}`)
.toEqual(toolSchemasSnapshot)
}
}
})
@@ -561,6 +683,8 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
.toBe(overridden === true)
expect(existsSync(join(dir, SYSTEM_PROMPT_SNAPSHOT)), `${name}/${SYSTEM_PROMPT_SNAPSHOT} presence must match \`pinsHeader\``)
.toBe(pinsHeader === true)
expect(existsSync(join(dir, TOOL_SCHEMAS_SNAPSHOT)), `${name}/${TOOL_SCHEMAS_SNAPSHOT} presence must match \`pinsHeader\``)
.toBe(pinsHeader === true)
// A nested-agent scenario ships one child fixture per recorded subagent
// session (`session.1.jsonl` …), the replay source for that child session.
for (const childFixture of childFixturePaths(dir, childSessions ?? 0)) {
@@ -584,7 +708,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
}
})
it('every pinning fixture carries one request/header, one readable prompt, and its declared deltas', async () => {
it('every pinning fixture carries one tokenized request/header, two sidecars, and its declared deltas', async () => {
// The live uniformity guard runs only in NON-pinning scenarios, so a class made of just
// its pinning scenario would otherwise accept a re-recorded pin with several headers or
// an undeclared mid-run header-delta — shapes the pin design cannot represent.
@@ -592,18 +716,24 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
const fixture = await readFile(join(snapshotsDir, scenario.name, 'session.jsonl'), 'utf8')
const headers = normalizedHeaders(fixture, fixtureContext(fixture))
const promptSnapshot = await readFile(join(snapshotsDir, scenario.name, SYSTEM_PROMPT_SNAPSHOT), 'utf8')
const toolSchemasSnapshot = await readFile(join(snapshotsDir, scenario.name, TOOL_SCHEMAS_SNAPSHOT), 'utf8')
const toolSchemas = parseToolSchemasSnapshot(toolSchemasSnapshot)
expect(headers.length, `${scenario.name}: a pinning fixture must carry exactly one request/header`).toBe(1)
expect(() => restorePinnedToolSchemas(headers[0], toolSchemas), `${scenario.name}: tools must use the sidecar token`)
.not.toThrow()
expect(promptSnapshot.length, `${scenario.name}/${SYSTEM_PROMPT_SNAPSHOT} must not be empty`).toBeGreaterThan(0)
expect(promptSnapshot.endsWith('\n'), `${scenario.name}/${SYSTEM_PROMPT_SNAPSHOT} must end in a newline`).toBe(true)
expect(toolSchemasSnapshot, `${scenario.name}/${TOOL_SCHEMAS_SNAPSHOT} must use canonical JSON formatting`)
.toBe(formatToolSchemasSnapshot(toolSchemas.initial, toolSchemas.deltas))
expect(headerDeltaCount(fixture), `${scenario.name}: a pinning fixture must carry exactly its declared request/header-deltas`)
.toBe(scenario.expectedHeaderDeltas ?? 0)
}
})
it('every committed JSONL has valid tool results and canonical header storage', async () => {
// System prompts always live in the readable Markdown artifact. Header
// pins keep tool schemas/prefixes in JSONL; every other fixture tokenizes
// all header bulk. Fixed-point checks make both storage rules fail loud.
// Prompts and schemas always leave JSONL. Header pins retain prefixes;
// every other fixture tokenizes those too. Fixed-point checks make both
// storage rules fail loud.
for (const scenario of scenarios) {
const dir = join(snapshotsDir, scenario.name)
const files = [
@@ -616,10 +746,9 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
.toEqual([])
expect(scrubSystemPrompts(fixture), `${scenario.name}/${file} carries an unscrubbed system prompt`)
.toEqual(fixture)
if (scenario.pinsHeader === true) {
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} must pin the non-system header content`)
.not.toEqual(fixture)
} else {
expect(scrubToolSchemas(fixture), `${scenario.name}/${file} carries unscrubbed tool schemas`)
.toEqual(fixture)
if (scenario.pinsHeader !== true) {
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} carries unscrubbed header content`)
.toEqual(fixture)
}

View File

@@ -1,2 +1,2 @@
{"type":"session","id":"ccdc749f-56f3-4267-9750-598b5c60b7b2","createdAt":600,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-nOQ4Gy"}
{"type":"request/header","seq":0,"time":4,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":[{"name":"t1","description":"D1","parameters":{"type":"object"}}]},"reason":"initial"}}
{"type":"request/header","seq":0,"time":4,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}

View File

@@ -0,0 +1,12 @@
{
"initial": [
{
"name": "t1",
"description": "D1",
"parameters": {
"type": "object"
}
}
],
"deltas": []
}

View File

@@ -1,4 +1,4 @@
{"type":"session","id":"12121212-3434-4545-8686-787878787878","createdAt":7,"cwd":"/rec/pin-cwd"}
{"type":"request/header","seq":0,"time":7,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":[{"name":"t1","description":"D1","parameters":{"type":"object"}}]},"reason":"initial"}}
{"type":"request/header","seq":0,"time":7,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
{"type":"request/header-delta","seq":1,"time":7,"data":{"system":{"keepStart":1,"keepEnd":0,"insert":["{{system}}"]}}}
{"type":"turn/start","seq":2,"time":7,"data":{"turn":1}}

View File

@@ -0,0 +1,12 @@
{
"initial": [
{
"name": "t1",
"description": "D1",
"parameters": {
"type": "object"
}
}
],
"deltas": []
}

View File

@@ -5,6 +5,7 @@ import {
normalizeStdout,
scrubRequestHeaders,
scrubSystemPrompts,
scrubToolSchemas,
} from '../src/normalize.ts'
/**
@@ -306,3 +307,45 @@ describe('scrubSystemPrompts', () => {
expect(scrubSystemPrompts(out)).toBe(out)
})
})
describe('scrubToolSchemas', () => {
it('scrubs only tool-schema payloads while keeping prompts and prefixes verbatim', () => {
const header = JSON.stringify({
type: 'request/header', seq: 1, time: 2,
data: {
header: {
system: 'full prompt',
tools: [{ name: 'read', description: 'full schema', parameters: { type: 'object' } }],
messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'full prefix' }] }],
},
reason: 'initial',
},
})
const delta = JSON.stringify({
type: 'request/header-delta', seq: 2, time: 3,
data: {
system: { keepStart: 1, keepEnd: 2, insert: ['new prompt line'] },
tools: { added: [{ name: 'grep', description: 'new schema' }], changed: [{ name: 'read', description: 'changed schema' }] },
messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'changed prefix' }] }],
},
})
const systemOnly = JSON.stringify({
type: 'request/header', seq: 3, time: 4,
data: { header: { system: 'prompt only' }, reason: 'resume' },
})
const out = scrubToolSchemas(`${header}\n${delta}\n${systemOnly}\n`)
expect(out).toContain('"tools":"{{tools}}"')
expect(out).toContain('"added":[{"name":"grep","description":"{{tools}}"}]')
expect(out).toContain('"changed":[{"name":"read","description":"{{tools}}"}]')
expect(out).not.toContain('full schema')
expect(out).not.toContain('new schema')
expect(out).not.toContain('changed schema')
expect(out).toContain('full prompt')
expect(out).toContain('new prompt line')
expect(out).toContain('full prefix')
expect(out).toContain('changed prefix')
expect(out.split('\n')[2]).toBe(systemOnly)
expect(scrubToolSchemas(out)).toBe(out)
})
})

View File

@@ -9,11 +9,16 @@ import {
childFixturePaths,
fixtureContext,
formatSystemPromptSnapshot,
formatToolSchemasSnapshot,
headerDeltaCount,
normalizedHeaders,
normalizedSystemPromptDeltas,
normalizedSystemPrompts,
normalizedToolSchemaDeltas,
normalizedToolSchemas,
parseToolSchemasSnapshot,
refreshFixtureReplacements,
restorePinnedToolSchemas,
stabilizeRefreshLog,
unknownToolCallIds,
} from '../src/suite.ts'
@@ -71,6 +76,7 @@ afterAll(async () => {
function staleRefreshFixtures(dir: string): void {
writeFileSync(join(dir, 'plain-turn', 'stdout.golden.jsonl'), 'stale stdout\n')
writeFileSync(join(dir, 'pin-turn', 'system-prompt.golden.md'), 'STALE PROMPT\n')
writeFileSync(join(dir, 'pin-turn', 'tool-schemas.golden.json'), '{"initial":[{"name":"stale"}],"deltas":[]}\n')
const plainBehaviorFile = join(dir, 'plain-turn', 'behavior.json')
const plainBehavior = JSON.parse(readFileSync(plainBehaviorFile, 'utf8')) as Record<string, unknown>
@@ -126,6 +132,9 @@ describe('defineAcpSnapshotSuite: refresh write-back', () => {
'NEW PROMPT LINE',
'',
].join('\n'))
const schemas = readFileSync(join(refreshDir, 'pin-turn', 'tool-schemas.golden.json'), 'utf8')
expect(schemas).toContain('"description": "D1"')
expect(schemas).not.toContain('"name":"stale"')
})
})
@@ -236,6 +245,40 @@ describe('normalizedSystemPrompts', () => {
})
})
describe('normalizedToolSchemas', () => {
it('extracts normalized schema arrays and omits absent or non-array fields', () => {
const log = [
'{"type":"session","id":"a","createdAt":5,"cwd":"/w"}',
'{"type":"request/header","seq":0,"time":9,"data":{"header":{"tools":[{"name":"read","description":"work in /w"}]}}}',
'{"type":"request/header","seq":1,"time":9,"data":{"header":{}}}',
'{"type":"request/header","seq":2,"time":9,"data":{"header":{"tools":null}}}',
'{"type":"request/header","seq":3,"time":9,"data":{"header":null}}',
'{"type":"request/header","seq":4,"time":9,"data":{"header":"invalid"}}',
'',
].join('\n')
expect(normalizedToolSchemas(log, { sessionIds: [], cwd: '/w' })).toEqual([
[{ name: 'read', description: 'work in {{cwd}}' }],
])
})
})
describe('normalizedToolSchemaDeltas', () => {
it('extracts and normalizes object-valued schema edits', () => {
const log = [
'{"type":"request/header-delta","data":{"tools":{"added":[{"name":"read","description":"work in /w"}]}}}',
'{"type":"request/header-delta","data":{"tools":null}}',
'{"type":"request/header-delta","data":{"tools":"invalid"}}',
'{"type":"request/header-delta","data":{"tools":[]}}',
'{"type":"request/header-delta","data":{"system":{"insert":[]}}}',
'{"type":"request/header","data":{"tools":{"added":[]}}}',
'',
].join('\n')
expect(normalizedToolSchemaDeltas(log, { sessionIds: [], cwd: '/w' })).toEqual([
{ added: [{ name: 'read', description: 'work in {{cwd}}' }] },
])
})
})
describe('normalizedSystemPromptDeltas', () => {
it('extracts and normalizes well-formed system edits', () => {
const log = [
@@ -270,6 +313,39 @@ describe('formatSystemPromptSnapshot', () => {
})
})
describe('tool-schema snapshots', () => {
const snapshot = {
initial: [{ name: 'read', description: 'Read a file.' }],
deltas: [{ added: [{ name: 'grep', description: 'Search files.' }] }],
}
it('formats and parses canonical structured JSON', () => {
const formatted = formatToolSchemasSnapshot(snapshot.initial, snapshot.deltas)
expect(formatted).toBe(`${JSON.stringify(snapshot, null, 2)}\n`)
expect(parseToolSchemasSnapshot(formatted)).toEqual(snapshot)
})
it('rejects invalid top-level and field shapes', () => {
expect(() => parseToolSchemasSnapshot('null')).toThrow(/must be an object/)
expect(() => parseToolSchemasSnapshot('"invalid"')).toThrow(/must be an object/)
expect(() => parseToolSchemasSnapshot('[]')).toThrow(/must be an object/)
expect(() => parseToolSchemasSnapshot('{"initial":{},"deltas":[]}')).toThrow(/array-valued/)
expect(() => parseToolSchemasSnapshot('{"initial":[],"deltas":{}}')).toThrow(/array-valued/)
})
it('restores initial schemas into the pinned header token', () => {
expect(restorePinnedToolSchemas({ system: '{{system}}', tools: '{{tools}}' }, snapshot))
.toEqual({ system: '{{system}}', tools: snapshot.initial })
})
it('rejects invalid headers and a missing tool token', () => {
expect(() => restorePinnedToolSchemas(null, snapshot)).toThrow(/must be an object/)
expect(() => restorePinnedToolSchemas('invalid', snapshot)).toThrow(/must be an object/)
expect(() => restorePinnedToolSchemas([], snapshot)).toThrow(/must be an object/)
expect(() => restorePinnedToolSchemas({ tools: [] }, snapshot)).toThrow(/must equal/)
})
})
describe('headerDeltaCount', () => {
it('counts request/header-delta events, ignoring blanks and other lines', () => {
const delta = JSON.stringify({ type: 'request/header-delta', seq: 2, time: 9, data: {} })