Pin request-header content in one snapshot scenario

Every session.jsonl fixture embedded the full composed system prompt and
complete tool-schema list in its request/header event (~8 KB on one line,
identical across the suite), so any prompt or tool-schema edit forced a
re-record or hand-edit of every fixture — see the dynamic-workflows PR for
the churn pattern this removes.

Now exactly one scenario (text-turn, flagged pinsHeader) commits and
compares that content verbatim; every other fixture stores and compares it
as {{system}}/{{tools}} tokens via the new pure scrubRequestHeaders
normalizer (applied to both compare sides and to record-mode writes, so a
re-record cannot reintroduce the content). request/header-delta payloads
are scrubbed the same way; config/reason stay verbatim — a model swap
SHOULD churn every fixture, a prompt edit should not. Replay is unaffected:
script derivation reads only assistant/chunk events.

Fixture meta-guards enforce the split: non-pinning fixtures must be fixed
points of the scrub, the pinning fixture must not be, and exactly one
scenario pins. Committed fixtures migrated through the same function.

Docs: pinned-header RFC (implemented/testing), base snapshot RFC + testing
policy + llm-replay module doc/README updated.
This commit is contained in:
Tianyi Cui
2026-07-06 23:41:20 +08:00
parent 0606cd559c
commit 1fd9e2a9ab
43 changed files with 254 additions and 48 deletions

View File

@@ -4,7 +4,7 @@ import { fileURLToPath } from 'node:url'
import { dirname, join } from 'node:path'
import { describe, expect, it } from 'vitest'
import { type HarvestedLog, type InputScript, runScenario } from './snapshot-harness.ts'
import { type NormalizeContext, normalizeSessionLog, normalizeStdout } from './snapshot-normalize.ts'
import { type NormalizeContext, normalizeSessionLog, normalizeStdout, scrubRequestHeaders } from './snapshot-normalize.ts'
/**
* ACP snapshot tests (REPLAY by default, keyless). Each scenario under
@@ -16,6 +16,13 @@ import { type NormalizeContext, normalizeSessionLog, normalizeStdout } from './s
* golden: the fixture doubles as the replay source (recorded scenarios) and the
* expected produced log (both sides normalized before comparing).
*
* Request-header content (the composed system prompt + tool schemas riding on
* `request/header` events) is pinned by exactly ONE scenario — the one with
* `pinsHeader` — and scrubbed to `{{system}}`/`{{tools}}` tokens in every
* other fixture and compare, so a prompt or tool-schema edit churns one
* committed line instead of every fixture (see the pinned-header RFC,
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
*
* `pnpm run test:snapshot:record` (DSH_SNAPSHOT=record + -u) re-records the
* `session.jsonl` fixtures against the real API and refreshes the stdout golden
* in one pass.
@@ -55,12 +62,29 @@ interface Scenario {
* harvested child logs back to those files. Defaults to 0.
*/
childSessions?: number
/**
* Whether THIS scenario's fixtures keep the full request-header content (the
* composed system prompt and tool schema list on `request/header` /
* `request/header-delta` events) and compare it verbatim. Exactly one
* scenario pins it; every other scenario stores and compares that content as
* `{{system}}`/`{{tools}}` tokens ({@link scrubRequestHeaders}), so a system
* prompt or tool-schema change shows up as ONE committed-fixture diff, not
* one per scenario. Today every fixture (parent, spawn child, fork child)
* carries the identical prompt-modulo-cwd and identical tools, so one pin
* covers the whole suite; if header composition ever becomes
* session-dependent (say, a restricted subagent toolset), pin one scenario
* per distinct header shape. Defaults to false.
*/
pinsHeader?: boolean
}
const SCENARIOS: Scenario[] = [
{ name: 'handshake', hasModelTurn: false, recorded: false },
{ name: 'reject-extra-dirs', hasModelTurn: false, recorded: false },
{ name: 'text-turn', hasModelTurn: true, recorded: true },
// text-turn is the pinned-header scenario: the minimal single text turn,
// whose fixture is the ONE place the full system prompt + tool schemas are
// committed and compared verbatim.
{ name: 'text-turn', hasModelTurn: true, recorded: true, pinsHeader: true },
{ name: 'tool-call-turn', hasModelTurn: true, recorded: true },
{ name: 'fs-terminal-card', hasModelTurn: true, recorded: true },
{ name: 'todo-plan', hasModelTurn: true, recorded: true },
@@ -181,14 +205,19 @@ for (const scenario of SCENARIOS) {
// RECORD mode (recorded model scenarios only): persist the freshly-harvested
// logs back to their fixtures — the primary to session.jsonl, each child to
// session.<n>.jsonl in harvest order. `--update` refreshes the Vitest
// goldens but NOT these fixtures, so write them here.
// goldens but NOT these fixtures, so write them here. A non-pinning
// scenario's fixtures are written header-scrubbed, so a re-record can
// never smuggle the full prompt/schema content back into every fixture.
const scrub = scenario.pinsHeader === true
? (log: string): string => log
: scrubRequestHeaders
if (RECORDING && scenario.recorded && scenario.hasModelTurn) {
expect(result.sessionLogs.length, 'record produced no session log to harvest').toBeGreaterThan(0)
expect(result.sessionLogs.length, `expected ${childSessions + 1} session logs (parent + children)`)
.toBe(childSessions + 1)
await writeFile(join(dir, 'session.jsonl'), (result.sessionLogs[0] as HarvestedLog).content)
await writeFile(join(dir, 'session.jsonl'), scrub((result.sessionLogs[0] as HarvestedLog).content))
for (let i = 1; i < result.sessionLogs.length; i++) {
await writeFile(join(dir, `session.${i}.jsonl`), (result.sessionLogs[i] as HarvestedLog).content)
await writeFile(join(dir, `session.${i}.jsonl`), scrub((result.sessionLogs[i] as HarvestedLog).content))
}
}
@@ -203,11 +232,17 @@ for (const scenario of SCENARIOS) {
// 1:1. Each side passes through normalizeSessionLog, scrubbed against ITS
// OWN volatile values — the live run's via `ctx`, the committed fixture's
// via its own header (a committed file cannot share the live run's ids).
// Unless this scenario pins the header, both sides ALSO pass through
// scrubRequestHeaders: the live log carries the real prompt/schemas, the
// fixture carries the `{{system}}`/`{{tools}}` tokens, and the scrub is
// idempotent — so the compare checks the header's presence, position,
// reason, and config, but not its bulk content (pinned once, in the
// `pinsHeader` scenario).
expect(result.sessionLogs.length, 'this scenario must persist a session log').toBe(childSessions + 1)
const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)]
for (let i = 0; i < fixtureFiles.length; i++) {
const harvested = (result.sessionLogs[i] as HarvestedLog).content
const fixture = await readFile(join(dir, fixtureFiles[i] as string), 'utf8')
const harvested = scrub((result.sessionLogs[i] as HarvestedLog).content)
const fixture = scrub(await readFile(join(dir, fixtureFiles[i] as string), 'utf8'))
expect(normalizeSessionLog(harvested, ctx), `${fixtureFiles[i]} mismatch`)
.toEqual(normalizeSessionLog(fixture, fixtureContext(fixture)))
}
@@ -253,4 +288,37 @@ describe('snapshot fixtures', () => {
}
}
})
it('exactly one scenario pins the request-header content', () => {
// Zero pins would drop the prompt/schema surface from the suite entirely;
// two would split it. The single pin is the design (pinned-header RFC).
expect(SCENARIOS.filter(s => s.pinsHeader === true).map(s => s.name)).toEqual(['text-turn'])
})
it('committed fixtures carry request-header content ONLY in the pinning scenario', async () => {
// The whole point of the pin: a system-prompt or tool-schema change must
// churn exactly one committed line. A non-pinning fixture that carries the
// full header (a hand-recorded file, or a header line hand-edited out of
// its canonical JSON form) silently reopens the suite-wide churn, so fail
// loud here: every non-pinning session*.jsonl must be a fixed point of
// scrubRequestHeaders (apply the scrub to fix a violation), and the
// pinning scenario's fixtures must NOT be (their content IS the pin).
for (const scenario of SCENARIOS) {
const dir = join(SNAPSHOTS_DIR, scenario.name)
const files = [
'session.jsonl',
...Array.from({ length: scenario.childSessions ?? 0 }, (_, i) => `session.${i + 1}.jsonl`),
]
for (const file of files) {
const fixture = await readFile(join(dir, file), 'utf8')
if (scenario.pinsHeader === true) {
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} must PIN the full header content`)
.not.toEqual(fixture)
} else {
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} carries unscrubbed header content`)
.toEqual(fixture)
}
}
}
})
})

View File

@@ -1,5 +1,5 @@
import { describe, expect, it } from 'vitest'
import { type NormalizeContext, normalizeSessionLog, normalizeStdout } from '../tests/snapshot-normalize.ts'
import { type NormalizeContext, normalizeSessionLog, normalizeStdout, scrubRequestHeaders } from '../tests/snapshot-normalize.ts'
/**
* Unit tests for the pure snapshot normalizers. Live as a *.spec.ts (runs in
@@ -108,3 +108,51 @@ describe('normalizeSessionLog', () => {
expect(out).toContain('"durationMs":88')
})
})
describe('scrubRequestHeaders', () => {
const headerLine = JSON.stringify({ type: 'session', version: 0, id: 's', createdAt: 1, cwd: '/w' })
const headerEvent = (header: object) =>
JSON.stringify({ type: 'request/header', seq: 3, time: 9, data: { header, reason: 'initial' } })
it('replaces header system and tools with tokens, keeping config and reason', () => {
const ev = headerEvent({
config: { model: 'm' },
system: 'You are an agent.\nBe brief.',
tools: [{ name: 'read', description: 'Read a file.', parameters: { type: 'object' } }],
})
const out = scrubRequestHeaders(`${headerLine}\n${ev}\n`)
expect(out).toContain('"system":"{{system}}"')
expect(out).toContain('"tools":"{{tools}}"')
expect(out).toContain('"config":{"model":"m"}')
expect(out).toContain('"reason":"initial"')
expect(out).not.toContain('You are an agent')
expect(out).not.toContain('Read a file')
})
it('keeps an absent system/tools absent (presence is behavior)', () => {
const out = scrubRequestHeaders(`${headerLine}\n${headerEvent({ config: { model: 'm' } })}\n`)
expect(out).not.toContain('{{system}}')
expect(out).not.toContain('{{tools}}')
})
it('scrubs a request/header-delta system/tools payload', () => {
const delta = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: { system: { keepStart: 1, keepEnd: 0, insert: ['leaked prompt line'] }, config: { model: 'm2' } },
})
const out = scrubRequestHeaders(`${headerLine}\n${delta}\n`)
expect(out).toContain('"system":"{{system}}"')
expect(out).toContain('"config":{"model":"m2"}')
expect(out).not.toContain('leaked prompt line')
expect(out).not.toContain('{{tools}}') // no tools delta → none invented
})
it('passes every other line through byte-for-byte and is idempotent', () => {
const other = JSON.stringify({ type: 'assistant/chunk', seq: 4, time: 9, data: { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'hi' } } })
const raw = `${headerLine}\n${headerEvent({ config: { model: 'm' }, system: 's', tools: [] })}\n${other}\n`
const once = scrubRequestHeaders(raw)
expect(once.split('\n')[0]).toBe(headerLine)
expect(once.split('\n')[2]).toBe(other)
expect(scrubRequestHeaders(once)).toBe(once)
})
})

View File

@@ -12,11 +12,21 @@
* `durationMs` (wall-clock hook runtime) → 0. NOT scrubbed: the log's `seq`
* (deterministic — `seq = log.length`, part of the event-log contract).
*
* A separate, composable normalizer — {@link scrubRequestHeaders} — replaces
* the bulky request-header CONTENT (the composed system prompt and the tool
* schema list) with `{{system}}`/`{{tools}}` tokens. It is deliberately NOT
* folded into {@link normalizeSessionLog}: the one header-pinning scenario
* compares that content verbatim, every other scenario composes the scrub in
* (the `pinsHeader` flag in acp.snapshot.ts; see the pinned-header RFC,
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
*
* See docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
*/
const SESSION_ID = '{{sessionId}}'
const CWD = '{{cwd}}'
const SYSTEM = '{{system}}'
const TOOLS = '{{tools}}'
/** A UUID v4 string, the shape `randomUUID()` produces for session ids. */
const UUID_RE = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi
@@ -110,3 +120,46 @@ export function normalizeSessionLog(rawLog: string, ctx: NormalizeContext): stri
})
return records.map(r => JSON.stringify(r)).join('\n') + '\n'
}
/**
* Replace request-header CONTENT in a session JSONL with stable tokens: a
* `request/header` event's `data.header.system` → `{{system}}` and
* `data.header.tools` → `{{tools}}`, and a `request/header-delta` event's
* `data.system`/`data.tools` payloads likewise (a delta embeds prompt/schema
* fragments, so leaving it raw would reopen the churn this scrub closes).
* Absent fields stay absent — WHETHER a header carried a system prompt or
* tools is behavior and stays visible; `config` and `reason` are small and
* stable, so they stay verbatim (a model swap SHOULD churn every fixture — it
* invalidates the recorded responses; a prompt/schema edit does NOT — replay
* never reads this content, see dsh-llm-replay).
*
* Only lines with something to scrub are re-serialized; every other line
* passes through byte-for-byte, so the transform is idempotent and applying
* it to an already-scrubbed fixture is a no-op — the on-disk-fixtures guard
* in acp.snapshot.ts relies on exactly that.
*/
export function scrubRequestHeaders(rawLog: string): string {
const lines = rawLog.split('\n')
const out = lines.map((line) => {
if (line.trim().length === 0) return line
const record = JSON.parse(line) as Record<string, unknown>
const data = record.data as Record<string, unknown> | null | undefined
if (data === null || typeof data !== 'object') return line
if (record.type === 'request/header') {
const header = data.header as Record<string, unknown> | null | undefined
if (header === null || typeof header !== 'object') return line
if (!('system' in header) && !('tools' in header)) return line
if ('system' in header) header.system = SYSTEM
if ('tools' in header) header.tools = TOOLS
return JSON.stringify(record)
}
if (record.type === 'request/header-delta') {
if (!('system' in data) && !('tools' in data)) return line
if ('system' in data) data.system = SYSTEM
if ('tools' in data) data.tools = TOOLS
return JSON.stringify(record)
}
return line
})
return out.join('\n')
}

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long