Pin request-header content in one snapshot scenario

Every session.jsonl fixture embedded the full composed system prompt and
complete tool-schema list in its request/header event (~8 KB on one line,
identical across the suite), so any prompt or tool-schema edit forced a
re-record or hand-edit of every fixture — see the dynamic-workflows PR for
the churn pattern this removes.

Now exactly one scenario (text-turn, flagged pinsHeader) commits and
compares that content verbatim; every other fixture stores and compares it
as {{system}}/{{tools}} tokens via the new pure scrubRequestHeaders
normalizer (applied to both compare sides and to record-mode writes, so a
re-record cannot reintroduce the content). request/header-delta payloads
are scrubbed the same way; config/reason stay verbatim — a model swap
SHOULD churn every fixture, a prompt edit should not. Replay is unaffected:
script derivation reads only assistant/chunk events.

Fixture meta-guards enforce the split: non-pinning fixtures must be fixed
points of the scrub, the pinning fixture must not be, and exactly one
scenario pins. Committed fixtures migrated through the same function.

Docs: pinned-header RFC (implemented/testing), base snapshot RFC + testing
policy + llm-replay module doc/README updated.
This commit is contained in:
Tianyi Cui
2026-07-06 23:41:20 +08:00
parent 0606cd559c
commit 1fd9e2a9ab
43 changed files with 254 additions and 48 deletions

View File

@@ -12,11 +12,21 @@
* `durationMs` (wall-clock hook runtime) → 0. NOT scrubbed: the log's `seq`
* (deterministic — `seq = log.length`, part of the event-log contract).
*
* A separate, composable normalizer — {@link scrubRequestHeaders} — replaces
* the bulky request-header CONTENT (the composed system prompt and the tool
* schema list) with `{{system}}`/`{{tools}}` tokens. It is deliberately NOT
* folded into {@link normalizeSessionLog}: the one header-pinning scenario
* compares that content verbatim, every other scenario composes the scrub in
* (the `pinsHeader` flag in acp.snapshot.ts; see the pinned-header RFC,
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
*
* See docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
*/
const SESSION_ID = '{{sessionId}}'
const CWD = '{{cwd}}'
const SYSTEM = '{{system}}'
const TOOLS = '{{tools}}'
/** A UUID v4 string, the shape `randomUUID()` produces for session ids. */
const UUID_RE = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi
@@ -110,3 +120,46 @@ export function normalizeSessionLog(rawLog: string, ctx: NormalizeContext): stri
})
return records.map(r => JSON.stringify(r)).join('\n') + '\n'
}
/**
* Replace request-header CONTENT in a session JSONL with stable tokens: a
* `request/header` event's `data.header.system` → `{{system}}` and
* `data.header.tools` → `{{tools}}`, and a `request/header-delta` event's
* `data.system`/`data.tools` payloads likewise (a delta embeds prompt/schema
* fragments, so leaving it raw would reopen the churn this scrub closes).
* Absent fields stay absent — WHETHER a header carried a system prompt or
* tools is behavior and stays visible; `config` and `reason` are small and
* stable, so they stay verbatim (a model swap SHOULD churn every fixture — it
* invalidates the recorded responses; a prompt/schema edit does NOT — replay
* never reads this content, see dsh-llm-replay).
*
* Only lines with something to scrub are re-serialized; every other line
* passes through byte-for-byte, so the transform is idempotent and applying
* it to an already-scrubbed fixture is a no-op — the on-disk-fixtures guard
* in acp.snapshot.ts relies on exactly that.
*/
export function scrubRequestHeaders(rawLog: string): string {
const lines = rawLog.split('\n')
const out = lines.map((line) => {
if (line.trim().length === 0) return line
const record = JSON.parse(line) as Record<string, unknown>
const data = record.data as Record<string, unknown> | null | undefined
if (data === null || typeof data !== 'object') return line
if (record.type === 'request/header') {
const header = data.header as Record<string, unknown> | null | undefined
if (header === null || typeof header !== 'object') return line
if (!('system' in header) && !('tools' in header)) return line
if ('system' in header) header.system = SYSTEM
if ('tools' in header) header.tools = TOOLS
return JSON.stringify(record)
}
if (record.type === 'request/header-delta') {
if (!('system' in data) && !('tools' in data)) return line
if ('system' in data) data.system = SYSTEM
if ('tools' in data) data.tools = TOOLS
return JSON.stringify(record)
}
return line
})
return out.join('\n')
}