Pin request-header content in one snapshot scenario
Every session.jsonl fixture embedded the full composed system prompt and
complete tool-schema list in its request/header event (~8 KB on one line,
identical across the suite), so any prompt or tool-schema edit forced a
re-record or hand-edit of every fixture — see the dynamic-workflows PR for
the churn pattern this removes.
Now exactly one scenario (text-turn, flagged pinsHeader) commits and
compares that content verbatim; every other fixture stores and compares it
as {{system}}/{{tools}} tokens via the new pure scrubRequestHeaders
normalizer (applied to both compare sides and to record-mode writes, so a
re-record cannot reintroduce the content). request/header-delta payloads
are scrubbed the same way; config/reason stay verbatim — a model swap
SHOULD churn every fixture, a prompt edit should not. Replay is unaffected:
script derivation reads only assistant/chunk events.
Fixture meta-guards enforce the split: non-pinning fixtures must be fixed
points of the scrub, the pinning fixture must not be, and exactly one
scenario pins. Committed fixtures migrated through the same function.
Docs: pinned-header RFC (implemented/testing), base snapshot RFC + testing
policy + llm-replay module doc/README updated.
This commit is contained in:
@@ -12,11 +12,21 @@
|
||||
* `durationMs` (wall-clock hook runtime) → 0. NOT scrubbed: the log's `seq`
|
||||
* (deterministic — `seq = log.length`, part of the event-log contract).
|
||||
*
|
||||
* A separate, composable normalizer — {@link scrubRequestHeaders} — replaces
|
||||
* the bulky request-header CONTENT (the composed system prompt and the tool
|
||||
* schema list) with `{{system}}`/`{{tools}}` tokens. It is deliberately NOT
|
||||
* folded into {@link normalizeSessionLog}: the one header-pinning scenario
|
||||
* compares that content verbatim, every other scenario composes the scrub in
|
||||
* (the `pinsHeader` flag in acp.snapshot.ts; see the pinned-header RFC,
|
||||
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
|
||||
*
|
||||
* See docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
|
||||
*/
|
||||
|
||||
const SESSION_ID = '{{sessionId}}'
|
||||
const CWD = '{{cwd}}'
|
||||
const SYSTEM = '{{system}}'
|
||||
const TOOLS = '{{tools}}'
|
||||
|
||||
/** A UUID v4 string, the shape `randomUUID()` produces for session ids. */
|
||||
const UUID_RE = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi
|
||||
@@ -110,3 +120,46 @@ export function normalizeSessionLog(rawLog: string, ctx: NormalizeContext): stri
|
||||
})
|
||||
return records.map(r => JSON.stringify(r)).join('\n') + '\n'
|
||||
}
|
||||
|
||||
/**
|
||||
* Replace request-header CONTENT in a session JSONL with stable tokens: a
|
||||
* `request/header` event's `data.header.system` → `{{system}}` and
|
||||
* `data.header.tools` → `{{tools}}`, and a `request/header-delta` event's
|
||||
* `data.system`/`data.tools` payloads likewise (a delta embeds prompt/schema
|
||||
* fragments, so leaving it raw would reopen the churn this scrub closes).
|
||||
* Absent fields stay absent — WHETHER a header carried a system prompt or
|
||||
* tools is behavior and stays visible; `config` and `reason` are small and
|
||||
* stable, so they stay verbatim (a model swap SHOULD churn every fixture — it
|
||||
* invalidates the recorded responses; a prompt/schema edit does NOT — replay
|
||||
* never reads this content, see dsh-llm-replay).
|
||||
*
|
||||
* Only lines with something to scrub are re-serialized; every other line
|
||||
* passes through byte-for-byte, so the transform is idempotent and applying
|
||||
* it to an already-scrubbed fixture is a no-op — the on-disk-fixtures guard
|
||||
* in acp.snapshot.ts relies on exactly that.
|
||||
*/
|
||||
export function scrubRequestHeaders(rawLog: string): string {
|
||||
const lines = rawLog.split('\n')
|
||||
const out = lines.map((line) => {
|
||||
if (line.trim().length === 0) return line
|
||||
const record = JSON.parse(line) as Record<string, unknown>
|
||||
const data = record.data as Record<string, unknown> | null | undefined
|
||||
if (data === null || typeof data !== 'object') return line
|
||||
if (record.type === 'request/header') {
|
||||
const header = data.header as Record<string, unknown> | null | undefined
|
||||
if (header === null || typeof header !== 'object') return line
|
||||
if (!('system' in header) && !('tools' in header)) return line
|
||||
if ('system' in header) header.system = SYSTEM
|
||||
if ('tools' in header) header.tools = TOOLS
|
||||
return JSON.stringify(record)
|
||||
}
|
||||
if (record.type === 'request/header-delta') {
|
||||
if (!('system' in data) && !('tools' in data)) return line
|
||||
if ('system' in data) data.system = SYSTEM
|
||||
if ('tools' in data) data.tools = TOOLS
|
||||
return JSON.stringify(record)
|
||||
}
|
||||
return line
|
||||
})
|
||||
return out.join('\n')
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user