Pin request-header content in one snapshot scenario

Every session.jsonl fixture embedded the full composed system prompt and
complete tool-schema list in its request/header event (~8 KB on one line,
identical across the suite), so any prompt or tool-schema edit forced a
re-record or hand-edit of every fixture — see the dynamic-workflows PR for
the churn pattern this removes.

Now exactly one scenario (text-turn, flagged pinsHeader) commits and
compares that content verbatim; every other fixture stores and compares it
as {{system}}/{{tools}} tokens via the new pure scrubRequestHeaders
normalizer (applied to both compare sides and to record-mode writes, so a
re-record cannot reintroduce the content). request/header-delta payloads
are scrubbed the same way; config/reason stay verbatim — a model swap
SHOULD churn every fixture, a prompt edit should not. Replay is unaffected:
script derivation reads only assistant/chunk events.

Fixture meta-guards enforce the split: non-pinning fixtures must be fixed
points of the scrub, the pinning fixture must not be, and exactly one
scenario pins. Committed fixtures migrated through the same function.

Docs: pinned-header RFC (implemented/testing), base snapshot RFC + testing
policy + llm-replay module doc/README updated.
This commit is contained in:
Tianyi Cui
2026-07-06 23:41:20 +08:00
parent 0606cd559c
commit 1fd9e2a9ab
43 changed files with 254 additions and 48 deletions

View File

@@ -158,6 +158,7 @@ Generated by `pnpm run gen-rfc-index` from the RFC tree — never edit by hand;
| [Per-session snapshot replay for nested agents](implemented/testing/2026-06-22-subagent-snapshot-replay.md) | 2026-06-22 |
| [Hook snapshot matrix — end-to-end goldens for both bridges](implemented/testing/2026-07-04-hook-snapshot-matrix.md) | 2026-07-04 |
| [Single-source the acp-agent replay config](implemented/testing/2026-07-04-single-source-acp-replay-config.md) | 2026-07-04 |
| [Pin request-header content in one snapshot scenario](implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md) | 2026-07-06 |
## Rejected

View File

@@ -51,7 +51,7 @@ The ACP server app loads `@deepseek-ai/dsh-llm-deepseek`, whose `apply` throws w
A snapshot run asserts **two** normalized surfaces, because the harness's external surfaces are distinct:
1. The **stdout transcript** — the framed `session/update` JSON-RPC the editor sees. Catches regressions in the ACP bridge's event→update translation (`streamSessionEventUpdate`). Compared against a committed `stdout.golden.jsonl`.
2. The **re-persisted session JSONL** — the log the replay run itself persists, compared against the scenario's `session.jsonl`. Catches regressions in the loop, tool dispatch, and turn/step structure that never surface on stdout. There is no separate session golden: `session.jsonl` is BOTH the replay source (recorded scenarios) and the expected produced log. Both sides pass through `normalizeSessionLog` before comparing — the fixture is raw-harvested (its own real session id / cwd / timestamps) and the replay output has fresh ones, so each is scrubbed against ITS OWN volatile values (the fixture's read from its header line) and the comparison is on normalized form. For an authored override scenario the same `session.jsonl` holds the expected produced log; `replay.override.json` drives the model, and `llm-replay` ignores the fixture for model chunks when an override exists, so committing the expected log there does not affect replay.
2. The **re-persisted session JSONL** — the log the replay run itself persists, compared against the scenario's `session.jsonl`. Catches regressions in the loop, tool dispatch, and turn/step structure that never surface on stdout. There is no separate session golden: `session.jsonl` is BOTH the replay source (recorded scenarios) and the expected produced log. Both sides pass through `normalizeSessionLog` before comparing — the fixture is raw-harvested (its own real session id / cwd / timestamps) and the replay output has fresh ones, so each is scrubbed against ITS OWN volatile values (the fixture's read from its header line) and the comparison is on normalized form. Request-header CONTENT (the composed system prompt + tool schemas) is additionally scrubbed to `{{system}}`/`{{tools}}` tokens on both sides — in the stored fixtures too — for every scenario except the one that pins it ([pinned-header RFC](2026-07-06-pin-request-header-content-in-one-scenario.md)). For an authored override scenario the same `session.jsonl` holds the expected produced log; `replay.override.json` drives the model, and `llm-replay` ignores the fixture for model chunks when an override exists, so committing the expected log there does not affect replay.
The two are genuinely additive: stdout is the bridge's *lossy projection* of the log (it drops `assistant/message.usage`, `step/*`, exact `seq`/`time`, and renders tool I/O differently), so a loop/tool/turn-structure regression can change the JSONL while leaving the stdout projection identical, and a bridge-translation regression can change stdout while the JSONL is untouched. Asserting the JSONL equality also echoes the proposed [universal replay fixture](../../proposed/testing/2026-06-11-deterministic-and-stress-testing.md) idea.

View File

@@ -0,0 +1,32 @@
# RFC: Pin request-header content in one snapshot scenario
Status: implemented
## Problem
Every model-driving ACP snapshot fixture (`session.jsonl`) embedded the full composed system prompt and the complete tool-schema list in its `request/header` event — roughly 8 KB on one line, per fixture. That content is identical across the suite (byte-identical tool list everywhere, including subagent children; identical prompt modulo each recording's temp cwd), so any PR touching a tool description or a system-prompt line had to update every fixture: re-record everything against the live API (churning model responses and stdout goldens along the way) or hand-edit ~35 giant header lines. The dynamic-workflows PR (#170) is the canonical example — adding one tool and one prompt paragraph rewrote every snapshot fixture in the repo, burying the behavioral diff a reviewer should be reading.
## Decision
Exactly one scenario — `text-turn`, flagged `pinsHeader` in `acp.snapshot.ts` — commits and compares the full request-header content. Every other fixture stores and compares that content as stable tokens: a `request/header` event's `header.system` becomes `"{{system}}"` and `header.tools` becomes `"{{tools}}"`, and a `request/header-delta`'s `system`/`tools` payloads likewise (a delta embeds prompt/schema fragments, so leaving it raw would reopen the churn). The scrub is a pure normalizer, `scrubRequestHeaders` in `snapshot-normalize.ts`, composed in front of `normalizeSessionLog` on BOTH sides of a non-pinning scenario's log compare and applied to the harvested logs record mode writes, so a re-record cannot smuggle the content back. Absent fields stay absent — WHETHER a header carried a prompt or tools is behavior and stays visible — and `config`/`reason` stay verbatim: a model swap SHOULD churn every fixture (it invalidates the recorded responses), while a prompt or schema edit does not (replay derives model behavior exclusively from `assistant/chunk` events and never reads header content — see `dsh-llm-replay`).
A system-prompt or tool-schema change therefore lands as exactly one committed-fixture diff — the pinned `text-turn` header line — updated by hand or by re-recording that one scenario (`pnpm run test:snapshot:record` with `-t text-turn`).
Guards in the fixture meta-tests make the split self-enforcing: every non-pinning `session*.jsonl` must be a fixed point of `scrubRequestHeaders` (unscrubbed content crept in — apply the scrub), the pinning scenario's fixture must NOT be one (the pin lost its content), and exactly one scenario must pin.
Today one pin covers the whole suite because every fixture — parent, spawn child, fork child — records the identical tool list and the identical prompt modulo cwd. If header composition ever becomes session-dependent (a restricted subagent toolset, say), the flag extends to one pinning scenario per distinct header shape.
## Alternatives considered
- **Re-record or hand-edit every fixture per change** — the status quo; the churn this RFC removes.
- **Scrub at compare time only, keeping fixtures raw** — the compares go green without fixture edits, but every committed fixture then carries a permanently stale copy of the prompt and schemas: dead weight that misleads readers and still rewrites wholesale on the next re-record. Storing the tokens keeps the fixture honest about what it does and does not pin.
- **Scrub everywhere, pin nowhere** — loses the only end-to-end record of the composed header as actually sent (prompt assembly, registered-tool order, full schemas). The generated tool catalog documents each tool in isolation; only a real fixture pins the composed set.
- **Slim the session log itself (log a content digest, store the header elsewhere)** — violates the reconstructability contract: the product log must reproduce each request bit-for-bit ([reconstructable-requests RFC](../architecture/2026-07-05-reconstructable-requests.md)). Header bulk is a test-artifact concern, solved in test normalization; the live log is untouched.
## Verification
All 37 snapshot scenarios replay green with the scrubbed fixtures (the committed fixtures were rewritten once through `scrubRequestHeaders` itself; `text-turn` untouched). The fixed-point, pin-retains-content, and exactly-one-pin guards run inside the suite, and `scrubRequestHeaders` has unit coverage for both header event types, absent-field preservation, config/reason retention, byte-for-byte pass-through of other lines, and idempotence.
## Consequences
A tool-description or system-prompt change churns one committed fixture line instead of every fixture in the suite, so snapshot diffs read as behavior again, and ~270 KB of duplicated header bytes leave the repo. The cost: fixtures no longer show per-scenario header content, so a non-uniform header would be invisible outside the pinned scenario — accepted while composition is provably suite-uniform (the recording is made by one `cordis.yml` for all scenarios), and revisited with additional pins if that ever changes.

View File

@@ -7,7 +7,7 @@ How this repo tests, tier by tier, and the rules that keep a green suite meaning
- **Unit** (`pnpm run test`): vitest over `packages|examples/*/tests/**/*.spec.ts`, colocated with what they test. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Excessive tests are welcome — err toward covering edge cases, error paths, event ordering, and concurrency races; review findings get regression tests (see `packages/core/agent-loop/tests/review-fixes.spec.ts`).
- **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped.
- **Real-API e2e** (`pnpm run test:e2e`): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (`EXA_API_KEY`, `PERPLEXITY_API_KEY`, …); each suite self-skips without its key so keyless CI stays green ([real-API e2e RFC](rfc/implemented/testing/2026-06-19-real-api-e2e-ci.md)).
- **Snapshot** (`pnpm run test:snapshot`): boots the real example subprocess, replays a recorded session keyless, diffs normalized stdout + the re-persisted log against committed goldens ([snapshot RFC](rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md)). Re-record with `pnpm run test:snapshot:record`; reviewing the golden diff is part of the review.
- **Snapshot** (`pnpm run test:snapshot`): boots the real example subprocess, replays a recorded session keyless, diffs normalized stdout + the re-persisted log against committed goldens ([snapshot RFC](rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md)). Re-record with `pnpm run test:snapshot:record`; reviewing the golden diff is part of the review. System-prompt/tool-schema content is pinned by ONE scenario (`text-turn`) and tokenized in every other fixture, so a prompt or schema edit churns one committed line ([pinned-header RFC](rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
## The with-key policy: inference is cheap here

View File

@@ -4,7 +4,7 @@ import { fileURLToPath } from 'node:url'
import { dirname, join } from 'node:path'
import { describe, expect, it } from 'vitest'
import { type HarvestedLog, type InputScript, runScenario } from './snapshot-harness.ts'
import { type NormalizeContext, normalizeSessionLog, normalizeStdout } from './snapshot-normalize.ts'
import { type NormalizeContext, normalizeSessionLog, normalizeStdout, scrubRequestHeaders } from './snapshot-normalize.ts'
/**
* ACP snapshot tests (REPLAY by default, keyless). Each scenario under
@@ -16,6 +16,13 @@ import { type NormalizeContext, normalizeSessionLog, normalizeStdout } from './s
* golden: the fixture doubles as the replay source (recorded scenarios) and the
* expected produced log (both sides normalized before comparing).
*
* Request-header content (the composed system prompt + tool schemas riding on
* `request/header` events) is pinned by exactly ONE scenario — the one with
* `pinsHeader` — and scrubbed to `{{system}}`/`{{tools}}` tokens in every
* other fixture and compare, so a prompt or tool-schema edit churns one
* committed line instead of every fixture (see the pinned-header RFC,
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
*
* `pnpm run test:snapshot:record` (DSH_SNAPSHOT=record + -u) re-records the
* `session.jsonl` fixtures against the real API and refreshes the stdout golden
* in one pass.
@@ -55,12 +62,29 @@ interface Scenario {
* harvested child logs back to those files. Defaults to 0.
*/
childSessions?: number
/**
* Whether THIS scenario's fixtures keep the full request-header content (the
* composed system prompt and tool schema list on `request/header` /
* `request/header-delta` events) and compare it verbatim. Exactly one
* scenario pins it; every other scenario stores and compares that content as
* `{{system}}`/`{{tools}}` tokens ({@link scrubRequestHeaders}), so a system
* prompt or tool-schema change shows up as ONE committed-fixture diff, not
* one per scenario. Today every fixture (parent, spawn child, fork child)
* carries the identical prompt-modulo-cwd and identical tools, so one pin
* covers the whole suite; if header composition ever becomes
* session-dependent (say, a restricted subagent toolset), pin one scenario
* per distinct header shape. Defaults to false.
*/
pinsHeader?: boolean
}
const SCENARIOS: Scenario[] = [
{ name: 'handshake', hasModelTurn: false, recorded: false },
{ name: 'reject-extra-dirs', hasModelTurn: false, recorded: false },
{ name: 'text-turn', hasModelTurn: true, recorded: true },
// text-turn is the pinned-header scenario: the minimal single text turn,
// whose fixture is the ONE place the full system prompt + tool schemas are
// committed and compared verbatim.
{ name: 'text-turn', hasModelTurn: true, recorded: true, pinsHeader: true },
{ name: 'tool-call-turn', hasModelTurn: true, recorded: true },
{ name: 'fs-terminal-card', hasModelTurn: true, recorded: true },
{ name: 'todo-plan', hasModelTurn: true, recorded: true },
@@ -181,14 +205,19 @@ for (const scenario of SCENARIOS) {
// RECORD mode (recorded model scenarios only): persist the freshly-harvested
// logs back to their fixtures — the primary to session.jsonl, each child to
// session.<n>.jsonl in harvest order. `--update` refreshes the Vitest
// goldens but NOT these fixtures, so write them here.
// goldens but NOT these fixtures, so write them here. A non-pinning
// scenario's fixtures are written header-scrubbed, so a re-record can
// never smuggle the full prompt/schema content back into every fixture.
const scrub = scenario.pinsHeader === true
? (log: string): string => log
: scrubRequestHeaders
if (RECORDING && scenario.recorded && scenario.hasModelTurn) {
expect(result.sessionLogs.length, 'record produced no session log to harvest').toBeGreaterThan(0)
expect(result.sessionLogs.length, `expected ${childSessions + 1} session logs (parent + children)`)
.toBe(childSessions + 1)
await writeFile(join(dir, 'session.jsonl'), (result.sessionLogs[0] as HarvestedLog).content)
await writeFile(join(dir, 'session.jsonl'), scrub((result.sessionLogs[0] as HarvestedLog).content))
for (let i = 1; i < result.sessionLogs.length; i++) {
await writeFile(join(dir, `session.${i}.jsonl`), (result.sessionLogs[i] as HarvestedLog).content)
await writeFile(join(dir, `session.${i}.jsonl`), scrub((result.sessionLogs[i] as HarvestedLog).content))
}
}
@@ -203,11 +232,17 @@ for (const scenario of SCENARIOS) {
// 1:1. Each side passes through normalizeSessionLog, scrubbed against ITS
// OWN volatile values — the live run's via `ctx`, the committed fixture's
// via its own header (a committed file cannot share the live run's ids).
// Unless this scenario pins the header, both sides ALSO pass through
// scrubRequestHeaders: the live log carries the real prompt/schemas, the
// fixture carries the `{{system}}`/`{{tools}}` tokens, and the scrub is
// idempotent — so the compare checks the header's presence, position,
// reason, and config, but not its bulk content (pinned once, in the
// `pinsHeader` scenario).
expect(result.sessionLogs.length, 'this scenario must persist a session log').toBe(childSessions + 1)
const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)]
for (let i = 0; i < fixtureFiles.length; i++) {
const harvested = (result.sessionLogs[i] as HarvestedLog).content
const fixture = await readFile(join(dir, fixtureFiles[i] as string), 'utf8')
const harvested = scrub((result.sessionLogs[i] as HarvestedLog).content)
const fixture = scrub(await readFile(join(dir, fixtureFiles[i] as string), 'utf8'))
expect(normalizeSessionLog(harvested, ctx), `${fixtureFiles[i]} mismatch`)
.toEqual(normalizeSessionLog(fixture, fixtureContext(fixture)))
}
@@ -253,4 +288,37 @@ describe('snapshot fixtures', () => {
}
}
})
it('exactly one scenario pins the request-header content', () => {
// Zero pins would drop the prompt/schema surface from the suite entirely;
// two would split it. The single pin is the design (pinned-header RFC).
expect(SCENARIOS.filter(s => s.pinsHeader === true).map(s => s.name)).toEqual(['text-turn'])
})
it('committed fixtures carry request-header content ONLY in the pinning scenario', async () => {
// The whole point of the pin: a system-prompt or tool-schema change must
// churn exactly one committed line. A non-pinning fixture that carries the
// full header (a hand-recorded file, or a header line hand-edited out of
// its canonical JSON form) silently reopens the suite-wide churn, so fail
// loud here: every non-pinning session*.jsonl must be a fixed point of
// scrubRequestHeaders (apply the scrub to fix a violation), and the
// pinning scenario's fixtures must NOT be (their content IS the pin).
for (const scenario of SCENARIOS) {
const dir = join(SNAPSHOTS_DIR, scenario.name)
const files = [
'session.jsonl',
...Array.from({ length: scenario.childSessions ?? 0 }, (_, i) => `session.${i + 1}.jsonl`),
]
for (const file of files) {
const fixture = await readFile(join(dir, file), 'utf8')
if (scenario.pinsHeader === true) {
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} must PIN the full header content`)
.not.toEqual(fixture)
} else {
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} carries unscrubbed header content`)
.toEqual(fixture)
}
}
}
})
})

View File

@@ -1,5 +1,5 @@
import { describe, expect, it } from 'vitest'
import { type NormalizeContext, normalizeSessionLog, normalizeStdout } from '../tests/snapshot-normalize.ts'
import { type NormalizeContext, normalizeSessionLog, normalizeStdout, scrubRequestHeaders } from '../tests/snapshot-normalize.ts'
/**
* Unit tests for the pure snapshot normalizers. Live as a *.spec.ts (runs in
@@ -108,3 +108,51 @@ describe('normalizeSessionLog', () => {
expect(out).toContain('"durationMs":88')
})
})
describe('scrubRequestHeaders', () => {
const headerLine = JSON.stringify({ type: 'session', version: 0, id: 's', createdAt: 1, cwd: '/w' })
const headerEvent = (header: object) =>
JSON.stringify({ type: 'request/header', seq: 3, time: 9, data: { header, reason: 'initial' } })
it('replaces header system and tools with tokens, keeping config and reason', () => {
const ev = headerEvent({
config: { model: 'm' },
system: 'You are an agent.\nBe brief.',
tools: [{ name: 'read', description: 'Read a file.', parameters: { type: 'object' } }],
})
const out = scrubRequestHeaders(`${headerLine}\n${ev}\n`)
expect(out).toContain('"system":"{{system}}"')
expect(out).toContain('"tools":"{{tools}}"')
expect(out).toContain('"config":{"model":"m"}')
expect(out).toContain('"reason":"initial"')
expect(out).not.toContain('You are an agent')
expect(out).not.toContain('Read a file')
})
it('keeps an absent system/tools absent (presence is behavior)', () => {
const out = scrubRequestHeaders(`${headerLine}\n${headerEvent({ config: { model: 'm' } })}\n`)
expect(out).not.toContain('{{system}}')
expect(out).not.toContain('{{tools}}')
})
it('scrubs a request/header-delta system/tools payload', () => {
const delta = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: { system: { keepStart: 1, keepEnd: 0, insert: ['leaked prompt line'] }, config: { model: 'm2' } },
})
const out = scrubRequestHeaders(`${headerLine}\n${delta}\n`)
expect(out).toContain('"system":"{{system}}"')
expect(out).toContain('"config":{"model":"m2"}')
expect(out).not.toContain('leaked prompt line')
expect(out).not.toContain('{{tools}}') // no tools delta → none invented
})
it('passes every other line through byte-for-byte and is idempotent', () => {
const other = JSON.stringify({ type: 'assistant/chunk', seq: 4, time: 9, data: { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'hi' } } })
const raw = `${headerLine}\n${headerEvent({ config: { model: 'm' }, system: 's', tools: [] })}\n${other}\n`
const once = scrubRequestHeaders(raw)
expect(once.split('\n')[0]).toBe(headerLine)
expect(once.split('\n')[2]).toBe(other)
expect(scrubRequestHeaders(once)).toBe(once)
})
})

View File

@@ -12,11 +12,21 @@
* `durationMs` (wall-clock hook runtime) → 0. NOT scrubbed: the log's `seq`
* (deterministic — `seq = log.length`, part of the event-log contract).
*
* A separate, composable normalizer — {@link scrubRequestHeaders} — replaces
* the bulky request-header CONTENT (the composed system prompt and the tool
* schema list) with `{{system}}`/`{{tools}}` tokens. It is deliberately NOT
* folded into {@link normalizeSessionLog}: the one header-pinning scenario
* compares that content verbatim, every other scenario composes the scrub in
* (the `pinsHeader` flag in acp.snapshot.ts; see the pinned-header RFC,
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
*
* See docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
*/
const SESSION_ID = '{{sessionId}}'
const CWD = '{{cwd}}'
const SYSTEM = '{{system}}'
const TOOLS = '{{tools}}'
/** A UUID v4 string, the shape `randomUUID()` produces for session ids. */
const UUID_RE = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi
@@ -110,3 +120,46 @@ export function normalizeSessionLog(rawLog: string, ctx: NormalizeContext): stri
})
return records.map(r => JSON.stringify(r)).join('\n') + '\n'
}
/**
* Replace request-header CONTENT in a session JSONL with stable tokens: a
* `request/header` event's `data.header.system` → `{{system}}` and
* `data.header.tools` → `{{tools}}`, and a `request/header-delta` event's
* `data.system`/`data.tools` payloads likewise (a delta embeds prompt/schema
* fragments, so leaving it raw would reopen the churn this scrub closes).
* Absent fields stay absent — WHETHER a header carried a system prompt or
* tools is behavior and stays visible; `config` and `reason` are small and
* stable, so they stay verbatim (a model swap SHOULD churn every fixture — it
* invalidates the recorded responses; a prompt/schema edit does NOT — replay
* never reads this content, see dsh-llm-replay).
*
* Only lines with something to scrub are re-serialized; every other line
* passes through byte-for-byte, so the transform is idempotent and applying
* it to an already-scrubbed fixture is a no-op — the on-disk-fixtures guard
* in acp.snapshot.ts relies on exactly that.
*/
export function scrubRequestHeaders(rawLog: string): string {
const lines = rawLog.split('\n')
const out = lines.map((line) => {
if (line.trim().length === 0) return line
const record = JSON.parse(line) as Record<string, unknown>
const data = record.data as Record<string, unknown> | null | undefined
if (data === null || typeof data !== 'object') return line
if (record.type === 'request/header') {
const header = data.header as Record<string, unknown> | null | undefined
if (header === null || typeof header !== 'object') return line
if (!('system' in header) && !('tools' in header)) return line
if ('system' in header) header.system = SYSTEM
if ('tools' in header) header.tools = TOOLS
return JSON.stringify(record)
}
if (record.type === 'request/header-delta') {
if (!('system' in data) && !('tools' in data)) return line
if ('system' in data) data.system = SYSTEM
if ('tools' in data) data.tools = TOOLS
return JSON.stringify(record)
}
return line
})
return out.join('\n')
}

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@@ -6,7 +6,7 @@ Its consumer is the ACP snapshot harness in `examples/acp-agent`, which loads th
## How the fixture works
The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`", done by the snapshot harness — this plugin does not record.
The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`", done by the snapshot harness — this plugin does not record. A fixture may carry its `request/header` content tokenized to `{{system}}`/`{{tools}}` (the harness pins that content in one scenario and scrubs the rest); replay is indifferent — derivation reads only `assistant/chunk` events and the line-0 session header.
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`: a `ReplayEntry[]`) that REPLACES the derived script.

View File

@@ -12,7 +12,11 @@
* `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model
* call per loop step — see packages/core/agent-loop/src/loop.ts). Recording is
* therefore "run the real agent once and harvest the `.jsonl`", done by the
* snapshot harness — this plugin does not record.
* snapshot harness — this plugin does not record. A fixture may carry its
* `request/header` content tokenized to `{{system}}`/`{{tools}}` (the harness
* pins that content in one scenario and scrubs the rest); replay is
* indifferent — derivation reads ONLY `assistant/chunk` events and the line-0
* session header.
*
* A NESTED-agent scenario records more than one log: the parent plus one per
* in-process subagent (each subagent runs as its own {@link Session} on the same