Merge remote-tracking branch 'origin/master' into codex/skill-system
This commit is contained in:
Yichen Jiang
@@ -4,8 +4,9 @@ Packages that exist to serve development, testing, and the examples rather than
|
||||
|
||||
| Package | Role | ctx key |
|
||||
|---|---|---|
|
||||
| `acp-snapshot/` | ACP snapshot suite kit: subprocess scenario harness + golden normalizers + the `defineAcpSnapshotSuite` factory | (library — imported by example `*.snapshot.ts` suites) |
|
||||
| `invariants/` | Dev-mode event-contract invariants + session-log freeze | (listens on `session/*`, `agent/*`) |
|
||||
| `llm-replay/` | Record/replay adapter: short-circuits `llm/stream` from a recorded session JSONL (keyless snapshot tests) | (listens on `llm/stream`) |
|
||||
| `subagent-mock/` | Scripted `SubagentProvider` for deterministic seam/tool tests | (registers on `ctx.subagents`) |
|
||||
|
||||
`invariants` runs only in dev mode (contract checks, not runtime behavior). `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate. `subagent-mock` exercises the real `ctx.subagents` load path without a model or child agent. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.
|
||||
`invariants` runs only in dev mode (contract checks, not runtime behavior). `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate. `acp-snapshot` carries the snapshot tier's harness/normalizer/suite machinery so every example's suite is a scenario table over one shared, gate-covered implementation. `subagent-mock` exercises the real `ctx.subagents` load path without a model or child agent. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.
|
||||
|
||||
36
packages/support/acp-snapshot/README.md
Normal file
36
packages/support/acp-snapshot/README.md
Normal file
@@ -0,0 +1,36 @@
|
||||
# `@deepseek-ai/dsh-acp-snapshot`
|
||||
|
||||
The ACP snapshot suite kit: the shared machinery behind the keyless snapshot tier (`pnpm run test:snapshot`, [testing policy](../../../docs/testing.md)). An example gets a full snapshot suite from a scenario table plus a fixtures directory; every compare/guard mechanic lives here, under the per-file coverage gate, instead of being copied per example.
|
||||
|
||||
Three layers, importable separately:
|
||||
|
||||
- **`runScenario` (harness)** — boots the real agent bin as a subprocess via tsx (unbuilt, Loader path), drives it over ACP JSON-RPC stdio from a deterministic `input.json` script, tees raw stdout for the golden + purity check, and harvests every persisted session JSONL (parent + subagent children, primary-first) after a graceful stdin-EOF shutdown. Parameterized by `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath` — absolute paths; the subprocess cwd is a temp dir outside the repo).
|
||||
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept), and the composable `scrubRequestHeaders` (header bulk → `{{system}}`/`{{tools}}`, structure kept — [pinned-header RFC](../../../docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
|
||||
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record-mode fixture write-back, the per-suite header pin with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin, non-pinning fixtures header-scrubbed). Must be called at vitest collection time.
|
||||
|
||||
A consuming `*.snapshot.ts` is the scenario table plus one factory call:
|
||||
|
||||
```ts
|
||||
import { dirname, join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { defineAcpSnapshotSuite, type Scenario } from '@deepseek-ai/dsh-acp-snapshot'
|
||||
|
||||
const SCENARIOS: Scenario[] = [
|
||||
{ name: 'text-turn', hasModelTurn: true, recorded: true, pinsHeader: true },
|
||||
]
|
||||
|
||||
defineAcpSnapshotSuite({
|
||||
agent: { // absolute paths, resolved from the suite's own location
|
||||
binScript: fileURLToPath(new URL('../../../packages/ui/acp-agent/src/bin.ts', import.meta.url)),
|
||||
configPath: fileURLToPath(new URL('../cordis.yml', import.meta.url)),
|
||||
tsconfigPath: fileURLToPath(new URL('../../../tsconfig.json', import.meta.url)),
|
||||
},
|
||||
snapshotsDir: join(dirname(fileURLToPath(import.meta.url)), 'snapshots'),
|
||||
scenarios: SCENARIOS, // exactly one entry sets pinsHeader
|
||||
mode: process.env.DSH_SNAPSHOT === 'record' ? 'record' : 'replay',
|
||||
})
|
||||
```
|
||||
|
||||
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config RFC](../../../docs/rfc/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. Fixture roles, record/replay semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
|
||||
|
||||
Constraints: `suite.ts` imports vitest, so the package is importable only inside a vitest run (the harness and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the harness speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug).
|
||||
35
packages/support/acp-snapshot/package.json
Normal file
35
packages/support/acp-snapshot/package.json
Normal file
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-acp-snapshot",
|
||||
"description": "ACP snapshot suite kit: real-subprocess scenario harness, golden normalizers, and the suite factory behind the keyless snapshot tier",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/types/**/*.d.ts",
|
||||
"lib/types/**/*.d.ts.map",
|
||||
"src"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"@agentclientprotocol/sdk": "0.25.1",
|
||||
"tsx": "^4.22.4",
|
||||
"vitest": "^4.1.8"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
},
|
||||
"devDependencies": {
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
}
|
||||
}
|
||||
471
packages/support/acp-snapshot/src/harness.ts
Normal file
471
packages/support/acp-snapshot/src/harness.ts
Normal file
@@ -0,0 +1,471 @@
|
||||
/**
|
||||
* Shared subprocess harness for ACP snapshot suites. A library module driven by
|
||||
* the suite factory in ./suite.ts (and directly by harness-level specs); each
|
||||
* example's `*.snapshot.ts` names its own agent-under-test paths.
|
||||
*
|
||||
* It boots the REAL agent bin subprocess via the cordis Loader (so the
|
||||
* export-shape bug class stays guarded — see docs/postmortem/0001), drives it
|
||||
* over real ACP JSON-RPC stdio with a deterministic input script, tees raw
|
||||
* stdout (for the golden + a purity check) into an SDK `ClientSideConnection`,
|
||||
* and — in record mode — harvests the persisted session JSONL after a graceful
|
||||
* shutdown flush. The pure normalizers in ./normalize.ts turn the captured
|
||||
* stdout frames and the session-log events into stable, snapshot-able text.
|
||||
*
|
||||
* See docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-acp-snapshot/harness
|
||||
*/
|
||||
|
||||
import { spawn, type ChildProcessWithoutNullStreams } from 'node:child_process'
|
||||
import { cp, mkdtemp, readFile, readdir, rm } from 'node:fs/promises'
|
||||
import { existsSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join, delimiter } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { Readable, Writable } from 'node:stream'
|
||||
import {
|
||||
ClientSideConnection,
|
||||
ndJsonStream,
|
||||
PROTOCOL_VERSION,
|
||||
type Agent as AcpAgent,
|
||||
type Client,
|
||||
type RequestPermissionRequest,
|
||||
type RequestPermissionResponse,
|
||||
type SessionNotification,
|
||||
} from '@agentclientprotocol/sdk'
|
||||
|
||||
// Resolve tsx's ESM loader to an ABSOLUTE path once: the child runs with its
|
||||
// cwd in a temp dir OUTSIDE the repo, where a bare `--import tsx` would not
|
||||
// resolve from node_modules. import.meta.resolve gives this package's tsx
|
||||
// regardless of the child cwd.
|
||||
const tsxLoader = fileURLToPath(import.meta.resolve('tsx'))
|
||||
|
||||
/**
|
||||
* The agent composition a scenario runs against: which bin to boot and which
|
||||
* leaf config it loads. All paths are ABSOLUTE — the subprocess cwd is a temp
|
||||
* dir outside the repo, so relative resolution would miss; a suite resolves
|
||||
* them from its own `import.meta.url`.
|
||||
*/
|
||||
export interface AgentUnderTest {
|
||||
/** The agent bin entry (e.g. `packages/ui/acp-agent/src/bin.ts`), run unbuilt via tsx. */
|
||||
binScript: string
|
||||
/**
|
||||
* The example's live `cordis.yml`. Under `DSH_SNAPSHOT=replay` the bin swaps
|
||||
* it for the sibling `cordis.snapshot.yml` (the keyless replay overlay), so
|
||||
* one path serves both modes.
|
||||
*/
|
||||
configPath: string
|
||||
/**
|
||||
* The repo-root tsconfig whose `paths` map resolves the unbuilt workspace
|
||||
* imports. Passed to the child as `TSX_TSCONFIG_PATH`: tsx finds a tsconfig
|
||||
* by searching UP from the child's cwd — a temp dir outside the repo — so
|
||||
* without the explicit pin the dsh-* imports fail before the bin writes a
|
||||
* byte.
|
||||
*/
|
||||
tsconfigPath: string
|
||||
}
|
||||
|
||||
/**
|
||||
* One step of a scenario's deterministic input script (`input.json`). The
|
||||
* harness interprets these in order. `newSession` captures the server-issued
|
||||
* (random) session id into a `{{sessionId}}` variable that later steps
|
||||
* reference, since a committed file cannot know the id in advance.
|
||||
*
|
||||
* `promptAndCancel` sends a prompt WITHOUT awaiting its response, waits until
|
||||
* the client observes the first streamed `agent_message_chunk` (so the emitted
|
||||
* frames deterministically precede the cancellation), then cancels the turn —
|
||||
* the only way to exercise a cancel deterministically (a plain `prompt` step
|
||||
* awaits the response, which a cancel/hang scenario would block on forever).
|
||||
*/
|
||||
export type InputStep =
|
||||
| { op: 'initialize'; terminalOutput?: boolean }
|
||||
| { op: 'newSession' }
|
||||
| { op: 'newSessionExpectError'; additionalDirectories?: string[] }
|
||||
| { op: 'prompt'; text: string }
|
||||
| { op: 'promptExpectError'; text: string }
|
||||
| { op: 'promptAndCancel'; text: string }
|
||||
| { op: 'cancel' }
|
||||
|
||||
/** A scenario's `input.json`: an ordered list of input steps. */
|
||||
export interface InputScript {
|
||||
steps: InputStep[]
|
||||
/**
|
||||
* Ordered answers for the agent's `session/request_permission` round-trips,
|
||||
* consumed FIFO — the Nth request gets the Nth answer. Each answer selects
|
||||
* by option KIND: option ids are agent-issued randoms a committed script
|
||||
* cannot know, while kinds are the ACP-stable vocabulary, so the client maps
|
||||
* kind → the offered `optionId` at answer time. A request beyond the queue
|
||||
* (or with no queue at all) is answered `cancelled` — the stub behavior a
|
||||
* scenario without approvals relies on. A scripted kind the request does
|
||||
* not offer REJECTS the run: the scenario scripted an impossible click,
|
||||
* and {@link runScenario} throws once the in-flight step settles (the
|
||||
* agent itself just sees `cancelled`, so it cannot absorb the bug).
|
||||
*/
|
||||
permissionAnswers?: PermissionAnswer[]
|
||||
}
|
||||
|
||||
/** One scripted answer to a permission request: which offered option kind to select. */
|
||||
export interface PermissionAnswer {
|
||||
/** The `PermissionOption.kind` to select (`allow_once`, `reject_always`, …). */
|
||||
kind: 'allow_once' | 'allow_always' | 'reject_once' | 'reject_always'
|
||||
}
|
||||
|
||||
/** One harvested session log plus the identifying facts off its header line. */
|
||||
export interface HarvestedLog {
|
||||
/** The recorded session id (header `id`). */
|
||||
id: string
|
||||
/** Session creation time (header `createdAt`) — the child-ordering key. */
|
||||
createdAt: number
|
||||
/** The parent session id, if this log is a subagent child (header `parentSession`). */
|
||||
parentSession?: string
|
||||
/** The full `.jsonl` file content. */
|
||||
content: string
|
||||
}
|
||||
|
||||
/** The result of running a scenario: raw stdout + the harvested session log(s). */
|
||||
export interface RunResult {
|
||||
/** Raw stdout bytes (decoded utf8), every newline-delimited JSON-RPC frame. */
|
||||
rawStdout: string
|
||||
/** stderr (for diagnostics on failure). */
|
||||
stderr: string
|
||||
/** The session id the server issued (undefined if no session was created). */
|
||||
sessionId?: string
|
||||
/** The temp cwd the session ran in (the bash workspace). */
|
||||
cwd: string
|
||||
/**
|
||||
* Every persisted session log harvested after the run, ordered primary-first:
|
||||
* the top-level (parent) session — the one with no `parentSession` — then each
|
||||
* subagent child by ascending `createdAt`. A single-session scenario harvests
|
||||
* exactly one; a nested-agent scenario harvests the parent plus one per child.
|
||||
*/
|
||||
sessionLogs: HarvestedLog[]
|
||||
}
|
||||
|
||||
/** How to run one scenario: the agent to boot, the mode, and the fixture wiring. */
|
||||
export interface RunOptions {
|
||||
/** The agent composition to boot. */
|
||||
agent: AgentUnderTest
|
||||
/** `replay` (default, keyless) or `record` (real API, harvests the log). */
|
||||
mode: 'replay' | 'record'
|
||||
/** The recorded session JSONL fixture path (replay reads it; record writes near it). */
|
||||
fixtureFile: string
|
||||
/** Optional sidecar override path (replay). */
|
||||
overrideFile?: string
|
||||
/**
|
||||
* Recorded SUBAGENT child-session fixture paths (replay). A nested-agent
|
||||
* scenario ships one per child (`session.1.jsonl`, …); the harness forwards
|
||||
* them to `dsh-llm-replay` via `$DSH_SNAPSHOT_CHILD_FILES` so each child
|
||||
* session replays from its own recorded script. Empty for single-session
|
||||
* scenarios. Ignored in record mode (children are harvested, not replayed).
|
||||
*/
|
||||
childFiles?: string[]
|
||||
/**
|
||||
* Optional `<scenario>/workspace/` directory whose contents are copied into
|
||||
* the temp cwd BEFORE the run — the standard way to seed files the agent
|
||||
* operates on (a file to read, edit, or grep). Absent for scenarios that
|
||||
* start from an empty workspace.
|
||||
*/
|
||||
workspaceDir?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Run a scenario end-to-end against a freshly-spawned subprocess. Owns the
|
||||
* child and its temp dirs; always tears them down. Returns the captured stdout
|
||||
* and (record mode) the harvested session-log path.
|
||||
*
|
||||
* @param input The scenario's input script (steps + optional permission answers).
|
||||
* @param opts The agent to boot, the mode, and the fixture wiring.
|
||||
* @returns The captured stdout/stderr, session id, temp cwd, and harvested logs.
|
||||
*/
|
||||
export async function runScenario(input: InputScript, opts: RunOptions): Promise<RunResult> {
|
||||
const cwd = await mkdtemp(join(tmpdir(), 'acp-snap-cwd-'))
|
||||
const sessionsRoot = await mkdtemp(join(tmpdir(), 'acp-snap-sessions-'))
|
||||
// Everything past the temp-dir creation runs under a try/finally that always
|
||||
// removes both dirs — so a failure in workspace seeding, spawn, or any step
|
||||
// never leaks them (the "e2e tests own their resources" rule).
|
||||
let child: ChildProcessWithoutNullStreams | undefined
|
||||
let sessionId: string | undefined
|
||||
let sessionLogs: HarvestedLog[] = []
|
||||
const rawBuffers: Buffer[] = []
|
||||
const stderrChunks: string[] = []
|
||||
try {
|
||||
// Seed the workspace if the scenario ships one (a file the agent reads/edits).
|
||||
// Copied into the temp cwd so the agent's bash tools see it; the goldens
|
||||
// normalize the cwd, so the seeded paths stay stable across runs.
|
||||
if (opts.workspaceDir !== undefined && existsSync(opts.workspaceDir)) {
|
||||
await cp(opts.workspaceDir, cwd, { recursive: true })
|
||||
}
|
||||
const env: NodeJS.ProcessEnv = {
|
||||
...process.env,
|
||||
TSX_TSCONFIG_PATH: opts.agent.tsconfigPath,
|
||||
DSH_SNAPSHOT: opts.mode,
|
||||
DSH_SNAPSHOT_FILE: opts.fixtureFile,
|
||||
DSH_SNAPSHOT_SESSIONS_ROOT: sessionsRoot,
|
||||
DSH_HOME: join(cwd, '.dsh'),
|
||||
DSH_AGENTS_HOME: join(cwd, '.agents'),
|
||||
...opts.overrideFile !== undefined ? { DSH_SNAPSHOT_OVERRIDE: opts.overrideFile } : {},
|
||||
...opts.childFiles !== undefined && opts.childFiles.length > 0
|
||||
? { DSH_SNAPSHOT_CHILD_FILES: opts.childFiles.join(delimiter) }
|
||||
: {},
|
||||
}
|
||||
|
||||
child = spawn(
|
||||
process.execPath,
|
||||
['--import', tsxLoader, opts.agent.binScript, opts.agent.configPath],
|
||||
{ cwd, env, stdio: ['pipe', 'pipe', 'pipe'] },
|
||||
)
|
||||
|
||||
child.stderr.setEncoding('utf8')
|
||||
child.stderr.on('data', (c: string) => stderrChunks.push(c))
|
||||
|
||||
// Tee raw stdout: accumulate the bytes for the golden + purity check, and ALSO
|
||||
// feed the same bytes to the SDK client through a passthrough. Buffer the raw
|
||||
// bytes (not per-chunk utf8 strings) and decode once at the end, so a
|
||||
// multibyte sequence split across two 'data' events can't corrupt the golden.
|
||||
const passthrough = new Readable({ read() {} })
|
||||
child.stdout.on('data', (buf: Buffer) => {
|
||||
rawBuffers.push(buf)
|
||||
passthrough.push(buf)
|
||||
})
|
||||
child.stdout.on('end', () => passthrough.push(null))
|
||||
|
||||
const stream = ndJsonStream(
|
||||
Writable.toWeb(child.stdin) as WritableStream<Uint8Array>,
|
||||
Readable.toWeb(passthrough) as ReadableStream<Uint8Array>,
|
||||
)
|
||||
// Watcher so a step can block until the client OBSERVES a particular
|
||||
// session/update — used by promptAndCancel to pin frame order (send cancel
|
||||
// only after the streamed agent_message_chunk has arrived, so those frames
|
||||
// deterministically precede the cancelled prompt response).
|
||||
const updateWaiters: { match: (u: SessionNotification['update']) => boolean; resolve: () => void }[] = []
|
||||
const waitForUpdate = (match: (u: SessionNotification['update']) => boolean): Promise<void> =>
|
||||
new Promise<void>(resolve => updateWaiters.push({ match, resolve }))
|
||||
|
||||
// Permission answers are consumed FIFO across the whole run; exhaustion
|
||||
// falls back to `cancelled` so approval-free scenarios keep the plain stub.
|
||||
const permissionQueue = [...input.permissionAnswers ?? []]
|
||||
// A scenario bug detected inside a client callback (a scripted permission
|
||||
// kind the agent never offered). It cannot fail the run from in there: a
|
||||
// callback throw only becomes a JSON-RPC error RESPONSE to the agent, and
|
||||
// a tolerant agent treats that as a denial and carries on — the run (or
|
||||
// worse, a record) would absorb the impossible click silently. So the
|
||||
// callback answers `cancelled` (a well-defined path for the agent),
|
||||
// captures the error here, and the step loop fails the run on it.
|
||||
let scriptError: Error | undefined
|
||||
const makeClient = (_agent: AcpAgent): Client => ({
|
||||
sessionUpdate(params: SessionNotification): Promise<void> {
|
||||
for (let i = updateWaiters.length - 1; i >= 0; i--) {
|
||||
const waiter = updateWaiters[i]
|
||||
// The index is always in-bounds (i only decreases; splice removes at
|
||||
// i, so lower entries stay valid); the guard satisfies
|
||||
// noUncheckedIndexedAccess.
|
||||
/* v8 ignore next 1 -- unreachable in-bounds guard, see above */
|
||||
if (waiter === undefined) continue
|
||||
if (waiter.match(params.update)) {
|
||||
updateWaiters.splice(i, 1)
|
||||
waiter.resolve()
|
||||
}
|
||||
}
|
||||
return Promise.resolve()
|
||||
},
|
||||
requestPermission(params: RequestPermissionRequest): Promise<RequestPermissionResponse> {
|
||||
const answer = permissionQueue.shift()
|
||||
if (answer === undefined) return Promise.resolve({ outcome: { outcome: 'cancelled' } })
|
||||
const option = params.options.find(o => o.kind === answer.kind)
|
||||
if (option === undefined) {
|
||||
// The scenario scripted a click the agent never offered — a scenario
|
||||
// bug. Captured (last one wins; same bug class either way) and
|
||||
// answered `cancelled`; the step loop rejects the run on it.
|
||||
scriptError = new Error(
|
||||
`snapshot-harness: scripted permission answer ${answer.kind} not among `
|
||||
+ `the offered options [${params.options.map(o => o.kind).join(', ')}]`,
|
||||
)
|
||||
return Promise.resolve({ outcome: { outcome: 'cancelled' } })
|
||||
}
|
||||
return Promise.resolve({ outcome: { outcome: 'selected', optionId: option.optionId } })
|
||||
},
|
||||
})
|
||||
const client = new ClientSideConnection(makeClient, stream)
|
||||
|
||||
for (const step of input.steps) {
|
||||
await runStep(client, step, cwd, waitForUpdate, () => sessionId, (id) => { sessionId = id })
|
||||
// A permission exchange happens while a step's request is in flight, so
|
||||
// by the time the step settles any script bug it exposed is captured —
|
||||
// fail the run HERE, as a harness error, rather than hoping the agent's
|
||||
// reaction to the answer perturbs the transcript.
|
||||
if (scriptError !== undefined) throw scriptError
|
||||
}
|
||||
// Done driving: close stdin so the server disposes gracefully (flushing
|
||||
// persistence) and exits. Then await exit so the harvested log is complete.
|
||||
child.stdin.end()
|
||||
await waitForExit(child)
|
||||
// Harvest EVERY persisted log (parent + any subagent children) while the
|
||||
// temp dirs still exist, ordered primary-first.
|
||||
sessionLogs = await harvestSessionLogs(sessionsRoot)
|
||||
} finally {
|
||||
// Failure-safe teardown: kill a still-running child and drop the temp dirs
|
||||
// even if seeding/spawn/a step/harvest threw, so a flaky run never leaks a
|
||||
// process or dir. `child` is undefined only if spawn itself threw.
|
||||
if (child !== undefined && child.exitCode === null && child.signalCode === null) {
|
||||
child.kill('SIGKILL')
|
||||
await waitForExit(child)
|
||||
}
|
||||
await rm(cwd, { recursive: true, force: true })
|
||||
await rm(sessionsRoot, { recursive: true, force: true })
|
||||
}
|
||||
|
||||
return {
|
||||
rawStdout: Buffer.concat(rawBuffers).toString('utf8'),
|
||||
stderr: stderrChunks.join(''),
|
||||
cwd,
|
||||
...sessionId !== undefined ? { sessionId } : {},
|
||||
sessionLogs,
|
||||
}
|
||||
}
|
||||
|
||||
/** Drive one input step over the client connection. */
|
||||
async function runStep(
|
||||
client: ClientSideConnection,
|
||||
step: InputStep,
|
||||
cwd: string,
|
||||
waitForUpdate: (match: (u: SessionNotification['update']) => boolean) => Promise<void>,
|
||||
getSessionId: () => string | undefined,
|
||||
setSessionId: (id: string) => void,
|
||||
): Promise<void> {
|
||||
switch (step.op) {
|
||||
case 'initialize':
|
||||
await client.initialize({
|
||||
protocolVersion: PROTOCOL_VERSION,
|
||||
clientCapabilities: step.terminalOutput === true ? { _meta: { terminal_output: true } } : {},
|
||||
})
|
||||
return
|
||||
case 'newSession': {
|
||||
const { sessionId } = await client.newSession({ cwd, mcpServers: [] })
|
||||
setSessionId(sessionId)
|
||||
return
|
||||
}
|
||||
case 'newSessionExpectError': {
|
||||
// The bridge rejects a session/new that widens the workspace scope
|
||||
// (non-empty additionalDirectories / mcpServers — unimplemented). The SDK
|
||||
// surfaces that as a rejected RPC; swallow it so the run completes and the
|
||||
// error frame is captured in the transcript.
|
||||
await client.newSession({
|
||||
cwd,
|
||||
mcpServers: [],
|
||||
...step.additionalDirectories !== undefined ? { additionalDirectories: step.additionalDirectories } : {},
|
||||
}).then(
|
||||
() => { throw new Error('snapshot-harness: expected session/new to be rejected but it succeeded') },
|
||||
() => { /* expected: the bridge rejected the unsupported workspace scope */ },
|
||||
)
|
||||
return
|
||||
}
|
||||
case 'prompt': {
|
||||
const sessionId = getSessionId()
|
||||
if (sessionId === undefined) throw new Error('snapshot-harness: prompt before newSession')
|
||||
await client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
|
||||
return
|
||||
}
|
||||
case 'promptExpectError': {
|
||||
const sessionId = getSessionId()
|
||||
if (sessionId === undefined) throw new Error('snapshot-harness: promptExpectError before newSession')
|
||||
// The model fails this turn (a recorded provider error), so the bridge
|
||||
// answers the prompt with a JSON-RPC error and the SDK rejects. That
|
||||
// rejection IS the expected editor experience — swallow it so the run
|
||||
// completes and the stdout transcript (the error frame) is captured.
|
||||
await client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
|
||||
.then(() => { throw new Error('snapshot-harness: expected the prompt to fail but it succeeded') },
|
||||
() => { /* expected: the turn failed and the bridge returned an error */ })
|
||||
return
|
||||
}
|
||||
case 'promptAndCancel': {
|
||||
const sessionId = getSessionId()
|
||||
if (sessionId === undefined) throw new Error('snapshot-harness: promptAndCancel before newSession')
|
||||
// Dispatch the prompt WITHOUT awaiting (a hang fixture never resolves on
|
||||
// its own). To pin frame order deterministically, wait until the client
|
||||
// has OBSERVED the hang's streamed agent_message_chunk before cancelling —
|
||||
// so those update frames always precede the cancelled prompt response in
|
||||
// the transcript (without this, the late chunk and the response race).
|
||||
// Then cancel and await the prompt, which the bridge settles as
|
||||
// `cancelled` once the abort propagates.
|
||||
const promptDone = client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
|
||||
await waitForUpdate(u => u.sessionUpdate === 'agent_message_chunk')
|
||||
await client.cancel({ sessionId })
|
||||
await promptDone
|
||||
return
|
||||
}
|
||||
case 'cancel': {
|
||||
const sessionId = getSessionId()
|
||||
if (sessionId === undefined) throw new Error('snapshot-harness: cancel before newSession')
|
||||
await client.cancel({ sessionId })
|
||||
return
|
||||
}
|
||||
default:
|
||||
throw new Error(`snapshot-harness: unknown input op ${JSON.stringify(step)}`)
|
||||
}
|
||||
}
|
||||
|
||||
/** Resolve once the child process exits (any code/signal). */
|
||||
function waitForExit(child: ChildProcessWithoutNullStreams): Promise<void> {
|
||||
// Race guard: both call sites run within one synchronous frame of
|
||||
// stdin.end()/kill(), so the exit event cannot have been delivered yet;
|
||||
// kept for any future caller that awaits in between.
|
||||
/* v8 ignore next 1 -- unreachable race guard, see above */
|
||||
if (child.exitCode !== null || child.signalCode !== null) return Promise.resolve()
|
||||
return new Promise<void>(resolve => child.once('exit', () => { resolve() }))
|
||||
}
|
||||
|
||||
/**
|
||||
* Harvest EVERY persisted `.jsonl` session log under a sessions root, parse each
|
||||
* header line, and return them ordered primary-first: the top-level session (no
|
||||
* `parentSession`) leads, then each subagent child by ascending `createdAt`.
|
||||
*
|
||||
* The JSONL backend lays sessions out as `<root>/<cwd-bucket>/<encoded-id>.jsonl`
|
||||
* (one bucket per cwd), so a parent and its same-cwd in-process child land in
|
||||
* the SAME bucket — collecting all files across all buckets catches both (a
|
||||
* first-match short-circuit would silently drop the child). Returns `[]` if no
|
||||
* log was produced (a no-session scenario).
|
||||
*/
|
||||
async function harvestSessionLogs(root: string): Promise<HarvestedLog[]> {
|
||||
let cwdDirs: string[]
|
||||
try {
|
||||
cwdDirs = await readdir(root)
|
||||
} catch {
|
||||
return []
|
||||
}
|
||||
const logs: HarvestedLog[] = []
|
||||
for (const dir of cwdDirs) {
|
||||
const sub = join(root, dir)
|
||||
let files: string[]
|
||||
try {
|
||||
files = await readdir(sub)
|
||||
} catch {
|
||||
continue
|
||||
}
|
||||
for (const f of files) {
|
||||
if (!f.endsWith('.jsonl')) continue
|
||||
const content = await readFile(join(sub, f), 'utf8')
|
||||
const firstLine = content.split('\n').find(line => line.trim().length > 0) ?? '{}'
|
||||
const header = JSON.parse(firstLine) as { id?: unknown; createdAt?: unknown; parentSession?: unknown }
|
||||
logs.push({
|
||||
id: typeof header.id === 'string' ? header.id : '',
|
||||
createdAt: typeof header.createdAt === 'number' ? header.createdAt : 0,
|
||||
...typeof header.parentSession === 'string' ? { parentSession: header.parentSession } : {},
|
||||
content,
|
||||
})
|
||||
}
|
||||
}
|
||||
// Primary (no parentSession) first, then children by ascending createdAt. A
|
||||
// scenario has exactly one top-level session. In the synchronous cut sibling
|
||||
// children are created strictly sequentially, so their createdAt values are
|
||||
// strictly ordered; the recordedId tiebreak only keeps a degenerate
|
||||
// same-millisecond collision (unreachable here) deterministic. This harvest
|
||||
// order must match the replay load order in dsh-llm-replay's loadSessionScripts
|
||||
// so session.<n>.jsonl maps to the same child on record and replay — replay
|
||||
// re-sorts childFiles by the same key, so the two stay consistent.
|
||||
logs.sort((a, b) => {
|
||||
const ap = a.parentSession === undefined ? 0 : 1
|
||||
const bp = b.parentSession === undefined ? 0 : 1
|
||||
return ap - bp || a.createdAt - b.createdAt || a.id.localeCompare(b.id)
|
||||
})
|
||||
return logs
|
||||
}
|
||||
38
packages/support/acp-snapshot/src/index.ts
Normal file
38
packages/support/acp-snapshot/src/index.ts
Normal file
@@ -0,0 +1,38 @@
|
||||
/**
|
||||
* ACP snapshot suite kit — the shared machinery behind the keyless snapshot
|
||||
* tier (`pnpm run test:snapshot`). Three layers, composable per example:
|
||||
* the subprocess scenario harness ({@link runScenario}), the pure golden
|
||||
* normalizers ({@link normalizeStdout} / {@link normalizeSessionLog} /
|
||||
* {@link scrubRequestHeaders}), and the suite factory
|
||||
* ({@link defineAcpSnapshotSuite}) that registers a scenario table as a full
|
||||
* describe/it tree. An example's `*.snapshot.ts` supplies only its
|
||||
* {@link AgentUnderTest} paths, its snapshots directory, and its
|
||||
* {@link Scenario} table.
|
||||
*
|
||||
* NOTE: ./suite.ts imports vitest, so this package is importable only inside a
|
||||
* vitest run — a support-tier constraint stated in the README.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-acp-snapshot
|
||||
*/
|
||||
|
||||
export {
|
||||
runScenario,
|
||||
type AgentUnderTest,
|
||||
type HarvestedLog,
|
||||
type InputScript,
|
||||
type InputStep,
|
||||
type PermissionAnswer,
|
||||
type RunOptions,
|
||||
type RunResult,
|
||||
} from './harness.ts'
|
||||
export {
|
||||
normalizeSessionLog,
|
||||
normalizeStdout,
|
||||
scrubRequestHeaders,
|
||||
type NormalizeContext,
|
||||
} from './normalize.ts'
|
||||
export {
|
||||
defineAcpSnapshotSuite,
|
||||
type Scenario,
|
||||
type SnapshotSuiteOptions,
|
||||
} from './suite.ts'
|
||||
213
packages/support/acp-snapshot/src/normalize.ts
Normal file
213
packages/support/acp-snapshot/src/normalize.ts
Normal file
@@ -0,0 +1,213 @@
|
||||
/**
|
||||
* Pure normalizers for the ACP snapshot goldens. They replace the
|
||||
* non-deterministic values in the two captured surfaces — the stdout JSON-RPC
|
||||
* transcript and the persisted session JSONL — with stable tokens, so a golden
|
||||
* compare reflects behavior, not run-to-run noise. Kept dependency-free and
|
||||
* side-effect-free so they unit-test trivially.
|
||||
*
|
||||
* Scrubbed: `randomUUID()` session ids → `{{sessionId}}`; the temp `mkdtemp`
|
||||
* cwd → `{{cwd}}` (it appears in terminal-card `_meta` and the log header);
|
||||
* JSON-RPC request `id` → a stable per-transcript sequence; the log's per-event
|
||||
* `time` (epoch ms) and header `createdAt` → 0; a `hook/result` event's
|
||||
* `durationMs` (wall-clock hook runtime) → 0. NOT scrubbed: the log's `seq`
|
||||
* (deterministic — `seq = log.length`, part of the event-log contract).
|
||||
*
|
||||
* A separate, composable normalizer — {@link scrubRequestHeaders} — replaces
|
||||
* the bulky request-header CONTENT (the composed system prompt, the tool
|
||||
* schema list, and the session prefix) with
|
||||
* `{{system}}`/`{{tools}}`/`{{messagePrefix}}` tokens. It is deliberately NOT
|
||||
* folded into {@link normalizeSessionLog}: each suite's one header-pinning
|
||||
* scenario compares that content verbatim, every other scenario composes the
|
||||
* scrub in (the `pinsHeader` flag on the scenario table, consumed by the suite
|
||||
* factory in ./suite.ts; see the pinned-header RFC,
|
||||
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
|
||||
*
|
||||
* See docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-acp-snapshot/normalize
|
||||
*/
|
||||
|
||||
const SESSION_ID = '{{sessionId}}'
|
||||
const CWD = '{{cwd}}'
|
||||
const SYSTEM = '{{system}}'
|
||||
const TOOLS = '{{tools}}'
|
||||
const MESSAGE_PREFIX = '{{messagePrefix}}'
|
||||
|
||||
/** A UUID v4 string, the shape `randomUUID()` produces for session ids. */
|
||||
const UUID_RE = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi
|
||||
|
||||
/** Inputs the normalizers need to recognize a run's volatile values. */
|
||||
export interface NormalizeContext {
|
||||
/** The session id(s) the run issued — replaced with `{{sessionId}}`. */
|
||||
sessionIds: string[]
|
||||
/** The temp cwd the run used — replaced with `{{cwd}}`. */
|
||||
cwd: string
|
||||
}
|
||||
|
||||
/** Replace cwd, session ids, and any stray UUID with stable tokens in a string. */
|
||||
function scrubString(value: string, ctx: NormalizeContext): string {
|
||||
let out = value
|
||||
// cwd first (longest, most specific), then explicit session ids, then any
|
||||
// residual UUID (covers ids that appear in places we didn't enumerate).
|
||||
out = out.split(ctx.cwd).join(CWD)
|
||||
for (const id of ctx.sessionIds) out = out.split(id).join(SESSION_ID)
|
||||
out = out.replace(UUID_RE, SESSION_ID)
|
||||
return out
|
||||
}
|
||||
|
||||
/** Recursively scrub a parsed JSON value (strings replaced; structure kept). */
|
||||
function scrubValue(value: unknown, ctx: NormalizeContext): unknown {
|
||||
if (typeof value === 'string') return scrubString(value, ctx)
|
||||
if (Array.isArray(value)) return value.map(v => scrubValue(v, ctx))
|
||||
if (value !== null && typeof value === 'object') {
|
||||
const out: Record<string, unknown> = {}
|
||||
for (const [k, v] of Object.entries(value)) out[k] = scrubValue(v, ctx)
|
||||
return out
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize a raw stdout transcript (newline-delimited JSON-RPC frames) into a
|
||||
* stable golden in the SAME shape as the wire: one compact JSON frame per line
|
||||
* (NDJSON), with the JSON-RPC `id` rewritten to a per-transcript sequence
|
||||
* (1, 2, 3, …) and all volatile strings scrubbed. Throws if any non-empty line
|
||||
* is not valid JSON — that doubles as the stdout-purity check (no logger leaked
|
||||
* onto the protocol).
|
||||
*
|
||||
* @param rawStdout The captured stdout bytes, decoded utf8.
|
||||
* @param ctx The run's volatile values to scrub.
|
||||
* @returns The normalized NDJSON transcript, one frame per line.
|
||||
*/
|
||||
export function normalizeStdout(rawStdout: string, ctx: NormalizeContext): string {
|
||||
const lines = rawStdout.split('\n').filter(line => line.trim().length > 0)
|
||||
// Map each distinct JSON-RPC id (request/response correlate by id) to a stable
|
||||
// sequence number, in first-seen order, so id churn doesn't perturb the golden.
|
||||
const idSeq = new Map<string, number>()
|
||||
const stableId = (id: unknown): number => {
|
||||
const key = JSON.stringify(id)
|
||||
let n = idSeq.get(key)
|
||||
if (n === undefined) { n = idSeq.size + 1; idSeq.set(key, n) }
|
||||
return n
|
||||
}
|
||||
const frames = lines.map((line) => {
|
||||
const frame = JSON.parse(line) as Record<string, unknown>
|
||||
if ('id' in frame && frame.id !== undefined && frame.id !== null) {
|
||||
frame.id = stableId(frame.id)
|
||||
}
|
||||
return scrubValue(frame, ctx) as Record<string, unknown>
|
||||
})
|
||||
return frames.map(f => JSON.stringify(f)).join('\n') + '\n'
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize a session JSONL log into a stable golden: the header line's
|
||||
* volatile fields (`createdAt`, `id`, `cwd`) and every event's `time` are
|
||||
* zeroed/scrubbed, all volatile strings scrubbed, and `seq` is LEFT INTACT
|
||||
* (deterministic by contract). Output is JSONL in the same shape as the input —
|
||||
* one compact record per line.
|
||||
*
|
||||
* @param rawLog The raw session `.jsonl` content.
|
||||
* @param ctx The run's volatile values to scrub.
|
||||
* @returns The normalized JSONL log, one record per line.
|
||||
*/
|
||||
export function normalizeSessionLog(rawLog: string, ctx: NormalizeContext): string {
|
||||
const lines = rawLog.split('\n').filter(line => line.trim().length > 0)
|
||||
const records = lines.map((line) => {
|
||||
const record = JSON.parse(line) as Record<string, unknown>
|
||||
// Header line: { type: 'session', createdAt, id, cwd, … }.
|
||||
if (record.type === 'session') {
|
||||
if ('createdAt' in record) record.createdAt = 0
|
||||
} else if ('time' in record) {
|
||||
// Event line: zero the epoch-ms timestamp; keep seq (deterministic).
|
||||
record.time = 0
|
||||
// A hook/result carries the hook's wall-clock runtime (`data.durationMs`),
|
||||
// which is run-to-run noise like `time` — zero it so the golden reflects
|
||||
// the hook's decision/exit, not how long the shell took.
|
||||
if (record.type === 'hook/result' && record.data !== null && typeof record.data === 'object') {
|
||||
const data = record.data as Record<string, unknown>
|
||||
if ('durationMs' in data) data.durationMs = 0
|
||||
}
|
||||
}
|
||||
return scrubValue(record, ctx) as Record<string, unknown>
|
||||
})
|
||||
return records.map(r => JSON.stringify(r)).join('\n') + '\n'
|
||||
}
|
||||
|
||||
/**
|
||||
* Replace request-header CONTENT in a session JSONL with stable tokens,
|
||||
* keeping its structure: a `request/header` event's `data.header.system` →
|
||||
* `{{system}}`, `data.header.tools` → `{{tools}}`, and
|
||||
* `data.header.messagePrefix` → one `{{messagePrefix}}` token per message
|
||||
* (the session prefix is model-visible bulk — an AGENTS digest, a skills
|
||||
* catalog — so its COUNT stays a structural fact while its text never lands
|
||||
* in a fixture); a
|
||||
* `request/header-delta` event keeps every structural fact — the system
|
||||
* delta's `keepStart`/`keepEnd` line positions and inserted-line COUNT (one
|
||||
* `{{system}}` token per inserted line), the tools delta's
|
||||
* added/removed/changed tool NAMES, the prefix replacement's message COUNT —
|
||||
* and tokenizes only the bulk (prompt
|
||||
* text; each added/changed schema's fields other than `name` → `{{tools}}`;
|
||||
* each replacement prefix message → `{{messagePrefix}}`),
|
||||
* so two different deltas still compare different.
|
||||
* Absent fields stay absent — WHETHER a header carried a system prompt,
|
||||
* tools, or a prefix is behavior and stays visible; `config` and `reason`
|
||||
* are small and
|
||||
* stable, so they stay verbatim (a model swap churns every fixture by design
|
||||
* — it invalidates the recorded responses; a prompt/schema edit churns none —
|
||||
* replay never reads this content, see dsh-llm-replay).
|
||||
*
|
||||
* Only lines with something to scrub are re-serialized; every other line
|
||||
* passes through byte-for-byte, so the transform is idempotent and applying
|
||||
* it to an already-scrubbed fixture is a no-op — the on-disk-fixtures guard
|
||||
* in ./suite.ts relies on exactly that.
|
||||
*
|
||||
* @param rawLog The raw session `.jsonl` content.
|
||||
* @returns The JSONL with header content tokenized, other lines byte-identical.
|
||||
*/
|
||||
export function scrubRequestHeaders(rawLog: string): string {
|
||||
const lines = rawLog.split('\n')
|
||||
const out = lines.map((line) => {
|
||||
if (line.trim().length === 0) return line
|
||||
const record = JSON.parse(line) as Record<string, unknown>
|
||||
const data = record.data as Record<string, unknown> | null | undefined
|
||||
if (data === null || typeof data !== 'object') return line
|
||||
if (record.type === 'request/header') {
|
||||
const header = data.header as Record<string, unknown> | null | undefined
|
||||
if (header === null || typeof header !== 'object') return line
|
||||
if (!('system' in header) && !('tools' in header) && !('messagePrefix' in header)) return line
|
||||
if ('system' in header) header.system = SYSTEM
|
||||
if ('tools' in header) header.tools = TOOLS
|
||||
if (Array.isArray(header.messagePrefix)) header.messagePrefix = header.messagePrefix.map(() => MESSAGE_PREFIX)
|
||||
return JSON.stringify(record)
|
||||
}
|
||||
if (record.type === 'request/header-delta') {
|
||||
let touched = false
|
||||
const system = data.system as Record<string, unknown> | null | undefined
|
||||
if (system !== null && typeof system === 'object' && Array.isArray(system.insert)) {
|
||||
system.insert = system.insert.map(() => SYSTEM)
|
||||
touched = true
|
||||
}
|
||||
const tools = data.tools as Record<string, unknown> | null | undefined
|
||||
if (tools !== null && typeof tools === 'object') {
|
||||
if (Array.isArray(tools.added)) { tools.added = tools.added.map(scrubToolSchema); touched = true }
|
||||
if (Array.isArray(tools.changed)) { tools.changed = tools.changed.map(scrubToolSchema); touched = true }
|
||||
}
|
||||
if (Array.isArray(data.messagePrefix)) {
|
||||
data.messagePrefix = data.messagePrefix.map(() => MESSAGE_PREFIX)
|
||||
touched = true
|
||||
}
|
||||
return touched ? JSON.stringify(record) : line
|
||||
}
|
||||
return line
|
||||
})
|
||||
return out.join('\n')
|
||||
}
|
||||
|
||||
/** Tokenize one tool schema's bulk (description, parameters, anything else), keeping its identifying `name`. */
|
||||
function scrubToolSchema(tool: unknown): unknown {
|
||||
if (tool === null || typeof tool !== 'object' || Array.isArray(tool)) return tool
|
||||
const out: Record<string, unknown> = {}
|
||||
for (const [k, v] of Object.entries(tool)) out[k] = k === 'name' ? v : TOOLS
|
||||
return out
|
||||
}
|
||||
393
packages/support/acp-snapshot/src/suite.ts
Normal file
393
packages/support/acp-snapshot/src/suite.ts
Normal file
@@ -0,0 +1,393 @@
|
||||
/**
|
||||
* The ACP snapshot suite factory (REPLAY by default, keyless). A suite is a
|
||||
* scenario table plus a snapshots directory: each scenario under
|
||||
* `<snapshotsDir>/<name>/` ships an `input.json` (the client stdin script) and
|
||||
* a `session.jsonl` fixture; replay boots the real agent subprocess
|
||||
* (./harness.ts), drives it, and diffs the normalized stdout transcript
|
||||
* against the committed `stdout.golden.jsonl`. For model scenarios it ALSO
|
||||
* checks the re-persisted session log — against the `session.jsonl` fixture
|
||||
* itself, not a separate golden: the fixture doubles as the replay source
|
||||
* (recorded scenarios) and the expected produced log (both sides normalized
|
||||
* before comparing).
|
||||
*
|
||||
* Request-header content (the composed system prompt + tool schemas riding on
|
||||
* `request/header` events) is pinned by exactly ONE scenario per suite — the
|
||||
* one with `pinsHeader` — and scrubbed to `{{system}}`/`{{tools}}` tokens in
|
||||
* every other fixture and compare, so a prompt or tool-schema edit churns one
|
||||
* committed line instead of every fixture. A per-run uniformity guard keeps
|
||||
* the single pin sound: every live header must equal the pinned one, and no
|
||||
* header-delta may appear outside the pinning scenario (see the
|
||||
* pinned-header RFC,
|
||||
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
|
||||
*
|
||||
* `pnpm run test:snapshot:record` (DSH_SNAPSHOT=record + -u) re-records the
|
||||
* `session.jsonl` fixtures against the real API and refreshes the stdout golden
|
||||
* in one pass; the caller resolves that env into {@link SnapshotSuiteOptions}
|
||||
* (env reading stays at the suite edge, not in this library).
|
||||
*
|
||||
* @module @deepseek-ai/dsh-acp-snapshot/suite
|
||||
*/
|
||||
|
||||
import { readFile, readdir, writeFile } from 'node:fs/promises'
|
||||
import { existsSync } from 'node:fs'
|
||||
import { join } from 'node:path'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { type AgentUnderTest, type HarvestedLog, type InputScript, runScenario } from './harness.ts'
|
||||
import { type NormalizeContext, normalizeSessionLog, normalizeStdout, scrubRequestHeaders } from './normalize.ts'
|
||||
|
||||
/** A snapshot scenario and how its fixtures are produced. */
|
||||
export interface Scenario {
|
||||
name: string
|
||||
/** Whether the scenario drives at least one model turn (so a JSONL golden applies). */
|
||||
hasModelTurn: boolean
|
||||
/**
|
||||
* Whether the run persists a comparable session log to diff against the
|
||||
* `session.jsonl` fixture. Defaults to {@link hasModelTurn} (a model turn
|
||||
* always produces a log worth comparing). Set it independently for a scenario
|
||||
* that produces a non-trivial log WITHOUT a model turn — e.g. a prompt blocked
|
||||
* by a `UserPromptSubmit` hook, which opens a `rejected` turn carrying `hook/*`
|
||||
* events but never calls the model.
|
||||
*/
|
||||
comparesLog?: boolean
|
||||
/**
|
||||
* Whether `test:snapshot:record` regenerates this scenario's `session.jsonl`
|
||||
* from the LIVE API. `recorded` scenarios are model-driven and reproducible;
|
||||
* `authored` scenarios (fixtures hand-written or hand-harvested — e.g. a
|
||||
* provider error or a cancel the live API can't be coaxed into
|
||||
* deterministically, a deterministic hook scenario, or a scripted repetition
|
||||
* a live model won't reproduce) are NEVER re-recorded.
|
||||
*/
|
||||
recorded: boolean
|
||||
/**
|
||||
* Whether replay is driven by a hand-written `replay.override.json` sidecar
|
||||
* (a `ReplayEntry[]` that REPLACES the script derived from `session.jsonl`)
|
||||
* — the throw/hang cases chunks cannot express. The fixture guard requires
|
||||
* the sidecar exactly when this is set: the harness forwards the file purely
|
||||
* on existence, so an unregistered stray sidecar would silently replace the
|
||||
* derived script — the guard fails loud on either mismatch. Defaults to
|
||||
* false (replay derives from the fixture's `assistant/chunk` events).
|
||||
*/
|
||||
overridden?: boolean
|
||||
/**
|
||||
* How many SUBAGENT child sessions this scenario records beyond the top-level
|
||||
* one (0 for a single-session scenario). Each child rides in a sibling fixture
|
||||
* `session.<n>.jsonl` (1-based); replay forwards them to `dsh-llm-replay` so
|
||||
* each child session replays from its own script, and record mode writes the
|
||||
* harvested child logs back to those files. Defaults to 0.
|
||||
*/
|
||||
childSessions?: number
|
||||
/**
|
||||
* Whether THIS scenario's fixtures keep the full request-header content (the
|
||||
* composed system prompt and tool schema list on `request/header` /
|
||||
* `request/header-delta` events) and compare it verbatim. Exactly one
|
||||
* scenario per suite pins it; every other scenario stores and compares that
|
||||
* content as `{{system}}`/`{{tools}}` tokens ({@link scrubRequestHeaders}),
|
||||
* so a system prompt or tool-schema change shows up as ONE committed-fixture
|
||||
* diff, not one per scenario. One pin suffices because header composition is
|
||||
* suite-uniform (parent, spawn child, and fork child all compose the same
|
||||
* prompt-modulo-cwd and the same tools) — and that premise is ASSERTED, not
|
||||
* assumed: every non-pinning run's live headers must equal the pinned
|
||||
* fixture's (normalized), so a session-dependent header (say, a restricted
|
||||
* subagent toolset) fails loud until it gets its own pinning scenario.
|
||||
* Defaults to false.
|
||||
*/
|
||||
pinsHeader?: boolean
|
||||
/**
|
||||
* Whether this scenario intentionally composes a different live request
|
||||
* header from the suite's pinned baseline while still storing scrubbed header
|
||||
* content in its fixture. Use only for cwd/project-sensitive context fixtures
|
||||
* whose header difference is the behavior under test; the fixture guard still
|
||||
* rejects committed full header content unless {@link pinsHeader} is true.
|
||||
* Defaults to false.
|
||||
*/
|
||||
variesHeader?: boolean
|
||||
}
|
||||
|
||||
/** One suite's inputs: the agent to boot, where its fixtures live, and its scenario table. */
|
||||
export interface SnapshotSuiteOptions {
|
||||
/** The agent composition every scenario boots. */
|
||||
agent: AgentUnderTest
|
||||
/** Absolute path of the suite's `snapshots/` directory (one subdir per scenario). */
|
||||
snapshotsDir: string
|
||||
/** The scenario table; exactly one entry must set `pinsHeader`. */
|
||||
scenarios: Scenario[]
|
||||
/**
|
||||
* `replay` (keyless, the default tier) or `record` (live API; re-records the
|
||||
* `recorded` scenarios' fixtures and refreshes the vitest goldens under
|
||||
* `--update`). The caller derives this from `$DSH_SNAPSHOT` — env reading
|
||||
* stays outside this library.
|
||||
*/
|
||||
mode: 'replay' | 'record'
|
||||
}
|
||||
|
||||
/**
|
||||
* The sibling child-fixture paths for a scenario (`session.1.jsonl` …).
|
||||
*
|
||||
* @param dir The scenario's snapshots directory (`<snapshotsDir>/<name>`).
|
||||
* @param childSessions How many subagent child sessions the scenario records.
|
||||
* @returns One path per child, 1-based, in fixture order.
|
||||
*/
|
||||
export function childFixturePaths(dir: string, childSessions: number): string[] {
|
||||
return Array.from({ length: childSessions }, (_, i) => join(dir, `session.${i + 1}.jsonl`))
|
||||
}
|
||||
|
||||
/**
|
||||
* Derive the {@link NormalizeContext} for a `session.jsonl` fixture from its own
|
||||
* header line (`{ type: 'session', id, cwd }`). A committed fixture carries the
|
||||
* session id and cwd of the run that harvested it — different from the live
|
||||
* replay run — so normalizing it against the live run's ctx would leave those
|
||||
* recorded values unscrubbed. Reading them from the header scrubs the fixture's
|
||||
* own id/cwd to the same `{{sessionId}}`/`{{cwd}}` tokens the replay output gets.
|
||||
* An authored fixture whose header is already normalized (`id:'{{sessionId}}'`,
|
||||
* `cwd:'{{cwd}}'`) yields those tokens as the volatile values, so scrubbing them
|
||||
* is an idempotent no-op. A header with no `cwd` falls back to a sentinel that
|
||||
* cannot occur in a log (NOT `''`, which `String.split` would match on every
|
||||
* character boundary and corrupt the output).
|
||||
*
|
||||
* @param fixture The committed `session.jsonl` content.
|
||||
* @returns The fixture's own volatile values, ready for {@link normalizeSessionLog}.
|
||||
*/
|
||||
export function fixtureContext(fixture: string): NormalizeContext {
|
||||
const firstLine = fixture.split('\n').find(line => line.trim().length > 0) ?? '{}'
|
||||
const header = JSON.parse(firstLine) as { id?: unknown; cwd?: unknown }
|
||||
return {
|
||||
sessionIds: typeof header.id === 'string' ? [header.id] : [],
|
||||
cwd: typeof header.cwd === 'string' ? header.cwd : '\0no-cwd\0',
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* The `data.header` payload of every `request/header` event in a session
|
||||
* JSONL, in log order, with the log's volatile values scrubbed first
|
||||
* ({@link normalizeSessionLog}) so headers harvested from different runs —
|
||||
* each embedding its own temp cwd in the composed prompt — compare on equal
|
||||
* footing.
|
||||
*
|
||||
* @param rawLog The session `.jsonl` content to extract headers from.
|
||||
* @param ctx The volatile values of the run that produced it.
|
||||
* @returns The normalized `data.header` payloads, in log order.
|
||||
*/
|
||||
export function normalizedHeaders(rawLog: string, ctx: NormalizeContext): unknown[] {
|
||||
return normalizeSessionLog(rawLog, ctx)
|
||||
.split('\n')
|
||||
.filter(line => line.trim().length > 0)
|
||||
.map(line => JSON.parse(line) as { type?: unknown; data?: { header?: unknown } })
|
||||
.filter(record => record.type === 'request/header')
|
||||
.map(record => record.data?.header)
|
||||
}
|
||||
|
||||
/**
|
||||
* Count the `request/header-delta` events in a session JSONL.
|
||||
*
|
||||
* @param rawLog The session `.jsonl` content.
|
||||
* @returns How many `request/header-delta` events the log carries.
|
||||
*/
|
||||
export function headerDeltaCount(rawLog: string): number {
|
||||
return rawLog.split('\n')
|
||||
.filter(line => line.trim().length > 0)
|
||||
.filter(line => (JSON.parse(line) as { type?: unknown }).type === 'request/header-delta')
|
||||
.length
|
||||
}
|
||||
|
||||
/**
|
||||
* Register the suite: one `describe` per scenario (the golden/log compares and
|
||||
* the header-uniformity guard) plus the fixture guard block (no orphan
|
||||
* scenario dirs, required files present, exactly one pin, non-pinning fixtures
|
||||
* header-scrubbed). Must run at vitest collection time — it calls
|
||||
* `describe`/`it`. Throws immediately if no scenario pins the header (the
|
||||
* uniformity guard would have nothing to compare against).
|
||||
*
|
||||
* @param options The agent, snapshots directory, scenario table, and mode.
|
||||
*/
|
||||
export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
const { agent, snapshotsDir, scenarios, mode } = options
|
||||
const RECORDING = mode === 'record'
|
||||
|
||||
/** The suite's single header-pinning scenario. Guarded here (and by a meta-test) so the pin cannot silently vanish. */
|
||||
const pinningScenario = scenarios.find(s => s.pinsHeader === true)
|
||||
if (pinningScenario === undefined) throw new Error('acp-snapshot: no scenario pins the request-header content')
|
||||
|
||||
for (const scenario of scenarios) {
|
||||
describe(`snapshot: ${scenario.name}`, () => {
|
||||
// In RECORD mode, only re-run the `recorded` (live-API) scenarios; the
|
||||
// `authored` ones (sidecar-driven errors/cancel) are never re-recorded.
|
||||
it.skipIf(RECORDING && !scenario.recorded)('matches the goldens', async () => {
|
||||
const dir = join(snapshotsDir, scenario.name)
|
||||
const input = JSON.parse(await readFile(join(dir, 'input.json'), 'utf8')) as InputScript
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
const workspaceDir = join(dir, 'workspace')
|
||||
const childSessions = scenario.childSessions ?? 0
|
||||
const result = await runScenario(input, {
|
||||
agent,
|
||||
mode,
|
||||
fixtureFile: join(dir, 'session.jsonl'),
|
||||
...existsSync(overrideFile) ? { overrideFile } : {},
|
||||
// In REPLAY, forward the recorded child fixtures so each subagent session
|
||||
// replays from its own script. In RECORD they are harvested, not read.
|
||||
...!RECORDING && childSessions > 0 ? { childFiles: childFixturePaths(dir, childSessions) } : {},
|
||||
...existsSync(workspaceDir) ? { workspaceDir } : {},
|
||||
})
|
||||
|
||||
// Scrub every volatile id the run produced: the ACP server-issued session
|
||||
// id plus every harvested log's recorded id (a subagent child id never
|
||||
// surfaces over ACP, but it appears in the child's own log header). The
|
||||
// normalizer's UUID catch-all covers any we don't enumerate.
|
||||
const ctx: NormalizeContext = {
|
||||
sessionIds: [
|
||||
...result.sessionId !== undefined ? [result.sessionId] : [],
|
||||
...result.sessionLogs.map(l => l.id),
|
||||
],
|
||||
cwd: result.cwd,
|
||||
}
|
||||
|
||||
// RECORD mode (recorded model scenarios only): persist the freshly-harvested
|
||||
// logs back to their fixtures — the primary to session.jsonl, each child to
|
||||
// session.<n>.jsonl in harvest order. `--update` refreshes the Vitest
|
||||
// goldens but NOT these fixtures, so write them here. A non-pinning
|
||||
// scenario's fixtures are written header-scrubbed, so a re-record can
|
||||
// never smuggle the full prompt/schema content back into every fixture.
|
||||
const scrub = scenario.pinsHeader === true
|
||||
? (log: string): string => log
|
||||
: scrubRequestHeaders
|
||||
if (RECORDING && scenario.recorded && scenario.hasModelTurn) {
|
||||
expect(result.sessionLogs.length, 'record produced no session log to harvest').toBeGreaterThan(0)
|
||||
expect(result.sessionLogs.length, `expected ${childSessions + 1} session logs (parent + children)`)
|
||||
.toBe(childSessions + 1)
|
||||
await writeFile(join(dir, 'session.jsonl'), scrub((result.sessionLogs[0] as HarvestedLog).content))
|
||||
for (let i = 1; i < result.sessionLogs.length; i++) {
|
||||
await writeFile(join(dir, `session.${i}.jsonl`), scrub((result.sessionLogs[i] as HarvestedLog).content))
|
||||
}
|
||||
}
|
||||
|
||||
await expect(normalizeStdout(result.rawStdout, ctx))
|
||||
.toMatchFileSnapshot(join(dir, 'stdout.golden.jsonl'))
|
||||
|
||||
// A model turn always produces a log worth comparing; a hook scenario can
|
||||
// produce one without a model turn (a `rejected` turn carrying `hook/*`).
|
||||
const comparesLog = scenario.comparesLog ?? scenario.hasModelTurn
|
||||
if (comparesLog) {
|
||||
// The harvested logs (primary-first) must match their committed fixtures
|
||||
// 1:1. Each side passes through normalizeSessionLog, scrubbed against ITS
|
||||
// OWN volatile values — the live run's via `ctx`, the committed fixture's
|
||||
// via its own header (a committed file cannot share the live run's ids).
|
||||
// Unless this scenario pins the header, both sides ALSO pass through
|
||||
// scrubRequestHeaders: the live log carries the real prompt/schemas, the
|
||||
// fixture carries the `{{system}}`/`{{tools}}` tokens, and the scrub is
|
||||
// idempotent — so the compare checks the header's presence, position,
|
||||
// reason, and config, but not its bulk content (pinned once, in the
|
||||
// `pinsHeader` scenario).
|
||||
expect(result.sessionLogs.length, 'this scenario must persist a session log').toBe(childSessions + 1)
|
||||
const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)]
|
||||
for (let i = 0; i < fixtureFiles.length; i++) {
|
||||
const harvested = scrub((result.sessionLogs[i] as HarvestedLog).content)
|
||||
const fixture = scrub(await readFile(join(dir, fixtureFiles[i] as string), 'utf8'))
|
||||
expect(normalizeSessionLog(harvested, ctx), `${fixtureFiles[i]} mismatch`)
|
||||
.toEqual(normalizeSessionLog(fixture, fixtureContext(fixture)))
|
||||
}
|
||||
}
|
||||
|
||||
// Header-uniformity guard: the single pin is sound only while every
|
||||
// session in the suite composes the SAME header and keeps it for the
|
||||
// whole run. Assert both halves live. (1) Every request/header the run
|
||||
// produced (parent, spawn child, fork child, initial or resume) must
|
||||
// equal the pinned fixture's header after each side is normalized
|
||||
// against its own volatile values. (2) No request/header-delta may
|
||||
// appear at all — a mid-run header change diverges from the pin by
|
||||
// construction, and its content would be invisible under the scrub. If
|
||||
// either fails, either the header changed (update the pin: re-record or
|
||||
// hand-edit the pinning scenario's fixture) or composition became
|
||||
// session-dependent by design (give the divergent shape its own
|
||||
// pinning scenario).
|
||||
if (scenario.pinsHeader !== true && scenario.variesHeader !== true) {
|
||||
const pinnedFixture = await readFile(join(snapshotsDir, pinningScenario.name, 'session.jsonl'), 'utf8')
|
||||
const pinned = normalizedHeaders(pinnedFixture, fixtureContext(pinnedFixture))
|
||||
expect(pinned.length, `the pinning fixture (${pinningScenario.name}) must carry exactly one request/header`)
|
||||
.toBe(1)
|
||||
for (const log of result.sessionLogs) {
|
||||
expect(headerDeltaCount(log.content), `session ${log.id}: a request/header-delta in a non-pinning scenario`)
|
||||
.toBe(0)
|
||||
const headers = normalizedHeaders(log.content, ctx)
|
||||
for (const [k, header] of headers.entries()) {
|
||||
expect(header, `session ${log.id}: request/header #${k + 1} diverged from the pinned (${pinningScenario.name}) header`)
|
||||
.toEqual(pinned[0])
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
describe('snapshot fixtures', () => {
|
||||
it('every scenario directory is registered (no orphans)', async () => {
|
||||
// toMatchFileSnapshot does not prune orphaned golden/fixture files, so a
|
||||
// renamed/removed scenario could leave a stale dir that nothing exercises.
|
||||
// Fail loud on any snapshots/<dir> not present in the scenario table.
|
||||
const entries = await readdir(snapshotsDir, { withFileTypes: true })
|
||||
const onDisk = entries.filter(e => e.isDirectory()).map(e => e.name).sort()
|
||||
const registered = scenarios.map(s => s.name).sort()
|
||||
expect(onDisk).toEqual(registered)
|
||||
})
|
||||
|
||||
it('every registered scenario has its required fixture files', () => {
|
||||
// Every scenario has an input script and an stdout golden. EVERY scenario
|
||||
// also needs `session.jsonl`: the suite boots `llm-replay` with that path
|
||||
// as the replay source for ALL scenarios (the factory passes
|
||||
// `fixtureFile: <dir>/session.jsonl` unconditionally), and `loadReplayScript`
|
||||
// throws "fixture not found" when it is absent and no override replaces it.
|
||||
// A no-model scenario ships a header-only `session.jsonl` (it derives to an
|
||||
// empty script — no model call is made); a model scenario's fixture also
|
||||
// doubles as the expected-log artifact the run is diffed against. The
|
||||
// `replay.override.json` sidecar is matched BOTH ways against the table's
|
||||
// `overridden` flag: required when set, forbidden when not — the harness
|
||||
// forwards the file purely on existence, so an unregistered stray sidecar
|
||||
// would silently replace the derived script.
|
||||
for (const { name, overridden, childSessions } of scenarios) {
|
||||
const dir = join(snapshotsDir, name)
|
||||
expect(existsSync(join(dir, 'input.json')), `${name}/input.json`).toBe(true)
|
||||
expect(existsSync(join(dir, 'stdout.golden.jsonl')), `${name}/stdout.golden.jsonl`).toBe(true)
|
||||
expect(existsSync(join(dir, 'session.jsonl')), `${name}/session.jsonl`).toBe(true)
|
||||
expect(existsSync(join(dir, 'replay.override.json')), `${name}/replay.override.json presence must match \`overridden\``)
|
||||
.toBe(overridden === true)
|
||||
// A nested-agent scenario ships one child fixture per recorded subagent
|
||||
// session (`session.1.jsonl` …), the replay source for that child session.
|
||||
for (const childFixture of childFixturePaths(dir, childSessions ?? 0)) {
|
||||
expect(existsSync(childFixture), childFixture).toBe(true)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
it('exactly one scenario pins the request-header content', () => {
|
||||
// Zero pins would drop the prompt/schema surface from the suite entirely;
|
||||
// two would split it. One pin per suite is the design (pinned-header RFC);
|
||||
// WHICH scenario pins is the scenario table's reviewable choice.
|
||||
expect(scenarios.filter(s => s.pinsHeader === true).map(s => s.name)).toEqual([pinningScenario.name])
|
||||
})
|
||||
|
||||
it('committed fixtures carry request-header content ONLY in the pinning scenario', async () => {
|
||||
// The whole point of the pin: a system-prompt or tool-schema change must
|
||||
// churn exactly one committed line. A non-pinning fixture that carries the
|
||||
// full header (a hand-recorded file, or a header line hand-edited out of
|
||||
// its canonical JSON form) silently reopens the suite-wide churn, so fail
|
||||
// loud here: every non-pinning session*.jsonl must be a fixed point of
|
||||
// scrubRequestHeaders (apply the scrub to fix a violation), and the
|
||||
// pinning scenario's fixtures must NOT be (their content IS the pin).
|
||||
for (const scenario of scenarios) {
|
||||
const dir = join(snapshotsDir, scenario.name)
|
||||
const files = [
|
||||
'session.jsonl',
|
||||
...Array.from({ length: scenario.childSessions ?? 0 }, (_, i) => `session.${i + 1}.jsonl`),
|
||||
]
|
||||
for (const file of files) {
|
||||
const fixture = await readFile(join(dir, file), 'utf8')
|
||||
if (scenario.pinsHeader === true) {
|
||||
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} must PIN the full header content`)
|
||||
.not.toEqual(fixture)
|
||||
} else {
|
||||
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} carries unscrubbed header content`)
|
||||
.toEqual(fixture)
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
232
packages/support/acp-snapshot/tests/fixtures/fake-acp-agent.ts
vendored
Normal file
232
packages/support/acp-snapshot/tests/fixtures/fake-acp-agent.ts
vendored
Normal file
@@ -0,0 +1,232 @@
|
||||
/**
|
||||
* Scripted fake ACP agent bin for `dsh-acp-snapshot`'s unit specs. Speaks
|
||||
* newline-delimited JSON-RPC on stdio like the real `dsh-acp-agent` bin, but
|
||||
* every behavior — how prompts settle, whether session/new rejects, which
|
||||
* session logs get persisted, what filesystem noise to leave — comes from a
|
||||
* `behavior.json` sitting NEXT to the `$DSH_SNAPSHOT_FILE` fixture, so a spec
|
||||
* scripts a whole subprocess run from data. The specs launch it through the
|
||||
* REAL `runScenario` spawn path (tsx loader, temp cwd, env plumbing), so the
|
||||
* harness plumbing is exercised for real; only the agent behind the protocol
|
||||
* is scripted.
|
||||
*
|
||||
* The specs (not the golden tier) own this bin: it asserts nothing, echoes
|
||||
* observable facts into `session/update` text chunks (env probe, permission
|
||||
* outcome, seeded-workspace listing) for the spec to read off `rawStdout`, and
|
||||
* exits 0 on stdin EOF after writing the scripted logs — mirroring the real
|
||||
* bin's dispose-flush-exit shape.
|
||||
*/
|
||||
|
||||
import { mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
|
||||
import { readdirSync } from 'node:fs'
|
||||
import { dirname, join } from 'node:path'
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import { createInterface } from 'node:readline'
|
||||
|
||||
/** One scripted session log: a file path under the sessions root plus its JSONL lines. */
|
||||
interface ScriptedLog {
|
||||
/** Path relative to `$DSH_SNAPSHOT_SESSIONS_ROOT`, e.g. `bucket/a.jsonl` (an empty dir segment is invalid). */
|
||||
file: string
|
||||
/**
|
||||
* The JSONL records. String templates `{{CWD}}` and `{{SID}}` are replaced
|
||||
* with the run's real cwd and the ACP session id this bin issued, so a
|
||||
* written log carries genuine volatile values for the normalizers to scrub.
|
||||
*/
|
||||
lines: unknown[]
|
||||
}
|
||||
|
||||
/** The whole scripted behavior for one run. Every field defaults to the least surprising choice. */
|
||||
interface Behavior {
|
||||
/** Reject every `session/new` (exercises the expect-error step without extra dirs). */
|
||||
rejectNewSession?: boolean
|
||||
/** Reject `session/new` only when `additionalDirectories` is non-empty (the real bridge's rule). */
|
||||
rejectExtraDirs?: boolean
|
||||
/** How `session/prompt` settles: a clean response, a JSON-RPC error, or a hang until `session/cancel`. */
|
||||
prompt?: 'respond' | 'error' | 'hang-until-cancel'
|
||||
/** Before responding to a prompt, send a `session/request_permission` request and echo its outcome as a chunk. */
|
||||
permissionProbe?: boolean
|
||||
/** Echo the `DSH_SNAPSHOT_*` env the harness set as a chunk (spec-side env-plumbing assertions). */
|
||||
echoEnv?: boolean
|
||||
/** Echo the sorted cwd listing as a chunk (spec-side workspace-seeding assertions). */
|
||||
echoWorkspace?: boolean
|
||||
/** Write a line to stderr on boot (spec-side stderr-capture assertions). */
|
||||
stderrNote?: string
|
||||
/** Session logs to persist on stdin EOF. */
|
||||
logs?: ScriptedLog[]
|
||||
/** Leave a stray FILE directly under the sessions root (harvest must skip it). */
|
||||
strayRootFile?: boolean
|
||||
/** Leave a stray non-`.jsonl` file inside a bucket (harvest must skip it). */
|
||||
strayBucketFile?: boolean
|
||||
/** Delete the sessions root entirely (harvest must yield no logs). */
|
||||
deleteSessionsRoot?: boolean
|
||||
}
|
||||
|
||||
const sessionsRoot = process.env.DSH_SNAPSHOT_SESSIONS_ROOT ?? ''
|
||||
const fixtureFile = process.env.DSH_SNAPSHOT_FILE ?? ''
|
||||
const behavior: Behavior = fixtureFile === ''
|
||||
? {}
|
||||
: JSON.parse(readFileSync(join(dirname(fixtureFile), 'behavior.json'), 'utf8')) as Behavior
|
||||
|
||||
if (behavior.stderrNote !== undefined) process.stderr.write(`${behavior.stderrNote}\n`)
|
||||
|
||||
let nextOutboundId = 1000
|
||||
let sessionId = ''
|
||||
/**
|
||||
* The cwd the client passed to `session/new` — used verbatim for `{{CWD}}`
|
||||
* substitution, mirroring the real bin (whose persisted header carries the
|
||||
* session cwd as given, NOT `process.cwd()`, which the OS realpaths — on
|
||||
* macOS `/var/folders/…` vs `/private/var/folders/…`).
|
||||
*/
|
||||
let sessionCwd = ''
|
||||
/** The parked prompt request id while `hang-until-cancel` waits for the cancel notification. */
|
||||
let parkedPromptId: number | string | null = null
|
||||
/** Resolvers for permission-probe responses, keyed by outbound request id. */
|
||||
const pendingPermission = new Map<number, (outcome: unknown) => void>()
|
||||
|
||||
function send(frame: Record<string, unknown>): void {
|
||||
process.stdout.write(`${JSON.stringify({ jsonrpc: '2.0', ...frame })}\n`)
|
||||
}
|
||||
|
||||
function respond(id: number | string, result: unknown): void {
|
||||
send({ id, result })
|
||||
}
|
||||
|
||||
function respondError(id: number | string, message: string): void {
|
||||
send({ id, error: { code: -32603, message } })
|
||||
}
|
||||
|
||||
function chunk(text: string): void {
|
||||
send({
|
||||
method: 'session/update',
|
||||
params: { sessionId, update: { sessionUpdate: 'agent_message_chunk', content: { type: 'text', text } } },
|
||||
})
|
||||
}
|
||||
|
||||
/** Substitute the `{{CWD}}`/`{{SID}}` templates through a scripted log record. */
|
||||
function instantiate(value: unknown): unknown {
|
||||
if (typeof value === 'string') return value.split('{{CWD}}').join(sessionCwd).split('{{SID}}').join(sessionId)
|
||||
if (Array.isArray(value)) return value.map(instantiate)
|
||||
if (value !== null && typeof value === 'object') {
|
||||
const out: Record<string, unknown> = {}
|
||||
for (const [k, v] of Object.entries(value)) out[k] = instantiate(v)
|
||||
return out
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
async function handlePrompt(id: number | string): Promise<void> {
|
||||
if ((behavior.prompt ?? 'respond') === 'hang-until-cancel') {
|
||||
// A thought chunk BEFORE any message chunk: a promptAndCancel waiter
|
||||
// watches for agent_message_chunk, so this exercises its non-matching
|
||||
// update path while the waiter is armed.
|
||||
send({
|
||||
method: 'session/update',
|
||||
params: { sessionId, update: { sessionUpdate: 'agent_thought_chunk', content: { type: 'text', text: 'mulling' } } },
|
||||
})
|
||||
}
|
||||
chunk('thinking about it')
|
||||
if (behavior.echoEnv === true) {
|
||||
chunk(`env:${JSON.stringify({
|
||||
mode: process.env.DSH_SNAPSHOT,
|
||||
override: process.env.DSH_SNAPSHOT_OVERRIDE ?? null,
|
||||
childFiles: process.env.DSH_SNAPSHOT_CHILD_FILES ?? null,
|
||||
})}`)
|
||||
}
|
||||
if (behavior.echoWorkspace === true) {
|
||||
chunk(`workspace:${readdirSync(process.cwd()).sort().join(',')}`)
|
||||
}
|
||||
if (behavior.permissionProbe === true) {
|
||||
const requestId = nextOutboundId++
|
||||
const outcome = await new Promise<unknown>((resolve) => {
|
||||
pendingPermission.set(requestId, resolve)
|
||||
send({
|
||||
id: requestId,
|
||||
method: 'session/request_permission',
|
||||
params: {
|
||||
sessionId,
|
||||
toolCall: { toolCallId: 'call_fake_1', title: 'fake tool', kind: 'execute', status: 'pending' },
|
||||
options: [
|
||||
{ optionId: 'opt-allow', name: 'Allow once', kind: 'allow_once' },
|
||||
{ optionId: 'opt-reject', name: 'Reject once', kind: 'reject_once' },
|
||||
],
|
||||
},
|
||||
})
|
||||
})
|
||||
chunk(`permission:${JSON.stringify(outcome)}`)
|
||||
}
|
||||
switch (behavior.prompt ?? 'respond') {
|
||||
case 'respond':
|
||||
respond(id, { stopReason: 'end_turn' })
|
||||
return
|
||||
case 'error':
|
||||
respondError(id, 'model exploded')
|
||||
return
|
||||
case 'hang-until-cancel':
|
||||
parkedPromptId = id
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
function handleFrame(frame: Record<string, unknown>): void {
|
||||
const id = frame.id as number | string | undefined
|
||||
const method = frame.method as string | undefined
|
||||
const params = (frame.params ?? {}) as Record<string, unknown>
|
||||
// A response to one of OUR outbound requests (the permission probe).
|
||||
if (method === undefined && id !== undefined && typeof id === 'number' && pendingPermission.has(id)) {
|
||||
const resolve = pendingPermission.get(id) as (outcome: unknown) => void
|
||||
pendingPermission.delete(id)
|
||||
resolve((frame.result as { outcome?: unknown } | undefined)?.outcome ?? null)
|
||||
return
|
||||
}
|
||||
switch (method) {
|
||||
case 'initialize':
|
||||
respond(id as number | string, { protocolVersion: 1, agentCapabilities: { loadSession: false } })
|
||||
return
|
||||
case 'session/new': {
|
||||
const extra = params.additionalDirectories as unknown[] | undefined
|
||||
if (behavior.rejectNewSession === true || (behavior.rejectExtraDirs === true && extra !== undefined && extra.length > 0)) {
|
||||
respondError(id as number | string, 'unsupported workspace scope')
|
||||
return
|
||||
}
|
||||
sessionId = randomUUID()
|
||||
sessionCwd = typeof params.cwd === 'string' ? params.cwd : process.cwd()
|
||||
respond(id as number | string, { sessionId })
|
||||
return
|
||||
}
|
||||
case 'session/prompt':
|
||||
void handlePrompt(id as number | string)
|
||||
return
|
||||
case 'session/cancel':
|
||||
if (parkedPromptId !== null) {
|
||||
const parked = parkedPromptId
|
||||
parkedPromptId = null
|
||||
respond(parked, { stopReason: 'cancelled' })
|
||||
}
|
||||
return
|
||||
default:
|
||||
// Unknown method: a notification is ignored; a request gets an error so
|
||||
// the SDK never waits forever on a frame this fake doesn't model.
|
||||
if (id !== undefined) respondError(id, `unhandled method ${String(method)}`)
|
||||
}
|
||||
}
|
||||
|
||||
function flushLogsAndExit(): void {
|
||||
for (const log of behavior.logs ?? []) {
|
||||
const target = join(sessionsRoot, log.file)
|
||||
mkdirSync(dirname(target), { recursive: true })
|
||||
writeFileSync(target, log.lines.map(l => JSON.stringify(instantiate(l))).join('\n') + '\n')
|
||||
}
|
||||
if (behavior.strayRootFile === true) writeFileSync(join(sessionsRoot, 'stray.txt'), 'not a bucket\n')
|
||||
if (behavior.strayBucketFile === true) {
|
||||
mkdirSync(join(sessionsRoot, 'bucket-noise'), { recursive: true })
|
||||
writeFileSync(join(sessionsRoot, 'bucket-noise', 'notes.txt'), 'not a session log\n')
|
||||
}
|
||||
if (behavior.deleteSessionsRoot === true) rmSync(sessionsRoot, { recursive: true, force: true })
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
const rl = createInterface({ input: process.stdin })
|
||||
rl.on('line', (line) => {
|
||||
if (line.trim().length === 0) return
|
||||
handleFrame(JSON.parse(line) as Record<string, unknown>)
|
||||
})
|
||||
rl.on('close', () => { flushLogsAndExit() })
|
||||
13
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-child/behavior.json
vendored
Normal file
13
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-child/behavior.json
vendored
Normal file
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"prompt": "respond",
|
||||
"logs": [
|
||||
{ "file": "b/parent.jsonl", "lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 700, "cwd": "{{CWD}}" },
|
||||
{ "type": "request/header", "seq": 0, "time": 3, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
|
||||
]},
|
||||
{ "file": "b/child.jsonl", "lines": [
|
||||
{ "type": "session", "id": "abababab-cdcd-4efe-8ada-badabadabada", "createdAt": 800, "cwd": "{{CWD}}", "parentSession": "{{SID}}" },
|
||||
{ "type": "request/header", "seq": 0, "time": 2, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
|
||||
]}
|
||||
]
|
||||
}
|
||||
1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-child/input.json
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-child/input.json
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{ "steps": [{ "op": "initialize" }, { "op": "newSession" }, { "op": "prompt", "text": "rec child" }] }
|
||||
2
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-child/session.1.jsonl
vendored
Normal file
2
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-child/session.1.jsonl
vendored
Normal file
@@ -0,0 +1,2 @@
|
||||
{"type":"session","id":"abababab-cdcd-4efe-8ada-badabadabada","createdAt":800,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-KBQJbW","parentSession":"f6fa7fcf-dd9c-4b39-8815-b25ddcebfd88"}
|
||||
{"type":"request/header","seq":0,"time":2,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
2
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-child/session.jsonl
vendored
Normal file
2
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-child/session.jsonl
vendored
Normal file
@@ -0,0 +1,2 @@
|
||||
{"type":"session","id":"f6fa7fcf-dd9c-4b39-8815-b25ddcebfd88","createdAt":700,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-KBQJbW"}
|
||||
{"type":"request/header","seq":0,"time":3,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
4
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-child/stdout.golden.jsonl
vendored
Normal file
4
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-child/stdout.golden.jsonl
vendored
Normal file
@@ -0,0 +1,4 @@
|
||||
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"loadSession":false}}}
|
||||
{"jsonrpc":"2.0","id":2,"result":{"sessionId":"{{sessionId}}"}}
|
||||
{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"{{sessionId}}","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"thinking about it"}}}}
|
||||
{"jsonrpc":"2.0","id":3,"result":{"stopReason":"end_turn"}}
|
||||
10
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/behavior.json
vendored
Normal file
10
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/behavior.json
vendored
Normal file
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"prompt": "respond",
|
||||
"logs": [{
|
||||
"file": "b/main.jsonl",
|
||||
"lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 600, "cwd": "{{CWD}}" },
|
||||
{ "type": "request/header", "seq": 0, "time": 4, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
|
||||
]
|
||||
}]
|
||||
}
|
||||
1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/input.json
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/input.json
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{ "steps": [{ "op": "initialize" }, { "op": "newSession" }, { "op": "prompt", "text": "rec pin" }] }
|
||||
2
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/session.jsonl
vendored
Normal file
2
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/session.jsonl
vendored
Normal file
@@ -0,0 +1,2 @@
|
||||
{"type":"session","id":"ccdc749f-56f3-4267-9750-598b5c60b7b2","createdAt":600,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-nOQ4Gy"}
|
||||
{"type":"request/header","seq":0,"time":4,"data":{"header":{"config":{"model":"fake"},"system":"SYS PROMPT","tools":[{"name":"t1","description":"D1","parameters":{"type":"object"}}]},"reason":"initial"}}
|
||||
4
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/stdout.golden.jsonl
vendored
Normal file
4
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/stdout.golden.jsonl
vendored
Normal file
@@ -0,0 +1,4 @@
|
||||
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"loadSession":false}}}
|
||||
{"jsonrpc":"2.0","id":2,"result":{"sessionId":"{{sessionId}}"}}
|
||||
{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"{{sessionId}}","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"thinking about it"}}}}
|
||||
{"jsonrpc":"2.0","id":3,"result":{"stopReason":"end_turn"}}
|
||||
1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-skip/behavior.json
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-skip/behavior.json
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{}
|
||||
1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-skip/input.json
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-skip/input.json
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{ "steps": [{ "op": "initialize" }] }
|
||||
1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-skip/replay.override.json
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-skip/replay.override.json
vendored
Normal file
@@ -0,0 +1 @@
|
||||
[{ "kind": "hang" }]
|
||||
1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-skip/session.jsonl
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-skip/session.jsonl
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{"type":"session","id":"{{sessionId}}","createdAt":0,"cwd":"{{cwd}}"}
|
||||
1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-skip/stdout.golden.jsonl
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-skip/stdout.golden.jsonl
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"loadSession":false}}}
|
||||
10
packages/support/acp-snapshot/tests/fixtures/suite/authored-error/behavior.json
vendored
Normal file
10
packages/support/acp-snapshot/tests/fixtures/suite/authored-error/behavior.json
vendored
Normal file
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"prompt": "error",
|
||||
"logs": [{
|
||||
"file": "b/main.jsonl",
|
||||
"lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 500, "cwd": "{{CWD}}" },
|
||||
{ "type": "turn/end", "seq": 1, "time": 9, "data": { "error": "model exploded" } }
|
||||
]
|
||||
}]
|
||||
}
|
||||
1
packages/support/acp-snapshot/tests/fixtures/suite/authored-error/input.json
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/suite/authored-error/input.json
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{ "steps": [{ "op": "initialize" }, { "op": "newSession" }, { "op": "promptExpectError", "text": "boom" }] }
|
||||
1
packages/support/acp-snapshot/tests/fixtures/suite/authored-error/replay.override.json
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/suite/authored-error/replay.override.json
vendored
Normal file
@@ -0,0 +1 @@
|
||||
[{ "kind": "throw", "chunks": [], "message": "model exploded", "code": "PROVIDER" }]
|
||||
2
packages/support/acp-snapshot/tests/fixtures/suite/authored-error/session.jsonl
vendored
Normal file
2
packages/support/acp-snapshot/tests/fixtures/suite/authored-error/session.jsonl
vendored
Normal file
@@ -0,0 +1,2 @@
|
||||
{"type":"session","id":"44444444-3333-4222-8111-000000000000","createdAt":17,"cwd":"/rec/authored-cwd"}
|
||||
{"type":"turn/end","seq":1,"time":17,"data":{"error":"model exploded"}}
|
||||
4
packages/support/acp-snapshot/tests/fixtures/suite/authored-error/stdout.golden.jsonl
vendored
Normal file
4
packages/support/acp-snapshot/tests/fixtures/suite/authored-error/stdout.golden.jsonl
vendored
Normal file
@@ -0,0 +1,4 @@
|
||||
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"loadSession":false}}}
|
||||
{"jsonrpc":"2.0","id":2,"result":{"sessionId":"{{sessionId}}"}}
|
||||
{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"{{sessionId}}","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"thinking about it"}}}}
|
||||
{"jsonrpc":"2.0","id":3,"error":{"code":-32603,"message":"model exploded"}}
|
||||
10
packages/support/acp-snapshot/tests/fixtures/suite/blocked-log/behavior.json
vendored
Normal file
10
packages/support/acp-snapshot/tests/fixtures/suite/blocked-log/behavior.json
vendored
Normal file
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"prompt": "error",
|
||||
"logs": [{
|
||||
"file": "b/main.jsonl",
|
||||
"lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 400, "cwd": "{{CWD}}" },
|
||||
{ "type": "hook/result", "seq": 1, "time": 8, "data": { "decision": "block", "durationMs": 37 } }
|
||||
]
|
||||
}]
|
||||
}
|
||||
1
packages/support/acp-snapshot/tests/fixtures/suite/blocked-log/input.json
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/suite/blocked-log/input.json
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{ "steps": [{ "op": "initialize" }, { "op": "newSession" }, { "op": "promptExpectError", "text": "blocked" }] }
|
||||
2
packages/support/acp-snapshot/tests/fixtures/suite/blocked-log/session.jsonl
vendored
Normal file
2
packages/support/acp-snapshot/tests/fixtures/suite/blocked-log/session.jsonl
vendored
Normal file
@@ -0,0 +1,2 @@
|
||||
{"type":"session","id":"99999999-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/blocked-cwd"}
|
||||
{"type":"hook/result","seq":1,"time":13,"data":{"decision":"block","durationMs":99}}
|
||||
4
packages/support/acp-snapshot/tests/fixtures/suite/blocked-log/stdout.golden.jsonl
vendored
Normal file
4
packages/support/acp-snapshot/tests/fixtures/suite/blocked-log/stdout.golden.jsonl
vendored
Normal file
@@ -0,0 +1,4 @@
|
||||
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"loadSession":false}}}
|
||||
{"jsonrpc":"2.0","id":2,"result":{"sessionId":"{{sessionId}}"}}
|
||||
{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"{{sessionId}}","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"thinking about it"}}}}
|
||||
{"jsonrpc":"2.0","id":3,"error":{"code":-32603,"message":"model exploded"}}
|
||||
1
packages/support/acp-snapshot/tests/fixtures/suite/no-model/behavior.json
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/suite/no-model/behavior.json
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{}
|
||||
1
packages/support/acp-snapshot/tests/fixtures/suite/no-model/input.json
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/suite/no-model/input.json
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{ "steps": [{ "op": "initialize" }] }
|
||||
1
packages/support/acp-snapshot/tests/fixtures/suite/no-model/session.jsonl
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/suite/no-model/session.jsonl
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{"type":"session","id":"{{sessionId}}","createdAt":0,"cwd":"{{cwd}}"}
|
||||
1
packages/support/acp-snapshot/tests/fixtures/suite/no-model/stdout.golden.jsonl
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/suite/no-model/stdout.golden.jsonl
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"loadSession":false}}}
|
||||
11
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/behavior.json
vendored
Normal file
11
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/behavior.json
vendored
Normal file
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"prompt": "respond",
|
||||
"logs": [{
|
||||
"file": "b/main.jsonl",
|
||||
"lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 100, "cwd": "{{CWD}}" },
|
||||
{ "type": "request/header", "seq": 0, "time": 100, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } },
|
||||
{ "type": "turn/start", "seq": 1, "time": 100, "data": { "turn": 1 } }
|
||||
]
|
||||
}]
|
||||
}
|
||||
1
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/input.json
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/input.json
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{ "steps": [{ "op": "initialize" }, { "op": "newSession" }, { "op": "prompt", "text": "pin" }] }
|
||||
3
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/session.jsonl
vendored
Normal file
3
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/session.jsonl
vendored
Normal file
@@ -0,0 +1,3 @@
|
||||
{"type":"session","id":"12121212-3434-4545-8686-787878787878","createdAt":7,"cwd":"/rec/pin-cwd"}
|
||||
{"type":"request/header","seq":0,"time":7,"data":{"header":{"config":{"model":"fake"},"system":"SYS PROMPT","tools":[{"name":"t1","description":"D1","parameters":{"type":"object"}}]},"reason":"initial"}}
|
||||
{"type":"turn/start","seq":1,"time":7,"data":{"turn":1}}
|
||||
4
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/stdout.golden.jsonl
vendored
Normal file
4
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/stdout.golden.jsonl
vendored
Normal file
@@ -0,0 +1,4 @@
|
||||
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"loadSession":false}}}
|
||||
{"jsonrpc":"2.0","id":2,"result":{"sessionId":"{{sessionId}}"}}
|
||||
{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"{{sessionId}}","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"thinking about it"}}}}
|
||||
{"jsonrpc":"2.0","id":3,"result":{"stopReason":"end_turn"}}
|
||||
15
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/behavior.json
vendored
Normal file
15
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/behavior.json
vendored
Normal file
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"prompt": "respond",
|
||||
"echoWorkspace": true,
|
||||
"logs": [
|
||||
{ "file": "b/parent.jsonl", "lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 200, "cwd": "{{CWD}}" },
|
||||
{ "type": "request/header", "seq": 0, "time": 5, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } },
|
||||
{ "type": "assistant/chunk", "seq": 1, "time": 5, "data": { "turn": 1, "step": 1, "chunk": { "type": "text-delta", "index": 0, "text": "hi" } } }
|
||||
]},
|
||||
{ "file": "b/child.jsonl", "lines": [
|
||||
{ "type": "session", "id": "eeeeeeee-1111-4222-8333-444444444444", "createdAt": 300, "cwd": "{{CWD}}", "parentSession": "{{SID}}" },
|
||||
{ "type": "request/header", "seq": 0, "time": 6, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
|
||||
]}
|
||||
]
|
||||
}
|
||||
1
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/input.json
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/input.json
vendored
Normal file
@@ -0,0 +1 @@
|
||||
{ "steps": [{ "op": "initialize" }, { "op": "newSession" }, { "op": "prompt", "text": "plain" }] }
|
||||
2
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/session.1.jsonl
vendored
Normal file
2
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/session.1.jsonl
vendored
Normal file
@@ -0,0 +1,2 @@
|
||||
{"type":"session","id":"eeeeeeee-1111-4222-8333-444444444444","createdAt":12,"cwd":"/rec/plain-cwd","parentSession":"56565656-7878-4989-8a9a-9b9b9b9b9b9b"}
|
||||
{"type":"request/header","seq":0,"time":12,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
3
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/session.jsonl
vendored
Normal file
3
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/session.jsonl
vendored
Normal file
@@ -0,0 +1,3 @@
|
||||
{"type":"session","id":"56565656-7878-4989-8a9a-9b9b9b9b9b9b","createdAt":11,"cwd":"/rec/plain-cwd"}
|
||||
{"type":"request/header","seq":0,"time":11,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
{"type":"assistant/chunk","seq":1,"time":11,"data":{"turn":1,"step":1,"chunk":{"type":"text-delta","index":0,"text":"hi"}}}
|
||||
5
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/stdout.golden.jsonl
vendored
Normal file
5
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/stdout.golden.jsonl
vendored
Normal file
@@ -0,0 +1,5 @@
|
||||
{"jsonrpc":"2.0","id":1,"result":{"protocolVersion":1,"agentCapabilities":{"loadSession":false}}}
|
||||
{"jsonrpc":"2.0","id":2,"result":{"sessionId":"{{sessionId}}"}}
|
||||
{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"{{sessionId}}","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"thinking about it"}}}}
|
||||
{"jsonrpc":"2.0","method":"session/update","params":{"sessionId":"{{sessionId}}","update":{"sessionUpdate":"agent_message_chunk","content":{"type":"text","text":"workspace:seed.txt"}}}}
|
||||
{"jsonrpc":"2.0","id":3,"result":{"stopReason":"end_turn"}}
|
||||
1
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/workspace/seed.txt
vendored
Normal file
1
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/workspace/seed.txt
vendored
Normal file
@@ -0,0 +1 @@
|
||||
seeded
|
||||
272
packages/support/acp-snapshot/tests/harness.spec.ts
Normal file
272
packages/support/acp-snapshot/tests/harness.spec.ts
Normal file
@@ -0,0 +1,272 @@
|
||||
import { mkdtemp, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { delimiter, join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { afterAll, describe, expect, it } from 'vitest'
|
||||
import { runScenario, type AgentUnderTest, type InputStep } from '../src/harness.ts'
|
||||
|
||||
/**
|
||||
* Unit tests for the subprocess harness, driven through the REAL spawn path
|
||||
* (tsx loader, temp cwd, env plumbing) against the scripted fake ACP bin in
|
||||
* ./fixtures/fake-acp-agent.ts. Each case writes a `behavior.json` next to a
|
||||
* throwaway fixture path; the fake bin echoes observable facts (env, seeded
|
||||
* workspace, permission outcomes) into `agent_message_chunk` text, so the
|
||||
* assertions read plain `rawStdout`.
|
||||
*/
|
||||
|
||||
const AGENT: AgentUnderTest = {
|
||||
binScript: fileURLToPath(new URL('./fixtures/fake-acp-agent.ts', import.meta.url)),
|
||||
// The fake bin ignores its config argv; any real path documents the shape.
|
||||
configPath: fileURLToPath(new URL('./fixtures/fake-acp-agent.ts', import.meta.url)),
|
||||
tsconfigPath: fileURLToPath(new URL('../../../../tsconfig.json', import.meta.url)),
|
||||
}
|
||||
|
||||
/** Temp scenario dirs to drop after the suite. */
|
||||
const tempDirs: string[] = []
|
||||
afterAll(async () => {
|
||||
for (const dir of tempDirs) await rm(dir, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
/** Write a behavior.json into a fresh temp dir; return the sibling fixture path the harness points the bin at. */
|
||||
async function scenario(behavior: object): Promise<{ dir: string; fixtureFile: string }> {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'acp-snap-spec-'))
|
||||
tempDirs.push(dir)
|
||||
await writeFile(join(dir, 'behavior.json'), JSON.stringify(behavior))
|
||||
return { dir, fixtureFile: join(dir, 'session.jsonl') }
|
||||
}
|
||||
|
||||
const boot: InputStep[] = [{ op: 'initialize' }, { op: 'newSession' }]
|
||||
|
||||
describe('runScenario', () => {
|
||||
it('drives a full turn: initialize (terminal caps), session, prompt, permission stub, harvest', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({
|
||||
permissionProbe: true,
|
||||
logs: [{
|
||||
file: 'bucket/main.jsonl',
|
||||
lines: [
|
||||
{ type: 'session', id: '{{SID}}', createdAt: 42, cwd: '{{CWD}}' },
|
||||
{ type: 'turn/start', seq: 1, time: 9, data: { turn: 1 } },
|
||||
],
|
||||
}],
|
||||
})
|
||||
const result = await runScenario(
|
||||
{ steps: [{ op: 'initialize', terminalOutput: true }, { op: 'newSession' }, { op: 'prompt', text: 'go' }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)
|
||||
expect(result.sessionId).toBeDefined()
|
||||
// The harness's client answers a permission request with `cancelled`; the
|
||||
// fake bin echoes the outcome it received back as a chunk.
|
||||
expect(result.rawStdout).toContain('permission:{\\"outcome\\":\\"cancelled\\"}')
|
||||
expect(result.sessionLogs).toHaveLength(1)
|
||||
expect(result.sessionLogs[0]?.id).toBe(result.sessionId)
|
||||
expect(result.sessionLogs[0]?.createdAt).toBe(42)
|
||||
expect(result.sessionLogs[0]?.content).toContain('turn/start')
|
||||
// The harvested log embeds the run's REAL temp cwd (template-substituted).
|
||||
expect(result.sessionLogs[0]?.content).toContain(result.cwd)
|
||||
})
|
||||
|
||||
it('forwards override/child fixture paths into the child env and captures stderr', { timeout: 20_000 }, async () => {
|
||||
const { dir, fixtureFile } = await scenario({ echoEnv: true, stderrNote: 'fake bin booted' })
|
||||
const childFiles = [join(dir, 'session.1.jsonl'), join(dir, 'session.2.jsonl')]
|
||||
const result = await runScenario(
|
||||
{ steps: [...boot, { op: 'prompt', text: 'env?' }] },
|
||||
{
|
||||
agent: AGENT,
|
||||
mode: 'replay',
|
||||
fixtureFile,
|
||||
overrideFile: join(dir, 'replay.override.json'),
|
||||
childFiles,
|
||||
// A workspaceDir that does not exist is skipped, not an error.
|
||||
workspaceDir: join(dir, 'no-such-workspace'),
|
||||
},
|
||||
)
|
||||
expect(result.stderr).toContain('fake bin booted')
|
||||
expect(result.rawStdout).toContain('replay.override.json')
|
||||
// Child paths ride one env var, joined with the platform delimiter.
|
||||
expect(result.rawStdout).toContain(JSON.stringify(childFiles.join(delimiter)).slice(1, -1))
|
||||
})
|
||||
|
||||
it('seeds the workspace dir into the temp cwd before the run', { timeout: 20_000 }, async () => {
|
||||
const { dir, fixtureFile } = await scenario({ echoWorkspace: true })
|
||||
const workspaceDir = join(dir, 'workspace')
|
||||
await writeFile(join(dir, 'behavior.json'), JSON.stringify({ echoWorkspace: true }))
|
||||
const { mkdir } = await import('node:fs/promises')
|
||||
await mkdir(workspaceDir, { recursive: true })
|
||||
await writeFile(join(workspaceDir, 'seeded.txt'), 'hello')
|
||||
const result = await runScenario(
|
||||
{ steps: [...boot, { op: 'prompt', text: 'ls' }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile, workspaceDir },
|
||||
)
|
||||
expect(result.rawStdout).toContain('workspace:seeded.txt')
|
||||
})
|
||||
|
||||
it('promptAndCancel waits for the streamed chunk, cancels, and settles the prompt', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({ prompt: 'hang-until-cancel' })
|
||||
const result = await runScenario(
|
||||
{ steps: [...boot, { op: 'promptAndCancel', text: 'hang' }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)
|
||||
expect(result.rawStdout).toContain('"stopReason":"cancelled"')
|
||||
// The streamed chunk deterministically precedes the cancelled response.
|
||||
expect(result.rawStdout.indexOf('thinking about it')).toBeLessThan(result.rawStdout.indexOf('cancelled'))
|
||||
})
|
||||
|
||||
it('promptExpectError swallows a model-error response as the expected outcome', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({ prompt: 'error' })
|
||||
const result = await runScenario(
|
||||
{ steps: [...boot, { op: 'promptExpectError', text: 'boom' }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)
|
||||
expect(result.rawStdout).toContain('model exploded')
|
||||
})
|
||||
|
||||
it('promptExpectError throws when the prompt unexpectedly succeeds (and teardown kills the live child)', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({ prompt: 'respond' })
|
||||
await expect(runScenario(
|
||||
{ steps: [...boot, { op: 'promptExpectError', text: 'fine' }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)).rejects.toThrow(/expected the prompt to fail/)
|
||||
})
|
||||
|
||||
it('newSessionExpectError swallows the rejection, with and without extra dirs', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({ rejectExtraDirs: true })
|
||||
const result = await runScenario(
|
||||
{ steps: [{ op: 'initialize' }, { op: 'newSessionExpectError', additionalDirectories: ['/elsewhere'] }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)
|
||||
// No session was created, so no id and no logs.
|
||||
expect(result.sessionId).toBeUndefined()
|
||||
expect(result.sessionLogs).toHaveLength(0)
|
||||
|
||||
const rejectAll = await scenario({ rejectNewSession: true })
|
||||
const second = await runScenario(
|
||||
{ steps: [{ op: 'initialize' }, { op: 'newSessionExpectError' }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile: rejectAll.fixtureFile },
|
||||
)
|
||||
expect(second.rawStdout).toContain('unsupported workspace scope')
|
||||
})
|
||||
|
||||
it('newSessionExpectError throws when session/new unexpectedly succeeds', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({})
|
||||
await expect(runScenario(
|
||||
{ steps: [{ op: 'initialize' }, { op: 'newSessionExpectError' }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)).rejects.toThrow(/expected session\/new to be rejected/)
|
||||
})
|
||||
|
||||
it('a plain cancel step is forwarded (and ignored by an idle agent)', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({})
|
||||
const result = await runScenario(
|
||||
{ steps: [...boot, { op: 'cancel' }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)
|
||||
expect(result.sessionId).toBeDefined()
|
||||
})
|
||||
|
||||
it.each([
|
||||
[{ op: 'prompt', text: 'x' }, /prompt before newSession/],
|
||||
[{ op: 'promptExpectError', text: 'x' }, /promptExpectError before newSession/],
|
||||
[{ op: 'promptAndCancel', text: 'x' }, /promptAndCancel before newSession/],
|
||||
[{ op: 'cancel' }, /cancel before newSession/],
|
||||
] as [InputStep, RegExp][])('rejects %j before newSession', { timeout: 20_000 }, async (step, message) => {
|
||||
const { fixtureFile } = await scenario({})
|
||||
await expect(runScenario(
|
||||
{ steps: [{ op: 'initialize' }, step] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)).rejects.toThrow(message)
|
||||
})
|
||||
|
||||
it('rejects an unknown input op', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({})
|
||||
const bogus = { op: 'reticulate' } as unknown as InputStep
|
||||
await expect(runScenario(
|
||||
{ steps: [bogus] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)).rejects.toThrow(/unknown input op/)
|
||||
})
|
||||
|
||||
it('harvests all logs primary-first, children by createdAt then id, skipping filesystem noise', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({
|
||||
strayRootFile: true,
|
||||
strayBucketFile: true,
|
||||
logs: [
|
||||
// File names chosen so readdir feeds the sort children-first AND
|
||||
// parent-in-the-middle: the comparator then sees a parent on both
|
||||
// sides of a pair, plus the same-createdAt (localeCompare) tiebreak.
|
||||
{ file: 'b1/aa-child-c.jsonl', lines: [{ type: 'session', id: 'cccccccc-0000-4000-8000-000000000000', createdAt: 500, parentSession: '{{SID}}' }] },
|
||||
{ file: 'b1/bb-parent.jsonl', lines: [{ type: 'session', id: '{{SID}}', createdAt: 900 }] },
|
||||
{ file: 'b1/cc-child-a.jsonl', lines: [{ type: 'session', id: 'aaaaaaaa-0000-4000-8000-000000000000', createdAt: 500, parentSession: '{{SID}}' }] },
|
||||
// Missing id/createdAt fall back to ''/0; earliest child by createdAt.
|
||||
{ file: 'b2/orphan-fields.jsonl', lines: [{ type: 'session', parentSession: '{{SID}}' }] },
|
||||
],
|
||||
})
|
||||
const result = await runScenario(
|
||||
{ steps: [...boot, { op: 'prompt', text: 'go' }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)
|
||||
expect(result.sessionLogs.map(l => [l.id, l.createdAt])).toEqual([
|
||||
[result.sessionId, 900],
|
||||
['', 0],
|
||||
['aaaaaaaa-0000-4000-8000-000000000000', 500],
|
||||
['cccccccc-0000-4000-8000-000000000000', 500],
|
||||
])
|
||||
expect(result.sessionLogs[1]?.parentSession).toBe(result.sessionId)
|
||||
})
|
||||
|
||||
it('treats an empty log file as a header-less primary with default fields', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({ logs: [{ file: 'b/empty.jsonl', lines: [] }] })
|
||||
const result = await runScenario(
|
||||
{ steps: boot },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)
|
||||
expect(result.sessionLogs.map(l => [l.id, l.createdAt, l.parentSession])).toEqual([['', 0, undefined]])
|
||||
})
|
||||
|
||||
it('yields no logs when the sessions root vanished', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({ deleteSessionsRoot: true })
|
||||
const result = await runScenario(
|
||||
{ steps: boot },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)
|
||||
expect(result.sessionLogs).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('answers permission requests from the scripted queue by option kind, falling back to cancelled', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({ permissionProbe: true })
|
||||
// Two prompts → two permission round-trips; one scripted answer, so the
|
||||
// second request exercises the exhausted-queue fallback.
|
||||
const result = await runScenario(
|
||||
{
|
||||
steps: [...boot, { op: 'prompt', text: 'one' }, { op: 'prompt', text: 'two' }],
|
||||
permissionAnswers: [{ kind: 'allow_once' }],
|
||||
},
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)
|
||||
const first = result.rawStdout.indexOf('permission:{\\"outcome\\":\\"selected\\",\\"optionId\\":\\"opt-allow\\"}')
|
||||
const second = result.rawStdout.indexOf('permission:{\\"outcome\\":\\"cancelled\\"}')
|
||||
expect(first).toBeGreaterThanOrEqual(0)
|
||||
expect(second).toBeGreaterThan(first)
|
||||
})
|
||||
|
||||
it('selects a non-first offered option by kind', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({ permissionProbe: true })
|
||||
const result = await runScenario(
|
||||
{ steps: [...boot, { op: 'prompt', text: 'deny it' }], permissionAnswers: [{ kind: 'reject_once' }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)
|
||||
expect(result.rawStdout).toContain('permission:{\\"outcome\\":\\"selected\\",\\"optionId\\":\\"opt-reject\\"}')
|
||||
})
|
||||
|
||||
it('rejects the run on a scripted permission kind the agent never offered', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({ permissionProbe: true })
|
||||
// The fake bin offers allow_once/reject_once; scripting allow_always is a
|
||||
// scenario bug. The agent is answered `cancelled` (it must not be able to
|
||||
// absorb the bug as an error-means-denial), and the RUN fails: a callback
|
||||
// throw would only reach the agent as a JSON-RPC error response, letting
|
||||
// a tolerant agent carry on and the scenario pass — or record.
|
||||
await expect(runScenario(
|
||||
{ steps: [...boot, { op: 'prompt', text: 'impossible click' }], permissionAnswers: [{ kind: 'allow_always' }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)).rejects.toThrow(/allow_always not among the offered options \[allow_once, reject_once\]/)
|
||||
})
|
||||
})
|
||||
262
packages/support/acp-snapshot/tests/normalize.spec.ts
Normal file
262
packages/support/acp-snapshot/tests/normalize.spec.ts
Normal file
@@ -0,0 +1,262 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { type NormalizeContext, normalizeSessionLog, normalizeStdout, scrubRequestHeaders } from '../src/normalize.ts'
|
||||
|
||||
/**
|
||||
* Unit tests for the pure snapshot normalizers. Live as a *.spec.ts (runs in
|
||||
* the default unit gate) and import the normalizers directly.
|
||||
*/
|
||||
|
||||
const ctx: NormalizeContext = {
|
||||
sessionIds: ['11111111-2222-3333-4444-555555555555'],
|
||||
cwd: '/tmp/acp-snap-cwd-abc123',
|
||||
}
|
||||
|
||||
describe('normalizeStdout', () => {
|
||||
it('rewrites JSON-RPC ids to a stable first-seen sequence', () => {
|
||||
const raw = [
|
||||
JSON.stringify({ jsonrpc: '2.0', id: 42, method: 'initialize' }),
|
||||
JSON.stringify({ jsonrpc: '2.0', id: 42, result: {} }),
|
||||
JSON.stringify({ jsonrpc: '2.0', id: 99, method: 'session/new' }),
|
||||
].join('\n')
|
||||
const out = normalizeStdout(raw, ctx)
|
||||
expect(out).toContain('"id":1')
|
||||
expect(out).toContain('"id":2')
|
||||
expect(out).not.toContain('42')
|
||||
expect(out).not.toContain('99')
|
||||
})
|
||||
|
||||
it('scrubs the cwd and session id anywhere they appear', () => {
|
||||
const raw = JSON.stringify({
|
||||
jsonrpc: '2.0', method: 'session/update',
|
||||
params: { sessionId: ctx.sessionIds[0], cwd: ctx.cwd, note: `at ${ctx.cwd}/x` },
|
||||
})
|
||||
const out = normalizeStdout(raw, ctx)
|
||||
expect(out).toContain('{{sessionId}}')
|
||||
expect(out).toContain('{{cwd}}')
|
||||
expect(out).not.toContain(ctx.cwd)
|
||||
expect(out).not.toContain(ctx.sessionIds[0] as string)
|
||||
})
|
||||
|
||||
it('scrubs a stray UUID not in the known list', () => {
|
||||
const raw = JSON.stringify({ jsonrpc: '2.0', method: 'x', params: { id: 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee' } })
|
||||
expect(normalizeStdout(raw, ctx)).toContain('{{sessionId}}')
|
||||
})
|
||||
|
||||
it('leaves notification frames without an id untouched in id-space', () => {
|
||||
const raw = JSON.stringify({ jsonrpc: '2.0', method: 'session/update', params: {} })
|
||||
const out = normalizeStdout(raw, ctx)
|
||||
expect(out).not.toContain('"id"')
|
||||
})
|
||||
|
||||
it('throws on a non-JSON stdout line (the purity check)', () => {
|
||||
const raw = `${JSON.stringify({ jsonrpc: '2.0', id: 1 })}\noops a log leaked\n`
|
||||
expect(() => normalizeStdout(raw, ctx)).toThrow()
|
||||
})
|
||||
|
||||
it('ignores blank lines', () => {
|
||||
const raw = `\n${JSON.stringify({ jsonrpc: '2.0', id: 1, method: 'm' })}\n\n`
|
||||
expect(() => normalizeStdout(raw, ctx)).not.toThrow()
|
||||
})
|
||||
})
|
||||
|
||||
describe('normalizeSessionLog', () => {
|
||||
const header = (over: object) => JSON.stringify({ type: 'session', version: 0, id: 's', createdAt: 123, ...over })
|
||||
const event = (over: object) => JSON.stringify({ type: 'turn/start', seq: 1, time: 999, data: { turn: 1 }, ...over })
|
||||
|
||||
it('zeroes the header createdAt', () => {
|
||||
const out = normalizeSessionLog(`${header({})}\n`, ctx)
|
||||
expect(out).toContain('"createdAt":0')
|
||||
expect(out).not.toContain('123')
|
||||
})
|
||||
|
||||
it('zeroes each event time but keeps seq', () => {
|
||||
const out = normalizeSessionLog(`${header({})}\n${event({ seq: 7, time: 999 })}\n`, ctx)
|
||||
expect(out).toContain('"time":0')
|
||||
expect(out).toContain('"seq":7') // seq is deterministic — NOT scrubbed
|
||||
expect(out).not.toContain('999')
|
||||
})
|
||||
|
||||
it('scrubs cwd and session id deep inside event data', () => {
|
||||
const ev = JSON.stringify({
|
||||
type: 'tool/result', seq: 2, time: 5,
|
||||
data: { content: [{ type: 'text', text: `wrote ${ctx.cwd}/proof.txt` }] },
|
||||
})
|
||||
const out = normalizeSessionLog(`${header({ cwd: ctx.cwd })}\n${ev}\n`, ctx)
|
||||
expect(out).toContain('{{cwd}}')
|
||||
expect(out).not.toContain(ctx.cwd)
|
||||
})
|
||||
|
||||
it('scrubs the session id in the header', () => {
|
||||
const out = normalizeSessionLog(`${header({ id: ctx.sessionIds[0] })}\n`, ctx)
|
||||
expect(out).toContain('{{sessionId}}')
|
||||
})
|
||||
|
||||
it('zeroes a hook/result durationMs (run-to-run noise) but keeps its decision', () => {
|
||||
const ev = JSON.stringify({
|
||||
type: 'hook/result', seq: 2, time: 5,
|
||||
data: { turn: 1, point: 'UserPromptSubmit', handlerId: 'h', decision: 'block', exitCode: 2, durationMs: 37 },
|
||||
})
|
||||
const out = normalizeSessionLog(`${header({})}\n${ev}\n`, ctx)
|
||||
expect(out).toContain('"durationMs":0')
|
||||
expect(out).not.toContain('37')
|
||||
expect(out).toContain('"decision":"block"') // the decision is the behavior — kept
|
||||
})
|
||||
|
||||
it('leaves a non-hook event durationMs untouched (only hook/result is scrubbed)', () => {
|
||||
const ev = JSON.stringify({ type: 'tool/result', seq: 2, time: 5, data: { durationMs: 88 } })
|
||||
const out = normalizeSessionLog(`${header({})}\n${ev}\n`, ctx)
|
||||
expect(out).toContain('"durationMs":88')
|
||||
})
|
||||
|
||||
it('tolerates records missing the volatile fields it would zero', () => {
|
||||
const bareHeader = JSON.stringify({ type: 'session', id: 's' })
|
||||
const timeless = JSON.stringify({ type: 'note', seq: 1 })
|
||||
const bareHook = JSON.stringify({ type: 'hook/result', seq: 2, time: 5, data: { decision: 'allow' } })
|
||||
const nullDataHook = JSON.stringify({ type: 'hook/result', seq: 3, time: 6, data: null })
|
||||
const out = normalizeSessionLog(`${bareHeader}\n${timeless}\n${bareHook}\n${nullDataHook}\n`, ctx)
|
||||
expect(out).toContain('"type":"note","seq":1')
|
||||
expect(out).toContain('"decision":"allow"')
|
||||
expect(out).not.toContain('durationMs')
|
||||
})
|
||||
})
|
||||
|
||||
describe('scrubRequestHeaders', () => {
|
||||
const headerLine = JSON.stringify({ type: 'session', version: 0, id: 's', createdAt: 1, cwd: '/w' })
|
||||
const headerEvent = (header: object) =>
|
||||
JSON.stringify({ type: 'request/header', seq: 3, time: 9, data: { header, reason: 'initial' } })
|
||||
|
||||
it('replaces header system and tools with tokens, keeping config and reason', () => {
|
||||
const ev = headerEvent({
|
||||
config: { model: 'm' },
|
||||
system: 'You are an agent.\nBe brief.',
|
||||
tools: [{ name: 'read', description: 'Read a file.', parameters: { type: 'object' } }],
|
||||
})
|
||||
const out = scrubRequestHeaders(`${headerLine}\n${ev}\n`)
|
||||
expect(out).toContain('"system":"{{system}}"')
|
||||
expect(out).toContain('"tools":"{{tools}}"')
|
||||
expect(out).toContain('"config":{"model":"m"}')
|
||||
expect(out).toContain('"reason":"initial"')
|
||||
expect(out).not.toContain('You are an agent')
|
||||
expect(out).not.toContain('Read a file')
|
||||
})
|
||||
|
||||
it('keeps an absent system/tools absent (presence is behavior)', () => {
|
||||
const out = scrubRequestHeaders(`${headerLine}\n${headerEvent({ config: { model: 'm' } })}\n`)
|
||||
expect(out).not.toContain('{{system}}')
|
||||
expect(out).not.toContain('{{tools}}')
|
||||
})
|
||||
|
||||
it('scrubs a header carrying only one of system/tools, leaving the other absent', () => {
|
||||
const systemOnly = scrubRequestHeaders(`${headerLine}\n${headerEvent({ system: 'secret prompt' })}\n`)
|
||||
expect(systemOnly).toContain('"system":"{{system}}"')
|
||||
expect(systemOnly).not.toContain('{{tools}}')
|
||||
const toolsOnly = scrubRequestHeaders(`${headerLine}\n${headerEvent({ tools: [{ name: 't' }] })}\n`)
|
||||
expect(toolsOnly).toContain('"tools":"{{tools}}"')
|
||||
expect(toolsOnly).not.toContain('{{system}}')
|
||||
})
|
||||
|
||||
it('scrubs the header session prefix to one token per message, keeping the count', () => {
|
||||
const ev = headerEvent({
|
||||
config: { model: 'm' },
|
||||
messagePrefix: [
|
||||
{ role: 'user', content: [{ type: 'text', text: 'workspace AGENTS digest' }] },
|
||||
{ role: 'user', content: [{ type: 'text', text: 'skills catalog' }] },
|
||||
],
|
||||
})
|
||||
const out = scrubRequestHeaders(`${headerLine}\n${ev}\n`)
|
||||
expect(out).toContain('"messagePrefix":["{{messagePrefix}}","{{messagePrefix}}"]')
|
||||
expect(out).not.toContain('AGENTS digest')
|
||||
expect(out).not.toContain('skills catalog')
|
||||
// Absence stays absent — a prefix-less header gains no token…
|
||||
expect(scrubRequestHeaders(`${headerLine}\n${headerEvent({ system: 's' })}\n`)).not.toContain('{{messagePrefix}}')
|
||||
// …and a non-array shape passes through untouched.
|
||||
const odd = JSON.stringify({ type: 'request/header', seq: 4, time: 9, data: { header: { config: { model: 'm' }, messagePrefix: 'weird' }, reason: 'initial' } })
|
||||
expect(scrubRequestHeaders(`${headerLine}\n${odd}\n`)).toContain('"messagePrefix":"weird"')
|
||||
})
|
||||
|
||||
it('scrubs a header-delta prefix replacement to one token per message', () => {
|
||||
const delta = JSON.stringify({
|
||||
type: 'request/header-delta', seq: 8, time: 9,
|
||||
data: { messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'leaked opener' }] }] },
|
||||
})
|
||||
const out = scrubRequestHeaders(`${headerLine}\n${delta}\n`)
|
||||
expect(out).toContain('"messagePrefix":["{{messagePrefix}}"]')
|
||||
expect(out).not.toContain('leaked opener')
|
||||
// The empty-array transition-to-absence stays a structural fact.
|
||||
const toNone = JSON.stringify({ type: 'request/header-delta', seq: 9, time: 9, data: { messagePrefix: [] } })
|
||||
expect(scrubRequestHeaders(`${headerLine}\n${toNone}\n`)).toContain('"messagePrefix":[]')
|
||||
})
|
||||
|
||||
it('leaves a delta with no scrubbable payload byte-identical (config-only, or non-array shapes)', () => {
|
||||
const configOnly = JSON.stringify({ type: 'request/header-delta', seq: 8, time: 9, data: { config: { model: 'm2' } } })
|
||||
const oddShapes = JSON.stringify({ type: 'request/header-delta', seq: 9, time: 9, data: { system: { insert: 'not-an-array' }, tools: null } })
|
||||
const headerless = JSON.stringify({ type: 'request/header', seq: 10, time: 9, data: { reason: 'initial' } })
|
||||
const nullData = JSON.stringify({ type: 'request/header', seq: 11, time: 9, data: null })
|
||||
const raw = `${headerLine}\n${configOnly}\n${oddShapes}\n${headerless}\n${nullData}\n`
|
||||
expect(scrubRequestHeaders(raw)).toBe(raw)
|
||||
})
|
||||
|
||||
it('scrubs a one-sided tools delta and passes non-object schema entries through', () => {
|
||||
const addedOnly = JSON.stringify({
|
||||
type: 'request/header-delta', seq: 8, time: 9,
|
||||
data: { tools: { added: [null, 'weird', { name: 'x', description: 'D' }] } },
|
||||
})
|
||||
const out = scrubRequestHeaders(`${headerLine}\n${addedOnly}\n`)
|
||||
// Non-object entries survive untouched; the object entry keeps only name.
|
||||
expect(out).toContain('"added":[null,"weird",{"name":"x","description":"{{tools}}"}]')
|
||||
const changedOnly = JSON.stringify({
|
||||
type: 'request/header-delta', seq: 8, time: 9,
|
||||
data: { tools: { changed: [{ name: 'y', parameters: {} }] } },
|
||||
})
|
||||
expect(scrubRequestHeaders(`${headerLine}\n${changedOnly}\n`))
|
||||
.toContain('"changed":[{"name":"y","parameters":"{{tools}}"}]')
|
||||
})
|
||||
|
||||
it('scrubs a header-delta system payload but keeps its line positions and arity', () => {
|
||||
const delta = JSON.stringify({
|
||||
type: 'request/header-delta', seq: 8, time: 9,
|
||||
data: { system: { keepStart: 1, keepEnd: 4, insert: ['leaked prompt line', 'second line'] }, config: { model: 'm2' } },
|
||||
})
|
||||
const out = scrubRequestHeaders(`${headerLine}\n${delta}\n`)
|
||||
// One token PER inserted line: the edit's position AND extent survive.
|
||||
expect(out).toContain('"insert":["{{system}}","{{system}}"]')
|
||||
expect(out).toContain('"keepStart":1')
|
||||
expect(out).toContain('"keepEnd":4')
|
||||
expect(out).toContain('"config":{"model":"m2"}')
|
||||
expect(out).not.toContain('leaked prompt line')
|
||||
expect(out).not.toContain('{{tools}}') // no tools delta → none invented
|
||||
})
|
||||
|
||||
it('scrubs a header-delta tools payload but keeps the added/removed/changed names', () => {
|
||||
const delta = JSON.stringify({
|
||||
type: 'request/header-delta', seq: 8, time: 9,
|
||||
data: {
|
||||
tools: {
|
||||
added: [{ name: 'grep', description: 'Search files.', parameters: { type: 'object' } }],
|
||||
removed: ['bash_kill'],
|
||||
changed: [{ name: 'read', description: 'Read v2.', parameters: { type: 'object' } }],
|
||||
},
|
||||
},
|
||||
})
|
||||
const out = scrubRequestHeaders(`${headerLine}\n${delta}\n`)
|
||||
// WHICH tools changed is behavior and survives; their bulk does not.
|
||||
expect(out).toContain('"added":[{"name":"grep","description":"{{tools}}","parameters":"{{tools}}"}]')
|
||||
expect(out).toContain('"removed":["bash_kill"]')
|
||||
expect(out).toContain('"changed":[{"name":"read","description":"{{tools}}","parameters":"{{tools}}"}]')
|
||||
expect(out).not.toContain('Search files')
|
||||
expect(out).not.toContain('Read v2')
|
||||
})
|
||||
|
||||
it('passes every other line through byte-for-byte and is idempotent', () => {
|
||||
const other = JSON.stringify({ type: 'assistant/chunk', seq: 4, time: 9, data: { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'hi' } } })
|
||||
const delta = JSON.stringify({
|
||||
type: 'request/header-delta', seq: 8, time: 9,
|
||||
data: { system: { keepStart: 0, keepEnd: 0, insert: ['x'] }, tools: { added: [{ name: 't', description: 'd', parameters: {} }], removed: [], changed: [] } },
|
||||
})
|
||||
const raw = `${headerLine}\n${headerEvent({ config: { model: 'm' }, system: 's', tools: [] })}\n${delta}\n${other}\n`
|
||||
const once = scrubRequestHeaders(raw)
|
||||
expect(once.split('\n')[0]).toBe(headerLine)
|
||||
expect(once.split('\n')[3]).toBe(other)
|
||||
expect(scrubRequestHeaders(once)).toBe(once)
|
||||
})
|
||||
})
|
||||
145
packages/support/acp-snapshot/tests/suite.spec.ts
Normal file
145
packages/support/acp-snapshot/tests/suite.spec.ts
Normal file
@@ -0,0 +1,145 @@
|
||||
import { cpSync, mkdtempSync } from 'node:fs'
|
||||
import { rm } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { afterAll, describe, expect, it } from 'vitest'
|
||||
import { defineAcpSnapshotSuite, type Scenario } from '../src/index.ts'
|
||||
import { childFixturePaths, fixtureContext, headerDeltaCount, normalizedHeaders } from '../src/suite.ts'
|
||||
|
||||
/**
|
||||
* Unit tests for the suite factory, by running it: two synthetic suites over
|
||||
* the scripted fake ACP bin (./fixtures/fake-acp-agent.ts) register REAL
|
||||
* describe/it trees at collection time, so every factory path — golden and log
|
||||
* compares, the per-suite header pin and its uniformity guard, record-mode
|
||||
* fixture write-back, skip semantics, and the fixture guard block — executes
|
||||
* as an ordinary green test. The pure helpers get direct cases below.
|
||||
*
|
||||
* The replay suite runs against the committed fixtures in ./fixtures/suite.
|
||||
* The record suite runs against a TEMP COPY of ./fixtures/record-suite
|
||||
* (record mode writes session fixtures back into its snapshots dir; a run must
|
||||
* never touch the committed tree). To re-bootstrap the record tree's goldens
|
||||
* after changing the fake bin's output, run this spec once with
|
||||
* `ACP_SNAPSHOT_SPEC_BOOTSTRAP=1` (points the record suite at the committed
|
||||
* tree so vitest creates/updates the goldens and the write-back lands there),
|
||||
* then commit the result.
|
||||
*/
|
||||
|
||||
const AGENT = {
|
||||
binScript: fileURLToPath(new URL('./fixtures/fake-acp-agent.ts', import.meta.url)),
|
||||
configPath: fileURLToPath(new URL('./fixtures/fake-acp-agent.ts', import.meta.url)),
|
||||
tsconfigPath: fileURLToPath(new URL('../../../../tsconfig.json', import.meta.url)),
|
||||
}
|
||||
|
||||
const REPLAY_DIR = fileURLToPath(new URL('./fixtures/suite', import.meta.url))
|
||||
const RECORD_SRC = fileURLToPath(new URL('./fixtures/record-suite', import.meta.url))
|
||||
|
||||
const REPLAY_SCENARIOS: Scenario[] = [
|
||||
{ name: 'pin-turn', hasModelTurn: true, recorded: true, pinsHeader: true },
|
||||
{ name: 'plain-turn', hasModelTurn: true, recorded: true, childSessions: 1 },
|
||||
{ name: 'no-model', hasModelTurn: false, recorded: false },
|
||||
{ name: 'blocked-log', hasModelTurn: false, comparesLog: true, recorded: false },
|
||||
{ name: 'authored-error', hasModelTurn: true, recorded: false, overridden: true },
|
||||
]
|
||||
|
||||
const RECORD_SCENARIOS: Scenario[] = [
|
||||
{ name: 'rec-pin', hasModelTurn: true, recorded: true, pinsHeader: true },
|
||||
{ name: 'rec-child', hasModelTurn: true, recorded: true, childSessions: 1 },
|
||||
// recorded:false in record mode → registered but skipped (never re-recorded).
|
||||
{ name: 'rec-skip', hasModelTurn: true, recorded: false, overridden: true },
|
||||
]
|
||||
|
||||
// Record mode mutates its snapshots dir, so run it on a throwaway copy —
|
||||
// except under the documented bootstrap knob, which regenerates the committed
|
||||
// fixtures/goldens in place.
|
||||
const BOOTSTRAP = process.env.ACP_SNAPSHOT_SPEC_BOOTSTRAP === '1'
|
||||
const recordDir = BOOTSTRAP ? RECORD_SRC : mkdtempSync(join(tmpdir(), 'acp-snap-record-suite-'))
|
||||
if (!BOOTSTRAP) cpSync(RECORD_SRC, recordDir, { recursive: true })
|
||||
afterAll(async () => {
|
||||
if (!BOOTSTRAP) await rm(recordDir, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
describe('defineAcpSnapshotSuite: replay mode', () => {
|
||||
defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: REPLAY_DIR, scenarios: REPLAY_SCENARIOS, mode: 'replay' })
|
||||
})
|
||||
|
||||
// The record suite's tests run in registration order: rec-pin re-records the
|
||||
// pinned fixture FIRST, so rec-child's uniformity guard reads the fresh pin.
|
||||
describe('defineAcpSnapshotSuite: record mode', () => {
|
||||
defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: recordDir, scenarios: RECORD_SCENARIOS, mode: 'record' })
|
||||
})
|
||||
|
||||
describe('defineAcpSnapshotSuite: registration contract', () => {
|
||||
it('throws when no scenario pins the request-header content', () => {
|
||||
expect(() => {
|
||||
defineAcpSnapshotSuite({
|
||||
agent: AGENT,
|
||||
snapshotsDir: REPLAY_DIR,
|
||||
scenarios: [{ name: 'pinless', hasModelTurn: true, recorded: true }],
|
||||
mode: 'replay',
|
||||
})
|
||||
}).toThrow(/no scenario pins/)
|
||||
})
|
||||
})
|
||||
|
||||
describe('childFixturePaths', () => {
|
||||
it('yields one sibling path per child, 1-based', () => {
|
||||
expect(childFixturePaths('/snap/s', 2)).toEqual(['/snap/s/session.1.jsonl', '/snap/s/session.2.jsonl'])
|
||||
})
|
||||
|
||||
it('yields nothing for a single-session scenario', () => {
|
||||
expect(childFixturePaths('/snap/s', 0)).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe('fixtureContext', () => {
|
||||
it('reads the fixture header id and cwd', () => {
|
||||
const ctx = fixtureContext('{"type":"session","id":"abc","cwd":"/rec"}\n{"type":"turn/start"}\n')
|
||||
expect(ctx).toEqual({ sessionIds: ['abc'], cwd: '/rec' })
|
||||
})
|
||||
|
||||
it('yields no session ids for a header without a string id', () => {
|
||||
expect(fixtureContext('{"type":"session","cwd":"/rec"}\n').sessionIds).toEqual([])
|
||||
})
|
||||
|
||||
it('falls back to an impossible sentinel cwd (never the empty string)', () => {
|
||||
const ctx = fixtureContext('{"type":"session","id":"abc"}\n')
|
||||
expect(ctx.cwd).toBe('\0no-cwd\0')
|
||||
expect(ctx.cwd).not.toBe('')
|
||||
})
|
||||
|
||||
it('treats an empty fixture as an empty header', () => {
|
||||
expect(fixtureContext('')).toEqual({ sessionIds: [], cwd: '\0no-cwd\0' })
|
||||
})
|
||||
})
|
||||
|
||||
describe('normalizedHeaders', () => {
|
||||
const header = (system: string): string => JSON.stringify({
|
||||
type: 'request/header', seq: 0, time: 9, data: { header: { config: { model: 'm' }, system }, reason: 'initial' },
|
||||
})
|
||||
|
||||
it('extracts every request/header payload in log order, normalized', () => {
|
||||
const id = '11111111-2222-4333-8444-555555555555'
|
||||
const log = `${JSON.stringify({ type: 'session', id, createdAt: 5, cwd: '/w' })}\n${header('one')}\n`
|
||||
+ `${JSON.stringify({ type: 'turn/start', seq: 1, time: 9, data: { turn: 1 } })}\n${header('two')}\n`
|
||||
const headers = normalizedHeaders(log, { sessionIds: [id], cwd: '/w' })
|
||||
expect(headers).toEqual([
|
||||
{ config: { model: 'm' }, system: 'one' },
|
||||
{ config: { model: 'm' }, system: 'two' },
|
||||
])
|
||||
})
|
||||
|
||||
it('yields nothing for a log without header events', () => {
|
||||
const log = `${JSON.stringify({ type: 'session', id: 'a', createdAt: 5 })}\n`
|
||||
expect(normalizedHeaders(log, { sessionIds: [], cwd: '/w' })).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe('headerDeltaCount', () => {
|
||||
it('counts request/header-delta events, ignoring blanks and other lines', () => {
|
||||
const delta = JSON.stringify({ type: 'request/header-delta', seq: 2, time: 9, data: {} })
|
||||
const other = JSON.stringify({ type: 'request/header', seq: 0, time: 9, data: {} })
|
||||
expect(headerDeltaCount(`${other}\n\n${delta}\n${delta}\n`)).toBe(2)
|
||||
expect(headerDeltaCount(`${other}\n`)).toBe(0)
|
||||
})
|
||||
})
|
||||
11
packages/support/acp-snapshot/tsconfig.json
Normal file
11
packages/support/acp-snapshot/tsconfig.json
Normal file
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": []
|
||||
}
|
||||
@@ -367,8 +367,11 @@ export function apply(ctx: Context, config: Config = {}): void {
|
||||
// hand-built one-shot (compaction summarize) is unfrozen and skipped — must
|
||||
// be EXACTLY what the session log reconstructs:
|
||||
//
|
||||
// - messages: the derivation over the log prefix strictly before the
|
||||
// in-flight step's `step/start` (the reconstruction boundary). Compared
|
||||
// - messages: the folded header's session prefix (messagePrefix — the
|
||||
// `agent/session-prefix` product, logged on the header because no
|
||||
// session event carries it) followed by the
|
||||
// derivation over the log prefix strictly before the in-flight step's
|
||||
// `step/start` (the reconstruction boundary). The derivation is compared
|
||||
// against a FRESH Session built over that prefix — the same projection
|
||||
// code with zero shared state, so the live cache under test cannot vouch
|
||||
// for itself. Boundary-correct by construction: content appended after
|
||||
@@ -408,18 +411,22 @@ export function apply(ctx: Context, config: Config = {}): void {
|
||||
if (boundary === -1) {
|
||||
throw new InvariantError('a loop-built request with no step/start in its session log')
|
||||
}
|
||||
const rebuilt = new Session(SessionId(`${String(session.id)}-invariant-rebuild`), structuredClone(events.slice(0, boundary)))
|
||||
// JSON equality is sound here: both sides are structuredClones produced by
|
||||
// the same projection code path, so key insertion order matches when the
|
||||
// values do.
|
||||
if (JSON.stringify(options.messages) !== JSON.stringify(rebuilt.deriveMessages())) {
|
||||
throw new InvariantError(`llm request for session "${String(session.id)}" diverges from the boundary derivation (log-reconstruction desync)`)
|
||||
}
|
||||
|
||||
const header = foldRequestHeader(events)
|
||||
if (header === undefined) {
|
||||
throw new InvariantError('a loop-built request with no request/header event in its session log')
|
||||
}
|
||||
const rebuilt = new Session(SessionId(`${String(session.id)}-invariant-rebuild`), structuredClone(events.slice(0, boundary)))
|
||||
// The reconstruction equation: the folded header's session prefix, then
|
||||
// the boundary derivation — the loop
|
||||
// logs the header event BEFORE dispatch, so the fold already covers this
|
||||
// request's prefix. JSON equality is sound here: both sides are
|
||||
// structuredClones produced by the same projection/build code path, so key
|
||||
// insertion order matches when the values do.
|
||||
const expected = [...header.messagePrefix ?? [], ...rebuilt.deriveMessages()]
|
||||
if (JSON.stringify(options.messages) !== JSON.stringify(expected)) {
|
||||
throw new InvariantError(`llm request for session "${String(session.id)}" diverges from the boundary derivation (log-reconstruction desync)`)
|
||||
}
|
||||
|
||||
const headerMatches = options.model === header.config.model
|
||||
&& options.system === header.system
|
||||
&& options.temperature === header.config.temperature
|
||||
|
||||
@@ -707,6 +707,21 @@ describe('request-reconstruction cross-check (llm/stream)', () => {
|
||||
expect(() => { dispatch(ctx, options) }).not.toThrow()
|
||||
})
|
||||
|
||||
it('expects the folded header\'s session prefix ahead of the derivation (prefix + derived)', async () => {
|
||||
const { ctx, session, boundary } = await requestSetup()
|
||||
const prefix = { role: 'user' as const, content: [{ type: 'text' as const, text: '<system-reminder>catalog</system-reminder>' }] }
|
||||
session.append('request/header-delta', { messagePrefix: [prefix] })
|
||||
// The prefixed request matches the fold…
|
||||
const prefixed = Object.freeze({ model: 'm', messages: Object.freeze([prefix, ...boundary]), sessionId: session.id })
|
||||
expect(() => { dispatch(ctx, prefixed) }).not.toThrow()
|
||||
// …a request that DROPPED the logged prefix diverges…
|
||||
const bare = Object.freeze({ model: 'm', messages: Object.freeze([...boundary]), sessionId: session.id })
|
||||
expect(() => { dispatch(ctx, bare) }).toThrow(/diverges from the boundary derivation/)
|
||||
// …and so does one that misplaced it (prefix sent after the history).
|
||||
const misplaced = Object.freeze({ model: 'm', messages: Object.freeze([...boundary, prefix]), sessionId: session.id })
|
||||
expect(() => { dispatch(ctx, misplaced) }).toThrow(/diverges from the boundary derivation/)
|
||||
})
|
||||
|
||||
it('rejects a frozen request whose messages diverge from the boundary derivation', async () => {
|
||||
const { ctx, session, boundary } = await requestSetup()
|
||||
const messages = [...boundary, { role: 'user', content: [{ type: 'text', text: 'phantom' }] }]
|
||||
|
||||
@@ -6,7 +6,7 @@ Its consumer is the ACP snapshot harness in `examples/acp-agent`, which loads th
|
||||
|
||||
## How the fixture works
|
||||
|
||||
The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`", done by the snapshot harness — this plugin does not record.
|
||||
The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`", done by the snapshot harness — this plugin does not record. A fixture may carry its `request/header` content tokenized to `{{system}}`/`{{tools}}` (the harness pins that content in one scenario and scrubs the rest); replay is indifferent — derivation reads only `assistant/chunk` events and the line-0 session header.
|
||||
|
||||
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`: a `ReplayEntry[]`) that REPLACES the derived script.
|
||||
|
||||
|
||||
@@ -12,7 +12,11 @@
|
||||
* `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model
|
||||
* call per loop step — see packages/core/agent-loop/src/loop.ts). Recording is
|
||||
* therefore "run the real agent once and harvest the `.jsonl`", done by the
|
||||
* snapshot harness — this plugin does not record.
|
||||
* snapshot harness — this plugin does not record. A fixture may carry its
|
||||
* `request/header` content tokenized to `{{system}}`/`{{tools}}` (the harness
|
||||
* pins that content in one scenario and scrubs the rest); replay is
|
||||
* indifferent — derivation reads ONLY `assistant/chunk` events and the line-0
|
||||
* session header.
|
||||
*
|
||||
* A NESTED-agent scenario records more than one log: the parent plus one per
|
||||
* in-process subagent (each subagent runs as its own {@link Session} on the same
|
||||
@@ -124,6 +128,8 @@ export interface SessionScript {
|
||||
* Parse a session `.jsonl` buffer into its event list. Line 0 is the session
|
||||
* header (a `{type:'session',…}` record), every subsequent non-empty line is a
|
||||
* {@link SessionEvent}. The header is skipped; malformed lines fail loud.
|
||||
* @param text - the raw `.jsonl` file contents.
|
||||
* @returns every event after the header, in log order.
|
||||
*/
|
||||
export function parseSessionLog(text: string): SessionEvent[] {
|
||||
const lines = text.split('\n').filter(line => line.trim().length > 0)
|
||||
@@ -147,6 +153,8 @@ export function parseSessionLog(text: string): SessionEvent[] {
|
||||
* own model calls; absent ⇒ 0). A header missing a field falls back to a stable
|
||||
* default (`''` / `0` / `0`) rather than throwing: a no-model fixture is
|
||||
* header-only and still orders fine as the single (primary) script.
|
||||
* @param text - the raw `.jsonl` file contents (only the header line is read).
|
||||
* @returns the header's `id`, `createdAt`, and `seedLength`, defaulted when absent.
|
||||
*/
|
||||
export function parseSessionHeader(text: string): { id: string; createdAt: number; seedLength: number } {
|
||||
const firstLine = text.split('\n').find(line => line.trim().length > 0) ?? '{}'
|
||||
@@ -176,6 +184,8 @@ export function parseSessionHeader(text: string): { id: string; createdAt: numbe
|
||||
* sidecar with an explicit `throw` (or `hang`) entry instead. {@link
|
||||
* deriveReplayScript} throws, naming the offending `(turn, step)`, so a missing
|
||||
* override fails loud rather than silently replaying a thrown call as success.
|
||||
* @param events - the recorded session's events; only `assistant/chunk` is consulted.
|
||||
* @returns one `chunks` entry per recorded model call, in call order.
|
||||
*/
|
||||
export function deriveReplayScript(events: SessionEvent[]): ReplayEntry[] {
|
||||
const script: ReplayEntry[] = []
|
||||
@@ -214,6 +224,8 @@ export function deriveReplayScript(events: SessionEvent[]): ReplayEntry[] {
|
||||
* Fail-loud if the JSONL fixture is missing (the scenario was never recorded) —
|
||||
* never silently returns an empty script, so a coverage hole can't masquerade
|
||||
* as a passing replay.
|
||||
* @param config - the fixture paths; only `file` and `overrideFile` are consulted.
|
||||
* @returns the primary session's replay entries.
|
||||
*/
|
||||
export function loadReplayScript(config: ReplayConfig): ReplayEntry[] {
|
||||
if (config.overrideFile !== undefined && existsSync(config.overrideFile)) {
|
||||
@@ -241,6 +253,8 @@ export function loadReplayScript(config: ReplayConfig): ReplayEntry[] {
|
||||
* the parent issues the FIRST model call (it must stream before it can delegate
|
||||
* in the synchronous nested cut), so binding it to the first live session is
|
||||
* correct regardless of a timestamp tie.
|
||||
* @param config - the fixture paths: the primary log plus any recorded child logs.
|
||||
* @returns the primary script first, then the child scripts in bind order.
|
||||
*/
|
||||
export function loadSessionScripts(config: ReplayConfig): SessionScript[] {
|
||||
const primaryEntries = loadReplayScript(config)
|
||||
@@ -355,6 +369,9 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined)
|
||||
* Each per-session cursor advances synchronously at listener-invocation time
|
||||
* (not lazily inside the generator) so call ORDER within a session, not
|
||||
* iteration order, fixes the mapping.
|
||||
* @param ctx - the context whose `llm/stream` waterfall the listener short-circuits.
|
||||
* @param config - the resolved fixture paths (env-var defaulting is `apply`'s job).
|
||||
* @returns the `ctx.on` disposer that removes the listener.
|
||||
*/
|
||||
export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void {
|
||||
const scripts = loadSessionScripts(config)
|
||||
@@ -408,6 +425,7 @@ export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void
|
||||
export const name = 'llm-replay'
|
||||
export const inject = ['llm']
|
||||
|
||||
/** Plugin config: the {@link ReplayConfig} inputs, each defaulting to its `DSH_SNAPSHOT_*` env var in `apply`. */
|
||||
export interface Config {
|
||||
/** Override the fixture path; defaults to `$DSH_SNAPSHOT_FILE`. */
|
||||
file?: string
|
||||
|
||||
@@ -41,13 +41,13 @@ describe('dsh-subagent-mock', () => {
|
||||
|
||||
it('surfaces a structured result when the request carries an outputSchema', async () => {
|
||||
const ctx = await mount({ reply: 'r', structured: { answer: 42 } })
|
||||
const run = ctx.subagents.start('mock', baseRequest({ outputSchema: { answer: { type: 'number' } } }))
|
||||
const run = ctx.subagents.start('mock', baseRequest({ outputSchema: { type: 'object', properties: { answer: { type: 'number' } } } }))
|
||||
await expect(run.result).resolves.toMatchObject({ structured: { answer: 42 } })
|
||||
})
|
||||
|
||||
it('defaults structured output to { reply } when outputSchema is requested but no structured value is configured', async () => {
|
||||
const ctx = await mount({ reply: 'fallback reply' })
|
||||
const run = ctx.subagents.start('mock', baseRequest({ outputSchema: { answer: { type: 'number' } } }))
|
||||
const run = ctx.subagents.start('mock', baseRequest({ outputSchema: { type: 'object', properties: { answer: { type: 'number' } } } }))
|
||||
await expect(run.result).resolves.toMatchObject({ structured: { reply: 'fallback reply' } })
|
||||
})
|
||||
|
||||
|
||||
Reference in New Issue
Block a user