Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge commit 'refs/codex-unblock/20260722-pr338-master' into HEAD

Browse Source 
# Conflicts:
#	docs/event-producer-consumer.md
#	examples/acp-agent/tests/acp.snapshot.ts
#	packages/core/session/src/index.ts
#	packages/examples/acp-demo/src/index.ts
#	packages/session-persistence/session-persistence-jsonl/src/format.ts
#	vitest.config.ts
This commit is contained in:
Tianyi Cui
2026-07-22 22:37:16 +08:00
parent 8f68ca93f5 54aaab6807
commit b0f5d9e313
499 changed files with 17918 additions and 3171 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
packages/support/acp-snapshot/README.md
View File

@@ -4,8 +4,8 @@ The ACP snapshot suite kit: the shared machinery behind the keyless snapshot tie
Four layers, importable separately:
- **`launchAcpTestAgent` (launcher)** — boots a source agent under tsx or a built `lib` agent under plain Node from a temp cwd, connects the SDK client over a raw-byte stdout tee, collects session updates and stderr, surfaces asynchronous spawn failures through startup, fails closed on unhandled permission requests, and owns graceful or signalled shutdown. Shutdown waits for process exit, inherited stdio closure, and ACP parser exhaustion before resolving or propagating a child error, so captures are complete and callers can remove owned paths after either outcome. When Windows accepts forced termination but publishes its exit marker asynchronously, shutdown gives that marker a bounded grace before treating fallback refusal as a second failure. Snapshot and ordinary e2e suites share this process boundary; a test supplies only agent paths, cwd, environment overrides, and any permission policy.
- **`runScenario` (harness)** — drives ACP JSON-RPC stdio from a deterministic `input.json` script through the launcher, tees raw stdout for the expected-output and purity checks, and harvests every persisted raw JSONL session log (parent and subagent children, primary-first) after graceful stdin EOF. `AgentUnderTest` supplies absolute `binScript`, optional `libBinScript`, `configPath`, and `tsconfigPath` paths because the subprocess cwd is outside the repo. Startup failures preserve captured agent stderr in the rejected diagnostic.
- **`launchAcpTestAgent` (launcher)** — boots a source agent under tsx or a built `lib` agent under plain Node from a supplied cwd, connects the SDK client over a raw-byte stdout tee, collects session updates and stderr, surfaces asynchronous spawn failures through startup, fails closed on unhandled permission requests, and owns graceful or signalled shutdown. Shutdown waits for process exit, inherited stdio closure, and ACP parser exhaustion before resolving or propagating a child error, so captures are complete and callers can remove owned paths after either outcome. When Windows accepts forced termination but publishes its exit marker asynchronously, shutdown gives that marker a bounded grace before treating fallback refusal as a second failure. Snapshot and ordinary e2e suites share this process boundary; a test supplies only agent paths, cwd, environment overrides, and any permission policy.
- **`runScenario` (harness)** — drives ACP JSON-RPC stdio from a deterministic `input.json` script through the launcher, tees raw stdout for the expected-output and purity checks, and harvests every persisted raw JSONL session log (parent and subagent children, primary-first) after graceful stdin EOF. `AgentUnderTest` supplies absolute `binScript`, optional `libBinScript`, `configPath`, and `tsconfigPath` paths because the subprocess cwd is outside the repo; `workspaceParent` may move the generated child cwd from the platform temp directory when that grant is itself under test. Startup failures preserve captured agent stderr in the rejected diagnostic.
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; cwd-rooted separators selected as canonical `/` or host-native; `session_info_update.updatedAt` → `{{updatedAt}}`; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept, the same cwd-path policy), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header Agent Note](../../../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario expected-output and re-persisted-log comparisons, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, the per-header-class pin (`system-prompt.expected.md` plus `tool-schemas.expected.json`) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt/schema-scrubbed, non-pinning fixtures fully header-scrubbed). Refresh expands packed timing envelopes before aligning existing volatile event times, so switching between packed and unpacked layouts cannot shift later records; fresh chunk-fragment arrays remain authoritative. A newly inserted `session/title` receives its preceding event's time so feature-driven insertions do not churn the remainder of a fixture. Each scenario directory's `session.jsonl` plus contiguous `session.<n>.jsonl` siblings are the ordered primary/child inventory; the scenario table does not duplicate their count. Must be called at vitest collection time.
@@ -36,7 +36,7 @@ defineAcpSnapshotSuite({
})
```
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode and filesystem scenarios are templates. Each pinning directory stores the normalized full prompt sequence in generated `system-prompt.expected.md` and the corresponding full tool-schema sequence in generated `tool-schemas.expected.json`; `session.jsonl` stores `"system":"{{system}}","tools":"{{tools}}"` while retaining config, reason, and any model-visible prefix. A pin with legitimate mid-run header changes declares `expectedHeaderChanges`, which fixes the length of both sidecar sequences.
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode and filesystem scenarios are templates. `workspaceParent` moves the generated cwd outside the platform temp area when temporary-directory grants are themselves under test; the harness still owns and removes only the generated child. Each pinning directory stores the normalized full prompt sequence in generated `system-prompt.expected.md` and the corresponding full tool-schema sequence in generated `tool-schemas.expected.json`; `session.jsonl` stores `"system":"{{system}}","tools":"{{tools}}"` while retaining config, reason, and any model-visible prefix. A pin with legitimate mid-run header changes declares `expectedHeaderChanges`, which fixes the length of both sidecar sequences.
Every scenario compares `stdout.expected.jsonl` with cwd-rooted separators canonicalized to `/`. On Windows, `pinsNativeWindowsStdout` additionally compares the complete `stdout.expected.windows.jsonl` after the shared expected output and requires that sidecar exactly when enabled. A scenario whose driven behavior needs POSIX process semantics (e.g. cancelling a live bash call kills a detached process group) declares `posixOnly`, which skips its run test on Windows while the fixture guards keep covering its committed files everywhere.

155
packages/support/acp-snapshot/src/harness.ts
View File

@@ -21,9 +21,12 @@ import { existsSync } from 'node:fs'
import { createHash } from 'node:crypto'
import { tmpdir } from 'node:os'
import { basename, dirname, join, delimiter } from 'node:path'
import { setTimeout as delay } from 'node:timers/promises'
import {
ClientSideConnection,
PROTOCOL_VERSION,
type CreateElicitationRequest,
type CreateElicitationResponse,
type RequestPermissionRequest,
type RequestPermissionResponse,
type SessionNotification,
@@ -32,6 +35,9 @@ import { launchAcpTestAgent, type AgentUnderTest, type LaunchedAcpTestAgent } fr
export type { AgentUnderTest } from './launcher.ts'
const DEFAULT_WAIT_TIMEOUT_MS = 10_000
const WAIT_POLL_INTERVAL_MS = 10
/**
* One step of a scenario's deterministic input script (`input.json`). The
* harness interprets these in order. `newSession` captures the server-issued
@@ -40,10 +46,13 @@ export type { AgentUnderTest } from './launcher.ts'
*
* `promptAndCancel` starts a prompt without awaiting completion, waits until
* the client observes the selected update (`agent_message_chunk` by default),
* then cancels and awaits completion. A named `waitForToolCallUpdate` keeps the
* step open for a terminal tool update that may follow the prompt response.
* then cancels and awaits completion. An optional `waitForFile` first observes
* a cwd-relative readiness marker, and a named `waitForToolCallUpdate` keeps
* the step open for a terminal tool update that may follow the prompt response.
* `promptAndWaitForAgentMessage` arms an exact text-chunk waiter before sending
* the prompt, then keeps the application live until that later update arrives.
* `waitForTurnEnd` holds the subprocess open until the selected session's latest
* complete raw-JSONL turn boundary is `turn/end`; its timeout defaults to 10s.
*/
export type InputStep =
| { op: 'initialize'; terminalOutput?: boolean }
@@ -56,9 +65,13 @@ export type InputStep =
op: 'promptAndCancel'
text: string
afterUpdate?: 'agent_message_chunk' | 'tool_call'
waitForFile?: { path: string; timeoutMs?: number }
waitForToolCallUpdate?: string
}
| { op: 'waitForTurnEnd'; timeoutMs?: number }
| { op: 'cancel' }
| { op: 'setMode'; modeId: string }
| { op: 'setModeExpectError'; modeId: string }
| { op: 'setConfigOption'; configId: string; value: string }
| { op: 'setConfigOptionExpectError'; configId: string; value: string }
@@ -78,6 +91,16 @@ export interface InputScript {
* agent itself just sees `cancelled`, so it cannot absorb the bug).
*/
permissionAnswers?: PermissionAnswer[]
/**
* Ordered answers for the agent's `elicitation/create` round-trips (the
* ask_user_question / plan-review forms), consumed FIFO — the Nth request
* gets the Nth answer. Exhaustion (or no queue) answers `cancel`, the same
* fail-closed stub an elicitation-free scenario relies on. Unlike permission
* kinds, the scripted strings are not validated against the offered form —
* a stray `choice` reaches the agent verbatim, which reads it as a custom
* (non-consenting) answer, so a scenario bug fails safe in the transcript.
*/
elicitationAnswers?: ElicitationAnswer[]
}
/** One scripted answer to a permission request: which offered option kind to select. */
@@ -86,6 +109,16 @@ export interface PermissionAnswer {
kind: 'allow_once' | 'allow_always' | 'reject_once' | 'reject_always'
}
/** One scripted answer to an elicitation form (accept with choice/custom content, or cancel). */
export interface ElicitationAnswer {
/** Accept the form with the content below, or cancel it. */
action: 'accept' | 'cancel'
/** The selected option label (the form's `choice` field). */
choice?: string
/** Free-form text (the form's `custom` field). */
custom?: string
}
/** One harvested session log plus the identifying facts off its header line. */
export interface HarvestedLog {
/** The recorded session id (header `id`). */
@@ -106,7 +139,7 @@ export interface RunResult {
stderr: string
/** The session id the server issued (undefined if no session was created). */
sessionId?: string
/** The temp cwd the session ran in (the bash workspace). */
/** The generated cwd the session ran in (the bash workspace). */
cwd: string
/**
* Every persisted session log harvested after the run, ordered primary-first:
@@ -137,11 +170,19 @@ export interface RunOptions {
childFiles?: string[]
/**
* Optional `<scenario>/workspace/` directory whose contents are copied into
* the temp cwd BEFORE the run — the standard way to seed files the agent
* the generated cwd BEFORE the run — the standard way to seed files the agent
* operates on (a file to read, edit, or grep). Absent for scenarios that
* start from an empty workspace.
*/
workspaceDir?: string
/**
* Parent directory for the generated session cwd. Defaults to
* `os.tmpdir()`. A scenario that must distinguish its workspace from the
* sandbox's always-writable temporary roots can place the generated child
* under `os.homedir()` instead. The harness removes only that generated
* child, never the supplied parent.
*/
workspaceParent?: string
/**
* Alternate LIVE config path for the boot (absolute), overriding
* {@link AgentUnderTest.configPath} for this run. A scenario needing a
@@ -172,15 +213,15 @@ export function snapshotSpillRoot(
/**
* Run a scenario end-to-end against a freshly-spawned subprocess. Owns the
* child and its temp dirs; always tears them down. Returns the captured stdout
* child and its generated dirs; always tears them down. Returns the captured stdout
* and (record mode) the harvested session-log path.
*
* @param input The scenario's input script (steps + optional permission answers).
* @param opts The agent to boot, the mode, and the fixture wiring.
* @returns The captured stdout/stderr, session id, temp cwd, and harvested logs.
* @returns The captured stdout/stderr, session id, generated cwd, and harvested logs.
*/
export async function runScenario(input: InputScript, opts: RunOptions): Promise<RunResult> {
const cwd = await mkdtemp(join(tmpdir(), 'acp-snap-cwd-'))
const cwd = await mkdtemp(join(opts.workspaceParent ?? tmpdir(), 'acp-snap-cwd-'))
const sessionsRoot = await mkdtemp(join(tmpdir(), 'acp-snap-sessions-'))
// Fixed path length: spill-policy budgets the preview against the REAL path
// before stdout normalization, so tmpdir() length differences churn expected outputs.
@@ -194,7 +235,7 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
let sessionLogs: HarvestedLog[] = []
const outcome = await (async (): Promise<RunResult> => {
// Seed the workspace if the scenario ships one (a file the agent reads/edits).
// Copied into the temp cwd so the agent's bash tools see it; the expected outputs
// Copied into the generated cwd so the agent's bash tools see it; the expected outputs
// normalize the cwd, so the seeded paths stay stable across runs.
if (opts.workspaceDir !== undefined && existsSync(opts.workspaceDir)) {
await cp(opts.workspaceDir, cwd, { recursive: true })
@@ -215,6 +256,8 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
// Permission answers are consumed FIFO across the whole run; exhaustion
// falls back to `cancelled` so approval-free scenarios keep the plain stub.
const permissionQueue = [...input.permissionAnswers ?? []]
// Elicitation answers mirror the permission queue: FIFO, cancel on exhaustion.
const elicitationQueue = [...input.elicitationAnswers ?? []]
// A scenario bug detected inside a client callback (a scripted permission
// kind the agent never offered). It cannot fail the run from in there: a
// callback throw only becomes a JSON-RPC error RESPONSE to the agent, and
@@ -244,13 +287,32 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
}
return Promise.resolve({ outcome: { outcome: 'selected', optionId: option.optionId } })
},
createElicitation(_params: CreateElicitationRequest): Promise<CreateElicitationResponse> {
const answer = elicitationQueue.shift()
if (answer === undefined || answer.action !== 'accept') return Promise.resolve({ action: 'cancel' })
return Promise.resolve({
action: 'accept',
content: {
...answer.choice !== undefined ? { choice: answer.choice } : {},
...answer.custom !== undefined ? { custom: answer.custom } : {},
},
})
},
})
const active = launched
await active.spawned
const { client } = active
for (const step of input.steps) {
await runStep(client, step, cwd, match => active.waitForUpdate(match), () => sessionId, (id) => { sessionId = id })
await runStep(
client,
step,
cwd,
match => active.waitForUpdate(match),
() => sessionId,
(id) => { sessionId = id },
(id, timeoutMs) => waitForPersistedTurnEnd(sessionsRoot, id, timeoutMs),
)
// A permission exchange happens while a step's request is in flight, so
// by the time the step settles any script bug it exposed is captured —
// fail the run HERE, as a harness error, rather than hoping the agent's
@@ -261,7 +323,7 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
// persistence) and exits. Then await exit so the harvested log is complete.
await active.close()
// Harvest EVERY persisted log (parent + any subagent children) while the
// temp dirs still exist, ordered primary-first.
// generated dirs still exist, ordered primary-first.
sessionLogs = await harvestSessionLogs(sessionsRoot)
return {
rawStdout: launched.rawStdout(),
@@ -320,6 +382,7 @@ async function runStep(
waitForUpdate: (match: (u: SessionNotification['update']) => boolean) => Promise<SessionNotification['update']>,
getSessionId: () => string | undefined,
setSessionId: (id: string) => void,
waitForTurnEnd: (sessionId: string, timeoutMs?: number) => Promise<void>,
): Promise<void> {
switch (step.op) {
case 'initialize':
@@ -384,6 +447,9 @@ async function runStep(
const promptDone = client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
const afterUpdate = step.afterUpdate ?? 'agent_message_chunk'
await waitForUpdate(u => u.sessionUpdate === afterUpdate)
if (step.waitForFile !== undefined) {
await waitForWorkspaceFile(cwd, step.waitForFile.path, step.waitForFile.timeoutMs)
}
// Arm this before cancellation so a fast tool drain cannot outrun the waiter.
const toolCallUpdateDone = step.waitForToolCallUpdate === undefined
? undefined
@@ -393,12 +459,36 @@ async function runStep(
if (toolCallUpdateDone !== undefined) await toolCallUpdateDone
return
}
case 'waitForTurnEnd': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: waitForTurnEnd before newSession')
await waitForTurnEnd(sessionId, step.timeoutMs)
return
}
case 'cancel': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: cancel before newSession')
await client.cancel({ sessionId })
return
}
case 'setMode': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: setMode before newSession')
await client.setSessionMode({ sessionId, modeId: step.modeId })
return
}
case 'setModeExpectError': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: setModeExpectError before newSession')
// The bridge rejects an unknown/uncomposed mode id with invalidParams;
// that rejection IS the expected wire behavior — swallow it so the run
// completes and the error frame is captured in the transcript.
await client.setSessionMode({ sessionId, modeId: step.modeId }).then(
() => { throw new Error('snapshot-harness: expected session/set_mode to be rejected but it succeeded') },
() => { /* expected: the bridge rejected the mode id */ },
)
return
}
case 'setConfigOption': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: setConfigOption before newSession')
@@ -422,6 +512,51 @@ async function runStep(
}
}
/**
* Wait until the raw JSONL backend exposes one complete closing turn boundary.
* The ACP cancel notification settles its prompt before the agent necessarily
* reaches quiescence, so cancellation snapshots use this external boundary to
* keep subprocess disposal from changing an `aborted` turn into `disposed`.
*/
async function waitForPersistedTurnEnd(
root: string,
sessionId: string,
timeoutMs = DEFAULT_WAIT_TIMEOUT_MS,
): Promise<void> {
const deadline = Date.now() + timeoutMs
while (true) {
const log = (await harvestSessionLogs(root)).find(candidate => candidate.id === sessionId)
if (log !== undefined && latestTurnIsClosed(log.content)) return
if (Date.now() >= deadline) {
throw new Error(`snapshot-harness: session "${sessionId}" did not persist turn/end within ${timeoutMs}ms`)
}
await delay(WAIT_POLL_INTERVAL_MS)
}
}
/** Wait for a cwd-relative marker proving an external action reached readiness. */
async function waitForWorkspaceFile(
cwd: string,
path: string,
timeoutMs = DEFAULT_WAIT_TIMEOUT_MS,
): Promise<void> {
const target = join(cwd, path)
const deadline = Date.now() + timeoutMs
while (!existsSync(target)) {
if (Date.now() >= deadline) {
throw new Error(`snapshot-harness: workspace file "${path}" did not appear within ${timeoutMs}ms`)
}
await delay(WAIT_POLL_INTERVAL_MS)
}
}
/** Return whether the last complete raw-JSONL turn boundary closes its turn. */
function latestTurnIsClosed(content: string): boolean {
const complete = content.slice(0, content.lastIndexOf('\n') + 1)
return complete.lastIndexOf('\n{"type":"turn/end",')
> complete.lastIndexOf('\n{"type":"turn/start",')
}
/**
* Harvest EVERY persisted `.jsonl` session log under a sessions root, parse each
* header line, and return them ordered primary-first: the top-level session (no

1
packages/support/acp-snapshot/src/index.ts
View File

@@ -18,6 +18,7 @@
export {
runScenario,
type ElicitationAnswer,
type HarvestedLog,
type InputScript,
type InputStep,

7
packages/support/acp-snapshot/src/launcher.ts
View File

@@ -15,6 +15,8 @@ import {
ndJsonStream,
type Agent as AcpAgent,
type Client,
type CreateElicitationRequest,
type CreateElicitationResponse,
type RequestPermissionRequest,
type RequestPermissionResponse,
type SessionNotification,
@@ -47,6 +49,8 @@ export interface AcpTestLaunchOptions {
env?: NodeJS.ProcessEnv
/** Permission handler; omitted requests fail closed as `cancelled`. */
requestPermission?: (params: RequestPermissionRequest) => Promise<RequestPermissionResponse>
/** Elicitation handler; omitted requests fail closed as `cancel`. */
createElicitation?: (params: CreateElicitationRequest) => Promise<CreateElicitationResponse>
}
/** A running ACP test process and its captured client-side surfaces. */
@@ -152,6 +156,8 @@ export function launchAcpTestAgent(options: AcpTestLaunchOptions): LaunchedAcpTe
}
const requestPermission = options.requestPermission
?? (() => Promise.resolve({ outcome: { outcome: 'cancelled' as const } }))
const createElicitation = options.createElicitation
?? (() => Promise.resolve({ action: 'cancel' as const }))
const makeClient = (_agent: AcpAgent): Client => ({
sessionUpdate(params: SessionNotification): Promise<void> {
return trackClientCallback(() => {
@@ -175,6 +181,7 @@ export function launchAcpTestAgent(options: AcpTestLaunchOptions): LaunchedAcpTe
})
},
requestPermission: params => trackClientCallback(() => requestPermission(params)),
unstable_createElicitation: params => trackClientCallback(() => createElicitation(params)),
})
const client = new ClientSideConnection(makeClient, stream)
// `exit` only reports the parent process's status. Descendants may retain

4
packages/support/acp-snapshot/src/normalize.ts
View File

@@ -1,5 +1,5 @@
/**
* Pure ACP transcript and session-log normalizers. They scrub session ids, temp cwd, RPC ids,
* Pure ACP transcript and session-log normalizers. They scrub session ids, run cwd, RPC ids,
* timestamps, and hook duration while preserving deterministic event sequence numbers.
* Request-header scrubbers stay composable so one scenario per header class can pin prompt and
* tool-schema sidecars while retaining any model-visible prefix in the session log.
@@ -44,7 +44,7 @@ function canonicalizeEmbeddedPaths(value: string): string {
export interface NormalizeContext {
/** The session id(s) the run issued — replaced with `{{sessionId}}`. */
sessionIds: string[]
/** The temp cwd the run used — replaced with `{{cwd}}`. */
/** The generated cwd the run used — replaced with `{{cwd}}`. */
cwd: string
}

9
packages/support/acp-snapshot/src/suite.ts
View File

@@ -106,6 +106,12 @@ export interface Scenario {
* {@link headerClass}.
*/
configPath?: string
/**
* Parent directory for the generated session cwd. Defaults to the platform
* temp directory; set this when temp is itself part of the behavior under
* test and the scenario needs an independent project location.
*/
workspaceParent?: string
/**
* Whether Windows additionally compares stdout with native separators against
* `stdout.expected.windows.jsonl`. The shared canonical stdout expected output is still
@@ -239,7 +245,7 @@ export function fixtureContext(fixture: string): NormalizeContext {
* The `data.header` payload of every `request/header` event in a session
* JSONL, in log order, with the log's volatile values scrubbed first
* ({@link normalizeSessionLog}) so headers harvested from different runs —
* each embedding its own temp cwd in the composed prompt — compare on equal
* each embedding its own generated cwd in the composed prompt — compare on equal
* footing.
*
* @param rawLog The session `.jsonl` content to extract headers from.
@@ -603,6 +609,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
// replays from its own script. In RECORD they are harvested, not read.
...!RECORDING && childFixtureFiles.length > 0 ? { childFiles: childFixtureFiles.map(file => join(dir, file)) } : {},
...existsSync(workspaceDir) ? { workspaceDir } : {},
...scenario.workspaceParent !== undefined ? { workspaceParent: scenario.workspaceParent } : {},
// A scenario booting an overlay tree passes its own live config; the
// bin's replay swap derives the sibling `*cordis.snapshot.yml` from it.
...scenario.configPath !== undefined ? { configPath: scenario.configPath } : {},

76
packages/support/acp-snapshot/tests/fixtures/fake-acp-agent.ts vendored
View File

@@ -1,7 +1,19 @@
/**
* Scripted ACP agent for snapshot-kit tests. A fixture-adjacent `behavior.json` controls the
* subprocess reached through the real harness path; the bin reports observations over ACP and
* writes scripted logs before exiting on stdin EOF.
* Scripted fake ACP agent bin for `dsh-acp-snapshot`'s unit specs. Speaks
* newline-delimited JSON-RPC on stdio like the real `dsh-acp-agent` bin, but
* every behavior — how prompts settle, whether session/new rejects, which
* session logs get persisted, what filesystem noise to leave — comes from a
* `behavior.json` sitting NEXT to the `$DSH_SNAPSHOT_FILE` fixture, so a spec
* scripts a whole subprocess run from data. The specs launch it through the
* REAL `runScenario` spawn path (tsx loader, temp cwd, env plumbing), so the
* harness plumbing is exercised for real; only the agent behind the protocol
* is scripted.
*
* The specs (not the golden tier) own this bin: it asserts nothing, echoes
* observable facts into `session/update` text chunks (env probe, permission
* outcome, seeded-workspace listing) for the spec to read off `rawStdout`, and
* exits 0 on stdin EOF after writing the scripted logs — mirroring the real
* bin's dispose-flush-exit shape.
*/
import { mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
@@ -37,8 +49,14 @@ interface Behavior {
cancelAtToolCall?: boolean
/** Emit the parked tool call's terminal update after answering cancellation. */
cancelToolCallUpdate?: boolean
/** Persist the scripted logs while handling cancellation, before stdin EOF. */
persistLogsOnCancel?: boolean
/** Before responding to a prompt, send a `session/request_permission` request and echo its outcome as a chunk. */
permissionProbe?: boolean
/** Before responding to a prompt, send an `elicitation/create` request and echo its response as a chunk. */
elicitationProbe?: boolean
/** How `session/set_mode` settles: an empty response (echoing the modeId as a chunk) or a JSON-RPC error. */
setMode?: 'respond' | 'error'
/** Echo the `DSH_SNAPSHOT_*` env the harness set as a chunk (spec-side env-plumbing assertions). */
echoEnv?: boolean
/** Echo the sorted cwd listing as a chunk (spec-side workspace-seeding assertions). */
@@ -47,7 +65,7 @@ interface Behavior {
stderrNote?: string
/** Let a short-lived descendant retain stdio and emit one final ACP update plus stderr line after this parent exits. */
lateInheritedOutput?: boolean
/** Session logs to persist on stdin EOF. */
/** Session logs to persist on stdin EOF and, when selected, on cancellation. */
logs?: ScriptedLog[]
/** Leave a stray FILE directly under the sessions root (harvest must skip it). */
strayRootFile?: boolean
@@ -84,8 +102,8 @@ let sessionId = ''
let sessionCwd = ''
/** The parked prompt request id while `hang-until-cancel` waits for the cancel notification. */
let parkedPromptId: number | string | null = null
/** Resolvers for permission-probe responses, keyed by outbound request id. */
const pendingPermission = new Map<number, (outcome: unknown) => void>()
/** Resolvers for outbound probe responses (permission/elicitation), keyed by request id. */
const pendingOutbound = new Map<number, (result: unknown) => void>()
/** Per-run `session/set_config_option` state: config id → current value (first vocabulary entry until set). */
const currentConfig: Record<string, string> = {}
@@ -160,8 +178,8 @@ async function handlePrompt(id: number | string): Promise<void> {
}
if (behavior.permissionProbe === true) {
const requestId = nextOutboundId++
const outcome = await new Promise<unknown>((resolve) => {
pendingPermission.set(requestId, resolve)
const result = await new Promise<unknown>((resolve) => {
pendingOutbound.set(requestId, resolve)
send({
id: requestId,
method: 'session/request_permission',
@@ -175,7 +193,24 @@ async function handlePrompt(id: number | string): Promise<void> {
},
})
})
chunk(`permission:${JSON.stringify(outcome)}`)
chunk(`permission:${JSON.stringify((result as { outcome?: unknown } | undefined)?.outcome ?? null)}`)
}
if (behavior.elicitationProbe === true) {
const requestId = nextOutboundId++
const result = await new Promise<unknown>((resolve) => {
pendingOutbound.set(requestId, resolve)
send({
id: requestId,
method: 'elicitation/create',
params: {
sessionId,
mode: 'form',
message: 'Approve this plan and leave plan mode?',
requestedSchema: { type: 'object', title: 'Plan review', properties: { choice: { type: 'string' }, custom: { type: 'string' } }, required: [] },
},
})
})
chunk(`elicitation:${JSON.stringify(result ?? null)}`)
}
switch (behavior.prompt ?? 'respond') {
case 'respond':
@@ -195,10 +230,10 @@ function handleFrame(frame: Record<string, unknown>): void {
const method = frame.method as string | undefined
const params = (frame.params ?? {}) as Record<string, unknown>
// A response to one of OUR outbound requests (the permission probe).
if (method === undefined && id !== undefined && typeof id === 'number' && pendingPermission.has(id)) {
const resolve = pendingPermission.get(id) as (outcome: unknown) => void
pendingPermission.delete(id)
resolve((frame.result as { outcome?: unknown } | undefined)?.outcome ?? null)
if (method === undefined && id !== undefined && typeof id === 'number' && pendingOutbound.has(id)) {
const resolve = pendingOutbound.get(id) as (result: unknown) => void
pendingOutbound.delete(id)
resolve(frame.result)
return
}
switch (method) {
@@ -219,6 +254,14 @@ function handleFrame(frame: Record<string, unknown>): void {
case 'session/prompt':
void handlePrompt(id as number | string)
return
case 'session/set_mode':
if ((behavior.setMode ?? 'respond') === 'error') {
respondError(id as number | string, 'unknown mode')
return
}
chunk(`setMode:${String(params.modeId)}`)
respond(id as number | string, {})
return
case 'session/set_config_option': {
const vocabulary = behavior.configOptions
const configId = params.configId as string
@@ -263,6 +306,7 @@ function handleFrame(frame: Record<string, unknown>): void {
},
})
}
if (behavior.persistLogsOnCancel === true) writeLogs()
}
return
default:
@@ -272,12 +316,16 @@ function handleFrame(frame: Record<string, unknown>): void {
}
}
function flushLogsAndExit(): void {
function writeLogs(): void {
for (const log of behavior.logs ?? []) {
const target = join(sessionsRoot, log.file)
mkdirSync(dirname(target), { recursive: true })
writeFileSync(target, log.lines.map(l => JSON.stringify(instantiate(l))).join('\n') + '\n')
}
}
function flushLogsAndExit(): void {
writeLogs()
if (behavior.strayRootFile === true) writeFileSync(join(sessionsRoot, 'stray.txt'), 'not a bucket\n')
if (behavior.strayBucketFile === true) {
mkdirSync(join(sessionsRoot, 'bucket-noise'), { recursive: true })

2
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/session.1.jsonl vendored Normal file
View File

@@ -0,0 +1,2 @@
{"type":"session","id":"abababab-cdcd-4efe-8ada-badabadabada","createdAt":800,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-KBQJbW","parentSession":"f6fa7fcf-dd9c-4b39-8815-b25ddcebfd88"}
{"type":"request/header","seq":0,"time":2,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}

174
packages/support/acp-snapshot/tests/harness.spec.ts
View File

@@ -1,7 +1,7 @@
import { mkdtemp, rm, writeFile } from 'node:fs/promises'
import { once } from 'node:events'
import { tmpdir } from 'node:os'
import { delimiter, join } from 'node:path'
import { delimiter, join, relative, sep } from 'node:path'
import { fileURLToPath } from 'node:url'
import { afterAll, describe, expect, it, vi } from 'vitest'
import { PROTOCOL_VERSION } from '@agentclientprotocol/sdk'
@@ -95,8 +95,8 @@ describe('runScenario', () => {
expect(clientClosed).toBe(true)
})
it('centralizes ACP boot, captures, updates, fail-closed permissions, and shutdown', { timeout: 20_000 }, async () => {
const { dir, fixtureFile } = await scenario({ permissionProbe: true, echoEnv: true, stderrNote: 'launcher stderr' })
it('centralizes ACP boot, captures, updates, fail-closed interactions, and shutdown', { timeout: 20_000 }, async () => {
const { dir, fixtureFile } = await scenario({ permissionProbe: true, elicitationProbe: true, echoEnv: true, stderrNote: 'launcher stderr' })
const sessionsRoot = await mkdtemp(join(tmpdir(), 'acp-launcher-sessions-'))
tempDirs.push(sessionsRoot)
const launched = launchAcpTestAgent({
@@ -120,6 +120,7 @@ describe('runScenario', () => {
expect((await nextChunk).sessionUpdate).toBe('agent_message_chunk')
expect(launched.updates.some(update => update.sessionUpdate === 'agent_message_chunk')).toBe(true)
expect(launched.rawStdout()).toContain('permission:{\\"outcome\\":\\"cancelled\\"}')
expect(launched.rawStdout()).toContain('elicitation:{\\"action\\":\\"cancel\\"}')
expect(launched.stderr()).toContain('launcher stderr')
const unmatched = expect(launched.waitForUpdate(() => false)).rejects.toThrow(/update stream closed/)
await launched.close()
@@ -465,6 +466,22 @@ describe('runScenario', () => {
expect(result.rawStdout).toContain('workspace:seeded.txt')
})
it('creates the generated workspace under an explicit parent', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({})
const workspaceParent = await mkdtemp(join(tmpdir(), 'acp-snap-parent-'))
tempDirs.push(workspaceParent)
const result = await runScenario(
{ steps: boot },
{ agent: AGENT, mode: 'replay', fixtureFile, workspaceParent },
)
const child = relative(workspaceParent, result.cwd)
expect(child).not.toBe('')
expect(child).not.toBe('..')
expect(child.startsWith(`..${sep}`)).toBe(false)
})
it('promptAndCancel waits for the streamed chunk, cancels, and settles the prompt', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ prompt: 'hang-until-cancel' })
const result = await runScenario(
@@ -476,6 +493,37 @@ describe('runScenario', () => {
expect(result.rawStdout.indexOf('thinking about it')).toBeLessThan(result.rawStdout.indexOf('cancelled'))
})
it('promptAndCancel can wait for cwd-relative readiness before cancelling', { timeout: 20_000 }, async () => {
const { dir, fixtureFile } = await scenario({ prompt: 'hang-until-cancel' })
const workspaceDir = join(dir, 'workspace')
const { mkdir } = await import('node:fs/promises')
await mkdir(workspaceDir, { recursive: true })
await writeFile(join(workspaceDir, 'started.txt'), 'started')
const result = await runScenario(
{
steps: [...boot, {
op: 'promptAndCancel',
text: 'hang',
waitForFile: { path: 'started.txt' },
}],
},
{ agent: AGENT, mode: 'replay', fixtureFile, workspaceDir },
)
expect(result.rawStdout).toContain('"stopReason":"cancelled"')
const missing = await scenario({ prompt: 'hang-until-cancel' })
await expect(runScenario(
{
steps: [...boot, {
op: 'promptAndCancel',
text: 'hang',
waitForFile: { path: 'never.txt', timeoutMs: 20 },
}],
},
{ agent: AGENT, mode: 'replay', fixtureFile: missing.fixtureFile },
)).rejects.toThrow(/workspace file "never\.txt" did not appear within 20ms/)
})
it('promptAndWaitForAgentMessage keeps the app live through a matching later update', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ prompt: 'respond' })
const result = await runScenario(
@@ -513,6 +561,55 @@ describe('runScenario', () => {
expect(result.rawStdout.indexOf('cancelled')).toBeLessThan(result.rawStdout.indexOf('"sessionUpdate":"tool_call_update"'))
})
it('waitForTurnEnd holds cancellation open through the persisted closing boundary', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({
prompt: 'hang-until-cancel',
persistLogsOnCancel: true,
logs: [{
file: 'bucket/session.jsonl',
lines: [
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
{ type: 'turn/end', seq: 1, time: 2, data: { turn: 1, reason: { kind: 'aborted' } } },
],
}],
})
const result = await runScenario(
{ steps: [...boot, { op: 'promptAndCancel', text: 'hang' }, { op: 'waitForTurnEnd' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.sessionLogs[0]?.content).toContain('"type":"turn/end"')
})
it('waitForTurnEnd times out for a missing log and an open logged turn', { timeout: 20_000 }, async () => {
const missing = await scenario({})
await expect(runScenario(
{ steps: [...boot, { op: 'waitForTurnEnd', timeoutMs: 20 }] },
{ agent: AGENT, mode: 'replay', fixtureFile: missing.fixtureFile },
)).rejects.toThrow(/did not persist turn\/end within 20ms/)
const open = await scenario({
prompt: 'hang-until-cancel',
persistLogsOnCancel: true,
logs: [{
file: 'bucket/session.jsonl',
lines: [
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } },
],
}],
})
await expect(runScenario(
{
steps: [
...boot,
{ op: 'promptAndCancel', text: 'hang' },
{ op: 'waitForTurnEnd', timeoutMs: 20 },
],
},
{ agent: AGENT, mode: 'replay', fixtureFile: open.fixtureFile },
)).rejects.toThrow(/did not persist turn\/end within 20ms/)
})
it('promptExpectError swallows a model-error response as the expected outcome', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ prompt: 'error' })
const result = await runScenario(
@@ -603,6 +700,7 @@ describe('runScenario', () => {
[{ op: 'promptAndWaitForAgentMessage', text: 'x', waitForText: 'later' }, /promptAndWaitForAgentMessage before newSession/],
[{ op: 'promptExpectError', text: 'x' }, /promptExpectError before newSession/],
[{ op: 'promptAndCancel', text: 'x' }, /promptAndCancel before newSession/],
[{ op: 'waitForTurnEnd' }, /waitForTurnEnd before newSession/],
[{ op: 'cancel' }, /cancel before newSession/],
[{ op: 'setConfigOption', configId: 'sandbox-mode', value: 'read-only' }, /setConfigOption before newSession/],
[{ op: 'setConfigOptionExpectError', configId: 'sandbox-mode', value: 'yolo' }, /setConfigOptionExpectError before newSession/],
@@ -716,6 +814,69 @@ describe('runScenario', () => {
expect(result.sessionLogs).toHaveLength(0)
})
it('drives session/set_mode and swallows the expected rejection of setModeExpectError', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({})
const result = await runScenario(
{ steps: [...boot, { op: 'setMode', modeId: 'plan' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.rawStdout).toContain('setMode:plan')
const rejecting = await scenario({ setMode: 'error' })
const rejected = await runScenario(
{ steps: [...boot, { op: 'setModeExpectError', modeId: 'yolo' }] },
{ agent: AGENT, mode: 'replay', fixtureFile: rejecting.fixtureFile },
)
expect(rejected.rawStdout).toContain('unknown mode')
})
it('fails the run when setModeExpectError unexpectedly succeeds, and both mode ops require a session', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({})
await expect(runScenario(
{ steps: [...boot, { op: 'setModeExpectError', modeId: 'plan' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/expected session\/set_mode to be rejected/)
await expect(runScenario(
{ steps: [{ op: 'initialize' }, { op: 'setMode', modeId: 'plan' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/setMode before newSession/)
await expect(runScenario(
{ steps: [{ op: 'initialize' }, { op: 'setModeExpectError', modeId: 'plan' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/setModeExpectError before newSession/)
})
it('answers elicitations from the scripted queue, falling back to cancel on exhaustion', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ elicitationProbe: true })
// Three prompts → three elicitations: an accept-with-choice, an
// accept-with-custom (feedback), then the exhausted-queue cancel.
const result = await runScenario(
{
steps: [...boot, { op: 'prompt', text: 'one' }, { op: 'prompt', text: 'two' }, { op: 'prompt', text: 'three' }],
elicitationAnswers: [
{ action: 'accept', choice: 'Approve' },
{ action: 'accept', custom: 'add tests first' },
],
},
{ agent: AGENT, mode: 'replay', fixtureFile },
)
const first = result.rawStdout.indexOf('elicitation:{\\"action\\":\\"accept\\",\\"content\\":{\\"choice\\":\\"Approve\\"}}')
const second = result.rawStdout.indexOf('elicitation:{\\"action\\":\\"accept\\",\\"content\\":{\\"custom\\":\\"add tests first\\"}}')
const third = result.rawStdout.indexOf('elicitation:{\\"action\\":\\"cancel\\"}')
expect(first).toBeGreaterThanOrEqual(0)
expect(second).toBeGreaterThan(first)
expect(third).toBeGreaterThan(second)
})
it('a scripted elicitation cancel answers cancel', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ elicitationProbe: true })
const result = await runScenario(
{ steps: [...boot, { op: 'prompt', text: 'one' }], elicitationAnswers: [{ action: 'cancel' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.rawStdout).toContain('elicitation:{\\"action\\":\\"cancel\\"}')
})
it('answers permission requests from the scripted queue by option kind, falling back to cancelled', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ permissionProbe: true })
// Two prompts → two permission round-trips; one scripted answer, so the
@@ -744,8 +905,11 @@ describe('runScenario', () => {
it('rejects the run on a scripted permission kind the agent never offered', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ permissionProbe: true })
// The fake offers only allow_once/reject_once. The harness must reject an impossible click,
// not merely send an RPC error that a tolerant agent could absorb.
// The fake bin offers allow_once/reject_once; scripting allow_always is a
// scenario bug. The agent is answered `cancelled` (it must not be able to
// absorb the bug as an error-means-denial), and the RUN fails: a callback
// throw would only reach the agent as a JSON-RPC error response, letting
// a tolerant agent carry on and the scenario pass — or record.
await expect(runScenario(
{ steps: [...boot, { op: 'prompt', text: 'impossible click' }], permissionAnswers: [{ kind: 'allow_always' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },

9
packages/support/acp-snapshot/tests/suite.spec.ts
View File

@@ -48,7 +48,14 @@ const RECORD_SRC = fileURLToPath(new URL('./fixtures/record-suite', import.meta.
// Replay pins explicit header classes; recording covers the default fallback.
const REPLAY_SCENARIOS: Scenario[] = [
{ name: 'pin-turn', hasModelTurn: true, recorded: true, pinsHeader: true, expectedHeaderChanges: 1, headerClass: 'main' },
{ name: 'plain-turn', hasModelTurn: true, recorded: true, headerClass: 'main', configPath: AGENT.configPath },
{
name: 'plain-turn',
hasModelTurn: true,
recorded: true,
headerClass: 'main',
configPath: AGENT.configPath,
workspaceParent: tmpdir(),
},
{ name: 'no-model', hasModelTurn: false, recorded: false, headerClass: 'main' },
{ name: 'blocked-log', hasModelTurn: false, comparesLog: true, recorded: false, headerClass: 'main' },
{ name: 'authored-error', hasModelTurn: true, recorded: false, overridden: true, headerClass: 'main' },

12
packages/support/loader-smoke/src/index.ts
View File

@@ -138,8 +138,6 @@ export interface LoaderSmokeOptions {
readonly mode?: ExampleMode
/** Environment overrides layered over the parent and isolated DSH homes. */
readonly env?: Readonly<NodeJS.ProcessEnv>
/** Lines written to stdin before EOF; omitted means immediate EOF. */
readonly stdinLines?: readonly string[]
/** Process deadline override for harness tests. */
readonly processTimeoutMs?: number
/** Optional world-state setup run in the isolated cwd before process start. */
@@ -157,10 +155,10 @@ export interface LoaderSmokeResult {
}
/**
* Boot one real Loader tree from an isolated cwd, write the requested stdin
* script, close stdin, and await a clean exit. The helper owns process kill and
* temp-directory cleanup on every outcome, and picks src/lib via {@link resolveExampleLaunch}.
* @param options - example paths, mode, environment, stdin, and diagnostic identity.
* Boot one real Loader tree from an isolated cwd, close stdin immediately, and
* await a clean exit. The helper owns process kill and temp-directory cleanup on
* every outcome, and picks src/lib via {@link resolveExampleLaunch}.
* @param options - example paths, mode, environment, and diagnostic identity.
* @returns captured stdout and stderr after a zero exit.
*/
export async function runLoaderSmoke(options: LoaderSmokeOptions): Promise<LoaderSmokeResult> {
@@ -220,7 +218,7 @@ export async function runLoaderSmoke(options: LoaderSmokeOptions): Promise<Loade
})
/* v8 ignore stop */
child.stdin.end((options.stdinLines ?? []).map(line => `${line}\n`).join(''))
child.stdin.end()
})
await options.inspect?.(cwd)
return result

5
packages/support/loader-smoke/tests/loader-smoke.spec.ts
View File

@@ -11,7 +11,7 @@ const fixture = (name: string): string => fileURLToPath(new URL(`./fixtures/${na
const canonicalTempPath = (path: string): string => path.replace(/^\/private(?=\/var\/)/, '')
describe('runLoaderSmoke', () => {
it('isolates the process, writes stdin, captures output, and removes the cwd', async () => {
it('isolates the process, closes stdin, captures output, and removes the cwd', async () => {
const result = await runLoaderSmoke({
label: 'success fixture',
tempDirPrefix: 'loader-smoke-success-',
@@ -20,7 +20,6 @@ describe('runLoaderSmoke', () => {
tsconfigPath,
mode: 'src',
env: { LOADER_SMOKE_MARKER: 'present' },
stdinLines: ['one', 'two'],
})
const output = JSON.parse(result.stdout) as {
configPath: string
@@ -35,7 +34,7 @@ describe('runLoaderSmoke', () => {
configPath,
args: [configPath],
marker: 'present',
input: 'one\ntwo\n',
input: '',
})
expect(canonicalTempPath(output.dshHome)).toBe(canonicalTempPath(join(output.cwd, '.dsh')))
expect(canonicalTempPath(output.agentsHome)).toBe(canonicalTempPath(join(output.cwd, '.agents')))