Merge branch 'master' into worktree/dsh-arg-parser

Integrate the Commander argument adapter and dsh-front-door work with master's
config-tree `dsh web` (#601: AppCLIEntry + apps/cli/cordis.yml) and the
packages/ui/acp → packages/acp/acp relocation.

- web.ts: keep master's AppCLIEntry-based boot, but take the adapter's parsed
  (host, port, dev) instead of an internal parseArgs. The adapter's host/port
  defaults (127.0.0.1/3080) match cordis.yml, so always passing them is
  behavior-equivalent to master's "undefined keeps the yml default".
- apps/cli/package.json: master's expanded config-tree dep set + commander.
- retire-readline Agent Note: point the TUI refusal proof at
  apps/cli/tests/built-bin.e2e.ts (both languages), re-record the pair.
- READMEs reconciled (demo-bin removal + master's ACP/channel rewording).
This commit is contained in:
Turtle
2026-07-25 14:37:57 +08:00
640 changed files with 13210 additions and 24114 deletions

View File

@@ -6,7 +6,7 @@ Four layers, importable separately:
- **`launchAcpTestAgent` (launcher)** — boots a source agent under tsx or a built `lib` agent under plain Node from a supplied cwd, connects the SDK client over a raw-byte stdout tee, collects session updates and stderr, surfaces asynchronous spawn failures through startup, fails closed on unhandled permission requests, and owns graceful or signalled shutdown. Shutdown waits for process exit, inherited stdio closure, and ACP parser exhaustion before resolving or propagating a child error, so captures are complete and callers can remove owned paths after either outcome. When Windows accepts forced termination but publishes its exit marker asynchronously, shutdown gives that marker a bounded grace before treating fallback refusal as a second failure. Snapshot and ordinary e2e suites share this process boundary; a test supplies only agent paths, cwd, environment overrides, and any permission policy.
- **`runScenario` (harness)** — drives ACP JSON-RPC stdio from a deterministic `input.json` script through the launcher, tees raw stdout for the expected-output and purity checks, and harvests every persisted raw JSONL session log (parent and subagent children, primary-first) after graceful stdin EOF. `AgentUnderTest` supplies absolute `binScript`, optional `libBinScript`, `configPath`, and `tsconfigPath` paths because the subprocess cwd is outside the repo; `workspaceParent` may move the generated child cwd from the platform temp directory when that grant is itself under test. Startup failures preserve captured agent stderr in the rejected diagnostic.
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs and every native/JavaScript filesystem spelling of the generated cwd → tokens, longest-first; cwd-rooted separators selected as canonical `/` or host-native; `session_info_update.updatedAt``{{updatedAt}}`; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept, the same cwd-path policy), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header Agent Note](../../../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs and every native/JavaScript filesystem spelling of the generated cwd → tokens, longest-first; cwd-rooted separators selected as canonical `/` or host-native; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept, the same cwd-path policy), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header Agent Note](../../../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario expected-output and re-persisted-log comparisons, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, the per-header-class pin (`system-prompt.expected.md` plus `tool-schemas.expected.json`) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt/schema-scrubbed, non-pinning fixtures fully header-scrubbed). Refresh expands packed timing envelopes before aligning existing volatile event times, so switching between packed and unpacked layouts cannot shift later records; fresh chunk-fragment arrays remain authoritative. A newly inserted `session/title` receives its preceding event's time so feature-driven insertions do not churn the remainder of a fixture. Each scenario directory's `session.jsonl` plus contiguous `session.<n>.jsonl` siblings are the ordered primary/child inventory; the scenario table does not duplicate their count. Must be called at vitest collection time.
A consuming `*.snapshot.ts` is the scenario table plus one factory call:
@@ -14,7 +14,22 @@ A consuming `*.snapshot.ts` is the scenario table plus one factory call:
```ts
import { dirname, join } from 'node:path'
import { fileURLToPath } from 'node:url'
import { defineAcpSnapshotSuite, type Scenario } from '@deepseek-ai/dsh-acp-snapshot'
import {
defineAcpSnapshotSuite,
type Scenario,
type SnapshotSuiteOptions,
} from '@deepseek-ai/dsh-acp-snapshot'
function snapshotMode(value: string | undefined): SnapshotSuiteOptions['mode'] {
switch (value) {
case undefined:
case '':
case 'replay': return 'replay'
case 'record': return 'record'
case 'refresh': return 'refresh'
default: throw new Error(`unknown DSH_SNAPSHOT mode: ${value}`)
}
}
const SCENARIOS: Scenario[] = [
{ name: 'text-turn', hasModelTurn: true, recorded: true, pinsHeader: true },
@@ -28,11 +43,7 @@ defineAcpSnapshotSuite({
},
snapshotsDir: join(dirname(fileURLToPath(import.meta.url)), 'snapshots'),
scenarios: SCENARIOS, // exactly one entry per header class sets pinsHeader
mode: process.env.DSH_SNAPSHOT === 'record'
? 'record'
: process.env.DSH_SNAPSHOT === 'refresh'
? 'refresh'
: 'replay',
mode: snapshotMode(process.env.DSH_SNAPSHOT),
})
```
@@ -42,7 +53,7 @@ Every scenario compares `stdout.expected.jsonl` with cwd-rooted separators canon
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config Agent Note](../../../.agents/notes/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log expected outputs, and each pin's prompt and tool-schema sidecars from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md).
Constraints: `suite.ts` imports vitest, so the package entry is importable only inside a vitest run (the launcher, harness, and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug). Session config options are scriptable too: the `setConfigOption` step switches a knob over `session/set_config_option`, and `setConfigOptionExpectError` asserts the bridge rejects an unknown id or out-of-vocabulary value (the error frame stays in the transcript).
Constraints: `suite.ts` imports vitest, so the package entry is importable only inside a vitest run (the launcher, harness, and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection`. Input scripts cover initialization, fresh-session creation, text prompting, cancellation, expected RPC failures, and durable turn-boundary waits. Permission round-trips are a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) mapped to the agent-issued `optionId`; an absent or exhausted queue answers `cancelled`, and an unoffered kind rejects the run.
## Model Experience
@@ -56,3 +67,4 @@ None; this package neither assembles nor sends a provider request.
- **Session harvest requires raw JSONL mode** — `runScenario` collects persisted `.jsonl` logs, so snapshot configs set `persistenceCompression: 'none'`; compressed JSONL and SQLite compositions have no snapshot-harvest path.
- **Built mode requires current artifacts** — run `pnpm run build` before selecting `DSH_EXAMPLE_MODE=lib`; source mode remains the zero-build path.
- **Backend coverage still rides an ACP driver** — see the [automation-only ACP decision](../../../.agents/notes/implemented/simplification/2026-07-23-acp-automation-only-protocol.md#snapshot-boundary) for why retained scenarios use this transport.

View File

@@ -25,8 +25,6 @@ import { setTimeout as delay } from 'node:timers/promises'
import {
ClientSideConnection,
PROTOCOL_VERSION,
type CreateElicitationRequest,
type CreateElicitationResponse,
type RequestPermissionRequest,
type RequestPermissionResponse,
type SessionNotification,
@@ -44,18 +42,19 @@ const WAIT_POLL_INTERVAL_MS = 10
* (random) session id into a `{{sessionId}}` variable that later steps
* reference, since a committed file cannot know the id in advance.
*
* `promptAndCancel` starts a prompt without awaiting completion, waits until
* the client observes the selected update (`agent_message_chunk` by default),
* then cancels and awaits completion. An optional `waitForFile` first observes
* a cwd-relative readiness marker, and a named `waitForToolCallUpdate` keeps
* the step open for a terminal tool update that may follow the prompt response.
* `promptAndCancel` starts a prompt without awaiting completion, waits for a
* readiness condition, then cancels and awaits completion. `waitForFile`
* observes a cwd-relative marker; the default observes the durable turn start.
* `promptAndWaitForAgentMessage` arms an exact text-chunk waiter before sending
* the prompt, then keeps the application live until that later update arrives.
* `waitForTurnEnd` holds the subprocess open until the selected session's latest
* complete raw-JSONL turn boundary is `turn/end`; its timeout defaults to 10s.
* `waitForTurnStart` waits for an open durable turn, optionally at or beyond a
* specified turn number. `waitForTurnEnd` holds the subprocess open until the
* selected session's latest complete raw-JSONL turn boundary is `turn/end`.
* A standalone `cancel` may also wait for a cwd-relative readiness marker.
* All wait timeouts default to 10s.
*/
export type InputStep =
| { op: 'initialize'; terminalOutput?: boolean }
| { op: 'initialize' }
| { op: 'newSession' }
| { op: 'newSessionExpectError'; additionalDirectories?: string[] }
| { op: 'prompt'; text: string }
@@ -64,16 +63,11 @@ export type InputStep =
| {
op: 'promptAndCancel'
text: string
afterUpdate?: 'agent_message_chunk' | 'tool_call'
waitForFile?: { path: string; timeoutMs?: number }
waitForToolCallUpdate?: string
}
| { op: 'waitForTurnStart'; minimumTurn?: number; timeoutMs?: number }
| { op: 'waitForTurnEnd'; timeoutMs?: number }
| { op: 'cancel' }
| { op: 'setMode'; modeId: string }
| { op: 'setModeExpectError'; modeId: string }
| { op: 'setConfigOption'; configId: string; value: string }
| { op: 'setConfigOptionExpectError'; configId: string; value: string }
| { op: 'cancel'; waitForFile?: { path: string; timeoutMs?: number } }
/** A scenario's `input.json`: an ordered list of input steps. */
export interface InputScript {
@@ -86,21 +80,11 @@ export interface InputScript {
* kind → the offered `optionId` at answer time. A request beyond the queue
* (or with no queue at all) is answered `cancelled` — the stub behavior a
* scenario without approvals relies on. A scripted kind the request does
* not offer REJECTS the run: the scenario scripted an impossible click,
* not offer REJECTS the run: the scenario scripted an impossible selection,
* and {@link runScenario} throws once the in-flight step settles (the
* agent itself just sees `cancelled`, so it cannot absorb the bug).
*/
permissionAnswers?: PermissionAnswer[]
/**
* Ordered answers for the agent's `elicitation/create` round-trips (the
* ask_user_question / plan-review forms), consumed FIFO — the Nth request
* gets the Nth answer. Exhaustion (or no queue) answers `cancel`, the same
* fail-closed stub an elicitation-free scenario relies on. Unlike permission
* kinds, the scripted strings are not validated against the offered form —
* a stray `choice` reaches the agent verbatim, which reads it as a custom
* (non-consenting) answer, so a scenario bug fails safe in the transcript.
*/
elicitationAnswers?: ElicitationAnswer[]
}
/** One scripted answer to a permission request: which offered option kind to select. */
@@ -109,16 +93,6 @@ export interface PermissionAnswer {
kind: 'allow_once' | 'allow_always' | 'reject_once' | 'reject_always'
}
/** One scripted answer to an elicitation form (accept with choice/custom content, or cancel). */
export interface ElicitationAnswer {
/** Accept the form with the content below, or cancel it. */
action: 'accept' | 'cancel'
/** The selected option label (the form's `choice` field). */
choice?: string
/** Free-form text (the form's `custom` field). */
custom?: string
}
/** One harvested session log plus the identifying facts off its header line. */
export interface HarvestedLog {
/** The recorded session id (header `id`). */
@@ -158,6 +132,8 @@ export interface RunOptions {
agent: AgentUnderTest
/** `replay` (default, keyless) or `record` (real API, harvests the log). */
mode: 'replay' | 'record'
/** Scenario-specific deployment environment layered into the subprocess. */
env?: NodeJS.ProcessEnv
/** The recorded session JSONL fixture path (replay reads it; record writes near it). */
fixtureFile: string
/** Optional sidecar override path (replay). */
@@ -244,6 +220,7 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
await cp(opts.workspaceDir, cwd, { recursive: true })
}
const env: NodeJS.ProcessEnv = {
...opts.env,
DSH_SNAPSHOT: opts.mode,
DSH_SNAPSHOT_FILE: opts.fixtureFile,
DSH_SNAPSHOT_SESSIONS_ROOT: sessionsRoot,
@@ -259,13 +236,11 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
// Permission answers are consumed FIFO across the whole run; exhaustion
// falls back to `cancelled` so approval-free scenarios keep the plain stub.
const permissionQueue = [...input.permissionAnswers ?? []]
// Elicitation answers mirror the permission queue: FIFO, cancel on exhaustion.
const elicitationQueue = [...input.elicitationAnswers ?? []]
// A scenario bug detected inside a client callback (a scripted permission
// kind the agent never offered). It cannot fail the run from in there: a
// callback throw only becomes a JSON-RPC error RESPONSE to the agent, and
// a tolerant agent treats that as a denial and carries on — the run (or
// worse, a record) would absorb the impossible click silently. So the
// worse, a record) would absorb the impossible selection silently. So the
// callback answers `cancelled` (a well-defined path for the agent),
// captures the error here, and the step loop fails the run on it.
let scriptError: Error | undefined
@@ -279,7 +254,7 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
if (answer === undefined) return Promise.resolve({ outcome: { outcome: 'cancelled' } })
const option = params.options.find(o => o.kind === answer.kind)
if (option === undefined) {
// The scenario scripted a click the agent never offered — a scenario
// The scenario scripted a selection the agent never offered — a scenario
// bug. Captured (last one wins; same bug class either way) and
// answered `cancelled`; the step loop rejects the run on it.
scriptError = new Error(
@@ -290,17 +265,6 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
}
return Promise.resolve({ outcome: { outcome: 'selected', optionId: option.optionId } })
},
createElicitation(_params: CreateElicitationRequest): Promise<CreateElicitationResponse> {
const answer = elicitationQueue.shift()
if (answer === undefined || answer.action !== 'accept') return Promise.resolve({ action: 'cancel' })
return Promise.resolve({
action: 'accept',
content: {
...answer.choice !== undefined ? { choice: answer.choice } : {},
...answer.custom !== undefined ? { custom: answer.custom } : {},
},
})
},
})
const active = launched
await active.spawned
@@ -314,6 +278,7 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
match => active.waitForUpdate(match),
() => sessionId,
(id) => { sessionId = id },
(id, timeoutMs, minimumTurn) => waitForPersistedTurnStart(sessionsRoot, id, timeoutMs, minimumTurn),
(id, timeoutMs) => waitForPersistedTurnEnd(sessionsRoot, id, timeoutMs),
)
// A permission exchange happens while a step's request is in flight, so
@@ -386,13 +351,14 @@ async function runStep(
waitForUpdate: (match: (u: SessionNotification['update']) => boolean) => Promise<SessionNotification['update']>,
getSessionId: () => string | undefined,
setSessionId: (id: string) => void,
waitForTurnStart: (sessionId: string, timeoutMs?: number, minimumTurn?: number) => Promise<void>,
waitForTurnEnd: (sessionId: string, timeoutMs?: number) => Promise<void>,
): Promise<void> {
switch (step.op) {
case 'initialize':
await client.initialize({
protocolVersion: PROTOCOL_VERSION,
clientCapabilities: step.terminalOutput === true ? { _meta: { terminal_output: true } } : {},
clientCapabilities: {},
})
return
case 'newSession': {
@@ -435,7 +401,7 @@ async function runStep(
if (sessionId === undefined) throw new Error('snapshot-harness: promptExpectError before newSession')
// The model fails this turn (a recorded provider error), so the bridge
// answers the prompt with a JSON-RPC error and the SDK rejects. That
// rejection IS the expected editor experience — swallow it so the run
// rejection IS the expected protocol result — swallow it so the run
// completes and the stdout transcript (the error frame) is captured.
await client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
.then(() => { throw new Error('snapshot-harness: expected the prompt to fail but it succeeded') },
@@ -446,21 +412,16 @@ async function runStep(
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: promptAndCancel before newSession')
// Dispatch without awaiting because the fixture does not settle on its
// own. Waiting for the selected update pins it before cancellation and
// the cancelled prompt response in the transcript.
// own. Wait for an external readiness marker or the durable turn start
// before sending cancellation.
const promptDone = client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
const afterUpdate = step.afterUpdate ?? 'agent_message_chunk'
await waitForUpdate(u => u.sessionUpdate === afterUpdate)
if (step.waitForFile !== undefined) {
await waitForWorkspaceFile(cwd, step.waitForFile.path, step.waitForFile.timeoutMs)
} else {
await waitForTurnStart(sessionId)
}
// Arm this before cancellation so a fast tool drain cannot outrun the waiter.
const toolCallUpdateDone = step.waitForToolCallUpdate === undefined
? undefined
: waitForUpdate(u => u.sessionUpdate === 'tool_call_update' && u.toolCallId === step.waitForToolCallUpdate)
await client.cancel({ sessionId })
await promptDone
if (toolCallUpdateDone !== undefined) await toolCallUpdateDone
return
}
case 'waitForTurnEnd': {
@@ -469,53 +430,46 @@ async function runStep(
await waitForTurnEnd(sessionId, step.timeoutMs)
return
}
case 'waitForTurnStart': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: waitForTurnStart before newSession')
await waitForTurnStart(sessionId, step.timeoutMs, step.minimumTurn)
return
}
case 'cancel': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: cancel before newSession')
if (step.waitForFile !== undefined) {
await waitForWorkspaceFile(cwd, step.waitForFile.path, step.waitForFile.timeoutMs)
}
await client.cancel({ sessionId })
return
}
case 'setMode': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: setMode before newSession')
await client.setSessionMode({ sessionId, modeId: step.modeId })
return
}
case 'setModeExpectError': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: setModeExpectError before newSession')
// The bridge rejects an unknown/uncomposed mode id with invalidParams;
// that rejection IS the expected wire behavior — swallow it so the run
// completes and the error frame is captured in the transcript.
await client.setSessionMode({ sessionId, modeId: step.modeId }).then(
() => { throw new Error('snapshot-harness: expected session/set_mode to be rejected but it succeeded') },
() => { /* expected: the bridge rejected the mode id */ },
)
return
}
case 'setConfigOption': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: setConfigOption before newSession')
await client.setSessionConfigOption({ sessionId, configId: step.configId, value: step.value })
return
}
case 'setConfigOptionExpectError': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: setConfigOptionExpectError before newSession')
// The bridge rejects an unknown id / out-of-vocabulary value; the SDK
// surfaces that as a rejected RPC — swallow it so the run completes and
// the error frame is captured in the transcript.
await client.setSessionConfigOption({ sessionId, configId: step.configId, value: step.value }).then(
() => { throw new Error('snapshot-harness: expected set_config_option to be rejected but it succeeded') },
() => { /* expected: the bridge rejected the id or value */ },
)
return
}
default:
throw new Error(`snapshot-harness: unknown input op ${JSON.stringify(step)}`)
}
}
/** Wait until persistence exposes an open turn for the selected session. */
async function waitForPersistedTurnStart(
root: string,
sessionId: string,
timeoutMs = DEFAULT_WAIT_TIMEOUT_MS,
minimumTurn?: number,
): Promise<void> {
const deadline = Date.now() + timeoutMs
while (true) {
const log = (await harvestSessionLogs(root)).find(candidate => candidate.id === sessionId)
const openTurn = log === undefined ? undefined : latestOpenTurn(log.content)
if (openTurn !== undefined && (minimumTurn === undefined || openTurn >= minimumTurn)) return
if (Date.now() >= deadline) {
const detail = minimumTurn === undefined ? 'turn/start' : `turn/start at or beyond turn ${minimumTurn}`
throw new Error(`snapshot-harness: session "${sessionId}" did not persist ${detail} within ${timeoutMs}ms`)
}
await delay(WAIT_POLL_INTERVAL_MS)
}
}
/**
* Wait until the raw JSONL backend exposes one complete closing turn boundary.
* The ACP cancel notification settles its prompt before the agent necessarily
@@ -561,6 +515,20 @@ function latestTurnIsClosed(content: string): boolean {
> complete.lastIndexOf('\n{"type":"turn/start",')
}
/** Return the latest open turn number, validating the persisted boundary record. */
function latestOpenTurn(content: string): number | undefined {
const complete = content.slice(0, content.lastIndexOf('\n') + 1)
const start = complete.lastIndexOf('\n{"type":"turn/start",')
if (start <= complete.lastIndexOf('\n{"type":"turn/end",')) return undefined
const end = complete.indexOf('\n', start + 1)
const record = JSON.parse(complete.slice(start + 1, end)) as { data?: { turn?: unknown } | null }
const turn = record.data?.turn
if (!Number.isSafeInteger(turn) || (turn as number) < 1) {
throw new Error('snapshot-harness: invalid persisted turn/start record')
}
return turn as number
}
/**
* Harvest EVERY persisted `.jsonl` session log under a sessions root, parse each
* header line, and return them ordered primary-first: the top-level session (no

View File

@@ -18,7 +18,6 @@
export {
runScenario,
type ElicitationAnswer,
type HarvestedLog,
type InputScript,
type InputStep,

View File

@@ -15,8 +15,6 @@ import {
ndJsonStream,
type Agent as AcpAgent,
type Client,
type CreateElicitationRequest,
type CreateElicitationResponse,
type RequestPermissionRequest,
type RequestPermissionResponse,
type SessionNotification,
@@ -49,8 +47,6 @@ export interface AcpTestLaunchOptions {
env?: NodeJS.ProcessEnv
/** Permission handler; omitted requests fail closed as `cancelled`. */
requestPermission?: (params: RequestPermissionRequest) => Promise<RequestPermissionResponse>
/** Elicitation handler; omitted requests fail closed as `cancel`. */
createElicitation?: (params: CreateElicitationRequest) => Promise<CreateElicitationResponse>
}
/** A running ACP test process and its captured client-side surfaces. */
@@ -156,8 +152,6 @@ export function launchAcpTestAgent(options: AcpTestLaunchOptions): LaunchedAcpTe
}
const requestPermission = options.requestPermission
?? (() => Promise.resolve({ outcome: { outcome: 'cancelled' as const } }))
const createElicitation = options.createElicitation
?? (() => Promise.resolve({ action: 'cancel' as const }))
const makeClient = (_agent: AcpAgent): Client => ({
sessionUpdate(params: SessionNotification): Promise<void> {
return trackClientCallback(() => {
@@ -181,7 +175,6 @@ export function launchAcpTestAgent(options: AcpTestLaunchOptions): LaunchedAcpTe
})
},
requestPermission: params => trackClientCallback(() => requestPermission(params)),
unstable_createElicitation: params => trackClientCallback(() => createElicitation(params)),
})
const client = new ClientSideConnection(makeClient, stream)
// `exit` only reports the parent process's status. Descendants may retain

View File

@@ -11,7 +11,6 @@ const CWD = '{{cwd}}'
const SYSTEM = '{{system}}'
const TOOLS = '{{tools}}'
const MESSAGE_PREFIX = '{{messagePrefix}}'
const UPDATED_AT = '{{updatedAt}}'
/** A cwd-rooted path after volatile cwd replacement, through its last separator-delimited segment. */
const CWD_ROOTED_PATH_RE = /\{\{cwd\}\}(?:[\\/][^\s<>"'`]+)+/g
@@ -130,8 +129,6 @@ export function normalizeStdout(
if ('id' in frame && frame.id !== undefined && frame.id !== null) {
frame.id = stableId(frame.id)
}
const update = (frame.params as { update?: Record<string, unknown> } | undefined)?.update
if (update?.sessionUpdate === 'session_info_update') update.updatedAt = UPDATED_AT
return scrubValue(frame, ctx, cwdPathMode) as Record<string, unknown>
})
return frames.map(f => JSON.stringify(f)).join('\n') + '\n'

View File

@@ -47,6 +47,8 @@ const PACKED_CHUNK_ROW_TYPES = new Set(['text-chunks', 'reasoning-chunks', 'tool
/** A snapshot scenario and how its fixtures are produced. */
export interface Scenario {
name: string
/** Deployment environment for this scenario's subprocess. */
env?: NodeJS.ProcessEnv
/** Whether the scenario drives at least one model turn (so a JSONL expected output applies). */
hasModelTurn: boolean
/**
@@ -604,6 +606,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
agent,
mode: childMode,
fixtureFile: join(dir, 'session.jsonl'),
...scenario.env !== undefined ? { env: scenario.env } : {},
...existsSync(overrideFile) ? { overrideFile } : {},
// In REPLAY, forward the recorded child fixtures so each subagent session
// replays from its own script. In RECORD they are harvested, not read.

View File

@@ -45,18 +45,10 @@ interface Behavior {
rejectExtraDirs?: boolean
/** How `session/prompt` settles: a clean response, a JSON-RPC error, or a hang until `session/cancel`. */
prompt?: 'respond' | 'error' | 'hang-until-cancel'
/** Emit a tool call instead of a message chunk before parking a cancellable prompt. */
cancelAtToolCall?: boolean
/** Emit the parked tool call's terminal update after answering cancellation. */
cancelToolCallUpdate?: boolean
/** Persist the scripted logs while handling cancellation, before stdin EOF. */
persistLogsOnCancel?: boolean
/** Before responding to a prompt, send a `session/request_permission` request and echo its outcome as a chunk. */
permissionProbe?: boolean
/** Before responding to a prompt, send an `elicitation/create` request and echo its response as a chunk. */
elicitationProbe?: boolean
/** How `session/set_mode` settles: an empty response (echoing the modeId as a chunk) or a JSON-RPC error. */
setMode?: 'respond' | 'error'
/** Echo the `DSH_SNAPSHOT_*` env the harness set as a chunk (spec-side env-plumbing assertions). */
echoEnv?: boolean
/** Echo the sorted cwd listing as a chunk (spec-side workspace-seeding assertions). */
@@ -73,13 +65,6 @@ interface Behavior {
strayBucketFile?: boolean
/** Delete the sessions root entirely (harvest must yield no logs). */
deleteSessionsRoot?: boolean
/**
* Vocabulary for `session/set_config_option`: allowed values per config id.
* A set naming an unknown id or an out-of-vocabulary value rejects (the
* real bridge's rule); a valid set answers with the complete refreshed
* option state, `currentValue` updated. Absent: every set rejects.
*/
configOptions?: Record<string, string[]>
}
const sessionsRoot = process.env.DSH_SNAPSHOT_SESSIONS_ROOT ?? ''
@@ -102,10 +87,10 @@ let sessionId = ''
let sessionCwd = ''
/** The parked prompt request id while `hang-until-cancel` waits for the cancel notification. */
let parkedPromptId: number | string | null = null
/** Resolvers for outbound probe responses (permission/elicitation), keyed by request id. */
/** The transient raw JSONL log that proves the parked turn started durably. */
let parkedTurnLog: string | undefined
/** Resolvers for outbound permission responses, keyed by request id. */
const pendingOutbound = new Map<number, (result: unknown) => void>()
/** Per-run `session/set_config_option` state: config id → current value (first vocabulary entry until set). */
const currentConfig: Record<string, string> = {}
function send(frame: Record<string, unknown>): void {
process.stdout.write(`${JSON.stringify({ jsonrpc: '2.0', ...frame })}\n`)
@@ -138,39 +123,34 @@ function instantiate(value: unknown): unknown {
return value
}
/** Persist an open turn so cancellation tests wait on agent state, not presentation output. */
function persistParkedTurnStart(): void {
parkedTurnLog = join(sessionsRoot, 'ready', 'open.jsonl')
mkdirSync(dirname(parkedTurnLog), { recursive: true })
writeFileSync(parkedTurnLog, [
JSON.stringify({ type: 'session', version: 0, id: sessionId, createdAt: 1, cwd: sessionCwd, delegationDepth: 0 }),
JSON.stringify({ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } }),
'',
].join('\n'))
}
/** Remove the transient open-turn log before publishing any scripted final logs. */
function clearParkedTurnStart(): void {
if (parkedTurnLog === undefined) return
rmSync(parkedTurnLog, { force: true })
parkedTurnLog = undefined
}
async function handlePrompt(id: number | string): Promise<void> {
if ((behavior.prompt ?? 'respond') === 'hang-until-cancel') {
// A thought chunk BEFORE any message chunk: a promptAndCancel waiter
// watches for agent_message_chunk, so this exercises its non-matching
// update path while the waiter is armed.
send({
method: 'session/update',
params: { sessionId, update: { sessionUpdate: 'agent_thought_chunk', content: { type: 'text', text: 'mulling' } } },
})
}
if (behavior.cancelAtToolCall === true) {
send({
method: 'session/update',
params: {
sessionId,
update: {
sessionUpdate: 'tool_call',
toolCallId: 'call_fake_1',
title: 'fake tool',
kind: 'execute',
status: 'in_progress',
},
},
})
} else {
chunk('thinking about it')
}
chunk('thinking about it')
if (behavior.echoEnv === true) {
chunk(`env:${JSON.stringify({
mode: process.env.DSH_SNAPSHOT,
override: process.env.DSH_SNAPSHOT_OVERRIDE ?? null,
childFiles: process.env.DSH_SNAPSHOT_CHILD_FILES ?? null,
spillRoot: process.env.DSH_SNAPSHOT_SPILL_ROOT ?? null,
// Scenario-supplied deployment env (the `Scenario.env` layering seam).
permissionMode: process.env.DSH_PERMISSION_MODE ?? null,
})}`)
}
if (behavior.echoWorkspace === true) {
@@ -185,7 +165,7 @@ async function handlePrompt(id: number | string): Promise<void> {
method: 'session/request_permission',
params: {
sessionId,
toolCall: { toolCallId: 'call_fake_1', title: 'fake tool', kind: 'execute', status: 'pending' },
toolCall: { toolCallId: 'call_fake_1' },
options: [
{ optionId: 'opt-allow', name: 'Allow once', kind: 'allow_once' },
{ optionId: 'opt-reject', name: 'Reject once', kind: 'reject_once' },
@@ -195,23 +175,6 @@ async function handlePrompt(id: number | string): Promise<void> {
})
chunk(`permission:${JSON.stringify((result as { outcome?: unknown } | undefined)?.outcome ?? null)}`)
}
if (behavior.elicitationProbe === true) {
const requestId = nextOutboundId++
const result = await new Promise<unknown>((resolve) => {
pendingOutbound.set(requestId, resolve)
send({
id: requestId,
method: 'elicitation/create',
params: {
sessionId,
mode: 'form',
message: 'Approve this plan and leave plan mode?',
requestedSchema: { type: 'object', title: 'Plan review', properties: { choice: { type: 'string' }, custom: { type: 'string' } }, required: [] },
},
})
})
chunk(`elicitation:${JSON.stringify(result ?? null)}`)
}
switch (behavior.prompt ?? 'respond') {
case 'respond':
respond(id, { stopReason: 'end_turn' })
@@ -220,6 +183,7 @@ async function handlePrompt(id: number | string): Promise<void> {
respondError(id, 'model exploded')
return
case 'hang-until-cancel':
persistParkedTurnStart()
parkedPromptId = id
return
}
@@ -254,59 +218,13 @@ function handleFrame(frame: Record<string, unknown>): void {
case 'session/prompt':
void handlePrompt(id as number | string)
return
case 'session/set_mode':
if ((behavior.setMode ?? 'respond') === 'error') {
respondError(id as number | string, 'unknown mode')
return
}
chunk(`setMode:${String(params.modeId)}`)
respond(id as number | string, {})
return
case 'session/set_config_option': {
const vocabulary = behavior.configOptions
const configId = params.configId as string
const value = params.value as string
const values = vocabulary?.[configId]
if (values === undefined) {
respondError(id as number | string, `unknown config option ${configId}`)
return
}
if (!values.includes(value)) {
respondError(id as number | string, `unknown ${configId} value ${value}`)
return
}
currentConfig[configId] = value
// The real bridge's contract: every set answers with the COMPLETE
// refreshed option state, not just the changed entry.
respond(id as number | string, {
configOptions: Object.entries(vocabulary as Record<string, string[]>).map(([cid, vs]) => ({
id: cid,
type: 'select',
currentValue: currentConfig[cid] ?? vs[0],
options: vs.map(v => ({ value: v, name: v })),
})),
})
return
}
case 'session/cancel':
if (parkedPromptId !== null) {
const parked = parkedPromptId
parkedPromptId = null
respond(parked, { stopReason: 'cancelled' })
if (behavior.cancelToolCallUpdate === true) {
send({
method: 'session/update',
params: {
sessionId,
update: {
sessionUpdate: 'tool_call_update',
toolCallId: 'call_fake_1',
status: 'failed',
},
},
})
}
clearParkedTurnStart()
if (behavior.persistLogsOnCancel === true) writeLogs()
respond(parked, { stopReason: 'cancelled' })
}
return
default:
@@ -325,6 +243,7 @@ function writeLogs(): void {
}
function flushLogsAndExit(): void {
clearParkedTurnStart()
writeLogs()
if (behavior.strayRootFile === true) writeFileSync(join(sessionsRoot, 'stray.txt'), 'not a bucket\n')
if (behavior.strayBucketFile === true) {

View File

@@ -95,8 +95,8 @@ describe('runScenario', () => {
expect(clientClosed).toBe(true)
})
it('centralizes ACP boot, captures, updates, fail-closed interactions, and shutdown', { timeout: 20_000 }, async () => {
const { dir, fixtureFile } = await scenario({ permissionProbe: true, elicitationProbe: true, echoEnv: true, stderrNote: 'launcher stderr' })
it('centralizes ACP boot, captures, updates, fail-closed permissions, and shutdown', { timeout: 20_000 }, async () => {
const { dir, fixtureFile } = await scenario({ permissionProbe: true, echoEnv: true, stderrNote: 'launcher stderr' })
const sessionsRoot = await mkdtemp(join(tmpdir(), 'acp-launcher-sessions-'))
tempDirs.push(sessionsRoot)
const launched = launchAcpTestAgent({
@@ -112,6 +112,8 @@ describe('runScenario', () => {
await launched.client.initialize({ protocolVersion: PROTOCOL_VERSION, clientCapabilities: {} })
const { sessionId } = await launched.client.newSession({ cwd: dir, mcpServers: [] })
const nextChunk = launched.waitForUpdate(update => update.sessionUpdate === 'agent_message_chunk')
const laterChunk = launched.waitForUpdate(update => update.sessionUpdate === 'agent_message_chunk'
&& update.content.type === 'text' && update.content.text === 'never this one')
const predicateFailure = new Error('predicate failed')
const failedPredicate = launched.waitForUpdate(() => { throw predicateFailure })
.catch((error: unknown): unknown => error)
@@ -120,8 +122,8 @@ describe('runScenario', () => {
expect((await nextChunk).sessionUpdate).toBe('agent_message_chunk')
expect(launched.updates.some(update => update.sessionUpdate === 'agent_message_chunk')).toBe(true)
expect(launched.rawStdout()).toContain('permission:{\\"outcome\\":\\"cancelled\\"}')
expect(launched.rawStdout()).toContain('elicitation:{\\"action\\":\\"cancel\\"}')
expect(launched.stderr()).toContain('launcher stderr')
void laterChunk.catch(() => undefined)
const unmatched = expect(launched.waitForUpdate(() => false)).rejects.toThrow(/update stream closed/)
await launched.close()
await unmatched
@@ -374,7 +376,7 @@ describe('runScenario', () => {
}
})
it('drives a full turn: initialize (terminal caps), session, prompt, permission stub, harvest', { timeout: 20_000 }, async () => {
it('drives a full turn: initialize, session, prompt, permission stub, harvest', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({
permissionProbe: true,
logs: [{
@@ -386,7 +388,7 @@ describe('runScenario', () => {
}],
})
const result = await runScenario(
{ steps: [{ op: 'initialize', terminalOutput: true }, { op: 'newSession' }, { op: 'prompt', text: 'go' }] },
{ steps: [{ op: 'initialize' }, { op: 'newSession' }, { op: 'prompt', text: 'go' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.sessionId).toBeDefined()
@@ -482,7 +484,7 @@ describe('runScenario', () => {
expect(child.startsWith(`..${sep}`)).toBe(false)
})
it('promptAndCancel waits for the streamed chunk, cancels, and settles the prompt', { timeout: 20_000 }, async () => {
it('promptAndCancel waits for the durable turn start, cancels, and settles the prompt', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ prompt: 'hang-until-cancel' })
const result = await runScenario(
{ steps: [...boot, { op: 'promptAndCancel', text: 'hang' }] },
@@ -539,28 +541,6 @@ describe('runScenario', () => {
expect(result.rawStdout).toContain('thinking about it')
})
it('promptAndCancel can bracket cancellation with tool-call updates', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({
prompt: 'hang-until-cancel',
cancelAtToolCall: true,
cancelToolCallUpdate: true,
})
const result = await runScenario(
{
steps: [...boot, {
op: 'promptAndCancel',
text: 'hang',
afterUpdate: 'tool_call',
waitForToolCallUpdate: 'call_fake_1',
}],
},
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.rawStdout).toContain('"sessionUpdate":"tool_call"')
expect(result.rawStdout.indexOf('"sessionUpdate":"tool_call"')).toBeLessThan(result.rawStdout.indexOf('cancelled'))
expect(result.rawStdout.indexOf('cancelled')).toBeLessThan(result.rawStdout.indexOf('"sessionUpdate":"tool_call_update"'))
})
it('waitForTurnEnd holds cancellation open through the persisted closing boundary', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({
prompt: 'hang-until-cancel',
@@ -580,6 +560,108 @@ describe('runScenario', () => {
expect(result.sessionLogs[0]?.content).toContain('"type":"turn/end"')
})
it('waitForTurnStart can require a later durable turn before continuing', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({
prompt: 'hang-until-cancel',
persistLogsOnCancel: true,
logs: [{
file: 'bucket/session.jsonl',
lines: [
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 3 } },
],
}],
})
const result = await runScenario(
{
steps: [
...boot,
{ op: 'promptAndCancel', text: 'hang' },
{ op: 'waitForTurnStart', minimumTurn: 3 },
],
},
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.sessionLogs[0]?.content).toContain('"turn":3')
})
it('waitForTurnStart rejects missing, earlier, and malformed durable turns', { timeout: 20_000 }, async () => {
const missing = await scenario({})
await expect(runScenario(
{ steps: [...boot, { op: 'waitForTurnStart', timeoutMs: 20 }] },
{ agent: AGENT, mode: 'replay', fixtureFile: missing.fixtureFile },
)).rejects.toThrow(/did not persist turn\/start within 20ms/)
const earlier = await scenario({
prompt: 'hang-until-cancel',
persistLogsOnCancel: true,
logs: [{
file: 'bucket/session.jsonl',
lines: [
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } },
],
}],
})
await expect(runScenario(
{
steps: [
...boot,
{ op: 'promptAndCancel', text: 'hang' },
{ op: 'waitForTurnStart', minimumTurn: 3, timeoutMs: 20 },
],
},
{ agent: AGENT, mode: 'replay', fixtureFile: earlier.fixtureFile },
)).rejects.toThrow(/turn\/start at or beyond turn 3 within 20ms/)
const closed = await scenario({
prompt: 'hang-until-cancel',
persistLogsOnCancel: true,
logs: [{
file: 'bucket/session.jsonl',
lines: [
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } },
{ type: 'turn/end', seq: 1, time: 2, data: { turn: 1, reason: { kind: 'stop' } } },
],
}],
})
await expect(runScenario(
{
steps: [
...boot,
{ op: 'promptAndCancel', text: 'hang' },
{ op: 'waitForTurnStart', timeoutMs: 20 },
],
},
{ agent: AGENT, mode: 'replay', fixtureFile: closed.fixtureFile },
)).rejects.toThrow(/did not persist turn\/start within 20ms/)
for (const turn of [undefined, 0]) {
const malformed = await scenario({
prompt: 'hang-until-cancel',
persistLogsOnCancel: true,
logs: [{
file: 'bucket/session.jsonl',
lines: [
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
{ type: 'turn/start', seq: 0, time: 1, data: turn === undefined ? {} : { turn } },
],
}],
})
await expect(runScenario(
{
steps: [
...boot,
{ op: 'promptAndCancel', text: 'hang' },
{ op: 'waitForTurnStart' },
],
},
{ agent: AGENT, mode: 'replay', fixtureFile: malformed.fixtureFile },
)).rejects.toThrow('invalid persisted turn/start record')
}
})
it('waitForTurnEnd times out for a missing log and an open logged turn', { timeout: 20_000 }, async () => {
const missing = await scenario({})
await expect(runScenario(
@@ -695,15 +777,27 @@ describe('runScenario', () => {
expect(result.sessionId).toBeDefined()
})
it('a standalone cancel can wait for cwd-relative readiness', { timeout: 20_000 }, async () => {
const { dir, fixtureFile } = await scenario({})
const workspaceDir = join(dir, 'workspace')
const { mkdir } = await import('node:fs/promises')
await mkdir(workspaceDir, { recursive: true })
await writeFile(join(workspaceDir, 'ready'), '')
const result = await runScenario(
{ steps: [...boot, { op: 'cancel', waitForFile: { path: 'ready' } }] },
{ agent: AGENT, mode: 'replay', fixtureFile, workspaceDir },
)
expect(result.sessionId).toBeDefined()
})
it.each([
[{ op: 'prompt', text: 'x' }, /prompt before newSession/],
[{ op: 'promptAndWaitForAgentMessage', text: 'x', waitForText: 'later' }, /promptAndWaitForAgentMessage before newSession/],
[{ op: 'promptExpectError', text: 'x' }, /promptExpectError before newSession/],
[{ op: 'promptAndCancel', text: 'x' }, /promptAndCancel before newSession/],
[{ op: 'waitForTurnStart' }, /waitForTurnStart before newSession/],
[{ op: 'waitForTurnEnd' }, /waitForTurnEnd before newSession/],
[{ op: 'cancel' }, /cancel before newSession/],
[{ op: 'setConfigOption', configId: 'sandbox-mode', value: 'read-only' }, /setConfigOption before newSession/],
[{ op: 'setConfigOptionExpectError', configId: 'sandbox-mode', value: 'yolo' }, /setConfigOptionExpectError before newSession/],
] as [InputStep, RegExp][])('rejects %j before newSession', { timeout: 20_000 }, async (step, message) => {
const { fixtureFile } = await scenario({})
await expect(runScenario(
@@ -712,53 +806,6 @@ describe('runScenario', () => {
)).rejects.toThrow(message)
})
it('setConfigOption switches a value and receives the complete refreshed option state', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({
configOptions: { 'sandbox-mode': ['read-only', 'workspace-write'], 'approval-policy': ['ask', 'never'] },
})
const result = await runScenario(
{
steps: [...boot,
{ op: 'setConfigOption', configId: 'sandbox-mode', value: 'workspace-write' },
{ op: 'setConfigOption', configId: 'approval-policy', value: 'never' }],
},
{ agent: AGENT, mode: 'replay', fixtureFile },
)
// Every set answers with the FULL state: the second response carries the
// first switch's value too — the complete-refreshed-state contract.
const frames = result.rawStdout.trim().split('\n').map(line => JSON.parse(line) as { result?: { configOptions?: { id: string; currentValue: string }[] } })
const states = frames
.map(f => f.result?.configOptions)
.filter(options => options !== undefined)
.map(options => Object.fromEntries((options as { id: string; currentValue: string }[]).map(o => [o.id, o.currentValue])))
expect(states).toEqual([
{ 'sandbox-mode': 'workspace-write', 'approval-policy': 'ask' },
{ 'sandbox-mode': 'workspace-write', 'approval-policy': 'never' },
])
})
it('setConfigOptionExpectError swallows the rejection for unknown ids and out-of-vocabulary values', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ configOptions: { 'sandbox-mode': ['read-only'] } })
const result = await runScenario(
{
steps: [...boot,
{ op: 'setConfigOptionExpectError', configId: 'sandbox-mode', value: 'yolo' },
{ op: 'setConfigOptionExpectError', configId: 'reasoning-effort', value: 'max' }],
},
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.rawStdout).toContain('unknown sandbox-mode value yolo')
expect(result.rawStdout).toContain('unknown config option reasoning-effort')
})
it('setConfigOptionExpectError throws when the set unexpectedly succeeds', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ configOptions: { 'sandbox-mode': ['read-only'] } })
await expect(runScenario(
{ steps: [...boot, { op: 'setConfigOptionExpectError', configId: 'sandbox-mode', value: 'read-only' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/expected set_config_option to be rejected/)
})
it('rejects an unknown input op', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({})
const bogus = { op: 'reticulate' } as unknown as InputStep
@@ -814,69 +861,6 @@ describe('runScenario', () => {
expect(result.sessionLogs).toHaveLength(0)
})
it('drives session/set_mode and swallows the expected rejection of setModeExpectError', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({})
const result = await runScenario(
{ steps: [...boot, { op: 'setMode', modeId: 'plan' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.rawStdout).toContain('setMode:plan')
const rejecting = await scenario({ setMode: 'error' })
const rejected = await runScenario(
{ steps: [...boot, { op: 'setModeExpectError', modeId: 'yolo' }] },
{ agent: AGENT, mode: 'replay', fixtureFile: rejecting.fixtureFile },
)
expect(rejected.rawStdout).toContain('unknown mode')
})
it('fails the run when setModeExpectError unexpectedly succeeds, and both mode ops require a session', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({})
await expect(runScenario(
{ steps: [...boot, { op: 'setModeExpectError', modeId: 'plan' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/expected session\/set_mode to be rejected/)
await expect(runScenario(
{ steps: [{ op: 'initialize' }, { op: 'setMode', modeId: 'plan' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/setMode before newSession/)
await expect(runScenario(
{ steps: [{ op: 'initialize' }, { op: 'setModeExpectError', modeId: 'plan' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/setModeExpectError before newSession/)
})
it('answers elicitations from the scripted queue, falling back to cancel on exhaustion', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ elicitationProbe: true })
// Three prompts → three elicitations: an accept-with-choice, an
// accept-with-custom (feedback), then the exhausted-queue cancel.
const result = await runScenario(
{
steps: [...boot, { op: 'prompt', text: 'one' }, { op: 'prompt', text: 'two' }, { op: 'prompt', text: 'three' }],
elicitationAnswers: [
{ action: 'accept', choice: 'Approve' },
{ action: 'accept', custom: 'add tests first' },
],
},
{ agent: AGENT, mode: 'replay', fixtureFile },
)
const first = result.rawStdout.indexOf('elicitation:{\\"action\\":\\"accept\\",\\"content\\":{\\"choice\\":\\"Approve\\"}}')
const second = result.rawStdout.indexOf('elicitation:{\\"action\\":\\"accept\\",\\"content\\":{\\"custom\\":\\"add tests first\\"}}')
const third = result.rawStdout.indexOf('elicitation:{\\"action\\":\\"cancel\\"}')
expect(first).toBeGreaterThanOrEqual(0)
expect(second).toBeGreaterThan(first)
expect(third).toBeGreaterThan(second)
})
it('a scripted elicitation cancel answers cancel', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ elicitationProbe: true })
const result = await runScenario(
{ steps: [...boot, { op: 'prompt', text: 'one' }], elicitationAnswers: [{ action: 'cancel' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.rawStdout).toContain('elicitation:{\\"action\\":\\"cancel\\"}')
})
it('answers permission requests from the scripted queue by option kind, falling back to cancelled', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ permissionProbe: true })
// Two prompts → two permission round-trips; one scripted answer, so the

View File

@@ -123,24 +123,6 @@ Additional instructions from: nested\AGENTS.md`,
expect(out).not.toContain('"id"')
})
it('stabilizes the timestamp carried by session title updates', () => {
const raw = JSON.stringify({
jsonrpc: '2.0',
method: 'session/update',
params: {
sessionId: ctx.sessionIds[0],
update: {
sessionUpdate: 'session_info_update',
title: 'Stable title',
updatedAt: '2026-07-20T17:03:13.689Z',
},
},
})
const out = normalizeStdout(raw, ctx)
expect(out).toContain('"updatedAt":"{{updatedAt}}"')
expect(out).not.toContain('2026-07-20T17:03:13.689Z')
})
it('throws on a non-JSON stdout line (the purity check)', () => {
const raw = `${JSON.stringify({ jsonrpc: '2.0', id: 1 })}\noops a log leaked\n`
expect(() => normalizeStdout(raw, ctx)).toThrow()

View File

@@ -53,6 +53,7 @@ const REPLAY_SCENARIOS: Scenario[] = [
hasModelTurn: true,
recorded: true,
headerClass: 'main',
env: { DSH_PERMISSION_MODE: 'never' },
configPath: AGENT.configPath,
workspaceParent: tmpdir(),
},
@@ -130,6 +131,8 @@ describe('defineAcpSnapshotSuite: refresh write-back', () => {
expect(stdout).not.toContain('stale stdout')
expect(stdout).toContain('env:{\\"mode\\":\\"replay\\"')
expect(stdout).not.toContain('\\"mode\\":\\"refresh\\"')
// The scenario's own env layer reached the subprocess.
expect(stdout).toContain('\\"permissionMode\\":\\"never\\"')
const blocked = readFileSync(join(refreshDir, 'blocked-log', 'session.jsonl'), 'utf8')
expect(blocked).toContain('"decision":"block"')

View File

@@ -1,6 +1,6 @@
# @deepseek-ai/dsh-llm-replay
A replay LLM plugin for keyless snapshot tests. It yields model streams reconstructed from a recorded **session JSONL** fixture, so a test can boot the real agent against a fixed model transcript with no API key. With `providers` configured it registers a replay-only adapter whose catalog is visible to clients such as ACP editors; without `providers` it installs the catch-all `llm/stream` waterfall used by tests that do not need discovery.
A replay LLM plugin for keyless snapshot tests. It yields model streams reconstructed from a recorded **session JSONL** fixture, so a test can boot the real agent against a fixed model transcript with no API key. With `providers` configured it registers a replay-only adapter whose catalog is available to scenarios that exercise model discovery; without `providers` it installs the catch-all `llm/stream` waterfall used by tests that do not need discovery.
Its consumers are the ACP snapshot harness in `examples/acp-agent` and the `stream-json` snapshot in `examples/headless-agent`; each loads this plugin in place of a real LLM adapter. Keeping derivation and replay here places that logic under the per-file 100% coverage gate on `packages/*/src`.
@@ -8,7 +8,7 @@ Its consumers are the ACP snapshot harness in `examples/acp-agent` and the `stre
The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`", done by the snapshot harness — this plugin does not record. A fixture may carry its `request/header` content tokenized to `{{system}}`/`{{tools}}` (the harness pins that content in one scenario and scrubs the rest); replay is indifferent — derivation reads only `assistant/chunk` events and the line-0 session header.
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`: a `ReplayEntry[]`) that REPLACES the derived script.
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`: a `ReplayEntry[]`) that REPLACES the derived script. A `hang` entry may name `readyFile`; replay writes that empty marker after its prefix chunks reach the loop and before it waits for cancellation, so an external driver can cancel deterministically without observing a presentation update.
## Nested agents: per-session keying

View File

@@ -6,7 +6,7 @@
* @module @deepseek-ai/dsh-llm-replay
*/
import { existsSync, readFileSync } from 'node:fs'
import { existsSync, readFileSync, writeFileSync } from 'node:fs'
import { delimiter as pathDelimiter } from 'node:path'
import type { Context } from 'cordis'
import { decodeStorageRecord } from '@deepseek-ai/dsh-session'
@@ -22,7 +22,11 @@ import { LlmAdapter, LlmError, assertNever } from '@deepseek-ai/dsh-llm'
export type ReplayEntry =
| { kind: 'chunks'; chunks: StreamChunk[] }
| { kind: 'throw'; chunks: StreamChunk[]; message: string; code: string }
| { kind: 'hang' }
| {
kind: 'hang'
/** Optional marker written after the prefix chunks are consumed and before the stream waits for cancellation. */
readyFile?: string
}
/** One model exposed by a replay-only provider catalog. */
export interface ReplayModelConfig {
@@ -42,7 +46,7 @@ export interface ReplayProviderConfig {
id: string
/** Selector label; defaults to {@link id}. */
name?: string
/** Advisory models exposed to clients such as ACP editors. */
/** Advisory models exposed to replay scenarios that exercise discovery. */
models?: ReplayModelConfig[]
}
@@ -301,6 +305,7 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined)
// chunk, then wait for abort and surface it as the consumer expects.
yield { type: 'block-start', index: 0, blockType: 'text' }
yield { type: 'text-delta', index: 0, text: 'partial' }
if (entry.readyFile !== undefined) writeFileSync(entry.readyFile, '')
await new Promise<void>((_resolve, reject) => {
if (signal?.aborted) { reject(new Error('aborted')); return }
signal?.addEventListener('abort', () => { reject(new Error('aborted')) }, { once: true })

View File

@@ -1,4 +1,4 @@
import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'
import { existsSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
@@ -379,7 +379,8 @@ describe('installLlmReplay (through the real LlmService)', () => {
it('rejects a hang entry when the signal fires DURING the wait (abort listener path)', async () => {
writeFileSync(file, sessionJsonl([]), 'utf8')
const overrideFile = join(dir, 'replay.override.json')
writeFileSync(overrideFile, JSON.stringify([{ kind: 'hang' }]), 'utf8')
const readyFile = join(dir, 'stream-ready')
writeFileSync(overrideFile, JSON.stringify([{ kind: 'hang', readyFile }]), 'utf8')
const ctx = new Context()
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file, overrideFile })
@@ -392,6 +393,7 @@ describe('installLlmReplay (through the real LlmService)', () => {
expect((await iterator.next()).value).toMatchObject({ type: 'text-delta' })
const pending = iterator.next()
await new Promise(r => setImmediate(r))
expect(existsSync(readyFile)).toBe(true)
controller.abort()
await expect(pending).rejects.toThrow('aborted')
})