Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into xtr/agent-loop-message-machine

Browse Source 
# Conflicts:
#	.agents/notes/implemented/feature/2026-06-14-acp-agent-client-protocol.md
#	.agents/notes/implemented/feature/2026-06-14-acp-agent-client-protocol.zh.md
#	.agents/notes/implemented/feature/2026-06-30-hook-bridges.i18n.yaml
#	.agents/notes/implemented/feature/2026-06-30-interception-seams.i18n.yaml
#	.agents/notes/implemented/feature/2026-07-06-sandbox.i18n.yaml
#	.agents/notes/implemented/feature/2026-07-06-sandbox.md
#	.agents/notes/implemented/feature/2026-07-06-sandbox.zh.md
#	.agents/notes/implemented/feature/2026-07-19-model-facing-goal-tools.i18n.yaml
#	.agents/notes/implemented/feature/2026-07-19-same-session-goal-round-driver.i18n.yaml
#	.agents/notes/implemented/feature/2026-07-25-workspace-ui-product-flow.i18n.yaml
#	.agents/notes/implemented/simplification/2026-07-02-remove-stream-chunk-mirror.i18n.yaml
#	docs/architecture.i18n.yaml
#	docs/cookbook/adding-a-tool.i18n.yaml
#	docs/cookbook/extension-cookbook.i18n.yaml
#	docs/core-data-structures/llm-streaming.i18n.yaml
#	docs/core-data-structures/session.i18n.yaml
#	docs/core-data-structures/tools.i18n.yaml
#	docs/event-producer-consumer.md
#	docs/persistence-catalog.md
#	examples/acp-agent/tests/snapshots/cordis-inspect-jsdoc/stdout.expected.jsonl
#	examples/acp-agent/tests/snapshots/escalation-approved/session.jsonl
#	examples/acp-agent/tests/snapshots/escalation-rejected/session.jsonl
#	examples/acp-agent/tests/snapshots/fs-escalation-approved/session.jsonl
#	examples/acp-agent/tests/snapshots/hook-cc-pretool-ask/session.jsonl
#	examples/acp-agent/tests/snapshots/permission-switching/session.jsonl
#	examples/acp-agent/tests/snapshots/plan-mode-reject/session.jsonl
#	examples/acp-agent/tests/snapshots/plan-mode/session.jsonl
#	examples/acp-agent/tests/snapshots/session-sandbox-root/session.jsonl
#	packages/context/session-reference/README.md
#	packages/core/agent-loop/tests/agent.spec.ts
#	packages/hooks/hooks-claude/tests/coverage-cases.ts
#	packages/host/runtime/tests/host-runtime.spec.ts
#	packages/llm/llm-retry/tests/retry.spec.ts
#	packages/session-persistence/session-persistence/src/coordinator.ts
#	packages/support/acp-snapshot/README.md
#	packages/support/acp-snapshot/src/normalize.ts
#	packages/ui/acp/acp-feature-support.md
#	packages/ui/acp/src/codec.ts
#	packages/ui/acp/src/index.ts
#	packages/ui/acp/tests/bridge.spec.ts
#	packages/ui/acp/tests/codec.spec.ts
#	packages/ui/acp/tests/config-options.spec.ts
#	packages/ui/acp/tests/dispose.spec.ts
#	packages/ui/acp/tests/edges.spec.ts
#	packages/ui/acp/tests/stream-update.spec.ts
#	packages/ui/acp/tests/turns.spec.ts
This commit is contained in:
_Kerman
2026-07-26 14:05:33 +08:00
parent af3311e1f6 3ccdf4ac46
commit 57d90c343d
1062 changed files with 33621 additions and 30750 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
packages/support/README.md
View File

@@ -8,6 +8,7 @@ Packages that exist to serve development, testing, and the examples rather than
| `agent-loop-testkit/` | Shared prerequisite mounting for tests that exercise the concrete agent loop | (library — imported by AgentLoop integration tests) |
| `invariants/` | Runtime event-contract assertions for development diagnostics | (listens on `session/*`, `agent/*`) |
| `loader-smoke/` | Shared real-Loader subprocess harness for keyless example smokes | (library — imported by example e2e suites) |
| `llm-mock-server/` | Scriptable OpenAI-compatible HTTP/SSE fault server + CLI for LLM recovery tests | (standalone server and test library) |
| `llm-replay/` | Record/replay adapter: short-circuits `llm/stream` from a recorded session JSONL (keyless snapshot tests) | (listens on `llm/stream`) |
`invariants` is development support but has no environment guard: it runs wherever registered, and the default `dsh-agent-spine-demo` bundle mounts it unconditionally. `agent-loop-testkit` centralizes the mandatory service spine for hand-built AgentLoop tests without owning their loop or scenario. `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate. `acp-snapshot` carries the ACP subprocess/client boundary plus the snapshot harness, normalizers, and suite machinery, while `loader-smoke` owns the parallel real-Loader launch boundary used by keyless example e2e suites. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.
`invariants` is development support but has no environment guard: it runs wherever registered, and the default `dsh-agent-spine-demo` bundle mounts it unconditionally. `agent-loop-testkit` centralizes the mandatory service spine for hand-built AgentLoop tests without owning their loop or scenario. `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate, while `llm-mock-server` drives real provider adapters through deterministic HTTP/SSE faults. `acp-snapshot` carries the ACP subprocess/client boundary plus the snapshot harness, normalizers, and suite machinery, while `loader-smoke` owns the parallel real-Loader launch boundary used by keyless example e2e suites. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.

28
packages/support/acp-snapshot/README.md
View File

@@ -6,7 +6,7 @@ Four layers, importable separately:
- **`launchAcpTestAgent` (launcher)** — boots a source agent under tsx or a built `lib` agent under plain Node from a supplied cwd, connects the SDK client over a raw-byte stdout tee, collects session updates and stderr, surfaces asynchronous spawn failures through startup, fails closed on unhandled permission requests, and owns graceful or signalled shutdown. Shutdown waits for process exit, inherited stdio closure, and ACP parser exhaustion before resolving or propagating a child error, so captures are complete and callers can remove owned paths after either outcome. When Windows accepts forced termination but publishes its exit marker asynchronously, shutdown gives that marker a bounded grace before treating fallback refusal as a second failure. Snapshot and ordinary e2e suites share this process boundary; a test supplies only agent paths, cwd, environment overrides, and any permission policy.
- **`runScenario` (harness)** — drives ACP JSON-RPC stdio from a deterministic `input.json` script through the launcher, tees raw stdout for the expected-output and purity checks, and harvests every persisted raw JSONL session log (parent and subagent children, primary-first) after graceful stdin EOF. `AgentUnderTest` supplies absolute `binScript`, optional `libBinScript`, `configPath`, and `tsconfigPath` paths because the subprocess cwd is outside the repo; `workspaceParent` may move the generated child cwd from the platform temp directory when that grant is itself under test. Startup failures preserve captured agent stderr in the rejected diagnostic.
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; cwd-rooted separators selected as canonical `/` or host-native; `session_info_update.updatedAt` → `{{updatedAt}}`; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept, the same cwd-path policy), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}` outside each pin, structure kept — [pinned-header Agent Note](../../../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs and every native/JavaScript filesystem spelling of the generated cwd → tokens, longest-first; cwd-rooted separators selected as canonical `/` or host-native; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept, the same cwd-path policy), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}` outside each pin, structure kept — [pinned-header Agent Note](../../../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario expected-output and re-persisted-log comparisons, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, the per-header-class pin (`system-prompt.expected.md` plus `tool-schemas.expected.json`) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt/schema-scrubbed, non-pinning fixtures fully header-scrubbed). Refresh expands packed timing envelopes before aligning existing volatile event times, so switching between packed and unpacked layouts cannot shift later records; fresh chunk-fragment arrays remain authoritative. A newly inserted `session/title` receives its preceding event's time so feature-driven insertions do not churn the remainder of a fixture. Each scenario directory's `session.jsonl` plus contiguous `session.<n>.jsonl` siblings are the ordered primary/child inventory; the scenario table does not duplicate their count. Must be called at vitest collection time.
A consuming `*.snapshot.ts` is the scenario table plus one factory call:
@@ -14,7 +14,22 @@ A consuming `*.snapshot.ts` is the scenario table plus one factory call:
```ts
import { dirname, join } from 'node:path'
import { fileURLToPath } from 'node:url'
import { defineAcpSnapshotSuite, type Scenario } from '@deepseek-ai/dsh-acp-snapshot'
import {
defineAcpSnapshotSuite,
type Scenario,
type SnapshotSuiteOptions,
} from '@deepseek-ai/dsh-acp-snapshot'
function snapshotMode(value: string | undefined): SnapshotSuiteOptions['mode'] {
switch (value) {
case undefined:
case '':
case 'replay': return 'replay'
case 'record': return 'record'
case 'refresh': return 'refresh'
default: throw new Error(`unknown DSH_SNAPSHOT mode: ${value}`)
}
}
const SCENARIOS: Scenario[] = [
{ name: 'text-turn', hasModelTurn: true, recorded: true, pinsHeader: true },
@@ -28,11 +43,7 @@ defineAcpSnapshotSuite({
},
snapshotsDir: join(dirname(fileURLToPath(import.meta.url)), 'snapshots'),
scenarios: SCENARIOS, // exactly one entry per header class sets pinsHeader
mode: process.env.DSH_SNAPSHOT === 'record'
? 'record'
: process.env.DSH_SNAPSHOT === 'refresh'
? 'refresh'
: 'replay',
mode: snapshotMode(process.env.DSH_SNAPSHOT),
})
```
@@ -42,7 +53,7 @@ Every scenario compares `stdout.expected.jsonl` with cwd-rooted separators canon
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config Agent Note](../../../.agents/notes/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log expected outputs, and each pin's prompt and tool-schema sidecars from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md).
Constraints: `suite.ts` imports vitest, so the package entry is importable only inside a vitest run (the launcher, harness, and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug). Session config options are scriptable too: the `setConfigOption` step switches a knob over `session/set_config_option`, and `setConfigOptionExpectError` asserts the bridge rejects an unknown id or out-of-vocabulary value (the error frame stays in the transcript).
Constraints: `suite.ts` imports vitest, so the package entry is importable only inside a vitest run (the launcher, harness, and normalizers have no such dependency but ship from the same entry). The launcher and suite factory are ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection` — while the normalizers are transport-neutral session-log/text helpers also consumed by the TUI snapshot suite and the web browser e2e lane. Input scripts cover initialization, fresh-session creation, text prompting, cancellation, expected RPC failures, and durable turn-boundary waits. Permission round-trips are a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) mapped to the agent-issued `optionId`; an absent or exhausted queue answers `cancelled`, and an unoffered kind rejects the run.
## Model Experience
@@ -56,3 +67,4 @@ None; this package neither assembles nor sends a provider request.
- **Session harvest requires raw JSONL mode** — `runScenario` collects persisted `.jsonl` logs, so snapshot configs set `persistenceCompression: 'none'`; compressed JSONL and SQLite compositions have no snapshot-harvest path.
- **Built mode requires current artifacts** — run `pnpm run build` before selecting `DSH_EXAMPLE_MODE=lib`; source mode remains the zero-build path.
- **Backend coverage still rides an ACP driver** — see the [automation-only ACP decision](../../../.agents/notes/implemented/simplification/2026-07-23-acp-automation-only-protocol.md#snapshot-boundary) for why retained scenarios use this transport.

216
packages/support/acp-snapshot/src/harness.ts
View File

@@ -17,7 +17,7 @@
*/
import { cp, mkdtemp, readFile, readdir, rm } from 'node:fs/promises'
import { existsSync } from 'node:fs'
import { existsSync, realpathSync } from 'node:fs'
import { createHash } from 'node:crypto'
import { tmpdir } from 'node:os'
import { basename, dirname, join, delimiter } from 'node:path'
@@ -25,8 +25,6 @@ import { setTimeout as delay } from 'node:timers/promises'
import {
ClientSideConnection,
PROTOCOL_VERSION,
type CreateElicitationRequest,
type CreateElicitationResponse,
type RequestPermissionRequest,
type RequestPermissionResponse,
type SessionNotification,
@@ -44,18 +42,19 @@ const WAIT_POLL_INTERVAL_MS = 10
* (random) session id into a `{{sessionId}}` variable that later steps
* reference, since a committed file cannot know the id in advance.
*
* `promptAndCancel` starts a prompt without awaiting completion, waits until
* the client observes the selected update (`agent_message_chunk` by default),
* then cancels and awaits completion. An optional `waitForFile` first observes
* a cwd-relative readiness marker, and a named `waitForToolCallUpdate` keeps
* the step open for a terminal tool update that may follow the prompt response.
* `promptAndCancel` starts a prompt without awaiting completion, waits for a
* readiness condition, then cancels and awaits completion. `waitForFile`
* observes a cwd-relative marker; the default observes the durable turn start.
* `promptAndWaitForAgentMessage` arms an exact text-chunk waiter before sending
* the prompt, then keeps the application live until that later update arrives.
* `waitForTurnEnd` holds the subprocess open until the selected session's latest
* complete raw-JSONL turn boundary is `turn/end`; its timeout defaults to 10s.
* `waitForTurnStart` waits for an open durable turn, optionally at or beyond a
* specified turn number. `waitForTurnEnd` holds the subprocess open until the
* selected session's latest complete raw-JSONL turn boundary is `turn/end`.
* A standalone `cancel` may also wait for a cwd-relative readiness marker.
* All wait timeouts default to 10s.
*/
export type InputStep =
| { op: 'initialize'; terminalOutput?: boolean }
| { op: 'initialize' }
| { op: 'newSession' }
| { op: 'newSessionExpectError'; additionalDirectories?: string[] }
| { op: 'prompt'; text: string }
@@ -64,16 +63,11 @@ export type InputStep =
| {
op: 'promptAndCancel'
text: string
afterUpdate?: 'agent_message_chunk' | 'tool_call'
waitForFile?: { path: string; timeoutMs?: number }
waitForToolCallUpdate?: string
}
| { op: 'waitForTurnStart'; minimumTurn?: number; timeoutMs?: number }
| { op: 'waitForTurnEnd'; timeoutMs?: number }
| { op: 'cancel' }
| { op: 'setMode'; modeId: string }
| { op: 'setModeExpectError'; modeId: string }
| { op: 'setConfigOption'; configId: string; value: string }
| { op: 'setConfigOptionExpectError'; configId: string; value: string }
| { op: 'cancel'; waitForFile?: { path: string; timeoutMs?: number } }
/** A scenario's `input.json`: an ordered list of input steps. */
export interface InputScript {
@@ -86,21 +80,11 @@ export interface InputScript {
* kind → the offered `optionId` at answer time. A request beyond the queue
* (or with no queue at all) is answered `cancelled` — the stub behavior a
* scenario without approvals relies on. A scripted kind the request does
* not offer REJECTS the run: the scenario scripted an impossible click,
* not offer REJECTS the run: the scenario scripted an impossible selection,
* and {@link runScenario} throws once the in-flight step settles (the
* agent itself just sees `cancelled`, so it cannot absorb the bug).
*/
permissionAnswers?: PermissionAnswer[]
/**
* Ordered answers for the agent's `elicitation/create` round-trips (the
* ask_user_question / plan-review forms), consumed FIFO — the Nth request
* gets the Nth answer. Exhaustion (or no queue) answers `cancel`, the same
* fail-closed stub an elicitation-free scenario relies on. Unlike permission
* kinds, the scripted strings are not validated against the offered form —
* a stray `choice` reaches the agent verbatim, which reads it as a custom
* (non-consenting) answer, so a scenario bug fails safe in the transcript.
*/
elicitationAnswers?: ElicitationAnswer[]
}
/** One scripted answer to a permission request: which offered option kind to select. */
@@ -109,16 +93,6 @@ export interface PermissionAnswer {
kind: 'allow_once' | 'allow_always' | 'reject_once' | 'reject_always'
}
/** One scripted answer to an elicitation form (accept with choice/custom content, or cancel). */
export interface ElicitationAnswer {
/** Accept the form with the content below, or cancel it. */
action: 'accept' | 'cancel'
/** The selected option label (the form's `choice` field). */
choice?: string
/** Free-form text (the form's `custom` field). */
custom?: string
}
/** One harvested session log plus the identifying facts off its header line. */
export interface HarvestedLog {
/** The recorded session id (header `id`). */
@@ -141,6 +115,8 @@ export interface RunResult {
sessionId?: string
/** The generated cwd the session ran in (the bash workspace). */
cwd: string
/** Filesystem-resolved spellings of {@link cwd} that child processes may report. */
cwdAliases: string[]
/**
* Every persisted session log harvested after the run, ordered primary-first:
* the top-level (parent) session — the one with no `parentSession` — then each
@@ -156,6 +132,8 @@ export interface RunOptions {
agent: AgentUnderTest
/** `replay` (default, keyless) or `record` (real API, harvests the log). */
mode: 'replay' | 'record'
/** Scenario-specific deployment environment layered into the subprocess. */
env?: NodeJS.ProcessEnv
/** The recorded session JSONL fixture path (replay reads it; record writes near it). */
fixtureFile: string
/** Optional sidecar override path (replay). */
@@ -222,6 +200,7 @@ export function snapshotSpillRoot(
*/
export async function runScenario(input: InputScript, opts: RunOptions): Promise<RunResult> {
const cwd = await mkdtemp(join(opts.workspaceParent ?? tmpdir(), 'acp-snap-cwd-'))
const cwdAliases = [...new Set([realpathSync(cwd), realpathSync.native(cwd)])]
const sessionsRoot = await mkdtemp(join(tmpdir(), 'acp-snap-sessions-'))
// Fixed path length: spill-policy budgets the preview against the REAL path
// before stdout normalization, so tmpdir() length differences churn expected outputs.
@@ -241,6 +220,7 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
await cp(opts.workspaceDir, cwd, { recursive: true })
}
const env: NodeJS.ProcessEnv = {
...opts.env,
DSH_SNAPSHOT: opts.mode,
DSH_SNAPSHOT_FILE: opts.fixtureFile,
DSH_SNAPSHOT_SESSIONS_ROOT: sessionsRoot,
@@ -256,13 +236,11 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
// Permission answers are consumed FIFO across the whole run; exhaustion
// falls back to `cancelled` so approval-free scenarios keep the plain stub.
const permissionQueue = [...input.permissionAnswers ?? []]
// Elicitation answers mirror the permission queue: FIFO, cancel on exhaustion.
const elicitationQueue = [...input.elicitationAnswers ?? []]
// A scenario bug detected inside a client callback (a scripted permission
// kind the agent never offered). It cannot fail the run from in there: a
// callback throw only becomes a JSON-RPC error RESPONSE to the agent, and
// a tolerant agent treats that as a denial and carries on — the run (or
// worse, a record) would absorb the impossible click silently. So the
// worse, a record) would absorb the impossible selection silently. So the
// callback answers `cancelled` (a well-defined path for the agent),
// captures the error here, and the step loop fails the run on it.
let scriptError: Error | undefined
@@ -276,7 +254,7 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
if (answer === undefined) return Promise.resolve({ outcome: { outcome: 'cancelled' } })
const option = params.options.find(o => o.kind === answer.kind)
if (option === undefined) {
// The scenario scripted a click the agent never offered — a scenario
// The scenario scripted a selection the agent never offered — a scenario
// bug. Captured (last one wins; same bug class either way) and
// answered `cancelled`; the step loop rejects the run on it.
scriptError = new Error(
@@ -287,17 +265,6 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
}
return Promise.resolve({ outcome: { outcome: 'selected', optionId: option.optionId } })
},
createElicitation(_params: CreateElicitationRequest): Promise<CreateElicitationResponse> {
const answer = elicitationQueue.shift()
if (answer === undefined || answer.action !== 'accept') return Promise.resolve({ action: 'cancel' })
return Promise.resolve({
action: 'accept',
content: {
...answer.choice !== undefined ? { choice: answer.choice } : {},
...answer.custom !== undefined ? { custom: answer.custom } : {},
},
})
},
})
const active = launched
await active.spawned
@@ -311,6 +278,7 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
match => active.waitForUpdate(match),
() => sessionId,
(id) => { sessionId = id },
(id, timeoutMs, minimumTurn) => waitForPersistedTurnStart(sessionsRoot, id, timeoutMs, minimumTurn),
(id, timeoutMs) => waitForPersistedTurnEnd(sessionsRoot, id, timeoutMs),
)
// A permission exchange happens while a step's request is in flight, so
@@ -329,6 +297,7 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
rawStdout: launched.rawStdout(),
stderr: launched.stderr(),
cwd,
cwdAliases,
...sessionId !== undefined ? { sessionId } : {},
sessionLogs,
}
@@ -382,13 +351,14 @@ async function runStep(
waitForUpdate: (match: (u: SessionNotification['update']) => boolean) => Promise<SessionNotification['update']>,
getSessionId: () => string | undefined,
setSessionId: (id: string) => void,
waitForTurnStart: (sessionId: string, timeoutMs?: number, minimumTurn?: number) => Promise<void>,
waitForTurnEnd: (sessionId: string, timeoutMs?: number) => Promise<void>,
): Promise<void> {
switch (step.op) {
case 'initialize':
await client.initialize({
protocolVersion: PROTOCOL_VERSION,
clientCapabilities: step.terminalOutput === true ? { _meta: { terminal_output: true } } : {},
clientCapabilities: {},
})
return
case 'newSession': {
@@ -431,7 +401,7 @@ async function runStep(
if (sessionId === undefined) throw new Error('snapshot-harness: promptExpectError before newSession')
// The model fails this turn (a recorded provider error), so the bridge
// answers the prompt with a JSON-RPC error and the SDK rejects. That
// rejection IS the expected editor experience — swallow it so the run
// rejection IS the expected protocol result — swallow it so the run
// completes and the stdout transcript (the error frame) is captured.
await client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
.then(() => { throw new Error('snapshot-harness: expected the prompt to fail but it succeeded') },
@@ -442,21 +412,16 @@ async function runStep(
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: promptAndCancel before newSession')
// Dispatch without awaiting because the fixture does not settle on its
// own. Waiting for the selected update pins it before cancellation and
// the cancelled prompt response in the transcript.
// own. Wait for an external readiness marker or the durable turn start
// before sending cancellation.
const promptDone = client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
const afterUpdate = step.afterUpdate ?? 'agent_message_chunk'
await waitForUpdate(u => u.sessionUpdate === afterUpdate)
if (step.waitForFile !== undefined) {
await waitForWorkspaceFile(cwd, step.waitForFile.path, step.waitForFile.timeoutMs)
} else {
await waitForTurnStart(sessionId)
}
// Arm this before cancellation so a fast tool drain cannot outrun the waiter.
const toolCallUpdateDone = step.waitForToolCallUpdate === undefined
? undefined
: waitForUpdate(u => u.sessionUpdate === 'tool_call_update' && u.toolCallId === step.waitForToolCallUpdate)
await client.cancel({ sessionId })
await promptDone
if (toolCallUpdateDone !== undefined) await toolCallUpdateDone
return
}
case 'waitForTurnEnd': {
@@ -465,53 +430,46 @@ async function runStep(
await waitForTurnEnd(sessionId, step.timeoutMs)
return
}
case 'waitForTurnStart': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: waitForTurnStart before newSession')
await waitForTurnStart(sessionId, step.timeoutMs, step.minimumTurn)
return
}
case 'cancel': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: cancel before newSession')
if (step.waitForFile !== undefined) {
await waitForWorkspaceFile(cwd, step.waitForFile.path, step.waitForFile.timeoutMs)
}
await client.cancel({ sessionId })
return
}
case 'setMode': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: setMode before newSession')
await client.setSessionMode({ sessionId, modeId: step.modeId })
return
}
case 'setModeExpectError': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: setModeExpectError before newSession')
// The bridge rejects an unknown/uncomposed mode id with invalidParams;
// that rejection IS the expected wire behavior — swallow it so the run
// completes and the error frame is captured in the transcript.
await client.setSessionMode({ sessionId, modeId: step.modeId }).then(
() => { throw new Error('snapshot-harness: expected session/set_mode to be rejected but it succeeded') },
() => { /* expected: the bridge rejected the mode id */ },
)
return
}
case 'setConfigOption': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: setConfigOption before newSession')
await client.setSessionConfigOption({ sessionId, configId: step.configId, value: step.value })
return
}
case 'setConfigOptionExpectError': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: setConfigOptionExpectError before newSession')
// The bridge rejects an unknown id / out-of-vocabulary value; the SDK
// surfaces that as a rejected RPC — swallow it so the run completes and
// the error frame is captured in the transcript.
await client.setSessionConfigOption({ sessionId, configId: step.configId, value: step.value }).then(
() => { throw new Error('snapshot-harness: expected set_config_option to be rejected but it succeeded') },
() => { /* expected: the bridge rejected the id or value */ },
)
return
}
default:
throw new Error(`snapshot-harness: unknown input op ${JSON.stringify(step)}`)
}
}
/** Wait until persistence exposes an open turn for the selected session. */
async function waitForPersistedTurnStart(
root: string,
sessionId: string,
timeoutMs = DEFAULT_WAIT_TIMEOUT_MS,
minimumTurn?: number,
): Promise<void> {
const deadline = Date.now() + timeoutMs
while (true) {
const log = (await harvestSessionLogs(root)).find(candidate => candidate.id === sessionId)
const openTurn = log === undefined ? undefined : latestOpenTurn(log.content)
if (openTurn !== undefined && (minimumTurn === undefined || openTurn >= minimumTurn)) return
if (Date.now() >= deadline) {
const detail = minimumTurn === undefined ? 'turn/start' : `turn/start at or beyond turn ${minimumTurn}`
throw new Error(`snapshot-harness: session "${sessionId}" did not persist ${detail} within ${timeoutMs}ms`)
}
await delay(WAIT_POLL_INTERVAL_MS)
}
}
/**
* Wait until the raw JSONL backend exposes one complete closing turn boundary.
* The ACP cancel notification settles its prompt before the agent necessarily
@@ -557,45 +515,49 @@ function latestTurnIsClosed(content: string): boolean {
> complete.lastIndexOf('\n{"type":"turn/start",')
}
/** Return the latest open turn number, validating the persisted boundary record. */
function latestOpenTurn(content: string): number | undefined {
const complete = content.slice(0, content.lastIndexOf('\n') + 1)
const start = complete.lastIndexOf('\n{"type":"turn/start",')
if (start <= complete.lastIndexOf('\n{"type":"turn/end",')) return undefined
const end = complete.indexOf('\n', start + 1)
const record = JSON.parse(complete.slice(start + 1, end)) as { data?: { turn?: unknown } | null }
const turn = record.data?.turn
if (!Number.isSafeInteger(turn) || (turn as number) < 1) {
throw new Error('snapshot-harness: invalid persisted turn/start record')
}
return turn as number
}
/**
* Harvest EVERY persisted `.jsonl` session log under a sessions root, parse each
* header line, and return them ordered primary-first: the top-level session (no
* `parentSession`) leads, then each subagent child by ascending `createdAt`.
*
* Snapshot configs select the JSONL backend's raw mode, which lays sessions
* out as `<root>/<cwd-bucket>/<encoded-id>.jsonl` (one bucket per cwd). A
* parent and its same-cwd in-process child land in the SAME bucket, so
* collecting all files across all buckets catches both. Returns `[]` if no log
* was produced (a no-session scenario).
* out as `<root>/<project>/<session-id>/session.jsonl`. Recursive collection
* catches the primary and every child session. Returns `[]` if no log was
* produced (a no-session scenario).
*/
async function harvestSessionLogs(root: string): Promise<HarvestedLog[]> {
let cwdDirs: string[]
let files: string[]
try {
cwdDirs = await readdir(root)
files = await readdir(root, { recursive: true })
} catch {
return []
}
const logs: HarvestedLog[] = []
for (const dir of cwdDirs) {
const sub = join(root, dir)
let files: string[]
try {
files = await readdir(sub)
} catch {
continue
}
for (const f of files) {
if (!f.endsWith('.jsonl')) continue
const content = await readFile(join(sub, f), 'utf8')
const firstLine = content.split('\n').find(line => line.trim().length > 0) ?? '{}'
const header = JSON.parse(firstLine) as { id?: unknown; createdAt?: unknown; parentSession?: unknown }
logs.push({
id: typeof header.id === 'string' ? header.id : '',
createdAt: typeof header.createdAt === 'number' ? header.createdAt : 0,
...typeof header.parentSession === 'string' ? { parentSession: header.parentSession } : {},
content,
})
}
for (const file of files) {
if (basename(file) !== 'session.jsonl') continue
const content = await readFile(join(root, file), 'utf8')
const firstLine = content.split('\n').find(line => line.trim().length > 0) ?? '{}'
const header = JSON.parse(firstLine) as { id?: unknown; createdAt?: unknown; parentSession?: unknown }
logs.push({
id: typeof header.id === 'string' ? header.id : '',
createdAt: typeof header.createdAt === 'number' ? header.createdAt : 0,
...typeof header.parentSession === 'string' ? { parentSession: header.parentSession } : {},
content,
})
}
// Primary (no parentSession) first, then children by ascending createdAt. A
// scenario has exactly one top-level session. In the synchronous cut sibling

1
packages/support/acp-snapshot/src/index.ts
View File

@@ -18,7 +18,6 @@
export {
runScenario,
type ElicitationAnswer,
type HarvestedLog,
type InputScript,
type InputStep,

7
packages/support/acp-snapshot/src/launcher.ts
View File

@@ -15,8 +15,6 @@ import {
ndJsonStream,
type Agent as AcpAgent,
type Client,
type CreateElicitationRequest,
type CreateElicitationResponse,
type RequestPermissionRequest,
type RequestPermissionResponse,
type SessionNotification,
@@ -49,8 +47,6 @@ export interface AcpTestLaunchOptions {
env?: NodeJS.ProcessEnv
/** Permission handler; omitted requests fail closed as `cancelled`. */
requestPermission?: (params: RequestPermissionRequest) => Promise<RequestPermissionResponse>
/** Elicitation handler; omitted requests fail closed as `cancel`. */
createElicitation?: (params: CreateElicitationRequest) => Promise<CreateElicitationResponse>
}
/** A running ACP test process and its captured client-side surfaces. */
@@ -156,8 +152,6 @@ export function launchAcpTestAgent(options: AcpTestLaunchOptions): LaunchedAcpTe
}
const requestPermission = options.requestPermission
?? (() => Promise.resolve({ outcome: { outcome: 'cancelled' as const } }))
const createElicitation = options.createElicitation
?? (() => Promise.resolve({ action: 'cancel' as const }))
const makeClient = (_agent: AcpAgent): Client => ({
sessionUpdate(params: SessionNotification): Promise<void> {
return trackClientCallback(() => {
@@ -181,7 +175,6 @@ export function launchAcpTestAgent(options: AcpTestLaunchOptions): LaunchedAcpTe
})
},
requestPermission: params => trackClientCallback(() => requestPermission(params)),
unstable_createElicitation: params => trackClientCallback(() => createElicitation(params)),
})
const client = new ClientSideConnection(makeClient, stream)
// `exit` only reports the parent process's status. Descendants may retain

41
packages/support/acp-snapshot/src/normalize.ts
View File

@@ -10,12 +10,17 @@ const SESSION_ID = '{{sessionId}}'
const CWD = '{{cwd}}'
const SYSTEM = '{{system}}'
const TOOLS = '{{tools}}'
const UPDATED_AT = '{{updatedAt}}'
const EVENT_TIME = '{{eventTime}}'
const EVENT_OMITTED_BYTES = '{{eventOmittedBytes}}'
/** A cwd-rooted path after volatile cwd replacement, through its last separator-delimited segment. */
const CWD_ROOTED_PATH_RE = /\{\{cwd\}\}(?:[\\/][^\s<>"'`]+)+/g
const PATH_TAG_RE = /(<path>)([^<]*)(<\/path>)/g
const ADDITIONAL_INSTRUCTIONS_PATH_RE = /(Additional instructions from: )([^\r\n]+)/g
const EMBEDDED_EVENT_TIME_RE = /^( "time": )\d+(?=,\r?$)/gm
const EVENT_READ_OMITTED_BYTES_RE = /(\r?\n\r?\n\(Omitted )\d+( bytes\.)/g
const EVENT_READ_TARGET_REGION_RE
= /^Session [^\r\n]+ — [^\r\n]+\r?\nTarget event seq \d+:\r?\n```json\r?\n\{\r?\n[\s\S]*?(?=\r?\n```(?:\r?\n|$)|\r?\n\r?\n\(Omitted )/
/** A UUID v4 string, the shape `randomUUID()` produces for session ids. */
const UUID_RE = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi
@@ -62,6 +67,8 @@ export interface NormalizeContext {
sessionIds: string[]
/** The generated cwd the run used — replaced with `{{cwd}}`. */
cwd: string
/** Other filesystem spellings of the same cwd (for example Windows short and long paths). */
cwdAliases?: readonly string[]
}
/** How cwd-rooted path separators are represented after the cwd is tokenized. */
@@ -76,14 +83,18 @@ export interface NormalizeOptions {
/** Replace cwd, session ids, and any stray UUID with stable tokens in a string. */
function scrubString(value: string, ctx: NormalizeContext, cwdPathMode: CwdPathMode): string {
let out = value
// cwd first (longest, most specific), then explicit session ids, then any
// residual UUID (covers ids that appear in places we didn't enumerate).
// macOS symlinks /tmp → /private/tmp and /var → /private/var. The session
// header cwd may be recorded without the /private prefix while fs tools
// resolve symlinks, producing paths with the prefix inside tool results.
// Strip the prefix before the cwd replacement so both forms match.
out = out.split(`/private${ctx.cwd}`).join(ctx.cwd)
out = out.split(ctx.cwd).join(CWD)
// Filesystem APIs can report one directory with several spellings. Replace
// every known spelling longest-first so a shorter alias cannot corrupt a
// longer one before it is tokenized. macOS additionally symlinks
// /tmp → /private/tmp and /var → /private/var: the session header cwd may
// omit the /private prefix while fs tools resolve symlinks, so cover the
// prefixed form of every spelling too, then collapse a residual prefixed
// token.
const cwdSpellings = [...new Set([ctx.cwd, ...ctx.cwdAliases ?? []])]
.filter(spelling => spelling.length > 0)
.flatMap(spelling => [`/private${spelling}`, spelling])
.sort((left, right) => right.length - left.length)
for (const spelling of cwdSpellings) out = out.split(spelling).join(CWD)
out = out.split(`/private${CWD}`).join(CWD)
if (cwdPathMode === 'canonical') {
// Restrict separator conversion to paths rooted at the cwd token. A global
@@ -93,6 +104,16 @@ function scrubString(value: string, ctx: NormalizeContext, cwdPathMode: CwdPathM
}
out = out.replace(LOCAL_SPILL_PATH_RE, (_match, name: string) => `{{spillLocator:${name}}}`)
out = out.replace(SNAPSHOT_SPILL_PATH_RE, (_match, name: string) => `{{spillLocator:${name}}}`)
// Exact event-read results render the target as pretty JSON inside a
// distinctive envelope. Restrict time scrubbing to that fenced target so
// neighbor, model, bash, and unrelated tool text remains regression-visible.
if (EVENT_READ_TARGET_REGION_RE.test(out)) {
out = out.replace(
EVENT_READ_TARGET_REGION_RE,
target => target.replace(EMBEDDED_EVENT_TIME_RE, `$1${EVENT_TIME}`),
)
out = out.replace(EVENT_READ_OMITTED_BYTES_RE, `$1${EVENT_OMITTED_BYTES}$2`)
}
for (const id of ctx.sessionIds) out = out.split(id).join(SESSION_ID)
out = out.replace(UUID_RE, SESSION_ID)
return out
@@ -145,8 +166,6 @@ export function normalizeStdout(
if ('id' in frame && frame.id !== undefined && frame.id !== null) {
frame.id = stableId(frame.id)
}
const update = (frame.params as { update?: Record<string, unknown> } | undefined)?.update
if (update?.sessionUpdate === 'session_info_update') update.updatedAt = UPDATED_AT
return scrubValue(frame, ctx, cwdPathMode) as Record<string, unknown>
})
return frames.map(f => JSON.stringify(f)).join('\n') + '\n'

4
packages/support/acp-snapshot/src/suite.ts
View File

@@ -48,6 +48,8 @@ const PACKED_CHUNK_ROW_TYPES = new Set(['text-chunks', 'reasoning-chunks', 'tool
/** A snapshot scenario and how its fixtures are produced. */
export interface Scenario {
name: string
/** Deployment environment for this scenario's subprocess. */
env?: NodeJS.ProcessEnv
/** Whether the scenario drives at least one model turn (so a JSONL expected output applies). */
hasModelTurn: boolean
/**
@@ -613,6 +615,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
agent,
mode: childMode,
fixtureFile: join(dir, 'session.jsonl'),
...scenario.env !== undefined ? { env: scenario.env } : {},
...existsSync(overrideFile) ? { overrideFile } : {},
// In REPLAY, forward the recorded child fixtures so each subagent session
// replays from its own script. In RECORD they are harvested, not read.
@@ -638,6 +641,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
...result.sessionLogs.map(l => l.id),
],
cwd: result.cwd,
cwdAliases: result.cwdAliases,
}
// Record writes live model fixtures; keyless refresh writes every comparable replayed

145
packages/support/acp-snapshot/tests/fixtures/fake-acp-agent.ts vendored
View File

@@ -23,9 +23,9 @@ import { dirname, join } from 'node:path'
import { randomUUID } from 'node:crypto'
import { createInterface } from 'node:readline'
/** One scripted session log: a file path under the sessions root plus its JSONL lines. */
/** One scripted session log: a transcript path under the sessions root plus its JSONL lines. */
interface ScriptedLog {
/** Path relative to `$DSH_SNAPSHOT_SESSIONS_ROOT`, e.g. `bucket/a.jsonl` (an empty dir segment is invalid). */
/** Path relative to `$DSH_SNAPSHOT_SESSIONS_ROOT`, e.g. `project/session/session.jsonl`. */
file: string
/**
* The JSONL records. String templates `{{CWD}}` and `{{SID}}` are replaced
@@ -45,18 +45,10 @@ interface Behavior {
rejectExtraDirs?: boolean
/** How `session/prompt` settles: a clean response, a JSON-RPC error, or a hang until `session/cancel`. */
prompt?: 'respond' | 'error' | 'hang-until-cancel'
/** Emit a tool call instead of a message chunk before parking a cancellable prompt. */
cancelAtToolCall?: boolean
/** Emit the parked tool call's terminal update after answering cancellation. */
cancelToolCallUpdate?: boolean
/** Persist the scripted logs while handling cancellation, before stdin EOF. */
persistLogsOnCancel?: boolean
/** Before responding to a prompt, send a `session/request_permission` request and echo its outcome as a chunk. */
permissionProbe?: boolean
/** Before responding to a prompt, send an `elicitation/create` request and echo its response as a chunk. */
elicitationProbe?: boolean
/** How `session/set_mode` settles: an empty response (echoing the modeId as a chunk) or a JSON-RPC error. */
setMode?: 'respond' | 'error'
/** Echo the `DSH_SNAPSHOT_*` env the harness set as a chunk (spec-side env-plumbing assertions). */
echoEnv?: boolean
/** Echo the sorted cwd listing as a chunk (spec-side workspace-seeding assertions). */
@@ -69,17 +61,10 @@ interface Behavior {
logs?: ScriptedLog[]
/** Leave a stray FILE directly under the sessions root (harvest must skip it). */
strayRootFile?: boolean
/** Leave a stray non-`.jsonl` file inside a bucket (harvest must skip it). */
/** Leave a stray non-transcript file inside a project directory (harvest must skip it). */
strayBucketFile?: boolean
/** Delete the sessions root entirely (harvest must yield no logs). */
deleteSessionsRoot?: boolean
/**
* Vocabulary for `session/set_config_option`: allowed values per config id.
* A set naming an unknown id or an out-of-vocabulary value rejects (the
* real bridge's rule); a valid set answers with the complete refreshed
* option state, `currentValue` updated. Absent: every set rejects.
*/
configOptions?: Record<string, string[]>
}
const sessionsRoot = process.env.DSH_SNAPSHOT_SESSIONS_ROOT ?? ''
@@ -102,10 +87,10 @@ let sessionId = ''
let sessionCwd = ''
/** The parked prompt request id while `hang-until-cancel` waits for the cancel notification. */
let parkedPromptId: number | string | null = null
/** Resolvers for outbound probe responses (permission/elicitation), keyed by request id. */
/** The transient raw JSONL log that proves the parked turn started durably. */
let parkedTurnLog: string | undefined
/** Resolvers for outbound permission responses, keyed by request id. */
const pendingOutbound = new Map<number, (result: unknown) => void>()
/** Per-run `session/set_config_option` state: config id → current value (first vocabulary entry until set). */
const currentConfig: Record<string, string> = {}
function send(frame: Record<string, unknown>): void {
process.stdout.write(`${JSON.stringify({ jsonrpc: '2.0', ...frame })}\n`)
@@ -138,39 +123,34 @@ function instantiate(value: unknown): unknown {
return value
}
/** Persist an open turn so cancellation tests wait on agent state, not presentation output. */
function persistParkedTurnStart(): void {
parkedTurnLog = join(sessionsRoot, 'ready', sessionId, 'session.jsonl')
mkdirSync(dirname(parkedTurnLog), { recursive: true })
writeFileSync(parkedTurnLog, [
JSON.stringify({ type: 'session', version: 0, id: sessionId, createdAt: 1, cwd: sessionCwd, delegationDepth: 0 }),
JSON.stringify({ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } }),
'',
].join('\n'))
}
/** Remove the transient open-turn log before publishing any scripted final logs. */
function clearParkedTurnStart(): void {
if (parkedTurnLog === undefined) return
rmSync(parkedTurnLog, { force: true })
parkedTurnLog = undefined
}
async function handlePrompt(id: number | string): Promise<void> {
if ((behavior.prompt ?? 'respond') === 'hang-until-cancel') {
// A thought chunk BEFORE any message chunk: a promptAndCancel waiter
// watches for agent_message_chunk, so this exercises its non-matching
// update path while the waiter is armed.
send({
method: 'session/update',
params: { sessionId, update: { sessionUpdate: 'agent_thought_chunk', content: { type: 'text', text: 'mulling' } } },
})
}
if (behavior.cancelAtToolCall === true) {
send({
method: 'session/update',
params: {
sessionId,
update: {
sessionUpdate: 'tool_call',
toolCallId: 'call_fake_1',
title: 'fake tool',
kind: 'execute',
status: 'in_progress',
},
},
})
} else {
chunk('thinking about it')
}
chunk('thinking about it')
if (behavior.echoEnv === true) {
chunk(`env:${JSON.stringify({
mode: process.env.DSH_SNAPSHOT,
override: process.env.DSH_SNAPSHOT_OVERRIDE ?? null,
childFiles: process.env.DSH_SNAPSHOT_CHILD_FILES ?? null,
spillRoot: process.env.DSH_SNAPSHOT_SPILL_ROOT ?? null,
// Scenario-supplied deployment env (the `Scenario.env` layering seam).
permissionMode: process.env.DSH_PERMISSION_MODE ?? null,
})}`)
}
if (behavior.echoWorkspace === true) {
@@ -185,7 +165,7 @@ async function handlePrompt(id: number | string): Promise<void> {
method: 'session/request_permission',
params: {
sessionId,
toolCall: { toolCallId: 'call_fake_1', title: 'fake tool', kind: 'execute', status: 'pending' },
toolCall: { toolCallId: 'call_fake_1' },
options: [
{ optionId: 'opt-allow', name: 'Allow once', kind: 'allow_once' },
{ optionId: 'opt-reject', name: 'Reject once', kind: 'reject_once' },
@@ -195,23 +175,6 @@ async function handlePrompt(id: number | string): Promise<void> {
})
chunk(`permission:${JSON.stringify((result as { outcome?: unknown } | undefined)?.outcome ?? null)}`)
}
if (behavior.elicitationProbe === true) {
const requestId = nextOutboundId++
const result = await new Promise<unknown>((resolve) => {
pendingOutbound.set(requestId, resolve)
send({
id: requestId,
method: 'elicitation/create',
params: {
sessionId,
mode: 'form',
message: 'Approve this plan and leave plan mode?',
requestedSchema: { type: 'object', title: 'Plan review', properties: { choice: { type: 'string' }, custom: { type: 'string' } }, required: [] },
},
})
})
chunk(`elicitation:${JSON.stringify(result ?? null)}`)
}
switch (behavior.prompt ?? 'respond') {
case 'respond':
respond(id, { stopReason: 'end_turn' })
@@ -220,6 +183,7 @@ async function handlePrompt(id: number | string): Promise<void> {
respondError(id, 'model exploded')
return
case 'hang-until-cancel':
persistParkedTurnStart()
parkedPromptId = id
return
}
@@ -254,59 +218,13 @@ function handleFrame(frame: Record<string, unknown>): void {
case 'session/prompt':
void handlePrompt(id as number | string)
return
case 'session/set_mode':
if ((behavior.setMode ?? 'respond') === 'error') {
respondError(id as number | string, 'unknown mode')
return
}
chunk(`setMode:${String(params.modeId)}`)
respond(id as number | string, {})
return
case 'session/set_config_option': {
const vocabulary = behavior.configOptions
const configId = params.configId as string
const value = params.value as string
const values = vocabulary?.[configId]
if (values === undefined) {
respondError(id as number | string, `unknown config option ${configId}`)
return
}
if (!values.includes(value)) {
respondError(id as number | string, `unknown ${configId} value ${value}`)
return
}
currentConfig[configId] = value
// The real bridge's contract: every set answers with the COMPLETE
// refreshed option state, not just the changed entry.
respond(id as number | string, {
configOptions: Object.entries(vocabulary as Record<string, string[]>).map(([cid, vs]) => ({
id: cid,
type: 'select',
currentValue: currentConfig[cid] ?? vs[0],
options: vs.map(v => ({ value: v, name: v })),
})),
})
return
}
case 'session/cancel':
if (parkedPromptId !== null) {
const parked = parkedPromptId
parkedPromptId = null
respond(parked, { stopReason: 'cancelled' })
if (behavior.cancelToolCallUpdate === true) {
send({
method: 'session/update',
params: {
sessionId,
update: {
sessionUpdate: 'tool_call_update',
toolCallId: 'call_fake_1',
status: 'failed',
},
},
})
}
clearParkedTurnStart()
if (behavior.persistLogsOnCancel === true) writeLogs()
respond(parked, { stopReason: 'cancelled' })
}
return
default:
@@ -325,6 +243,7 @@ function writeLogs(): void {
}
function flushLogsAndExit(): void {
clearParkedTurnStart()
writeLogs()
if (behavior.strayRootFile === true) writeFileSync(join(sessionsRoot, 'stray.txt'), 'not a bucket\n')
if (behavior.strayBucketFile === true) {

4
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-child/behavior.json vendored
View File

@@ -1,11 +1,11 @@
{
"prompt": "respond",
"logs": [
{ "file": "b/parent.jsonl", "lines": [
{ "file": "b/parent/session.jsonl", "lines": [
{ "type": "session", "id": "{{SID}}", "createdAt": 700, "cwd": "{{CWD}}", "delegationDepth": 0 },
{ "type": "request/header", "seq": 0, "time": 3, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
]},
{ "file": "b/child.jsonl", "lines": [
{ "file": "b/child/session.jsonl", "lines": [
{ "type": "session", "id": "abababab-cdcd-4efe-8ada-badabadabada", "createdAt": 800, "cwd": "{{CWD}}", "parentSession": "{{SID}}", "delegationDepth": 1 },
{ "type": "request/header", "seq": 0, "time": 2, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
]}

2
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/behavior.json vendored
View File

@@ -1,7 +1,7 @@
{
"prompt": "respond",
"logs": [{
"file": "b/main.jsonl",
"file": "b/main/session.jsonl",
"lines": [
{ "type": "session", "id": "{{SID}}", "createdAt": 600, "cwd": "{{CWD}}", "delegationDepth": 0 },
{ "type": "request/header", "seq": 0, "time": 4, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }

2
packages/support/acp-snapshot/tests/fixtures/suite/authored-error/behavior.json vendored
View File

@@ -1,7 +1,7 @@
{
"prompt": "error",
"logs": [{
"file": "b/main.jsonl",
"file": "b/main/session.jsonl",
"lines": [
{ "type": "session", "id": "{{SID}}", "createdAt": 500, "cwd": "{{CWD}}", "delegationDepth": 0 },
{ "type": "turn/end", "seq": 1, "time": 9, "data": { "error": "model exploded" } }

2
packages/support/acp-snapshot/tests/fixtures/suite/blocked-log/behavior.json vendored
View File

@@ -1,7 +1,7 @@
{
"prompt": "error",
"logs": [{
"file": "b/main.jsonl",
"file": "b/main/session.jsonl",
"lines": [
{ "type": "session", "id": "{{SID}}", "createdAt": 400, "cwd": "{{CWD}}", "delegationDepth": 0 },
{ "type": "hook/result", "seq": 1, "time": 8, "data": { "decision": "block", "durationMs": 37 } }

2
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/behavior.json vendored
View File

@@ -1,7 +1,7 @@
{
"prompt": "respond",
"logs": [{
"file": "b/main.jsonl",
"file": "b/main/session.jsonl",
"lines": [
{ "type": "session", "id": "{{SID}}", "createdAt": 100, "cwd": "{{CWD}}", "delegationDepth": 0 },
{ "type": "request/header", "seq": 0, "time": 100, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } },

4
packages/support/acp-snapshot/tests/fixtures/suite/plain-turn/behavior.json vendored
View File

@@ -2,12 +2,12 @@
"prompt": "respond",
"echoWorkspace": true,
"logs": [
{ "file": "b/parent.jsonl", "lines": [
{ "file": "b/parent/session.jsonl", "lines": [
{ "type": "session", "id": "{{SID}}", "createdAt": 200, "cwd": "{{CWD}}", "delegationDepth": 0 },
{ "type": "request/header", "seq": 0, "time": 5, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } },
{ "type": "assistant/chunk", "seq": 1, "time": 5, "data": { "turn": 1, "step": 1, "chunk": { "type": "text-delta", "index": 0, "text": "hi" } } }
]},
{ "file": "b/child.jsonl", "lines": [
{ "file": "b/child/session.jsonl", "lines": [
{ "type": "session", "id": "eeeeeeee-1111-4222-8333-444444444444", "createdAt": 300, "cwd": "{{CWD}}", "parentSession": "{{SID}}", "delegationDepth": 1 },
{ "type": "request/header", "seq": 0, "time": 6, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
]}

280
packages/support/acp-snapshot/tests/harness.spec.ts
View File

@@ -95,8 +95,8 @@ describe('runScenario', () => {
expect(clientClosed).toBe(true)
})
it('centralizes ACP boot, captures, updates, fail-closed interactions, and shutdown', { timeout: 20_000 }, async () => {
const { dir, fixtureFile } = await scenario({ permissionProbe: true, elicitationProbe: true, echoEnv: true, stderrNote: 'launcher stderr' })
it('centralizes ACP boot, captures, updates, fail-closed permissions, and shutdown', { timeout: 20_000 }, async () => {
const { dir, fixtureFile } = await scenario({ permissionProbe: true, echoEnv: true, stderrNote: 'launcher stderr' })
const sessionsRoot = await mkdtemp(join(tmpdir(), 'acp-launcher-sessions-'))
tempDirs.push(sessionsRoot)
const launched = launchAcpTestAgent({
@@ -112,6 +112,8 @@ describe('runScenario', () => {
await launched.client.initialize({ protocolVersion: PROTOCOL_VERSION, clientCapabilities: {} })
const { sessionId } = await launched.client.newSession({ cwd: dir, mcpServers: [] })
const nextChunk = launched.waitForUpdate(update => update.sessionUpdate === 'agent_message_chunk')
const laterChunk = launched.waitForUpdate(update => update.sessionUpdate === 'agent_message_chunk'
&& update.content.type === 'text' && update.content.text === 'never this one')
const predicateFailure = new Error('predicate failed')
const failedPredicate = launched.waitForUpdate(() => { throw predicateFailure })
.catch((error: unknown): unknown => error)
@@ -120,8 +122,8 @@ describe('runScenario', () => {
expect((await nextChunk).sessionUpdate).toBe('agent_message_chunk')
expect(launched.updates.some(update => update.sessionUpdate === 'agent_message_chunk')).toBe(true)
expect(launched.rawStdout()).toContain('permission:{\\"outcome\\":\\"cancelled\\"}')
expect(launched.rawStdout()).toContain('elicitation:{\\"action\\":\\"cancel\\"}')
expect(launched.stderr()).toContain('launcher stderr')
void laterChunk.catch(() => undefined)
const unmatched = expect(launched.waitForUpdate(() => false)).rejects.toThrow(/update stream closed/)
await launched.close()
await unmatched
@@ -374,11 +376,11 @@ describe('runScenario', () => {
}
})
it('drives a full turn: initialize (terminal caps), session, prompt, permission stub, harvest', { timeout: 20_000 }, async () => {
it('drives a full turn: initialize, session, prompt, permission stub, harvest', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({
permissionProbe: true,
logs: [{
file: 'bucket/main.jsonl',
file: 'project/main/session.jsonl',
lines: [
{ type: 'session', id: '{{SID}}', createdAt: 42, cwd: '{{CWD}}' },
{ type: 'turn/start', seq: 1, time: 9, data: { turn: 1 } },
@@ -386,7 +388,7 @@ describe('runScenario', () => {
}],
})
const result = await runScenario(
{ steps: [{ op: 'initialize', terminalOutput: true }, { op: 'newSession' }, { op: 'prompt', text: 'go' }] },
{ steps: [{ op: 'initialize' }, { op: 'newSession' }, { op: 'prompt', text: 'go' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.sessionId).toBeDefined()
@@ -482,7 +484,7 @@ describe('runScenario', () => {
expect(child.startsWith(`..${sep}`)).toBe(false)
})
it('promptAndCancel waits for the streamed chunk, cancels, and settles the prompt', { timeout: 20_000 }, async () => {
it('promptAndCancel waits for the durable turn start, cancels, and settles the prompt', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ prompt: 'hang-until-cancel' })
const result = await runScenario(
{ steps: [...boot, { op: 'promptAndCancel', text: 'hang' }] },
@@ -539,34 +541,12 @@ describe('runScenario', () => {
expect(result.rawStdout).toContain('thinking about it')
})
it('promptAndCancel can bracket cancellation with tool-call updates', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({
prompt: 'hang-until-cancel',
cancelAtToolCall: true,
cancelToolCallUpdate: true,
})
const result = await runScenario(
{
steps: [...boot, {
op: 'promptAndCancel',
text: 'hang',
afterUpdate: 'tool_call',
waitForToolCallUpdate: 'call_fake_1',
}],
},
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.rawStdout).toContain('"sessionUpdate":"tool_call"')
expect(result.rawStdout.indexOf('"sessionUpdate":"tool_call"')).toBeLessThan(result.rawStdout.indexOf('cancelled'))
expect(result.rawStdout.indexOf('cancelled')).toBeLessThan(result.rawStdout.indexOf('"sessionUpdate":"tool_call_update"'))
})
it('waitForTurnEnd holds cancellation open through the persisted closing boundary', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({
prompt: 'hang-until-cancel',
persistLogsOnCancel: true,
logs: [{
file: 'bucket/session.jsonl',
file: 'project/main/session.jsonl',
lines: [
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
{ type: 'turn/end', seq: 1, time: 2, data: { turn: 1, reason: { kind: 'aborted' } } },
@@ -580,6 +560,108 @@ describe('runScenario', () => {
expect(result.sessionLogs[0]?.content).toContain('"type":"turn/end"')
})
it('waitForTurnStart can require a later durable turn before continuing', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({
prompt: 'hang-until-cancel',
persistLogsOnCancel: true,
logs: [{
file: 'project/main/session.jsonl',
lines: [
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 3 } },
],
}],
})
const result = await runScenario(
{
steps: [
...boot,
{ op: 'promptAndCancel', text: 'hang' },
{ op: 'waitForTurnStart', minimumTurn: 3 },
],
},
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.sessionLogs[0]?.content).toContain('"turn":3')
})
it('waitForTurnStart rejects missing, earlier, and malformed durable turns', { timeout: 20_000 }, async () => {
const missing = await scenario({})
await expect(runScenario(
{ steps: [...boot, { op: 'waitForTurnStart', timeoutMs: 20 }] },
{ agent: AGENT, mode: 'replay', fixtureFile: missing.fixtureFile },
)).rejects.toThrow(/did not persist turn\/start within 20ms/)
const earlier = await scenario({
prompt: 'hang-until-cancel',
persistLogsOnCancel: true,
logs: [{
file: 'project/main/session.jsonl',
lines: [
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } },
],
}],
})
await expect(runScenario(
{
steps: [
...boot,
{ op: 'promptAndCancel', text: 'hang' },
{ op: 'waitForTurnStart', minimumTurn: 3, timeoutMs: 20 },
],
},
{ agent: AGENT, mode: 'replay', fixtureFile: earlier.fixtureFile },
)).rejects.toThrow(/turn\/start at or beyond turn 3 within 20ms/)
const closed = await scenario({
prompt: 'hang-until-cancel',
persistLogsOnCancel: true,
logs: [{
file: 'project/main/session.jsonl',
lines: [
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } },
{ type: 'turn/end', seq: 1, time: 2, data: { turn: 1, reason: { kind: 'stop' } } },
],
}],
})
await expect(runScenario(
{
steps: [
...boot,
{ op: 'promptAndCancel', text: 'hang' },
{ op: 'waitForTurnStart', timeoutMs: 20 },
],
},
{ agent: AGENT, mode: 'replay', fixtureFile: closed.fixtureFile },
)).rejects.toThrow(/did not persist turn\/start within 20ms/)
for (const turn of [undefined, 0]) {
const malformed = await scenario({
prompt: 'hang-until-cancel',
persistLogsOnCancel: true,
logs: [{
file: 'project/main/session.jsonl',
lines: [
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
{ type: 'turn/start', seq: 0, time: 1, data: turn === undefined ? {} : { turn } },
],
}],
})
await expect(runScenario(
{
steps: [
...boot,
{ op: 'promptAndCancel', text: 'hang' },
{ op: 'waitForTurnStart' },
],
},
{ agent: AGENT, mode: 'replay', fixtureFile: malformed.fixtureFile },
)).rejects.toThrow('invalid persisted turn/start record')
}
})
it('waitForTurnEnd times out for a missing log and an open logged turn', { timeout: 20_000 }, async () => {
const missing = await scenario({})
await expect(runScenario(
@@ -591,7 +673,7 @@ describe('runScenario', () => {
prompt: 'hang-until-cancel',
persistLogsOnCancel: true,
logs: [{
file: 'bucket/session.jsonl',
file: 'project/main/session.jsonl',
lines: [
{ type: 'session', version: 0, id: '{{SID}}', createdAt: 1, delegationDepth: 0 },
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } },
@@ -695,15 +777,27 @@ describe('runScenario', () => {
expect(result.sessionId).toBeDefined()
})
it('a standalone cancel can wait for cwd-relative readiness', { timeout: 20_000 }, async () => {
const { dir, fixtureFile } = await scenario({})
const workspaceDir = join(dir, 'workspace')
const { mkdir } = await import('node:fs/promises')
await mkdir(workspaceDir, { recursive: true })
await writeFile(join(workspaceDir, 'ready'), '')
const result = await runScenario(
{ steps: [...boot, { op: 'cancel', waitForFile: { path: 'ready' } }] },
{ agent: AGENT, mode: 'replay', fixtureFile, workspaceDir },
)
expect(result.sessionId).toBeDefined()
})
it.each([
[{ op: 'prompt', text: 'x' }, /prompt before newSession/],
[{ op: 'promptAndWaitForAgentMessage', text: 'x', waitForText: 'later' }, /promptAndWaitForAgentMessage before newSession/],
[{ op: 'promptExpectError', text: 'x' }, /promptExpectError before newSession/],
[{ op: 'promptAndCancel', text: 'x' }, /promptAndCancel before newSession/],
[{ op: 'waitForTurnStart' }, /waitForTurnStart before newSession/],
[{ op: 'waitForTurnEnd' }, /waitForTurnEnd before newSession/],
[{ op: 'cancel' }, /cancel before newSession/],
[{ op: 'setConfigOption', configId: 'sandbox-mode', value: 'read-only' }, /setConfigOption before newSession/],
[{ op: 'setConfigOptionExpectError', configId: 'sandbox-mode', value: 'yolo' }, /setConfigOptionExpectError before newSession/],
] as [InputStep, RegExp][])('rejects %j before newSession', { timeout: 20_000 }, async (step, message) => {
const { fixtureFile } = await scenario({})
await expect(runScenario(
@@ -712,53 +806,6 @@ describe('runScenario', () => {
)).rejects.toThrow(message)
})
it('setConfigOption switches a value and receives the complete refreshed option state', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({
configOptions: { 'sandbox-mode': ['read-only', 'workspace-write'], 'approval-policy': ['ask', 'never'] },
})
const result = await runScenario(
{
steps: [...boot,
{ op: 'setConfigOption', configId: 'sandbox-mode', value: 'workspace-write' },
{ op: 'setConfigOption', configId: 'approval-policy', value: 'never' }],
},
{ agent: AGENT, mode: 'replay', fixtureFile },
)
// Every set answers with the FULL state: the second response carries the
// first switch's value too — the complete-refreshed-state contract.
const frames = result.rawStdout.trim().split('\n').map(line => JSON.parse(line) as { result?: { configOptions?: { id: string; currentValue: string }[] } })
const states = frames
.map(f => f.result?.configOptions)
.filter(options => options !== undefined)
.map(options => Object.fromEntries((options as { id: string; currentValue: string }[]).map(o => [o.id, o.currentValue])))
expect(states).toEqual([
{ 'sandbox-mode': 'workspace-write', 'approval-policy': 'ask' },
{ 'sandbox-mode': 'workspace-write', 'approval-policy': 'never' },
])
})
it('setConfigOptionExpectError swallows the rejection for unknown ids and out-of-vocabulary values', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ configOptions: { 'sandbox-mode': ['read-only'] } })
const result = await runScenario(
{
steps: [...boot,
{ op: 'setConfigOptionExpectError', configId: 'sandbox-mode', value: 'yolo' },
{ op: 'setConfigOptionExpectError', configId: 'reasoning-effort', value: 'max' }],
},
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.rawStdout).toContain('unknown sandbox-mode value yolo')
expect(result.rawStdout).toContain('unknown config option reasoning-effort')
})
it('setConfigOptionExpectError throws when the set unexpectedly succeeds', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ configOptions: { 'sandbox-mode': ['read-only'] } })
await expect(runScenario(
{ steps: [...boot, { op: 'setConfigOptionExpectError', configId: 'sandbox-mode', value: 'read-only' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/expected set_config_option to be rejected/)
})
it('rejects an unknown input op', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({})
const bogus = { op: 'reticulate' } as unknown as InputStep
@@ -776,11 +823,11 @@ describe('runScenario', () => {
// File names chosen so readdir feeds the sort children-first AND
// parent-in-the-middle: the comparator then sees a parent on both
// sides of a pair, plus the same-createdAt (localeCompare) tiebreak.
{ file: 'b1/aa-child-c.jsonl', lines: [{ type: 'session', id: 'cccccccc-0000-4000-8000-000000000000', createdAt: 500, parentSession: '{{SID}}' }] },
{ file: 'b1/bb-parent.jsonl', lines: [{ type: 'session', id: '{{SID}}', createdAt: 900 }] },
{ file: 'b1/cc-child-a.jsonl', lines: [{ type: 'session', id: 'aaaaaaaa-0000-4000-8000-000000000000', createdAt: 500, parentSession: '{{SID}}' }] },
{ file: 'b1/aa-child-c/session.jsonl', lines: [{ type: 'session', id: 'cccccccc-0000-4000-8000-000000000000', createdAt: 500, parentSession: '{{SID}}' }] },
{ file: 'b1/bb-parent/session.jsonl', lines: [{ type: 'session', id: '{{SID}}', createdAt: 900 }] },
{ file: 'b1/cc-child-a/session.jsonl', lines: [{ type: 'session', id: 'aaaaaaaa-0000-4000-8000-000000000000', createdAt: 500, parentSession: '{{SID}}' }] },
// Missing id/createdAt fall back to ''/0; earliest child by createdAt.
{ file: 'b2/orphan-fields.jsonl', lines: [{ type: 'session', parentSession: '{{SID}}' }] },
{ file: 'b2/orphan/session.jsonl', lines: [{ type: 'session', parentSession: '{{SID}}' }] },
],
})
const result = await runScenario(
@@ -797,7 +844,7 @@ describe('runScenario', () => {
})
it('treats an empty log file as a header-less primary with default fields', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ logs: [{ file: 'b/empty.jsonl', lines: [] }] })
const { fixtureFile } = await scenario({ logs: [{ file: 'b/empty/session.jsonl', lines: [] }] })
const result = await runScenario(
{ steps: boot },
{ agent: AGENT, mode: 'replay', fixtureFile },
@@ -814,69 +861,6 @@ describe('runScenario', () => {
expect(result.sessionLogs).toHaveLength(0)
})
it('drives session/set_mode and swallows the expected rejection of setModeExpectError', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({})
const result = await runScenario(
{ steps: [...boot, { op: 'setMode', modeId: 'plan' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.rawStdout).toContain('setMode:plan')
const rejecting = await scenario({ setMode: 'error' })
const rejected = await runScenario(
{ steps: [...boot, { op: 'setModeExpectError', modeId: 'yolo' }] },
{ agent: AGENT, mode: 'replay', fixtureFile: rejecting.fixtureFile },
)
expect(rejected.rawStdout).toContain('unknown mode')
})
it('fails the run when setModeExpectError unexpectedly succeeds, and both mode ops require a session', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({})
await expect(runScenario(
{ steps: [...boot, { op: 'setModeExpectError', modeId: 'plan' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/expected session\/set_mode to be rejected/)
await expect(runScenario(
{ steps: [{ op: 'initialize' }, { op: 'setMode', modeId: 'plan' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/setMode before newSession/)
await expect(runScenario(
{ steps: [{ op: 'initialize' }, { op: 'setModeExpectError', modeId: 'plan' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/setModeExpectError before newSession/)
})
it('answers elicitations from the scripted queue, falling back to cancel on exhaustion', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ elicitationProbe: true })
// Three prompts → three elicitations: an accept-with-choice, an
// accept-with-custom (feedback), then the exhausted-queue cancel.
const result = await runScenario(
{
steps: [...boot, { op: 'prompt', text: 'one' }, { op: 'prompt', text: 'two' }, { op: 'prompt', text: 'three' }],
elicitationAnswers: [
{ action: 'accept', choice: 'Approve' },
{ action: 'accept', custom: 'add tests first' },
],
},
{ agent: AGENT, mode: 'replay', fixtureFile },
)
const first = result.rawStdout.indexOf('elicitation:{\\"action\\":\\"accept\\",\\"content\\":{\\"choice\\":\\"Approve\\"}}')
const second = result.rawStdout.indexOf('elicitation:{\\"action\\":\\"accept\\",\\"content\\":{\\"custom\\":\\"add tests first\\"}}')
const third = result.rawStdout.indexOf('elicitation:{\\"action\\":\\"cancel\\"}')
expect(first).toBeGreaterThanOrEqual(0)
expect(second).toBeGreaterThan(first)
expect(third).toBeGreaterThan(second)
})
it('a scripted elicitation cancel answers cancel', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ elicitationProbe: true })
const result = await runScenario(
{ steps: [...boot, { op: 'prompt', text: 'one' }], elicitationAnswers: [{ action: 'cancel' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)
expect(result.rawStdout).toContain('elicitation:{\\"action\\":\\"cancel\\"}')
})
it('answers permission requests from the scripted queue by option kind, falling back to cancelled', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ permissionProbe: true })
// Two prompts → two permission round-trips; one scripted answer, so the

64
packages/support/acp-snapshot/tests/normalize.spec.ts
View File

@@ -45,6 +45,24 @@ describe('normalizeStdout', () => {
expect(out).not.toContain(ctx.sessionIds[0] as string)
})
it('scrubs every filesystem spelling of the cwd longest-first', () => {
const longCwd = String.raw`C:\Users\runneradmin\AppData\Local\Temp\acp-snapshot`
const aliasedCtx: NormalizeContext = {
sessionIds: [],
cwd: String.raw`C:\Users\RUNNER~1\AppData\Local\Temp\acp-snapshot`,
cwdAliases: [
longCwd,
String.raw`C:\Users\runneradmin\AppData\Local\Temp\acp`,
],
}
const raw = JSON.stringify({
cwd: longCwd,
path: `${longCwd}\\nested\\proof.txt`,
})
const frame = JSON.parse(normalizeStdout(raw, aliasedCtx)) as { cwd: string; path: string }
expect(frame).toEqual({ cwd: '{{cwd}}', path: '{{cwd}}/nested/proof.txt' })
})
it('canonicalizes only cwd-rooted path separators', () => {
const windowsCtx: NormalizeContext = {
sessionIds: [],
@@ -106,22 +124,54 @@ Additional instructions from: nested\AGENTS.md`,
expect(out).not.toContain('"id"')
})
it('stabilizes the timestamp carried by session title updates', () => {
it('stabilizes only the top-level event timestamp and spill byte count in event-read text', () => {
const raw = JSON.stringify({
jsonrpc: '2.0',
method: 'session/update',
params: {
sessionId: ctx.sessionIds[0],
update: {
sessionUpdate: 'session_info_update',
title: 'Stable title',
updatedAt: '2026-07-20T17:03:13.689Z',
sessionUpdate: 'tool_call_update',
content: [{
type: 'content',
content: {
type: 'text',
text: 'Session prior — title\nTarget event seq 4:\n```json\n{\n "seq": 4,\n "time": 1784876275593,\n "data": {\n "time": 31337,\n "note": "model-visible"\n }\n}\n```\n\nAfter:\n "time": 424242,\n neighbor semantic text\n\n(Omitted 39387 bytes. Full formatted result stored at: /tmp/result.txt.)',
},
}],
},
},
})
const out = normalizeStdout(raw, ctx)
expect(out).toContain('"updatedAt":"{{updatedAt}}"')
expect(out).not.toContain('2026-07-20T17:03:13.689Z')
expect(out).toContain('\\"time\\": {{eventTime}}')
expect(out).toContain('\\"time\\": 31337')
expect(out).toContain('\\"time\\": 424242')
expect(out).toContain('Omitted {{eventOmittedBytes}} bytes')
expect(out).not.toContain('1784876275593')
expect(out).not.toContain('39387')
})
it('preserves event-like timestamps in unrelated output text', () => {
const raw = JSON.stringify({
jsonrpc: '2.0',
method: 'session/update',
params: {
update: {
sessionUpdate: 'tool_call_update',
content: [{
type: 'content',
content: {
type: 'text',
text: 'bash output:\n```json\n{\n "time": 1784876275593,\n "data": {}\n}\n```\n\n(Omitted 39387 bytes. Full formatted result stored at: /tmp/result.txt.)',
},
}],
},
},
})
const out = normalizeStdout(raw, ctx)
expect(out).toContain('1784876275593')
expect(out).toContain('39387')
expect(out).not.toContain('{{eventTime}}')
expect(out).not.toContain('{{eventOmittedBytes}}')
})
it('throws on a non-JSON stdout line (the purity check)', () => {

3
packages/support/acp-snapshot/tests/suite.spec.ts
View File

@@ -53,6 +53,7 @@ const REPLAY_SCENARIOS: Scenario[] = [
hasModelTurn: true,
recorded: true,
headerClass: 'main',
env: { DSH_PERMISSION_MODE: 'never' },
configPath: AGENT.configPath,
workspaceParent: tmpdir(),
},
@@ -130,6 +131,8 @@ describe('defineAcpSnapshotSuite: refresh write-back', () => {
expect(stdout).not.toContain('stale stdout')
expect(stdout).toContain('env:{\\"mode\\":\\"replay\\"')
expect(stdout).not.toContain('\\"mode\\":\\"refresh\\"')
// The scenario's own env layer reached the subprocess.
expect(stdout).toContain('\\"permissionMode\\":\\"never\\"')
const blocked = readFileSync(join(refreshDir, 'blocked-log', 'session.jsonl'), 'utf8')
expect(blocked).toContain('"decision":"block"')

84
packages/support/llm-mock-server/README.md Normal file
View File

@@ -0,0 +1,84 @@
# `@deepseek-ai/dsh-llm-mock-server`
A scriptable OpenAI-compatible HTTP/SSE server for exercising real LLM adapters, the agent loop, and recovery policy without a provider key. It accepts `POST /chat/completions` and `POST /v1/chat/completions`; each accepted request consumes one configured behavior in arrival order. Invalid methods, paths, bearer tokens, and JSON do not consume the script.
The library entry exports `startMockLlmServer(options)`, behavior and telemetry types, the default random stress weights, the accepted Node timer bound, and a running handle with the bound `baseURL`, generated or configured `randomSeed`, captured requests, and idempotent `close()`. Closing force-terminates stalled connections.
## Standalone use
Run the source entry from this repository:
```sh
pnpm run mock:llm -- \
--port 8000 \
--api-key mock-key \
--sequence partial_disconnect,success \
--partial-text "discard this half"
```
Point the shipping DeepSeek adapter at the server; it appends `/chat/completions` to the configured base:
```sh
DEEPSEEK_BASE_URL=http://127.0.0.1:8000/v1 \
DEEPSEEK_API_KEY=mock-key \
pnpm run demo:headless "test provider recovery"
```
The built package also exposes `dsh-llm-mock-server`. Stdout is JSONL: a `ready` record carries the `/v1` base URL and random seed, followed by request/result records that name both the scripted behavior and the concrete selected behavior.
## Behavior script
`--sequence` is a comma-separated FIFO. Exhaustion returns a structured HTTP 500; `--repeat-last` explicitly reuses the last entry.
| Behavior | Wire result |
|---|---|
| `connection_reset` | Destroy the socket before HTTP headers |
| `stream_disconnect` | Send SSE headers, then reset before the first event |
| `partial_disconnect` | Send text deltas, then reset the socket |
| `stall` | Send SSE headers and remain idle until client/server cancellation |
| `empty` | Send a valid content-less stop and `[DONE]` |
| `empty_body` / `stream_eof` / `partial_eof` | End cleanly without the required `[DONE]` boundary |
| `malformed_json` / `malformed_event` | Send invalid SSE JSON or an invalid provider chunk shape |
| `rate_limit` / `server_error` / `service_unavailable` | Return retry-oriented 429/500/503 JSON errors |
| `auth_error` / `invalid_request` / `context_overflow` / `quota_exceeded` | Return terminal or separately recovered provider errors |
| `success` / `slow_success` / `reasoning_success` | Stream a complete text response, optionally delayed or preceded by reasoning |
| `tool_call_success` / `max_tokens` | Complete with a tool call or `length` finish |
| `wrong_content_type` | Send a valid SSE body under `application/json` |
| `random` | Select a concrete request behavior from weighted seeded randomness |
`connection_refused` is CLI-only and must be the first entry. It delays binding a caller-specified nonzero port, so requests during `--listen-delay-ms` receive a real TCP refusal; the remaining entries begin after the listener starts.
## Random mode
Use a repeating `random` entry for an open-ended mixed run:
```sh
pnpm run mock:llm -- \
--port 8000 \
--sequence random \
--repeat-last \
--seed 42 \
--random-weights 'success=60,slow_success=10,connection_reset=5,stream_disconnect=5,partial_disconnect=10,empty=5,server_error=5'
```
Omitting `--seed` generates one and prints it in the `ready` record. `--random-weights` accepts non-negative relative `behavior=weight` entries and requires at least one positive concrete behavior. The exported default is a success-heavy stress profile containing reset, disconnect, partial output, empty completion, stall, 429/5xx, clean truncation, and malformed JSON; it is test pressure, not an estimate of production incident frequency. `connection_refused` is excluded because a bound request handler cannot produce a true refusal.
When random weights include `stall`, configure the client under test with a short stream-idle timeout so the scenario terminates promptly.
## Timing and content controls
The CLI exposes `--success-text`, `--partial-text`, `--reasoning-text`, `--chunk-size`, `--chunk-delay-ms`, `--disconnect-delay-ms`, `--retry-after-ms`, `--request-id`, `--tool-name`, and `--tool-arguments`. Millisecond delays are bounded integers within Node's timer range; `retryAfterMs` must also be positive. The library accepts the same camel-case options. An optional exact `apiKey` validates `Authorization: Bearer <token>`; omission accepts any token.
## Model Experience
None, as this test server substitutes provider wire behavior without invoking a real model.
#### KV Cache effect
None; requests terminate locally and never reach a provider cache.
## Known Limitations and Deferred Work
- **Random weights model test pressure, not production incidence** — callers that want an environment-specific distribution must provide measured weights and record the emitted seed.
- **Request scripts are arrival-ordered** — concurrent callers share one cursor, so deterministic per-session fault assignment requires separate server instances.
- **True connection refusal is a listener lifecycle phase** — the CLI delay must overlap the client attempt; request-level random selection can only reset an accepted connection.

45
packages/support/llm-mock-server/package.json Normal file
View File

@@ -0,0 +1,45 @@
{
"name": "@deepseek-ai/dsh-llm-mock-server",
"description": "Scriptable OpenAI-compatible HTTP/SSE fault server for LLM recovery tests",
"version": "0.0.1",
"private": true,
"type": "module",
"main": "lib/index.js",
"types": "lib/types/index.d.ts",
"bin": {
"dsh-llm-mock-server": "lib/bin.js"
},
"exports": {
".": {
"types": "./lib/types/index.d.ts",
"default": "./lib/index.js"
},
"./invariant": {
"types": "./lib/types/invariant.d.ts",
"default": "./lib/invariant.js"
},
"./bin": {
"types": "./lib/types/bin.d.ts",
"default": "./lib/bin.js"
},
"./src/*": "./src/*",
"./package.json": "./package.json"
},
"files": [
"lib/index.js",
"lib/invariant.js",
"lib/bin.js",
"lib/types/**/*.d.ts",
"lib/types/**/*.d.ts.map",
"src"
],
"license": "BSD-3-Clause",
"peerDependencies": {
"@deepseek-ai/dsh-invariants": "^0.0.1",
"cordis": "^4.0.0-rc.7"
},
"devDependencies": {
"@deepseek-ai/dsh-invariants": "workspace:^",
"cordis": "^4.0.0-rc.7"
}
}

50
packages/support/llm-mock-server/src/bin.ts Normal file
View File

@@ -0,0 +1,50 @@
#!/usr/bin/env node
/**
* Standalone process wrapper for the scriptable mock LLM server.
* @module @deepseek-ai/dsh-llm-mock-server/bin
*/
import { setTimeout as delay } from 'node:timers/promises'
import { MOCK_LLM_CLI_USAGE, parseMockLlmCliArgs } from './cli.ts'
import { startMockLlmServer } from './index.ts'
/* v8 ignore start -- thin process/signal glue; parser and server behavior are covered directly */
try {
const parsed = parseMockLlmCliArgs(process.argv.slice(2))
if (parsed.kind === 'help') {
process.stdout.write(MOCK_LLM_CLI_USAGE)
} else {
const { server: serverOptions, listenDelayMs, startsUnavailable } = parsed.config
const host = serverOptions.host ?? '127.0.0.1'
const port = serverOptions.port ?? 8_000
if (startsUnavailable) {
process.stdout.write(`${JSON.stringify({
type: 'unavailable',
baseURL: `http://${host}:${port}/v1`,
listenDelayMs,
})}\n`)
await delay(listenDelayMs)
}
const server = await startMockLlmServer({
...serverOptions,
onEvent: (event) => { process.stdout.write(`${JSON.stringify(event)}\n`) },
})
process.stdout.write(`${JSON.stringify({
type: 'ready',
baseURL: `${server.baseURL}/v1`,
randomSeed: server.randomSeed,
})}\n`)
let closing = false
const close = (code: number): void => {
if (closing) return
closing = true
void server.close().finally(() => { process.exit(code) })
}
process.on('SIGINT', () => { close(130) })
process.on('SIGTERM', () => { close(143) })
}
} catch (error: unknown) {
process.stderr.write(`${error instanceof Error ? error.message : String(error)}\n\n${MOCK_LLM_CLI_USAGE}`)
process.exitCode = 1
}
/* v8 ignore stop */

222
packages/support/llm-mock-server/src/cli.ts Normal file
View File

@@ -0,0 +1,222 @@
/**
* Dependency-free CLI parsing for the standalone mock LLM server.
* @module @deepseek-ai/dsh-llm-mock-server/cli
*/
import { MAX_MOCK_LLM_TIMER_DELAY_MS, MOCK_LLM_BEHAVIORS } from './index.ts'
import type {
ConcreteMockLlmBehavior,
MockLlmBehavior,
MockLlmRandomWeights,
MockLlmServerOptions,
} from './index.ts'
/** Listener lifecycle behavior understood only by the standalone CLI. */
export const CONNECTION_REFUSED_BEHAVIOR = 'connection_refused'
/** Parsed CLI configuration, including a pre-listen unavailable interval. */
export interface MockLlmCliConfig {
/** Server options after removing the lifecycle-only `connection_refused` entry. */
readonly server: MockLlmServerOptions
/** Delay before binding the model port; an integer from zero through the Node timer maximum. */
readonly listenDelayMs: number
/** Whether the original sequence requested a true pre-listen refusal phase. */
readonly startsUnavailable: boolean
}
/** Result of parsing `dsh-llm-mock-server` arguments. */
export type MockLlmCliParseResult =
| { readonly kind: 'help' }
| { readonly kind: 'run'; readonly config: MockLlmCliConfig }
const BEHAVIORS = new Set<string>(MOCK_LLM_BEHAVIORS)
const DEFAULT_LISTEN_DELAY_MS = 750
/** Command usage written for `--help` and invalid arguments. */
export const MOCK_LLM_CLI_USAGE = `Usage: dsh-llm-mock-server [options]
Required:
--sequence <a,b,...> Ordered behaviors; connection_refused is allowed first
Listener:
--host <host> Default 127.0.0.1
--port <port> Default 8000; required and nonzero for connection_refused
--api-key <token> Validate exact Bearer token when present
--listen-delay-ms <ms> Unavailable interval (default 750 with connection_refused)
--repeat-last Repeat the final request behavior after exhaustion
--seed <uint32> Reproduce random selections
--random-weights <a=n,...> Relative weights for concrete behaviors
Response:
--success-text <text>
--partial-text <text>
--reasoning-text <text>
--chunk-size <count>
--chunk-delay-ms <ms>
--disconnect-delay-ms <ms>
--retry-after-ms <ms>
--request-id <id>
--tool-name <name>
--tool-arguments <json>
Other:
--help
`
function optionValue(argv: readonly string[], index: number, option: string): string {
const value = argv[index + 1]
if (value === undefined || value.startsWith('--')) {
throw new Error(`dsh-llm-mock-server: ${option} requires a value`)
}
return value
}
function numberValue(option: string, value: string): number {
const parsed = Number(value)
if (!Number.isFinite(parsed)) throw new Error(`dsh-llm-mock-server: ${option} must be a finite number`)
return parsed
}
function boundedIntegerValue(option: string, value: string, min: number, max: number): number {
const parsed = numberValue(option, value)
if (!Number.isInteger(parsed) || parsed < min || parsed > max) {
throw new Error(`dsh-llm-mock-server: ${option} must be an integer between ${min} and ${max}`)
}
return parsed
}
function parseSequence(raw: string): { startsUnavailable: boolean; sequence: MockLlmBehavior[] } {
const entries = raw.split(',').map(entry => entry.trim())
if (entries.some(entry => entry.length === 0)) {
throw new Error('dsh-llm-mock-server: --sequence must contain non-empty comma-separated behaviors')
}
const startsUnavailable = entries[0] === CONNECTION_REFUSED_BEHAVIOR
if (entries.slice(1).includes(CONNECTION_REFUSED_BEHAVIOR)) {
throw new Error('dsh-llm-mock-server: connection_refused is allowed only as the first behavior')
}
const requestEntries = startsUnavailable ? entries.slice(1) : entries
if (requestEntries.length === 0) {
throw new Error('dsh-llm-mock-server: connection_refused must be followed by a request behavior')
}
for (const entry of requestEntries) {
if (!BEHAVIORS.has(entry)) throw new Error(`dsh-llm-mock-server: unknown behavior ${JSON.stringify(entry)}`)
}
return { startsUnavailable, sequence: requestEntries as MockLlmBehavior[] }
}
function parseRandomWeights(raw: string): MockLlmRandomWeights {
const weights: MockLlmRandomWeights = {}
for (const entry of raw.split(',')) {
const [behavior, rawWeight, ...extra] = entry.split('=')
if (behavior === undefined || behavior === '' || rawWeight === undefined || rawWeight === '' || extra.length > 0) {
throw new Error('dsh-llm-mock-server: --random-weights expects behavior=weight comma-separated entries')
}
if (!BEHAVIORS.has(behavior) || behavior === 'random') {
throw new Error(`dsh-llm-mock-server: random weight requires a concrete behavior, got ${JSON.stringify(behavior)}`)
}
if (Object.hasOwn(weights, behavior)) {
throw new Error(`dsh-llm-mock-server: duplicate random weight for ${JSON.stringify(behavior)}`)
}
weights[behavior as ConcreteMockLlmBehavior] = numberValue('--random-weights', rawWeight)
}
return weights
}
/**
* Parse standalone server arguments without starting a process or listener.
* @param argv - arguments after the executable name.
* @returns help or validated run configuration.
*/
export function parseMockLlmCliArgs(argv: readonly string[]): MockLlmCliParseResult {
if (argv.includes('--help')) return { kind: 'help' }
let sequenceRaw: string | undefined
let host: string | undefined
let port = 8_000
let apiKey: string | undefined
let listenDelayMs: number | undefined
let repeatLast = false
let randomSeed: number | undefined
let randomWeights: MockLlmRandomWeights | undefined
let successText: string | undefined
let partialText: string | undefined
let reasoningText: string | undefined
let chunkSize: number | undefined
let chunkDelayMs: number | undefined
let disconnectDelayMs: number | undefined
let retryAfterMs: number | undefined
let requestId: string | undefined
let toolName: string | undefined
let toolArguments: string | undefined
for (let index = 0; index < argv.length; index += 1) {
const option = argv[index] as string
if (option === '--repeat-last') {
repeatLast = true
continue
}
const value = optionValue(argv, index, option)
index += 1
switch (option) {
case '--sequence': sequenceRaw = value; break
case '--host': host = value; break
case '--port': port = numberValue(option, value); break
case '--api-key': apiKey = value; break
case '--listen-delay-ms':
listenDelayMs = boundedIntegerValue(option, value, 0, MAX_MOCK_LLM_TIMER_DELAY_MS)
break
case '--seed': randomSeed = numberValue(option, value); break
case '--random-weights': randomWeights = parseRandomWeights(value); break
case '--success-text': successText = value; break
case '--partial-text': partialText = value; break
case '--reasoning-text': reasoningText = value; break
case '--chunk-size': chunkSize = numberValue(option, value); break
case '--chunk-delay-ms': chunkDelayMs = numberValue(option, value); break
case '--disconnect-delay-ms': disconnectDelayMs = numberValue(option, value); break
case '--retry-after-ms': retryAfterMs = numberValue(option, value); break
case '--request-id': requestId = value; break
case '--tool-name': toolName = value; break
case '--tool-arguments': toolArguments = value; break
default: throw new Error(`dsh-llm-mock-server: unknown option ${JSON.stringify(option)}`)
}
}
if (sequenceRaw === undefined) throw new Error('dsh-llm-mock-server: --sequence is required')
const parsedSequence = parseSequence(sequenceRaw)
if (parsedSequence.startsUnavailable && port === 0) {
throw new Error('dsh-llm-mock-server: connection_refused requires an explicit nonzero --port')
}
if (!parsedSequence.startsUnavailable && listenDelayMs !== undefined) {
throw new Error('dsh-llm-mock-server: --listen-delay-ms requires connection_refused first in --sequence')
}
if (!parsedSequence.sequence.includes('random') && (randomSeed !== undefined || randomWeights !== undefined)) {
throw new Error('dsh-llm-mock-server: --seed and --random-weights require random in --sequence')
}
return {
kind: 'run',
config: {
server: {
sequence: parsedSequence.sequence,
port,
repeatLast,
...randomSeed === undefined ? {} : { randomSeed },
...randomWeights === undefined ? {} : { randomWeights },
...host === undefined ? {} : { host },
...apiKey === undefined ? {} : { apiKey },
...successText === undefined ? {} : { successText },
...partialText === undefined ? {} : { partialText },
...reasoningText === undefined ? {} : { reasoningText },
...chunkSize === undefined ? {} : { chunkSize },
...chunkDelayMs === undefined ? {} : { chunkDelayMs },
...disconnectDelayMs === undefined ? {} : { disconnectDelayMs },
...retryAfterMs === undefined ? {} : { retryAfterMs },
...requestId === undefined ? {} : { requestId },
...toolName === undefined ? {} : { toolName },
...toolArguments === undefined ? {} : { toolArguments },
},
listenDelayMs: parsedSequence.startsUnavailable ? listenDelayMs ?? DEFAULT_LISTEN_DELAY_MS : 0,
startsUnavailable: parsedSequence.startsUnavailable,
},
}
}

738
packages/support/llm-mock-server/src/index.ts Normal file
View File

@@ -0,0 +1,738 @@
/**
* Scriptable OpenAI-compatible HTTP/SSE server for transport, protocol, and
* semantic-empty LLM recovery tests. Each accepted chat-completions request
* consumes one behavior; the server never retries or interprets harness policy.
*
* @module @deepseek-ai/dsh-llm-mock-server
*/
import { createServer } from 'node:http'
import type { IncomingHttpHeaders, IncomingMessage, ServerResponse } from 'node:http'
import { randomBytes } from 'node:crypto'
import { isIP, type AddressInfo } from 'node:net'
import { setTimeout as delay } from 'node:timers/promises'
/** Request-scoped behaviors accepted by {@link startMockLlmServer}. */
export const MOCK_LLM_BEHAVIORS = [
'connection_reset',
'stream_disconnect',
'empty',
'empty_body',
'stream_eof',
'partial_eof',
'partial_disconnect',
'stall',
'malformed_json',
'malformed_event',
'wrong_content_type',
'rate_limit',
'server_error',
'service_unavailable',
'auth_error',
'invalid_request',
'context_overflow',
'quota_exceeded',
'success',
'reasoning_success',
'tool_call_success',
'max_tokens',
'slow_success',
'random',
] as const
/** One scripted mock behavior name; `random` selects a concrete behavior per request. */
export type MockLlmBehavior = typeof MOCK_LLM_BEHAVIORS[number]
/** One concrete request behavior after resolving a `random` script entry. */
export type ConcreteMockLlmBehavior = Exclude<MockLlmBehavior, 'random'>
/** Relative non-negative weights for random request behavior selection. */
export type MockLlmRandomWeights = Partial<Record<ConcreteMockLlmBehavior, number>>
/**
* Default stress profile for `random`. Weights are configurable test pressure,
* not a claim about production incident frequency.
*/
export const DEFAULT_MOCK_LLM_RANDOM_WEIGHTS: Readonly<MockLlmRandomWeights> = Object.freeze({
success: 48,
slow_success: 10,
max_tokens: 2,
connection_reset: 5,
stream_disconnect: 5,
partial_disconnect: 10,
empty: 5,
stall: 2,
rate_limit: 5,
server_error: 4,
service_unavailable: 2,
partial_eof: 1,
malformed_json: 1,
})
/** Largest millisecond delay accepted by Node timers without truncation. */
export const MAX_MOCK_LLM_TIMER_DELAY_MS = 2_147_483_647
/** How one accepted request ended at the mock boundary. */
export type MockLlmRequestOutcome = 'completed' | 'reset' | 'stalled' | 'client_closed' | 'server_error'
/** Immutable telemetry emitted when a request starts or reaches an outcome. */
export type MockLlmServerEvent =
| {
readonly type: 'request'
readonly attempt: number
readonly scriptBehavior: MockLlmBehavior | 'script_exhausted'
readonly behavior: ConcreteMockLlmBehavior | 'script_exhausted'
readonly path: string
}
| {
readonly type: 'result'
readonly attempt: number
readonly scriptBehavior: MockLlmBehavior | 'script_exhausted'
readonly behavior: ConcreteMockLlmBehavior | 'script_exhausted'
readonly outcome: MockLlmRequestOutcome
readonly chunksSent: number
}
/** Captured wire request and its final server-side outcome. */
export interface MockLlmRequestRecord {
/** One-based accepted chat-completions request number. */
readonly attempt: number
/** Script entry consumed for this request before random resolution. */
readonly scriptBehavior: MockLlmBehavior | 'script_exhausted'
/** Concrete behavior selected for this request, or exhaustion after the configured script. */
readonly behavior: ConcreteMockLlmBehavior | 'script_exhausted'
/** Original request path, including a `/v1` prefix when the client supplied one. */
readonly path: string
/** Detached request headers. */
readonly headers: Readonly<IncomingHttpHeaders>
/** Parsed JSON request body. */
readonly body: unknown
/** Number of SSE `data:` events handed to Node before the outcome. */
chunksSent: number
/** Final server-side outcome; absent while a stalled request remains open. */
outcome?: MockLlmRequestOutcome
}
/** Configuration for one mock server instance. */
export interface MockLlmServerOptions {
/** Loopback host by default. */
readonly host?: string
/** TCP port; zero requests an OS-assigned port. */
readonly port?: number
/** Optional exact bearer token; omission accepts any authorization header. */
readonly apiKey?: string
/** Ordered request behaviors; exhaustion fails loud unless `repeatLast` is true. */
readonly sequence: readonly MockLlmBehavior[]
/** Reuse the final behavior after the sequence is consumed. */
readonly repeatLast?: boolean
/** Optional deterministic unsigned 32-bit seed; omission generates and exposes one. */
readonly randomSeed?: number
/** Relative weights used whenever a script entry is `random`. */
readonly randomWeights?: Readonly<MockLlmRandomWeights>
/** Complete text returned by success-shaped behaviors. */
readonly successText?: string
/** Text emitted before partial EOF/reset behaviors terminate. */
readonly partialText?: string
/** Reasoning text emitted by `reasoning_success`. */
readonly reasoningText?: string
/** Unicode code-point count per text or reasoning SSE delta. */
readonly chunkSize?: number
/** Inter-chunk delay for `slow_success`, in milliseconds. */
readonly chunkDelayMs?: number
/** Delay after headers/deltas before a forced disconnect, in milliseconds. */
readonly disconnectDelayMs?: number
/** Provider retry delay; the wire `Retry-After` value rounds up to whole seconds. */
readonly retryAfterMs?: number
/** Optional provider request id returned on HTTP failures. */
readonly requestId?: string
/** Tool name emitted by `tool_call_success`. */
readonly toolName?: string
/** Raw JSON arguments emitted by `tool_call_success`. */
readonly toolArguments?: string
/** Optional observer for JSONL CLI telemetry; observer failures never affect wire behavior. */
readonly onEvent?: (event: MockLlmServerEvent) => void
}
/** Running mock server and captured request state. */
export interface MockLlmServer {
/** Base URL without `/v1`; both root and `/v1` chat-completions paths are accepted. */
readonly baseURL: string
/** Actual bound port, including an OS-assigned value. */
readonly port: number
/** Seed used for random behavior selection, including the generated default. */
readonly randomSeed: number
/** Live request records in arrival order. */
readonly requests: readonly MockLlmRequestRecord[]
/** Stop accepting requests and force-close stalled/streaming connections; idempotent. */
close(): Promise<void>
}
interface ResolvedOptions {
readonly host: string
readonly port: number
readonly apiKey?: string
readonly sequence: readonly MockLlmBehavior[]
readonly lastBehavior: MockLlmBehavior
readonly repeatLast: boolean
readonly randomSeed: number
readonly randomWeights: readonly (readonly [ConcreteMockLlmBehavior, number])[]
readonly successText: string
readonly partialText: string
readonly reasoningText: string
readonly chunkSize: number
readonly chunkDelayMs: number
readonly disconnectDelayMs: number
readonly retryAfterMs: number
readonly requestId?: string
readonly toolName: string
readonly toolArguments: string
readonly onEvent?: (event: MockLlmServerEvent) => void
}
const DEFAULT_SUCCESS_TEXT = 'mock response recovered'
const DEFAULT_PARTIAL_TEXT = 'discarded partial response'
const DEFAULT_REASONING_TEXT = 'mock reasoning'
const CONCRETE_BEHAVIORS = new Set<string>(MOCK_LLM_BEHAVIORS.filter(behavior => behavior !== 'random'))
function boundedInteger(name: string, value: number, min: number, max: number): number {
if (!Number.isInteger(value) || value < min || value > max) {
throw new Error(`llm-mock-server: ${name} must be an integer between ${min} and ${max}`)
}
return value
}
function resolveOptions(options: MockLlmServerOptions): ResolvedOptions {
const host = options.host ?? '127.0.0.1'
const port = boundedInteger('port', options.port ?? 0, 0, 65_535)
const chunkSize = boundedInteger('chunkSize', options.chunkSize ?? 8, 1, Number.MAX_SAFE_INTEGER)
const chunkDelayMs = boundedInteger(
'chunkDelayMs',
options.chunkDelayMs ?? 25,
0,
MAX_MOCK_LLM_TIMER_DELAY_MS,
)
const disconnectDelayMs = boundedInteger(
'disconnectDelayMs',
options.disconnectDelayMs ?? 10,
0,
MAX_MOCK_LLM_TIMER_DELAY_MS,
)
const retryAfterMs = boundedInteger(
'retryAfterMs',
options.retryAfterMs ?? 1_000,
1,
MAX_MOCK_LLM_TIMER_DELAY_MS,
)
const randomSeed = boundedInteger(
'randomSeed',
options.randomSeed ?? randomBytes(4).readUInt32LE(0),
0,
0xffff_ffff,
)
const successText = options.successText ?? DEFAULT_SUCCESS_TEXT
const partialText = options.partialText ?? DEFAULT_PARTIAL_TEXT
const reasoningText = options.reasoningText ?? DEFAULT_REASONING_TEXT
const toolName = options.toolName ?? 'mock_tool'
const toolArguments = options.toolArguments ?? '{"value":"mock"}'
if (host.length === 0) throw new Error('llm-mock-server: host must not be empty')
if (options.sequence.length === 0) throw new Error('llm-mock-server: sequence must not be empty')
const lastBehavior = options.sequence.reduce((_previous, behavior) => behavior)
if (options.apiKey === '') throw new Error('llm-mock-server: apiKey must not be empty')
if (successText.length === 0) throw new Error('llm-mock-server: successText must not be empty')
if (partialText.length === 0) throw new Error('llm-mock-server: partialText must not be empty')
if (reasoningText.length === 0) throw new Error('llm-mock-server: reasoningText must not be empty')
if (toolName.length === 0) throw new Error('llm-mock-server: toolName must not be empty')
if (options.requestId === '') throw new Error('llm-mock-server: requestId must not be empty')
try {
JSON.parse(toolArguments)
} catch {
throw new Error('llm-mock-server: toolArguments must be valid JSON')
}
const configuredWeights = options.randomWeights ?? DEFAULT_MOCK_LLM_RANDOM_WEIGHTS
const randomWeights: Array<readonly [ConcreteMockLlmBehavior, number]> = []
for (const [behavior, weight] of Object.entries(configuredWeights)) {
if (!CONCRETE_BEHAVIORS.has(behavior)) {
throw new Error(`llm-mock-server: randomWeights contains unknown concrete behavior ${JSON.stringify(behavior)}`)
}
if (!Number.isFinite(weight) || weight < 0) {
throw new Error(`llm-mock-server: random weight for ${behavior} must be a non-negative finite number`)
}
if (weight > 0) randomWeights.push([behavior as ConcreteMockLlmBehavior, weight])
}
if (randomWeights.length === 0) {
throw new Error('llm-mock-server: randomWeights must contain at least one positive weight')
}
return {
host,
port,
...options.apiKey === undefined ? {} : { apiKey: options.apiKey },
sequence: [...options.sequence],
lastBehavior,
repeatLast: options.repeatLast ?? false,
randomSeed,
randomWeights,
successText,
partialText,
reasoningText,
chunkSize,
chunkDelayMs,
disconnectDelayMs,
retryAfterMs,
...options.requestId === undefined ? {} : { requestId: options.requestId },
toolName,
toolArguments,
...options.onEvent === undefined ? {} : { onEvent: options.onEvent },
}
}
function emit(options: ResolvedOptions, event: MockLlmServerEvent): void {
try {
options.onEvent?.(Object.freeze(event))
} catch (_telemetryObserverFailure) {
// Test telemetry is observational; a broken observer cannot change provider wire behavior.
}
}
async function readJsonBody(request: IncomingMessage): Promise<unknown> {
const chunks: Buffer[] = []
for await (const chunk of request) chunks.push(Buffer.from(chunk as Uint8Array))
const body = Buffer.concat(chunks).toString('utf8')
return body.length === 0 ? undefined : JSON.parse(body)
}
function splitText(text: string, size: number): string[] {
const points = Array.from(text)
const chunks: string[] = []
for (let index = 0; index < points.length; index += size) chunks.push(points.slice(index, index + size).join(''))
return chunks
}
function openSse(response: ServerResponse, contentType = 'text/event-stream; charset=utf-8'): void {
response.writeHead(200, {
'content-type': contentType,
'cache-control': 'no-cache',
'connection': 'keep-alive',
})
response.flushHeaders()
}
function writeSse(record: MockLlmRequestRecord, response: ServerResponse, payload: unknown): void {
response.write(`data: ${typeof payload === 'string' ? payload : JSON.stringify(payload)}\n\n`)
record.chunksSent += 1
}
function writeDone(record: MockLlmRequestRecord, response: ServerResponse): void {
writeSse(record, response, '[DONE]')
}
function finishRecord(
options: ResolvedOptions,
record: MockLlmRequestRecord,
outcome: MockLlmRequestOutcome,
): void {
if (record.outcome !== undefined) return
record.outcome = outcome
emit(options, {
type: 'result',
attempt: record.attempt,
scriptBehavior: record.scriptBehavior,
behavior: record.behavior,
outcome,
chunksSent: record.chunksSent,
})
}
function httpError(
options: ResolvedOptions,
record: MockLlmRequestRecord,
response: ServerResponse,
status: number,
message: string,
code: string,
type = 'mock_error',
): void {
const headers: Record<string, string> = { 'content-type': 'application/json' }
if (record.behavior === 'rate_limit') {
headers['retry-after'] = String(Math.ceil(options.retryAfterMs / 1_000))
}
if (options.requestId !== undefined) headers['x-request-id'] = options.requestId
response.writeHead(status, headers)
response.end(JSON.stringify({ error: { message, type, code } }))
finishRecord(options, record, 'completed')
}
function terminalChunk(reason: string, outputTokens: number): unknown {
return {
choices: [{ index: 0, delta: { content: '' }, finish_reason: reason }],
usage: { prompt_tokens: 3, completion_tokens: outputTokens },
}
}
async function pause(milliseconds: number, response: ServerResponse): Promise<boolean> {
if (milliseconds === 0) return !response.destroyed
const controller = new AbortController()
const stop = (): void => { controller.abort() }
response.once('close', stop)
try {
await delay(milliseconds, undefined, { signal: controller.signal })
return true
} catch (_responseClosed) {
// The timer only receives this response-owned abort signal; closing the response cancels its wait.
return false
} finally {
response.off('close', stop)
}
}
async function streamText(
options: ResolvedOptions,
record: MockLlmRequestRecord,
response: ServerResponse,
text: string,
delayMs: number,
): Promise<boolean> {
for (const chunk of splitText(text, options.chunkSize)) {
writeSse(record, response, { choices: [{ index: 0, delta: { content: chunk }, finish_reason: null }] })
if (!await pause(delayMs, response)) return false
}
return true
}
async function completeText(
options: ResolvedOptions,
record: MockLlmRequestRecord,
response: ServerResponse,
reason: 'stop' | 'length',
delayMs: number,
): Promise<void> {
if (!await streamText(options, record, response, options.successText, delayMs)) {
finishRecord(options, record, 'client_closed')
return
}
writeSse(record, response, terminalChunk(reason, Array.from(options.successText).length))
writeDone(record, response)
response.end()
finishRecord(options, record, 'completed')
}
async function disconnect(
options: ResolvedOptions,
record: MockLlmRequestRecord,
response: ServerResponse,
): Promise<void> {
if (!await pause(options.disconnectDelayMs, response)) {
finishRecord(options, record, 'client_closed')
return
}
finishRecord(options, record, 'reset')
response.destroy()
}
function toolCallChunks(options: ResolvedOptions): readonly unknown[] {
const midpoint = Math.max(1, Math.floor(options.toolArguments.length / 2))
return [
{
choices: [{
index: 0,
delta: {
tool_calls: [{
index: 0,
id: 'mock-call-1',
type: 'function',
function: { name: options.toolName, arguments: options.toolArguments.slice(0, midpoint) },
}],
},
finish_reason: null,
}],
},
{
choices: [{
index: 0,
delta: { tool_calls: [{ index: 0, function: { arguments: options.toolArguments.slice(midpoint) } }] },
finish_reason: null,
}],
},
]
}
async function runBehavior(
options: ResolvedOptions,
record: MockLlmRequestRecord,
request: IncomingMessage,
response: ServerResponse,
): Promise<void> {
switch (record.behavior) {
case 'script_exhausted':
httpError(options, record, response, 500, 'mock script exhausted', 'MOCK_SCRIPT_EXHAUSTED')
return
case 'connection_reset':
finishRecord(options, record, 'reset')
request.socket.destroy()
return
case 'stream_disconnect':
openSse(response)
await disconnect(options, record, response)
return
case 'empty':
openSse(response)
writeSse(record, response, terminalChunk('stop', 0))
writeDone(record, response)
response.end()
finishRecord(options, record, 'completed')
return
case 'empty_body':
openSse(response)
response.end()
finishRecord(options, record, 'completed')
return
case 'stream_eof':
openSse(response)
writeSse(record, response, { choices: [{ index: 0, delta: { role: 'assistant' }, finish_reason: null }] })
response.end()
finishRecord(options, record, 'completed')
return
case 'partial_eof':
openSse(response)
await streamText(options, record, response, options.partialText, 0)
response.end()
finishRecord(options, record, 'completed')
return
case 'partial_disconnect':
openSse(response)
if (!await streamText(options, record, response, options.partialText, options.chunkDelayMs)) return
await disconnect(options, record, response)
return
case 'stall':
openSse(response)
finishRecord(options, record, 'stalled')
return
case 'malformed_json':
openSse(response)
writeSse(record, response, '{not-json')
writeDone(record, response)
response.end()
finishRecord(options, record, 'completed')
return
case 'malformed_event':
openSse(response)
writeSse(record, response, { choices: [null] })
writeDone(record, response)
response.end()
finishRecord(options, record, 'completed')
return
case 'wrong_content_type':
openSse(response, 'application/json')
await completeText(options, record, response, 'stop', 0)
return
case 'rate_limit':
httpError(options, record, response, 429, 'mock rate limit', 'rate_limit')
return
case 'server_error':
httpError(options, record, response, 500, 'mock server error', 'server_error')
return
case 'service_unavailable':
httpError(options, record, response, 503, 'mock service unavailable', 'service_unavailable')
return
case 'auth_error':
httpError(options, record, response, 401, 'mock authentication failed', 'invalid_api_key')
return
case 'invalid_request':
httpError(options, record, response, 400, 'mock invalid request', 'invalid_request')
return
case 'context_overflow':
httpError(
options,
record,
response,
400,
'mock input exceeds the model context window',
'context_length_exceeded',
'invalid_request_error',
)
return
case 'quota_exceeded':
httpError(options, record, response, 429, 'mock insufficient quota', 'insufficient_quota')
return
case 'success':
openSse(response)
await completeText(options, record, response, 'stop', 0)
return
case 'reasoning_success':
openSse(response)
for (const chunk of splitText(options.reasoningText, options.chunkSize)) {
writeSse(record, response, {
choices: [{ index: 0, delta: { reasoning_content: chunk }, finish_reason: null }],
})
}
await completeText(options, record, response, 'stop', 0)
return
case 'tool_call_success':
openSse(response)
for (const chunk of toolCallChunks(options)) writeSse(record, response, chunk)
writeSse(record, response, terminalChunk('tool_calls', 2))
writeDone(record, response)
response.end()
finishRecord(options, record, 'completed')
return
case 'max_tokens':
openSse(response)
await completeText(options, record, response, 'length', 0)
return
case 'slow_success':
openSse(response)
await completeText(options, record, response, 'stop', options.chunkDelayMs)
return
}
}
function seededRandom(seed: number): () => number {
let state = seed
return () => {
state = (state + 0x6d2b_79f5) >>> 0
let mixed = state
mixed = Math.imul(mixed ^ mixed >>> 15, mixed | 1)
mixed ^= mixed + Math.imul(mixed ^ mixed >>> 7, mixed | 61)
return ((mixed ^ mixed >>> 14) >>> 0) / 0x1_0000_0000
}
}
function chooseRandomBehavior(
weights: readonly (readonly [ConcreteMockLlmBehavior, number])[],
random: () => number,
): ConcreteMockLlmBehavior {
const total = weights.reduce((sum, entry) => sum + entry[1], 0)
let draw = random() * total
for (const [behavior, weight] of weights) {
if (draw < weight) return behavior
draw -= weight
}
// Floating-point subtraction can only leave a rounding residue at the upper boundary.
/* v8 ignore next -- seededRandom is strictly less than one; this guards floating-point residue only */
return (weights.at(-1) as readonly [ConcreteMockLlmBehavior, number])[0]
}
/**
* Start a local chat-completions server that consumes one configured behavior
* per accepted request. Only a `POST` path ending in `/chat/completions` consumes the script;
* invalid routes, methods, authorization, and JSON receive ordinary 4xx
* responses. Closing the handle terminates stalled connections.
*
* @param options - listener, script, response content, timing, and telemetry options.
* @returns the listening handle after the port is bound.
*/
export async function startMockLlmServer(options: MockLlmServerOptions): Promise<MockLlmServer> {
const resolved = resolveOptions(options)
const requests: MockLlmRequestRecord[] = []
const random = seededRandom(resolved.randomSeed)
let cursor = 0
const selectBehavior = (): {
scriptBehavior: MockLlmBehavior | 'script_exhausted'
behavior: ConcreteMockLlmBehavior | 'script_exhausted'
} => {
const selected = resolved.sequence[cursor]
cursor += 1
const scriptBehavior = selected
?? (resolved.repeatLast ? resolved.lastBehavior : 'script_exhausted')
return {
scriptBehavior,
behavior: scriptBehavior === 'random'
? chooseRandomBehavior(resolved.randomWeights, random)
: scriptBehavior,
}
}
const handle = async (request: IncomingMessage, response: ServerResponse): Promise<void> => {
/* v8 ignore next -- node:http server requests always carry a URL despite the shared optional type */
const path = new URL(request.url ?? '/', 'http://mock.invalid').pathname
if (request.method !== 'POST') {
response.writeHead(405, { allow: 'POST' }).end()
return
}
if (!path.endsWith('/chat/completions')) {
response.writeHead(404).end()
return
}
if (resolved.apiKey !== undefined && request.headers.authorization !== `Bearer ${resolved.apiKey}`) {
response.writeHead(401, { 'content-type': 'application/json' })
response.end(JSON.stringify({ error: { message: 'invalid mock bearer token', code: 'invalid_api_key' } }))
return
}
let body: unknown
try {
body = await readJsonBody(request)
} catch {
response.writeHead(400, { 'content-type': 'application/json' })
response.end(JSON.stringify({ error: { message: 'request body must be valid JSON', code: 'invalid_json' } }))
return
}
const selected = selectBehavior()
const record: MockLlmRequestRecord = {
attempt: requests.length + 1,
scriptBehavior: selected.scriptBehavior,
behavior: selected.behavior,
path,
headers: { ...request.headers },
body,
chunksSent: 0,
}
requests.push(record)
response.once('close', () => {
if (!response.writableFinished && record.outcome === undefined) {
finishRecord(resolved, record, 'client_closed')
}
})
emit(resolved, {
type: 'request',
attempt: record.attempt,
scriptBehavior: record.scriptBehavior,
behavior: record.behavior,
path,
})
await runBehavior(resolved, record, request, response)
}
const server = createServer((request, response) => {
/* v8 ignore start -- last-resort containment for Node response failures after validated test inputs */
handle(request, response).catch((error: unknown) => {
const record = requests.at(-1)
if (record !== undefined) finishRecord(resolved, record, 'server_error')
if (response.headersSent) {
response.destroy(error instanceof Error ? error : new Error(String(error)))
return
}
response.writeHead(500, { 'content-type': 'application/json' })
response.end(JSON.stringify({ error: { message: 'mock server handler failed', code: 'MOCK_HANDLER_FAILED' } }))
})
/* v8 ignore stop */
})
let closing: Promise<void> | undefined
const close = (): Promise<void> => (closing ??= new Promise((resolveClose) => {
server.close(() => { resolveClose() })
server.closeAllConnections()
}))
await new Promise<void>((resolveListen, rejectListen) => {
server.once('error', rejectListen)
server.listen(resolved.port, resolved.host, () => {
server.off('error', rejectListen)
resolveListen()
})
})
const address = server.address() as AddressInfo
const advertisedHost = isIP(resolved.host) === 6 ? `[${resolved.host}]` : resolved.host
return {
baseURL: `http://${advertisedHost}:${address.port}`,
port: address.port,
randomSeed: resolved.randomSeed,
requests,
close,
}
}

30
packages/support/llm-mock-server/src/invariant.ts Normal file
View File

@@ -0,0 +1,30 @@
/**
* Package-owned invariant companion for `@deepseek-ai/dsh-llm-mock-server`.
* @module @deepseek-ai/dsh-llm-mock-server/invariant
*/
/* jscpd:ignore-start */
import type { Context } from 'cordis'
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
const PACKAGE_NAME = '@deepseek-ai/dsh-llm-mock-server'
/** Cordis companion plugin name. */
export const name = 'llm-mock-server-invariant'
/** Service required before the companion can reserve package ownership. */
export const inject = ['invariants']
/**
* No runtime invariant: this standalone test server owns no Cordis event stream or shared data;
* its wire behavior and lifecycle are exercised through direct HTTP and assembled-loop tests.
*/
const install: InvariantInstaller = () => {}
/**
* Register this package's invariant companion.
* @param ctx - Cordis context carrying the invariant service.
* @returns the installed registration's disposer after setup succeeds.
*/
export const apply = (ctx: Context): Promise<() => void> =>
Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
/* jscpd:ignore-end */

124
packages/support/llm-mock-server/tests/cli.spec.ts Normal file
View File

@@ -0,0 +1,124 @@
import { describe, expect, it } from 'vitest'
import {
MOCK_LLM_CLI_USAGE,
parseMockLlmCliArgs,
} from '../src/cli.ts'
describe('mock LLM server CLI parser', () => {
it('returns help without requiring a sequence', () => {
expect(parseMockLlmCliArgs(['--help'])).toEqual({ kind: 'help' })
expect(MOCK_LLM_CLI_USAGE).toContain('--sequence')
})
it('parses every request and listener option', () => {
expect(parseMockLlmCliArgs([
'--sequence', 'connection_refused,partial_disconnect,success',
'--host', 'localhost',
'--port', '9010',
'--api-key', 'mock-key',
'--listen-delay-ms', '100',
'--repeat-last',
'--success-text', 'done',
'--partial-text', 'half',
'--reasoning-text', 'think',
'--chunk-size', '2',
'--chunk-delay-ms', '3',
'--disconnect-delay-ms', '4',
'--retry-after-ms', '5000',
'--request-id', 'request-1',
'--tool-name', 'lookup',
'--tool-arguments', '{"id":1}',
])).toEqual({
kind: 'run',
config: {
startsUnavailable: true,
listenDelayMs: 100,
server: {
sequence: ['partial_disconnect', 'success'],
host: 'localhost',
port: 9010,
apiKey: 'mock-key',
repeatLast: true,
successText: 'done',
partialText: 'half',
reasoningText: 'think',
chunkSize: 2,
chunkDelayMs: 3,
disconnectDelayMs: 4,
retryAfterMs: 5000,
requestId: 'request-1',
toolName: 'lookup',
toolArguments: '{"id":1}',
},
},
})
})
it('uses standalone defaults for an ordinary sequence', () => {
expect(parseMockLlmCliArgs(['--sequence', 'success'])).toEqual({
kind: 'run',
config: {
startsUnavailable: false,
listenDelayMs: 0,
server: {
sequence: ['success'],
port: 8000,
repeatLast: false,
},
},
})
})
it('uses the default unavailable interval', () => {
const result = parseMockLlmCliArgs(['--sequence', 'connection_refused,success', '--port', '8001'])
expect(result).toMatchObject({
kind: 'run',
config: { startsUnavailable: true, listenDelayMs: 750 },
})
})
it('parses a reproducible weighted random profile', () => {
expect(parseMockLlmCliArgs([
'--sequence', 'random',
'--repeat-last',
'--seed', '42',
'--random-weights', 'success=8,partial_disconnect=2',
])).toEqual({
kind: 'run',
config: {
startsUnavailable: false,
listenDelayMs: 0,
server: {
sequence: ['random'],
port: 8000,
repeatLast: true,
randomSeed: 42,
randomWeights: { success: 8, partial_disconnect: 2 },
},
},
})
})
it.each([
[[], /--sequence is required/],
[['--wat'], /requires a value/],
[['--wat', 'x'], /unknown option/],
[['--port', 'NaN', '--sequence', 'success'], /finite number/],
[['--sequence', 'success,'], /non-empty/],
[['--sequence', 'success,connection_refused'], /only as the first/],
[['--sequence', 'connection_refused'], /must be followed/],
[['--sequence', 'unknown'], /unknown behavior/],
[['--sequence', 'connection_refused,success', '--port', '0'], /nonzero/],
[['--sequence', 'success', '--listen-delay-ms', '5'], /requires connection_refused/],
[['--sequence', 'connection_refused,success', '--listen-delay-ms', '-1'], /integer between 0 and 2147483647/],
[['--sequence', 'connection_refused,success', '--listen-delay-ms', '1.5'], /integer between 0 and 2147483647/],
[['--sequence', 'connection_refused,success', '--listen-delay-ms', '2147483648'], /integer between 0 and 2147483647/],
[['--sequence', 'success', '--seed', '1'], /require random/],
[['--sequence', 'random', '--random-weights', 'success'], /expects behavior=weight/],
[['--sequence', 'random', '--random-weights', 'random=1'], /concrete behavior/],
[['--sequence', 'random', '--random-weights', 'success=1,success=2'], /duplicate/],
[['--sequence', 'random', '--random-weights', 'success=nope'], /finite number/],
])('rejects invalid argv %#', (argv, expected) => {
expect(() => parseMockLlmCliArgs(argv)).toThrow(expected)
})
})

18
packages/support/llm-mock-server/tests/invariant.spec.ts Normal file
View File

@@ -0,0 +1,18 @@
import { describe, expect, it } from 'vitest'
import { Context } from 'cordis'
import InvariantService from '@deepseek-ai/dsh-invariants'
import * as MockServerInvariant from '../src/invariant.ts'
describe('mock LLM server invariant companion', () => {
it('registers its explained empty runtime invariant', async () => {
const ctx = new Context()
await ctx.plugin(InvariantService)
const fiber = await ctx.plugin(MockServerInvariant)
expect(() => {
ctx.invariants.register('@deepseek-ai/dsh-llm-mock-server', () => {})
}).toThrow(/already registered/)
await fiber.dispose()
await ctx.fiber.dispose()
})
})

355
packages/support/llm-mock-server/tests/server.spec.ts Normal file
View File

@@ -0,0 +1,355 @@
import { request } from 'node:http'
import { afterEach, describe, expect, it } from 'vitest'
import type { MockLlmBehavior, MockLlmServer, MockLlmServerEvent } from '../src/index.ts'
import { startMockLlmServer } from '../src/index.ts'
const running: MockLlmServer[] = []
afterEach(async () => {
await Promise.all(running.splice(0).map(server => server.close()))
})
async function start(
sequence: readonly MockLlmBehavior[],
options: Omit<Parameters<typeof startMockLlmServer>[0], 'sequence'> = {},
): Promise<MockLlmServer> {
const server = await startMockLlmServer({ sequence, ...options })
running.push(server)
return server
}
function chat(
server: MockLlmServer,
options: { path?: string; key?: string; body?: string; signal?: AbortSignal } = {},
): Promise<Response> {
return fetch(`${server.baseURL}${options.path ?? '/v1/chat/completions'}`, {
method: 'POST',
headers: {
'content-type': 'application/json',
...options.key === undefined ? {} : { authorization: `Bearer ${options.key}` },
},
body: options.body ?? JSON.stringify({ model: 'mock', messages: [], stream: true }),
...options.signal === undefined ? {} : { signal: options.signal },
})
}
function rawChat(server: MockLlmServer, chunks: readonly Buffer[]): Promise<void> {
return new Promise((resolve, reject) => {
const outgoing = request(`${server.baseURL}/v1/chat/completions`, {
method: 'POST',
headers: { 'content-type': 'application/json' },
}, (response) => {
response.once('error', reject)
response.once('end', resolve)
response.resume()
})
outgoing.once('error', reject)
for (const chunk of chunks) outgoing.write(chunk)
outgoing.end()
})
}
describe('mock LLM server wire behaviors', () => {
it('streams a complete text response and captures the request', async () => {
const events: MockLlmServerEvent[] = []
const server = await start(['success'], {
apiKey: 'mock-key',
successText: 'recovered',
chunkSize: 3,
onEvent: (event) => { events.push(event) },
})
const response = await chat(server, { key: 'mock-key' })
const body = await response.text()
expect(response.status).toBe(200)
expect(response.headers.get('content-type')).toContain('text/event-stream')
expect(body).toContain('"content":"rec"')
expect(body).toContain('"content":"ove"')
expect(body).toContain('"content":"red"')
expect(body).toContain('"finish_reason":"stop"')
expect(body).toContain('data: [DONE]')
expect(server.requests).toEqual([expect.objectContaining({
attempt: 1,
behavior: 'success',
path: '/v1/chat/completions',
body: { model: 'mock', messages: [], stream: true },
chunksSent: 5,
outcome: 'completed',
})])
expect(events).toEqual([
{
type: 'request',
attempt: 1,
scriptBehavior: 'success',
behavior: 'success',
path: '/v1/chat/completions',
},
{
type: 'result',
attempt: 1,
scriptBehavior: 'success',
behavior: 'success',
outcome: 'completed',
chunksSent: 5,
},
])
})
it('supports root paths and intentionally ignores telemetry observer failures', async () => {
const server = await start(['empty'], {
onEvent() {
throw new Error('observer failed')
},
})
const response = await chat(server, { path: '/chat/completions' })
expect(response.status).toBe(200)
expect(await response.text()).toContain('data: [DONE]')
expect(server.requests[0]).toMatchObject({ path: '/chat/completions', outcome: 'completed' })
})
it.each([
['empty_body', 0, ''] as const,
['stream_eof', 1, '"role":"assistant"'] as const,
['partial_eof', 1, 'discarded partial response'] as const,
['malformed_json', 2, 'data: {not-json'] as const,
['malformed_event', 2, '"choices":[null]'] as const,
])('serves %s without inventing a terminal completion', async (behavior, chunks, marker) => {
const server = await start([behavior], { chunkSize: 100 })
const response = await chat(server)
const body = await response.text()
expect(response.status).toBe(200)
expect(body).toContain(marker)
if (behavior !== 'malformed_json' && behavior !== 'malformed_event') {
expect(body).not.toContain('[DONE]')
}
expect(server.requests[0]).toMatchObject({ behavior, chunksSent: chunks, outcome: 'completed' })
})
it.each([
['connection_reset', false] as const,
['stream_disconnect', true] as const,
['partial_disconnect', true] as const,
])('forces the %s transport boundary', async (behavior, receivesHeaders) => {
const server = await start([behavior], { disconnectDelayMs: 20, partialText: 'half' })
let headersReceived = false
await expect((async () => {
const response = await chat(server)
headersReceived = true
await response.text()
})()).rejects.toThrow()
expect(headersReceived).toBe(receivesHeaders)
expect(server.requests[0]).toMatchObject({
behavior,
chunksSent: behavior === 'partial_disconnect' ? 1 : 0,
outcome: 'reset',
})
})
it('holds a stalled stream until the client aborts and server close remains idempotent', async () => {
const server = await start(['stall'])
const controller = new AbortController()
const response = await chat(server, { signal: controller.signal })
expect(response.status).toBe(200)
expect(server.requests[0]).toMatchObject({ behavior: 'stall', outcome: 'stalled' })
controller.abort()
await expect(response.text()).rejects.toThrow()
await server.close()
await server.close()
})
it.each([
['slow_success', 100] as const,
['stream_disconnect', 100] as const,
['partial_disconnect', 100] as const,
])('records a client that closes during %s', async (behavior, delayMs) => {
const events: MockLlmServerEvent[] = []
const server = await start([behavior], {
chunkDelayMs: delayMs,
disconnectDelayMs: delayMs,
chunkSize: 1,
onEvent: (event) => { events.push(event) },
})
const controller = new AbortController()
const response = await chat(server, { signal: controller.signal })
controller.abort()
await expect(response.text()).rejects.toThrow()
await new Promise((resolve) => { setTimeout(resolve, 5) })
expect(server.requests[0]).toMatchObject({ behavior, outcome: 'client_closed' })
expect(events.filter(event => event.type === 'result')).toEqual([
expect.objectContaining({ behavior, outcome: 'client_closed' }),
])
})
it('preserves UTF-8 code points split across request chunks', async () => {
const server = await start(['success'])
const encoded = Buffer.from(JSON.stringify({ messages: [{ role: 'user', content: '你好' }] }))
const characterOffset = encoded.indexOf(Buffer.from('你'))
expect(characterOffset).toBeGreaterThanOrEqual(0)
await rawChat(server, [
encoded.subarray(0, characterOffset + 1),
encoded.subarray(characterOffset + 1),
])
expect(server.requests[0]?.body).toEqual({ messages: [{ role: 'user', content: '你好' }] })
})
it('formats an IPv6 listener as a valid base URL', async () => {
const server = await start(['success'], { host: '::1' })
expect(server.baseURL).toMatch(/^http:\/\/\[::1\]:\d+$/)
expect((await chat(server)).status).toBe(200)
})
it('emits reasoning, tool calls, max-token finishes, slow chunks, and a wrong content type', async () => {
const server = await start([
'reasoning_success',
'tool_call_success',
'max_tokens',
'slow_success',
'wrong_content_type',
], {
successText: 'answer',
reasoningText: 'think',
toolName: 'lookup',
toolArguments: '{"id":7}',
chunkDelayMs: 1,
chunkSize: 2,
})
const bodies: string[] = []
const contentTypes: Array<string | null> = []
for (let index = 0; index < 5; index += 1) {
const response = await chat(server)
contentTypes.push(response.headers.get('content-type'))
bodies.push(await response.text())
}
expect(bodies[0]).toContain('"reasoning_content":"th"')
expect(bodies[1]).toContain('"name":"lookup"')
expect(bodies[1]).toContain('"arguments":"{\\"id"')
expect(bodies[1]).toContain('"finish_reason":"tool_calls"')
expect(bodies[2]).toContain('"finish_reason":"length"')
expect(bodies[3]).toContain('"finish_reason":"stop"')
expect(contentTypes[4]).toBe('application/json')
expect(server.requests).toHaveLength(5)
expect(server.requests.every(record => record.outcome === 'completed')).toBe(true)
})
it.each([
['rate_limit', 429, 'mock rate limit'] as const,
['server_error', 500, 'mock server error'] as const,
['service_unavailable', 503, 'mock service unavailable'] as const,
['auth_error', 401, 'mock authentication failed'] as const,
['invalid_request', 400, 'mock invalid request'] as const,
['context_overflow', 400, 'context_length_exceeded'] as const,
['quota_exceeded', 429, 'insufficient_quota'] as const,
])('serves %s as a structured HTTP error', async (behavior, status, marker) => {
const server = await start([behavior], { retryAfterMs: 1_001, requestId: 'mock-request-1' })
const response = await chat(server)
const body = await response.text()
expect(response.status).toBe(status)
expect(body).toContain(marker)
expect(response.headers.get('x-request-id')).toBe('mock-request-1')
if (behavior === 'rate_limit') expect(response.headers.get('retry-after')).toBe('2')
else expect(response.headers.get('retry-after')).toBeNull()
expect(server.requests[0]?.outcome).toBe('completed')
})
it('fails loud on script exhaustion and can explicitly repeat the final behavior', async () => {
const exhausted = await start(['success'], { successText: 'once' })
await (await chat(exhausted)).text()
const exhaustedResponse = await chat(exhausted)
expect(exhaustedResponse.status).toBe(500)
expect(await exhaustedResponse.text()).toContain('mock script exhausted')
expect(exhausted.requests.map(record => record.behavior)).toEqual(['success', 'script_exhausted'])
const repeating = await start(['empty'], { repeatLast: true })
await (await chat(repeating)).text()
await (await chat(repeating)).text()
expect(repeating.requests.map(record => record.behavior)).toEqual(['empty', 'empty'])
})
it('selects weighted random behaviors reproducibly and reports the concrete choice', async () => {
const options = {
sequence: ['random'] as const,
repeatLast: true,
randomSeed: 42,
randomWeights: { success: 1, empty: 1 },
successText: 'random success',
}
const first = await startMockLlmServer(options)
const second = await startMockLlmServer(options)
running.push(first, second)
for (let attempt = 0; attempt < 12; attempt += 1) {
await (await chat(first)).text()
await (await chat(second)).text()
}
const firstChoices = first.requests.map(record => record.behavior)
expect(first.randomSeed).toBe(42)
expect(second.randomSeed).toBe(42)
expect(firstChoices).toEqual(second.requests.map(record => record.behavior))
expect(new Set(firstChoices)).toEqual(new Set(['success', 'empty']))
expect(first.requests.every(record => record.scriptBehavior === 'random')).toBe(true)
})
it('rejects invalid method, route, bearer token, and JSON without consuming the script', async () => {
const server = await start(['success'], { apiKey: 'expected' })
const method = await fetch(`${server.baseURL}/v1/chat/completions`)
const route = await fetch(`${server.baseURL}/v1/other`, { method: 'POST', body: '{}' })
const auth = await chat(server, { key: 'wrong' })
const json = await chat(server, { key: 'expected', body: '{' })
expect(method.status).toBe(405)
expect(method.headers.get('allow')).toBe('POST')
expect(route.status).toBe(404)
expect(auth.status).toBe(401)
expect(json.status).toBe(400)
expect(server.requests).toHaveLength(0)
const emptyRequest = await fetch(`${server.baseURL}/v1/chat/completions`, {
method: 'POST',
headers: { authorization: 'Bearer expected' },
})
expect(emptyRequest.status).toBe(200)
expect(server.requests[0]?.behavior).toBe('success')
expect(server.requests[0]?.body).toBeUndefined()
})
})
describe('mock LLM server option validation', () => {
it.each([
[{ sequence: [] }, /sequence/],
[{ sequence: ['success'], host: '' }, /host/],
[{ sequence: ['success'], port: -1 }, /port/],
[{ sequence: ['success'], port: 65_536 }, /port/],
[{ sequence: ['success'], apiKey: '' }, /apiKey/],
[{ sequence: ['success'], successText: '' }, /successText/],
[{ sequence: ['success'], partialText: '' }, /partialText/],
[{ sequence: ['success'], reasoningText: '' }, /reasoningText/],
[{ sequence: ['success'], chunkSize: 0 }, /chunkSize/],
[{ sequence: ['success'], chunkDelayMs: -1 }, /chunkDelayMs/],
[{ sequence: ['success'], disconnectDelayMs: Number.POSITIVE_INFINITY }, /disconnectDelayMs/],
[{ sequence: ['success'], retryAfterMs: 0 }, /retryAfterMs/],
[{ sequence: ['success'], requestId: '' }, /requestId/],
[{ sequence: ['success'], toolName: '' }, /toolName/],
[{ sequence: ['success'], toolArguments: '{' }, /toolArguments/],
[{ sequence: ['random'], randomSeed: -1 }, /randomSeed/],
[{ sequence: ['random'], randomWeights: { random: 1 } }, /unknown concrete behavior/],
[{ sequence: ['random'], randomWeights: { success: -1 } }, /non-negative/],
[{ sequence: ['random'], randomWeights: { success: 0 } }, /positive weight/],
] as const)('rejects invalid options %#', async (options, expected) => {
await expect(startMockLlmServer(options as Parameters<typeof startMockLlmServer>[0]))
.rejects.toThrow(expected)
})
})

15
packages/support/llm-mock-server/tsconfig.json Normal file
View File

@@ -0,0 +1,15 @@
{
"extends": "../../../tsconfig.base.json",
"compilerOptions": {
"rootDir": "src",
"outDir": "lib/types"
},
"include": [
"src"
],
"references": [
{
"path": "../../support/invariants"
}
]
}

17
packages/support/llm-mock-server/tsdown.config.ts Normal file
View File

@@ -0,0 +1,17 @@
import { defineConfig } from 'tsdown'
/** Builds each public entry as a self-contained file admitted by the package whitelist. */
export default defineConfig([
{
entry: ['lib/types/index.js'], outDir: 'lib', format: ['esm'], platform: 'node', target: 'es2024',
fixedExtension: false, outputOptions: { codeSplitting: false }, dts: false, clean: false,
},
{
entry: ['lib/types/invariant.js'], outDir: 'lib', format: ['esm'], platform: 'node', target: 'es2024',
fixedExtension: false, outputOptions: { codeSplitting: false }, dts: false, clean: false,
},
{
entry: ['lib/types/bin.js'], outDir: 'lib', format: ['esm'], platform: 'node', target: 'es2024',
fixedExtension: false, outputOptions: { codeSplitting: false }, dts: false, clean: false,
},
])

11
packages/support/llm-replay/README.md
View File

@@ -1,14 +1,14 @@
# @deepseek-ai/dsh-llm-replay
A replay LLM plugin for keyless snapshot tests. It yields model streams reconstructed from a recorded **session JSONL** fixture, so a test can boot the real agent against a fixed model transcript with no API key. With `providers` configured it registers a replay-only adapter whose catalog is visible to clients such as ACP editors; without `providers` it installs the catch-all `llm/stream` waterfall used by tests that do not need discovery.
A replay LLM plugin for keyless snapshot tests. It yields model streams reconstructed from a recorded **session JSONL** fixture, so a test can boot the real agent against a fixed model transcript with no API key. With `providers` configured it registers a replay-only adapter whose catalog is available to scenarios that exercise model discovery; without `providers` it installs the catch-all `llm/stream` waterfall used by tests that do not need discovery.
Its consumers are the ACP snapshot harness in `examples/acp-agent` and the `stream-json` snapshot in `examples/headless-agent`; each loads this plugin in place of a real LLM adapter. Keeping derivation and replay here places that logic under the per-file 100% coverage gate on `packages/*/src`.
Its consumers are the ACP, headless `stream-json`, and TUI snapshot suites plus the web browser e2e lane. Loader-driven suites mount this plugin in place of a real LLM adapter; the web lane installs it directly to retain the teardown consumption handle. Keeping derivation and replay here places that logic under the per-file 100% coverage gate on `packages/*/src`.
## How the fixture works
The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`", done by the snapshot harness — this plugin does not record. A fixture may carry its `request/header` content tokenized to `{{system}}`/`{{tools}}` (the harness pins that content in one scenario and scrubs the rest); replay is indifferent — derivation reads only `assistant/chunk` events and the line-0 session header.
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`: a `ReplayEntry[]`) that REPLACES the derived script.
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`: a `ReplayEntry[]`) that REPLACES the derived script. A `hang` entry may name `readyFile`; replay writes that empty marker after its prefix chunks reach the loop and before it waits for cancellation, so an external driver can cancel deterministically without observing a presentation update.
## Nested agents: per-session keying
@@ -24,6 +24,7 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | Optional path to a `ReplayEntry[]` sidecar that replaces the PRIMARY session's derived script. |
| `childFiles` | string[] | `$DSH_SNAPSHOT_CHILD_FILES` (path-delimited) | Recorded subagent child-session logs for a nested scenario; empty for a single-session scenario. |
| `providers` | `ReplayProviderConfig[]` | — | Optional replay-only provider and model catalog. Each model may publish `contextWindow`; configured routes dispatch through the replay adapter and never perform provider I/O. |
| `paceMs` | number | — (burst) | Optional per-chunk delay in ms so downstream transports (e.g. the web SSE mux observed by a real browser) see genuinely incremental delivery. A realism knob only — tests must not depend on it for correctness. Non-negative integer; abort during a pace wait cancels the stream promptly. |
```yaml
- id: llm-replay
@@ -43,11 +44,11 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
## Exports
- `installLlmReplay(ctx, config)` — install the configured replay adapter or catch-all `llm/stream` listener; returns the disposer (HMR safety). Use this in tests to drive replay without the Loader or env vars.
- `installLlmReplay(ctx, config)` — install the configured replay adapter or catch-all `llm/stream` listener; returns a `ReplayHandle` (`dispose()` for HMR safety plus `assertConsumed()`, the teardown check that every recorded script bound to a live session and every bound cursor drained — turning a scenario that silently drove fewer model calls than recorded into a crisp diagnostic). Use this in tests to drive replay without the Loader or env vars.
- `loadSessionScripts(config)` — resolve the ordered `SessionScript[]` (primary + children) for a scenario, ready to bind to live sessions in first-call order.
- `loadReplayScript(config)` — resolve the `ReplayEntry[]` for the PRIMARY session only (sidecar override if present, else derived from the JSONL; fail-loud if the fixture is missing).
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)` — the pure helpers that turn a recorded session log into a script and read its header `id`/`createdAt`. A derived group must end in a `finish` chunk; a group without one is the fingerprint of a thrown `stream()` and must instead be expressed via an override sidecar.
- Types `ReplayEntry` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `Config`.
- Types `ReplayEntry` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`.
## Plugin export shape

99
packages/support/llm-replay/src/index.ts
View File

@@ -6,7 +6,7 @@
* @module @deepseek-ai/dsh-llm-replay
*/
import { existsSync, readFileSync } from 'node:fs'
import { existsSync, readFileSync, writeFileSync } from 'node:fs'
import { delimiter as pathDelimiter } from 'node:path'
import type { Context } from 'cordis'
import { decodeStorageRecord } from '@deepseek-ai/dsh-session'
@@ -22,7 +22,11 @@ import { LlmAdapter, LlmError, assertNever } from '@deepseek-ai/dsh-llm'
export type ReplayEntry =
| { kind: 'chunks'; chunks: StreamChunk[] }
| { kind: 'throw'; chunks: StreamChunk[]; message: string; code: string }
| { kind: 'hang' }
| {
kind: 'hang'
/** Optional marker written after the prefix chunks are consumed and before the stream waits for cancellation. */
readyFile?: string
}
/** One model exposed by a replay-only provider catalog. */
export interface ReplayModelConfig {
@@ -42,7 +46,7 @@ export interface ReplayProviderConfig {
id: string
/** Selector label; defaults to {@link id}. */
name?: string
/** Advisory models exposed to clients such as ACP editors. */
/** Advisory models exposed to replay scenarios that exercise discovery. */
models?: ReplayModelConfig[]
}
@@ -74,6 +78,32 @@ export interface ReplayConfig {
* by tests that do not need discovery.
*/
providers?: ReplayProviderConfig[]
/**
* Optional per-chunk pacing delay in milliseconds: each replayed chunk waits
* this long before yielding, so a downstream transport (e.g. the web SSE
* mux observed by a browser) sees genuinely incremental delivery. A realism
* knob only — correctness must never depend on it. Absent or `0` keeps
* today's synchronous burst yield. Must be a non-negative finite integer;
* aborting mid-wait cancels the stream like any other abort.
*/
paceMs?: number
}
/**
* Handle returned by {@link installLlmReplay}: removal plus the end-of-run
* consumption check that turns silent fixture underruns (a scenario that
* issued fewer calls than recorded, or never bound a recorded child script)
* into a crisp diagnostic at teardown.
*/
export interface ReplayHandle {
/** Remove the registered adapter or waterfall listener (HMR safety). Freestanding closure — safe to destructure. */
dispose(this: void): void
/**
* Throw unless every recorded script was bound to a live session and every
* bound cursor consumed its full entry list. Call at scenario teardown.
* Freestanding closure — safe to destructure.
*/
assertConsumed(this: void): void
}
/**
@@ -277,12 +307,32 @@ class ReplayAdapter extends LlmAdapter {
}
}
/**
* Wait `paceMs` between chunk yields, aborting the wait (and the stream) the
* moment the signal fires — a paced replay must cancel as promptly as a burst
* one.
*/
function paceDelay(paceMs: number, signal: AbortSignal | undefined): Promise<void> {
return new Promise<void>((resolve, reject) => {
const timer = setTimeout(() => {
signal?.removeEventListener('abort', onAbort)
resolve()
}, paceMs)
const onAbort = (): void => {
clearTimeout(timer)
reject(new Error('aborted'))
}
signal?.addEventListener('abort', onAbort, { once: true })
})
}
/** Yield a recorded stream back, honoring abort like a real adapter. */
async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined): AsyncIterable<StreamChunk> {
async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined, paceMs: number): AsyncIterable<StreamChunk> {
switch (entry.kind) {
case 'chunks':
for (const chunk of entry.chunks) {
if (signal?.aborted) throw new Error('aborted')
if (paceMs > 0) await paceDelay(paceMs, signal)
yield chunk
}
return
@@ -293,6 +343,7 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined)
// mid-stream STREAM_CLOSED after partial chunks).
for (const chunk of entry.chunks) {
if (signal?.aborted) throw new Error('aborted')
if (paceMs > 0) await paceDelay(paceMs, signal)
yield chunk
}
throw new LlmError(entry.message, entry.code)
@@ -301,6 +352,7 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined)
// chunk, then wait for abort and surface it as the consumer expects.
yield { type: 'block-start', index: 0, blockType: 'text' }
yield { type: 'text-delta', index: 0, text: 'partial' }
if (entry.readyFile !== undefined) writeFileSync(entry.readyFile, '')
await new Promise<void>((_resolve, reject) => {
if (signal?.aborted) { reject(new Error('aborted')); return }
signal?.addEventListener('abort', () => { reject(new Error('aborted')) }, { once: true })
@@ -319,14 +371,17 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined)
* next ordered recorded script, then advances its own cursor synchronously at
* invocation time; calls without `sessionId` share one anonymous session. A
* non-empty provider catalog registers a routed replay adapter; otherwise a
* catch-all waterfall intercepts requests. Returns the effect disposer for
* HMR-safe removal.
* catch-all waterfall intercepts requests.
*
* @param ctx - the context whose LLM service receives the replay route or waterfall.
* @param config - the resolved fixture paths (env-var defaulting is `apply`'s job).
* @returns the disposer that removes the registered adapter or listener.
* @returns the {@link ReplayHandle} carrying the disposer and the teardown consumption check.
*/
export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void {
export function installLlmReplay(ctx: Context, config: ReplayConfig): ReplayHandle {
const paceMs = config.paceMs ?? 0
if (!Number.isInteger(paceMs) || paceMs < 0) {
throw new Error(`llm-replay: paceMs must be a non-negative integer, got ${String(config.paceMs)}`)
}
const scripts = loadSessionScripts(config)
// Live-session → its bound script + cursor. A new live session id claims the
// next not-yet-bound script (scripts are in bind order); `nextScript` is the
@@ -370,14 +425,31 @@ export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void
+ `but its script has only ${boundState.entries.length}; re-record the scenario`,
)
}
yield* replayEntry(entry, options.signal)
yield* replayEntry(entry, options.signal, paceMs)
})()
}
const providers = config.providers ?? []
if (providers.length > 0) {
return ctx.llm.registerAdapter(providers.map(provider => provider.id), new ReplayAdapter(providers, replay))
const dispose = providers.length > 0
? ctx.llm.registerAdapter(providers.map(provider => provider.id), new ReplayAdapter(providers, replay))
: ctx.on('llm/stream', (options: GenerateOptions, _next) => replay(options))
return {
dispose,
assertConsumed(): void {
const problems: string[] = []
if (nextScript < scripts.length) {
problems.push(`${scripts.length - nextScript} recorded script(s) never bound to a live session`)
}
for (const [key, state] of bound) {
if (state.cursor < state.entries.length) {
const who = key === ANON ? 'the anonymous session' : `session ${key}`
problems.push(`${who} consumed ${state.cursor}/${state.entries.length} recorded call(s)`)
}
}
if (problems.length > 0) {
throw new Error(`llm-replay: fixture not fully consumed — ${problems.join('; ')}; the scenario drove fewer model calls than recorded`)
}
},
}
return ctx.on('llm/stream', (options: GenerateOptions, _next) => replay(options))
}
export const name = 'llm-replay'
@@ -397,6 +469,8 @@ export interface Config {
childFiles?: string[]
/** Optional replay-only provider catalog; absent or empty selects catch-all waterfall replay. */
providers?: ReplayProviderConfig[]
/** Optional per-chunk pacing delay in ms (see {@link ReplayConfig.paceMs}); absent keeps burst yield. */
paceMs?: number
}
export function apply(ctx: Context, config: Config = {}): void {
@@ -413,5 +487,6 @@ export function apply(ctx: Context, config: Config = {}): void {
...overrideFile !== undefined && overrideFile.length > 0 ? { overrideFile } : {},
...childFiles.length > 0 ? { childFiles } : {},
...config.providers !== undefined ? { providers: config.providers } : {},
...config.paceMs !== undefined ? { paceMs: config.paceMs } : {},
})
}

96
packages/support/llm-replay/tests/llm-replay.spec.ts
View File

@@ -1,4 +1,4 @@
import { mkdtempSync, rmSync, writeFileSync } from 'node:fs'
import { existsSync, mkdtempSync, rmSync, writeFileSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
@@ -234,7 +234,7 @@ describe('installLlmReplay (through the real LlmService)', () => {
writeLog(TEXT_CHUNKS)
const ctx = new Context()
await ctx.plugin(LlmService)
const dispose = installLlmReplay(ctx, {
const { dispose } = installLlmReplay(ctx, {
file,
providers: [
{
@@ -379,7 +379,8 @@ describe('installLlmReplay (through the real LlmService)', () => {
it('rejects a hang entry when the signal fires DURING the wait (abort listener path)', async () => {
writeFileSync(file, sessionJsonl([]), 'utf8')
const overrideFile = join(dir, 'replay.override.json')
writeFileSync(overrideFile, JSON.stringify([{ kind: 'hang' }]), 'utf8')
const readyFile = join(dir, 'stream-ready')
writeFileSync(overrideFile, JSON.stringify([{ kind: 'hang', readyFile }]), 'utf8')
const ctx = new Context()
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file, overrideFile })
@@ -392,6 +393,7 @@ describe('installLlmReplay (through the real LlmService)', () => {
expect((await iterator.next()).value).toMatchObject({ type: 'text-delta' })
const pending = iterator.next()
await new Promise(r => setImmediate(r))
expect(existsSync(readyFile)).toBe(true)
controller.abort()
await expect(pending).rejects.toThrow('aborted')
})
@@ -429,6 +431,92 @@ describe('installLlmReplay (through the real LlmService)', () => {
await iterator.next()
await expect(iterator.next()).rejects.toThrow('aborted')
})
it('rejects a paceMs that is not a non-negative integer', async () => {
writeLog(TEXT_CHUNKS)
const ctx = new Context()
await ctx.plugin(LlmService)
expect(() => installLlmReplay(ctx, { file, paceMs: -1 })).toThrow(/paceMs/)
expect(() => installLlmReplay(ctx, { file, paceMs: 1.5 })).toThrow(/paceMs/)
})
it('paces chunk yields when paceMs is set (each chunk waits at least the pace)', async () => {
writeLog(TEXT_CHUNKS)
const ctx = new Context()
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file, paceMs: 10 })
const started = performance.now()
const chunks = await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))
expect(chunks).toEqual(TEXT_CHUNKS)
// N chunks × 10ms; allow generous scheduling slack, assert the floor only.
expect(performance.now() - started).toBeGreaterThanOrEqual(TEXT_CHUNKS.length * 10 - 5)
})
it('aborting DURING a pace wait cancels the stream promptly', async () => {
writeLog(TEXT_CHUNKS)
const ctx = new Context()
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file, paceMs: 60_000 })
const controller = new AbortController()
const pending = drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal }))
// Let the generator park inside the pace timer, then abort — the reject
// must come from the abort listener, not the (distant) timer.
await new Promise(r => setImmediate(r))
controller.abort()
await expect(pending).rejects.toThrow('aborted')
})
it('assertConsumed passes only after every recorded call replayed', async () => {
writeLog(TEXT_CHUNKS, TEXT_CHUNKS)
const ctx = new Context()
await ctx.plugin(LlmService)
const handle = installLlmReplay(ctx, { file })
await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))
// One of two recorded calls consumed — the underrun must name the gap.
expect(() => { handle.assertConsumed() }).toThrow(/consumed 1\/2 recorded call/)
await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))
expect(() => { handle.assertConsumed() }).not.toThrow()
})
it('paces a throw-entry prefix too (the recorded partial streams at the same cadence)', async () => {
writeFileSync(file, sessionJsonl([]), 'utf8')
const overrideFile = join(dir, 'replay.override.json')
const partial: StreamChunk[] = [{ type: 'block-start', index: 0, blockType: 'text' }]
writeFileSync(overrideFile, JSON.stringify([
{ kind: 'throw', chunks: partial, message: 'boom', code: 'STREAM_CLOSED' },
]), 'utf8')
const ctx = new Context()
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file, overrideFile, paceMs: 10 })
const started = performance.now()
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).rejects.toThrow('boom')
expect(performance.now() - started).toBeGreaterThanOrEqual(5)
})
it('assertConsumed names an underrunning identified session by its id', async () => {
writeLog(TEXT_CHUNKS, TEXT_CHUNKS)
const ctx = new Context()
await ctx.plugin(LlmService)
const handle = installLlmReplay(ctx, { file })
const sessionId = 'live-underrun' as NonNullable<GenerateOptions['sessionId']>
await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [], sessionId }))
expect(() => { handle.assertConsumed() }).toThrow(/session live-underrun consumed 1\/2/)
})
it('assertConsumed reports recorded scripts no live session ever bound', async () => {
writeLog(TEXT_CHUNKS)
const childFile = join(dir, 'session.1.jsonl')
writeFileSync(childFile, sessionJsonl(
TEXT_CHUNKS.map((chunk, i) => chunkEvent(i + 1, 1, 1, chunk)),
{ id: 'child', createdAt: 10 },
), 'utf8')
const ctx = new Context()
await ctx.plugin(LlmService)
const handle = installLlmReplay(ctx, { file, childFiles: [childFile] })
await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [], sessionId: 'live-parent' as NonNullable<GenerateOptions['sessionId']> }))
// The child script never bound: the scenario drove fewer sessions than recorded.
expect(() => { handle.assertConsumed() }).toThrow(/1 recorded script\(s\) never bound/)
})
})
describe('parseSessionHeader', () => {
@@ -629,7 +717,7 @@ describe('apply (the plugin entry)', () => {
writeFileSync(file, sessionJsonl(TEXT_CHUNKS.map((c, i) => chunkEvent(i + 1, 1, 1, c))), 'utf8')
const ctx = new Context()
await ctx.plugin(LlmService)
apply(ctx, { file, providers: [{ id: 'm', models: [{ id: 'm' }] }] })
apply(ctx, { file, providers: [{ id: 'm', models: [{ id: 'm' }] }], paceMs: 1 })
expect(ctx.llm.listProviders()).toEqual([{ id: 'm', name: 'm' }])
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
})

6
packages/support/loader-smoke/tests/example-launch.spec.ts
View File

@@ -5,7 +5,7 @@ import {
resolveExampleMode,
} from '@deepseek-ai/dsh-loader-smoke'
const SRC_BIN = '/repo/packages/examples/tui-demo/src/bin.ts'
const SRC_BIN = '/repo/packages/examples/cli-demo/src/bin.ts'
const TSCONFIG = '/repo/tsconfig.json'
const originalMode = process.env[EXAMPLE_MODE_ENV]
@@ -65,7 +65,7 @@ describe('resolveExampleLaunch', () => {
env: { DSH_HOME: '/tmp/home' },
})
expect(args).not.toContain('--import')
expect(args).toContain('/repo/packages/examples/tui-demo/lib/bin.js')
expect(args).toContain('/repo/packages/examples/cli-demo/lib/bin.js')
expect(args.slice(-2)).toEqual(['--config', './cordis.yml'])
expect(env.TSX_TSCONFIG_PATH).toBeUndefined()
expect(env.DSH_HOME).toBe('/tmp/home')
@@ -100,6 +100,6 @@ describe('resolveExampleLaunch', () => {
it('defaults the mode from the environment', () => {
process.env[EXAMPLE_MODE_ENV] = 'lib'
const { args } = resolveExampleLaunch({ srcBin: SRC_BIN })
expect(args).toContain('/repo/packages/examples/tui-demo/lib/bin.js')
expect(args).toContain('/repo/packages/examples/cli-demo/lib/bin.js')
})
})