Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Merge latest origin/master into compact-post-step-overflow-recovery

Browse Source 
# Conflicts:
#	docs/agent-lifecycle.md
#	docs/architecture.md
#	docs/cookbook/extension-cookbook.i18n.yaml
#	docs/cordis-catalog/events.md
#	docs/cordis-catalog/services.md
#	docs/event-producer-consumer.md
#	docs/rfc/INDEX.md
#	docs/rfc/implemented/architecture/2026-07-05-reconstructable-requests.md
#	docs/rfc/implemented/architecture/2026-07-15-replay-token-meter-service.i18n.yaml
#	docs/rfc/implemented/architecture/2026-07-15-replay-token-meter-service.md
#	docs/rfc/implemented/architecture/2026-07-15-replay-token-meter-service.zh.md
#	docs/rfc/implemented/feature/2026-06-18-compaction-capability-seam.md
#	docs/rfc/implemented/feature/2026-07-07-session-prefix.md
#	packages/compact/compact-basic/README.md
#	packages/compact/compact-basic/src/config.ts
#	packages/compact/compact-basic/src/index.ts
#	packages/compact/compact-basic/src/summarizer.ts
#	packages/compact/compact-basic/tests/compact-basic.spec.ts
#	packages/compact/compact-basic/tests/compact-loop-repro.spec.ts
#	packages/compact/compact/README.md
#	packages/cordis/tool-cordis/src/api-catalog.ts
#	packages/core/agent-loop/src/loop.ts
#	packages/core/agent-loop/tests/cancel.spec.ts
#	packages/llm/llm-deepseek/src/adapter.ts
#	packages/llm/llm-pi-ai/README.md
#	packages/llm/llm-pi-ai/src/stream.ts
#	packages/llm/llm-pi-ai/tests/convert.spec.ts
#	packages/llm/llm/README.md
#	packages/llm/llm/src/index.ts
#	packages/llm/llm/tests/service.spec.ts
#	scripts/gen-doc-graphs.ts
This commit is contained in:
Tianyi Cui
2026-07-19 12:06:23 +08:00
parent 231aeabe55 383a305bd4
commit 7be78c7d4c
814 changed files with 42348 additions and 10067 deletions
Download Patch File Download Diff File Expand all files Collapse all files

5
packages/support/README.md
View File

@@ -4,10 +4,11 @@ Packages that exist to serve development, testing, and the examples rather than
| Package | Role | ctx key |
|---|---|---|
| `acp-snapshot/` | ACP snapshot suite kit: subprocess scenario harness + golden normalizers + the `defineAcpSnapshotSuite` factory | (library — imported by example `*.snapshot.ts` suites) |
| `acp-snapshot/` | ACP test kit: shared subprocess/client launcher + snapshot harness, normalizers, and suite factory | (library — imported by ACP e2e and `*.snapshot.ts` suites) |
| `agent-loop-testkit/` | Shared prerequisite mounting for tests that exercise the concrete agent loop | (library — imported by AgentLoop integration tests) |
| `invariants/` | Runtime event-contract assertions for development diagnostics | (listens on `session/*`, `agent/*`) |
| `loader-smoke/` | Shared real-Loader subprocess harness for keyless example smokes | (library — imported by example e2e suites) |
| `llm-replay/` | Record/replay adapter: short-circuits `llm/stream` from a recorded session JSONL (keyless snapshot tests) | (listens on `llm/stream`) |
| `subagent-mock/` | Scripted `SubagentProvider` for deterministic seam/tool tests | (registers on `ctx.subagents`) |
`invariants` is development support but has no environment guard: it runs wherever registered, and the default `dsh-agent-spine-demo` bundle mounts it unconditionally. `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate. `acp-snapshot` carries the snapshot tier's harness/normalizer/suite machinery, while `loader-smoke` owns the parallel stdio/Loader process boundary used by keyless example e2e suites. `subagent-mock` exercises the real `ctx.subagents` load path without a model or child agent. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.
`invariants` is development support but has no environment guard: it runs wherever registered, and the default `dsh-agent-spine-demo` bundle mounts it unconditionally. `agent-loop-testkit` centralizes the mandatory service spine for hand-built AgentLoop tests without owning their loop or scenario. `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate. `acp-snapshot` carries the ACP subprocess/client boundary plus the snapshot harness, normalizers, and suite machinery, while `loader-smoke` owns the parallel stdio/Loader process boundary used by keyless example e2e suites. `subagent-mock` exercises the real `ctx.subagents` load path without a model or child agent. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.

13
packages/support/acp-snapshot/README.md
View File

@@ -2,11 +2,12 @@
The ACP snapshot suite kit: the shared machinery behind the keyless snapshot tier (`pnpm run test:snapshot`, [testing policy](../../../docs/testing.md)). An example gets a full snapshot suite from a scenario table plus a fixtures directory; every compare/guard mechanic lives here, under the per-file coverage gate, instead of being copied per example.
Three layers, importable separately:
Four layers, importable separately:
- **`runScenario` (harness)** — boots the real agent bin as a subprocess via tsx (unbuilt, Loader path), drives it over ACP JSON-RPC stdio from a deterministic `input.json` script, tees raw stdout for the golden + purity check, and harvests every persisted session JSONL (parent + subagent children, primary-first) after a graceful stdin-EOF shutdown. Parameterized by `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath` — absolute paths; the subprocess cwd is a temp dir outside the repo).
- **`launchAcpTestAgent` (launcher)** — boots an unbuilt ACP agent from a temp cwd, pins tsx to the repo tsconfig, connects the SDK client over a raw-byte stdout tee, collects session updates and stderr, surfaces asynchronous spawn failures through its startup lifecycle, fails closed on unhandled permission requests, and owns graceful or signalled shutdown. Shutdown waits for process exit, inherited stdio closure, and ACP parser exhaustion before resolving or propagating a child error, so captures are complete and callers can remove owned paths after either outcome. Snapshot and ordinary e2e suites share this process boundary; a test supplies only agent paths, cwd, environment overrides, and any permission policy.
- **`runScenario` (harness)** — boots the real agent bin as a subprocess via tsx (unbuilt, Loader path), drives it over ACP JSON-RPC stdio from a deterministic `input.json` script, tees raw stdout for the golden + purity check, and harvests every persisted session JSONL (parent + subagent children, primary-first) after a graceful stdin-EOF shutdown. Parameterized by `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath` — absolute paths; the subprocess cwd is a temp dir outside the repo). Startup failures preserve captured agent stderr in the rejected diagnostic.
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header RFC](../../../docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, the per-header-class pin (`system-prompt.golden.md` plus `tool-schemas.golden.json`) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt/schema-scrubbed, non-pinning fixtures fully header-scrubbed). Must be called at vitest collection time.
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, the per-header-class pin (`system-prompt.golden.md` plus `tool-schemas.golden.json`) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt/schema-scrubbed, non-pinning fixtures fully header-scrubbed). Each scenario directory's `session.jsonl` plus contiguous `session.<n>.jsonl` siblings are the ordered primary/child inventory; the scenario table does not duplicate their count. Must be called at vitest collection time.
A consuming `*.snapshot.ts` is the scenario table plus one factory call:
@@ -35,11 +36,11 @@ defineAcpSnapshotSuite({
})
```
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode and filesystem scenarios are templates. Each pinning directory stores the normalized composed prompt in generated `system-prompt.golden.md` and the initial schemas plus schema deltas in generated `tool-schemas.golden.json`; `session.jsonl` stores `"system":"{{system}}","tools":"{{tools}}"` while retaining config, reason, and any model-visible prefix.
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode and filesystem scenarios are templates. Each pinning directory stores the normalized full prompt sequence in generated `system-prompt.golden.md` and the corresponding full tool-schema sequence in generated `tool-schemas.golden.json`; `session.jsonl` stores `"system":"{{system}}","tools":"{{tools}}"` while retaining config, reason, and any model-visible prefix. A pin with legitimate mid-run header changes declares `expectedHeaderChanges`, which fixes the length of both sidecar sequences.
Examples use a `cordis.snapshot.yml` overlay with [`dsh-llm-replay`](../llm-replay/README.md). Recording calls the live model and updates model fixtures; keyless refresh replays those fixtures and updates derived stdout, session-log, prompt, and tool-schema snapshots. See the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config RFC](../../../docs/rfc/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log goldens, and each pin's prompt and tool-schema sidecars from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
`suite.ts` imports Vitest, so use this package only inside a Vitest run. The ACP-specific script queues permission answers by stable option kind and maps them to current option ids; a missing answer cancels, while an unavailable kind fails the scenario after cancelling the agent request. It can also set session config options or assert that unknown ids and values are rejected in the transcript.
Constraints: `suite.ts` imports vitest, so the package entry is importable only inside a vitest run (the launcher, harness, and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug). Session config options are scriptable too: the `setConfigOption` step switches a knob over `session/set_config_option`, and `setConfigOptionExpectError` asserts the bridge rejects an unknown id or out-of-vocabulary value (the error frame stays in the transcript).
## Model Experience

4
packages/support/acp-snapshot/package.json
View File

@@ -1,6 +1,6 @@
{
"name": "@deepseek-ai/dsh-acp-snapshot",
"description": "ACP snapshot suite kit: real-subprocess scenario harness, golden normalizers, and the suite factory behind the keyless snapshot tier",
"description": "ACP test kit: shared subprocess launcher, snapshot scenario harness, golden normalizers, and suite factory",
"version": "0.0.1",
"private": true,
"type": "module",
@@ -23,7 +23,7 @@
"license": "BSD-3-Clause",
"dependencies": {
"@agentclientprotocol/sdk": "0.25.1",
"tsx": "^4.22.4",
"@deepseek-ai/dsh-loader-smoke": "workspace:*",
"vitest": "^4.1.8"
},
"peerDependencies": {

274
packages/support/acp-snapshot/src/harness.ts
View File

@@ -1,65 +1,47 @@
/**
* Shared ACP snapshot subprocess harness. It boots the real agent bin through the Cordis
* loader, drives deterministic ACP JSON-RPC over stdio, captures protocol-pure stdout, and
* harvests persisted session logs after graceful shutdown. Normalization stays in
* `normalize.ts`; suite registration stays in `suite.ts`.
* Shared subprocess harness for ACP snapshot suites. A library module driven by
* the suite factory in ./suite.ts (and directly by harness-level specs); each
* example's `*.snapshot.ts` names its own agent-under-test paths.
*
* It boots the REAL agent bin subprocess via the cordis Loader (so the
* export-shape bug class stays guarded — see docs/postmortem/0001), drives it
* over real ACP JSON-RPC stdio with a deterministic input script, tees raw
* stdout (for the golden + a purity check) into an SDK `ClientSideConnection`,
* and — in record mode — harvests the persisted session JSONL after a graceful
* shutdown flush. The pure normalizers in ./normalize.ts turn the captured
* stdout frames and the session-log events into stable, snapshot-able text.
*
* See docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
*
* @module @deepseek-ai/dsh-acp-snapshot/harness
*/
import { spawn, type ChildProcessWithoutNullStreams } from 'node:child_process'
import { cp, mkdtemp, readFile, readdir, rm } from 'node:fs/promises'
import { existsSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join, delimiter } from 'node:path'
import { fileURLToPath } from 'node:url'
import { Readable, Writable } from 'node:stream'
import {
ClientSideConnection,
ndJsonStream,
PROTOCOL_VERSION,
type Agent as AcpAgent,
type Client,
type RequestPermissionRequest,
type RequestPermissionResponse,
type SessionNotification,
} from '@agentclientprotocol/sdk'
import { launchAcpTestAgent, type AgentUnderTest, type LaunchedAcpTestAgent } from './launcher.ts'
// Resolve tsx's ESM loader to an ABSOLUTE path once: the child runs with its
// cwd in a temp dir OUTSIDE the repo, where a bare `--import tsx` would not
// resolve from node_modules. import.meta.resolve gives this package's tsx
// regardless of the child cwd.
const tsxLoader = fileURLToPath(import.meta.resolve('tsx'))
export type { AgentUnderTest } from './launcher.ts'
/**
* The agent composition a scenario runs against: which bin to boot and which
* leaf config it loads. All paths are ABSOLUTE — the subprocess cwd is a temp
* dir outside the repo, so relative resolution would miss; a suite resolves
* them from its own `import.meta.url`.
*/
export interface AgentUnderTest {
/** The agent bin entry (e.g. `packages/examples/acp-demo/src/bin.ts`), run unbuilt via tsx. */
binScript: string
/**
* The example's live `cordis.yml`. Under `DSH_SNAPSHOT=replay` the bin swaps
* it for the sibling `cordis.snapshot.yml` (the keyless replay overlay), so
* one path serves both modes.
*/
configPath: string
/**
* The repo-root tsconfig whose `paths` map resolves the unbuilt workspace
* imports. Passed to the child as `TSX_TSCONFIG_PATH`: tsx finds a tsconfig
* by searching UP from the child's cwd — a temp dir outside the repo — so
* without the explicit pin the dsh-* imports fail before the bin writes a
* byte.
*/
tsconfigPath: string
}
/**
* One step of a scenario's deterministic input script (`input.json`). The harness interprets
* these in order. `newSession` captures the server-issued (random) session id into a
* `{{sessionId}}` variable that later steps reference. `promptAndCancel` sends without awaiting,
* waits for the first streamed message, then cancels, making transcript order deterministic.
* One step of a scenario's deterministic input script (`input.json`). The
* harness interprets these in order. `newSession` captures the server-issued
* (random) session id into a `{{sessionId}}` variable that later steps
* reference, since a committed file cannot know the id in advance.
*
* `promptAndCancel` sends a prompt WITHOUT awaiting its response, waits until
* the client observes the first streamed `agent_message_chunk` (so the emitted
* frames deterministically precede the cancellation), then cancels the turn —
* the only way to exercise a cancel deterministically (a plain `prompt` step
* awaits the response, which a cancel/hang scenario would block on forever).
*/
export type InputStep =
| { op: 'initialize'; terminalOutput?: boolean }
@@ -76,9 +58,16 @@ export type InputStep =
export interface InputScript {
steps: InputStep[]
/**
* FIFO permission answers selected by stable option kind; the harness maps each kind to the
* agent-issued option id. Exhaustion cancels, while a kind the agent did not offer fails the
* scenario.
* Ordered answers for the agent's `session/request_permission` round-trips,
* consumed FIFO — the Nth request gets the Nth answer. Each answer selects
* by option KIND: option ids are agent-issued randoms a committed script
* cannot know, while kinds are the ACP-stable vocabulary, so the client maps
* kind → the offered `optionId` at answer time. A request beyond the queue
* (or with no queue at all) is answered `cancelled` — the stub behavior a
* scenario without approvals relies on. A scripted kind the request does
* not offer REJECTS the run: the scenario scripted an impossible click,
* and {@link runScenario} throws once the in-flight step settles (the
* agent itself just sees `cancelled`, so it cannot absorb the bug).
*/
permissionAnswers?: PermissionAnswer[]
}
@@ -168,25 +157,26 @@ export interface RunOptions {
export async function runScenario(input: InputScript, opts: RunOptions): Promise<RunResult> {
const cwd = await mkdtemp(join(tmpdir(), 'acp-snap-cwd-'))
const sessionsRoot = await mkdtemp(join(tmpdir(), 'acp-snap-sessions-'))
// Everything past the temp-dir creation runs under a try/finally that always
// removes both dirs — so a failure in workspace seeding, spawn, or any step
// never leaks them (the "e2e tests own their resources" rule).
let child: ChildProcessWithoutNullStreams | undefined
// Fixed path length: spill-policy budgets the preview against the REAL path
// before stdout normalization, so tmpdir() length differences churn goldens.
const spillRoot = '/tmp/dsh-acp-snapshot-spill'
// Everything past the temp-dir creation is followed by failure-safe cleanup,
// so a failure in workspace seeding, spawn, or any step never leaks resources.
let launched: LaunchedAcpTestAgent | undefined
let sessionId: string | undefined
let sessionLogs: HarvestedLog[] = []
const rawBuffers: Buffer[] = []
const stderrChunks: string[] = []
try {
const outcome = await (async (): Promise<RunResult> => {
// Seed the workspace if the scenario ships one (a file the agent reads/edits).
// Copied into the temp cwd so the agent's bash tools see it; the goldens
// normalize the cwd, so the seeded paths stay stable across runs.
if (opts.workspaceDir !== undefined && existsSync(opts.workspaceDir)) {
await cp(opts.workspaceDir, cwd, { recursive: true })
}
const env: NodeJS.ProcessEnv = {
...process.env,
TSX_TSCONFIG_PATH: opts.agent.tsconfigPath,
DSH_SNAPSHOT: opts.mode,
DSH_SNAPSHOT_FILE: opts.fixtureFile,
DSH_SNAPSHOT_SESSIONS_ROOT: sessionsRoot,
DSH_SNAPSHOT_SPILL_ROOT: spillRoot,
DSH_HOME: join(cwd, '.dsh'),
DSH_AGENTS_HOME: join(cwd, '.agents'),
...opts.overrideFile !== undefined ? { DSH_SNAPSHOT_OVERRIDE: opts.overrideFile } : {},
@@ -195,58 +185,22 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
: {},
}
child = spawn(
process.execPath,
['--import', tsxLoader, opts.agent.binScript, '--config', opts.configPath ?? opts.agent.configPath],
{ cwd, env, stdio: ['pipe', 'pipe', 'pipe'] },
)
child.stderr.setEncoding('utf8')
child.stderr.on('data', (c: string) => stderrChunks.push(c))
// Tee the same raw bytes to the golden and SDK client. Decode once at the end so a UTF-8
// sequence split across stream chunks cannot corrupt the transcript.
const passthrough = new Readable({ read() {} })
child.stdout.on('data', (buf: Buffer) => {
rawBuffers.push(buf)
passthrough.push(buf)
})
child.stdout.on('end', () => passthrough.push(null))
const stream = ndJsonStream(
Writable.toWeb(child.stdin) as WritableStream<Uint8Array>,
Readable.toWeb(passthrough) as ReadableStream<Uint8Array>,
)
// Watcher so a step can block until the client OBSERVES a particular
// session/update — used by promptAndCancel to pin frame order (send cancel
// only after the streamed agent_message_chunk has arrived, so those frames
// deterministically precede the cancelled prompt response).
const updateWaiters: { match: (u: SessionNotification['update']) => boolean; resolve: () => void }[] = []
const waitForUpdate = (match: (u: SessionNotification['update']) => boolean): Promise<void> =>
new Promise<void>(resolve => updateWaiters.push({ match, resolve }))
// Permission answers are consumed FIFO across the whole run; exhaustion
// falls back to `cancelled` so approval-free scenarios keep the plain stub.
const permissionQueue = [...input.permissionAnswers ?? []]
// A callback throw would become only an RPC error the agent could absorb. Record an
// impossible permission choice here, answer cancelled, and fail the outer scenario.
// A scenario bug detected inside a client callback (a scripted permission
// kind the agent never offered). It cannot fail the run from in there: a
// callback throw only becomes a JSON-RPC error RESPONSE to the agent, and
// a tolerant agent treats that as a denial and carries on — the run (or
// worse, a record) would absorb the impossible click silently. So the
// callback answers `cancelled` (a well-defined path for the agent),
// captures the error here, and the step loop fails the run on it.
let scriptError: Error | undefined
const makeClient = (_agent: AcpAgent): Client => ({
sessionUpdate(params: SessionNotification): Promise<void> {
for (let i = updateWaiters.length - 1; i >= 0; i--) {
const waiter = updateWaiters[i]
// The index is always in-bounds (i only decreases; splice removes at
// i, so lower entries stay valid); the guard satisfies
// noUncheckedIndexedAccess.
/* v8 ignore next 1 -- unreachable in-bounds guard, see above */
if (waiter === undefined) continue
if (waiter.match(params.update)) {
updateWaiters.splice(i, 1)
waiter.resolve()
}
}
return Promise.resolve()
},
launched = launchAcpTestAgent({
agent: opts.agent,
cwd,
...opts.configPath !== undefined ? { configPath: opts.configPath } : {},
env,
requestPermission(params: RequestPermissionRequest): Promise<RequestPermissionResponse> {
const answer = permissionQueue.shift()
if (answer === undefined) return Promise.resolve({ outcome: { outcome: 'cancelled' } })
@@ -264,10 +218,12 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
return Promise.resolve({ outcome: { outcome: 'selected', optionId: option.optionId } })
},
})
const client = new ClientSideConnection(makeClient, stream)
const active = launched
await active.spawned
const { client } = active
for (const step of input.steps) {
await runStep(client, step, cwd, waitForUpdate, () => sessionId, (id) => { sessionId = id })
await runStep(client, step, cwd, match => active.waitForUpdate(match), () => sessionId, (id) => { sessionId = id })
// A permission exchange happens while a step's request is in flight, so
// by the time the step settles any script bug it exposed is captured —
// fail the run HERE, as a harness error, rather than hoping the agent's
@@ -276,30 +232,57 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
}
// Done driving: close stdin so the server disposes gracefully (flushing
// persistence) and exits. Then await exit so the harvested log is complete.
child.stdin.end()
await waitForExit(child)
await active.close()
// Harvest EVERY persisted log (parent + any subagent children) while the
// temp dirs still exist, ordered primary-first.
sessionLogs = await harvestSessionLogs(sessionsRoot)
} finally {
// Failure-safe teardown: kill a still-running child and drop the temp dirs
// even if seeding/spawn/a step/harvest threw, so a flaky run never leaks a
// process or dir. `child` is undefined only if spawn itself threw.
if (child !== undefined && child.exitCode === null && child.signalCode === null) {
child.kill('SIGKILL')
await waitForExit(child)
return {
rawStdout: launched.rawStdout(),
stderr: launched.stderr(),
cwd,
...sessionId !== undefined ? { sessionId } : {},
sessionLogs,
}
await rm(cwd, { recursive: true, force: true })
await rm(sessionsRoot, { recursive: true, force: true })
}
})().then(
value => ({ status: 'fulfilled', value } as const),
(error: unknown) => {
const stderr = launched?.stderr() ?? ''
return {
status: 'rejected',
error: stderr === ''
? error
: new Error(`snapshot-harness: scenario failed: ${String(error)}\nagent stderr:\n${stderr}`, { cause: error }),
} as const
},
)
return {
rawStdout: Buffer.concat(rawBuffers).toString('utf8'),
stderr: stderrChunks.join(''),
cwd,
...sessionId !== undefined ? { sessionId } : {},
sessionLogs,
// Failure-safe teardown: wait for a still-running child, then attempt every
// owned-path removal even when an earlier cleanup rejects. Report every
// teardown failure alongside a scenario failure so neither orthogonal
// outcome hides the other.
const cleanupResults: PromiseSettledResult<unknown>[] = []
const cleanup = async (action: () => Promise<unknown>): Promise<void> => {
cleanupResults.push(...await Promise.allSettled([action()]))
}
/* v8 ignore next 1 -- launch itself can only throw on a defensive synchronous spawn API failure */
await cleanup(() => launched?.close('SIGKILL') ?? Promise.resolve())
await cleanup(() => rm(cwd, { recursive: true, force: true }))
await cleanup(() => rm(sessionsRoot, { recursive: true, force: true }))
await cleanup(() => rm(spillRoot, { recursive: true, force: true }))
const cleanupFailures = cleanupResults
.filter((result): result is PromiseRejectedResult => result.status === 'rejected')
.map(result => result.reason as unknown)
if (cleanupFailures.length > 0) {
throw new AggregateError(
outcome.status === 'rejected' ? [outcome.error, ...cleanupFailures] : cleanupFailures,
outcome.status === 'rejected'
? 'snapshot scenario and cleanup failed'
: 'snapshot cleanup failed',
)
}
if (outcome.status === 'rejected') throw outcome.error
return outcome.value
}
/** Drive one input step over the client connection. */
@@ -307,7 +290,7 @@ async function runStep(
client: ClientSideConnection,
step: InputStep,
cwd: string,
waitForUpdate: (match: (u: SessionNotification['update']) => boolean) => Promise<void>,
waitForUpdate: (match: (u: SessionNotification['update']) => boolean) => Promise<SessionNotification['update']>,
getSessionId: () => string | undefined,
setSessionId: (id: string) => void,
): Promise<void> {
@@ -324,8 +307,10 @@ async function runStep(
return
}
case 'newSessionExpectError': {
// The bridge rejects a session/new that widens the workspace scope (non-empty
// additionalDirectories / mcpServers — unimplemented).
// The bridge rejects a session/new that widens the workspace scope
// (non-empty additionalDirectories / mcpServers — unimplemented). The SDK
// surfaces that as a rejected RPC; swallow it so the run completes and the
// error frame is captured in the transcript.
await client.newSession({
cwd,
mcpServers: [],
@@ -345,8 +330,10 @@ async function runStep(
case 'promptExpectError': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: promptExpectError before newSession')
// The model fails this turn (a recorded provider error), so the bridge answers the prompt
// with a JSON-RPC error and the SDK rejects.
// The model fails this turn (a recorded provider error), so the bridge
// answers the prompt with a JSON-RPC error and the SDK rejects. That
// rejection IS the expected editor experience — swallow it so the run
// completes and the stdout transcript (the error frame) is captured.
await client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
.then(() => { throw new Error('snapshot-harness: expected the prompt to fail but it succeeded') },
() => { /* expected: the turn failed and the bridge returned an error */ })
@@ -355,8 +342,13 @@ async function runStep(
case 'promptAndCancel': {
const sessionId = getSessionId()
if (sessionId === undefined) throw new Error('snapshot-harness: promptAndCancel before newSession')
// A hang fixture never resolves alone. Wait for its streamed chunk before cancellation
// so updates deterministically precede the cancelled prompt response.
// Dispatch the prompt WITHOUT awaiting (a hang fixture never resolves on
// its own). To pin frame order deterministically, wait until the client
// has OBSERVED the hang's streamed agent_message_chunk before cancelling —
// so those update frames always precede the cancelled prompt response in
// the transcript (without this, the late chunk and the response race).
// Then cancel and await the prompt, which the bridge settles as
// `cancelled` once the abort propagates.
const promptDone = client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
await waitForUpdate(u => u.sessionUpdate === 'agent_message_chunk')
await client.cancel({ sessionId })
@@ -392,16 +384,6 @@ async function runStep(
}
}
/** Resolve once the child process exits (any code/signal). */
function waitForExit(child: ChildProcessWithoutNullStreams): Promise<void> {
// Race guard: both call sites run within one synchronous frame of
// stdin.end()/kill(), so the exit event cannot have been delivered yet;
// kept for any future caller that awaits in between.
/* v8 ignore next 1 -- unreachable race guard, see above */
if (child.exitCode !== null || child.signalCode !== null) return Promise.resolve()
return new Promise<void>(resolve => child.once('exit', () => { resolve() }))
}
/**
* Harvest EVERY persisted `.jsonl` session log under a sessions root, parse each
* header line, and return them ordered primary-first: the top-level session (no
@@ -442,8 +424,14 @@ async function harvestSessionLogs(root: string): Promise<HarvestedLog[]> {
})
}
}
// Match replay fixture assignment: primary first, then children by creation time, with id as
// a deterministic collision tiebreaker.
// Primary (no parentSession) first, then children by ascending createdAt. A
// scenario has exactly one top-level session. In the synchronous cut sibling
// children are created strictly sequentially, so their createdAt values are
// strictly ordered; the recordedId tiebreak only keeps a degenerate
// same-millisecond collision (unreachable here) deterministic. This harvest
// order must match the replay load order in dsh-llm-replay's loadSessionScripts
// so session.<n>.jsonl maps to the same child on record and replay — replay
// re-sorts childFiles by the same key, so the two stay consistent.
logs.sort((a, b) => {
const ap = a.parentSession === undefined ? 0 : 1
const bp = b.parentSession === undefined ? 0 : 1

24
packages/support/acp-snapshot/src/index.ts
View File

@@ -1,13 +1,23 @@
/**
* ACP snapshot suite kit: subprocess scenario harness, pure golden normalizers, and the Vitest
* suite factory behind `pnpm run test:snapshot`. Because this entry exports `suite.ts`, importing
* it requires a Vitest run.
* ACP snapshot suite kit — the shared machinery behind the keyless snapshot
* tier (`pnpm run test:snapshot`). Four layers, composable per example: the
* shared subprocess/client launcher ({@link launchAcpTestAgent}), the scripted
* scenario harness ({@link runScenario}), the pure golden normalizers
* ({@link normalizeStdout} / {@link normalizeSessionLog} /
* {@link scrubRequestHeaders} / {@link scrubSystemPrompts}), and the suite
* factory ({@link defineAcpSnapshotSuite}) that registers a scenario table as a
* full describe/it tree. Ordinary ACP e2e tests can use the launcher directly;
* an example's `*.snapshot.ts` supplies only its {@link AgentUnderTest} paths,
* snapshots directory, and {@link Scenario} table.
*
* NOTE: ./suite.ts imports vitest, so this package is importable only inside a
* vitest run — a support-tier constraint stated in the README.
*
* @module @deepseek-ai/dsh-acp-snapshot
*/
export {
runScenario,
type AgentUnderTest,
type HarvestedLog,
type InputScript,
type InputStep,
@@ -15,6 +25,12 @@ export {
type RunOptions,
type RunResult,
} from './harness.ts'
export {
launchAcpTestAgent,
type AcpTestLaunchOptions,
type AgentUnderTest,
type LaunchedAcpTestAgent,
} from './launcher.ts'
export {
normalizeSessionLog,
normalizeStdout,

276
packages/support/acp-snapshot/src/launcher.ts Normal file
View File

@@ -0,0 +1,276 @@
/**
* Shared launcher for ACP tests that drive an agent subprocess over JSON-RPC
* stdio. It owns source-or-built launch resolution, workspace environment,
* stdout tee, SDK client, update collection, permission fallback, and process
* shutdown so e2e and snapshot suites do not each reconstruct that boundary.
*
* @module @deepseek-ai/dsh-acp-snapshot/launcher
*/
import { spawn, type ChildProcessWithoutNullStreams } from 'node:child_process'
import { join } from 'node:path'
import { Readable, Writable } from 'node:stream'
import {
ClientSideConnection,
ndJsonStream,
type Agent as AcpAgent,
type Client,
type RequestPermissionRequest,
type RequestPermissionResponse,
type SessionNotification,
} from '@agentclientprotocol/sdk'
import { resolveExampleLaunch } from '@deepseek-ai/dsh-loader-smoke'
/** The source/built agent entry, leaf config, and workspace tsconfig an ACP test boots. */
export interface AgentUnderTest {
/** The agent source bin entry (for example `packages/examples/acp-demo/src/bin.ts`). */
binScript: string
/** Explicit built-mode entry for fixtures whose source path is not under `src/`. */
libBinScript?: string | undefined
/** The leaf `cordis.yml` loaded by the bin. */
configPath: string
/** The repo tsconfig whose paths resolve unbuilt workspace imports. */
tsconfigPath: string
}
/** Options for one ACP test subprocess. */
export interface AcpTestLaunchOptions {
/** The agent composition to boot. */
agent: AgentUnderTest
/** Process cwd and default session-home root. */
cwd: string
/** Alternate leaf config for this launch. */
configPath?: string
/** Extra environment values layered over the parent environment. */
env?: NodeJS.ProcessEnv
/** Permission handler; omitted requests fail closed as `cancelled`. */
requestPermission?: (params: RequestPermissionRequest) => Promise<RequestPermissionResponse>
}
/** A running ACP test process and its captured client-side surfaces. */
export interface LaunchedAcpTestAgent {
/** The child process, exposed for process-level assertions. */
child: ChildProcessWithoutNullStreams
/** Resolve when the OS spawns the child; reject with its asynchronous spawn failure. */
spawned: Promise<void>
/** The SDK connection backed by the child's stdio. */
client: ClientSideConnection
/** Session updates in receive order. */
updates: SessionNotification['update'][]
/** Decode all stdout bytes captured so far. */
rawStdout(): string
/** Decode all stderr chunks captured so far. */
stderr(): string
/** Resolve when a future session update matches the predicate. */
waitForUpdate(match: (update: SessionNotification['update']) => boolean): Promise<SessionNotification['update']>
/** Close the process and drain its streams and callbacks; rejects promptly if fallback termination is refused. */
close(signal?: NodeJS.Signals): Promise<void>
}
/**
* Boot an ACP agent subprocess and connect an SDK client to its stdio.
*
* @param options Agent paths, cwd, environment, and optional permission handler.
* @returns The running process, connected client, captures, and shutdown handle.
*/
export function launchAcpTestAgent(options: AcpTestLaunchOptions): LaunchedAcpTestAgent {
const { agent, cwd } = options
const launch = resolveExampleLaunch({
srcBin: agent.binScript,
libBin: agent.libBinScript,
configArgs: ['--config', options.configPath ?? agent.configPath],
tsconfigPath: agent.tsconfigPath,
env: {
...options.env,
DSH_HOME: join(cwd, '.dsh'),
DSH_AGENTS_HOME: join(cwd, '.agents'),
},
})
const child = spawn(
launch.command,
launch.args,
{
cwd,
env: { ...process.env, ...launch.env },
stdio: ['pipe', 'pipe', 'pipe'],
},
)
// A spawn-level failure is an asynchronous `error` event. Observe it in the
// same tick as spawn so a missing cwd or OS rejection cannot crash the test
// runner, then make startup and shutdown surface the original error.
// Keep observing after the first error: a fallback kill attempted during
// shutdown may itself report another process error, which must not become an
// unhandled EventEmitter error after the promise has already settled.
const childFailure = new Promise<Error>(resolve => child.on('error', resolve))
const spawned = Promise.race([
new Promise<void>(resolve => child.once('spawn', resolve)),
childFailure.then((error): never => { throw error }),
])
// `spawned` is public and close() also awaits it, but a caller may ignore both.
// Keep that misuse from turning the already-observed child error into an
// unhandled promise rejection.
void spawned.catch(() => undefined)
const stderrChunks: string[] = []
child.stderr.setEncoding('utf8')
child.stderr.on('data', (chunk: string) => stderrChunks.push(chunk))
const rawBuffers: Buffer[] = []
const passthrough = new Readable({ read() {} })
const updates: SessionNotification['update'][] = []
const updateWaiters: {
match: (update: SessionNotification['update']) => boolean
resolve: (update: SessionNotification['update']) => void
reject: (reason: unknown) => void
}[] = []
let updateStreamFailure: Error | undefined
const closeUpdateStream = (): void => {
if (updateStreamFailure !== undefined) return
updateStreamFailure = new Error('ACP test agent update stream closed before a matching session update arrived')
for (const waiter of updateWaiters.splice(0)) waiter.reject(updateStreamFailure)
}
child.stdout.on('data', (buffer: Buffer) => {
rawBuffers.push(buffer)
passthrough.push(buffer)
})
child.stdout.on('end', () => {
passthrough.push(null)
})
const stream = ndJsonStream(
Writable.toWeb(child.stdin) as WritableStream<Uint8Array>,
Readable.toWeb(passthrough) as ReadableStream<Uint8Array>,
)
const inFlightClientCallbacks = new Set<Promise<unknown>>()
const trackClientCallback = <T>(callback: () => T | PromiseLike<T>): Promise<T> => {
const pending = Promise.resolve().then(callback)
inFlightClientCallbacks.add(pending)
const untrack = (): void => { inFlightClientCallbacks.delete(pending) }
void pending.then(untrack, untrack)
return pending
}
const requestPermission = options.requestPermission
?? (() => Promise.resolve({ outcome: { outcome: 'cancelled' as const } }))
const makeClient = (_agent: AcpAgent): Client => ({
sessionUpdate(params: SessionNotification): Promise<void> {
return trackClientCallback(() => {
updates.push(params.update)
for (let index = updateWaiters.length - 1; index >= 0; index--) {
const waiter = updateWaiters[index]
/* v8 ignore next 1 -- index is bounded by the array length */
if (waiter === undefined) continue
let matches: boolean
try {
matches = waiter.match(params.update)
} catch (error: unknown) {
updateWaiters.splice(index, 1)
waiter.reject(error)
continue
}
if (!matches) continue
updateWaiters.splice(index, 1)
waiter.resolve(params.update)
}
})
},
requestPermission: params => trackClientCallback(() => requestPermission(params)),
})
const client = new ClientSideConnection(makeClient, stream)
// `exit` only reports the parent process's status. Descendants may retain
// inherited stdout/stderr handles and buffered ACP frames may still be
// crossing the SDK parser. Node's `close` follows stdio closure; the SDK's
// `closed` follows parser exhaustion. Capture both eagerly so a caller that
// invokes close after process exit still joins the complete drain boundary.
const stdioClosed = new Promise<void>(resolve => child.once('close', () => { resolve() }))
const drained = Promise.all([stdioClosed, client.closed]).then(async () => {
// The ACP SDK's readable loop dispatches client callbacks without awaiting
// them. Once `closed` settles no new callbacks can start, but callbacks
// already in flight still belong to this launch's teardown boundary.
while (inFlightClientCallbacks.size > 0) {
await Promise.allSettled([...inFlightClientCallbacks])
}
})
// A caller may await a pending update without calling close(). Make natural
// stream exhaustion terminal for those waiters too, but only after the
// parser has dispatched every buffered frame.
void client.closed.then(closeUpdateStream)
return {
child,
spawned,
client,
updates,
rawStdout: () => Buffer.concat(rawBuffers).toString('utf8'),
stderr: () => stderrChunks.join(''),
waitForUpdate(match): Promise<SessionNotification['update']> {
if (updateStreamFailure !== undefined) return Promise.reject(updateStreamFailure)
return new Promise((resolve, reject) => updateWaiters.push({ match, resolve, reject }))
},
async close(signal?: NodeJS.Signals): Promise<void> {
try {
await spawned
} catch (error: unknown) {
await drained
closeUpdateStream()
throw error
}
if (!isRunning(child)) {
await drained
closeUpdateStream()
return
}
const exited = waitForExit(child)
if (signal === undefined) child.stdin.end()
else child.kill(signal)
const failure = await Promise.race([
exited.then((): undefined => undefined),
childFailure,
])
if (failure === undefined) {
await drained
closeUpdateStream()
return
}
// An `error` after spawn is not an exit edge: in particular, a failed
// signal can leave the subprocess live. Force termination, await the
// already-observed exit edge, and only then propagate the child error so
// callers may safely remove cwd/session resources after close rejects.
const fallbackError = Promise.withResolvers<Error>()
const observeFallbackError = (error: Error): void => { fallbackError.resolve(error) }
child.once('error', observeFallbackError)
if (!child.kill('SIGKILL')) {
child.off('error', observeFallbackError)
closeUpdateStream()
throw new AggregateError(
[failure, new Error('Fallback SIGKILL was not accepted by the child process')],
'ACP test agent failed and fallback termination was refused',
)
}
const fallbackFailure = await Promise.race([
exited.then((): undefined => undefined),
fallbackError.promise,
])
child.off('error', observeFallbackError)
if (fallbackFailure !== undefined) {
closeUpdateStream()
throw new AggregateError(
[failure, fallbackFailure],
'ACP test agent failed and fallback termination was refused',
)
}
await drained
closeUpdateStream()
throw failure
},
}
}
/** Resolve once a running child exits. */
function waitForExit(child: ChildProcessWithoutNullStreams): Promise<void> {
return new Promise<void>(resolve => child.once('exit', () => { resolve() }))
}
/** Whether the child still lacks either OS termination marker. */
function isRunning(child: ChildProcessWithoutNullStreams): boolean {
return child.exitCode === null && child.signalCode === null
}

59
packages/support/acp-snapshot/src/normalize.ts
View File

@@ -14,6 +14,16 @@ const MESSAGE_PREFIX = '{{messagePrefix}}'
/** A UUID v4 string, the shape `randomUUID()` produces for session ids. */
const UUID_RE = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi
const LOCAL_SPILL_PATH_RE = new RegExp(
String.raw`\{\{cwd\}\}/\.spill/session-[0-9a-f]{12}/[0-9a-f]{12}-([A-Za-z0-9._~-]+?)`
+ String.raw`(?=\. Use read with offset/limit|[\s)]|$)`,
'g',
)
const SNAPSHOT_SPILL_PATH_RE = new RegExp(
String.raw`/tmp/dsh-acp-snapshot-spill/session-[0-9a-f]{12}/[0-9a-f]{12}-([A-Za-z0-9._~-]+?)`
+ String.raw`(?=\. Use read with offset/limit|[\s)]|$)`,
'g',
)
/** Inputs the normalizers need to recognize a run's volatile values. */
export interface NormalizeContext {
@@ -29,6 +39,9 @@ function scrubString(value: string, ctx: NormalizeContext): string {
// cwd first (longest, most specific), then explicit session ids, then any
// residual UUID (covers ids that appear in places we didn't enumerate).
out = out.split(ctx.cwd).join(CWD)
out = out.split(`/private${CWD}`).join(CWD)
out = out.replace(LOCAL_SPILL_PATH_RE, (_match, name: string) => `{{spillLocator:${name}}}`)
out = out.replace(SNAPSHOT_SPILL_PATH_RE, (_match, name: string) => `{{spillLocator:${name}}}`)
for (const id of ctx.sessionIds) out = out.split(id).join(SESSION_ID)
out = out.replace(UUID_RE, SESSION_ID)
return out
@@ -112,8 +125,8 @@ export function normalizeSessionLog(rawLog: string, ctx: NormalizeContext): stri
}
/**
* Replace system-prompt content in request headers and header deltas with
* `{{system}}` tokens while retaining field presence and delta structure.
* Replace system-prompt content in request headers with `{{system}}` tokens
* while retaining field presence.
* Other header content stays verbatim, so a header-pinning fixture can keep
* its complete tool schemas while every JSONL fixture omits the prompt text.
* Lines without a system payload pass through byte-for-byte; the transform is
@@ -127,11 +140,11 @@ export function scrubSystemPrompts(rawLog: string): string {
}
/**
* Replace tool schemas in request headers and header deltas with `{{tools}}`
* tokens while retaining field presence, tool names, and delta structure.
* System prompts and session-prefix messages stay verbatim so pinning fixtures
* can move only schema bulk into their dedicated JSON sidecar. Lines without a
* tool payload pass through byte-for-byte; the transform is idempotent.
* Replace tool schemas in full request-header snapshots with `{{tools}}`
* tokens while retaining field presence. System prompts and session-prefix
* messages stay verbatim so pinning fixtures can move only schema bulk into
* their dedicated JSON sidecar. Lines without a tool payload pass through
* byte-for-byte; the transform is idempotent.
*
* @param rawLog The raw session `.jsonl` content.
* @returns The JSONL with tool-schema content tokenized.
@@ -144,9 +157,9 @@ export function scrubToolSchemas(rawLog: string): string {
* Replace all bulky request-header content in a session JSONL with stable
* tokens. This includes the system-prompt fields handled by
* {@link scrubSystemPrompts}, tool schemas, and session-prefix messages. It
* keeps system-delta line positions and arity, tool-delta names, prefix
* message counts, field presence, config, and reason. Lines without content
* to scrub pass through byte-for-byte, and the transform is idempotent.
* keeps prefix message counts, field presence, config, and reason. Lines
* without content to scrub pass through byte-for-byte, and the transform is
* idempotent.
*
* @param rawLog The raw session `.jsonl` content.
* @returns The JSONL with all header bulk tokenized, other lines byte-identical.
@@ -182,33 +195,7 @@ function scrubHeaderContent(rawLog: string, options: HeaderScrubOptions): string
}
return touched ? JSON.stringify(record) : line
}
if (record.type === 'request/header-delta') {
let touched = false
const system = data.system as Record<string, unknown> | null | undefined
if (options.system === true && system !== null && typeof system === 'object' && Array.isArray(system.insert)) {
system.insert = system.insert.map(() => SYSTEM)
touched = true
}
const tools = data.tools as Record<string, unknown> | null | undefined
if (options.tools === true && tools !== null && typeof tools === 'object') {
if (Array.isArray(tools.added)) { tools.added = tools.added.map(scrubToolSchema); touched = true }
if (Array.isArray(tools.changed)) { tools.changed = tools.changed.map(scrubToolSchema); touched = true }
}
if (options.prefix === true && Array.isArray(data.messagePrefix)) {
data.messagePrefix = data.messagePrefix.map(() => MESSAGE_PREFIX)
touched = true
}
return touched ? JSON.stringify(record) : line
}
return line
})
return out.join('\n')
}
/** Tokenize one tool schema's bulk (description, parameters, anything else), keeping its identifying `name`. */
function scrubToolSchema(tool: unknown): unknown {
if (tool === null || typeof tool !== 'object' || Array.isArray(tool)) return tool
const out: Record<string, unknown> = {}
for (const [k, v] of Object.entries(tool)) out[k] = k === 'name' ? v : TOOLS
return out
}

352
packages/support/acp-snapshot/src/suite.ts
View File

@@ -1,16 +1,21 @@
/**
* Keyless-by-default ACP snapshot suite factory. Each scenario drives the real subprocess and
* compares normalized stdout; comparable session fixtures are both replay input and expected
* output. Record mode refreshes reproducible model scenarios from the live API, while refresh
* mode replays committed scripts and rewrites derived artifacts without a key.
* Keyless-by-default ACP snapshot suite factory. Each scenario drives the real
* subprocess and compares normalized stdout; comparable session fixtures are
* both replay input and expected output. Record mode refreshes reproducible
* model scenarios from the live API, while refresh mode replays committed
* scripts and rewrites derived artifacts without a key.
* Replay scenarios run concurrently because each subprocess owns unique temp
* cwd and persistence roots and reads only committed fixtures. Record and
* refresh stay serial while writing.
*
* Exactly one scenario per header-composition class pins the system prompt and tool schemas in
* dedicated sidecars. Every live header is checked against that pin, so session-dependent
* composition must declare a separate class instead of escaping coverage.
* Exactly one scenario per header-composition class pins the full prompt and
* tool-schema sequences in dedicated sidecars. Every live header is checked
* against that pin, so session-dependent composition must declare a separate
* class instead of escaping coverage.
* @module @deepseek-ai/dsh-acp-snapshot/suite
*/
import { readFile, readdir, writeFile } from 'node:fs/promises'
import { readFile, readdir, rm, writeFile } from 'node:fs/promises'
import { existsSync } from 'node:fs'
import { join } from 'node:path'
import { describe, expect, it } from 'vitest'
@@ -66,28 +71,17 @@ export interface Scenario {
* false (replay derives from the fixture's `assistant/chunk` events).
*/
overridden?: boolean
/**
* How many SUBAGENT child sessions this scenario records beyond the top-level
* one (0 for a single-session scenario). Each child rides in a sibling fixture
* `session.<n>.jsonl` (1-based); replay forwards them to `dsh-llm-replay` so
* each child session replays from its own script, and record mode writes the
* harvested child logs back to those files. Defaults to 0.
*/
childSessions?: number
/**
* Whether this scenario is its header class's sole request-header pin. Dedicated sidecars own
* the prompt and tool schemas, while every classmate is checked for equality.
*/
pinsHeader?: boolean
/**
* How many `request/header-delta` events this PINNING scenario's fixture
* legitimately carries (default 0). A recorded mid-run header change — a
* config-option switch rewriting a prompt section — is part of the pinned
* surface, with readable prompt text in Markdown; any OTHER count
* still fails, so fixture rot stays caught. Meaningless off the pin (the
* live uniformity guard keeps non-pinning scenarios delta-free).
* How many changed `request/header` snapshots this PINNING scenario's primary
* fixture legitimately carries (default 0). Their full prompt text is kept in
* the readable Markdown pin; any other count fails. Meaningless off the pin.
*/
expectedHeaderDeltas?: number
expectedHeaderChanges?: number
/**
* Which header-composition class this scenario belongs to. Scenarios that
* boot the same config compose the same header; each class has exactly one
@@ -127,14 +121,40 @@ export interface SnapshotSuiteOptions {
}
/**
* The sibling child-fixture paths for a scenario (`session.1.jsonl` …).
* Validate and order a scenario directory's session-fixture filenames.
*
* @param dir The scenario's snapshots directory (`<snapshotsDir>/<name>`).
* @param childSessions How many subagent child sessions the scenario records.
* @returns One path per child, 1-based, in fixture order.
* The primary fixture is always `session.jsonl`; child sessions are discovered
* from contiguous `session.1.jsonl` … filenames. The directory is the source of
* truth, so scenario tables do not duplicate a child count that can drift from
* the files. A session-like JSONL with any other suffix fails loud.
*
* @param names File names in one scenario directory.
* @returns The primary and child fixture names in replay/harvest order.
*/
export function childFixturePaths(dir: string, childSessions: number): string[] {
return Array.from({ length: childSessions }, (_, i) => join(dir, `session.${i + 1}.jsonl`))
export function sessionFixtureNames(names: readonly string[]): string[] {
if (!names.includes('session.jsonl')) throw new Error('missing session.jsonl')
const children: { name: string; index: number }[] = []
for (const name of names) {
if (name === 'session.jsonl') continue
if (!name.startsWith('session.') || !name.endsWith('.jsonl')) continue
const match = /^session\.([1-9]\d*)\.jsonl$/.exec(name)
if (match === null) throw new Error(`invalid child session fixture name: ${name}`)
children.push({ name, index: Number(match[1]) })
}
children.sort((a, b) => a.index - b.index)
for (const [offset, child] of children.entries()) {
const expected = offset + 1
if (child.index !== expected) {
throw new Error(`child session fixtures must be contiguous: expected session.${expected}.jsonl, found ${child.name}`)
}
}
return ['session.jsonl', ...children.map(child => child.name)]
}
/** Read one scenario directory's validated session-fixture inventory. */
async function sessionFixtures(dir: string): Promise<string[]> {
const entries = await readdir(dir, { withFileTypes: true })
return sessionFixtureNames(entries.filter(entry => entry.isFile()).map(entry => entry.name))
}
/**
@@ -208,114 +228,58 @@ export function normalizedToolSchemas(rawLog: string, ctx: NormalizeContext): un
})
}
/**
* Extract normalized tool-schema edits from request-header deltas in log order.
* Deltas without an object-valued tools edit are omitted; their remaining
* structure stays pinned in the session JSONL.
*
* @param rawLog The session `.jsonl` content to inspect.
* @param ctx The volatile values of the run that produced it.
* @returns The normalized tool-schema edits, in event order.
*/
export function normalizedToolSchemaDeltas(rawLog: string, ctx: NormalizeContext): unknown[] {
return normalizeSessionLog(rawLog, ctx)
.split('\n')
.filter(line => line.trim().length > 0)
.map(line => JSON.parse(line) as { type?: unknown; data?: { tools?: unknown } })
.filter(record => record.type === 'request/header-delta')
.flatMap((record) => {
const tools = record.data?.tools
return tools !== null && typeof tools === 'object' && !Array.isArray(tools) ? [tools] : []
})
}
/** The structured contents of a tool-schema sidecar. */
export interface ToolSchemasSnapshot {
/** The complete tool schemas from the pinned request header. */
initial: unknown[]
/** Complete tool-schema edits from subsequent request-header deltas. */
deltas: unknown[]
/** Complete tool schemas from subsequent changed-header snapshots. */
changes: unknown[][]
}
/**
* Render tool schemas and later schema edits as canonical, readable JSON.
* Render the full tool-schema sequence as canonical, readable JSON.
*
* @param initial The pinned request header's complete tool schemas.
* @param deltas Complete tool-schema edits from request-header deltas.
* @param changes Complete tool schemas from later changed headers.
* @returns A pretty-printed JSON snapshot ending in one newline.
*/
export function formatToolSchemasSnapshot(initial: readonly unknown[], deltas: readonly unknown[] = []): string {
return `${JSON.stringify({ initial, deltas }, null, 2)}\n`
export function formatToolSchemasSnapshot(initial: readonly unknown[], changes: readonly unknown[][] = []): string {
return `${JSON.stringify({ initial, changes }, null, 2)}\n`
}
/**
* Parse and validate the stable top-level shape of a tool-schema sidecar.
*
* @param snapshot The JSON sidecar text.
* @returns Its initial schemas and schema deltas.
* @returns Its initial and changed-header schema sets.
*/
export function parseToolSchemasSnapshot(snapshot: string): ToolSchemasSnapshot {
const parsed = JSON.parse(snapshot) as unknown
if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) {
throw new Error('acp-snapshot: tool-schema snapshot must be an object')
}
const { initial, deltas } = parsed as { initial?: unknown; deltas?: unknown }
if (!Array.isArray(initial) || !Array.isArray(deltas)) {
throw new Error('acp-snapshot: tool-schema snapshot must carry array-valued initial and deltas fields')
const { initial, changes } = parsed as { initial?: unknown; changes?: unknown }
if (!Array.isArray(initial) || !Array.isArray(changes) || !changes.every(Array.isArray)) {
throw new Error('acp-snapshot: tool-schema snapshot must carry array-valued initial and changes fields')
}
return { initial, deltas }
return { initial, changes }
}
/**
* Restore a sidecar's initial schemas into a tokenized pinned header.
* Restore one sidecar schema set into a tokenized pinned header.
*
* @param header The parsed request header carrying `tools: "{{tools}}"`.
* @param snapshot The parsed tool-schema sidecar.
* @returns A copy of the header with its complete initial schemas restored.
* @param schemas The complete schemas for this full header snapshot.
* @returns A copy of the header with its complete schemas restored.
*/
export function restorePinnedToolSchemas(header: unknown, snapshot: ToolSchemasSnapshot): unknown {
export function restorePinnedToolSchemas(header: unknown, schemas: readonly unknown[]): unknown {
if (header === null || typeof header !== 'object' || Array.isArray(header)) {
throw new Error('acp-snapshot: pinned request header must be an object')
}
if ((header as { tools?: unknown }).tools !== TOOLS_TOKEN) {
throw new Error(`acp-snapshot: pinned request header tools must equal ${TOOLS_TOKEN}`)
}
return { ...header, tools: snapshot.initial }
}
/** One normalized system-prompt edit carried by a `request/header-delta`. */
export interface SystemPromptDeltaSnapshot {
/** How many leading lines remain from the prior prompt. */
keepStart: number
/** How many trailing lines remain from the prior prompt. */
keepEnd: number
/** The normalized replacement lines inserted between the retained ranges. */
insert: string[]
}
/**
* Extract normalized system-prompt edits from request-header deltas in log
* order. Deltas without a well-formed system edit are omitted; their non-prompt
* structure remains pinned in JSONL.
*
* @param rawLog The session `.jsonl` content to inspect.
* @param ctx The volatile values of the run that produced it.
* @returns The normalized system-prompt edits, in event order.
*/
export function normalizedSystemPromptDeltas(rawLog: string, ctx: NormalizeContext): SystemPromptDeltaSnapshot[] {
return normalizeSessionLog(rawLog, ctx)
.split('\n')
.filter(line => line.trim().length > 0)
.map(line => JSON.parse(line) as { type?: unknown; data?: { system?: unknown } })
.filter(record => record.type === 'request/header-delta')
.flatMap((record) => {
const system = record.data?.system
if (system === null || typeof system !== 'object') return []
const { keepStart, keepEnd, insert } = system as { keepStart?: unknown; keepEnd?: unknown; insert?: unknown }
if (typeof keepStart !== 'number' || typeof keepEnd !== 'number' || !Array.isArray(insert)) return []
if (!insert.every(line => typeof line === 'string')) return []
return [{ keepStart, keepEnd, insert: insert }]
})
return { ...header, tools: schemas }
}
/**
@@ -324,38 +288,40 @@ export function normalizedSystemPromptDeltas(rawLog: string, ctx: NormalizeConte
* the committed file follows the repository newline contract.
*
* @param prompt The normalized system prompt.
* @param deltas Normalized prompt edits to append as readable sections.
* @param changes Full normalized prompts from later changed-header snapshots.
* @returns Markdown snapshot text ending in a newline.
*/
export function formatSystemPromptSnapshot(
prompt: string,
deltas: readonly SystemPromptDeltaSnapshot[] = [],
changes: readonly string[] = [],
): string {
let snapshot = prompt.endsWith('\n') ? prompt : `${prompt}\n`
for (const [index, delta] of deltas.entries()) {
snapshot += `\n<!-- request/header-delta ${index + 1}: keepStart=${delta.keepStart}, keepEnd=${delta.keepEnd} -->\n\n`
const insert = delta.insert.join('\n')
snapshot += insert.endsWith('\n') ? insert : `${insert}\n`
for (const [index, change] of changes.entries()) {
snapshot += `\n<!-- request/header change ${index + 1} -->\n\n`
snapshot += change.endsWith('\n') ? change : `${change}\n`
}
return snapshot
}
/** Return the initial-prompt portion of a possibly delta-bearing snapshot. */
/** Return the initial-prompt portion of a possibly multi-header snapshot. */
function initialSystemPromptSnapshot(snapshot: string): string {
const marker = snapshot.indexOf('\n<!-- request/header-delta ')
const marker = snapshot.indexOf('\n<!-- request/header change ')
return marker < 0 ? snapshot : snapshot.slice(0, marker)
}
/**
* Count the `request/header-delta` events in a session JSONL.
* Count changed `request/header` snapshots in a session JSONL.
*
* @param rawLog The session `.jsonl` content.
* @returns How many `request/header-delta` events the log carries.
* @returns How many headers carry reason `change`.
*/
export function headerDeltaCount(rawLog: string): number {
export function headerChangeCount(rawLog: string): number {
return rawLog.split('\n')
.filter(line => line.trim().length > 0)
.filter(line => (JSON.parse(line) as { type?: unknown }).type === 'request/header-delta')
.filter((line) => {
const record = JSON.parse(line) as { type?: unknown; data?: { reason?: unknown } }
return record.type === 'request/header' && record.data?.reason === 'change'
})
.length
}
@@ -461,7 +427,7 @@ export function stabilizeRefreshLog(fresh: string, existing: string, replacement
}
/**
* Register the suite: one `describe` per scenario (the golden/log compares and
* Register the suite: one test per scenario (the golden/log compares and
* the header-uniformity guard) plus the fixture guard block (no orphan
* scenario dirs, required files present, exactly one pin per header class,
* pinning fixtures well-formed, every JSONL prompt-scrubbed, non-pinning
@@ -477,6 +443,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
const RECORDING = mode === 'record'
const REFRESHING = mode === 'refresh'
const childMode: 'replay' | 'record' = RECORDING ? 'record' : 'replay'
const scenarioSuite = mode === 'replay' ? describe.concurrent : describe
/** The class a scenario's header composition belongs to (see {@link Scenario.headerClass}). */
const classOf = (scenario: Scenario): string => scenario.headerClass ?? 'default'
@@ -496,16 +463,21 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
}
}
for (const scenario of scenarios) {
describe(`snapshot: ${scenario.name}`, () => {
scenarioSuite('snapshot scenarios', () => {
for (const scenario of scenarios) {
// In RECORD mode, only re-run the `recorded` (live-API) scenarios; the `authored` ones
// (sidecar-driven errors/cancel) are never re-recorded.
it.skipIf(RECORDING && !scenario.recorded)('matches the goldens', async () => {
it.skipIf(RECORDING && !scenario.recorded)(`snapshot: ${scenario.name} matches the goldens`, async ({ expect }) => {
const dir = join(snapshotsDir, scenario.name)
const input = JSON.parse(await readFile(join(dir, 'input.json'), 'utf8')) as InputScript
const overrideFile = join(dir, 'replay.override.json')
const workspaceDir = join(dir, 'workspace')
const childSessions = scenario.childSessions ?? 0
// Replay/refresh need the committed inventory up front because those
// files drive the model scripts. Record mode creates that inventory
// from the harvested live logs, so it must also work for a brand-new
// scenario with no session.jsonl yet.
let fixtureFiles = RECORDING ? [] : await sessionFixtures(dir)
const childFixtureFiles = fixtureFiles.slice(1)
const comparesLog = scenario.comparesLog ?? scenario.hasModelTurn
const result = await runScenario(input, {
agent,
@@ -514,7 +486,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
...existsSync(overrideFile) ? { overrideFile } : {},
// In REPLAY, forward the recorded child fixtures so each subagent session
// replays from its own script. In RECORD they are harvested, not read.
...!RECORDING && childSessions > 0 ? { childFiles: childFixturePaths(dir, childSessions) } : {},
...!RECORDING && childFixtureFiles.length > 0 ? { childFiles: childFixtureFiles.map(file => join(dir, file)) } : {},
...existsSync(workspaceDir) ? { workspaceDir } : {},
// A scenario booting an overlay tree passes its own live config; the
// bin's replay swap derives the sibling `*cordis.snapshot.yml` from it.
@@ -542,7 +514,6 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
const scrub = scenario.pinsHeader === true
? (log: string): string => scrubToolSchemas(scrubSystemPrompts(log))
: scrubRequestHeaders
const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)]
const existingFixtures = REFRESHING
? await Promise.all(fixtureFiles.map(file => readFile(join(dir, file), 'utf8')))
: []
@@ -551,43 +522,51 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|| (REFRESHING && comparesLog)
if (writesSessionFixtures) {
expect(result.sessionLogs.length, `${mode} produced no session log to harvest`).toBeGreaterThan(0)
expect(result.sessionLogs.length, `expected ${childSessions + 1} session logs (parent + children)`)
.toBe(childSessions + 1)
if (REFRESHING) {
expect(result.sessionLogs.length, `expected ${fixtureFiles.length} session logs (parent + children)`)
.toBe(fixtureFiles.length)
}
const outputFixtureFiles = [
'session.jsonl',
...Array.from({ length: result.sessionLogs.length - 1 }, (_, i) => `session.${i + 1}.jsonl`),
]
const primary = (result.sessionLogs[0] as HarvestedLog).content
await writeFile(join(dir, 'session.jsonl'), scrub(
await writeFile(join(dir, outputFixtureFiles[0] as string), scrub(
REFRESHING ? stabilizeRefreshLog(primary, existingFixtures[0] as string, replacements) : primary,
))
for (let i = 1; i < result.sessionLogs.length; i++) {
const child = (result.sessionLogs[i] as HarvestedLog).content
await writeFile(join(dir, `session.${i}.jsonl`), scrub(
await writeFile(join(dir, outputFixtureFiles[i] as string), scrub(
REFRESHING ? stabilizeRefreshLog(child, existingFixtures[i] as string, replacements) : child,
))
}
if (RECORDING) {
const outputNames = new Set(outputFixtureFiles)
const entries = await readdir(dir, { withFileTypes: true })
await Promise.all(entries
.filter(entry => entry.isFile()
// Only valid numbered children are record-owned stale output.
// Malformed session-like names stay for the inventory guard to
// reject instead of being silently deleted during mutation.
&& /^session\.[1-9]\d*\.jsonl$/.test(entry.name)
&& !outputNames.has(entry.name))
.map(entry => rm(join(dir, entry.name))))
fixtureFiles = outputFixtureFiles
}
if (scenario.pinsHeader === true) {
const prompts = result.sessionLogs.flatMap(log => normalizedSystemPrompts(log.content, ctx))
expect(prompts.length, `${mode} produced no system prompt to snapshot`).toBeGreaterThan(0)
const initialSnapshot = formatSystemPromptSnapshot(prompts[0] as string)
for (const prompt of prompts) {
expect(formatSystemPromptSnapshot(prompt), 'the pinning run produced divergent system prompts')
.toEqual(initialSnapshot)
}
const primary = result.sessionLogs[0] as HarvestedLog
const snapshot = formatSystemPromptSnapshot(
prompts[0] as string,
normalizedSystemPromptDeltas(primary.content, ctx),
)
const prompts = normalizedSystemPrompts(primary.content, ctx)
expect(prompts.length, `${mode} produced no system prompt to snapshot`).toBeGreaterThan(0)
const snapshot = formatSystemPromptSnapshot(prompts[0] as string, prompts.slice(1))
await writeFile(join(dir, SYSTEM_PROMPT_SNAPSHOT), snapshot)
const schemaSets = result.sessionLogs.flatMap(log => normalizedToolSchemas(log.content, ctx))
const schemaSets = normalizedToolSchemas(primary.content, ctx)
expect(schemaSets.length, `${mode} produced no tool schemas to snapshot`).toBeGreaterThan(0)
const initialSchemaSnapshot = formatToolSchemasSnapshot(schemaSets[0] as unknown[])
for (const schemas of schemaSets) {
expect(formatToolSchemasSnapshot(schemas), 'the pinning run produced divergent tool schemas')
.toEqual(initialSchemaSnapshot)
}
expect(schemaSets.length, `${mode} produced a tool-schema sequence that differs from its prompt sequence`)
.toBe(prompts.length)
await writeFile(join(dir, TOOL_SCHEMAS_SNAPSHOT), formatToolSchemasSnapshot(
schemaSets[0] as unknown[],
normalizedToolSchemaDeltas(primary.content, ctx),
schemaSets.slice(1),
))
}
}
@@ -602,7 +581,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
// produce one without a model turn (a `rejected` turn carrying `hook/*`).
if (comparesLog) {
// The harvested logs (primary-first) must match their committed fixtures 1:1.
expect(result.sessionLogs.length, 'this scenario must persist a session log').toBe(childSessions + 1)
expect(result.sessionLogs.length, 'this scenario must persist one log per session fixture').toBe(fixtureFiles.length)
for (let i = 0; i < fixtureFiles.length; i++) {
const harvested = scrub((result.sessionLogs[i] as HarvestedLog).content)
const fixture = scrub(await readFile(join(dir, fixtureFiles[i] as string), 'utf8'))
@@ -611,8 +590,8 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
}
}
// Header-uniformity guard: every live header in a class must equal the class pin split
// across tokenized JSONL plus readable prompt and structured schema sidecars.
// Every live full header must equal its class pin reconstructed from
// tokenized JSONL plus readable prompt and structured schema sidecars.
/* v8 ignore next -- construction guarantees the pin exists; a miss would fail the one-header assertion loudly. */
const pinningScenario = pinningByClass.get(classOf(scenario)) ?? scenario
const pinningDir = join(snapshotsDir, pinningScenario.name)
@@ -620,17 +599,23 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
const pinned = normalizedHeaders(pinnedFixture, fixtureContext(pinnedFixture))
const promptSnapshot = await readFile(join(pinningDir, SYSTEM_PROMPT_SNAPSHOT), 'utf8')
const initialPromptSnapshot = initialSystemPromptSnapshot(promptSnapshot)
expect(pinned.length, `the pinning fixture (${pinningScenario.name}) has an unexpected request/header count`)
.toBe(1 + (pinningScenario.expectedHeaderChanges ?? 0))
const toolSchemasSnapshot = await readFile(join(pinningDir, TOOL_SCHEMAS_SNAPSHOT), 'utf8')
const toolSchemas = parseToolSchemasSnapshot(toolSchemasSnapshot)
expect(pinned.length, `the pinning fixture (${pinningScenario.name}) must carry exactly one request/header`)
.toBe(1)
const pinnedHeader = restorePinnedToolSchemas(pinned[0], toolSchemas)
const pinnedSchemaSets = [toolSchemas.initial, ...toolSchemas.changes]
expect(pinnedSchemaSets.length, `the pinning fixture (${pinningScenario.name}) has an unexpected tool-schema count`)
.toBe(pinned.length)
const pinnedHeaders = pinned.map((header, index) => restorePinnedToolSchemas(
header,
pinnedSchemaSets[index] as unknown[],
))
for (const [logIndex, log] of result.sessionLogs.entries()) {
const expectedDeltas = scenario.pinsHeader === true && logIndex === 0
? scenario.expectedHeaderDeltas ?? 0
const expectedChanges = scenario.pinsHeader === true && logIndex === 0
? scenario.expectedHeaderChanges ?? 0
: 0
expect(headerDeltaCount(log.content), `session ${log.id}: request/header-delta count`)
.toBe(expectedDeltas)
expect(headerChangeCount(log.content), `session ${log.id}: changed request/header count`)
.toBe(expectedChanges)
const headers = normalizedHeaders(scrubSystemPrompts(log.content), ctx)
const prompts = normalizedSystemPrompts(log.content, ctx)
const schemaSets = normalizedToolSchemas(log.content, ctx)
@@ -639,27 +624,30 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
expect(schemaSets.length, `session ${log.id}: every request/header must carry an array-valued tools field`)
.toBe(headers.length)
for (const [k, header] of headers.entries()) {
const expected = expectedChanges > 0 ? pinnedHeaders[k] : pinnedHeaders[0]
expect(header, `session ${log.id}: request/header #${k + 1} diverged from the pinned (${pinningScenario.name}) header`)
.toEqual(pinnedHeader)
expect(formatSystemPromptSnapshot(prompts[k] as string), `session ${log.id}: initial system prompt #${k + 1} diverged from ${pinningScenario.name}/${SYSTEM_PROMPT_SNAPSHOT}`)
.toEqual(initialPromptSnapshot)
.toEqual(expected)
if (expectedChanges === 0) {
expect(formatSystemPromptSnapshot(prompts[k] as string), `session ${log.id}: initial system prompt #${k + 1} diverged from ${pinningScenario.name}/${SYSTEM_PROMPT_SNAPSHOT}`)
.toEqual(initialPromptSnapshot)
}
}
if (scenario.pinsHeader === true && logIndex === 0) {
expect(formatSystemPromptSnapshot(
prompts[0] as string,
normalizedSystemPromptDeltas(log.content, ctx),
), `session ${log.id}: system-prompt deltas diverged from ${pinningScenario.name}/${SYSTEM_PROMPT_SNAPSHOT}`)
prompts.slice(1),
), `session ${log.id}: changed system prompts diverged from ${pinningScenario.name}/${SYSTEM_PROMPT_SNAPSHOT}`)
.toEqual(promptSnapshot)
expect(formatToolSchemasSnapshot(
schemaSets[0] as unknown[],
normalizedToolSchemaDeltas(log.content, ctx),
), `session ${log.id}: tool-schema deltas diverged from ${pinningScenario.name}/${TOOL_SCHEMAS_SNAPSHOT}`)
schemaSets.slice(1),
), `session ${log.id}: changed tool schemas diverged from ${pinningScenario.name}/${TOOL_SCHEMAS_SNAPSHOT}`)
.toEqual(toolSchemasSnapshot)
}
}
})
})
}
}
})
describe('snapshot fixtures', () => {
it('every scenario directory is registered (no orphans)', async () => {
@@ -672,9 +660,9 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
expect(onDisk).toEqual(registered)
})
it('every registered scenario has its required fixture files', () => {
// Every scenario has an input script and an stdout golden.
for (const { name, overridden, childSessions, pinsHeader } of scenarios) {
it('every registered scenario has its required fixture files', async () => {
// Every scenario needs input, stdout, a primary session fixture, and matching optional sidecars.
for (const { name, overridden, pinsHeader } of scenarios) {
const dir = join(snapshotsDir, name)
expect(existsSync(join(dir, 'input.json')), `${name}/input.json`).toBe(true)
expect(existsSync(join(dir, 'stdout.golden.jsonl')), `${name}/stdout.golden.jsonl`).toBe(true)
@@ -685,11 +673,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
.toBe(pinsHeader === true)
expect(existsSync(join(dir, TOOL_SCHEMAS_SNAPSHOT)), `${name}/${TOOL_SCHEMAS_SNAPSHOT} presence must match \`pinsHeader\``)
.toBe(pinsHeader === true)
// A nested-agent scenario ships one child fixture per recorded subagent
// session (`session.1.jsonl` …), the replay source for that child session.
for (const childFixture of childFixturePaths(dir, childSessions ?? 0)) {
expect(existsSync(childFixture), childFixture).toBe(true)
}
await expect(sessionFixtures(dir), `${name}: session fixture inventory`).resolves.toBeDefined()
}
})
@@ -708,25 +692,30 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
}
})
it('every pinning fixture carries one tokenized request/header, two sidecars, and its declared deltas', async () => {
// The live uniformity guard runs only in NON-pinning scenarios, so a class made of just
// its pinning scenario would otherwise accept a re-recorded pin with several headers or
// an undeclared mid-run header-delta — shapes the pin design cannot represent.
it('every pinning fixture carries one tokenized header sequence and two sidecars', async () => {
// Assert the committed pin directly because a class containing only its
// pinning scenario has no non-pinning live run to catch undeclared changes.
for (const scenario of pinningByClass.values()) {
const fixture = await readFile(join(snapshotsDir, scenario.name, 'session.jsonl'), 'utf8')
const headers = normalizedHeaders(fixture, fixtureContext(fixture))
const promptSnapshot = await readFile(join(snapshotsDir, scenario.name, SYSTEM_PROMPT_SNAPSHOT), 'utf8')
expect(headers.length, `${scenario.name}: unexpected request/header count`)
.toBe(1 + (scenario.expectedHeaderChanges ?? 0))
const toolSchemasSnapshot = await readFile(join(snapshotsDir, scenario.name, TOOL_SCHEMAS_SNAPSHOT), 'utf8')
const toolSchemas = parseToolSchemasSnapshot(toolSchemasSnapshot)
expect(headers.length, `${scenario.name}: a pinning fixture must carry exactly one request/header`).toBe(1)
expect(() => restorePinnedToolSchemas(headers[0], toolSchemas), `${scenario.name}: tools must use the sidecar token`)
.not.toThrow()
const schemaSets = [toolSchemas.initial, ...toolSchemas.changes]
expect(schemaSets.length, `${scenario.name}: tool-schema sequence must match the header sequence`)
.toBe(headers.length)
for (const [index, header] of headers.entries()) {
expect(() => restorePinnedToolSchemas(header, schemaSets[index] as unknown[]), `${scenario.name}: tools must use the sidecar token`)
.not.toThrow()
}
expect(promptSnapshot.length, `${scenario.name}/${SYSTEM_PROMPT_SNAPSHOT} must not be empty`).toBeGreaterThan(0)
expect(promptSnapshot.endsWith('\n'), `${scenario.name}/${SYSTEM_PROMPT_SNAPSHOT} must end in a newline`).toBe(true)
expect(toolSchemasSnapshot, `${scenario.name}/${TOOL_SCHEMAS_SNAPSHOT} must use canonical JSON formatting`)
.toBe(formatToolSchemasSnapshot(toolSchemas.initial, toolSchemas.deltas))
expect(headerDeltaCount(fixture), `${scenario.name}: a pinning fixture must carry exactly its declared request/header-deltas`)
.toBe(scenario.expectedHeaderDeltas ?? 0)
.toBe(formatToolSchemasSnapshot(toolSchemas.initial, toolSchemas.changes))
expect(headerChangeCount(fixture), `${scenario.name}: a pinning fixture must carry exactly its declared changed headers`)
.toBe(scenario.expectedHeaderChanges ?? 0)
}
})
@@ -736,10 +725,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
// storage rules fail loud.
for (const scenario of scenarios) {
const dir = join(snapshotsDir, scenario.name)
const files = [
'session.jsonl',
...Array.from({ length: scenario.childSessions ?? 0 }, (_, i) => `session.${i + 1}.jsonl`),
]
const files = await sessionFixtures(dir)
for (const file of files) {
const fixture = await readFile(join(dir, file), 'utf8')
expect(unknownToolCallIds(fixture), `${scenario.name}/${file} contains UNKNOWN_TOOL`)

24
packages/support/acp-snapshot/tests/fixtures/fake-acp-agent.ts vendored
View File

@@ -6,6 +6,7 @@
import { mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
import { readdirSync } from 'node:fs'
import { spawn } from 'node:child_process'
import { dirname, join } from 'node:path'
import { randomUUID } from 'node:crypto'
import { createInterface } from 'node:readline'
@@ -24,6 +25,8 @@ interface ScriptedLog {
/** The whole scripted behavior for one run. Every field defaults to the least surprising choice. */
interface Behavior {
/** Exit during startup after writing any configured stderr note. */
failOnBoot?: boolean
/** Reject every `session/new` (exercises the expect-error step without extra dirs). */
rejectNewSession?: boolean
/** Reject `session/new` only when `additionalDirectories` is non-empty (the real bridge's rule). */
@@ -38,6 +41,8 @@ interface Behavior {
echoWorkspace?: boolean
/** Write a line to stderr on boot (spec-side stderr-capture assertions). */
stderrNote?: string
/** Let a short-lived descendant retain stdio and emit one final ACP update plus stderr line after this parent exits. */
lateInheritedOutput?: boolean
/** Session logs to persist on stdin EOF. */
logs?: ScriptedLog[]
/** Leave a stray FILE directly under the sessions root (harvest must skip it). */
@@ -62,6 +67,7 @@ const behavior: Behavior = fixtureFile === ''
: JSON.parse(readFileSync(join(dirname(fixtureFile), 'behavior.json'), 'utf8')) as Behavior
if (behavior.stderrNote !== undefined) process.stderr.write(`${behavior.stderrNote}\n`)
if (behavior.failOnBoot === true) process.exit(7)
let nextOutboundId = 1000
let sessionId = ''
@@ -244,6 +250,24 @@ function flushLogsAndExit(): void {
writeFileSync(join(sessionsRoot, 'bucket-noise', 'notes.txt'), 'not a session log\n')
}
if (behavior.deleteSessionsRoot === true) rmSync(sessionsRoot, { recursive: true, force: true })
if (behavior.lateInheritedOutput === true) {
const frame = JSON.stringify({
jsonrpc: '2.0',
method: 'session/update',
params: {
sessionId,
update: {
sessionUpdate: 'agent_message_chunk',
content: { type: 'text', text: 'late inherited stdout' },
},
},
})
const code = [
`setTimeout(() => process.stdout.write(${JSON.stringify(`${frame}\n`)}), 50)`,
`setTimeout(() => process.stderr.write(${JSON.stringify('late inherited stderr\n')}), 75)`,
].join(';')
spawn(process.execPath, ['-e', code], { stdio: ['ignore', 1, 2] }).unref()
}
process.exit(0)
}

2
packages/support/acp-snapshot/tests/fixtures/record-suite/rec-pin/tool-schemas.golden.json vendored
View File

@@ -8,5 +8,5 @@
}
}
],
"deltas": []
"changes": []
}

2
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/behavior.json vendored
View File

@@ -5,7 +5,7 @@
"lines": [
{ "type": "session", "id": "{{SID}}", "createdAt": 100, "cwd": "{{CWD}}" },
{ "type": "request/header", "seq": 0, "time": 100, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } },
{ "type": "request/header-delta", "seq": 1, "time": 100, "data": { "system": { "keepStart": 1, "keepEnd": 0, "insert": ["NEW PROMPT LINE"] } } },
{ "type": "request/header", "seq": 1, "time": 100, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT\n\nNEW PROMPT LINE", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "change" } },
{ "type": "turn/start", "seq": 2, "time": 100, "data": { "turn": 1 } }
]
}]

2
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/session.jsonl vendored
View File

@@ -1,4 +1,4 @@
{"type":"session","id":"12121212-3434-4545-8686-787878787878","createdAt":7,"cwd":"/rec/pin-cwd"}
{"type":"request/header","seq":0,"time":7,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
{"type":"request/header-delta","seq":1,"time":7,"data":{"system":{"keepStart":1,"keepEnd":0,"insert":["{{system}}"]}}}
{"type":"request/header","seq":1,"time":7,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"change"}}
{"type":"turn/start","seq":2,"time":7,"data":{"turn":1}}

4
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/system-prompt.golden.md vendored
View File

@@ -1,5 +1,7 @@
SYS PROMPT
<!-- request/header-delta 1: keepStart=1, keepEnd=0 -->
<!-- request/header change 1 -->
SYS PROMPT
NEW PROMPT LINE

12
packages/support/acp-snapshot/tests/fixtures/suite/pin-turn/tool-schemas.golden.json vendored
View File

@@ -8,5 +8,15 @@
}
}
],
"deltas": []
"changes": [
[
{
"name": "t1",
"description": "D1",
"parameters": {
"type": "object"
}
}
]
]
}

264
packages/support/acp-snapshot/tests/harness.spec.ts
View File

@@ -1,23 +1,46 @@
import { mkdtemp, rm, writeFile } from 'node:fs/promises'
import { once } from 'node:events'
import { tmpdir } from 'node:os'
import { delimiter, join } from 'node:path'
import { fileURLToPath } from 'node:url'
import { afterAll, describe, expect, it } from 'vitest'
import { afterAll, describe, expect, it, vi } from 'vitest'
import { PROTOCOL_VERSION } from '@agentclientprotocol/sdk'
import { runScenario, type AgentUnderTest, type InputStep } from '../src/harness.ts'
import { launchAcpTestAgent } from '../src/launcher.ts'
const fsControl = vi.hoisted(() => ({ cleanupFailure: undefined as Error | undefined }))
vi.mock('node:fs/promises', async (importOriginal) => {
const actual = await importOriginal<typeof import('node:fs/promises')>()
return {
...actual,
async rm(...args: Parameters<typeof actual.rm>): Promise<void> {
if (String(args[0]).includes('acp-snap-cwd-') && fsControl.cleanupFailure !== undefined) {
const failure = fsControl.cleanupFailure
fsControl.cleanupFailure = undefined
await actual.rm(...args)
throw failure
}
await actual.rm(...args)
},
}
})
/**
* Unit tests for the subprocess harness, driven through the REAL spawn path
* (tsx loader, temp cwd, env plumbing) against the scripted fake ACP bin in
* (mode-aware launcher, temp cwd, env plumbing) against the scripted fake ACP bin in
* ./fixtures/fake-acp-agent.ts. Each case writes a `behavior.json` next to a
* throwaway fixture path; the fake bin echoes observable facts (env, seeded
* workspace, permission outcomes) into `agent_message_chunk` text, so the
* assertions read plain `rawStdout`.
*/
const fakeAgent = fileURLToPath(new URL('./fixtures/fake-acp-agent.ts', import.meta.url))
const AGENT: AgentUnderTest = {
binScript: fileURLToPath(new URL('./fixtures/fake-acp-agent.ts', import.meta.url)),
binScript: fakeAgent,
libBinScript: fakeAgent,
// The fake bin ignores its config argv; any real path documents the shape.
configPath: fileURLToPath(new URL('./fixtures/fake-acp-agent.ts', import.meta.url)),
configPath: fakeAgent,
tsconfigPath: fileURLToPath(new URL('../../../../tsconfig.json', import.meta.url)),
}
@@ -38,6 +61,206 @@ async function scenario(behavior: object): Promise<{ dir: string; fixtureFile: s
const boot: InputStep[] = [{ op: 'initialize' }, { op: 'newSession' }]
describe('runScenario', () => {
it('surfaces an asynchronous child spawn failure through startup and close', async () => {
const { dir } = await scenario({})
const launched = launchAcpTestAgent({ agent: AGENT, cwd: join(dir, 'missing') })
let stdioClosed = false
let clientClosed = false
launched.child.once('close', () => { stdioClosed = true })
void launched.client.closed.then(
() => { clientClosed = true },
() => { clientClosed = true },
)
await expect(launched.spawned).rejects.toMatchObject({ code: 'ENOENT' })
await expect(launched.close()).rejects.toMatchObject({ code: 'ENOENT' })
expect(stdioClosed).toBe(true)
expect(clientClosed).toBe(true)
})
it('centralizes ACP boot, captures, updates, fail-closed permissions, and shutdown', { timeout: 20_000 }, async () => {
const { dir, fixtureFile } = await scenario({ permissionProbe: true, echoEnv: true, stderrNote: 'launcher stderr' })
const sessionsRoot = await mkdtemp(join(tmpdir(), 'acp-launcher-sessions-'))
tempDirs.push(sessionsRoot)
const launched = launchAcpTestAgent({
agent: AGENT,
cwd: dir,
configPath: AGENT.configPath,
env: {
DSH_SNAPSHOT: 'replay',
DSH_SNAPSHOT_FILE: fixtureFile,
DSH_SNAPSHOT_SESSIONS_ROOT: sessionsRoot,
},
})
await launched.client.initialize({ protocolVersion: PROTOCOL_VERSION, clientCapabilities: {} })
const { sessionId } = await launched.client.newSession({ cwd: dir, mcpServers: [] })
const nextChunk = launched.waitForUpdate(update => update.sessionUpdate === 'agent_message_chunk')
const predicateFailure = new Error('predicate failed')
const failedPredicate = launched.waitForUpdate(() => { throw predicateFailure })
.catch((error: unknown): unknown => error)
await launched.client.prompt({ sessionId, prompt: [{ type: 'text', text: 'go' }] })
expect(await failedPredicate).toBe(predicateFailure)
expect((await nextChunk).sessionUpdate).toBe('agent_message_chunk')
expect(launched.updates.some(update => update.sessionUpdate === 'agent_message_chunk')).toBe(true)
expect(launched.rawStdout()).toContain('permission:{\\"outcome\\":\\"cancelled\\"}')
expect(launched.stderr()).toContain('launcher stderr')
const unmatched = expect(launched.waitForUpdate(() => false)).rejects.toThrow(/update stream closed/)
await launched.close()
await unmatched
await expect(launched.waitForUpdate(() => true)).rejects.toThrow(/update stream closed/)
await launched.close('SIGKILL')
// The minimal shape needs no environment or config override.
const minimal = launchAcpTestAgent({ agent: AGENT, cwd: dir })
await minimal.client.initialize({ protocolVersion: PROTOCOL_VERSION, clientCapabilities: {} })
const childFailure = new Error('child process failed')
let exited = false
minimal.child.once('exit', () => { exited = true })
minimal.child.emit('error', childFailure)
await expect(minimal.close('SIGTERM')).rejects.toBe(childFailure)
// close rejects only after the fallback SIGKILL has produced an exit edge.
expect(exited).toBe(true)
})
it('waits for inherited stdio and buffered ACP parsing after the parent exits', { timeout: 20_000 }, async () => {
const { dir, fixtureFile } = await scenario({ lateInheritedOutput: true })
const launched = launchAcpTestAgent({
agent: AGENT,
cwd: dir,
env: { DSH_SNAPSHOT_FILE: fixtureFile },
})
await launched.client.initialize({ protocolVersion: PROTOCOL_VERSION, clientCapabilities: {} })
await launched.client.newSession({ cwd: dir, mcpServers: [] })
const lateUpdate = launched.waitForUpdate(update =>
update.sessionUpdate === 'agent_message_chunk'
&& update.content.type === 'text'
&& update.content.text === 'late inherited stdout')
await launched.close()
await expect(lateUpdate).resolves.toMatchObject({ sessionUpdate: 'agent_message_chunk' })
expect(launched.rawStdout()).toContain('late inherited stdout')
expect(launched.stderr()).toContain('late inherited stderr')
})
it('rejects promptly when fallback termination is refused', async () => {
const { dir } = await scenario({})
const launched = launchAcpTestAgent({ agent: AGENT, cwd: dir })
await launched.spawned
const childFailure = Object.assign(new Error('signal refused'), { code: 'EPERM' })
const originalKill = launched.child.kill.bind(launched.child)
const kill = vi.spyOn(launched.child, 'kill').mockReturnValue(false)
const closed = new Promise<void>(resolve => launched.child.once('close', () => { resolve() }))
try {
launched.child.emit('error', childFailure)
const rejection = await launched.close('SIGTERM').catch((error: unknown): unknown => error)
expect(rejection).toBeInstanceOf(AggregateError)
expect(rejection).toMatchObject({
message: 'ACP test agent failed and fallback termination was refused',
errors: [
childFailure,
expect.objectContaining({ message: 'Fallback SIGKILL was not accepted by the child process' }),
],
})
expect(kill).toHaveBeenNthCalledWith(1, 'SIGTERM')
expect(kill).toHaveBeenNthCalledWith(2, 'SIGKILL')
} finally {
kill.mockRestore()
originalKill('SIGKILL')
await closed
}
})
it('rejects promptly when fallback termination emits an error', async () => {
const { dir } = await scenario({})
const launched = launchAcpTestAgent({ agent: AGENT, cwd: dir })
await launched.spawned
const childFailure = Object.assign(new Error('signal refused'), { code: 'EPERM' })
const fallbackFailure = Object.assign(new Error('fallback signal refused'), { code: 'EPERM' })
const originalKill = launched.child.kill.bind(launched.child)
const kill = vi.spyOn(launched.child, 'kill').mockImplementation((signal) => {
if (signal === 'SIGKILL') queueMicrotask(() => launched.child.emit('error', fallbackFailure))
return signal === 'SIGKILL'
})
const closed = new Promise<void>(resolve => launched.child.once('close', () => { resolve() }))
try {
launched.child.emit('error', childFailure)
const rejection = await launched.close('SIGTERM').catch((error: unknown): unknown => error)
expect(rejection).toBeInstanceOf(AggregateError)
expect(rejection).toMatchObject({
message: 'ACP test agent failed and fallback termination was refused',
errors: [childFailure, fallbackFailure],
})
expect(kill).toHaveBeenNthCalledWith(1, 'SIGTERM')
expect(kill).toHaveBeenNthCalledWith(2, 'SIGKILL')
} finally {
kill.mockRestore()
originalKill('SIGKILL')
await closed
}
})
it('waits for in-flight client callbacks after the ACP stream closes', { timeout: 20_000 }, async () => {
const { dir, fixtureFile } = await scenario({ permissionProbe: true })
let releasePermission: (() => void) | undefined
const permissionReleased = new Promise<void>((resolve) => { releasePermission = resolve })
let markPermissionStarted: (() => void) | undefined
const permissionStarted = new Promise<void>((resolve) => { markPermissionStarted = resolve })
let permissionFinished = false
const launched = launchAcpTestAgent({
agent: AGENT,
cwd: dir,
env: { DSH_SNAPSHOT_FILE: fixtureFile },
async requestPermission() {
markPermissionStarted?.()
await permissionReleased
permissionFinished = true
return { outcome: { outcome: 'cancelled' } }
},
})
await launched.client.initialize({ protocolVersion: PROTOCOL_VERSION, clientCapabilities: {} })
const { sessionId } = await launched.client.newSession({ cwd: dir, mcpServers: [] })
void launched.client.prompt({ sessionId, prompt: [{ type: 'text', text: 'go' }] }).catch(() => undefined)
await permissionStarted
const childClosed = once(launched.child, 'close')
let closeSettled = false
const closing = launched.close('SIGKILL').then(() => { closeSettled = true })
await childClosed
await launched.client.closed
expect(closeSettled).toBe(false)
releasePermission?.()
await closing
expect(permissionFinished).toBe(true)
})
it('includes agent stderr when the ACP connection closes during startup', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ failOnBoot: true, stderrNote: 'fake agent requested startup failure' })
await expect(runScenario(
{ steps: [{ op: 'initialize' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/agent stderr:\nfake agent requested startup failure/)
})
it('preserves launch-resolution errors when no child process exists', async () => {
const { dir, fixtureFile } = await scenario({})
vi.stubEnv('DSH_EXAMPLE_MODE', 'lib')
try {
await expect(runScenario(
{ steps: [] },
{
agent: { ...AGENT, binScript: join(dir, 'outside-src.ts'), libBinScript: undefined },
mode: 'replay',
fixtureFile,
},
)).rejects.toThrow(/expected a "\/src\/" segment/)
} finally {
vi.unstubAllEnvs()
}
})
it('drives a full turn: initialize (terminal caps), session, prompt, permission stub, harvest', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({
permissionProbe: true,
@@ -128,6 +351,39 @@ describe('runScenario', () => {
)).rejects.toThrow(/expected the prompt to fail/)
})
it('reports scenario and cleanup failures together', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ prompt: 'respond' })
const cleanupFailure = new Error('cleanup failed')
fsControl.cleanupFailure = cleanupFailure
const failure = await runScenario(
{ steps: [...boot, { op: 'promptExpectError', text: 'fine' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
).catch((error: unknown): unknown => error)
expect(failure).toBeInstanceOf(AggregateError)
const failures = (failure as AggregateError).errors as unknown[]
expect(failures).toHaveLength(2)
expect(failures[0]).toBeInstanceOf(Error)
expect((failures[0] as Error).message).toMatch(/expected the prompt to fail/)
expect(failures[1]).toBe(cleanupFailure)
})
it('reports cleanup failure after an otherwise successful scenario', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({})
const cleanupFailure = new Error('cleanup failed')
fsControl.cleanupFailure = cleanupFailure
const failure = await runScenario(
{ steps: boot },
{ agent: AGENT, mode: 'replay', fixtureFile },
).catch((error: unknown): unknown => error)
expect(failure).toBeInstanceOf(AggregateError)
expect((failure as AggregateError).message).toBe('snapshot cleanup failed')
expect((failure as AggregateError).errors as unknown[]).toEqual([cleanupFailure])
})
it('newSessionExpectError swallows the rejection, with and without extra dirs', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ rejectExtraDirs: true })
const result = await runScenario(

164
packages/support/acp-snapshot/tests/normalize.spec.ts
View File

@@ -93,6 +93,52 @@ describe('normalizeSessionLog', () => {
expect(out).not.toContain(ctx.cwd)
})
it('scrubs random local spill paths under the snapshot cwd', () => {
const ev = JSON.stringify({
type: 'tool/result', seq: 2, time: 5,
data: {
content: [{
type: 'text',
text: `Full formatted result stored at: ${ctx.cwd}/.spill/session-c22bc3f1d2af/8a7b6c5d4e3f-bash.txt. Use read with offset/limit, or grep this path to search within it.`,
}],
},
})
const out = normalizeSessionLog(`${header({ cwd: ctx.cwd })}\n${ev}\n`, ctx)
expect(out).toContain('{{spillLocator:bash.txt}}')
expect(out).not.toContain('session-c22bc3f1d2af')
expect(out).not.toContain('8a7b6c5d4e3f')
})
it('scrubs macOS /private aliases for local spill paths', () => {
const ev = JSON.stringify({
type: 'tool/result', seq: 2, time: 5,
data: {
content: [{
type: 'text',
text: `Full formatted result stored at: /private${ctx.cwd}/.spill/session-c22bc3f1d2af/8a7b6c5d4e3f-bash.txt. Use read with offset/limit, or grep this path to search within it.`,
}],
},
})
const out = normalizeSessionLog(`${header({ cwd: ctx.cwd })}\n${ev}\n`, ctx)
expect(out).toContain('{{spillLocator:bash.txt}}')
expect(out).not.toContain('/private{{spillLocator')
})
it('scrubs fixed snapshot spill paths', () => {
const ev = JSON.stringify({
type: 'tool/result', seq: 2, time: 5,
data: {
content: [{
type: 'text',
text: 'Full formatted result stored at: /tmp/dsh-acp-snapshot-spill/session-c22bc3f1d2af/8a7b6c5d4e3f-bash.txt. Use read with offset/limit, or grep this path to search within it.',
}],
},
})
const out = normalizeSessionLog(`${header({ cwd: ctx.cwd })}\n${ev}\n`, ctx)
expect(out).toContain('{{spillLocator:bash.txt}}')
expect(out).not.toContain('/tmp/dsh-acp-snapshot-spill')
})
it('scrubs the session id in the header', () => {
const out = normalizeSessionLog(`${header({ id: ctx.sessionIds[0] })}\n`, ctx)
expect(out).toContain('{{sessionId}}')
@@ -181,89 +227,19 @@ describe('scrubRequestHeaders', () => {
expect(scrubRequestHeaders(`${headerLine}\n${odd}\n`)).toContain('"messagePrefix":"weird"')
})
it('scrubs a header-delta prefix replacement to one token per message', () => {
const delta = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: { messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'leaked opener' }] }] },
})
const out = scrubRequestHeaders(`${headerLine}\n${delta}\n`)
expect(out).toContain('"messagePrefix":["{{messagePrefix}}"]')
expect(out).not.toContain('leaked opener')
// The empty-array transition-to-absence stays a structural fact.
const toNone = JSON.stringify({ type: 'request/header-delta', seq: 9, time: 9, data: { messagePrefix: [] } })
expect(scrubRequestHeaders(`${headerLine}\n${toNone}\n`)).toContain('"messagePrefix":[]')
})
it('leaves a delta with no scrubbable payload byte-identical (config-only, or non-array shapes)', () => {
const configOnly = JSON.stringify({ type: 'request/header-delta', seq: 8, time: 9, data: { config: { model: 'm2' } } })
const oddShapes = JSON.stringify({ type: 'request/header-delta', seq: 9, time: 9, data: { system: { insert: 'not-an-array' }, tools: null } })
it('leaves malformed headers with no scrubbable payload byte-identical', () => {
const headerless = JSON.stringify({ type: 'request/header', seq: 10, time: 9, data: { reason: 'initial' } })
const nullData = JSON.stringify({ type: 'request/header', seq: 11, time: 9, data: null })
const raw = `${headerLine}\n${configOnly}\n${oddShapes}\n${headerless}\n${nullData}\n`
const raw = `${headerLine}\n${headerless}\n${nullData}\n`
expect(scrubRequestHeaders(raw)).toBe(raw)
})
it('scrubs a one-sided tools delta and passes non-object schema entries through', () => {
const addedOnly = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: { tools: { added: [null, 'weird', { name: 'x', description: 'D' }] } },
})
const out = scrubRequestHeaders(`${headerLine}\n${addedOnly}\n`)
// Non-object entries survive untouched; the object entry keeps only name.
expect(out).toContain('"added":[null,"weird",{"name":"x","description":"{{tools}}"}]')
const changedOnly = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: { tools: { changed: [{ name: 'y', parameters: {} }] } },
})
expect(scrubRequestHeaders(`${headerLine}\n${changedOnly}\n`))
.toContain('"changed":[{"name":"y","parameters":"{{tools}}"}]')
})
it('scrubs a header-delta system payload but keeps its line positions and arity', () => {
const delta = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: { system: { keepStart: 1, keepEnd: 4, insert: ['leaked prompt line', 'second line'] }, config: { model: 'm2' } },
})
const out = scrubRequestHeaders(`${headerLine}\n${delta}\n`)
// One token PER inserted line: the edit's position AND extent survive.
expect(out).toContain('"insert":["{{system}}","{{system}}"]')
expect(out).toContain('"keepStart":1')
expect(out).toContain('"keepEnd":4')
expect(out).toContain('"config":{"model":"m2"}')
expect(out).not.toContain('leaked prompt line')
expect(out).not.toContain('{{tools}}') // no tools delta → none invented
})
it('scrubs a header-delta tools payload but keeps the added/removed/changed names', () => {
const delta = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: {
tools: {
added: [{ name: 'grep', description: 'Search files.', parameters: { type: 'object' } }],
removed: ['bash_kill'],
changed: [{ name: 'read', description: 'Read v2.', parameters: { type: 'object' } }],
},
},
})
const out = scrubRequestHeaders(`${headerLine}\n${delta}\n`)
// WHICH tools changed is behavior and survives; their bulk does not.
expect(out).toContain('"added":[{"name":"grep","description":"{{tools}}","parameters":"{{tools}}"}]')
expect(out).toContain('"removed":["bash_kill"]')
expect(out).toContain('"changed":[{"name":"read","description":"{{tools}}","parameters":"{{tools}}"}]')
expect(out).not.toContain('Search files')
expect(out).not.toContain('Read v2')
})
it('passes every other line through byte-for-byte and is idempotent', () => {
const other = JSON.stringify({ type: 'assistant/chunk', seq: 4, time: 9, data: { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'hi' } } })
const delta = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: { system: { keepStart: 0, keepEnd: 0, insert: ['x'] }, tools: { added: [{ name: 't', description: 'd', parameters: {} }], removed: [], changed: [] } },
})
const raw = `${headerLine}\n${headerEvent({ config: { model: 'm' }, system: 's', tools: [] })}\n${delta}\n${other}\n`
const raw = `${headerLine}\n${headerEvent({ config: { model: 'm' }, system: 's', tools: [] })}\n${other}\n`
const once = scrubRequestHeaders(raw)
expect(once.split('\n')[0]).toBe(headerLine)
expect(once.split('\n')[3]).toBe(other)
expect(once.split('\n')[2]).toBe(other)
expect(scrubRequestHeaders(once)).toBe(once)
})
})
@@ -281,12 +257,15 @@ describe('scrubSystemPrompts', () => {
reason: 'initial',
},
})
const delta = JSON.stringify({
type: 'request/header-delta', seq: 2, time: 3,
const changed = JSON.stringify({
type: 'request/header', seq: 2, time: 3,
data: {
system: { keepStart: 1, keepEnd: 2, insert: ['new prompt line'] },
tools: { changed: [{ name: 'read', description: 'changed schema' }] },
messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'changed prefix' }] }],
header: {
system: 'new prompt',
tools: [{ name: 'read', description: 'changed schema' }],
messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'changed prefix' }] }],
},
reason: 'change',
},
})
const toolsOnly = JSON.stringify({
@@ -294,11 +273,10 @@ describe('scrubSystemPrompts', () => {
data: { header: { tools: [{ name: 'read', description: 'schema only' }] }, reason: 'resume' },
})
const out = scrubSystemPrompts(`${header}\n${delta}\n${toolsOnly}\n`)
const out = scrubSystemPrompts(`${header}\n${changed}\n${toolsOnly}\n`)
expect(out).toContain('"system":"{{system}}"')
expect(out).toContain('"insert":["{{system}}"]')
expect(out).not.toContain('full prompt')
expect(out).not.toContain('new prompt line')
expect(out).not.toContain('new prompt')
expect(out).toContain('full schema')
expect(out).toContain('full prefix')
expect(out).toContain('changed schema')
@@ -321,12 +299,15 @@ describe('scrubToolSchemas', () => {
reason: 'initial',
},
})
const delta = JSON.stringify({
type: 'request/header-delta', seq: 2, time: 3,
const changed = JSON.stringify({
type: 'request/header', seq: 2, time: 3,
data: {
system: { keepStart: 1, keepEnd: 2, insert: ['new prompt line'] },
tools: { added: [{ name: 'grep', description: 'new schema' }], changed: [{ name: 'read', description: 'changed schema' }] },
messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'changed prefix' }] }],
header: {
system: 'new prompt',
tools: [{ name: 'grep', description: 'new schema' }],
messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'changed prefix' }] }],
},
reason: 'change',
},
})
const systemOnly = JSON.stringify({
@@ -334,15 +315,12 @@ describe('scrubToolSchemas', () => {
data: { header: { system: 'prompt only' }, reason: 'resume' },
})
const out = scrubToolSchemas(`${header}\n${delta}\n${systemOnly}\n`)
expect(out).toContain('"tools":"{{tools}}"')
expect(out).toContain('"added":[{"name":"grep","description":"{{tools}}"}]')
expect(out).toContain('"changed":[{"name":"read","description":"{{tools}}"}]')
const out = scrubToolSchemas(`${header}\n${changed}\n${systemOnly}\n`)
expect(out.match(/"tools":"{{tools}}"/g)).toHaveLength(2)
expect(out).not.toContain('full schema')
expect(out).not.toContain('new schema')
expect(out).not.toContain('changed schema')
expect(out).toContain('full prompt')
expect(out).toContain('new prompt line')
expect(out).toContain('new prompt')
expect(out).toContain('full prefix')
expect(out).toContain('changed prefix')
expect(out.split('\n')[2]).toBe(systemOnly)

163
packages/support/acp-snapshot/tests/suite.spec.ts
View File

@@ -1,4 +1,4 @@
import { cpSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs'
import { cpSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
import { rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
@@ -6,18 +6,16 @@ import { fileURLToPath } from 'node:url'
import { afterAll, describe, expect, it } from 'vitest'
import { defineAcpSnapshotSuite, type HarvestedLog, type Scenario } from '../src/index.ts'
import {
childFixturePaths,
fixtureContext,
formatSystemPromptSnapshot,
headerChangeCount,
formatToolSchemasSnapshot,
headerDeltaCount,
normalizedHeaders,
normalizedSystemPromptDeltas,
normalizedSystemPrompts,
normalizedToolSchemaDeltas,
normalizedToolSchemas,
parseToolSchemasSnapshot,
refreshFixtureReplacements,
sessionFixtureNames,
restorePinnedToolSchemas,
stabilizeRefreshLog,
unknownToolCallIds,
@@ -34,9 +32,11 @@ import {
* spec once with `ACP_SNAPSHOT_SPEC_BOOTSTRAP=1`, then review and commit the resulting tree.
*/
const fakeAgent = fileURLToPath(new URL('./fixtures/fake-acp-agent.ts', import.meta.url))
const AGENT = {
binScript: fileURLToPath(new URL('./fixtures/fake-acp-agent.ts', import.meta.url)),
configPath: fileURLToPath(new URL('./fixtures/fake-acp-agent.ts', import.meta.url)),
binScript: fakeAgent,
libBinScript: fakeAgent,
configPath: fakeAgent,
tsconfigPath: fileURLToPath(new URL('../../../../tsconfig.json', import.meta.url)),
}
@@ -45,8 +45,8 @@ const RECORD_SRC = fileURLToPath(new URL('./fixtures/record-suite', import.meta.
// Replay pins explicit header classes; recording covers the default fallback.
const REPLAY_SCENARIOS: Scenario[] = [
{ name: 'pin-turn', hasModelTurn: true, recorded: true, pinsHeader: true, expectedHeaderDeltas: 1, headerClass: 'main' },
{ name: 'plain-turn', hasModelTurn: true, recorded: true, childSessions: 1, headerClass: 'main', configPath: AGENT.configPath },
{ name: 'pin-turn', hasModelTurn: true, recorded: true, pinsHeader: true, expectedHeaderChanges: 1, headerClass: 'main' },
{ name: 'plain-turn', hasModelTurn: true, recorded: true, headerClass: 'main', configPath: AGENT.configPath },
{ name: 'no-model', hasModelTurn: false, recorded: false, headerClass: 'main' },
{ name: 'blocked-log', hasModelTurn: false, comparesLog: true, recorded: false, headerClass: 'main' },
{ name: 'authored-error', hasModelTurn: true, recorded: false, overridden: true, headerClass: 'main' },
@@ -54,7 +54,7 @@ const REPLAY_SCENARIOS: Scenario[] = [
const RECORD_SCENARIOS: Scenario[] = [
{ name: 'rec-pin', hasModelTurn: true, recorded: true, pinsHeader: true },
{ name: 'rec-child', hasModelTurn: true, recorded: true, childSessions: 1 },
{ name: 'rec-child', hasModelTurn: true, recorded: true },
// recorded:false in record mode → registered but skipped (never re-recorded).
{ name: 'rec-skip', hasModelTurn: true, recorded: false, overridden: true },
]
@@ -64,7 +64,13 @@ const RECORD_SCENARIOS: Scenario[] = [
// committed record fixtures/goldens in place.
const BOOTSTRAP = process.env.ACP_SNAPSHOT_SPEC_BOOTSTRAP === '1'
const recordDir = BOOTSTRAP ? RECORD_SRC : mkdtempSync(join(tmpdir(), 'acp-snap-record-suite-'))
if (!BOOTSTRAP) cpSync(RECORD_SRC, recordDir, { recursive: true })
if (!BOOTSTRAP) {
cpSync(RECORD_SRC, recordDir, { recursive: true })
// Record mode owns its output inventory: a new scenario has no primary yet,
// while a changed child count can leave old numbered fixtures behind.
rmSync(join(recordDir, 'rec-pin', 'session.jsonl'))
writeFileSync(join(recordDir, 'rec-child', 'session.2.jsonl'), 'stale child\n')
}
const refreshDir = mkdtempSync(join(tmpdir(), 'acp-snap-refresh-suite-'))
cpSync(REPLAY_DIR, refreshDir, { recursive: true })
staleRefreshFixtures(refreshDir)
@@ -76,7 +82,7 @@ afterAll(async () => {
function staleRefreshFixtures(dir: string): void {
writeFileSync(join(dir, 'plain-turn', 'stdout.golden.jsonl'), 'stale stdout\n')
writeFileSync(join(dir, 'pin-turn', 'system-prompt.golden.md'), 'STALE PROMPT\n')
writeFileSync(join(dir, 'pin-turn', 'tool-schemas.golden.json'), '{"initial":[{"name":"stale"}],"deltas":[]}\n')
writeFileSync(join(dir, 'pin-turn', 'tool-schemas.golden.json'), '{"initial":[{"name":"stale"}],"changes":[]}\n')
const plainBehaviorFile = join(dir, 'plain-turn', 'behavior.json')
const plainBehavior = JSON.parse(readFileSync(plainBehaviorFile, 'utf8')) as Record<string, unknown>
@@ -127,7 +133,9 @@ describe('defineAcpSnapshotSuite: refresh write-back', () => {
expect(readFileSync(join(refreshDir, 'pin-turn', 'system-prompt.golden.md'), 'utf8')).toBe([
'SYS PROMPT',
'',
'<!-- request/header-delta 1: keepStart=1, keepEnd=0 -->',
'<!-- request/header change 1 -->',
'',
'SYS PROMPT',
'',
'NEW PROMPT LINE',
'',
@@ -138,6 +146,13 @@ describe('defineAcpSnapshotSuite: refresh write-back', () => {
})
})
describe('defineAcpSnapshotSuite: record inventory write-back', () => {
it('creates a missing primary fixture and prunes stale child fixtures', () => {
expect(readFileSync(join(recordDir, 'rec-pin', 'session.jsonl'), 'utf8')).toContain('"type":"session"')
expect(() => readFileSync(join(recordDir, 'rec-child', 'session.2.jsonl'), 'utf8')).toThrow()
})
})
describe('defineAcpSnapshotSuite: registration contract', () => {
it("throws when a scenario's header class has no pinning scenario", () => {
expect(() => {
@@ -177,13 +192,41 @@ describe('defineAcpSnapshotSuite: registration contract', () => {
})
})
describe('childFixturePaths', () => {
it('yields one sibling path per child, 1-based', () => {
expect(childFixturePaths('/snap/s', 2)).toEqual(['/snap/s/session.1.jsonl', '/snap/s/session.2.jsonl'])
describe('sessionFixtureNames', () => {
it('orders the primary and contiguous child fixtures while ignoring other files', () => {
expect(sessionFixtureNames([
'stdout.golden.jsonl',
'session.2.jsonl',
'session.jsonl',
'session.1.jsonl',
'input.json',
])).toEqual(['session.jsonl', 'session.1.jsonl', 'session.2.jsonl'])
})
it('yields nothing for a single-session scenario', () => {
expect(childFixturePaths('/snap/s', 0)).toEqual([])
it('accepts a primary-only scenario', () => {
expect(sessionFixtureNames(['session.jsonl'])).toEqual(['session.jsonl'])
})
it('rejects a directory without the primary fixture', () => {
expect(() => sessionFixtureNames(['session.1.jsonl'])).toThrow('missing session.jsonl')
})
it('rejects gapped child fixtures', () => {
expect(() => sessionFixtureNames(['session.jsonl', 'session.2.jsonl']))
.toThrow('expected session.1.jsonl, found session.2.jsonl')
})
it.each(['session.0.jsonl', 'session.child.jsonl', 'session.01.jsonl'])(
'rejects invalid child fixture name %s',
(name) => {
expect(() => sessionFixtureNames(['session.jsonl', name]))
.toThrow(`invalid child session fixture name: ${name}`)
},
)
it('rejects duplicate child indexes', () => {
expect(() => sessionFixtureNames(['session.jsonl', 'session.1.jsonl', 'session.1.jsonl']))
.toThrow('expected session.2.jsonl, found session.1.jsonl')
})
})
@@ -262,65 +305,41 @@ describe('normalizedToolSchemas', () => {
})
})
describe('normalizedToolSchemaDeltas', () => {
it('extracts and normalizes object-valued schema edits', () => {
const log = [
'{"type":"request/header-delta","data":{"tools":{"added":[{"name":"read","description":"work in /w"}]}}}',
'{"type":"request/header-delta","data":{"tools":null}}',
'{"type":"request/header-delta","data":{"tools":"invalid"}}',
'{"type":"request/header-delta","data":{"tools":[]}}',
'{"type":"request/header-delta","data":{"system":{"insert":[]}}}',
'{"type":"request/header","data":{"tools":{"added":[]}}}',
'',
].join('\n')
expect(normalizedToolSchemaDeltas(log, { sessionIds: [], cwd: '/w' })).toEqual([
{ added: [{ name: 'read', description: 'work in {{cwd}}' }] },
])
})
})
describe('normalizedSystemPromptDeltas', () => {
it('extracts and normalizes well-formed system edits', () => {
const log = [
'{"type":"request/header-delta","data":{"system":{"keepStart":1,"keepEnd":0,"insert":["work in /w"]}}}',
'{"type":"request/header-delta","data":{"tools":{"replace":[]}}}',
'{"type":"request/header-delta","data":{"system":{"keepStart":"1","keepEnd":0,"insert":[]}}}',
'{"type":"request/header-delta","data":{"system":{"keepStart":1,"keepEnd":0,"insert":[null]}}}',
'',
].join('\n')
expect(normalizedSystemPromptDeltas(log, { sessionIds: [], cwd: '/w' })).toEqual([
{ keepStart: 1, keepEnd: 0, insert: ['work in {{cwd}}'] },
])
})
})
describe('formatSystemPromptSnapshot', () => {
it('adds a missing terminal newline without changing an existing one', () => {
expect(formatSystemPromptSnapshot('prompt')).toBe('prompt\n')
expect(formatSystemPromptSnapshot('prompt\n')).toBe('prompt\n')
})
it('renders readable system-prompt delta sections', () => {
expect(formatSystemPromptSnapshot('prompt', [
{ keepStart: 1, keepEnd: 0, insert: ['new', 'lines'] },
])).toBe('prompt\n\n<!-- request/header-delta 1: keepStart=1, keepEnd=0 -->\n\nnew\nlines\n')
it('renders readable changed-prompt sections', () => {
expect(formatSystemPromptSnapshot('prompt', ['new\nlines']))
.toBe('prompt\n\n<!-- request/header change 1 -->\n\nnew\nlines\n')
})
it('does not double the newline of a delta insert with a trailing blank line', () => {
expect(formatSystemPromptSnapshot('prompt\n', [
{ keepStart: 2, keepEnd: 1, insert: ['tail', ''] },
])).toBe('prompt\n\n<!-- request/header-delta 1: keepStart=2, keepEnd=1 -->\n\ntail\n')
it('does not double the newline of a changed prompt', () => {
expect(formatSystemPromptSnapshot('prompt\n', ['changed\n']))
.toBe('prompt\n\n<!-- request/header change 1 -->\n\nchanged\n')
})
})
describe('headerChangeCount', () => {
it('counts changed request headers, ignoring anchors, blanks, and other lines', () => {
const change = JSON.stringify({ type: 'request/header', seq: 2, time: 9, data: { reason: 'change' } })
const anchor = JSON.stringify({ type: 'request/header', seq: 0, time: 9, data: { reason: 'initial' } })
const other = JSON.stringify({ type: 'turn/start', seq: 1, time: 9, data: {} })
expect(headerChangeCount(`${anchor}\n${other}\n\n${change}\n${change}\n`)).toBe(2)
expect(headerChangeCount(`${anchor}\n`)).toBe(0)
})
})
describe('tool-schema snapshots', () => {
const snapshot = {
initial: [{ name: 'read', description: 'Read a file.' }],
deltas: [{ added: [{ name: 'grep', description: 'Search files.' }] }],
changes: [[{ name: 'grep', description: 'Search files.' }]],
}
it('formats and parses canonical structured JSON', () => {
const formatted = formatToolSchemasSnapshot(snapshot.initial, snapshot.deltas)
const formatted = formatToolSchemasSnapshot(snapshot.initial, snapshot.changes)
expect(formatted).toBe(`${JSON.stringify(snapshot, null, 2)}\n`)
expect(parseToolSchemasSnapshot(formatted)).toEqual(snapshot)
})
@@ -329,29 +348,21 @@ describe('tool-schema snapshots', () => {
expect(() => parseToolSchemasSnapshot('null')).toThrow(/must be an object/)
expect(() => parseToolSchemasSnapshot('"invalid"')).toThrow(/must be an object/)
expect(() => parseToolSchemasSnapshot('[]')).toThrow(/must be an object/)
expect(() => parseToolSchemasSnapshot('{"initial":{},"deltas":[]}')).toThrow(/array-valued/)
expect(() => parseToolSchemasSnapshot('{"initial":[],"deltas":{}}')).toThrow(/array-valued/)
expect(() => parseToolSchemasSnapshot('{"initial":{},"changes":[]}')).toThrow(/array-valued/)
expect(() => parseToolSchemasSnapshot('{"initial":[],"changes":{}}')).toThrow(/array-valued/)
expect(() => parseToolSchemasSnapshot('{"initial":[],"changes":[{}]}')).toThrow(/array-valued/)
})
it('restores initial schemas into the pinned header token', () => {
expect(restorePinnedToolSchemas({ system: '{{system}}', tools: '{{tools}}' }, snapshot))
expect(restorePinnedToolSchemas({ system: '{{system}}', tools: '{{tools}}' }, snapshot.initial))
.toEqual({ system: '{{system}}', tools: snapshot.initial })
})
it('rejects invalid headers and a missing tool token', () => {
expect(() => restorePinnedToolSchemas(null, snapshot)).toThrow(/must be an object/)
expect(() => restorePinnedToolSchemas('invalid', snapshot)).toThrow(/must be an object/)
expect(() => restorePinnedToolSchemas([], snapshot)).toThrow(/must be an object/)
expect(() => restorePinnedToolSchemas({ tools: [] }, snapshot)).toThrow(/must equal/)
})
})
describe('headerDeltaCount', () => {
it('counts request/header-delta events, ignoring blanks and other lines', () => {
const delta = JSON.stringify({ type: 'request/header-delta', seq: 2, time: 9, data: {} })
const other = JSON.stringify({ type: 'request/header', seq: 0, time: 9, data: {} })
expect(headerDeltaCount(`${other}\n\n${delta}\n${delta}\n`)).toBe(2)
expect(headerDeltaCount(`${other}\n`)).toBe(0)
expect(() => restorePinnedToolSchemas(null, snapshot.initial)).toThrow(/must be an object/)
expect(() => restorePinnedToolSchemas('invalid', snapshot.initial)).toThrow(/must be an object/)
expect(() => restorePinnedToolSchemas([], snapshot.initial)).toThrow(/must be an object/)
expect(() => restorePinnedToolSchemas({ tools: [] }, snapshot.initial)).toThrow(/must equal/)
})
})

4
packages/support/acp-snapshot/tsconfig.json
View File

@@ -7,5 +7,7 @@
"include": [
"src"
],
"references": []
"references": [
{ "path": "../loader-smoke" }
]
}

27
packages/support/agent-loop-testkit/README.md Normal file
View File

@@ -0,0 +1,27 @@
# `@deepseek-ai/dsh-agent-loop-testkit`
Shared prerequisite mounting for tests that exercise the concrete `AgentLoop`. `mountAgentLoopTestDependencies(ctx, options?)` installs the LLM, session, system-prompt, tool, and agent services in dependency order, then returns before the loop is mounted.
The caller registers adapters and optional plugins, mounts `AgentLoop` with the configuration under test, and disposes its own Context. System-prompt and tool-registry configuration can be forwarded through `options`; the helper does not provide test defaults beyond those owned by the services. A plugin-load failure rejects the helper call, while services activated earlier in the sequence remain owned by the caller's Context.
```ts
import { Context } from 'cordis'
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
import { mountAgentLoopTestDependencies } from '@deepseek-ai/dsh-agent-loop-testkit'
const ctx = new Context()
await mountAgentLoopTestDependencies(ctx)
// Register the test adapter and any optional plugins here.
await ctx.plugin(AgentLoop, { agents: [] })
```
Tests of injection failures, partial topology, service load order, or service teardown mount their dependencies directly instead of using this helper.
## Model Experience
None, as this test-only composition helper neither drives nor modifies model requests.
## Known Limitations and Deferred Work
- **Only the mandatory prerequisite spine is shared** — adapters, optional plugins, `AgentLoop`, agents, and Context teardown remain caller-owned so scenario-specific ordering stays visible.

41
packages/support/agent-loop-testkit/package.json Normal file
View File

@@ -0,0 +1,41 @@
{
"name": "@deepseek-ai/dsh-agent-loop-testkit",
"description": "Shared prerequisite mounting for tests that exercise the concrete agent loop",
"version": "0.0.1",
"private": true,
"type": "module",
"main": "lib/index.js",
"types": "lib/types/index.d.ts",
"exports": {
".": {
"types": "./lib/types/index.d.ts",
"default": "./lib/index.js"
},
"./src/*": "./src/*",
"./package.json": "./package.json"
},
"files": [
"lib/index.js",
"lib/types/**/*.d.ts",
"lib/types/**/*.d.ts.map",
"src"
],
"license": "BSD-3-Clause",
"peerDependencies": {
"@deepseek-ai/dsh-agent": "^0.0.1",
"@deepseek-ai/dsh-llm": "^0.0.1",
"@deepseek-ai/dsh-session": "^0.0.1",
"@deepseek-ai/dsh-system-prompt": "^0.0.1",
"@deepseek-ai/dsh-tools": "^0.0.1",
"cordis": "^4.0.0-rc.7"
},
"devDependencies": {
"@deepseek-ai/dsh-agent": "workspace:^",
"@deepseek-ai/dsh-agent-loop": "workspace:^",
"@deepseek-ai/dsh-llm": "workspace:^",
"@deepseek-ai/dsh-session": "workspace:^",
"@deepseek-ai/dsh-system-prompt": "workspace:^",
"@deepseek-ai/dsh-tools": "workspace:^",
"cordis": "^4.0.0-rc.7"
}
}

46
packages/support/agent-loop-testkit/src/index.ts Normal file
View File

@@ -0,0 +1,46 @@
/**
* Shared mounting for the services required before tests load the concrete
* agent loop. The caller retains ownership of the context, loop, adapters,
* optional plugins, and teardown.
* @module @deepseek-ai/dsh-agent-loop-testkit
*/
import type { Context } from 'cordis'
import AgentRegistry from '@deepseek-ai/dsh-agent'
import LlmService from '@deepseek-ai/dsh-llm'
import SessionStore from '@deepseek-ai/dsh-session'
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
import type { Config as SystemPromptConfig } from '@deepseek-ai/dsh-system-prompt'
import ToolRegistry from '@deepseek-ai/dsh-tools'
import type { Config as ToolRegistryConfig } from '@deepseek-ai/dsh-tools'
/** Configuration forwarded to the prerequisite service plugins. */
export interface AgentLoopTestDependenciesOptions {
/** Configuration for the system-prompt registry. */
readonly systemPrompt?: SystemPromptConfig
/** Configuration for the tool registry. */
readonly tools?: ToolRegistryConfig
}
/**
* Mount the standard prerequisite services for an AgentLoop test.
*
* The function deliberately does not mount AgentLoop or register an adapter,
* so tests retain control of load order and the topology under test. The
* context owns every mounted service and remains responsible for disposal. A
* plugin-load failure rejects the promise; services activated earlier in the
* sequence remain context-owned and unwind with that context.
* @param ctx - test context that owns the mounted services.
* @param options - optional service configuration forwarded without mutation.
* @returns after every prerequisite service has activated.
*/
export async function mountAgentLoopTestDependencies(
ctx: Context,
options: AgentLoopTestDependenciesOptions = {},
): Promise<void> {
await ctx.plugin(LlmService)
await ctx.plugin(SessionStore)
await ctx.plugin(SystemPrompt, options.systemPrompt ?? {})
await ctx.plugin(ToolRegistry, options.tools ?? {})
await ctx.plugin(AgentRegistry)
}

20
packages/support/agent-loop-testkit/tests/agent-loop-testkit.spec.ts Normal file
View File

@@ -0,0 +1,20 @@
import { describe, expect, it } from 'vitest'
import { Context } from 'cordis'
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
import { renderPrompt } from '@deepseek-ai/dsh-system-prompt'
import { mountAgentLoopTestDependencies } from '../src/index.ts'
describe('dsh-agent-loop-testkit', () => {
it('mounts a configurable prerequisite spine that can activate AgentLoop', async () => {
const ctx = new Context()
await mountAgentLoopTestDependencies(ctx, {
systemPrompt: { persona: 'Test persona.' },
tools: { mode: 'native' },
})
expect(renderPrompt(await ctx.systemPrompt.assemble())).toContain('Test persona.')
await expect(ctx.plugin(AgentLoop, { agents: [] })).resolves.toBeDefined()
await ctx.fiber.dispose()
})
})

33
packages/support/agent-loop-testkit/tsconfig.json Normal file
View File

@@ -0,0 +1,33 @@
{
"extends": "../../../tsconfig.base.json",
"compilerOptions": {
"rootDir": "src",
"outDir": "lib/types"
},
"include": [
"src"
],
"references": [
{
"path": "../../../vendor/cosmokit"
},
{
"path": "../../../vendor/cordis"
},
{
"path": "../../core/agent"
},
{
"path": "../../llm/llm"
},
{
"path": "../../core/session"
},
{
"path": "../../core/system-prompt"
},
{
"path": "../../core/tools"
}
]
}

4
packages/support/invariants/README.md
View File

@@ -4,7 +4,7 @@ Runtime event-contract assertions intended for development diagnostics. This pur
The plugin has no environment guard: it is active wherever it is registered. The default [`dsh-agent-spine-demo`](../../examples/agent-spine-demo/README.md) bundle mounts it unconditionally; a custom composition can omit it when the runtime cost is undesirable. It doubles as executable documentation of the event taxonomy — the assertions *are* the contract.
Session itself owns immutable log storage in every composition: it takes one lossless JSON snapshot of each accepted event, deep-freezes that record, and exposes the log through immutable array snapshots. The invariants plugin checks the cross-record and cross-seam rules that storage immutability cannot express.
Session itself owns immutable, surface-valid log storage in every composition: it takes one lossless JSON snapshot of each candidate, validates the complete surface transition, deep-freezes the accepted record, and exposes the log through immutable array snapshots. The invariants plugin checks the remaining cross-record and cross-seam rules that Session does not own.
Session-log assertions run during Cordis `internal/dispatch`, while `Session.append()` is resolving the `session/event` callback snapshot but before it pushes the candidate into the log. A valid transition is staged by exact event identity and applied to the live trace only when that same committed event reaches the plugin's contained post-commit listener. A later internal dispatch check can therefore veto without advancing either the log or the invariant trace, while ordinary `session/event` observer failures remain observe-only.
@@ -40,7 +40,7 @@ Agent status (per agent):
Model requests (on `llm/stream`):
- **a loop-built request is exactly what the log reconstructs** — a frozen request with a live `sessionId` is rebuilt through a fresh `Session` from the prefix before its in-flight `step/start`; later content belongs to the next request, and hand-built unfrozen one-shots are excluded. Frozen messages must match that derivation, while every other field matches folded `request/header*` events. The prepended check runs before ordinary short-circuiting stream listeners, but correctness comes from the sequence boundary rather than listener timing. See the [reconstructability RFC](../../../docs/rfc/implemented/architecture/2026-07-05-reconstructable-requests.md).
- **a loop-built request is exactly what the log reconstructs** — a frozen request with a live `sessionId` (the loop-built marker; hand-built one-shots like compaction's summarize are unfrozen and skipped) must carry frozen `messages` deep-equal to the derivation over the log prefix strictly before the in-flight step's `step/start` (rebuilt through a FRESH `Session`, so the live cache cannot vouch for itself — and boundary-correct: content logged after `step/start` legitimately belongs to the next request), and every non-content field must equal the latest logged `request/header` (see [the reconstructability RFC](../../../docs/rfc/implemented/architecture/2026-07-05-reconstructable-requests.md)). Registered with `prepend: true` so a short-circuiting `llm/stream` listener (the replay adapter) cannot silence it; prepend orders it against append-registered listeners only — correctness rests on the seq-bounded rebuild, never listener timing.
On any violation it throws `InvariantError` (`code: 'INVARIANT'`).

107
packages/support/invariants/src/index.ts
View File

@@ -3,7 +3,8 @@
* turn and step nesting, scoped dispatch, status transitions, and request
* reconstruction. The plugin has no environment guard and is active wherever
* mounted, including the default `dsh-agent-spine-demo` bundle; custom compositions
* may omit it. Sessions still own event snapshots and freezing.
* may omit it. Sessions own immutable, surface-valid event storage; this plugin
* checks only relationships that event acceptance cannot express.
* @module @deepseek-ai/dsh-invariants
*/
@@ -13,7 +14,7 @@ import { assertNever, HarnessError } from '@deepseek-ai/dsh-llm'
import type { CallId, GenerateOptions } from '@deepseek-ai/dsh-llm'
import type { Agent, AgentStatus } from '@deepseek-ai/dsh-agent'
import { Session, SessionId, foldRequestHeader } from '@deepseek-ai/dsh-session'
import type { SessionEvent, SurfaceEventType } from '@deepseek-ai/dsh-session'
import type { SessionEvent } from '@deepseek-ai/dsh-session'
import { scopedSubjectResolverFor } from './scoped-events.generated.ts'
export const name = 'invariants'
@@ -48,15 +49,6 @@ interface SessionTrace {
* `step/end` — a result must arrive in the same step as its call.
*/
pendingCalls: Set<CallId>
/** Every seq seen so far — validates `sourceEventSeqs` references. */
knownSeqs: Set<number>
/**
* The seqs currently on the surface linked list, in linked-list order
* (head to tail). A replace reorders this relative to seq order (the new
* node takes the replaced range's position), so range validation is
* positional, not by seq comparison.
*/
surface: number[]
}
/** One accepted event's deferred mutation of a live session trace. */
@@ -68,12 +60,6 @@ interface SessionTraceTransition {
| { kind: 'none' }
| { kind: 'add' | 'delete'; callId: CallId }
| { kind: 'clear' }
/** The event's mutation of the derived surface order. */
surface:
| { kind: 'none' | 'append' }
| { kind: 'replace'; start: number; count: number }
/** The committed event sequence to add to the known-sequence set. */
seq: number
}
/** Assert that a step-scoped event names the currently open turn and step. */
@@ -97,73 +83,6 @@ function validateEvent(trace: SessionTrace, event: SessionEvent): SessionTraceTr
let nextTurn = trace.nextTurn
let nextStep = trace.nextStep
let pendingCalls: SessionTraceTransition['pendingCalls'] = { kind: 'none' }
let surface: SessionTraceTransition['surface'] = { kind: 'none' }
// --- Surface invariants ---
// Surface metadata (sourceEventSeqs, surfaceOp) is only valid on
// surface-eligible event types. The compiler enforces this at append()
// call sites; this runtime check catches casts and persisted data.
const SURFACE_TYPES = new Set<string>(['user/message', 'assistant/message', 'tool/result', 'context/message', 'steering/message'])
// Cast to surface-eligible event type so we can access surfaceOp and
// sourceEventSeqs (optional on SessionEvent, mandatory on SurfaceEvent).
// SurfaceEvent's mandatory surfaceOp is too strict here — we need to
// CHECK whether surface metadata is present, not assume it.
const se = event as SessionEvent<SurfaceEventType>
if (!SURFACE_TYPES.has(event.type)) {
if (se.sourceEventSeqs !== undefined) {
throw new InvariantError(`${event.type} cannot carry sourceEventSeqs (non-surface event)`)
}
if (se.surfaceOp !== undefined) {
throw new InvariantError(`${event.type} cannot carry surfaceOp (non-surface event)`)
}
}
if (se.sourceEventSeqs !== undefined) {
if (se.sourceEventSeqs.length === 0 && event.type !== 'assistant/message') {
throw new InvariantError('sourceEventSeqs must not be empty except on assistant/message')
}
const unique = new Set(se.sourceEventSeqs)
if (unique.size !== se.sourceEventSeqs.length) {
throw new InvariantError('sourceEventSeqs must not contain duplicates')
}
for (const ref of se.sourceEventSeqs) {
if (ref >= event.seq) {
throw new InvariantError(`sourceEventSeqs must reference earlier events: ${ref} >= current seq ${event.seq}`)
}
if (!trace.knownSeqs.has(ref)) {
throw new InvariantError(`sourceEventSeqs references unknown seq ${ref}`)
}
}
}
// Fold this event into the tracked surface linked list, validating the
// replace contract as we go. `append` adds a tail node; `replace` shadows a
// positional range — every shadowed node must appear in sourceEventSeqs.
if (se.surfaceOp !== undefined) {
if (se.surfaceOp === 'append') {
surface = { kind: 'append' }
} else {
const { start, end } = se.surfaceOp
const startIdx = trace.surface.indexOf(start)
if (startIdx === -1) {
throw new InvariantError(`surface replace: start seq ${start} is not on the surface`)
}
const endIdx = trace.surface.indexOf(end)
if (endIdx === -1) {
throw new InvariantError(`surface replace: end seq ${end} is not on the surface`)
}
if (startIdx > endIdx) {
throw new InvariantError(`surface replace: start seq ${start} (pos ${startIdx}) is after end seq ${end} (pos ${endIdx}) on the surface`)
}
// Every node the replace shadows (surface positions [startIdx, endIdx]
// inclusive) must appear in sourceEventSeqs — the provenance contract.
const shadowed = trace.surface.slice(startIdx, endIdx + 1)
const recorded = new Set(se.sourceEventSeqs ?? [])
const missing = shadowed.filter(seq => !recorded.has(seq))
if (missing.length > 0) {
throw new InvariantError(`surface replace: sourceEventSeqs must include every shadowed surface node; missing ${missing.join(', ')}`)
}
surface = { kind: 'replace', start: startIdx, count: shadowed.length }
}
}
// Boundary/step-scoped events have explicit cases; every OTHER event type —
// including plugin-added (merge-extensible) SessionEventMap keys — is caught
@@ -263,8 +182,6 @@ function validateEvent(trace: SessionTrace, event: SessionEvent): SessionTraceTr
return {
scalars: { lastSeq: event.seq, openTurn, openStep, nextTurn, nextStep },
pendingCalls,
surface,
seq: event.seq,
}
}
@@ -287,20 +204,6 @@ function applyTransition(trace: SessionTrace, transition: SessionTraceTransition
default:
assertNever(transition.pendingCalls, 'session trace pending-call transition')
}
switch (transition.surface.kind) {
case 'none':
break
case 'append':
trace.surface.push(transition.seq)
break
case 'replace':
trace.surface.splice(transition.surface.start, transition.surface.count, transition.seq)
break
/* v8 ignore next -- validateEvent produces this closed transition union */
default:
assertNever(transition.surface, 'session trace surface transition')
}
trace.knownSeqs.add(transition.seq)
}
/** Validate and apply one event while rebuilding an already-committed log. */
@@ -345,8 +248,6 @@ export function apply(ctx: Context): void {
nextTurn: 1,
nextStep: 1,
pendingCalls: new Set(),
knownSeqs: new Set(),
surface: [],
})
/** Build (or rebuild) a session's trace by replaying its whole log. */
@@ -445,7 +346,7 @@ export function apply(ctx: Context): void {
// the boundary (an `agent/request`-window inject) is legitimately absent
// from this request, and a current-surface comparison would false-fire.
// - header: every non-content field must equal the fold of the log's
// `request/header*` events — the loop logs the header event BEFORE
// `request/header` events — the loop logs the header event BEFORE
// dispatch, so the fold already covers this request.
//
// Registered with `prepend: true` so a short-circuiting llm/stream listener

103
packages/support/invariants/tests/invariants.spec.ts
View File

@@ -48,7 +48,7 @@ describe('session-log invariants', () => {
session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
session.append('step/start', { turn: 1, step: 1 })
session.append('assistant/chunk', { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'h' } })
session.append('assistant/message', { turn: 1, step: 1, content: [{ type: 'tool-call', id: CallId('c1'), name: 'echo', arguments: '{}' }] }, { surfaceOp: 'append' })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [{ type: 'tool-call', id: CallId('c1'), name: 'echo', arguments: '{}' }] }, { surfaceOp: 'append' })
session.append('tool/call', { turn: 1, step: 1, callId: CallId('c1'), name: 'echo', arguments: '{}' })
session.append('tool/result', { turn: 1, step: 1, callId: CallId('c1'), content: [{ type: 'text', text: 'ok' }], isError: false }, { surfaceOp: 'append' })
session.append('step/end', { turn: 1, step: 1 })
@@ -195,7 +195,7 @@ describe('session-log invariants', () => {
expect(() => {
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('step/start', { turn: 1, step: 1 })
session.append('assistant/message', { turn: 1, step: 1, content: [
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [
{ type: 'tool-call', id: CallId('crashed'), name: 'bash', arguments: '{}' },
] }, { surfaceOp: 'append' })
session.append('tool/result', {
@@ -251,10 +251,10 @@ describe('session-log invariants', () => {
expect(() => {
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('step/start', { turn: 1, step: 1 })
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append' })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append' })
session.append('step/end', { turn: 1, step: 1 })
session.append('step/start', { turn: 1, step: 2 })
session.append('assistant/message', { turn: 1, step: 2, content: [] }, { surfaceOp: 'append' })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 2, content: [] }, { surfaceOp: 'append' })
session.append('step/end', { turn: 1, step: 2 })
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
session.append('turn/start', { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } })
@@ -316,7 +316,7 @@ describe('session-log invariants', () => {
const session = ctx.sessions.create()
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('step/start', { turn: 1, step: 1 })
expect(() => session.append('assistant/message', { turn: 1, step: 2, content: [] }, { surfaceOp: 'append' }))
expect(() => session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 2, content: [] }, { surfaceOp: 'append' }))
.toThrow(/open is turn 1\/step 1/)
})
})
@@ -464,7 +464,7 @@ describe('HMR safety', () => {
})
})
describe('surface invariants', () => {
describe('surface contract under the invariants composition', () => {
it('accepts well-formed surface metadata', async () => {
const { ctx } = await setup()
const session = ctx.sessions.create()
@@ -473,7 +473,7 @@ describe('surface invariants', () => {
session.append('step/start', { turn: 1, step: 1 })
expect(() => {
session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
}).not.toThrow()
})
@@ -483,7 +483,7 @@ describe('surface invariants', () => {
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('step/start', { turn: 1, step: 1 })
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] })
// no throw — well-formed replace op
})
@@ -493,7 +493,7 @@ describe('surface invariants', () => {
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('step/start', { turn: 1, step: 1 })
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [] })
}).not.toThrow()
expect(() => {
session.append('user/message', { content: [], source: { kind: 'user' } }, { surfaceOp: 'append', sourceEventSeqs: [] })
@@ -506,7 +506,7 @@ describe('surface invariants', () => {
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1, 1] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1, 1] })
}).toThrow(/must not contain duplicates/)
})
@@ -517,19 +517,20 @@ describe('surface invariants', () => {
// The next event is seq 1. Referencing its own seq fails on "must reference
// earlier events" (the check order is: earlier first, then unknown).
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
}).toThrow(/must reference earlier/)
})
it('accepts sourceEventSeqs referencing a valid earlier event', async () => {
// Positive test: ref < current seq and ref is in knownSeqs → passes.
// Session seqs are contiguous, so every non-negative ref below the current
// seq necessarily names an existing earlier event.
const { ctx } = await setup()
const session = ctx.sessions.create()
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('step/start', { turn: 1, step: 1 })
// seqs so far: 0, 1. The next event at seq 2 references seq 1 → valid.
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
}).not.toThrow()
})
@@ -538,27 +539,10 @@ describe('surface invariants', () => {
const session = ctx.sessions.create()
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [99] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [99] })
}).toThrow(/must reference earlier/)
})
it('rejects sourceEventSeqs referencing unknown seq (gap in event log)', async () => {
// Create an impossible-through-public-API gap so seq 2 is earlier but unknown.
const { ctx } = await setup()
const session = ctx.sessions.create()
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('step/start', { turn: 1, step: 1 })
;(session as unknown as { log: unknown[] }).log.push({
type: 'assistant/chunk',
seq: 3,
time: Date.now(),
data: { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'x' } },
})
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [2] })
}).toThrow(/unknown seq 2/)
})
it('rejects a replace whose start is positioned after its end on the surface', async () => {
const { ctx } = await setup()
const session = ctx.sessions.create()
@@ -568,8 +552,8 @@ describe('surface invariants', () => {
session.append('user/message', { content: [{ type: 'text', text: 'b' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 3
// Reversed range: start seq 3 is at a later surface position than end seq 2.
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 3, end: 2 }, sourceEventSeqs: [2, 3] })
}).toThrow(/is after end seq 2 .* on the surface/)
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 3, end: 2 }, sourceEventSeqs: [2, 3] })
}).toThrow(/is after end seq 2/)
})
it('rejects a replace whose sourceEventSeqs omits a shadowed surface node', async () => {
@@ -581,7 +565,7 @@ describe('surface invariants', () => {
session.append('user/message', { content: [{ type: 'text', text: 'b' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 3
// Replace shadows surface nodes [2, 3] but records provenance for only [2].
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [{ type: 'text', text: 'sum' }] }, { surfaceOp: { op: 'replace', start: 2, end: 3 }, sourceEventSeqs: [2] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [{ type: 'text', text: 'sum' }] }, { surfaceOp: { op: 'replace', start: 2, end: 3 }, sourceEventSeqs: [2] })
}).toThrow(/must include every shadowed surface node; missing 3/)
})
@@ -593,7 +577,7 @@ describe('surface invariants', () => {
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 2
session.append('user/message', { content: [{ type: 'text', text: 'b' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 3
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [{ type: 'text', text: 'sum' }] }, { surfaceOp: { op: 'replace', start: 2, end: 3 }, sourceEventSeqs: [2, 3] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [{ type: 'text', text: 'sum' }] }, { surfaceOp: { op: 'replace', start: 2, end: 3 }, sourceEventSeqs: [2, 3] })
}).not.toThrow()
})
@@ -605,8 +589,8 @@ describe('surface invariants', () => {
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 2
// seq 1 (step/start) is a real earlier event but never entered the surface.
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 1, end: 2 }, sourceEventSeqs: [1, 2] })
}).toThrow(/start seq 1 is not on the surface/)
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 1, end: 2 }, sourceEventSeqs: [1, 2] })
}).toThrow(/start seq 1 not found in surface/)
})
it('rejects a replace naming an end seq that is not on the surface', async () => {
@@ -617,8 +601,8 @@ describe('surface invariants', () => {
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 2
// start (2) is on the surface but end (99) never entered it.
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 99 }, sourceEventSeqs: [2] })
}).toThrow(/end seq 99 is not on the surface/)
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 99 }, sourceEventSeqs: [2] })
}).toThrow(/end seq 99 not found in surface/)
})
it('rejects a replace whose range is reversed in surface position after a prior replace reordered it', async () => {
@@ -629,13 +613,13 @@ describe('surface invariants', () => {
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 2
session.append('user/message', { content: [{ type: 'text', text: 'b' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 3
// Replace node 2 (position 0) with seq 4 — surface is now [4, 3], so seq 4
// precedes seq 3 in linked-list order even though 4 > 3 numerically.
session.append('assistant/message', { turn: 1, step: 1, content: [{ type: 'text', text: 's' }] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] }) // seq 4
// precedes seq 3 in surface order even though 4 > 3 numerically.
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [{ type: 'text', text: 's' }] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] }) // seq 4
// A replace with start=3, end=4 passes the seq check (3 <= 4) but is
// reversed positionally (3 is at pos 1, 4 is at pos 0).
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 3, end: 4 }, sourceEventSeqs: [3, 4] }) // seq 5
}).toThrow(/is after end seq 4 .* on the surface/)
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 3, end: 4 }, sourceEventSeqs: [3, 4] }) // seq 5
}).toThrow(/is after end seq 4/)
})
it('accepts a replace whose start seq exceeds its end seq when the surface position order is valid', async () => {
@@ -649,9 +633,9 @@ describe('surface invariants', () => {
// head seq (4) is numerically GREATER than the tail seq (3): the surface is
// not seq-ordered. A replace spanning start=4 (pos 0) … end=3 (pos 1) is
// valid positionally and must be accepted even though start seq > end seq.
session.append('assistant/message', { turn: 1, step: 1, content: [{ type: 'text', text: 's' }] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] }) // seq 4
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [{ type: 'text', text: 's' }] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] }) // seq 4
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 4, end: 3 }, sourceEventSeqs: [4, 3] }) // seq 5
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 4, end: 3 }, sourceEventSeqs: [4, 3] }) // seq 5
}).not.toThrow()
})
@@ -663,7 +647,7 @@ describe('surface invariants', () => {
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 2
// A replace with no sourceEventSeqs records no provenance for the node it shadows.
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 2 } })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 2 } })
}).toThrow(/must include every shadowed surface node; missing 2/)
})
@@ -674,30 +658,11 @@ describe('surface invariants', () => {
{ type: 'step/start' as const, seq: 1, time: 0, data: { turn: 1, step: 1 } },
{ type: 'user/message' as const, seq: 2, time: 0, data: { content: [{ type: 'text' as const, text: 'a' }], source: { kind: 'user' as const } }, surfaceOp: 'append' as const },
{ type: 'user/message' as const, seq: 3, time: 0, data: { content: [{ type: 'text' as const, text: 'b' }], source: { kind: 'user' as const } }, surfaceOp: 'append' as const },
{ type: 'assistant/message' as const, seq: 4, time: 0, data: { turn: 1, step: 1, content: [{ type: 'text' as const, text: 'sum' }] }, surfaceOp: { op: 'replace' as const, start: 2, end: 3 }, sourceEventSeqs: [2] },
{ type: 'assistant/message' as const, seq: 4, time: 0, data: { turn: 1, step: 1, content: [{ type: 'text' as const, text: 'sum' }], provenance: { provider: 'mock', model: 'mock' } }, surfaceOp: { op: 'replace' as const, start: 2, end: 3 }, sourceEventSeqs: [2] },
]
expect(() => ctx.sessions.create(undefined, { seed: badSeed })).toThrow(/must include every shadowed surface node; missing 3/)
})
it('rejects sourceEventSeqs on a non-surface event', async () => {
const { ctx } = await setup()
const session = ctx.sessions.create()
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
// Session rejects this at its own acceptance boundary. Emit a hand-built
// record to cover the listener's defensive check for alternate producers.
const event = { type: 'turn/end', seq: 1, time: 1, data: { turn: 1, reason: { kind: 'completed' } }, sourceEventSeqs: [0] }
expect(() => { ctx.emit(scopeTarget(session, undefined), 'session/event', session, event as never) })
.toThrow(/cannot carry sourceEventSeqs/)
})
it('rejects surfaceOp on a non-surface event', async () => {
const { ctx } = await setup()
const session = ctx.sessions.create()
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
const event = { type: 'turn/end', seq: 1, time: 1, data: { turn: 1, reason: { kind: 'completed' } }, surfaceOp: 'append' }
expect(() => { ctx.emit(scopeTarget(session, undefined), 'session/event', session, event as never) })
.toThrow(/cannot carry surfaceOp/)
})
})
describe('request-reconstruction cross-check (llm/stream)', () => {
@@ -709,7 +674,7 @@ describe('request-reconstruction cross-check (llm/stream)', () => {
session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
const boundary = session.deriveMessages()
session.append('step/start', { turn: 1, step: 1 })
session.append('request/header', { header: { config: { model: 'm' } }, reason: 'initial' })
session.append('request/header', { header: { config: { provider: 'mock', model: 'm' } }, reason: 'initial' })
return { ctx, session, boundary }
}
@@ -739,7 +704,7 @@ describe('request-reconstruction cross-check (llm/stream)', () => {
it('expects the folded header\'s session prefix ahead of the derivation (prefix + derived)', async () => {
const { ctx, session, boundary } = await requestSetup()
const prefix = { role: 'user' as const, content: [{ type: 'text' as const, text: '<system-reminder>catalog</system-reminder>' }] }
session.append('request/header-delta', { messagePrefix: [prefix] })
session.append('request/header', { header: { config: { provider: 'mock', model: 'm' }, messagePrefix: [prefix] }, reason: 'change' })
// The prefixed request matches the fold…
const prefixed = Object.freeze({ model: 'm', messages: Object.freeze([prefix, ...boundary]), sessionId: session.id })
expect(() => { dispatch(ctx, prefixed) }).not.toThrow()
@@ -808,7 +773,7 @@ describe('request cross-check ordering (prepend)', () => {
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
session.append('step/start', { turn: 1, step: 1 })
session.append('request/header', { header: { config: { model: 'm' } }, reason: 'initial' })
session.append('request/header', { header: { config: { provider: 'mock', model: 'm' } }, reason: 'initial' })
const divergent = Object.freeze({
model: 'm',

14
packages/support/llm-replay/README.md
View File

@@ -1,6 +1,6 @@
# @deepseek-ai/dsh-llm-replay
A replay LLM plugin for keyless snapshot tests. It installs a single `llm/stream` waterfall listener that short-circuits the waterfall (never calls `next()`) and yields model streams reconstructed from a recorded **session JSONL** fixture — so a test can boot the real agent against a fixed model transcript with no API key.
A replay LLM plugin for keyless snapshot tests. It yields model streams reconstructed from a recorded **session JSONL** fixture, so a test can boot the real agent against a fixed model transcript with no API key. With `providers` configured it registers a replay-only adapter whose catalog is visible to clients such as ACP editors; without `providers` it installs the catch-all `llm/stream` waterfall used by tests that do not need discovery.
Its consumer is the ACP snapshot harness in `examples/acp-agent`, which loads this plugin (via `cordis.snapshot.yml`) in place of a real LLM adapter. The package exists so its derive/parse/replay logic falls under the per-file 100% coverage gate on `packages/*/src` (the same logic, while it lived under `examples/`, was outside the gate).
@@ -23,10 +23,18 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
| `file` | string | `$DSH_SNAPSHOT_FILE` | Path to the primary (parent) `session.jsonl` fixture. Required (config or env). |
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | Optional path to a `ReplayEntry[]` sidecar that replaces the PRIMARY session's derived script. |
| `childFiles` | string[] | `$DSH_SNAPSHOT_CHILD_FILES` (path-delimited) | Recorded subagent child-session logs for a nested scenario; empty for a single-session scenario. |
| `providers` | `ReplayProviderConfig[]` | — | Optional replay-only provider and model catalog. Configured routes dispatch through the replay adapter and never perform provider I/O. |
```yaml
- id: llm-replay
name: '@deepseek-ai/dsh-llm-replay'
config:
providers:
- id: deepseek
name: DeepSeek
models:
- id: deepseek-v4-flash
- id: deepseek-v4-pro
# file/overrideFile/childFiles default to $DSH_SNAPSHOT_FILE /
# $DSH_SNAPSHOT_OVERRIDE / $DSH_SNAPSHOT_CHILD_FILES, set by the snapshot
# harness per scenario.
@@ -34,11 +42,11 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
## Exports
- `installLlmReplay(ctx, config)` — install the `llm/stream` listener; returns the disposer (HMR safety). Use this in tests to drive replay without the Loader or env vars.
- `installLlmReplay(ctx, config)` — install the configured replay adapter or catch-all `llm/stream` listener; returns the disposer (HMR safety). Use this in tests to drive replay without the Loader or env vars.
- `loadSessionScripts(config)` — resolve the ordered `SessionScript[]` (primary + children) for a scenario, ready to bind to live sessions in first-call order.
- `loadReplayScript(config)` — resolve the `ReplayEntry[]` for the PRIMARY session only (sidecar override if present, else derived from the JSONL; fail-loud if the fixture is missing).
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)` — the pure helpers that turn a recorded session log into a script and read its header `id`/`createdAt`. A derived group must end in a `finish` chunk; a group without one is the fingerprint of a thrown `stream()` and must instead be expressed via an override sidecar.
- Types `ReplayEntry` / `SessionScript` / `ReplayConfig` / `Config`.
- Types `ReplayEntry` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `Config`.
## Plugin export shape

92
packages/support/llm-replay/src/index.ts
View File

@@ -10,8 +10,8 @@ import { existsSync, readFileSync } from 'node:fs'
import { delimiter as pathDelimiter } from 'node:path'
import type { Context } from 'cordis'
import type { SessionEvent } from '@deepseek-ai/dsh-session'
import type { GenerateOptions, StreamChunk } from '@deepseek-ai/dsh-llm'
import { LlmError, assertNever } from '@deepseek-ai/dsh-llm'
import type { GenerateOptions, LlmModelInfo, LlmProviderInfo, StreamChunk } from '@deepseek-ai/dsh-llm'
import { LlmAdapter, LlmError, assertNever } from '@deepseek-ai/dsh-llm'
/**
* One recorded model call. `throw` may replay prefix chunks before failing;
@@ -20,9 +20,29 @@ import { LlmError, assertNever } from '@deepseek-ai/dsh-llm'
*/
export type ReplayEntry =
| { kind: 'chunks'; chunks: StreamChunk[] }
| { kind: 'throw'; chunks: StreamChunk[]; message: string; code: string; status?: number }
| { kind: 'throw'; chunks: StreamChunk[]; message: string; code: string }
| { kind: 'hang' }
/** One model exposed by a replay-only provider catalog. */
export interface ReplayModelConfig {
/** Model id used for replay requests. */
id: string
/** Selector label; defaults to {@link id}. */
name?: string
/** Optional selector description. */
description?: string
}
/** One provider route exposed by the replay adapter. */
export interface ReplayProviderConfig {
/** Provider route used for replay requests. */
id: string
/** Selector label; defaults to {@link id}. */
name?: string
/** Advisory models exposed to clients such as ACP editors. */
models?: ReplayModelConfig[]
}
/** Resolved plugin configuration. */
export interface ReplayConfig {
/**
@@ -45,6 +65,12 @@ export interface ReplayConfig {
* for a single-session scenario.
*/
childFiles?: string[]
/**
* Optional provider catalog. When non-empty, replay registers an adapter for
* these routes; when absent or empty, it retains the catch-all waterfall used
* by tests that do not need discovery.
*/
providers?: ReplayProviderConfig[]
}
/**
@@ -203,6 +229,42 @@ export function loadSessionScripts(config: ReplayConfig): SessionScript[] {
return [primary, ...children]
}
/** Replay adapter that makes a configured provider catalog discoverable without provider I/O. */
class ReplayAdapter extends LlmAdapter {
private readonly providers: ReadonlyMap<string, ReplayProviderConfig>
constructor(
providers: readonly ReplayProviderConfig[],
private readonly replay: (options: GenerateOptions) => AsyncIterable<StreamChunk>,
) {
super()
this.providers = new Map(providers.map(provider => [provider.id, provider]))
}
override providerInfo(provider: string): LlmProviderInfo {
const configured = this.providers.get(provider)
/* v8 ignore next -- LlmService only asks about routes registered from this same map. */
if (configured === undefined) return super.providerInfo(provider)
return { id: provider, name: configured.name ?? provider }
}
override listModels(provider: string): Promise<readonly LlmModelInfo[]> {
const configured = this.providers.get(provider)
/* v8 ignore next -- LlmService only asks about routes registered from this same map. */
if (configured === undefined) return Promise.resolve([])
return Promise.resolve((configured.models ?? []).map(model => ({
provider,
id: model.id,
name: model.name ?? model.id,
...model.description === undefined ? {} : { description: model.description },
})))
}
override stream(options: GenerateOptions): AsyncIterable<StreamChunk> {
return this.replay(options)
}
}
/** Yield a recorded stream back, honoring abort like a real adapter. */
async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined): AsyncIterable<StreamChunk> {
switch (entry.kind) {
@@ -221,7 +283,7 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined)
if (signal?.aborted) throw new Error('aborted')
yield chunk
}
throw new LlmError(entry.message, entry.code, entry.status)
throw new LlmError(entry.message, entry.code)
case 'hang':
// Replay a stream that stalls until cancelled (mirrors MockAdapter): one
// chunk, then wait for abort and surface it as the consumer expects.
@@ -243,12 +305,14 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined)
/**
* Install per-session positional replay. A newly seen live session takes the
* next ordered recorded script, then advances its own cursor synchronously at
* invocation time; calls without `sessionId` share one anonymous session.
* Returns the effect disposer for HMR-safe removal.
* invocation time; calls without `sessionId` share one anonymous session. A
* non-empty provider catalog registers a routed replay adapter; otherwise a
* catch-all waterfall intercepts requests. Returns the effect disposer for
* HMR-safe removal.
*
* @param ctx - the context whose `llm/stream` waterfall the listener short-circuits.
* @param ctx - the context whose LLM service receives the replay route or waterfall.
* @param config - the resolved fixture paths (env-var defaulting is `apply`'s job).
* @returns the `ctx.on` disposer that removes the listener.
* @returns the disposer that removes the registered adapter or listener.
*/
export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void {
const scripts = loadSessionScripts(config)
@@ -258,7 +322,7 @@ export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void
const bound = new Map<string, { entries: ReplayEntry[]; cursor: number }>()
let nextScript = 0
const ANON = '\0anon\0' // the key for a call that carries no sessionId
return ctx.on('llm/stream', (options: GenerateOptions, _next) => {
const replay = (options: GenerateOptions): AsyncIterable<StreamChunk> => {
const key = options.sessionId ?? ANON
let state = bound.get(key)
let unrecorded = false
@@ -296,7 +360,12 @@ export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void
}
yield* replayEntry(entry, options.signal)
})()
})
}
const providers = config.providers ?? []
if (providers.length > 0) {
return ctx.llm.registerAdapter(providers.map(provider => provider.id), new ReplayAdapter(providers, replay))
}
return ctx.on('llm/stream', (options: GenerateOptions, _next) => replay(options))
}
export const name = 'llm-replay'
@@ -314,6 +383,8 @@ export interface Config {
* a nested-agent scenario; absent/empty for a single-session scenario.
*/
childFiles?: string[]
/** Optional replay-only provider catalog; absent or empty selects catch-all waterfall replay. */
providers?: ReplayProviderConfig[]
}
export function apply(ctx: Context, config: Config = {}): void {
@@ -329,5 +400,6 @@ export function apply(ctx: Context, config: Config = {}): void {
file,
...overrideFile !== undefined && overrideFile.length > 0 ? { overrideFile } : {},
...childFiles.length > 0 ? { childFiles } : {},
...config.providers !== undefined ? { providers: config.providers } : {},
})
}

95
packages/support/llm-replay/tests/llm-replay.spec.ts
View File

@@ -175,7 +175,7 @@ describe('loadReplayScript', () => {
it('uses the sidecar override when present, ignoring the JSONL', () => {
writeFileSync(file, sessionJsonl([]), 'utf8')
const overrideFile = join(dir, 'replay.override.json')
const override: ReplayEntry[] = [{ kind: 'throw', chunks: [], message: '401', code: 'AUTH', status: 401 }]
const override: ReplayEntry[] = [{ kind: 'throw', chunks: [], message: '401', code: 'AUTH' }]
writeFileSync(overrideFile, JSON.stringify(override), 'utf8')
expect(loadReplayScript({ file, overrideFile })).toEqual(override)
})
@@ -198,7 +198,7 @@ describe('loadReplayScript', () => {
})
})
describe('installLlmReplay (through the real waterfall)', () => {
describe('installLlmReplay (through the real LlmService)', () => {
function writeLog(...calls: StreamChunk[][]): void {
let seq = 1
const events: SessionEvent[] = []
@@ -214,7 +214,41 @@ describe('installLlmReplay (through the real waterfall)', () => {
await ctx.plugin(LlmService)
// No adapter registered for 'm' — replay must not reach it.
installLlmReplay(ctx, { file })
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
})
it('registers a replay-only provider catalog when configured', async () => {
writeLog(TEXT_CHUNKS)
const ctx = new Context()
await ctx.plugin(LlmService)
const dispose = installLlmReplay(ctx, {
file,
providers: [
{
id: 'deepseek',
name: 'DeepSeek',
models: [
{ id: 'flash' },
{ id: 'pro', name: 'Pro', description: 'Larger model' },
],
},
{ id: 'empty' },
],
})
expect(ctx.llm.listProviders()).toEqual([
{ id: 'deepseek', name: 'DeepSeek' },
{ id: 'empty', name: 'empty' },
])
await expect(ctx.llm.listModels('deepseek')).resolves.toEqual([
{ provider: 'deepseek', id: 'flash', name: 'flash' },
{ provider: 'deepseek', id: 'pro', name: 'Pro', description: 'Larger model' },
])
await expect(ctx.llm.listModels('empty')).resolves.toEqual([])
expect(await drain(ctx.llm.stream({ provider: 'deepseek', model: 'pro', messages: [] }))).toEqual(TEXT_CHUNKS)
dispose()
expect(ctx.llm.listProviders()).toEqual([])
})
it('serves the Nth call the Nth derived entry (positional)', async () => {
@@ -227,16 +261,16 @@ describe('installLlmReplay (through the real waterfall)', () => {
const ctx = new Context()
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file })
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(second)
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(second)
})
it('replays a sidecar throw-entry as an LlmError with code/status, after its prefix chunks', async () => {
it('replays a sidecar throw-entry as an LlmError with its stable code, after its prefix chunks', async () => {
writeFileSync(file, sessionJsonl([]), 'utf8')
const overrideFile = join(dir, 'replay.override.json')
const partial: StreamChunk[] = [{ type: 'block-start', index: 0, blockType: 'text' }]
writeFileSync(overrideFile, JSON.stringify([
{ kind: 'throw', chunks: partial, message: 'unauthorized', code: 'AUTH', status: 401 },
{ kind: 'throw', chunks: partial, message: 'unauthorized', code: 'AUTH' },
]), 'utf8')
const ctx = new Context()
await ctx.plugin(LlmService)
@@ -244,8 +278,8 @@ describe('installLlmReplay (through the real waterfall)', () => {
const seen: StreamChunk[] = []
await expect((async () => {
for await (const c of ctx.llm.stream({ model: 'm', messages: [] })) seen.push(c)
})()).rejects.toMatchObject({ message: 'unauthorized', code: 'AUTH', status: 401 })
for await (const c of ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })) seen.push(c)
})()).rejects.toMatchObject({ message: 'unauthorized', code: 'AUTH' })
expect(seen).toEqual(partial)
})
@@ -258,7 +292,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
installLlmReplay(ctx, { file, overrideFile })
const controller = new AbortController()
const iterator = ctx.llm.stream({ model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
const iterator = ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
// Deterministically consume the two pre-hang chunks (no sleep), then abort
// and assert the next pull rejects — event-driven, per the no-sleeps rule.
expect((await iterator.next()).value).toMatchObject({ type: 'block-start' })
@@ -272,8 +306,8 @@ describe('installLlmReplay (through the real waterfall)', () => {
const ctx = new Context()
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file })
await drain(ctx.llm.stream({ model: 'm', messages: [] }))
await expect(drain(ctx.llm.stream({ model: 'm', messages: [] }))).rejects.toThrow(/exhausted/)
await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).rejects.toThrow(/exhausted/)
})
it('aborts mid-replay when the signal is already set', async () => {
@@ -283,7 +317,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
installLlmReplay(ctx, { file })
const controller = new AbortController()
controller.abort()
await expect(drain(ctx.llm.stream({ model: 'm', messages: [], signal: controller.signal })))
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal })))
.rejects.toThrow('aborted')
})
@@ -305,11 +339,11 @@ describe('installLlmReplay (through the real waterfall)', () => {
}, { inject: ['llm'] }))
// While installed, replay short-circuits to the derived fixture ('hi').
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
await fiber.dispose()
// After dispose the listener is gone; the call reaches the real adapter.
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] })))
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })))
.toEqual([{ type: 'finish', reason: { kind: 'stop' } }])
})
@@ -321,7 +355,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
const ctx = new Context()
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file, overrideFile })
await expect(drain(ctx.llm.stream({ model: 'm', messages: [] })))
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })))
.rejects.toThrow(/llm-replay replay entry/)
})
@@ -333,7 +367,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file, overrideFile })
const controller = new AbortController()
const iterator = ctx.llm.stream({ model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
const iterator = ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
// Consume the two pre-hang chunks, then start the third pull so the generator
// is parked inside the await (signal NOT yet aborted — exercises the
// addEventListener('abort') registration), and only THEN abort.
@@ -350,7 +384,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
const overrideFile = join(dir, 'replay.override.json')
const partial: StreamChunk[] = [{ type: 'block-start', index: 0, blockType: 'text' }]
writeFileSync(overrideFile, JSON.stringify([
{ kind: 'throw', chunks: partial, message: 'unauthorized', code: 'AUTH', status: 401 },
{ kind: 'throw', chunks: partial, message: 'unauthorized', code: 'AUTH' },
]), 'utf8')
const ctx = new Context()
await ctx.plugin(LlmService)
@@ -359,7 +393,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
controller.abort()
// Already aborted: the throw-entry's prefix loop surfaces 'aborted' before
// it can reach the recorded LlmError.
await expect(drain(ctx.llm.stream({ model: 'm', messages: [], signal: controller.signal })))
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal })))
.rejects.toThrow('aborted')
})
@@ -373,7 +407,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
const controller = new AbortController()
controller.abort()
// The two pre-hang chunks still flow; the abort surfaces at the await.
const iterator = ctx.llm.stream({ model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
const iterator = ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
await iterator.next()
await iterator.next()
await expect(iterator.next()).rejects.toThrow('aborted')
@@ -503,7 +537,7 @@ describe('installLlmReplay (per-session keying)', () => {
]
const live = (id: string): GenerateOptions =>
({ model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
({ provider: 'm', model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
it('routes each live session to its own script by FIRST-CALL order', async () => {
const parentFile = writeSession('session.jsonl', { id: 'rec-parent', createdAt: 100 }, [TEXT_CHUNKS])
@@ -540,7 +574,7 @@ describe('installLlmReplay (per-session keying)', () => {
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file: parentFile })
// No sessionId at all — the legacy single-session path.
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
})
it('fails loud when more distinct live sessions call than were recorded', async () => {
@@ -574,12 +608,13 @@ describe('apply (the plugin entry)', () => {
expect(inject).toEqual(['llm'])
})
it('installs replay from an explicit config.file', async () => {
it('installs replay and its catalog from explicit config', async () => {
writeFileSync(file, sessionJsonl(TEXT_CHUNKS.map((c, i) => chunkEvent(i + 1, 1, 1, c))), 'utf8')
const ctx = new Context()
await ctx.plugin(LlmService)
apply(ctx, { file })
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
apply(ctx, { file, providers: [{ id: 'm', models: [{ id: 'm' }] }] })
expect(ctx.llm.listProviders()).toEqual([{ id: 'm', name: 'm' }])
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
})
it('falls back to $DSH_SNAPSHOT_FILE / $DSH_SNAPSHOT_OVERRIDE when config is empty', async () => {
@@ -591,7 +626,7 @@ describe('apply (the plugin entry)', () => {
const ctx = new Context()
await ctx.plugin(LlmService)
apply(ctx)
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
})
it('uses only the file when no override path is configured or in the env', async () => {
@@ -601,7 +636,7 @@ describe('apply (the plugin entry)', () => {
const ctx = new Context()
await ctx.plugin(LlmService)
apply(ctx)
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
})
it('throws when no fixture path is given by config or env', async () => {
@@ -631,7 +666,7 @@ describe('apply (the plugin entry)', () => {
await ctx.plugin(LlmService)
apply(ctx, { file, childFiles: [childFile] })
const live = (id: string): GenerateOptions =>
({ model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
({ provider: 'm', model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
expect(await drain(ctx.llm.stream(live('A')))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream(live('B')))).toEqual(childSecond)
})
@@ -651,7 +686,7 @@ describe('apply (the plugin entry)', () => {
await ctx.plugin(LlmService)
apply(ctx)
const live = (id: string): GenerateOptions =>
({ model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
({ provider: 'm', model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
expect(await drain(ctx.llm.stream(live('A')))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream(live('B')))).toEqual(childChunks)
})
@@ -663,6 +698,6 @@ describe('apply (the plugin entry)', () => {
const ctx = new Context()
await ctx.plugin(LlmService)
apply(ctx)
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
})
})

8
packages/support/loader-smoke/README.md
View File

@@ -1,10 +1,10 @@
# `@deepseek-ai/dsh-loader-smoke`
Shared subprocess harness for keyless example smokes that boot the real stdio-agent bin and a real `cordis.yml` through the Cordis Loader. A test supplies absolute bin/config/tsconfig paths, optional environment overrides, and stdin lines; `runLoaderSmoke` owns the isolated cwd, DSH homes, tsx path resolution, 30-second process deadline, captured diagnostics, forced kill, EOF, and cleanup.
Shared subprocess harness for tests that boot an app and `cordis.yml` through the Cordis Loader. `resolveExampleLaunch` selects local `src` mode (tsx and root tsconfig paths) or CI `lib` mode (plain Node and package exports) from an explicit mode or `DSH_EXAMPLE_MODE`.
Successful runs return stdout and stderr only after a zero exit. Non-zero exits and deadlines reject with both captured streams. `LOADER_SMOKE_TEST_TIMEOUT_MS` leaves Vitest enough room for the process-owned diagnostic timeout to fire first.
`runLoaderSmoke` owns the isolated cwd, DSH homes, stdin, diagnostics, deadline, termination, and cleanup. It returns both streams after a zero exit and rejects with both streams on failure.
This is support-tier test infrastructure, not product API. The consumers are the Loader-path smokes under `examples/{echo-agent,coding-agent,cordis-agent}`.
This is support-tier test infrastructure, not product API.
## Model Experience
@@ -12,6 +12,6 @@ None, as this test-only harness boots example processes and inspects their strea
## Known Limitations and Deferred Work
- **Only the unbuilt tsx/Loader path is exercised** — built-bin artifacts remain the responsibility of their separate e2e smokes.
- **Built mode requires a prior build** — the config must also resolve every named package upward through `examples/node_modules`.
- **Captured stdout and stderr are unbounded** — a runaway child can consume memory until the deadline kills it.
- **Timeout kills only the direct child** — a process tree spawned by a faulty fixture can outlive the smoke and needs external cleanup.

148
packages/support/loader-smoke/src/index.ts
View File

@@ -2,6 +2,14 @@
* Shared subprocess harness for keyless example smokes that boot a real
* `cordis.yml` through the stdio-agent bin and Cordis Loader.
*
* It also owns the mode-aware launch resolver every example subprocess harness shares
* ({@link resolveExampleLaunch}): booting an example bin from TypeScript source under `tsx` (the
* zero-build dev path, resolving `@deepseek-ai/dsh-*` / `@cordisjs/*` through the tsconfig `paths`
* map) or from built `lib/` under plain Node (resolving bare packages through real `exports`, as an
* installed consumer does, while Node type-strips relative example-local TypeScript plugins).
* Consolidating that spawn glue here retires the copies in the ACP snapshot harness and the example
* e2e drivers (the `TODO(acp-test-harness)`).
*
* @module @deepseek-ai/dsh-loader-smoke
*/
@@ -9,26 +17,125 @@ import { spawn } from 'node:child_process'
import { mkdtemp, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { fileURLToPath } from 'node:url'
const DEFAULT_PROCESS_TIMEOUT_MS = 30_000
const TSX_LOADER = fileURLToPath(import.meta.resolve('tsx'))
/** Vitest deadline that leaves room for the subprocess-owned 30-second diagnostic timeout. */
export const LOADER_SMOKE_TEST_TIMEOUT_MS = DEFAULT_PROCESS_TIMEOUT_MS + 15_000
/** Which artifact an example bin is booted from: unbuilt `src` via tsx, or built `lib` via plain Node. */
export type ExampleMode = 'src' | 'lib'
/** Environment variable selecting the mode; CI and pre-push set it to `lib`, dev leaves it unset (`src`). */
export const EXAMPLE_MODE_ENV = 'DSH_EXAMPLE_MODE'
/**
* Parse an {@link ExampleMode} from a raw string, defaulting to `src` when absent so an unset
* environment reproduces the dev/tsx behavior. Throws on any other value rather than silently
* falling back, so a typo in a gate's env fails loud.
* @param raw - the raw value; defaults to `process.env.DSH_EXAMPLE_MODE`.
* @returns the validated mode.
*/
export function resolveExampleMode(raw: string | undefined = process.env[EXAMPLE_MODE_ENV]): ExampleMode {
switch (raw) {
case undefined:
case '':
case 'src':
return 'src'
case 'lib':
return 'lib'
default:
throw new Error(`${EXAMPLE_MODE_ENV} must be 'src' or 'lib', got ${JSON.stringify(raw)}.`)
}
}
/** Inputs to {@link resolveExampleLaunch}. */
export interface ExampleLaunchOptions {
/** Absolute path to the example bin's TypeScript source entry (`<pkg>/src/bin.ts`); the `lib` bin is derived from it. */
readonly srcBin: string
/** Explicit plain-Node entry for `lib` mode; test fixtures may point this at Node-type-strippable TypeScript. */
readonly libBin?: string | undefined
/** Arguments passed after the bin — the config, positional (`[configPath]`) or flagged (`['--config', configPath]`). */
readonly configArgs?: readonly string[]
/** The mode to launch in; defaults to {@link resolveExampleMode} of the environment. */
readonly mode?: ExampleMode
/** Absolute repo tsconfig whose `paths` map resolves unbuilt workspace imports. Required in `src` mode, ignored in `lib`. */
readonly tsconfigPath?: string
/** Prepend `--expose-internals` (the Cordis Loader's bare-plugin resolver needs it for some bins); defaults to `false`. */
readonly exposeInternals?: boolean
/** Extra environment entries the mode-specific ones layer over; the caller then merges the result over `process.env`. */
readonly env?: NodeJS.ProcessEnv
}
/** The resolved spawn: `spawn(command, args, { env: { ...process.env, ...env } })`. */
export interface ExampleLaunch {
/** The executable to spawn — always the current Node binary. */
readonly command: string
/** Node flags, the resolved bin, then the caller's `configArgs`. */
readonly args: string[]
/** Mode-specific environment (`TSX_TSCONFIG_PATH` in `src`, nothing added in `lib`) layered over the caller's `env`. */
readonly env: NodeJS.ProcessEnv
}
/** Derive the built-lib bin (`<pkg>/lib/<name>.js`) from a source bin (`<pkg>/src/<name>.ts`). */
function toLibBin(srcBin: string): string {
const markerLength = '/src/'.length
const cut = Math.max(srcBin.lastIndexOf('/src/'), srcBin.lastIndexOf('\\src\\'))
if (cut === -1) {
throw new Error(`resolveExampleLaunch: expected a "/src/" segment or Windows equivalent in bin path ${JSON.stringify(srcBin)}.`)
}
const separator = srcBin.slice(cut, cut + 1)
const tail = srcBin.slice(cut + markerLength).replace(/\.ts$/, '.js')
return `${srcBin.slice(0, cut)}${separator}lib${separator}${tail}`
}
/**
* Resolve how to spawn an example bin in the selected mode.
*
* `src` yields `node [--expose-internals] --import <tsx> <srcBin> <configArgs>` with `TSX_TSCONFIG_PATH`
* set so the tsconfig `paths` map resolves workspace imports to source. `lib` yields
* `node [--expose-internals] <libBin> <configArgs>` under plain Node with no tsx and no paths map, so
* bare package plugins resolve through real package `exports` into built `lib/`; relative example-local
* TypeScript plugins remain source files loaded through Node's built-in type stripping. Bare resolution
* requires the config to live below a workspace that declares its `cordis.yml` package dependencies.
*
* @param options - the source bin, config arguments, mode, and environment.
* @returns the command, argument vector, and mode-specific environment to spawn with.
*/
export function resolveExampleLaunch(options: ExampleLaunchOptions): ExampleLaunch {
const mode = options.mode ?? resolveExampleMode()
const configArgs = options.configArgs ?? []
const flags = options.exposeInternals === true ? ['--expose-internals'] : []
const env: NodeJS.ProcessEnv = { ...options.env }
if (mode === 'src') {
if (options.tsconfigPath === undefined) {
throw new Error("resolveExampleLaunch: 'src' mode needs tsconfigPath for the workspace paths map.")
}
const tsxLoader = import.meta.resolve('tsx')
env.TSX_TSCONFIG_PATH = options.tsconfigPath
return { command: process.execPath, args: [...flags, '--import', tsxLoader, options.srcBin, ...configArgs], env }
}
return { command: process.execPath, args: [...flags, options.libBin ?? toLibBin(options.srcBin), ...configArgs], env }
}
/** Inputs that vary between real-Loader example smokes. */
export interface LoaderSmokeOptions {
/** Human-readable example name used in failure diagnostics. */
readonly label: string
/** Prefix for the isolated temporary process cwd. */
readonly tempDirPrefix: string
/** Absolute stdio-agent bin path. */
/** Absolute stdio-agent bin SOURCE path (`<pkg>/src/bin.ts`); the `lib` bin is derived from it. */
readonly binScript: string
/** Explicit plain-Node entry for `lib` mode; intended for test fixtures outside a package `src/` tree. */
readonly libBinScript?: string | undefined
/** Absolute real Loader config path. */
readonly configPath: string
/** Absolute repo tsconfig path used for unbuilt workspace-package resolution. */
/** Absolute repo tsconfig path used for unbuilt workspace-package resolution (required in `src` mode). */
readonly tsconfigPath: string
/** Boot from source via tsx (`src`) or built lib via plain Node (`lib`); defaults to the environment's mode. */
readonly mode?: ExampleMode
/** Environment overrides layered over the parent and isolated DSH homes. */
readonly env?: Readonly<NodeJS.ProcessEnv>
/** Lines written to stdin before EOF; omitted means immediate EOF. */
@@ -48,30 +155,29 @@ export interface LoaderSmokeResult {
/**
* Boot one real Loader tree from an isolated cwd, write the requested stdin
* script, close stdin, and await a clean exit. The helper owns process kill and
* temp-directory cleanup on every outcome.
* @param options - example paths, environment, stdin, and diagnostic identity.
* temp-directory cleanup on every outcome, and picks src/lib via {@link resolveExampleLaunch}.
* @param options - example paths, mode, environment, stdin, and diagnostic identity.
* @returns captured stdout and stderr after a zero exit.
*/
export async function runLoaderSmoke(options: LoaderSmokeOptions): Promise<LoaderSmokeResult> {
const cwd = await mkdtemp(join(tmpdir(), options.tempDirPrefix))
const processTimeoutMs = options.processTimeoutMs ?? DEFAULT_PROCESS_TIMEOUT_MS
const launch = resolveExampleLaunch({
srcBin: options.binScript,
libBin: options.libBinScript,
configArgs: [options.configPath],
...options.mode !== undefined ? { mode: options.mode } : {},
tsconfigPath: options.tsconfigPath,
exposeInternals: true,
env: { DSH_HOME: join(cwd, '.dsh'), DSH_AGENTS_HOME: join(cwd, '.agents'), ...options.env },
})
try {
return await new Promise((resolve, reject) => {
const child = spawn(
process.execPath,
['--expose-internals', '--import', TSX_LOADER, options.binScript, options.configPath],
{
cwd,
env: {
...process.env,
DSH_HOME: join(cwd, '.dsh'),
DSH_AGENTS_HOME: join(cwd, '.agents'),
...options.env,
TSX_TSCONFIG_PATH: options.tsconfigPath,
},
stdio: ['pipe', 'pipe', 'pipe'],
},
)
const child = spawn(launch.command, launch.args, {
cwd,
env: { ...process.env, ...launch.env },
stdio: ['pipe', 'pipe', 'pipe'],
})
let stdout = ''
let stderr = ''
let deferredFailure: Error | undefined

111
packages/support/loader-smoke/tests/example-launch.spec.ts Normal file
View File

@@ -0,0 +1,111 @@
import { afterEach, describe, expect, it } from 'vitest'
import {
EXAMPLE_MODE_ENV,
resolveExampleLaunch,
resolveExampleMode,
} from '@deepseek-ai/dsh-loader-smoke'
const SRC_BIN = '/repo/packages/examples/stdio-demo/src/bin.ts'
const TSCONFIG = '/repo/tsconfig.json'
const originalMode = process.env[EXAMPLE_MODE_ENV]
afterEach(() => {
if (originalMode === undefined) Reflect.deleteProperty(process.env, EXAMPLE_MODE_ENV)
else process.env[EXAMPLE_MODE_ENV] = originalMode
})
describe('resolveExampleMode', () => {
it('defaults absent/empty/src to src', () => {
Reflect.deleteProperty(process.env, EXAMPLE_MODE_ENV)
expect(resolveExampleMode()).toBe('src')
expect(resolveExampleMode('')).toBe('src')
expect(resolveExampleMode('src')).toBe('src')
})
it('accepts lib', () => {
expect(resolveExampleMode('lib')).toBe('lib')
})
it('throws on any other value', () => {
expect(() => resolveExampleMode('prod')).toThrow(/must be 'src' or 'lib'/)
})
it('reads the environment when no argument is given', () => {
process.env[EXAMPLE_MODE_ENV] = 'lib'
expect(resolveExampleMode()).toBe('lib')
Reflect.deleteProperty(process.env, EXAMPLE_MODE_ENV)
expect(resolveExampleMode()).toBe('src')
})
})
describe('resolveExampleLaunch', () => {
it('src mode: --import tsx on the source bin with the tsconfig paths env', () => {
const { command, args, env } = resolveExampleLaunch({
srcBin: SRC_BIN,
configArgs: ['./cordis.yml'],
mode: 'src',
tsconfigPath: TSCONFIG,
})
expect(command).toBe(process.execPath)
expect(args).toContain('--import')
expect(args).toContain(SRC_BIN)
expect(args[args.length - 1]).toBe('./cordis.yml')
expect(args).not.toContain('--expose-internals')
expect(env.TSX_TSCONFIG_PATH).toBe(TSCONFIG)
})
it('src mode: throws without a tsconfig path', () => {
expect(() => resolveExampleLaunch({ srcBin: SRC_BIN, mode: 'src' })).toThrow(/needs tsconfigPath/)
})
it('lib mode: plain node on the derived lib bin, no tsx and no paths env', () => {
const { args, env } = resolveExampleLaunch({
srcBin: SRC_BIN,
configArgs: ['--config', './cordis.yml'],
mode: 'lib',
env: { DSH_HOME: '/tmp/home' },
})
expect(args).not.toContain('--import')
expect(args).toContain('/repo/packages/examples/stdio-demo/lib/bin.js')
expect(args.slice(-2)).toEqual(['--config', './cordis.yml'])
expect(env.TSX_TSCONFIG_PATH).toBeUndefined()
expect(env.DSH_HOME).toBe('/tmp/home')
})
it('lib mode: uses an explicit plain-Node bin when provided', () => {
const fixture = '/repo/fixture.ts'
const { args } = resolveExampleLaunch({ srcBin: fixture, libBin: fixture, mode: 'lib' })
expect(args).toContain(fixture)
})
it('prepends --expose-internals when requested', () => {
const { args } = resolveExampleLaunch({ srcBin: SRC_BIN, mode: 'lib', exposeInternals: true })
expect(args[0]).toBe('--expose-internals')
})
it('lib mode: rewrites only the last /src/ segment', () => {
const { args } = resolveExampleLaunch({
srcBin: '/repo/src/packages/examples/acp-demo/src/bin.ts',
mode: 'lib',
})
expect(args).toContain('/repo/src/packages/examples/acp-demo/lib/bin.js')
})
it('lib mode: derives the built bin from a Windows source path', () => {
const { args } = resolveExampleLaunch({
srcBin: String.raw`D:\repo\src\packages\examples\acp-demo\src\bin.ts`,
mode: 'lib',
})
expect(args).toContain(String.raw`D:\repo\src\packages\examples\acp-demo\lib\bin.js`)
})
it('lib mode: throws when the bin has no /src/ segment', () => {
expect(() => resolveExampleLaunch({ srcBin: '/repo/lib/bin.js', mode: 'lib' })).toThrow(/"\/src\/" segment/)
})
it('defaults the mode from the environment', () => {
process.env[EXAMPLE_MODE_ENV] = 'lib'
const { args } = resolveExampleLaunch({ srcBin: SRC_BIN })
expect(args).toContain('/repo/packages/examples/stdio-demo/lib/bin.js')
})
})

3
packages/support/loader-smoke/tests/loader-smoke.spec.ts
View File

@@ -16,6 +16,7 @@ describe('runLoaderSmoke', () => {
binScript: fixture('success'),
configPath,
tsconfigPath,
mode: 'src',
env: { LOADER_SMOKE_MARKER: 'present' },
stdinLines: ['one', 'two'],
})
@@ -43,6 +44,7 @@ describe('runLoaderSmoke', () => {
label: 'failure fixture',
tempDirPrefix: 'loader-smoke-fail-',
binScript: fixture('fail'),
libBinScript: fixture('fail'),
configPath,
tsconfigPath,
})).rejects.toThrow('failure fixture exited 7. stdout:\n\nstderr:\nfixture failed')
@@ -53,6 +55,7 @@ describe('runLoaderSmoke', () => {
label: 'hanging fixture',
tempDirPrefix: 'loader-smoke-hang-',
binScript: fixture('hang'),
libBinScript: fixture('hang'),
configPath,
tsconfigPath,
processTimeoutMs: 100,

2
packages/support/subagent-mock/package.json
View File

@@ -24,6 +24,7 @@
"peerDependencies": {
"@deepseek-ai/dsh-agent": "^0.0.1",
"@deepseek-ai/dsh-llm": "^0.0.1",
"@deepseek-ai/dsh-session": "^0.0.1",
"@deepseek-ai/dsh-subagent": "^0.0.1",
"cordis": "^4.0.0-rc.7"
},
@@ -33,6 +34,7 @@
"devDependencies": {
"@deepseek-ai/dsh-agent": "workspace:^",
"@deepseek-ai/dsh-llm": "workspace:^",
"@deepseek-ai/dsh-session": "workspace:^",
"@deepseek-ai/dsh-subagent": "workspace:^",
"@cordisjs/plugin-loader": "^1.0.0-rc.5",
"cordis": "^4.0.0-rc.7"

5
packages/support/subagent-mock/src/index.ts
View File

@@ -7,8 +7,8 @@
import type { Context } from 'cordis'
import z from 'schemastery'
import { AgentId } from '@deepseek-ai/dsh-agent'
import type { ContentBlock } from '@deepseek-ai/dsh-llm'
import { SessionId } from '@deepseek-ai/dsh-session'
import type {
SubagentCapabilities,
SubagentProvider,
@@ -54,7 +54,7 @@ class MockSubagentProvider implements SubagentProvider {
// A deterministic child id derived from the parent — no clock/random (both
// banned in deterministic paths here, and unnecessary for a scripted run).
const id = AgentId(`mock-subagent:${this.name}:${request.parent.id}`)
const id = SessionId(`mock-subagent:${this.name}:${request.parent.id}`)
const resultFor = (): SubagentResult => ({
output,
@@ -69,6 +69,7 @@ class MockSubagentProvider implements SubagentProvider {
})
return {
id,
localAgent: undefined,
result,
dispose(): Promise<void> {
flags.cancelled = true

6
packages/support/subagent-mock/tests/subagent-mock.spec.ts
View File

@@ -1,13 +1,15 @@
import { describe, expect, it } from 'vitest'
import { Context } from 'cordis'
import Loader from '@cordisjs/plugin-loader'
import { AgentId, type Agent } from '@deepseek-ai/dsh-agent'
import { type Agent } from '@deepseek-ai/dsh-agent'
import SubagentService, { type SubagentStartRequest } from '@deepseek-ai/dsh-subagent'
import * as mock from '../src/index.ts'
import { SessionId } from '@deepseek-ai/dsh-session'
/** A minimal parent — the mock provider only reads `parent.id`. */
function fakeParent(id = 'parent-1'): Agent {
return { id: AgentId(id) } as unknown as Agent
return { id: SessionId(id) } as unknown as Agent
}
function baseRequest(over: Partial<SubagentStartRequest> = {}): SubagentStartRequest {