Merge origin/master into worktree/explicit-turn-signal
This commit is contained in:
Yichen Jiang
@@ -4,11 +4,10 @@ Packages that exist to serve development, testing, and the examples rather than
|
||||
|
||||
| Package | Role | ctx key |
|
||||
|---|---|---|
|
||||
| `acp-snapshot/` | ACP snapshot suite kit: subprocess scenario harness + golden normalizers + the `defineAcpSnapshotSuite` factory | (library — imported by example `*.snapshot.ts` suites) |
|
||||
| `acp-snapshot/` | ACP test kit: shared subprocess/client launcher + snapshot harness, normalizers, and suite factory | (library — imported by ACP e2e and `*.snapshot.ts` suites) |
|
||||
| `agent-loop-testkit/` | Shared prerequisite mounting for tests that exercise the concrete agent loop | (library — imported by AgentLoop integration tests) |
|
||||
| `invariants/` | Runtime event-contract assertions for development diagnostics | (listens on `session/*`, `agent/*`) |
|
||||
| `loader-smoke/` | Shared real-Loader subprocess harness for keyless example smokes | (library — imported by example e2e suites) |
|
||||
| `llm-replay/` | Record/replay adapter: short-circuits `llm/stream` from a recorded session JSONL (keyless snapshot tests) | (listens on `llm/stream`) |
|
||||
| `subagent-mock/` | Scripted `SubagentProvider` for deterministic seam/tool tests | (registers on `ctx.subagents`) |
|
||||
|
||||
`invariants` is development support but has no environment guard: it runs wherever registered, and the default `dsh-agent-spine-demo` bundle mounts it unconditionally. `agent-loop-testkit` centralizes the mandatory service spine for hand-built AgentLoop tests without owning their loop or scenario. `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate. `acp-snapshot` carries the snapshot tier's harness/normalizer/suite machinery, while `loader-smoke` owns the parallel stdio/Loader process boundary used by keyless example e2e suites. `subagent-mock` exercises the real `ctx.subagents` load path without a model or child agent. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.
|
||||
`invariants` is development support but has no environment guard: it runs wherever registered, and the default `dsh-agent-spine-demo` bundle mounts it unconditionally. `agent-loop-testkit` centralizes the mandatory service spine for hand-built AgentLoop tests without owning their loop or scenario. `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate. `acp-snapshot` carries the ACP subprocess/client boundary plus the snapshot harness, normalizers, and suite machinery, while `loader-smoke` owns the parallel real-Loader launch boundary used by keyless example e2e suites. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.
|
||||
|
||||
@@ -2,11 +2,12 @@
|
||||
|
||||
The ACP snapshot suite kit: the shared machinery behind the keyless snapshot tier (`pnpm run test:snapshot`, [testing policy](../../../docs/testing.md)). An example gets a full snapshot suite from a scenario table plus a fixtures directory; every compare/guard mechanic lives here, under the per-file coverage gate, instead of being copied per example.
|
||||
|
||||
Three layers, importable separately:
|
||||
Four layers, importable separately:
|
||||
|
||||
- **`runScenario` (harness)** — boots the real agent bin as a subprocess via tsx (unbuilt, Loader path), drives it over ACP JSON-RPC stdio from a deterministic `input.json` script, tees raw stdout for the golden + purity check, and harvests every persisted session JSONL (parent + subagent children, primary-first) after a graceful stdin-EOF shutdown. Parameterized by `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath` — absolute paths; the subprocess cwd is a temp dir outside the repo). Startup failures preserve captured agent stderr in the rejected diagnostic.
|
||||
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header RFC](../../../docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
|
||||
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, the per-header-class pin (`system-prompt.golden.md` plus `tool-schemas.golden.json`) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt/schema-scrubbed, non-pinning fixtures fully header-scrubbed). Must be called at vitest collection time.
|
||||
- **`launchAcpTestAgent` (launcher)** — boots an unbuilt ACP agent from a temp cwd, pins tsx to the repo tsconfig, connects the SDK client over a raw-byte stdout tee, collects session updates and stderr, surfaces asynchronous spawn failures through its startup lifecycle, fails closed on unhandled permission requests, and owns graceful or signalled shutdown. Shutdown waits for process exit, inherited stdio closure, and ACP parser exhaustion before resolving or propagating a child error, so captures are complete and callers can remove owned paths after either outcome. Snapshot and ordinary e2e suites share this process boundary; a test supplies only agent paths, cwd, environment overrides, and any permission policy.
|
||||
- **`runScenario` (harness)** — boots the real agent bin as a subprocess via tsx (unbuilt, Loader path), drives it over ACP JSON-RPC stdio from a deterministic `input.json` script, tees raw stdout for the expected-output and purity checks, and harvests every persisted session JSONL (parent + subagent children, primary-first) after a graceful stdin-EOF shutdown. Parameterized by `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath` — absolute paths; the subprocess cwd is a temp dir outside the repo). Startup failures preserve captured agent stderr in the rejected diagnostic.
|
||||
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header Agent Note](../../../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
|
||||
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario expected-output and re-persisted-log comparisons, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, the per-header-class pin (`system-prompt.expected.md` plus `tool-schemas.expected.json`) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt/schema-scrubbed, non-pinning fixtures fully header-scrubbed). Each scenario directory's `session.jsonl` plus contiguous `session.<n>.jsonl` siblings are the ordered primary/child inventory; the scenario table does not duplicate their count. Must be called at vitest collection time.
|
||||
|
||||
A consuming `*.snapshot.ts` is the scenario table plus one factory call:
|
||||
|
||||
@@ -35,17 +36,21 @@ defineAcpSnapshotSuite({
|
||||
})
|
||||
```
|
||||
|
||||
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode and filesystem scenarios are templates. Each pinning directory stores the normalized full prompt sequence in generated `system-prompt.golden.md` and the corresponding full tool-schema sequence in generated `tool-schemas.golden.json`; `session.jsonl` stores `"system":"{{system}}","tools":"{{tools}}"` while retaining config, reason, and any model-visible prefix. A pin with legitimate mid-run header changes declares `expectedHeaderChanges`, which fixes the length of both sidecar sequences.
|
||||
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode and filesystem scenarios are templates. Each pinning directory stores the normalized full prompt sequence in generated `system-prompt.expected.md` and the corresponding full tool-schema sequence in generated `tool-schemas.expected.json`; `session.jsonl` stores `"system":"{{system}}","tools":"{{tools}}"` while retaining config, reason, and any model-visible prefix. A pin with legitimate mid-run header changes declares `expectedHeaderChanges`, which fixes the length of both sidecar sequences.
|
||||
|
||||
Examples use a `cordis.snapshot.yml` overlay with [`dsh-llm-replay`](../llm-replay/README.md). Recording calls the live model and updates model fixtures; keyless refresh replays those fixtures and updates derived stdout, session-log, prompt, and tool-schema snapshots. See the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
|
||||
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config Agent Note](../../../.agents/notes/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log expected outputs, and each pin's prompt and tool-schema sidecars from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md).
|
||||
|
||||
`suite.ts` imports Vitest, so use this package only inside a Vitest run. The ACP-specific script queues permission answers by stable option kind and maps them to current option ids; a missing answer cancels, while an unavailable kind fails the scenario after cancelling the agent request. It can also set session config options or assert that unknown ids and values are rejected in the transcript.
|
||||
Constraints: `suite.ts` imports vitest, so the package entry is importable only inside a vitest run (the launcher, harness, and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug). Session config options are scriptable too: the `setConfigOption` step switches a knob over `session/set_config_option`, and `setConfigOptionExpectError` asserts the bridge rejects an unknown id or out-of-vocabulary value (the error frame stays in the transcript).
|
||||
|
||||
## Model Experience
|
||||
|
||||
None, as this test-only harness records, normalizes, and compares ACP transcripts without changing the agent's assembled model request.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
None; this package neither assembles nor sends a provider request.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **Session harvest is JSONL-only** — `runScenario` collects persisted `.jsonl` logs, so an example composed over the SQLite persistence backend has no snapshot path.
|
||||
- **Session harvest requires raw JSONL mode** — `runScenario` collects persisted `.jsonl` logs, so snapshot configs set `persistenceCompression: 'none'`; compressed JSONL and SQLite compositions have no snapshot-harvest path.
|
||||
- **The subprocess boots the unbuilt tsx/Loader path only** — the built-bin artifact is guarded by the separate `built-bin` e2e smokes, never by this tier.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-acp-snapshot",
|
||||
"description": "ACP snapshot suite kit: real-subprocess scenario harness, golden normalizers, and the suite factory behind the keyless snapshot tier",
|
||||
"description": "ACP test kit: shared subprocess launcher, snapshot scenario harness, expected-output normalizers, and suite factory",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
|
||||
@@ -1,59 +1,46 @@
|
||||
/**
|
||||
* Shared ACP snapshot subprocess harness. It boots the real agent bin through the Cordis
|
||||
* loader, drives deterministic ACP JSON-RPC over stdio, captures protocol-pure stdout, and
|
||||
* harvests persisted session logs after graceful shutdown. Normalization stays in
|
||||
* `normalize.ts`; suite registration stays in `suite.ts`.
|
||||
* Shared subprocess harness for ACP snapshot suites. A library module driven by
|
||||
* the suite factory in ./suite.ts (and directly by harness-level specs); each
|
||||
* example's `*.snapshot.ts` names its own agent-under-test paths.
|
||||
*
|
||||
* It boots the REAL agent bin subprocess via the cordis Loader (so the
|
||||
* export-shape bug class stays guarded — see docs/postmortem/0001), drives it
|
||||
* over real ACP JSON-RPC stdio with a deterministic input script, tees raw
|
||||
* stdout (for the expected-output and purity checks) into an SDK `ClientSideConnection`,
|
||||
* and — in record mode — harvests the persisted session JSONL after a graceful
|
||||
* shutdown flush. The pure normalizers in ./normalize.ts turn the captured
|
||||
* stdout frames and the session-log events into stable, snapshot-able text.
|
||||
*
|
||||
* See .agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-acp-snapshot/harness
|
||||
*/
|
||||
|
||||
import { spawn, type ChildProcessWithoutNullStreams } from 'node:child_process'
|
||||
import { cp, mkdtemp, readFile, readdir, rm } from 'node:fs/promises'
|
||||
import { existsSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join, delimiter } from 'node:path'
|
||||
import { Readable, Writable } from 'node:stream'
|
||||
import {
|
||||
ClientSideConnection,
|
||||
ndJsonStream,
|
||||
PROTOCOL_VERSION,
|
||||
type Agent as AcpAgent,
|
||||
type Client,
|
||||
type RequestPermissionRequest,
|
||||
type RequestPermissionResponse,
|
||||
type SessionNotification,
|
||||
} from '@agentclientprotocol/sdk'
|
||||
import { resolveExampleLaunch } from '@deepseek-ai/dsh-loader-smoke'
|
||||
import { launchAcpTestAgent, type AgentUnderTest, type LaunchedAcpTestAgent } from './launcher.ts'
|
||||
|
||||
export type { AgentUnderTest } from './launcher.ts'
|
||||
|
||||
/**
|
||||
* The agent composition a scenario runs against: which bin to boot and which
|
||||
* leaf config it loads. All paths are ABSOLUTE — the subprocess cwd is a temp
|
||||
* dir outside the repo, so relative resolution would miss; a suite resolves
|
||||
* them from its own `import.meta.url`.
|
||||
*/
|
||||
export interface AgentUnderTest {
|
||||
/** The agent bin's SOURCE entry (e.g. `packages/examples/acp-demo/src/bin.ts`); the `lib` bin is derived from it. */
|
||||
binScript: string
|
||||
/** Explicit plain-Node entry for `lib` mode; intended for test fixtures outside a package `src/` tree. */
|
||||
libBinScript?: string | undefined
|
||||
/**
|
||||
* The example's live `cordis.yml`. Under `DSH_SNAPSHOT=replay` the bin swaps
|
||||
* it for the sibling `cordis.snapshot.yml` (the keyless replay overlay), so
|
||||
* one path serves both modes.
|
||||
*/
|
||||
configPath: string
|
||||
/**
|
||||
* The repo-root tsconfig whose `paths` map resolves the unbuilt workspace
|
||||
* imports in `src` mode (passed to the child as `TSX_TSCONFIG_PATH`). Ignored
|
||||
* in `lib` mode, where the example resolves plugins through real `exports`.
|
||||
*/
|
||||
tsconfigPath: string
|
||||
}
|
||||
|
||||
/**
|
||||
* One step of a scenario's deterministic input script (`input.json`). The harness interprets
|
||||
* these in order. `newSession` captures the server-issued (random) session id into a
|
||||
* `{{sessionId}}` variable that later steps reference. `promptAndCancel` sends without awaiting,
|
||||
* waits for the first streamed message, then cancels, making transcript order deterministic.
|
||||
* One step of a scenario's deterministic input script (`input.json`). The
|
||||
* harness interprets these in order. `newSession` captures the server-issued
|
||||
* (random) session id into a `{{sessionId}}` variable that later steps
|
||||
* reference, since a committed file cannot know the id in advance.
|
||||
*
|
||||
* `promptAndCancel` starts a prompt without awaiting completion, waits until
|
||||
* the client observes the selected update (`agent_message_chunk` by default),
|
||||
* then cancels and awaits completion. A named `waitForToolCallUpdate` keeps the
|
||||
* step open for a terminal tool update that may follow the prompt response.
|
||||
*/
|
||||
export type InputStep =
|
||||
| { op: 'initialize'; terminalOutput?: boolean }
|
||||
@@ -61,7 +48,12 @@ export type InputStep =
|
||||
| { op: 'newSessionExpectError'; additionalDirectories?: string[] }
|
||||
| { op: 'prompt'; text: string }
|
||||
| { op: 'promptExpectError'; text: string }
|
||||
| { op: 'promptAndCancel'; text: string }
|
||||
| {
|
||||
op: 'promptAndCancel'
|
||||
text: string
|
||||
afterUpdate?: 'agent_message_chunk' | 'tool_call'
|
||||
waitForToolCallUpdate?: string
|
||||
}
|
||||
| { op: 'cancel' }
|
||||
| { op: 'setConfigOption'; configId: string; value: string }
|
||||
| { op: 'setConfigOptionExpectError'; configId: string; value: string }
|
||||
@@ -70,9 +62,16 @@ export type InputStep =
|
||||
export interface InputScript {
|
||||
steps: InputStep[]
|
||||
/**
|
||||
* FIFO permission answers selected by stable option kind; the harness maps each kind to the
|
||||
* agent-issued option id. Exhaustion cancels, while a kind the agent did not offer fails the
|
||||
* scenario.
|
||||
* Ordered answers for the agent's `session/request_permission` round-trips,
|
||||
* consumed FIFO — the Nth request gets the Nth answer. Each answer selects
|
||||
* by option KIND: option ids are agent-issued randoms a committed script
|
||||
* cannot know, while kinds are the ACP-stable vocabulary, so the client maps
|
||||
* kind → the offered `optionId` at answer time. A request beyond the queue
|
||||
* (or with no queue at all) is answered `cancelled` — the stub behavior a
|
||||
* scenario without approvals relies on. A scripted kind the request does
|
||||
* not offer REJECTS the run: the scenario scripted an impossible click,
|
||||
* and {@link runScenario} throws once the in-flight step settles (the
|
||||
* agent itself just sees `cancelled`, so it cannot absorb the bug).
|
||||
*/
|
||||
permissionAnswers?: PermissionAnswer[]
|
||||
}
|
||||
@@ -163,95 +162,49 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
|
||||
const cwd = await mkdtemp(join(tmpdir(), 'acp-snap-cwd-'))
|
||||
const sessionsRoot = await mkdtemp(join(tmpdir(), 'acp-snap-sessions-'))
|
||||
// Fixed path length: spill-policy budgets the preview against the REAL path
|
||||
// before stdout normalization, so tmpdir() length differences churn goldens.
|
||||
// before stdout normalization, so tmpdir() length differences churn expected outputs.
|
||||
const spillRoot = '/tmp/dsh-acp-snapshot-spill'
|
||||
// Everything past the temp-dir creation runs under a try/finally that always
|
||||
// removes both dirs — so a failure in workspace seeding, spawn, or any step
|
||||
// never leaks them (the "e2e tests own their resources" rule).
|
||||
let child: ChildProcessWithoutNullStreams | undefined
|
||||
// Everything past the temp-dir creation is followed by failure-safe cleanup,
|
||||
// so a failure in workspace seeding, spawn, or any step never leaks resources.
|
||||
let launched: LaunchedAcpTestAgent | undefined
|
||||
let sessionId: string | undefined
|
||||
let sessionLogs: HarvestedLog[] = []
|
||||
const rawBuffers: Buffer[] = []
|
||||
const stderrChunks: string[] = []
|
||||
try {
|
||||
const outcome = await (async (): Promise<RunResult> => {
|
||||
// Seed the workspace if the scenario ships one (a file the agent reads/edits).
|
||||
// Copied into the temp cwd so the agent's bash tools see it; the expected outputs
|
||||
// normalize the cwd, so the seeded paths stay stable across runs.
|
||||
if (opts.workspaceDir !== undefined && existsSync(opts.workspaceDir)) {
|
||||
await cp(opts.workspaceDir, cwd, { recursive: true })
|
||||
}
|
||||
// Boot the agent in the environment's mode (DSH_EXAMPLE_MODE): `src` runs the
|
||||
// source bin under tsx with the paths map; `lib` runs the built bin under plain
|
||||
// Node, resolving plugins through the example's workspace node_modules → lib.
|
||||
const launch = resolveExampleLaunch({
|
||||
srcBin: opts.agent.binScript,
|
||||
libBin: opts.agent.libBinScript,
|
||||
configArgs: ['--config', opts.configPath ?? opts.agent.configPath],
|
||||
tsconfigPath: opts.agent.tsconfigPath,
|
||||
env: {
|
||||
DSH_SNAPSHOT: opts.mode,
|
||||
DSH_SNAPSHOT_FILE: opts.fixtureFile,
|
||||
DSH_SNAPSHOT_SESSIONS_ROOT: sessionsRoot,
|
||||
DSH_SNAPSHOT_SPILL_ROOT: spillRoot,
|
||||
DSH_HOME: join(cwd, '.dsh'),
|
||||
DSH_AGENTS_HOME: join(cwd, '.agents'),
|
||||
...opts.overrideFile !== undefined ? { DSH_SNAPSHOT_OVERRIDE: opts.overrideFile } : {},
|
||||
...opts.childFiles !== undefined && opts.childFiles.length > 0
|
||||
? { DSH_SNAPSHOT_CHILD_FILES: opts.childFiles.join(delimiter) }
|
||||
: {},
|
||||
},
|
||||
})
|
||||
|
||||
child = spawn(
|
||||
launch.command,
|
||||
launch.args,
|
||||
{ cwd, env: { ...process.env, ...launch.env }, stdio: ['pipe', 'pipe', 'pipe'] },
|
||||
)
|
||||
|
||||
child.stderr.setEncoding('utf8')
|
||||
child.stderr.on('data', (c: string) => stderrChunks.push(c))
|
||||
|
||||
// Tee the same raw bytes to the golden and SDK client. Decode once at the end so a UTF-8
|
||||
// sequence split across stream chunks cannot corrupt the transcript.
|
||||
const passthrough = new Readable({ read() {} })
|
||||
child.stdout.on('data', (buf: Buffer) => {
|
||||
rawBuffers.push(buf)
|
||||
passthrough.push(buf)
|
||||
})
|
||||
child.stdout.on('end', () => passthrough.push(null))
|
||||
|
||||
const stream = ndJsonStream(
|
||||
Writable.toWeb(child.stdin) as WritableStream<Uint8Array>,
|
||||
Readable.toWeb(passthrough) as ReadableStream<Uint8Array>,
|
||||
)
|
||||
// Watcher so a step can block until the client OBSERVES a particular
|
||||
// session/update — used by promptAndCancel to pin frame order (send cancel
|
||||
// only after the streamed agent_message_chunk has arrived, so those frames
|
||||
// deterministically precede the cancelled prompt response).
|
||||
const updateWaiters: { match: (u: SessionNotification['update']) => boolean; resolve: () => void }[] = []
|
||||
const waitForUpdate = (match: (u: SessionNotification['update']) => boolean): Promise<void> =>
|
||||
new Promise<void>(resolve => updateWaiters.push({ match, resolve }))
|
||||
const env: NodeJS.ProcessEnv = {
|
||||
DSH_SNAPSHOT: opts.mode,
|
||||
DSH_SNAPSHOT_FILE: opts.fixtureFile,
|
||||
DSH_SNAPSHOT_SESSIONS_ROOT: sessionsRoot,
|
||||
DSH_SNAPSHOT_SPILL_ROOT: spillRoot,
|
||||
DSH_HOME: join(cwd, '.dsh'),
|
||||
DSH_AGENTS_HOME: join(cwd, '.agents'),
|
||||
...opts.overrideFile !== undefined ? { DSH_SNAPSHOT_OVERRIDE: opts.overrideFile } : {},
|
||||
...opts.childFiles !== undefined && opts.childFiles.length > 0
|
||||
? { DSH_SNAPSHOT_CHILD_FILES: opts.childFiles.join(delimiter) }
|
||||
: {},
|
||||
}
|
||||
|
||||
// Permission answers are consumed FIFO across the whole run; exhaustion
|
||||
// falls back to `cancelled` so approval-free scenarios keep the plain stub.
|
||||
const permissionQueue = [...input.permissionAnswers ?? []]
|
||||
// A callback throw would become only an RPC error the agent could absorb. Record an
|
||||
// impossible permission choice here, answer cancelled, and fail the outer scenario.
|
||||
// A scenario bug detected inside a client callback (a scripted permission
|
||||
// kind the agent never offered). It cannot fail the run from in there: a
|
||||
// callback throw only becomes a JSON-RPC error RESPONSE to the agent, and
|
||||
// a tolerant agent treats that as a denial and carries on — the run (or
|
||||
// worse, a record) would absorb the impossible click silently. So the
|
||||
// callback answers `cancelled` (a well-defined path for the agent),
|
||||
// captures the error here, and the step loop fails the run on it.
|
||||
let scriptError: Error | undefined
|
||||
const makeClient = (_agent: AcpAgent): Client => ({
|
||||
sessionUpdate(params: SessionNotification): Promise<void> {
|
||||
for (let i = updateWaiters.length - 1; i >= 0; i--) {
|
||||
const waiter = updateWaiters[i]
|
||||
// The index is always in-bounds (i only decreases; splice removes at
|
||||
// i, so lower entries stay valid); the guard satisfies
|
||||
// noUncheckedIndexedAccess.
|
||||
/* v8 ignore next 1 -- unreachable in-bounds guard, see above */
|
||||
if (waiter === undefined) continue
|
||||
if (waiter.match(params.update)) {
|
||||
updateWaiters.splice(i, 1)
|
||||
waiter.resolve()
|
||||
}
|
||||
}
|
||||
return Promise.resolve()
|
||||
},
|
||||
launched = launchAcpTestAgent({
|
||||
agent: opts.agent,
|
||||
cwd,
|
||||
...opts.configPath !== undefined ? { configPath: opts.configPath } : {},
|
||||
env,
|
||||
requestPermission(params: RequestPermissionRequest): Promise<RequestPermissionResponse> {
|
||||
const answer = permissionQueue.shift()
|
||||
if (answer === undefined) return Promise.resolve({ outcome: { outcome: 'cancelled' } })
|
||||
@@ -269,10 +222,12 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
|
||||
return Promise.resolve({ outcome: { outcome: 'selected', optionId: option.optionId } })
|
||||
},
|
||||
})
|
||||
const client = new ClientSideConnection(makeClient, stream)
|
||||
const active = launched
|
||||
await active.spawned
|
||||
const { client } = active
|
||||
|
||||
for (const step of input.steps) {
|
||||
await runStep(client, step, cwd, waitForUpdate, () => sessionId, (id) => { sessionId = id })
|
||||
await runStep(client, step, cwd, match => active.waitForUpdate(match), () => sessionId, (id) => { sessionId = id })
|
||||
// A permission exchange happens while a step's request is in flight, so
|
||||
// by the time the step settles any script bug it exposed is captured —
|
||||
// fail the run HERE, as a harness error, rather than hoping the agent's
|
||||
@@ -281,35 +236,57 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
|
||||
}
|
||||
// Done driving: close stdin so the server disposes gracefully (flushing
|
||||
// persistence) and exits. Then await exit so the harvested log is complete.
|
||||
child.stdin.end()
|
||||
await waitForExit(child)
|
||||
await active.close()
|
||||
// Harvest EVERY persisted log (parent + any subagent children) while the
|
||||
// temp dirs still exist, ordered primary-first.
|
||||
sessionLogs = await harvestSessionLogs(sessionsRoot)
|
||||
} catch (error: unknown) {
|
||||
const stderr = stderrChunks.join('')
|
||||
if (stderr === '') throw error
|
||||
throw new Error(`snapshot-harness: scenario failed: ${String(error)}\nagent stderr:\n${stderr}`, { cause: error })
|
||||
} finally {
|
||||
// Failure-safe teardown: kill a still-running child and drop the temp dirs
|
||||
// even if seeding/spawn/a step/harvest threw, so a flaky run never leaks a
|
||||
// process or dir. `child` is undefined only if spawn itself threw.
|
||||
if (child !== undefined && child.exitCode === null && child.signalCode === null) {
|
||||
child.kill('SIGKILL')
|
||||
await waitForExit(child)
|
||||
return {
|
||||
rawStdout: launched.rawStdout(),
|
||||
stderr: launched.stderr(),
|
||||
cwd,
|
||||
...sessionId !== undefined ? { sessionId } : {},
|
||||
sessionLogs,
|
||||
}
|
||||
await rm(cwd, { recursive: true, force: true })
|
||||
await rm(sessionsRoot, { recursive: true, force: true })
|
||||
await rm(spillRoot, { recursive: true, force: true })
|
||||
}
|
||||
})().then(
|
||||
value => ({ status: 'fulfilled', value } as const),
|
||||
(error: unknown) => {
|
||||
const stderr = launched?.stderr() ?? ''
|
||||
return {
|
||||
status: 'rejected',
|
||||
error: stderr === ''
|
||||
? error
|
||||
: new Error(`snapshot-harness: scenario failed: ${String(error)}\nagent stderr:\n${stderr}`, { cause: error }),
|
||||
} as const
|
||||
},
|
||||
)
|
||||
|
||||
return {
|
||||
rawStdout: Buffer.concat(rawBuffers).toString('utf8'),
|
||||
stderr: stderrChunks.join(''),
|
||||
cwd,
|
||||
...sessionId !== undefined ? { sessionId } : {},
|
||||
sessionLogs,
|
||||
// Failure-safe teardown: wait for a still-running child, then attempt every
|
||||
// owned-path removal even when an earlier cleanup rejects. Report every
|
||||
// teardown failure alongside a scenario failure so neither orthogonal
|
||||
// outcome hides the other.
|
||||
const cleanupResults: PromiseSettledResult<unknown>[] = []
|
||||
const cleanup = async (action: () => Promise<unknown>): Promise<void> => {
|
||||
cleanupResults.push(...await Promise.allSettled([action()]))
|
||||
}
|
||||
/* v8 ignore next 1 -- launch itself can only throw on a defensive synchronous spawn API failure */
|
||||
await cleanup(() => launched?.close('SIGKILL') ?? Promise.resolve())
|
||||
await cleanup(() => rm(cwd, { recursive: true, force: true }))
|
||||
await cleanup(() => rm(sessionsRoot, { recursive: true, force: true }))
|
||||
await cleanup(() => rm(spillRoot, { recursive: true, force: true }))
|
||||
|
||||
const cleanupFailures = cleanupResults
|
||||
.filter((result): result is PromiseRejectedResult => result.status === 'rejected')
|
||||
.map(result => result.reason as unknown)
|
||||
if (cleanupFailures.length > 0) {
|
||||
throw new AggregateError(
|
||||
outcome.status === 'rejected' ? [outcome.error, ...cleanupFailures] : cleanupFailures,
|
||||
outcome.status === 'rejected'
|
||||
? 'snapshot scenario and cleanup failed'
|
||||
: 'snapshot cleanup failed',
|
||||
)
|
||||
}
|
||||
if (outcome.status === 'rejected') throw outcome.error
|
||||
return outcome.value
|
||||
}
|
||||
|
||||
/** Drive one input step over the client connection. */
|
||||
@@ -317,7 +294,7 @@ async function runStep(
|
||||
client: ClientSideConnection,
|
||||
step: InputStep,
|
||||
cwd: string,
|
||||
waitForUpdate: (match: (u: SessionNotification['update']) => boolean) => Promise<void>,
|
||||
waitForUpdate: (match: (u: SessionNotification['update']) => boolean) => Promise<SessionNotification['update']>,
|
||||
getSessionId: () => string | undefined,
|
||||
setSessionId: (id: string) => void,
|
||||
): Promise<void> {
|
||||
@@ -334,8 +311,10 @@ async function runStep(
|
||||
return
|
||||
}
|
||||
case 'newSessionExpectError': {
|
||||
// The bridge rejects a session/new that widens the workspace scope (non-empty
|
||||
// additionalDirectories / mcpServers — unimplemented).
|
||||
// The bridge rejects a session/new that widens the workspace scope
|
||||
// (non-empty additionalDirectories / mcpServers — unimplemented). The SDK
|
||||
// surfaces that as a rejected RPC; swallow it so the run completes and the
|
||||
// error frame is captured in the transcript.
|
||||
await client.newSession({
|
||||
cwd,
|
||||
mcpServers: [],
|
||||
@@ -355,8 +334,10 @@ async function runStep(
|
||||
case 'promptExpectError': {
|
||||
const sessionId = getSessionId()
|
||||
if (sessionId === undefined) throw new Error('snapshot-harness: promptExpectError before newSession')
|
||||
// The model fails this turn (a recorded provider error), so the bridge answers the prompt
|
||||
// with a JSON-RPC error and the SDK rejects.
|
||||
// The model fails this turn (a recorded provider error), so the bridge
|
||||
// answers the prompt with a JSON-RPC error and the SDK rejects. That
|
||||
// rejection IS the expected editor experience — swallow it so the run
|
||||
// completes and the stdout transcript (the error frame) is captured.
|
||||
await client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
|
||||
.then(() => { throw new Error('snapshot-harness: expected the prompt to fail but it succeeded') },
|
||||
() => { /* expected: the turn failed and the bridge returned an error */ })
|
||||
@@ -365,12 +346,19 @@ async function runStep(
|
||||
case 'promptAndCancel': {
|
||||
const sessionId = getSessionId()
|
||||
if (sessionId === undefined) throw new Error('snapshot-harness: promptAndCancel before newSession')
|
||||
// A hang fixture never resolves alone. Wait for its streamed chunk before cancellation
|
||||
// so updates deterministically precede the cancelled prompt response.
|
||||
// Dispatch without awaiting because the fixture does not settle on its
|
||||
// own. Waiting for the selected update pins it before cancellation and
|
||||
// the cancelled prompt response in the transcript.
|
||||
const promptDone = client.prompt({ sessionId, prompt: [{ type: 'text', text: step.text }] })
|
||||
await waitForUpdate(u => u.sessionUpdate === 'agent_message_chunk')
|
||||
const afterUpdate = step.afterUpdate ?? 'agent_message_chunk'
|
||||
await waitForUpdate(u => u.sessionUpdate === afterUpdate)
|
||||
// Arm this before cancellation so a fast tool drain cannot outrun the waiter.
|
||||
const toolCallUpdateDone = step.waitForToolCallUpdate === undefined
|
||||
? undefined
|
||||
: waitForUpdate(u => u.sessionUpdate === 'tool_call_update' && u.toolCallId === step.waitForToolCallUpdate)
|
||||
await client.cancel({ sessionId })
|
||||
await promptDone
|
||||
if (toolCallUpdateDone !== undefined) await toolCallUpdateDone
|
||||
return
|
||||
}
|
||||
case 'cancel': {
|
||||
@@ -402,26 +390,16 @@ async function runStep(
|
||||
}
|
||||
}
|
||||
|
||||
/** Resolve once the child process exits (any code/signal). */
|
||||
function waitForExit(child: ChildProcessWithoutNullStreams): Promise<void> {
|
||||
// Race guard: both call sites run within one synchronous frame of
|
||||
// stdin.end()/kill(), so the exit event cannot have been delivered yet;
|
||||
// kept for any future caller that awaits in between.
|
||||
/* v8 ignore next 1 -- unreachable race guard, see above */
|
||||
if (child.exitCode !== null || child.signalCode !== null) return Promise.resolve()
|
||||
return new Promise<void>(resolve => child.once('exit', () => { resolve() }))
|
||||
}
|
||||
|
||||
/**
|
||||
* Harvest EVERY persisted `.jsonl` session log under a sessions root, parse each
|
||||
* header line, and return them ordered primary-first: the top-level session (no
|
||||
* `parentSession`) leads, then each subagent child by ascending `createdAt`.
|
||||
*
|
||||
* The JSONL backend lays sessions out as `<root>/<cwd-bucket>/<encoded-id>.jsonl`
|
||||
* (one bucket per cwd), so a parent and its same-cwd in-process child land in
|
||||
* the SAME bucket — collecting all files across all buckets catches both (a
|
||||
* first-match short-circuit would silently drop the child). Returns `[]` if no
|
||||
* log was produced (a no-session scenario).
|
||||
* Snapshot configs select the JSONL backend's raw mode, which lays sessions
|
||||
* out as `<root>/<cwd-bucket>/<encoded-id>.jsonl` (one bucket per cwd). A
|
||||
* parent and its same-cwd in-process child land in the SAME bucket, so
|
||||
* collecting all files across all buckets catches both. Returns `[]` if no log
|
||||
* was produced (a no-session scenario).
|
||||
*/
|
||||
async function harvestSessionLogs(root: string): Promise<HarvestedLog[]> {
|
||||
let cwdDirs: string[]
|
||||
@@ -452,8 +430,14 @@ async function harvestSessionLogs(root: string): Promise<HarvestedLog[]> {
|
||||
})
|
||||
}
|
||||
}
|
||||
// Match replay fixture assignment: primary first, then children by creation time, with id as
|
||||
// a deterministic collision tiebreaker.
|
||||
// Primary (no parentSession) first, then children by ascending createdAt. A
|
||||
// scenario has exactly one top-level session. In the synchronous cut sibling
|
||||
// children are created strictly sequentially, so their createdAt values are
|
||||
// strictly ordered; the recordedId tiebreak only keeps a degenerate
|
||||
// same-millisecond collision (unreachable here) deterministic. This harvest
|
||||
// order must match the replay load order in dsh-llm-replay's loadSessionScripts
|
||||
// so session.<n>.jsonl maps to the same child on record and replay — replay
|
||||
// re-sorts childFiles by the same key, so the two stay consistent.
|
||||
logs.sort((a, b) => {
|
||||
const ap = a.parentSession === undefined ? 0 : 1
|
||||
const bp = b.parentSession === undefined ? 0 : 1
|
||||
|
||||
@@ -1,13 +1,23 @@
|
||||
/**
|
||||
* ACP snapshot suite kit: subprocess scenario harness, pure golden normalizers, and the Vitest
|
||||
* suite factory behind `pnpm run test:snapshot`. Because this entry exports `suite.ts`, importing
|
||||
* it requires a Vitest run.
|
||||
* ACP snapshot suite kit — the shared machinery behind the keyless snapshot
|
||||
* tier (`pnpm run test:snapshot`). Four layers, composable per example: the
|
||||
* shared subprocess/client launcher ({@link launchAcpTestAgent}), the scripted
|
||||
* scenario harness ({@link runScenario}), the pure expected-output normalizers
|
||||
* ({@link normalizeStdout} / {@link normalizeSessionLog} /
|
||||
* {@link scrubRequestHeaders} / {@link scrubSystemPrompts}), and the suite
|
||||
* factory ({@link defineAcpSnapshotSuite}) that registers a scenario table as a
|
||||
* full describe/it tree. Ordinary ACP e2e tests can use the launcher directly;
|
||||
* an example's `*.snapshot.ts` supplies only its {@link AgentUnderTest} paths,
|
||||
* snapshots directory, and {@link Scenario} table.
|
||||
*
|
||||
* NOTE: ./suite.ts imports vitest, so this package is importable only inside a
|
||||
* vitest run — a support-tier constraint stated in the README.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-acp-snapshot
|
||||
*/
|
||||
|
||||
export {
|
||||
runScenario,
|
||||
type AgentUnderTest,
|
||||
type HarvestedLog,
|
||||
type InputScript,
|
||||
type InputStep,
|
||||
@@ -15,6 +25,12 @@ export {
|
||||
type RunOptions,
|
||||
type RunResult,
|
||||
} from './harness.ts'
|
||||
export {
|
||||
launchAcpTestAgent,
|
||||
type AcpTestLaunchOptions,
|
||||
type AgentUnderTest,
|
||||
type LaunchedAcpTestAgent,
|
||||
} from './launcher.ts'
|
||||
export {
|
||||
normalizeSessionLog,
|
||||
normalizeStdout,
|
||||
|
||||
276
packages/support/acp-snapshot/src/launcher.ts
Normal file
276
packages/support/acp-snapshot/src/launcher.ts
Normal file
@@ -0,0 +1,276 @@
|
||||
/**
|
||||
* Shared launcher for ACP tests that drive an agent subprocess over JSON-RPC
|
||||
* stdio. It owns source-or-built launch resolution, workspace environment,
|
||||
* stdout tee, SDK client, update collection, permission fallback, and process
|
||||
* shutdown so e2e and snapshot suites do not each reconstruct that boundary.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-acp-snapshot/launcher
|
||||
*/
|
||||
|
||||
import { spawn, type ChildProcessWithoutNullStreams } from 'node:child_process'
|
||||
import { join } from 'node:path'
|
||||
import { Readable, Writable } from 'node:stream'
|
||||
import {
|
||||
ClientSideConnection,
|
||||
ndJsonStream,
|
||||
type Agent as AcpAgent,
|
||||
type Client,
|
||||
type RequestPermissionRequest,
|
||||
type RequestPermissionResponse,
|
||||
type SessionNotification,
|
||||
} from '@agentclientprotocol/sdk'
|
||||
import { resolveExampleLaunch } from '@deepseek-ai/dsh-loader-smoke'
|
||||
|
||||
/** The source/built agent entry, leaf config, and workspace tsconfig an ACP test boots. */
|
||||
export interface AgentUnderTest {
|
||||
/** The agent source bin entry (for example `packages/examples/acp-demo/src/bin.ts`). */
|
||||
binScript: string
|
||||
/** Explicit built-mode entry for fixtures whose source path is not under `src/`. */
|
||||
libBinScript?: string | undefined
|
||||
/** The leaf `cordis.yml` loaded by the bin. */
|
||||
configPath: string
|
||||
/** The repo tsconfig whose paths resolve unbuilt workspace imports. */
|
||||
tsconfigPath: string
|
||||
}
|
||||
|
||||
/** Options for one ACP test subprocess. */
|
||||
export interface AcpTestLaunchOptions {
|
||||
/** The agent composition to boot. */
|
||||
agent: AgentUnderTest
|
||||
/** Process cwd and default session-home root. */
|
||||
cwd: string
|
||||
/** Alternate leaf config for this launch. */
|
||||
configPath?: string
|
||||
/** Extra environment values layered over the parent environment. */
|
||||
env?: NodeJS.ProcessEnv
|
||||
/** Permission handler; omitted requests fail closed as `cancelled`. */
|
||||
requestPermission?: (params: RequestPermissionRequest) => Promise<RequestPermissionResponse>
|
||||
}
|
||||
|
||||
/** A running ACP test process and its captured client-side surfaces. */
|
||||
export interface LaunchedAcpTestAgent {
|
||||
/** The child process, exposed for process-level assertions. */
|
||||
child: ChildProcessWithoutNullStreams
|
||||
/** Resolve when the OS spawns the child; reject with its asynchronous spawn failure. */
|
||||
spawned: Promise<void>
|
||||
/** The SDK connection backed by the child's stdio. */
|
||||
client: ClientSideConnection
|
||||
/** Session updates in receive order. */
|
||||
updates: SessionNotification['update'][]
|
||||
/** Decode all stdout bytes captured so far. */
|
||||
rawStdout(): string
|
||||
/** Decode all stderr chunks captured so far. */
|
||||
stderr(): string
|
||||
/** Resolve when a future session update matches the predicate. */
|
||||
waitForUpdate(match: (update: SessionNotification['update']) => boolean): Promise<SessionNotification['update']>
|
||||
/** Close the process and drain its streams and callbacks; rejects promptly if fallback termination is refused. */
|
||||
close(signal?: NodeJS.Signals): Promise<void>
|
||||
}
|
||||
|
||||
/**
|
||||
* Boot an ACP agent subprocess and connect an SDK client to its stdio.
|
||||
*
|
||||
* @param options Agent paths, cwd, environment, and optional permission handler.
|
||||
* @returns The running process, connected client, captures, and shutdown handle.
|
||||
*/
|
||||
export function launchAcpTestAgent(options: AcpTestLaunchOptions): LaunchedAcpTestAgent {
|
||||
const { agent, cwd } = options
|
||||
const launch = resolveExampleLaunch({
|
||||
srcBin: agent.binScript,
|
||||
libBin: agent.libBinScript,
|
||||
configArgs: ['--config', options.configPath ?? agent.configPath],
|
||||
tsconfigPath: agent.tsconfigPath,
|
||||
env: {
|
||||
...options.env,
|
||||
DSH_HOME: join(cwd, '.dsh'),
|
||||
DSH_AGENTS_HOME: join(cwd, '.agents'),
|
||||
},
|
||||
})
|
||||
const child = spawn(
|
||||
launch.command,
|
||||
launch.args,
|
||||
{
|
||||
cwd,
|
||||
env: { ...process.env, ...launch.env },
|
||||
stdio: ['pipe', 'pipe', 'pipe'],
|
||||
},
|
||||
)
|
||||
// A spawn-level failure is an asynchronous `error` event. Observe it in the
|
||||
// same tick as spawn so a missing cwd or OS rejection cannot crash the test
|
||||
// runner, then make startup and shutdown surface the original error.
|
||||
// Keep observing after the first error: a fallback kill attempted during
|
||||
// shutdown may itself report another process error, which must not become an
|
||||
// unhandled EventEmitter error after the promise has already settled.
|
||||
const childFailure = new Promise<Error>(resolve => child.on('error', resolve))
|
||||
const spawned = Promise.race([
|
||||
new Promise<void>(resolve => child.once('spawn', resolve)),
|
||||
childFailure.then((error): never => { throw error }),
|
||||
])
|
||||
// `spawned` is public and close() also awaits it, but a caller may ignore both.
|
||||
// Keep that misuse from turning the already-observed child error into an
|
||||
// unhandled promise rejection.
|
||||
void spawned.catch(() => undefined)
|
||||
|
||||
const stderrChunks: string[] = []
|
||||
child.stderr.setEncoding('utf8')
|
||||
child.stderr.on('data', (chunk: string) => stderrChunks.push(chunk))
|
||||
|
||||
const rawBuffers: Buffer[] = []
|
||||
const passthrough = new Readable({ read() {} })
|
||||
const updates: SessionNotification['update'][] = []
|
||||
const updateWaiters: {
|
||||
match: (update: SessionNotification['update']) => boolean
|
||||
resolve: (update: SessionNotification['update']) => void
|
||||
reject: (reason: unknown) => void
|
||||
}[] = []
|
||||
let updateStreamFailure: Error | undefined
|
||||
const closeUpdateStream = (): void => {
|
||||
if (updateStreamFailure !== undefined) return
|
||||
updateStreamFailure = new Error('ACP test agent update stream closed before a matching session update arrived')
|
||||
for (const waiter of updateWaiters.splice(0)) waiter.reject(updateStreamFailure)
|
||||
}
|
||||
child.stdout.on('data', (buffer: Buffer) => {
|
||||
rawBuffers.push(buffer)
|
||||
passthrough.push(buffer)
|
||||
})
|
||||
child.stdout.on('end', () => {
|
||||
passthrough.push(null)
|
||||
})
|
||||
const stream = ndJsonStream(
|
||||
Writable.toWeb(child.stdin) as WritableStream<Uint8Array>,
|
||||
Readable.toWeb(passthrough) as ReadableStream<Uint8Array>,
|
||||
)
|
||||
const inFlightClientCallbacks = new Set<Promise<unknown>>()
|
||||
const trackClientCallback = <T>(callback: () => T | PromiseLike<T>): Promise<T> => {
|
||||
const pending = Promise.resolve().then(callback)
|
||||
inFlightClientCallbacks.add(pending)
|
||||
const untrack = (): void => { inFlightClientCallbacks.delete(pending) }
|
||||
void pending.then(untrack, untrack)
|
||||
return pending
|
||||
}
|
||||
const requestPermission = options.requestPermission
|
||||
?? (() => Promise.resolve({ outcome: { outcome: 'cancelled' as const } }))
|
||||
const makeClient = (_agent: AcpAgent): Client => ({
|
||||
sessionUpdate(params: SessionNotification): Promise<void> {
|
||||
return trackClientCallback(() => {
|
||||
updates.push(params.update)
|
||||
for (let index = updateWaiters.length - 1; index >= 0; index--) {
|
||||
const waiter = updateWaiters[index]
|
||||
/* v8 ignore next 1 -- index is bounded by the array length */
|
||||
if (waiter === undefined) continue
|
||||
let matches: boolean
|
||||
try {
|
||||
matches = waiter.match(params.update)
|
||||
} catch (error: unknown) {
|
||||
updateWaiters.splice(index, 1)
|
||||
waiter.reject(error)
|
||||
continue
|
||||
}
|
||||
if (!matches) continue
|
||||
updateWaiters.splice(index, 1)
|
||||
waiter.resolve(params.update)
|
||||
}
|
||||
})
|
||||
},
|
||||
requestPermission: params => trackClientCallback(() => requestPermission(params)),
|
||||
})
|
||||
const client = new ClientSideConnection(makeClient, stream)
|
||||
// `exit` only reports the parent process's status. Descendants may retain
|
||||
// inherited stdout/stderr handles and buffered ACP frames may still be
|
||||
// crossing the SDK parser. Node's `close` follows stdio closure; the SDK's
|
||||
// `closed` follows parser exhaustion. Capture both eagerly so a caller that
|
||||
// invokes close after process exit still joins the complete drain boundary.
|
||||
const stdioClosed = new Promise<void>(resolve => child.once('close', () => { resolve() }))
|
||||
const drained = Promise.all([stdioClosed, client.closed]).then(async () => {
|
||||
// The ACP SDK's readable loop dispatches client callbacks without awaiting
|
||||
// them. Once `closed` settles no new callbacks can start, but callbacks
|
||||
// already in flight still belong to this launch's teardown boundary.
|
||||
while (inFlightClientCallbacks.size > 0) {
|
||||
await Promise.allSettled([...inFlightClientCallbacks])
|
||||
}
|
||||
})
|
||||
// A caller may await a pending update without calling close(). Make natural
|
||||
// stream exhaustion terminal for those waiters too, but only after the
|
||||
// parser has dispatched every buffered frame.
|
||||
void client.closed.then(closeUpdateStream)
|
||||
|
||||
return {
|
||||
child,
|
||||
spawned,
|
||||
client,
|
||||
updates,
|
||||
rawStdout: () => Buffer.concat(rawBuffers).toString('utf8'),
|
||||
stderr: () => stderrChunks.join(''),
|
||||
waitForUpdate(match): Promise<SessionNotification['update']> {
|
||||
if (updateStreamFailure !== undefined) return Promise.reject(updateStreamFailure)
|
||||
return new Promise((resolve, reject) => updateWaiters.push({ match, resolve, reject }))
|
||||
},
|
||||
async close(signal?: NodeJS.Signals): Promise<void> {
|
||||
try {
|
||||
await spawned
|
||||
} catch (error: unknown) {
|
||||
await drained
|
||||
closeUpdateStream()
|
||||
throw error
|
||||
}
|
||||
if (!isRunning(child)) {
|
||||
await drained
|
||||
closeUpdateStream()
|
||||
return
|
||||
}
|
||||
const exited = waitForExit(child)
|
||||
if (signal === undefined) child.stdin.end()
|
||||
else child.kill(signal)
|
||||
const failure = await Promise.race([
|
||||
exited.then((): undefined => undefined),
|
||||
childFailure,
|
||||
])
|
||||
if (failure === undefined) {
|
||||
await drained
|
||||
closeUpdateStream()
|
||||
return
|
||||
}
|
||||
|
||||
// An `error` after spawn is not an exit edge: in particular, a failed
|
||||
// signal can leave the subprocess live. Force termination, await the
|
||||
// already-observed exit edge, and only then propagate the child error so
|
||||
// callers may safely remove cwd/session resources after close rejects.
|
||||
const fallbackError = Promise.withResolvers<Error>()
|
||||
const observeFallbackError = (error: Error): void => { fallbackError.resolve(error) }
|
||||
child.once('error', observeFallbackError)
|
||||
if (!child.kill('SIGKILL')) {
|
||||
child.off('error', observeFallbackError)
|
||||
closeUpdateStream()
|
||||
throw new AggregateError(
|
||||
[failure, new Error('Fallback SIGKILL was not accepted by the child process')],
|
||||
'ACP test agent failed and fallback termination was refused',
|
||||
)
|
||||
}
|
||||
const fallbackFailure = await Promise.race([
|
||||
exited.then((): undefined => undefined),
|
||||
fallbackError.promise,
|
||||
])
|
||||
child.off('error', observeFallbackError)
|
||||
if (fallbackFailure !== undefined) {
|
||||
closeUpdateStream()
|
||||
throw new AggregateError(
|
||||
[failure, fallbackFailure],
|
||||
'ACP test agent failed and fallback termination was refused',
|
||||
)
|
||||
}
|
||||
await drained
|
||||
closeUpdateStream()
|
||||
throw failure
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/** Resolve once a running child exits. */
|
||||
function waitForExit(child: ChildProcessWithoutNullStreams): Promise<void> {
|
||||
return new Promise<void>(resolve => child.once('exit', () => { resolve() }))
|
||||
}
|
||||
|
||||
/** Whether the child still lacks either OS termination marker. */
|
||||
function isRunning(child: ChildProcessWithoutNullStreams): boolean {
|
||||
return child.exitCode === null && child.signalCode === null
|
||||
}
|
||||
@@ -60,7 +60,7 @@ function scrubValue(value: unknown, ctx: NormalizeContext): unknown {
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize a raw stdout transcript (newline-delimited JSON-RPC frames) into a stable golden
|
||||
* Normalize a raw stdout transcript (newline-delimited JSON-RPC frames) into a stable expected output
|
||||
* in the same shape as the wire: one compact JSON frame per line (NDJSON), with the JSON-RPC
|
||||
* `id` rewritten to a per-transcript sequence (1, 2, 3, …) and all volatile strings scrubbed.
|
||||
* Invalid JSON throws, doubling as a protocol-stdout purity check.
|
||||
@@ -72,7 +72,7 @@ function scrubValue(value: unknown, ctx: NormalizeContext): unknown {
|
||||
export function normalizeStdout(rawStdout: string, ctx: NormalizeContext): string {
|
||||
const lines = rawStdout.split('\n').filter(line => line.trim().length > 0)
|
||||
// Map each distinct JSON-RPC id (request/response correlate by id) to a stable
|
||||
// sequence number, in first-seen order, so id churn doesn't perturb the golden.
|
||||
// sequence number, in first-seen order, so id churn doesn't perturb the expected output.
|
||||
const idSeq = new Map<string, number>()
|
||||
const stableId = (id: unknown): number => {
|
||||
const key = JSON.stringify(id)
|
||||
@@ -91,7 +91,7 @@ export function normalizeStdout(rawStdout: string, ctx: NormalizeContext): strin
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize a session JSONL log into a stable golden: the header line's
|
||||
* Normalize a session JSONL log into a stable expected output: the header line's
|
||||
* volatile fields (`createdAt`, `id`, `cwd`) and every event's `time` are
|
||||
* zeroed/scrubbed, all volatile strings scrubbed, and `seq` is LEFT INTACT
|
||||
* (deterministic by contract). Output is JSONL in the same shape as the input —
|
||||
@@ -112,7 +112,7 @@ export function normalizeSessionLog(rawLog: string, ctx: NormalizeContext): stri
|
||||
// Event line: zero the epoch-ms timestamp; keep seq (deterministic).
|
||||
record.time = 0
|
||||
// A hook/result carries the hook's wall-clock runtime (`data.durationMs`),
|
||||
// which is run-to-run noise like `time` — zero it so the golden reflects
|
||||
// which is run-to-run noise like `time` — zero it so the expected output reflects
|
||||
// the hook's decision/exit, not how long the shell took.
|
||||
if (record.type === 'hook/result' && record.data !== null && typeof record.data === 'object') {
|
||||
const data = record.data as Record<string, unknown>
|
||||
|
||||
@@ -15,7 +15,7 @@
|
||||
* @module @deepseek-ai/dsh-acp-snapshot/suite
|
||||
*/
|
||||
|
||||
import { readFile, readdir, writeFile } from 'node:fs/promises'
|
||||
import { readFile, readdir, rm, writeFile } from 'node:fs/promises'
|
||||
import { existsSync } from 'node:fs'
|
||||
import { join } from 'node:path'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
@@ -30,10 +30,10 @@ import {
|
||||
} from './normalize.ts'
|
||||
|
||||
/** The readable system-prompt snapshot beside each header-pinning fixture. */
|
||||
const SYSTEM_PROMPT_SNAPSHOT = 'system-prompt.golden.md'
|
||||
const SYSTEM_PROMPT_SNAPSHOT = 'system-prompt.expected.md'
|
||||
|
||||
/** The structured tool-schema snapshot beside each header-pinning fixture. */
|
||||
const TOOL_SCHEMAS_SNAPSHOT = 'tool-schemas.golden.json'
|
||||
const TOOL_SCHEMAS_SNAPSHOT = 'tool-schemas.expected.json'
|
||||
|
||||
/** Stable session-log token standing in for the sidecar's initial schemas. */
|
||||
const TOOLS_TOKEN = '{{tools}}'
|
||||
@@ -41,7 +41,7 @@ const TOOLS_TOKEN = '{{tools}}'
|
||||
/** A snapshot scenario and how its fixtures are produced. */
|
||||
export interface Scenario {
|
||||
name: string
|
||||
/** Whether the scenario drives at least one model turn (so a JSONL golden applies). */
|
||||
/** Whether the scenario drives at least one model turn (so a JSONL expected output applies). */
|
||||
hasModelTurn: boolean
|
||||
/**
|
||||
* Whether the run persists a comparable session log to diff against the
|
||||
@@ -71,14 +71,6 @@ export interface Scenario {
|
||||
* false (replay derives from the fixture's `assistant/chunk` events).
|
||||
*/
|
||||
overridden?: boolean
|
||||
/**
|
||||
* How many SUBAGENT child sessions this scenario records beyond the top-level
|
||||
* one (0 for a single-session scenario). Each child rides in a sibling fixture
|
||||
* `session.<n>.jsonl` (1-based); replay forwards them to `dsh-llm-replay` so
|
||||
* each child session replays from its own script, and record mode writes the
|
||||
* harvested child logs back to those files. Defaults to 0.
|
||||
*/
|
||||
childSessions?: number
|
||||
/**
|
||||
* Whether this scenario is its header class's sole request-header pin. Dedicated sidecars own
|
||||
* the prompt and tool schemas, while every classmate is checked for equality.
|
||||
@@ -120,8 +112,8 @@ export interface SnapshotSuiteOptions {
|
||||
scenarios: Scenario[]
|
||||
/**
|
||||
* `replay` (keyless, the default tier), `record` (live API; re-records the
|
||||
* `recorded` scenarios' fixtures and refreshes the Vitest goldens under
|
||||
* `--update`), or `refresh` (keyless replay that rewrites stdout goldens and
|
||||
* `recorded` scenarios' fixtures and refreshes the Vitest expected outputs under
|
||||
* `--update`), or `refresh` (keyless replay that rewrites stdout expected outputs and
|
||||
* comparable session fixtures from the replay run). The caller derives this
|
||||
* from `$DSH_SNAPSHOT` — env reading stays outside this library.
|
||||
*/
|
||||
@@ -129,14 +121,40 @@ export interface SnapshotSuiteOptions {
|
||||
}
|
||||
|
||||
/**
|
||||
* The sibling child-fixture paths for a scenario (`session.1.jsonl` …).
|
||||
* Validate and order a scenario directory's session-fixture filenames.
|
||||
*
|
||||
* @param dir The scenario's snapshots directory (`<snapshotsDir>/<name>`).
|
||||
* @param childSessions How many subagent child sessions the scenario records.
|
||||
* @returns One path per child, 1-based, in fixture order.
|
||||
* The primary fixture is always `session.jsonl`; child sessions are discovered
|
||||
* from contiguous `session.1.jsonl` … filenames. The directory is the source of
|
||||
* truth, so scenario tables do not duplicate a child count that can drift from
|
||||
* the files. A session-like JSONL with any other suffix fails loud.
|
||||
*
|
||||
* @param names File names in one scenario directory.
|
||||
* @returns The primary and child fixture names in replay/harvest order.
|
||||
*/
|
||||
export function childFixturePaths(dir: string, childSessions: number): string[] {
|
||||
return Array.from({ length: childSessions }, (_, i) => join(dir, `session.${i + 1}.jsonl`))
|
||||
export function sessionFixtureNames(names: readonly string[]): string[] {
|
||||
if (!names.includes('session.jsonl')) throw new Error('missing session.jsonl')
|
||||
const children: { name: string; index: number }[] = []
|
||||
for (const name of names) {
|
||||
if (name === 'session.jsonl') continue
|
||||
if (!name.startsWith('session.') || !name.endsWith('.jsonl')) continue
|
||||
const match = /^session\.([1-9]\d*)\.jsonl$/.exec(name)
|
||||
if (match === null) throw new Error(`invalid child session fixture name: ${name}`)
|
||||
children.push({ name, index: Number(match[1]) })
|
||||
}
|
||||
children.sort((a, b) => a.index - b.index)
|
||||
for (const [offset, child] of children.entries()) {
|
||||
const expected = offset + 1
|
||||
if (child.index !== expected) {
|
||||
throw new Error(`child session fixtures must be contiguous: expected session.${expected}.jsonl, found ${child.name}`)
|
||||
}
|
||||
}
|
||||
return ['session.jsonl', ...children.map(child => child.name)]
|
||||
}
|
||||
|
||||
/** Read one scenario directory's validated session-fixture inventory. */
|
||||
async function sessionFixtures(dir: string): Promise<string[]> {
|
||||
const entries = await readdir(dir, { withFileTypes: true })
|
||||
return sessionFixtureNames(entries.filter(entry => entry.isFile()).map(entry => entry.name))
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -409,7 +427,7 @@ export function stabilizeRefreshLog(fresh: string, existing: string, replacement
|
||||
}
|
||||
|
||||
/**
|
||||
* Register the suite: one test per scenario (the golden/log compares and
|
||||
* Register the suite: one test per scenario (the expected-output and log comparisons and
|
||||
* the header-uniformity guard) plus the fixture guard block (no orphan
|
||||
* scenario dirs, required files present, exactly one pin per header class,
|
||||
* pinning fixtures well-formed, every JSONL prompt-scrubbed, non-pinning
|
||||
@@ -449,12 +467,17 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
for (const scenario of scenarios) {
|
||||
// In RECORD mode, only re-run the `recorded` (live-API) scenarios; the `authored` ones
|
||||
// (sidecar-driven errors/cancel) are never re-recorded.
|
||||
it.skipIf(RECORDING && !scenario.recorded)(`snapshot: ${scenario.name} matches the goldens`, async ({ expect }) => {
|
||||
it.skipIf(RECORDING && !scenario.recorded)(`snapshot: ${scenario.name} matches the expected outputs`, async ({ expect }) => {
|
||||
const dir = join(snapshotsDir, scenario.name)
|
||||
const input = JSON.parse(await readFile(join(dir, 'input.json'), 'utf8')) as InputScript
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
const workspaceDir = join(dir, 'workspace')
|
||||
const childSessions = scenario.childSessions ?? 0
|
||||
// Replay/refresh need the committed inventory up front because those
|
||||
// files drive the model scripts. Record mode creates that inventory
|
||||
// from the harvested live logs, so it must also work for a brand-new
|
||||
// scenario with no session.jsonl yet.
|
||||
let fixtureFiles = RECORDING ? [] : await sessionFixtures(dir)
|
||||
const childFixtureFiles = fixtureFiles.slice(1)
|
||||
const comparesLog = scenario.comparesLog ?? scenario.hasModelTurn
|
||||
const result = await runScenario(input, {
|
||||
agent,
|
||||
@@ -463,7 +486,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
...existsSync(overrideFile) ? { overrideFile } : {},
|
||||
// In REPLAY, forward the recorded child fixtures so each subagent session
|
||||
// replays from its own script. In RECORD they are harvested, not read.
|
||||
...!RECORDING && childSessions > 0 ? { childFiles: childFixturePaths(dir, childSessions) } : {},
|
||||
...!RECORDING && childFixtureFiles.length > 0 ? { childFiles: childFixtureFiles.map(file => join(dir, file)) } : {},
|
||||
...existsSync(workspaceDir) ? { workspaceDir } : {},
|
||||
// A scenario booting an overlay tree passes its own live config; the
|
||||
// bin's replay swap derives the sibling `*cordis.snapshot.yml` from it.
|
||||
@@ -491,7 +514,6 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
const scrub = scenario.pinsHeader === true
|
||||
? (log: string): string => scrubToolSchemas(scrubSystemPrompts(log))
|
||||
: scrubRequestHeaders
|
||||
const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)]
|
||||
const existingFixtures = REFRESHING
|
||||
? await Promise.all(fixtureFiles.map(file => readFile(join(dir, file), 'utf8')))
|
||||
: []
|
||||
@@ -500,18 +522,37 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
|| (REFRESHING && comparesLog)
|
||||
if (writesSessionFixtures) {
|
||||
expect(result.sessionLogs.length, `${mode} produced no session log to harvest`).toBeGreaterThan(0)
|
||||
expect(result.sessionLogs.length, `expected ${childSessions + 1} session logs (parent + children)`)
|
||||
.toBe(childSessions + 1)
|
||||
if (REFRESHING) {
|
||||
expect(result.sessionLogs.length, `expected ${fixtureFiles.length} session logs (parent + children)`)
|
||||
.toBe(fixtureFiles.length)
|
||||
}
|
||||
const outputFixtureFiles = [
|
||||
'session.jsonl',
|
||||
...Array.from({ length: result.sessionLogs.length - 1 }, (_, i) => `session.${i + 1}.jsonl`),
|
||||
]
|
||||
const primary = (result.sessionLogs[0] as HarvestedLog).content
|
||||
await writeFile(join(dir, 'session.jsonl'), scrub(
|
||||
await writeFile(join(dir, outputFixtureFiles[0] as string), scrub(
|
||||
REFRESHING ? stabilizeRefreshLog(primary, existingFixtures[0] as string, replacements) : primary,
|
||||
))
|
||||
for (let i = 1; i < result.sessionLogs.length; i++) {
|
||||
const child = (result.sessionLogs[i] as HarvestedLog).content
|
||||
await writeFile(join(dir, `session.${i}.jsonl`), scrub(
|
||||
await writeFile(join(dir, outputFixtureFiles[i] as string), scrub(
|
||||
REFRESHING ? stabilizeRefreshLog(child, existingFixtures[i] as string, replacements) : child,
|
||||
))
|
||||
}
|
||||
if (RECORDING) {
|
||||
const outputNames = new Set(outputFixtureFiles)
|
||||
const entries = await readdir(dir, { withFileTypes: true })
|
||||
await Promise.all(entries
|
||||
.filter(entry => entry.isFile()
|
||||
// Only valid numbered children are record-owned stale output.
|
||||
// Malformed session-like names stay for the inventory guard to
|
||||
// reject instead of being silently deleted during mutation.
|
||||
&& /^session\.[1-9]\d*\.jsonl$/.test(entry.name)
|
||||
&& !outputNames.has(entry.name))
|
||||
.map(entry => rm(join(dir, entry.name))))
|
||||
fixtureFiles = outputFixtureFiles
|
||||
}
|
||||
if (scenario.pinsHeader === true) {
|
||||
const primary = result.sessionLogs[0] as HarvestedLog
|
||||
const prompts = normalizedSystemPrompts(primary.content, ctx)
|
||||
@@ -532,15 +573,15 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
|
||||
const stdout = normalizeStdout(result.rawStdout, ctx)
|
||||
if (REFRESHING) {
|
||||
await writeFile(join(dir, 'stdout.golden.jsonl'), stdout)
|
||||
await writeFile(join(dir, 'stdout.expected.jsonl'), stdout)
|
||||
}
|
||||
await expect(stdout).toMatchFileSnapshot(join(dir, 'stdout.golden.jsonl'))
|
||||
await expect(stdout).toMatchFileSnapshot(join(dir, 'stdout.expected.jsonl'))
|
||||
|
||||
// A model turn always produces a log worth comparing; a hook scenario can
|
||||
// produce one without a model turn (a `rejected` turn carrying `hook/*`).
|
||||
if (comparesLog) {
|
||||
// The harvested logs (primary-first) must match their committed fixtures 1:1.
|
||||
expect(result.sessionLogs.length, 'this scenario must persist a session log').toBe(childSessions + 1)
|
||||
expect(result.sessionLogs.length, 'this scenario must persist one log per session fixture').toBe(fixtureFiles.length)
|
||||
for (let i = 0; i < fixtureFiles.length; i++) {
|
||||
const harvested = scrub((result.sessionLogs[i] as HarvestedLog).content)
|
||||
const fixture = scrub(await readFile(join(dir, fixtureFiles[i] as string), 'utf8'))
|
||||
@@ -610,7 +651,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
|
||||
describe('snapshot fixtures', () => {
|
||||
it('every scenario directory is registered (no orphans)', async () => {
|
||||
// toMatchFileSnapshot does not prune orphaned golden/fixture files, so a
|
||||
// toMatchFileSnapshot does not prune orphaned expected-output or fixture files, so a
|
||||
// renamed/removed scenario could leave a stale dir that nothing exercises.
|
||||
// Fail loud on any snapshots/<dir> not present in the scenario table.
|
||||
const entries = await readdir(snapshotsDir, { withFileTypes: true })
|
||||
@@ -619,12 +660,12 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
expect(onDisk).toEqual(registered)
|
||||
})
|
||||
|
||||
it('every registered scenario has its required fixture files', () => {
|
||||
// Every scenario has an input script and an stdout golden.
|
||||
for (const { name, overridden, childSessions, pinsHeader } of scenarios) {
|
||||
it('every registered scenario has its required fixture files', async () => {
|
||||
// Every scenario needs input, stdout, a primary session fixture, and matching optional sidecars.
|
||||
for (const { name, overridden, pinsHeader } of scenarios) {
|
||||
const dir = join(snapshotsDir, name)
|
||||
expect(existsSync(join(dir, 'input.json')), `${name}/input.json`).toBe(true)
|
||||
expect(existsSync(join(dir, 'stdout.golden.jsonl')), `${name}/stdout.golden.jsonl`).toBe(true)
|
||||
expect(existsSync(join(dir, 'stdout.expected.jsonl')), `${name}/stdout.expected.jsonl`).toBe(true)
|
||||
expect(existsSync(join(dir, 'session.jsonl')), `${name}/session.jsonl`).toBe(true)
|
||||
expect(existsSync(join(dir, 'replay.override.json')), `${name}/replay.override.json presence must match \`overridden\``)
|
||||
.toBe(overridden === true)
|
||||
@@ -632,11 +673,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
.toBe(pinsHeader === true)
|
||||
expect(existsSync(join(dir, TOOL_SCHEMAS_SNAPSHOT)), `${name}/${TOOL_SCHEMAS_SNAPSHOT} presence must match \`pinsHeader\``)
|
||||
.toBe(pinsHeader === true)
|
||||
// A nested-agent scenario ships one child fixture per recorded subagent
|
||||
// session (`session.1.jsonl` …), the replay source for that child session.
|
||||
for (const childFixture of childFixturePaths(dir, childSessions ?? 0)) {
|
||||
expect(existsSync(childFixture), childFixture).toBe(true)
|
||||
}
|
||||
await expect(sessionFixtures(dir), `${name}: session fixture inventory`).resolves.toBeDefined()
|
||||
}
|
||||
})
|
||||
|
||||
@@ -688,10 +725,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
// storage rules fail loud.
|
||||
for (const scenario of scenarios) {
|
||||
const dir = join(snapshotsDir, scenario.name)
|
||||
const files = [
|
||||
'session.jsonl',
|
||||
...Array.from({ length: scenario.childSessions ?? 0 }, (_, i) => `session.${i + 1}.jsonl`),
|
||||
]
|
||||
const files = await sessionFixtures(dir)
|
||||
for (const file of files) {
|
||||
const fixture = await readFile(join(dir, file), 'utf8')
|
||||
expect(unknownToolCallIds(fixture), `${scenario.name}/${file} contains UNKNOWN_TOOL`)
|
||||
|
||||
@@ -6,6 +6,7 @@
|
||||
|
||||
import { mkdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
|
||||
import { readdirSync } from 'node:fs'
|
||||
import { spawn } from 'node:child_process'
|
||||
import { dirname, join } from 'node:path'
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import { createInterface } from 'node:readline'
|
||||
@@ -32,6 +33,10 @@ interface Behavior {
|
||||
rejectExtraDirs?: boolean
|
||||
/** How `session/prompt` settles: a clean response, a JSON-RPC error, or a hang until `session/cancel`. */
|
||||
prompt?: 'respond' | 'error' | 'hang-until-cancel'
|
||||
/** Emit a tool call instead of a message chunk before parking a cancellable prompt. */
|
||||
cancelAtToolCall?: boolean
|
||||
/** Emit the parked tool call's terminal update after answering cancellation. */
|
||||
cancelToolCallUpdate?: boolean
|
||||
/** Before responding to a prompt, send a `session/request_permission` request and echo its outcome as a chunk. */
|
||||
permissionProbe?: boolean
|
||||
/** Echo the `DSH_SNAPSHOT_*` env the harness set as a chunk (spec-side env-plumbing assertions). */
|
||||
@@ -40,6 +45,8 @@ interface Behavior {
|
||||
echoWorkspace?: boolean
|
||||
/** Write a line to stderr on boot (spec-side stderr-capture assertions). */
|
||||
stderrNote?: string
|
||||
/** Let a short-lived descendant retain stdio and emit one final ACP update plus stderr line after this parent exits. */
|
||||
lateInheritedOutput?: boolean
|
||||
/** Session logs to persist on stdin EOF. */
|
||||
logs?: ScriptedLog[]
|
||||
/** Leave a stray FILE directly under the sessions root (harvest must skip it). */
|
||||
@@ -123,7 +130,23 @@ async function handlePrompt(id: number | string): Promise<void> {
|
||||
params: { sessionId, update: { sessionUpdate: 'agent_thought_chunk', content: { type: 'text', text: 'mulling' } } },
|
||||
})
|
||||
}
|
||||
chunk('thinking about it')
|
||||
if (behavior.cancelAtToolCall === true) {
|
||||
send({
|
||||
method: 'session/update',
|
||||
params: {
|
||||
sessionId,
|
||||
update: {
|
||||
sessionUpdate: 'tool_call',
|
||||
toolCallId: 'call_fake_1',
|
||||
title: 'fake tool',
|
||||
kind: 'execute',
|
||||
status: 'in_progress',
|
||||
},
|
||||
},
|
||||
})
|
||||
} else {
|
||||
chunk('thinking about it')
|
||||
}
|
||||
if (behavior.echoEnv === true) {
|
||||
chunk(`env:${JSON.stringify({
|
||||
mode: process.env.DSH_SNAPSHOT,
|
||||
@@ -226,6 +249,19 @@ function handleFrame(frame: Record<string, unknown>): void {
|
||||
const parked = parkedPromptId
|
||||
parkedPromptId = null
|
||||
respond(parked, { stopReason: 'cancelled' })
|
||||
if (behavior.cancelToolCallUpdate === true) {
|
||||
send({
|
||||
method: 'session/update',
|
||||
params: {
|
||||
sessionId,
|
||||
update: {
|
||||
sessionUpdate: 'tool_call_update',
|
||||
toolCallId: 'call_fake_1',
|
||||
status: 'failed',
|
||||
},
|
||||
},
|
||||
})
|
||||
}
|
||||
}
|
||||
return
|
||||
default:
|
||||
@@ -247,6 +283,24 @@ function flushLogsAndExit(): void {
|
||||
writeFileSync(join(sessionsRoot, 'bucket-noise', 'notes.txt'), 'not a session log\n')
|
||||
}
|
||||
if (behavior.deleteSessionsRoot === true) rmSync(sessionsRoot, { recursive: true, force: true })
|
||||
if (behavior.lateInheritedOutput === true) {
|
||||
const frame = JSON.stringify({
|
||||
jsonrpc: '2.0',
|
||||
method: 'session/update',
|
||||
params: {
|
||||
sessionId,
|
||||
update: {
|
||||
sessionUpdate: 'agent_message_chunk',
|
||||
content: { type: 'text', text: 'late inherited stdout' },
|
||||
},
|
||||
},
|
||||
})
|
||||
const code = [
|
||||
`setTimeout(() => process.stdout.write(${JSON.stringify(`${frame}\n`)}), 50)`,
|
||||
`setTimeout(() => process.stderr.write(${JSON.stringify('late inherited stderr\n')}), 75)`,
|
||||
].join(';')
|
||||
spawn(process.execPath, ['-e', code], { stdio: ['ignore', 1, 2] }).unref()
|
||||
}
|
||||
process.exit(0)
|
||||
}
|
||||
|
||||
|
||||
@@ -2,11 +2,11 @@
|
||||
"prompt": "respond",
|
||||
"logs": [
|
||||
{ "file": "b/parent.jsonl", "lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 700, "cwd": "{{CWD}}" },
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 700, "cwd": "{{CWD}}", "delegationDepth": 0 },
|
||||
{ "type": "request/header", "seq": 0, "time": 3, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
|
||||
]},
|
||||
{ "file": "b/child.jsonl", "lines": [
|
||||
{ "type": "session", "id": "abababab-cdcd-4efe-8ada-badabadabada", "createdAt": 800, "cwd": "{{CWD}}", "parentSession": "{{SID}}" },
|
||||
{ "type": "session", "id": "abababab-cdcd-4efe-8ada-badabadabada", "createdAt": 800, "cwd": "{{CWD}}", "parentSession": "{{SID}}", "delegationDepth": 1 },
|
||||
{ "type": "request/header", "seq": 0, "time": 2, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
|
||||
]}
|
||||
]
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
{"type":"session","id":"abababab-cdcd-4efe-8ada-badabadabada","createdAt":800,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-KBQJbW","parentSession":"f6fa7fcf-dd9c-4b39-8815-b25ddcebfd88"}
|
||||
{"type":"session","id":"abababab-cdcd-4efe-8ada-badabadabada","createdAt":800,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-KBQJbW","parentSession":"f6fa7fcf-dd9c-4b39-8815-b25ddcebfd88","delegationDepth":1}
|
||||
{"type":"request/header","seq":0,"time":2,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
{"type":"session","id":"f6fa7fcf-dd9c-4b39-8815-b25ddcebfd88","createdAt":700,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-KBQJbW"}
|
||||
{"type":"session","id":"f6fa7fcf-dd9c-4b39-8815-b25ddcebfd88","createdAt":700,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-KBQJbW","delegationDepth":0}
|
||||
{"type":"request/header","seq":0,"time":3,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
"logs": [{
|
||||
"file": "b/main.jsonl",
|
||||
"lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 600, "cwd": "{{CWD}}" },
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 600, "cwd": "{{CWD}}", "delegationDepth": 0 },
|
||||
{ "type": "request/header", "seq": 0, "time": 4, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
|
||||
]
|
||||
}]
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
{"type":"session","id":"ccdc749f-56f3-4267-9750-598b5c60b7b2","createdAt":600,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-nOQ4Gy"}
|
||||
{"type":"session","id":"ccdc749f-56f3-4267-9750-598b5c60b7b2","createdAt":600,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-nOQ4Gy","delegationDepth":0}
|
||||
{"type":"request/header","seq":0,"time":4,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
|
||||
@@ -1 +1 @@
|
||||
{"type":"session","id":"{{sessionId}}","createdAt":0,"cwd":"{{cwd}}"}
|
||||
{"type":"session","id":"{{sessionId}}","createdAt":0,"cwd":"{{cwd}}","delegationDepth":0}
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
"logs": [{
|
||||
"file": "b/main.jsonl",
|
||||
"lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 500, "cwd": "{{CWD}}" },
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 500, "cwd": "{{CWD}}", "delegationDepth": 0 },
|
||||
{ "type": "turn/end", "seq": 1, "time": 9, "data": { "error": "model exploded" } }
|
||||
]
|
||||
}]
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
{"type":"session","id":"44444444-3333-4222-8111-000000000000","createdAt":17,"cwd":"/rec/authored-cwd"}
|
||||
{"type":"session","id":"44444444-3333-4222-8111-000000000000","createdAt":17,"cwd":"/rec/authored-cwd","delegationDepth":0}
|
||||
{"type":"turn/end","seq":1,"time":17,"data":{"error":"model exploded"}}
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
"logs": [{
|
||||
"file": "b/main.jsonl",
|
||||
"lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 400, "cwd": "{{CWD}}" },
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 400, "cwd": "{{CWD}}", "delegationDepth": 0 },
|
||||
{ "type": "hook/result", "seq": 1, "time": 8, "data": { "decision": "block", "durationMs": 37 } }
|
||||
]
|
||||
}]
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
{"type":"session","id":"99999999-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/blocked-cwd"}
|
||||
{"type":"session","id":"99999999-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/blocked-cwd","delegationDepth":0}
|
||||
{"type":"hook/result","seq":1,"time":13,"data":{"decision":"block","durationMs":99}}
|
||||
|
||||
@@ -1 +1 @@
|
||||
{"type":"session","id":"{{sessionId}}","createdAt":0,"cwd":"{{cwd}}"}
|
||||
{"type":"session","id":"{{sessionId}}","createdAt":0,"cwd":"{{cwd}}","delegationDepth":0}
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
"logs": [{
|
||||
"file": "b/main.jsonl",
|
||||
"lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 100, "cwd": "{{CWD}}" },
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 100, "cwd": "{{CWD}}", "delegationDepth": 0 },
|
||||
{ "type": "request/header", "seq": 0, "time": 100, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } },
|
||||
{ "type": "request/header", "seq": 1, "time": 100, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT\n\nNEW PROMPT LINE", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "change" } },
|
||||
{ "type": "turn/start", "seq": 2, "time": 100, "data": { "turn": 1 } }
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
{"type":"session","id":"12121212-3434-4545-8686-787878787878","createdAt":7,"cwd":"/rec/pin-cwd"}
|
||||
{"type":"session","id":"12121212-3434-4545-8686-787878787878","createdAt":7,"cwd":"/rec/pin-cwd","delegationDepth":0}
|
||||
{"type":"request/header","seq":0,"time":7,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
{"type":"request/header","seq":1,"time":7,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"change"}}
|
||||
{"type":"turn/start","seq":2,"time":7,"data":{"turn":1}}
|
||||
|
||||
@@ -3,12 +3,12 @@
|
||||
"echoWorkspace": true,
|
||||
"logs": [
|
||||
{ "file": "b/parent.jsonl", "lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 200, "cwd": "{{CWD}}" },
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 200, "cwd": "{{CWD}}", "delegationDepth": 0 },
|
||||
{ "type": "request/header", "seq": 0, "time": 5, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } },
|
||||
{ "type": "assistant/chunk", "seq": 1, "time": 5, "data": { "turn": 1, "step": 1, "chunk": { "type": "text-delta", "index": 0, "text": "hi" } } }
|
||||
]},
|
||||
{ "file": "b/child.jsonl", "lines": [
|
||||
{ "type": "session", "id": "eeeeeeee-1111-4222-8333-444444444444", "createdAt": 300, "cwd": "{{CWD}}", "parentSession": "{{SID}}" },
|
||||
{ "type": "session", "id": "eeeeeeee-1111-4222-8333-444444444444", "createdAt": 300, "cwd": "{{CWD}}", "parentSession": "{{SID}}", "delegationDepth": 1 },
|
||||
{ "type": "request/header", "seq": 0, "time": 6, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
|
||||
]}
|
||||
]
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
{"type":"session","id":"eeeeeeee-1111-4222-8333-444444444444","createdAt":12,"cwd":"/rec/plain-cwd","parentSession":"56565656-7878-4989-8a9a-9b9b9b9b9b9b"}
|
||||
{"type":"session","id":"eeeeeeee-1111-4222-8333-444444444444","createdAt":12,"cwd":"/rec/plain-cwd","parentSession":"56565656-7878-4989-8a9a-9b9b9b9b9b9b","delegationDepth":1}
|
||||
{"type":"request/header","seq":0,"time":12,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
{"type":"session","id":"56565656-7878-4989-8a9a-9b9b9b9b9b9b","createdAt":11,"cwd":"/rec/plain-cwd"}
|
||||
{"type":"session","id":"56565656-7878-4989-8a9a-9b9b9b9b9b9b","createdAt":11,"cwd":"/rec/plain-cwd","delegationDepth":0}
|
||||
{"type":"request/header","seq":0,"time":11,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
{"type":"assistant/chunk","seq":1,"time":11,"data":{"turn":1,"step":1,"chunk":{"type":"text-delta","index":0,"text":"hi"}}}
|
||||
|
||||
@@ -1,13 +1,34 @@
|
||||
import { mkdtemp, rm, writeFile } from 'node:fs/promises'
|
||||
import { once } from 'node:events'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { delimiter, join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { afterAll, describe, expect, it } from 'vitest'
|
||||
import { afterAll, describe, expect, it, vi } from 'vitest'
|
||||
import { PROTOCOL_VERSION } from '@agentclientprotocol/sdk'
|
||||
import { runScenario, type AgentUnderTest, type InputStep } from '../src/harness.ts'
|
||||
import { launchAcpTestAgent } from '../src/launcher.ts'
|
||||
|
||||
const fsControl = vi.hoisted(() => ({ cleanupFailure: undefined as Error | undefined }))
|
||||
|
||||
vi.mock('node:fs/promises', async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import('node:fs/promises')>()
|
||||
return {
|
||||
...actual,
|
||||
async rm(...args: Parameters<typeof actual.rm>): Promise<void> {
|
||||
if (String(args[0]).includes('acp-snap-cwd-') && fsControl.cleanupFailure !== undefined) {
|
||||
const failure = fsControl.cleanupFailure
|
||||
fsControl.cleanupFailure = undefined
|
||||
await actual.rm(...args)
|
||||
throw failure
|
||||
}
|
||||
await actual.rm(...args)
|
||||
},
|
||||
}
|
||||
})
|
||||
|
||||
/**
|
||||
* Unit tests for the subprocess harness, driven through the REAL spawn path
|
||||
* (tsx loader, temp cwd, env plumbing) against the scripted fake ACP bin in
|
||||
* (mode-aware launcher, temp cwd, env plumbing) against the scripted fake ACP bin in
|
||||
* ./fixtures/fake-acp-agent.ts. Each case writes a `behavior.json` next to a
|
||||
* throwaway fixture path; the fake bin echoes observable facts (env, seeded
|
||||
* workspace, permission outcomes) into `agent_message_chunk` text, so the
|
||||
@@ -40,6 +61,181 @@ async function scenario(behavior: object): Promise<{ dir: string; fixtureFile: s
|
||||
const boot: InputStep[] = [{ op: 'initialize' }, { op: 'newSession' }]
|
||||
|
||||
describe('runScenario', () => {
|
||||
it('surfaces an asynchronous child spawn failure through startup and close', async () => {
|
||||
const { dir } = await scenario({})
|
||||
const launched = launchAcpTestAgent({ agent: AGENT, cwd: join(dir, 'missing') })
|
||||
let stdioClosed = false
|
||||
let clientClosed = false
|
||||
launched.child.once('close', () => { stdioClosed = true })
|
||||
void launched.client.closed.then(
|
||||
() => { clientClosed = true },
|
||||
() => { clientClosed = true },
|
||||
)
|
||||
await expect(launched.spawned).rejects.toMatchObject({ code: 'ENOENT' })
|
||||
await expect(launched.close()).rejects.toMatchObject({ code: 'ENOENT' })
|
||||
expect(stdioClosed).toBe(true)
|
||||
expect(clientClosed).toBe(true)
|
||||
})
|
||||
|
||||
it('centralizes ACP boot, captures, updates, fail-closed permissions, and shutdown', { timeout: 20_000 }, async () => {
|
||||
const { dir, fixtureFile } = await scenario({ permissionProbe: true, echoEnv: true, stderrNote: 'launcher stderr' })
|
||||
const sessionsRoot = await mkdtemp(join(tmpdir(), 'acp-launcher-sessions-'))
|
||||
tempDirs.push(sessionsRoot)
|
||||
const launched = launchAcpTestAgent({
|
||||
agent: AGENT,
|
||||
cwd: dir,
|
||||
configPath: AGENT.configPath,
|
||||
env: {
|
||||
DSH_SNAPSHOT: 'replay',
|
||||
DSH_SNAPSHOT_FILE: fixtureFile,
|
||||
DSH_SNAPSHOT_SESSIONS_ROOT: sessionsRoot,
|
||||
},
|
||||
})
|
||||
await launched.client.initialize({ protocolVersion: PROTOCOL_VERSION, clientCapabilities: {} })
|
||||
const { sessionId } = await launched.client.newSession({ cwd: dir, mcpServers: [] })
|
||||
const nextChunk = launched.waitForUpdate(update => update.sessionUpdate === 'agent_message_chunk')
|
||||
const predicateFailure = new Error('predicate failed')
|
||||
const failedPredicate = launched.waitForUpdate(() => { throw predicateFailure })
|
||||
.catch((error: unknown): unknown => error)
|
||||
await launched.client.prompt({ sessionId, prompt: [{ type: 'text', text: 'go' }] })
|
||||
expect(await failedPredicate).toBe(predicateFailure)
|
||||
expect((await nextChunk).sessionUpdate).toBe('agent_message_chunk')
|
||||
expect(launched.updates.some(update => update.sessionUpdate === 'agent_message_chunk')).toBe(true)
|
||||
expect(launched.rawStdout()).toContain('permission:{\\"outcome\\":\\"cancelled\\"}')
|
||||
expect(launched.stderr()).toContain('launcher stderr')
|
||||
const unmatched = expect(launched.waitForUpdate(() => false)).rejects.toThrow(/update stream closed/)
|
||||
await launched.close()
|
||||
await unmatched
|
||||
await expect(launched.waitForUpdate(() => true)).rejects.toThrow(/update stream closed/)
|
||||
await launched.close('SIGKILL')
|
||||
|
||||
// The minimal shape needs no environment or config override.
|
||||
const minimal = launchAcpTestAgent({ agent: AGENT, cwd: dir })
|
||||
await minimal.client.initialize({ protocolVersion: PROTOCOL_VERSION, clientCapabilities: {} })
|
||||
const childFailure = new Error('child process failed')
|
||||
let exited = false
|
||||
minimal.child.once('exit', () => { exited = true })
|
||||
minimal.child.emit('error', childFailure)
|
||||
await expect(minimal.close('SIGTERM')).rejects.toBe(childFailure)
|
||||
// close rejects only after the fallback SIGKILL has produced an exit edge.
|
||||
expect(exited).toBe(true)
|
||||
})
|
||||
|
||||
it('waits for inherited stdio and buffered ACP parsing after the parent exits', { timeout: 20_000 }, async () => {
|
||||
const { dir, fixtureFile } = await scenario({ lateInheritedOutput: true })
|
||||
const launched = launchAcpTestAgent({
|
||||
agent: AGENT,
|
||||
cwd: dir,
|
||||
env: { DSH_SNAPSHOT_FILE: fixtureFile },
|
||||
})
|
||||
await launched.client.initialize({ protocolVersion: PROTOCOL_VERSION, clientCapabilities: {} })
|
||||
await launched.client.newSession({ cwd: dir, mcpServers: [] })
|
||||
const lateUpdate = launched.waitForUpdate(update =>
|
||||
update.sessionUpdate === 'agent_message_chunk'
|
||||
&& update.content.type === 'text'
|
||||
&& update.content.text === 'late inherited stdout')
|
||||
|
||||
await launched.close()
|
||||
|
||||
await expect(lateUpdate).resolves.toMatchObject({ sessionUpdate: 'agent_message_chunk' })
|
||||
expect(launched.rawStdout()).toContain('late inherited stdout')
|
||||
expect(launched.stderr()).toContain('late inherited stderr')
|
||||
})
|
||||
|
||||
it('rejects promptly when fallback termination is refused', async () => {
|
||||
const { dir } = await scenario({})
|
||||
const launched = launchAcpTestAgent({ agent: AGENT, cwd: dir })
|
||||
await launched.spawned
|
||||
|
||||
const childFailure = Object.assign(new Error('signal refused'), { code: 'EPERM' })
|
||||
const originalKill = launched.child.kill.bind(launched.child)
|
||||
const kill = vi.spyOn(launched.child, 'kill').mockReturnValue(false)
|
||||
const closed = new Promise<void>(resolve => launched.child.once('close', () => { resolve() }))
|
||||
try {
|
||||
launched.child.emit('error', childFailure)
|
||||
const rejection = await launched.close('SIGTERM').catch((error: unknown): unknown => error)
|
||||
expect(rejection).toBeInstanceOf(AggregateError)
|
||||
expect(rejection).toMatchObject({
|
||||
message: 'ACP test agent failed and fallback termination was refused',
|
||||
errors: [
|
||||
childFailure,
|
||||
expect.objectContaining({ message: 'Fallback SIGKILL was not accepted by the child process' }),
|
||||
],
|
||||
})
|
||||
expect(kill).toHaveBeenNthCalledWith(1, 'SIGTERM')
|
||||
expect(kill).toHaveBeenNthCalledWith(2, 'SIGKILL')
|
||||
} finally {
|
||||
kill.mockRestore()
|
||||
originalKill('SIGKILL')
|
||||
await closed
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects promptly when fallback termination emits an error', async () => {
|
||||
const { dir } = await scenario({})
|
||||
const launched = launchAcpTestAgent({ agent: AGENT, cwd: dir })
|
||||
await launched.spawned
|
||||
|
||||
const childFailure = Object.assign(new Error('signal refused'), { code: 'EPERM' })
|
||||
const fallbackFailure = Object.assign(new Error('fallback signal refused'), { code: 'EPERM' })
|
||||
const originalKill = launched.child.kill.bind(launched.child)
|
||||
const kill = vi.spyOn(launched.child, 'kill').mockImplementation((signal) => {
|
||||
if (signal === 'SIGKILL') queueMicrotask(() => launched.child.emit('error', fallbackFailure))
|
||||
return signal === 'SIGKILL'
|
||||
})
|
||||
const closed = new Promise<void>(resolve => launched.child.once('close', () => { resolve() }))
|
||||
try {
|
||||
launched.child.emit('error', childFailure)
|
||||
const rejection = await launched.close('SIGTERM').catch((error: unknown): unknown => error)
|
||||
expect(rejection).toBeInstanceOf(AggregateError)
|
||||
expect(rejection).toMatchObject({
|
||||
message: 'ACP test agent failed and fallback termination was refused',
|
||||
errors: [childFailure, fallbackFailure],
|
||||
})
|
||||
expect(kill).toHaveBeenNthCalledWith(1, 'SIGTERM')
|
||||
expect(kill).toHaveBeenNthCalledWith(2, 'SIGKILL')
|
||||
} finally {
|
||||
kill.mockRestore()
|
||||
originalKill('SIGKILL')
|
||||
await closed
|
||||
}
|
||||
})
|
||||
|
||||
it('waits for in-flight client callbacks after the ACP stream closes', { timeout: 20_000 }, async () => {
|
||||
const { dir, fixtureFile } = await scenario({ permissionProbe: true })
|
||||
let releasePermission: (() => void) | undefined
|
||||
const permissionReleased = new Promise<void>((resolve) => { releasePermission = resolve })
|
||||
let markPermissionStarted: (() => void) | undefined
|
||||
const permissionStarted = new Promise<void>((resolve) => { markPermissionStarted = resolve })
|
||||
let permissionFinished = false
|
||||
const launched = launchAcpTestAgent({
|
||||
agent: AGENT,
|
||||
cwd: dir,
|
||||
env: { DSH_SNAPSHOT_FILE: fixtureFile },
|
||||
async requestPermission() {
|
||||
markPermissionStarted?.()
|
||||
await permissionReleased
|
||||
permissionFinished = true
|
||||
return { outcome: { outcome: 'cancelled' } }
|
||||
},
|
||||
})
|
||||
await launched.client.initialize({ protocolVersion: PROTOCOL_VERSION, clientCapabilities: {} })
|
||||
const { sessionId } = await launched.client.newSession({ cwd: dir, mcpServers: [] })
|
||||
void launched.client.prompt({ sessionId, prompt: [{ type: 'text', text: 'go' }] }).catch(() => undefined)
|
||||
await permissionStarted
|
||||
|
||||
const childClosed = once(launched.child, 'close')
|
||||
let closeSettled = false
|
||||
const closing = launched.close('SIGKILL').then(() => { closeSettled = true })
|
||||
await childClosed
|
||||
await launched.client.closed
|
||||
expect(closeSettled).toBe(false)
|
||||
|
||||
releasePermission?.()
|
||||
await closing
|
||||
expect(permissionFinished).toBe(true)
|
||||
})
|
||||
|
||||
it('includes agent stderr when the ACP connection closes during startup', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({ failOnBoot: true, stderrNote: 'fake agent requested startup failure' })
|
||||
await expect(runScenario(
|
||||
@@ -48,6 +244,23 @@ describe('runScenario', () => {
|
||||
)).rejects.toThrow(/agent stderr:\nfake agent requested startup failure/)
|
||||
})
|
||||
|
||||
it('preserves launch-resolution errors when no child process exists', async () => {
|
||||
const { dir, fixtureFile } = await scenario({})
|
||||
vi.stubEnv('DSH_EXAMPLE_MODE', 'lib')
|
||||
try {
|
||||
await expect(runScenario(
|
||||
{ steps: [] },
|
||||
{
|
||||
agent: { ...AGENT, binScript: join(dir, 'outside-src.ts'), libBinScript: undefined },
|
||||
mode: 'replay',
|
||||
fixtureFile,
|
||||
},
|
||||
)).rejects.toThrow(/expected a "\/src\/" segment/)
|
||||
} finally {
|
||||
vi.unstubAllEnvs()
|
||||
}
|
||||
})
|
||||
|
||||
it('drives a full turn: initialize (terminal caps), session, prompt, permission stub, harvest', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({
|
||||
permissionProbe: true,
|
||||
@@ -121,6 +334,28 @@ describe('runScenario', () => {
|
||||
expect(result.rawStdout.indexOf('thinking about it')).toBeLessThan(result.rawStdout.indexOf('cancelled'))
|
||||
})
|
||||
|
||||
it('promptAndCancel can bracket cancellation with tool-call updates', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({
|
||||
prompt: 'hang-until-cancel',
|
||||
cancelAtToolCall: true,
|
||||
cancelToolCallUpdate: true,
|
||||
})
|
||||
const result = await runScenario(
|
||||
{
|
||||
steps: [...boot, {
|
||||
op: 'promptAndCancel',
|
||||
text: 'hang',
|
||||
afterUpdate: 'tool_call',
|
||||
waitForToolCallUpdate: 'call_fake_1',
|
||||
}],
|
||||
},
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)
|
||||
expect(result.rawStdout).toContain('"sessionUpdate":"tool_call"')
|
||||
expect(result.rawStdout.indexOf('"sessionUpdate":"tool_call"')).toBeLessThan(result.rawStdout.indexOf('cancelled'))
|
||||
expect(result.rawStdout.indexOf('cancelled')).toBeLessThan(result.rawStdout.indexOf('"sessionUpdate":"tool_call_update"'))
|
||||
})
|
||||
|
||||
it('promptExpectError swallows a model-error response as the expected outcome', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({ prompt: 'error' })
|
||||
const result = await runScenario(
|
||||
@@ -138,6 +373,39 @@ describe('runScenario', () => {
|
||||
)).rejects.toThrow(/expected the prompt to fail/)
|
||||
})
|
||||
|
||||
it('reports scenario and cleanup failures together', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({ prompt: 'respond' })
|
||||
const cleanupFailure = new Error('cleanup failed')
|
||||
fsControl.cleanupFailure = cleanupFailure
|
||||
|
||||
const failure = await runScenario(
|
||||
{ steps: [...boot, { op: 'promptExpectError', text: 'fine' }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
).catch((error: unknown): unknown => error)
|
||||
|
||||
expect(failure).toBeInstanceOf(AggregateError)
|
||||
const failures = (failure as AggregateError).errors as unknown[]
|
||||
expect(failures).toHaveLength(2)
|
||||
expect(failures[0]).toBeInstanceOf(Error)
|
||||
expect((failures[0] as Error).message).toMatch(/expected the prompt to fail/)
|
||||
expect(failures[1]).toBe(cleanupFailure)
|
||||
})
|
||||
|
||||
it('reports cleanup failure after an otherwise successful scenario', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({})
|
||||
const cleanupFailure = new Error('cleanup failed')
|
||||
fsControl.cleanupFailure = cleanupFailure
|
||||
|
||||
const failure = await runScenario(
|
||||
{ steps: boot },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
).catch((error: unknown): unknown => error)
|
||||
|
||||
expect(failure).toBeInstanceOf(AggregateError)
|
||||
expect((failure as AggregateError).message).toBe('snapshot cleanup failed')
|
||||
expect((failure as AggregateError).errors as unknown[]).toEqual([cleanupFailure])
|
||||
})
|
||||
|
||||
it('newSessionExpectError swallows the rejection, with and without extra dirs', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({ rejectExtraDirs: true })
|
||||
const result = await runScenario(
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { cpSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs'
|
||||
import { cpSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
|
||||
import { rm } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
@@ -6,7 +6,6 @@ import { fileURLToPath } from 'node:url'
|
||||
import { afterAll, describe, expect, it } from 'vitest'
|
||||
import { defineAcpSnapshotSuite, type HarvestedLog, type Scenario } from '../src/index.ts'
|
||||
import {
|
||||
childFixturePaths,
|
||||
fixtureContext,
|
||||
formatSystemPromptSnapshot,
|
||||
headerChangeCount,
|
||||
@@ -16,6 +15,7 @@ import {
|
||||
normalizedToolSchemas,
|
||||
parseToolSchemasSnapshot,
|
||||
refreshFixtureReplacements,
|
||||
sessionFixtureNames,
|
||||
restorePinnedToolSchemas,
|
||||
stabilizeRefreshLog,
|
||||
unknownToolCallIds,
|
||||
@@ -24,7 +24,7 @@ import {
|
||||
/**
|
||||
* Unit tests for the suite factory, by running it: two synthetic suites over the scripted fake
|
||||
* ACP bin (./fixtures/fake-acp-agent.ts) register real describe/it trees at collection time,
|
||||
* so every factory path — golden and log compares, the per-suite header pin and its uniformity
|
||||
* so every factory path — expected-output and log comparisons, the per-suite header pin and its uniformity
|
||||
* guard, record-mode fixture write-back, skip semantics, and the fixture guard block —
|
||||
* executes as an ordinary green test.
|
||||
*
|
||||
@@ -46,7 +46,7 @@ const RECORD_SRC = fileURLToPath(new URL('./fixtures/record-suite', import.meta.
|
||||
// Replay pins explicit header classes; recording covers the default fallback.
|
||||
const REPLAY_SCENARIOS: Scenario[] = [
|
||||
{ name: 'pin-turn', hasModelTurn: true, recorded: true, pinsHeader: true, expectedHeaderChanges: 1, headerClass: 'main' },
|
||||
{ name: 'plain-turn', hasModelTurn: true, recorded: true, childSessions: 1, headerClass: 'main', configPath: AGENT.configPath },
|
||||
{ name: 'plain-turn', hasModelTurn: true, recorded: true, headerClass: 'main', configPath: AGENT.configPath },
|
||||
{ name: 'no-model', hasModelTurn: false, recorded: false, headerClass: 'main' },
|
||||
{ name: 'blocked-log', hasModelTurn: false, comparesLog: true, recorded: false, headerClass: 'main' },
|
||||
{ name: 'authored-error', hasModelTurn: true, recorded: false, overridden: true, headerClass: 'main' },
|
||||
@@ -54,17 +54,23 @@ const REPLAY_SCENARIOS: Scenario[] = [
|
||||
|
||||
const RECORD_SCENARIOS: Scenario[] = [
|
||||
{ name: 'rec-pin', hasModelTurn: true, recorded: true, pinsHeader: true },
|
||||
{ name: 'rec-child', hasModelTurn: true, recorded: true, childSessions: 1 },
|
||||
{ name: 'rec-child', hasModelTurn: true, recorded: true },
|
||||
// recorded:false in record mode → registered but skipped (never re-recorded).
|
||||
{ name: 'rec-skip', hasModelTurn: true, recorded: false, overridden: true },
|
||||
]
|
||||
|
||||
// Record/refresh modes mutate their snapshots dir, so run them on throwaway
|
||||
// copies — except record's documented bootstrap knob, which regenerates the
|
||||
// committed record fixtures/goldens in place.
|
||||
// committed record fixtures and expected outputs in place.
|
||||
const BOOTSTRAP = process.env.ACP_SNAPSHOT_SPEC_BOOTSTRAP === '1'
|
||||
const recordDir = BOOTSTRAP ? RECORD_SRC : mkdtempSync(join(tmpdir(), 'acp-snap-record-suite-'))
|
||||
if (!BOOTSTRAP) cpSync(RECORD_SRC, recordDir, { recursive: true })
|
||||
if (!BOOTSTRAP) {
|
||||
cpSync(RECORD_SRC, recordDir, { recursive: true })
|
||||
// Record mode owns its output inventory: a new scenario has no primary yet,
|
||||
// while a changed child count can leave old numbered fixtures behind.
|
||||
rmSync(join(recordDir, 'rec-pin', 'session.jsonl'))
|
||||
writeFileSync(join(recordDir, 'rec-child', 'session.2.jsonl'), 'stale child\n')
|
||||
}
|
||||
const refreshDir = mkdtempSync(join(tmpdir(), 'acp-snap-refresh-suite-'))
|
||||
cpSync(REPLAY_DIR, refreshDir, { recursive: true })
|
||||
staleRefreshFixtures(refreshDir)
|
||||
@@ -74,9 +80,9 @@ afterAll(async () => {
|
||||
})
|
||||
|
||||
function staleRefreshFixtures(dir: string): void {
|
||||
writeFileSync(join(dir, 'plain-turn', 'stdout.golden.jsonl'), 'stale stdout\n')
|
||||
writeFileSync(join(dir, 'pin-turn', 'system-prompt.golden.md'), 'STALE PROMPT\n')
|
||||
writeFileSync(join(dir, 'pin-turn', 'tool-schemas.golden.json'), '{"initial":[{"name":"stale"}],"changes":[]}\n')
|
||||
writeFileSync(join(dir, 'plain-turn', 'stdout.expected.jsonl'), 'stale stdout\n')
|
||||
writeFileSync(join(dir, 'pin-turn', 'system-prompt.expected.md'), 'STALE PROMPT\n')
|
||||
writeFileSync(join(dir, 'pin-turn', 'tool-schemas.expected.json'), '{"initial":[{"name":"stale"}],"changes":[]}\n')
|
||||
|
||||
const plainBehaviorFile = join(dir, 'plain-turn', 'behavior.json')
|
||||
const plainBehavior = JSON.parse(readFileSync(plainBehaviorFile, 'utf8')) as Record<string, unknown>
|
||||
@@ -84,12 +90,12 @@ function staleRefreshFixtures(dir: string): void {
|
||||
writeFileSync(plainBehaviorFile, `${JSON.stringify(plainBehavior, null, 2)}\n`)
|
||||
|
||||
writeFileSync(join(dir, 'blocked-log', 'session.jsonl'), [
|
||||
'{"type":"session","id":"99999999-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/blocked-cwd"}',
|
||||
'{"type":"session","id":"99999999-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/blocked-cwd","delegationDepth":0}',
|
||||
'{"type":"hook/result","seq":1,"time":13,"data":{"decision":"stale","durationMs":99}}',
|
||||
'',
|
||||
].join('\n'))
|
||||
writeFileSync(join(dir, 'authored-error', 'session.jsonl'), [
|
||||
'{"type":"session","id":"77777777-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/error-cwd"}',
|
||||
'{"type":"session","id":"77777777-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/error-cwd","delegationDepth":0}',
|
||||
'{"type":"turn/end","seq":1,"time":9,"data":{"error":"stale"}}',
|
||||
'',
|
||||
].join('\n'))
|
||||
@@ -111,7 +117,7 @@ describe('defineAcpSnapshotSuite: refresh mode', () => {
|
||||
|
||||
describe('defineAcpSnapshotSuite: refresh write-back', () => {
|
||||
it('rewrites stdout and comparable logs from a replay-mode child run', () => {
|
||||
const stdout = readFileSync(join(refreshDir, 'plain-turn', 'stdout.golden.jsonl'), 'utf8')
|
||||
const stdout = readFileSync(join(refreshDir, 'plain-turn', 'stdout.expected.jsonl'), 'utf8')
|
||||
expect(stdout).not.toContain('stale stdout')
|
||||
expect(stdout).toContain('env:{\\"mode\\":\\"replay\\"')
|
||||
expect(stdout).not.toContain('\\"mode\\":\\"refresh\\"')
|
||||
@@ -124,7 +130,7 @@ describe('defineAcpSnapshotSuite: refresh write-back', () => {
|
||||
expect(authored).toContain('"error":"model exploded"')
|
||||
expect(authored).not.toContain('"error":"stale"')
|
||||
|
||||
expect(readFileSync(join(refreshDir, 'pin-turn', 'system-prompt.golden.md'), 'utf8')).toBe([
|
||||
expect(readFileSync(join(refreshDir, 'pin-turn', 'system-prompt.expected.md'), 'utf8')).toBe([
|
||||
'SYS PROMPT',
|
||||
'',
|
||||
'<!-- request/header change 1 -->',
|
||||
@@ -134,12 +140,19 @@ describe('defineAcpSnapshotSuite: refresh write-back', () => {
|
||||
'NEW PROMPT LINE',
|
||||
'',
|
||||
].join('\n'))
|
||||
const schemas = readFileSync(join(refreshDir, 'pin-turn', 'tool-schemas.golden.json'), 'utf8')
|
||||
const schemas = readFileSync(join(refreshDir, 'pin-turn', 'tool-schemas.expected.json'), 'utf8')
|
||||
expect(schemas).toContain('"description": "D1"')
|
||||
expect(schemas).not.toContain('"name":"stale"')
|
||||
})
|
||||
})
|
||||
|
||||
describe('defineAcpSnapshotSuite: record inventory write-back', () => {
|
||||
it('creates a missing primary fixture and prunes stale child fixtures', () => {
|
||||
expect(readFileSync(join(recordDir, 'rec-pin', 'session.jsonl'), 'utf8')).toContain('"type":"session"')
|
||||
expect(() => readFileSync(join(recordDir, 'rec-child', 'session.2.jsonl'), 'utf8')).toThrow()
|
||||
})
|
||||
})
|
||||
|
||||
describe('defineAcpSnapshotSuite: registration contract', () => {
|
||||
it("throws when a scenario's header class has no pinning scenario", () => {
|
||||
expect(() => {
|
||||
@@ -179,13 +192,41 @@ describe('defineAcpSnapshotSuite: registration contract', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('childFixturePaths', () => {
|
||||
it('yields one sibling path per child, 1-based', () => {
|
||||
expect(childFixturePaths('/snap/s', 2)).toEqual(['/snap/s/session.1.jsonl', '/snap/s/session.2.jsonl'])
|
||||
describe('sessionFixtureNames', () => {
|
||||
it('orders the primary and contiguous child fixtures while ignoring other files', () => {
|
||||
expect(sessionFixtureNames([
|
||||
'stdout.expected.jsonl',
|
||||
'session.2.jsonl',
|
||||
'session.jsonl',
|
||||
'session.1.jsonl',
|
||||
'input.json',
|
||||
])).toEqual(['session.jsonl', 'session.1.jsonl', 'session.2.jsonl'])
|
||||
})
|
||||
|
||||
it('yields nothing for a single-session scenario', () => {
|
||||
expect(childFixturePaths('/snap/s', 0)).toEqual([])
|
||||
it('accepts a primary-only scenario', () => {
|
||||
expect(sessionFixtureNames(['session.jsonl'])).toEqual(['session.jsonl'])
|
||||
})
|
||||
|
||||
it('rejects a directory without the primary fixture', () => {
|
||||
expect(() => sessionFixtureNames(['session.1.jsonl'])).toThrow('missing session.jsonl')
|
||||
})
|
||||
|
||||
it('rejects gapped child fixtures', () => {
|
||||
expect(() => sessionFixtureNames(['session.jsonl', 'session.2.jsonl']))
|
||||
.toThrow('expected session.1.jsonl, found session.2.jsonl')
|
||||
})
|
||||
|
||||
it.each(['session.0.jsonl', 'session.child.jsonl', 'session.01.jsonl'])(
|
||||
'rejects invalid child fixture name %s',
|
||||
(name) => {
|
||||
expect(() => sessionFixtureNames(['session.jsonl', name]))
|
||||
.toThrow(`invalid child session fixture name: ${name}`)
|
||||
},
|
||||
)
|
||||
|
||||
it('rejects duplicate child indexes', () => {
|
||||
expect(() => sessionFixtureNames(['session.jsonl', 'session.1.jsonl', 'session.1.jsonl']))
|
||||
.toThrow('expected session.2.jsonl, found session.1.jsonl')
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# `@deepseek-ai/dsh-agent-loop-testkit`
|
||||
|
||||
Shared prerequisite mounting for tests that exercise the concrete `AgentLoop`. `mountAgentLoopTestDependencies(ctx, options?)` installs the LLM, session, system-prompt, tool, agent, and agent-execution services in dependency order, then returns before the loop is mounted.
|
||||
Shared prerequisite mounting for tests that exercise the concrete `AgentLoop`. `mountAgentLoopTestDependencies(ctx, options?)` installs the LLM, session, system-prompt, tool, and agent services in dependency order, then returns before the loop is mounted.
|
||||
|
||||
The caller registers adapters and optional plugins, mounts `AgentLoop` with the configuration under test, and disposes its own Context. System-prompt and tool-registry configuration can be forwarded through `options`; the helper does not provide test defaults beyond those owned by the services. A plugin-load failure rejects the helper call, while services activated earlier in the sequence remain owned by the caller's Context.
|
||||
|
||||
@@ -22,6 +22,10 @@ Tests of injection failures, partial topology, service load order, or service te
|
||||
|
||||
None, as this test-only composition helper neither drives nor modifies model requests.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
None; this package neither assembles nor sends a provider request.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **Only the mandatory prerequisite spine is shared** — adapters, optional plugins, `AgentLoop`, agents, and Context teardown remain caller-owned so scenario-specific ordering stays visible.
|
||||
|
||||
@@ -23,7 +23,6 @@
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-agent": "^0.0.1",
|
||||
"@deepseek-ai/dsh-agent-execution": "^0.0.1",
|
||||
"@deepseek-ai/dsh-llm": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session": "^0.0.1",
|
||||
"@deepseek-ai/dsh-system-prompt": "^0.0.1",
|
||||
@@ -32,7 +31,6 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-agent": "workspace:^",
|
||||
"@deepseek-ai/dsh-agent-execution": "workspace:^",
|
||||
"@deepseek-ai/dsh-agent-loop": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
|
||||
@@ -7,7 +7,6 @@
|
||||
|
||||
import type { Context } from 'cordis'
|
||||
import AgentRegistry from '@deepseek-ai/dsh-agent'
|
||||
import AgentExecutionProvider from '@deepseek-ai/dsh-agent-execution'
|
||||
import LlmService from '@deepseek-ai/dsh-llm'
|
||||
import SessionStore from '@deepseek-ai/dsh-session'
|
||||
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
|
||||
@@ -44,5 +43,4 @@ export async function mountAgentLoopTestDependencies(
|
||||
await ctx.plugin(SystemPrompt, options.systemPrompt ?? {})
|
||||
await ctx.plugin(ToolRegistry, options.tools ?? {})
|
||||
await ctx.plugin(AgentRegistry)
|
||||
await ctx.plugin(AgentExecutionProvider)
|
||||
}
|
||||
|
||||
@@ -17,9 +17,6 @@
|
||||
{
|
||||
"path": "../../core/agent"
|
||||
},
|
||||
{
|
||||
"path": "../../core/agent-execution"
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm"
|
||||
},
|
||||
|
||||
@@ -4,7 +4,7 @@ Runtime event-contract assertions intended for development diagnostics. This pur
|
||||
|
||||
The plugin has no environment guard: it is active wherever it is registered. The default [`dsh-agent-spine-demo`](../../examples/agent-spine-demo/README.md) bundle mounts it unconditionally; a custom composition can omit it when the runtime cost is undesirable. It doubles as executable documentation of the event taxonomy — the assertions *are* the contract.
|
||||
|
||||
Session itself owns immutable, surface-valid log storage in every composition: it takes one lossless JSON snapshot of each candidate, validates the complete surface transition, deep-freezes the accepted record, and exposes the log through immutable array snapshots. The invariants plugin checks the remaining cross-record and cross-seam rules that Session does not own.
|
||||
Session itself owns immutable, surface-valid log storage in every composition: it takes one lossless JSON snapshot of each candidate, validates complete provenance and positional replacement, restricts `tool/result` replacement to one current result's `content`, deep-freezes the accepted record, and exposes the log through immutable array snapshots. The invariants plugin checks the remaining cross-record and cross-seam rules that Session does not own.
|
||||
|
||||
Session-log assertions run during Cordis `internal/dispatch`, while `Session.append()` is resolving the `session/event` callback snapshot but before it pushes the candidate into the log. A valid transition is staged by exact event identity and applied to the live trace only when that same committed event reaches the plugin's contained post-commit listener. A later internal dispatch check can therefore veto without advancing either the log or the invariant trace, while ordinary `session/event` observer failures remain observe-only.
|
||||
|
||||
@@ -31,8 +31,7 @@ Session log (per session):
|
||||
- **turns pair and nest** — `turn/start` opens a turn, `turn/end` closes the matching one; no overlapping turns.
|
||||
- **steps nest in turns** — `step/start` opens a step in the open turn; `step/end` closes the matching step.
|
||||
- **chunks belong to an open step** — `step/start` precedes its `assistant/chunk`s.
|
||||
- **a `tool/result` needs a prior `tool/call`** — but NOT the converse: a `tool/call` may have no result (a thrown tool-execution pipeline step ends the turn with no `tool/result`, which is legal).
|
||||
- **provenance sources are valid and unambiguous** — `sourceEventSeqs` contains unique earlier known seqs; only `assistant/message` may carry an explicit empty list, which denotes a known empty provider stream rather than absent legacy provenance.
|
||||
- **an appended `tool/result` needs a prior `tool/call`** — fresh `surfaceOp: 'append'` results name the open step and consume its pending call. A Session-validated replacement is a turn-enclosed rewrite, not another execution. A `tool/call` may still have no result when the execution pipeline throws.
|
||||
|
||||
Agent status (per agent):
|
||||
|
||||
@@ -40,13 +39,13 @@ Agent status (per agent):
|
||||
|
||||
Model requests (on `llm/stream`):
|
||||
|
||||
- **a loop-built request is exactly what the log reconstructs** — a frozen request with a live `sessionId` (the loop-built marker; hand-built one-shots like compaction's summarize are unfrozen and skipped) must carry frozen `messages` deep-equal to the derivation over the log prefix strictly before the in-flight step's `step/start` (rebuilt through a FRESH `Session`, so the live cache cannot vouch for itself — and boundary-correct: content logged after `step/start` legitimately belongs to the next request), and every non-content field must equal the latest logged `request/header` (see [the reconstructability RFC](../../../docs/rfc/implemented/architecture/2026-07-05-reconstructable-requests.md)). Registered with `prepend: true` so a short-circuiting `llm/stream` listener (the replay adapter) cannot silence it; prepend orders it against append-registered listeners only — correctness rests on the seq-bounded rebuild, never listener timing.
|
||||
- **a loop-built request is exactly what the log reconstructs** — a frozen request with a live `sessionId` (the loop-built marker; hand-built one-shots like compaction's summarize are unfrozen and skipped) must carry frozen `messages` deep-equal to the derivation over the log prefix strictly before the in-flight step's `step/start` (rebuilt through a FRESH `Session`, so the live cache cannot vouch for itself — and boundary-correct: content logged after `step/start` legitimately belongs to the next request), and every non-content field must equal the latest logged `request/header` (see [the reconstructability Agent Note](../../../.agents/notes/implemented/architecture/2026-07-05-reconstructable-requests.md)). Registered with `prepend: true` so a short-circuiting `llm/stream` listener (the replay adapter) cannot silence it; prepend orders it against append-registered listeners only — correctness rests on the seq-bounded rebuild, never listener timing.
|
||||
|
||||
On any violation it throws `InvariantError` (`code: 'INVARIANT'`).
|
||||
|
||||
## Why runtime assertions remain useful
|
||||
|
||||
Session enforces the per-record storage boundary at runtime, where a cast cannot bypass it. Pervasive `DeepReadonly<SessionEvent>` types would add noise across consumers without expressing relationships such as turn/step nesting, subject-correct scoped dispatch, or equality between a request and its log reconstruction. This plugin checks those relationships wherever it is mounted while `dsh-session` keeps history immutable in every composition. See [source-owned session immutability and dev-mode invariants](../../../docs/rfc/implemented/architecture/2026-06-11-dev-invariants-over-deep-readonly.md).
|
||||
Session enforces the per-record storage boundary at runtime, where a cast cannot bypass it. Pervasive `DeepReadonly<SessionEvent>` types would add noise across consumers without expressing relationships such as turn/step nesting, subject-correct scoped dispatch, or equality between a request and its log reconstruction. This plugin checks those relationships wherever it is mounted while `dsh-session` keeps history immutable in every composition. See [source-owned session immutability and dev-mode invariants](../../../.agents/notes/implemented/architecture/2026-06-11-dev-invariants-over-deep-readonly.md).
|
||||
|
||||
## Seeded sessions
|
||||
|
||||
@@ -56,6 +55,10 @@ A seeded or forked session arrives with events already in its log because constr
|
||||
|
||||
None, as this observer only validates events and frozen requests and never rewrites prompts, schemas, messages, or streams.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
None; this package neither assembles nor sends a provider request.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **The request-reconstructability assertion covers loop-built requests only** — hand-built one-shots (e.g. compaction's summarize call) carry no live `sessionId` marker and are skipped.
|
||||
|
||||
@@ -86,7 +86,7 @@ function validateEvent(trace: SessionTrace, event: SessionEvent): SessionTraceTr
|
||||
|
||||
// Boundary/step-scoped events have explicit cases; every OTHER event type —
|
||||
// including plugin-added (merge-extensible) SessionEventMap keys — is caught
|
||||
// by the `default` and must be turn-enclosed (the turn-enclosure RFC). No assertNever: an
|
||||
// by the `default` and must be turn-enclosed (the turn-enclosure Agent Note). No assertNever: an
|
||||
// unknown variant is valid, not a compile error.
|
||||
switch (event.type) {
|
||||
case 'turn/start': {
|
||||
@@ -151,6 +151,16 @@ function validateEvent(trace: SessionTrace, event: SessionEvent): SessionTraceTr
|
||||
break
|
||||
}
|
||||
case 'tool/result': {
|
||||
// Session has already validated a provenance-backed content rewrite.
|
||||
// It is durable turn work, not a second execution of the original call.
|
||||
if (event.surfaceOp !== 'append') {
|
||||
if (trace.openTurn === null) {
|
||||
throw new InvariantError(
|
||||
'tool/result surface replacement appended outside any open turn',
|
||||
)
|
||||
}
|
||||
break
|
||||
}
|
||||
requireOpenStep(trace, 'tool/result', event.data.turn, event.data.step)
|
||||
// A result needs a prior matching call in the same step. (The converse
|
||||
// does NOT hold: a call may have no result — a throwing tool-execution
|
||||
@@ -162,7 +172,7 @@ function validateEvent(trace: SessionTrace, event: SessionEvent): SessionTraceTr
|
||||
pendingCalls = { kind: 'delete', callId: event.data.callId }
|
||||
break
|
||||
}
|
||||
// Turn-enclosure (the turn-enclosure RFC): EVERY session event not handled by a boundary
|
||||
// Turn-enclosure (the turn-enclosure Agent Note): EVERY session event not handled by a boundary
|
||||
// case above must sit inside an open turn. The durable session log uses the
|
||||
// turn as its commit/replay boundary (the JSONL backend treats anything
|
||||
// after the last turn/end as a crash tail), so a bare event between turns is
|
||||
@@ -330,7 +340,7 @@ export function apply(ctx: Context): void {
|
||||
}
|
||||
}, { global: true })
|
||||
|
||||
// Request-reconstruction cross-check (the reconstructability RFC): a
|
||||
// Request-reconstruction cross-check (the reconstructability Agent Note): a
|
||||
// loop-built request — frozen envelope + live sessionId is the marker; a
|
||||
// hand-built one-shot (compaction summarize) is unfrozen and skipped — must
|
||||
// be EXACTLY what the session log reconstructs:
|
||||
|
||||
@@ -30,10 +30,12 @@ const scopedSubjectResolvers = Object.freeze({
|
||||
'agent/created': adapt<'agent/created'>(args => args[0]),
|
||||
'agent/disposed': adapt<'agent/disposed'>(args => args[0]),
|
||||
'agent/error': adapt<'agent/error'>(args => args[0]),
|
||||
'agent/post-step': adapt<'agent/post-step'>(args => args[0]),
|
||||
'agent/pre-step': adapt<'agent/pre-step'>(args => args[0]),
|
||||
'agent/prompt-submit': adapt<'agent/prompt-submit'>(args => args[0]),
|
||||
'agent/queued': adapt<'agent/queued'>(args => args[0]),
|
||||
'agent/request': adapt<'agent/request'>(args => args[0]),
|
||||
'agent/request-error': adapt<'agent/request-error'>(args => args[0]),
|
||||
'agent/session-prefix': adapt<'agent/session-prefix'>(args => args[0]),
|
||||
'agent/session-start': adapt<'agent/session-start'>(args => args[0]),
|
||||
'agent/status': adapt<'agent/status'>(args => args[0]),
|
||||
|
||||
@@ -149,7 +149,7 @@ describe('session-log invariants', () => {
|
||||
it('rejects a message event appended outside any open turn (turn-enclosure)', async () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
// No turn open: every message-bearing event must be turn-enclosed (the turn-enclosure RFC).
|
||||
// No turn open: every message-bearing event must be turn-enclosed (the turn-enclosure Agent Note).
|
||||
expect(() => session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }, { surfaceOp: 'append' }))
|
||||
.toThrow(/outside any open turn/)
|
||||
expect(() => session.append('context/message', { content: [{ type: 'text', text: 'ctx' }], source: { kind: 'user' } }, { surfaceOp: 'append' }))
|
||||
@@ -160,7 +160,7 @@ describe('session-log invariants', () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
// steering/message is turn-scoped: outside a turn it would land past the
|
||||
// commit boundary and be dropped on resume (the turn-enclosure RFC).
|
||||
// commit boundary and be dropped on resume (the turn-enclosure Agent Note).
|
||||
expect(() => session.append('steering/message', { turn: 1, content: [{ type: 'text', text: 'go' }], source: { kind: 'user' } }, { surfaceOp: 'append' }))
|
||||
.toThrow(/outside any open turn/)
|
||||
// A PLUGIN-added (merge-extensible) event type is caught by the default too.
|
||||
@@ -189,6 +189,19 @@ describe('session-log invariants', () => {
|
||||
.toThrow(/no prior tool\/call/)
|
||||
})
|
||||
|
||||
it('keeps fresh tool-result appends open-step and pending-call checked', async () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
expect(() => session.append('tool/result', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
callId: CallId('closed'),
|
||||
content: [],
|
||||
isError: false,
|
||||
}, { surfaceOp: 'append' })).toThrow(/open is turn 1\/step null/)
|
||||
})
|
||||
|
||||
it('allows a synthetic interrupted tool/result from crash repair without a prior tool/call event', async () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
@@ -465,6 +478,41 @@ describe('HMR safety', () => {
|
||||
})
|
||||
|
||||
describe('surface contract under the invariants composition', () => {
|
||||
async function toolResultRewriteFixture(openRewriteTurn = true) {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
const unrelated = session.append('user/message', {
|
||||
content: [{ type: 'text', text: 'request' }],
|
||||
source: { kind: 'user' },
|
||||
}, { surfaceOp: 'append' })
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
session.append('tool/call', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
callId: CallId('rewrite'),
|
||||
name: 'echo',
|
||||
arguments: '{}',
|
||||
})
|
||||
const originalData = {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
callId: CallId('rewrite'),
|
||||
content: [{ type: 'text' as const, text: 'original' }],
|
||||
isError: true,
|
||||
error: { name: 'ExitError', code: 'EXIT_1' },
|
||||
meta: { presentation: { kind: 'terminal', output: 'full output' } },
|
||||
futureField: { nested: ['preserve', 1] },
|
||||
}
|
||||
const original = session.append('tool/result', originalData, { surfaceOp: 'append' })
|
||||
session.append('step/end', { turn: 1, step: 1 })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
if (openRewriteTurn) {
|
||||
session.append('turn/start', { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
}
|
||||
return { session, unrelated, original }
|
||||
}
|
||||
|
||||
it('accepts well-formed surface metadata', async () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
@@ -487,6 +535,71 @@ describe('surface contract under the invariants composition', () => {
|
||||
// no throw — well-formed replace op
|
||||
})
|
||||
|
||||
it('treats a provenance-backed tool-result replacement as a turn-enclosed rewrite', async () => {
|
||||
const { session, original } = await toolResultRewriteFixture()
|
||||
|
||||
expect(() => session.append('tool/result', {
|
||||
...original.data,
|
||||
content: [{ type: 'text', text: 'pruned' }],
|
||||
}, {
|
||||
surfaceOp: { op: 'replace', start: original.seq, end: original.seq },
|
||||
sourceEventSeqs: [original.seq],
|
||||
})).not.toThrow()
|
||||
})
|
||||
|
||||
it('rejects a tool-result replacement outside a turn', async () => {
|
||||
const { session, original } = await toolResultRewriteFixture(false)
|
||||
|
||||
expect(() => session.append('tool/result', {
|
||||
...original.data,
|
||||
content: [{ type: 'text', text: 'pruned' }],
|
||||
}, {
|
||||
surfaceOp: { op: 'replace', start: original.seq, end: original.seq },
|
||||
sourceEventSeqs: [original.seq],
|
||||
})).toThrow(/outside any open turn/)
|
||||
})
|
||||
|
||||
it('rejects a tool-result replacement targeting an unrelated current node', async () => {
|
||||
const { session, unrelated, original } = await toolResultRewriteFixture()
|
||||
expect(() => session.append('tool/result', {
|
||||
...original.data,
|
||||
content: [{ type: 'text', text: 'forged' }],
|
||||
}, {
|
||||
surfaceOp: { op: 'replace', start: unrelated.seq, end: unrelated.seq },
|
||||
sourceEventSeqs: [unrelated.seq],
|
||||
})).toThrow(/must target a current tool\/result/)
|
||||
})
|
||||
|
||||
it('rejects a multi-node tool-result replacement even with complete provenance', async () => {
|
||||
const { session, unrelated, original } = await toolResultRewriteFixture()
|
||||
expect(() => session.append('tool/result', {
|
||||
...original.data,
|
||||
content: [{ type: 'text', text: 'forged' }],
|
||||
}, {
|
||||
surfaceOp: { op: 'replace', start: unrelated.seq, end: original.seq },
|
||||
sourceEventSeqs: [unrelated.seq, original.seq],
|
||||
})).toThrow(/must rewrite exactly one current node/)
|
||||
})
|
||||
|
||||
it.each([
|
||||
['callId', { callId: CallId('forged') }],
|
||||
['turn', { turn: 2 }],
|
||||
['step', { step: 2 }],
|
||||
['error', { error: { name: 'ExitError', code: 'DIFFERENT' } }],
|
||||
['meta', { meta: { presentation: { kind: 'generic' } } }],
|
||||
['future data', { futureField: { nested: ['changed'] } }],
|
||||
])('rejects a content rewrite with altered %s', async (_label, altered) => {
|
||||
const { session, original } = await toolResultRewriteFixture()
|
||||
expect(() => session.append('tool/result', {
|
||||
...original.data,
|
||||
...altered,
|
||||
content: [{ type: 'text', text: 'pruned' }],
|
||||
}, {
|
||||
surfaceOp: { op: 'replace', start: original.seq, end: original.seq },
|
||||
sourceEventSeqs: [original.seq],
|
||||
})).toThrow(/may change only content/)
|
||||
})
|
||||
|
||||
it('accepts known-empty assistant provenance and rejects empty provenance elsewhere', async () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
@@ -814,8 +927,8 @@ describe('scoped-dispatch invariants', () => {
|
||||
['agent/status', [agent, 'idle']],
|
||||
['agent/queued', [agent, [], { source: { kind: 'user' }, steering: false }]],
|
||||
['agent/session-start', [agent, 'startup']],
|
||||
['agent/pre-step', [agent, 1, 1, '', new AbortController().signal]],
|
||||
['agent/prompt-submit', [agent, [], { kind: 'user' }, () => Promise.resolve({ kind: 'allow' })]],
|
||||
['agent/pre-step', [agent, 1, 1, new AbortController().signal]],
|
||||
['agent/prompt-submit', [agent, [], { kind: 'user' }, new AbortController().signal, () => Promise.resolve({ kind: 'allow' })]],
|
||||
['agent/request', [agent, 1, 1, { model: 'm' }, () => Promise.resolve({ model: 'm' })]],
|
||||
['agent/session-prefix', [agent, [], new AbortController().signal, () => Promise.resolve([])]],
|
||||
['agent/step-result', [agent, 1, 1, { role: 'assistant', content: [] }, () => Promise.resolve({ role: 'assistant', content: [] })]],
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
A replay LLM plugin for keyless snapshot tests. It yields model streams reconstructed from a recorded **session JSONL** fixture, so a test can boot the real agent against a fixed model transcript with no API key. With `providers` configured it registers a replay-only adapter whose catalog is visible to clients such as ACP editors; without `providers` it installs the catch-all `llm/stream` waterfall used by tests that do not need discovery.
|
||||
|
||||
Its consumer is the ACP snapshot harness in `examples/acp-agent`, which loads this plugin (via `cordis.snapshot.yml`) in place of a real LLM adapter. The package exists so its derive/parse/replay logic falls under the per-file 100% coverage gate on `packages/*/src` (the same logic, while it lived under `examples/`, was outside the gate).
|
||||
Its consumers are the ACP snapshot harness in `examples/acp-agent` and the `stream-json` snapshot in `examples/headless-agent`; each loads this plugin in place of a real LLM adapter. Keeping derivation and replay here places that logic under the per-file 100% coverage gate on `packages/*/src`.
|
||||
|
||||
## How the fixture works
|
||||
|
||||
@@ -56,6 +56,10 @@ Named `name` / `inject` / `Config` / `apply`, with **no default export**: the co
|
||||
|
||||
None, as this keyless test adapter sends no request to a provider model; it only replays recorded assistant chunks into the test loop.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
None; this package neither assembles nor sends a provider request.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **First-call-order script binding assumes sequential delegation** — a cut that runs sibling subagents concurrently (or a compaction summarize call landing mid-run) would bind live sessions to recorded scripts non-deterministically; a stronger keying is deferred until such a scenario exists (`XXX(concurrent-subagents)`).
|
||||
|
||||
@@ -20,7 +20,7 @@ import { LlmAdapter, LlmError, assertNever } from '@deepseek-ai/dsh-llm'
|
||||
*/
|
||||
export type ReplayEntry =
|
||||
| { kind: 'chunks'; chunks: StreamChunk[] }
|
||||
| { kind: 'throw'; chunks: StreamChunk[]; message: string; code: string; status?: number }
|
||||
| { kind: 'throw'; chunks: StreamChunk[]; message: string; code: string }
|
||||
| { kind: 'hang' }
|
||||
|
||||
/** One model exposed by a replay-only provider catalog. */
|
||||
@@ -283,7 +283,7 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined)
|
||||
if (signal?.aborted) throw new Error('aborted')
|
||||
yield chunk
|
||||
}
|
||||
throw new LlmError(entry.message, entry.code, entry.status)
|
||||
throw new LlmError(entry.message, entry.code)
|
||||
case 'hang':
|
||||
// Replay a stream that stalls until cancelled (mirrors MockAdapter): one
|
||||
// chunk, then wait for abort and surface it as the consumer expects.
|
||||
|
||||
@@ -175,7 +175,7 @@ describe('loadReplayScript', () => {
|
||||
it('uses the sidecar override when present, ignoring the JSONL', () => {
|
||||
writeFileSync(file, sessionJsonl([]), 'utf8')
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
const override: ReplayEntry[] = [{ kind: 'throw', chunks: [], message: '401', code: 'AUTH', status: 401 }]
|
||||
const override: ReplayEntry[] = [{ kind: 'throw', chunks: [], message: '401', code: 'AUTH' }]
|
||||
writeFileSync(overrideFile, JSON.stringify(override), 'utf8')
|
||||
expect(loadReplayScript({ file, overrideFile })).toEqual(override)
|
||||
})
|
||||
@@ -265,12 +265,12 @@ describe('installLlmReplay (through the real LlmService)', () => {
|
||||
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(second)
|
||||
})
|
||||
|
||||
it('replays a sidecar throw-entry as an LlmError with code/status, after its prefix chunks', async () => {
|
||||
it('replays a sidecar throw-entry as an LlmError with its stable code, after its prefix chunks', async () => {
|
||||
writeFileSync(file, sessionJsonl([]), 'utf8')
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
const partial: StreamChunk[] = [{ type: 'block-start', index: 0, blockType: 'text' }]
|
||||
writeFileSync(overrideFile, JSON.stringify([
|
||||
{ kind: 'throw', chunks: partial, message: 'unauthorized', code: 'AUTH', status: 401 },
|
||||
{ kind: 'throw', chunks: partial, message: 'unauthorized', code: 'AUTH' },
|
||||
]), 'utf8')
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
@@ -279,7 +279,7 @@ describe('installLlmReplay (through the real LlmService)', () => {
|
||||
const seen: StreamChunk[] = []
|
||||
await expect((async () => {
|
||||
for await (const c of ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })) seen.push(c)
|
||||
})()).rejects.toMatchObject({ message: 'unauthorized', code: 'AUTH', status: 401 })
|
||||
})()).rejects.toMatchObject({ message: 'unauthorized', code: 'AUTH' })
|
||||
expect(seen).toEqual(partial)
|
||||
})
|
||||
|
||||
@@ -384,7 +384,7 @@ describe('installLlmReplay (through the real LlmService)', () => {
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
const partial: StreamChunk[] = [{ type: 'block-start', index: 0, blockType: 'text' }]
|
||||
writeFileSync(overrideFile, JSON.stringify([
|
||||
{ kind: 'throw', chunks: partial, message: 'unauthorized', code: 'AUTH', status: 401 },
|
||||
{ kind: 'throw', chunks: partial, message: 'unauthorized', code: 'AUTH' },
|
||||
]), 'utf8')
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
Shared subprocess harness for tests that boot an app and `cordis.yml` through the Cordis Loader. `resolveExampleLaunch` selects local `src` mode (tsx and root tsconfig paths) or CI `lib` mode (plain Node and package exports) from an explicit mode or `DSH_EXAMPLE_MODE`.
|
||||
|
||||
`runLoaderSmoke` owns the isolated cwd, DSH homes, stdin, diagnostics, deadline, termination, and cleanup. It returns both streams after a zero exit and rejects with both streams on failure.
|
||||
`runLoaderSmoke` accepts bin and config paths, optional complete bin arguments, environment overrides, stdin, pre-run setup, and pre-cleanup inspection. It owns the isolated cwd, DSH homes, diagnostics, deadline, termination, EOF, and cleanup; it returns both streams after a zero exit and rejects with both streams on failure.
|
||||
|
||||
This is support-tier test infrastructure, not product API.
|
||||
|
||||
@@ -10,6 +10,10 @@ This is support-tier test infrastructure, not product API.
|
||||
|
||||
None, as this test-only harness boots example processes and inspects their streams without changing an assembled model request.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
None; this package neither assembles nor sends a provider request.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **Built mode requires a prior build** — the config must also resolve every named package upward through `examples/node_modules`.
|
||||
|
||||
@@ -1,14 +1,12 @@
|
||||
/**
|
||||
* Shared subprocess harness for keyless example smokes that boot a real
|
||||
* `cordis.yml` through the stdio-agent bin and Cordis Loader.
|
||||
* `cordis.yml` through an app bin and Cordis Loader.
|
||||
*
|
||||
* It also owns the mode-aware launch resolver every example subprocess harness shares
|
||||
* ({@link resolveExampleLaunch}): booting an example bin from TypeScript source under `tsx` (the
|
||||
* zero-build dev path, resolving `@deepseek-ai/dsh-*` / `@cordisjs/*` through the tsconfig `paths`
|
||||
* map) or from built `lib/` under plain Node (resolving bare packages through real `exports`, as an
|
||||
* installed consumer does, while Node type-strips relative example-local TypeScript plugins).
|
||||
* Consolidating that spawn glue here retires the copies in the ACP snapshot harness and the example
|
||||
* e2e drivers (the `TODO(acp-test-harness)`).
|
||||
*
|
||||
* @module @deepseek-ai/dsh-loader-smoke
|
||||
*/
|
||||
@@ -126,12 +124,14 @@ export interface LoaderSmokeOptions {
|
||||
readonly label: string
|
||||
/** Prefix for the isolated temporary process cwd. */
|
||||
readonly tempDirPrefix: string
|
||||
/** Absolute stdio-agent bin SOURCE path (`<pkg>/src/bin.ts`); the `lib` bin is derived from it. */
|
||||
/** Absolute app-bin source path (`<pkg>/src/bin.ts`); the `lib` bin is derived from it. */
|
||||
readonly binScript: string
|
||||
/** Explicit plain-Node entry for `lib` mode; intended for test fixtures outside a package `src/` tree. */
|
||||
readonly libBinScript?: string | undefined
|
||||
/** Absolute real Loader config path. */
|
||||
/** Absolute real Loader config path, passed as the sole bin argument by default. */
|
||||
readonly configPath: string
|
||||
/** Complete argv after the bin path; overrides the default `[configPath]`. */
|
||||
readonly binArgs?: readonly string[]
|
||||
/** Absolute repo tsconfig path used for unbuilt workspace-package resolution (required in `src` mode). */
|
||||
readonly tsconfigPath: string
|
||||
/** Boot from source via tsx (`src`) or built lib via plain Node (`lib`); defaults to the environment's mode. */
|
||||
@@ -142,6 +142,10 @@ export interface LoaderSmokeOptions {
|
||||
readonly stdinLines?: readonly string[]
|
||||
/** Process deadline override for harness tests. */
|
||||
readonly processTimeoutMs?: number
|
||||
/** Optional world-state setup run in the isolated cwd before process start. */
|
||||
readonly prepare?: (cwd: string) => Promise<void> | void
|
||||
/** Optional world-state assertion run in the isolated cwd before cleanup. */
|
||||
readonly inspect?: (cwd: string) => Promise<void> | void
|
||||
}
|
||||
|
||||
/** Captured output from a Loader smoke that exited successfully. */
|
||||
@@ -162,17 +166,18 @@ export interface LoaderSmokeResult {
|
||||
export async function runLoaderSmoke(options: LoaderSmokeOptions): Promise<LoaderSmokeResult> {
|
||||
const cwd = await mkdtemp(join(tmpdir(), options.tempDirPrefix))
|
||||
const processTimeoutMs = options.processTimeoutMs ?? DEFAULT_PROCESS_TIMEOUT_MS
|
||||
const launch = resolveExampleLaunch({
|
||||
srcBin: options.binScript,
|
||||
libBin: options.libBinScript,
|
||||
configArgs: [options.configPath],
|
||||
...options.mode !== undefined ? { mode: options.mode } : {},
|
||||
tsconfigPath: options.tsconfigPath,
|
||||
exposeInternals: true,
|
||||
env: { DSH_HOME: join(cwd, '.dsh'), DSH_AGENTS_HOME: join(cwd, '.agents'), ...options.env },
|
||||
})
|
||||
try {
|
||||
return await new Promise((resolve, reject) => {
|
||||
await options.prepare?.(cwd)
|
||||
const launch = resolveExampleLaunch({
|
||||
srcBin: options.binScript,
|
||||
libBin: options.libBinScript,
|
||||
configArgs: options.binArgs ?? [options.configPath],
|
||||
...options.mode !== undefined ? { mode: options.mode } : {},
|
||||
tsconfigPath: options.tsconfigPath,
|
||||
exposeInternals: true,
|
||||
env: { DSH_HOME: join(cwd, '.dsh'), DSH_AGENTS_HOME: join(cwd, '.agents'), ...options.env },
|
||||
})
|
||||
const result = await new Promise<LoaderSmokeResult>((resolve, reject) => {
|
||||
const child = spawn(launch.command, launch.args, {
|
||||
cwd,
|
||||
env: { ...process.env, ...launch.env },
|
||||
@@ -217,6 +222,8 @@ export async function runLoaderSmoke(options: LoaderSmokeOptions): Promise<Loade
|
||||
|
||||
child.stdin.end((options.stdinLines ?? []).map(line => `${line}\n`).join(''))
|
||||
})
|
||||
await options.inspect?.(cwd)
|
||||
return result
|
||||
} finally {
|
||||
await rm(cwd, { recursive: true, force: true })
|
||||
}
|
||||
|
||||
@@ -5,7 +5,7 @@ import {
|
||||
resolveExampleMode,
|
||||
} from '@deepseek-ai/dsh-loader-smoke'
|
||||
|
||||
const SRC_BIN = '/repo/packages/examples/stdio-demo/src/bin.ts'
|
||||
const SRC_BIN = '/repo/packages/examples/tui-demo/src/bin.ts'
|
||||
const TSCONFIG = '/repo/tsconfig.json'
|
||||
|
||||
const originalMode = process.env[EXAMPLE_MODE_ENV]
|
||||
@@ -66,7 +66,7 @@ describe('resolveExampleLaunch', () => {
|
||||
env: { DSH_HOME: '/tmp/home' },
|
||||
})
|
||||
expect(args).not.toContain('--import')
|
||||
expect(args).toContain('/repo/packages/examples/stdio-demo/lib/bin.js')
|
||||
expect(args).toContain('/repo/packages/examples/tui-demo/lib/bin.js')
|
||||
expect(args.slice(-2)).toEqual(['--config', './cordis.yml'])
|
||||
expect(env.TSX_TSCONFIG_PATH).toBeUndefined()
|
||||
expect(env.DSH_HOME).toBe('/tmp/home')
|
||||
@@ -106,6 +106,6 @@ describe('resolveExampleLaunch', () => {
|
||||
it('defaults the mode from the environment', () => {
|
||||
process.env[EXAMPLE_MODE_ENV] = 'lib'
|
||||
const { args } = resolveExampleLaunch({ srcBin: SRC_BIN })
|
||||
expect(args).toContain('/repo/packages/examples/stdio-demo/lib/bin.js')
|
||||
expect(args).toContain('/repo/packages/examples/tui-demo/lib/bin.js')
|
||||
})
|
||||
})
|
||||
|
||||
@@ -6,6 +6,7 @@ process.stdin.on('data', (chunk: string) => { input += chunk })
|
||||
process.stdin.on('end', () => {
|
||||
console.log(JSON.stringify({
|
||||
configPath: process.argv[2],
|
||||
args: process.argv.slice(2),
|
||||
cwd: process.cwd(),
|
||||
dshHome: process.env.DSH_HOME,
|
||||
agentsHome: process.env.DSH_AGENTS_HOME,
|
||||
|
||||
@@ -1,4 +1,6 @@
|
||||
import { existsSync } from 'node:fs'
|
||||
import { readFile, writeFile } from 'node:fs/promises'
|
||||
import { join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { LOADER_SMOKE_TEST_TIMEOUT_MS, runLoaderSmoke } from '@deepseek-ai/dsh-loader-smoke'
|
||||
@@ -22,6 +24,7 @@ describe('runLoaderSmoke', () => {
|
||||
})
|
||||
const output = JSON.parse(result.stdout) as {
|
||||
configPath: string
|
||||
args: string[]
|
||||
cwd: string
|
||||
dshHome: string
|
||||
agentsHome: string
|
||||
@@ -30,6 +33,7 @@ describe('runLoaderSmoke', () => {
|
||||
}
|
||||
expect(output).toMatchObject({
|
||||
configPath,
|
||||
args: [configPath],
|
||||
marker: 'present',
|
||||
input: 'one\ntwo\n',
|
||||
})
|
||||
@@ -39,6 +43,30 @@ describe('runLoaderSmoke', () => {
|
||||
expect(existsSync(output.cwd)).toBe(false)
|
||||
}, LOADER_SMOKE_TEST_TIMEOUT_MS)
|
||||
|
||||
it('passes an arbitrary bin argv and inspects world state before cleanup', async () => {
|
||||
let inspected = ''
|
||||
let marker = ''
|
||||
const result = await runLoaderSmoke({
|
||||
label: 'argv fixture',
|
||||
tempDirPrefix: 'loader-smoke-argv-',
|
||||
binScript: fixture('success'),
|
||||
libBinScript: fixture('success'),
|
||||
configPath,
|
||||
binArgs: ['--config', configPath, '--output-format', 'json', 'task with spaces'],
|
||||
tsconfigPath,
|
||||
prepare: cwd => writeFile(join(cwd, 'marker.txt'), 'prepared'),
|
||||
inspect: async (cwd) => {
|
||||
inspected = cwd
|
||||
marker = await readFile(join(cwd, 'marker.txt'), 'utf8')
|
||||
},
|
||||
})
|
||||
const output = JSON.parse(result.stdout) as { args: string[]; cwd: string }
|
||||
expect(output.args).toEqual(['--config', configPath, '--output-format', 'json', 'task with spaces'])
|
||||
expect(canonicalTempPath(inspected)).toBe(canonicalTempPath(output.cwd))
|
||||
expect(marker).toBe('prepared')
|
||||
expect(existsSync(inspected)).toBe(false)
|
||||
}, LOADER_SMOKE_TEST_TIMEOUT_MS)
|
||||
|
||||
it('rejects a non-zero exit with captured diagnostics', async () => {
|
||||
await expect(runLoaderSmoke({
|
||||
label: 'failure fixture',
|
||||
|
||||
@@ -1,29 +0,0 @@
|
||||
# @deepseek-ai/dsh-subagent-mock
|
||||
|
||||
A scripted `SubagentProvider` for testing the [subagent seam](../../subagent/subagent/README.md) without a model or a real child agent — the subagent analog of [`dsh-llm-replay`](../llm-replay/README.md).
|
||||
|
||||
It lets a test drive `ctx.subagents` and the model-facing `dsh-tool-subagent` through the real Cordis loader/export path, exercising provider registration, async start, start-time capability validation, required-signal cancellation, `result`, `dispose`, and structured output deterministically and keylessly.
|
||||
|
||||
## Usage
|
||||
|
||||
Load it as a plugin (functional shape: `name`/`inject`/`Config`/`apply`, no default). Config (all optional):
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `name` | `mock` | Registry name to register the provider under. |
|
||||
| `reply` | `mock subagent reply` | The scripted child's final answer text. |
|
||||
| `stopReason` | `completed` | The stop reason `result` settles with. |
|
||||
| `capabilities` | all `true` | Which start-time capabilities (`outputSchema`, `depthLimit`, `toolFilter`, and `persona`) the provider advertises. |
|
||||
| `inheritsParentContext` | `false` | Conversation-history descriptor: `false` means fresh, while `true` exercises seeded/fork wording. It says nothing about tool, service, scope, or authority inheritance. |
|
||||
| `structured` | `{ reply }` | Structured value surfaced when a request carries an `outputSchema` and the capability is on. |
|
||||
|
||||
Aborting the required request signal or disposing before `result` settles flips the stop reason to `aborted`, so both holder-facing cancellation paths are observable.
|
||||
|
||||
## Model Experience
|
||||
|
||||
Indirectly, through `dsh-tool-subagent`, which renders this test provider's configured reply or stop-reason error into the parent test history.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **Scripted provider only** — it does not run a model, create a child agent, or exercise real prompt/tool-loop behavior.
|
||||
- **One synthetic outcome per run** — it models no multi-turn, streaming, steering, resume, or subprocess transport behavior.
|
||||
@@ -1,40 +0,0 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-subagent-mock",
|
||||
"description": "Scripted subagent provider for testing the subagent seam (keyless, deterministic)",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/types/**/*.d.ts",
|
||||
"lib/types/**/*.d.ts.map",
|
||||
"src"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-agent": "^0.0.1",
|
||||
"@deepseek-ai/dsh-llm": "^0.0.1",
|
||||
"@deepseek-ai/dsh-subagent": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"dependencies": {
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-agent": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-subagent": "workspace:^",
|
||||
"@cordisjs/plugin-loader": "^1.0.0-rc.5",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
@@ -1,126 +0,0 @@
|
||||
/**
|
||||
* Scripted, model-free subagent provider for deterministic coverage of registration,
|
||||
* capability checks, lifecycle, the model-facing tool, and structured results through the real
|
||||
* loader path. It is a named-export functional plugin; no default export.
|
||||
* @module @deepseek-ai/dsh-subagent-mock
|
||||
*/
|
||||
|
||||
import type { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import { AgentId } from '@deepseek-ai/dsh-agent'
|
||||
import type { ContentBlock } from '@deepseek-ai/dsh-llm'
|
||||
import type {
|
||||
SubagentCapabilities,
|
||||
SubagentProvider,
|
||||
SubagentResult,
|
||||
SubagentRun,
|
||||
SubagentStartRequest,
|
||||
SubagentStopReason,
|
||||
} from '@deepseek-ai/dsh-subagent'
|
||||
|
||||
const STOP_REASONS = ['completed', 'aborted', 'error', 'max-tokens', 'refusal'] as const
|
||||
|
||||
const DEFAULT_CAPS: SubagentCapabilities = { outputSchema: true, depthLimit: true, toolFilter: true, persona: true }
|
||||
|
||||
/** Scripted provider whose configured result aborts if disposed or signalled first. */
|
||||
class MockSubagentProvider implements SubagentProvider {
|
||||
readonly capabilities: SubagentCapabilities
|
||||
readonly inheritsParentContext: boolean
|
||||
|
||||
constructor(
|
||||
readonly name: string,
|
||||
private readonly config: Config,
|
||||
) {
|
||||
this.capabilities = { ...DEFAULT_CAPS, ...config.capabilities }
|
||||
this.inheritsParentContext = config.inheritsParentContext ?? false
|
||||
}
|
||||
|
||||
async start(request: SubagentStartRequest): Promise<SubagentRun> {
|
||||
if (request.signal.aborted) throw new Error('mock subagent start aborted before publication')
|
||||
const reply = this.config.reply ?? 'mock subagent reply'
|
||||
const output: ContentBlock[] = [{ type: 'text', text: reply }]
|
||||
const wantsStructured = request.outputSchema !== undefined && this.capabilities.outputSchema
|
||||
const baseStop: SubagentStopReason = this.config.stopReason ?? 'completed'
|
||||
const flags = { cancelled: false }
|
||||
const onAbort = (): void => { flags.cancelled = true }
|
||||
request.signal.addEventListener('abort', onAbort, { once: true })
|
||||
// Make publication genuinely asynchronous so a same-turn abort is still
|
||||
// a provider-owned startup failure rather than a returned live run.
|
||||
await Promise.resolve()
|
||||
if (flags.cancelled) {
|
||||
request.signal.removeEventListener('abort', onAbort)
|
||||
throw new Error('mock subagent start aborted before publication')
|
||||
}
|
||||
|
||||
// A deterministic child id derived from the parent — no clock/random (both
|
||||
// banned in deterministic paths here, and unnecessary for a scripted run).
|
||||
const id = AgentId(`mock-subagent:${this.name}:${request.parent.id}`)
|
||||
|
||||
const resultFor = (): SubagentResult => ({
|
||||
output,
|
||||
...wantsStructured ? { structured: this.config.structured ?? { reply } } : {},
|
||||
stopReason: flags.cancelled ? 'aborted' : baseStop,
|
||||
})
|
||||
|
||||
const result = new Promise<SubagentResult>((resolve) => {
|
||||
setTimeout(() => { resolve(resultFor()) }, 0)
|
||||
}).finally(() => {
|
||||
request.signal.removeEventListener('abort', onAbort)
|
||||
})
|
||||
return {
|
||||
id,
|
||||
result,
|
||||
dispose(): Promise<void> {
|
||||
flags.cancelled = true
|
||||
request.signal.removeEventListener('abort', onAbort)
|
||||
return Promise.resolve()
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export const name = 'subagent-mock'
|
||||
export const inject = ['subagents']
|
||||
|
||||
/** Config for the mock provider; all optional with test-friendly defaults. */
|
||||
export interface Config {
|
||||
/** Registry name to register under. */
|
||||
name: string
|
||||
/** The text the scripted child "returns" as its final answer. */
|
||||
reply?: string
|
||||
/** The stop reason the run settles with. */
|
||||
stopReason?: SubagentStopReason
|
||||
/** Which start-time capabilities to advertise (default: all `true`). */
|
||||
capabilities?: Partial<SubagentCapabilities>
|
||||
/**
|
||||
* The conversation-history descriptor to declare
|
||||
* ({@link SubagentProvider.inheritsParentContext}); default `false` (fresh
|
||||
* conversation). Set `true` to exercise seeded/fork wording in consumer
|
||||
* tests. This flag says nothing about tool, service, scope, or authority
|
||||
* inheritance.
|
||||
*/
|
||||
inheritsParentContext?: boolean
|
||||
/**
|
||||
* Structured value surfaced when a request carries an `outputSchema` and the
|
||||
* `outputSchema` capability is on (default: `{ reply }`).
|
||||
*/
|
||||
structured?: unknown
|
||||
}
|
||||
|
||||
export const Config: z<Config> = z.object({
|
||||
name: z.string().default('mock'),
|
||||
reply: z.string(),
|
||||
stopReason: z.union(STOP_REASONS),
|
||||
capabilities: z.object({
|
||||
outputSchema: z.boolean(),
|
||||
depthLimit: z.boolean(),
|
||||
toolFilter: z.boolean(),
|
||||
persona: z.boolean(),
|
||||
}),
|
||||
inheritsParentContext: z.boolean(),
|
||||
structured: z.any(),
|
||||
})
|
||||
|
||||
export function apply(ctx: Context, config: Config): void {
|
||||
ctx.subagents.registerProvider(new MockSubagentProvider(config.name, config))
|
||||
}
|
||||
@@ -1,120 +0,0 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import Loader from '@cordisjs/plugin-loader'
|
||||
import { AgentId, type Agent } from '@deepseek-ai/dsh-agent'
|
||||
import SubagentService, { type SubagentStartRequest } from '@deepseek-ai/dsh-subagent'
|
||||
import * as mock from '../src/index.ts'
|
||||
|
||||
/** A minimal parent — the mock provider only reads `parent.id`. */
|
||||
function fakeParent(id = 'parent-1'): Agent {
|
||||
return { id: AgentId(id) } as unknown as Agent
|
||||
}
|
||||
|
||||
function baseRequest(over: Partial<SubagentStartRequest> = {}): SubagentStartRequest {
|
||||
return { prompt: [{ type: 'text', text: 'task' }], parent: fakeParent(), signal: new AbortController().signal, ...over }
|
||||
}
|
||||
|
||||
async function mount(config: Partial<mock.Config> = {}): Promise<Context> {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SubagentService)
|
||||
await ctx.plugin(mock, { name: 'mock', ...config })
|
||||
return ctx
|
||||
}
|
||||
|
||||
describe('dsh-subagent-mock', () => {
|
||||
it('registers a provider on ctx.subagents and returns the scripted reply', async () => {
|
||||
const ctx = await mount({ reply: 'hello from mock' })
|
||||
expect(ctx.subagents.list()).toEqual(['mock'])
|
||||
|
||||
const run = await ctx.subagents.start('mock', baseRequest())
|
||||
await expect(run.result).resolves.toEqual({
|
||||
output: [{ type: 'text', text: 'hello from mock' }],
|
||||
structured: undefined,
|
||||
stopReason: 'completed',
|
||||
})
|
||||
await run.dispose()
|
||||
})
|
||||
|
||||
it('registers under a configurable name', async () => {
|
||||
const ctx = await mount({ name: 'spawn' })
|
||||
expect(ctx.subagents.list()).toEqual(['spawn'])
|
||||
})
|
||||
|
||||
it('surfaces a structured result when the request carries an outputSchema', async () => {
|
||||
const ctx = await mount({ reply: 'r', structured: { answer: 42 } })
|
||||
const run = await ctx.subagents.start('mock', baseRequest({ outputSchema: { type: 'object', properties: { answer: { type: 'number' } } } }))
|
||||
await expect(run.result).resolves.toMatchObject({ structured: { answer: 42 } })
|
||||
})
|
||||
|
||||
it('defaults structured output to { reply } when outputSchema is requested but no structured value is configured', async () => {
|
||||
const ctx = await mount({ reply: 'fallback reply' })
|
||||
const run = await ctx.subagents.start('mock', baseRequest({ outputSchema: { type: 'object', properties: { answer: { type: 'number' } } } }))
|
||||
await expect(run.result).resolves.toMatchObject({ structured: { reply: 'fallback reply' } })
|
||||
})
|
||||
|
||||
it('omits structured output when outputSchema capability is off', async () => {
|
||||
const ctx = await mount({ capabilities: { outputSchema: false } })
|
||||
// The service rejects an outputSchema request against a no-cap provider, so
|
||||
// the structured path is only reachable when the cap is on; with it off and
|
||||
// no schema requested, the result has no structured field.
|
||||
const run = await ctx.subagents.start('mock', baseRequest())
|
||||
const result = await run.result
|
||||
expect(result).not.toHaveProperty('structured')
|
||||
})
|
||||
|
||||
it('honors a configured stop reason', async () => {
|
||||
const ctx = await mount({ stopReason: 'refusal' })
|
||||
const run = await ctx.subagents.start('mock', baseRequest())
|
||||
await expect(run.result).resolves.toMatchObject({ stopReason: 'refusal' })
|
||||
})
|
||||
|
||||
it('flips the stop reason to aborted when the signal fires before the result settles', async () => {
|
||||
const ctx = await mount()
|
||||
const controller = new AbortController()
|
||||
const run = await ctx.subagents.start('mock', baseRequest({ signal: controller.signal }))
|
||||
controller.abort()
|
||||
await expect(run.result).resolves.toMatchObject({ stopReason: 'aborted' })
|
||||
})
|
||||
|
||||
it('rejects an already-aborted request before starting publication', async () => {
|
||||
const ctx = await mount()
|
||||
const controller = new AbortController()
|
||||
controller.abort()
|
||||
|
||||
await expect(ctx.subagents.start('mock', baseRequest({ signal: controller.signal })))
|
||||
.rejects.toThrow('mock subagent start aborted before publication')
|
||||
})
|
||||
|
||||
it('rejects when cancellation wins the asynchronous publication handoff', async () => {
|
||||
const ctx = await mount()
|
||||
const controller = new AbortController()
|
||||
const pending = ctx.subagents.start('mock', baseRequest({ signal: controller.signal }))
|
||||
|
||||
controller.abort()
|
||||
|
||||
await expect(pending).rejects.toThrow('mock subagent start aborted before publication')
|
||||
})
|
||||
|
||||
it('unregisters the provider when the owning fiber is disposed (HMR safety)', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SubagentService)
|
||||
const fiber = await ctx.plugin(mock, { name: 'mock' })
|
||||
expect(ctx.subagents.list()).toEqual(['mock'])
|
||||
await fiber.dispose()
|
||||
expect(ctx.subagents.list()).toEqual([])
|
||||
})
|
||||
|
||||
it('has the namespace-plugin export shape (no stray default) so the Loader keeps name/inject/Config/apply', () => {
|
||||
// A default export would make Loader unwrap only that value and drop `inject`.
|
||||
expect('default' in mock).toBe(false)
|
||||
expect(mock.name).toBe('subagent-mock')
|
||||
expect(mock.inject).toEqual(['subagents'])
|
||||
|
||||
const loader = Object.create(Loader.prototype) as Loader
|
||||
const unwrapped = loader.unwrapExports(mock) as Record<string, unknown>
|
||||
expect(unwrapped).toBe(mock)
|
||||
expect(unwrapped.name).toBe('subagent-mock')
|
||||
expect(unwrapped.inject).toEqual(['subagents'])
|
||||
expect(typeof unwrapped.apply).toBe('function')
|
||||
})
|
||||
})
|
||||
@@ -1,30 +0,0 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../../core/agent"
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm"
|
||||
},
|
||||
{
|
||||
"path": "../../subagent/subagent"
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user