Merge branch 'master' into worktree-windows-runtime
This commit is contained in:
Tianyi Cui
@@ -5,9 +5,9 @@ The ACP snapshot suite kit: the shared machinery behind the keyless snapshot tie
|
||||
Four layers, importable separately:
|
||||
|
||||
- **`launchAcpTestAgent` (launcher)** — boots a source agent under tsx or a built `lib` agent under plain Node from a temp cwd, connects the SDK client over a raw-byte stdout tee, collects session updates and stderr, surfaces asynchronous spawn failures through startup, fails closed on unhandled permission requests, and owns graceful or signalled shutdown. Shutdown waits for process exit, inherited stdio closure, and ACP parser exhaustion before resolving or propagating a child error, so captures are complete and callers can remove owned paths after either outcome. When Windows accepts forced termination but publishes its exit marker asynchronously, shutdown gives that marker a bounded grace before treating fallback refusal as a second failure. Snapshot and ordinary e2e suites share this process boundary; a test supplies only agent paths, cwd, environment overrides, and any permission policy.
|
||||
- **`runScenario` (harness)** — drives ACP JSON-RPC stdio from a deterministic `input.json` script through the launcher, tees raw stdout for the golden and purity check, and harvests every persisted session JSONL (parent and subagent children, primary-first) after graceful stdin EOF. `AgentUnderTest` supplies absolute `binScript`, optional `libBinScript`, `configPath`, and `tsconfigPath` paths because the subprocess cwd is outside the repo. Startup failures preserve captured agent stderr in the rejected diagnostic.
|
||||
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; cwd-rooted separators selected as canonical `/` or host-native; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept, the same cwd-path policy), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header RFC](../../../docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
|
||||
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario shared golden and re-persisted-log compares, optional Windows-native stdout sidecars, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, the per-header-class pin (`system-prompt.golden.md` plus `tool-schemas.golden.json`) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt/schema-scrubbed, non-pinning fixtures fully header-scrubbed). Each scenario directory's `session.jsonl` plus contiguous `session.<n>.jsonl` siblings are the ordered primary/child inventory; the scenario table does not duplicate their count. Must be called at vitest collection time.
|
||||
- **`runScenario` (harness)** — drives ACP JSON-RPC stdio from a deterministic `input.json` script through the launcher, tees raw stdout for the expected-output and purity checks, and harvests every persisted raw JSONL session log (parent and subagent children, primary-first) after graceful stdin EOF. `AgentUnderTest` supplies absolute `binScript`, optional `libBinScript`, `configPath`, and `tsconfigPath` paths because the subprocess cwd is outside the repo. Startup failures preserve captured agent stderr in the rejected diagnostic.
|
||||
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; cwd-rooted separators selected as canonical `/` or host-native; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept, the same cwd-path policy), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header Agent Note](../../../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
|
||||
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario expected-output and re-persisted-log comparisons, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, the per-header-class pin (`system-prompt.expected.md` plus `tool-schemas.expected.json`) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt/schema-scrubbed, non-pinning fixtures fully header-scrubbed). Each scenario directory's `session.jsonl` plus contiguous `session.<n>.jsonl` siblings are the ordered primary/child inventory; the scenario table does not duplicate their count. Must be called at vitest collection time.
|
||||
|
||||
A consuming `*.snapshot.ts` is the scenario table plus one factory call:
|
||||
|
||||
@@ -36,11 +36,11 @@ defineAcpSnapshotSuite({
|
||||
})
|
||||
```
|
||||
|
||||
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode and filesystem scenarios are templates. Each pinning directory stores the normalized full prompt sequence in generated `system-prompt.golden.md` and the corresponding full tool-schema sequence in generated `tool-schemas.golden.json`; `session.jsonl` stores `"system":"{{system}}","tools":"{{tools}}"` while retaining config, reason, and any model-visible prefix. A pin with legitimate mid-run header changes declares `expectedHeaderChanges`, which fixes the length of both sidecar sequences.
|
||||
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode and filesystem scenarios are templates. Each pinning directory stores the normalized full prompt sequence in generated `system-prompt.expected.md` and the corresponding full tool-schema sequence in generated `tool-schemas.expected.json`; `session.jsonl` stores `"system":"{{system}}","tools":"{{tools}}"` while retaining config, reason, and any model-visible prefix. A pin with legitimate mid-run header changes declares `expectedHeaderChanges`, which fixes the length of both sidecar sequences.
|
||||
|
||||
Every scenario compares `stdout.golden.jsonl` with cwd-rooted separators canonicalized to `/`. On Windows, `pinsNativeWindowsStdout` additionally compares the complete `stdout.golden.windows.jsonl` after the shared golden and requires that sidecar exactly when enabled. A scenario whose driven behavior needs POSIX process semantics (e.g. cancelling a live bash call kills a detached process group) declares `posixOnly`, which skips its run test on Windows while the fixture guards keep covering its committed files everywhere.
|
||||
Every scenario compares `stdout.expected.jsonl` with cwd-rooted separators canonicalized to `/`. On Windows, `pinsNativeWindowsStdout` additionally compares the complete `stdout.expected.windows.jsonl` after the shared expected output and requires that sidecar exactly when enabled. A scenario whose driven behavior needs POSIX process semantics (e.g. cancelling a live bash call kills a detached process group) declares `posixOnly`, which skips its run test on Windows while the fixture guards keep covering its committed files everywhere.
|
||||
|
||||
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config RFC](../../../docs/rfc/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log goldens, and each pin's prompt and tool-schema sidecars from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
|
||||
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config Agent Note](../../../.agents/notes/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log expected outputs, and each pin's prompt and tool-schema sidecars from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md).
|
||||
|
||||
Constraints: `suite.ts` imports vitest, so the package entry is importable only inside a vitest run (the launcher, harness, and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug). Session config options are scriptable too: the `setConfigOption` step switches a knob over `session/set_config_option`, and `setConfigOptionExpectError` asserts the bridge rejects an unknown id or out-of-vocabulary value (the error frame stays in the transcript).
|
||||
|
||||
@@ -54,5 +54,5 @@ None; this package neither assembles nor sends a provider request.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **Session harvest is JSONL-only** — `runScenario` collects persisted `.jsonl` logs, so an example composed over the SQLite persistence backend has no snapshot path.
|
||||
- **Session harvest requires raw JSONL mode** — `runScenario` collects persisted `.jsonl` logs, so snapshot configs set `persistenceCompression: 'none'`; compressed JSONL and SQLite compositions have no snapshot-harvest path.
|
||||
- **Built mode requires current artifacts** — run `pnpm run build` before selecting `DSH_EXAMPLE_MODE=lib`; source mode remains the zero-build path.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-acp-snapshot",
|
||||
"description": "ACP test kit: shared subprocess launcher, snapshot scenario harness, golden normalizers, and suite factory",
|
||||
"description": "ACP test kit: shared subprocess launcher, snapshot scenario harness, expected-output normalizers, and suite factory",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
|
||||
@@ -6,12 +6,12 @@
|
||||
* It boots the REAL agent bin subprocess via the cordis Loader (so the
|
||||
* export-shape bug class stays guarded — see docs/postmortem/0001), drives it
|
||||
* over real ACP JSON-RPC stdio with a deterministic input script, tees raw
|
||||
* stdout (for the golden + a purity check) into an SDK `ClientSideConnection`,
|
||||
* stdout (for the expected-output and purity checks) into an SDK `ClientSideConnection`,
|
||||
* and — in record mode — harvests the persisted session JSONL after a graceful
|
||||
* shutdown flush. The pure normalizers in ./normalize.ts turn the captured
|
||||
* stdout frames and the session-log events into stable, snapshot-able text.
|
||||
*
|
||||
* See docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
|
||||
* See .agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-acp-snapshot/harness
|
||||
*/
|
||||
@@ -171,7 +171,7 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
|
||||
const cwd = await mkdtemp(join(tmpdir(), 'acp-snap-cwd-'))
|
||||
const sessionsRoot = await mkdtemp(join(tmpdir(), 'acp-snap-sessions-'))
|
||||
// Fixed path length: spill-policy budgets the preview against the REAL path
|
||||
// before stdout normalization, so tmpdir() length differences churn goldens.
|
||||
// before stdout normalization, so tmpdir() length differences churn expected outputs.
|
||||
const spillRoot = snapshotSpillRoot()
|
||||
// Everything past the temp-dir creation is followed by failure-safe cleanup,
|
||||
// so a failure in workspace seeding, spawn, or any step never leaks resources.
|
||||
@@ -180,7 +180,7 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
|
||||
let sessionLogs: HarvestedLog[] = []
|
||||
const outcome = await (async (): Promise<RunResult> => {
|
||||
// Seed the workspace if the scenario ships one (a file the agent reads/edits).
|
||||
// Copied into the temp cwd so the agent's bash tools see it; the goldens
|
||||
// Copied into the temp cwd so the agent's bash tools see it; the expected outputs
|
||||
// normalize the cwd, so the seeded paths stay stable across runs.
|
||||
if (opts.workspaceDir !== undefined && existsSync(opts.workspaceDir)) {
|
||||
await cp(opts.workspaceDir, cwd, { recursive: true })
|
||||
@@ -404,11 +404,11 @@ async function runStep(
|
||||
* header line, and return them ordered primary-first: the top-level session (no
|
||||
* `parentSession`) leads, then each subagent child by ascending `createdAt`.
|
||||
*
|
||||
* The JSONL backend lays sessions out as `<root>/<cwd-bucket>/<encoded-id>.jsonl`
|
||||
* (one bucket per cwd), so a parent and its same-cwd in-process child land in
|
||||
* the SAME bucket — collecting all files across all buckets catches both (a
|
||||
* first-match short-circuit would silently drop the child). Returns `[]` if no
|
||||
* log was produced (a no-session scenario).
|
||||
* Snapshot configs select the JSONL backend's raw mode, which lays sessions
|
||||
* out as `<root>/<cwd-bucket>/<encoded-id>.jsonl` (one bucket per cwd). A
|
||||
* parent and its same-cwd in-process child land in the SAME bucket, so
|
||||
* collecting all files across all buckets catches both. Returns `[]` if no log
|
||||
* was produced (a no-session scenario).
|
||||
*/
|
||||
async function harvestSessionLogs(root: string): Promise<HarvestedLog[]> {
|
||||
let cwdDirs: string[]
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
* ACP snapshot suite kit — the shared machinery behind the keyless snapshot
|
||||
* tier (`pnpm run test:snapshot`). Four layers, composable per example: the
|
||||
* shared subprocess/client launcher ({@link launchAcpTestAgent}), the scripted
|
||||
* scenario harness ({@link runScenario}), the pure golden normalizers
|
||||
* scenario harness ({@link runScenario}), the pure expected-output normalizers
|
||||
* ({@link normalizeStdout} / {@link normalizeSessionLog} /
|
||||
* {@link scrubRequestHeaders} / {@link scrubSystemPrompts}), and the suite
|
||||
* factory ({@link defineAcpSnapshotSuite}) that registers a scenario table as a
|
||||
|
||||
@@ -92,7 +92,7 @@ function scrubValue(value: unknown, ctx: NormalizeContext, cwdPathMode: CwdPathM
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize a raw stdout transcript (newline-delimited JSON-RPC frames) into a stable golden
|
||||
* Normalize a raw stdout transcript (newline-delimited JSON-RPC frames) into a stable expected output
|
||||
* in the same shape as the wire: one compact JSON frame per line (NDJSON), with the JSON-RPC
|
||||
* `id` rewritten to a per-transcript sequence (1, 2, 3, …) and all volatile strings scrubbed.
|
||||
* Invalid JSON throws, doubling as a protocol-stdout purity check.
|
||||
@@ -110,7 +110,7 @@ export function normalizeStdout(
|
||||
const cwdPathMode = options.cwdPathMode ?? 'canonical'
|
||||
const lines = rawStdout.split('\n').filter(line => line.trim().length > 0)
|
||||
// Map each distinct JSON-RPC id (request/response correlate by id) to a stable
|
||||
// sequence number, in first-seen order, so id churn doesn't perturb the golden.
|
||||
// sequence number, in first-seen order, so id churn doesn't perturb the expected output.
|
||||
const idSeq = new Map<string, number>()
|
||||
const stableId = (id: unknown): number => {
|
||||
const key = JSON.stringify(id)
|
||||
@@ -129,7 +129,7 @@ export function normalizeStdout(
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize a session JSONL log into a stable golden: the header line's
|
||||
* Normalize a session JSONL log into a stable expected output: the header line's
|
||||
* volatile fields (`createdAt`, `id`, `cwd`) and every event's `time` are
|
||||
* zeroed/scrubbed, all volatile strings scrubbed, and `seq` is LEFT INTACT
|
||||
* (deterministic by contract). Output is JSONL in the same shape as the input —
|
||||
@@ -156,7 +156,7 @@ export function normalizeSessionLog(
|
||||
// Event line: zero the epoch-ms timestamp; keep seq (deterministic).
|
||||
record.time = 0
|
||||
// A hook/result carries the hook's wall-clock runtime (`data.durationMs`),
|
||||
// which is run-to-run noise like `time` — zero it so the golden reflects
|
||||
// which is run-to-run noise like `time` — zero it so the expected output reflects
|
||||
// the hook's decision/exit, not how long the shell took.
|
||||
if (record.type === 'hook/result' && record.data !== null && typeof record.data === 'object') {
|
||||
const data = record.data as Record<string, unknown>
|
||||
|
||||
@@ -31,13 +31,13 @@ import {
|
||||
} from './normalize.ts'
|
||||
|
||||
/** The readable system-prompt snapshot beside each header-pinning fixture. */
|
||||
const SYSTEM_PROMPT_SNAPSHOT = 'system-prompt.golden.md'
|
||||
const SYSTEM_PROMPT_SNAPSHOT = 'system-prompt.expected.md'
|
||||
|
||||
/** The structured tool-schema snapshot beside each header-pinning fixture. */
|
||||
const TOOL_SCHEMAS_SNAPSHOT = 'tool-schemas.golden.json'
|
||||
const TOOL_SCHEMAS_SNAPSHOT = 'tool-schemas.expected.json'
|
||||
|
||||
/** The optional full Windows-native stdout transcript. */
|
||||
const WINDOWS_STDOUT_SNAPSHOT = 'stdout.golden.windows.jsonl'
|
||||
const WINDOWS_STDOUT_SNAPSHOT = 'stdout.expected.windows.jsonl'
|
||||
|
||||
/** Stable session-log token standing in for the sidecar's initial schemas. */
|
||||
const TOOLS_TOKEN = '{{tools}}'
|
||||
@@ -45,7 +45,7 @@ const TOOLS_TOKEN = '{{tools}}'
|
||||
/** A snapshot scenario and how its fixtures are produced. */
|
||||
export interface Scenario {
|
||||
name: string
|
||||
/** Whether the scenario drives at least one model turn (so a JSONL golden applies). */
|
||||
/** Whether the scenario drives at least one model turn (so a JSONL expected output applies). */
|
||||
hasModelTurn: boolean
|
||||
/**
|
||||
* Whether the run persists a comparable session log to diff against the
|
||||
@@ -106,7 +106,7 @@ export interface Scenario {
|
||||
configPath?: string
|
||||
/**
|
||||
* Whether Windows additionally compares stdout with native separators against
|
||||
* `stdout.golden.windows.jsonl`. The shared canonical stdout golden is still
|
||||
* `stdout.expected.windows.jsonl`. The shared canonical stdout expected output is still
|
||||
* compared on every platform, and the fixture guard requires this sidecar
|
||||
* exactly when the option is set.
|
||||
*/
|
||||
@@ -139,24 +139,24 @@ export function scenarioSkipped(
|
||||
return scenario.posixOnly === true && platform === 'win32'
|
||||
}
|
||||
|
||||
/** One stdout golden selected for a platform run. */
|
||||
interface StdoutGoldenVariant {
|
||||
/** One stdout expected output selected for a platform run. */
|
||||
interface StdoutExpectedVariant {
|
||||
file: string
|
||||
cwdPathMode: CwdPathMode
|
||||
}
|
||||
|
||||
/**
|
||||
* Select the shared stdout golden plus any platform-native assertion declared by a scenario.
|
||||
* Select the shared stdout expected output plus any platform-native assertion declared by a scenario.
|
||||
*
|
||||
* @param scenario The scenario whose stdout contract is being selected.
|
||||
* @param platform The running Node platform, injectable for unit coverage.
|
||||
* @returns The ordered golden variants: shared canonical first, then optional Windows native.
|
||||
* @returns The ordered expected-output variants: shared canonical first, then optional Windows native.
|
||||
*/
|
||||
export function stdoutGoldenVariants(
|
||||
export function stdoutExpectedVariants(
|
||||
scenario: Scenario,
|
||||
platform: NodeJS.Platform = process.platform,
|
||||
): StdoutGoldenVariant[] {
|
||||
const canonical: StdoutGoldenVariant = { file: 'stdout.golden.jsonl', cwdPathMode: 'canonical' }
|
||||
): StdoutExpectedVariant[] {
|
||||
const canonical: StdoutExpectedVariant = { file: 'stdout.expected.jsonl', cwdPathMode: 'canonical' }
|
||||
if (platform !== 'win32' || scenario.pinsNativeWindowsStdout !== true) return [canonical]
|
||||
return [canonical, { file: WINDOWS_STDOUT_SNAPSHOT, cwdPathMode: 'native' }]
|
||||
}
|
||||
@@ -171,8 +171,8 @@ export interface SnapshotSuiteOptions {
|
||||
scenarios: Scenario[]
|
||||
/**
|
||||
* `replay` (keyless, the default tier), `record` (live API; re-records the
|
||||
* `recorded` scenarios' fixtures and refreshes the Vitest goldens under
|
||||
* `--update`), or `refresh` (keyless replay that rewrites stdout goldens and
|
||||
* `recorded` scenarios' fixtures and refreshes the Vitest expected outputs under
|
||||
* `--update`), or `refresh` (keyless replay that rewrites stdout expected outputs and
|
||||
* comparable session fixtures from the replay run). The caller derives this
|
||||
* from `$DSH_SNAPSHOT` — env reading stays outside this library.
|
||||
*/
|
||||
@@ -486,7 +486,7 @@ export function stabilizeRefreshLog(fresh: string, existing: string, replacement
|
||||
}
|
||||
|
||||
/**
|
||||
* Register the suite: one test per scenario (the golden/log compares and
|
||||
* Register the suite: one test per scenario (the expected-output and log comparisons and
|
||||
* the header-uniformity guard) plus the fixture guard block (no orphan
|
||||
* scenario dirs, required files present, exactly one pin per header class,
|
||||
* pinning fixtures well-formed, every JSONL prompt-scrubbed, non-pinning
|
||||
@@ -527,7 +527,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
// In RECORD mode, only re-run the `recorded` (live-API) scenarios; the `authored` ones
|
||||
// (sidecar-driven errors/cancel) are never re-recorded. `posixOnly` scenarios skip on
|
||||
// Windows, where their process semantics cannot be driven.
|
||||
it.skipIf(scenarioSkipped(scenario, RECORDING))(`snapshot: ${scenario.name} matches the goldens`, async ({ expect }) => {
|
||||
it.skipIf(scenarioSkipped(scenario, RECORDING))(`snapshot: ${scenario.name} matches the expected outputs`, async ({ expect }) => {
|
||||
const dir = join(snapshotsDir, scenario.name)
|
||||
const input = JSON.parse(await readFile(join(dir, 'input.json'), 'utf8')) as InputScript
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
@@ -631,12 +631,12 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
}
|
||||
}
|
||||
|
||||
for (const golden of stdoutGoldenVariants(scenario)) {
|
||||
const stdout = normalizeStdout(result.rawStdout, ctx, { cwdPathMode: golden.cwdPathMode })
|
||||
for (const expected of stdoutExpectedVariants(scenario)) {
|
||||
const stdout = normalizeStdout(result.rawStdout, ctx, { cwdPathMode: expected.cwdPathMode })
|
||||
if (REFRESHING) {
|
||||
await writeFile(join(dir, golden.file), stdout)
|
||||
await writeFile(join(dir, expected.file), stdout)
|
||||
}
|
||||
await expect(stdout, `${golden.file} mismatch`).toMatchFileSnapshot(join(dir, golden.file))
|
||||
await expect(stdout, `${expected.file} mismatch`).toMatchFileSnapshot(join(dir, expected.file))
|
||||
}
|
||||
|
||||
// A model turn always produces a log worth comparing; a hook scenario can
|
||||
@@ -713,7 +713,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
|
||||
describe('snapshot fixtures', () => {
|
||||
it('every scenario directory is registered (no orphans)', async () => {
|
||||
// toMatchFileSnapshot does not prune orphaned golden/fixture files, so a
|
||||
// toMatchFileSnapshot does not prune orphaned expected-output or fixture files, so a
|
||||
// renamed/removed scenario could leave a stale dir that nothing exercises.
|
||||
// Fail loud on any snapshots/<dir> not present in the scenario table.
|
||||
const entries = await readdir(snapshotsDir, { withFileTypes: true })
|
||||
@@ -727,7 +727,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
for (const { name, overridden, pinsHeader, pinsNativeWindowsStdout } of scenarios) {
|
||||
const dir = join(snapshotsDir, name)
|
||||
expect(existsSync(join(dir, 'input.json')), `${name}/input.json`).toBe(true)
|
||||
expect(existsSync(join(dir, 'stdout.golden.jsonl')), `${name}/stdout.golden.jsonl`).toBe(true)
|
||||
expect(existsSync(join(dir, 'stdout.expected.jsonl')), `${name}/stdout.expected.jsonl`).toBe(true)
|
||||
expect(
|
||||
existsSync(join(dir, WINDOWS_STDOUT_SNAPSHOT)),
|
||||
`${name}/${WINDOWS_STDOUT_SNAPSHOT} presence must match \`pinsNativeWindowsStdout\``,
|
||||
|
||||
@@ -2,11 +2,11 @@
|
||||
"prompt": "respond",
|
||||
"logs": [
|
||||
{ "file": "b/parent.jsonl", "lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 700, "cwd": "{{CWD}}" },
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 700, "cwd": "{{CWD}}", "delegationDepth": 0 },
|
||||
{ "type": "request/header", "seq": 0, "time": 3, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
|
||||
]},
|
||||
{ "file": "b/child.jsonl", "lines": [
|
||||
{ "type": "session", "id": "abababab-cdcd-4efe-8ada-badabadabada", "createdAt": 800, "cwd": "{{CWD}}", "parentSession": "{{SID}}" },
|
||||
{ "type": "session", "id": "abababab-cdcd-4efe-8ada-badabadabada", "createdAt": 800, "cwd": "{{CWD}}", "parentSession": "{{SID}}", "delegationDepth": 1 },
|
||||
{ "type": "request/header", "seq": 0, "time": 2, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
|
||||
]}
|
||||
]
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
{"type":"session","id":"abababab-cdcd-4efe-8ada-badabadabada","createdAt":800,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-KBQJbW","parentSession":"f6fa7fcf-dd9c-4b39-8815-b25ddcebfd88"}
|
||||
{"type":"session","id":"abababab-cdcd-4efe-8ada-badabadabada","createdAt":800,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-KBQJbW","parentSession":"f6fa7fcf-dd9c-4b39-8815-b25ddcebfd88","delegationDepth":1}
|
||||
{"type":"request/header","seq":0,"time":2,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
{"type":"session","id":"f6fa7fcf-dd9c-4b39-8815-b25ddcebfd88","createdAt":700,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-KBQJbW"}
|
||||
{"type":"session","id":"f6fa7fcf-dd9c-4b39-8815-b25ddcebfd88","createdAt":700,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-KBQJbW","delegationDepth":0}
|
||||
{"type":"request/header","seq":0,"time":3,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
"logs": [{
|
||||
"file": "b/main.jsonl",
|
||||
"lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 600, "cwd": "{{CWD}}" },
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 600, "cwd": "{{CWD}}", "delegationDepth": 0 },
|
||||
{ "type": "request/header", "seq": 0, "time": 4, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
|
||||
]
|
||||
}]
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
{"type":"session","id":"ccdc749f-56f3-4267-9750-598b5c60b7b2","createdAt":600,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-nOQ4Gy"}
|
||||
{"type":"session","id":"ccdc749f-56f3-4267-9750-598b5c60b7b2","createdAt":600,"cwd":"/var/folders/2g/b32ct0qn1d728l_v6tdkjytr0000gn/T/acp-snap-cwd-nOQ4Gy","delegationDepth":0}
|
||||
{"type":"request/header","seq":0,"time":4,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
|
||||
@@ -1 +1 @@
|
||||
{"type":"session","id":"{{sessionId}}","createdAt":0,"cwd":"{{cwd}}"}
|
||||
{"type":"session","id":"{{sessionId}}","createdAt":0,"cwd":"{{cwd}}","delegationDepth":0}
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
"logs": [{
|
||||
"file": "b/main.jsonl",
|
||||
"lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 500, "cwd": "{{CWD}}" },
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 500, "cwd": "{{CWD}}", "delegationDepth": 0 },
|
||||
{ "type": "turn/end", "seq": 1, "time": 9, "data": { "error": "model exploded" } }
|
||||
]
|
||||
}]
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
{"type":"session","id":"44444444-3333-4222-8111-000000000000","createdAt":17,"cwd":"/rec/authored-cwd"}
|
||||
{"type":"session","id":"44444444-3333-4222-8111-000000000000","createdAt":17,"cwd":"/rec/authored-cwd","delegationDepth":0}
|
||||
{"type":"turn/end","seq":1,"time":17,"data":{"error":"model exploded"}}
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
"logs": [{
|
||||
"file": "b/main.jsonl",
|
||||
"lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 400, "cwd": "{{CWD}}" },
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 400, "cwd": "{{CWD}}", "delegationDepth": 0 },
|
||||
{ "type": "hook/result", "seq": 1, "time": 8, "data": { "decision": "block", "durationMs": 37 } }
|
||||
]
|
||||
}]
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
{"type":"session","id":"99999999-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/blocked-cwd"}
|
||||
{"type":"session","id":"99999999-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/blocked-cwd","delegationDepth":0}
|
||||
{"type":"hook/result","seq":1,"time":13,"data":{"decision":"block","durationMs":99}}
|
||||
|
||||
@@ -1 +1 @@
|
||||
{"type":"session","id":"{{sessionId}}","createdAt":0,"cwd":"{{cwd}}"}
|
||||
{"type":"session","id":"{{sessionId}}","createdAt":0,"cwd":"{{cwd}}","delegationDepth":0}
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
"logs": [{
|
||||
"file": "b/main.jsonl",
|
||||
"lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 100, "cwd": "{{CWD}}" },
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 100, "cwd": "{{CWD}}", "delegationDepth": 0 },
|
||||
{ "type": "request/header", "seq": 0, "time": 100, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } },
|
||||
{ "type": "request/header", "seq": 1, "time": 100, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT\n\nNEW PROMPT LINE", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "change" } },
|
||||
{ "type": "turn/start", "seq": 2, "time": 100, "data": { "turn": 1 } }
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
{"type":"session","id":"12121212-3434-4545-8686-787878787878","createdAt":7,"cwd":"/rec/pin-cwd"}
|
||||
{"type":"session","id":"12121212-3434-4545-8686-787878787878","createdAt":7,"cwd":"/rec/pin-cwd","delegationDepth":0}
|
||||
{"type":"request/header","seq":0,"time":7,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
{"type":"request/header","seq":1,"time":7,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"change"}}
|
||||
{"type":"turn/start","seq":2,"time":7,"data":{"turn":1}}
|
||||
|
||||
@@ -3,12 +3,12 @@
|
||||
"echoWorkspace": true,
|
||||
"logs": [
|
||||
{ "file": "b/parent.jsonl", "lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 200, "cwd": "{{CWD}}" },
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 200, "cwd": "{{CWD}}", "delegationDepth": 0 },
|
||||
{ "type": "request/header", "seq": 0, "time": 5, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } },
|
||||
{ "type": "assistant/chunk", "seq": 1, "time": 5, "data": { "turn": 1, "step": 1, "chunk": { "type": "text-delta", "index": 0, "text": "hi" } } }
|
||||
]},
|
||||
{ "file": "b/child.jsonl", "lines": [
|
||||
{ "type": "session", "id": "eeeeeeee-1111-4222-8333-444444444444", "createdAt": 300, "cwd": "{{CWD}}", "parentSession": "{{SID}}" },
|
||||
{ "type": "session", "id": "eeeeeeee-1111-4222-8333-444444444444", "createdAt": 300, "cwd": "{{CWD}}", "parentSession": "{{SID}}", "delegationDepth": 1 },
|
||||
{ "type": "request/header", "seq": 0, "time": 6, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } }
|
||||
]}
|
||||
]
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
{"type":"session","id":"eeeeeeee-1111-4222-8333-444444444444","createdAt":12,"cwd":"/rec/plain-cwd","parentSession":"56565656-7878-4989-8a9a-9b9b9b9b9b9b"}
|
||||
{"type":"session","id":"eeeeeeee-1111-4222-8333-444444444444","createdAt":12,"cwd":"/rec/plain-cwd","parentSession":"56565656-7878-4989-8a9a-9b9b9b9b9b9b","delegationDepth":1}
|
||||
{"type":"request/header","seq":0,"time":12,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
{"type":"session","id":"56565656-7878-4989-8a9a-9b9b9b9b9b9b","createdAt":11,"cwd":"/rec/plain-cwd"}
|
||||
{"type":"session","id":"56565656-7878-4989-8a9a-9b9b9b9b9b9b","createdAt":11,"cwd":"/rec/plain-cwd","delegationDepth":0}
|
||||
{"type":"request/header","seq":0,"time":11,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
{"type":"assistant/chunk","seq":1,"time":11,"data":{"turn":1,"step":1,"chunk":{"type":"text-delta","index":0,"text":"hi"}}}
|
||||
|
||||
@@ -19,14 +19,14 @@ import {
|
||||
sessionFixtureNames,
|
||||
restorePinnedToolSchemas,
|
||||
stabilizeRefreshLog,
|
||||
stdoutGoldenVariants,
|
||||
stdoutExpectedVariants,
|
||||
unknownToolCallIds,
|
||||
} from '../src/suite.ts'
|
||||
|
||||
/**
|
||||
* Unit tests for the suite factory, by running it: two synthetic suites over the scripted fake
|
||||
* ACP bin (./fixtures/fake-acp-agent.ts) register real describe/it trees at collection time,
|
||||
* so every factory path — golden and log compares, the per-suite header pin and its uniformity
|
||||
* so every factory path — expected-output and log comparisons, the per-suite header pin and its uniformity
|
||||
* guard, record-mode fixture write-back, skip semantics, and the fixture guard block —
|
||||
* executes as an ordinary green test.
|
||||
*
|
||||
@@ -63,7 +63,7 @@ const RECORD_SCENARIOS: Scenario[] = [
|
||||
|
||||
// Record/refresh modes mutate their snapshots dir, so run them on throwaway
|
||||
// copies — except record's documented bootstrap knob, which regenerates the
|
||||
// committed record fixtures/goldens in place.
|
||||
// committed record fixtures and expected outputs in place.
|
||||
const BOOTSTRAP = process.env.ACP_SNAPSHOT_SPEC_BOOTSTRAP === '1'
|
||||
const recordDir = BOOTSTRAP ? RECORD_SRC : mkdtempSync(join(tmpdir(), 'acp-snap-record-suite-'))
|
||||
if (!BOOTSTRAP) {
|
||||
@@ -82,9 +82,9 @@ afterAll(async () => {
|
||||
})
|
||||
|
||||
function staleRefreshFixtures(dir: string): void {
|
||||
writeFileSync(join(dir, 'plain-turn', 'stdout.golden.jsonl'), 'stale stdout\n')
|
||||
writeFileSync(join(dir, 'pin-turn', 'system-prompt.golden.md'), 'STALE PROMPT\n')
|
||||
writeFileSync(join(dir, 'pin-turn', 'tool-schemas.golden.json'), '{"initial":[{"name":"stale"}],"changes":[]}\n')
|
||||
writeFileSync(join(dir, 'plain-turn', 'stdout.expected.jsonl'), 'stale stdout\n')
|
||||
writeFileSync(join(dir, 'pin-turn', 'system-prompt.expected.md'), 'STALE PROMPT\n')
|
||||
writeFileSync(join(dir, 'pin-turn', 'tool-schemas.expected.json'), '{"initial":[{"name":"stale"}],"changes":[]}\n')
|
||||
|
||||
const plainBehaviorFile = join(dir, 'plain-turn', 'behavior.json')
|
||||
const plainBehavior = JSON.parse(readFileSync(plainBehaviorFile, 'utf8')) as Record<string, unknown>
|
||||
@@ -92,12 +92,12 @@ function staleRefreshFixtures(dir: string): void {
|
||||
writeFileSync(plainBehaviorFile, `${JSON.stringify(plainBehavior, null, 2)}\n`)
|
||||
|
||||
writeFileSync(join(dir, 'blocked-log', 'session.jsonl'), [
|
||||
'{"type":"session","id":"99999999-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/blocked-cwd"}',
|
||||
'{"type":"session","id":"99999999-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/blocked-cwd","delegationDepth":0}',
|
||||
'{"type":"hook/result","seq":1,"time":13,"data":{"decision":"stale","durationMs":99}}',
|
||||
'',
|
||||
].join('\n'))
|
||||
writeFileSync(join(dir, 'authored-error', 'session.jsonl'), [
|
||||
'{"type":"session","id":"77777777-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/error-cwd"}',
|
||||
'{"type":"session","id":"77777777-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/error-cwd","delegationDepth":0}',
|
||||
'{"type":"turn/end","seq":1,"time":9,"data":{"error":"stale"}}',
|
||||
'',
|
||||
].join('\n'))
|
||||
@@ -119,7 +119,7 @@ describe('defineAcpSnapshotSuite: refresh mode', () => {
|
||||
|
||||
describe('defineAcpSnapshotSuite: refresh write-back', () => {
|
||||
it('rewrites stdout and comparable logs from a replay-mode child run', () => {
|
||||
const stdout = readFileSync(join(refreshDir, 'plain-turn', 'stdout.golden.jsonl'), 'utf8')
|
||||
const stdout = readFileSync(join(refreshDir, 'plain-turn', 'stdout.expected.jsonl'), 'utf8')
|
||||
expect(stdout).not.toContain('stale stdout')
|
||||
expect(stdout).toContain('env:{\\"mode\\":\\"replay\\"')
|
||||
expect(stdout).not.toContain('\\"mode\\":\\"refresh\\"')
|
||||
@@ -132,7 +132,7 @@ describe('defineAcpSnapshotSuite: refresh write-back', () => {
|
||||
expect(authored).toContain('"error":"model exploded"')
|
||||
expect(authored).not.toContain('"error":"stale"')
|
||||
|
||||
expect(readFileSync(join(refreshDir, 'pin-turn', 'system-prompt.golden.md'), 'utf8')).toBe([
|
||||
expect(readFileSync(join(refreshDir, 'pin-turn', 'system-prompt.expected.md'), 'utf8')).toBe([
|
||||
'SYS PROMPT',
|
||||
'',
|
||||
'<!-- request/header change 1 -->',
|
||||
@@ -142,7 +142,7 @@ describe('defineAcpSnapshotSuite: refresh write-back', () => {
|
||||
'NEW PROMPT LINE',
|
||||
'',
|
||||
].join('\n'))
|
||||
const schemas = readFileSync(join(refreshDir, 'pin-turn', 'tool-schemas.golden.json'), 'utf8')
|
||||
const schemas = readFileSync(join(refreshDir, 'pin-turn', 'tool-schemas.expected.json'), 'utf8')
|
||||
expect(schemas).toContain('"description": "D1"')
|
||||
expect(schemas).not.toContain('"name":"stale"')
|
||||
})
|
||||
@@ -197,7 +197,7 @@ describe('defineAcpSnapshotSuite: registration contract', () => {
|
||||
describe('sessionFixtureNames', () => {
|
||||
it('orders the primary and contiguous child fixtures while ignoring other files', () => {
|
||||
expect(sessionFixtureNames([
|
||||
'stdout.golden.jsonl',
|
||||
'stdout.expected.jsonl',
|
||||
'session.2.jsonl',
|
||||
'session.jsonl',
|
||||
'session.1.jsonl',
|
||||
@@ -232,7 +232,7 @@ describe('sessionFixtureNames', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('stdoutGoldenVariants', () => {
|
||||
describe('stdoutExpectedVariants', () => {
|
||||
const scenario: Scenario = {
|
||||
name: 'windows-native',
|
||||
hasModelTurn: true,
|
||||
@@ -241,18 +241,18 @@ describe('stdoutGoldenVariants', () => {
|
||||
}
|
||||
|
||||
it('adds the native sidecar after the shared golden on Windows', () => {
|
||||
expect(stdoutGoldenVariants(scenario, 'win32')).toEqual([
|
||||
{ file: 'stdout.golden.jsonl', cwdPathMode: 'canonical' },
|
||||
{ file: 'stdout.golden.windows.jsonl', cwdPathMode: 'native' },
|
||||
expect(stdoutExpectedVariants(scenario, 'win32')).toEqual([
|
||||
{ file: 'stdout.expected.jsonl', cwdPathMode: 'canonical' },
|
||||
{ file: 'stdout.expected.windows.jsonl', cwdPathMode: 'native' },
|
||||
])
|
||||
})
|
||||
|
||||
it('keeps only the shared golden on other platforms or without the declaration', () => {
|
||||
expect(stdoutGoldenVariants(scenario, 'linux')).toEqual([
|
||||
{ file: 'stdout.golden.jsonl', cwdPathMode: 'canonical' },
|
||||
expect(stdoutExpectedVariants(scenario, 'linux')).toEqual([
|
||||
{ file: 'stdout.expected.jsonl', cwdPathMode: 'canonical' },
|
||||
])
|
||||
expect(stdoutGoldenVariants({ ...scenario, pinsNativeWindowsStdout: false }, 'win32')).toEqual([
|
||||
{ file: 'stdout.golden.jsonl', cwdPathMode: 'canonical' },
|
||||
expect(stdoutExpectedVariants({ ...scenario, pinsNativeWindowsStdout: false }, 'win32')).toEqual([
|
||||
{ file: 'stdout.expected.jsonl', cwdPathMode: 'canonical' },
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
@@ -4,7 +4,7 @@ Runtime event-contract assertions intended for development diagnostics. This pur
|
||||
|
||||
The plugin has no environment guard: it is active wherever it is registered. The default [`dsh-agent-spine-demo`](../../examples/agent-spine-demo/README.md) bundle mounts it unconditionally; a custom composition can omit it when the runtime cost is undesirable. It doubles as executable documentation of the event taxonomy — the assertions *are* the contract.
|
||||
|
||||
Session itself owns immutable, surface-valid log storage in every composition: it takes one lossless JSON snapshot of each candidate, validates the complete surface transition, deep-freezes the accepted record, and exposes the log through immutable array snapshots. The invariants plugin checks the remaining cross-record and cross-seam rules that Session does not own.
|
||||
Session itself owns immutable, surface-valid log storage in every composition: it takes one lossless JSON snapshot of each candidate, validates complete provenance and positional replacement, restricts `tool/result` replacement to one current result's `content`, deep-freezes the accepted record, and exposes the log through immutable array snapshots. The invariants plugin checks the remaining cross-record and cross-seam rules that Session does not own.
|
||||
|
||||
Session-log assertions run during Cordis `internal/dispatch`, while `Session.append()` is resolving the `session/event` callback snapshot but before it pushes the candidate into the log. A valid transition is staged by exact event identity and applied to the live trace only when that same committed event reaches the plugin's contained post-commit listener. A later internal dispatch check can therefore veto without advancing either the log or the invariant trace, while ordinary `session/event` observer failures remain observe-only.
|
||||
|
||||
@@ -31,8 +31,7 @@ Session log (per session):
|
||||
- **turns pair and nest** — `turn/start` opens a turn, `turn/end` closes the matching one; no overlapping turns.
|
||||
- **steps nest in turns** — `step/start` opens a step in the open turn; `step/end` closes the matching step.
|
||||
- **chunks belong to an open step** — `step/start` precedes its `assistant/chunk`s.
|
||||
- **a `tool/result` needs a prior `tool/call`** — but NOT the converse: a `tool/call` may have no result (a thrown tool-execution pipeline step ends the turn with no `tool/result`, which is legal).
|
||||
- **provenance sources are valid and unambiguous** — `sourceEventSeqs` contains unique earlier known seqs; only `assistant/message` may carry an explicit empty list, which denotes a known empty provider stream rather than absent legacy provenance.
|
||||
- **an appended `tool/result` needs a prior `tool/call`** — fresh `surfaceOp: 'append'` results name the open step and consume its pending call. A Session-validated replacement is a turn-enclosed rewrite, not another execution. A `tool/call` may still have no result when the execution pipeline throws.
|
||||
|
||||
Agent status (per agent):
|
||||
|
||||
@@ -40,13 +39,13 @@ Agent status (per agent):
|
||||
|
||||
Model requests (on `llm/stream`):
|
||||
|
||||
- **a loop-built request is exactly what the log reconstructs** — a frozen request with a live `sessionId` (the loop-built marker; hand-built one-shots like compaction's summarize are unfrozen and skipped) must carry frozen `messages` deep-equal to the derivation over the log prefix strictly before the in-flight step's `step/start` (rebuilt through a FRESH `Session`, so the live cache cannot vouch for itself — and boundary-correct: content logged after `step/start` legitimately belongs to the next request), and every non-content field must equal the latest logged `request/header` (see [the reconstructability RFC](../../../docs/rfc/implemented/architecture/2026-07-05-reconstructable-requests.md)). Registered with `prepend: true` so a short-circuiting `llm/stream` listener (the replay adapter) cannot silence it; prepend orders it against append-registered listeners only — correctness rests on the seq-bounded rebuild, never listener timing.
|
||||
- **a loop-built request is exactly what the log reconstructs** — a frozen request with a live `sessionId` (the loop-built marker; hand-built one-shots like compaction's summarize are unfrozen and skipped) must carry frozen `messages` deep-equal to the derivation over the log prefix strictly before the in-flight step's `step/start` (rebuilt through a FRESH `Session`, so the live cache cannot vouch for itself — and boundary-correct: content logged after `step/start` legitimately belongs to the next request), and every non-content field must equal the latest logged `request/header` (see [the reconstructability Agent Note](../../../.agents/notes/implemented/architecture/2026-07-05-reconstructable-requests.md)). Registered with `prepend: true` so a short-circuiting `llm/stream` listener (the replay adapter) cannot silence it; prepend orders it against append-registered listeners only — correctness rests on the seq-bounded rebuild, never listener timing.
|
||||
|
||||
On any violation it throws `InvariantError` (`code: 'INVARIANT'`).
|
||||
|
||||
## Why runtime assertions remain useful
|
||||
|
||||
Session enforces the per-record storage boundary at runtime, where a cast cannot bypass it. Pervasive `DeepReadonly<SessionEvent>` types would add noise across consumers without expressing relationships such as turn/step nesting, subject-correct scoped dispatch, or equality between a request and its log reconstruction. This plugin checks those relationships wherever it is mounted while `dsh-session` keeps history immutable in every composition. See [source-owned session immutability and dev-mode invariants](../../../docs/rfc/implemented/architecture/2026-06-11-dev-invariants-over-deep-readonly.md).
|
||||
Session enforces the per-record storage boundary at runtime, where a cast cannot bypass it. Pervasive `DeepReadonly<SessionEvent>` types would add noise across consumers without expressing relationships such as turn/step nesting, subject-correct scoped dispatch, or equality between a request and its log reconstruction. This plugin checks those relationships wherever it is mounted while `dsh-session` keeps history immutable in every composition. See [source-owned session immutability and dev-mode invariants](../../../.agents/notes/implemented/architecture/2026-06-11-dev-invariants-over-deep-readonly.md).
|
||||
|
||||
## Seeded sessions
|
||||
|
||||
|
||||
@@ -86,7 +86,7 @@ function validateEvent(trace: SessionTrace, event: SessionEvent): SessionTraceTr
|
||||
|
||||
// Boundary/step-scoped events have explicit cases; every OTHER event type —
|
||||
// including plugin-added (merge-extensible) SessionEventMap keys — is caught
|
||||
// by the `default` and must be turn-enclosed (the turn-enclosure RFC). No assertNever: an
|
||||
// by the `default` and must be turn-enclosed (the turn-enclosure Agent Note). No assertNever: an
|
||||
// unknown variant is valid, not a compile error.
|
||||
switch (event.type) {
|
||||
case 'turn/start': {
|
||||
@@ -151,6 +151,16 @@ function validateEvent(trace: SessionTrace, event: SessionEvent): SessionTraceTr
|
||||
break
|
||||
}
|
||||
case 'tool/result': {
|
||||
// Session has already validated a provenance-backed content rewrite.
|
||||
// It is durable turn work, not a second execution of the original call.
|
||||
if (event.surfaceOp !== 'append') {
|
||||
if (trace.openTurn === null) {
|
||||
throw new InvariantError(
|
||||
'tool/result surface replacement appended outside any open turn',
|
||||
)
|
||||
}
|
||||
break
|
||||
}
|
||||
requireOpenStep(trace, 'tool/result', event.data.turn, event.data.step)
|
||||
// A result needs a prior matching call in the same step. (The converse
|
||||
// does NOT hold: a call may have no result — a throwing tool-execution
|
||||
@@ -162,7 +172,7 @@ function validateEvent(trace: SessionTrace, event: SessionEvent): SessionTraceTr
|
||||
pendingCalls = { kind: 'delete', callId: event.data.callId }
|
||||
break
|
||||
}
|
||||
// Turn-enclosure (the turn-enclosure RFC): EVERY session event not handled by a boundary
|
||||
// Turn-enclosure (the turn-enclosure Agent Note): EVERY session event not handled by a boundary
|
||||
// case above must sit inside an open turn. The durable session log uses the
|
||||
// turn as its commit/replay boundary (the JSONL backend treats anything
|
||||
// after the last turn/end as a crash tail), so a bare event between turns is
|
||||
@@ -330,7 +340,7 @@ export function apply(ctx: Context): void {
|
||||
}
|
||||
}, { global: true })
|
||||
|
||||
// Request-reconstruction cross-check (the reconstructability RFC): a
|
||||
// Request-reconstruction cross-check (the reconstructability Agent Note): a
|
||||
// loop-built request — frozen envelope + live sessionId is the marker; a
|
||||
// hand-built one-shot (compaction summarize) is unfrozen and skipped — must
|
||||
// be EXACTLY what the session log reconstructs:
|
||||
|
||||
@@ -149,7 +149,7 @@ describe('session-log invariants', () => {
|
||||
it('rejects a message event appended outside any open turn (turn-enclosure)', async () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
// No turn open: every message-bearing event must be turn-enclosed (the turn-enclosure RFC).
|
||||
// No turn open: every message-bearing event must be turn-enclosed (the turn-enclosure Agent Note).
|
||||
expect(() => session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }, { surfaceOp: 'append' }))
|
||||
.toThrow(/outside any open turn/)
|
||||
expect(() => session.append('context/message', { content: [{ type: 'text', text: 'ctx' }], source: { kind: 'user' } }, { surfaceOp: 'append' }))
|
||||
@@ -160,7 +160,7 @@ describe('session-log invariants', () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
// steering/message is turn-scoped: outside a turn it would land past the
|
||||
// commit boundary and be dropped on resume (the turn-enclosure RFC).
|
||||
// commit boundary and be dropped on resume (the turn-enclosure Agent Note).
|
||||
expect(() => session.append('steering/message', { turn: 1, content: [{ type: 'text', text: 'go' }], source: { kind: 'user' } }, { surfaceOp: 'append' }))
|
||||
.toThrow(/outside any open turn/)
|
||||
// A PLUGIN-added (merge-extensible) event type is caught by the default too.
|
||||
@@ -189,6 +189,19 @@ describe('session-log invariants', () => {
|
||||
.toThrow(/no prior tool\/call/)
|
||||
})
|
||||
|
||||
it('keeps fresh tool-result appends open-step and pending-call checked', async () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
expect(() => session.append('tool/result', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
callId: CallId('closed'),
|
||||
content: [],
|
||||
isError: false,
|
||||
}, { surfaceOp: 'append' })).toThrow(/open is turn 1\/step null/)
|
||||
})
|
||||
|
||||
it('allows a synthetic interrupted tool/result from crash repair without a prior tool/call event', async () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
@@ -465,6 +478,41 @@ describe('HMR safety', () => {
|
||||
})
|
||||
|
||||
describe('surface contract under the invariants composition', () => {
|
||||
async function toolResultRewriteFixture(openRewriteTurn = true) {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
const unrelated = session.append('user/message', {
|
||||
content: [{ type: 'text', text: 'request' }],
|
||||
source: { kind: 'user' },
|
||||
}, { surfaceOp: 'append' })
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
session.append('tool/call', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
callId: CallId('rewrite'),
|
||||
name: 'echo',
|
||||
arguments: '{}',
|
||||
})
|
||||
const originalData = {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
callId: CallId('rewrite'),
|
||||
content: [{ type: 'text' as const, text: 'original' }],
|
||||
isError: true,
|
||||
error: { name: 'ExitError', code: 'EXIT_1' },
|
||||
meta: { presentation: { kind: 'terminal', output: 'full output' } },
|
||||
futureField: { nested: ['preserve', 1] },
|
||||
}
|
||||
const original = session.append('tool/result', originalData, { surfaceOp: 'append' })
|
||||
session.append('step/end', { turn: 1, step: 1 })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
if (openRewriteTurn) {
|
||||
session.append('turn/start', { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
}
|
||||
return { session, unrelated, original }
|
||||
}
|
||||
|
||||
it('accepts well-formed surface metadata', async () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
@@ -487,6 +535,71 @@ describe('surface contract under the invariants composition', () => {
|
||||
// no throw — well-formed replace op
|
||||
})
|
||||
|
||||
it('treats a provenance-backed tool-result replacement as a turn-enclosed rewrite', async () => {
|
||||
const { session, original } = await toolResultRewriteFixture()
|
||||
|
||||
expect(() => session.append('tool/result', {
|
||||
...original.data,
|
||||
content: [{ type: 'text', text: 'pruned' }],
|
||||
}, {
|
||||
surfaceOp: { op: 'replace', start: original.seq, end: original.seq },
|
||||
sourceEventSeqs: [original.seq],
|
||||
})).not.toThrow()
|
||||
})
|
||||
|
||||
it('rejects a tool-result replacement outside a turn', async () => {
|
||||
const { session, original } = await toolResultRewriteFixture(false)
|
||||
|
||||
expect(() => session.append('tool/result', {
|
||||
...original.data,
|
||||
content: [{ type: 'text', text: 'pruned' }],
|
||||
}, {
|
||||
surfaceOp: { op: 'replace', start: original.seq, end: original.seq },
|
||||
sourceEventSeqs: [original.seq],
|
||||
})).toThrow(/outside any open turn/)
|
||||
})
|
||||
|
||||
it('rejects a tool-result replacement targeting an unrelated current node', async () => {
|
||||
const { session, unrelated, original } = await toolResultRewriteFixture()
|
||||
expect(() => session.append('tool/result', {
|
||||
...original.data,
|
||||
content: [{ type: 'text', text: 'forged' }],
|
||||
}, {
|
||||
surfaceOp: { op: 'replace', start: unrelated.seq, end: unrelated.seq },
|
||||
sourceEventSeqs: [unrelated.seq],
|
||||
})).toThrow(/must target a current tool\/result/)
|
||||
})
|
||||
|
||||
it('rejects a multi-node tool-result replacement even with complete provenance', async () => {
|
||||
const { session, unrelated, original } = await toolResultRewriteFixture()
|
||||
expect(() => session.append('tool/result', {
|
||||
...original.data,
|
||||
content: [{ type: 'text', text: 'forged' }],
|
||||
}, {
|
||||
surfaceOp: { op: 'replace', start: unrelated.seq, end: original.seq },
|
||||
sourceEventSeqs: [unrelated.seq, original.seq],
|
||||
})).toThrow(/must rewrite exactly one current node/)
|
||||
})
|
||||
|
||||
it.each([
|
||||
['callId', { callId: CallId('forged') }],
|
||||
['turn', { turn: 2 }],
|
||||
['step', { step: 2 }],
|
||||
['error', { error: { name: 'ExitError', code: 'DIFFERENT' } }],
|
||||
['meta', { meta: { presentation: { kind: 'generic' } } }],
|
||||
['future data', { futureField: { nested: ['changed'] } }],
|
||||
])('rejects a content rewrite with altered %s', async (_label, altered) => {
|
||||
const { session, original } = await toolResultRewriteFixture()
|
||||
expect(() => session.append('tool/result', {
|
||||
...original.data,
|
||||
...altered,
|
||||
content: [{ type: 'text', text: 'pruned' }],
|
||||
}, {
|
||||
surfaceOp: { op: 'replace', start: original.seq, end: original.seq },
|
||||
sourceEventSeqs: [original.seq],
|
||||
})).toThrow(/may change only content/)
|
||||
})
|
||||
|
||||
it('accepts known-empty assistant provenance and rejects empty provenance elsewhere', async () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
|
||||
Reference in New Issue
Block a user