Merge branch 'codex/simp-snapshot-fixture-inventory' into codex/simp-shared-acp-test-launcher

# Conflicts:
#	examples/acp-agent/tests/escalation.e2e.ts
#	packages/support/README.md
#	packages/support/acp-snapshot/README.md
#	packages/support/acp-snapshot/src/harness.ts
#	packages/support/acp-snapshot/tests/harness.spec.ts
This commit is contained in:
Tianyi Cui
2026-07-18 11:58:15 +08:00
657 changed files with 36667 additions and 7474 deletions

View File

@@ -5,9 +5,10 @@ Packages that exist to serve development, testing, and the examples rather than
| Package | Role | ctx key |
|---|---|---|
| `acp-snapshot/` | ACP test kit: shared subprocess/client launcher + snapshot harness, normalizers, and suite factory | (library — imported by ACP e2e and `*.snapshot.ts` suites) |
| `agent-loop-testkit/` | Shared prerequisite mounting for tests that exercise the concrete agent loop | (library — imported by AgentLoop integration tests) |
| `invariants/` | Runtime event-contract assertions for development diagnostics | (listens on `session/*`, `agent/*`) |
| `loader-smoke/` | Shared real-Loader subprocess harness for keyless example smokes | (library — imported by example e2e suites) |
| `llm-replay/` | Record/replay adapter: short-circuits `llm/stream` from a recorded session JSONL (keyless snapshot tests) | (listens on `llm/stream`) |
| `subagent-mock/` | Scripted `SubagentProvider` for deterministic seam/tool tests | (registers on `ctx.subagents`) |
`invariants` is development support but has no environment guard: it runs wherever registered, and the default `dsh-agent-spine-demo` bundle mounts it unconditionally. `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate. `acp-snapshot` carries the ACP subprocess/client boundary plus the snapshot harness, normalizers, and suite machinery, while `loader-smoke` owns the parallel stdio/Loader process boundary used by keyless example e2e suites. `subagent-mock` exercises the real `ctx.subagents` load path without a model or child agent. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.
`invariants` is development support but has no environment guard: it runs wherever registered, and the default `dsh-agent-spine-demo` bundle mounts it unconditionally. `agent-loop-testkit` centralizes the mandatory service spine for hand-built AgentLoop tests without owning their loop or scenario. `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate. `acp-snapshot` carries the ACP subprocess/client boundary plus the snapshot harness, normalizers, and suite machinery, while `loader-smoke` owns the parallel stdio/Loader process boundary used by keyless example e2e suites. `subagent-mock` exercises the real `ctx.subagents` load path without a model or child agent. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.

View File

@@ -5,7 +5,7 @@ The ACP snapshot suite kit: the shared machinery behind the keyless snapshot tie
Four layers, importable separately:
- **`launchAcpTestAgent` (launcher)** — boots an unbuilt ACP agent from a temp cwd, pins tsx to the repo tsconfig, connects the SDK client over a raw-byte stdout tee, collects session updates and stderr, surfaces asynchronous spawn failures through its startup lifecycle, fails closed on unhandled permission requests, and owns graceful or signalled shutdown. Shutdown waits for process exit, inherited stdio closure, and ACP parser exhaustion before resolving or propagating a child error, so captures are complete and callers can remove owned paths after either outcome. Snapshot and ordinary e2e suites share this process boundary; a test supplies only agent paths, cwd, environment overrides, and any permission policy.
- **`runScenario` (harness)** — boots the real agent bin as a subprocess via tsx (unbuilt, Loader path), drives it over ACP JSON-RPC stdio from a deterministic `input.json` script, tees raw stdout for the golden + purity check, and harvests every persisted session JSONL (parent + subagent children, primary-first) after a graceful stdin-EOF shutdown. Parameterized by `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath` — absolute paths; the subprocess cwd is a temp dir outside the repo).
- **`runScenario` (harness)** — boots the real agent bin as a subprocess via tsx (unbuilt, Loader path), drives it over ACP JSON-RPC stdio from a deterministic `input.json` script, tees raw stdout for the golden + purity check, and harvests every persisted session JSONL (parent + subagent children, primary-first) after a graceful stdin-EOF shutdown. Parameterized by `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath` — absolute paths; the subprocess cwd is a temp dir outside the repo). Startup failures preserve captured agent stderr in the rejected diagnostic.
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header RFC](../../../docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, the per-header-class pin (`system-prompt.golden.md` plus `tool-schemas.golden.json`) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt/schema-scrubbed, non-pinning fixtures fully header-scrubbed). Each scenario directory's `session.jsonl` plus contiguous `session.<n>.jsonl` siblings are the ordered primary/child inventory; the scenario table does not duplicate their count. Must be called at vitest collection time.

View File

@@ -157,9 +157,11 @@ export interface RunOptions {
export async function runScenario(input: InputScript, opts: RunOptions): Promise<RunResult> {
const cwd = await mkdtemp(join(tmpdir(), 'acp-snap-cwd-'))
const sessionsRoot = await mkdtemp(join(tmpdir(), 'acp-snap-sessions-'))
// Everything past the temp-dir creation runs under a try/finally that always
// removes both dirs — so a failure in workspace seeding, spawn, or any step
// never leaks them (the "e2e tests own their resources" rule).
// Fixed path length: spill-policy budgets the preview against the REAL path
// before stdout normalization, so tmpdir() length differences churn goldens.
const spillRoot = '/tmp/dsh-acp-snapshot-spill'
// Everything past the temp-dir creation is followed by failure-safe cleanup,
// so a failure in workspace seeding, spawn, or any step never leaks resources.
let launched: LaunchedAcpTestAgent | undefined
let sessionId: string | undefined
let sessionLogs: HarvestedLog[] = []
@@ -174,6 +176,9 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
DSH_SNAPSHOT: opts.mode,
DSH_SNAPSHOT_FILE: opts.fixtureFile,
DSH_SNAPSHOT_SESSIONS_ROOT: sessionsRoot,
DSH_SNAPSHOT_SPILL_ROOT: spillRoot,
DSH_HOME: join(cwd, '.dsh'),
DSH_AGENTS_HOME: join(cwd, '.agents'),
...opts.overrideFile !== undefined ? { DSH_SNAPSHOT_OVERRIDE: opts.overrideFile } : {},
...opts.childFiles !== undefined && opts.childFiles.length > 0
? { DSH_SNAPSHOT_CHILD_FILES: opts.childFiles.join(delimiter) }
@@ -240,11 +245,19 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
}
})().then(
value => ({ status: 'fulfilled', value } as const),
(error: unknown) => ({ status: 'rejected', error } as const),
(error: unknown) => {
const stderr = launched?.stderr() ?? ''
return {
status: 'rejected',
error: stderr === ''
? error
: new Error(`snapshot-harness: scenario failed: ${String(error)}\nagent stderr:\n${stderr}`, { cause: error }),
} as const
},
)
// Failure-safe teardown: wait for a still-running child, then attempt BOTH
// directory removals even when an earlier cleanup rejects. Report every
// Failure-safe teardown: wait for a still-running child, then attempt every
// owned-path removal even when an earlier cleanup rejects. Report every
// teardown failure alongside a scenario failure so neither orthogonal
// outcome hides the other.
const cleanupResults: PromiseSettledResult<unknown>[] = []
@@ -255,6 +268,7 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
await cleanup(() => launched?.close('SIGKILL') ?? Promise.resolve())
await cleanup(() => rm(cwd, { recursive: true, force: true }))
await cleanup(() => rm(sessionsRoot, { recursive: true, force: true }))
await cleanup(() => rm(spillRoot, { recursive: true, force: true }))
const cleanupFailures = cleanupResults
.filter((result): result is PromiseRejectedResult => result.status === 'rejected')

View File

@@ -14,6 +14,16 @@ const MESSAGE_PREFIX = '{{messagePrefix}}'
/** A UUID v4 string, the shape `randomUUID()` produces for session ids. */
const UUID_RE = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi
const LOCAL_SPILL_PATH_RE = new RegExp(
String.raw`\{\{cwd\}\}/\.spill/session-[0-9a-f]{12}/[0-9a-f]{12}-([A-Za-z0-9._~-]+?)`
+ String.raw`(?=\. Use read with offset/limit|[\s)]|$)`,
'g',
)
const SNAPSHOT_SPILL_PATH_RE = new RegExp(
String.raw`/tmp/dsh-acp-snapshot-spill/session-[0-9a-f]{12}/[0-9a-f]{12}-([A-Za-z0-9._~-]+?)`
+ String.raw`(?=\. Use read with offset/limit|[\s)]|$)`,
'g',
)
/** Inputs the normalizers need to recognize a run's volatile values. */
export interface NormalizeContext {
@@ -29,6 +39,9 @@ function scrubString(value: string, ctx: NormalizeContext): string {
// cwd first (longest, most specific), then explicit session ids, then any
// residual UUID (covers ids that appear in places we didn't enumerate).
out = out.split(ctx.cwd).join(CWD)
out = out.split(`/private${CWD}`).join(CWD)
out = out.replace(LOCAL_SPILL_PATH_RE, (_match, name: string) => `{{spillLocator:${name}}}`)
out = out.replace(SNAPSHOT_SPILL_PATH_RE, (_match, name: string) => `{{spillLocator:${name}}}`)
for (const id of ctx.sessionIds) out = out.split(id).join(SESSION_ID)
out = out.replace(UUID_RE, SESSION_ID)
return out

View File

@@ -4,6 +4,9 @@
* both replay input and expected output. Record mode refreshes reproducible
* model scenarios from the live API, while refresh mode replays committed
* scripts and rewrites derived artifacts without a key.
* Replay scenarios run concurrently because each subprocess owns unique temp
* cwd and persistence roots and reads only committed fixtures. Record and
* refresh stay serial while writing.
*
* Exactly one scenario per header-composition class pins the full prompt and
* tool-schema sequences in dedicated sidecars. Every live header is checked
@@ -424,7 +427,7 @@ export function stabilizeRefreshLog(fresh: string, existing: string, replacement
}
/**
* Register the suite: one `describe` per scenario (the golden/log compares and
* Register the suite: one test per scenario (the golden/log compares and
* the header-uniformity guard) plus the fixture guard block (no orphan
* scenario dirs, required files present, exactly one pin per header class,
* pinning fixtures well-formed, every JSONL prompt-scrubbed, non-pinning
@@ -440,6 +443,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
const RECORDING = mode === 'record'
const REFRESHING = mode === 'refresh'
const childMode: 'replay' | 'record' = RECORDING ? 'record' : 'replay'
const scenarioSuite = mode === 'replay' ? describe.concurrent : describe
/** The class a scenario's header composition belongs to (see {@link Scenario.headerClass}). */
const classOf = (scenario: Scenario): string => scenario.headerClass ?? 'default'
@@ -459,11 +463,11 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
}
}
for (const scenario of scenarios) {
describe(`snapshot: ${scenario.name}`, () => {
scenarioSuite('snapshot scenarios', () => {
for (const scenario of scenarios) {
// In RECORD mode, only re-run the `recorded` (live-API) scenarios; the `authored` ones
// (sidecar-driven errors/cancel) are never re-recorded.
it.skipIf(RECORDING && !scenario.recorded)('matches the goldens', async () => {
it.skipIf(RECORDING && !scenario.recorded)(`snapshot: ${scenario.name} matches the goldens`, async ({ expect }) => {
const dir = join(snapshotsDir, scenario.name)
const input = JSON.parse(await readFile(join(dir, 'input.json'), 'utf8')) as InputScript
const overrideFile = join(dir, 'replay.override.json')
@@ -642,8 +646,8 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
}
}
})
})
}
}
})
describe('snapshot fixtures', () => {
it('every scenario directory is registered (no orphans)', async () => {

View File

@@ -25,6 +25,8 @@ interface ScriptedLog {
/** The whole scripted behavior for one run. Every field defaults to the least surprising choice. */
interface Behavior {
/** Exit during startup after writing any configured stderr note. */
failOnBoot?: boolean
/** Reject every `session/new` (exercises the expect-error step without extra dirs). */
rejectNewSession?: boolean
/** Reject `session/new` only when `additionalDirectories` is non-empty (the real bridge's rule). */
@@ -65,6 +67,7 @@ const behavior: Behavior = fixtureFile === ''
: JSON.parse(readFileSync(join(dirname(fixtureFile), 'behavior.json'), 'utf8')) as Behavior
if (behavior.stderrNote !== undefined) process.stderr.write(`${behavior.stderrNote}\n`)
if (behavior.failOnBoot === true) process.exit(7)
let nextOutboundId = 1000
let sessionId = ''

View File

@@ -234,6 +234,14 @@ describe('runScenario', () => {
expect(permissionFinished).toBe(true)
})
it('includes agent stderr when the ACP connection closes during startup', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({ failOnBoot: true, stderrNote: 'fake agent requested startup failure' })
await expect(runScenario(
{ steps: [{ op: 'initialize' }] },
{ agent: AGENT, mode: 'replay', fixtureFile },
)).rejects.toThrow(/agent stderr:\nfake agent requested startup failure/)
})
it('drives a full turn: initialize (terminal caps), session, prompt, permission stub, harvest', { timeout: 20_000 }, async () => {
const { fixtureFile } = await scenario({
permissionProbe: true,

View File

@@ -93,6 +93,52 @@ describe('normalizeSessionLog', () => {
expect(out).not.toContain(ctx.cwd)
})
it('scrubs random local spill paths under the snapshot cwd', () => {
const ev = JSON.stringify({
type: 'tool/result', seq: 2, time: 5,
data: {
content: [{
type: 'text',
text: `Full formatted result stored at: ${ctx.cwd}/.spill/session-c22bc3f1d2af/8a7b6c5d4e3f-bash.txt. Use read with offset/limit, or grep this path to search within it.`,
}],
},
})
const out = normalizeSessionLog(`${header({ cwd: ctx.cwd })}\n${ev}\n`, ctx)
expect(out).toContain('{{spillLocator:bash.txt}}')
expect(out).not.toContain('session-c22bc3f1d2af')
expect(out).not.toContain('8a7b6c5d4e3f')
})
it('scrubs macOS /private aliases for local spill paths', () => {
const ev = JSON.stringify({
type: 'tool/result', seq: 2, time: 5,
data: {
content: [{
type: 'text',
text: `Full formatted result stored at: /private${ctx.cwd}/.spill/session-c22bc3f1d2af/8a7b6c5d4e3f-bash.txt. Use read with offset/limit, or grep this path to search within it.`,
}],
},
})
const out = normalizeSessionLog(`${header({ cwd: ctx.cwd })}\n${ev}\n`, ctx)
expect(out).toContain('{{spillLocator:bash.txt}}')
expect(out).not.toContain('/private{{spillLocator')
})
it('scrubs fixed snapshot spill paths', () => {
const ev = JSON.stringify({
type: 'tool/result', seq: 2, time: 5,
data: {
content: [{
type: 'text',
text: 'Full formatted result stored at: /tmp/dsh-acp-snapshot-spill/session-c22bc3f1d2af/8a7b6c5d4e3f-bash.txt. Use read with offset/limit, or grep this path to search within it.',
}],
},
})
const out = normalizeSessionLog(`${header({ cwd: ctx.cwd })}\n${ev}\n`, ctx)
expect(out).toContain('{{spillLocator:bash.txt}}')
expect(out).not.toContain('/tmp/dsh-acp-snapshot-spill')
})
it('scrubs the session id in the header', () => {
const out = normalizeSessionLog(`${header({ id: ctx.sessionIds[0] })}\n`, ctx)
expect(out).toContain('{{sessionId}}')

View File

@@ -0,0 +1,27 @@
# `@deepseek-ai/dsh-agent-loop-testkit`
Shared prerequisite mounting for tests that exercise the concrete `AgentLoop`. `mountAgentLoopTestDependencies(ctx, options?)` installs the LLM, session, system-prompt, tool, and agent services in dependency order, then returns before the loop is mounted.
The caller registers adapters and optional plugins, mounts `AgentLoop` with the configuration under test, and disposes its own Context. System-prompt and tool-registry configuration can be forwarded through `options`; the helper does not provide test defaults beyond those owned by the services. A plugin-load failure rejects the helper call, while services activated earlier in the sequence remain owned by the caller's Context.
```ts
import { Context } from 'cordis'
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
import { mountAgentLoopTestDependencies } from '@deepseek-ai/dsh-agent-loop-testkit'
const ctx = new Context()
await mountAgentLoopTestDependencies(ctx)
// Register the test adapter and any optional plugins here.
await ctx.plugin(AgentLoop, { agents: [] })
```
Tests of injection failures, partial topology, service load order, or service teardown mount their dependencies directly instead of using this helper.
## Model Experience
None, as this test-only composition helper neither drives nor modifies model requests.
## Known Limitations and Deferred Work
- **Only the mandatory prerequisite spine is shared** — adapters, optional plugins, `AgentLoop`, agents, and Context teardown remain caller-owned so scenario-specific ordering stays visible.

View File

@@ -0,0 +1,41 @@
{
"name": "@deepseek-ai/dsh-agent-loop-testkit",
"description": "Shared prerequisite mounting for tests that exercise the concrete agent loop",
"version": "0.0.1",
"private": true,
"type": "module",
"main": "lib/index.js",
"types": "lib/types/index.d.ts",
"exports": {
".": {
"types": "./lib/types/index.d.ts",
"default": "./lib/index.js"
},
"./src/*": "./src/*",
"./package.json": "./package.json"
},
"files": [
"lib/index.js",
"lib/types/**/*.d.ts",
"lib/types/**/*.d.ts.map",
"src"
],
"license": "BSD-3-Clause",
"peerDependencies": {
"@deepseek-ai/dsh-agent": "^0.0.1",
"@deepseek-ai/dsh-llm": "^0.0.1",
"@deepseek-ai/dsh-session": "^0.0.1",
"@deepseek-ai/dsh-system-prompt": "^0.0.1",
"@deepseek-ai/dsh-tools": "^0.0.1",
"cordis": "^4.0.0-rc.7"
},
"devDependencies": {
"@deepseek-ai/dsh-agent": "workspace:^",
"@deepseek-ai/dsh-agent-loop": "workspace:^",
"@deepseek-ai/dsh-llm": "workspace:^",
"@deepseek-ai/dsh-session": "workspace:^",
"@deepseek-ai/dsh-system-prompt": "workspace:^",
"@deepseek-ai/dsh-tools": "workspace:^",
"cordis": "^4.0.0-rc.7"
}
}

View File

@@ -0,0 +1,46 @@
/**
* Shared mounting for the services required before tests load the concrete
* agent loop. The caller retains ownership of the context, loop, adapters,
* optional plugins, and teardown.
* @module @deepseek-ai/dsh-agent-loop-testkit
*/
import type { Context } from 'cordis'
import AgentRegistry from '@deepseek-ai/dsh-agent'
import LlmService from '@deepseek-ai/dsh-llm'
import SessionStore from '@deepseek-ai/dsh-session'
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
import type { Config as SystemPromptConfig } from '@deepseek-ai/dsh-system-prompt'
import ToolRegistry from '@deepseek-ai/dsh-tools'
import type { Config as ToolRegistryConfig } from '@deepseek-ai/dsh-tools'
/** Configuration forwarded to the prerequisite service plugins. */
export interface AgentLoopTestDependenciesOptions {
/** Configuration for the system-prompt registry. */
readonly systemPrompt?: SystemPromptConfig
/** Configuration for the tool registry. */
readonly tools?: ToolRegistryConfig
}
/**
* Mount the standard prerequisite services for an AgentLoop test.
*
* The function deliberately does not mount AgentLoop or register an adapter,
* so tests retain control of load order and the topology under test. The
* context owns every mounted service and remains responsible for disposal. A
* plugin-load failure rejects the promise; services activated earlier in the
* sequence remain context-owned and unwind with that context.
* @param ctx - test context that owns the mounted services.
* @param options - optional service configuration forwarded without mutation.
* @returns after every prerequisite service has activated.
*/
export async function mountAgentLoopTestDependencies(
ctx: Context,
options: AgentLoopTestDependenciesOptions = {},
): Promise<void> {
await ctx.plugin(LlmService)
await ctx.plugin(SessionStore)
await ctx.plugin(SystemPrompt, options.systemPrompt ?? {})
await ctx.plugin(ToolRegistry, options.tools ?? {})
await ctx.plugin(AgentRegistry)
}

View File

@@ -0,0 +1,20 @@
import { describe, expect, it } from 'vitest'
import { Context } from 'cordis'
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
import { renderPrompt } from '@deepseek-ai/dsh-system-prompt'
import { mountAgentLoopTestDependencies } from '../src/index.ts'
describe('dsh-agent-loop-testkit', () => {
it('mounts a configurable prerequisite spine that can activate AgentLoop', async () => {
const ctx = new Context()
await mountAgentLoopTestDependencies(ctx, {
systemPrompt: { persona: 'Test persona.' },
tools: { mode: 'native' },
})
expect(renderPrompt(await ctx.systemPrompt.assemble())).toContain('Test persona.')
await expect(ctx.plugin(AgentLoop, { agents: [] })).resolves.toBeDefined()
await ctx.fiber.dispose()
})
})

View File

@@ -0,0 +1,33 @@
{
"extends": "../../../tsconfig.base.json",
"compilerOptions": {
"rootDir": "src",
"outDir": "lib/types"
},
"include": [
"src"
],
"references": [
{
"path": "../../../vendor/cosmokit"
},
{
"path": "../../../vendor/cordis"
},
{
"path": "../../core/agent"
},
{
"path": "../../llm/llm"
},
{
"path": "../../core/session"
},
{
"path": "../../core/system-prompt"
},
{
"path": "../../core/tools"
}
]
}

View File

@@ -4,7 +4,7 @@ Runtime event-contract assertions intended for development diagnostics. This pur
The plugin has no environment guard: it is active wherever it is registered. The default [`dsh-agent-spine-demo`](../../examples/agent-spine-demo/README.md) bundle mounts it unconditionally; a custom composition can omit it when the runtime cost is undesirable. It doubles as executable documentation of the event taxonomy — the assertions *are* the contract.
Session itself owns immutable log storage in every composition: it takes one lossless JSON snapshot of each accepted event, deep-freezes that record, and exposes the log through immutable array snapshots. The invariants plugin checks the cross-record and cross-seam rules that storage immutability cannot express.
Session itself owns immutable, surface-valid log storage in every composition: it takes one lossless JSON snapshot of each candidate, validates the complete surface transition, deep-freezes the accepted record, and exposes the log through immutable array snapshots. The invariants plugin checks the remaining cross-record and cross-seam rules that Session does not own.
Session-log assertions run during Cordis `internal/dispatch`, while `Session.append()` is resolving the `session/event` callback snapshot but before it pushes the candidate into the log. A valid transition is staged by exact event identity and applied to the live trace only when that same committed event reaches the plugin's contained post-commit listener. A later internal dispatch check can therefore veto without advancing either the log or the invariant trace, while ordinary `session/event` observer failures remain observe-only.
@@ -32,6 +32,7 @@ Session log (per session):
- **steps nest in turns** — `step/start` opens a step in the open turn; `step/end` closes the matching step.
- **chunks belong to an open step** — `step/start` precedes its `assistant/chunk`s.
- **a `tool/result` needs a prior `tool/call`** — but NOT the converse: a `tool/call` may have no result (a thrown tool-execution pipeline step ends the turn with no `tool/result`, which is legal).
- **provenance sources are valid and unambiguous** — `sourceEventSeqs` contains unique earlier known seqs; only `assistant/message` may carry an explicit empty list, which denotes a known empty provider stream rather than absent legacy provenance.
Agent status (per agent):

View File

@@ -3,7 +3,8 @@
* turn and step nesting, scoped dispatch, status transitions, and request
* reconstruction. The plugin has no environment guard and is active wherever
* mounted, including the default `dsh-agent-spine-demo` bundle; custom compositions
* may omit it. Sessions still own event snapshots and freezing.
* may omit it. Sessions own immutable, surface-valid event storage; this plugin
* checks only relationships that event acceptance cannot express.
* @module @deepseek-ai/dsh-invariants
*/
@@ -13,7 +14,7 @@ import { assertNever, HarnessError } from '@deepseek-ai/dsh-llm'
import type { CallId, GenerateOptions } from '@deepseek-ai/dsh-llm'
import type { Agent, AgentStatus } from '@deepseek-ai/dsh-agent'
import { Session, SessionId, foldRequestHeader } from '@deepseek-ai/dsh-session'
import type { SessionEvent, SurfaceEventType } from '@deepseek-ai/dsh-session'
import type { SessionEvent } from '@deepseek-ai/dsh-session'
import { scopedSubjectResolverFor } from './scoped-events.generated.ts'
export const name = 'invariants'
@@ -48,15 +49,6 @@ interface SessionTrace {
* `step/end` — a result must arrive in the same step as its call.
*/
pendingCalls: Set<CallId>
/** Every seq seen so far — validates `sourceEventSeqs` references. */
knownSeqs: Set<number>
/**
* The seqs currently on the surface, in derived-message order. A replace
* reorders this relative to seq order (the new
* node takes the replaced range's position), so range validation is
* positional, not by seq comparison.
*/
surface: number[]
}
/** One accepted event's deferred mutation of a live session trace. */
@@ -68,12 +60,6 @@ interface SessionTraceTransition {
| { kind: 'none' }
| { kind: 'add' | 'delete'; callId: CallId }
| { kind: 'clear' }
/** The event's mutation of the derived surface order. */
surface:
| { kind: 'none' | 'append' }
| { kind: 'replace'; start: number; count: number }
/** The committed event sequence to add to the known-sequence set. */
seq: number
}
/** Assert that a step-scoped event names the currently open turn and step. */
@@ -97,73 +83,6 @@ function validateEvent(trace: SessionTrace, event: SessionEvent): SessionTraceTr
let nextTurn = trace.nextTurn
let nextStep = trace.nextStep
let pendingCalls: SessionTraceTransition['pendingCalls'] = { kind: 'none' }
let surface: SessionTraceTransition['surface'] = { kind: 'none' }
// --- Surface invariants ---
// Surface metadata (sourceEventSeqs, surfaceOp) is only valid on
// surface-eligible event types. The compiler enforces this at append()
// call sites; this runtime check catches casts and persisted data.
const SURFACE_TYPES = new Set<string>(['user/message', 'assistant/message', 'tool/result', 'context/message', 'steering/message'])
// Cast to surface-eligible event type so we can access surfaceOp and
// sourceEventSeqs (optional on SessionEvent, mandatory on SurfaceEvent).
// SurfaceEvent's mandatory surfaceOp is too strict here — we need to
// CHECK whether surface metadata is present, not assume it.
const se = event as SessionEvent<SurfaceEventType>
if (!SURFACE_TYPES.has(event.type)) {
if (se.sourceEventSeqs !== undefined) {
throw new InvariantError(`${event.type} cannot carry sourceEventSeqs (non-surface event)`)
}
if (se.surfaceOp !== undefined) {
throw new InvariantError(`${event.type} cannot carry surfaceOp (non-surface event)`)
}
}
if (se.sourceEventSeqs !== undefined) {
if (se.sourceEventSeqs.length === 0) {
throw new InvariantError('sourceEventSeqs must not be empty when present')
}
const unique = new Set(se.sourceEventSeqs)
if (unique.size !== se.sourceEventSeqs.length) {
throw new InvariantError('sourceEventSeqs must not contain duplicates')
}
for (const ref of se.sourceEventSeqs) {
if (ref >= event.seq) {
throw new InvariantError(`sourceEventSeqs must reference earlier events: ${ref} >= current seq ${event.seq}`)
}
if (!trace.knownSeqs.has(ref)) {
throw new InvariantError(`sourceEventSeqs references unknown seq ${ref}`)
}
}
}
// Fold this event into the tracked surface order, validating the
// replace contract as we go. `append` adds a tail node; `replace` shadows a
// positional range — every shadowed node must appear in sourceEventSeqs.
if (se.surfaceOp !== undefined) {
if (se.surfaceOp === 'append') {
surface = { kind: 'append' }
} else {
const { start, end } = se.surfaceOp
const startIdx = trace.surface.indexOf(start)
if (startIdx === -1) {
throw new InvariantError(`surface replace: start seq ${start} is not on the surface`)
}
const endIdx = trace.surface.indexOf(end)
if (endIdx === -1) {
throw new InvariantError(`surface replace: end seq ${end} is not on the surface`)
}
if (startIdx > endIdx) {
throw new InvariantError(`surface replace: start seq ${start} (pos ${startIdx}) is after end seq ${end} (pos ${endIdx}) on the surface`)
}
// Every node the replace shadows (surface positions [startIdx, endIdx]
// inclusive) must appear in sourceEventSeqs — the provenance contract.
const shadowed = trace.surface.slice(startIdx, endIdx + 1)
const recorded = new Set(se.sourceEventSeqs ?? [])
const missing = shadowed.filter(seq => !recorded.has(seq))
if (missing.length > 0) {
throw new InvariantError(`surface replace: sourceEventSeqs must include every shadowed surface node; missing ${missing.join(', ')}`)
}
surface = { kind: 'replace', start: startIdx, count: shadowed.length }
}
}
// Boundary/step-scoped events have explicit cases; every OTHER event type —
// including plugin-added (merge-extensible) SessionEventMap keys — is caught
@@ -263,8 +182,6 @@ function validateEvent(trace: SessionTrace, event: SessionEvent): SessionTraceTr
return {
scalars: { lastSeq: event.seq, openTurn, openStep, nextTurn, nextStep },
pendingCalls,
surface,
seq: event.seq,
}
}
@@ -287,20 +204,6 @@ function applyTransition(trace: SessionTrace, transition: SessionTraceTransition
default:
assertNever(transition.pendingCalls, 'session trace pending-call transition')
}
switch (transition.surface.kind) {
case 'none':
break
case 'append':
trace.surface.push(transition.seq)
break
case 'replace':
trace.surface.splice(transition.surface.start, transition.surface.count, transition.seq)
break
/* v8 ignore next -- validateEvent produces this closed transition union */
default:
assertNever(transition.surface, 'session trace surface transition')
}
trace.knownSeqs.add(transition.seq)
}
/** Validate and apply one event while rebuilding an already-committed log. */
@@ -345,8 +248,6 @@ export function apply(ctx: Context): void {
nextTurn: 1,
nextStep: 1,
pendingCalls: new Set(),
knownSeqs: new Set(),
surface: [],
})
/** Build (or rebuild) a session's trace by replaying its whole log. */

View File

@@ -48,7 +48,7 @@ describe('session-log invariants', () => {
session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
session.append('step/start', { turn: 1, step: 1 })
session.append('assistant/chunk', { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'h' } })
session.append('assistant/message', { turn: 1, step: 1, content: [{ type: 'tool-call', id: CallId('c1'), name: 'echo', arguments: '{}' }] }, { surfaceOp: 'append' })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [{ type: 'tool-call', id: CallId('c1'), name: 'echo', arguments: '{}' }] }, { surfaceOp: 'append' })
session.append('tool/call', { turn: 1, step: 1, callId: CallId('c1'), name: 'echo', arguments: '{}' })
session.append('tool/result', { turn: 1, step: 1, callId: CallId('c1'), content: [{ type: 'text', text: 'ok' }], isError: false }, { surfaceOp: 'append' })
session.append('step/end', { turn: 1, step: 1 })
@@ -195,7 +195,7 @@ describe('session-log invariants', () => {
expect(() => {
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('step/start', { turn: 1, step: 1 })
session.append('assistant/message', { turn: 1, step: 1, content: [
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [
{ type: 'tool-call', id: CallId('crashed'), name: 'bash', arguments: '{}' },
] }, { surfaceOp: 'append' })
session.append('tool/result', {
@@ -251,10 +251,10 @@ describe('session-log invariants', () => {
expect(() => {
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('step/start', { turn: 1, step: 1 })
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append' })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append' })
session.append('step/end', { turn: 1, step: 1 })
session.append('step/start', { turn: 1, step: 2 })
session.append('assistant/message', { turn: 1, step: 2, content: [] }, { surfaceOp: 'append' })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 2, content: [] }, { surfaceOp: 'append' })
session.append('step/end', { turn: 1, step: 2 })
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
session.append('turn/start', { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } })
@@ -316,7 +316,7 @@ describe('session-log invariants', () => {
const session = ctx.sessions.create()
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('step/start', { turn: 1, step: 1 })
expect(() => session.append('assistant/message', { turn: 1, step: 2, content: [] }, { surfaceOp: 'append' }))
expect(() => session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 2, content: [] }, { surfaceOp: 'append' }))
.toThrow(/open is turn 1\/step 1/)
})
})
@@ -464,7 +464,7 @@ describe('HMR safety', () => {
})
})
describe('surface invariants', () => {
describe('surface contract under the invariants composition', () => {
it('accepts well-formed surface metadata', async () => {
const { ctx } = await setup()
const session = ctx.sessions.create()
@@ -473,7 +473,7 @@ describe('surface invariants', () => {
session.append('step/start', { turn: 1, step: 1 })
expect(() => {
session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
}).not.toThrow()
})
@@ -483,17 +483,21 @@ describe('surface invariants', () => {
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('step/start', { turn: 1, step: 1 })
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] })
// no throw — well-formed replace op
})
it('rejects empty sourceEventSeqs', async () => {
it('accepts known-empty assistant provenance and rejects empty provenance elsewhere', async () => {
const { ctx } = await setup()
const session = ctx.sessions.create()
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('step/start', { turn: 1, step: 1 })
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [] })
}).toThrow(InvariantError)
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [] })
}).not.toThrow()
expect(() => {
session.append('user/message', { content: [], source: { kind: 'user' } }, { surfaceOp: 'append', sourceEventSeqs: [] })
}).toThrow(/must not be empty except on assistant\/message/)
})
it('rejects duplicate sourceEventSeqs', async () => {
@@ -502,7 +506,7 @@ describe('surface invariants', () => {
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1, 1] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1, 1] })
}).toThrow(/must not contain duplicates/)
})
@@ -513,19 +517,20 @@ describe('surface invariants', () => {
// The next event is seq 1. Referencing its own seq fails on "must reference
// earlier events" (the check order is: earlier first, then unknown).
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
}).toThrow(/must reference earlier/)
})
it('accepts sourceEventSeqs referencing a valid earlier event', async () => {
// Positive test: ref < current seq and ref is in knownSeqs → passes.
// Session seqs are contiguous, so every non-negative ref below the current
// seq necessarily names an existing earlier event.
const { ctx } = await setup()
const session = ctx.sessions.create()
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('step/start', { turn: 1, step: 1 })
// seqs so far: 0, 1. The next event at seq 2 references seq 1 → valid.
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
}).not.toThrow()
})
@@ -534,27 +539,10 @@ describe('surface invariants', () => {
const session = ctx.sessions.create()
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [99] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [99] })
}).toThrow(/must reference earlier/)
})
it('rejects sourceEventSeqs referencing unknown seq (gap in event log)', async () => {
// Create an impossible-through-public-API gap so seq 2 is earlier but unknown.
const { ctx } = await setup()
const session = ctx.sessions.create()
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('step/start', { turn: 1, step: 1 })
;(session as unknown as { log: unknown[] }).log.push({
type: 'assistant/chunk',
seq: 3,
time: Date.now(),
data: { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'x' } },
})
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [2] })
}).toThrow(/unknown seq 2/)
})
it('rejects a replace whose start is positioned after its end on the surface', async () => {
const { ctx } = await setup()
const session = ctx.sessions.create()
@@ -564,8 +552,8 @@ describe('surface invariants', () => {
session.append('user/message', { content: [{ type: 'text', text: 'b' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 3
// Reversed range: start seq 3 is at a later surface position than end seq 2.
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 3, end: 2 }, sourceEventSeqs: [2, 3] })
}).toThrow(/is after end seq 2 .* on the surface/)
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 3, end: 2 }, sourceEventSeqs: [2, 3] })
}).toThrow(/is after end seq 2/)
})
it('rejects a replace whose sourceEventSeqs omits a shadowed surface node', async () => {
@@ -577,7 +565,7 @@ describe('surface invariants', () => {
session.append('user/message', { content: [{ type: 'text', text: 'b' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 3
// Replace shadows surface nodes [2, 3] but records provenance for only [2].
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [{ type: 'text', text: 'sum' }] }, { surfaceOp: { op: 'replace', start: 2, end: 3 }, sourceEventSeqs: [2] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [{ type: 'text', text: 'sum' }] }, { surfaceOp: { op: 'replace', start: 2, end: 3 }, sourceEventSeqs: [2] })
}).toThrow(/must include every shadowed surface node; missing 3/)
})
@@ -589,7 +577,7 @@ describe('surface invariants', () => {
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 2
session.append('user/message', { content: [{ type: 'text', text: 'b' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 3
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [{ type: 'text', text: 'sum' }] }, { surfaceOp: { op: 'replace', start: 2, end: 3 }, sourceEventSeqs: [2, 3] })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [{ type: 'text', text: 'sum' }] }, { surfaceOp: { op: 'replace', start: 2, end: 3 }, sourceEventSeqs: [2, 3] })
}).not.toThrow()
})
@@ -601,8 +589,8 @@ describe('surface invariants', () => {
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 2
// seq 1 (step/start) is a real earlier event but never entered the surface.
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 1, end: 2 }, sourceEventSeqs: [1, 2] })
}).toThrow(/start seq 1 is not on the surface/)
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 1, end: 2 }, sourceEventSeqs: [1, 2] })
}).toThrow(/start seq 1 not found in surface/)
})
it('rejects a replace naming an end seq that is not on the surface', async () => {
@@ -613,8 +601,8 @@ describe('surface invariants', () => {
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 2
// start (2) is on the surface but end (99) never entered it.
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 99 }, sourceEventSeqs: [2] })
}).toThrow(/end seq 99 is not on the surface/)
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 99 }, sourceEventSeqs: [2] })
}).toThrow(/end seq 99 not found in surface/)
})
it('rejects a replace whose range is reversed in surface position after a prior replace reordered it', async () => {
@@ -626,12 +614,12 @@ describe('surface invariants', () => {
session.append('user/message', { content: [{ type: 'text', text: 'b' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 3
// Replace node 2 (position 0) with seq 4 — surface is now [4, 3], so seq 4
// precedes seq 3 in surface order even though 4 > 3 numerically.
session.append('assistant/message', { turn: 1, step: 1, content: [{ type: 'text', text: 's' }] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] }) // seq 4
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [{ type: 'text', text: 's' }] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] }) // seq 4
// A replace with start=3, end=4 passes the seq check (3 <= 4) but is
// reversed positionally (3 is at pos 1, 4 is at pos 0).
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 3, end: 4 }, sourceEventSeqs: [3, 4] }) // seq 5
}).toThrow(/is after end seq 4 .* on the surface/)
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 3, end: 4 }, sourceEventSeqs: [3, 4] }) // seq 5
}).toThrow(/is after end seq 4/)
})
it('accepts a replace whose start seq exceeds its end seq when the surface position order is valid', async () => {
@@ -645,9 +633,9 @@ describe('surface invariants', () => {
// head seq (4) is numerically GREATER than the tail seq (3): the surface is
// not seq-ordered. A replace spanning start=4 (pos 0) … end=3 (pos 1) is
// valid positionally and must be accepted even though start seq > end seq.
session.append('assistant/message', { turn: 1, step: 1, content: [{ type: 'text', text: 's' }] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] }) // seq 4
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [{ type: 'text', text: 's' }] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] }) // seq 4
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 4, end: 3 }, sourceEventSeqs: [4, 3] }) // seq 5
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 4, end: 3 }, sourceEventSeqs: [4, 3] }) // seq 5
}).not.toThrow()
})
@@ -659,7 +647,7 @@ describe('surface invariants', () => {
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 2
// A replace with no sourceEventSeqs records no provenance for the node it shadows.
expect(() => {
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 2 } })
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 2 } })
}).toThrow(/must include every shadowed surface node; missing 2/)
})
@@ -670,30 +658,11 @@ describe('surface invariants', () => {
{ type: 'step/start' as const, seq: 1, time: 0, data: { turn: 1, step: 1 } },
{ type: 'user/message' as const, seq: 2, time: 0, data: { content: [{ type: 'text' as const, text: 'a' }], source: { kind: 'user' as const } }, surfaceOp: 'append' as const },
{ type: 'user/message' as const, seq: 3, time: 0, data: { content: [{ type: 'text' as const, text: 'b' }], source: { kind: 'user' as const } }, surfaceOp: 'append' as const },
{ type: 'assistant/message' as const, seq: 4, time: 0, data: { turn: 1, step: 1, content: [{ type: 'text' as const, text: 'sum' }] }, surfaceOp: { op: 'replace' as const, start: 2, end: 3 }, sourceEventSeqs: [2] },
{ type: 'assistant/message' as const, seq: 4, time: 0, data: { turn: 1, step: 1, content: [{ type: 'text' as const, text: 'sum' }], provenance: { provider: 'mock', model: 'mock' } }, surfaceOp: { op: 'replace' as const, start: 2, end: 3 }, sourceEventSeqs: [2] },
]
expect(() => ctx.sessions.create(undefined, { seed: badSeed })).toThrow(/must include every shadowed surface node; missing 3/)
})
it('rejects sourceEventSeqs on a non-surface event', async () => {
const { ctx } = await setup()
const session = ctx.sessions.create()
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
// Session rejects this at its own acceptance boundary. Emit a hand-built
// record to cover the listener's defensive check for alternate producers.
const event = { type: 'turn/end', seq: 1, time: 1, data: { turn: 1, reason: { kind: 'completed' } }, sourceEventSeqs: [0] }
expect(() => { ctx.emit(scopeTarget(session, undefined), 'session/event', session, event as never) })
.toThrow(/cannot carry sourceEventSeqs/)
})
it('rejects surfaceOp on a non-surface event', async () => {
const { ctx } = await setup()
const session = ctx.sessions.create()
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
const event = { type: 'turn/end', seq: 1, time: 1, data: { turn: 1, reason: { kind: 'completed' } }, surfaceOp: 'append' }
expect(() => { ctx.emit(scopeTarget(session, undefined), 'session/event', session, event as never) })
.toThrow(/cannot carry surfaceOp/)
})
})
describe('request-reconstruction cross-check (llm/stream)', () => {
@@ -705,7 +674,7 @@ describe('request-reconstruction cross-check (llm/stream)', () => {
session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
const boundary = session.deriveMessages()
session.append('step/start', { turn: 1, step: 1 })
session.append('request/header', { header: { config: { model: 'm' } }, reason: 'initial' })
session.append('request/header', { header: { config: { provider: 'mock', model: 'm' } }, reason: 'initial' })
return { ctx, session, boundary }
}
@@ -735,7 +704,7 @@ describe('request-reconstruction cross-check (llm/stream)', () => {
it('expects the folded header\'s session prefix ahead of the derivation (prefix + derived)', async () => {
const { ctx, session, boundary } = await requestSetup()
const prefix = { role: 'user' as const, content: [{ type: 'text' as const, text: '<system-reminder>catalog</system-reminder>' }] }
session.append('request/header', { header: { config: { model: 'm' }, messagePrefix: [prefix] }, reason: 'change' })
session.append('request/header', { header: { config: { provider: 'mock', model: 'm' }, messagePrefix: [prefix] }, reason: 'change' })
// The prefixed request matches the fold…
const prefixed = Object.freeze({ model: 'm', messages: Object.freeze([prefix, ...boundary]), sessionId: session.id })
expect(() => { dispatch(ctx, prefixed) }).not.toThrow()
@@ -804,7 +773,7 @@ describe('request cross-check ordering (prepend)', () => {
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
session.append('step/start', { turn: 1, step: 1 })
session.append('request/header', { header: { config: { model: 'm' } }, reason: 'initial' })
session.append('request/header', { header: { config: { provider: 'mock', model: 'm' } }, reason: 'initial' })
const divergent = Object.freeze({
model: 'm',

View File

@@ -1,6 +1,6 @@
# @deepseek-ai/dsh-llm-replay
A replay LLM plugin for keyless snapshot tests. It installs a single `llm/stream` waterfall listener that short-circuits the waterfall (never calls `next()`) and yields model streams reconstructed from a recorded **session JSONL** fixture so a test can boot the real agent against a fixed model transcript with no API key.
A replay LLM plugin for keyless snapshot tests. It yields model streams reconstructed from a recorded **session JSONL** fixture, so a test can boot the real agent against a fixed model transcript with no API key. With `providers` configured it registers a replay-only adapter whose catalog is visible to clients such as ACP editors; without `providers` it installs the catch-all `llm/stream` waterfall used by tests that do not need discovery.
Its consumer is the ACP snapshot harness in `examples/acp-agent`, which loads this plugin (via `cordis.snapshot.yml`) in place of a real LLM adapter. The package exists so its derive/parse/replay logic falls under the per-file 100% coverage gate on `packages/*/src` (the same logic, while it lived under `examples/`, was outside the gate).
@@ -23,10 +23,18 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
| `file` | string | `$DSH_SNAPSHOT_FILE` | Path to the primary (parent) `session.jsonl` fixture. Required (config or env). |
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | Optional path to a `ReplayEntry[]` sidecar that replaces the PRIMARY session's derived script. |
| `childFiles` | string[] | `$DSH_SNAPSHOT_CHILD_FILES` (path-delimited) | Recorded subagent child-session logs for a nested scenario; empty for a single-session scenario. |
| `providers` | `ReplayProviderConfig[]` | — | Optional replay-only provider and model catalog. Configured routes dispatch through the replay adapter and never perform provider I/O. |
```yaml
- id: llm-replay
name: '@deepseek-ai/dsh-llm-replay'
config:
providers:
- id: deepseek
name: DeepSeek
models:
- id: deepseek-v4-flash
- id: deepseek-v4-pro
# file/overrideFile/childFiles default to $DSH_SNAPSHOT_FILE /
# $DSH_SNAPSHOT_OVERRIDE / $DSH_SNAPSHOT_CHILD_FILES, set by the snapshot
# harness per scenario.
@@ -34,11 +42,11 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
## Exports
- `installLlmReplay(ctx, config)` — install the `llm/stream` listener; returns the disposer (HMR safety). Use this in tests to drive replay without the Loader or env vars.
- `installLlmReplay(ctx, config)` — install the configured replay adapter or catch-all `llm/stream` listener; returns the disposer (HMR safety). Use this in tests to drive replay without the Loader or env vars.
- `loadSessionScripts(config)` — resolve the ordered `SessionScript[]` (primary + children) for a scenario, ready to bind to live sessions in first-call order.
- `loadReplayScript(config)` — resolve the `ReplayEntry[]` for the PRIMARY session only (sidecar override if present, else derived from the JSONL; fail-loud if the fixture is missing).
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)` — the pure helpers that turn a recorded session log into a script and read its header `id`/`createdAt`. A derived group must end in a `finish` chunk; a group without one is the fingerprint of a thrown `stream()` and must instead be expressed via an override sidecar.
- Types `ReplayEntry` / `SessionScript` / `ReplayConfig` / `Config`.
- Types `ReplayEntry` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `Config`.
## Plugin export shape

View File

@@ -10,8 +10,8 @@ import { existsSync, readFileSync } from 'node:fs'
import { delimiter as pathDelimiter } from 'node:path'
import type { Context } from 'cordis'
import type { SessionEvent } from '@deepseek-ai/dsh-session'
import type { GenerateOptions, StreamChunk } from '@deepseek-ai/dsh-llm'
import { LlmError, assertNever } from '@deepseek-ai/dsh-llm'
import type { GenerateOptions, LlmModelInfo, LlmProviderInfo, StreamChunk } from '@deepseek-ai/dsh-llm'
import { LlmAdapter, LlmError, assertNever } from '@deepseek-ai/dsh-llm'
/**
* One recorded model call. `throw` may replay prefix chunks before failing;
@@ -23,6 +23,26 @@ export type ReplayEntry =
| { kind: 'throw'; chunks: StreamChunk[]; message: string; code: string; status?: number }
| { kind: 'hang' }
/** One model exposed by a replay-only provider catalog. */
export interface ReplayModelConfig {
/** Model id used for replay requests. */
id: string
/** Selector label; defaults to {@link id}. */
name?: string
/** Optional selector description. */
description?: string
}
/** One provider route exposed by the replay adapter. */
export interface ReplayProviderConfig {
/** Provider route used for replay requests. */
id: string
/** Selector label; defaults to {@link id}. */
name?: string
/** Advisory models exposed to clients such as ACP editors. */
models?: ReplayModelConfig[]
}
/** Resolved plugin configuration. */
export interface ReplayConfig {
/**
@@ -45,6 +65,12 @@ export interface ReplayConfig {
* for a single-session scenario.
*/
childFiles?: string[]
/**
* Optional provider catalog. When non-empty, replay registers an adapter for
* these routes; when absent or empty, it retains the catch-all waterfall used
* by tests that do not need discovery.
*/
providers?: ReplayProviderConfig[]
}
/**
@@ -203,6 +229,42 @@ export function loadSessionScripts(config: ReplayConfig): SessionScript[] {
return [primary, ...children]
}
/** Replay adapter that makes a configured provider catalog discoverable without provider I/O. */
class ReplayAdapter extends LlmAdapter {
private readonly providers: ReadonlyMap<string, ReplayProviderConfig>
constructor(
providers: readonly ReplayProviderConfig[],
private readonly replay: (options: GenerateOptions) => AsyncIterable<StreamChunk>,
) {
super()
this.providers = new Map(providers.map(provider => [provider.id, provider]))
}
override providerInfo(provider: string): LlmProviderInfo {
const configured = this.providers.get(provider)
/* v8 ignore next -- LlmService only asks about routes registered from this same map. */
if (configured === undefined) return super.providerInfo(provider)
return { id: provider, name: configured.name ?? provider }
}
override listModels(provider: string): Promise<readonly LlmModelInfo[]> {
const configured = this.providers.get(provider)
/* v8 ignore next -- LlmService only asks about routes registered from this same map. */
if (configured === undefined) return Promise.resolve([])
return Promise.resolve((configured.models ?? []).map(model => ({
provider,
id: model.id,
name: model.name ?? model.id,
...model.description === undefined ? {} : { description: model.description },
})))
}
override stream(options: GenerateOptions): AsyncIterable<StreamChunk> {
return this.replay(options)
}
}
/** Yield a recorded stream back, honoring abort like a real adapter. */
async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined): AsyncIterable<StreamChunk> {
switch (entry.kind) {
@@ -243,12 +305,14 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined)
/**
* Install per-session positional replay. A newly seen live session takes the
* next ordered recorded script, then advances its own cursor synchronously at
* invocation time; calls without `sessionId` share one anonymous session.
* Returns the effect disposer for HMR-safe removal.
* invocation time; calls without `sessionId` share one anonymous session. A
* non-empty provider catalog registers a routed replay adapter; otherwise a
* catch-all waterfall intercepts requests. Returns the effect disposer for
* HMR-safe removal.
*
* @param ctx - the context whose `llm/stream` waterfall the listener short-circuits.
* @param ctx - the context whose LLM service receives the replay route or waterfall.
* @param config - the resolved fixture paths (env-var defaulting is `apply`'s job).
* @returns the `ctx.on` disposer that removes the listener.
* @returns the disposer that removes the registered adapter or listener.
*/
export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void {
const scripts = loadSessionScripts(config)
@@ -258,7 +322,7 @@ export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void
const bound = new Map<string, { entries: ReplayEntry[]; cursor: number }>()
let nextScript = 0
const ANON = '\0anon\0' // the key for a call that carries no sessionId
return ctx.on('llm/stream', (options: GenerateOptions, _next) => {
const replay = (options: GenerateOptions): AsyncIterable<StreamChunk> => {
const key = options.sessionId ?? ANON
let state = bound.get(key)
let unrecorded = false
@@ -296,7 +360,12 @@ export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void
}
yield* replayEntry(entry, options.signal)
})()
})
}
const providers = config.providers ?? []
if (providers.length > 0) {
return ctx.llm.registerAdapter(providers.map(provider => provider.id), new ReplayAdapter(providers, replay))
}
return ctx.on('llm/stream', (options: GenerateOptions, _next) => replay(options))
}
export const name = 'llm-replay'
@@ -314,6 +383,8 @@ export interface Config {
* a nested-agent scenario; absent/empty for a single-session scenario.
*/
childFiles?: string[]
/** Optional replay-only provider catalog; absent or empty selects catch-all waterfall replay. */
providers?: ReplayProviderConfig[]
}
export function apply(ctx: Context, config: Config = {}): void {
@@ -329,5 +400,6 @@ export function apply(ctx: Context, config: Config = {}): void {
file,
...overrideFile !== undefined && overrideFile.length > 0 ? { overrideFile } : {},
...childFiles.length > 0 ? { childFiles } : {},
...config.providers !== undefined ? { providers: config.providers } : {},
})
}

View File

@@ -198,7 +198,7 @@ describe('loadReplayScript', () => {
})
})
describe('installLlmReplay (through the real waterfall)', () => {
describe('installLlmReplay (through the real LlmService)', () => {
function writeLog(...calls: StreamChunk[][]): void {
let seq = 1
const events: SessionEvent[] = []
@@ -214,7 +214,41 @@ describe('installLlmReplay (through the real waterfall)', () => {
await ctx.plugin(LlmService)
// No adapter registered for 'm' — replay must not reach it.
installLlmReplay(ctx, { file })
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
})
it('registers a replay-only provider catalog when configured', async () => {
writeLog(TEXT_CHUNKS)
const ctx = new Context()
await ctx.plugin(LlmService)
const dispose = installLlmReplay(ctx, {
file,
providers: [
{
id: 'deepseek',
name: 'DeepSeek',
models: [
{ id: 'flash' },
{ id: 'pro', name: 'Pro', description: 'Larger model' },
],
},
{ id: 'empty' },
],
})
expect(ctx.llm.listProviders()).toEqual([
{ id: 'deepseek', name: 'DeepSeek' },
{ id: 'empty', name: 'empty' },
])
await expect(ctx.llm.listModels('deepseek')).resolves.toEqual([
{ provider: 'deepseek', id: 'flash', name: 'flash' },
{ provider: 'deepseek', id: 'pro', name: 'Pro', description: 'Larger model' },
])
await expect(ctx.llm.listModels('empty')).resolves.toEqual([])
expect(await drain(ctx.llm.stream({ provider: 'deepseek', model: 'pro', messages: [] }))).toEqual(TEXT_CHUNKS)
dispose()
expect(ctx.llm.listProviders()).toEqual([])
})
it('serves the Nth call the Nth derived entry (positional)', async () => {
@@ -227,8 +261,8 @@ describe('installLlmReplay (through the real waterfall)', () => {
const ctx = new Context()
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file })
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(second)
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(second)
})
it('replays a sidecar throw-entry as an LlmError with code/status, after its prefix chunks', async () => {
@@ -244,7 +278,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
const seen: StreamChunk[] = []
await expect((async () => {
for await (const c of ctx.llm.stream({ model: 'm', messages: [] })) seen.push(c)
for await (const c of ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })) seen.push(c)
})()).rejects.toMatchObject({ message: 'unauthorized', code: 'AUTH', status: 401 })
expect(seen).toEqual(partial)
})
@@ -258,7 +292,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
installLlmReplay(ctx, { file, overrideFile })
const controller = new AbortController()
const iterator = ctx.llm.stream({ model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
const iterator = ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
// Deterministically consume the two pre-hang chunks (no sleep), then abort
// and assert the next pull rejects — event-driven, per the no-sleeps rule.
expect((await iterator.next()).value).toMatchObject({ type: 'block-start' })
@@ -272,8 +306,8 @@ describe('installLlmReplay (through the real waterfall)', () => {
const ctx = new Context()
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file })
await drain(ctx.llm.stream({ model: 'm', messages: [] }))
await expect(drain(ctx.llm.stream({ model: 'm', messages: [] }))).rejects.toThrow(/exhausted/)
await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).rejects.toThrow(/exhausted/)
})
it('aborts mid-replay when the signal is already set', async () => {
@@ -283,7 +317,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
installLlmReplay(ctx, { file })
const controller = new AbortController()
controller.abort()
await expect(drain(ctx.llm.stream({ model: 'm', messages: [], signal: controller.signal })))
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal })))
.rejects.toThrow('aborted')
})
@@ -305,11 +339,11 @@ describe('installLlmReplay (through the real waterfall)', () => {
}, { inject: ['llm'] }))
// While installed, replay short-circuits to the derived fixture ('hi').
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
await fiber.dispose()
// After dispose the listener is gone; the call reaches the real adapter.
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] })))
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })))
.toEqual([{ type: 'finish', reason: { kind: 'stop' } }])
})
@@ -321,7 +355,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
const ctx = new Context()
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file, overrideFile })
await expect(drain(ctx.llm.stream({ model: 'm', messages: [] })))
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })))
.rejects.toThrow(/llm-replay replay entry/)
})
@@ -333,7 +367,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file, overrideFile })
const controller = new AbortController()
const iterator = ctx.llm.stream({ model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
const iterator = ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
// Consume the two pre-hang chunks, then start the third pull so the generator
// is parked inside the await (signal NOT yet aborted — exercises the
// addEventListener('abort') registration), and only THEN abort.
@@ -359,7 +393,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
controller.abort()
// Already aborted: the throw-entry's prefix loop surfaces 'aborted' before
// it can reach the recorded LlmError.
await expect(drain(ctx.llm.stream({ model: 'm', messages: [], signal: controller.signal })))
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal })))
.rejects.toThrow('aborted')
})
@@ -373,7 +407,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
const controller = new AbortController()
controller.abort()
// The two pre-hang chunks still flow; the abort surfaces at the await.
const iterator = ctx.llm.stream({ model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
const iterator = ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
await iterator.next()
await iterator.next()
await expect(iterator.next()).rejects.toThrow('aborted')
@@ -503,7 +537,7 @@ describe('installLlmReplay (per-session keying)', () => {
]
const live = (id: string): GenerateOptions =>
({ model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
({ provider: 'm', model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
it('routes each live session to its own script by FIRST-CALL order', async () => {
const parentFile = writeSession('session.jsonl', { id: 'rec-parent', createdAt: 100 }, [TEXT_CHUNKS])
@@ -540,7 +574,7 @@ describe('installLlmReplay (per-session keying)', () => {
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file: parentFile })
// No sessionId at all — the legacy single-session path.
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
})
it('fails loud when more distinct live sessions call than were recorded', async () => {
@@ -574,12 +608,13 @@ describe('apply (the plugin entry)', () => {
expect(inject).toEqual(['llm'])
})
it('installs replay from an explicit config.file', async () => {
it('installs replay and its catalog from explicit config', async () => {
writeFileSync(file, sessionJsonl(TEXT_CHUNKS.map((c, i) => chunkEvent(i + 1, 1, 1, c))), 'utf8')
const ctx = new Context()
await ctx.plugin(LlmService)
apply(ctx, { file })
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
apply(ctx, { file, providers: [{ id: 'm', models: [{ id: 'm' }] }] })
expect(ctx.llm.listProviders()).toEqual([{ id: 'm', name: 'm' }])
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
})
it('falls back to $DSH_SNAPSHOT_FILE / $DSH_SNAPSHOT_OVERRIDE when config is empty', async () => {
@@ -591,7 +626,7 @@ describe('apply (the plugin entry)', () => {
const ctx = new Context()
await ctx.plugin(LlmService)
apply(ctx)
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
})
it('uses only the file when no override path is configured or in the env', async () => {
@@ -601,7 +636,7 @@ describe('apply (the plugin entry)', () => {
const ctx = new Context()
await ctx.plugin(LlmService)
apply(ctx)
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
})
it('throws when no fixture path is given by config or env', async () => {
@@ -631,7 +666,7 @@ describe('apply (the plugin entry)', () => {
await ctx.plugin(LlmService)
apply(ctx, { file, childFiles: [childFile] })
const live = (id: string): GenerateOptions =>
({ model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
({ provider: 'm', model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
expect(await drain(ctx.llm.stream(live('A')))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream(live('B')))).toEqual(childSecond)
})
@@ -651,7 +686,7 @@ describe('apply (the plugin entry)', () => {
await ctx.plugin(LlmService)
apply(ctx)
const live = (id: string): GenerateOptions =>
({ model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
({ provider: 'm', model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
expect(await drain(ctx.llm.stream(live('A')))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream(live('B')))).toEqual(childChunks)
})
@@ -663,6 +698,6 @@ describe('apply (the plugin entry)', () => {
const ctx = new Context()
await ctx.plugin(LlmService)
apply(ctx)
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
})
})