Merge remote-tracking branch 'origin/master' into codex/simp-prune-llm-contract
# Conflicts: # docs/cordis-catalog/services.md # packages/llm/llm-pi-ai/README.md # packages/support/llm-replay/tests/llm-replay.spec.ts
This commit is contained in:
Tianyi Cui
@@ -5,9 +5,10 @@ Packages that exist to serve development, testing, and the examples rather than
|
||||
| Package | Role | ctx key |
|
||||
|---|---|---|
|
||||
| `acp-snapshot/` | ACP snapshot suite kit: subprocess scenario harness + golden normalizers + the `defineAcpSnapshotSuite` factory | (library — imported by example `*.snapshot.ts` suites) |
|
||||
| `agent-loop-testkit/` | Shared prerequisite mounting for tests that exercise the concrete agent loop | (library — imported by AgentLoop integration tests) |
|
||||
| `invariants/` | Runtime event-contract assertions for development diagnostics | (listens on `session/*`, `agent/*`) |
|
||||
| `loader-smoke/` | Shared real-Loader subprocess harness for keyless example smokes | (library — imported by example e2e suites) |
|
||||
| `llm-replay/` | Record/replay adapter: short-circuits `llm/stream` from a recorded session JSONL (keyless snapshot tests) | (listens on `llm/stream`) |
|
||||
| `subagent-mock/` | Scripted `SubagentProvider` for deterministic seam/tool tests | (registers on `ctx.subagents`) |
|
||||
|
||||
`invariants` is development support but has no environment guard: it runs wherever registered, and the default `dsh-agent-spine-demo` bundle mounts it unconditionally. `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate. `acp-snapshot` carries the snapshot tier's harness/normalizer/suite machinery, while `loader-smoke` owns the parallel stdio/Loader process boundary used by keyless example e2e suites. `subagent-mock` exercises the real `ctx.subagents` load path without a model or child agent. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.
|
||||
`invariants` is development support but has no environment guard: it runs wherever registered, and the default `dsh-agent-spine-demo` bundle mounts it unconditionally. `agent-loop-testkit` centralizes the mandatory service spine for hand-built AgentLoop tests without owning their loop or scenario. `llm-replay` backs the demos and the snapshot test tier under the per-file coverage gate. `acp-snapshot` carries the snapshot tier's harness/normalizer/suite machinery, while `loader-smoke` owns the parallel stdio/Loader process boundary used by keyless example e2e suites. `subagent-mock` exercises the real `ctx.subagents` load path without a model or child agent. A package graduates OUT of `support/` into a product group only when it gains documented product consumers.
|
||||
|
||||
@@ -4,7 +4,7 @@ The ACP snapshot suite kit: the shared machinery behind the keyless snapshot tie
|
||||
|
||||
Three layers, importable separately:
|
||||
|
||||
- **`runScenario` (harness)** — boots the real agent bin as a subprocess via tsx (unbuilt, Loader path), drives it over ACP JSON-RPC stdio from a deterministic `input.json` script, tees raw stdout for the golden + purity check, and harvests every persisted session JSONL (parent + subagent children, primary-first) after a graceful stdin-EOF shutdown. Parameterized by `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath` — absolute paths; the subprocess cwd is a temp dir outside the repo).
|
||||
- **`runScenario` (harness)** — boots the real agent bin as a subprocess via tsx (unbuilt, Loader path), drives it over ACP JSON-RPC stdio from a deterministic `input.json` script, tees raw stdout for the golden + purity check, and harvests every persisted session JSONL (parent + subagent children, primary-first) after a graceful stdin-EOF shutdown. Parameterized by `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath` — absolute paths; the subprocess cwd is a temp dir outside the repo). Startup failures preserve captured agent stderr in the rejected diagnostic.
|
||||
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept), `scrubSystemPrompts` (prompt text → `{{system}}`), `scrubToolSchemas` (schema bulk → `{{tools}}`), and `scrubRequestHeaders` (all header bulk → `{{system}}`/`{{tools}}`/`{{messagePrefix}}` outside each pin, structure kept — [pinned-header RFC](../../../docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
|
||||
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record/refresh fixture write-back, rejection of structured `UNKNOWN_TOOL` results, the per-header-class pin (`system-prompt.golden.md` plus `tool-schemas.golden.json`) with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, every JSONL prompt/schema-scrubbed, non-pinning fixtures fully header-scrubbed). Must be called at vitest collection time.
|
||||
|
||||
@@ -35,7 +35,7 @@ defineAcpSnapshotSuite({
|
||||
})
|
||||
```
|
||||
|
||||
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode and filesystem scenarios are templates. Each pinning directory stores the normalized composed prompt in generated `system-prompt.golden.md` and the initial schemas plus schema deltas in generated `tool-schemas.golden.json`; `session.jsonl` stores `"system":"{{system}}","tools":"{{tools}}"` while retaining config, reason, and any model-visible prefix.
|
||||
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode and filesystem scenarios are templates. Each pinning directory stores the normalized full prompt sequence in generated `system-prompt.golden.md` and the corresponding full tool-schema sequence in generated `tool-schemas.golden.json`; `session.jsonl` stores `"system":"{{system}}","tools":"{{tools}}"` while retaining config, reason, and any model-visible prefix. A pin with legitimate mid-run header changes declares `expectedHeaderChanges`, which fixes the length of both sidecar sequences.
|
||||
|
||||
Examples use a `cordis.snapshot.yml` overlay with [`dsh-llm-replay`](../llm-replay/README.md). Recording calls the live model and updates model fixtures; keyless refresh replays those fixtures and updates derived stdout, session-log, prompt, and tool-schema snapshots. See the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
|
||||
|
||||
|
||||
@@ -168,6 +168,9 @@ export interface RunOptions {
|
||||
export async function runScenario(input: InputScript, opts: RunOptions): Promise<RunResult> {
|
||||
const cwd = await mkdtemp(join(tmpdir(), 'acp-snap-cwd-'))
|
||||
const sessionsRoot = await mkdtemp(join(tmpdir(), 'acp-snap-sessions-'))
|
||||
// Fixed path length: spill-policy budgets the preview against the REAL path
|
||||
// before stdout normalization, so tmpdir() length differences churn goldens.
|
||||
const spillRoot = '/tmp/dsh-acp-snapshot-spill'
|
||||
// Everything past the temp-dir creation runs under a try/finally that always
|
||||
// removes both dirs — so a failure in workspace seeding, spawn, or any step
|
||||
// never leaks them (the "e2e tests own their resources" rule).
|
||||
@@ -187,6 +190,7 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
|
||||
DSH_SNAPSHOT: opts.mode,
|
||||
DSH_SNAPSHOT_FILE: opts.fixtureFile,
|
||||
DSH_SNAPSHOT_SESSIONS_ROOT: sessionsRoot,
|
||||
DSH_SNAPSHOT_SPILL_ROOT: spillRoot,
|
||||
DSH_HOME: join(cwd, '.dsh'),
|
||||
DSH_AGENTS_HOME: join(cwd, '.agents'),
|
||||
...opts.overrideFile !== undefined ? { DSH_SNAPSHOT_OVERRIDE: opts.overrideFile } : {},
|
||||
@@ -281,6 +285,10 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
|
||||
// Harvest EVERY persisted log (parent + any subagent children) while the
|
||||
// temp dirs still exist, ordered primary-first.
|
||||
sessionLogs = await harvestSessionLogs(sessionsRoot)
|
||||
} catch (error: unknown) {
|
||||
const stderr = stderrChunks.join('')
|
||||
if (stderr === '') throw error
|
||||
throw new Error(`snapshot-harness: scenario failed: ${String(error)}\nagent stderr:\n${stderr}`, { cause: error })
|
||||
} finally {
|
||||
// Failure-safe teardown: kill a still-running child and drop the temp dirs
|
||||
// even if seeding/spawn/a step/harvest threw, so a flaky run never leaks a
|
||||
@@ -291,6 +299,7 @@ export async function runScenario(input: InputScript, opts: RunOptions): Promise
|
||||
}
|
||||
await rm(cwd, { recursive: true, force: true })
|
||||
await rm(sessionsRoot, { recursive: true, force: true })
|
||||
await rm(spillRoot, { recursive: true, force: true })
|
||||
}
|
||||
|
||||
return {
|
||||
|
||||
@@ -14,6 +14,16 @@ const MESSAGE_PREFIX = '{{messagePrefix}}'
|
||||
|
||||
/** A UUID v4 string, the shape `randomUUID()` produces for session ids. */
|
||||
const UUID_RE = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi
|
||||
const LOCAL_SPILL_PATH_RE = new RegExp(
|
||||
String.raw`\{\{cwd\}\}/\.spill/session-[0-9a-f]{12}/[0-9a-f]{12}-([A-Za-z0-9._~-]+?)`
|
||||
+ String.raw`(?=\. Use read with offset/limit|[\s)]|$)`,
|
||||
'g',
|
||||
)
|
||||
const SNAPSHOT_SPILL_PATH_RE = new RegExp(
|
||||
String.raw`/tmp/dsh-acp-snapshot-spill/session-[0-9a-f]{12}/[0-9a-f]{12}-([A-Za-z0-9._~-]+?)`
|
||||
+ String.raw`(?=\. Use read with offset/limit|[\s)]|$)`,
|
||||
'g',
|
||||
)
|
||||
|
||||
/** Inputs the normalizers need to recognize a run's volatile values. */
|
||||
export interface NormalizeContext {
|
||||
@@ -29,6 +39,9 @@ function scrubString(value: string, ctx: NormalizeContext): string {
|
||||
// cwd first (longest, most specific), then explicit session ids, then any
|
||||
// residual UUID (covers ids that appear in places we didn't enumerate).
|
||||
out = out.split(ctx.cwd).join(CWD)
|
||||
out = out.split(`/private${CWD}`).join(CWD)
|
||||
out = out.replace(LOCAL_SPILL_PATH_RE, (_match, name: string) => `{{spillLocator:${name}}}`)
|
||||
out = out.replace(SNAPSHOT_SPILL_PATH_RE, (_match, name: string) => `{{spillLocator:${name}}}`)
|
||||
for (const id of ctx.sessionIds) out = out.split(id).join(SESSION_ID)
|
||||
out = out.replace(UUID_RE, SESSION_ID)
|
||||
return out
|
||||
@@ -112,8 +125,8 @@ export function normalizeSessionLog(rawLog: string, ctx: NormalizeContext): stri
|
||||
}
|
||||
|
||||
/**
|
||||
* Replace system-prompt content in request headers and header deltas with
|
||||
* `{{system}}` tokens while retaining field presence and delta structure.
|
||||
* Replace system-prompt content in request headers with `{{system}}` tokens
|
||||
* while retaining field presence.
|
||||
* Other header content stays verbatim, so a header-pinning fixture can keep
|
||||
* its complete tool schemas while every JSONL fixture omits the prompt text.
|
||||
* Lines without a system payload pass through byte-for-byte; the transform is
|
||||
@@ -127,11 +140,11 @@ export function scrubSystemPrompts(rawLog: string): string {
|
||||
}
|
||||
|
||||
/**
|
||||
* Replace tool schemas in request headers and header deltas with `{{tools}}`
|
||||
* tokens while retaining field presence, tool names, and delta structure.
|
||||
* System prompts and session-prefix messages stay verbatim so pinning fixtures
|
||||
* can move only schema bulk into their dedicated JSON sidecar. Lines without a
|
||||
* tool payload pass through byte-for-byte; the transform is idempotent.
|
||||
* Replace tool schemas in full request-header snapshots with `{{tools}}`
|
||||
* tokens while retaining field presence. System prompts and session-prefix
|
||||
* messages stay verbatim so pinning fixtures can move only schema bulk into
|
||||
* their dedicated JSON sidecar. Lines without a tool payload pass through
|
||||
* byte-for-byte; the transform is idempotent.
|
||||
*
|
||||
* @param rawLog The raw session `.jsonl` content.
|
||||
* @returns The JSONL with tool-schema content tokenized.
|
||||
@@ -144,9 +157,9 @@ export function scrubToolSchemas(rawLog: string): string {
|
||||
* Replace all bulky request-header content in a session JSONL with stable
|
||||
* tokens. This includes the system-prompt fields handled by
|
||||
* {@link scrubSystemPrompts}, tool schemas, and session-prefix messages. It
|
||||
* keeps system-delta line positions and arity, tool-delta names, prefix
|
||||
* message counts, field presence, config, and reason. Lines without content
|
||||
* to scrub pass through byte-for-byte, and the transform is idempotent.
|
||||
* keeps prefix message counts, field presence, config, and reason. Lines
|
||||
* without content to scrub pass through byte-for-byte, and the transform is
|
||||
* idempotent.
|
||||
*
|
||||
* @param rawLog The raw session `.jsonl` content.
|
||||
* @returns The JSONL with all header bulk tokenized, other lines byte-identical.
|
||||
@@ -182,33 +195,7 @@ function scrubHeaderContent(rawLog: string, options: HeaderScrubOptions): string
|
||||
}
|
||||
return touched ? JSON.stringify(record) : line
|
||||
}
|
||||
if (record.type === 'request/header-delta') {
|
||||
let touched = false
|
||||
const system = data.system as Record<string, unknown> | null | undefined
|
||||
if (options.system === true && system !== null && typeof system === 'object' && Array.isArray(system.insert)) {
|
||||
system.insert = system.insert.map(() => SYSTEM)
|
||||
touched = true
|
||||
}
|
||||
const tools = data.tools as Record<string, unknown> | null | undefined
|
||||
if (options.tools === true && tools !== null && typeof tools === 'object') {
|
||||
if (Array.isArray(tools.added)) { tools.added = tools.added.map(scrubToolSchema); touched = true }
|
||||
if (Array.isArray(tools.changed)) { tools.changed = tools.changed.map(scrubToolSchema); touched = true }
|
||||
}
|
||||
if (options.prefix === true && Array.isArray(data.messagePrefix)) {
|
||||
data.messagePrefix = data.messagePrefix.map(() => MESSAGE_PREFIX)
|
||||
touched = true
|
||||
}
|
||||
return touched ? JSON.stringify(record) : line
|
||||
}
|
||||
return line
|
||||
})
|
||||
return out.join('\n')
|
||||
}
|
||||
|
||||
/** Tokenize one tool schema's bulk (description, parameters, anything else), keeping its identifying `name`. */
|
||||
function scrubToolSchema(tool: unknown): unknown {
|
||||
if (tool === null || typeof tool !== 'object' || Array.isArray(tool)) return tool
|
||||
const out: Record<string, unknown> = {}
|
||||
for (const [k, v] of Object.entries(tool)) out[k] = k === 'name' ? v : TOOLS
|
||||
return out
|
||||
}
|
||||
|
||||
@@ -1,12 +1,17 @@
|
||||
/**
|
||||
* Keyless-by-default ACP snapshot suite factory. Each scenario drives the real subprocess and
|
||||
* compares normalized stdout; comparable session fixtures are both replay input and expected
|
||||
* output. Record mode refreshes reproducible model scenarios from the live API, while refresh
|
||||
* mode replays committed scripts and rewrites derived artifacts without a key.
|
||||
* Keyless-by-default ACP snapshot suite factory. Each scenario drives the real
|
||||
* subprocess and compares normalized stdout; comparable session fixtures are
|
||||
* both replay input and expected output. Record mode refreshes reproducible
|
||||
* model scenarios from the live API, while refresh mode replays committed
|
||||
* scripts and rewrites derived artifacts without a key.
|
||||
* Replay scenarios run concurrently because each subprocess owns unique temp
|
||||
* cwd and persistence roots and reads only committed fixtures. Record and
|
||||
* refresh stay serial while writing.
|
||||
*
|
||||
* Exactly one scenario per header-composition class pins the system prompt and tool schemas in
|
||||
* dedicated sidecars. Every live header is checked against that pin, so session-dependent
|
||||
* composition must declare a separate class instead of escaping coverage.
|
||||
* Exactly one scenario per header-composition class pins the full prompt and
|
||||
* tool-schema sequences in dedicated sidecars. Every live header is checked
|
||||
* against that pin, so session-dependent composition must declare a separate
|
||||
* class instead of escaping coverage.
|
||||
* @module @deepseek-ai/dsh-acp-snapshot/suite
|
||||
*/
|
||||
|
||||
@@ -80,14 +85,11 @@ export interface Scenario {
|
||||
*/
|
||||
pinsHeader?: boolean
|
||||
/**
|
||||
* How many `request/header-delta` events this PINNING scenario's fixture
|
||||
* legitimately carries (default 0). A recorded mid-run header change — a
|
||||
* config-option switch rewriting a prompt section — is part of the pinned
|
||||
* surface, with readable prompt text in Markdown; any OTHER count
|
||||
* still fails, so fixture rot stays caught. Meaningless off the pin (the
|
||||
* live uniformity guard keeps non-pinning scenarios delta-free).
|
||||
* How many changed `request/header` snapshots this PINNING scenario's primary
|
||||
* fixture legitimately carries (default 0). Their full prompt text is kept in
|
||||
* the readable Markdown pin; any other count fails. Meaningless off the pin.
|
||||
*/
|
||||
expectedHeaderDeltas?: number
|
||||
expectedHeaderChanges?: number
|
||||
/**
|
||||
* Which header-composition class this scenario belongs to. Scenarios that
|
||||
* boot the same config compose the same header; each class has exactly one
|
||||
@@ -208,114 +210,58 @@ export function normalizedToolSchemas(rawLog: string, ctx: NormalizeContext): un
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract normalized tool-schema edits from request-header deltas in log order.
|
||||
* Deltas without an object-valued tools edit are omitted; their remaining
|
||||
* structure stays pinned in the session JSONL.
|
||||
*
|
||||
* @param rawLog The session `.jsonl` content to inspect.
|
||||
* @param ctx The volatile values of the run that produced it.
|
||||
* @returns The normalized tool-schema edits, in event order.
|
||||
*/
|
||||
export function normalizedToolSchemaDeltas(rawLog: string, ctx: NormalizeContext): unknown[] {
|
||||
return normalizeSessionLog(rawLog, ctx)
|
||||
.split('\n')
|
||||
.filter(line => line.trim().length > 0)
|
||||
.map(line => JSON.parse(line) as { type?: unknown; data?: { tools?: unknown } })
|
||||
.filter(record => record.type === 'request/header-delta')
|
||||
.flatMap((record) => {
|
||||
const tools = record.data?.tools
|
||||
return tools !== null && typeof tools === 'object' && !Array.isArray(tools) ? [tools] : []
|
||||
})
|
||||
}
|
||||
|
||||
/** The structured contents of a tool-schema sidecar. */
|
||||
export interface ToolSchemasSnapshot {
|
||||
/** The complete tool schemas from the pinned request header. */
|
||||
initial: unknown[]
|
||||
/** Complete tool-schema edits from subsequent request-header deltas. */
|
||||
deltas: unknown[]
|
||||
/** Complete tool schemas from subsequent changed-header snapshots. */
|
||||
changes: unknown[][]
|
||||
}
|
||||
|
||||
/**
|
||||
* Render tool schemas and later schema edits as canonical, readable JSON.
|
||||
* Render the full tool-schema sequence as canonical, readable JSON.
|
||||
*
|
||||
* @param initial The pinned request header's complete tool schemas.
|
||||
* @param deltas Complete tool-schema edits from request-header deltas.
|
||||
* @param changes Complete tool schemas from later changed headers.
|
||||
* @returns A pretty-printed JSON snapshot ending in one newline.
|
||||
*/
|
||||
export function formatToolSchemasSnapshot(initial: readonly unknown[], deltas: readonly unknown[] = []): string {
|
||||
return `${JSON.stringify({ initial, deltas }, null, 2)}\n`
|
||||
export function formatToolSchemasSnapshot(initial: readonly unknown[], changes: readonly unknown[][] = []): string {
|
||||
return `${JSON.stringify({ initial, changes }, null, 2)}\n`
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse and validate the stable top-level shape of a tool-schema sidecar.
|
||||
*
|
||||
* @param snapshot The JSON sidecar text.
|
||||
* @returns Its initial schemas and schema deltas.
|
||||
* @returns Its initial and changed-header schema sets.
|
||||
*/
|
||||
export function parseToolSchemasSnapshot(snapshot: string): ToolSchemasSnapshot {
|
||||
const parsed = JSON.parse(snapshot) as unknown
|
||||
if (parsed === null || typeof parsed !== 'object' || Array.isArray(parsed)) {
|
||||
throw new Error('acp-snapshot: tool-schema snapshot must be an object')
|
||||
}
|
||||
const { initial, deltas } = parsed as { initial?: unknown; deltas?: unknown }
|
||||
if (!Array.isArray(initial) || !Array.isArray(deltas)) {
|
||||
throw new Error('acp-snapshot: tool-schema snapshot must carry array-valued initial and deltas fields')
|
||||
const { initial, changes } = parsed as { initial?: unknown; changes?: unknown }
|
||||
if (!Array.isArray(initial) || !Array.isArray(changes) || !changes.every(Array.isArray)) {
|
||||
throw new Error('acp-snapshot: tool-schema snapshot must carry array-valued initial and changes fields')
|
||||
}
|
||||
return { initial, deltas }
|
||||
return { initial, changes }
|
||||
}
|
||||
|
||||
/**
|
||||
* Restore a sidecar's initial schemas into a tokenized pinned header.
|
||||
* Restore one sidecar schema set into a tokenized pinned header.
|
||||
*
|
||||
* @param header The parsed request header carrying `tools: "{{tools}}"`.
|
||||
* @param snapshot The parsed tool-schema sidecar.
|
||||
* @returns A copy of the header with its complete initial schemas restored.
|
||||
* @param schemas The complete schemas for this full header snapshot.
|
||||
* @returns A copy of the header with its complete schemas restored.
|
||||
*/
|
||||
export function restorePinnedToolSchemas(header: unknown, snapshot: ToolSchemasSnapshot): unknown {
|
||||
export function restorePinnedToolSchemas(header: unknown, schemas: readonly unknown[]): unknown {
|
||||
if (header === null || typeof header !== 'object' || Array.isArray(header)) {
|
||||
throw new Error('acp-snapshot: pinned request header must be an object')
|
||||
}
|
||||
if ((header as { tools?: unknown }).tools !== TOOLS_TOKEN) {
|
||||
throw new Error(`acp-snapshot: pinned request header tools must equal ${TOOLS_TOKEN}`)
|
||||
}
|
||||
return { ...header, tools: snapshot.initial }
|
||||
}
|
||||
|
||||
/** One normalized system-prompt edit carried by a `request/header-delta`. */
|
||||
export interface SystemPromptDeltaSnapshot {
|
||||
/** How many leading lines remain from the prior prompt. */
|
||||
keepStart: number
|
||||
/** How many trailing lines remain from the prior prompt. */
|
||||
keepEnd: number
|
||||
/** The normalized replacement lines inserted between the retained ranges. */
|
||||
insert: string[]
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract normalized system-prompt edits from request-header deltas in log
|
||||
* order. Deltas without a well-formed system edit are omitted; their non-prompt
|
||||
* structure remains pinned in JSONL.
|
||||
*
|
||||
* @param rawLog The session `.jsonl` content to inspect.
|
||||
* @param ctx The volatile values of the run that produced it.
|
||||
* @returns The normalized system-prompt edits, in event order.
|
||||
*/
|
||||
export function normalizedSystemPromptDeltas(rawLog: string, ctx: NormalizeContext): SystemPromptDeltaSnapshot[] {
|
||||
return normalizeSessionLog(rawLog, ctx)
|
||||
.split('\n')
|
||||
.filter(line => line.trim().length > 0)
|
||||
.map(line => JSON.parse(line) as { type?: unknown; data?: { system?: unknown } })
|
||||
.filter(record => record.type === 'request/header-delta')
|
||||
.flatMap((record) => {
|
||||
const system = record.data?.system
|
||||
if (system === null || typeof system !== 'object') return []
|
||||
const { keepStart, keepEnd, insert } = system as { keepStart?: unknown; keepEnd?: unknown; insert?: unknown }
|
||||
if (typeof keepStart !== 'number' || typeof keepEnd !== 'number' || !Array.isArray(insert)) return []
|
||||
if (!insert.every(line => typeof line === 'string')) return []
|
||||
return [{ keepStart, keepEnd, insert: insert }]
|
||||
})
|
||||
return { ...header, tools: schemas }
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -324,38 +270,40 @@ export function normalizedSystemPromptDeltas(rawLog: string, ctx: NormalizeConte
|
||||
* the committed file follows the repository newline contract.
|
||||
*
|
||||
* @param prompt The normalized system prompt.
|
||||
* @param deltas Normalized prompt edits to append as readable sections.
|
||||
* @param changes Full normalized prompts from later changed-header snapshots.
|
||||
* @returns Markdown snapshot text ending in a newline.
|
||||
*/
|
||||
export function formatSystemPromptSnapshot(
|
||||
prompt: string,
|
||||
deltas: readonly SystemPromptDeltaSnapshot[] = [],
|
||||
changes: readonly string[] = [],
|
||||
): string {
|
||||
let snapshot = prompt.endsWith('\n') ? prompt : `${prompt}\n`
|
||||
for (const [index, delta] of deltas.entries()) {
|
||||
snapshot += `\n<!-- request/header-delta ${index + 1}: keepStart=${delta.keepStart}, keepEnd=${delta.keepEnd} -->\n\n`
|
||||
const insert = delta.insert.join('\n')
|
||||
snapshot += insert.endsWith('\n') ? insert : `${insert}\n`
|
||||
for (const [index, change] of changes.entries()) {
|
||||
snapshot += `\n<!-- request/header change ${index + 1} -->\n\n`
|
||||
snapshot += change.endsWith('\n') ? change : `${change}\n`
|
||||
}
|
||||
return snapshot
|
||||
}
|
||||
|
||||
/** Return the initial-prompt portion of a possibly delta-bearing snapshot. */
|
||||
/** Return the initial-prompt portion of a possibly multi-header snapshot. */
|
||||
function initialSystemPromptSnapshot(snapshot: string): string {
|
||||
const marker = snapshot.indexOf('\n<!-- request/header-delta ')
|
||||
const marker = snapshot.indexOf('\n<!-- request/header change ')
|
||||
return marker < 0 ? snapshot : snapshot.slice(0, marker)
|
||||
}
|
||||
|
||||
/**
|
||||
* Count the `request/header-delta` events in a session JSONL.
|
||||
* Count changed `request/header` snapshots in a session JSONL.
|
||||
*
|
||||
* @param rawLog The session `.jsonl` content.
|
||||
* @returns How many `request/header-delta` events the log carries.
|
||||
* @returns How many headers carry reason `change`.
|
||||
*/
|
||||
export function headerDeltaCount(rawLog: string): number {
|
||||
export function headerChangeCount(rawLog: string): number {
|
||||
return rawLog.split('\n')
|
||||
.filter(line => line.trim().length > 0)
|
||||
.filter(line => (JSON.parse(line) as { type?: unknown }).type === 'request/header-delta')
|
||||
.filter((line) => {
|
||||
const record = JSON.parse(line) as { type?: unknown; data?: { reason?: unknown } }
|
||||
return record.type === 'request/header' && record.data?.reason === 'change'
|
||||
})
|
||||
.length
|
||||
}
|
||||
|
||||
@@ -461,7 +409,7 @@ export function stabilizeRefreshLog(fresh: string, existing: string, replacement
|
||||
}
|
||||
|
||||
/**
|
||||
* Register the suite: one `describe` per scenario (the golden/log compares and
|
||||
* Register the suite: one test per scenario (the golden/log compares and
|
||||
* the header-uniformity guard) plus the fixture guard block (no orphan
|
||||
* scenario dirs, required files present, exactly one pin per header class,
|
||||
* pinning fixtures well-formed, every JSONL prompt-scrubbed, non-pinning
|
||||
@@ -477,6 +425,7 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
const RECORDING = mode === 'record'
|
||||
const REFRESHING = mode === 'refresh'
|
||||
const childMode: 'replay' | 'record' = RECORDING ? 'record' : 'replay'
|
||||
const scenarioSuite = mode === 'replay' ? describe.concurrent : describe
|
||||
|
||||
/** The class a scenario's header composition belongs to (see {@link Scenario.headerClass}). */
|
||||
const classOf = (scenario: Scenario): string => scenario.headerClass ?? 'default'
|
||||
@@ -496,11 +445,11 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
}
|
||||
}
|
||||
|
||||
for (const scenario of scenarios) {
|
||||
describe(`snapshot: ${scenario.name}`, () => {
|
||||
scenarioSuite('snapshot scenarios', () => {
|
||||
for (const scenario of scenarios) {
|
||||
// In RECORD mode, only re-run the `recorded` (live-API) scenarios; the `authored` ones
|
||||
// (sidecar-driven errors/cancel) are never re-recorded.
|
||||
it.skipIf(RECORDING && !scenario.recorded)('matches the goldens', async () => {
|
||||
it.skipIf(RECORDING && !scenario.recorded)(`snapshot: ${scenario.name} matches the goldens`, async ({ expect }) => {
|
||||
const dir = join(snapshotsDir, scenario.name)
|
||||
const input = JSON.parse(await readFile(join(dir, 'input.json'), 'utf8')) as InputScript
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
@@ -564,30 +513,19 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
))
|
||||
}
|
||||
if (scenario.pinsHeader === true) {
|
||||
const prompts = result.sessionLogs.flatMap(log => normalizedSystemPrompts(log.content, ctx))
|
||||
expect(prompts.length, `${mode} produced no system prompt to snapshot`).toBeGreaterThan(0)
|
||||
const initialSnapshot = formatSystemPromptSnapshot(prompts[0] as string)
|
||||
for (const prompt of prompts) {
|
||||
expect(formatSystemPromptSnapshot(prompt), 'the pinning run produced divergent system prompts')
|
||||
.toEqual(initialSnapshot)
|
||||
}
|
||||
const primary = result.sessionLogs[0] as HarvestedLog
|
||||
const snapshot = formatSystemPromptSnapshot(
|
||||
prompts[0] as string,
|
||||
normalizedSystemPromptDeltas(primary.content, ctx),
|
||||
)
|
||||
const prompts = normalizedSystemPrompts(primary.content, ctx)
|
||||
expect(prompts.length, `${mode} produced no system prompt to snapshot`).toBeGreaterThan(0)
|
||||
const snapshot = formatSystemPromptSnapshot(prompts[0] as string, prompts.slice(1))
|
||||
await writeFile(join(dir, SYSTEM_PROMPT_SNAPSHOT), snapshot)
|
||||
|
||||
const schemaSets = result.sessionLogs.flatMap(log => normalizedToolSchemas(log.content, ctx))
|
||||
const schemaSets = normalizedToolSchemas(primary.content, ctx)
|
||||
expect(schemaSets.length, `${mode} produced no tool schemas to snapshot`).toBeGreaterThan(0)
|
||||
const initialSchemaSnapshot = formatToolSchemasSnapshot(schemaSets[0] as unknown[])
|
||||
for (const schemas of schemaSets) {
|
||||
expect(formatToolSchemasSnapshot(schemas), 'the pinning run produced divergent tool schemas')
|
||||
.toEqual(initialSchemaSnapshot)
|
||||
}
|
||||
expect(schemaSets.length, `${mode} produced a tool-schema sequence that differs from its prompt sequence`)
|
||||
.toBe(prompts.length)
|
||||
await writeFile(join(dir, TOOL_SCHEMAS_SNAPSHOT), formatToolSchemasSnapshot(
|
||||
schemaSets[0] as unknown[],
|
||||
normalizedToolSchemaDeltas(primary.content, ctx),
|
||||
schemaSets.slice(1),
|
||||
))
|
||||
}
|
||||
}
|
||||
@@ -611,8 +549,8 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
}
|
||||
}
|
||||
|
||||
// Header-uniformity guard: every live header in a class must equal the class pin split
|
||||
// across tokenized JSONL plus readable prompt and structured schema sidecars.
|
||||
// Every live full header must equal its class pin reconstructed from
|
||||
// tokenized JSONL plus readable prompt and structured schema sidecars.
|
||||
/* v8 ignore next -- construction guarantees the pin exists; a miss would fail the one-header assertion loudly. */
|
||||
const pinningScenario = pinningByClass.get(classOf(scenario)) ?? scenario
|
||||
const pinningDir = join(snapshotsDir, pinningScenario.name)
|
||||
@@ -620,17 +558,23 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
const pinned = normalizedHeaders(pinnedFixture, fixtureContext(pinnedFixture))
|
||||
const promptSnapshot = await readFile(join(pinningDir, SYSTEM_PROMPT_SNAPSHOT), 'utf8')
|
||||
const initialPromptSnapshot = initialSystemPromptSnapshot(promptSnapshot)
|
||||
expect(pinned.length, `the pinning fixture (${pinningScenario.name}) has an unexpected request/header count`)
|
||||
.toBe(1 + (pinningScenario.expectedHeaderChanges ?? 0))
|
||||
const toolSchemasSnapshot = await readFile(join(pinningDir, TOOL_SCHEMAS_SNAPSHOT), 'utf8')
|
||||
const toolSchemas = parseToolSchemasSnapshot(toolSchemasSnapshot)
|
||||
expect(pinned.length, `the pinning fixture (${pinningScenario.name}) must carry exactly one request/header`)
|
||||
.toBe(1)
|
||||
const pinnedHeader = restorePinnedToolSchemas(pinned[0], toolSchemas)
|
||||
const pinnedSchemaSets = [toolSchemas.initial, ...toolSchemas.changes]
|
||||
expect(pinnedSchemaSets.length, `the pinning fixture (${pinningScenario.name}) has an unexpected tool-schema count`)
|
||||
.toBe(pinned.length)
|
||||
const pinnedHeaders = pinned.map((header, index) => restorePinnedToolSchemas(
|
||||
header,
|
||||
pinnedSchemaSets[index] as unknown[],
|
||||
))
|
||||
for (const [logIndex, log] of result.sessionLogs.entries()) {
|
||||
const expectedDeltas = scenario.pinsHeader === true && logIndex === 0
|
||||
? scenario.expectedHeaderDeltas ?? 0
|
||||
const expectedChanges = scenario.pinsHeader === true && logIndex === 0
|
||||
? scenario.expectedHeaderChanges ?? 0
|
||||
: 0
|
||||
expect(headerDeltaCount(log.content), `session ${log.id}: request/header-delta count`)
|
||||
.toBe(expectedDeltas)
|
||||
expect(headerChangeCount(log.content), `session ${log.id}: changed request/header count`)
|
||||
.toBe(expectedChanges)
|
||||
const headers = normalizedHeaders(scrubSystemPrompts(log.content), ctx)
|
||||
const prompts = normalizedSystemPrompts(log.content, ctx)
|
||||
const schemaSets = normalizedToolSchemas(log.content, ctx)
|
||||
@@ -639,27 +583,30 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
expect(schemaSets.length, `session ${log.id}: every request/header must carry an array-valued tools field`)
|
||||
.toBe(headers.length)
|
||||
for (const [k, header] of headers.entries()) {
|
||||
const expected = expectedChanges > 0 ? pinnedHeaders[k] : pinnedHeaders[0]
|
||||
expect(header, `session ${log.id}: request/header #${k + 1} diverged from the pinned (${pinningScenario.name}) header`)
|
||||
.toEqual(pinnedHeader)
|
||||
expect(formatSystemPromptSnapshot(prompts[k] as string), `session ${log.id}: initial system prompt #${k + 1} diverged from ${pinningScenario.name}/${SYSTEM_PROMPT_SNAPSHOT}`)
|
||||
.toEqual(initialPromptSnapshot)
|
||||
.toEqual(expected)
|
||||
if (expectedChanges === 0) {
|
||||
expect(formatSystemPromptSnapshot(prompts[k] as string), `session ${log.id}: initial system prompt #${k + 1} diverged from ${pinningScenario.name}/${SYSTEM_PROMPT_SNAPSHOT}`)
|
||||
.toEqual(initialPromptSnapshot)
|
||||
}
|
||||
}
|
||||
if (scenario.pinsHeader === true && logIndex === 0) {
|
||||
expect(formatSystemPromptSnapshot(
|
||||
prompts[0] as string,
|
||||
normalizedSystemPromptDeltas(log.content, ctx),
|
||||
), `session ${log.id}: system-prompt deltas diverged from ${pinningScenario.name}/${SYSTEM_PROMPT_SNAPSHOT}`)
|
||||
prompts.slice(1),
|
||||
), `session ${log.id}: changed system prompts diverged from ${pinningScenario.name}/${SYSTEM_PROMPT_SNAPSHOT}`)
|
||||
.toEqual(promptSnapshot)
|
||||
expect(formatToolSchemasSnapshot(
|
||||
schemaSets[0] as unknown[],
|
||||
normalizedToolSchemaDeltas(log.content, ctx),
|
||||
), `session ${log.id}: tool-schema deltas diverged from ${pinningScenario.name}/${TOOL_SCHEMAS_SNAPSHOT}`)
|
||||
schemaSets.slice(1),
|
||||
), `session ${log.id}: changed tool schemas diverged from ${pinningScenario.name}/${TOOL_SCHEMAS_SNAPSHOT}`)
|
||||
.toEqual(toolSchemasSnapshot)
|
||||
}
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
describe('snapshot fixtures', () => {
|
||||
it('every scenario directory is registered (no orphans)', async () => {
|
||||
@@ -708,25 +655,30 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
}
|
||||
})
|
||||
|
||||
it('every pinning fixture carries one tokenized request/header, two sidecars, and its declared deltas', async () => {
|
||||
// The live uniformity guard runs only in NON-pinning scenarios, so a class made of just
|
||||
// its pinning scenario would otherwise accept a re-recorded pin with several headers or
|
||||
// an undeclared mid-run header-delta — shapes the pin design cannot represent.
|
||||
it('every pinning fixture carries one tokenized header sequence and two sidecars', async () => {
|
||||
// Assert the committed pin directly because a class containing only its
|
||||
// pinning scenario has no non-pinning live run to catch undeclared changes.
|
||||
for (const scenario of pinningByClass.values()) {
|
||||
const fixture = await readFile(join(snapshotsDir, scenario.name, 'session.jsonl'), 'utf8')
|
||||
const headers = normalizedHeaders(fixture, fixtureContext(fixture))
|
||||
const promptSnapshot = await readFile(join(snapshotsDir, scenario.name, SYSTEM_PROMPT_SNAPSHOT), 'utf8')
|
||||
expect(headers.length, `${scenario.name}: unexpected request/header count`)
|
||||
.toBe(1 + (scenario.expectedHeaderChanges ?? 0))
|
||||
const toolSchemasSnapshot = await readFile(join(snapshotsDir, scenario.name, TOOL_SCHEMAS_SNAPSHOT), 'utf8')
|
||||
const toolSchemas = parseToolSchemasSnapshot(toolSchemasSnapshot)
|
||||
expect(headers.length, `${scenario.name}: a pinning fixture must carry exactly one request/header`).toBe(1)
|
||||
expect(() => restorePinnedToolSchemas(headers[0], toolSchemas), `${scenario.name}: tools must use the sidecar token`)
|
||||
.not.toThrow()
|
||||
const schemaSets = [toolSchemas.initial, ...toolSchemas.changes]
|
||||
expect(schemaSets.length, `${scenario.name}: tool-schema sequence must match the header sequence`)
|
||||
.toBe(headers.length)
|
||||
for (const [index, header] of headers.entries()) {
|
||||
expect(() => restorePinnedToolSchemas(header, schemaSets[index] as unknown[]), `${scenario.name}: tools must use the sidecar token`)
|
||||
.not.toThrow()
|
||||
}
|
||||
expect(promptSnapshot.length, `${scenario.name}/${SYSTEM_PROMPT_SNAPSHOT} must not be empty`).toBeGreaterThan(0)
|
||||
expect(promptSnapshot.endsWith('\n'), `${scenario.name}/${SYSTEM_PROMPT_SNAPSHOT} must end in a newline`).toBe(true)
|
||||
expect(toolSchemasSnapshot, `${scenario.name}/${TOOL_SCHEMAS_SNAPSHOT} must use canonical JSON formatting`)
|
||||
.toBe(formatToolSchemasSnapshot(toolSchemas.initial, toolSchemas.deltas))
|
||||
expect(headerDeltaCount(fixture), `${scenario.name}: a pinning fixture must carry exactly its declared request/header-deltas`)
|
||||
.toBe(scenario.expectedHeaderDeltas ?? 0)
|
||||
.toBe(formatToolSchemasSnapshot(toolSchemas.initial, toolSchemas.changes))
|
||||
expect(headerChangeCount(fixture), `${scenario.name}: a pinning fixture must carry exactly its declared changed headers`)
|
||||
.toBe(scenario.expectedHeaderChanges ?? 0)
|
||||
}
|
||||
})
|
||||
|
||||
|
||||
@@ -24,6 +24,8 @@ interface ScriptedLog {
|
||||
|
||||
/** The whole scripted behavior for one run. Every field defaults to the least surprising choice. */
|
||||
interface Behavior {
|
||||
/** Exit during startup after writing any configured stderr note. */
|
||||
failOnBoot?: boolean
|
||||
/** Reject every `session/new` (exercises the expect-error step without extra dirs). */
|
||||
rejectNewSession?: boolean
|
||||
/** Reject `session/new` only when `additionalDirectories` is non-empty (the real bridge's rule). */
|
||||
@@ -62,6 +64,7 @@ const behavior: Behavior = fixtureFile === ''
|
||||
: JSON.parse(readFileSync(join(dirname(fixtureFile), 'behavior.json'), 'utf8')) as Behavior
|
||||
|
||||
if (behavior.stderrNote !== undefined) process.stderr.write(`${behavior.stderrNote}\n`)
|
||||
if (behavior.failOnBoot === true) process.exit(7)
|
||||
|
||||
let nextOutboundId = 1000
|
||||
let sessionId = ''
|
||||
|
||||
@@ -8,5 +8,5 @@
|
||||
}
|
||||
}
|
||||
],
|
||||
"deltas": []
|
||||
"changes": []
|
||||
}
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
"lines": [
|
||||
{ "type": "session", "id": "{{SID}}", "createdAt": 100, "cwd": "{{CWD}}" },
|
||||
{ "type": "request/header", "seq": 0, "time": 100, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "initial" } },
|
||||
{ "type": "request/header-delta", "seq": 1, "time": 100, "data": { "system": { "keepStart": 1, "keepEnd": 0, "insert": ["NEW PROMPT LINE"] } } },
|
||||
{ "type": "request/header", "seq": 1, "time": 100, "data": { "header": { "config": { "model": "fake" }, "system": "SYS PROMPT\n\nNEW PROMPT LINE", "tools": [{ "name": "t1", "description": "D1", "parameters": { "type": "object" } }] }, "reason": "change" } },
|
||||
{ "type": "turn/start", "seq": 2, "time": 100, "data": { "turn": 1 } }
|
||||
]
|
||||
}]
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
{"type":"session","id":"12121212-3434-4545-8686-787878787878","createdAt":7,"cwd":"/rec/pin-cwd"}
|
||||
{"type":"request/header","seq":0,"time":7,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
||||
{"type":"request/header-delta","seq":1,"time":7,"data":{"system":{"keepStart":1,"keepEnd":0,"insert":["{{system}}"]}}}
|
||||
{"type":"request/header","seq":1,"time":7,"data":{"header":{"config":{"model":"fake"},"system":"{{system}}","tools":"{{tools}}"},"reason":"change"}}
|
||||
{"type":"turn/start","seq":2,"time":7,"data":{"turn":1}}
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
SYS PROMPT
|
||||
|
||||
<!-- request/header-delta 1: keepStart=1, keepEnd=0 -->
|
||||
<!-- request/header change 1 -->
|
||||
|
||||
SYS PROMPT
|
||||
|
||||
NEW PROMPT LINE
|
||||
|
||||
@@ -8,5 +8,15 @@
|
||||
}
|
||||
}
|
||||
],
|
||||
"deltas": []
|
||||
"changes": [
|
||||
[
|
||||
{
|
||||
"name": "t1",
|
||||
"description": "D1",
|
||||
"parameters": {
|
||||
"type": "object"
|
||||
}
|
||||
}
|
||||
]
|
||||
]
|
||||
}
|
||||
|
||||
@@ -38,6 +38,14 @@ async function scenario(behavior: object): Promise<{ dir: string; fixtureFile: s
|
||||
const boot: InputStep[] = [{ op: 'initialize' }, { op: 'newSession' }]
|
||||
|
||||
describe('runScenario', () => {
|
||||
it('includes agent stderr when the ACP connection closes during startup', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({ failOnBoot: true, stderrNote: 'fake agent requested startup failure' })
|
||||
await expect(runScenario(
|
||||
{ steps: [{ op: 'initialize' }] },
|
||||
{ agent: AGENT, mode: 'replay', fixtureFile },
|
||||
)).rejects.toThrow(/agent stderr:\nfake agent requested startup failure/)
|
||||
})
|
||||
|
||||
it('drives a full turn: initialize (terminal caps), session, prompt, permission stub, harvest', { timeout: 20_000 }, async () => {
|
||||
const { fixtureFile } = await scenario({
|
||||
permissionProbe: true,
|
||||
|
||||
@@ -93,6 +93,52 @@ describe('normalizeSessionLog', () => {
|
||||
expect(out).not.toContain(ctx.cwd)
|
||||
})
|
||||
|
||||
it('scrubs random local spill paths under the snapshot cwd', () => {
|
||||
const ev = JSON.stringify({
|
||||
type: 'tool/result', seq: 2, time: 5,
|
||||
data: {
|
||||
content: [{
|
||||
type: 'text',
|
||||
text: `Full formatted result stored at: ${ctx.cwd}/.spill/session-c22bc3f1d2af/8a7b6c5d4e3f-bash.txt. Use read with offset/limit, or grep this path to search within it.`,
|
||||
}],
|
||||
},
|
||||
})
|
||||
const out = normalizeSessionLog(`${header({ cwd: ctx.cwd })}\n${ev}\n`, ctx)
|
||||
expect(out).toContain('{{spillLocator:bash.txt}}')
|
||||
expect(out).not.toContain('session-c22bc3f1d2af')
|
||||
expect(out).not.toContain('8a7b6c5d4e3f')
|
||||
})
|
||||
|
||||
it('scrubs macOS /private aliases for local spill paths', () => {
|
||||
const ev = JSON.stringify({
|
||||
type: 'tool/result', seq: 2, time: 5,
|
||||
data: {
|
||||
content: [{
|
||||
type: 'text',
|
||||
text: `Full formatted result stored at: /private${ctx.cwd}/.spill/session-c22bc3f1d2af/8a7b6c5d4e3f-bash.txt. Use read with offset/limit, or grep this path to search within it.`,
|
||||
}],
|
||||
},
|
||||
})
|
||||
const out = normalizeSessionLog(`${header({ cwd: ctx.cwd })}\n${ev}\n`, ctx)
|
||||
expect(out).toContain('{{spillLocator:bash.txt}}')
|
||||
expect(out).not.toContain('/private{{spillLocator')
|
||||
})
|
||||
|
||||
it('scrubs fixed snapshot spill paths', () => {
|
||||
const ev = JSON.stringify({
|
||||
type: 'tool/result', seq: 2, time: 5,
|
||||
data: {
|
||||
content: [{
|
||||
type: 'text',
|
||||
text: 'Full formatted result stored at: /tmp/dsh-acp-snapshot-spill/session-c22bc3f1d2af/8a7b6c5d4e3f-bash.txt. Use read with offset/limit, or grep this path to search within it.',
|
||||
}],
|
||||
},
|
||||
})
|
||||
const out = normalizeSessionLog(`${header({ cwd: ctx.cwd })}\n${ev}\n`, ctx)
|
||||
expect(out).toContain('{{spillLocator:bash.txt}}')
|
||||
expect(out).not.toContain('/tmp/dsh-acp-snapshot-spill')
|
||||
})
|
||||
|
||||
it('scrubs the session id in the header', () => {
|
||||
const out = normalizeSessionLog(`${header({ id: ctx.sessionIds[0] })}\n`, ctx)
|
||||
expect(out).toContain('{{sessionId}}')
|
||||
@@ -181,89 +227,19 @@ describe('scrubRequestHeaders', () => {
|
||||
expect(scrubRequestHeaders(`${headerLine}\n${odd}\n`)).toContain('"messagePrefix":"weird"')
|
||||
})
|
||||
|
||||
it('scrubs a header-delta prefix replacement to one token per message', () => {
|
||||
const delta = JSON.stringify({
|
||||
type: 'request/header-delta', seq: 8, time: 9,
|
||||
data: { messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'leaked opener' }] }] },
|
||||
})
|
||||
const out = scrubRequestHeaders(`${headerLine}\n${delta}\n`)
|
||||
expect(out).toContain('"messagePrefix":["{{messagePrefix}}"]')
|
||||
expect(out).not.toContain('leaked opener')
|
||||
// The empty-array transition-to-absence stays a structural fact.
|
||||
const toNone = JSON.stringify({ type: 'request/header-delta', seq: 9, time: 9, data: { messagePrefix: [] } })
|
||||
expect(scrubRequestHeaders(`${headerLine}\n${toNone}\n`)).toContain('"messagePrefix":[]')
|
||||
})
|
||||
|
||||
it('leaves a delta with no scrubbable payload byte-identical (config-only, or non-array shapes)', () => {
|
||||
const configOnly = JSON.stringify({ type: 'request/header-delta', seq: 8, time: 9, data: { config: { model: 'm2' } } })
|
||||
const oddShapes = JSON.stringify({ type: 'request/header-delta', seq: 9, time: 9, data: { system: { insert: 'not-an-array' }, tools: null } })
|
||||
it('leaves malformed headers with no scrubbable payload byte-identical', () => {
|
||||
const headerless = JSON.stringify({ type: 'request/header', seq: 10, time: 9, data: { reason: 'initial' } })
|
||||
const nullData = JSON.stringify({ type: 'request/header', seq: 11, time: 9, data: null })
|
||||
const raw = `${headerLine}\n${configOnly}\n${oddShapes}\n${headerless}\n${nullData}\n`
|
||||
const raw = `${headerLine}\n${headerless}\n${nullData}\n`
|
||||
expect(scrubRequestHeaders(raw)).toBe(raw)
|
||||
})
|
||||
|
||||
it('scrubs a one-sided tools delta and passes non-object schema entries through', () => {
|
||||
const addedOnly = JSON.stringify({
|
||||
type: 'request/header-delta', seq: 8, time: 9,
|
||||
data: { tools: { added: [null, 'weird', { name: 'x', description: 'D' }] } },
|
||||
})
|
||||
const out = scrubRequestHeaders(`${headerLine}\n${addedOnly}\n`)
|
||||
// Non-object entries survive untouched; the object entry keeps only name.
|
||||
expect(out).toContain('"added":[null,"weird",{"name":"x","description":"{{tools}}"}]')
|
||||
const changedOnly = JSON.stringify({
|
||||
type: 'request/header-delta', seq: 8, time: 9,
|
||||
data: { tools: { changed: [{ name: 'y', parameters: {} }] } },
|
||||
})
|
||||
expect(scrubRequestHeaders(`${headerLine}\n${changedOnly}\n`))
|
||||
.toContain('"changed":[{"name":"y","parameters":"{{tools}}"}]')
|
||||
})
|
||||
|
||||
it('scrubs a header-delta system payload but keeps its line positions and arity', () => {
|
||||
const delta = JSON.stringify({
|
||||
type: 'request/header-delta', seq: 8, time: 9,
|
||||
data: { system: { keepStart: 1, keepEnd: 4, insert: ['leaked prompt line', 'second line'] }, config: { model: 'm2' } },
|
||||
})
|
||||
const out = scrubRequestHeaders(`${headerLine}\n${delta}\n`)
|
||||
// One token PER inserted line: the edit's position AND extent survive.
|
||||
expect(out).toContain('"insert":["{{system}}","{{system}}"]')
|
||||
expect(out).toContain('"keepStart":1')
|
||||
expect(out).toContain('"keepEnd":4')
|
||||
expect(out).toContain('"config":{"model":"m2"}')
|
||||
expect(out).not.toContain('leaked prompt line')
|
||||
expect(out).not.toContain('{{tools}}') // no tools delta → none invented
|
||||
})
|
||||
|
||||
it('scrubs a header-delta tools payload but keeps the added/removed/changed names', () => {
|
||||
const delta = JSON.stringify({
|
||||
type: 'request/header-delta', seq: 8, time: 9,
|
||||
data: {
|
||||
tools: {
|
||||
added: [{ name: 'grep', description: 'Search files.', parameters: { type: 'object' } }],
|
||||
removed: ['bash_kill'],
|
||||
changed: [{ name: 'read', description: 'Read v2.', parameters: { type: 'object' } }],
|
||||
},
|
||||
},
|
||||
})
|
||||
const out = scrubRequestHeaders(`${headerLine}\n${delta}\n`)
|
||||
// WHICH tools changed is behavior and survives; their bulk does not.
|
||||
expect(out).toContain('"added":[{"name":"grep","description":"{{tools}}","parameters":"{{tools}}"}]')
|
||||
expect(out).toContain('"removed":["bash_kill"]')
|
||||
expect(out).toContain('"changed":[{"name":"read","description":"{{tools}}","parameters":"{{tools}}"}]')
|
||||
expect(out).not.toContain('Search files')
|
||||
expect(out).not.toContain('Read v2')
|
||||
})
|
||||
|
||||
it('passes every other line through byte-for-byte and is idempotent', () => {
|
||||
const other = JSON.stringify({ type: 'assistant/chunk', seq: 4, time: 9, data: { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'hi' } } })
|
||||
const delta = JSON.stringify({
|
||||
type: 'request/header-delta', seq: 8, time: 9,
|
||||
data: { system: { keepStart: 0, keepEnd: 0, insert: ['x'] }, tools: { added: [{ name: 't', description: 'd', parameters: {} }], removed: [], changed: [] } },
|
||||
})
|
||||
const raw = `${headerLine}\n${headerEvent({ config: { model: 'm' }, system: 's', tools: [] })}\n${delta}\n${other}\n`
|
||||
const raw = `${headerLine}\n${headerEvent({ config: { model: 'm' }, system: 's', tools: [] })}\n${other}\n`
|
||||
const once = scrubRequestHeaders(raw)
|
||||
expect(once.split('\n')[0]).toBe(headerLine)
|
||||
expect(once.split('\n')[3]).toBe(other)
|
||||
expect(once.split('\n')[2]).toBe(other)
|
||||
expect(scrubRequestHeaders(once)).toBe(once)
|
||||
})
|
||||
})
|
||||
@@ -281,12 +257,15 @@ describe('scrubSystemPrompts', () => {
|
||||
reason: 'initial',
|
||||
},
|
||||
})
|
||||
const delta = JSON.stringify({
|
||||
type: 'request/header-delta', seq: 2, time: 3,
|
||||
const changed = JSON.stringify({
|
||||
type: 'request/header', seq: 2, time: 3,
|
||||
data: {
|
||||
system: { keepStart: 1, keepEnd: 2, insert: ['new prompt line'] },
|
||||
tools: { changed: [{ name: 'read', description: 'changed schema' }] },
|
||||
messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'changed prefix' }] }],
|
||||
header: {
|
||||
system: 'new prompt',
|
||||
tools: [{ name: 'read', description: 'changed schema' }],
|
||||
messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'changed prefix' }] }],
|
||||
},
|
||||
reason: 'change',
|
||||
},
|
||||
})
|
||||
const toolsOnly = JSON.stringify({
|
||||
@@ -294,11 +273,10 @@ describe('scrubSystemPrompts', () => {
|
||||
data: { header: { tools: [{ name: 'read', description: 'schema only' }] }, reason: 'resume' },
|
||||
})
|
||||
|
||||
const out = scrubSystemPrompts(`${header}\n${delta}\n${toolsOnly}\n`)
|
||||
const out = scrubSystemPrompts(`${header}\n${changed}\n${toolsOnly}\n`)
|
||||
expect(out).toContain('"system":"{{system}}"')
|
||||
expect(out).toContain('"insert":["{{system}}"]')
|
||||
expect(out).not.toContain('full prompt')
|
||||
expect(out).not.toContain('new prompt line')
|
||||
expect(out).not.toContain('new prompt')
|
||||
expect(out).toContain('full schema')
|
||||
expect(out).toContain('full prefix')
|
||||
expect(out).toContain('changed schema')
|
||||
@@ -321,12 +299,15 @@ describe('scrubToolSchemas', () => {
|
||||
reason: 'initial',
|
||||
},
|
||||
})
|
||||
const delta = JSON.stringify({
|
||||
type: 'request/header-delta', seq: 2, time: 3,
|
||||
const changed = JSON.stringify({
|
||||
type: 'request/header', seq: 2, time: 3,
|
||||
data: {
|
||||
system: { keepStart: 1, keepEnd: 2, insert: ['new prompt line'] },
|
||||
tools: { added: [{ name: 'grep', description: 'new schema' }], changed: [{ name: 'read', description: 'changed schema' }] },
|
||||
messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'changed prefix' }] }],
|
||||
header: {
|
||||
system: 'new prompt',
|
||||
tools: [{ name: 'grep', description: 'new schema' }],
|
||||
messagePrefix: [{ role: 'user', content: [{ type: 'text', text: 'changed prefix' }] }],
|
||||
},
|
||||
reason: 'change',
|
||||
},
|
||||
})
|
||||
const systemOnly = JSON.stringify({
|
||||
@@ -334,15 +315,12 @@ describe('scrubToolSchemas', () => {
|
||||
data: { header: { system: 'prompt only' }, reason: 'resume' },
|
||||
})
|
||||
|
||||
const out = scrubToolSchemas(`${header}\n${delta}\n${systemOnly}\n`)
|
||||
expect(out).toContain('"tools":"{{tools}}"')
|
||||
expect(out).toContain('"added":[{"name":"grep","description":"{{tools}}"}]')
|
||||
expect(out).toContain('"changed":[{"name":"read","description":"{{tools}}"}]')
|
||||
const out = scrubToolSchemas(`${header}\n${changed}\n${systemOnly}\n`)
|
||||
expect(out.match(/"tools":"{{tools}}"/g)).toHaveLength(2)
|
||||
expect(out).not.toContain('full schema')
|
||||
expect(out).not.toContain('new schema')
|
||||
expect(out).not.toContain('changed schema')
|
||||
expect(out).toContain('full prompt')
|
||||
expect(out).toContain('new prompt line')
|
||||
expect(out).toContain('new prompt')
|
||||
expect(out).toContain('full prefix')
|
||||
expect(out).toContain('changed prefix')
|
||||
expect(out.split('\n')[2]).toBe(systemOnly)
|
||||
|
||||
@@ -9,12 +9,10 @@ import {
|
||||
childFixturePaths,
|
||||
fixtureContext,
|
||||
formatSystemPromptSnapshot,
|
||||
headerChangeCount,
|
||||
formatToolSchemasSnapshot,
|
||||
headerDeltaCount,
|
||||
normalizedHeaders,
|
||||
normalizedSystemPromptDeltas,
|
||||
normalizedSystemPrompts,
|
||||
normalizedToolSchemaDeltas,
|
||||
normalizedToolSchemas,
|
||||
parseToolSchemasSnapshot,
|
||||
refreshFixtureReplacements,
|
||||
@@ -45,7 +43,7 @@ const RECORD_SRC = fileURLToPath(new URL('./fixtures/record-suite', import.meta.
|
||||
|
||||
// Replay pins explicit header classes; recording covers the default fallback.
|
||||
const REPLAY_SCENARIOS: Scenario[] = [
|
||||
{ name: 'pin-turn', hasModelTurn: true, recorded: true, pinsHeader: true, expectedHeaderDeltas: 1, headerClass: 'main' },
|
||||
{ name: 'pin-turn', hasModelTurn: true, recorded: true, pinsHeader: true, expectedHeaderChanges: 1, headerClass: 'main' },
|
||||
{ name: 'plain-turn', hasModelTurn: true, recorded: true, childSessions: 1, headerClass: 'main', configPath: AGENT.configPath },
|
||||
{ name: 'no-model', hasModelTurn: false, recorded: false, headerClass: 'main' },
|
||||
{ name: 'blocked-log', hasModelTurn: false, comparesLog: true, recorded: false, headerClass: 'main' },
|
||||
@@ -76,7 +74,7 @@ afterAll(async () => {
|
||||
function staleRefreshFixtures(dir: string): void {
|
||||
writeFileSync(join(dir, 'plain-turn', 'stdout.golden.jsonl'), 'stale stdout\n')
|
||||
writeFileSync(join(dir, 'pin-turn', 'system-prompt.golden.md'), 'STALE PROMPT\n')
|
||||
writeFileSync(join(dir, 'pin-turn', 'tool-schemas.golden.json'), '{"initial":[{"name":"stale"}],"deltas":[]}\n')
|
||||
writeFileSync(join(dir, 'pin-turn', 'tool-schemas.golden.json'), '{"initial":[{"name":"stale"}],"changes":[]}\n')
|
||||
|
||||
const plainBehaviorFile = join(dir, 'plain-turn', 'behavior.json')
|
||||
const plainBehavior = JSON.parse(readFileSync(plainBehaviorFile, 'utf8')) as Record<string, unknown>
|
||||
@@ -127,7 +125,9 @@ describe('defineAcpSnapshotSuite: refresh write-back', () => {
|
||||
expect(readFileSync(join(refreshDir, 'pin-turn', 'system-prompt.golden.md'), 'utf8')).toBe([
|
||||
'SYS PROMPT',
|
||||
'',
|
||||
'<!-- request/header-delta 1: keepStart=1, keepEnd=0 -->',
|
||||
'<!-- request/header change 1 -->',
|
||||
'',
|
||||
'SYS PROMPT',
|
||||
'',
|
||||
'NEW PROMPT LINE',
|
||||
'',
|
||||
@@ -262,65 +262,41 @@ describe('normalizedToolSchemas', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('normalizedToolSchemaDeltas', () => {
|
||||
it('extracts and normalizes object-valued schema edits', () => {
|
||||
const log = [
|
||||
'{"type":"request/header-delta","data":{"tools":{"added":[{"name":"read","description":"work in /w"}]}}}',
|
||||
'{"type":"request/header-delta","data":{"tools":null}}',
|
||||
'{"type":"request/header-delta","data":{"tools":"invalid"}}',
|
||||
'{"type":"request/header-delta","data":{"tools":[]}}',
|
||||
'{"type":"request/header-delta","data":{"system":{"insert":[]}}}',
|
||||
'{"type":"request/header","data":{"tools":{"added":[]}}}',
|
||||
'',
|
||||
].join('\n')
|
||||
expect(normalizedToolSchemaDeltas(log, { sessionIds: [], cwd: '/w' })).toEqual([
|
||||
{ added: [{ name: 'read', description: 'work in {{cwd}}' }] },
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
describe('normalizedSystemPromptDeltas', () => {
|
||||
it('extracts and normalizes well-formed system edits', () => {
|
||||
const log = [
|
||||
'{"type":"request/header-delta","data":{"system":{"keepStart":1,"keepEnd":0,"insert":["work in /w"]}}}',
|
||||
'{"type":"request/header-delta","data":{"tools":{"replace":[]}}}',
|
||||
'{"type":"request/header-delta","data":{"system":{"keepStart":"1","keepEnd":0,"insert":[]}}}',
|
||||
'{"type":"request/header-delta","data":{"system":{"keepStart":1,"keepEnd":0,"insert":[null]}}}',
|
||||
'',
|
||||
].join('\n')
|
||||
expect(normalizedSystemPromptDeltas(log, { sessionIds: [], cwd: '/w' })).toEqual([
|
||||
{ keepStart: 1, keepEnd: 0, insert: ['work in {{cwd}}'] },
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
describe('formatSystemPromptSnapshot', () => {
|
||||
it('adds a missing terminal newline without changing an existing one', () => {
|
||||
expect(formatSystemPromptSnapshot('prompt')).toBe('prompt\n')
|
||||
expect(formatSystemPromptSnapshot('prompt\n')).toBe('prompt\n')
|
||||
})
|
||||
|
||||
it('renders readable system-prompt delta sections', () => {
|
||||
expect(formatSystemPromptSnapshot('prompt', [
|
||||
{ keepStart: 1, keepEnd: 0, insert: ['new', 'lines'] },
|
||||
])).toBe('prompt\n\n<!-- request/header-delta 1: keepStart=1, keepEnd=0 -->\n\nnew\nlines\n')
|
||||
it('renders readable changed-prompt sections', () => {
|
||||
expect(formatSystemPromptSnapshot('prompt', ['new\nlines']))
|
||||
.toBe('prompt\n\n<!-- request/header change 1 -->\n\nnew\nlines\n')
|
||||
})
|
||||
|
||||
it('does not double the newline of a delta insert with a trailing blank line', () => {
|
||||
expect(formatSystemPromptSnapshot('prompt\n', [
|
||||
{ keepStart: 2, keepEnd: 1, insert: ['tail', ''] },
|
||||
])).toBe('prompt\n\n<!-- request/header-delta 1: keepStart=2, keepEnd=1 -->\n\ntail\n')
|
||||
it('does not double the newline of a changed prompt', () => {
|
||||
expect(formatSystemPromptSnapshot('prompt\n', ['changed\n']))
|
||||
.toBe('prompt\n\n<!-- request/header change 1 -->\n\nchanged\n')
|
||||
})
|
||||
})
|
||||
|
||||
describe('headerChangeCount', () => {
|
||||
it('counts changed request headers, ignoring anchors, blanks, and other lines', () => {
|
||||
const change = JSON.stringify({ type: 'request/header', seq: 2, time: 9, data: { reason: 'change' } })
|
||||
const anchor = JSON.stringify({ type: 'request/header', seq: 0, time: 9, data: { reason: 'initial' } })
|
||||
const other = JSON.stringify({ type: 'turn/start', seq: 1, time: 9, data: {} })
|
||||
expect(headerChangeCount(`${anchor}\n${other}\n\n${change}\n${change}\n`)).toBe(2)
|
||||
expect(headerChangeCount(`${anchor}\n`)).toBe(0)
|
||||
})
|
||||
})
|
||||
|
||||
describe('tool-schema snapshots', () => {
|
||||
const snapshot = {
|
||||
initial: [{ name: 'read', description: 'Read a file.' }],
|
||||
deltas: [{ added: [{ name: 'grep', description: 'Search files.' }] }],
|
||||
changes: [[{ name: 'grep', description: 'Search files.' }]],
|
||||
}
|
||||
|
||||
it('formats and parses canonical structured JSON', () => {
|
||||
const formatted = formatToolSchemasSnapshot(snapshot.initial, snapshot.deltas)
|
||||
const formatted = formatToolSchemasSnapshot(snapshot.initial, snapshot.changes)
|
||||
expect(formatted).toBe(`${JSON.stringify(snapshot, null, 2)}\n`)
|
||||
expect(parseToolSchemasSnapshot(formatted)).toEqual(snapshot)
|
||||
})
|
||||
@@ -329,29 +305,21 @@ describe('tool-schema snapshots', () => {
|
||||
expect(() => parseToolSchemasSnapshot('null')).toThrow(/must be an object/)
|
||||
expect(() => parseToolSchemasSnapshot('"invalid"')).toThrow(/must be an object/)
|
||||
expect(() => parseToolSchemasSnapshot('[]')).toThrow(/must be an object/)
|
||||
expect(() => parseToolSchemasSnapshot('{"initial":{},"deltas":[]}')).toThrow(/array-valued/)
|
||||
expect(() => parseToolSchemasSnapshot('{"initial":[],"deltas":{}}')).toThrow(/array-valued/)
|
||||
expect(() => parseToolSchemasSnapshot('{"initial":{},"changes":[]}')).toThrow(/array-valued/)
|
||||
expect(() => parseToolSchemasSnapshot('{"initial":[],"changes":{}}')).toThrow(/array-valued/)
|
||||
expect(() => parseToolSchemasSnapshot('{"initial":[],"changes":[{}]}')).toThrow(/array-valued/)
|
||||
})
|
||||
|
||||
it('restores initial schemas into the pinned header token', () => {
|
||||
expect(restorePinnedToolSchemas({ system: '{{system}}', tools: '{{tools}}' }, snapshot))
|
||||
expect(restorePinnedToolSchemas({ system: '{{system}}', tools: '{{tools}}' }, snapshot.initial))
|
||||
.toEqual({ system: '{{system}}', tools: snapshot.initial })
|
||||
})
|
||||
|
||||
it('rejects invalid headers and a missing tool token', () => {
|
||||
expect(() => restorePinnedToolSchemas(null, snapshot)).toThrow(/must be an object/)
|
||||
expect(() => restorePinnedToolSchemas('invalid', snapshot)).toThrow(/must be an object/)
|
||||
expect(() => restorePinnedToolSchemas([], snapshot)).toThrow(/must be an object/)
|
||||
expect(() => restorePinnedToolSchemas({ tools: [] }, snapshot)).toThrow(/must equal/)
|
||||
})
|
||||
})
|
||||
|
||||
describe('headerDeltaCount', () => {
|
||||
it('counts request/header-delta events, ignoring blanks and other lines', () => {
|
||||
const delta = JSON.stringify({ type: 'request/header-delta', seq: 2, time: 9, data: {} })
|
||||
const other = JSON.stringify({ type: 'request/header', seq: 0, time: 9, data: {} })
|
||||
expect(headerDeltaCount(`${other}\n\n${delta}\n${delta}\n`)).toBe(2)
|
||||
expect(headerDeltaCount(`${other}\n`)).toBe(0)
|
||||
expect(() => restorePinnedToolSchemas(null, snapshot.initial)).toThrow(/must be an object/)
|
||||
expect(() => restorePinnedToolSchemas('invalid', snapshot.initial)).toThrow(/must be an object/)
|
||||
expect(() => restorePinnedToolSchemas([], snapshot.initial)).toThrow(/must be an object/)
|
||||
expect(() => restorePinnedToolSchemas({ tools: [] }, snapshot.initial)).toThrow(/must equal/)
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
27
packages/support/agent-loop-testkit/README.md
Normal file
27
packages/support/agent-loop-testkit/README.md
Normal file
@@ -0,0 +1,27 @@
|
||||
# `@deepseek-ai/dsh-agent-loop-testkit`
|
||||
|
||||
Shared prerequisite mounting for tests that exercise the concrete `AgentLoop`. `mountAgentLoopTestDependencies(ctx, options?)` installs the LLM, session, system-prompt, tool, and agent services in dependency order, then returns before the loop is mounted.
|
||||
|
||||
The caller registers adapters and optional plugins, mounts `AgentLoop` with the configuration under test, and disposes its own Context. System-prompt and tool-registry configuration can be forwarded through `options`; the helper does not provide test defaults beyond those owned by the services. A plugin-load failure rejects the helper call, while services activated earlier in the sequence remain owned by the caller's Context.
|
||||
|
||||
```ts
|
||||
import { Context } from 'cordis'
|
||||
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
|
||||
import { mountAgentLoopTestDependencies } from '@deepseek-ai/dsh-agent-loop-testkit'
|
||||
|
||||
const ctx = new Context()
|
||||
|
||||
await mountAgentLoopTestDependencies(ctx)
|
||||
// Register the test adapter and any optional plugins here.
|
||||
await ctx.plugin(AgentLoop, { agents: [] })
|
||||
```
|
||||
|
||||
Tests of injection failures, partial topology, service load order, or service teardown mount their dependencies directly instead of using this helper.
|
||||
|
||||
## Model Experience
|
||||
|
||||
None, as this test-only composition helper neither drives nor modifies model requests.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **Only the mandatory prerequisite spine is shared** — adapters, optional plugins, `AgentLoop`, agents, and Context teardown remain caller-owned so scenario-specific ordering stays visible.
|
||||
41
packages/support/agent-loop-testkit/package.json
Normal file
41
packages/support/agent-loop-testkit/package.json
Normal file
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-agent-loop-testkit",
|
||||
"description": "Shared prerequisite mounting for tests that exercise the concrete agent loop",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/types/**/*.d.ts",
|
||||
"lib/types/**/*.d.ts.map",
|
||||
"src"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-agent": "^0.0.1",
|
||||
"@deepseek-ai/dsh-llm": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session": "^0.0.1",
|
||||
"@deepseek-ai/dsh-system-prompt": "^0.0.1",
|
||||
"@deepseek-ai/dsh-tools": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-agent": "workspace:^",
|
||||
"@deepseek-ai/dsh-agent-loop": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-system-prompt": "workspace:^",
|
||||
"@deepseek-ai/dsh-tools": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
46
packages/support/agent-loop-testkit/src/index.ts
Normal file
46
packages/support/agent-loop-testkit/src/index.ts
Normal file
@@ -0,0 +1,46 @@
|
||||
/**
|
||||
* Shared mounting for the services required before tests load the concrete
|
||||
* agent loop. The caller retains ownership of the context, loop, adapters,
|
||||
* optional plugins, and teardown.
|
||||
* @module @deepseek-ai/dsh-agent-loop-testkit
|
||||
*/
|
||||
|
||||
import type { Context } from 'cordis'
|
||||
import AgentRegistry from '@deepseek-ai/dsh-agent'
|
||||
import LlmService from '@deepseek-ai/dsh-llm'
|
||||
import SessionStore from '@deepseek-ai/dsh-session'
|
||||
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
|
||||
import type { Config as SystemPromptConfig } from '@deepseek-ai/dsh-system-prompt'
|
||||
import ToolRegistry from '@deepseek-ai/dsh-tools'
|
||||
import type { Config as ToolRegistryConfig } from '@deepseek-ai/dsh-tools'
|
||||
|
||||
/** Configuration forwarded to the prerequisite service plugins. */
|
||||
export interface AgentLoopTestDependenciesOptions {
|
||||
/** Configuration for the system-prompt registry. */
|
||||
readonly systemPrompt?: SystemPromptConfig
|
||||
/** Configuration for the tool registry. */
|
||||
readonly tools?: ToolRegistryConfig
|
||||
}
|
||||
|
||||
/**
|
||||
* Mount the standard prerequisite services for an AgentLoop test.
|
||||
*
|
||||
* The function deliberately does not mount AgentLoop or register an adapter,
|
||||
* so tests retain control of load order and the topology under test. The
|
||||
* context owns every mounted service and remains responsible for disposal. A
|
||||
* plugin-load failure rejects the promise; services activated earlier in the
|
||||
* sequence remain context-owned and unwind with that context.
|
||||
* @param ctx - test context that owns the mounted services.
|
||||
* @param options - optional service configuration forwarded without mutation.
|
||||
* @returns after every prerequisite service has activated.
|
||||
*/
|
||||
export async function mountAgentLoopTestDependencies(
|
||||
ctx: Context,
|
||||
options: AgentLoopTestDependenciesOptions = {},
|
||||
): Promise<void> {
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SystemPrompt, options.systemPrompt ?? {})
|
||||
await ctx.plugin(ToolRegistry, options.tools ?? {})
|
||||
await ctx.plugin(AgentRegistry)
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
|
||||
import { renderPrompt } from '@deepseek-ai/dsh-system-prompt'
|
||||
import { mountAgentLoopTestDependencies } from '../src/index.ts'
|
||||
|
||||
describe('dsh-agent-loop-testkit', () => {
|
||||
it('mounts a configurable prerequisite spine that can activate AgentLoop', async () => {
|
||||
const ctx = new Context()
|
||||
await mountAgentLoopTestDependencies(ctx, {
|
||||
systemPrompt: { persona: 'Test persona.' },
|
||||
tools: { mode: 'native' },
|
||||
})
|
||||
|
||||
expect(renderPrompt(await ctx.systemPrompt.assemble())).toContain('Test persona.')
|
||||
await expect(ctx.plugin(AgentLoop, { agents: [] })).resolves.toBeDefined()
|
||||
|
||||
await ctx.fiber.dispose()
|
||||
})
|
||||
})
|
||||
33
packages/support/agent-loop-testkit/tsconfig.json
Normal file
33
packages/support/agent-loop-testkit/tsconfig.json
Normal file
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../core/agent"
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
{
|
||||
"path": "../../core/system-prompt"
|
||||
},
|
||||
{
|
||||
"path": "../../core/tools"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -4,7 +4,7 @@ Runtime event-contract assertions intended for development diagnostics. This pur
|
||||
|
||||
The plugin has no environment guard: it is active wherever it is registered. The default [`dsh-agent-spine-demo`](../../examples/agent-spine-demo/README.md) bundle mounts it unconditionally; a custom composition can omit it when the runtime cost is undesirable. It doubles as executable documentation of the event taxonomy — the assertions *are* the contract.
|
||||
|
||||
Session itself owns immutable log storage in every composition: it takes one lossless JSON snapshot of each accepted event, deep-freezes that record, and exposes the log through immutable array snapshots. The invariants plugin checks the cross-record and cross-seam rules that storage immutability cannot express.
|
||||
Session itself owns immutable, surface-valid log storage in every composition: it takes one lossless JSON snapshot of each candidate, validates the complete surface transition, deep-freezes the accepted record, and exposes the log through immutable array snapshots. The invariants plugin checks the remaining cross-record and cross-seam rules that Session does not own.
|
||||
|
||||
Session-log assertions run during Cordis `internal/dispatch`, while `Session.append()` is resolving the `session/event` callback snapshot but before it pushes the candidate into the log. A valid transition is staged by exact event identity and applied to the live trace only when that same committed event reaches the plugin's contained post-commit listener. A later internal dispatch check can therefore veto without advancing either the log or the invariant trace, while ordinary `session/event` observer failures remain observe-only.
|
||||
|
||||
@@ -32,6 +32,7 @@ Session log (per session):
|
||||
- **steps nest in turns** — `step/start` opens a step in the open turn; `step/end` closes the matching step.
|
||||
- **chunks belong to an open step** — `step/start` precedes its `assistant/chunk`s.
|
||||
- **a `tool/result` needs a prior `tool/call`** — but NOT the converse: a `tool/call` may have no result (a thrown tool-execution pipeline step ends the turn with no `tool/result`, which is legal).
|
||||
- **provenance sources are valid and unambiguous** — `sourceEventSeqs` contains unique earlier known seqs; only `assistant/message` may carry an explicit empty list, which denotes a known empty provider stream rather than absent legacy provenance.
|
||||
|
||||
Agent status (per agent):
|
||||
|
||||
@@ -39,7 +40,7 @@ Agent status (per agent):
|
||||
|
||||
Model requests (on `llm/stream`):
|
||||
|
||||
- **a loop-built request is exactly what the log reconstructs** — a frozen request with a live `sessionId` is rebuilt through a fresh `Session` from the prefix before its in-flight `step/start`; later content belongs to the next request, and hand-built unfrozen one-shots are excluded. Frozen messages must match that derivation, while every other field matches folded `request/header*` events. The prepended check runs before ordinary short-circuiting stream listeners, but correctness comes from the sequence boundary rather than listener timing. See the [reconstructability RFC](../../../docs/rfc/implemented/architecture/2026-07-05-reconstructable-requests.md).
|
||||
- **a loop-built request is exactly what the log reconstructs** — a frozen request with a live `sessionId` (the loop-built marker; hand-built one-shots like compaction's summarize are unfrozen and skipped) must carry frozen `messages` deep-equal to the derivation over the log prefix strictly before the in-flight step's `step/start` (rebuilt through a FRESH `Session`, so the live cache cannot vouch for itself — and boundary-correct: content logged after `step/start` legitimately belongs to the next request), and every non-content field must equal the latest logged `request/header` (see [the reconstructability RFC](../../../docs/rfc/implemented/architecture/2026-07-05-reconstructable-requests.md)). Registered with `prepend: true` so a short-circuiting `llm/stream` listener (the replay adapter) cannot silence it; prepend orders it against append-registered listeners only — correctness rests on the seq-bounded rebuild, never listener timing.
|
||||
|
||||
On any violation it throws `InvariantError` (`code: 'INVARIANT'`).
|
||||
|
||||
|
||||
@@ -3,7 +3,8 @@
|
||||
* turn and step nesting, scoped dispatch, status transitions, and request
|
||||
* reconstruction. The plugin has no environment guard and is active wherever
|
||||
* mounted, including the default `dsh-agent-spine-demo` bundle; custom compositions
|
||||
* may omit it. Sessions still own event snapshots and freezing.
|
||||
* may omit it. Sessions own immutable, surface-valid event storage; this plugin
|
||||
* checks only relationships that event acceptance cannot express.
|
||||
* @module @deepseek-ai/dsh-invariants
|
||||
*/
|
||||
|
||||
@@ -13,7 +14,7 @@ import { assertNever, HarnessError } from '@deepseek-ai/dsh-llm'
|
||||
import type { CallId, GenerateOptions } from '@deepseek-ai/dsh-llm'
|
||||
import type { Agent, AgentStatus } from '@deepseek-ai/dsh-agent'
|
||||
import { Session, SessionId, foldRequestHeader } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionEvent, SurfaceEventType } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import { scopedSubjectResolverFor } from './scoped-events.generated.ts'
|
||||
|
||||
export const name = 'invariants'
|
||||
@@ -48,15 +49,6 @@ interface SessionTrace {
|
||||
* `step/end` — a result must arrive in the same step as its call.
|
||||
*/
|
||||
pendingCalls: Set<CallId>
|
||||
/** Every seq seen so far — validates `sourceEventSeqs` references. */
|
||||
knownSeqs: Set<number>
|
||||
/**
|
||||
* The seqs currently on the surface linked list, in linked-list order
|
||||
* (head to tail). A replace reorders this relative to seq order (the new
|
||||
* node takes the replaced range's position), so range validation is
|
||||
* positional, not by seq comparison.
|
||||
*/
|
||||
surface: number[]
|
||||
}
|
||||
|
||||
/** One accepted event's deferred mutation of a live session trace. */
|
||||
@@ -68,12 +60,6 @@ interface SessionTraceTransition {
|
||||
| { kind: 'none' }
|
||||
| { kind: 'add' | 'delete'; callId: CallId }
|
||||
| { kind: 'clear' }
|
||||
/** The event's mutation of the derived surface order. */
|
||||
surface:
|
||||
| { kind: 'none' | 'append' }
|
||||
| { kind: 'replace'; start: number; count: number }
|
||||
/** The committed event sequence to add to the known-sequence set. */
|
||||
seq: number
|
||||
}
|
||||
|
||||
/** Assert that a step-scoped event names the currently open turn and step. */
|
||||
@@ -97,73 +83,6 @@ function validateEvent(trace: SessionTrace, event: SessionEvent): SessionTraceTr
|
||||
let nextTurn = trace.nextTurn
|
||||
let nextStep = trace.nextStep
|
||||
let pendingCalls: SessionTraceTransition['pendingCalls'] = { kind: 'none' }
|
||||
let surface: SessionTraceTransition['surface'] = { kind: 'none' }
|
||||
|
||||
// --- Surface invariants ---
|
||||
// Surface metadata (sourceEventSeqs, surfaceOp) is only valid on
|
||||
// surface-eligible event types. The compiler enforces this at append()
|
||||
// call sites; this runtime check catches casts and persisted data.
|
||||
const SURFACE_TYPES = new Set<string>(['user/message', 'assistant/message', 'tool/result', 'context/message', 'steering/message'])
|
||||
// Cast to surface-eligible event type so we can access surfaceOp and
|
||||
// sourceEventSeqs (optional on SessionEvent, mandatory on SurfaceEvent).
|
||||
// SurfaceEvent's mandatory surfaceOp is too strict here — we need to
|
||||
// CHECK whether surface metadata is present, not assume it.
|
||||
const se = event as SessionEvent<SurfaceEventType>
|
||||
if (!SURFACE_TYPES.has(event.type)) {
|
||||
if (se.sourceEventSeqs !== undefined) {
|
||||
throw new InvariantError(`${event.type} cannot carry sourceEventSeqs (non-surface event)`)
|
||||
}
|
||||
if (se.surfaceOp !== undefined) {
|
||||
throw new InvariantError(`${event.type} cannot carry surfaceOp (non-surface event)`)
|
||||
}
|
||||
}
|
||||
if (se.sourceEventSeqs !== undefined) {
|
||||
if (se.sourceEventSeqs.length === 0) {
|
||||
throw new InvariantError('sourceEventSeqs must not be empty when present')
|
||||
}
|
||||
const unique = new Set(se.sourceEventSeqs)
|
||||
if (unique.size !== se.sourceEventSeqs.length) {
|
||||
throw new InvariantError('sourceEventSeqs must not contain duplicates')
|
||||
}
|
||||
for (const ref of se.sourceEventSeqs) {
|
||||
if (ref >= event.seq) {
|
||||
throw new InvariantError(`sourceEventSeqs must reference earlier events: ${ref} >= current seq ${event.seq}`)
|
||||
}
|
||||
if (!trace.knownSeqs.has(ref)) {
|
||||
throw new InvariantError(`sourceEventSeqs references unknown seq ${ref}`)
|
||||
}
|
||||
}
|
||||
}
|
||||
// Fold this event into the tracked surface linked list, validating the
|
||||
// replace contract as we go. `append` adds a tail node; `replace` shadows a
|
||||
// positional range — every shadowed node must appear in sourceEventSeqs.
|
||||
if (se.surfaceOp !== undefined) {
|
||||
if (se.surfaceOp === 'append') {
|
||||
surface = { kind: 'append' }
|
||||
} else {
|
||||
const { start, end } = se.surfaceOp
|
||||
const startIdx = trace.surface.indexOf(start)
|
||||
if (startIdx === -1) {
|
||||
throw new InvariantError(`surface replace: start seq ${start} is not on the surface`)
|
||||
}
|
||||
const endIdx = trace.surface.indexOf(end)
|
||||
if (endIdx === -1) {
|
||||
throw new InvariantError(`surface replace: end seq ${end} is not on the surface`)
|
||||
}
|
||||
if (startIdx > endIdx) {
|
||||
throw new InvariantError(`surface replace: start seq ${start} (pos ${startIdx}) is after end seq ${end} (pos ${endIdx}) on the surface`)
|
||||
}
|
||||
// Every node the replace shadows (surface positions [startIdx, endIdx]
|
||||
// inclusive) must appear in sourceEventSeqs — the provenance contract.
|
||||
const shadowed = trace.surface.slice(startIdx, endIdx + 1)
|
||||
const recorded = new Set(se.sourceEventSeqs ?? [])
|
||||
const missing = shadowed.filter(seq => !recorded.has(seq))
|
||||
if (missing.length > 0) {
|
||||
throw new InvariantError(`surface replace: sourceEventSeqs must include every shadowed surface node; missing ${missing.join(', ')}`)
|
||||
}
|
||||
surface = { kind: 'replace', start: startIdx, count: shadowed.length }
|
||||
}
|
||||
}
|
||||
|
||||
// Boundary/step-scoped events have explicit cases; every OTHER event type —
|
||||
// including plugin-added (merge-extensible) SessionEventMap keys — is caught
|
||||
@@ -263,8 +182,6 @@ function validateEvent(trace: SessionTrace, event: SessionEvent): SessionTraceTr
|
||||
return {
|
||||
scalars: { lastSeq: event.seq, openTurn, openStep, nextTurn, nextStep },
|
||||
pendingCalls,
|
||||
surface,
|
||||
seq: event.seq,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -287,20 +204,6 @@ function applyTransition(trace: SessionTrace, transition: SessionTraceTransition
|
||||
default:
|
||||
assertNever(transition.pendingCalls, 'session trace pending-call transition')
|
||||
}
|
||||
switch (transition.surface.kind) {
|
||||
case 'none':
|
||||
break
|
||||
case 'append':
|
||||
trace.surface.push(transition.seq)
|
||||
break
|
||||
case 'replace':
|
||||
trace.surface.splice(transition.surface.start, transition.surface.count, transition.seq)
|
||||
break
|
||||
/* v8 ignore next -- validateEvent produces this closed transition union */
|
||||
default:
|
||||
assertNever(transition.surface, 'session trace surface transition')
|
||||
}
|
||||
trace.knownSeqs.add(transition.seq)
|
||||
}
|
||||
|
||||
/** Validate and apply one event while rebuilding an already-committed log. */
|
||||
@@ -345,8 +248,6 @@ export function apply(ctx: Context): void {
|
||||
nextTurn: 1,
|
||||
nextStep: 1,
|
||||
pendingCalls: new Set(),
|
||||
knownSeqs: new Set(),
|
||||
surface: [],
|
||||
})
|
||||
|
||||
/** Build (or rebuild) a session's trace by replaying its whole log. */
|
||||
@@ -445,7 +346,7 @@ export function apply(ctx: Context): void {
|
||||
// the boundary (an `agent/request`-window inject) is legitimately absent
|
||||
// from this request, and a current-surface comparison would false-fire.
|
||||
// - header: every non-content field must equal the fold of the log's
|
||||
// `request/header*` events — the loop logs the header event BEFORE
|
||||
// `request/header` events — the loop logs the header event BEFORE
|
||||
// dispatch, so the fold already covers this request.
|
||||
//
|
||||
// Registered with `prepend: true` so a short-circuiting llm/stream listener
|
||||
|
||||
@@ -48,7 +48,7 @@ describe('session-log invariants', () => {
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
session.append('assistant/chunk', { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'h' } })
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [{ type: 'tool-call', id: CallId('c1'), name: 'echo', arguments: '{}' }] }, { surfaceOp: 'append' })
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [{ type: 'tool-call', id: CallId('c1'), name: 'echo', arguments: '{}' }] }, { surfaceOp: 'append' })
|
||||
session.append('tool/call', { turn: 1, step: 1, callId: CallId('c1'), name: 'echo', arguments: '{}' })
|
||||
session.append('tool/result', { turn: 1, step: 1, callId: CallId('c1'), content: [{ type: 'text', text: 'ok' }], isError: false }, { surfaceOp: 'append' })
|
||||
session.append('step/end', { turn: 1, step: 1 })
|
||||
@@ -195,7 +195,7 @@ describe('session-log invariants', () => {
|
||||
expect(() => {
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [
|
||||
{ type: 'tool-call', id: CallId('crashed'), name: 'bash', arguments: '{}' },
|
||||
] }, { surfaceOp: 'append' })
|
||||
session.append('tool/result', {
|
||||
@@ -251,10 +251,10 @@ describe('session-log invariants', () => {
|
||||
expect(() => {
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append' })
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append' })
|
||||
session.append('step/end', { turn: 1, step: 1 })
|
||||
session.append('step/start', { turn: 1, step: 2 })
|
||||
session.append('assistant/message', { turn: 1, step: 2, content: [] }, { surfaceOp: 'append' })
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 2, content: [] }, { surfaceOp: 'append' })
|
||||
session.append('step/end', { turn: 1, step: 2 })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
session.append('turn/start', { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
@@ -316,7 +316,7 @@ describe('session-log invariants', () => {
|
||||
const session = ctx.sessions.create()
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
expect(() => session.append('assistant/message', { turn: 1, step: 2, content: [] }, { surfaceOp: 'append' }))
|
||||
expect(() => session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 2, content: [] }, { surfaceOp: 'append' }))
|
||||
.toThrow(/open is turn 1\/step 1/)
|
||||
})
|
||||
})
|
||||
@@ -464,7 +464,7 @@ describe('HMR safety', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('surface invariants', () => {
|
||||
describe('surface contract under the invariants composition', () => {
|
||||
it('accepts well-formed surface metadata', async () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
@@ -473,7 +473,7 @@ describe('surface invariants', () => {
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
expect(() => {
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
|
||||
}).not.toThrow()
|
||||
})
|
||||
|
||||
@@ -483,17 +483,21 @@ describe('surface invariants', () => {
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] })
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] })
|
||||
// no throw — well-formed replace op
|
||||
})
|
||||
|
||||
it('rejects empty sourceEventSeqs', async () => {
|
||||
it('accepts known-empty assistant provenance and rejects empty provenance elsewhere', async () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
expect(() => {
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [] })
|
||||
}).toThrow(InvariantError)
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [] })
|
||||
}).not.toThrow()
|
||||
expect(() => {
|
||||
session.append('user/message', { content: [], source: { kind: 'user' } }, { surfaceOp: 'append', sourceEventSeqs: [] })
|
||||
}).toThrow(/must not be empty except on assistant\/message/)
|
||||
})
|
||||
|
||||
it('rejects duplicate sourceEventSeqs', async () => {
|
||||
@@ -502,7 +506,7 @@ describe('surface invariants', () => {
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
|
||||
expect(() => {
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1, 1] })
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1, 1] })
|
||||
}).toThrow(/must not contain duplicates/)
|
||||
})
|
||||
|
||||
@@ -513,19 +517,20 @@ describe('surface invariants', () => {
|
||||
// The next event is seq 1. Referencing its own seq fails on "must reference
|
||||
// earlier events" (the check order is: earlier first, then unknown).
|
||||
expect(() => {
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
|
||||
}).toThrow(/must reference earlier/)
|
||||
})
|
||||
|
||||
it('accepts sourceEventSeqs referencing a valid earlier event', async () => {
|
||||
// Positive test: ref < current seq and ref is in knownSeqs → passes.
|
||||
// Session seqs are contiguous, so every non-negative ref below the current
|
||||
// seq necessarily names an existing earlier event.
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
// seqs so far: 0, 1. The next event at seq 2 references seq 1 → valid.
|
||||
expect(() => {
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [1] })
|
||||
}).not.toThrow()
|
||||
})
|
||||
|
||||
@@ -534,27 +539,10 @@ describe('surface invariants', () => {
|
||||
const session = ctx.sessions.create()
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
expect(() => {
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [99] })
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [99] })
|
||||
}).toThrow(/must reference earlier/)
|
||||
})
|
||||
|
||||
it('rejects sourceEventSeqs referencing unknown seq (gap in event log)', async () => {
|
||||
// Create an impossible-through-public-API gap so seq 2 is earlier but unknown.
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
;(session as unknown as { log: unknown[] }).log.push({
|
||||
type: 'assistant/chunk',
|
||||
seq: 3,
|
||||
time: Date.now(),
|
||||
data: { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'x' } },
|
||||
})
|
||||
expect(() => {
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [2] })
|
||||
}).toThrow(/unknown seq 2/)
|
||||
})
|
||||
|
||||
it('rejects a replace whose start is positioned after its end on the surface', async () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
@@ -564,8 +552,8 @@ describe('surface invariants', () => {
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'b' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 3
|
||||
// Reversed range: start seq 3 is at a later surface position than end seq 2.
|
||||
expect(() => {
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 3, end: 2 }, sourceEventSeqs: [2, 3] })
|
||||
}).toThrow(/is after end seq 2 .* on the surface/)
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 3, end: 2 }, sourceEventSeqs: [2, 3] })
|
||||
}).toThrow(/is after end seq 2/)
|
||||
})
|
||||
|
||||
it('rejects a replace whose sourceEventSeqs omits a shadowed surface node', async () => {
|
||||
@@ -577,7 +565,7 @@ describe('surface invariants', () => {
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'b' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 3
|
||||
// Replace shadows surface nodes [2, 3] but records provenance for only [2].
|
||||
expect(() => {
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [{ type: 'text', text: 'sum' }] }, { surfaceOp: { op: 'replace', start: 2, end: 3 }, sourceEventSeqs: [2] })
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [{ type: 'text', text: 'sum' }] }, { surfaceOp: { op: 'replace', start: 2, end: 3 }, sourceEventSeqs: [2] })
|
||||
}).toThrow(/must include every shadowed surface node; missing 3/)
|
||||
})
|
||||
|
||||
@@ -589,7 +577,7 @@ describe('surface invariants', () => {
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 2
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'b' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 3
|
||||
expect(() => {
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [{ type: 'text', text: 'sum' }] }, { surfaceOp: { op: 'replace', start: 2, end: 3 }, sourceEventSeqs: [2, 3] })
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [{ type: 'text', text: 'sum' }] }, { surfaceOp: { op: 'replace', start: 2, end: 3 }, sourceEventSeqs: [2, 3] })
|
||||
}).not.toThrow()
|
||||
})
|
||||
|
||||
@@ -601,8 +589,8 @@ describe('surface invariants', () => {
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 2
|
||||
// seq 1 (step/start) is a real earlier event but never entered the surface.
|
||||
expect(() => {
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 1, end: 2 }, sourceEventSeqs: [1, 2] })
|
||||
}).toThrow(/start seq 1 is not on the surface/)
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 1, end: 2 }, sourceEventSeqs: [1, 2] })
|
||||
}).toThrow(/start seq 1 not found in surface/)
|
||||
})
|
||||
|
||||
it('rejects a replace naming an end seq that is not on the surface', async () => {
|
||||
@@ -613,8 +601,8 @@ describe('surface invariants', () => {
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 2
|
||||
// start (2) is on the surface but end (99) never entered it.
|
||||
expect(() => {
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 99 }, sourceEventSeqs: [2] })
|
||||
}).toThrow(/end seq 99 is not on the surface/)
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 99 }, sourceEventSeqs: [2] })
|
||||
}).toThrow(/end seq 99 not found in surface/)
|
||||
})
|
||||
|
||||
it('rejects a replace whose range is reversed in surface position after a prior replace reordered it', async () => {
|
||||
@@ -625,13 +613,13 @@ describe('surface invariants', () => {
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 2
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'b' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 3
|
||||
// Replace node 2 (position 0) with seq 4 — surface is now [4, 3], so seq 4
|
||||
// precedes seq 3 in linked-list order even though 4 > 3 numerically.
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [{ type: 'text', text: 's' }] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] }) // seq 4
|
||||
// precedes seq 3 in surface order even though 4 > 3 numerically.
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [{ type: 'text', text: 's' }] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] }) // seq 4
|
||||
// A replace with start=3, end=4 passes the seq check (3 <= 4) but is
|
||||
// reversed positionally (3 is at pos 1, 4 is at pos 0).
|
||||
expect(() => {
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 3, end: 4 }, sourceEventSeqs: [3, 4] }) // seq 5
|
||||
}).toThrow(/is after end seq 4 .* on the surface/)
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 3, end: 4 }, sourceEventSeqs: [3, 4] }) // seq 5
|
||||
}).toThrow(/is after end seq 4/)
|
||||
})
|
||||
|
||||
it('accepts a replace whose start seq exceeds its end seq when the surface position order is valid', async () => {
|
||||
@@ -645,9 +633,9 @@ describe('surface invariants', () => {
|
||||
// head seq (4) is numerically GREATER than the tail seq (3): the surface is
|
||||
// not seq-ordered. A replace spanning start=4 (pos 0) … end=3 (pos 1) is
|
||||
// valid positionally and must be accepted even though start seq > end seq.
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [{ type: 'text', text: 's' }] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] }) // seq 4
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [{ type: 'text', text: 's' }] }, { surfaceOp: { op: 'replace', start: 2, end: 2 }, sourceEventSeqs: [2] }) // seq 4
|
||||
expect(() => {
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 4, end: 3 }, sourceEventSeqs: [4, 3] }) // seq 5
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 4, end: 3 }, sourceEventSeqs: [4, 3] }) // seq 5
|
||||
}).not.toThrow()
|
||||
})
|
||||
|
||||
@@ -659,7 +647,7 @@ describe('surface invariants', () => {
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'a' }], source: { kind: 'user' } }, { surfaceOp: 'append' }) // seq 2
|
||||
// A replace with no sourceEventSeqs records no provenance for the node it shadows.
|
||||
expect(() => {
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 2 } })
|
||||
session.append('assistant/message', { provenance: { provider: 'mock', model: 'mock' }, turn: 1, step: 1, content: [] }, { surfaceOp: { op: 'replace', start: 2, end: 2 } })
|
||||
}).toThrow(/must include every shadowed surface node; missing 2/)
|
||||
})
|
||||
|
||||
@@ -670,30 +658,11 @@ describe('surface invariants', () => {
|
||||
{ type: 'step/start' as const, seq: 1, time: 0, data: { turn: 1, step: 1 } },
|
||||
{ type: 'user/message' as const, seq: 2, time: 0, data: { content: [{ type: 'text' as const, text: 'a' }], source: { kind: 'user' as const } }, surfaceOp: 'append' as const },
|
||||
{ type: 'user/message' as const, seq: 3, time: 0, data: { content: [{ type: 'text' as const, text: 'b' }], source: { kind: 'user' as const } }, surfaceOp: 'append' as const },
|
||||
{ type: 'assistant/message' as const, seq: 4, time: 0, data: { turn: 1, step: 1, content: [{ type: 'text' as const, text: 'sum' }] }, surfaceOp: { op: 'replace' as const, start: 2, end: 3 }, sourceEventSeqs: [2] },
|
||||
{ type: 'assistant/message' as const, seq: 4, time: 0, data: { turn: 1, step: 1, content: [{ type: 'text' as const, text: 'sum' }], provenance: { provider: 'mock', model: 'mock' } }, surfaceOp: { op: 'replace' as const, start: 2, end: 3 }, sourceEventSeqs: [2] },
|
||||
]
|
||||
expect(() => ctx.sessions.create(undefined, { seed: badSeed })).toThrow(/must include every shadowed surface node; missing 3/)
|
||||
})
|
||||
|
||||
it('rejects sourceEventSeqs on a non-surface event', async () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
// Session rejects this at its own acceptance boundary. Emit a hand-built
|
||||
// record to cover the listener's defensive check for alternate producers.
|
||||
const event = { type: 'turn/end', seq: 1, time: 1, data: { turn: 1, reason: { kind: 'completed' } }, sourceEventSeqs: [0] }
|
||||
expect(() => { ctx.emit(scopeTarget(session, undefined), 'session/event', session, event as never) })
|
||||
.toThrow(/cannot carry sourceEventSeqs/)
|
||||
})
|
||||
|
||||
it('rejects surfaceOp on a non-surface event', async () => {
|
||||
const { ctx } = await setup()
|
||||
const session = ctx.sessions.create()
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
const event = { type: 'turn/end', seq: 1, time: 1, data: { turn: 1, reason: { kind: 'completed' } }, surfaceOp: 'append' }
|
||||
expect(() => { ctx.emit(scopeTarget(session, undefined), 'session/event', session, event as never) })
|
||||
.toThrow(/cannot carry surfaceOp/)
|
||||
})
|
||||
})
|
||||
|
||||
describe('request-reconstruction cross-check (llm/stream)', () => {
|
||||
@@ -705,7 +674,7 @@ describe('request-reconstruction cross-check (llm/stream)', () => {
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
|
||||
const boundary = session.deriveMessages()
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
session.append('request/header', { header: { config: { model: 'm' } }, reason: 'initial' })
|
||||
session.append('request/header', { header: { config: { provider: 'mock', model: 'm' } }, reason: 'initial' })
|
||||
return { ctx, session, boundary }
|
||||
}
|
||||
|
||||
@@ -735,7 +704,7 @@ describe('request-reconstruction cross-check (llm/stream)', () => {
|
||||
it('expects the folded header\'s session prefix ahead of the derivation (prefix + derived)', async () => {
|
||||
const { ctx, session, boundary } = await requestSetup()
|
||||
const prefix = { role: 'user' as const, content: [{ type: 'text' as const, text: '<system-reminder>catalog</system-reminder>' }] }
|
||||
session.append('request/header-delta', { messagePrefix: [prefix] })
|
||||
session.append('request/header', { header: { config: { provider: 'mock', model: 'm' }, messagePrefix: [prefix] }, reason: 'change' })
|
||||
// The prefixed request matches the fold…
|
||||
const prefixed = Object.freeze({ model: 'm', messages: Object.freeze([prefix, ...boundary]), sessionId: session.id })
|
||||
expect(() => { dispatch(ctx, prefixed) }).not.toThrow()
|
||||
@@ -804,7 +773,7 @@ describe('request cross-check ordering (prepend)', () => {
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
session.append('request/header', { header: { config: { model: 'm' } }, reason: 'initial' })
|
||||
session.append('request/header', { header: { config: { provider: 'mock', model: 'm' } }, reason: 'initial' })
|
||||
|
||||
const divergent = Object.freeze({
|
||||
model: 'm',
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# @deepseek-ai/dsh-llm-replay
|
||||
|
||||
A replay LLM plugin for keyless snapshot tests. It installs a single `llm/stream` waterfall listener that short-circuits the waterfall (never calls `next()`) and yields model streams reconstructed from a recorded **session JSONL** fixture — so a test can boot the real agent against a fixed model transcript with no API key.
|
||||
A replay LLM plugin for keyless snapshot tests. It yields model streams reconstructed from a recorded **session JSONL** fixture, so a test can boot the real agent against a fixed model transcript with no API key. With `providers` configured it registers a replay-only adapter whose catalog is visible to clients such as ACP editors; without `providers` it installs the catch-all `llm/stream` waterfall used by tests that do not need discovery.
|
||||
|
||||
Its consumer is the ACP snapshot harness in `examples/acp-agent`, which loads this plugin (via `cordis.snapshot.yml`) in place of a real LLM adapter. The package exists so its derive/parse/replay logic falls under the per-file 100% coverage gate on `packages/*/src` (the same logic, while it lived under `examples/`, was outside the gate).
|
||||
|
||||
@@ -23,10 +23,18 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
|
||||
| `file` | string | `$DSH_SNAPSHOT_FILE` | Path to the primary (parent) `session.jsonl` fixture. Required (config or env). |
|
||||
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | Optional path to a `ReplayEntry[]` sidecar that replaces the PRIMARY session's derived script. |
|
||||
| `childFiles` | string[] | `$DSH_SNAPSHOT_CHILD_FILES` (path-delimited) | Recorded subagent child-session logs for a nested scenario; empty for a single-session scenario. |
|
||||
| `providers` | `ReplayProviderConfig[]` | — | Optional replay-only provider and model catalog. Configured routes dispatch through the replay adapter and never perform provider I/O. |
|
||||
|
||||
```yaml
|
||||
- id: llm-replay
|
||||
name: '@deepseek-ai/dsh-llm-replay'
|
||||
config:
|
||||
providers:
|
||||
- id: deepseek
|
||||
name: DeepSeek
|
||||
models:
|
||||
- id: deepseek-v4-flash
|
||||
- id: deepseek-v4-pro
|
||||
# file/overrideFile/childFiles default to $DSH_SNAPSHOT_FILE /
|
||||
# $DSH_SNAPSHOT_OVERRIDE / $DSH_SNAPSHOT_CHILD_FILES, set by the snapshot
|
||||
# harness per scenario.
|
||||
@@ -34,11 +42,11 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
|
||||
|
||||
## Exports
|
||||
|
||||
- `installLlmReplay(ctx, config)` — install the `llm/stream` listener; returns the disposer (HMR safety). Use this in tests to drive replay without the Loader or env vars.
|
||||
- `installLlmReplay(ctx, config)` — install the configured replay adapter or catch-all `llm/stream` listener; returns the disposer (HMR safety). Use this in tests to drive replay without the Loader or env vars.
|
||||
- `loadSessionScripts(config)` — resolve the ordered `SessionScript[]` (primary + children) for a scenario, ready to bind to live sessions in first-call order.
|
||||
- `loadReplayScript(config)` — resolve the `ReplayEntry[]` for the PRIMARY session only (sidecar override if present, else derived from the JSONL; fail-loud if the fixture is missing).
|
||||
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)` — the pure helpers that turn a recorded session log into a script and read its header `id`/`createdAt`. A derived group must end in a `finish` chunk; a group without one is the fingerprint of a thrown `stream()` and must instead be expressed via an override sidecar.
|
||||
- Types `ReplayEntry` / `SessionScript` / `ReplayConfig` / `Config`.
|
||||
- Types `ReplayEntry` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `Config`.
|
||||
|
||||
## Plugin export shape
|
||||
|
||||
|
||||
@@ -10,8 +10,8 @@ import { existsSync, readFileSync } from 'node:fs'
|
||||
import { delimiter as pathDelimiter } from 'node:path'
|
||||
import type { Context } from 'cordis'
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import type { GenerateOptions, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import { LlmError, assertNever } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, LlmModelInfo, LlmProviderInfo, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import { LlmAdapter, LlmError, assertNever } from '@deepseek-ai/dsh-llm'
|
||||
|
||||
/**
|
||||
* One recorded model call. `throw` may replay prefix chunks before failing;
|
||||
@@ -23,6 +23,26 @@ export type ReplayEntry =
|
||||
| { kind: 'throw'; chunks: StreamChunk[]; message: string; code: string }
|
||||
| { kind: 'hang' }
|
||||
|
||||
/** One model exposed by a replay-only provider catalog. */
|
||||
export interface ReplayModelConfig {
|
||||
/** Model id used for replay requests. */
|
||||
id: string
|
||||
/** Selector label; defaults to {@link id}. */
|
||||
name?: string
|
||||
/** Optional selector description. */
|
||||
description?: string
|
||||
}
|
||||
|
||||
/** One provider route exposed by the replay adapter. */
|
||||
export interface ReplayProviderConfig {
|
||||
/** Provider route used for replay requests. */
|
||||
id: string
|
||||
/** Selector label; defaults to {@link id}. */
|
||||
name?: string
|
||||
/** Advisory models exposed to clients such as ACP editors. */
|
||||
models?: ReplayModelConfig[]
|
||||
}
|
||||
|
||||
/** Resolved plugin configuration. */
|
||||
export interface ReplayConfig {
|
||||
/**
|
||||
@@ -45,6 +65,12 @@ export interface ReplayConfig {
|
||||
* for a single-session scenario.
|
||||
*/
|
||||
childFiles?: string[]
|
||||
/**
|
||||
* Optional provider catalog. When non-empty, replay registers an adapter for
|
||||
* these routes; when absent or empty, it retains the catch-all waterfall used
|
||||
* by tests that do not need discovery.
|
||||
*/
|
||||
providers?: ReplayProviderConfig[]
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -203,6 +229,42 @@ export function loadSessionScripts(config: ReplayConfig): SessionScript[] {
|
||||
return [primary, ...children]
|
||||
}
|
||||
|
||||
/** Replay adapter that makes a configured provider catalog discoverable without provider I/O. */
|
||||
class ReplayAdapter extends LlmAdapter {
|
||||
private readonly providers: ReadonlyMap<string, ReplayProviderConfig>
|
||||
|
||||
constructor(
|
||||
providers: readonly ReplayProviderConfig[],
|
||||
private readonly replay: (options: GenerateOptions) => AsyncIterable<StreamChunk>,
|
||||
) {
|
||||
super()
|
||||
this.providers = new Map(providers.map(provider => [provider.id, provider]))
|
||||
}
|
||||
|
||||
override providerInfo(provider: string): LlmProviderInfo {
|
||||
const configured = this.providers.get(provider)
|
||||
/* v8 ignore next -- LlmService only asks about routes registered from this same map. */
|
||||
if (configured === undefined) return super.providerInfo(provider)
|
||||
return { id: provider, name: configured.name ?? provider }
|
||||
}
|
||||
|
||||
override listModels(provider: string): Promise<readonly LlmModelInfo[]> {
|
||||
const configured = this.providers.get(provider)
|
||||
/* v8 ignore next -- LlmService only asks about routes registered from this same map. */
|
||||
if (configured === undefined) return Promise.resolve([])
|
||||
return Promise.resolve((configured.models ?? []).map(model => ({
|
||||
provider,
|
||||
id: model.id,
|
||||
name: model.name ?? model.id,
|
||||
...model.description === undefined ? {} : { description: model.description },
|
||||
})))
|
||||
}
|
||||
|
||||
override stream(options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
return this.replay(options)
|
||||
}
|
||||
}
|
||||
|
||||
/** Yield a recorded stream back, honoring abort like a real adapter. */
|
||||
async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined): AsyncIterable<StreamChunk> {
|
||||
switch (entry.kind) {
|
||||
@@ -243,12 +305,14 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined)
|
||||
/**
|
||||
* Install per-session positional replay. A newly seen live session takes the
|
||||
* next ordered recorded script, then advances its own cursor synchronously at
|
||||
* invocation time; calls without `sessionId` share one anonymous session.
|
||||
* Returns the effect disposer for HMR-safe removal.
|
||||
* invocation time; calls without `sessionId` share one anonymous session. A
|
||||
* non-empty provider catalog registers a routed replay adapter; otherwise a
|
||||
* catch-all waterfall intercepts requests. Returns the effect disposer for
|
||||
* HMR-safe removal.
|
||||
*
|
||||
* @param ctx - the context whose `llm/stream` waterfall the listener short-circuits.
|
||||
* @param ctx - the context whose LLM service receives the replay route or waterfall.
|
||||
* @param config - the resolved fixture paths (env-var defaulting is `apply`'s job).
|
||||
* @returns the `ctx.on` disposer that removes the listener.
|
||||
* @returns the disposer that removes the registered adapter or listener.
|
||||
*/
|
||||
export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void {
|
||||
const scripts = loadSessionScripts(config)
|
||||
@@ -258,7 +322,7 @@ export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void
|
||||
const bound = new Map<string, { entries: ReplayEntry[]; cursor: number }>()
|
||||
let nextScript = 0
|
||||
const ANON = '\0anon\0' // the key for a call that carries no sessionId
|
||||
return ctx.on('llm/stream', (options: GenerateOptions, _next) => {
|
||||
const replay = (options: GenerateOptions): AsyncIterable<StreamChunk> => {
|
||||
const key = options.sessionId ?? ANON
|
||||
let state = bound.get(key)
|
||||
let unrecorded = false
|
||||
@@ -296,7 +360,12 @@ export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void
|
||||
}
|
||||
yield* replayEntry(entry, options.signal)
|
||||
})()
|
||||
})
|
||||
}
|
||||
const providers = config.providers ?? []
|
||||
if (providers.length > 0) {
|
||||
return ctx.llm.registerAdapter(providers.map(provider => provider.id), new ReplayAdapter(providers, replay))
|
||||
}
|
||||
return ctx.on('llm/stream', (options: GenerateOptions, _next) => replay(options))
|
||||
}
|
||||
|
||||
export const name = 'llm-replay'
|
||||
@@ -314,6 +383,8 @@ export interface Config {
|
||||
* a nested-agent scenario; absent/empty for a single-session scenario.
|
||||
*/
|
||||
childFiles?: string[]
|
||||
/** Optional replay-only provider catalog; absent or empty selects catch-all waterfall replay. */
|
||||
providers?: ReplayProviderConfig[]
|
||||
}
|
||||
|
||||
export function apply(ctx: Context, config: Config = {}): void {
|
||||
@@ -329,5 +400,6 @@ export function apply(ctx: Context, config: Config = {}): void {
|
||||
file,
|
||||
...overrideFile !== undefined && overrideFile.length > 0 ? { overrideFile } : {},
|
||||
...childFiles.length > 0 ? { childFiles } : {},
|
||||
...config.providers !== undefined ? { providers: config.providers } : {},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -198,7 +198,7 @@ describe('loadReplayScript', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('installLlmReplay (through the real waterfall)', () => {
|
||||
describe('installLlmReplay (through the real LlmService)', () => {
|
||||
function writeLog(...calls: StreamChunk[][]): void {
|
||||
let seq = 1
|
||||
const events: SessionEvent[] = []
|
||||
@@ -214,7 +214,41 @@ describe('installLlmReplay (through the real waterfall)', () => {
|
||||
await ctx.plugin(LlmService)
|
||||
// No adapter registered for 'm' — replay must not reach it.
|
||||
installLlmReplay(ctx, { file })
|
||||
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
})
|
||||
|
||||
it('registers a replay-only provider catalog when configured', async () => {
|
||||
writeLog(TEXT_CHUNKS)
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
const dispose = installLlmReplay(ctx, {
|
||||
file,
|
||||
providers: [
|
||||
{
|
||||
id: 'deepseek',
|
||||
name: 'DeepSeek',
|
||||
models: [
|
||||
{ id: 'flash' },
|
||||
{ id: 'pro', name: 'Pro', description: 'Larger model' },
|
||||
],
|
||||
},
|
||||
{ id: 'empty' },
|
||||
],
|
||||
})
|
||||
|
||||
expect(ctx.llm.listProviders()).toEqual([
|
||||
{ id: 'deepseek', name: 'DeepSeek' },
|
||||
{ id: 'empty', name: 'empty' },
|
||||
])
|
||||
await expect(ctx.llm.listModels('deepseek')).resolves.toEqual([
|
||||
{ provider: 'deepseek', id: 'flash', name: 'flash' },
|
||||
{ provider: 'deepseek', id: 'pro', name: 'Pro', description: 'Larger model' },
|
||||
])
|
||||
await expect(ctx.llm.listModels('empty')).resolves.toEqual([])
|
||||
expect(await drain(ctx.llm.stream({ provider: 'deepseek', model: 'pro', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
|
||||
dispose()
|
||||
expect(ctx.llm.listProviders()).toEqual([])
|
||||
})
|
||||
|
||||
it('serves the Nth call the Nth derived entry (positional)', async () => {
|
||||
@@ -227,8 +261,8 @@ describe('installLlmReplay (through the real waterfall)', () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
installLlmReplay(ctx, { file })
|
||||
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(second)
|
||||
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(second)
|
||||
})
|
||||
|
||||
it('replays a sidecar throw-entry as an LlmError with its stable code, after its prefix chunks', async () => {
|
||||
@@ -244,7 +278,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
|
||||
|
||||
const seen: StreamChunk[] = []
|
||||
await expect((async () => {
|
||||
for await (const c of ctx.llm.stream({ model: 'm', messages: [] })) seen.push(c)
|
||||
for await (const c of ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })) seen.push(c)
|
||||
})()).rejects.toMatchObject({ message: 'unauthorized', code: 'AUTH' })
|
||||
expect(seen).toEqual(partial)
|
||||
})
|
||||
@@ -258,7 +292,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
|
||||
installLlmReplay(ctx, { file, overrideFile })
|
||||
|
||||
const controller = new AbortController()
|
||||
const iterator = ctx.llm.stream({ model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
|
||||
const iterator = ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
|
||||
// Deterministically consume the two pre-hang chunks (no sleep), then abort
|
||||
// and assert the next pull rejects — event-driven, per the no-sleeps rule.
|
||||
expect((await iterator.next()).value).toMatchObject({ type: 'block-start' })
|
||||
@@ -272,8 +306,8 @@ describe('installLlmReplay (through the real waterfall)', () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
installLlmReplay(ctx, { file })
|
||||
await drain(ctx.llm.stream({ model: 'm', messages: [] }))
|
||||
await expect(drain(ctx.llm.stream({ model: 'm', messages: [] }))).rejects.toThrow(/exhausted/)
|
||||
await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))
|
||||
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).rejects.toThrow(/exhausted/)
|
||||
})
|
||||
|
||||
it('aborts mid-replay when the signal is already set', async () => {
|
||||
@@ -283,7 +317,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
|
||||
installLlmReplay(ctx, { file })
|
||||
const controller = new AbortController()
|
||||
controller.abort()
|
||||
await expect(drain(ctx.llm.stream({ model: 'm', messages: [], signal: controller.signal })))
|
||||
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal })))
|
||||
.rejects.toThrow('aborted')
|
||||
})
|
||||
|
||||
@@ -305,11 +339,11 @@ describe('installLlmReplay (through the real waterfall)', () => {
|
||||
}, { inject: ['llm'] }))
|
||||
|
||||
// While installed, replay short-circuits to the derived fixture ('hi').
|
||||
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
|
||||
await fiber.dispose()
|
||||
// After dispose the listener is gone; the call reaches the real adapter.
|
||||
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] })))
|
||||
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })))
|
||||
.toEqual([{ type: 'finish', reason: { kind: 'stop' } }])
|
||||
})
|
||||
|
||||
@@ -321,7 +355,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
installLlmReplay(ctx, { file, overrideFile })
|
||||
await expect(drain(ctx.llm.stream({ model: 'm', messages: [] })))
|
||||
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })))
|
||||
.rejects.toThrow(/llm-replay replay entry/)
|
||||
})
|
||||
|
||||
@@ -333,7 +367,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
|
||||
await ctx.plugin(LlmService)
|
||||
installLlmReplay(ctx, { file, overrideFile })
|
||||
const controller = new AbortController()
|
||||
const iterator = ctx.llm.stream({ model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
|
||||
const iterator = ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
|
||||
// Consume the two pre-hang chunks, then start the third pull so the generator
|
||||
// is parked inside the await (signal NOT yet aborted — exercises the
|
||||
// addEventListener('abort') registration), and only THEN abort.
|
||||
@@ -359,7 +393,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
|
||||
controller.abort()
|
||||
// Already aborted: the throw-entry's prefix loop surfaces 'aborted' before
|
||||
// it can reach the recorded LlmError.
|
||||
await expect(drain(ctx.llm.stream({ model: 'm', messages: [], signal: controller.signal })))
|
||||
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal })))
|
||||
.rejects.toThrow('aborted')
|
||||
})
|
||||
|
||||
@@ -373,7 +407,7 @@ describe('installLlmReplay (through the real waterfall)', () => {
|
||||
const controller = new AbortController()
|
||||
controller.abort()
|
||||
// The two pre-hang chunks still flow; the abort surfaces at the await.
|
||||
const iterator = ctx.llm.stream({ model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
|
||||
const iterator = ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal })[Symbol.asyncIterator]()
|
||||
await iterator.next()
|
||||
await iterator.next()
|
||||
await expect(iterator.next()).rejects.toThrow('aborted')
|
||||
@@ -503,7 +537,7 @@ describe('installLlmReplay (per-session keying)', () => {
|
||||
]
|
||||
|
||||
const live = (id: string): GenerateOptions =>
|
||||
({ model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
|
||||
({ provider: 'm', model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
|
||||
|
||||
it('routes each live session to its own script by FIRST-CALL order', async () => {
|
||||
const parentFile = writeSession('session.jsonl', { id: 'rec-parent', createdAt: 100 }, [TEXT_CHUNKS])
|
||||
@@ -540,7 +574,7 @@ describe('installLlmReplay (per-session keying)', () => {
|
||||
await ctx.plugin(LlmService)
|
||||
installLlmReplay(ctx, { file: parentFile })
|
||||
// No sessionId at all — the legacy single-session path.
|
||||
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
})
|
||||
|
||||
it('fails loud when more distinct live sessions call than were recorded', async () => {
|
||||
@@ -574,12 +608,13 @@ describe('apply (the plugin entry)', () => {
|
||||
expect(inject).toEqual(['llm'])
|
||||
})
|
||||
|
||||
it('installs replay from an explicit config.file', async () => {
|
||||
it('installs replay and its catalog from explicit config', async () => {
|
||||
writeFileSync(file, sessionJsonl(TEXT_CHUNKS.map((c, i) => chunkEvent(i + 1, 1, 1, c))), 'utf8')
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
apply(ctx, { file })
|
||||
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
apply(ctx, { file, providers: [{ id: 'm', models: [{ id: 'm' }] }] })
|
||||
expect(ctx.llm.listProviders()).toEqual([{ id: 'm', name: 'm' }])
|
||||
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
})
|
||||
|
||||
it('falls back to $DSH_SNAPSHOT_FILE / $DSH_SNAPSHOT_OVERRIDE when config is empty', async () => {
|
||||
@@ -591,7 +626,7 @@ describe('apply (the plugin entry)', () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
apply(ctx)
|
||||
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
})
|
||||
|
||||
it('uses only the file when no override path is configured or in the env', async () => {
|
||||
@@ -601,7 +636,7 @@ describe('apply (the plugin entry)', () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
apply(ctx)
|
||||
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
})
|
||||
|
||||
it('throws when no fixture path is given by config or env', async () => {
|
||||
@@ -631,7 +666,7 @@ describe('apply (the plugin entry)', () => {
|
||||
await ctx.plugin(LlmService)
|
||||
apply(ctx, { file, childFiles: [childFile] })
|
||||
const live = (id: string): GenerateOptions =>
|
||||
({ model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
|
||||
({ provider: 'm', model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
|
||||
expect(await drain(ctx.llm.stream(live('A')))).toEqual(TEXT_CHUNKS)
|
||||
expect(await drain(ctx.llm.stream(live('B')))).toEqual(childSecond)
|
||||
})
|
||||
@@ -651,7 +686,7 @@ describe('apply (the plugin entry)', () => {
|
||||
await ctx.plugin(LlmService)
|
||||
apply(ctx)
|
||||
const live = (id: string): GenerateOptions =>
|
||||
({ model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
|
||||
({ provider: 'm', model: 'm', messages: [], sessionId: id as NonNullable<GenerateOptions['sessionId']> })
|
||||
expect(await drain(ctx.llm.stream(live('A')))).toEqual(TEXT_CHUNKS)
|
||||
expect(await drain(ctx.llm.stream(live('B')))).toEqual(childChunks)
|
||||
})
|
||||
@@ -663,6 +698,6 @@ describe('apply (the plugin entry)', () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
apply(ctx)
|
||||
expect(await drain(ctx.llm.stream({ model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
})
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user