Merge pull request #1156 from deepseek-harness/fix/goal-complete-wrapup

fix(tool-goal): deliver a user-facing wrap-up message after goal-round complete/blocked
This commit is contained in:
Ziya
2026-08-02 11:03:22 -04:00
committed by GitHub
32 changed files with 549 additions and 53 deletions

View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write packages/core/tools/README.md
README.md: 15fc5839a3b0e3fa2d20c5a9cc50577e9807ffda
README.zh.md: 8547ee4a796dcd93945dfa40373c14c10d7d0c8a
README.md: 80ea3cc93437d48a7ea0ffba0ff4d2ef2407755f
README.zh.md: 1f0791c5df7afd4a3479afdd827c4fc148cf8883

View File

@@ -43,7 +43,7 @@ The live registry pipeline has three transformable waterfalls, then the definiti
- `ToolExecutionInput` — the caller-supplied call description: `{ callId, name, arguments, signal, agent?, parent? }`; `signal` is required and readonly, callers may pass an enclosing execution's opaque token as `parent`, and callers never choose the new execution's own token.
- `ToolExecutionToken` — a fresh branded `Symbol` assigned by the registry. It supports equality correlation only and never crosses a model, log, or worker boundary.
- `ToolExecution` — the readonly pipeline view: immutable `{ token, callId, name, arguments, signal, agent?, parent? }`; the registry separately retains and re-fuses the original caller signal. `ToolDispatchExecution` is the `tools/execute`-only view whose required signal is mutable, so a wrapper may replace and restore it but cannot delete it. A nested call's `parent` is a `ToolExecutionToken`, not an execution object.
- `ToolRunContext` — the execution passed to a tool body, extending `ToolExecution` with `deferContext(context)`. Composite tools use it to ferry context produced by nested dispatches to the outer result even when the tool later throws or cancellation wins; it never injects immediately.
- `ToolRunContext` — the execution passed to a tool body, extending `ToolExecution` with `deferContext(context)`. It defers one context until the tool's final result reaches the loop — typically a nested-dispatch context ferried by a composite tool, or a fresh plugin-sourced instruction minted by a leaf tool (`tool-goal`'s wrap-up) — even when the tool later throws or cancellation wins; it never injects immediately.
- `ToolExecutionResult` — discriminated execution-local outcome. Success is `{ isError:false, value:JsonValue, content, meta?, additionalContexts? }`; failure is `{ isError:true, error:{ message, info? }, content, meta?, additionalContexts? }` and has no value. Call identity stays on the immutable `ToolExecution`. The registry snapshots, validates, and freezes the canonical value before rendering, then materializes the durable presentation fields before final observation. `ToolFailure.info` carries an internal `{ name, code }` for a `HarnessError`; `additionalContexts` preserves every deferred or post-execute identified `UserMessage` for the loop's post-result FIFO.
- `PreToolDecision``{kind:'allow'}` | `{kind:'deny', reason}` | `{kind:'ask', reason?}`. Input rewrite is deliberately not offered; `ask` is serviced by [`ctx.approval`](../../ui/user-approval/README.md) when mounted and otherwise degrades to deny.
- `PostToolDecision` — accept may replace `content` or `value`, never both, and may attach `additionalContexts`; block turns feedback into a valueless failure. Content replacement preserves the canonical value and metadata. Value replacement is revalidated and rerenders content/metadata. Accept preserves tool-deferred contexts before decision contexts; block discards tool-deferred contexts and exposes only contexts explicitly supplied by the blocking decision.

View File

@@ -43,7 +43,7 @@ tools:
- `ToolExecutionInput`:调用方提供的调用描述:`{ callId, name, arguments, signal, agent?, parent? }``signal` 必填且只读,调用方可以将外层执行的不透明 token 作为 `parent` 传入,但绝不能选择新执行自身的 token。
- `ToolExecutionToken`:注册表分配的全新带品牌 `Symbol`。它只支持通过相等性进行关联,绝不会跨越模型、日志或 worker 边界。
- `ToolExecution`:只读流水线视图:不可变的 `{ token, callId, name, arguments, signal, agent?, parent? }`;注册表会另行保留并重新融合调用方的原始信号。`ToolDispatchExecution` 是仅供 `tools/execute` 使用的视图,其必填信号可变,因此包装层可以替换并还原它,但不能删除它。嵌套调用的 `parent``ToolExecutionToken`,而不是执行对象。
- `ToolRunContext`:传给工具主体的执行上下文,在 `ToolExecution` 基础上增加 `deferContext(context)`。组合工具借此把嵌套分发产生的上下文传递到外层结果,即使工具后来抛出或取消胜出也不例外;该方法绝不会立即注入上下文。
- `ToolRunContext`:传给工具主体的执行上下文,在 `ToolExecution` 基础上增加 `deferContext(context)`它把一条上下文推迟到该工具的最终结果抵达循环时——通常是组合工具转运的嵌套分发上下文,也可以是叶子工具铸造的全新插件来源指令(如 `tool-goal` 的收尾注入)——即使工具后来抛出或取消胜出也不例外;该方法绝不会立即注入上下文。
- `ToolExecutionResult`:可辨识的执行局部结果。成功形态为 `{ isError:false, value:JsonValue, content, meta?, additionalContexts? }`;失败形态为 `{ isError:true, error:{ message, info? }, content, meta?, additionalContexts? }`,且不含值。调用身份保留在不可变的 `ToolExecution` 上。注册表会在呈现前快照、验证并冻结规范值,随后在最终观测前实体化持久呈现字段。`ToolFailure.info` 携带内部的 `{ name, code }`,用于表示 `HarnessError``additionalContexts` 会保留每个通过延迟或 post-execute 加入且带标识的 `UserMessage`,供循环在结果后按 FIFO 顺序处理。
- `PreToolDecision``{kind:'allow'}` | `{kind:'deny', reason}` | `{kind:'ask', reason?}`。该类型有意不提供输入改写;`ask` 在挂载 [`ctx.approval`](../../ui/user-approval/README.md) 时由它处理,否则退化为拒绝。
- `PostToolDecision`:接受决定可以替换 `content``value`(不能同时替换),并可附加 `additionalContexts`;阻止决定会把反馈变成无值失败。替换内容会保留规范值和元数据。替换值会重新验证,并重新呈现内容/元数据。接受决定会先保留工具延迟的上下文,再附加决定上下文;阻止决定会丢弃工具延迟的上下文,只公开阻止决定显式提供的上下文。

View File

@@ -344,15 +344,18 @@ export interface ToolDispatchExecution extends Omit<ToolExecution, 'signal'> {
/**
* Runtime context handed to a tool implementation after the registry has
* accepted a {@link ToolExecution}. A composite tool uses
* {@link deferContext} to ferry context produced by nested dispatches back to
* the outer result; the loop appends it only after the outer `tool/result`.
* accepted a {@link ToolExecution}. {@link deferContext} attaches context to
* this execution's own result — a composite tool ferries nested-dispatch
* context back to the outer result, and a leaf tool may mint a fresh
* plugin-sourced instruction; the loop appends it only after the
* `tool/result`.
*/
export interface ToolRunContext extends ToolExecution {
/**
* Defer one nested-dispatch context until this tool's final result reaches
* the agent loop. Contexts retain their individual source and metadata and
* are emitted in call order.
* Defer one context — typically a nested-dispatch context ferried by a
* composite tool, or a fresh plugin-sourced instruction — until this tool's
* final result reaches the agent loop. Contexts retain their individual
* source and metadata and are emitted in call order.
*/
deferContext(context: UserMessage): void
/**

View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write packages/goal/tool-goal/README.md
README.md: aaed61dd517aeb2f94efa22c34c64d1068155d46
README.zh.md: 5365b64ef65fb3d3f00e19357327479ebd8285a8
README.md: 2fa80c2e5fa3d675a48fc18506635fd811ac8f80
README.zh.md: c6c39e3cc739fb39a4a36080db5246e7c7349147

View File

@@ -14,7 +14,7 @@ All calls are exclusive, so a model-ordered batch observes earlier mutations and
All three canonical values match the compact JSON already rendered to Native callers: `{ goal: null }` or `{ goal: { id, revision, objective, phase, roundsStarted, maxGoalRounds, blockedReason? }, activation }`. Programmatic consumers therefore receive the same domain structure without parsing the rendered JSON.
An autonomous goal round that successfully reports `complete` or `blocked` marks that tool execution with `concludeTurn()` so the physical turn stops after the step. Direct-human mutations never contribute this stop: the assistant may acknowledge the change and concurrent human steering remains available to the loop.
An autonomous goal round that successfully reports `complete` or `blocked` defers one wrap-up context onto that tool result: an injected instruction telling the model to write a final closing message to the user and call no more tools, after which the turn ends through the ordinary no-tool-calls stop. Direct-human mutations receive no instruction: the assistant may acknowledge the change and concurrent human steering remains available to the loop.
## Authority
@@ -61,11 +61,11 @@ Prefix-stable while the plugin scope, configured threshold, and guidance text ar
#### What the model sees
The generated [`get_goal`, `create_goal`, and `update_goal` schemas](../../../docs/tool-catalog.md#deepseek-aidsh-tool-goal). Successful results are compact JSON. Mutation results are followed by the goal domain's raw `<goal_state>` snapshot after the tool batch. `activation` in a result is a live observation and never becomes replay authority.
The generated [`get_goal`, `create_goal`, and `update_goal` schemas](../../../docs/tool-catalog.md#deepseek-aidsh-tool-goal). Successful results are compact JSON. Mutation results are followed by the goal domain's raw `<goal_state>` snapshot after the tool batch. `activation` in a result is a live observation and never becomes replay authority. A goal-round `complete` or `blocked` result additionally injects one `<goal_complete>`/`<goal_blocked>` wrap-up instruction that asks for a grounded closing message to the user without further tool calls.
#### Token effect
Fixed schema cost plus one compact result per call. Mutations also retain the domain snapshot until compaction.
Fixed schema cost plus one compact result per call. Mutations also retain the domain snapshot until compaction. A goal-round terminal update adds the injected wrap-up instruction and one further model request for the closing message — once per goal lifecycle, not per round.
#### KV Cache effect

View File

@@ -14,7 +14,7 @@
3 个规范值都与已经渲染给 Native 调用方的紧凑 JSON 一致:`{ goal: null }``{ goal: { id, revision, objective, phase, roundsStarted, maxGoalRounds, blockedReason? }, activation }`。因此,编程消费方无需解析渲染后的 JSON即可收到相同领域结构。
自主 Goal Round 成功报告 `complete``blocked` 时,会`concludeTurn()` 标记该次工具执行,使物理轮次在该步骤后停止。人类直接变更不会导致这种停止assistant 可以确认变更,循环仍可接收并发的人类 steering中途引导
自主 Goal Round 成功报告 `complete``blocked` 时,会在该次工具结果上附带一条收尾注入指令,要求模型面向用户写出最终收尾消息、不再调用工具,之后轮次经由常规的无工具调用停止路径结束。人类直接变更不会收到这条指令assistant 可以确认变更,循环仍可接收并发的人类 steering中途引导
## 权限
@@ -61,11 +61,11 @@ Use goal tools for one long-running completion objective in the current session.
#### 模型看到的内容
生成的 [`get_goal`、`create_goal` 和 `update_goal` schema](../../../docs/tool-catalog.md#deepseek-aidsh-tool-goal)。成功结果是紧凑 JSON。变更结果之后是工具批次结束后由 goal 领域产生的原始 `<goal_state>` 快照。结果中的 `activation` 是实时观察值,绝不会成为回放权限依据。
生成的 [`get_goal`、`create_goal` 和 `update_goal` schema](../../../docs/tool-catalog.md#deepseek-aidsh-tool-goal)。成功结果是紧凑 JSON。变更结果之后是工具批次结束后由 goal 领域产生的原始 `<goal_state>` 快照。结果中的 `activation` 是实时观察值,绝不会成为回放权限依据。Goal Round 的 `complete`/`blocked` 结果还会额外注入一条 `<goal_complete>`/`<goal_blocked>` 收尾指令,要求模型向用户写出有依据的收尾消息且不再调用工具。
#### Token 影响
固定 schema 成本加上每次调用的一条紧凑结果。变更还会保留领域快照直到压缩compaction
固定 schema 成本加上每次调用的一条紧凑结果。变更还会保留领域快照直到压缩compactionGoal Round 的终态更新会增加注入的收尾指令和一次额外的模型请求用于收尾消息——每个 goal 生命周期一次,而非每轮一次。
#### KV Cache 影响

View File

@@ -8,7 +8,7 @@ import type { Context } from 'cordis'
import z from 'schemastery'
import { GoalId } from '@deepseek-ai/dsh-goal'
import type { GoalRef, GoalView } from '@deepseek-ai/dsh-goal'
import { HarnessError } from '@deepseek-ai/dsh-llm'
import { createUserMessage, HarnessError } from '@deepseek-ai/dsh-llm'
import { defineTool } from '@deepseek-ai/dsh-tools'
import type { GenericCallView } from '@deepseek-ai/dsh-tools'
import type {} from '@deepseek-ai/dsh-system-prompt'
@@ -17,6 +17,7 @@ import {
goalToolExecution,
requireDirectHuman,
} from './authority.ts'
import { renderWrapupContext } from './wrapup.ts'
export const name = 'tool-goal'
export const inject = ['agents', 'goals', 'tools', 'systemPrompt']
@@ -309,7 +310,14 @@ export function apply(ctx: Context, config: Config): void {
code: 'model-reported',
message: args.blocked_reason as string,
})
if (authority.kind === 'goal-round') exec.concludeTurn()
if (authority.kind === 'goal-round') {
exec.deferContext(createUserMessage({
content: args.action === 'complete'
? renderWrapupContext(goal.objective)
: renderWrapupContext(goal.objective, args.blocked_reason as string),
source: { kind: 'plugin', plugin: 'tool-goal' },
}))
}
return Promise.resolve(goalValue(goal))
},
presentCall: args => present(

View File

@@ -0,0 +1,41 @@
/** Model-visible wrap-up instruction for a terminal autonomous goal update. */
import type { ContentBlock } from '@deepseek-ai/dsh-llm'
const GROUNDING =
'Report only what earlier rounds and tool results in this session actually establish; '
+ 'when a detail is not in the session, say so instead of inventing it. '
/**
* Render the closing-message instruction injected after an autonomous goal
* round reports `complete` or `blocked`, replacing the former hard turn stop
* so the model still addresses the user once before the turn ends.
* @param objective - the terminal goal's objective, echoed for grounding.
* @param blockedReason - the validated report for `blocked`; omitted for `complete`.
* @returns a fresh one-block context for `ToolRunContext.deferContext()`.
*/
export function renderWrapupContext(objective: string, blockedReason?: string): ContentBlock[] {
const heading = `Objective: ${JSON.stringify(objective)}\n`
const text = blockedReason === undefined
? '<goal_complete>\n'
+ heading
+ 'The goal is marked complete and this autonomous run is ending. Write the closing '
+ 'message to the user now: state the outcome, summarize what was done and how it was '
+ 'verified, and point to the concrete results (files, commits, or other artifacts). '
+ GROUNDING
+ 'Note anything the user should review or do next. Address the user directly. Do not '
+ "call any more tools in this run; further work waits for the user's next instruction.\n"
+ '</goal_complete>'
: '<goal_blocked>\n'
+ heading
+ `Blocked: ${JSON.stringify(blockedReason)}\n`
+ 'The goal is marked blocked and this autonomous run is ending. Write the closing '
+ 'message to the user now: state what has been completed so far, describe the concrete '
+ 'blocking condition and what you tried, and say exactly what you need from the user to '
+ 'continue. '
+ GROUNDING
+ 'Address the user directly. Do not call any more tools in this run; further work '
+ "waits for the user's next instruction.\n"
+ '</goal_blocked>'
return [{ type: 'text', text }]
}

View File

@@ -347,7 +347,7 @@ describe('goal tool state transitions', () => {
expect(goal).toMatchObject({ phase: 'active', revision: 4 })
})
it('terminal-stops an autonomous completion but leaves a human pause interactive', async () => {
it('injects one wrap-up instruction for an autonomous completion but leaves a human pause interactive', async () => {
const { ctx, root } = await harness()
const humanTurn = openTurn(root, { kind: 'user' })
const created = ctx.goals.create(root.agent, { objective: 'pause cleanly' })
@@ -356,6 +356,7 @@ describe('goal tool state transitions', () => {
}, root.agent)
expect(resultGoal(paused)).toMatchObject({ phase: 'paused' })
expect(paused.concludesTurn).toBeUndefined()
expect(paused.additionalContexts).toBeUndefined()
const resumed = resultGoal(await execute(ctx, 'update_goal', {
goal_id: created.id, revision: 2, action: 'resume',
}, root.agent))
@@ -368,7 +369,27 @@ describe('goal tool state transitions', () => {
goal_id: created.id, revision: resumed['revision'], action: 'complete',
}, root.agent)
expect(resultGoal(complete)).toMatchObject({ phase: 'complete' })
expect(complete.concludesTurn).toBe(true)
expect(complete.concludesTurn).toBeUndefined()
const contexts = complete.additionalContexts ?? []
expect(contexts).toHaveLength(1)
expect(contexts[0]?.source).toEqual({ kind: 'plugin', plugin: 'tool-goal' })
const block = contexts[0]?.content[0]
if (block?.type !== 'text') throw new Error('expected one text wrap-up block')
expect(block.text).toContain('<goal_complete>')
expect(block.text).toContain('"pause cleanly"')
expect(block.text).toContain("Do not call any more tools in this run; further work waits for the user's next instruction.")
})
it('completes without a wrap-up instruction under direct human authority', async () => {
const { ctx, root } = await harness()
openTurn(root, { kind: 'user' })
const created = ctx.goals.create(root.agent, { objective: 'finish now' })
const complete = await execute(ctx, 'update_goal', {
goal_id: created.id, revision: created.revision, action: 'complete',
}, root.agent)
expect(resultGoal(complete)).toMatchObject({ phase: 'complete' })
expect(complete.concludesTurn).toBeUndefined()
expect(complete.additionalContexts).toBeUndefined()
})
it('rearms a restored active goal only after a new direct human prompt', async () => {
@@ -550,6 +571,14 @@ describe('goal tool state transitions', () => {
blockedReason: { code: 'model-reported', message: 'The required credential is still unavailable.' },
roundsStarted: 3,
})
expect(blocked.concludesTurn).toBeUndefined()
const contexts = blocked.additionalContexts ?? []
expect(contexts).toHaveLength(1)
const block = contexts[0]?.content[0]
if (block?.type !== 'text') throw new Error('expected one text wrap-up block')
expect(block.text).toContain('<goal_blocked>')
expect(block.text).toContain('The required credential is still unavailable.')
expect(block.text).toContain("Do not call any more tools in this run; further work waits for the user's next instruction.")
})
it('lets direct human authority block before the model threshold', async () => {
@@ -570,5 +599,7 @@ describe('goal tool state transitions', () => {
},
roundsStarted: 0,
})
expect(blocked.concludesTurn).toBeUndefined()
expect(blocked.additionalContexts).toBeUndefined()
})
})

View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write packages/support/llm-replay/README.md
README.md: 0deb6e76b29d40483b754ac01c98ee0e01bfcbe8
README.zh.md: 7720e2d1bc6eb7bc5c89d5c1708767a54a7b0080
README.md: 85aa56705929e7630e4cfb6c2a3c9cbbd0d843a6
README.zh.md: 751f75dea197ffb112cfa703e3a5dbfaffb8c0b2

View File

@@ -12,6 +12,8 @@ The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assi
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`) that either replaces the derived script (a bare `ReplayEntry[]`) or augments it (`{ patches: [{ at, entry }] }`: keep every JSONL-derived call and swap the named 0-based call indexes; `at` equal to the derived length appends the retry attempt after an injected transient throw). Patch indexes must be unique. The override document, each patch and entry, and every chunk discriminant are validated when the file loads. A `hang` entry may name `readyFile`; replay writes that empty marker after its prefix chunks reach the loop and before it waits for cancellation, so an external driver can cancel deterministically without observing a presentation update.
A scripted string may embed `{{fromRequest:<regex>}}` to fill a value no static sidecar can know — for example a randomly minted goal id the model must echo back into `update_goal`. At stream time every placeholder resolves against the live request: the corpus is every string leaf of the request messages joined by newlines, the pattern's LAST corpus match wins, and its first capture group (or the whole match without one) substitutes in place. A pattern that matches nothing, an invalid pattern, and an unterminated placeholder each fail loud. The last two braces of a consecutive `}` run terminate the placeholder, so a pattern may end with a brace quantifier (`[0-9a-f]{4}`) but cannot contain `}}` followed by further pattern content. Resolution applies to every scripted entry, including ones derived from the recorded JSONL — a recorded fixture whose text legitimately contains the literal marker must be expressed through a sidecar without it.
## Nested agents: per-session keying
A scenario where a parent agent delegates to in-process subagents records more than one log: the parent (`session.jsonl`) plus one per child (`session.1.jsonl`, …). Each agent runs as its own `Session` on the same context, so replay must serve each one its own script.
@@ -55,7 +57,7 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
- `installLlmReplay(ctx, config)` — install the configured replay adapter or catch-all `llm/stream` listener; returns a `ReplayHandle` (`dispose()` for HMR safety plus `assertConsumed()`, the teardown check that every recorded script bound to a live session and every bound cursor drained — turning a scenario that silently drove fewer model calls than recorded into a crisp diagnostic). Use this in tests to drive replay without the Loader or env vars.
- `loadSessionScripts(config)` — resolve the ordered `SessionScript[]` (primary + children) for a scenario, ready to bind to live sessions in first-call order.
- `loadReplayScript(config)` — resolve the `ReplayEntry[]` for the primary session only (validated sidecar replacement/patches if present, else derived from the JSONL; fail-loud if the fixture is missing).
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)` — the pure helpers that turn a recorded session log into a script and read its header `id`/`createdAt`. A derived group must end in a `finish` chunk; a group without one is the fingerprint of a thrown `stream()` and must instead be expressed via an override sidecar.
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)` / `resolveScriptedEntry(entry, messages)` — the pure helpers that turn a recorded session log into a script, read its header `id`/`createdAt`, and resolve `{{fromRequest:...}}` placeholders against one live request. A derived group must end in a `finish` chunk; a group without one is the fingerprint of a thrown `stream()` and must instead be expressed via an override sidecar.
- Types `ReplayEntry` / `ReplayOverrideDoc` / `ReplayOverridePatch` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`.
## Plugin export shape

View File

@@ -12,6 +12,8 @@ fixture 就是持久化的会话日志(`<scenario>/session.jsonl`)。其 `as
有两种失败模式无法仅根据 `assistant/chunk` 重建:在产生任何分片前直接抛出异常(例如 HTTP 401此时日志只有 `turn/end {error}` 而没有分片),以及取消或挂起(差异在时序,而非分片内容)。需要这些行为的场景可提供伴随文件(`<scenario>/replay.override.json`):它可以替换派生脚本(裸 `ReplayEntry[]`),也可以增补派生脚本(`{ patches: [{ at, entry }] }`:保留所有从 JSONL 派生的调用,只替换指定的从 0 开始计数的调用索引;当 `at` 等于派生长度时,则在注入瞬态异常后的重试位置追加一次调用)。补丁索引不得重复。文件加载时会校验覆写文档、每个补丁和条目,以及每个分片的判别标签。`hang` 条目可以指定 `readyFile`;当前缀分片到达循环后、开始等待取消前,回放会写入这个空标记,使外部驱动程序无需观察展示层更新即可确定性地取消。
脚本字符串可以内嵌 `{{fromRequest:<regex>}}`,用来填入静态伴随文件不可能预知的值——例如模型必须原样回填到 `update_goal` 的随机生成 goal id。回放时每个占位符针对实时请求解析语料是请求消息的所有字符串叶子按换行拼接的结果取该模式在语料中的最后一次匹配用其第一个捕获组无捕获组时用整个匹配原位替换。模式匹配不到内容、模式非法、占位符未闭合都会明确报错。连续右花括号串的最后两个花括号才是占位符结束符因此模式可以以花括号量词收尾`[0-9a-f]{4}`),但不能在 `}}` 之后还有后续模式内容。解析作用于所有脚本条目,包括从已记录 JSONL 派生的条目——若录制文本本身合法地含有该字面量标记,需改用不含标记的伴随文件表达。
## 嵌套 agent每会话键控
父 agent 委托给进程内 subagent子 agent的场景会记录多个日志父会话使用 `session.jsonl`,每个子会话各使用一个日志(`session.1.jsonl` 等)。每个 agent 都在同一上下文中作为独立的 `Session` 运行,因此回放必须为每个 agent 提供各自的脚本。
@@ -55,7 +57,7 @@ fixture 就是持久化的会话日志(`<scenario>/session.jsonl`)。其 `as
- `installLlmReplay(ctx, config)`:安装已配置回放适配器或 catch-all `llm/stream` 监听器;返回 `ReplayHandle`(包含用于保证 HMR热模块替换安全的 `dispose()`,以及清理阶段执行的 `assertConsumed()` 检查;后者确保每个已记录脚本都绑定到实时会话,且每个已绑定游标都已耗尽,从而将场景静默驱动的模型调用少于记录数转换为明确诊断)。在测试中使用它,可以不通过 Loader 或 env var 驱动回放。
- `loadSessionScripts(config)`:解析场景的有序的 `SessionScript[]`(主会话 + 子会话),准备按首次调用顺序绑定到实时会话。
- `loadReplayScript(config)`:只解析主会话的 `ReplayEntry[]`(如果伴随文件存在,则使用经校验的替换或补丁;否则从 JSONL 派生fixture 缺失时明确报错)。
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)`:将已记录会话日志转换为脚本读取其 header `id`/`createdAt` 的纯辅助工具。派生分组必须以 `finish` 分片结束;没有该分片的分组是 `stream()` 抛出异常的指纹,必须改用 override sidecar 表达。
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)` / `resolveScriptedEntry(entry, messages)`:将已记录会话日志转换为脚本读取其 header `id`/`createdAt`、并针对单次实时请求解析 `{{fromRequest:...}}` 占位符的纯辅助工具。派生分组必须以 `finish` 分片结束;没有该分片的分组是 `stream()` 抛出异常的指纹,必须改用 override sidecar 表达。
- 类型 `ReplayEntry` / `ReplayOverrideDoc` / `ReplayOverridePatch` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`
## 插件导出形态

View File

@@ -241,6 +241,96 @@ const REPLAY_CHUNK_TYPES = new Set<StreamChunk['type']>([
'finish',
])
const FROM_REQUEST_OPEN = '{{fromRequest:'
const FROM_REQUEST_CLOSE = '}}'
/** Collect every string leaf of one JSON-shaped value, in traversal order. */
function collectStrings(value: unknown, out: string[]): void {
if (typeof value === 'string') {
out.push(value)
return
}
if (Array.isArray(value)) {
for (const item of value) collectStrings(item, out)
return
}
if (value !== null && typeof value === 'object') {
for (const item of Object.values(value)) collectStrings(item, out)
}
}
/** Resolve one placeholder pattern against the request corpus; the LAST match wins. */
function resolveFromRequest(pattern: string, corpus: string): string {
let regex: RegExp
try {
regex = new RegExp(pattern, 'g')
} catch (error) {
// RegExp construction only throws SyntaxError; String() carries its message.
throw new Error(`llm-replay: fromRequest has an invalid pattern ${JSON.stringify(pattern)}: ${String(error)}`)
}
let last: RegExpExecArray | undefined
for (const match of corpus.matchAll(regex)) last = match
if (last === undefined) {
throw new Error(`llm-replay: fromRequest pattern ${JSON.stringify(pattern)} matched nothing in the request`)
}
return last[1] ?? last[0]
}
/** Replace every `{{fromRequest:<pattern>}}` occurrence in one scripted string. */
function substituteString(text: string, corpus: string): string {
let result = ''
let cursor = 0
while (true) {
const open = text.indexOf(FROM_REQUEST_OPEN, cursor)
if (open === -1) return result + text.slice(cursor)
let close = text.indexOf(FROM_REQUEST_CLOSE, open + FROM_REQUEST_OPEN.length)
if (close === -1) {
throw new Error(`llm-replay: fromRequest placeholder is unterminated in ${JSON.stringify(text)}`)
}
// The last two braces of a consecutive `}` run terminate the placeholder,
// so a pattern may end with a brace quantifier like `[0-9a-f]{4}`.
while (text[close + FROM_REQUEST_CLOSE.length] === '}') close += 1
const pattern = text.slice(open + FROM_REQUEST_OPEN.length, close)
result += text.slice(cursor, open) + resolveFromRequest(pattern, corpus)
cursor = close + FROM_REQUEST_CLOSE.length
}
}
/** Deep-copy one JSON-shaped value with scripted placeholders resolved. */
function substituteValue(value: unknown, corpus: string): unknown {
if (typeof value === 'string') {
return value.includes(FROM_REQUEST_OPEN) ? substituteString(value, corpus) : value
}
if (Array.isArray(value)) return value.map(item => substituteValue(item, corpus))
if (value !== null && typeof value === 'object') {
return Object.fromEntries(Object.entries(value).map(([key, item]) => [key, substituteValue(item, corpus)]))
}
return value
}
/**
* Resolve every `{{fromRequest:<regex>}}` placeholder in one scripted entry
* against the live request. The corpus is every string leaf of the request
* messages joined by newlines; the pattern's LAST corpus match wins and its
* first capture group (or, without one, the whole match) substitutes in place.
* Scenario sidecars use this to script arguments no static file can know,
* such as a randomly minted goal id the model must echo back. A pattern that
* matches nothing, an invalid pattern, and an unterminated placeholder each
* fail loud. The last two braces of a consecutive `}` run terminate the
* placeholder, so a pattern may end with a brace quantifier but cannot
* contain `}}` followed by further pattern content. Derived entries pass
* through the same resolution as sidecar entries.
* @param entry - the scripted entry about to replay.
* @param messages - the live request messages searched by the placeholders.
* @returns the entry itself when no placeholder appears, else a resolved deep copy.
*/
export function resolveScriptedEntry(entry: ReplayEntry, messages: GenerateOptions['messages']): ReplayEntry {
if (!JSON.stringify(entry).includes(FROM_REQUEST_OPEN)) return entry
const leaves: string[] = []
collectStrings(messages, leaves)
return substituteValue(entry, leaves.join('\n')) as ReplayEntry
}
function isRecord(value: unknown): value is Record<string, unknown> {
return typeof value === 'object' && value !== null && !Array.isArray(value)
}
@@ -583,7 +673,7 @@ export function installLlmReplay(ctx: Context, config: ReplayConfig): ReplayHand
+ `but its script has only ${boundState.entries.length}; re-record the scenario`,
)
}
yield* replayEntry(entry, options.signal, paceMs)
yield* replayEntry(resolveScriptedEntry(entry, options.messages), options.signal, paceMs)
})()
}
const providers = config.providers ?? []

View File

@@ -4,7 +4,7 @@ import { join } from 'node:path'
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
import { Context } from 'cordis'
import type { SessionEvent } from '@deepseek-ai/dsh-session'
import LlmService, { GenerateOptions, LlmAdapter, StreamChunk } from '@deepseek-ai/dsh-llm'
import LlmService, { CallId, createUserMessage, GenerateOptions, LlmAdapter, StreamChunk } from '@deepseek-ai/dsh-llm'
import {
type ReplayEntry,
type SessionScript,
@@ -17,6 +17,7 @@ import {
name,
parseSessionHeader,
parseSessionLog,
resolveScriptedEntry,
} from '../src/index.ts'
/**
@@ -310,6 +311,80 @@ describe('installLlmReplay (through the real LlmService)', () => {
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
})
describe('{{fromRequest:...}} substitution', () => {
const requestMessages = [createUserMessage({
content: [{ type: 'text' as const, text: 'stale {"goal":{"id":"goal-old"}} then {"goal":{"id":"goal-42ab"}}' }],
source: { kind: 'user' as const },
})]
function scriptedCall(argumentsDelta: string): StreamChunk[] {
return [
{ type: 'block-start', index: 0, blockType: 'tool-call' },
{ type: 'tool-call-delta', index: 0, id: CallId('c1'), name: 'update_goal', argumentsDelta },
{ type: 'block-end', index: 0, block: { type: 'tool-call', id: CallId('c1'), name: 'update_goal', arguments: argumentsDelta } },
{ type: 'finish', reason: { kind: 'tool-calls' } },
]
}
async function streamScripted(argumentsDelta: string): Promise<StreamChunk[]> {
writeLog(TEXT_CHUNKS)
const overrideFile = join(dir, 'replay.override.json')
writeFileSync(overrideFile, JSON.stringify([{ kind: 'chunks', chunks: scriptedCall(argumentsDelta) }]), 'utf8')
const ctx = new Context()
await ctx.plugin(LlmService)
installLlmReplay(ctx, { file, overrideFile })
return drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: requestMessages }))
}
it('resolves the capture group from the LAST request match in every scripted string field', async () => {
const streamed = await streamScripted('{"goal_id":"{{fromRequest:"id":"(goal-[^"]+)"}}","revision":1}')
const delta = streamed.find(chunk => chunk.type === 'tool-call-delta')
expect(delta).toMatchObject({ argumentsDelta: '{"goal_id":"goal-42ab","revision":1}' })
const end = streamed.find(chunk => chunk.type === 'block-end')
expect(end).toMatchObject({ block: { arguments: '{"goal_id":"goal-42ab","revision":1}' } })
})
it('substitutes the whole match when the pattern has no capture group', async () => {
const streamed = await streamScripted('{"goal_id":"{{fromRequest:goal-[0-9a-z]+}}"}')
const delta = streamed.find(chunk => chunk.type === 'tool-call-delta')
expect(delta).toMatchObject({ argumentsDelta: '{"goal_id":"goal-42ab"}' })
})
it('keeps a trailing brace quantifier inside the pattern (terminator is the run tail)', async () => {
const streamed = await streamScripted('{"goal_id":"{{fromRequest:goal-[0-9a-z]{4}}}"}')
const delta = streamed.find(chunk => chunk.type === 'tool-call-delta')
expect(delta).toMatchObject({ argumentsDelta: '{"goal_id":"goal-42ab"}' })
})
it('fails loud when a placeholder matches nothing in the request', async () => {
await expect(streamScripted('{"goal_id":"{{fromRequest:task-[0-9]+}}"}'))
.rejects.toThrow(/fromRequest.*matched nothing/)
})
it('fails loud on an invalid placeholder pattern', async () => {
await expect(streamScripted('{"goal_id":"{{fromRequest:(goal-}}"}'))
.rejects.toThrow(/fromRequest.*invalid pattern/)
})
it('fails loud on an unterminated placeholder', () => {
const entry: ReplayEntry = { kind: 'chunks', chunks: scriptedCall('{"goal_id":"{{fromRequest:goal-1"}') }
expect(() => resolveScriptedEntry(entry, requestMessages)).toThrow(/fromRequest placeholder is unterminated/)
})
it('returns the exact same entry when no placeholder appears', () => {
const entry: ReplayEntry = { kind: 'chunks', chunks: TEXT_CHUNKS }
expect(resolveScriptedEntry(entry, requestMessages)).toBe(entry)
})
it('skips non-string request leaves when building the corpus', () => {
const messages = requestMessages.map(message => ({ ...message, seq: 7 })) as unknown as GenerateOptions['messages']
const entry: ReplayEntry = { kind: 'chunks', chunks: scriptedCall('{"goal_id":"{{fromRequest:goal-42[a-z]+}}"}') }
const resolved = resolveScriptedEntry(entry, messages)
if (resolved.kind !== 'chunks') throw new Error('expected chunks entry')
expect(resolved.chunks[1]).toMatchObject({ argumentsDelta: '{"goal_id":"goal-42ab"}' })
})
})
it('registers a replay-only provider catalog when configured', async () => {
writeLog(TEXT_CHUNKS)
const ctx = new Context()