fix(tools): forward nested image results in code mode

This commit is contained in:
Tianyi Cui
2026-08-11 15:34:13 +08:00
parent 219d2a1fb9
commit e00146be73
19 changed files with 117 additions and 42 deletions

View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-07-20-code-mode-typed-tool-returns.md
2026-07-20-code-mode-typed-tool-returns.md: 6b31dbca21a22a24f2bbb0ef61097622884165c3
2026-07-20-code-mode-typed-tool-returns.zh.md: 1bb88d29ffd96d65ce06033202499d2780814abd
2026-07-20-code-mode-typed-tool-returns.md: 6bb4ccdeb81172ec9102a8656d380dfed09b63ae
2026-07-20-code-mode-typed-tool-returns.zh.md: 49084d6104ce2f733810126e4bb7cf78e92740e5

View File

@@ -14,7 +14,7 @@ The [canonical tool-output contract](../architecture/2026-07-20-canonical-tool-o
## Decision
Code Mode is a typed projection of the visible tool registry. Each successful binding resolves to the final canonical `JsonValue` after post-execute policy, while a failed binding rejects with a real `ToolCallError`. Intermediate values remain inside the run and cross the worker boundary whole. Only the outer `run_code` logs, completion value, or failure diagnostic enter the configurable output ledger and the model-facing spill pipeline.
Code Mode is a typed projection of the visible tool registry. Each successful binding resolves to the final canonical `JsonValue` after post-execute policy, while a failed binding rejects with a real `ToolCallError`. Intermediate values remain inside the run and cross the worker boundary whole. The outer `run_code` logs, completion value, or failure diagnostic enter the configurable output ledger and model-facing spill pipeline; a successfully settled sub-call whose final Native content contains an image additionally defers that complete ordered content through the parent result as logged, source-attributed context.
This note owns the return and failure contract layered on the original [Code Mode foundation](2026-06-15-code-mode.md). The unified schema vocabulary is owned by the [JSON-value schema DSL note](../architecture/2026-07-20-unified-json-value-schema-dsl.md), and Native rendering and policy projection remain owned by the canonical-output note.
@@ -49,7 +49,7 @@ declare const tools: {
### Binding values and failures
Before dispatch the bridge snapshots binding arguments as lossless JSON and snapshots the detached value again for an independent durable summary event. Host-side detachment, immutable execution, and output-schema projection all use iterative traversals rather than nested structured clone or recursive freezing. `undefined`, non-finite numbers, `-0`, sparse arrays, cycles, functions, and exotic objects reject that call before the tool runs. Successful dispatch returns `ToolExecutionResult.value`; Native `content`, metadata, and internal error information do not cross to the program.
Before dispatch the bridge snapshots binding arguments as lossless JSON and snapshots the detached value again for an independent durable summary event. Host-side detachment, immutable execution, and output-schema projection all use iterative traversals rather than nested structured clone or recursive freezing. `undefined`, non-finite numbers, `-0`, sparse arrays, cycles, functions, and exotic objects reject that call before the tool runs. Successful dispatch returns `ToolExecutionResult.value`; Native `content`, metadata, and internal error information do not cross to the program. Image-bearing final content is not a second binding value: the bridge ferries it after the outer result so the next model request can see the durable image, while post-execute block/content replacement remains authoritative and text-only results are not duplicated.
Code Mode declares its rejection capability on the runtime request as `{ name: "ToolCallError", memberNameProperty: "toolName" }`. The runtime Service Definition treats those names as data: the worker materializes and injects the actual constructor used for `tools` binding failures, so `error instanceof ToolCallError` works without making a generic runtime know about tools. The worker constructs failures and defines their public fields through module-captured error and property-definition intrinsics plus null-prototype descriptors, so model mutations cannot replace the promised rejection with a worker failure. The error has the standard `Error` message plus the exact `toolName`; it deliberately omits `ToolFailure.info`, error codes, and Native content. This is an exception contract for control flow, not a failure union for programmatic classification.
@@ -73,13 +73,13 @@ Temporary Cordis Plugins follow the same rule: `cordis_mount` returns `{ id, plu
### Persistence, metadata, and spill
Nested dispatch logs the sub-call's full rendered `content`/`isError` on `tool/code-dispatch` but does not persist canonical values. `tool/result` continues to persist only rendered content, error, and optional metadata. `SESSION_FORMAT_VERSION` remains unchanged (pre-release shape churn does not bump it) and replay cannot recreate intermediate canonical program values.
Nested dispatch logs the sub-call's full rendered `content`/`isError` on `tool/code-dispatch` but does not persist canonical values. `tool/result` continues to persist only rendered content, error, and optional metadata. A successful final content sequence containing an image is also wrapped in a source-attributed user message and deferred through the outer result; the normal session event makes that model-visible input reconstructable. `SESSION_FORMAT_VERSION` remains unchanged (pre-release shape churn does not bump it) and replay cannot recreate intermediate canonical program values.
The opaque `exec.parent` token marks nested calls. Presentation metadata and generic or tool-owned spill projections skip those calls because they have no direct result card and their canonical values never enter context. The outer `run_code` call alone produces one card and may spill its final post-policy presentation; `run_code` intentionally declares neither a result presenter nor presentation metadata, so UI adapters complete the card through their generic raw-content fallback using durable `tool/result.content`.
## Testing
Compile-time and snapshot tests pin exact `ToolArgsMap`, `ToolOutputMap`, `ToolName`, schema-to-TypeScript coverage, and exotic names. Registry and real-worker tests cover scalar, array, object, and null values; raw string rendering; absent `undefined`; consumer-declared real rejection classes, including `ToolCallError`; invalid arguments and completions, including intrinsic-looking forged prototypes; model-mutated JSON-boundary globals, prototype methods, constructor slots, and inherited descriptor fields; typed binding failures after those mutations; large uncapped intermediate bindings; nested spill suppression; exact and over-limit 64 MiB accounting; combined logs/value/diagnostic accounting; giant thrown stacks; bounded failure spill; hostile forged traffic; and built-package execution.
Compile-time and snapshot tests pin exact `ToolArgsMap`, `ToolOutputMap`, `ToolName`, schema-to-TypeScript coverage, exotic names, and assembled Code Mode image forwarding. Registry and real-worker tests cover scalar, array, object, and null values; raw string rendering; absent `undefined`; consumer-declared real rejection classes, including `ToolCallError`; invalid arguments and completions, including intrinsic-looking forged prototypes; model-mutated JSON-boundary globals, prototype methods, constructor slots, and inherited descriptor fields; typed binding failures after those mutations; large uncapped intermediate bindings; nested spill suppression; generic image-bearing context deferral plus post-execute replacement/block precedence; exact and over-limit 64 MiB accounting; combined logs/value/diagnostic accounting; giant thrown stacks; bounded failure spill; hostile forged traffic; and built-package execution.
Keyless real-worker integration tests pin the two handle workflows that prose results could not safely support. A background bash call returns its task id, the outer run settles, and a later run polls that id to completion; separate cases prove pre-abort creates no task, post-publication call abort preserves the task, foreground execution stays signal-coupled, and `task_kill` owns cancellation. A Cordis program reads an active or pending mount's id and `waitingFor` fields directly, unmounts by that id, and confirms removal without parsing rendered text.
@@ -93,6 +93,10 @@ Keyless real-worker integration tests pin the two handle workflows that prose re
**Silently inspect or truncate an oversized completion.** Rejected because changing a JSON value into a string is lossy and type-incorrect. The explicit `output-limit` failure lets the model choose a smaller result, while the retained logs and diagnostic can still use normal outer spill.
**Require each rich leaf tool to inspect `exec.parent` and defer itself.** Rejected because it couples leaf tools to Code Mode internals, duplicates policy handling, and misses future rich tools. The dispatch bridge owns generic forwarding from the already settled final result.
**Expose Native rich content as part of every binding's canonical value.** Rejected because a canonical value is lossless JSON and tool-specific; attachment blocks are a model projection with durable lifecycle semantics. Keeping the value and projection separate preserves typed programs without dropping images from later model context.
## Consequences
Code programs can compose tools through stable values instead of reverse-engineering Native prose. Native and Both Mode retain their existing text and UI presentation, while Code Mode receives output-schema types and exact runtime JSON. Tool authors must treat the canonical value as their programmatic API and put display-only formatting in the renderer.
@@ -107,6 +111,6 @@ The worker performs bounded-depth flat-wire transport and lossless validation bu
- Intermediate values have no byte cap and can exhaust process or worker memory through retention, flat-wire copies, or structured-clone cost.
- The 64 MiB hard cap applies only to the outer variable payloads, excluding fixed result-envelope syntax and presentation whitespace; spill cannot recover bytes rejected beyond that cap.
- Provider or executor acquisition limits may already have discarded source data before a canonical value reaches Code Mode.
- Unsupported MCP output schemas fall back to `JsonValue`; richer Native multimedia projection is deferred.
- Unsupported MCP output schemas fall back to `JsonValue`; admitted MCP images use the generic deferred projection, while audio and embedded-resource payloads remain diagnostic-only.
- There is one result card per outer `run_code`, never per nested call.
- Code failures expose `ToolCallError` message and tool name only, without a programmatic error-code union.

View File

@@ -14,7 +14,7 @@ Code Mode 过去会把每个嵌套工具的结果从 `ContentBlock[]` 重新投
## 决策
Code Mode 是可见工具注册表的类型化投影。每个成功的绑定调用都会解析为 post-execute 策略处理后的最终规范 `JsonValue`,失败的绑定调用则会以真正的 `ToolCallError` 拒绝 Promise。中间值只存在于本次运行中并完整跨越 worker 边界。只有外层 `run_code` 的日志、完成值或失败诊断会进入可配置的输出账本以及面向模型的输出落盘流水线。
Code Mode 是可见工具注册表的类型化投影。每个成功的绑定调用都会解析为 post-execute 策略处理后的最终规范 `JsonValue`,失败的绑定调用则会以真正的 `ToolCallError` 拒绝 Promise。中间值只存在于本次运行中并完整跨越 worker 边界。外层 `run_code` 的日志、完成值或失败诊断会进入可配置的输出账本以及面向模型的输出落盘流水线;如果成功结算的子调用最终 Native 内容包含图片,其完整有序内容还会经父结果延后为写入日志且带来源归属的上下文
本文档定义叠加在原始 [Code Mode 基础](2026-06-15-code-mode.md)之上的返回值与失败约定。统一 schema 词汇由 [JSON 值 schema DSL Agent Note](../architecture/2026-07-20-unified-json-value-schema-dsl.md)负责定义Native 渲染与策略投影仍由规范输出 Agent Note 负责定义。
@@ -49,7 +49,7 @@ declare const tools: {
### 绑定值与失败
分发前,桥接层会把绑定参数快照为无损 JSON再对分离后的值生成一次快照供独立的持久摘要事件使用。宿主侧的值分离、执行数据的不可变处理与输出 schema 投影均采用迭代遍历,而不使用嵌套结构化克隆或递归冻结。`undefined`、非有限数、`-0`、稀疏数组、循环引用、函数和非普通对象都会使该调用在工具运行前被拒绝。成功分发会返回 `ToolExecutionResult.value`Native `content`、元数据和内部错误信息不会传入程序。
分发前,桥接层会把绑定参数快照为无损 JSON再对分离后的值生成一次快照供独立的持久摘要事件使用。宿主侧的值分离、执行数据的不可变处理与输出 schema 投影均采用迭代遍历,而不使用嵌套结构化克隆或递归冻结。`undefined`、非有限数、`-0`、稀疏数组、循环引用、函数和非普通对象都会使该调用在工具运行前被拒绝。成功分发会返回 `ToolExecutionResult.value`Native `content`、元数据和内部错误信息不会传入程序。含图片的最终内容不是第二份绑定值桥接层会在外层结果之后转运它使下一次模型请求可以看到持久图片post-execute 阻止/内容替换仍具有权威性,纯文本结果不会重复。
Code Mode 通过运行时请求中的 `{ name: "ToolCallError", memberNameProperty: "toolName" }` 声明其以异常拒绝 Promise 的能力。运行时 Service Definition 只把这些名称视为数据worker 会动态生成并注入真正用于 `tools` 绑定失败的构造函数,因此无需让通用运行时了解工具,`error instanceof ToolCallError` 也能成立。worker 使用模块初始化时捕获的 Error 构造函数与属性定义内建方法,配合原型为 null 的属性描述符,构造失败对象并定义其公开字段,因此模型代码的修改不会把约定承诺的 reject 变成 worker 失败。该错误包含标准的 `Error` 消息和确切的 `toolName`,并有意省略 `ToolFailure.info`、错误代码与 Native 内容。这是一项用于控制流的异常约定,而不是供程序分类的失败联合。
@@ -73,13 +73,13 @@ Code Mode 通过运行时请求中的 `{ name: "ToolCallError", memberNameProper
### 持久化、元数据与输出落盘
嵌套分发在 `tool/code-dispatch` 上记录子调用完整渲染后的 `content`/`isError`,但不会持久化规范值。`tool/result` 继续只持久化渲染后的内容、错误和可选元数据。`SESSION_FORMAT_VERSION` 保持不变(预发布阶段的形状变动不递增版本号),回放也无法重建程序的规范中间值。
嵌套分发在 `tool/code-dispatch` 上记录子调用完整渲染后的 `content`/`isError`,但不会持久化规范值。`tool/result` 继续只持久化渲染后的内容、错误和可选元数据。包含图片的成功最终内容序列还会包装成带来源归属的用户消息,并经外层结果延后;普通会话事件使该模型可见输入可以重建。`SESSION_FORMAT_VERSION` 保持不变(预发布阶段的形状变动不递增版本号),回放也无法重建程序的规范中间值。
不透明的 `exec.parent` token 用于标识嵌套调用。由于这些调用没有直接对应的结果卡片,而且其规范值永远不会进入上下文,展示元数据以及通用或工具自有的输出落盘投影都会跳过它们。只有外层 `run_code` 调用会生成一张卡片,并且可能将 post-policy 处理后的最终展示写入落盘文件;`run_code` 有意既不声明结果展示器,也不声明展示元数据,因此 UI 适配器会通过通用的原始内容回退机制,使用持久化的 `tool/result.content` 补全该卡片。
## 测试
编译期测试与快照测试锁定了精确的 `ToolArgsMap`、`ToolOutputMap`、`ToolName`、schema 到 TypeScript 的覆盖范围以及特殊名称。注册表与真实 worker 测试覆盖标量、数组、对象和 null 值;字符串原文渲染;缺席的 `undefined`;消费方声明、实际用于拒绝 Promise 的异常类,包括 `ToolCallError`;无效参数与完成值,包括伪装为内建原型的伪造原型;模型代码修改过的 JSON 边界全局对象、原型方法、构造函数槽位以及继承而来的属性描述符字段上述修改后的类型化绑定失败不设上限的大型中间绑定值嵌套输出落盘抑制64 MiB 上限内外的精确计量;日志、值与诊断的组合计量;抛出的超大堆栈;有界失败的输出落盘;不可信对端伪造的流量;以及构建后包的执行。
编译期测试与快照测试锁定了精确的 `ToolArgsMap`、`ToolOutputMap`、`ToolName`、schema 到 TypeScript 的覆盖范围特殊名称,以及组装后的 Code Mode 图片转发。注册表与真实 worker 测试覆盖标量、数组、对象和 null 值;字符串原文渲染;缺席的 `undefined`;消费方声明、实际用于拒绝 Promise 的异常类,包括 `ToolCallError`;无效参数与完成值,包括伪装为内建原型的伪造原型;模型代码修改过的 JSON 边界全局对象、原型方法、构造函数槽位,以及继承而来的属性描述符字段;上述修改后的类型化绑定失败;不设上限的大型中间绑定值;嵌套输出落盘抑制;通用含图片上下文延后以及 post-execute 替换/阻止优先级;64 MiB 上限内外的精确计量;日志、值与诊断的组合计量;抛出的超大堆栈;有界失败的输出落盘;不可信对端伪造的流量;以及构建后包的执行。
无密钥的真实 worker 集成测试锁定了自然语言结果无法安全支持的两种句柄工作流。后台 bash 调用返回 task id外层运行结束之后的运行再根据该 id 轮询直至任务完成;其他用例分别证明,预先中止不会创建任务、发布后的调用取消会保留任务、前台执行仍与信号耦合,并且取消归 `task_kill` 所有。Cordis 程序会直接读取 active 或 pending 挂载的 id 和 `waitingFor` 字段,按该 id 卸载,并在不解析渲染文本的情况下确认挂载已移除。
@@ -93,6 +93,10 @@ Code Mode 通过运行时请求中的 `{ name: "ToolCallError", memberNameProper
**静默检查格式化或截断过大的完成值:**不予采纳。把 JSON 值改成字符串既有损又违反类型。显式的 `output-limit` 失败让模型可以选择返回更小的结果,而保留的日志和诊断仍可使用普通的外层输出落盘机制。
**要求每个丰富叶子工具检查 `exec.parent` 并自行延后。** 不予采用,因为这会把叶子工具与 Code Mode 内部机制耦合、重复策略处理,并遗漏未来丰富工具。分发桥接层负责从已经结算的最终结果通用转发。
**把 Native 丰富内容暴露为每个绑定规范值的一部分。** 不予采用,因为规范值是无损 JSON 且由工具定义;附件块是具有持久生命周期语义的模型投影。保持值与投影分离,既能保留类型化程序,也不会从后续模型上下文中丢弃图片。
## 后果
Code Mode 程序可以通过稳定值组合工具,无需逆向解析 Native 自然语言。Native 和 Both Mode 保留现有文本与 UI 展示Code Mode 则获得输出 schema 类型和精确的运行时 JSON。工具作者必须把规范值视为程序化 API并将仅用于展示的格式化放入渲染器。
@@ -107,6 +111,6 @@ worker 会以嵌套深度有界的扁平协议格式传输数据并执行无损
- 中间值没有字节上限,可能因值的保留、扁平协议格式副本或结构化克隆开销而耗尽进程或 worker 内存。
- 64 MiB 硬上限只适用于外层可变负载,不计固定的结果封装语法与展示空白;输出落盘无法恢复超出该上限后被拒绝的字节。
- 提供方或执行器的采集上限可能在规范值到达 Code Mode 前就已丢弃部分源数据。
- 不支持的 MCP 输出 schema 会回退为 `JsonValue`更丰富的 Native 多媒体投影留待后续实现
- 不支持的 MCP 输出 schema 会回退为 `JsonValue`已准入的 MCP 图片使用通用延后投影,而音频和嵌入资源载荷仍只提供诊断
- 每个外层 `run_code` 只有一张结果卡片,嵌套调用不会各自生成卡片。
- Code Mode 失败只暴露 `ToolCallError` 的消息与工具名,不提供程序可用的错误代码联合。