Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(fs): add a minimal read_image tool over the attachment and fs seams

Browse Source 
The model reads a PNG/JPEG/WebP/GIF file, the bytes commit through the
durable attachment lifecycle, and the tool result carries the real
ImageBlock so the image enters context from the next request onward.
FileSystem gains a bounded readBytes primitive (local + E2B providers);
registration is conditional on the attachment store, and a strict
execution gate refuses routes that do not declare image input, so a
text route's durable history stays free of image blocks. llm-replay
models may declare inputModalities, letting keyless ACP snapshots pin
both the sha256-referenced success and the verbatim refusal.

Supersedes the withdrawn route-scoped design of PR #598; the decision
record is .agents/notes/implemented/feature/2026-08-10-minimal-read-image-tool.md.
This commit is contained in:
creatixchu
2026-08-10 15:09:07 +08:00
parent 3764ce62a5
commit 1861a3fc7c
65 changed files with 1973 additions and 67 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
.agents/notes/implemented/feature/2026-08-10-minimal-read-image-tool.i18n.yaml Normal file
View File

@@ -0,0 +1,6 @@
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-08-10-minimal-read-image-tool.md
2026-08-10-minimal-read-image-tool.md: a43e53d70e98bac7a50aa6bbabbb1e177237df01
2026-08-10-minimal-read-image-tool.zh.md: a94e4b296425ad50876b0b45a689442c896a85a1

33
.agents/notes/implemented/feature/2026-08-10-minimal-read-image-tool.md Normal file
View File

@@ -0,0 +1,33 @@
# Agent Note: A minimal read_image tool over existing seams
Status: implemented
English | [中文](2026-08-10-minimal-read-image-tool.zh.md)
## Problem
The multimodal attachment work gave user uploads a complete durable path — bytes committed to the content-addressed attachment store before the owning `user/message`, an `ImageBlock` carrying only the `sha256:` reference, and the pi-ai route re-reading verified bytes per request — but the model itself had no way to look at an image on disk. `read` rejects binary content by contract, so an agent asked about a screenshot or a rendered chart either failed or shelled out to lossy workarounds. A first standalone attempt (PR #598) solved this together with loop-level route scoping: an `agent/request-ready` extension point publishing exact-model modalities before assembly, per-route schema/guidance visibility, and a reversible `image-placeholder-v1` history projection so text routes could continue over placeholder text. That design worked but coupled a tool to new agent-loop machinery, three new session-log concepts, and per-step registration churn — far more surface than the capability needs.
## Decision
Ship the smallest tool that loads an image into the next request's context, entirely over existing seams; the withdrawn PR #598 design is the explicit counter-example this note records.
- **`read_image` lives in `dsh-tool-fs`** beside `read`/`write`/`edit`. Extension selects the declared PNG/JPEG/WebP/GIF media type; the attachment store's magic-byte and pixel validation stays authoritative. Bytes travel `ctx.fs.stat` → bounded `ctx.fs.readBytes` → `ctx.attachments.saveImage` → `fs/observed`, and the tool result is the metadata envelope plus a real `ImageBlock` — `ToolResultBlock.content` already admits image blocks, the pi-ai adapter already renders them, and the Web host's model-switch guard already scans tool results, so nothing downstream changes.
- **`FileSystem.readBytes(target, signal, maxBytes)`** is a new required provider primitive: the byte bound lives at the seam so no backend can buffer an unbounded file, with the stat-size short-circuit and a one-byte-past-cap stream guard against post-stat growth (`FS_TOO_LARGE`).
- **Registration is composition-conditional, execution is route-gated.** The tool registers only under `ctx.inject(['attachments'], …)` — no store, no tool. At execution, before any I/O, the strict gate resolves the calling route (latest `request/header` config, falling back to agent options) through `ctx.llm.resolveModelInfo` and requires `image` in `inputModalities`; unknown capability refuses. A refusal is a plain `isError` result, so a text route's durable history never acquires an image block and the session cannot brick its own route.
- **Code Mode forwards the image out-of-band**: a nested dispatch returns the canonical value (execution-local, no image block) and defers a `user`-role context message carrying the envelope and image, so the picture still reaches the next request.
- **llm-replay models may declare `inputModalities`**, which is what lets the two keyless ACP snapshots pin both sides of the gate — the sha256-referenced success on an image-capable replay route and the verbatim refusal on a text-only one.
## Alternatives considered
- **PR #598's route-scoped design** (request-ready seam, per-route schema/guidance visibility, reversible history projection) — withdrawn in favor of this note's shape. What it bought: text routes could keep running after images entered history, and the tool disappeared from prompts where it cannot succeed. What it cost: agent-loop changes, three new durable concepts (`agent/request-ready`, `messageProjection`, availability notices), and registration that churned per step. The capability itself — see an image on the next request — never needed any of it. If per-route projection becomes a real requirement, that PR's history is the reference implementation.
- **`agent.inject()` instead of the image-bearing tool result** — routes the image around the tool result as a separate injected user message. Rejected: the image *is* the tool's result; splitting them adds a second logged message with no gain, and the tool-result path already works end to end.
- **Magic-byte sniffing instead of extension declaration** — sniffing duplicates detection the attachment store already owns (sharp-backed, authoritative). The extension is only a *declaration*; a mismatch fails closed with a rename remedy rather than being silently accepted, which also keeps the model's mental map (file name ↔ content) honest.
- **Registering unconditionally and failing on a missing store** — rejected; a deployment without an attachment store cannot ever satisfy the tool, so its schema would be a standing lie. The route gate, by contrast, is per-call state and correctly lives at the execution boundary.
## Consequences
- A text-only route refuses instead of degrading: no placeholder projection means no delegated-viewing story here — that is deliberately the next PR (subagent image readback rebuilt on the current subagent seams).
- The route gate races a concurrent model switch; the Web host's image-aware switch guard covers its surface, and other front doors own their equivalent. Recorded as a tool-fs Known Limitation.
- Repeated image results accumulate request-token cost until compaction; content addressing deduplicates bytes only.
- The tool-result card renders the durable reference, not pixels; inline preview is deferred to the UI packages.

33
.agents/notes/implemented/feature/2026-08-10-minimal-read-image-tool.zh.md Normal file
View File

@@ -0,0 +1,33 @@
# Agent Note: 基于既有 seam 的最小 read_image 工具
Status: implemented
[English](2026-08-10-minimal-read-image-tool.md) | 中文
## 问题
多模态附件工作为用户上传建立了完整的持久路径：字节在所属 `user/message` 之前提交到内容寻址的附件存储，`ImageBlock` 只携带 `sha256:` 引用，pi-ai 路由在每次请求时重新读取并校验字节。但模型自己没有查看磁盘图像的手段。`read` 按约定拒绝二进制内容，因此被问到截图或渲染图表的 agent 要么失败，要么退到有损的变通做法。第一次独立尝试（PR #598）把这个问题与循环级路由作用域一起解决：新增在组装前发布确切模型模态的 `agent/request-ready` 扩展点、按路由控制 schema／指导可见性，以及可逆的 `image-placeholder-v1` 历史投影让文本路由能在占位符上继续。该设计可行，但让一个工具耦合了新的 agent-loop 机制、三个新的会话日志概念和每步的注册变动，远超这项能力本身的需要。
## 决定
只交付能把图像载入下一次请求上下文的最小工具，完全建立在既有 seam 之上；撤回的 PR #598 设计是本记录明确保留的反例。
- **`read_image` 放在 `dsh-tool-fs`**，与 `read`/`write`/`edit` 并列。扩展名选择声明的 PNG/JPEG/WebP/GIF 媒体类型；附件存储的魔数与像素校验保持权威。字节沿 `ctx.fs.stat` → 有界 `ctx.fs.readBytes` → `ctx.attachments.saveImage` → `fs/observed` 流动，工具结果是元数据信封加真正的 `ImageBlock`——`ToolResultBlock.content` 本就允许图像块，pi-ai 适配器本就会渲染它们，Web 宿主的模型切换防护本就会扫描工具结果，下游无需任何改动。
- **`FileSystem.readBytes(target, signal, maxBytes)`** 是新的必备提供方原语：字节上限放在 seam 上，任何后端都无法无界缓冲文件；stat 大小先短路，随后的流最多多读一个字节以防 stat 之后的增长（`FS_TOO_LARGE`）。
- **注册随组合条件挂载，执行按路由门禁。** 工具只在 `ctx.inject(['attachments'], …)` 作用域内注册——没有存储就没有工具。执行时在任何 I/O 之前，严格门禁通过 `ctx.llm.resolveModelInfo` 解析调用路由（最新 `request/header` 配置，缺失时回退到 agent 选项），要求 `inputModalities` 包含 `image`；能力未知即拒绝。拒绝是普通的 `isError` 结果，因此文本路由的持久历史绝不会出现图像块，会话不会毁掉自己的路由。
- **Code Mode 以带外方式转发图像**：嵌套分派返回规范值（仅限本次执行，不含图像块），并延迟提交一条携带信封和图像的 `user` 角色上下文消息，图片仍会到达下一次请求。
- **llm-replay 模型可以声明 `inputModalities`**，这正是两个 keyless ACP 快照能钉住门禁两侧的原因：图像路由上以 sha256 引用的成功结果，和纯文本路由上逐字的拒绝。
## 考虑过的替代方案
- **PR #598 的路由作用域设计**（request-ready 扩展点、按路由的 schema／指导可见性、可逆历史投影）——被本记录的形态取代后撤回。它换来的是：图像进入历史后文本路由仍能运行，工具在注定失败的提示词里消失。它付出的是：改动 agent-loop、三个新的持久概念（`agent/request-ready`、`messageProjection`、可用性通知）和每步变动的注册。而这项能力本身——下一次请求看到图像——从不需要这些。如果按路由投影将来成为真实需求，该 PR 的历史就是参考实现。
- **用 `agent.inject()` 代替带图像的工具结果**——把图像绕过工具结果，作为单独注入的用户消息。拒绝：图像就是工具的结果；拆开只会多一条无收益的日志消息，而工具结果路径本就端到端可用。
- **用魔数嗅探代替扩展名声明**——嗅探重复了附件存储已拥有的检测（基于 sharp，权威）。扩展名只是声明；不匹配时按改名修复提示失败关闭，而不是被静默接受，这也让模型对文件名与内容的对应保持诚实。
- **无条件注册、缺存储时执行报错**——拒绝；没有附件存储的部署永远无法满足该工具，其 schema 会是常态谎言。相反，路由门禁是逐调用状态，正确的位置就是执行边界。
## 后果
- 纯文本路由得到拒绝而不是降级：没有占位符投影意味着这里没有委托查看的方案——那有意留给下一个 PR（基于当前 subagent seam 重建的 subagent image readback）。
- 路由门禁与并发模型切换存在竞态；Web 宿主的图像感知切换防护覆盖其表面，其他前端拥有各自的等价防护。已记入 tool-fs 的已知限制。
- 重复的图像结果在压缩之前持续累积请求 token 成本；内容寻址只去重字节。
- 工具结果卡片渲染持久引用而非像素；内嵌预览延后到 UI 包处理。