fix(gui): harden multimodal image attachments

This commit is contained in:
Yichen Jiang
2026-07-23 19:38:37 +08:00
parent eea595fcb4
commit 580e05b794
61 changed files with 1700 additions and 214 deletions

View File

@@ -1,4 +1,4 @@
# Agent Note: Every LLM request is reconstructable from the session log
# Agent Note: Every LLM request is reconstructable from durable session state
Status: implemented
@@ -12,7 +12,7 @@ The reference shape for the happy path is MiniCode's `LLMClient`: a stateful con
### The principle
**Model-visible ⟺ logged.** Anything that reaches a model request must be recorded in the session log. The checkable consequence: **every conversation request the loop sends is a pure function of the session log** — anyone holding the log reconstructs it byte-for-byte. Scope, stated precisely: the guarantee covers the loop-built `GenerateOptions`; provider wire bytes follow from it because both adapters' serialization is a pure per-message function at a pinned code version; direct one-shots (compaction's summarize call) log their envelope scalars (`compact/summary.{provider, model, maxTokens}`) and their input is deterministic code over the logged region — reconstructable from log + code, outside the invariant because only the loop marks request ownership.
**Model-visible ⟺ durably recorded.** Text and structured references that reach a model request are recorded in the session log; referenced binary bytes live in immutable content-addressed attachment storage and are integrity-checked against the logged metadata. The checkable consequence: **every conversation `GenerateOptions` the loop sends is a pure function of the session log**, while provider wire bytes are a function of that envelope plus the immutable attachment objects at a pinned code version. Text-only adapter serialization remains pure in-memory conversion; image-capable adapters perform verified `readImage()` I/O during serialization. Direct one-shots such as compaction log their envelope scalars (`compact/summary.{provider, model, maxTokens}`), derive input deterministically from the logged region, and resolve any referenced images through the same durable store; they remain outside the loop invariant because only the loop marks request ownership.
Prefix-cache stability is corollary #1, not the headline: an append-only log projected by a per-node pure function yields requests that are append-extensions of their predecessors whenever the header is unchanged — stability is emergent, not managed. Byte-exact audit/replay is corollary #2; resume and fork with *attributable* drift is corollary #3.
@@ -26,7 +26,7 @@ Each step rebuilds prompt assembly. On the instance's first step, `agent/session
**`step/start` is the reconstruction boundary.** A step derives messages from events before that sequence. Injection after the snapshot joins the next request, and reentrant appends are rejected during event publication. `agent/pre-step(agent, turn, step, signal)` remains the generic seam for content needed by the current request. Header reconstruction selects the step's `request/header`, or carries the prior snapshot when no new header is written.
**Enforcement.** The `dsh-agent-loop/invariant` companion registers with `ctx.invariants` and, when selected, independently rebuilds each loop request through a fresh `Session`, so the live cache cannot vouch for itself, then compares messages and folded header fields at `llm/stream`. The loop records the exact frozen request through `markAgentLoopRequest()` in `dsh-llm`; the process-local identity lets the companion and other request observers recognize conversation work, while direct one-shots remain excluded regardless of their frozen shape or session id. Correctness depends on sequence-bounded reconstruction rather than listener order. A with-key e2e requires positive cache-read tokens after the first request; per-step usage is the production signal, and a header change or compaction appears as a cache-read drop on the next step.
**Enforcement.** The `dsh-agent-loop/invariant` companion registers with `ctx.invariants` and, when selected, independently rebuilds each loop request through a fresh `Session`, so the live cache cannot vouch for itself, then compares messages and folded header fields at `llm/stream`. Attachment backends separately verify referenced bytes against their content digest and logged media metadata before an adapter may serialize them. The loop records the exact frozen request through `markAgentLoopRequest()` in `dsh-llm`; the process-local identity lets the companion and other request observers recognize conversation work, while direct one-shots remain excluded regardless of their frozen shape or session id. Correctness depends on sequence-bounded reconstruction rather than listener order. A with-key e2e requires positive cache-read tokens after the first request; per-step usage is the production signal, and a header change or compaction appears as a cache-read drop on the next step.
### The MiniCode shape: adopted, with the provenance arrow inverted
@@ -44,7 +44,7 @@ Like MiniCode, the conversation advances append-only and resets only when model-
## Consequences
- A request that is not explained by the log cannot be constructed by accident — not by the loop, not by a listener; mutating a built request throws; every header change is a durable, diffable log event.
- A request envelope that is not explained by the log cannot be constructed by accident — not by the loop, not by a listener; mutating a built request throws; every header change is a durable, diffable log event. Exact replay of an image request additionally requires the immutable object named by its logged attachment reference.
- Choosing between the advisory channels is a change-frequency decision, and the design makes the stable one structural: an `agent/session-prefix` contribution is composed once per loop instance and reused verbatim, so it extends the cacheable prefix at zero marginal cost and CANNOT bust the provider cache mid-session; content that changes mid-session flows through the append-only history channels — `agent.inject()` and tool/prompt-submit `additionalContexts` — each a durable `context/message` paid once and prefix-cached thereafter, at the price of accumulating in history and the log. Route session-frozen openers to the prefix and change notices to the history channels; a per-step request-only tail slot was deliberately dropped (no consumer, and a durable append covers every current update pattern).
- What still costs full price at the provider is inherent and logged: compaction (its `compact/*` events and replacement entry), a real prompt, tool, or config change (`request/header` with reason `change`), or a process boundary with drift (a differing `resume` snapshot). The provider's own reasoning-content exclusion is managed server-side.
- The `step/start`-listener behavior change (above) is the one observable semantics change for plugins; `agent/pre-step` is the current-request seam.

View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write
2026-07-22-web-multimodal-image-input-and-durable-attachments.md: 14ad4e5ceb28fd7df8b639db403be8accd8c2028
2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md: 5752247b080ee1be0b0bfbbf7a9fb7486ff6c41b
2026-07-22-web-multimodal-image-input-and-durable-attachments.md: 156bec4a4f9bb05490847cbe775368bc472a33e9
2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md: fb0f8d3031bb2d0e6af97136892cd07c22d575b3

View File

@@ -1,12 +1,12 @@
# Agent Note: Web multimodal image input and durable attachments
Status: proposed
Status: implemented
English | [中文](2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md)
## Problem
The Web composer accepts only text: `InputBar` receives a string draft, `ConversationService.send()` creates text content, and the host forwards that content to the agent. Users cannot paste an image, inspect it before sending, submit an image-only prompt, or recover sent images from history.
Before this change, the Web composer accepted only text: `InputBar` received a string draft, `ConversationService.send()` created text content, and the host forwarded that content to the agent. Users could not paste an image, inspect it before sending, submit an image-only prompt, or recover sent images from history.
This is not only a composer gap. Core needs a durable image content block, providers need explicit modality handling, and the session log must reconstruct everything visible to a model. [The previous image-block removal](../../implemented/simplification/2026-07-04-drop-image-content-block.md) rejected a partial design that could silently lose or flatten images. A browser object URL, local path, provider URL, or base64 payload cannot be canonical session content.
@@ -14,19 +14,19 @@ The [Web client architecture](../../implemented/architecture/2026-07-19-gui-web-
Peer products converge on an attachment rail above the editor, but their storage choices differ. Codex-style paths such as `/var/folders/.../codex-clipboard-*.png` are reasonable intake staging locations, not durable message identities: the operating system may delete them, another host cannot read them, and a resumed session cannot rely on them.
## Proposal
## Decision
Add pasted or dropped raster images to the Web composer as the first consumer of a durable attachment capability. Unsent files remain temporary client-owned draft state. The host validates and durably commits every accepted user image before appending its message event. A provider adapter that produces structured image output must durably commit the output before appending its assistant block. Canonical user and assistant content contains only role-neutral `ImageBlock` references.
Pasted or dropped raster images are the Web composer's first consumer of a durable attachment capability. Unsent files remain temporary client-owned draft state. The host validates and durably commits every accepted user image before appending its message event. A provider adapter that produces structured image output must durably commit the output before appending its assistant block. Canonical user and assistant content contains only role-neutral `ImageBlock` references.
Version one supports PNG, JPEG, WebP, and GIF paste and drag-and-drop, image-only or mixed prompts, historical user and assistant image rendering, and original-image preview on double-click. File picking, generic files, PDF, audio, video, image copying, and a custom context menu are separate follow-ups.
Version one supports PNG, JPEG, WebP, and GIF paste and drag-and-drop, image-only or mixed prompts, historical user and assistant image rendering, and original-image preview on double-click. File picking, generic files, PDF, audio, video, image copying, and a custom context menu remain separate follow-ups.
### Product behavior
- Pasting or dropping one or more supported images adds ordered thumbnails above the textarea without inserting placeholder text. Dragging files over the composer highlights the drop target.
- The rail is shared by the empty-state and resident composers, is hidden when empty, and scrolls horizontally instead of widening the composer.
- Each approximately 72-by-72-pixel thumbnail has a remove action and opens its original draft image on double-click.
- A prompt may contain text and images or images only. Pure text paste remains native browser behavior; the paste handler prevents the default only when it accepts an image file. File drops on the composer always prevent browser navigation, accept supported images, and report unsupported files locally.
- A failed send restores the complete text and image draft. Removal, successful send, and session-scope disposal revoke obsolete object URLs.
- A prompt may contain text and images or images only. Pure text paste remains native browser behavior; mixed clipboard content inserts its text normally while adding its files to the rail, and file-only paste prevents default browser handling. File drops on the composer always prevent browser navigation and report unsupported files locally.
- A failed send restores the complete text and image draft. Removal, successful send, empty-state disposal, rendered-session disposal, and application disposal revoke the object URLs they own.
- Historical user and assistant images use one `MessageImage` control. Inline images preserve intrinsic aspect ratio, do not upscale, and stay within a 240-by-240-pixel box.
- Double-clicking a message image opens the stored original in a viewport-bounded modal. Escape, the close control, and backdrop activation close it and restore focus.
- Version one does not override the browser context menu and provides no explicit image-copy action.
@@ -54,6 +54,7 @@ interface ChatStoreState {
}
interface ComposerAttachment {
kind: 'image'
id: string
file: File
previewUrl: string
@@ -64,7 +65,7 @@ This split uses the slots framework's store seat and bound actions as the single
The local attachment backend resolves an explicit `dshHome`, then `$DSH_HOME`, then `~/.dsh`. It stores content-addressed objects below `$DSH_HOME/attachments/v1/objects/<prefix>/<sha256>` with owner-only directory and file permissions. A temporary file is written, synchronized, and atomically published before the service returns a reference. The content digest is encoded in the opaque `sha256:<digest>` identifier, and every read verifies the digest, media type, byte length, width, and height.
The store performs no automatic deletion in version one. Sent user images and model-generated images remain reachable for history, resume, and fork. Reference-aware garbage collection needs a separate design because an age-only rule can delete data still referenced by a durable session.
The store performs no automatic deletion in version one. Sent user images and model-generated images remain reachable for history, resume, and fork. Reference-aware garbage collection needs a separate design because an age-only rule can delete data still referenced by a durable session. Deployment byte and pixel limits are admission policy on writes; reads verify the digest and recorded metadata without reapplying current admission limits, so lowering policy does not invalidate older history.
### Durable content and prompt wire
@@ -90,7 +91,7 @@ interface ImageBlock {
}
```
`ImageBlock` joins the merge-extensible core `ContentBlockMap` and is valid in either user or assistant content. It never carries base64, an object URL, a filesystem path, or a provider-owned locator. This keeps the session event plus immutable object store sufficient to reconstruct the exact model-visible image.
`ImageBlock` joins the merge-extensible core `ContentBlockMap` and is valid in either user or assistant content. It never carries base64, an object URL, a filesystem path, or a provider-owned locator. This keeps the session event plus immutable object store sufficient to reconstruct the exact model-visible image. The LLM vocabulary therefore has a type-only dependency on the attachment seam; provider runtime dependencies remain adapter-specific.
The browser cannot mint a durable reference, so `session.prompt` accepts a narrow intake union rather than canonical `ContentBlock[]`:
@@ -109,29 +110,31 @@ type PromptInputPart =
Base64 crosses JSON-RPC once and is discarded after persistence. The host validates canonical base64, image count, aggregate bytes, individual bytes, magic-byte MIME, intrinsic dimensions, and decoded-pixel count. Only after every image succeeds does it call the agent with normalized text and durable image blocks. A failure appends no user event and exposes no attachment path or raw bytes.
`session.attachment` is a read-only, session-scoped endpoint. The host serves bytes only when a durable event in that session references the requested attachment identifier. The client caches the resulting object URL by session and attachment identifier for its service lifetime and revokes it on disposal.
`session.attachment` is a read-only, session-scoped endpoint. The host serves bytes only when a durable event in that session references the requested attachment identifier. The client deduplicates loads by session and attachment identifier while that session is rendered, revokes resolved URLs on rendered-session disposal, and invalidates late loads so an unmounted session cannot repopulate the cache.
### Model capabilities and provider behavior
Model catalog entries gain optional merge-extensible input and output modality declarations. A missing declaration means unknown; a present list without `image` is an explicit negative capability.
The host is the authoritative preflight boundary. If the selected model explicitly excludes image input, it rejects the prompt before writing any attachment or event, and the client restores the draft. Unknown capability proceeds to the adapter guard so uncatalogued model identifiers remain usable. Immediate intake-time rejection in the UI may be added after model selection is exposed consistently across every Web entry path.
The host is the authoritative preflight boundary. It resolves the session's latest routed provider/model, falling back through agent options to host defaults; if that model explicitly excludes image input, it rejects the prompt before writing any attachment or event, and the client restores the draft. Unknown capability proceeds to the adapter guard so uncatalogued model identifiers remain usable. `host.describe` projects the default model and image limits into `SessionsService`; both composers use the limits before allocating object URLs or base64, while only the no-session composer uses the default model for early explicit text-only feedback. Decoded-pixel validation and every resident session's actual route remain authoritative on the host.
The Pi-AI adapter is the first visual-input route: it resolves each durable reference through `ctx.attachments` and emits native image content only for models that declare image input. The hand-written DeepSeek adapter throws typed `UNSUPPORTED_CONTENT` for an image anywhere in the request, including nested tool results. No adapter may flatten or skip an image.
The Pi-AI adapter is the first visual-input route: it resolves `ctx.attachments` at request time, then resolves each durable reference and emits native image content only for models that declare image input. Request-time service resolution keeps Cordis load order from freezing optional attachment availability. The hand-written DeepSeek adapter throws typed `UNSUPPORTED_CONTENT` for an image anywhere in the request, including nested tool results. No adapter may flatten or skip an image.
Core supports structured assistant image blocks, but no current production provider route is certified for image output. Any future output-capable adapter must retrieve provider bytes under bounded size and time policy, validate them through the same attachment service, persist them, and only then publish the atomic `ImageBlock`. A URL in assistant Markdown remains text and is never downloaded automatically.
Token estimation accounts for image dimensions without counting base64 or attachment locators as text. Provider-reported usage remains authoritative. ACP renders an explicit image marker until that protocol surface gains native image support rather than silently omitting the block.
Compaction replays the selected conversation prefix, including image references, into the configured summarization route. A visual-capable route resolves those references through its adapter; a text-only route fails explicitly instead of silently dropping the visual context. The synthesized checkpoint remains text-only, and `compact-basic` rejects image summary output with `UNSUPPORTED_CONTENT`.
### History rendering and original preview
History folding preserves `ImageBlock` in both user and assistant messages. User images align to the trailing edge above their text; assistant images align to the leading narration flow. `MessageImage` derives a stable inline box from recorded dimensions, resolves bytes through the session-authorized loader, uses `object-fit: contain`, and turns a missing or corrupt object into a retryable error control.
History folding preserves `ImageBlock` in both user and assistant messages. User images align to the trailing edge above their text; assistant images remain in their original content-block position in the leading narration flow. `MessageImage` derives a stable inline box from recorded dimensions, resolves bytes through the session-authorized loader, uses `object-fit: contain`, and turns a missing or corrupt object into a retryable error control.
Composer thumbnails and each `MessageImage` own ephemeral original-preview state and invoke the same pure `ImageLightbox`. The modal uses the already resolved original object URL, constrains only display size, focuses its close control, and restores the previous focus target when closed.
### Limits and trust boundaries
Version one accepts PNG, JPEG, WebP, and GIF only. SVG and remote URLs are excluded. Default limits are 5 MiB per image, 10 images and 20 MiB aggregate image bytes per message, and 40 million intrinsic pixels per image. These deployment-varying limits are validated backend configuration and are projected to the client for fast-path guidance; host validation remains authoritative.
Version one accepts PNG, JPEG, WebP, and GIF only. SVG and remote URLs are excluded. Default limits are 5 MiB per image, 10 images and 20 MiB aggregate image bytes per message, and 40 million intrinsic pixels per image. These deployment-varying limits are validated backend configuration and are projected to the client for fast-path guidance; host validation remains authoritative. The Web carrier independently caps buffered API request bodies, with `dsh web` deriving its default from the aggregate image limit plus base64/envelope expansion and allowing an explicit override.
Malformed base64, unsupported or mismatched media, truncated headers, excess bytes, excess image count, excess pixels, missing objects, and integrity mismatches return stable structured failures. Original filenames are reduced to a display basename, control characters are removed, and no local path is logged or returned to the browser.
@@ -144,19 +147,18 @@ Malformed base64, unsupported or mismatched media, truncated headers, excess byt
| `packages/llm/llm` and `packages/llm/token-meter` | Role-neutral `ImageBlock`, modality metadata, and image cost estimation. |
| `packages/llm/llm-pi-ai` | Resolve durable supported image input into native provider content. |
| `packages/llm/llm-deepseek` | Reject image content explicitly. |
| `packages/compact/compact-basic` | Preserve images in summary input and reject non-text checkpoint output explicitly. |
| `packages/host/apiproxy` and `packages/host/runtime` | Narrow upload wire, persist-before-event ordering, session-authorized reads, limits, and model preflight. |
| `packages/host/webserver` | Bound buffered API request bodies before the fetch carrier. |
| `packages/client/connection` and `packages/client/runtime` | Wire types, fixture images, prompt uploads, attachment reads, and durable-reference folding. |
| `packages/client/ui-conversation` | Per-session draft images, attachment rail, user and assistant image controls, and original preview. |
| `packages/ui/acp` | Explicit fallback rendering for image blocks. |
The attachment packages form the interface/implementation side of one capability seam. Composer behavior stays in the conversation object layer, provider conversion stays in adapters, and no change is required in `agent-loop`.
### Delivery
### Implementation
1. Land the attachment seam, role-neutral image block, image-aware token estimation, Pi-AI input conversion, DeepSeek rejection, and durable host ordering.
2. Land the Web upload/read protocol, in-memory draft images, paste/drop rail, user and assistant history rendering, double-click preview, and assembled keyless Web coverage.
3. Add immediate intake-time capability feedback when active model selection is consistently available to the composer.
4. Propose file picking, generic files/PDF, audio/video, durable draft staging, output-provider certification, and reference-aware garbage collection independently.
The implemented slice includes the attachment seam, role-neutral image block, image-aware token estimation, Pi-AI input conversion, DeepSeek rejection, durable host ordering, Web upload/read protocol, host-capability projection, bounded Web request bodies, in-memory draft images, paste/drop rail, user and assistant history rendering, double-click preview, compaction handling, and keyless assembled Web coverage.
No compatibility shim is required for the pre-release prompt wire; all call sites and fixtures change with the introducing slice.
@@ -186,20 +188,16 @@ Composer presentation can use a generic attachment rail, but provider semantics
UI state can be stale and does not protect direct SDK, ACP, replay, or uncatalogued model paths. Silent filtering changes user intent. Provider enforcement remains mandatory, while UI checks are optional earlier feedback.
## Acceptance criteria
## Testing
- Pasting or dropping one or more supported images shows ordered removable thumbnails above both composer variants without changing textarea text; drag-over highlights the target, unsupported drops cannot navigate away, and image-only send works.
- Unsent browser images exist only as `File` and object URLs, survive session switches in memory, do not enter `localStorage`, and are revoked after removal, accepted send, or service disposal.
- Every accepted user image is committed below resolved `DSH_HOME` before its `user/message` event. The event contains only `ImageBlock` references and never base64 or temporary paths.
- Structured assistant images can be represented only by a durable `ImageBlock`; a future output adapter must persist bytes before emitting the assistant event, while Markdown image URLs remain text.
- Cold history renders user and assistant image references through the same bounded control. Double-click opens the original; Escape, backdrop, and close control dismiss it without a custom context menu.
- Session attachment reads fail unless the same session log references the identifier. Missing or corrupt objects fail explicitly and never return unverified bytes.
- Pi-AI emits native input images for a compatible route. DeepSeek and every non-implementing consumer return an explicit unsupported-content failure rather than dropping the block.
- An explicitly text-only active model rejects image send before attachment persistence or session event append; unknown metadata still reaches adapter enforcement and a failed send restores the draft.
- Keyless unit, host integration, client integration, and assembled Chromium coverage exercise persistence ordering, absence of base64 in logs, authorization, paste and drop, image-only send, historical user and assistant images, original preview, and object-URL cleanup.
- The current production adapter set declares text-only output; output-provider certification, file picking, non-image files, video, persistent drafts, and garbage collection remain outside version one.
- Storage tests cover content-addressed deduplication, private permissions, admission failures, corruption/missing-object failures, and reading history after deployment limits are lowered.
- Host and protocol tests cover persist-before-event ordering, absence of base64 in logs, session-scoped authorization, capability rejection, upload limits, and bounded HTTP request bodies.
- Client unit and assembled Chromium tests cover paste and drop, mixed clipboard text, image-only send, draft restoration, historical user and assistant images, original preview, ordering, and draft/session/application object-URL cleanup.
- Adapter and compaction tests cover native Pi-AI image conversion, late attachment-service composition, text-only rejection, nested tool-result images, preserved summary input, and explicit image-output rejection.
- A credentialed real-API test sends a PNG through the Anthropic `claude-opus-4-8` route and requires the model to identify its QR code.
- The current production adapter set declares text-only output; output-provider certification remains outside version one.
## Risks
## Consequences
- Durable storage grows without garbage collection. Version one chooses replay safety over premature deletion.
- A missing or corrupt object makes exact model reconstruction fail. Failing loud preserves integrity but may prevent that session from continuing until repaired.
@@ -208,3 +206,4 @@ UI state can be stale and does not protect direct SDK, ACP, replay, or uncatalog
- Original preview decodes more pixels than the inline control displays. Pixel limits, one clicked preview, and object-URL disposal bound but do not eliminate transient browser memory.
- Capability metadata may be missing or stale. Host preflight improves feedback, while adapter enforcement remains authoritative.
- A future output provider may require authenticated retrieval before an assistant image can complete, adding latency and a new failure point. Persist-before-event ordering favors replay integrity.
- File picking, generic files/PDF, audio/video, durable draft staging, image copying, custom context menus, output-provider certification, and reference-aware garbage collection remain independent designs.

View File

@@ -1,12 +1,12 @@
# Agent Note: Web 多模态图片输入与持久附件
Status: proposed
Status: implemented
[English](2026-07-22-web-multimodal-image-input-and-durable-attachments.md) | 中文
## 问题
Web 输入区目前仅接受文本:`InputBar` 接收字符串草稿,`ConversationService.send()` 创建文本内容,宿主再把该内容转发给 agent(智能体)。用户无法粘贴图片、在发送前查看图片、提交仅含图片的提示词,也无法从历史记录中恢复已发送图片。
在此变更之前,Web 输入区仅接受文本:`InputBar` 接收字符串草稿,`ConversationService.send()` 创建文本内容,宿主再把该内容转发给 agent(智能体)。用户无法粘贴图片、在发送前查看图片、提交仅含图片的提示词,也无法从历史记录中恢复已发送图片。
这不只是输入区功能缺失。核心层需要持久图片内容块,提供方需要明确处理模态,会话日志则必须重建模型可见的全部内容。[此前移除图片块的决策](../../implemented/simplification/2026-07-04-drop-image-content-block.md)否决了可能静默丢失图片或将其展平的不完整设计。浏览器对象 URL、本地路径、提供方 URL 或 base64 数据都不能成为规范会话内容。
@@ -14,19 +14,19 @@ Web 输入区目前仅接受文本:`InputBar` 接收字符串草稿,`Convers
同类产品普遍在编辑器上方设置附件栏,但存储方案各不相同。诸如 `/var/folders/.../codex-clipboard-*.png` 的 Codex 式路径适合作为接收输入时的暂存位置,却不能作为持久消息身份:操作系统可能删除文件,另一台宿主无法读取文件,恢复后的会话也不能依赖文件仍然存在。
## 提案
## 决策
把粘贴或拖放的光栅图片作为持久附件能力的首个消费方,接入 Web 输入区。未发送文件仍是由客户端持有的临时草稿状态。宿主在追加相应消息事件前,校验并持久提交每张已接受的用户图片。生成结构化图片输出的提供方适配器在追加相应助手块前,也必须持久提交输出。规范用户内容与助手内容只包含角色无关的 `ImageBlock` 引用。
粘贴或拖放的光栅图片是 Web 输入区对持久附件能力的首个应用场景。未发送文件仍是由客户端持有的临时草稿状态。宿主在追加相应消息事件前,校验并持久提交每张已接受的用户图片。生成结构化图片输出的提供方适配器在追加相应助手块前,也必须持久提交输出。规范用户内容与助手内容只包含角色无关的 `ImageBlock` 引用。
第一版支持粘贴和拖放 PNG、JPEG、WebP 与 GIF,支持仅图片或混合提示词,支持渲染历史用户图片与助手图片,并支持双击预览原图。文件选择、通用文件、PDF、音频、视频、图片复制和自定义上下文菜单分别作为后续工作。
第一版支持粘贴和拖放 PNG、JPEG、WebP 与 GIF,支持仅图片或混合提示词,支持渲染历史用户图片与助手图片,并支持双击预览原图。文件选择、通用文件、PDF、音频、视频、图片复制和自定义上下文菜单仍分别作为后续工作。
### 产品行为
- 粘贴或拖放一张或多张受支持的图片后,文本框上方会按顺序显示缩略图,但不会插入占位文本。文件拖入输入区时会高亮放置目标。
- 空状态输入区与常驻输入区共用附件栏;附件栏为空时隐藏,通过横向滚动避免撑宽输入区。
- 每个缩略图约为 72 × 72 像素,带有移除操作;双击时打开草稿原图。
- 提示词可同时包含文本与图片,也可仅包含图片。粘贴纯文本时保持浏览器原生行为;粘贴处理器只有接受图片文件后才阻止默认行为。无论文件是否受支持,在输入区放置文件时都会阻止浏览器导航;系统会接受受支持的图片,并在本地提示哪些文件不受支持。
- 发送失败时恢复完整的文本与图片草稿。移除、发送成功和会话作用域释放都会撤销过期的对象 URL。
- 提示词可同时包含文本与图片,也可仅包含图片。粘贴纯文本时保持浏览器原生行为;粘贴混合的剪贴板内容时,文本会正常插入,文件则同时添加到附件栏;仅粘贴文件时才阻止浏览器的默认处理。在输入区放置文件时总会阻止浏览器导航,并在本地报告不受支持的文件。
- 发送失败时恢复完整的文本与图片草稿。移除、发送成功、空状态释放、已渲染会话释放和应用释放都会撤销各自持有的对象 URL。
- 历史用户图片与助手图片共用一个 `MessageImage` 控件。行内图片保持固有宽高比、不放大,并限制在 240 × 240 像素的边界框内。
- 双击消息图片会在不超出视口的模态框中打开存储的原图。按 Escape、激活关闭控件或激活背景区域都会关闭模态框并恢复焦点。
- 第一版不覆盖浏览器上下文菜单,也不提供明确的图片复制操作。
@@ -54,6 +54,7 @@ interface ChatStoreState {
}
interface ComposerAttachment {
kind: 'image'
id: string
file: File
previewUrl: string
@@ -64,7 +65,7 @@ interface ComposerAttachment {
本地附件后端依次解析显式 `dshHome`、`$DSH_HOME` 和 `~/.dsh`。它把内容寻址对象存储在 `$DSH_HOME/attachments/v1/objects/<prefix>/<sha256>` 下,并为目录和文件设置仅所有者可访问的权限。服务先写入并同步临时文件,再以原子方式发布,之后才返回引用。内容摘要编码在不透明的 `sha256:<digest>` 标识符中,每次读取都会校验摘要、媒体类型、字节长度、宽度和高度。
第一版不对存储执行自动删除。已发送的用户图片和模型生成图片会一直保留,以供历史记录、恢复和 fork 使用。按引用感知的垃圾回收需要单独设计,因为仅按时间清理可能删除仍被持久会话引用的数据。
第一版不对存储执行自动删除。已发送的用户图片和模型生成图片会一直保留,以供历史记录、恢复和 fork 使用。按引用感知的垃圾回收需要单独设计,因为仅按时间清理可能删除仍被持久会话引用的数据。部署的字节和像素限制是写入时的准入策略;读取时会校验摘要和已记录的元数据,但不重新应用当前准入限制,因此收紧策略不会导致旧历史记录失效。
### 持久内容与提示词协议
@@ -90,7 +91,7 @@ interface ImageBlock {
}
```
`ImageBlock` 加入可合并扩展的核心 `ContentBlockMap`,在用户内容和助手内容中都有效。它绝不携带 base64、对象 URL、文件系统路径或提供方持有的定位符。因此,会话事件与不可变对象存储足以共同重建模型可见的确切图片。
`ImageBlock` 加入可合并扩展的核心 `ContentBlockMap`,在用户内容和助手内容中都有效。它绝不携带 base64、对象 URL、文件系统路径或提供方持有的定位符。因此,会话事件与不可变对象存储足以共同重建模型可见的确切图片。LLM 词汇由此仅在类型层面依赖附件服务边界;提供方运行时依赖仍由各适配器持有。
浏览器无法生成持久引用,因此 `session.prompt` 接受范围狭窄的接收联合类型,而不是规范 `ContentBlock[]`:
@@ -109,29 +110,31 @@ type PromptInputPart =
Base64 只跨越一次 JSON-RPC,并在持久化后丢弃。宿主会校验规范 base64、图片数量、总字节数、单张图片字节数、与魔数字节一致的 MIME、固有尺寸和解码像素数。只有每张图片都成功后,宿主才会用规范化文本和持久图片块调用 agent。失败时不追加用户事件,也不公开任何附件路径或原始字节。
`session.attachment` 是只读且限定于会话作用域的端点。只有该会话中的持久事件引用了所请求的附件标识符,宿主才提供字节。客户端在服务生命周期内,以会话和附件标识符为键缓存生成的对象 URL,并在释放时撤销它。
`session.attachment` 是只读且限定于会话作用域的端点。只有该会话中的持久事件引用了所请求的附件标识符,宿主才提供字节。会话处于渲染状态时,客户端会按会话和附件标识符对加载操作去重;已渲染会话释放时会撤销已解析的 URL,并使延迟完成的加载失效,以免已卸载的会话重新写入缓存。
### 模型能力与提供方行为
模型目录项增加可选且可合并扩展的输入与输出模态声明。缺少声明表示未知;声明存在但不含 `image`,则明确表示不支持图片。
宿主是权威的前置检查边界。如果所选模型明确排除图片输入,宿主会在写入任何附件或事件前拒绝提示词,客户端则恢复草稿。能力未知时继续进入适配器强制检查,使未收录的模型标识符仍然可用。在所有 Web 入口路径都能一致公开模型选择后,可以在 UI 中增加粘贴时立即拒绝的反馈。
宿主是权威的前置检查边界。它会解析会话最新路由到的提供方和模型,并在缺失时依次回退到 agent 选项和宿主默认值;如果该模型明确排除图片输入,宿主会在写入任何附件或事件前拒绝提示词,客户端则恢复草稿。能力未知时继续进入适配器强制检查,使未收录的模型标识符仍然可用。`host.describe` 把默认模型和图片限制投影到 `SessionsService`;两种输入区都会在分配对象 URL 或 base64 前使用这些限制,只有无会话输入区会使用默认模型,针对明确仅支持文本的情况提前反馈。解码像素校验与每个常驻会话的实际路由均由宿主作出权威判定。
Pi-AI 适配器是首条视觉输入路径:它通过 `ctx.attachments` 解析每个持久引用,并且仅为声明支持图片输入的模型生成提供方原生图片内容。手写 DeepSeek 适配器遇到请求中任何位置的图片时都会抛出类型化的 `UNSUPPORTED_CONTENT` 错误,包括嵌套工具结果中的图片。任何适配器都不得将图片展平或跳过。
Pi-AI 适配器是首条视觉输入路径:它在请求时解析 `ctx.attachments`,再解析每个持久引用,并且仅为声明支持图片输入的模型生成提供方原生图片内容。在请求时解析服务,可避免 Cordis 加载顺序将可选附件服务的可用性固化。手写 DeepSeek 适配器遇到请求中任何位置的图片时都会抛出类型化的 `UNSUPPORTED_CONTENT` 错误,包括嵌套工具结果中的图片。任何适配器都不得将图片展平或跳过。
核心层支持结构化助手图片块,但当前没有任何生产提供方路径通过图片输出认证。未来任何支持输出的适配器都必须在有界的大小和时间策略下获取提供方字节,通过同一个附件服务校验并持久化字节,之后才能以原子方式发布 `ImageBlock`。助手 Markdown 中的 URL 仍是文本,绝不自动下载。
token 估算根据图片尺寸计量,但不把 base64 或附件定位符作为文本计数。提供方返回的用量仍是权威值。在 ACP(Agent Client Protocol)接口原生支持图片前,ACP 会渲染明确的图片标记,而不是静默省略该块。
压缩(compaction)会把选定的会话前缀(包含图片引用)回放到已配置的摘要生成路径中。支持视觉的路径会通过适配器解析这些引用;仅文本路径会明确失败,而不是静默丢弃视觉上下文。合成的检查点仍仅包含文本,`compact-basic` 会以 `UNSUPPORTED_CONTENT` 拒绝包含图片的摘要输出。
### 历史渲染与原图预览
历史记录折叠会在用户消息和助手消息中保留 `ImageBlock`。用户图片在文本上方靠尾端对齐;助手图片在叙述流中靠前端对齐。`MessageImage` 根据记录的尺寸派生稳定的行内边界框,通过会话授权加载器解析字节,使用 `object-fit: contain`,并将对象缺失或损坏转换为可重试的错误控件。
历史记录折叠会在用户消息和助手消息中保留 `ImageBlock`。用户图片在文本上方靠尾端对齐;助手图片则保留在靠前端叙述流中的原内容块位置。`MessageImage` 根据记录的尺寸派生稳定的行内边界框,通过会话授权加载器解析字节,使用 `object-fit: contain`,并将对象缺失或损坏转换为可重试的错误控件。
输入区缩略图和每个 `MessageImage` 各自持有临时原图预览状态,并调用同一个纯 `ImageLightbox`。模态框使用已经解析的原始对象 URL,只限制显示尺寸;它会聚焦关闭控件,并在关闭时把焦点恢复到先前的目标。
### 限制与信任边界
第一版仅接受 PNG、JPEG、WebP 和 GIF。不接受 SVG 和远程 URL。默认限制为每张图片 5 MiB、每条消息 10 张图片和 20 MiB 图片总字节数,以及每张图片 4,000 万个固有像素。这些随部署变化的限制属于经过校验的后端配置,并会投影给客户端以提供快速路径指引;宿主校验仍是权威结果。
第一版仅接受 PNG、JPEG、WebP 和 GIF。不接受 SVG 和远程 URL。默认限制为每张图片 5 MiB、每条消息 10 张图片和 20 MiB 图片总字节数,以及每张图片 4,000 万个固有像素。这些随部署变化的限制属于经过校验的后端配置,并会投影给客户端以提供快速路径指引;宿主校验仍是权威结果。Web 载体会独立限制 API 请求体的缓冲大小;`dsh web` 根据图片总量限制加上 base64 和请求封装的膨胀量推导默认值,并允许显式覆盖。
格式错误的 base64、不支持或不匹配的媒体、截断的文件头、超出字节限制、超出图片数量、超出像素限制、对象缺失和完整性不匹配都会返回稳定的结构化错误。原始文件名只保留用于显示的末段,控制字符会被移除,并且任何本地路径都不会写入日志或返回浏览器。
@@ -144,19 +147,18 @@ token 估算根据图片尺寸计量,但不把 base64 或附件定位符作为
| `packages/llm/llm` 和 `packages/llm/token-meter` | 角色无关的 `ImageBlock`、模态元数据和图片成本估算。 |
| `packages/llm/llm-pi-ai` | 将持久且受支持的图片输入解析为提供方原生内容。 |
| `packages/llm/llm-deepseek` | 明确拒绝图片内容。 |
| `packages/compact/compact-basic` | 在摘要输入中保留图片,并明确拒绝非文本检查点输出。 |
| `packages/host/apiproxy` 和 `packages/host/runtime` | 范围狭窄的上传协议、先持久化再追加事件的顺序、会话授权读取、限制和模型前置检查。 |
| `packages/host/webserver` | 在 fetch 载体之前限制 API 请求体的缓冲大小。 |
| `packages/client/connection` 和 `packages/client/runtime` | 协议类型、fixture(测试前置数据)图片、提示词上传、附件读取和持久引用折叠。 |
| `packages/client/ui-conversation` | 每个会话的草稿图片、附件栏、用户与助手图片控件和原图预览。 |
| `packages/ui/acp` | 图片块的明确兜底渲染。 |
附件包(package)构成一个能力服务边界的接口与实现侧。输入区行为留在会话对象层,提供方转换留在适配器中,无需修改 `agent-loop`。
### 交付
### 实现
1. 交付附件服务边界、角色无关的图片块、图片感知的 token 估算、Pi-AI 输入转换、DeepSeek 拒绝和宿主侧的持久化顺序。
2. 交付 Web 上传与读取协议、内存草稿图片、支持粘贴与拖放的附件栏、用户与助手历史图片渲染、双击预览,以及组装后无需密钥的 Web 覆盖。
3. 在输入区能够一致获取当前模型选择后,增加接收图片时立即提供的能力反馈。
4. 分别为文件选择、通用文件与 PDF、音频与视频、持久草稿暂存、输出提供方认证和按引用感知的垃圾回收提出方案。
已实现的范围包括附件服务边界、角色无关的图片块、图片感知的 token 估算、Pi-AI 输入转换、DeepSeek 拒绝、宿主持久化顺序、Web 上传与读取协议、宿主能力投影、大小受限的 Web 请求体、内存草稿图片、粘贴与拖放附件栏、用户与助手历史图片渲染、双击预览、压缩处理,以及组装后无需密钥的 Web 覆盖。
预发布提示词协议不需要兼容包装层;引入相应切片时会同时修改所有调用点和 fixture。
@@ -186,20 +188,16 @@ token 估算根据图片尺寸计量,但不把 base64 或附件定位符作为
UI 状态可能陈旧,也无法保护直接 SDK、ACP、回放或未收录模型的路径。静默过滤会改变用户意图。提供方强制检查仍是必需项,UI 检查则是可选的提前反馈。
## 验收标准
## 测试
- 在两种输入区中粘贴或拖放一张或多张受支持的图片时,文本框上方会按顺序显示可移除缩略图,且不更改文本框内容;拖入时会高亮目标,放置不支持的文件也不会触发页面跳转,仅图片的发送可正常工作。
- 未发送的浏览器图片仅以 `File` 与对象 URL 的形式存在,可以在内存中跨会话切换保留,不进入 `localStorage`,并在移除、发送被接受或服务释放后撤销。
- 每张已接受的用户图片都会提交到解析所得 `DSH_HOME` 下,之后才会追加相应的 `user/message` 事件。事件只包含 `ImageBlock` 引用,绝不包含 base64 或临时路径。
- 结构化助手图片只能由持久 `ImageBlock` 表示;未来的输出适配器必须在发出助手事件前持久化字节,而 Markdown 图片 URL 仍是文本。
- 冷启动历史记录通过同一个有界控件渲染用户与助手图片引用。双击打开原图;Escape、激活背景区域和激活关闭控件可以关闭预览,且不提供自定义上下文菜单。
- 除非同一个会话日志引用了该标识符,否则会话附件读取会失败。对象缺失或损坏会明确失败,绝不返回未经校验的字节。
- Pi-AI 会为兼容路径生成提供方原生输入图片。DeepSeek 与所有未实现该能力的消费方返回明确的不支持内容错误,而不是丢弃该块。
- 明确仅支持文本的当前模型会在持久化附件或追加会话事件前拒绝图片发送;未知元数据仍会到达适配器强制检查,发送失败则恢复草稿。
- 无需密钥的单元测试、宿主集成测试、客户端集成测试和组装应用的 Chromium 覆盖会验证持久化顺序、日志中不含 base64、授权、粘贴与拖放、仅图片发送、历史用户与助手图片、原图预览和对象 URL 清理。
- 当前生产适配器集合声明仅支持文本输出;输出提供方认证、文件选择、非图片文件、视频、持久草稿和垃圾回收不在第一版范围内。
- 存储测试覆盖内容寻址去重、私有权限、准入失败、对象损坏或缺失时的失败,以及收紧部署限制后读取历史数据。
- 宿主与协议测试覆盖先持久化再追加事件的顺序、日志中不含 base64、会话作用域授权、能力拒绝、上传限制,以及大小受限的 HTTP 请求体。
- 客户端单元测试和组装后的 Chromium 测试覆盖粘贴与拖放、混合剪贴板文本、仅图片发送、草稿恢复、历史用户与助手图片、原图预览、顺序,以及草稿、会话和应用层级的对象 URL 清理。
- 适配器与压缩测试覆盖 Pi-AI 原生图片转换、后置附件服务组合、仅文本拒绝、嵌套工具结果中的图片、保留摘要输入,以及明确拒绝图片输出。
- 需要凭据的实际 API 测试会通过 Anthropic `claude-opus-4-8` 路径发送一张 PNG,并要求模型识别其中的二维码。
- 当前生产适配器集合声明仅支持文本输出;输出提供方认证不在第一版范围内。
## 风险
## 后果
- 持久存储会在没有垃圾回收时持续增长。第一版选择回放安全,而不是过早删除。
- 对象缺失或损坏会让模型请求无法精确重建。明确失败可以保持完整性,但在修复前可能阻止该会话继续运行。
@@ -208,3 +206,4 @@ UI 状态可能陈旧,也无法保护直接 SDK、ACP、回放或未收录模
- 原图预览解码的像素多于行内控件显示的像素。像素限制、一次只打开一个预览和对象 URL 释放可以约束但无法消除浏览器瞬时内存占用。
- 能力元数据可能缺失或陈旧。宿主前置检查可以改善反馈,适配器强制检查仍是权威结果。
- 未来输出提供方可能需要经过身份认证的下载,助手图片才能完成,这会增加延迟与新的故障点。先持久化再追加事件的顺序优先保障回放完整性。
- 文件选择、通用文件与 PDF、音频与视频、持久草稿暂存、图片复制、自定义上下文菜单、输出提供方认证和按引用感知的垃圾回收仍是相互独立的设计。