Merge remote-tracking branch 'origin/worktree/web-multimodal-image-input' into worktree/web-multimodal-image-input
# Conflicts: # .agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.i18n.yaml # apps/web/tests/code-mode-fixture.snapshot.ts # apps/web/tests/image-display.snapshot.ts # docs/architecture.i18n.yaml # docs/module-graph.md # packages/client/test-runtime/tests/runtime.spec.tsx # packages/client/ui-conversation/src/client/skeleton/ConversationSession.tsx # packages/client/ui-conversation/tests/terminal-card.spec.tsx # packages/client/ui-trajectory/src/client/layout.ts
This commit is contained in:
creatixchu
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.md
|
||||
2026-07-22-web-multimodal-image-input-and-durable-attachments.md: 5b5425f794fd1581520524cf1cfa0f8ae71bde38
|
||||
2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md: b3ff73c96c82d31ed6b5b8a2051e6703f607f64e
|
||||
2026-07-22-web-multimodal-image-input-and-durable-attachments.md: c1959ecc5d3a89af57bade3a58c3a8edc2679e0d
|
||||
2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md: b61053de3b83ee2b4f27907b9deb059b9e531177
|
||||
|
||||
@@ -120,9 +120,9 @@ Base64 crosses JSON-RPC once and is discarded after persistence. The host valida
|
||||
|
||||
Model catalog entries gain optional merge-extensible input and output modality declarations. A missing declaration means unknown; a present list without `image` is an explicit negative capability.
|
||||
|
||||
The host is the authoritative preflight boundary. It resolves the session's latest routed provider/model, falling back through agent options to host defaults; if that model explicitly excludes image input, it rejects the prompt before writing any attachment or event, and the client restores the draft. Unknown capability proceeds to the adapter guard so uncatalogued model identifiers remain usable. `host.describe` projects the host-default active model and the image limits into `SessionsService`; the composer applies only the deployment limits before allocating object URLs or base64. Model capability is deliberately not gated client-side: the handshake snapshot cannot represent a session's current target after `session.selectModel`, so the host preflight is the sole capability authority and its rejection renders through the composer error strip. Decoded-pixel validation and the session's actual route remain authoritative on the host.
|
||||
The host is the authoritative preflight boundary. It resolves the session's latest routed provider/model, falling back through agent options to host defaults; if that model explicitly excludes image input, it rejects the prompt before writing any attachment or event, and the client restores the draft. Unknown capability proceeds to the adapter guard so uncatalogued model identifiers remain usable. The browser rejects unsupported declared image media types before allocating preview URLs, but it does not snapshot deployment limits or model capability: a handshake snapshot cannot represent a session's current target after `session.selectModel`, and deployment policy may change independently. The host validates the complete batch against current byte, count, aggregate, media, dimension, pixel, and routed-model policy before writing any attachment or event; its rejection renders through the composer error strip.
|
||||
|
||||
The Pi-AI adapter is the first visual-input route: it resolves `ctx.attachments` at request time, then resolves each durable reference and emits native image content only for models that declare image input. The shipped Web assembly reaches it through `dsh web --provider <name> --model <id>`: the yml pi-ai row already routes openai/anthropic with ambient credentials, and only a catalog provider absent from that row is mounted dynamically; the DeepSeek-only default remains text-only. Request-time service resolution keeps Cordis load order from freezing optional attachment availability. The hand-written DeepSeek adapter throws typed `UNSUPPORTED_CONTENT` for an image anywhere in the request, including nested tool results. No adapter may flatten or skip an image.
|
||||
The Pi-AI adapter is the first visual-input route: it resolves `ctx.attachments` at request time, then resolves each durable reference and emits native image content only for models that declare image input. The shipped composition registers Pi-AI OpenAI and Anthropic routes alongside the text-only default DeepSeek route; selecting the active provider/model remains a host composition or profile concern rather than an image-input CLI feature. Request-time service resolution keeps Cordis load order from freezing optional attachment availability. The hand-written DeepSeek adapter throws typed `UNSUPPORTED_CONTENT` for an image anywhere in the request, including nested tool results. No adapter may flatten or skip an image.
|
||||
|
||||
Core supports structured assistant image blocks, but no current production provider route is certified for image output. Any future output-capable adapter must retrieve provider bytes under bounded size and time policy, validate them through the same attachment service, persist them, and only then publish the atomic `ImageBlock`. A URL in assistant Markdown remains text and is never downloaded automatically.
|
||||
|
||||
@@ -138,7 +138,7 @@ Composer thumbnails and each `MessageImage` own ephemeral original-preview state
|
||||
|
||||
### Limits and trust boundaries
|
||||
|
||||
Version one accepts PNG, JPEG, WebP, and GIF only. SVG and remote URLs are excluded. Default limits are 5 MiB per image, 10 images and 20 MiB aggregate image bytes per message, and 40 million intrinsic pixels per image. These deployment-varying limits are validated backend configuration and are projected to the client for fast-path guidance; host validation remains authoritative. The client connection carrier independently caps buffered API request bodies, deriving the cap from the aggregate image limit plus base64 and envelope expansion; a body without a declared length is rejected the moment it crosses the cap rather than drained to its end.
|
||||
Version one accepts PNG, JPEG, WebP, and GIF only. SVG and remote URLs are excluded. Default limits are 5 MiB per image, 10 images and 20 MiB aggregate image bytes per message, and 40 million intrinsic pixels per image. These deployment-varying limits are validated backend configuration and enforced by the host before persistence. The client connection carrier independently caps buffered API request bodies, deriving the cap from the host attachment service's aggregate image limit plus base64 and envelope expansion; a body without a declared length is rejected the moment it crosses the cap rather than drained to its end.
|
||||
|
||||
Malformed base64, unsupported or mismatched media, truncated headers, excess bytes, excess image count, excess pixels, missing objects, and integrity mismatches return stable structured failures. Original filenames are reduced to a display basename, control characters are removed, and no local path is logged or returned to the browser.
|
||||
|
||||
|
||||
@@ -120,9 +120,9 @@ Base64 只跨越一次 JSON-RPC，并在持久化后丢弃。宿主会校验规
|
||||
|
||||
模型目录项增加可选且可合并扩展的输入与输出模态声明。缺少声明表示未知；声明存在但不含 `image`，则明确表示不支持图片。
|
||||
|
||||
宿主是权威的前置检查边界。它会解析会话最新路由到的提供方和模型，并在缺失时依次回退到 agent 选项和宿主默认值；如果该模型明确排除图片输入，宿主会在写入任何附件或事件前拒绝提示词，客户端则恢复草稿。能力未知时继续进入适配器强制检查，使未收录的模型标识符仍然可用。`host.describe` 把宿主默认的当前模型和图片限制投影到 `SessionsService`；composer 在分配对象 URL 或 base64 前只应用部署级限制。模型能力刻意不在客户端把关：握手快照无法表达 `session.selectModel` 之后会话的当前目标，因此宿主前置检查是唯一的能力权威，其拒绝通过 composer 错误条呈现。解码像素校验与会话的实际路由仍由宿主作出权威判定。
|
||||
宿主是权威的前置检查边界。它会解析会话最新路由到的提供方和模型，并在缺失时依次回退到 agent 选项和宿主默认值；如果该模型明确排除图片输入，宿主会在写入任何附件或事件前拒绝提示词，客户端则恢复草稿。能力未知时继续进入适配器强制检查，使未收录的模型标识符仍然可用。浏览器会在分配预览 URL 前拒绝声明不支持的图片媒体类型，但不会为部署限制或模型能力保留快照：握手快照无法表达 `session.selectModel` 之后会话的当前目标，部署策略也可能独立变化。宿主会根据当前的单张字节数、图片数量、总字节数、媒体类型、尺寸、像素数和路由模型策略校验整个批次，再写入任何附件或事件；其拒绝通过 composer 错误条呈现。
|
||||
|
||||
Pi-AI 适配器是首条视觉输入路径：它在请求时解析 `ctx.attachments`，再解析每个持久引用，并且仅为声明支持图片输入的模型生成提供方原生图片内容。交付的 Web 组装通过 `dsh web --provider <name> --model <id>` 到达这条路径：yml 的 pi-ai row 已用环境凭据路由 openai/anthropic，只有该 row 之外的目录 provider 才会动态挂载；仅含 DeepSeek 的默认组装仍是纯文本。在请求时解析服务，可避免 Cordis 加载顺序将可选附件服务的可用性固化。手写 DeepSeek 适配器遇到请求中任何位置的图片时都会抛出类型化的 `UNSUPPORTED_CONTENT` 错误，包括嵌套工具结果中的图片。任何适配器都不得将图片展平或跳过。
|
||||
Pi-AI 适配器是首条视觉输入路径：它在请求时解析 `ctx.attachments`，再解析每个持久引用，并且仅为声明支持图片输入的模型生成提供方原生图片内容。交付的组合会同时注册 Pi-AI OpenAI、Anthropic 路由和仅支持文本的默认 DeepSeek 路由；选择当前提供方／模型仍由宿主组合或配置承担，而不是图片输入 CLI（命令行界面）的功能。在请求时解析服务，可避免 Cordis 加载顺序将可选附件服务的可用性固化。手写 DeepSeek 适配器遇到请求中任何位置的图片时都会抛出类型化的 `UNSUPPORTED_CONTENT` 错误，包括嵌套工具结果中的图片。任何适配器都不得将图片展平或跳过。
|
||||
|
||||
核心层支持结构化助手图片块，但当前没有任何生产提供方路径通过图片输出认证。未来任何支持输出的适配器都必须在有界的大小和时间策略下获取提供方字节，通过同一个附件服务校验并持久化字节，之后才能以原子方式发布 `ImageBlock`。助手 Markdown 中的 URL 仍是文本，绝不自动下载。
|
||||
|
||||
@@ -138,7 +138,7 @@ token 估算根据图片尺寸计量，但不把 base64 或附件定位符作为
|
||||
|
||||
### 限制与信任边界
|
||||
|
||||
第一版仅接受 PNG、JPEG、WebP 和 GIF。不接受 SVG 和远程 URL。默认限制为每张图片 5 MiB、每条消息 10 张图片和 20 MiB 图片总字节数，以及每张图片 4,000 万个固有像素。这些随部署变化的限制属于经过校验的后端配置，并会投影给客户端以提供快速路径指引；宿主校验仍是权威结果。客户端连接载体会独立限制 API 请求体的缓冲大小，并根据图片总量限制加上 base64 和请求封装的膨胀量推导上限；未声明长度的请求体在越过上限的瞬间即被拒绝，而不是先读完再拒。
|
||||
第一版仅接受 PNG、JPEG、WebP 和 GIF。不接受 SVG 和远程 URL。默认限制为每张图片 5 MiB、每条消息 10 张图片和 20 MiB 图片总字节数，以及每张图片 4,000 万个固有像素。这些随部署变化的限制属于经过校验的后端配置，并由宿主在持久化前强制执行。客户端连接载体会独立限制 API 请求体的缓冲大小，并根据宿主附件服务的图片总量限制，加上 base64 和请求封装的膨胀量推导上限；未声明长度的请求体在越过上限的瞬间即被拒绝，而不是先读完再拒。
|
||||
|
||||
格式错误的 base64、不支持或不匹配的媒体、截断的文件头、超出字节限制、超出图片数量、超出像素限制、对象缺失和完整性不匹配都会返回稳定的结构化错误。原始文件名只保留用于显示的末段，控制字符会被移除，并且任何本地路径都不会写入日志或返回浏览器。
|
||||
|
||||
|
||||
Reference in New Issue
Block a user