Add web multimodal image attachments

This commit is contained in:
Yichen Jiang
2026-07-23 15:20:47 +08:00
parent 3e3ea47296
commit cb4c11b869
116 changed files with 3177 additions and 151 deletions

View File

@@ -7,10 +7,14 @@ A service can be a core spine service, a swappable capability seam, or a bundle/
```mermaid
flowchart LR
pkg_attachment["attachment"]
svc_attachments["ctx.attachments<br/>Durable binary attachment storage"]
pkg_attachment_local["attachment-local"]
pkg_host_runtime["host-runtime"]
pkg_llm_pi_ai["llm-pi-ai"]
pkg_llm["llm"]
svc_llm["ctx.llm<br/>LLM adapter registry"]
pkg_llm_deepseek["llm-deepseek"]
pkg_llm_pi_ai["llm-pi-ai"]
pkg_llm_replay["llm-replay"]
pkg_agent_loop["agent-loop"]
pkg_compact_basic["compact-basic"]
@@ -125,6 +129,8 @@ flowchart LR
pkg_agent --> svc_agents
pkg_agent_loop --> svc_agentLoop
pkg_approval --> svc_approval
pkg_attachment --> svc_attachments
pkg_attachment_local --> svc_attachments
pkg_bash --> svc_bash
pkg_bash_local --> svc_bash
pkg_bash_sandbox --> svc_bash
@@ -189,6 +195,8 @@ flowchart LR
svc_agents --> pkg_tui_demo
svc_approval --> pkg_tool_bash
svc_approval --> pkg_tools
svc_attachments --> pkg_host_runtime
svc_attachments --> pkg_llm_pi_ai
svc_bash --> pkg_hooks_claude
svc_bash --> pkg_hooks_codex
svc_bash --> pkg_tool_bash
@@ -263,6 +271,7 @@ flowchart LR
| ctx key | Role | Owner | Implementations | Direct consumers | Companion plugins | Note |
| --- | --- | --- | --- | --- | --- | --- |
| `ctx.attachments` | `seam` | [`attachment`](../packages/attachment/attachment) | [`attachment-local`](../packages/attachment/attachment-local) | [`host-runtime`](../packages/host/runtime), [`llm-pi-ai`](../packages/llm/llm-pi-ai) | - | The host commits accepted images before session events; provider adapters resolve authorized durable references into provider-native content. |
| `ctx.llm` | `seam` | [`llm`](../packages/llm/llm) | [`llm-deepseek`](../packages/llm/llm-deepseek), [`llm-pi-ai`](../packages/llm/llm-pi-ai), [`llm-replay`](../packages/support/llm-replay) | [`agent-loop`](../packages/core/agent-loop), [`compact-basic`](../packages/compact/compact-basic) | - | Adapters register provider implementations; the loop and compaction call the provider-neutral stream service. |
| `ctx.tokenMeter` | `core` | [`token-meter`](../packages/llm/token-meter) | - | [`compact-basic`](../packages/compact/compact-basic) | - | Owns isolated per-session replay folds; pressure consumers share immutable revisioned measurements. |
| `ctx.toolResultPrune` | `core` | [`compact-tool-result-prune`](../packages/compact/compact-tool-result-prune) | - | [`compact-basic`](../packages/compact/compact-basic) | - | Rewrites oversized current tool results through replayable single-node surface replacements before summary compaction. |

View File

@@ -193,6 +193,26 @@ Depends on: [`AgentLoopConfig`](#deepseek-aidsh-agent-loop) · [`GoalDomainConfi
Source: [`packages/examples/agent-spine-demo/src/index.ts:87`](../packages/examples/agent-spine-demo/src/index.ts)
## `@deepseek-ai/dsh-attachment-local`
```ts config-catalog
/** Local attachment backend configuration. */
export interface Config {
/** Explicit harness home; omitted follows `DSH_HOME`, then `~/.dsh`. */
dshHome?: string
/** Maximum encoded bytes accepted for one image. */
maxImageBytes?: number
/** Maximum image count accepted in one submitted message. */
maxImagesPerMessage?: number
/** Maximum aggregate encoded image bytes accepted in one submitted message. */
maxMessageImageBytes?: number
/** Maximum intrinsic width multiplied by height accepted for one image. */
maxImagePixels?: number
}
```
Source: [`packages/attachment/attachment-local/src/index.ts:26`](../packages/attachment/attachment-local/src/index.ts)
## `@deepseek-ai/dsh-bash-local`
```ts config-catalog
@@ -1881,6 +1901,7 @@ These load from a `cordis.yml` entry with no `config:` block; they declare no co
Abstract service classes — a deployment loads a concrete implementation package instead ([capability seams](../.agents/notes/implemented/architecture/2026-06-13-capability-seams.md)).
- `@deepseek-ai/dsh-attachment` — abstract `AttachmentStore` ([`packages/attachment/attachment/src/index.ts`](../packages/attachment/attachment/src/index.ts))
- `@deepseek-ai/dsh-bash` — abstract `BashExecutor` ([`packages/bash/bash/src/index.ts`](../packages/bash/bash/src/index.ts))
- `@deepseek-ai/dsh-code-runtime` — abstract `CodeRuntime` ([`packages/code-runtime/code-runtime/src/index.ts`](../packages/code-runtime/code-runtime/src/index.ts))
- `@deepseek-ai/dsh-compact` — abstract `CompactService` ([`packages/compact/compact/src/index.ts`](../packages/compact/compact/src/index.ts))

View File

@@ -248,6 +248,30 @@ Types: [ApprovalOutcome](../core-data-structures/approval.md) · [ApprovalReques
Source: [`packages/ui/user-approval/src/index.ts:213`](../../packages/ui/user-approval/src/index.ts)
## `ctx.attachments` — `AttachmentStore` (abstract seam)
Immutable binary attachment service. Implementations validate bytes before publishing a reference.
```ts cordis-catalog
/**
* Validate and durably commit one image before its owning session event is appended.
* @param input - encoded bytes, declared media type, and optional display name.
* @returns a durable content-addressed reference.
*/
abstract saveImage(input: SaveImageAttachment): Promise<ImageAttachmentRef>
/**
* Read one image and verify that bytes still match the recorded reference.
* @param ref - durable reference from the session log.
* @returns the verified bytes and canonical reference.
*/
abstract readImage(ref: ImageAttachmentRef): Promise<StoredImageAttachment>
```
Types: [ImageAttachmentRef](../core-data-structures/attachment.md) · [SaveImageAttachment](../core-data-structures/attachment.md) · [StoredImageAttachment](../core-data-structures/attachment.md)
Source: [`packages/attachment/attachment/src/index.ts:28`](../../packages/attachment/attachment/src/index.ts)
## `ctx.bash` — `BashExecutor` (abstract seam)
Abstract bash execution service. Subclass, implement the abstract methods, and load the subclass as a plugin — it registers as `ctx.bash` (one implementation per context; loading a second throws, which is cordis' standard duplicate-service behavior).
@@ -1493,7 +1517,7 @@ estimateMessage(message: Message): number
Types: [EpochHeader](../core-data-structures/session.md) · [Message](../core-data-structures/core.md) · [Session](../core-data-structures/session.md) · [TokenMeasurement](../core-data-structures/token-meter.md)
Source: [`packages/llm/token-meter/src/index.ts:82`](../../packages/llm/token-meter/src/index.ts)
Source: [`packages/llm/token-meter/src/index.ts:87`](../../packages/llm/token-meter/src/index.ts)
## `ctx.toolResultPrune` — `ToolResultPruneService`

View File

@@ -0,0 +1,70 @@
# Durable Image Attachments
The attachment seam separates binary image ownership from the session log. A producer gives validated encoded bytes to [`ctx.attachments`](../cordis-catalog/services.md#ctxattachments); the service publishes an immutable content-addressed reference only after the object is durable. Session events and model-visible `ImageBlock`s contain that reference and metadata, never a browser object URL, host temporary path, provider URL, or base64 payload.
Unsent browser drafts may stay in memory and native clients may stage them in operating-system temporary storage. Once the host accepts a user message, its images move below `<DSH_HOME>/attachments/v1` before the user event is appended. Structured model image output follows the same persist-before-event rule.
Source: [`packages/attachment/attachment/src/types.ts`](../../packages/attachment/attachment/src/types.ts)
## Identity and verified metadata
`AttachmentId` is a branded opaque string. The local backend currently emits `sha256:<digest>`, but consumers must neither parse that representation nor derive a filesystem path from it.
```ts type-equiv
/** Raster image formats accepted by the version-one attachment path. */
type ImageMediaType = 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif'
```
```ts type-equiv
/** Durable, serializable metadata for one immutable image object. */
interface ImageAttachmentRef {
/** Opaque storage identifier; never a filesystem path or bearer URL. */
attachmentId: AttachmentId
/** Media type verified from the stored bytes. */
mediaType: ImageMediaType
/** Exact encoded byte length. */
bytes: number
/** Intrinsic encoded width in pixels. */
width: number
/** Intrinsic encoded height in pixels. */
height: number
/** Optional display name stripped of local path information. */
name?: string
}
```
```ts type-equiv
/** Deployment-resolved limits shared by upload consumers and UI preflight. */
interface ImageAttachmentLimits {
maxImageBytes: number
maxImagesPerMessage: number
maxMessageImageBytes: number
maxImagePixels: number
mediaTypes: readonly ImageMediaType[]
}
```
The reference records intrinsic dimensions and encoded length so clients can lay out history without decoding first, while every authoritative read still re-checks digest, media signature, dimensions, and metadata against the object.
## Commit and verified-read payloads
```ts type-equiv
/** Request to validate and durably commit one image. */
interface SaveImageAttachment {
data: Uint8Array
/** Caller-declared media type, checked against magic bytes. */
mediaType: ImageMediaType
/** Optional browser/provider display name; it is never interpreted as a path. */
name?: string
}
```
```ts type-equiv
/** Stored image bytes returned after reference and digest verification. */
interface StoredImageAttachment {
ref: ImageAttachmentRef
data: Uint8Array
}
```
`saveImage()` validates bytes and atomically commits one object before returning its reference. `readImage()` accepts a reference from an authorized session path and returns bytes only after integrity verification. The service is deliberately retention-neutral: resumed and forked sessions may share objects, so reference-aware garbage collection is deferred rather than tied to any one session's deletion.

View File

@@ -28,6 +28,7 @@ Everything else is documented on a **sub-page**, not here. The rule that draws t
| [tools.md](tools.md) | `ToolDefinition` full fields, the schema DSL, `ToolExecution`/`ToolResult`, tool-presentation UI types, and the guarded execution pipeline |
| [user-interaction.md](user-interaction.md) | the UI-backed human question/answer seam: `AskUserQuestionRequest`, answer/options vocabulary, provider API, error taxonomy |
| [approval.md](approval.md) | the one-shot user-approval seam: `ApprovalRequest`, `ApprovalOutcome`, per-session policy, audit and answerer contracts |
| [attachment.md](attachment.md) | durable image identity and metadata, validation inputs, verified reads, and the `AttachmentStore` seam |
| [bash.md](bash.md) | the bash executor seam: `BashExecRequest`/`Spec`, `BashRunResult`, background `BashProcess` handles |
| [pty.md](pty.md) | persistent terminal ids, backend/session contracts, send readiness, bounded reads, and owner-visible snapshots |
| [sandbox.md](sandbox.md) | per-session policy resolution and the process-confinement seam: file-effect modes, execution/provider policies, `ConfinedArgv`, enforcement and fail-closed errors |
@@ -106,12 +107,13 @@ Source: [`packages/llm/llm/src/types.ts`](../../packages/llm/llm/src/types.ts)
interface ContentBlockMap {
'text': TextBlock
'reasoning': ReasoningBlock
'image': ImageBlock
'tool-call': ToolCallBlock
'tool-result': ToolResultBlock
}
```
The block interfaces (full fields in source): `TextBlock` (`text`), `ReasoningBlock` (thinking, distinct from visible text), `ToolCallBlock` (`id: CallId`, `name`, raw-JSON `arguments`), `ToolResultBlock` (`toolCallId`, nested `content: ContentBlock[]`, `isError?`). `ContentBlock = ContentBlockMap[ContentBlockType]`. The core set is limited to blocks every shipping path honors — multimodal content (images, audio, …) has no core block type; a feature that needs one adds it via the merge-extensible map together with the adapter/UI/compaction support that honors it.
The block interfaces (full fields in source): `TextBlock` (`text`), `ReasoningBlock` (thinking, distinct from visible text), `ImageBlock` (a durable [image attachment](attachment.md) plus optional alternative text), `ToolCallBlock` (`id: CallId`, `name`, raw-JSON `arguments`), and `ToolResultBlock` (`toolCallId`, nested `content: ContentBlock[]`, `isError?`). `ContentBlock = ContentBlockMap[ContentBlockType]`. A new modality belongs in the merge-extensible map only when its adapter, UI, compaction, and durable replay paths honor it.
A `Message` is a role plus blocks. Loop-derived assistant messages carry their durable provider/model identity and optional adapter-private replay metadata:
@@ -192,6 +194,10 @@ interface LlmModelInfo {
name: string
/** Optional user-facing distinction from otherwise similar models. */
description?: string
/** Accepted request modalities; absent means unknown, while an explicit omission is negative capability. */
inputModalities?: readonly ModelModality[]
/** Structured response modalities; absent means unknown, while an explicit omission is negative capability. */
outputModalities?: readonly ModelModality[]
}
```

View File

@@ -208,6 +208,7 @@ declare abstract class LlmAdapter {
interface ContentBlockMap {
'text': TextBlock
'reasoning': ReasoningBlock
'image': ImageBlock
'tool-call': ToolCallBlock
'tool-result': ToolResultBlock
}