refactor(llm): drop the image content block until a path can honor it
ImageBlock had no production producer and every consumer dropped it: the deepseek serializer skipped it, the pi-ai converter skipped it as unrepresentable, the ACP bridge neither advertises image prompt capability nor forwards image blocks, and compact-basic charged a flat 85-token estimate and rendered an [image] placeholder. A block constructed today would silently vanish from the wire — the vocabulary advertised a capability no path honors, the silent-data-loss shape the defensive patterns warn against. The only constructors were tests pinning the skip/estimate branches. Remove ImageBlock and its ContentBlockMap entry (its cache?: CacheHint field leaves with it; CacheHint itself and the other two cache? fields are out of scope). compact-basic loses its explicit image estimate and placeholder arms (the merge-extensible default arms absorb the case); the deepseek serializer, pi-ai converter, and ACP codec already handled image in their default arms, so only their image-naming comments change. The codec's inbound rejection of ACP-protocol image prompt content stays — that guards wire content a client can send regardless of our vocabulary. Tests that constructed harness image blocks to pin the removed branches are dropped (the 85-token estimate pin) or retargeted onto plugin-added block types / other non-text blocks, which the surviving default arms own. Docs, the type-equiv pastes, and the content-block vocabulary RFC's block list and multimodal-home consequence are updated in the same change; the RFC moves to implemented/ and the index is regenerated. A real multimodal feature reintroduces image via declaration merging together with the adapter mapping, ACP advertisement, and compaction pricing that honor it.
This commit is contained in:
@@ -25,7 +25,7 @@ An adapter registry plus a single streaming call surface, interceptable via a wa
|
||||
|
||||
### Content-block vocabulary (`types.ts`)
|
||||
|
||||
Messages are arrays of typed content blocks: `text`, `reasoning`, `tool-call`, `tool-result`, `image`. The union is derived from the merge-extensible `ContentBlockMap`, so plugins can add block types via declaration merging.
|
||||
Messages are arrays of typed content blocks: `text`, `reasoning`, `tool-call`, `tool-result`. The union is derived from the merge-extensible `ContentBlockMap`, so plugins can add block types via declaration merging. The core set is limited to blocks every shipping path honors — multimodal content (images, audio, …) has no core block type; a feature that needs one adds it via the map together with the adapter/UI/compaction support that honors it.
|
||||
|
||||
Streaming is a raw chunk protocol (`block-start`, `text-delta`, `reasoning-delta`, `tool-call-delta`, `block-end`, `usage`, `finish`). `BlockAssembler` is the single shared implementation that assembles chunks into blocks/messages.
|
||||
|
||||
|
||||
@@ -57,24 +57,22 @@ export interface ToolResultBlock {
|
||||
cache?: CacheHint
|
||||
}
|
||||
|
||||
/** An image, by URL or data URL. */
|
||||
export interface ImageBlock {
|
||||
type: 'image'
|
||||
url: string
|
||||
mimeType?: string
|
||||
cache?: CacheHint
|
||||
}
|
||||
|
||||
/**
|
||||
* All known content block shapes, keyed by their `type` tag.
|
||||
* Merge-extensible: plugins add new block types via declaration merging.
|
||||
*
|
||||
* The core set is deliberately limited to blocks every shipping path honors.
|
||||
* Multimodal content (images, audio, …) has no core block type: a feature
|
||||
* that needs one adds it via declaration merging in the same coordinated
|
||||
* change that maps it in the adapters, surfaces it in the UI bridges, and
|
||||
* prices it in compaction — a producer never lands without its consumers
|
||||
* (see docs/rfc/implemented/simplification/2026-07-04-drop-image-content-block.md).
|
||||
*/
|
||||
export interface ContentBlockMap {
|
||||
'text': TextBlock
|
||||
'reasoning': ReasoningBlock
|
||||
'tool-call': ToolCallBlock
|
||||
'tool-result': ToolResultBlock
|
||||
'image': ImageBlock
|
||||
}
|
||||
|
||||
export type ContentBlockType = keyof ContentBlockMap
|
||||
|
||||
@@ -63,12 +63,12 @@ describe('BlockAssembler', () => {
|
||||
|
||||
it('throws from assemble() when a partial has an unhandled blockType', () => {
|
||||
const assembler = new BlockAssembler()
|
||||
// Directly push a block-end for an image block whose block-start never
|
||||
// called ensure — but the image block-type flows through normally.
|
||||
// What we really need is a partial whose blockType is not text/reasoning/tool-call.
|
||||
// We can achieve this via a block-start for 'image' followed by blocks().
|
||||
assembler.push({ type: 'block-start', index: 0, blockType: 'image' } as unknown as StreamChunk)
|
||||
expect(() => assembler.blocks()).toThrow('cannot assemble incomplete block of type "image"')
|
||||
// A partial whose blockType is not text/reasoning/tool-call cannot be
|
||||
// assembled without its block-end. A plugin-added block type (here
|
||||
// 'video', via the merge-extensible ContentBlockMap) opened by a
|
||||
// block-start with no closing block-end exercises that throw.
|
||||
assembler.push({ type: 'block-start', index: 0, blockType: 'video' } as unknown as StreamChunk)
|
||||
expect(() => assembler.blocks()).toThrow('cannot assemble incomplete block of type "video"')
|
||||
})
|
||||
|
||||
it('mustGet throws when an index is missing from the partials map (invariant violation)', () => {
|
||||
|
||||
@@ -81,7 +81,7 @@ describe('BlockAssembler properties', () => {
|
||||
fc.assert(fc.property(streamArb, (chunks) => {
|
||||
const blocks = feed(chunks).blocks()
|
||||
for (const block of blocks) {
|
||||
expect(['text', 'reasoning', 'tool-call', 'tool-result', 'image']).toContain(block.type)
|
||||
expect(['text', 'reasoning', 'tool-call', 'tool-result']).toContain(block.type)
|
||||
}
|
||||
}))
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user