refactor(llm): drop the image content block until a path can honor it
ImageBlock had no production producer and every consumer dropped it: the deepseek serializer skipped it, the pi-ai converter skipped it as unrepresentable, the ACP bridge neither advertises image prompt capability nor forwards image blocks, and compact-basic charged a flat 85-token estimate and rendered an [image] placeholder. A block constructed today would silently vanish from the wire — the vocabulary advertised a capability no path honors, the silent-data-loss shape the defensive patterns warn against. The only constructors were tests pinning the skip/estimate branches. Remove ImageBlock and its ContentBlockMap entry (its cache?: CacheHint field leaves with it; CacheHint itself and the other two cache? fields are out of scope). compact-basic loses its explicit image estimate and placeholder arms (the merge-extensible default arms absorb the case); the deepseek serializer, pi-ai converter, and ACP codec already handled image in their default arms, so only their image-naming comments change. The codec's inbound rejection of ACP-protocol image prompt content stays — that guards wire content a client can send regardless of our vocabulary. Tests that constructed harness image blocks to pin the removed branches are dropped (the 85-token estimate pin) or retargeted onto plugin-added block types / other non-text blocks, which the surviving default arms own. Docs, the type-equiv pastes, and the content-block vocabulary RFC's block list and multimodal-home consequence are updated in the same change; the RFC moves to implemented/ and the index is regenerated. A real multimodal feature reintroduces image via declaration merging together with the adapter mapping, ACP advertisement, and compaction pricing that honor it.
This commit is contained in:
@@ -8,10 +8,10 @@ This is the implementation tier of the compaction capability — see the [interf
|
||||
|
||||
The abstract contract states only WHAT compaction does; this backend owns every HOW decision:
|
||||
|
||||
- **Token estimation** — `estimateContentTokens()`: char/4 with per-block structural overhead (`text`/`reasoning` = `ceil(len/4) + 4`, `tool-call` from name + arguments, `tool-result` recursive, `image` = 85, unknown blocks via JSON length).
|
||||
- **Token estimation** — `estimateContentTokens()`: char/4 with per-block structural overhead (`text`/`reasoning` = `ceil(len/4) + 4`, `tool-call` from name + arguments, `tool-result` recursive, unknown blocks via JSON length).
|
||||
- **Retention policy** — `compactIfNeeded()` walks the surface nodes tail→head summing per-node token estimates, and retains the smallest tail-run of WHOLE units (a closed step, or a single no-step node such as a pre-step `user/message` or inter-step `steering/message`) whose total reaches `retainTokens`; everything older is compacted. Retention is **turn-agnostic** — turn boundaries play no role, so a single runaway turn that alone exceeds the window compacts its OWN early closed steps rather than being retained verbatim (the failure mode that motivated dropping turn-protection: a tool-heavy turn must stay compactable or the harness dies exactly when compaction is needed). The only structural guard is **tool-pairing balance**: the compacted region's edges are balanced cuts on the surface (no unanswered tool-call crosses either edge), so it never splits a step's `assistant/message` tool-calls from their `tool/result`s. When the only compactable content left is an un-splittable open tail step, it declines (returns `null`) and retries once an older step closes. **Single-unit overflow is out of scope, by design**: if one retained unit (a single closed step, or a large pasted `user/message`) ALONE exceeds the budget, compaction cannot help and the call may go out over-budget — bounding an individual unit's size is a separate concern. `compactRegion()` enforces tool-pairing balance strictly, throwing on a boundary that would split a step. `dsh-session` exports `isToolPairingBalanced` for the check.
|
||||
- **Dynamic convergence** — no static summary-length config pretends to bound what the model will write. If framing/estimator/system overhead leaves the compacted surface above threshold, `compactIfNeeded()` re-compacts the head checkpoint up to `compactionRetries` extra times; if it still cannot get below threshold, it throws. A summary whose estimated stored size is not smaller than the shadowed content fails closed before it mutates the surface.
|
||||
- **Summarization** — `summarize()`: a `GenerateOptions` request assembled via `BlockAssembler` with a fixed system prompt that asks for a structured checkpoint (Primary Request and Intent · Key Technical Concepts · Files and Code · Errors and Fixes · Pending Tasks · Current Work · Next Step · Critical Context), every section mandatory, exact paths/commands/identifiers preserved. The request runs through the `agent/request` waterfall before `ctx.llm.stream()`, so router agents that choose the concrete model there also route compaction summaries. `maxTokens` is the provider-side generation cap; only text blocks from the model's reply are kept before the checkpoint is stored (reasoning is dropped so private chain-of-thought never leaks into the durable summary, and a stray `tool-call` is dropped so the synthesized `user/message` summary cannot land an orphaned call with no matching `tool-result`). The compacted region is flattened to a plain-text transcript first: text and reasoning contribute their text, and every non-text block (image, tool-call, tool-result, plugin-added types) contributes a type-tagged placeholder (`[image]`, `[tool-call: name(args)]`, …) so the summarizer is told what existed rather than silently dropping it.
|
||||
- **Summarization** — `summarize()`: a `GenerateOptions` request assembled via `BlockAssembler` with a fixed system prompt that asks for a structured checkpoint (Primary Request and Intent · Key Technical Concepts · Files and Code · Errors and Fixes · Pending Tasks · Current Work · Next Step · Critical Context), every section mandatory, exact paths/commands/identifiers preserved. The request runs through the `agent/request` waterfall before `ctx.llm.stream()`, so router agents that choose the concrete model there also route compaction summaries. `maxTokens` is the provider-side generation cap; only text blocks from the model's reply are kept before the checkpoint is stored (reasoning is dropped so private chain-of-thought never leaks into the durable summary, and a stray `tool-call` is dropped so the synthesized `user/message` summary cannot land an orphaned call with no matching `tool-result`). The compacted region is flattened to a plain-text transcript first: text and reasoning contribute their text, and every non-text block (tool-call, tool-result, plugin-added types) contributes a type-tagged placeholder (`[tool-call: name(args)]`, `[tool-result: …]`, …) so the summarizer is told what existed rather than silently dropping it.
|
||||
- **Checkpoint framing** — the raw summary is not landed directly. `compactRegion()` wraps it in a checkpoint preamble (so a resuming model reads it as a checkpoint, not a fresh user request, and builds on the captured context rather than restating it) plus `<compacted-summary>…</compacted-summary>` tags. Because region compaction can be invoked manually, a surface may hold several checkpoints, so the framing does not claim everything after it is recent or verbatim. The tags make a prior checkpoint detectable in the transcript on the next compaction cycle: the summarization prompt then instructs the model to merge it in place (preserve still-true facts, drop stale ones) rather than re-summarize it verbatim — a cheap incremental merge that needs no extra log/event machinery. The unframed summary stays on the `compact/summary` provenance event.
|
||||
- **Surface mutation** — `compactRegion()` appends the `compact/start` → `compact/summary` → `compact/end` log records and the single `user/message` replace node carrying the framed summary (see the interface README).
|
||||
- **Auto-compaction** — an `agent/pre-step` listener delegates to `compactIfNeeded()` before every step (not just a turn's first — a tool-heavy turn grows the surface mid-turn, so a runaway turn still compacts, and per-step firing is the only moment to rescue it before overflow). `agent/pre-step` is a serial (awaited, in-order) surface-mutation checkpoint that fires after `turn/start` and BEFORE the step opens (`step/start`) and its request history is derived, so compaction mutates the surface — with its log-only `compact/*` records landing cleanly outside any step — and the loop derives once from the result: no double-derive, and the listener cannot see (or need to rewrite) an already-assembled `messages` array. The listener owns no threshold logic of its own (the single token-pressure check lives in `compactIfNeeded()`); because Cordis `serial` bails early on non-void return values, the listener returns `void` and does not use the dispatcher's bail channel as a veto surface.
|
||||
|
||||
@@ -44,9 +44,6 @@ export { resolveConfig } from './types.ts'
|
||||
/** Per-block structural overhead for JSON framing / type tag. */
|
||||
const BLOCK_OVERHEAD = 4
|
||||
|
||||
/** Heuristic token count for an image block (~85 tokens for low-res URL). */
|
||||
const IMAGE_TOKEN_COST = 85
|
||||
|
||||
/** Role-field framing overhead added per message in {@link BasicCompactService.estimateTokens}. */
|
||||
const ROLE_OVERHEAD = 4
|
||||
|
||||
@@ -230,9 +227,6 @@ export class BasicCompactService extends CompactService {
|
||||
case 'tool-result':
|
||||
tokens += this.estimateContentTokens(block.content) + BLOCK_OVERHEAD
|
||||
break
|
||||
case 'image':
|
||||
tokens += IMAGE_TOKEN_COST
|
||||
break
|
||||
default:
|
||||
// Unknown block types (merge-extensible ContentBlockMap):
|
||||
// estimate conservatively via JSON stringify.
|
||||
@@ -706,10 +700,10 @@ export class BasicCompactService extends CompactService {
|
||||
/**
|
||||
* Render content blocks to a single plain-text string for the summarization
|
||||
* prompt. Text and reasoning contribute their text; every other block type
|
||||
* contributes a type-tagged placeholder (`[image]`, `[tool-call: name(args)]`,
|
||||
* …) so the summarizer is told what non-text content existed in the region
|
||||
* rather than silently losing it. Blocks join with newlines; empty-text
|
||||
* blocks contribute nothing.
|
||||
* contributes a type-tagged placeholder (`[tool-call: name(args)]`,
|
||||
* `[tool-result: …]`, …) so the summarizer is told what non-text content
|
||||
* existed in the region rather than silently losing it. Blocks join with
|
||||
* newlines; empty-text blocks contribute nothing.
|
||||
*/
|
||||
private _blocksToText(blocks: readonly ContentBlock[]): string {
|
||||
const parts: string[] = []
|
||||
@@ -729,9 +723,6 @@ export class BasicCompactService extends CompactService {
|
||||
parts.push(inner ? `[tool-result: ${inner}]` : '[tool-result]')
|
||||
break
|
||||
}
|
||||
case 'image':
|
||||
parts.push('[image]')
|
||||
break
|
||||
// ContentBlockMap is merge-extensible — render an unknown block as a
|
||||
// bare type-tagged placeholder so a plugin-added block type is still
|
||||
// signalled to the summarizer rather than dropped.
|
||||
|
||||
@@ -811,11 +811,6 @@ describe('BasicCompactService token estimation (char/4 heuristic)', () => {
|
||||
])).toBe(10)
|
||||
})
|
||||
|
||||
it('estimates image blocks at fixed 85 tokens', () => {
|
||||
const svc = new BasicCompactService(new Context(), cfg({ auto: false }))
|
||||
expect(svc.estimateContentTokens([{ type: 'image', url: 'https://example.com/img.png' }])).toBe(85)
|
||||
})
|
||||
|
||||
it('returns 0 for empty content blocks', () => {
|
||||
const svc = new BasicCompactService(new Context(), cfg({ auto: false }))
|
||||
expect(svc.estimateContentTokens([])).toBe(0)
|
||||
@@ -1324,7 +1319,7 @@ describe('BasicCompactService edge cases', () => {
|
||||
s.append('assistant/message', {
|
||||
turn: 1, step: 1,
|
||||
content: [
|
||||
{ type: 'tool-result', toolCallId: CallId('n1'), content: [{ type: 'image', url: 'https://x/n.png' }] },
|
||||
{ type: 'tool-result', toolCallId: CallId('n1'), content: [{ type: 'chart', data: 'x' } as unknown as ContentBlock] },
|
||||
{ type: 'custom-widget', payload: 'x' } as unknown as ContentBlock,
|
||||
{ type: 'tool-call', id: CallId('b1'), name: 'bash', arguments: '{}' },
|
||||
],
|
||||
@@ -1343,7 +1338,7 @@ describe('BasicCompactService edge cases', () => {
|
||||
const nodes = s.surface.nodes
|
||||
await compactRegion(svc, s, nodes[0]!.seq, nodes[nodes.length - 1]!.seq, 'm')
|
||||
const { text } = svc.summarizeCalls[0]!
|
||||
expect(text).toContain('[tool-result: [image]]') // nested tool-result with content
|
||||
expect(text).toContain('[tool-result: [chart]]') // nested tool-result with content
|
||||
expect(text).toContain('[custom-widget]') // unknown block placeholder
|
||||
expect(text).toContain('Tool result (call b1): [tool-result]') // empty nested → bare placeholder
|
||||
})
|
||||
@@ -1510,25 +1505,28 @@ describe('BasicCompactService edge cases', () => {
|
||||
it('renders non-text blocks as type-tagged placeholders across all message kinds', async () => {
|
||||
const svc = createTestService()
|
||||
const s = new Session(SessionId('placeholders'))
|
||||
// A plugin-added block type (merge-extensible ContentBlockMap) — the
|
||||
// placeholder path must cover every message kind, not just assistant.
|
||||
const chart = (id: string): ContentBlock => ({ type: 'chart', data: id } as unknown as ContentBlock)
|
||||
s.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
s.append('step/start', { turn: 1, step: 1 })
|
||||
// user/message with only an image block → '[image]' placeholder.
|
||||
s.append('user/message', { content: [{ type: 'image', url: 'https://x/y.png' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
|
||||
// assistant/message with an image block AND the tool-call its tool/result
|
||||
// answers (so the surface is tool-pairing balanced) → '[image]' placeholder.
|
||||
// user/message with only a plugin-added block → '[chart]' placeholder.
|
||||
s.append('user/message', { content: [chart('y')], source: { kind: 'user' } }, { surfaceOp: 'append' })
|
||||
// assistant/message with a plugin-added block AND the tool-call its
|
||||
// tool/result answers (so the surface is tool-pairing balanced).
|
||||
s.append('assistant/message', {
|
||||
turn: 1, step: 1,
|
||||
content: [
|
||||
{ type: 'image', url: 'https://x/z.png' },
|
||||
chart('z'),
|
||||
{ type: 'tool-call', id: CallId('e1'), name: 'bash', arguments: '{}' },
|
||||
],
|
||||
}, { surfaceOp: 'append' })
|
||||
// tool/result with an image block → '[image]' placeholder.
|
||||
// tool/result with a plugin-added block → '[chart]' placeholder.
|
||||
s.append('tool/call', { turn: 1, step: 1, callId: CallId('e1'), name: 'bash', arguments: '{}' })
|
||||
s.append('tool/result', { turn: 1, step: 1, callId: CallId('e1'), content: [{ type: 'image', url: 'https://x/r.png' }], isError: false }, { surfaceOp: 'append' })
|
||||
// context/message and steering/message with image content.
|
||||
s.append('context/message', { content: [{ type: 'image', url: 'https://x/c.png' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
|
||||
s.append('steering/message', { turn: 1, content: [{ type: 'image', url: 'https://x/s.png' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
|
||||
s.append('tool/result', { turn: 1, step: 1, callId: CallId('e1'), content: [chart('r')], isError: false }, { surfaceOp: 'append' })
|
||||
// context/message and steering/message with plugin-added content.
|
||||
s.append('context/message', { content: [chart('c')], source: { kind: 'user' } }, { surfaceOp: 'append' })
|
||||
s.append('steering/message', { turn: 1, content: [chart('s')], source: { kind: 'user' } }, { surfaceOp: 'append' })
|
||||
s.append('step/end', { turn: 1, step: 1 })
|
||||
s.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
s.append('turn/start', { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
@@ -1537,11 +1535,11 @@ describe('BasicCompactService edge cases', () => {
|
||||
await compactRegion(svc, s, nodes[0]!.seq, nodes[nodes.length - 1]!.seq, 'm')
|
||||
const { text } = svc.summarizeCalls[0]!
|
||||
// Every non-text block surfaces as a placeholder rather than being dropped.
|
||||
expect(text).toContain('User: [image]')
|
||||
expect(text).toContain('Assistant: [image]')
|
||||
expect(text).toContain('Tool result (call e1): [image]')
|
||||
expect(text).toContain('[Context: [image]]')
|
||||
expect(text).toContain('[Steering: [image]]')
|
||||
expect(text).toContain('User: [chart]')
|
||||
expect(text).toContain('Assistant: [chart]')
|
||||
expect(text).toContain('Tool result (call e1): [chart]')
|
||||
expect(text).toContain('[Context: [chart]]')
|
||||
expect(text).toContain('[Steering: [chart]]')
|
||||
})
|
||||
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user