fix: address ds-review-bot v6/v7 findings on the image-input assembly
- resolveLlmRoute: reuse the yml pi-ai row for providers it already routes (DUPLICATE_ADAPTER boot failure) and detect an unset model by origin, not by comparison against one deployment default; covered by a new spec. - LlmService.resolveModelInfoFor preserves (and validates) modality metadata, arming the host image preflight for exact-route resolution. - session.selectModel refuses a text-only target once the session log carries an image on any replayed route; an accepted switch would strand every later turn with no in-product recovery. - The composer no longer gates image intake on the handshake activeModel snapshot (wrong authority for a per-session decision); the host preflight plus the error strip own capability, deployment limits stay client-side. - InputHub shell teardown releases the scope's draft images (File objects and object URLs leaked for the page lifetime). - session.prompt image parts carry optional alt into the durable block; ImageBlock documents assistant-side rendering as forward compatibility. - Assembled built-client lane apps/web/tests/image-display.snapshot.ts pins the history galleries over the authorized attachment route, the lightbox, and the composer paste rail; the attachment rail is an accessible group. - Docs: validateImage on the seam page, fixture byte metadata matches its PNG, and the Agent Note claims now match the shipped coverage.
This commit is contained in:
creatixchu
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-07-22-web-multimodal-image-input-and-durable-attachments.md
|
||||
2026-07-22-web-multimodal-image-input-and-durable-attachments.md: 746ae8a5ffb111866c99c6aad5cb0906f252481e
|
||||
2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md: 5061e8e29a14790d22456a44e44fa162d244f116
|
||||
2026-07-22-web-multimodal-image-input-and-durable-attachments.md: 2b676c1965dbef0bdc24ba6d5b4af5bf66840780
|
||||
2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md: f13648051bcf4c81c8b032897645c9b6aeb0d32a
|
||||
|
||||
@@ -120,7 +120,7 @@ Base64 crosses JSON-RPC once and is discarded after persistence. The host valida
|
||||
|
||||
Model catalog entries gain optional merge-extensible input and output modality declarations. A missing declaration means unknown; a present list without `image` is an explicit negative capability.
|
||||
|
||||
The host is the authoritative preflight boundary. It resolves the session's latest routed provider/model, falling back through agent options to host defaults; if that model explicitly excludes image input, it rejects the prompt before writing any attachment or event, and the client restores the draft. Unknown capability proceeds to the adapter guard so uncatalogued model identifiers remain usable. `host.describe` projects the active model and image limits into `SessionsService`; the resident `InputBar` uses them before allocating object URLs or base64 for fast feedback. Decoded-pixel validation and the session's actual route remain authoritative on the host.
|
||||
The host is the authoritative preflight boundary. It resolves the session's latest routed provider/model, falling back through agent options to host defaults; if that model explicitly excludes image input, it rejects the prompt before writing any attachment or event, and the client restores the draft. Unknown capability proceeds to the adapter guard so uncatalogued model identifiers remain usable. `host.describe` projects the host-default active model and the image limits into `SessionsService`; the composer applies only the deployment limits before allocating object URLs or base64. Model capability is deliberately not gated client-side: the handshake snapshot cannot represent a session's current target after `session.selectModel`, so the host preflight is the sole capability authority and its rejection renders through the composer error strip. Decoded-pixel validation and the session's actual route remain authoritative on the host.
|
||||
|
||||
The Pi-AI adapter is the first visual-input route: it resolves `ctx.attachments` at request time, then resolves each durable reference and emits native image content only for models that declare image input. The shipped Web assembly reaches it through `dsh web --provider <name> --model <id>`, which mounts that pi-ai catalog route with the provider's ambient credentials; the DeepSeek-only default remains text-only. Request-time service resolution keeps Cordis load order from freezing optional attachment availability. The hand-written DeepSeek adapter throws typed `UNSUPPORTED_CONTENT` for an image anywhere in the request, including nested tool results. No adapter may flatten or skip an image.
|
||||
|
||||
@@ -194,8 +194,8 @@ UI state can be stale and does not protect direct SDK, ACP, replay, or uncatalog
|
||||
## Testing
|
||||
|
||||
- Storage tests cover content-addressed deduplication, private permissions, admission failures, corruption/missing-object failures, and reading history after deployment limits are lowered.
|
||||
- Host and protocol tests cover persist-before-event ordering, absence of base64 in logs, session-scoped authorization, capability rejection, upload limits, and bounded HTTP request bodies.
|
||||
- Client unit and assembled Chromium tests cover paste and drop, mixed clipboard text, image-only send, draft restoration, historical user and assistant images, original preview, ordering, and draft/session/application object-URL cleanup.
|
||||
- Host and protocol tests cover persist-before-event ordering, absence of base64 in logs, session-scoped authorization, capability rejection, upload limits, bounded HTTP request bodies, and refusal of a text-only `session.selectModel` once the session log carries an image (an accepted switch would strand every later turn).
|
||||
- Client unit tests cover paste and drop, mixed clipboard text, image-only send, draft restoration, ordering, and draft/session-scope/application object-URL cleanup; the keyless assembled built-client lane (`apps/web/tests/image-display.snapshot.ts`, `DSH_EXAMPLE_MODE=lib pnpm run test:snapshot`) covers the historical user and assistant galleries over the authorized attachment route, the original-size lightbox, and the composer paste rail.
|
||||
- Adapter and compaction tests cover native Pi-AI image conversion, late attachment-service composition, text-only rejection, nested tool-result images, preserved summary input, and explicit image-output rejection.
|
||||
- A credentialed real-API test sends a PNG through the Anthropic `claude-opus-4-8` route and requires the model to identify its QR code.
|
||||
- The current production adapter set declares text-only output; output-provider certification remains outside version one.
|
||||
|
||||
@@ -120,7 +120,7 @@ Base64 只跨越一次 JSON-RPC，并在持久化后丢弃。宿主会校验规
|
||||
|
||||
模型目录项增加可选且可合并扩展的输入与输出模态声明。缺少声明表示未知；声明存在但不含 `image`，则明确表示不支持图片。
|
||||
|
||||
宿主是权威的前置检查边界。它会解析会话最新路由到的提供方和模型，并在缺失时依次回退到 agent 选项和宿主默认值；如果该模型明确排除图片输入，宿主会在写入任何附件或事件前拒绝提示词，客户端则恢复草稿。能力未知时继续进入适配器强制检查，使未收录的模型标识符仍然可用。`host.describe` 把当前模型和图片限制投影到 `SessionsService`；常驻 `InputBar` 会在分配对象 URL 或 base64 前使用这些信息提供快速反馈。解码像素校验与会话的实际路由仍由宿主作出权威判定。
|
||||
宿主是权威的前置检查边界。它会解析会话最新路由到的提供方和模型，并在缺失时依次回退到 agent 选项和宿主默认值；如果该模型明确排除图片输入，宿主会在写入任何附件或事件前拒绝提示词，客户端则恢复草稿。能力未知时继续进入适配器强制检查，使未收录的模型标识符仍然可用。`host.describe` 把宿主默认的当前模型和图片限制投影到 `SessionsService`；composer 在分配对象 URL 或 base64 前只应用部署级限制。模型能力刻意不在客户端把关：握手快照无法表达 `session.selectModel` 之后会话的当前目标，因此宿主前置检查是唯一的能力权威，其拒绝通过 composer 错误条呈现。解码像素校验与会话的实际路由仍由宿主作出权威判定。
|
||||
|
||||
Pi-AI 适配器是首条视觉输入路径：它在请求时解析 `ctx.attachments`，再解析每个持久引用，并且仅为声明支持图片输入的模型生成提供方原生图片内容。交付的 Web 组装通过 `dsh web --provider <name> --model <id>` 到达这条路径——该命令用提供方的环境凭据挂载对应的 pi-ai 目录路由；仅含 DeepSeek 的默认组装仍是纯文本。在请求时解析服务，可避免 Cordis 加载顺序将可选附件服务的可用性固化。手写 DeepSeek 适配器遇到请求中任何位置的图片时都会抛出类型化的 `UNSUPPORTED_CONTENT` 错误，包括嵌套工具结果中的图片。任何适配器都不得将图片展平或跳过。
|
||||
|
||||
@@ -194,8 +194,8 @@ UI 状态可能陈旧，也无法保护直接 SDK、ACP、回放或未收录模
|
||||
## 测试
|
||||
|
||||
- 存储测试覆盖内容寻址去重、私有权限、准入失败、对象损坏或缺失时的失败，以及收紧部署限制后读取历史数据。
|
||||
- 宿主与协议测试覆盖先持久化再追加事件的顺序、日志中不含 base64、会话作用域授权、能力拒绝、上传限制，以及大小受限的 HTTP 请求体。
|
||||
- 客户端单元测试和组装后的 Chromium 测试覆盖粘贴与拖放、混合剪贴板文本、仅图片发送、草稿恢复、历史用户与助手图片、原图预览、顺序，以及草稿、会话和应用层级的对象 URL 清理。
|
||||
- 宿主与协议测试覆盖先持久化再追加事件的顺序、日志中不含 base64、会话作用域授权、能力拒绝、上传限制、大小受限的 HTTP 请求体，以及在会话日志已含图片时拒绝切换到纯文本模型的 `session.selectModel`（接受该切换会让此后每一轮都失败）。
|
||||
- 客户端单元测试覆盖粘贴与拖放、混合剪贴板文本、仅图片发送、草稿恢复、顺序，以及草稿、会话作用域和应用层级的对象 URL 清理；keyless 的组装后构建产物通道（`apps/web/tests/image-display.snapshot.ts`，`DSH_EXAMPLE_MODE=lib pnpm run test:snapshot`）覆盖经授权附件路由渲染的历史用户与助手图片画廊、原图 lightbox，以及 composer 粘贴缩略图条。
|
||||
- 适配器与压缩测试覆盖 Pi-AI 原生图片转换、后置附件服务组合、仅文本拒绝、嵌套工具结果中的图片、保留摘要输入，以及明确拒绝图片输出。
|
||||
- 需要凭据的实际 API 测试会通过 Anthropic `claude-opus-4-8` 路径发送一张 PNG，并要求模型识别其中的二维码。
|
||||
- 当前生产适配器集合声明仅支持文本输出；输出提供方认证不在第一版范围内。
|
||||
|
||||
@@ -61,6 +61,61 @@ export function resolveLanTrust(
|
||||
return { lanAddresses, trustedHosts: [...lanAddresses, ...extra] }
|
||||
}
|
||||
|
||||
/** One provider/model source layer for {@link resolveLlmRoute}, in override order. */
|
||||
export interface LlmRouteInput {
|
||||
/** CLI flag values (highest precedence). */
|
||||
cli: { provider?: string | undefined; model?: string | undefined }
|
||||
/** Profile-json values (parsed JSON — validated here, the config boundary). */
|
||||
profile: { provider?: unknown; model?: unknown }
|
||||
/** The api-gateway yml row's config values (deployment defaults). */
|
||||
gateway: { provider?: unknown; model?: unknown }
|
||||
/** Providers the shipped yml already routes through its static pi-ai row. */
|
||||
ymlPiAiProviders: readonly string[]
|
||||
}
|
||||
|
||||
/** The boot's resolved LLM routing decision. */
|
||||
export interface LlmRoute {
|
||||
/** Effective api-gateway provider. */
|
||||
provider: string
|
||||
/** Pi-ai provider to mount dynamically; undefined when DeepSeek or a yml-routed provider serves the request. */
|
||||
dynamicPiAiProvider: string | undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the boot's LLM route from the layered provider/model sources.
|
||||
* A non-DeepSeek provider requires a model set at least as explicitly as the
|
||||
* provider itself (flag/profile) — origin decides, never a comparison against
|
||||
* any deployment's default model value, so editing the yml default cannot
|
||||
* silently disarm the guard. Providers the shipped yml pi-ai row already
|
||||
* routes are NOT mounted again: `LlmService.registerAdapter` rejects
|
||||
* duplicate routes, so the gateway provider/model patch alone selects them.
|
||||
* @param input - the layered provider/model sources and the yml pi-ai roster.
|
||||
* @returns the effective provider and the dynamic pi-ai mount decision.
|
||||
*/
|
||||
export function resolveLlmRoute(input: LlmRouteInput): LlmRoute {
|
||||
const provider = input.cli.provider ?? input.profile.provider ?? input.gateway.provider
|
||||
if (typeof provider !== 'string' || provider === '') {
|
||||
throw new Error('dsh: api-gateway provider must be a non-empty string')
|
||||
}
|
||||
if (provider !== 'deepseek') {
|
||||
const providerFromYml = input.cli.provider === undefined && input.profile.provider === undefined
|
||||
// A yml-set provider trusts its own row pairing; an override must bring
|
||||
// its model along instead of inheriting the yml default's.
|
||||
const model = providerFromYml
|
||||
? input.gateway.model
|
||||
: input.cli.model ?? input.profile.model
|
||||
if (typeof model !== 'string' || model === '') {
|
||||
throw new Error(`dsh: provider ${provider} requires an explicit model`)
|
||||
}
|
||||
}
|
||||
return {
|
||||
provider,
|
||||
dynamicPiAiProvider: provider === 'deepseek' || input.ymlPiAiProviders.includes(provider)
|
||||
? undefined
|
||||
: provider,
|
||||
}
|
||||
}
|
||||
|
||||
/** One profile-json key mapped onto a yml row's config field. */
|
||||
interface ProfileMapping {
|
||||
jsonPath: string
|
||||
@@ -207,15 +262,16 @@ export class AppCLIEntry {
|
||||
if (this.options.model !== undefined) put('api-gateway', 'model', this.options.model)
|
||||
|
||||
const gatewayConfig = rows.get('api-gateway')?.config as Record<string, unknown> | undefined
|
||||
const provider = this.options.provider ?? profile.provider ?? gatewayConfig?.provider
|
||||
const model = this.options.model ?? profile.model ?? gatewayConfig?.model
|
||||
if (typeof provider !== 'string' || provider === '') {
|
||||
throw new Error('dsh: api-gateway provider must be a non-empty string')
|
||||
}
|
||||
if (provider !== 'deepseek' && (typeof model !== 'string' || model === '' || model === 'deepseek-v4-flash')) {
|
||||
throw new Error(`dsh: provider ${provider} requires an explicit model`)
|
||||
}
|
||||
this.piAiProvider = provider === 'deepseek' ? undefined : provider
|
||||
const piAiRow = rows.get('llm-pi-ai')?.config as { providers?: { provider?: unknown }[] } | undefined
|
||||
const route = resolveLlmRoute({
|
||||
cli: { provider: this.options.provider, model: this.options.model },
|
||||
profile: { provider: profile.provider, model: profile.model },
|
||||
gateway: { provider: gatewayConfig?.provider, model: gatewayConfig?.model },
|
||||
ymlPiAiProviders: (piAiRow?.providers ?? [])
|
||||
.map(p => p.provider)
|
||||
.filter((value): value is string => typeof value === 'string'),
|
||||
})
|
||||
this.piAiProvider = route.dynamicPiAiProvider
|
||||
|
||||
// Source 2b: authorities for the /api browser-trust fence (rationale on
|
||||
// resolveLanTrust).
|
||||
|
||||
61
apps/cli/tests/llm-route.spec.ts
Normal file
61
apps/cli/tests/llm-route.spec.ts
Normal file
@@ -0,0 +1,61 @@
|
||||
/** resolveLlmRoute: layered provider/model resolution and the dynamic pi-ai mount decision. */
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { resolveLlmRoute } from '../src/app-cli-entry.ts'
|
||||
|
||||
/** The shipped yml shape: DeepSeek gateway default plus a pi-ai row routing openai/anthropic. */
|
||||
const SHIPPED = {
|
||||
gateway: { provider: 'deepseek', model: 'deepseek-v4-flash' },
|
||||
ymlPiAiProviders: ['openai', 'anthropic'],
|
||||
}
|
||||
|
||||
describe('resolveLlmRoute', () => {
|
||||
it('keeps the DeepSeek default without any dynamic mount', () => {
|
||||
expect(resolveLlmRoute({ cli: {}, profile: {}, ...SHIPPED }))
|
||||
.toEqual({ provider: 'deepseek', dynamicPiAiProvider: undefined })
|
||||
})
|
||||
|
||||
it('reuses the yml pi-ai row for providers it already routes (no duplicate adapter)', () => {
|
||||
expect(resolveLlmRoute({
|
||||
cli: { provider: 'anthropic', model: 'claude-opus-4-8' }, profile: {}, ...SHIPPED,
|
||||
})).toEqual({ provider: 'anthropic', dynamicPiAiProvider: undefined })
|
||||
})
|
||||
|
||||
it('mounts pi-ai dynamically only for providers absent from the yml row', () => {
|
||||
expect(resolveLlmRoute({
|
||||
cli: { provider: 'google', model: 'gemini-3-pro' }, profile: {}, ...SHIPPED,
|
||||
})).toEqual({ provider: 'google', dynamicPiAiProvider: 'google' })
|
||||
})
|
||||
|
||||
it('requires an explicit model wherever the provider override came from, by origin', () => {
|
||||
// CLI provider with no CLI/profile model: the yml DeepSeek default must not leak in.
|
||||
expect(() => resolveLlmRoute({ cli: { provider: 'anthropic' }, profile: {}, ...SHIPPED }))
|
||||
.toThrow(/provider anthropic requires an explicit model/)
|
||||
// Profile provider paired with a profile model is explicit enough.
|
||||
expect(resolveLlmRoute({
|
||||
cli: {}, profile: { provider: 'openai', model: 'gpt-5' }, ...SHIPPED,
|
||||
})).toEqual({ provider: 'openai', dynamicPiAiProvider: undefined })
|
||||
// Profile provider with only the yml default model: same gap, same refusal.
|
||||
expect(() => resolveLlmRoute({ cli: {}, profile: { provider: 'openai' }, ...SHIPPED }))
|
||||
.toThrow(/provider openai requires an explicit model/)
|
||||
})
|
||||
|
||||
it('trusts a yml-set non-DeepSeek provider only when its own row carries the model', () => {
|
||||
expect(resolveLlmRoute({
|
||||
cli: {}, profile: {},
|
||||
gateway: { provider: 'anthropic', model: 'claude-opus-4-8' },
|
||||
ymlPiAiProviders: ['openai', 'anthropic'],
|
||||
})).toEqual({ provider: 'anthropic', dynamicPiAiProvider: undefined })
|
||||
expect(() => resolveLlmRoute({
|
||||
cli: {}, profile: {},
|
||||
gateway: { provider: 'anthropic' },
|
||||
ymlPiAiProviders: ['openai', 'anthropic'],
|
||||
})).toThrow(/provider anthropic requires an explicit model/)
|
||||
})
|
||||
|
||||
it('fails loud on a missing or empty provider', () => {
|
||||
expect(() => resolveLlmRoute({ cli: {}, profile: {}, gateway: {}, ymlPiAiProviders: [] }))
|
||||
.toThrow(/provider must be a non-empty string/)
|
||||
expect(() => resolveLlmRoute({ cli: { provider: '' }, profile: {}, ...SHIPPED }))
|
||||
.toThrow(/provider must be a non-empty string/)
|
||||
})
|
||||
})
|
||||
177
apps/web/tests/image-display.snapshot.ts
Normal file
177
apps/web/tests/image-display.snapshot.ts
Normal file
@@ -0,0 +1,177 @@
|
||||
// @vitest-environment jsdom
|
||||
// Multimodal image surfaces over the BUILT client graph (the code-mode-fixture
|
||||
// idiom: real bundles via AppWebEntry, keyless FixtureApiClient transport).
|
||||
// Opens the fixture history session whose turn 65 carries an image in BOTH a
|
||||
// user message and an assistant message, and pins the product surfaces: the
|
||||
// history ImageGallery loading real fixture bytes through the authorized
|
||||
// sessions.attachment route, the double-click ImageLightbox, and the composer
|
||||
// intake chain (paste → thumbnail rail → image-only send enablement → remove).
|
||||
import { readFileSync } from 'node:fs'
|
||||
import { join } from 'node:path'
|
||||
import { act, cleanup, fireEvent, screen, waitFor, within } from '@testing-library/react'
|
||||
import { afterEach, beforeEach, expect, it, vi } from 'vitest'
|
||||
import type { WebBootEntry } from '@deepseek-ai/dsh-client-modules/client'
|
||||
import { AppWebEntry } from '@deepseek-ai/dsh-client-web'
|
||||
|
||||
const PLUGINS: readonly (WebBootEntry & { dir: string })[] = [
|
||||
{ id: '@deepseek-ai/dsh-client-connection', dir: 'connection', url: '/plugins/connection.js', rev: 'fx', inject: [], immediately: true },
|
||||
{ id: '@deepseek-ai/dsh-client-runtime', dir: 'runtime', url: '/plugins/runtime.js', rev: 'fx', inject: ['@deepseek-ai/dsh-client-connection'], immediately: true },
|
||||
{ id: '@deepseek-ai/dsh-client-ui-theme', dir: 'ui-theme', url: '/plugins/ui-theme.js', rev: 'fx', inject: [], immediately: true },
|
||||
{ id: '@deepseek-ai/dsh-client-locale', dir: 'locale', url: '/plugins/locale.js', rev: 'fx', inject: [], immediately: true },
|
||||
{ id: '@deepseek-ai/dsh-client-ui-layout', dir: 'ui-layout', url: '/plugins/ui-layout.js', rev: 'fx', inject: ['@deepseek-ai/dsh-client-runtime'] },
|
||||
{ id: '@deepseek-ai/dsh-client-ui-sidebar', dir: 'ui-sidebar', url: '/plugins/ui-sidebar.js', rev: 'fx', inject: ['@deepseek-ai/dsh-client-ui-layout'] },
|
||||
{ id: '@deepseek-ai/dsh-client-ui-conversation', dir: 'ui-conversation', url: '/plugins/ui-conversation.js', rev: 'fx', inject: ['@deepseek-ai/dsh-client-ui-layout'] },
|
||||
{
|
||||
id: '@deepseek-ai/dsh-client-ui-workspace',
|
||||
dir: 'ui-workspace',
|
||||
url: '/plugins/ui-workspace.js',
|
||||
rev: 'fx',
|
||||
inject: [
|
||||
'@deepseek-ai/dsh-client-runtime',
|
||||
'@deepseek-ai/dsh-client-ui-conversation',
|
||||
'@deepseek-ai/dsh-client-ui-sidebar',
|
||||
],
|
||||
},
|
||||
]
|
||||
|
||||
const bundles = new Map(PLUGINS.map(plugin => [
|
||||
plugin.url,
|
||||
readFileSync(join(process.cwd(), 'packages/client', plugin.dir, 'lib/client.js'), 'utf8'),
|
||||
]))
|
||||
|
||||
interface FixtureWindow extends Window {
|
||||
__DSH_BOOT__?: { rev: string; entries: WebBootEntry[] }
|
||||
__ModuleLoader__?: unknown
|
||||
}
|
||||
|
||||
class ResizeObserverStub {
|
||||
observe(): void {}
|
||||
disconnect(): void {}
|
||||
unobserve(): void {}
|
||||
}
|
||||
|
||||
const win = window as FixtureWindow
|
||||
let unmount: (() => void) | undefined
|
||||
|
||||
beforeEach(() => {
|
||||
localStorage.clear()
|
||||
document.title = 'DeepSeek Harness'
|
||||
vi.stubGlobal('ResizeObserver', ResizeObserverStub)
|
||||
vi.stubGlobal('requestAnimationFrame', (callback: FrameRequestCallback) =>
|
||||
setTimeout(() => { callback(0) }, 0) as unknown as number)
|
||||
vi.stubGlobal('cancelAnimationFrame', (id: number) => { clearTimeout(id) })
|
||||
})
|
||||
|
||||
afterEach(() => {
|
||||
act(() => { unmount?.() })
|
||||
unmount = undefined
|
||||
cleanup()
|
||||
delete win.__DSH_BOOT__
|
||||
delete win.__ModuleLoader__
|
||||
document.body.innerHTML = ''
|
||||
document.head.querySelectorAll('style[data-plugin]').forEach((style) => { style.remove() })
|
||||
document.title = ''
|
||||
history.replaceState(null, '', '/')
|
||||
vi.unstubAllGlobals()
|
||||
})
|
||||
|
||||
/** Boot the complete built client graph against the populated fixture branch. */
|
||||
function boot(): void {
|
||||
history.replaceState(null, '', '/?fixture')
|
||||
const root = document.createElement('div')
|
||||
root.id = 'root'
|
||||
document.body.appendChild(root)
|
||||
win.__DSH_BOOT__ = { rev: 'fx', entries: PLUGINS.map(({ dir: _dir, ...plugin }) => plugin) }
|
||||
act(() => {
|
||||
const entry = new AppWebEntry(root, {
|
||||
fetchBundle: (url) => {
|
||||
const code = bundles.get(url)
|
||||
return code === undefined ? Promise.reject(new Error(`missing built bundle ${url}`)) : Promise.resolve(code)
|
||||
},
|
||||
executeBundle: (code) => { (0, eval)(code) },
|
||||
})
|
||||
void entry.run()
|
||||
unmount = () => { entry.dispose() }
|
||||
})
|
||||
}
|
||||
|
||||
/** Open the fixture history session (the alpha log carrying the turn-65 image pair) and wait for its gallery. */
|
||||
async function openFixtureSession(): Promise<void> {
|
||||
const tree = await screen.findByRole('tree', { name: 'Sessions' }, { timeout: 10_000 })
|
||||
const group = (await within(tree).findAllByText('fixture'))
|
||||
.map(el => el.closest<HTMLElement>('[role="treeitem"]'))
|
||||
.find(el => el?.getAttribute('aria-expanded') !== null)
|
||||
if (group === null || group === undefined) throw new Error('fixture Workspace group missing')
|
||||
if (group.getAttribute('aria-expanded') === 'false') {
|
||||
fireEvent.click(within(group).getByText('fixture'))
|
||||
await waitFor(() => {
|
||||
expect(group.getAttribute('aria-expanded')).toBe('true')
|
||||
})
|
||||
}
|
||||
const session = await within(tree).findByText('Fixture 历史会话')
|
||||
fireEvent.click(session)
|
||||
await waitFor(() => {
|
||||
expect(document.querySelectorAll('[data-align] img').length).toBeGreaterThan(0)
|
||||
}, { timeout: 10_000 })
|
||||
}
|
||||
|
||||
it('renders the history image pair through the authorized attachment route and opens the lightbox', async () => {
|
||||
boot()
|
||||
await openFixtureSession()
|
||||
|
||||
// Both the user-side (align=end) and assistant-side (align=start) galleries
|
||||
// load real fixture bytes over sessions.attachment (data: fallback in jsdom).
|
||||
await waitFor(() => {
|
||||
const user = document.querySelector('[data-align="end"] img')
|
||||
const assistant = document.querySelector('[data-align="start"] img')
|
||||
if (user === null || assistant === null) throw new Error('history image galleries missing')
|
||||
// jsdom serves object URLs; environments without createObjectURL fall back to data:.
|
||||
expect(user.getAttribute('src')).toMatch(/^(blob:|data:image\/png;base64,)/)
|
||||
expect(assistant.getAttribute('src')).toMatch(/^(blob:|data:image\/png;base64,)/)
|
||||
}, { timeout: 10_000 })
|
||||
const userImage = document.querySelector<HTMLElement>('[data-align="end"] img')!
|
||||
expect(userImage.getAttribute('alt')).toBe('fixture-image.png')
|
||||
|
||||
// Double-click opens the original-size lightbox; Escape/close dismisses it.
|
||||
const frame = userImage.closest('button')
|
||||
if (frame === null) throw new Error('image frame button missing')
|
||||
fireEvent.doubleClick(frame)
|
||||
const lightbox = await screen.findByRole('dialog')
|
||||
expect(within(lightbox).getByRole('img').getAttribute('src')).toMatch(/^(blob:|data:image\/png;base64,)/)
|
||||
fireEvent.click(within(lightbox).getByRole('button', { name: /关闭/ }))
|
||||
await waitFor(() => {
|
||||
expect(screen.queryByRole('dialog')).toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
it('accepts a pasted image into the composer rail and removes it', async () => {
|
||||
boot()
|
||||
await openFixtureSession()
|
||||
|
||||
// Image-only send arming is pinned at package level (input-bar.spec.tsx);
|
||||
// this assembled lane pins the intake chain over the built graph.
|
||||
const textarea = await screen.findByPlaceholderText('Message the agent', {}, { timeout: 10_000 })
|
||||
const image = new File([new Uint8Array([137, 80, 78, 71])], 'pasted.png', { type: 'image/png' })
|
||||
fireEvent.paste(textarea, {
|
||||
clipboardData: {
|
||||
items: [{ kind: 'file', type: 'image/png', getAsFile: () => image }],
|
||||
getData: () => '',
|
||||
},
|
||||
})
|
||||
|
||||
// The rail is an accessible group holding the draft thumbnail (queried via
|
||||
// DOM: jsdom's a11y-visibility computation hides the composer subtree).
|
||||
const rail = await waitFor(() => {
|
||||
const el = document.querySelector('[role="group"][aria-label="待发送图片"]')
|
||||
if (el === null) throw new Error('attachment rail missing')
|
||||
return el
|
||||
}, { timeout: 5_000 })
|
||||
expect(rail.querySelector('img')?.getAttribute('src')).toMatch(/^(blob:|data:)/)
|
||||
|
||||
const remove = rail.querySelector('button[aria-label^="移除图片"]')
|
||||
if (remove === null) throw new Error('remove button missing')
|
||||
fireEvent.click(remove)
|
||||
await waitFor(() => {
|
||||
expect(document.querySelector('[role="group"][aria-label="待发送图片"]')).toBeNull()
|
||||
})
|
||||
})
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write docs/core-data-structures/attachment.md
|
||||
attachment.md: bb25b62b3260dff0ae85a03819c8655213c25563
|
||||
attachment.zh.md: b9bf182ef7623d2d121f6d6190fd14de49401f67
|
||||
attachment.md: c4c974b075300025868e90f9905193d62d68c4ee
|
||||
attachment.zh.md: 12802a414018d081460459ad96ee4aa91801da62
|
||||
|
||||
@@ -69,4 +69,4 @@ interface StoredImageAttachment {
|
||||
}
|
||||
```
|
||||
|
||||
`saveImage()` validates bytes and atomically commits one object before returning its reference. `readImage()` accepts a reference from an authorized session path and returns bytes only after integrity verification. The service is deliberately retention-neutral: resumed and forked sessions may share objects, so reference-aware garbage collection is deferred rather than tied to any one session's deletion.
|
||||
`saveImage()` validates bytes and atomically commits one object before returning its reference. `validateImage()` runs the same admission checks (magic bytes, declared media type, size and pixel limits) without persisting anything — batch callers MUST validate every member through it before persisting any, so a rejected batch leaves no partial objects behind. `readImage()` accepts a reference from an authorized session path and returns bytes only after integrity verification. The service is deliberately retention-neutral: resumed and forked sessions may share objects, so reference-aware garbage collection is deferred rather than tied to any one session's deletion.
|
||||
|
||||
@@ -69,4 +69,4 @@ interface StoredImageAttachment {
|
||||
}
|
||||
```
|
||||
|
||||
`saveImage()` 校验字节并以原子方式提交一个对象，之后才返回其引用。`readImage()` 接受来自已授权会话路径的引用，只在完整性校验通过后返回字节。该服务刻意不规定保留策略：恢复和 fork 后的会话可能共享对象，因此基于引用的垃圾回收会延期实现，而不是与任何一个会话的删除绑定。
|
||||
`saveImage()` 校验字节并以原子方式提交一个对象，之后才返回其引用。`validateImage()` 执行同一套准入检查（magic bytes、声明的媒体类型、大小与像素上限）但不落任何持久化——批量调用方必须先对每个成员通过它校验、再持久化任何一个，从而保证被拒绝的批次不会留下部分对象。`readImage()` 接受来自已授权会话路径的引用，只在完整性校验通过后返回字节。该服务刻意不规定保留策略：恢复和 fork 后的会话可能共享对象，因此基于引用的垃圾回收会延期实现，而不是与任何一个会话的删除绑定。
|
||||
|
||||
@@ -108,7 +108,9 @@ const FIXTURE_IMAGE_DATA = 'iVBORw0KGgoAAAANSUhEUgAAAKAAAABaCAYAAAA/xl1SAAAAvklE
|
||||
const FIXTURE_IMAGE_REF: ImageAttachmentRef = {
|
||||
attachmentId: 'fixture:image' as AttachmentIdType,
|
||||
mediaType: 'image/png',
|
||||
bytes: 68,
|
||||
// Matches the decoded FIXTURE_IMAGE_DATA exactly (the real backend serves
|
||||
// verified metadata; a mismatched fixture would mislead comparisons).
|
||||
bytes: 247,
|
||||
width: 160,
|
||||
height: 90,
|
||||
name: 'fixture-image.png',
|
||||
|
||||
@@ -28,6 +28,7 @@ interface ConversationAttachmentFace {
|
||||
mode: 'queue' | 'steer',
|
||||
imageIds: readonly string[],
|
||||
): Promise<void>
|
||||
releaseDraftImage(id: string): void
|
||||
}
|
||||
|
||||
/** Session-addressed input facade registry (InputService face + composer-layer extras). */
|
||||
@@ -84,8 +85,13 @@ export class InputHub implements InputService {
|
||||
]
|
||||
return () => {
|
||||
for (const off of offs) off()
|
||||
// Draft attachments die with the scope: the shell only holds ids, so
|
||||
// the service-owned File objects and object URLs must be released
|
||||
// here or they leak for the page lifetime.
|
||||
const drafts = shell.snapshot.imageIds
|
||||
shell.dispose()
|
||||
this.shells.delete(id)
|
||||
for (const imageId of drafts) this.conversation().releaseDraftImage(imageId)
|
||||
}
|
||||
}, 'conversation.input: session shell')
|
||||
return shell
|
||||
|
||||
@@ -330,11 +330,13 @@ export class ConversationService extends Service implements IConversation {
|
||||
current: readonly ComposerAttachment[],
|
||||
): void {
|
||||
if (files.length === 0 && current.length === 0) return
|
||||
// Deployment-wide limits only. Model capability is deliberately NOT
|
||||
// checked here: the handshake's activeModel is the host default, not the
|
||||
// session's current target (session.selectModel never refreshes it), so a
|
||||
// client-side modality gate refuses sessions the host would accept and
|
||||
// vice versa. The host preflight on session.prompt is the authority; its
|
||||
// rejection renders through the composer error strip.
|
||||
const description = this.requireSessions().hostDescription()
|
||||
const modalities = description?.activeModel?.inputModalities
|
||||
if (modalities !== undefined && !modalities.includes('image')) {
|
||||
throw new Error('当前模型不支持图片输入')
|
||||
}
|
||||
const limits = description?.imageLimits
|
||||
const all = [...current.map(attachment => attachment.file), ...files]
|
||||
if (limits !== undefined && all.length > limits.maxImagesPerMessage) {
|
||||
|
||||
@@ -385,7 +385,7 @@ export function InputBar({
|
||||
{overlay !== undefined && <div className={css.overlayAnchor}>{overlay}</div>}
|
||||
{accessory !== undefined && <div className={css.accessory}>{accessory}</div>}
|
||||
{attachments.length > 0 && (
|
||||
<div className={css.attachments} aria-label="待发送图片">
|
||||
<div className={css.attachments} role="group" aria-label="待发送图片">
|
||||
{attachments.map(attachment => (
|
||||
<div key={attachment.id} className={css.attachment}>
|
||||
<button
|
||||
|
||||
@@ -20,13 +20,12 @@ async function bench() {
|
||||
})
|
||||
// config.input is required (the apply shares its hub with the inject
|
||||
// factories); the bench passes its own instance explicitly.
|
||||
const fiber = runtime.ctx.plugin(ConversationService, {
|
||||
input: new InputHub(runtime.ctx),
|
||||
})
|
||||
const hub = new InputHub(runtime.ctx)
|
||||
const fiber = runtime.ctx.plugin(ConversationService, { input: hub })
|
||||
await fiber.await()
|
||||
const root = runtime.ctx.get('conversation') as ConversationService
|
||||
const scoped = runtime.sessions.scope('s1')!.get('conversation') as ConversationService
|
||||
return { runtime, root, scoped, prompt, cancel, loadOlder }
|
||||
return { runtime, hub, root, scoped, prompt, cancel, loadOlder }
|
||||
}
|
||||
|
||||
describe('ConversationService', () => {
|
||||
@@ -50,6 +49,26 @@ describe('ConversationService', () => {
|
||||
await b.runtime.dispose()
|
||||
})
|
||||
|
||||
it('releases draft images when the session scope is disposed', async () => {
|
||||
const b = await bench()
|
||||
const created = vi.spyOn(URL, 'createObjectURL').mockReturnValue('blob:draft-1')
|
||||
const revoked = vi.spyOn(URL, 'revokeObjectURL').mockReturnValue(undefined)
|
||||
try {
|
||||
const [attachment] = b.root.createDraftImages([new File([new Uint8Array(4)], 'a.png', { type: 'image/png' })])
|
||||
if (attachment === undefined) throw new Error('draft attachment missing')
|
||||
// Land the id in the session shell exactly as the composer does.
|
||||
b.hub.shell(b.runtime.sessions.behavior('s1').sessionId).addImages([attachment.id])
|
||||
await b.runtime.sessions.remove('s1')
|
||||
// Scope teardown released the service-held File and its object URL.
|
||||
expect(b.root.draftImages([attachment.id])).toEqual([])
|
||||
expect(revoked).toHaveBeenCalledWith('blob:draft-1')
|
||||
} finally {
|
||||
created.mockRestore()
|
||||
revoked.mockRestore()
|
||||
}
|
||||
await b.runtime.dispose()
|
||||
})
|
||||
|
||||
it('fails loudly from the root scope, on an unbound session, or without SessionsService', async () => {
|
||||
const b = await bench()
|
||||
await expect(b.root.send('x', 'queue')).rejects.toThrow(/requires a session scope/)
|
||||
|
||||
@@ -106,7 +106,7 @@ async function durablePromptContent(ctx: Context, content: readonly PromptConten
|
||||
mediaType: item.part.mediaType,
|
||||
...item.part.name === undefined ? {} : { name: item.part.name },
|
||||
})
|
||||
return { type: 'image', attachment }
|
||||
return { type: 'image', attachment, ...item.part.alt === undefined ? {} : { alt: item.part.alt } }
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -127,6 +127,32 @@ function imageInContent(content: unknown, attachmentId: string): ImageAttachment
|
||||
return undefined
|
||||
}
|
||||
|
||||
/** True when any block (nested tool-result content included) is an image block. */
|
||||
function contentHasImage(content: unknown): boolean {
|
||||
if (!Array.isArray(content)) return false
|
||||
for (const value of content) {
|
||||
if (typeof value !== 'object' || value === null || Array.isArray(value)) continue
|
||||
const block = value as { type?: unknown; content?: unknown }
|
||||
if (block.type === 'image') return true
|
||||
if (block.type === 'tool-result' && contentHasImage(block.content)) return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
/**
|
||||
* True when the session log already carries image content on any route a
|
||||
* model request replays (message content, wrapped messages, streamed blocks).
|
||||
* The log is immutable, so a true here is permanent for the session's life.
|
||||
*/
|
||||
function sessionHasImage(events: readonly SessionEvent[]): boolean {
|
||||
return events.some((event) => {
|
||||
const data = event.data as { content?: unknown; message?: { content?: unknown }; chunk?: { type?: unknown; block?: unknown } }
|
||||
if (contentHasImage(data.content)) return true
|
||||
if (data.message !== undefined && contentHasImage(data.message.content)) return true
|
||||
return event.type === 'assistant/chunk' && data.chunk?.type === 'block-end' && contentHasImage([data.chunk.block])
|
||||
})
|
||||
}
|
||||
|
||||
function referencedImage(events: readonly SessionEvent[], attachmentId: string): ImageAttachmentRef | undefined {
|
||||
for (const event of events) {
|
||||
const data = event.data as { content?: unknown; chunk?: { type?: unknown; block?: unknown } }
|
||||
@@ -1111,6 +1137,20 @@ export function createApiProxy(ctx: Context, defaults: ApiProxyDefaults): ApiPro
|
||||
? {}
|
||||
: { reasoningEffort: ReasoningEffortId(reasoningEffort) },
|
||||
})
|
||||
// An image-bearing log replays into every later request, and both
|
||||
// wire routes reject image content on text-only models — accepting
|
||||
// this selection would strand the session (every turn fails, no
|
||||
// in-product recovery). Refuse at the selection boundary instead.
|
||||
if (sessionHasImage(found.agent.session.events)) {
|
||||
const info = await ctx.llm.resolveModelInfo(resolved.provider, resolved.model)
|
||||
if (info.inputModalities !== undefined && !info.inputModalities.includes('image')) {
|
||||
return err(request, {
|
||||
code: 'model-unavailable',
|
||||
message: `Model "${resolved.model}" does not accept image input, but this session's history already contains images; select an image-capable model.`,
|
||||
details: { provider, model },
|
||||
})
|
||||
}
|
||||
}
|
||||
const selected: AgentLlmTarget = {
|
||||
provider: resolved.provider,
|
||||
model: resolved.model,
|
||||
|
||||
@@ -198,7 +198,7 @@ export const imageMediaTypeSchema = z.union([
|
||||
/** Prompt wire content is intentionally narrower than merge-extensible durable core content. */
|
||||
export const promptContentPartSchema = z.discriminatedUnion('type', [
|
||||
z.object({ type: z.literal('text'), text: z.string() }),
|
||||
z.object({ type: z.literal('image'), mediaType: imageMediaTypeSchema, data: z.string(), name: z.string().optional() }),
|
||||
z.object({ type: z.literal('image'), mediaType: imageMediaTypeSchema, data: z.string(), name: z.string().optional(), alt: z.string().optional() }),
|
||||
])
|
||||
|
||||
/** session.prompt request payload. */
|
||||
|
||||
@@ -162,7 +162,7 @@ export interface SessionSummary {
|
||||
/** Browser-submitted prompt content; image bytes are promoted to durable references by the host. */
|
||||
export type PromptContentPart =
|
||||
| { type: 'text'; text: string }
|
||||
| { type: 'image'; mediaType: ImageMediaType; data: string; name?: string }
|
||||
| { type: 'image'; mediaType: ImageMediaType; data: string; name?: string; alt?: string }
|
||||
|
||||
/** Session-domain unary methods (the map keys session.* of RpcMethodMap). */
|
||||
export interface SessionsApi {
|
||||
|
||||
@@ -230,4 +230,91 @@ describe('Web session model selection', () => {
|
||||
.toEqual({ provider: 'deepseek', model: 'private-preview', reasoningEffort: 'max' })
|
||||
await ctx.fiber.dispose()
|
||||
})
|
||||
|
||||
it('refuses a text-only selection once the session log carries an image', async () => {
|
||||
const { ctx, sessionId, agent } = await harness()
|
||||
ctx.llm.registerAdapter(['text-only'], new class extends CatalogAdapter {
|
||||
override resolveModel(provider: string, model: string): Promise<LlmResolvedModelInfo> {
|
||||
return Promise.resolve({ provider, id: model, name: model, inputModalities: ['text'] })
|
||||
}
|
||||
}('Text Only', []))
|
||||
ctx.llm.registerAdapter(['vision'], new class extends CatalogAdapter {
|
||||
override resolveModel(provider: string, model: string): Promise<LlmResolvedModelInfo> {
|
||||
return Promise.resolve({ provider, id: model, name: model, inputModalities: ['text', 'image'] })
|
||||
}
|
||||
}('Vision', []))
|
||||
const api = createApiProxy(ctx, { provider: 'deepseek', model: 'deepseek-chat', cwd: '/tmp', workspaceRoot: '/tmp' })
|
||||
|
||||
// Before any image lands, a text-only selection is legitimate.
|
||||
expect(expectValue(await api.sessions.selectModel(request({
|
||||
sessionId, provider: 'text-only', model: 'plain',
|
||||
}))).selected).toEqual({ provider: 'text-only', model: 'plain' })
|
||||
|
||||
agent.session.append('user/message', {
|
||||
id: 'msg-image', role: 'user', source: { kind: 'user' },
|
||||
content: [{ type: 'image', attachment: { attachmentId: 'att-1', mediaType: 'image/png', bytes: 8, width: 1, height: 1 } }],
|
||||
} as never, { surfaceOp: 'append' })
|
||||
|
||||
// The log is immutable: a text-only route would fail every later turn.
|
||||
const stranded = await api.sessions.selectModel(request({
|
||||
sessionId, provider: 'text-only', model: 'plain',
|
||||
}))
|
||||
expect(stranded.result).toMatchObject({
|
||||
ok: false,
|
||||
error: { code: 'model-unavailable', message: expect.stringMatching(/history already contains images/) as unknown },
|
||||
})
|
||||
|
||||
// Image-capable and modality-unknown routes stay selectable.
|
||||
expect(expectValue(await api.sessions.selectModel(request({
|
||||
sessionId, provider: 'vision', model: 'sees',
|
||||
}))).selected).toEqual({ provider: 'vision', model: 'sees' })
|
||||
expect(expectValue(await api.sessions.selectModel(request({
|
||||
sessionId, provider: 'deepseek', model: 'deepseek-chat',
|
||||
}))).selected).toEqual({ provider: 'deepseek', model: 'deepseek-chat', reasoningEffort: 'high' })
|
||||
await ctx.fiber.dispose()
|
||||
})
|
||||
|
||||
it('detects images on every replayed route: wrapped messages, streamed blocks, nested tool results', async () => {
|
||||
const image = { type: 'image', attachment: { attachmentId: 'att-x', mediaType: 'image/png', bytes: 8, width: 1, height: 1 } }
|
||||
const cases: { label: string; append: (agent: Agent) => void }[] = [
|
||||
{
|
||||
label: 'steering message wrapper',
|
||||
append: (agent) => {
|
||||
agent.session.append('steering/message', {
|
||||
turn: 1, message: { id: 'st-1', role: 'user', source: { kind: 'user' }, content: [image] },
|
||||
} as never, { surfaceOp: 'append' })
|
||||
},
|
||||
},
|
||||
{
|
||||
label: 'streamed assistant block',
|
||||
append: (agent) => {
|
||||
agent.session.append('assistant/chunk', {
|
||||
turn: 1, step: 0, chunk: { type: 'block-end', index: 0, block: image },
|
||||
} as never)
|
||||
},
|
||||
},
|
||||
{
|
||||
label: 'nested tool-result content',
|
||||
append: (agent) => {
|
||||
agent.session.append('user/message', {
|
||||
id: 'tr-1', role: 'user', source: { kind: 'tool', callId: 'c1' },
|
||||
content: [{ type: 'tool-result', toolCallId: 'c1', content: [image], isError: false }],
|
||||
} as never, { surfaceOp: 'append' })
|
||||
},
|
||||
},
|
||||
]
|
||||
for (const { label, append } of cases) {
|
||||
const { ctx, sessionId, agent } = await harness()
|
||||
ctx.llm.registerAdapter(['text-only'], new class extends CatalogAdapter {
|
||||
override resolveModel(provider: string, model: string): Promise<LlmResolvedModelInfo> {
|
||||
return Promise.resolve({ provider, id: model, name: model, inputModalities: ['text'] })
|
||||
}
|
||||
}('Text Only', []))
|
||||
const api = createApiProxy(ctx, { provider: 'deepseek', model: 'deepseek-chat', cwd: '/tmp', workspaceRoot: '/tmp' })
|
||||
append(agent)
|
||||
const stranded = await api.sessions.selectModel(request({ sessionId, provider: 'text-only', model: 'plain' }))
|
||||
expect(stranded.result.ok, label).toBe(false)
|
||||
await ctx.fiber.dispose()
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
@@ -333,11 +333,23 @@ export class LlmService extends Service {
|
||||
'INVALID_MODEL_CONTEXT',
|
||||
)
|
||||
}
|
||||
for (const modalities of [resolved.inputModalities, resolved.outputModalities]) {
|
||||
if (modalities !== undefined && (!Array.isArray(modalities) || modalities.some(m => typeof m !== 'string'))) {
|
||||
throw new LlmError(
|
||||
`adapter returned invalid modality metadata for provider "${provider}" model "${model}"`,
|
||||
'INVALID_MODEL_INFO',
|
||||
)
|
||||
}
|
||||
}
|
||||
const info: LlmResolvedModelInfo = {
|
||||
provider,
|
||||
id: model,
|
||||
name: resolved.name,
|
||||
...resolved.description === undefined ? {} : { description: resolved.description },
|
||||
// Capability metadata rides through: an explicit modality omission is
|
||||
// negative capability downstream preflights act on (image admission).
|
||||
...resolved.inputModalities === undefined ? {} : { inputModalities: resolved.inputModalities },
|
||||
...resolved.outputModalities === undefined ? {} : { outputModalities: resolved.outputModalities },
|
||||
...context === undefined ? {} : { context: { contextWindow: context.contextWindow } },
|
||||
}
|
||||
const reasoning = resolved.reasoning
|
||||
|
||||
@@ -47,12 +47,17 @@ export interface ReasoningBlock {
|
||||
text: string
|
||||
}
|
||||
|
||||
/** A durable raster image reference, valid in user or assistant content. */
|
||||
/**
|
||||
* A durable raster image reference, valid in user or assistant content. The
|
||||
* block is deliberately role-neutral; assistant-side rendering is forward
|
||||
* compatibility — the current production adapters declare text-only output,
|
||||
* so only user content carries images today.
|
||||
*/
|
||||
export interface ImageBlock {
|
||||
type: 'image'
|
||||
/** Immutable bytes and intrinsic display metadata owned by the attachment service. */
|
||||
attachment: ImageAttachmentRef
|
||||
/** Optional provider- and UI-facing alternative text. */
|
||||
/** Optional provider- and UI-facing alternative text, carried from the prompt wire's image part. */
|
||||
alt?: string
|
||||
}
|
||||
|
||||
|
||||
@@ -857,6 +857,8 @@ describe('LlmService', () => {
|
||||
[{ provider: 'route', id: 'model', name: 1 }, 'non-string name'],
|
||||
[{ provider: 'route', id: 'model', name: '' }, 'empty name'],
|
||||
[{ provider: 'route', id: 'model', name: 'Model', description: 1 }, 'non-string description'],
|
||||
[{ provider: 'route', id: 'model', name: 'Model', inputModalities: 'text' }, 'non-array input modalities'],
|
||||
[{ provider: 'route', id: 'model', name: 'Model', outputModalities: [1] }, 'non-string output modality'],
|
||||
] as const)('rejects invalid exact model metadata (%s: %s)', async (metadata, _label) => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
@@ -871,6 +873,27 @@ describe('LlmService', () => {
|
||||
.rejects.toMatchObject({ code: 'INVALID_MODEL_INFO' })
|
||||
})
|
||||
|
||||
it('preserves modality metadata through exact model resolution', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
const adapter = new class extends ScriptedAdapter {
|
||||
override resolveModel(): Promise<LlmResolvedModelInfo> {
|
||||
return Promise.resolve({
|
||||
provider: 'route', id: 'model', name: 'Model',
|
||||
inputModalities: ['text', 'image'], outputModalities: ['text'],
|
||||
})
|
||||
}
|
||||
}(SCRIPT)
|
||||
ctx.llm.registerAdapter(['route'], adapter)
|
||||
|
||||
// Downstream preflights (image admission) act on this exact field; a
|
||||
// rebuild that drops it silently reads as "modalities unknown".
|
||||
await expect(ctx.llm.resolveModelInfo('route', 'model')).resolves.toEqual({
|
||||
provider: 'route', id: 'model', name: 'Model',
|
||||
inputModalities: ['text', 'image'], outputModalities: ['text'],
|
||||
})
|
||||
})
|
||||
|
||||
it('resolves detached model context independently of advisory catalog membership', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
|
||||
Reference in New Issue
Block a user