Merge origin/master into feat/plan-mode
This commit is contained in:
@@ -5,7 +5,9 @@ The LLM seam and its provider adapters. The interface package (`llm`) owns the a
|
||||
| Package | Role | ctx key |
|
||||
|---|---|---|
|
||||
| `llm/` | Abstract LLM service + content-block vocabulary + chunk assembler | `ctx.llm` |
|
||||
| `token-meter/` | Replay-aware request and surface token measurement | `ctx.tokenMeter` |
|
||||
| `llm-retry/` | Bounded transient request retry policy | (listens to `agent/request-error`) |
|
||||
| `llm-deepseek/` | DeepSeek API adapter (hand-rolled fetch/SSE) | (registers on `ctx.llm`) |
|
||||
| `llm-pi-ai/` | DeepSeek adapter via `@earendil-works/pi-ai` (design twin) | (registers on `ctx.llm`) |
|
||||
| `llm-pi-ai/` | Multi-provider adapter via `@earendil-works/pi-ai` | (registers on `ctx.llm`) |
|
||||
|
||||
The interface lives at `llm/llm/`; adapters are flat siblings under the group. A new provider adapter joins here and registers on `ctx.llm` without touching the interface. See [twin LLM adapters](../../docs/rfc/implemented/architecture/2026-06-13-twin-llm-adapters.md) for why two adapters exist.
|
||||
The interface lives at `llm/llm/`; adapters, retry policy, and reusable token meter are flat siblings under the group. Requests route by `provider`, while `model` is passed through to the selected adapter. A new provider adapter joins here and registers one or more provider routes on `ctx.llm` without touching the interface. See [twin LLM adapters](../../.agents/notes/implemented/architecture/2026-06-13-twin-llm-adapters.md) for the contract-validation origin of the two shipping implementations and the [replay token meter Agent Note](../../.agents/notes/implemented/architecture/2026-07-15-replay-token-meter-service.md) for measurement ownership.
|
||||
|
||||
@@ -2,7 +2,9 @@
|
||||
|
||||
DeepSeek chat-completions adapter for the harness LLM seam: hand-rolled `fetch` + SSE translation from the official wire format (source of truth: the API docs — guides/thinking_mode, guides/tool_calls, api/create-chat-completion) into the `StreamChunk` protocol.
|
||||
|
||||
A second, independent implementation of the same seam exists in `@deepseek-ai/dsh-llm-pi-ai` (library-backed). Same Config shape — pick one per context (registering both for the same model names throws by design).
|
||||
A second, library-backed implementation of the same seam exists in `@deepseek-ai/dsh-llm-pi-ai`. This package always owns the `deepseek` provider route; mounting a pi-ai profile with `provider: deepseek` in the same context throws `LlmError('DUPLICATE_ADAPTER')` by design.
|
||||
|
||||
The package root exposes the Cordis plugin contract and `DeepSeekAdapter`; wire serialization, SSE parsing, and chunk translation helpers are not part of that root contract.
|
||||
|
||||
## Config
|
||||
|
||||
@@ -12,17 +14,24 @@ A second, independent implementation of the same seam exists in `@deepseek-ai/ds
|
||||
config:
|
||||
apiKey: !!js process.env.DEEPSEEK_API_KEY # or rely on the env fallback
|
||||
baseURL: !!js process.env.DEEPSEEK_BASE_URL # default: https://api.deepseek.com
|
||||
models: [deepseek-v4-flash, deepseek-v4-pro] # one adapter, registered for each name
|
||||
thinking: enabled # optional; provider default is enabled
|
||||
reasoningEffort: high # optional; high | max — omitted ⇒ not sent
|
||||
streamIdleTimeoutMs: 300000 # optional; positive finite Node timer delay; five-minute default
|
||||
models: # optional; defaults to V4 Flash and V4 Pro
|
||||
- id: deepseek-v4-flash
|
||||
name: DeepSeek V4 Flash
|
||||
- id: private-reasoner
|
||||
description: Company-hosted reasoning model
|
||||
```
|
||||
|
||||
`models` lists every model name this one adapter instance serves: the adapter registers itself for each (the harness model name IS the wire `model` string), so a `generate`/`stream` call routes to it whenever `options.model` is any of them. Registering a second adapter for a name already taken throws `LlmError('DUPLICATE_ADAPTER')` (the LLM service enforces one adapter per model, all-or-nothing).
|
||||
The plugin registers the single provider route `deepseek`. A request selects it with `provider: deepseek`; its `model` is passed through as the wire `model` string, so changing DeepSeek models does not require lifecycle-time registration. Omitting `models` advertises `deepseek-v4-flash` and `deepseek-v4-pro`; an explicit list replaces those defaults, while `models: []` advertises none. Catalog entries are exposed through `ctx.llm.listModels('deepseek')` for clients such as ACP editors, but remain advisory: unlisted model ids still pass through unchanged. An omitted entry name defaults to its id. Registering another adapter for `deepseek` throws `LlmError('DUPLICATE_ADAPTER')`.
|
||||
|
||||
`reasoningEffort` is **omitted by default** — when unset, the `reasoning_effort` wire field is not sent and the server applies its own default for the model. The only accepted values are `high` and `max` (DeepSeek's official effort levels). It is meaningful only with thinking enabled (the provider default).
|
||||
|
||||
`thinking`/`reasoningEffort` are adapter-level request defaults serialized as the official top-level `thinking: {type}` / `reasoning_effort` wire fields. They live in adapter config (not `GenerateOptions`) to keep the core vocabulary provider-neutral.
|
||||
|
||||
`streamIdleTimeoutMs` bounds each outstanding provider read, including the initial `fetch`, without counting time the consumer spends between chunks. One stable abort signal reaches the request and body reader for the whole call; expiry stops the transport and throws `LlmError('TIMEOUT')`, while an earlier caller abort throws `LlmError('ABORTED')`. The adapter makes exactly one provider request per `stream()` call; agent-level retry is a separate plugin policy.
|
||||
|
||||
## App attribution
|
||||
|
||||
Every request carries the shared attribution header from dsh-llm's `attributionHeaders()` - the mandatory `User-Agent` baseline identifying the harness (see [dsh-llm § App attribution](../llm/README.md#app-attribution-attributionts)). Direct DeepSeek requests and OpenAI-compatible gateway requests get no provider-specific app-attribution headers under this adapter contract; OpenRouter app attribution is deferred to a future explicit OpenRouter adapter or mode.
|
||||
@@ -36,25 +45,41 @@ Every request carries the shared attribution header from dsh-llm's `attributionH
|
||||
|
||||
## Errors
|
||||
|
||||
Non-2xx responses throw `LlmError` with stable codes: `AUTH` (401/403), `RATE_LIMIT` (429), `INVALID_REQUEST` (400), `SERVER` (5xx), `HTTP_<status>` otherwise. Protocol violations throw `STREAM_CLOSED` (no `[DONE]`) or `MALFORMED_RESPONSE` (bad JSON payload). Unknown wire `finish_reason`s (e.g. `content_filter`, `insufficient_system_resource`) become `finish {kind: 'error', code: <REASON>}` chunks.
|
||||
Non-2xx responses throw `LlmError` with stable codes: `AUTH` (401/403), `QUOTA` (a response whose provider details identify exhausted quota, balance, or credits), `RATE_LIMIT` (other 429s), `CONTEXT_WINDOW_EXCEEDED` (a 400 whose provider code, type, or message identifies context overflow), `INVALID_REQUEST` (other 400s), `SERVER` (5xx), `HTTP_<status>` otherwise. Its serializable `failure` retains the HTTP status plus a valid positive `Retry-After` seconds/date delay and `x-request-id` / `x-deepseek-request-id` when present. A pre-response transport failure (DNS, refused connection, TLS, proxy) throws `TRANSPORT` naming the configured endpoint and chaining the original rejection as `cause`; caller aborts throw `ABORTED`, and the loop's cancellation signal remains authoritative. Protocol violations throw `STREAM_CLOSED` (no `[DONE]`) or `MALFORMED_RESPONSE` (bad JSON payload). Unknown wire `finish_reason`s (e.g. `content_filter`, `insufficient_system_resource`) become `finish {kind: 'error', failure}` chunks.
|
||||
|
||||
## Testing
|
||||
|
||||
Unit suites run against a local `node:http` mock SSE server (no network). Real-API coverage lives in `tests/adapter.e2e.ts` (`pnpm run test:e2e`, key-gated): V4 Flash + V4 Pro across thinking enabled/disabled and both official effort levels, including the thinking+tools round trip with reasoning passback.
|
||||
Unit suites run against a local `node:http` mock SSE server (no network), including structured HTTP facts, malformed/truncated streams, caller abort, connection failure, and proof that idle timeout aborts the actual body. Real-API coverage lives in `tests/adapter.e2e.ts` (`pnpm run test:e2e`, key-gated): V4 Flash + V4 Pro across thinking enabled/disabled and both official effort levels, including the thinking+tools round trip with reasoning passback.
|
||||
|
||||
## Model Experience
|
||||
|
||||
### DeepSeek request
|
||||
|
||||
**What the model sees**: The selected DeepSeek model receives the harness system prompt, message history, tool schemas, stop sequences, and call config without adapter-authored prompt prose. On a prior assistant turn with tool calls, its reasoning content is passed back as required; reasoning from tool-call-free turns is omitted.
|
||||
#### What the model sees
|
||||
|
||||
**Token effect**: Provider tokenization governs exact input. Conditional reasoning passback increases tool-round-trip context, while dropping other reasoning avoids paying those tokens again; cache-read usage is reported when available.
|
||||
The selected DeepSeek model receives the harness system prompt, message history, tool schemas, stop sequences, and call config without adapter-authored prompt prose. On a prior assistant turn with tool calls, its reasoning content is passed back as required; reasoning from tool-call-free turns is omitted.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Provider tokenization governs exact input. Conditional reasoning passback increases tool-round-trip context, while dropping other reasoning avoids paying those tokens again; cache-read usage is reported when available.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
An unchanged assembled prefix is eligible for DeepSeek cache reuse, which this adapter reports in usage. A model-route change or any upstream prompt, schema, prefix, or history change may prevent reuse from the first changed token; reasoning passback appends during tool round trips.
|
||||
|
||||
### DeepSeek response
|
||||
|
||||
**What the model sees**: Reasoning, text, and raw-string tool arguments are translated into harness chunks for the loop to log and assemble.
|
||||
#### What the model sees
|
||||
|
||||
**Token effect**: Generated tokens follow provider thinking and effort settings plus the request's `maxTokens`; only loop-retained blocks affect later input.
|
||||
Reasoning, text, and raw-string tool arguments are translated into harness chunks for the loop to log and assemble.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Generated tokens follow provider thinking and effort settings plus the request's `maxTokens`; only loop-retained blocks affect later input.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
Loop-retained response blocks append to the next request and preserve its earlier reusable prefix; dropped blocks have no later cache effect. Changing the provider or model selects a different cache domain.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
|
||||
@@ -23,6 +23,7 @@
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-llm": "^0.0.1",
|
||||
"@deepseek-ai/dsh-timeout": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"dependencies": {
|
||||
@@ -30,6 +31,7 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-timeout": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -5,14 +5,25 @@
|
||||
* @module dsh-llm-deepseek/adapter
|
||||
*/
|
||||
|
||||
import { attributionHeaders, LlmAdapter, LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import { attributionHeaders, CONTEXT_WINDOW_EXCEEDED_CODE, isContextWindowExceededError, isQuotaExceededError, LlmAdapter, LlmError, ProviderRequestId, QUOTA_EXCEEDED_CODE } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, LlmModelInfo, LlmProviderInfo, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import { idleWatchdog, MAX_TIMER_DELAY_MS, timeoutOf } from '@deepseek-ai/dsh-timeout'
|
||||
import { serializeRequest } from './serialize.ts'
|
||||
import type { RequestDefaults } from './serialize.ts'
|
||||
import { parseSse } from './sse.ts'
|
||||
import { translate } from './translate.ts'
|
||||
import type { WireError } from './types.ts'
|
||||
|
||||
/** One optional model entry advertised by the hand-written adapter. */
|
||||
export interface DeepSeekCatalogModel {
|
||||
/** Wire model id accepted by the configured endpoint. */
|
||||
id: string
|
||||
/** Selector label; defaults to {@link id}. */
|
||||
name?: string
|
||||
/** Optional selector detail for deployments with similar model variants. */
|
||||
description?: string
|
||||
}
|
||||
|
||||
/** Constructor options for {@link DeepSeekAdapter}; the plugin's `apply` resolves them from Config + environment. */
|
||||
export interface DeepSeekAdapterOptions {
|
||||
/** Bearer token sent in the `authorization` header on every request. */
|
||||
@@ -21,17 +32,46 @@ export interface DeepSeekAdapterOptions {
|
||||
baseURL: string
|
||||
/** Request defaults applied to every call (thinking mode, effort). */
|
||||
defaults?: RequestDefaults
|
||||
/** Advisory models exposed to discovery consumers; requests remain unrestricted. */
|
||||
models?: readonly DeepSeekCatalogModel[]
|
||||
/** Maximum provider idle time while one stream read is outstanding. */
|
||||
streamIdleTimeoutMs?: number
|
||||
}
|
||||
|
||||
/** Default maximum idle interval while an adapter stream read is outstanding. */
|
||||
export const DEFAULT_STREAM_IDLE_TIMEOUT_MS = 300_000
|
||||
const STREAM_IDLE_TIMEOUT_CODE = 'LLM_STREAM_IDLE_TIMEOUT'
|
||||
|
||||
function providerRetryAfterMs(value: string | null): number | undefined {
|
||||
if (value === null) return undefined
|
||||
if (/^\d+$/.test(value)) {
|
||||
const delay = Number(value) * 1_000
|
||||
return Number.isFinite(delay) && delay > 0 ? delay : undefined
|
||||
}
|
||||
const delay = Date.parse(value) - Date.now()
|
||||
return Number.isFinite(delay) && delay > 0 ? delay : undefined
|
||||
}
|
||||
|
||||
function requestId(headers: Headers): ReturnType<typeof ProviderRequestId> | undefined {
|
||||
const value = headers.get('x-request-id') ?? headers.get('x-deepseek-request-id')
|
||||
return value === null || value.length === 0 ? undefined : ProviderRequestId(value)
|
||||
}
|
||||
|
||||
/**
|
||||
* Map an HTTP status to a stable LlmError code.
|
||||
* @param status - status of a non-2xx provider response.
|
||||
* @returns `AUTH` (401/403), `RATE_LIMIT` (429), `INVALID_REQUEST` (400), `SERVER` (5xx), or `HTTP_<status>` for anything else.
|
||||
* @param error - parsed provider error body, when available.
|
||||
* @returns the normalized harness error code.
|
||||
*/
|
||||
export function httpErrorCode(status: number): string {
|
||||
export function httpErrorCode(status: number, error?: WireError['error']): string {
|
||||
if (status === 401 || status === 403) return 'AUTH'
|
||||
const detail = [error?.code, error?.type, error?.message].filter(Boolean).join(' ')
|
||||
if (isQuotaExceededError(detail)) return QUOTA_EXCEEDED_CODE
|
||||
if (status === 429) return 'RATE_LIMIT'
|
||||
if (status === 400) return 'INVALID_REQUEST'
|
||||
if (status === 400) {
|
||||
if (isContextWindowExceededError(detail)) return CONTEXT_WINDOW_EXCEEDED_CODE
|
||||
return 'INVALID_REQUEST'
|
||||
}
|
||||
if (status >= 500) return 'SERVER'
|
||||
return `HTTP_${status}`
|
||||
}
|
||||
@@ -40,43 +80,136 @@ export function httpErrorCode(status: number): string {
|
||||
* The first real `LlmAdapter`. One instance serves every model name it was
|
||||
* registered under (the harness model name IS the wire model name).
|
||||
*
|
||||
* Abort: `options.signal` is handed to fetch — both the initial request and
|
||||
* the body stream reject on abort, which surfaces to the loop as a rejected
|
||||
* step (the loop already contains step errors).
|
||||
* One stable signal reaches both initial fetch and body reads. Caller aborts
|
||||
* map to `ABORTED`; the configured per-read idle watchdog maps to `TIMEOUT`.
|
||||
*/
|
||||
export class DeepSeekAdapter extends LlmAdapter {
|
||||
private readonly streamIdleTimeoutMs: number
|
||||
|
||||
constructor(private readonly options: DeepSeekAdapterOptions) {
|
||||
super()
|
||||
this.streamIdleTimeoutMs = options.streamIdleTimeoutMs ?? DEFAULT_STREAM_IDLE_TIMEOUT_MS
|
||||
if (!Number.isFinite(this.streamIdleTimeoutMs)
|
||||
|| this.streamIdleTimeoutMs <= 0
|
||||
|| this.streamIdleTimeoutMs > MAX_TIMER_DELAY_MS) {
|
||||
throw new Error(
|
||||
`llm-deepseek: streamIdleTimeoutMs must be a positive finite number no greater than ${MAX_TIMER_DELAY_MS}`,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
override providerInfo(provider: string): LlmProviderInfo {
|
||||
return { id: provider, name: 'DeepSeek' }
|
||||
}
|
||||
|
||||
override listModels(provider: string): Promise<readonly LlmModelInfo[]> {
|
||||
return Promise.resolve((this.options.models ?? []).map(model => ({
|
||||
provider,
|
||||
id: model.id,
|
||||
name: model.name ?? model.id,
|
||||
...model.description === undefined ? {} : { description: model.description },
|
||||
})))
|
||||
}
|
||||
|
||||
async * stream(options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
const consumer = new AbortController()
|
||||
const upstream = options.signal === undefined
|
||||
? consumer.signal
|
||||
: AbortSignal.any([options.signal, consumer.signal])
|
||||
using watchdog = idleWatchdog(upstream, this.streamIdleTimeoutMs, STREAM_IDLE_TIMEOUT_CODE)
|
||||
const iterator = this.request(options, watchdog.signal)[Symbol.asyncIterator]()
|
||||
let exhausted = false
|
||||
try {
|
||||
while (true) {
|
||||
const result = await watchdog.next(iterator)
|
||||
if (result.done) {
|
||||
exhausted = true
|
||||
return
|
||||
}
|
||||
yield result.value
|
||||
}
|
||||
} catch (error: unknown) {
|
||||
if (timeoutOf(watchdog.signal, STREAM_IDLE_TIMEOUT_CODE) !== undefined) {
|
||||
throw new LlmError(
|
||||
`DeepSeek stream idle timeout after ${this.streamIdleTimeoutMs}ms`,
|
||||
'TIMEOUT',
|
||||
{ cause: error },
|
||||
)
|
||||
}
|
||||
if (options.signal?.aborted) {
|
||||
throw new LlmError('DeepSeek request aborted by caller', 'ABORTED', { cause: error })
|
||||
}
|
||||
if (error instanceof LlmError) throw error
|
||||
throw new LlmError(`DeepSeek API stream from ${this.options.baseURL} failed`, 'TRANSPORT', { cause: error })
|
||||
} finally {
|
||||
consumer.abort('DeepSeek stream consumer stopped')
|
||||
if (!exhausted && iterator.return !== undefined) {
|
||||
try {
|
||||
await iterator.return()
|
||||
} catch (_abortedTransportTeardown) {
|
||||
// The consumer controller already owns termination; a return-time abort cannot add a second outcome.
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async * request(options: GenerateOptions, signal: AbortSignal): AsyncIterable<StreamChunk> {
|
||||
const body = serializeRequest(options, this.options.defaults ?? {})
|
||||
// Prepared outside the try so the TRANSPORT label below covers exactly the
|
||||
// transport boundary, never a serialization failure.
|
||||
const payload = JSON.stringify(body)
|
||||
const headers = {
|
||||
'authorization': `Bearer ${this.options.apiKey}`,
|
||||
'content-type': 'application/json',
|
||||
'accept': 'text/event-stream',
|
||||
...attributionHeaders(),
|
||||
...options.sessionId !== undefined
|
||||
? { 'x-deepseek-harness-session-id': String(options.sessionId) }
|
||||
: {},
|
||||
}
|
||||
|
||||
// TODO(http): adopt the Cordis HTTP service when shared transport configuration
|
||||
// outweighs its additional runtime dependencies.
|
||||
const response = await fetch(`${this.options.baseURL}/chat/completions`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'authorization': `Bearer ${this.options.apiKey}`,
|
||||
'content-type': 'application/json',
|
||||
'accept': 'text/event-stream',
|
||||
...attributionHeaders(),
|
||||
},
|
||||
body: JSON.stringify(body),
|
||||
...options.signal ? { signal: options.signal } : {},
|
||||
})
|
||||
let response: Response
|
||||
try {
|
||||
response = await fetch(`${this.options.baseURL}/chat/completions`, {
|
||||
method: 'POST',
|
||||
headers,
|
||||
body: payload,
|
||||
signal,
|
||||
})
|
||||
} catch (error: unknown) {
|
||||
// The outer stream distinguishes caller cancellation and watchdog expiry.
|
||||
if (signal.aborted) throw error
|
||||
// fetch wraps every transport failure (DNS, refused connection, TLS,
|
||||
// proxy) in a bare `TypeError: fetch failed` whose actionable detail
|
||||
// lives on `cause`. Wrapping with the endpoint and chaining the cause
|
||||
// lets `errorChain` render the full diagnosis at every reporting seam.
|
||||
throw new LlmError(
|
||||
`DeepSeek API request to ${this.options.baseURL} failed`,
|
||||
'TRANSPORT',
|
||||
{ cause: error },
|
||||
)
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
const code = httpErrorCode(response.status)
|
||||
let message = `DeepSeek API error (HTTP ${response.status})`
|
||||
let providerError: WireError['error']
|
||||
try {
|
||||
const parsed = await response.json() as WireError
|
||||
if (parsed.error?.message) message = parsed.error.message
|
||||
providerError = parsed.error
|
||||
if (providerError?.message) message = providerError.message
|
||||
} catch {
|
||||
// Only swallow error-body parsing: status and code are already captured,
|
||||
// so malformed gateway JSON must not mask the actionable HTTP failure.
|
||||
// Only swallow error-body parsing: the HTTP status still identifies the
|
||||
// failure, so malformed gateway JSON must not mask it.
|
||||
}
|
||||
throw new LlmError(message, code, response.status)
|
||||
const delay = providerRetryAfterMs(response.headers.get('retry-after'))
|
||||
const id = requestId(response.headers)
|
||||
throw new LlmError(message, httpErrorCode(response.status, providerError), {
|
||||
status: response.status,
|
||||
...delay === undefined ? {} : { providerRetryAfterMs: delay },
|
||||
...id === undefined ? {} : { requestId: id },
|
||||
})
|
||||
}
|
||||
if (!response.body) {
|
||||
throw new LlmError('DeepSeek API returned no response body', 'EMPTY_RESPONSE')
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
/**
|
||||
* Register a {@link DeepSeekAdapter} for configured model names on `ctx.llm`. Configuration uses
|
||||
* Register a {@link DeepSeekAdapter} for the `deepseek` provider route on `ctx.llm`. Configuration uses
|
||||
* Cordis schemastery; pass secrets from environment variables through `cordis.yml` with `!!js`,
|
||||
* as shown in the package README, rather than reading ad hoc files.
|
||||
* @module @deepseek-ai/dsh-llm-deepseek
|
||||
@@ -8,19 +8,23 @@
|
||||
import type { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import type {} from '@deepseek-ai/dsh-llm'
|
||||
import { DeepSeekAdapter } from './adapter.ts'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
import { DEFAULT_STREAM_IDLE_TIMEOUT_MS, DeepSeekAdapter } from './adapter.ts'
|
||||
import type { DeepSeekCatalogModel } from './adapter.ts'
|
||||
|
||||
export { DeepSeekAdapter, httpErrorCode } from './adapter.ts'
|
||||
export type { DeepSeekAdapterOptions } from './adapter.ts'
|
||||
export { serializeMessages, serializeRequest } from './serialize.ts'
|
||||
export { DeepSeekAdapter } from './adapter.ts'
|
||||
export type { DeepSeekAdapterOptions, DeepSeekCatalogModel } from './adapter.ts'
|
||||
export type { RequestDefaults } from './serialize.ts'
|
||||
export { DONE, parseSse } from './sse.ts'
|
||||
export { mapFinishReason, mapUsage, translate } from './translate.ts'
|
||||
export type * from './types.ts'
|
||||
|
||||
export const name = 'llm-deepseek'
|
||||
export const inject = ['llm']
|
||||
|
||||
const DEFAULT_MODELS: DeepSeekCatalogModel[] = [
|
||||
{ id: 'deepseek-v4-flash' },
|
||||
{ id: 'deepseek-v4-pro' },
|
||||
]
|
||||
|
||||
/**
|
||||
* Plugin config, validated by the same-named schemastery schema. Every field
|
||||
* is optional in yml: credentials/endpoint fall back to the environment (a
|
||||
@@ -32,40 +36,66 @@ export interface Config {
|
||||
apiKey?: string
|
||||
/** Endpoint base; falls back to $DEEPSEEK_BASE_URL, then the public API. */
|
||||
baseURL?: string
|
||||
/** Model names to register (sent verbatim on the wire). */
|
||||
models?: string[]
|
||||
/** Thinking-mode default for every request (provider default: enabled). */
|
||||
thinking?: 'enabled' | 'disabled'
|
||||
/** Thinking effort (only meaningful with thinking enabled). */
|
||||
reasoningEffort?: 'high' | 'max'
|
||||
/** Advisory models shown by discovery consumers; defaults to V4 Flash and V4 Pro. */
|
||||
models?: DeepSeekCatalogModel[]
|
||||
/** Maximum provider idle time while one stream read is outstanding (default five minutes). */
|
||||
streamIdleTimeoutMs?: number
|
||||
}
|
||||
|
||||
const catalogModel: z<DeepSeekCatalogModel> = z.object({
|
||||
id: z.string().required(),
|
||||
name: z.string(),
|
||||
description: z.string(),
|
||||
})
|
||||
|
||||
export const Config: z<Config> = z.object({
|
||||
apiKey: z.string(),
|
||||
baseURL: z.string(),
|
||||
models: z.array(z.string()).default(['deepseek-v4-flash', 'deepseek-v4-pro']),
|
||||
thinking: z.union(['enabled', 'disabled']),
|
||||
reasoningEffort: z.union(['high', 'max']),
|
||||
models: z.array(catalogModel).default(DEFAULT_MODELS),
|
||||
streamIdleTimeoutMs: z.number().min(Number.MIN_VALUE).max(MAX_TIMER_DELAY_MS).default(DEFAULT_STREAM_IDLE_TIMEOUT_MS),
|
||||
})
|
||||
|
||||
/** Public API default; the internal endpoint comes from $DEEPSEEK_BASE_URL. */
|
||||
export const PUBLIC_BASE_URL = 'https://api.deepseek.com'
|
||||
|
||||
/** Resolve, validate, and detach the advisory model catalog. */
|
||||
function resolveModels(models: readonly DeepSeekCatalogModel[] | undefined): DeepSeekCatalogModel[] {
|
||||
const seen = new Set<string>()
|
||||
return (models ?? DEFAULT_MODELS).map((model) => {
|
||||
if (model.id.length === 0) throw new Error('llm-deepseek: catalog model ids must be non-empty')
|
||||
if (model.name !== undefined && model.name.length === 0) {
|
||||
throw new Error(`llm-deepseek: catalog model "${model.id}" has an empty name`)
|
||||
}
|
||||
if (seen.has(model.id)) throw new Error(`llm-deepseek: duplicate catalog model "${model.id}"`)
|
||||
seen.add(model.id)
|
||||
return {
|
||||
id: model.id,
|
||||
...model.name === undefined ? {} : { name: model.name },
|
||||
...model.description === undefined ? {} : { description: model.description },
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
export function apply(ctx: Context, config: Config): void {
|
||||
const apiKey = config.apiKey ?? process.env.DEEPSEEK_API_KEY
|
||||
if (apiKey === undefined || apiKey.length === 0) {
|
||||
throw new Error('llm-deepseek: an API key is required (Config.apiKey or $DEEPSEEK_API_KEY)')
|
||||
}
|
||||
const baseURL = config.baseURL ?? process.env.DEEPSEEK_BASE_URL ?? PUBLIC_BASE_URL
|
||||
// schemastery's .default() guarantees models is set after validation.
|
||||
const models = config.models as string[]
|
||||
|
||||
ctx.llm.registerAdapter(models, new DeepSeekAdapter({
|
||||
ctx.llm.registerAdapter(['deepseek'], new DeepSeekAdapter({
|
||||
apiKey,
|
||||
baseURL,
|
||||
defaults: {
|
||||
thinking: config.thinking,
|
||||
reasoningEffort: config.reasoningEffort,
|
||||
},
|
||||
models: resolveModels(config.models),
|
||||
streamIdleTimeoutMs: config.streamIdleTimeoutMs ?? DEFAULT_STREAM_IDLE_TIMEOUT_MS,
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -35,7 +35,10 @@ export function mapFinishReason(reason: string): FinishReason {
|
||||
case 'length': return { kind: 'max-tokens' }
|
||||
default:
|
||||
// content_filter, insufficient_system_resource, future additions.
|
||||
return { kind: 'error', message: `model stopped: ${reason}`, code: reason.toUpperCase() }
|
||||
return {
|
||||
kind: 'error',
|
||||
failure: { message: `model stopped: ${reason}`, code: reason.toUpperCase() },
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -16,11 +16,11 @@ const FLASH = 'deepseek-v4-flash'
|
||||
const PRO = 'deepseek-v4-pro'
|
||||
const contexts: Context[] = []
|
||||
|
||||
async function harness(model: string, config: Partial<Config> = {}) {
|
||||
async function harness(_model: string, config: Partial<Config> = {}) {
|
||||
const ctx = new Context()
|
||||
contexts.push(ctx)
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmDeepSeek, { models: [model], ...config })
|
||||
await ctx.plugin(LlmDeepSeek, config)
|
||||
return ctx
|
||||
}
|
||||
|
||||
@@ -134,6 +134,7 @@ describe.skipIf(!process.env.DEEPSEEK_API_KEY)('llm-deepseek e2e (real API)', ()
|
||||
const ctx = await harness(FLASH, { thinking: 'disabled' })
|
||||
const kinds: string[] = []
|
||||
for await (const chunk of ctx.llm.stream({
|
||||
provider: 'deepseek',
|
||||
model: FLASH,
|
||||
messages: ask('Count from 1 to 5, digits only.'),
|
||||
maxTokens: 50,
|
||||
|
||||
@@ -2,15 +2,25 @@ import { createServer } from 'node:http'
|
||||
import type { IncomingMessage, Server, ServerResponse } from 'node:http'
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import LlmService, { LlmError, userAgent } from '@deepseek-ai/dsh-llm'
|
||||
import LlmService, {
|
||||
CONTEXT_WINDOW_EXCEEDED_CODE,
|
||||
errorChain,
|
||||
LlmError,
|
||||
ProviderRequestId,
|
||||
QUOTA_EXCEEDED_CODE,
|
||||
userAgent,
|
||||
} from '@deepseek-ai/dsh-llm'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
import { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import * as LlmDeepSeek from '@deepseek-ai/dsh-llm-deepseek'
|
||||
import { DeepSeekAdapter, httpErrorCode } from '@deepseek-ai/dsh-llm-deepseek'
|
||||
import { DeepSeekAdapter } from '@deepseek-ai/dsh-llm-deepseek'
|
||||
import { httpErrorCode } from '../src/adapter.ts'
|
||||
import { assemble } from './assemble.ts'
|
||||
|
||||
/** One scripted behavior for the next request the mock server receives. */
|
||||
type Behavior =
|
||||
| { kind: 'sse'; events: string[]; delayMs?: number }
|
||||
| { kind: 'http-error'; status: number; body: string; contentType?: string }
|
||||
| { kind: 'http-error'; status: number; body: string; contentType?: string; headers?: Record<string, string> }
|
||||
| { kind: 'close-early'; events: string[] }
|
||||
|
||||
interface MockServer {
|
||||
@@ -28,6 +38,7 @@ const servers: Server[] = []
|
||||
afterEach(async () => {
|
||||
await Promise.all(servers.splice(0).map(server => new Promise(resolve => server.close(resolve))))
|
||||
vi.unstubAllEnvs()
|
||||
vi.useRealTimers()
|
||||
})
|
||||
|
||||
/** Local chat-completions stand-in: replays scripted behaviors per request. */
|
||||
@@ -46,7 +57,10 @@ async function mockServer(script: Behavior[]): Promise<MockServer> {
|
||||
return
|
||||
}
|
||||
if (behavior.kind === 'http-error') {
|
||||
response.writeHead(behavior.status, { 'content-type': behavior.contentType ?? 'application/json' })
|
||||
response.writeHead(behavior.status, {
|
||||
'content-type': behavior.contentType ?? 'application/json',
|
||||
...behavior.headers,
|
||||
})
|
||||
response.end(behavior.body)
|
||||
return
|
||||
}
|
||||
@@ -86,7 +100,7 @@ const textEvents = [
|
||||
async function harness(baseURL: string, config: object = {}) {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmDeepSeek, { apiKey: 'test-key', baseURL, models: ['deepseek-v4-flash'], ...config })
|
||||
await ctx.plugin(LlmDeepSeek, { apiKey: 'test-key', baseURL, ...config })
|
||||
return ctx
|
||||
}
|
||||
|
||||
@@ -123,6 +137,7 @@ describe('DeepSeekAdapter against a mock server', () => {
|
||||
|
||||
const kinds: string[] = []
|
||||
for await (const chunk of ctx.llm.stream({
|
||||
provider: 'deepseek',
|
||||
model: 'deepseek-v4-flash',
|
||||
messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }],
|
||||
})) {
|
||||
@@ -131,6 +146,19 @@ describe('DeepSeekAdapter against a mock server', () => {
|
||||
expect(kinds).toEqual(['block-start', 'text-delta', 'block-end', 'usage', 'finish'])
|
||||
})
|
||||
|
||||
it('forwards the harness session id for host-side trajectory routing', async () => {
|
||||
const server = await mockServer([{ kind: 'sse', events: textEvents }])
|
||||
const ctx = await harness(server.url)
|
||||
|
||||
await assemble(ctx, {
|
||||
model: 'deepseek-v4-flash',
|
||||
messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }],
|
||||
sessionId: SessionId('child-session'),
|
||||
})
|
||||
|
||||
expect(server.headers[0]?.['x-deepseek-harness-session-id']).toBe('child-session')
|
||||
})
|
||||
|
||||
it('forwards thinking config onto the wire', async () => {
|
||||
const server = await mockServer([{ kind: 'sse', events: textEvents }])
|
||||
const ctx = await harness(server.url, { thinking: 'disabled', reasoningEffort: 'high' })
|
||||
@@ -158,7 +186,7 @@ describe('DeepSeekAdapter against a mock server', () => {
|
||||
status,
|
||||
body: JSON.stringify({ error: { message: `failed with ${status}`, type: 't', code: 'c' } }),
|
||||
}
|
||||
const server = await mockServer([behavior, behavior, behavior])
|
||||
const server = await mockServer([behavior, behavior])
|
||||
const ctx = await harness(server.url)
|
||||
await expect(assemble(ctx,{ model: 'deepseek-v4-flash', messages: [] }))
|
||||
.rejects.toThrow(`failed with ${status}`)
|
||||
@@ -166,11 +194,116 @@ describe('DeepSeekAdapter against a mock server', () => {
|
||||
assemble(ctx,{ model: 'deepseek-v4-flash', messages: [] })
|
||||
.catch((error: unknown) => (error as LlmError).code),
|
||||
).resolves.toBe(code)
|
||||
// The numeric HTTP status is carried on the error for explicit handling.
|
||||
await expect(
|
||||
assemble(ctx,{ model: 'deepseek-v4-flash', messages: [] })
|
||||
.catch((error: unknown) => (error as LlmError).status),
|
||||
).resolves.toBe(status)
|
||||
})
|
||||
|
||||
it('classifies a thrown HTTP context-window rejection with the canonical code', async () => {
|
||||
const server = await mockServer([{
|
||||
kind: 'http-error',
|
||||
status: 400,
|
||||
body: JSON.stringify({
|
||||
error: {
|
||||
message: 'This model maximum context length is 128000 tokens; your input exceeds that limit.',
|
||||
type: 'invalid_request_error',
|
||||
code: 'context_length_exceeded',
|
||||
},
|
||||
}),
|
||||
}])
|
||||
const ctx = await harness(server.url)
|
||||
const code = await assemble(ctx, { model: 'deepseek-v4-flash', messages: [] })
|
||||
.catch((error: unknown) => (error as LlmError).code)
|
||||
expect(code).toBe(CONTEXT_WINDOW_EXCEEDED_CODE)
|
||||
})
|
||||
|
||||
it('retains status, Retry-After seconds, and provider request id as structured facts', async () => {
|
||||
const server = await mockServer([{
|
||||
kind: 'http-error',
|
||||
status: 429,
|
||||
body: JSON.stringify({ error: { message: 'slow down' } }),
|
||||
headers: { 'retry-after': '2', 'x-request-id': 'req-429' },
|
||||
}])
|
||||
const ctx = await harness(server.url)
|
||||
let thrown: unknown
|
||||
try {
|
||||
await assemble(ctx, { model: 'deepseek-v4-flash', messages: [] })
|
||||
} catch (error: unknown) {
|
||||
thrown = error
|
||||
}
|
||||
expect(thrown).toBeInstanceOf(LlmError)
|
||||
expect((thrown as LlmError).failure).toEqual({
|
||||
message: 'slow down',
|
||||
code: 'RATE_LIMIT',
|
||||
status: 429,
|
||||
providerRetryAfterMs: 2_000,
|
||||
requestId: ProviderRequestId('req-429'),
|
||||
})
|
||||
})
|
||||
|
||||
it('parses a future Retry-After HTTP date and the DeepSeek request-id fallback', async () => {
|
||||
const now = 1_800_000_000_000
|
||||
const dateNow = vi.spyOn(Date, 'now').mockReturnValue(now)
|
||||
try {
|
||||
const server = await mockServer([{
|
||||
kind: 'http-error',
|
||||
status: 503,
|
||||
body: JSON.stringify({ error: { message: 'come back later' } }),
|
||||
headers: {
|
||||
'retry-after': new Date(now + 3_000).toUTCString(),
|
||||
'x-deepseek-request-id': 'deepseek-503',
|
||||
},
|
||||
}])
|
||||
const ctx = await harness(server.url)
|
||||
await expect(assemble(ctx, { model: 'deepseek-v4-flash', messages: [] }))
|
||||
.rejects.toMatchObject({
|
||||
failure: {
|
||||
message: 'come back later',
|
||||
code: 'SERVER',
|
||||
status: 503,
|
||||
providerRetryAfterMs: 3_000,
|
||||
requestId: ProviderRequestId('deepseek-503'),
|
||||
},
|
||||
})
|
||||
} finally {
|
||||
dateNow.mockRestore()
|
||||
}
|
||||
})
|
||||
|
||||
it('omits zero, non-finite, invalid, and past Retry-After values', async () => {
|
||||
const values = [
|
||||
'0',
|
||||
'9'.repeat(400),
|
||||
'not-a-date',
|
||||
new Date(0).toUTCString(),
|
||||
]
|
||||
for (const value of values) {
|
||||
const server = await mockServer([{
|
||||
kind: 'http-error',
|
||||
status: 429,
|
||||
body: JSON.stringify({ error: { message: 'retry later' } }),
|
||||
headers: { 'retry-after': value },
|
||||
}])
|
||||
const ctx = await harness(server.url)
|
||||
let thrown: LlmError | undefined
|
||||
try {
|
||||
await assemble(ctx, { model: 'deepseek-v4-flash', messages: [] })
|
||||
} catch (error: unknown) {
|
||||
if (error instanceof LlmError) thrown = error
|
||||
}
|
||||
expect(thrown?.failure).toEqual({ message: 'retry later', code: 'RATE_LIMIT', status: 429 })
|
||||
}
|
||||
})
|
||||
|
||||
it('classifies only context-capacity HTTP 400 details as context overflow', () => {
|
||||
expect(httpErrorCode(400, { message: 'request too large for model context' }))
|
||||
.toBe(CONTEXT_WINDOW_EXCEEDED_CODE)
|
||||
expect(httpErrorCode(400, { message: 'invalid input: temperature exceeds maximum allowed value' }))
|
||||
.toBe('INVALID_REQUEST')
|
||||
expect(httpErrorCode(413, { code: 'context_length_exceeded' })).toBe('HTTP_413')
|
||||
})
|
||||
|
||||
it('distinguishes terminal quota exhaustion from transient HTTP 429 throttling', () => {
|
||||
expect(httpErrorCode(429, { code: 'insufficient_quota', message: 'account credits exhausted' }))
|
||||
.toBe(QUOTA_EXCEEDED_CODE)
|
||||
expect(httpErrorCode(429, { message: 'request rate limit exceeded' })).toBe('RATE_LIMIT')
|
||||
})
|
||||
|
||||
it('keeps the status-line message for JSON error bodies without a message', async () => {
|
||||
@@ -191,6 +324,40 @@ describe('DeepSeekAdapter against a mock server', () => {
|
||||
expect(httpErrorCode(418)).toBe('HTTP_418')
|
||||
})
|
||||
|
||||
it('wraps a transport failure in TRANSPORT with the fetch cause chain in the message', async () => {
|
||||
// Port 1 is reserved/unbound: fetch rejects with `TypeError: fetch failed`
|
||||
// whose actionable detail (ECONNREFUSED) lives on `cause`.
|
||||
const ctx = await harness('http://127.0.0.1:1')
|
||||
let caught: unknown
|
||||
try {
|
||||
await assemble(ctx, { model: 'deepseek-v4-flash', messages: [] })
|
||||
} catch (error: unknown) {
|
||||
caught = error
|
||||
}
|
||||
expect(caught).toBeInstanceOf(LlmError)
|
||||
const llmError = caught as LlmError
|
||||
expect(llmError.code).toBe('TRANSPORT')
|
||||
expect(llmError.message).toContain('http://127.0.0.1:1')
|
||||
expect(llmError.cause).toBeInstanceOf(TypeError)
|
||||
// The chain renderer reaches the transport diagnosis through the cause.
|
||||
expect(errorChain(llmError)).toMatch(/ECONNREFUSED|EADDRNOTAVAIL|bad port/)
|
||||
})
|
||||
|
||||
it('classifies an aborted request without losing the transport rejection', async () => {
|
||||
const controller = new AbortController()
|
||||
controller.abort()
|
||||
const ctx = await harness('http://127.0.0.1:1')
|
||||
let caught: unknown
|
||||
try {
|
||||
await assemble(ctx, { model: 'deepseek-v4-flash', messages: [], signal: controller.signal })
|
||||
} catch (error: unknown) {
|
||||
caught = error
|
||||
}
|
||||
expect(caught).toBeInstanceOf(LlmError)
|
||||
expect(caught).toMatchObject({ code: 'ABORTED' })
|
||||
expect((caught as LlmError).cause).toMatchObject({ name: 'AbortError' })
|
||||
})
|
||||
|
||||
it('throws EMPTY_RESPONSE when the response has no body', async () => {
|
||||
const adapter = new DeepSeekAdapter({ apiKey: 'k', baseURL: 'http://127.0.0.1:1' })
|
||||
const fetchSpy = vi.spyOn(globalThis, 'fetch').mockResolvedValue(
|
||||
@@ -198,7 +365,7 @@ describe('DeepSeekAdapter against a mock server', () => {
|
||||
)
|
||||
try {
|
||||
const iterate = async (): Promise<void> => {
|
||||
for await (const _chunk of adapter.stream({ model: 'm', messages: [] })) { /* drain */ }
|
||||
for await (const _chunk of adapter.stream({ provider: 'deepseek', model: 'm', messages: [] })) { /* drain */ }
|
||||
}
|
||||
await expect(iterate()).rejects.toThrow(/no response body/)
|
||||
} finally {
|
||||
@@ -206,14 +373,20 @@ describe('DeepSeekAdapter against a mock server', () => {
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects with STREAM_CLOSED when the server drops mid-stream', async () => {
|
||||
it('classifies an abrupt body close as TRANSPORT and retains its cause', async () => {
|
||||
const server = await mockServer([{
|
||||
kind: 'close-early',
|
||||
events: ['{"choices":[{"delta":{"content":"par"}}]}'],
|
||||
}])
|
||||
const ctx = await harness(server.url)
|
||||
await expect(assemble(ctx,{ model: 'deepseek-v4-flash', messages: [] }))
|
||||
.rejects.toThrow(/terminated|socket|without \[DONE\]/)
|
||||
let caught: unknown
|
||||
try {
|
||||
await assemble(ctx,{ model: 'deepseek-v4-flash', messages: [] })
|
||||
} catch (error: unknown) {
|
||||
caught = error
|
||||
}
|
||||
expect(caught).toMatchObject({ code: 'TRANSPORT' })
|
||||
expect(errorChain(caught)).toMatch(/terminated|socket|without \[DONE\]/)
|
||||
})
|
||||
|
||||
it('aborts mid-stream via the request signal', async () => {
|
||||
@@ -224,6 +397,7 @@ describe('DeepSeekAdapter against a mock server', () => {
|
||||
const pending = (async () => {
|
||||
const chunks = []
|
||||
for await (const chunk of ctx.llm.stream({
|
||||
provider: 'deepseek',
|
||||
model: 'deepseek-v4-flash',
|
||||
messages: [],
|
||||
signal: controller.signal,
|
||||
@@ -234,30 +408,168 @@ describe('DeepSeekAdapter against a mock server', () => {
|
||||
})()
|
||||
|
||||
setTimeout(() => { controller.abort() }, 30)
|
||||
await expect(pending).rejects.toThrow()
|
||||
await expect(pending).rejects.toMatchObject({ code: 'ABORTED' })
|
||||
})
|
||||
|
||||
it('maps connection failures to TRANSPORT without losing the cause', async () => {
|
||||
const cause = new TypeError('connection refused')
|
||||
const fetchSpy = vi.spyOn(globalThis, 'fetch').mockRejectedValue(cause)
|
||||
const adapter = new DeepSeekAdapter({ apiKey: 'k', baseURL: 'https://example.invalid' })
|
||||
try {
|
||||
const drain = async (): Promise<void> => {
|
||||
for await (const _chunk of adapter.stream({ provider: 'deepseek', model: 'm', messages: [] })) { /* drain */ }
|
||||
}
|
||||
await expect(drain()).rejects.toMatchObject({ code: 'TRANSPORT', cause })
|
||||
} finally {
|
||||
fetchSpy.mockRestore()
|
||||
}
|
||||
})
|
||||
|
||||
it('renders a non-Error transport rejection without losing its cause', async () => {
|
||||
const fetchSpy = vi.spyOn(globalThis, 'fetch').mockImplementation(() => {
|
||||
const failed = Promise.withResolvers<Response>()
|
||||
failed.reject('offline')
|
||||
return failed.promise
|
||||
})
|
||||
const adapter = new DeepSeekAdapter({ apiKey: 'k', baseURL: 'https://example.invalid' })
|
||||
try {
|
||||
const drain = async (): Promise<void> => {
|
||||
for await (const _chunk of adapter.stream({ provider: 'deepseek', model: 'm', messages: [] })) { /* drain */ }
|
||||
}
|
||||
await expect(drain()).rejects.toMatchObject({
|
||||
message: 'DeepSeek API request to https://example.invalid failed',
|
||||
code: 'TRANSPORT',
|
||||
cause: 'offline',
|
||||
})
|
||||
} finally {
|
||||
fetchSpy.mockRestore()
|
||||
}
|
||||
})
|
||||
|
||||
it('aborts the underlying body when the stream stays idle past its watchdog', async () => {
|
||||
vi.useFakeTimers()
|
||||
let stopped = false
|
||||
const fetchSpy = vi.spyOn(globalThis, 'fetch').mockImplementation((_input, init) => {
|
||||
const signal = init?.signal
|
||||
const body = new ReadableStream<Uint8Array>({
|
||||
start(controller) {
|
||||
signal?.addEventListener('abort', () => {
|
||||
stopped = true
|
||||
controller.error(signal.reason)
|
||||
}, { once: true })
|
||||
},
|
||||
})
|
||||
return Promise.resolve(new Response(body, { status: 200 }))
|
||||
})
|
||||
const adapter = new DeepSeekAdapter({
|
||||
apiKey: 'k',
|
||||
baseURL: 'https://example.invalid',
|
||||
streamIdleTimeoutMs: 100,
|
||||
})
|
||||
try {
|
||||
const drain = (async () => {
|
||||
for await (const _chunk of adapter.stream({ provider: 'deepseek', model: 'm', messages: [] })) { /* drain */ }
|
||||
})()
|
||||
const rejected = expect(drain).rejects.toMatchObject({ code: 'TIMEOUT' })
|
||||
await vi.advanceTimersByTimeAsync(0)
|
||||
await vi.advanceTimersByTimeAsync(100)
|
||||
await rejected
|
||||
expect(stopped).toBe(true)
|
||||
} finally {
|
||||
fetchSpy.mockRestore()
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('plugin registration and config', () => {
|
||||
it('registers the configured models and unregisters on dispose (HMR safety)', async () => {
|
||||
it('keeps wire helpers off the package root', () => {
|
||||
for (const helper of [
|
||||
'httpErrorCode',
|
||||
'serializeMessages',
|
||||
'serializeRequest',
|
||||
'DONE',
|
||||
'parseSse',
|
||||
'mapFinishReason',
|
||||
'mapUsage',
|
||||
'translate',
|
||||
]) expect(LlmDeepSeek).not.toHaveProperty(helper)
|
||||
})
|
||||
|
||||
it('registers the deepseek provider and unregisters on dispose (HMR safety)', async () => {
|
||||
const server = await mockServer([])
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
const fiber = await ctx.plugin(LlmDeepSeek, {
|
||||
apiKey: 'k',
|
||||
baseURL: server.url,
|
||||
models: ['deepseek-v4-flash', 'deepseek-v4-pro'],
|
||||
})
|
||||
expect(ctx.llm.models().sort()).toEqual(['deepseek-v4-flash', 'deepseek-v4-pro'])
|
||||
expect(ctx.llm.listProviders()).toEqual([{ id: 'deepseek', name: 'DeepSeek' }])
|
||||
await fiber.dispose()
|
||||
expect(ctx.llm.models()).toEqual([])
|
||||
expect(ctx.llm.listProviders()).toEqual([])
|
||||
})
|
||||
|
||||
it('defaults the model list', async () => {
|
||||
it('owns the deepseek provider and advertises the default models', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmDeepSeek, { apiKey: 'k', baseURL: 'http://127.0.0.1:1' })
|
||||
expect(ctx.llm.models().sort()).toEqual(['deepseek-v4-flash', 'deepseek-v4-pro'])
|
||||
expect(ctx.llm.listProviders()).toEqual([{ id: 'deepseek', name: 'DeepSeek' }])
|
||||
await expect(ctx.llm.listModels('deepseek')).resolves.toEqual([
|
||||
{ provider: 'deepseek', id: 'deepseek-v4-flash', name: 'deepseek-v4-flash' },
|
||||
{ provider: 'deepseek', id: 'deepseek-v4-pro', name: 'deepseek-v4-pro' },
|
||||
])
|
||||
})
|
||||
|
||||
it('uses the default model catalog when apply is called directly', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
LlmDeepSeek.apply(ctx, { apiKey: 'k', baseURL: 'http://127.0.0.1:1' })
|
||||
await expect(ctx.llm.listModels('deepseek')).resolves.toEqual([
|
||||
{ provider: 'deepseek', id: 'deepseek-v4-flash', name: 'deepseek-v4-flash' },
|
||||
{ provider: 'deepseek', id: 'deepseek-v4-pro', name: 'deepseek-v4-pro' },
|
||||
])
|
||||
})
|
||||
|
||||
it('advertises configured models without restricting arbitrary request ids', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmDeepSeek, {
|
||||
apiKey: 'k',
|
||||
baseURL: 'http://127.0.0.1:1',
|
||||
models: [
|
||||
{ id: 'private-fast' },
|
||||
{ id: 'private-reasoner', name: 'Private Reasoner', description: 'Higher reasoning budget' },
|
||||
],
|
||||
})
|
||||
await expect(ctx.llm.listModels('deepseek')).resolves.toEqual([
|
||||
{ provider: 'deepseek', id: 'private-fast', name: 'private-fast' },
|
||||
{ provider: 'deepseek', id: 'private-reasoner', name: 'Private Reasoner', description: 'Higher reasoning budget' },
|
||||
])
|
||||
})
|
||||
|
||||
it('allows an explicit empty model catalog', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmDeepSeek, {
|
||||
apiKey: 'k',
|
||||
baseURL: 'http://127.0.0.1:1',
|
||||
models: [],
|
||||
})
|
||||
await expect(ctx.llm.listModels('deepseek')).resolves.toEqual([])
|
||||
})
|
||||
|
||||
it.each([
|
||||
[[{ id: '' }], /ids must be non-empty/],
|
||||
[[{ id: 'm', name: '' }], /empty name/],
|
||||
[[{ id: 'm' }, { id: 'm' }], /duplicate catalog model/],
|
||||
] as const)('rejects invalid advisory model config', async (models, message) => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await expect(ctx.plugin(LlmDeepSeek, {
|
||||
apiKey: 'k',
|
||||
baseURL: 'http://127.0.0.1:1',
|
||||
models: [...models],
|
||||
})).rejects.toThrow(message)
|
||||
expect(ctx.llm.listProviders()).toEqual([])
|
||||
})
|
||||
|
||||
it('falls back to DEEPSEEK_API_KEY and DEEPSEEK_BASE_URL env vars', async () => {
|
||||
@@ -266,7 +578,7 @@ describe('plugin registration and config', () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmDeepSeek, {})
|
||||
expect(ctx.llm.models().length).toBeGreaterThan(0)
|
||||
expect(ctx.llm.listProviders()).toEqual([{ id: 'deepseek', name: 'DeepSeek' }])
|
||||
})
|
||||
|
||||
it('throws a clear error when no API key is available', async () => {
|
||||
@@ -275,7 +587,7 @@ describe('plugin registration and config', () => {
|
||||
await ctx.plugin(LlmService)
|
||||
await expect(ctx.plugin(LlmDeepSeek, {}))
|
||||
.rejects.toThrow(/an API key is required/)
|
||||
expect(ctx.llm.models()).toEqual([])
|
||||
expect(ctx.llm.listProviders()).toEqual([])
|
||||
})
|
||||
|
||||
it('prefers explicit config over env for key and base URL', async () => {
|
||||
@@ -292,7 +604,7 @@ describe('plugin registration and config', () => {
|
||||
vi.stubEnv('DEEPSEEK_BASE_URL', server.url)
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmDeepSeek, { apiKey: 'k', models: ['deepseek-v4-flash'] })
|
||||
await ctx.plugin(LlmDeepSeek, { apiKey: 'k' })
|
||||
await assemble(ctx,{ model: 'deepseek-v4-flash', messages: [] })
|
||||
expect(server.requests).toHaveLength(1)
|
||||
})
|
||||
@@ -304,11 +616,38 @@ describe('plugin registration and config', () => {
|
||||
await ctx.plugin(LlmService)
|
||||
// Registration succeeds; no call is made (would hit api.deepseek.com).
|
||||
await ctx.plugin(LlmDeepSeek, {})
|
||||
expect(ctx.llm.models().length).toBeGreaterThan(0)
|
||||
expect(ctx.llm.listProviders()).toEqual([{ id: 'deepseek', name: 'DeepSeek' }])
|
||||
})
|
||||
|
||||
it('adapter is constructible directly for embedding', () => {
|
||||
it('adapter is constructible directly for embedding', async () => {
|
||||
const adapter = new DeepSeekAdapter({ apiKey: 'k', baseURL: 'http://127.0.0.1:1' })
|
||||
expect(adapter).toBeInstanceOf(DeepSeekAdapter)
|
||||
await expect(adapter.listModels('deepseek')).resolves.toEqual([])
|
||||
})
|
||||
|
||||
it('rejects invalid idle watchdog bounds for direct and plugin composition', async () => {
|
||||
expect(() => new DeepSeekAdapter({
|
||||
apiKey: 'k',
|
||||
baseURL: 'http://127.0.0.1:1',
|
||||
streamIdleTimeoutMs: Number.POSITIVE_INFINITY,
|
||||
})).toThrow(/streamIdleTimeoutMs.*positive finite/)
|
||||
expect(() => new DeepSeekAdapter({
|
||||
apiKey: 'k',
|
||||
baseURL: 'http://127.0.0.1:1',
|
||||
streamIdleTimeoutMs: MAX_TIMER_DELAY_MS + 1,
|
||||
})).toThrow(/streamIdleTimeoutMs.*no greater/)
|
||||
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await expect(ctx.plugin(LlmDeepSeek, {
|
||||
apiKey: 'k',
|
||||
baseURL: 'http://127.0.0.1:1',
|
||||
streamIdleTimeoutMs: 0,
|
||||
})).rejects.toThrow(/streamIdleTimeoutMs/)
|
||||
await expect(ctx.plugin(LlmDeepSeek, {
|
||||
apiKey: 'k',
|
||||
baseURL: 'http://127.0.0.1:1',
|
||||
streamIdleTimeoutMs: MAX_TIMER_DELAY_MS + 1,
|
||||
})).rejects.toThrow(/streamIdleTimeoutMs/)
|
||||
})
|
||||
})
|
||||
|
||||
@@ -15,11 +15,19 @@ export interface AssembledResult {
|
||||
finish: FinishReason
|
||||
}
|
||||
|
||||
export async function assemble(ctx: Context, options: GenerateOptions): Promise<AssembledResult> {
|
||||
export async function assemble(ctx: Context, options: Omit<GenerateOptions, 'provider'> & { provider?: string }): Promise<AssembledResult> {
|
||||
const assembler = new BlockAssembler()
|
||||
for await (const chunk of ctx.llm.stream(options)) assembler.push(chunk)
|
||||
const request = { provider: 'deepseek', ...options }
|
||||
for await (const chunk of ctx.llm.stream(request)) assembler.push(chunk)
|
||||
return {
|
||||
message: assembler.message(),
|
||||
message: {
|
||||
...assembler.message(),
|
||||
provenance: {
|
||||
provider: request.provider,
|
||||
model: request.model,
|
||||
...assembler.replayState === undefined ? {} : { replayState: assembler.replayState },
|
||||
},
|
||||
},
|
||||
...assembler.usage !== undefined ? { usage: assembler.usage } : {},
|
||||
finish: assembler.finish,
|
||||
}
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { CallId } from '@deepseek-ai/dsh-llm'
|
||||
import type { ContentBlock, GenerateOptions, Message } from '@deepseek-ai/dsh-llm'
|
||||
import { serializeMessages, serializeRequest } from '@deepseek-ai/dsh-llm-deepseek'
|
||||
import { serializeMessages, serializeRequest } from '../src/serialize.ts'
|
||||
|
||||
function request(overrides: Partial<GenerateOptions> = {}): GenerateOptions {
|
||||
return { model: 'deepseek-v4-flash', messages: [], ...overrides }
|
||||
return { provider: 'deepseek', model: 'deepseek-v4-flash', messages: [], ...overrides }
|
||||
}
|
||||
|
||||
describe('serializeMessages', () => {
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import { DONE, parseSse } from '@deepseek-ai/dsh-llm-deepseek'
|
||||
import { DONE, parseSse } from '../src/sse.ts'
|
||||
|
||||
/** Build a byte stream from string fragments (fragments = network reads). */
|
||||
async function* bytes(...fragments: (string | Uint8Array)[]): AsyncGenerator<Uint8Array> {
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { BlockAssembler, LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import type { StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import { DONE, mapFinishReason, mapUsage, translate } from '@deepseek-ai/dsh-llm-deepseek'
|
||||
import { DONE } from '../src/sse.ts'
|
||||
import { mapFinishReason, mapUsage, translate } from '../src/translate.ts'
|
||||
|
||||
async function* feed(...payloads: (string | object)[]): AsyncGenerator<string> {
|
||||
for (const payload of payloads) {
|
||||
@@ -231,8 +232,7 @@ describe('mapFinishReason', () => {
|
||||
(wire) => {
|
||||
expect(mapFinishReason(wire)).toEqual({
|
||||
kind: 'error',
|
||||
message: `model stopped: ${wire}`,
|
||||
code: wire.toUpperCase(),
|
||||
failure: { message: `model stopped: ${wire}`, code: wire.toUpperCase() },
|
||||
})
|
||||
},
|
||||
)
|
||||
|
||||
@@ -19,6 +19,9 @@
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm"
|
||||
},
|
||||
{
|
||||
"path": "../../util/timeout"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
@@ -1,60 +1,100 @@
|
||||
# @deepseek-ai/dsh-llm-pi-ai
|
||||
|
||||
DeepSeek adapter for the harness LLM seam backed by [`@earendil-works/pi-ai`](https://www.npmjs.com/package/@earendil-works/pi-ai) (the LLM library behind the pi agent).
|
||||
Generic multi-provider adapter for the harness LLM seam backed by [`@earendil-works/pi-ai`](https://www.npmjs.com/package/@earendil-works/pi-ai). One plugin instance owns an explicit list of provider profiles; every request selects a profile with `GenerateOptions.provider` and resolves `GenerateOptions.model` dynamically from pi-ai's installed catalog.
|
||||
|
||||
## Why a second adapter exists
|
||||
|
||||
`@deepseek-ai/dsh-llm-deepseek` already talks to the same endpoint. This package is its **design-verification twin**: same models, same wire protocol, completely different internals — a unified LLM library with its own event vocabulary versus hand-rolled fetch/SSE. Anything the harness `StreamChunk` protocol cannot express for BOTH implementations is a core-vocabulary bug. The differences it exercised on purpose:
|
||||
|
||||
- pi-ai hands tool-call `arguments` around as **parsed objects**; the harness keeps raw JSON strings. The adapter patches replay payloads back to the original raw strings before sending them, and re-stringifies parsed output tool calls at `block-end`.
|
||||
- pi-ai reports failures as **in-stream error events** (it never throws mid-stream); these map to `finish {kind:'error'|'aborted'}` chunks — the protocol's other sanctioned error path besides throwing (which llm-deepseek uses).
|
||||
- pi-ai folds reasoning tokens into `usage.output`; there is no separate reasoning count to map.
|
||||
- pi-ai's options omit some DeepSeek/OpenAI-compatible details; the adapter uses its `onPayload` hook to preserve the harness contract (`stop`, scrubbing pi-ai's own per-tool `strict` default — the hand-rolled twin sends no such field — omitted reasoning effort, raw replayed tool arguments).
|
||||
The package root exposes the Cordis plugin contract and `PiAiAdapter`; profile resolution, model construction, replay conversion, and stream conversion remain package-internal.
|
||||
|
||||
## Config
|
||||
|
||||
Same shape as llm-deepseek (one-line swap in cordis.yml), with pi-ai's thinking-level vocabulary:
|
||||
Configure credentials and deployment-specific transport settings per provider. Omitting `apiKey` delegates authentication to pi-ai's provider-native ambient discovery. `baseURL` overrides only the endpoint of the selected catalog model, preserving its API family and compatibility metadata, so private proxies such as `https://proxy.example.com:8443` remain supported.
|
||||
|
||||
```yaml
|
||||
- id: llm
|
||||
name: '@deepseek-ai/dsh-llm-pi-ai'
|
||||
config:
|
||||
apiKey: !!js process.env.DEEPSEEK_API_KEY
|
||||
baseURL: !!js process.env.DEEPSEEK_BASE_URL
|
||||
models: [deepseek-v4-flash, deepseek-v4-pro]
|
||||
reasoning: high # off | high | xhigh (xhigh → wire 'max')
|
||||
providers:
|
||||
- provider: openai
|
||||
apiKey: !!js process.env.OPENAI_API_KEY
|
||||
baseURL: https://proxy.example.com:8443
|
||||
reasoning: high
|
||||
- provider: anthropic
|
||||
apiKey: !!js process.env.ANTHROPIC_API_KEY
|
||||
streamIdleTimeoutMs: 300000
|
||||
- provider: openrouter
|
||||
apiKey: !!js process.env.OPENROUTER_API_KEY
|
||||
headers:
|
||||
X-Deployment: production
|
||||
```
|
||||
|
||||
Each provider name must exist in pi-ai's installed catalog and may appear only once in this plugin instance. Registration with `ctx.llm` is atomic: a collision with any provider route already owned by another adapter fails plugin loading without registering the remaining routes. Model ids are not lifecycle config; an unknown model fails before any provider request with `LlmError('UNKNOWN_MODEL')`.
|
||||
|
||||
The adapter exposes each configured provider's installed pi-ai models through `ctx.llm.listModels(provider)`. This is provider-neutral selector metadata derived from `getModels(provider)`; request-time resolution still performs the authoritative catalog lookup, so discovery does not create a second model registry.
|
||||
|
||||
Supported profile fields are `provider`, `apiKey`, `baseURL`, `headers`, `reasoning`, `thinkingBudgets`, `cacheRetention`, `transport`, `timeoutMs`, `websocketConnectTimeoutMs`, and `streamIdleTimeoutMs`. The stream-idle interval is a positive finite Node timer delay, defaults to five minutes, and covers only an outstanding provider read, not consumer think time. Harness app attribution wins a conflicting configured header name.
|
||||
|
||||
The adapter forces pi-ai's SDK `maxRetries` to zero so one `stream()` call makes one provider request. The removed profile fields `maxRetries` and `maxRetryDelayMs` fail load instead of silently multiplying or hiding the separately composed agent-level retry budget. Idle expiry aborts the SDK's stable request signal and surfaces `TIMEOUT`; an earlier caller abort remains `ABORTED`.
|
||||
|
||||
## Provider/model routing and replay
|
||||
|
||||
The selected pi-ai catalog descriptor supplies the protocol implementation. This includes native API differences such as OpenAI models whose descriptor uses the Responses API rather than Chat Completions; the harness adapter does not hardcode endpoint selection by model name.
|
||||
|
||||
Successful assistant responses store a versioned, lossless-JSON replay state beside their durable provider/model provenance. At request time, `LlmService` passes replay state only when the historical provider route and target provider route are currently owned by this same `PiAiAdapter` instance. The adapter validates the state and restores pi-ai response ids and provider signatures even when the target provider or model changes; pi-ai then decides which metadata its target API can reuse. History without replay state is translated as foreign provider-neutral content and never impersonates a native pi-ai response.
|
||||
|
||||
If a listener rewrites assembled assistant content, the loop drops replay state before logging the message because its provider metadata no longer describes the content. Invalid versions, malformed metadata, provenance provider/model mismatches, and content/block mismatches fail explicitly with `LlmError('INVALID_REPLAY_STATE')`.
|
||||
|
||||
## Vocabulary differences
|
||||
|
||||
- pi-ai tool-call arguments are parsed objects; the harness stores raw JSON strings. The adapter parses input and re-stringifies output.
|
||||
- pi-ai reports failures as in-stream error events; these map to `finish {kind:'error'|'aborted', failure}` chunks. Provider-specific error text distinguishes terminal `QUOTA` from transient `RATE_LIMIT`, while text and usage signals evaluated against the resolved model's context window normalize overflow to `CONTEXT_WINDOW_EXCEEDED`.
|
||||
- pi-ai folds reasoning tokens into output usage; there is no separate reasoning count to map.
|
||||
- `GenerateOptions.stop` is rejected with `UNSUPPORTED_OPTION` because pi-ai's common streaming surface cannot guarantee it across providers.
|
||||
|
||||
## App attribution
|
||||
|
||||
Every request carries the shared attribution header from dsh-llm's `attributionHeaders()`, passed through pi-ai's `headers` stream option (pi-ai merges caller headers last, so it always reaches the wire - the unit suite asserts arrival on the mock server, same as llm-deepseek). OpenRouter-specific app attribution headers are intentionally not sent by this adapter contract; they are deferred to a future explicit OpenRouter adapter or mode. See [dsh-llm § App attribution](../llm/README.md#app-attribution-attributionts).
|
||||
Every request carries the shared attribution header from dsh-llm's `attributionHeaders()`, merged through pi-ai's `headers` stream option. Provider-specific app-attribution headers are not synthesized. See [dsh-llm § App attribution](../llm/README.md#app-attribution-attributionts).
|
||||
|
||||
## Dependency weight
|
||||
|
||||
pi-ai declares the openai/anthropic/google/mistral/AWS SDKs as install-time dependencies. They are lazy-loaded — only the openai SDK actually loads for this adapter — but they do land in `node_modules`. Accepted for a package whose purpose is design verification.
|
||||
pi-ai installs several provider SDKs and lazy-loads the one selected by the catalog model. The dependency weight is isolated to this opt-in adapter package.
|
||||
|
||||
## Testing
|
||||
|
||||
Unit suites run against a local `node:http` mock SSE server (pi-ai's openai SDK happily talks to any base URL). Real-API coverage in `tests/adapter.e2e.ts` (`pnpm run test:e2e`, key-gated): V4 Flash + V4 Pro across all exposed reasoning levels (off/high/xhigh), the thinking+tools round trip, and a cross-adapter structural-equivalence check against llm-deepseek.
|
||||
Unit tests use pi-ai catalog models redirected to local mock servers and cover provider/profile routing, one wire request per adapter call, idle-timeout response termination, caller abort, native API selection, endpoint overrides, attribution, conversion, replay-state validation, and cross-provider/model replay within one adapter instance. Real-API coverage remains key-gated under `pnpm run test:e2e`.
|
||||
|
||||
## Model Experience
|
||||
|
||||
### DeepSeek request through pi-ai
|
||||
### Provider request through pi-ai
|
||||
|
||||
**What the model sees**: The selected model receives the same logical system prompt, history, tools, stop sequences, and raw replayed tool arguments as the hand-written adapter. This package adds no prompt prose and removes pi-ai's own per-tool `strict` default to preserve that contract.
|
||||
#### What the model sees
|
||||
|
||||
**Token effect**: Provider tokenization governs exact input. Reasoning level changes generated and passback content; pi-ai reports reasoning inside output usage rather than as a separate count.
|
||||
The selected catalog model receives `GenerateOptions.system`, history, tools, and sampling fields supported by pi-ai's common streaming API. This package adds no prompt prose. Provider-native replay metadata is restored only when the adapter validates it for the historical content.
|
||||
|
||||
### DeepSeek response
|
||||
#### Token effect
|
||||
|
||||
**What the model sees**: pi-ai events become harness reasoning, text, tool-call, usage, and finish chunks; parsed tool arguments are restored to raw JSON strings at the harness boundary.
|
||||
Provider tokenization governs exact input. Conversion adds no model-visible text; replay metadata may let a native API reuse provider-side state.
|
||||
|
||||
**Token effect**: Generated content affects later inputs only after the loop records it; adapter conversion adds no model-visible text.
|
||||
#### KV Cache effect
|
||||
|
||||
Conversion preserves logical request order without adding text, while the selected provider's serialization and replay state determine reuse. Changing adapter instance, provider, model, or any upstream request token may prevent reuse from the first difference.
|
||||
|
||||
### Provider response
|
||||
|
||||
#### What the model sees
|
||||
|
||||
pi-ai events become harness reasoning, text, tool-call, usage, and finish chunks. Parsed tool arguments cross the harness boundary as raw JSON strings.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Generated content affects later inputs only after the loop records it. pi-ai folds reasoning tokens into output usage when the provider does not report them separately.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
Recorded response content appends to the next request and does not invalidate its earlier reusable prefix. Unrecorded transport metadata and usage accounting do not affect cache identity.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **`tool_choice` is not mapped** — same MVP contract as llm-deepseek.
|
||||
- **In-history `system`-role messages fold into `user`-role wire messages** — pi-ai exposes a single `systemPrompt` slot, diverging from the hand-rolled twin's `role: 'system'` passthrough.
|
||||
- **`LlmError.status` is never set** — pi-ai reports failures as in-stream events with no HTTP status, so error codes are regex-classified from the error text.
|
||||
- **`buildModel` hardcodes descriptor metadata** — `contextWindow: 128000`, `maxTokens: 64000`, zero cost, identically for every registered model name; not configurable.
|
||||
- **pi-ai's built-in retries are disabled (`maxRetries: 0`)** — failures surface immediately; retry policy belongs to `llm/stream` listeners.
|
||||
- **Catalog membership is required** — custom model ids that are absent from the installed pi-ai catalog fail with `UNKNOWN_MODEL`, even when a provider profile supplies a custom endpoint.
|
||||
- **`GenerateOptions.stop` is unsupported** — pi-ai's common stream options cannot guarantee stop-sequence behavior across providers, so the adapter rejects the field.
|
||||
- **In-history `system` messages use pi-ai's common context conversion** — provider-specific placement follows pi-ai rather than a harness-owned wire override.
|
||||
- **Provider HTTP status is unavailable** — pi-ai error events do not expose a stable HTTP status across providers; failures expose only stable harness error codes.
|
||||
- **Retry policy is not an adapter option** — SDK retries are disabled so durable agent steps and `llm/retry` events own every visible attempt; direct `ctx.llm.stream()` calls remain single-attempt.
|
||||
|
||||
@@ -23,6 +23,7 @@
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-llm": "^0.0.1",
|
||||
"@deepseek-ai/dsh-timeout": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"dependencies": {
|
||||
@@ -32,6 +33,7 @@
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm-deepseek": "workspace:^",
|
||||
"@deepseek-ai/dsh-timeout": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,173 +1,155 @@
|
||||
/**
|
||||
* Pi-ai-backed DeepSeek adapter and design twin of the hand-rolled adapter.
|
||||
* Both implementations must fit the same provider-neutral stream vocabulary.
|
||||
* Generic pi-ai-backed implementation of the Harness LLM seam.
|
||||
*
|
||||
* @module dsh-llm-pi-ai/adapter
|
||||
*/
|
||||
|
||||
import { stream as piStream } from '@earendil-works/pi-ai'
|
||||
import type { Model } from '@earendil-works/pi-ai'
|
||||
import { attributionHeaders, LlmAdapter } from '@deepseek-ai/dsh-llm'
|
||||
import { CallId } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import { toPiContext, toStreamChunks } from './convert.ts'
|
||||
import {
|
||||
getModels,
|
||||
streamSimple,
|
||||
} from '@earendil-works/pi-ai'
|
||||
import type {
|
||||
Api,
|
||||
KnownProvider,
|
||||
Model,
|
||||
SimpleStreamOptions,
|
||||
} from '@earendil-works/pi-ai'
|
||||
import { attributionHeaders, LlmAdapter, LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, LlmModelInfo, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import { idleWatchdog, timeoutOf } from '@deepseek-ai/dsh-timeout'
|
||||
import { resolveProfiles } from './config.ts'
|
||||
import type { PiAiProviderProfile, ResolvedPiAiProviderProfile } from './config.ts'
|
||||
import { toPiContext } from './context.ts'
|
||||
import { toStreamChunks } from './stream.ts'
|
||||
|
||||
/** Reasoning levels surfaced by this adapter (DeepSeek wire: high|max). */
|
||||
export type PiAiReasoning = 'off' | 'high' | 'xhigh'
|
||||
|
||||
/** Constructor options for {@link PiAiAdapter}; the plugin's `apply` resolves them from Config + environment. */
|
||||
/** Constructor options for {@link PiAiAdapter}. */
|
||||
export interface PiAiAdapterOptions {
|
||||
/** Bearer token pi-ai sends on every request. */
|
||||
apiKey: string
|
||||
/** Endpoint base; `/chat/completions` is appended. */
|
||||
baseURL: string
|
||||
/** Thinking level applied to every request ('off' disables thinking). */
|
||||
reasoning?: PiAiReasoning | undefined
|
||||
/** Validated provider profiles this adapter instance owns. */
|
||||
profiles: readonly PiAiProviderProfile[]
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the inline pi-ai model descriptor for one DeepSeek model name.
|
||||
* @param modelId - harness model name; sent verbatim on the wire.
|
||||
* @param options - adapter options; only `baseURL` is read here (key and reasoning apply per request, not per descriptor).
|
||||
* @returns a descriptor with every DeepSeek compat flag explicit — pi-ai's URL-based auto-detection is never relied on.
|
||||
* Resolve a catalog model dynamically and apply only the configured endpoint
|
||||
* override, preserving the catalog's API/capability/compatibility metadata.
|
||||
*/
|
||||
export function buildModel(modelId: string, options: PiAiAdapterOptions): Model<'openai-completions'> {
|
||||
function resolveModel(profile: PiAiProviderProfile, modelId: string): Model<Api> {
|
||||
const model = getModels(profile.provider as KnownProvider).find(candidate => candidate.id === modelId) as Model<Api> | undefined
|
||||
if (model === undefined) {
|
||||
throw new LlmError(`pi-ai provider "${profile.provider}" has no catalog model "${modelId}"`, 'UNKNOWN_MODEL')
|
||||
}
|
||||
return profile.baseURL === undefined ? model : { ...model, baseUrl: profile.baseURL }
|
||||
}
|
||||
|
||||
/** Copy profile stream knobs into pi-ai's common option vocabulary. */
|
||||
function profileOptions(profile: PiAiProviderProfile): SimpleStreamOptions {
|
||||
return {
|
||||
id: modelId,
|
||||
name: modelId,
|
||||
api: 'openai-completions',
|
||||
provider: 'deepseek',
|
||||
baseUrl: options.baseURL,
|
||||
// Keep reasoning support enabled so `off` can send DeepSeek's explicit
|
||||
// disabled marker rather than falling back to the provider's enabled default.
|
||||
reasoning: true,
|
||||
// DeepSeek's official effort levels: high|max (xhigh maps to max).
|
||||
thinkingLevelMap: { minimal: null, low: null, medium: null, high: 'high', xhigh: 'max' },
|
||||
input: ['text'],
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
|
||||
contextWindow: 128_000,
|
||||
maxTokens: 64_000,
|
||||
compat: {
|
||||
// Auto-detection only fires for *.deepseek.com base URLs; the internal
|
||||
// endpoint (and test mocks) need these set explicitly.
|
||||
thinkingFormat: 'deepseek',
|
||||
requiresReasoningContentOnAssistantMessages: true,
|
||||
supportsReasoningEffort: true,
|
||||
// DeepSeek documents max_tokens (not OpenAI's max_completion_tokens).
|
||||
maxTokensField: 'max_tokens',
|
||||
},
|
||||
...profile.apiKey === undefined ? {} : { apiKey: profile.apiKey },
|
||||
...profile.reasoning === undefined ? {} : { reasoning: profile.reasoning },
|
||||
...profile.thinkingBudgets === undefined ? {} : { thinkingBudgets: profile.thinkingBudgets },
|
||||
...profile.cacheRetention === undefined ? {} : { cacheRetention: profile.cacheRetention },
|
||||
...profile.transport === undefined ? {} : { transport: profile.transport },
|
||||
...profile.timeoutMs === undefined ? {} : { timeoutMs: profile.timeoutMs },
|
||||
...profile.websocketConnectTimeoutMs === undefined ? {} : { websocketConnectTimeoutMs: profile.websocketConnectTimeoutMs },
|
||||
// The agent recovery layer owns visible attempts; one adapter call is one SDK attempt.
|
||||
maxRetries: 0,
|
||||
}
|
||||
}
|
||||
|
||||
type Payload = {
|
||||
tools?: { function?: { strict?: unknown } }[]
|
||||
messages?: {
|
||||
role?: unknown
|
||||
tool_calls?: { id?: unknown; function?: { arguments?: unknown } }[]
|
||||
}[]
|
||||
reasoning_effort?: unknown
|
||||
stop?: unknown
|
||||
}
|
||||
|
||||
function rawToolArguments(options: GenerateOptions): Map<CallId, string> {
|
||||
const raw = new Map<CallId, string>()
|
||||
for (const message of options.messages) {
|
||||
if (message.role !== 'assistant') continue
|
||||
for (const block of message.content) {
|
||||
if (block.type === 'tool-call') raw.set(block.id, block.arguments)
|
||||
}
|
||||
/** Merge deployment headers while removing case-insensitive attribution collisions. */
|
||||
function requestHeaders(headers: Readonly<Record<string, string>> | undefined): Record<string, string> {
|
||||
const attribution = attributionHeaders()
|
||||
const reserved = new Set(Object.keys(attribution).map(name => name.toLowerCase()))
|
||||
return {
|
||||
...Object.fromEntries(Object.entries(headers ?? {}).filter(([name]) => !reserved.has(name.toLowerCase()))),
|
||||
...attribution,
|
||||
}
|
||||
return raw
|
||||
}
|
||||
|
||||
function patchPayload(payload: unknown, options: GenerateOptions, reasoning: PiAiReasoning | undefined): unknown {
|
||||
/* v8 ignore next -- pi-ai onPayload always receives an object; tolerate unusual future hooks defensively */
|
||||
if (typeof payload !== 'object' || payload === null) return payload
|
||||
const body = payload as Payload
|
||||
|
||||
if (reasoning === undefined) {
|
||||
delete body.reasoning_effort
|
||||
}
|
||||
if (options.stop !== undefined) {
|
||||
body.stop = options.stop
|
||||
}
|
||||
|
||||
// pi-ai stamps its own `strict` default on every serialized tool; the
|
||||
// harness tool contract has no strict field and the hand-rolled twin sends
|
||||
// none, so scrub it for wire parity.
|
||||
for (const tool of body.tools ?? []) {
|
||||
/* v8 ignore next -- malformed pi-ai payload guard: real tool entries always carry function */
|
||||
if (tool.function === undefined) continue
|
||||
delete tool.function.strict
|
||||
}
|
||||
|
||||
const rawById = rawToolArguments(options)
|
||||
/* v8 ignore next -- defensive for non-chat payloads; OpenAI chat payloads always carry messages */
|
||||
for (const message of body.messages ?? []) {
|
||||
if (message.role !== 'assistant') continue
|
||||
/* v8 ignore next -- assistant messages without tool_calls need no raw-argument patch */
|
||||
for (const call of message.tool_calls ?? []) {
|
||||
/* v8 ignore next -- malformed pi-ai payload guard: real tool calls always carry a string id */
|
||||
if (typeof call.id !== 'string') continue
|
||||
const raw = rawById.get(CallId(call.id))
|
||||
/* v8 ignore next -- pi-ai always emits a function object for assistant tool_calls; guard malformed payloads defensively */
|
||||
if (raw !== undefined && call.function !== undefined) call.function.arguments = raw
|
||||
}
|
||||
}
|
||||
|
||||
return body
|
||||
}
|
||||
|
||||
/**
|
||||
* pi-ai-backed adapter. One instance serves every registered model name.
|
||||
*
|
||||
* Implementation notes:
|
||||
* - `onPayload` patches provider payload details pi-ai cannot express directly:
|
||||
* stop sequences, scrubbing pi-ai's own per-tool `strict` default (the
|
||||
* hand-rolled twin sends no such field), omitted reasoning effort, and raw
|
||||
* replayed tool-call arguments.
|
||||
* - pi-ai reports request failures as in-stream error events; convert.ts
|
||||
* maps them to `finish {kind:'error'|'aborted'}` chunks rather than
|
||||
* throwing — both are sanctioned StreamChunk error paths.
|
||||
* pi-ai-backed multi-provider adapter. Model descriptors are resolved for each
|
||||
* request, so models need not be registered during the Cordis lifecycle.
|
||||
*/
|
||||
export class PiAiAdapter extends LlmAdapter {
|
||||
constructor(private readonly options: PiAiAdapterOptions) {
|
||||
private readonly profiles: ReadonlyMap<string, ResolvedPiAiProviderProfile>
|
||||
|
||||
constructor(options: PiAiAdapterOptions) {
|
||||
super()
|
||||
this.profiles = new Map(resolveProfiles(options.profiles).map(profile => [profile.provider, profile]))
|
||||
}
|
||||
|
||||
override listModels(provider: string): Promise<readonly LlmModelInfo[]> {
|
||||
const profile = this.profiles.get(provider)
|
||||
if (profile === undefined) {
|
||||
return Promise.reject(new LlmError(`pi-ai adapter does not own provider "${provider}"`, 'NO_ADAPTER'))
|
||||
}
|
||||
return Promise.resolve(getModels(profile.provider as KnownProvider).map(model => ({
|
||||
provider,
|
||||
id: model.id,
|
||||
name: model.name,
|
||||
})))
|
||||
}
|
||||
|
||||
async * stream(options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
const model = buildModel(options.model, this.options)
|
||||
// Undefined config means "provider default" (DeepSeek: thinking ENABLED),
|
||||
// matching llm-deepseek's omission semantics. pi-ai derives the wire
|
||||
// thinking toggle from whether reasoningEffort is passed, so undefined maps
|
||||
// internally to 'high' to get `thinking: enabled`; patchPayload then removes
|
||||
// `reasoning_effort` so the provider chooses its default effort.
|
||||
const reasoning = this.options.reasoning ?? 'high'
|
||||
if (options.stop !== undefined) {
|
||||
throw new LlmError('llm-pi-ai does not support GenerateOptions.stop', 'UNSUPPORTED_OPTION')
|
||||
}
|
||||
const profile = this.profiles.get(options.provider)
|
||||
if (profile === undefined) {
|
||||
throw new LlmError(`pi-ai adapter does not own provider "${options.provider}"`, 'NO_ADAPTER')
|
||||
}
|
||||
const model = resolveModel(profile, options.model)
|
||||
|
||||
// Pi-ai has no iterator-return cancellation hook. Chain an internal signal
|
||||
// and abort it when this generator exits so early consumers stop the HTTP stream.
|
||||
const controller = new AbortController()
|
||||
const onCallerAbort = (): void => { controller.abort(options.signal?.reason) }
|
||||
if (options.signal?.aborted) controller.abort(options.signal.reason)
|
||||
else options.signal?.addEventListener('abort', onCallerAbort, { once: true })
|
||||
const consumer = new AbortController()
|
||||
const upstream = options.signal === undefined
|
||||
? consumer.signal
|
||||
: AbortSignal.any([options.signal, consumer.signal])
|
||||
const streamIdleTimeoutMs = profile.streamIdleTimeoutMs
|
||||
using watchdog = idleWatchdog(upstream, streamIdleTimeoutMs, 'LLM_STREAM_IDLE_TIMEOUT')
|
||||
|
||||
try {
|
||||
const events = piStream(model, toPiContext(options), {
|
||||
apiKey: this.options.apiKey,
|
||||
// pi-ai merges caller headers last over its provider defaults, so the
|
||||
// harness attribution always reaches the wire.
|
||||
headers: attributionHeaders(),
|
||||
...options.temperature !== undefined ? { temperature: options.temperature } : {},
|
||||
...options.maxTokens !== undefined ? { maxTokens: options.maxTokens } : {},
|
||||
signal: controller.signal,
|
||||
...reasoning !== 'off' ? { reasoningEffort: reasoning } : {},
|
||||
onPayload: payload => patchPayload(payload, options, this.options.reasoning),
|
||||
maxRetries: 0,
|
||||
const events = streamSimple(model, toPiContext(options), {
|
||||
...profileOptions(profile),
|
||||
...options.temperature === undefined ? {} : { temperature: options.temperature },
|
||||
...options.maxTokens === undefined ? {} : { maxTokens: options.maxTokens },
|
||||
...options.sessionId === undefined ? {} : { sessionId: String(options.sessionId) },
|
||||
signal: watchdog.signal,
|
||||
// Profile headers are deployment-owned; attribution names are
|
||||
// Harness-owned and therefore win collisions.
|
||||
headers: requestHeaders(profile.headers),
|
||||
})
|
||||
|
||||
yield* toStreamChunks(events)
|
||||
const iterator = toStreamChunks(events, model.contextWindow)[Symbol.asyncIterator]()
|
||||
let exhausted = false
|
||||
try {
|
||||
while (true) {
|
||||
const result = await watchdog.next(iterator)
|
||||
const timeout = timeoutOf(watchdog.signal, 'LLM_STREAM_IDLE_TIMEOUT')
|
||||
if (timeout !== undefined) throw timeout
|
||||
if (result.done) {
|
||||
exhausted = true
|
||||
return
|
||||
}
|
||||
yield result.value
|
||||
}
|
||||
} finally {
|
||||
if (!exhausted) {
|
||||
consumer.abort('pi-ai stream consumer stopped')
|
||||
try {
|
||||
await iterator.return(undefined)
|
||||
} catch (_abortedSdkTeardown) {
|
||||
// The stable signal already owns SDK termination; return-time abort cannot add an outcome.
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch (error: unknown) {
|
||||
if (timeoutOf(watchdog.signal, 'LLM_STREAM_IDLE_TIMEOUT') !== undefined) {
|
||||
throw new LlmError(`pi-ai stream idle timeout after ${streamIdleTimeoutMs}ms`, 'TIMEOUT', { cause: error })
|
||||
}
|
||||
if (options.signal?.aborted) {
|
||||
throw new LlmError('pi-ai request aborted by caller', 'ABORTED', { cause: error })
|
||||
}
|
||||
throw error
|
||||
} finally {
|
||||
options.signal?.removeEventListener('abort', onCallerAbort)
|
||||
controller.abort('consumer stopped streaming')
|
||||
consumer.abort('pi-ai stream consumer stopped')
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
122
packages/llm/llm-pi-ai/src/config.ts
Normal file
122
packages/llm/llm-pi-ai/src/config.ts
Normal file
@@ -0,0 +1,122 @@
|
||||
/**
|
||||
* Configuration schema and provider-profile validation for the pi-ai adapter.
|
||||
*
|
||||
* @module dsh-llm-pi-ai/config
|
||||
*/
|
||||
|
||||
import { getProviders } from '@earendil-works/pi-ai'
|
||||
import type { CacheRetention, ThinkingBudgets, ThinkingLevel, Transport } from '@earendil-works/pi-ai'
|
||||
import z from 'schemastery'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
|
||||
/** Default maximum idle interval while an adapter stream read is outstanding. */
|
||||
export const DEFAULT_STREAM_IDLE_TIMEOUT_MS = 300_000
|
||||
|
||||
/** Configuration for one pi-ai provider route. */
|
||||
export interface PiAiProviderProfile {
|
||||
/** pi-ai provider catalog name and Harness route key. */
|
||||
provider: string
|
||||
/** Provider credential; when absent pi-ai uses its provider-native ambient discovery. */
|
||||
apiKey?: string
|
||||
/** Override the selected catalog model's endpoint without changing its protocol metadata. */
|
||||
baseURL?: string
|
||||
/** Provider request headers; Harness attribution wins reserved names. */
|
||||
headers?: Record<string, string>
|
||||
/** Provider-neutral pi-ai reasoning level. */
|
||||
reasoning?: ThinkingLevel
|
||||
/** Token budgets used by reasoning providers that support them. */
|
||||
thinkingBudgets?: ThinkingBudgets
|
||||
/** Prompt-cache retention preference. */
|
||||
cacheRetention?: CacheRetention
|
||||
/** Streaming transport preference. */
|
||||
transport?: Transport
|
||||
/** HTTP/provider SDK timeout in milliseconds. */
|
||||
timeoutMs?: number
|
||||
/** WebSocket connection timeout in milliseconds. */
|
||||
websocketConnectTimeoutMs?: number
|
||||
/** Maximum provider idle time while one stream read is outstanding. */
|
||||
streamIdleTimeoutMs?: number
|
||||
}
|
||||
|
||||
/** Validated profile with every adapter-owned default resolved. */
|
||||
export interface ResolvedPiAiProviderProfile extends PiAiProviderProfile {
|
||||
/** Positive finite provider-idle interval after defaulting. */
|
||||
streamIdleTimeoutMs: number
|
||||
}
|
||||
|
||||
/** Plugin configuration: the non-empty provider profiles this instance owns. */
|
||||
export interface Config {
|
||||
/** Non-empty set of pi-ai provider routes this adapter instance owns. */
|
||||
providers: PiAiProviderProfile[]
|
||||
}
|
||||
|
||||
const thinkingBudgets = z.object({
|
||||
minimal: z.number(),
|
||||
low: z.number(),
|
||||
medium: z.number(),
|
||||
high: z.number(),
|
||||
})
|
||||
|
||||
const profile = z.object({
|
||||
provider: z.string().required(),
|
||||
apiKey: z.string(),
|
||||
baseURL: z.string(),
|
||||
headers: z.dict(z.string()),
|
||||
reasoning: z.union(['minimal', 'low', 'medium', 'high', 'xhigh']),
|
||||
thinkingBudgets,
|
||||
cacheRetention: z.union(['none', 'short', 'long']),
|
||||
transport: z.union(['sse', 'websocket', 'websocket-cached', 'auto']),
|
||||
timeoutMs: z.natural(),
|
||||
websocketConnectTimeoutMs: z.natural(),
|
||||
streamIdleTimeoutMs: z.number().min(Number.MIN_VALUE).max(MAX_TIMER_DELAY_MS).default(DEFAULT_STREAM_IDLE_TIMEOUT_MS),
|
||||
})
|
||||
|
||||
/** Runtime schema for {@link Config}. */
|
||||
export const Config: z<Config> = z.object({
|
||||
providers: z.array(profile).required(),
|
||||
})
|
||||
|
||||
/**
|
||||
* Validate profiles against the installed pi-ai catalog and return a detached
|
||||
* shallow copy suitable for adapter construction.
|
||||
* @param profiles - configured provider profiles.
|
||||
* @returns validated profiles in configuration order.
|
||||
*/
|
||||
export function resolveProfiles(profiles: readonly PiAiProviderProfile[]): ResolvedPiAiProviderProfile[] {
|
||||
if (profiles.length === 0) throw new Error('llm-pi-ai: providers must contain at least one profile')
|
||||
const supported = new Set<string>(getProviders())
|
||||
const seen = new Set<string>()
|
||||
return profiles.map((source) => {
|
||||
const legacy = source as PiAiProviderProfile & {
|
||||
maxRetries?: unknown
|
||||
maxRetryDelayMs?: unknown
|
||||
}
|
||||
if ('maxRetries' in legacy || 'maxRetryDelayMs' in legacy) {
|
||||
throw new Error('llm-pi-ai: maxRetries and maxRetryDelayMs were removed; compose agent recovery with dsh-llm-retry')
|
||||
}
|
||||
if (source.provider.length === 0) throw new Error('llm-pi-ai: provider names must be non-empty')
|
||||
if (!supported.has(source.provider)) throw new Error(`llm-pi-ai: unknown pi-ai provider "${source.provider}"`)
|
||||
if (seen.has(source.provider)) throw new Error(`llm-pi-ai: duplicate provider profile "${source.provider}"`)
|
||||
if (source.apiKey !== undefined && source.apiKey.trim().length === 0) {
|
||||
throw new Error(`llm-pi-ai: provider "${source.provider}" has an empty apiKey; omit it to use ambient authentication`)
|
||||
}
|
||||
if (source.baseURL !== undefined && source.baseURL.length === 0) {
|
||||
throw new Error(`llm-pi-ai: provider "${source.provider}" has an empty baseURL`)
|
||||
}
|
||||
const streamIdleTimeoutMs = source.streamIdleTimeoutMs ?? DEFAULT_STREAM_IDLE_TIMEOUT_MS
|
||||
if (!Number.isFinite(streamIdleTimeoutMs)
|
||||
|| streamIdleTimeoutMs <= 0
|
||||
|| streamIdleTimeoutMs > MAX_TIMER_DELAY_MS) {
|
||||
throw new Error(
|
||||
`llm-pi-ai: provider "${source.provider}" streamIdleTimeoutMs must be a positive finite number no greater than ${MAX_TIMER_DELAY_MS}`,
|
||||
)
|
||||
}
|
||||
seen.add(source.provider)
|
||||
return {
|
||||
...source,
|
||||
streamIdleTimeoutMs,
|
||||
...source.headers === undefined ? {} : { headers: { ...source.headers } },
|
||||
...source.thinkingBudgets === undefined ? {} : { thinkingBudgets: { ...source.thinkingBudgets } },
|
||||
}
|
||||
})
|
||||
}
|
||||
85
packages/llm/llm-pi-ai/src/context.ts
Normal file
85
packages/llm/llm-pi-ai/src/context.ts
Normal file
@@ -0,0 +1,85 @@
|
||||
/**
|
||||
* Harness request-history conversion into pi-ai's Context vocabulary.
|
||||
*
|
||||
* @module dsh-llm-pi-ai/context
|
||||
*/
|
||||
|
||||
import { CallId } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, Message } from '@deepseek-ai/dsh-llm'
|
||||
import type { Context as PiContext, Message as PiMessage, Tool as PiTool } from '@earendil-works/pi-ai'
|
||||
import { toPiAssistant } from './replay.ts'
|
||||
|
||||
/** Join the text blocks of a harness message. */
|
||||
function flattenText(message: Message): string {
|
||||
return message.content
|
||||
.filter(block => block.type === 'text')
|
||||
.map(block => block.text)
|
||||
.join('')
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert harness history to a pi-ai Context. Tool results need the tool
|
||||
* NAME (pi-ai's `toolName`), which the harness doesn't carry on the result
|
||||
* block — it is recovered from the preceding assistant tool-call with the
|
||||
* same id.
|
||||
* @param options - the harness request; `options.system` maps to pi-ai's single `systemPrompt` slot.
|
||||
* @returns the pi-ai context; `tools` is omitted entirely when the request declares none.
|
||||
*/
|
||||
export function toPiContext(options: GenerateOptions): PiContext {
|
||||
const toolNames = new Map<CallId, string>()
|
||||
const messages: PiMessage[] = []
|
||||
|
||||
for (const message of options.messages) {
|
||||
if (message.role === 'system') {
|
||||
// pi-ai has a single systemPrompt slot; in-history system messages are
|
||||
// folded into user messages to preserve order (rare in practice — the
|
||||
// harness sends the system prompt via options.system).
|
||||
messages.push({ role: 'user', content: flattenText(message), timestamp: 0 })
|
||||
continue
|
||||
}
|
||||
if (message.role === 'assistant') {
|
||||
const assistant = toPiAssistant(message)
|
||||
for (const block of assistant.content) {
|
||||
if (block.type === 'toolCall') toolNames.set(CallId(block.id), block.name)
|
||||
}
|
||||
messages.push(assistant)
|
||||
continue
|
||||
}
|
||||
// user role: text + tool results (each result becomes its own message).
|
||||
const text = flattenText(message)
|
||||
const results = message.content.filter(block => block.type === 'tool-result')
|
||||
if (text.length > 0 || results.length === 0) {
|
||||
messages.push({ role: 'user', content: text, timestamp: 0 })
|
||||
}
|
||||
for (const result of results) {
|
||||
messages.push({
|
||||
role: 'toolResult',
|
||||
toolCallId: result.toolCallId,
|
||||
toolName: toolNames.get(result.toolCallId) ?? 'unknown',
|
||||
content: [{
|
||||
type: 'text',
|
||||
text: result.content
|
||||
.filter(block => block.type === 'text')
|
||||
.map(block => block.text)
|
||||
.join('') || '(no output)',
|
||||
}],
|
||||
isError: result.isError ?? false,
|
||||
timestamp: 0,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
const tools: PiTool[] | undefined = options.tools?.map(tool => ({
|
||||
name: tool.name,
|
||||
description: tool.description,
|
||||
// ToolSchema.parameters is a JSON Schema object; pi-ai's TSchema
|
||||
// (TypeBox) is structurally JSON Schema, so it assigns directly.
|
||||
parameters: tool.parameters,
|
||||
}))
|
||||
|
||||
return {
|
||||
...options.system !== undefined ? { systemPrompt: options.system } : {},
|
||||
messages,
|
||||
...tools !== undefined && tools.length > 0 ? { tools } : {},
|
||||
}
|
||||
}
|
||||
@@ -1,276 +0,0 @@
|
||||
/**
|
||||
* Bidirectional mapping between the harness vocabulary and pi-ai's:
|
||||
* Convert harness requests to pi-ai context and pi-ai assistant events to harness stream chunks.
|
||||
* pi-ai parses tool arguments while the harness preserves raw JSON, so conversion parses inbound
|
||||
* arguments and re-stringifies outbound values while the adapter restores provider payloads.
|
||||
* In-stream pi-ai errors become harness error/aborted finishes, and its reasoning tokens remain
|
||||
* folded into output usage because it reports no separate count.
|
||||
* @module dsh-llm-pi-ai/convert
|
||||
*/
|
||||
|
||||
import { CallId, LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import type { FinishReason, GenerateOptions, Message, StreamChunk, TokenUsage } from '@deepseek-ai/dsh-llm'
|
||||
import type {
|
||||
AssistantMessage,
|
||||
AssistantMessageEvent,
|
||||
Context as PiContext,
|
||||
Message as PiMessage,
|
||||
Tool as PiTool,
|
||||
Usage as PiUsage,
|
||||
} from '@earendil-works/pi-ai'
|
||||
|
||||
/** Join the text blocks of a harness message. */
|
||||
function flattenText(message: Message): string {
|
||||
return message.content
|
||||
.filter(block => block.type === 'text')
|
||||
.map(block => block.text)
|
||||
.join('')
|
||||
}
|
||||
|
||||
/** Parse tool-call argument JSON; tolerate model malformations with {}. */
|
||||
function parseArguments(raw: string): Record<string, unknown> {
|
||||
try {
|
||||
const parsed: unknown = JSON.parse(raw)
|
||||
if (typeof parsed === 'object' && parsed !== null && !Array.isArray(parsed)) {
|
||||
return parsed as Record<string, unknown>
|
||||
}
|
||||
} catch {
|
||||
// fall through
|
||||
}
|
||||
return {}
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert harness history to a pi-ai Context. Tool results need the tool
|
||||
* NAME (pi-ai's `toolName`), which the harness doesn't carry on the result
|
||||
* block — it is recovered from the preceding assistant tool-call with the
|
||||
* same id.
|
||||
* @param options - the harness request; `options.system` maps to pi-ai's single `systemPrompt` slot.
|
||||
* @returns the pi-ai context; `tools` is omitted entirely when the request declares none.
|
||||
*/
|
||||
export function toPiContext(options: GenerateOptions): PiContext {
|
||||
const toolNames = new Map<CallId, string>()
|
||||
const messages: PiMessage[] = []
|
||||
|
||||
for (const message of options.messages) {
|
||||
if (message.role === 'system') {
|
||||
// pi-ai has a single systemPrompt slot; in-history system messages are
|
||||
// folded into user messages to preserve order (rare in practice — the
|
||||
// harness sends the system prompt via options.system).
|
||||
messages.push({ role: 'user', content: flattenText(message), timestamp: 0 })
|
||||
continue
|
||||
}
|
||||
if (message.role === 'assistant') {
|
||||
const content: AssistantMessage['content'] = []
|
||||
for (const block of message.content) {
|
||||
switch (block.type) {
|
||||
case 'text':
|
||||
content.push({ type: 'text', text: block.text })
|
||||
break
|
||||
case 'reasoning':
|
||||
// Without this wire-field name, pi-ai replays an empty `reasoning_content`, violating
|
||||
// DeepSeek's thinking-mode passback rule on tool-call turns.
|
||||
content.push({ type: 'thinking', thinking: block.text, thinkingSignature: 'reasoning_content' })
|
||||
break
|
||||
case 'tool-call':
|
||||
toolNames.set(block.id, block.name)
|
||||
content.push({
|
||||
type: 'toolCall',
|
||||
id: block.id,
|
||||
name: block.name,
|
||||
arguments: parseArguments(block.arguments),
|
||||
})
|
||||
break
|
||||
default:
|
||||
// plugin-added block types: not representable here.
|
||||
break
|
||||
}
|
||||
}
|
||||
messages.push({
|
||||
role: 'assistant',
|
||||
content,
|
||||
api: 'openai-completions',
|
||||
provider: 'deepseek',
|
||||
model: options.model,
|
||||
usage: emptyPiUsage(),
|
||||
stopReason: content.some(piece => piece.type === 'toolCall') ? 'toolUse' : 'stop',
|
||||
timestamp: 0,
|
||||
})
|
||||
continue
|
||||
}
|
||||
// user role: text + tool results (each result becomes its own message).
|
||||
const text = flattenText(message)
|
||||
const results = message.content.filter(block => block.type === 'tool-result')
|
||||
if (text.length > 0 || results.length === 0) {
|
||||
messages.push({ role: 'user', content: text, timestamp: 0 })
|
||||
}
|
||||
for (const result of results) {
|
||||
messages.push({
|
||||
role: 'toolResult',
|
||||
toolCallId: result.toolCallId,
|
||||
toolName: toolNames.get(result.toolCallId) ?? 'unknown',
|
||||
content: [{
|
||||
type: 'text',
|
||||
text: result.content
|
||||
.filter(block => block.type === 'text')
|
||||
.map(block => block.text)
|
||||
.join('') || '(no output)',
|
||||
}],
|
||||
isError: result.isError ?? false,
|
||||
timestamp: 0,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
const tools: PiTool[] | undefined = options.tools?.map(tool => ({
|
||||
name: tool.name,
|
||||
description: tool.description,
|
||||
// ToolSchema.parameters is a JSON Schema object; pi-ai's TSchema
|
||||
// (TypeBox) is structurally JSON Schema, so it assigns directly.
|
||||
parameters: tool.parameters,
|
||||
}))
|
||||
|
||||
return {
|
||||
...options.system !== undefined ? { systemPrompt: options.system } : {},
|
||||
messages,
|
||||
...tools !== undefined && tools.length > 0 ? { tools } : {},
|
||||
}
|
||||
}
|
||||
|
||||
function emptyPiUsage(): PiUsage {
|
||||
return {
|
||||
input: 0,
|
||||
output: 0,
|
||||
cacheRead: 0,
|
||||
cacheWrite: 0,
|
||||
totalTokens: 0,
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Map pi-ai usage (reasoning folded into output by pi-ai).
|
||||
* @param usage - cumulative usage from the terminal pi-ai event.
|
||||
* @returns harness counts; cache fields appear only when non-zero (pi-ai reports zeros, not absence).
|
||||
*/
|
||||
export function mapUsage(usage: PiUsage): TokenUsage {
|
||||
return {
|
||||
inputTokens: usage.input,
|
||||
outputTokens: usage.output,
|
||||
...usage.cacheRead > 0 ? { cacheReadTokens: usage.cacheRead } : {},
|
||||
...usage.cacheWrite > 0 ? { cacheWriteTokens: usage.cacheWrite } : {},
|
||||
}
|
||||
}
|
||||
|
||||
function classifyPiAiError(message: string): string {
|
||||
if (/\b(?:401|403)\b/.test(message)) return 'AUTH'
|
||||
if (/\b429\b|rate.?limit/i.test(message)) return 'RATE_LIMIT'
|
||||
if (/\b400\b|invalid.?request/i.test(message)) return 'INVALID_REQUEST'
|
||||
if (/\b5\d\d\b/.test(message)) return 'SERVER'
|
||||
return 'PI_AI_ERROR'
|
||||
}
|
||||
|
||||
/**
|
||||
* Map a terminal pi-ai event to the harness finish reason.
|
||||
* @param message - the assistant message carried by the `done` or `error` event.
|
||||
* @returns the harness reason; `error` yields `{kind: 'error'}` with a code classified from the error text.
|
||||
*/
|
||||
export function mapStopReason(message: AssistantMessage): FinishReason {
|
||||
switch (message.stopReason) {
|
||||
case 'stop': return { kind: 'stop' }
|
||||
case 'length': return { kind: 'max-tokens' }
|
||||
case 'toolUse': return { kind: 'tool-calls' }
|
||||
case 'aborted': return { kind: 'aborted' }
|
||||
case 'error': {
|
||||
const text = message.errorMessage ?? 'pi-ai stream error'
|
||||
return { kind: 'error', message: text, code: classifyPiAiError(text) }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Translate the pi-ai event stream into StreamChunks. pi-ai never throws
|
||||
* mid-stream — failures arrive as `error` events, which become error/aborted
|
||||
* `finish` chunks (the harness protocol's other error-delivery style).
|
||||
* @param events - one assistant turn's pi-ai event stream.
|
||||
* @returns the harness chunks, ending with `usage` then `finish`; throws
|
||||
* `LlmError` (`STREAM_CLOSED`) if the source ends without a terminal event.
|
||||
*/
|
||||
export async function* toStreamChunks(events: AsyncIterable<AssistantMessageEvent>): AsyncGenerator<StreamChunk> {
|
||||
// pi-ai contentIndex ↔ our block index map 1:1 (both count blocks from 0
|
||||
// in stream order), but we track ids per index for tool calls.
|
||||
const toolIds = new Map<number, { id: string; name: string }>()
|
||||
|
||||
for await (const event of events) {
|
||||
switch (event.type) {
|
||||
case 'start':
|
||||
break
|
||||
case 'text_start':
|
||||
yield { type: 'block-start', index: event.contentIndex, blockType: 'text' }
|
||||
break
|
||||
case 'text_delta':
|
||||
yield { type: 'text-delta', index: event.contentIndex, text: event.delta }
|
||||
break
|
||||
case 'text_end':
|
||||
yield { type: 'block-end', index: event.contentIndex, block: { type: 'text', text: event.content } }
|
||||
break
|
||||
case 'thinking_start':
|
||||
yield { type: 'block-start', index: event.contentIndex, blockType: 'reasoning' }
|
||||
break
|
||||
case 'thinking_delta':
|
||||
yield { type: 'reasoning-delta', index: event.contentIndex, text: event.delta }
|
||||
break
|
||||
case 'thinking_end':
|
||||
yield { type: 'block-end', index: event.contentIndex, block: { type: 'reasoning', text: event.content } }
|
||||
break
|
||||
case 'toolcall_start': {
|
||||
// The id/name live on the partial's content at this index.
|
||||
const partial = event.partial.content[event.contentIndex]
|
||||
const id = partial?.type === 'toolCall' ? partial.id : ''
|
||||
const name = partial?.type === 'toolCall' ? partial.name : ''
|
||||
toolIds.set(event.contentIndex, { id, name })
|
||||
yield { type: 'block-start', index: event.contentIndex, blockType: 'tool-call' }
|
||||
break
|
||||
}
|
||||
case 'toolcall_delta': {
|
||||
const known = toolIds.get(event.contentIndex)
|
||||
yield {
|
||||
type: 'tool-call-delta',
|
||||
index: event.contentIndex,
|
||||
id: CallId(known?.id ?? ''),
|
||||
...known?.name !== undefined && known.name.length > 0 ? { name: known.name } : {},
|
||||
argumentsDelta: event.delta,
|
||||
}
|
||||
break
|
||||
}
|
||||
case 'toolcall_end':
|
||||
yield {
|
||||
type: 'block-end',
|
||||
index: event.contentIndex,
|
||||
block: {
|
||||
type: 'tool-call',
|
||||
id: CallId(event.toolCall.id),
|
||||
name: event.toolCall.name,
|
||||
// pi-ai hands back the PARSED arguments; the harness vocabulary
|
||||
// keeps the raw string.
|
||||
arguments: JSON.stringify(event.toolCall.arguments),
|
||||
},
|
||||
}
|
||||
break
|
||||
case 'done':
|
||||
yield { type: 'usage', usage: mapUsage(event.message.usage) }
|
||||
yield { type: 'finish', reason: mapStopReason(event.message) }
|
||||
return
|
||||
case 'error':
|
||||
// In-stream error delivery (pi-ai's style) → error finish chunk
|
||||
// (the harness's other sanctioned error path besides throwing).
|
||||
yield { type: 'usage', usage: mapUsage(event.error.usage) }
|
||||
yield { type: 'finish', reason: mapStopReason(event.error) }
|
||||
return
|
||||
// no default: AssistantMessageEvent is pi-ai's closed union; a new
|
||||
// event type should fail compilation here via tsc's exhaustiveness
|
||||
// when one is added (switch covers all current variants).
|
||||
}
|
||||
}
|
||||
throw new LlmError('pi-ai event stream ended without done/error', 'STREAM_CLOSED')
|
||||
}
|
||||
@@ -1,76 +1,41 @@
|
||||
/**
|
||||
* pi-ai-backed DeepSeek adapter plugin. Same Config shape as
|
||||
* `@deepseek-ai/dsh-llm-deepseek` (one-line swap in cordis.yml), different
|
||||
* implementation underneath — see `./adapter.ts` for why both exist.
|
||||
* Generic pi-ai-backed LLM adapter plugin. One plugin instance registers an
|
||||
* explicit set of provider profiles; requests select a profile by provider and
|
||||
* resolve the model dynamically from pi-ai's installed catalog.
|
||||
*
|
||||
* ```yaml
|
||||
* - id: llm
|
||||
* name: '@deepseek-ai/dsh-llm-pi-ai'
|
||||
* config:
|
||||
* apiKey: !!js process.env.DEEPSEEK_API_KEY
|
||||
* baseURL: !!js process.env.DEEPSEEK_BASE_URL
|
||||
* models: [deepseek-v4-flash, deepseek-v4-pro]
|
||||
* reasoning: high
|
||||
* providers:
|
||||
* - provider: openai
|
||||
* apiKey: !!js process.env.OPENAI_API_KEY
|
||||
* - provider: anthropic
|
||||
* apiKey: !!js process.env.ANTHROPIC_API_KEY
|
||||
* - provider: openrouter
|
||||
* apiKey: !!js process.env.OPENROUTER_API_KEY
|
||||
* baseURL: https://proxy.example.com/v1
|
||||
* ```
|
||||
*
|
||||
* @module @deepseek-ai/dsh-llm-pi-ai
|
||||
*/
|
||||
|
||||
import type { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import type {} from '@deepseek-ai/dsh-llm'
|
||||
import { PiAiAdapter } from './adapter.ts'
|
||||
import type { PiAiReasoning } from './adapter.ts'
|
||||
import { Config, resolveProfiles } from './config.ts'
|
||||
|
||||
export { buildModel, PiAiAdapter } from './adapter.ts'
|
||||
export type { PiAiAdapterOptions, PiAiReasoning } from './adapter.ts'
|
||||
export { mapStopReason, mapUsage, toPiContext, toStreamChunks } from './convert.ts'
|
||||
export { PiAiAdapter } from './adapter.ts'
|
||||
export type { PiAiAdapterOptions } from './adapter.ts'
|
||||
export { Config } from './config.ts'
|
||||
export type { PiAiProviderProfile } from './config.ts'
|
||||
|
||||
export const name = 'llm-pi-ai'
|
||||
export const inject = ['llm']
|
||||
|
||||
/**
|
||||
* Plugin config, validated by the same-named schemastery schema. Every field
|
||||
* is optional in yml: credentials/endpoint fall back to the environment (a
|
||||
* missing API key fails plugin load, not the first call).
|
||||
*/
|
||||
export interface Config {
|
||||
/** API key; falls back to $DEEPSEEK_API_KEY. Required one way or the other. */
|
||||
apiKey?: string
|
||||
/** Endpoint base; falls back to $DEEPSEEK_BASE_URL, then the public API. */
|
||||
baseURL?: string
|
||||
/** Model names to register (sent verbatim on the wire). */
|
||||
models?: string[]
|
||||
/**
|
||||
* Thinking level for every request: 'off' disables thinking mode; 'high'
|
||||
* and 'xhigh' (wire 'max') set the effort. Omitted = provider default
|
||||
* (thinking enabled), matching llm-deepseek's omission semantics.
|
||||
*/
|
||||
reasoning?: PiAiReasoning
|
||||
}
|
||||
|
||||
export const Config: z<Config> = z.object({
|
||||
apiKey: z.string(),
|
||||
baseURL: z.string(),
|
||||
models: z.array(z.string()).default(['deepseek-v4-flash', 'deepseek-v4-pro']),
|
||||
reasoning: z.union(['off', 'high', 'xhigh']),
|
||||
})
|
||||
|
||||
/** Public API default; the internal endpoint comes from $DEEPSEEK_BASE_URL. */
|
||||
export const PUBLIC_BASE_URL = 'https://api.deepseek.com'
|
||||
|
||||
/** Register one generic pi-ai adapter for all configured provider routes. */
|
||||
export function apply(ctx: Context, config: Config): void {
|
||||
const apiKey = config.apiKey ?? process.env.DEEPSEEK_API_KEY
|
||||
if (apiKey === undefined || apiKey.length === 0) {
|
||||
throw new Error('llm-pi-ai: an API key is required (Config.apiKey or $DEEPSEEK_API_KEY)')
|
||||
}
|
||||
const baseURL = config.baseURL ?? process.env.DEEPSEEK_BASE_URL ?? PUBLIC_BASE_URL
|
||||
// schemastery's .default() guarantees models is set after validation.
|
||||
const models = config.models as string[]
|
||||
|
||||
ctx.llm.registerAdapter(models, new PiAiAdapter({
|
||||
apiKey,
|
||||
baseURL,
|
||||
reasoning: config.reasoning,
|
||||
}))
|
||||
const profiles = resolveProfiles(config.providers)
|
||||
const adapter = new PiAiAdapter({ profiles })
|
||||
ctx.llm.registerAdapter(profiles.map(entry => entry.provider), adapter)
|
||||
}
|
||||
|
||||
211
packages/llm/llm-pi-ai/src/replay.ts
Normal file
211
packages/llm/llm-pi-ai/src/replay.ts
Normal file
@@ -0,0 +1,211 @@
|
||||
/**
|
||||
* Durable pi-ai replay metadata and assistant-history reconstruction.
|
||||
*
|
||||
* Harness content remains the durable source for text and tool calls. This
|
||||
* module stores only the provider-native metadata needed to reconstruct a
|
||||
* pi-ai assistant message on a later request.
|
||||
*
|
||||
* @module dsh-llm-pi-ai/replay
|
||||
*/
|
||||
|
||||
import { LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import type { Message } from '@deepseek-ai/dsh-llm'
|
||||
import type { Api, AssistantMessage, Usage as PiUsage } from '@earendil-works/pi-ai'
|
||||
|
||||
type PiAiReplayBlock =
|
||||
| { type: 'text'; textSignature?: string }
|
||||
| { type: 'reasoning'; thinkingSignature?: string; redacted?: boolean }
|
||||
| { type: 'tool-call'; thoughtSignature?: string }
|
||||
|
||||
/** Versioned adapter-private projection required to replay a pi-ai response. */
|
||||
export interface PiAiReplayState {
|
||||
kind: 'pi-ai'
|
||||
version: 1
|
||||
api: Api
|
||||
provider: string
|
||||
model: string
|
||||
responseModel?: string
|
||||
responseId?: string
|
||||
stopReason: AssistantMessage['stopReason']
|
||||
blocks: PiAiReplayBlock[]
|
||||
}
|
||||
|
||||
/** Parse tool-call argument JSON; tolerate model malformations with {}. */
|
||||
function parseArguments(raw: string): Record<string, unknown> {
|
||||
try {
|
||||
const parsed: unknown = JSON.parse(raw)
|
||||
if (typeof parsed === 'object' && parsed !== null && !Array.isArray(parsed)) {
|
||||
return parsed as Record<string, unknown>
|
||||
}
|
||||
} catch {
|
||||
// fall through
|
||||
}
|
||||
return {}
|
||||
}
|
||||
|
||||
/** Construct the zero usage value required by historical pi-ai messages. */
|
||||
function emptyPiUsage(): PiUsage {
|
||||
return {
|
||||
input: 0,
|
||||
output: 0,
|
||||
cacheRead: 0,
|
||||
cacheWrite: 0,
|
||||
totalTokens: 0,
|
||||
cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0, total: 0 },
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Project a successful pi-ai response into the minimal durable replay state.
|
||||
* @param message - completed native pi-ai assistant response.
|
||||
* @returns the versioned lossless-JSON replay projection.
|
||||
*/
|
||||
export function toPiReplayState(message: AssistantMessage): PiAiReplayState {
|
||||
return {
|
||||
kind: 'pi-ai',
|
||||
version: 1,
|
||||
api: message.api,
|
||||
provider: message.provider,
|
||||
model: message.model,
|
||||
...message.responseModel === undefined ? {} : { responseModel: message.responseModel },
|
||||
...message.responseId === undefined ? {} : { responseId: message.responseId },
|
||||
stopReason: message.stopReason,
|
||||
blocks: message.content.map((block): PiAiReplayBlock => {
|
||||
switch (block.type) {
|
||||
case 'text': return {
|
||||
type: 'text',
|
||||
...block.textSignature === undefined ? {} : { textSignature: block.textSignature },
|
||||
}
|
||||
case 'thinking': return {
|
||||
type: 'reasoning',
|
||||
...block.thinkingSignature === undefined ? {} : { thinkingSignature: block.thinkingSignature },
|
||||
...block.redacted === undefined ? {} : { redacted: block.redacted },
|
||||
}
|
||||
case 'toolCall': return {
|
||||
type: 'tool-call',
|
||||
...block.thoughtSignature === undefined ? {} : { thoughtSignature: block.thoughtSignature },
|
||||
}
|
||||
}
|
||||
}),
|
||||
}
|
||||
}
|
||||
|
||||
function invalidReplay(message: string): never {
|
||||
throw new LlmError(`invalid pi-ai replay state: ${message}`, 'INVALID_REPLAY_STATE')
|
||||
}
|
||||
|
||||
/** Validate the adapter-private state before it reaches pi-ai. */
|
||||
function readReplayState(value: unknown): PiAiReplayState {
|
||||
if (typeof value !== 'object' || value === null || Array.isArray(value)) return invalidReplay('expected an object')
|
||||
const state = value as Record<string, unknown>
|
||||
if (state['kind'] !== 'pi-ai') return invalidReplay('unknown state kind')
|
||||
if (state['version'] !== 1) return invalidReplay(`unsupported version ${String(state['version'])}`)
|
||||
for (const key of ['api', 'provider', 'model'] as const) {
|
||||
if (typeof state[key] !== 'string' || state[key].length === 0) return invalidReplay(`${key} must be a non-empty string`)
|
||||
}
|
||||
if (!['stop', 'length', 'toolUse', 'error', 'aborted'].includes(String(state['stopReason']))) {
|
||||
return invalidReplay('unknown stopReason')
|
||||
}
|
||||
if (state['responseModel'] !== undefined && typeof state['responseModel'] !== 'string') return invalidReplay('responseModel must be a string')
|
||||
if (state['responseId'] !== undefined && typeof state['responseId'] !== 'string') return invalidReplay('responseId must be a string')
|
||||
if (!Array.isArray(state['blocks'])) return invalidReplay('blocks must be an array')
|
||||
for (const [index, value] of state['blocks'].entries()) {
|
||||
if (typeof value !== 'object' || value === null || Array.isArray(value)) return invalidReplay(`block ${index} must be an object`)
|
||||
const block = value as Record<string, unknown>
|
||||
if (!['text', 'reasoning', 'tool-call'].includes(String(block['type']))) return invalidReplay(`block ${index} has an unknown type`)
|
||||
for (const signature of ['textSignature', 'thinkingSignature', 'thoughtSignature'] as const) {
|
||||
if (block[signature] !== undefined && typeof block[signature] !== 'string') return invalidReplay(`block ${index} ${signature} must be a string`)
|
||||
}
|
||||
if (block['redacted'] !== undefined && typeof block['redacted'] !== 'boolean') return invalidReplay(`block ${index} redacted must be boolean`)
|
||||
}
|
||||
return state as unknown as PiAiReplayState
|
||||
}
|
||||
|
||||
/** Convert provider-neutral blocks without trusting them as same-model replay. */
|
||||
function foreignAssistant(message: Message): AssistantMessage {
|
||||
const content: AssistantMessage['content'] = []
|
||||
for (const block of message.content) {
|
||||
switch (block.type) {
|
||||
case 'text': content.push({ type: 'text', text: block.text }); break
|
||||
case 'reasoning': content.push({ type: 'thinking', thinking: block.text }); break
|
||||
case 'tool-call': content.push({
|
||||
type: 'toolCall',
|
||||
id: block.id,
|
||||
name: block.name,
|
||||
arguments: parseArguments(block.arguments),
|
||||
}); break
|
||||
default:
|
||||
// plugin-added block types are not representable in pi-ai.
|
||||
break
|
||||
}
|
||||
}
|
||||
return {
|
||||
role: 'assistant',
|
||||
content,
|
||||
// Deliberately never equals a catalog API: absent replay state is foreign
|
||||
// even if provenance names the same provider/model as this request.
|
||||
api: 'dsh-foreign',
|
||||
provider: message.provenance?.provider ?? 'dsh-foreign',
|
||||
model: message.provenance?.model ?? 'dsh-foreign',
|
||||
usage: emptyPiUsage(),
|
||||
stopReason: content.some(piece => piece.type === 'toolCall') ? 'toolUse' : 'stop',
|
||||
timestamp: 0,
|
||||
}
|
||||
}
|
||||
|
||||
/** Recombine durable Harness content with validated pi-ai replay metadata. */
|
||||
function replayedAssistant(message: Message, rawState: unknown): AssistantMessage {
|
||||
const state = readReplayState(rawState)
|
||||
const provenance = message.provenance
|
||||
if (state.provider !== provenance?.provider) return invalidReplay('provider does not match assistant provenance')
|
||||
if (state.model !== provenance.model) return invalidReplay('model does not match assistant provenance')
|
||||
if (state.blocks.length !== message.content.length) return invalidReplay('block count does not match assistant content')
|
||||
const content: AssistantMessage['content'] = message.content.map((block, index) => {
|
||||
const replay = state.blocks[index]
|
||||
if (replay === undefined || replay.type !== block.type) return invalidReplay(`block ${index} does not match assistant content`)
|
||||
switch (block.type) {
|
||||
case 'text': return {
|
||||
type: 'text',
|
||||
text: block.text,
|
||||
...replay.type === 'text' && replay.textSignature !== undefined ? { textSignature: replay.textSignature } : {},
|
||||
}
|
||||
case 'reasoning': return {
|
||||
type: 'thinking',
|
||||
thinking: block.text,
|
||||
...replay.type === 'reasoning' && replay.thinkingSignature !== undefined ? { thinkingSignature: replay.thinkingSignature } : {},
|
||||
...replay.type === 'reasoning' && replay.redacted !== undefined ? { redacted: replay.redacted } : {},
|
||||
}
|
||||
case 'tool-call': return {
|
||||
type: 'toolCall',
|
||||
id: block.id,
|
||||
name: block.name,
|
||||
arguments: parseArguments(block.arguments),
|
||||
...replay.type === 'tool-call' && replay.thoughtSignature !== undefined ? { thoughtSignature: replay.thoughtSignature } : {},
|
||||
}
|
||||
/* v8 ignore next -- readReplayState rejects unknown replay tags, so an equal plugin-added Harness tag cannot reach this switch */
|
||||
default: return invalidReplay(`block ${index} has an unsupported Harness type`)
|
||||
}
|
||||
})
|
||||
return {
|
||||
role: 'assistant',
|
||||
content,
|
||||
api: state.api,
|
||||
provider: state.provider,
|
||||
model: state.model,
|
||||
...state.responseModel === undefined ? {} : { responseModel: state.responseModel },
|
||||
...state.responseId === undefined ? {} : { responseId: state.responseId },
|
||||
usage: emptyPiUsage(),
|
||||
stopReason: state.stopReason,
|
||||
timestamp: 0,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert one durable Harness assistant message into pi-ai history.
|
||||
* @param message - assistant content with optional adapter-owned replay metadata.
|
||||
* @returns a native pi-ai assistant message reconstructed from durable content.
|
||||
*/
|
||||
export function toPiAssistant(message: Message): AssistantMessage {
|
||||
const replayState = message.provenance?.replayState
|
||||
return replayState === undefined ? foreignAssistant(message) : replayedAssistant(message, replayState)
|
||||
}
|
||||
176
packages/llm/llm-pi-ai/src/stream.ts
Normal file
176
packages/llm/llm-pi-ai/src/stream.ts
Normal file
@@ -0,0 +1,176 @@
|
||||
/**
|
||||
* pi-ai assistant event translation into the Harness streaming protocol.
|
||||
*
|
||||
* pi-ai tool-call arguments are parsed objects while the Harness keeps their
|
||||
* raw JSON representation. pi-ai also reports failures as terminal stream
|
||||
* events, which this module maps into Harness finish chunks.
|
||||
*
|
||||
* @module dsh-llm-pi-ai/stream
|
||||
*/
|
||||
|
||||
import { CallId, CONTEXT_WINDOW_EXCEEDED_CODE, isContextWindowExceededError, isQuotaExceededError, LlmError, QUOTA_EXCEEDED_CODE } from '@deepseek-ai/dsh-llm'
|
||||
import type { FinishReason, StreamChunk, TokenUsage } from '@deepseek-ai/dsh-llm'
|
||||
import { isContextOverflow } from '@earendil-works/pi-ai'
|
||||
import type { AssistantMessage, AssistantMessageEvent, Usage as PiUsage } from '@earendil-works/pi-ai'
|
||||
import { toPiReplayState } from './replay.ts'
|
||||
|
||||
/**
|
||||
* Map pi-ai usage (reasoning folded into output by pi-ai).
|
||||
* @param usage - cumulative usage from the terminal pi-ai event.
|
||||
* @returns harness counts; cache fields appear only when non-zero (pi-ai reports zeros, not absence).
|
||||
*/
|
||||
export function mapUsage(usage: PiUsage): TokenUsage {
|
||||
return {
|
||||
inputTokens: usage.input,
|
||||
outputTokens: usage.output,
|
||||
...usage.cacheRead > 0 ? { cacheReadTokens: usage.cacheRead } : {},
|
||||
...usage.cacheWrite > 0 ? { cacheWriteTokens: usage.cacheWrite } : {},
|
||||
}
|
||||
}
|
||||
|
||||
function classifyPiAiError(message: string): string {
|
||||
if (/\b(?:401|403)\b/.test(message)) return 'AUTH'
|
||||
if (isQuotaExceededError(message)) return QUOTA_EXCEEDED_CODE
|
||||
if (/\b429\b|rate.?limit/i.test(message)) return 'RATE_LIMIT'
|
||||
if (/\b400\b|invalid.?request/i.test(message)) return 'INVALID_REQUEST'
|
||||
if (/\b5\d\d\b/.test(message)) return 'SERVER'
|
||||
if (/\btime(?:d)?\s*out\b|timeout/i.test(message)) return 'TIMEOUT'
|
||||
if (/\b(?:network|connection|socket|fetch)\b|\bECONN[A-Z]+\b/i.test(message)
|
||||
|| /\b(?:other side closed|HTTP2 request did not get a response|WebSocket closed unexpectedly)\b/i.test(message)) {
|
||||
return 'TRANSPORT'
|
||||
}
|
||||
return 'PI_AI_ERROR'
|
||||
}
|
||||
|
||||
/**
|
||||
* Map a terminal pi-ai event to the harness finish reason.
|
||||
* @param message - the assistant message carried by the `done` or `error` event.
|
||||
* @param contextWindow - resolved catalog capacity for usage-based overflow detection.
|
||||
* @returns the mapped harness reason. Recognized error text, `stop` usage above
|
||||
* `contextWindow`, and zero-output `length` usage that fills the window map
|
||||
* to `CONTEXT_WINDOW_EXCEEDED`.
|
||||
*/
|
||||
export function mapStopReason(message: AssistantMessage, contextWindow?: number): FinishReason {
|
||||
const piAiOverflow = isContextOverflow(message, contextWindow)
|
||||
const harnessOverflow = message.stopReason === 'error'
|
||||
&& message.errorMessage !== undefined
|
||||
&& isContextWindowExceededError(message.errorMessage)
|
||||
if (piAiOverflow || harnessOverflow) {
|
||||
return {
|
||||
kind: 'error',
|
||||
failure: {
|
||||
message: message.errorMessage ?? `pi-ai detected context overflow for model "${message.model}"`,
|
||||
code: CONTEXT_WINDOW_EXCEEDED_CODE,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
switch (message.stopReason) {
|
||||
case 'stop': return { kind: 'stop' }
|
||||
case 'length': return { kind: 'max-tokens' }
|
||||
case 'toolUse': return { kind: 'tool-calls' }
|
||||
case 'aborted': return {
|
||||
kind: 'aborted',
|
||||
failure: { message: message.errorMessage ?? 'pi-ai stream aborted', code: 'ABORTED' },
|
||||
}
|
||||
case 'error': {
|
||||
const text = message.errorMessage ?? 'pi-ai stream error'
|
||||
return { kind: 'error', failure: { message: text, code: classifyPiAiError(text) } }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Translate the pi-ai event stream into StreamChunks. pi-ai never throws
|
||||
* mid-stream — failures arrive as `error` events, which become error/aborted
|
||||
* `finish` chunks (the harness protocol's other error-delivery style).
|
||||
* @param events - one assistant turn's pi-ai event stream.
|
||||
* @param contextWindow - resolved catalog capacity for usage-based overflow detection.
|
||||
* @returns the harness chunks, ending with `usage` then `finish`; throws
|
||||
* `LlmError` (`STREAM_CLOSED`) if the source ends without a terminal event.
|
||||
*/
|
||||
export async function* toStreamChunks(
|
||||
events: AsyncIterable<AssistantMessageEvent>,
|
||||
contextWindow?: number,
|
||||
): AsyncGenerator<StreamChunk> {
|
||||
// pi-ai contentIndex ↔ our block index map 1:1 (both count blocks from 0
|
||||
// in stream order), but we track ids per index for tool calls.
|
||||
const toolIds = new Map<number, { id: string; name: string }>()
|
||||
|
||||
for await (const event of events) {
|
||||
switch (event.type) {
|
||||
case 'start':
|
||||
break
|
||||
case 'text_start':
|
||||
yield { type: 'block-start', index: event.contentIndex, blockType: 'text' }
|
||||
break
|
||||
case 'text_delta':
|
||||
yield { type: 'text-delta', index: event.contentIndex, text: event.delta }
|
||||
break
|
||||
case 'text_end':
|
||||
yield { type: 'block-end', index: event.contentIndex, block: { type: 'text', text: event.content } }
|
||||
break
|
||||
case 'thinking_start':
|
||||
yield { type: 'block-start', index: event.contentIndex, blockType: 'reasoning' }
|
||||
break
|
||||
case 'thinking_delta':
|
||||
yield { type: 'reasoning-delta', index: event.contentIndex, text: event.delta }
|
||||
break
|
||||
case 'thinking_end':
|
||||
yield { type: 'block-end', index: event.contentIndex, block: { type: 'reasoning', text: event.content } }
|
||||
break
|
||||
case 'toolcall_start': {
|
||||
// The id/name live on the partial's content at this index.
|
||||
const partial = event.partial.content[event.contentIndex]
|
||||
const id = partial?.type === 'toolCall' ? partial.id : ''
|
||||
const name = partial?.type === 'toolCall' ? partial.name : ''
|
||||
toolIds.set(event.contentIndex, { id, name })
|
||||
yield { type: 'block-start', index: event.contentIndex, blockType: 'tool-call' }
|
||||
break
|
||||
}
|
||||
case 'toolcall_delta': {
|
||||
const known = toolIds.get(event.contentIndex)
|
||||
yield {
|
||||
type: 'tool-call-delta',
|
||||
index: event.contentIndex,
|
||||
id: CallId(known?.id ?? ''),
|
||||
...known?.name !== undefined && known.name.length > 0 ? { name: known.name } : {},
|
||||
argumentsDelta: event.delta,
|
||||
}
|
||||
break
|
||||
}
|
||||
case 'toolcall_end':
|
||||
yield {
|
||||
type: 'block-end',
|
||||
index: event.contentIndex,
|
||||
block: {
|
||||
type: 'tool-call',
|
||||
id: CallId(event.toolCall.id),
|
||||
name: event.toolCall.name,
|
||||
// pi-ai hands back the PARSED arguments; the harness vocabulary
|
||||
// keeps the raw string.
|
||||
arguments: JSON.stringify(event.toolCall.arguments),
|
||||
},
|
||||
}
|
||||
break
|
||||
case 'done':
|
||||
yield { type: 'usage', usage: mapUsage(event.message.usage) }
|
||||
yield {
|
||||
type: 'finish',
|
||||
reason: mapStopReason(event.message, contextWindow),
|
||||
replayState: toPiReplayState(event.message),
|
||||
}
|
||||
return
|
||||
case 'error':
|
||||
// In-stream error delivery (pi-ai's style) → error finish chunk
|
||||
// (the harness's other sanctioned error path besides throwing).
|
||||
yield { type: 'usage', usage: mapUsage(event.error.usage) }
|
||||
yield { type: 'finish', reason: mapStopReason(event.error, contextWindow) }
|
||||
return
|
||||
// no default: AssistantMessageEvent is pi-ai's closed union; a new
|
||||
// event type should fail compilation here via tsc's exhaustiveness
|
||||
// when one is added (switch covers all current variants).
|
||||
}
|
||||
}
|
||||
throw new LlmError('pi-ai event stream ended without done/error', 'STREAM_CLOSED')
|
||||
}
|
||||
@@ -3,26 +3,33 @@ import { Context } from 'cordis'
|
||||
import LlmService, { CallId } from '@deepseek-ai/dsh-llm'
|
||||
import type { Message, ToolSchema } from '@deepseek-ai/dsh-llm'
|
||||
import * as LlmPiAi from '@deepseek-ai/dsh-llm-pi-ai'
|
||||
import type { Config } from '@deepseek-ai/dsh-llm-pi-ai'
|
||||
import type { PiAiProviderProfile } from '@deepseek-ai/dsh-llm-pi-ai'
|
||||
import * as LlmDeepSeek from '@deepseek-ai/dsh-llm-deepseek'
|
||||
import { assemble, type AssembledResult } from './assemble.ts'
|
||||
|
||||
/**
|
||||
* Real-API e2e for the pi-ai-backed adapter: V4 Flash + V4 Pro across all
|
||||
* reasoning levels the adapter exposes (off / high / xhigh→wire 'max').
|
||||
* Mirrors the llm-deepseek matrix so the two independent implementations
|
||||
* verify the same StreamChunk contract. Key-gated.
|
||||
* Real-API e2e for the pi-ai-backed adapter: V4 Flash + V4 Pro with provider
|
||||
* defaults and representative high/xhigh reasoning. Mirrors the native
|
||||
* adapter's StreamChunk contract and exercises a replayed tool follow-up.
|
||||
* Key-gated.
|
||||
*/
|
||||
|
||||
const FLASH = 'deepseek-v4-flash'
|
||||
const PRO = 'deepseek-v4-pro'
|
||||
const contexts: Context[] = []
|
||||
|
||||
async function harness(model: string, config: Partial<Config> = {}) {
|
||||
async function harness(_model: string, config: Partial<PiAiProviderProfile> = {}) {
|
||||
const ctx = new Context()
|
||||
contexts.push(ctx)
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmPiAi, { models: [model], ...config })
|
||||
await ctx.plugin(LlmPiAi, {
|
||||
providers: [{
|
||||
provider: 'deepseek',
|
||||
...process.env.DEEPSEEK_API_KEY === undefined ? {} : { apiKey: process.env.DEEPSEEK_API_KEY },
|
||||
...process.env.DEEPSEEK_BASE_URL === undefined ? {} : { baseURL: process.env.DEEPSEEK_BASE_URL },
|
||||
...config,
|
||||
}],
|
||||
})
|
||||
return ctx
|
||||
}
|
||||
|
||||
@@ -56,8 +63,8 @@ const weatherTool: ToolSchema = {
|
||||
}
|
||||
|
||||
describe.skipIf(!process.env.DEEPSEEK_API_KEY)('llm-pi-ai e2e (real API)', () => {
|
||||
it.each([FLASH, PRO])('%s + reasoning off: plain text generation', async (model) => {
|
||||
const ctx = await harness(model, { reasoning: 'off' })
|
||||
it.each([FLASH, PRO])('%s + provider-default reasoning: plain text generation', async (model) => {
|
||||
const ctx = await harness(model)
|
||||
const result = await assemble(ctx,{
|
||||
model,
|
||||
messages: ask('Reply with exactly the word: pong'),
|
||||
@@ -65,7 +72,6 @@ describe.skipIf(!process.env.DEEPSEEK_API_KEY)('llm-pi-ai e2e (real API)', () =>
|
||||
})
|
||||
expect(result.finish.kind).toBe('stop')
|
||||
expect(textOf(result).toLowerCase()).toContain('pong')
|
||||
expect(result.message.content.some(block => block.type === 'reasoning')).toBe(false)
|
||||
})
|
||||
|
||||
it.each([FLASH, PRO])('%s + reasoning high: reasoning blocks present', async (model) => {
|
||||
@@ -99,7 +105,7 @@ describe.skipIf(!process.env.DEEPSEEK_API_KEY)('llm-pi-ai e2e (real API)', () =>
|
||||
model: PRO,
|
||||
messages: [
|
||||
...ask('What is the weather in Paris right now? Use the get_weather tool.'),
|
||||
{ role: 'assistant', content: first.message.content },
|
||||
first.message,
|
||||
{
|
||||
role: 'user',
|
||||
content: [{
|
||||
@@ -123,9 +129,9 @@ describe.skipIf(!process.env.DEEPSEEK_API_KEY)('llm-pi-ai e2e (real API)', () =>
|
||||
const deepseekCtx = new Context()
|
||||
contexts.push(deepseekCtx)
|
||||
await deepseekCtx.plugin(LlmService)
|
||||
await deepseekCtx.plugin(LlmDeepSeek, { models: [FLASH], thinking: 'disabled' })
|
||||
await deepseekCtx.plugin(LlmDeepSeek, { thinking: 'disabled' })
|
||||
|
||||
const piCtx = await harness(FLASH, { reasoning: 'off' })
|
||||
const piCtx = await harness(FLASH)
|
||||
|
||||
const prompt = ask('Reply with exactly the word: pong')
|
||||
const [fromDeepSeek, fromPiAi] = await Promise.all([
|
||||
|
||||
@@ -2,44 +2,69 @@ import { createServer } from 'node:http'
|
||||
import type { IncomingMessage, Server, ServerResponse } from 'node:http'
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import LlmService, { CallId, userAgent } from '@deepseek-ai/dsh-llm'
|
||||
import LlmService, { CONTEXT_WINDOW_EXCEEDED_CODE, LlmError, userAgent } from '@deepseek-ai/dsh-llm'
|
||||
import * as LlmPiAi from '@deepseek-ai/dsh-llm-pi-ai'
|
||||
import { buildModel, PiAiAdapter } from '@deepseek-ai/dsh-llm-pi-ai'
|
||||
import { PiAiAdapter } from '@deepseek-ai/dsh-llm-pi-ai'
|
||||
import { getModels } from '@earendil-works/pi-ai'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
import { resolveProfiles } from '../src/config.ts'
|
||||
import { assemble } from './assemble.ts'
|
||||
|
||||
/** Scripted SSE responses, one per request (OpenAI chat-completions shape). */
|
||||
interface MockServer {
|
||||
url: string
|
||||
paths: string[]
|
||||
requests: unknown[]
|
||||
/** Header bags of received requests, in order (parallel to `requests`). */
|
||||
headers: IncomingMessage['headers'][]
|
||||
close(): Promise<void>
|
||||
readonly closedResponses: number
|
||||
responseClosed: Promise<void>
|
||||
}
|
||||
|
||||
const servers: Server[] = []
|
||||
|
||||
afterEach(async () => {
|
||||
vi.unstubAllEnvs()
|
||||
await Promise.all(servers.splice(0).map(server => new Promise(resolve => server.close(resolve))))
|
||||
})
|
||||
|
||||
async function mockServer(script: { status?: number; events?: string[]; body?: string }[]): Promise<MockServer> {
|
||||
async function mockServer(script: {
|
||||
status?: number
|
||||
events?: string[]
|
||||
body?: string
|
||||
delayMs?: number
|
||||
headers?: Record<string, string>
|
||||
}[]): Promise<MockServer> {
|
||||
const paths: string[] = []
|
||||
const requests: unknown[] = []
|
||||
const headers: IncomingMessage['headers'][] = []
|
||||
let closedResponses = 0
|
||||
const responseClosed = Promise.withResolvers<undefined>()
|
||||
const server = createServer((request: IncomingMessage, response: ServerResponse) => {
|
||||
response.on('close', () => {
|
||||
closedResponses += 1
|
||||
responseClosed.resolve(undefined)
|
||||
})
|
||||
let body = ''
|
||||
request.on('data', (chunk: Buffer) => { body += chunk.toString('utf8') })
|
||||
request.on('end', () => {
|
||||
requests.push(JSON.parse(body))
|
||||
paths.push(request.url ?? '')
|
||||
requests.push(body.length === 0 ? undefined : JSON.parse(body))
|
||||
headers.push(request.headers)
|
||||
const behavior = script.shift() ?? { status: 500, body: 'script exhausted' }
|
||||
if (behavior.status !== undefined && behavior.status !== 200) {
|
||||
response.writeHead(behavior.status, { 'content-type': 'application/json' })
|
||||
response.writeHead(behavior.status, { 'content-type': 'application/json', ...behavior.headers })
|
||||
response.end(behavior.body ?? '{}')
|
||||
return
|
||||
}
|
||||
response.writeHead(200, { 'content-type': 'text/event-stream' })
|
||||
for (const event of behavior.events ?? []) response.write(`data: ${event}\n\n`)
|
||||
response.end()
|
||||
let index = 0
|
||||
const writeNext = (): void => {
|
||||
const event = behavior.events?.[index++]
|
||||
if (event === undefined) { response.end(); return }
|
||||
response.write(`data: ${event}\n\n`)
|
||||
if (behavior.delayMs === undefined) writeNext()
|
||||
else setTimeout(writeNext, behavior.delayMs)
|
||||
}
|
||||
writeNext()
|
||||
})
|
||||
})
|
||||
servers.push(server)
|
||||
@@ -48,9 +73,11 @@ async function mockServer(script: { status?: number; events?: string[]; body?: s
|
||||
if (address === null || typeof address === 'string') throw new Error('no port')
|
||||
return {
|
||||
url: `http://127.0.0.1:${address.port}`,
|
||||
paths,
|
||||
requests,
|
||||
headers,
|
||||
close: () => new Promise(resolve => server.close(() => { resolve() })),
|
||||
responseClosed: responseClosed.promise,
|
||||
get closedResponses() { return closedResponses },
|
||||
}
|
||||
}
|
||||
|
||||
@@ -61,347 +88,389 @@ const textEvents = [
|
||||
'[DONE]',
|
||||
]
|
||||
|
||||
const toolEvents = [
|
||||
'{"choices":[{"delta":{"role":"assistant","content":null},"index":0,"finish_reason":null}]}',
|
||||
'{"choices":[{"delta":{"tool_calls":[{"index":0,"id":"call-1","type":"function","function":{"name":"get_weather","arguments":""}}]},"index":0,"finish_reason":null}]}',
|
||||
'{"choices":[{"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\\"city\\":\\"Paris\\"}"}}]},"index":0,"finish_reason":null}]}',
|
||||
'{"choices":[{"delta":{},"index":0,"finish_reason":"tool_calls"}],"usage":{"prompt_tokens":20,"completion_tokens":6}}',
|
||||
'[DONE]',
|
||||
]
|
||||
|
||||
const thinkingEvents = [
|
||||
'{"choices":[{"delta":{"role":"assistant","content":null,"reasoning_content":""},"index":0,"finish_reason":null}]}',
|
||||
'{"choices":[{"delta":{"reasoning_content":"pondering"},"index":0,"finish_reason":null}]}',
|
||||
'{"choices":[{"delta":{"content":"answer","reasoning_content":null},"index":0,"finish_reason":null}]}',
|
||||
'{"choices":[{"delta":{},"index":0,"finish_reason":"stop"}],"usage":{"prompt_tokens":5,"completion_tokens":9}}',
|
||||
'[DONE]',
|
||||
]
|
||||
|
||||
async function harness(baseURL: string, config: object = {}) {
|
||||
async function harness(baseURL: string, overrides: Record<string, unknown> = {}): Promise<Context> {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmPiAi, { apiKey: 'test-key', baseURL, models: ['deepseek-v4-flash'], ...config })
|
||||
await ctx.plugin(LlmPiAi, {
|
||||
providers: [{ provider: 'deepseek', apiKey: 'test-key', baseURL, ...overrides }],
|
||||
})
|
||||
return ctx
|
||||
}
|
||||
|
||||
describe('PiAiAdapter against a mock server', () => {
|
||||
it('streams a text generation through the assembler', async () => {
|
||||
describe('PiAiAdapter provider routing', () => {
|
||||
it('resolves a catalog model dynamically and uses a private endpoint', async () => {
|
||||
const server = await mockServer([{ events: textEvents }])
|
||||
const ctx = await harness(server.url)
|
||||
|
||||
const result = await assemble(ctx, {
|
||||
model: 'deepseek-v4-flash',
|
||||
messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }],
|
||||
})
|
||||
expect(result.message.content).toEqual([{ type: 'text', text: 'hello' }])
|
||||
expect(result.finish).toEqual({ kind: 'stop' })
|
||||
expect(result.usage).toMatchObject({ inputTokens: 3, outputTokens: 1 })
|
||||
expect(result.usage).toEqual({ inputTokens: 3, outputTokens: 1 })
|
||||
expect(server.paths).toEqual(['/chat/completions'])
|
||||
})
|
||||
|
||||
// Attribution reaches the wire through pi-ai's headers hook: the exact
|
||||
// shared User-Agent, and no provider-specific headers under the
|
||||
// User-Agent-only contract.
|
||||
it('merges profile headers with Harness attribution winning', async () => {
|
||||
const server = await mockServer([{ events: textEvents }])
|
||||
const ctx = await harness(server.url, {
|
||||
headers: { 'x-company': 'private', 'User-Agent': 'wrong' },
|
||||
})
|
||||
await assemble(ctx, { model: 'deepseek-v4-flash', messages: [] })
|
||||
expect(server.headers[0]?.['x-company']).toBe('private')
|
||||
expect(server.headers[0]?.['user-agent']).toBe(userAgent())
|
||||
expect(server.headers[0]).not.toHaveProperty('http-referer')
|
||||
expect(server.headers[0]).not.toHaveProperty('x-openrouter-title')
|
||||
expect(server.headers[0]).not.toHaveProperty('x-openrouter-categories')
|
||||
})
|
||||
|
||||
it('streams tool calls with re-stringified arguments', async () => {
|
||||
const server = await mockServer([{ events: toolEvents }])
|
||||
const ctx = await harness(server.url)
|
||||
|
||||
const result = await assemble(ctx,{
|
||||
model: 'deepseek-v4-flash',
|
||||
messages: [{ role: 'user', content: [{ type: 'text', text: 'weather?' }] }],
|
||||
tools: [{
|
||||
name: 'get_weather',
|
||||
description: 'Get weather',
|
||||
parameters: { type: 'object', properties: { city: { type: 'string' } } },
|
||||
}],
|
||||
it('forwards common stream options and profile reasoning', async () => {
|
||||
const server = await mockServer([{ events: textEvents }])
|
||||
const ctx = await harness(server.url, {
|
||||
reasoning: 'xhigh',
|
||||
cacheRetention: 'none',
|
||||
transport: 'sse',
|
||||
timeoutMs: 5000,
|
||||
websocketConnectTimeoutMs: 3000,
|
||||
streamIdleTimeoutMs: 10_000,
|
||||
thinkingBudgets: { high: 2048 },
|
||||
})
|
||||
expect(result.finish).toEqual({ kind: 'tool-calls' })
|
||||
const call = result.message.content.find(block => block.type === 'tool-call')
|
||||
expect(call).toMatchObject({ name: 'get_weather', arguments: '{"city":"Paris"}' })
|
||||
})
|
||||
|
||||
it('maps reasoning_content streams to reasoning blocks', async () => {
|
||||
const server = await mockServer([{ events: thinkingEvents }])
|
||||
const ctx = await harness(server.url, { reasoning: 'high' })
|
||||
|
||||
const result = await assemble(ctx,{
|
||||
model: 'deepseek-v4-flash',
|
||||
messages: [{ role: 'user', content: [{ type: 'text', text: 'think' }] }],
|
||||
})
|
||||
expect(result.message.content).toEqual([
|
||||
{ type: 'reasoning', text: 'pondering' },
|
||||
{ type: 'text', text: 'answer' },
|
||||
])
|
||||
})
|
||||
|
||||
it('sends DeepSeek thinking fields when reasoning is configured', async () => {
|
||||
const server = await mockServer([{ events: textEvents }])
|
||||
const ctx = await harness(server.url, { reasoning: 'xhigh' })
|
||||
await assemble(ctx,{ model: 'deepseek-v4-flash', messages: [] })
|
||||
expect(server.requests[0]).toMatchObject({
|
||||
thinking: { type: 'enabled' },
|
||||
reasoning_effort: 'max', // xhigh maps to max via thinkingLevelMap
|
||||
})
|
||||
})
|
||||
|
||||
it('disables thinking for reasoning: off', async () => {
|
||||
const server = await mockServer([{ events: textEvents }])
|
||||
const ctx = await harness(server.url, { reasoning: 'off' })
|
||||
await assemble(ctx,{ model: 'deepseek-v4-flash', messages: [] })
|
||||
expect(server.requests[0]).toMatchObject({ thinking: { type: 'disabled' } })
|
||||
})
|
||||
|
||||
it('injects stop sequences through onPayload', async () => {
|
||||
const server = await mockServer([{ events: textEvents }])
|
||||
const ctx = await harness(server.url)
|
||||
await assemble(ctx,{ model: 'deepseek-v4-flash', messages: [], stop: ['END'] })
|
||||
expect(server.requests[0]).toMatchObject({ stop: ['END'] })
|
||||
})
|
||||
|
||||
it('scrubs pi-ai\'s own per-tool strict default through onPayload', async () => {
|
||||
const server = await mockServer([{ events: textEvents }])
|
||||
const ctx = await harness(server.url)
|
||||
await assemble(ctx,{
|
||||
await assemble(ctx, {
|
||||
model: 'deepseek-v4-flash',
|
||||
messages: [],
|
||||
tools: [
|
||||
{ name: 'alpha', description: 'a', parameters: {} },
|
||||
{ name: 'beta', description: 'b', parameters: {} },
|
||||
],
|
||||
temperature: 0.2,
|
||||
maxTokens: 77,
|
||||
sessionId: 'session-for-pi' as never,
|
||||
})
|
||||
expect(server.requests[0]).toMatchObject({
|
||||
model: 'deepseek-v4-flash',
|
||||
temperature: 0.2,
|
||||
max_completion_tokens: 77,
|
||||
thinking: { type: 'enabled' },
|
||||
reasoning_effort: 'max',
|
||||
})
|
||||
|
||||
// pi-ai stamps `strict` on every serialized tool function; the harness
|
||||
// contract has none and the hand-rolled twin sends no such field, so the
|
||||
// payload fixup must have deleted it from every tool.
|
||||
const request = server.requests[0] as { tools: { function: { name: string; strict?: boolean } }[] }
|
||||
expect(request.tools.map(tool => tool.function.name)).toEqual(['alpha', 'beta'])
|
||||
for (const tool of request.tools) {
|
||||
expect('strict' in tool.function).toBe(false)
|
||||
}
|
||||
})
|
||||
|
||||
it('preserves raw replayed tool-call arguments in the provider payload', async () => {
|
||||
it('preserves omitted profile options when constructing the adapter directly', async () => {
|
||||
const server = await mockServer([{ events: textEvents }])
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['deepseek'], new PiAiAdapter({
|
||||
profiles: [{ provider: 'deepseek', apiKey: 'test-key', baseURL: server.url }],
|
||||
}))
|
||||
|
||||
const result = await assemble(ctx, { model: 'deepseek-v4-flash', messages: [] })
|
||||
|
||||
expect(result.message.content).toEqual([{ type: 'text', text: 'hello' }])
|
||||
})
|
||||
|
||||
it('rejects stop sequences rather than silently ignoring them', async () => {
|
||||
const server = await mockServer([])
|
||||
const ctx = await harness(server.url)
|
||||
await assemble(ctx,{
|
||||
model: 'deepseek-v4-flash',
|
||||
messages: [{
|
||||
role: 'assistant',
|
||||
content: [{ type: 'tool-call', id: CallId('broken'), name: 'f', arguments: '{broken' }],
|
||||
await expect(assemble(ctx, { model: 'deepseek-v4-flash', messages: [], stop: ['END'] }))
|
||||
.rejects.toMatchObject({ code: 'UNSUPPORTED_OPTION' })
|
||||
expect(server.requests).toEqual([])
|
||||
})
|
||||
|
||||
it('rejects unknown catalog models before network I/O', async () => {
|
||||
const server = await mockServer([])
|
||||
const ctx = await harness(server.url)
|
||||
await expect(assemble(ctx, { model: 'not-in-the-catalog', messages: [] }))
|
||||
.rejects.toMatchObject({ code: 'UNKNOWN_MODEL' })
|
||||
expect(server.requests).toEqual([])
|
||||
})
|
||||
|
||||
it('uses the catalog API implementation, including OpenAI Responses', async () => {
|
||||
const server = await mockServer([{ status: 401, body: JSON.stringify({ error: { message: 'expected mock failure' } }) }])
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmPiAi, {
|
||||
providers: [{ provider: 'openai', apiKey: 'test-key', baseURL: `${server.url}/v1` }],
|
||||
})
|
||||
const result = await assemble(ctx, { provider: 'openai', model: 'gpt-4.1', messages: [] })
|
||||
expect(result.finish.kind).toBe('error')
|
||||
expect(server.paths).toEqual(['/v1/responses'])
|
||||
})
|
||||
|
||||
it('forces one wire request for an SDK-retryable provider failure', async () => {
|
||||
const server = await mockServer([
|
||||
{
|
||||
status: 429,
|
||||
headers: { 'retry-after-ms': '1' },
|
||||
body: JSON.stringify({ error: { message: 'retryable provider failure' } }),
|
||||
},
|
||||
{ status: 500, body: JSON.stringify({ error: { message: 'hidden SDK retry' } }) },
|
||||
{ status: 500, body: JSON.stringify({ error: { message: 'second hidden SDK retry' } }) },
|
||||
])
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmPiAi, {
|
||||
providers: [{ provider: 'openai', apiKey: 'test-key', baseURL: `${server.url}/v1` }],
|
||||
})
|
||||
|
||||
const result = await assemble(ctx, { provider: 'openai', model: 'gpt-4.1', messages: [] })
|
||||
|
||||
expect(result.finish).toMatchObject({ kind: 'error' })
|
||||
expect(server.paths).toEqual(['/v1/responses'])
|
||||
})
|
||||
|
||||
it('uses OpenAI Responses against an Azure project v1 path with its API key header', async () => {
|
||||
const server = await mockServer([{ status: 401, body: JSON.stringify({ error: { message: 'expected mock failure' } }) }])
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmPiAi, {
|
||||
providers: [{
|
||||
provider: 'openai',
|
||||
apiKey: 'test-key',
|
||||
baseURL: `${server.url}/api/projects/openai/openai/v1`,
|
||||
headers: { 'api-key': 'test-key', Authorization: '' },
|
||||
}],
|
||||
})
|
||||
|
||||
const request = server.requests[0] as { messages: { role: string; tool_calls?: { id: string; function: { arguments: string } }[] }[] }
|
||||
const assistant = request.messages.find(message => message.role === 'assistant')
|
||||
expect(assistant?.tool_calls?.[0]?.function.arguments).toBe('{broken')
|
||||
})
|
||||
|
||||
it('maps HTTP errors to error finish chunks (pi-ai in-stream style)', async () => {
|
||||
const server = await mockServer([{
|
||||
status: 401,
|
||||
body: JSON.stringify({ error: { message: 'bad key' } }),
|
||||
}])
|
||||
const ctx = await harness(server.url)
|
||||
const result = await assemble(ctx,{ model: 'deepseek-v4-flash', messages: [] })
|
||||
expect(result.finish).toMatchObject({ kind: 'error', code: 'AUTH' })
|
||||
expect((result.finish as { message: string }).message).toMatch(/bad key|401/)
|
||||
const result = await assemble(ctx, { provider: 'openai', model: 'gpt-5.5', messages: [] })
|
||||
expect(result.finish.kind).toBe('error')
|
||||
expect(server.paths).toEqual(['/api/projects/openai/openai/v1/responses'])
|
||||
expect(server.headers[0]?.['api-key']).toBe('test-key')
|
||||
expect(server.headers[0]?.authorization).toBe('')
|
||||
})
|
||||
|
||||
it.each([
|
||||
[401, 'AUTH'],
|
||||
[400, 'INVALID_REQUEST'],
|
||||
[429, 'RATE_LIMIT'],
|
||||
[500, 'SERVER'],
|
||||
] as const)('maps HTTP %s to stable error code %s', async (status, code) => {
|
||||
] as const)('maps HTTP %s failures to %s', async (status, code) => {
|
||||
const server = await mockServer([{ status, body: JSON.stringify({ error: { message: `provider ${status}` } }) }])
|
||||
const ctx = await harness(server.url)
|
||||
const result = await assemble(ctx,{ model: 'deepseek-v4-flash', messages: [] })
|
||||
expect(result.finish).toMatchObject({ kind: 'error', code })
|
||||
const result = await assemble(ctx, { model: 'deepseek-v4-flash', messages: [] })
|
||||
expect(result.finish).toMatchObject({ kind: 'error', failure: { code } })
|
||||
expect(server.paths).toEqual(['/chat/completions'])
|
||||
})
|
||||
|
||||
it('registers/unregisters models on the llm service (HMR safety)', async () => {
|
||||
it('uses the resolved catalog context window for usage-based overflow detection', async () => {
|
||||
const model = getModels('deepseek').find(candidate => candidate.id === 'deepseek-v4-flash')
|
||||
if (model === undefined) throw new Error('deepseek-v4-flash missing from pi-ai test catalog')
|
||||
const events = [
|
||||
'{"choices":[{"delta":{"role":"assistant","content":""},"index":0,"finish_reason":null}]}',
|
||||
JSON.stringify({
|
||||
choices: [{ delta: {}, index: 0, finish_reason: 'stop' }],
|
||||
usage: { prompt_tokens: model.contextWindow + 1, completion_tokens: 0 },
|
||||
}),
|
||||
'[DONE]',
|
||||
]
|
||||
const server = await mockServer([{ events }])
|
||||
const ctx = await harness(server.url)
|
||||
|
||||
const result = await assemble(ctx, { model: model.id, messages: [] })
|
||||
|
||||
expect(result.finish).toEqual({
|
||||
kind: 'error',
|
||||
failure: {
|
||||
message: `pi-ai detected context overflow for model "${model.id}"`,
|
||||
code: CONTEXT_WINDOW_EXCEEDED_CODE,
|
||||
},
|
||||
})
|
||||
})
|
||||
|
||||
it('stops the SDK request when the adapter idle watchdog expires', async () => {
|
||||
const server = await mockServer([{ events: textEvents, delayMs: 200 }])
|
||||
const ctx = await harness(server.url, { streamIdleTimeoutMs: 20 })
|
||||
|
||||
await expect(assemble(ctx, { model: 'deepseek-v4-flash', messages: [] }))
|
||||
.rejects.toMatchObject({ code: 'TIMEOUT' })
|
||||
await Promise.race([
|
||||
server.responseClosed,
|
||||
new Promise<never>((_resolve, reject) => {
|
||||
setTimeout(() => { reject(new Error('SDK request did not close after idle timeout')) }, 100)
|
||||
}),
|
||||
])
|
||||
|
||||
expect(server.paths).toEqual(['/chat/completions'])
|
||||
expect(server.closedResponses).toBe(1)
|
||||
})
|
||||
})
|
||||
|
||||
describe('provider profile lifecycle', () => {
|
||||
it('keeps adapter helpers off the package root', () => {
|
||||
for (const helper of [
|
||||
'resolveProfiles',
|
||||
'toPiContext',
|
||||
'toPiReplayState',
|
||||
'toPiAssistant',
|
||||
'mapStopReason',
|
||||
'mapUsage',
|
||||
'toStreamChunks',
|
||||
]) expect(LlmPiAi).not.toHaveProperty(helper)
|
||||
})
|
||||
|
||||
it('registers every profile atomically and unregisters on dispose', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
const fiber = await ctx.plugin(LlmPiAi, { apiKey: 'k', baseURL: 'http://127.0.0.1:1' })
|
||||
expect(ctx.llm.models().sort()).toEqual(['deepseek-v4-flash', 'deepseek-v4-pro'])
|
||||
const fiber = await ctx.plugin(LlmPiAi, {
|
||||
providers: [{ provider: 'openai' }, { provider: 'anthropic' }],
|
||||
})
|
||||
expect(ctx.llm.listProviders()).toEqual([
|
||||
{ id: 'openai', name: 'openai' },
|
||||
{ id: 'anthropic', name: 'anthropic' },
|
||||
])
|
||||
await fiber.dispose()
|
||||
expect(ctx.llm.models()).toEqual([])
|
||||
expect(ctx.llm.listProviders()).toEqual([])
|
||||
})
|
||||
|
||||
it('throws a clear error when no API key is available', async () => {
|
||||
const previous = process.env.DEEPSEEK_API_KEY
|
||||
delete process.env.DEEPSEEK_API_KEY
|
||||
try {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await expect(ctx.plugin(LlmPiAi, {})).rejects.toThrow(/an API key is required/)
|
||||
} finally {
|
||||
if (previous !== undefined) process.env.DEEPSEEK_API_KEY = previous
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('option spreads and env fallbacks', () => {
|
||||
it('forwards temperature, maxTokens, and signal', async () => {
|
||||
const server = await mockServer([{ events: textEvents }])
|
||||
const ctx = await harness(server.url)
|
||||
const controller = new AbortController()
|
||||
await assemble(ctx,{
|
||||
model: 'deepseek-v4-flash',
|
||||
messages: [],
|
||||
temperature: 0.5,
|
||||
maxTokens: 40,
|
||||
signal: controller.signal,
|
||||
it('exposes the installed pi-ai model catalog through provider-neutral metadata', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmPiAi, { providers: [{ provider: 'openai' }] })
|
||||
const models = await ctx.llm.listModels('openai')
|
||||
expect(models.find(model => model.id === 'gpt-4.1')).toEqual({
|
||||
provider: 'openai', id: 'gpt-4.1', name: 'GPT-4.1',
|
||||
})
|
||||
expect(server.requests[0]).toMatchObject({ temperature: 0.5, max_tokens: 40 })
|
||||
expect(models.every(model => model.provider === 'openai')).toBe(true)
|
||||
})
|
||||
|
||||
it('falls back to DEEPSEEK_API_KEY / DEEPSEEK_BASE_URL env vars', async () => {
|
||||
it('accepts absent credentials for pi-ai ambient authentication', async () => {
|
||||
vi.stubEnv('DEEPSEEK_API_KEY', 'ambient-key')
|
||||
const server = await mockServer([{ events: textEvents }])
|
||||
vi.stubEnv('DEEPSEEK_API_KEY', 'env-key')
|
||||
vi.stubEnv('DEEPSEEK_BASE_URL', server.url)
|
||||
try {
|
||||
const ctx = await harness(server.url, { apiKey: undefined })
|
||||
await assemble(ctx, { model: 'deepseek-v4-flash', messages: [] })
|
||||
expect(server.headers[0]?.authorization).toBe('Bearer ambient-key')
|
||||
})
|
||||
|
||||
it('validates empty, duplicate, unknown, and explicitly blank profiles', () => {
|
||||
expect(() => resolveProfiles([])).toThrow(/at least one/)
|
||||
expect(() => resolveProfiles([{ provider: '' }])).toThrow(/non-empty/)
|
||||
expect(() => resolveProfiles([{ provider: 'not-real' }])).toThrow(/unknown/)
|
||||
expect(() => resolveProfiles([{ provider: 'openai' }, { provider: 'openai' }])).toThrow(/duplicate/)
|
||||
expect(() => resolveProfiles([{ provider: 'openai', apiKey: '' }])).toThrow(/empty apiKey/)
|
||||
expect(() => resolveProfiles([{ provider: 'openai', apiKey: ' ' }])).toThrow(/empty apiKey/)
|
||||
expect(() => resolveProfiles([{ provider: 'openai', baseURL: '' }])).toThrow(/empty baseURL/)
|
||||
})
|
||||
|
||||
it.each(['maxRetries', 'maxRetryDelayMs'] as const)(
|
||||
'rejects removed profile field %s instead of silently restoring hidden SDK retries',
|
||||
async (field) => {
|
||||
const legacy = { provider: 'openai', [field]: 2 }
|
||||
expect(() => resolveProfiles([legacy as never])).toThrow(/removed.*agent recovery/i)
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmPiAi, { models: ['deepseek-v4-flash'] })
|
||||
await assemble(ctx,{ model: 'deepseek-v4-flash', messages: [] })
|
||||
expect(server.requests).toHaveLength(1)
|
||||
} finally {
|
||||
vi.unstubAllEnvs()
|
||||
await expect(ctx.plugin(LlmPiAi, { providers: [legacy as never] }))
|
||||
.rejects.toThrow(/removed.*agent recovery/i)
|
||||
},
|
||||
)
|
||||
|
||||
it('rejects invalid stream tunables at plugin load', async () => {
|
||||
const invalid = [
|
||||
{ timeoutMs: -1 },
|
||||
{ websocketConnectTimeoutMs: -1 },
|
||||
{ streamIdleTimeoutMs: 0 },
|
||||
{ streamIdleTimeoutMs: Number.NaN },
|
||||
{ streamIdleTimeoutMs: MAX_TIMER_DELAY_MS + 1 },
|
||||
]
|
||||
for (const entry of invalid) {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await expect(ctx.plugin(LlmPiAi, { providers: [{ provider: 'openai', ...entry }] }))
|
||||
.rejects.toThrow()
|
||||
}
|
||||
})
|
||||
|
||||
it('defaults to the public base URL without config or env', async () => {
|
||||
vi.stubEnv('DEEPSEEK_API_KEY', 'k')
|
||||
vi.stubEnv('DEEPSEEK_BASE_URL', undefined)
|
||||
try {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmPiAi, {})
|
||||
expect(ctx.llm.models().length).toBeGreaterThan(0)
|
||||
} finally {
|
||||
vi.unstubAllEnvs()
|
||||
}
|
||||
it('constructs the adapter directly and rejects routes it does not own', async () => {
|
||||
const adapter = new PiAiAdapter({ profiles: [{ provider: 'openai' }] })
|
||||
await expect(adapter.listModels('anthropic')).rejects.toMatchObject({ code: 'NO_ADAPTER' })
|
||||
await expect((async () => {
|
||||
for await (const _chunk of adapter.stream({ provider: 'anthropic', model: 'claude-sonnet-4', messages: [] })) { /* drain */ }
|
||||
})()).rejects.toMatchObject({ code: 'NO_ADAPTER' })
|
||||
expect(new LlmError('x', 'X')).toBeInstanceOf(Error)
|
||||
})
|
||||
|
||||
it('validates direct-constructor profiles at the embedding boundary', () => {
|
||||
expect(() => new PiAiAdapter({
|
||||
profiles: [{ provider: 'openai', streamIdleTimeoutMs: 0 }],
|
||||
})).toThrow(/streamIdleTimeoutMs.*positive finite/)
|
||||
expect(() => new PiAiAdapter({
|
||||
profiles: [{ provider: 'openai', streamIdleTimeoutMs: MAX_TIMER_DELAY_MS + 1 }],
|
||||
})).toThrow(/streamIdleTimeoutMs.*no greater/)
|
||||
})
|
||||
})
|
||||
|
||||
describe('buildModel', () => {
|
||||
it('builds a DeepSeek-compat openai-completions model descriptor', () => {
|
||||
const model = buildModel('deepseek-v4-pro', { apiKey: 'k', baseURL: 'http://x', reasoning: 'high' })
|
||||
expect(model).toMatchObject({
|
||||
id: 'deepseek-v4-pro',
|
||||
api: 'openai-completions',
|
||||
describe('abort wiring', () => {
|
||||
it('preserves an unknown pre-dispatch adapter Error exactly', async () => {
|
||||
const original = new Error('SDK context conversion exploded')
|
||||
const message = Object.defineProperty({}, 'role', {
|
||||
get() { throw original },
|
||||
})
|
||||
const adapter = new PiAiAdapter({ profiles: [{ provider: 'deepseek', apiKey: 'test-key' }] })
|
||||
const drain = async (): Promise<void> => {
|
||||
for await (const _chunk of adapter.stream({
|
||||
provider: 'deepseek',
|
||||
model: 'deepseek-v4-flash',
|
||||
messages: [message as never],
|
||||
})) { /* drain */ }
|
||||
}
|
||||
|
||||
await expect(drain()).rejects.toBe(original)
|
||||
})
|
||||
|
||||
it('lets a concurrent caller abort classify a pre-dispatch adapter failure', async () => {
|
||||
const controller = new AbortController()
|
||||
const original = new Error('conversion lost its caller')
|
||||
const message = Object.defineProperty({}, 'role', {
|
||||
get() {
|
||||
controller.abort('caller cancelled during conversion')
|
||||
throw original
|
||||
},
|
||||
})
|
||||
const adapter = new PiAiAdapter({ profiles: [{ provider: 'deepseek', apiKey: 'test-key' }] })
|
||||
const drain = async (): Promise<void> => {
|
||||
for await (const _chunk of adapter.stream({
|
||||
provider: 'deepseek',
|
||||
model: 'deepseek-v4-flash',
|
||||
messages: [message as never],
|
||||
signal: controller.signal,
|
||||
})) { /* drain */ }
|
||||
}
|
||||
|
||||
await expect(drain()).rejects.toMatchObject({ code: 'ABORTED', cause: original })
|
||||
})
|
||||
|
||||
it('resolves catalog endpoints without an override before honoring pre-abort', async () => {
|
||||
const adapter = new PiAiAdapter({ profiles: [{ provider: 'deepseek', apiKey: 'test-key' }] })
|
||||
const controller = new AbortController()
|
||||
controller.abort('already stopped')
|
||||
const chunks = []
|
||||
for await (const chunk of adapter.stream({
|
||||
provider: 'deepseek',
|
||||
baseUrl: 'http://x',
|
||||
reasoning: true,
|
||||
compat: { thinkingFormat: 'deepseek', requiresReasoningContentOnAssistantMessages: true },
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps reasoning true even for off (pi-ai gates the thinking field on it)', () => {
|
||||
// 'off' yields {thinking: {type: 'disabled'}} on the wire — pi-ai only
|
||||
// emits the field at all when model.reasoning is true.
|
||||
expect(buildModel('m', { apiKey: 'k', baseURL: 'http://x', reasoning: 'off' }).reasoning).toBe(true)
|
||||
})
|
||||
|
||||
it('adapter is constructible directly for embedding', () => {
|
||||
expect(new PiAiAdapter({ apiKey: 'k', baseURL: 'http://x' })).toBeInstanceOf(PiAiAdapter)
|
||||
})
|
||||
})
|
||||
|
||||
describe('provider reasoning, passback, and early-stream cancellation', () => {
|
||||
it('defaults omitted reasoning config to thinking ENABLED (provider default)', async () => {
|
||||
const server = await mockServer([{ events: textEvents }])
|
||||
const ctx = await harness(server.url) // no reasoning key at all
|
||||
await assemble(ctx,{ model: 'deepseek-v4-flash', messages: [] })
|
||||
const request = server.requests[0] as Record<string, unknown>
|
||||
expect(request.thinking).toEqual({ type: 'enabled' })
|
||||
expect('reasoning_effort' in request).toBe(false)
|
||||
})
|
||||
|
||||
it('replays reasoning_content on assistant tool-call turns (passback rule)', async () => {
|
||||
const server = await mockServer([{ events: textEvents }])
|
||||
const ctx = await harness(server.url)
|
||||
await assemble(ctx,{
|
||||
model: 'deepseek-v4-flash',
|
||||
messages: [
|
||||
{ role: 'user', content: [{ type: 'text', text: 'weather?' }] },
|
||||
{
|
||||
role: 'assistant',
|
||||
content: [
|
||||
{ type: 'reasoning', text: 'I should check.' },
|
||||
{ type: 'tool-call', id: CallId('c1'), name: 'get_weather', arguments: '{"city":"Paris"}' },
|
||||
],
|
||||
},
|
||||
{
|
||||
role: 'user',
|
||||
content: [{ type: 'tool-result', toolCallId: CallId('c1'), content: [{ type: 'text', text: 'Sunny' }] }],
|
||||
},
|
||||
],
|
||||
})
|
||||
const request = server.requests[0] as { messages: { role: string; reasoning_content?: string }[] }
|
||||
const assistant = request.messages.find(message => message.role === 'assistant')
|
||||
expect(assistant?.reasoning_content).toBe('I should check.')
|
||||
})
|
||||
|
||||
it('aborts the upstream request when the consumer stops streaming early', async () => {
|
||||
// Slow server: write one chunk, then hold the connection open and record
|
||||
// whether the socket closes (the adapter must cancel on early break).
|
||||
let socketClosed = false
|
||||
const server = createServer((request: IncomingMessage, response: ServerResponse) => {
|
||||
request.on('data', () => undefined)
|
||||
request.on('end', () => {
|
||||
response.writeHead(200, { 'content-type': 'text/event-stream' })
|
||||
response.write(`data: ${textEvents[0]}\n\n`)
|
||||
response.write(`data: ${textEvents[1]}\n\n`)
|
||||
// never finish; rely on client abort
|
||||
request.socket.on('close', () => { socketClosed = true })
|
||||
})
|
||||
})
|
||||
servers.push(server)
|
||||
await new Promise<void>(resolve => server.listen(0, '127.0.0.1', resolve))
|
||||
const address = server.address()
|
||||
if (address === null || typeof address === 'string') throw new Error('no port')
|
||||
const ctx = await harness(`http://127.0.0.1:${address.port}`)
|
||||
|
||||
for await (const chunk of ctx.llm.stream({ model: 'deepseek-v4-flash', messages: [] })) {
|
||||
if (chunk.type === 'text-delta') break // stop early mid-stream
|
||||
}
|
||||
// The finally-abort must reach the server as a closed socket.
|
||||
await vi.waitFor(() => { expect(socketClosed).toBe(true) }, { timeout: 5_000 })
|
||||
})
|
||||
})
|
||||
|
||||
describe('caller cancellation', () => {
|
||||
it('honors a pre-aborted caller signal', async () => {
|
||||
const ctx = await harness('http://127.0.0.1:1')
|
||||
const controller = new AbortController()
|
||||
controller.abort('already cancelled')
|
||||
// pi-ai surfaces the abort as an in-stream error event → aborted finish.
|
||||
const result = await assemble(ctx,{
|
||||
model: 'deepseek-v4-flash',
|
||||
messages: [],
|
||||
signal: controller.signal,
|
||||
})
|
||||
})) chunks.push(chunk)
|
||||
expect(chunks.at(-1)).toMatchObject({ type: 'finish', reason: { kind: 'aborted' } })
|
||||
})
|
||||
|
||||
it('honors a pre-aborted caller signal', async () => {
|
||||
const server = await mockServer([{ events: textEvents, delayMs: 20 }])
|
||||
const ctx = await harness(server.url)
|
||||
const controller = new AbortController()
|
||||
controller.abort('already stopped')
|
||||
const result = await assemble(ctx, { model: 'deepseek-v4-flash', messages: [], signal: controller.signal })
|
||||
expect(result.finish.kind).toBe('aborted')
|
||||
})
|
||||
|
||||
it('propagates a mid-stream caller abort to the upstream request', async () => {
|
||||
const server = await mockServer([{ events: textEvents }])
|
||||
it('forwards an abort that arrives while provider streaming is active', async () => {
|
||||
const server = await mockServer([{ events: textEvents, delayMs: 30 }])
|
||||
const ctx = await harness(server.url)
|
||||
const controller = new AbortController()
|
||||
const pending = assemble(ctx,{
|
||||
model: 'deepseek-v4-flash',
|
||||
messages: [],
|
||||
signal: controller.signal,
|
||||
const resultPromise = assemble(ctx, {
|
||||
model: 'deepseek-v4-flash', messages: [], signal: controller.signal,
|
||||
})
|
||||
controller.abort()
|
||||
const result = await pending
|
||||
// Either the abort lands before any chunk (aborted) or after the tiny
|
||||
// mock stream finished (stop) — both are valid races; never a hang.
|
||||
expect(['aborted', 'stop']).toContain(result.finish.kind)
|
||||
setTimeout(() => { controller.abort('stopped during stream') }, 10)
|
||||
const result = await resultPromise
|
||||
expect(result.finish.kind).toBe('aborted')
|
||||
})
|
||||
|
||||
it('aborts upstream when a consumer stops early', async () => {
|
||||
const server = await mockServer([{ events: textEvents, delayMs: 30 }])
|
||||
const ctx = await harness(server.url)
|
||||
for await (const chunk of ctx.llm.stream({ provider: 'deepseek', model: 'deepseek-v4-flash', messages: [] })) {
|
||||
if (chunk.type === 'block-start') break
|
||||
}
|
||||
await new Promise(resolve => setTimeout(resolve, 20))
|
||||
expect(server.requests).toHaveLength(1)
|
||||
})
|
||||
})
|
||||
|
||||
@@ -15,11 +15,19 @@ export interface AssembledResult {
|
||||
finish: FinishReason
|
||||
}
|
||||
|
||||
export async function assemble(ctx: Context, options: GenerateOptions): Promise<AssembledResult> {
|
||||
export async function assemble(ctx: Context, options: Omit<GenerateOptions, 'provider'> & { provider?: string }): Promise<AssembledResult> {
|
||||
const assembler = new BlockAssembler()
|
||||
for await (const chunk of ctx.llm.stream(options)) assembler.push(chunk)
|
||||
const request = { provider: 'deepseek', ...options }
|
||||
for await (const chunk of ctx.llm.stream(request)) assembler.push(chunk)
|
||||
return {
|
||||
message: assembler.message(),
|
||||
message: {
|
||||
...assembler.message(),
|
||||
provenance: {
|
||||
provider: request.provider,
|
||||
model: request.model,
|
||||
...assembler.replayState === undefined ? {} : { replayState: assembler.replayState },
|
||||
},
|
||||
},
|
||||
...assembler.usage !== undefined ? { usage: assembler.usage } : {},
|
||||
finish: assembler.finish,
|
||||
}
|
||||
|
||||
@@ -1,8 +1,10 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { CallId } from '@deepseek-ai/dsh-llm'
|
||||
import { CallId, CONTEXT_WINDOW_EXCEEDED_CODE, LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import type { ContentBlock, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import type { AssistantMessage, AssistantMessageEvent, Usage } from '@earendil-works/pi-ai'
|
||||
import { mapStopReason, mapUsage, toPiContext, toStreamChunks } from '@deepseek-ai/dsh-llm-pi-ai'
|
||||
import { toPiContext } from '../src/context.ts'
|
||||
import { toPiReplayState } from '../src/replay.ts'
|
||||
import { mapStopReason, mapUsage, toStreamChunks } from '../src/stream.ts'
|
||||
|
||||
function usage(input = 0, output = 0, cacheRead = 0, cacheWrite = 0): Usage {
|
||||
return {
|
||||
@@ -42,6 +44,7 @@ async function collect(stream: AsyncIterable<StreamChunk>): Promise<StreamChunk[
|
||||
describe('toPiContext', () => {
|
||||
it('maps system prompt, user text, and tools', () => {
|
||||
const context = toPiContext({
|
||||
provider: 'deepseek',
|
||||
model: 'deepseek-v4-flash',
|
||||
system: 'be helpful',
|
||||
messages: [{ role: 'user', content: [{ type: 'text', text: 'hi' }] }],
|
||||
@@ -55,13 +58,14 @@ describe('toPiContext', () => {
|
||||
})
|
||||
|
||||
it('omits empty tools and absent system prompt', () => {
|
||||
const context = toPiContext({ model: 'm', messages: [], tools: [] })
|
||||
const context = toPiContext({ provider: 'deepseek', model: 'm', messages: [], tools: [] })
|
||||
expect(context.systemPrompt).toBeUndefined()
|
||||
expect(context.tools).toBeUndefined()
|
||||
})
|
||||
|
||||
it('maps assistant text/reasoning/tool-call blocks', () => {
|
||||
const context = toPiContext({
|
||||
provider: 'deepseek',
|
||||
model: 'm',
|
||||
messages: [{
|
||||
role: 'assistant',
|
||||
@@ -76,8 +80,7 @@ describe('toPiContext', () => {
|
||||
expect(message.role).toBe('assistant')
|
||||
expect(message.stopReason).toBe('toolUse')
|
||||
expect(message.content).toEqual([
|
||||
// thinkingSignature names the replay field — DeepSeek's passback rule.
|
||||
{ type: 'thinking', thinking: 'hmm', thinkingSignature: 'reasoning_content' },
|
||||
{ type: 'thinking', thinking: 'hmm' },
|
||||
{ type: 'text', text: 'calling' },
|
||||
{ type: 'toolCall', id: 'c1', name: 'f', arguments: { a: 1 } },
|
||||
])
|
||||
@@ -85,6 +88,7 @@ describe('toPiContext', () => {
|
||||
|
||||
it('marks tool-call-free assistant messages with stopReason stop', () => {
|
||||
const context = toPiContext({
|
||||
provider: 'deepseek',
|
||||
model: 'm',
|
||||
messages: [{ role: 'assistant', content: [{ type: 'text', text: 'done' }] }],
|
||||
})
|
||||
@@ -93,6 +97,7 @@ describe('toPiContext', () => {
|
||||
|
||||
it('parses malformed tool-call arguments to {}', () => {
|
||||
const context = toPiContext({
|
||||
provider: 'deepseek',
|
||||
model: 'm',
|
||||
messages: [{
|
||||
role: 'assistant',
|
||||
@@ -105,6 +110,7 @@ describe('toPiContext', () => {
|
||||
|
||||
it('parses non-object argument JSON (arrays, scalars) to {}', () => {
|
||||
const context = toPiContext({
|
||||
provider: 'deepseek',
|
||||
model: 'm',
|
||||
messages: [{
|
||||
role: 'assistant',
|
||||
@@ -116,6 +122,7 @@ describe('toPiContext', () => {
|
||||
|
||||
it('recovers toolName for tool results from the preceding assistant call', () => {
|
||||
const context = toPiContext({
|
||||
provider: 'deepseek',
|
||||
model: 'm',
|
||||
messages: [
|
||||
{
|
||||
@@ -140,6 +147,7 @@ describe('toPiContext', () => {
|
||||
|
||||
it('labels unmatched tool results with toolName unknown and keeps isError', () => {
|
||||
const context = toPiContext({
|
||||
provider: 'deepseek',
|
||||
model: 'm',
|
||||
messages: [{
|
||||
role: 'user',
|
||||
@@ -156,6 +164,7 @@ describe('toPiContext', () => {
|
||||
|
||||
it('splits mixed user text + tool results and folds history system messages', () => {
|
||||
const context = toPiContext({
|
||||
provider: 'deepseek',
|
||||
model: 'm',
|
||||
messages: [
|
||||
{ role: 'system', content: [{ type: 'text', text: 'rule' }] },
|
||||
@@ -173,6 +182,7 @@ describe('toPiContext', () => {
|
||||
|
||||
it('skips plugin-added (unknown) blocks in assistant content', () => {
|
||||
const context = toPiContext({
|
||||
provider: 'deepseek',
|
||||
model: 'm',
|
||||
messages: [{
|
||||
role: 'assistant',
|
||||
@@ -184,6 +194,195 @@ describe('toPiContext', () => {
|
||||
})
|
||||
expect((context.messages[0] as AssistantMessage).content).toEqual([{ type: 'text', text: 'visible' }])
|
||||
})
|
||||
|
||||
it('recombines durable content with pi-ai replay metadata across target providers and models', () => {
|
||||
const state = toPiReplayState(assistant({
|
||||
api: 'openai-responses',
|
||||
provider: 'openai',
|
||||
model: 'gpt-5',
|
||||
responseModel: 'gpt-5-2026-01-01',
|
||||
responseId: 'resp_123',
|
||||
stopReason: 'toolUse',
|
||||
content: [
|
||||
{ type: 'thinking', thinking: 'private reasoning', thinkingSignature: 'think-sig', redacted: true },
|
||||
{ type: 'text', text: 'calling', textSignature: 'text-sig' },
|
||||
{ type: 'toolCall', id: 'c1', name: 'f', arguments: { a: 1 }, thoughtSignature: 'tool-sig' },
|
||||
],
|
||||
}))
|
||||
const context = toPiContext({
|
||||
provider: 'anthropic',
|
||||
model: 'claude-next',
|
||||
messages: [{
|
||||
role: 'assistant',
|
||||
content: [
|
||||
{ type: 'reasoning', text: 'private reasoning' },
|
||||
{ type: 'text', text: 'calling' },
|
||||
{ type: 'tool-call', id: CallId('c1'), name: 'f', arguments: '{"a":1}' },
|
||||
],
|
||||
provenance: { provider: 'openai', model: 'gpt-5', replayState: state },
|
||||
}],
|
||||
})
|
||||
|
||||
expect(context.messages[0]).toMatchObject({
|
||||
role: 'assistant',
|
||||
api: 'openai-responses',
|
||||
provider: 'openai',
|
||||
model: 'gpt-5',
|
||||
responseModel: 'gpt-5-2026-01-01',
|
||||
responseId: 'resp_123',
|
||||
stopReason: 'toolUse',
|
||||
content: [
|
||||
{ type: 'thinking', thinking: 'private reasoning', thinkingSignature: 'think-sig', redacted: true },
|
||||
{ type: 'text', text: 'calling', textSignature: 'text-sig' },
|
||||
{ type: 'toolCall', id: 'c1', name: 'f', arguments: { a: 1 }, thoughtSignature: 'tool-sig' },
|
||||
],
|
||||
})
|
||||
})
|
||||
|
||||
it('replays all native block kinds when optional metadata is absent', () => {
|
||||
const state = toPiReplayState(assistant({
|
||||
content: [
|
||||
{ type: 'thinking', thinking: 'private reasoning' },
|
||||
{ type: 'text', text: 'calling' },
|
||||
{ type: 'toolCall', id: 'c1', name: 'f', arguments: { a: 1 } },
|
||||
],
|
||||
}))
|
||||
const context = toPiContext({
|
||||
provider: 'deepseek',
|
||||
model: 'new-model',
|
||||
messages: [{
|
||||
role: 'assistant',
|
||||
content: [
|
||||
{ type: 'reasoning', text: 'private reasoning' },
|
||||
{ type: 'text', text: 'calling' },
|
||||
{ type: 'tool-call', id: CallId('c1'), name: 'f', arguments: '{"a":1}' },
|
||||
],
|
||||
provenance: { provider: 'deepseek', model: 'deepseek-v4-flash', replayState: state },
|
||||
}],
|
||||
})
|
||||
|
||||
expect(context.messages[0]).toMatchObject({
|
||||
role: 'assistant',
|
||||
content: [
|
||||
{ type: 'thinking', thinking: 'private reasoning' },
|
||||
{ type: 'text', text: 'calling' },
|
||||
{ type: 'toolCall', id: 'c1', name: 'f', arguments: { a: 1 } },
|
||||
],
|
||||
})
|
||||
expect(context.messages[0]).not.toHaveProperty('responseModel')
|
||||
expect(context.messages[0]).not.toHaveProperty('responseId')
|
||||
})
|
||||
|
||||
it('rejects unsupported replay-state versions with a stable error code', () => {
|
||||
try {
|
||||
toPiContext({
|
||||
provider: 'deepseek',
|
||||
model: 'm',
|
||||
messages: [{
|
||||
role: 'assistant',
|
||||
content: [{ type: 'text', text: 'done' }],
|
||||
provenance: {
|
||||
provider: 'deepseek',
|
||||
model: 'old',
|
||||
replayState: { kind: 'pi-ai', version: 2 },
|
||||
},
|
||||
}],
|
||||
})
|
||||
expect.fail('expected invalid replay state')
|
||||
} catch (error: unknown) {
|
||||
expect(error).toBeInstanceOf(LlmError)
|
||||
expect((error as LlmError).code).toBe('INVALID_REPLAY_STATE')
|
||||
expect((error as Error).message).toContain('unsupported version 2')
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects replay metadata whose blocks do not match the durable content', () => {
|
||||
const state = toPiReplayState(assistant({ content: [{ type: 'text', text: 'done' }] }))
|
||||
expect(() => toPiContext({
|
||||
provider: 'deepseek',
|
||||
model: 'm',
|
||||
messages: [{
|
||||
role: 'assistant',
|
||||
content: [{ type: 'reasoning', text: 'done' }],
|
||||
provenance: { provider: 'deepseek', model: 'deepseek-v4-flash', replayState: state },
|
||||
}],
|
||||
})).toThrow(/block 0 does not match assistant content/)
|
||||
})
|
||||
|
||||
it('rejects replay metadata whose block count differs from durable content', () => {
|
||||
const state = toPiReplayState(assistant())
|
||||
expect(() => toPiContext({
|
||||
provider: 'deepseek',
|
||||
model: 'm',
|
||||
messages: [{
|
||||
role: 'assistant',
|
||||
content: [{ type: 'text', text: 'done' }],
|
||||
provenance: { provider: 'deepseek', model: 'deepseek-v4-flash', replayState: state },
|
||||
}],
|
||||
})).toThrow(/block count does not match assistant content/)
|
||||
})
|
||||
|
||||
const validReplay = {
|
||||
kind: 'pi-ai',
|
||||
version: 1,
|
||||
api: 'openai-completions',
|
||||
provider: 'deepseek',
|
||||
model: 'deepseek-v4-flash',
|
||||
stopReason: 'stop',
|
||||
blocks: [{ type: 'text' }],
|
||||
}
|
||||
|
||||
it.each([
|
||||
['provider', { ...validReplay, provider: 'openai' }],
|
||||
['model', { ...validReplay, model: 'deepseek-v4-pro' }],
|
||||
])('rejects replay metadata whose %s differs from assistant provenance', (field, replayState) => {
|
||||
try {
|
||||
toPiContext({
|
||||
provider: 'deepseek',
|
||||
model: 'next-model',
|
||||
messages: [{
|
||||
role: 'assistant',
|
||||
content: [{ type: 'text', text: 'done' }],
|
||||
provenance: { provider: 'deepseek', model: 'deepseek-v4-flash', replayState },
|
||||
}],
|
||||
})
|
||||
expect.fail('expected invalid replay state')
|
||||
} catch (error: unknown) {
|
||||
expect(error).toBeInstanceOf(LlmError)
|
||||
expect((error as LlmError).code).toBe('INVALID_REPLAY_STATE')
|
||||
expect((error as Error).message).toContain(`${field} does not match assistant provenance`)
|
||||
}
|
||||
})
|
||||
|
||||
it.each([
|
||||
['number state', 1, 'expected an object'],
|
||||
['null state', null, 'expected an object'],
|
||||
['array state', [], 'expected an object'],
|
||||
['unknown kind', { ...validReplay, kind: 'other' }, 'unknown state kind'],
|
||||
['non-string api', { ...validReplay, api: 1 }, 'api must be a non-empty string'],
|
||||
['empty provider', { ...validReplay, provider: '' }, 'provider must be a non-empty string'],
|
||||
['missing model', { ...validReplay, model: undefined }, 'model must be a non-empty string'],
|
||||
['unknown stop reason', { ...validReplay, stopReason: 'pause' }, 'unknown stopReason'],
|
||||
['non-string response model', { ...validReplay, responseModel: 1 }, 'responseModel must be a string'],
|
||||
['non-string response id', { ...validReplay, responseId: 1 }, 'responseId must be a string'],
|
||||
['non-array blocks', { ...validReplay, blocks: 'text' }, 'blocks must be an array'],
|
||||
['number block', { ...validReplay, blocks: [1] }, 'block 0 must be an object'],
|
||||
['null block', { ...validReplay, blocks: [null] }, 'block 0 must be an object'],
|
||||
['array block', { ...validReplay, blocks: [[]] }, 'block 0 must be an object'],
|
||||
['unknown block type', { ...validReplay, blocks: [{ type: 'audio' }] }, 'block 0 has an unknown type'],
|
||||
['non-string signature', { ...validReplay, blocks: [{ type: 'text', textSignature: 1 }] }, 'textSignature must be a string'],
|
||||
['non-boolean redaction', { ...validReplay, blocks: [{ type: 'reasoning', redacted: 'yes' }] }, 'redacted must be boolean'],
|
||||
])('rejects malformed replay state: %s', (_name, replayState, message) => {
|
||||
expect(() => toPiContext({
|
||||
provider: 'deepseek',
|
||||
model: 'm',
|
||||
messages: [{
|
||||
role: 'assistant',
|
||||
content: [{ type: 'text', text: 'done' }],
|
||||
provenance: { provider: 'deepseek', model: 'deepseek-v4-flash', replayState },
|
||||
}],
|
||||
})).toThrow(message)
|
||||
})
|
||||
})
|
||||
|
||||
describe('toStreamChunks', () => {
|
||||
@@ -205,7 +404,19 @@ describe('toStreamChunks', () => {
|
||||
{ type: 'text-delta', index: 0, text: 'hi' },
|
||||
{ type: 'block-end', index: 0, block: { type: 'text', text: 'hi' } },
|
||||
{ type: 'usage', usage: { inputTokens: 3, outputTokens: 2 } },
|
||||
{ type: 'finish', reason: { kind: 'stop' } },
|
||||
{
|
||||
type: 'finish',
|
||||
reason: { kind: 'stop' },
|
||||
replayState: {
|
||||
kind: 'pi-ai',
|
||||
version: 1,
|
||||
api: 'openai-completions',
|
||||
provider: 'deepseek',
|
||||
model: 'deepseek-v4-flash',
|
||||
stopReason: 'stop',
|
||||
blocks: [{ type: 'text' }],
|
||||
},
|
||||
},
|
||||
])
|
||||
})
|
||||
|
||||
@@ -234,7 +445,7 @@ describe('toStreamChunks', () => {
|
||||
toolCall: { type: 'toolCall', id: 'call-1', name: 'f', arguments: { a: 1 } },
|
||||
partial: partialWithToolCall,
|
||||
},
|
||||
{ type: 'done', reason: 'toolUse', message: assistant({ stopReason: 'toolUse' }) },
|
||||
{ type: 'done', reason: 'toolUse', message: assistant({ content: partialWithToolCall.content, stopReason: 'toolUse' }) },
|
||||
)))
|
||||
expect(chunks).toEqual([
|
||||
{ type: 'block-start', index: 0, blockType: 'tool-call' },
|
||||
@@ -242,7 +453,19 @@ describe('toStreamChunks', () => {
|
||||
{ type: 'tool-call-delta', index: 0, id: 'call-1', name: 'f', argumentsDelta: ':1}' },
|
||||
{ type: 'block-end', index: 0, block: { type: 'tool-call', id: 'call-1', name: 'f', arguments: '{"a":1}' } },
|
||||
{ type: 'usage', usage: { inputTokens: 0, outputTokens: 0 } },
|
||||
{ type: 'finish', reason: { kind: 'tool-calls' } },
|
||||
{
|
||||
type: 'finish',
|
||||
reason: { kind: 'tool-calls' },
|
||||
replayState: {
|
||||
kind: 'pi-ai',
|
||||
version: 1,
|
||||
api: 'openai-completions',
|
||||
provider: 'deepseek',
|
||||
model: 'deepseek-v4-flash',
|
||||
stopReason: 'toolUse',
|
||||
blocks: [{ type: 'tool-call' }],
|
||||
},
|
||||
},
|
||||
])
|
||||
})
|
||||
|
||||
@@ -262,20 +485,32 @@ describe('toStreamChunks', () => {
|
||||
)))
|
||||
expect(chunks).toEqual([
|
||||
{ type: 'usage', usage: { inputTokens: 1, outputTokens: 0 } },
|
||||
{ type: 'finish', reason: { kind: 'error', message: 'boom', code: 'PI_AI_ERROR' } },
|
||||
{ type: 'finish', reason: { kind: 'error', failure: { message: 'boom', code: 'PI_AI_ERROR' } } },
|
||||
])
|
||||
})
|
||||
|
||||
it('maps aborted error events to aborted finish', async () => {
|
||||
const error = assistant({ stopReason: 'aborted' })
|
||||
const chunks = await collect(toStreamChunks(feed({ type: 'error', reason: 'aborted', error })))
|
||||
expect(chunks.at(-1)).toEqual({ type: 'finish', reason: { kind: 'aborted' } })
|
||||
expect(chunks.at(-1)).toEqual({
|
||||
type: 'finish',
|
||||
reason: { kind: 'aborted', failure: { message: 'pi-ai stream aborted', code: 'ABORTED' } },
|
||||
})
|
||||
})
|
||||
|
||||
it('rejects a stream that ends without done or error', async () => {
|
||||
await expect(collect(toStreamChunks(feed({ type: 'start', partial: assistant() }))))
|
||||
.rejects.toThrow(/without done\/error/)
|
||||
})
|
||||
|
||||
it('preserves an unknown SDK iterator Error exactly', async () => {
|
||||
const original = Object.assign(new Error('SDK transport exploded'), { code: 'ECONNRESET' })
|
||||
async function* failedSdkStream(): AsyncGenerator<AssistantMessageEvent> {
|
||||
throw original
|
||||
}
|
||||
|
||||
await expect(collect(toStreamChunks(failedSdkStream()))).rejects.toBe(original)
|
||||
})
|
||||
})
|
||||
|
||||
describe('mapStopReason / mapUsage', () => {
|
||||
@@ -283,23 +518,84 @@ describe('mapStopReason / mapUsage', () => {
|
||||
['stop', { kind: 'stop' }],
|
||||
['length', { kind: 'max-tokens' }],
|
||||
['toolUse', { kind: 'tool-calls' }],
|
||||
['aborted', { kind: 'aborted' }],
|
||||
['aborted', { kind: 'aborted', failure: { message: 'pi-ai stream aborted', code: 'ABORTED' } }],
|
||||
] as const)('maps %s', (stopReason, expected) => {
|
||||
expect(mapStopReason(assistant({ stopReason }))).toEqual(expected)
|
||||
})
|
||||
|
||||
it('defaults the error message when pi-ai omits it', () => {
|
||||
expect(mapStopReason(assistant({ stopReason: 'error' })))
|
||||
.toEqual({ kind: 'error', message: 'pi-ai stream error', code: 'PI_AI_ERROR' })
|
||||
.toEqual({ kind: 'error', failure: { message: 'pi-ai stream error', code: 'PI_AI_ERROR' } })
|
||||
})
|
||||
|
||||
it('maps routable HTTP-ish error messages to stable codes', () => {
|
||||
expect(mapStopReason(assistant({ stopReason: 'error', errorMessage: 'HTTP 401: bad key' })))
|
||||
.toMatchObject({ kind: 'error', code: 'AUTH' })
|
||||
.toMatchObject({ kind: 'error', failure: { code: 'AUTH' } })
|
||||
expect(mapStopReason(assistant({ stopReason: 'error', errorMessage: 'HTTP 429: rate limit' })))
|
||||
.toMatchObject({ kind: 'error', code: 'RATE_LIMIT' })
|
||||
.toMatchObject({ kind: 'error', failure: { code: 'RATE_LIMIT' } })
|
||||
expect(mapStopReason(assistant({ stopReason: 'error', errorMessage: 'HTTP 429: insufficient_quota' })))
|
||||
.toMatchObject({ kind: 'error', failure: { code: 'QUOTA' } })
|
||||
expect(mapStopReason(assistant({
|
||||
stopReason: 'error',
|
||||
errorMessage: 'OpenAI API error (429): You exceeded your current quota, please check your plan and billing details.',
|
||||
}))).toMatchObject({ kind: 'error', failure: { code: 'QUOTA' } })
|
||||
expect(mapStopReason(assistant({ stopReason: 'error', errorMessage: 'HTTP 500: backend down' })))
|
||||
.toMatchObject({ kind: 'error', code: 'SERVER' })
|
||||
.toMatchObject({ kind: 'error', failure: { code: 'SERVER' } })
|
||||
expect(mapStopReason(assistant({ stopReason: 'error', errorMessage: 'provider timed out' })))
|
||||
.toMatchObject({ kind: 'error', failure: { code: 'TIMEOUT' } })
|
||||
expect(mapStopReason(assistant({ stopReason: 'error', errorMessage: 'ECONNRESET socket closed' })))
|
||||
.toMatchObject({ kind: 'error', failure: { code: 'TRANSPORT' } })
|
||||
expect(mapStopReason(assistant({
|
||||
stopReason: 'error',
|
||||
errorMessage: 'HTTP 400: input exceeds the model context window limit',
|
||||
}))).toMatchObject({ kind: 'error', failure: { code: CONTEXT_WINDOW_EXCEEDED_CODE } })
|
||||
expect(mapStopReason(assistant({
|
||||
stopReason: 'error',
|
||||
errorMessage: 'HTTP 400: request too large for model context',
|
||||
}))).toMatchObject({ kind: 'error', failure: { code: CONTEXT_WINDOW_EXCEEDED_CODE } })
|
||||
expect(mapStopReason(assistant({
|
||||
stopReason: 'error',
|
||||
errorMessage: 'HTTP 400: invalid input: temperature exceeds maximum allowed value',
|
||||
}))).toMatchObject({ kind: 'error', failure: { code: 'INVALID_REQUEST' } })
|
||||
})
|
||||
|
||||
it.each([
|
||||
'other side closed',
|
||||
'HTTP2 request did not get a response',
|
||||
'WebSocket closed unexpectedly',
|
||||
])('maps pi-ai transport wording %j', (errorMessage) => {
|
||||
expect(mapStopReason(assistant({ stopReason: 'error', errorMessage })))
|
||||
.toMatchObject({ kind: 'error', failure: { code: 'TRANSPORT' } })
|
||||
})
|
||||
|
||||
it('uses pi-ai provider-specific overflow classification without losing rate-limit exclusions', () => {
|
||||
expect(mapStopReason(assistant({
|
||||
stopReason: 'error',
|
||||
errorMessage: 'prompt is too long: 213462 tokens > 200000 maximum',
|
||||
}))).toMatchObject({ kind: 'error', failure: { code: CONTEXT_WINDOW_EXCEEDED_CODE } })
|
||||
expect(mapStopReason(assistant({
|
||||
stopReason: 'error',
|
||||
errorMessage: 'ThrottlingException: Too many tokens, rate limit reached',
|
||||
}))).toMatchObject({ kind: 'error', failure: { code: 'RATE_LIMIT' } })
|
||||
})
|
||||
|
||||
it('uses the resolved context window for silent and length-stop overflows', () => {
|
||||
const silent = assistant({ stopReason: 'stop', usage: usage(101, 0) })
|
||||
expect(mapStopReason(silent)).toEqual({ kind: 'stop' })
|
||||
expect(mapStopReason(silent, 100)).toEqual({
|
||||
kind: 'error',
|
||||
failure: {
|
||||
message: 'pi-ai detected context overflow for model "deepseek-v4-flash"',
|
||||
code: CONTEXT_WINDOW_EXCEEDED_CODE,
|
||||
},
|
||||
})
|
||||
|
||||
const truncated = assistant({ stopReason: 'length', usage: usage(80, 0, 19) })
|
||||
expect(mapStopReason(truncated)).toEqual({ kind: 'max-tokens' })
|
||||
expect(mapStopReason(truncated, 100)).toMatchObject({
|
||||
kind: 'error',
|
||||
failure: { code: CONTEXT_WINDOW_EXCEEDED_CODE },
|
||||
})
|
||||
})
|
||||
|
||||
it('maps cache fields only when nonzero', () => {
|
||||
|
||||
163
packages/llm/llm-pi-ai/tests/provider-apis.e2e.ts
Normal file
163
packages/llm/llm-pi-ai/tests/provider-apis.e2e.ts
Normal file
@@ -0,0 +1,163 @@
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import LlmService, { CallId } from '@deepseek-ai/dsh-llm'
|
||||
import type { Message, ToolSchema } from '@deepseek-ai/dsh-llm'
|
||||
import * as LlmPiAi from '@deepseek-ai/dsh-llm-pi-ai'
|
||||
import type { PiAiReplayState } from '../src/replay.ts'
|
||||
import { assemble, type AssembledResult } from './assemble.ts'
|
||||
|
||||
interface ProviderCase {
|
||||
provider: 'openai' | 'anthropic'
|
||||
api: 'openai-responses' | 'anthropic-messages'
|
||||
model: string
|
||||
apiKey?: string
|
||||
baseURL?: string
|
||||
headers?: Record<string, string>
|
||||
}
|
||||
|
||||
const openAIBaseURL = process.env.DSH_PI_AI_OPENAI_BASE_URL
|
||||
const azureOpenAIKey = process.env.AZURE_OPENAI_API_KEY
|
||||
|
||||
const providerCases: ProviderCase[] = [
|
||||
{
|
||||
provider: 'openai',
|
||||
api: 'openai-responses',
|
||||
model: process.env.DSH_PI_AI_OPENAI_MODEL ?? 'gpt-5.5',
|
||||
...azureOpenAIKey
|
||||
? { apiKey: azureOpenAIKey, headers: { 'api-key': azureOpenAIKey, Authorization: '' } }
|
||||
: {},
|
||||
...openAIBaseURL ? { baseURL: openAIBaseURL } : {},
|
||||
},
|
||||
{
|
||||
provider: 'anthropic',
|
||||
api: 'anthropic-messages',
|
||||
model: process.env.DSH_PI_AI_ANTHROPIC_MODEL ?? 'claude-opus-4-8',
|
||||
...process.env.ANTHROPIC_API_KEY ? { apiKey: process.env.ANTHROPIC_API_KEY } : {},
|
||||
},
|
||||
]
|
||||
|
||||
const contexts: Context[] = []
|
||||
|
||||
async function harness(): Promise<Context> {
|
||||
const ctx = new Context()
|
||||
contexts.push(ctx)
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmPiAi, {
|
||||
providers: providerCases.map(profile => ({
|
||||
provider: profile.provider,
|
||||
...profile.apiKey === undefined ? {} : { apiKey: profile.apiKey },
|
||||
...profile.baseURL === undefined ? {} : { baseURL: profile.baseURL },
|
||||
...profile.headers === undefined ? {} : { headers: profile.headers },
|
||||
})),
|
||||
})
|
||||
return ctx
|
||||
}
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(contexts.splice(0).map(ctx => ctx.fiber.dispose()))
|
||||
})
|
||||
|
||||
function ask(text: string): Message[] {
|
||||
return [{ role: 'user', content: [{ type: 'text', text }] }]
|
||||
}
|
||||
|
||||
function textOf(result: AssembledResult): string {
|
||||
return result.message.content
|
||||
.filter(block => block.type === 'text')
|
||||
.map(block => block.text)
|
||||
.join('')
|
||||
}
|
||||
|
||||
function expectFinish(result: AssembledResult, expected: 'stop' | 'tool-calls'): void {
|
||||
if (result.finish.kind === 'error') {
|
||||
throw new Error(`provider request failed (${result.finish.failure.code}): ${result.finish.failure.message}`)
|
||||
}
|
||||
expect(result.finish.kind).toBe(expected)
|
||||
}
|
||||
|
||||
function expectNativeReplay(result: AssembledResult, profile: ProviderCase): PiAiReplayState {
|
||||
const replayState = result.message.provenance?.replayState
|
||||
expect(replayState).toMatchObject({
|
||||
kind: 'pi-ai',
|
||||
version: 1,
|
||||
api: profile.api,
|
||||
provider: profile.provider,
|
||||
model: profile.model,
|
||||
})
|
||||
return replayState as PiAiReplayState
|
||||
}
|
||||
|
||||
const lookupTool: ToolSchema = {
|
||||
name: 'lookup_code',
|
||||
description: 'Look up the word represented by a short code.',
|
||||
parameters: {
|
||||
type: 'object',
|
||||
properties: { code: { type: 'string', description: 'The code to look up.' } },
|
||||
required: ['code'],
|
||||
},
|
||||
}
|
||||
|
||||
for (const profile of providerCases) {
|
||||
describe.skipIf(profile.apiKey === undefined)(
|
||||
`llm-pi-ai ${profile.provider} e2e (${profile.api})`,
|
||||
() => {
|
||||
it('streams text with usage and native replay metadata', async () => {
|
||||
const ctx = await harness()
|
||||
const result = await assemble(ctx, {
|
||||
provider: profile.provider,
|
||||
model: profile.model,
|
||||
messages: ask('Reply with exactly the word: pong'),
|
||||
maxTokens: 1024,
|
||||
})
|
||||
|
||||
expectFinish(result, 'stop')
|
||||
expect(textOf(result).toLowerCase()).toContain('pong')
|
||||
expect(result.usage?.inputTokens).toBeGreaterThan(0)
|
||||
expect(result.usage?.outputTokens).toBeGreaterThan(0)
|
||||
expect(expectNativeReplay(result, profile).stopReason).toBe('stop')
|
||||
})
|
||||
|
||||
it('round-trips a tool call with provider-native replay metadata', async () => {
|
||||
const ctx = await harness()
|
||||
const prompt = ask('Use lookup_code with code "blue". Do not answer without calling the tool.')
|
||||
const first = await assemble(ctx, {
|
||||
provider: profile.provider,
|
||||
model: profile.model,
|
||||
messages: prompt,
|
||||
tools: [lookupTool],
|
||||
maxTokens: 2048,
|
||||
})
|
||||
|
||||
expectFinish(first, 'tool-calls')
|
||||
const call = first.message.content.find(block => block.type === 'tool-call')
|
||||
expect(call).toBeDefined()
|
||||
expect(call!.name).toBe('lookup_code')
|
||||
expect(JSON.parse(call!.arguments)).toMatchObject({ code: 'blue' })
|
||||
expect(expectNativeReplay(first, profile).stopReason).toBe('toolUse')
|
||||
|
||||
const second = await assemble(ctx, {
|
||||
provider: profile.provider,
|
||||
model: profile.model,
|
||||
messages: [
|
||||
...prompt,
|
||||
first.message,
|
||||
{
|
||||
role: 'user',
|
||||
content: [{
|
||||
type: 'tool-result',
|
||||
toolCallId: CallId(call!.id),
|
||||
content: [{ type: 'text', text: 'The code blue means ocean.' }],
|
||||
}],
|
||||
},
|
||||
],
|
||||
tools: [lookupTool],
|
||||
maxTokens: 2048,
|
||||
})
|
||||
|
||||
expectFinish(second, 'stop')
|
||||
expect(textOf(second).toLowerCase()).toContain('ocean')
|
||||
expect(expectNativeReplay(second, profile).stopReason).toBe('stop')
|
||||
})
|
||||
},
|
||||
)
|
||||
}
|
||||
35
packages/llm/llm-pi-ai/tests/sdk-options.spec.ts
Normal file
35
packages/llm/llm-pi-ai/tests/sdk-options.spec.ts
Normal file
@@ -0,0 +1,35 @@
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
|
||||
const streamSimple = vi.hoisted(() => vi.fn())
|
||||
|
||||
vi.mock('@earendil-works/pi-ai', async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import('@earendil-works/pi-ai')>()
|
||||
return { ...actual, streamSimple }
|
||||
})
|
||||
|
||||
import { PiAiAdapter } from '../src/adapter.ts'
|
||||
|
||||
afterEach(() => { streamSimple.mockReset() })
|
||||
|
||||
describe('pi-ai SDK retry boundary', () => {
|
||||
it('pins one SDK attempt even when the installed provider currently defaults to zero retries', async () => {
|
||||
const failure = new Error('mock SDK boundary')
|
||||
streamSimple.mockReturnValue({
|
||||
async * [Symbol.asyncIterator](): AsyncGenerator<never> {
|
||||
throw failure
|
||||
},
|
||||
})
|
||||
const adapter = new PiAiAdapter({ profiles: [{ provider: 'openai', apiKey: 'test-key' }] })
|
||||
const drain = async (): Promise<void> => {
|
||||
for await (const _chunk of adapter.stream({
|
||||
provider: 'openai',
|
||||
model: 'gpt-4.1',
|
||||
messages: [],
|
||||
})) { /* drain */ }
|
||||
}
|
||||
|
||||
await expect(drain()).rejects.toBe(failure)
|
||||
expect(streamSimple).toHaveBeenCalledOnce()
|
||||
expect(streamSimple.mock.calls[0]?.[2]).toMatchObject({ maxRetries: 0 })
|
||||
})
|
||||
})
|
||||
@@ -19,6 +19,9 @@
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm"
|
||||
},
|
||||
{
|
||||
"path": "../../util/timeout"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
39
packages/llm/llm-retry/README.md
Normal file
39
packages/llm/llm-retry/README.md
Normal file
@@ -0,0 +1,39 @@
|
||||
# `@deepseek-ai/dsh-llm-retry`
|
||||
|
||||
Function plugin that retries selected transient model-request failures on the agent loop's closed-step recovery seam. It does not wrap `ctx.llm.stream()`: every adapter call remains one provider attempt, and every retry opens a fresh numbered step.
|
||||
|
||||
The default policy permits two retries for `RATE_LIMIT`, `SERVER`, `TIMEOUT`, and `TRANSPORT`, using bounded exponential backoff from 500 ms to 10 seconds with 10 percent jitter. Delay bounds must fit Node's supported timer range. A valid `providerRetryAfterMs` replaces local backoff when it is within the configured cap; an over-cap instruction delegates to the next recovery policy instead.
|
||||
|
||||
Before waiting, the plugin appends a non-surface `llm/retry` event with the failure and scheduled delay. Cancellation and plugin disposal abort the wait; disposal drains the plugin's active backoffs, and a callback captured before disposal fails closed if invoked afterward.
|
||||
|
||||
```yaml
|
||||
- name: '@deepseek-ai/dsh-llm-retry'
|
||||
config:
|
||||
maxTransientRetries: 2
|
||||
initialDelayMs: 500
|
||||
maxDelayMs: 10000
|
||||
jitterRatio: 0.1
|
||||
retryableCodes: [RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT]
|
||||
```
|
||||
|
||||
## Model Experience
|
||||
|
||||
### Transient request recovery
|
||||
|
||||
#### What the model sees
|
||||
|
||||
No retry event, delay, or failure prose is model-visible. After a retry, the next numbered step reconstructs the same explicit provider/model request from durable session history; failed chunks never enter derived messages.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Each retry is a new provider request and may repeat input-token billing. The finite budget caps attempts; `llm/retry` itself contributes no tokens.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
The reconstructed request preserves the prior prefix and is eligible for provider cache reuse under that provider's rules. The non-surface status event does not change cache identity.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **Agent steps are the only retry boundary** — direct `ctx.llm.stream()` consumers remain single-attempt because a raw stream cannot separate already-emitted chunks durably.
|
||||
- **Finite plugin budgets add** — this policy counts only configured transient codes; context-overflow compaction counts only its own code. A future policy with overlapping codes must document and test registration-order behavior.
|
||||
- **`llm/retry` records scheduling, not completion** — later step and turn events establish success, exhaustion, or cancellation.
|
||||
47
packages/llm/llm-retry/package.json
Normal file
47
packages/llm/llm-retry/package.json
Normal file
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-llm-retry",
|
||||
"description": "Bounded transient LLM request retry policy for the DeepSeek Harness",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/types/**/*.d.ts",
|
||||
"lib/types/**/*.d.ts.map",
|
||||
"src"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-agent": "^0.0.1",
|
||||
"@deepseek-ai/dsh-llm": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session": "^0.0.1",
|
||||
"@deepseek-ai/dsh-timeout": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"dependencies": {
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@cordisjs/plugin-include": "workspace:^",
|
||||
"@cordisjs/plugin-loader": "workspace:^",
|
||||
"@deepseek-ai/dsh-agent": "workspace:^",
|
||||
"@deepseek-ai/dsh-agent-loop": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-persistence-jsonl": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-persistence-sqlite": "workspace:^",
|
||||
"@deepseek-ai/dsh-system-prompt": "workspace:^",
|
||||
"@deepseek-ai/dsh-timeout": "workspace:^",
|
||||
"@deepseek-ai/dsh-tools": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
213
packages/llm/llm-retry/src/index.ts
Normal file
213
packages/llm/llm-retry/src/index.ts
Normal file
@@ -0,0 +1,213 @@
|
||||
/**
|
||||
* Bounded transient model-request retry policy on the agent loop's closed-step
|
||||
* recovery seam. Each scheduled retry is durable before its cancellable wait.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-llm-retry
|
||||
*/
|
||||
|
||||
import type { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import type { Agent, RequestError, RequestErrorDecision } from '@deepseek-ai/dsh-agent'
|
||||
import type { LlmFailure } from '@deepseek-ai/dsh-llm'
|
||||
import type {} from '@deepseek-ai/dsh-session'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
|
||||
declare module '@deepseek-ai/dsh-session' {
|
||||
interface SessionEventMap {
|
||||
/** Durable, non-surface record of one transient retry scheduled after a closed failed step. */
|
||||
'llm/retry': {
|
||||
turn: number
|
||||
step: number
|
||||
retry: number
|
||||
maxRetries: number
|
||||
delayMs: number
|
||||
failure: LlmFailure
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export const name = 'llm-retry'
|
||||
export const inject = ['agents']
|
||||
|
||||
const DEFAULT_MAX_TRANSIENT_RETRIES = 2
|
||||
const DEFAULT_INITIAL_DELAY_MS = 500
|
||||
const DEFAULT_MAX_DELAY_MS = 10_000
|
||||
const DEFAULT_JITTER_RATIO = 0.1
|
||||
const DEFAULT_RETRYABLE_CODES = Object.freeze(['RATE_LIMIT', 'SERVER', 'TIMEOUT', 'TRANSPORT'])
|
||||
|
||||
/** Deployment-owned limits and classification for transient request recovery. */
|
||||
export interface Config {
|
||||
/** Maximum transient retries after the first request (default 2). */
|
||||
maxTransientRetries?: number
|
||||
/** Initial local exponential-backoff delay in milliseconds (default 500). */
|
||||
initialDelayMs?: number
|
||||
/** Maximum accepted or locally scheduled delay in milliseconds (default 10000). */
|
||||
maxDelayMs?: number
|
||||
/** Symmetric random multiplier range around one (default 0.1). */
|
||||
jitterRatio?: number
|
||||
/** Stable failure codes eligible for this policy. */
|
||||
retryableCodes?: string[]
|
||||
}
|
||||
|
||||
/** Runtime schema for {@link Config}. */
|
||||
export const Config: z<Config> = z.object({
|
||||
maxTransientRetries: z.number().step(1).min(0).default(DEFAULT_MAX_TRANSIENT_RETRIES),
|
||||
initialDelayMs: z.number().max(MAX_TIMER_DELAY_MS).default(DEFAULT_INITIAL_DELAY_MS),
|
||||
maxDelayMs: z.number().max(MAX_TIMER_DELAY_MS).default(DEFAULT_MAX_DELAY_MS),
|
||||
jitterRatio: z.number().min(0).max(1).default(DEFAULT_JITTER_RATIO),
|
||||
retryableCodes: z.array(z.string()).default([...DEFAULT_RETRYABLE_CODES]),
|
||||
})
|
||||
|
||||
interface ResolvedConfig {
|
||||
readonly maxTransientRetries: number
|
||||
readonly initialDelayMs: number
|
||||
readonly maxDelayMs: number
|
||||
readonly jitterRatio: number
|
||||
readonly retryableCodes: ReadonlySet<string>
|
||||
}
|
||||
|
||||
function resolveConfig(config: Config): ResolvedConfig {
|
||||
const maxTransientRetries = config.maxTransientRetries ?? DEFAULT_MAX_TRANSIENT_RETRIES
|
||||
const initialDelayMs = config.initialDelayMs ?? DEFAULT_INITIAL_DELAY_MS
|
||||
const maxDelayMs = config.maxDelayMs ?? DEFAULT_MAX_DELAY_MS
|
||||
const jitterRatio = config.jitterRatio ?? DEFAULT_JITTER_RATIO
|
||||
const codes = config.retryableCodes ?? [...DEFAULT_RETRYABLE_CODES]
|
||||
|
||||
if (!Number.isInteger(maxTransientRetries) || maxTransientRetries < 0) {
|
||||
throw new Error('llm-retry: maxTransientRetries must be a non-negative integer')
|
||||
}
|
||||
if (!Number.isFinite(initialDelayMs) || initialDelayMs <= 0 || initialDelayMs > MAX_TIMER_DELAY_MS) {
|
||||
throw new Error(`llm-retry: initialDelayMs must be a positive finite number no greater than ${MAX_TIMER_DELAY_MS}`)
|
||||
}
|
||||
if (!Number.isFinite(maxDelayMs) || maxDelayMs <= 0 || maxDelayMs > MAX_TIMER_DELAY_MS) {
|
||||
throw new Error(`llm-retry: maxDelayMs must be a positive finite number no greater than ${MAX_TIMER_DELAY_MS}`)
|
||||
}
|
||||
if (initialDelayMs > maxDelayMs) {
|
||||
throw new Error('llm-retry: initialDelayMs must be less than or equal to maxDelayMs')
|
||||
}
|
||||
if (!Number.isFinite(jitterRatio) || jitterRatio < 0 || jitterRatio > 1) {
|
||||
throw new Error('llm-retry: jitterRatio must be between 0 and 1')
|
||||
}
|
||||
if (codes.length === 0) {
|
||||
throw new Error('llm-retry: retryableCodes must not be empty')
|
||||
}
|
||||
if (codes.some(code => code.length === 0)) {
|
||||
throw new Error('llm-retry: retryableCodes must contain only non-empty strings')
|
||||
}
|
||||
if (new Set(codes).size !== codes.length) {
|
||||
throw new Error('llm-retry: retryableCodes must not contain duplicates')
|
||||
}
|
||||
|
||||
return Object.freeze({
|
||||
maxTransientRetries,
|
||||
initialDelayMs,
|
||||
maxDelayMs,
|
||||
jitterRatio,
|
||||
retryableCodes: new Set(codes),
|
||||
})
|
||||
}
|
||||
|
||||
/** Non-serializable seams used to make timing policy deterministic in tests. */
|
||||
export interface RetryInternals {
|
||||
/** Random sample in the inclusive zero-to-one range used for jitter. */
|
||||
random?: () => number
|
||||
}
|
||||
|
||||
function localDelay(config: ResolvedConfig, retry: number, random: () => number): number {
|
||||
const exponent = Math.min(retry - 1, 1024)
|
||||
const exponential = Math.min(config.initialDelayMs * 2 ** exponent, config.maxDelayMs)
|
||||
const jitter = 1 - config.jitterRatio + 2 * config.jitterRatio * random()
|
||||
return Math.min(exponential * jitter, config.maxDelayMs)
|
||||
}
|
||||
|
||||
function cancellableDelay(delayMs: number, signal: AbortSignal): Promise<boolean> {
|
||||
if (signal.aborted) return Promise.resolve(false)
|
||||
return new Promise((resolve) => {
|
||||
const timer = setTimeout(() => {
|
||||
signal.removeEventListener('abort', onAbort)
|
||||
resolve(true)
|
||||
}, delayMs)
|
||||
function onAbort(): void {
|
||||
clearTimeout(timer)
|
||||
resolve(false)
|
||||
}
|
||||
signal.addEventListener('abort', onAbort, { once: true })
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Install bounded transient request recovery.
|
||||
* @param ctx - plugin context that owns the listener and active waits.
|
||||
* @param config - retry budget, delay bounds, jitter, and eligible codes.
|
||||
* @param internals - non-serializable deterministic seams for tests.
|
||||
*/
|
||||
export function apply(ctx: Context, config: Config = {}, internals: RetryInternals = {}): void {
|
||||
const resolved = resolveConfig(config)
|
||||
const random = internals.random ?? Math.random
|
||||
const lifetime = new AbortController()
|
||||
const active = new Set<Promise<RequestErrorDecision>>()
|
||||
|
||||
async function backoff(
|
||||
agent: Agent,
|
||||
turn: number,
|
||||
step: number,
|
||||
failure: LlmFailure,
|
||||
retry: number,
|
||||
delayMs: number,
|
||||
signal: AbortSignal,
|
||||
): Promise<RequestErrorDecision> {
|
||||
const fusedSignal = AbortSignal.any([signal, lifetime.signal])
|
||||
if (fusedSignal.aborted) return { action: 'fail' }
|
||||
agent.session.append('llm/retry', {
|
||||
turn,
|
||||
step,
|
||||
retry,
|
||||
maxRetries: resolved.maxTransientRetries,
|
||||
delayMs,
|
||||
failure,
|
||||
})
|
||||
if (!await cancellableDelay(delayMs, fusedSignal)) return { action: 'fail' }
|
||||
return { action: 'retry' }
|
||||
}
|
||||
|
||||
const disposeListener = ctx.on('agent/request-error', (
|
||||
agent: Agent,
|
||||
turn: number,
|
||||
step: number,
|
||||
_error: RequestError,
|
||||
failure: LlmFailure,
|
||||
priorFailures: readonly LlmFailure[],
|
||||
signal: AbortSignal,
|
||||
next: () => Promise<RequestErrorDecision>,
|
||||
) => {
|
||||
// A waterfall may have captured this callback before its registration was
|
||||
// removed. Lifetime cancellation must prevent that stale callback from
|
||||
// entering a downstream policy after disposal.
|
||||
if (lifetime.signal.aborted) return Promise.resolve<RequestErrorDecision>({ action: 'fail' })
|
||||
if (!resolved.retryableCodes.has(failure.code)) return next()
|
||||
const priorTransientFailures = priorFailures.filter(item => resolved.retryableCodes.has(item.code)).length
|
||||
if (priorTransientFailures >= resolved.maxTransientRetries) return next()
|
||||
|
||||
const retry = priorTransientFailures + 1
|
||||
let delayMs: number
|
||||
if (failure.providerRetryAfterMs !== undefined
|
||||
&& Number.isFinite(failure.providerRetryAfterMs)
|
||||
&& failure.providerRetryAfterMs > 0) {
|
||||
if (failure.providerRetryAfterMs > resolved.maxDelayMs) return next()
|
||||
delayMs = failure.providerRetryAfterMs
|
||||
} else {
|
||||
delayMs = localDelay(resolved, retry, random)
|
||||
}
|
||||
|
||||
const tracked = backoff(agent, turn, step, failure, retry, delayMs, signal)
|
||||
.finally(() => active.delete(tracked))
|
||||
active.add(tracked)
|
||||
return tracked
|
||||
})
|
||||
|
||||
ctx.effect(() => async () => {
|
||||
disposeListener()
|
||||
lifetime.abort(new Error('llm-retry plugin disposed'))
|
||||
await Promise.allSettled([...active])
|
||||
}, 'llm-retry: abort and drain backoffs')
|
||||
}
|
||||
124
packages/llm/llm-retry/tests/loader-composition.spec.ts
Normal file
124
packages/llm/llm-retry/tests/loader-composition.spec.ts
Normal file
@@ -0,0 +1,124 @@
|
||||
import { mkdtemp, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import Loader from '@cordisjs/plugin-loader'
|
||||
import Include from '@cordisjs/plugin-include'
|
||||
import AgentRegistry from '@deepseek-ai/dsh-agent'
|
||||
import type { Agent } from '@deepseek-ai/dsh-agent'
|
||||
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
|
||||
import LlmService, { LlmAdapter, LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
|
||||
import ToolRegistry from '@deepseek-ai/dsh-tools'
|
||||
import * as retry from '../src/index.ts'
|
||||
|
||||
let root: string | undefined
|
||||
let context: Context | undefined
|
||||
|
||||
class TransientOnceAdapter extends LlmAdapter {
|
||||
requests = 0
|
||||
|
||||
async * stream(_options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
this.requests += 1
|
||||
if (this.requests === 1) throw new LlmError('temporary outage', 'SERVER')
|
||||
yield { type: 'block-start', index: 0, blockType: 'text' }
|
||||
yield { type: 'text-delta', index: 0, text: 'recovered' }
|
||||
yield { type: 'block-end', index: 0, block: { type: 'text', text: 'recovered' } }
|
||||
yield { type: 'finish', reason: { kind: 'stop' } }
|
||||
}
|
||||
}
|
||||
|
||||
function waitForIdle(ctx: Context, agent: Agent): Promise<void> {
|
||||
return new Promise((resolve) => {
|
||||
const dispose = ctx.on('agent/status', (subject, status) => {
|
||||
if (subject === agent && status === 'idle') {
|
||||
dispose()
|
||||
resolve()
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
afterEach(async () => {
|
||||
await context?.fiber.dispose()
|
||||
context = undefined
|
||||
if (root !== undefined) await rm(root, { recursive: true, force: true })
|
||||
root = undefined
|
||||
})
|
||||
|
||||
async function loadYaml(lines: readonly string[]): Promise<Context> {
|
||||
root = await mkdtemp(join(tmpdir(), 'dsh-llm-retry-loader-'))
|
||||
const configPath = join(root, 'cordis.yml')
|
||||
await writeFile(configPath, [...lines, ''].join('\n'))
|
||||
|
||||
context = new Context()
|
||||
context.baseUrl = pathToFileURL(root).href + '/'
|
||||
await context.plugin(Loader)
|
||||
context.loader.builtins.include = Include
|
||||
const modules = new Map<string, unknown>([
|
||||
['@deepseek-ai/dsh-llm', LlmService],
|
||||
['@deepseek-ai/dsh-session', SessionStore],
|
||||
['@deepseek-ai/dsh-system-prompt', SystemPrompt],
|
||||
['@deepseek-ai/dsh-tools', ToolRegistry],
|
||||
['@deepseek-ai/dsh-agent', AgentRegistry],
|
||||
['@deepseek-ai/dsh-llm-retry', retry],
|
||||
['@deepseek-ai/dsh-agent-loop', AgentLoop],
|
||||
])
|
||||
context.loader.internal = {
|
||||
version: 'v2',
|
||||
async import(specifier: string) {
|
||||
if (!modules.has(specifier)) throw new Error(`unexpected Loader import: ${specifier}`)
|
||||
return modules.get(specifier)
|
||||
},
|
||||
} as unknown as NonNullable<typeof context.loader.internal>
|
||||
await context.loader.create({
|
||||
name: 'cordis:include',
|
||||
config: { path: pathToFileURL(configPath).href },
|
||||
})
|
||||
await context.loader.await()
|
||||
return context
|
||||
}
|
||||
|
||||
describe('real Loader composition', () => {
|
||||
it('loads the flat policy and records recovery through the shipping loop', async () => {
|
||||
const loaded = await loadYaml([
|
||||
"- name: '@deepseek-ai/dsh-llm'",
|
||||
"- name: '@deepseek-ai/dsh-session'",
|
||||
"- name: '@deepseek-ai/dsh-system-prompt'",
|
||||
"- name: '@deepseek-ai/dsh-tools'",
|
||||
"- name: '@deepseek-ai/dsh-agent'",
|
||||
"- name: '@deepseek-ai/dsh-llm-retry'",
|
||||
' config:',
|
||||
' maxTransientRetries: 1',
|
||||
' initialDelayMs: 1',
|
||||
' maxDelayMs: 1',
|
||||
' jitterRatio: 0',
|
||||
' retryableCodes: [RATE_LIMIT, SERVER]',
|
||||
"- name: '@deepseek-ai/dsh-agent-loop'",
|
||||
])
|
||||
|
||||
const unloaded = [...loaded.loader.entries()]
|
||||
.filter(entry => entry.fiber === undefined && !entry.disabled)
|
||||
.map(entry => entry.options.name)
|
||||
expect(unloaded).toEqual([])
|
||||
expect(loaded.agents).toBeInstanceOf(AgentRegistry)
|
||||
|
||||
const adapter = new TransientOnceAdapter()
|
||||
loaded.llm.registerAdapter(['mock'], adapter)
|
||||
const agent = loaded.agentLoop.create(SessionId('loader-retry'), { provider: 'mock', model: 'mock' })
|
||||
const idle = waitForIdle(loaded, agent)
|
||||
agent.send([{ type: 'text', text: 'recover' }])
|
||||
await idle
|
||||
|
||||
expect(adapter.requests).toBe(2)
|
||||
expect(agent.session.events.filter(event => event.type === 'llm/retry')).toHaveLength(1)
|
||||
expect(agent.session.deriveMessages().at(-1)).toMatchObject({
|
||||
role: 'assistant',
|
||||
content: [{ type: 'text', text: 'recovered' }],
|
||||
})
|
||||
})
|
||||
})
|
||||
57
packages/llm/llm-retry/tests/persistence.spec.ts
Normal file
57
packages/llm/llm-retry/tests/persistence.spec.ts
Normal file
@@ -0,0 +1,57 @@
|
||||
import { mkdtemp, rm } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import SessionPersistenceJsonl from '@deepseek-ai/dsh-session-persistence-jsonl'
|
||||
import SessionPersistenceSqlite from '@deepseek-ai/dsh-session-persistence-sqlite'
|
||||
import type {} from '../src/index.ts'
|
||||
|
||||
const dirs: string[] = []
|
||||
|
||||
afterEach(async () => {
|
||||
for (const dir of dirs.splice(0)) await rm(dir, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
async function backend(kind: 'jsonl' | 'sqlite'): Promise<Context> {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
if (kind === 'jsonl') {
|
||||
const root = await mkdtemp(join(tmpdir(), 'dsh-llm-retry-jsonl-'))
|
||||
dirs.push(root)
|
||||
await ctx.plugin(SessionPersistenceJsonl, { root })
|
||||
} else {
|
||||
await ctx.plugin(SessionPersistenceSqlite, { path: ':memory:' })
|
||||
}
|
||||
return ctx
|
||||
}
|
||||
|
||||
describe.each(['jsonl', 'sqlite'] as const)('%s retry-event persistence', (kind) => {
|
||||
it('round-trips the event losslessly without adding a model message', async () => {
|
||||
const ctx = await backend(kind)
|
||||
try {
|
||||
const session = ctx.sessions.create(SessionId(`retry-${kind}`))
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
session.append('step/end', { turn: 1, step: 1 })
|
||||
const event = session.append('llm/retry', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
retry: 1,
|
||||
maxRetries: 2,
|
||||
delayMs: 750,
|
||||
failure: { message: 'provider busy', code: 'RATE_LIMIT', status: 429 },
|
||||
})
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'aborted', reason: 'cancelled in backoff' } })
|
||||
|
||||
expect(session.deriveMessages()).toEqual([])
|
||||
await ctx.sessions.flush(session)
|
||||
const loaded = await ctx.sessionPersistence.load(session.id)
|
||||
|
||||
expect(loaded.events.find(item => item.type === 'llm/retry')).toEqual(event)
|
||||
} finally {
|
||||
await ctx.fiber.dispose()
|
||||
}
|
||||
})
|
||||
})
|
||||
453
packages/llm/llm-retry/tests/retry.spec.ts
Normal file
453
packages/llm/llm-retry/tests/retry.spec.ts
Normal file
@@ -0,0 +1,453 @@
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import type { Fiber } from 'cordis'
|
||||
import LlmService, { CallId, LlmAdapter, LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
|
||||
import ToolRegistry, { defineTool } from '@deepseek-ai/dsh-tools'
|
||||
import AgentRegistry from '@deepseek-ai/dsh-agent'
|
||||
import type { Agent, RequestErrorDecision } from '@deepseek-ai/dsh-agent'
|
||||
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
import * as retry from '../src/index.ts'
|
||||
|
||||
type ScriptEntry = Error | Iterable<StreamChunk> | AsyncIterable<StreamChunk>
|
||||
|
||||
class ScriptedAdapter extends LlmAdapter {
|
||||
readonly requests: GenerateOptions[] = []
|
||||
|
||||
constructor(private readonly entries: ScriptEntry[]) {
|
||||
super()
|
||||
}
|
||||
|
||||
async * stream(options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
this.requests.push(options)
|
||||
const entry = this.entries.shift()
|
||||
if (entry === undefined) throw new Error('retry test script exhausted')
|
||||
if (entry instanceof Error) throw entry
|
||||
yield* entry
|
||||
}
|
||||
}
|
||||
|
||||
async function* partialToolFailure(error: Error): AsyncGenerator<StreamChunk> {
|
||||
const id = CallId('discarded-call')
|
||||
yield { type: 'block-start', index: 0, blockType: 'text' }
|
||||
yield { type: 'text-delta', index: 0, text: 'discarded partial output' }
|
||||
yield { type: 'block-end', index: 0, block: { type: 'text', text: 'discarded partial output' } }
|
||||
yield { type: 'block-start', index: 1, blockType: 'tool-call' }
|
||||
yield { type: 'tool-call-delta', index: 1, id, name: 'danger', argumentsDelta: '{}' }
|
||||
yield { type: 'block-end', index: 1, block: { type: 'tool-call', id, name: 'danger', arguments: '{}' } }
|
||||
throw error
|
||||
}
|
||||
|
||||
function textResponse(text: string): StreamChunk[] {
|
||||
return [
|
||||
{ type: 'block-start', index: 0, blockType: 'text' },
|
||||
{ type: 'text-delta', index: 0, text },
|
||||
{ type: 'block-end', index: 0, block: { type: 'text', text } },
|
||||
{ type: 'finish', reason: { kind: 'stop' } },
|
||||
]
|
||||
}
|
||||
|
||||
async function harness(
|
||||
adapter: LlmAdapter,
|
||||
config: retry.Config = {},
|
||||
beforeRetry?: (ctx: Context) => void,
|
||||
internals: retry.RetryInternals = {},
|
||||
): Promise<{ ctx: Context; retryFiber: Fiber }> {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SystemPrompt)
|
||||
await ctx.plugin(ToolRegistry)
|
||||
await ctx.plugin(AgentRegistry)
|
||||
beforeRetry?.(ctx)
|
||||
const resolvedConfig = Object.assign({
|
||||
maxTransientRetries: 2,
|
||||
initialDelayMs: 500,
|
||||
maxDelayMs: 10_000,
|
||||
jitterRatio: 0,
|
||||
}, config)
|
||||
const retryFiber = await ctx.plugin(Object.assign((inner: Context) => {
|
||||
retry.apply(inner, resolvedConfig, internals)
|
||||
}, { inject: retry.inject }))
|
||||
await ctx.plugin(AgentLoop, { agents: [] })
|
||||
ctx.llm.registerAdapter(['mock'], adapter)
|
||||
return { ctx, retryFiber }
|
||||
}
|
||||
|
||||
function waitForIdle(ctx: Context, agent: Agent): Promise<void> {
|
||||
return new Promise((resolve) => {
|
||||
const dispose = ctx.on('agent/status', (subject, status) => {
|
||||
if (subject === agent && status === 'idle') {
|
||||
dispose()
|
||||
resolve()
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
function waitForRetry(ctx: Context, agent: Agent, retryNumber: number): Promise<Extract<SessionEvent, { type: 'llm/retry' }>> {
|
||||
return new Promise((resolve) => {
|
||||
const dispose = ctx.on('session/event', (session, event) => {
|
||||
if (session === agent.session && event.type === 'llm/retry' && event.data.retry === retryNumber) {
|
||||
dispose()
|
||||
resolve(event)
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
let context: Context | undefined
|
||||
|
||||
afterEach(async () => {
|
||||
vi.useRealTimers()
|
||||
await context?.fiber.dispose()
|
||||
context = undefined
|
||||
})
|
||||
|
||||
describe('bounded transient retry policy', () => {
|
||||
it('records the scheduled delay before opening a fresh request attempt', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('busy', 'RATE_LIMIT', { status: 429 }),
|
||||
textResponse('done'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter))
|
||||
const agent = context.agentLoop.create(SessionId('retry-success'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const scheduled = new Promise<Extract<(typeof agent.session.events)[number], { type: 'llm/retry' }>>((resolve) => {
|
||||
const dispose = context?.on('session/event', (session, event) => {
|
||||
if (session === agent.session && event.type === 'llm/retry') {
|
||||
dispose?.()
|
||||
resolve(event)
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
const event = await scheduled
|
||||
|
||||
expect(event.data).toEqual({
|
||||
turn: 1,
|
||||
step: 1,
|
||||
retry: 1,
|
||||
maxRetries: 2,
|
||||
delayMs: 500,
|
||||
failure: { message: 'busy', code: 'RATE_LIMIT', status: 429 },
|
||||
})
|
||||
expect(adapter.requests).toHaveLength(1)
|
||||
await vi.advanceTimersByTimeAsync(499)
|
||||
expect(adapter.requests).toHaveLength(1)
|
||||
|
||||
const idle = waitForIdle(context, agent)
|
||||
await vi.advanceTimersByTimeAsync(1)
|
||||
await idle
|
||||
|
||||
expect(adapter.requests).toHaveLength(2)
|
||||
expect(agent.session.events.filter(item => item.type === 'step/start').map(item => item.data.step))
|
||||
.toEqual([1, 2])
|
||||
expect(agent.session.deriveMessages().at(-1)).toEqual({
|
||||
role: 'assistant',
|
||||
content: [{ type: 'text', text: 'done' }],
|
||||
provenance: { provider: 'mock', model: 'mock' },
|
||||
})
|
||||
})
|
||||
|
||||
it('leaves partial failed chunks on their step without committing a message or tool side effect', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
partialToolFailure(new LlmError('stream interrupted', 'TRANSPORT')),
|
||||
textResponse('recovered'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter))
|
||||
let toolExecutions = 0
|
||||
context.tools.register(defineTool({
|
||||
name: 'danger',
|
||||
description: 'must not run for a failed provider attempt',
|
||||
parameters: {},
|
||||
async execute() {
|
||||
toolExecutions += 1
|
||||
return [{ type: 'text', text: 'unexpected' }]
|
||||
},
|
||||
}))
|
||||
const agent = context.agentLoop.create(SessionId('retry-partial'), { provider: 'mock', model: 'mock' })
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
await scheduled
|
||||
const idle = waitForIdle(context, agent)
|
||||
await vi.advanceTimersByTimeAsync(500)
|
||||
await idle
|
||||
|
||||
const failedChunks = agent.session.events.filter(event =>
|
||||
event.type === 'assistant/chunk' && event.data.step === 1,
|
||||
)
|
||||
expect(failedChunks).toHaveLength(6)
|
||||
expect(agent.session.events.filter(event => event.type === 'assistant/message').map(event => event.data.step))
|
||||
.toEqual([2])
|
||||
expect(agent.session.events.some(event => event.type === 'tool/call')).toBe(false)
|
||||
expect(toolExecutions).toBe(0)
|
||||
expect(agent.session.deriveMessages().at(-1)).toMatchObject({
|
||||
role: 'assistant',
|
||||
content: [{ type: 'text', text: 'recovered' }],
|
||||
provenance: { provider: 'mock', model: 'mock' },
|
||||
})
|
||||
})
|
||||
|
||||
it('applies bounded exponential jitter and stops after the configured budget', async () => {
|
||||
vi.useFakeTimers()
|
||||
const samples = [0, 1]
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('busy one', 'SERVER'),
|
||||
new LlmError('busy two', 'SERVER'),
|
||||
new LlmError('busy three', 'SERVER'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, { jitterRatio: 0.1 }, undefined, {
|
||||
random: () => samples.shift() ?? 0.5,
|
||||
}))
|
||||
const agent = context.agentLoop.create(SessionId('retry-exhausted'), { provider: 'mock', model: 'mock' })
|
||||
const first = waitForRetry(context, agent, 1)
|
||||
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
expect((await first).data.delayMs).toBe(450)
|
||||
|
||||
const second = waitForRetry(context, agent, 2)
|
||||
await vi.advanceTimersByTimeAsync(450)
|
||||
expect((await second).data.delayMs).toBe(1_100)
|
||||
|
||||
const idle = waitForIdle(context, agent)
|
||||
await vi.advanceTimersByTimeAsync(1_100)
|
||||
await idle
|
||||
|
||||
expect(adapter.requests).toHaveLength(3)
|
||||
expect(agent.session.events.filter(event => event.type === 'llm/retry')).toHaveLength(2)
|
||||
expect(agent.session.events.at(-1)).toMatchObject({
|
||||
type: 'turn/end',
|
||||
data: { reason: { kind: 'error', failure: { message: 'busy three', code: 'SERVER' } } },
|
||||
})
|
||||
})
|
||||
|
||||
it('uses a bounded provider Retry-After verbatim and delegates an over-cap instruction', async () => {
|
||||
vi.useFakeTimers()
|
||||
const accepted = new ScriptedAdapter([
|
||||
new LlmError('wait', 'RATE_LIMIT', { providerRetryAfterMs: 2_000 }),
|
||||
textResponse('done'),
|
||||
])
|
||||
;({ ctx: context } = await harness(accepted, { jitterRatio: 1 }))
|
||||
const acceptedAgent = context.agentLoop.create(SessionId('retry-after-accepted'), { provider: 'mock', model: 'mock' })
|
||||
const scheduled = waitForRetry(context, acceptedAgent, 1)
|
||||
acceptedAgent.send([{ type: 'text', text: 'go' }])
|
||||
expect((await scheduled).data.delayMs).toBe(2_000)
|
||||
const acceptedIdle = waitForIdle(context, acceptedAgent)
|
||||
await vi.advanceTimersByTimeAsync(2_000)
|
||||
await acceptedIdle
|
||||
expect(accepted.requests).toHaveLength(2)
|
||||
|
||||
await context.fiber.dispose()
|
||||
const rejected = new ScriptedAdapter([
|
||||
new LlmError('wait too long', 'RATE_LIMIT', { providerRetryAfterMs: 10_001 }),
|
||||
])
|
||||
;({ ctx: context } = await harness(rejected))
|
||||
const rejectedAgent = context.agentLoop.create(SessionId('retry-after-rejected'), { provider: 'mock', model: 'mock' })
|
||||
const rejectedIdle = waitForIdle(context, rejectedAgent)
|
||||
rejectedAgent.send([{ type: 'text', text: 'go' }])
|
||||
await rejectedIdle
|
||||
expect(rejected.requests).toHaveLength(1)
|
||||
expect(rejectedAgent.session.events.some(event => event.type === 'llm/retry')).toBe(false)
|
||||
})
|
||||
|
||||
it('delegates non-transient failures without scheduling a timer', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([new LlmError('bad key', 'AUTH')])
|
||||
;({ ctx: context } = await harness(adapter))
|
||||
const agent = context.agentLoop.create(SessionId('retry-auth'), { provider: 'mock', model: 'mock' })
|
||||
const idle = waitForIdle(context, agent)
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
await idle
|
||||
expect(adapter.requests).toHaveLength(1)
|
||||
expect(agent.session.events.some(event => event.type === 'llm/retry')).toBe(false)
|
||||
expect(vi.getTimerCount()).toBe(0)
|
||||
})
|
||||
|
||||
it('aborts and drains a captured backoff before plugin disposal completes', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('temporary', 'TRANSPORT'),
|
||||
textResponse('must not run'),
|
||||
])
|
||||
const mounted = await harness(adapter)
|
||||
context = mounted.ctx
|
||||
const agent = context.agentLoop.create(SessionId('retry-hmr'), { provider: 'mock', model: 'mock' })
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
await scheduled
|
||||
const idle = waitForIdle(context, agent)
|
||||
|
||||
await mounted.retryFiber.dispose()
|
||||
await idle
|
||||
await vi.advanceTimersByTimeAsync(60_000)
|
||||
|
||||
expect(adapter.requests).toHaveLength(1)
|
||||
expect(agent.session.events.filter(event => event.type === 'step/start')).toHaveLength(1)
|
||||
expect(vi.getTimerCount()).toBe(0)
|
||||
})
|
||||
|
||||
it('does not make plugin disposal wait for a delegated recovery policy', async () => {
|
||||
const adapter = new ScriptedAdapter([new LlmError('bad key', 'AUTH')])
|
||||
const mounted = await harness(adapter)
|
||||
context = mounted.ctx
|
||||
const downstream = Promise.withResolvers<RequestErrorDecision>()
|
||||
const entered = Promise.withResolvers<undefined>()
|
||||
context.on('agent/request-error', () => {
|
||||
entered.resolve(undefined)
|
||||
return downstream.promise
|
||||
})
|
||||
const agent = context.agentLoop.create(SessionId('retry-delegated-disposal'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const idle = waitForIdle(context, agent)
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
await entered.promise
|
||||
|
||||
const disposing = mounted.retryFiber.dispose()
|
||||
let timer: ReturnType<typeof setTimeout> | undefined
|
||||
const outcome = await Promise.race([
|
||||
disposing.then(() => 'disposed' as const),
|
||||
new Promise<'blocked'>((resolve) => { timer = setTimeout(() => { resolve('blocked') }, 100) }),
|
||||
])
|
||||
if (timer !== undefined) clearTimeout(timer)
|
||||
downstream.resolve({ action: 'fail' })
|
||||
await disposing
|
||||
await idle
|
||||
|
||||
expect(outcome).toBe('disposed')
|
||||
expect(adapter.requests).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('fails a captured callback after disposal without entering downstream policy', async () => {
|
||||
const adapter = new ScriptedAdapter([new LlmError('bad key', 'AUTH')])
|
||||
const captured = Promise.withResolvers<undefined>()
|
||||
let invokeCaptured: (() => Promise<void>) | undefined
|
||||
const mounted = await harness(adapter, {}, (ctx) => {
|
||||
ctx.on('agent/request-error', (_agent, _turn, _step, _error, _failure, _history, _signal, next) => {
|
||||
return new Promise<RequestErrorDecision>((resolve) => {
|
||||
invokeCaptured = async () => { resolve(await next()) }
|
||||
captured.resolve(undefined)
|
||||
})
|
||||
})
|
||||
})
|
||||
context = mounted.ctx
|
||||
let downstreamCalls = 0
|
||||
context.on('agent/request-error', async (_agent, _turn, _step, _error, _failure, _history, _signal, next) => {
|
||||
downstreamCalls += 1
|
||||
return next()
|
||||
})
|
||||
const agent = context.agentLoop.create(SessionId('retry-captured-disposal'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const idle = waitForIdle(context, agent)
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
await captured.promise
|
||||
|
||||
await mounted.retryFiber.dispose()
|
||||
if (invokeCaptured === undefined) throw new Error('request-error waterfall did not capture retry callback')
|
||||
await invokeCaptured()
|
||||
await idle
|
||||
|
||||
expect(downstreamCalls).toBe(0)
|
||||
expect(adapter.requests).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('lets turn cancellation win during backoff without opening another step', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('temporary', 'TIMEOUT'),
|
||||
textResponse('must not run'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter))
|
||||
const agent = context.agentLoop.create(SessionId('retry-cancel'), { provider: 'mock', model: 'mock' })
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
await scheduled
|
||||
const idle = waitForIdle(context, agent)
|
||||
agent.cancel('user cancelled during retry')
|
||||
await idle
|
||||
|
||||
expect(adapter.requests).toHaveLength(1)
|
||||
expect(agent.session.events.at(-1)).toMatchObject({
|
||||
type: 'turn/end',
|
||||
data: { reason: { kind: 'aborted', reason: 'user cancelled during retry' } },
|
||||
})
|
||||
expect(vi.getTimerCount()).toBe(0)
|
||||
})
|
||||
|
||||
it('lets an earlier recovery listener cancel before retry policy runs', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('temporary', 'SERVER'),
|
||||
textResponse('must not run'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, {}, (ctx) => {
|
||||
ctx.on('agent/request-error', async (agent, _turn, _step, _error, _failure, _history, _signal, next) => {
|
||||
agent.cancel('cancelled by earlier recovery policy')
|
||||
return next()
|
||||
})
|
||||
}))
|
||||
const agent = context.agentLoop.create(SessionId('retry-pre-cancel'), { provider: 'mock', model: 'mock' })
|
||||
const idle = waitForIdle(context, agent)
|
||||
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
await idle
|
||||
|
||||
expect(adapter.requests).toHaveLength(1)
|
||||
expect(agent.session.events.some(event => event.type === 'llm/retry')).toBe(false)
|
||||
expect(agent.session.events.at(-1)).toMatchObject({
|
||||
type: 'turn/end',
|
||||
data: { reason: { kind: 'aborted', reason: 'cancelled by earlier recovery policy' } },
|
||||
})
|
||||
})
|
||||
|
||||
it('handles synchronous cancellation from the retry status event', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('temporary', 'SERVER'),
|
||||
textResponse('must not run'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter))
|
||||
const agent = context.agentLoop.create(SessionId('retry-event-cancel'), { provider: 'mock', model: 'mock' })
|
||||
context.on('session/event', (session, event) => {
|
||||
if (session === agent.session && event.type === 'llm/retry') agent.cancel('cancelled by retry observer')
|
||||
})
|
||||
const idle = waitForIdle(context, agent)
|
||||
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
await idle
|
||||
|
||||
expect(adapter.requests).toHaveLength(1)
|
||||
expect(agent.session.events.filter(event => event.type === 'llm/retry')).toHaveLength(1)
|
||||
expect(vi.getTimerCount()).toBe(0)
|
||||
})
|
||||
|
||||
it.each([
|
||||
[{ maxTransientRetries: -1 }, /maxTransientRetries/],
|
||||
[{ maxTransientRetries: 1.5 }, /maxTransientRetries/],
|
||||
[{ initialDelayMs: 0 }, /initialDelayMs/],
|
||||
[{ maxDelayMs: Number.POSITIVE_INFINITY }, /maxDelayMs/],
|
||||
[{ initialDelayMs: MAX_TIMER_DELAY_MS + 1 }, /initialDelayMs/],
|
||||
[{ maxDelayMs: MAX_TIMER_DELAY_MS + 1 }, /maxDelayMs/],
|
||||
[{ initialDelayMs: 20, maxDelayMs: 10 }, /less than or equal/],
|
||||
[{ jitterRatio: 1.1 }, /jitterRatio/],
|
||||
[{ retryableCodes: [] }, /must not be empty/],
|
||||
[{ retryableCodes: ['SERVER', 'SERVER'] }, /duplicates/],
|
||||
[{ retryableCodes: [''] }, /non-empty strings/],
|
||||
] as const)('fails direct composition for invalid config %#', (config, message) => {
|
||||
expect(() => { retry.apply(new Context(), config as retry.Config) }).toThrow(message)
|
||||
})
|
||||
})
|
||||
33
packages/llm/llm-retry/tsconfig.json
Normal file
33
packages/llm/llm-retry/tsconfig.json
Normal file
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
{
|
||||
"path": "../../core/agent"
|
||||
},
|
||||
{
|
||||
"path": "../../util/timeout"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -8,55 +8,67 @@ An adapter registry plus a single streaming call surface, interceptable via a wa
|
||||
|
||||
### Public API
|
||||
|
||||
- `ctx.llm.registerAdapter(models: string[], adapter: LlmAdapter): () => void` Register an adapter for the given model names. Disposed with the calling fiber.
|
||||
- `ctx.llm.models(): string[]` — model names with a registered adapter.
|
||||
- `ctx.llm.registerAdapter(providers: string[], adapter: LlmAdapter): () => void` Register one adapter instance for the given provider routes. Registration is all-or-nothing, and is disposed with the calling fiber.
|
||||
- `ctx.llm.listProviders(): LlmProviderInfo[]` Describe registered provider routes in registration order.
|
||||
- `ctx.llm.listModels(provider: string): Promise<LlmModelInfo[]>` Discover the models one registered provider currently advertises.
|
||||
- `ctx.llm.stream(options: GenerateOptions): AsyncIterable<StreamChunk>` Stream one model call as raw chunks (token-level deltas). Consumers assemble the chunks into blocks/messages with `BlockAssembler`.
|
||||
|
||||
`LlmService` preserves errors from final adapter selection, synchronous dispatch, iterator construction, and iteration, and binds their provenance to the exact stream handle returned for that model call. `isLlmAdapterFailure(stream, value)` reports only errors from that call's final adapter boundary; `llmFailureOf(stream, value)` returns the adjacent immutable `LlmFailure`. Nested model calls, `llm/stream` middleware, and downstream consumer failures remain unclassified for the outer call. Classification never replaces or mutates the adapter's original coded `Error`.
|
||||
|
||||
Provider and model metadata is a discovery surface, not a routing whitelist. `registerAdapter()` still owns provider exclusivity, while an adapter may accept model ids absent from `listModels()`; consumers must not reject a request because its model is unlisted. Returned metadata is detached and invalid or duplicate adapter entries fail with `INVALID_ADAPTER` or `INVALID_CATALOG`.
|
||||
|
||||
### Events
|
||||
|
||||
| Event | Mode | Purpose |
|
||||
|---|---|---|
|
||||
| `llm/stream` | waterfall | Intercept/wrap every streaming model call (retry, caching, routing) |
|
||||
| `llm/stream` | waterfall | Intercept/wrap every streaming model call for caching, logging, or routing |
|
||||
|
||||
### Extension points
|
||||
|
||||
- Subclass `LlmAdapter` and call `ctx.llm.registerAdapter(models, adapter)` to add a new model provider.
|
||||
- Wrap `llm/stream` via `ctx.on()` waterfall listeners for caching, retry, logging, rate-limiting, etc.
|
||||
- Subclass `LlmAdapter` and call `ctx.llm.registerAdapter(providers, adapter)` to add one or more provider routes. `GenerateOptions.provider` selects the adapter; `GenerateOptions.model` is adapter-owned and may be resolved dynamically. Override `providerInfo()` and asynchronous `listModels()` to expose selector metadata; their defaults use the route id as its name and advertise no models.
|
||||
- Wrap `llm/stream` via `ctx.on()` waterfall listeners for caching, logging, or routing. A wrapper that retries after emitting a chunk has no durable attempt boundary; shipped agent retry policy therefore uses `agent/request-error` instead.
|
||||
|
||||
### Content-block vocabulary (`types.ts`)
|
||||
|
||||
Messages are arrays of typed content blocks: `text`, `reasoning`, `tool-call`, `tool-result`. The union is derived from the merge-extensible `ContentBlockMap`, so plugins can add block types via declaration merging. The core set is limited to blocks every shipping path honors — multimodal content (images, audio, …) has no core block type; a feature that needs one adds it via the map together with the adapter/UI/compaction support that honors it.
|
||||
Messages are arrays of typed content blocks: `text`, `reasoning`, `tool-call`, `tool-result`. The union is derived from the merge-extensible `ContentBlockMap`, so plugins can add block types via declaration merging. Assistant messages produced by the loop also carry provider/model provenance and optional adapter-private replay state. Before dispatch, `LlmService` retains that state only when the historical provider route and target provider route are currently owned by the exact same adapter instance; the adapter then decides whether it can restore or convert the state across models/providers. The core block set is limited to blocks every shipping path honors — multimodal content (images, audio, …) has no core block type; a feature that needs one adds it via the map together with the adapter/UI/compaction support that honors it.
|
||||
|
||||
Streaming is a raw chunk protocol (`block-start`, `text-delta`, `reasoning-delta`, `tool-call-delta`, `block-end`, `usage`, `finish`). `BlockAssembler` is the single shared implementation that assembles chunks into blocks/messages.
|
||||
|
||||
### Call configuration (`call-config.ts`)
|
||||
|
||||
`LlmCallConfig` is the model + sampling scalars of one conversation's requests (`model`, `temperature`, `maxTokens`, `stop` — each mapping 1:1 onto the same-named `GenerateOptions` field). It is per-conversation state recorded in the session log as part of the request header (see the dsh-session `request/header` events), never a silently-adjustable per-call knob: the `agent/request` waterfall proposes a replacement and the loop logs a real change. `callConfigEquals(a, b)` is the field-wise real-change detector; `deepFreeze(value)` is the ownership helper the loop applies to every built request before dispatch (`llm/stream` listeners and adapters read, never rewrite).
|
||||
`LlmCallConfig` is the provider + model + sampling scalars of one conversation's requests (`provider`, `model`, `temperature`, `maxTokens`, `stop` — each mapping 1:1 onto the same-named `GenerateOptions` field). It is per-conversation state recorded in the session log as part of the request header (see the dsh-session `request/header` events), never a silently-adjustable per-call knob: the `agent/request` waterfall proposes a replacement and the loop logs a real change. `callConfigEquals(a, b)` is the field-wise real-change detector; `deepFreeze(value)` is the ownership helper the loop applies to every built request before dispatch (`llm/stream` listeners and adapters read, never rewrite).
|
||||
|
||||
### App attribution (`attribution.ts`)
|
||||
|
||||
Every product adapter sends application identity on provider HTTP requests. `attributionHeaders(identity?)` builds the standard `User-Agent`, defaulting to public `APP_IDENTITY`; white-label deployments may replace but not suppress it. Adapters verify the wire header directly or through their library hook. See [the attribution RFC](../../../docs/rfc/implemented/architecture/2026-06-21-mandatory-app-attribution-headers.md).
|
||||
Every product adapter sends application identity on provider HTTP requests. `attributionHeaders(identity?)` builds the standard `User-Agent`, defaulting to public `APP_IDENTITY`; white-label deployments may replace but not suppress it. Adapters verify the wire header directly or through their library hook. See [the attribution Agent Note](../../../.agents/notes/implemented/architecture/2026-06-21-mandatory-app-attribution-headers.md).
|
||||
|
||||
### Classes
|
||||
|
||||
- `LlmAdapter` — abstract base class for provider adapters. The only required method is `stream()`.
|
||||
- `BlockAssembler` — incrementally assembles raw chunks into complete content blocks and an assistant message. The agent loop feeds it raw chunks (logging them for replay) while reading the assembled blocks/message for history.
|
||||
- `HarnessError` — base class for the harness error taxonomy: a stable `code` string (distinct from the human `message`) plus `cause` chaining. Lives here, in the leaf package every other imports, so a single base is shared without a new dependency edge. Per-package errors (`LlmError`, `ToolArgsError`, `InvariantError`, …) extend it. `isHarnessError(value)` narrows at seams.
|
||||
- `LlmError` — extends `HarnessError`; `code` string (`NO_ADAPTER`, `DUPLICATE_ADAPTER`, and adapter codes like `AUTH`/`RATE_LIMIT`) plus an optional numeric `status` when the failure came from a non-2xx provider response.
|
||||
- `LlmError` — extends `HarnessError`; its stable `code` string (`NO_ADAPTER`, `DUPLICATE_ADAPTER`, and adapter codes like `AUTH`/`RATE_LIMIT`) matches its frozen serializable `failure.code`. The payload may also retain validated status, `Retry-After`, and branded provider request id facts; policy remains outside the error.
|
||||
- `errorChain(value)` — renders a thrown value with its full `cause` chain and AggregateError members for diagnostic surfaces (UI notices, logger lines, durable `turn/end` messages), so transport wrappers like undici's `TypeError: fetch failed` surface the underlying `ECONNREFUSED`/DNS/TLS detail instead of masking it. Rendering only — route on `code`, never by parsing the result.
|
||||
- `CONTEXT_WINDOW_EXCEEDED_CODE` — the provider-neutral code both DeepSeek adapters use when a request exceeds the model context window, regardless of thrown-HTTP versus in-band finish delivery. `isContextWindowExceededError(detail)` is their shared conservative classifier for OpenAI-compatible provider detail.
|
||||
- `QUOTA_EXCEEDED_CODE` — the non-transient provider-neutral code for exhausted account quota, balance, credits, budget, or usage limits. `isQuotaExceededError(detail)` keeps those failures distinct from request-rate limits.
|
||||
|
||||
### Real adapters
|
||||
|
||||
Two adapters implement `LlmAdapter` on different internals: [`@deepseek-ai/dsh-llm-deepseek`](../llm-deepseek) uses hand-rolled fetch/SSE, while [`@deepseek-ai/dsh-llm-pi-ai`](../llm-pi-ai) uses `@earendil-works/pi-ai`. Both follow the `StreamChunk` conventions in `types.ts`: usage precedes finish, tool arguments remain raw strings, and errors take one of two sanctioned paths. See [the twin LLM adapters](../../../docs/rfc/implemented/architecture/2026-06-13-twin-llm-adapters.md) for the design rationale.
|
||||
Two adapters implement `LlmAdapter` on different internals: [`@deepseek-ai/dsh-llm-deepseek`](../llm-deepseek) uses hand-rolled fetch/SSE for the `deepseek` route, while [`@deepseek-ai/dsh-llm-pi-ai`](../llm-pi-ai) dynamically resolves configured provider/model pairs through `@earendil-works/pi-ai`. Both follow the `StreamChunk` conventions in `types.ts`: usage precedes finish, tool arguments remain raw strings, and errors take one of two sanctioned paths. See [the twin LLM adapters](../../../.agents/notes/implemented/architecture/2026-06-13-twin-llm-adapters.md) for the design rationale.
|
||||
|
||||
## Model Experience
|
||||
|
||||
None, as this adapter registry forwards an already assembled request without adding or changing any model-bound text, schema, or message.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
Pass-through; the registry preserves the assembled request prefix, while the selected adapter and provider own cache reuse and routing boundaries.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **No retry/caching/rate-limit layer ships** — `llm/stream` is the intended wrap seam and has no production listener, so provider 429/5xx failures surface immediately.
|
||||
- **`GenerateOptions` sampling is `temperature`/`maxTokens`/`stop` only** — no `tool_choice`, `top_p`, or penalty fields; the vocabulary grows when a producer lands ([dropped inert knobs](../../../docs/rfc/implemented/simplification/2026-07-04-drop-inert-request-knobs.md)).
|
||||
- **Producer-gated variants stay out until produced** — `prefill`, per-tool `strict`, block `cache` hints, and the `agent` message-source variant were pruned as producerless ([RFC](../../../docs/rfc/implemented/simplification/2026-07-04-prune-producerless-vocabulary-variants.md)).
|
||||
- **No default retry/caching/rate-limit policy ships in this service** — `llm/stream` remains a single-attempt call-wrapper seam; the agent loop separately offers proven model-request failures to `agent/request-error`, whose default preserves the original failure. `@deepseek-ai/dsh-llm-retry` is an optional policy plugin loaded by the shared example spine.
|
||||
- **`GenerateOptions` sampling is `temperature`/`maxTokens`/`stop` only** — no `tool_choice`, `top_p`, or penalty fields; the vocabulary grows when a producer lands ([dropped inert knobs](../../../.agents/notes/implemented/simplification/2026-07-04-drop-inert-request-knobs.md)).
|
||||
- **Producer-gated variants stay out until produced** — `prefill`, per-tool `strict`, block `cache` hints, and the `agent` message-source variant were pruned as producerless ([Agent Note](../../../.agents/notes/implemented/simplification/2026-07-04-prune-producerless-vocabulary-variants.md)).
|
||||
- **`BlockAssembler` handles core block kinds only** — a plugin-added block type whose stream is never closed by `block-end` makes `blocks()` throw.
|
||||
- **`APP_IDENTITY.url` names a repository that does not exist yet** — `FIXME`: creating the public `deepseek-ai/deepseek-harness-sdk` repo gates the first release.
|
||||
- **`GenerateOptions.sessionId` is a locally-declared brand** — importing dsh-session's `SessionId` would cycle; a future ids-owning package would dissolve the workaround.
|
||||
|
||||
143
packages/llm/llm/src/adapter-failure.ts
Normal file
143
packages/llm/llm/src/adapter-failure.ts
Normal file
@@ -0,0 +1,143 @@
|
||||
/**
|
||||
* Private provider-failure tagging shared by `LlmService` and its consumers.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-llm/adapter-failure
|
||||
*/
|
||||
|
||||
import { HarnessError } from './error.ts'
|
||||
import type { LlmFailure, StreamChunk } from './types.ts'
|
||||
|
||||
/** Errors and normalized facts proven to originate in one model call's final adapter boundary. */
|
||||
export type AdapterFailureScope = WeakMap<Error, LlmFailure>
|
||||
|
||||
/** Call-local failure scopes keyed by the exact stream handle returned to a consumer. */
|
||||
const adapterFailureScopes = new WeakMap<AsyncIterable<StreamChunk>, AdapterFailureScope>()
|
||||
|
||||
/**
|
||||
* Bind one call's adapter-failure scope to a unique returned stream handle.
|
||||
* @param stream - the waterfall-selected stream for this call.
|
||||
* @param failures - errors tagged by this call's final adapter boundary.
|
||||
* @returns a unique stream handle that delegates iteration to `stream`.
|
||||
* @internal
|
||||
*/
|
||||
export function bindAdapterFailureScope(
|
||||
stream: AsyncIterable<StreamChunk>,
|
||||
failures: AdapterFailureScope,
|
||||
): AsyncIterable<StreamChunk> {
|
||||
const call = {
|
||||
[Symbol.asyncIterator](): AsyncIterator<StreamChunk> {
|
||||
return stream[Symbol.asyncIterator]()
|
||||
},
|
||||
}
|
||||
adapterFailureScopes.set(call, failures)
|
||||
return call
|
||||
}
|
||||
|
||||
/**
|
||||
* Preserve an adapter's Error identity while tagging its provider origin.
|
||||
* @param failures - the call-local final-adapter failure scope.
|
||||
* @param value - arbitrary value thrown by adapter dispatch or iteration.
|
||||
* @returns the original Error, or a coded Error wrapping a non-Error throw.
|
||||
* @internal
|
||||
*/
|
||||
export function markLlmAdapterFailure(
|
||||
failures: AdapterFailureScope,
|
||||
value: unknown,
|
||||
): Error & { code?: string } {
|
||||
const error = value instanceof Error
|
||||
? value as Error & { code?: string }
|
||||
: new HarnessError(String(value), 'UNKNOWN', { cause: value })
|
||||
const carried = error instanceof HarnessError ? ownFailureSnapshot(error) : undefined
|
||||
const failure = carried !== undefined && carried.code === error.code ? carried : Object.freeze({
|
||||
message: errorMessage(error),
|
||||
code: harnessErrorCode(error),
|
||||
})
|
||||
failures.set(error, failure)
|
||||
return error
|
||||
}
|
||||
|
||||
/** Snapshot an own data property without invoking an SDK-defined accessor. */
|
||||
function ownFailureSnapshot(error: Error): LlmFailure | undefined {
|
||||
try {
|
||||
const descriptor = Object.getOwnPropertyDescriptor(error, 'failure')
|
||||
return descriptor !== undefined && 'value' in descriptor
|
||||
? failureSnapshot(descriptor.value)
|
||||
: undefined
|
||||
} catch (_sdkPropertyTrap) {
|
||||
return undefined
|
||||
}
|
||||
}
|
||||
|
||||
/** Validate and detach an arbitrary serializable failure payload. */
|
||||
function failureSnapshot(value: unknown): LlmFailure | undefined {
|
||||
if (typeof value !== 'object' || value === null) return undefined
|
||||
try {
|
||||
const candidate = value as Partial<LlmFailure>
|
||||
const message = candidate.message
|
||||
const code = candidate.code
|
||||
const status = candidate.status
|
||||
const providerRetryAfterMs = candidate.providerRetryAfterMs
|
||||
const requestId = candidate.requestId
|
||||
if (typeof message !== 'string' || message.length === 0
|
||||
|| typeof code !== 'string' || code.length === 0
|
||||
|| (status !== undefined && (!Number.isInteger(status) || status < 100 || status > 599))
|
||||
|| (providerRetryAfterMs !== undefined
|
||||
&& (!Number.isFinite(providerRetryAfterMs) || providerRetryAfterMs <= 0))
|
||||
|| (requestId !== undefined && (typeof requestId !== 'string' || requestId.length === 0))) return undefined
|
||||
return Object.freeze({
|
||||
message,
|
||||
code,
|
||||
...status === undefined ? {} : { status },
|
||||
...providerRetryAfterMs === undefined ? {} : { providerRetryAfterMs },
|
||||
...requestId === undefined ? {} : { requestId },
|
||||
})
|
||||
} catch (_sdkFailureGetter) {
|
||||
return undefined
|
||||
}
|
||||
}
|
||||
|
||||
/** Read an SDK error message without letting an accessor replace the primary failure. */
|
||||
function errorMessage(error: Error): string {
|
||||
try {
|
||||
const message: unknown = error.message
|
||||
if (typeof message === 'string' && message.length > 0) return message
|
||||
} catch (_sdkMessageGetter) {
|
||||
// The fallback below preserves a serializable failure beside the original Error.
|
||||
}
|
||||
return 'LLM adapter failed'
|
||||
}
|
||||
|
||||
/** Trust only Harness-owned codes; third-party SDK codes are not our taxonomy. */
|
||||
function harnessErrorCode(error: Error): string {
|
||||
return error instanceof HarnessError ? error.code : 'UNKNOWN'
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a failure came from final adapter dispatch, iterator construction,
|
||||
* or iteration for the call represented by the exact returned stream handle.
|
||||
* @param stream - the exact stream returned by the model call being classified.
|
||||
* @param value - arbitrary failure caught by a model-call consumer.
|
||||
* @returns true only for errors tagged at that call's final adapter boundary.
|
||||
*/
|
||||
export function isLlmAdapterFailure(
|
||||
stream: AsyncIterable<StreamChunk>,
|
||||
value: unknown,
|
||||
): value is Error & { code?: string } {
|
||||
const failures = adapterFailureScopes.get(stream)
|
||||
return value instanceof Error && failures !== undefined && failures.has(value)
|
||||
}
|
||||
|
||||
/**
|
||||
* Retrieve normalized provider facts only for an Error tagged by this exact
|
||||
* model call's final adapter boundary.
|
||||
* @param stream - the exact stream returned to the consumer.
|
||||
* @param value - the caught failure.
|
||||
* @returns the immutable facts for that call, or `undefined` for middleware, nested, or consumer failures.
|
||||
*/
|
||||
export function llmFailureOf(
|
||||
stream: AsyncIterable<StreamChunk>,
|
||||
value: unknown,
|
||||
): LlmFailure | undefined {
|
||||
const failures = adapterFailureScopes.get(stream)
|
||||
return value instanceof Error ? failures?.get(value) : undefined
|
||||
}
|
||||
@@ -36,14 +36,13 @@ export class BlockAssembler {
|
||||
private order: number[] = []
|
||||
private _usage: TokenUsage | undefined
|
||||
private _finish: FinishReason | undefined
|
||||
private _replayState: unknown = undefined
|
||||
|
||||
/**
|
||||
* Feed one chunk. Returns the completed block when the chunk closes one
|
||||
* (an explicit `block-end`), otherwise undefined.
|
||||
* Feed one chunk into the assembly state.
|
||||
* @param chunk - the next raw chunk, in stream order.
|
||||
* @returns the authoritative block from the first `block-end` at its index; undefined for every other chunk.
|
||||
*/
|
||||
push(chunk: StreamChunk): ContentBlock | undefined {
|
||||
push(chunk: StreamChunk): void {
|
||||
switch (chunk.type) {
|
||||
case 'block-start': {
|
||||
if (!this.partials.has(chunk.index)) {
|
||||
@@ -77,7 +76,7 @@ export class BlockAssembler {
|
||||
// and the final assembled block in agreement.
|
||||
if (partial.block) return
|
||||
partial.block = chunk.block
|
||||
return chunk.block
|
||||
return
|
||||
}
|
||||
case 'usage': {
|
||||
this._usage = chunk.usage
|
||||
@@ -85,6 +84,7 @@ export class BlockAssembler {
|
||||
}
|
||||
case 'finish': {
|
||||
this._finish = chunk.reason
|
||||
this._replayState = chunk.replayState
|
||||
return
|
||||
}
|
||||
default: return assertNever(chunk, 'BlockAssembler.push')
|
||||
@@ -142,6 +142,11 @@ export class BlockAssembler {
|
||||
return this._finish ?? { kind: 'stop' }
|
||||
}
|
||||
|
||||
/** Adapter-private replay state from the terminal finish chunk, if any. */
|
||||
get replayState(): unknown {
|
||||
return this._replayState
|
||||
}
|
||||
|
||||
/**
|
||||
* The assembled assistant message.
|
||||
* @returns an assistant-role message over `blocks()` (same open-block assembly rules).
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
/**
|
||||
* Centralize the non-secret product identity every provider request sends as `User-Agent`, keeping
|
||||
* adapters from drifting. See
|
||||
* `docs/rfc/implemented/architecture/2026-06-21-mandatory-app-attribution-headers.md`.
|
||||
* `.agents/notes/implemented/architecture/2026-06-21-mandatory-app-attribution-headers.md`.
|
||||
*
|
||||
* App-attribution vocabulary for provider requests.
|
||||
* @module @deepseek-ai/dsh-llm/attribution
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
/**
|
||||
* dsh-llm's owned branded id: `CallId` (tool-call correlation).
|
||||
* dsh-llm's owned branded ids: tool-call correlation and provider request
|
||||
* diagnostics.
|
||||
*
|
||||
* The `Branded<B>` primitive itself lives in `@deepseek-ai/dsh-brand` (a
|
||||
* zero-dependency type-only package) so every owner of a cross-boundary id can
|
||||
@@ -25,3 +26,15 @@ export type CallId = Branded<'CallId'>
|
||||
export function CallId(id: string): CallId {
|
||||
return id as CallId
|
||||
}
|
||||
|
||||
/** Provider-issued request identifier retained for diagnostics across package boundaries. */
|
||||
export type ProviderRequestId = Branded<'ProviderRequestId'>
|
||||
|
||||
/**
|
||||
* Brand a provider-issued request identifier.
|
||||
* @param id - the opaque provider-issued string.
|
||||
* @returns the same string, branded; no validation is performed.
|
||||
*/
|
||||
export function ProviderRequestId(id: string): ProviderRequestId {
|
||||
return id as ProviderRequestId
|
||||
}
|
||||
|
||||
@@ -1,17 +1,18 @@
|
||||
/**
|
||||
* Conversation call configuration and freeze utilities. Model and sampling
|
||||
* values are request-header state that can affect cache reuse; request
|
||||
* waterfalls replace them and the loop logs changes instead of allowing
|
||||
* silent per-call drift.
|
||||
* Conversation call configuration and freeze utilities. Provider routing,
|
||||
* model, and sampling values are request-header state that can affect cache
|
||||
* reuse; request waterfalls replace them and the loop logs changed snapshots
|
||||
* instead of allowing silent per-call drift.
|
||||
* @module dsh-llm/call-config
|
||||
*/
|
||||
|
||||
/**
|
||||
* Model + sampling scalars of one conversation's requests. Every field maps
|
||||
* Provider + model + sampling scalars of one conversation's requests. Every field maps
|
||||
* 1:1 onto the same-named `GenerateOptions` field; the loop builds requests
|
||||
* from the logged header rather than accepting these per call.
|
||||
*/
|
||||
export interface LlmCallConfig {
|
||||
provider: string
|
||||
model: string
|
||||
temperature?: number
|
||||
maxTokens?: number
|
||||
@@ -21,13 +22,13 @@ export interface LlmCallConfig {
|
||||
/**
|
||||
* Field-wise equality over {@link LlmCallConfig} — the comparison a caller
|
||||
* runs to decide whether a proposed configuration is a real change (worth a
|
||||
* logged header delta) or the held one restated.
|
||||
* logged header snapshot) or the held one restated.
|
||||
* @param a - one configuration.
|
||||
* @param b - the other.
|
||||
* @returns whether every field (including the `stop` list, element-wise) matches.
|
||||
*/
|
||||
export function callConfigEquals(a: LlmCallConfig, b: LlmCallConfig): boolean {
|
||||
if (a.model !== b.model || a.temperature !== b.temperature || a.maxTokens !== b.maxTokens) return false
|
||||
if (a.provider !== b.provider || a.model !== b.model || a.temperature !== b.temperature || a.maxTokens !== b.maxTokens) return false
|
||||
if (a.stop === undefined || b.stop === undefined) return a.stop === b.stop
|
||||
return a.stop.length === b.stop.length && a.stop.every((s, i) => s === b.stop?.[i])
|
||||
}
|
||||
|
||||
@@ -21,6 +21,109 @@ export class HarnessError extends Error {
|
||||
}
|
||||
}
|
||||
|
||||
/** Canonical provider-neutral code for a model request rejected because its context window was exceeded. */
|
||||
export const CONTEXT_WINDOW_EXCEEDED_CODE = 'CONTEXT_WINDOW_EXCEEDED'
|
||||
|
||||
/** Canonical provider-neutral code for an exhausted account quota or balance. */
|
||||
export const QUOTA_EXCEEDED_CODE = 'QUOTA'
|
||||
|
||||
/** Structured codes and plain phrases that explicitly name a context bound being exceeded. */
|
||||
const STRUCTURED_CONTEXT_OVERFLOW = new RegExp(
|
||||
String.raw`(?:^|[^a-z0-9])context[\s_-](?:length|window)[\s_-]`
|
||||
+ String.raw`(?:exceed(?:ed|s)?|overflow(?:ed)?|limit[\s_-]exceeded)(?:$|[^a-z0-9])`,
|
||||
'i',
|
||||
)
|
||||
|
||||
/** Request-size wording that ties "too large" directly to model context capacity. */
|
||||
const TOO_LARGE_FOR_CONTEXT = new RegExp(
|
||||
String.raw`\b(?:request|prompt|input|messages?)\s+(?:is\s+|are\s+)?`
|
||||
+ String.raw`too\s+(?:large|long)\s+for\s+(?:(?:this|the)\s+)?`
|
||||
+ String.raw`(?:model(?:'s)?\s+)?context(?:\s+window)?\b`,
|
||||
'i',
|
||||
)
|
||||
|
||||
/** "Exceeds" wording is safe only when its object is explicitly the model context. */
|
||||
const EXCEEDS_MODEL_CONTEXT = new RegExp(
|
||||
String.raw`\b(?:input|prompt|request|messages?)\b.{0,40}`
|
||||
+ String.raw`\b(?:exceed(?:s|ed)?|overflows?|is\s+larger\s+than)\b.{0,40}`
|
||||
+ String.raw`\b(?:the\s+)?(?:model(?:'s)?\s+)?context(?:\s+(?:length|window))?\b`,
|
||||
'i',
|
||||
)
|
||||
|
||||
/**
|
||||
* Recognize the context-overflow wording used by OpenAI-compatible providers
|
||||
* and library adapters. Adapters pass all available provider code, type, and
|
||||
* message text so both thrown and in-band delivery styles share one classifier.
|
||||
* @param detail - provider error code/type/message text joined into one string.
|
||||
* @returns true when the detail identifies a request exceeding the model context window.
|
||||
*/
|
||||
export function isContextWindowExceededError(detail: string): boolean {
|
||||
return STRUCTURED_CONTEXT_OVERFLOW.test(detail)
|
||||
|| /\b(?:maximum|max)(?:\s+(?:allowed|supported))?\s+context\s+(?:length|window)\b/i.test(detail)
|
||||
|| TOO_LARGE_FOR_CONTEXT.test(detail)
|
||||
|| /\b(?:input|prompt|request)\s+(?:is\s+)?too\s+(?:long|large)\s+for\s+(?:this|the)\s+model\b/i.test(detail)
|
||||
|| EXCEEDS_MODEL_CONTEXT.test(detail)
|
||||
}
|
||||
|
||||
/**
|
||||
* Recognize provider wording that identifies an exhausted account quota rather
|
||||
* than a transient request-rate limit.
|
||||
* @param detail - provider error code/type/message text joined into one string.
|
||||
* @returns true only for terminal quota, balance, credit, budget, or usage-limit wording.
|
||||
*/
|
||||
export function isQuotaExceededError(detail: string): boolean {
|
||||
return /\binsufficient[\s_-]+(?:quota|balance|credits?)\b/i.test(detail)
|
||||
|| /\b(?:quota|usage[\s_-]+limit)[\s_-]+(?:exceeded|exhausted|reached)\b/i.test(detail)
|
||||
|| /\bexceed(?:ed|s)?[\s_-]+(?:(?:your|the)[\s_-]+)?(?:current[\s_-]+)?quota\b/i.test(detail)
|
||||
|| /\b(?:balance|credits?)[\s_-]+(?:exhausted|depleted)\b/i.test(detail)
|
||||
|| /\bout[\s_-]+of[\s_-]+(?:credits?|budget)\b/i.test(detail)
|
||||
}
|
||||
|
||||
/**
|
||||
* Render a thrown value with its full `cause` chain and AggregateError
|
||||
* members, so transport wrappers like undici's `TypeError: fetch failed`
|
||||
* surface the underlying failure instead of masking it. Diagnostic-surface
|
||||
* rendering only (messages, notices, logs) — never parse the result; route on
|
||||
* {@link HarnessError.code}.
|
||||
* @param value - the caught value (`unknown` in catch clauses).
|
||||
* @returns the outermost message first, each cause appended with `: ` (skipped
|
||||
* when it repeats the wrapper message verbatim), and AggregateError members
|
||||
* bracketed and `; `-joined.
|
||||
*/
|
||||
export function errorChain(value: unknown): string {
|
||||
// Tracks the active recursion path (entries removed on exit), so only true
|
||||
// cycles are flagged and a diamond-shared cause still renders in full.
|
||||
const path = new Set<unknown>()
|
||||
const render = (current: unknown): string => {
|
||||
if (path.has(current)) return '<circular cause>'
|
||||
path.add(current)
|
||||
try {
|
||||
if (!(current instanceof Error)) return String(current)
|
||||
const message = current.message === '' ? current.name : current.message
|
||||
const members = current instanceof AggregateError && current.errors.length > 0
|
||||
? ` [${current.errors.map(render).join('; ')}]`
|
||||
: ''
|
||||
const causeText = current.cause === undefined || current.cause === null
|
||||
? ''
|
||||
: render(current.cause)
|
||||
// Wrappers like `new HarnessError(String(value), code, { cause: value })`
|
||||
// repeat their cause verbatim; rendering it again would only add noise.
|
||||
const cause = causeText === '' || causeText === message ? '' : `: ${causeText}`
|
||||
return `${message}${members}${cause}`
|
||||
} catch {
|
||||
// Only hostile coercion or hostile accessors (a throwing toString /
|
||||
// Symbol.toPrimitive on a non-Error, or a throwing message/name/cause/
|
||||
// errors getter on an Error subclass): this renderer feeds UI notices
|
||||
// and logs, so nothing may escape. Inner frames catch their own throws,
|
||||
// so only the hostile node collapses, not the whole chain.
|
||||
return '<unrenderable value>'
|
||||
} finally {
|
||||
path.delete(current)
|
||||
}
|
||||
}
|
||||
return render(value)
|
||||
}
|
||||
|
||||
/**
|
||||
* Narrow an arbitrary thrown value to a HarnessError (for `instanceof` at seams).
|
||||
* @param value - the caught value (`unknown` in catch clauses).
|
||||
|
||||
@@ -7,8 +7,12 @@
|
||||
*/
|
||||
|
||||
import { Context, Service } from 'cordis'
|
||||
import type { GenerateOptions, StreamChunk } from './types.ts'
|
||||
import type { GenerateOptions, LlmFailure, LlmModelInfo, LlmProviderInfo, Message, StreamChunk } from './types.ts'
|
||||
import type { ProviderRequestId } from './brand.ts'
|
||||
import { deepFreeze } from './call-config.ts'
|
||||
import { HarnessError } from './error.ts'
|
||||
import { bindAdapterFailureScope, markLlmAdapterFailure } from './adapter-failure.ts'
|
||||
import type { AdapterFailureScope } from './adapter-failure.ts'
|
||||
|
||||
export * from './attribution.ts'
|
||||
export * from './brand.ts'
|
||||
@@ -18,6 +22,7 @@ export * from './types.ts'
|
||||
export { BlockAssembler } from './assembler.ts'
|
||||
export { callConfigEquals, deepFreeze } from './call-config.ts'
|
||||
export type { LlmCallConfig } from './call-config.ts'
|
||||
export { isLlmAdapterFailure, llmFailureOf } from './adapter-failure.ts'
|
||||
|
||||
declare module 'cordis' {
|
||||
interface Context {
|
||||
@@ -31,7 +36,7 @@ declare module 'cordis' {
|
||||
* adapter's stream, or yield your own chunks to short-circuit.
|
||||
* @param options - the full request. A LOOP-built request arrives
|
||||
* deep-frozen (mutation throws): its content is a pure function of the
|
||||
* session log (the reconstructability RFC), so listeners read it, never
|
||||
* session log (the reconstructability Agent Note), so listeners read it, never
|
||||
* rewrite it. A hand-built one-shot (compaction summarize) is the
|
||||
* caller's own object and stays mutable here.
|
||||
* @mode waterfall
|
||||
@@ -40,26 +45,83 @@ declare module 'cordis' {
|
||||
}
|
||||
}
|
||||
|
||||
/** Structured provider facts and cause accepted by {@link LlmError}. */
|
||||
export interface LlmErrorOptions extends ErrorOptions {
|
||||
/** Valid HTTP status observed at the provider boundary. */
|
||||
status?: number
|
||||
/** Positive finite provider-requested delay in milliseconds. */
|
||||
providerRetryAfterMs?: number
|
||||
/** Non-empty opaque provider request id. */
|
||||
requestId?: ProviderRequestId
|
||||
}
|
||||
|
||||
/**
|
||||
* Typed error for LLM-related failures. Extends {@link HarnessError}, so the
|
||||
* `code` string (e.g. `AUTH`, `RATE_LIMIT`, `NO_ADAPTER`) is shared taxonomy;
|
||||
* `status` carries the HTTP status when the error originated from a non-2xx
|
||||
* provider response (absent for protocol/usage errors that have no HTTP status).
|
||||
* `code` string (e.g. `AUTH`, `RATE_LIMIT`, `NO_ADAPTER`) is shared taxonomy.
|
||||
*/
|
||||
export class LlmError extends HarnessError {
|
||||
constructor(message: string, code: string, public status?: number, options?: ErrorOptions) {
|
||||
/** Serializable facts retained beside this live Error. */
|
||||
readonly failure: LlmFailure
|
||||
|
||||
/**
|
||||
* @param message - non-empty human-readable failure summary.
|
||||
* @param code - non-empty stable provider-neutral machine code.
|
||||
* @param options - optional cause and validated serializable provider facts.
|
||||
*/
|
||||
constructor(message: string, code: string, options?: LlmErrorOptions) {
|
||||
if (typeof message !== 'string' || message.length === 0) throw new Error('LlmError message must be a non-empty string')
|
||||
if (typeof code !== 'string' || code.length === 0) throw new Error('LlmError code must be a non-empty string')
|
||||
if (options?.status !== undefined
|
||||
&& (!Number.isInteger(options.status) || options.status < 100 || options.status > 599)) {
|
||||
throw new Error('LlmError status must be an integer from 100 through 599')
|
||||
}
|
||||
if (options?.providerRetryAfterMs !== undefined
|
||||
&& (!Number.isFinite(options.providerRetryAfterMs) || options.providerRetryAfterMs <= 0)) {
|
||||
throw new Error('LlmError providerRetryAfterMs must be a positive finite number')
|
||||
}
|
||||
if (options?.requestId !== undefined
|
||||
&& (typeof options.requestId !== 'string' || options.requestId.length === 0)) {
|
||||
throw new Error('LlmError requestId must be a non-empty string')
|
||||
}
|
||||
super(message, code, options)
|
||||
this.name = 'LlmError'
|
||||
this.failure = Object.freeze({
|
||||
message,
|
||||
code,
|
||||
...options?.status === undefined ? {} : { status: options.status },
|
||||
...options?.providerRetryAfterMs === undefined ? {} : { providerRetryAfterMs: options.providerRetryAfterMs },
|
||||
...options?.requestId === undefined ? {} : { requestId: options.requestId },
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Provider-wire adapter for the harness message and stream vocabulary. Register implementations
|
||||
* with `ctx.llm.registerAdapter(models, adapter)`. Every provider HTTP request must include
|
||||
* with `ctx.llm.registerAdapter(providers, adapter)`. Every provider HTTP request must include
|
||||
* `attributionHeaders()`; prove that at the wire or library header-hook boundary. The hand-rolled
|
||||
* DeepSeek and pi-ai adapters intentionally exercise this contract through different internals.
|
||||
*/
|
||||
export abstract class LlmAdapter {
|
||||
/**
|
||||
* Describe one provider route owned by this adapter.
|
||||
* @param provider - a route passed to `registerAdapter()` for this instance.
|
||||
* @returns detached display metadata whose id must equal `provider`.
|
||||
*/
|
||||
providerInfo(provider: string): LlmProviderInfo {
|
||||
return { id: provider, name: provider }
|
||||
}
|
||||
|
||||
/**
|
||||
* List models this adapter can currently advertise for one owned provider.
|
||||
* The result is advisory: an adapter may accept unlisted model ids, and
|
||||
* consumers must not turn absence into request rejection.
|
||||
* @param _provider - one provider route owned by this adapter.
|
||||
* @returns discoverable models in adapter-preferred order.
|
||||
*/
|
||||
listModels(_provider: string): Promise<readonly LlmModelInfo[]> {
|
||||
return Promise.resolve([])
|
||||
}
|
||||
|
||||
/**
|
||||
* Stream one model call as raw chunks. The only required method.
|
||||
* @param options - the fully-assembled request; implementations must honor `options.signal`.
|
||||
@@ -73,30 +135,40 @@ export abstract class LlmAdapter {
|
||||
* surface, interceptable via the `llm/stream` waterfall.
|
||||
*/
|
||||
export class LlmService extends Service {
|
||||
private adapters = new Map<string, LlmAdapter>()
|
||||
private adapters = new Map<string, { adapter: LlmAdapter; provider: LlmProviderInfo }>()
|
||||
|
||||
constructor(ctx: Context) {
|
||||
super(ctx, 'llm')
|
||||
}
|
||||
|
||||
/**
|
||||
* Register an adapter for the given model names. Throws `LlmError` with code
|
||||
* `DUPLICATE_ADAPTER` if any model already has an adapter (all-or-nothing).
|
||||
* Register an adapter for the given provider routes. Throws `LlmError` with code
|
||||
* `DUPLICATE_ADAPTER` if any provider already has an adapter (all-or-nothing).
|
||||
* Disposed with the fiber.
|
||||
* @param models - every model name this adapter should serve.
|
||||
* @param adapter - the adapter that streams calls for those models.
|
||||
* @param providers - every provider route this adapter should serve.
|
||||
* @param adapter - the adapter that streams calls for those providers.
|
||||
* @returns the disposer that unregisters all of them.
|
||||
*/
|
||||
registerAdapter(models: string[], adapter: LlmAdapter): () => void {
|
||||
registerAdapter(providers: string[], adapter: LlmAdapter): () => void {
|
||||
const dispose = this.ctx.effect(function* (this: LlmService) {
|
||||
for (const model of models) {
|
||||
if (this.adapters.has(model)) {
|
||||
throw new LlmError(`an adapter for model "${model}" is already registered`, 'DUPLICATE_ADAPTER')
|
||||
if (providers.length === 0) throw new LlmError('an adapter must register at least one provider', 'INVALID_ADAPTER')
|
||||
const unique = new Set<string>()
|
||||
const registrations: { adapter: LlmAdapter; provider: LlmProviderInfo }[] = []
|
||||
for (const provider of providers) {
|
||||
if (provider.length === 0) throw new LlmError('adapter provider names must be non-empty', 'INVALID_ADAPTER')
|
||||
if (unique.has(provider) || this.adapters.has(provider)) {
|
||||
throw new LlmError(`an adapter for provider "${provider}" is already registered`, 'DUPLICATE_ADAPTER')
|
||||
}
|
||||
const info = adapter.providerInfo(provider)
|
||||
if (typeof info.id !== 'string' || info.id !== provider || typeof info.name !== 'string' || info.name.length === 0) {
|
||||
throw new LlmError(`adapter metadata for provider "${provider}" must preserve its id and have a non-empty name`, 'INVALID_ADAPTER')
|
||||
}
|
||||
unique.add(provider)
|
||||
registrations.push({ adapter, provider: { id: info.id, name: info.name } })
|
||||
}
|
||||
for (const model of models) this.adapters.set(model, adapter)
|
||||
for (const registration of registrations) this.adapters.set(registration.provider.id, registration)
|
||||
yield () => {
|
||||
for (const model of models) this.adapters.delete(model)
|
||||
for (const provider of providers) this.adapters.delete(provider)
|
||||
}
|
||||
}.bind(this), 'llm.registerAdapter()')
|
||||
// ctx.effect's disposer returns Promise<void>; our disposer API is
|
||||
@@ -105,30 +177,134 @@ export class LlmService extends Service {
|
||||
}
|
||||
|
||||
/**
|
||||
* Model names with a registered adapter.
|
||||
* @returns the registered names, in registration order.
|
||||
* Describe provider routes with a registered adapter.
|
||||
* @returns detached provider metadata in registration order.
|
||||
*/
|
||||
models(): string[] {
|
||||
return [...this.adapters.keys()]
|
||||
listProviders(): LlmProviderInfo[] {
|
||||
return [...this.adapters.values()].map(({ provider }) => ({ ...provider }))
|
||||
}
|
||||
|
||||
private adapter(model: string): LlmAdapter {
|
||||
const adapter = this.adapters.get(model)
|
||||
if (!adapter) throw new LlmError(`no adapter registered for model "${model}"`, 'NO_ADAPTER')
|
||||
return adapter
|
||||
/**
|
||||
* Discover models advertised by one registered provider. Catalog membership
|
||||
* is advisory and never changes routing or request validation.
|
||||
* @param provider - registered provider route to inspect.
|
||||
* @returns detached model metadata in adapter-preferred order.
|
||||
*/
|
||||
async listModels(provider: string): Promise<LlmModelInfo[]> {
|
||||
const adapter = this.registration(provider).adapter
|
||||
const models = await adapter.listModels(provider)
|
||||
const seen = new Set<string>()
|
||||
return models.map((model) => {
|
||||
if (
|
||||
typeof model.provider !== 'string'
|
||||
|| model.provider !== provider
|
||||
|| typeof model.id !== 'string'
|
||||
|| model.id.length === 0
|
||||
|| typeof model.name !== 'string'
|
||||
|| model.name.length === 0
|
||||
|| (model.description !== undefined && typeof model.description !== 'string')
|
||||
|| seen.has(model.id)
|
||||
) {
|
||||
throw new LlmError(`adapter returned invalid or duplicate model metadata for provider "${provider}"`, 'INVALID_CATALOG')
|
||||
}
|
||||
seen.add(model.id)
|
||||
return {
|
||||
provider: model.provider,
|
||||
id: model.id,
|
||||
name: model.name,
|
||||
...model.description === undefined ? {} : { description: model.description },
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
private registration(provider: string): { adapter: LlmAdapter; provider: LlmProviderInfo } {
|
||||
const registration = this.adapters.get(provider)
|
||||
if (!registration) throw new LlmError(`no adapter registered for provider "${provider}"`, 'NO_ADAPTER')
|
||||
return registration
|
||||
}
|
||||
|
||||
/** Remove replay state whose historical route is owned by another adapter. */
|
||||
private forAdapter(options: GenerateOptions, adapter: LlmAdapter): GenerateOptions {
|
||||
const messages: Message[] = options.messages.map((message) => {
|
||||
const provenance = message.provenance
|
||||
if (message.role !== 'assistant' || provenance?.replayState === undefined) return message
|
||||
if (this.adapters.get(provenance.provider)?.adapter === adapter) return message
|
||||
return {
|
||||
...message,
|
||||
provenance: { provider: provenance.provider, model: provenance.model },
|
||||
}
|
||||
})
|
||||
if (messages.every((message, index) => message === options.messages[index])) return options
|
||||
const filtered = { ...options, messages }
|
||||
return Object.isFrozen(options) ? deepFreeze(filtered) : filtered
|
||||
}
|
||||
|
||||
/**
|
||||
* Final adapter boundary. It tags only failures from adapter selection,
|
||||
* synchronous dispatch, iterator construction, or iteration while preserving
|
||||
* the original Error object. Middleware outside this generator remains
|
||||
* distinguishable as plugin work. An iteration failure skips adapter cleanup
|
||||
* so it cannot suppress the primary provider error. A downstream close awaits
|
||||
* adapter cleanup, whose failures remain ordinary untagged work.
|
||||
*/
|
||||
private async * adapterStream(
|
||||
options: GenerateOptions,
|
||||
failures: AdapterFailureScope,
|
||||
): AsyncGenerator<StreamChunk> {
|
||||
let iterator: AsyncIterator<StreamChunk>
|
||||
try {
|
||||
const adapter = this.registration(options.provider).adapter
|
||||
const stream = adapter.stream(this.forAdapter(options, adapter))
|
||||
iterator = stream[Symbol.asyncIterator]()
|
||||
} catch (error: unknown) {
|
||||
throw markLlmAdapterFailure(failures, error)
|
||||
}
|
||||
|
||||
let completed = false
|
||||
let iterationFailed = false
|
||||
try {
|
||||
while (true) {
|
||||
let value: StreamChunk
|
||||
try {
|
||||
const item = await iterator.next()
|
||||
if (item.done) {
|
||||
completed = true
|
||||
return
|
||||
}
|
||||
value = item.value
|
||||
} catch (error: unknown) {
|
||||
iterationFailed = true
|
||||
throw markLlmAdapterFailure(failures, error)
|
||||
}
|
||||
// End the adapter-owned try before yielding: consumer/middleware
|
||||
// failures resumed into this generator must remain untagged.
|
||||
yield value
|
||||
}
|
||||
} finally {
|
||||
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition -- the iteration catch sets its latch before entering finally.
|
||||
if (!completed && !iterationFailed) {
|
||||
const close = iterator.return?.bind(iterator)
|
||||
if (close) await close()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Stream one model call as raw chunks (token-level deltas). Throws
|
||||
* `LlmError` with code `NO_ADAPTER` if no adapter is registered for
|
||||
* `options.model`. Dispatches through the `llm/stream` waterfall.
|
||||
* @param options - the full request; `options.model` selects the adapter.
|
||||
* `options.provider`. Replay state is retained only when the same adapter
|
||||
* instance owns its historical provider and the target provider. Final
|
||||
* adapter selection, dispatch, and iteration failures retain their original
|
||||
* Error identity and are tagged in a call-local scope for narrow agent-loop
|
||||
* request recovery; middleware and nested-call failures remain untagged for
|
||||
* the outer call.
|
||||
* @param options - the full request; `options.provider` selects the adapter.
|
||||
* @returns the chunk stream, possibly wrapped by `llm/stream` listeners.
|
||||
*/
|
||||
stream(options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
return this.ctx.waterfall(this, 'llm/stream', options, () => {
|
||||
return this.adapter(options.model).stream(options)
|
||||
})
|
||||
const failures: AdapterFailureScope = new WeakMap<Error, LlmFailure>()
|
||||
const stream = this.ctx.waterfall(this, 'llm/stream', options, () => this.adapterStream(options, failures))
|
||||
return bindAdapterFailureScope(stream, failures)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -5,7 +5,21 @@
|
||||
*/
|
||||
|
||||
import type { Branded } from '@deepseek-ai/dsh-brand'
|
||||
import type { CallId } from './brand.ts'
|
||||
import type { CallId, ProviderRequestId } from './brand.ts'
|
||||
|
||||
/** Serializable provider-boundary facts; policy decides whether they are retryable. */
|
||||
export interface LlmFailure {
|
||||
/** Human-readable provider or transport failure. */
|
||||
readonly message: string
|
||||
/** Stable provider-neutral machine-routing code. */
|
||||
readonly code: string
|
||||
/** HTTP status observed at the provider boundary, when available. */
|
||||
readonly status?: number
|
||||
/** Provider-requested delay in milliseconds, when valid and available. */
|
||||
readonly providerRetryAfterMs?: number
|
||||
/** Opaque provider-issued request identifier for diagnostics. */
|
||||
readonly requestId?: ProviderRequestId
|
||||
}
|
||||
|
||||
/** Plain text visible to the end user. */
|
||||
export interface TextBlock {
|
||||
@@ -53,10 +67,29 @@ export type ContentBlockType = keyof ContentBlockMap
|
||||
/** Any known content block, derived from {@link ContentBlockMap}; switch on `type` and fall through unknowns (merge-extensible). */
|
||||
export type ContentBlock = ContentBlockMap[ContentBlockType]
|
||||
|
||||
/** A single message in a conversation history. */
|
||||
/** Provider ownership and adapter-private replay data for an assistant message. */
|
||||
export interface AssistantProvenance {
|
||||
/** Provider route that produced the message. */
|
||||
provider: string
|
||||
/** Provider model id that produced the message. */
|
||||
model: string
|
||||
/**
|
||||
* Lossless-JSON adapter state needed to replay the provider response.
|
||||
* `LlmService` exposes it to a target adapter only when that adapter instance
|
||||
* currently owns both this historical provider and the target provider.
|
||||
*/
|
||||
replayState?: unknown
|
||||
}
|
||||
|
||||
/**
|
||||
* A single message in a conversation history. Loop-derived assistant messages
|
||||
* always carry provenance; callers may omit it on hand-built foreign history.
|
||||
*/
|
||||
export interface Message {
|
||||
role: 'system' | 'user' | 'assistant'
|
||||
content: ContentBlock[]
|
||||
/** Present only on assistant messages produced by a routed adapter. */
|
||||
provenance?: AssistantProvenance
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -79,8 +112,8 @@ export interface FinishReasonMap {
|
||||
'stop': { kind: 'stop' }
|
||||
'tool-calls': { kind: 'tool-calls' }
|
||||
'max-tokens': { kind: 'max-tokens' }
|
||||
'aborted': { kind: 'aborted' }
|
||||
'error': { kind: 'error'; message: string; code?: string }
|
||||
'aborted': { kind: 'aborted'; failure: LlmFailure }
|
||||
'error': { kind: 'error'; failure: LlmFailure }
|
||||
}
|
||||
|
||||
/** Any known finish reason, derived from {@link FinishReasonMap}; switch on `kind` and fall through unknowns (merge-extensible). */
|
||||
@@ -102,6 +135,26 @@ export interface TokenUsage {
|
||||
reasoningTokens?: number
|
||||
}
|
||||
|
||||
/** Display metadata for one registered provider route. */
|
||||
export interface LlmProviderInfo {
|
||||
/** Provider route key used by {@link GenerateOptions.provider}. */
|
||||
id: string
|
||||
/** Human-readable provider name for selectors and diagnostics. */
|
||||
name: string
|
||||
}
|
||||
|
||||
/** One adapter-discovered model; catalog membership is advisory, not request validation. */
|
||||
export interface LlmModelInfo {
|
||||
/** Provider route that owns this model entry. */
|
||||
provider: string
|
||||
/** Model id passed to {@link GenerateOptions.model}. */
|
||||
id: string
|
||||
/** Human-readable model name for selectors. */
|
||||
name: string
|
||||
/** Optional user-facing distinction from otherwise similar models. */
|
||||
description?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Raw streaming protocol emitted by adapters.
|
||||
* Block indexes correlate interleaved deltas, and `block-end` carries the
|
||||
@@ -116,7 +169,12 @@ export type StreamChunk =
|
||||
| { type: 'tool-call-delta'; index: number; id: CallId; name?: string; argumentsDelta: string }
|
||||
| { type: 'block-end'; index: number; block: ContentBlock }
|
||||
| { type: 'usage'; usage: TokenUsage }
|
||||
| { type: 'finish'; reason: FinishReason }
|
||||
| {
|
||||
type: 'finish'
|
||||
reason: FinishReason
|
||||
/** Adapter-private lossless-JSON state for replaying a successful response. */
|
||||
replayState?: unknown
|
||||
}
|
||||
|
||||
/**
|
||||
* JSON-schema description of a tool, as sent to the model.
|
||||
@@ -134,6 +192,8 @@ export interface ToolSchema {
|
||||
|
||||
/** A single model request, fully assembled. */
|
||||
export interface GenerateOptions {
|
||||
/** Registered provider route selecting the adapter instance. */
|
||||
provider: string
|
||||
model: string
|
||||
/**
|
||||
* Ordered conversation messages, exactly as the provider sees them (after
|
||||
|
||||
@@ -29,12 +29,12 @@ describe('BlockAssembler', () => {
|
||||
expect(assembler.message().role).toBe('assistant')
|
||||
})
|
||||
|
||||
it('returns the completed block from push() on block-end', () => {
|
||||
it('records the completed block from block-end', () => {
|
||||
const assembler = new BlockAssembler()
|
||||
expect(assembler.push({ type: 'block-start', index: 0, blockType: 'text' })).toBeUndefined()
|
||||
expect(assembler.push({ type: 'text-delta', index: 0, text: 'hi' })).toBeUndefined()
|
||||
const block = assembler.push({ type: 'block-end', index: 0, block: { type: 'text', text: 'hi' } })
|
||||
expect(block).toEqual({ type: 'text', text: 'hi' })
|
||||
assembler.push({ type: 'block-start', index: 0, blockType: 'text' })
|
||||
assembler.push({ type: 'text-delta', index: 0, text: 'hi' })
|
||||
assembler.push({ type: 'block-end', index: 0, block: { type: 'text', text: 'hi' } })
|
||||
expect(assembler.blocks()).toEqual([{ type: 'text', text: 'hi' }])
|
||||
})
|
||||
|
||||
it('tolerates deltas without explicit block-start/end', () => {
|
||||
@@ -57,8 +57,8 @@ describe('BlockAssembler', () => {
|
||||
// push a delta first to guarantee the partial exists
|
||||
assembler.push({ type: 'text-delta', index: 0, text: 'hi' })
|
||||
// block-end's ensure() must find the existing partial (the second branch path)
|
||||
const block = assembler.push({ type: 'block-end', index: 0, block: { type: 'text', text: 'hi' } })
|
||||
expect(block).toEqual({ type: 'text', text: 'hi' })
|
||||
assembler.push({ type: 'block-end', index: 0, block: { type: 'text', text: 'hi' } })
|
||||
expect(assembler.blocks()).toEqual([{ type: 'text', text: 'hi' }])
|
||||
})
|
||||
|
||||
it('throws from assemble() when a partial has an unhandled blockType', () => {
|
||||
@@ -128,7 +128,7 @@ describe('assertNever', () => {
|
||||
|
||||
it('BlockAssembler.push rejects chunks outside the closed StreamChunk union', () => {
|
||||
const assembler = new BlockAssembler()
|
||||
expect(() => assembler.push({ type: 'rogue-chunk' } as unknown as StreamChunk))
|
||||
expect(() => { assembler.push({ type: 'rogue-chunk' } as unknown as StreamChunk) })
|
||||
.toThrow('unreachable variant in BlockAssembler.push')
|
||||
})
|
||||
})
|
||||
@@ -140,26 +140,8 @@ describe('BlockAssembler duplicate-close contract', () => {
|
||||
{ type: 'block-end', index: 0, block: { type: 'reasoning', text: 'first' } },
|
||||
{ type: 'block-end', index: 0, block: { type: 'text', text: 'second' } },
|
||||
]
|
||||
const streaming = new BlockAssembler()
|
||||
const closed = []
|
||||
for (const chunk of chunks) {
|
||||
const block = streaming.push(chunk)
|
||||
if (block) closed.push(block)
|
||||
}
|
||||
|
||||
const oneShot = new BlockAssembler()
|
||||
for (const chunk of chunks) oneShot.push(chunk)
|
||||
|
||||
expect(closed).toEqual([{ type: 'reasoning', text: 'first' }])
|
||||
expect(oneShot.blocks()).toEqual([{ type: 'reasoning', text: 'first' }])
|
||||
expect(closed).toEqual(oneShot.blocks())
|
||||
})
|
||||
|
||||
it('push returns undefined for a duplicate block-end (it closed nothing)', () => {
|
||||
const a = new BlockAssembler()
|
||||
expect(a.push({ type: 'block-end', index: 0, block: { type: 'text', text: 'x' } }))
|
||||
.toEqual({ type: 'text', text: 'x' })
|
||||
expect(a.push({ type: 'block-end', index: 0, block: { type: 'text', text: 'y' } }))
|
||||
.toBeUndefined()
|
||||
const assembler = new BlockAssembler()
|
||||
for (const chunk of chunks) assembler.push(chunk)
|
||||
expect(assembler.blocks()).toEqual([{ type: 'reasoning', text: 'first' }])
|
||||
})
|
||||
})
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
/**
|
||||
* call-config unit tests: field-wise LlmCallConfig equality (the real-change
|
||||
* detector behind logged header deltas) and the deepFreeze ownership helper
|
||||
* detector behind logged changed headers) and the deepFreeze ownership helper
|
||||
* the loop applies to every built request.
|
||||
*/
|
||||
|
||||
@@ -9,14 +9,16 @@ import { callConfigEquals, deepFreeze } from '../src/call-config.ts'
|
||||
|
||||
describe('callConfigEquals', () => {
|
||||
it('compares every field, including the stop list element-wise', () => {
|
||||
expect(callConfigEquals({ model: 'm' }, { model: 'm' })).toBe(true)
|
||||
expect(callConfigEquals({ model: 'm' }, { model: 'x' })).toBe(false)
|
||||
expect(callConfigEquals({ model: 'm', temperature: 0.5 }, { model: 'm' })).toBe(false)
|
||||
expect(callConfigEquals({ model: 'm', maxTokens: 1 }, { model: 'm', maxTokens: 2 })).toBe(false)
|
||||
expect(callConfigEquals({ model: 'm', stop: ['a'] }, { model: 'm' })).toBe(false)
|
||||
expect(callConfigEquals({ model: 'm', stop: ['a'] }, { model: 'm', stop: ['a', 'b'] })).toBe(false)
|
||||
expect(callConfigEquals({ model: 'm', stop: ['a'] }, { model: 'm', stop: ['b'] })).toBe(false)
|
||||
expect(callConfigEquals({ model: 'm', stop: ['a', 'b'] }, { model: 'm', stop: ['a', 'b'] })).toBe(true)
|
||||
const base = { provider: 'p', model: 'm' }
|
||||
expect(callConfigEquals(base, base)).toBe(true)
|
||||
expect(callConfigEquals(base, { provider: 'x', model: 'm' })).toBe(false)
|
||||
expect(callConfigEquals(base, { provider: 'p', model: 'x' })).toBe(false)
|
||||
expect(callConfigEquals({ ...base, temperature: 0.5 }, base)).toBe(false)
|
||||
expect(callConfigEquals({ ...base, maxTokens: 1 }, { ...base, maxTokens: 2 })).toBe(false)
|
||||
expect(callConfigEquals({ ...base, stop: ['a'] }, base)).toBe(false)
|
||||
expect(callConfigEquals({ ...base, stop: ['a'] }, { ...base, stop: ['a', 'b'] })).toBe(false)
|
||||
expect(callConfigEquals({ ...base, stop: ['a'] }, { ...base, stop: ['b'] })).toBe(false)
|
||||
expect(callConfigEquals({ ...base, stop: ['a', 'b'] }, { ...base, stop: ['a', 'b'] })).toBe(true)
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
/**
|
||||
* Property-based tests for the BlockAssembler (the property-testing RFC).
|
||||
* Property-based tests for the BlockAssembler (the property-testing Agent Note).
|
||||
*
|
||||
* The assembler is protocol-shaped: arbitrary interleavings of block-start,
|
||||
* deltas, block-end, usage, and finish — valid and malformed (duplicate
|
||||
@@ -41,7 +41,10 @@ const chunkArb: fc.Arbitrary<StreamChunk> = indexArb.chain(index => fc.oneof(
|
||||
fc.constant<StreamChunk>({ type: 'usage', usage: { inputTokens: 1, outputTokens: 1 } }),
|
||||
fc.constant<StreamChunk>({ type: 'finish', reason: { kind: 'stop' } }),
|
||||
fc.constant<StreamChunk>({ type: 'finish', reason: { kind: 'tool-calls' } }),
|
||||
fc.string().map((message): StreamChunk => ({ type: 'finish', reason: { kind: 'error', message } })),
|
||||
fc.string({ minLength: 1 }).map((message): StreamChunk => ({
|
||||
type: 'finish',
|
||||
reason: { kind: 'error', failure: { message, code: 'UNKNOWN' } },
|
||||
})),
|
||||
))
|
||||
|
||||
/** A stream is an arbitrary list of chunks (we do NOT force a terminal finish). */
|
||||
|
||||
@@ -1,6 +1,19 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import LlmService, { GenerateOptions, LlmAdapter, LlmError, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import LlmService, {
|
||||
errorChain,
|
||||
GenerateOptions,
|
||||
HarnessError,
|
||||
isContextWindowExceededError,
|
||||
isQuotaExceededError,
|
||||
isLlmAdapterFailure,
|
||||
LlmAdapter,
|
||||
LlmError,
|
||||
llmFailureOf,
|
||||
ProviderRequestId,
|
||||
StreamChunk,
|
||||
} from '@deepseek-ai/dsh-llm'
|
||||
import type { LlmModelInfo, LlmProviderInfo } from '@deepseek-ai/dsh-llm'
|
||||
|
||||
class ScriptedAdapter extends LlmAdapter {
|
||||
constructor(private script: StreamChunk[]) {
|
||||
@@ -12,6 +25,42 @@ class ScriptedAdapter extends LlmAdapter {
|
||||
}
|
||||
}
|
||||
|
||||
class RecordingAdapter extends ScriptedAdapter {
|
||||
lastOptions: GenerateOptions | undefined
|
||||
|
||||
override async * stream(options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
this.lastOptions = options
|
||||
yield * super.stream(options)
|
||||
}
|
||||
}
|
||||
|
||||
class ThrowingAdapter extends LlmAdapter {
|
||||
constructor(private readonly failure: Error) {
|
||||
super()
|
||||
}
|
||||
|
||||
stream(_options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
throw this.failure
|
||||
}
|
||||
}
|
||||
|
||||
class CatalogAdapter extends ScriptedAdapter {
|
||||
constructor(
|
||||
private readonly provider: LlmProviderInfo,
|
||||
private readonly models: readonly LlmModelInfo[],
|
||||
) {
|
||||
super(SCRIPT)
|
||||
}
|
||||
|
||||
override providerInfo(_provider: string): LlmProviderInfo {
|
||||
return this.provider
|
||||
}
|
||||
|
||||
override listModels(_provider: string): Promise<readonly LlmModelInfo[]> {
|
||||
return Promise.resolve(this.models)
|
||||
}
|
||||
}
|
||||
|
||||
const SCRIPT: StreamChunk[] = [
|
||||
{ type: 'block-start', index: 0, blockType: 'text' },
|
||||
{ type: 'text-delta', index: 0, text: 'hi' },
|
||||
@@ -19,22 +68,538 @@ const SCRIPT: StreamChunk[] = [
|
||||
]
|
||||
|
||||
describe('LlmService', () => {
|
||||
it('recognizes structured and model-capacity context-window overflow details', () => {
|
||||
expect(isContextWindowExceededError('context_length_exceeded maximum context length')).toBe(true)
|
||||
expect(isContextWindowExceededError('context-window-overflowed')).toBe(true)
|
||||
expect(isContextWindowExceededError('This model maximum context length is 128000 tokens')).toBe(true)
|
||||
expect(isContextWindowExceededError('input is too long for this model')).toBe(true)
|
||||
expect(isContextWindowExceededError('request too large for model context')).toBe(true)
|
||||
expect(isContextWindowExceededError('input exceeds the model context window limit')).toBe(true)
|
||||
})
|
||||
|
||||
it('does not mistake unrelated input validation for context-window overflow', () => {
|
||||
expect(isContextWindowExceededError('invalid request: malformed tool arguments')).toBe(false)
|
||||
expect(isContextWindowExceededError('invalid input: temperature exceeds maximum allowed value')).toBe(false)
|
||||
expect(isContextWindowExceededError('input exceeds maximum allowed value')).toBe(false)
|
||||
expect(isContextWindowExceededError('context window size must be positive')).toBe(false)
|
||||
})
|
||||
|
||||
it('distinguishes exhausted account quota from transient rate limiting', () => {
|
||||
for (const detail of [
|
||||
'insufficient_quota',
|
||||
'account balance depleted',
|
||||
'usage-limit-exceeded',
|
||||
'out of credits',
|
||||
'OpenAI API error (429): You exceeded your current quota, please check your plan and billing details.',
|
||||
]) expect(isQuotaExceededError(detail)).toBe(true)
|
||||
expect(isQuotaExceededError('HTTP 429: rate limit reached')).toBe(false)
|
||||
expect(isQuotaExceededError('quota resets in one minute')).toBe(false)
|
||||
})
|
||||
|
||||
it('errorChain renders the full cause chain of a wrapped transport failure', () => {
|
||||
const chain = new TypeError('fetch failed', { cause: new Error('connect ECONNREFUSED 127.0.0.1:443') })
|
||||
expect(errorChain(chain)).toBe('fetch failed: connect ECONNREFUSED 127.0.0.1:443')
|
||||
})
|
||||
|
||||
it('errorChain renders AggregateError members (Happy Eyeballs multi-address failures)', () => {
|
||||
const aggregate = new AggregateError(
|
||||
[new Error('connect ECONNREFUSED ::1:443'), new Error('connect ECONNREFUSED 127.0.0.1:443')],
|
||||
'',
|
||||
)
|
||||
const wrapped = new TypeError('fetch failed', { cause: aggregate })
|
||||
expect(errorChain(wrapped)).toBe(
|
||||
'fetch failed: AggregateError [connect ECONNREFUSED ::1:443; connect ECONNREFUSED 127.0.0.1:443]',
|
||||
)
|
||||
})
|
||||
|
||||
it('errorChain survives non-Error values, hostile coercion, and circular causes', () => {
|
||||
expect(errorChain('plain string')).toBe('plain string')
|
||||
expect(errorChain({ toString: () => { throw new Error('hostile') } })).toBe('<unrenderable value>')
|
||||
const circular = new Error('outer')
|
||||
circular.cause = circular
|
||||
expect(errorChain(circular)).toBe('outer: <circular cause>')
|
||||
// A hostile accessor collapses only its own node, not the whole chain.
|
||||
const hostileNode = new Error('node')
|
||||
Object.defineProperty(hostileNode, 'message', { get() { throw new Error('hostile getter') } })
|
||||
expect(errorChain(new Error('outer', { cause: hostileNode }))).toBe('outer: <unrenderable value>')
|
||||
// A diamond-shared (non-cyclic) cause renders in full on both paths.
|
||||
const shared = new Error('shared')
|
||||
const diamond = new AggregateError([new Error('a', { cause: shared }), new Error('b', { cause: shared })], 'agg')
|
||||
expect(errorChain(diamond)).toBe('agg [a: shared; b: shared]')
|
||||
})
|
||||
|
||||
it('errorChain falls back to the error name, skips empty aggregates, and stops at null causes', () => {
|
||||
expect(errorChain(new TypeError('', { cause: null }))).toBe('TypeError')
|
||||
expect(errorChain(new AggregateError([], 'all failed'))).toBe('all failed')
|
||||
})
|
||||
|
||||
it('errorChain collapses a cause that repeats the wrapper message verbatim', () => {
|
||||
// The `new HarnessError(String(value), code, { cause: value })` normalization
|
||||
// pattern repeats its cause; rendering it twice would only add noise.
|
||||
const wrapped = new HarnessError('boom', 'UNKNOWN', { cause: 'boom' })
|
||||
expect(errorChain(wrapped)).toBe('boom')
|
||||
})
|
||||
|
||||
it('routes stream() to the registered adapter', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-provider'], new ScriptedAdapter(SCRIPT))
|
||||
|
||||
const chunks: StreamChunk[] = []
|
||||
for await (const chunk of ctx.llm.stream({ provider: 'test-provider', model: 'test-model', messages: [] })) chunks.push(chunk)
|
||||
expect(chunks).toEqual(SCRIPT)
|
||||
})
|
||||
|
||||
it('throws NO_ADAPTER for unregistered providers', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
const stream = ctx.llm.stream({ provider: 'nope', model: 'any-model', messages: [] })
|
||||
let caught: unknown
|
||||
try {
|
||||
for await (const _ of stream) { /* drain */ }
|
||||
} catch (error: unknown) {
|
||||
caught = error
|
||||
}
|
||||
expect(caught).toBeInstanceOf(LlmError)
|
||||
expect((caught as LlmError).code).toBe('NO_ADAPTER')
|
||||
expect((caught as LlmError).message).toContain('no adapter registered')
|
||||
expect(isLlmAdapterFailure(stream, caught)).toBe(true)
|
||||
})
|
||||
|
||||
it.each(['done', 'value'] as const)('tags a throwing IteratorResult.%s getter without replacing its Error', async (field) => {
|
||||
const original = new LlmError(`${field} getter failed`, 'RESULT_GETTER_FAILED')
|
||||
const result = field === 'done' ? {} : { done: false }
|
||||
Object.defineProperty(result, field, { get: () => { throw original } })
|
||||
let cleanupLookups = 0
|
||||
const iterator: AsyncIterator<StreamChunk> = {
|
||||
next: () => Promise.resolve(result as unknown as IteratorResult<StreamChunk>),
|
||||
}
|
||||
Object.defineProperty(iterator, 'return', {
|
||||
get: () => {
|
||||
cleanupLookups += 1
|
||||
throw new Error('return getter must not run after iteration fails')
|
||||
},
|
||||
})
|
||||
const adapter = new class extends LlmAdapter {
|
||||
stream(_options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
return {
|
||||
[Symbol.asyncIterator](): AsyncIterator<StreamChunk> {
|
||||
return iterator
|
||||
},
|
||||
}
|
||||
}
|
||||
}()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-model'], adapter)
|
||||
|
||||
const stream = ctx.llm.stream({ provider: 'test-model', model: 'test-model', messages: [] })
|
||||
let caught: unknown
|
||||
try {
|
||||
for await (const _chunk of stream) { /* drain */ }
|
||||
} catch (error: unknown) {
|
||||
caught = error
|
||||
}
|
||||
|
||||
expect(caught).toBe(original)
|
||||
expect(isLlmAdapterFailure(stream, caught)).toBe(true)
|
||||
expect(cleanupLookups).toBe(0)
|
||||
})
|
||||
|
||||
it.each(['dispatch', 'iterator'] as const)('tags synchronous adapter %s failures without replacing their Error', async (boundary) => {
|
||||
const original = new LlmError(`${boundary} failed`, 'BOUNDARY_FAILED')
|
||||
const adapter = new class extends LlmAdapter {
|
||||
stream(_options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
if (boundary === 'dispatch') throw original
|
||||
return { [Symbol.asyncIterator]: () => { throw original } }
|
||||
}
|
||||
}()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-model'], adapter)
|
||||
|
||||
const stream = ctx.llm.stream({ provider: 'test-model', model: 'test-model', messages: [] })
|
||||
let caught: unknown
|
||||
try {
|
||||
for await (const _chunk of stream) { /* drain */ }
|
||||
} catch (error: unknown) {
|
||||
caught = error
|
||||
}
|
||||
|
||||
expect(caught).toBe(original)
|
||||
expect(isLlmAdapterFailure(stream, caught)).toBe(true)
|
||||
expect(llmFailureOf(stream, caught)).toEqual({
|
||||
message: `${boundary} failed`,
|
||||
code: 'BOUNDARY_FAILED',
|
||||
})
|
||||
})
|
||||
|
||||
it('keeps structured provider facts beside a frozen third-party Error', async () => {
|
||||
const original = new LlmError('provider busy', 'RATE_LIMIT', {
|
||||
status: 429,
|
||||
providerRetryAfterMs: 1_500,
|
||||
requestId: ProviderRequestId('req-7'),
|
||||
})
|
||||
Object.freeze(original)
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-provider'], new ThrowingAdapter(original))
|
||||
|
||||
const stream = ctx.llm.stream({ provider: 'test-provider', model: 'test-model', messages: [] })
|
||||
let caught: unknown
|
||||
try {
|
||||
for await (const _chunk of stream) { /* drain */ }
|
||||
} catch (error: unknown) {
|
||||
caught = error
|
||||
}
|
||||
|
||||
expect(caught).toBe(original)
|
||||
expect(llmFailureOf(stream, caught)).toEqual({
|
||||
message: 'provider busy',
|
||||
code: 'RATE_LIMIT',
|
||||
status: 429,
|
||||
providerRetryAfterMs: 1_500,
|
||||
requestId: ProviderRequestId('req-7'),
|
||||
})
|
||||
})
|
||||
|
||||
it('does not trust retry facts carried by an unknown third-party Error', async () => {
|
||||
const carried = { message: 'busy', code: 'SERVER', status: 503 }
|
||||
const original = Object.assign(new Error('busy'), { failure: carried })
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-provider'], new ThrowingAdapter(original))
|
||||
|
||||
const stream = ctx.llm.stream({ provider: 'test-provider', model: 'test-model', messages: [] })
|
||||
await expect((async () => {
|
||||
for await (const _chunk of stream) { /* drain */ }
|
||||
})()).rejects.toBe(original)
|
||||
const facts = llmFailureOf(stream, original)
|
||||
carried.status = 500
|
||||
|
||||
expect(facts).toEqual({ message: 'busy', code: 'UNKNOWN' })
|
||||
expect(Object.isFrozen(facts)).toBe(true)
|
||||
expect(facts).not.toBe(carried)
|
||||
})
|
||||
|
||||
it('keeps an unknown SDK Error exact without trusting its private code or accessors', async () => {
|
||||
const original = Object.assign(new Error('socket closed'), { code: 'ECONNRESET' })
|
||||
Object.defineProperty(original, 'failure', {
|
||||
get() { throw new Error('SDK failure accessor must not run') },
|
||||
})
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-provider'], new ThrowingAdapter(original))
|
||||
|
||||
const stream = ctx.llm.stream({ provider: 'test-provider', model: 'test-model', messages: [] })
|
||||
await expect((async () => {
|
||||
for await (const _chunk of stream) { /* drain */ }
|
||||
})()).rejects.toBe(original)
|
||||
|
||||
expect(original.code).toBe('ECONNRESET')
|
||||
expect(llmFailureOf(stream, original)).toEqual({ message: 'socket closed', code: 'UNKNOWN' })
|
||||
})
|
||||
|
||||
it('keeps an SDK Error exact when its message accessor is hostile', async () => {
|
||||
const original = Object.defineProperty(new Error(), 'message', {
|
||||
get() { throw new Error('SDK message accessor trap') },
|
||||
})
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-provider'], new ThrowingAdapter(original))
|
||||
const stream = ctx.llm.stream({ provider: 'test-provider', model: 'test-model', messages: [] })
|
||||
|
||||
await expect((async () => {
|
||||
for await (const _chunk of stream) { /* drain */ }
|
||||
})()).rejects.toBe(original)
|
||||
expect(llmFailureOf(stream, original)).toEqual({ message: 'LLM adapter failed', code: 'UNKNOWN' })
|
||||
})
|
||||
|
||||
it('falls back safely when SDK objects trap failure inspection or expose malformed facts', async () => {
|
||||
const propertyTrap = new Proxy(new HarnessError('descriptor trapped', 'SERVER'), {
|
||||
getOwnPropertyDescriptor(target, property) {
|
||||
if (property === 'failure') throw new Error('SDK descriptor trap')
|
||||
return Reflect.getOwnPropertyDescriptor(target, property)
|
||||
},
|
||||
})
|
||||
const throwingFacts = Object.create(null) as Record<string, unknown>
|
||||
Object.defineProperty(throwingFacts, 'message', {
|
||||
get() { throw new Error('SDK fact getter trap') },
|
||||
})
|
||||
const carrying = (message: string, failure: unknown): HarnessError => Object.defineProperty(
|
||||
new HarnessError(message, 'SERVER'),
|
||||
'failure',
|
||||
{ value: failure },
|
||||
)
|
||||
const factGetter = carrying('fact getter failed', throwingFacts)
|
||||
const malformed = carrying('malformed facts', { message: 'provider busy', code: 'SERVER', requestId: 1 })
|
||||
const primitive = carrying('primitive facts', 1)
|
||||
const nullFacts = carrying('null facts', null)
|
||||
const mismatched = carrying('mismatched facts', { message: 'busy', code: 'RATE_LIMIT' })
|
||||
|
||||
for (const [original, expectedMessage] of [
|
||||
[propertyTrap, 'descriptor trapped'],
|
||||
[factGetter, 'fact getter failed'],
|
||||
[malformed, 'malformed facts'],
|
||||
[primitive, 'primitive facts'],
|
||||
[nullFacts, 'null facts'],
|
||||
[mismatched, 'mismatched facts'],
|
||||
] as const) {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-provider'], new ThrowingAdapter(original))
|
||||
const stream = ctx.llm.stream({ provider: 'test-provider', model: 'test-model', messages: [] })
|
||||
|
||||
await expect((async () => {
|
||||
for await (const _chunk of stream) { /* drain */ }
|
||||
})()).rejects.toBe(original)
|
||||
expect(llmFailureOf(stream, original)).toEqual({ message: expectedMessage, code: 'SERVER' })
|
||||
}
|
||||
})
|
||||
|
||||
it('retains a stable code from a HarnessError without requiring LlmError facts', async () => {
|
||||
const original = new HarnessError('stable adapter failure', 'ADAPTER_STABLE')
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-provider'], new ThrowingAdapter(original))
|
||||
const stream = ctx.llm.stream({ provider: 'test-provider', model: 'test-model', messages: [] })
|
||||
|
||||
await expect((async () => {
|
||||
for await (const _chunk of stream) { /* drain */ }
|
||||
})()).rejects.toBe(original)
|
||||
expect(llmFailureOf(stream, original)).toEqual({
|
||||
message: 'stable adapter failure',
|
||||
code: 'ADAPTER_STABLE',
|
||||
})
|
||||
expect(llmFailureOf(stream, 'not an Error')).toBeUndefined()
|
||||
expect(llmFailureOf({ [Symbol.asyncIterator]: () => stream[Symbol.asyncIterator]() }, original)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('keeps a nested adapter failure scoped to the nested model call', async () => {
|
||||
const original = new LlmError('nested provider failed', 'NESTED_FAILED')
|
||||
const outer = new RecordingAdapter(SCRIPT)
|
||||
const nested = new ThrowingAdapter(original)
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['outer'], outer)
|
||||
ctx.llm.registerAdapter(['nested'], nested)
|
||||
let nestedStream: AsyncIterable<StreamChunk> | undefined
|
||||
ctx.on('llm/stream', (options, next) => {
|
||||
if (options.provider !== 'outer') return next()
|
||||
return (async function* () {
|
||||
nestedStream = ctx.llm.stream({ provider: 'nested', model: 'nested', messages: [] })
|
||||
yield * nestedStream
|
||||
})()
|
||||
})
|
||||
|
||||
const outerStream = ctx.llm.stream({ provider: 'outer', model: 'outer', messages: [] })
|
||||
let caught: unknown
|
||||
try {
|
||||
for await (const _chunk of outerStream) { /* drain */ }
|
||||
} catch (error: unknown) {
|
||||
caught = error
|
||||
}
|
||||
|
||||
expect(caught).toBe(original)
|
||||
expect(nestedStream).toBeDefined()
|
||||
expect(isLlmAdapterFailure(nestedStream!, caught)).toBe(true)
|
||||
expect(isLlmAdapterFailure(outerStream, caught)).toBe(false)
|
||||
expect(outer.lastOptions).toBeUndefined()
|
||||
})
|
||||
|
||||
it('keeps call scopes distinct when middleware reuses an iterable', async () => {
|
||||
const firstFailure = new LlmError('first provider failed', 'FIRST_FAILED')
|
||||
const secondFailure = new LlmError('second provider failed', 'SECOND_FAILED')
|
||||
const delegates: AsyncIterable<StreamChunk>[] = []
|
||||
const shared: AsyncIterable<StreamChunk> = {
|
||||
[Symbol.asyncIterator](): AsyncIterator<StreamChunk> {
|
||||
const delegate = delegates.shift()
|
||||
if (delegate === undefined) throw new Error('shared stream has no call delegate')
|
||||
return delegate[Symbol.asyncIterator]()
|
||||
},
|
||||
}
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['first'], new ThrowingAdapter(firstFailure))
|
||||
ctx.llm.registerAdapter(['second'], new ThrowingAdapter(secondFailure))
|
||||
ctx.on('llm/stream', (_options, next) => {
|
||||
delegates.push(next())
|
||||
return shared
|
||||
})
|
||||
|
||||
const firstStream = ctx.llm.stream({ provider: 'first', model: 'first', messages: [] })
|
||||
const secondStream = ctx.llm.stream({ provider: 'second', model: 'second', messages: [] })
|
||||
const catchFailure = async (stream: AsyncIterable<StreamChunk>): Promise<unknown> => {
|
||||
try {
|
||||
for await (const _chunk of stream) { /* drain */ }
|
||||
} catch (error: unknown) {
|
||||
return error
|
||||
}
|
||||
return new Error('expected adapter to fail')
|
||||
}
|
||||
|
||||
expect(firstStream).not.toBe(secondStream)
|
||||
const firstCaught = await catchFailure(firstStream)
|
||||
expect(firstCaught).toBe(firstFailure)
|
||||
expect(isLlmAdapterFailure(firstStream, firstCaught)).toBe(true)
|
||||
expect(isLlmAdapterFailure(secondStream, firstCaught)).toBe(false)
|
||||
const secondCaught = await catchFailure(secondStream)
|
||||
expect(secondCaught).toBe(secondFailure)
|
||||
expect(isLlmAdapterFailure(secondStream, secondCaught)).toBe(true)
|
||||
expect(isLlmAdapterFailure(firstStream, secondCaught)).toBe(false)
|
||||
expect(delegates).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('propagates a rejected next promptly without awaiting a non-settling return', async () => {
|
||||
const original = new LlmError('provider failed', 'PROVIDER_FAILED')
|
||||
let cleanupCalls = 0
|
||||
const adapter = new class extends LlmAdapter {
|
||||
stream(_options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
return {
|
||||
[Symbol.asyncIterator](): AsyncIterator<StreamChunk> {
|
||||
return {
|
||||
next: () => Promise.reject(original),
|
||||
return: () => {
|
||||
cleanupCalls += 1
|
||||
return new Promise<IteratorResult<StreamChunk>>(() => {})
|
||||
},
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
}()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-model'], adapter)
|
||||
|
||||
const stream = ctx.llm.stream({ provider: 'test-model', model: 'test-model', messages: [] })
|
||||
const failure = (async (): Promise<unknown> => {
|
||||
try {
|
||||
for await (const _chunk of stream) { /* drain */ }
|
||||
} catch (error: unknown) {
|
||||
return error
|
||||
}
|
||||
return new Error('expected adapter iteration to fail')
|
||||
})()
|
||||
let timer: ReturnType<typeof setTimeout> | undefined
|
||||
const timeout = new Promise<Error>((resolve) => {
|
||||
timer = setTimeout(() => { resolve(new Error('adapter failure did not settle promptly')) }, 100)
|
||||
})
|
||||
const caught = await Promise.race([failure, timeout])
|
||||
if (timer !== undefined) clearTimeout(timer)
|
||||
|
||||
expect(caught).toBe(original)
|
||||
expect(isLlmAdapterFailure(stream, caught)).toBe(true)
|
||||
expect(cleanupCalls).toBe(0)
|
||||
})
|
||||
|
||||
it('awaits one adapter return on downstream close and leaves its rejection unclassified', async () => {
|
||||
const cleanup = new Error('cleanup failed')
|
||||
let cleanupCalls = 0
|
||||
const adapter = new class extends LlmAdapter {
|
||||
stream(_options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
return {
|
||||
[Symbol.asyncIterator](): AsyncIterator<StreamChunk> {
|
||||
return {
|
||||
next: () => Promise.resolve({ done: false, value: SCRIPT[0]! }),
|
||||
return: () => {
|
||||
cleanupCalls += 1
|
||||
return Promise.reject(cleanup)
|
||||
},
|
||||
}
|
||||
},
|
||||
}
|
||||
}
|
||||
}()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-model'], adapter)
|
||||
|
||||
const stream = ctx.llm.stream({ provider: 'test-model', model: 'test-model', messages: [] })
|
||||
let caught: unknown
|
||||
try {
|
||||
for await (const _chunk of stream) break
|
||||
} catch (error: unknown) {
|
||||
caught = error
|
||||
}
|
||||
|
||||
expect(caught).toBe(cleanup)
|
||||
expect(isLlmAdapterFailure(stream, caught)).toBe(false)
|
||||
expect(cleanupCalls).toBe(1)
|
||||
})
|
||||
|
||||
it('allows downstream close when the adapter iterator has no return method', async () => {
|
||||
const adapter = new class extends LlmAdapter {
|
||||
stream(_options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
return {
|
||||
[Symbol.asyncIterator](): AsyncIterator<StreamChunk> {
|
||||
return { next: () => Promise.resolve({ done: false, value: SCRIPT[0]! }) }
|
||||
},
|
||||
}
|
||||
}
|
||||
}()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-model'], adapter)
|
||||
|
||||
let chunks = 0
|
||||
for await (const _chunk of ctx.llm.stream({ provider: 'test-model', model: 'test-model', messages: [] })) {
|
||||
chunks += 1
|
||||
break
|
||||
}
|
||||
|
||||
expect(chunks).toBe(1)
|
||||
})
|
||||
|
||||
it('normalizes and tags non-Error adapter failures once', async () => {
|
||||
const adapter = new class extends LlmAdapter {
|
||||
stream(_options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
return {
|
||||
[Symbol.asyncIterator](): AsyncIterator<StreamChunk> {
|
||||
// Third-party adapters can reject with arbitrary values.
|
||||
// eslint-disable-next-line @typescript-eslint/prefer-promise-reject-errors
|
||||
return { next: () => Promise.reject('plain provider failure') }
|
||||
},
|
||||
}
|
||||
}
|
||||
}()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-model'], adapter)
|
||||
|
||||
const stream = ctx.llm.stream({ provider: 'test-model', model: 'test-model', messages: [] })
|
||||
let caught: unknown
|
||||
try {
|
||||
for await (const _chunk of stream) { /* drain */ }
|
||||
} catch (error: unknown) {
|
||||
caught = error
|
||||
}
|
||||
|
||||
expect(caught).toBeInstanceOf(HarnessError)
|
||||
expect(caught).toMatchObject({ code: 'UNKNOWN', cause: 'plain provider failure' })
|
||||
expect(isLlmAdapterFailure(stream, caught)).toBe(true)
|
||||
})
|
||||
|
||||
it('does not tag a failure thrown downstream while consuming adapter output', async () => {
|
||||
const downstream = new Error('consumer failed')
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['test-model'], new ScriptedAdapter(SCRIPT))
|
||||
|
||||
const chunks: StreamChunk[] = []
|
||||
for await (const chunk of ctx.llm.stream({ model: 'test-model', messages: [] })) chunks.push(chunk)
|
||||
expect(chunks).toEqual(SCRIPT)
|
||||
})
|
||||
const stream = ctx.llm.stream({ provider: 'test-model', model: 'test-model', messages: [] })
|
||||
let caught: unknown
|
||||
try {
|
||||
for await (const _chunk of stream) throw downstream
|
||||
} catch (error: unknown) {
|
||||
caught = error
|
||||
}
|
||||
|
||||
it('throws NO_ADAPTER for unregistered models', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await expect((async () => {
|
||||
for await (const _ of ctx.llm.stream({ model: 'nope', messages: [] })) { /* drain */ }
|
||||
})()).rejects.toThrow('no adapter registered')
|
||||
expect(caught).toBe(downstream)
|
||||
expect(isLlmAdapterFailure(stream, caught)).toBe(false)
|
||||
expect(isLlmAdapterFailure(new ScriptedAdapter(SCRIPT).stream({
|
||||
provider: 'unbound', model: 'unbound', messages: [],
|
||||
}), caught)).toBe(false)
|
||||
expect(isLlmAdapterFailure(stream, 'consumer failed')).toBe(false)
|
||||
})
|
||||
|
||||
it('unregisters adapters when the owning fiber is disposed (HMR safety)', async () => {
|
||||
@@ -44,10 +609,80 @@ describe('LlmService', () => {
|
||||
const fiber = await ctx.plugin(Object.assign((inner: Context) => {
|
||||
inner.llm.registerAdapter(['scoped-model'], new ScriptedAdapter(SCRIPT))
|
||||
}, { inject: ['llm'] }))
|
||||
expect(ctx.llm.models()).toEqual(['scoped-model'])
|
||||
expect(ctx.llm.listProviders()).toEqual([{ id: 'scoped-model', name: 'scoped-model' }])
|
||||
|
||||
await fiber.dispose()
|
||||
expect(ctx.llm.models()).toEqual([])
|
||||
expect(ctx.llm.listProviders()).toEqual([])
|
||||
})
|
||||
|
||||
it('discovers detached provider and advisory model metadata', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
const provider = { id: 'catalog', name: 'Catalog Provider' }
|
||||
const model = { provider: 'catalog', id: 'fast', name: 'Fast', description: 'Low latency' }
|
||||
ctx.llm.registerAdapter(['catalog'], new CatalogAdapter(provider, [model]))
|
||||
|
||||
const providers = ctx.llm.listProviders()
|
||||
const models = await ctx.llm.listModels('catalog')
|
||||
expect(providers).toEqual([provider])
|
||||
expect(models).toEqual([model])
|
||||
|
||||
providers[0]!.name = 'mutated'
|
||||
models[0]!.name = 'mutated'
|
||||
provider.name = 'source mutated'
|
||||
model.name = 'source mutated'
|
||||
expect(ctx.llm.listProviders()).toEqual([{ id: 'catalog', name: 'Catalog Provider' }])
|
||||
await expect(ctx.llm.listModels('catalog')).resolves.toEqual([{
|
||||
provider: 'catalog', id: 'fast', name: 'source mutated', description: 'Low latency',
|
||||
}])
|
||||
})
|
||||
|
||||
it('defaults adapters to their route name and an empty advisory model list', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['plain'], new ScriptedAdapter(SCRIPT))
|
||||
expect(ctx.llm.listProviders()).toEqual([{ id: 'plain', name: 'plain' }])
|
||||
await expect(ctx.llm.listModels('plain')).resolves.toEqual([])
|
||||
await expect(ctx.llm.listModels('missing')).rejects.toMatchObject({ code: 'NO_ADAPTER' })
|
||||
})
|
||||
|
||||
it.each([
|
||||
[{ id: 1, name: 'Name' }, 'non-string id'],
|
||||
[{ id: 'other', name: 'Name' }, 'mismatched id'],
|
||||
[{ id: 'route', name: 1 }, 'non-string name'],
|
||||
[{ id: 'route', name: '' }, 'empty name'],
|
||||
] as const)('rejects invalid provider metadata atomically (%s: %s)', async (metadata, _label) => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
const adapter = new CatalogAdapter(metadata as unknown as LlmProviderInfo, [])
|
||||
expect(() => ctx.llm.registerAdapter(['route'], adapter)).toThrow(expect.objectContaining({ code: 'INVALID_ADAPTER' }))
|
||||
expect(ctx.llm.listProviders()).toEqual([])
|
||||
})
|
||||
|
||||
it.each([
|
||||
[{ provider: 1, id: 'm', name: 'M' }, 'non-string provider'],
|
||||
[{ provider: 'other', id: 'm', name: 'M' }, 'mismatched provider'],
|
||||
[{ provider: 'route', id: 1, name: 'M' }, 'non-string id'],
|
||||
[{ provider: 'route', id: '', name: 'M' }, 'empty id'],
|
||||
[{ provider: 'route', id: 'm', name: 1 }, 'non-string name'],
|
||||
[{ provider: 'route', id: 'm', name: '' }, 'empty name'],
|
||||
[{ provider: 'route', id: 'm', name: 'M', description: 1 }, 'non-string description'],
|
||||
] as const)('rejects invalid model metadata (%s: %s)', async (metadata, _label) => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['route'], new CatalogAdapter(
|
||||
{ id: 'route', name: 'Route' },
|
||||
[metadata as unknown as LlmModelInfo],
|
||||
))
|
||||
await expect(ctx.llm.listModels('route')).rejects.toMatchObject({ code: 'INVALID_CATALOG' })
|
||||
})
|
||||
|
||||
it('rejects duplicate model ids in one provider catalog', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
const model = { provider: 'route', id: 'same', name: 'Same' }
|
||||
ctx.llm.registerAdapter(['route'], new CatalogAdapter({ id: 'route', name: 'Route' }, [model, model]))
|
||||
await expect(ctx.llm.listModels('route')).rejects.toMatchObject({ code: 'INVALID_CATALOG' })
|
||||
})
|
||||
|
||||
it('lets llm/stream waterfall listeners wrap the underlying stream', async () => {
|
||||
@@ -64,11 +699,90 @@ describe('LlmService', () => {
|
||||
})
|
||||
|
||||
const chunks: StreamChunk[] = []
|
||||
for await (const chunk of ctx.llm.stream({ model: 'test-model', messages: [] })) chunks.push(chunk)
|
||||
for await (const chunk of ctx.llm.stream({ provider: 'test-model', model: 'dynamic-model', messages: [] })) chunks.push(chunk)
|
||||
expect(chunks).toHaveLength(4)
|
||||
expect(chunks[0]).toMatchObject({ index: 99 })
|
||||
})
|
||||
|
||||
it('resolves the provider after llm/stream listeners have had a chance to route it', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
const adapter = new RecordingAdapter(SCRIPT)
|
||||
ctx.llm.registerAdapter(['routed'], adapter)
|
||||
ctx.on('llm/stream', (options, next) => {
|
||||
options.provider = 'routed'
|
||||
return next()
|
||||
})
|
||||
|
||||
for await (const _chunk of ctx.llm.stream({ provider: 'initial', model: 'm', messages: [] })) { /* drain */ }
|
||||
expect(adapter.lastOptions?.provider).toBe('routed')
|
||||
})
|
||||
|
||||
it('keeps replay state when historical and target providers belong to the same adapter instance', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
const adapter = new RecordingAdapter(SCRIPT)
|
||||
ctx.llm.registerAdapter(['historical', 'target'], adapter)
|
||||
const replayState = { private: 'state' }
|
||||
|
||||
for await (const _chunk of ctx.llm.stream({
|
||||
provider: 'target',
|
||||
model: 'new-model',
|
||||
messages: [{
|
||||
role: 'assistant',
|
||||
content: [{ type: 'text', text: 'old response' }],
|
||||
provenance: { provider: 'historical', model: 'old-model', replayState },
|
||||
}],
|
||||
})) { /* drain */ }
|
||||
|
||||
expect(adapter.lastOptions?.messages[0]?.provenance).toEqual({
|
||||
provider: 'historical', model: 'old-model', replayState,
|
||||
})
|
||||
})
|
||||
|
||||
it('strips replay state but preserves provenance when the target uses a different adapter instance', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['historical'], new RecordingAdapter(SCRIPT))
|
||||
const target = new RecordingAdapter(SCRIPT)
|
||||
ctx.llm.registerAdapter(['target'], target)
|
||||
|
||||
for await (const _chunk of ctx.llm.stream({
|
||||
provider: 'target',
|
||||
model: 'new-model',
|
||||
messages: [{
|
||||
role: 'assistant',
|
||||
content: [{ type: 'text', text: 'old response' }],
|
||||
provenance: { provider: 'historical', model: 'old-model', replayState: { private: 'state' } },
|
||||
}],
|
||||
})) { /* drain */ }
|
||||
|
||||
expect(target.lastOptions?.messages[0]?.provenance).toEqual({ provider: 'historical', model: 'old-model' })
|
||||
})
|
||||
|
||||
it('preserves immutability while stripping replay state from frozen requests', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['historical'], new RecordingAdapter(SCRIPT))
|
||||
const target = new RecordingAdapter(SCRIPT)
|
||||
ctx.llm.registerAdapter(['target'], target)
|
||||
const options = Object.freeze({
|
||||
provider: 'target',
|
||||
model: 'new-model',
|
||||
messages: [{
|
||||
role: 'assistant' as const,
|
||||
content: [{ type: 'text' as const, text: 'old response' }],
|
||||
provenance: { provider: 'historical', model: 'old-model', replayState: { private: 'state' } },
|
||||
}],
|
||||
})
|
||||
|
||||
for await (const _chunk of ctx.llm.stream(options)) { /* drain */ }
|
||||
|
||||
expect(target.lastOptions).not.toBe(options)
|
||||
expect(Object.isFrozen(target.lastOptions)).toBe(true)
|
||||
expect(target.lastOptions?.messages[0]?.provenance).toEqual({ provider: 'historical', model: 'old-model' })
|
||||
})
|
||||
|
||||
it('creates LlmError with a code for programmatic handling', () => {
|
||||
const err = new LlmError('something went wrong', 'CUSTOM_CODE')
|
||||
expect(err).toBeInstanceOf(Error)
|
||||
@@ -77,13 +791,24 @@ describe('LlmService', () => {
|
||||
expect(err.code).toBe('CUSTOM_CODE')
|
||||
})
|
||||
|
||||
it('rejects non-serializable structured failure facts at construction', () => {
|
||||
expect(() => new LlmError('busy', 'RATE_LIMIT', { status: 42 })).toThrow(/status/)
|
||||
expect(() => new LlmError('busy', 'RATE_LIMIT', { providerRetryAfterMs: Number.NaN }))
|
||||
.toThrow(/providerRetryAfterMs/)
|
||||
expect(() => new LlmError('busy', 'RATE_LIMIT', { requestId: ProviderRequestId('') })).toThrow(/requestId/)
|
||||
expect(() => new LlmError(1 as never, 'RATE_LIMIT')).toThrow(/message/)
|
||||
expect(() => new LlmError('busy', 1 as never)).toThrow(/code/)
|
||||
expect(() => new LlmError('busy', 'RATE_LIMIT', { requestId: 1 as never })).toThrow(/requestId/)
|
||||
})
|
||||
|
||||
it('LlmError extends the shared HarnessError base', async () => {
|
||||
const { HarnessError, isHarnessError } = await import('@deepseek-ai/dsh-llm')
|
||||
const err = new LlmError('boom', 'AUTH', 401)
|
||||
const cause = new Error('root cause')
|
||||
const err = new LlmError('boom', 'AUTH', { cause })
|
||||
expect(err).toBeInstanceOf(HarnessError)
|
||||
expect(isHarnessError(err)).toBe(true)
|
||||
expect(err.code).toBe('AUTH')
|
||||
expect(err.status).toBe(401)
|
||||
expect(err.cause).toBe(cause)
|
||||
})
|
||||
|
||||
it('HarnessError carries a code, names itself by subclass, and chains cause', async () => {
|
||||
@@ -104,9 +829,9 @@ describe('LlmService', () => {
|
||||
await ctx.plugin(LlmService)
|
||||
|
||||
const dispose = ctx.llm.registerAdapter(['m1'], new ScriptedAdapter(SCRIPT))
|
||||
expect(ctx.llm.models()).toEqual(['m1'])
|
||||
expect(ctx.llm.listProviders()).toEqual([{ id: 'm1', name: 'm1' }])
|
||||
dispose()
|
||||
expect(ctx.llm.models()).toEqual([])
|
||||
expect(ctx.llm.listProviders()).toEqual([])
|
||||
})
|
||||
|
||||
it('rejects duplicate adapter registration with DUPLICATE_ADAPTER code', async () => {
|
||||
@@ -123,19 +848,30 @@ describe('LlmService', () => {
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects empty and internally duplicated provider registrations atomically', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
const adapter = new ScriptedAdapter(SCRIPT)
|
||||
|
||||
expect(() => ctx.llm.registerAdapter([], adapter)).toThrow(expect.objectContaining({ code: 'INVALID_ADAPTER' }))
|
||||
expect(() => ctx.llm.registerAdapter([''], adapter)).toThrow(expect.objectContaining({ code: 'INVALID_ADAPTER' }))
|
||||
expect(() => ctx.llm.registerAdapter(['first', 'first'], adapter)).toThrow(expect.objectContaining({ code: 'DUPLICATE_ADAPTER' }))
|
||||
expect(ctx.llm.listProviders()).toEqual([])
|
||||
})
|
||||
|
||||
it('re-registers a model after its prior registration is disposed', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
|
||||
const dispose = ctx.llm.registerAdapter(['m1'], new ScriptedAdapter(SCRIPT))
|
||||
expect(ctx.llm.models()).toEqual(['m1'])
|
||||
expect(ctx.llm.listProviders()).toEqual([{ id: 'm1', name: 'm1' }])
|
||||
dispose()
|
||||
expect(ctx.llm.models()).toEqual([])
|
||||
expect(ctx.llm.listProviders()).toEqual([])
|
||||
|
||||
// The duplicate check is not wedged: the same model registers cleanly again.
|
||||
const disposeAgain = ctx.llm.registerAdapter(['m1'], new ScriptedAdapter(SCRIPT))
|
||||
expect(ctx.llm.models()).toEqual(['m1'])
|
||||
expect(ctx.llm.listProviders()).toEqual([{ id: 'm1', name: 'm1' }])
|
||||
disposeAgain()
|
||||
expect(ctx.llm.models()).toEqual([])
|
||||
expect(ctx.llm.listProviders()).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
54
packages/llm/token-meter/README.md
Normal file
54
packages/llm/token-meter/README.md
Normal file
@@ -0,0 +1,54 @@
|
||||
# @deepseek-ai/dsh-token-meter
|
||||
|
||||
Replay-aware token measurement through the singleton `ctx.tokenMeter` service. It advances one isolated fold per session from the durable log, so compaction and other pressure-sensitive plugins can share accounting without depending on `CompactService`.
|
||||
|
||||
## Configuration
|
||||
|
||||
| Key | Default | Contract |
|
||||
|---|---:|---|
|
||||
| `contextWindow` | `128000` | Positive integer service-wide context capacity. |
|
||||
|
||||
The estimator intentionally uses one fixed heuristic: four characters per token plus structural overhead for roles, blocks, and request-envelope fields. `contextWindow` is the only deployment setting. Direct construction validates it; Loader mounts first apply the package's Schemastery shape validation. Unrecognized top-level keys are rejected.
|
||||
|
||||
## Measurement contract
|
||||
|
||||
`ctx.tokenMeter` directly exposes two operations:
|
||||
|
||||
- `measure(session, requestHeader?)` returns request pressure and the current priced surface at one consumed-log revision.
|
||||
- `estimateMessage(message)` prices one message with the fixed heuristic.
|
||||
|
||||
`measure()` synchronizes once and returns one detached, deeply immutable snapshot. `totalTokens` is request-and-response pressure, while `surfaceTokens` is the surface-only heuristic total and equals the sum of `nodes[].tokens`. A `requestHeader` override affects pressure fields only; the surface fields still describe the current session. Every call clones the positional nodes, so measurement is O(surface).
|
||||
|
||||
The fold tracks full request-header snapshots, step boundaries, surface appends and replacements, successful assistant messages, provider usage, and assistant-chunk provenance. Provider usage is reused only when the latest successful call's canonical request envelope matches the measured envelope and its total is no lower than that call's full heuristic anchor; a later success replaces the earlier anchor. Otherwise the complete current envelope and surface are estimated. Surface changes remain signed relative to a matching anchor, including negative deltas after shrinking replacements.
|
||||
|
||||
Usage accounting sums disjoint input, cache-read, cache-write, and output buckets; reasoning is not added again. Every successful call records an assistant anchor, including content-less calls. An explicit empty provenance list means a known empty provider stream, while absent legacy provenance conservatively treats the durable assistant output as provider output.
|
||||
|
||||
## Composition
|
||||
|
||||
```yaml
|
||||
- name: '@deepseek-ai/dsh-token-meter'
|
||||
- name: '@deepseek-ai/dsh-compact-basic'
|
||||
```
|
||||
|
||||
Both plugins have usable defaults. A deployment with a different capacity configures the meter once:
|
||||
|
||||
```yaml
|
||||
- name: '@deepseek-ai/dsh-token-meter'
|
||||
config:
|
||||
contextWindow: 32768
|
||||
```
|
||||
|
||||
## Model Experience
|
||||
|
||||
Indirectly, through consumers such as `dsh-compact-basic`; the service itself adds no prompt, message, schema, tool, or model call.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
No direct invalidation; the named consumer owns any request-prefix changes.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **The fixed heuristic is approximate** — content without reusable provider usage is priced by character count plus structural overhead, not an exact provider tokenizer or request serializer.
|
||||
- **Every measurement clones the current surface** — coherent immutable snapshots make reads O(surface), including below-threshold pressure checks.
|
||||
- **Provider usage is only reusable for an identical canonical envelope** — prompt, prefix, tools, provider, model, or call-config changes deliberately fall back to full heuristic estimation.
|
||||
- **Legacy provenance is conservative** — assistant messages without `sourceEventSeqs` cannot distinguish provider output from listener rewrites, so the fold avoids claiming a known empty or exact chunk stream.
|
||||
37
packages/llm/token-meter/package.json
Normal file
37
packages/llm/token-meter/package.json
Normal file
@@ -0,0 +1,37 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-token-meter",
|
||||
"description": "Replay-aware token measurement service (ctx.tokenMeter) for the DeepSeek Harness",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/types/**/*.d.ts",
|
||||
"lib/types/**/*.d.ts.map",
|
||||
"src"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-llm": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"dependencies": {
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
419
packages/llm/token-meter/src/index.ts
Normal file
419
packages/llm/token-meter/src/index.ts
Normal file
@@ -0,0 +1,419 @@
|
||||
/**
|
||||
* Single replay-aware token-meter service for request and surface pressure.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-token-meter
|
||||
*/
|
||||
|
||||
import { Context, Service } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import { BlockAssembler, deepFreeze } from '@deepseek-ai/dsh-llm'
|
||||
import type { ContentBlock, Message, TokenUsage } from '@deepseek-ai/dsh-llm'
|
||||
import type { EpochHeader, Session, SessionEvent, SurfaceEvent } from '@deepseek-ai/dsh-session'
|
||||
import { canonicalHeader, headerEquals, isSurfaceEvent } from '@deepseek-ai/dsh-session'
|
||||
import type {
|
||||
TokenMeasurement,
|
||||
TokenMeasurementBaseline,
|
||||
TokenMeterConfig,
|
||||
TokenSurfaceNode,
|
||||
} from './types.ts'
|
||||
|
||||
export type * from './types.ts'
|
||||
|
||||
/** Default service-wide provider context capacity. */
|
||||
const DEFAULT_CONTEXT_WINDOW = 128_000
|
||||
|
||||
/** Complete public configuration key set. */
|
||||
const TOKEN_METER_CONFIG_KEYS: ReadonlySet<string> = new Set(['contextWindow'])
|
||||
|
||||
/** Fixed text-density estimate used until exact tokenization is needed. */
|
||||
const CHARS_PER_TOKEN = 4
|
||||
|
||||
/** Per-block structural overhead for JSON framing and type tags. */
|
||||
const BLOCK_OVERHEAD = 4
|
||||
|
||||
/** Role-field framing overhead added to every priced message. */
|
||||
const ROLE_OVERHEAD = 4
|
||||
|
||||
interface MeasurementAnchor {
|
||||
readonly header: EpochHeader | undefined
|
||||
readonly surfaceTokens: number
|
||||
readonly baseline: Exclude<TokenMeasurementBaseline, { kind: 'none' }>
|
||||
}
|
||||
|
||||
interface ReplayState {
|
||||
consumedEvents: number
|
||||
header: EpochHeader | undefined
|
||||
surface: TokenSurfaceNode[]
|
||||
surfaceTokens: number
|
||||
stepStart: { turn: number; step: number; surfaceTokens: number } | undefined
|
||||
anchor: MeasurementAnchor | undefined
|
||||
}
|
||||
|
||||
interface PreparedSurfaceMutation {
|
||||
readonly tokens: number
|
||||
commit(state: ReplayState): void
|
||||
}
|
||||
|
||||
/** Sum disjoint provider usage buckets without double-counting reasoning output. */
|
||||
function usageTokens(usage: TokenUsage): number {
|
||||
return usage.inputTokens
|
||||
+ (usage.cacheReadTokens ?? 0)
|
||||
+ (usage.cacheWriteTokens ?? 0)
|
||||
+ usage.outputTokens
|
||||
}
|
||||
|
||||
/** Compare optional envelopes so a headerless estimate can track later surface deltas. */
|
||||
function optionalHeaderEquals(
|
||||
left: EpochHeader | undefined,
|
||||
right: EpochHeader | undefined,
|
||||
): boolean {
|
||||
if (left === undefined || right === undefined) return left === right
|
||||
return headerEquals(left, right)
|
||||
}
|
||||
|
||||
/** Reject stale or misspelled keys before defaults can hide them. */
|
||||
function validateConfigKeys(config: TokenMeterConfig): void {
|
||||
for (const key of Object.keys(config)) {
|
||||
if (!TOKEN_METER_CONFIG_KEYS.has(key)) {
|
||||
throw new Error(
|
||||
`TokenMeterConfig: unknown key "${key}" (allowed: contextWindow)`,
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Resolve and validate the one service-wide context capacity. */
|
||||
function resolveContextWindow(config: TokenMeterConfig): number {
|
||||
validateConfigKeys(config)
|
||||
const contextWindow = config.contextWindow === undefined
|
||||
? DEFAULT_CONTEXT_WINDOW
|
||||
: config.contextWindow
|
||||
if (!Number.isInteger(contextWindow) || contextWindow <= 0) {
|
||||
throw new Error(
|
||||
`TokenMeterConfig: contextWindow (${contextWindow}) must be a positive integer`,
|
||||
)
|
||||
}
|
||||
return contextWindow
|
||||
}
|
||||
|
||||
declare module 'cordis' {
|
||||
interface Context {
|
||||
tokenMeter: TokenMeterService
|
||||
}
|
||||
}
|
||||
|
||||
/** Replay owner for one service-wide estimator and isolated per-session folds. */
|
||||
export class TokenMeterService extends Service {
|
||||
static Config: z<TokenMeterConfig> = z.object({
|
||||
contextWindow: z.number().step(1).min(1).default(DEFAULT_CONTEXT_WINDOW),
|
||||
})
|
||||
|
||||
/** Provider context-window capacity used by pressure consumers. */
|
||||
readonly contextWindow: number
|
||||
|
||||
private readonly states = new WeakMap<Session, ReplayState>()
|
||||
|
||||
constructor(ctx: Context, config: TokenMeterConfig = {}) {
|
||||
super(ctx, 'tokenMeter')
|
||||
this.contextWindow = resolveContextWindow(config)
|
||||
|
||||
// Readers catch up independently, while eager observation bounds ordinary
|
||||
// read latency without creating state for sessions no consumer has read.
|
||||
ctx.on('session/event', (session) => {
|
||||
if (this.states.has(session)) this._sync(session)
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Measure current request pressure and surface through the durable tail.
|
||||
*
|
||||
* Provider usage is reused only when the latest successful call's canonical
|
||||
* request envelope matches `requestHeader` and its total is no lower than
|
||||
* that call's full heuristic anchor; otherwise the complete envelope and
|
||||
* surface are heuristically repriced.
|
||||
*
|
||||
* `requestHeader` affects request pressure only; surface fields always
|
||||
* describe the current session surface. Every call clones those positional
|
||||
* nodes, so measurement is O(surface).
|
||||
*
|
||||
* @param session - session to replay through its current durable tail.
|
||||
* @param requestHeader - optional effective request envelope replacing the latest logged header.
|
||||
* @returns a detached deeply immutable pressure and surface measurement.
|
||||
*/
|
||||
measure(session: Session, requestHeader?: EpochHeader): TokenMeasurement {
|
||||
const state = this._sync(session)
|
||||
const header = requestHeader === undefined
|
||||
? state.header
|
||||
: canonicalHeader(requestHeader)
|
||||
const anchor = state.anchor
|
||||
|
||||
let baseline: TokenMeasurementBaseline
|
||||
let surfaceDeltaTokens: number
|
||||
if (anchor !== undefined && optionalHeaderEquals(anchor.header, header)) {
|
||||
baseline = anchor.baseline
|
||||
surfaceDeltaTokens = state.surfaceTokens - anchor.surfaceTokens
|
||||
} else if (header === undefined && state.surfaceTokens === 0) {
|
||||
baseline = { kind: 'none', tokens: 0 }
|
||||
surfaceDeltaTokens = 0
|
||||
} else {
|
||||
baseline = {
|
||||
kind: 'estimated',
|
||||
tokens: this._estimateHeader(header) + state.surfaceTokens,
|
||||
}
|
||||
surfaceDeltaTokens = 0
|
||||
}
|
||||
|
||||
return deepFreeze(structuredClone({
|
||||
logRevision: state.consumedEvents,
|
||||
baseline,
|
||||
surfaceDeltaTokens,
|
||||
totalTokens: Math.max(0, baseline.tokens + surfaceDeltaTokens),
|
||||
surfaceTokens: state.surfaceTokens,
|
||||
nodes: state.surface,
|
||||
}))
|
||||
}
|
||||
|
||||
/**
|
||||
* Heuristically price one model-visible message.
|
||||
* @param message - message to price without mutation.
|
||||
* @returns content and role-framing tokens under the fixed service heuristic.
|
||||
*/
|
||||
estimateMessage(message: Message): number {
|
||||
return this._estimateContent(message.content) + ROLE_OVERHEAD
|
||||
}
|
||||
|
||||
/** Catch one session's fold up to the current durable tail. */
|
||||
private _sync(session: Session): ReplayState {
|
||||
let state = this.states.get(session)
|
||||
if (state === undefined) {
|
||||
state = {
|
||||
consumedEvents: 0,
|
||||
header: undefined,
|
||||
surface: [],
|
||||
surfaceTokens: 0,
|
||||
stepStart: undefined,
|
||||
anchor: undefined,
|
||||
}
|
||||
this.states.set(session, state)
|
||||
}
|
||||
|
||||
while (state.consumedEvents < session.events.length) {
|
||||
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion -- contiguous session seqs index the durable log
|
||||
const event = session.events[state.consumedEvents]!
|
||||
this._foldEvent(session, state, event)
|
||||
state.consumedEvents += 1
|
||||
}
|
||||
return state
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate and prepare every fallible part before mutating replay state.
|
||||
* A malformed event remains unread on every retry instead of partially
|
||||
* applying the same mutation more than once.
|
||||
*/
|
||||
private _foldEvent(session: Session, state: ReplayState, event: SessionEvent): void {
|
||||
let nextHeader = state.header
|
||||
let nextStepStart = state.stepStart
|
||||
let nextAnchor = state.anchor
|
||||
|
||||
switch (event.type) {
|
||||
case 'request/header':
|
||||
nextHeader = canonicalHeader(event.data.header)
|
||||
break
|
||||
case 'step/start':
|
||||
if (state.stepStart !== undefined) {
|
||||
throw new Error(
|
||||
`token meter: step/start at seq ${event.seq} arrived before turn ${state.stepStart.turn}/step ${state.stepStart.step} ended`,
|
||||
)
|
||||
}
|
||||
nextStepStart = { ...event.data, surfaceTokens: state.surfaceTokens }
|
||||
break
|
||||
case 'step/end':
|
||||
if (state.stepStart === undefined
|
||||
|| state.stepStart.turn !== event.data.turn
|
||||
|| state.stepStart.step !== event.data.step) {
|
||||
throw new Error(`token meter: step/end at seq ${event.seq} has no matching step/start boundary`)
|
||||
}
|
||||
nextStepStart = undefined
|
||||
break
|
||||
default:
|
||||
break
|
||||
}
|
||||
|
||||
const surface = isSurfaceEvent(event)
|
||||
? this._prepareSurfaceMutation(session, state, event)
|
||||
: undefined
|
||||
|
||||
if (event.type === 'assistant/message') {
|
||||
const stepStart = state.stepStart
|
||||
if (stepStart === undefined
|
||||
|| stepStart.turn !== event.data.turn
|
||||
|| stepStart.step !== event.data.step) {
|
||||
throw new Error(`token meter: assistant/message at seq ${event.seq} has no matching step/start boundary`)
|
||||
}
|
||||
|
||||
// assistant/message is surface-mandatory at every append/seed boundary.
|
||||
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
|
||||
const eventTokens = surface!.tokens
|
||||
if (event.data.usage !== undefined && nextHeader !== undefined) {
|
||||
const providerAssistantTokens = this._estimateProviderAssistant(
|
||||
session,
|
||||
event,
|
||||
eventTokens,
|
||||
)
|
||||
const anchorSurfaceTokens = stepStart.surfaceTokens + providerAssistantTokens
|
||||
const providerTokens = usageTokens(event.data.usage)
|
||||
const estimatedAnchorTokens = this._estimateHeader(nextHeader) + anchorSurfaceTokens
|
||||
nextAnchor = {
|
||||
header: nextHeader,
|
||||
surfaceTokens: anchorSurfaceTokens,
|
||||
// Signed heuristic deltas remain conservative only from an anchor
|
||||
// that is at least as large as the matching full heuristic price.
|
||||
baseline: providerTokens >= estimatedAnchorTokens
|
||||
? { kind: 'usage', tokens: providerTokens, usage: event.data.usage }
|
||||
: { kind: 'estimated', tokens: estimatedAnchorTokens },
|
||||
}
|
||||
} else {
|
||||
const anchorSurfaceTokens = stepStart.surfaceTokens + eventTokens
|
||||
nextAnchor = {
|
||||
header: nextHeader,
|
||||
surfaceTokens: anchorSurfaceTokens,
|
||||
baseline: {
|
||||
kind: 'estimated',
|
||||
tokens: this._estimateHeader(nextHeader) + anchorSurfaceTokens,
|
||||
},
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
state.header = nextHeader
|
||||
state.stepStart = nextStepStart
|
||||
if (surface !== undefined) surface.commit(state)
|
||||
state.anchor = nextAnchor
|
||||
}
|
||||
|
||||
/** Validate one surface operation and return its allocation-light commit. */
|
||||
private _prepareSurfaceMutation(
|
||||
session: Session,
|
||||
state: ReplayState,
|
||||
event: SurfaceEvent,
|
||||
): PreparedSurfaceMutation {
|
||||
const tokens = this._estimateSurfaceEvent(session, event)
|
||||
const op = event.surfaceOp
|
||||
if (op === 'append') {
|
||||
return {
|
||||
tokens,
|
||||
commit(target) {
|
||||
target.surface.push({ seq: event.seq, tokens })
|
||||
target.surfaceTokens += tokens
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
const startIdx = state.surface.findIndex(node => node.seq === op.start)
|
||||
const endIdx = state.surface.findIndex(node => node.seq === op.end)
|
||||
if (startIdx === -1 || endIdx === -1 || startIdx > endIdx) {
|
||||
throw new Error(
|
||||
`token meter: replace at seq ${event.seq} has invalid current range ${op.start}-${op.end}`,
|
||||
)
|
||||
}
|
||||
const removedTokens = state.surface
|
||||
.slice(startIdx, endIdx + 1)
|
||||
.reduce((total, node) => total + node.tokens, 0)
|
||||
return {
|
||||
tokens,
|
||||
commit(target) {
|
||||
target.surface.splice(startIdx, endIdx - startIdx + 1, { seq: event.seq, tokens })
|
||||
target.surfaceTokens += tokens - removedTokens
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/** Price one current surface event exactly as it projects to a request. */
|
||||
private _estimateSurfaceEvent(session: Session, event: SurfaceEvent): number {
|
||||
const message = session.deriveEventMessage(event)
|
||||
return message === null ? 0 : this.estimateMessage(message)
|
||||
}
|
||||
|
||||
/**
|
||||
* Reassemble provider output from exact chunk provenance for a usage anchor.
|
||||
* Missing legacy provenance conservatively treats the durable output as the
|
||||
* provider output; explicit empty provenance prices a known empty stream.
|
||||
*/
|
||||
private _estimateProviderAssistant(
|
||||
session: Session,
|
||||
event: SessionEvent<'assistant/message'>,
|
||||
durableEventTokens: number,
|
||||
): number {
|
||||
const sourceSeqs = event.sourceEventSeqs
|
||||
if (sourceSeqs === undefined) return durableEventTokens
|
||||
|
||||
const assembler = new BlockAssembler()
|
||||
const seen = new Set<number>()
|
||||
for (const seq of sourceSeqs) {
|
||||
if (seq >= event.seq) {
|
||||
throw new Error(`token meter: assistant/message at seq ${event.seq} source seq ${seq} is not earlier`)
|
||||
}
|
||||
if (seen.has(seq)) {
|
||||
throw new Error(`token meter: assistant/message at seq ${event.seq} repeats source seq ${seq}`)
|
||||
}
|
||||
seen.add(seq)
|
||||
// Session construction validates contiguous seqs, and the explicit
|
||||
// earlier-than-assistant check above therefore guarantees existence.
|
||||
const source = session.events[seq]
|
||||
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
|
||||
const sourceEvent = source!
|
||||
if (sourceEvent.type !== 'assistant/chunk') {
|
||||
throw new Error(`token meter: assistant/message at seq ${event.seq} source seq ${seq} is not assistant/chunk`)
|
||||
}
|
||||
if (sourceEvent.data.turn !== event.data.turn || sourceEvent.data.step !== event.data.step) {
|
||||
throw new Error(`token meter: assistant/message at seq ${event.seq} source seq ${seq} belongs to another step`)
|
||||
}
|
||||
assembler.push(sourceEvent.data.chunk)
|
||||
}
|
||||
const providerMessage = assembler.message()
|
||||
return providerMessage.content.length === 0 ? 0 : this.estimateMessage(providerMessage)
|
||||
}
|
||||
|
||||
/** Price content blocks recursively under the fixed density heuristic. */
|
||||
private _estimateContent(blocks: readonly ContentBlock[]): number {
|
||||
let tokens = 0
|
||||
for (const block of blocks) {
|
||||
switch (block.type) {
|
||||
case 'text':
|
||||
case 'reasoning':
|
||||
tokens += Math.ceil(block.text.length / CHARS_PER_TOKEN) + BLOCK_OVERHEAD
|
||||
break
|
||||
case 'tool-call':
|
||||
tokens += Math.ceil(block.name.length / CHARS_PER_TOKEN)
|
||||
+ Math.ceil(block.arguments.length / CHARS_PER_TOKEN)
|
||||
+ BLOCK_OVERHEAD
|
||||
break
|
||||
case 'tool-result':
|
||||
tokens += this._estimateContent(block.content) + BLOCK_OVERHEAD
|
||||
break
|
||||
default:
|
||||
// ContentBlockMap is merge-extensible; unknown blocks retain a
|
||||
// conservative structural JSON price under the fixed heuristic.
|
||||
tokens += BLOCK_OVERHEAD + Math.ceil(JSON.stringify(block).length / CHARS_PER_TOKEN)
|
||||
}
|
||||
}
|
||||
return tokens
|
||||
}
|
||||
|
||||
/** Price the canonical non-surface request envelope. */
|
||||
private _estimateHeader(header: EpochHeader | undefined): number {
|
||||
if (header === undefined) return 0
|
||||
let tokens = 0
|
||||
for (const message of header.messagePrefix ?? []) tokens += this.estimateMessage(message)
|
||||
if (header.system !== undefined) {
|
||||
tokens += Math.ceil(header.system.length / CHARS_PER_TOKEN) + ROLE_OVERHEAD
|
||||
}
|
||||
if (header.tools !== undefined && header.tools.length > 0) {
|
||||
tokens += Math.ceil(JSON.stringify(header.tools).length / CHARS_PER_TOKEN) + BLOCK_OVERHEAD
|
||||
}
|
||||
return tokens
|
||||
}
|
||||
}
|
||||
|
||||
export default TokenMeterService
|
||||
43
packages/llm/token-meter/src/types.ts
Normal file
43
packages/llm/token-meter/src/types.ts
Normal file
@@ -0,0 +1,43 @@
|
||||
/**
|
||||
* Public configuration and measurement vocabulary for replay token metering.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-token-meter/types
|
||||
*/
|
||||
|
||||
import type { TokenUsage } from '@deepseek-ai/dsh-llm'
|
||||
|
||||
/** Token-meter plugin configuration. */
|
||||
export interface TokenMeterConfig {
|
||||
/** Service-wide context-window capacity in tokens. Defaults to `128000`. */
|
||||
contextWindow?: number
|
||||
}
|
||||
|
||||
/** The baseline from which a signed surface delta produces current pressure. */
|
||||
export type TokenMeasurementBaseline =
|
||||
| { readonly kind: 'none'; readonly tokens: 0 }
|
||||
| { readonly kind: 'estimated'; readonly tokens: number }
|
||||
| { readonly kind: 'usage'; readonly tokens: number; readonly usage: Readonly<TokenUsage> }
|
||||
|
||||
/** Detached immutable request-pressure and surface snapshot at one consumed log revision. */
|
||||
export interface TokenMeasurement {
|
||||
/** Number of durable events consumed; equal to the next unread event seq. */
|
||||
readonly logRevision: number
|
||||
/** Provider or heuristic anchor used for this measurement. */
|
||||
readonly baseline: TokenMeasurementBaseline
|
||||
/** Signed repricing of current surface content relative to the baseline anchor. */
|
||||
readonly surfaceDeltaTokens: number
|
||||
/** Non-negative current request-and-response pressure. */
|
||||
readonly totalTokens: number
|
||||
/** Total heuristic tokens across the current surface. */
|
||||
readonly surfaceTokens: number
|
||||
/** Current surface nodes in positional head-to-tail order. */
|
||||
readonly nodes: readonly TokenSurfaceNode[]
|
||||
}
|
||||
|
||||
/** One token-priced node in the current ordered session surface. */
|
||||
export interface TokenSurfaceNode {
|
||||
/** Durable sequence number of the surface event. */
|
||||
readonly seq: number
|
||||
/** Heuristic tokens for the exact message projected by this node. */
|
||||
readonly tokens: number
|
||||
}
|
||||
649
packages/llm/token-meter/tests/token-meter.spec.ts
Normal file
649
packages/llm/token-meter/tests/token-meter.spec.ts
Normal file
@@ -0,0 +1,649 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import { CallId } from '@deepseek-ai/dsh-llm'
|
||||
import type { ContentBlock, Message, TokenUsage } from '@deepseek-ai/dsh-llm'
|
||||
import SessionStore, { Session, SessionId, canonicalHeader } from '@deepseek-ai/dsh-session'
|
||||
import type { EpochHeader, SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import TokenMeterService from '@deepseek-ai/dsh-token-meter'
|
||||
import type { TokenMeasurement, TokenMeterConfig } from '@deepseek-ai/dsh-token-meter'
|
||||
|
||||
function header(model: string, extras: Omit<EpochHeader, 'config'> = {}): EpochHeader {
|
||||
return canonicalHeader({ config: { provider: 'mock', model }, ...extras })
|
||||
}
|
||||
|
||||
function textMessage(text: string, role: Message['role'] = 'user'): Message {
|
||||
return { role, content: [{ type: 'text', text }] }
|
||||
}
|
||||
|
||||
function appendHeader(session: Session, value: EpochHeader): void {
|
||||
session.append('request/header', { header: value, reason: 'initial' })
|
||||
}
|
||||
|
||||
/** Inject malformed persisted history after the live append boundary for defensive replay tests. */
|
||||
function appendUnchecked(session: Session, event: SessionEvent): void {
|
||||
const log = (session as unknown as { log: SessionEvent[] }).log
|
||||
log.push(event)
|
||||
}
|
||||
|
||||
interface SuccessfulCallOptions {
|
||||
turn?: number
|
||||
step?: number
|
||||
providerText?: string
|
||||
durableText?: string
|
||||
usage?: TokenUsage
|
||||
provenance?: 'exact' | 'empty' | 'absent'
|
||||
}
|
||||
|
||||
function appendSuccessfulCall(
|
||||
session: Session,
|
||||
value: EpochHeader,
|
||||
options: SuccessfulCallOptions = {},
|
||||
): void {
|
||||
const turn = options.turn ?? 1
|
||||
const step = options.step ?? 1
|
||||
const providerText = options.providerText ?? 'provider answer'
|
||||
const durableText = options.durableText ?? providerText
|
||||
const provenance = options.provenance ?? 'exact'
|
||||
session.append('step/start', { turn, step })
|
||||
appendHeader(session, value)
|
||||
|
||||
const sources: number[] = []
|
||||
if (provenance === 'exact') {
|
||||
const chunks = [
|
||||
{ type: 'block-start' as const, index: 0, blockType: 'text' as const },
|
||||
{ type: 'text-delta' as const, index: 0, text: providerText },
|
||||
{ type: 'block-end' as const, index: 0, block: { type: 'text' as const, text: providerText } },
|
||||
...options.usage === undefined ? [] : [{ type: 'usage' as const, usage: options.usage }],
|
||||
{ type: 'finish' as const, reason: { kind: 'stop' as const } },
|
||||
]
|
||||
for (const chunk of chunks) {
|
||||
sources.push(session.append('assistant/chunk', { turn, step, chunk }).seq)
|
||||
}
|
||||
}
|
||||
|
||||
const intent = provenance === 'absent'
|
||||
? { surfaceOp: 'append' as const }
|
||||
: { surfaceOp: 'append' as const, sourceEventSeqs: provenance === 'empty' ? [] : sources }
|
||||
session.append('assistant/message', {
|
||||
provenance: {
|
||||
provider: value.config.provider,
|
||||
model: value.config.model,
|
||||
},
|
||||
turn,
|
||||
step,
|
||||
content: durableText.length === 0 ? [] : [{ type: 'text', text: durableText }],
|
||||
...options.usage === undefined ? {} : { usage: options.usage },
|
||||
}, intent)
|
||||
session.append('step/end', { turn, step })
|
||||
}
|
||||
|
||||
function meter(config: TokenMeterConfig = {}): TokenMeterService {
|
||||
return new TokenMeterService(new Context(), config)
|
||||
}
|
||||
|
||||
function expectSurfaceTotal(measurement: TokenMeasurement): void {
|
||||
expect(measurement.nodes.reduce((total, node) => total + node.tokens, 0))
|
||||
.toBe(measurement.surfaceTokens)
|
||||
}
|
||||
|
||||
describe('TokenMeterService configuration and registration', () => {
|
||||
it('provides one zero-config context window', () => {
|
||||
const service = meter()
|
||||
expect(service.contextWindow).toBe(128_000)
|
||||
})
|
||||
|
||||
it('accepts one service-wide context-window override', () => {
|
||||
expect(meter({ contextWindow: 32_000 }).contextWindow).toBe(32_000)
|
||||
})
|
||||
|
||||
it.each(['models', 'contextWidow'])('rejects unknown top-level config key %s', (key) => {
|
||||
expect(() => meter({ [key]: {} }))
|
||||
.toThrow(`TokenMeterConfig: unknown key "${key}"`)
|
||||
})
|
||||
|
||||
it.each([
|
||||
{ contextWindow: 0 },
|
||||
{ contextWindow: -1 },
|
||||
{ contextWindow: 1.5 },
|
||||
{ contextWindow: Number.NaN },
|
||||
{ contextWindow: null },
|
||||
] as unknown as TokenMeterConfig[])('rejects invalid context capacity %#', (config) => {
|
||||
expect(() => meter(config)).toThrow(/contextWindow .* positive integer/)
|
||||
})
|
||||
|
||||
it('registers and unregisters ctx.tokenMeter with its plugin fiber', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(TokenMeterService)
|
||||
expect(ctx.get('tokenMeter')).toBeInstanceOf(TokenMeterService)
|
||||
await fiber.dispose()
|
||||
expect(ctx.get('tokenMeter')).toBeUndefined()
|
||||
})
|
||||
})
|
||||
|
||||
describe('TokenMeterService pricing', () => {
|
||||
it('prices every built-in content shape and merge-extended blocks with one fixed heuristic', () => {
|
||||
const service = meter({ contextWindow: 100 })
|
||||
const blocks: ContentBlock[] = [
|
||||
{ type: 'text', text: 'abcd' },
|
||||
{ type: 'reasoning', text: 'ab' },
|
||||
{ type: 'tool-call', id: CallId('c'), name: 'read', arguments: '{"x":1}' },
|
||||
{
|
||||
type: 'tool-result',
|
||||
toolCallId: CallId('c'),
|
||||
content: [{ type: 'text', text: 'xy' }],
|
||||
isError: false,
|
||||
},
|
||||
{ type: 'future-block', payload: 'abcd' } as unknown as ContentBlock,
|
||||
]
|
||||
const estimated = service.estimateMessage({ role: 'assistant', content: blocks })
|
||||
expect(estimated).toBeGreaterThan(30)
|
||||
expect(service.estimateMessage(textMessage('abcd'))).toBe(9)
|
||||
})
|
||||
|
||||
it('returns a detached deeply immutable empty measurement', () => {
|
||||
const service = meter()
|
||||
const session = new Session(SessionId('empty'))
|
||||
const result = service.measure(session)
|
||||
expect(result).toEqual({
|
||||
logRevision: 0,
|
||||
baseline: { kind: 'none', tokens: 0 },
|
||||
surfaceDeltaTokens: 0,
|
||||
totalTokens: 0,
|
||||
surfaceTokens: 0,
|
||||
nodes: [],
|
||||
})
|
||||
expect(Object.isFrozen(result)).toBe(true)
|
||||
expect(Object.isFrozen(result.baseline)).toBe(true)
|
||||
expect(Object.isFrozen(result.nodes)).toBe(true)
|
||||
expectSurfaceTotal(result)
|
||||
expect(() => {
|
||||
;(result as { totalTokens: number }).totalTokens = 1
|
||||
}).toThrow(TypeError)
|
||||
})
|
||||
|
||||
it('keeps an earlier unified snapshot detached from later replay', () => {
|
||||
const service = meter()
|
||||
const session = new Session(SessionId('detached'))
|
||||
session.append('user/message', {
|
||||
content: [{ type: 'text', text: 'first' }],
|
||||
source: { kind: 'user' },
|
||||
}, { surfaceOp: 'append' })
|
||||
const snapshot = service.measure(session)
|
||||
const snapshotCopy = structuredClone(snapshot)
|
||||
expect(Object.isFrozen(snapshot.nodes)).toBe(true)
|
||||
expect(Object.isFrozen(snapshot.nodes[0])).toBe(true)
|
||||
expectSurfaceTotal(snapshot)
|
||||
expect(() => {
|
||||
;(snapshot.nodes as Array<{ seq: number; tokens: number }>).push({ seq: 99, tokens: 1 })
|
||||
}).toThrow(TypeError)
|
||||
expect(() => {
|
||||
;(snapshot.nodes[0] as { seq: number; tokens: number }).tokens = 1
|
||||
}).toThrow(TypeError)
|
||||
|
||||
session.append('user/message', {
|
||||
content: [{ type: 'text', text: 'second' }],
|
||||
source: { kind: 'user' },
|
||||
}, { surfaceOp: 'append' })
|
||||
const advanced = service.measure(session)
|
||||
expect(advanced.logRevision).toBe(2)
|
||||
expect(advanced.nodes).toHaveLength(2)
|
||||
expectSurfaceTotal(advanced)
|
||||
expect(snapshot).toEqual(snapshotCopy)
|
||||
expect(snapshot.logRevision).toBe(1)
|
||||
expect(snapshot.nodes).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('prices header, prefix, tools, and surface when no reusable usage exists', () => {
|
||||
const service = meter()
|
||||
const session = new Session(SessionId('heuristic'))
|
||||
session.append('user/message', {
|
||||
content: [{ type: 'text', text: 'question' }],
|
||||
source: { kind: 'user' },
|
||||
}, { surfaceOp: 'append' })
|
||||
appendHeader(session, header('deepseek-v4-flash', {
|
||||
system: 'system',
|
||||
messagePrefix: [textMessage('prefix')],
|
||||
tools: [{ name: 'read', description: 'read', parameters: { type: 'object' } }],
|
||||
}))
|
||||
const result = service.measure(session)
|
||||
expect(result.baseline.kind).toBe('estimated')
|
||||
expect(result.totalTokens).toBeGreaterThan(result.surfaceTokens)
|
||||
expect(result.logRevision).toBe(session.events.length)
|
||||
expectSurfaceTotal(result)
|
||||
})
|
||||
|
||||
it('keeps request-header overrides out of the returned surface', () => {
|
||||
const service = meter()
|
||||
const session = new Session(SessionId('override-surface'))
|
||||
session.append('user/message', {
|
||||
content: [{ type: 'text', text: 'question' }],
|
||||
source: { kind: 'user' },
|
||||
}, { surfaceOp: 'append' })
|
||||
|
||||
const logged = service.measure(session)
|
||||
const overridden = service.measure(session, header('another-model', {
|
||||
system: 'large override '.repeat(100),
|
||||
}))
|
||||
expect(overridden.totalTokens).toBeGreaterThan(logged.totalTokens)
|
||||
expect(overridden.surfaceTokens).toBe(logged.surfaceTokens)
|
||||
expect(overridden.nodes).toEqual(logged.nodes)
|
||||
expectSurfaceTotal(overridden)
|
||||
})
|
||||
})
|
||||
|
||||
describe('replay anchors and surface folds', () => {
|
||||
const USAGE: TokenUsage = {
|
||||
inputTokens: 20,
|
||||
cacheReadTokens: 3,
|
||||
cacheWriteTokens: 4,
|
||||
outputTokens: 7,
|
||||
reasoningTokens: 6,
|
||||
}
|
||||
|
||||
it('uses disjoint provider usage and signed durable-output rewrites', () => {
|
||||
const service = meter()
|
||||
const session = new Session(SessionId('usage'))
|
||||
session.append('user/message', {
|
||||
content: [{ type: 'text', text: 'before' }],
|
||||
source: { kind: 'user' },
|
||||
}, { surfaceOp: 'append' })
|
||||
appendSuccessfulCall(session, header('deepseek-v4-flash'), {
|
||||
providerText: 'short',
|
||||
durableText: 'a much longer rewritten durable assistant answer',
|
||||
usage: USAGE,
|
||||
})
|
||||
const result = service.measure(session)
|
||||
expect(result.baseline).toMatchObject({ kind: 'usage', tokens: 34, usage: USAGE })
|
||||
expect(result.surfaceDeltaTokens).toBeGreaterThan(0)
|
||||
expect(result.totalTokens).toBe(34 + result.surfaceDeltaTokens)
|
||||
expect(() => {
|
||||
;((result.baseline as { usage: { inputTokens: number } }).usage.inputTokens) = 1
|
||||
}).toThrow(TypeError)
|
||||
})
|
||||
|
||||
it('selects a heuristic anchor when provider usage would undercut its scale', () => {
|
||||
const service = meter()
|
||||
const session = new Session(SessionId('low-usage-anchor'))
|
||||
const system = 'system context'
|
||||
const requestHeader = header('deepseek-v4-flash', { system })
|
||||
appendSuccessfulCall(session, requestHeader, {
|
||||
providerText: 'abcd'.repeat(512),
|
||||
usage: { inputTokens: 20, outputTokens: 7 },
|
||||
})
|
||||
|
||||
const anchored = service.measure(session)
|
||||
expect(anchored.baseline.kind).toBe('estimated')
|
||||
const assistant = anchored.nodes[0]!.seq
|
||||
session.append('user/message', {
|
||||
content: [{ type: 'text', text: 'short' }],
|
||||
source: { kind: 'plugin', plugin: 'test' },
|
||||
}, {
|
||||
surfaceOp: { op: 'replace', start: assistant, end: assistant },
|
||||
sourceEventSeqs: [assistant],
|
||||
})
|
||||
|
||||
const shrunken = service.measure(session)
|
||||
expect(27 + shrunken.surfaceDeltaTokens).toBeLessThan(0)
|
||||
expect(shrunken.totalTokens).toBeGreaterThan(0)
|
||||
expect(shrunken.totalTokens).toBe(service.measure(
|
||||
session,
|
||||
header('different-model', { system }),
|
||||
).totalTokens)
|
||||
})
|
||||
|
||||
it('uses an estimated anchor when provider usage is absent', () => {
|
||||
const service = meter()
|
||||
const session = new Session(SessionId('missing-usage'))
|
||||
appendSuccessfulCall(session, header('deepseek-v4-flash', { system: 's' }), {
|
||||
providerText: 'provider',
|
||||
durableText: 'rewritten',
|
||||
})
|
||||
const anchored = service.measure(session)
|
||||
expect(anchored.baseline.kind).toBe('estimated')
|
||||
expect(anchored.surfaceDeltaTokens).toBe(0)
|
||||
session.append('user/message', {
|
||||
content: [{ type: 'text', text: 'later' }],
|
||||
source: { kind: 'user' },
|
||||
}, { surfaceOp: 'append' })
|
||||
const advanced = service.measure(session)
|
||||
expect(advanced.surfaceDeltaTokens).toBeGreaterThan(0)
|
||||
})
|
||||
|
||||
it('distinguishes explicit empty provenance from absent legacy provenance', () => {
|
||||
const explicit = new Session(SessionId('explicit-empty'))
|
||||
const legacy = new Session(SessionId('legacy-absent'))
|
||||
appendSuccessfulCall(explicit, header('deepseek-v4-flash'), {
|
||||
durableText: 'listener injected text',
|
||||
providerText: '',
|
||||
usage: USAGE,
|
||||
provenance: 'empty',
|
||||
})
|
||||
appendSuccessfulCall(legacy, header('deepseek-v4-flash'), {
|
||||
durableText: 'listener injected text',
|
||||
providerText: '',
|
||||
usage: USAGE,
|
||||
provenance: 'absent',
|
||||
})
|
||||
const service = meter()
|
||||
expect(service.measure(explicit).surfaceDeltaTokens).toBeGreaterThan(0)
|
||||
expect(service.measure(legacy).surfaceDeltaTokens).toBe(0)
|
||||
})
|
||||
|
||||
it('keeps only the latest successful request anchor across model switches', () => {
|
||||
const service = meter({ contextWindow: 1_000 })
|
||||
const session = new Session(SessionId('switch'))
|
||||
const alphaHeader = header('alpha', { system: 'same envelope' })
|
||||
appendSuccessfulCall(session, alphaHeader, { usage: USAGE, providerText: 'alpha' })
|
||||
expect(service.measure(session).baseline).toMatchObject({ kind: 'usage', tokens: 34 })
|
||||
|
||||
appendSuccessfulCall(session, header('beta'), {
|
||||
turn: 1,
|
||||
step: 2,
|
||||
usage: { inputTokens: 100, outputTokens: 50 },
|
||||
providerText: 'beta response',
|
||||
})
|
||||
expect(service.measure(session).baseline).toMatchObject({ kind: 'usage', tokens: 150 })
|
||||
|
||||
appendHeader(session, alphaHeader)
|
||||
const switchedBack = service.measure(session)
|
||||
expect(switchedBack.baseline.kind).toBe('estimated')
|
||||
expect(switchedBack.surfaceDeltaTokens).toBe(0)
|
||||
})
|
||||
|
||||
it('invalidates usage for any canonical envelope change or explicit override', () => {
|
||||
const service = meter()
|
||||
const session = new Session(SessionId('envelope'))
|
||||
const anchoredHeader = header('deepseek-v4-flash', { system: 'one' })
|
||||
appendSuccessfulCall(session, anchoredHeader, { usage: USAGE })
|
||||
expect(service.measure(session, { ...anchoredHeader, tools: [] }).baseline.kind).toBe('usage')
|
||||
expect(service.measure(session, header('deepseek-v4-flash', { system: 'two' })).baseline.kind)
|
||||
.toBe('estimated')
|
||||
expect(service.measure(session, header('deepseek-v4-pro', { system: 'one' })).baseline.kind)
|
||||
.toBe('estimated')
|
||||
expect(service.measure(session, {
|
||||
...anchoredHeader,
|
||||
config: { ...anchoredHeader.config, temperature: 0.2 },
|
||||
}).baseline.kind).toBe('estimated')
|
||||
expect(service.measure(session, {
|
||||
...anchoredHeader,
|
||||
messagePrefix: [textMessage('prefix')],
|
||||
}).baseline.kind).toBe('estimated')
|
||||
expect(service.measure(session, {
|
||||
...anchoredHeader,
|
||||
tools: [{ name: 'read', description: 'read', parameters: { type: 'object' } }],
|
||||
}).baseline.kind).toBe('estimated')
|
||||
})
|
||||
|
||||
it('folds the latest full header snapshot into the effective envelope', () => {
|
||||
const session = new Session(SessionId('header-snapshot'))
|
||||
appendHeader(session, header('deepseek-v4-flash'))
|
||||
session.append('request/header', {
|
||||
header: header('deepseek-v4-pro'),
|
||||
reason: 'change',
|
||||
})
|
||||
const result = meter().measure(session)
|
||||
expect(result.baseline.kind).toBe('estimated')
|
||||
expect(result.logRevision).toBe(2)
|
||||
})
|
||||
|
||||
it('replays seeded append and replace operations with signed deltas', () => {
|
||||
const service = meter()
|
||||
const original = new Session(SessionId('surface-original'))
|
||||
appendSuccessfulCall(original, header('deepseek-v4-flash'), {
|
||||
usage: USAGE,
|
||||
providerText: 'long provider answer '.repeat(100),
|
||||
})
|
||||
original.append('user/message', {
|
||||
content: [{ type: 'text', text: 'new tail' }],
|
||||
source: { kind: 'user' },
|
||||
}, { surfaceOp: 'append' })
|
||||
const seeded = new Session(SessionId('surface-seeded'), original.events)
|
||||
const before = service.measure(seeded)
|
||||
expect(before.nodes).toHaveLength(2)
|
||||
expect(before.surfaceDeltaTokens).toBeGreaterThan(0)
|
||||
expectSurfaceTotal(before)
|
||||
|
||||
const first = seeded.surface.nodes[0]!
|
||||
seeded.append('user/message', {
|
||||
content: [{ type: 'text', text: 'replacement' }],
|
||||
source: { kind: 'plugin', plugin: 'test' },
|
||||
}, { surfaceOp: { op: 'replace', start: first, end: first }, sourceEventSeqs: [first] })
|
||||
const after = service.measure(seeded)
|
||||
expect(after.nodes).toHaveLength(2)
|
||||
expect(after.nodes[0]!.seq).toBe(seeded.events.length - 1)
|
||||
expect(after.logRevision).toBe(seeded.events.length)
|
||||
expect(Object.isFrozen(after.nodes)).toBe(true)
|
||||
expect(Object.isFrozen(after.nodes[0])).toBe(true)
|
||||
expect(after.surfaceDeltaTokens).toBeLessThan(0)
|
||||
expectSurfaceTotal(after)
|
||||
expect(before.nodes).toHaveLength(2)
|
||||
expect(before.logRevision).toBe(original.events.length)
|
||||
expect(before.surfaceDeltaTokens).toBeGreaterThan(0)
|
||||
})
|
||||
|
||||
it('prices an empty assistant surface anchor as zero', () => {
|
||||
const session = new Session(SessionId('empty-assistant'))
|
||||
appendSuccessfulCall(session, header('deepseek-v4-flash'), {
|
||||
providerText: '',
|
||||
durableText: '',
|
||||
provenance: 'empty',
|
||||
})
|
||||
const measurement = meter().measure(session)
|
||||
const assistant = session.events.find(event => event.type === 'assistant/message')!
|
||||
expect(measurement.nodes).toEqual([{ seq: assistant.seq, tokens: 0 }])
|
||||
expect(measurement.surfaceTokens).toBe(0)
|
||||
expectSurfaceTotal(measurement)
|
||||
})
|
||||
})
|
||||
|
||||
describe('malformed replay and listener lifecycle', () => {
|
||||
function expectRepeatedFailure(service: TokenMeterService, session: Session, pattern: RegExp): void {
|
||||
expect(() => service.measure(session)).toThrow(pattern)
|
||||
expect(() => service.measure(session)).toThrow(pattern)
|
||||
}
|
||||
|
||||
it('rejects an assistant without its step boundary transactionally', () => {
|
||||
const session = new Session(SessionId('bad-step'))
|
||||
appendHeader(session, header('deepseek-v4-flash'))
|
||||
session.append('assistant/message', {
|
||||
provenance: { provider: 'mock', model: 'deepseek-v4-flash' },
|
||||
turn: 1,
|
||||
step: 1,
|
||||
content: [{ type: 'text', text: 'bad' }],
|
||||
}, { surfaceOp: 'append', sourceEventSeqs: [] })
|
||||
expectRepeatedFailure(meter(), session, /no matching step\/start/)
|
||||
})
|
||||
|
||||
it('clears completed step boundaries and rejects overlapping or late step events', () => {
|
||||
const overlapping = new Session(SessionId('overlapping-step'))
|
||||
overlapping.append('step/start', { turn: 1, step: 1 })
|
||||
overlapping.append('step/start', { turn: 1, step: 2 })
|
||||
expectRepeatedFailure(
|
||||
meter(),
|
||||
overlapping,
|
||||
/arrived before turn 1\/step 1 ended/,
|
||||
)
|
||||
|
||||
const late = new Session(SessionId('late-assistant'))
|
||||
late.append('step/start', { turn: 1, step: 1 })
|
||||
appendHeader(late, header('deepseek-v4-flash'))
|
||||
late.append('step/end', { turn: 1, step: 1 })
|
||||
late.append('assistant/message', {
|
||||
provenance: { provider: 'mock', model: 'deepseek-v4-flash' },
|
||||
turn: 1,
|
||||
step: 1,
|
||||
content: [],
|
||||
}, { surfaceOp: 'append', sourceEventSeqs: [] })
|
||||
expectRepeatedFailure(
|
||||
meter(),
|
||||
late,
|
||||
/no matching step\/start/,
|
||||
)
|
||||
|
||||
const mismatchedEnd = new Session(SessionId('mismatched-end'))
|
||||
mismatchedEnd.append('step/start', { turn: 1, step: 1 })
|
||||
mismatchedEnd.append('step/end', { turn: 1, step: 2 })
|
||||
expectRepeatedFailure(
|
||||
meter(),
|
||||
mismatchedEnd,
|
||||
/step\/end .* no matching step\/start/,
|
||||
)
|
||||
})
|
||||
|
||||
it('rejects invalid assistant provenance', () => {
|
||||
const cases: Array<{
|
||||
name: string
|
||||
appendSource(session: Session): number[]
|
||||
pattern: RegExp
|
||||
}> = [
|
||||
{
|
||||
name: 'non-chunk',
|
||||
appendSource(session) {
|
||||
return [session.append('user/message', {
|
||||
content: [{ type: 'text', text: 'x' }],
|
||||
source: { kind: 'user' },
|
||||
}, { surfaceOp: 'append' }).seq]
|
||||
},
|
||||
pattern: /is not assistant\/chunk/,
|
||||
},
|
||||
{
|
||||
name: 'wrong-step',
|
||||
appendSource(session) {
|
||||
return [session.append('assistant/chunk', {
|
||||
turn: 1,
|
||||
step: 2,
|
||||
chunk: { type: 'finish', reason: { kind: 'stop' } },
|
||||
}).seq]
|
||||
},
|
||||
pattern: /belongs to another step/,
|
||||
},
|
||||
]
|
||||
for (const testCase of cases) {
|
||||
const session = new Session(SessionId(`bad-source-${testCase.name}`))
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
appendHeader(session, header('deepseek-v4-flash'))
|
||||
const sourceEventSeqs = testCase.appendSource(session)
|
||||
session.append('assistant/message', {
|
||||
provenance: { provider: 'mock', model: 'deepseek-v4-flash' },
|
||||
turn: 1,
|
||||
step: 1,
|
||||
content: [{ type: 'text', text: 'bad' }],
|
||||
usage: { inputTokens: 1, outputTokens: 1 },
|
||||
}, { surfaceOp: 'append', sourceEventSeqs })
|
||||
expect(() => meter().measure(session)).toThrow(testCase.pattern)
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects repeated and non-earlier assistant provenance', () => {
|
||||
const duplicate = new Session(SessionId('duplicate-source'))
|
||||
duplicate.append('step/start', { turn: 1, step: 1 })
|
||||
appendHeader(duplicate, header('deepseek-v4-flash'))
|
||||
const source = duplicate.append('assistant/chunk', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
chunk: { type: 'finish', reason: { kind: 'stop' } },
|
||||
}).seq
|
||||
appendUnchecked(duplicate, {
|
||||
type: 'assistant/message',
|
||||
seq: duplicate.seq,
|
||||
time: 0,
|
||||
data: {
|
||||
provenance: { provider: 'mock', model: 'deepseek-v4-flash' },
|
||||
turn: 1,
|
||||
step: 1,
|
||||
content: [],
|
||||
usage: { inputTokens: 1, outputTokens: 0 },
|
||||
},
|
||||
surfaceOp: 'append',
|
||||
sourceEventSeqs: [source, source],
|
||||
})
|
||||
expect(() => meter().measure(duplicate)).toThrow(/repeats source seq/)
|
||||
|
||||
const future = new Session(SessionId('future-source'))
|
||||
future.append('step/start', { turn: 1, step: 1 })
|
||||
appendHeader(future, header('deepseek-v4-flash'))
|
||||
appendUnchecked(future, {
|
||||
type: 'assistant/message',
|
||||
seq: future.seq,
|
||||
time: 0,
|
||||
data: {
|
||||
provenance: { provider: 'mock', model: 'deepseek-v4-flash' },
|
||||
turn: 1,
|
||||
step: 1,
|
||||
content: [],
|
||||
usage: { inputTokens: 1, outputTokens: 0 },
|
||||
},
|
||||
surfaceOp: 'append',
|
||||
sourceEventSeqs: [99],
|
||||
})
|
||||
expect(() => meter().measure(future)).toThrow(/is not earlier/)
|
||||
})
|
||||
|
||||
it('does not partially apply a malformed assistant replacement', () => {
|
||||
const session = new Session(SessionId('transactional-replace'))
|
||||
session.append('user/message', {
|
||||
content: [{ type: 'text', text: 'head' }],
|
||||
source: { kind: 'user' },
|
||||
}, { surfaceOp: 'append' })
|
||||
appendHeader(session, header('deepseek-v4-flash'))
|
||||
const head = session.events[0]!.seq
|
||||
session.append('assistant/message', {
|
||||
provenance: { provider: 'mock', model: 'deepseek-v4-flash' },
|
||||
turn: 1,
|
||||
step: 1,
|
||||
content: [{ type: 'text', text: 'replacement' }],
|
||||
}, { surfaceOp: { op: 'replace', start: head, end: head }, sourceEventSeqs: [head] })
|
||||
expectRepeatedFailure(
|
||||
meter(),
|
||||
session,
|
||||
/no matching step\/start/,
|
||||
)
|
||||
})
|
||||
|
||||
it('rejects corrupt replacement ranges without advancing the replay cursor', () => {
|
||||
const session = new Session(SessionId('bad-replace'))
|
||||
const head = session.append('user/message', {
|
||||
content: [{ type: 'text', text: 'head' }],
|
||||
source: { kind: 'user' },
|
||||
}, { surfaceOp: 'append' }).seq
|
||||
appendUnchecked(session, {
|
||||
type: 'user/message',
|
||||
seq: session.seq,
|
||||
time: 0,
|
||||
data: {
|
||||
content: [{ type: 'text', text: 'bad' }],
|
||||
source: { kind: 'user' },
|
||||
},
|
||||
surfaceOp: { op: 'replace', start: 99, end: 99 },
|
||||
sourceEventSeqs: [head],
|
||||
})
|
||||
expectRepeatedFailure(meter(), session, /invalid current range/)
|
||||
})
|
||||
|
||||
it('handles earlier-reader catch-up, eager observation, and service reload', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
let activeMeter: TokenMeterService | undefined
|
||||
const revisions: number[] = []
|
||||
ctx.on('session/event', (session) => {
|
||||
if (activeMeter !== undefined) revisions.push(activeMeter.measure(session).logRevision)
|
||||
})
|
||||
const firstFiber = await ctx.plugin(TokenMeterService)
|
||||
activeMeter = ctx.tokenMeter
|
||||
const session = ctx.sessions.create(SessionId('listener-order'))
|
||||
activeMeter.measure(session)
|
||||
session.append('user/message', {
|
||||
content: [{ type: 'text', text: 'one' }],
|
||||
source: { kind: 'user' },
|
||||
}, { surfaceOp: 'append' })
|
||||
expect(revisions).toEqual([1])
|
||||
expect(activeMeter.measure(session).logRevision).toBe(1)
|
||||
|
||||
await firstFiber.dispose()
|
||||
const secondFiber = await ctx.plugin(TokenMeterService)
|
||||
activeMeter = ctx.tokenMeter
|
||||
expect(activeMeter.measure(session).logRevision).toBe(1)
|
||||
await secondFiber.dispose()
|
||||
})
|
||||
})
|
||||
27
packages/llm/token-meter/tsconfig.json
Normal file
27
packages/llm/token-meter/tsconfig.json
Normal file
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user