feat(llm): add per-provider retry policies
This commit is contained in:
Turtle
@@ -6,8 +6,8 @@ The LLM seam and its provider adapters. The interface package (`llm`) owns the a
|
||||
|---|---|---|
|
||||
| `llm/` | Abstract LLM service + content-block vocabulary + chunk assembler | `ctx.llm` |
|
||||
| `token-meter/` | Replay-aware request and surface token measurement | `ctx.tokenMeter` |
|
||||
| `llm-retry/` | Bounded transient request retry policy | (listens to `agent/request-error`) |
|
||||
| `llm-retry/` | Exact-provider normal or unbounded request retry policy | (listens to `agent/request-error`) |
|
||||
| `llm-deepseek/` | DeepSeek API adapter (hand-rolled fetch/SSE) | (registers on `ctx.llm`) |
|
||||
| `llm-pi-ai/` | Multi-provider adapter via `@earendil-works/pi-ai` | (registers on `ctx.llm`) |
|
||||
|
||||
The interface lives at `llm/llm/`; adapters, retry policy, and the reusable token meter are flat siblings under the group. Requests route by `provider`, while `model` is passed through to the selected adapter. The route-owning adapter optionally resolves exact provider/model context capacity; the token meter remains model-agnostic. A new provider adapter registers one or more provider routes on `ctx.llm` without touching the interface or consumers. See [twin LLM adapters](../../.agents/notes/implemented/architecture/2026-06-13-twin-llm-adapters.md) for the two shipping implementations, the [replay token meter Agent Note](../../.agents/notes/implemented/architecture/2026-07-15-replay-token-meter-service.md) for measurement ownership, and the [routed model context Agent Note](../../.agents/notes/implemented/architecture/2026-07-20-routed-model-context-and-compaction-policy.md) for capacity and compaction-policy ownership.
|
||||
The interface lives at `llm/llm/`; adapters, retry policy, and the reusable token meter are flat siblings under the group. Requests route by `provider`, while `model` is passed through to the selected adapter. The route-owning adapter supplies retry policy and optionally resolves exact provider/model context capacity; the retry executor and token meter remain provider-agnostic. A new provider adapter registers one or more provider routes on `ctx.llm` without touching the consumers. See [twin LLM adapters](../../.agents/notes/implemented/architecture/2026-06-13-twin-llm-adapters.md) for the two shipping implementations, the [replay token meter Agent Note](../../.agents/notes/implemented/architecture/2026-07-15-replay-token-meter-service.md) for measurement ownership, and the [routed model context Agent Note](../../.agents/notes/implemented/architecture/2026-07-20-routed-model-context-and-compaction-policy.md) for capacity and compaction-policy ownership.
|
||||
|
||||
@@ -17,6 +17,12 @@ The package root exposes the Cordis plugin contract and `DeepSeekAdapter`; wire
|
||||
thinking: enabled # optional; provider default is enabled
|
||||
reasoningEffort: high # optional; high | max — omitted ⇒ not sent
|
||||
streamIdleTimeoutMs: 300000 # optional; positive finite Node timer delay; five-minute default
|
||||
retryPolicy: # optional; omission uses bounded normal defaults
|
||||
mode: always # normal | always
|
||||
backoff:
|
||||
initialDelayMs: 500
|
||||
maxDelayMs: 10000
|
||||
jitterRatio: 0.1
|
||||
defaultContextWindow: 256000 # optional positive-integer fallback for models without an exact value
|
||||
models: # optional; defaults to V4 Flash and V4 Pro
|
||||
- id: deepseek-v4-flash
|
||||
@@ -26,7 +32,7 @@ The package root exposes the Cordis plugin contract and `DeepSeekAdapter`; wire
|
||||
contextWindow: 64000
|
||||
```
|
||||
|
||||
The plugin registers the single provider route `deepseek`. A request selects it with `provider: deepseek`; its `model` is passed through as the wire `model` string, so changing DeepSeek models does not require lifecycle-time registration. Omitting `models` advertises `deepseek-v4-flash` and `deepseek-v4-pro`, each with a 128,000-token context window; an explicit list replaces those defaults, while `models: []` advertises none. Catalog entries are exposed through `ctx.llm.listModels('deepseek')` for clients such as ACP editors, but remain advisory: unlisted model ids still pass through unchanged. An omitted entry name defaults to its id.
|
||||
The plugin registers the single provider route `deepseek` together with its resolved `retryPolicy`. A request selects it with `provider: deepseek`; its `model` is passed through as the wire `model` string, so changing DeepSeek models does not require lifecycle-time registration. Omitting `models` advertises `deepseek-v4-flash` and `deepseek-v4-pro`, each with a 128,000-token context window; an explicit list replaces those defaults, while `models: []` advertises none. Catalog entries are exposed through `ctx.llm.listModels('deepseek')` for clients such as ACP editors, but remain advisory: unlisted model ids still pass through unchanged. An omitted entry name defaults to its id.
|
||||
|
||||
`contextWindow` is optional per configured model and is not exposed through the advisory catalog. `ctx.llm.resolveModelContext('deepseek', model)` returns an exact model value first, then `defaultContextWindow` for an entry without capacity or an unlisted pass-through id. When neither value exists it returns `undefined` without invalidating routing. Pressure-sensitive plugins therefore get deployment-owned capacity without treating the model selector as authoritative. Registering another adapter for `deepseek` throws `LlmError('DUPLICATE_ADAPTER')`.
|
||||
|
||||
@@ -34,7 +40,7 @@ The plugin registers the single provider route `deepseek`. A request selects it
|
||||
|
||||
`thinking`/`reasoningEffort` are adapter-level request defaults serialized as the official top-level `thinking: {type}` / `reasoning_effort` wire fields. They live in adapter config (not `GenerateOptions`) to keep the core vocabulary provider-neutral. A request with `GenerateOptions.purpose: 'session-title'` forces thinking disabled and omits `reasoning_effort`, reserving its bounded output for visible title text without changing conversation or compaction defaults.
|
||||
|
||||
`streamIdleTimeoutMs` bounds each outstanding provider read, including the initial `fetch`, without counting time the consumer spends between chunks. One stable abort signal reaches the request and body reader for the whole call; expiry stops the transport and throws `LlmError('TIMEOUT')`, while an earlier caller abort throws `LlmError('ABORTED')`. The adapter makes exactly one provider request per `stream()` call; agent-level retry is a separate plugin policy.
|
||||
`streamIdleTimeoutMs` bounds each outstanding provider read, including the initial `fetch`, without counting time the consumer spends between chunks. One stable abort signal reaches the request and body reader for the whole call; expiry stops the transport and throws `LlmError('TIMEOUT')`, while an earlier caller abort throws `LlmError('ABORTED')`. The adapter makes exactly one provider request per `stream()` call; it registers the configured policy as provider metadata, and `dsh-llm-retry` separately executes it at durable agent-step boundaries.
|
||||
|
||||
## App attribution
|
||||
|
||||
|
||||
@@ -5,12 +5,14 @@
|
||||
* @module dsh-llm-deepseek/adapter
|
||||
*/
|
||||
|
||||
import { attributionHeaders, CONTEXT_WINDOW_EXCEEDED_CODE, isContextWindowExceededError, isQuotaExceededError, LlmAdapter, LlmError, ProviderRequestId, QUOTA_EXCEEDED_CODE } from '@deepseek-ai/dsh-llm'
|
||||
import { attributionHeaders, CONTEXT_WINDOW_EXCEEDED_CODE, isContextWindowExceededError, isQuotaExceededError, LlmAdapter, LlmError, ProviderRequestId, QUOTA_EXCEEDED_CODE, resolveRetryPolicy } from '@deepseek-ai/dsh-llm'
|
||||
import type {
|
||||
GenerateOptions,
|
||||
LlmModelContext,
|
||||
LlmModelInfo,
|
||||
LlmProviderInfo,
|
||||
ResolvedRetryPolicy,
|
||||
RetryPolicyConfig,
|
||||
StreamChunk,
|
||||
} from '@deepseek-ai/dsh-llm'
|
||||
import { idleWatchdog, MAX_TIMER_DELAY_MS, timeoutOf } from '@deepseek-ai/dsh-timeout'
|
||||
@@ -46,6 +48,8 @@ export interface DeepSeekAdapterOptions {
|
||||
models?: readonly DeepSeekCatalogModel[]
|
||||
/** Maximum provider idle time while one stream read is outstanding. */
|
||||
streamIdleTimeoutMs?: number
|
||||
/** Provider-owned model-request retry policy; omission uses normal defaults. */
|
||||
retryPolicy?: RetryPolicyConfig
|
||||
}
|
||||
|
||||
/** Default maximum idle interval while an adapter stream read is outstanding. */
|
||||
@@ -95,6 +99,7 @@ export function httpErrorCode(status: number, error?: WireError['error']): strin
|
||||
*/
|
||||
export class DeepSeekAdapter extends LlmAdapter {
|
||||
private readonly streamIdleTimeoutMs: number
|
||||
private readonly retryPolicy: ResolvedRetryPolicy
|
||||
|
||||
constructor(private readonly options: DeepSeekAdapterOptions) {
|
||||
super()
|
||||
@@ -110,12 +115,17 @@ export class DeepSeekAdapter extends LlmAdapter {
|
||||
`llm-deepseek: streamIdleTimeoutMs must be a positive finite number no greater than ${MAX_TIMER_DELAY_MS}`,
|
||||
)
|
||||
}
|
||||
this.retryPolicy = resolveRetryPolicy(options.retryPolicy, 'llm-deepseek: retryPolicy')
|
||||
}
|
||||
|
||||
override providerInfo(provider: string): LlmProviderInfo {
|
||||
return { id: provider, name: 'DeepSeek' }
|
||||
}
|
||||
|
||||
override providerRetryPolicy(_provider: string): ResolvedRetryPolicy {
|
||||
return this.retryPolicy
|
||||
}
|
||||
|
||||
override listModels(provider: string): Promise<readonly LlmModelInfo[]> {
|
||||
return Promise.resolve((this.options.models ?? []).map(model => ({
|
||||
provider,
|
||||
|
||||
@@ -7,7 +7,8 @@
|
||||
|
||||
import type { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import type {} from '@deepseek-ai/dsh-llm'
|
||||
import { RetryPolicySchema } from '@deepseek-ai/dsh-llm'
|
||||
import type { RetryPolicyConfig } from '@deepseek-ai/dsh-llm'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
import { DEFAULT_STREAM_IDLE_TIMEOUT_MS, DeepSeekAdapter } from './adapter.ts'
|
||||
import type { DeepSeekCatalogModel } from './adapter.ts'
|
||||
@@ -46,6 +47,8 @@ export interface Config {
|
||||
models?: DeepSeekCatalogModel[]
|
||||
/** Maximum provider idle time while one stream read is outstanding (default five minutes). */
|
||||
streamIdleTimeoutMs?: number
|
||||
/** Provider-owned model-request retry policy; omission uses normal defaults. */
|
||||
retryPolicy?: RetryPolicyConfig
|
||||
}
|
||||
|
||||
const catalogModel: z<DeepSeekCatalogModel> = z.object({
|
||||
@@ -63,6 +66,7 @@ export const Config: z<Config> = z.object({
|
||||
defaultContextWindow: z.number().step(1).min(1),
|
||||
models: z.array(catalogModel).default(DEFAULT_MODELS),
|
||||
streamIdleTimeoutMs: z.number().min(Number.MIN_VALUE).max(MAX_TIMER_DELAY_MS).default(DEFAULT_STREAM_IDLE_TIMEOUT_MS),
|
||||
retryPolicy: RetryPolicySchema,
|
||||
})
|
||||
|
||||
/** Public API default; the internal endpoint comes from $DEEPSEEK_BASE_URL. */
|
||||
@@ -111,5 +115,6 @@ export function apply(ctx: Context, config: Config): void {
|
||||
: { defaultContextWindow: config.defaultContextWindow },
|
||||
models: resolveModels(config.models),
|
||||
streamIdleTimeoutMs: config.streamIdleTimeoutMs ?? DEFAULT_STREAM_IDLE_TIMEOUT_MS,
|
||||
...config.retryPolicy === undefined ? {} : { retryPolicy: config.retryPolicy },
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -522,6 +522,26 @@ describe('plugin registration and config', () => {
|
||||
expect(ctx.llm.listProviders()).toEqual([])
|
||||
})
|
||||
|
||||
it('registers retryPolicy from the provider config', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LlmDeepSeek, {
|
||||
apiKey: 'k',
|
||||
baseURL: 'http://127.0.0.1:1',
|
||||
retryPolicy: {
|
||||
mode: 'always',
|
||||
backoff: { initialDelayMs: 25, maxDelayMs: 100, jitterRatio: 0.2 },
|
||||
},
|
||||
})
|
||||
|
||||
expect(ctx.llm.providerRetryPolicy('deepseek')).toEqual({
|
||||
mode: 'always',
|
||||
initialDelayMs: 25,
|
||||
maxDelayMs: 100,
|
||||
jitterRatio: 0.2,
|
||||
})
|
||||
})
|
||||
|
||||
it('owns the deepseek provider and advertises the default models', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
@@ -731,4 +751,16 @@ describe('plugin registration and config', () => {
|
||||
streamIdleTimeoutMs: MAX_TIMER_DELAY_MS + 1,
|
||||
})).rejects.toThrow(/streamIdleTimeoutMs/)
|
||||
})
|
||||
|
||||
it('rejects invalid nested retryPolicy before registering the provider', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
|
||||
await expect(ctx.plugin(LlmDeepSeek, {
|
||||
apiKey: 'k',
|
||||
baseURL: 'http://127.0.0.1:1',
|
||||
retryPolicy: { mode: 'normal', maxRetries: -1 },
|
||||
})).rejects.toThrow(/retryPolicy/)
|
||||
expect(ctx.llm.listProviders()).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
@@ -17,6 +17,13 @@ Configure credentials and deployment-specific transport settings per provider. O
|
||||
apiKey: !!js process.env.OPENAI_API_KEY
|
||||
baseURL: https://proxy.example.com:8443
|
||||
reasoning: high
|
||||
retryPolicy:
|
||||
mode: normal
|
||||
maxRetries: 3
|
||||
backoff:
|
||||
initialDelayMs: 500
|
||||
maxDelayMs: 10000
|
||||
jitterRatio: 0.1
|
||||
- provider: anthropic
|
||||
apiKey: !!js process.env.ANTHROPIC_API_KEY
|
||||
streamIdleTimeoutMs: 300000
|
||||
@@ -30,7 +37,7 @@ Each provider name must exist in pi-ai's installed catalog and may appear only o
|
||||
|
||||
The adapter exposes each configured provider's installed pi-ai models through `ctx.llm.listModels(provider)`. This is provider-neutral selector metadata derived from `getModels(provider)`; request-time resolution still performs the authoritative catalog lookup, so discovery does not create a second model registry. `ctx.llm.resolveModelContext(provider, model)` performs the same exact descriptor lookup and returns its context window, keeping capacity metadata on the route-owning adapter rather than a consuming plugin.
|
||||
|
||||
Supported profile fields are `provider`, `apiKey`, `baseURL`, `headers`, `reasoning`, `thinkingBudgets`, `cacheRetention`, `transport`, `timeoutMs`, `websocketConnectTimeoutMs`, and `streamIdleTimeoutMs`. The stream-idle interval is a positive finite Node timer delay, defaults to five minutes, and covers only an outstanding provider read, not consumer think time. Harness app attribution wins a conflicting configured header name.
|
||||
Supported profile fields are `provider`, `apiKey`, `baseURL`, `headers`, `reasoning`, `thinkingBudgets`, `cacheRetention`, `transport`, `timeoutMs`, `websocketConnectTimeoutMs`, `streamIdleTimeoutMs`, and `retryPolicy`. Each profile's optional retry policy is captured with that provider route; omission uses bounded normal defaults. The stream-idle interval is a positive finite Node timer delay, defaults to five minutes, and covers only an outstanding provider read, not consumer think time. Harness app attribution wins a conflicting configured header name.
|
||||
|
||||
The adapter forces pi-ai's SDK `maxRetries` to zero so one `stream()` call makes one provider request. The removed profile fields `maxRetries` and `maxRetryDelayMs` fail load instead of silently multiplying or hiding the separately composed agent-level retry budget. Idle expiry aborts the SDK's stable request signal and surfaces `TIMEOUT`; an earlier caller abort remains `ABORTED`.
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@ import type {
|
||||
SimpleStreamOptions,
|
||||
} from '@earendil-works/pi-ai'
|
||||
import { attributionHeaders, LlmAdapter, LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, LlmModelContext, LlmModelInfo, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, LlmModelContext, LlmModelInfo, ResolvedRetryPolicy, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import { idleWatchdog, timeoutOf } from '@deepseek-ai/dsh-timeout'
|
||||
import { resolveProfiles } from './config.ts'
|
||||
import type { PiAiProviderProfile, ResolvedPiAiProviderProfile } from './config.ts'
|
||||
@@ -30,7 +30,10 @@ export interface PiAiAdapterOptions {
|
||||
* Resolve a catalog model dynamically and apply only the configured endpoint
|
||||
* override, preserving the catalog's API/capability/compatibility metadata.
|
||||
*/
|
||||
function resolveModel(profile: PiAiProviderProfile, modelId: string): Model<Api> {
|
||||
function resolveModel(
|
||||
profile: Omit<PiAiProviderProfile, 'retryPolicy'>,
|
||||
modelId: string,
|
||||
): Model<Api> {
|
||||
const model = getBuiltinModels(profile.provider as BuiltinProvider).find(candidate => candidate.id === modelId) as Model<Api> | undefined
|
||||
if (model === undefined) {
|
||||
throw new LlmError(`pi-ai provider "${profile.provider}" has no catalog model "${modelId}"`, 'UNKNOWN_MODEL')
|
||||
@@ -39,7 +42,7 @@ function resolveModel(profile: PiAiProviderProfile, modelId: string): Model<Api>
|
||||
}
|
||||
|
||||
/** Copy profile stream knobs into pi-ai's common option vocabulary. */
|
||||
function profileOptions(profile: PiAiProviderProfile): SimpleStreamOptions {
|
||||
function profileOptions(profile: Omit<PiAiProviderProfile, 'retryPolicy'>): SimpleStreamOptions {
|
||||
return {
|
||||
...profile.apiKey === undefined ? {} : { apiKey: profile.apiKey },
|
||||
...profile.reasoning === undefined ? {} : { reasoning: profile.reasoning },
|
||||
@@ -75,6 +78,10 @@ export class PiAiAdapter extends LlmAdapter {
|
||||
this.profiles = new Map(resolveProfiles(options.profiles).map(profile => [profile.provider, profile]))
|
||||
}
|
||||
|
||||
override providerRetryPolicy(provider: string): ResolvedRetryPolicy | undefined {
|
||||
return this.profiles.get(provider)?.retryPolicy
|
||||
}
|
||||
|
||||
override listModels(provider: string): Promise<readonly LlmModelInfo[]> {
|
||||
const profile = this.profiles.get(provider)
|
||||
if (profile === undefined) {
|
||||
|
||||
@@ -8,6 +8,8 @@ import { getBuiltinProviders } from '@earendil-works/pi-ai/providers/all'
|
||||
import type { CacheRetention, ThinkingBudgets, ThinkingLevel, Transport } from '@earendil-works/pi-ai'
|
||||
import z from 'schemastery'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
import { resolveRetryPolicy, RetryPolicySchema } from '@deepseek-ai/dsh-llm'
|
||||
import type { ResolvedRetryPolicy, RetryPolicyConfig } from '@deepseek-ai/dsh-llm'
|
||||
|
||||
/** Default maximum idle interval while an adapter stream read is outstanding. */
|
||||
export const DEFAULT_STREAM_IDLE_TIMEOUT_MS = 300_000
|
||||
@@ -36,12 +38,16 @@ export interface PiAiProviderProfile {
|
||||
websocketConnectTimeoutMs?: number
|
||||
/** Maximum provider idle time while one stream read is outstanding. */
|
||||
streamIdleTimeoutMs?: number
|
||||
/** Provider-owned model-request retry policy; omission uses normal defaults. */
|
||||
retryPolicy?: RetryPolicyConfig
|
||||
}
|
||||
|
||||
/** Validated profile with every adapter-owned default resolved. */
|
||||
export interface ResolvedPiAiProviderProfile extends PiAiProviderProfile {
|
||||
export interface ResolvedPiAiProviderProfile extends Omit<PiAiProviderProfile, 'retryPolicy'> {
|
||||
/** Positive finite provider-idle interval after defaulting. */
|
||||
streamIdleTimeoutMs: number
|
||||
/** Immutable retry policy captured with this provider route. */
|
||||
retryPolicy: ResolvedRetryPolicy
|
||||
}
|
||||
|
||||
/** Plugin configuration: the non-empty provider profiles this instance owns. */
|
||||
@@ -69,6 +75,7 @@ const profile = z.object({
|
||||
timeoutMs: z.natural(),
|
||||
websocketConnectTimeoutMs: z.natural(),
|
||||
streamIdleTimeoutMs: z.number().min(Number.MIN_VALUE).max(MAX_TIMER_DELAY_MS).default(DEFAULT_STREAM_IDLE_TIMEOUT_MS),
|
||||
retryPolicy: RetryPolicySchema,
|
||||
})
|
||||
|
||||
/** Runtime schema for {@link Config}. */
|
||||
@@ -115,6 +122,10 @@ export function resolveProfiles(profiles: readonly PiAiProviderProfile[]): Resol
|
||||
return {
|
||||
...source,
|
||||
streamIdleTimeoutMs,
|
||||
retryPolicy: resolveRetryPolicy(
|
||||
source.retryPolicy,
|
||||
`llm-pi-ai: provider "${source.provider}" retryPolicy`,
|
||||
),
|
||||
...source.headers === undefined ? {} : { headers: { ...source.headers } },
|
||||
...source.thinkingBudgets === undefined ? {} : { thinkingBudgets: { ...source.thinkingBudgets } },
|
||||
}
|
||||
|
||||
@@ -10,6 +10,9 @@
|
||||
* providers:
|
||||
* - provider: openai
|
||||
* apiKey: !!js process.env.OPENAI_API_KEY
|
||||
* retryPolicy:
|
||||
* mode: normal
|
||||
* maxRetries: 2
|
||||
* - provider: anthropic
|
||||
* apiKey: !!js process.env.ANTHROPIC_API_KEY
|
||||
* - provider: openrouter
|
||||
@@ -36,6 +39,6 @@ export const inject = ['llm']
|
||||
/** Register one generic pi-ai adapter for all configured provider routes. */
|
||||
export function apply(ctx: Context, config: Config): void {
|
||||
const profiles = resolveProfiles(config.providers)
|
||||
const adapter = new PiAiAdapter({ profiles })
|
||||
const adapter = new PiAiAdapter({ profiles: config.providers })
|
||||
ctx.llm.registerAdapter(profiles.map(entry => entry.provider), adapter)
|
||||
}
|
||||
|
||||
@@ -303,12 +303,31 @@ describe('provider profile lifecycle', () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
const fiber = await ctx.plugin(LlmPiAi, {
|
||||
providers: [{ provider: 'openai' }, { provider: 'anthropic' }],
|
||||
providers: [
|
||||
{
|
||||
provider: 'openai',
|
||||
retryPolicy: {
|
||||
mode: 'always',
|
||||
backoff: { initialDelayMs: 25, maxDelayMs: 100, jitterRatio: 0.2 },
|
||||
},
|
||||
},
|
||||
{ provider: 'anthropic' },
|
||||
],
|
||||
})
|
||||
expect(ctx.llm.listProviders()).toEqual([
|
||||
{ id: 'openai', name: 'openai' },
|
||||
{ id: 'anthropic', name: 'anthropic' },
|
||||
])
|
||||
expect(ctx.llm.providerRetryPolicy('openai')).toEqual({
|
||||
mode: 'always',
|
||||
initialDelayMs: 25,
|
||||
maxDelayMs: 100,
|
||||
jitterRatio: 0.2,
|
||||
})
|
||||
expect(ctx.llm.providerRetryPolicy('anthropic')).toMatchObject({
|
||||
mode: 'normal',
|
||||
maxRetries: 2,
|
||||
})
|
||||
await fiber.dispose()
|
||||
expect(ctx.llm.listProviders()).toEqual([])
|
||||
})
|
||||
@@ -373,6 +392,23 @@ describe('provider profile lifecycle', () => {
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects invalid nested retryPolicy at the provider-profile boundary', async () => {
|
||||
expect(() => resolveProfiles([{
|
||||
provider: 'openai',
|
||||
retryPolicy: { mode: 'always', backoff: { jitterRatio: -1 } },
|
||||
}])).toThrow(/retryPolicy\.backoff\.jitterRatio/)
|
||||
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await expect(ctx.plugin(LlmPiAi, {
|
||||
providers: [{
|
||||
provider: 'openai',
|
||||
retryPolicy: { mode: 'normal', maxRetries: -1 },
|
||||
}],
|
||||
})).rejects.toThrow(/retryPolicy/)
|
||||
expect(ctx.llm.listProviders()).toEqual([])
|
||||
})
|
||||
|
||||
it('constructs the adapter directly and rejects routes it does not own', async () => {
|
||||
const adapter = new PiAiAdapter({ profiles: [{ provider: 'openai' }] })
|
||||
await expect(adapter.listModels('anthropic')).rejects.toMatchObject({ code: 'NO_ADAPTER' })
|
||||
|
||||
@@ -1,41 +1,50 @@
|
||||
# `@deepseek-ai/dsh-llm-retry`
|
||||
|
||||
Function plugin that retries selected transient model-request failures on the agent loop's closed-step recovery seam. It does not wrap `ctx.llm.stream()`: every adapter call remains one provider attempt, and every retry opens a fresh numbered step.
|
||||
Function plugin that applies exact-provider retry policy on the agent loop's closed-step recovery seam. It does not wrap `ctx.llm.stream()`: every adapter call remains one provider attempt, and every retry opens a fresh numbered step.
|
||||
|
||||
The default policy permits two retries for `RATE_LIMIT`, `SERVER`, `TIMEOUT`, and `TRANSPORT`, using bounded exponential backoff from 500 ms to 10 seconds with 10 percent jitter. Delay bounds must fit Node's supported timer range. A valid `providerRetryAfterMs` replaces local backoff when it is within the configured cap; an over-cap instruction delegates to the next recovery policy instead.
|
||||
Each provider adapter owns an optional nested `retryPolicy`, captured when its route registers on `ctx.llm`. Omission uses normal mode: two retries for `RATE_LIMIT`, `SERVER`, `TIMEOUT`, and `TRANSPORT`. A normal policy can change its finite budget, eligible codes, and backoff. Always mode asks downstream recovery first, then retries every model-request failure without an attempt limit; success, cancellation, or plugin disposal stops it.
|
||||
|
||||
Before waiting, the plugin appends a non-surface `llm/retry` event with the failure and scheduled delay. Cancellation and plugin disposal abort the wait; disposal drains the plugin's active backoffs, and a callback captured before disposal fails closed if invoked afterward.
|
||||
Both modes use bounded exponential backoff with symmetric jitter. A valid `providerRetryAfterMs` at or below `maxDelayMs` replaces local backoff without jitter. An over-cap provider delay makes normal mode delegate, while always mode uses its configured local backoff so it cannot terminate on that instruction.
|
||||
|
||||
The separately published `./invariant` companion checks that every retry record names the current open turn and its latest closed step, has a unique step record and increasing retry number, and carries a positive bounded retry budget and non-negative bounded timer delay. Full jitter may schedule zero milliseconds at its lower boundary.
|
||||
Before waiting, the plugin appends a non-surface `llm/retry` event with the provider, mode, failure, and scheduled delay. Normal events include the finite maximum; always events omit it, and UIs render `∞`. Cancellation and plugin disposal abort the wait; disposal drains active backoffs, and a callback captured before disposal fails closed.
|
||||
|
||||
The separately published `./invariant` companion checks that every retry record names the current open turn and latest closed step, matches the failed request's durable provider, has a unique step record and correct provider-policy retry number, and carries a valid mode-specific budget and bounded timer delay. Full jitter may schedule zero milliseconds at its lower boundary.
|
||||
|
||||
```yaml
|
||||
- name: '@deepseek-ai/dsh-llm-retry'
|
||||
- name: '@deepseek-ai/dsh-llm-deepseek'
|
||||
config:
|
||||
maxTransientRetries: 2
|
||||
initialDelayMs: 500
|
||||
maxDelayMs: 10000
|
||||
jitterRatio: 0.1
|
||||
retryableCodes: [RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT]
|
||||
apiKey: !!js process.env.DEEPSEEK_API_KEY
|
||||
retryPolicy:
|
||||
mode: always
|
||||
backoff:
|
||||
initialDelayMs: 1000
|
||||
maxDelayMs: 30000
|
||||
jitterRatio: 0.2
|
||||
|
||||
- name: '@deepseek-ai/dsh-llm-retry'
|
||||
```
|
||||
|
||||
The executor has no policy config. Multi-provider adapters such as `dsh-llm-pi-ai` place `retryPolicy` inside each provider profile, avoiding a second provider-name list.
|
||||
|
||||
## Model Experience
|
||||
|
||||
### Transient request recovery
|
||||
### Model-request recovery
|
||||
|
||||
#### What the model sees
|
||||
|
||||
No retry event, delay, or failure prose is model-visible. After a retry, the next numbered step reconstructs the same explicit provider/model request from durable session history; failed chunks never enter derived messages.
|
||||
No retry event, delay, provider error, or failed partial output is model-visible. The next numbered step reconstructs the same explicit provider/model request from durable surface history unless a downstream recovery policy deliberately changes that surface.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Each retry is a new provider request and may repeat input-token billing. The finite budget caps attempts; `llm/retry` itself contributes no tokens.
|
||||
Each retry is a new provider request and may repeat input-token billing. Normal mode has a finite budget; always mode can consume unbounded requests until success or cancellation. `llm/retry` itself contributes no tokens.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
The reconstructed request preserves the prior prefix and is eligible for provider cache reuse under that provider's rules. The non-surface status event does not change cache identity.
|
||||
The reconstructed request preserves the prior prefix and is eligible for provider cache reuse under that provider's rules. The non-surface retry event does not change cache identity.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **Agent steps are the only retry boundary** — direct `ctx.llm.stream()` consumers remain single-attempt because a raw stream cannot separate already-emitted chunks durably.
|
||||
- **Finite plugin budgets add** — this policy counts only configured transient codes; context-overflow compaction counts only its own code. A future policy with overlapping codes must document and test registration-order behavior.
|
||||
- **Always mode retries permanent failures** — authentication, quota, invalid-request, protocol, and unrecoverable context errors continue until success, cancellation, or disposal; deployments own provider-specific cost and latency controls.
|
||||
- **Recovery policies compose by waterfall order** — always mode accepts a downstream retry before applying its fallback. A later policy that never settles also prevents the fallback from running.
|
||||
- **`llm/retry` records scheduling, not completion** — later step and turn events establish success, exhaustion, or cancellation.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-llm-retry",
|
||||
"description": "Bounded transient LLM request retry policy for the DeepSeek Harness",
|
||||
"description": "Provider-routed LLM request retry policy for the DeepSeek Harness",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
|
||||
38
packages/llm/llm-retry/src/history.ts
Normal file
38
packages/llm/llm-retry/src/history.ts
Normal file
@@ -0,0 +1,38 @@
|
||||
/** Durable request-route lookup for one closed model step. @module @deepseek-ai/dsh-llm-retry/history */
|
||||
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
|
||||
/**
|
||||
* Find the provider in force when one step closed, excluding later recovery mutations.
|
||||
* A preceding retry is also a route marker because every provider change
|
||||
* requires a newer full request-header snapshot.
|
||||
* @param events - session events containing the closed step.
|
||||
* @param turn - turn that owns the failed step.
|
||||
* @param step - failed step whose provider is required.
|
||||
* @returns the provider from the request header in force at that step boundary.
|
||||
*/
|
||||
export function providerForClosedStep(
|
||||
events: readonly SessionEvent[],
|
||||
turn: number,
|
||||
step: number,
|
||||
): string | undefined {
|
||||
const stepEndIndex = events.findLastIndex(event =>
|
||||
event.type === 'step/end'
|
||||
&& event.data.turn === turn
|
||||
&& event.data.step === step,
|
||||
)
|
||||
if (stepEndIndex < 0) return undefined
|
||||
for (let index = stepEndIndex; index >= 0; index -= 1) {
|
||||
// The loop bounds prove this indexed read exists.
|
||||
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
|
||||
const event = events[index]!
|
||||
if (event.type === 'request/header') return event.data.header.config.provider
|
||||
if (event.type === 'llm/retry'
|
||||
&& event.data.turn === turn
|
||||
&& event.data.step < step) {
|
||||
return event.data.provider
|
||||
}
|
||||
if (event.type === 'turn/start' || event.type === 'turn/end') return undefined
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
@@ -1,5 +1,5 @@
|
||||
/**
|
||||
* Bounded transient model-request retry policy on the agent loop's closed-step
|
||||
* Provider-routed model-request retry policy on the agent loop's closed-step
|
||||
* recovery seam. Each scheduled retry is durable before its cancellable wait.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-llm-retry
|
||||
@@ -8,103 +8,50 @@
|
||||
import type { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import type { Agent, RequestError, RequestErrorDecision } from '@deepseek-ai/dsh-agent'
|
||||
import type { LlmFailure } from '@deepseek-ai/dsh-llm'
|
||||
import type {} from '@deepseek-ai/dsh-session'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
import type { LlmFailure, ResolvedRetryPolicy } from '@deepseek-ai/dsh-llm'
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import { providerForClosedStep } from './history.ts'
|
||||
|
||||
declare module '@deepseek-ai/dsh-session' {
|
||||
interface SessionEventMap {
|
||||
/** Durable, non-surface record of one transient retry scheduled after a closed failed step. */
|
||||
/** Durable, non-surface record of one provider-routed retry scheduled after a closed failed step. */
|
||||
'llm/retry': {
|
||||
turn: number
|
||||
step: number
|
||||
provider: string
|
||||
mode: 'normal'
|
||||
retry: number
|
||||
maxRetries: number
|
||||
delayMs: number
|
||||
failure: LlmFailure
|
||||
} | {
|
||||
turn: number
|
||||
step: number
|
||||
provider: string
|
||||
mode: 'always'
|
||||
retry: number
|
||||
delayMs: number
|
||||
failure: LlmFailure
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export const name = 'llm-retry'
|
||||
export const inject = ['agents']
|
||||
export const inject = ['agents', 'llm']
|
||||
|
||||
const DEFAULT_MAX_TRANSIENT_RETRIES = 2
|
||||
const DEFAULT_INITIAL_DELAY_MS = 500
|
||||
const DEFAULT_MAX_DELAY_MS = 10_000
|
||||
const DEFAULT_JITTER_RATIO = 0.1
|
||||
const DEFAULT_RETRYABLE_CODES = Object.freeze(['RATE_LIMIT', 'SERVER', 'TIMEOUT', 'TRANSPORT'])
|
||||
|
||||
/** Deployment-owned limits and classification for transient request recovery. */
|
||||
export interface Config {
|
||||
/** Maximum transient retries after the first request (default 2). */
|
||||
maxTransientRetries?: number
|
||||
/** Initial local exponential-backoff delay in milliseconds (default 500). */
|
||||
initialDelayMs?: number
|
||||
/** Maximum accepted or locally scheduled delay in milliseconds (default 10000). */
|
||||
maxDelayMs?: number
|
||||
/** Symmetric random multiplier range around one (default 0.1). */
|
||||
jitterRatio?: number
|
||||
/** Stable failure codes eligible for this policy. */
|
||||
retryableCodes?: string[]
|
||||
}
|
||||
/** This policy executor has no config; providers own `retryPolicy`. */
|
||||
export type Config = Readonly<Record<never, never>>
|
||||
|
||||
/** Runtime schema for {@link Config}. */
|
||||
export const Config: z<Config> = z.object({
|
||||
maxTransientRetries: z.number().step(1).min(0).default(DEFAULT_MAX_TRANSIENT_RETRIES),
|
||||
initialDelayMs: z.number().max(MAX_TIMER_DELAY_MS).default(DEFAULT_INITIAL_DELAY_MS),
|
||||
maxDelayMs: z.number().max(MAX_TIMER_DELAY_MS).default(DEFAULT_MAX_DELAY_MS),
|
||||
jitterRatio: z.number().min(0).max(1).default(DEFAULT_JITTER_RATIO),
|
||||
retryableCodes: z.array(z.string()).default([...DEFAULT_RETRYABLE_CODES]),
|
||||
})
|
||||
export const Config: z<Config> = z.object({})
|
||||
|
||||
interface ResolvedConfig {
|
||||
readonly maxTransientRetries: number
|
||||
readonly initialDelayMs: number
|
||||
readonly maxDelayMs: number
|
||||
readonly jitterRatio: number
|
||||
readonly retryableCodes: ReadonlySet<string>
|
||||
}
|
||||
|
||||
function resolveConfig(config: Config): ResolvedConfig {
|
||||
const maxTransientRetries = config.maxTransientRetries ?? DEFAULT_MAX_TRANSIENT_RETRIES
|
||||
const initialDelayMs = config.initialDelayMs ?? DEFAULT_INITIAL_DELAY_MS
|
||||
const maxDelayMs = config.maxDelayMs ?? DEFAULT_MAX_DELAY_MS
|
||||
const jitterRatio = config.jitterRatio ?? DEFAULT_JITTER_RATIO
|
||||
const codes = config.retryableCodes ?? [...DEFAULT_RETRYABLE_CODES]
|
||||
|
||||
if (!Number.isInteger(maxTransientRetries) || maxTransientRetries < 0) {
|
||||
throw new Error('llm-retry: maxTransientRetries must be a non-negative integer')
|
||||
function validateConfig(config: Config): void {
|
||||
const [key] = Object.keys(config)
|
||||
if (key === undefined) return
|
||||
if (key === 'retryPolicy') {
|
||||
throw new Error('llm-retry: retryPolicy belongs under each provider configuration')
|
||||
}
|
||||
if (!Number.isFinite(initialDelayMs) || initialDelayMs <= 0 || initialDelayMs > MAX_TIMER_DELAY_MS) {
|
||||
throw new Error(`llm-retry: initialDelayMs must be a positive finite number no greater than ${MAX_TIMER_DELAY_MS}`)
|
||||
}
|
||||
if (!Number.isFinite(maxDelayMs) || maxDelayMs <= 0 || maxDelayMs > MAX_TIMER_DELAY_MS) {
|
||||
throw new Error(`llm-retry: maxDelayMs must be a positive finite number no greater than ${MAX_TIMER_DELAY_MS}`)
|
||||
}
|
||||
if (initialDelayMs > maxDelayMs) {
|
||||
throw new Error('llm-retry: initialDelayMs must be less than or equal to maxDelayMs')
|
||||
}
|
||||
if (!Number.isFinite(jitterRatio) || jitterRatio < 0 || jitterRatio > 1) {
|
||||
throw new Error('llm-retry: jitterRatio must be between 0 and 1')
|
||||
}
|
||||
if (codes.length === 0) {
|
||||
throw new Error('llm-retry: retryableCodes must not be empty')
|
||||
}
|
||||
if (codes.some(code => code.length === 0)) {
|
||||
throw new Error('llm-retry: retryableCodes must contain only non-empty strings')
|
||||
}
|
||||
if (new Set(codes).size !== codes.length) {
|
||||
throw new Error('llm-retry: retryableCodes must not contain duplicates')
|
||||
}
|
||||
|
||||
return Object.freeze({
|
||||
maxTransientRetries,
|
||||
initialDelayMs,
|
||||
maxDelayMs,
|
||||
jitterRatio,
|
||||
retryableCodes: new Set(codes),
|
||||
})
|
||||
throw new Error(`llm-retry: unknown key "${key}"`)
|
||||
}
|
||||
|
||||
/** Non-serializable seams used to make timing policy deterministic in tests. */
|
||||
@@ -113,7 +60,38 @@ export interface RetryInternals {
|
||||
random?: () => number
|
||||
}
|
||||
|
||||
function localDelay(config: ResolvedConfig, retry: number, random: () => number): number {
|
||||
type DownstreamOutcome =
|
||||
| { readonly type: 'decision'; readonly decision: RequestErrorDecision }
|
||||
| { readonly type: 'error'; readonly error: unknown }
|
||||
| { readonly type: 'aborted' }
|
||||
|
||||
function downstreamUntilAbort(
|
||||
next: () => Promise<RequestErrorDecision>,
|
||||
signal: AbortSignal,
|
||||
): Promise<DownstreamOutcome> {
|
||||
if (signal.aborted) return Promise.resolve({ type: 'aborted' })
|
||||
return new Promise((resolve) => {
|
||||
const finish = (outcome: DownstreamOutcome): void => {
|
||||
signal.removeEventListener('abort', onAbort)
|
||||
resolve(outcome)
|
||||
}
|
||||
const onAbort = (): void => { finish({ type: 'aborted' }) }
|
||||
signal.addEventListener('abort', onAbort, { once: true })
|
||||
let downstream: Promise<RequestErrorDecision>
|
||||
try {
|
||||
downstream = next()
|
||||
} catch (error: unknown) {
|
||||
finish({ type: 'error', error })
|
||||
return
|
||||
}
|
||||
void downstream.then(
|
||||
(decision) => { finish({ type: 'decision', decision }) },
|
||||
(error: unknown) => { finish({ type: 'error', error }) },
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
function localDelay(config: ResolvedRetryPolicy, retry: number, random: () => number): number {
|
||||
const exponent = Math.min(retry - 1, 1024)
|
||||
const exponential = Math.min(config.initialDelayMs * 2 ** exponent, config.maxDelayMs)
|
||||
const jitter = 1 - config.jitterRatio + 2 * config.jitterRatio * random()
|
||||
@@ -136,13 +114,13 @@ function cancellableDelay(delayMs: number, signal: AbortSignal): Promise<boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Install bounded transient request recovery.
|
||||
* Install provider-routed normal or unbounded request recovery.
|
||||
* @param ctx - plugin context that owns the listener and active waits.
|
||||
* @param config - retry budget, delay bounds, jitter, and eligible codes.
|
||||
* @param config - empty executor config; provider registrations own policy.
|
||||
* @param internals - non-serializable deterministic seams for tests.
|
||||
*/
|
||||
export function apply(ctx: Context, config: Config = {}, internals: RetryInternals = {}): void {
|
||||
const resolved = resolveConfig(config)
|
||||
validateConfig(config)
|
||||
const random = internals.random ?? Math.random
|
||||
const lifetime = new AbortController()
|
||||
const active = new Set<Promise<RequestErrorDecision>>()
|
||||
@@ -152,25 +130,40 @@ export function apply(ctx: Context, config: Config = {}, internals: RetryInterna
|
||||
turn: number,
|
||||
step: number,
|
||||
failure: LlmFailure,
|
||||
provider: string,
|
||||
policy: ResolvedRetryPolicy,
|
||||
retry: number,
|
||||
delayMs: number,
|
||||
signal: AbortSignal,
|
||||
): Promise<RequestErrorDecision> {
|
||||
const fusedSignal = AbortSignal.any([signal, lifetime.signal])
|
||||
if (fusedSignal.aborted) return { action: 'fail' }
|
||||
agent.session.append('llm/retry', {
|
||||
turn,
|
||||
step,
|
||||
retry,
|
||||
maxRetries: resolved.maxTransientRetries,
|
||||
delayMs,
|
||||
failure,
|
||||
})
|
||||
const eventData = policy.mode === 'normal'
|
||||
? {
|
||||
turn,
|
||||
step,
|
||||
provider,
|
||||
mode: policy.mode,
|
||||
retry,
|
||||
maxRetries: policy.maxRetries,
|
||||
delayMs,
|
||||
failure,
|
||||
}
|
||||
: {
|
||||
turn,
|
||||
step,
|
||||
provider,
|
||||
mode: policy.mode,
|
||||
retry,
|
||||
delayMs,
|
||||
failure,
|
||||
}
|
||||
agent.session.append('llm/retry', eventData)
|
||||
if (!await cancellableDelay(delayMs, fusedSignal)) return { action: 'fail' }
|
||||
return { action: 'retry' }
|
||||
}
|
||||
|
||||
const disposeListener = ctx.on('agent/request-error', (
|
||||
const disposeListener = ctx.on('agent/request-error', async (
|
||||
agent: Agent,
|
||||
turn: number,
|
||||
step: number,
|
||||
@@ -184,22 +177,61 @@ export function apply(ctx: Context, config: Config = {}, internals: RetryInterna
|
||||
// removed. Lifetime cancellation must prevent that stale callback from
|
||||
// entering a downstream policy after disposal.
|
||||
if (lifetime.signal.aborted) return Promise.resolve<RequestErrorDecision>({ action: 'fail' })
|
||||
if (!resolved.retryableCodes.has(failure.code)) return next()
|
||||
const priorTransientFailures = priorFailures.filter(item => resolved.retryableCodes.has(item.code)).length
|
||||
if (priorTransientFailures >= resolved.maxTransientRetries) return next()
|
||||
// Bind policy to the header in force when this step closed. Downstream
|
||||
// recovery may append later state before an always fallback runs.
|
||||
const provider = providerForClosedStep(agent.session.events, turn, step)
|
||||
/* v8 ignore next 3 -- agent-loop closes only steps whose request header was recorded */
|
||||
if (provider === undefined) {
|
||||
throw new Error(`llm-retry: no request provider for closed turn ${turn}/step ${step}`)
|
||||
}
|
||||
const policy = ctx.llm.providerRetryPolicy(provider)
|
||||
|
||||
const retry = priorTransientFailures + 1
|
||||
if (policy.mode === 'always') {
|
||||
const downstream = await downstreamUntilAbort(
|
||||
next,
|
||||
AbortSignal.any([signal, lifetime.signal]),
|
||||
)
|
||||
if (downstream.type === 'aborted') return { action: 'fail' }
|
||||
if (downstream.type === 'error') {
|
||||
ctx.logger.warn(
|
||||
`llm-retry: provider "${provider}" always policy ignored a downstream recovery failure: %o`,
|
||||
downstream.error,
|
||||
)
|
||||
}
|
||||
if (downstream.type === 'decision' && downstream.decision.action === 'retry') {
|
||||
return downstream.decision
|
||||
}
|
||||
} else if (!policy.retryableCodes.includes(failure.code)) {
|
||||
return next()
|
||||
}
|
||||
|
||||
const firstPriorStep = step - priorFailures.length
|
||||
const priorPolicyRetry = agent.session.events.findLast((event): event is SessionEvent<'llm/retry'> =>
|
||||
event.type === 'llm/retry'
|
||||
&& event.data.turn === turn
|
||||
&& event.data.step >= firstPriorStep
|
||||
&& event.data.step < step
|
||||
&& event.data.provider === provider
|
||||
&& event.data.mode === policy.mode,
|
||||
)
|
||||
const previousRetry = priorPolicyRetry?.data.retry ?? 0
|
||||
if (policy.mode === 'normal' && previousRetry >= policy.maxRetries) return next()
|
||||
const retry = previousRetry + 1
|
||||
let delayMs: number
|
||||
if (failure.providerRetryAfterMs !== undefined
|
||||
&& Number.isFinite(failure.providerRetryAfterMs)
|
||||
&& failure.providerRetryAfterMs > 0) {
|
||||
if (failure.providerRetryAfterMs > resolved.maxDelayMs) return next()
|
||||
delayMs = failure.providerRetryAfterMs
|
||||
if (failure.providerRetryAfterMs > policy.maxDelayMs) {
|
||||
if (policy.mode === 'normal') return next()
|
||||
delayMs = localDelay(policy, retry, random)
|
||||
} else {
|
||||
delayMs = failure.providerRetryAfterMs
|
||||
}
|
||||
} else {
|
||||
delayMs = localDelay(resolved, retry, random)
|
||||
delayMs = localDelay(policy, retry, random)
|
||||
}
|
||||
|
||||
const tracked = backoff(agent, turn, step, failure, retry, delayMs, signal)
|
||||
const tracked = backoff(agent, turn, step, failure, provider, policy, retry, delayMs, signal)
|
||||
.finally(() => active.delete(tracked))
|
||||
active.add(tracked)
|
||||
return tracked
|
||||
|
||||
@@ -4,6 +4,7 @@ import type { Context } from 'cordis'
|
||||
import type { Session, SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
import type { InvariantFailure, InvariantInstaller } from '@deepseek-ai/dsh-invariants'
|
||||
import { providerForClosedStep } from './history.ts'
|
||||
import type {} from './index.ts'
|
||||
|
||||
const PACKAGE_NAME = '@deepseek-ai/dsh-llm-retry'
|
||||
@@ -19,34 +20,46 @@ function validateRetry(
|
||||
event: SessionEvent<'llm/retry'>,
|
||||
fail: InvariantFailure,
|
||||
): void {
|
||||
const { turn, step, retry, maxRetries, delayMs } = event.data
|
||||
const { turn, step, provider, mode, retry, delayMs } = event.data
|
||||
if (!Number.isSafeInteger(retry) || retry < 1) {
|
||||
fail('llm/retry retry must be a positive safe integer')
|
||||
}
|
||||
if (!Number.isSafeInteger(maxRetries) || maxRetries < 1 || retry > maxRetries) {
|
||||
fail(`llm/retry retry ${retry} must not exceed a positive safe maxRetries ${maxRetries}`)
|
||||
if (typeof provider !== 'string' || provider.length === 0) {
|
||||
fail('llm/retry provider must be non-empty string')
|
||||
}
|
||||
if (!(delayMs >= 0 && delayMs <= MAX_TIMER_DELAY_MS)) {
|
||||
fail(`llm/retry delayMs must be within 0..${MAX_TIMER_DELAY_MS}`)
|
||||
}
|
||||
|
||||
const currentTurnEvents: SessionEvent[] = []
|
||||
let openTurn: number | undefined
|
||||
for (const prior of history.slice().reverse()) {
|
||||
if (prior.type === 'turn/end') fail('llm/retry must be appended inside an open turn')
|
||||
if (prior.type === 'turn/start') {
|
||||
openTurn = prior.data.turn
|
||||
switch (mode) {
|
||||
case 'normal': {
|
||||
const { maxRetries } = event.data
|
||||
if (!Number.isSafeInteger(maxRetries) || maxRetries < 1 || retry > maxRetries) {
|
||||
fail(`llm/retry retry ${retry} must not exceed a positive safe maxRetries ${maxRetries}`)
|
||||
}
|
||||
break
|
||||
}
|
||||
currentTurnEvents.push(prior)
|
||||
case 'always':
|
||||
if ('maxRetries' in event.data) fail('llm/retry always mode must omit maxRetries')
|
||||
break
|
||||
default:
|
||||
fail(`llm/retry mode must be normal or always, got ${String(mode)}`)
|
||||
}
|
||||
if (openTurn === undefined) fail('llm/retry must be appended inside an open turn')
|
||||
if (typeof delayMs !== 'number' || !Number.isFinite(delayMs)
|
||||
|| delayMs < 0 || delayMs > MAX_TIMER_DELAY_MS) {
|
||||
fail(`llm/retry delayMs must be a finite number within 0..${MAX_TIMER_DELAY_MS}`)
|
||||
}
|
||||
|
||||
const turnStartIndex = history.findLastIndex(prior =>
|
||||
prior.type === 'turn/start' || prior.type === 'turn/end')
|
||||
const turnBoundary = history[turnStartIndex]
|
||||
if (turnBoundary?.type !== 'turn/start') {
|
||||
fail('llm/retry must be appended inside an open turn')
|
||||
}
|
||||
const openTurn = turnBoundary.data.turn
|
||||
if (turn !== openTurn) {
|
||||
fail(`llm/retry names turn ${turn}, but the open turn is ${openTurn}`)
|
||||
}
|
||||
|
||||
const currentTurnEvents = history.slice(turnStartIndex + 1)
|
||||
let closedStep: number | undefined
|
||||
for (const prior of currentTurnEvents) {
|
||||
for (const prior of currentTurnEvents.slice().reverse()) {
|
||||
if (prior.type === 'step/start') {
|
||||
fail(`llm/retry must follow step/end, but step ${prior.data.step} is still open`)
|
||||
}
|
||||
@@ -58,15 +71,26 @@ function validateRetry(
|
||||
if (closedStep === undefined || step !== closedStep) {
|
||||
fail(`llm/retry names step ${step}, but the latest closed step is ${String(closedStep)}`)
|
||||
}
|
||||
const routedProvider = providerForClosedStep(history, turn, step)
|
||||
if (routedProvider !== provider) {
|
||||
fail(`llm/retry provider ${provider} does not match the failed request provider ${String(routedProvider)}`)
|
||||
}
|
||||
|
||||
const priorRetries = currentTurnEvents
|
||||
.filter((prior): prior is SessionEvent<'llm/retry'> => prior.type === 'llm/retry')
|
||||
if (priorRetries.some(prior => prior.data.step === step)) {
|
||||
fail(`llm/retry duplicates the retry record for turn ${turn}/step ${step}`)
|
||||
}
|
||||
const priorRetry = priorRetries[0]
|
||||
if (priorRetry !== undefined && retry <= priorRetry.data.retry) {
|
||||
fail(`llm/retry retry ${retry} must increase after retry ${priorRetry.data.retry}`)
|
||||
const lastSuccessIndex = currentTurnEvents.findLastIndex(prior => prior.type === 'assistant/message')
|
||||
const priorPolicyRetry = currentTurnEvents.findLast((prior, index): prior is SessionEvent<'llm/retry'> => (
|
||||
index > lastSuccessIndex
|
||||
&& prior.type === 'llm/retry'
|
||||
&& prior.data.provider === provider
|
||||
&& prior.data.mode === mode
|
||||
))
|
||||
const expectedRetry = (priorPolicyRetry?.data.retry ?? 0) + 1
|
||||
if (retry !== expectedRetry) {
|
||||
fail(`llm/retry retry ${retry} must equal provider policy retry ${expectedRetry}`)
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@ import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
import InvariantService from '@deepseek-ai/dsh-invariants'
|
||||
import * as RetryInvariant from '@deepseek-ai/dsh-llm-retry/invariant'
|
||||
import { providerForClosedStep } from '../src/history.ts'
|
||||
|
||||
async function setup(): Promise<Context> {
|
||||
const ctx = new Context()
|
||||
@@ -17,33 +18,116 @@ function closeStep(ctx: Context, id: string, turn = 1, step = 1) {
|
||||
const session = ctx.sessions.create(SessionId(id))
|
||||
session.append('turn/start', { turn, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('step/start', { turn, step })
|
||||
session.append('request/header', {
|
||||
header: { config: { provider: 'mock', model: 'mock' } },
|
||||
reason: 'initial',
|
||||
})
|
||||
session.append('step/end', { turn, step })
|
||||
return session
|
||||
}
|
||||
|
||||
const failure = { message: 'provider busy', code: 'RATE_LIMIT', status: 429 }
|
||||
const normal = { provider: 'mock', mode: 'normal' as const }
|
||||
|
||||
describe('llm-retry invariants', () => {
|
||||
it('has no provider without the requested closed step', () => {
|
||||
expect(providerForClosedStep([], 1, 1)).toBeUndefined()
|
||||
expect(providerForClosedStep([{
|
||||
type: 'step/end',
|
||||
data: { turn: 1, step: 1 },
|
||||
}] as never, 1, 1)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('does not inherit a provider across a turn boundary', () => {
|
||||
expect(providerForClosedStep([
|
||||
{ type: 'turn/start', data: { turn: 1 } },
|
||||
{
|
||||
type: 'request/header',
|
||||
data: { header: { config: { provider: 'prior' } } },
|
||||
},
|
||||
{ type: 'turn/end', data: { turn: 1 } },
|
||||
{ type: 'turn/start', data: { turn: 2 } },
|
||||
{ type: 'step/end', data: { turn: 2, step: 1 } },
|
||||
] as never, 2, 1)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('accepts increasing retry records for successive closed steps and ignores unrelated events', async () => {
|
||||
const ctx = await setup()
|
||||
const session = closeStep(ctx, 'retry-invariant-valid')
|
||||
expect(() => {
|
||||
session.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 2, delayMs: 500, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 2, delayMs: 500, failure,
|
||||
})
|
||||
session.append('step/start', { turn: 1, step: 2 })
|
||||
session.append('step/end', { turn: 1, step: 2 })
|
||||
session.append('llm/retry', {
|
||||
turn: 1, step: 2, retry: 2, maxRetries: 2, delayMs: 1_000, failure,
|
||||
turn: 1, step: 2, ...normal, retry: 2, maxRetries: 2, delayMs: 1_000, failure,
|
||||
})
|
||||
const zeroDelay = closeStep(ctx, 'retry-invariant-zero-delay')
|
||||
zeroDelay.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 1, delayMs: 0, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 1, delayMs: 0, failure,
|
||||
})
|
||||
}).not.toThrow()
|
||||
expect(() => { ctx.emit('tools/change') }).not.toThrow()
|
||||
})
|
||||
|
||||
it('accepts unbounded always records without serializing an infinite maximum', async () => {
|
||||
const ctx = await setup()
|
||||
const session = closeStep(ctx, 'retry-invariant-always')
|
||||
expect(() => {
|
||||
session.append('llm/retry', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
provider: 'mock',
|
||||
mode: 'always',
|
||||
retry: 1,
|
||||
delayMs: 500,
|
||||
failure,
|
||||
})
|
||||
}).not.toThrow()
|
||||
expect(() => {
|
||||
session.append('llm/retry', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
provider: 'mock',
|
||||
mode: 'always',
|
||||
retry: 1,
|
||||
maxRetries: 2,
|
||||
delayMs: 500,
|
||||
failure,
|
||||
} as never)
|
||||
}).toThrow(/always mode must omit maxRetries/)
|
||||
})
|
||||
|
||||
it('rejects empty providers and unknown modes from hostile durable input', async () => {
|
||||
const ctx = await setup()
|
||||
const emptyProvider = closeStep(ctx, 'retry-invariant-empty-provider')
|
||||
expect(() => {
|
||||
emptyProvider.append('llm/retry', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
provider: '',
|
||||
mode: 'always',
|
||||
retry: 1,
|
||||
delayMs: 1,
|
||||
failure,
|
||||
})
|
||||
}).toThrow(/provider must be non-empty/)
|
||||
|
||||
const unknownMode = closeStep(ctx, 'retry-invariant-unknown-mode')
|
||||
expect(() => {
|
||||
unknownMode.append('llm/retry', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
provider: 'mock',
|
||||
mode: 'sometimes',
|
||||
retry: 1,
|
||||
delayMs: 1,
|
||||
failure,
|
||||
} as never)
|
||||
}).toThrow(/mode must be normal or always/)
|
||||
})
|
||||
|
||||
it.each([
|
||||
[{ retry: 0, maxRetries: 2, delayMs: 1 }, /positive safe integer/],
|
||||
[{ retry: 1.5, maxRetries: 2, delayMs: 1 }, /positive safe integer/],
|
||||
@@ -56,7 +140,7 @@ describe('llm-retry invariants', () => {
|
||||
const ctx = await setup()
|
||||
const session = closeStep(ctx, `retry-invariant-bounds-${data.retry}-${data.maxRetries}-${data.delayMs}`)
|
||||
expect(() => {
|
||||
session.append('llm/retry', { turn: 1, step: 1, ...data, failure })
|
||||
session.append('llm/retry', { turn: 1, step: 1, ...normal, ...data, failure })
|
||||
}).toThrow(message)
|
||||
})
|
||||
|
||||
@@ -65,14 +149,14 @@ describe('llm-retry invariants', () => {
|
||||
const absent = ctx.sessions.create(SessionId('retry-invariant-no-turn'))
|
||||
expect(() => {
|
||||
absent.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
})
|
||||
}).toThrow(/inside an open turn/)
|
||||
|
||||
const wrongTurn = closeStep(ctx, 'retry-invariant-wrong-turn')
|
||||
expect(() => {
|
||||
wrongTurn.append('llm/retry', {
|
||||
turn: 2, step: 1, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
turn: 2, step: 1, ...normal, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
})
|
||||
}).toThrow(/open turn is 1/)
|
||||
|
||||
@@ -81,7 +165,7 @@ describe('llm-retry invariants', () => {
|
||||
openStep.append('step/start', { turn: 1, step: 1 })
|
||||
expect(() => {
|
||||
openStep.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
})
|
||||
}).toThrow(/step 1 is still open/)
|
||||
|
||||
@@ -89,14 +173,14 @@ describe('llm-retry invariants', () => {
|
||||
noStep.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
expect(() => {
|
||||
noStep.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
})
|
||||
}).toThrow(/latest closed step is undefined/)
|
||||
|
||||
const wrongStep = closeStep(ctx, 'retry-invariant-wrong-step')
|
||||
expect(() => {
|
||||
wrongStep.append('llm/retry', {
|
||||
turn: 1, step: 2, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
turn: 1, step: 2, ...normal, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
})
|
||||
}).toThrow(/latest closed step is 1/)
|
||||
|
||||
@@ -104,34 +188,97 @@ describe('llm-retry invariants', () => {
|
||||
closedTurn.append('turn/end', { turn: 1, reason: { kind: 'aborted' } })
|
||||
expect(() => {
|
||||
closedTurn.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
})
|
||||
}).toThrow(/inside an open turn/)
|
||||
})
|
||||
|
||||
it('binds the policy provider to the failed step rather than a later header', async () => {
|
||||
const ctx = await setup()
|
||||
const session = closeStep(ctx, 'retry-invariant-provider')
|
||||
session.append('request/header', {
|
||||
header: { config: { provider: 'other', model: 'mock' } },
|
||||
reason: 'change',
|
||||
})
|
||||
expect(() => {
|
||||
session.append('llm/retry', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
provider: 'mock',
|
||||
mode: 'always',
|
||||
retry: 1,
|
||||
delayMs: 1,
|
||||
failure,
|
||||
})
|
||||
}).not.toThrow()
|
||||
|
||||
const mismatch = closeStep(ctx, 'retry-invariant-provider-mismatch')
|
||||
expect(() => {
|
||||
mismatch.append('llm/retry', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
provider: 'other',
|
||||
mode: 'always',
|
||||
retry: 1,
|
||||
delayMs: 1,
|
||||
failure,
|
||||
})
|
||||
}).toThrow(/does not match the failed request provider mock/)
|
||||
})
|
||||
|
||||
it('rejects a current-turn retry without a current-turn provider route', async () => {
|
||||
const ctx = await setup()
|
||||
const session = closeStep(ctx, 'retry-invariant-prior-route')
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
session.append('turn/start', { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('step/start', { turn: 2, step: 1 })
|
||||
session.append('step/end', { turn: 2, step: 1 })
|
||||
expect(() => {
|
||||
session.append('llm/retry', {
|
||||
turn: 2,
|
||||
step: 1,
|
||||
provider: 'mock',
|
||||
mode: 'always',
|
||||
retry: 1,
|
||||
delayMs: 1,
|
||||
failure,
|
||||
})
|
||||
}).toThrow(/does not match the failed request provider undefined/)
|
||||
})
|
||||
|
||||
it('rejects non-numeric durable delays', async () => {
|
||||
const ctx = await setup()
|
||||
const session = closeStep(ctx, 'retry-invariant-delay-type')
|
||||
expect(() => {
|
||||
session.append('llm/retry', {
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 2, delayMs: '1', failure,
|
||||
} as never)
|
||||
}).toThrow(/delayMs must be a finite number/)
|
||||
})
|
||||
|
||||
it('rejects duplicate and non-increasing retry records', async () => {
|
||||
const ctx = await setup()
|
||||
const duplicate = closeStep(ctx, 'retry-invariant-duplicate')
|
||||
duplicate.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 3, delayMs: 1, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 3, delayMs: 1, failure,
|
||||
})
|
||||
expect(() => {
|
||||
duplicate.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 2, maxRetries: 3, delayMs: 1, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 2, maxRetries: 3, delayMs: 1, failure,
|
||||
})
|
||||
}).toThrow(/duplicates the retry record/)
|
||||
|
||||
const nonIncreasing = closeStep(ctx, 'retry-invariant-non-increasing')
|
||||
nonIncreasing.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 3, delayMs: 1, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 3, delayMs: 1, failure,
|
||||
})
|
||||
nonIncreasing.append('step/start', { turn: 1, step: 2 })
|
||||
nonIncreasing.append('step/end', { turn: 1, step: 2 })
|
||||
expect(() => {
|
||||
nonIncreasing.append('llm/retry', {
|
||||
turn: 1, step: 2, retry: 1, maxRetries: 3, delayMs: 1, failure,
|
||||
turn: 1, step: 2, ...normal, retry: 1, maxRetries: 3, delayMs: 1, failure,
|
||||
})
|
||||
}).toThrow(/must increase/)
|
||||
}).toThrow(/must equal provider policy retry 2/)
|
||||
})
|
||||
|
||||
it('validates existing histories on late registration', async () => {
|
||||
@@ -140,7 +287,7 @@ describe('llm-retry invariants', () => {
|
||||
const session = ctx.sessions.create(SessionId('retry-invariant-late'))
|
||||
session.append('step/end', { turn: 1, step: 1 })
|
||||
session.append('llm/retry', {
|
||||
turn: 1, step: 1, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
turn: 1, step: 1, ...normal, retry: 1, maxRetries: 2, delayMs: 1, failure,
|
||||
})
|
||||
await ctx.plugin(InvariantService)
|
||||
await expect(ctx.plugin(RetryInvariant)).rejects.toThrow(/inside an open turn/)
|
||||
|
||||
@@ -9,8 +9,8 @@ import Include from '@cordisjs/plugin-include'
|
||||
import AgentRegistry from '@deepseek-ai/dsh-agent'
|
||||
import type { Agent } from '@deepseek-ai/dsh-agent'
|
||||
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
|
||||
import LlmService, { LlmAdapter, LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import LlmService, { LlmAdapter, LlmError, resolveRetryPolicy } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, ResolvedRetryPolicy, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
|
||||
import ToolRegistry from '@deepseek-ai/dsh-tools'
|
||||
@@ -21,6 +21,16 @@ let context: Context | undefined
|
||||
|
||||
class TransientOnceAdapter extends LlmAdapter {
|
||||
requests = 0
|
||||
private readonly retryPolicy = resolveRetryPolicy({
|
||||
mode: 'normal',
|
||||
maxRetries: 1,
|
||||
retryableCodes: ['RATE_LIMIT', 'SERVER'],
|
||||
backoff: { initialDelayMs: 1, maxDelayMs: 1, jitterRatio: 0 },
|
||||
}, 'loader test provider retryPolicy')
|
||||
|
||||
override providerRetryPolicy(_provider: string): ResolvedRetryPolicy {
|
||||
return this.retryPolicy
|
||||
}
|
||||
|
||||
async * stream(_options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
this.requests += 1
|
||||
@@ -87,7 +97,7 @@ describe('real Loader composition', () => {
|
||||
// Real-Loader composition resolves workspace packages through tsx at test
|
||||
// time; first resolution after the host/client program split is slow enough
|
||||
// to trip the default 5s budget on cold caches.
|
||||
it('loads the flat policy and records recovery through the shipping loop', { timeout: 60_000 }, async () => {
|
||||
it('loads provider-supplied policy and records recovery through the shipping loop', { timeout: 60_000 }, async () => {
|
||||
const loaded = await loadYaml([
|
||||
"- name: '@deepseek-ai/dsh-llm'",
|
||||
"- name: '@deepseek-ai/dsh-session'",
|
||||
@@ -95,12 +105,6 @@ describe('real Loader composition', () => {
|
||||
"- name: '@deepseek-ai/dsh-tools'",
|
||||
"- name: '@deepseek-ai/dsh-agent'",
|
||||
"- name: '@deepseek-ai/dsh-llm-retry'",
|
||||
' config:',
|
||||
' maxTransientRetries: 1',
|
||||
' initialDelayMs: 1',
|
||||
' maxDelayMs: 1',
|
||||
' jitterRatio: 0',
|
||||
' retryableCodes: [RATE_LIMIT, SERVER]',
|
||||
"- name: '@deepseek-ai/dsh-agent-loop'",
|
||||
])
|
||||
|
||||
|
||||
@@ -34,12 +34,17 @@ describe.each(['jsonl', 'sqlite'] as const)('%s retry-event persistence', (kind)
|
||||
const session = ctx.sessions.create(SessionId(`retry-${kind}`))
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
session.append('request/header', {
|
||||
header: { config: { provider: 'mock', model: 'mock' } },
|
||||
reason: 'initial',
|
||||
})
|
||||
session.append('step/end', { turn: 1, step: 1 })
|
||||
const event = session.append('llm/retry', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
provider: 'mock',
|
||||
mode: 'always',
|
||||
retry: 1,
|
||||
maxRetries: 2,
|
||||
delayMs: 750,
|
||||
failure: { message: 'provider busy', code: 'RATE_LIMIT', status: 429 },
|
||||
})
|
||||
|
||||
@@ -1,8 +1,16 @@
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import type { Fiber } from 'cordis'
|
||||
import LlmService, { CallId, LlmAdapter, LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import LlmService, { CallId, LlmAdapter, LlmError, resolveRetryPolicy } from '@deepseek-ai/dsh-llm'
|
||||
import type {
|
||||
AlwaysRetryPolicyConfig,
|
||||
BackoffConfig,
|
||||
GenerateOptions,
|
||||
NormalRetryPolicyConfig,
|
||||
ResolvedRetryPolicy,
|
||||
RetryPolicyConfig,
|
||||
StreamChunk,
|
||||
} from '@deepseek-ai/dsh-llm'
|
||||
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
|
||||
@@ -10,13 +18,13 @@ import ToolRegistry, { defineContentToolFixture } from '@deepseek-ai/dsh-tools'
|
||||
import AgentRegistry from '@deepseek-ai/dsh-agent'
|
||||
import type { Agent, RequestErrorDecision } from '@deepseek-ai/dsh-agent'
|
||||
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
import * as retry from '../src/index.ts'
|
||||
|
||||
type ScriptEntry = Error | Iterable<StreamChunk> | AsyncIterable<StreamChunk>
|
||||
|
||||
class ScriptedAdapter extends LlmAdapter {
|
||||
readonly requests: GenerateOptions[] = []
|
||||
private retryPolicies: Readonly<Record<string, ResolvedRetryPolicy | undefined>> = {}
|
||||
|
||||
constructor(private readonly entries: ScriptEntry[]) {
|
||||
super()
|
||||
@@ -29,6 +37,21 @@ class ScriptedAdapter extends LlmAdapter {
|
||||
if (entry instanceof Error) throw entry
|
||||
yield* entry
|
||||
}
|
||||
|
||||
configureRetryPolicies(
|
||||
policies: Readonly<Record<string, RetryPolicyConfig | undefined>>,
|
||||
): void {
|
||||
this.retryPolicies = Object.fromEntries(Object.entries(policies).map(([provider, policy]) => [
|
||||
provider,
|
||||
policy === undefined
|
||||
? undefined
|
||||
: resolveRetryPolicy(policy, `retry test provider "${provider}" retryPolicy`),
|
||||
]))
|
||||
}
|
||||
|
||||
override providerRetryPolicy(provider: string): ResolvedRetryPolicy | undefined {
|
||||
return this.retryPolicies[provider]
|
||||
}
|
||||
}
|
||||
|
||||
async function* partialToolFailure(error: Error): AsyncGenerator<StreamChunk> {
|
||||
@@ -52,8 +75,8 @@ function textResponse(text: string): StreamChunk[] {
|
||||
}
|
||||
|
||||
async function harness(
|
||||
adapter: LlmAdapter,
|
||||
config: retry.Config = {},
|
||||
adapter: ScriptedAdapter,
|
||||
policies: Readonly<Record<string, RetryPolicyConfig | undefined>> = { mock: normalConfig() },
|
||||
beforeRetry?: (ctx: Context) => void,
|
||||
internals: retry.RetryInternals = {},
|
||||
): Promise<{ ctx: Context; retryFiber: Fiber }> {
|
||||
@@ -64,20 +87,44 @@ async function harness(
|
||||
await ctx.plugin(ToolRegistry)
|
||||
await ctx.plugin(AgentRegistry)
|
||||
beforeRetry?.(ctx)
|
||||
const resolvedConfig = Object.assign({
|
||||
maxTransientRetries: 2,
|
||||
initialDelayMs: 500,
|
||||
maxDelayMs: 10_000,
|
||||
jitterRatio: 0,
|
||||
}, config)
|
||||
adapter.configureRetryPolicies(policies)
|
||||
const retryFiber = await ctx.plugin(Object.assign((inner: Context) => {
|
||||
retry.apply(inner, resolvedConfig, internals)
|
||||
retry.apply(inner, {}, internals)
|
||||
}, { inject: retry.inject }))
|
||||
await ctx.plugin(AgentLoop, { agents: [] })
|
||||
ctx.llm.registerAdapter(['mock'], adapter)
|
||||
ctx.llm.registerAdapter(['mock', 'other'], adapter)
|
||||
return { ctx, retryFiber }
|
||||
}
|
||||
|
||||
function normalConfig(
|
||||
overrides: Partial<Omit<NormalRetryPolicyConfig, 'mode'>> = {},
|
||||
): NormalRetryPolicyConfig {
|
||||
const { backoff, ...policy } = overrides
|
||||
return {
|
||||
mode: 'normal',
|
||||
maxRetries: 2,
|
||||
...policy,
|
||||
backoff: {
|
||||
initialDelayMs: 500,
|
||||
maxDelayMs: 10_000,
|
||||
jitterRatio: 0,
|
||||
...backoff,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function alwaysConfig(backoff: BackoffConfig = {}): AlwaysRetryPolicyConfig {
|
||||
return {
|
||||
mode: 'always',
|
||||
backoff: {
|
||||
initialDelayMs: 500,
|
||||
maxDelayMs: 10_000,
|
||||
jitterRatio: 0,
|
||||
...backoff,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
function waitForIdle(ctx: Context, agent: Agent): Promise<void> {
|
||||
return new Promise((resolve) => {
|
||||
const dispose = ctx.on('agent/status', (subject, status) => {
|
||||
@@ -108,14 +155,14 @@ afterEach(async () => {
|
||||
context = undefined
|
||||
})
|
||||
|
||||
describe('bounded transient retry policy', () => {
|
||||
describe('provider-routed retry policy', () => {
|
||||
it('records the scheduled delay before opening a fresh request attempt', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('busy', 'RATE_LIMIT', { status: 429 }),
|
||||
textResponse('done'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter))
|
||||
;({ ctx: context } = await harness(adapter, {}, undefined, { random: () => 0.5 }))
|
||||
const agent = context.agentLoop.create(SessionId('retry-success'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
@@ -135,6 +182,8 @@ describe('bounded transient retry policy', () => {
|
||||
expect(event.data).toEqual({
|
||||
turn: 1,
|
||||
step: 1,
|
||||
provider: 'mock',
|
||||
mode: 'normal',
|
||||
retry: 1,
|
||||
maxRetries: 2,
|
||||
delayMs: 500,
|
||||
@@ -207,7 +256,9 @@ describe('bounded transient retry policy', () => {
|
||||
new LlmError('busy two', 'SERVER'),
|
||||
new LlmError('busy three', 'SERVER'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, { jitterRatio: 0.1 }, undefined, {
|
||||
;({ ctx: context } = await harness(adapter, { mock: normalConfig({
|
||||
backoff: { jitterRatio: 0.1 },
|
||||
}) }, undefined, {
|
||||
random: () => samples.shift() ?? 0.5,
|
||||
}))
|
||||
const agent = context.agentLoop.create(SessionId('retry-exhausted'), { provider: 'mock', model: 'mock' })
|
||||
@@ -238,11 +289,9 @@ describe('bounded transient retry policy', () => {
|
||||
new LlmError('busy', 'SERVER'),
|
||||
textResponse('done'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, {
|
||||
initialDelayMs: 1,
|
||||
maxDelayMs: 1,
|
||||
jitterRatio: 1,
|
||||
}, undefined, { random: () => 0 }))
|
||||
;({ ctx: context } = await harness(adapter, { mock: normalConfig({
|
||||
backoff: { initialDelayMs: 1, maxDelayMs: 1, jitterRatio: 1 },
|
||||
}) }, undefined, { random: () => 0 }))
|
||||
const agent = context.agentLoop.create(SessionId('retry-zero-delay'), { provider: 'mock', model: 'mock' })
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
|
||||
@@ -261,7 +310,9 @@ describe('bounded transient retry policy', () => {
|
||||
new LlmError('wait', 'RATE_LIMIT', { providerRetryAfterMs: 2_000 }),
|
||||
textResponse('done'),
|
||||
])
|
||||
;({ ctx: context } = await harness(accepted, { jitterRatio: 1 }))
|
||||
;({ ctx: context } = await harness(accepted, { mock: normalConfig({
|
||||
backoff: { jitterRatio: 1 },
|
||||
}) }))
|
||||
const acceptedAgent = context.agentLoop.create(SessionId('retry-after-accepted'), { provider: 'mock', model: 'mock' })
|
||||
const scheduled = waitForRetry(context, acceptedAgent, 1)
|
||||
acceptedAgent.send([{ type: 'text', text: 'go' }])
|
||||
@@ -284,6 +335,32 @@ describe('bounded transient retry policy', () => {
|
||||
expect(rejectedAgent.session.events.some(event => event.type === 'llm/retry')).toBe(false)
|
||||
})
|
||||
|
||||
it('uses local jittered backoff when always mode receives an over-cap Retry-After', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('wait too long', 'AUTH', { providerRetryAfterMs: 10 }),
|
||||
textResponse('done'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, { mock: alwaysConfig({
|
||||
initialDelayMs: 2,
|
||||
maxDelayMs: 4,
|
||||
jitterRatio: 0.5,
|
||||
}) }, undefined, { random: () => 1 }))
|
||||
const agent = context.agentLoop.create(SessionId('retry-always-over-cap'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
expect((await scheduled).data.delayMs).toBe(3)
|
||||
const idle = waitForIdle(context, agent)
|
||||
await vi.advanceTimersByTimeAsync(3)
|
||||
await idle
|
||||
|
||||
expect(adapter.requests).toHaveLength(2)
|
||||
})
|
||||
|
||||
it('delegates non-transient failures without scheduling a timer', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([new LlmError('bad key', 'AUTH')])
|
||||
@@ -297,13 +374,203 @@ describe('bounded transient retry policy', () => {
|
||||
expect(vi.getTimerCount()).toBe(0)
|
||||
})
|
||||
|
||||
it('selects policy by the failed request provider', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('mock auth failed', 'AUTH'),
|
||||
new LlmError('other auth failed', 'AUTH'),
|
||||
textResponse('other recovered'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, {
|
||||
other: alwaysConfig({ initialDelayMs: 1, maxDelayMs: 1, jitterRatio: 0 }),
|
||||
}))
|
||||
|
||||
const normalAgent = context.agentLoop.create(SessionId('retry-provider-normal'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const normalIdle = waitForIdle(context, normalAgent)
|
||||
normalAgent.send([{ type: 'text', text: 'normal' }])
|
||||
await normalIdle
|
||||
expect(normalAgent.session.events.some(event => event.type === 'llm/retry')).toBe(false)
|
||||
|
||||
const alwaysAgent = context.agentLoop.create(SessionId('retry-provider-always'), {
|
||||
provider: 'other',
|
||||
model: 'mock',
|
||||
})
|
||||
const scheduled = waitForRetry(context, alwaysAgent, 1)
|
||||
alwaysAgent.send([{ type: 'text', text: 'always' }])
|
||||
expect((await scheduled).data).toMatchObject({
|
||||
provider: 'other',
|
||||
mode: 'always',
|
||||
retry: 1,
|
||||
delayMs: 1,
|
||||
})
|
||||
const alwaysIdle = waitForIdle(context, alwaysAgent)
|
||||
await vi.advanceTimersByTimeAsync(1)
|
||||
await alwaysIdle
|
||||
|
||||
expect(adapter.requests.map(request => request.provider)).toEqual(['mock', 'other', 'other'])
|
||||
})
|
||||
|
||||
it('selects an always policy from the provider chosen by agent/request', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('rerouted auth failed', 'AUTH'),
|
||||
textResponse('rerouted recovery'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, {
|
||||
other: alwaysConfig({ initialDelayMs: 1, maxDelayMs: 1, jitterRatio: 0 }),
|
||||
}, (ctx) => {
|
||||
ctx.on('agent/request', async (_agent, _turn, _step, config) => ({
|
||||
...config,
|
||||
provider: 'other',
|
||||
}))
|
||||
}))
|
||||
const agent = context.agentLoop.create(SessionId('retry-provider-rerouted'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
|
||||
agent.send([{ type: 'text', text: 'reroute' }])
|
||||
expect((await scheduled).data).toMatchObject({ provider: 'other', mode: 'always' })
|
||||
const idle = waitForIdle(context, agent)
|
||||
await vi.advanceTimersByTimeAsync(1)
|
||||
await idle
|
||||
|
||||
expect(adapter.requests.map(request => request.provider)).toEqual(['other', 'other'])
|
||||
})
|
||||
|
||||
it('keeps always mode unbounded while preserving cancellable jittered backoff', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('auth one', 'AUTH'),
|
||||
new LlmError('auth two', 'AUTH'),
|
||||
new LlmError('auth three', 'AUTH'),
|
||||
new LlmError('auth four', 'AUTH'),
|
||||
textResponse('eventually recovered'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, { mock: alwaysConfig({
|
||||
initialDelayMs: 1,
|
||||
maxDelayMs: 4,
|
||||
jitterRatio: 0.1,
|
||||
}) }, undefined, { random: () => 1 }))
|
||||
const agent = context.agentLoop.create(SessionId('retry-always-unbounded'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const idle = waitForIdle(context, agent)
|
||||
|
||||
agent.send([{ type: 'text', text: 'keep trying' }])
|
||||
await vi.runAllTimersAsync()
|
||||
await idle
|
||||
|
||||
const events = agent.session.events.filter(event => event.type === 'llm/retry')
|
||||
expect(adapter.requests).toHaveLength(5)
|
||||
expect(events.map(event => ({
|
||||
provider: event.data.provider,
|
||||
mode: event.data.mode,
|
||||
retry: event.data.retry,
|
||||
delayMs: event.data.delayMs,
|
||||
hasMax: 'maxRetries' in event.data,
|
||||
}))).toEqual([
|
||||
{ provider: 'mock', mode: 'always', retry: 1, delayMs: 1.1, hasMax: false },
|
||||
{ provider: 'mock', mode: 'always', retry: 2, delayMs: 2.2, hasMax: false },
|
||||
{ provider: 'mock', mode: 'always', retry: 3, delayMs: 4, hasMax: false },
|
||||
{ provider: 'mock', mode: 'always', retry: 4, delayMs: 4, hasMax: false },
|
||||
])
|
||||
})
|
||||
|
||||
it('keeps failed error text and partial output out of every retried model context', async () => {
|
||||
vi.useFakeTimers()
|
||||
const diagnostic = 'private provider diagnostic must not enter context'
|
||||
const adapter = new ScriptedAdapter([
|
||||
partialToolFailure(new LlmError(diagnostic, 'AUTH')),
|
||||
textResponse('recovered without leaked context'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, { mock: alwaysConfig({
|
||||
initialDelayMs: 1,
|
||||
maxDelayMs: 1,
|
||||
}) }))
|
||||
const agent = context.agentLoop.create(SessionId('retry-always-context-isolation'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
|
||||
agent.send([{ type: 'text', text: 'safe input' }])
|
||||
await scheduled
|
||||
const idle = waitForIdle(context, agent)
|
||||
await vi.advanceTimersByTimeAsync(1)
|
||||
await idle
|
||||
|
||||
expect(adapter.requests).toHaveLength(2)
|
||||
expect(adapter.requests[1]?.messages).toEqual(adapter.requests[0]?.messages)
|
||||
const retriedContext = JSON.stringify(adapter.requests[1]?.messages)
|
||||
expect(retriedContext).not.toContain(diagnostic)
|
||||
expect(retriedContext).not.toContain('discarded partial output')
|
||||
expect(agent.session.events.some(event =>
|
||||
event.type === 'llm/retry' && event.data.failure.message === diagnostic,
|
||||
)).toBe(true)
|
||||
})
|
||||
|
||||
it('lets downstream specialized recovery run before always fallback', async () => {
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('requires specialized recovery', 'AUTH'),
|
||||
textResponse('specialized recovery won'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, { mock: alwaysConfig() }))
|
||||
context.on('agent/request-error', async () => ({ action: 'retry' }))
|
||||
const agent = context.agentLoop.create(SessionId('retry-always-composition'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const idle = waitForIdle(context, agent)
|
||||
|
||||
agent.send([{ type: 'text', text: 'recover' }])
|
||||
await idle
|
||||
|
||||
expect(adapter.requests).toHaveLength(2)
|
||||
expect(agent.session.events.some(event => event.type === 'llm/retry')).toBe(false)
|
||||
})
|
||||
|
||||
it.each([
|
||||
['synchronously', () => { throw new Error('downstream recovery failed') }],
|
||||
['asynchronously', async () => { throw new Error('downstream recovery failed') }],
|
||||
])('falls back to always retry when downstream recovery throws %s', async (_kind, failDownstream) => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('requires fallback', 'AUTH'),
|
||||
textResponse('always recovered'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, { mock: alwaysConfig({
|
||||
initialDelayMs: 1,
|
||||
maxDelayMs: 1,
|
||||
}) }))
|
||||
context.on('agent/request-error', failDownstream)
|
||||
const agent = context.agentLoop.create(SessionId('retry-always-downstream-error'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
|
||||
agent.send([{ type: 'text', text: 'recover' }])
|
||||
await scheduled
|
||||
const idle = waitForIdle(context, agent)
|
||||
await vi.advanceTimersByTimeAsync(1)
|
||||
await idle
|
||||
|
||||
expect(adapter.requests).toHaveLength(2)
|
||||
})
|
||||
|
||||
it('aborts and drains a captured backoff before plugin disposal completes', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('temporary', 'TRANSPORT'),
|
||||
textResponse('must not run'),
|
||||
])
|
||||
const mounted = await harness(adapter)
|
||||
const mounted = await harness(adapter, { mock: alwaysConfig() })
|
||||
context = mounted.ctx
|
||||
const agent = context.agentLoop.create(SessionId('retry-hmr'), { provider: 'mock', model: 'mock' })
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
@@ -322,7 +589,7 @@ describe('bounded transient retry policy', () => {
|
||||
|
||||
it('does not make plugin disposal wait for a delegated recovery policy', async () => {
|
||||
const adapter = new ScriptedAdapter([new LlmError('bad key', 'AUTH')])
|
||||
const mounted = await harness(adapter)
|
||||
const mounted = await harness(adapter, { mock: alwaysConfig() })
|
||||
context = mounted.ctx
|
||||
const downstream = Promise.withResolvers<RequestErrorDecision>()
|
||||
const entered = Promise.withResolvers<undefined>()
|
||||
@@ -353,6 +620,61 @@ describe('bounded transient retry policy', () => {
|
||||
expect(adapter.requests).toHaveLength(1)
|
||||
})
|
||||
|
||||
it('lets turn cancellation interrupt a delegated recovery policy', async () => {
|
||||
const adapter = new ScriptedAdapter([new LlmError('bad key', 'AUTH')])
|
||||
const mounted = await harness(adapter, { mock: alwaysConfig() })
|
||||
context = mounted.ctx
|
||||
const downstream = Promise.withResolvers<RequestErrorDecision>()
|
||||
const entered = Promise.withResolvers<undefined>()
|
||||
context.on('agent/request-error', () => {
|
||||
entered.resolve(undefined)
|
||||
return downstream.promise
|
||||
})
|
||||
const agent = context.agentLoop.create(SessionId('retry-delegated-cancel'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const idle = waitForIdle(context, agent)
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
await entered.promise
|
||||
|
||||
agent.cancel({ kind: 'user' })
|
||||
await idle
|
||||
downstream.resolve({ action: 'fail' })
|
||||
|
||||
expect(adapter.requests).toHaveLength(1)
|
||||
expect(agent.session.events.at(-1)).toMatchObject({
|
||||
type: 'turn/end',
|
||||
data: { reason: { kind: 'aborted' } },
|
||||
})
|
||||
})
|
||||
|
||||
it('handles synchronous cancellation while entering delegated recovery', async () => {
|
||||
const adapter = new ScriptedAdapter([new LlmError('bad key', 'AUTH')])
|
||||
const mounted = await harness(adapter, { mock: alwaysConfig() })
|
||||
context = mounted.ctx
|
||||
const downstream = Promise.withResolvers<RequestErrorDecision>()
|
||||
context.on('agent/request-error', (agent) => {
|
||||
agent.cancel({ kind: 'user' })
|
||||
return downstream.promise
|
||||
})
|
||||
const agent = context.agentLoop.create(SessionId('retry-delegated-sync-cancel'), {
|
||||
provider: 'mock',
|
||||
model: 'mock',
|
||||
})
|
||||
const idle = waitForIdle(context, agent)
|
||||
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
await idle
|
||||
downstream.resolve({ action: 'fail' })
|
||||
|
||||
expect(adapter.requests).toHaveLength(1)
|
||||
expect(agent.session.events.at(-1)).toMatchObject({
|
||||
type: 'turn/end',
|
||||
data: { reason: { kind: 'aborted' } },
|
||||
})
|
||||
})
|
||||
|
||||
it('fails a captured callback after disposal without entering downstream policy', async () => {
|
||||
const adapter = new ScriptedAdapter([new LlmError('bad key', 'AUTH')])
|
||||
const captured = Promise.withResolvers<undefined>()
|
||||
@@ -391,10 +713,10 @@ describe('bounded transient retry policy', () => {
|
||||
it('lets turn cancellation win during backoff without opening another step', async () => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('temporary', 'TIMEOUT'),
|
||||
new LlmError('permanent', 'AUTH'),
|
||||
textResponse('must not run'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter))
|
||||
;({ ctx: context } = await harness(adapter, { mock: alwaysConfig() }))
|
||||
const agent = context.agentLoop.create(SessionId('retry-cancel'), { provider: 'mock', model: 'mock' })
|
||||
const scheduled = waitForRetry(context, agent, 1)
|
||||
agent.send([{ type: 'text', text: 'go' }])
|
||||
@@ -411,13 +733,16 @@ describe('bounded transient retry policy', () => {
|
||||
expect(vi.getTimerCount()).toBe(0)
|
||||
})
|
||||
|
||||
it('lets an earlier recovery listener cancel before retry policy runs', async () => {
|
||||
it.each([
|
||||
['normal', normalConfig()],
|
||||
['always', alwaysConfig()],
|
||||
])('lets an earlier recovery listener cancel before %s retry policy runs', async (_mode, policy) => {
|
||||
vi.useFakeTimers()
|
||||
const adapter = new ScriptedAdapter([
|
||||
new LlmError('temporary', 'SERVER'),
|
||||
textResponse('must not run'),
|
||||
])
|
||||
;({ ctx: context } = await harness(adapter, {}, (ctx) => {
|
||||
;({ ctx: context } = await harness(adapter, { mock: policy }, (ctx) => {
|
||||
ctx.on('agent/request-error', async (agent, _turn, _step, _error, _failure, _history, _signal, next) => {
|
||||
agent.cancel({ kind: 'user' })
|
||||
return next()
|
||||
@@ -458,19 +783,15 @@ describe('bounded transient retry policy', () => {
|
||||
expect(vi.getTimerCount()).toBe(0)
|
||||
})
|
||||
|
||||
it.each([
|
||||
[{ maxTransientRetries: -1 }, /maxTransientRetries/],
|
||||
[{ maxTransientRetries: 1.5 }, /maxTransientRetries/],
|
||||
[{ initialDelayMs: 0 }, /initialDelayMs/],
|
||||
[{ maxDelayMs: Number.POSITIVE_INFINITY }, /maxDelayMs/],
|
||||
[{ initialDelayMs: MAX_TIMER_DELAY_MS + 1 }, /initialDelayMs/],
|
||||
[{ maxDelayMs: MAX_TIMER_DELAY_MS + 1 }, /maxDelayMs/],
|
||||
[{ initialDelayMs: 20, maxDelayMs: 10 }, /less than or equal/],
|
||||
[{ jitterRatio: 1.1 }, /jitterRatio/],
|
||||
[{ retryableCodes: [] }, /must not be empty/],
|
||||
[{ retryableCodes: ['SERVER', 'SERVER'] }, /duplicates/],
|
||||
[{ retryableCodes: [''] }, /non-empty strings/],
|
||||
] as const)('fails direct composition for invalid config %#', (config, message) => {
|
||||
expect(() => { retry.apply(new Context(), config as retry.Config) }).toThrow(message)
|
||||
it('rejects retry policy configured on the executor instead of a provider', () => {
|
||||
expect(() => {
|
||||
retry.apply(new Context(), { retryPolicy: { mode: 'always' } })
|
||||
}).toThrow(/retryPolicy belongs under each provider/)
|
||||
})
|
||||
|
||||
it('rejects unknown executor config', () => {
|
||||
expect(() => {
|
||||
retry.apply(new Context(), { retryPolciy: {} })
|
||||
}).toThrow(/unknown key "retryPolciy"/)
|
||||
})
|
||||
})
|
||||
|
||||
25
packages/llm/llm-retry/tsdown.config.ts
Normal file
25
packages/llm/llm-retry/tsdown.config.ts
Normal file
@@ -0,0 +1,25 @@
|
||||
import { defineConfig } from 'tsdown'
|
||||
|
||||
/** Build the package root and invariant companion as independent bundles. */
|
||||
export default defineConfig([
|
||||
{
|
||||
entry: ['lib/types/index.js'],
|
||||
outDir: 'lib',
|
||||
format: ['esm'],
|
||||
platform: 'node',
|
||||
target: 'es2024',
|
||||
fixedExtension: false,
|
||||
dts: false,
|
||||
clean: false,
|
||||
},
|
||||
{
|
||||
entry: ['lib/types/invariant.js'],
|
||||
outDir: 'lib',
|
||||
format: ['esm'],
|
||||
platform: 'node',
|
||||
target: 'es2024',
|
||||
fixedExtension: false,
|
||||
dts: false,
|
||||
clean: false,
|
||||
},
|
||||
])
|
||||
@@ -10,13 +10,14 @@ An adapter registry plus a single streaming call surface, interceptable via a wa
|
||||
|
||||
- `ctx.llm.registerAdapter(providers: string[], adapter: LlmAdapter): () => void` Register one adapter instance for the given provider routes. Registration is all-or-nothing, and is disposed with the calling fiber.
|
||||
- `ctx.llm.listProviders(): LlmProviderInfo[]` Describe registered provider routes in registration order.
|
||||
- `ctx.llm.providerRetryPolicy(provider: string): ResolvedRetryPolicy` Return the provider-owned retry policy captured during registration, with normal defaults resolved.
|
||||
- `ctx.llm.listModels(provider: string): Promise<LlmModelInfo[]>` Discover the models one registered provider currently advertises.
|
||||
- `ctx.llm.resolveModelContext(provider: string, model: string): Promise<LlmModelContext | undefined>` Resolve authoritative context capacity for one exact route from its owning adapter.
|
||||
- `ctx.llm.stream(options: GenerateOptions): AsyncIterable<StreamChunk>` Stream one model call as raw chunks (token-level deltas). Consumers assemble the chunks into blocks/messages with `BlockAssembler`.
|
||||
|
||||
`LlmService` preserves errors from final adapter selection, synchronous dispatch, iterator construction, and iteration, and binds their provenance to the exact stream handle returned for that model call. `isLlmAdapterFailure(stream, value)` reports only errors from that call's final adapter boundary; `llmFailureOf(stream, value)` returns the adjacent immutable `LlmFailure`. Nested model calls, `llm/stream` middleware, and downstream consumer failures remain unclassified for the outer call. Classification never replaces or mutates the adapter's original coded `Error`.
|
||||
|
||||
Provider and model metadata is a discovery surface, not a routing whitelist. `registerAdapter()` still owns provider exclusivity, while an adapter may accept model ids absent from `listModels()`; consumers must not reject a request because its model is unlisted. Returned metadata is detached and invalid or duplicate adapter entries fail with `INVALID_ADAPTER` or `INVALID_CATALOG`.
|
||||
Provider and model metadata is a discovery surface, not a routing whitelist. `registerAdapter()` still owns provider exclusivity and captures the adapter's retry policy for each route, while an adapter may accept model ids absent from `listModels()`; consumers must not reject a request because its model is unlisted. Returned selector metadata is detached and invalid or duplicate adapter entries fail with `INVALID_ADAPTER` or `INVALID_CATALOG`.
|
||||
|
||||
Context capacity is a separate correctness query, not a catalog decoration or global LLM setting. `resolveModelContext()` asks the adapter that owns the exact provider/model route; an adapter can describe an unlisted dynamic model, and `undefined` means only that capacity is unavailable. Invalid returned capacity fails with `INVALID_MODEL_CONTEXT`.
|
||||
|
||||
@@ -28,7 +29,7 @@ Context capacity is a separate correctness query, not a catalog decoration or gl
|
||||
|
||||
### Extension points
|
||||
|
||||
- Subclass `LlmAdapter` and call `ctx.llm.registerAdapter(providers, adapter)` to add one or more provider routes. `GenerateOptions.provider` selects the adapter; `GenerateOptions.model` is adapter-owned and may be resolved dynamically. Override `providerInfo()` and asynchronous `listModels()` to expose selector metadata, and `resolveModelContext()` when exact capacity is known; the defaults use the route id as its name, advertise no models, and return no capacity.
|
||||
- Subclass `LlmAdapter` and call `ctx.llm.registerAdapter(providers, adapter)` to add one or more provider routes. `GenerateOptions.provider` selects the adapter; `GenerateOptions.model` is adapter-owned and may be resolved dynamically. Override `providerRetryPolicy()` to supply provider-owned recovery configuration, `providerInfo()` and asynchronous `listModels()` to expose selector metadata, and `resolveModelContext()` when exact capacity is known; the defaults use bounded normal retry policy, use the route id as its name, advertise no models, and return no capacity.
|
||||
- Wrap `llm/stream` via `ctx.on()` waterfall listeners for caching, logging, or routing. A wrapper that retries after emitting a chunk has no durable attempt boundary; shipped agent retry policy therefore uses `agent/request-error` instead.
|
||||
|
||||
### Content-block vocabulary (`types.ts`)
|
||||
@@ -69,7 +70,7 @@ Pass-through; the registry preserves the assembled request prefix, while the sel
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **No default retry/caching/rate-limit policy ships in this service** — `llm/stream` remains a single-attempt call-wrapper seam; the agent loop separately offers proven model-request failures to `agent/request-error`, whose default preserves the original failure. `@deepseek-ai/dsh-llm-retry` is an optional policy plugin loaded by the shared example spine.
|
||||
- **No retry execution, caching, or rate limiting ships in this service** — provider registration stores retry policy, but `llm/stream` remains a single-attempt call-wrapper seam. The agent loop separately offers proven model-request failures to `agent/request-error`, whose default preserves the original failure; `@deepseek-ai/dsh-llm-retry` is the optional executor loaded by the shared example spine.
|
||||
- **`GenerateOptions` sampling is `temperature`/`maxTokens`/`stop` only** — no `tool_choice`, `top_p`, or penalty fields; the vocabulary grows when a producer lands ([dropped inert knobs](../../../.agents/notes/implemented/simplification/2026-07-04-drop-inert-request-knobs.md)).
|
||||
- **Producer-gated variants stay out until produced** — `prefill`, per-tool `strict`, block `cache` hints, and the `agent` message-source variant were pruned as producerless ([Agent Note](../../../.agents/notes/implemented/simplification/2026-07-04-prune-producerless-vocabulary-variants.md)).
|
||||
- **`BlockAssembler` handles core block kinds only** — a plugin-added block type whose stream is never closed by `block-end` makes `blocks()` throw.
|
||||
|
||||
@@ -38,11 +38,16 @@
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-brand": "^0.0.1",
|
||||
"@deepseek-ai/dsh-invariants": "^0.0.1",
|
||||
"@deepseek-ai/dsh-timeout": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"dependencies": {
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-brand": "workspace:^",
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-timeout": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -16,6 +16,8 @@ import type {
|
||||
Message,
|
||||
StreamChunk,
|
||||
} from './types.ts'
|
||||
import { resolveRetryPolicy } from './retry-policy.ts'
|
||||
import type { ResolvedRetryPolicy } from './retry-policy.ts'
|
||||
import type { ProviderRequestId } from './brand.ts'
|
||||
import { deepFreeze } from './call-config.ts'
|
||||
import { HarnessError } from './error.ts'
|
||||
@@ -27,6 +29,7 @@ export * from './brand.ts'
|
||||
export * from './never.ts'
|
||||
export * from './error.ts'
|
||||
export * from './types.ts'
|
||||
export * from './retry-policy.ts'
|
||||
export { BlockAssembler } from './assembler.ts'
|
||||
export { callConfigEquals, deepFreeze, isAgentLoopRequest, markAgentLoopRequest } from './call-config.ts'
|
||||
export type { LlmCallConfig } from './call-config.ts'
|
||||
@@ -119,6 +122,15 @@ export abstract class LlmAdapter {
|
||||
return { id: provider, name: provider }
|
||||
}
|
||||
|
||||
/**
|
||||
* Return the provider-owned retry policy captured with this route.
|
||||
* @param _provider - a route passed to `registerAdapter()` for this instance.
|
||||
* @returns a resolved policy, or `undefined` to use the normal defaults.
|
||||
*/
|
||||
providerRetryPolicy(_provider: string): ResolvedRetryPolicy | undefined {
|
||||
return undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* List models this adapter can currently advertise for one owned provider.
|
||||
* The result is advisory: an adapter may accept unlisted model ids, and
|
||||
@@ -157,7 +169,11 @@ export abstract class LlmAdapter {
|
||||
* surface, interceptable via the `llm/stream` waterfall.
|
||||
*/
|
||||
export class LlmService extends Service {
|
||||
private adapters = new Map<string, { adapter: LlmAdapter; provider: LlmProviderInfo }>()
|
||||
private adapters = new Map<string, {
|
||||
adapter: LlmAdapter
|
||||
provider: LlmProviderInfo
|
||||
retryPolicy: ResolvedRetryPolicy
|
||||
}>()
|
||||
|
||||
constructor(ctx: Context) {
|
||||
super(ctx, 'llm')
|
||||
@@ -175,7 +191,11 @@ export class LlmService extends Service {
|
||||
const dispose = this.ctx.effect(function* (this: LlmService) {
|
||||
if (providers.length === 0) throw new LlmError('an adapter must register at least one provider', 'INVALID_ADAPTER')
|
||||
const unique = new Set<string>()
|
||||
const registrations: { adapter: LlmAdapter; provider: LlmProviderInfo }[] = []
|
||||
const registrations: {
|
||||
adapter: LlmAdapter
|
||||
provider: LlmProviderInfo
|
||||
retryPolicy: ResolvedRetryPolicy
|
||||
}[] = []
|
||||
for (const provider of providers) {
|
||||
if (provider.length === 0) throw new LlmError('adapter provider names must be non-empty', 'INVALID_ADAPTER')
|
||||
if (unique.has(provider) || this.adapters.has(provider)) {
|
||||
@@ -186,7 +206,13 @@ export class LlmService extends Service {
|
||||
throw new LlmError(`adapter metadata for provider "${provider}" must preserve its id and have a non-empty name`, 'INVALID_ADAPTER')
|
||||
}
|
||||
unique.add(provider)
|
||||
registrations.push({ adapter, provider: { id: info.id, name: info.name } })
|
||||
const retryPolicy = adapter.providerRetryPolicy(provider)
|
||||
?? resolveRetryPolicy(undefined, `llm: provider "${provider}" retryPolicy`)
|
||||
registrations.push({
|
||||
adapter,
|
||||
provider: { id: info.id, name: info.name },
|
||||
retryPolicy,
|
||||
})
|
||||
}
|
||||
for (const registration of registrations) this.adapters.set(registration.provider.id, registration)
|
||||
yield () => {
|
||||
@@ -206,6 +232,15 @@ export class LlmService extends Service {
|
||||
return [...this.adapters.values()].map(({ provider }) => ({ ...provider }))
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the retry policy captured when one provider route was registered.
|
||||
* @param provider - registered provider route to inspect.
|
||||
* @returns the provider-owned policy, with normal defaults already resolved.
|
||||
*/
|
||||
providerRetryPolicy(provider: string): ResolvedRetryPolicy {
|
||||
return this.registration(provider).retryPolicy
|
||||
}
|
||||
|
||||
/**
|
||||
* Discover models advertised by one registered provider. Catalog membership
|
||||
* is advisory and never changes routing or request validation.
|
||||
@@ -262,7 +297,11 @@ export class LlmService extends Service {
|
||||
return { contextWindow: context.contextWindow }
|
||||
}
|
||||
|
||||
private registration(provider: string): { adapter: LlmAdapter; provider: LlmProviderInfo } {
|
||||
private registration(provider: string): {
|
||||
adapter: LlmAdapter
|
||||
provider: LlmProviderInfo
|
||||
retryPolicy: ResolvedRetryPolicy
|
||||
} {
|
||||
const registration = this.adapters.get(provider)
|
||||
if (!registration) throw new LlmError(`no adapter registered for provider "${provider}"`, 'NO_ADAPTER')
|
||||
return registration
|
||||
|
||||
184
packages/llm/llm/src/retry-policy.ts
Normal file
184
packages/llm/llm/src/retry-policy.ts
Normal file
@@ -0,0 +1,184 @@
|
||||
/**
|
||||
* Provider-owned request-retry policy configuration and resolution.
|
||||
*
|
||||
* Adapters expose one resolved policy per registered provider route; the
|
||||
* optional dsh-llm-retry plugin executes it on the agent's failed-step seam.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-llm/retry-policy
|
||||
*/
|
||||
|
||||
import z from 'schemastery'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
|
||||
const DEFAULT_MAX_RETRIES = 2
|
||||
const DEFAULT_INITIAL_DELAY_MS = 500
|
||||
const DEFAULT_MAX_DELAY_MS = 10_000
|
||||
const DEFAULT_JITTER_RATIO = 0.1
|
||||
const DEFAULT_RETRYABLE_CODES = Object.freeze(['RATE_LIMIT', 'SERVER', 'TIMEOUT', 'TRANSPORT'])
|
||||
|
||||
/** Bounded exponential backoff with symmetric jitter around each local delay. */
|
||||
export interface BackoffConfig {
|
||||
/** Initial local exponential-backoff delay in milliseconds (default 500). */
|
||||
initialDelayMs?: number
|
||||
/** Maximum locally scheduled or accepted provider delay in milliseconds (default 10000). */
|
||||
maxDelayMs?: number
|
||||
/** Symmetric random multiplier range around one (default 0.1). */
|
||||
jitterRatio?: number
|
||||
}
|
||||
|
||||
/** Current bounded transient retry behavior for one provider route. */
|
||||
export interface NormalRetryPolicyConfig {
|
||||
/** Retry only configured transient failure codes. */
|
||||
mode: 'normal'
|
||||
/** Maximum eligible retries after the first request (default 2). */
|
||||
maxRetries?: number
|
||||
/** Stable failure codes eligible for this policy. */
|
||||
retryableCodes?: string[]
|
||||
/** Local exponential-backoff and jitter configuration. */
|
||||
backoff?: BackoffConfig
|
||||
}
|
||||
|
||||
/** Unbounded retry behavior for every model-request failure on one provider route. */
|
||||
export interface AlwaysRetryPolicyConfig {
|
||||
/** Retry every model-request failure until success, cancellation, or disposal. */
|
||||
mode: 'always'
|
||||
/** Local exponential-backoff and jitter configuration. */
|
||||
backoff?: BackoffConfig
|
||||
}
|
||||
|
||||
/** Provider-owned model-request retry policy configuration. */
|
||||
export type RetryPolicyConfig = NormalRetryPolicyConfig | AlwaysRetryPolicyConfig
|
||||
|
||||
/** Fully resolved backoff shared by both retry modes. */
|
||||
export interface ResolvedRetryBackoff {
|
||||
readonly initialDelayMs: number
|
||||
readonly maxDelayMs: number
|
||||
readonly jitterRatio: number
|
||||
}
|
||||
|
||||
/** Fully resolved bounded transient retry policy. */
|
||||
export interface ResolvedNormalRetryPolicy extends ResolvedRetryBackoff {
|
||||
readonly mode: 'normal'
|
||||
readonly maxRetries: number
|
||||
readonly retryableCodes: readonly string[]
|
||||
}
|
||||
|
||||
/** Fully resolved unbounded retry policy. */
|
||||
export interface ResolvedAlwaysRetryPolicy extends ResolvedRetryBackoff {
|
||||
readonly mode: 'always'
|
||||
}
|
||||
|
||||
/** Immutable provider policy captured when its adapter route is registered. */
|
||||
export type ResolvedRetryPolicy = ResolvedNormalRetryPolicy | ResolvedAlwaysRetryPolicy
|
||||
|
||||
const backoffSchema: z<BackoffConfig> = z.object({
|
||||
initialDelayMs: z.number().max(MAX_TIMER_DELAY_MS).default(DEFAULT_INITIAL_DELAY_MS),
|
||||
maxDelayMs: z.number().max(MAX_TIMER_DELAY_MS).default(DEFAULT_MAX_DELAY_MS),
|
||||
jitterRatio: z.number().min(0).max(1).default(DEFAULT_JITTER_RATIO),
|
||||
})
|
||||
|
||||
const normalPolicySchema: z<NormalRetryPolicyConfig> = z.object({
|
||||
mode: z.const('normal').required(),
|
||||
maxRetries: z.number().step(1).min(0).max(Number.MAX_SAFE_INTEGER).default(DEFAULT_MAX_RETRIES),
|
||||
retryableCodes: z.array(z.string()).default([...DEFAULT_RETRYABLE_CODES]),
|
||||
backoff: backoffSchema,
|
||||
})
|
||||
|
||||
const alwaysPolicySchema: z<AlwaysRetryPolicyConfig> = z.object({
|
||||
mode: z.const('always').required(),
|
||||
backoff: backoffSchema,
|
||||
})
|
||||
|
||||
/** Cordis schema embedded by each concrete provider configuration. */
|
||||
export const RetryPolicySchema: z<RetryPolicyConfig> = z.union([
|
||||
normalPolicySchema,
|
||||
alwaysPolicySchema,
|
||||
])
|
||||
|
||||
const NORMAL_POLICY_KEYS: ReadonlySet<string> = new Set([
|
||||
'mode', 'maxRetries', 'retryableCodes', 'backoff',
|
||||
])
|
||||
const ALWAYS_POLICY_KEYS: ReadonlySet<string> = new Set(['mode', 'backoff'])
|
||||
const BACKOFF_KEYS: ReadonlySet<string> = new Set(['initialDelayMs', 'maxDelayMs', 'jitterRatio'])
|
||||
|
||||
function validateKeys(value: object, allowed: ReadonlySet<string>, path: string): void {
|
||||
for (const key of Object.keys(value)) {
|
||||
if (!allowed.has(key)) throw new Error(`${path}: unknown key "${key}"`)
|
||||
}
|
||||
}
|
||||
|
||||
function resolveBackoff(config: BackoffConfig | undefined, path: string): ResolvedRetryBackoff {
|
||||
if (config !== undefined) validateKeys(config, BACKOFF_KEYS, path)
|
||||
const initialDelayMs = config?.initialDelayMs ?? DEFAULT_INITIAL_DELAY_MS
|
||||
const maxDelayMs = config?.maxDelayMs ?? DEFAULT_MAX_DELAY_MS
|
||||
const jitterRatio = config?.jitterRatio ?? DEFAULT_JITTER_RATIO
|
||||
|
||||
if (!Number.isFinite(initialDelayMs) || initialDelayMs <= 0 || initialDelayMs > MAX_TIMER_DELAY_MS) {
|
||||
throw new Error(`${path}.initialDelayMs must be a positive finite number no greater than ${MAX_TIMER_DELAY_MS}`)
|
||||
}
|
||||
if (!Number.isFinite(maxDelayMs) || maxDelayMs <= 0 || maxDelayMs > MAX_TIMER_DELAY_MS) {
|
||||
throw new Error(`${path}.maxDelayMs must be a positive finite number no greater than ${MAX_TIMER_DELAY_MS}`)
|
||||
}
|
||||
if (initialDelayMs > maxDelayMs) {
|
||||
throw new Error(`${path}.initialDelayMs must be less than or equal to maxDelayMs`)
|
||||
}
|
||||
if (!Number.isFinite(jitterRatio) || jitterRatio < 0 || jitterRatio > 1) {
|
||||
throw new Error(`${path}.jitterRatio must be between 0 and 1`)
|
||||
}
|
||||
|
||||
return Object.freeze({ initialDelayMs, maxDelayMs, jitterRatio })
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate, default, and detach one provider-owned retry policy.
|
||||
* @param config - optional provider configuration; omission selects normal defaults.
|
||||
* @param path - diagnostic path naming the provider config that owns the value.
|
||||
* @returns an immutable policy safe to capture in provider registration state.
|
||||
*/
|
||||
export function resolveRetryPolicy(
|
||||
config: RetryPolicyConfig | undefined,
|
||||
path: string,
|
||||
): ResolvedRetryPolicy {
|
||||
if (config === undefined) {
|
||||
return Object.freeze({
|
||||
mode: 'normal',
|
||||
maxRetries: DEFAULT_MAX_RETRIES,
|
||||
retryableCodes: DEFAULT_RETRYABLE_CODES,
|
||||
...resolveBackoff(undefined, `${path}.backoff`),
|
||||
})
|
||||
}
|
||||
|
||||
switch (config.mode) {
|
||||
case 'normal': {
|
||||
validateKeys(config, NORMAL_POLICY_KEYS, path)
|
||||
const maxRetries = config.maxRetries ?? DEFAULT_MAX_RETRIES
|
||||
const retryableCodes = config.retryableCodes ?? [...DEFAULT_RETRYABLE_CODES]
|
||||
if (!Number.isSafeInteger(maxRetries) || maxRetries < 0) {
|
||||
throw new Error(`${path}.maxRetries must be a non-negative safe integer`)
|
||||
}
|
||||
if (retryableCodes.length === 0) {
|
||||
throw new Error(`${path}.retryableCodes must not be empty`)
|
||||
}
|
||||
if (retryableCodes.some(code => code.length === 0)) {
|
||||
throw new Error(`${path}.retryableCodes must contain only non-empty strings`)
|
||||
}
|
||||
if (new Set(retryableCodes).size !== retryableCodes.length) {
|
||||
throw new Error(`${path}.retryableCodes must not contain duplicates`)
|
||||
}
|
||||
return Object.freeze({
|
||||
mode: 'normal',
|
||||
maxRetries,
|
||||
retryableCodes: Object.freeze([...retryableCodes]),
|
||||
...resolveBackoff(config.backoff, `${path}.backoff`),
|
||||
})
|
||||
}
|
||||
case 'always':
|
||||
validateKeys(config, ALWAYS_POLICY_KEYS, path)
|
||||
return Object.freeze({
|
||||
mode: 'always',
|
||||
...resolveBackoff(config.backoff, `${path}.backoff`),
|
||||
})
|
||||
default:
|
||||
throw new Error(`${path}.mode must be "normal" or "always"`)
|
||||
}
|
||||
}
|
||||
84
packages/llm/llm/tests/retry-policy.spec.ts
Normal file
84
packages/llm/llm/tests/retry-policy.spec.ts
Normal file
@@ -0,0 +1,84 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
resolveRetryPolicy,
|
||||
RetryPolicySchema,
|
||||
} from '@deepseek-ai/dsh-llm'
|
||||
import type { RetryPolicyConfig } from '@deepseek-ai/dsh-llm'
|
||||
import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout'
|
||||
|
||||
describe('provider retry policy', () => {
|
||||
it('resolves immutable normal defaults', () => {
|
||||
const policy = resolveRetryPolicy(undefined, 'provider.retryPolicy')
|
||||
|
||||
expect(policy).toEqual({
|
||||
mode: 'normal',
|
||||
maxRetries: 2,
|
||||
retryableCodes: ['RATE_LIMIT', 'SERVER', 'TIMEOUT', 'TRANSPORT'],
|
||||
initialDelayMs: 500,
|
||||
maxDelayMs: 10_000,
|
||||
jitterRatio: 0.1,
|
||||
})
|
||||
expect(Object.isFrozen(policy)).toBe(true)
|
||||
if (policy.mode !== 'normal') throw new Error('expected normal policy')
|
||||
expect(Object.isFrozen(policy.retryableCodes)).toBe(true)
|
||||
})
|
||||
|
||||
it('resolves and detaches a configured normal policy', () => {
|
||||
const retryableCodes = ['BUSY']
|
||||
const config: RetryPolicyConfig = {
|
||||
mode: 'normal',
|
||||
maxRetries: 4,
|
||||
retryableCodes,
|
||||
backoff: {
|
||||
initialDelayMs: 25,
|
||||
maxDelayMs: 100,
|
||||
jitterRatio: 0,
|
||||
},
|
||||
}
|
||||
|
||||
const policy = resolveRetryPolicy(config, 'provider.retryPolicy')
|
||||
retryableCodes.push('LATE')
|
||||
|
||||
expect(policy).toEqual({
|
||||
mode: 'normal',
|
||||
maxRetries: 4,
|
||||
retryableCodes: ['BUSY'],
|
||||
initialDelayMs: 25,
|
||||
maxDelayMs: 100,
|
||||
jitterRatio: 0,
|
||||
})
|
||||
})
|
||||
|
||||
it('resolves always mode with default backoff', () => {
|
||||
expect(resolveRetryPolicy({ mode: 'always' }, 'provider.retryPolicy')).toEqual({
|
||||
mode: 'always',
|
||||
initialDelayMs: 500,
|
||||
maxDelayMs: 10_000,
|
||||
jitterRatio: 0.1,
|
||||
})
|
||||
expect(RetryPolicySchema).toBeDefined()
|
||||
})
|
||||
|
||||
it.each([
|
||||
[{ mode: 'normal', maxRetries: -1 }, /maxRetries/],
|
||||
[{ mode: 'normal', maxRetries: 1.5 }, /maxRetries/],
|
||||
[{ mode: 'normal', maxRetries: Number.MAX_SAFE_INTEGER + 1 }, /maxRetries/],
|
||||
[{ mode: 'always', backoff: { initialDelayMs: 0 } }, /initialDelayMs/],
|
||||
[{ mode: 'normal', backoff: { maxDelayMs: Number.POSITIVE_INFINITY } }, /maxDelayMs/],
|
||||
[{ mode: 'normal', backoff: { initialDelayMs: MAX_TIMER_DELAY_MS + 1 } }, /initialDelayMs/],
|
||||
[{ mode: 'always', backoff: { maxDelayMs: MAX_TIMER_DELAY_MS + 1 } }, /maxDelayMs/],
|
||||
[{ mode: 'normal', backoff: { initialDelayMs: 20, maxDelayMs: 10 } }, /less than or equal/],
|
||||
[{ mode: 'always', backoff: { jitterRatio: 1.1 } }, /jitterRatio/],
|
||||
[{ mode: 'normal', retryableCodes: [] }, /must not be empty/],
|
||||
[{ mode: 'normal', retryableCodes: ['SERVER', 'SERVER'] }, /duplicates/],
|
||||
[{ mode: 'normal', retryableCodes: [''] }, /non-empty strings/],
|
||||
[{ mode: 'normal', maxRetires: 1 }, /unknown key "maxRetires"/],
|
||||
[{ mode: 'always', maxRetries: 1 }, /unknown key "maxRetries"/],
|
||||
[{ mode: 'always', backoff: { initialDelay: 1 } }, /unknown key "initialDelay"/],
|
||||
[{ mode: 'sometimes' }, /mode must be "normal" or "always"/],
|
||||
] as const)('rejects invalid policy %#', (config, message) => {
|
||||
expect(() => {
|
||||
resolveRetryPolicy(config as unknown as RetryPolicyConfig, 'provider.retryPolicy')
|
||||
}).toThrow(message)
|
||||
})
|
||||
})
|
||||
@@ -11,6 +11,7 @@ import LlmService, {
|
||||
LlmError,
|
||||
llmFailureOf,
|
||||
ProviderRequestId,
|
||||
resolveRetryPolicy,
|
||||
StreamChunk,
|
||||
} from '@deepseek-ai/dsh-llm'
|
||||
import type { LlmModelContext, LlmModelInfo, LlmProviderInfo } from '@deepseek-ai/dsh-llm'
|
||||
@@ -159,6 +160,27 @@ describe('LlmService', () => {
|
||||
expect(chunks).toEqual(SCRIPT)
|
||||
})
|
||||
|
||||
it('captures provider-owned retry policy at registration and defaults omission', async () => {
|
||||
const configured = resolveRetryPolicy({ mode: 'always' }, 'test retryPolicy')
|
||||
const adapter = new class extends ScriptedAdapter {
|
||||
override providerRetryPolicy(provider: string) {
|
||||
return provider === 'configured' ? configured : undefined
|
||||
}
|
||||
}(SCRIPT)
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
ctx.llm.registerAdapter(['configured', 'defaulted'], adapter)
|
||||
|
||||
expect(ctx.llm.providerRetryPolicy('configured')).toBe(configured)
|
||||
expect(ctx.llm.providerRetryPolicy('defaulted')).toMatchObject({
|
||||
mode: 'normal',
|
||||
maxRetries: 2,
|
||||
})
|
||||
expect(() => ctx.llm.providerRetryPolicy('missing')).toThrow(
|
||||
expect.objectContaining({ code: 'NO_ADAPTER' }),
|
||||
)
|
||||
})
|
||||
|
||||
it('throws NO_ADAPTER for unregistered providers', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
|
||||
@@ -19,6 +19,9 @@
|
||||
},
|
||||
{
|
||||
"path": "../../support/invariants"
|
||||
},
|
||||
{
|
||||
"path": "../../util/timeout"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user