feat(llm): add per-provider retry policies

This commit is contained in:
Turtle
2026-07-25 10:18:16 +08:00
parent fe84b446b0
commit b58e33268a
60 changed files with 1606 additions and 334 deletions

View File

@@ -2,6 +2,8 @@
Status: implemented
The [per-provider request retry policy](../feature/2026-07-24-provider-retry-policies.md) extends this foundation with exact-provider configuration and an explicit unbounded mode. This note continues to own structured failure facts, the closed-step recovery boundary, normal mode's transient defaults, visible single attempts, and durable retry status.
## Problem
`dsh-llm` can report provider failures either by throwing during adapter dispatch or iteration or by ending with `finish { kind: 'error' | 'aborted' }`. The final adapter boundary tags thrown failures so `dsh-agent-loop` can distinguish them from middleware and result-processing defects, and the loop normalizes both delivery forms into `agent/request-error` after closing the failed step. The default decision is `fail`; `dsh-compact-basic` is the only shipped recovery listener, and it retries a canonical context-window overflow only after compaction proves that the durable surface shrank.
@@ -14,7 +16,7 @@ The prior boundary left three narrower gaps.
- Retry ownership differs by adapter. The hand-written DeepSeek adapter makes one attempt, while pi-ai profiles can enable opaque library retries. Combining hidden transport retries with an `agent/request-error` listener would multiply attempts and omit intermediate failures from the session log.
- A recovered failure has no durable status fact. The failed step and chunks remain reconstructable, but an observer cannot tell whether the agent is deliberately backing off, for how long, or why. A long silent wait looks like a stalled loop.
The goal is bounded recovery from transient failures of the same explicit provider/model request. Provider or model failover, response splicing, and semantic output repair are different problems and have no current consumer.
The default policy provides bounded recovery from transient failures of the same explicit provider/model request. Provider or model failover, response splicing, and semantic output repair are different problems and have no current consumer.
## Decision
@@ -48,31 +50,19 @@ The initial shared transient-code set is intentionally small: the adapters' exis
`@deepseek-ai/dsh-llm-retry` is a function plugin that listens to `agent/request-error`. It introduces no service or new loop branch; the agent-loop package changes only the data carried through its existing failed-step recovery control flow.
The `agent/request-error` seam carries the current `LlmFailure` and an immutable list of prior failures that led to another request attempt in this consecutive recovery sequence. `dsh-llm-retry` counts only prior failures whose codes are in its configured transient set, while `dsh-compact-basic` counts only prior context-overflow failures. A successful model request clears the history. Alternating transient and context-overflow failures therefore consume their owning policy budgets independently; the maximum request count is one plus the sum of the finite budgets of the loaded recovery policies.
The `agent/request-error` seam carries the current `LlmFailure` and an immutable list of prior failures that led to another request attempt in this consecutive recovery sequence. Normal `dsh-llm-retry` policy counts retry records scheduled by the same exact-provider policy, while `dsh-compact-basic` counts prior context-overflow failures. A successful model request clears the history. Alternating transient and context-overflow failures therefore consume their owning finite budgets independently.
The plugin resolves and validates this deployment configuration at load:
```ts ignore-check
interface Config {
maxTransientRetries?: number
initialDelayMs?: number
maxDelayMs?: number
jitterRatio?: number
retryableCodes?: string[]
}
```
The defaults are two transient retries, a 500 millisecond initial delay, a 10 second delay cap, 10 percent jitter, and the four transient codes above. The count and delay bounds match the conservative edge of the inspected implementations: [OpenCode uses two request retries with 500 ms/10 s bounds](https://github.com/anomalyco/opencode/blob/9976269ab1accfc9f9dc98a4a688c516934de422/%70ackages/llm/src/route/executor.ts#L36-L39), [Pi separates three agent-level retries from provider retries and defaults provider retries to zero](https://github.com/earendil-works/pi/blob/3da591ab74ab9ab407e72ed882600b2c851fae21/%70ackages/coding-agent/docs/settings.md#L139-L147), and [Codex uses finite request/stream budgets plus a five-minute idle timeout](https://github.com/openai/codex/blob/0fb559f0f6e231a88ac02ea002d3ecd248e2b515/codex-rs/model-provider-info/src/lib.rs#L25-L33). Ten percent follows [Codex's bounded jitter](https://github.com/openai/codex/blob/0fb559f0f6e231a88ac02ea002d3ecd248e2b515/codex-rs/codex-client/src/retry.rs#L40-L47). Two retries mean at most three provider requests when no other recovery policy applies. `maxTransientRetries` is a non-negative integer, delays are positive finite numbers with `initialDelayMs <= maxDelayMs`, `jitterRatio` is in `[0, 1]`, and codes are non-empty and unique. These are Cordis config fields rather than hidden constants so deployments can choose different cost and latency budgets.
The [provider-policy decision](../feature/2026-07-24-provider-retry-policies.md) owns the current configuration shape. Provider adapters register their nested `retryPolicy`; omission uses normal defaults: two transient retries, a 500 millisecond initial delay, a 10 second delay cap, 10 percent jitter, and the four transient codes above. The count and delay bounds match the conservative edge of the inspected implementations: [OpenCode uses two request retries with 500 ms/10 s bounds](https://github.com/anomalyco/opencode/blob/9976269ab1accfc9f9dc98a4a688c516934de422/%70ackages/llm/src/route/executor.ts#L36-L39), [Pi separates three agent-level retries from provider retries and defaults provider retries to zero](https://github.com/earendil-works/pi/blob/3da591ab74ab9ab407e72ed882600b2c851fae21/%70ackages/coding-agent/docs/settings.md#L139-L147), and [Codex uses finite request/stream budgets plus a five-minute idle timeout](https://github.com/openai/codex/blob/0fb559f0f6e231a88ac02ea002d3ecd248e2b515/codex-rs/model-provider-info/src/lib.rs#L25-L33). Ten percent follows [Codex's bounded jitter](https://github.com/openai/codex/blob/0fb559f0f6e231a88ac02ea002d3ecd248e2b515/codex-rs/codex-client/src/retry.rs#L40-L47).
For an eligible failure with budget remaining, the one-based transient retry count uses bounded exponential backoff. A valid `providerRetryAfterMs` replaces exponential backoff only when it does not exceed `maxDelayMs`; a longer provider delay causes delegation instead of an earlier retry that violates the provider instruction. Local backoff multiplies by an injected random factor in `[1 - jitterRatio, 1 + jitterRatio]` and clamps the final value to `maxDelayMs`; provider delay is not jittered.
The plugin owns a lifetime `AbortController` and tracks every active backoff callback. Each wait fuses the waterfall's turn signal with that lifetime signal. Effect cleanup first unregisters the listener, then aborts and awaits the active callbacks; a captured callback whose lifetime signal aborts returns `fail` and can neither retry nor enter the rest of its captured waterfall after disposal. This makes HMR disposal quiescent even though Cordis has already captured the listener.
Before sleeping, `dsh-llm-retry` appends one non-surface `llm/retry` session event containing the turn, failed step, one-based transient retry number, configured maximum, scheduled delay, and `LlmFailure`. The plugin owns the `SessionEventMap` augmentation; `dsh-session` remains generic persistence and does not absorb the optional policy's vocabulary. The event says what was scheduled, not that the next request completed; cancellation during the delay is subsequently visible on `turn/end`. The event ships only with a production renderer and replay/snapshot coverage, because its purpose is operational state rather than trace collection.
Before sleeping, `dsh-llm-retry` appends one non-surface `llm/retry` session event containing the turn, failed step, provider, policy mode, provider-policy retry number, mode-specific finite maximum when present, scheduled delay, and `LlmFailure`. The plugin owns the `SessionEventMap` augmentation; `dsh-session` remains generic persistence and does not absorb the optional policy's vocabulary. The event says what was scheduled, not that the next request completed; cancellation during the delay is subsequently visible on `turn/end`. The event ships only with a production renderer and replay/snapshot coverage, because its purpose is operational state rather than trace collection.
The listener calls `next()` for a non-transient code, an exhausted policy budget, or an over-cap provider delay. This preserves composition with context-overflow recovery and later policy plugins. It returns `{ action: 'retry' }` only after the delay completes under both signals; turn cancellation and plugin disposal return `fail`, after which the loop's cancellation/disposal checks remain authoritative.
The agent-spine demo bundle loads the plugin so the shared stdio/TUI, one-shot CLI, and ACP example compositions use the same bounded policy. Library consumers retain explicit plugin composition: omitting the plugin leaves `agent/request-error` at its current fail default.
The agent-spine demo bundle loads the plugin so the shared stdio/TUI, one-shot CLI, and ACP example compositions use the same provider-routed policy. Library consumers retain explicit plugin composition: omitting the plugin leaves `agent/request-error` at its current fail default.
### Make one layer own visible attempts
@@ -99,7 +89,7 @@ If recovery is exhausted, the final failure is stored once on `turn/end.reason`
- Automatic provider or model failover. Requests already select one explicit provider and model, and the provider registry deliberately has one adapter owner per provider.
- Retrying or continuing after a successful terminal finish, or splicing chunks from two attempts into one assistant message.
- Repairing malformed tool arguments, refusals, content filters, or other semantic model output.
- Unbounded retries, unattended retry-until-cancelled behavior, circuit breakers, shared provider health, or cross-agent retry budgets.
- Circuit breakers, shared provider health, or cross-agent retry budgets.
- Changing `llm/stream` into a response lifecycle or adding convenience generation APIs without a production consumer.
## Alternatives considered
@@ -108,7 +98,7 @@ If recovery is exhausted, the final failure is stored once on `turn/end.reason`
- **Add response start, interrupted, discarded, failed, and committed events to `dsh-llm`** — rejected because the agent log already separates raw chunks, successful messages, and numbered attempts. A second state machine would duplicate ownership without enabling the bounded same-route retry.
- **Add logical routes, capability matrices, and failover selection** — rejected because current requests already name provider and model explicitly, one adapter owns each provider, and no current consumer requires automatic fallback or can prove semantic compatibility.
- **Put `retryable` or `failover` on `LlmFailure`** — rejected because adapters report facts while deployment policy decides action. The same 429 may be retried in an interactive bundle and rejected in a cost-capped batch.
- **Retry forever while the caller remains active** — rejected because it gives one request unbounded cost and latency. Visible status makes bounded waiting understandable; it does not make an unlimited budget safe.
- **Retry forever while the caller remains active** — the [per-provider policy](../feature/2026-07-24-provider-retry-policies.md) supersedes this rejection for explicit `always` entries while retaining bounded normal mode as the default.
- **Log retry status only through the process logger** — rejected because process logs do not reconstruct session behavior and cannot drive replayed UI state.
- **Keep only flat codes** — rejected because retry delay and provider request id are structured provider facts, and HTTP status is necessary for diagnosis when different wire failures share one stable code.
@@ -119,7 +109,7 @@ If recovery is exhausted, the final failure is stored once on `turn/end.reason`
- DeepSeek and pi-ai adapter tests cover representative 400, 401/403, 429, 5xx, connection, malformed/truncated stream, timeout, abort, retry-after seconds/date, request-id, and unknown-SDK-error paths without recovery policy parsing message text.
- Pi-ai pins the SDK option to zero retries and performs one observed wire attempt for a retryable provider response; separate tests make removing either boundary fail.
- `agent/request-error` carries current failure facts plus immutable prior-retried failure facts; a success clears that history, and alternating transient/context-overflow integration tests prove the two policies consume only their own finite budgets.
- `dsh-llm-retry` validates every config field at Loader startup, delegates all ineligible paths with `next()`, and makes at most `maxTransientRetries + 1` provider requests when no other policy applies.
- Each provider adapter validates its nested retry policy at Loader startup, and `ctx.llm` captures it with the route; normal mode delegates ineligible paths and makes at most `maxRetries + 1` provider requests when no other policy applies.
- HMR-during-backoff tests prove disposal unregisters the listener, aborts and awaits its captured callbacks, emits no retry decision after disposal, and leaves no timer or promise alive.
- Pure unit tests cover transient-code selection, exponential backoff and jitter bounds, valid and over-cap `Retry-After`, exhausted budgets, deterministic timer/random seams, and abort during backoff.
- Real agent-loop tests cover failure before chunks, partial chunks then failure, thrown and in-band failures, retry to success in a new step, exhaustion to structured `turn/end.reason`, and composition with `dsh-compact-basic` context-overflow recovery.
@@ -130,12 +120,12 @@ If recovery is exhausted, the final failure is stored once on `turn/end.reason`
## Consequences
- Every transient recovery attempt is visible as a closed step plus `llm/retry`, and the bounded policy prevents hidden SDK retries from multiplying cost. A retry can still duplicate provider billing even when no chunk arrived; the finite attempt budget limits but cannot remove that risk.
- Every retry attempt is visible as a closed step plus `llm/retry`, and adapter-level single-attempt behavior prevents hidden SDK retries from multiplying policy decisions. A retry can still duplicate provider billing even when no chunk arrived; normal mode limits that risk, while explicit always mode accepts it until cancellation or success.
- Provider SDKs may hide status or retry headers. Those adapters retain the stable facts they expose and otherwise use a coarse code rather than letting recovery policy parse fragile text.
- Durable retry events expand the session protocol and UI state machine. Shipping the event and its consumer together prevents an unused telemetry vocabulary, but later schema changes still require persistence and replay work.
- Clearing a failed step's live chunks can visibly retract output. That is preferable to presenting discarded text or partial tool JSON as committed history, and snapshots pin the transition.
- Adapter-local idle enforcement stops stalled transports without counting consumer think time. Contract tests at each transport boundary guard against SDK drift.
- Multiple recovery plugins add their finite budgets. Their classifiers remain disjoint here; an overlapping classifier would be registration-order policy and must be documented and tested by the plugins that introduce it.
- Multiple normal recovery plugins add their finite budgets. Always mode delegates first and then supplies an unbounded fallback; overlapping classifiers remain registration-order policy and must be documented and tested by the plugins that introduce them.
## Related

View File

@@ -0,0 +1,6 @@
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write
2026-07-24-provider-retry-policies.md: 0ff304715aa5e1bfaf296e3f53e1da28d8b4042d
2026-07-24-provider-retry-policies.zh.md: 3b1d8f12a5fb37424e8b964f7bbb2be278263056

View File

@@ -0,0 +1,65 @@
# Agent Note: Per-provider request retry policies
Status: implemented
English | [中文](2026-07-24-provider-retry-policies.zh.md)
## Problem
One process may route model requests to providers with different reliability and cost constraints. A single transient classifier and finite retry budget cannot express a deployment that wants bounded recovery for most providers but requires one provider to keep retrying every model-request failure until the request succeeds or the caller cancels it.
Provider policy must follow the request that actually failed, including a route selected by `agent/request`, rather than the agent's initial options. Unbounded policy also cannot store JavaScript `Infinity` in the durable session event, and neither provider error text nor discarded partial output may enter the next model request.
## Decision
Each concrete adapter accepts an optional `retryPolicy` inside its provider configuration. The adapter validates and resolves the policy, and `ctx.llm` captures it when that exact provider route registers. `@deepseek-ai/dsh-llm-retry` reads the registered policy for the provider whose step failed. A provider without `retryPolicy` uses the normal defaults.
```yaml
providers:
- provider: deepseek
retryPolicy:
mode: normal
maxRetries: 2
retryableCodes: [RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT]
backoff:
initialDelayMs: 500
maxDelayMs: 10000
jitterRatio: 0.1
- provider: internal
retryPolicy:
mode: always
backoff:
initialDelayMs: 1000
maxDelayMs: 30000
jitterRatio: 0.2
```
The listener selects the policy from the durable `request/header` in force when the failed step closed, excluding later recovery mutations. Normal mode retains the bounded transient behavior: it retries configured codes up to `maxRetries`, counts retries scheduled by the same provider policy in the current consecutive failure sequence, and otherwise delegates.
Always mode asks downstream recovery first so a specialized policy such as context-overflow compaction can make progress. A downstream retry wins. A downstream failure decision or thrown recovery error falls back to an unbounded retry of the same provider request; the thrown error is logged. Success, turn cancellation, and plugin disposal are the only termination paths.
Both modes use exponential local delays from `initialDelayMs` to `maxDelayMs`. `jitterRatio` multiplies each target by a uniform sample in `[1 - jitterRatio, 1 + jitterRatio]`, then applies the cap. A positive provider `Retry-After` within the cap remains exact and unjittered. An over-cap provider delay makes normal mode delegate; always mode retains its guarantee by using the configured local backoff.
Each scheduled retry appends a non-surface `llm/retry` event with the failed provider, policy mode, provider-policy retry number, delay, and failure facts. Normal events carry finite `maxRetries`; always events omit it, and UIs render the limit as `∞`. The event and failed `assistant/chunk` records do not contribute surface messages, so the next request contains the same derived context as the failed request unless another recovery policy deliberately changes the surface.
## Alternatives considered
**One global `always` switch** — rejected because it cannot isolate the unbounded cost and latency risk to the provider that needs it and can silently apply after runtime rerouting.
**A separate exact-provider list on `dsh-llm-retry`** — rejected because it duplicates provider route names outside their owning adapter configuration and lets provider registration drift from recovery policy.
**A very large finite retry count** — rejected because it eventually violates the requested keep-retrying contract and serializes an arbitrary operational limit as if it were meaningful.
**Provider-SDK retries** — rejected because hidden attempts multiply agent-level budgets, cannot use the closed-step durability boundary, and may splice or discard streamed output without a reconstructable retry record.
**Put the error into model context** — rejected because a transport or provider diagnostic is operational state, not conversation content. It can expose sensitive provider details and changes the retried request instead of repeating the failed request.
## Verification
Adapter tests validate nested policies at provider load and prove registration captures configured and default policies. Unit and real-Loader composition tests select policies from the failed request's provider, exercise always mode beyond the normal budget, pin jitter and delay caps, prove downstream recovery ordering, prove cancellation interrupts stalled downstream recovery, and prove cancellation and disposal stop active backoff waits. Request-level coverage compares the complete messages of failed and retried attempts and rejects both provider error text and discarded partial output. JSONL and SQLite tests round-trip an always event without `Infinity`; invariant tests bind its provider to the request header and its retry number to the active provider policy; ACP and TUI tests render finite and infinite limits.
## Consequences
Normal mode remains a finite default, while an explicit always policy can spend unbounded requests and time on permanent authentication, quota, invalid-request, protocol, or context failures. Operators must pair always mode with a cancellable caller and provider-specific cost controls. Retry state stays observable and durable without becoming model-visible, and exact-provider selection keeps one provider's exceptional policy from changing another provider's recovery behavior.
This decision extends the closed-step recovery, single visible adapter attempt, structured failure, and durable status design in [bounded recovery for transient LLM request failures](../architecture/2026-06-21-bounded-llm-request-recovery.md).

View File

@@ -0,0 +1,65 @@
# Agent Note: 逐提供方请求重试策略
Status: implemented
[English](2026-07-24-provider-retry-policies.md) | 中文
## 问题
同一进程可能把模型请求路由到可靠性和成本约束各不相同的提供方。单一的瞬态错误分类器与有限重试预算无法表达这种部署需求:大多数提供方只需有界恢复,但其中一个提供方必须持续重试每次模型请求失败,直到请求成功或调用方取消。
提供方策略必须跟随实际失败的请求,包括 `agent/request` 选择的路由,而不能跟随 agent智能体的初始选项。无界策略也不能把 JavaScript `Infinity` 存入持久会话事件;提供方错误文本与丢弃的部分输出都不得进入下一次模型请求。
## 决策
每个具体适配器都在其提供方配置中接受可选的 `retryPolicy`。适配器负责校验并解析策略,`ctx.llm` 则在该精确提供方路由注册时捕获策略。`@deepseek-ai/dsh-llm-retry` 读取失败步骤对应提供方的已注册策略。未配置 `retryPolicy` 的提供方使用 normal 默认值。
```yaml
providers:
- provider: deepseek
retryPolicy:
mode: normal
maxRetries: 2
retryableCodes: [RATE_LIMIT, SERVER, TIMEOUT, TRANSPORT]
backoff:
initialDelayMs: 500
maxDelayMs: 10000
jitterRatio: 0.1
- provider: internal
retryPolicy:
mode: always
backoff:
initialDelayMs: 1000
maxDelayMs: 30000
jitterRatio: 0.2
```
监听器根据失败步骤关闭时生效的持久 `request/header` 选择策略后续恢复产生的改动不参与选择。normal 模式保留有界瞬态错误处理行为:它重试配置的错误代码,次数不超过 `maxRetries`;在当前连续失败序列中,同一提供方策略安排的重试都计入次数;其他情况委托后续处理。
always 模式先请求下游恢复使上下文溢出压缩compaction之类的专用策略有机会取得进展。下游若决定重试则以该决定为准。下游若决定失败或恢复过程抛出错误则回退为无界重试同一提供方请求抛出的错误会写入日志。成功、轮次取消和插件 dispose资源释放是仅有的终止路径。
两种模式的本地延迟都按指数增长,从 `initialDelayMs` 增至 `maxDelayMs``jitterRatio``[1 - jitterRatio, 1 + jitterRatio]` 区间内的均匀随机样本乘以每次目标值,再应用上限。提供方给出的正数 `Retry-After` 若未超过上限则保持精确且不加抖动。若提供方延迟超过上限normal 模式会委托后续处理always 模式则改用配置的本地退避,以维持无限重试保证。
每次安排重试都会追加一条不进入表层的 `llm/retry` 事件其中包含失败的提供方、策略模式、提供方策略内的重试编号、延迟和失败事实。normal 事件包含有限的 `maxRetries`always 事件省略该字段UI 将上限渲染为 `∞`。该事件与失败的 `assistant/chunk` 记录都不会生成表层消息,因此除非其他恢复策略有意改变表层,否则下一次请求包含的派生上下文与失败请求相同。
## 曾考虑的替代方案
**单一全局 `always` 开关**:不予采纳,因为它无法把无界成本与延迟风险限制在确有需要的提供方,还可能在运行时重新路由后悄然生效。
**在 `dsh-llm-retry` 上维护单独的精确提供方列表**:不予采纳,因为它会在所属适配器配置之外重复提供方路由名称,并让提供方注册与恢复策略发生偏差。
**设置很大的有限重试次数**:不予采纳,因为它最终仍会违反持续重试的契约,并把任意选取的运维上限序列化成看似有意义的数值。
**使用提供方 SDK 重试**:不予采纳,因为隐藏尝试会叠加 agent 层预算,无法利用已关闭 step 的持久性边界,还可能在没有可重建重试记录的情况下拼接或丢弃流式输出。
**把错误放入模型上下文**:不予采纳,因为传输或提供方诊断信息属于运维状态,而非对话内容。它可能暴露敏感的提供方细节,并会改变重试请求,无法重复原本失败的请求。
## 验证
适配器测试会在提供方加载时校验嵌套策略,并证明注册流程会捕获已配置策略和默认策略。单元测试与真实 Loader 组合测试根据失败请求的提供方选择策略、验证 always 模式可越过 normal 预算、固定抖动和延迟上限、证明下游恢复顺序、证明取消会中断停滞的下游恢复,并证明取消与 dispose 会停止正在进行的退避等待。请求级覆盖会比较失败尝试与重试尝试的完整消息并排除提供方错误文本和丢弃的部分输出。JSONL 与 SQLite 测试会往返读写不含 `Infinity` 的 always 事件不变式测试会将事件中的提供方绑定到请求头并将重试编号绑定到活跃的提供方策略ACP 与 TUI 测试会分别渲染有限和无限上限。
## 后果
normal 模式仍是有限的默认策略;显式的 always 策略可能在永久性的身份验证、配额、无效请求、协议或上下文错误上耗费无限次请求和无限时间。运维方必须为 always 模式配备可取消的调用方和针对提供方的成本控制。重试状态保持可观察且持久,但不会对模型可见;精确提供方选择也能避免某个提供方的例外策略改变其他提供方的恢复行为。
本决策扩展了[瞬态 LLM大语言模型请求失败的有界恢复](../architecture/2026-06-21-bounded-llm-request-recovery.md)中确定的已关闭 step 恢复、单次可见适配器尝试、结构化失败与持久状态设计。