feat: implement bounded LLM request recovery

packages/examples/agent-spine-demo/README.md

@@ -17,6 +17,7 @@ Read this package for the whole plugin tree and its composition order.
 @deepseek-ai/dsh-skill            skill provider registry
 @deepseek-ai/dsh-skill-local      local filesystem skill provider
 @deepseek-ai/dsh-agent            agent registry + initiator scope + agent/* events
+@deepseek-ai/dsh-llm-retry        bounded transient request retry policy
 @deepseek-ai/dsh-tasks            generic background-task registry
 @deepseek-ai/dsh-invariants       dev-mode event-contract assertions
 @deepseek-ai/dsh-tool-bash        the model-facing bash schema
@@ -42,19 +43,21 @@ This is the [interface/implementation/consumer seam](../../../.agents/notes/impl
 
 ```ts
 import type { Config } from '@deepseek-ai/dsh-agent-spine-demo'
-// { agents?, maxParallelToolCalls?, persona?, toolOrder?, tools?, dshHome?, skills?, workspaceContext, toolBash?, toolTasks? }
+// { agents?, maxParallelToolCalls?, persona?, toolOrder?, tools?, dshHome?, skills?, workspaceContext, toolBash?, toolTasks?, llmRetry? }
 // workspaceContext requires { maxBytes } or false; the other owner schemas supply defaults.
 ```
 
-The bundle FORWARDS each field to the child that owns it: `agents` and `maxParallelToolCalls` to `agent-loop` (`agents` defaults to `[]`; the cap defaults there), so each app supplies its own pre-created agents — a stdio app pre-creates `main`, while the ACP app creates agents on demand at `session/new`; `persona` and `toolOrder` to `dsh-system-prompt`; `tools` to the tool registry for its presentation mode; `skills.registry`, `skills.local`, and `skills.tool` to the skill registry, local provider, and model-facing consumer; the required `workspaceContext` choice to `dsh-workspace-context` (`{ maxBytes }` enables loading and `false` disables it); and `toolBash`/`toolTasks` to the two model-facing tool plugins the bundle owns. Set `skills.enabled: false` to omit both the local provider and model-facing skill tool, and set `toolTasks: false` to retain the task service for foreground producers without exposing `task_output` / `task_list` / `task_kill`. It resolves `dshHome` once through [`@deepseek-ai/dsh-home`](../../util/home/README.md) and forwards that absolute value to tool-bash's managed environment and enabled local skill discovery. An absent top-level `dshHome` adopts `skills.local.dshHome`; supplying both with different resolved paths fails loudly. `toolBash.enableRunInBackground` controls only the bash producer; independently loaded producers keep their own config. Workspace instructions register before the skill catalog so their session-prefix message renders first. App packages use `pickSpineConfig()` to copy only these bundle-owned fields.
+The bundle FORWARDS each field to the child that owns it: `agents` and `maxParallelToolCalls` to `agent-loop` (`agents` defaults to `[]`; the cap defaults there), so each app supplies its own pre-created agents — a stdio app pre-creates `main`, while the ACP app creates agents on demand at `session/new`; `llmRetry` to the bounded retry policy; `persona` and `toolOrder` to `dsh-system-prompt`; `tools` to the tool registry for its presentation mode; `skills.registry`, `skills.local`, and `skills.tool` to the skill registry, local provider, and model-facing consumer; the required `workspaceContext` choice to `dsh-workspace-context` (`{ maxBytes }` enables loading and `false` disables it); and `toolBash`/`toolTasks` to the two model-facing tool plugins the bundle owns. Set `skills.enabled: false` to omit both the local provider and model-facing skill tool, and set `toolTasks: false` to retain the task service for foreground producers without exposing `task_output` / `task_list` / `task_kill`. It resolves `dshHome` once through [`@deepseek-ai/dsh-home`](../../util/home/README.md) and forwards that absolute value to tool-bash's managed environment and enabled local skill discovery. An absent top-level `dshHome` adopts `skills.local.dshHome`; supplying both with different resolved paths fails loudly. `toolBash.enableRunInBackground` controls only the bash producer; independently loaded producers keep their own config. Workspace instructions register before the skill catalog so their session-prefix message renders first. App packages use `pickSpineConfig()` to copy only these bundle-owned fields.
 
 ## Why a code bundle, not a shared YAML include
 
 A YAML include can deduplicate config but cannot own a bin or provide front-door defaults. App packages make stdout-safe ACP wiring the default, though a leaf can still add an unsafe logger. Bundle children register services in the root isolate-keyed store, so injected leaf siblings see them without load-order coupling.
 
+The bounded retry policy may repeat a transiently failed request in a new numbered step. Retry status and failed partial chunks stay outside model history, each provider attempt can still incur billing, front doors derive usage across every logged step, and the reconstructed request preserves the prior prefix for provider cache reuse.
+
 ## Model Experience
 
-Indirectly, through `dsh-system-prompt`, `dsh-tool-skill`, `dsh-tool-bash`, and `dsh-tools`, which this bundle mounts without adding model-bound wrapper content.
+Indirectly, through `dsh-system-prompt`, `dsh-tool-skill`, `dsh-tool-bash`, `dsh-tools`, and `dsh-llm-retry`, which this bundle mounts without adding model-bound wrapper content.
 
 #### KV Cache effect
 

packages/examples/agent-spine-demo/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@deepseek-ai/dsh-agent-spine-demo",
-  "description": "The default executor-less/UI-less agent spine as one Cordis bundle plugin (timer + llm + sessions + system-prompt + tools + skills + agents + tasks + invariants + tool-bash + workspace-context + tool-skill + tool-tasks + agent-loop)",
+  "description": "The default executor-less/UI-less agent spine as one Cordis bundle plugin (timer + llm + sessions + system-prompt + tools + skills + agents + bounded retry + tasks + invariants + tool-bash + workspace-context + tool-skill + tool-tasks + agent-loop)",
   "version": "0.0.1",
   "private": true,
   "type": "module",
@@ -28,6 +28,7 @@
     "@deepseek-ai/dsh-invariants": "^0.0.1",
     "@deepseek-ai/dsh-home": "^0.0.1",
     "@deepseek-ai/dsh-llm": "^0.0.1",
+    "@deepseek-ai/dsh-llm-retry": "^0.0.1",
     "@deepseek-ai/dsh-workspace-context": "^0.0.1",
     "@deepseek-ai/dsh-session": "^0.0.1",
     "@deepseek-ai/dsh-skill": "^0.0.1",
@@ -48,6 +49,7 @@
     "@deepseek-ai/dsh-invariants": "workspace:^",
     "@deepseek-ai/dsh-home": "workspace:^",
     "@deepseek-ai/dsh-llm": "workspace:^",
+    "@deepseek-ai/dsh-llm-retry": "workspace:^",
     "@deepseek-ai/dsh-workspace-context": "workspace:^",
     "@deepseek-ai/dsh-session": "workspace:^",
     "@deepseek-ai/dsh-skill": "workspace:^",

packages/examples/agent-spine-demo/src/index.ts

@@ -25,6 +25,7 @@ import * as workspaceContext from '@deepseek-ai/dsh-workspace-context'
 import * as toolSkill from '@deepseek-ai/dsh-tool-skill'
 import * as toolTasks from '@deepseek-ai/dsh-tool-tasks'
 import AgentLoop, { type Config as AgentLoopConfig } from '@deepseek-ai/dsh-agent-loop'
+import * as llmRetry from '@deepseek-ai/dsh-llm-retry'
 import { resolveDshHome } from '@deepseek-ai/dsh-home'
 
 export const name = 'agent-spine-demo'
@@ -77,6 +78,8 @@ export interface Config {
   toolBash?: toolBash.Config
   /** Generic background-task controls; set false to keep the task service without model-facing task tools. */
   toolTasks?: toolTasks.Config | false
+  /** Bounded transient model-request retry policy. */
+  llmRetry?: llmRetry.Config
 }
 
 /** The skill config schema exported for app packages that forward `skills`. */
@@ -93,6 +96,9 @@ export const ToolBashConfigSchema: z<toolBash.Config> = toolBash.Config
 /** The task-control-tool config schema exported for app packages that forward `toolTasks`. */
 export const ToolTasksConfigSchema: z<toolTasks.Config> = toolTasks.Config
 
+/** The bounded LLM retry schema exported for app packages that forward `llmRetry`. */
+export const LlmRetryConfigSchema: z<llmRetry.Config> = llmRetry.Config
+
 /** Intersect the owners' schemas so validation + defaulting stay identical. */
 export const Config = z.intersect([
   AgentLoop.Config,
@@ -104,7 +110,8 @@ export const Config = z.intersect([
     workspaceContext: z.union([z.const(false), workspaceContext.Config]).required(),
     toolBash: ToolBashConfigSchema,
     toolTasks: z.union([z.const(false), ToolTasksConfigSchema]),
-  }) as unknown as z<Pick<Config, 'tools' | 'dshHome' | 'skills' | 'workspaceContext' | 'toolBash' | 'toolTasks'>>,
+    llmRetry: LlmRetryConfigSchema,
+  }) as unknown as z<Pick<Config, 'tools' | 'dshHome' | 'skills' | 'workspaceContext' | 'toolBash' | 'toolTasks' | 'llmRetry'>>,
 ]) as unknown as z<Config>
 
 /**
@@ -123,6 +130,7 @@ export function pickSpineConfig(config: Omit<Config, 'agents'>): Omit<Config, 'a
     ...config.skills !== undefined ? { skills: config.skills } : {},
     ...config.toolBash !== undefined ? { toolBash: config.toolBash } : {},
     ...config.toolTasks !== undefined ? { toolTasks: config.toolTasks } : {},
+    ...config.llmRetry !== undefined ? { llmRetry: config.llmRetry } : {},
   }
 }
 
@@ -159,6 +167,7 @@ export function apply(ctx: Context, config: Config): void {
     ctx.plugin(SkillLocal, Object.assign({}, config.skills?.local, { dshHome }))
   }
   ctx.plugin(AgentRegistry)
+  ctx.plugin(llmRetry, config.llmRetry ?? {})
   ctx.plugin(TaskService)
   ctx.plugin(invariants)
   ctx.plugin(toolBash, Object.assign({}, config.toolBash, { dshHome }))

packages/examples/agent-spine-demo/tests/agent-core.spec.ts

@@ -10,7 +10,7 @@ import { agentEvents, type Agent } from '@deepseek-ai/dsh-agent'
 import { SessionId } from '@deepseek-ai/dsh-session'
 import LocalFileSystem from '@deepseek-ai/dsh-fs-local'
 import { MockAdapter, textResponse } from '../../../core/agent-loop/tests/mock-adapter.ts'
-import { CallId, type Message } from '@deepseek-ai/dsh-llm'
+import { CallId, LlmAdapter, LlmError, type GenerateOptions, type Message, type StreamChunk } from '@deepseek-ai/dsh-llm'
 import type { ToolExecution } from '@deepseek-ai/dsh-tools'
 
 declare module '@deepseek-ai/dsh-tasks' {
@@ -101,6 +101,16 @@ function messageText(message: Message | undefined): string {
   return message?.content.map(block => block.type === 'text' ? block.text : '').join('\n') ?? ''
 }
 
+class TransientOnceAdapter extends LlmAdapter {
+  requests = 0
+
+  async * stream(_options: GenerateOptions): AsyncIterable<StreamChunk> {
+    this.requests += 1
+    if (this.requests === 1) throw new LlmError('temporary outage', 'SERVER')
+    yield* textResponse('recovered by bundled policy')
+  }
+}
+
 describe('dsh-agent-spine-demo bundle', () => {
   it('brings up the full default spine', async () => {
     const ctx = await mount({ workspaceContext: false })
@@ -117,6 +127,37 @@ describe('dsh-agent-spine-demo bundle', () => {
     await ctx.fiber.dispose()
   })
 
+  it('loads and configures bounded request recovery for every bundled front door', async () => {
+    const adapter = new TransientOnceAdapter()
+    const ctx = await mount({
+      workspaceContext: false,
+      llmRetry: {
+        maxTransientRetries: 1,
+        initialDelayMs: 1,
+        maxDelayMs: 1,
+        jitterRatio: 0,
+      },
+    })
+    ctx.llm.registerAdapter(['mock'], adapter)
+    const handle = await ctx.agents.create({
+      sessionId: SessionId('bundled-retry-session'),
+      meta: { cwd: process.cwd() },
+      agentOptions: { provider: 'mock', model: 'mock' },
+    })
+
+    handle.agent.send([{ type: 'text', text: 'recover' }])
+    await waitForIdle(ctx, handle.agent)
+
+    expect(adapter.requests).toBe(2)
+    const retryEvents = handle.agent.session.events.filter(event => event.type === 'llm/retry')
+    expect(retryEvents).toHaveLength(1)
+    expect(retryEvents[0]?.data.retry).toBe(1)
+    expect(retryEvents[0]?.data.maxRetries).toBe(1)
+    expect(messageText(handle.agent.session.deriveMessages().at(-1))).toBe('recovered by bundled policy')
+    await handle.dispose()
+    await ctx.fiber.dispose()
+  })
+
   it('includes the skill registry, local provider, and skill tool without builtin skills', async () => {
     const ctx = await mount({ workspaceContext: false })
 
@@ -370,6 +411,7 @@ describe('dsh-agent-spine-demo bundle', () => {
       skills: { enabled: false },
       toolBash: { enableRunInBackground: false },
       toolTasks: false as const,
+      llmRetry: { maxTransientRetries: 1, jitterRatio: 0 },
     }
 
     expect(agentCore.pickSpineConfig(appConfig)).toEqual({
@@ -381,6 +423,7 @@ describe('dsh-agent-spine-demo bundle', () => {
       skills: appConfig.skills,
       toolBash: appConfig.toolBash,
       toolTasks: appConfig.toolTasks,
+      llmRetry: appConfig.llmRetry,
     })
     expect(agentCore.pickSpineConfig({ workspaceContext: false })).toEqual({ workspaceContext: false })
   })

packages/examples/agent-spine-demo/tsconfig.json

@@ -47,6 +47,9 @@
     {
       "path": "../../core/agent-loop"
     },
+    {
+      "path": "../../llm/llm-retry"
+    },
     {
       "path": "../../support/invariants"
     },