Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(llm): preserve serving retry policy

Browse Source
This commit is contained in:
Turtle
2026-07-25 13:30:34 +08:00
parent a623ed5183
commit 015ba14bae
27 changed files with 278 additions and 82 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
packages/core/agent-loop/README.md
View File

@@ -60,7 +60,7 @@ The driver owns one agent for its lifetime and runs inside `ctx.agents.withIniti
Every provider call that reaches a successful finish appends exactly one `assistant/message` completion anchor, including content-less calls and `max-tokens` finishes. A successful `agent/step-result` stores its transformed content; a rejected result records empty content before the original failure continues. The anchor retains exact chunk provenance (`[]` for a stream with no chunks) and usage when available, while empty content stays out of derived message history.
Plugin failure ends the current turn, not the loop. Only final adapter dispatch/iteration failures and terminal in-band error or aborted finishes enter `agent/request-error`; middleware, result processing, tools, and `agent/post-step` remain ordinary turn failures. Recovery receives the exact live error, immutable provider facts, and immutable prior failures after the failed step closes. A retry rebuilds from the durable log in a new numbered step, success clears the consecutive history, and exhaustion records the structured failure once on `turn/end`. AgentLoop privately owns one cancellation holder whose explicit signal spans prompt policy, assembly, every step, model and tool work, recovery, continuation, and terminal stop; it retires the holder immediately before publishing `turn/end`, while the driver may remain `running` through the durability flush. An effective `cancel()` emits the typed runtime-only `user | parent` cause before clearing pending work and cooperatively aborting the holder; notification failures cannot veto cancellation, work queued by a notification observer is cleared, work queued by a later abort observer belongs to the next turn, and idle cancellation emits nothing. Durable `turn/end` remains coarse `aborted`; undispatched model tool calls receive synthetic `tool/call` and `ABORTED_BEFORE_DISPATCH` result pairs. Disposal wins terminal classification, and work that ignores the signal must settle before quiescence. The [explicit-cancellation decision](../../../.agents/notes/implemented/architecture/2026-07-16-explicit-turn-cancellation.md) owns the lifecycle and race contract. Terminal continuation stops remain authoritative through turn close and durability flush.
Plugin failure ends the current turn, not the loop. Only final adapter dispatch/iteration failures and terminal in-band error or aborted finishes enter `agent/request-error`; middleware, result processing, tools, and `agent/post-step` remain ordinary turn failures. Recovery receives the exact live error, immutable provider facts, immutable prior failures, and the immutable retry policy of the adapter registration that served the request after the failed step closes; the policy is absent if no final adapter served it. A retry rebuilds from the durable log in a new numbered step, success clears the consecutive history, and exhaustion records the structured failure once on `turn/end`. AgentLoop privately owns one cancellation holder whose explicit signal spans prompt policy, assembly, every step, model and tool work, recovery, continuation, and terminal stop; it retires the holder immediately before publishing `turn/end`, while the driver may remain `running` through the durability flush. An effective `cancel()` emits the typed runtime-only `user | parent` cause before clearing pending work and cooperatively aborting the holder; notification failures cannot veto cancellation, work queued by a notification observer is cleared, work queued by a later abort observer belongs to the next turn, and idle cancellation emits nothing. Durable `turn/end` remains coarse `aborted`; undispatched model tool calls receive synthetic `tool/call` and `ABORTED_BEFORE_DISPATCH` result pairs. Disposal wins terminal classification, and work that ignores the signal must settle before quiescence. The [explicit-cancellation decision](../../../.agents/notes/implemented/architecture/2026-07-16-explicit-turn-cancellation.md) owns the lifecycle and race contract. Terminal continuation stops remain authoritative through turn close and durability flush.
Within a step, exclusive calls form barriers; parallel-safe calls use a bounded rolling pool and are reclassified before start. Only dispatch/body overlaps. Policy, durable results, and result context remain model-ordered. Abort stops new calls, drains started results, then drains accepted batch context before the turn closes through the normal abort path.

23
packages/core/agent-loop/src/loop.ts
View File

@@ -7,9 +7,9 @@
import { randomUUID } from 'node:crypto'
import type { Context } from 'cordis'
import type { ContentBlock, FinishReason, GenerateOptions, LlmCallConfig, LlmFailure, Message } from '@deepseek-ai/dsh-llm'
import type { ContentBlock, FinishReason, GenerateOptions, LlmCallConfig, LlmFailure, Message, ResolvedRetryPolicy } from '@deepseek-ai/dsh-llm'
import { isDeepStrictEqual } from 'node:util'
import { BlockAssembler, HarnessError, LlmError, assertNever, deepFreeze, errorChain, llmFailureOf, markAgentLoopRequest } from '@deepseek-ai/dsh-llm'
import { BlockAssembler, HarnessError, LlmError, assertNever, deepFreeze, errorChain, llmFailureOf, llmRetryPolicyOf, markAgentLoopRequest } from '@deepseek-ai/dsh-llm'
import { agentEvents, agentInterruptReasonOf, assembleContextFor, AgentMessageId } from '@deepseek-ai/dsh-agent'
import type { AgentEventDispatch, ContinuationDecision, HookContext, PromptDecision, RequestError, RequestErrorDecision } from '@deepseek-ai/dsh-agent'
import { canonicalHeader } from '@deepseek-ai/dsh-session'
@@ -33,6 +33,7 @@ class TerminalModelRequestFailure extends Error {
constructor(
readonly requestError: RequestError,
readonly failure: LlmFailure,
readonly retryPolicy: ResolvedRetryPolicy | undefined,
) {
super(failure.message, { cause: requestError })
this.name = 'TerminalModelRequestFailure'
@@ -442,14 +443,18 @@ async function runTurn(
let stepOutcome:
| { hadToolCalls: boolean; finish: FinishReason }
| { requestError: RequestError; failure: LlmFailure }
| { requestError: RequestError; failure: LlmFailure; retryPolicy: ResolvedRetryPolicy | undefined }
| { error: RequestError }
try {
stepOutcome = await runStep(
ctx, events, handle, turn, step, assembly, fullSystemPrompt, boundaryMessages, transmission, signal)
} catch (error: unknown) {
if (error instanceof TerminalModelRequestFailure) {
stepOutcome = { requestError: error.requestError, failure: error.failure }
stepOutcome = {
requestError: error.requestError,
failure: error.failure,
retryPolicy: error.retryPolicy,
}
} else {
stepOutcome = { error: toError(error) }
}
@@ -470,7 +475,7 @@ async function runTurn(
try {
recoveryDecision = await events.waterfall(
'agent/request-error', turn, step, stepOutcome.requestError,
stepOutcome.failure, requestFailureHistory, signal,
stepOutcome.failure, requestFailureHistory, stepOutcome.retryPolicy, signal,
() => Promise.resolve(defaultDecision),
)
} catch (recoveryError: unknown) {
@@ -714,14 +719,18 @@ async function runStep(
}
} catch (error: unknown) {
const failure = llmFailureOf(stream, error)
if (failure !== undefined && error instanceof Error) throw new TerminalModelRequestFailure(error, failure)
if (failure !== undefined && error instanceof Error) {
throw new TerminalModelRequestFailure(error, failure, llmRetryPolicyOf(stream))
}
throw error
}
interruptionCheckpoint(signal)
// Normalize failure finish chunks into the same path as thrown stream errors.
const stepError = finishError(assembler.finish)
if (stepError) throw new TerminalModelRequestFailure(stepError.error, stepError.failure)
if (stepError) {
throw new TerminalModelRequestFailure(stepError.error, stepError.failure, llmRetryPolicyOf(stream))
}
const recordAssistantMessage = (
assembledContent: ContentBlock[],

44
packages/core/agent-loop/tests/request-recovery.spec.ts
View File

@@ -269,10 +269,11 @@ describe('agent post-step and request-error lifecycle', () => {
const ctx = await harness(adapter)
const agent = ctx.agentLoop.create(SessionId(`recover-${_style}`), { provider: 'mock', model: 'mock' })
const attempts: number[] = []
ctx.on('agent/request-error', async (subject, turn, step, error, facts, history) => {
ctx.on('agent/request-error', async (subject, turn, step, error, facts, history, retryPolicy) => {
expect(subject).toBe(agent)
expect({ turn, step, code: error.code }).toEqual({ turn: 1, step: 1, code: CONTEXT_WINDOW_EXCEEDED_CODE })
expect(facts.code).toBe(CONTEXT_WINDOW_EXCEEDED_CODE)
expect(retryPolicy).toMatchObject({ mode: 'normal', maxRetries: 2 })
attempts.push(history.length)
subject.session.append('user/message', {
content: [{ type: 'text', text: 'RECOVERY SURFACE MUTATION' }],
@@ -301,7 +302,9 @@ describe('agent post-step and request-error lifecycle', () => {
const agent = ctx.agentLoop.create(SessionId(`stream-plugin-${_name.replaceAll(' ', '-')}`), { provider: 'mock', model: 'mock' })
let recoveries = 0
install(ctx)
ctx.on('agent/request-error', async (_agent, _turn, _step, _error, _failure, _history, _signal, next) => {
ctx.on('agent/request-error', async (
_agent, _turn, _step, _error, _failure, _history, _retryPolicy, _signal, next,
) => {
recoveries += 1
return next()
})
@@ -332,7 +335,9 @@ describe('agent post-step and request-error lifecycle', () => {
})
const agent = ctx.agentLoop.create(SessionId('nested-stream-not-recoverable'), { provider: 'mock', model: 'mock' })
let recoveries = 0
ctx.on('agent/request-error', async (_agent, _turn, _step, _error, _failure, _history, _signal, next) => {
ctx.on('agent/request-error', async (
_agent, _turn, _step, _error, _failure, _history, _retryPolicy, _signal, next,
) => {
recoveries += 1
return next()
})
@@ -365,7 +370,9 @@ describe('agent post-step and request-error lifecycle', () => {
}
const agent = ctx.agentLoop.create(SessionId(`${boundary}-not-recoverable`), { provider: 'mock', model: 'mock' })
let recoveries = 0
ctx.on('agent/request-error', async (_agent, _turn, _step, _error, _failure, _history, _signal, next) => {
ctx.on('agent/request-error', async (
_agent, _turn, _step, _error, _failure, _history, _retryPolicy, _signal, next,
) => {
recoveries += 1
return next()
})
@@ -393,7 +400,9 @@ describe('agent post-step and request-error lifecycle', () => {
}
const agent = ctx.agentLoop.create(SessionId(`${failure}-not-recoverable`), { provider: 'mock', model: 'mock' })
let recoveries = 0
ctx.on('agent/request-error', async (_agent, _turn, _step, _error, _failure, _history, _signal, next) => {
ctx.on('agent/request-error', async (
_agent, _turn, _step, _error, _failure, _history, _retryPolicy, _signal, next,
) => {
recoveries += 1
return next()
})
@@ -412,7 +421,9 @@ describe('agent post-step and request-error lifecycle', () => {
const ctx = await harness(makeAdapter(original))
const agent = ctx.agentLoop.create(SessionId(`identity-${_name.replaceAll(' ', '-')}`), { provider: 'mock', model: 'mock' })
let seen: Error | undefined
ctx.on('agent/request-error', async (_agent, _turn, _step, error, _failure, _history, _signal, next) => {
ctx.on('agent/request-error', async (
_agent, _turn, _step, error, _failure, _history, _retryPolicy, _signal, next,
) => {
seen = error
return next()
})
@@ -431,7 +442,9 @@ describe('agent post-step and request-error lifecycle', () => {
const agent = ctx.agentLoop.create(SessionId('hostile-message-recovery'), { provider: 'mock', model: 'mock' })
let seenError: Error | undefined
let seenFailure: LlmFailure | undefined
ctx.on('agent/request-error', async (_agent, _turn, _step, error, failure, _history, _signal, next) => {
ctx.on('agent/request-error', async (
_agent, _turn, _step, error, failure, _history, _retryPolicy, _signal, next,
) => {
seenError = error
seenFailure = failure
return next()
@@ -462,7 +475,7 @@ describe('agent post-step and request-error lifecycle', () => {
let seenFailure: LlmFailure | undefined
let seenHistory: readonly LlmFailure[] | undefined
ctx.on('agent/request-error', async (
_agent, _turn, _step, error, failure, history, _signal, next,
_agent, _turn, _step, error, failure, history, _retryPolicy, _signal, next,
) => {
seenError = error
seenFailure = failure
@@ -506,13 +519,18 @@ describe('agent post-step and request-error lifecycle', () => {
const ctx = scenario === 'iterator' ? await harness(new IteratorConstructionFailureAdapter()) : await harness()
const agent = ctx.agentLoop.create(SessionId(`request-boundary-${scenario}`), { provider: 'mock', model: 'mock' })
let seen = ''
ctx.on('agent/request-error', async (_agent, _turn, _step, error, _failure, _history, _signal, next) => {
let sawServingPolicy = false
ctx.on('agent/request-error', async (
_agent, _turn, _step, error, _failure, _history, retryPolicy, _signal, next,
) => {
seen = error.code ?? ''
sawServingPolicy = retryPolicy !== undefined
return next()
})
send(agent)
await waitForIdle(ctx, agent)
expect(seen).toBe(scenario === 'iterator' ? 'ITERATOR_CONSTRUCTION' : 'NO_ADAPTER')
expect(sawServingPolicy).toBe(scenario === 'iterator')
}
})
@@ -522,7 +540,7 @@ describe('agent post-step and request-error lifecycle', () => {
const cappedAgent = cappedCtx.agentLoop.create(SessionId('retry-cap'), { provider: 'mock', model: 'mock' })
const cappedHistories: string[][] = []
cappedCtx.on('agent/request-error', async (
_agent, _turn, _step, _error, _failure, history, _signal, next,
_agent, _turn, _step, _error, _failure, history, _retryPolicy, _signal, next,
) => {
const codes = history.map(entry => entry.code)
cappedHistories.push(codes)
@@ -547,7 +565,7 @@ describe('agent post-step and request-error lifecycle', () => {
const resetAgent = resetCtx.agentLoop.create(SessionId('retry-reset'), { provider: 'mock', model: 'mock' })
const resetHistories: { step: number; codes: string[] }[] = []
resetCtx.on('agent/request-error', async (
_agent, _turn, step, _error, _failure, history, _signal, next,
_agent, _turn, step, _error, _failure, history, _retryPolicy, _signal, next,
) => {
resetHistories.push({ step, codes: history.map(entry => entry.code) })
return resetHistories.length === 1 ? { action: 'retry' } : next()
@@ -578,7 +596,9 @@ describe('agent post-step and request-error lifecycle', () => {
const agent = ctx.agentLoop.create(SessionId(`${action}-recovery`), { provider: 'mock', model: 'mock' })
let entered!: () => void
const recoveryEntered = new Promise<void>((resolve) => { entered = resolve })
ctx.on('agent/request-error', async (_agent, _turn, _step, _error, _failure, _history, signal) => {
ctx.on('agent/request-error', async (
_agent, _turn, _step, _error, _failure, _history, _retryPolicy, signal,
) => {
entered()
await new Promise<void>((resolve) => {
signal.addEventListener('abort', () => { resolve() }, { once: true })

6
packages/core/agent/src/types.ts
View File

@@ -8,7 +8,7 @@
import type { Context } from 'cordis'
import type { Branded } from '@deepseek-ai/dsh-brand'
import type { Scoped } from '@deepseek-ai/dsh-scope'
import type { ContentBlock, LlmCallConfig, LlmFailure, Message, MessageSource } from '@deepseek-ai/dsh-llm'
import type { ContentBlock, LlmCallConfig, LlmFailure, Message, MessageSource, ResolvedRetryPolicy } from '@deepseek-ai/dsh-llm'
import type { JsonValue, Session, SessionId } from '@deepseek-ai/dsh-session'
import type {} from '@deepseek-ai/dsh-system-prompt'
declare module '@deepseek-ai/dsh-system-prompt' {
@@ -456,11 +456,13 @@ declare module 'cordis' {
* @param error - the original model-request failure.
* @param failure - serializable facts normalized at the final adapter boundary.
* @param priorFailures - immutable failures that already authorized another request in this consecutive sequence.
* @param retryPolicy - immutable policy of the adapter registration that served
* the failed request, or `undefined` if no final adapter served it.
* @param signal - the turn abort signal.
* Scope-filtered dispatch (`@deepseek-ai/dsh-scope`): agent-scoped listeners receive only that agent.
* @mode waterfall
*/
'agent/request-error'(this: Scoped<Agent>, agent: Agent, turn: number, step: number, error: RequestError, failure: LlmFailure, priorFailures: readonly LlmFailure[], signal: AbortSignal, next: () => Promise<RequestErrorDecision>): Promise<RequestErrorDecision>
'agent/request-error'(this: Scoped<Agent>, agent: Agent, turn: number, step: number, error: RequestError, failure: LlmFailure, priorFailures: readonly LlmFailure[], retryPolicy: ResolvedRetryPolicy | undefined, signal: AbortSignal, next: () => Promise<RequestErrorDecision>): Promise<RequestErrorDecision>
/**
* Override whether the turn continues. The default continues after tool
* calls or steering and stops otherwise; a continue reason becomes steering.

2
packages/core/scope/tests/invariant.spec.ts
View File

@@ -51,7 +51,7 @@ describe('scoped-dispatch invariants', () => {
'agent/post-step': [agent, 1, 1, signal],
'agent/prompt-submit': [agent, [], { kind: 'user' }, signal, () => Promise.resolve({ kind: 'allow' })],
'agent/request': [agent, 1, 1, config, signal, () => Promise.resolve(config)],
'agent/request-error': [agent, 1, 1, new Error('request failed'), { message: 'request failed', code: 'UNKNOWN' }, [], signal, () => Promise.resolve({ action: 'fail' })],
'agent/request-error': [agent, 1, 1, new Error('request failed'), { message: 'request failed', code: 'UNKNOWN' }, [], undefined, signal, () => Promise.resolve({ action: 'fail' })],
'agent/session-prefix': [agent, [], signal, () => Promise.resolve([])],
'agent/step-result': [agent, 1, 1, message, signal, () => Promise.resolve(message)],
'agent/turn-continuation': [agent, 1, { action: 'stop' }, signal, () => Promise.resolve({ action: 'stop' })],