Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(llm): simplify live request telemetry (round 2)

Browse Source
This commit is contained in:
Hypatia May
2026-07-28 19:03:45 +08:00
parent 35b9c454e5
commit e6ce6abd7d
31 changed files with 159 additions and 187 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
packages/llm/llm/README.i18n.yaml
View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write packages/llm/llm/README.md
README.md: d28a5632a3fbdbf11c7dba2ee0c57a704f2ba6f4
README.zh.md: fd93fa43d5bfabd6e8751d4efbad229096ced08d
README.md: 5d0459b722c4c5f472231ee86e775bbc4ff7b7f2
README.zh.md: 3dd7153d63c8c4b40b737ed58d5d6d1a29f3153a

2
packages/llm/llm/README.md
View File

@@ -25,7 +25,7 @@ Provider and model metadata is a discovery surface, not a routing whitelist. `re
Exact-model metadata is a separate correctness query, not a catalog decoration or global LLM setting. `resolveModelInfo()` asks the adapter that owns the exact provider/model route once; an adapter can describe an unlisted dynamic model, and absent `context` or `reasoning` fields mean only that those capabilities are unavailable. Invalid identity, context, or reasoning metadata fails with `INVALID_MODEL_INFO`, `INVALID_MODEL_CONTEXT`, or `INVALID_MODEL_REASONING`.
Reasoning identifiers are opaque adapter-owned strings rather than a core enum. An adapter publishes its ordered selectable list, including an `off` id when that model's capability API exposes one. `resolveCallConfig()` accepts only an exact advertised identifier, materializes `defaultEffort` when present, and otherwise preserves the provider default. Asynchronous model resolvers receive the caller's signal and must settle promptly after cancellation. `prepareCall()` additionally exposes the detached context metadata from that same lookup and retains the exact adapter registration through header logging and terminal dispatch, so HMR cannot combine one adapter's capability result with another adapter's request; reusing its one-shot handle or changing its call-config fields fails with `INVALID_PREPARED_CALL`. Its dispatch observer runs after a final stream handle is constructed and before adapter iteration. An unsupported explicit or configured effort fails with `UNSUPPORTED_REASONING_EFFORT` before provider I/O.
Reasoning identifiers are opaque adapter-owned strings rather than a core enum. An adapter publishes its ordered selectable list, including an `off` id when that model's capability API exposes one. `resolveCallConfig()` accepts only an exact advertised identifier, materializes `defaultEffort` when present, and otherwise preserves the provider default. Asynchronous model resolvers receive the caller's signal and must settle promptly after cancellation. `prepareCall()` additionally exposes the detached context metadata from that same lookup and retains the exact adapter registration through header logging and terminal dispatch, so HMR cannot combine one adapter's capability result with another adapter's request; reusing its one-shot handle or changing its call-config fields fails with `INVALID_PREPARED_CALL`. An unsupported explicit or configured effort fails with `UNSUPPORTED_REASONING_EFFORT` before provider I/O.
### Events

2
packages/llm/llm/README.zh.md
View File

@@ -25,7 +25,7 @@
确切模型元数据是独立的正确性查询，不是 catalog 装饰或全局 LLM 设置。`resolveModelInfo()` 会向拥有精确提供方／模型路由的适配器查询一次；适配器可以描述未列出的动态模型，缺少 `context` 或 `reasoning` 字段只表示相应能力不可用。无效的身份、上下文或推理元数据会以 `INVALID_MODEL_INFO`、`INVALID_MODEL_CONTEXT` 或 `INVALID_MODEL_REASONING` 失败。
推理标识符是由适配器持有的不透明字符串，而非核心枚举。适配器会公布有序可选列表；模型能力 API 提供 `off` id 时，列表也会包含它。`resolveCallConfig()` 只接受与已公布标识符完全一致的值，在存在 `defaultEffort` 时填入它，否则保留提供方默认值。异步模型解析器会接收调用方的 signal，并且必须在取消后迅速完成结算。`prepareCall()` 还会公开同一次查询得到的脱耦上下文元数据，并让精确适配器注册跨越请求头记录和最终分派，因此 HMR（热模块替换）不会将一个适配器的能力结果与另一个适配器的请求混用；复用其一次性句柄或更改调用配置字段会以 `INVALID_PREPARED_CALL` 失败。其分派观察器在最终流句柄构造完成后、适配器开始迭代前运行。不支持的显式或配置推理强度会在提供方 I/O 前以 `UNSUPPORTED_REASONING_EFFORT` 失败。
推理标识符是由适配器持有的不透明字符串，而非核心枚举。适配器会公布有序可选列表；模型能力 API 提供 `off` id 时，列表也会包含它。`resolveCallConfig()` 只接受与已公布标识符完全一致的值，在存在 `defaultEffort` 时填入它，否则保留提供方默认值。异步模型解析器会接收调用方的 signal，并且必须在取消后迅速完成结算。`prepareCall()` 还会公开同一次查询得到的脱耦上下文元数据，并让精确适配器注册跨越请求头记录和最终分派，因此 HMR（热模块替换）不会将一个适配器的能力结果与另一个适配器的请求混用；复用其一次性句柄或更改调用配置字段会以 `INVALID_PREPARED_CALL` 失败。不支持的显式或配置推理强度会在提供方 I/O 前以 `UNSUPPORTED_REASONING_EFFORT` 失败。
### 事件

117
packages/llm/llm/src/index.ts
View File

@@ -119,11 +119,9 @@ export interface PreparedLlmCall {
* preparation. The request's call-config fields must match {@link config};
* reuse or mismatch fails with `INVALID_PREPARED_CALL`.
* @param options - fully assembled request carrying the prepared config.
* @param onDispatched - contained Agent-loop notification hook invoked after
* a stream handle is constructed and before its adapter is iterated.
* @returns the chunk stream, including the `llm/stream` waterfall.
*/
stream(options: GenerateOptions, onDispatched?: () => void): AsyncIterable<StreamChunk>
stream(options: GenerateOptions): AsyncIterable<StreamChunk>
}
/**
@@ -408,6 +406,7 @@ export class LlmService extends Service {
const resolved = await this.resolveModelInfoFor(registration, config.model, signal)
const reasoning = resolved.reasoning
const requested = config.reasoningEffort
let resolvedConfig = config
if (reasoning === undefined) {
if (requested !== undefined) {
throw new LlmError(
@@ -415,26 +414,20 @@ export class LlmService extends Service {
'UNSUPPORTED_REASONING_EFFORT',
)
}
return {
config,
...resolved.context === undefined ? {} : { context: resolved.context },
} else {
const effective = requested ?? reasoning.defaultEffort
if (effective !== undefined) {
if (!reasoning.efforts.some(effort => effort.id === effective)) {
throw new LlmError(
`provider "${config.provider}" model "${config.model}" does not support reasoning effort "${effective}"`,
'UNSUPPORTED_REASONING_EFFORT',
)
}
if (requested !== effective) resolvedConfig = { ...config, reasoningEffort: effective }
}
}
const effective = requested ?? reasoning.defaultEffort
if (effective === undefined) {
return {
config,
...resolved.context === undefined ? {} : { context: resolved.context },
}
}
if (!reasoning.efforts.some(effort => effort.id === effective)) {
throw new LlmError(
`provider "${config.provider}" model "${config.model}" does not support reasoning effort "${effective}"`,
'UNSUPPORTED_REASONING_EFFORT',
)
}
return {
config: requested === effective ? config : { ...config, reasoningEffort: effective },
config: resolvedConfig,
...resolved.context === undefined ? {} : { context: resolved.context },
}
}
@@ -458,16 +451,12 @@ export class LlmService extends Service {
return Object.freeze({
config: resolvedConfig,
...context === undefined ? {} : { context },
stream: (options: GenerateOptions, onDispatched?: () => void): AsyncIterable<StreamChunk> => {
stream: (options: GenerateOptions): AsyncIterable<StreamChunk> => {
if (dispatched) {
throw new LlmError('a prepared LLM call can only be dispatched once', 'INVALID_PREPARED_CALL')
}
dispatched = true
return this.streamWithRegistration(
options,
{ registration, config: resolvedConfig },
onDispatched,
)
return this.streamWithRegistration(options, { registration, config: resolvedConfig })
},
})
}
@@ -502,50 +491,24 @@ export class LlmService extends Service {
* so it cannot suppress the primary provider error. A downstream close awaits
* adapter cleanup, whose failures remain ordinary untagged work.
*/
private adapterStream(
private async * adapterStream(
options: GenerateOptions,
failures: AdapterFailureScope,
prepared?: { registration: AdapterRegistration; config: LlmCallConfig },
onDispatched?: () => void,
): AsyncIterable<StreamChunk> {
if (prepared === undefined) {
return this.resolveAndStream(options, failures, onDispatched)
}
): AsyncGenerator<StreamChunk> {
let iterator: AsyncIterator<StreamChunk>
try {
const registration = prepared.registration
const registration = prepared?.registration ?? this.registration(options.provider)
failures.retryPolicy = registration.retryPolicy
const resolvedConfig = prepared.config
if (!callConfigEquals(options, resolvedConfig)) {
const resolvedConfig = prepared === undefined
? (await this.resolveCallFor(registration, options, options.signal)).config
: prepared.config
if (prepared !== undefined && !callConfigEquals(options, resolvedConfig)) {
throw new LlmError(
'prepared LLM call config changed before adapter dispatch',
'INVALID_PREPARED_CALL',
)
}
const adapter = registration.adapter
const stream = adapter.stream(this.forAdapter(options, adapter))
iterator = stream[Symbol.asyncIterator]()
} catch (error: unknown) {
return this.failedAdapterStream(markLlmAdapterFailure(failures, error))
}
this.notifyDispatched(onDispatched)
return this.iterateAdapter(iterator, failures)
}
private async * resolveAndStream(
options: GenerateOptions,
failures: AdapterFailureScope,
onDispatched?: () => void,
): AsyncGenerator<StreamChunk> {
let iterator: AsyncIterator<StreamChunk>
try {
const registration = this.registration(options.provider)
failures.retryPolicy = registration.retryPolicy
const resolvedConfig = (await this.resolveCallFor(
registration,
options,
options.signal,
)).config
const resolvedOptions = callConfigEquals(options, resolvedConfig)
? options
: Object.isFrozen(options)
@@ -557,19 +520,7 @@ export class LlmService extends Service {
} catch (error: unknown) {
throw markLlmAdapterFailure(failures, error)
}
this.notifyDispatched(onDispatched)
yield* this.iterateAdapter(iterator, failures)
}
private async * failedAdapterStream(error: Error): AsyncGenerator<StreamChunk> {
await Promise.resolve()
throw error
}
private async * iterateAdapter(
iterator: AsyncIterator<StreamChunk>,
failures: AdapterFailureScope,
): AsyncGenerator<StreamChunk> {
let completed = false
let iterationFailed = false
try {
@@ -599,15 +550,6 @@ export class LlmService extends Service {
}
}
private notifyDispatched(onDispatched: (() => void) | undefined): void {
if (onDispatched === undefined) return
try {
onDispatched()
} catch (error: unknown) {
this.ctx.logger.warn(`llm dispatch observer threw: ${String(error)}`)
}
}
/**
* Stream one model call as raw chunks (token-level deltas). Throws
* `LlmError` with code `NO_ADAPTER` if no adapter is registered for
@@ -619,32 +561,23 @@ export class LlmService extends Service {
* agent-loop request recovery; middleware and nested-call failures remain
* untagged for the outer call.
* @param options - the full request; `options.provider` selects the adapter.
* @param onDispatched - contained Agent-loop notification hook invoked after
* a stream handle is constructed and before its adapter is iterated.
* @returns the chunk stream, possibly wrapped by `llm/stream` listeners.
*/
stream(options: GenerateOptions, onDispatched?: () => void): AsyncIterable<StreamChunk> {
return this.streamWithRegistration(options, undefined, onDispatched)
stream(options: GenerateOptions): AsyncIterable<StreamChunk> {
return this.streamWithRegistration(options)
}
private streamWithRegistration(
options: GenerateOptions,
prepared?: { registration: AdapterRegistration; config: LlmCallConfig },
onDispatched?: () => void,
): AsyncIterable<StreamChunk> {
const failures: AdapterFailureScope = { failures: new WeakMap<Error, LlmFailure>() }
let terminalEntered = false
const stream = this.ctx.waterfall(
this,
'llm/stream',
options,
() => {
terminalEntered = true
return this.adapterStream(options, failures, prepared, onDispatched)
},
() => this.adapterStream(options, failures, prepared),
)
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition -- waterfall mutates this latch.
if (!terminalEntered) this.notifyDispatched(onDispatched)
return bindAdapterFailureScope(stream, failures)
}
}

16
packages/llm/llm/tests/service.spec.ts
View File

@@ -1070,11 +1070,14 @@ describe('LlmService', () => {
provider,
id: model,
name: model,
description: 'Resolved model',
context: source,
reasoning: {
efforts: [{ id: ReasoningEffortId('high'), name: 'High' }],
defaultEffort: ReasoningEffortId('high'),
},
reasoning: model === 'no-default'
? { efforts: [{ id: ReasoningEffortId('high'), name: 'High' }] }
: {
efforts: [{ id: ReasoningEffortId('high'), name: 'High' }],
defaultEffort: ReasoningEffortId('high'),
},
})
}
}(SCRIPT)
@@ -1090,6 +1093,11 @@ describe('LlmService', () => {
messages: [],
})) { /* drain */ }
expect(resolutions).toBe(1)
const noDefault = await ctx.llm.prepareCall({ provider: 'route', model: 'no-default' })
expect(noDefault.config).toEqual({ provider: 'route', model: 'no-default' })
expect(noDefault.context).toEqual({ contextWindow: 64_000 })
expect(resolutions).toBe(2)
})
it('passes cancellation through exact-model resolution', async () => {