refactor(telemetry): remove the OTel backend's flush forwarding
Three review rounds each found a new silent-loss path in the same wrapper state (dispose racing an in-flight flush, overlapping hints displacing the retained promise, the provider's fixed 30s flush timeout rejecting while the processor still drains). Every path exists only because forwarding the seam's turn-boundary hint to forceFlush() made this backend the process's second flusher against undocumented SDK internals from the upstream experimental tree. The backend now implements no flush(): the batch processor is the only flusher, its scheduledDelayMillis (already deployment-tunable through the processor passthrough) governs export cadence, and shutdown()'s drain is complete by construction. The two race-pin tests collapse into one dispose-during-in-flight-batch drain pin; the seam's optional flush() contract now tells implementers they own the concurrent-flush/ shutdown interaction. Removal rationale and the reinstatement trigger (a stated turn-boundary latency requirement scheduledDelayMillis cannot meet — and then via the processor's own forceFlush(), never the provider's timeout-wrapped one) are recorded in the revival Agent Note, both languages.
This commit is contained in:
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
README.md: 2e0c902e7b16726829319c852eab3f26a68847b3
|
||||
README.zh.md: 4917ee1fcff91f07d29bbcb1b6957b80d9e73197
|
||||
README.md: 28fffc5f43f960de1a44700aa70050d792b81d4f
|
||||
README.zh.md: 821969fea09487d9d4450e981a0d791a66dad3a1
|
||||
|
||||
@@ -17,7 +17,7 @@ The OpenTelemetry backend for [the telemetry seam](../session-telemetry/) — th
|
||||
processor: {} # optional; passed verbatim to BatchLogRecordProcessor
|
||||
```
|
||||
|
||||
`exporter.url` is the one field this package validates itself — required, no default, must parse as `http(s)` — so a missing endpoint fails at plugin load. Everything else is the SDK's option shape, owned and documented by the SDK, and both blocks pass through whole: every `OTLPExporterNodeConfigBase` field (`headers`, `timeoutMillis`, `compression`, `keepAlive`, …) reaches the exporter, and batching, retry, queue bounds, and loss policy under sustained failure are the SDK's documented behavior, tuned through the `processor` passthrough. Removing this block from `cordis.yml` is the opt-out: no residual state, no `enabled` flag.
|
||||
`exporter.url` is the one field this package validates itself — required, no default, must parse as `http(s)` — so a missing endpoint fails at plugin load. Everything else is the SDK's option shape, owned and documented by the SDK, and both blocks pass through whole: every `OTLPExporterNodeConfigBase` field (`headers`, `timeoutMillis`, `compression`, `keepAlive`, …) reaches the exporter, and batching, export cadence (`scheduledDelayMillis`), retry, queue bounds, and loss policy under sustained failure are the SDK's documented behavior, tuned through the `processor` passthrough. The backend deliberately implements no `flush()`: the batch processor is the only flusher in the process, which is what makes `shutdown()`'s drain complete. Removing this block from `cordis.yml` is the opt-out: no residual state, no `enabled` flag.
|
||||
|
||||
## What leaves the machine
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@
|
||||
processor: {} # optional; passed verbatim to BatchLogRecordProcessor
|
||||
```
|
||||
|
||||
`exporter.url` 是本包唯一自行校验的字段:必填、无默认值、必须能解析为 `http(s)`,因此缺失端点会在插件加载时失败。其余全部是 SDK 自己的选项形态,由 SDK 拥有并在 SDK 文档中说明,两个配置块都整体透传(passthrough):`OTLPExporterNodeConfigBase` 的每个字段(`headers`、`timeoutMillis`、`compression`、`keepAlive` 等)都会到达导出器;批处理、重试、队列上限,以及持续失败下的丢失策略,都是 SDK 的文档化行为,经 `processor` 透传调优。从 `cordis.yml` 中删除该配置块即为退出方式:无残留状态,也没有 `enabled` 开关。
|
||||
`exporter.url` 是本包唯一自行校验的字段:必填、无默认值、必须能解析为 `http(s)`,因此缺失端点会在插件加载时失败。其余全部是 SDK 自己的选项形态,由 SDK 拥有并在 SDK 文档中说明,两个配置块都整体透传(passthrough):`OTLPExporterNodeConfigBase` 的每个字段(`headers`、`timeoutMillis`、`compression`、`keepAlive` 等)都会到达导出器;批处理、导出节奏(`scheduledDelayMillis`)、重试、队列上限,以及持续失败下的丢失策略,都是 SDK 的文档化行为,经 `processor` 透传调优。该后端刻意不实现 `flush()`:批处理器是进程内唯一执行 flush 的组件,`shutdown()` 的排空正因如此才是完整的。从 `cordis.yml` 中删除该配置块即为退出方式:无残留状态,也没有 `enabled` 开关。
|
||||
|
||||
## 哪些数据会离开本机
|
||||
|
||||
|
||||
@@ -147,36 +147,26 @@ export class TelemetryOtel extends Telemetry {
|
||||
})
|
||||
}
|
||||
|
||||
/** Every not-yet-settled turn-boundary flush, retained so {@link shutdown} can order behind ALL of them. */
|
||||
private inflightFlush: Promise<void> = Promise.resolve()
|
||||
|
||||
/** Forward the turn-boundary hint to the SDK's flush, fire-and-forget. */
|
||||
override flush(): void {
|
||||
// Best-effort hint: the SDK resolves forceFlush even when exports fail
|
||||
// (failures go to its own diagnostics), and the coordinator stops calling
|
||||
// this once the fiber is disposed — a rejection would be SDK drift. The
|
||||
// settled promise is retained (not awaited): the SDK's concurrent-flush
|
||||
// guard makes a flush that overlaps another return WITHOUT draining, so
|
||||
// an overlapping hint resolves instantly and must JOIN the outstanding
|
||||
// one, not displace it — shutdown orders behind the whole set.
|
||||
/* v8 ignore next -- unreachable guard: forceFlush does not reject while the provider is alive */
|
||||
const flush = this.provider.forceFlush().catch(() => {})
|
||||
this.inflightFlush = Promise.all([this.inflightFlush, flush]).then(() => undefined)
|
||||
}
|
||||
// The seam's optional flush() hint is deliberately NOT implemented. The
|
||||
// batch processor exports on its own cadence (`processor.scheduledDelayMillis`,
|
||||
// the SDK's documented knob), and this backend is the SDK pipeline's only
|
||||
// caller — forwarding the hint to `forceFlush()` was the sole source of
|
||||
// concurrent flushes, whose undocumented interactions with shutdown's
|
||||
// internal drain (concurrent-flush guard, provider-level flush timeout)
|
||||
// silently dropped tail records. Removal history and the revival trigger:
|
||||
// the revival Agent Note.
|
||||
|
||||
/**
|
||||
* Delegate disposal to the SDK's shutdown contract: flush the queue and
|
||||
* quiesce. Orders behind every outstanding turn-boundary flush first —
|
||||
* shutdown's internal flush is a no-op while one is in flight (the SDK's
|
||||
* concurrent-flush guard), which would silently drop everything enqueued
|
||||
* after that flush snapshot, including the coordinator's dispose-time
|
||||
* `shutdown` markers. Awaited (and error-contained) by the coordinator's
|
||||
* disposer.
|
||||
* Delegate disposal to the SDK's shutdown contract: drain the queue and
|
||||
* quiesce. With no concurrent `forceFlush()` in the process (see above),
|
||||
* shutdown's internal drain is complete — everything emitted before this
|
||||
* call, including the coordinator's dispose-time `shutdown` markers, is
|
||||
* exported before the exporter closes. Awaited (and error-contained) by
|
||||
* the coordinator's disposer.
|
||||
* @returns resolves when the SDK pipeline has quiesced.
|
||||
*/
|
||||
async shutdown(): Promise<void> {
|
||||
await this.inflightFlush
|
||||
await this.provider.shutdown()
|
||||
shutdown(): Promise<void> {
|
||||
return this.provider.shutdown()
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -123,12 +123,14 @@ describe('TelemetryOtel wire', () => {
|
||||
expect(ops[0]!.record.attributes).toContainEqual({ key: 'telemetry.op', value: { stringValue: 'shutdown' } })
|
||||
})
|
||||
|
||||
it('delivers records enqueued while a turn-boundary flush is in flight (flush/shutdown race)', async () => {
|
||||
// Hold the collector's response to the flush-triggered export open until
|
||||
// after disposal has begun: the SDK's concurrent-flush guard makes the
|
||||
// shutdown-internal flush return early while another flush is running, so
|
||||
// without ordering in the backend the coordinator's dispose-time shutdown
|
||||
// marker (enqueued after the flush snapshot) would be dropped silently.
|
||||
it('drains records enqueued after a timer export began: dispose during an in-flight batch', async () => {
|
||||
// The backend implements NO flush() — the batch processor exports on its
|
||||
// own cadence, and shutdown's internal drain is complete exactly because
|
||||
// nothing in the process calls forceFlush() concurrently (the SDK's
|
||||
// concurrent-flush guard skips draining otherwise). Pin that: hold the
|
||||
// collector's response to the timer-triggered export open across
|
||||
// disposal, and the dispose-time shutdown marker (enqueued after that
|
||||
// batch's snapshot) must still arrive.
|
||||
const gate = Promise.withResolvers<boolean>()
|
||||
const arrived = Promise.withResolvers<boolean>()
|
||||
const { url, captures } = await mockCollector(async (index) => {
|
||||
@@ -137,10 +139,14 @@ describe('TelemetryOtel wire', () => {
|
||||
await gate.promise
|
||||
}
|
||||
})
|
||||
const { ctx, fiber } = await boot(url)
|
||||
const session = ctx.sessions.create(SessionId('race'), { meta: {} })
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(TelemetryOtel, {
|
||||
exporter: { url },
|
||||
processor: { scheduledDelayMillis: 10 },
|
||||
})
|
||||
const session = ctx.sessions.create(SessionId('drain'), { meta: {} })
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
ctx.telemetry.flush!()
|
||||
await arrived.promise
|
||||
|
||||
const disposal = fiber.dispose()
|
||||
@@ -149,40 +155,6 @@ describe('TelemetryOtel wire', () => {
|
||||
gate.resolve(true)
|
||||
await disposal
|
||||
|
||||
const records = allRecords(captures)
|
||||
const ops = records.filter(r => r.scope === '@deepseek-ai/dsh-session-telemetry-otel/ops')
|
||||
expect(ops).toHaveLength(1)
|
||||
expect(ops[0]!.record.attributes).toContainEqual({ key: 'telemetry.op', value: { stringValue: 'shutdown' } })
|
||||
})
|
||||
|
||||
it('orders shutdown behind the OLDEST in-flight flush when hints overlap', async () => {
|
||||
// The SDK's concurrent-flush guard resolves an overlapping forceFlush()
|
||||
// immediately; if the backend RETAINS only the latest flush promise, two
|
||||
// back-to-back turn flushes leave shutdown awaiting the instantly-resolved
|
||||
// second one while the first still exports — reopening the same silent
|
||||
// drop the single-flush race test pins.
|
||||
const gate = Promise.withResolvers<boolean>()
|
||||
const arrived = Promise.withResolvers<boolean>()
|
||||
const { url, captures } = await mockCollector(async (index) => {
|
||||
if (index === 0) {
|
||||
arrived.resolve(true)
|
||||
await gate.promise
|
||||
}
|
||||
})
|
||||
const { ctx, fiber } = await boot(url)
|
||||
const session = ctx.sessions.create(SessionId('race2'), { meta: {} })
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
ctx.telemetry.flush!()
|
||||
await arrived.promise
|
||||
// Second hint while the first export is held open: resolves immediately
|
||||
// under the SDK's guard and must not displace the outstanding one.
|
||||
ctx.telemetry.flush!()
|
||||
|
||||
const disposal = fiber.dispose()
|
||||
await new Promise(resolve => setTimeout(resolve, 50))
|
||||
gate.resolve(true)
|
||||
await disposal
|
||||
|
||||
const ops = allRecords(captures).filter(r => r.scope === '@deepseek-ai/dsh-session-telemetry-otel/ops')
|
||||
expect(ops).toHaveLength(1)
|
||||
expect(ops[0]!.record.attributes).toContainEqual({ key: 'telemetry.op', value: { stringValue: 'shutdown' } })
|
||||
@@ -209,15 +181,14 @@ describe('TelemetryOtel wire', () => {
|
||||
expect(types).toContain('turn/start')
|
||||
})
|
||||
|
||||
it('maps the warn severity and forwards the flush hint to the SDK', async () => {
|
||||
it('maps the warn severity and leaves the seam flush hint unimplemented', async () => {
|
||||
const { url, captures } = await mockCollector()
|
||||
const { ctx, fiber } = await boot(url)
|
||||
const session = ctx.sessions.create(SessionId('warn'), { meta: {} })
|
||||
session.append('prompt/blocked', { content: [], source: { kind: 'user' }, reason: 'vetoed' })
|
||||
// The turn-boundary hint: safe, non-blocking, and enough to push the batch out.
|
||||
expect(() => {
|
||||
ctx.telemetry.flush!()
|
||||
}).not.toThrow()
|
||||
// No flush(): the coordinator's optional-call forwarding no-ops, and the
|
||||
// batch processor owns export cadence end to end (see the backend note).
|
||||
expect('flush' in ctx.telemetry && ctx.telemetry.flush !== undefined).toBe(false)
|
||||
await fiber.dispose()
|
||||
const blocked = allRecords(captures).find(r =>
|
||||
r.record.attributes?.some(a => a.key === 'event.type' && a.value.stringValue === 'prompt/blocked'))
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
README.md: 73bc22053ad08725fecade43fb33fe944dfe3c60
|
||||
README.zh.md: 8e63e14bd8e099980133c9d6d864135fba7f39af
|
||||
README.md: df0384d0528f0b3a95ded414444e1b11ea7d52bf
|
||||
README.zh.md: d86fede206a4b2363f1deda86ff32d868d1db71d
|
||||
|
||||
@@ -6,7 +6,7 @@ The telemetry seam: the CAPTURE side of session-event reporting, behind a backen
|
||||
|
||||
## The backend contract
|
||||
|
||||
`TelemetryBackend` is three members: `emit(record)` (MUST be a non-blocking enqueue — it runs synchronously on the `session/event` hot path), optional `flush()` (a turn-boundary hint, fire-and-forget), and `shutdown()` (the lifecycle forward: flush-and-quiesce, awaited at dispose). `Telemetry` is its service-registered form under the `telemetry` context key — one implementation per context, duplicate load throws. A backend composes `TelemetryCoordinator` in its constructor.
|
||||
`TelemetryBackend` is three members: `emit(record)` (MUST be a non-blocking enqueue — it runs synchronously on the `session/event` hot path), optional `flush()` (a turn-boundary hint, fire-and-forget; most backends leave it unimplemented and let their SDK's batching cadence govern export timing — an implementer owns the interaction between concurrent flushes and `shutdown()`'s drain), and `shutdown()` (the lifecycle forward: drain-and-quiesce, awaited at dispose). `Telemetry` is its service-registered form under the `telemetry` context key — one implementation per context, duplicate load throws. A backend composes `TelemetryCoordinator` in its constructor.
|
||||
|
||||
## Capture points
|
||||
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
## 后端契约
|
||||
|
||||
`TelemetryBackend` 只有三个成员:`emit(record)`(必须是非阻塞入队;它在 `session/event` 热路径上同步执行)、可选的 `flush()`(轮次边界提示,触发后不等待结果)、以及 `shutdown()`(生命周期转发点:flush 并完全停稳,在 dispose(资源释放)时被等待)。`Telemetry` 是它注册在 `telemetry` 上下文键下的服务形态:每个上下文只允许一个实现,重复加载会抛出异常。后端在其构造函数中组合 `TelemetryCoordinator`。
|
||||
`TelemetryBackend` 只有三个成员:`emit(record)`(必须是非阻塞入队;它在 `session/event` 热路径上同步执行)、可选的 `flush()`(轮次边界提示,触发后不等待结果;多数后端不实现它,而由其 SDK 的批处理节奏决定导出时机;并发 flush 与 `shutdown()` 的排空之间的交互由实现方自行负责)、以及 `shutdown()`(生命周期转发点:排空并完全停稳,在 dispose(资源释放)时被等待)。`Telemetry` 是它注册在 `telemetry` 上下文键下的服务形态:每个上下文只允许一个实现,重复加载会抛出异常。后端在其构造函数中组合 `TelemetryCoordinator`。
|
||||
|
||||
## 捕获点
|
||||
|
||||
|
||||
@@ -103,7 +103,12 @@ export interface TelemetryBackend {
|
||||
* Optional hint that a natural boundary (turn end) passed — a backend may
|
||||
* forward it to its SDK's flush so records land at turn boundaries. Called
|
||||
* fire-and-forget; implementations must not block and must not throw
|
||||
* meaningfully (the coordinator contains exceptions).
|
||||
* meaningfully (the coordinator contains exceptions). Most backends should
|
||||
* leave this unimplemented and let their SDK's own batching cadence govern
|
||||
* export timing: a backend that does implement it owns the interaction
|
||||
* between its concurrent flushes and {@link shutdown}'s drain (the OTel
|
||||
* backend removed its implementation for exactly that hazard — see the
|
||||
* revival Agent Note).
|
||||
*/
|
||||
flush?(): void
|
||||
/**
|
||||
|
||||
Reference in New Issue
Block a user