fix(cli): bound telemetry shutdown on signals

Refs #1356
This commit is contained in:
fz
2026-08-03 22:08:47 +08:00
parent 8eb97ab47e
commit ef23b88ad4
22 changed files with 454 additions and 70 deletions

View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-07-31-web-telemetry-default-mount.md
2026-07-31-web-telemetry-default-mount.md: e9ec7d0cda37db44e753c9aee572763b7e24ada6
2026-07-31-web-telemetry-default-mount.zh.md: 68b411d0668772ce81d7f323c2d286714a223ca4
2026-07-31-web-telemetry-default-mount.md: 9e7a28d52742535f86d22ba4892b2298575eae99
2026-07-31-web-telemetry-default-mount.zh.md: 20f2262ad36a6b5c8d1325526a3e6d572af12172

View File

@@ -10,7 +10,7 @@ The telemetry seam and OTel backend ([revival Note](2026-07-23-session-telemetry
## Decision
The shared `dsh` core (`apps/cli/config/base.cordis.yml`) mounts the `telemetry-otel` row by default with a baked-in production endpoint, so every surface — TUI, web, and headless — reports; this is the **internal-testing deployment stance** — reporting is on when an endpoint exists, and users opt out through the environment. Each surface's exit path drains the queue: web/headless dispose on SIGINT/SIGTERM (headless gained those handlers in this change), and the TUI's normal exit runs `disposeRootAndExit` (root dispose, 5s bounded — above the ~1s drain ceiling configured here) while its `/resume` handoff disposes the root before `execve`.
The shared `dsh` core (`apps/cli/config/base.cordis.yml`) mounts the `telemetry-otel` row by default with a baked-in production endpoint, so every surface — TUI, web, and headless — reports; this is the **internal-testing deployment stance** — reporting is on when an endpoint exists, and users opt out through the environment. Each surface's exit path drains the queue: web/headless use the [bounded, escalating process-shutdown controller](../bug-fix/2026-08-03-cli-signal-shutdown-escalation.md) on SIGINT/SIGTERM, and the TUI's normal exit runs `disposeRootAndExit` (root dispose, 5s bounded — above the backend's 3s shutdown deadline) while its `/resume` handoff disposes the root before `execve`.
| Ruling | Value | Rationale |
|---|---|---|
@@ -18,7 +18,7 @@ The shared `dsh` core (`apps/cli/config/base.cordis.yml`) mounts the `telemetry-
| Endpoint | `DSH_TELEMETRY_OTLP_URL`, default `https://harness-telemetry.deepseeksvc.com/v1/logs` | Internal collector; the env override serves local/dev runs |
| Opt-out switch | any non-empty `DSH_TELEMETRY_DISABLED` (including `0`/`false`) disables | A privacy switch prefers off-by-mistake over on-by-mistake; a row can only be disabled at AppCLIEntry's patch layer (config has no disable semantic, and the switch must precede the load-time `exporter.url` validation) |
| Cadence | `processor.scheduledDelayMillis: 10000` (10s/batch) | Streaming while the session runs, never exit-time-only; a crash loses at most the last unexported interval |
| Exit-drain bound | `exporter.timeoutMillis: 1000` + `maxExportBatchSize: 2048` (== maxQueueSize) + `exportTimeoutMillis: 1500` | Dispose must release within ~1s against an unreachable collector: timeoutMillis doubles as the per-attempt socket timeout and the retry deadline (1s effectively disables the SDK's 5-try backoff), and aligning batch size with the queue cap makes the drain a single batch; SDK defaults can stall 40s+ |
| Exit-drain bound | `exporter.timeoutMillis: 1000` + `maxExportBatchSize: 2048` (== maxQueueSize) + `exportTimeoutMillis: 1500` + `shutdownTimeoutMillis: 3000` | Ordinary unreachable-collector failure releases in ~1s: timeoutMillis is the per-attempt socket timeout and retry deadline, while one queue-sized batch avoids sequential drain multiplication. The DSH-owned 3s outer bound covers the SDK's preceding unbounded `forceFlush()` wait when the transport Promise never obtains a socket. |
| Compression | `compression: gzip` | Event bodies carry full content; cross-datacenter bandwidth |
| CI isolation | top-level `env: DSH_TELEMETRY_DISABLED: '1'` in all 8 GitHub workflows | Every CI channel that boots the web composition (e2e/snapshot/built smokes) must not stream test sessions to the production endpoint |
@@ -30,7 +30,7 @@ The keyless integration test `apps/cli/tests/telemetry-web.e2e.ts` pins the depl
**A config field instead of an env patch for the switch.** Infeasible: cordis rows have no config-level disable semantic, and `exporter.url` validation fails loud at plugin construction, so the switch must take effect before the Loader — AppCLIEntry's patch layer is the only seat.
**A `Promise.race` timeout backstop around exit.** Deferred: the parameter set already bounds the worst-case drain to ~1.5-3s (typically <100ms), measured SIGINT-to-exit 110ms-1.1s; the unbounded drip-feed-response risk stays under observation, and on real evidence the race lands inside the backend's `shutdown()` (never the coordinator — that would decide loss semantics for every backend).
**A `Promise.race` timeout backstop around exit.** Originally deferred because the SDK parameters appeared to bound the backend's drain to ~1.5-3s (typically <100ms), with measured SIGINT-to-exit of 110ms-1.1s. A Linux sandbox reproduction later proved that `BatchLogRecordProcessor.shutdown()` can wait forever in `exporter.forceFlush()` before reaching its `exportTimeoutMillis`-bounded completion Promise. The [CLI shutdown fix](../bug-fix/2026-08-03-cli-signal-shutdown-escalation.md) therefore adds both a three-second backend bound for that specific gap and a five-second process-level bound plus repeated-signal escape for the whole plugin tree.
## Consequences

View File

@@ -10,7 +10,7 @@ Status: implemented
## Decision
`dsh` 共享核心(`apps/cli/config/base.cordis.yml`)默认挂载 `telemetry-otel` 行,内置生产 endpoint因此所有 surface——TUI、web、headless——都上报这是**内部测试期的部署立场**——有 endpoint 就报,用户可经环境变量退出。各 surface 的退出路径都会排空队列web/headless 在 SIGINT/SIGTERM 上 disposeheadless 的信号处理是本次补上的)TUI 的正常退出走 `disposeRootAndExit`(根 dispose5s 兜底——高于此处配置的 ~1s drain 上界),其 `/resume` 移交也在 `execve` 前 dispose 根。
`dsh` 共享核心(`apps/cli/config/base.cordis.yml`)默认挂载 `telemetry-otel` 行,内置生产 endpoint因此所有 surface——TUI、web、headless——都上报这是**内部测试期的部署立场**——有 endpoint 就报,用户可经环境变量退出。各 surface 的退出路径都会排空队列web/headless 在 SIGINT/SIGTERM 时使用[有界、可升级的进程关闭控制器](../bug-fix/2026-08-03-cli-signal-shutdown-escalation.md)TUI 的正常退出走 `disposeRootAndExit`(根 dispose5s 兜底——高于后端的 3s 关闭截止时间),其 `/resume` 移交也在 `execve` 前 dispose 根。
| 决策项 | 取值 | 理由 |
|---|---|---|
@@ -18,7 +18,7 @@ Status: implemented
| endpoint | `DSH_TELEMETRY_OTLP_URL`,缺省 `https://harness-telemetry.deepseeksvc.com/v1/logs` | 内部 collectorenv 覆盖供本地/联调 |
| 退出开关 | `DSH_TELEMETRY_DISABLED` 非空(含 `0`/`false`)即关 | 隐私向开关取「宁关勿误开」;行级 disable 只能在 AppCLIEntry 的 patch 层做config 无 disable 语义,且必须先于 `exporter.url` 的加载期校验生效) |
| 上报节奏 | `processor.scheduledDelayMillis: 10000`10s/批) | 流式回流,非退出才报;崩溃至多丢最后一个未导出间隔 |
| 退出 drain 上界 | `exporter.timeoutMillis: 1000` + `maxExportBatchSize: 2048== maxQueueSize` + `exportTimeoutMillis: 1500` | collector 不可达时 dispose 必须 ~1s 内放行timeoutMillis 同时是单次 socket 超时与重试 deadline1s 等效关掉 SDK 5 次 backoff批大小对齐队列上限使 drain 恒为单批;默认参数下最坏可卡 40s+ |
| 退出 drain 上界 | `exporter.timeoutMillis: 1000` + `maxExportBatchSize: 2048== maxQueueSize` + `exportTimeoutMillis: 1500` + `shutdownTimeoutMillis: 3000` | collector 不可达的常规故障会在约 1s 内放行timeoutMillis 是单次 socket 超时与重试 deadline,使用与队列等大的单批可避免依次排空导致耗时倍增。由 DSH 管理的 3s 外层上限覆盖 SDK 先执行的无界 `forceFlush()` 等待,即传输 Promise 始终无法取得 socket 的情况。 |
| 压缩 | `compression: gzip` | 事件 body 含全文,跨机房带宽 |
| CI 隔离 | 全部 8 个 GitHub workflow 顶层 `env: DSH_TELEMETRY_DISABLED: '1'` | CI 启动 web 组合的所有通道e2e/snapshot/built smoke不得向生产 endpoint 泄测试会话 |
@@ -30,7 +30,7 @@ Status: implemented
**开关做成 config 字段而非 env patch。** 不可行cordis 行没有 config 层的 disable 语义,且 `exporter.url` 校验在插件构造期 fail-loud开关必须在 Loader 之前生效——AppCLIEntry patch 层是唯一落点。
**退出时 `Promise.race` 兜底超时。** 暂缓:参数组合已把最坏 drain 压到 ~1.5-3s典型 <100ms实测 SIGINT→退出 110ms-1.1sdrip-feed 慢滴响应的无界等待风险留观,出现实证再在 backend `shutdown()` 内加 race不放 coordinator——那会替所有 backend 决定丢失语义)
**退出时 `Promise.race` 兜底超时。** 最初暂缓,是因为 SDK 参数看似已经将后端排空耗时限制在约 1.5-3s通常 <100ms实测 SIGINT 到退出耗时 110ms-1.1s。后来在 Linux 沙箱中复现并证明,`BatchLogRecordProcessor.shutdown()` 可能在 `exporter.forceFlush()` 中永久等待,无法进入受 `exportTimeoutMillis` 限制的完成 Promise。因此[CLI 关闭修复](../bug-fix/2026-08-03-cli-signal-shutdown-escalation.md) 既为这一特定缺口增加 3 秒后端上限,也为整棵插件树增加 5 秒进程级上限和重复信号退出途径
## Consequences