fix(cli): bound telemetry shutdown on signals

Refs #1356
This commit is contained in:
fz
2026-08-03 22:08:47 +08:00
parent 8eb97ab47e
commit ef23b88ad4
22 changed files with 454 additions and 70 deletions

View File

@@ -0,0 +1,6 @@
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/bug-fix/2026-08-03-cli-signal-shutdown-escalation.md
2026-08-03-cli-signal-shutdown-escalation.md: 62abe7a8e6887345232bd5c429a9abc61a424b5d
2026-08-03-cli-signal-shutdown-escalation.zh.md: b8ccf94dc14a9363d8e373ebd13c4a89cd03e23c

View File

@@ -0,0 +1,52 @@
# Agent Note: Bounded, escalating signal shutdown for Web and headless
Status: implemented
English | [中文](2026-08-03-cli-signal-shutdown-escalation.zh.md)
## Problem
The default telemetry mount added SIGINT/SIGTERM handlers to `dsh web` and `dsh -p` so process exit could drain the Cordis tree instead of dropping queued telemetry. Each handler used a one-way boolean latch and exited only after `ctx.fiber.dispose()` settled. Headless normal completion also awaited that disposal without a bound.
A user then reproduced `dsh -p` hanging immediately after the observation URL and ignoring repeated `Ctrl+C`; `DSH_TELEMETRY_DISABLED=1` removed the hang, while a standalone Node handler in the same Linux sandbox received SIGINT. This isolated the pending disposer to telemetry rather than terminal signal forwarding. OTel's `BatchLogRecordProcessor.shutdown()` awaits `exporter.forceFlush()` before the `exportTimeoutMillis`-bounded completion promise, and the OTLP exporter's `forceFlush()` waits directly on its in-flight HTTP Promise. A proxy/sandbox connection that never obtains a socket can therefore leave provider shutdown pending despite both configured SDK timeouts.
The latch then turned that telemetry defect into an unkillable CLI: normal completion was already awaiting the single-shot root disposal; the first SIGINT joined the same pending disposal and set the signal latch; later SIGINTs returned at the latch, so the process had no remaining escape. A signal received before normal completion had the same unbounded wait. Web used the same latch shape.
Telemetry's own timeouts cannot prove that the whole plugin tree settles. Any current or future disposer can wedge, and the process boundary must preserve both a graceful first attempt and a user-controlled way out.
## Decision
The fix has two ownership layers. The OTel backend adds `shutdownTimeoutMillis` (default and shipped value: three seconds) around the SDK provider's complete shutdown Promise. Crossing it rejects into the telemetry coordinator's existing contained-failure path, allowing the Cordis tree to finish disposal; pending records may be lost because OTel exposes no cancellation for the transport Promise.
Web and headless share `createProcessShutdown`, one process-level controller around root disposal:
- Normal shutdown calls coalesce onto one disposal and retain the first requested exit code; they never escalate one another.
- The first signal starts the same graceful disposal and a referenced five-second exit backstop. Disposal success or failure exits once; neither can cancel the process exit.
- A signal received while shutdown is pending forces immediate exit with that signal path's code. This includes the first `Ctrl+C` after headless normal completion has already entered disposal, and a second signal after a signal initiated the drain.
- The five-second bound is a process-safety invariant, not a deployment tunable. It matches the existing TUI root-disposal allowance and is long enough for the telemetry deployment's ordinary drain ceiling.
Headless preserves exit 0 for a completed turn, exit 1 for another turn-end reason or API business error, 130 for SIGINT, and 143 for SIGTERM. Web preserves its existing SIGTERM exit 0 and SIGINT exit 130 behavior.
This supersedes the [telemetry deployment Note's](../feature/2026-07-31-web-telemetry-default-mount.md) assumption that SDK exporter/processor timeouts bound complete provider shutdown, and its earlier decision to defer a process-level backstop. The backend owns its export loss/latency policy and closes the known SDK `forceFlush()` gap; the launcher owns the outer guarantee that no plugin can trap the process indefinitely.
## Alternatives considered
**Bound only the telemetry backend's `shutdown()`.** Insufficient because it protects the known OTel wait but cannot protect the launcher from another plugin's disposer.
**Restore Node's default immediate signal exit.** Rejected because a healthy first signal should still flush telemetry and release other resources. Immediate exit is the explicit escalation path, not the default.
**Add only the five-second timeout.** Rejected because a user pressing `Ctrl+C` again is asking to stop waiting now. Swallowing that intent for the rest of the grace period recreates the reported behavior at a shorter duration.
## Consequences
A healthy exit still disposes the complete Cordis tree. The known telemetry wait releases after at most three seconds; any other wedged exit lasts at most five seconds without further input, and a repeated signal ends it immediately. Forced or deadline-bounded exit can interrupt telemetry export or remaining cleanup, which is intentional only after the graceful contract has failed or the user has explicitly escalated.
The controller is launcher infrastructure rather than a Cordis plugin: it makes no claim that disposal completed, and it does not weaken the lifecycle rule that ordinary disposers must reach quiescence.
## Testing
`apps/cli/tests/process-shutdown.spec.ts` pins resolved and rejected disposal, the five-second backstop, normal-call coalescing, a signal interrupting normal disposal, and second-signal escalation.
`apps/cli/tests/headless-shutdown.e2e.ts` boots the real shipped Web/headless Loader tree in a PTY with a test-only plugin whose disposer announces entry and never settles. The test sends SIGINT after the observation URL, waits for proof that disposal started, sends SIGINT again, and requires exit 130. The source/artifact launch resolver keeps the same regression on both execution planes. This PTY case covers the user-visible process state; no model-output snapshot changes.
`packages/telemetry/session-telemetry-otel/tests/otel.spec.ts` holds a real OTLP request open after timer export begins and pins that Cordis disposal returns at `shutdownTimeoutMillis`, despite the SDK's `forceFlush()` remaining pending. The collector is then released so the still-observed provider Promise settles cleanly.

View File

@@ -0,0 +1,52 @@
# Agent NoteWeb 与 headless 的有界信号关闭和重复信号强制退出
状态:已实现
[English](2026-08-03-cli-signal-shutdown-escalation.md) | 中文
## 问题
默认挂载遥测后,`dsh web``dsh -p` 新增了 SIGINT/SIGTERM 处理器,使进程退出时可以排空 Cordis 插件树而不是丢弃排队中的遥测数据。每个处理器都使用单向布尔闩锁latch并且只有在 `ctx.fiber.dispose()` 结算后才退出。headless 正常完成时同样会无界等待整棵树执行 dispose资源释放
随后有用户复现,`dsh -p` 在打印观察 URL 后立即卡死,重复按 `Ctrl+C` 也没有反应;设置 `DSH_TELEMETRY_DISABLED=1` 后不再卡死,而同一 Linux 沙箱中的独立 Node 信号处理器能够收到 SIGINT。这将待结算的 disposer 定位到遥测而非终端信号转发。OTel 的 `BatchLogRecordProcessor.shutdown()` 会先等待 `exporter.forceFlush()`,再进入受 `exportTimeoutMillis` 限制的完成 promiseOTLP 导出器的 `forceFlush()` 则直接等待正在进行的 HTTP Promise。因此代理沙箱连接始终无法取得 socket 时,即使已经配置两项 SDK 超时,也会让提供方关闭一直待结算。
闩锁随后把这个遥测缺陷变成无法终止的 CLI命令行界面正常完成流程已经在等待单次根级 dispose第一次 SIGINT 会加入同一个待结算的 dispose并设置信号闩锁后续 SIGINT 在闩锁处直接返回因此进程再无退出途径。正常完成之前收到信号时同样会陷入无界等待。Web 使用的闩锁结构与此相同。
遥测自身的超时无法证明整棵插件树都能结算。任何当前或未来的 disposer 都可能卡死;进程边界既要保留第一次优雅关闭的机会,也必须给用户留下强制退出的途径。
## 决策
修复分为两层归属。OTel 后端围绕 SDK 提供方的完整关闭 Promise 增加 `shutdownTimeoutMillis`(默认值和交付值均为 3 秒)。超过该截止时间时会 reject并进入遥测协调器现有的失败隔离路径使 Cordis 插件树能够完成 dispose由于 OTel 未公开取消传输 Promise 的能力,待处理记录可能丢失。
Web 与 headless 共用 `createProcessShutdown`,它是围绕根级 dispose 建立的进程级控制器:
- 多次正常关闭调用会汇合到同一次 dispose并保留首次请求的退出码这些调用不会相互触发强制退出。
- 第一个信号会启动同一次优雅 dispose并设置一个带引用的 5 秒退出兜底。dispose 无论成功或失败都会触发且仅触发一次退出;任何一种结果都无法取消进程退出。
- 关闭待结算期间收到信号时,会立即按该信号路径的退出码强制退出。这既包括 headless 正常完成已经进入 dispose 后收到的第一次 `Ctrl+C`,也包括由信号启动排空后收到的第二个信号。
- 5 秒上限是进程安全不变式,而不是部署调节项。它与现有 TUI 根级 dispose 的等待上限一致,也足以覆盖遥测部署的常规排空时限。
headless 对完成的轮次仍以 0 退出,对其他轮次结束原因或 API 业务错误仍以 1 退出,对 SIGINT 以 130 退出,对 SIGTERM 以 143 退出。Web 保留现有行为SIGTERM 以 0 退出SIGINT 以 130 退出。
这项决策取代了[遥测部署 Agent Note](../feature/2026-07-31-web-telemetry-default-mount.md) 中 SDK 导出器/处理器超时能够限制提供方完整关闭流程的假设,也取代了其中暂缓进程级退出兜底的决定。后端负责导出数据丢失与延迟策略,并封住已知的 SDK `forceFlush()` 缺口;启动器负责最外层保证,确保任何插件都无法无限期困住进程。
## 考虑过的替代方案
**只限制遥测后端的 `shutdown()`。** 仍不充分:它能保护已知的 OTel 等待,但无法保护启动器免受其他插件 disposer 的影响。
**恢复 Node 默认的信号即时退出。** 不予采纳:收到第一个信号时,健康流程仍应刷新遥测数据并释放其他资源。即时退出是显式的强制退出路径,而非默认行为。
**只增加 5 秒超时。** 不予采纳:用户再次按下 `Ctrl+C`,就是要求立即停止等待。若在剩余宽限期内继续吞掉这一意图,只是缩短了报告中故障的持续时间,并未解决问题。
## 后果
健康的退出流程仍会对整棵 Cordis 插件树执行 dispose。已知的遥测等待最多会在 3 秒后解除;其他退出流程卡死时,如无进一步输入,最多等待 5 秒,再次收到信号则立即结束进程。强制退出或受截止时间限制的退出可能中断遥测导出或尚未完成的清理工作;只有优雅关闭契约已经失败,或用户明确要求强制退出时,才会有意接受这一结果。
该控制器属于启动器基础设施,而不是 Cordis 插件:它不会声称 dispose 已经完成,也不会削弱普通 disposer 必须达到完全停稳状态的生命周期规则。
## 测试
`apps/cli/tests/process-shutdown.spec.ts` 固定了 dispose 成功与失败、5 秒退出兜底、正常调用汇合、信号中断正常 dispose以及第二次信号强制退出的行为。
`apps/cli/tests/headless-shutdown.e2e.ts` 在 PTY 中启动真实交付的 Web/headless Loader 插件树,并挂载一个仅用于测试的插件;该插件的 disposer 会声明已经进入清理流程,但永不结算。测试在观察地址出现后发送 SIGINT等待 dispose 已启动的证据,再次发送 SIGINT并要求进程以 130 退出。源码/产物启动解析器使两个执行平面都覆盖同一项回归。该 PTY 用例覆盖用户可见的进程状态;模型输出快照没有变化。
`packages/telemetry/session-telemetry-otel/tests/otel.spec.ts` 在定时器导出开始后保持一条真实 OTLP 请求打开,并固定以下行为:即使 SDK 的 `forceFlush()` 仍待结算Cordis dispose 也会在 `shutdownTimeoutMillis` 到期时返回。随后测试释放 collector使仍受观察的提供方 Promise 干净结算。

View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-07-31-web-telemetry-default-mount.md
2026-07-31-web-telemetry-default-mount.md: e9ec7d0cda37db44e753c9aee572763b7e24ada6
2026-07-31-web-telemetry-default-mount.zh.md: 68b411d0668772ce81d7f323c2d286714a223ca4
2026-07-31-web-telemetry-default-mount.md: 9e7a28d52742535f86d22ba4892b2298575eae99
2026-07-31-web-telemetry-default-mount.zh.md: 20f2262ad36a6b5c8d1325526a3e6d572af12172

View File

@@ -10,7 +10,7 @@ The telemetry seam and OTel backend ([revival Note](2026-07-23-session-telemetry
## Decision
The shared `dsh` core (`apps/cli/config/base.cordis.yml`) mounts the `telemetry-otel` row by default with a baked-in production endpoint, so every surface — TUI, web, and headless — reports; this is the **internal-testing deployment stance** — reporting is on when an endpoint exists, and users opt out through the environment. Each surface's exit path drains the queue: web/headless dispose on SIGINT/SIGTERM (headless gained those handlers in this change), and the TUI's normal exit runs `disposeRootAndExit` (root dispose, 5s bounded — above the ~1s drain ceiling configured here) while its `/resume` handoff disposes the root before `execve`.
The shared `dsh` core (`apps/cli/config/base.cordis.yml`) mounts the `telemetry-otel` row by default with a baked-in production endpoint, so every surface — TUI, web, and headless — reports; this is the **internal-testing deployment stance** — reporting is on when an endpoint exists, and users opt out through the environment. Each surface's exit path drains the queue: web/headless use the [bounded, escalating process-shutdown controller](../bug-fix/2026-08-03-cli-signal-shutdown-escalation.md) on SIGINT/SIGTERM, and the TUI's normal exit runs `disposeRootAndExit` (root dispose, 5s bounded — above the backend's 3s shutdown deadline) while its `/resume` handoff disposes the root before `execve`.
| Ruling | Value | Rationale |
|---|---|---|
@@ -18,7 +18,7 @@ The shared `dsh` core (`apps/cli/config/base.cordis.yml`) mounts the `telemetry-
| Endpoint | `DSH_TELEMETRY_OTLP_URL`, default `https://harness-telemetry.deepseeksvc.com/v1/logs` | Internal collector; the env override serves local/dev runs |
| Opt-out switch | any non-empty `DSH_TELEMETRY_DISABLED` (including `0`/`false`) disables | A privacy switch prefers off-by-mistake over on-by-mistake; a row can only be disabled at AppCLIEntry's patch layer (config has no disable semantic, and the switch must precede the load-time `exporter.url` validation) |
| Cadence | `processor.scheduledDelayMillis: 10000` (10s/batch) | Streaming while the session runs, never exit-time-only; a crash loses at most the last unexported interval |
| Exit-drain bound | `exporter.timeoutMillis: 1000` + `maxExportBatchSize: 2048` (== maxQueueSize) + `exportTimeoutMillis: 1500` | Dispose must release within ~1s against an unreachable collector: timeoutMillis doubles as the per-attempt socket timeout and the retry deadline (1s effectively disables the SDK's 5-try backoff), and aligning batch size with the queue cap makes the drain a single batch; SDK defaults can stall 40s+ |
| Exit-drain bound | `exporter.timeoutMillis: 1000` + `maxExportBatchSize: 2048` (== maxQueueSize) + `exportTimeoutMillis: 1500` + `shutdownTimeoutMillis: 3000` | Ordinary unreachable-collector failure releases in ~1s: timeoutMillis is the per-attempt socket timeout and retry deadline, while one queue-sized batch avoids sequential drain multiplication. The DSH-owned 3s outer bound covers the SDK's preceding unbounded `forceFlush()` wait when the transport Promise never obtains a socket. |
| Compression | `compression: gzip` | Event bodies carry full content; cross-datacenter bandwidth |
| CI isolation | top-level `env: DSH_TELEMETRY_DISABLED: '1'` in all 8 GitHub workflows | Every CI channel that boots the web composition (e2e/snapshot/built smokes) must not stream test sessions to the production endpoint |
@@ -30,7 +30,7 @@ The keyless integration test `apps/cli/tests/telemetry-web.e2e.ts` pins the depl
**A config field instead of an env patch for the switch.** Infeasible: cordis rows have no config-level disable semantic, and `exporter.url` validation fails loud at plugin construction, so the switch must take effect before the Loader — AppCLIEntry's patch layer is the only seat.
**A `Promise.race` timeout backstop around exit.** Deferred: the parameter set already bounds the worst-case drain to ~1.5-3s (typically <100ms), measured SIGINT-to-exit 110ms-1.1s; the unbounded drip-feed-response risk stays under observation, and on real evidence the race lands inside the backend's `shutdown()` (never the coordinator — that would decide loss semantics for every backend).
**A `Promise.race` timeout backstop around exit.** Originally deferred because the SDK parameters appeared to bound the backend's drain to ~1.5-3s (typically <100ms), with measured SIGINT-to-exit of 110ms-1.1s. A Linux sandbox reproduction later proved that `BatchLogRecordProcessor.shutdown()` can wait forever in `exporter.forceFlush()` before reaching its `exportTimeoutMillis`-bounded completion Promise. The [CLI shutdown fix](../bug-fix/2026-08-03-cli-signal-shutdown-escalation.md) therefore adds both a three-second backend bound for that specific gap and a five-second process-level bound plus repeated-signal escape for the whole plugin tree.
## Consequences

View File

@@ -10,7 +10,7 @@ Status: implemented
## Decision
`dsh` 共享核心(`apps/cli/config/base.cordis.yml`)默认挂载 `telemetry-otel` 行,内置生产 endpoint因此所有 surface——TUI、web、headless——都上报这是**内部测试期的部署立场**——有 endpoint 就报,用户可经环境变量退出。各 surface 的退出路径都会排空队列web/headless 在 SIGINT/SIGTERM 上 disposeheadless 的信号处理是本次补上的)TUI 的正常退出走 `disposeRootAndExit`(根 dispose5s 兜底——高于此处配置的 ~1s drain 上界),其 `/resume` 移交也在 `execve` 前 dispose 根。
`dsh` 共享核心(`apps/cli/config/base.cordis.yml`)默认挂载 `telemetry-otel` 行,内置生产 endpoint因此所有 surface——TUI、web、headless——都上报这是**内部测试期的部署立场**——有 endpoint 就报,用户可经环境变量退出。各 surface 的退出路径都会排空队列web/headless 在 SIGINT/SIGTERM 时使用[有界、可升级的进程关闭控制器](../bug-fix/2026-08-03-cli-signal-shutdown-escalation.md)TUI 的正常退出走 `disposeRootAndExit`(根 dispose5s 兜底——高于后端的 3s 关闭截止时间),其 `/resume` 移交也在 `execve` 前 dispose 根。
| 决策项 | 取值 | 理由 |
|---|---|---|
@@ -18,7 +18,7 @@ Status: implemented
| endpoint | `DSH_TELEMETRY_OTLP_URL`,缺省 `https://harness-telemetry.deepseeksvc.com/v1/logs` | 内部 collectorenv 覆盖供本地/联调 |
| 退出开关 | `DSH_TELEMETRY_DISABLED` 非空(含 `0`/`false`)即关 | 隐私向开关取「宁关勿误开」;行级 disable 只能在 AppCLIEntry 的 patch 层做config 无 disable 语义,且必须先于 `exporter.url` 的加载期校验生效) |
| 上报节奏 | `processor.scheduledDelayMillis: 10000`10s/批) | 流式回流,非退出才报;崩溃至多丢最后一个未导出间隔 |
| 退出 drain 上界 | `exporter.timeoutMillis: 1000` + `maxExportBatchSize: 2048== maxQueueSize` + `exportTimeoutMillis: 1500` | collector 不可达时 dispose 必须 ~1s 内放行timeoutMillis 同时是单次 socket 超时与重试 deadline1s 等效关掉 SDK 5 次 backoff批大小对齐队列上限使 drain 恒为单批;默认参数下最坏可卡 40s+ |
| 退出 drain 上界 | `exporter.timeoutMillis: 1000` + `maxExportBatchSize: 2048== maxQueueSize` + `exportTimeoutMillis: 1500` + `shutdownTimeoutMillis: 3000` | collector 不可达的常规故障会在约 1s 内放行timeoutMillis 是单次 socket 超时与重试 deadline,使用与队列等大的单批可避免依次排空导致耗时倍增。由 DSH 管理的 3s 外层上限覆盖 SDK 先执行的无界 `forceFlush()` 等待,即传输 Promise 始终无法取得 socket 的情况。 |
| 压缩 | `compression: gzip` | 事件 body 含全文,跨机房带宽 |
| CI 隔离 | 全部 8 个 GitHub workflow 顶层 `env: DSH_TELEMETRY_DISABLED: '1'` | CI 启动 web 组合的所有通道e2e/snapshot/built smoke不得向生产 endpoint 泄测试会话 |
@@ -30,7 +30,7 @@ Status: implemented
**开关做成 config 字段而非 env patch。** 不可行cordis 行没有 config 层的 disable 语义,且 `exporter.url` 校验在插件构造期 fail-loud开关必须在 Loader 之前生效——AppCLIEntry patch 层是唯一落点。
**退出时 `Promise.race` 兜底超时。** 暂缓:参数组合已把最坏 drain 压到 ~1.5-3s典型 <100ms实测 SIGINT→退出 110ms-1.1sdrip-feed 慢滴响应的无界等待风险留观,出现实证再在 backend `shutdown()` 内加 race不放 coordinator——那会替所有 backend 决定丢失语义)
**退出时 `Promise.race` 兜底超时。** 最初暂缓,是因为 SDK 参数看似已经将后端排空耗时限制在约 1.5-3s通常 <100ms实测 SIGINT 到退出耗时 110ms-1.1s。后来在 Linux 沙箱中复现并证明,`BatchLogRecordProcessor.shutdown()` 可能在 `exporter.forceFlush()` 中永久等待,无法进入受 `exportTimeoutMillis` 限制的完成 Promise。因此[CLI 关闭修复](../bug-fix/2026-08-03-cli-signal-shutdown-escalation.md) 既为这一特定缺口增加 3 秒后端上限,也为整棵插件树增加 5 秒进程级上限和重复信号退出途径
## Consequences