feat(telemetry): add feedback-gated OTEL modes
This commit is contained in:
Turtle
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/telemetry/session-telemetry-otel/README.md
|
||||
README.md: 9b208e291e77bee50d9d4fd14808268dca75f2db
|
||||
README.zh.md: 76de1bf1ad58a0239907f3b63c672177874c7966
|
||||
README.md: fab2461477b2174bded42ed6f05ae55c7c5f697c
|
||||
README.zh.md: ab0191188836e03434adbce527d31b62ead848a3
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
The OpenTelemetry backend for [the telemetry seam](../session-telemetry/) — the only entry a deployment loads. It composes the OTel JS SDK as-is (`LoggerProvider` → `BatchLogRecordProcessor` → OTLP/HTTP log exporter) and maps each record the seam hands over onto `logger.emit()`, under two instrumentation scopes: ledger records on `@deepseek-ai/dsh-session-telemetry-otel`, operational records on `@deepseek-ai/dsh-session-telemetry-otel/ops`. Resource identity (`service.name`/`service.version`) comes from `dsh-llm`'s `APP_IDENTITY`, the same source the attribution headers use.
|
||||
The OpenTelemetry backend for [the telemetry seam](../session-telemetry/) — the only entry a deployment loads. Its `mode` decides whether the seam hands records over immediately, releases them only at recorded feedback, or keeps telemetry local. Uploading modes compose the OTel JS SDK as-is (`LoggerProvider` → `BatchLogRecordProcessor` → OTLP/HTTP log exporter) and map each handed-over record onto `logger.emit()`, under two instrumentation scopes: ledger records on `@deepseek-ai/dsh-session-telemetry-otel`, operational records on `@deepseek-ai/dsh-session-telemetry-otel/ops`. Resource identity (`service.name`/`service.version`) comes from `dsh-llm`'s `APP_IDENTITY`, the same source the attribution headers use.
|
||||
|
||||
## Config
|
||||
|
||||
@@ -10,6 +10,7 @@ The OpenTelemetry backend for [the telemetry seam](../session-telemetry/) — th
|
||||
- id: telemetry-otel
|
||||
name: '@deepseek-ai/dsh-session-telemetry-otel'
|
||||
config:
|
||||
mode: FULL # FULL (default), FEEDBACK_ONLY, or DISABLED
|
||||
exporter: # passed verbatim to the SDK's OTLP/HTTP log exporter
|
||||
url: https://collector.example.com/v1/logs
|
||||
headers:
|
||||
@@ -17,15 +18,21 @@ The OpenTelemetry backend for [the telemetry seam](../session-telemetry/) — th
|
||||
processor: {} # optional; passed verbatim to BatchLogRecordProcessor
|
||||
```
|
||||
|
||||
`exporter.url` is the one field this package validates itself — required, no default, must parse as `http(s)` — so a missing endpoint fails at plugin load (as does a non-positive-integer `processor.maxExportBatchSize`, which the SDK accepts but then hangs on at shutdown). Everything else is the SDK's option shape, owned and documented by the SDK, and both blocks pass through whole: every `OTLPExporterNodeConfigBase` field (`headers`, `timeoutMillis`, `compression`, `keepAlive`, …) reaches the exporter, and batching, export cadence (`scheduledDelayMillis`), retry, queue bounds, and loss policy under sustained failure are the SDK's documented behavior, tuned through the `processor` passthrough. The backend deliberately implements no `flush()`: the batch processor is the only flusher in the process, which is what makes `shutdown()`'s drain complete. Removing this block from `cordis.yml` is the opt-out: no residual state, no `enabled` flag.
|
||||
| `mode` | Behavior |
|
||||
|---|---|
|
||||
| `FULL` | Default. Each projected record, including lifecycle ops records, is handed to the OTel SDK immediately. |
|
||||
| `FEEDBACK_ONLY` | Each `feedback/record` releases the redacted, projected session prefix through that event. Later records wait for another feedback event and remain local if none arrives. |
|
||||
| `DISABLED` | No coordinator, provider, processor, or exporter is constructed. No telemetry record leaves the process. A `feedback/record` logs `session telemetry is DISABLED; nothing will be shared and this feedback remains local`; the event remains in the local session log. |
|
||||
|
||||
`exporter.url` is required in `FULL` and `FEEDBACK_ONLY`, has no default, and must parse as `http(s)`; it is optional and unused in `DISABLED`. Uploading modes also reject a non-positive-integer `processor.maxExportBatchSize`, which the SDK accepts but then hangs on at shutdown. Everything else is the SDK's option shape, owned and documented by the SDK, and both blocks pass through whole: every `OTLPExporterNodeConfigBase` field (`headers`, `timeoutMillis`, `compression`, `keepAlive`, …) reaches the exporter, and batching, export cadence (`scheduledDelayMillis`), retry, queue bounds, and loss policy under sustained failure are the SDK's documented behavior, tuned through the `processor` passthrough. The backend deliberately implements no `flush()`: the batch processor is the only flusher in the process, which is what makes `shutdown()`'s drain complete.
|
||||
|
||||
## What leaves the machine
|
||||
|
||||
Records carry the complete `event.data` as the seam's `telemetry/record` waterfall returns it — user and assistant message content, tool arguments and results (command output, file contents), the full system prompt and tool schemas (`request/header`), todo text, compaction summaries, hook `stderrSummary`, and the session `cwd` (a local path). The seam ships no redaction rules: with no `telemetry/record` listener mounted, that is the raw captured copy, so a deployment exporting beyond a trusted boundary mounts its own rules (see [the seam README](../session-telemetry/README.md#the-redact-waterfall)). Provider credentials never appear regardless: adapter API keys are constructor parameters, not session events, so they are structurally absent from the log and therefore from telemetry.
|
||||
In uploading modes, records carry the complete `event.data` as the seam's `telemetry/record` waterfall returns it — user and assistant message content, tool arguments and results (command output, file contents), the full system prompt and tool schemas (`request/header`), todo text, compaction summaries, hook `stderrSummary`, feedback text, and the session `cwd` (a local path). The seam ships no redaction rules: with no `telemetry/record` listener mounted, that is the raw captured copy, so a deployment exporting beyond a trusted boundary mounts its own rules (see [the seam README](../session-telemetry/README.md#the-redact-waterfall)). Provider credentials never appear regardless: adapter API keys are constructor parameters, not session events, so they are structurally absent from the log and therefore from telemetry. `DISABLED` does not construct the SDK pipeline or hand any capture to a backend.
|
||||
|
||||
## Field mapping
|
||||
|
||||
Seam record → SDK log record: `time` → `timestamp`/`observedTimestamp`; `severity` → `severityNumber`/`severityText` (INFO 9 / WARN 13 / ERROR 17); `body` → the structured log body; `attributes` verbatim. Receivers dedupe on `(session.id, event.seq)`, alert on severity, and detect crashes by `shutdown`-record absence (a session with activity, no `shutdown` ops record, gone stale ended uncleanly). The marker means telemetry stopped observing the session cleanly — emitted at the session's own disposal, or at application teardown for sessions still running then; a marker followed by more of that session's events is a telemetry reload, not a session restart. Streams are not self-contained across lineage: a resumed session continues its own id's stream from where the previous process left off, and a forked session's stream starts at its inherited boundary — its prefix lives in the parent's stream, stitched via `session.parent_id` + `session.seed_length`. One consequence of continuing rather than replaying: a turn left open mid-stream and never closed marks the previous process dying inside it. The local log is repaired with synthetic closers at resume, but those repairs are never exported — the wire stream stays faithful to what the crashed process actually shipped, and a later clean `shutdown` marker attests only to the resumed process's own exit.
|
||||
Seam record → SDK log record: `time` → `timestamp`/`observedTimestamp`; `severity` → `severityNumber`/`severityText` (INFO 9 / WARN 13 / ERROR 17); `body` → the structured log body; `attributes` verbatim. Receivers dedupe on `(session.id, event.seq)` and alert on severity. In `FULL`, they may also detect crashes by `shutdown`-record absence: the marker is emitted at the session's own disposal or application teardown, and a marker followed by more events is a telemetry reload. In `FEEDBACK_ONLY`, a released prefix normally has no later `shutdown` marker, so its absence is not a crash signal. Streams are not self-contained across lineage: a resumed session continues its own id's stream from where the previous process left off, and a forked session's stream starts at its inherited boundary — its prefix lives in the parent's stream, stitched via `session.parent_id` + `session.seed_length`. A resumed local log may contain synthetic closers that were never exported; the wire stream stays faithful to records actually handed to the SDK.
|
||||
|
||||
## Model Experience
|
||||
|
||||
@@ -39,3 +46,4 @@ None; this package neither assembles nor sends a provider request.
|
||||
|
||||
- **Upstream experimental tree** — `@opentelemetry/sdk-logs` is still published from the upstream experimental tree; SDK API churn lands here and only here — the seam contract does not move.
|
||||
- **No live-collector coverage** — every test exports to a local mock collector; the keyless Loader-composition e2e (`tests/loader-composition.e2e.ts`) covers the wire shape on every run, and behavior against a real OTLP deployment (auth, TLS, throttling) is the SDK exporter's documented territory.
|
||||
- **Feedback-only memory** — each session retains deep-copied, redacted projected records in memory until feedback releases them or the session becomes unreachable. There is no durable pre-feedback spool; a crash before feedback uploads nothing.
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
[遥测（telemetry）seam](../session-telemetry/) 的 OpenTelemetry 后端，也是部署方唯一要加载的条目。它原样组合 OTel JS SDK（`LoggerProvider` → `BatchLogRecordProcessor` → OTLP/HTTP 日志导出器），把 seam 交接过来的每条记录映射到 `logger.emit()`，并使用两个插桩作用域（instrumentation scope）：ledger 记录挂在 `@deepseek-ai/dsh-session-telemetry-otel` 下，运维记录挂在 `@deepseek-ai/dsh-session-telemetry-otel/ops` 下。资源身份（`service.name`/`service.version`）来自 `dsh-llm` 的 `APP_IDENTITY`，与归因标头同源。
|
||||
[遥测（telemetry）seam](../session-telemetry/) 的 OpenTelemetry 后端，也是部署方唯一要加载的条目。其 `mode` 决定 seam 是立即交接记录、仅在记录反馈时释放记录，还是将遥测留在本地。上传模式会原样组合 OTel JS SDK（`LoggerProvider` → `BatchLogRecordProcessor` → OTLP/HTTP 日志导出器），把每条已交接记录映射到 `logger.emit()`，并使用两个插桩作用域（instrumentation scope）：ledger 记录挂在 `@deepseek-ai/dsh-session-telemetry-otel` 下，运维记录挂在 `@deepseek-ai/dsh-session-telemetry-otel/ops` 下。资源身份（`service.name`/`service.version`）来自 `dsh-llm` 的 `APP_IDENTITY`，与归因标头同源。
|
||||
|
||||
## 配置
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
- id: telemetry-otel
|
||||
name: '@deepseek-ai/dsh-session-telemetry-otel'
|
||||
config:
|
||||
mode: FULL # FULL (default), FEEDBACK_ONLY, or DISABLED
|
||||
exporter: # passed verbatim to the SDK's OTLP/HTTP log exporter
|
||||
url: https://collector.example.com/v1/logs
|
||||
headers:
|
||||
@@ -17,15 +18,21 @@
|
||||
processor: {} # optional; passed verbatim to BatchLogRecordProcessor
|
||||
```
|
||||
|
||||
`exporter.url` 是本包（package）唯一自行校验的字段：必填、无默认值、必须能解析为 `http(s)`，因此缺失端点会在插件加载时失败（`processor.maxExportBatchSize` 不是正整数时同样如此：SDK 会接受该值，随后却在关闭时因它挂起）。其余全部是 SDK 自己的选项形态，由 SDK 拥有并在 SDK 文档中说明，两个配置块都整体透传（passthrough）：`OTLPExporterNodeConfigBase` 的每个字段（`headers`、`timeoutMillis`、`compression`、`keepAlive` 等）都会到达导出器；批处理、导出节奏（`scheduledDelayMillis`）、重试、队列上限，以及持续失败下的丢失策略，都是 SDK 的文档化行为，经 `processor` 透传调优。该后端刻意不实现 `flush()`：批处理器是进程内唯一执行 flush 的组件，`shutdown()` 的排空正因如此才是完整的。从 `cordis.yml` 中删除该配置块即为退出方式：无残留状态，也没有 `enabled` 开关。
|
||||
| `mode` | 行为 |
|
||||
|---|---|
|
||||
| `FULL` | 默认值。每条已投影记录都立即交给 OTel SDK，包括生命周期运维记录。 |
|
||||
| `FEEDBACK_ONLY` | 每个 `feedback/record` 都会释放截至该事件的已脱敏、已投影会话前缀。后续记录等待下一个反馈事件；如果没有后续反馈，则留在本地。 |
|
||||
| `DISABLED` | 不构造协调器、提供方、处理器或导出器。没有遥测记录会离开进程。`feedback/record` 会记录 `session telemetry is DISABLED; nothing will be shared and this feedback remains local`；该事件留在本地会话日志中。 |
|
||||
|
||||
`exporter.url` 在 `FULL` 与 `FEEDBACK_ONLY` 中必填，无默认值，且必须能解析为 `http(s)`；在 `DISABLED` 中可省略且不使用。上传模式也会拒绝不是正整数的 `processor.maxExportBatchSize`，SDK 虽会接受该值，但随后会在关闭时挂起。其余全部是 SDK 自己的选项形态，由 SDK 拥有并在 SDK 文档中说明，两个配置块都整体透传（passthrough）：`OTLPExporterNodeConfigBase` 的每个字段（`headers`、`timeoutMillis`、`compression`、`keepAlive` 等）都会到达导出器；批处理、导出节奏（`scheduledDelayMillis`）、重试、队列上限，以及持续失败下的丢失策略，都是 SDK 的文档化行为，经 `processor` 透传调优。该后端刻意不实现 `flush()`：批处理器是进程内唯一执行 flush 的组件，`shutdown()` 的排空正因如此才是完整的。
|
||||
|
||||
## 哪些数据会离开本机
|
||||
|
||||
记录携带完整的 `event.data`，内容以 seam 的 `telemetry/record` waterfall（瀑布式事件）返回的结果为准：用户与 assistant 消息内容、工具参数与工具结果（命令输出、文件内容）、完整的系统提示词与工具 schema（`request/header`）、todo 文本、压缩（compaction）摘要、钩子的 `stderrSummary`，以及会话 `cwd`（一个本地路径）。seam 不带任何脱敏规则：未挂载 `telemetry/record` 监听器时，导出的就是捕获原样的副本，因此向可信边界之外导出的部署方要挂载自己的规则（见 [seam README](../session-telemetry/README.md#the-redact-waterfall)）。无论如何，提供方凭据都不会出现：适配器的 API key 是构造函数参数而非会话事件，因此它们在结构上就不存在于日志中，也就不存在于遥测中。
|
||||
在上传模式中，记录携带完整的 `event.data`，内容以 seam 的 `telemetry/record` waterfall（瀑布式事件）返回的结果为准：用户与 assistant 消息内容、工具参数与工具结果（命令输出、文件内容）、完整的系统提示词与工具 schema（`request/header`）、todo 文本、压缩（compaction）摘要、钩子的 `stderrSummary`、反馈文本，以及会话 `cwd`（一个本地路径）。seam 不带任何脱敏规则：未挂载 `telemetry/record` 监听器时，导出的就是捕获原样的副本，因此向可信边界之外导出的部署方要挂载自己的规则（见 [seam README](../session-telemetry/README.md#the-redact-waterfall)）。无论如何，提供方凭据都不会出现：适配器的 API key 是构造函数参数而非会话事件，因此它们在结构上就不存在于日志中，也就不存在于遥测中。`DISABLED` 不会构造 SDK 流水线，也不会将任何捕获内容交给后端。
|
||||
|
||||
## 字段映射
|
||||
|
||||
seam 记录 → SDK 日志记录：`time` → `timestamp`/`observedTimestamp`；`severity` → `severityNumber`/`severityText`（INFO 9 / WARN 13 / ERROR 17）；`body` → 结构化日志 body；`attributes` 原样照搬。接收端基于 `(session.id, event.seq)` 去重、按严重级别告警，并通过 `shutdown` 记录的缺失检测崩溃（一个曾有活动、没有 `shutdown` 运维记录、且已然陈旧的会话，就是未干净结束的会话）。该标记的含义是遥测干净地停止了对该会话的观察：它在会话自身 dispose（资源释放）时发出，对于届时仍在运行的会话，则在应用关闭时发出；标记之后又出现该会话的更多事件，说明发生的是遥测重载，而不是会话重启。跨谱系（lineage）的流并不自足：恢复的会话在其自身 id 的流上从上一个进程停止之处继续；fork 出的会话，其流从继承边界开始，前缀位于父会话的流中，由接收端基于 `session.parent_id` + `session.seed_length` 拼接。继续而非回放的一个后果：流中一个开启后再未关闭的轮次，标志着上一个进程死在了该轮次之内。恢复时本地日志会以合成的关闭事件修复，但这些修复绝不导出：导出的流忠实于崩溃进程实际发出的内容，其后干净的 `shutdown` 标记也只证明恢复后进程自身的退出。
|
||||
seam 记录 → SDK 日志记录：`time` → `timestamp`/`observedTimestamp`；`severity` → `severityNumber`/`severityText`（INFO 9 / WARN 13 / ERROR 17）；`body` → 结构化日志 body；`attributes` 原样照搬。接收端基于 `(session.id, event.seq)` 去重，并按严重级别告警。在 `FULL` 中，接收端还可通过缺少 `shutdown` 记录检测崩溃：该标记在会话自身 dispose（资源释放）或应用关闭时发出；标记之后出现更多事件，说明遥测发生了重载。在 `FEEDBACK_ONLY` 中，已释放的前缀通常不包含随后的 `shutdown` 标记，因此缺少该标记不是崩溃信号。跨谱系（lineage）的流并不自足：恢复的会话在其自身 id 的流上从上一个进程停止之处继续；fork 出的会话的流从继承边界开始，其前缀位于父会话的流中，由接收端基于 `session.parent_id` + `session.seed_length` 拼接。恢复后的本地日志可能包含从未导出的合成关闭事件；协议流忠实于实际交给 SDK 的记录。
|
||||
|
||||
## 模型体验
|
||||
|
||||
@@ -39,3 +46,4 @@ seam 记录 → SDK 日志记录：`time` → `timestamp`/`observedTimestamp`；
|
||||
|
||||
- **上游实验性源码树**：`@opentelemetry/sdk-logs` 仍从上游实验性（experimental）源码树发布；SDK API 的变动只会落在本包，也仅落在本包；seam 契约不动。
|
||||
- **无真实 collector 覆盖**：所有测试都导出到本地 mock collector；无密钥的 Loader 组合 e2e（`tests/loader-composition.e2e.ts`）在每次运行中都覆盖协议格式（wire format）形态，而面对真实 OTLP 部署的行为（认证、TLS、限流）属于 SDK 导出器文档的职责范围。
|
||||
- **仅反馈模式的内存占用**：每个会话都会在内存中保留已深拷贝、已脱敏的投影记录，直到反馈将其释放或会话变得不可达。反馈前不存在持久化 spool；如果在反馈前崩溃，则什么都不上传。
|
||||
|
||||
@@ -36,6 +36,7 @@
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-command-feedback": "^0.0.1",
|
||||
"@deepseek-ai/dsh-invariants": "^0.0.1",
|
||||
"@deepseek-ai/dsh-llm": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session": "^0.0.1",
|
||||
@@ -44,6 +45,7 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@cordisjs/plugin-loader": "workspace:^",
|
||||
"@deepseek-ai/dsh-command-feedback": "workspace:^",
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
|
||||
@@ -6,8 +6,8 @@
|
||||
* record handed over by the seam onto `logger.emit()`. Per the seam's
|
||||
* boundary axiom, everything downstream of that call (batching, retry,
|
||||
* queueing, loss policy) is the SDK's documented behavior, configured
|
||||
* verbatim through the `exporter`/`processor` passthroughs; this package
|
||||
* adds no knobs of its own on top of them.
|
||||
* verbatim through the `exporter`/`processor` passthroughs. This package owns
|
||||
* only whether capture is immediate, feedback-released, or disabled.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-session-telemetry-otel
|
||||
*/
|
||||
@@ -15,7 +15,14 @@
|
||||
import { createRequire } from 'node:module'
|
||||
import z from 'schemastery'
|
||||
import type { Context } from 'cordis'
|
||||
import { Telemetry, TelemetryCoordinator, type TelemetryRecord, type TelemetrySeverity } from '@deepseek-ai/dsh-session-telemetry'
|
||||
import type {} from '@deepseek-ai/dsh-command-feedback'
|
||||
import {
|
||||
Telemetry,
|
||||
TelemetryCoordinator,
|
||||
type TelemetryDelivery,
|
||||
type TelemetryRecord,
|
||||
type TelemetrySeverity,
|
||||
} from '@deepseek-ai/dsh-session-telemetry'
|
||||
import { APP_IDENTITY } from '@deepseek-ai/dsh-llm'
|
||||
import {
|
||||
BatchLogRecordProcessor,
|
||||
@@ -31,13 +38,22 @@ import { resourceFromAttributes } from '@opentelemetry/resources'
|
||||
// version (same pattern as dsh-llm's attribution identity).
|
||||
const { version } = createRequire(import.meta.url)('../package.json') as { version: string }
|
||||
|
||||
/** Supported session-sharing policies for the OTel backend. */
|
||||
export const TELEMETRY_MODES = ['FULL', 'FEEDBACK_ONLY', 'DISABLED'] as const
|
||||
|
||||
/** Session-sharing policy selected by {@link Config.mode}. */
|
||||
export type TelemetryMode = typeof TELEMETRY_MODES[number]
|
||||
|
||||
const DISABLED_FEEDBACK_WARNING = 'session telemetry is DISABLED; nothing will be shared and this feedback remains local'
|
||||
|
||||
/**
|
||||
* Plugin configuration: two verbatim SDK option shapes plus nothing else.
|
||||
* `exporter.url` is the one field this package validates itself — required,
|
||||
* no default, must parse as an `http(s)` URL — because a missing endpoint
|
||||
* must fail at plugin load, not at first export.
|
||||
* Plugin configuration: one sharing policy plus two verbatim SDK option
|
||||
* shapes. `exporter.url` is required for modes that upload and unused for
|
||||
* `DISABLED`.
|
||||
*/
|
||||
export interface Config {
|
||||
/** Sharing policy; defaults to immediate `FULL` delivery. */
|
||||
mode?: TelemetryMode
|
||||
/**
|
||||
* Passed verbatim to the SDK's OTLP/HTTP log exporter — the complete
|
||||
* `OTLPExporterNodeConfigBase` shape (`headers`, `timeoutMillis`,
|
||||
@@ -45,7 +61,7 @@ export interface Config {
|
||||
* is the one field this package requires and validates itself.
|
||||
*/
|
||||
exporter?: OTLPExporterNodeConfigBase & {
|
||||
/** Full logs endpoint (e.g. `https://collector.example.com/v1/logs`). Required; validated at plugin load. */
|
||||
/** Full logs endpoint (e.g. `https://collector.example.com/v1/logs`). Required outside `DISABLED`; validated at load. */
|
||||
url?: string
|
||||
}
|
||||
/**
|
||||
@@ -57,13 +73,14 @@ export interface Config {
|
||||
|
||||
/**
|
||||
* Schemastery validator for {@link Config}; cordis runs it before the plugin
|
||||
* starts. Shape-level only — the load-bearing `exporter.url` check lives in
|
||||
* the constructor so its error message names the field. Both slots are opaque
|
||||
* passthroughs: the SDK owns their shapes and validates its own options;
|
||||
* re-declaring them field-by-field here would violate the boundary axiom
|
||||
* (and silently drop every field not re-declared).
|
||||
* starts. Shape-level only — the mode-dependent `exporter.url` check lives in
|
||||
* the constructor so its error message names the field. Both SDK slots are
|
||||
* opaque passthroughs: the SDK owns their shapes and validates its own
|
||||
* options; re-declaring them field-by-field here would violate the boundary
|
||||
* axiom (and silently drop every field not re-declared).
|
||||
*/
|
||||
export const Config: z<Config> = z.object({
|
||||
mode: z.union(TELEMETRY_MODES).default('FULL'),
|
||||
exporter: z.any(),
|
||||
processor: z.any(),
|
||||
})
|
||||
@@ -76,22 +93,32 @@ const SEVERITY: Record<TelemetrySeverity, { severityNumber: SeverityNumber; seve
|
||||
}
|
||||
|
||||
/**
|
||||
* The backend plugin — the only entry a deployment loads. Constructing it
|
||||
* wires the SDK pipeline, registers the `telemetry` service (duplicate load
|
||||
* throws, cordis' standard duplicate-service behavior), and composes the
|
||||
* seam's {@link TelemetryCoordinator}, which installs the capture side onto
|
||||
* this fiber.
|
||||
* The backend plugin — the only entry a deployment loads. It always registers
|
||||
* the `telemetry` service (duplicate load throws). Uploading modes wire the SDK
|
||||
* pipeline and compose {@link TelemetryCoordinator}; `DISABLED` constructs no
|
||||
* SDK state and listens only to warn when recorded feedback stays local.
|
||||
*/
|
||||
export class TelemetryOtel extends Telemetry {
|
||||
static inject = ['sessions']
|
||||
static Config = Config
|
||||
|
||||
private readonly provider: LoggerProvider
|
||||
private readonly ledger: Logger
|
||||
private readonly ops: Logger
|
||||
private readonly provider: LoggerProvider | undefined
|
||||
private readonly ledger: Logger | undefined
|
||||
private readonly ops: Logger | undefined
|
||||
|
||||
constructor(ctx: Context, config: Config) {
|
||||
super(ctx)
|
||||
const mode = config.mode ?? 'FULL'
|
||||
if (mode === 'DISABLED') {
|
||||
this.provider = undefined
|
||||
this.ledger = undefined
|
||||
this.ops = undefined
|
||||
ctx.on('session/event', (_session, event) => {
|
||||
if (event.type === 'feedback/record') ctx.logger.warn(DISABLED_FEEDBACK_WARNING)
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
const url = config.exporter?.url
|
||||
if (url === undefined || url.length === 0) {
|
||||
throw new Error('session-telemetry-otel: exporter.url is required (the full OTLP logs endpoint)')
|
||||
@@ -134,16 +161,26 @@ export class TelemetryOtel extends Telemetry {
|
||||
})
|
||||
this.ledger = this.provider.getLogger('@deepseek-ai/dsh-session-telemetry-otel', version)
|
||||
this.ops = this.provider.getLogger('@deepseek-ai/dsh-session-telemetry-otel/ops', version)
|
||||
new TelemetryCoordinator(ctx, this)
|
||||
const delivery: TelemetryDelivery = mode === 'FULL' ? 'immediate' : 'held'
|
||||
const coordinator = new TelemetryCoordinator(ctx, this, delivery)
|
||||
if (mode === 'FEEDBACK_ONLY') {
|
||||
// The coordinator listener is registered first, so a feedback event
|
||||
// enters the held prefix before this listener releases that exact prefix.
|
||||
ctx.on('session/event', (session, event) => {
|
||||
if (event.type === 'feedback/record') coordinator.release(session)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Map one seam record onto the SDK logger for its channel — a synchronous
|
||||
* enqueue into the batch processor's queue.
|
||||
* enqueue into the batch processor's queue. Direct calls are no-ops in
|
||||
* `DISABLED`, where no coordinator or SDK pipeline exists.
|
||||
* @param record - the logical record handed over by the coordinator.
|
||||
*/
|
||||
emit(record: TelemetryRecord): void {
|
||||
const logger = record.channel === 'ops' ? this.ops : this.ledger
|
||||
if (logger === undefined) return
|
||||
logger.emit({
|
||||
timestamp: record.time,
|
||||
observedTimestamp: record.time,
|
||||
@@ -167,14 +204,15 @@ export class TelemetryOtel extends Telemetry {
|
||||
/**
|
||||
* Delegate disposal to the SDK's shutdown contract: drain the queue and
|
||||
* quiesce. With no concurrent `forceFlush()` in the process (see above),
|
||||
* shutdown's internal drain is complete — everything emitted before this
|
||||
* call, including the coordinator's dispose-time `shutdown` markers, is
|
||||
* exported before the exporter closes. Awaited (and error-contained) by
|
||||
* the coordinator's disposer.
|
||||
* shutdown's internal drain is complete — everything handed to the SDK
|
||||
* before this call is exported before the exporter closes. In `FULL`, that
|
||||
* includes dispose-time `shutdown` markers; held suffixes never reach the
|
||||
* SDK. Awaited (and error-contained) by the coordinator's disposer. A
|
||||
* disabled backend resolves immediately.
|
||||
* @returns resolves when the SDK pipeline has quiesced.
|
||||
*/
|
||||
shutdown(): Promise<void> {
|
||||
return this.provider.shutdown()
|
||||
return this.provider === undefined ? Promise.resolve() : this.provider.shutdown()
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -15,10 +15,9 @@ export const name = 'session-telemetry-otel-invariant'
|
||||
export const inject = ['invariants']
|
||||
|
||||
/**
|
||||
* No runtime invariant: the backend forwards seam records into the OTel SDK's
|
||||
* in-process pipeline and appends nothing to any session; its only observable
|
||||
* effects (batching, export) happen inside the SDK past the seam's boundary
|
||||
* axiom, out of reach of an independent companion.
|
||||
* No runtime invariant: mode selection changes capture handoff, SDK setup, and
|
||||
* local diagnostics without mutating session or service state an independent
|
||||
* companion can compare. Export remains inside the SDK past the seam boundary.
|
||||
*/
|
||||
const install: InvariantInstaller = () => {}
|
||||
|
||||
|
||||
@@ -40,6 +40,11 @@ interface OtlpCapture {
|
||||
}[]
|
||||
}
|
||||
|
||||
interface FixtureOutput {
|
||||
captures: OtlpCapture[]
|
||||
logContent: string
|
||||
}
|
||||
|
||||
async function jsonlFiles(dir: string): Promise<string[]> {
|
||||
const entries = await readdir(dir, { withFileTypes: true })
|
||||
const paths = await Promise.all(entries.map(async (entry) => {
|
||||
@@ -50,10 +55,29 @@ async function jsonlFiles(dir: string): Promise<string[]> {
|
||||
return paths.flat()
|
||||
}
|
||||
|
||||
async function readFixtureOutput(cwd: string): Promise<FixtureOutput> {
|
||||
const captures = JSON.parse(await readFile(join(cwd, 'otlp-captures.json'), 'utf8')) as OtlpCapture[]
|
||||
const logs = await jsonlFiles(join(cwd, '.sessions'))
|
||||
expect(logs).toHaveLength(1)
|
||||
return { captures, logContent: await readFile(logs[0] as string, 'utf8') }
|
||||
}
|
||||
|
||||
function allRecords(captures: OtlpCapture[]) {
|
||||
return captures.flatMap(capture => capture.resourceLogs.flatMap(resource =>
|
||||
resource.scopeLogs.flatMap(scoped => scoped.logRecords.map(record => ({ scope: scoped.scope.name, record })))))
|
||||
}
|
||||
|
||||
function eventTypes(captures: OtlpCapture[]): string[] {
|
||||
return allRecords(captures).flatMap(({ record }) =>
|
||||
record.attributes?.flatMap(attribute =>
|
||||
attribute.key === 'event.type' && typeof attribute.value['stringValue'] === 'string'
|
||||
? [attribute.value['stringValue']]
|
||||
: []) ?? [])
|
||||
}
|
||||
|
||||
describe('session-telemetry-otel through a real headless cordis.yml', () => {
|
||||
it('exports redacted ledger records to the collector while the canonical log keeps the secret', async () => {
|
||||
let captures: OtlpCapture[] = []
|
||||
let logContent = ''
|
||||
let output!: FixtureOutput
|
||||
const { stderr } = await runLoaderSmoke({
|
||||
label: 'session-telemetry-otel loader smoke',
|
||||
tempDirPrefix: 'telemetry-otel-e2e-',
|
||||
@@ -61,39 +85,70 @@ describe('session-telemetry-otel through a real headless cordis.yml', () => {
|
||||
libBinScript: driver,
|
||||
configPath,
|
||||
tsconfigPath: repoTsconfig,
|
||||
inspect: async (cwd) => {
|
||||
captures = JSON.parse(await readFile(join(cwd, 'otlp-captures.json'), 'utf8')) as OtlpCapture[]
|
||||
const logs = await jsonlFiles(join(cwd, '.sessions'))
|
||||
expect(logs).toHaveLength(1)
|
||||
logContent = await readFile(logs[0] as string, 'utf8')
|
||||
},
|
||||
inspect: async (cwd) => { output = await readFixtureOutput(cwd) },
|
||||
})
|
||||
expect(stderr).not.toContain('UNHANDLED')
|
||||
|
||||
const records = captures.flatMap(capture => capture.resourceLogs.flatMap(resource =>
|
||||
resource.scopeLogs.flatMap(scoped => scoped.logRecords.map(record => ({ scope: scoped.scope.name, record })))))
|
||||
const records = allRecords(output.captures)
|
||||
expect(records.length).toBeGreaterThan(0)
|
||||
|
||||
const eventTypes = records.flatMap(({ record }) =>
|
||||
record.attributes?.flatMap(attribute =>
|
||||
attribute.key === 'event.type' && typeof attribute.value['stringValue'] === 'string'
|
||||
? [attribute.value['stringValue']]
|
||||
: []) ?? [])
|
||||
const types = eventTypes(output.captures)
|
||||
for (const expected of ['turn/start', 'user/message', 'tool/call', 'tool/result', 'assistant/message', 'turn/end']) {
|
||||
expect(eventTypes, expected).toContain(expected)
|
||||
expect(types, expected).toContain(expected)
|
||||
}
|
||||
expect(records.some(({ scope }) => scope.endsWith('/ops'))).toBe(true)
|
||||
|
||||
// The deployment-mounted rule on the wire: the fixture credential never
|
||||
// leaves the process, its surrounding prose does, and the placeholder
|
||||
// marks the spot — the seam itself ships no rules.
|
||||
const wire = JSON.stringify(captures)
|
||||
const wire = JSON.stringify(output.captures)
|
||||
expect(wire).not.toContain(FIXTURE_SECRET)
|
||||
expect(wire).toContain(FIXTURE_PLACEHOLDER)
|
||||
expect(wire).toContain('prove telemetry with key')
|
||||
|
||||
// The canonical session log is never rewritten.
|
||||
expect(logContent).toContain(FIXTURE_SECRET)
|
||||
expect(logContent).not.toContain(FIXTURE_PLACEHOLDER)
|
||||
expect(output.logContent).toContain(FIXTURE_SECRET)
|
||||
expect(output.logContent).not.toContain(FIXTURE_PLACEHOLDER)
|
||||
}, LOADER_SMOKE_TEST_TIMEOUT_MS)
|
||||
|
||||
it('exports only prefixes ending in feedback under feedback-only mode', async () => {
|
||||
let output!: FixtureOutput
|
||||
const { stderr } = await runLoaderSmoke({
|
||||
label: 'session-telemetry-otel feedback-only loader smoke',
|
||||
tempDirPrefix: 'telemetry-otel-feedback-e2e-',
|
||||
binScript: driver,
|
||||
libBinScript: driver,
|
||||
configPath,
|
||||
tsconfigPath: repoTsconfig,
|
||||
env: { DSH_TELEMETRY_E2E_MODE: 'FEEDBACK_ONLY' },
|
||||
inspect: async (cwd) => { output = await readFixtureOutput(cwd) },
|
||||
})
|
||||
expect(stderr).not.toContain('UNHANDLED')
|
||||
|
||||
const wire = JSON.stringify(output.captures)
|
||||
expect(eventTypes(output.captures)).toContain('feedback/record')
|
||||
expect(wire).toContain('fixture feedback')
|
||||
expect(wire).toContain('prove telemetry with key')
|
||||
expect(wire).not.toContain('post-feedback private suffix')
|
||||
expect(output.logContent).toContain('post-feedback private suffix')
|
||||
}, LOADER_SMOKE_TEST_TIMEOUT_MS)
|
||||
|
||||
it('keeps disabled feedback local and prints the stable warning', async () => {
|
||||
let output!: FixtureOutput
|
||||
const { stdout } = await runLoaderSmoke({
|
||||
label: 'session-telemetry-otel disabled loader smoke',
|
||||
tempDirPrefix: 'telemetry-otel-disabled-e2e-',
|
||||
binScript: driver,
|
||||
libBinScript: driver,
|
||||
configPath,
|
||||
tsconfigPath: repoTsconfig,
|
||||
env: { DSH_TELEMETRY_E2E_MODE: 'DISABLED' },
|
||||
inspect: async (cwd) => { output = await readFixtureOutput(cwd) },
|
||||
})
|
||||
|
||||
expect(output.captures).toEqual([])
|
||||
expect(output.logContent).toContain('fixture feedback')
|
||||
expect(stdout.match(/session telemetry is DISABLED; nothing will be shared and this feedback remains local/)?.[0])
|
||||
.toMatchInlineSnapshot('"session telemetry is DISABLED; nothing will be shared and this feedback remains local"')
|
||||
}, LOADER_SMOKE_TEST_TIMEOUT_MS)
|
||||
})
|
||||
|
||||
@@ -5,12 +5,13 @@
|
||||
* for the default-exported Service class.
|
||||
*/
|
||||
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { createServer, type Server } from 'node:http'
|
||||
import { once } from 'node:events'
|
||||
import { gunzipSync } from 'node:zlib'
|
||||
import { Context } from 'cordis'
|
||||
import Loader from '@cordisjs/plugin-loader'
|
||||
import { recordFeedback } from '@deepseek-ai/dsh-command-feedback'
|
||||
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import TelemetryOtel, { Config } from '../src/index.ts'
|
||||
|
||||
@@ -30,6 +31,7 @@ interface OtlpLogsRequest {
|
||||
severityNumber: number
|
||||
severityText: string
|
||||
attributes?: { key: string; value: Record<string, unknown> }[]
|
||||
body?: unknown
|
||||
}[]
|
||||
}[]
|
||||
}[]
|
||||
@@ -88,6 +90,14 @@ function allRecords(captures: Capture[]) {
|
||||
s.logRecords.map(record => ({ scope: s.scope.name, record })))))
|
||||
}
|
||||
|
||||
function eventTypes(captures: Capture[]): string[] {
|
||||
return allRecords(captures).flatMap(({ record }) =>
|
||||
record.attributes?.flatMap(attribute =>
|
||||
attribute.key === 'event.type' && typeof attribute.value['stringValue'] === 'string'
|
||||
? [attribute.value['stringValue']]
|
||||
: []) ?? [])
|
||||
}
|
||||
|
||||
describe('TelemetryOtel wire', () => {
|
||||
it('ships session records and the ops shutdown marker through the real SDK pipeline', async () => {
|
||||
const { url, captures } = await mockCollector()
|
||||
@@ -195,6 +205,82 @@ describe('TelemetryOtel wire', () => {
|
||||
r.record.attributes?.some(a => a.key === 'event.type' && a.value.stringValue === 'turn/start'))
|
||||
expect(start?.record.severityNumber).toBe(13)
|
||||
})
|
||||
|
||||
it('holds each session suffix until the next feedback event', async () => {
|
||||
const { url, captures } = await mockCollector()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(TelemetryOtel, {
|
||||
mode: 'FEEDBACK_ONLY',
|
||||
exporter: { url },
|
||||
})
|
||||
const session = ctx.sessions.create(SessionId('feedback-only'), { meta: {} })
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
recordFeedback(session, 'first report')
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
recordFeedback(session, 'second report')
|
||||
session.append('turn/start', { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
await fiber.dispose()
|
||||
|
||||
const types = allRecords(captures).flatMap(({ record }) =>
|
||||
record.attributes?.flatMap(attribute =>
|
||||
attribute.key === 'event.type' ? [attribute.value.stringValue] : []) ?? [])
|
||||
expect(types).toEqual(['turn/start', 'feedback/record', 'turn/end', 'feedback/record'])
|
||||
expect(JSON.stringify(captures)).toContain('first report')
|
||||
expect(JSON.stringify(captures)).toContain('second report')
|
||||
expect(allRecords(captures).some(({ scope }) => scope.endsWith('/ops'))).toBe(false)
|
||||
})
|
||||
|
||||
it('sends no request when feedback-only mode ends without feedback', async () => {
|
||||
const { url, captures } = await mockCollector()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(TelemetryOtel, {
|
||||
mode: 'FEEDBACK_ONLY',
|
||||
exporter: { url },
|
||||
})
|
||||
const session = ctx.sessions.create(SessionId('no-feedback'), { meta: {} })
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
await fiber.dispose()
|
||||
expect(captures).toEqual([])
|
||||
})
|
||||
|
||||
it('boots disabled without exporter config and warns when feedback stays local', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const warn = vi.spyOn(ctx.logger, 'warn').mockImplementation(() => {})
|
||||
const fiber = await ctx.plugin(TelemetryOtel, { mode: 'DISABLED' })
|
||||
const session = ctx.sessions.create(SessionId('disabled'), { meta: {} })
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
recordFeedback(session, 'local report')
|
||||
|
||||
expect(warn).toHaveBeenCalledWith(
|
||||
'session telemetry is DISABLED; nothing will be shared and this feedback remains local',
|
||||
)
|
||||
ctx.telemetry.emit({
|
||||
channel: 'ledger',
|
||||
time: 0,
|
||||
severity: 'info',
|
||||
attributes: {},
|
||||
body: null,
|
||||
})
|
||||
await ctx.telemetry.shutdown()
|
||||
await fiber.dispose()
|
||||
recordFeedback(session, 'after disposal')
|
||||
expect(warn).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('defaults direct construction to full delivery', async () => {
|
||||
const { url, captures } = await mockCollector()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
new TelemetryOtel(ctx, { exporter: { url } })
|
||||
const session = ctx.sessions.create(SessionId('direct-default'), { meta: {} })
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
await ctx.fiber.dispose()
|
||||
|
||||
expect(eventTypes(captures)).toContain('turn/start')
|
||||
})
|
||||
})
|
||||
|
||||
describe('TelemetryOtel config fails loud', () => {
|
||||
@@ -203,6 +289,8 @@ describe('TelemetryOtel config fails loud', () => {
|
||||
[{ exporter: { url: '' } }, /exporter\.url is required/],
|
||||
[{ exporter: { url: 'not a url' } }, /not a valid URL/],
|
||||
[{ exporter: { url: 'ftp://collector' } }, /must be http\(s\)/],
|
||||
[{ mode: 'FEEDBACK_ONLY' }, /exporter\.url is required/],
|
||||
[{ mode: 'INVALID' }, /INVALID/],
|
||||
// The SDK accepts a non-positive batch size but its shutdown drain then
|
||||
// splices empty batches forever — dispose would hang, so reject at load.
|
||||
[{ exporter: { url: 'http://c/v1/logs' }, processor: { maxExportBatchSize: 0 } }, /maxExportBatchSize/],
|
||||
|
||||
@@ -20,6 +20,9 @@
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
{
|
||||
"path": "../../feedback/command-feedback"
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm"
|
||||
},
|
||||
|
||||
Reference in New Issue
Block a user