Merge remote-tracking branch 'origin/worktree/ci-native-windows-20260808' into worktree/ci-native-windows-coverage-20260808
This commit is contained in:
Tianyi Cui
6
packages/session/README.i18n.yaml
Normal file
6
packages/session/README.i18n.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/session/README.md
|
||||
README.md: 586d1be0286a0de935b0b08313e6965452b85376
|
||||
README.zh.md: 60e58e6d471a48d3ced03a518e96aeeae79110e8
|
||||
51
packages/session/README.md
Normal file
51
packages/session/README.md
Normal file
@@ -0,0 +1,51 @@
|
||||
# session/ — durable session data plane
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
The durable family around `core/session`'s live in-memory service: the persistence seam with its storage backends and checkpoint policy, the projection seam that serves whole log-derived values, log-backed titles, and outbound session telemetry. All **product** packages. `session-query/` remains a sibling group: the read/tool surface is consumed independently of persistence internals.
|
||||
|
||||
## Persistence
|
||||
|
||||
Durable session persistence, semantic checkpoint policy, and the shipped storage backends.
|
||||
|
||||
| Package | Role | ctx key |
|
||||
|---|---|---|
|
||||
| [`session-persistence/`](session-persistence/README.md) | Defines the persistence service and shared write coordination | `ctx.sessionPersistence` |
|
||||
| [`session-checkpoint-policy/`](session-checkpoint-policy/README.md) | Applies semantic durability checkpoints | wraps `ctx.llm` and `ctx.tools` |
|
||||
| [`session-persistence-jsonl/`](session-persistence-jsonl/README.md) | Persists sessions in JSONL files | registers on `ctx.sessionPersistence` |
|
||||
| [`session-persistence-sqlite/`](session-persistence-sqlite/README.md) | Persists sessions in SQLite | registers on `ctx.sessionPersistence` |
|
||||
|
||||
The [session-persistence decision](../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md) records the persistence design.
|
||||
|
||||
## Projection
|
||||
|
||||
Serves current, log-derived per-session state to client carriers.
|
||||
|
||||
| Package | Role | ctx key |
|
||||
|---|---|---|
|
||||
| [`session-projection/`](session-projection/README.md) | Defines and drives session projection units | `ctx.sessionProjections` |
|
||||
| [`session-projection-cache/`](session-projection-cache/README.md) | Persists and restores projection checkpoints | `ctx.sessionProjectionCache` |
|
||||
|
||||
## Titles
|
||||
|
||||
Derives durable session titles from the session log, with an optional model-backed provider.
|
||||
|
||||
| Package | Role | ctx key |
|
||||
|---|---|---|
|
||||
| [`session-title/`](session-title/README.md) | Owns title state, fallback behavior, provider registration, and refresh | `ctx.sessionTitle` |
|
||||
| [`session-title-llm/`](session-title-llm/README.md) | Provides shared model-backed title generation | — |
|
||||
| [`session-title-first-message-llm/`](session-title-first-message-llm/README.md) | Titles a session from its first eligible human message | registers on `ctx.sessionTitle` |
|
||||
| [`session-title-all-messages-llm/`](session-title-all-messages-llm/README.md) | Titles a session from all eligible human messages | registers on `ctx.sessionTitle` |
|
||||
|
||||
Deployments may register one model-backed provider; the service retains a deterministic fallback when none is present.
|
||||
|
||||
## Telemetry
|
||||
|
||||
Projects session activity into outbound telemetry and delegates delivery to a configured reporting backend. The [telemetry decision](../../.agents/notes/implemented/feature/2026-07-23-session-telemetry-otel-revival.md) records the reporting boundary; the [mode decision](../../.agents/notes/implemented/feature/2026-08-05-feedback-gated-session-telemetry.md) records immediate, feedback-gated, and disabled delivery.
|
||||
|
||||
| Package | Role |
|
||||
|---|---|
|
||||
| [`session-telemetry/`](session-telemetry/README.md) | Defines capture, redaction, projection, and live or on-demand backend delivery. |
|
||||
| [`session-telemetry-otel/`](session-telemetry-otel/README.md) | Delivers telemetry through OpenTelemetry logs in `FULL`, `FEEDBACK_ONLY`, or `DISABLED` mode. |
|
||||
|
||||
The subsystem references: [persistence.md](../../docs/subsystems/persistence.md), [session-projection.md](../../docs/subsystems/session-projection.md), [session-title.md](../../docs/subsystems/session-title.md), and [telemetry.md](../../docs/subsystems/telemetry.md). Only one title provider may register at a time; the demo spine mounts the fallback service and leaves both model providers out of default composition.
|
||||
51
packages/session/README.zh.md
Normal file
51
packages/session/README.zh.md
Normal file
@@ -0,0 +1,51 @@
|
||||
# session/：持久会话数据平面
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
围绕 `core/session` 常驻内存实时服务的持久家族：持久化 seam 连同其存储后端与检查点策略、供出日志派生全量值的投影 seam、日志支持的标题，以及外发会话遥测。全部都是**产品**包（package）。`session-query/` 仍是同级独立组：读取／工具面的消费不依赖持久化内部实现。
|
||||
|
||||
## 持久化
|
||||
|
||||
持久会话数据的持久化机制、语义检查点策略以及随产品交付的存储后端。
|
||||
|
||||
| 包 | 职责 | ctx 键 |
|
||||
|---|---|---|
|
||||
| [`session-persistence/`](session-persistence/README.md) | 定义持久化服务和共享写入协调机制 | `ctx.sessionPersistence` |
|
||||
| [`session-checkpoint-policy/`](session-checkpoint-policy/README.md) | 应用语义持久性检查点 | 包装 `ctx.llm` 和 `ctx.tools` |
|
||||
| [`session-persistence-jsonl/`](session-persistence-jsonl/README.md) | 将会话持久化到 JSONL 文件 | 注册到 `ctx.sessionPersistence` |
|
||||
| [`session-persistence-sqlite/`](session-persistence-sqlite/README.md) | 将会话持久化到 SQLite | 注册到 `ctx.sessionPersistence` |
|
||||
|
||||
[会话持久化决策](../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md)记录了持久化设计。
|
||||
|
||||
## 投影
|
||||
|
||||
向客户端载体提供从日志派生的当前逐会话状态。
|
||||
|
||||
| 包 | 职责 | ctx 键 |
|
||||
|---|---|---|
|
||||
| [`session-projection/`](session-projection/README.md) | 定义并驱动会话投影单元 | `ctx.sessionProjections` |
|
||||
| [`session-projection-cache/`](session-projection-cache/README.md) | 持久化并恢复投影检查点 | `ctx.sessionProjectionCache` |
|
||||
|
||||
## 标题
|
||||
|
||||
从会话日志派生持久会话标题，并支持可选的模型后端 provider。
|
||||
|
||||
| 包 | 职责 | ctx 键 |
|
||||
|---|---|---|
|
||||
| [`session-title/`](session-title/README.md) | 负责标题状态、回退行为、provider 注册与刷新 | `ctx.sessionTitle` |
|
||||
| [`session-title-llm/`](session-title-llm/README.md) | 提供共享的模型标题生成能力 | — |
|
||||
| [`session-title-first-message-llm/`](session-title-first-message-llm/README.md) | 根据第一条合格的人类消息生成会话标题 | 注册到 `ctx.sessionTitle` |
|
||||
| [`session-title-all-messages-llm/`](session-title-all-messages-llm/README.md) | 根据所有合格的人类消息生成会话标题 | 注册到 `ctx.sessionTitle` |
|
||||
|
||||
部署可注册一个模型后端 provider；未注册时，服务仍提供确定性回退。
|
||||
|
||||
## 遥测
|
||||
|
||||
将会话活动投影为外发遥测，并将投递委派给配置的上报后端。[遥测决策](../../.agents/notes/implemented/feature/2026-07-23-session-telemetry-otel-revival.md)记录上报边界；[模式决策](../../.agents/notes/implemented/feature/2026-08-05-feedback-gated-session-telemetry.md)记录即时、反馈门控与禁用投递。
|
||||
|
||||
| 包 | 职责 |
|
||||
|---|---|
|
||||
| [`session-telemetry/`](session-telemetry/README.md) | 定义捕获、脱敏、投影，以及实时或按需后端投递。 |
|
||||
| [`session-telemetry-otel/`](session-telemetry-otel/README.md) | 通过 OpenTelemetry 日志以 `FULL`、`FEEDBACK_ONLY` 或 `DISABLED` 模式投递遥测。 |
|
||||
|
||||
子系统参考：[persistence.md](../../docs/subsystems/persistence.md)、[session-projection.md](../../docs/subsystems/session-projection.md)、[session-title.md](../../docs/subsystems/session-title.md) 与 [telemetry.md](../../docs/subsystems/telemetry.md)。同一时间只允许一个标题提供方注册；demo 主干挂载回退服务，两个模型提供方都留在默认组合之外。
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/session-persistence/session-checkpoint-policy/README.md
|
||||
README.md: 01ed3c694967b1a86245d01b9d9f7eecd1193348
|
||||
README.zh.md: e2c197c0cc1d942a045c1ea22c1fb257f6e4059d
|
||||
45
packages/session/session-checkpoint-policy/README.md
Normal file
45
packages/session/session-checkpoint-policy/README.md
Normal file
@@ -0,0 +1,45 @@
|
||||
# dsh-session-checkpoint-policy
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
Semantic durability policy for persisted agents. It checkpoints the event-sourced session before a model adapter receives a request, before a top-level tool body may produce an external side effect, and at each `agent/pre-step` boundary so the preceding response and ordered tool results are durable before the next request.
|
||||
|
||||
## Plugin (namespace: `session-checkpoint-policy`)
|
||||
|
||||
This zero-config function plugin consumes `ctx.sessions`, `ctx.llm`, `ctx.tools`, and the presence of `ctx.sessionPersistence`. Load it beside one persistence backend:
|
||||
|
||||
```yaml
|
||||
- id: session-persistence
|
||||
name: '@deepseek-ai/dsh-session-persistence-jsonl'
|
||||
|
||||
- id: session-checkpoints
|
||||
name: '@deepseek-ai/dsh-session-checkpoint-policy'
|
||||
```
|
||||
|
||||
Persistence and checkpoint scheduling are intentionally separate Cordis plugins. A persistence backend starts bounded background batches for `session/event` appends and makes each requested `session/flush` an immediate quiescence barrier; this policy chooses the request, tool-dispatch, and next-step barriers. Loading a backend without this policy is valid, but a crash may lose events still inside the configured batching window or an outstanding write. First-party persisted apps and runtimes mount both plugins explicitly; a specialized deployment may deliberately omit or replace the policy.
|
||||
|
||||
The policy wraps `llm/stream` lazily, so the downstream stream is not constructed until the live session's buffered request events are durable. It wraps `tools/execute` after pre-execute policy and guards; a top-level tool body runs only after its recorded call is durable. If cancellation lands while that flush is pending, the wrapper returns the canonical `ABORTED_BEFORE_DISPATCH` result without entering the tool body. Nested tool dispatches reuse the outer model-visible call's checkpoint. `agent/pre-step` persists the preceding response/result batch before request derivation.
|
||||
|
||||
Checkpoint rejection is fail-closed at the model and tool boundaries: neither the adapter nor the top-level tool body runs. A step-boundary rejection fails the turn before another request starts. Concurrent tool checkpoints share the session store's serialized persistence drain and cannot duplicate sequence numbers.
|
||||
|
||||
## Model Experience
|
||||
|
||||
### Interrupted calls
|
||||
|
||||
#### What the model sees
|
||||
|
||||
The plugin adds no prompt or tool schema. A hard crash after a tool checkpoint but before its result leaves a durable unmatched call; session recovery supplies the model-visible `TOOL_OUTCOME_UNKNOWN` result owned by `dsh-session`. The message permits retry for read-only or idempotent work and requires state verification or user confirmation for calls that may have side effects.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Successful checkpoints add no tokens and do not change the request. Recovery adds one short tool-result message to balance the interrupted transcript.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
The repair result is appended after the reusable prefix, so it does not invalidate earlier cache entries.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- The policy durably records execution intent, not generic exactly-once effects. Side-effecting tools should forward `exec.callId` as an idempotency key when their provider supports one.
|
||||
- Streaming `assistant/chunk` events have no per-chunk checkpoint. Bounded background batches normally persist them before the next semantic checkpoint, but a hard crash may lose the current in-memory batch or outstanding write.
|
||||
- A persisted call without a result cannot prove whether its external effect completed. Recovery therefore records an unknown outcome instead of retrying automatically.
|
||||
45
packages/session/session-checkpoint-policy/README.zh.md
Normal file
45
packages/session/session-checkpoint-policy/README.zh.md
Normal file
@@ -0,0 +1,45 @@
|
||||
# dsh-session-checkpoint-policy
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
已持久化的 agent（智能体）的语义持久性策略。它会在模型适配器收到请求前、顶层工具正文可产生外部副作用前，以及每个 `agent/pre-step` 边界为事件溯源会话创建检查点，使前一响应与有序工具结果在下一个请求前已持久化。
|
||||
|
||||
## 插件（命名空间：`session-checkpoint-policy`）
|
||||
|
||||
该零配置函数插件消费 `ctx.sessions`、`ctx.llm`、`ctx.tools` 以及 `ctx.sessionPersistence` 的存在性。将其与一个持久化后端一起加载：
|
||||
|
||||
```yaml
|
||||
- id: session-persistence
|
||||
name: '@deepseek-ai/dsh-session-persistence-jsonl'
|
||||
|
||||
- id: session-checkpoints
|
||||
name: '@deepseek-ai/dsh-session-checkpoint-policy'
|
||||
```
|
||||
|
||||
持久化与检查点调度刻意拆分为独立 Cordis 插件。持久化后端会为追加的 `session/event` 启动有界后台批次，并把每个已请求的 `session/flush` 变成即时完全停稳屏障；该策略选择请求、工具分派和下一步骤屏障。不带此策略加载后端是有效的，但崩溃可能丢失仍位于已配置批处理窗口内的事件，或尚未完成的写入。第一方持久化应用和运行时显式挂载两个插件；专用部署可以刻意省略或替换策略。
|
||||
|
||||
策略延迟包装 `llm/stream`，因此下游流只会在活动会话中缓冲的请求事件已持久化后构造。它在预执行策略和防护机制之后包装 `tools/execute`；只有在已记录调用已持久化后，顶层工具正文才会运行。如果取消在 flush 等待期间到达，包装层会返回规范的 `ABORTED_BEFORE_DISPATCH` 结果，不进入工具正文。嵌套工具分派重用外层模型可见调用的检查点。`agent/pre-step` 在派生请求前持久化前一响应/结果批次。
|
||||
|
||||
在模型和工具边界，检查点被拒绝时会按失败即阻止原则处理：适配器和顶层工具正文都不运行。步骤边界处的检查点被拒绝会在另一个请求开始前使轮次失败。并发工具检查点共享会话存储的串行持久化排空流程，不会产生重复的序列号。
|
||||
|
||||
## 模型体验
|
||||
|
||||
### 中断调用
|
||||
|
||||
#### 模型看到的内容
|
||||
|
||||
插件不添加提示词或工具 schema。工具检查点后、结果前的硬崩溃会留下持久的未匹配调用；会话恢复会提供模型可见的 `TOOL_OUTCOME_UNKNOWN` 结果，该结果由 `dsh-session` 负责。该消息允许重试只读或幂等工作，并要求对可能有副作用的调用验证状态或请求用户确认。
|
||||
|
||||
#### Token 影响
|
||||
|
||||
成功检查点不添加 token，也不改变请求。恢复会添加一条短工具结果消息，以平衡中断的 transcript（文本记录）。
|
||||
|
||||
#### KV Cache 影响
|
||||
|
||||
修复结果追加在可重用前缀之后，因此不会使较早的缓存条目失效。
|
||||
|
||||
## 已知限制与暂缓事项
|
||||
|
||||
- 该策略以持久方式记录执行意图，而非为通用副作用提供恰好一次保证。当提供方支持时，有副作用的工具应将 `exec.callId` 作为幂等键转发。
|
||||
- 流式 `assistant/chunk` 事件没有逐分片检查点。有界后台批次通常会在下一个语义检查点之前将其持久化，但硬崩溃可能丢失当前内存批次或尚未完成的写入。
|
||||
- 已持久化的调用没有结果时，无法证明其外部副作用是否完成。因此，恢复会记录未知结果，而不是自动重试。
|
||||
50
packages/session/session-checkpoint-policy/package.json
Normal file
50
packages/session/session-checkpoint-policy/package.json
Normal file
@@ -0,0 +1,50 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-session-checkpoint-policy",
|
||||
"description": "Semantic session durability checkpoints before model requests and tool side effects",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./invariant": {
|
||||
"types": "./lib/types/invariant.d.ts",
|
||||
"default": "./lib/invariant.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-agent": "^0.0.1",
|
||||
"@deepseek-ai/dsh-invariants": "^0.0.1",
|
||||
"@deepseek-ai/dsh-llm": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session-persistence": "^0.0.1",
|
||||
"@deepseek-ai/dsh-tools": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@cordisjs/plugin-loader": "workspace:^",
|
||||
"@deepseek-ai/dsh-agent": "workspace:^",
|
||||
"@deepseek-ai/dsh-agent-loop": "workspace:^",
|
||||
"@deepseek-ai/dsh-agent-loop-testkit": "workspace:^",
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-persistence": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-persistence-jsonl": "workspace:^",
|
||||
"@deepseek-ai/dsh-system-prompt": "workspace:^",
|
||||
"@deepseek-ai/dsh-tools": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
83
packages/session/session-checkpoint-policy/src/index.ts
Normal file
83
packages/session/session-checkpoint-policy/src/index.ts
Normal file
@@ -0,0 +1,83 @@
|
||||
/**
|
||||
* Semantic durability checkpoints for model requests, top-level tool dispatch,
|
||||
* and completed agent steps.
|
||||
* @module @deepseek-ai/dsh-session-checkpoint-policy
|
||||
*/
|
||||
|
||||
import type { Context } from 'cordis'
|
||||
import type { Session } from '@deepseek-ai/dsh-session'
|
||||
import type { StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import { TOOL_ABORTED_BEFORE_DISPATCH, type ToolExecutionResult } from '@deepseek-ai/dsh-tools'
|
||||
import type { PreStepDecision } from '@deepseek-ai/dsh-agent'
|
||||
import type {} from '@deepseek-ai/dsh-session-persistence'
|
||||
|
||||
/** Cordis plugin name used by Loader diagnostics. */
|
||||
export const name = 'session-checkpoint-policy'
|
||||
|
||||
/** Services whose request, tool, session, and persistence boundaries this policy joins. */
|
||||
export const inject = ['llm', 'sessionPersistence', 'sessions', 'tools']
|
||||
|
||||
/**
|
||||
* Delay construction of the downstream model stream until the complete logged
|
||||
* request prefix is durable. A checkpoint rejection prevents adapter dispatch.
|
||||
*
|
||||
* @param ctx - plugin context that owns the session store.
|
||||
* @param session - live session named by the model request.
|
||||
* @param next - downstream `llm/stream` chain.
|
||||
* @returns a stream that checkpoints before requesting its first chunk.
|
||||
*/
|
||||
function afterCheckpoint(
|
||||
ctx: Context,
|
||||
session: Session,
|
||||
next: () => AsyncIterable<StreamChunk>,
|
||||
): AsyncIterable<StreamChunk> {
|
||||
return (async function* (): AsyncIterable<StreamChunk> {
|
||||
await ctx.sessions.flush(session)
|
||||
yield* next()
|
||||
})()
|
||||
}
|
||||
|
||||
/** Materialize the canonical result for a call cancelled before tool dispatch. */
|
||||
function abortedBeforeDispatchResult(): ToolExecutionResult {
|
||||
return {
|
||||
content: [{ type: 'text', text: 'Error: tool call aborted before dispatch' }],
|
||||
isError: true,
|
||||
error: {
|
||||
message: 'tool call aborted before dispatch',
|
||||
info: { name: 'AbortError', code: TOOL_ABORTED_BEFORE_DISPATCH },
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Install semantic checkpoint listeners. Loop-built model calls checkpoint the
|
||||
* logged request before adapter dispatch; top-level tool calls checkpoint their
|
||||
* recorded call before the tool body; the next request boundary checkpoints
|
||||
* the preceding response/result batch. Nested tool dispatches reuse the durable outer call.
|
||||
*
|
||||
* Checkpoint failures are fail-closed at the model and tool side-effect
|
||||
* boundaries: the downstream adapter or tool body is not invoked.
|
||||
*
|
||||
* @param ctx - plugin context that owns the listeners.
|
||||
*/
|
||||
export function apply(ctx: Context): void {
|
||||
ctx.on('llm/stream', (options, next): AsyncIterable<StreamChunk> => {
|
||||
if (options.sessionId === undefined) return next()
|
||||
const session = ctx.sessions.get(options.sessionId)
|
||||
return session === undefined ? next() : afterCheckpoint(ctx, session, next)
|
||||
})
|
||||
|
||||
ctx.on('tools/execute', async (exec, next): Promise<ToolExecutionResult> => {
|
||||
if (exec.agent === undefined || exec.parent !== undefined) return next()
|
||||
await ctx.sessions.flush(exec.agent.session)
|
||||
if (exec.signal.aborted) return abortedBeforeDispatchResult()
|
||||
return next()
|
||||
})
|
||||
|
||||
// Before each request, persist everything committed by the preceding step;
|
||||
// the first step's call is an intentional no-op beyond any prompt intake.
|
||||
ctx.on('agent/pre-step', async ({ agent }, next): Promise<PreStepDecision> => {
|
||||
await ctx.sessions.flush(agent.session)
|
||||
return next()
|
||||
})
|
||||
}
|
||||
30
packages/session/session-checkpoint-policy/src/invariant.ts
Normal file
30
packages/session/session-checkpoint-policy/src/invariant.ts
Normal file
@@ -0,0 +1,30 @@
|
||||
/**
|
||||
* Package-owned invariant companion for `@deepseek-ai/dsh-session-checkpoint-policy`.
|
||||
* @module @deepseek-ai/dsh-session-checkpoint-policy/invariant
|
||||
*/
|
||||
|
||||
/* jscpd:ignore-start */
|
||||
import type { Context } from 'cordis'
|
||||
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
|
||||
|
||||
const PACKAGE_NAME = '@deepseek-ai/dsh-session-checkpoint-policy'
|
||||
|
||||
/** Cordis companion plugin name. */
|
||||
export const name = 'session-checkpoint-policy-invariant'
|
||||
/** Service required before the companion can reserve package ownership. */
|
||||
export const inject = ['invariants']
|
||||
|
||||
/**
|
||||
* No runtime invariant: checkpoint ordering is enforced at the intercepted waterfall and
|
||||
* persistence seams; this stateless policy owns no independent mutable relation.
|
||||
*/
|
||||
const install: InvariantInstaller = () => {}
|
||||
|
||||
/**
|
||||
* Register this package's invariant companion.
|
||||
* @param ctx - Cordis context carrying the invariant service.
|
||||
* @returns the installed registration's disposer after setup succeeds.
|
||||
*/
|
||||
export const apply = (ctx: Context): Promise<() => void> =>
|
||||
Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
|
||||
/* jscpd:ignore-end */
|
||||
@@ -0,0 +1,111 @@
|
||||
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { execa } from 'execa'
|
||||
import { Context } from 'cordis'
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import SessionStore, {
|
||||
SessionId, TOOL_OUTCOME_UNKNOWN,
|
||||
type SessionEvent,
|
||||
} from '@deepseek-ai/dsh-session'
|
||||
import SessionPersistenceJsonl from '@deepseek-ai/dsh-session-persistence-jsonl'
|
||||
|
||||
const repoRoot = fileURLToPath(new URL('../../../../', import.meta.url))
|
||||
const childScript = fileURLToPath(new URL('./fixtures/crash-child.ts', import.meta.url))
|
||||
const tsxLoader = fileURLToPath(import.meta.resolve('tsx'))
|
||||
const sessionId = SessionId('semantic-checkpoint-crash')
|
||||
const roots: string[] = []
|
||||
const CHILD_FAILPOINT_TIMEOUT_MS = 30_000
|
||||
|
||||
async function waitForMarker(path: string, expected: string): Promise<string> {
|
||||
// vi.waitFor retries every callback throw, so terminal states RESOLVE out
|
||||
// of the retry loop (complete marker, or content that can no longer become
|
||||
// the expected marker) and only the still-in-progress states throw-to-retry.
|
||||
const content = await vi.waitFor(async () => {
|
||||
const current = await readFile(path, 'utf8').catch((error: unknown) => {
|
||||
if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error
|
||||
throw new Error(`crash child did not publish failpoint ${JSON.stringify(expected)} at ${path}`, { cause: error })
|
||||
})
|
||||
if (current === expected || !expected.startsWith(current)) return current
|
||||
throw new Error(`crash child has not finished publishing failpoint ${JSON.stringify(expected)}`)
|
||||
}, { interval: 10, timeout: CHILD_FAILPOINT_TIMEOUT_MS })
|
||||
if (content !== expected) {
|
||||
throw new Error(`crash child wrote unexpected failpoint ${JSON.stringify(content)}`)
|
||||
}
|
||||
return content
|
||||
}
|
||||
|
||||
async function crashAt(mode: 'request' | 'tool'): Promise<{ root: string; markerText: string }> {
|
||||
const root = await mkdtemp(join(tmpdir(), `dsh-semantic-${mode}-`))
|
||||
roots.push(root)
|
||||
const marker = join(root, 'failpoint')
|
||||
// Keep the open-before-write window deterministic: readiness is marker content, not path existence.
|
||||
await writeFile(marker, '')
|
||||
const expectedMarker = mode === 'request' ? 'request-dispatched' : 'tool-side-effect'
|
||||
// The SIGKILL-at-failpoint choreography stays custom: the child must die
|
||||
// mid-write, so no timeout or graceful termination may reach it first.
|
||||
const child = execa(process.execPath, ['--import', tsxLoader, childScript, mode, root, marker], {
|
||||
cwd: repoRoot,
|
||||
env: { TSX_TSCONFIG_PATH: join(repoRoot, 'tsconfig.json') },
|
||||
stdin: 'ignore',
|
||||
stdout: 'ignore',
|
||||
reject: false,
|
||||
})
|
||||
try {
|
||||
const markerText = await waitForMarker(marker, expectedMarker)
|
||||
child.kill('SIGKILL')
|
||||
const exit = await child
|
||||
expect({ code: exit.exitCode ?? null, signal: exit.signal ?? null }).toEqual({ code: null, signal: 'SIGKILL' })
|
||||
return { root, markerText }
|
||||
} catch (error: unknown) {
|
||||
child.kill('SIGKILL')
|
||||
throw new Error(`crash child failed: ${(await child).stderr}`, { cause: error })
|
||||
}
|
||||
}
|
||||
|
||||
async function load(root: string): Promise<SessionEvent[]> {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SessionPersistenceJsonl, { root, compression: 'none' })
|
||||
try {
|
||||
return [...(await ctx.sessionPersistence.load(sessionId)).events]
|
||||
} finally {
|
||||
await ctx.fiber.dispose()
|
||||
}
|
||||
}
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true })))
|
||||
})
|
||||
|
||||
describe.skipIf(process.platform === 'win32')('semantic checkpoint hard-crash recovery', () => {
|
||||
it('persists the complete request before model dispatch', async () => {
|
||||
const crashed = await crashAt('request')
|
||||
expect(crashed.markerText).toBe('request-dispatched')
|
||||
const events = await load(crashed.root)
|
||||
expect(events.map(event => event.type)).toEqual([
|
||||
'agent/inbox/spliced', 'turn/start', 'agent/inbox/spliced',
|
||||
'step/start', 'user/message', 'request/header', 'request/context', 'step/end', 'turn/end',
|
||||
])
|
||||
expect(events.at(-1)).toMatchObject({
|
||||
type: 'turn/end', data: { reason: { kind: 'interrupted' } },
|
||||
})
|
||||
})
|
||||
|
||||
it('persists tool intent before a side effect and repairs its missing result as unknown', async () => {
|
||||
const crashed = await crashAt('tool')
|
||||
expect(crashed.markerText).toBe('tool-side-effect')
|
||||
const events = await load(crashed.root)
|
||||
expect(events.some(event => event.type === 'assistant/message')).toBe(true)
|
||||
expect(events.some(event => event.type === 'tool/call')).toBe(true)
|
||||
const result = events.find(event => event.type === 'tool/result')
|
||||
expect(result?.type === 'tool/result' && result.data.error).toEqual({
|
||||
name: 'ToolOutcomeUnknownError', code: TOOL_OUTCOME_UNKNOWN,
|
||||
})
|
||||
if (result?.type !== 'tool/result' || result.data.message.content[0].content[0]?.type !== 'text') {
|
||||
throw new Error('expected a text tool result')
|
||||
}
|
||||
expect(result.data.message.content[0].content[0].text).toContain('Do not retry blindly.')
|
||||
})
|
||||
})
|
||||
60
packages/session/session-checkpoint-policy/tests/fixtures/crash-child.ts
vendored
Normal file
60
packages/session/session-checkpoint-policy/tests/fixtures/crash-child.ts
vendored
Normal file
@@ -0,0 +1,60 @@
|
||||
import { writeFile } from 'node:fs/promises'
|
||||
import { Context } from 'cordis'
|
||||
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
|
||||
import { mountAgentLoopTestDependencies } from '@deepseek-ai/dsh-agent-loop-testkit'
|
||||
import { createUserMessage, CallId, type GenerateOptions, LlmAdapter, type StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import SessionPersistenceJsonl from '@deepseek-ai/dsh-session-persistence-jsonl'
|
||||
import * as checkpointPolicy from '../../src/index.ts'
|
||||
|
||||
function waitForCrash(): Promise<never> {
|
||||
return new Promise(() => { setInterval(() => {}, 60_000) })
|
||||
}
|
||||
|
||||
const [mode, root, marker] = process.argv.slice(2)
|
||||
if ((mode !== 'request' && mode !== 'tool') || root === undefined || marker === undefined) {
|
||||
throw new Error('usage: crash-child.ts <request|tool> <persistence-root> <marker>')
|
||||
}
|
||||
const persistenceRoot = root
|
||||
const failpoint = marker
|
||||
|
||||
class CrashAdapter extends LlmAdapter {
|
||||
async * stream(_options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
if (mode === 'request') {
|
||||
await writeFile(failpoint, 'request-dispatched')
|
||||
await waitForCrash()
|
||||
return
|
||||
}
|
||||
yield { type: 'block-start', index: 0, blockType: 'tool-call' }
|
||||
yield {
|
||||
type: 'block-end',
|
||||
index: 0,
|
||||
block: { type: 'tool-call', id: CallId('crash-call'), name: 'crash_tool', arguments: '{}' },
|
||||
}
|
||||
yield { type: 'finish', reason: { kind: 'tool-calls' } }
|
||||
}
|
||||
}
|
||||
|
||||
const ctx = new Context()
|
||||
await mountAgentLoopTestDependencies(ctx)
|
||||
await ctx.plugin(AgentLoop, { agents: [] })
|
||||
await ctx.plugin(SessionPersistenceJsonl, { root: persistenceRoot, compression: 'none' })
|
||||
await ctx.plugin(checkpointPolicy)
|
||||
ctx.llm.registerAdapter(['crash'], new CrashAdapter())
|
||||
ctx.tools.register({
|
||||
name: 'crash_tool',
|
||||
description: 'records an external effect and never returns',
|
||||
parameters: {},
|
||||
output: { schema: { type: 'null' }, render: () => [] },
|
||||
async execute() {
|
||||
await writeFile(failpoint, 'tool-side-effect')
|
||||
return waitForCrash()
|
||||
},
|
||||
})
|
||||
|
||||
const handle = await ctx.agents.create({
|
||||
sessionId: SessionId('semantic-checkpoint-crash'),
|
||||
agentOptions: { provider: 'crash', model: 'crash' },
|
||||
})
|
||||
handle.agent.followup(createUserMessage({ content: [{ type: 'text', text: 'exercise the crash boundary' }], source: { kind: 'user' } }))
|
||||
await waitForCrash()
|
||||
@@ -0,0 +1,268 @@
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import Loader from '@cordisjs/plugin-loader'
|
||||
import { agentEvents, type Agent } from '@deepseek-ai/dsh-agent'
|
||||
import LlmService, { CallId, type GenerateOptions, LlmAdapter, type StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionEvent, SessionHeader } from '@deepseek-ai/dsh-session'
|
||||
import SessionPersistence from '@deepseek-ai/dsh-session-persistence'
|
||||
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
|
||||
import ToolRegistry, { TOOL_ABORTED_BEFORE_DISPATCH } from '@deepseek-ai/dsh-tools'
|
||||
import * as checkpointPolicy from '../src/index.ts'
|
||||
|
||||
const contexts: Context[] = []
|
||||
|
||||
class TestPersistence extends SessionPersistence {
|
||||
locate(_meta: SessionHeader): undefined { return undefined }
|
||||
create(_meta: SessionHeader): Promise<void> { return Promise.resolve() }
|
||||
append(_id: SessionId, _events: readonly SessionEvent[]): Promise<void> { return Promise.resolve() }
|
||||
load(_id: SessionId): Promise<{ meta: SessionHeader; events: SessionEvent[] }> {
|
||||
return Promise.reject(new Error('not used'))
|
||||
}
|
||||
inspect(_id: SessionId): Promise<{ meta: SessionHeader; events: SessionEvent[] }> {
|
||||
return Promise.reject(new Error('not used'))
|
||||
}
|
||||
readFrom(_id: SessionId, _fromSeq: number): Promise<{ meta: SessionHeader; events: SessionEvent[] }> {
|
||||
return Promise.reject(new Error('not used'))
|
||||
}
|
||||
list(): Promise<SessionHeader[]> { return Promise.resolve([]) }
|
||||
listSnapshots(): Promise<never[]> { return Promise.resolve([]) }
|
||||
}
|
||||
|
||||
class RecordingAdapter extends LlmAdapter {
|
||||
constructor(private readonly order: string[]) { super() }
|
||||
async * stream(_options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
this.order.push('adapter')
|
||||
yield { type: 'finish', reason: { kind: 'stop' } }
|
||||
}
|
||||
}
|
||||
|
||||
async function setup(): Promise<Context> {
|
||||
const ctx = new Context()
|
||||
contexts.push(ctx)
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(SystemPrompt)
|
||||
await ctx.plugin(ToolRegistry)
|
||||
await ctx.plugin(TestPersistence)
|
||||
await ctx.plugin(checkpointPolicy)
|
||||
return ctx
|
||||
}
|
||||
|
||||
async function drain(stream: AsyncIterable<StreamChunk>): Promise<void> {
|
||||
for await (const _chunk of stream) { /* drain */ }
|
||||
}
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(contexts.splice(0).map(ctx => ctx.fiber.dispose()))
|
||||
})
|
||||
|
||||
describe('session-checkpoint-policy request boundary', () => {
|
||||
it('awaits the live session checkpoint before constructing the downstream model stream', async () => {
|
||||
const ctx = await setup()
|
||||
const session = ctx.sessions.create(SessionId('request-checkpoint'))
|
||||
session.append('turn/start', { turn: 1 })
|
||||
const gate = Promise.withResolvers<undefined>()
|
||||
const order: string[] = []
|
||||
ctx.on('session/flush', async () => {
|
||||
order.push('flush:start')
|
||||
await gate.promise
|
||||
order.push('flush:end')
|
||||
})
|
||||
ctx.llm.registerAdapter(['mock'], new RecordingAdapter(order))
|
||||
|
||||
const pending = drain(ctx.llm.stream({
|
||||
provider: 'mock', model: 'mock', messages: [], sessionId: session.id,
|
||||
}))
|
||||
await Promise.resolve()
|
||||
expect(order).toEqual(['flush:start'])
|
||||
gate.resolve(undefined)
|
||||
await pending
|
||||
expect(order).toEqual(['flush:start', 'flush:end', 'adapter'])
|
||||
})
|
||||
|
||||
it('delegates a request without a live session without checkpointing', async () => {
|
||||
const ctx = await setup()
|
||||
const order: string[] = []
|
||||
ctx.on('session/flush', () => { order.push('flush') })
|
||||
ctx.llm.registerAdapter(['mock'], new RecordingAdapter(order))
|
||||
await drain(ctx.llm.stream({ provider: 'mock', model: 'mock', messages: [] }))
|
||||
expect(order).toEqual(['adapter'])
|
||||
})
|
||||
|
||||
it('delegates an already-detached session id without checkpointing', async () => {
|
||||
const ctx = await setup()
|
||||
const order: string[] = []
|
||||
ctx.on('session/flush', () => { order.push('flush') })
|
||||
ctx.llm.registerAdapter(['mock'], new RecordingAdapter(order))
|
||||
await drain(ctx.llm.stream({
|
||||
provider: 'mock', model: 'mock', messages: [], sessionId: SessionId('detached'),
|
||||
}))
|
||||
expect(order).toEqual(['adapter'])
|
||||
})
|
||||
|
||||
it('does not dispatch the adapter when the checkpoint rejects', async () => {
|
||||
const ctx = await setup()
|
||||
const session = ctx.sessions.create(SessionId('request-failure'))
|
||||
const order: string[] = []
|
||||
ctx.on('session/flush', () => Promise.reject(new Error('disk unavailable')))
|
||||
ctx.llm.registerAdapter(['mock'], new RecordingAdapter(order))
|
||||
await expect(drain(ctx.llm.stream({
|
||||
provider: 'mock', model: 'mock', messages: [], sessionId: session.id,
|
||||
}))).rejects.toThrow('disk unavailable')
|
||||
expect(order).toEqual([])
|
||||
})
|
||||
})
|
||||
|
||||
describe('session-checkpoint-policy tool and step boundaries', () => {
|
||||
it('awaits the checkpoint before a top-level tool body', async () => {
|
||||
const ctx = await setup()
|
||||
const session = ctx.sessions.create(SessionId('tool-checkpoint'))
|
||||
const agent = { session } as Agent
|
||||
const gate = Promise.withResolvers<undefined>()
|
||||
const order: string[] = []
|
||||
ctx.on('session/flush', async () => {
|
||||
order.push('flush:start')
|
||||
await gate.promise
|
||||
order.push('flush:end')
|
||||
})
|
||||
ctx.tools.register({
|
||||
name: 'write', description: 'side effect', parameters: {},
|
||||
output: { schema: { type: 'null' }, render: () => [] },
|
||||
execute: async () => { order.push('tool'); return null },
|
||||
})
|
||||
|
||||
const pending = ctx.tools.execute({
|
||||
callId: CallId('write-1'), name: 'write', arguments: {}, agent,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
await Promise.resolve()
|
||||
expect(order).toEqual(['flush:start'])
|
||||
gate.resolve(undefined)
|
||||
await expect(pending).resolves.toMatchObject({ isError: false })
|
||||
expect(order).toEqual(['flush:start', 'flush:end', 'tool'])
|
||||
})
|
||||
|
||||
it('does not dispatch when cancellation lands during the tool checkpoint', async () => {
|
||||
const ctx = await setup()
|
||||
const session = ctx.sessions.create(SessionId('tool-checkpoint-cancel'))
|
||||
const agent = { session } as Agent
|
||||
const controller = new AbortController()
|
||||
const gate = Promise.withResolvers<undefined>()
|
||||
const order: string[] = []
|
||||
ctx.on('session/flush', async () => {
|
||||
order.push('flush:start')
|
||||
await gate.promise
|
||||
order.push('flush:end')
|
||||
})
|
||||
ctx.tools.register({
|
||||
name: 'write', description: 'side effect', parameters: {},
|
||||
output: { schema: { type: 'null' }, render: () => [] },
|
||||
execute: async () => { order.push('tool'); return null },
|
||||
})
|
||||
|
||||
const pending = ctx.tools.execute({
|
||||
callId: CallId('write-cancelled'), name: 'write', arguments: {}, agent,
|
||||
signal: controller.signal,
|
||||
})
|
||||
await Promise.resolve()
|
||||
expect(order).toEqual(['flush:start'])
|
||||
controller.abort('cancelled during checkpoint')
|
||||
gate.resolve(undefined)
|
||||
|
||||
await expect(pending).resolves.toEqual({
|
||||
content: [{ type: 'text', text: 'Error: tool call aborted before dispatch' }],
|
||||
isError: true,
|
||||
error: {
|
||||
message: 'tool call aborted before dispatch',
|
||||
info: { name: 'AbortError', code: TOOL_ABORTED_BEFORE_DISPATCH },
|
||||
},
|
||||
})
|
||||
expect(order).toEqual(['flush:start', 'flush:end'])
|
||||
})
|
||||
|
||||
it('turns a rejected checkpoint into an error result without running the tool body', async () => {
|
||||
const ctx = await setup()
|
||||
const session = ctx.sessions.create(SessionId('tool-failure'))
|
||||
const agent = { session } as Agent
|
||||
let ran = false
|
||||
ctx.on('session/flush', () => Promise.reject(new Error('disk unavailable')))
|
||||
ctx.tools.register({
|
||||
name: 'write', description: 'side effect', parameters: {},
|
||||
output: { schema: { type: 'null' }, render: () => [] },
|
||||
execute: async () => { ran = true; return null },
|
||||
})
|
||||
const result = await ctx.tools.execute({
|
||||
callId: CallId('write-2'), name: 'write', arguments: {}, agent,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
expect(result.isError).toBe(true)
|
||||
expect(result.content).toEqual([{ type: 'text', text: 'Error: disk unavailable' }])
|
||||
expect(ran).toBe(false)
|
||||
})
|
||||
|
||||
it('reuses the outer checkpoint for a nested tool dispatch', async () => {
|
||||
const ctx = await setup()
|
||||
const session = ctx.sessions.create(SessionId('nested-tool'))
|
||||
const agent = { session } as Agent
|
||||
let flushes = 0
|
||||
ctx.on('session/flush', () => { flushes += 1 })
|
||||
ctx.tools.register({
|
||||
name: 'nested', description: 'nested', parameters: {},
|
||||
output: { schema: { type: 'null' }, render: () => [] },
|
||||
execute: async () => null,
|
||||
})
|
||||
await ctx.tools.execute({
|
||||
callId: CallId('nested-1'), name: 'nested', arguments: {}, agent,
|
||||
parent: Symbol('outer') as never,
|
||||
signal: new AbortController().signal,
|
||||
})
|
||||
expect(flushes).toBe(0)
|
||||
})
|
||||
|
||||
it('checkpoints during pre-step processing', async () => {
|
||||
const ctx = await setup()
|
||||
const session = ctx.sessions.create(SessionId('post-step'))
|
||||
const agent = { session } as Agent
|
||||
const flushed: string[] = []
|
||||
ctx.on('session/flush', (current) => { flushed.push(current.id) })
|
||||
const signal = new AbortController().signal
|
||||
await agentEvents(ctx, agent).waterfall(
|
||||
'agent/pre-step', { messages: [], turn: 1, step: 1, signal },
|
||||
() => Promise.resolve({ kind: 'enter', messages: [] }),
|
||||
)
|
||||
expect(flushed).toEqual([session.id])
|
||||
})
|
||||
})
|
||||
|
||||
describe('session-checkpoint-policy lifecycle', () => {
|
||||
it('removes its wrappers when the owning fiber is disposed', async () => {
|
||||
const ctx = new Context()
|
||||
contexts.push(ctx)
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(SystemPrompt)
|
||||
await ctx.plugin(ToolRegistry)
|
||||
await ctx.plugin(TestPersistence)
|
||||
const session = ctx.sessions.create(SessionId('disposed-policy'))
|
||||
let flushes = 0
|
||||
ctx.on('session/flush', () => { flushes += 1 })
|
||||
ctx.llm.registerAdapter(['mock'], new RecordingAdapter([]))
|
||||
const fiber = await ctx.plugin(checkpointPolicy)
|
||||
await drain(ctx.llm.stream({ provider: 'mock', model: 'mock', messages: [], sessionId: session.id }))
|
||||
expect(flushes).toBe(1)
|
||||
await fiber.dispose()
|
||||
await drain(ctx.llm.stream({ provider: 'mock', model: 'mock', messages: [], sessionId: session.id }))
|
||||
expect(flushes).toBe(1)
|
||||
})
|
||||
|
||||
it('keeps the Loader-safe namespace plugin shape', () => {
|
||||
expect('default' in checkpointPolicy).toBe(false)
|
||||
const loader = Object.create(Loader.prototype) as Loader
|
||||
const unwrapped = loader.unwrapExports(checkpointPolicy) as Record<string, unknown>
|
||||
expect(unwrapped).toBe(checkpointPolicy)
|
||||
expect(unwrapped.name).toBe('session-checkpoint-policy')
|
||||
expect(unwrapped.inject).toEqual(['llm', 'sessionPersistence', 'sessions', 'tools'])
|
||||
expect(typeof unwrapped.apply).toBe('function')
|
||||
})
|
||||
})
|
||||
36
packages/session/session-checkpoint-policy/tsconfig.json
Normal file
36
packages/session/session-checkpoint-policy/tsconfig.json
Normal file
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../core/agent"
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
{
|
||||
"path": "../session-persistence"
|
||||
},
|
||||
{
|
||||
"path": "../../support/invariants"
|
||||
},
|
||||
{
|
||||
"path": "../../core/tools"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/session-persistence/session-persistence-jsonl/README.md
|
||||
README.md: b7fa8fc2918711dd24eeba67132a267452d401f9
|
||||
README.zh.md: 59d21c6171e857849c4ab48940d66962fffbfdeb
|
||||
77
packages/session/session-persistence-jsonl/README.md
Normal file
77
packages/session/session-persistence-jsonl/README.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# @deepseek-ai/dsh-session-persistence-jsonl
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
The JSONL durable session-persistence backend — a concrete `SessionPersistence` (the `dsh-session-persistence` seam). Each session has one append-only logical JSONL log, stored as `.jsonl.zstd` by default or raw `.jsonl` when compression is disabled.
|
||||
|
||||
## On-disk layout
|
||||
|
||||
```
|
||||
<root>/
|
||||
--<normalized-cwd>--/ # readable project directory (or _no-cwd/)
|
||||
<encoded-id>/ # session-owned directory
|
||||
session.jsonl.zstd # default: checksummed header frame + append frames
|
||||
session.jsonl # only with compression: 'none'
|
||||
```
|
||||
|
||||
- The first logical line is the immutable `SessionHeader` tagged `{ type: 'session', version, id, cwd?, createdAt, parentSession?, seedLength?, origin?, delegationDepth }`. `delegationDepth` is required on disk and is `0` for a top-level session; a missing or invalid value rejects the log. Every subsequent logical line is one storage record; `assistant/chunk` events are never dropped, and `seq` stays contiguous across the decoded log (`events[i].seq === i`).
|
||||
- A storage record is a `SessionEvent` JSON verbatim, or — for an eligible run when `packChunks` is enabled — a **packed chunk row** (`text-chunks` / `reasoning-chunks` / `tool-call-chunks`; bare slash-less tags like the header's `session`, so row tags cannot be confused with event types): one line holding a run of ≥3 consecutive same-block `assistant/chunk` delta events, `seq0`/`time0` plus per-member `dt` gaps reconstructing every member's `seq`/`time` exactly. The lossless codec lives in `@deepseek-ai/dsh-session` (`packChunkRuns`/`decodeStorageRecord`) and whitelists exact shapes — anything unrecognized stores verbatim. Reading is layout-blind: `load` always decodes rows, so packed, unpacked, and mixed files load identically.
|
||||
- The project directory keeps the normalized cwd readable for navigation and is bounded for filesystem component limits. Separator replacement and truncation are intentionally lossy, so cwd strings that normalize alike share a project directory; session ids still select distinct session directories. On a case-insensitive filesystem, identity validation accepts an alternate path spelling only when filesystem canonicalization resolves both spellings to the same transcript. The configured root remains deployment-controlled: it may be project-local, shared, temporary, or centralized. The [project-session directory decision](../../../.agents/notes/implemented/architecture/2026-07-24-project-session-directories.md) records this tradeoff.
|
||||
- Session ids are unvalidated branded strings, so they are injectively escaped to a single safe path segment before use (no traversal, no collision). The resulting directory is reserved for additional session-owned artifacts; discovery reads only the fixed transcript filename.
|
||||
|
||||
## Config
|
||||
|
||||
| Key | Type | Notes |
|
||||
|---|---|---|
|
||||
| `root` | `string` (required) | Root directory for all session files. **No default** — a `process.cwd()` default would scatter files as the process's cwd changes (bash calls, subprocesses). An existing root must be a readable directory; an absent root is created on first materialization. |
|
||||
| `packChunks` | `boolean` (default `true`) | Write eligible delta-chunk runs as packed rows (~60% smaller logical logs measured on a real coding session). Set `false` for one-event-per-line diagnostics; reading packed rows works regardless of this write-side switch. |
|
||||
| `compression` | `'zstd' \| 'none'` | Defaults to `'zstd'`; `'none'` retains newline-delimited UTF-8 text. |
|
||||
| `preparedSessionCacheSize` | positive integer (default `5`) | Maximum unpublished Sessions retained after cold history inspection for reuse by resume. |
|
||||
| `writeBatchMaxDelayMs` | positive integer (default `200`) | Fixed coalescing window after an idle live-event queue receives work. Later events do not reset it; flush and teardown bypass it. It does not bound event-loop, serialized-operation, or backend latency. At most Node's `2_147_483_647` ms timer limit. |
|
||||
|
||||
`locate(meta)` returns `{ kind: 'jsonl', path }` for the fixed transcript inside the resolved project/session directories. It performs no filesystem I/O: the target can be returned before the directory or file exists, and an existing file contains only the last flushed prefix.
|
||||
|
||||
## Physical encoding
|
||||
|
||||
The default artifact is a standard concatenation of independent [Zstandard frames](../../../.agents/notes/implemented/architecture/2026-07-19-zstandard-jsonl-session-logs.md): one checksummed frame containing only the header line, followed by one checksummed frame per durable append batch. The backend uses Node's built-in Zstandard API with its default compression level and exposes no level knob. Listing reads and validates only the header frame. `compression: 'none'` keeps the same logical lines in the original raw representation.
|
||||
|
||||
A root belongs to one encoding. Startup discovery and targeted lookup reject the opposite suffix with an error naming the incompatible artifact and instructing the caller to select the matching mode or a separate root. Flat `<project>/<id>.jsonl*` artifacts are also rejected instead of ignored. There is no migration, mixed-root fallback, or dual write.
|
||||
|
||||
## Durability and crash semantics
|
||||
|
||||
- **Bound storage identity.** Lookup requires one matching session directory across the readable project directories, then verifies that the header id equals the requested id and that the header's id/cwd derive the selected transcript path. Listing applies the same path check and rejects duplicate ids. Identity failures occur before repair or append.
|
||||
- **Lazy materialization.** `create(meta)` writes nothing; on the first `append`, the backend writes and `fsync`s the encoded header and first batch in a temporary file. POSIX publishes it without overwrite via a hard link and `fsync`s the parent directory. Windows publishes it without overwrite via `MoveFileExW(..., MOVEFILE_WRITE_THROUGH)` and creates missing directories through the same write-through pattern. A created-but-never-appended session leaves nothing on disk and is absent from `list`.
|
||||
- **Append-only.** Flushed events are never rewritten. Subsequent raw batches append lines; compressed batches append one frame. Both paths `fsync`, and a caught write or sync failure rolls the file back to its prior byte length.
|
||||
- **Crash recovery — preserve valid tail work.** `load` validates every complete compressed frame and scans their decompressed JSONL. If the last frame is structurally incomplete, the reader keeps its complete decoded records, truncates from that frame's start, and re-encodes those records with the synthetic tool, step, and turn closers required by the shared [persistence contract](../../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md). Raw mode truncates from its first incomplete line. A checksum/decompression failure in a complete frame, or a defect at or before the last committed `turn/end`, is corruption and rejects.
|
||||
- **Non-mutating inspection.** `inspect()` returns an immutable balanced logical view and may synthesize recovery closers in memory, without truncating an incomplete tail or changing the lightweight revision.
|
||||
- **Contiguous-seq.** `append` rejects a batch whose first `seq` does not continue the stored log, and rejects non-JSON-serializable `event.data` naming the offending event type.
|
||||
- **Lightweight revisions.** `listSnapshots(signal?)` identifies a log by its device, inode, size, and nanosecond timestamps, avoiding a full-log parse while changing after append, repair, replacement, or store changes. A full-prefix read requires the same identity before and after reading the bytes, and `readStoredRevision()` uses that identity to validate retained preparations without loading the log. Snapshot listing forwards the exact signal through artifact discovery and checks cancellation around every `stat`; because filesystem `stat` is not interruptible, cancellation waits for the active call to settle, then rejects without starting another.
|
||||
|
||||
## Write path
|
||||
|
||||
The plugin copies frozen session events into one controller per live session. The first pending event starts the configured fixed batching window, and later events join without resetting it. Expiry starts one durable append; events admitted during that write form a separately bounded follow-up batch. `session/flush` cancels the wait and drains current and pending batches. A per-session cursor prevents resumed sessions from re-appending stored events, and live sessions are seeded when the plugin loads. The owning backend instance serializes operations for one session; disposal drains every retained controller before teardown. Every logical event remains present: batching only lets one compressed frame or raw fsync carry more records.
|
||||
|
||||
## Model Experience
|
||||
|
||||
### Resumed conversation history
|
||||
|
||||
#### What the model sees
|
||||
|
||||
JSONL storage contributes no live prompt or schema. Loading restores stored surface history and preserves prior request headers for reconstruction; the new loop composes its current envelope. Recovery balances an assistant request without a durable call with `TOOL_NOT_STARTED`; a durable call without a result becomes `TOOL_OUTCOME_UNKNOWN`, which tells the model to retry only read-only or idempotent work and to verify possible side effects or ask the user. Raw `assistant/chunk` records do not duplicate messages.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Zero live-request tokens. A resumed agent pays for retained history and its current envelope, plus the quoted repair result for each interrupted call.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
JSONL storage does not mutate live request prefixes. A resumed loop can reuse provider cache only when its reconstructed history, current envelope, and model route match; crash-repair results append.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **Only the configured encoding and current `SESSION_FORMAT_VERSION` (v0) load** — changing compression requires a separate/fresh root or selecting the legacy raw mode; the pre-release format has no migration.
|
||||
- **The flat-file storage layout does not load** — use a separate root or move pre-release artifacts into the project/session directory layout before loading.
|
||||
- **Compressed files are not directly line-readable** — use the backend to load them, or select `compression: 'none'` before writing a fresh root when external line readers are required.
|
||||
- **Nothing deletes session files** — logs accumulate under `root` until removed externally (the seam has no deletion surface).
|
||||
- **One live writer per session** — append and repair are coordinated only inside the owning backend instance. Another backend instance or process must not write the same session until that owner reaches quiescent disposal; initial same-id publication remains collision-safe through the POSIX no-overwrite hard link or Windows write-through rename without replacement.
|
||||
- **POSIX materialization requires hard-link support** — first append uses `link()` so same-id races fail instead of overwriting a committed log; Windows uses write-through rename without replacement.
|
||||
77
packages/session/session-persistence-jsonl/README.zh.md
Normal file
77
packages/session/session-persistence-jsonl/README.zh.md
Normal file
@@ -0,0 +1,77 @@
|
||||
# @deepseek-ai/dsh-session-persistence-jsonl
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
JSONL 持久会话存储后端：`SessionPersistence` 的一个具体实现（`dsh-session-persistence` seam）。每个会话有一个仅追加的逻辑 JSONL 日志，默认存储为 `.jsonl.zstd`；禁用压缩时使用原始 `.jsonl`。
|
||||
|
||||
## 磁盘布局
|
||||
|
||||
```
|
||||
<root>/
|
||||
--<normalized-cwd>--/ # readable project directory (or _no-cwd/)
|
||||
<encoded-id>/ # session-owned directory
|
||||
session.jsonl.zstd # default: checksummed header frame + append frames
|
||||
session.jsonl # only with compression: 'none'
|
||||
```
|
||||
|
||||
- 第一个逻辑行是不可变的 `SessionHeader`，标记为 `{ type: 'session', version, id, cwd?, createdAt, parentSession?, seedLength?, origin?, delegationDepth }`。`delegationDepth` 在磁盘上必需，顶层会话为 `0`；缺失或无效值会拒绝日志。后续每个逻辑行是一条存储记录；`assistant/chunk` 事件绝不丢弃，且 `seq` 在解码日志中保持连续（`events[i].seq === i`）。
|
||||
- 存储记录是原样 `SessionEvent` JSON，或在 `packChunks` 已启用且连续段符合条件时写入的**打包分片行**（`text-chunks` / `reasoning-chunks` / `tool-call-chunks`；像 header 的 `session` 一样不带斜杠，因此行 tag 不会与事件类型混淆）：一行保存至少 3 个连续同 block `assistant/chunk` delta 事件，`seq0`/`time0` 和每成员 `dt` 间隔精确重建每个成员的 `seq`/`time`。无损 codec 位于 `@deepseek-ai/dsh-session`（`packChunkRuns`/`decodeStorageRecord`），并使用精确形态 allowlist：任何未识别内容原样存储。读取与布局无关：`load` 始终解码行，因此打包、非打包和混合文件加载结果一致。
|
||||
- 项目目录保留规范化 cwd 可读，并限制在文件系统组件上限内。分隔符替换和截断刻意有损，因此规范化相同的 cwd 字符串共享项目目录；会话 id 仍选择不同会话目录。在不区分大小写的文件系统上，只有文件系统规范化将两种写法解析到同一 transcript（文本记录）时，身份验证才接受备选路径写法。配置根仍由部署控制：可以是项目本地、共享、临时或集中式。[项目会话目录决策](../../../.agents/notes/implemented/architecture/2026-07-24-project-session-directories.md) 记录这项取舍。
|
||||
- 会话 id 是未验证的带品牌类型的字符串，因此在使用前单射转义为一个安全路径段（无遍历、无冲突）。结果目录保留给其他会话自有产物；发现只读取固定 transcript 文件名。
|
||||
|
||||
## 配置
|
||||
|
||||
| 键 | 类型 | 说明 |
|
||||
|---|---|---|
|
||||
| `root` | `string`（必需） | 所有会话文件的根目录。**无默认值**：`process.cwd()` 默认值会随进程 cwd 变更（bash 调用、子进程）而分散文件。现有根必须是可读目录；缺失根在第一次实体化时创建。 |
|
||||
| `packChunks` | `boolean`（默认 `true`） | 将符合条件的 delta 分片连续段写为打包行（在真实编码会话上测得逻辑日志约小 60%）。设为 `false` 可用于每事件一行诊断；无论该写入侧开关如何，都能读取打包行。 |
|
||||
| `compression` | `'zstd' \| 'none'` | 默认 `'zstd'`；`'none'` 保留换行分隔 UTF-8 文本。 |
|
||||
| `preparedSessionCacheSize` | 正整数（默认 `5`） | 冷历史检查后保留、供恢复复用的未发布 Session 数量上限。 |
|
||||
| `writeBatchMaxDelayMs` | 正整数（默认 `200`） | 空闲的活动事件队列收到待写入事件后开启的固定合并窗口。后续事件不会重置窗口；flush 与 teardown 会绕过它。该值不限制事件循环、串行化操作或后端延迟。最大值为 Node 计时器上限 `2_147_483_647` ms。 |
|
||||
|
||||
`locate(meta)` 返回已解析项目/会话目录内固定 transcript 的 `{ kind: 'jsonl', path }`。它不执行文件系统 I/O：可以在目录或文件存在前返回目标，现有文件也只包含最近一次 flush 完成的前缀。
|
||||
|
||||
## 物理编码
|
||||
|
||||
默认产物是独立 [Zstandard frame](../../../.agents/notes/implemented/architecture/2026-07-19-zstandard-jsonl-session-logs.md) 的标准拼接：一个仅包含 header 行的带 checksum frame，后跟每个持久 append 批次一个带 checksum frame。后端使用 Node 内置 Zstandard API 和默认压缩级别，不提供级别开关。列表只读取并验证 header frame。`compression: 'none'` 在原始表示中保留相同逻辑行。
|
||||
|
||||
一个根只属于一种编码。启动发现和定向查找会拒绝相反 suffix，错误会命名不兼容产物，并指示调用方选择匹配 mode 或独立根。平铺 `<project>/<id>.jsonl*` 产物也会被拒绝，而不是忽略。不提供迁移、混合根回退或双写。
|
||||
|
||||
## 持久性与崩溃语义
|
||||
|
||||
- **绑定存储身份。** 查找要求可读项目目录中只有一个匹配会话目录，然后验证 header id 等于请求 id，且 header id/cwd 派生所选 transcript 路径。列表应用同一路径检查，并拒绝重复 id。身份失败发生在修复或 append 前。
|
||||
- **延迟实体化。**`create(meta)` 不写入；第一次 `append` 将编码 header 和第一批写入临时文件并执行 `fsync`。POSIX 通过硬链接无覆盖发布，并对父目录 `fsync`。Windows 通过 `MoveFileExW(..., MOVEFILE_WRITE_THROUGH)` 无覆盖发布，并通过同一 write-through pattern 创建缺失目录。已创建但从未 append 的会话不留下磁盘内容，不在 `list` 中。
|
||||
- **仅追加。** 已 flush 事件绝不重写。后续原始批次 append 行；压缩批次 append 一个 frame。两条路径都执行 `fsync`，并在捕获到写入或同步失败时回滚到之前字节长度。
|
||||
- **崩溃恢复：保留有效尾部工作。**`load` 验证每个完整压缩 frame，并扫描解压 JSONL。最后 frame 结构不完整时，读取器保留其完整解码记录，从 frame 开头截断，并使用共享[持久化契约](../../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md) 需要的合成工具、步骤和轮次 closer 重新编码这些记录。原始 mode 从第一个不完整行截断。完整 frame 中的 checksum/解压失败，或位于最后已提交的 `turn/end` 处或之前的缺陷属于损坏，会被拒绝。
|
||||
- **非变更检查。**`inspect()` 返回不可变、平衡的逻辑视图，并可在内存中合成恢复 closer，但不会截断不完整尾部或更改轻量修订。
|
||||
- **连续 seq。**`append` 拒绝第一个 `seq` 不继续已存储日志的批次，并拒绝非 JSON 可序列化 `event.data`，同时命名违规事件类型。
|
||||
- **轻量修订。**`listSnapshots(signal?)` 使用 device、inode、size 和纳秒时间戳标识日志，避免解析完整日志；该标识会在 append、修复、替换或存储变更后改变。完整前缀读取要求读取字节前后的身份一致，`readStoredRevision()` 使用同一身份校验保留的 preparation，而不加载日志。快照列表通过产物发现转发精确信号，并在每个 `stat` 前后检查取消；由于文件系统 `stat` 不可中断，取消会等待活动调用完成，然后在不启动另一次调用的情况下拒绝。
|
||||
|
||||
## 写入路径
|
||||
|
||||
插件将冻结的会话事件复制到每个活动会话各自的 controller。第一个待处理事件会开启配置的固定批处理窗口，后续事件会加入但不会重置截止时间。窗口到期后会启动一次持久化追加；该次写入期间接纳的事件会形成另一个独立有界的后续批次。`session/flush` 会取消等待并排空当前与待处理批次。每会话游标防止恢复后的会话重新 append 已存储事件，插件加载时会为活动会话设置初始状态。所属后端实例串行化单会话操作；dispose（资源释放）会在拆卸前排空每个保留的 controller。每个逻辑事件都会保留：批处理只让单个压缩帧或一次原始 JSONL fsync 承载更多记录。
|
||||
|
||||
## 模型体验
|
||||
|
||||
### 恢复的对话历史
|
||||
|
||||
#### 模型看到的内容
|
||||
|
||||
JSONL 存储不影响当前提示词或 schema。加载会恢复已存储的表层历史，并保留之前的请求 header 用于重建；新 loop 组合当前 envelope。恢复会用 `TOOL_NOT_STARTED` 平衡没有已持久化调用的 assistant 请求；已有已持久化调用但无结果时则变为 `TOOL_OUTCOME_UNKNOWN`，它要求模型只重试只读或幂等工作，并验证可能的副作用或询问用户。原始 `assistant/chunk` 记录不会重复生成消息。
|
||||
|
||||
#### Token 影响
|
||||
|
||||
当前请求不会新增 token。恢复后的 agent（智能体）会因保留的历史、当前 envelope，以及每个中断调用中以引用形式加入的修复结果文本而消耗 token。
|
||||
|
||||
#### KV Cache 影响
|
||||
|
||||
JSONL 存储不修改实时请求前缀。只有重建历史、当前 envelope 和模型路由匹配时，恢复 loop 才能重用提供方缓存；崩溃修复结果仅追加。
|
||||
|
||||
## 已知限制与暂缓事项
|
||||
|
||||
- **只加载已配置编码和当前 `SESSION_FORMAT_VERSION` (v0)**：更改压缩需要独立/全新根，或选择遗留原始 mode；预发布格式没有迁移。
|
||||
- **平铺文件存储布局不加载**：加载前使用独立根，或将预发布产物移入项目/会话目录布局。
|
||||
- **压缩文件不能直接按行读取**：使用后端加载；或在写入新根前选择 `compression: 'none'`，以便外部行 reader 使用。
|
||||
- **不删除会话文件**：日志在 `root` 下累积，直到外部移除（seam 无删除接口）。
|
||||
- **每会话一个实时 writer**：append 和修复只在所属后端实例内协调。在所有者完成完全停稳的 dispose 前，其他后端实例或进程不得写入同一会话；初始同 id 发布仍通过 POSIX 无覆盖硬链接或 Windows 无替换 write-through rename 保持冲突安全。
|
||||
- **POSIX 实体化需要硬链接支持**：第一次 append 使用 `link()`，使同 id 竞态失败，而不覆盖已提交日志；Windows 使用无替换 write-through rename。
|
||||
43
packages/session/session-persistence-jsonl/package.json
Normal file
43
packages/session/session-persistence-jsonl/package.json
Normal file
@@ -0,0 +1,43 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-session-persistence-jsonl",
|
||||
"description": "JSONL durable session persistence backend for the DeepSeek Harness",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./invariant": {
|
||||
"types": "./lib/types/invariant.d.ts",
|
||||
"default": "./lib/invariant.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-invariants": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session-persistence": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"dependencies": {
|
||||
"koffi": "^3.1.0",
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-persistence": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
390
packages/session/session-persistence-jsonl/src/format.ts
Normal file
390
packages/session/session-persistence-jsonl/src/format.ts
Normal file
@@ -0,0 +1,390 @@
|
||||
/**
|
||||
* On-disk format helpers for the JSONL session-persistence backend: path
|
||||
* sanitization (a {@link SessionId} is an unvalidated branded string, so it
|
||||
* MUST be encoded before use in a path — no traversal, no collision), the
|
||||
* per-project/session directory layout, header-line (de)serialization, and the
|
||||
* truncation-repair offset computation.
|
||||
*
|
||||
* @module dsh-session-persistence-jsonl/format
|
||||
*/
|
||||
|
||||
import { join } from 'node:path'
|
||||
import { decodeStorageRecord, packChunkRuns } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionEvent, SessionHeader, SessionId, StorageRecord } from '@deepseek-ai/dsh-session'
|
||||
|
||||
/** Physical encoding selected for JSONL session artifacts. */
|
||||
export type JsonlCompression = 'zstd' | 'none'
|
||||
|
||||
/**
|
||||
* Return the artifact suffix for one physical encoding.
|
||||
* @param compression - configured JSONL artifact encoding.
|
||||
* @returns `.jsonl.zstd` for Zstandard or `.jsonl` for plaintext.
|
||||
*/
|
||||
export function logSuffix(compression: JsonlCompression): '.jsonl.zstd' | '.jsonl' {
|
||||
return compression === 'zstd' ? '.jsonl.zstd' : '.jsonl'
|
||||
}
|
||||
|
||||
/**
|
||||
* The first JSONL record of a session artifact: the immutable
|
||||
* {@link SessionHeader} tagged as a `session` record so a reader can tell it
|
||||
* apart from an event line.
|
||||
*/
|
||||
export interface HeaderLine {
|
||||
type: 'session'
|
||||
version: number
|
||||
id: SessionId
|
||||
createdAt: number
|
||||
cwd?: string
|
||||
parentSession?: SessionId
|
||||
seedLength?: number
|
||||
origin?: 'subagent'
|
||||
delegationDepth: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the header line object from a {@link SessionHeader}.
|
||||
* @param header - the immutable session metadata to serialize.
|
||||
* @returns the `type: 'session'`-tagged line object, absent optional fields omitted (never null).
|
||||
*/
|
||||
export function toHeaderLine(header: SessionHeader): HeaderLine {
|
||||
return {
|
||||
type: 'session',
|
||||
version: header.version,
|
||||
id: header.id,
|
||||
createdAt: header.createdAt,
|
||||
...header.cwd !== undefined ? { cwd: header.cwd } : {},
|
||||
...header.parentSession !== undefined ? { parentSession: header.parentSession } : {},
|
||||
...header.seedLength !== undefined ? { seedLength: header.seedLength } : {},
|
||||
...header.origin !== undefined ? { origin: header.origin } : {},
|
||||
delegationDepth: header.delegationDepth ?? 0,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a header line back into a {@link SessionHeader}.
|
||||
* @param line - the shape-checked first line of a log (see the `isHeaderLine` guard).
|
||||
* @returns the header, absent optional fields omitted.
|
||||
*/
|
||||
export function fromHeaderLine(line: HeaderLine): SessionHeader {
|
||||
if (Object.hasOwn(line, 'sandboxMode') || Object.hasOwn(line, 'approvalPolicy')) {
|
||||
throw new Error('session header uses retired policy baseline fields')
|
||||
}
|
||||
return {
|
||||
version: line.version,
|
||||
id: line.id,
|
||||
createdAt: line.createdAt,
|
||||
...line.cwd !== undefined ? { cwd: line.cwd } : {},
|
||||
...line.parentSession !== undefined ? { parentSession: line.parentSession } : {},
|
||||
...line.seedLength !== undefined ? { seedLength: line.seedLength } : {},
|
||||
...line.origin !== undefined ? { origin: line.origin } : {},
|
||||
delegationDepth: line.delegationDepth,
|
||||
}
|
||||
}
|
||||
|
||||
/** Type guard: a parsed first line is a well-formed session header. */
|
||||
function isHeaderLine(value: unknown): value is HeaderLine {
|
||||
return (
|
||||
typeof value === 'object' && value !== null
|
||||
&& (value as { type?: unknown }).type === 'session'
|
||||
&& typeof (value as { version?: unknown }).version === 'number'
|
||||
&& typeof (value as { id?: unknown }).id === 'string'
|
||||
&& typeof (value as { createdAt?: unknown }).createdAt === 'number'
|
||||
&& Number.isSafeInteger((value as { createdAt: number }).createdAt)
|
||||
&& (value as { createdAt: number }).createdAt >= 0
|
||||
&& !Object.is((value as { createdAt: number }).createdAt, -0)
|
||||
&& typeof (value as { delegationDepth?: unknown }).delegationDepth === 'number'
|
||||
&& Number.isSafeInteger((value as { delegationDepth: number }).delegationDepth)
|
||||
&& (value as { delegationDepth: number }).delegationDepth >= 0
|
||||
&& !Object.is((value as { delegationDepth: number }).delegationDepth, -0)
|
||||
&& ((value as { origin?: unknown }).origin === undefined
|
||||
|| (value as { origin?: unknown }).origin === 'subagent')
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Encode an arbitrary string as a single safe path segment, injectively over ALL JS (UTF-16)
|
||||
* strings — including lone surrogates. A {@link SessionId} is an unvalidated branded string,
|
||||
* so this neutralizes `../`, absolute paths, NUL, and separators before any filesystem use.
|
||||
* Safe code units remain literal; every other unit, including `~`, becomes
|
||||
* `~XXXX`. Operating on code units preserves lone surrogates, while special-
|
||||
* casing `.` and `..` prevents traversal by an otherwise safe whole segment.
|
||||
*
|
||||
* @param raw - the string to encode; must be non-empty (throws on `''`).
|
||||
* @returns the escaped single path segment, decodable back to `raw`.
|
||||
*/
|
||||
export function encodeSegment(raw: string): string {
|
||||
if (raw.length === 0) throw new Error('cannot encode an empty path segment')
|
||||
if (raw === '.') return '~002E'
|
||||
if (raw === '..') return '~002E~002E'
|
||||
let out = ''
|
||||
for (let i = 0; i < raw.length; i++) {
|
||||
const code = raw.charCodeAt(i)
|
||||
const ch = String.fromCharCode(code)
|
||||
if (ch !== '~' && /^[A-Za-z0-9._-]$/.test(ch)) {
|
||||
out += ch
|
||||
} else {
|
||||
out += '~' + code.toString(16).toUpperCase().padStart(4, '0')
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the readable directory key for a project path.
|
||||
* Filesystem separators and drive separators become `-`; unsafe code units use
|
||||
* the same `~XXXX` escape as session ids. The key is bounded for filesystem
|
||||
* component limits. Separator replacement and truncation are intentionally
|
||||
* lossy, following the common human-navigable project-directory convention.
|
||||
* @param cwd - the session's project directory.
|
||||
* @returns a single filesystem-safe project directory name.
|
||||
*/
|
||||
export function projectKey(cwd: string): string {
|
||||
if (cwd.length === 0) throw new Error('cannot encode an empty project path')
|
||||
let readable = ''
|
||||
let separatorRun = false
|
||||
for (let i = 0; i < cwd.length; i++) {
|
||||
const code = cwd.charCodeAt(i)
|
||||
const ch = String.fromCharCode(code)
|
||||
if (ch === '/' || ch === '\\' || ch === ':') {
|
||||
if (!separatorRun) readable += '-'
|
||||
separatorRun = true
|
||||
} else if (ch !== '~' && /^[A-Za-z0-9._-]$/.test(ch)) {
|
||||
readable += ch
|
||||
separatorRun = false
|
||||
} else {
|
||||
readable += '~' + code.toString(16).toUpperCase().padStart(4, '0')
|
||||
separatorRun = false
|
||||
}
|
||||
}
|
||||
const slug = readable.replace(/^-+/, '') || 'root'
|
||||
return `--${slug.slice(0, 251)}--`
|
||||
}
|
||||
|
||||
/**
|
||||
* The configured root's human-navigable project directory. A configured root
|
||||
* may be local or shared; this grouping does not prescribe its deployment.
|
||||
* @param root - the backend's session root directory.
|
||||
* @param cwd - the session's project directory; `undefined` selects `_no-cwd`.
|
||||
* @returns the project directory path under `root`.
|
||||
*/
|
||||
export function projectDir(root: string, cwd: string | undefined): string {
|
||||
if (cwd === undefined) return join(root, '_no-cwd')
|
||||
return join(root, projectKey(cwd))
|
||||
}
|
||||
|
||||
/**
|
||||
* The directory owned by one session and available for future session-local
|
||||
* artifacts.
|
||||
* @param root - the backend's session root directory.
|
||||
* @param cwd - the session's project directory.
|
||||
* @param id - the session id, encoded to one safe path segment.
|
||||
* @returns the session directory beneath its project directory.
|
||||
*/
|
||||
export function sessionDir(root: string, cwd: string | undefined, id: SessionId): string {
|
||||
return join(projectDir(root, cwd), encodeSegment(id))
|
||||
}
|
||||
|
||||
/**
|
||||
* The append-only event-log file path for a session.
|
||||
* @param root - the backend's session root directory.
|
||||
* @param cwd - the session's project directory (`undefined` → `_no-cwd`).
|
||||
* @param id - the session id, path-encoded via {@link encodeSegment} before filesystem use.
|
||||
* @param compression - physical artifact encoding and filename suffix.
|
||||
* @returns the session's configured JSONL artifact path.
|
||||
*/
|
||||
export function logPath(
|
||||
root: string,
|
||||
cwd: string | undefined,
|
||||
id: SessionId,
|
||||
compression: JsonlCompression,
|
||||
): string {
|
||||
return join(sessionDir(root, cwd, id), `session${logSuffix(compression)}`)
|
||||
}
|
||||
|
||||
/**
|
||||
* Serialize an event batch as JSONL lines (no trailing newline). With
|
||||
* `packChunks` on, delta-chunk runs pack into `text-chunks` /
|
||||
* `reasoning-chunks` / `tool-call-chunks` storage rows; off writes one event
|
||||
* per line, byte-identical to the pre-packing layout. Reading is layout-blind
|
||||
* either way ({@link scanLog} always decodes rows), so the switch only shapes
|
||||
* NEW bytes.
|
||||
* @param events - the batch to serialize, in log order.
|
||||
* @param packChunks - whether to pack delta runs into storage rows.
|
||||
* @returns the batch's JSONL text; the writer adds the final newline.
|
||||
*/
|
||||
export function eventLines(events: readonly SessionEvent[], packChunks: boolean): string {
|
||||
const records: readonly StorageRecord[] = packChunks ? packChunkRuns(events) : events
|
||||
return records.map(record => JSON.stringify(record)).join('\n')
|
||||
}
|
||||
|
||||
interface SessionLogScan {
|
||||
meta: SessionHeader
|
||||
events: SessionEvent[]
|
||||
committedBytes: number
|
||||
}
|
||||
|
||||
/** Parse one complete header record supplied independently from event rows. */
|
||||
function parseHeaderRecord(record: Buffer): SessionHeader {
|
||||
if (record.length === 0 || record.at(-1) !== 0x0A || record.indexOf(0x0A) !== record.length - 1) {
|
||||
throw new Error('empty or header-less session log')
|
||||
}
|
||||
let parsed: unknown
|
||||
try {
|
||||
parsed = JSON.parse(record.subarray(0, -1).toString('utf8'))
|
||||
} catch {
|
||||
throw new Error('corrupt session log: header line is not valid JSON')
|
||||
}
|
||||
if (!isHeaderLine(parsed)) {
|
||||
throw new Error('corrupt session log: first line is not a session header')
|
||||
}
|
||||
return fromHeaderLine(parsed)
|
||||
}
|
||||
|
||||
/**
|
||||
* Incrementally scan complete JSONL event records after an independently
|
||||
* supplied header record. Newline search and byte offsets stay on raw buffers;
|
||||
* only complete records are decoded to UTF-8. A fragment crossing writes is
|
||||
* copied because a decoder may reuse its output buffer after `write()` returns.
|
||||
*/
|
||||
export class SessionLogScanner {
|
||||
private readonly meta: SessionHeader
|
||||
private readonly events: SessionEvent[] = []
|
||||
private fragments: Buffer[] = []
|
||||
private fragmentBytes = 0
|
||||
private inputBytes: number
|
||||
private committedBytes: number
|
||||
private eventLine = 0
|
||||
private issue: Error | undefined
|
||||
private finished = false
|
||||
|
||||
/**
|
||||
* Create an event scanner from exactly one newline-terminated header record.
|
||||
* @param headerRecord - the complete first JSONL record, including its newline.
|
||||
*/
|
||||
constructor(headerRecord: Buffer) {
|
||||
this.meta = parseHeaderRecord(headerRecord)
|
||||
this.inputBytes = headerRecord.length
|
||||
this.committedBytes = headerRecord.length
|
||||
}
|
||||
|
||||
/**
|
||||
* Consume the next raw plaintext chunk, retaining only an incomplete final record.
|
||||
* @param chunk - bytes immediately following all previously supplied bytes.
|
||||
*/
|
||||
write(chunk: Buffer): void {
|
||||
if (this.finished) throw new Error('cannot write to a finished session log scanner')
|
||||
const chunkStart = this.inputBytes
|
||||
this.inputBytes += chunk.length
|
||||
let lineStart = 0
|
||||
for (
|
||||
let newline = chunk.indexOf(0x0A);
|
||||
newline !== -1;
|
||||
newline = chunk.indexOf(0x0A, lineStart)
|
||||
) {
|
||||
const fragment = chunk.subarray(lineStart, newline)
|
||||
let line = fragment
|
||||
if (this.fragments.length > 0) {
|
||||
if (fragment.length > 0) this.fragments.push(fragment)
|
||||
line = Buffer.concat(this.fragments, this.fragmentBytes + fragment.length)
|
||||
this.fragments = []
|
||||
this.fragmentBytes = 0
|
||||
}
|
||||
this.consumeEventLine(line, chunkStart + newline + 1)
|
||||
lineStart = newline + 1
|
||||
}
|
||||
if (lineStart < chunk.length) {
|
||||
const fragment = Buffer.from(chunk.subarray(lineStart))
|
||||
this.fragments.push(fragment)
|
||||
this.fragmentBytes += fragment.length
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Snapshot progress before appending a recoverable torn-frame prefix.
|
||||
* @returns byte, committed-prefix, and expanded-event cursors.
|
||||
*/
|
||||
checkpoint(): { inputBytes: number; committedBytes: number; eventCount: number } {
|
||||
return {
|
||||
inputBytes: this.inputBytes,
|
||||
committedBytes: this.committedBytes,
|
||||
eventCount: this.events.length,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Finish scanning, ignoring a final record without a newline as a torn tail.
|
||||
* @returns the header, contiguous event prefix, and safe truncation offset.
|
||||
*/
|
||||
finish(): SessionLogScan {
|
||||
this.finished = true
|
||||
return { meta: this.meta, events: this.events, committedBytes: this.committedBytes }
|
||||
}
|
||||
|
||||
/** Decode one complete event row and update the contiguous prefix. */
|
||||
private consumeEventLine(line: Buffer, endByte: number): void {
|
||||
this.eventLine += 1
|
||||
let decoded: SessionEvent[]
|
||||
try {
|
||||
decoded = decodeStorageRecord(JSON.parse(line.toString('utf8')))
|
||||
} catch {
|
||||
this.issue ??= new Error(`corrupt session log: unparsable committed event at line ${this.eventLine}`)
|
||||
return
|
||||
}
|
||||
|
||||
if (this.issue !== undefined) {
|
||||
if (decoded.some(event => event.type === 'turn/end')) throw this.issue
|
||||
return
|
||||
}
|
||||
|
||||
const rowStart = this.events.length
|
||||
for (const event of decoded) {
|
||||
if (event.seq !== this.events.length) {
|
||||
const expected = this.events.length
|
||||
this.events.length = rowStart
|
||||
this.issue = new Error(
|
||||
`corrupt session log: seq gap in committed region at line ${this.eventLine} `
|
||||
+ `(expected ${expected}, got ${event.seq})`,
|
||||
)
|
||||
if (decoded.some(candidate => candidate.type === 'turn/end')) throw this.issue
|
||||
return
|
||||
}
|
||||
this.events.push(event)
|
||||
}
|
||||
this.committedBytes = endByte
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a complete or torn JSONL buffer into its preserved event prefix. This
|
||||
* compatibility wrapper supplies the first record separately, then delegates
|
||||
* event rows to {@link SessionLogScanner}.
|
||||
*
|
||||
* @param buffer - the raw bytes of the log file (header line first).
|
||||
* @returns the header, preserved event prefix, and byte offset safe to append at.
|
||||
*/
|
||||
export function scanLog(buffer: Buffer): SessionLogScan {
|
||||
const headerEnd = buffer.indexOf(0x0A)
|
||||
if (headerEnd === -1) throw new Error('empty or header-less session log')
|
||||
const scanner = new SessionLogScanner(buffer.subarray(0, headerEnd + 1))
|
||||
scanner.write(buffer.subarray(headerEnd + 1))
|
||||
return scanner.finish()
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse just the header line of a log into a {@link SessionHeader}, or
|
||||
* `undefined` if it is missing/not a header. Used by `list()` to read session
|
||||
* metadata WITHOUT parsing the whole log: a session picker scales with the
|
||||
* number of sessions, not the total size of every conversation.
|
||||
* @param firstLine - the first line of a log file (without its trailing newline).
|
||||
* @returns the parsed header, or `undefined` when the line is not a well-formed session header.
|
||||
*/
|
||||
export function parseHeaderMeta(firstLine: string): SessionHeader | undefined {
|
||||
let parsed: unknown
|
||||
try {
|
||||
parsed = JSON.parse(firstLine)
|
||||
} catch {
|
||||
return undefined
|
||||
}
|
||||
if (!isHeaderLine(parsed)) return undefined
|
||||
return fromHeaderLine(parsed)
|
||||
}
|
||||
899
packages/session/session-persistence-jsonl/src/index.ts
Normal file
899
packages/session/session-persistence-jsonl/src/index.ts
Normal file
@@ -0,0 +1,899 @@
|
||||
/**
|
||||
* JSONL durable session-persistence backend. It stores a header and contiguous
|
||||
* events in one append-only file per session, and delegates orchestration to
|
||||
* {@link PersistenceCoordinator}. Its side-effect-free locator returns the
|
||||
* absolute per-session log target before materialization.
|
||||
* @module @deepseek-ai/dsh-session-persistence-jsonl
|
||||
*/
|
||||
|
||||
import { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import { readdirSync } from 'node:fs'
|
||||
import { open, mkdir, readFile, readdir, realpath, link, rm, stat, truncate } from 'node:fs/promises'
|
||||
import { dirname, join, resolve } from 'node:path'
|
||||
import { performance } from 'node:perf_hooks'
|
||||
import { scheduler } from 'node:timers/promises'
|
||||
import { randomBytes } from 'node:crypto'
|
||||
import {
|
||||
DEFAULT_PREPARED_SESSION_CACHE_SIZE, DEFAULT_WRITE_BATCH_MAX_DELAY_MS, MAX_WRITE_BATCH_DELAY_MS,
|
||||
SessionPersistence, SessionPersistenceRevision, PersistenceCoordinator,
|
||||
type PersistenceBackend, type SessionLocation, type SessionPersistenceSnapshot,
|
||||
type SessionInspection, type SessionPersistenceRevision as PersistenceRevision, type StoredPrefix,
|
||||
} from '@deepseek-ai/dsh-session-persistence'
|
||||
import type { SessionEvent, SessionId, SessionHeader, SessionPreparation } from '@deepseek-ai/dsh-session'
|
||||
import {
|
||||
encodeSegment, eventLines, logPath, logSuffix, parseHeaderMeta, projectDir, scanLog, sessionDir,
|
||||
SessionLogScanner, toHeaderLine,
|
||||
type JsonlCompression,
|
||||
} from './format.ts'
|
||||
import {
|
||||
compressZstdFrame, createZstdFrameDecoder, decompressZstdFrame, decompressZstdPrefix, scanZstdFrames,
|
||||
} from './zstd.ts'
|
||||
import { ensureDurableDirectoryWin32, publishNewFileWin32 } from './win32.ts'
|
||||
|
||||
export type { JsonlCompression } from './format.ts'
|
||||
|
||||
const DEFAULT_PACK_CHUNKS = true
|
||||
const DEFAULT_COMPRESSION: JsonlCompression = 'zstd'
|
||||
/**
|
||||
* Internal scheduling constant, not deployment configuration: balance
|
||||
* frame-boundary event-loop yields against `setImmediate` overhead. One frame
|
||||
* remains an indivisible synchronous decode.
|
||||
*/
|
||||
const ZSTD_DECODE_YIELD_INTERVAL_MS = 500
|
||||
|
||||
/** Assert that the independently decodable first frame contains only the header record. */
|
||||
function assertZstdHeaderFrame(plaintext: Buffer): void {
|
||||
if (plaintext.length === 0 || plaintext.indexOf(0x0A) !== plaintext.length - 1) {
|
||||
throw new Error('corrupt Zstandard session log: first frame is not exactly one header line')
|
||||
}
|
||||
}
|
||||
|
||||
/** Loader schema for the JSONL artifact's physical encoding. */
|
||||
export const JsonlCompressionSchema: z<JsonlCompression> = z.union([
|
||||
z.const('zstd'),
|
||||
z.const('none'),
|
||||
]).default(DEFAULT_COMPRESSION)
|
||||
|
||||
/** Plugin config: where the JSONL backend keeps its session logs, and the packed-row write switch. */
|
||||
export interface Config {
|
||||
/**
|
||||
* Root directory for all session files. Required (no default): a default of
|
||||
* `process.cwd()` would scatter session files as the process's cwd changes
|
||||
* (bash calls, subprocesses). Sessions group under human-readable project
|
||||
* directories, then per-session directories. An existing root must be a
|
||||
* readable directory; an absent root is created on first materialization.
|
||||
*/
|
||||
root: string
|
||||
/**
|
||||
* Write runs of consecutive `assistant/chunk` delta events as packed
|
||||
* `text-chunks`/`reasoning-chunks`/`tool-call-chunks` rows (lossless,
|
||||
* ~60% smaller logs measured on a real session). Defaults to true; false
|
||||
* keeps one `SessionEvent` per line for diagnostics. Reading packed rows is
|
||||
* unconditional: a log's layout never depends on this switch.
|
||||
*/
|
||||
packChunks?: boolean
|
||||
/** Physical encoding; defaults to checksummed Zstandard frames. */
|
||||
compression?: JsonlCompression
|
||||
/** Maximum cold Session preparations retained for history-to-resume reuse. */
|
||||
preparedSessionCacheSize?: number
|
||||
/** Fixed live-event coalescing window; not a backend completion deadline. */
|
||||
writeBatchMaxDelayMs?: number
|
||||
}
|
||||
|
||||
/** Opaque coordinator token for replacing bytes recovered from a torn frame. */
|
||||
interface JsonlTornMarker {
|
||||
truncateTo: number
|
||||
recoveredEvents: SessionEvent[]
|
||||
}
|
||||
|
||||
interface FileRevisionIdentity {
|
||||
readonly dev: bigint
|
||||
readonly ino: bigint
|
||||
readonly size: bigint
|
||||
readonly mtimeNs: bigint
|
||||
readonly ctimeNs: bigint
|
||||
}
|
||||
|
||||
/** Build the source-qualified revision shared by full and lightweight reads. */
|
||||
function fileRevision(identity: FileRevisionIdentity): PersistenceRevision {
|
||||
return SessionPersistenceRevision([
|
||||
identity.dev,
|
||||
identity.ino,
|
||||
identity.size,
|
||||
identity.mtimeNs,
|
||||
identity.ctimeNs,
|
||||
].join(':'))
|
||||
}
|
||||
|
||||
/** Whether a filesystem error means absence; every non-ENOENT failure must surface. */
|
||||
function isENOENT(error: unknown): boolean {
|
||||
return (error as NodeJS.ErrnoException | null)?.code === 'ENOENT'
|
||||
}
|
||||
|
||||
/**
|
||||
* The JSONL persistence backend. Load as a plugin; it registers as
|
||||
* `ctx.sessionPersistence` and (via the coordinator) installs the write-path
|
||||
* listeners. Its torn-tail marker carries the byte offset and any events
|
||||
* recovered from an incomplete final Zstandard frame.
|
||||
*/
|
||||
export class SessionPersistenceJsonl extends SessionPersistence implements PersistenceBackend<JsonlTornMarker> {
|
||||
static inject = ['sessions']
|
||||
|
||||
static Config: z<Config> = z.object({
|
||||
root: z.string().required(),
|
||||
packChunks: z.boolean().default(DEFAULT_PACK_CHUNKS),
|
||||
compression: JsonlCompressionSchema,
|
||||
preparedSessionCacheSize: z.number().step(1).min(1).default(DEFAULT_PREPARED_SESSION_CACHE_SIZE),
|
||||
writeBatchMaxDelayMs: z.number().step(1).min(1).max(MAX_WRITE_BATCH_DELAY_MS)
|
||||
.default(DEFAULT_WRITE_BATCH_MAX_DELAY_MS),
|
||||
})
|
||||
|
||||
/**
|
||||
* Backend label for coordinator diagnostics and effects. It shadows
|
||||
* `Service.name` without changing the service key captured by the base
|
||||
* constructor.
|
||||
*/
|
||||
override readonly name = 'session-persistence-jsonl'
|
||||
|
||||
private root: string
|
||||
private packChunks: boolean
|
||||
private compression: JsonlCompression
|
||||
private coordinator: PersistenceCoordinator<JsonlTornMarker>
|
||||
private rootEncodingCheck: Promise<void> | undefined
|
||||
|
||||
constructor(ctx: Context, public config: Config) {
|
||||
super(ctx)
|
||||
// Resolve once so later process.cwd() changes cannot split one backend across roots.
|
||||
this.root = resolve(config.root)
|
||||
// Programmatic wrappers may construct the backend without Schemastery normalization.
|
||||
const preparedSessionCacheSize = config.preparedSessionCacheSize
|
||||
?? DEFAULT_PREPARED_SESSION_CACHE_SIZE
|
||||
const writeBatchMaxDelayMs = config.writeBatchMaxDelayMs
|
||||
?? DEFAULT_WRITE_BATCH_MAX_DELAY_MS
|
||||
this.packChunks = config.packChunks ?? DEFAULT_PACK_CHUNKS
|
||||
this.compression = config.compression ?? DEFAULT_COMPRESSION
|
||||
this.assertUsableRoot()
|
||||
this.coordinator = new PersistenceCoordinator<JsonlTornMarker>(this.ctx, this, {
|
||||
preparedSessionCacheSize,
|
||||
writeBatchMaxDelayMs,
|
||||
})
|
||||
}
|
||||
|
||||
// Each backend keeps the typed service surface beside its storage hooks;
|
||||
// extracting these trivial forwards would add an inheritance seam.
|
||||
/* jscpd:ignore-start */
|
||||
// --- SessionPersistence service surface (delegated to the coordinator) ---
|
||||
|
||||
/** Resolve the absolute target path without touching the filesystem. */
|
||||
locate(meta: SessionHeader): SessionLocation {
|
||||
return { kind: 'jsonl', path: logPath(this.root, meta.cwd, meta.id, this.compression) }
|
||||
}
|
||||
|
||||
create(meta: SessionHeader): Promise<void> {
|
||||
return this.coordinator.create(meta)
|
||||
}
|
||||
|
||||
append(id: SessionId, events: readonly SessionEvent[]): Promise<void> {
|
||||
return this.coordinator.append(id, events)
|
||||
}
|
||||
|
||||
override prepare(id: SessionId, signal?: AbortSignal): Promise<SessionPreparation> {
|
||||
return this.coordinator.prepare(id, signal)
|
||||
}
|
||||
|
||||
load(id: SessionId): Promise<SessionInspection> {
|
||||
return this.coordinator.load(id)
|
||||
}
|
||||
|
||||
inspect(id: SessionId, signal?: AbortSignal): Promise<SessionInspection> {
|
||||
return this.coordinator.inspect(id, signal)
|
||||
}
|
||||
|
||||
// JSONL is sequential media: no loadStoredFrom hook, so the coordinator
|
||||
// parses the stored prefix (both encodings) and skips forward to fromSeq.
|
||||
readFrom(id: SessionId, fromSeq: number, signal?: AbortSignal): Promise<{ meta: SessionHeader; events: SessionEvent[] }> {
|
||||
return this.coordinator.readFrom(id, fromSeq, signal)
|
||||
}
|
||||
|
||||
// One method serves both public `list` and the backend hook; delegating it to
|
||||
// the coordinator would call this hook recursively.
|
||||
|
||||
/* jscpd:ignore-end */
|
||||
// --- PersistenceBackend hooks (the file-bytes storage primitives) ---
|
||||
|
||||
/** Read a stored prefix by id across all project directories when cwd is unknown. */
|
||||
async loadStored(id: SessionId, signal?: AbortSignal): Promise<StoredPrefix<JsonlTornMarker> | undefined> {
|
||||
signal?.throwIfAborted()
|
||||
await this.ensureRootEncoding()
|
||||
signal?.throwIfAborted()
|
||||
const path = await this.findLog(id, signal)
|
||||
if (path === undefined) return undefined
|
||||
return this.readPrefix(path, id, signal)
|
||||
}
|
||||
|
||||
/**
|
||||
* Read one log's stat-derived revision without loading its event bytes.
|
||||
* Resolving an id with unknown cwd still scans the project directories.
|
||||
*/
|
||||
async readStoredRevision(id: SessionId, signal?: AbortSignal): Promise<PersistenceRevision | undefined> {
|
||||
signal?.throwIfAborted()
|
||||
await this.ensureRootEncoding()
|
||||
signal?.throwIfAborted()
|
||||
const path = await this.findLog(id, signal)
|
||||
if (path === undefined) return undefined
|
||||
try {
|
||||
const identity = await stat(path, { bigint: true })
|
||||
signal?.throwIfAborted()
|
||||
return fileRevision(identity)
|
||||
} catch (error: unknown) {
|
||||
signal?.throwIfAborted()
|
||||
if (isENOENT(error)) return undefined
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a stored prefix and convert torn-tail state to the opaque marker the
|
||||
* coordinator can round-trip without knowing the physical encoding.
|
||||
*/
|
||||
private async readPrefix(
|
||||
path: string,
|
||||
expectedId?: SessionId,
|
||||
signal?: AbortSignal,
|
||||
): Promise<StoredPrefix<JsonlTornMarker>> {
|
||||
let buffer: Buffer
|
||||
let revision: PersistenceRevision
|
||||
for (;;) {
|
||||
signal?.throwIfAborted()
|
||||
const before = fileRevision(await stat(path, { bigint: true }))
|
||||
buffer = await readFile(path, { signal })
|
||||
signal?.throwIfAborted()
|
||||
const after = fileRevision(await stat(path, { bigint: true }))
|
||||
if (before === after) {
|
||||
revision = after
|
||||
break
|
||||
}
|
||||
}
|
||||
let prefix: Omit<StoredPrefix<JsonlTornMarker>, 'revision'>
|
||||
if (this.compression === 'zstd') {
|
||||
prefix = await this.readZstdPrefix(buffer, signal)
|
||||
} else {
|
||||
signal?.throwIfAborted()
|
||||
const { meta, events, committedBytes } = scanLog(buffer)
|
||||
signal?.throwIfAborted()
|
||||
prefix = {
|
||||
meta,
|
||||
events,
|
||||
...committedBytes < buffer.byteLength
|
||||
? { tornMarker: { truncateTo: committedBytes, recoveredEvents: [] } }
|
||||
: {},
|
||||
}
|
||||
}
|
||||
signal?.throwIfAborted()
|
||||
await this.assertStoredIdentity(path, prefix.meta, expectedId, signal)
|
||||
signal?.throwIfAborted()
|
||||
return { ...prefix, revision }
|
||||
}
|
||||
|
||||
/** Decode complete frames and retain complete JSONL records from a torn final frame. */
|
||||
private async readZstdPrefix(
|
||||
buffer: Buffer,
|
||||
signal?: AbortSignal,
|
||||
): Promise<Omit<StoredPrefix<JsonlTornMarker>, 'revision'>> {
|
||||
signal?.throwIfAborted()
|
||||
const { frames, tornStart } = scanZstdFrames(buffer)
|
||||
signal?.throwIfAborted()
|
||||
if (frames.length === 0) throw new Error('empty or header-less Zstandard session log')
|
||||
|
||||
const decoder = createZstdFrameDecoder()
|
||||
let yieldDeadline = performance.now() + ZSTD_DECODE_YIELD_INTERVAL_MS
|
||||
try {
|
||||
const decodedFrames = decoder.decode(buffer, frames)
|
||||
signal?.throwIfAborted()
|
||||
const headerFrame = decodedFrames.next()
|
||||
signal?.throwIfAborted()
|
||||
/* v8 ignore next -- a non-empty structural frame list makes the decoder yield its first frame or throw. */
|
||||
if (headerFrame.done) throw new Error('empty or header-less Zstandard session log')
|
||||
assertZstdHeaderFrame(headerFrame.value)
|
||||
const scanner = new SessionLogScanner(headerFrame.value)
|
||||
|
||||
let remainingFrames = frames.length - 1
|
||||
for (const plaintext of decodedFrames) {
|
||||
signal?.throwIfAborted()
|
||||
scanner.write(plaintext)
|
||||
remainingFrames -= 1
|
||||
if (remainingFrames > 0 && performance.now() >= yieldDeadline) {
|
||||
await scheduler.yield()
|
||||
signal?.throwIfAborted()
|
||||
yieldDeadline = performance.now() + ZSTD_DECODE_YIELD_INTERVAL_MS
|
||||
}
|
||||
}
|
||||
signal?.throwIfAborted()
|
||||
const complete = scanner.checkpoint()
|
||||
if (complete.committedBytes !== complete.inputBytes) {
|
||||
throw new Error('corrupt Zstandard session log: complete frame contains a torn JSONL record')
|
||||
}
|
||||
if (tornStart === undefined) {
|
||||
const prefix = scanner.finish()
|
||||
return { meta: prefix.meta, events: prefix.events }
|
||||
}
|
||||
|
||||
let recoveredPlaintext: Buffer = Buffer.alloc(0)
|
||||
try {
|
||||
signal?.throwIfAborted()
|
||||
recoveredPlaintext = await decompressZstdPrefix(buffer.subarray(tornStart))
|
||||
} catch {
|
||||
/* v8 ignore next -- decoder failure plus concurrent abort is timing-dependent */
|
||||
if (signal?.aborted) signal.throwIfAborted()
|
||||
// A structurally incomplete final frame may end before Node's decoder can
|
||||
// emit any plaintext; the complete prior frames remain recoverable.
|
||||
}
|
||||
signal?.throwIfAborted()
|
||||
scanner.write(recoveredPlaintext)
|
||||
const recoveredPrefix = scanner.finish()
|
||||
signal?.throwIfAborted()
|
||||
return {
|
||||
meta: recoveredPrefix.meta,
|
||||
events: recoveredPrefix.events,
|
||||
tornMarker: {
|
||||
truncateTo: tornStart,
|
||||
recoveredEvents: recoveredPrefix.events.slice(complete.eventCount),
|
||||
},
|
||||
}
|
||||
} catch (error) {
|
||||
/* v8 ignore next -- decoder failure plus concurrent abort is timing-dependent */
|
||||
if (signal?.aborted) signal.throwIfAborted()
|
||||
throw error
|
||||
} finally {
|
||||
decoder.close()
|
||||
}
|
||||
}
|
||||
|
||||
/** Durably append a batch, lazily materializing the file when not yet present. */
|
||||
async appendBatch(meta: SessionHeader, events: readonly SessionEvent[], isMaterialized: boolean): Promise<void> {
|
||||
await this.ensureRootEncoding()
|
||||
if (isMaterialized) {
|
||||
await this.appendLines(meta, events)
|
||||
} else {
|
||||
await this.materialize(meta, events)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Make a crash repair durable: truncate a torn tail, restore complete events
|
||||
* decoded from it, then append synthetic closers. Two fsync'd steps — the seam
|
||||
* does not require this to be atomic.
|
||||
*/
|
||||
async commitRepair(
|
||||
meta: SessionHeader,
|
||||
tornMarker: JsonlTornMarker | undefined,
|
||||
closers: readonly SessionEvent[],
|
||||
): Promise<void> {
|
||||
if (tornMarker !== undefined) await this.repair(meta, tornMarker.truncateTo)
|
||||
const repairedEvents = [...(tornMarker?.recoveredEvents ?? []), ...closers]
|
||||
if (repairedEvents.length > 0) await this.appendLines(meta, repairedEvents)
|
||||
}
|
||||
|
||||
/** List valid unique stored sessions' metadata (header line only — no full-log parse). */
|
||||
async list(signal?: AbortSignal): Promise<SessionHeader[]> {
|
||||
return (await this.listArtifacts(signal)).map(artifact => artifact.header)
|
||||
}
|
||||
|
||||
/** List metadata plus a stat-derived identity for each append-only log. */
|
||||
async listSnapshots(signal?: AbortSignal): Promise<SessionPersistenceSnapshot[]> {
|
||||
const snapshots: SessionPersistenceSnapshot[] = []
|
||||
for (const artifact of await this.listArtifacts(signal)) {
|
||||
signal?.throwIfAborted()
|
||||
try {
|
||||
const identity = await stat(artifact.path, { bigint: true })
|
||||
signal?.throwIfAborted()
|
||||
snapshots.push({
|
||||
header: artifact.header,
|
||||
revision: fileRevision(identity),
|
||||
})
|
||||
} catch (error: unknown) {
|
||||
signal?.throwIfAborted()
|
||||
if (!isENOENT(error)) throw error
|
||||
}
|
||||
}
|
||||
signal?.throwIfAborted()
|
||||
return snapshots
|
||||
}
|
||||
|
||||
private async listArtifacts(signal?: AbortSignal): Promise<Array<{ header: SessionHeader; path: string }>> {
|
||||
signal?.throwIfAborted()
|
||||
await this.ensureRootEncoding()
|
||||
signal?.throwIfAborted()
|
||||
const artifacts: Array<{ header: SessionHeader; path: string }> = []
|
||||
const ids = new Set<SessionId>()
|
||||
for (const project of await this.listProjectDirs(signal)) {
|
||||
signal?.throwIfAborted()
|
||||
for (const dir of await this.listSessionDirs(project, signal)) {
|
||||
signal?.throwIfAborted()
|
||||
const opposite = join(dir, `session${logSuffix(this.oppositeCompression())}`)
|
||||
const oppositeExists = await this.exists(opposite)
|
||||
signal?.throwIfAborted()
|
||||
if (oppositeExists) throw this.encodingMismatch(opposite)
|
||||
const path = join(dir, `session${logSuffix(this.compression)}`)
|
||||
const pathExists = await this.exists(path)
|
||||
signal?.throwIfAborted()
|
||||
if (!pathExists) continue
|
||||
// Read only headers so listing scales with session count, not log size.
|
||||
const first = this.compression === 'zstd'
|
||||
? await this.readFirstZstdLine(path, signal)
|
||||
: await this.readFirstLine(path, signal)
|
||||
signal?.throwIfAborted()
|
||||
if (first === undefined) continue // empty/half-written file
|
||||
const meta = parseHeaderMeta(first)
|
||||
if (meta === undefined) continue // not a session header
|
||||
await this.assertStoredIdentity(path, meta, undefined, signal)
|
||||
signal?.throwIfAborted()
|
||||
if (ids.has(meta.id)) {
|
||||
throw new Error(`duplicate JSONL session id "${meta.id}" appears in multiple project directories`)
|
||||
}
|
||||
ids.add(meta.id)
|
||||
artifacts.push({ header: meta, path })
|
||||
}
|
||||
}
|
||||
signal?.throwIfAborted()
|
||||
return artifacts
|
||||
}
|
||||
|
||||
// --- materialization / append / repair (file mechanics) ---
|
||||
|
||||
/** Atomically write the header line + first batch (temp-write, fsync, publish). */
|
||||
private async materialize(meta: SessionHeader, events: readonly SessionEvent[]): Promise<void> {
|
||||
const project = projectDir(this.root, meta.cwd)
|
||||
const dir = sessionDir(this.root, meta.cwd, meta.id)
|
||||
const finalPath = logPath(this.root, meta.cwd, meta.id, this.compression)
|
||||
await this.rejectOppositeArtifact(meta.cwd, meta.id)
|
||||
const content = await this.encodeMaterialization(meta, events)
|
||||
/* v8 ignore next -- native Windows coverage exercises this platform dispatch; Linux covers the POSIX peer */
|
||||
if (process.platform === 'win32') {
|
||||
await this.materializeWin32(project, dir, finalPath, meta.id, content)
|
||||
} else {
|
||||
await this.materializePosix(project, dir, finalPath, meta.id, content)
|
||||
}
|
||||
}
|
||||
|
||||
/* v8 ignore start -- Windows uses the Win32 durable-publish path; POSIX coverage exercises this peer. */
|
||||
private async materializePosix(
|
||||
project: string,
|
||||
dir: string,
|
||||
finalPath: string,
|
||||
id: SessionId,
|
||||
content: Buffer | string,
|
||||
): Promise<void> {
|
||||
await mkdir(this.root, { recursive: true, mode: 0o700 })
|
||||
await this.syncDirPosix(dirname(this.root))
|
||||
await mkdir(project, { recursive: true, mode: 0o700 })
|
||||
await this.syncDirPosix(this.root)
|
||||
await mkdir(dir, { recursive: true, mode: 0o700 })
|
||||
await this.syncDirPosix(project)
|
||||
await this.rejectExistingLog(finalPath, id)
|
||||
const tmp = await this.writeSyncedTempFile(finalPath, content)
|
||||
// Publish via link()+unlink(), NOT rename(): link fails with EEXIST if the
|
||||
// final path already exists, so two processes materializing the same id
|
||||
// concurrently cannot clobber each other. rename() would silently overwrite.
|
||||
let linked = false
|
||||
try {
|
||||
await link(tmp, finalPath)
|
||||
linked = true
|
||||
} finally {
|
||||
// Remove an unpublished temp on failure. After publication, defer cleanup
|
||||
// until the directory entry is durable so cleanup cannot reject a live log.
|
||||
/* v8 ignore next -- link failure is the TOCTOU/IO race guarded above; not reachable in test */
|
||||
if (!linked) await rm(tmp, { force: true })
|
||||
}
|
||||
// link() succeeded — the log is published. fsync the directory so the new
|
||||
// entry survives a power loss: the new link is not crash-durable until the
|
||||
// parent directory's metadata is synced.
|
||||
await this.syncDirPosix(dir)
|
||||
// Best-effort temp cleanup: the log is already published and durable, so a
|
||||
// failure to remove the (now-redundant) temp hard link must NOT reject the
|
||||
// append. Swallow only the rm failure; nothing else of consequence runs here.
|
||||
try {
|
||||
await rm(tmp, { force: true })
|
||||
} catch {
|
||||
/* v8 ignore next -- redundant temp link; publish already durable, rm failure is an unreachable IO edge */
|
||||
}
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
|
||||
/* v8 ignore start -- native Windows coverage exercises this integration path */
|
||||
private async materializeWin32(
|
||||
project: string,
|
||||
dir: string,
|
||||
finalPath: string,
|
||||
id: SessionId,
|
||||
content: Buffer | string,
|
||||
): Promise<void> {
|
||||
await ensureDurableDirectoryWin32(this.root)
|
||||
await ensureDurableDirectoryWin32(project)
|
||||
await ensureDurableDirectoryWin32(dir)
|
||||
await this.rejectExistingLog(finalPath, id)
|
||||
const tmp = await this.writeSyncedTempFile(finalPath, content)
|
||||
try {
|
||||
await publishNewFileWin32(tmp, finalPath)
|
||||
} catch (error) {
|
||||
await rm(tmp, { force: true })
|
||||
throw error
|
||||
}
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
|
||||
private async rejectExistingLog(finalPath: string, id: SessionId): Promise<void> {
|
||||
// Never publish over an existing committed log: materialize is the first
|
||||
// write of a session the backend believes is new. A file here means a
|
||||
// different session shares this id on disk — reject loudly. (createCore
|
||||
// already guards the create path, so this is unreachable-in-practice TOCTOU
|
||||
// defense.)
|
||||
/* v8 ignore next 3 -- createCore guards collisions before materialize; this is a TOCTOU backstop */
|
||||
if (await this.exists(finalPath)) {
|
||||
throw new Error(`refusing to materialize "${id}": a log already exists on disk (load/resume it instead)`)
|
||||
}
|
||||
}
|
||||
|
||||
private async writeSyncedTempFile(finalPath: string, content: Buffer | string): Promise<string> {
|
||||
const tmp = `${finalPath}.${randomBytes(6).toString('hex')}.tmp`
|
||||
const handle = await open(tmp, 'wx', 0o600)
|
||||
try {
|
||||
await handle.writeFile(content)
|
||||
await handle.sync()
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
return tmp
|
||||
}
|
||||
|
||||
/** Encode the header and first batch without combining their frame boundaries. */
|
||||
private async encodeMaterialization(meta: SessionHeader, events: readonly SessionEvent[]): Promise<Buffer | string> {
|
||||
const header = JSON.stringify(toHeaderLine(meta)) + '\n'
|
||||
const body = eventLines(events, this.packChunks) + '\n'
|
||||
if (this.compression === 'none') return header + body
|
||||
const headerFrame = await compressZstdFrame(header)
|
||||
const eventFrame = await compressZstdFrame(body)
|
||||
return Buffer.concat([headerFrame, eventFrame])
|
||||
}
|
||||
|
||||
/** Encode one durable append batch in the configured physical representation. */
|
||||
private async encodeEventBatch(events: readonly SessionEvent[]): Promise<Buffer | string> {
|
||||
const body = eventLines(events, this.packChunks) + '\n'
|
||||
return this.compression === 'zstd' ? compressZstdFrame(body) : body
|
||||
}
|
||||
|
||||
/** fsync a POSIX directory so a just-created/renamed entry is crash-durable. */
|
||||
/* v8 ignore start -- Windows uses write-through namespace operations; POSIX coverage exercises directory fsync. */
|
||||
private async syncDirPosix(dir: string): Promise<void> {
|
||||
const handle = await open(dir, 'r')
|
||||
try {
|
||||
await handle.sync()
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
|
||||
/**
|
||||
* Append and fsync event lines. On a partial write or sync failure, restore the
|
||||
* previous size before rethrowing because the unchanged cursor will retry the
|
||||
* batch; leaving partial bytes would create duplicate sequence numbers.
|
||||
*/
|
||||
private async appendLines(meta: SessionHeader, events: readonly SessionEvent[]): Promise<void> {
|
||||
const content = await this.encodeEventBatch(events)
|
||||
const path = logPath(this.root, meta.cwd, meta.id, this.compression)
|
||||
const handle = await open(path, 'a')
|
||||
let closed = false
|
||||
const closeAppendHandle = async (): Promise<void> => {
|
||||
if (closed) return
|
||||
closed = true
|
||||
await handle.close()
|
||||
}
|
||||
|
||||
try {
|
||||
const { size: before } = await handle.stat()
|
||||
try {
|
||||
await handle.writeFile(content)
|
||||
await handle.sync()
|
||||
} catch (error) {
|
||||
try {
|
||||
await closeAppendHandle()
|
||||
await this.rollbackAppend(path, before)
|
||||
} catch (rollbackError) {
|
||||
throw new AggregateError([error, rollbackError], `failed to roll back append to "${path}"`)
|
||||
}
|
||||
throw error
|
||||
}
|
||||
} finally {
|
||||
await closeAppendHandle()
|
||||
}
|
||||
}
|
||||
|
||||
private async rollbackAppend(path: string, size: number): Promise<void> {
|
||||
const handle = await open(path, 'r+')
|
||||
try {
|
||||
await handle.truncate(size)
|
||||
await handle.sync()
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
}
|
||||
|
||||
/** Truncate the log file to `offset` bytes and fsync (discard the crash tail). */
|
||||
private async repair(meta: SessionHeader, offset: number): Promise<void> {
|
||||
const path = logPath(this.root, meta.cwd, meta.id, this.compression)
|
||||
await truncate(path, offset)
|
||||
const handle = await open(path, 'r+')
|
||||
try {
|
||||
await handle.sync()
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
}
|
||||
|
||||
// --- discovery helpers ---
|
||||
|
||||
/**
|
||||
* Read the first newline-terminated line of a file without loading the whole
|
||||
* file. Returns undefined if the file is empty or has no complete first line.
|
||||
* Reads in bounded chunks so a huge log costs only the header read.
|
||||
*/
|
||||
private async readFirstLine(path: string, signal?: AbortSignal): Promise<string | undefined> {
|
||||
signal?.throwIfAborted()
|
||||
const handle = await open(path, 'r')
|
||||
try {
|
||||
signal?.throwIfAborted()
|
||||
const chunks: Buffer[] = []
|
||||
const buf = Buffer.alloc(8192)
|
||||
for (;;) {
|
||||
signal?.throwIfAborted()
|
||||
const { bytesRead } = await handle.read(buf, 0, buf.length, null)
|
||||
signal?.throwIfAborted()
|
||||
if (bytesRead === 0) return undefined // EOF with no newline → no complete line
|
||||
const slice = buf.subarray(0, bytesRead)
|
||||
const nl = slice.indexOf(0x0a)
|
||||
if (nl !== -1) {
|
||||
chunks.push(slice.subarray(0, nl))
|
||||
signal?.throwIfAborted()
|
||||
return Buffer.concat(chunks).toString('utf8')
|
||||
}
|
||||
chunks.push(Buffer.from(slice))
|
||||
}
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
}
|
||||
|
||||
/** Read and validate only the independently compressed header frame. */
|
||||
private async readFirstZstdLine(path: string, signal?: AbortSignal): Promise<string | undefined> {
|
||||
signal?.throwIfAborted()
|
||||
const handle = await open(path, 'r')
|
||||
try {
|
||||
signal?.throwIfAborted()
|
||||
let content = Buffer.alloc(0)
|
||||
const chunk = Buffer.alloc(8192)
|
||||
for (;;) {
|
||||
signal?.throwIfAborted()
|
||||
const { bytesRead } = await handle.read(chunk, 0, chunk.length, null)
|
||||
signal?.throwIfAborted()
|
||||
if (bytesRead === 0) return undefined
|
||||
signal?.throwIfAborted()
|
||||
content = Buffer.concat([content, chunk.subarray(0, bytesRead)])
|
||||
signal?.throwIfAborted()
|
||||
const first = scanZstdFrames(content, 1).frames[0]
|
||||
signal?.throwIfAborted()
|
||||
if (first === undefined) continue
|
||||
let plaintext: Buffer
|
||||
try {
|
||||
signal?.throwIfAborted()
|
||||
plaintext = await decompressZstdFrame(content.subarray(first.start, first.end))
|
||||
} catch (error) {
|
||||
/* v8 ignore next -- decoder failure plus concurrent abort is timing-dependent */
|
||||
if (signal?.aborted) signal.throwIfAborted()
|
||||
throw new Error('corrupt Zstandard session log: header frame failed validation', { cause: error })
|
||||
}
|
||||
signal?.throwIfAborted()
|
||||
assertZstdHeaderFrame(plaintext)
|
||||
return plaintext.subarray(0, -1).toString('utf8')
|
||||
}
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
}
|
||||
|
||||
/** Find the unique physical log for an id across every project directory. */
|
||||
private async findLog(id: SessionId, signal?: AbortSignal): Promise<string | undefined> {
|
||||
const matches: string[] = []
|
||||
for (const project of await this.listProjectDirs(signal)) {
|
||||
signal?.throwIfAborted()
|
||||
await this.rejectLegacyFlatArtifact(project, id, signal)
|
||||
signal?.throwIfAborted()
|
||||
const dir = join(project, encodeSegment(id))
|
||||
const path = join(dir, `session${logSuffix(this.compression)}`)
|
||||
const opposite = join(dir, `session${logSuffix(this.oppositeCompression())}`)
|
||||
const oppositeExists = await this.exists(opposite)
|
||||
signal?.throwIfAborted()
|
||||
if (oppositeExists) throw this.encodingMismatch(opposite)
|
||||
const pathExists = await this.exists(path)
|
||||
signal?.throwIfAborted()
|
||||
if (pathExists) matches.push(path)
|
||||
}
|
||||
if (matches.length > 1) {
|
||||
throw new Error(`duplicate JSONL session id "${id}" appears in multiple project directories`)
|
||||
}
|
||||
signal?.throwIfAborted()
|
||||
return matches[0]
|
||||
}
|
||||
|
||||
/** Require an existing configured root to be a readable directory. */
|
||||
private assertUsableRoot(): void {
|
||||
try {
|
||||
readdirSync(this.root)
|
||||
} catch (error) {
|
||||
if (isENOENT(error)) return
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/** Reject metadata that does not identify the selected physical log. */
|
||||
private async assertStoredIdentity(
|
||||
path: string,
|
||||
meta: SessionHeader,
|
||||
expectedId?: SessionId,
|
||||
signal?: AbortSignal,
|
||||
): Promise<void> {
|
||||
signal?.throwIfAborted()
|
||||
if (expectedId !== undefined && meta.id !== expectedId) {
|
||||
throw new Error(`corrupt session log "${path}": requested id "${expectedId}" does not match header id "${meta.id}"`)
|
||||
}
|
||||
let expectedPath: string
|
||||
try {
|
||||
expectedPath = logPath(this.root, meta.cwd, meta.id, this.compression)
|
||||
} catch (error) {
|
||||
throw new Error(`corrupt session log "${path}": header id cannot name a storage path`, { cause: error })
|
||||
}
|
||||
if (path !== expectedPath && !await this.sameFile(path, expectedPath, signal)) {
|
||||
throw new Error(`corrupt session log "${path}": header id "${meta.id}" and cwd identify "${expectedPath}"`)
|
||||
}
|
||||
signal?.throwIfAborted()
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether two path spellings resolve to the same physical file. This admits
|
||||
* case aliases on case-insensitive filesystems without weakening identity
|
||||
* checks on case-sensitive stores.
|
||||
*/
|
||||
private async sameFile(path: string, expectedPath: string, signal?: AbortSignal): Promise<boolean> {
|
||||
signal?.throwIfAborted()
|
||||
try {
|
||||
const [actual, expected] = await Promise.all([realpath(path), realpath(expectedPath)])
|
||||
signal?.throwIfAborted()
|
||||
return actual === expected
|
||||
} catch (error) {
|
||||
signal?.throwIfAborted()
|
||||
/* v8 ignore else -- non-ENOENT realpath failures require an external permission or I/O fault */
|
||||
if (isENOENT(error)) return false
|
||||
/* v8 ignore next -- non-ENOENT realpath failures are external I/O faults, propagated unchanged */
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/** The human-readable project directories under the configured root. */
|
||||
private async listProjectDirs(signal?: AbortSignal): Promise<string[]> {
|
||||
try {
|
||||
signal?.throwIfAborted()
|
||||
const entries = await readdir(this.root, { withFileTypes: true })
|
||||
signal?.throwIfAborted()
|
||||
return entries.filter(e => e.isDirectory()).map(e => join(this.root, e.name))
|
||||
} catch (error) {
|
||||
// Only an absent root means no sessions; rethrow every other I/O failure.
|
||||
if (isENOENT(error)) return []
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/** List session-owned directories and reject the obsolete flat-file layout. */
|
||||
private async listSessionDirs(project: string, signal?: AbortSignal): Promise<string[]> {
|
||||
signal?.throwIfAborted()
|
||||
const entries = await readdir(project, { withFileTypes: true })
|
||||
signal?.throwIfAborted()
|
||||
const legacy = entries.find(entry =>
|
||||
entry.isFile() && (entry.name.endsWith('.jsonl') || entry.name.endsWith('.jsonl.zstd')))
|
||||
if (legacy !== undefined) throw this.legacyLayout(join(project, legacy.name))
|
||||
return entries.filter(entry => entry.isDirectory()).map(entry => join(project, entry.name))
|
||||
}
|
||||
|
||||
/** Reject a root that already belongs to the other physical encoding. */
|
||||
private ensureRootEncoding(): Promise<void> {
|
||||
this.rootEncodingCheck ??= this.checkRootEncoding()
|
||||
return this.rootEncodingCheck
|
||||
}
|
||||
|
||||
private async checkRootEncoding(): Promise<void> {
|
||||
for (const project of await this.listProjectDirs()) {
|
||||
for (const dir of await this.listSessionDirs(project)) {
|
||||
const incompatible = join(dir, `session${logSuffix(this.oppositeCompression())}`)
|
||||
if (await this.exists(incompatible)) throw this.encodingMismatch(incompatible)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async rejectLegacyFlatArtifact(
|
||||
project: string,
|
||||
id: SessionId,
|
||||
signal?: AbortSignal,
|
||||
): Promise<void> {
|
||||
signal?.throwIfAborted()
|
||||
const encoded = encodeSegment(id)
|
||||
for (const compression of ['zstd', 'none'] as const) {
|
||||
const path = join(project, encoded + logSuffix(compression))
|
||||
const artifactExists = await this.exists(path)
|
||||
signal?.throwIfAborted()
|
||||
if (artifactExists) throw this.legacyLayout(path)
|
||||
}
|
||||
}
|
||||
|
||||
private async rejectOppositeArtifact(cwd: string | undefined, id: SessionId): Promise<void> {
|
||||
const path = logPath(this.root, cwd, id, this.oppositeCompression())
|
||||
if (await this.exists(path)) throw this.encodingMismatch(path)
|
||||
}
|
||||
|
||||
private oppositeCompression(): JsonlCompression {
|
||||
return this.compression === 'zstd' ? 'none' : 'zstd'
|
||||
}
|
||||
|
||||
private encodingMismatch(path: string): Error {
|
||||
return new Error(
|
||||
`session artifact ${JSON.stringify(path)} uses ${logSuffix(this.oppositeCompression())}, `
|
||||
+ `but this backend is configured for compression ${JSON.stringify(this.compression)}; `
|
||||
+ 'use a separate root or select the matching compression mode',
|
||||
)
|
||||
}
|
||||
|
||||
private legacyLayout(path: string): Error {
|
||||
return new Error(
|
||||
`session artifact ${JSON.stringify(path)} uses the unsupported flat-file layout; `
|
||||
+ 'use a separate root or move it into a project/session directory before loading',
|
||||
)
|
||||
}
|
||||
|
||||
private async exists(path: string): Promise<boolean> {
|
||||
try {
|
||||
const handle = await open(path, 'r')
|
||||
await handle.close()
|
||||
return true
|
||||
} catch (error) {
|
||||
// Only ENOENT means absent. A permission/I/O error must surface rather
|
||||
// than letting load or collision checks proceed under false absence.
|
||||
// Windows reports ENOENT, not ENOTDIR, for `regular-file/child`; verify
|
||||
// the immediate parent so a blocked session directory remains a storage fault.
|
||||
/* v8 ignore else -- Windows reports file-valued parents as ENOENT; POSIX covers direct ENOTDIR. */
|
||||
if (isENOENT(error)) {
|
||||
await this.assertLogParentAllowsAbsence(path)
|
||||
return false
|
||||
}
|
||||
/* v8 ignore next -- Windows repairs ENOTDIR from ENOENT above; POSIX covers direct ENOTDIR. */
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/* v8 ignore start -- native Windows coverage exercises this repair; POSIX open reports ENOTDIR before this point. */
|
||||
private async assertLogParentAllowsAbsence(path: string): Promise<void> {
|
||||
try {
|
||||
const parent = dirname(path)
|
||||
const info = await stat(parent)
|
||||
if (info.isDirectory()) return
|
||||
const error = new Error(`ENOTDIR: parent path exists but is not a directory: ${parent}`) as NodeJS.ErrnoException
|
||||
error.code = 'ENOTDIR'
|
||||
error.path = parent
|
||||
throw error
|
||||
} catch (error) {
|
||||
if (isENOENT(error)) return
|
||||
throw error
|
||||
}
|
||||
}
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
|
||||
export default SessionPersistenceJsonl
|
||||
30
packages/session/session-persistence-jsonl/src/invariant.ts
Normal file
30
packages/session/session-persistence-jsonl/src/invariant.ts
Normal file
@@ -0,0 +1,30 @@
|
||||
/**
|
||||
* Package-owned invariant companion for `@deepseek-ai/dsh-session-persistence-jsonl`.
|
||||
* @module @deepseek-ai/dsh-session-persistence-jsonl/invariant
|
||||
*/
|
||||
|
||||
/* jscpd:ignore-start */
|
||||
import type { Context } from 'cordis'
|
||||
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
|
||||
|
||||
const PACKAGE_NAME = '@deepseek-ai/dsh-session-persistence-jsonl'
|
||||
|
||||
/** Cordis companion plugin name. */
|
||||
export const name = 'session-persistence-jsonl-invariant'
|
||||
/** Service required before the companion can reserve package ownership. */
|
||||
export const inject = ['invariants']
|
||||
|
||||
/**
|
||||
* No runtime invariant: persistence correctness requires backend round-trip and crash-tail tests;
|
||||
* this package exposes no continuously observable in-process relation.
|
||||
*/
|
||||
const install: InvariantInstaller = () => {}
|
||||
|
||||
/**
|
||||
* Register this package's invariant companion.
|
||||
* @param ctx - Cordis context carrying the invariant service.
|
||||
* @returns the installed registration's disposer after setup succeeds.
|
||||
*/
|
||||
export const apply = (ctx: Context): Promise<() => void> =>
|
||||
Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
|
||||
/* jscpd:ignore-end */
|
||||
155
packages/session/session-persistence-jsonl/src/win32.ts
Normal file
155
packages/session/session-persistence-jsonl/src/win32.ts
Normal file
@@ -0,0 +1,155 @@
|
||||
/**
|
||||
* Windows durable namespace helpers for the JSONL backend.
|
||||
*
|
||||
* POSIX publishes a newly-created log by creating a directory entry and then
|
||||
* fsyncing the parent directory. Windows does not expose that parent-directory
|
||||
* fsync contract through Node, so the Windows path uses the native durable
|
||||
* namespace primitive instead: create a staging object in the target directory
|
||||
* and publish it with `MoveFileExW(..., MOVEFILE_WRITE_THROUGH)` without
|
||||
* replacement or cross-volume copy fallback.
|
||||
*
|
||||
* @module dsh-session-persistence-jsonl/win32
|
||||
*/
|
||||
|
||||
import { mkdtemp, rm, stat } from 'node:fs/promises'
|
||||
import { join, parse, resolve, toNamespacedPath } from 'node:path'
|
||||
|
||||
type MoveFileExW = (existing: string, replacement: string, flags: number) => number
|
||||
type GetLastError = () => number
|
||||
|
||||
interface Win32Bindings {
|
||||
moveFileExW: MoveFileExW
|
||||
getLastError: GetLastError
|
||||
}
|
||||
|
||||
interface Win32ErrnoException extends NodeJS.ErrnoException {
|
||||
win32Code: number
|
||||
dest: string
|
||||
}
|
||||
|
||||
const MOVEFILE_WRITE_THROUGH = 0x00000008
|
||||
const ERROR_FILE_NOT_FOUND = 2
|
||||
const ERROR_PATH_NOT_FOUND = 3
|
||||
const ERROR_ACCESS_DENIED = 5
|
||||
const ERROR_NOT_SAME_DEVICE = 17
|
||||
const ERROR_FILE_EXISTS = 80
|
||||
const ERROR_INVALID_NAME = 123
|
||||
const ERROR_ALREADY_EXISTS = 183
|
||||
|
||||
let bindings: Win32Bindings | undefined
|
||||
|
||||
/** Load the small Win32 surface lazily so non-Windows processes never load Koffi. */
|
||||
async function win32(): Promise<Win32Bindings> {
|
||||
if (bindings !== undefined) return bindings
|
||||
const koffi = (await import('koffi')).default
|
||||
const kernel32 = koffi.load('kernel32.dll')
|
||||
bindings = {
|
||||
moveFileExW: kernel32.func('__stdcall', 'MoveFileExW', 'int', ['str16', 'str16', 'uint']) as MoveFileExW,
|
||||
getLastError: kernel32.func('__stdcall', 'GetLastError', 'uint', []) as GetLastError,
|
||||
}
|
||||
return bindings
|
||||
}
|
||||
|
||||
function errnoCode(win32Code: number): string {
|
||||
switch (win32Code) {
|
||||
case ERROR_FILE_NOT_FOUND:
|
||||
case ERROR_PATH_NOT_FOUND:
|
||||
return 'ENOENT'
|
||||
case ERROR_ACCESS_DENIED:
|
||||
return 'EACCES'
|
||||
case ERROR_NOT_SAME_DEVICE:
|
||||
return 'EXDEV'
|
||||
case ERROR_FILE_EXISTS:
|
||||
case ERROR_ALREADY_EXISTS:
|
||||
return 'EEXIST'
|
||||
case ERROR_INVALID_NAME:
|
||||
return 'EINVAL'
|
||||
default:
|
||||
return 'EIO'
|
||||
}
|
||||
}
|
||||
|
||||
function win32Error(syscall: string, win32Code: number, path: string, dest: string): Win32ErrnoException {
|
||||
const code = errnoCode(win32Code)
|
||||
const error = new Error(`${syscall} ${code} (Win32 ${win32Code}): ${path} -> ${dest}`) as Win32ErrnoException
|
||||
error.code = code
|
||||
error.errno = win32Code
|
||||
error.syscall = syscall
|
||||
error.path = path
|
||||
error.dest = dest
|
||||
error.win32Code = win32Code
|
||||
return error
|
||||
}
|
||||
|
||||
function isENOENT(error: unknown): boolean {
|
||||
return (error as NodeJS.ErrnoException | null)?.code === 'ENOENT'
|
||||
}
|
||||
|
||||
function isEEXIST(error: unknown): boolean {
|
||||
return (error as NodeJS.ErrnoException | null)?.code === 'EEXIST'
|
||||
}
|
||||
|
||||
async function assertDirectory(path: string): Promise<boolean> {
|
||||
try {
|
||||
// A bare drive root is already short, and Node rejects its extended-length
|
||||
// spelling as EISDIR. Descendants retain the namespace for long-path probes.
|
||||
const probe = path === parse(path).root ? path : toNamespacedPath(path)
|
||||
const info = await stat(probe)
|
||||
if (info.isDirectory()) return true
|
||||
const error = new Error(`path exists but is not a directory: ${path}`) as NodeJS.ErrnoException
|
||||
error.code = 'ENOTDIR'
|
||||
error.path = path
|
||||
throw error
|
||||
} catch (error) {
|
||||
if (isENOENT(error)) return false
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Publish `existing` at `replacement` with Windows write-through rename
|
||||
* semantics. The destination must not already exist; the move must stay within
|
||||
* the volume (no copy fallback flag is set).
|
||||
* @param existing - the synced staging path to move.
|
||||
* @param replacement - the final path, which must not already exist.
|
||||
*/
|
||||
export async function publishNewFileWin32(existing: string, replacement: string): Promise<void> {
|
||||
const api = await win32()
|
||||
const ok = api.moveFileExW(toNamespacedPath(existing), toNamespacedPath(replacement), MOVEFILE_WRITE_THROUGH)
|
||||
if (ok === 0) throw win32Error('MoveFileExW', api.getLastError(), existing, replacement)
|
||||
}
|
||||
|
||||
/**
|
||||
* Create `target` and its missing ancestors with durable Windows namespace
|
||||
* publication. Each missing directory is first created as a random staging
|
||||
* sibling, then moved to its final name with `MOVEFILE_WRITE_THROUGH`; races
|
||||
* with another creator are accepted only after verifying the winner is a
|
||||
* directory.
|
||||
* @param target - the absolute directory path to create durably when absent.
|
||||
*/
|
||||
export async function ensureDurableDirectoryWin32(target: string): Promise<void> {
|
||||
const absolute = resolve(target)
|
||||
const root = parse(absolute).root
|
||||
await assertDirectory(root)
|
||||
|
||||
const segments = absolute.slice(root.length).split(/[\\/]+/).filter(part => part.length > 0)
|
||||
let current = root
|
||||
for (const segment of segments) {
|
||||
const next = join(current, segment)
|
||||
if (!await assertDirectory(next)) await createLeafDirectoryWin32(current, next)
|
||||
current = next
|
||||
}
|
||||
}
|
||||
|
||||
async function createLeafDirectoryWin32(parent: string, target: string): Promise<void> {
|
||||
// Keep the staging component independent of the target basename so a legal
|
||||
// 255-byte target component does not make mkdtemp's sibling name too long.
|
||||
const staging = await mkdtemp(toNamespacedPath(join(parent, '.dsh-mkdir-')))
|
||||
try {
|
||||
await publishNewFileWin32(staging, target)
|
||||
} catch (error) {
|
||||
await rm(staging, { recursive: true, force: true })
|
||||
if (isEEXIST(error) && await assertDirectory(target)) return
|
||||
throw error
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,178 @@
|
||||
/**
|
||||
* Node-private synchronous Zstandard frame decoder optimization.
|
||||
* @module dsh-session-persistence-jsonl/zstd-private-decoder
|
||||
*/
|
||||
|
||||
import { constants as bufferConstants } from 'node:buffer'
|
||||
import { createZstdDecompress } from 'node:zlib'
|
||||
import type { ZstdFrameDecoder, ZstdFrameRange } from './zstd.ts'
|
||||
|
||||
const DECODE_CHUNK_SIZE = 1024 * 1024
|
||||
|
||||
interface NodeZstdPrivateHandle {
|
||||
writeSync(
|
||||
flushFlag: number,
|
||||
input: Buffer,
|
||||
inputOffset: number,
|
||||
inputLength: number,
|
||||
output: Buffer,
|
||||
outputOffset: number,
|
||||
outputLength: number,
|
||||
): void
|
||||
}
|
||||
|
||||
type NodeZstdPrivateWriteState = Uint32Array & { 0: number; 1: number }
|
||||
|
||||
interface NodeZstdPrivateState {
|
||||
[key: symbol]: unknown
|
||||
_handle: NodeZstdPrivateHandle | null
|
||||
_writeState: NodeZstdPrivateWriteState
|
||||
_defaultFlushFlag: number
|
||||
}
|
||||
|
||||
type NodeZstdPrivateStream = ReturnType<typeof createZstdDecompress> & NodeZstdPrivateState
|
||||
|
||||
/** Return the stream with its observed private Node contract, or reject that optimization. */
|
||||
function privateZstdStream(
|
||||
stream: ReturnType<typeof createZstdDecompress>,
|
||||
): { stream: NodeZstdPrivateStream; errorKey: symbol } | undefined {
|
||||
const candidate = stream as unknown as Partial<NodeZstdPrivateState>
|
||||
const handle = candidate._handle
|
||||
const errorKey = Reflect.ownKeys(stream).find((key): key is symbol => (
|
||||
typeof key === 'symbol' && key.description === 'kError'
|
||||
))
|
||||
/* v8 ignore next -- one test runtime exposes one Node-private shape; the Node 22/24/26 matrix checks compatibility. */
|
||||
if (
|
||||
typeof handle !== 'object' || handle === null
|
||||
|| typeof (handle as { writeSync?: unknown }).writeSync !== 'function'
|
||||
|| !(candidate._writeState instanceof Uint32Array)
|
||||
|| candidate._writeState.length < 2
|
||||
|| typeof candidate._defaultFlushFlag !== 'number'
|
||||
|| errorKey === undefined
|
||||
|| candidate[errorKey] !== null
|
||||
) return undefined
|
||||
return { stream: stream as NodeZstdPrivateStream, errorKey }
|
||||
}
|
||||
|
||||
/**
|
||||
* Synchronous multi-frame decoder backed by one Node Zstd stream handle. Node
|
||||
* exposes synchronous decoding only as a one-shot API, so this adapter uses
|
||||
* the stream's private handle contract to reuse its native context and output
|
||||
* chunks across frames.
|
||||
*/
|
||||
export class NodePrivateZstdFrameDecoder implements ZstdFrameDecoder {
|
||||
private readonly output = Buffer.allocUnsafe(DECODE_CHUNK_SIZE)
|
||||
private decoderError?: Error
|
||||
private started = false
|
||||
private closed = false
|
||||
|
||||
private constructor(
|
||||
private readonly stream: NodeZstdPrivateStream,
|
||||
private readonly errorKey: symbol,
|
||||
) {
|
||||
this.stream.on('error', (error: Error) => {
|
||||
this.decoderError ??= error
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Create the optimized decoder when this Node release exposes the expected
|
||||
* private stream shape.
|
||||
* @returns a shared decoder, or `undefined` when callers must use the public fallback.
|
||||
*/
|
||||
static create(): NodePrivateZstdFrameDecoder | undefined {
|
||||
const stream = createZstdDecompress({ chunkSize: DECODE_CHUNK_SIZE })
|
||||
const privateAccess = privateZstdStream(stream)
|
||||
/* v8 ignore next -- reached only when a supported Node release changes its private stream shape. */
|
||||
if (privateAccess !== undefined) {
|
||||
return new NodePrivateZstdFrameDecoder(privateAccess.stream, privateAccess.errorKey)
|
||||
}
|
||||
/* v8 ignore next -- the active Node runtime passed the private-shape probe above. */
|
||||
stream.close()
|
||||
/* v8 ignore next -- the active Node runtime passed the private-shape probe above. */
|
||||
return undefined
|
||||
}
|
||||
|
||||
/** @inheritdoc */
|
||||
public *decode(source: Buffer, frames: readonly ZstdFrameRange[]): Generator<Buffer, void, void> {
|
||||
if (this.started) throw new Error('Zstandard frame decoder was already started')
|
||||
if (this.closed) throw new Error('cannot start a closed Zstandard frame decoder')
|
||||
this.started = true
|
||||
try {
|
||||
for (const frame of frames) {
|
||||
try {
|
||||
yield this.decodeFrame(source.subarray(frame.start, frame.end))
|
||||
} catch (error) {
|
||||
throw new Error(`corrupt Zstandard session log: frame at byte ${frame.start} failed validation`, {
|
||||
cause: error,
|
||||
})
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
this.close()
|
||||
}
|
||||
}
|
||||
|
||||
/** Decode one frame; its returned scratch view remains valid until the next call. */
|
||||
private decodeFrame(input: Buffer): Buffer {
|
||||
const handle = this.stream._handle
|
||||
/* v8 ignore next -- decode() rejects closed instances before entering this private frame operation. */
|
||||
if (this.closed || handle === null) throw new Error('cannot decode with a closed Zstandard frame decoder')
|
||||
|
||||
let inputOffset = 0
|
||||
let inputRemaining = input.length
|
||||
let outputBytes = 0
|
||||
const fullChunks: Buffer[] = []
|
||||
for (;;) {
|
||||
handle.writeSync(
|
||||
this.stream._defaultFlushFlag,
|
||||
input,
|
||||
inputOffset,
|
||||
inputRemaining,
|
||||
this.output,
|
||||
0,
|
||||
this.output.length,
|
||||
)
|
||||
if (this.decoderError !== undefined) throw this.decoderError
|
||||
const internalError = this.stream[this.errorKey]
|
||||
if (internalError !== null) {
|
||||
if (internalError instanceof Error) throw internalError
|
||||
throw new Error('Zstandard decoder exposed a non-Error internal failure')
|
||||
}
|
||||
|
||||
const outputAfter = this.stream._writeState[0]
|
||||
const inputAfter = this.stream._writeState[1]
|
||||
const consumed = inputRemaining - inputAfter
|
||||
const produced = this.output.length - outputAfter
|
||||
if (produced > 0) {
|
||||
outputBytes += produced
|
||||
/* v8 ignore next -- Buffer cannot materialize a frame beyond its own process-wide maximum length. */
|
||||
if (outputBytes > bufferConstants.MAX_LENGTH) {
|
||||
throw new Error(`Zstandard frame output exceeds ${bufferConstants.MAX_LENGTH} bytes`)
|
||||
}
|
||||
}
|
||||
|
||||
if (outputAfter !== 0) {
|
||||
/* v8 ignore next -- structurally scanned ranges contain exactly one complete frame and no trailing bytes. */
|
||||
if (inputAfter !== 0) throw new Error('Zstandard frame decoder left trailing input')
|
||||
const finalChunk = this.output.subarray(0, produced)
|
||||
if (fullChunks.length === 0) return finalChunk
|
||||
if (produced > 0) fullChunks.push(Buffer.from(finalChunk))
|
||||
const onlyChunk = fullChunks[0] as Buffer
|
||||
return fullChunks.length === 1
|
||||
? onlyChunk
|
||||
: Buffer.concat(fullChunks, outputBytes)
|
||||
}
|
||||
fullChunks.push(Buffer.from(this.output))
|
||||
inputOffset += consumed
|
||||
inputRemaining = inputAfter
|
||||
}
|
||||
}
|
||||
|
||||
/** @inheritdoc */
|
||||
close(): void {
|
||||
if (this.closed) return
|
||||
this.closed = true
|
||||
this.stream.close()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
/**
|
||||
* Public-API synchronous Zstandard frame decoder fallback.
|
||||
* @module dsh-session-persistence-jsonl/zstd-public-decoder
|
||||
*/
|
||||
|
||||
import { zstdDecompressSync } from 'node:zlib'
|
||||
import type { ZstdFrameDecoder, ZstdFrameRange } from './zstd.ts'
|
||||
|
||||
/** Multi-frame adapter built exclusively from Node's supported one-shot API. */
|
||||
export class PublicZstdFrameDecoder implements ZstdFrameDecoder {
|
||||
private started = false
|
||||
private closed = false
|
||||
|
||||
/** @inheritdoc */
|
||||
public *decode(source: Buffer, frames: readonly ZstdFrameRange[]): Generator<Buffer, void, void> {
|
||||
if (this.started) throw new Error('Zstandard frame decoder was already started')
|
||||
if (this.closed) throw new Error('cannot start a closed Zstandard frame decoder')
|
||||
this.started = true
|
||||
try {
|
||||
for (const { start, end } of frames) {
|
||||
let decoded: Buffer
|
||||
try {
|
||||
decoded = zstdDecompressSync(source.subarray(start, end))
|
||||
} catch (error) {
|
||||
throw new Error(`corrupt Zstandard session log: frame at byte ${start} failed validation`, {
|
||||
cause: error,
|
||||
})
|
||||
}
|
||||
yield decoded
|
||||
}
|
||||
} finally {
|
||||
this.close()
|
||||
}
|
||||
}
|
||||
|
||||
/** @inheritdoc */
|
||||
close(): void {
|
||||
this.closed = true
|
||||
}
|
||||
}
|
||||
156
packages/session/session-persistence-jsonl/src/zstd.ts
Normal file
156
packages/session/session-persistence-jsonl/src/zstd.ts
Normal file
@@ -0,0 +1,156 @@
|
||||
/**
|
||||
* Zstandard frame primitives for the JSONL persistence backend. The backend
|
||||
* owns a concatenated-frame container so it can append and recover batches
|
||||
* without exposing compression mechanics through the persistence seam.
|
||||
* @module dsh-session-persistence-jsonl/zstd
|
||||
*/
|
||||
|
||||
import {
|
||||
constants, zstdCompress, zstdDecompress, type ZstdOptions,
|
||||
} from 'node:zlib'
|
||||
import { promisify } from 'node:util'
|
||||
import { NodePrivateZstdFrameDecoder } from './zstd-private-decoder.ts'
|
||||
import { PublicZstdFrameDecoder } from './zstd-public-decoder.ts'
|
||||
|
||||
const ZSTD_MAGIC = 0xFD2FB528
|
||||
const zstdCompressAsync = promisify(zstdCompress)
|
||||
const zstdDecompressAsync = promisify(zstdDecompress)
|
||||
const CHECKSUM_OPTIONS: ZstdOptions = {
|
||||
params: { [constants.ZSTD_c_checksumFlag]: 1 },
|
||||
}
|
||||
const INCOMPLETE_FRAME_OPTIONS: ZstdOptions = {
|
||||
finishFlush: constants.ZSTD_e_flush,
|
||||
}
|
||||
|
||||
/** Byte range occupied by one structurally complete Zstandard frame. */
|
||||
export interface ZstdFrameRange {
|
||||
/** Inclusive frame start. */
|
||||
start: number
|
||||
/** Exclusive frame end. */
|
||||
end: number
|
||||
}
|
||||
|
||||
/** Structural scan result for a concatenated Zstandard stream. */
|
||||
export interface ZstdFrameScan {
|
||||
/** Complete frames in file order. */
|
||||
frames: ZstdFrameRange[]
|
||||
/** Start of an incomplete final frame, when EOF interrupts one. */
|
||||
tornStart?: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Locate complete frames without decompressing their blocks. Invalid complete
|
||||
* structure rejects; EOF inside the final frame returns its start for repair.
|
||||
* @param buffer - complete bytes currently present in the session artifact.
|
||||
* @param maxFrames - optional complete-frame limit for metadata-only readers.
|
||||
* @returns complete frame ranges and an optional incomplete-final-frame start.
|
||||
*/
|
||||
export function scanZstdFrames(buffer: Buffer, maxFrames = Number.POSITIVE_INFINITY): ZstdFrameScan {
|
||||
const frames: ZstdFrameRange[] = []
|
||||
let offset = 0
|
||||
|
||||
while (offset < buffer.length) {
|
||||
const start = offset
|
||||
if (buffer.length - offset < 4) return { frames, tornStart: start }
|
||||
if (buffer.readUInt32LE(offset) !== ZSTD_MAGIC) {
|
||||
throw new Error(`corrupt Zstandard session log: invalid frame magic at byte ${offset}`)
|
||||
}
|
||||
offset += 4
|
||||
|
||||
if (offset === buffer.length) return { frames, tornStart: start }
|
||||
const descriptor = buffer.readUInt8(offset)
|
||||
offset += 1
|
||||
if ((descriptor & 0x18) !== 0) {
|
||||
throw new Error(`corrupt Zstandard session log: reserved frame-header bit at byte ${offset - 1}`)
|
||||
}
|
||||
|
||||
const contentSizeFlag = descriptor >>> 6
|
||||
const singleSegment = (descriptor & 0x20) !== 0
|
||||
const checksum = (descriptor & 0x04) !== 0
|
||||
const dictionaryFlag = descriptor & 0x03
|
||||
const dictionaryBytes = dictionaryFlag === 3 ? 4 : dictionaryFlag
|
||||
const contentSizeBytes = contentSizeFlag === 0
|
||||
? (singleSegment ? 1 : 0)
|
||||
: 1 << contentSizeFlag
|
||||
const remainingHeaderBytes = (singleSegment ? 0 : 1) + dictionaryBytes + contentSizeBytes
|
||||
if (buffer.length - offset < remainingHeaderBytes) return { frames, tornStart: start }
|
||||
offset += remainingHeaderBytes
|
||||
|
||||
for (;;) {
|
||||
if (buffer.length - offset < 3) return { frames, tornStart: start }
|
||||
const blockHeader = buffer.readUIntLE(offset, 3)
|
||||
offset += 3
|
||||
const lastBlock = (blockHeader & 1) !== 0
|
||||
const blockType = (blockHeader >>> 1) & 0x03
|
||||
const blockSize = blockHeader >>> 3
|
||||
if (blockType === 0x03) {
|
||||
throw new Error(`corrupt Zstandard session log: reserved block type at byte ${offset - 3}`)
|
||||
}
|
||||
const payloadBytes = blockType === 0x01 ? 1 : blockSize
|
||||
if (buffer.length - offset < payloadBytes) return { frames, tornStart: start }
|
||||
offset += payloadBytes
|
||||
if (lastBlock) break
|
||||
}
|
||||
|
||||
if (checksum) {
|
||||
if (buffer.length - offset < 4) return { frames, tornStart: start }
|
||||
offset += 4
|
||||
}
|
||||
frames.push({ start, end: offset })
|
||||
if (frames.length === maxFrames) return { frames }
|
||||
}
|
||||
|
||||
return { frames }
|
||||
}
|
||||
|
||||
/**
|
||||
* Compress one independently decodable, checksummed Zstandard frame.
|
||||
* @param input - JSONL bytes for a header or durable event batch.
|
||||
* @returns the complete encoded frame.
|
||||
*/
|
||||
export async function compressZstdFrame(input: Buffer | string): Promise<Buffer> {
|
||||
return zstdCompressAsync(input, CHECKSUM_OPTIONS)
|
||||
}
|
||||
|
||||
/**
|
||||
* Decompress one complete frame and validate its checksum.
|
||||
* @param input - one structurally complete Zstandard frame.
|
||||
* @returns the frame plaintext.
|
||||
*/
|
||||
export async function decompressZstdFrame(input: Buffer): Promise<Buffer> {
|
||||
return zstdDecompressAsync(input)
|
||||
}
|
||||
|
||||
/** Common lifecycle for interchangeable synchronous multi-frame decoders. */
|
||||
export interface ZstdFrameDecoder {
|
||||
/**
|
||||
* Decode and checksum complete frames in source order. Each yielded buffer
|
||||
* remains valid only until the iterator advances to the next frame.
|
||||
* @param source - concatenated Zstandard frame bytes.
|
||||
* @param frames - structurally complete ranges within `source`.
|
||||
* @returns one plaintext buffer per frame.
|
||||
*/
|
||||
decode(source: Buffer, frames: readonly ZstdFrameRange[]): Generator<Buffer, void, void>
|
||||
/** Release decoder-owned resources; repeated calls are harmless. */
|
||||
close(): void
|
||||
}
|
||||
|
||||
/**
|
||||
* Select the shared private decoder when the running Node 22/24/26 shape is
|
||||
* compatible, otherwise preserve correctness with the public one-shot API.
|
||||
* @returns a synchronous decoder with an implementation-independent lifecycle.
|
||||
*/
|
||||
export function createZstdFrameDecoder(): ZstdFrameDecoder {
|
||||
return NodePrivateZstdFrameDecoder.create() ?? new PublicZstdFrameDecoder()
|
||||
}
|
||||
|
||||
/**
|
||||
* Recover available plaintext from a structurally incomplete final frame.
|
||||
* `ZSTD_e_flush` deliberately suppresses final-frame and checksum completion;
|
||||
* callers must establish the torn frame boundary before using this helper.
|
||||
* @param input - available bytes from a known incomplete Zstandard frame.
|
||||
* @returns plaintext produced from the available input.
|
||||
*/
|
||||
export async function decompressZstdPrefix(input: Buffer): Promise<Buffer> {
|
||||
return zstdDecompressAsync(input, INCOMPLETE_FRAME_OPTIONS)
|
||||
}
|
||||
1491
packages/session/session-persistence-jsonl/tests/jsonl.spec.ts
Normal file
1491
packages/session/session-persistence-jsonl/tests/jsonl.spec.ts
Normal file
File diff suppressed because it is too large
Load Diff
210
packages/session/session-persistence-jsonl/tests/win32.spec.ts
Normal file
210
packages/session/session-persistence-jsonl/tests/win32.spec.ts
Normal file
@@ -0,0 +1,210 @@
|
||||
/**
|
||||
* Unit tests for the Windows durable namespace helper with a mocked kernel32
|
||||
* binding. The real JSONL suite exercises the helper on native Windows; these
|
||||
* tests keep the Win32 error mapping and race handling covered on every host.
|
||||
*/
|
||||
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from 'node:fs'
|
||||
import { mkdtemp, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
|
||||
const MOVEFILE_WRITE_THROUGH = 0x00000008
|
||||
const ERROR_FILE_NOT_FOUND = 2
|
||||
const ERROR_PATH_NOT_FOUND = 3
|
||||
const ERROR_ACCESS_DENIED = 5
|
||||
const ERROR_NOT_SAME_DEVICE = 17
|
||||
const ERROR_FILE_EXISTS = 80
|
||||
const ERROR_INVALID_NAME = 123
|
||||
const ERROR_ALREADY_EXISTS = 183
|
||||
|
||||
type MoveFileExW = (existing: string, replacement: string, flags: number, setLastError: (code: number) => void) => number
|
||||
|
||||
const roots: string[] = []
|
||||
|
||||
function stripNamespace(path: string): string {
|
||||
if (path.startsWith('\\\\?\\UNC\\')) return `\\\\${path.slice('\\\\?\\UNC\\'.length)}`
|
||||
if (path.startsWith('\\\\?\\')) return path.slice('\\\\?\\'.length)
|
||||
return path
|
||||
}
|
||||
|
||||
async function tempRoot(): Promise<string> {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'dsh-jsonl-win32-'))
|
||||
roots.push(dir)
|
||||
return dir
|
||||
}
|
||||
|
||||
async function importWithMove(moveFileExW: MoveFileExW): Promise<typeof import('../src/win32.ts')> {
|
||||
vi.resetModules()
|
||||
vi.doMock('koffi', () => {
|
||||
let lastError = 0
|
||||
const setLastError = (code: number): void => { lastError = code }
|
||||
const move: MoveFileExW = (existing, replacement, flags, setError) => {
|
||||
const ok = moveFileExW(existing, replacement, flags, setError)
|
||||
lastError = ok === 0 ? lastError : 0
|
||||
return ok
|
||||
}
|
||||
return {
|
||||
default: {
|
||||
load: () => ({
|
||||
func: (_convention: string, name: string, result: string) => {
|
||||
if (name === 'MoveFileExW') return (existing: string, replacement: string, flags: number) => {
|
||||
expect(result).toBe('int')
|
||||
const ok = move(existing, replacement, flags, setLastError)
|
||||
return ok
|
||||
}
|
||||
return () => lastError
|
||||
},
|
||||
}),
|
||||
},
|
||||
}
|
||||
})
|
||||
return import('../src/win32.ts')
|
||||
}
|
||||
|
||||
async function importWithError(code: number): Promise<typeof import('../src/win32.ts')> {
|
||||
vi.resetModules()
|
||||
vi.doMock('koffi', () => ({
|
||||
default: {
|
||||
load: () => ({
|
||||
func: (_convention: string, name: string) => {
|
||||
if (name === 'MoveFileExW') return () => 0
|
||||
return () => code
|
||||
},
|
||||
}),
|
||||
},
|
||||
}))
|
||||
return import('../src/win32.ts')
|
||||
}
|
||||
|
||||
async function importWithFilesystemMove(): Promise<typeof import('../src/win32.ts')> {
|
||||
return importWithMove((existing, replacement, flags, setLastError) => {
|
||||
expect(flags).toBe(MOVEFILE_WRITE_THROUGH)
|
||||
const from = stripNamespace(existing)
|
||||
const to = stripNamespace(replacement)
|
||||
if (!existsSync(from)) { setLastError(ERROR_FILE_NOT_FOUND); return 0 }
|
||||
if (existsSync(to)) { setLastError(ERROR_ALREADY_EXISTS); return 0 }
|
||||
renameSync(from, to)
|
||||
return 1
|
||||
})
|
||||
}
|
||||
|
||||
afterEach(async () => {
|
||||
vi.doUnmock('koffi')
|
||||
vi.doUnmock('node:fs/promises')
|
||||
vi.doUnmock('node:path')
|
||||
vi.resetModules()
|
||||
for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
describe('Windows durable namespace helpers', () => {
|
||||
it('keeps drive-root probes native while namespacing descendants', async () => {
|
||||
const probes: string[] = []
|
||||
vi.resetModules()
|
||||
vi.doMock('node:fs/promises', async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import('node:fs/promises')>()
|
||||
return {
|
||||
...actual,
|
||||
stat: async (path: string) => {
|
||||
probes.push(path)
|
||||
return { isDirectory: () => true }
|
||||
},
|
||||
}
|
||||
})
|
||||
vi.doMock('node:path', async (importOriginal) => {
|
||||
const actual = await importOriginal<typeof import('node:path')>()
|
||||
return {
|
||||
...actual,
|
||||
join: (...paths: string[]) => actual.win32.join(...paths),
|
||||
parse: (path: string) => actual.win32.parse(path),
|
||||
resolve: (...paths: string[]) => actual.win32.resolve(...paths),
|
||||
toNamespacedPath: (path: string) => actual.win32.toNamespacedPath(path),
|
||||
}
|
||||
})
|
||||
const { ensureDurableDirectoryWin32 } = await import('../src/win32.ts')
|
||||
|
||||
await ensureDurableDirectoryWin32('C:\\existing')
|
||||
|
||||
expect(probes).toEqual(['C:\\', '\\\\?\\C:\\existing'])
|
||||
})
|
||||
|
||||
it('publishes a new file with write-through MoveFileExW semantics', async () => {
|
||||
const { publishNewFileWin32 } = await importWithFilesystemMove()
|
||||
const root = await tempRoot()
|
||||
const tmp = join(root, 'log.tmp')
|
||||
const final = join(root, 'log.jsonl')
|
||||
await writeFile(tmp, 'content')
|
||||
|
||||
await publishNewFileWin32(tmp, final)
|
||||
expect(existsSync(tmp)).toBe(false)
|
||||
expect(readFileSync(final, 'utf8')).toBe('content')
|
||||
})
|
||||
|
||||
it('maps Win32 publish failures to Node-style errno codes', async () => {
|
||||
const cases = [
|
||||
[ERROR_FILE_NOT_FOUND, 'ENOENT'],
|
||||
[ERROR_PATH_NOT_FOUND, 'ENOENT'],
|
||||
[ERROR_ACCESS_DENIED, 'EACCES'],
|
||||
[ERROR_NOT_SAME_DEVICE, 'EXDEV'],
|
||||
[ERROR_FILE_EXISTS, 'EEXIST'],
|
||||
[ERROR_ALREADY_EXISTS, 'EEXIST'],
|
||||
[ERROR_INVALID_NAME, 'EINVAL'],
|
||||
[9999, 'EIO'],
|
||||
] as const
|
||||
for (const [win32Code, code] of cases) {
|
||||
const { publishNewFileWin32 } = await importWithError(win32Code)
|
||||
await expect(publishNewFileWin32('from', 'to')).rejects.toMatchObject({ code, win32Code, path: 'from', dest: 'to' })
|
||||
}
|
||||
})
|
||||
|
||||
it('creates missing directories through staging siblings and tolerates an already-created race', async () => {
|
||||
const root = await tempRoot()
|
||||
const raced = join(root, 'raced')
|
||||
const { ensureDurableDirectoryWin32 } = await importWithMove((existing, replacement, flags, setLastError) => {
|
||||
expect(flags).toBe(MOVEFILE_WRITE_THROUGH)
|
||||
const from = stripNamespace(existing)
|
||||
const to = stripNamespace(replacement)
|
||||
if (to === raced) {
|
||||
mkdirSync(to)
|
||||
setLastError(ERROR_ALREADY_EXISTS)
|
||||
return 0
|
||||
}
|
||||
if (!existsSync(from)) { setLastError(ERROR_FILE_NOT_FOUND); return 0 }
|
||||
if (existsSync(to)) { setLastError(ERROR_ALREADY_EXISTS); return 0 }
|
||||
renameSync(from, to)
|
||||
return 1
|
||||
})
|
||||
|
||||
await ensureDurableDirectoryWin32(join(root, 'a', 'b'))
|
||||
expect(existsSync(join(root, 'a', 'b'))).toBe(true)
|
||||
await ensureDurableDirectoryWin32(join(root, 'a', 'b'))
|
||||
await ensureDurableDirectoryWin32(raced)
|
||||
expect(existsSync(raced)).toBe(true)
|
||||
})
|
||||
|
||||
it('keeps staging names valid for a maximum-length target component', async () => {
|
||||
const { ensureDurableDirectoryWin32 } = await importWithFilesystemMove()
|
||||
const root = await tempRoot()
|
||||
const target = join(root, 'x'.repeat(255))
|
||||
|
||||
await ensureDurableDirectoryWin32(target)
|
||||
expect(existsSync(target)).toBe(true)
|
||||
})
|
||||
|
||||
it('surfaces directory publication failures other than an existing-target race', async () => {
|
||||
const { ensureDurableDirectoryWin32 } = await importWithError(ERROR_ACCESS_DENIED)
|
||||
const root = await tempRoot()
|
||||
|
||||
await expect(ensureDurableDirectoryWin32(join(root, 'denied'))).rejects.toMatchObject({ code: 'EACCES' })
|
||||
})
|
||||
|
||||
it('rejects a non-directory component instead of treating it as missing', async () => {
|
||||
const { ensureDurableDirectoryWin32 } = await importWithFilesystemMove()
|
||||
const root = await tempRoot()
|
||||
const blocked = join(root, 'blocked')
|
||||
writeFileSync(blocked, 'x')
|
||||
|
||||
await expect(ensureDurableDirectoryWin32(join(blocked, 'child'))).rejects.toMatchObject({ code: 'ENOTDIR' })
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,39 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import {
|
||||
compressZstdFrame, createZstdFrameDecoder, decompressZstdFrame, decompressZstdPrefix, scanZstdFrames,
|
||||
} from '../src/zstd.ts'
|
||||
import { NodePrivateZstdFrameDecoder } from '../src/zstd-private-decoder.ts'
|
||||
import { PublicZstdFrameDecoder } from '../src/zstd-public-decoder.ts'
|
||||
|
||||
describe('JSONL Zstandard compatibility', () => {
|
||||
it('round-trips concatenated checksummed frames through the built-in Node API', async () => {
|
||||
const encoded = Buffer.concat([
|
||||
await compressZstdFrame('{"type":"session","version":0,"id":"compat","createdAt":1}\n'),
|
||||
await compressZstdFrame('{"type":"turn/start","seq":0,"turn":1}\n'),
|
||||
])
|
||||
const { frames, tornStart } = scanZstdFrames(encoded)
|
||||
|
||||
expect(tornStart).toBeUndefined()
|
||||
expect(frames).toHaveLength(2)
|
||||
expect(frames.map(frame => encoded.subarray(frame.start, frame.start + 4).toString('hex')))
|
||||
.toEqual(['28b52ffd', '28b52ffd'])
|
||||
const decoded = await Promise.all(frames.map(frame => decompressZstdFrame(encoded.subarray(frame.start, frame.end))))
|
||||
expect(Buffer.concat(decoded).toString()).toContain('"type":"turn/start"')
|
||||
|
||||
const preferred = createZstdFrameDecoder()
|
||||
expect(preferred).toBeInstanceOf(NodePrivateZstdFrameDecoder)
|
||||
for (const decoder of [preferred, new PublicZstdFrameDecoder()]) {
|
||||
try {
|
||||
const plaintext = Array.from(decoder.decode(encoded, frames), chunk => Buffer.from(chunk))
|
||||
expect(Buffer.concat(plaintext).toString()).toContain('"type":"turn/start"')
|
||||
} finally {
|
||||
decoder.close()
|
||||
}
|
||||
}
|
||||
|
||||
const eventFrame = encoded.subarray(frames[1]!.start, frames[1]!.end)
|
||||
const missingChecksumByte = eventFrame.subarray(0, -1)
|
||||
expect(scanZstdFrames(missingChecksumByte)).toEqual({ frames: [], tornStart: 0 })
|
||||
expect((await decompressZstdPrefix(missingChecksumByte)).toString()).toContain('"type":"turn/start"')
|
||||
})
|
||||
})
|
||||
717
packages/session/session-persistence-jsonl/tests/zstd.spec.ts
Normal file
717
packages/session/session-persistence-jsonl/tests/zstd.spec.ts
Normal file
@@ -0,0 +1,717 @@
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import { appendFile, mkdir, mkdtemp, open, readFile, readdir, rm, stat, writeFile } from 'node:fs/promises'
|
||||
import type { FileHandle } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { performance } from 'node:perf_hooks'
|
||||
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import SessionPersistenceJsonl from '@deepseek-ai/dsh-session-persistence-jsonl'
|
||||
import { logPath, scanLog, sessionDir, toHeaderLine, type JsonlCompression } from '../src/format.ts'
|
||||
import {
|
||||
compressZstdFrame, createZstdFrameDecoder, decompressZstdFrame, decompressZstdPrefix, scanZstdFrames,
|
||||
type ZstdFrameDecoder,
|
||||
} from '../src/zstd.ts'
|
||||
import { NodePrivateZstdFrameDecoder } from '../src/zstd-private-decoder.ts'
|
||||
import { PublicZstdFrameDecoder } from '../src/zstd-public-decoder.ts'
|
||||
import { runPersistenceContract, meta, oneTurnLog } from '../../session-persistence/tests/contract.ts'
|
||||
import { runCoordinatorContract, type CoordinatorFixture } from '../../session-persistence/tests/coordinator-contract.ts'
|
||||
|
||||
const MAGIC = Buffer.from([0x28, 0xB5, 0x2F, 0xFD])
|
||||
const roots: string[] = []
|
||||
const contexts: Context[] = []
|
||||
|
||||
interface ZstdReaderInternals {
|
||||
readZstdPrefix(buffer: Buffer, signal?: AbortSignal): Promise<{ events: SessionEvent[] }>
|
||||
}
|
||||
|
||||
type HeaderRead = (
|
||||
this: FileHandle,
|
||||
buffer: Buffer,
|
||||
offset: number,
|
||||
length: number,
|
||||
position: number | null,
|
||||
) => Promise<{ bytesRead: number; buffer: Buffer }>
|
||||
|
||||
async function freshRoot(prefix = 'dsh-jsonl-zstd-'): Promise<string> {
|
||||
const root = await mkdtemp(join(tmpdir(), prefix))
|
||||
roots.push(root)
|
||||
return root
|
||||
}
|
||||
|
||||
async function mount(root: string, compression?: JsonlCompression): Promise<Context> {
|
||||
const ctx = new Context()
|
||||
contexts.push(ctx)
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SessionPersistenceJsonl, {
|
||||
root,
|
||||
...(compression === undefined ? {} : { compression }),
|
||||
})
|
||||
return ctx
|
||||
}
|
||||
|
||||
async function decodeCompleteFrames(buffer: Buffer): Promise<Buffer> {
|
||||
const { frames, tornStart } = scanZstdFrames(buffer)
|
||||
expect(tornStart).toBeUndefined()
|
||||
const plaintext: Buffer[] = []
|
||||
for (const frame of frames) {
|
||||
plaintext.push(await decompressZstdFrame(buffer.subarray(frame.start, frame.end)))
|
||||
}
|
||||
return Buffer.concat(plaintext)
|
||||
}
|
||||
|
||||
async function tornFrame(
|
||||
plaintext: string,
|
||||
accepts: (decoded: string) => boolean,
|
||||
): Promise<Buffer> {
|
||||
const frame = await compressZstdFrame(plaintext)
|
||||
const candidateEnds = [
|
||||
frame.length - 1,
|
||||
frame.length - 4,
|
||||
...[0.9, 0.75, 0.6, 0.5, 0.4, 0.25].map(ratio => Math.floor(frame.length * ratio)),
|
||||
]
|
||||
for (const end of candidateEnds) {
|
||||
const candidate = frame.subarray(0, end)
|
||||
if (scanZstdFrames(candidate).tornStart !== 0) continue
|
||||
try {
|
||||
const decoded = (await decompressZstdPrefix(candidate)).toString('utf8')
|
||||
if (accepts(decoded)) return candidate
|
||||
} catch {
|
||||
// Some early cuts precede the first decodable block; keep searching for
|
||||
// a cut that exercises partial-plaintext recovery.
|
||||
}
|
||||
}
|
||||
throw new Error('test fixture could not produce the requested torn Zstandard frame')
|
||||
}
|
||||
|
||||
function deterministicNoise(length: number): string {
|
||||
let state = 0x12345678
|
||||
let output = ''
|
||||
for (let index = 0; index < length; index++) {
|
||||
state = (Math.imul(state, 1_664_525) + 1_013_904_223) >>> 0
|
||||
output += String.fromCharCode(33 + (state % 90))
|
||||
}
|
||||
return output
|
||||
}
|
||||
|
||||
function emptyStructuralFrame(descriptor: number): Buffer {
|
||||
const contentSizeFlag = descriptor >>> 6
|
||||
const singleSegment = (descriptor & 0x20) !== 0
|
||||
const dictionaryBytes = [0, 1, 2, 4][descriptor & 0x03]!
|
||||
const contentSizeBytes = contentSizeFlag === 0 ? (singleSegment ? 1 : 0) : 1 << contentSizeFlag
|
||||
const variableHeader = Buffer.alloc((singleSegment ? 0 : 1) + dictionaryBytes + contentSizeBytes)
|
||||
const lastEmptyRawBlock = Buffer.from([1, 0, 0])
|
||||
const checksum = (descriptor & 0x04) === 0 ? Buffer.alloc(0) : Buffer.alloc(4)
|
||||
return Buffer.concat([MAGIC, Buffer.from([descriptor]), variableHeader, lastEmptyRawBlock, checksum])
|
||||
}
|
||||
|
||||
afterEach(async () => {
|
||||
vi.restoreAllMocks()
|
||||
for (const ctx of contexts.splice(0).reverse()) await ctx.fiber.dispose()
|
||||
for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
runPersistenceContract('jsonl-zstd', async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'dsh-jsonl-zstd-contract-'))
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(SessionPersistenceJsonl, { root })
|
||||
return {
|
||||
persistence: ctx.sessionPersistence,
|
||||
dispose: async () => {
|
||||
await fiber.dispose()
|
||||
await rm(root, { recursive: true, force: true })
|
||||
},
|
||||
}
|
||||
})
|
||||
|
||||
runCoordinatorContract('jsonl-zstd', async (): Promise<CoordinatorFixture> => {
|
||||
const root = await mkdtemp(join(tmpdir(), 'dsh-jsonl-zstd-coordinator-'))
|
||||
return {
|
||||
mount: async ctx => ctx.plugin(SessionPersistenceJsonl, { root }),
|
||||
corruptTail: async (id, cwd) => {
|
||||
const line = JSON.stringify({
|
||||
type: 'assistant/chunk',
|
||||
seq: 8,
|
||||
time: 9,
|
||||
data: { turn: 2, step: 1, chunk: { type: 'text-delta', index: 0, text: deterministicNoise(300_000) } },
|
||||
}) + '\n'
|
||||
const partial = await tornFrame(line, decoded => decoded.length > 0 && !decoded.endsWith('\n'))
|
||||
await appendFile(logPath(root, cwd, id, 'zstd'), partial)
|
||||
},
|
||||
cleanup: async () => { await rm(root, { recursive: true, force: true }) },
|
||||
}
|
||||
})
|
||||
|
||||
describe('Zstandard frame structure', () => {
|
||||
it('scans concatenated checksummed frames and honors a frame limit', async () => {
|
||||
const first = await compressZstdFrame('header\n')
|
||||
const second = await compressZstdFrame('event\n')
|
||||
const stream = Buffer.concat([first, second])
|
||||
expect(scanZstdFrames(Buffer.alloc(0))).toEqual({ frames: [] })
|
||||
expect(scanZstdFrames(stream)).toEqual({
|
||||
frames: [{ start: 0, end: first.length }, { start: first.length, end: stream.length }],
|
||||
})
|
||||
expect(scanZstdFrames(stream, 1)).toEqual({ frames: [{ start: 0, end: first.length }] })
|
||||
expect(first[4]! & 0x04).toBe(0x04)
|
||||
expect(second[4]! & 0x04).toBe(0x04)
|
||||
expect((await decompressZstdFrame(first)).toString()).toBe('header\n')
|
||||
const decoder = createZstdFrameDecoder()
|
||||
try {
|
||||
const plaintext = Array.from(decoder.decode(stream, scanZstdFrames(stream).frames), chunk => Buffer.from(chunk))
|
||||
expect(Buffer.concat(plaintext).toString()).toBe('header\nevent\n')
|
||||
} finally {
|
||||
decoder.close()
|
||||
}
|
||||
})
|
||||
|
||||
it('keeps the public and Node-private synchronous decoders interchangeable', async () => {
|
||||
const frames = [await compressZstdFrame('first\n'), await compressZstdFrame('second\n')]
|
||||
const stream = Buffer.concat(frames)
|
||||
const ranges = scanZstdFrames(stream).frames
|
||||
const privateDecoder = NodePrivateZstdFrameDecoder.create()
|
||||
expect(privateDecoder).toBeDefined()
|
||||
|
||||
for (const decoder of [new PublicZstdFrameDecoder(), privateDecoder!]) {
|
||||
try {
|
||||
const plaintext = Array.from(decoder.decode(stream, ranges), chunk => Buffer.from(chunk))
|
||||
expect(plaintext).toHaveLength(2)
|
||||
expect(Buffer.concat(plaintext).toString()).toBe('first\nsecond\n')
|
||||
} finally {
|
||||
decoder.close()
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
it('falls back to the public decoder when the private Node contract is unavailable', () => {
|
||||
vi.spyOn(NodePrivateZstdFrameDecoder, 'create').mockReturnValue(undefined)
|
||||
const decoder = createZstdFrameDecoder()
|
||||
expect(decoder).toBeInstanceOf(PublicZstdFrameDecoder)
|
||||
decoder.close()
|
||||
})
|
||||
|
||||
it('enforces decoder lifecycle and checksum errors through both implementations', async () => {
|
||||
const frame = await compressZstdFrame('frame\n')
|
||||
const range = [{ start: 0, end: frame.length }]
|
||||
const corrupt = Buffer.from(frame)
|
||||
corrupt[corrupt.length - 1] = corrupt[corrupt.length - 1]! ^ 0xFF
|
||||
const factories: Array<() => ZstdFrameDecoder> = [
|
||||
() => new PublicZstdFrameDecoder(),
|
||||
() => NodePrivateZstdFrameDecoder.create()!,
|
||||
]
|
||||
|
||||
for (const create of factories) {
|
||||
const interrupted = create()
|
||||
const iterator = interrupted.decode(frame, range)
|
||||
expect(iterator.next().value?.toString()).toBe('frame\n')
|
||||
iterator.return()
|
||||
expect(() => Array.from(interrupted.decode(frame, range))).toThrow(/already started/)
|
||||
interrupted.close()
|
||||
|
||||
const closed = create()
|
||||
closed.close()
|
||||
closed.close()
|
||||
expect(() => Array.from(closed.decode(frame, range))).toThrow(/closed/)
|
||||
|
||||
const invalid = create()
|
||||
expect(() => Array.from(invalid.decode(corrupt, range))).toThrow(/frame at byte 0 failed validation/)
|
||||
}
|
||||
})
|
||||
|
||||
it('assembles private-decoder output at and beyond its reusable chunk boundary', async () => {
|
||||
for (const length of [8, 9]) {
|
||||
const plaintext = Buffer.alloc(length, 0x61)
|
||||
const frame = await compressZstdFrame(plaintext)
|
||||
const decoder = NodePrivateZstdFrameDecoder.create()!
|
||||
;(decoder as unknown as { output: Buffer }).output = Buffer.allocUnsafe(8)
|
||||
const [decoded] = Array.from(
|
||||
decoder.decode(frame, [{ start: 0, end: frame.length }]),
|
||||
chunk => Buffer.from(chunk),
|
||||
)
|
||||
expect(decoded).toEqual(plaintext)
|
||||
}
|
||||
})
|
||||
|
||||
it('normalizes private decoder stream failures', async () => {
|
||||
interface PrivateDecoderInternals {
|
||||
stream: {
|
||||
[key: symbol]: unknown
|
||||
emit(event: string, error: Error): boolean
|
||||
}
|
||||
errorKey: symbol
|
||||
}
|
||||
const frame = await compressZstdFrame('frame\n')
|
||||
const range = [{ start: 0, end: frame.length }]
|
||||
|
||||
const emitted = NodePrivateZstdFrameDecoder.create()!
|
||||
const emittedInternals = emitted as unknown as PrivateDecoderInternals
|
||||
const first = new Error('first emitted decoder failure')
|
||||
emittedInternals.stream.emit('error', first)
|
||||
emittedInternals.stream.emit('error', new Error('later emitted decoder failure'))
|
||||
try {
|
||||
Array.from(emitted.decode(frame, range))
|
||||
throw new Error('expected emitted decoder failure')
|
||||
} catch (error) {
|
||||
expect((error as Error).cause).toBe(first)
|
||||
}
|
||||
|
||||
for (const internalFailure of [new Error('internal decoder failure'), 'not an Error']) {
|
||||
const decoder = NodePrivateZstdFrameDecoder.create()!
|
||||
const internals = decoder as unknown as PrivateDecoderInternals
|
||||
internals.stream[internals.errorKey] = internalFailure
|
||||
try {
|
||||
Array.from(decoder.decode(frame, range))
|
||||
throw new Error('expected internal decoder failure')
|
||||
} catch (error) {
|
||||
const cause = (error as Error).cause
|
||||
if (internalFailure instanceof Error) {
|
||||
expect(cause).toBe(internalFailure)
|
||||
} else {
|
||||
expect(cause).toMatchObject({ message: 'Zstandard decoder exposed a non-Error internal failure' })
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
it('distinguishes incomplete frame regions from invalid complete structure', () => {
|
||||
expect(scanZstdFrames(MAGIC.subarray(0, 2))).toEqual({ frames: [], tornStart: 0 })
|
||||
expect(scanZstdFrames(MAGIC)).toEqual({ frames: [], tornStart: 0 })
|
||||
expect(() => scanZstdFrames(Buffer.alloc(4))).toThrow(/invalid frame magic/)
|
||||
expect(() => scanZstdFrames(Buffer.concat([MAGIC, Buffer.from([0x08])]))).toThrow(/reserved frame-header bit/)
|
||||
|
||||
// Non-single-segment descriptor with no window descriptor.
|
||||
expect(scanZstdFrames(Buffer.concat([MAGIC, Buffer.from([0x00])]))).toEqual({ frames: [], tornStart: 0 })
|
||||
// Single-segment header followed by only two bytes of the three-byte block header.
|
||||
expect(scanZstdFrames(Buffer.concat([MAGIC, Buffer.from([0x20, 0x00, 0x01, 0x00])]))).toEqual({
|
||||
frames: [],
|
||||
tornStart: 0,
|
||||
})
|
||||
|
||||
const rawFiveBytes = Buffer.from([(5 << 3) | 1, 0, 0])
|
||||
expect(scanZstdFrames(Buffer.concat([
|
||||
MAGIC,
|
||||
Buffer.from([0x20, 0x00]),
|
||||
rawFiveBytes,
|
||||
Buffer.from([0x01, 0x02]),
|
||||
]))).toEqual({ frames: [], tornStart: 0 })
|
||||
|
||||
const reservedBlock = Buffer.concat([
|
||||
MAGIC,
|
||||
Buffer.from([0x20, 0x00, 0x07, 0x00, 0x00]),
|
||||
])
|
||||
expect(() => scanZstdFrames(reservedBlock)).toThrow(/reserved block type/)
|
||||
})
|
||||
|
||||
it('covers standard header variants, RLE blocks, multiple blocks, and checksums', () => {
|
||||
for (const descriptor of [0x00, 0x21, 0x42, 0x83, 0xE3]) {
|
||||
const frame = emptyStructuralFrame(descriptor)
|
||||
expect(scanZstdFrames(frame)).toEqual({ frames: [{ start: 0, end: frame.length }] })
|
||||
}
|
||||
|
||||
const rle = Buffer.concat([
|
||||
MAGIC,
|
||||
Buffer.from([0x20, 0x01]),
|
||||
Buffer.from([(1 << 3) | (1 << 1) | 1, 0, 0]),
|
||||
Buffer.from([0x41]),
|
||||
])
|
||||
expect(scanZstdFrames(rle)).toEqual({ frames: [{ start: 0, end: rle.length }] })
|
||||
|
||||
const twoBlocks = Buffer.concat([
|
||||
MAGIC,
|
||||
Buffer.from([0x20, 0x00]),
|
||||
Buffer.from([0, 0, 0]),
|
||||
Buffer.from([1, 0, 0]),
|
||||
])
|
||||
expect(scanZstdFrames(twoBlocks)).toEqual({ frames: [{ start: 0, end: twoBlocks.length }] })
|
||||
|
||||
const checksummed = emptyStructuralFrame(0x24)
|
||||
expect(scanZstdFrames(checksummed.subarray(0, -1))).toEqual({ frames: [], tornStart: 0 })
|
||||
expect(scanZstdFrames(checksummed)).toEqual({ frames: [{ start: 0, end: checksummed.length }] })
|
||||
})
|
||||
})
|
||||
|
||||
describe('SessionPersistenceJsonl: default Zstandard encoding', () => {
|
||||
it('writes .jsonl.zstd by default with one header frame and one first-batch frame', async () => {
|
||||
const root = await freshRoot()
|
||||
const ctx = await mount(root)
|
||||
const header = meta('default-zstd', '/work')
|
||||
await ctx.sessionPersistence.create(header)
|
||||
await ctx.sessionPersistence.append(header.id, oneTurnLog())
|
||||
|
||||
const path = logPath(root, header.cwd, header.id, 'zstd')
|
||||
const buffer = await readFile(path)
|
||||
expect(buffer.subarray(0, 4)).toEqual(MAGIC)
|
||||
await expect(stat(logPath(root, header.cwd, header.id, 'none'))).rejects.toThrow()
|
||||
expect(ctx.sessionPersistence.locate(header)).toEqual({ kind: 'jsonl', path })
|
||||
|
||||
const scan = scanZstdFrames(buffer)
|
||||
expect(scan.frames).toHaveLength(2)
|
||||
const plaintext = await decodeCompleteFrames(buffer)
|
||||
expect(plaintext.toString()).toBe([
|
||||
JSON.stringify(toHeaderLine(header)),
|
||||
...oneTurnLog().map(e => JSON.stringify(e)),
|
||||
'',
|
||||
].join('\n'))
|
||||
expect((await ctx.sessionPersistence.load(header.id)).events).toEqual(oneTurnLog())
|
||||
})
|
||||
|
||||
it('resolves the default when a programmatic wrapper bypasses Loader schema normalization', async () => {
|
||||
const root = await freshRoot()
|
||||
const ctx = new Context()
|
||||
contexts.push(ctx)
|
||||
await ctx.plugin(SessionStore)
|
||||
let backend!: SessionPersistenceJsonl
|
||||
await ctx.plugin(Object.assign((inner: Context) => {
|
||||
backend = new SessionPersistenceJsonl(inner, { root })
|
||||
}, { inject: ['sessions'] }))
|
||||
const header = meta('direct-default')
|
||||
const path = logPath(root, header.cwd, header.id, 'zstd')
|
||||
expect(backend.locate(header)).toEqual({
|
||||
kind: 'jsonl',
|
||||
path,
|
||||
})
|
||||
|
||||
const base = oneTurnLog()
|
||||
const events: SessionEvent[] = [
|
||||
...base.slice(0, 3),
|
||||
...Array.from({ length: 3 }, (_, index): SessionEvent => ({
|
||||
type: 'assistant/chunk',
|
||||
seq: 3 + index,
|
||||
time: 4 + index,
|
||||
data: { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: `part-${index}` } },
|
||||
})),
|
||||
...base.slice(3).map((event): SessionEvent => ({
|
||||
...event,
|
||||
seq: event.seq + 3,
|
||||
time: event.time + 3,
|
||||
})),
|
||||
]
|
||||
await backend.create(header)
|
||||
await backend.append(header.id, events)
|
||||
|
||||
const plaintext = (await decodeCompleteFrames(await readFile(path))).toString()
|
||||
const recordTypes = plaintext.trimEnd().split('\n')
|
||||
.map(line => (JSON.parse(line) as { type: string }).type)
|
||||
expect(recordTypes).toContain('text-chunks')
|
||||
expect((await backend.load(header.id)).events).toEqual(events)
|
||||
})
|
||||
|
||||
it('appends one frame per durable batch without rewriting prior bytes', async () => {
|
||||
const root = await freshRoot()
|
||||
const ctx = await mount(root)
|
||||
const header = meta('append-frame')
|
||||
await ctx.sessionPersistence.create(header)
|
||||
await ctx.sessionPersistence.append(header.id, oneTurnLog())
|
||||
const path = logPath(root, header.cwd, header.id, 'zstd')
|
||||
const before = await readFile(path)
|
||||
const secondTurn = [
|
||||
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2 } },
|
||||
{ type: 'turn/end', seq: 7, time: 8, data: { turn: 2, reason: { kind: 'completed' } } },
|
||||
] as SessionEvent[]
|
||||
await ctx.sessionPersistence.append(header.id, secondTurn)
|
||||
|
||||
const after = await readFile(path)
|
||||
expect(after.subarray(0, before.length)).toEqual(before)
|
||||
expect(scanZstdFrames(after).frames).toHaveLength(3)
|
||||
expect((await ctx.sessionPersistence.load(header.id)).events).toEqual([...oneTurnLog(), ...secondTurn])
|
||||
})
|
||||
|
||||
it('lists from a multi-chunk header frame without decoding a corrupt event frame', async () => {
|
||||
const root = await freshRoot()
|
||||
const ctx = await mount(root)
|
||||
const header = meta('large-header', `/work/${'x'.repeat(24_000)}`)
|
||||
await ctx.sessionPersistence.create(header)
|
||||
await ctx.sessionPersistence.append(header.id, oneTurnLog())
|
||||
const path = logPath(root, header.cwd, header.id, 'zstd')
|
||||
const buffer = Buffer.from(await readFile(path))
|
||||
const eventFrame = scanZstdFrames(buffer).frames[1]!
|
||||
buffer[eventFrame.end - 1] = buffer[eventFrame.end - 1]! ^ 0xFF
|
||||
await writeFile(path, buffer)
|
||||
|
||||
expect((await ctx.sessionPersistence.list()).map(item => item.id)).toEqual([header.id])
|
||||
await expect(ctx.sessionPersistence.load(header.id)).rejects.toThrow(/frame at byte .* failed validation/)
|
||||
})
|
||||
|
||||
it('stops multi-frame inspection when cancellation arrives at a slice deadline', async () => {
|
||||
const root = await freshRoot()
|
||||
const ctx = await mount(root)
|
||||
const header = meta('cancel-zstd-frames')
|
||||
const headerFrame = await compressZstdFrame(`${JSON.stringify(toHeaderLine(header))}\n`)
|
||||
const eventFrame = await compressZstdFrame(`${JSON.stringify(oneTurnLog()[0])}\n`)
|
||||
const laterFrame = await compressZstdFrame(`${JSON.stringify(oneTurnLog()[1])}\n`)
|
||||
const stream = Buffer.concat([headerFrame, eventFrame, laterFrame])
|
||||
const controller = new AbortController()
|
||||
const reason = new Error('cancel after Zstandard decode starts')
|
||||
const reader = ctx.sessionPersistence as unknown as ZstdReaderInternals
|
||||
vi.spyOn(performance, 'now').mockReturnValueOnce(0).mockReturnValue(501)
|
||||
const pending = reader.readZstdPrefix(stream, controller.signal)
|
||||
queueMicrotask(() => { controller.abort(reason) })
|
||||
|
||||
await expect(pending).rejects.toBe(reason)
|
||||
})
|
||||
|
||||
it('continues decoding every frame after a slice deadline yields', async () => {
|
||||
const root = await freshRoot()
|
||||
const ctx = await mount(root)
|
||||
const header = meta('yield-zstd-frames')
|
||||
const events = oneTurnLog().slice(0, 2)
|
||||
const headerFrame = await compressZstdFrame(`${JSON.stringify(toHeaderLine(header))}\n`)
|
||||
const eventFrames = await Promise.all(events.map(async event => (
|
||||
compressZstdFrame(`${JSON.stringify(event)}\n`)
|
||||
)))
|
||||
const stream = Buffer.concat([headerFrame, ...eventFrames])
|
||||
const reader = ctx.sessionPersistence as unknown as ZstdReaderInternals
|
||||
vi.spyOn(performance, 'now').mockReturnValueOnce(0).mockReturnValue(501)
|
||||
|
||||
const prefix = await reader.readZstdPrefix(stream)
|
||||
|
||||
expect(prefix.events).toEqual(events)
|
||||
})
|
||||
|
||||
it.each(['none', 'zstd'] as const)(
|
||||
'observes cancellation after each async %s header read during listing',
|
||||
async (compression) => {
|
||||
const root = await freshRoot()
|
||||
const ctx = await mount(root, compression)
|
||||
const header = meta(`cancel-${compression}-header-read`, '/work')
|
||||
await ctx.sessionPersistence.create(header)
|
||||
await ctx.sessionPersistence.append(header.id, oneTurnLog())
|
||||
await ctx.sessionPersistence.list()
|
||||
const path = logPath(root, header.cwd, header.id, compression)
|
||||
const probe = await open(path, 'r')
|
||||
const prototype = Object.getPrototypeOf(probe) as { read: HeaderRead }
|
||||
const originalRead = prototype.read
|
||||
await probe.close()
|
||||
const controller = new AbortController()
|
||||
const reason = new Error(`cancel ${compression} header read`)
|
||||
const read = vi.spyOn(prototype, 'read').mockImplementation(async function (
|
||||
this: FileHandle,
|
||||
buffer: Buffer,
|
||||
offset: number,
|
||||
length: number,
|
||||
position: number | null,
|
||||
) {
|
||||
const result = await originalRead.call(this, buffer, offset, length, position)
|
||||
controller.abort(reason)
|
||||
return result
|
||||
})
|
||||
|
||||
await expect(ctx.sessionPersistence.list(controller.signal)).rejects.toBe(reason)
|
||||
expect(read).toHaveBeenCalledTimes(1)
|
||||
},
|
||||
)
|
||||
|
||||
it('preserves complete records from a torn frame and re-encodes them with crash closers', async () => {
|
||||
const root = await freshRoot()
|
||||
const ctx = await mount(root)
|
||||
const header = meta('recover-torn', '/proj')
|
||||
await ctx.sessionPersistence.create(header)
|
||||
await ctx.sessionPersistence.append(header.id, oneTurnLog())
|
||||
const path = logPath(root, header.cwd, header.id, 'zstd')
|
||||
const committed = await readFile(path)
|
||||
const openTurn = [
|
||||
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2 } },
|
||||
{ type: 'step/start', seq: 7, time: 8, data: { turn: 2, step: 1 } },
|
||||
{ type: 'assistant/chunk', seq: 8, time: 9, data: { turn: 2, step: 1, chunk: { type: 'text-delta', index: 0, text: deterministicNoise(300_000) } } },
|
||||
] as SessionEvent[]
|
||||
const plaintext = openTurn.map(e => JSON.stringify(e)).join('\n') + '\n'
|
||||
const partial = await tornFrame(plaintext, (decoded) => {
|
||||
const newlines = decoded.match(/\n/g)?.length ?? 0
|
||||
return newlines >= 2 && !decoded.endsWith('\n')
|
||||
})
|
||||
await appendFile(path, partial)
|
||||
|
||||
const loaded = await ctx.sessionPersistence.load(header.id)
|
||||
expect(loaded.events.map(event => event.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
|
||||
expect(loaded.events[6]).toEqual(openTurn[0])
|
||||
expect(loaded.events[7]).toEqual(openTurn[1])
|
||||
expect(loaded.events.some(event => event.type === 'assistant/chunk' && event.seq === 8)).toBe(false)
|
||||
expect(loaded.events[8]?.type).toBe('step/end')
|
||||
expect(loaded.events[9]?.type).toBe('turn/end')
|
||||
|
||||
const repaired = await readFile(path)
|
||||
expect(repaired.subarray(0, committed.length)).toEqual(committed)
|
||||
expect(scanZstdFrames(repaired).tornStart).toBeUndefined()
|
||||
expect(scanLog(await decodeCompleteFrames(repaired)).events).toEqual(loaded.events)
|
||||
})
|
||||
|
||||
it('drops a frame torn in its header before it has produced plaintext', async () => {
|
||||
const root = await freshRoot()
|
||||
const ctx = await mount(root)
|
||||
const header = meta('partial-magic')
|
||||
await ctx.sessionPersistence.create(header)
|
||||
await ctx.sessionPersistence.append(header.id, oneTurnLog())
|
||||
const path = logPath(root, header.cwd, header.id, 'zstd')
|
||||
const committed = await readFile(path)
|
||||
await appendFile(path, MAGIC.subarray(0, 2))
|
||||
|
||||
expect((await ctx.sessionPersistence.load(header.id)).events).toEqual(oneTurnLog())
|
||||
expect(await readFile(path)).toEqual(committed)
|
||||
})
|
||||
|
||||
it('recovers complete events when EOF tears only the final frame checksum', async () => {
|
||||
const root = await freshRoot()
|
||||
const ctx = await mount(root)
|
||||
const header = meta('partial-checksum')
|
||||
await ctx.sessionPersistence.create(header)
|
||||
await ctx.sessionPersistence.append(header.id, oneTurnLog())
|
||||
const path = logPath(root, header.cwd, header.id, 'zstd')
|
||||
const secondTurn = [
|
||||
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2 } },
|
||||
{ type: 'turn/end', seq: 7, time: 8, data: { turn: 2, reason: { kind: 'completed' } } },
|
||||
] as SessionEvent[]
|
||||
const frame = await compressZstdFrame(secondTurn.map(e => JSON.stringify(e)).join('\n') + '\n')
|
||||
await appendFile(path, frame.subarray(0, -1))
|
||||
|
||||
const loaded = await ctx.sessionPersistence.load(header.id)
|
||||
expect(loaded.events).toEqual([...oneTurnLog(), ...secondTurn])
|
||||
const repaired = await readFile(path)
|
||||
expect(scanZstdFrames(repaired).tornStart).toBeUndefined()
|
||||
expect(scanLog(await decodeCompleteFrames(repaired)).events).toEqual(loaded.events)
|
||||
})
|
||||
|
||||
it('rejects a complete frame containing a torn JSONL record', async () => {
|
||||
const root = await freshRoot()
|
||||
const ctx = await mount(root)
|
||||
const header = meta('complete-bad-jsonl')
|
||||
await ctx.sessionPersistence.create(header)
|
||||
await ctx.sessionPersistence.append(header.id, oneTurnLog())
|
||||
await appendFile(
|
||||
logPath(root, header.cwd, header.id, 'zstd'),
|
||||
await compressZstdFrame('{"type":"turn/start"'),
|
||||
)
|
||||
await expect(ctx.sessionPersistence.load(header.id)).rejects.toThrow(/complete frame contains a torn JSONL record/)
|
||||
})
|
||||
|
||||
it('rolls back a checksummed append frame when fsync fails', async () => {
|
||||
const root = await freshRoot()
|
||||
const ctx = await mount(root)
|
||||
const header = meta('zstd-fsync-rollback')
|
||||
await ctx.sessionPersistence.create(header)
|
||||
await ctx.sessionPersistence.append(header.id, oneTurnLog())
|
||||
const path = logPath(root, header.cwd, header.id, 'zstd')
|
||||
const before = await readFile(path)
|
||||
|
||||
const handle = await open(path, 'r')
|
||||
const prototype = Object.getPrototypeOf(handle) as { sync: () => Promise<void> }
|
||||
await handle.close()
|
||||
const realSync = prototype.sync
|
||||
let failed = false
|
||||
const spy = vi.spyOn(prototype, 'sync').mockImplementation(async function (this: FileHandle) {
|
||||
if (!failed) {
|
||||
failed = true
|
||||
throw new Error('simulated Zstandard fsync failure')
|
||||
}
|
||||
return realSync.call(this)
|
||||
})
|
||||
const secondTurn = [
|
||||
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2 } },
|
||||
{ type: 'turn/end', seq: 7, time: 8, data: { turn: 2, reason: { kind: 'completed' } } },
|
||||
] as SessionEvent[]
|
||||
await expect(ctx.sessionPersistence.append(header.id, secondTurn)).rejects.toThrow(/simulated Zstandard fsync failure/)
|
||||
expect(await readFile(path)).toEqual(before)
|
||||
spy.mockRestore()
|
||||
await ctx.sessionPersistence.append(header.id, secondTurn)
|
||||
expect((await ctx.sessionPersistence.load(header.id)).events).toEqual([...oneTurnLog(), ...secondTurn])
|
||||
})
|
||||
|
||||
it('skips empty, incomplete, and non-header compressed artifacts while rejecting malformed header frames', async () => {
|
||||
const root = await freshRoot()
|
||||
for (const [id, content] of [
|
||||
['empty', Buffer.alloc(0)],
|
||||
['partial', MAGIC],
|
||||
['not-header', await compressZstdFrame('{"type":"turn/start"}\n')],
|
||||
] as const) {
|
||||
const sessionId = SessionId(id)
|
||||
await mkdir(sessionDir(root, undefined, sessionId), { recursive: true })
|
||||
await writeFile(logPath(root, undefined, sessionId, 'zstd'), content)
|
||||
}
|
||||
const ctx = await mount(root)
|
||||
expect(await ctx.sessionPersistence.list()).toEqual([])
|
||||
|
||||
const twoLinesId = SessionId('two-lines')
|
||||
await mkdir(sessionDir(root, undefined, twoLinesId), { recursive: true })
|
||||
await writeFile(logPath(root, undefined, twoLinesId, 'zstd'), await compressZstdFrame([
|
||||
JSON.stringify(toHeaderLine(meta('two-lines'))),
|
||||
JSON.stringify({ type: 'turn/start' }),
|
||||
'',
|
||||
].join('\n')))
|
||||
await expect(ctx.sessionPersistence.list()).rejects.toThrow(/first frame is not exactly one header line/)
|
||||
await expect(ctx.sessionPersistence.load(SessionId('two-lines')))
|
||||
.rejects.toThrow(/first frame is not exactly one header line/)
|
||||
})
|
||||
|
||||
it('rejects missing, empty, and checksum-corrupt header frames on targeted reads', async () => {
|
||||
const root = await freshRoot()
|
||||
for (const id of ['partial-only', 'empty-header', 'bad-checksum']) {
|
||||
await mkdir(sessionDir(root, undefined, SessionId(id)), { recursive: true })
|
||||
}
|
||||
await writeFile(logPath(root, undefined, SessionId('partial-only'), 'zstd'), MAGIC)
|
||||
await writeFile(logPath(root, undefined, SessionId('empty-header'), 'zstd'), await compressZstdFrame(''))
|
||||
const corruptHeader = Buffer.from(await compressZstdFrame(`${JSON.stringify(toHeaderLine(meta('bad-checksum')))}\n`))
|
||||
corruptHeader[corruptHeader.length - 1] = corruptHeader[corruptHeader.length - 1]! ^ 0xFF
|
||||
await writeFile(logPath(root, undefined, SessionId('bad-checksum'), 'zstd'), corruptHeader)
|
||||
const ctx = await mount(root)
|
||||
|
||||
await expect(ctx.sessionPersistence.load(SessionId('partial-only')))
|
||||
.rejects.toThrow(/empty or header-less Zstandard session log/)
|
||||
await expect(ctx.sessionPersistence.load(SessionId('empty-header')))
|
||||
.rejects.toThrow(/first frame is not exactly one header line/)
|
||||
await expect(ctx.sessionPersistence.list()).rejects.toThrow(/header frame failed validation/)
|
||||
})
|
||||
})
|
||||
|
||||
describe('SessionPersistenceJsonl: encoding selection', () => {
|
||||
it('rejects roots owned by the opposite encoding in both directions', async () => {
|
||||
const rawRoot = await freshRoot('dsh-jsonl-raw-mismatch-')
|
||||
const raw = await mount(rawRoot, 'none')
|
||||
const rawHeader = meta('raw-log')
|
||||
await raw.sessionPersistence.create(rawHeader)
|
||||
await raw.sessionPersistence.append(rawHeader.id, oneTurnLog())
|
||||
const defaultBackend = await mount(rawRoot)
|
||||
await expect(defaultBackend.sessionPersistence.list()).rejects.toThrow(/configured for compression "zstd"/)
|
||||
|
||||
const zstdRoot = await freshRoot('dsh-jsonl-zstd-mismatch-')
|
||||
const zstd = await mount(zstdRoot)
|
||||
const zstdHeader = meta('zstd-log')
|
||||
await zstd.sessionPersistence.create(zstdHeader)
|
||||
await zstd.sessionPersistence.append(zstdHeader.id, oneTurnLog())
|
||||
const rawBackend = await mount(zstdRoot, 'none')
|
||||
await expect(rawBackend.sessionPersistence.list()).rejects.toThrow(/configured for compression "none"/)
|
||||
})
|
||||
|
||||
it('rechecks targeted artifacts and listing after an initially empty root', async () => {
|
||||
const root = await freshRoot()
|
||||
const ctx = await mount(root)
|
||||
expect(await ctx.sessionPersistence.list()).toEqual([])
|
||||
|
||||
const loadHeader = meta('late-raw-load', '/late')
|
||||
await mkdir(sessionDir(root, loadHeader.cwd, loadHeader.id), { recursive: true })
|
||||
await writeFile(logPath(root, loadHeader.cwd, loadHeader.id, 'none'), [
|
||||
JSON.stringify(toHeaderLine(loadHeader)),
|
||||
...oneTurnLog().map(e => JSON.stringify(e)),
|
||||
'',
|
||||
].join('\n'))
|
||||
await expect(ctx.sessionPersistence.load(loadHeader.id)).rejects.toThrow(/uses \.jsonl/)
|
||||
await expect((ctx.sessionPersistence as SessionPersistenceJsonl).loadStored(loadHeader.id))
|
||||
.rejects.toThrow(/uses \.jsonl/)
|
||||
await expect(ctx.sessionPersistence.list()).rejects.toThrow(/uses \.jsonl/)
|
||||
})
|
||||
|
||||
it('refuses materialization when an opposite artifact appears after create', async () => {
|
||||
const root = await freshRoot()
|
||||
const ctx = await mount(root)
|
||||
await ctx.sessionPersistence.list()
|
||||
const header = meta('late-raw-materialize', '/late')
|
||||
await ctx.sessionPersistence.create(header)
|
||||
await mkdir(sessionDir(root, header.cwd, header.id), { recursive: true })
|
||||
await writeFile(logPath(root, header.cwd, header.id, 'none'), [
|
||||
JSON.stringify(toHeaderLine(header)),
|
||||
...oneTurnLog().map(e => JSON.stringify(e)),
|
||||
'',
|
||||
].join('\n'))
|
||||
await expect(ctx.sessionPersistence.append(header.id, oneTurnLog())).rejects.toThrow(/uses \.jsonl/)
|
||||
expect((await readdir(sessionDir(root, header.cwd, header.id))).some(name => name.endsWith('.jsonl.zstd'))).toBe(false)
|
||||
})
|
||||
})
|
||||
30
packages/session/session-persistence-jsonl/tsconfig.json
Normal file
30
packages/session/session-persistence-jsonl/tsconfig.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
{
|
||||
"path": "../session-persistence"
|
||||
},
|
||||
{
|
||||
"path": "../../support/invariants"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/session-persistence/session-persistence-sqlite/README.md
|
||||
README.md: 4961f62bf6854c343d35b8406c9d721590274534
|
||||
README.zh.md: 8610ef56f737bac8781536ea26a5b48dbf67ed48
|
||||
63
packages/session/session-persistence-sqlite/README.md
Normal file
63
packages/session/session-persistence-sqlite/README.md
Normal file
@@ -0,0 +1,63 @@
|
||||
# @deepseek-ai/dsh-session-persistence-sqlite
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
A SQLite durable session-persistence backend — a second `SessionPersistence` implementation ([session persistence](../../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md)), built to validate that the abstract seam and the shared `runPersistenceContract` suite are genuinely backend-agnostic. It satisfies the SAME contract as `dsh-session-persistence-jsonl` (append-only, contiguous-seq, lazy materialization, interrupted-turn close on load), expressed over `node:sqlite` rows instead of file bytes.
|
||||
|
||||
`locate(meta)` returns `undefined`: all sessions share one database, so there is no honest independent per-session transcript path.
|
||||
|
||||
> **TODO:** this backend talks to `node:sqlite` directly. If a cordis database service (`cordis/db` / a `@cordisjs` SQL driver plugin) is adopted, route through that instead of holding a raw `DatabaseSync` here — the contract surface (`SessionPersistence`) would not change, only the storage driver.
|
||||
|
||||
## Storage model
|
||||
|
||||
Each `SessionEvent` maps 1:1 onto a row in an `events` table `(session_id, seq, type, time, data, source_event_seqs, surface_op)` — `data` is the event payload as JSON text, so the row shape is the event verbatim (including `assistant/chunk`, keeping `seq` contiguous). The two `TEXT` columns `source_event_seqs` and `surface_op` are nullable; they store the event's optional surface-metadata fields (see [session surface](../../../.agents/notes/implemented/architecture/2026-06-18-session-surface.md)). Out-of-log metadata (`SessionHeader`), a per-materialization incarnation id, and a monotonic per-log revision live in a `sessions` row; `createdAt` is a non-negative safe integer stored in a strict `INTEGER` column. A singleton state row carries the immutable store id. A `sessions` row is written only by the first `append` — its existence is the lazy-materialization signal (`list` reports exactly the sessions that have a row).
|
||||
|
||||
The repository's Node range supports unflagged `node:sqlite`. The database enables foreign keys and uses the configured journal mode (`wal` by default; use a rollback mode where WAL shared-memory files are unsuitable). `PRAGMA application_id` identifies the canonical persistence database, and `PRAGMA user_version` stores its layout version. A fresh database must have no application identity or user-defined schema objects; initialization creates every table and stamps both pragmas in one transaction. Non-pristine unversioned databases, foreign application identities, and every non-current version reject before journal-mode mutation because this unreleased format has no migrations.
|
||||
|
||||
On filesystems with POSIX modes, the backend requests mode `0700` for missing directories and exclusively creates a missing database with mode `0600` before SQLite opens it; the process umask may further restrict both. New WAL, shared-memory, and persistent rollback-journal sidecars receive the database's resulting owner-only mode. Existing directories, database files, and sidecars keep their modes; filesystem setup errors other than an existing database fail initialization. These defaults prevent incidental exposure through a permissive process umask, but do not protect database confidentiality or integrity when another principal can replace the database entry in its parent directory.
|
||||
|
||||
## Contract semantics over rows
|
||||
|
||||
- **Append = a transaction.** `append` runs `BEGIN`/`COMMIT` around the batch: it materializes the `sessions` row (if still lazy) and INSERTs every event, asserting the contiguous-seq contract first (the first event's `seq` must equal the stored next-seq). A mid-batch failure (a UNIQUE violation on a duplicated seq) rolls back entirely, so the stored log and the in-memory cursor stay consistent. (`load()` already balanced the stored log, so `append` never has to repair a crash tail.)
|
||||
- **Lazy materialization.** `create()` records intent in memory only — no row is written until the first `append`. A created-but-never-appended session has no `sessions` row, so it is absent from `list()` (which reports exactly the sessions that have a row).
|
||||
- **Interrupted-turn close on load.** `load()` implements the shared [crash-recovery contract](../../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md): preserve the valid interrupted turn, append its synthetic closing events in one transaction, and remove only a torn tail row. Committed parse errors or sequence gaps make the session unloadable. Because recovery mutates stored rows, the next append starts from a balanced log and accurate cursor.
|
||||
- **Non-mutating inspection.** `inspect()` returns an immutable balanced logical view and may synthesize recovery closers in memory, without deleting a torn tail row, appending recovery rows, or changing the lightweight revision.
|
||||
- **Lightweight revisions.** `listSnapshots(signal?)` combines the immutable store and database-file identity, a per-materialization incarnation id, and a per-session counter incremented in each mutating transaction. A full-prefix read captures that revision and its event rows in one read transaction, while `readStoredRevision()` queries only the session row to validate retained preparations. This keeps unchanged observations stable without parsing event rows and distinguishes independent stores and recreated same-id logs. It checks cancellation before and after shared readiness and the synchronous metadata query; the query itself is non-preemptible.
|
||||
|
||||
## Configuration (schemastery)
|
||||
|
||||
```ts
|
||||
interface Config {
|
||||
path: string // SQLite database file path, or ':memory:' for an in-process DB
|
||||
journalMode?: 'wal' | 'delete' | 'truncate' | 'persist' // journal_mode pragma; default 'wal'
|
||||
preparedSessionCacheSize?: number // positive integer; default 5
|
||||
writeBatchMaxDelayMs?: number // positive integer; default 200; maximum 2_147_483_647
|
||||
}
|
||||
```
|
||||
|
||||
## Write path
|
||||
|
||||
Like the JSONL backend, the plugin copies each frozen `session/event` into one controller per live session. The first pending event starts the configured fixed batching window, and later events join without resetting it. Expiry starts one transaction; events admitted during that write form a separately bounded follow-up batch. `session/flush` cancels the wait and drains current and pending batches. The controller persists a fork's seed once, keeps a write cursor so resume never re-appends stored events, and seeds live sessions on apply because HMR does not replay `session/created`. Dispose drains every retained controller before closing the database. Every event remains a separate SQLite row; batching only groups more INSERTs into one transaction and revision increment.
|
||||
|
||||
## Model Experience
|
||||
|
||||
### Resumed conversation history
|
||||
|
||||
#### What the model sees
|
||||
|
||||
SQLite storage contributes no live prompt or schema. Loading restores the same surface history as JSONL and preserves prior headers for reconstruction; the new loop composes its current envelope. Recovery balances an assistant request without a durable call with `TOOL_NOT_STARTED`; a durable call without a result becomes `TOOL_OUTCOME_UNKNOWN`, which tells the model to retry only read-only or idempotent work and to verify possible side effects or ask the user. Row metadata and raw chunks are not messages.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Zero live-request tokens. Resume restores retained history and pays the current envelope, plus the quoted repair result for each interrupted call.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
SQLite storage does not mutate live request prefixes. A resumed loop can reuse provider cache only when its reconstructed history, current envelope, and model route match; crash-repair results append.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **`DatabaseSync` is synchronous** — every append transaction blocks the event loop for its duration; acceptable for local stores, a throughput ceiling for busy multi-session servers.
|
||||
- **Write contention has no wait or retry policy** — the backend sets no busy timeout and retries no locked-database error, so another connection holding a write transaction makes the operation reject immediately.
|
||||
- **Only a pristine new database or the current owned `SCHEMA_VERSION` opens** — unversioned schema objects, foreign application identities, and every other schema version are rejected rather than migrated (unreleased software; no persisted user data to preserve).
|
||||
- **Nothing deletes stored sessions** — rows accumulate until removed externally (the seam has no deletion surface; `ON DELETE CASCADE` is wired for such out-of-band cleanup).
|
||||
63
packages/session/session-persistence-sqlite/README.zh.md
Normal file
63
packages/session/session-persistence-sqlite/README.zh.md
Normal file
@@ -0,0 +1,63 @@
|
||||
# @deepseek-ai/dsh-session-persistence-sqlite
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
SQLite 持久会话存储后端：第二个 `SessionPersistence` 实现（见[会话持久化](../../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md)），用于验证抽象 seam 和共享 `runPersistenceContract` 套件真正与后端无关。它满足与 `dsh-session-persistence-jsonl` 相同的契约（仅追加、连续 seq、延迟实体化、在 load 时关闭中断轮次），但用 `node:sqlite` 行而非文件字节表达。
|
||||
|
||||
`locate(meta)` 返回 `undefined`：所有会话共享一个数据库，因此不存在真实、独立的逐会话 transcript（文本记录）路径。
|
||||
|
||||
> **TODO：** 该后端直接调用 `node:sqlite`。如果采用 Cordis 数据库服务（`cordis/db` / `@cordisjs` SQL driver 插件），应改为通过该服务路由，而不在此直接持有 `DatabaseSync`；契约接口（`SessionPersistence`）不会变，只更换存储驱动。
|
||||
|
||||
## 存储模型
|
||||
|
||||
每个 `SessionEvent` 1:1 映射到 `events` 表中的一行 `(session_id, seq, type, time, data, source_event_seqs, surface_op)`；`data` 是作为 JSON 文本的事件 payload，因此行结构就是原始事件本身（包括 `assistant/chunk`，保持 `seq` 连续）。两个 `TEXT` 列 `source_event_seqs` 和 `surface_op` 可为空，存储事件可选接口元数据字段（见[会话接口](../../../.agents/notes/implemented/architecture/2026-06-18-session-surface.md)）。日志外元数据（`SessionHeader`）、每实体化 incarnation id 和每日志单调修订位于 `sessions` 行；`createdAt` 是存储在 strict `INTEGER` 列中的非负安全整数。单例状态行携带不可变存储 id。`sessions` 行只由第一次 `append` 写入，其存在性是延迟实体化信号（`list` 精确报告有行的会话）。
|
||||
|
||||
仓库支持的 Node 范围可不加 flag 使用 `node:sqlite`。数据库启用外键，并使用已配置 journal mode（默认 `wal`；WAL 共享内存文件不适用时使用 rollback mode）。`PRAGMA application_id` 标识规范持久化数据库，`PRAGMA user_version` 存储布局版本。新数据库必须没有 application identity 或用户定义 schema 对象；初始化在一个事务中创建全部表并盖上两个 pragma。非 pristine 无版本数据库、外部 application identity 和所有非当前版本在 journal-mode 变更前均会被拒绝，因为该未发布格式无迁移。
|
||||
|
||||
在具有 POSIX mode 的文件系统上，后端为缺失目录请求 mode `0700`，并在 SQLite 打开前以 mode `0600` 排他创建缺失数据库；进程 umask 可进一步限制两者。新 WAL、共享内存和持久 rollback-journal sidecar 获得数据库最终的仅所有者 mode。现有目录、数据库文件和 sidecar 保留原 mode；除已存在数据库外的文件系统设置错误会使初始化失败。这些默认值防止宽松进程 umask 造成的意外暴露，但当其他 principal 能替换父目录中的数据库条目时，不保护数据库机密性或完整性。
|
||||
|
||||
## 行上的契约语义
|
||||
|
||||
- **Append = 事务。**`append` 围绕批次运行 `BEGIN`/`COMMIT`：它实体化 `sessions` 行（如果仍延迟），并 INSERT 每个事件，首先断言连续 seq 契约（第一个事件 `seq` 必须等于已存储 next-seq）。批次中失败（重复 seq 上的 UNIQUE 违规）会完全回滚，使已存储日志和内存游标保持一致。（`load()` 已平衡已存储日志，因此 `append` 不必修复崩溃尾部。）
|
||||
- **延迟实体化。**`create()` 只在内存记录意图，第一次 `append` 前不写行。从未 append 的会话没有 `sessions` 行，因此不在 `list()` 中（它精确报告有行的会话）。
|
||||
- **在 load 时关闭中断轮次。**`load()` 实现共享[崩溃恢复契约](../../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md)：保留有效中断轮次，在一个事务中追加合成关闭事件，并只移除撕裂尾部行。已提交解析错误或序列缺口使会话无法加载。恢复会变更已存储行，因此下一次 append 从平衡日志和准确游标开始。
|
||||
- **非变更检查。**`inspect()` 返回不可变、平衡的逻辑视图，并可在内存中合成恢复 closer，但不会删除撕裂尾部行、追加恢复行或更改轻量修订。
|
||||
- **轻量修订。**`listSnapshots(signal?)` 组合不可变存储与数据库文件身份、每实体化 incarnation id，以及在每个变更事务中递增的每会话计数器。完整前缀读取在同一个读事务中捕获该 revision 及其事件行，`readStoredRevision()` 则只查询 session 行来校验保留的 preparation。它在不解析事件行的情况下保持未变观察稳定，并区分独立存储和重建的同 id 日志。它在共享就绪和同步元数据查询前后检查取消；查询本身不可抢占。
|
||||
|
||||
## 配置（schemastery）
|
||||
|
||||
```ts
|
||||
interface Config {
|
||||
path: string // SQLite database file path, or ':memory:' for an in-process DB
|
||||
journalMode?: 'wal' | 'delete' | 'truncate' | 'persist' // journal_mode pragma; default 'wal'
|
||||
preparedSessionCacheSize?: number // positive integer; default 5
|
||||
writeBatchMaxDelayMs?: number // positive integer; default 200; maximum 2_147_483_647
|
||||
}
|
||||
```
|
||||
|
||||
## 写入路径
|
||||
|
||||
与 JSONL 后端一样，插件将每个冻结的 `session/event` 复制到每个活动会话各自的 controller。第一个待处理事件会开启配置的固定批处理窗口，后续事件会加入但不会重置截止时间。窗口到期后会启动一个事务；该次写入期间接纳的事件会形成另一个独立有界的后续批次。`session/flush` 会取消等待并排空当前与待处理批次。Controller 会持久化一次 fork 种子，并保留写入游标，使恢复操作绝不重新 append 已存储事件；它还会在 apply 时为活动会话设置初始状态，因为 HMR（热模块替换）不回放 `session/created`。dispose（资源释放）会在关闭数据库前排空每个保留的 controller。每个事件仍各占一行 SQLite 记录；批处理只把更多 INSERT 归入同一个事务和同一次修订版本递增。
|
||||
|
||||
## 模型体验
|
||||
|
||||
### 恢复的对话历史
|
||||
|
||||
#### 模型看到的内容
|
||||
|
||||
SQLite 存储不影响当前提示词或 schema。加载会恢复与 JSONL 相同的呈现历史，并保留之前的 header 用于重建；新 loop 组合当前 envelope。恢复会用 `TOOL_NOT_STARTED` 平衡没有已持久化调用的 assistant 请求；已有已持久化调用但无结果时则变为 `TOOL_OUTCOME_UNKNOWN`，它要求模型只重试只读或幂等工作，并验证可能的副作用或询问用户。行元数据和原始分片不会成为消息。
|
||||
|
||||
#### Token 影响
|
||||
|
||||
当前请求不会新增 token。恢复会还原已保留的历史，并产生当前 envelope 以及每个中断调用所附修复结果文本的 token 开销。
|
||||
|
||||
#### KV Cache 影响
|
||||
|
||||
SQLite 存储不修改当前请求前缀。只有重建历史、当前 envelope 和模型路由匹配时，恢复 loop 才能重用提供方缓存；崩溃修复结果会追加到末尾。
|
||||
|
||||
## 已知限制与暂缓事项
|
||||
|
||||
- **`DatabaseSync` 是同步的**：每个 append 事务在整个期间阻塞事件循环；对本地存储可接受，对繁忙多会话服务器是吞吐上限。
|
||||
- **写入争用无等待或重试策略**：后端不设置 busy timeout，也不重试 locked-database 错误，因此其他连接持有写事务时操作立即拒绝。
|
||||
- **只有 pristine 新数据库或当前自有 `SCHEMA_VERSION` 才能打开**：无版本 schema 对象、外部 application identity 和所有其他 schema 版本被拒绝，而不是迁移（未发布软件，无持久用户数据需要保留）。
|
||||
- **不删除已存储会话**：行会累积，直到外部移除（seam 无删除接口；`ON DELETE CASCADE` 已为这种带外清理配置）。
|
||||
42
packages/session/session-persistence-sqlite/package.json
Normal file
42
packages/session/session-persistence-sqlite/package.json
Normal file
@@ -0,0 +1,42 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-session-persistence-sqlite",
|
||||
"description": "SQLite durable session persistence backend for the DeepSeek Harness",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./invariant": {
|
||||
"types": "./lib/types/invariant.d.ts",
|
||||
"default": "./lib/invariant.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-invariants": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session-persistence": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"dependencies": {
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-persistence": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
407
packages/session/session-persistence-sqlite/src/index.ts
Normal file
407
packages/session/session-persistence-sqlite/src/index.ts
Normal file
@@ -0,0 +1,407 @@
|
||||
/**
|
||||
* SQLite durable session-persistence backend. It maps each session header and
|
||||
* event to rows, and delegates write-path orchestration to
|
||||
* {@link PersistenceCoordinator}. It has no independent per-session artifact,
|
||||
* so its locator returns `undefined`.
|
||||
* @module @deepseek-ai/dsh-session-persistence-sqlite
|
||||
*/
|
||||
|
||||
import { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import { statSync } from 'node:fs'
|
||||
import { DatabaseSync } from 'node:sqlite'
|
||||
import { mkdir, open } from 'node:fs/promises'
|
||||
import { dirname, resolve } from 'node:path'
|
||||
import {
|
||||
DEFAULT_PREPARED_SESSION_CACHE_SIZE, DEFAULT_WRITE_BATCH_MAX_DELAY_MS, MAX_WRITE_BATCH_DELAY_MS,
|
||||
SessionPersistence, SessionPersistenceRevision, PersistenceCoordinator,
|
||||
type PersistenceBackend, type SessionLocation, type SessionPersistenceSnapshot,
|
||||
type SessionInspection, type SessionPersistenceRevision as PersistenceRevision,
|
||||
type StoredPrefix, type StoredSuffix,
|
||||
} from '@deepseek-ai/dsh-session-persistence'
|
||||
import type { SessionEvent, SurfaceEventType, SessionId, SessionHeader, SessionPreparation } from '@deepseek-ai/dsh-session'
|
||||
import {
|
||||
type JournalMode, openDatabase, rowToMeta, scanRows, type EventRow, type SessionRow,
|
||||
} from './schema.ts'
|
||||
|
||||
export { SCHEMA_VERSION } from './schema.ts'
|
||||
|
||||
/**
|
||||
* Serialize an event's surface-metadata fields for SQL binding. Both fields are
|
||||
* nullable TEXT columns — null when the event has no surface metadata (non-surface
|
||||
* events, events written before surface support).
|
||||
*/
|
||||
function surfaceBindings(event: SessionEvent): [string | null, string | null] {
|
||||
const se = event as SessionEvent<SurfaceEventType>
|
||||
return [
|
||||
se.sourceEventSeqs ? JSON.stringify(se.sourceEventSeqs) : null,
|
||||
se.surfaceOp !== undefined ? JSON.stringify(se.surfaceOp) : null,
|
||||
]
|
||||
}
|
||||
|
||||
/** Build the source-qualified revision shared by full and lightweight reads. */
|
||||
function sqliteRevision(storeIdentity: string, row: SessionRow): PersistenceRevision {
|
||||
return SessionPersistenceRevision(
|
||||
`${storeIdentity}:incarnation:${row.incarnation}:revision:${row.revision}`,
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Exclusively create a missing database file with owner-only permissions.
|
||||
* Existing files retain their modes, and errors other than `EEXIST` propagate.
|
||||
* `DatabaseSync` reopens by path, so this does not protect confidentiality or
|
||||
* integrity when another principal can replace the database entry in its parent
|
||||
* directory.
|
||||
*/
|
||||
async function createDatabaseFile(path: string): Promise<void> {
|
||||
try {
|
||||
const handle = await open(path, 'wx', 0o600)
|
||||
await handle.close()
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code !== 'EEXIST') throw error
|
||||
}
|
||||
}
|
||||
|
||||
/** Plugin configuration. */
|
||||
export interface Config {
|
||||
/**
|
||||
* Filesystem path to the SQLite database file. The special value `:memory:`
|
||||
* opens an in-process database (tests). On filesystems with POSIX modes,
|
||||
* missing directories and databases are created owner-only; existing path
|
||||
* modes are preserved. Filesystem setup errors other than an existing database
|
||||
* fail initialization. The backend does not protect confidentiality or
|
||||
* integrity when another principal can replace the database entry in its
|
||||
* parent directory.
|
||||
*/
|
||||
path: string
|
||||
/**
|
||||
* SQLite `journal_mode` pragma. `wal` (the default) is the recorded
|
||||
* durability model; pick a rollback-journal mode (`delete`/`truncate`/
|
||||
* `persist`) on filesystems where WAL's shared-memory files do not work
|
||||
* (network mounts). See {@link JournalMode}.
|
||||
*/
|
||||
journalMode?: JournalMode
|
||||
/** Maximum cold Session preparations retained for history-to-resume reuse. */
|
||||
preparedSessionCacheSize?: number
|
||||
/** Fixed live-event coalescing window; not a backend completion deadline. */
|
||||
writeBatchMaxDelayMs?: number
|
||||
}
|
||||
|
||||
/**
|
||||
* The SQLite persistence backend. Load as a plugin; it registers as
|
||||
* `ctx.sessionPersistence` and (via the coordinator) installs the write-path
|
||||
* listeners. Its torn-tail marker is the seq to delete from.
|
||||
*/
|
||||
export class SessionPersistenceSqlite extends SessionPersistence implements PersistenceBackend<number> {
|
||||
static inject = ['sessions']
|
||||
|
||||
static Config: z<Config> = z.object({
|
||||
path: z.string().required(),
|
||||
journalMode: z.union(['wal', 'delete', 'truncate', 'persist'] as const).default('wal'),
|
||||
preparedSessionCacheSize: z.number().step(1).min(1).default(DEFAULT_PREPARED_SESSION_CACHE_SIZE),
|
||||
writeBatchMaxDelayMs: z.number().step(1).min(1).max(MAX_WRITE_BATCH_DELAY_MS)
|
||||
.default(DEFAULT_WRITE_BATCH_MAX_DELAY_MS),
|
||||
})
|
||||
|
||||
/**
|
||||
* Backend label for the coordinator's dispose diagnostics. Intentionally
|
||||
* shadows cordis `Service.name` (set to `'sessionPersistence'` by the base);
|
||||
* see the JSONL backend for why this does not affect service resolution.
|
||||
*/
|
||||
override readonly name = 'session-persistence-sqlite'
|
||||
|
||||
private db!: DatabaseSync
|
||||
private storeIdentity!: string
|
||||
private ready: Promise<void>
|
||||
private coordinator: PersistenceCoordinator<number>
|
||||
|
||||
constructor(ctx: Context, public config: Config) {
|
||||
super(ctx)
|
||||
// Programmatic wrappers may construct the backend without Schemastery normalization.
|
||||
const preparedSessionCacheSize = config.preparedSessionCacheSize
|
||||
?? DEFAULT_PREPARED_SESSION_CACHE_SIZE
|
||||
const writeBatchMaxDelayMs = config.writeBatchMaxDelayMs
|
||||
?? DEFAULT_WRITE_BATCH_MAX_DELAY_MS
|
||||
// Open asynchronously so directory creation does not block plugin apply;
|
||||
// every storage hook awaits the same readiness promise.
|
||||
this.ready = this.openDb(config.path, (config as Required<Config>).journalMode)
|
||||
this.coordinator = new PersistenceCoordinator<number>(this.ctx, this, {
|
||||
preparedSessionCacheSize,
|
||||
writeBatchMaxDelayMs,
|
||||
})
|
||||
}
|
||||
|
||||
private async openDb(path: string, journalMode: JournalMode): Promise<void> {
|
||||
const actual = path === ':memory:' ? path : resolve(path)
|
||||
if (actual !== ':memory:') {
|
||||
await mkdir(dirname(actual), { recursive: true, mode: 0o700 })
|
||||
await createDatabaseFile(actual)
|
||||
}
|
||||
this.db = openDatabase(actual, journalMode)
|
||||
try {
|
||||
const row = this.db.prepare(
|
||||
'SELECT store_id FROM persistence_state WHERE singleton = 1',
|
||||
).get() as { store_id: string } | undefined
|
||||
/* v8 ignore next -- openDatabase inserts the singleton before returning. */
|
||||
if (row === undefined) {
|
||||
throw new Error(`session database at "${actual}" has no store identity`)
|
||||
}
|
||||
if (row.store_id.length === 0) {
|
||||
throw new Error(`session database at "${actual}" has no valid store identity`)
|
||||
}
|
||||
if (actual !== ':memory:') {
|
||||
const identity = statSync(actual, { bigint: true })
|
||||
this.storeIdentity = `file:${identity.dev}:${identity.ino}:${identity.birthtimeNs}:store:${row.store_id}`
|
||||
} else {
|
||||
this.storeIdentity = `memory:store:${row.store_id}`
|
||||
}
|
||||
} catch (error: unknown) {
|
||||
this.db.close()
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
// --- SessionPersistence service surface (delegated to the coordinator) ---
|
||||
|
||||
/** SQLite has one database, not an independent local artifact per session. */
|
||||
locate(_meta: SessionHeader): SessionLocation | undefined {
|
||||
return undefined
|
||||
}
|
||||
|
||||
create(meta: SessionHeader): Promise<void> {
|
||||
return this.coordinator.create(meta)
|
||||
}
|
||||
|
||||
append(id: SessionId, events: readonly SessionEvent[]): Promise<void> {
|
||||
return this.coordinator.append(id, events)
|
||||
}
|
||||
|
||||
override prepare(id: SessionId, signal?: AbortSignal): Promise<SessionPreparation> {
|
||||
return this.coordinator.prepare(id, signal)
|
||||
}
|
||||
|
||||
load(id: SessionId): Promise<SessionInspection> {
|
||||
return this.coordinator.load(id)
|
||||
}
|
||||
|
||||
inspect(id: SessionId, signal?: AbortSignal): Promise<SessionInspection> {
|
||||
return this.coordinator.inspect(id, signal)
|
||||
}
|
||||
|
||||
readFrom(id: SessionId, fromSeq: number, signal?: AbortSignal): Promise<{ meta: SessionHeader; events: SessionEvent[] }> {
|
||||
return this.coordinator.readFrom(id, fromSeq, signal)
|
||||
}
|
||||
|
||||
// One method serves both public `list` and the backend hook; delegating it to
|
||||
// the coordinator would call this hook recursively.
|
||||
|
||||
// --- PersistenceBackend hooks (the SQLite storage primitives) ---
|
||||
|
||||
/** Read a stored prefix by id (ids are globally unique — no scope to scan). */
|
||||
loadStored(id: SessionId, signal?: AbortSignal): Promise<StoredPrefix<number> | undefined> {
|
||||
return this.readPrefix(id, signal)
|
||||
}
|
||||
|
||||
/** Read one row's revision without loading its events. */
|
||||
async readStoredRevision(id: SessionId, signal?: AbortSignal): Promise<PersistenceRevision | undefined> {
|
||||
signal?.throwIfAborted()
|
||||
await this.ready
|
||||
signal?.throwIfAborted()
|
||||
const row = this.rowFor(id)
|
||||
return row === undefined ? undefined : sqliteRevision(this.storeIdentity, row)
|
||||
}
|
||||
|
||||
/**
|
||||
* Seek-capable suffix read: SQL selects `seq >= fromSeq` directly, so the
|
||||
* read scales with the suffix, not the log. Torn rows past the preserved
|
||||
* region are dropped, never repaired (non-mutating read).
|
||||
*/
|
||||
async loadStoredFrom(id: SessionId, fromSeq: number, signal?: AbortSignal): Promise<StoredSuffix | undefined> {
|
||||
signal?.throwIfAborted()
|
||||
await this.ready
|
||||
signal?.throwIfAborted()
|
||||
const row = this.rowFor(id)
|
||||
if (row === undefined) return undefined
|
||||
const meta = rowToMeta(row)
|
||||
const eventRows = this.db
|
||||
.prepare('SELECT seq, type, time, data, source_event_seqs, surface_op FROM events WHERE session_id = ? AND seq >= ? ORDER BY seq')
|
||||
.all(id, fromSeq) as unknown as EventRow[]
|
||||
signal?.throwIfAborted()
|
||||
const { preserved } = scanRows(eventRows, fromSeq)
|
||||
return { meta, events: preserved }
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a session's row + ordered events into a {@link StoredPrefix}. The
|
||||
* torn-tail marker is the seq from which a never-committed tail must be deleted
|
||||
* (`scanRows` already returns it as `number | undefined`).
|
||||
*/
|
||||
private async readPrefix(id: SessionId, signal?: AbortSignal): Promise<StoredPrefix<number> | undefined> {
|
||||
signal?.throwIfAborted()
|
||||
await this.ready
|
||||
signal?.throwIfAborted()
|
||||
this.db.exec('BEGIN')
|
||||
let snapshot: { row: SessionRow; eventRows: EventRow[] } | undefined
|
||||
try {
|
||||
const row = this.rowFor(id)
|
||||
if (row !== undefined) {
|
||||
const eventRows = this.db
|
||||
.prepare('SELECT seq, type, time, data, source_event_seqs, surface_op FROM events WHERE session_id = ? ORDER BY seq')
|
||||
.all(id) as unknown as EventRow[]
|
||||
snapshot = { row, eventRows }
|
||||
}
|
||||
this.db.exec('COMMIT')
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore start -- synchronous read failures only need transaction cleanup before propagation. */
|
||||
this.db.exec('ROLLBACK')
|
||||
throw error
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
signal?.throwIfAborted()
|
||||
if (snapshot === undefined) return undefined
|
||||
const { row, eventRows } = snapshot
|
||||
const { preserved, tornFrom } = scanRows(eventRows)
|
||||
return {
|
||||
meta: rowToMeta(row),
|
||||
events: preserved,
|
||||
revision: sqliteRevision(this.storeIdentity, row),
|
||||
...tornFrom !== undefined ? { tornMarker: tornFrom } : {},
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Durably append a batch in ONE transaction: materialize the sessions row (if
|
||||
* lazy) and INSERT every event, or roll back entirely. The transaction is the
|
||||
* atomicity + durability boundary, so a mid-batch failure (a UNIQUE violation
|
||||
* on a duplicated seq) leaves the stored log untouched.
|
||||
*/
|
||||
async appendBatch(meta: SessionHeader, events: readonly SessionEvent[], isMaterialized: boolean): Promise<void> {
|
||||
await this.ready
|
||||
const insertEvent = this.db.prepare(
|
||||
'INSERT INTO events (session_id, seq, type, time, data, source_event_seqs, surface_op) VALUES (?, ?, ?, ?, ?, ?, ?)',
|
||||
)
|
||||
this.db.exec('BEGIN')
|
||||
try {
|
||||
if (!isMaterialized) this.writeRow(meta)
|
||||
for (const event of events) {
|
||||
const [surfaceSeqs, surfaceOp] = surfaceBindings(event)
|
||||
insertEvent.run(meta.id, event.seq, event.type, event.time, JSON.stringify(event.data), surfaceSeqs, surfaceOp)
|
||||
}
|
||||
this.db.prepare('UPDATE sessions SET revision = revision + 1 WHERE id = ?').run(meta.id)
|
||||
this.db.exec('COMMIT')
|
||||
} catch (error) {
|
||||
this.db.exec('ROLLBACK')
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Make a crash repair durable in ONE transaction: DELETE the torn tail (from
|
||||
* `tornMarker`) and INSERT the synthetic `closers`. After COMMIT the stored rows
|
||||
* == the balanced log.
|
||||
*/
|
||||
async commitRepair(meta: SessionHeader, tornMarker: number | undefined, closers: readonly SessionEvent[]): Promise<void> {
|
||||
await this.ready
|
||||
this.db.exec('BEGIN')
|
||||
try {
|
||||
if (tornMarker !== undefined) {
|
||||
this.db.prepare('DELETE FROM events WHERE session_id = ? AND seq >= ?').run(meta.id, tornMarker)
|
||||
}
|
||||
if (closers.length > 0) {
|
||||
const insertEvent = this.db.prepare(
|
||||
'INSERT INTO events (session_id, seq, type, time, data, source_event_seqs, surface_op) VALUES (?, ?, ?, ?, ?, ?, ?)',
|
||||
)
|
||||
for (const event of closers) {
|
||||
const [surfaceSeqs, surfaceOp] = surfaceBindings(event)
|
||||
insertEvent.run(meta.id, event.seq, event.type, event.time, JSON.stringify(event.data), surfaceSeqs, surfaceOp)
|
||||
}
|
||||
}
|
||||
if (tornMarker !== undefined || closers.length > 0) {
|
||||
this.db.prepare('UPDATE sessions SET revision = revision + 1 WHERE id = ?').run(meta.id)
|
||||
}
|
||||
this.db.exec('COMMIT')
|
||||
} catch (error) {
|
||||
// The DELETE+INSERT cannot collide (a row at a closer's seq is preserved or
|
||||
// deleted as torn first); this rolls back a DB-level failure (disk full,
|
||||
// etc.), unreachable in test.
|
||||
/* v8 ignore start */
|
||||
this.db.exec('ROLLBACK')
|
||||
throw error
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
}
|
||||
|
||||
/** List all materialized sessions' metadata (every row is a materialized session). */
|
||||
async list(signal?: AbortSignal): Promise<SessionHeader[]> {
|
||||
signal?.throwIfAborted()
|
||||
await this.ready
|
||||
signal?.throwIfAborted()
|
||||
const rows = this.db
|
||||
.prepare('SELECT * FROM sessions')
|
||||
.all() as unknown as SessionRow[]
|
||||
signal?.throwIfAborted()
|
||||
return rows.map(rowToMeta)
|
||||
}
|
||||
|
||||
/** List metadata with a source-qualified monotonic revision per session. */
|
||||
async listSnapshots(signal?: AbortSignal): Promise<SessionPersistenceSnapshot[]> {
|
||||
signal?.throwIfAborted()
|
||||
await this.ready
|
||||
signal?.throwIfAborted()
|
||||
const rows = this.db.prepare('SELECT * FROM sessions').all() as unknown as SessionRow[]
|
||||
signal?.throwIfAborted()
|
||||
return rows.map(row => ({
|
||||
header: rowToMeta(row),
|
||||
revision: SessionPersistenceRevision(
|
||||
`${this.storeIdentity}:incarnation:${row.incarnation}:revision:${row.revision}`,
|
||||
),
|
||||
}))
|
||||
}
|
||||
|
||||
/** Close the database handle (awaited by the coordinator's dispose, post-drain). */
|
||||
async close(): Promise<void> {
|
||||
await this.ready
|
||||
this.db.close()
|
||||
}
|
||||
|
||||
// --- row helpers ---
|
||||
|
||||
/** Fetch a session's row, or undefined if absent. */
|
||||
private rowFor(id: SessionId): SessionRow | undefined {
|
||||
return this.db.prepare('SELECT * FROM sessions WHERE id = ?').get(id) as unknown as SessionRow | undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Insert-or-replace a session's metadata row. The only caller is the first
|
||||
* materializing `appendBatch`, so writing the row IS the materialization (its
|
||||
* existence is the signal `list` reads).
|
||||
*/
|
||||
private writeRow(meta: SessionHeader): void {
|
||||
this.db.prepare(`
|
||||
INSERT INTO sessions
|
||||
(id, version, created_at, cwd, parent_session, seed_length, origin, delegation_depth, incarnation, revision)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, 0)
|
||||
ON CONFLICT(id) DO UPDATE SET
|
||||
version = excluded.version,
|
||||
created_at = excluded.created_at,
|
||||
cwd = excluded.cwd,
|
||||
parent_session = excluded.parent_session,
|
||||
seed_length = excluded.seed_length,
|
||||
origin = excluded.origin,
|
||||
delegation_depth = excluded.delegation_depth
|
||||
`).run(
|
||||
meta.id,
|
||||
meta.version,
|
||||
meta.createdAt,
|
||||
meta.cwd ?? null,
|
||||
meta.parentSession ?? null,
|
||||
meta.seedLength ?? null,
|
||||
meta.origin ?? null,
|
||||
meta.delegationDepth ?? null,
|
||||
randomUUID(),
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
export default SessionPersistenceSqlite
|
||||
30
packages/session/session-persistence-sqlite/src/invariant.ts
Normal file
30
packages/session/session-persistence-sqlite/src/invariant.ts
Normal file
@@ -0,0 +1,30 @@
|
||||
/**
|
||||
* Package-owned invariant companion for `@deepseek-ai/dsh-session-persistence-sqlite`.
|
||||
* @module @deepseek-ai/dsh-session-persistence-sqlite/invariant
|
||||
*/
|
||||
|
||||
/* jscpd:ignore-start */
|
||||
import type { Context } from 'cordis'
|
||||
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
|
||||
|
||||
const PACKAGE_NAME = '@deepseek-ai/dsh-session-persistence-sqlite'
|
||||
|
||||
/** Cordis companion plugin name. */
|
||||
export const name = 'session-persistence-sqlite-invariant'
|
||||
/** Service required before the companion can reserve package ownership. */
|
||||
export const inject = ['invariants']
|
||||
|
||||
/**
|
||||
* No runtime invariant: persistence correctness requires backend round-trip and crash-tail tests;
|
||||
* this package exposes no continuously observable in-process relation.
|
||||
*/
|
||||
const install: InvariantInstaller = () => {}
|
||||
|
||||
/**
|
||||
* Register this package's invariant companion.
|
||||
* @param ctx - Cordis context carrying the invariant service.
|
||||
* @returns the installed registration's disposer after setup succeeds.
|
||||
*/
|
||||
export const apply = (ctx: Context): Promise<() => void> =>
|
||||
Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
|
||||
/* jscpd:ignore-end */
|
||||
262
packages/session/session-persistence-sqlite/src/schema.ts
Normal file
262
packages/session/session-persistence-sqlite/src/schema.ts
Normal file
@@ -0,0 +1,262 @@
|
||||
/**
|
||||
* Schema + load-time helpers for the SQLite session-persistence backend: the
|
||||
* DDL (a store-identity row, `sessions` metadata, and a 1:1 `events` row per
|
||||
* `SessionEvent`), the database open/configure step, and the last-`turn/end`
|
||||
* cut that gives the SQLite backend the SAME crash-tail-on-load semantics as
|
||||
* the JSONL backend.
|
||||
*
|
||||
* @module dsh-session-persistence-sqlite/schema
|
||||
*/
|
||||
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import { DatabaseSync } from 'node:sqlite'
|
||||
import type { SessionEvent, SessionId, SessionHeader, SurfaceOp } from '@deepseek-ai/dsh-session'
|
||||
|
||||
/**
|
||||
* The on-disk schema version. Bumped only on a breaking change to the table
|
||||
* layout; orthogonal to a session's own `version` (which versions the EVENT
|
||||
* vocabulary, stored per session in the `sessions` row).
|
||||
*/
|
||||
export const SCHEMA_VERSION = 13
|
||||
|
||||
/** SQLite application id protecting unrelated databases from persistence writes. */
|
||||
export const SESSION_PERSISTENCE_SQLITE_APPLICATION_ID = 0x44534850
|
||||
|
||||
/**
|
||||
* A row of the `sessions` table — the out-of-log metadata ({@link SessionHeader}).
|
||||
* The row's EXISTENCE is the materialization signal: it is written only by the
|
||||
* first `append` (lazy materialization), so a created-but-never-appended
|
||||
* session has no row and is absent from `list`, mirroring the JSONL
|
||||
* backend's "no file until first append".
|
||||
*/
|
||||
export interface SessionRow {
|
||||
id: string
|
||||
version: number
|
||||
created_at: number
|
||||
cwd: string | null
|
||||
parent_session: string | null
|
||||
seed_length: number | null
|
||||
origin: 'subagent' | null
|
||||
/** Stable identity assigned when this log is materialized. */
|
||||
incarnation: string
|
||||
/** Monotonic log-change token incremented in each mutating transaction. */
|
||||
revision: number
|
||||
delegation_depth: number | null
|
||||
}
|
||||
|
||||
/** An `events` table row: one `SessionEvent` mapped 1:1 (`data` is JSON text). */
|
||||
export interface EventRow {
|
||||
seq: number
|
||||
type: string
|
||||
time: number
|
||||
data: string
|
||||
/** JSON-encoded `number[]` — the event's sourceEventSeqs, or null. */
|
||||
source_event_seqs: string | null
|
||||
/** JSON-encoded `SurfaceOp` — how the event entered the surface, or null. */
|
||||
surface_op: string | null
|
||||
}
|
||||
|
||||
/**
|
||||
* Journal modes the backend will run under. `wal` is the default and the
|
||||
* durability model the persistence ADR records; the rollback-journal modes
|
||||
* (`delete`/`truncate`/`persist`) exist for filesystems where WAL's
|
||||
* shared-memory files do not work (network mounts). `memory`/`off` are
|
||||
* excluded: dropping journal durability silently contradicts what this
|
||||
* backend promises.
|
||||
*/
|
||||
export type JournalMode = 'wal' | 'delete' | 'truncate' | 'persist'
|
||||
|
||||
/**
|
||||
* Open the database and apply its schema and pragmas. An empty database with a
|
||||
* zero `user_version` is initialized at {@link SCHEMA_VERSION}; a nonempty
|
||||
* unversioned database and every other non-current version reject rather than
|
||||
* being migrated in place.
|
||||
* @param path - the SQLite database file to open (created when absent).
|
||||
* @param journalMode - validated journal pragma.
|
||||
* @returns the open handle with pragmas applied and all three tables ensured.
|
||||
*/
|
||||
export function openDatabase(path: string, journalMode: JournalMode): DatabaseSync {
|
||||
const db = new DatabaseSync(path)
|
||||
try {
|
||||
configureDatabase(db, path, journalMode)
|
||||
return db
|
||||
} catch (error: unknown) {
|
||||
db.close()
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
function configureDatabase(db: DatabaseSync, path: string, journalMode: JournalMode): void {
|
||||
db.exec('PRAGMA foreign_keys = ON')
|
||||
let began = false
|
||||
try {
|
||||
db.exec('BEGIN IMMEDIATE')
|
||||
began = true
|
||||
// Validate while holding the write lock so no other connection can change
|
||||
// schema ownership between inspection and initialization.
|
||||
const { user_version: onDisk } = db.prepare('PRAGMA user_version').get() as { user_version: number }
|
||||
const { application_id: applicationId } = db.prepare('PRAGMA application_id').get() as { application_id: number }
|
||||
const { count: userObjectCount } = db.prepare(
|
||||
"SELECT COUNT(*) AS count FROM sqlite_schema WHERE name NOT GLOB 'sqlite_*'",
|
||||
).get() as { count: number }
|
||||
if (onDisk === 0 && (applicationId !== 0 || userObjectCount > 0)) {
|
||||
throw new Error(`session database at "${path}" has an unversioned schema or application identity`)
|
||||
}
|
||||
if (onDisk !== 0 && onDisk !== SCHEMA_VERSION) {
|
||||
throw new Error(`session database at "${path}" has schema version ${onDisk}, incompatible with this build (${SCHEMA_VERSION})`)
|
||||
}
|
||||
if (onDisk === SCHEMA_VERSION && applicationId !== SESSION_PERSISTENCE_SQLITE_APPLICATION_ID) {
|
||||
throw new Error(
|
||||
`session database at "${path}" has application id ${applicationId}, expected ${SESSION_PERSISTENCE_SQLITE_APPLICATION_ID}`,
|
||||
)
|
||||
}
|
||||
db.exec(`
|
||||
CREATE TABLE IF NOT EXISTS persistence_state (
|
||||
singleton INTEGER PRIMARY KEY CHECK (singleton = 1),
|
||||
store_id TEXT NOT NULL
|
||||
) STRICT;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS sessions (
|
||||
id TEXT PRIMARY KEY,
|
||||
version INTEGER NOT NULL,
|
||||
created_at INTEGER NOT NULL,
|
||||
cwd TEXT,
|
||||
parent_session TEXT,
|
||||
seed_length INTEGER,
|
||||
origin TEXT,
|
||||
delegation_depth INTEGER,
|
||||
incarnation TEXT NOT NULL,
|
||||
revision INTEGER NOT NULL
|
||||
) STRICT;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS events (
|
||||
session_id TEXT NOT NULL REFERENCES sessions(id) ON DELETE CASCADE,
|
||||
seq INTEGER NOT NULL,
|
||||
type TEXT NOT NULL,
|
||||
time INTEGER NOT NULL,
|
||||
data TEXT NOT NULL,
|
||||
source_event_seqs TEXT,
|
||||
surface_op TEXT,
|
||||
PRIMARY KEY (session_id, seq)
|
||||
) STRICT
|
||||
`)
|
||||
db.prepare(
|
||||
'INSERT OR IGNORE INTO persistence_state (singleton, store_id) VALUES (1, ?)',
|
||||
).run(randomUUID())
|
||||
if (onDisk === 0) {
|
||||
db.exec(`PRAGMA application_id = ${SESSION_PERSISTENCE_SQLITE_APPLICATION_ID}`)
|
||||
db.exec(`PRAGMA user_version = ${SCHEMA_VERSION}`)
|
||||
}
|
||||
db.exec('COMMIT')
|
||||
began = false
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore next -- a BEGIN failure leaves no transaction to roll back. */
|
||||
if (began) {
|
||||
/* v8 ignore next 5 -- preserve the original schema failure if SQLite also refuses rollback. */
|
||||
try {
|
||||
db.exec('ROLLBACK')
|
||||
} catch {
|
||||
// The original SQLite failure remains the actionable cause.
|
||||
}
|
||||
}
|
||||
throw error
|
||||
}
|
||||
// The validated union is safe to interpolate into a non-bindable PRAGMA.
|
||||
// Apply it only after ownership validation and initialization commit.
|
||||
db.exec(`PRAGMA journal_mode = ${journalMode.toUpperCase()}`)
|
||||
}
|
||||
|
||||
/**
|
||||
* Reconstruct the {@link SessionHeader} from a `sessions` row.
|
||||
* @param row - the `sessions` table row.
|
||||
* @returns the header, `NULL` columns mapped to omitted optional fields.
|
||||
*/
|
||||
export function rowToMeta(row: SessionRow): SessionHeader {
|
||||
if (!Number.isSafeInteger(row.created_at) || row.created_at < 0) {
|
||||
throw new Error('stored session createdAt must be a non-negative safe integer')
|
||||
}
|
||||
return {
|
||||
version: row.version,
|
||||
id: row.id as SessionId,
|
||||
createdAt: row.created_at,
|
||||
...row.cwd !== null ? { cwd: row.cwd } : {},
|
||||
...row.parent_session !== null ? { parentSession: row.parent_session as SessionId } : {},
|
||||
...row.seed_length !== null ? { seedLength: row.seed_length } : {},
|
||||
...row.origin !== null ? { origin: row.origin } : {},
|
||||
...row.delegation_depth !== null ? { delegationDepth: row.delegation_depth } : {},
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reconstruct a {@link SessionEvent} from an `events` row (parses `data`).
|
||||
* @param row - the `events` table row; `data` and the surface columns hold JSON text.
|
||||
* @returns the reconstructed event; throws when a JSON column fails to parse
|
||||
* ({@link scanRows} treats that as a hole, not corruption, in the tail).
|
||||
*/
|
||||
export function rowToEvent(row: EventRow): SessionEvent {
|
||||
// Surface-metadata fields are conditional on the event type in the type
|
||||
// system; spread them so each variant gets only the fields it declares.
|
||||
const surfaceFields = {
|
||||
...row.source_event_seqs !== null ? { sourceEventSeqs: JSON.parse(row.source_event_seqs) as number[] } : {},
|
||||
...row.surface_op !== null ? { surfaceOp: JSON.parse(row.surface_op) as SurfaceOp } : {},
|
||||
}
|
||||
return {
|
||||
type: row.type as SessionEvent['type'],
|
||||
seq: row.seq,
|
||||
time: row.time,
|
||||
data: JSON.parse(row.data) as SessionEvent['data'],
|
||||
...surfaceFields,
|
||||
} as SessionEvent
|
||||
}
|
||||
|
||||
/**
|
||||
* Find the preserved prefix of ordered event rows. Fully written rows in an
|
||||
* interrupted final turn remain in the prefix. The first unparsable row or seq
|
||||
* gap after the last `turn/end` marks a tolerated torn tail; the same hole in
|
||||
* the committed region rejects.
|
||||
*
|
||||
* @param rows - one session's event rows, ordered by seq ascending.
|
||||
* @param base - the seq the first row is expected to carry; `0` for a whole
|
||||
* log, the requested `fromSeq` for a suffix read (`loadStoredFrom`).
|
||||
* @returns the preserved event prefix, plus `tornFrom` — the seq the physical
|
||||
* delete starts at — when a torn tail exists.
|
||||
*/
|
||||
export function scanRows(rows: readonly EventRow[], base = 0): { preserved: SessionEvent[]; tornFrom?: number } {
|
||||
// Pass 1: parse each row's data; a row whose data is not valid JSON is a hole.
|
||||
// (The seq/type COLUMNS are always present even when `data` is corrupt.)
|
||||
interface Parsed { ok: boolean; event?: SessionEvent }
|
||||
const parsed: Parsed[] = rows.map((row) => {
|
||||
try {
|
||||
return { ok: true, event: rowToEvent(row) }
|
||||
} catch {
|
||||
return { ok: false }
|
||||
}
|
||||
})
|
||||
|
||||
// The last index that is a valid `turn/end` — holes through a closed turn
|
||||
// are always committed corruption.
|
||||
let lastTurnEnd = -1
|
||||
for (let i = parsed.length - 1; i >= 0; i--) {
|
||||
if (parsed[i]?.ok && rows[i]?.type === 'turn/end') { lastTurnEnd = i; break }
|
||||
}
|
||||
|
||||
// Preserve the contiguous prefix, including a complete interrupted turn;
|
||||
// holes through the last committed boundary throw, while later holes stop.
|
||||
const preserved: SessionEvent[] = []
|
||||
for (let i = 0; i < rows.length; i++) {
|
||||
const p = parsed[i]
|
||||
if (!p?.ok || p.event === undefined) {
|
||||
if (i <= lastTurnEnd) throw new Error(`corrupt session log: unparsable committed event at seq ${rows[i]?.seq}`)
|
||||
break // torn tail fragment after the last turn/end — stop, tolerate
|
||||
}
|
||||
if (p.event.seq !== base + i) {
|
||||
if (i <= lastTurnEnd) throw new Error(`corrupt session log: seq gap in committed region (expected ${base + i}, got ${p.event.seq})`)
|
||||
break // gap after the last turn/end — torn tail, stop
|
||||
}
|
||||
preserved.push(p.event)
|
||||
}
|
||||
|
||||
// Any rows past the preserved prefix are a never-committed torn tail; their
|
||||
// first seq is the deletion point for load's physical repair.
|
||||
return preserved.length < rows.length ? { preserved, tornFrom: base + preserved.length } : { preserved }
|
||||
}
|
||||
926
packages/session/session-persistence-sqlite/tests/sqlite.spec.ts
Normal file
926
packages/session/session-persistence-sqlite/tests/sqlite.spec.ts
Normal file
@@ -0,0 +1,926 @@
|
||||
import { createUserMessage, createMessage } from '@deepseek-ai/dsh-llm'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import { existsSync } from 'node:fs'
|
||||
import { chmod, mkdtemp, rm, stat, symlink, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { dirname, join } from 'node:path'
|
||||
import { DatabaseSync } from 'node:sqlite'
|
||||
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type { Session, SessionEvent, SurfaceEvent, SurfaceEventType } from '@deepseek-ai/dsh-session'
|
||||
import SessionPersistenceSqlite, { SCHEMA_VERSION } from '@deepseek-ai/dsh-session-persistence-sqlite'
|
||||
import {
|
||||
openDatabase,
|
||||
rowToEvent,
|
||||
rowToMeta,
|
||||
scanRows,
|
||||
SESSION_PERSISTENCE_SQLITE_APPLICATION_ID,
|
||||
type EventRow,
|
||||
} from '../src/schema.ts'
|
||||
import { runPersistenceContract, meta, oneTurnLog, appendLog } from '../../session-persistence/tests/contract.ts'
|
||||
import { runCoordinatorContract, type CoordinatorFixture } from '../../session-persistence/tests/coordinator-contract.ts'
|
||||
|
||||
const dirs: string[] = []
|
||||
afterEach(async () => { for (const d of dirs.splice(0)) await rm(d, { recursive: true, force: true }) })
|
||||
|
||||
async function expectFlushError(promise: Promise<unknown>, message: RegExp): Promise<void> {
|
||||
try {
|
||||
await promise
|
||||
} catch (error) {
|
||||
expect(error).toBeInstanceOf(Error)
|
||||
expect((error as Error).message).toMatch(message)
|
||||
return
|
||||
}
|
||||
throw new Error('expected flush to reject')
|
||||
}
|
||||
|
||||
async function freshDbPath(): Promise<string> {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'dsh-sqlite-'))
|
||||
dirs.push(dir)
|
||||
return join(dir, 'sessions.db')
|
||||
}
|
||||
|
||||
/** A context with the session store + SQLite backend, plus a teardown. */
|
||||
async function backend(path = ':memory:'): Promise<{ ctx: Context; dispose: () => Promise<void> }> {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(SessionPersistenceSqlite, { path })
|
||||
return { ctx, dispose: () => fiber.dispose() }
|
||||
}
|
||||
|
||||
// Run the same backend-agnostic contract as JSONL to pin identical semantics.
|
||||
runPersistenceContract('sqlite', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(SessionPersistenceSqlite, { path: ':memory:' })
|
||||
return {
|
||||
persistence: ctx.sessionPersistence,
|
||||
dispose: async () => { await fiber.dispose() },
|
||||
}
|
||||
})
|
||||
|
||||
// A file-backed database lets two mounts share rows across reload. `corruptTail` inserts invalid
|
||||
// JSON past the committed seq, exercising coordinator repair against real database rows.
|
||||
runCoordinatorContract('sqlite', async (): Promise<CoordinatorFixture> => {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'dsh-sqlite-coord-'))
|
||||
const path = join(dir, 'sessions.db')
|
||||
return {
|
||||
mount: async ctx => ctx.plugin(SessionPersistenceSqlite, { path }),
|
||||
corruptTail: async (id) => {
|
||||
// A row past the committed region whose `data` does not parse: scanRows
|
||||
// bounds the preserved prefix at it and returns its seq as tornFrom, which
|
||||
// the backend surfaces to the coordinator as the tornMarker to delete from.
|
||||
const db = openDatabase(path, 'wal')
|
||||
const next = (db.prepare('SELECT COALESCE(MAX(seq), -1) + 1 AS n FROM events WHERE session_id = ?')
|
||||
.get(id) as { n: number }).n
|
||||
db.prepare('INSERT INTO events (session_id, seq, type, time, data) VALUES (?, ?, ?, ?, ?)')
|
||||
.run(id, next, 'assistant/chunk', 99, '{not valid json')
|
||||
db.close()
|
||||
},
|
||||
cleanup: async () => { await rm(dir, { recursive: true, force: true }) },
|
||||
}
|
||||
})
|
||||
|
||||
describe('scanRows', () => {
|
||||
// scanRows works off EventRows (data is a JSON string column); build them from SessionEvents
|
||||
// so the unit tests read in terms of the event vocabulary. Surface metadata is serialized to
|
||||
// its nullable columns so the conversion remains faithful.
|
||||
const rows = (events: SessionEvent[]): EventRow[] =>
|
||||
events.map((e) => {
|
||||
const se = e as SessionEvent<SurfaceEventType>
|
||||
return {
|
||||
seq: e.seq, type: e.type, time: e.time, data: JSON.stringify(e.data),
|
||||
source_event_seqs: se.sourceEventSeqs !== undefined ? JSON.stringify(se.sourceEventSeqs) : null,
|
||||
surface_op: se.surfaceOp !== undefined ? JSON.stringify(se.surfaceOp) : null,
|
||||
}
|
||||
})
|
||||
|
||||
it('preserves the full log when it ends exactly on a turn/end (no torn tail)', () => {
|
||||
const { preserved, tornFrom } = scanRows(rows(oneTurnLog()))
|
||||
expect(preserved).toEqual(oneTurnLog())
|
||||
expect(tornFrom).toBeUndefined()
|
||||
})
|
||||
|
||||
it('PRESERVES the real events of an interrupted turn after the last turn/end', () => {
|
||||
// turn 1 committed (0..5) + a crashed turn 2 (turn/start 6, step/start 7, no
|
||||
// close): all 8 rows are intact, so the whole prefix is preserved and there
|
||||
// is no torn fragment to delete. (load() then synthesizes the closers.)
|
||||
const withOpenTurn: SessionEvent[] = [
|
||||
...oneTurnLog(),
|
||||
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2 } },
|
||||
{ type: 'step/start', seq: 7, time: 8, data: { turn: 2, step: 1 } },
|
||||
]
|
||||
const { preserved, tornFrom } = scanRows(rows(withOpenTurn))
|
||||
expect(preserved.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
expect(tornFrom).toBeUndefined()
|
||||
})
|
||||
|
||||
it('preserves the contiguous prefix and flags a torn tail at a seq gap', () => {
|
||||
// A gap after seq 0 (no committed turn/end): seq 0 is the preserved
|
||||
// interrupted-turn event; the gap bounds it and marks the torn fragment.
|
||||
const gapped: SessionEvent[] = [
|
||||
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } },
|
||||
{ type: 'step/start', seq: 2, time: 2, data: { turn: 1, step: 1 } }, // seq 1 missing
|
||||
]
|
||||
const { preserved, tornFrom } = scanRows(rows(gapped))
|
||||
expect(preserved.map(e => e.seq)).toEqual([0])
|
||||
expect(tornFrom).toBe(1)
|
||||
})
|
||||
|
||||
it('an empty log preserves nothing and has no torn tail', () => {
|
||||
expect(scanRows([])).toEqual({ preserved: [] })
|
||||
})
|
||||
|
||||
it('throws on a seq gap inside the committed region (before the last turn/end)', () => {
|
||||
const gapped: SessionEvent[] = [
|
||||
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } },
|
||||
{ type: 'step/start', seq: 2, time: 2, data: { turn: 1, step: 1 } }, // seq 1 missing
|
||||
{ type: 'turn/end', seq: 3, time: 3, data: { turn: 1, reason: { kind: 'completed' } } },
|
||||
]
|
||||
expect(() => scanRows(rows(gapped))).toThrow(/seq gap in committed region/)
|
||||
})
|
||||
|
||||
it('throws on an unparsable row inside the committed region', () => {
|
||||
const withCorruptCommitted: EventRow[] = [
|
||||
{ seq: 0, type: 'turn/start', time: 1, data: '{not json', source_event_seqs: null, surface_op: null }, // corrupt, sits before a turn/end
|
||||
{ seq: 1, type: 'turn/end', time: 2, data: JSON.stringify({ turn: 1, reason: { kind: 'completed' } }), source_event_seqs: null, surface_op: null },
|
||||
]
|
||||
expect(() => scanRows(withCorruptCommitted)).toThrow(/unparsable committed event/)
|
||||
})
|
||||
|
||||
it('tolerates an unparsable torn-tail row after the last turn/end', () => {
|
||||
const withCorruptTail: EventRow[] = [
|
||||
...rows(oneTurnLog()),
|
||||
{ seq: 6, type: 'turn/start', time: 7, data: '{not json', source_event_seqs: null, surface_op: null }, // torn fragment, no committed turn/end after
|
||||
]
|
||||
const { preserved, tornFrom } = scanRows(withCorruptTail)
|
||||
expect(preserved).toEqual(oneTurnLog())
|
||||
expect(tornFrom).toBe(6)
|
||||
})
|
||||
})
|
||||
|
||||
describe('rowToMeta', () => {
|
||||
it('restores optional origin metadata', () => {
|
||||
expect(rowToMeta({
|
||||
id: 'with-origin',
|
||||
version: 0,
|
||||
created_at: 1,
|
||||
cwd: null,
|
||||
parent_session: null,
|
||||
seed_length: null,
|
||||
origin: 'subagent',
|
||||
incarnation: 'with-origin',
|
||||
revision: 1,
|
||||
delegation_depth: null,
|
||||
})).toMatchObject({ id: 'with-origin', origin: 'subagent' })
|
||||
})
|
||||
|
||||
it('rejects fractional stored creation metadata', () => {
|
||||
expect(() => rowToMeta({
|
||||
id: 'fractional',
|
||||
version: 0,
|
||||
created_at: 1.5,
|
||||
cwd: null,
|
||||
parent_session: null,
|
||||
seed_length: null,
|
||||
origin: null,
|
||||
incarnation: 'fractional',
|
||||
revision: 1,
|
||||
delegation_depth: null,
|
||||
})).toThrow('stored session createdAt must be a non-negative safe integer')
|
||||
})
|
||||
})
|
||||
|
||||
describe('SessionPersistenceSqlite: durability and crash semantics', () => {
|
||||
it('rejects a stored v0 log containing a legacy request/header-delta event', async () => {
|
||||
const path = await freshDbPath()
|
||||
const m = meta('legacy-header-delta', '/legacy')
|
||||
const db = openDatabase(path, 'wal')
|
||||
db.prepare('INSERT INTO sessions (id, version, created_at, cwd, parent_session, seed_length, delegation_depth, incarnation, revision) VALUES (?, ?, ?, ?, NULL, NULL, NULL, ?, 1)')
|
||||
.run(m.id, m.version, m.createdAt, m.cwd ?? null, 'legacy-header-delta')
|
||||
const insert = db.prepare('INSERT INTO events (session_id, seq, type, time, data) VALUES (?, ?, ?, ?, ?)')
|
||||
insert.run(m.id, 0, 'turn/start', 1, JSON.stringify({ turn: 1 }))
|
||||
insert.run(m.id, 1, 'request/header-delta', 2, JSON.stringify({ config: { model: 'legacy' } }))
|
||||
insert.run(m.id, 2, 'turn/end', 3, JSON.stringify({ turn: 1, reason: { kind: 'completed' } }))
|
||||
db.close()
|
||||
|
||||
const mounted = await backend(path)
|
||||
await expect(mounted.ctx.sessionPersistence.load(m.id)).rejects.toThrow(/unsupported legacy request\/header-delta event at seq 1/)
|
||||
await mounted.dispose()
|
||||
})
|
||||
|
||||
it('rejects a stored v0 full header carrying the legacy fallback reason', async () => {
|
||||
const path = await freshDbPath()
|
||||
const m = meta('legacy-header-fallback', '/legacy')
|
||||
const db = openDatabase(path, 'wal')
|
||||
db.prepare('INSERT INTO sessions (id, version, created_at, cwd, parent_session, seed_length, delegation_depth, incarnation, revision) VALUES (?, ?, ?, ?, NULL, NULL, NULL, ?, 1)')
|
||||
.run(m.id, m.version, m.createdAt, m.cwd ?? null, 'legacy-header-fallback')
|
||||
db.prepare('INSERT INTO events (session_id, seq, type, time, data) VALUES (?, ?, ?, ?, ?)')
|
||||
.run(m.id, 0, 'request/header', 1, JSON.stringify({
|
||||
header: { config: { model: 'legacy' } },
|
||||
reason: 'fallback',
|
||||
}))
|
||||
db.close()
|
||||
|
||||
const mounted = await backend(path)
|
||||
await expect(mounted.ctx.sessionPersistence.load(m.id))
|
||||
.rejects.toThrow(/unsupported legacy request\/header reason "fallback" at seq 0/)
|
||||
await mounted.dispose()
|
||||
})
|
||||
|
||||
it('has no independent per-session log location', async () => {
|
||||
const { ctx, dispose } = await backend()
|
||||
expect(ctx.sessionPersistence.locate(meta('sqlite-location'))).toBeUndefined()
|
||||
await dispose()
|
||||
})
|
||||
|
||||
it('an interrupted turn (rows after the last turn/end) is PRESERVED and closed during load', async () => {
|
||||
const path = await freshDbPath()
|
||||
const m = meta('crash')
|
||||
// Run 1: persist a complete turn, then a half-written second turn (no turn/end).
|
||||
const ctx1 = new Context()
|
||||
await ctx1.plugin(SessionStore)
|
||||
const fiber1 = await ctx1.plugin(SessionPersistenceSqlite, { path })
|
||||
await ctx1.sessionPersistence.create(m)
|
||||
await ctx1.sessionPersistence.append(m.id, oneTurnLog())
|
||||
await ctx1.sessionPersistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2 } },
|
||||
{ type: 'step/start', seq: 7, time: 8, data: { turn: 2, step: 1 } },
|
||||
])
|
||||
await fiber1.dispose()
|
||||
|
||||
// Run 2: load PRESERVES the interrupted turn's real events (a turn can be huge
|
||||
// — never truncated) and closes the orphaned turn with synthetic boundary
|
||||
// events: step/end (the step was open) then turn/end {interrupted}.
|
||||
const ctx2 = new Context()
|
||||
await ctx2.plugin(SessionStore)
|
||||
const fiber2 = await ctx2.plugin(SessionPersistenceSqlite, { path })
|
||||
const loaded = await ctx2.sessionPersistence.load(m.id)
|
||||
expect(loaded.events.map(e => e.type)).toEqual([
|
||||
'turn/start', 'user/message', 'step/start', 'assistant/message', 'step/end', 'turn/end', // turn 1
|
||||
'turn/start', 'step/start', 'step/end', 'turn/end', // turn 2: real events + synthetic closers
|
||||
])
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
|
||||
const last = loaded.events.at(-1)!
|
||||
expect(last.type === 'turn/end' && last.data.reason).toEqual({ kind: 'interrupted' })
|
||||
|
||||
// load durably closed the turn, so the next append continues at the balanced
|
||||
// length (seq 10) and a reload round-trips identically.
|
||||
await ctx2.sessionPersistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 10, time: 9, data: { turn: 3 } },
|
||||
{ type: 'turn/end', seq: 11, time: 10, data: { turn: 3, reason: { kind: 'completed' } } },
|
||||
])
|
||||
const reloaded = await ctx2.sessionPersistence.load(m.id)
|
||||
expect(reloaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
|
||||
await fiber2.dispose()
|
||||
})
|
||||
|
||||
it('load() durably closes the interrupted turn: the synthetic closers are on disk after load', async () => {
|
||||
const path = await freshDbPath()
|
||||
const m = meta('load-closes')
|
||||
const b1 = await backend(path)
|
||||
await b1.ctx.sessionPersistence.create(m)
|
||||
await b1.ctx.sessionPersistence.append(m.id, oneTurnLog()) // seqs 0..5
|
||||
await b1.dispose()
|
||||
// Hand-write an interrupted turn (turn/start seq 6, no turn/end).
|
||||
const db = openDatabase(path, 'wal')
|
||||
db.prepare('INSERT INTO events (session_id, seq, type, time, data) VALUES (?, 6, ?, 7, ?)')
|
||||
.run(m.id, 'turn/start', JSON.stringify({ turn: 2 }))
|
||||
db.close()
|
||||
|
||||
const b2 = await backend(path)
|
||||
const loaded = await b2.ctx.sessionPersistence.load(m.id)
|
||||
// turn 2's real turn/start (seq 6) is preserved + a synthetic turn/end (seq 7).
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
expect(loaded.events.at(-1)!.type).toBe('turn/end')
|
||||
// load() is mutating: the synthetic turn/end MUST be on disk so the stored log
|
||||
// is balanced and the cursor is truthful (contract: load closes, not defers).
|
||||
const probe = openDatabase(path, 'wal')
|
||||
const stored = probe.prepare('SELECT seq, type FROM events WHERE session_id = ? ORDER BY seq').all(m.id) as { seq: number; type: string }[]
|
||||
probe.close()
|
||||
expect(stored.map(r => r.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
expect(stored.at(-1)!.type).toBe('turn/end')
|
||||
await b2.dispose()
|
||||
})
|
||||
|
||||
it('all-tail load: a session whose only turn never closed is preserved and closed on load', async () => {
|
||||
const path = await freshDbPath()
|
||||
const m = meta('all-tail')
|
||||
const b1 = await backend(path)
|
||||
await b1.ctx.sessionPersistence.create(m)
|
||||
// A first turn that NEVER completed: turn/start + user/message, no turn/end.
|
||||
await b1.ctx.sessionPersistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } },
|
||||
{ type: 'user/message', seq: 1, time: 2, data: createUserMessage({
|
||||
content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' },
|
||||
}), surfaceOp: 'append' },
|
||||
])
|
||||
await b1.dispose()
|
||||
|
||||
// A fresh backend loads it: the interrupted (only) turn's real events are
|
||||
// preserved and closed with a synthetic turn/end {interrupted} — NOT
|
||||
// truncated. The session was materialized, so list() reports it present.
|
||||
const b2 = await backend(path)
|
||||
const loaded = await b2.ctx.sessionPersistence.load(m.id)
|
||||
expect(loaded.events.map(e => e.type)).toEqual(['turn/start', 'user/message', 'turn/end'])
|
||||
expect(loaded.events.at(-1)!.type === 'turn/end' && loaded.events.at(-1)!.data).toMatchObject({ reason: { kind: 'interrupted' } })
|
||||
expect((await b2.ctx.sessionPersistence.list()).map(x => x.id)).toContain(m.id)
|
||||
await b2.dispose()
|
||||
})
|
||||
|
||||
it('rejects opening a database whose schema version is not the current build (newer OR older)', async () => {
|
||||
const path = await freshDbPath()
|
||||
openDatabase(path, 'wal').close() // stamp user_version = SCHEMA_VERSION
|
||||
// Bump user_version past what this build supports.
|
||||
const dbNewer = openDatabase(path, 'wal')
|
||||
dbNewer.exec(`PRAGMA user_version = ${SCHEMA_VERSION + 1}`)
|
||||
dbNewer.close()
|
||||
expect(() => openDatabase(path, 'wal')).toThrow(/incompatible with this build/)
|
||||
|
||||
// The immediately preceding layout lacks the required store identity and is
|
||||
// rejected rather than migrated (unreleased software, no backward-compat).
|
||||
const olderPath = await freshDbPath()
|
||||
openDatabase(olderPath, 'wal').close()
|
||||
const dbOlder = openDatabase(olderPath, 'wal')
|
||||
dbOlder.exec(`PRAGMA user_version = ${SCHEMA_VERSION - 1}`)
|
||||
dbOlder.close()
|
||||
expect(() => openDatabase(olderPath, 'wal')).toThrow(/incompatible with this build/)
|
||||
})
|
||||
|
||||
it('rejects a table-backed unversioned database before stamping or changing journal mode', async () => {
|
||||
const path = await freshDbPath()
|
||||
const legacy = new DatabaseSync(path)
|
||||
legacy.exec('CREATE TABLE sessions (id TEXT PRIMARY KEY)')
|
||||
legacy.close()
|
||||
|
||||
expect(() => openDatabase(path, 'wal')).toThrow(/unversioned schema or application identity/)
|
||||
|
||||
const unchanged = new DatabaseSync(path)
|
||||
expect(unchanged.prepare('PRAGMA user_version').get()).toEqual({ user_version: 0 })
|
||||
expect(unchanged.prepare('PRAGMA journal_mode').get()).toEqual({ journal_mode: 'delete' })
|
||||
expect(unchanged.prepare(
|
||||
"SELECT name FROM sqlite_schema WHERE type = 'table' AND name = 'sessions'",
|
||||
).get()).toEqual({ name: 'sessions' })
|
||||
unchanged.close()
|
||||
})
|
||||
|
||||
it('counts a sqliteX table as user-owned instead of mistaking it for SQLite metadata', async () => {
|
||||
const path = await freshDbPath()
|
||||
const unrelated = new DatabaseSync(path)
|
||||
unrelated.exec('CREATE TABLE sqliteX (value TEXT)')
|
||||
unrelated.exec("INSERT INTO sqliteX VALUES ('safe')")
|
||||
unrelated.close()
|
||||
|
||||
expect(() => openDatabase(path, 'wal')).toThrow(/unversioned schema or application identity/)
|
||||
|
||||
const unchanged = new DatabaseSync(path)
|
||||
expect(unchanged.prepare('SELECT value FROM sqliteX').get()).toEqual({ value: 'safe' })
|
||||
expect(unchanged.prepare('PRAGMA application_id').get()).toEqual({ application_id: 0 })
|
||||
expect(unchanged.prepare('PRAGMA user_version').get()).toEqual({ user_version: 0 })
|
||||
expect(unchanged.prepare('PRAGMA journal_mode').get()).toEqual({ journal_mode: 'delete' })
|
||||
unchanged.close()
|
||||
})
|
||||
|
||||
it('rejects view-only and foreign-application unversioned databases without mutation', async () => {
|
||||
const viewPath = await freshDbPath()
|
||||
const viewOnly = new DatabaseSync(viewPath)
|
||||
viewOnly.exec('CREATE VIEW foreign_view AS SELECT 1 AS value')
|
||||
viewOnly.close()
|
||||
|
||||
expect(() => openDatabase(viewPath, 'wal')).toThrow(/unversioned schema or application identity/)
|
||||
const unchangedView = new DatabaseSync(viewPath)
|
||||
expect(unchangedView.prepare('PRAGMA journal_mode').get()).toEqual({ journal_mode: 'delete' })
|
||||
expect(unchangedView.prepare(
|
||||
"SELECT type FROM sqlite_schema WHERE name = 'foreign_view'",
|
||||
).get()).toEqual({ type: 'view' })
|
||||
unchangedView.close()
|
||||
|
||||
const applicationPath = await freshDbPath()
|
||||
const foreignApplication = new DatabaseSync(applicationPath)
|
||||
foreignApplication.exec('PRAGMA application_id = 12345')
|
||||
foreignApplication.close()
|
||||
|
||||
expect(() => openDatabase(applicationPath, 'wal')).toThrow(/unversioned schema or application identity/)
|
||||
const unchangedApplication = new DatabaseSync(applicationPath)
|
||||
expect(unchangedApplication.prepare('PRAGMA application_id').get()).toEqual({ application_id: 12345 })
|
||||
expect(unchangedApplication.prepare('PRAGMA user_version').get()).toEqual({ user_version: 0 })
|
||||
expect(unchangedApplication.prepare('PRAGMA journal_mode').get()).toEqual({ journal_mode: 'delete' })
|
||||
unchangedApplication.close()
|
||||
})
|
||||
|
||||
it('rejects a current-version database with a foreign application identity', async () => {
|
||||
const path = await freshDbPath()
|
||||
const foreign = new DatabaseSync(path)
|
||||
foreign.exec('PRAGMA application_id = 12345')
|
||||
foreign.exec(`PRAGMA user_version = ${SCHEMA_VERSION}`)
|
||||
foreign.close()
|
||||
|
||||
expect(() => openDatabase(path, 'wal')).toThrow(/has application id 12345/)
|
||||
|
||||
const unchanged = new DatabaseSync(path)
|
||||
expect(unchanged.prepare('PRAGMA application_id').get()).toEqual({ application_id: 12345 })
|
||||
expect(unchanged.prepare('PRAGMA user_version').get()).toEqual({ user_version: SCHEMA_VERSION })
|
||||
expect(unchanged.prepare('PRAGMA journal_mode').get()).toEqual({ journal_mode: 'delete' })
|
||||
unchanged.close()
|
||||
})
|
||||
|
||||
it('rolls back schema objects and identity stamps when initialization fails', async () => {
|
||||
const path = await freshDbPath()
|
||||
const conflicting = new DatabaseSync(path)
|
||||
conflicting.exec(`PRAGMA application_id = ${SESSION_PERSISTENCE_SQLITE_APPLICATION_ID}`)
|
||||
conflicting.exec(`PRAGMA user_version = ${SCHEMA_VERSION}`)
|
||||
conflicting.exec("CREATE VIEW persistence_state AS SELECT 1 AS singleton, 'foreign' AS store_id")
|
||||
conflicting.close()
|
||||
|
||||
expect(() => openDatabase(path, 'wal')).toThrow()
|
||||
|
||||
const unchanged = new DatabaseSync(path)
|
||||
expect(unchanged.prepare(
|
||||
"SELECT type FROM sqlite_schema WHERE name = 'persistence_state'",
|
||||
).get()).toEqual({ type: 'view' })
|
||||
expect(unchanged.prepare(
|
||||
"SELECT type FROM sqlite_schema WHERE name = 'sessions'",
|
||||
).get()).toBeUndefined()
|
||||
expect(unchanged.prepare(
|
||||
"SELECT type FROM sqlite_schema WHERE name = 'events'",
|
||||
).get()).toBeUndefined()
|
||||
expect(unchanged.prepare('PRAGMA application_id').get())
|
||||
.toEqual({ application_id: SESSION_PERSISTENCE_SQLITE_APPLICATION_ID })
|
||||
expect(unchanged.prepare('PRAGMA user_version').get()).toEqual({ user_version: SCHEMA_VERSION })
|
||||
expect(unchanged.prepare('PRAGMA journal_mode').get()).toEqual({ journal_mode: 'delete' })
|
||||
unchanged.close()
|
||||
})
|
||||
|
||||
it('stamps the persistence application identity with the schema version', async () => {
|
||||
const path = await freshDbPath()
|
||||
openDatabase(path, 'wal').close()
|
||||
|
||||
const db = new DatabaseSync(path)
|
||||
expect(db.prepare('PRAGMA application_id').get())
|
||||
.toEqual({ application_id: SESSION_PERSISTENCE_SQLITE_APPLICATION_ID })
|
||||
expect(db.prepare('PRAGMA user_version').get()).toEqual({ user_version: SCHEMA_VERSION })
|
||||
db.close()
|
||||
})
|
||||
|
||||
it('rejects a sibling v3 database (the merge-collided version) rather than opening it against missing columns', async () => {
|
||||
// Version 3 identified two incompatible sibling layouts, so it is always rejected.
|
||||
const path = await freshDbPath()
|
||||
openDatabase(path, 'wal').close() // creates + stamps user_version = SCHEMA_VERSION
|
||||
const db = openDatabase(path, 'wal')
|
||||
db.exec('PRAGMA user_version = 3')
|
||||
db.close()
|
||||
expect(() => openDatabase(path, 'wal')).toThrow(/schema version 3, incompatible with this build/)
|
||||
})
|
||||
|
||||
it('a corrupt-JSON row in the uncommitted tail is discarded on load, not unloadable', async () => {
|
||||
const path = await freshDbPath()
|
||||
const m = meta('corrupt-tail')
|
||||
const b1 = await backend(path)
|
||||
await b1.ctx.sessionPersistence.create(m)
|
||||
await b1.ctx.sessionPersistence.append(m.id, oneTurnLog()) // committed: seqs 0..5
|
||||
await b1.dispose()
|
||||
|
||||
// A torn row after the last committed turn has invalid JSON. `scanRows` locates the boundary
|
||||
// from seq/type columns without parsing the tail, preserves the committed prefix, and load
|
||||
// deletes the row; invalid JSON inside the committed region would remain fatal.
|
||||
const db = openDatabase(path, 'wal')
|
||||
db.prepare('INSERT INTO events (session_id, seq, type, time, data) VALUES (?, 6, ?, 7, ?)')
|
||||
.run(m.id, 'turn/start', '{not valid json')
|
||||
db.close()
|
||||
|
||||
const b2 = await backend(path)
|
||||
const loaded = await b2.ctx.sessionPersistence.load(m.id)
|
||||
expect(loaded.events).toEqual(oneTurnLog()) // torn tail discarded, committed intact (turn 1 already balanced → no closers)
|
||||
// load physically deleted the corrupt tail row, so a fresh append continues.
|
||||
await b2.ctx.sessionPersistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 6, time: 8, data: { turn: 2 } },
|
||||
{ type: 'turn/end', seq: 7, time: 9, data: { turn: 2, reason: { kind: 'completed' } } },
|
||||
])
|
||||
const reloaded = await b2.ctx.sessionPersistence.load(m.id)
|
||||
expect(reloaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
await b2.dispose()
|
||||
})
|
||||
|
||||
it('append rolls back the whole batch on a mid-batch seq collision (transaction)', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(SessionPersistenceSqlite, { path: ':memory:' })
|
||||
const m = meta('rollback')
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, oneTurnLog()) // seqs 0..5
|
||||
|
||||
// A batch that re-states an already-stored seq must be rejected and leave
|
||||
// the stored log unchanged (the UNIQUE (session_id, seq) constraint fires
|
||||
// inside the transaction → ROLLBACK).
|
||||
await expect(ctx.sessionPersistence.append(m.id, oneTurnLog())).rejects.toThrow()
|
||||
const loaded = await ctx.sessionPersistence.load(m.id)
|
||||
expect(loaded.events).toEqual(oneTurnLog()) // unchanged
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('persists across separate backend instances over the same file', async () => {
|
||||
const path = await freshDbPath()
|
||||
const m = meta('persist', '/proj')
|
||||
const ctx1 = new Context()
|
||||
await ctx1.plugin(SessionStore)
|
||||
const fiber1 = await ctx1.plugin(SessionPersistenceSqlite, { path })
|
||||
await ctx1.sessionPersistence.create(m)
|
||||
await ctx1.sessionPersistence.append(m.id, oneTurnLog())
|
||||
await fiber1.dispose()
|
||||
|
||||
const ctx2 = new Context()
|
||||
await ctx2.plugin(SessionStore)
|
||||
const fiber2 = await ctx2.plugin(SessionPersistenceSqlite, { path })
|
||||
expect((await ctx2.sessionPersistence.list()).map(x => x.id)).toContain(m.id)
|
||||
const loaded = await ctx2.sessionPersistence.load(m.id)
|
||||
expect(loaded.meta).toMatchObject({ id: m.id, cwd: '/proj' })
|
||||
expect(loaded.events).toEqual(oneTurnLog())
|
||||
await fiber2.dispose()
|
||||
})
|
||||
|
||||
it('source-qualifies revisions across stores while preserving same-file reopen identity', async () => {
|
||||
const pathA = await freshDbPath()
|
||||
const pathB = await freshDbPath()
|
||||
const m = meta('revision-source')
|
||||
const a = await backend(pathA)
|
||||
await a.ctx.sessionPersistence.create(m)
|
||||
await a.ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
const revisionA = (await a.ctx.sessionPersistence.listSnapshots())[0]?.revision
|
||||
await a.dispose()
|
||||
|
||||
const probeA = openDatabase(pathA, 'wal')
|
||||
const storeIdA = (probeA.prepare(
|
||||
'SELECT store_id FROM persistence_state WHERE singleton = 1',
|
||||
).get() as { store_id: string }).store_id
|
||||
probeA.close()
|
||||
|
||||
const aliasA = `${pathA}.alias`
|
||||
await symlink(pathA, aliasA)
|
||||
const reopenedA = await backend(aliasA)
|
||||
expect((await reopenedA.ctx.sessionPersistence.listSnapshots())[0]?.revision).toBe(revisionA)
|
||||
await reopenedA.dispose()
|
||||
|
||||
const b = await backend(pathB)
|
||||
await b.ctx.sessionPersistence.create(m)
|
||||
await b.ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
const revisionB = (await b.ctx.sessionPersistence.listSnapshots())[0]?.revision
|
||||
const probeB = openDatabase(pathB, 'wal')
|
||||
const storeIdB = (probeB.prepare(
|
||||
'SELECT store_id FROM persistence_state WHERE singleton = 1',
|
||||
).get() as { store_id: string }).store_id
|
||||
probeB.close()
|
||||
expect(storeIdB).not.toBe(storeIdA)
|
||||
expect(revisionB).not.toBe(revisionA)
|
||||
expect(String(revisionA)).toMatch(/:revision:1$/)
|
||||
expect(String(revisionB)).toMatch(/:revision:1$/)
|
||||
await b.dispose()
|
||||
})
|
||||
|
||||
it('binds a full stored prefix to the same revision as a lightweight read', async () => {
|
||||
const b = await backend()
|
||||
const m = meta('stored-prefix-revision')
|
||||
await b.ctx.sessionPersistence.create(m)
|
||||
await b.ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
const persistence = b.ctx.sessionPersistence as SessionPersistenceSqlite
|
||||
|
||||
const stored = await persistence.loadStored(m.id)
|
||||
expect(stored?.revision).toBe(await persistence.readStoredRevision(m.id))
|
||||
expect(await persistence.readStoredRevision(SessionId('missing-revision'))).toBeUndefined()
|
||||
await b.dispose()
|
||||
})
|
||||
|
||||
it('changes revisions when a deleted session id is materialized again in the same database', async () => {
|
||||
const path = await freshDbPath()
|
||||
const m = meta('recreated-revision')
|
||||
const first = await backend(path)
|
||||
await first.ctx.sessionPersistence.create(m)
|
||||
await first.ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
const before = (await first.ctx.sessionPersistence.listSnapshots())[0]?.revision
|
||||
await first.dispose()
|
||||
|
||||
const cleanup = openDatabase(path, 'wal')
|
||||
cleanup.prepare('DELETE FROM sessions WHERE id = ?').run(m.id)
|
||||
cleanup.close()
|
||||
|
||||
const second = await backend(path)
|
||||
await second.ctx.sessionPersistence.create(m)
|
||||
await second.ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
const after = (await second.ctx.sessionPersistence.listSnapshots())[0]?.revision
|
||||
expect(after).not.toBe(before)
|
||||
expect(String(before)).toMatch(/:revision:1$/)
|
||||
expect(String(after)).toMatch(/:revision:1$/)
|
||||
await second.dispose()
|
||||
})
|
||||
|
||||
it('awaits in-flight readiness before surfacing snapshot-list cancellation', async () => {
|
||||
const b = await backend()
|
||||
const internals = b.ctx.sessionPersistence as unknown as { ready: Promise<void> }
|
||||
const originalReady = internals.ready
|
||||
const readiness = Promise.withResolvers<undefined>()
|
||||
internals.ready = readiness.promise
|
||||
const reason = new Error('SQLite snapshot readiness cancelled')
|
||||
const controller = new AbortController()
|
||||
const pending = b.ctx.sessionPersistence.listSnapshots(controller.signal)
|
||||
let settled = false
|
||||
void pending.then(
|
||||
() => { settled = true },
|
||||
() => { settled = true },
|
||||
)
|
||||
|
||||
controller.abort(reason)
|
||||
await Promise.resolve()
|
||||
expect(settled).toBe(false)
|
||||
|
||||
readiness.resolve(undefined)
|
||||
await expect(pending).rejects.toBe(reason)
|
||||
internals.ready = originalReady
|
||||
await b.dispose()
|
||||
})
|
||||
|
||||
it('exposes the schema version constant', () => {
|
||||
expect(SCHEMA_VERSION).toBe(13)
|
||||
})
|
||||
|
||||
it('keeps the revision stable for an empty repair hook', async () => {
|
||||
const b = await backend()
|
||||
const m = meta('empty-repair')
|
||||
await b.ctx.sessionPersistence.create(m)
|
||||
await b.ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
const before = await b.ctx.sessionPersistence.listSnapshots()
|
||||
await (b.ctx.sessionPersistence as SessionPersistenceSqlite).commitRepair(m, undefined, [])
|
||||
expect(await b.ctx.sessionPersistence.listSnapshots()).toEqual(before)
|
||||
await b.dispose()
|
||||
})
|
||||
})
|
||||
|
||||
describe('SessionPersistenceSqlite: edge cases', () => {
|
||||
it('resolves the preparation-cache default without schema normalization', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
let persistence!: SessionPersistenceSqlite
|
||||
await ctx.plugin(Object.assign((inner: Context) => {
|
||||
persistence = new SessionPersistenceSqlite(inner, {
|
||||
path: ':memory:',
|
||||
journalMode: 'wal',
|
||||
})
|
||||
}, { inject: ['sessions'] }))
|
||||
|
||||
expect(await persistence.list()).toEqual([])
|
||||
await ctx.fiber.dispose()
|
||||
})
|
||||
|
||||
it('uses the configured preparation cache through the public service', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(SessionPersistenceSqlite, {
|
||||
path: ':memory:',
|
||||
preparedSessionCacheSize: 1,
|
||||
writeBatchMaxDelayMs: 1,
|
||||
})
|
||||
const m = meta('sqlite-preparation-cache')
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
|
||||
const preparation = await ctx.sessionPersistence.prepare(m.id)
|
||||
expect(preparation.session.header).toEqual(m)
|
||||
preparation[Symbol.dispose]()
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('rejects and closes a current-schema database with an invalid store identity', async () => {
|
||||
const path = await freshDbPath()
|
||||
const db = openDatabase(path, 'wal')
|
||||
db.exec("UPDATE persistence_state SET store_id = '' WHERE singleton = 1")
|
||||
db.close()
|
||||
|
||||
const b = await backend(path)
|
||||
await expect(b.ctx.sessionPersistence.listSnapshots()).rejects.toThrow(/no valid store identity/)
|
||||
await expect(b.dispose()).resolves.toBeUndefined()
|
||||
})
|
||||
|
||||
it('creates a new database and WAL sidecars with owner-only modes without changing its parent mode', async () => {
|
||||
if (process.platform === 'win32') return
|
||||
const path = await freshDbPath()
|
||||
const dir = dirname(path)
|
||||
await chmod(dir, 0o755)
|
||||
|
||||
const b = await backend(path)
|
||||
await b.ctx.sessionPersistence.list()
|
||||
|
||||
expect((await stat(dir)).mode & 0o777).toBe(0o755)
|
||||
expect((await stat(path)).mode & 0o777).toBe(0o600)
|
||||
expect((await stat(`${path}-wal`)).mode & 0o777).toBe(0o600)
|
||||
expect((await stat(`${path}-shm`)).mode & 0o777).toBe(0o600)
|
||||
await b.dispose()
|
||||
})
|
||||
|
||||
it('creates a persistent rollback journal with owner-only mode', async () => {
|
||||
if (process.platform === 'win32') return
|
||||
const path = await freshDbPath()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(SessionPersistenceSqlite, { path, journalMode: 'persist' })
|
||||
const m = meta('persist-permissions')
|
||||
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
|
||||
expect((await stat(path)).mode & 0o777).toBe(0o600)
|
||||
expect((await stat(`${path}-journal`)).mode & 0o777).toBe(0o600)
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('preserves the mode of an existing database file', async () => {
|
||||
if (process.platform === 'win32') return
|
||||
const path = await freshDbPath()
|
||||
await writeFile(path, '', { mode: 0o644 })
|
||||
await chmod(path, 0o644)
|
||||
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(SessionPersistenceSqlite, { path, journalMode: 'delete' })
|
||||
await ctx.sessionPersistence.list()
|
||||
|
||||
expect((await stat(path)).mode & 0o777).toBe(0o644)
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('surfaces an invalid database path during pre-creation', async () => {
|
||||
const path = await freshDbPath()
|
||||
const b = await backend(`${path}\0`)
|
||||
|
||||
await expect(b.ctx.sessionPersistence.list()).rejects.toMatchObject({ code: 'ERR_INVALID_ARG_VALUE' })
|
||||
await b.dispose()
|
||||
})
|
||||
|
||||
it('append rolls back and rethrows when an event INSERT fails inside the transaction', async () => {
|
||||
const path = await freshDbPath()
|
||||
const m = meta('rollback-insert')
|
||||
const b1 = await backend(path)
|
||||
await b1.ctx.sessionPersistence.create(m)
|
||||
await b1.ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
|
||||
// A SECOND backend over the same file loads the session first, so it adopts
|
||||
// cursor 6 (the committed length) into its OWN in-memory state.
|
||||
const b2 = await backend(path)
|
||||
await b2.ctx.sessionPersistence.load(m.id) // cursor 6 in b2
|
||||
const turn2: SessionEvent[] = [
|
||||
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2 } },
|
||||
{ type: 'turn/end', seq: 7, time: 8, data: { turn: 2, reason: { kind: 'completed' } } },
|
||||
]
|
||||
// b1 commits seq 6..7 first.
|
||||
await b1.ctx.sessionPersistence.append(m.id, turn2)
|
||||
// b2 still thinks its cursor is 6, so this batch passes the contiguity check
|
||||
// but its INSERT of seq 6 hits the UNIQUE (session_id, seq) constraint
|
||||
// mid-transaction → ROLLBACK + rethrow.
|
||||
await expect(b2.ctx.sessionPersistence.append(m.id, turn2)).rejects.toThrow(/UNIQUE/)
|
||||
// b1's turn is intact; b2's rolled-back attempt left nothing extra.
|
||||
const loaded = await b1.ctx.sessionPersistence.load(m.id)
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
await b1.dispose()
|
||||
await b2.dispose()
|
||||
})
|
||||
|
||||
it('journalMode config reaches the database (default wal, rollback modes selectable)', async () => {
|
||||
// :memory: databases always report journal_mode=memory, so probe file DBs.
|
||||
const walPath = await freshDbPath()
|
||||
const bWal = await backend(walPath)
|
||||
await bWal.ctx.sessionPersistence.create(meta('jm-wal'))
|
||||
const probe = openDatabase(walPath, 'wal')
|
||||
expect((probe.prepare('PRAGMA journal_mode').get() as { journal_mode: string }).journal_mode).toBe('wal')
|
||||
probe.close()
|
||||
await bWal.dispose()
|
||||
|
||||
const deletePath = await freshDbPath()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(SessionPersistenceSqlite, { path: deletePath, journalMode: 'delete' })
|
||||
await ctx.sessionPersistence.create(meta('jm-delete'))
|
||||
// Probe through a second connection: journal_mode=delete is a per-database
|
||||
// property only insofar as no WAL files exist — assert the world, not the
|
||||
// backend's self-report (no -wal sidecar after writes in delete mode).
|
||||
const db = openDatabase(deletePath, 'delete')
|
||||
expect((db.prepare('PRAGMA journal_mode').get() as { journal_mode: string }).journal_mode).toBe('delete')
|
||||
db.close()
|
||||
expect(existsSync(`${deletePath}-wal`)).toBe(false)
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('HMR: a DIFFERENT session colliding with a materialized on-disk id is rejected', async () => {
|
||||
const path = await freshDbPath()
|
||||
// Instance 1 materializes a session and disposes.
|
||||
const b1 = await backend(path)
|
||||
const s1 = b1.ctx.sessions.create(SessionId('hmr-collide'))
|
||||
appendLog(s1, oneTurnLog())
|
||||
await b1.ctx.sessions.flush(s1)
|
||||
await b1.dispose()
|
||||
|
||||
// A fresh context with an UNRELATED live session reusing the id meets a
|
||||
// materialized row that is NOT a prefix of its events → reject.
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
let session!: Session
|
||||
await ctx.plugin(Object.assign((inner: Context) => {
|
||||
session = inner.sessions.create(SessionId('hmr-collide'))
|
||||
}, { inject: ['sessions'] }))
|
||||
session.append('turn/start', { turn: 1 })
|
||||
await ctx.plugin(SessionPersistenceSqlite, { path })
|
||||
await expectFlushError(ctx.sessions.flush(session), /id collision/)
|
||||
await ctx.fiber.dispose()
|
||||
})
|
||||
})
|
||||
|
||||
describe('surface field round-trip', () => {
|
||||
it('rowToEvent parses surface fields from EventRow columns', () => {
|
||||
const row: EventRow = {
|
||||
seq: 0, type: 'assistant/message', time: 1,
|
||||
data: JSON.stringify({ turn: 1, step: 1, content: [] }),
|
||||
source_event_seqs: JSON.stringify([3, 5]),
|
||||
surface_op: JSON.stringify('append'),
|
||||
}
|
||||
const event = rowToEvent(row)
|
||||
expect((event as SurfaceEvent).sourceEventSeqs).toEqual([3, 5])
|
||||
expect((event as SurfaceEvent).surfaceOp).toBe('append')
|
||||
})
|
||||
|
||||
it('rowToEvent handles replace surfaceOp object', () => {
|
||||
const row: EventRow = {
|
||||
seq: 0, type: 'assistant/message', time: 1,
|
||||
data: JSON.stringify({ turn: 1, step: 1, content: [] }),
|
||||
source_event_seqs: JSON.stringify([0, 1]),
|
||||
surface_op: JSON.stringify({ op: 'replace', start: 0, end: 1 }),
|
||||
}
|
||||
const event = rowToEvent(row)
|
||||
expect((event as SurfaceEvent).sourceEventSeqs).toEqual([0, 1])
|
||||
expect((event as SurfaceEvent).surfaceOp).toEqual({ op: 'replace', start: 0, end: 1 })
|
||||
})
|
||||
|
||||
it('scanRows with surface columns reconstructs events with surface fields', () => {
|
||||
const rows: EventRow[] = [
|
||||
{ seq: 0, type: 'user/message', time: 1,
|
||||
data: JSON.stringify({ content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }),
|
||||
source_event_seqs: null, surface_op: '{"op":"replace","start":0,"end":0}' },
|
||||
{ seq: 1, type: 'turn/end', time: 2,
|
||||
data: JSON.stringify({ turn: 1, reason: { kind: 'completed' } }),
|
||||
source_event_seqs: null, surface_op: null },
|
||||
]
|
||||
const { preserved } = scanRows(rows)
|
||||
expect(preserved).toHaveLength(2)
|
||||
expect((preserved[0]! as SurfaceEvent).surfaceOp).toEqual({ op: 'replace', start: 0, end: 0 })
|
||||
expect((preserved[0]! as SurfaceEvent).sourceEventSeqs).toBeUndefined()
|
||||
expect((preserved[1] as SessionEvent<SurfaceEventType>).surfaceOp).toBeUndefined()
|
||||
})
|
||||
|
||||
it('append and load round-trips surface fields through SQLite', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(SessionPersistenceSqlite, { path: ':memory:' })
|
||||
const session = ctx.sessions.create(SessionId('roundtrip-surface'))
|
||||
session.append('turn/start', { turn: 1 })
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
session.append('user/message', createUserMessage({
|
||||
content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' },
|
||||
}), { surfaceOp: 'append' })
|
||||
session.append('assistant/message', {
|
||||
turn: 1, step: 1,
|
||||
message: createMessage({
|
||||
role: 'assistant',
|
||||
content: [],
|
||||
source: {
|
||||
kind: 'model',
|
||||
...{ provider: 'mock', model: 'mock' },
|
||||
},
|
||||
}),
|
||||
}, { surfaceOp: 'append', sourceEventSeqs: [2] })
|
||||
session.append('step/end', { turn: 1, step: 1 })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
await ctx.sessions.flush(session)
|
||||
const loaded = await ctx.sessionPersistence.load(SessionId('roundtrip-surface'))
|
||||
expect(loaded.events).toHaveLength(6)
|
||||
const um = loaded.events[2]!
|
||||
expect((um as SurfaceEvent).surfaceOp).toBe('append')
|
||||
expect((um as SurfaceEvent).sourceEventSeqs).toBeUndefined()
|
||||
const am = loaded.events[3]!
|
||||
expect((am as SurfaceEvent).surfaceOp).toBe('append')
|
||||
expect((am as SurfaceEvent).sourceEventSeqs).toEqual([2])
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('persists events with surfaceOp but no sourceEventSeqs (covers null branch in surfaceBindings)', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(SessionPersistenceSqlite, { path: ':memory:' })
|
||||
const session = ctx.sessions.create(SessionId('surface-noseq'))
|
||||
session.append('turn/start', { turn: 1 })
|
||||
session.append('user/message', createUserMessage({
|
||||
content: [],
|
||||
source: { kind: 'user' },
|
||||
}), { surfaceOp: 'append' })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
await ctx.sessions.flush(session)
|
||||
const loaded = await ctx.sessionPersistence.load(SessionId('surface-noseq'))
|
||||
expect((loaded.events[1]! as SurfaceEvent).surfaceOp).toBe('append')
|
||||
expect((loaded.events[1]! as SurfaceEvent).sourceEventSeqs).toBeUndefined()
|
||||
await fiber.dispose()
|
||||
})
|
||||
})
|
||||
30
packages/session/session-persistence-sqlite/tsconfig.json
Normal file
30
packages/session/session-persistence-sqlite/tsconfig.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
{
|
||||
"path": "../session-persistence"
|
||||
},
|
||||
{
|
||||
"path": "../../support/invariants"
|
||||
}
|
||||
]
|
||||
}
|
||||
6
packages/session/session-persistence/README.i18n.yaml
Normal file
6
packages/session/session-persistence/README.i18n.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/session-persistence/session-persistence/README.md
|
||||
README.md: 89c7cd5ebaff6f9ce9df9b60a50121dff4ddeeb5
|
||||
README.zh.md: 1ef7eb6c6c507167f61bb5df7c4ce8183aac5d0d
|
||||
83
packages/session/session-persistence/README.md
Normal file
83
packages/session/session-persistence/README.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# @deepseek-ai/dsh-session-persistence
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
The abstract durable session-persistence seam (`ctx.sessionPersistence`). Defines WHAT a persistence backend does — durably store, reload, and list sessions — without saying HOW. Mirrors the `dsh-bash` capability-seam template ([capability seams](../../../.agents/notes/implemented/architecture/2026-06-13-capability-seams.md)): an abstract service here, a concrete implementation in a sibling package, consumers that inject the interface.
|
||||
|
||||
The persisted unit IS the existing `SessionEvent` (event-sourced model — the log is the single source of truth), so there is no parallel "persisted message" type. Metadata that is NOT replayable conversation state (format version, cwd, lineage, seed boundary, origin, delegation depth) travels separately as `SessionHeader`, owned by `dsh-session` and re-exported here.
|
||||
|
||||
## Service API (`ctx.sessionPersistence`)
|
||||
|
||||
| Method | Contract |
|
||||
|---|---|
|
||||
| `locate(meta): SessionLocation \| undefined` | Resolve an absolute per-session artifact target without I/O or materialization. Backends without an independent local artifact return `undefined`. |
|
||||
| `create(meta): Promise<void>` | Register a new session's metadata. MAY defer the physical write until the first `append` (lazy materialization). |
|
||||
| `append(id, events): Promise<void>` | Durably persist a batch. Append-only; first event `seq` == stored next-seq after any repair; rejects non-JSON-serializable data naming the offending type. |
|
||||
| `prepare(id, signal?): Promise<SessionPreparation>` | Reserve the exact unpublished Session used by resume. A coordinator reuses an earlier inspection when available, commits pending recovery, and releases an unpublished reservation back to its bounded cache on disposal. |
|
||||
| `load(id): Promise<{ meta; events }>` | Return an immutable balanced logical log after supported same-version shape upgrades and commit cold recovery. A live load first flushes its snapshot and rejects while its turn is open; a cold load preserves an interrupted final turn and durably closes it with synthetic `tool/result`/`step/end?`/`turn/end {interrupted}` events. Only a torn tail fragment is dropped; committed corruption, malformed shapes, and unknown `version` reject. |
|
||||
| `inspect(id, signal?): Promise<{ meta; events }>` | Return an upgraded, validated, deeply frozen logical view without committing recovery or publishing a Session. A cold view receives in-memory synthetic recovery closers while its physical torn tail remains untouched; an already-live view is its current immutable snapshot and may contain an open turn. Coordinator-backed implementations retain the exact cold unpublished Session in a bounded LRU for later `prepare`, but discard and reload it when the stored revision changes. Same-id inspections share an in-flight read. |
|
||||
| `readFrom(id, fromSeq, signal?): Promise<{ meta; events }>` | The detached physical-suffix primitive: return valid stored events with `seq >= fromSeq` without preparation caching, truncation, closers, or coordinator state. A `fromSeq` at or past the stored end returns an empty event list; a negative or non-safe-integer `fromSeq` rejects. Seek-capable backends (SQLite) read only the suffix unless a supported old shape requires prefix context for normalization; sequential backends (JSONL) parse the whole artifact and skip forward. Intended for checkpoint consumers that fold only the tail past a watermark. |
|
||||
| `list(signal?): Promise<SessionHeader[]>` | Lightweight listing from metadata, no full-log parse. The optional signal cancels backend listing work. A zero-event lazily-materialized session is absent from `list`. |
|
||||
| `listSnapshots(signal?): Promise<SessionPersistenceSnapshot[]>` | Lightweight metadata plus an opaque branded per-log revision, without loading event logs. A revision stays equal while that log and its backing store are unchanged, changes after append or mutating load repair, and cannot collide solely because two stores use the same local counter. The optional signal requests cancellation of backend discovery work; first-party backends settle any started listing work before rejecting so an awaited call is quiescent. |
|
||||
|
||||
## Invariants every backend must honor
|
||||
|
||||
- **Append-only; a crashed turn is closed, not truncated.** Flushed events are never rewritten. A crash can leave an unclosed final turn whose events are real and possibly large; `load` preserves them and durably appends synthetic closers (a risk-classified error `tool/result` per unanswered assistant call, then `step/end?`+`turn/end {interrupted}`) to balance the log and keep the rehydrated history a valid provider transcript. Only a never-fully-written torn tail fragment is discarded.
|
||||
- **Contiguous seq.** `load` rejects a `seq` gap/parse error in the MIDDLE of the log; `append`'s first `seq` must equal the stored next-seq.
|
||||
- **JSON-serializable data.** `append` materializes each direct/replay batch through the shared one-pass lossless-JSON boundary. Live `Session` events are already deep-frozen, but the write coordinator still copies each event into a persistence-owned buffer.
|
||||
- **Durability.** `append` returns only once the batch is durable.
|
||||
|
||||
## The write coordinator
|
||||
|
||||
`PersistenceCoordinator` owns per-id state and serialization, one bounded write controller per live session, lazy materialization, crash-tail repair, session adoption, and quiescent disposal. A first-party backend composes one, implements the small `PersistenceBackend` storage hook interface, and delegates its stateful methods. JSONL and SQLite therefore share lifecycle correctness while retaining different storage primitives; see the [coordinator Agent Note](../../../.agents/notes/implemented/architecture/2026-06-18-shared-persistence-write-coordinator.md), [flush-controller simplification](../../../.agents/notes/implemented/simplification/2026-07-23-collapse-persistence-flush-state.md), and [bounded batching decision](../../../.agents/notes/implemented/architecture/2026-08-08-bounded-session-persistence-write-batching.md).
|
||||
|
||||
Each `session/event` copies its event into the session controller. The first pending event starts a fixed batching window; later events join without resetting its deadline. The configured `writeBatchMaxDelayMs` bounds this intentional wait, not event-loop, initialization, serialized-operation, or backend latency. Events admitted during a write form a new bounded batch. `session/flush` cancels the wait and is a shared quiescence barrier that drains events admitted while it runs. A background failure is logged once, retains the ordered batch, and pauses automatic retry; a new event starts a fresh window, while explicit flush or backend teardown retries immediately and surfaces a repeated failure.
|
||||
|
||||
Crash repair is cold-only. For a live id, `load(id)` snapshots the authoritative in-memory log, waits for that snapshot to become durable, and returns it only when balanced; an open live turn rejects instead of receiving synthetic interruption closers. For a cold id, inspection reads, validates, freezes, and constructs one unpublished Session; repeated inspection reuses that object graph only while its source revision remains current. `prepare(id)` performs the same check before repair, reserves the exact Session, commits any pending torn-tail/interrupted-turn repair, and returns it for publication. HMR adoption reads through `loadStored`, applies the coordinator's cwd check, and never closes the active turn.
|
||||
|
||||
Backend reads normalize the exact supported same-version shapes before current-shape validation. Pre-identity messages receive the deterministic id `legacy-message:<session-id>:<event-seq>`; a tool-result content replacement inherits its target's imported id. A pre-react-loop `turn/start` loses its obsolete trigger, a removed `steering/message` becomes the same identified `user/message`, and an older `turn/end` maps its terminal reason without inventing unavailable cancellation provenance. The coordinator uses the same normalized view for `load`, `inspect`, `readFrom`, ownerless-state claims, and HMR prefix adoption. Storage remains append-only: reads do not rewrite old records, and later appends use the current shape. These are narrow import exceptions from the [pre-identity message](../../../.agents/notes/implemented/bug-fix/2026-07-28-load-pre-identity-session-messages.md) and [pre-react-loop session](../../../.agents/notes/implemented/bug-fix/2026-08-04-load-pre-react-loop-sessions.md) decisions, not a general v0 migration promise.
|
||||
|
||||
When a live session emits `session/disposed`, the coordinator waits for its controller, serializes a final drain, then releases state owned by that exact `Session` object. Failed retirement leaves the controller in the live-session map, so backend teardown can retry it. Backend teardown stops event admission first, flushes every remaining controller, awaits per-id operations, and only then closes the storage handle.
|
||||
|
||||
The side-effect-free `locate`, lightweight `listSnapshots`, and per-id `readStoredRevision` queries remain backend-owned because they describe storage topology and revision identity rather than write orchestration. `listSnapshots(signal?)` passes the caller's exact signal into backend discovery so observers can cancel that work without detaching it.
|
||||
|
||||
The `PersistenceBackend<TornMarker>` hooks (the only seam between the coordinator and storage):
|
||||
|
||||
| Hook | Role |
|
||||
|---|---|
|
||||
| `name` | Backend label for the dispose-failure `AggregateError`. |
|
||||
| `loadStored(id, signal?)` | Read a stored prefix by id across every storage scope. Used by resume/load, non-mutating inspect, live adoption, and the create-collision probe. The optional signal belongs to observation-only reads. Returned metadata identifies `id`; `revision` identifies exactly the returned header and events; an opaque `tornMarker` is present iff a torn tail must be truncated. |
|
||||
| `readStoredRevision(id, signal?)` | Read the current source-qualified revision for one id without loading its event log. It uses the same revision representation as `loadStored` and returns `undefined` when the id is absent. |
|
||||
| `loadStoredFrom?(id, fromSeq, signal?)` | Optional seek-capable suffix read behind the service's `readFrom`: the header plus stored events with `seq >= fromSeq`, non-mutating, no torn marker. SQLite implements it (`WHERE seq >= ?`); a backend that omits it gets the coordinator's fallback — `loadStored` plus a forward skip. |
|
||||
| `appendBatch(meta, events, isMaterialized)` | Durably append a contiguous batch, lazily materializing ATOMICALLY when not yet materialized. |
|
||||
| `commitRepair(meta, tornMarker, closers)` | Make a crash repair durable: truncate the torn tail (iff `tornMarker !== undefined` — a marker may be falsy, e.g. seq/offset `0`) and append `closers`. NOT required to be atomic. Used by load (truncate + closers) and live-adoption (truncate only). |
|
||||
| `list(signal?)` | List all stored metadata, observing optional cancellation. |
|
||||
| `close?()` | Optional lifecycle teardown (e.g. close a db handle), awaited after the dispose drain. |
|
||||
|
||||
The coordinator asserts the stored id and compares stored/live cwd before repair or live adoption. Its `inspect()` path takes ownership of fresh backend values, validates and freezes them once, and retains at most the configured number of unpublished Sessions without calling `commitRepair`. A retained source is reused or repaired only when its revision still equals `readStoredRevision`; otherwise the coordinator reloads it. This freshness check does not add cross-process writer exclusion. Revision retries converge when the durable log remains unchanged for one read/check round trip; continuous external writers can delay `load`, `inspect`, or `prepare`. The `tornMarker` is fully OPAQUE: the coordinator only tests `!== undefined` and round-trips it to `commitRepair`, never inspecting its value (the JSONL backend uses the byte offset to truncate to, the SQLite backend the seq to delete from). A third-party backend MAY implement the abstract service directly without the coordinator, but it must provide the same non-mutating inspection and trustworthy lightweight snapshot revisions. See [the write-coordinator Agent Note](../../../.agents/notes/implemented/architecture/2026-06-18-shared-persistence-write-coordinator.md).
|
||||
|
||||
## Metadata and location types
|
||||
|
||||
Re-exported from `dsh-session`: `SessionHeader` (immutable session metadata: `version`, `id`, `createdAt`, `cwd?`, `parentSession?`, `seedLength?`, `origin?`, `delegationDepth?`). `SessionLocation` is `{ readonly kind: string; readonly path: string }`; its path is an absolute backend target, not proof that the artifact exists or contains an unflushed turn.
|
||||
|
||||
## Model Experience
|
||||
|
||||
### Resumed conversation history
|
||||
|
||||
#### What the model sees
|
||||
|
||||
This seam adds no prompt or schema. Resume restores stored surface events as message history; stored request headers reconstruct earlier calls, while the new loop composes the current system prompt, tools, and session prefix for its next request. Crash repair marks an assistant request without a durable call as `TOOL_NOT_STARTED`; a durable call without a result becomes `TOOL_OUTCOME_UNKNOWN`, whose text lets the model retry read-only or idempotent work but directs it to verify side effects or ask the user instead of retrying blindly.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Zero tokens during ordinary persistence. Resume restores retained history cost and pays the current request envelope normally; each repaired call adds the quoted retained error text.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
Persistence does not mutate live request prefixes. A resumed loop can reuse provider cache only when its reconstructed history, current envelope, and model route match; crash-repair results append without rewriting earlier history.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **No deletion or retention surface** — pruning stored sessions is out-of-band backend maintenance.
|
||||
- **`list()` is unpaginated and unfiltered** — it returns every stored session's header; fine for local stores, unindexed at scale.
|
||||
- **Repair-time synthetic closers are the only crash story** — a backend must synthesize `tool/result`/`step/end`/`turn/end` closers on load; there is no partial-turn resume that continues an interrupted turn instead of closing it.
|
||||
83
packages/session/session-persistence/README.zh.md
Normal file
83
packages/session/session-persistence/README.zh.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# @deepseek-ai/dsh-session-persistence
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
这是用于持久保存会话的抽象 seam（`ctx.sessionPersistence`）。它定义持久化后端做什么：持久存储、重新加载和列出会话，而不规定如何实现。它与 `dsh-bash` 能力 seam 模板一致（见[能力 seam](../../../.agents/notes/implemented/architecture/2026-06-13-capability-seams.md)）：本包提供抽象服务，同级包提供具体实现，消费方注入接口。
|
||||
|
||||
持久化单元就是现有 `SessionEvent`（事件溯源模型：日志是唯一真源），因此不存在另一套并行的「持久消息」类型。不属于可回放对话状态的元数据（格式版本、cwd、血缘、种子边界、origin、委托深度）作为 `SessionHeader` 单独传输，该类型归 `dsh-session` 所有，并在此重新导出。
|
||||
|
||||
## 服务 API（`ctx.sessionPersistence`）
|
||||
|
||||
| 方法 | 契约 |
|
||||
|---|---|
|
||||
| `locate(meta): SessionLocation \| undefined` | 在不执行 I/O 或实体化的情况下解析绝对的每会话产物目标。没有独立本地产物的后端返回 `undefined`。 |
|
||||
| `create(meta): Promise<void>` | 注册新会话元数据。可以将物理写入延迟到第一次 `append`（延迟实体化）。 |
|
||||
| `append(id, events): Promise<void>` | 持久保存一个批次。仅追加；任何修复后，第一个事件 `seq` == 已存储 next-seq；非 JSON 可序列化数据会被拒绝，并命名违规类型。 |
|
||||
| `prepare(id, signal?): Promise<SessionPreparation>` | 预留恢复使用的精确未发布 Session。协调器会尽可能复用之前的检查结果、提交待处理恢复，并在 dispose 时将未发布 reservation 释放回有界缓存。 |
|
||||
| `load(id): Promise<{ meta; events }>` | 在升级受支持的同版本形状后返回不可变、平衡的逻辑日志，并提交冷恢复。实时 load 先 flush 其快照，并在轮次开放时拒绝；冷 load 保留中断的最终轮次，并用合成 `tool/result`/`step/end?`/`turn/end {interrupted}` 事件持久关闭它。只丢弃撕裂尾部碎片；已提交损坏、格式错误的形状和未知 `version` 会被拒绝。 |
|
||||
| `inspect(id, signal?): Promise<{ meta; events }>` | 返回已经升级、验证和深度冻结的逻辑视图，但不提交恢复或发布 Session。冷视图会获得仅存在于内存的合成恢复 closer，物理撕裂尾部保持不变；已经实时存在的视图则是当前不可变快照，可能包含打开的 turn。基于协调器的实现会在有界 LRU 中保留精确的冷未发布 Session，供后续 `prepare` 使用，但已存储 revision 变化后会丢弃并重新读取。同 id 检查共享进行中的读取。 |
|
||||
| `readFrom(id, fromSeq, signal?): Promise<{ meta; events }>` | 脱离的物理后缀原语：返回 `seq >= fromSeq` 的有效已存储事件，不进入 preparation 缓存、不截断、不合成 closer，也不发布协调器状态。`fromSeq` 达到或超过已存储末尾时返回空事件列表；负数或非安全整数 `fromSeq` 会被拒绝。可寻址后端（SQLite）只读后缀，除非受支持的旧形状需要前缀上下文才能完成规范化；顺序后端（JSONL）解析整个产物并向前跳过。用于只续折水位之后尾部的 checkpoint 消费方。 |
|
||||
| `list(signal?): Promise<SessionHeader[]>` | 从元数据轻量列出，不解析完整日志。可选信号取消后端列表工作。零事件延迟实体化会话不在 `list` 中。 |
|
||||
| `listSnapshots(signal?): Promise<SessionPersistenceSnapshot[]>` | 返回轻量元数据和每份日志一个不透明、带品牌类型的修订值，不加载事件日志。日志及其后端存储不变时，修订保持相等；append 或变更性 load 修复后会改变；不会仅因两个存储使用相同本地计数器而冲突。可选信号请求取消后端发现工作；第一方后端会先等待所有已启动的列出工作结束，再予以拒绝，因此调用返回拒绝时，相关工作已完全停稳。 |
|
||||
|
||||
## 每个后端必须遵守的不变量
|
||||
|
||||
- **仅追加；崩溃轮次会被关闭，而非截断。** 已 flush 事件绝不重写。崩溃可留下未关闭最终轮次，其事件真实且可能很大；`load` 保留它们，并持久追加合成 closer（为每个未回答 assistant 调用添加按风险分类错误 `tool/result`，再添加 `step/end?`+`turn/end {interrupted}`），以平衡日志，并确保重新载入的历史仍是有效的提供方 transcript（文本记录）。只丢弃从未完整写入的撕裂尾部碎片。
|
||||
- **连续 seq。**`load` 拒绝日志中间的 `seq` 缺口/解析错误；`append` 的第一个 `seq` 必须等于已存储 next-seq。
|
||||
- **JSON 可序列化数据。**`append` 通过共享单遍无损 JSON 边界实体化每个直接/回放批次。实时 `Session` 事件已深度冻结，但写入协调器仍将每个事件复制到持久化自有缓冲区。
|
||||
- **持久性。**`append` 只在批次持久后返回。
|
||||
|
||||
## 写入协调器
|
||||
|
||||
`PersistenceCoordinator` 负责每 id 状态和串行化、每个活动会话各自的有界写入 controller、延迟实体化、崩溃尾部修复、会话接管和完全停稳的 dispose（资源释放）。第一方后端组合一个协调器，实现小型 `PersistenceBackend` 存储钩子接口，并委托其有状态方法。因此 JSONL 和 SQLite 共享生命周期正确性，同时保留不同存储原语；见[协调器 Agent Note](../../../.agents/notes/implemented/architecture/2026-06-18-shared-persistence-write-coordinator.md)、[flush controller 简化](../../../.agents/notes/implemented/simplification/2026-07-23-collapse-persistence-flush-state.md)和[有界批处理决策](../../../.agents/notes/implemented/architecture/2026-08-08-bounded-session-persistence-write-batching.md)。
|
||||
|
||||
每个 `session/event` 将事件复制到会话 controller。第一个待处理事件会开启固定批处理窗口；后续事件会加入该批次，但不会重置截止时间。配置的 `writeBatchMaxDelayMs` 只限制这段有意等待，而不限制事件循环、初始化、串行化操作或后端延迟。写入期间接纳的事件会形成一个新的有界批次。`session/flush` 会取消等待，并作为共享的完全停稳屏障，排空屏障运行期间接纳的事件。后台写入失败只记录一次日志，保留顺序不变的批次，并暂停自动重试；新事件会开启新的固定窗口，而显式 flush 或后端拆卸会立即重试，并在失败再次发生时向调用方暴露失败。
|
||||
|
||||
崩溃修复只适用于冷状态。对于实时 id，`load(id)` 为权威内存日志制作快照，等待该快照持久，并只在平衡时返回；开放实时轮次会被拒绝，而不会收到合成中断 closer。对于冷 id，检查只读取、验证、冻结并构造一次未发布 Session；只有来源 revision 仍然是当前值时，重复检查才会复用该对象图。`prepare(id)` 在修复前执行相同校验，预留精确 Session，提交任何待处理的撕裂尾部或中断轮次修复，并将其返回用于发布。HMR 接管通过 `loadStored` 读取，应用协调器 cwd 检查，并绝不关闭活动轮次。
|
||||
|
||||
后端读取会在当前形状验证前，规范化明确受支持的同版本形状。消息标识机制引入前的消息会获得确定性的 id `legacy-message:<session-id>:<event-seq>`；工具结果的内容替换会继承其目标导入后的 id。react-loop 重构前的 `turn/start` 会移除过时的 trigger，已移除的 steering（中途引导）事件 `steering/message` 会转换为同一条带标识的 `user/message`；旧版 `turn/end` 会在不虚构无法获得的取消来源的前提下映射终止原因。协调器对 `load`、`inspect`、`readFrom`、无 owner 状态的认领和 HMR 前缀接管使用同一份规范化视图。存储仍然仅追加：读取不会重写旧记录，此后追加的事件使用当前形状。这些是[消息标识机制引入前的消息](../../../.agents/notes/implemented/bug-fix/2026-07-28-load-pre-identity-session-messages.md)与 [react-loop 重构前会话](../../../.agents/notes/implemented/bug-fix/2026-08-04-load-pre-react-loop-sessions.md)决策所规定的范围受限的导入例外，并不构成通用的 v0 迁移承诺。
|
||||
|
||||
实时会话发出 `session/disposed` 时，协调器等待其 controller，串行化最终 drain，然后释放该精确 `Session` 对象拥有的状态。失败退役会将 controller 保留在实时会话 map 中，使后端拆卸可重试。后端拆卸先停止事件接纳，flush 每个剩余 controller，等待每 id 操作，最后才关闭存储句柄。
|
||||
|
||||
无副作用 `locate`、轻量 `listSnapshots` 和按 id 查询的 `readStoredRevision` 仍由后端负责，因为它们描述存储拓扑和 revision 身份，而非写入编排。`listSnapshots(signal?)` 将调用方传入的同一个信号传给后端发现流程，使观察者可在不脱离该工作的情况下取消。
|
||||
|
||||
`PersistenceBackend<TornMarker>` 钩子（协调器与存储之间的唯一 seam）：
|
||||
|
||||
| 钩子 | 职责 |
|
||||
|---|---|
|
||||
| `name` | dispose 失败 `AggregateError` 的后端标签。 |
|
||||
| `loadStored(id, signal?)` | 在全部存储范围中按 id 读取已存储前缀。用于恢复／加载、非修改式 inspect、实时接管和 create 冲突探测。可选信号属于仅观察读取。返回元数据标识 `id`；`revision` 精确标识返回的 header 和事件；当且仅当必须截断撕裂尾部时才存在不透明 `tornMarker`。 |
|
||||
| `readStoredRevision(id, signal?)` | 在不加载事件日志的情况下读取一个 id 当前的来源限定 revision。它使用与 `loadStored` 相同的 revision 表示；id 不存在时返回 `undefined`。 |
|
||||
| `loadStoredFrom?(id, fromSeq, signal?)` | 服务 `readFrom` 背后的可选可寻址后缀读取：返回 header 和 `seq >= fromSeq` 的已存储事件，非变更、无撕裂标记。SQLite 实现它（`WHERE seq >= ?`）；不实现的后端使用协调器回退——`loadStored` 加向前跳过。 |
|
||||
| `appendBatch(meta, events, isMaterialized)` | 持久追加连续批次；尚未实体化时以原子方式延迟实体化。 |
|
||||
| `commitRepair(meta, tornMarker, closers)` | 使崩溃修复持久：截断撕裂尾部（当且仅当 `tornMarker !== undefined`；标记可为 falsy，例如 seq/offset `0`），并追加 `closers`。不要求原子性。由 load（截断 + closer）和实时接管（仅截断）使用。 |
|
||||
| `list(signal?)` | 列出全部已存储元数据，并遵循可选的取消信号。 |
|
||||
| `close?()` | 可选生命周期拆卸（例如关闭 db 句柄），在 dispose drain 后等待。 |
|
||||
|
||||
协调器断言已存储 id，并在修复或实时接管前比较已存储/实时 cwd。其 `inspect()` 路径取得新鲜后端值的所有权，只验证和冻结一次，并在不调用 `commitRepair` 的情况下最多保留配置数量的未发布 Session。只有保留源的 revision 仍等于 `readStoredRevision` 时，系统才会复用或修复它；否则协调器会重新读取。该新鲜性校验不会增加跨进程写入排他。持久日志在一次读取与复核往返内保持不变时，revision 重试才能收敛；持续的外部写入可能延迟 `load`、`inspect` 或 `prepare`。`tornMarker` 完全不透明：协调器只测试 `!== undefined`，并将其原样往返给 `commitRepair`，绝不检查值（JSONL 后端使用待截断字节偏移，SQLite 后端使用待删除 seq）。第三方后端可以不用协调器直接实现抽象服务，但必须提供相同非变更检查和可信轻量快照修订。详见[写入协调器 Agent Note](../../../.agents/notes/implemented/architecture/2026-06-18-shared-persistence-write-coordinator.md)。
|
||||
|
||||
## 元数据与位置类型
|
||||
|
||||
从 `dsh-session` 重新导出：`SessionHeader`（不可变会话元数据：`version`、`id`、`createdAt`、`cwd?`、`parentSession?`、`seedLength?`、`origin?`、`delegationDepth?`）。`SessionLocation` 是 `{ readonly kind: string; readonly path: string }`；其 path 是绝对后端目标，不证明产物已存在或包含未 flush 轮次。
|
||||
|
||||
## 模型体验
|
||||
|
||||
### 恢复的对话历史
|
||||
|
||||
#### 模型所见
|
||||
|
||||
该 seam 不添加提示词或 schema。恢复会将已存储的表层事件还原为消息历史；已存储请求 header 重建较早调用，新 loop 则为下一次请求组合当前系统提示词、工具和会话前缀。崩溃修复将没有持久调用的 assistant 请求标记为 `TOOL_NOT_STARTED`；有持久调用但无结果时变为 `TOOL_OUTCOME_UNKNOWN`，其文本允许模型重试只读或幂等工作，但要求验证副作用或询问用户，而不是盲目重试。
|
||||
|
||||
#### Token 影响
|
||||
|
||||
普通持久化期间为零 token。恢复后会重新计入保留历史的 token 用量，并照常计入当前请求 envelope 的 token 用量；每个已修复调用都会增加一段以引用形式保留的错误文本。
|
||||
|
||||
#### KV Cache 影响
|
||||
|
||||
持久化不修改实时请求前缀。只有当重建历史、当前 envelope 和模型路由匹配时，恢复 loop 才能重用提供方缓存；崩溃修复结果仅追加，不重写较早历史。
|
||||
|
||||
## 已知限制与暂缓事项
|
||||
|
||||
- **无删除或保留接口**：剪枝已存储会话是带外后端维护。
|
||||
- **`list()` 无分页且无过滤**：它返回每个已存储会话的 header；适合本地存储，大规模时无索引。
|
||||
- **修复时合成 closer 是唯一崩溃方案**：后端必须在 load 时合成 `tool/result`/`step/end`/`turn/end` closer；没有继续中断轮次而不先关闭它的部分轮次恢复。
|
||||
42
packages/session/session-persistence/package.json
Normal file
42
packages/session/session-persistence/package.json
Normal file
@@ -0,0 +1,42 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-session-persistence",
|
||||
"description": "Abstract durable session persistence seam (ctx.sessionPersistence) for the DeepSeek Harness",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./invariant": {
|
||||
"types": "./lib/types/invariant.d.ts",
|
||||
"default": "./lib/invariant.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-brand": "^0.0.1",
|
||||
"@deepseek-ai/dsh-invariants": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session": "^0.0.1",
|
||||
"@deepseek-ai/dsh-timeout": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-brand": "workspace:^",
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-scope": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-timeout": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
1273
packages/session/session-persistence/src/coordinator.ts
Normal file
1273
packages/session/session-persistence/src/coordinator.ts
Normal file
File diff suppressed because it is too large
Load Diff
203
packages/session/session-persistence/src/index.ts
Normal file
203
packages/session/session-persistence/src/index.ts
Normal file
@@ -0,0 +1,203 @@
|
||||
/**
|
||||
* Durable session-persistence seam (`ctx.sessionPersistence`). Backends store
|
||||
* {@link SessionEvent}s as the event-sourced log and carry non-replayable
|
||||
* {@link SessionHeader} metadata separately.
|
||||
* @module @deepseek-ai/dsh-session-persistence
|
||||
*/
|
||||
|
||||
import { Context, Service } from 'cordis'
|
||||
import { SessionPreparation } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionEvent, SessionId, SessionHeader } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionPersistenceRevision } from './revision.ts'
|
||||
|
||||
// Re-export the metadata vocabulary so consumers import it from the seam.
|
||||
export type { SessionHeader } from '@deepseek-ai/dsh-session'
|
||||
export { SessionPersistenceRevision } from './revision.ts'
|
||||
|
||||
/** Lightweight immutable source identity returned without loading a full log. */
|
||||
export interface SessionPersistenceSnapshot {
|
||||
/** Detached metadata for one materialized session. */
|
||||
header: SessionHeader
|
||||
/** Opaque source-qualified token that changes whenever this stored log changes. */
|
||||
revision: SessionPersistenceRevision
|
||||
}
|
||||
|
||||
/** Immutable logical session prepared from persistence or a live owner. */
|
||||
export interface SessionInspection {
|
||||
/** Validated immutable session metadata. */
|
||||
readonly meta: SessionHeader
|
||||
/** Validated contiguous logical event log. */
|
||||
readonly events: readonly SessionEvent[]
|
||||
}
|
||||
|
||||
// The backend-agnostic write-path orchestration first-party backends compose.
|
||||
export {
|
||||
DEFAULT_PREPARED_SESSION_CACHE_SIZE,
|
||||
DEFAULT_WRITE_BATCH_MAX_DELAY_MS,
|
||||
MAX_WRITE_BATCH_DELAY_MS,
|
||||
PersistenceCoordinator,
|
||||
SessionPersistenceCorruptionError,
|
||||
} from './coordinator.ts'
|
||||
export type {
|
||||
PersistenceBackend,
|
||||
PersistenceCoordinatorOptions,
|
||||
StoredPrefix,
|
||||
StoredSuffix,
|
||||
} from './coordinator.ts'
|
||||
|
||||
declare module 'cordis' {
|
||||
interface Context {
|
||||
sessionPersistence: SessionPersistence
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* A backend-resolved, per-session local artifact location. The path is an
|
||||
* absolute target path and can name an artifact that has not materialized yet.
|
||||
* Consumers must treat it as a location hint, never as an authorization token.
|
||||
*/
|
||||
export interface SessionLocation {
|
||||
/** Backend-specific artifact kind, for example `jsonl`. */
|
||||
readonly kind: string
|
||||
/** Absolute path to this session's backend-owned artifact. */
|
||||
readonly path: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Durable append-only session storage. Implementations preserve contiguous,
|
||||
* losslessly JSON-serializable events; {@link append} resolves only after
|
||||
* durability, and {@link load} balances a complete interrupted tail without
|
||||
* rewriting committed events.
|
||||
*/
|
||||
export abstract class SessionPersistence extends Service {
|
||||
constructor(ctx: Context) {
|
||||
super(ctx, 'sessionPersistence')
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve this backend's independent local artifact for a session without
|
||||
* reading, creating, flushing, or otherwise materializing it. Backends such
|
||||
* as SQLite that do not own one artifact per session return `undefined`.
|
||||
* @param meta - the immutable session header whose artifact is requested.
|
||||
* @returns the backend-specific absolute location, when one exists.
|
||||
*/
|
||||
abstract locate(meta: SessionHeader): SessionLocation | undefined
|
||||
|
||||
/**
|
||||
* Register a new session's metadata. A backend MAY defer the physical write
|
||||
* until the first {@link append} (lazy materialization), in which case a
|
||||
* created-but-never-appended session is absent from {@link list}
|
||||
* — abandoned sessions leave nothing behind.
|
||||
* @param meta - the immutable header (id, version, cwd, lineage) to record.
|
||||
*/
|
||||
abstract create(meta: SessionHeader): Promise<void>
|
||||
|
||||
/**
|
||||
* Durably persist a batch of events. Honors the append-only and contiguous-
|
||||
* seq contracts: the first event's `seq` MUST equal the stored next-seq
|
||||
* (after `load` has durably closed any interrupted turn). Rejects non-JSON-
|
||||
* serializable `event.data` with an error naming the offending event type.
|
||||
* @param id - the session the batch belongs to.
|
||||
* @param events - the contiguous batch to persist, in seq order.
|
||||
*/
|
||||
abstract append(id: SessionId, events: readonly SessionEvent[]): Promise<void>
|
||||
|
||||
/**
|
||||
* Prepare the exact unpublished Session used by resume. Implementations may
|
||||
* reuse object graphs retained by an earlier {@link inspect} after confirming
|
||||
* their durable revision is still current; disposal releases an unpublished
|
||||
* reservation. Revision retries require the durable log to remain unchanged
|
||||
* for one read/check round trip; continuous external writers may delay completion.
|
||||
* @param id - persisted session to prepare.
|
||||
* @param signal - optional cancellation for preparation work.
|
||||
* @returns one owned unpublished Session preparation.
|
||||
*/
|
||||
async prepare(id: SessionId, signal?: AbortSignal): Promise<SessionPreparation> {
|
||||
signal?.throwIfAborted()
|
||||
const loaded = await this.load(id)
|
||||
signal?.throwIfAborted()
|
||||
const sessions = this.ctx.get('sessions')
|
||||
if (sessions === undefined) {
|
||||
throw new Error('cannot prepare a session: SessionStore is not configured')
|
||||
}
|
||||
return SessionPreparation.create(sessions.prepare(id, {
|
||||
seed: loaded.events.map(event => structuredClone(event)),
|
||||
meta: structuredClone(loaded.meta),
|
||||
seedSource: 'persistence',
|
||||
}))
|
||||
}
|
||||
|
||||
/**
|
||||
* Load an immutable balanced logical view and commit any required cold
|
||||
* recovery. A complete interrupted final turn is preserved and durably
|
||||
* closed with missing tool errors plus any open step and turn boundaries;
|
||||
* only a torn final record is discarded. Unknown versions and corruption in
|
||||
* the committed prefix reject. Implementations MUST NOT crash-repair an
|
||||
* identity still bound to a live Session: a balanced live log may return as a
|
||||
* durable snapshot, while an open live turn rejects. Returned values may be
|
||||
* shared with immutable live or prepared state and must not be mutated.
|
||||
* Revision-based implementations may wait for one stable read/check round trip.
|
||||
* @param id - the persisted session to reload.
|
||||
* @returns the header and a log ending on a balanced `turn/end`.
|
||||
*/
|
||||
abstract load(id: SessionId): Promise<SessionInspection>
|
||||
|
||||
/**
|
||||
* Inspect an immutable logical session without committing recovery or
|
||||
* publishing it. A cold complete interrupted turn receives synthetic closers
|
||||
* in memory and a torn physical tail remains untouched. An already-live
|
||||
* Session instead yields its current immutable snapshot, which may contain an
|
||||
* open turn and its `session/end-seed` boundary. Coordinator-backed
|
||||
* implementations retain the exact cold unpublished Session for bounded
|
||||
* reuse by a later {@link prepare}. A stale ready source is reloaded; a source
|
||||
* already committing or reserved for resume remains exclusive, and inspection
|
||||
* may borrow its immutable view. Callers borrow only the immutable header and
|
||||
* log. Continuous external writers may delay revision convergence.
|
||||
* @param id - the persisted session to inspect.
|
||||
* @param signal - optional cancellation for queued and backend read work.
|
||||
* @returns the validated header and current logical event log.
|
||||
*/
|
||||
abstract inspect(id: SessionId, signal?: AbortSignal): Promise<SessionInspection>
|
||||
|
||||
/**
|
||||
* Read the stored events from `fromSeq` onward — the read-from-seq
|
||||
* primitive for read models that resume from a watermark (e.g. a persisted
|
||||
* projection cache folding only the tail past its checkpoint). Unlike
|
||||
* {@link inspect}, it is a detached physical suffix read: no preparation
|
||||
* cache, torn-tail truncation, synthetic closers, or coordinator-state
|
||||
* publication. Only events from the valid contiguous stored prefix are
|
||||
* returned, so a torn fragment never reaches the caller. `fromSeq` at or
|
||||
* beyond the stored prefix returns an empty event list (never an error).
|
||||
* Backends whose medium can seek by seq
|
||||
* (SQLite) read only the suffix; sequential media (JSONL, both encodings)
|
||||
* still parse the whole artifact and skip forward — the primitive bounds
|
||||
* what is RETURNED and refolded, not every backend's physical read.
|
||||
* @param id - the persisted session to read.
|
||||
* @param fromSeq - first event seq to include; a non-negative safe integer.
|
||||
* @param signal - optional cancellation for queued and backend read work.
|
||||
* @returns the header and the stored events with `seq >= fromSeq`.
|
||||
*/
|
||||
abstract readFrom(id: SessionId, fromSeq: number, signal?: AbortSignal):
|
||||
Promise<{ meta: SessionHeader; events: SessionEvent[] }>
|
||||
|
||||
/**
|
||||
* Lightweight listing from metadata, without a full-log parse.
|
||||
* @param signal - optional cancellation for backend listing work.
|
||||
* @returns one header per materialized session.
|
||||
*/
|
||||
abstract list(signal?: AbortSignal): Promise<SessionHeader[]>
|
||||
|
||||
/**
|
||||
* List materialized sessions with cheap per-log change tokens.
|
||||
*
|
||||
* Repeated observations of an unchanged log return the same revision. A
|
||||
* successful mutating {@link load} repair changes the next listed revision.
|
||||
* Revisions also distinguish independently backed stores so backend-local
|
||||
* counters cannot compare equal across different persistence sources.
|
||||
* @param signal - optional cancellation for backend snapshot-listing work.
|
||||
* @returns one header and opaque revision per materialized session without loading full logs.
|
||||
*/
|
||||
abstract listSnapshots(signal?: AbortSignal): Promise<SessionPersistenceSnapshot[]>
|
||||
}
|
||||
|
||||
export default SessionPersistence
|
||||
30
packages/session/session-persistence/src/invariant.ts
Normal file
30
packages/session/session-persistence/src/invariant.ts
Normal file
@@ -0,0 +1,30 @@
|
||||
/**
|
||||
* Package-owned invariant companion for `@deepseek-ai/dsh-session-persistence`.
|
||||
* @module @deepseek-ai/dsh-session-persistence/invariant
|
||||
*/
|
||||
|
||||
/* jscpd:ignore-start */
|
||||
import type { Context } from 'cordis'
|
||||
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
|
||||
|
||||
const PACKAGE_NAME = '@deepseek-ai/dsh-session-persistence'
|
||||
|
||||
/** Cordis companion plugin name. */
|
||||
export const name = 'session-persistence-invariant'
|
||||
/** Service required before the companion can reserve package ownership. */
|
||||
export const inject = ['invariants']
|
||||
|
||||
/**
|
||||
* No runtime invariant: persistence correctness requires backend round-trip and crash-tail tests;
|
||||
* this package exposes no continuously observable in-process relation.
|
||||
*/
|
||||
const install: InvariantInstaller = () => {}
|
||||
|
||||
/**
|
||||
* Register this package's invariant companion.
|
||||
* @param ctx - Cordis context carrying the invariant service.
|
||||
* @returns the installed registration's disposer after setup succeeds.
|
||||
*/
|
||||
export const apply = (ctx: Context): Promise<() => void> =>
|
||||
Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
|
||||
/* jscpd:ignore-end */
|
||||
348
packages/session/session-persistence/src/preparations.ts
Normal file
348
packages/session/session-persistence/src/preparations.ts
Normal file
@@ -0,0 +1,348 @@
|
||||
/**
|
||||
* Bounded sharing and exclusive reservation of unpublished Sessions.
|
||||
* @module @deepseek-ai/dsh-session-persistence/preparations
|
||||
*/
|
||||
|
||||
import type { Session, SessionId } from '@deepseek-ai/dsh-session'
|
||||
|
||||
interface PreparedSource {
|
||||
readonly session: Session
|
||||
}
|
||||
|
||||
type PreparationPhase = 'loading' | 'ready' | 'committing' | 'reserved'
|
||||
|
||||
interface PreparationEntry<Source, CommitState> {
|
||||
readonly id: SessionId
|
||||
readonly result: Promise<Source>
|
||||
phase: PreparationPhase
|
||||
source?: Source
|
||||
reservation?: SessionPreparationReservation<Source, CommitState>
|
||||
reservationSettled?: Promise<void>
|
||||
settleReservation?: () => void
|
||||
}
|
||||
|
||||
/** One exclusively held prepared source and its committed persistence state. */
|
||||
export interface SessionPreparationReservation<Source, CommitState> {
|
||||
readonly entry: PreparationEntry<Source, CommitState>
|
||||
readonly source: Source
|
||||
readonly state: CommitState
|
||||
}
|
||||
|
||||
/** Per-coordinator cold-read sharing, exclusive reservation, and ready-entry LRU. */
|
||||
export class SessionPreparations<Source extends PreparedSource, CommitState> {
|
||||
private readonly entries = new Map<SessionId, PreparationEntry<Source, CommitState>>()
|
||||
|
||||
constructor(private readonly capacity: number) {}
|
||||
|
||||
/**
|
||||
* Whether this pool currently knows about an unpublished identity.
|
||||
* @param id - session identity.
|
||||
* @returns whether an entry exists for the identity.
|
||||
*/
|
||||
has(id: SessionId): boolean {
|
||||
return this.entries.has(id)
|
||||
}
|
||||
|
||||
/**
|
||||
* Observe one prepared source, sharing an in-flight read for the same id.
|
||||
* @param id - session identity.
|
||||
* @param load - cold loader used when no entry exists.
|
||||
* @param signal - optional cancellation signal while waiting.
|
||||
* @returns the shared prepared source.
|
||||
*/
|
||||
async inspect(
|
||||
id: SessionId,
|
||||
load: () => Promise<Source>,
|
||||
signal?: AbortSignal,
|
||||
): Promise<Source> {
|
||||
const entry = this.entryFor(id, load)
|
||||
const loaded = signal === undefined
|
||||
? await entry.result
|
||||
: await observeQueuedAbort(entry.result, signal)
|
||||
const source = entry.source ?? loaded
|
||||
if (this.entries.get(id) === entry && entry.phase === 'ready') this.touch(entry)
|
||||
return source
|
||||
}
|
||||
|
||||
/**
|
||||
* Reserve one ready source after committing its pending durable repair.
|
||||
* @param id - session identity.
|
||||
* @param load - cold loader used when no entry exists.
|
||||
* @param commit - durable repair and cursor-state commit.
|
||||
* @param signal - optional cancellation signal while waiting.
|
||||
* @returns the exclusive reservation, or undefined if its entry was invalidated.
|
||||
*/
|
||||
async reserve(
|
||||
id: SessionId,
|
||||
load: () => Promise<Source>,
|
||||
commit: (source: Source) => Promise<{ source: Source; state: CommitState } | undefined>,
|
||||
signal?: AbortSignal,
|
||||
): Promise<SessionPreparationReservation<Source, CommitState> | undefined> {
|
||||
const entry = this.entryFor(id, load)
|
||||
await (signal === undefined ? entry.result : observeQueuedAbort(entry.result, signal))
|
||||
while (this.entries.get(id) === entry && entry.phase !== 'ready') {
|
||||
const settled = entry.reservationSettled
|
||||
/* v8 ignore next -- committing/reserved transitions install this waiter synchronously. */
|
||||
if (settled === undefined) throw new Error(`session "${id}" preparation lost its reservation waiter`)
|
||||
if (signal === undefined) await settled
|
||||
else await observeQueuedAbort(settled, signal)
|
||||
}
|
||||
if (this.entries.get(id) !== entry) return undefined
|
||||
const source = entry.source as Source
|
||||
const reservationSettled = Promise.withResolvers<void>()
|
||||
entry.phase = 'committing'
|
||||
entry.reservationSettled = reservationSettled.promise
|
||||
entry.settleReservation = reservationSettled.resolve
|
||||
let committed: { source: Source; state: CommitState } | undefined
|
||||
try {
|
||||
committed = await commit(source)
|
||||
} catch (error: unknown) {
|
||||
this.remove(entry)
|
||||
throw error
|
||||
}
|
||||
if (committed === undefined) {
|
||||
this.remove(entry)
|
||||
return undefined
|
||||
}
|
||||
entry.source = committed.source
|
||||
try {
|
||||
signal?.throwIfAborted()
|
||||
} catch (error: unknown) {
|
||||
this.makeReady(entry)
|
||||
throw error
|
||||
}
|
||||
if (this.entries.get(id) !== entry) return undefined
|
||||
const reservation: SessionPreparationReservation<Source, CommitState> = {
|
||||
entry,
|
||||
source: committed.source,
|
||||
state: committed.state,
|
||||
}
|
||||
entry.phase = 'reserved'
|
||||
entry.reservation = reservation
|
||||
return reservation
|
||||
}
|
||||
|
||||
/**
|
||||
* Return the exact reservation for Session publication, rejecting aliases.
|
||||
* @param session - exact Session candidate for publication.
|
||||
* @returns its reservation, or undefined when no preparation exists.
|
||||
*/
|
||||
reservationFor(session: Session): SessionPreparationReservation<Source, CommitState> | undefined {
|
||||
const entry = this.entries.get(session.id)
|
||||
if (entry === undefined) return undefined
|
||||
if (entry.phase === 'reserved'
|
||||
&& entry.source?.session === session
|
||||
&& entry.reservation !== undefined) {
|
||||
return entry.reservation
|
||||
}
|
||||
throw new Error(`cannot publish session "${session.id}": persisted state already owns this identity`)
|
||||
}
|
||||
|
||||
/**
|
||||
* Consume a reservation after its exact Session has attached.
|
||||
* @param reservation - reservation to consume.
|
||||
*/
|
||||
attach(reservation: SessionPreparationReservation<Source, CommitState>): void {
|
||||
const { entry } = reservation
|
||||
if (this.entries.get(entry.id) !== entry || entry.reservation !== reservation) {
|
||||
throw new Error(`session "${entry.id}" preparation is no longer reserved`)
|
||||
}
|
||||
this.remove(entry)
|
||||
}
|
||||
|
||||
/**
|
||||
* Consume a reservation whose caller only needs the committed inspection.
|
||||
* @param reservation - reservation to consume.
|
||||
*/
|
||||
discard(reservation: SessionPreparationReservation<Source, CommitState>): void {
|
||||
const { entry } = reservation
|
||||
if (this.entries.get(entry.id) !== entry || entry.reservation !== reservation) return
|
||||
this.remove(entry)
|
||||
}
|
||||
|
||||
/**
|
||||
* Return a reusable unpublished reservation to the ready LRU.
|
||||
* @param reservation - reservation to release.
|
||||
* @param reusable - whether the source remains valid for reuse.
|
||||
*/
|
||||
release(
|
||||
reservation: SessionPreparationReservation<Source, CommitState>,
|
||||
reusable: boolean,
|
||||
): void {
|
||||
const { entry } = reservation
|
||||
if (this.entries.get(entry.id) !== entry
|
||||
|| entry.reservation !== reservation
|
||||
|| entry.phase !== 'reserved') return
|
||||
if (!reusable) {
|
||||
this.remove(entry)
|
||||
return
|
||||
}
|
||||
delete entry.reservation
|
||||
this.makeReady(entry)
|
||||
}
|
||||
|
||||
/**
|
||||
* Discard a prepared view after the durable log changes.
|
||||
* @param id - changed session identity.
|
||||
*/
|
||||
invalidate(id: SessionId): void {
|
||||
const entry = this.entries.get(id)
|
||||
if (entry !== undefined) this.remove(entry)
|
||||
}
|
||||
|
||||
/**
|
||||
* Discard an exact stale ready source without disturbing an exclusive owner.
|
||||
* @param id - changed session identity.
|
||||
* @param expected - exact source observed before its revision check.
|
||||
* @returns whether the source was discarded, retained by a reservation, or is absent.
|
||||
*/
|
||||
discardReady(id: SessionId, expected: Source): 'discarded' | 'retained' | 'missing' {
|
||||
const entry = this.entries.get(id)
|
||||
if (entry === undefined || entry.source !== expected) return 'missing'
|
||||
if (entry.phase !== 'ready') return 'retained'
|
||||
this.remove(entry)
|
||||
return 'discarded'
|
||||
}
|
||||
|
||||
/**
|
||||
* Reject writes while an unpublished Session exclusively reserves the id.
|
||||
* @param id - session identity to check.
|
||||
*/
|
||||
assertWritable(id: SessionId): void {
|
||||
const phase = this.entries.get(id)?.phase
|
||||
if (phase === 'committing' || phase === 'reserved') {
|
||||
throw new Error(`cannot append session "${id}" while its persisted preparation is reserved`)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove a completed entry for an already-serialized append adoption.
|
||||
* @param id - adopted session identity.
|
||||
* @returns the prepared source, or undefined when no ready entry exists.
|
||||
*/
|
||||
takeReady(id: SessionId): Source | undefined {
|
||||
const entry = this.entries.get(id)
|
||||
if (entry === undefined || entry.phase !== 'ready' || entry.source === undefined) return undefined
|
||||
this.remove(entry)
|
||||
return entry.source
|
||||
}
|
||||
|
||||
private entryFor(
|
||||
id: SessionId,
|
||||
load: () => Promise<Source>,
|
||||
): PreparationEntry<Source, CommitState> {
|
||||
const existing = this.entries.get(id)
|
||||
if (existing !== undefined) return existing
|
||||
const deferred = Promise.withResolvers<Source>()
|
||||
const entry: PreparationEntry<Source, CommitState> = {
|
||||
id,
|
||||
result: deferred.promise,
|
||||
phase: 'loading',
|
||||
}
|
||||
this.entries.set(id, entry)
|
||||
let loading: Promise<Source>
|
||||
try {
|
||||
// Start immediately so a same-tick serialized append queues behind this
|
||||
// read. The deferred result settles only after the entry becomes ready.
|
||||
loading = load()
|
||||
} catch (error: unknown) {
|
||||
this.remove(entry)
|
||||
deferred.reject(error)
|
||||
return entry
|
||||
}
|
||||
void loading.then((source) => {
|
||||
if (this.entries.get(id) === entry) {
|
||||
entry.source = source
|
||||
this.makeReady(entry)
|
||||
}
|
||||
deferred.resolve(source)
|
||||
}, (error: unknown) => {
|
||||
this.remove(entry)
|
||||
deferred.reject(error)
|
||||
})
|
||||
return entry
|
||||
}
|
||||
|
||||
private makeReady(entry: PreparationEntry<Source, CommitState>): void {
|
||||
if (this.entries.get(entry.id) !== entry) return
|
||||
entry.phase = 'ready'
|
||||
const settle = entry.settleReservation
|
||||
delete entry.reservationSettled
|
||||
delete entry.settleReservation
|
||||
settle?.()
|
||||
this.touch(entry)
|
||||
}
|
||||
|
||||
private remove(entry: PreparationEntry<Source, CommitState>): void {
|
||||
if (this.entries.get(entry.id) !== entry) return
|
||||
this.entries.delete(entry.id)
|
||||
const settle = entry.settleReservation
|
||||
delete entry.reservationSettled
|
||||
delete entry.settleReservation
|
||||
settle?.()
|
||||
}
|
||||
|
||||
private touch(entry: PreparationEntry<Source, CommitState>): void {
|
||||
this.entries.delete(entry.id)
|
||||
this.entries.set(entry.id, entry)
|
||||
let readyCount = 0
|
||||
for (const candidate of this.entries.values()) {
|
||||
if (candidate.phase === 'ready') readyCount += 1
|
||||
}
|
||||
if (readyCount <= this.capacity) return
|
||||
for (const [id, candidate] of this.entries) {
|
||||
if (candidate.phase !== 'ready') continue
|
||||
this.entries.delete(id)
|
||||
return
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Give a queued observer a prompt cancellation view without cancelling shared work.
|
||||
* @param operation - shared operation whose settlement remains authoritative.
|
||||
* @param signal - observer-local cancellation signal.
|
||||
* @param started - whether the operation has crossed its cancellation cutoff.
|
||||
* @returns the operation result or the observer's prompt cancellation.
|
||||
*/
|
||||
export function observeQueuedAbort<T>(
|
||||
operation: Promise<T>,
|
||||
signal: AbortSignal,
|
||||
started: () => boolean = () => false,
|
||||
): Promise<T> {
|
||||
return new Promise<T>((resolve, reject) => {
|
||||
let settled = false
|
||||
const finish = (callback: () => void): void => {
|
||||
if (settled) return
|
||||
settled = true
|
||||
signal.removeEventListener('abort', onAbort)
|
||||
callback()
|
||||
}
|
||||
const onAbort = (): void => {
|
||||
if (started()) return
|
||||
finish(() => {
|
||||
try {
|
||||
signal.throwIfAborted()
|
||||
} catch (reason: unknown) {
|
||||
rejectObservation(reject, reason)
|
||||
return
|
||||
}
|
||||
/* v8 ignore next -- a native AbortSignal emits abort only after becoming aborted. */
|
||||
reject(new Error('queued observation abort event lacked an aborted signal'))
|
||||
})
|
||||
}
|
||||
signal.addEventListener('abort', onAbort, { once: true })
|
||||
operation.then(
|
||||
(value) => { finish(() => { resolve(value) }) },
|
||||
(reason: unknown) => {
|
||||
finish(() => { rejectObservation(reject, reason) })
|
||||
},
|
||||
)
|
||||
if (signal.aborted) onAbort()
|
||||
})
|
||||
}
|
||||
|
||||
/** Preserve an exact loader or AbortSignal reason, including legacy non-Error values. */
|
||||
function rejectObservation(reject: (reason?: unknown) => void, reason: unknown): void {
|
||||
reject(reason)
|
||||
}
|
||||
18
packages/session/session-persistence/src/revision.ts
Normal file
18
packages/session/session-persistence/src/revision.ts
Normal file
@@ -0,0 +1,18 @@
|
||||
/** Opaque revision identity for lightweight persistence observations. */
|
||||
|
||||
import type { Branded } from '@deepseek-ai/dsh-brand'
|
||||
|
||||
/**
|
||||
* Backend-owned token that identifies both one storage source and one revision
|
||||
* of a persisted session log.
|
||||
*/
|
||||
export type SessionPersistenceRevision = Branded<'SessionPersistenceRevision'>
|
||||
|
||||
/**
|
||||
* Brand a backend revision for the provider-neutral persistence contract.
|
||||
* @param value - backend-owned opaque revision representation.
|
||||
* @returns the same runtime string with persistence-revision identity.
|
||||
*/
|
||||
export function SessionPersistenceRevision(value: string): SessionPersistenceRevision {
|
||||
return value as SessionPersistenceRevision
|
||||
}
|
||||
159
packages/session/session-persistence/src/write-behind.ts
Normal file
159
packages/session/session-persistence/src/write-behind.ts
Normal file
@@ -0,0 +1,159 @@
|
||||
/**
|
||||
* Bounded per-session write batching for the shared persistence coordinator.
|
||||
* @module @deepseek-ai/dsh-session-persistence/write-behind
|
||||
*/
|
||||
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
|
||||
/** Dependencies and scheduling policy for one live session's write controller. */
|
||||
export interface SessionWriteBehindOptions {
|
||||
/** Maximum intentional batching wait after an idle queue receives work. */
|
||||
readonly maxDelayMs: number
|
||||
/** Persist one stable ordered prefix; resolves only after backend durability. */
|
||||
readonly write: (events: readonly SessionEvent[]) => Promise<void>
|
||||
/** Observe a detached background write failure without rejecting the producer. */
|
||||
readonly reportBackgroundFailure: (error: unknown) => void
|
||||
}
|
||||
|
||||
/**
|
||||
* Owns one live session's pending events, fixed batching deadline, active write,
|
||||
* failure retention, and explicit quiescence barrier.
|
||||
*/
|
||||
export class SessionWriteBehind {
|
||||
private pending: SessionEvent[] = []
|
||||
private timer: ReturnType<typeof setTimeout> | undefined
|
||||
private active: Promise<void> | undefined
|
||||
private barrier: Promise<void> | undefined
|
||||
private deadlineExpired = false
|
||||
private automaticPaused = false
|
||||
|
||||
/**
|
||||
* @param options - fixed scheduling policy and durable batch sink.
|
||||
*/
|
||||
constructor(private readonly options: SessionWriteBehindOptions) {}
|
||||
|
||||
/** Whether this controller owns queued events or an active durable write. */
|
||||
get hasWork(): boolean {
|
||||
return this.pending.length > 0 || this.active !== undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Copy one event into the persistence-owned queue and start a fixed deadline
|
||||
* when the automatic path is idle.
|
||||
* @param event - frozen live event to retain independently of its producer.
|
||||
*/
|
||||
enqueue(event: SessionEvent): void {
|
||||
const wasEmpty = this.pending.length === 0
|
||||
this.pending.push(structuredClone(event))
|
||||
if (this.barrier !== undefined) return
|
||||
if (this.automaticPaused) {
|
||||
this.automaticPaused = false
|
||||
this.deadlineExpired = false
|
||||
this.armTimer()
|
||||
} else if (wasEmpty) {
|
||||
this.armTimer()
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Cancel the batching wait and durably drain through a quiescent point.
|
||||
* Concurrent callers join the same barrier.
|
||||
* @returns a promise that rejects if the barrier's durable retry fails.
|
||||
*/
|
||||
flush(): Promise<void> {
|
||||
if (this.barrier !== undefined) return this.barrier
|
||||
this.cancelTimer()
|
||||
this.deadlineExpired = false
|
||||
this.automaticPaused = false
|
||||
const barrier = Promise.withResolvers<void>()
|
||||
this.barrier = barrier.promise
|
||||
void this.drainBarrier(barrier.resolve, barrier.reject)
|
||||
return barrier.promise
|
||||
}
|
||||
|
||||
/** Cancel the current automatic deadline without draining retained work. */
|
||||
cancelAutomaticWait(): void {
|
||||
this.cancelTimer()
|
||||
this.deadlineExpired = false
|
||||
}
|
||||
|
||||
/** Start the one fixed window for the current pending prefix. */
|
||||
private armTimer(): void {
|
||||
this.timer = setTimeout(() => { this.onDeadline() }, this.options.maxDelayMs)
|
||||
}
|
||||
|
||||
/** Cancel any pending automatic deadline. */
|
||||
private cancelTimer(): void {
|
||||
if (this.timer === undefined) return
|
||||
clearTimeout(this.timer)
|
||||
this.timer = undefined
|
||||
}
|
||||
|
||||
/** Start a background write now, or remember that an active write used the budget. */
|
||||
private onDeadline(): void {
|
||||
this.timer = undefined
|
||||
if (this.active !== undefined) {
|
||||
this.deadlineExpired = true
|
||||
return
|
||||
}
|
||||
this.startBackground()
|
||||
}
|
||||
|
||||
/** Start one detached write whose failure is reported and retained. */
|
||||
private startBackground(): void {
|
||||
const active = this.startWrite(true)
|
||||
void active.then(() => { this.continueAutomatic() }, () => {})
|
||||
}
|
||||
|
||||
/** Continue immediately after an over-budget active write, otherwise keep its timer. */
|
||||
private continueAutomatic(): void {
|
||||
if (this.barrier !== undefined || this.pending.length === 0) return
|
||||
if (this.deadlineExpired) {
|
||||
this.deadlineExpired = false
|
||||
this.startBackground()
|
||||
}
|
||||
}
|
||||
|
||||
/** Await overlapping work, drain to quiescence, and settle the shared barrier. */
|
||||
private async drainBarrier(resolve: () => void, reject: (reason?: unknown) => void): Promise<void> {
|
||||
try {
|
||||
const overlapping = this.active
|
||||
if (overlapping !== undefined) {
|
||||
await Promise.allSettled([overlapping])
|
||||
this.automaticPaused = false
|
||||
}
|
||||
while (this.pending.length > 0) await this.startWrite(false)
|
||||
} catch (error: unknown) {
|
||||
this.barrier = undefined
|
||||
reject(error)
|
||||
return
|
||||
}
|
||||
// Close admission to this barrier in the same job that observes the empty
|
||||
// queue, before resolving callers. A later enqueue therefore starts its own
|
||||
// automatic window instead of being stranded behind a settled barrier.
|
||||
this.barrier = undefined
|
||||
resolve()
|
||||
}
|
||||
|
||||
/** Start one stable pending prefix, retaining it in order if durability fails. */
|
||||
private startWrite(background: boolean): Promise<void> {
|
||||
const batch = this.pending.splice(0)
|
||||
this.cancelTimer()
|
||||
this.deadlineExpired = false
|
||||
const operation = Promise.resolve().then(() => this.options.write(batch))
|
||||
const active = operation
|
||||
.catch((error: unknown) => {
|
||||
this.pending = batch.concat(this.pending)
|
||||
this.cancelTimer()
|
||||
this.deadlineExpired = false
|
||||
this.automaticPaused = true
|
||||
if (background) this.options.reportBackgroundFailure(error)
|
||||
throw error
|
||||
})
|
||||
.finally(() => {
|
||||
this.active = undefined
|
||||
})
|
||||
this.active = active
|
||||
return active
|
||||
}
|
||||
}
|
||||
432
packages/session/session-persistence/tests/contract.ts
Normal file
432
packages/session/session-persistence/tests/contract.ts
Normal file
@@ -0,0 +1,432 @@
|
||||
/**
|
||||
* Reusable contract test for any {@link SessionPersistence} backend. A backend
|
||||
* package imports {@link runPersistenceContract} and calls it with a factory
|
||||
* that yields a fresh, empty backend (and a teardown), so every backend is held
|
||||
* to the same append-only / contiguous-seq / lazy-materialization / crash
|
||||
* semantics. The JSONL backend's own spec adds file-specific tests on top.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-session-persistence/tests/contract
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { SESSION_FORMAT_VERSION, Session, SessionId, TOOL_NOT_STARTED, TOOL_OUTCOME_UNKNOWN } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionEvent, SessionHeader, SurfaceEventType, SurfaceIntent } from '@deepseek-ai/dsh-session'
|
||||
import { CallId, MessageId, createMessage, freezeMessage } from '@deepseek-ai/dsh-llm'
|
||||
import type { SessionPersistence } from '../src/index.ts'
|
||||
|
||||
/** A backend under test plus its teardown. */
|
||||
export interface ContractBackend {
|
||||
persistence: SessionPersistence
|
||||
dispose: () => Promise<void>
|
||||
}
|
||||
|
||||
/** Build a minimal {@link SessionHeader} for a session id. */
|
||||
export function meta(id: string, cwd?: string): SessionHeader {
|
||||
return {
|
||||
version: SESSION_FORMAT_VERSION,
|
||||
id: SessionId(id),
|
||||
createdAt: 1000,
|
||||
...cwd !== undefined ? { cwd } : {},
|
||||
}
|
||||
}
|
||||
|
||||
/** A well-formed one-turn event log (contiguous seqs from 0). */
|
||||
export function oneTurnLog(): SessionEvent[] {
|
||||
return [
|
||||
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } },
|
||||
{ type: 'user/message', seq: 1, time: 2, data: freezeMessage({
|
||||
id: MessageId('one-turn-user'),
|
||||
role: 'user',
|
||||
content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' },
|
||||
}), surfaceOp: 'append' },
|
||||
{ type: 'step/start', seq: 2, time: 3, data: { turn: 1, step: 1 } },
|
||||
{ type: 'assistant/message', seq: 3, time: 4, data: {
|
||||
turn: 1, step: 1,
|
||||
message: freezeMessage({
|
||||
id: MessageId('one-turn-assistant'),
|
||||
role: 'assistant',
|
||||
content: [{ type: 'text', text: 'hello' }],
|
||||
source: {
|
||||
kind: 'model',
|
||||
...{ provider: 'mock', model: 'mock' },
|
||||
},
|
||||
}),
|
||||
}, surfaceOp: 'append' },
|
||||
{ type: 'step/end', seq: 4, time: 5, data: { turn: 1, step: 1 } },
|
||||
{ type: 'turn/end', seq: 5, time: 6, data: { turn: 1, reason: { kind: 'completed' } } },
|
||||
]
|
||||
}
|
||||
|
||||
/**
|
||||
* Append recorded events to a live session while forwarding surface metadata verbatim. The broad
|
||||
* `SessionEvent` union makes the typed marker optional, but the runtime guard must still reject a
|
||||
* surface event whose fixture omitted it; this helper never synthesizes a default.
|
||||
*/
|
||||
export function appendLog(session: Session, events: readonly SessionEvent[]): void {
|
||||
for (const e of events) {
|
||||
const se = e as SessionEvent<SurfaceEventType>
|
||||
if (se.surfaceOp !== undefined) {
|
||||
const intent: SurfaceIntent = {
|
||||
surfaceOp: se.surfaceOp,
|
||||
...se.sourceEventSeqs !== undefined ? { sourceEventSeqs: se.sourceEventSeqs } : {},
|
||||
}
|
||||
session.append(e.type, e.data, intent)
|
||||
} else {
|
||||
session.append(e.type, e.data)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the backend-agnostic contract suite. `make()` MUST return a fresh, empty
|
||||
* backend each call.
|
||||
*/
|
||||
export function runPersistenceContract(name: string, make: () => Promise<ContractBackend>): void {
|
||||
describe(`SessionPersistence contract: ${name}`, () => {
|
||||
it('round-trips a session: create + append → load returns identical meta and byte-identical events', async () => {
|
||||
const { persistence, dispose } = await make()
|
||||
try {
|
||||
const m = meta('s1', '/work')
|
||||
const log = oneTurnLog()
|
||||
await persistence.create(m)
|
||||
await persistence.append(m.id, log)
|
||||
|
||||
const loaded = await persistence.load(m.id)
|
||||
expect(loaded.meta).toMatchObject({ version: SESSION_FORMAT_VERSION, id: m.id, cwd: '/work' })
|
||||
expect(loaded.events).toEqual(log)
|
||||
} finally {
|
||||
await dispose()
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects a fractional creation timestamp without reserving its session id', async () => {
|
||||
const { persistence, dispose } = await make()
|
||||
try {
|
||||
const m = { ...meta('fractional-created-at'), createdAt: 1.5 }
|
||||
await expect(persistence.create(m))
|
||||
.rejects.toThrow('session metadata createdAt must be a non-negative safe integer')
|
||||
|
||||
const valid = meta('fractional-created-at')
|
||||
await persistence.create(valid)
|
||||
await persistence.append(valid.id, oneTurnLog())
|
||||
expect((await persistence.load(valid.id)).meta.createdAt).toBe(valid.createdAt)
|
||||
} finally {
|
||||
await dispose()
|
||||
}
|
||||
})
|
||||
|
||||
it('crash recovery: load preserves an interrupted (unclosed) turn and closes it with turn/end {interrupted}', async () => {
|
||||
const { persistence, dispose } = await make()
|
||||
try {
|
||||
const m = meta('interrupted')
|
||||
await persistence.create(m)
|
||||
await persistence.append(m.id, oneTurnLog()) // turn 1, committed (seqs 0..5)
|
||||
// A second turn that crashed mid-flight: turn/start + step/start were
|
||||
// durably written, but no step/end / turn/end ever arrived.
|
||||
await persistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2 } },
|
||||
{ type: 'step/start', seq: 7, time: 8, data: { turn: 2, step: 1 } },
|
||||
])
|
||||
const beforeRepair = (await persistence.listSnapshots())
|
||||
.find(snapshot => snapshot.header.id === m.id)?.revision
|
||||
|
||||
const inspected = await persistence.inspect(m.id)
|
||||
const afterInspect = (await persistence.listSnapshots())
|
||||
.find(snapshot => snapshot.header.id === m.id)?.revision
|
||||
expect(afterInspect).toBe(beforeRepair)
|
||||
expect(inspected.events.map(e => e.type)).toEqual([
|
||||
'turn/start', 'user/message', 'step/start', 'assistant/message', 'step/end', 'turn/end',
|
||||
'turn/start', 'step/start', 'step/end', 'turn/end',
|
||||
])
|
||||
|
||||
// load PRESERVES the interrupted turn's events (a turn can be huge — they
|
||||
// must not be truncated) and closes the orphaned turn with synthetic
|
||||
// boundary events: step/end (the step was open) then turn/end {interrupted}.
|
||||
const loaded = await persistence.load(m.id)
|
||||
const afterRepair = (await persistence.listSnapshots())
|
||||
.find(snapshot => snapshot.header.id === m.id)?.revision
|
||||
expect(afterRepair).not.toBe(beforeRepair)
|
||||
expect(loaded.events.map(e => e.type)).toEqual([
|
||||
'turn/start', 'user/message', 'step/start', 'assistant/message', 'step/end', 'turn/end', // turn 1
|
||||
'turn/start', 'step/start', 'step/end', 'turn/end', // turn 2: real events + synthetic closers
|
||||
])
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
|
||||
const last = loaded.events.at(-1)!
|
||||
expect(last.type === 'turn/end' && last.data.reason).toEqual({ kind: 'interrupted' })
|
||||
|
||||
// The closed log is durable and continuable: a fresh append continues at
|
||||
// the balanced length (seq 10), and a reload round-trips identically.
|
||||
await persistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 10, time: 9, data: { turn: 3 } },
|
||||
{ type: 'turn/end', seq: 11, time: 10, data: { turn: 3, reason: { kind: 'completed' } } },
|
||||
])
|
||||
const reloaded = await persistence.load(m.id)
|
||||
expect(reloaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
|
||||
} finally {
|
||||
await dispose()
|
||||
}
|
||||
})
|
||||
|
||||
it('crash recovery: an unstarted assistant tool request gets a retryable synthetic result', async () => {
|
||||
const { persistence, dispose } = await make()
|
||||
try {
|
||||
const m = meta('interrupted-toolcall')
|
||||
await persistence.create(m)
|
||||
await persistence.append(m.id, oneTurnLog()) // turn 1, committed (seqs 0..5)
|
||||
// Turn 2 crashed AFTER the assistant message asked for a tool call but
|
||||
// BEFORE the tool/result was written (the loop runs tools after logging
|
||||
// the assistant message — a process killed mid-tool lands exactly here).
|
||||
await persistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2 } },
|
||||
{ type: 'step/start', seq: 7, time: 8, data: { turn: 2, step: 1 } },
|
||||
{ type: 'assistant/message', seq: 8, time: 9, data: {
|
||||
turn: 2, step: 1,
|
||||
message: createMessage({
|
||||
role: 'assistant',
|
||||
content: [
|
||||
{ type: 'tool-call', id: CallId('call-x'), name: 'bash', arguments: '{}' },
|
||||
],
|
||||
source: {
|
||||
kind: 'model',
|
||||
...{ provider: 'mock', model: 'mock' },
|
||||
},
|
||||
}),
|
||||
}, surfaceOp: 'append' },
|
||||
])
|
||||
|
||||
const loaded = await persistence.load(m.id)
|
||||
// The orphaned call is answered by a synthetic error tool/result BEFORE
|
||||
// step/end + turn/end {interrupted}, so the step (and turn) are balanced
|
||||
// and a resumed session derives a valid transcript (no dangling call).
|
||||
expect(loaded.events.map(e => e.type)).toEqual([
|
||||
'turn/start', 'user/message', 'step/start', 'assistant/message', 'step/end', 'turn/end', // turn 1
|
||||
'turn/start', 'step/start', 'assistant/message', 'tool/result', 'step/end', 'turn/end', // turn 2
|
||||
])
|
||||
const synthetic = loaded.events.find(e => e.type === 'tool/result')
|
||||
expect(synthetic?.type === 'tool/result' && synthetic.data).toMatchObject({
|
||||
message: {
|
||||
source: { kind: 'tool', callId: CallId('call-x') },
|
||||
content: [{ type: 'tool-result', toolCallId: CallId('call-x'), isError: true }],
|
||||
},
|
||||
error: { code: TOOL_NOT_STARTED },
|
||||
})
|
||||
// The synthetic result carries the SAME callId as the orphaned tool-call,
|
||||
// so deriveMessages() pairs them — no provider-invalid dangling call.
|
||||
const call = loaded.events.findLast(e => e.type === 'assistant/message')
|
||||
const callId = call?.type === 'assistant/message'
|
||||
&& call.data.message.content.find(b => b.type === 'tool-call')
|
||||
expect(callId && callId.type === 'tool-call' && callId.id).toBe(CallId('call-x'))
|
||||
} finally {
|
||||
await dispose()
|
||||
}
|
||||
})
|
||||
|
||||
it('crash recovery: a recorded tool call with no result tells the model to assess retry risk', async () => {
|
||||
const { persistence, dispose } = await make()
|
||||
try {
|
||||
const m = meta('unknown-tool-outcome')
|
||||
await persistence.create(m)
|
||||
await persistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } },
|
||||
{ type: 'step/start', seq: 1, time: 2, data: { turn: 1, step: 1 } },
|
||||
{ type: 'assistant/message', seq: 2, time: 3, data: {
|
||||
turn: 1, step: 1,
|
||||
message: createMessage({
|
||||
role: 'assistant',
|
||||
content: [
|
||||
{ type: 'tool-call', id: CallId('call-risk'), name: 'write', arguments: '{}' },
|
||||
],
|
||||
source: {
|
||||
kind: 'model',
|
||||
...{ provider: 'mock', model: 'mock' },
|
||||
},
|
||||
}),
|
||||
}, surfaceOp: 'append' },
|
||||
{ type: 'tool/call', seq: 3, time: 4, data: { turn: 1, step: 1, callId: CallId('call-risk'), name: 'write', arguments: '{}' } },
|
||||
])
|
||||
|
||||
const loaded = await persistence.load(m.id)
|
||||
const synthetic = loaded.events.find(e => e.type === 'tool/result')
|
||||
expect(synthetic?.type === 'tool/result' && synthetic.data.error).toEqual({
|
||||
name: 'ToolOutcomeUnknownError', code: TOOL_OUTCOME_UNKNOWN,
|
||||
})
|
||||
if (synthetic?.type !== 'tool/result' || synthetic.data.message.content[0].content[0]?.type !== 'text') {
|
||||
throw new Error('expected a text tool result')
|
||||
}
|
||||
expect(synthetic.data.message.content[0].content[0].text).toContain('retry only if the operation is read-only or idempotent')
|
||||
expect(synthetic.data.message.content[0].content[0].text).toContain('if it may have side effects, first verify external state or ask the user')
|
||||
const resumed = Session.create(m.id, loaded.events, loaded.meta)
|
||||
const resumedResult = resumed.deriveMessages().find(message => message.content.some(block => block.type === 'tool-result'))
|
||||
expect(resumedResult?.content[0]).toMatchObject({
|
||||
type: 'tool-result', toolCallId: CallId('call-risk'), isError: true,
|
||||
})
|
||||
} finally {
|
||||
await dispose()
|
||||
}
|
||||
})
|
||||
|
||||
it('list() excludes a created-but-never-appended (zero-event) session', async () => {
|
||||
const { persistence, dispose } = await make()
|
||||
try {
|
||||
await persistence.create(meta('empty'))
|
||||
expect((await persistence.list()).map(m => m.id)).not.toContain(SessionId('empty'))
|
||||
expect((await persistence.listSnapshots()).map(snapshot => snapshot.header.id))
|
||||
.not.toContain(SessionId('empty'))
|
||||
} finally {
|
||||
await dispose()
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects pre-aborted observation reads with the exact cancellation reason', async () => {
|
||||
const { persistence, dispose } = await make()
|
||||
try {
|
||||
const reason = new Error('persistence observation cancelled')
|
||||
const controller = new AbortController()
|
||||
await expect(persistence.listSnapshots(controller.signal)).resolves.toEqual([])
|
||||
controller.abort(reason)
|
||||
|
||||
await expect(persistence.list(controller.signal)).rejects.toBe(reason)
|
||||
await expect(persistence.listSnapshots(controller.signal)).rejects.toBe(reason)
|
||||
await expect(persistence.inspect(SessionId('cancelled-inspect'), controller.signal))
|
||||
.rejects.toBe(reason)
|
||||
await expect(persistence.readFrom(SessionId('cancelled-read-from'), 0, controller.signal))
|
||||
.rejects.toBe(reason)
|
||||
} finally {
|
||||
await dispose()
|
||||
}
|
||||
})
|
||||
|
||||
it('readFrom returns exactly the stored suffix from the requested seq, without mutating the log', async () => {
|
||||
const { persistence, dispose } = await make()
|
||||
try {
|
||||
const m = meta('read-from', '/work')
|
||||
const log = oneTurnLog()
|
||||
await persistence.create(m)
|
||||
await persistence.append(m.id, log)
|
||||
|
||||
const whole = await persistence.readFrom(m.id, 0)
|
||||
expect(whole.meta).toMatchObject({ id: m.id, cwd: '/work' })
|
||||
expect(whole.events).toEqual(log)
|
||||
|
||||
const suffix = await persistence.readFrom(m.id, 3)
|
||||
expect(suffix.events).toEqual(log.slice(3))
|
||||
expect(suffix.events[0]?.seq).toBe(3)
|
||||
|
||||
// At/past the stored end: an empty tail, never an error.
|
||||
await expect(persistence.readFrom(m.id, log.length)).resolves.toMatchObject({ events: [] })
|
||||
await expect(persistence.readFrom(m.id, log.length + 100)).resolves.toMatchObject({ events: [] })
|
||||
|
||||
// Non-mutating: an interrupted-turn log is served as stored, no closers.
|
||||
await persistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2 } },
|
||||
])
|
||||
const tail = await persistence.readFrom(m.id, 6)
|
||||
expect(tail.events.map(event => event.type)).toEqual(['turn/start'])
|
||||
|
||||
await expect(persistence.readFrom(SessionId('absent-read-from'), 0)).rejects.toThrow('not found')
|
||||
await expect(persistence.readFrom(m.id, -1)).rejects.toThrow('non-negative safe integer')
|
||||
await expect(persistence.readFrom(m.id, 1.5)).rejects.toThrow('non-negative safe integer')
|
||||
} finally {
|
||||
await dispose()
|
||||
}
|
||||
})
|
||||
|
||||
it('lists stable lightweight revisions that change after an append', async () => {
|
||||
const { persistence, dispose } = await make()
|
||||
try {
|
||||
const m = meta('s2')
|
||||
await persistence.create(m)
|
||||
await persistence.append(m.id, oneTurnLog())
|
||||
expect((await persistence.list()).map(x => x.id)).toContain(m.id)
|
||||
const first = (await persistence.listSnapshots()).find(snapshot => snapshot.header.id === m.id)
|
||||
const repeated = (await persistence.listSnapshots()).find(snapshot => snapshot.header.id === m.id)
|
||||
expect(first).toBeDefined()
|
||||
expect(repeated?.revision).toBe(first?.revision)
|
||||
|
||||
await persistence.append(m.id, [{
|
||||
type: 'turn/start',
|
||||
seq: 6,
|
||||
time: 7,
|
||||
data: { turn: 2 },
|
||||
}])
|
||||
const changed = (await persistence.listSnapshots()).find(snapshot => snapshot.header.id === m.id)
|
||||
expect(changed?.revision).not.toBe(first?.revision)
|
||||
} finally {
|
||||
await dispose()
|
||||
}
|
||||
})
|
||||
|
||||
it('append rejects a batch whose first seq does not match the stored next-seq', async () => {
|
||||
const { persistence, dispose } = await make()
|
||||
try {
|
||||
const m = meta('s3')
|
||||
await persistence.create(m)
|
||||
await persistence.append(m.id, oneTurnLog()) // seqs 0..5, next-seq = 6
|
||||
// A re-append of an already-stored seq must be rejected, not duplicated.
|
||||
const restated = oneTurnLog()
|
||||
await expect(persistence.append(m.id, restated)).rejects.toThrow()
|
||||
} finally {
|
||||
await dispose()
|
||||
}
|
||||
})
|
||||
|
||||
it('append rejects a mid-batch seq gap', async () => {
|
||||
const { persistence, dispose } = await make()
|
||||
try {
|
||||
const m = meta('s4')
|
||||
await persistence.create(m)
|
||||
const gapped: SessionEvent[] = [
|
||||
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1 } },
|
||||
{ type: 'step/start', seq: 2, time: 2, data: { turn: 1, step: 1 } }, // gap: missing seq 1
|
||||
]
|
||||
await expect(persistence.append(m.id, gapped)).rejects.toThrow()
|
||||
} finally {
|
||||
await dispose()
|
||||
}
|
||||
})
|
||||
|
||||
it('append rejects non-JSON-serializable event data, naming the event type', async () => {
|
||||
const { persistence, dispose } = await make()
|
||||
try {
|
||||
// Every value `isJsonValue` rejects must be rejected by the backend, not just BigInt —
|
||||
// otherwise a backend could pass this contract while still accepting values that
|
||||
// corrupt the durable round-trip. Each value is carried in a plugin-added field on one
|
||||
// user message so the contract covers the complete JSON-value boundary.
|
||||
const cyclic: Record<string, unknown> = { type: 'text', text: 'x' }
|
||||
cyclic['self'] = cyclic
|
||||
const badValues: unknown[] = [
|
||||
1n, // BigInt
|
||||
undefined, // dropped by JSON.stringify
|
||||
Infinity, // → null
|
||||
() => 0, // function
|
||||
Symbol('s'), // symbol
|
||||
new Map(), // exotic object
|
||||
cyclic, // circular ref
|
||||
]
|
||||
for (const [i, bad] of badValues.entries()) {
|
||||
// A fresh session per value isolates each rejection (a rejected append
|
||||
// must leave no state behind, but isolating keeps the assertion clean).
|
||||
const mi = meta(`s5-${i}`)
|
||||
await persistence.create(mi)
|
||||
const events = [
|
||||
{
|
||||
type: 'user/message',
|
||||
seq: 0,
|
||||
time: 1,
|
||||
data: {
|
||||
id: MessageId(`invalid-json-${i}`),
|
||||
role: 'user',
|
||||
content: [{ type: 'text', text: 'x' }],
|
||||
source: { kind: 'user' },
|
||||
extra: bad,
|
||||
},
|
||||
},
|
||||
] as unknown as SessionEvent[]
|
||||
await expect(persistence.append(mi.id, events)).rejects.toThrow(/losslessly JSON-serializable/)
|
||||
}
|
||||
} finally {
|
||||
await dispose()
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
1432
packages/session/session-persistence/tests/coordinator-contract.ts
Normal file
1432
packages/session/session-persistence/tests/coordinator-contract.ts
Normal file
File diff suppressed because it is too large
Load Diff
1915
packages/session/session-persistence/tests/persistence.spec.ts
Normal file
1915
packages/session/session-persistence/tests/persistence.spec.ts
Normal file
File diff suppressed because it is too large
Load Diff
360
packages/session/session-persistence/tests/preparations.spec.ts
Normal file
360
packages/session/session-persistence/tests/preparations.spec.ts
Normal file
@@ -0,0 +1,360 @@
|
||||
/** Unit coverage for unpublished Session preparation ownership and sharing. */
|
||||
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { Session, SessionId } from '@deepseek-ai/dsh-session'
|
||||
import { observeQueuedAbort, SessionPreparations } from '../src/preparations.ts'
|
||||
|
||||
interface PreparedSource {
|
||||
readonly session: Session
|
||||
readonly label: string
|
||||
}
|
||||
|
||||
function prepared(label: string): PreparedSource {
|
||||
return { session: Session.create(SessionId(label)), label }
|
||||
}
|
||||
|
||||
function committed(source: PreparedSource): Promise<{ source: PreparedSource; state: string }> {
|
||||
return Promise.resolve({ source, state: source.label })
|
||||
}
|
||||
|
||||
describe('SessionPreparations inspection', () => {
|
||||
it('shares in-flight and ready sources, then invalidates them', async () => {
|
||||
const preparations = new SessionPreparations<PreparedSource, string>(2)
|
||||
const id = SessionId('shared-inspection')
|
||||
const gate = Promise.withResolvers<PreparedSource>()
|
||||
const load = vi.fn(() => gate.promise)
|
||||
const first = preparations.inspect(id, load)
|
||||
const second = preparations.inspect(id, load, new AbortController().signal)
|
||||
const source = prepared(id)
|
||||
|
||||
expect(preparations.has(id)).toBe(true)
|
||||
gate.resolve(source)
|
||||
await expect(first).resolves.toBe(source)
|
||||
await expect(second).resolves.toBe(source)
|
||||
await expect(preparations.inspect(id, load)).resolves.toBe(source)
|
||||
expect(load).toHaveBeenCalledOnce()
|
||||
|
||||
preparations.invalidate(id)
|
||||
preparations.invalidate(id)
|
||||
expect(preparations.has(id)).toBe(false)
|
||||
})
|
||||
|
||||
it('keeps a shared load alive when its first observer cancels', async () => {
|
||||
const preparations = new SessionPreparations<PreparedSource, string>(1)
|
||||
const id = SessionId('cancelled-first-observer')
|
||||
const gate = Promise.withResolvers<PreparedSource>()
|
||||
const load = vi.fn(() => gate.promise)
|
||||
const controller = new AbortController()
|
||||
const reason = new Error('first observer cancelled')
|
||||
const first = preparations.inspect(id, load, controller.signal)
|
||||
const joined = preparations.inspect(id, load)
|
||||
|
||||
controller.abort(reason)
|
||||
await expect(first).rejects.toBe(reason)
|
||||
const source = prepared(id)
|
||||
gate.resolve(source)
|
||||
await expect(joined).resolves.toBe(source)
|
||||
await expect(preparations.inspect(id, load)).resolves.toBe(source)
|
||||
expect(load).toHaveBeenCalledOnce()
|
||||
})
|
||||
|
||||
it('evicts completed loads whose observers cancelled before readiness', async () => {
|
||||
const preparations = new SessionPreparations<PreparedSource, string>(1)
|
||||
const firstId = SessionId('cancelled-ready-first')
|
||||
const secondId = SessionId('cancelled-ready-second')
|
||||
const firstGate = Promise.withResolvers<PreparedSource>()
|
||||
const secondGate = Promise.withResolvers<PreparedSource>()
|
||||
const firstController = new AbortController()
|
||||
const secondController = new AbortController()
|
||||
const first = preparations.inspect(firstId, () => firstGate.promise, firstController.signal)
|
||||
const second = preparations.inspect(secondId, () => secondGate.promise, secondController.signal)
|
||||
|
||||
firstController.abort(new Error('first observer cancelled'))
|
||||
secondController.abort(new Error('second observer cancelled'))
|
||||
await expect(first).rejects.toThrow('first observer cancelled')
|
||||
await expect(second).rejects.toThrow('second observer cancelled')
|
||||
|
||||
firstGate.resolve(prepared(firstId))
|
||||
await firstGate.promise
|
||||
secondGate.resolve(prepared(secondId))
|
||||
await secondGate.promise
|
||||
await Promise.resolve()
|
||||
|
||||
expect(preparations.has(firstId)).toBe(false)
|
||||
expect(preparations.has(secondId)).toBe(true)
|
||||
})
|
||||
|
||||
it('removes failed and invalidated in-flight loads without changing their observers', async () => {
|
||||
const preparations = new SessionPreparations<PreparedSource, string>(1)
|
||||
const failedId = SessionId('failed-inspection')
|
||||
const failure = new Error('load failed')
|
||||
await expect(preparations.inspect(failedId, () => Promise.reject(failure))).rejects.toBe(failure)
|
||||
expect(preparations.has(failedId)).toBe(false)
|
||||
|
||||
const invalidatedId = SessionId('invalidated-inspection')
|
||||
const gate = Promise.withResolvers<PreparedSource>()
|
||||
const inspection = preparations.inspect(invalidatedId, () => gate.promise)
|
||||
preparations.invalidate(invalidatedId)
|
||||
const source = prepared(invalidatedId)
|
||||
gate.resolve(source)
|
||||
await expect(inspection).resolves.toBe(source)
|
||||
expect(preparations.has(invalidatedId)).toBe(false)
|
||||
|
||||
const rejectedId = SessionId('invalidated-rejection')
|
||||
const rejectedGate = Promise.withResolvers<PreparedSource>()
|
||||
const rejected = preparations.inspect(rejectedId, () => rejectedGate.promise)
|
||||
preparations.invalidate(rejectedId)
|
||||
rejectedGate.reject(failure)
|
||||
await expect(rejected).rejects.toBe(failure)
|
||||
})
|
||||
|
||||
it('removes a load that throws before returning its promise', async () => {
|
||||
const preparations = new SessionPreparations<PreparedSource, string>(1)
|
||||
const id = SessionId('synchronous-load-failure')
|
||||
const failure = new Error('synchronous load failure')
|
||||
|
||||
await expect(preparations.inspect(id, () => { throw failure })).rejects.toBe(failure)
|
||||
expect(preparations.has(id)).toBe(false)
|
||||
})
|
||||
|
||||
it('evicts ready entries while leaving reserved entries alone', async () => {
|
||||
const preparations = new SessionPreparations<PreparedSource, string>(1)
|
||||
const reservedA = await preparations.reserve(
|
||||
SessionId('reserved-a'),
|
||||
() => Promise.resolve(prepared('reserved-a')),
|
||||
committed,
|
||||
)
|
||||
const reservedB = await preparations.reserve(
|
||||
SessionId('reserved-b'),
|
||||
() => Promise.resolve(prepared('reserved-b')),
|
||||
committed,
|
||||
)
|
||||
expect(reservedA).toBeDefined()
|
||||
expect(reservedB).toBeDefined()
|
||||
|
||||
await preparations.inspect(SessionId('ready-c'), () => Promise.resolve(prepared('ready-c')))
|
||||
preparations.release(reservedA!, true)
|
||||
expect(preparations.has(SessionId('reserved-b'))).toBe(true)
|
||||
expect(preparations.has(SessionId('ready-c'))).toBe(false)
|
||||
expect(preparations.has(SessionId('reserved-a'))).toBe(true)
|
||||
|
||||
preparations.discard(reservedB!)
|
||||
preparations.invalidate(SessionId('reserved-a'))
|
||||
})
|
||||
|
||||
it('discards only the exact ready source and retains exclusive reservations', async () => {
|
||||
const preparations = new SessionPreparations<PreparedSource, string>(1)
|
||||
const ready = prepared('discard-ready')
|
||||
expect(preparations.discardReady(ready.session.id, ready)).toBe('missing')
|
||||
await preparations.inspect(ready.session.id, () => Promise.resolve(ready))
|
||||
expect(preparations.discardReady(ready.session.id, prepared('different'))).toBe('missing')
|
||||
expect(preparations.discardReady(ready.session.id, ready)).toBe('discarded')
|
||||
|
||||
const reserved = await preparations.reserve(
|
||||
ready.session.id,
|
||||
() => Promise.resolve(ready),
|
||||
committed,
|
||||
)
|
||||
expect(preparations.discardReady(ready.session.id, ready)).toBe('retained')
|
||||
preparations.release(reserved!, false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('SessionPreparations reservation', () => {
|
||||
it('waits for an existing reservation, republishes the exact Session, and attaches once', async () => {
|
||||
const preparations = new SessionPreparations<PreparedSource, string>(2)
|
||||
const id = SessionId('reservation-wait')
|
||||
const source = prepared(id)
|
||||
const first = await preparations.reserve(id, () => Promise.resolve(source), committed)
|
||||
expect(first).toBeDefined()
|
||||
expect(preparations.reservationFor(source.session)).toBe(first)
|
||||
expect(() => preparations.reservationFor(Session.create(id))).toThrow(/cannot publish/)
|
||||
expect(() => { preparations.assertWritable(id) }).toThrow(/is reserved/)
|
||||
|
||||
let secondSettled = false
|
||||
const secondPromise = preparations.reserve(id, () => Promise.resolve(prepared('unused')), committed)
|
||||
.then((reservation) => {
|
||||
secondSettled = true
|
||||
return reservation
|
||||
})
|
||||
await Promise.resolve()
|
||||
await Promise.resolve()
|
||||
await Promise.resolve()
|
||||
expect(secondSettled).toBe(false)
|
||||
|
||||
preparations.release(first!, true)
|
||||
const second = await secondPromise
|
||||
expect(second?.source).toBe(source)
|
||||
preparations.attach(second!)
|
||||
expect(preparations.reservationFor(source.session)).toBeUndefined()
|
||||
expect(() => { preparations.attach(second!) }).toThrow(/no longer reserved/)
|
||||
preparations.discard(second!)
|
||||
preparations.release(second!, true)
|
||||
expect(() => { preparations.assertWritable(id) }).not.toThrow()
|
||||
})
|
||||
|
||||
it('supports abortable reservation waits without cancelling the held reservation', async () => {
|
||||
const preparations = new SessionPreparations<PreparedSource, string>(1)
|
||||
const id = SessionId('abortable-reservation-wait')
|
||||
const first = await preparations.reserve(id, () => Promise.resolve(prepared(id)), committed)
|
||||
const controller = new AbortController()
|
||||
const reason = { kind: 'cancelled' }
|
||||
const waiting = preparations.reserve(id, () => Promise.resolve(prepared('unused')), committed, controller.signal)
|
||||
|
||||
await Promise.resolve()
|
||||
await Promise.resolve()
|
||||
await Promise.resolve()
|
||||
controller.abort(reason)
|
||||
await expect(waiting).rejects.toBe(reason)
|
||||
expect(preparations.reservationFor(first!.source.session)).toBe(first)
|
||||
preparations.release(first!, false)
|
||||
expect(preparations.has(id)).toBe(false)
|
||||
})
|
||||
|
||||
it('removes a failed commit and wakes another waiter as invalidated', async () => {
|
||||
const preparations = new SessionPreparations<PreparedSource, string>(1)
|
||||
const id = SessionId('failed-commit')
|
||||
const commitStarted = Promise.withResolvers<undefined>()
|
||||
const commitGate = Promise.withResolvers<{ source: PreparedSource; state: string }>()
|
||||
const source = prepared(id)
|
||||
const failure = new Error('commit failed')
|
||||
const first = preparations.reserve(id, () => Promise.resolve(source), () => {
|
||||
commitStarted.resolve(undefined)
|
||||
return commitGate.promise
|
||||
})
|
||||
await commitStarted.promise
|
||||
expect(() => { preparations.assertWritable(id) }).toThrow(/is reserved/)
|
||||
const second = preparations.reserve(id, () => Promise.resolve(prepared('unused')), committed)
|
||||
|
||||
commitGate.reject(failure)
|
||||
await expect(first).rejects.toBe(failure)
|
||||
await expect(second).resolves.toBeUndefined()
|
||||
expect(preparations.has(id)).toBe(false)
|
||||
})
|
||||
|
||||
it('returns a post-commit cancellation to the ready pool', async () => {
|
||||
const preparations = new SessionPreparations<PreparedSource, string>(1)
|
||||
const id = SessionId('post-commit-cancel')
|
||||
const source = prepared(id)
|
||||
const controller = new AbortController()
|
||||
const reason = new Error('cancel after commit')
|
||||
|
||||
await expect(preparations.reserve(id, () => Promise.resolve(source), async (value) => {
|
||||
controller.abort(reason)
|
||||
return { source: value, state: value.label }
|
||||
}, controller.signal)).rejects.toBe(reason)
|
||||
|
||||
expect(preparations.takeReady(id)).toBe(source)
|
||||
expect(preparations.takeReady(id)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('does not revive an invalidated commit after post-commit cancellation', async () => {
|
||||
const preparations = new SessionPreparations<PreparedSource, string>(1)
|
||||
const id = SessionId('invalidated-commit-cancel')
|
||||
const source = prepared(id)
|
||||
const commitStarted = Promise.withResolvers<undefined>()
|
||||
const commitGate = Promise.withResolvers<undefined>()
|
||||
const controller = new AbortController()
|
||||
const reason = new Error('cancel invalidated commit')
|
||||
const reservation = preparations.reserve(id, () => Promise.resolve(source), async (value) => {
|
||||
commitStarted.resolve(undefined)
|
||||
await commitGate.promise
|
||||
return { source: value, state: value.label }
|
||||
}, controller.signal)
|
||||
|
||||
await commitStarted.promise
|
||||
preparations.invalidate(id)
|
||||
controller.abort(reason)
|
||||
commitGate.resolve(undefined)
|
||||
await expect(reservation).rejects.toBe(reason)
|
||||
expect(preparations.has(id)).toBe(false)
|
||||
})
|
||||
|
||||
it('does not reserve an entry invalidated while its commit succeeds', async () => {
|
||||
const preparations = new SessionPreparations<PreparedSource, string>(1)
|
||||
const id = SessionId('invalidated-successful-commit')
|
||||
const source = prepared(id)
|
||||
const commitStarted = Promise.withResolvers<undefined>()
|
||||
const commitGate = Promise.withResolvers<undefined>()
|
||||
const reservation = preparations.reserve(id, () => Promise.resolve(source), async (value) => {
|
||||
commitStarted.resolve(undefined)
|
||||
await commitGate.promise
|
||||
return { source: value, state: value.label }
|
||||
})
|
||||
|
||||
await commitStarted.promise
|
||||
preparations.invalidate(id)
|
||||
commitGate.resolve(undefined)
|
||||
|
||||
await expect(reservation).resolves.toBeUndefined()
|
||||
expect(preparations.has(id)).toBe(false)
|
||||
})
|
||||
|
||||
it('returns undefined when a load is invalidated before reservation', async () => {
|
||||
const preparations = new SessionPreparations<PreparedSource, string>(1)
|
||||
const id = SessionId('invalidated-reservation')
|
||||
const gate = Promise.withResolvers<PreparedSource>()
|
||||
const reservation = preparations.reserve(id, () => gate.promise, committed)
|
||||
preparations.invalidate(id)
|
||||
gate.resolve(prepared(id))
|
||||
await expect(reservation).resolves.toBeUndefined()
|
||||
})
|
||||
|
||||
it('skips pending adoption and accepts a ready source exactly once', async () => {
|
||||
const preparations = new SessionPreparations<PreparedSource, string>(1)
|
||||
const id = SessionId('take-ready')
|
||||
const gate = Promise.withResolvers<PreparedSource>()
|
||||
const inspection = preparations.inspect(id, () => gate.promise)
|
||||
expect(preparations.takeReady(id)).toBeUndefined()
|
||||
const source = prepared(id)
|
||||
gate.resolve(source)
|
||||
await inspection
|
||||
expect(preparations.takeReady(id)).toBe(source)
|
||||
expect(preparations.takeReady(id)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('rejects publication while only an inspection exists', async () => {
|
||||
const preparations = new SessionPreparations<PreparedSource, string>(1)
|
||||
const source = prepared('inspection-publication')
|
||||
await preparations.inspect(source.session.id, () => Promise.resolve(source))
|
||||
expect(() => preparations.reservationFor(source.session)).toThrow(/cannot publish/)
|
||||
})
|
||||
})
|
||||
|
||||
describe('observeQueuedAbort', () => {
|
||||
it('relays fulfillment and rejection exactly', async () => {
|
||||
const signal = new AbortController().signal
|
||||
await expect(observeQueuedAbort(Promise.resolve('value'), signal)).resolves.toBe('value')
|
||||
const failure = { kind: 'failed' }
|
||||
const rejected = Promise.withResolvers<never>()
|
||||
rejected.reject(failure)
|
||||
await expect(observeQueuedAbort(rejected.promise, signal)).rejects.toBe(failure)
|
||||
})
|
||||
|
||||
it('rejects promptly with an exact abort reason and ignores later settlement', async () => {
|
||||
const operation = Promise.withResolvers<string>()
|
||||
const controller = new AbortController()
|
||||
const reason = { kind: 'aborted' }
|
||||
const observed = observeQueuedAbort(operation.promise, controller.signal)
|
||||
controller.abort(reason)
|
||||
await expect(observed).rejects.toBe(reason)
|
||||
operation.resolve('late')
|
||||
await Promise.resolve()
|
||||
})
|
||||
|
||||
it('observes a pre-aborted signal through the default start predicate', async () => {
|
||||
const controller = new AbortController()
|
||||
controller.abort('pre-aborted')
|
||||
await expect(observeQueuedAbort(new Promise<never>(() => {}), controller.signal))
|
||||
.rejects.toBe('pre-aborted')
|
||||
})
|
||||
|
||||
it('lets an operation that already started own cancellation settlement', async () => {
|
||||
const operation = Promise.withResolvers<string>()
|
||||
const controller = new AbortController()
|
||||
const observed = observeQueuedAbort(operation.promise, controller.signal, () => true)
|
||||
controller.abort(new Error('too late'))
|
||||
operation.resolve('owned')
|
||||
await expect(observed).resolves.toBe('owned')
|
||||
})
|
||||
})
|
||||
275
packages/session/session-persistence/tests/write-behind.spec.ts
Normal file
275
packages/session/session-persistence/tests/write-behind.spec.ts
Normal file
@@ -0,0 +1,275 @@
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import { SessionWriteBehind } from '../src/write-behind.ts'
|
||||
|
||||
/** Minimal ordered event fixture; batching does not interpret event vocabulary. */
|
||||
function event(seq: number): SessionEvent<'turn/start'> {
|
||||
return {
|
||||
type: 'turn/start',
|
||||
seq,
|
||||
time: seq,
|
||||
data: { turn: seq + 1 },
|
||||
}
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
vi.useRealTimers()
|
||||
})
|
||||
|
||||
describe('SessionWriteBehind', () => {
|
||||
it('uses one fixed window from the first queued event and owns its copy', async () => {
|
||||
vi.useFakeTimers()
|
||||
const batches: SessionEvent[][] = []
|
||||
const controller = new SessionWriteBehind({
|
||||
maxDelayMs: 200,
|
||||
write: async (events) => { batches.push(structuredClone(events) as SessionEvent[]) },
|
||||
reportBackgroundFailure: vi.fn(),
|
||||
})
|
||||
const first = event(0)
|
||||
|
||||
controller.enqueue(first)
|
||||
first.data.turn = 99
|
||||
await vi.advanceTimersByTimeAsync(150)
|
||||
controller.enqueue(event(1))
|
||||
await vi.advanceTimersByTimeAsync(49)
|
||||
expect(batches).toEqual([])
|
||||
|
||||
await vi.advanceTimersByTimeAsync(1)
|
||||
expect(batches).toEqual([[
|
||||
expect.objectContaining({ seq: 0, data: { turn: 1 } }),
|
||||
expect.objectContaining({ seq: 1 }),
|
||||
]])
|
||||
expect(controller.hasWork).toBe(false)
|
||||
})
|
||||
|
||||
it('coalesces twenty events admitted ten milliseconds apart into one 200 ms batch', async () => {
|
||||
vi.useFakeTimers()
|
||||
const batches: number[][] = []
|
||||
const controller = new SessionWriteBehind({
|
||||
maxDelayMs: 200,
|
||||
write: async (events) => { batches.push(events.map(item => item.seq)) },
|
||||
reportBackgroundFailure: vi.fn(),
|
||||
})
|
||||
|
||||
controller.enqueue(event(0))
|
||||
for (let seq = 1; seq < 20; seq += 1) {
|
||||
await vi.advanceTimersByTimeAsync(10)
|
||||
controller.enqueue(event(seq))
|
||||
}
|
||||
expect(batches).toEqual([])
|
||||
|
||||
await vi.advanceTimersByTimeAsync(10)
|
||||
expect(batches).toEqual([Array.from({ length: 20 }, (_, seq) => seq)])
|
||||
await controller.flush()
|
||||
})
|
||||
|
||||
it('makes concurrent flushes one immediate barrier that drains admitted tails', async () => {
|
||||
vi.useFakeTimers()
|
||||
const gate = Promise.withResolvers<boolean>()
|
||||
const batches: number[][] = []
|
||||
const controller = new SessionWriteBehind({
|
||||
maxDelayMs: 200,
|
||||
write: async (events) => {
|
||||
batches.push(events.map(item => item.seq))
|
||||
if (batches.length === 1) await gate.promise
|
||||
},
|
||||
reportBackgroundFailure: vi.fn(),
|
||||
})
|
||||
|
||||
controller.enqueue(event(0))
|
||||
const first = controller.flush()
|
||||
const second = controller.flush()
|
||||
expect(second).toBe(first)
|
||||
await Promise.resolve()
|
||||
expect(batches).toEqual([[0]])
|
||||
|
||||
controller.enqueue(event(1))
|
||||
gate.resolve(true)
|
||||
await first
|
||||
expect(batches).toEqual([[0], [1]])
|
||||
expect(controller.hasWork).toBe(false)
|
||||
expect(vi.getTimerCount()).toBe(0)
|
||||
})
|
||||
|
||||
it('starts a new window for work admitted after an already-quiescent barrier', async () => {
|
||||
vi.useFakeTimers()
|
||||
const batches: number[][] = []
|
||||
const controller = new SessionWriteBehind({
|
||||
maxDelayMs: 200,
|
||||
write: async (events) => { batches.push(events.map(item => item.seq)) },
|
||||
reportBackgroundFailure: vi.fn(),
|
||||
})
|
||||
|
||||
const barrier = controller.flush()
|
||||
controller.enqueue(event(0))
|
||||
await barrier
|
||||
expect(batches).toEqual([])
|
||||
expect(vi.getTimerCount()).toBe(1)
|
||||
|
||||
await vi.advanceTimersByTimeAsync(200)
|
||||
expect(batches).toEqual([[0]])
|
||||
expect(controller.hasWork).toBe(false)
|
||||
})
|
||||
|
||||
it('starts an over-budget tail immediately after the active write', async () => {
|
||||
vi.useFakeTimers()
|
||||
const gate = Promise.withResolvers<boolean>()
|
||||
const batches: number[][] = []
|
||||
const controller = new SessionWriteBehind({
|
||||
maxDelayMs: 200,
|
||||
write: async (events) => {
|
||||
batches.push(events.map(item => item.seq))
|
||||
if (batches.length === 1) await gate.promise
|
||||
},
|
||||
reportBackgroundFailure: vi.fn(),
|
||||
})
|
||||
|
||||
controller.enqueue(event(0))
|
||||
await vi.advanceTimersByTimeAsync(200)
|
||||
expect(batches).toEqual([[0]])
|
||||
controller.enqueue(event(1))
|
||||
await vi.advanceTimersByTimeAsync(200)
|
||||
expect(batches).toEqual([[0]])
|
||||
|
||||
gate.resolve(true)
|
||||
await vi.advanceTimersByTimeAsync(0)
|
||||
expect(batches).toEqual([[0], [1]])
|
||||
await controller.flush()
|
||||
})
|
||||
|
||||
it('keeps a tail deadline that has not expired when the active write finishes', async () => {
|
||||
vi.useFakeTimers()
|
||||
const gate = Promise.withResolvers<boolean>()
|
||||
const batches: number[][] = []
|
||||
const controller = new SessionWriteBehind({
|
||||
maxDelayMs: 200,
|
||||
write: async (events) => {
|
||||
batches.push(events.map(item => item.seq))
|
||||
if (batches.length === 1) await gate.promise
|
||||
},
|
||||
reportBackgroundFailure: vi.fn(),
|
||||
})
|
||||
|
||||
controller.enqueue(event(0))
|
||||
await vi.advanceTimersByTimeAsync(200)
|
||||
controller.enqueue(event(1))
|
||||
await vi.advanceTimersByTimeAsync(50)
|
||||
gate.resolve(true)
|
||||
await vi.advanceTimersByTimeAsync(0)
|
||||
expect(batches).toEqual([[0]])
|
||||
|
||||
await vi.advanceTimersByTimeAsync(149)
|
||||
expect(batches).toEqual([[0]])
|
||||
await vi.advanceTimersByTimeAsync(1)
|
||||
expect(batches).toEqual([[0], [1]])
|
||||
await controller.flush()
|
||||
})
|
||||
|
||||
it('pauses automatic retries after failure and preserves order for new work', async () => {
|
||||
vi.useFakeTimers()
|
||||
const failure = new Error('storage unavailable')
|
||||
const report = vi.fn()
|
||||
const batches: number[][] = []
|
||||
let attempt = 0
|
||||
const controller = new SessionWriteBehind({
|
||||
maxDelayMs: 200,
|
||||
write: async (events) => {
|
||||
batches.push(events.map(item => item.seq))
|
||||
if (++attempt === 1) throw failure
|
||||
},
|
||||
reportBackgroundFailure: report,
|
||||
})
|
||||
|
||||
controller.enqueue(event(0))
|
||||
await vi.advanceTimersByTimeAsync(200)
|
||||
expect(report).toHaveBeenCalledWith(failure)
|
||||
expect(controller.hasWork).toBe(true)
|
||||
await vi.advanceTimersByTimeAsync(1_000)
|
||||
expect(batches).toEqual([[0]])
|
||||
|
||||
controller.enqueue(event(1))
|
||||
await vi.advanceTimersByTimeAsync(199)
|
||||
expect(batches).toEqual([[0]])
|
||||
await vi.advanceTimersByTimeAsync(1)
|
||||
expect(batches).toEqual([[0], [0, 1]])
|
||||
await controller.flush()
|
||||
})
|
||||
|
||||
it('observes an overlapping background failure and retries it inside flush', async () => {
|
||||
vi.useFakeTimers()
|
||||
const gate = Promise.withResolvers<boolean>()
|
||||
const report = vi.fn()
|
||||
const batches: number[][] = []
|
||||
const controller = new SessionWriteBehind({
|
||||
maxDelayMs: 200,
|
||||
write: async (events) => {
|
||||
batches.push(events.map(item => item.seq))
|
||||
if (batches.length === 1) {
|
||||
await gate.promise
|
||||
throw new Error('transient')
|
||||
}
|
||||
},
|
||||
reportBackgroundFailure: report,
|
||||
})
|
||||
|
||||
controller.enqueue(event(0))
|
||||
await vi.advanceTimersByTimeAsync(200)
|
||||
const first = controller.flush()
|
||||
const second = controller.flush()
|
||||
gate.resolve(true)
|
||||
|
||||
await expect(Promise.all([first, second])).resolves.toEqual([undefined, undefined])
|
||||
expect(batches).toEqual([[0], [0]])
|
||||
expect(report).toHaveBeenCalledOnce()
|
||||
expect(controller.hasWork).toBe(false)
|
||||
})
|
||||
|
||||
it('surfaces a barrier failure without detached logging and retains its batch', async () => {
|
||||
vi.useFakeTimers()
|
||||
const failure = new Error('durability failed')
|
||||
const report = vi.fn()
|
||||
const batches: number[][] = []
|
||||
let attempt = 0
|
||||
const controller = new SessionWriteBehind({
|
||||
maxDelayMs: 200,
|
||||
write: async (events) => {
|
||||
batches.push(events.map(item => item.seq))
|
||||
if (++attempt === 1) throw failure
|
||||
},
|
||||
reportBackgroundFailure: report,
|
||||
})
|
||||
|
||||
controller.enqueue(event(0))
|
||||
await expect(controller.flush()).rejects.toBe(failure)
|
||||
expect(report).not.toHaveBeenCalled()
|
||||
expect(controller.hasWork).toBe(true)
|
||||
|
||||
controller.enqueue(event(1))
|
||||
await vi.advanceTimersByTimeAsync(200)
|
||||
expect(batches).toEqual([[0], [0, 1]])
|
||||
await controller.flush()
|
||||
})
|
||||
|
||||
it('retains a failed batch larger than the engine call-argument limit', async () => {
|
||||
const failure = new Error('durability failed')
|
||||
const batchSize = 150_000
|
||||
const sizes: number[] = []
|
||||
let attempt = 0
|
||||
const controller = new SessionWriteBehind({
|
||||
maxDelayMs: 200,
|
||||
write: async (events) => {
|
||||
sizes.push(events.length)
|
||||
if (++attempt === 1) throw failure
|
||||
},
|
||||
reportBackgroundFailure: vi.fn(),
|
||||
})
|
||||
|
||||
for (let seq = 0; seq < batchSize; seq += 1) controller.enqueue(event(seq))
|
||||
await expect(controller.flush()).rejects.toBe(failure)
|
||||
expect(controller.hasWork).toBe(true)
|
||||
|
||||
await controller.flush()
|
||||
expect(sizes).toEqual([batchSize, batchSize])
|
||||
expect(controller.hasWork).toBe(false)
|
||||
})
|
||||
})
|
||||
27
packages/session/session-persistence/tsconfig.json
Normal file
27
packages/session/session-persistence/tsconfig.json
Normal file
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../util/brand"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
{
|
||||
"path": "../../support/invariants"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/session-projection/session-projection-cache/README.md
|
||||
README.md: 5d4ad07fab6648acdb40c6aa86d32cc78b4c016e
|
||||
README.zh.md: 827480658b8c69647380a782c1a983b390380108
|
||||
62
packages/session/session-projection-cache/README.md
Normal file
62
packages/session/session-projection-cache/README.md
Normal file
@@ -0,0 +1,62 @@
|
||||
# @deepseek-ai/dsh-session-projection-cache
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
The persisted projection cache (`ctx.sessionProjectionCache`): durable checkpoints of every registered projection unit's state, one record per session on the domain data form (`session_projcache` domain — the shipped json backend lands it beside `workspace.json` under the configured storage root). Design authority: the [session-projection RFC](../../../.agents/notes/proposed/architecture/2026-07-27-session-projection-and-command-log.md) (persisted projection cache section).
|
||||
|
||||
A stored row `(key → {ver, seq, val})` is a fold shortcut, never an authority: possibly stale (`seq` says exactly how stale) but never wrong. Consequences the implementation commits to:
|
||||
|
||||
- **Every background write is fail-soft.** A failed durable write logs a warning and keeps the cache stale; the next write or cold read self-heals. A crash between writes costs a longer tail replay, never a wrong value.
|
||||
- **A `ver` mismatch against the live unit's `stateVersion` discards, never migrates.** A unit bump invalidates its rows at read time; the key refolds from the log.
|
||||
- **Whole-record writes.** Each write replaces the session's full checkpoint (the registry cut is always complete), snapshotted through the lossless-JSON boundary — a unit state violating the plain-JSON contract fails loud.
|
||||
- **Records are bound to a log lifecycle, not just an id.** Each record stores the header identity (`createdAt`, `cwd`) it was folded from; every read validates it (the live or stored header is the witness) before accepting a row, so a deleted-then-recreated id or a persistence store swapped under a surviving cache discards the unrelated record instead of seeding phantom values.
|
||||
- **The log leads, the cache follows.** A live checkpoint flushes the session's buffered events durably BEFORE the cache row lands, so a crash can leave the cache behind the log (a longer tail replay) but never ahead of it.
|
||||
|
||||
## Write policy
|
||||
|
||||
Two mandatory points, throttled in between:
|
||||
|
||||
| Trigger | Nature |
|
||||
|---|---|
|
||||
| `turn/end` | Mandatory — the turn-final value is what cold reads want. |
|
||||
| Session disposal (detach) | Mandatory — the live-to-cold moment; after it the cold ladder serves this session. |
|
||||
| `writeEveryEvents` committed events | Config throttle (count). |
|
||||
| `writeIntervalMs` since the first dirty event | Config throttle (interval). |
|
||||
|
||||
Both `Config` fields are required (no defaults): flush cadence is a deployment choice with no universally correct value, stated in cordis.yml.
|
||||
|
||||
## Listing read (`cachedSnapshot(meta)`)
|
||||
|
||||
The zero-I/O rung: whole values viewed straight from the identity-matching stored record (version-matching keys only), returned as a `{asOfSeq, values}` cut — `asOfSeq` is the lowest served-row watermark, so a client seeding its per-session value store under higher-seq-wins can never let a stale list block overwrite a newer push frame. `undefined` when no usable record exists (unknown id, unrelated lifecycle, or no version-matching rows); the api-proxy list carrier turns that into an absent column.
|
||||
|
||||
## Cold read (`coldSnapshot(id, signal?)`)
|
||||
|
||||
The read ladder, zero full-log load on the happy path: cached rows → `sessionProjections.restoreFloor` (anchored one event below the lowest usable watermark) → persistence `readFrom(id, floor)` → `sessionProjections.restore` → fail-soft write-back of the refreshed rows. The anchor makes a shrunk log (crash-repair truncation) provable: an overreaching row triggers exactly one full re-read from seq 0 instead of serving a ghost value. No registered units serve `{asOfSeq: -1, values: {}}` without touching persistence; a session with no persisted log rejects with the seam's `not found`.
|
||||
|
||||
`write(session)` is the synchronous-cut checkpoint both mandatory points use; carriers may call it directly (not fail-soft — the fail-soft wrappers own containment).
|
||||
|
||||
## Composition
|
||||
|
||||
```yaml
|
||||
- id: session-projection-cache
|
||||
name: '@deepseek-ai/dsh-session-projection-cache'
|
||||
config:
|
||||
writeEveryEvents: 200
|
||||
writeIntervalMs: 5000
|
||||
```
|
||||
|
||||
Injects `storageDomain`, `sessionProjections`, `sessionPersistence`, `sessions`. Without this row the projection system runs live-only (watermark cache; cold reads fall back to full log loads wherever a carrier implements them).
|
||||
|
||||
## Model Experience
|
||||
|
||||
None, as the cache only persists and restores host-side read models of already-logged session state and touches no prompt, message, schema, stream, or tool result.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
None; the cache never assembles or sends provider requests.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **No eviction or retention surface** — records accumulate per session; pruning stored checkpoints is out-of-band maintenance, same stance as session persistence itself.
|
||||
- **Interval throttle is per-session coarse** — the timer arms at the first dirty event after a clean write; a steady sub-threshold trickle writes once per interval, not a sliding window.
|
||||
- **`coldSnapshot` reads are not deduplicated** — two concurrent cold reads of one session each run the ladder; last write-back wins (rows are equivalent), acceptable for listing-scale call rates.
|
||||
62
packages/session/session-projection-cache/README.zh.md
Normal file
62
packages/session/session-projection-cache/README.zh.md
Normal file
@@ -0,0 +1,62 @@
|
||||
# @deepseek-ai/dsh-session-projection-cache
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
持久投影缓存（`ctx.sessionProjectionCache`）：把每个已注册投影单元的状态持久化为检查点，基于域数据形态（domain data form）每会话一条记录（`session_projcache` 域——出厂 JSON 后端将其落在配置的存储根目录下、`workspace.json` 旁边）。设计权威：[session-projection RFC](../../../.agents/notes/proposed/architecture/2026-07-27-session-projection-and-command-log.md)（persisted projection cache 一节）。
|
||||
|
||||
一条存储行 `(key → {ver, seq, val})` 是折叠捷径，绝不是权威：可能陈旧（`seq` 精确说明陈旧到哪），但绝不会错。实现据此承诺：
|
||||
|
||||
- **每次后台写入都 fail-soft。** 持久写失败只记一条警告并保持缓存陈旧；下一次写入或冷读自愈。两次写之间崩溃的代价是更长的尾部回放，绝不是错误的值。
|
||||
- **`ver` 与活单元 `stateVersion` 不匹配即丢弃，绝不迁移。** 单元递增版本会在读取时使其行失效；该 key 从日志重新折叠。
|
||||
- **整记录写入。** 每次写入替换该会话的完整检查点（注册表切面始终是完整的），并经无损 JSON 边界快照——违反纯 JSON 契约的单元状态会大声失败。
|
||||
- **记录绑定到日志生命周期，而不只是 id。** 每条记录存储其折叠来源的 header 身份（`createdAt`、`cwd`）；每次读取先以活 header 或存储 header 为证验证它，再接受任何行——被删后重建的 id、或缓存幸存而持久化存储被换掉时，无关记录被整体丢弃，绝不播种幻影值。
|
||||
- **日志领先，缓存跟随。** 活会话检查点先把缓冲事件持久 flush，缓存行才落地，因此崩溃只会让缓存落后于日志（更长的尾部回放），绝不领先于它。
|
||||
|
||||
## 写策略
|
||||
|
||||
两个必写点，其间节流：
|
||||
|
||||
| 触发 | 性质 |
|
||||
|---|---|
|
||||
| `turn/end` | 必写——冷读要的正是轮次终值。 |
|
||||
| 会话销毁（detach） | 必写——live 转 cold 的时刻；此后冷读阶梯接管该会话。 |
|
||||
| 累计 `writeEveryEvents` 个已提交事件 | 配置节流（条数）。 |
|
||||
| 距首个脏事件 `writeIntervalMs` 毫秒 | 配置节流（间隔）。 |
|
||||
|
||||
两个 `Config` 字段均必填（无默认值）：写入节奏是部署选择，没有普适正确值，由 cordis.yml 明示。
|
||||
|
||||
## 列表读（`cachedSnapshot(meta)`）
|
||||
|
||||
零 I/O 一档：从身份匹配的存储记录直接 view 全量值（仅版本匹配的 key），以 `{asOfSeq, values}` 切面返回——`asOfSeq` 取所服务行的最低水位，客户端在 higher-seq-wins 规则下播种值存储时，陈旧列表块永远压不过更新的推送帧。无可用记录（未知 id、无关生命周期、无版本匹配行）时返回 `undefined`；api-proxy 列表载体将其转为列缺席。
|
||||
|
||||
## 冷读（`coldSnapshot(id, signal?)`）
|
||||
|
||||
读取阶梯，正常路径无需加载全量日志：缓存行 → `sessionProjections.restoreFloor`（锚定在最低可用水位之前一个事件的位置）→ 持久化 `readFrom(id, floor)` → `sessionProjections.restore` → 刷新行的 fail-soft 写回。这个锚使缩短的日志（崩溃修复截断）可被证明：越界的行恰好触发一次从 seq 0 的全量重读，而不是把幽灵值当现值服务。无已注册单元时直接服务 `{asOfSeq: -1, values: {}}`，不触碰持久化；无持久日志的会话以 seam 的 `not found` 拒绝。
|
||||
|
||||
`write(session)` 是两个必写点共用的同步切面检查点；载体可以直接调用（非 fail-soft——由 fail-soft 包装层负责遏制）。
|
||||
|
||||
## 组合
|
||||
|
||||
```yaml
|
||||
- id: session-projection-cache
|
||||
name: '@deepseek-ai/dsh-session-projection-cache'
|
||||
config:
|
||||
writeEveryEvents: 200
|
||||
writeIntervalMs: 5000
|
||||
```
|
||||
|
||||
注入 `storageDomain`、`sessionProjections`、`sessionPersistence`、`sessions`。没有这一行时，投影系统只跑 live（水位缓存；冷读在实现了它的载体处退回全量日志加载）。
|
||||
|
||||
## 模型体验
|
||||
|
||||
无，因为缓存只持久化并恢复 host 侧的、由已写入日志的会话状态派生的读模型，不触碰任何提示词、消息、schema、流或工具结果。
|
||||
|
||||
#### KV Cache 影响
|
||||
|
||||
无；缓存从不组装或发送提供方请求。
|
||||
|
||||
## 已知局限与延后工作
|
||||
|
||||
- **没有淘汰或保留面**——记录按会话累积；清理存储的检查点是带外维护，与会话持久化本身同一立场。
|
||||
- **间隔节流按会话粗粒度**——计时器在一次干净写入后的首个脏事件时武装；持续的低于阈值的涓流每个间隔写一次，不是滑动窗口。
|
||||
- **`coldSnapshot` 读取不去重**——同一会话的两个并发冷读各跑一遍阶梯；写回最后者胜（行等价），对列表级调用频率可接受。
|
||||
48
packages/session/session-projection-cache/package.json
Normal file
48
packages/session/session-projection-cache/package.json
Normal file
@@ -0,0 +1,48 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-session-projection-cache",
|
||||
"description": "Persisted projection cache (ctx.sessionProjectionCache): durable per-session projection checkpoints over the domain data form, throttled write-behind, and the cold-read ladder (cache row + persistence tail replay)",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./invariant": {
|
||||
"types": "./lib/types/invariant.d.ts",
|
||||
"default": "./lib/invariant.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"schemastery": "^3.18.0",
|
||||
"zod": "^4.4.3"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-invariants": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session-persistence": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session-projection": "^0.0.1",
|
||||
"@deepseek-ai/dsh-storage-domain": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-persistence": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-projection": "workspace:^",
|
||||
"@deepseek-ai/dsh-storage": "workspace:^",
|
||||
"@deepseek-ai/dsh-storage-domain": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
300
packages/session/session-projection-cache/src/index.ts
Normal file
300
packages/session/session-projection-cache/src/index.ts
Normal file
@@ -0,0 +1,300 @@
|
||||
/**
|
||||
* Persisted projection cache (`ctx.sessionProjectionCache`): durable
|
||||
* checkpoints of every registered projection unit's state, one record per
|
||||
* session on the domain data form (`session_projcache` domain — the shipped
|
||||
* json backend lands it beside `workspace.json`). The cache is a fold
|
||||
* shortcut, never an authority: a row is possibly stale (its `seq`
|
||||
* says how stale) but never wrong, so every write path is fail-soft (a lost
|
||||
* write costs a longer tail replay on the next cold read) and a
|
||||
* `ver` mismatch discards the row instead of migrating it. Design
|
||||
* authority: the session-projection RFC
|
||||
* (.agents/notes/proposed/architecture/2026-07-27-session-projection-and-command-log.md).
|
||||
* @module @deepseek-ai/dsh-session-projection-cache
|
||||
*/
|
||||
|
||||
import { Context, Service } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import { snapshotJsonValue } from '@deepseek-ai/dsh-session'
|
||||
import type { Session, SessionEvent, SessionHeader, SessionId } from '@deepseek-ai/dsh-session'
|
||||
// Empty type import: applies the package's cordis Context merge
|
||||
// (`ctx.sessionPersistence`), which this service reads on the cold path.
|
||||
import type {} from '@deepseek-ai/dsh-session-persistence'
|
||||
import type { ProjectionCheckpoint, ProjectionSnapshot } from '@deepseek-ai/dsh-session-projection'
|
||||
import type { KvTable } from '@deepseek-ai/dsh-storage-domain'
|
||||
import { projectionCacheDomainSpec } from './spec.ts'
|
||||
import type { CheckpointIdentity, CheckpointRecord } from './spec.ts'
|
||||
|
||||
export { checkpointIdentity, checkpointRecord, checkpointRow, projectionCacheDomainSpec } from './spec.ts'
|
||||
export type { CheckpointIdentity, CheckpointRecord } from './spec.ts'
|
||||
|
||||
declare module 'cordis' {
|
||||
interface Context {
|
||||
sessionProjectionCache: SessionProjectionCache
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Plugin config. Both throttle triggers are deployment choices with no
|
||||
* universally correct value, so the composition states them explicitly
|
||||
* (cordis.yml); the two mandatory write points (`turn/end` and session
|
||||
* disposal) are policy, not tunables, and always fire.
|
||||
*/
|
||||
export interface Config {
|
||||
/** Committed events per session that force a durable checkpoint write between mandatory points. */
|
||||
writeEveryEvents: number
|
||||
/** Longest time (milliseconds) a dirty checkpoint may stay unwritten between mandatory points. */
|
||||
writeIntervalMs: number
|
||||
}
|
||||
|
||||
export const Config: z<Config> = z.object({
|
||||
writeEveryEvents: z.natural().min(1).required(),
|
||||
writeIntervalMs: z.natural().min(1).required(),
|
||||
})
|
||||
|
||||
/** Per-session write-behind bookkeeping (live sessions only; dropped at retire). */
|
||||
interface DirtyState {
|
||||
/** Committed events since the last durable write. */
|
||||
pending: number
|
||||
/** Interval trigger armed at the first dirty event after a clean write. */
|
||||
timer: ReturnType<typeof setTimeout> | undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* The persisted projection cache service. Opens the `session_projcache`
|
||||
* domain at init, checkpoints live sessions on a throttled write-behind
|
||||
* (count/interval triggers from {@link Config}) plus two mandatory points —
|
||||
* `turn/end` and session disposal (the live-to-cold moment) — and serves the
|
||||
* cold-read ladder: cached row, persistence `readFrom` tail, registry
|
||||
* `restore`, durable write-back. Every durable write is fail-soft: failures
|
||||
* log a warning and the cache self-heals on the next write or cold read.
|
||||
*/
|
||||
export class SessionProjectionCache extends Service {
|
||||
static inject = ['storageDomain', 'sessionProjections', 'sessionPersistence', 'sessions']
|
||||
|
||||
static Config: z<Config> = Config
|
||||
|
||||
private table?: KvTable<SessionId, CheckpointRecord>
|
||||
private readonly dirty = new Map<Session, DirtyState>()
|
||||
|
||||
constructor(ctx: Context, public config: Config) {
|
||||
super(ctx, 'sessionProjectionCache')
|
||||
}
|
||||
|
||||
/** Open the domain and install the write-behind listeners. */
|
||||
protected async [Service.init](): Promise<void> {
|
||||
const domain = await this.ctx.storageDomain.open(projectionCacheDomainSpec)
|
||||
this.ctx.effect(() => () => domain.close(), 'sessionProjectionCache.domainClose')
|
||||
this.table = domain.table('sessions')
|
||||
this.installWritePath()
|
||||
}
|
||||
|
||||
/**
|
||||
* The stored record for one session, accepted only when its bound log
|
||||
* identity matches `expected`. A session id names a slot, not a lifecycle:
|
||||
* a recreated id or a persistence store swapped under a surviving cache
|
||||
* must not let an old record seed state folded from an unrelated log.
|
||||
* Synchronous from the domain's in-memory state.
|
||||
* @param id - the session whose record is read.
|
||||
* @param expected - the log identity the caller holds (live or stored header).
|
||||
* @returns the identity-matching record, or `undefined` (absent or unrelated).
|
||||
*/
|
||||
private recordFor(id: SessionId, expected: CheckpointIdentity): CheckpointRecord | undefined {
|
||||
const record = this.requireTable().get(id)
|
||||
if (record === undefined) return undefined
|
||||
return identityMatches(record.identity, expected) ? record : undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* The zero-I/O listing read: whole values viewed straight from the stored
|
||||
* rows (version-matching keys only), each cut carried with its watermark
|
||||
* so a client value store can seed under its higher-seq-wins rule — as
|
||||
* stale as the last durable checkpoint but never wrong, and never from an
|
||||
* unrelated log (the caller's header is the identity witness). Fresher
|
||||
* paths (the history tail baseline, {@link coldSnapshot}) supersede these
|
||||
* values whenever a session is actually opened.
|
||||
* @param meta - the listed session's header (identity witness; no log read).
|
||||
* @returns the cut (`asOfSeq` = lowest served-row watermark), or
|
||||
* `undefined` when no usable row exists for this lifecycle.
|
||||
*/
|
||||
cachedSnapshot(meta: SessionHeader): ProjectionSnapshot | undefined {
|
||||
const record = this.recordFor(meta.id, identityOf(meta))
|
||||
if (record === undefined) return undefined
|
||||
const values = this.ctx.sessionProjections.viewCheckpoint(record.rows)
|
||||
const keys = Object.keys(values)
|
||||
if (keys.length === 0) return undefined
|
||||
// The block carries ONE cut: the lowest served watermark is the seq every
|
||||
// value is at least current as of (under-claiming is safe under
|
||||
// higher-seq-wins; over-claiming would let a stale value outrank pushes).
|
||||
const asOfSeq = Math.min(...keys.map(key => (record.rows[key] as { seq: number }).seq))
|
||||
return { asOfSeq, values }
|
||||
}
|
||||
|
||||
/**
|
||||
* Durably checkpoint one live session NOW (both mandatory points call
|
||||
* this; tests and carriers may too). The registry cut is snapshotted at
|
||||
* this boundary (states are live references), then the whole record is
|
||||
* replaced. NOT fail-soft — callers on the fail-soft paths contain it.
|
||||
* @param session - the live session to checkpoint.
|
||||
* @returns resolution after durability and event emission.
|
||||
*/
|
||||
async write(session: Session): Promise<void> {
|
||||
const rows = this.ctx.sessionProjections.checkpoint(session)
|
||||
this.markClean(session)
|
||||
// Durability barrier: the checkpoint cut was taken above, so flushing
|
||||
// AFTER it guarantees every event inside the cut is durably logged
|
||||
// before the cache row lands — a crash can leave the cache behind the
|
||||
// log (longer tail replay) but never ahead of it (phantom values folded
|
||||
// from events no stored log contains). At detach the store entry is
|
||||
// already gone; persistence's own retirement drain covers that path and
|
||||
// any residual overreach is caught by the cold read's anchored floor.
|
||||
if (this.ctx.sessions.get(session.id) === session) await this.ctx.sessions.flush(session)
|
||||
await this.put(session.id, identityOf(session.header), rows)
|
||||
}
|
||||
|
||||
/**
|
||||
* Cold-read one persisted session's projections with zero full-log load:
|
||||
* cached rows + a persistence `readFrom` tail from the registry's restore
|
||||
* floor, refolded by the registry and written back (fail-soft) so the next
|
||||
* cold read starts closer. A cache row invalidated by a shrunk log
|
||||
* (crash-repair truncation) triggers one full re-read from seq 0 — the
|
||||
* ladder's slow rung, still no crash. Rejects when the session has no
|
||||
* persisted log (`not found` from the persistence seam).
|
||||
* @param id - the persisted session to read.
|
||||
* @param signal - optional cancellation for the persistence reads.
|
||||
* @returns the snapshot cut at the stored log end.
|
||||
*/
|
||||
async coldSnapshot(id: SessionId, signal?: AbortSignal): Promise<ProjectionSnapshot> {
|
||||
const record = this.requireTable().get(id)
|
||||
const cached = record?.rows ?? {}
|
||||
const floor = this.ctx.sessionProjections.restoreFloor(cached)
|
||||
const persistence = this.ctx.sessionPersistence
|
||||
if (floor === undefined) {
|
||||
// No unit registered: nothing to fold, but the not-found contract must
|
||||
// hold in this topology too — the probe read rejects for an absent log
|
||||
// and dates the empty cut for a present one.
|
||||
const probe = await persistence.readFrom(id, 0, signal)
|
||||
return { asOfSeq: probe.events.at(-1)?.seq ?? -1, values: {} }
|
||||
}
|
||||
let restored: { snapshot: ProjectionSnapshot; checkpoint: ProjectionCheckpoint }
|
||||
const tail = await persistence.readFrom(id, floor, signal)
|
||||
// The tail's stored header is the identity witness: a record bound to a
|
||||
// different lifecycle (recreated id, swapped store) is discarded whole
|
||||
// before any of its rows can seed a fold.
|
||||
const related = record === undefined || identityMatches(record.identity, identityOf(tail.meta))
|
||||
try {
|
||||
if (!related) throw new Error('unrelated log identity')
|
||||
restored = this.ctx.sessionProjections.restore(cached, tail.events, floor)
|
||||
} catch {
|
||||
// The recoverable restore failures: an unrelated record, or a row
|
||||
// overreaching the stored log end (or predating the floor). Both imply
|
||||
// floor > 0 (baseSeq-0 restores never throw and an unrelated record
|
||||
// still carried a usable watermark), so the full log is a fresh read.
|
||||
const whole = await persistence.readFrom(id, 0, signal)
|
||||
restored = this.ctx.sessionProjections.restore({}, whole.events, 0)
|
||||
}
|
||||
await this.putSoft(id, identityOf(tail.meta), restored.checkpoint, 'cold-read write-back')
|
||||
return restored.snapshot
|
||||
}
|
||||
|
||||
// --- write-behind (throttle + mandatory points) ---
|
||||
|
||||
private installWritePath(): void {
|
||||
// Every committed event advances the dirty counter; turn/end is a
|
||||
// mandatory point (the durable value most reads want is the turn-final
|
||||
// one), count/interval throttle the in-turn stream.
|
||||
this.ctx.on('session/event', (session: Session, event: SessionEvent) => {
|
||||
if (event.type === 'turn/end') {
|
||||
void this.flushSoft(session, 'turn/end')
|
||||
return
|
||||
}
|
||||
const state = this.dirty.get(session) ?? { pending: 0, timer: undefined }
|
||||
this.dirty.set(session, state)
|
||||
state.pending += 1
|
||||
if (state.pending >= this.config.writeEveryEvents) {
|
||||
void this.flushSoft(session, 'count threshold')
|
||||
return
|
||||
}
|
||||
state.timer ??= setTimeout(() => {
|
||||
void this.flushSoft(session, 'interval')
|
||||
}, this.config.writeIntervalMs)
|
||||
})
|
||||
|
||||
// Detach (the live-to-cold moment): the second mandatory point. After
|
||||
// this write the cold-read ladder serves the session from the cache.
|
||||
// flushSoft's synchronous prefix reads and resets the dirty state, so
|
||||
// dropping it (timer already cleared by markClean) right after is safe.
|
||||
this.ctx.on('session/disposed', (session: Session) => {
|
||||
void this.flushSoft(session, 'detach')
|
||||
this.markClean(session)
|
||||
this.dirty.delete(session)
|
||||
})
|
||||
|
||||
// Clear pending timers with the plugin (their sessions outlive the cache).
|
||||
this.ctx.effect(() => () => {
|
||||
for (const state of this.dirty.values()) {
|
||||
if (state.timer !== undefined) clearTimeout(state.timer)
|
||||
}
|
||||
this.dirty.clear()
|
||||
}, 'sessionProjectionCache.timers')
|
||||
}
|
||||
|
||||
/**
|
||||
* One fail-soft durable checkpoint. Every caller has work by construction:
|
||||
* the throttle triggers only fire dirty (markClean clears the timer with
|
||||
* the counter) and the two mandatory points write unconditionally.
|
||||
*/
|
||||
private async flushSoft(session: Session, trigger: string): Promise<void> {
|
||||
try {
|
||||
await this.write(session)
|
||||
} catch (error) {
|
||||
this.ctx.logger.warn(`session projection cache: ${trigger} write for "${session.id}" failed (cache stays stale): ${String(error)}`)
|
||||
}
|
||||
}
|
||||
|
||||
/** Reset one session's dirty bookkeeping (its checkpoint is being written). */
|
||||
private markClean(session: Session): void {
|
||||
const state = this.dirty.get(session)
|
||||
if (state === undefined) return
|
||||
state.pending = 0
|
||||
if (state.timer !== undefined) {
|
||||
clearTimeout(state.timer)
|
||||
state.timer = undefined
|
||||
}
|
||||
}
|
||||
|
||||
/** Replace one session's stored record with its log identity and a detached snapshot of `rows`. */
|
||||
private async put(id: SessionId, identity: CheckpointIdentity, rows: ProjectionCheckpoint): Promise<void> {
|
||||
const detached = snapshotJsonValue(rows)
|
||||
if (detached === undefined) {
|
||||
throw new TypeError('projection checkpoint is not losslessly JSON-serializable (a unit state violates the plain-JSON contract)')
|
||||
}
|
||||
await this.requireTable().put(id, { identity, rows: detached as CheckpointRecord['rows'] })
|
||||
}
|
||||
|
||||
/** Fail-soft {@link put}: cache writes must never fail their caller's read or event path. */
|
||||
private async putSoft(id: SessionId, identity: CheckpointIdentity, rows: ProjectionCheckpoint, what: string): Promise<void> {
|
||||
try {
|
||||
await this.put(id, identity, rows)
|
||||
} catch (error) {
|
||||
this.ctx.logger.warn(`session projection cache: ${what} for "${id}" failed (cache stays stale): ${String(error)}`)
|
||||
}
|
||||
}
|
||||
|
||||
private requireTable(): KvTable<SessionId, CheckpointRecord> {
|
||||
/* v8 ignore next -- Service.init assigns the table before the service becomes injectable */
|
||||
if (this.table === undefined) throw new Error('session projection cache is not initialized')
|
||||
return this.table
|
||||
}
|
||||
}
|
||||
|
||||
/** Project a header onto the identity fields a record is bound to. */
|
||||
function identityOf(header: SessionHeader): CheckpointIdentity {
|
||||
return { createdAt: header.createdAt, ...header.cwd === undefined ? {} : { cwd: header.cwd } }
|
||||
}
|
||||
|
||||
/** Whether a stored record's bound identity names the caller's lifecycle. */
|
||||
function identityMatches(stored: CheckpointIdentity, expected: CheckpointIdentity): boolean {
|
||||
return stored.createdAt === expected.createdAt && stored.cwd === expected.cwd
|
||||
}
|
||||
|
||||
export default SessionProjectionCache
|
||||
35
packages/session/session-projection-cache/src/invariant.ts
Normal file
35
packages/session/session-projection-cache/src/invariant.ts
Normal file
@@ -0,0 +1,35 @@
|
||||
/**
|
||||
* Package-owned invariant companion for `@deepseek-ai/dsh-session-projection-cache`.
|
||||
* @module @deepseek-ai/dsh-session-projection-cache/invariant
|
||||
*/
|
||||
|
||||
/* jscpd:ignore-start */
|
||||
import type { Context } from 'cordis'
|
||||
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
|
||||
|
||||
const PACKAGE_NAME = '@deepseek-ai/dsh-session-projection-cache'
|
||||
|
||||
/** Cordis companion plugin name. */
|
||||
export const name = 'session-projection-cache-invariant'
|
||||
/** Service required before the companion can reserve package ownership. */
|
||||
export const inject = ['invariants']
|
||||
|
||||
/**
|
||||
* No runtime invariant: the cache's correctness relation (a stored row equals
|
||||
* the registry fold at its `seq` watermark) is only checkable by re-running the
|
||||
* fold over the persisted log — duplicating the implementation rather than
|
||||
* detecting drift — and its staleness is by design (fail-soft writes). The
|
||||
* durable boundary is already schema-validated by the storage-domain layer
|
||||
* on every reopen, and the read ladder's version/watermark guards are proven
|
||||
* by the package spec.
|
||||
*/
|
||||
const install: InvariantInstaller = () => {}
|
||||
|
||||
/**
|
||||
* Register this package's invariant companion.
|
||||
* @param ctx - Cordis context carrying the invariant service.
|
||||
* @returns the installed registration's disposer after setup succeeds.
|
||||
*/
|
||||
export const apply = (ctx: Context): Promise<() => void> =>
|
||||
Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
|
||||
/* jscpd:ignore-end */
|
||||
71
packages/session/session-projection-cache/src/spec.ts
Normal file
71
packages/session/session-projection-cache/src/spec.ts
Normal file
@@ -0,0 +1,71 @@
|
||||
/**
|
||||
* The session-projcache domain declaration: one `sessions` table keyed by
|
||||
* {@link SessionId}, each record the full projection checkpoint for one
|
||||
* session (`key → {ver, seq, val}` rows). The spec object
|
||||
* is the single source of the domain's identity, version, and record schema;
|
||||
* the storage-domain routing decides the medium (the shipped composition's
|
||||
* json backend lands it at `<root>/session_projcache.json`, beside
|
||||
* `workspace.json`).
|
||||
* @module @deepseek-ai/dsh-session-projection-cache/src/spec
|
||||
*/
|
||||
|
||||
import { z } from 'zod'
|
||||
import { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import { defineDomain, domainTable } from '@deepseek-ai/dsh-storage-domain'
|
||||
|
||||
/**
|
||||
* One persisted checkpoint row (the RFC's `(sessionId, key, ver, seq, val)`
|
||||
* minus the two record keys). `val` is the unit's internal state — plain
|
||||
* JSON by the unit contract; `z.json()` enforces that at the durable
|
||||
* boundary. A row is never wrong, only possibly stale: `seq` says exactly
|
||||
* how stale, and a `ver` mismatch against the live unit's `stateVersion`
|
||||
* discards it at read time (never a migration).
|
||||
*/
|
||||
export const checkpointRow = z.object({
|
||||
ver: z.number().int().nonnegative(),
|
||||
seq: z.number().int().gte(-1),
|
||||
val: z.json(),
|
||||
})
|
||||
|
||||
/**
|
||||
* The stored-log identity a record is bound to: the immutable header fields
|
||||
* that distinguish one session lifecycle from another under the same id. A
|
||||
* session id names a slot, not a lifecycle — a deleted-then-recreated id, or
|
||||
* a persistence root swapped under a surviving cache, would otherwise let an
|
||||
* old row pass every watermark check and seed state folded from an unrelated
|
||||
* log. Reads validate this against the live header (listing) or the stored
|
||||
* header (cold read) before accepting any row.
|
||||
*/
|
||||
export const checkpointIdentity = z.object({
|
||||
createdAt: z.number().int().nonnegative(),
|
||||
cwd: z.string().optional(),
|
||||
})
|
||||
|
||||
/** The identity fields a record is bound to, inferred from {@link checkpointIdentity}. */
|
||||
export type CheckpointIdentity = z.infer<typeof checkpointIdentity>
|
||||
|
||||
/**
|
||||
* One session's stored record: the log identity it was folded from plus its
|
||||
* checkpoint rows keyed by projection key. The whole record is replaced on
|
||||
* every write (whole-value discipline — the registry checkpoint is always
|
||||
* the complete per-session cut).
|
||||
*/
|
||||
export const checkpointRecord = z.object({
|
||||
identity: checkpointIdentity,
|
||||
rows: z.record(z.string(), checkpointRow),
|
||||
})
|
||||
|
||||
/** One stored per-session checkpoint record, inferred from {@link checkpointRecord}. */
|
||||
export type CheckpointRecord = z.infer<typeof checkpointRecord>
|
||||
|
||||
/**
|
||||
* The session-projcache domain spec. Version bumps discard the whole medium
|
||||
* (cache semantics: a stale or unreadable cache costs a longer tail replay,
|
||||
* never a wrong value). v2 added the record's log-identity binding; v3
|
||||
* renamed the row fields to `ver`/`seq`/`val`.
|
||||
*/
|
||||
export const projectionCacheDomainSpec = defineDomain({
|
||||
name: 'session_projcache',
|
||||
version: 3,
|
||||
tables: { sessions: domainTable<SessionId, CheckpointRecord>(checkpointRecord) },
|
||||
})
|
||||
387
packages/session/session-projection-cache/tests/cache.spec.ts
Normal file
387
packages/session/session-projection-cache/tests/cache.spec.ts
Normal file
@@ -0,0 +1,387 @@
|
||||
/**
|
||||
* SessionProjectionCache behavior: mandatory-point writes (turn/end, detach),
|
||||
* count/interval throttling between them, fail-soft durability (a failed
|
||||
* write logs and stays stale, never throws into the event path), and the
|
||||
* cold-read ladder (cached row + readFrom tail + registry restore +
|
||||
* write-back; version bump and shrunk-log rows degrade to a full re-read).
|
||||
*/
|
||||
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import { z } from 'zod'
|
||||
import Storage from '@deepseek-ai/dsh-storage'
|
||||
import { DomainFacility } from '@deepseek-ai/dsh-storage-domain'
|
||||
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type { Session, SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import SessionProjectionRegistry from '@deepseek-ai/dsh-session-projection'
|
||||
import type { ProjectionDefinition } from '@deepseek-ai/dsh-session-projection'
|
||||
import { MemoryMediaPool, MemoryStorageBackend } from '../../../storage/storage-domain/tests/helpers/memory-backend.ts'
|
||||
import SessionProjectionCache from '../src/index.ts'
|
||||
|
||||
declare module '@deepseek-ai/dsh-session-projection/types' {
|
||||
interface SessionProjectionMap {
|
||||
'cache-test/marks': { marks: string[] }
|
||||
}
|
||||
}
|
||||
|
||||
declare module '@deepseek-ai/dsh-session' {
|
||||
interface SessionEventMap {
|
||||
'cache-test/mark': { marks: string[] }
|
||||
}
|
||||
|
||||
interface OutOfBandSessionEventMap {
|
||||
'cache-test/mark': true
|
||||
}
|
||||
}
|
||||
|
||||
type MarksState = { marks: string[] } | null
|
||||
const marksUnit = (stateVersion = 1): ProjectionDefinition<'cache-test/marks', MarksState> => ({
|
||||
key: 'cache-test/marks',
|
||||
schema: z.object({ marks: z.array(z.string()) }),
|
||||
init: () => null,
|
||||
apply: (state, event) => (event.type === 'cache-test/mark' ? (event).data : state),
|
||||
view: state => state ?? { marks: [] },
|
||||
stateVersion,
|
||||
})
|
||||
|
||||
/** A persistence double serving readFrom over a fixed per-id stored log (headers stamp createdAt 0). */
|
||||
function fakePersistence(logs: Map<string, SessionEvent[]>) {
|
||||
const readFrom = vi.fn(async (id: SessionId, fromSeq: number) => {
|
||||
const events = logs.get(String(id))
|
||||
if (events === undefined) throw new Error(`session "${id}" not found`)
|
||||
return {
|
||||
meta: { version: 0, id, createdAt: 0 },
|
||||
events: events.filter(event => event.seq >= fromSeq),
|
||||
}
|
||||
})
|
||||
return { readFrom }
|
||||
}
|
||||
|
||||
/** Header shape for cachedSnapshot calls (fake logs stamp createdAt 0, no cwd). */
|
||||
const headerOf = (id: SessionId, createdAt = 0, cwd?: string) =>
|
||||
({ version: 0, id, createdAt, ...cwd === undefined ? {} : { cwd } })
|
||||
|
||||
interface HarnessOptions {
|
||||
pool?: MemoryMediaPool
|
||||
config?: { writeEveryEvents: number; writeIntervalMs: number }
|
||||
stateVersion?: number
|
||||
logs?: Map<string, SessionEvent[]>
|
||||
}
|
||||
|
||||
const contexts: Context[] = []
|
||||
|
||||
async function harness(options: HarnessOptions = {}) {
|
||||
const pool = options.pool ?? new MemoryMediaPool()
|
||||
const logs = options.logs ?? new Map<string, SessionEvent[]>()
|
||||
const ctx = new Context()
|
||||
contexts.push(ctx)
|
||||
await ctx.plugin(Storage)
|
||||
ctx.storage.backend.register('memory', new MemoryStorageBackend(pool))
|
||||
const facility = new DomainFacility(ctx, { backend: 'memory', routes: {} })
|
||||
ctx.storage.mount('domain', facility)
|
||||
ctx.provide('storageDomain', facility)
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SessionProjectionRegistry)
|
||||
ctx.sessionProjections.register(marksUnit(options.stateVersion))
|
||||
const persistence = fakePersistence(logs)
|
||||
ctx.provide('sessionPersistence', persistence as never)
|
||||
const fiber = await ctx.plugin(SessionProjectionCache, options.config ?? { writeEveryEvents: 100, writeIntervalMs: 60_000 })
|
||||
return { ctx, pool, logs, fiber, persistence, cache: ctx.sessionProjectionCache }
|
||||
}
|
||||
|
||||
const mark = (session: Session, marks: string[]): SessionEvent =>
|
||||
session.append('cache-test/mark', { marks })
|
||||
|
||||
const endTurn = (session: Session): SessionEvent =>
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
|
||||
/** The stored medium record for one session id (undefined = never written). */
|
||||
function storedRecord(pool: MemoryMediaPool, id: Session['id']) {
|
||||
return pool.media.get('session_projcache')?.tables.get('sessions')?.get(String(id)) as
|
||||
{
|
||||
identity: { createdAt: number; cwd?: string }
|
||||
rows: Record<string, { ver: number; seq: number; val: unknown }>
|
||||
} | undefined
|
||||
}
|
||||
|
||||
/** The stored medium rows for one session id (undefined = never written). */
|
||||
function storedRows(pool: MemoryMediaPool, id: Session['id']) {
|
||||
return storedRecord(pool, id)?.rows
|
||||
}
|
||||
|
||||
/** Wait until queued fail-soft writes (event-listener fire-and-forget) drain. */
|
||||
const settle = () => new Promise(resolve => setTimeout(resolve, 0))
|
||||
|
||||
afterEach(async () => {
|
||||
vi.useRealTimers()
|
||||
await Promise.all(contexts.splice(0).map(ctx => ctx.fiber.dispose()))
|
||||
})
|
||||
|
||||
describe('SessionProjectionCache write policy', () => {
|
||||
it('writes a durable checkpoint at turn/end (mandatory point)', async () => {
|
||||
const { ctx, pool } = await harness()
|
||||
const session = ctx.sessions.create(SessionId('turn-end'))
|
||||
mark(session, ['a'])
|
||||
expect(storedRows(pool, session.id)).toBeUndefined() // throttled: no write yet
|
||||
const end = endTurn(session)
|
||||
await settle()
|
||||
const rows = storedRows(pool, session.id)
|
||||
expect(rows?.['cache-test/marks']).toEqual({ ver: 1, seq: end.seq, val: { marks: ['a'] } })
|
||||
})
|
||||
|
||||
it('writes at session disposal (detach, the live-to-cold moment)', async () => {
|
||||
const { ctx, pool } = await harness()
|
||||
// Sessions dispose with their owning fiber: create in a child plugin.
|
||||
let session: Session | undefined
|
||||
const owner = await ctx.plugin(Object.assign((inner: Context) => {
|
||||
session = inner.sessions.create(SessionId('detach'))
|
||||
}, { inject: ['sessions'] }))
|
||||
if (session === undefined) throw new Error('session was not created')
|
||||
mark(session, ['live'])
|
||||
await owner.dispose()
|
||||
await settle()
|
||||
expect(storedRows(pool, session.id)?.['cache-test/marks']?.val).toEqual({ marks: ['live'] })
|
||||
})
|
||||
|
||||
it('flushes when the in-turn event count reaches the configured threshold', async () => {
|
||||
const { ctx, pool } = await harness({ config: { writeEveryEvents: 3, writeIntervalMs: 60_000 } })
|
||||
const session = ctx.sessions.create(SessionId('count'))
|
||||
mark(session, ['1'])
|
||||
mark(session, ['2'])
|
||||
await settle()
|
||||
expect(storedRows(pool, session.id)).toBeUndefined()
|
||||
mark(session, ['3'])
|
||||
await settle()
|
||||
expect(storedRows(pool, session.id)?.['cache-test/marks']?.val).toEqual({ marks: ['3'] })
|
||||
})
|
||||
|
||||
it('flushes on the configured interval when the count threshold is not reached', async () => {
|
||||
vi.useFakeTimers()
|
||||
const { ctx, pool } = await harness({ config: { writeEveryEvents: 100, writeIntervalMs: 250 } })
|
||||
const session = ctx.sessions.create(SessionId('interval'))
|
||||
mark(session, ['slow'])
|
||||
await vi.advanceTimersByTimeAsync(249)
|
||||
expect(storedRows(pool, session.id)).toBeUndefined()
|
||||
await vi.advanceTimersByTimeAsync(1)
|
||||
await vi.advanceTimersByTimeAsync(0)
|
||||
expect(storedRows(pool, session.id)?.['cache-test/marks']?.val).toEqual({ marks: ['slow'] })
|
||||
})
|
||||
|
||||
it('write() on a never-dirty session checkpoints directly and rejects a non-JSON unit state', async () => {
|
||||
const { ctx, pool } = await harness()
|
||||
// Never dirtied: no events — write() still lands the init-derived cut.
|
||||
const clean = ctx.sessions.create(SessionId('clean-write'))
|
||||
await ctx.sessionProjectionCache.write(clean)
|
||||
expect(storedRows(pool, clean.id)?.['cache-test/marks']).toEqual({ ver: 1, seq: -1, val: null })
|
||||
// A unit whose state violates the plain-JSON contract fails the write loud.
|
||||
ctx.sessionProjections.register({
|
||||
key: 'cache-test/marks2' as never,
|
||||
schema: { parse: (value: unknown) => value } as never,
|
||||
init: () => new Map<string, string>(),
|
||||
apply: (state: unknown) => state,
|
||||
view: () => null as never,
|
||||
stateVersion: 1,
|
||||
})
|
||||
await expect(ctx.sessionProjectionCache.write(clean)).rejects.toThrow('not losslessly JSON-serializable')
|
||||
})
|
||||
|
||||
it('plugin disposal clears armed interval timers and leaves cleaned sessions alone', async () => {
|
||||
vi.useFakeTimers()
|
||||
const { ctx, pool, fiber } = await harness({ config: { writeEveryEvents: 100, writeIntervalMs: 5000 } })
|
||||
const armed = ctx.sessions.create(SessionId('armed'))
|
||||
const cleaned = ctx.sessions.create(SessionId('cleaned'))
|
||||
mark(armed, ['pending']) // timer armed, no write yet
|
||||
mark(cleaned, ['done'])
|
||||
endTurn(cleaned) // mandatory write; markClean leaves {pending: 0, timer: undefined} in the map
|
||||
await vi.advanceTimersByTimeAsync(0)
|
||||
await fiber.dispose()
|
||||
// The armed timer died with the plugin: advancing time writes nothing.
|
||||
await vi.advanceTimersByTimeAsync(10_000)
|
||||
expect(storedRows(pool, armed.id)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('contains a durable write failure: logs a warning, event path unharmed, next write self-heals', async () => {
|
||||
const { ctx, pool } = await harness()
|
||||
const warn = vi.spyOn(ctx.logger, 'warn').mockImplementation(() => {})
|
||||
const session = ctx.sessions.create(SessionId('fail-soft'))
|
||||
mark(session, ['x'])
|
||||
pool.failNextWrites = 1
|
||||
endTurn(session)
|
||||
await settle()
|
||||
expect(storedRows(pool, session.id)).toBeUndefined()
|
||||
expect(warn).toHaveBeenCalledWith(expect.stringContaining('turn/end write for "fail-soft" failed'))
|
||||
// Self-heal: the next mandatory point writes the current cut.
|
||||
mark(session, ['y'])
|
||||
endTurn(session)
|
||||
await settle()
|
||||
expect(storedRows(pool, session.id)?.['cache-test/marks']?.val).toEqual({ marks: ['y'] })
|
||||
})
|
||||
})
|
||||
|
||||
describe('SessionProjectionCache cold read', () => {
|
||||
const storedLog = (marks: string[][]): SessionEvent[] => {
|
||||
const events: SessionEvent[] = [
|
||||
{ type: 'turn/start', seq: 0, time: 0, data: { turn: 1 } },
|
||||
]
|
||||
for (const m of marks) {
|
||||
events.push({ type: 'cache-test/mark', seq: events.length, time: events.length, data: { marks: m } })
|
||||
}
|
||||
events.push({ type: 'turn/end', seq: events.length, time: events.length, data: { turn: 1, reason: { kind: 'completed' } } })
|
||||
return events
|
||||
}
|
||||
|
||||
/** Pre-seed the medium with one stored checkpoint record (before the domain opens). */
|
||||
function seedRow(
|
||||
pool: MemoryMediaPool,
|
||||
id: string,
|
||||
row: { ver: number; seq: number; val: unknown },
|
||||
identity: { createdAt: number; cwd?: string } = { createdAt: 0 },
|
||||
): void {
|
||||
pool.versions.set('session_projcache', 3)
|
||||
pool.media.set('session_projcache', {
|
||||
tables: new Map([['sessions', new Map([[id, { identity, rows: { 'cache-test/marks': row } }]])]]),
|
||||
global: null,
|
||||
})
|
||||
}
|
||||
|
||||
it('serves a cold session from the cache row plus a bounded tail read, and writes the refresh back', async () => {
|
||||
const pool = new MemoryMediaPool()
|
||||
const logs = new Map([['cold', storedLog([['a'], ['a', 'b']])]])
|
||||
// A warm-era checkpoint at watermark 1 (only ['a'] folded).
|
||||
seedRow(pool, 'cold', { ver: 1, seq: 1, val: { marks: ['a'] } })
|
||||
const { cache, persistence, pool: samePool } = await harness({ pool, logs })
|
||||
const id = SessionId('cold')
|
||||
const snapshot = await cache.coldSnapshot(id)
|
||||
expect(snapshot.values['cache-test/marks']).toEqual({ marks: ['a', 'b'] })
|
||||
expect(snapshot.asOfSeq).toBe(3)
|
||||
// The tail read was bounded by the anchored floor (watermark 1 -> floor 1), not 0.
|
||||
expect(persistence.readFrom).toHaveBeenCalledWith(id, 1, undefined)
|
||||
// Write-back: the stored row advanced to the served cut.
|
||||
expect(storedRows(samePool, id)?.['cache-test/marks'])
|
||||
.toEqual({ ver: 1, seq: 3, val: { marks: ['a', 'b'] } })
|
||||
})
|
||||
|
||||
it('discards a version-mismatched row and refolds the full log', async () => {
|
||||
const pool = new MemoryMediaPool()
|
||||
const logs = new Map([['bumped', storedLog([['a']])]])
|
||||
seedRow(pool, 'bumped', { ver: 1, seq: 2, val: { marks: ['stale'] } })
|
||||
const { cache, persistence } = await harness({ pool, logs, stateVersion: 2 })
|
||||
const snapshot = await cache.coldSnapshot(SessionId('bumped'))
|
||||
expect(snapshot.values['cache-test/marks']).toEqual({ marks: ['a'] })
|
||||
// Mismatch pulls the floor to 0: one full read, no second pass needed.
|
||||
expect(persistence.readFrom).toHaveBeenCalledTimes(1)
|
||||
expect(persistence.readFrom).toHaveBeenCalledWith(SessionId('bumped'), 0, undefined)
|
||||
})
|
||||
|
||||
it('detects a log shrunk below the row watermark and degrades to one full re-read', async () => {
|
||||
const pool = new MemoryMediaPool()
|
||||
const logs = new Map([['shrunk', storedLog([['a']])]]) // seqs 0..2
|
||||
seedRow(pool, 'shrunk', { ver: 1, seq: 9, val: { marks: ['ghost'] } })
|
||||
const { cache, persistence } = await harness({ pool, logs })
|
||||
const snapshot = await cache.coldSnapshot(SessionId('shrunk'))
|
||||
expect(snapshot.values['cache-test/marks']).toEqual({ marks: ['a'] })
|
||||
expect(snapshot.asOfSeq).toBe(2)
|
||||
// Anchored tail read (floor 9) came back empty -> full re-read from 0.
|
||||
expect(persistence.readFrom).toHaveBeenNthCalledWith(1, SessionId('shrunk'), 9, undefined)
|
||||
expect(persistence.readFrom).toHaveBeenNthCalledWith(2, SessionId('shrunk'), 0, undefined)
|
||||
})
|
||||
|
||||
it('write-back failure is contained: the snapshot is still served', async () => {
|
||||
const pool = new MemoryMediaPool()
|
||||
const logs = new Map([['soft', storedLog([['a']])]])
|
||||
const { ctx, cache } = await harness({ pool, logs })
|
||||
const warn = vi.spyOn(ctx.logger, 'warn').mockImplementation(() => {})
|
||||
pool.failNextWrites = 1
|
||||
const snapshot = await cache.coldSnapshot(SessionId('soft'))
|
||||
expect(snapshot.values['cache-test/marks']).toEqual({ marks: ['a'] })
|
||||
expect(warn).toHaveBeenCalledWith(expect.stringContaining('cold-read write-back for "soft" failed'))
|
||||
})
|
||||
|
||||
it('rejects for a session with no persisted log', async () => {
|
||||
const { cache } = await harness()
|
||||
await expect(cache.coldSnapshot(SessionId('absent'))).rejects.toThrow('not found')
|
||||
})
|
||||
|
||||
it('discards a record bound to a different log lifecycle and refolds from the actual log', async () => {
|
||||
const pool = new MemoryMediaPool()
|
||||
const logs = new Map([['reborn', storedLog([['real']])]]) // stored header stamps createdAt 0
|
||||
// A checkpoint from a PRIOR lifecycle of the same id (different createdAt):
|
||||
// its rows pass every watermark check, but the identity does not match.
|
||||
seedRow(pool, 'reborn', { ver: 1, seq: 2, val: { marks: ['phantom'] } }, { createdAt: 999 })
|
||||
const { cache, pool: samePool } = await harness({ pool, logs })
|
||||
const snapshot = await cache.coldSnapshot(SessionId('reborn'))
|
||||
expect(snapshot.values['cache-test/marks']).toEqual({ marks: ['real'] })
|
||||
// The write-back rebinds the record to the actual log's identity.
|
||||
expect(storedRecord(samePool, SessionId('reborn'))?.identity).toEqual({ createdAt: 0 })
|
||||
})
|
||||
|
||||
it('cachedSnapshot returns undefined when every stored row is version-mismatched', async () => {
|
||||
const pool = new MemoryMediaPool()
|
||||
seedRow(pool, 'all-stale', { ver: 99, seq: 4, val: { marks: ['old'] } })
|
||||
const { cache } = await harness({ pool })
|
||||
expect(cache.cachedSnapshot(headerOf(SessionId('all-stale')))).toBeUndefined()
|
||||
})
|
||||
|
||||
it('binds identity on cwd too: a matching cwd serves, a moved session does not', async () => {
|
||||
const pool = new MemoryMediaPool()
|
||||
seedRow(pool, 'homed', { ver: 1, seq: 2, val: { marks: ['w'] } }, { createdAt: 0, cwd: '/work' })
|
||||
const { cache } = await harness({ pool })
|
||||
const id = SessionId('homed')
|
||||
expect(cache.cachedSnapshot(headerOf(id, 0, '/work'))?.values['cache-test/marks']).toEqual({ marks: ['w'] })
|
||||
expect(cache.cachedSnapshot(headerOf(id, 0, '/elsewhere'))).toBeUndefined()
|
||||
expect(cache.cachedSnapshot(headerOf(id, 0))).toBeUndefined()
|
||||
})
|
||||
|
||||
it('dates an empty stored log at -1 in the zero-units topology', async () => {
|
||||
const pool = new MemoryMediaPool()
|
||||
const logs = new Map([['empty', [] as SessionEvent[]]])
|
||||
const ctx = new Context()
|
||||
contexts.push(ctx)
|
||||
await ctx.plugin(Storage)
|
||||
ctx.storage.backend.register('memory', new MemoryStorageBackend(pool))
|
||||
const facility = new DomainFacility(ctx, { backend: 'memory', routes: {} })
|
||||
ctx.storage.mount('domain', facility)
|
||||
ctx.provide('storageDomain', facility)
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SessionProjectionRegistry)
|
||||
ctx.provide('sessionPersistence', fakePersistence(logs) as never)
|
||||
await ctx.plugin(SessionProjectionCache, { writeEveryEvents: 100, writeIntervalMs: 60_000 })
|
||||
await expect(ctx.sessionProjectionCache.coldSnapshot(SessionId('empty')))
|
||||
.resolves.toEqual({ asOfSeq: -1, values: {} })
|
||||
})
|
||||
|
||||
it('cachedSnapshot serves identity-matching rows with the cut watermark and refuses unrelated ones', async () => {
|
||||
const pool = new MemoryMediaPool()
|
||||
seedRow(pool, 'listed', { ver: 1, seq: 4, val: { marks: ['t'] } })
|
||||
const { cache } = await harness({ pool })
|
||||
const id = SessionId('listed')
|
||||
// Matching header: values plus the watermark the client seeds under.
|
||||
expect(cache.cachedSnapshot(headerOf(id))).toEqual({ asOfSeq: 4, values: { 'cache-test/marks': { marks: ['t'] } } })
|
||||
// A recreated id (different createdAt): the record is unrelated — no block.
|
||||
expect(cache.cachedSnapshot(headerOf(id, 777))).toBeUndefined()
|
||||
// Unknown id: no block.
|
||||
expect(cache.cachedSnapshot(headerOf(SessionId('never-cached')))).toBeUndefined()
|
||||
})
|
||||
|
||||
it('holds the not-found contract with zero registered units, and dates the empty cut for a present log', async () => {
|
||||
// Same composition minus any registered unit: restoreFloor is undefined,
|
||||
// yet coldSnapshot must still reject for an absent log (probe read) and
|
||||
// serve an empty cut at the stored end for a present one.
|
||||
const pool = new MemoryMediaPool()
|
||||
const logs = new Map([['bare', storedLog([['a']])]]) // seqs 0..2
|
||||
const ctx = new Context()
|
||||
contexts.push(ctx)
|
||||
await ctx.plugin(Storage)
|
||||
ctx.storage.backend.register('memory', new MemoryStorageBackend(pool))
|
||||
const facility = new DomainFacility(ctx, { backend: 'memory', routes: {} })
|
||||
ctx.storage.mount('domain', facility)
|
||||
ctx.provide('storageDomain', facility)
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SessionProjectionRegistry)
|
||||
ctx.provide('sessionPersistence', fakePersistence(logs) as never)
|
||||
await ctx.plugin(SessionProjectionCache, { writeEveryEvents: 100, writeIntervalMs: 60_000 })
|
||||
await expect(ctx.sessionProjectionCache.coldSnapshot(SessionId('absent'))).rejects.toThrow('not found')
|
||||
await expect(ctx.sessionProjectionCache.coldSnapshot(SessionId('bare')))
|
||||
.resolves.toEqual({ asOfSeq: 2, values: {} })
|
||||
})
|
||||
})
|
||||
39
packages/session/session-projection-cache/tsconfig.json
Normal file
39
packages/session/session-projection-cache/tsconfig.json
Normal file
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
{
|
||||
"path": "../session-persistence"
|
||||
},
|
||||
{
|
||||
"path": "../session-projection"
|
||||
},
|
||||
{
|
||||
"path": "../../storage/storage"
|
||||
},
|
||||
{
|
||||
"path": "../../storage/storage-domain"
|
||||
},
|
||||
{
|
||||
"path": "../../support/invariants"
|
||||
}
|
||||
]
|
||||
}
|
||||
6
packages/session/session-projection/README.i18n.yaml
Normal file
6
packages/session/session-projection/README.i18n.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/session-projection/session-projection/README.md
|
||||
README.md: a42e88c262915fc5e53cd72205079cbc8029a8df
|
||||
README.zh.md: f60a48bd41f0c33edb85a495ada9b6818ec46ee5
|
||||
47
packages/session/session-projection/README.md
Normal file
47
packages/session/session-projection/README.md
Normal file
@@ -0,0 +1,47 @@
|
||||
# @deepseek-ai/dsh-session-projection
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
Session-projection seam. It owns `ctx.sessionProjections`, the registry that drives every registered projection unit over committed session events and serves finished whole values to carriers, currently the api-proxy history tail page and `session/projection` push frame. A domain registers pure mathematics; the framework owns the drive. The [session-projection RFC](../../../.agents/notes/proposed/architecture/2026-07-27-session-projection-and-command-log.md) records the design rationale.
|
||||
|
||||
## Service: `SessionProjectionRegistry` (ctx key: `sessionProjections`)
|
||||
|
||||
### Public API
|
||||
|
||||
- `ctx.sessionProjections.register(definition): () => void` Register one domain's unit. Duplicate keys and invalid `stateVersion` throw; the registration is an effect on the calling fiber, so an unloaded domain plugin's key (with its cached cells) disappears from subsequent drives and snapshots — clients read that as capability absence.
|
||||
- `ctx.sessionProjections.onChanged(listener): () => void` Subscribe to the change feed: one call per unit whose state reference changed, per committed event, carrying the schema-validated view and the causing seq. Effect-tied like `register`.
|
||||
- `ctx.sessionProjections.snapshot(session): ProjectionSnapshot` One consistent synchronous cut over every registered unit — `{ asOfSeq, values }` with `asOfSeq` = the seq of the last event every value reflects (`-1` for an empty log).
|
||||
|
||||
### Key Types
|
||||
|
||||
- `SessionProjectionMap` — the single merge-extensible type table for the whole chain (host unit, wire block, React hook). Values are wire-JSON whole values; rendering belongs to the slot system, never this layer.
|
||||
- `ProjectionDefinition<K, S>` — `{ key, schema, init(), apply(state, event), view(state), stateVersion }`: a state-driven computation unit of three pure synchronous functions plus declarations, never an opaque getter.
|
||||
|
||||
## Contract
|
||||
|
||||
- **The framework drives, the domain computes.** The registry subscribes to `session/event` once; every committed event passes every unit's `apply` eagerly. Domains hold no subscriptions. Cells (`{state, observedSeq}` per unit per session, WeakMap-keyed) build lazily — a unit registered after events flowed, or a read of a session predating the registration, folds `init` over the in-memory log on first touch.
|
||||
- **Same-reference means no work.** `apply` MUST return the same state reference for events that do not concern the unit; the drive gates the change feed on `Object.is`, so non-matching events cost one call and nothing downstream.
|
||||
- **Whole-value event rule (load-bearing).** A state-carrying log event MUST carry the complete post-change state, never a bare delta — it keeps every transition trivially cheap and every served value self-describing (last-wins for consumers).
|
||||
- **Synchronous unit discipline.** `init`/`apply`/`view` MUST be synchronous; carriers read `snapshot()` in the same tick as their page slice, which is what makes `asOfSeq` one consistent cut. An accidentally-async `view` returns a Promise, which fails the boundary `schema.parse` loudly.
|
||||
- **State is plain JSON, `stateVersion` is its invalidation anchor.** The persisted projection cache stores `(sessionId, key, ver, seq, val)` rows; bump `stateVersion` whenever the state shape or the fold semantics change so stale rows are discarded instead of forward-applied into garbage.
|
||||
- **No wire vocabulary here.** The registry exposes only the change feed and the snapshot read face; carriers (api-proxy) mint their own frames (`session/projection`) and blocks from them.
|
||||
- **Optional seam.** Domain plugins register under `ctx.inject(['sessionProjections'], …)` so headless assemblies without the registry stay unaffected; carriers use `ctx.get('sessionProjections')` and omit their block/frames entirely when the registry is absent.
|
||||
|
||||
## Role
|
||||
|
||||
This is the interface-plus-drive package of the capability-seam split: domain host plugins (e.g. `dsh-tool-todo`) contribute units, carriers (`dsh-host-apiproxy`) consume the snapshot and change feed, and neither knows the other.
|
||||
|
||||
## Model Experience
|
||||
|
||||
None, as the registry only computes client-facing read models of already-logged session state and touches no prompt, message, schema, stream, or tool result.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
None; projections never assemble or send provider requests.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **Every tail page carries every registered key** — there is no per-key opt-out or lazy-key request shape yet; acceptable while values are UI-scale whole states (a todo list, a goal snapshot), revisit if a domain's value grows large.
|
||||
- **Eager drive touches every unit per event** — cheap by construction (whole-value rule, same-reference gate), but a hot path would justify per-unit event-type prefilters, addable without contract change.
|
||||
- **Registry cells live in memory only** — a restart rebuilds by folding the log on first touch; compositions that mount `dsh-session-projection-cache` seed that fold from persisted rows instead.
|
||||
- **Synchronous unit discipline is only partially mechanical** — the boundary `schema.parse` rejects a Promise-returning `view`, but an `apply` that blocks or reads torn non-session state is a review concern; the invariant companion documents why no runtime check exists.
|
||||
47
packages/session/session-projection/README.zh.md
Normal file
47
packages/session/session-projection/README.zh.md
Normal file
@@ -0,0 +1,47 @@
|
||||
# @deepseek-ai/dsh-session-projection
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
会话投影 seam。它拥有 `ctx.sessionProjections`：该注册表在已提交的会话事件上驱动每个已注册的投影单元，并向载体提供完整的最终值，目前包括 api-proxy 历史尾页和 `session/projection` 推送帧。领域注册的只是纯数学；驱动权归框架。[session-projection RFC](../../../.agents/notes/proposed/architecture/2026-07-27-session-projection-and-command-log.md)记录了设计理由。
|
||||
|
||||
## 服务：`SessionProjectionRegistry`（ctx 键：`sessionProjections`）
|
||||
|
||||
### 公开 API
|
||||
|
||||
- `ctx.sessionProjections.register(definition): () => void` 注册一个领域的单元。key 重复或 `stateVersion` 非法都会 throw；注册是挂在调用方 fiber 上的 effect，领域插件卸载后其 key（连同缓存的 cell）从后续驱动与快照中消失——客户端将其读作能力缺失。
|
||||
- `ctx.sessionProjections.onChanged(listener): () => void` 订阅变更流：每个已提交事件、每个状态引用发生变化的单元各回调一次，携带经 schema 校验的 view 与致因 seq。与 `register` 一样绑定 effect。
|
||||
- `ctx.sessionProjections.snapshot(session): ProjectionSnapshot` 对全部已注册单元做一次一致的同步切面——`{ asOfSeq, values }`，其中 `asOfSeq` = 所有值共同反映到的最后一个事件的 seq（空日志为 `-1`）。
|
||||
|
||||
### 关键类型
|
||||
|
||||
- `SessionProjectionMap`——整条链路唯一的 merge-extensible 类型表（host 侧单元、协议块、React 钩子）。值是协议层 JSON 全量值；渲染归 slot 体系管，永远不归本层。
|
||||
- `ProjectionDefinition<K, S>`——`{ key, schema, init(), apply(state, event), view(state), stateVersion }`：由三个纯同步函数外加若干声明构成的状态驱动计算单元（state-driven computation unit），绝不是一个不透明的 getter。
|
||||
|
||||
## 契约
|
||||
|
||||
- **框架负责驱动，领域负责计算。** 注册表只订阅一次 `session/event`；每个已提交事件都会主动经过每个单元的 `apply`。领域不持有任何订阅。cell（每会话每单元一份 `{state, observedSeq}`，以 WeakMap 为键）惰性构建——在事件流过之后才注册的单元，或读取一个早于该注册的会话，都在首次触达时从 `init` 出发在内存日志上折叠。
|
||||
- **同引用即无工作。** 对与单元无关的事件，`apply` 必须返回同一个状态引用；驱动以 `Object.is` 把守变更流，因此不匹配的事件只花一次调用，不产生任何下游工作。
|
||||
- **全量值事件规则（承重）。** 携带状态的日志事件必须携带变更后的完整状态，绝不携带裸增量——这让每次状态转移始终足够廉价，也让每个被供给的值自描述（对消费方即 last-wins）。
|
||||
- **单元的同步纪律。**`init`/`apply`/`view` 必须是同步的；载体在切出页面切片的同一 tick 内读取 `snapshot()`，`asOfSeq` 之所以是一个一致切面正系于此。误写成异步的 `view` 会返回 Promise，让边界的 `schema.parse` 当场大声失败。
|
||||
- **状态是纯 JSON，`stateVersion` 是其失效锚点。** 持久投影缓存（persisted projection cache）存储 `(sessionId, key, ver, seq, val)` 行；状态形状或折叠语义一旦变化就递增 `stateVersion`，使陈旧行被丢弃，而不是被正向 apply 成垃圾。
|
||||
- **本层没有协议词汇。** 注册表只暴露变更流与快照读取面；载体（api-proxy）据此自铸各自的帧（`session/projection`）与块。
|
||||
- **可选 seam。** 领域插件在 `ctx.inject(['sessionProjections'], …)` 下注册，因此不带注册表的 headless 组装完全不受影响；载体使用 `ctx.get('sessionProjections')`，注册表缺席时完全省略自己的块与帧。
|
||||
|
||||
## 职责
|
||||
|
||||
这是能力 seam 拆分中「接口 + 驱动」的那个包：领域 host 插件（如 `dsh-tool-todo`）贡献单元，载体（`dsh-host-apiproxy`）消费快照与变更流，两侧互不相识。
|
||||
|
||||
## 模型体验
|
||||
|
||||
无——注册表只对已入日志的会话状态计算面向客户端的读模型，不触碰任何提示词、消息、schema、流或工具结果。
|
||||
|
||||
#### KV Cache 影响
|
||||
|
||||
无；投影从不组装或发送提供方请求。
|
||||
|
||||
## 已知限制与暂缓事项
|
||||
|
||||
- **每个尾页携带每个已注册的 key**——尚无逐 key 的 opt-out 或惰性 key 请求形状；在值都是 UI 量级的全量状态（一张 todo 清单、一份 goal 快照）时可以接受，若某领域的值变大再重议。
|
||||
- **主动驱动（eager drive）逐事件触达每个单元**——按构造开销很低（全量值规则、同引用闸门），但若出现热点路径，可加按单元的事件类型预过滤，契约不变。
|
||||
- **注册表 cell 只活在内存里**——重启后首次触达时靠折叠日志重建；挂载了 `dsh-session-projection-cache` 的组合改由持久行播种该折叠。
|
||||
- **单元同步纪律只有部分可机械把关**——边界 `schema.parse` 能拒绝返回 Promise 的 `view`，但阻塞的 `apply`、或读取撕裂的非会话状态的 `apply`，只能靠评审把关；invariant 配套记载了为何不存在运行时检查。
|
||||
45
packages/session/session-projection/package.json
Normal file
45
packages/session/session-projection/package.json
Normal file
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-session-projection",
|
||||
"description": "Session-projection seam: the merge-extensible projection type table, the provider contract, and the ctx.sessionProjections registry serving whole current values of log-derived per-session state",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./invariant": {
|
||||
"types": "./lib/types/invariant.d.ts",
|
||||
"default": "./lib/invariant.js"
|
||||
},
|
||||
"./types": {
|
||||
"types": "./lib/types/types.d.ts",
|
||||
"default": "./lib/types/types.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"lib/types/**/*.js",
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"zod": "^4.4.3"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-invariants": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
400
packages/session/session-projection/src/index.ts
Normal file
400
packages/session/session-projection/src/index.ts
Normal file
@@ -0,0 +1,400 @@
|
||||
/**
|
||||
* Session-projection seam: the merge-extensible `SessionProjectionMap` type
|
||||
* table, the `ProjectionDefinition` state-driven computation unit contract,
|
||||
* and the `ctx.sessionProjections` registry that DRIVES every registered unit
|
||||
* forward eagerly over committed session events. Domain host plugins
|
||||
* contribute pure mathematics (init/apply/view); the framework owns the
|
||||
* subscription, the per-session watermark cache, and change notification;
|
||||
* carriers consume the snapshot read face and the change feed. Neither side
|
||||
* knows the other
|
||||
* (capability-seam three-way split). Design authority: the session-projection
|
||||
* RFC (.agents/notes/proposed/architecture/2026-07-27-session-projection-and-command-log.md).
|
||||
*
|
||||
* Whole-value event rule (load-bearing): a state-carrying log event MUST
|
||||
* carry the complete post-change state, never a bare delta — it keeps every
|
||||
* unit's transition trivially cheap and every served value self-describing.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-session-projection
|
||||
*/
|
||||
|
||||
import { Context, Service } from 'cordis'
|
||||
import type { ZodType } from 'zod'
|
||||
import type { Session, SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
|
||||
declare module 'cordis' {
|
||||
interface Context {
|
||||
sessionProjections: SessionProjectionRegistry
|
||||
}
|
||||
}
|
||||
|
||||
import type { SessionProjectionMap } from './types.ts'
|
||||
|
||||
export type { SessionProjectionMap } from './types.ts'
|
||||
|
||||
/**
|
||||
* One domain's state-driven computation unit: three pure synchronous
|
||||
* functions plus declarations — never an opaque getter. The framework drives
|
||||
* `apply` on every committed session event; the domain holds no
|
||||
* subscriptions and owns only the mathematics. All three functions MUST be
|
||||
* synchronous (an async unit would tear the carriers' consistency cut) and
|
||||
* `state` MUST be plain JSON (the persisted-cache precondition).
|
||||
*/
|
||||
export interface ProjectionDefinition<K extends keyof SessionProjectionMap, S> {
|
||||
/** The projection key this unit owns (its `SessionProjectionMap` entry). */
|
||||
key: K
|
||||
/** Validates the wire payload (`view` output) before it leaves the host. */
|
||||
schema: ZodType<SessionProjectionMap[K]>
|
||||
/**
|
||||
* State for the empty log.
|
||||
* @returns the initial state.
|
||||
*/
|
||||
init(): S
|
||||
/**
|
||||
* Pure transition: previous state + one committed event → next state. A
|
||||
* unit uninterested in an event MUST return the same state reference — an
|
||||
* unchanged reference (`Object.is`) produces zero downstream work.
|
||||
* @param state - the state covering all prior events.
|
||||
* @param event - the next committed session event.
|
||||
* @returns the next state (same reference when the event is not the unit's).
|
||||
*/
|
||||
apply(state: S, event: SessionEvent): S
|
||||
/**
|
||||
* State → wire payload (the read-side projection).
|
||||
* @param state - the current state.
|
||||
* @returns the whole current value for this unit's key.
|
||||
*/
|
||||
view(state: S): SessionProjectionMap[K]
|
||||
/**
|
||||
* Persisted-cache invalidation anchor: bump whenever the state shape or the
|
||||
* fold semantics change, so persisted `(sessionId, key, ver, seq, val)`
|
||||
* rows from an older unit are discarded instead of being forward-applied
|
||||
* into garbage. Non-negative integer.
|
||||
*/
|
||||
stateVersion: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Change-feed listener: one unit's value changed for one session. `value` is
|
||||
* the schema-validated `view` output; `seq` is the unit's watermark at
|
||||
* emission (the seq of the event that caused the change).
|
||||
*/
|
||||
export type ProjectionChangeListener = (
|
||||
session: Session,
|
||||
key: Extract<keyof SessionProjectionMap, string>,
|
||||
value: unknown,
|
||||
seq: number,
|
||||
) => void
|
||||
|
||||
/**
|
||||
* One consistent read cut over every registered unit for one session.
|
||||
* `asOfSeq` is the shared watermark — the seq of the last event every value
|
||||
* reflects (`-1` for an empty log, mirroring `session/subscribed.lastSeq`).
|
||||
*/
|
||||
export interface ProjectionSnapshot {
|
||||
/** Seq of the last event the values reflect; -1 for an empty log. */
|
||||
asOfSeq: number
|
||||
/** Whole current value per registered key. */
|
||||
values: Partial<SessionProjectionMap>
|
||||
}
|
||||
|
||||
/**
|
||||
* One unit's checkpoint: its internal state (plain JSON by the unit
|
||||
* contract), the seq of the last event folded into it, and the unit
|
||||
* `stateVersion` that produced it — the persisted projection-cache row
|
||||
* `(sessionId, key, ver, seq, val)` minus the two outer keys. A row is
|
||||
* never authoritative, only a fold shortcut: `restore` discards it on a
|
||||
* version mismatch or when it claims events past the stored log end.
|
||||
*/
|
||||
export interface ProjectionCheckpointRow {
|
||||
/** The registering unit's `stateVersion` at fold time. */
|
||||
ver: number
|
||||
/** Seq of the last event folded into `val`; -1 for the empty log. */
|
||||
seq: number
|
||||
/** The unit's internal state — plain JSON per the unit contract. */
|
||||
val: unknown
|
||||
}
|
||||
|
||||
/** Checkpoint rows keyed by projection key (one session's persisted cache value). */
|
||||
export type ProjectionCheckpoint = Record<string, ProjectionCheckpointRow>
|
||||
|
||||
/** Type-erased unit view the drive machinery works with (the register seam already proved the typed contract). */
|
||||
interface ErasedDefinition {
|
||||
key: string
|
||||
schema: { parse(value: unknown): unknown }
|
||||
init(): unknown
|
||||
apply(state: unknown, event: SessionEvent): unknown
|
||||
view(state: unknown): unknown
|
||||
stateVersion: number
|
||||
}
|
||||
|
||||
/** Per-session per-unit watermark cache row. */
|
||||
interface UnitCell {
|
||||
state: unknown
|
||||
/** Seq of the last event passed through `apply` (regardless of change). */
|
||||
observedSeq: number
|
||||
}
|
||||
|
||||
/** One live registration: the unit plus its per-session cells (dropped whole on disposal). */
|
||||
interface Registration {
|
||||
readonly def: ErasedDefinition
|
||||
readonly cells: WeakMap<Session, UnitCell>
|
||||
}
|
||||
|
||||
/**
|
||||
* `ctx.sessionProjections`: the projection unit table and its drive. The
|
||||
* service subscribes to `session/event` once; every committed event passes
|
||||
* every registered unit's `apply` (eager drive), and a changed state
|
||||
* reference notifies the change feed with the schema-validated view.
|
||||
* Cells build lazily — a unit registered after events flowed, or a session
|
||||
* older than the registry, folds `init` over the in-memory log on first
|
||||
* touch (event or read). Registration is an effect (disposer rides the
|
||||
* calling fiber): an unloaded domain plugin's key disappears from snapshots
|
||||
* and clients read it as capability absence. Duplicate keys throw. Domain
|
||||
* plugins register under `ctx.inject(['sessionProjections'], …)` so headless
|
||||
* assemblies without the registry stay unaffected.
|
||||
*/
|
||||
export class SessionProjectionRegistry extends Service {
|
||||
private readonly registrations = new Map<string, Registration>()
|
||||
private readonly listeners = new Set<ProjectionChangeListener>()
|
||||
|
||||
/**
|
||||
* Create and install the registry as `ctx.sessionProjections`.
|
||||
* @param ctx - Cordis context that owns the service.
|
||||
*/
|
||||
constructor(ctx: Context) {
|
||||
super(ctx, 'sessionProjections')
|
||||
ctx.on('session/event', (session: Session, event: SessionEvent) => {
|
||||
this.drive(session, event)
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Register one domain's unit. The registration is an effect on the calling
|
||||
* context's fiber: disposing the fiber (or calling the returned disposer)
|
||||
* removes the key — and the unit's cached cells — from subsequent drives
|
||||
* and snapshots.
|
||||
* @param definition - key, boundary schema, pure unit functions, and stateVersion.
|
||||
* @returns the exact disposer that unregisters this unit.
|
||||
*/
|
||||
register<K extends keyof SessionProjectionMap, S>(definition: ProjectionDefinition<K, S>): () => void {
|
||||
if (!Number.isSafeInteger(definition.stateVersion) || definition.stateVersion < 0) {
|
||||
throw new Error(`session projection ${JSON.stringify(definition.key)} stateVersion must be a non-negative integer, got ${String(definition.stateVersion)}`)
|
||||
}
|
||||
const dispose = this.ctx.effect(function* (this: SessionProjectionRegistry) {
|
||||
const key = definition.key as string
|
||||
if (this.registrations.has(key)) {
|
||||
throw new Error(`session projection key ${JSON.stringify(key)} is already registered`)
|
||||
}
|
||||
this.registrations.set(key, { def: definition, cells: new WeakMap() })
|
||||
yield () => {
|
||||
this.registrations.delete(key)
|
||||
}
|
||||
}.bind(this), 'sessionProjections.register()')
|
||||
return () => void dispose()
|
||||
}
|
||||
|
||||
/**
|
||||
* Subscribe to the change feed. The registration is an effect on the
|
||||
* calling context's fiber.
|
||||
* @param listener - called once per unit whose state reference changed, per committed event.
|
||||
* @returns the exact disposer that unsubscribes.
|
||||
*/
|
||||
onChanged(listener: ProjectionChangeListener): () => void {
|
||||
const dispose = this.ctx.effect(() => {
|
||||
this.listeners.add(listener)
|
||||
return () => {
|
||||
this.listeners.delete(listener)
|
||||
}
|
||||
}, 'sessionProjections.onChanged()')
|
||||
return () => void dispose()
|
||||
}
|
||||
|
||||
/**
|
||||
* One consistent cut over every registered unit for one session, read from
|
||||
* the watermark cache (missing cells fold lazily over the in-memory log).
|
||||
* Fully synchronous — every value and `asOfSeq` reflect the same log
|
||||
* position. Each value passes its unit's schema before leaving.
|
||||
* @param session - the session whose projection values are read.
|
||||
* @returns the snapshot; `values` is empty when no unit is registered.
|
||||
*/
|
||||
snapshot(session: Session): ProjectionSnapshot {
|
||||
const values: Record<string, unknown> = {}
|
||||
for (const registration of this.registrations.values()) {
|
||||
const cell = this.cellFor(registration, session)
|
||||
values[registration.def.key] = registration.def.schema.parse(registration.def.view(cell.state))
|
||||
}
|
||||
return { asOfSeq: session.seq - 1, values: values }
|
||||
}
|
||||
|
||||
/**
|
||||
* State-level checkpoint of every registered unit for one session, read
|
||||
* from the watermark cache (missing cells fold lazily over the in-memory
|
||||
* log). This is the write side of the persisted projection cache: the
|
||||
* returned rows are the `(key → {ver, seq, val})` part of the durable
|
||||
* `(sessionId, key, ver, seq, val)`
|
||||
* rows. Every `val` is a DETACHED structured clone — never the live
|
||||
* cell reference: the watermark cache is this registry's authoritative
|
||||
* mutable state, and a caller reaching the live reference could corrupt
|
||||
* every subsequent snapshot and frame through it (plain JSON by the unit
|
||||
* contract, so the clone is total).
|
||||
* @param session - the session whose unit states are checkpointed.
|
||||
* @returns one row per registered key; empty when no unit is registered.
|
||||
*/
|
||||
checkpoint(session: Session): ProjectionCheckpoint {
|
||||
const rows: ProjectionCheckpoint = {}
|
||||
for (const registration of this.registrations.values()) {
|
||||
const cell = this.cellFor(registration, session)
|
||||
rows[registration.def.key] = {
|
||||
ver: registration.def.stateVersion,
|
||||
seq: cell.observedSeq,
|
||||
val: structuredClone(cell.state),
|
||||
}
|
||||
}
|
||||
return rows
|
||||
}
|
||||
|
||||
/**
|
||||
* The stored seq a {@link restore} tail read over `checkpoint` must start
|
||||
* at: one event BELOW the lowest usable watermark (a row is usable when
|
||||
* its `ver` matches the live unit's `stateVersion`; an absent or mismatched row
|
||||
* pulls the floor to `0` — that key must refold the full log). The
|
||||
* one-below anchor is load-bearing: the tail then proves how far the
|
||||
* stored log still extends, so {@link restore} can detect a log that
|
||||
* shrank below a row's watermark (crash-repair truncation) instead of
|
||||
* serving the stale row as current — an empty tail read from the anchor
|
||||
* yields an end below every watermark and the restore rejects for a full
|
||||
* re-read.
|
||||
* @param checkpoint - persisted rows for one session (possibly stale or empty).
|
||||
* @returns the seq to hand the persistence `readFrom`, or `undefined`
|
||||
* when no unit is registered (no read needed — {@link restore} would
|
||||
* serve empty values regardless).
|
||||
*/
|
||||
restoreFloor(checkpoint: ProjectionCheckpoint): number | undefined {
|
||||
let floor: number | undefined
|
||||
for (const registration of this.registrations.values()) {
|
||||
const row = checkpoint[registration.def.key]
|
||||
const need = row !== undefined && row.ver === registration.def.stateVersion
|
||||
? Math.max(row.seq + 1, 0)
|
||||
: 0
|
||||
floor = floor === undefined ? need : Math.min(floor, need)
|
||||
}
|
||||
return floor === undefined ? undefined : Math.max(floor - 1, 0)
|
||||
}
|
||||
|
||||
/**
|
||||
* View a checkpoint's rows without any log read: for every registered
|
||||
* unit whose row's `ver` matches, serve the schema-validated
|
||||
* `view` of the stored state; mismatched or absent rows leave their key
|
||||
* absent (a cold or listing consumer treats it as not-yet-available and a
|
||||
* fuller read path refolds it). The zero-I/O rung of the read ladder —
|
||||
* values are as stale as their rows, never wrong.
|
||||
* @param checkpoint - persisted rows for one session (possibly stale or empty).
|
||||
* @returns whole values per key with a usable row; empty when none.
|
||||
*/
|
||||
viewCheckpoint(checkpoint: ProjectionCheckpoint): Partial<SessionProjectionMap> {
|
||||
const values: Record<string, unknown> = {}
|
||||
for (const registration of this.registrations.values()) {
|
||||
const def = registration.def
|
||||
const row = checkpoint[def.key]
|
||||
if (row === undefined || row.ver !== def.stateVersion) continue
|
||||
values[def.key] = def.schema.parse(def.view(row.val))
|
||||
}
|
||||
return values
|
||||
}
|
||||
|
||||
/**
|
||||
* Cold read: fold every registered unit over a stored log suffix, seeding
|
||||
* each from its checkpoint row when usable — the one read recipe (cached
|
||||
* state + forward tail replay + `view`) applied without a live `Session`.
|
||||
* Call with the events returned by a persistence
|
||||
* `readFrom(id, restoreFloor(checkpoint))` and that same floor as
|
||||
* `baseSeq`; the floor's one-below anchor makes the supplied end honest,
|
||||
* so a shrunk log is detected here. A row is usable iff its
|
||||
* `ver` matches the live unit's `stateVersion`, it does not predate `baseSeq`
|
||||
* (`seq >= baseSeq - 1`), and it does not claim events past the
|
||||
* supplied end (`seq <= endSeq`); an unusable row is discarded
|
||||
* and its key refolds from `init` — which is only sound over the full
|
||||
* log, so a discarded row with `baseSeq > 0` throws (the caller re-reads
|
||||
* from seq 0, e.g. after a crash-repair truncation shrank the log below
|
||||
* a row's watermark).
|
||||
* @param checkpoint - persisted rows for one session (possibly stale or empty).
|
||||
* @param events - the stored events with `seq >= baseSeq`, in seq order.
|
||||
* @param baseSeq - the seq `events` starts at (its first event's seq when non-empty).
|
||||
* @returns the snapshot cut at the supplied log end (`asOfSeq` is the last
|
||||
* supplied event's seq, `baseSeq - 1` for an empty tail) plus the
|
||||
* refreshed checkpoint rows at that cut, ready for a durable write-back.
|
||||
*/
|
||||
restore(checkpoint: ProjectionCheckpoint, events: readonly SessionEvent[], baseSeq: number):
|
||||
{ snapshot: ProjectionSnapshot; checkpoint: ProjectionCheckpoint } {
|
||||
const endSeq = events.at(-1)?.seq ?? baseSeq - 1
|
||||
const values: Record<string, unknown> = {}
|
||||
const refreshed: ProjectionCheckpoint = {}
|
||||
for (const registration of this.registrations.values()) {
|
||||
const def = registration.def
|
||||
const row = checkpoint[def.key]
|
||||
const usable = row !== undefined
|
||||
&& row.ver === def.stateVersion
|
||||
&& row.seq >= baseSeq - 1
|
||||
&& row.seq <= endSeq
|
||||
if (!usable && baseSeq > 0) {
|
||||
throw new Error(
|
||||
`session projection ${JSON.stringify(def.key)} cannot restore from seq ${baseSeq}: `
|
||||
+ 'its checkpoint row is missing, version-mismatched, or beyond the supplied log end; re-read from seq 0',
|
||||
)
|
||||
}
|
||||
let state = usable ? row.val : def.init()
|
||||
const from = usable ? row.seq : baseSeq - 1
|
||||
for (const event of events) {
|
||||
if (event.seq > from) state = def.apply(state, event)
|
||||
}
|
||||
values[def.key] = def.schema.parse(def.view(state))
|
||||
refreshed[def.key] = { ver: def.stateVersion, seq: endSeq, val: state }
|
||||
}
|
||||
return {
|
||||
snapshot: { asOfSeq: endSeq, values: values },
|
||||
checkpoint: refreshed,
|
||||
}
|
||||
}
|
||||
|
||||
/** Fold one unit from init over `events`, producing a cell watermarked at the last folded event. */
|
||||
private buildCell(def: ErasedDefinition, events: readonly SessionEvent[]): UnitCell {
|
||||
let state = def.init()
|
||||
for (const event of events) state = def.apply(state, event)
|
||||
return { state, observedSeq: (events.at(-1)?.seq ?? -1) }
|
||||
}
|
||||
|
||||
/** Read (or lazily build, folding the full in-memory log) one unit's cell. */
|
||||
private cellFor(registration: Registration, session: Session): UnitCell {
|
||||
let cell = registration.cells.get(session)
|
||||
if (cell === undefined) {
|
||||
cell = this.buildCell(registration.def, session.events)
|
||||
registration.cells.set(session, cell)
|
||||
}
|
||||
return cell
|
||||
}
|
||||
|
||||
/** Eager drive: pass one committed event through every registered unit; notify on changed references. */
|
||||
private drive(session: Session, event: SessionEvent): void {
|
||||
for (const registration of this.registrations.values()) {
|
||||
let cell = registration.cells.get(session)
|
||||
if (cell === undefined) {
|
||||
// Late build mid-stream: fold history before this event (seq = log
|
||||
// index, so the prefix slice is exact), then take the normal gate.
|
||||
cell = this.buildCell(registration.def, session.events.slice(0, event.seq))
|
||||
registration.cells.set(session, cell)
|
||||
}
|
||||
const next = registration.def.apply(cell.state, event)
|
||||
const changed = !Object.is(next, cell.state)
|
||||
cell.state = next
|
||||
cell.observedSeq = event.seq
|
||||
if (changed && this.listeners.size > 0) {
|
||||
const value = registration.def.schema.parse(registration.def.view(next))
|
||||
for (const listener of this.listeners) {
|
||||
listener(session, registration.def.key as Extract<keyof SessionProjectionMap, string>, value, event.seq)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export default SessionProjectionRegistry
|
||||
38
packages/session/session-projection/src/invariant.ts
Normal file
38
packages/session/session-projection/src/invariant.ts
Normal file
@@ -0,0 +1,38 @@
|
||||
/**
|
||||
* Package-owned invariant companion for `@deepseek-ai/dsh-session-projection`.
|
||||
* @module @deepseek-ai/dsh-session-projection/invariant
|
||||
*/
|
||||
|
||||
/* jscpd:ignore-start */
|
||||
import type { Context } from 'cordis'
|
||||
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
|
||||
|
||||
const PACKAGE_NAME = '@deepseek-ai/dsh-session-projection'
|
||||
|
||||
/** Cordis companion plugin name. */
|
||||
export const name = 'session-projection-invariant'
|
||||
/** Service required before the companion can reserve package ownership. */
|
||||
export const inject = ['invariants']
|
||||
|
||||
/**
|
||||
* No runtime invariant: the registry's own contracts (duplicate-key and
|
||||
* stateVersion rejection, effect-tied removal, the Object.is change gate) are
|
||||
* enforced synchronously inside the service and proven by its spec, the
|
||||
* drive relation (every committed `session/event` passes every unit) would
|
||||
* require re-running the drive to check — duplicating the implementation
|
||||
* rather than detecting drift — and the served-value relation (every served
|
||||
* key has a live registration) lives on each carrier's wire path, which
|
||||
* emits no cordis event this companion could observe; carrier specs assert
|
||||
* it. Synchronous-unit discipline is enforced as far as practical by the
|
||||
* boundary `schema.parse` (a Promise-returning view fails loudly).
|
||||
*/
|
||||
const install: InvariantInstaller = () => {}
|
||||
|
||||
/**
|
||||
* Register this package's invariant companion.
|
||||
* @param ctx - Cordis context carrying the invariant service.
|
||||
* @returns the installed registration's disposer after setup succeeds.
|
||||
*/
|
||||
export const apply = (ctx: Context): Promise<() => void> =>
|
||||
Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
|
||||
/* jscpd:ignore-end */
|
||||
17
packages/session/session-projection/src/types.ts
Normal file
17
packages/session/session-projection/src/types.ts
Normal file
@@ -0,0 +1,17 @@
|
||||
/**
|
||||
* Pure-type outlet of the session-projection seam: the one projection type
|
||||
* table, importable from client aggregates without dragging the host-side
|
||||
* cordis Context merges of the package root (dsh-agent → dsh-session). Domain
|
||||
* packages may declare-merge through either the package root or this outlet —
|
||||
* re-export preserves symbol identity, so both land on the same table.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-session-projection/types
|
||||
*/
|
||||
|
||||
/**
|
||||
* The single projection type table for the whole chain (host provider, wire
|
||||
* block, client cell, React hook). Domain packages merge their key here via
|
||||
* declaration merging; values are wire-JSON whole values. How a value is
|
||||
* rendered is the slot system's business, never this layer's.
|
||||
*/
|
||||
export interface SessionProjectionMap {}
|
||||
334
packages/session/session-projection/tests/registry.spec.ts
Normal file
334
packages/session/session-projection/tests/registry.spec.ts
Normal file
@@ -0,0 +1,334 @@
|
||||
/**
|
||||
* SessionProjectionRegistry unit drive: eager apply on committed events with
|
||||
* lazy cell build (registration after events, session after registration),
|
||||
* the Object.is no-change gate (same reference ⇒ zero change-feed work),
|
||||
* snapshot consistency (asOfSeq = last event seq; values from the watermark
|
||||
* cache), duplicate-key rejection, stateVersion validation, and effect-tied
|
||||
* removal of registrations and change listeners (HMR safety).
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import { z } from 'zod'
|
||||
import SessionStore from '@deepseek-ai/dsh-session'
|
||||
import type { Session, SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import SessionProjectionRegistry from '@deepseek-ai/dsh-session-projection'
|
||||
import type { ProjectionDefinition } from '@deepseek-ai/dsh-session-projection'
|
||||
|
||||
declare module '@deepseek-ai/dsh-session-projection/types' {
|
||||
interface SessionProjectionMap {
|
||||
'test/marks': { marks: string[] }
|
||||
'test/count': number
|
||||
}
|
||||
}
|
||||
|
||||
declare module '@deepseek-ai/dsh-session' {
|
||||
interface SessionEventMap {
|
||||
'test/mark': { marks: string[] }
|
||||
}
|
||||
}
|
||||
|
||||
/** Whole-value unit: latest test/mark event wins; unrelated events return the same reference. */
|
||||
type MarksState = { marks: string[] } | null
|
||||
const marksUnit = (): ProjectionDefinition<'test/marks', MarksState> => ({
|
||||
key: 'test/marks',
|
||||
schema: z.object({ marks: z.array(z.string()) }),
|
||||
init: () => null,
|
||||
apply: (state, event) => (event.type === 'test/mark' ? (event).data : state),
|
||||
view: state => state ?? { marks: [] },
|
||||
stateVersion: 1,
|
||||
})
|
||||
|
||||
/** Counting unit over every event — state changes on each apply. */
|
||||
const countUnit = (): ProjectionDefinition<'test/count', number> => ({
|
||||
key: 'test/count',
|
||||
schema: z.number().int().nonnegative(),
|
||||
init: () => 0,
|
||||
apply: state => state + 1,
|
||||
view: state => state,
|
||||
stateVersion: 1,
|
||||
})
|
||||
|
||||
async function harness(): Promise<{ ctx: Context; session: Session }> {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SessionProjectionRegistry)
|
||||
return { ctx, session: ctx.sessions.create() }
|
||||
}
|
||||
|
||||
const mark = (session: Session, marks: string[]): SessionEvent =>
|
||||
session.append('test/mark', { marks })
|
||||
|
||||
describe('SessionProjectionRegistry drive', () => {
|
||||
it('drives a registered unit over committed events and snapshots the current value', async () => {
|
||||
const { ctx, session } = await harness()
|
||||
ctx.sessionProjections.register(marksUnit())
|
||||
mark(session, ['a'])
|
||||
mark(session, ['a', 'b'])
|
||||
const snapshot = ctx.sessionProjections.snapshot(session)
|
||||
expect(snapshot.values['test/marks']).toEqual({ marks: ['a', 'b'] })
|
||||
expect(snapshot.asOfSeq).toBe(session.seq - 1)
|
||||
})
|
||||
|
||||
it('builds the cell lazily from the full log for a unit registered after events flowed', async () => {
|
||||
const { ctx, session } = await harness()
|
||||
mark(session, ['pre-registration'])
|
||||
ctx.sessionProjections.register(marksUnit())
|
||||
expect(ctx.sessionProjections.snapshot(session).values['test/marks']).toEqual({ marks: ['pre-registration'] })
|
||||
// The lazily-built cell then continues on the live drive path.
|
||||
mark(session, ['after'])
|
||||
expect(ctx.sessionProjections.snapshot(session).values['test/marks']).toEqual({ marks: ['after'] })
|
||||
})
|
||||
|
||||
it('serves init-derived state and asOfSeq -1 for an empty log', async () => {
|
||||
const { ctx, session } = await harness()
|
||||
ctx.sessionProjections.register(marksUnit())
|
||||
const snapshot = ctx.sessionProjections.snapshot(session)
|
||||
expect(snapshot.asOfSeq).toBe(-1)
|
||||
expect(snapshot.values['test/marks']).toEqual({ marks: [] })
|
||||
})
|
||||
|
||||
it('notifies onChanged with the validated view and the causing seq, and skips same-reference applies', async () => {
|
||||
const { ctx, session } = await harness()
|
||||
ctx.sessionProjections.register(marksUnit())
|
||||
const seen: { key: string; value: unknown; seq: number; sessionId: string }[] = []
|
||||
ctx.sessionProjections.onChanged((changedSession, key, value, seq) => {
|
||||
seen.push({ key, value, seq, sessionId: String(changedSession.id) })
|
||||
})
|
||||
const event = mark(session, ['a'])
|
||||
// Non-matching event: apply returns the same reference — no notification.
|
||||
session.append('turn/start', { turn: 1 })
|
||||
expect(seen).toEqual([{ key: 'test/marks', value: { marks: ['a'] }, seq: event.seq, sessionId: String(session.id) }])
|
||||
})
|
||||
|
||||
it('drives independently per session (cells are per-session watermarks)', async () => {
|
||||
const { ctx, session } = await harness()
|
||||
const other = ctx.sessions.create()
|
||||
ctx.sessionProjections.register(marksUnit())
|
||||
mark(session, ['one'])
|
||||
mark(other, ['two'])
|
||||
expect(ctx.sessionProjections.snapshot(session).values['test/marks']).toEqual({ marks: ['one'] })
|
||||
expect(ctx.sessionProjections.snapshot(other).values['test/marks']).toEqual({ marks: ['two'] })
|
||||
})
|
||||
|
||||
it('runs every registered unit — a changing unit notifies while a same-reference unit stays silent', async () => {
|
||||
const { ctx, session } = await harness()
|
||||
ctx.sessionProjections.register(marksUnit())
|
||||
ctx.sessionProjections.register(countUnit())
|
||||
const changedKeys: string[] = []
|
||||
ctx.sessionProjections.onChanged((_session, key) => {
|
||||
changedKeys.push(key)
|
||||
})
|
||||
session.append('turn/start', { turn: 1 })
|
||||
// count applied (+1 change), marks returned the same reference.
|
||||
expect(changedKeys).toEqual(['test/count'])
|
||||
const snapshot = ctx.sessionProjections.snapshot(session)
|
||||
expect(snapshot.values['test/count']).toBe(1)
|
||||
expect(snapshot.values['test/marks']).toEqual({ marks: [] })
|
||||
})
|
||||
|
||||
it('rejects duplicate keys loud and keeps the first unit', async () => {
|
||||
const { ctx, session } = await harness()
|
||||
ctx.sessionProjections.register(marksUnit())
|
||||
expect(() => ctx.sessionProjections.register(marksUnit())).toThrow(/"test\/marks" is already registered/)
|
||||
mark(session, ['kept'])
|
||||
expect(ctx.sessionProjections.snapshot(session).values['test/marks']).toEqual({ marks: ['kept'] })
|
||||
})
|
||||
|
||||
it('rejects a non-integer or negative stateVersion at register time', async () => {
|
||||
const { ctx } = await harness()
|
||||
expect(() => ctx.sessionProjections.register({ ...marksUnit(), stateVersion: -1 })).toThrow(/stateVersion/)
|
||||
expect(() => ctx.sessionProjections.register({ ...marksUnit(), stateVersion: 1.5 })).toThrow(/stateVersion/)
|
||||
})
|
||||
|
||||
it('register() disposer removes the key (with its cells) and frees it for re-registration', async () => {
|
||||
const { ctx, session } = await harness()
|
||||
const dispose = ctx.sessionProjections.register(marksUnit())
|
||||
mark(session, ['cached'])
|
||||
dispose()
|
||||
expect(ctx.sessionProjections.snapshot(session).values).toEqual({})
|
||||
ctx.sessionProjections.register(marksUnit())
|
||||
// Fresh registration rebuilds from the log, not from a stale cell.
|
||||
expect(ctx.sessionProjections.snapshot(session).values['test/marks']).toEqual({ marks: ['cached'] })
|
||||
})
|
||||
|
||||
it('removes registrations and change listeners when their owning fiber unloads (HMR safety)', async () => {
|
||||
const { ctx, session } = await harness()
|
||||
const notifications: string[] = []
|
||||
const fiber = await ctx.plugin(Object.assign((inner: Context) => {
|
||||
inner.sessionProjections.register(marksUnit())
|
||||
inner.sessionProjections.onChanged((_session, key) => {
|
||||
notifications.push(key)
|
||||
})
|
||||
}, { inject: ['sessionProjections'] }))
|
||||
mark(session, ['live'])
|
||||
expect(notifications).toEqual(['test/marks'])
|
||||
await fiber.dispose()
|
||||
mark(session, ['after-dispose'])
|
||||
expect(notifications).toEqual(['test/marks'])
|
||||
expect(ctx.sessionProjections.snapshot(session).values).toEqual({})
|
||||
})
|
||||
|
||||
it('checkpoints every registered unit with its stateVersion and per-cell watermark', async () => {
|
||||
const { ctx, session } = await harness()
|
||||
ctx.sessionProjections.register(marksUnit())
|
||||
ctx.sessionProjections.register({ ...countUnit(), stateVersion: 7 })
|
||||
const markEvent = mark(session, ['a'])
|
||||
const rows = ctx.sessionProjections.checkpoint(session)
|
||||
expect(rows['test/marks']).toEqual({ ver: 1, seq: markEvent.seq, val: { marks: ['a'] } })
|
||||
expect(rows['test/count']).toEqual({ ver: 7, seq: markEvent.seq, val: 1 })
|
||||
// Empty log: init-derived state at watermark -1.
|
||||
const fresh = ctx.sessions.create()
|
||||
expect(ctx.sessionProjections.checkpoint(fresh)['test/marks']).toEqual({ ver: 1, seq: -1, val: null })
|
||||
})
|
||||
|
||||
it('checkpoint states are detached clones — mutating them cannot corrupt the watermark cache', async () => {
|
||||
const { ctx, session } = await harness()
|
||||
ctx.sessionProjections.register(marksUnit())
|
||||
mark(session, ['a'])
|
||||
const rows = ctx.sessionProjections.checkpoint(session)
|
||||
// Hostile (or merely careless) consumer mutates the handed-out state.
|
||||
;(rows['test/marks']?.val as { marks: string[] }).marks.push('INJECTED')
|
||||
// The registry's authoritative cell is untouched: snapshot and a fresh
|
||||
// checkpoint both still serve the committed value.
|
||||
expect(ctx.sessionProjections.snapshot(session).values['test/marks']).toEqual({ marks: ['a'] })
|
||||
expect(ctx.sessionProjections.checkpoint(session)['test/marks']?.val).toEqual({ marks: ['a'] })
|
||||
})
|
||||
|
||||
it('restoreFloor anchors one below the lowest usable watermark and at 0 for missing or mismatched rows', async () => {
|
||||
const { ctx } = await harness()
|
||||
expect(ctx.sessionProjections.restoreFloor({})).toBeUndefined() // no unit registered
|
||||
ctx.sessionProjections.register(marksUnit())
|
||||
ctx.sessionProjections.register(countUnit())
|
||||
expect(ctx.sessionProjections.restoreFloor({})).toBe(0)
|
||||
// Lowest usable watermark is count's 5 → the anchored tail starts AT 5
|
||||
// (one below the first needed seq 6), so the read proves seq 5 still exists.
|
||||
expect(ctx.sessionProjections.restoreFloor({
|
||||
'test/marks': { ver: 1, seq: 10, val: { marks: [] } },
|
||||
'test/count': { ver: 1, seq: 5, val: 6 },
|
||||
})).toBe(5)
|
||||
// A version-mismatched row forces that key back to a full refold.
|
||||
expect(ctx.sessionProjections.restoreFloor({
|
||||
'test/marks': { ver: 2, seq: 10, val: { marks: [] } },
|
||||
'test/count': { ver: 1, seq: 5, val: 6 },
|
||||
})).toBe(0)
|
||||
// A fresh (-1) row still needs the whole tail from 0.
|
||||
expect(ctx.sessionProjections.restoreFloor({
|
||||
'test/marks': { ver: 1, seq: -1, val: null },
|
||||
'test/count': { ver: 1, seq: -1, val: 0 },
|
||||
})).toBe(0)
|
||||
})
|
||||
|
||||
it('restore folds the tail past each usable row and refolds from init on version mismatch', async () => {
|
||||
const { ctx } = await harness()
|
||||
ctx.sessionProjections.register(marksUnit())
|
||||
ctx.sessionProjections.register(countUnit())
|
||||
const tail: SessionEvent[] = [
|
||||
{ type: 'test/mark', seq: 3, time: 3, data: { marks: ['new'] } },
|
||||
{ type: 'turn/end', seq: 4, time: 4, data: { turn: 1, reason: { kind: 'completed' } } },
|
||||
]
|
||||
// marks row usable (watermark 2, tail starts at 3); count row mismatched — but
|
||||
// a mismatch with baseSeq > 0 cannot silently refold: it throws for a re-read.
|
||||
expect(() => ctx.sessionProjections.restore({
|
||||
'test/marks': { ver: 1, seq: 2, val: { marks: ['old'] } },
|
||||
'test/count': { ver: 99, seq: 2, val: 3 },
|
||||
}, tail, 3)).toThrow(/re-read from seq 0/)
|
||||
// The full-log re-read (baseSeq 0) refolds the mismatched key from init.
|
||||
const full: SessionEvent[] = [
|
||||
{ type: 'turn/start', seq: 0, time: 0, data: { turn: 1 } },
|
||||
{ type: 'test/mark', seq: 1, time: 1, data: { marks: ['old'] } },
|
||||
{ type: 'test/mark', seq: 2, time: 2, data: { marks: ['old', '2'] } },
|
||||
...tail,
|
||||
]
|
||||
const { snapshot, checkpoint } = ctx.sessionProjections.restore({
|
||||
'test/marks': { ver: 1, seq: 2, val: { marks: ['old', '2'] } },
|
||||
'test/count': { ver: 99, seq: 2, val: 3 },
|
||||
}, full, 0)
|
||||
expect(snapshot.asOfSeq).toBe(4)
|
||||
expect(snapshot.values['test/marks']).toEqual({ marks: ['new'] })
|
||||
expect(snapshot.values['test/count']).toBe(5) // refolded from init over all 5 events
|
||||
// The refreshed rows sit at the served cut, ready for a durable write-back.
|
||||
expect(checkpoint['test/marks']).toEqual({ ver: 1, seq: 4, val: { marks: ['new'] } })
|
||||
expect(checkpoint['test/count']).toEqual({ ver: 1, seq: 4, val: 5 })
|
||||
})
|
||||
|
||||
it('restore over a suffix folds only past each row watermark and serves an exact empty-tail cut', async () => {
|
||||
const { ctx } = await harness()
|
||||
ctx.sessionProjections.register(marksUnit())
|
||||
ctx.sessionProjections.register(countUnit())
|
||||
const rows = {
|
||||
'test/marks': { ver: 1, seq: 4, val: { marks: ['done'] } },
|
||||
'test/count': { ver: 1, seq: 2, val: 3 },
|
||||
}
|
||||
const tail: SessionEvent[] = [
|
||||
{ type: 'turn/start', seq: 3, time: 3, data: { turn: 2 } },
|
||||
{ type: 'turn/end', seq: 4, time: 4, data: { turn: 2, reason: { kind: 'completed' } } },
|
||||
]
|
||||
const { snapshot } = ctx.sessionProjections.restore(rows, tail, 3)
|
||||
expect(snapshot.asOfSeq).toBe(4)
|
||||
// marks already covers the tail (watermark 4): nothing re-applied.
|
||||
expect(snapshot.values['test/marks']).toEqual({ marks: ['done'] })
|
||||
// count folds exactly seqs 3 and 4 on top of its checkpoint.
|
||||
expect(snapshot.values['test/count']).toBe(5)
|
||||
|
||||
// Empty tail (checkpoint is current): the cut sits at baseSeq - 1.
|
||||
const { snapshot: current } = ctx.sessionProjections.restore({
|
||||
'test/marks': { ver: 1, seq: 4, val: { marks: ['done'] } },
|
||||
'test/count': { ver: 1, seq: 4, val: 5 },
|
||||
}, [], 5)
|
||||
expect(current.asOfSeq).toBe(4)
|
||||
expect(current.values['test/count']).toBe(5)
|
||||
})
|
||||
|
||||
it('viewCheckpoint serves version-matching rows without any log and skips mismatched keys', async () => {
|
||||
const { ctx } = await harness()
|
||||
ctx.sessionProjections.register(marksUnit())
|
||||
ctx.sessionProjections.register(countUnit())
|
||||
const values = ctx.sessionProjections.viewCheckpoint({
|
||||
'test/marks': { ver: 1, seq: 4, val: { marks: ['stored'] } },
|
||||
'test/count': { ver: 99, seq: 4, val: 5 }, // mismatched: absent
|
||||
})
|
||||
expect(values['test/marks']).toEqual({ marks: ['stored'] })
|
||||
expect('test/count' in values).toBe(false)
|
||||
expect(ctx.sessionProjections.viewCheckpoint({})).toEqual({})
|
||||
})
|
||||
|
||||
it('restore rejects a row claiming events past the supplied log end (shrunk log ⇒ re-read)', async () => {
|
||||
const { ctx } = await harness()
|
||||
ctx.sessionProjections.register(countUnit())
|
||||
const rows = { 'test/count': { ver: 1, seq: 9, val: 10 } }
|
||||
// The anchored floor sits ON the watermark, so the tail read must return
|
||||
// at least seq 9 from an intact log…
|
||||
const floor = ctx.sessionProjections.restoreFloor(rows)
|
||||
expect(floor).toBe(9)
|
||||
// …an intact log serves the anchor event and the checkpoint stands as-is.
|
||||
const anchor: SessionEvent = { type: 'turn/end', seq: 9, time: 9, data: { turn: 2, reason: { kind: 'completed' } } }
|
||||
expect(ctx.sessionProjections.restore(rows, [anchor], 9).snapshot.values['test/count']).toBe(10)
|
||||
// …while a log crash-repaired down to fewer events returns an empty tail:
|
||||
// the row overreaches the proven end and a tail read cannot fix this key.
|
||||
expect(() => ctx.sessionProjections.restore(rows, [], 9)).toThrow(/re-read from seq 0/)
|
||||
// The full re-read discards the overreaching row and refolds from init.
|
||||
const events: SessionEvent[] = [
|
||||
{ type: 'turn/start', seq: 0, time: 0, data: { turn: 1 } },
|
||||
{ type: 'turn/end', seq: 1, time: 1, data: { turn: 1, reason: { kind: 'completed' } } },
|
||||
]
|
||||
const { snapshot } = ctx.sessionProjections.restore(rows, events, 0)
|
||||
expect(snapshot.asOfSeq).toBe(1)
|
||||
expect(snapshot.values['test/count']).toBe(2)
|
||||
})
|
||||
|
||||
it('fails loud when a unit view violates its own schema (async unit output is unrepresentable)', async () => {
|
||||
const { ctx, session } = await harness()
|
||||
ctx.sessionProjections.register({
|
||||
key: 'test/marks',
|
||||
schema: z.object({ marks: z.array(z.string()) }),
|
||||
init: () => null as MarksState,
|
||||
apply: state => state,
|
||||
// A Promise (what an accidentally-async view would return) is not the
|
||||
// declared shape: the boundary parse rejects it before it leaves.
|
||||
view: () => Promise.resolve({ marks: [] }) as never,
|
||||
stateVersion: 1,
|
||||
})
|
||||
expect(() => ctx.sessionProjections.snapshot(session)).toThrow()
|
||||
})
|
||||
})
|
||||
24
packages/session/session-projection/tsconfig.json
Normal file
24
packages/session/session-projection/tsconfig.json
Normal file
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
{
|
||||
"path": "../../support/invariants"
|
||||
}
|
||||
]
|
||||
}
|
||||
6
packages/session/session-telemetry-otel/README.i18n.yaml
Normal file
6
packages/session/session-telemetry-otel/README.i18n.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/telemetry/session-telemetry-otel/README.md
|
||||
README.md: 585995ce409255df9608bc33b76625374bc67669
|
||||
README.zh.md: 6d2cfa4d492cee7f90d557c83f5c1ab3c730c6e5
|
||||
54
packages/session/session-telemetry-otel/README.md
Normal file
54
packages/session/session-telemetry-otel/README.md
Normal file
@@ -0,0 +1,54 @@
|
||||
# @deepseek-ai/dsh-session-telemetry-otel
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
The OpenTelemetry backend for [the telemetry seam](../session-telemetry/) — the only entry a deployment loads. Its `mode` decides whether the seam follows session events live, replays the canonical log only at recorded feedback, or keeps telemetry local. Uploading modes compose the OTel JS SDK as-is (`LoggerProvider` → `BatchLogRecordProcessor` → OTLP/HTTP log exporter) and map each handed-over record onto `logger.emit()`, under two instrumentation scopes: ledger records on `@deepseek-ai/dsh-session-telemetry-otel`, operational records on `@deepseek-ai/dsh-session-telemetry-otel/ops`. Resource identity contains `service.name`/`service.version` from `dsh-llm`'s `APP_IDENTITY` plus this package's anonymous `user.id` (`$DSH_HOME/.userid`, a random UUID created on first use and reset by deleting the file), carried once per export batch rather than per record.
|
||||
|
||||
## Config
|
||||
|
||||
```yaml
|
||||
- id: telemetry-otel
|
||||
name: '@deepseek-ai/dsh-session-telemetry-otel'
|
||||
config:
|
||||
mode: FULL # FULL (default), FEEDBACK_ONLY, or DISABLED
|
||||
shutdownTimeoutMillis: 3000 # optional; defaults to 3000
|
||||
exporter: # passed verbatim to the SDK's OTLP/HTTP log exporter
|
||||
url: https://collector.example.com/v1/logs
|
||||
headers:
|
||||
authorization: !!js `Bearer ${process.env.OTLP_TOKEN}`
|
||||
processor: {} # optional; passed verbatim to BatchLogRecordProcessor
|
||||
```
|
||||
|
||||
| `mode` | Behavior |
|
||||
|---|---|
|
||||
| `FULL` | Default. Each projected record, including lifecycle ops records, is handed to the OTel SDK immediately. |
|
||||
| `FEEDBACK_ONLY` | Each `feedback/record` replays, projects, and redacts the canonical session-log suffix through that event. Later records wait for another feedback event and remain local if none arrives. |
|
||||
| `DISABLED` | No coordinator, provider, processor, or exporter is constructed. No telemetry record leaves the process. A `feedback/record` logs `session telemetry is DISABLED; nothing will be shared and this feedback remains local`; the event remains in the local session log. |
|
||||
|
||||
Programmatic TypeScript configuration uses the exported `TelemetryMode` enum (`TelemetryMode.FULL`, `TelemetryMode.FEEDBACK_ONLY`, or `TelemetryMode.DISABLED`); raw string literals are not assignable. Serialized Cordis configuration continues to use the string values shown above.
|
||||
|
||||
Upload authorization is positive and fail-closed. An unknown direct-construction mode fails before transport configuration is read. Only `FULL` accepts direct `ctx.telemetry.emit()` calls. `FEEDBACK_ONLY` gives its on-demand coordinator a private backend capability and treats only the exact `feedback/record` object already stored at `session.events[event.seq]` as consent; an independently emitted bus value is ignored. `DISABLED` never constructs the SDK pipeline, even when exporter options are present.
|
||||
|
||||
`exporter.url` is required in `FULL` and `FEEDBACK_ONLY`, has no default, and must parse as `http(s)`; it is optional and unused in `DISABLED`. In uploading modes, `shutdownTimeoutMillis` is a positive finite DSH-owned outer deadline that defaults to 3000 ms, and a non-positive-integer `processor.maxExportBatchSize` also fails at plugin load because the SDK accepts it but then hangs on shutdown. Both SDK blocks pass through whole: every `OTLPExporterNodeConfigBase` field (`headers`, `timeoutMillis`, `compression`, `keepAlive`, …) reaches the exporter, and batching, export cadence (`scheduledDelayMillis`), retry, queue bounds, and loss policy under sustained failure are SDK behavior tuned through `processor`. The backend implements no `flush()`: the batch processor owns ordinary flushing. During shutdown, OTel awaits `exporter.forceFlush()` before the processor's `exportTimeoutMillis`-bounded completion promise; if that transport promise never settles, this package abandons the wait at `shutdownTimeoutMillis`, logs the contained shutdown failure through the coordinator, and lets application teardown continue. The deadline cannot cancel the SDK transport, so records still pending then may be lost at process exit.
|
||||
|
||||
## What leaves the machine
|
||||
|
||||
In uploading modes, records carry the complete `event.data` as the seam's `telemetry/record` waterfall returns it — user and assistant message content, tool arguments and results (command output, file contents), the full system prompt and tool schemas (`request/header`), todo text, compaction summaries, hook `stderrSummary`, feedback text, and the session `cwd` (a local path). The seam ships no redaction rules: with no `telemetry/record` listener mounted, that is the raw captured copy, so a deployment exporting beyond a trusted boundary mounts its own rules (see [the seam README](../session-telemetry/README.md#the-redact-waterfall)). `FULL` runs redaction at append time; `FEEDBACK_ONLY` retains no telemetry copy and runs the currently mounted rules when feedback triggers canonical-log replay. Provider credentials never appear regardless: adapter API keys are constructor parameters, not session events, so they are structurally absent from the log and therefore from telemetry. `DISABLED` does not construct the SDK pipeline or hand any capture to a backend.
|
||||
|
||||
## Field mapping
|
||||
|
||||
Seam record → SDK log record: `time` → `timestamp`/`observedTimestamp`; `severity` → `severityNumber`/`severityText` (INFO 9 / WARN 13 / ERROR 17); `body` → the structured log body; `attributes` verbatim. Receivers dedupe on `(session.id, event.seq)` and alert on severity. In `FULL`, they may also detect crashes by `shutdown`-record absence: the marker is emitted at the session's own disposal or application teardown, and a marker followed by more events is a telemetry reload. In `FEEDBACK_ONLY`, a released prefix normally has no later `shutdown` marker, so its absence is not a crash signal. Streams are not self-contained across lineage: a resumed session continues its own id's stream from where the previous process left off, and a forked session's stream starts at its inherited boundary — its prefix lives in the parent's stream, stitched via `session.parent_id` + `session.seed_length`. A resumed local log may contain synthetic closers that were never exported; the wire stream stays faithful to records actually handed to the SDK.
|
||||
|
||||
## Model Experience
|
||||
|
||||
None, as the backend only forwards the seam's redacted records into the OTel SDK pipeline; it never contributes to a model request.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
None; this package neither assembles nor sends a provider request.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **Upstream experimental tree** — `@opentelemetry/sdk-logs` is still published from the upstream experimental tree; SDK API churn lands here and only here — the seam contract does not move.
|
||||
- **Live-collector behavior belongs to the SDK exporter** — authentication, TLS, throttling, and other real OTLP deployment behavior follow the upstream SDK rather than a package-owned compatibility layer.
|
||||
- **Feedback-time snapshot** — `FEEDBACK_ONLY` retains no telemetry-owned copy before feedback. It reads and redacts the current canonical log when feedback is recorded; a crash before feedback uploads nothing, and policy changes before feedback affect what that replay exports.
|
||||
54
packages/session/session-telemetry-otel/README.zh.md
Normal file
54
packages/session/session-telemetry-otel/README.zh.md
Normal file
@@ -0,0 +1,54 @@
|
||||
# @deepseek-ai/dsh-session-telemetry-otel
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
[遥测（telemetry）seam](../session-telemetry/) 的 OpenTelemetry 后端，也是部署方唯一要加载的条目。其 `mode` 决定 seam 是实时跟随会话事件、仅在记录反馈时回放权威日志，还是将遥测留在本地。上传模式会原样组合 OTel JS SDK（`LoggerProvider` → `BatchLogRecordProcessor` → OTLP/HTTP 日志导出器），把每条已交接记录映射到 `logger.emit()`，并使用两个插桩作用域（instrumentation scope）：ledger 记录挂在 `@deepseek-ai/dsh-session-telemetry-otel` 下，运维记录挂在 `@deepseek-ai/dsh-session-telemetry-otel/ops` 下。资源身份包含 `service.name`/`service.version`（来自 `dsh-llm` 的 `APP_IDENTITY`），以及本包的匿名 `user.id`（`$DSH_HOME/.userid`；首次使用时创建的随机 UUID，删除该文件可重置）；这些身份随每个导出批次携带一次，而非逐条记录携带。
|
||||
|
||||
## 配置
|
||||
|
||||
```yaml
|
||||
- id: telemetry-otel
|
||||
name: '@deepseek-ai/dsh-session-telemetry-otel'
|
||||
config:
|
||||
mode: FULL # FULL (default), FEEDBACK_ONLY, or DISABLED
|
||||
shutdownTimeoutMillis: 3000 # optional; defaults to 3000
|
||||
exporter: # passed verbatim to the SDK's OTLP/HTTP log exporter
|
||||
url: https://collector.example.com/v1/logs
|
||||
headers:
|
||||
authorization: !!js `Bearer ${process.env.OTLP_TOKEN}`
|
||||
processor: {} # optional; passed verbatim to BatchLogRecordProcessor
|
||||
```
|
||||
|
||||
| `mode` | 行为 |
|
||||
|---|---|
|
||||
| `FULL` | 默认值。每条已投影记录都立即交给 OTel SDK，包括生命周期运维记录。 |
|
||||
| `FEEDBACK_ONLY` | 每个 `feedback/record` 都会回放权威会话日志中截至该事件的后缀，并进行投影与脱敏。后续记录等待下一个反馈事件；如果没有后续反馈，则留在本地。 |
|
||||
| `DISABLED` | 不构造协调器、提供方、处理器或导出器。没有遥测记录会离开进程。`feedback/record` 会记录 `session telemetry is DISABLED; nothing will be shared and this feedback remains local`；该事件留在本地会话日志中。 |
|
||||
|
||||
程序化 TypeScript 配置使用导出的 `TelemetryMode` 枚举（`TelemetryMode.FULL`、`TelemetryMode.FEEDBACK_ONLY` 或 `TelemetryMode.DISABLED`）；原始字符串字面量不可赋值。序列化后的 Cordis 配置继续使用上表所示的字符串值。
|
||||
|
||||
上传授权采用显式许可，且为 fail-closed。通过直接构造传入未知模式时，会在读取传输配置前失败。只有 `FULL` 接受对 `ctx.telemetry.emit()` 的直接调用。`FEEDBACK_ONLY` 向其按需协调器提供私有后端能力，并且仅在 `feedback/record` 对象已经存储于 `session.events[event.seq]` 且对象身份完全相同时，才将其视为同意；独立发出的总线值会被忽略。即使存在导出器选项，`DISABLED` 也绝不会构造 SDK 流水线。
|
||||
|
||||
`exporter.url` 在 `FULL` 与 `FEEDBACK_ONLY` 中必填，无默认值，且必须能解析为 `http(s)`；在 `DISABLED` 中可省略且不使用。在上传模式中，`shutdownTimeoutMillis` 是由 DSH 管理的有限正数外层截止时间，默认值为 3000 ms；`processor.maxExportBatchSize` 不是正整数时也会在插件加载时失败，因为 SDK 会接受该值，随后却在关闭时挂起。两个 SDK 配置块都整体透传（passthrough）：`OTLPExporterNodeConfigBase` 的每个字段（`headers`、`timeoutMillis`、`compression`、`keepAlive` 等）都会到达导出器；批处理、导出节奏（`scheduledDelayMillis`）、重试、队列上限，以及持续失败下的丢失策略，都是通过 `processor` 调节的 SDK 行为。该后端不实现 `flush()`：常规 flush 由批处理器负责。关闭期间，OTel 会先等待 `exporter.forceFlush()`，再进入受处理器 `exportTimeoutMillis` 限制的完成 promise；如果该传输 promise 始终不结算，本包会在 `shutdownTimeoutMillis` 到期时放弃等待，通过协调器记录已隔离的关闭失败，并让应用继续拆卸。该截止时间无法取消 SDK 传输，因此届时仍待处理的记录可能在进程退出时丢失。
|
||||
|
||||
## 哪些数据会离开本机
|
||||
|
||||
在上传模式中，记录携带完整的 `event.data`，内容以 seam 的 `telemetry/record` waterfall（瀑布式事件）返回的结果为准：用户与 assistant 消息内容、工具参数与工具结果（命令输出、文件内容）、完整的系统提示词与工具 schema（`request/header`）、todo 文本、压缩（compaction）摘要、钩子的 `stderrSummary`、反馈文本，以及会话 `cwd`（一个本地路径）。seam 不带任何脱敏规则：未挂载 `telemetry/record` 监听器时，导出的就是捕获原样的副本，因此向可信边界之外导出的部署方要挂载自己的规则（见 [seam README](../session-telemetry/README.md#the-redact-waterfall)）。`FULL` 在追加时运行脱敏；`FEEDBACK_ONLY` 不保留遥测副本，而是在反馈触发权威日志回放时运行当时挂载的规则。无论如何，提供方凭据都不会出现：适配器的 API key 是构造函数参数而非会话事件，因此它们在结构上就不存在于日志中，也就不存在于遥测中。`DISABLED` 不会构造 SDK 流水线，也不会将任何捕获内容交给后端。
|
||||
|
||||
## 字段映射
|
||||
|
||||
seam 记录 → SDK 日志记录：`time` → `timestamp`/`observedTimestamp`；`severity` → `severityNumber`/`severityText`（INFO 9 / WARN 13 / ERROR 17）；`body` → 结构化日志 body；`attributes` 原样照搬。接收端基于 `(session.id, event.seq)` 去重，并按严重级别告警。在 `FULL` 中，接收端还可通过缺少 `shutdown` 记录检测崩溃：该标记在会话自身 dispose（资源释放）或应用关闭时发出；标记之后出现更多事件，说明遥测发生了重载。在 `FEEDBACK_ONLY` 中，已释放的前缀通常不包含随后的 `shutdown` 标记，因此缺少该标记不是崩溃信号。跨谱系（lineage）的流并不自足：恢复的会话在其自身 id 的流上从上一个进程停止之处继续；fork 出的会话的流从继承边界开始，其前缀位于父会话的流中，由接收端基于 `session.parent_id` + `session.seed_length` 拼接。恢复后的本地日志可能包含从未导出的合成关闭事件；协议流忠实于实际交给 SDK 的记录。
|
||||
|
||||
## 模型体验
|
||||
|
||||
无。该后端只把 seam 脱敏后的记录转发进 OTel SDK 流水线；它绝不向模型请求贡献任何内容。
|
||||
|
||||
#### KV Cache 影响
|
||||
|
||||
无；本包既不组装也不发送提供方请求。
|
||||
|
||||
## 已知限制与暂缓事项
|
||||
|
||||
- **上游实验性源码树**：`@opentelemetry/sdk-logs` 仍从上游实验性（experimental）源码树发布；SDK API 的变动只会落在本包，也仅落在本包；seam 契约不动。
|
||||
- **真实 collector 行为属于 SDK 导出器**：身份验证、TLS、限流及其他真实 OTLP 部署行为遵循上游 SDK，不由本包自有兼容层处理。
|
||||
- **反馈时快照**：`FEEDBACK_ONLY` 在反馈前不保留遥测自有副本。记录反馈时，它读取并脱敏当前的权威日志；反馈前发生崩溃时什么都不上传，而反馈前的策略变更会影响该次回放的导出内容。
|
||||
57
packages/session/session-telemetry-otel/package.json
Normal file
57
packages/session/session-telemetry-otel/package.json
Normal file
@@ -0,0 +1,57 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-session-telemetry-otel",
|
||||
"description": "OpenTelemetry backend for the DeepSeek Harness telemetry seam: hands captured session records to the OTel JS SDK's log pipeline",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./invariant": {
|
||||
"types": "./lib/types/invariant.d.ts",
|
||||
"default": "./lib/invariant.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"@opentelemetry/api": "^1.9.1",
|
||||
"@opentelemetry/api-logs": "^0.220.0",
|
||||
"@opentelemetry/exporter-logs-otlp-http": "^0.220.0",
|
||||
"@opentelemetry/otlp-exporter-base": "^0.220.0",
|
||||
"@opentelemetry/resources": "^2.9.0",
|
||||
"@opentelemetry/sdk-logs": "^0.220.0",
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-command-feedback": "^0.0.1",
|
||||
"@deepseek-ai/dsh-brand": "^0.0.1",
|
||||
"@deepseek-ai/dsh-invariants": "^0.0.1",
|
||||
"@deepseek-ai/dsh-llm": "^0.0.1",
|
||||
"@deepseek-ai/dsh-paths": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session-telemetry": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@cordisjs/plugin-loader": "workspace:^",
|
||||
"@deepseek-ai/dsh-command-feedback": "workspace:^",
|
||||
"@deepseek-ai/dsh-brand": "workspace:^",
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-paths": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-telemetry": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
289
packages/session/session-telemetry-otel/src/index.ts
Normal file
289
packages/session/session-telemetry-otel/src/index.ts
Normal file
@@ -0,0 +1,289 @@
|
||||
/**
|
||||
* OpenTelemetry backend for the DeepSeek Harness telemetry seam.
|
||||
*
|
||||
* Composes the OTel JS SDK as-is — a `LoggerProvider` with a
|
||||
* `BatchLogRecordProcessor` and an OTLP/HTTP log exporter — and maps each
|
||||
* record handed over by the seam onto `logger.emit()`. Per the seam's
|
||||
* boundary axiom, everything downstream of that call (batching, retry,
|
||||
* queueing, loss policy) is the SDK's documented behavior, configured
|
||||
* verbatim through the `exporter`/`processor` passthroughs. This package owns
|
||||
* capture mode and an outer shutdown deadline: the SDK's export timeout does
|
||||
* not bound its preceding `forceFlush()` wait.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-session-telemetry-otel
|
||||
*/
|
||||
|
||||
import { createRequire } from 'node:module'
|
||||
import z from 'schemastery'
|
||||
import type { Context } from 'cordis'
|
||||
import type {} from '@deepseek-ai/dsh-command-feedback'
|
||||
import {
|
||||
Telemetry,
|
||||
TelemetryCoordinator,
|
||||
type TelemetryBackend,
|
||||
type TelemetryRecord,
|
||||
type TelemetrySeverity,
|
||||
} from '@deepseek-ai/dsh-session-telemetry'
|
||||
import { APP_IDENTITY } from '@deepseek-ai/dsh-llm'
|
||||
import { getOrCreateAnonymousUserId } from './user-id.ts'
|
||||
import {
|
||||
BatchLogRecordProcessor,
|
||||
LoggerProvider,
|
||||
type BatchLogRecordProcessorOptions,
|
||||
} from '@opentelemetry/sdk-logs'
|
||||
import { OTLPLogExporter } from '@opentelemetry/exporter-logs-otlp-http'
|
||||
import type { OTLPExporterNodeConfigBase } from '@opentelemetry/otlp-exporter-base'
|
||||
import { SeverityNumber, type AnyValue, type Logger } from '@opentelemetry/api-logs'
|
||||
import { resourceFromAttributes } from '@opentelemetry/resources'
|
||||
|
||||
// The package's own manifest is the single source of the instrumentation-scope
|
||||
// version (same pattern as dsh-llm's attribution identity).
|
||||
const { version } = createRequire(import.meta.url)('../package.json') as { version: string }
|
||||
|
||||
/** Session-sharing policy selected by {@link Config.mode}. */
|
||||
export enum TelemetryMode {
|
||||
FULL = 'FULL',
|
||||
FEEDBACK_ONLY = 'FEEDBACK_ONLY',
|
||||
DISABLED = 'DISABLED',
|
||||
}
|
||||
|
||||
/** Default session-sharing policy for schema and direct construction. */
|
||||
export const DEFAULT_TELEMETRY_MODE = TelemetryMode.FULL
|
||||
|
||||
const DISABLED_FEEDBACK_WARNING = 'session telemetry is DISABLED; nothing will be shared and this feedback remains local'
|
||||
const NON_CANONICAL_FEEDBACK_WARNING = 'session telemetry ignored a feedback event absent from the canonical session log'
|
||||
const DROP_RECORD: TelemetryBackend['emit'] = () => {}
|
||||
|
||||
/** Resolve the default and reject unknown runtime values before transport setup. */
|
||||
function resolveMode(mode: TelemetryMode | undefined): TelemetryMode {
|
||||
const resolved = mode ?? DEFAULT_TELEMETRY_MODE
|
||||
switch (resolved) {
|
||||
case TelemetryMode.FULL:
|
||||
case TelemetryMode.FEEDBACK_ONLY:
|
||||
case TelemetryMode.DISABLED:
|
||||
return resolved
|
||||
default:
|
||||
return assertNever(resolved)
|
||||
}
|
||||
}
|
||||
|
||||
/** Fail closed when direct construction bypasses the runtime config schema. */
|
||||
function assertNever(value: never): never {
|
||||
throw new Error(`session-telemetry-otel: unsupported mode ${JSON.stringify(value)}`)
|
||||
}
|
||||
|
||||
/**
|
||||
* Plugin configuration: one sharing policy, two verbatim SDK option shapes,
|
||||
* and one DSH-owned shutdown bound. Uploading modes validate their endpoint
|
||||
* and shutdown deadline at plugin load; `DISABLED` reads neither.
|
||||
*/
|
||||
export interface Config {
|
||||
/** Sharing policy; defaults to immediate `FULL` delivery. */
|
||||
mode?: TelemetryMode
|
||||
/**
|
||||
* Passed verbatim to the SDK's OTLP/HTTP log exporter — the complete
|
||||
* `OTLPExporterNodeConfigBase` shape (`headers`, `timeoutMillis`,
|
||||
* `compression`, `keepAlive`, …), owned and documented by the SDK. `url`
|
||||
* is the one field this package requires and validates itself.
|
||||
*/
|
||||
exporter?: OTLPExporterNodeConfigBase & {
|
||||
/** Full logs endpoint (e.g. `https://collector.example.com/v1/logs`). Required outside `DISABLED`; validated at load. */
|
||||
url?: string
|
||||
}
|
||||
/**
|
||||
* Passed verbatim to `BatchLogRecordProcessor` (minus the exporter slot,
|
||||
* which this plugin fills); the SDK owns and documents these knobs.
|
||||
*/
|
||||
processor?: Omit<BatchLogRecordProcessorOptions, 'exporter'>
|
||||
/** Maximum time spent awaiting the SDK provider's complete shutdown path. */
|
||||
shutdownTimeoutMillis?: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Schemastery validator for {@link Config}; cordis runs it before the plugin
|
||||
* starts. Shape-level only — load-bearing value checks live in the constructor
|
||||
* so their errors name the fields. Both SDK slots are opaque passthroughs:
|
||||
* the SDK owns their shapes and validates its own options;
|
||||
* re-declaring them field-by-field here would violate the boundary axiom
|
||||
* (and silently drop every field not re-declared).
|
||||
*/
|
||||
export const Config: z<Config> = z.object({
|
||||
mode: z.union(Object.values(TelemetryMode)).default(DEFAULT_TELEMETRY_MODE),
|
||||
exporter: z.any(),
|
||||
processor: z.any(),
|
||||
shutdownTimeoutMillis: z.number(),
|
||||
})
|
||||
|
||||
/** Default outer allowance for the SDK's complete shutdown sequence. */
|
||||
export const DEFAULT_SHUTDOWN_TIMEOUT_MILLIS = 3_000
|
||||
|
||||
// Node clamps larger timer delays to one millisecond. This is a runtime
|
||||
// protocol limit, not a deployment default.
|
||||
const MAX_TIMER_DELAY_MILLIS = 2_147_483_647
|
||||
|
||||
/** Severity mapping from the seam's three-level vocabulary to OTel severity numbers. */
|
||||
const SEVERITY: Record<TelemetrySeverity, { severityNumber: SeverityNumber; severityText: string }> = {
|
||||
info: { severityNumber: SeverityNumber.INFO, severityText: 'INFO' },
|
||||
warn: { severityNumber: SeverityNumber.WARN, severityText: 'WARN' },
|
||||
error: { severityNumber: SeverityNumber.ERROR, severityText: 'ERROR' },
|
||||
}
|
||||
|
||||
/**
|
||||
* The backend plugin — the only entry a deployment loads. It always registers
|
||||
* the `telemetry` service (duplicate load throws). Uploading modes wire the SDK
|
||||
* pipeline and compose {@link TelemetryCoordinator}; `DISABLED` constructs no
|
||||
* SDK state and listens only to warn when recorded feedback stays local.
|
||||
*/
|
||||
export class TelemetryOtel extends Telemetry {
|
||||
static inject = ['sessions']
|
||||
static Config = Config
|
||||
|
||||
private readonly directEmit: TelemetryBackend['emit']
|
||||
private readonly provider: LoggerProvider | undefined
|
||||
private readonly shutdownTimeoutMillis: number
|
||||
|
||||
constructor(ctx: Context, config: Config) {
|
||||
const mode = resolveMode(config.mode)
|
||||
super(ctx)
|
||||
if (mode === TelemetryMode.DISABLED) {
|
||||
this.directEmit = DROP_RECORD
|
||||
this.provider = undefined
|
||||
this.shutdownTimeoutMillis = DEFAULT_SHUTDOWN_TIMEOUT_MILLIS
|
||||
ctx.on('session/event', (_session, event) => {
|
||||
if (event.type === 'feedback/record') ctx.logger.warn(DISABLED_FEEDBACK_WARNING)
|
||||
})
|
||||
return
|
||||
}
|
||||
|
||||
const url = config.exporter?.url
|
||||
if (url === undefined || url.length === 0) {
|
||||
throw new Error('session-telemetry-otel: exporter.url is required (the full OTLP logs endpoint)')
|
||||
}
|
||||
let parsed: URL
|
||||
try {
|
||||
parsed = new URL(url)
|
||||
} catch {
|
||||
// Re-thrown as a config error: the only way here is a malformed url string.
|
||||
throw new Error(`session-telemetry-otel: exporter.url is not a valid URL: ${JSON.stringify(url)}`)
|
||||
}
|
||||
if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') {
|
||||
throw new Error(`session-telemetry-otel: exporter.url must be http(s), got ${parsed.protocol}`)
|
||||
}
|
||||
// The one processor field checked beyond the SDK's own validation: the
|
||||
// SDK accepts a non-positive batch size, but its shutdown drain then
|
||||
// splices empty batches without consuming the queue — dispose would hang
|
||||
// forever with records queued. Misconfiguration fails at load instead.
|
||||
const batchSize = config.processor?.maxExportBatchSize
|
||||
if (batchSize !== undefined && (!Number.isInteger(batchSize) || batchSize < 1)) {
|
||||
throw new Error(`session-telemetry-otel: processor.maxExportBatchSize must be a positive integer, got ${String(batchSize)}`)
|
||||
}
|
||||
const shutdownTimeoutMillis = config.shutdownTimeoutMillis ?? DEFAULT_SHUTDOWN_TIMEOUT_MILLIS
|
||||
if (!Number.isFinite(shutdownTimeoutMillis) || shutdownTimeoutMillis <= 0 || shutdownTimeoutMillis > MAX_TIMER_DELAY_MILLIS) {
|
||||
throw new Error(`session-telemetry-otel: shutdownTimeoutMillis must be a positive finite number no greater than ${MAX_TIMER_DELAY_MILLIS}, got ${String(shutdownTimeoutMillis)}`)
|
||||
}
|
||||
this.shutdownTimeoutMillis = shutdownTimeoutMillis
|
||||
this.provider = new LoggerProvider({
|
||||
resource: resourceFromAttributes({
|
||||
'service.name': APP_IDENTITY.product,
|
||||
'service.version': APP_IDENTITY.version,
|
||||
// OTel semconv's standard user attribute, carried once per export
|
||||
// batch on the Resource rather than per record: the collector
|
||||
// aggregates by Resource, and the id is process-stable anyway.
|
||||
'user.id': getOrCreateAnonymousUserId(),
|
||||
}),
|
||||
processors: [
|
||||
new BatchLogRecordProcessor({
|
||||
...config.processor,
|
||||
// The complete validated exporter object, verbatim: every SDK
|
||||
// option (`timeoutMillis`, `compression`, `keepAlive`, …) reaches
|
||||
// the exporter — rebuilding selected fields here would silently
|
||||
// ignore the rest. App identity travels in the Resource
|
||||
// (service.name/version); the transport-level user-agent is the
|
||||
// SDK's own, per the axiom.
|
||||
exporter: new OTLPLogExporter(config.exporter),
|
||||
}),
|
||||
],
|
||||
})
|
||||
const ledger = this.provider.getLogger('@deepseek-ai/dsh-session-telemetry-otel', version)
|
||||
const ops = this.provider.getLogger('@deepseek-ai/dsh-session-telemetry-otel/ops', version)
|
||||
const enqueue: TelemetryBackend['emit'] = (record) => {
|
||||
const logger: Logger = record.channel === 'ops' ? ops : ledger
|
||||
logger.emit({
|
||||
timestamp: record.time,
|
||||
observedTimestamp: record.time,
|
||||
...SEVERITY[record.severity],
|
||||
// JSON-serializable by the seam's contract (validated at Session.append),
|
||||
// which is exactly the AnyValue subset.
|
||||
body: record.body as AnyValue,
|
||||
attributes: record.attributes,
|
||||
})
|
||||
}
|
||||
const backend: TelemetryBackend = {
|
||||
emit: enqueue,
|
||||
shutdown: () => this.shutdown(),
|
||||
}
|
||||
if (mode === TelemetryMode.FULL) {
|
||||
this.directEmit = enqueue
|
||||
new TelemetryCoordinator(ctx, backend, 'live')
|
||||
return
|
||||
}
|
||||
this.directEmit = DROP_RECORD
|
||||
const coordinator = new TelemetryCoordinator(ctx, backend, 'on-demand')
|
||||
ctx.on('session/event', (session, event) => {
|
||||
if (event.type !== 'feedback/record') return
|
||||
// Consent is the committed record, not an independently emitted bus value.
|
||||
if (session.events[event.seq] !== event) {
|
||||
ctx.logger.warn(NON_CANONICAL_FEEDBACK_WARNING)
|
||||
return
|
||||
}
|
||||
coordinator.captureSession(session, event.seq)
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Hand a direct service record to the SDK only in `FULL`. Direct calls are
|
||||
* no-ops in `FEEDBACK_ONLY` and `DISABLED`; feedback replay uses a private
|
||||
* backend capability created only for the canonical feedback listener.
|
||||
* @param record - the logical record offered directly to the service.
|
||||
*/
|
||||
emit(record: TelemetryRecord): void {
|
||||
this.directEmit(record)
|
||||
}
|
||||
|
||||
// The seam's optional flush() hint is deliberately NOT implemented. The
|
||||
// batch processor exports on its own cadence (`processor.scheduledDelayMillis`,
|
||||
// the SDK's documented knob), and this backend is the SDK pipeline's only
|
||||
// caller — forwarding the hint to `forceFlush()` was the sole source of
|
||||
// concurrent flushes, whose undocumented interactions with shutdown's
|
||||
// internal drain (concurrent-flush guard, provider-level flush timeout)
|
||||
// silently dropped tail records. Removal history and the revival trigger:
|
||||
// the revival Agent Note.
|
||||
|
||||
/**
|
||||
* Ask the SDK to drain and quiesce, but reject after the backend-owned
|
||||
* deadline. OTel's processor export timeout wraps `exportCompleted` only;
|
||||
* shutdown awaits `exporter.forceFlush()` first, which can remain pending
|
||||
* when the transport never obtains a socket. The provider promise remains
|
||||
* observed after the deadline so a later rejection cannot become unhandled.
|
||||
* `DISABLED` has no provider and resolves immediately.
|
||||
* @returns resolves when the SDK pipeline quiesces or is disabled, or rejects at the configured deadline.
|
||||
*/
|
||||
async shutdown(): Promise<void> {
|
||||
if (this.provider === undefined) return
|
||||
const providerShutdown = this.provider.shutdown()
|
||||
let timer: ReturnType<typeof setTimeout> | undefined
|
||||
const deadline = new Promise<never>((_resolve, reject) => {
|
||||
timer = setTimeout(() => {
|
||||
reject(new Error(`session-telemetry-otel: provider shutdown exceeded ${this.shutdownTimeoutMillis}ms`))
|
||||
}, this.shutdownTimeoutMillis)
|
||||
})
|
||||
try {
|
||||
await Promise.race([providerShutdown, deadline])
|
||||
} finally {
|
||||
/* v8 ignore else -- the Promise executor assigns timer synchronously before this race starts. */
|
||||
if (timer !== undefined) clearTimeout(timer)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export default TelemetryOtel
|
||||
31
packages/session/session-telemetry-otel/src/invariant.ts
Normal file
31
packages/session/session-telemetry-otel/src/invariant.ts
Normal file
@@ -0,0 +1,31 @@
|
||||
/**
|
||||
* Package-owned invariant companion for `@deepseek-ai/dsh-session-telemetry-otel`.
|
||||
* @module @deepseek-ai/dsh-session-telemetry-otel/invariant
|
||||
*/
|
||||
|
||||
/* jscpd:ignore-start */
|
||||
import type { Context } from 'cordis'
|
||||
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
|
||||
|
||||
const PACKAGE_NAME = '@deepseek-ai/dsh-session-telemetry-otel'
|
||||
|
||||
/** Cordis companion plugin name. */
|
||||
export const name = 'session-telemetry-otel-invariant'
|
||||
/** Service required before the companion can reserve package ownership. */
|
||||
export const inject = ['invariants']
|
||||
|
||||
/**
|
||||
* No runtime invariant: mode selection changes capture handoff, SDK setup, and
|
||||
* local diagnostics without mutating session or service state an independent
|
||||
* companion can compare. Export remains inside the SDK past the seam boundary.
|
||||
*/
|
||||
const install: InvariantInstaller = () => {}
|
||||
|
||||
/**
|
||||
* Register this package's invariant companion.
|
||||
* @param ctx - Cordis context carrying the invariant service.
|
||||
* @returns the installed registration's disposer after setup succeeds.
|
||||
*/
|
||||
export const apply = (ctx: Context): Promise<() => void> =>
|
||||
Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
|
||||
/* jscpd:ignore-end */
|
||||
103
packages/session/session-telemetry-otel/src/user-id.ts
Normal file
103
packages/session/session-telemetry-otel/src/user-id.ts
Normal file
@@ -0,0 +1,103 @@
|
||||
/**
|
||||
* Per-harness-home anonymous user id for the OTel Resource.
|
||||
*
|
||||
* The id is a random UUID persisted as a bare line in `.userid` inside the
|
||||
* harness home resolved by {@link resolveDshHome} (`$DSH_HOME` > `~/.dsh`),
|
||||
* and never derived from the hostname, network address, git remote, or any
|
||||
* other identifying source — a derived id would make "anonymous" a fiction.
|
||||
* The id is scoped to the harness home, not the machine: every process
|
||||
* sharing one `$DSH_HOME` reports the same id, and deleting the file simply
|
||||
* mints a fresh identity on the next launch (loss is accepted by design).
|
||||
* This identity belongs to the OTel feed alone; the dsh-sdk launcher
|
||||
* telemetry keeps its own separate store.
|
||||
*
|
||||
* Reads and writes are synchronous so the backend constructor can call this
|
||||
* on its boot path, and the result is memoized per resolved file path: one
|
||||
* process touches the disk once, and a file deleted mid-run keeps the
|
||||
* process's id until the next launch.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-session-telemetry-otel/user-id
|
||||
*/
|
||||
|
||||
import { randomUUID } from 'node:crypto'
|
||||
import { mkdirSync, readFileSync, writeFileSync } from 'node:fs'
|
||||
import { dirname, join } from 'node:path'
|
||||
import type { Branded } from '@deepseek-ai/dsh-brand'
|
||||
import { resolveDshHome } from '@deepseek-ai/dsh-paths'
|
||||
|
||||
/** A harness-home-scoped anonymous user id (random UUID v4). */
|
||||
export type AnonymousUserId = Branded<'AnonymousUserId'>
|
||||
|
||||
/** File inside the harness home storing the id: a bare UUID line, no wrapper format. */
|
||||
export const USER_ID_FILE_NAME = '.userid'
|
||||
|
||||
const UUID_PATTERN = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i
|
||||
|
||||
/** Ambient seams for locating and generating the id; every field has a default. */
|
||||
export interface AnonymousUserIdOptions {
|
||||
/** Environment consulted for `DSH_HOME`; defaults to `process.env`. */
|
||||
env?: NodeJS.ProcessEnv
|
||||
/** UUID generator; defaults to `crypto.randomUUID` (test seam). */
|
||||
randomUUID?: () => string
|
||||
}
|
||||
|
||||
/** Process-lifetime memo keyed by resolved file path, so distinct test homes never share an id. */
|
||||
const memo = new Map<string, AnonymousUserId>()
|
||||
|
||||
/** Read a valid persisted id from the file, or `undefined` when absent/corrupt. */
|
||||
function readPersistedId(file: string): AnonymousUserId | undefined {
|
||||
let text: string
|
||||
try {
|
||||
text = readFileSync(file, 'utf8')
|
||||
} catch {
|
||||
// Absent or unreadable: the caller mints and persists a fresh id.
|
||||
return undefined
|
||||
}
|
||||
const value = text.trim()
|
||||
return UUID_PATTERN.test(value) ? (value as AnonymousUserId) : undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Return the harness home's anonymous user id, creating and persisting one on
|
||||
* first use. A concurrent first launch is settled by an exclusive-create
|
||||
* write: the loser rereads the winner's id. (A reread landing in the winner's
|
||||
* narrow create-to-write window can still yield two per-process ids for that
|
||||
* run; the next launch converges on the persisted one.) Persistence is
|
||||
* best-effort — a write failure (read-only home) still returns a usable id
|
||||
* for the current run so telemetry is never blocked.
|
||||
* @param options - home-location and UUID-generation seams.
|
||||
* @returns the stable per-harness-home anonymous user id.
|
||||
*/
|
||||
export function getOrCreateAnonymousUserId(options: AnonymousUserIdOptions = {}): AnonymousUserId {
|
||||
const file = join(resolveDshHome(undefined, options.env ?? process.env), USER_ID_FILE_NAME)
|
||||
const cached = memo.get(file)
|
||||
if (cached !== undefined) return cached
|
||||
|
||||
let id = readPersistedId(file)
|
||||
if (id === undefined) {
|
||||
const generate = options.randomUUID ?? randomUUID
|
||||
const created = generate() as AnonymousUserId
|
||||
try {
|
||||
mkdirSync(dirname(file), { recursive: true })
|
||||
writeFileSync(file, `${created}\n`, { encoding: 'utf8', flag: 'wx' })
|
||||
id = created
|
||||
} catch {
|
||||
// A wx refusal (EEXIST) covers both a concurrent winner and a
|
||||
// pre-existing corrupt file: the reread adopts a valid winner, and an
|
||||
// invalid reread falls through to the overwrite path. Non-EEXIST
|
||||
// failures (read-only home) land there too, accepted best-effort below.
|
||||
id = readPersistedId(file)
|
||||
if (id === undefined) {
|
||||
try {
|
||||
writeFileSync(file, `${created}\n`, 'utf8')
|
||||
} catch {
|
||||
// Best-effort persistence: keep the fresh id in memory even when the
|
||||
// home is unwritable, so this run still reports a consistent id.
|
||||
}
|
||||
id = created
|
||||
}
|
||||
}
|
||||
}
|
||||
memo.set(file, id)
|
||||
return id
|
||||
}
|
||||
@@ -0,0 +1,154 @@
|
||||
/**
|
||||
* REAL-composition tier: boot the examples-owned telemetry Loader fixture as
|
||||
* a subprocess (per testing policy, through the same app/boot path a
|
||||
* deployment uses), run one mocked-model turn with a real bash round trip,
|
||||
* and assert against what the mock OTLP collector actually received on the
|
||||
* wire: ledger mirroring, the deployment-mounted redact rule applied to the
|
||||
* exported copy, ops markers, and the untouched canonical log.
|
||||
*/
|
||||
|
||||
import { readFile, readdir } from 'node:fs/promises'
|
||||
import { join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { LOADER_SMOKE_TEST_TIMEOUT_MS, runLoaderSmoke } from '@deepseek-ai/dsh-loader-smoke'
|
||||
|
||||
const driver = fileURLToPath(new URL(
|
||||
'../../../../examples/headless-agent/tests/fixtures/telemetry-otel-driver.ts',
|
||||
import.meta.url,
|
||||
))
|
||||
const configPath = fileURLToPath(new URL(
|
||||
'../../../../examples/headless-agent/tests/fixtures/telemetry-otel.cordis.yml',
|
||||
import.meta.url,
|
||||
))
|
||||
const repoTsconfig = fileURLToPath(new URL('../../../../tsconfig.json', import.meta.url))
|
||||
|
||||
const FIXTURE_SECRET = 'sk-e2efixture1234567890'
|
||||
const FIXTURE_PLACEHOLDER = '[E2E-REDACTED]'
|
||||
|
||||
interface OtlpLogRecord {
|
||||
attributes?: { key: string; value: Record<string, unknown> }[]
|
||||
body?: unknown
|
||||
}
|
||||
|
||||
interface OtlpCapture {
|
||||
resourceLogs: {
|
||||
scopeLogs: {
|
||||
scope: { name: string }
|
||||
logRecords: OtlpLogRecord[]
|
||||
}[]
|
||||
}[]
|
||||
}
|
||||
|
||||
interface FixtureOutput {
|
||||
captures: OtlpCapture[]
|
||||
logContent: string
|
||||
}
|
||||
|
||||
async function jsonlFiles(dir: string): Promise<string[]> {
|
||||
const entries = await readdir(dir, { withFileTypes: true })
|
||||
const paths = await Promise.all(entries.map(async (entry) => {
|
||||
const path = join(dir, entry.name)
|
||||
if (entry.isDirectory()) return jsonlFiles(path)
|
||||
return entry.isFile() && entry.name.endsWith('.jsonl') ? [path] : []
|
||||
}))
|
||||
return paths.flat()
|
||||
}
|
||||
|
||||
async function readFixtureOutput(cwd: string): Promise<FixtureOutput> {
|
||||
const captures = JSON.parse(await readFile(join(cwd, 'otlp-captures.json'), 'utf8')) as OtlpCapture[]
|
||||
const logs = await jsonlFiles(join(cwd, '.sessions'))
|
||||
expect(logs).toHaveLength(1)
|
||||
return { captures, logContent: await readFile(logs[0] as string, 'utf8') }
|
||||
}
|
||||
|
||||
function allRecords(captures: OtlpCapture[]) {
|
||||
return captures.flatMap(capture => capture.resourceLogs.flatMap(resource =>
|
||||
resource.scopeLogs.flatMap(scoped => scoped.logRecords.map(record => ({ scope: scoped.scope.name, record })))))
|
||||
}
|
||||
|
||||
function eventTypes(captures: OtlpCapture[]): string[] {
|
||||
return allRecords(captures).flatMap(({ record }) =>
|
||||
record.attributes?.flatMap(attribute =>
|
||||
attribute.key === 'event.type' && typeof attribute.value['stringValue'] === 'string'
|
||||
? [attribute.value['stringValue']]
|
||||
: []) ?? [])
|
||||
}
|
||||
|
||||
describe('session-telemetry-otel through a real headless cordis.yml', () => {
|
||||
it('exports redacted ledger records to the collector while the canonical log keeps the secret', async () => {
|
||||
let output!: FixtureOutput
|
||||
const { stderr } = await runLoaderSmoke({
|
||||
label: 'session-telemetry-otel loader smoke',
|
||||
tempDirPrefix: 'telemetry-otel-e2e-',
|
||||
binScript: driver,
|
||||
libBinScript: driver,
|
||||
configPath,
|
||||
tsconfigPath: repoTsconfig,
|
||||
inspect: async (cwd) => { output = await readFixtureOutput(cwd) },
|
||||
})
|
||||
expect(stderr).not.toContain('UNHANDLED')
|
||||
|
||||
const records = allRecords(output.captures)
|
||||
expect(records.length).toBeGreaterThan(0)
|
||||
|
||||
const types = eventTypes(output.captures)
|
||||
for (const expected of ['turn/start', 'user/message', 'tool/call', 'tool/result', 'assistant/message', 'turn/end']) {
|
||||
expect(types, expected).toContain(expected)
|
||||
}
|
||||
expect(records.some(({ scope }) => scope.endsWith('/ops'))).toBe(true)
|
||||
|
||||
// The deployment-mounted rule on the wire: the fixture credential never
|
||||
// leaves the process, its surrounding prose does, and the placeholder
|
||||
// marks the spot — the seam itself ships no rules.
|
||||
const wire = JSON.stringify(output.captures)
|
||||
expect(wire).not.toContain(FIXTURE_SECRET)
|
||||
expect(wire).toContain(FIXTURE_PLACEHOLDER)
|
||||
expect(wire).toContain('prove telemetry with key')
|
||||
|
||||
// The canonical session log is never rewritten.
|
||||
expect(output.logContent).toContain(FIXTURE_SECRET)
|
||||
expect(output.logContent).not.toContain(FIXTURE_PLACEHOLDER)
|
||||
}, LOADER_SMOKE_TEST_TIMEOUT_MS)
|
||||
|
||||
it('exports only prefixes ending in feedback under feedback-only mode', async () => {
|
||||
let output!: FixtureOutput
|
||||
const { stderr } = await runLoaderSmoke({
|
||||
label: 'session-telemetry-otel feedback-only loader smoke',
|
||||
tempDirPrefix: 'telemetry-otel-feedback-e2e-',
|
||||
binScript: driver,
|
||||
libBinScript: driver,
|
||||
configPath,
|
||||
tsconfigPath: repoTsconfig,
|
||||
env: { DSH_TELEMETRY_E2E_MODE: 'FEEDBACK_ONLY' },
|
||||
inspect: async (cwd) => { output = await readFixtureOutput(cwd) },
|
||||
})
|
||||
expect(stderr).not.toContain('UNHANDLED')
|
||||
|
||||
const wire = JSON.stringify(output.captures)
|
||||
expect(eventTypes(output.captures)).toContain('feedback/record')
|
||||
expect(wire).toContain('fixture feedback')
|
||||
expect(wire).toContain('prove telemetry with key')
|
||||
expect(wire).not.toContain('post-feedback private suffix')
|
||||
expect(output.logContent).toContain('post-feedback private suffix')
|
||||
}, LOADER_SMOKE_TEST_TIMEOUT_MS)
|
||||
|
||||
it('keeps disabled feedback local and prints the stable warning', async () => {
|
||||
let output!: FixtureOutput
|
||||
const { stdout } = await runLoaderSmoke({
|
||||
label: 'session-telemetry-otel disabled loader smoke',
|
||||
tempDirPrefix: 'telemetry-otel-disabled-e2e-',
|
||||
binScript: driver,
|
||||
libBinScript: driver,
|
||||
configPath,
|
||||
tsconfigPath: repoTsconfig,
|
||||
env: { DSH_TELEMETRY_E2E_MODE: 'DISABLED' },
|
||||
inspect: async (cwd) => { output = await readFixtureOutput(cwd) },
|
||||
})
|
||||
|
||||
expect(output.captures).toEqual([])
|
||||
expect(output.logContent).toContain('fixture feedback')
|
||||
expect(stdout.match(/session telemetry is DISABLED; nothing will be shared and this feedback remains local/)?.[0])
|
||||
.toMatchInlineSnapshot('"session telemetry is DISABLED; nothing will be shared and this feedback remains local"')
|
||||
}, LOADER_SMOKE_TEST_TIMEOUT_MS)
|
||||
})
|
||||
470
packages/session/session-telemetry-otel/tests/otel.spec.ts
Normal file
470
packages/session/session-telemetry-otel/tests/otel.spec.ts
Normal file
@@ -0,0 +1,470 @@
|
||||
/**
|
||||
* OTel backend unit tier: wire assertions against a scripted `node:http`
|
||||
* mock collector through the SDK's REAL pipeline (BatchLogRecordProcessor →
|
||||
* OTLP/HTTP JSON), config fail-loud cases, and the real-Loader-path guard
|
||||
* for the default-exported Service class.
|
||||
*/
|
||||
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, expectTypeOf, it, vi } from 'vitest'
|
||||
import { createServer, type Server } from 'node:http'
|
||||
import { once } from 'node:events'
|
||||
import { mkdtempSync, rmSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { gunzipSync } from 'node:zlib'
|
||||
import { Context } from 'cordis'
|
||||
import { getOrCreateAnonymousUserId } from '../src/user-id.ts'
|
||||
import Loader from '@cordisjs/plugin-loader'
|
||||
import { recordFeedback } from '@deepseek-ai/dsh-command-feedback'
|
||||
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import TelemetryOtel, { Config, DEFAULT_TELEMETRY_MODE, TelemetryMode } from '../src/index.ts'
|
||||
|
||||
interface Capture {
|
||||
headers: import('node:http').IncomingHttpHeaders
|
||||
body: OtlpLogsRequest
|
||||
}
|
||||
|
||||
/** Just the slice of ExportLogsServiceRequest JSON these assertions touch. */
|
||||
interface OtlpLogsRequest {
|
||||
resourceLogs: {
|
||||
resource: { attributes: { key: string; value: { stringValue?: string } }[] }
|
||||
scopeLogs: {
|
||||
scope: { name: string }
|
||||
logRecords: {
|
||||
timeUnixNano: string
|
||||
severityNumber: number
|
||||
severityText: string
|
||||
attributes?: { key: string; value: Record<string, unknown> }[]
|
||||
body?: unknown
|
||||
}[]
|
||||
}[]
|
||||
}[]
|
||||
}
|
||||
|
||||
const servers: Server[] = []
|
||||
|
||||
// The backend resolves the harness home's anonymous user id at construction;
|
||||
// pin DSH_HOME to a temp dir so the suite never touches the ambient ~/.dsh.
|
||||
let tempHome: string
|
||||
let previousDshHome: string | undefined
|
||||
beforeAll(() => {
|
||||
tempHome = mkdtempSync(join(tmpdir(), 'dsh-otel-home-'))
|
||||
previousDshHome = process.env.DSH_HOME
|
||||
process.env.DSH_HOME = tempHome
|
||||
})
|
||||
afterAll(() => {
|
||||
if (previousDshHome === undefined) delete process.env.DSH_HOME
|
||||
else process.env.DSH_HOME = previousDshHome
|
||||
rmSync(tempHome, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
for (const server of servers.splice(0)) {
|
||||
server.close()
|
||||
server.closeAllConnections()
|
||||
}
|
||||
})
|
||||
|
||||
async function mockCollector(
|
||||
beforeRespond?: (requestIndex: number) => Promise<void> | void,
|
||||
): Promise<{ url: string; captures: Capture[] }> {
|
||||
const captures: Capture[] = []
|
||||
let requestIndex = 0
|
||||
const server = createServer((request, response) => {
|
||||
const chunks: Buffer[] = []
|
||||
request.on('data', chunk => chunks.push(chunk as Buffer))
|
||||
request.on('end', () => {
|
||||
const index = requestIndex++
|
||||
void (async () => {
|
||||
await beforeRespond?.(index)
|
||||
const raw = Buffer.concat(chunks)
|
||||
const body = request.headers['content-encoding'] === 'gzip' ? gunzipSync(raw) : raw
|
||||
captures.push({
|
||||
headers: request.headers,
|
||||
body: JSON.parse(body.toString()) as OtlpLogsRequest,
|
||||
})
|
||||
response.writeHead(200, { 'content-type': 'application/json' }).end('{}')
|
||||
})()
|
||||
})
|
||||
})
|
||||
servers.push(server)
|
||||
server.listen(0, '127.0.0.1')
|
||||
await once(server, 'listening')
|
||||
const address = server.address()
|
||||
if (address === null || typeof address === 'string') throw new Error('no port')
|
||||
return { url: `http://127.0.0.1:${address.port}/v1/logs`, captures }
|
||||
}
|
||||
|
||||
async function boot(url: string) {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(TelemetryOtel, {
|
||||
exporter: { url, headers: { authorization: 'Bearer test-token' } },
|
||||
})
|
||||
return { ctx, fiber }
|
||||
}
|
||||
|
||||
function allRecords(captures: Capture[]) {
|
||||
return captures.flatMap(c => c.body.resourceLogs.flatMap(r => r.scopeLogs.flatMap(s =>
|
||||
s.logRecords.map(record => ({ scope: s.scope.name, record })))))
|
||||
}
|
||||
|
||||
function eventTypes(captures: Capture[]): string[] {
|
||||
return allRecords(captures).flatMap(({ record }) =>
|
||||
record.attributes?.flatMap(attribute =>
|
||||
attribute.key === 'event.type' && typeof attribute.value['stringValue'] === 'string'
|
||||
? [attribute.value['stringValue']]
|
||||
: []) ?? [])
|
||||
}
|
||||
|
||||
describe('TelemetryOtel wire', () => {
|
||||
it('ships session records and the ops shutdown marker through the real SDK pipeline', async () => {
|
||||
const { url, captures } = await mockCollector()
|
||||
const { ctx, fiber } = await boot(url)
|
||||
const session = ctx.sessions.create(SessionId('wire'), { meta: { cwd: '/tmp/w' } })
|
||||
session.append('turn/start', { turn: 1 })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'error', error: { message: 'boom', code: 'UNKNOWN' } } })
|
||||
ctx.telemetry.emit({
|
||||
channel: 'ledger',
|
||||
time: Date.now(),
|
||||
severity: 'info',
|
||||
attributes: { 'session.id': 'wire', 'event.type': 'manual', 'event.seq': 99 },
|
||||
body: { direct: true },
|
||||
})
|
||||
await fiber.dispose()
|
||||
|
||||
expect(captures.length).toBeGreaterThan(0)
|
||||
const first = captures[0]!
|
||||
const authorization: string | undefined = first.headers.authorization
|
||||
expect(authorization).toBe('Bearer test-token')
|
||||
|
||||
const resource = first.body.resourceLogs[0]!.resource.attributes
|
||||
expect(resource).toContainEqual({ key: 'service.name', value: { stringValue: 'deepseek-harness' } })
|
||||
expect(resource).toContainEqual({ key: 'user.id', value: { stringValue: getOrCreateAnonymousUserId() } })
|
||||
|
||||
const records = allRecords(captures)
|
||||
const ledger = records.filter(r => r.scope === '@deepseek-ai/dsh-session-telemetry-otel')
|
||||
const ops = records.filter(r => r.scope === '@deepseek-ai/dsh-session-telemetry-otel/ops')
|
||||
|
||||
const start = ledger.find(r => r.record.attributes?.some(a => a.key === 'event.type' && a.value.stringValue === 'turn/start'))
|
||||
expect(start).toBeDefined()
|
||||
expect(start?.record.severityNumber).toBe(9)
|
||||
expect(BigInt(start!.record.timeUnixNano)).toBe(BigInt(session.events[0]!.time) * 1_000_000n)
|
||||
expect(start?.record.attributes).toContainEqual({ key: 'session.cwd', value: { stringValue: '/tmp/w' } })
|
||||
|
||||
const end = ledger.find(r => r.record.attributes?.some(a => a.key === 'event.type' && a.value.stringValue === 'turn/end'))
|
||||
expect(end?.record.severityNumber).toBe(17)
|
||||
expect(end?.record.severityText).toBe('ERROR')
|
||||
expect(eventTypes(captures)).toContain('manual')
|
||||
|
||||
expect(ops).toHaveLength(1)
|
||||
expect(ops[0]!.record.attributes).toContainEqual({ key: 'telemetry.op', value: { stringValue: 'shutdown' } })
|
||||
})
|
||||
|
||||
it('drains records enqueued after a timer export began: dispose during an in-flight batch', async () => {
|
||||
// The backend implements NO flush() — the batch processor exports on its
|
||||
// own cadence, and shutdown's internal drain is complete exactly because
|
||||
// nothing in the process calls forceFlush() concurrently (the SDK's
|
||||
// concurrent-flush guard skips draining otherwise). Pin that: hold the
|
||||
// collector's response to the timer-triggered export open across
|
||||
// disposal, and the dispose-time shutdown marker (enqueued after that
|
||||
// batch's snapshot) must still arrive.
|
||||
const gate = Promise.withResolvers<boolean>()
|
||||
const arrived = Promise.withResolvers<boolean>()
|
||||
const { url, captures } = await mockCollector(async (index) => {
|
||||
if (index === 0) {
|
||||
arrived.resolve(true)
|
||||
await gate.promise
|
||||
}
|
||||
})
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(TelemetryOtel, {
|
||||
exporter: { url },
|
||||
processor: { scheduledDelayMillis: 10 },
|
||||
})
|
||||
const session = ctx.sessions.create(SessionId('drain'), { meta: {} })
|
||||
session.append('turn/start', { turn: 1 })
|
||||
await arrived.promise
|
||||
|
||||
const disposal = fiber.dispose()
|
||||
// Let disposal reach the backend's shutdown while the export is held open.
|
||||
await new Promise(resolve => setTimeout(resolve, 50))
|
||||
gate.resolve(true)
|
||||
await disposal
|
||||
|
||||
const ops = allRecords(captures).filter(r => r.scope === '@deepseek-ai/dsh-session-telemetry-otel/ops')
|
||||
expect(ops).toHaveLength(1)
|
||||
expect(ops[0]!.record.attributes).toContainEqual({ key: 'telemetry.op', value: { stringValue: 'shutdown' } })
|
||||
})
|
||||
|
||||
it('bounds the SDK forceFlush wait when an in-flight transport never settles', async () => {
|
||||
const gate = Promise.withResolvers<boolean>()
|
||||
const arrived = Promise.withResolvers<boolean>()
|
||||
const { url, captures } = await mockCollector(async (index) => {
|
||||
if (index === 0) {
|
||||
arrived.resolve(true)
|
||||
await gate.promise
|
||||
}
|
||||
})
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(TelemetryOtel, {
|
||||
exporter: { url, timeoutMillis: 60_000 },
|
||||
processor: { scheduledDelayMillis: 10, exportTimeoutMillis: 60_000 },
|
||||
shutdownTimeoutMillis: 50,
|
||||
})
|
||||
const session = ctx.sessions.create(SessionId('bounded-shutdown'), { meta: {} })
|
||||
session.append('turn/start', { turn: 1 })
|
||||
await arrived.promise
|
||||
|
||||
const started = performance.now()
|
||||
await fiber.dispose()
|
||||
expect(performance.now() - started).toBeLessThan(1_000)
|
||||
expect(captures).toHaveLength(0)
|
||||
|
||||
// The outer deadline cannot cancel the SDK transport. Let it finish so
|
||||
// the real provider promise remains clean after the test has proved the
|
||||
// Cordis disposer no longer waits for it.
|
||||
gate.resolve(true)
|
||||
await expect.poll(() => captures.length).toBeGreaterThanOrEqual(2)
|
||||
})
|
||||
|
||||
it('passes exporter options beyond url and headers through to the SDK exporter', async () => {
|
||||
const { url, captures } = await mockCollector()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
// `compression` is a documented SDK exporter option; the advertised
|
||||
// verbatim passthrough must hand it (and every other field) to the
|
||||
// exporter rather than silently rebuilding url/headers only.
|
||||
const fiber = await ctx.plugin(TelemetryOtel, {
|
||||
exporter: { url, compression: 'gzip' },
|
||||
} as Config)
|
||||
const session = ctx.sessions.create(SessionId('gzip'), { meta: {} })
|
||||
session.append('turn/start', { turn: 1 })
|
||||
await fiber.dispose()
|
||||
|
||||
expect(captures.length).toBeGreaterThan(0)
|
||||
expect(captures[0]!.headers['content-encoding']).toBe('gzip')
|
||||
const types = allRecords(captures).flatMap(({ record }) =>
|
||||
record.attributes?.flatMap(a => a.key === 'event.type' ? [a.value.stringValue] : []) ?? [])
|
||||
expect(types).toContain('turn/start')
|
||||
})
|
||||
|
||||
it('maps warn severity from record policy and leaves the seam flush hint unimplemented', async () => {
|
||||
const { url, captures } = await mockCollector()
|
||||
const { ctx, fiber } = await boot(url)
|
||||
ctx.on('telemetry/record', (_record, next) => ({ ...next(), severity: 'warn' }))
|
||||
const session = ctx.sessions.create(SessionId('warn'), { meta: {} })
|
||||
session.append('turn/start', { turn: 1 })
|
||||
// No flush(): the coordinator's optional-call forwarding no-ops, and the
|
||||
// batch processor owns export cadence end to end (see the backend note).
|
||||
expect('flush' in ctx.telemetry && ctx.telemetry.flush !== undefined).toBe(false)
|
||||
await fiber.dispose()
|
||||
const start = allRecords(captures).find(r =>
|
||||
r.record.attributes?.some(a => a.key === 'event.type' && a.value.stringValue === 'turn/start'))
|
||||
expect(start?.record.severityNumber).toBe(13)
|
||||
})
|
||||
|
||||
it('replays each session suffix only at the next feedback event', async () => {
|
||||
const { url, captures } = await mockCollector()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(TelemetryOtel, {
|
||||
mode: TelemetryMode.FEEDBACK_ONLY,
|
||||
exporter: { url },
|
||||
})
|
||||
ctx.on('telemetry/record', (_record, next) => {
|
||||
ctx.telemetry.emit({
|
||||
channel: 'ledger',
|
||||
time: Date.now(),
|
||||
severity: 'info',
|
||||
attributes: { 'session.id': 'feedback-only', 'event.type': 'direct-bypass', 'event.seq': 99 },
|
||||
body: { mustStayLocal: true },
|
||||
})
|
||||
return next()
|
||||
})
|
||||
const session = ctx.sessions.create(SessionId('feedback-only'), { meta: {} })
|
||||
session.append('turn/start', { turn: 1 })
|
||||
recordFeedback(session, 'first report')
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
recordFeedback(session, 'second report')
|
||||
session.append('turn/start', { turn: 2 })
|
||||
await fiber.dispose()
|
||||
|
||||
const types = allRecords(captures).flatMap(({ record }) =>
|
||||
record.attributes?.flatMap(attribute =>
|
||||
attribute.key === 'event.type' ? [attribute.value.stringValue] : []) ?? [])
|
||||
expect(types).toEqual(['turn/start', 'feedback/record', 'turn/end', 'feedback/record'])
|
||||
expect(JSON.stringify(captures)).toContain('first report')
|
||||
expect(JSON.stringify(captures)).toContain('second report')
|
||||
expect(allRecords(captures).some(({ scope }) => scope.endsWith('/ops'))).toBe(false)
|
||||
})
|
||||
|
||||
it('ignores direct emits and non-canonical feedback in feedback-only mode', async () => {
|
||||
const { url, captures } = await mockCollector()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const warn = vi.spyOn(ctx.logger, 'warn').mockImplementation(() => {})
|
||||
const fiber = await ctx.plugin(TelemetryOtel, {
|
||||
mode: TelemetryMode.FEEDBACK_ONLY,
|
||||
exporter: { url },
|
||||
})
|
||||
const session = ctx.sessions.create(SessionId('no-feedback'), { meta: {} })
|
||||
session.append('turn/start', { turn: 1 })
|
||||
ctx.telemetry.emit({
|
||||
channel: 'ledger',
|
||||
time: Date.now(),
|
||||
severity: 'info',
|
||||
attributes: { 'session.id': 'no-feedback', 'event.type': 'direct', 'event.seq': 99 },
|
||||
body: { mustStayLocal: true },
|
||||
})
|
||||
ctx.emit('session/event', session, {
|
||||
type: 'feedback/record',
|
||||
seq: session.events.length,
|
||||
time: Date.now(),
|
||||
data: { text: 'not committed' },
|
||||
})
|
||||
await fiber.dispose()
|
||||
|
||||
expect(warn).toHaveBeenCalledWith(
|
||||
'session telemetry ignored a feedback event absent from the canonical session log',
|
||||
)
|
||||
expect(captures).toEqual([])
|
||||
})
|
||||
|
||||
it('constructs no disabled transport even when exporter options are present', async () => {
|
||||
const { url, captures } = await mockCollector()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const warn = vi.spyOn(ctx.logger, 'warn').mockImplementation(() => {})
|
||||
const fiber = await ctx.plugin(TelemetryOtel, {
|
||||
mode: TelemetryMode.DISABLED,
|
||||
exporter: { url },
|
||||
processor: { maxExportBatchSize: 0 },
|
||||
})
|
||||
const session = ctx.sessions.create(SessionId('disabled'), { meta: {} })
|
||||
session.append('turn/start', { turn: 1 })
|
||||
recordFeedback(session, 'local report')
|
||||
|
||||
expect(warn).toHaveBeenCalledWith(
|
||||
'session telemetry is DISABLED; nothing will be shared and this feedback remains local',
|
||||
)
|
||||
ctx.telemetry.emit({
|
||||
channel: 'ledger',
|
||||
time: 0,
|
||||
severity: 'info',
|
||||
attributes: {},
|
||||
body: null,
|
||||
})
|
||||
await ctx.telemetry.shutdown()
|
||||
await fiber.dispose()
|
||||
recordFeedback(session, 'after disposal')
|
||||
expect(warn).toHaveBeenCalledTimes(1)
|
||||
expect(captures).toEqual([])
|
||||
})
|
||||
|
||||
it('defaults direct construction to full delivery', async () => {
|
||||
const { url, captures } = await mockCollector()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
new TelemetryOtel(ctx, { exporter: { url } })
|
||||
const session = ctx.sessions.create(SessionId('direct-default'), { meta: {} })
|
||||
session.append('turn/start', { turn: 1 })
|
||||
await ctx.fiber.dispose()
|
||||
|
||||
expect(eventTypes(captures)).toContain('turn/start')
|
||||
})
|
||||
})
|
||||
|
||||
describe('TelemetryOtel config fails loud', () => {
|
||||
it('exposes modes through the nominal enum', () => {
|
||||
expectTypeOf<Config['mode']>().toEqualTypeOf<TelemetryMode | undefined>()
|
||||
expectTypeOf<'FULL'>().not.toExtend<TelemetryMode>()
|
||||
expectTypeOf<TelemetryMode.FULL>().toExtend<TelemetryMode>()
|
||||
expect(DEFAULT_TELEMETRY_MODE).toBe(TelemetryMode.FULL)
|
||||
expect(Config({}).mode).toBe(DEFAULT_TELEMETRY_MODE)
|
||||
})
|
||||
|
||||
it.each([
|
||||
[{}, /exporter\.url is required/],
|
||||
[{ exporter: { url: '' } }, /exporter\.url is required/],
|
||||
[{ exporter: { url: 'not a url' } }, /not a valid URL/],
|
||||
[{ exporter: { url: 'ftp://collector' } }, /must be http\(s\)/],
|
||||
[{ mode: TelemetryMode.FEEDBACK_ONLY }, /exporter\.url is required/],
|
||||
[{ mode: 'INVALID' }, /INVALID/],
|
||||
// The SDK accepts a non-positive batch size but its shutdown drain then
|
||||
// splices empty batches forever — dispose would hang, so reject at load.
|
||||
[{ exporter: { url: 'http://c/v1/logs' }, processor: { maxExportBatchSize: 0 } }, /maxExportBatchSize/],
|
||||
[{ exporter: { url: 'http://c/v1/logs' }, processor: { maxExportBatchSize: 0.5 } }, /maxExportBatchSize/],
|
||||
[{ exporter: { url: 'http://c/v1/logs' }, shutdownTimeoutMillis: 0 }, /shutdownTimeoutMillis/],
|
||||
[{ exporter: { url: 'http://c/v1/logs' }, shutdownTimeoutMillis: Number.POSITIVE_INFINITY }, /shutdownTimeoutMillis/],
|
||||
])('rejects %j at plugin load', async (config, message) => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
await expect(ctx.plugin(TelemetryOtel, config as Config)).rejects.toThrow(message)
|
||||
})
|
||||
|
||||
it('rejects an unknown direct mode before reading transport config', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
let exporterRead = false
|
||||
const config = {
|
||||
mode: 'INVALID',
|
||||
get exporter() {
|
||||
exporterRead = true
|
||||
throw new Error('transport config was read')
|
||||
},
|
||||
} as unknown as Config
|
||||
|
||||
expect(() => new TelemetryOtel(ctx, config)).toThrow(/unsupported mode "INVALID"/)
|
||||
expect(exporterRead).toBe(false)
|
||||
})
|
||||
|
||||
it('does not read any transport setting in disabled mode', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const transportRead = vi.fn(() => {
|
||||
throw new Error('transport config was read')
|
||||
})
|
||||
const config = {
|
||||
mode: TelemetryMode.DISABLED,
|
||||
get exporter() {
|
||||
return transportRead()
|
||||
},
|
||||
get processor() {
|
||||
return transportRead()
|
||||
},
|
||||
get shutdownTimeoutMillis() {
|
||||
return transportRead()
|
||||
},
|
||||
} as unknown as Config
|
||||
|
||||
new TelemetryOtel(ctx, config)
|
||||
expect(transportRead).not.toHaveBeenCalled()
|
||||
await ctx.fiber.dispose()
|
||||
})
|
||||
})
|
||||
|
||||
describe('dsh-session-telemetry-otel real-load-path guard', () => {
|
||||
it('keeps the Service class with inject/Config through unwrapExports', async () => {
|
||||
const module = await import('../src/index.ts')
|
||||
const loader = Object.create(Loader.prototype) as Loader
|
||||
const unwrapped = loader.unwrapExports(module) as typeof TelemetryOtel
|
||||
expect(unwrapped).toBe(TelemetryOtel)
|
||||
expect(unwrapped.inject).toEqual(['sessions'])
|
||||
expect(typeof unwrapped.Config).toBe('function')
|
||||
})
|
||||
|
||||
it('boots through the unwrapped class and registers ctx.telemetry', async () => {
|
||||
const { url } = await mockCollector()
|
||||
const module = await import('../src/index.ts')
|
||||
const loader = Object.create(Loader.prototype) as Loader
|
||||
const unwrapped = loader.unwrapExports(module) as Parameters<Context['plugin']>[0]
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(unwrapped, { exporter: { url } })
|
||||
expect(ctx.telemetry).toBeInstanceOf(TelemetryOtel)
|
||||
await fiber.dispose()
|
||||
})
|
||||
})
|
||||
105
packages/session/session-telemetry-otel/tests/user-id.spec.ts
Normal file
105
packages/session/session-telemetry-otel/tests/user-id.spec.ts
Normal file
@@ -0,0 +1,105 @@
|
||||
import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import {
|
||||
USER_ID_FILE_NAME,
|
||||
getOrCreateAnonymousUserId,
|
||||
} from '../src/user-id.ts'
|
||||
|
||||
const dirs: string[] = []
|
||||
|
||||
function tempHome(): string {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'dsh-userid-'))
|
||||
dirs.push(dir)
|
||||
return dir
|
||||
}
|
||||
|
||||
afterEach(() => {
|
||||
for (const dir of dirs.splice(0)) {
|
||||
rmSync(dir, { recursive: true, force: true })
|
||||
}
|
||||
})
|
||||
|
||||
const UUID = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i
|
||||
|
||||
describe('getOrCreateAnonymousUserId', () => {
|
||||
it('creates, persists, and returns a bare UUID line on first use', () => {
|
||||
const home = tempHome()
|
||||
const id = getOrCreateAnonymousUserId({ env: { DSH_HOME: home } })
|
||||
expect(id).toMatch(UUID)
|
||||
expect(readFileSync(join(home, USER_ID_FILE_NAME), 'utf8')).toBe(`${id}\n`)
|
||||
})
|
||||
|
||||
it('creates the home directory when missing', () => {
|
||||
const home = join(tempHome(), 'nested', 'home')
|
||||
const id = getOrCreateAnonymousUserId({ env: { DSH_HOME: home } })
|
||||
expect(readFileSync(join(home, USER_ID_FILE_NAME), 'utf8')).toBe(`${id}\n`)
|
||||
})
|
||||
|
||||
it('returns the persisted id on subsequent calls, tolerating surrounding whitespace', () => {
|
||||
const home = tempHome()
|
||||
const existing = '01234567-89ab-4cde-8f01-23456789abcd'
|
||||
writeFileSync(join(home, USER_ID_FILE_NAME), ` ${existing}\n\n`, 'utf8')
|
||||
expect(getOrCreateAnonymousUserId({ env: { DSH_HOME: home } })).toBe(existing)
|
||||
})
|
||||
|
||||
it('overwrites a corrupt file with a fresh id', () => {
|
||||
const home = tempHome()
|
||||
writeFileSync(join(home, USER_ID_FILE_NAME), 'not-a-uuid\n', 'utf8')
|
||||
const id = getOrCreateAnonymousUserId({ env: { DSH_HOME: home } })
|
||||
expect(id).toMatch(UUID)
|
||||
expect(readFileSync(join(home, USER_ID_FILE_NAME), 'utf8')).toBe(`${id}\n`)
|
||||
})
|
||||
|
||||
it('adopts a concurrent winner: exclusive create loses to an id written after the initial read', () => {
|
||||
const home = tempHome()
|
||||
const winner = 'aaaaaaaa-bbbb-4ccc-8ddd-eeeeeeeeeeee'
|
||||
const file = join(home, USER_ID_FILE_NAME)
|
||||
// The generator seam runs between the initial read (absent) and the wx
|
||||
// write, so planting the winner here simulates the concurrent first launch.
|
||||
const id = getOrCreateAnonymousUserId({
|
||||
env: { DSH_HOME: home },
|
||||
randomUUID: () => {
|
||||
writeFileSync(file, `${winner}\n`, 'utf8')
|
||||
return 'ffffffff-0000-4000-8000-000000000000'
|
||||
},
|
||||
})
|
||||
expect(id).toBe(winner)
|
||||
})
|
||||
|
||||
it('returns a usable id when the home cannot contain files, without persisting', () => {
|
||||
const home = tempHome()
|
||||
const blocked = join(home, 'blocked')
|
||||
writeFileSync(blocked, 'occupied\n')
|
||||
const id = getOrCreateAnonymousUserId({ env: { DSH_HOME: blocked } })
|
||||
expect(id).toMatch(UUID)
|
||||
expect(existsSync(join(blocked, USER_ID_FILE_NAME))).toBe(false)
|
||||
})
|
||||
|
||||
it('memoizes per resolved home for the process lifetime: one read, deletion-proof', () => {
|
||||
const home = tempHome()
|
||||
const first = getOrCreateAnonymousUserId({ env: { DSH_HOME: home } })
|
||||
rmSync(join(home, USER_ID_FILE_NAME))
|
||||
expect(getOrCreateAnonymousUserId({ env: { DSH_HOME: home } })).toBe(first)
|
||||
})
|
||||
|
||||
it('keeps distinct homes on distinct ids', () => {
|
||||
const a = getOrCreateAnonymousUserId({ env: { DSH_HOME: tempHome() } })
|
||||
const b = getOrCreateAnonymousUserId({ env: { DSH_HOME: tempHome() } })
|
||||
expect(a).not.toBe(b)
|
||||
})
|
||||
|
||||
it('reads process.env by default', () => {
|
||||
const home = tempHome()
|
||||
const previous = process.env.DSH_HOME
|
||||
process.env.DSH_HOME = home
|
||||
try {
|
||||
const id = getOrCreateAnonymousUserId()
|
||||
expect(readFileSync(join(home, USER_ID_FILE_NAME), 'utf8')).toBe(`${id}\n`)
|
||||
} finally {
|
||||
if (previous === undefined) delete process.env.DSH_HOME
|
||||
else process.env.DSH_HOME = previous
|
||||
}
|
||||
})
|
||||
})
|
||||
42
packages/session/session-telemetry-otel/tsconfig.json
Normal file
42
packages/session/session-telemetry-otel/tsconfig.json
Normal file
@@ -0,0 +1,42 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
{
|
||||
"path": "../../feedback/command-feedback"
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm"
|
||||
},
|
||||
{
|
||||
"path": "../session-telemetry"
|
||||
},
|
||||
{
|
||||
"path": "../../util/brand"
|
||||
},
|
||||
{
|
||||
"path": "../../util/paths"
|
||||
},
|
||||
{
|
||||
"path": "../../support/invariants"
|
||||
}
|
||||
]
|
||||
}
|
||||
6
packages/session/session-telemetry/README.i18n.yaml
Normal file
6
packages/session/session-telemetry/README.i18n.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/telemetry/session-telemetry/README.md
|
||||
README.md: 67d95bcc62bbf6783f8dcd11f0236d8c926b557b
|
||||
README.zh.md: 1ee0e0eb14bb06c8ac669cd417f2ee2ce46ca430
|
||||
43
packages/session/session-telemetry/README.md
Normal file
43
packages/session/session-telemetry/README.md
Normal file
@@ -0,0 +1,43 @@
|
||||
# @deepseek-ai/dsh-session-telemetry
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
The telemetry seam: the capture side of session-event reporting, behind a backend contract any reporting SDK satisfies with zero bending. Capture can follow live session events or replay a canonical session-log prefix on demand. The boundary axiom that shapes everything here: **this package's aspect ends at `emit()`** — batching, retry, queueing, and loss policy belong to the backend's SDK and are neither specified nor wrapped. Rationale and rejected alternatives: [the revival Agent Note](../../../.agents/notes/implemented/feature/2026-07-23-session-telemetry-otel-revival.md), [feedback-gated delivery](../../../.agents/notes/implemented/feature/2026-08-05-feedback-gated-session-telemetry.md), and [buffer-free feedback replay](../../../.agents/notes/implemented/simplification/2026-08-06-buffer-free-feedback-telemetry.md).
|
||||
|
||||
## The backend contract
|
||||
|
||||
`TelemetryBackend` is three members: `emit(record)` (MUST be a non-blocking enqueue — it runs synchronously on the `session/event` hot path or during an explicit canonical-log replay), optional `flush()` (a turn-boundary hint, fire-and-forget; most backends leave it unimplemented and let their SDK's batching cadence govern export timing — an implementer owns the interaction between concurrent flushes and `shutdown()`'s drain), and `shutdown()` (the lifecycle forward: drain-and-quiesce, awaited at dispose). `Telemetry` is its service-registered form under the `telemetry` context key — one implementation per context, duplicate load throws. A backend composes `TelemetryCoordinator` with `live` capture or `on-demand` capture and calls `captureSession(session, throughSeq?)` at its owning trigger.
|
||||
|
||||
## Capture points
|
||||
|
||||
In `live` mode the coordinator registers, all through the composing fiber's effects: `session/created` (adopt: record the header, read the log back through the projection from the construction boundary — constructor seeds from fork/resume never re-emit on the firehose and never re-export), `session/event` (project, deep-copy, redact, then hand off; zero I/O), `session/flush` (forward the optional `flush()` hint and return void — the loop's awaited parallel must never wait on telemetry), `session/disposed` (capture the session's `shutdown` operational record at its termination edge, then retire it), `agent/error` (the one live-bus relay; the session event vocabulary intentionally has no operational-error record), a dispose effect (capture shutdown for each still-live session, then await the backend's `shutdown()`; failures warn instead of throwing), and an adoption sweep of `ctx.sessions.list()` (a hot reload does not replay `session/created`). In `on-demand` mode it registers only the dispose effect: `captureSession()` reads the canonical log through an optional inclusive sequence boundary, while flush hints and operational events remain local.
|
||||
|
||||
## The redact waterfall
|
||||
|
||||
Every record passes the `telemetry/record` waterfall immediately after projection — the seam's scrubbing extension point. The seam ships NO rules of its own: the innermost `next()` passes the record through unchanged, so with no listener mounted records reach the backend exactly as captured, and exported data is precisely as clean as the rules a deployment mounts. Listeners stack by transforming `next()`'s return value; returning without `next()` replaces everything beneath, and a throwing listener withholds that one record fail-closed inside the coordinator's containment. Live capture runs the waterfall at append time; on-demand capture runs it while replaying the canonical log, using the rules mounted at that time. Redaction applies to the outbound copy only; the canonical session log is never rewritten.
|
||||
|
||||
## The handoff cursor
|
||||
|
||||
A module-scope `WeakMap<Session, seq>` marks the highest seq HANDED OFF (not delivered) per session. Live capture advances it at append time; on-demand capture advances it only while `captureSession()` hands a requested prefix to the backend. An uncaptured prefix remains solely in the canonical log, so a coordinator reload adds no telemetry-owned recovery state. On replay the coordinator re-hands only events past the cursor (events at or below it still rebuild the chunk-projection state); a missing cursor safely degrades to a re-hand from the session's construction boundary (`Session.firstLiveSeq` — seq 0 for a session born in this process), absorbed by receiver-side dedupe on `(session.id, event.seq)`. Constructor seeds never re-export: a resumed session's history shipped from the previous process under the same id, and a fork's inherited prefix lives in the parent's stream (receivers stitch on `session.parent_id` + `session.seed_length`). The accepted cost, consistent with at-most-once delivery: a resume does not backfill records a previous process failed to deliver — a deployment with a backfill requirement needs the deferred outbox, not replay. This is a deliberate, narrow exception to the registrations-are-effects discipline: entries die with their sessions, the value is a monotonic watermark, and losing it is never an error.
|
||||
|
||||
## The fixed chunk projection
|
||||
|
||||
Only the first `assistant/chunk` of each `(turn, step)` ships; the rest are dropped at capture and never advance the cursor. That one chunk is the stream-started signal: `step/start` + first-chunk presence + `assistant/message` presence + the `turn/end` reason distinguish "the request never started" from "the stream died midway" without chunk volume, and time-to-first-token stays computable. Chunk elision makes `seq` gaps routine on the wire — a gap is never a loss signal. Every other event type, including ones merged by plugins this package never heard of, passes through whole.
|
||||
|
||||
## The logical record
|
||||
|
||||
`TelemetryRecord`: `channel` (`ledger` | `ops`), `time` (epoch ms), `severity` (pre-mapped: ERROR for `tool/result.isError`, `turn/end` error reasons, and `agent-error`; INFO for other captured records, while `telemetry/record` policies may assign WARN), identity-only `attributes` (`session.id`, `event.type`, `event.seq`, plus `session.cwd`/`session.parent_id`/`session.seed_length` when the header has them), and the complete deep-copied `event.data` as `body` — post-redaction. Operational records carry `telemetry.op` (`agent-error` | `shutdown`) and `session.id`, and deliberately NO `event.seq`/`event.type` — signals to alert on, not entries to sum; `agent-error` normalizes its arbitrary thrown value into a stable `{ name, message }` body. Delivery downstream of the handoff is the backend SDK's; duplicates remain possible (cursor-less re-adoption, SDK retries), so receivers dedupe on `(session.id, event.seq)`.
|
||||
|
||||
## Model Experience
|
||||
|
||||
None, as the seam only observes the session stream and hands redacted copies to a reporting backend; it never contributes to a model request.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
None; this package neither assembles nor sends a provider request.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **Best-effort delivery** — the cursor marks handed-off, not delivered; a session torn down inside a reload window cannot be re-adopted; whatever sits in a backend queue at crash time is lost. A durable outbox (spool, per-sink cursors, at-least-once) is deferred until a deployment states a crash-loss requirement — see [the revival Agent Note](../../../.agents/notes/implemented/feature/2026-07-23-session-telemetry-otel-revival.md).
|
||||
- **No built-in redaction rules** — with no `telemetry/record` listener mounted, records leave the process exactly as captured, including any credentials embedded in file contents or command output; a deployment exporting to a shared collector owns its rule set.
|
||||
- **On-demand redaction uses current state** — uncaptured events exist only in the canonical session log. A later `captureSession()` deep-copies and redacts their current values with the policy mounted at that time; there is no capture-time telemetry snapshot or durable pre-capture spool.
|
||||
43
packages/session/session-telemetry/README.zh.md
Normal file
43
packages/session/session-telemetry/README.zh.md
Normal file
@@ -0,0 +1,43 @@
|
||||
# @deepseek-ai/dsh-session-telemetry
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
遥测（telemetry）seam：会话事件上报的捕获侧，隔在一个后端契约之后，任何上报 SDK 都无需变形即可满足该契约。捕获侧可跟随实时会话事件，也可按需回放权威会话日志前缀。塑造本包（package）一切设计的边界公理：**本包的职责止于 `emit()`**。批处理、重试、排队与丢失策略都属于后端自身的 SDK，本包既不为其立规，也不做包装。设计依据与被否决的替代方案见[复活 Agent Note（agent 决策记录）](../../../.agents/notes/implemented/feature/2026-07-23-session-telemetry-otel-revival.md)、[反馈门控投递](../../../.agents/notes/implemented/feature/2026-08-05-feedback-gated-session-telemetry.md)与[无缓冲反馈回放](../../../.agents/notes/implemented/simplification/2026-08-06-buffer-free-feedback-telemetry.md)。
|
||||
|
||||
## 后端契约
|
||||
|
||||
`TelemetryBackend` 只有三个成员：`emit(record)`（必须是非阻塞入队；它在 `session/event` 热路径或显式权威日志回放期间同步执行）、可选的 `flush()`（轮次边界提示，触发后不等待结果；多数后端不实现它，而由其 SDK 的批处理节奏决定导出时机；并发 flush 与 `shutdown()` 的排空之间的交互由实现方自行负责）、以及 `shutdown()`（生命周期转发点：排空并完全停稳，在 dispose（资源释放）时被等待）。`Telemetry` 是它注册在 `telemetry` 上下文键下的服务形态：每个上下文只允许一个实现，重复加载会抛出异常。后端以 `live` 或 `on-demand` 模式组合 `TelemetryCoordinator`，并在自身所属的触发器中调用 `captureSession(session, throughSeq?)`。
|
||||
|
||||
## 捕获点
|
||||
|
||||
在 `live` 模式中，协调器的全部注册都经由组合方 fiber 的 effect 完成：`session/created`（收养：记录 header，并经投影从构造边界起回读日志；来自 fork 或恢复的构造函数种子绝不会在 firehose 上再次发出，也绝不会再次导出）、`session/event`（投影、深拷贝、脱敏，再交接；零 I/O）、`session/flush`（转发可选的 `flush()` 提示并返回 void；循环所等待的并行任务绝不能等待遥测）、`session/disposed`（在会话自身的终止边缘捕获该会话的 `shutdown` 运维记录，然后将其退役）、`agent/error`（唯一的实时总线转发；会话事件词汇有意不包含运维错误记录）、一个 dispose effect（捕获每个仍存活会话的 shutdown，再等待后端的 `shutdown()`；失败只发出警告而不抛出），以及对 `ctx.sessions.list()` 的收养扫描（热重载不会重放 `session/created`）。在 `on-demand` 模式中，协调器只注册 dispose effect：`captureSession()` 读取权威日志，直至可选的序列号边界（含边界）；flush 提示与运维事件留在本地。
|
||||
|
||||
## 脱敏 waterfall（瀑布式事件）
|
||||
|
||||
每条记录在投影后立即经过 `telemetry/record` waterfall，这是该 seam 的脱敏扩展点。seam 自身不带任何规则：最内层的 `next()` 原样透传记录，因此未挂载监听器时，记录以捕获时的原样到达后端；导出数据能干净到什么程度，恰恰取决于部署方挂载了什么规则。监听器通过变换 `next()` 的返回值来堆叠；不调用 `next()` 就返回，即替换其下方的全部逻辑；抛出异常的监听器会在协调器的隔离范围内以 fail-closed 方式拦下这一条记录。实时捕获在追加时运行 waterfall；按需捕获则在回放权威日志时使用当时挂载的规则运行 waterfall。脱敏只作用于外发副本；权威会话日志永不改写。
|
||||
|
||||
## handoff 游标
|
||||
|
||||
一个模块作用域的 `WeakMap<Session, seq>` 记录每个会话已交接（而非已投递）的最高 seq。实时捕获在追加时推进游标；按需捕获只有在 `captureSession()` 将请求的前缀交给后端时才推进游标。未捕获的前缀只留在权威日志中，因此协调器重载不会增加遥测自有的恢复状态。回放时，协调器只重新交接游标之后的事件（游标及其之前的事件仍用于重建分片投影状态）；游标缺失时安全退化为从会话构造边界起的重新交接（`Session.firstLiveSeq`，对在本进程中诞生的会话即 seq 0），由接收端基于 `(session.id, event.seq)` 的去重吸收。构造函数种子绝不会再次导出：恢复会话的历史已由上一个进程以同一 id 发出，fork 继承的前缀则位于父会话的流中（接收端基于 `session.parent_id` + `session.seed_length` 拼接）。由此接受的代价与至多一次（at-most-once）投递一致：恢复不会回填上一个进程未能投递的记录；有回填要求的部署需要的是已推迟的 outbox，而不是回放。这是对「注册即 effect」纪律的一次有意且范围极窄的例外：条目随其会话消亡，值是单调水位线，丢失它绝不是错误。
|
||||
|
||||
## 固定分片投影
|
||||
|
||||
每个 `(turn, step)` 只发出第一条 `assistant/chunk`；其余分片在捕获时丢弃，且绝不推进游标。这一条分片就是「流已开始」的信号：`step/start`、首分片是否存在、`assistant/message` 是否存在，加上 `turn/end` 的原因，无需分片流量即可区分「请求从未开始」与「流中途夭折」，首个 token 延迟（time-to-first-token）也仍然可以计算。分片省略使导出流中的 `seq` 缺口成为常态：缺口绝不是丢失信号。其余所有事件类型都会完整透传，包括本包从未听说过的插件所合并的事件类型。
|
||||
|
||||
## 逻辑记录
|
||||
|
||||
`TelemetryRecord` 包含：`channel`（`ledger` | `ops`）、`time`（epoch 毫秒）、`severity`（预先映射好的严重级别：`tool/result.isError`、`turn/end` 的错误原因与 `agent-error` 映射为 ERROR，其他已捕获记录映射为 INFO，而 `telemetry/record` 策略可以指定 WARN）、只含身份信息的 `attributes`（`session.id`、`event.type`、`event.seq`，header 中存在时再加 `session.cwd`/`session.parent_id`/`session.seed_length`），以及作为 `body` 的完整深拷贝 `event.data`，且以脱敏后的内容为准。运维记录携带 `telemetry.op`（`agent-error` | `shutdown`）和 `session.id`，并刻意不带 `event.seq`/`event.type`：它们是用来告警的信号，不是用来累加的条目；`agent-error` 会把任意抛出值规范化为稳定的 `{ name, message }` 记录主体。交接之后的投递由后端 SDK 负责；重复仍然可能出现（无游标的重新收养、SDK 重试），因此接收端基于 `(session.id, event.seq)` 去重。
|
||||
|
||||
## 模型体验
|
||||
|
||||
无。该 seam 只观察会话流，并把脱敏后的副本交给上报后端；它绝不向模型请求贡献任何内容。
|
||||
|
||||
#### KV Cache 影响
|
||||
|
||||
无；本包既不组装也不发送提供方请求。
|
||||
|
||||
## 已知限制与暂缓事项
|
||||
|
||||
- **尽力而为的投递**：游标标记的是已交接而非已投递；在重载窗口内被拆除的会话无法重新收养；崩溃时留在后端队列中的内容会丢失。持久化 outbox（spool、每 sink 游标、at-least-once）推迟到有部署方提出明确的崩溃丢失要求时再实现；见[复活 Agent Note](../../../.agents/notes/implemented/feature/2026-07-23-session-telemetry-otel-revival.md)。
|
||||
- **不内置脱敏规则**：未挂载 `telemetry/record` 监听器时，记录以捕获时的原样离开进程，包括文件内容或命令输出中内嵌的任何凭据；向共享 collector 导出的部署方自行负责其规则集。
|
||||
- **按需脱敏使用当前状态**：未捕获的事件只存在于权威会话日志中。后续的 `captureSession()` 会使用当时挂载的策略，深拷贝并脱敏其当前值；不存在捕获时的遥测快照或持久化的捕获前 spool。
|
||||
39
packages/session/session-telemetry/package.json
Normal file
39
packages/session/session-telemetry/package.json
Normal file
@@ -0,0 +1,39 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-session-telemetry",
|
||||
"description": "Telemetry seam for the DeepSeek Harness: session-event capture, projection, redaction, and handoff to a reporting backend",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./invariant": {
|
||||
"types": "./lib/types/invariant.d.ts",
|
||||
"default": "./lib/invariant.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-agent": "^0.0.1",
|
||||
"@deepseek-ai/dsh-invariants": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-agent": "workspace:^",
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
319
packages/session/session-telemetry/src/coordinator.ts
Normal file
319
packages/session/session-telemetry/src/coordinator.ts
Normal file
@@ -0,0 +1,319 @@
|
||||
/**
|
||||
* Capture coordinator: the seam's upstream half. Live capture subscribes to
|
||||
* the session firehose plus the one live-bus relay (`agent/error`). Both
|
||||
* capture paths apply the fixed chunk projection, build logical records, and
|
||||
* run each through the
|
||||
* `telemetry/record` waterfall (deployment-mounted redaction rules;
|
||||
* pass-through when none), then hands the result to the backend. Live capture
|
||||
* follows the session firehose; on-demand capture replays the canonical log
|
||||
* only when requested. Every synchronous handler is self-contained so a
|
||||
* failing backend can never starve other subscribers (cordis `emit` is
|
||||
* stop-on-throw) or touch the agent loop. Composed by a backend in its
|
||||
* constructor.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-session-telemetry/coordinator
|
||||
*/
|
||||
|
||||
import type { Context } from 'cordis'
|
||||
import type { Session, SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import type { Agent } from '@deepseek-ai/dsh-agent'
|
||||
import type { TelemetryBackend, TelemetryRecord, TelemetrySeverity } from './index.ts'
|
||||
|
||||
/** Whether capture follows live events or reads the canonical log only when requested. */
|
||||
export type TelemetryCapture = 'live' | 'on-demand'
|
||||
|
||||
/** One projected record ready for backend handoff. */
|
||||
interface ProjectedRecord {
|
||||
readonly record: TelemetryRecord
|
||||
/** Ledger cursor advanced only after the backend accepts this record. */
|
||||
readonly seq?: number
|
||||
}
|
||||
|
||||
/**
|
||||
* The handoff cursor: per session, the highest `seq` handed to a backend.
|
||||
* Deliberately MODULE-scope ambient state — a narrow, documented exception
|
||||
* to the registrations-are-effects discipline: cordis has no HMR
|
||||
* state-handover API, and keying by the `Session` object (which belongs to
|
||||
* the session store and outlives any telemetry fiber) is the only in-process
|
||||
* lifetime that lets a re-adopting fiber resume instead of re-handing
|
||||
* history. Entries die with their sessions; a missing entry safely means
|
||||
* "re-hand everything". Advanced only at emit time — the cursor marks
|
||||
* handed-off, not delivered.
|
||||
*/
|
||||
const handoffCursor = new WeakMap<Session, number>()
|
||||
|
||||
/**
|
||||
* Install the telemetry capture side onto a context for one backend.
|
||||
*
|
||||
* Live capture registers the persistence-coordinator listener set plus the
|
||||
* `agent/error` relay, all through `ctx.effect()`/`ctx.on()` on the composing
|
||||
* fiber, and sweeps already-live sessions (a hot reload does not replay
|
||||
* `session/created`). A `session/disposed` captures the session's `shutdown`
|
||||
* operational record at its own termination edge and retires it from the
|
||||
* adopted set. On-demand capture registers none of those continuous listeners;
|
||||
* {@link captureSession} reads the canonical log explicitly and never creates
|
||||
* operational records. Disposal captures shutdown markers for live-adopted
|
||||
* sessions, then awaits the backend's `shutdown()`; a failure there warns
|
||||
* instead of throwing — best-effort reporting must not fail application
|
||||
* teardown.
|
||||
*/
|
||||
export class TelemetryCoordinator {
|
||||
/**
|
||||
* Sessions adopted by THIS fiber and still live, for double-adoption
|
||||
* protection and the teardown sweep of unmarked sessions;
|
||||
* `session/disposed` marks and retires entries.
|
||||
*/
|
||||
private readonly adopted = new Set<Session>()
|
||||
/** Per session, the `turn:step` keys whose first chunk already shipped; rebuilt from the log on re-adoption. */
|
||||
private readonly chunkSeen = new WeakMap<Session, Set<string>>()
|
||||
/**
|
||||
* @param ctx - the composing backend's context; listeners bind to its fiber.
|
||||
* @param backend - the backend receiving records; owned elsewhere, never disposed here beyond `shutdown()` forwarding.
|
||||
* @param capture - follow live events, or wait for explicit canonical-log capture.
|
||||
*/
|
||||
constructor(
|
||||
private readonly ctx: Context,
|
||||
private readonly backend: TelemetryBackend,
|
||||
capture: TelemetryCapture = 'live',
|
||||
) {
|
||||
if (capture === 'live') {
|
||||
ctx.on('session/created', (session) => {
|
||||
this.adopt(session)
|
||||
})
|
||||
// Capture the shutdown marker at the session's own termination edge,
|
||||
// then retire the only strong reference owned by this coordinator.
|
||||
ctx.on('session/disposed', (session) => {
|
||||
this.contain(() => {
|
||||
if (!this.adopted.delete(session)) return
|
||||
this.deliver(session, { record: this.redact(shutdownRecord(session)) })
|
||||
})
|
||||
})
|
||||
ctx.on('session/event', (session, event) => {
|
||||
this.contain(() => {
|
||||
this.captureEvent(session, event)
|
||||
})
|
||||
})
|
||||
// Parallel listeners are awaited by the loop at turn end; returning void
|
||||
// (not the SDK's flush promise) is the turn-latency contract.
|
||||
ctx.on('session/flush', (session) => {
|
||||
this.contain(() => {
|
||||
this.hintFlush(session)
|
||||
})
|
||||
})
|
||||
ctx.on('agent/error', ({ agent, turn, step, error }) => {
|
||||
this.contain(() => {
|
||||
this.relayAgentError(agent, turn, step, error)
|
||||
})
|
||||
})
|
||||
for (const session of ctx.sessions.list()) {
|
||||
this.adopt(session)
|
||||
}
|
||||
}
|
||||
ctx.effect(() => async () => {
|
||||
// Sessions still adopted here are alive through whole-application
|
||||
// teardown, so capture the marker before the backend quiesces.
|
||||
for (const session of this.adopted) {
|
||||
this.contain(() => {
|
||||
this.deliver(session, { record: this.redact(shutdownRecord(session)) })
|
||||
})
|
||||
}
|
||||
try {
|
||||
await this.backend.shutdown()
|
||||
} catch (error) {
|
||||
this.ctx.logger.warn(`telemetry: backend shutdown failed: ${String(error)}`)
|
||||
}
|
||||
}, 'telemetry capture')
|
||||
}
|
||||
|
||||
/**
|
||||
* Project and hand over the canonical session-log suffix after the handoff
|
||||
* cursor, optionally stopping at an inclusive sequence boundary. Redaction
|
||||
* runs during this call, so an on-demand caller retains no copied records
|
||||
* before requesting capture and uses the policy mounted at that time.
|
||||
* Backend and policy failures remain contained per event and do not starve
|
||||
* later events in the same replay.
|
||||
* @param session - session whose current canonical-log prefix may be handed over.
|
||||
* @param throughSeq - optional last sequence included in this capture.
|
||||
*/
|
||||
captureSession(session: Session, throughSeq?: number): void {
|
||||
const cursor = handoffCursor.get(session) ?? session.firstLiveSeq - 1
|
||||
// Containment is PER EVENT: one rejected record is withheld fail-closed
|
||||
// while the rest of the historical replay proceeds.
|
||||
for (const event of session.events) {
|
||||
if (throughSeq !== undefined && event.seq > throughSeq) break
|
||||
this.contain(() => {
|
||||
if (event.seq <= cursor) this.track(session, event)
|
||||
else this.captureEvent(session, event)
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Adopt a session: replay its log THROUGH the projection from the handoff
|
||||
* cursor, then rely on the firehose for everything after. When no cursor
|
||||
* survived, replay starts at the session's construction boundary
|
||||
* (`firstLiveSeq`), not seq 0: constructor seeds never publish on the
|
||||
* firehose, and their content already left the process under another
|
||||
* identity — the same id in a previous process (resume) or the parent's
|
||||
* stream (fork, stitched by receivers via `session.seed_length`). Events
|
||||
* at or below the start still feed the projection state (first-chunk
|
||||
* tracking) without being re-handed, so a resumed fiber drops mid-step
|
||||
* chunk continuations exactly like the fiber that saw the step begin. The
|
||||
* cost, accepted with the seam's at-most-once stance: a resume no longer
|
||||
* backfills records a previous process failed to deliver.
|
||||
* @param session - the live session to adopt; a second adoption is a no-op.
|
||||
*/
|
||||
private adopt(session: Session): void {
|
||||
if (this.adopted.has(session)) return
|
||||
this.adopted.add(session)
|
||||
this.captureSession(session)
|
||||
}
|
||||
|
||||
/** Feed the chunk projection without handing off — the ≤cursor half of re-adoption. */
|
||||
private track(session: Session, event: SessionEvent): void {
|
||||
if (event.type === 'assistant/chunk') {
|
||||
this.seen(session).add(`${event.data.turn}:${event.data.step}`)
|
||||
}
|
||||
}
|
||||
|
||||
/** Project, redact, and hand one event to the backend. */
|
||||
private captureEvent(session: Session, event: SessionEvent): void {
|
||||
if (event.type === 'assistant/chunk') {
|
||||
const key = `${event.data.turn}:${event.data.step}`
|
||||
const seen = this.seen(session)
|
||||
// Fixed chunk projection: only the first chunk of each (turn, step)
|
||||
// ships — the stream-started signal; content is byte-complete in the
|
||||
// step's assembled assistant/message. Dropped chunks do not advance
|
||||
// the cursor, so re-adoption re-drops them deterministically.
|
||||
if (seen.has(key)) return
|
||||
seen.add(key)
|
||||
}
|
||||
this.deliver(session, {
|
||||
record: this.redact({
|
||||
channel: 'ledger',
|
||||
time: event.time,
|
||||
severity: severityOf(event),
|
||||
attributes: identityOf(session, event),
|
||||
// The canonical event object is mutable and the backend serializes
|
||||
// later; append-time validation guarantees this clone cannot throw.
|
||||
body: structuredClone(event.data),
|
||||
}),
|
||||
seq: event.seq,
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the `telemetry/record` waterfall at capture time. The innermost `next`
|
||||
* passes the record through unchanged — the seam ships no rules; exported
|
||||
* data is as clean as the listeners a deployment mounts. Callers run inside
|
||||
* {@link contain}, so a throwing rule withholds the record instead of
|
||||
* reaching the loop (fail-closed). On-demand capture invokes this waterfall
|
||||
* while reading the canonical session log, not when the event was appended.
|
||||
*/
|
||||
private redact(record: TelemetryRecord): TelemetryRecord {
|
||||
return this.ctx.waterfall('telemetry/record', record, () => record)
|
||||
}
|
||||
|
||||
/** Hand one redacted record to the backend, then advance its ledger cursor. */
|
||||
private deliver(session: Session, pending: ProjectedRecord): void {
|
||||
this.backend.emit(pending.record)
|
||||
if (pending.seq !== undefined) handoffCursor.set(session, pending.seq)
|
||||
}
|
||||
|
||||
/** Forward the turn-end boundary to the backend's optional flush hint. */
|
||||
private hintFlush(session: Session): void {
|
||||
if (this.adopted.has(session)) this.backend.flush?.()
|
||||
}
|
||||
|
||||
/** Relay one `agent/error` bus emission as an `agent-error` operational record. */
|
||||
private relayAgentError(agent: Agent, turn: number, step: number, error: unknown): void {
|
||||
const detail = errorDetail(error)
|
||||
this.deliver(agent.session, {
|
||||
record: this.redact({
|
||||
channel: 'ops',
|
||||
time: Date.now(),
|
||||
severity: 'error',
|
||||
attributes: {
|
||||
'telemetry.op': 'agent-error',
|
||||
'session.id': String(agent.session.id),
|
||||
'agent.id': agent.id,
|
||||
'error.name': detail.name,
|
||||
turn,
|
||||
step,
|
||||
},
|
||||
body: detail,
|
||||
}),
|
||||
})
|
||||
}
|
||||
|
||||
/** Lazily create the per-session first-chunk tracking set. */
|
||||
private seen(session: Session): Set<string> {
|
||||
let set = this.chunkSeen.get(session)
|
||||
if (!set) this.chunkSeen.set(session, set = new Set())
|
||||
return set
|
||||
}
|
||||
|
||||
/**
|
||||
* Run one capture-side step with its exception contained: cordis `emit`
|
||||
* is stop-on-throw, so a throwing listener would starve every subscriber
|
||||
* registered after this plugin — nothing from the backend may escape.
|
||||
*/
|
||||
private contain(step: () => void): void {
|
||||
try {
|
||||
step()
|
||||
} catch (error) {
|
||||
this.ctx.logger.warn(`telemetry: capture step failed: ${String(error)}`)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the per-session clean-exit marker: emitted at the session's own
|
||||
* disposal edge, or at coordinator dispose for sessions still alive then.
|
||||
*/
|
||||
function shutdownRecord(session: Session): TelemetryRecord {
|
||||
return {
|
||||
channel: 'ops',
|
||||
time: Date.now(),
|
||||
severity: 'info',
|
||||
attributes: { 'telemetry.op': 'shutdown', 'session.id': String(session.id) },
|
||||
body: { op: 'shutdown' },
|
||||
}
|
||||
}
|
||||
|
||||
/** Map an event's own outcome flag to the pre-baked alerting severity. */
|
||||
function severityOf(event: SessionEvent): TelemetrySeverity {
|
||||
switch (event.type) {
|
||||
case 'tool/result':
|
||||
return event.data.message.content[0].isError === true ? 'error' : 'info'
|
||||
case 'turn/end':
|
||||
return event.data.reason.kind === 'error' ? 'error' : 'info'
|
||||
default:
|
||||
// Merge-extensible fall-through (no assertNever): event types this seam
|
||||
// does not depend on — including plugin-merged ones it never heard of —
|
||||
// pass through as info; their owners' outcome semantics stay theirs.
|
||||
return 'info'
|
||||
}
|
||||
}
|
||||
|
||||
/** Normalize the live bus's arbitrary thrown value into the stable operational-record shape. */
|
||||
function errorDetail(error: unknown): { name: string; message: string } {
|
||||
const normalized = error instanceof Error ? error : new Error(String(error))
|
||||
return { name: normalized.name, message: normalized.message }
|
||||
}
|
||||
|
||||
/** Build the minimal identity attributes: envelope plus self-contained header facts. */
|
||||
function identityOf(session: Session, event: SessionEvent): Record<string, string | number> {
|
||||
const attributes: Record<string, string | number> = {
|
||||
'session.id': String(session.id),
|
||||
'event.type': event.type,
|
||||
'event.seq': event.seq,
|
||||
}
|
||||
const { cwd, parentSession, seedLength } = session.header
|
||||
if (cwd !== undefined) attributes['session.cwd'] = cwd
|
||||
if (parentSession !== undefined) attributes['session.parent_id'] = String(parentSession)
|
||||
// The durable fork boundary: a forked stream starts here, and its prefix
|
||||
// lives in the parent's stream — receivers stitch on (parent_id, seed_length).
|
||||
if (seedLength !== undefined) attributes['session.seed_length'] = seedLength
|
||||
return attributes
|
||||
}
|
||||
161
packages/session/session-telemetry/src/index.ts
Normal file
161
packages/session/session-telemetry/src/index.ts
Normal file
@@ -0,0 +1,161 @@
|
||||
/**
|
||||
* Telemetry seam for the DeepSeek Harness.
|
||||
*
|
||||
* The seam owns the CAPTURE side of session-event reporting — which records
|
||||
* exist (the chunk projection), what they carry (the logical record), when
|
||||
* they are captured (adoption, the per-append firehose, lifecycle
|
||||
* forwarding), live versus on-demand canonical-log capture, and the HMR
|
||||
* cursor. Everything downstream of
|
||||
* {@link Telemetry.emit} — batching, retry, queueing, and loss policy — is the
|
||||
* reporting SDK's territory and is deliberately not modelled here. The
|
||||
* design and its trade-offs are pinned in
|
||||
* .agents/notes/implemented/feature/2026-07-23-session-telemetry-otel-revival.md.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-session-telemetry
|
||||
*/
|
||||
|
||||
import { Context, Service } from 'cordis'
|
||||
|
||||
declare module 'cordis' {
|
||||
interface Context {
|
||||
telemetry: Telemetry
|
||||
}
|
||||
|
||||
interface Events {
|
||||
/**
|
||||
* Transform one outbound record before it reaches the backend. This
|
||||
* waterfall is the seam's redaction extension point. It ships NO rules
|
||||
* of its own: the
|
||||
* innermost `next()` passes the record through unchanged, and with no
|
||||
* listener mounted records reach the backend as captured, so exported
|
||||
* data is exactly as clean as the rules a deployment mounts. Listeners
|
||||
* stack by transforming `next()`'s return value; returning without
|
||||
* `next()` replaces everything beneath. Dispatched synchronously on the
|
||||
* capture hot path inside the coordinator's containment: a throwing
|
||||
* listener withholds that one record (fail-closed) and never reaches the
|
||||
* agent loop. Live capture dispatches at append time; on-demand capture
|
||||
* dispatches while reading the canonical log. Redaction applies to the
|
||||
* exported copy only; the canonical session log is never rewritten.
|
||||
* @param record - the candidate record, already the coordinator's own deep
|
||||
* copy; listeners return a (possibly new) record and must not mutate it.
|
||||
* @mode waterfall
|
||||
*/
|
||||
'telemetry/record'(record: TelemetryRecord, next: () => TelemetryRecord): TelemetryRecord
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Severity of a telemetry record, pre-mapped at capture so a receiver can
|
||||
* alert with zero configuration: `error` for events whose own outcome flag
|
||||
* says so (the tool-result block's `isError`, `turn/end` error reasons) and for
|
||||
* `agent-error` operational records. Captured events otherwise default to
|
||||
* `info`; `warn` remains available to `telemetry/record` policies and
|
||||
* backends.
|
||||
*/
|
||||
export type TelemetrySeverity = 'info' | 'warn' | 'error'
|
||||
|
||||
/**
|
||||
* One logical record handed to a backend — the seam's whole outbound
|
||||
* vocabulary. Ledger records mirror session-log events one-to-one;
|
||||
* operational records (`channel: 'ops'`) carry the two signals with no log
|
||||
* home (`agent-error`, `shutdown`) and deliberately omit `event.seq`-style
|
||||
* identity so they can never be mistaken for ledger rows.
|
||||
*/
|
||||
export interface TelemetryRecord {
|
||||
/** Ledger (session-log mirror) or ops (operational signal) channel; backends keep the two under separate instrumentation scopes. */
|
||||
channel: 'ledger' | 'ops'
|
||||
/** Unix epoch milliseconds — the source event's append time for ledger records, the emission time for ops records. */
|
||||
time: number
|
||||
/** Pre-mapped alerting severity; see {@link TelemetrySeverity}. */
|
||||
severity: TelemetrySeverity
|
||||
/**
|
||||
* Identity attributes, deliberately minimal: ledger records carry
|
||||
* `session.id`, `event.type`, `event.seq`, plus `session.cwd` /
|
||||
* `session.parent_id` / `session.seed_length` when the header has them;
|
||||
* ops records carry `telemetry.op`, `session.id`, and (for `agent-error`)
|
||||
* `agent.id`, `turn`, `step`, `error.name`. Anything recoverable from the
|
||||
* body is intentionally NOT duplicated here.
|
||||
*/
|
||||
attributes: Record<string, string | number>
|
||||
/**
|
||||
* The complete payload: a deep copy of the session event's `data` for
|
||||
* ledger records (JSON-serializable by `Session.append`'s own
|
||||
* validation), or the op payload for ops records. Never mutated after
|
||||
* handoff.
|
||||
*/
|
||||
body: unknown
|
||||
}
|
||||
|
||||
/**
|
||||
* The backend contract the coordinator hands records to — the minimum any
|
||||
* reporting SDK satisfies with zero bending. {@link Telemetry} is its
|
||||
* service-registered form; tests compose the coordinator with a bare
|
||||
* implementation of this interface.
|
||||
*/
|
||||
export interface TelemetryBackend {
|
||||
/**
|
||||
* Hand one record to the backend's pipeline. MUST be a non-blocking
|
||||
* enqueue — the coordinator calls this synchronously from the
|
||||
* `session/event` hot path or an explicit canonical-log capture, so anything
|
||||
* slower than a queue push would tax the agent loop or feedback handling.
|
||||
* Errors thrown here are contained by the coordinator and logged; they
|
||||
* never reach the loop.
|
||||
* @param record - the logical record to report; owned by the backend after the call.
|
||||
*/
|
||||
emit(record: TelemetryRecord): void
|
||||
/**
|
||||
* Optional hint that a natural boundary (turn end) passed — a backend may
|
||||
* forward it to its SDK's flush so records land at turn boundaries. Called
|
||||
* fire-and-forget; implementations must not block and must not throw
|
||||
* meaningfully (the coordinator contains exceptions). Most backends should
|
||||
* leave this unimplemented and let their SDK's own batching cadence govern
|
||||
* export timing: a backend that does implement it owns the interaction
|
||||
* between its concurrent flushes and {@link shutdown}'s drain (the OTel
|
||||
* backend removed its implementation for exactly that hazard — see the
|
||||
* revival Agent Note).
|
||||
*/
|
||||
flush?(): void
|
||||
/**
|
||||
* Forward the fiber's disposal to the SDK: flush whatever is queued and
|
||||
* reach quiescence, per the SDK's own shutdown contract. Everything
|
||||
* emitted before this call must still be delivered — including records
|
||||
* enqueued while a {@link flush} hint is in flight, so a backend whose SDK
|
||||
* guards against concurrent flushes orders behind the outstanding one (the
|
||||
* coordinator emits its dispose-time `shutdown` markers immediately before
|
||||
* calling this). Awaited by the coordinator's dispose; a rejection is
|
||||
* logged as a warning and never fails application teardown.
|
||||
* The coordinator captures dispose-time shutdown markers immediately before
|
||||
* this call for live capture; on-demand capture creates no ops records.
|
||||
* @returns resolves when the backend's pipeline has quiesced.
|
||||
*/
|
||||
shutdown(): Promise<void>
|
||||
}
|
||||
|
||||
/**
|
||||
* The backend contract in its loadable form: one implementation per context —
|
||||
* the cordis `Service` registration under the `telemetry` key throws on a
|
||||
* duplicate, cordis' standard behavior. A backend composes a
|
||||
* {@link TelemetryCoordinator} in its constructor to install the capture side.
|
||||
*/
|
||||
export abstract class Telemetry extends Service implements TelemetryBackend {
|
||||
constructor(ctx: Context) {
|
||||
super(ctx, 'telemetry')
|
||||
}
|
||||
|
||||
/**
|
||||
* See {@link TelemetryBackend.emit} — the seam declaration is the contract's one home.
|
||||
* @param record - the logical record to report; owned by the backend after the call.
|
||||
*/
|
||||
abstract emit(record: TelemetryRecord): void
|
||||
|
||||
/** See {@link TelemetryBackend.flush}. */
|
||||
flush?(): void
|
||||
|
||||
/**
|
||||
* See {@link TelemetryBackend.shutdown}.
|
||||
* @returns resolves when the backend's pipeline has quiesced.
|
||||
*/
|
||||
abstract shutdown(): Promise<void>
|
||||
}
|
||||
|
||||
export { TelemetryCoordinator, type TelemetryCapture } from './coordinator.ts'
|
||||
32
packages/session/session-telemetry/src/invariant.ts
Normal file
32
packages/session/session-telemetry/src/invariant.ts
Normal file
@@ -0,0 +1,32 @@
|
||||
/**
|
||||
* Package-owned invariant companion for `@deepseek-ai/dsh-session-telemetry`.
|
||||
* @module @deepseek-ai/dsh-session-telemetry/invariant
|
||||
*/
|
||||
|
||||
/* jscpd:ignore-start */
|
||||
import type { Context } from 'cordis'
|
||||
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
|
||||
|
||||
const PACKAGE_NAME = '@deepseek-ai/dsh-session-telemetry'
|
||||
|
||||
/** Cordis companion plugin name. */
|
||||
export const name = 'session-telemetry-invariant'
|
||||
/** Service required before the companion can reserve package ownership. */
|
||||
export const inject = ['invariants']
|
||||
|
||||
/**
|
||||
* No runtime invariant: the seam's whole output is the backend handoff — a
|
||||
* synchronous `emit()` call outside every authoritative event stream — and its
|
||||
* capture side never appends session events, so no event/data relation exists
|
||||
* for an independent companion to observe.
|
||||
*/
|
||||
const install: InvariantInstaller = () => {}
|
||||
|
||||
/**
|
||||
* Register this package's invariant companion.
|
||||
* @param ctx - Cordis context carrying the invariant service.
|
||||
* @returns the installed registration's disposer after setup succeeds.
|
||||
*/
|
||||
export const apply = (ctx: Context): Promise<() => void> =>
|
||||
Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
|
||||
/* jscpd:ignore-end */
|
||||
129
packages/session/session-telemetry/tests/redact.spec.ts
Normal file
129
packages/session/session-telemetry/tests/redact.spec.ts
Normal file
@@ -0,0 +1,129 @@
|
||||
import { createUserMessage } from '@deepseek-ai/dsh-llm'
|
||||
/**
|
||||
* The `telemetry/record` waterfall contract: pass-through when no listener is
|
||||
* mounted, listener stacking and replacement, ops-record coverage, the
|
||||
* untouched canonical log, and the fail-closed containment of a throwing rule.
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import {
|
||||
TelemetryCoordinator,
|
||||
type TelemetryBackend,
|
||||
type TelemetryRecord,
|
||||
} from '../src/index.ts'
|
||||
|
||||
const FIXTURE_SECRET = 'sk-fixture1234567890'
|
||||
|
||||
class CollectingBackend implements TelemetryBackend {
|
||||
records: TelemetryRecord[] = []
|
||||
emit(record: TelemetryRecord): void {
|
||||
this.records.push(record)
|
||||
}
|
||||
async shutdown(): Promise<void> {}
|
||||
}
|
||||
|
||||
async function setup() {
|
||||
const backend = new CollectingBackend()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin({
|
||||
name: 'fake-telemetry',
|
||||
inject: ['sessions'],
|
||||
apply: (inner: Context) => void new TelemetryCoordinator(inner, backend),
|
||||
})
|
||||
return { ctx, backend, fiber }
|
||||
}
|
||||
|
||||
describe('telemetry/record waterfall', () => {
|
||||
it('passes records through unchanged when no listener is mounted', async () => {
|
||||
const { ctx, backend } = await setup()
|
||||
const session = ctx.sessions.create(SessionId('w'))
|
||||
session.append('user/message', createUserMessage({
|
||||
content: [{ type: 'text', text: `key ${FIXTURE_SECRET}` }], source: { kind: 'user' },
|
||||
}), { surfaceOp: 'append' })
|
||||
const body = backend.records[0]!.body as { content: { text: string }[] }
|
||||
expect(body.content[0]!.text).toBe(`key ${FIXTURE_SECRET}`)
|
||||
})
|
||||
|
||||
it('applies a mounted rule to every outbound record, ops records included', async () => {
|
||||
const { ctx, backend, fiber } = await setup()
|
||||
ctx.on('telemetry/record', (_record, next) => {
|
||||
const record = next()
|
||||
return { ...record, body: { scrubbed: true } }
|
||||
})
|
||||
const session = ctx.sessions.create(SessionId('rule'))
|
||||
session.append('user/message', createUserMessage({
|
||||
content: [{ type: 'text', text: FIXTURE_SECRET }], source: { kind: 'user' },
|
||||
}), { surfaceOp: 'append' })
|
||||
expect(backend.records[0]!.body).toEqual({ scrubbed: true })
|
||||
// The dispose-time shutdown ops record passes through the same waterfall.
|
||||
await fiber.dispose()
|
||||
const ops = backend.records.filter(record => record.channel === 'ops')
|
||||
expect(ops).toHaveLength(1)
|
||||
expect(ops[0]!.body).toEqual({ scrubbed: true })
|
||||
})
|
||||
|
||||
it('keeps the canonical log untouched by a mounted rule', async () => {
|
||||
const { ctx } = await setup()
|
||||
ctx.on('telemetry/record', (_record, next) => ({ ...next(), body: null }))
|
||||
const session = ctx.sessions.create(SessionId('log'))
|
||||
session.append('user/message', createUserMessage({
|
||||
content: [{ type: 'text', text: FIXTURE_SECRET }], source: { kind: 'user' },
|
||||
}), { surfaceOp: 'append' })
|
||||
const logged = session.events[0]!.data as { content: { text: string }[] }
|
||||
expect(logged.content[0]!.text).toBe(FIXTURE_SECRET)
|
||||
})
|
||||
|
||||
it('stacks listeners outermost-first around next()', async () => {
|
||||
const { ctx, backend } = await setup()
|
||||
const order: string[] = []
|
||||
ctx.on('telemetry/record', (_record, next) => {
|
||||
order.push('outer-before')
|
||||
const record = next()
|
||||
order.push('outer-after')
|
||||
return { ...record, attributes: { ...record.attributes, outer: 1 } }
|
||||
})
|
||||
ctx.on('telemetry/record', (_record, next) => {
|
||||
order.push('inner')
|
||||
const record = next()
|
||||
return { ...record, attributes: { ...record.attributes, inner: 1 } }
|
||||
})
|
||||
const session = ctx.sessions.create(SessionId('stack'))
|
||||
session.append('user/message', createUserMessage({
|
||||
content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' },
|
||||
}), { surfaceOp: 'append' })
|
||||
expect(order).toEqual(['outer-before', 'inner', 'outer-after'])
|
||||
expect(backend.records[0]!.attributes).toMatchObject({ outer: 1, inner: 1 })
|
||||
})
|
||||
|
||||
it('a listener that skips next() replaces everything beneath it', async () => {
|
||||
const { ctx, backend } = await setup()
|
||||
const inner = { called: false }
|
||||
ctx.on('telemetry/record', () => ({ channel: 'ops', time: 0, severity: 'info', attributes: {}, body: 'replaced' } satisfies TelemetryRecord))
|
||||
ctx.on('telemetry/record', (_record, next) => {
|
||||
inner.called = true
|
||||
return next()
|
||||
})
|
||||
const session = ctx.sessions.create(SessionId('veto'))
|
||||
session.append('user/message', createUserMessage({
|
||||
content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' },
|
||||
}), { surfaceOp: 'append' })
|
||||
expect(backend.records[0]!.body).toBe('replaced')
|
||||
expect(inner.called).toBe(false)
|
||||
})
|
||||
|
||||
it('a throwing rule withholds the record fail-closed without disturbing the log', async () => {
|
||||
const { ctx, backend } = await setup()
|
||||
ctx.on('telemetry/record', () => {
|
||||
throw new Error('rule exploded')
|
||||
})
|
||||
const session = ctx.sessions.create(SessionId('closed'))
|
||||
session.append('user/message', createUserMessage({
|
||||
content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' },
|
||||
}), { surfaceOp: 'append' })
|
||||
expect(backend.records).toHaveLength(0)
|
||||
expect(session.events).toHaveLength(1)
|
||||
})
|
||||
})
|
||||
552
packages/session/session-telemetry/tests/telemetry.spec.ts
Normal file
552
packages/session/session-telemetry/tests/telemetry.spec.ts
Normal file
@@ -0,0 +1,552 @@
|
||||
import { createToolResultMessage, createUserMessage } from '@deepseek-ai/dsh-llm'
|
||||
/**
|
||||
* Coordinator semantics against a bare fake backend — the RFC's named unit
|
||||
* tier for the seam: adoption (fresh, seeded, re-adoption via the handoff
|
||||
* cursor), the fixed chunk projection, deep-copy isolation, turn-latency and
|
||||
* dispose-ordering pins, failure containment, and the `agent/error` relay.
|
||||
*/
|
||||
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import SessionStore, { SessionId, type Session, type SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import type { Agent } from '@deepseek-ai/dsh-agent'
|
||||
import {
|
||||
TelemetryCoordinator,
|
||||
type TelemetryBackend,
|
||||
type TelemetryCapture,
|
||||
type TelemetryRecord,
|
||||
} from '../src/index.ts'
|
||||
|
||||
declare module '@deepseek-ai/dsh-session' {
|
||||
interface SessionEventMap {
|
||||
/**
|
||||
* Test-only merged event proving unknown types flow through unchanged.
|
||||
* @mode emit
|
||||
* @param payload - opaque test payload
|
||||
*/
|
||||
'telemetry-test/opaque': { payload: { nested: string[] } }
|
||||
}
|
||||
}
|
||||
|
||||
class FakeBackend implements TelemetryBackend {
|
||||
records: TelemetryRecord[] = []
|
||||
calls: string[] = []
|
||||
emitError: Error | undefined
|
||||
rejectSeq: number | undefined
|
||||
shutdownError: Error | undefined
|
||||
shutdownResolved = false
|
||||
|
||||
emit(record: TelemetryRecord): void {
|
||||
if (this.emitError) throw this.emitError
|
||||
if (this.rejectSeq !== undefined && record.attributes['event.seq'] === this.rejectSeq) {
|
||||
throw new Error(`backend rejected seq ${this.rejectSeq}`)
|
||||
}
|
||||
this.records.push(record)
|
||||
this.calls.push(`emit:${String(record.attributes['event.seq'] ?? record.attributes['telemetry.op'])}`)
|
||||
}
|
||||
|
||||
flush = vi.fn()
|
||||
|
||||
async shutdown(): Promise<void> {
|
||||
this.calls.push('shutdown')
|
||||
await new Promise(resolve => setTimeout(resolve, 5))
|
||||
if (this.shutdownError) throw this.shutdownError
|
||||
this.shutdownResolved = true
|
||||
}
|
||||
|
||||
ledger(): TelemetryRecord[] {
|
||||
return this.records.filter(r => r.channel === 'ledger')
|
||||
}
|
||||
}
|
||||
|
||||
async function setup(
|
||||
backend: FakeBackend = new FakeBackend(),
|
||||
capture: TelemetryCapture = 'live',
|
||||
) {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
let coordinator!: TelemetryCoordinator
|
||||
const fiber = await ctx.plugin({
|
||||
name: 'fake-telemetry',
|
||||
inject: ['sessions'],
|
||||
apply: (inner: Context) => {
|
||||
coordinator = new TelemetryCoordinator(inner, backend, capture)
|
||||
},
|
||||
})
|
||||
return { ctx, backend, coordinator, fiber }
|
||||
}
|
||||
|
||||
function liveSession(ctx: Context, id = `s-${Math.random().toString(36).slice(2)}`): Session {
|
||||
return ctx.sessions.create(SessionId(id), { meta: {} })
|
||||
}
|
||||
|
||||
function appendTurn(session: Session): void {
|
||||
session.append('turn/start', { turn: 1 })
|
||||
session.append('user/message', createUserMessage({
|
||||
content: [{ type: 'text', text: 'hello' }], source: { kind: 'user' },
|
||||
}), { surfaceOp: 'append' })
|
||||
}
|
||||
|
||||
describe('TelemetryCoordinator capture', () => {
|
||||
it('hands every appended event over with envelope identity and cloned body', async () => {
|
||||
const { ctx, backend } = await setup()
|
||||
const session = liveSession(ctx, 'cap')
|
||||
appendTurn(session)
|
||||
|
||||
const start = backend.ledger()[0]!
|
||||
const message = backend.ledger()[1]!
|
||||
expect(start.attributes).toMatchObject({ 'session.id': 'cap', 'event.type': 'turn/start', 'event.seq': 0 })
|
||||
expect(start.time).toBe(session.events[0]!.time)
|
||||
expect(start.severity).toBe('info')
|
||||
expect(message.attributes['event.seq']).toBe(1)
|
||||
// Deep-copy isolation: mutating the handed-off body never reaches the log.
|
||||
;(message.body as { content: { text: string }[] }).content[0]!.text = 'tampered'
|
||||
const logged = session.events[1] as SessionEvent<'user/message'>
|
||||
expect(logged.data.content[0]).toMatchObject({ text: 'hello' })
|
||||
})
|
||||
|
||||
it('stamps header facts on every record when present', async () => {
|
||||
const { ctx, backend } = await setup()
|
||||
const parent = SessionId('parent')
|
||||
const session = ctx.sessions.create(SessionId('child'), { meta: { cwd: '/tmp/proj', parentSession: parent } })
|
||||
appendTurn(session)
|
||||
for (const record of backend.ledger()) {
|
||||
expect(record.attributes['session.cwd']).toBe('/tmp/proj')
|
||||
expect(record.attributes['session.parent_id']).toBe('parent')
|
||||
}
|
||||
})
|
||||
|
||||
it('maps outcome flags to severity, unknown types falling through as info', async () => {
|
||||
const { ctx, backend } = await setup()
|
||||
const session = liveSession(ctx)
|
||||
session.append('turn/start', { turn: 1 })
|
||||
session.append('tool/result', {
|
||||
turn: 1, step: 1,
|
||||
message: createToolResultMessage({
|
||||
callId: 'c1' as never,
|
||||
content: [],
|
||||
isError: true,
|
||||
}),
|
||||
}, { surfaceOp: 'append' })
|
||||
session.append('tool/result', {
|
||||
turn: 1, step: 1,
|
||||
message: createToolResultMessage({
|
||||
callId: 'c2' as never,
|
||||
content: [],
|
||||
isError: false,
|
||||
}),
|
||||
}, { surfaceOp: 'append' })
|
||||
session.append('telemetry-test/opaque', { payload: { nested: [] } })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'error', error: { message: 'boom', code: 'UNKNOWN' } } })
|
||||
const severities = backend.ledger().map(r => [r.attributes['event.type'], r.severity])
|
||||
expect(severities).toEqual([
|
||||
['turn/start', 'info'],
|
||||
['tool/result', 'error'],
|
||||
['tool/result', 'info'],
|
||||
['telemetry-test/opaque', 'info'],
|
||||
['turn/end', 'error'],
|
||||
])
|
||||
})
|
||||
|
||||
it('passes unknown merged event types through unchanged', async () => {
|
||||
const { ctx, backend } = await setup()
|
||||
const session = liveSession(ctx)
|
||||
session.append('telemetry-test/opaque', { payload: { nested: ['a', 'b'] } })
|
||||
const record = backend.ledger()[0]!
|
||||
expect(record.attributes['event.type']).toBe('telemetry-test/opaque')
|
||||
expect(record.severity).toBe('info')
|
||||
expect(record.body).toEqual({ payload: { nested: ['a', 'b'] } })
|
||||
})
|
||||
|
||||
it('ships only the first chunk of each (turn, step), per session', async () => {
|
||||
const { ctx, backend } = await setup()
|
||||
const a = liveSession(ctx, 'a')
|
||||
const b = liveSession(ctx, 'b')
|
||||
const chunk = (s: Session, turn: number, step: number, text: string) =>
|
||||
s.append('assistant/chunk', { turn, step, chunk: { type: 'text-delta', index: 0, text } })
|
||||
chunk(a, 1, 1, 'a11-first')
|
||||
chunk(a, 1, 1, 'a11-second')
|
||||
chunk(a, 1, 2, 'a12-first')
|
||||
chunk(b, 1, 1, 'b11-first')
|
||||
chunk(b, 1, 1, 'b11-second')
|
||||
const shipped = backend.ledger().map(r => [r.attributes['session.id'], (r.body as { chunk: { text: string } }).chunk.text])
|
||||
expect(shipped).toEqual([
|
||||
['a', 'a11-first'],
|
||||
['a', 'a12-first'],
|
||||
['b', 'b11-first'],
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
describe('TelemetryCoordinator on-demand capture', () => {
|
||||
it('captures one canonical-log prefix at a time without following later events', async () => {
|
||||
const { ctx, backend, coordinator } = await setup(new FakeBackend(), 'on-demand')
|
||||
const session = liveSession(ctx, 'on-demand-prefix')
|
||||
appendTurn(session)
|
||||
const firstBoundary = session.events[1]!.seq
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
expect(backend.records).toEqual([])
|
||||
|
||||
coordinator.captureSession(session, firstBoundary)
|
||||
expect(backend.ledger().map(record => record.attributes['event.type'])).toEqual([
|
||||
'turn/start',
|
||||
'user/message',
|
||||
])
|
||||
|
||||
expect(backend.ledger()).toHaveLength(2)
|
||||
coordinator.captureSession(session)
|
||||
coordinator.captureSession(session)
|
||||
expect(backend.ledger().map(record => record.attributes['event.type'])).toEqual([
|
||||
'turn/start',
|
||||
'user/message',
|
||||
'turn/end',
|
||||
])
|
||||
})
|
||||
|
||||
it('runs the currently mounted redaction policy during canonical-log capture', async () => {
|
||||
const { ctx, backend, coordinator } = await setup(new FakeBackend(), 'on-demand')
|
||||
const session = liveSession(ctx, 'on-demand-redacted')
|
||||
session.append('turn/start', { turn: 1 })
|
||||
const disposeRule = ctx.on('telemetry/record', (_record, next) => ({
|
||||
...next(),
|
||||
body: { scrubbed: true },
|
||||
}))
|
||||
|
||||
coordinator.captureSession(session)
|
||||
expect(backend.ledger()[0]!.body).toEqual({ scrubbed: true })
|
||||
disposeRule()
|
||||
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
coordinator.captureSession(session)
|
||||
expect(backend.ledger()[1]!.body).toEqual({ turn: 1, reason: { kind: 'completed' } })
|
||||
})
|
||||
|
||||
it('contains each backend failure independently while replaying a prefix', async () => {
|
||||
const backend = new FakeBackend()
|
||||
backend.rejectSeq = 1
|
||||
const { ctx, coordinator } = await setup(backend, 'on-demand')
|
||||
const warn = vi.spyOn(ctx.logger, 'warn').mockImplementation(() => {})
|
||||
const session = liveSession(ctx, 'on-demand-failure')
|
||||
appendTurn(session)
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
|
||||
coordinator.captureSession(session)
|
||||
expect(backend.ledger().map(record => record.attributes['event.seq'])).toEqual([0, 2])
|
||||
expect(warn).toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('captures a pending prefix after coordinator reload without retained records', async () => {
|
||||
const first = new FakeBackend()
|
||||
const { ctx, fiber } = await setup(first, 'on-demand')
|
||||
const session = liveSession(ctx, 'on-demand-reload')
|
||||
session.append('turn/start', { turn: 1 })
|
||||
await fiber.dispose()
|
||||
expect(first.records).toEqual([])
|
||||
|
||||
const second = new FakeBackend()
|
||||
let coordinator!: TelemetryCoordinator
|
||||
await ctx.plugin({
|
||||
name: 'fake-telemetry-after-on-demand-reload',
|
||||
inject: ['sessions'],
|
||||
apply: (inner: Context) => {
|
||||
coordinator = new TelemetryCoordinator(inner, second, 'on-demand')
|
||||
},
|
||||
})
|
||||
coordinator.captureSession(session)
|
||||
expect(second.ledger().map(record => record.attributes['event.seq'])).toEqual([0])
|
||||
})
|
||||
|
||||
it('registers no continuous capture, flush, or ops listeners', async () => {
|
||||
const { ctx, backend, coordinator, fiber } = await setup(new FakeBackend(), 'on-demand')
|
||||
const redact = vi.fn((_record: TelemetryRecord, next: () => TelemetryRecord) => next())
|
||||
ctx.on('telemetry/record', redact)
|
||||
const session = liveSession(ctx, 'on-demand-ledger-only')
|
||||
session.append('turn/start', { turn: 1 })
|
||||
await ctx.parallel('session/flush', session)
|
||||
const agent = { id: 'agent-1', session } as Agent
|
||||
ctx.emit('agent/error', { agent, turn: 1, step: 1, error: new Error('local only') })
|
||||
expect(backend.flush).not.toHaveBeenCalled()
|
||||
expect(backend.records).toEqual([])
|
||||
expect(redact).not.toHaveBeenCalled()
|
||||
|
||||
coordinator.captureSession(session)
|
||||
expect(redact).toHaveBeenCalledTimes(1)
|
||||
await fiber.dispose()
|
||||
expect(backend.records.map(record => record.channel)).toEqual(['ledger'])
|
||||
})
|
||||
})
|
||||
|
||||
describe('TelemetryCoordinator adoption', () => {
|
||||
it('exports an unpublished suffix without re-exporting constructor history', async () => {
|
||||
const backend = new FakeBackend()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const parent = liveSession(ctx, 'seed-parent')
|
||||
appendTurn(parent)
|
||||
await ctx.plugin({
|
||||
name: 'fake-telemetry',
|
||||
inject: ['sessions'],
|
||||
apply: (inner: Context) => void new TelemetryCoordinator(inner, backend),
|
||||
})
|
||||
const child = ctx.sessions.prepare(SessionId('seeded'), { seed: [...parent.events], meta: {} })
|
||||
child.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
ctx.sessions.enter(child)
|
||||
ctx.sessions.announce(child)
|
||||
|
||||
const seqs = backend.ledger().map(r => [r.attributes['session.id'], r.attributes['event.seq']])
|
||||
expect(seqs).toEqual(expect.arrayContaining([['seed-parent', 0], ['seed-parent', 1]]))
|
||||
// 2 end-seed, 3 turn/end: both this lifecycle's own writes, while
|
||||
// inherited 0-1 stay with the parent stream.
|
||||
expect(seqs.filter(([id]) => id === 'seeded')).toEqual([['seeded', 2], ['seeded', 3]])
|
||||
})
|
||||
|
||||
it('resume shape: a full-log seed exports only its own end-seed and rebuilds the chunk projection', async () => {
|
||||
const backend = new FakeBackend()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const donor = ctx.sessions.create(SessionId('donor'), { meta: {} })
|
||||
donor.append('turn/start', { turn: 1 })
|
||||
donor.append('assistant/chunk', { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'first' } })
|
||||
const resumed = ctx.sessions.create(SessionId('resumed'), { seed: [...donor.events], meta: {} })
|
||||
await ctx.plugin({
|
||||
name: 'fake-telemetry',
|
||||
inject: ['sessions'],
|
||||
apply: (inner: Context) => void new TelemetryCoordinator(inner, backend),
|
||||
})
|
||||
const ofResumed = () => backend.ledger()
|
||||
.filter(r => r.attributes['session.id'] === 'resumed')
|
||||
.map(r => r.attributes['event.seq'])
|
||||
// Nothing inherited is re-exported; seq 2 is this session's own first
|
||||
// write — the end-seed event its constructor appended after the seed.
|
||||
expect(ofResumed()).toEqual([2])
|
||||
// The seed fed the projection: the (turn 1, step 1) first chunk already
|
||||
// shipped from the original process, so its continuation is re-dropped…
|
||||
resumed.append('assistant/chunk', { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'continuation' } })
|
||||
expect(ofResumed()).toEqual([2])
|
||||
// …while a new step's first chunk exports normally.
|
||||
resumed.append('assistant/chunk', { turn: 1, step: 2, chunk: { type: 'text-delta', index: 0, text: 'next step' } })
|
||||
expect(ofResumed()).toEqual([2, 4])
|
||||
})
|
||||
|
||||
it('stamps session.seed_length from the header so receivers can stitch fork streams', async () => {
|
||||
const backend = new FakeBackend()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const parent = liveSession(ctx, 'stitch-parent')
|
||||
appendTurn(parent)
|
||||
const child = ctx.sessions.create(SessionId('stitch-child'), {
|
||||
seed: [...parent.events],
|
||||
meta: { parentSession: SessionId('stitch-parent'), seedLength: 2 },
|
||||
})
|
||||
await ctx.plugin({
|
||||
name: 'fake-telemetry',
|
||||
inject: ['sessions'],
|
||||
apply: (inner: Context) => void new TelemetryCoordinator(inner, backend),
|
||||
})
|
||||
child.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
const record = backend.ledger().find(r => r.attributes['session.id'] === 'stitch-child')!
|
||||
expect(record.attributes['session.parent_id']).toBe('stitch-parent')
|
||||
expect(record.attributes['session.seed_length']).toBe(2)
|
||||
})
|
||||
|
||||
it('adopts exactly once when created fires after the sweep', async () => {
|
||||
const backend = new FakeBackend()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
// The enter/announce window: prepare+enter puts the session in the store
|
||||
// (visible to the constructor sweep) before `session/created` fires, so a
|
||||
// coordinator loaded inside that window sees the session twice — sweep
|
||||
// first, created second. The second adoption must be a no-op.
|
||||
const session = ctx.sessions.prepare(SessionId('overlap'))
|
||||
appendTurn(session)
|
||||
ctx.sessions.enter(session)
|
||||
await ctx.plugin({
|
||||
name: 'fake-telemetry',
|
||||
inject: ['sessions'],
|
||||
apply: (inner: Context) => void new TelemetryCoordinator(inner, backend),
|
||||
})
|
||||
expect(backend.ledger()).toHaveLength(2)
|
||||
ctx.sessions.announce(session)
|
||||
expect(backend.ledger()).toHaveLength(2)
|
||||
})
|
||||
|
||||
it('resumes from the handoff cursor across a reload, re-dropping mid-step chunks', async () => {
|
||||
const backend = new FakeBackend()
|
||||
const { ctx, fiber } = await setup(backend)
|
||||
const session = liveSession(ctx, 'hmr')
|
||||
session.append('turn/start', { turn: 1 })
|
||||
session.append('assistant/chunk', { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'first' } })
|
||||
expect(backend.ledger()).toHaveLength(2)
|
||||
|
||||
await fiber.dispose()
|
||||
// The reload window: appends while no telemetry listener is registered.
|
||||
session.append('assistant/chunk', { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'mid-step continuation' } })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
|
||||
const second = new FakeBackend()
|
||||
await ctx.plugin({
|
||||
name: 'fake-telemetry-2',
|
||||
inject: ['sessions'],
|
||||
apply: (inner: Context) => void new TelemetryCoordinator(inner, second),
|
||||
})
|
||||
// Only the window events past the cursor are re-handed, and the mid-step
|
||||
// continuation is re-dropped because ≤cursor events rebuilt the projection.
|
||||
expect(second.ledger().map(r => r.attributes['event.type'])).toEqual(['turn/end'])
|
||||
})
|
||||
|
||||
it('replays past a record the backend rejects: one event withheld, the rest adopted', async () => {
|
||||
const backend = new FakeBackend()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const warn = vi.spyOn(ctx.logger, 'warn').mockImplementation(() => {})
|
||||
const session = liveSession(ctx, 'partial')
|
||||
appendTurn(session)
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
// The backend rejects exactly the middle historical event: fail-closed
|
||||
// must withhold THAT record only — an adoption replay that dies on the
|
||||
// first contained failure would silently skip the rest of the log while
|
||||
// the session stays marked adopted.
|
||||
backend.rejectSeq = 1
|
||||
await ctx.plugin({
|
||||
name: 'fake-telemetry',
|
||||
inject: ['sessions'],
|
||||
apply: (inner: Context) => void new TelemetryCoordinator(inner, backend),
|
||||
})
|
||||
expect(backend.ledger().map(r => r.attributes['event.seq'])).toEqual([0, 2])
|
||||
expect(warn).toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('re-hands the full log when no cursor survived (fresh session object)', async () => {
|
||||
const backend = new FakeBackend()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const session = liveSession(ctx, 'fresh')
|
||||
appendTurn(session)
|
||||
await ctx.plugin({
|
||||
name: 'fake-telemetry',
|
||||
inject: ['sessions'],
|
||||
apply: (inner: Context) => void new TelemetryCoordinator(inner, backend),
|
||||
})
|
||||
expect(backend.ledger().map(r => r.attributes['event.seq'])).toEqual([0, 1])
|
||||
})
|
||||
})
|
||||
|
||||
describe('TelemetryCoordinator lifecycle and containment', () => {
|
||||
it('forwards session/flush as a hint without awaiting backend work', async () => {
|
||||
const { ctx, backend } = await setup()
|
||||
const session = liveSession(ctx)
|
||||
let settled = false
|
||||
backend.flush.mockImplementation(() => {
|
||||
// The backend may kick off arbitrary async work; the loop's parallel must not wait for it.
|
||||
void new Promise(resolve => setTimeout(resolve, 50)).then(() => { settled = true })
|
||||
})
|
||||
await ctx.parallel('session/flush', session)
|
||||
expect(backend.flush).toHaveBeenCalledTimes(1)
|
||||
expect(settled).toBe(false)
|
||||
})
|
||||
|
||||
it('ignores flush hints for sessions it never adopted', async () => {
|
||||
const { ctx, backend } = await setup()
|
||||
const stranger = ctx.sessions.prepare(SessionId('stranger'), { meta: {} })
|
||||
await ctx.parallel('session/flush', stranger)
|
||||
expect(backend.flush).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('emits no marker for a session whose announcement was vetoed before adoption', async () => {
|
||||
const backend = new FakeBackend()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
// A listener registered BEFORE the coordinator vetoes publication: the
|
||||
// store still emits the paired `session/disposed` for rollback, but the
|
||||
// coordinator never saw `session/created` — a marker for a session the
|
||||
// receiver saw no activity from would be noise, not signal.
|
||||
ctx.on('session/created', () => {
|
||||
throw new Error('vetoed by an earlier listener')
|
||||
})
|
||||
await ctx.plugin({
|
||||
name: 'fake-telemetry',
|
||||
inject: ['sessions'],
|
||||
apply: (inner: Context) => void new TelemetryCoordinator(inner, backend),
|
||||
})
|
||||
expect(() => ctx.sessions.create(SessionId('vetoed'), { meta: {} })).toThrow('vetoed')
|
||||
expect(backend.records.filter(r => r.channel === 'ops')).toHaveLength(0)
|
||||
})
|
||||
|
||||
it('emits each adopted session’s shutdown record before awaiting backend shutdown', async () => {
|
||||
const { ctx, backend, fiber } = await setup()
|
||||
liveSession(ctx, 's1')
|
||||
liveSession(ctx, 's2')
|
||||
await fiber.dispose()
|
||||
expect(backend.calls).toEqual(['emit:shutdown', 'emit:shutdown', 'shutdown'])
|
||||
expect(backend.shutdownResolved).toBe(true)
|
||||
const ops = backend.records.filter(r => r.channel === 'ops')
|
||||
expect(ops.map(r => r.attributes['session.id']).sort()).toEqual(['s1', 's2'])
|
||||
expect(ops.every(r => r.attributes['telemetry.op'] === 'shutdown' && r.severity === 'info')).toBe(true)
|
||||
expect(ops.every(r => !('event.seq' in r.attributes) && !('event.type' in r.attributes))).toBe(true)
|
||||
})
|
||||
|
||||
it('emits the shutdown marker at the session’s own disposal edge, then retires it', async () => {
|
||||
const { ctx, backend, fiber } = await setup()
|
||||
liveSession(ctx, 'survivor')
|
||||
// A session owned by its own fiber: disposing the fiber detaches it from
|
||||
// the store and emits `session/disposed` — the authoritative termination
|
||||
// edge. The marker must ride THAT edge (receivers classify a session with
|
||||
// activity and no marker as crashed, so a normally closed session in a
|
||||
// long-running host must not look like a crash), and the session retires
|
||||
// from the adopted set so unload neither retains it nor re-marks it.
|
||||
const owner = await ctx.plugin(Object.assign((inner: Context) => {
|
||||
inner.sessions.create(SessionId('ephemeral'), { meta: {} })
|
||||
}, { inject: ['sessions'] }))
|
||||
await owner.dispose()
|
||||
const atEdge = backend.records.filter(r => r.channel === 'ops')
|
||||
expect(atEdge.map(r => r.attributes['session.id'])).toEqual(['ephemeral'])
|
||||
expect(atEdge[0]!.attributes['telemetry.op']).toBe('shutdown')
|
||||
await fiber.dispose()
|
||||
const ops = backend.records.filter(r => r.channel === 'ops')
|
||||
expect(ops.map(r => r.attributes['session.id'])).toEqual(['ephemeral', 'survivor'])
|
||||
})
|
||||
|
||||
it('warns instead of throwing when backend shutdown fails', async () => {
|
||||
const backend = new FakeBackend()
|
||||
backend.shutdownError = new Error('exporter unreachable')
|
||||
const { ctx, fiber } = await setup(backend)
|
||||
const warn = vi.spyOn(ctx.logger, 'warn').mockImplementation(() => {})
|
||||
liveSession(ctx)
|
||||
await expect(fiber.dispose()).resolves.not.toThrow()
|
||||
expect(warn.mock.calls.some(args => String(args[0]).includes('shutdown failed'))).toBe(true)
|
||||
})
|
||||
|
||||
it('contains emit failures: the append succeeds and capture heals', async () => {
|
||||
const { ctx, backend } = await setup()
|
||||
const warn = vi.spyOn(ctx.logger, 'warn').mockImplementation(() => {})
|
||||
const session = liveSession(ctx)
|
||||
backend.emitError = new Error('backend broke')
|
||||
expect(() => session.append('turn/start', { turn: 1 })).not.toThrow()
|
||||
expect(warn).toHaveBeenCalled()
|
||||
backend.emitError = undefined
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
expect(backend.ledger().map(r => r.attributes['event.type'])).toEqual(['turn/end'])
|
||||
})
|
||||
|
||||
it.each([
|
||||
['Error values', new TypeError('adapter exploded'), 'TypeError', 'adapter exploded'],
|
||||
['non-Error values', 'plain failure', 'Error', 'plain failure'],
|
||||
])('relays agent/error %s as an ops record with normalized identity', async (_label, error, name, message) => {
|
||||
const { ctx, backend } = await setup()
|
||||
const session = liveSession(ctx, 'erring')
|
||||
// Only the members the relay reads; the full Agent surface is irrelevant here.
|
||||
const agent = { id: 'agent-1', session } as Agent
|
||||
ctx.emit('agent/error', { agent, turn: 3, step: 2, error })
|
||||
const record = backend.records.find(r => r.channel === 'ops')!
|
||||
expect(record.severity).toBe('error')
|
||||
expect(record.attributes).toMatchObject({
|
||||
'telemetry.op': 'agent-error',
|
||||
'session.id': 'erring',
|
||||
'agent.id': 'agent-1',
|
||||
'error.name': name,
|
||||
turn: 3,
|
||||
step: 2,
|
||||
})
|
||||
expect(record.body).toEqual({ name, message })
|
||||
})
|
||||
})
|
||||
27
packages/session/session-telemetry/tsconfig.json
Normal file
27
packages/session/session-telemetry/tsconfig.json
Normal file
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
{
|
||||
"path": "../../core/agent"
|
||||
},
|
||||
{
|
||||
"path": "../../support/invariants"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/session-title/session-title-all-messages-llm/README.md
|
||||
README.md: 25ec92432b5c2d18624f2f4d851552ee2b3bf5d0
|
||||
README.zh.md: 947aa614741b2b0bd50e70f1d273a625a5624cc6
|
||||
28
packages/session/session-title-all-messages-llm/README.md
Normal file
28
packages/session/session-title-all-messages-llm/README.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# @deepseek-ai/dsh-session-title-all-messages-llm
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
Optional `ctx.sessionTitle` provider that summarizes every eligible human message through `ctx.llm`. It registers the `all-user-messages` cadence and starts a new revision after each new human prompt, using seeded history as well as child-session prompts. A newer revision aborts and supersedes older work; even a provider that ignores cancellation cannot commit stale output.
|
||||
|
||||
The plugin uses the complete required [shared LLM configuration](../session-title-llm/README.md#configuration). Omit both `provider` and `model` to inherit the exact route from each current logged main request, or set both to route title generation independently. If the final framed aggregate prompt exceeds `maxInputBytes`, the request fails instead of truncating history; automatic use warns and keeps the prior title.
|
||||
|
||||
## Model Experience
|
||||
|
||||
### All-messages title request
|
||||
|
||||
#### What the model sees
|
||||
|
||||
The title model receives the shared title instruction and a JSON array of all eligible human messages through the current revision, in log order with exact seqs. Seeded history is included.
|
||||
|
||||
#### Token effect
|
||||
|
||||
One auxiliary request may follow every new eligible prompt, bounded per request by `maxInputBytes` and `maxOutputTokens`; explicit refreshes may add calls. The main agent request gains zero tokens.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
No main-request invalidation. Auxiliary input grows or changes after each prompt, so provider-specific cache reuse ends at the first changed JSON token.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- Input overflow retains the prior title; this provider has no summarization-of-summaries or retention policy for very long sessions.
|
||||
- It treats all eligible human messages equally and offers no weighting, filtering, or manual-title precedence.
|
||||
28
packages/session/session-title-all-messages-llm/README.zh.md
Normal file
28
packages/session/session-title-all-messages-llm/README.zh.md
Normal file
@@ -0,0 +1,28 @@
|
||||
# @deepseek-ai/dsh-session-title-all-messages-llm
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
可选的 `ctx.sessionTitle` 提供方，通过 `ctx.llm` 总结所有符合条件的用户消息。它注册 `all-user-messages` 节奏，并在每条新用户提示词后启动新 revision，同时使用预置历史与子会话提示词。较新的 revision 会中止并取代旧工作；即使提供方忽略取消，也无法提交陈旧输出。
|
||||
|
||||
该插件使用完整且必填的[共享大语言模型（LLM）配置](../session-title-llm/README.md#configuration)。同时省略 `provider` 与 `model` 时，会继承每个当前已记录主请求的确切路由；也可以同时设置二者，使标题生成使用独立路由。如果最终封装的聚合提示词超过 `maxInputBytes`，请求会失败而不是截断历史；自动使用时会发出警告并保留先前标题。
|
||||
|
||||
## 模型体验
|
||||
|
||||
### 全消息标题请求
|
||||
|
||||
#### 模型看到的内容
|
||||
|
||||
标题模型会收到共享标题指令，以及一个 JSON 数组，其中按日志顺序包含截至当前 revision 的所有符合条件用户消息和确切 seq。预置历史也包含在内。
|
||||
|
||||
#### Token 影响
|
||||
|
||||
每条符合条件的新提示词之后都可能发送一次辅助请求，每次请求受 `maxInputBytes` 和 `maxOutputTokens` 约束；显式刷新可能增加调用。主 agent（智能体）请求不会增加 token。
|
||||
|
||||
#### KV Cache 影响
|
||||
|
||||
不会使主请求的 KV Cache 失效。每条提示词后，辅助输入都会增长或变化，因此提供方专用缓存复用会在第一个变化的 JSON token 处结束。
|
||||
|
||||
## 已知限制与暂缓事项
|
||||
|
||||
- 输入溢出时保留先前标题；对于很长的会话，此提供方没有基于摘要继续生成摘要的机制或保留策略。
|
||||
- 它平等对待所有符合条件的用户消息，不提供权重、过滤或手动标题优先级。
|
||||
41
packages/session/session-title-all-messages-llm/package.json
Normal file
41
packages/session/session-title-all-messages-llm/package.json
Normal file
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-session-title-all-messages-llm",
|
||||
"description": "All-user-messages LLM provider plugin for DeepSeek Harness session titles",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./invariant": {
|
||||
"types": "./lib/types/invariant.d.ts",
|
||||
"default": "./lib/invariant.js"
|
||||
},
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": ["lib/index.js", "lib/invariant.js", "lib/types/**/*.d.ts"],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-invariants": "^0.0.1",
|
||||
"@deepseek-ai/dsh-llm": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session-title": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session-title-llm": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
},
|
||||
"dependencies": {
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-title": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-title-llm": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.7"
|
||||
}
|
||||
}
|
||||
36
packages/session/session-title-all-messages-llm/src/index.ts
Normal file
36
packages/session/session-title-all-messages-llm/src/index.ts
Normal file
@@ -0,0 +1,36 @@
|
||||
/** All-human-messages model provider for `ctx.sessionTitle`. */
|
||||
|
||||
import type { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import {
|
||||
registerSessionTitleLlmProvider,
|
||||
SessionTitleLlmConfigFields,
|
||||
} from '@deepseek-ai/dsh-session-title-llm'
|
||||
import type { SessionTitleLlmConfig } from '@deepseek-ai/dsh-session-title-llm'
|
||||
|
||||
export const name = 'session-title-all-messages-llm'
|
||||
export const inject = ['sessionTitle', 'llm', 'sessions']
|
||||
|
||||
/** Required LLM policy; this plugin adds no defaults. */
|
||||
export type Config = SessionTitleLlmConfig
|
||||
/** Loader schema shared with the first-message provider. */
|
||||
/* jscpd:ignore-start -- Loader requires each plugin to export its own statically walkable schema; the field validators remain shared. */
|
||||
export const Config: z<Config> = z.object({
|
||||
targetWords: SessionTitleLlmConfigFields.targetWords,
|
||||
targetCjkCharacters: SessionTitleLlmConfigFields.targetCjkCharacters,
|
||||
maxInputBytes: SessionTitleLlmConfigFields.maxInputBytes,
|
||||
maxOutputTokens: SessionTitleLlmConfigFields.maxOutputTokens,
|
||||
timeoutMs: SessionTitleLlmConfigFields.timeoutMs,
|
||||
provider: SessionTitleLlmConfigFields.provider,
|
||||
model: SessionTitleLlmConfigFields.model,
|
||||
})
|
||||
/* jscpd:ignore-end */
|
||||
|
||||
/**
|
||||
* Register the all-user-messages model provider.
|
||||
* @param ctx - context exposing session-title, LLM, and session services.
|
||||
* @param config - required route, target, byte, token, and timeout policy.
|
||||
*/
|
||||
export function apply(ctx: Context, config: Config): void {
|
||||
registerSessionTitleLlmProvider(ctx, config, name, 'all-user-messages', messages => messages)
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
/**
|
||||
* Package-owned invariant companion for `@deepseek-ai/dsh-session-title-all-messages-llm`.
|
||||
* @module @deepseek-ai/dsh-session-title-all-messages-llm/invariant
|
||||
*/
|
||||
|
||||
/* jscpd:ignore-start */
|
||||
import type { Context } from 'cordis'
|
||||
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
|
||||
|
||||
const PACKAGE_NAME = '@deepseek-ai/dsh-session-title-all-messages-llm'
|
||||
|
||||
/** Cordis companion plugin name. */
|
||||
export const name = 'session-title-all-messages-llm-invariant'
|
||||
/** Service required before the companion can reserve package ownership. */
|
||||
export const inject = ['invariants']
|
||||
|
||||
/**
|
||||
* No runtime invariant: this thin provider delegates request and result validation to the shared
|
||||
* title service and LLM helper and retains no independent mutable state.
|
||||
*/
|
||||
const install: InvariantInstaller = () => {}
|
||||
|
||||
/**
|
||||
* Register this package's invariant companion.
|
||||
* @param ctx - Cordis context carrying the invariant service.
|
||||
* @returns the installed registration's disposer after setup succeeds.
|
||||
*/
|
||||
export const apply = (ctx: Context): Promise<() => void> =>
|
||||
Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
|
||||
/* jscpd:ignore-end */
|
||||
@@ -0,0 +1,73 @@
|
||||
import { Context } from 'cordis'
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import LlmService, { createUserMessage, LlmAdapter } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import SessionStore, { Session, SessionId } from '@deepseek-ai/dsh-session'
|
||||
import SessionTitleService from '@deepseek-ai/dsh-session-title'
|
||||
import * as providerPlugin from '@deepseek-ai/dsh-session-title-all-messages-llm'
|
||||
|
||||
class RecordingAdapter extends LlmAdapter {
|
||||
readonly requests: GenerateOptions[] = []
|
||||
|
||||
override async * stream(options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
this.requests.push(options)
|
||||
yield { type: 'text-delta', index: 0, text: 'All messages model title' }
|
||||
yield { type: 'finish', reason: { kind: 'stop' } }
|
||||
}
|
||||
}
|
||||
|
||||
const TITLE_CONFIG = { fallbackMaxWords: 5, fallbackMaxBytes: 40, maxTitleBytes: 80 } as const
|
||||
const LLM_CONFIG = {
|
||||
targetWords: 5,
|
||||
targetCjkCharacters: 10,
|
||||
maxInputBytes: 1_000,
|
||||
maxOutputTokens: 32,
|
||||
timeoutMs: 1_000,
|
||||
} as const
|
||||
|
||||
async function settle(): Promise<void> {
|
||||
await new Promise(resolve => setTimeout(resolve, 0))
|
||||
}
|
||||
|
||||
describe('all-messages LLM title provider', () => {
|
||||
it('includes seeded history and the latest prompt while inheriting the logged request route', async () => {
|
||||
const seeded = Session.create(SessionId('seed-source'))
|
||||
seeded.append('turn/start', { turn: 1 })
|
||||
const inherited = seeded.append('user/message', createUserMessage({
|
||||
content: [{ type: 'text', text: 'inherited prompt' }], source: { kind: 'user' },
|
||||
}), { surfaceOp: 'append' })
|
||||
seeded.append('session/title', {
|
||||
title: 'Inherited fallback', messageSeqs: [inherited.seq], source: { kind: 'fallback' },
|
||||
})
|
||||
seeded.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SessionTitleService, TITLE_CONFIG)
|
||||
const adapter = new RecordingAdapter()
|
||||
ctx.llm.registerAdapter(['current-route'], adapter)
|
||||
await ctx.plugin(providerPlugin, LLM_CONFIG)
|
||||
const session = ctx.sessions.create(SessionId('all-plugin'), {
|
||||
seed: seeded.events,
|
||||
meta: { parentSession: seeded.id, seedLength: seeded.seq },
|
||||
})
|
||||
session.append('turn/start', { turn: 2 })
|
||||
const latest = session.append('user/message', createUserMessage({
|
||||
content: [{ type: 'text', text: 'latest prompt' }], source: { kind: 'user' },
|
||||
}), { surfaceOp: 'append' })
|
||||
await settle()
|
||||
session.append('request/header', {
|
||||
header: { config: { provider: 'current-route', model: 'current-model' } }, reason: 'resume',
|
||||
})
|
||||
await settle()
|
||||
|
||||
expect(adapter.requests[0]).toMatchObject({ provider: 'current-route', model: 'current-model' })
|
||||
const content = adapter.requests[0]?.messages[0]?.content[0]
|
||||
expect(content?.type === 'text' && content.text).toContain('inherited prompt')
|
||||
expect(content?.type === 'text' && content.text).toContain('latest prompt')
|
||||
expect(ctx.sessionTitle.get(session)).toMatchObject({
|
||||
messageSeqs: [inherited.seq, latest.seq],
|
||||
})
|
||||
})
|
||||
})
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user