refactor(session): fold the session family into packages/session/

git mv the 12 packages from session-persistence/, session-projection/,
session-title/, and telemetry/ into one session/ group per the
regrouping RFC; merge the four group READMEs into one bilingual
triplet; rewrite the group segment in tsconfig references (intra-group
references shorten to ../<pkg>), tsconfig.base.json paths/globs,
knip.json keys, vitest include, gate scripts, and authored doc/note
citations; regenerate module graph, doc graphs, catalogs, and the
lockfile importer keys. No npm names change.

Full unit suite: 8779 passed; the 18 reported failures reproduce as
env flakes (ambient-proxy IPv6 tunneling, watched-dir inotify
timeouts under parallel load) — each passes in isolation with
NO_PROXY set, matching their known pre-existing behavior on master.
This commit is contained in:
Tianyi Cui
2026-07-30 01:52:06 +08:00
parent 645fcf5713
commit 7e445c3a67
220 changed files with 258 additions and 286 deletions

View File

@@ -0,0 +1,6 @@
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write packages/session-persistence/session-persistence-jsonl/README.md
README.md: b7fa8fc2918711dd24eeba67132a267452d401f9
README.zh.md: 59d21c6171e857849c4ab48940d66962fffbfdeb

View File

@@ -0,0 +1,77 @@
# @deepseek-ai/dsh-session-persistence-jsonl
English | [中文](README.zh.md)
The JSONL durable session-persistence backend — a concrete `SessionPersistence` (the `dsh-session-persistence` seam). Each session has one append-only logical JSONL log, stored as `.jsonl.zstd` by default or raw `.jsonl` when compression is disabled.
## On-disk layout
```
<root>/
--<normalized-cwd>--/ # readable project directory (or _no-cwd/)
<encoded-id>/ # session-owned directory
session.jsonl.zstd # default: checksummed header frame + append frames
session.jsonl # only with compression: 'none'
```
- The first logical line is the immutable `SessionHeader` tagged `{ type: 'session', version, id, cwd?, createdAt, parentSession?, seedLength?, origin?, delegationDepth }`. `delegationDepth` is required on disk and is `0` for a top-level session; a missing or invalid value rejects the log. Every subsequent logical line is one storage record; `assistant/chunk` events are never dropped, and `seq` stays contiguous across the decoded log (`events[i].seq === i`).
- A storage record is a `SessionEvent` JSON verbatim, or — for an eligible run when `packChunks` is enabled — a **packed chunk row** (`text-chunks` / `reasoning-chunks` / `tool-call-chunks`; bare slash-less tags like the header's `session`, so row tags cannot be confused with event types): one line holding a run of ≥3 consecutive same-block `assistant/chunk` delta events, `seq0`/`time0` plus per-member `dt` gaps reconstructing every member's `seq`/`time` exactly. The lossless codec lives in `@deepseek-ai/dsh-session` (`packChunkRuns`/`decodeStorageRecord`) and whitelists exact shapes — anything unrecognized stores verbatim. Reading is layout-blind: `load` always decodes rows, so packed, unpacked, and mixed files load identically.
- The project directory keeps the normalized cwd readable for navigation and is bounded for filesystem component limits. Separator replacement and truncation are intentionally lossy, so cwd strings that normalize alike share a project directory; session ids still select distinct session directories. On a case-insensitive filesystem, identity validation accepts an alternate path spelling only when filesystem canonicalization resolves both spellings to the same transcript. The configured root remains deployment-controlled: it may be project-local, shared, temporary, or centralized. The [project-session directory decision](../../../.agents/notes/implemented/architecture/2026-07-24-project-session-directories.md) records this tradeoff.
- Session ids are unvalidated branded strings, so they are injectively escaped to a single safe path segment before use (no traversal, no collision). The resulting directory is reserved for additional session-owned artifacts; discovery reads only the fixed transcript filename.
## Config
| Key | Type | Notes |
|---|---|---|
| `root` | `string` (required) | Root directory for all session files. **No default** — a `process.cwd()` default would scatter files as the process's cwd changes (bash calls, subprocesses). An existing root must be a readable directory; an absent root is created on first materialization. |
| `packChunks` | `boolean` (default `true`) | Write eligible delta-chunk runs as packed rows (~60% smaller logical logs measured on a real coding session). Set `false` for one-event-per-line diagnostics; reading packed rows works regardless of this write-side switch. |
| `compression` | `'zstd' \| 'none'` | Defaults to `'zstd'`; `'none'` retains newline-delimited UTF-8 text. |
| `preparedSessionCacheSize` | positive integer (default `5`) | Maximum unpublished Sessions retained after cold history inspection for reuse by resume. |
| `writeBatchMaxDelayMs` | positive integer (default `200`) | Fixed coalescing window after an idle live-event queue receives work. Later events do not reset it; flush and teardown bypass it. It does not bound event-loop, serialized-operation, or backend latency. At most Node's `2_147_483_647` ms timer limit. |
`locate(meta)` returns `{ kind: 'jsonl', path }` for the fixed transcript inside the resolved project/session directories. It performs no filesystem I/O: the target can be returned before the directory or file exists, and an existing file contains only the last flushed prefix.
## Physical encoding
The default artifact is a standard concatenation of independent [Zstandard frames](../../../.agents/notes/implemented/architecture/2026-07-19-zstandard-jsonl-session-logs.md): one checksummed frame containing only the header line, followed by one checksummed frame per durable append batch. The backend uses Node's built-in Zstandard API with its default compression level and exposes no level knob. Listing reads and validates only the header frame. `compression: 'none'` keeps the same logical lines in the original raw representation.
A root belongs to one encoding. Startup discovery and targeted lookup reject the opposite suffix with an error naming the incompatible artifact and instructing the caller to select the matching mode or a separate root. Flat `<project>/<id>.jsonl*` artifacts are also rejected instead of ignored. There is no migration, mixed-root fallback, or dual write.
## Durability and crash semantics
- **Bound storage identity.** Lookup requires one matching session directory across the readable project directories, then verifies that the header id equals the requested id and that the header's id/cwd derive the selected transcript path. Listing applies the same path check and rejects duplicate ids. Identity failures occur before repair or append.
- **Lazy materialization.** `create(meta)` writes nothing; on the first `append`, the backend writes and `fsync`s the encoded header and first batch in a temporary file. POSIX publishes it without overwrite via a hard link and `fsync`s the parent directory. Windows publishes it without overwrite via `MoveFileExW(..., MOVEFILE_WRITE_THROUGH)` and creates missing directories through the same write-through pattern. A created-but-never-appended session leaves nothing on disk and is absent from `list`.
- **Append-only.** Flushed events are never rewritten. Subsequent raw batches append lines; compressed batches append one frame. Both paths `fsync`, and a caught write or sync failure rolls the file back to its prior byte length.
- **Crash recovery — preserve valid tail work.** `load` validates every complete compressed frame and scans their decompressed JSONL. If the last frame is structurally incomplete, the reader keeps its complete decoded records, truncates from that frame's start, and re-encodes those records with the synthetic tool, step, and turn closers required by the shared [persistence contract](../../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md). Raw mode truncates from its first incomplete line. A checksum/decompression failure in a complete frame, or a defect at or before the last committed `turn/end`, is corruption and rejects.
- **Non-mutating inspection.** `inspect()` returns an immutable balanced logical view and may synthesize recovery closers in memory, without truncating an incomplete tail or changing the lightweight revision.
- **Contiguous-seq.** `append` rejects a batch whose first `seq` does not continue the stored log, and rejects non-JSON-serializable `event.data` naming the offending event type.
- **Lightweight revisions.** `listSnapshots(signal?)` identifies a log by its device, inode, size, and nanosecond timestamps, avoiding a full-log parse while changing after append, repair, replacement, or store changes. A full-prefix read requires the same identity before and after reading the bytes, and `readStoredRevision()` uses that identity to validate retained preparations without loading the log. Snapshot listing forwards the exact signal through artifact discovery and checks cancellation around every `stat`; because filesystem `stat` is not interruptible, cancellation waits for the active call to settle, then rejects without starting another.
## Write path
The plugin copies frozen session events into one controller per live session. The first pending event starts the configured fixed batching window, and later events join without resetting it. Expiry starts one durable append; events admitted during that write form a separately bounded follow-up batch. `session/flush` cancels the wait and drains current and pending batches. A per-session cursor prevents resumed sessions from re-appending stored events, and live sessions are seeded when the plugin loads. The owning backend instance serializes operations for one session; disposal drains every retained controller before teardown. Every logical event remains present: batching only lets one compressed frame or raw fsync carry more records.
## Model Experience
### Resumed conversation history
#### What the model sees
JSONL storage contributes no live prompt or schema. Loading restores stored surface history and preserves prior request headers for reconstruction; the new loop composes its current envelope. Recovery balances an assistant request without a durable call with `TOOL_NOT_STARTED`; a durable call without a result becomes `TOOL_OUTCOME_UNKNOWN`, which tells the model to retry only read-only or idempotent work and to verify possible side effects or ask the user. Raw `assistant/chunk` records do not duplicate messages.
#### Token effect
Zero live-request tokens. A resumed agent pays for retained history and its current envelope, plus the quoted repair result for each interrupted call.
#### KV Cache effect
JSONL storage does not mutate live request prefixes. A resumed loop can reuse provider cache only when its reconstructed history, current envelope, and model route match; crash-repair results append.
## Known Limitations and Deferred Work
- **Only the configured encoding and current `SESSION_FORMAT_VERSION` (v0) load** — changing compression requires a separate/fresh root or selecting the legacy raw mode; the pre-release format has no migration.
- **The flat-file storage layout does not load** — use a separate root or move pre-release artifacts into the project/session directory layout before loading.
- **Compressed files are not directly line-readable** — use the backend to load them, or select `compression: 'none'` before writing a fresh root when external line readers are required.
- **Nothing deletes session files** — logs accumulate under `root` until removed externally (the seam has no deletion surface).
- **One live writer per session** — append and repair are coordinated only inside the owning backend instance. Another backend instance or process must not write the same session until that owner reaches quiescent disposal; initial same-id publication remains collision-safe through the POSIX no-overwrite hard link or Windows write-through rename without replacement.
- **POSIX materialization requires hard-link support** — first append uses `link()` so same-id races fail instead of overwriting a committed log; Windows uses write-through rename without replacement.

View File

@@ -0,0 +1,77 @@
# @deepseek-ai/dsh-session-persistence-jsonl
[English](README.md) | 中文
JSONL 持久会话存储后端:`SessionPersistence` 的一个具体实现(`dsh-session-persistence` seam。每个会话有一个仅追加的逻辑 JSONL 日志,默认存储为 `.jsonl.zstd`;禁用压缩时使用原始 `.jsonl`
## 磁盘布局
```
<root>/
--<normalized-cwd>--/ # readable project directory (or _no-cwd/)
<encoded-id>/ # session-owned directory
session.jsonl.zstd # default: checksummed header frame + append frames
session.jsonl # only with compression: 'none'
```
- 第一个逻辑行是不可变的 `SessionHeader`,标记为 `{ type: 'session', version, id, cwd?, createdAt, parentSession?, seedLength?, origin?, delegationDepth }``delegationDepth` 在磁盘上必需,顶层会话为 `0`;缺失或无效值会拒绝日志。后续每个逻辑行是一条存储记录;`assistant/chunk` 事件绝不丢弃,且 `seq` 在解码日志中保持连续(`events[i].seq === i`)。
- 存储记录是原样 `SessionEvent` JSON或在 `packChunks` 已启用且连续段符合条件时写入的**打包分片行**`text-chunks` / `reasoning-chunks` / `tool-call-chunks`;像 header 的 `session` 一样不带斜杠,因此行 tag 不会与事件类型混淆):一行保存至少 3 个连续同 block `assistant/chunk` delta 事件,`seq0`/`time0` 和每成员 `dt` 间隔精确重建每个成员的 `seq`/`time`。无损 codec 位于 `@deepseek-ai/dsh-session``packChunkRuns`/`decodeStorageRecord`),并使用精确形态 allowlist任何未识别内容原样存储。读取与布局无关`load` 始终解码行,因此打包、非打包和混合文件加载结果一致。
- 项目目录保留规范化 cwd 可读,并限制在文件系统组件上限内。分隔符替换和截断刻意有损,因此规范化相同的 cwd 字符串共享项目目录;会话 id 仍选择不同会话目录。在不区分大小写的文件系统上,只有文件系统规范化将两种写法解析到同一 transcript文本记录身份验证才接受备选路径写法。配置根仍由部署控制可以是项目本地、共享、临时或集中式。[项目会话目录决策](../../../.agents/notes/implemented/architecture/2026-07-24-project-session-directories.md) 记录这项取舍。
- 会话 id 是未验证的带品牌类型的字符串,因此在使用前单射转义为一个安全路径段(无遍历、无冲突)。结果目录保留给其他会话自有产物;发现只读取固定 transcript 文件名。
## 配置
| 键 | 类型 | 说明 |
|---|---|---|
| `root` | `string`(必需) | 所有会话文件的根目录。**无默认值**`process.cwd()` 默认值会随进程 cwd 变更bash 调用、子进程)而分散文件。现有根必须是可读目录;缺失根在第一次实体化时创建。 |
| `packChunks` | `boolean`(默认 `true` | 将符合条件的 delta 分片连续段写为打包行(在真实编码会话上测得逻辑日志约小 60%)。设为 `false` 可用于每事件一行诊断;无论该写入侧开关如何,都能读取打包行。 |
| `compression` | `'zstd' \| 'none'` | 默认 `'zstd'``'none'` 保留换行分隔 UTF-8 文本。 |
| `preparedSessionCacheSize` | 正整数(默认 `5` | 冷历史检查后保留、供恢复复用的未发布 Session 数量上限。 |
| `writeBatchMaxDelayMs` | 正整数(默认 `200` | 空闲的活动事件队列收到待写入事件后开启的固定合并窗口。后续事件不会重置窗口flush 与 teardown 会绕过它。该值不限制事件循环、串行化操作或后端延迟。最大值为 Node 计时器上限 `2_147_483_647` ms。 |
`locate(meta)` 返回已解析项目/会话目录内固定 transcript 的 `{ kind: 'jsonl', path }`。它不执行文件系统 I/O可以在目录或文件存在前返回目标现有文件也只包含最近一次 flush 完成的前缀。
## 物理编码
默认产物是独立 [Zstandard frame](../../../.agents/notes/implemented/architecture/2026-07-19-zstandard-jsonl-session-logs.md) 的标准拼接:一个仅包含 header 行的带 checksum frame后跟每个持久 append 批次一个带 checksum frame。后端使用 Node 内置 Zstandard API 和默认压缩级别,不提供级别开关。列表只读取并验证 header frame。`compression: 'none'` 在原始表示中保留相同逻辑行。
一个根只属于一种编码。启动发现和定向查找会拒绝相反 suffix错误会命名不兼容产物并指示调用方选择匹配 mode 或独立根。平铺 `<project>/<id>.jsonl*` 产物也会被拒绝,而不是忽略。不提供迁移、混合根回退或双写。
## 持久性与崩溃语义
- **绑定存储身份。** 查找要求可读项目目录中只有一个匹配会话目录,然后验证 header id 等于请求 id且 header id/cwd 派生所选 transcript 路径。列表应用同一路径检查,并拒绝重复 id。身份失败发生在修复或 append 前。
- **延迟实体化。**`create(meta)` 不写入;第一次 `append` 将编码 header 和第一批写入临时文件并执行 `fsync`。POSIX 通过硬链接无覆盖发布,并对父目录 `fsync`。Windows 通过 `MoveFileExW(..., MOVEFILE_WRITE_THROUGH)` 无覆盖发布,并通过同一 write-through pattern 创建缺失目录。已创建但从未 append 的会话不留下磁盘内容,不在 `list` 中。
- **仅追加。** 已 flush 事件绝不重写。后续原始批次 append 行;压缩批次 append 一个 frame。两条路径都执行 `fsync`,并在捕获到写入或同步失败时回滚到之前字节长度。
- **崩溃恢复:保留有效尾部工作。**`load` 验证每个完整压缩 frame并扫描解压 JSONL。最后 frame 结构不完整时,读取器保留其完整解码记录,从 frame 开头截断,并使用共享[持久化契约](../../../.agents/notes/implemented/architecture/2026-06-14-session-persistence.md) 需要的合成工具、步骤和轮次 closer 重新编码这些记录。原始 mode 从第一个不完整行截断。完整 frame 中的 checksum/解压失败,或位于最后已提交的 `turn/end` 处或之前的缺陷属于损坏,会被拒绝。
- **非变更检查。**`inspect()` 返回不可变、平衡的逻辑视图,并可在内存中合成恢复 closer但不会截断不完整尾部或更改轻量修订。
- **连续 seq。**`append` 拒绝第一个 `seq` 不继续已存储日志的批次,并拒绝非 JSON 可序列化 `event.data`,同时命名违规事件类型。
- **轻量修订。**`listSnapshots(signal?)` 使用 device、inode、size 和纳秒时间戳标识日志,避免解析完整日志;该标识会在 append、修复、替换或存储变更后改变。完整前缀读取要求读取字节前后的身份一致`readStoredRevision()` 使用同一身份校验保留的 preparation而不加载日志。快照列表通过产物发现转发精确信号并在每个 `stat` 前后检查取消;由于文件系统 `stat` 不可中断,取消会等待活动调用完成,然后在不启动另一次调用的情况下拒绝。
## 写入路径
插件将冻结的会话事件复制到每个活动会话各自的 controller。第一个待处理事件会开启配置的固定批处理窗口后续事件会加入但不会重置截止时间。窗口到期后会启动一次持久化追加该次写入期间接纳的事件会形成另一个独立有界的后续批次。`session/flush` 会取消等待并排空当前与待处理批次。每会话游标防止恢复后的会话重新 append 已存储事件插件加载时会为活动会话设置初始状态。所属后端实例串行化单会话操作dispose资源释放会在拆卸前排空每个保留的 controller。每个逻辑事件都会保留批处理只让单个压缩帧或一次原始 JSONL fsync 承载更多记录。
## 模型体验
### 恢复的对话历史
#### 模型看到的内容
JSONL 存储不影响当前提示词或 schema。加载会恢复已存储的表层历史并保留之前的请求 header 用于重建;新 loop 组合当前 envelope。恢复会用 `TOOL_NOT_STARTED` 平衡没有已持久化调用的 assistant 请求;已有已持久化调用但无结果时则变为 `TOOL_OUTCOME_UNKNOWN`,它要求模型只重试只读或幂等工作,并验证可能的副作用或询问用户。原始 `assistant/chunk` 记录不会重复生成消息。
#### Token 影响
当前请求不会新增 token。恢复后的 agent智能体会因保留的历史、当前 envelope以及每个中断调用中以引用形式加入的修复结果文本而消耗 token。
#### KV Cache 影响
JSONL 存储不修改实时请求前缀。只有重建历史、当前 envelope 和模型路由匹配时,恢复 loop 才能重用提供方缓存;崩溃修复结果仅追加。
## 已知限制与暂缓事项
- **只加载已配置编码和当前 `SESSION_FORMAT_VERSION` (v0)**:更改压缩需要独立/全新根,或选择遗留原始 mode预发布格式没有迁移。
- **平铺文件存储布局不加载**:加载前使用独立根,或将预发布产物移入项目/会话目录布局。
- **压缩文件不能直接按行读取**:使用后端加载;或在写入新根前选择 `compression: 'none'`,以便外部行 reader 使用。
- **不删除会话文件**:日志在 `root` 下累积直到外部移除seam 无删除接口)。
- **每会话一个实时 writer**append 和修复只在所属后端实例内协调。在所有者完成完全停稳的 dispose 前,其他后端实例或进程不得写入同一会话;初始同 id 发布仍通过 POSIX 无覆盖硬链接或 Windows 无替换 write-through rename 保持冲突安全。
- **POSIX 实体化需要硬链接支持**:第一次 append 使用 `link()`,使同 id 竞态失败而不覆盖已提交日志Windows 使用无替换 write-through rename。

View File

@@ -0,0 +1,43 @@
{
"name": "@deepseek-ai/dsh-session-persistence-jsonl",
"description": "JSONL durable session persistence backend for the DeepSeek Harness",
"version": "0.0.1",
"private": true,
"type": "module",
"main": "lib/index.js",
"types": "lib/types/index.d.ts",
"exports": {
".": {
"types": "./lib/types/index.d.ts",
"default": "./lib/index.js"
},
"./invariant": {
"types": "./lib/types/invariant.d.ts",
"default": "./lib/invariant.js"
},
"./src/*": "./src/*",
"./package.json": "./package.json"
},
"files": [
"lib/index.js",
"lib/invariant.js",
"lib/types/**/*.d.ts"
],
"license": "BSD-3-Clause",
"peerDependencies": {
"@deepseek-ai/dsh-invariants": "^0.0.1",
"@deepseek-ai/dsh-session": "^0.0.1",
"@deepseek-ai/dsh-session-persistence": "^0.0.1",
"cordis": "^4.0.0-rc.7"
},
"dependencies": {
"koffi": "^3.1.0",
"schemastery": "^3.18.0"
},
"devDependencies": {
"@deepseek-ai/dsh-invariants": "workspace:^",
"@deepseek-ai/dsh-session": "workspace:^",
"@deepseek-ai/dsh-session-persistence": "workspace:^",
"cordis": "^4.0.0-rc.7"
}
}

View File

@@ -0,0 +1,390 @@
/**
* On-disk format helpers for the JSONL session-persistence backend: path
* sanitization (a {@link SessionId} is an unvalidated branded string, so it
* MUST be encoded before use in a path — no traversal, no collision), the
* per-project/session directory layout, header-line (de)serialization, and the
* truncation-repair offset computation.
*
* @module dsh-session-persistence-jsonl/format
*/
import { join } from 'node:path'
import { decodeStorageRecord, packChunkRuns } from '@deepseek-ai/dsh-session'
import type { SessionEvent, SessionHeader, SessionId, StorageRecord } from '@deepseek-ai/dsh-session'
/** Physical encoding selected for JSONL session artifacts. */
export type JsonlCompression = 'zstd' | 'none'
/**
* Return the artifact suffix for one physical encoding.
* @param compression - configured JSONL artifact encoding.
* @returns `.jsonl.zstd` for Zstandard or `.jsonl` for plaintext.
*/
export function logSuffix(compression: JsonlCompression): '.jsonl.zstd' | '.jsonl' {
return compression === 'zstd' ? '.jsonl.zstd' : '.jsonl'
}
/**
* The first JSONL record of a session artifact: the immutable
* {@link SessionHeader} tagged as a `session` record so a reader can tell it
* apart from an event line.
*/
export interface HeaderLine {
type: 'session'
version: number
id: SessionId
createdAt: number
cwd?: string
parentSession?: SessionId
seedLength?: number
origin?: 'subagent'
delegationDepth: number
}
/**
* Build the header line object from a {@link SessionHeader}.
* @param header - the immutable session metadata to serialize.
* @returns the `type: 'session'`-tagged line object, absent optional fields omitted (never null).
*/
export function toHeaderLine(header: SessionHeader): HeaderLine {
return {
type: 'session',
version: header.version,
id: header.id,
createdAt: header.createdAt,
...header.cwd !== undefined ? { cwd: header.cwd } : {},
...header.parentSession !== undefined ? { parentSession: header.parentSession } : {},
...header.seedLength !== undefined ? { seedLength: header.seedLength } : {},
...header.origin !== undefined ? { origin: header.origin } : {},
delegationDepth: header.delegationDepth ?? 0,
}
}
/**
* Parse a header line back into a {@link SessionHeader}.
* @param line - the shape-checked first line of a log (see the `isHeaderLine` guard).
* @returns the header, absent optional fields omitted.
*/
export function fromHeaderLine(line: HeaderLine): SessionHeader {
if (Object.hasOwn(line, 'sandboxMode') || Object.hasOwn(line, 'approvalPolicy')) {
throw new Error('session header uses retired policy baseline fields')
}
return {
version: line.version,
id: line.id,
createdAt: line.createdAt,
...line.cwd !== undefined ? { cwd: line.cwd } : {},
...line.parentSession !== undefined ? { parentSession: line.parentSession } : {},
...line.seedLength !== undefined ? { seedLength: line.seedLength } : {},
...line.origin !== undefined ? { origin: line.origin } : {},
delegationDepth: line.delegationDepth,
}
}
/** Type guard: a parsed first line is a well-formed session header. */
function isHeaderLine(value: unknown): value is HeaderLine {
return (
typeof value === 'object' && value !== null
&& (value as { type?: unknown }).type === 'session'
&& typeof (value as { version?: unknown }).version === 'number'
&& typeof (value as { id?: unknown }).id === 'string'
&& typeof (value as { createdAt?: unknown }).createdAt === 'number'
&& Number.isSafeInteger((value as { createdAt: number }).createdAt)
&& (value as { createdAt: number }).createdAt >= 0
&& !Object.is((value as { createdAt: number }).createdAt, -0)
&& typeof (value as { delegationDepth?: unknown }).delegationDepth === 'number'
&& Number.isSafeInteger((value as { delegationDepth: number }).delegationDepth)
&& (value as { delegationDepth: number }).delegationDepth >= 0
&& !Object.is((value as { delegationDepth: number }).delegationDepth, -0)
&& ((value as { origin?: unknown }).origin === undefined
|| (value as { origin?: unknown }).origin === 'subagent')
)
}
/**
* Encode an arbitrary string as a single safe path segment, injectively over ALL JS (UTF-16)
* strings — including lone surrogates. A {@link SessionId} is an unvalidated branded string,
* so this neutralizes `../`, absolute paths, NUL, and separators before any filesystem use.
* Safe code units remain literal; every other unit, including `~`, becomes
* `~XXXX`. Operating on code units preserves lone surrogates, while special-
* casing `.` and `..` prevents traversal by an otherwise safe whole segment.
*
* @param raw - the string to encode; must be non-empty (throws on `''`).
* @returns the escaped single path segment, decodable back to `raw`.
*/
export function encodeSegment(raw: string): string {
if (raw.length === 0) throw new Error('cannot encode an empty path segment')
if (raw === '.') return '~002E'
if (raw === '..') return '~002E~002E'
let out = ''
for (let i = 0; i < raw.length; i++) {
const code = raw.charCodeAt(i)
const ch = String.fromCharCode(code)
if (ch !== '~' && /^[A-Za-z0-9._-]$/.test(ch)) {
out += ch
} else {
out += '~' + code.toString(16).toUpperCase().padStart(4, '0')
}
}
return out
}
/**
* Build the readable directory key for a project path.
* Filesystem separators and drive separators become `-`; unsafe code units use
* the same `~XXXX` escape as session ids. The key is bounded for filesystem
* component limits. Separator replacement and truncation are intentionally
* lossy, following the common human-navigable project-directory convention.
* @param cwd - the session's project directory.
* @returns a single filesystem-safe project directory name.
*/
export function projectKey(cwd: string): string {
if (cwd.length === 0) throw new Error('cannot encode an empty project path')
let readable = ''
let separatorRun = false
for (let i = 0; i < cwd.length; i++) {
const code = cwd.charCodeAt(i)
const ch = String.fromCharCode(code)
if (ch === '/' || ch === '\\' || ch === ':') {
if (!separatorRun) readable += '-'
separatorRun = true
} else if (ch !== '~' && /^[A-Za-z0-9._-]$/.test(ch)) {
readable += ch
separatorRun = false
} else {
readable += '~' + code.toString(16).toUpperCase().padStart(4, '0')
separatorRun = false
}
}
const slug = readable.replace(/^-+/, '') || 'root'
return `--${slug.slice(0, 251)}--`
}
/**
* The configured root's human-navigable project directory. A configured root
* may be local or shared; this grouping does not prescribe its deployment.
* @param root - the backend's session root directory.
* @param cwd - the session's project directory; `undefined` selects `_no-cwd`.
* @returns the project directory path under `root`.
*/
export function projectDir(root: string, cwd: string | undefined): string {
if (cwd === undefined) return join(root, '_no-cwd')
return join(root, projectKey(cwd))
}
/**
* The directory owned by one session and available for future session-local
* artifacts.
* @param root - the backend's session root directory.
* @param cwd - the session's project directory.
* @param id - the session id, encoded to one safe path segment.
* @returns the session directory beneath its project directory.
*/
export function sessionDir(root: string, cwd: string | undefined, id: SessionId): string {
return join(projectDir(root, cwd), encodeSegment(id))
}
/**
* The append-only event-log file path for a session.
* @param root - the backend's session root directory.
* @param cwd - the session's project directory (`undefined` → `_no-cwd`).
* @param id - the session id, path-encoded via {@link encodeSegment} before filesystem use.
* @param compression - physical artifact encoding and filename suffix.
* @returns the session's configured JSONL artifact path.
*/
export function logPath(
root: string,
cwd: string | undefined,
id: SessionId,
compression: JsonlCompression,
): string {
return join(sessionDir(root, cwd, id), `session${logSuffix(compression)}`)
}
/**
* Serialize an event batch as JSONL lines (no trailing newline). With
* `packChunks` on, delta-chunk runs pack into `text-chunks` /
* `reasoning-chunks` / `tool-call-chunks` storage rows; off writes one event
* per line, byte-identical to the pre-packing layout. Reading is layout-blind
* either way ({@link scanLog} always decodes rows), so the switch only shapes
* NEW bytes.
* @param events - the batch to serialize, in log order.
* @param packChunks - whether to pack delta runs into storage rows.
* @returns the batch's JSONL text; the writer adds the final newline.
*/
export function eventLines(events: readonly SessionEvent[], packChunks: boolean): string {
const records: readonly StorageRecord[] = packChunks ? packChunkRuns(events) : events
return records.map(record => JSON.stringify(record)).join('\n')
}
interface SessionLogScan {
meta: SessionHeader
events: SessionEvent[]
committedBytes: number
}
/** Parse one complete header record supplied independently from event rows. */
function parseHeaderRecord(record: Buffer): SessionHeader {
if (record.length === 0 || record.at(-1) !== 0x0A || record.indexOf(0x0A) !== record.length - 1) {
throw new Error('empty or header-less session log')
}
let parsed: unknown
try {
parsed = JSON.parse(record.subarray(0, -1).toString('utf8'))
} catch {
throw new Error('corrupt session log: header line is not valid JSON')
}
if (!isHeaderLine(parsed)) {
throw new Error('corrupt session log: first line is not a session header')
}
return fromHeaderLine(parsed)
}
/**
* Incrementally scan complete JSONL event records after an independently
* supplied header record. Newline search and byte offsets stay on raw buffers;
* only complete records are decoded to UTF-8. A fragment crossing writes is
* copied because a decoder may reuse its output buffer after `write()` returns.
*/
export class SessionLogScanner {
private readonly meta: SessionHeader
private readonly events: SessionEvent[] = []
private fragments: Buffer[] = []
private fragmentBytes = 0
private inputBytes: number
private committedBytes: number
private eventLine = 0
private issue: Error | undefined
private finished = false
/**
* Create an event scanner from exactly one newline-terminated header record.
* @param headerRecord - the complete first JSONL record, including its newline.
*/
constructor(headerRecord: Buffer) {
this.meta = parseHeaderRecord(headerRecord)
this.inputBytes = headerRecord.length
this.committedBytes = headerRecord.length
}
/**
* Consume the next raw plaintext chunk, retaining only an incomplete final record.
* @param chunk - bytes immediately following all previously supplied bytes.
*/
write(chunk: Buffer): void {
if (this.finished) throw new Error('cannot write to a finished session log scanner')
const chunkStart = this.inputBytes
this.inputBytes += chunk.length
let lineStart = 0
for (
let newline = chunk.indexOf(0x0A);
newline !== -1;
newline = chunk.indexOf(0x0A, lineStart)
) {
const fragment = chunk.subarray(lineStart, newline)
let line = fragment
if (this.fragments.length > 0) {
if (fragment.length > 0) this.fragments.push(fragment)
line = Buffer.concat(this.fragments, this.fragmentBytes + fragment.length)
this.fragments = []
this.fragmentBytes = 0
}
this.consumeEventLine(line, chunkStart + newline + 1)
lineStart = newline + 1
}
if (lineStart < chunk.length) {
const fragment = Buffer.from(chunk.subarray(lineStart))
this.fragments.push(fragment)
this.fragmentBytes += fragment.length
}
}
/**
* Snapshot progress before appending a recoverable torn-frame prefix.
* @returns byte, committed-prefix, and expanded-event cursors.
*/
checkpoint(): { inputBytes: number; committedBytes: number; eventCount: number } {
return {
inputBytes: this.inputBytes,
committedBytes: this.committedBytes,
eventCount: this.events.length,
}
}
/**
* Finish scanning, ignoring a final record without a newline as a torn tail.
* @returns the header, contiguous event prefix, and safe truncation offset.
*/
finish(): SessionLogScan {
this.finished = true
return { meta: this.meta, events: this.events, committedBytes: this.committedBytes }
}
/** Decode one complete event row and update the contiguous prefix. */
private consumeEventLine(line: Buffer, endByte: number): void {
this.eventLine += 1
let decoded: SessionEvent[]
try {
decoded = decodeStorageRecord(JSON.parse(line.toString('utf8')))
} catch {
this.issue ??= new Error(`corrupt session log: unparsable committed event at line ${this.eventLine}`)
return
}
if (this.issue !== undefined) {
if (decoded.some(event => event.type === 'turn/end')) throw this.issue
return
}
const rowStart = this.events.length
for (const event of decoded) {
if (event.seq !== this.events.length) {
const expected = this.events.length
this.events.length = rowStart
this.issue = new Error(
`corrupt session log: seq gap in committed region at line ${this.eventLine} `
+ `(expected ${expected}, got ${event.seq})`,
)
if (decoded.some(candidate => candidate.type === 'turn/end')) throw this.issue
return
}
this.events.push(event)
}
this.committedBytes = endByte
}
}
/**
* Parse a complete or torn JSONL buffer into its preserved event prefix. This
* compatibility wrapper supplies the first record separately, then delegates
* event rows to {@link SessionLogScanner}.
*
* @param buffer - the raw bytes of the log file (header line first).
* @returns the header, preserved event prefix, and byte offset safe to append at.
*/
export function scanLog(buffer: Buffer): SessionLogScan {
const headerEnd = buffer.indexOf(0x0A)
if (headerEnd === -1) throw new Error('empty or header-less session log')
const scanner = new SessionLogScanner(buffer.subarray(0, headerEnd + 1))
scanner.write(buffer.subarray(headerEnd + 1))
return scanner.finish()
}
/**
* Parse just the header line of a log into a {@link SessionHeader}, or
* `undefined` if it is missing/not a header. Used by `list()` to read session
* metadata WITHOUT parsing the whole log: a session picker scales with the
* number of sessions, not the total size of every conversation.
* @param firstLine - the first line of a log file (without its trailing newline).
* @returns the parsed header, or `undefined` when the line is not a well-formed session header.
*/
export function parseHeaderMeta(firstLine: string): SessionHeader | undefined {
let parsed: unknown
try {
parsed = JSON.parse(firstLine)
} catch {
return undefined
}
if (!isHeaderLine(parsed)) return undefined
return fromHeaderLine(parsed)
}

View File

@@ -0,0 +1,899 @@
/**
* JSONL durable session-persistence backend. It stores a header and contiguous
* events in one append-only file per session, and delegates orchestration to
* {@link PersistenceCoordinator}. Its side-effect-free locator returns the
* absolute per-session log target before materialization.
* @module @deepseek-ai/dsh-session-persistence-jsonl
*/
import { Context } from 'cordis'
import z from 'schemastery'
import { readdirSync } from 'node:fs'
import { open, mkdir, readFile, readdir, realpath, link, rm, stat, truncate } from 'node:fs/promises'
import { dirname, join, resolve } from 'node:path'
import { performance } from 'node:perf_hooks'
import { scheduler } from 'node:timers/promises'
import { randomBytes } from 'node:crypto'
import {
DEFAULT_PREPARED_SESSION_CACHE_SIZE, DEFAULT_WRITE_BATCH_MAX_DELAY_MS, MAX_WRITE_BATCH_DELAY_MS,
SessionPersistence, SessionPersistenceRevision, PersistenceCoordinator,
type PersistenceBackend, type SessionLocation, type SessionPersistenceSnapshot,
type SessionInspection, type SessionPersistenceRevision as PersistenceRevision, type StoredPrefix,
} from '@deepseek-ai/dsh-session-persistence'
import type { SessionEvent, SessionId, SessionHeader, SessionPreparation } from '@deepseek-ai/dsh-session'
import {
encodeSegment, eventLines, logPath, logSuffix, parseHeaderMeta, projectDir, scanLog, sessionDir,
SessionLogScanner, toHeaderLine,
type JsonlCompression,
} from './format.ts'
import {
compressZstdFrame, createZstdFrameDecoder, decompressZstdFrame, decompressZstdPrefix, scanZstdFrames,
} from './zstd.ts'
import { ensureDurableDirectoryWin32, publishNewFileWin32 } from './win32.ts'
export type { JsonlCompression } from './format.ts'
const DEFAULT_PACK_CHUNKS = true
const DEFAULT_COMPRESSION: JsonlCompression = 'zstd'
/**
* Internal scheduling constant, not deployment configuration: balance
* frame-boundary event-loop yields against `setImmediate` overhead. One frame
* remains an indivisible synchronous decode.
*/
const ZSTD_DECODE_YIELD_INTERVAL_MS = 500
/** Assert that the independently decodable first frame contains only the header record. */
function assertZstdHeaderFrame(plaintext: Buffer): void {
if (plaintext.length === 0 || plaintext.indexOf(0x0A) !== plaintext.length - 1) {
throw new Error('corrupt Zstandard session log: first frame is not exactly one header line')
}
}
/** Loader schema for the JSONL artifact's physical encoding. */
export const JsonlCompressionSchema: z<JsonlCompression> = z.union([
z.const('zstd'),
z.const('none'),
]).default(DEFAULT_COMPRESSION)
/** Plugin config: where the JSONL backend keeps its session logs, and the packed-row write switch. */
export interface Config {
/**
* Root directory for all session files. Required (no default): a default of
* `process.cwd()` would scatter session files as the process's cwd changes
* (bash calls, subprocesses). Sessions group under human-readable project
* directories, then per-session directories. An existing root must be a
* readable directory; an absent root is created on first materialization.
*/
root: string
/**
* Write runs of consecutive `assistant/chunk` delta events as packed
* `text-chunks`/`reasoning-chunks`/`tool-call-chunks` rows (lossless,
* ~60% smaller logs measured on a real session). Defaults to true; false
* keeps one `SessionEvent` per line for diagnostics. Reading packed rows is
* unconditional: a log's layout never depends on this switch.
*/
packChunks?: boolean
/** Physical encoding; defaults to checksummed Zstandard frames. */
compression?: JsonlCompression
/** Maximum cold Session preparations retained for history-to-resume reuse. */
preparedSessionCacheSize?: number
/** Fixed live-event coalescing window; not a backend completion deadline. */
writeBatchMaxDelayMs?: number
}
/** Opaque coordinator token for replacing bytes recovered from a torn frame. */
interface JsonlTornMarker {
truncateTo: number
recoveredEvents: SessionEvent[]
}
interface FileRevisionIdentity {
readonly dev: bigint
readonly ino: bigint
readonly size: bigint
readonly mtimeNs: bigint
readonly ctimeNs: bigint
}
/** Build the source-qualified revision shared by full and lightweight reads. */
function fileRevision(identity: FileRevisionIdentity): PersistenceRevision {
return SessionPersistenceRevision([
identity.dev,
identity.ino,
identity.size,
identity.mtimeNs,
identity.ctimeNs,
].join(':'))
}
/** Whether a filesystem error means absence; every non-ENOENT failure must surface. */
function isENOENT(error: unknown): boolean {
return (error as NodeJS.ErrnoException | null)?.code === 'ENOENT'
}
/**
* The JSONL persistence backend. Load as a plugin; it registers as
* `ctx.sessionPersistence` and (via the coordinator) installs the write-path
* listeners. Its torn-tail marker carries the byte offset and any events
* recovered from an incomplete final Zstandard frame.
*/
export class SessionPersistenceJsonl extends SessionPersistence implements PersistenceBackend<JsonlTornMarker> {
static inject = ['sessions']
static Config: z<Config> = z.object({
root: z.string().required(),
packChunks: z.boolean().default(DEFAULT_PACK_CHUNKS),
compression: JsonlCompressionSchema,
preparedSessionCacheSize: z.number().step(1).min(1).default(DEFAULT_PREPARED_SESSION_CACHE_SIZE),
writeBatchMaxDelayMs: z.number().step(1).min(1).max(MAX_WRITE_BATCH_DELAY_MS)
.default(DEFAULT_WRITE_BATCH_MAX_DELAY_MS),
})
/**
* Backend label for coordinator diagnostics and effects. It shadows
* `Service.name` without changing the service key captured by the base
* constructor.
*/
override readonly name = 'session-persistence-jsonl'
private root: string
private packChunks: boolean
private compression: JsonlCompression
private coordinator: PersistenceCoordinator<JsonlTornMarker>
private rootEncodingCheck: Promise<void> | undefined
constructor(ctx: Context, public config: Config) {
super(ctx)
// Resolve once so later process.cwd() changes cannot split one backend across roots.
this.root = resolve(config.root)
// Programmatic wrappers may construct the backend without Schemastery normalization.
const preparedSessionCacheSize = config.preparedSessionCacheSize
?? DEFAULT_PREPARED_SESSION_CACHE_SIZE
const writeBatchMaxDelayMs = config.writeBatchMaxDelayMs
?? DEFAULT_WRITE_BATCH_MAX_DELAY_MS
this.packChunks = config.packChunks ?? DEFAULT_PACK_CHUNKS
this.compression = config.compression ?? DEFAULT_COMPRESSION
this.assertUsableRoot()
this.coordinator = new PersistenceCoordinator<JsonlTornMarker>(this.ctx, this, {
preparedSessionCacheSize,
writeBatchMaxDelayMs,
})
}
// Each backend keeps the typed service surface beside its storage hooks;
// extracting these trivial forwards would add an inheritance seam.
/* jscpd:ignore-start */
// --- SessionPersistence service surface (delegated to the coordinator) ---
/** Resolve the absolute target path without touching the filesystem. */
locate(meta: SessionHeader): SessionLocation {
return { kind: 'jsonl', path: logPath(this.root, meta.cwd, meta.id, this.compression) }
}
create(meta: SessionHeader): Promise<void> {
return this.coordinator.create(meta)
}
append(id: SessionId, events: readonly SessionEvent[]): Promise<void> {
return this.coordinator.append(id, events)
}
override prepare(id: SessionId, signal?: AbortSignal): Promise<SessionPreparation> {
return this.coordinator.prepare(id, signal)
}
load(id: SessionId): Promise<SessionInspection> {
return this.coordinator.load(id)
}
inspect(id: SessionId, signal?: AbortSignal): Promise<SessionInspection> {
return this.coordinator.inspect(id, signal)
}
// JSONL is sequential media: no loadStoredFrom hook, so the coordinator
// parses the stored prefix (both encodings) and skips forward to fromSeq.
readFrom(id: SessionId, fromSeq: number, signal?: AbortSignal): Promise<{ meta: SessionHeader; events: SessionEvent[] }> {
return this.coordinator.readFrom(id, fromSeq, signal)
}
// One method serves both public `list` and the backend hook; delegating it to
// the coordinator would call this hook recursively.
/* jscpd:ignore-end */
// --- PersistenceBackend hooks (the file-bytes storage primitives) ---
/** Read a stored prefix by id across all project directories when cwd is unknown. */
async loadStored(id: SessionId, signal?: AbortSignal): Promise<StoredPrefix<JsonlTornMarker> | undefined> {
signal?.throwIfAborted()
await this.ensureRootEncoding()
signal?.throwIfAborted()
const path = await this.findLog(id, signal)
if (path === undefined) return undefined
return this.readPrefix(path, id, signal)
}
/**
* Read one log's stat-derived revision without loading its event bytes.
* Resolving an id with unknown cwd still scans the project directories.
*/
async readStoredRevision(id: SessionId, signal?: AbortSignal): Promise<PersistenceRevision | undefined> {
signal?.throwIfAborted()
await this.ensureRootEncoding()
signal?.throwIfAborted()
const path = await this.findLog(id, signal)
if (path === undefined) return undefined
try {
const identity = await stat(path, { bigint: true })
signal?.throwIfAborted()
return fileRevision(identity)
} catch (error: unknown) {
signal?.throwIfAborted()
if (isENOENT(error)) return undefined
throw error
}
}
/**
* Read a stored prefix and convert torn-tail state to the opaque marker the
* coordinator can round-trip without knowing the physical encoding.
*/
private async readPrefix(
path: string,
expectedId?: SessionId,
signal?: AbortSignal,
): Promise<StoredPrefix<JsonlTornMarker>> {
let buffer: Buffer
let revision: PersistenceRevision
for (;;) {
signal?.throwIfAborted()
const before = fileRevision(await stat(path, { bigint: true }))
buffer = await readFile(path, { signal })
signal?.throwIfAborted()
const after = fileRevision(await stat(path, { bigint: true }))
if (before === after) {
revision = after
break
}
}
let prefix: Omit<StoredPrefix<JsonlTornMarker>, 'revision'>
if (this.compression === 'zstd') {
prefix = await this.readZstdPrefix(buffer, signal)
} else {
signal?.throwIfAborted()
const { meta, events, committedBytes } = scanLog(buffer)
signal?.throwIfAborted()
prefix = {
meta,
events,
...committedBytes < buffer.byteLength
? { tornMarker: { truncateTo: committedBytes, recoveredEvents: [] } }
: {},
}
}
signal?.throwIfAborted()
await this.assertStoredIdentity(path, prefix.meta, expectedId, signal)
signal?.throwIfAborted()
return { ...prefix, revision }
}
/** Decode complete frames and retain complete JSONL records from a torn final frame. */
private async readZstdPrefix(
buffer: Buffer,
signal?: AbortSignal,
): Promise<Omit<StoredPrefix<JsonlTornMarker>, 'revision'>> {
signal?.throwIfAborted()
const { frames, tornStart } = scanZstdFrames(buffer)
signal?.throwIfAborted()
if (frames.length === 0) throw new Error('empty or header-less Zstandard session log')
const decoder = createZstdFrameDecoder()
let yieldDeadline = performance.now() + ZSTD_DECODE_YIELD_INTERVAL_MS
try {
const decodedFrames = decoder.decode(buffer, frames)
signal?.throwIfAborted()
const headerFrame = decodedFrames.next()
signal?.throwIfAborted()
/* v8 ignore next -- a non-empty structural frame list makes the decoder yield its first frame or throw. */
if (headerFrame.done) throw new Error('empty or header-less Zstandard session log')
assertZstdHeaderFrame(headerFrame.value)
const scanner = new SessionLogScanner(headerFrame.value)
let remainingFrames = frames.length - 1
for (const plaintext of decodedFrames) {
signal?.throwIfAborted()
scanner.write(plaintext)
remainingFrames -= 1
if (remainingFrames > 0 && performance.now() >= yieldDeadline) {
await scheduler.yield()
signal?.throwIfAborted()
yieldDeadline = performance.now() + ZSTD_DECODE_YIELD_INTERVAL_MS
}
}
signal?.throwIfAborted()
const complete = scanner.checkpoint()
if (complete.committedBytes !== complete.inputBytes) {
throw new Error('corrupt Zstandard session log: complete frame contains a torn JSONL record')
}
if (tornStart === undefined) {
const prefix = scanner.finish()
return { meta: prefix.meta, events: prefix.events }
}
let recoveredPlaintext: Buffer = Buffer.alloc(0)
try {
signal?.throwIfAborted()
recoveredPlaintext = await decompressZstdPrefix(buffer.subarray(tornStart))
} catch {
/* v8 ignore next -- decoder failure plus concurrent abort is timing-dependent */
if (signal?.aborted) signal.throwIfAborted()
// A structurally incomplete final frame may end before Node's decoder can
// emit any plaintext; the complete prior frames remain recoverable.
}
signal?.throwIfAborted()
scanner.write(recoveredPlaintext)
const recoveredPrefix = scanner.finish()
signal?.throwIfAborted()
return {
meta: recoveredPrefix.meta,
events: recoveredPrefix.events,
tornMarker: {
truncateTo: tornStart,
recoveredEvents: recoveredPrefix.events.slice(complete.eventCount),
},
}
} catch (error) {
/* v8 ignore next -- decoder failure plus concurrent abort is timing-dependent */
if (signal?.aborted) signal.throwIfAborted()
throw error
} finally {
decoder.close()
}
}
/** Durably append a batch, lazily materializing the file when not yet present. */
async appendBatch(meta: SessionHeader, events: readonly SessionEvent[], isMaterialized: boolean): Promise<void> {
await this.ensureRootEncoding()
if (isMaterialized) {
await this.appendLines(meta, events)
} else {
await this.materialize(meta, events)
}
}
/**
* Make a crash repair durable: truncate a torn tail, restore complete events
* decoded from it, then append synthetic closers. Two fsync'd steps — the seam
* does not require this to be atomic.
*/
async commitRepair(
meta: SessionHeader,
tornMarker: JsonlTornMarker | undefined,
closers: readonly SessionEvent[],
): Promise<void> {
if (tornMarker !== undefined) await this.repair(meta, tornMarker.truncateTo)
const repairedEvents = [...(tornMarker?.recoveredEvents ?? []), ...closers]
if (repairedEvents.length > 0) await this.appendLines(meta, repairedEvents)
}
/** List valid unique stored sessions' metadata (header line only — no full-log parse). */
async list(signal?: AbortSignal): Promise<SessionHeader[]> {
return (await this.listArtifacts(signal)).map(artifact => artifact.header)
}
/** List metadata plus a stat-derived identity for each append-only log. */
async listSnapshots(signal?: AbortSignal): Promise<SessionPersistenceSnapshot[]> {
const snapshots: SessionPersistenceSnapshot[] = []
for (const artifact of await this.listArtifacts(signal)) {
signal?.throwIfAborted()
try {
const identity = await stat(artifact.path, { bigint: true })
signal?.throwIfAborted()
snapshots.push({
header: artifact.header,
revision: fileRevision(identity),
})
} catch (error: unknown) {
signal?.throwIfAborted()
if (!isENOENT(error)) throw error
}
}
signal?.throwIfAborted()
return snapshots
}
private async listArtifacts(signal?: AbortSignal): Promise<Array<{ header: SessionHeader; path: string }>> {
signal?.throwIfAborted()
await this.ensureRootEncoding()
signal?.throwIfAborted()
const artifacts: Array<{ header: SessionHeader; path: string }> = []
const ids = new Set<SessionId>()
for (const project of await this.listProjectDirs(signal)) {
signal?.throwIfAborted()
for (const dir of await this.listSessionDirs(project, signal)) {
signal?.throwIfAborted()
const opposite = join(dir, `session${logSuffix(this.oppositeCompression())}`)
const oppositeExists = await this.exists(opposite)
signal?.throwIfAborted()
if (oppositeExists) throw this.encodingMismatch(opposite)
const path = join(dir, `session${logSuffix(this.compression)}`)
const pathExists = await this.exists(path)
signal?.throwIfAborted()
if (!pathExists) continue
// Read only headers so listing scales with session count, not log size.
const first = this.compression === 'zstd'
? await this.readFirstZstdLine(path, signal)
: await this.readFirstLine(path, signal)
signal?.throwIfAborted()
if (first === undefined) continue // empty/half-written file
const meta = parseHeaderMeta(first)
if (meta === undefined) continue // not a session header
await this.assertStoredIdentity(path, meta, undefined, signal)
signal?.throwIfAborted()
if (ids.has(meta.id)) {
throw new Error(`duplicate JSONL session id "${meta.id}" appears in multiple project directories`)
}
ids.add(meta.id)
artifacts.push({ header: meta, path })
}
}
signal?.throwIfAborted()
return artifacts
}
// --- materialization / append / repair (file mechanics) ---
/** Atomically write the header line + first batch (temp-write, fsync, publish). */
private async materialize(meta: SessionHeader, events: readonly SessionEvent[]): Promise<void> {
const project = projectDir(this.root, meta.cwd)
const dir = sessionDir(this.root, meta.cwd, meta.id)
const finalPath = logPath(this.root, meta.cwd, meta.id, this.compression)
await this.rejectOppositeArtifact(meta.cwd, meta.id)
const content = await this.encodeMaterialization(meta, events)
/* v8 ignore next -- native Windows coverage exercises this platform dispatch; Linux covers the POSIX peer */
if (process.platform === 'win32') {
await this.materializeWin32(project, dir, finalPath, meta.id, content)
} else {
await this.materializePosix(project, dir, finalPath, meta.id, content)
}
}
/* v8 ignore start -- Windows uses the Win32 durable-publish path; POSIX coverage exercises this peer. */
private async materializePosix(
project: string,
dir: string,
finalPath: string,
id: SessionId,
content: Buffer | string,
): Promise<void> {
await mkdir(this.root, { recursive: true, mode: 0o700 })
await this.syncDirPosix(dirname(this.root))
await mkdir(project, { recursive: true, mode: 0o700 })
await this.syncDirPosix(this.root)
await mkdir(dir, { recursive: true, mode: 0o700 })
await this.syncDirPosix(project)
await this.rejectExistingLog(finalPath, id)
const tmp = await this.writeSyncedTempFile(finalPath, content)
// Publish via link()+unlink(), NOT rename(): link fails with EEXIST if the
// final path already exists, so two processes materializing the same id
// concurrently cannot clobber each other. rename() would silently overwrite.
let linked = false
try {
await link(tmp, finalPath)
linked = true
} finally {
// Remove an unpublished temp on failure. After publication, defer cleanup
// until the directory entry is durable so cleanup cannot reject a live log.
/* v8 ignore next -- link failure is the TOCTOU/IO race guarded above; not reachable in test */
if (!linked) await rm(tmp, { force: true })
}
// link() succeeded — the log is published. fsync the directory so the new
// entry survives a power loss: the new link is not crash-durable until the
// parent directory's metadata is synced.
await this.syncDirPosix(dir)
// Best-effort temp cleanup: the log is already published and durable, so a
// failure to remove the (now-redundant) temp hard link must NOT reject the
// append. Swallow only the rm failure; nothing else of consequence runs here.
try {
await rm(tmp, { force: true })
} catch {
/* v8 ignore next -- redundant temp link; publish already durable, rm failure is an unreachable IO edge */
}
}
/* v8 ignore stop */
/* v8 ignore start -- native Windows coverage exercises this integration path */
private async materializeWin32(
project: string,
dir: string,
finalPath: string,
id: SessionId,
content: Buffer | string,
): Promise<void> {
await ensureDurableDirectoryWin32(this.root)
await ensureDurableDirectoryWin32(project)
await ensureDurableDirectoryWin32(dir)
await this.rejectExistingLog(finalPath, id)
const tmp = await this.writeSyncedTempFile(finalPath, content)
try {
await publishNewFileWin32(tmp, finalPath)
} catch (error) {
await rm(tmp, { force: true })
throw error
}
}
/* v8 ignore stop */
private async rejectExistingLog(finalPath: string, id: SessionId): Promise<void> {
// Never publish over an existing committed log: materialize is the first
// write of a session the backend believes is new. A file here means a
// different session shares this id on disk — reject loudly. (createCore
// already guards the create path, so this is unreachable-in-practice TOCTOU
// defense.)
/* v8 ignore next 3 -- createCore guards collisions before materialize; this is a TOCTOU backstop */
if (await this.exists(finalPath)) {
throw new Error(`refusing to materialize "${id}": a log already exists on disk (load/resume it instead)`)
}
}
private async writeSyncedTempFile(finalPath: string, content: Buffer | string): Promise<string> {
const tmp = `${finalPath}.${randomBytes(6).toString('hex')}.tmp`
const handle = await open(tmp, 'wx', 0o600)
try {
await handle.writeFile(content)
await handle.sync()
} finally {
await handle.close()
}
return tmp
}
/** Encode the header and first batch without combining their frame boundaries. */
private async encodeMaterialization(meta: SessionHeader, events: readonly SessionEvent[]): Promise<Buffer | string> {
const header = JSON.stringify(toHeaderLine(meta)) + '\n'
const body = eventLines(events, this.packChunks) + '\n'
if (this.compression === 'none') return header + body
const headerFrame = await compressZstdFrame(header)
const eventFrame = await compressZstdFrame(body)
return Buffer.concat([headerFrame, eventFrame])
}
/** Encode one durable append batch in the configured physical representation. */
private async encodeEventBatch(events: readonly SessionEvent[]): Promise<Buffer | string> {
const body = eventLines(events, this.packChunks) + '\n'
return this.compression === 'zstd' ? compressZstdFrame(body) : body
}
/** fsync a POSIX directory so a just-created/renamed entry is crash-durable. */
/* v8 ignore start -- Windows uses write-through namespace operations; POSIX coverage exercises directory fsync. */
private async syncDirPosix(dir: string): Promise<void> {
const handle = await open(dir, 'r')
try {
await handle.sync()
} finally {
await handle.close()
}
}
/* v8 ignore stop */
/**
* Append and fsync event lines. On a partial write or sync failure, restore the
* previous size before rethrowing because the unchanged cursor will retry the
* batch; leaving partial bytes would create duplicate sequence numbers.
*/
private async appendLines(meta: SessionHeader, events: readonly SessionEvent[]): Promise<void> {
const content = await this.encodeEventBatch(events)
const path = logPath(this.root, meta.cwd, meta.id, this.compression)
const handle = await open(path, 'a')
let closed = false
const closeAppendHandle = async (): Promise<void> => {
if (closed) return
closed = true
await handle.close()
}
try {
const { size: before } = await handle.stat()
try {
await handle.writeFile(content)
await handle.sync()
} catch (error) {
try {
await closeAppendHandle()
await this.rollbackAppend(path, before)
} catch (rollbackError) {
throw new AggregateError([error, rollbackError], `failed to roll back append to "${path}"`)
}
throw error
}
} finally {
await closeAppendHandle()
}
}
private async rollbackAppend(path: string, size: number): Promise<void> {
const handle = await open(path, 'r+')
try {
await handle.truncate(size)
await handle.sync()
} finally {
await handle.close()
}
}
/** Truncate the log file to `offset` bytes and fsync (discard the crash tail). */
private async repair(meta: SessionHeader, offset: number): Promise<void> {
const path = logPath(this.root, meta.cwd, meta.id, this.compression)
await truncate(path, offset)
const handle = await open(path, 'r+')
try {
await handle.sync()
} finally {
await handle.close()
}
}
// --- discovery helpers ---
/**
* Read the first newline-terminated line of a file without loading the whole
* file. Returns undefined if the file is empty or has no complete first line.
* Reads in bounded chunks so a huge log costs only the header read.
*/
private async readFirstLine(path: string, signal?: AbortSignal): Promise<string | undefined> {
signal?.throwIfAborted()
const handle = await open(path, 'r')
try {
signal?.throwIfAborted()
const chunks: Buffer[] = []
const buf = Buffer.alloc(8192)
for (;;) {
signal?.throwIfAborted()
const { bytesRead } = await handle.read(buf, 0, buf.length, null)
signal?.throwIfAborted()
if (bytesRead === 0) return undefined // EOF with no newline → no complete line
const slice = buf.subarray(0, bytesRead)
const nl = slice.indexOf(0x0a)
if (nl !== -1) {
chunks.push(slice.subarray(0, nl))
signal?.throwIfAborted()
return Buffer.concat(chunks).toString('utf8')
}
chunks.push(Buffer.from(slice))
}
} finally {
await handle.close()
}
}
/** Read and validate only the independently compressed header frame. */
private async readFirstZstdLine(path: string, signal?: AbortSignal): Promise<string | undefined> {
signal?.throwIfAborted()
const handle = await open(path, 'r')
try {
signal?.throwIfAborted()
let content = Buffer.alloc(0)
const chunk = Buffer.alloc(8192)
for (;;) {
signal?.throwIfAborted()
const { bytesRead } = await handle.read(chunk, 0, chunk.length, null)
signal?.throwIfAborted()
if (bytesRead === 0) return undefined
signal?.throwIfAborted()
content = Buffer.concat([content, chunk.subarray(0, bytesRead)])
signal?.throwIfAborted()
const first = scanZstdFrames(content, 1).frames[0]
signal?.throwIfAborted()
if (first === undefined) continue
let plaintext: Buffer
try {
signal?.throwIfAborted()
plaintext = await decompressZstdFrame(content.subarray(first.start, first.end))
} catch (error) {
/* v8 ignore next -- decoder failure plus concurrent abort is timing-dependent */
if (signal?.aborted) signal.throwIfAborted()
throw new Error('corrupt Zstandard session log: header frame failed validation', { cause: error })
}
signal?.throwIfAborted()
assertZstdHeaderFrame(plaintext)
return plaintext.subarray(0, -1).toString('utf8')
}
} finally {
await handle.close()
}
}
/** Find the unique physical log for an id across every project directory. */
private async findLog(id: SessionId, signal?: AbortSignal): Promise<string | undefined> {
const matches: string[] = []
for (const project of await this.listProjectDirs(signal)) {
signal?.throwIfAborted()
await this.rejectLegacyFlatArtifact(project, id, signal)
signal?.throwIfAborted()
const dir = join(project, encodeSegment(id))
const path = join(dir, `session${logSuffix(this.compression)}`)
const opposite = join(dir, `session${logSuffix(this.oppositeCompression())}`)
const oppositeExists = await this.exists(opposite)
signal?.throwIfAborted()
if (oppositeExists) throw this.encodingMismatch(opposite)
const pathExists = await this.exists(path)
signal?.throwIfAborted()
if (pathExists) matches.push(path)
}
if (matches.length > 1) {
throw new Error(`duplicate JSONL session id "${id}" appears in multiple project directories`)
}
signal?.throwIfAborted()
return matches[0]
}
/** Require an existing configured root to be a readable directory. */
private assertUsableRoot(): void {
try {
readdirSync(this.root)
} catch (error) {
if (isENOENT(error)) return
throw error
}
}
/** Reject metadata that does not identify the selected physical log. */
private async assertStoredIdentity(
path: string,
meta: SessionHeader,
expectedId?: SessionId,
signal?: AbortSignal,
): Promise<void> {
signal?.throwIfAborted()
if (expectedId !== undefined && meta.id !== expectedId) {
throw new Error(`corrupt session log "${path}": requested id "${expectedId}" does not match header id "${meta.id}"`)
}
let expectedPath: string
try {
expectedPath = logPath(this.root, meta.cwd, meta.id, this.compression)
} catch (error) {
throw new Error(`corrupt session log "${path}": header id cannot name a storage path`, { cause: error })
}
if (path !== expectedPath && !await this.sameFile(path, expectedPath, signal)) {
throw new Error(`corrupt session log "${path}": header id "${meta.id}" and cwd identify "${expectedPath}"`)
}
signal?.throwIfAborted()
}
/**
* Whether two path spellings resolve to the same physical file. This admits
* case aliases on case-insensitive filesystems without weakening identity
* checks on case-sensitive stores.
*/
private async sameFile(path: string, expectedPath: string, signal?: AbortSignal): Promise<boolean> {
signal?.throwIfAborted()
try {
const [actual, expected] = await Promise.all([realpath(path), realpath(expectedPath)])
signal?.throwIfAborted()
return actual === expected
} catch (error) {
signal?.throwIfAborted()
/* v8 ignore else -- non-ENOENT realpath failures require an external permission or I/O fault */
if (isENOENT(error)) return false
/* v8 ignore next -- non-ENOENT realpath failures are external I/O faults, propagated unchanged */
throw error
}
}
/** The human-readable project directories under the configured root. */
private async listProjectDirs(signal?: AbortSignal): Promise<string[]> {
try {
signal?.throwIfAborted()
const entries = await readdir(this.root, { withFileTypes: true })
signal?.throwIfAborted()
return entries.filter(e => e.isDirectory()).map(e => join(this.root, e.name))
} catch (error) {
// Only an absent root means no sessions; rethrow every other I/O failure.
if (isENOENT(error)) return []
throw error
}
}
/** List session-owned directories and reject the obsolete flat-file layout. */
private async listSessionDirs(project: string, signal?: AbortSignal): Promise<string[]> {
signal?.throwIfAborted()
const entries = await readdir(project, { withFileTypes: true })
signal?.throwIfAborted()
const legacy = entries.find(entry =>
entry.isFile() && (entry.name.endsWith('.jsonl') || entry.name.endsWith('.jsonl.zstd')))
if (legacy !== undefined) throw this.legacyLayout(join(project, legacy.name))
return entries.filter(entry => entry.isDirectory()).map(entry => join(project, entry.name))
}
/** Reject a root that already belongs to the other physical encoding. */
private ensureRootEncoding(): Promise<void> {
this.rootEncodingCheck ??= this.checkRootEncoding()
return this.rootEncodingCheck
}
private async checkRootEncoding(): Promise<void> {
for (const project of await this.listProjectDirs()) {
for (const dir of await this.listSessionDirs(project)) {
const incompatible = join(dir, `session${logSuffix(this.oppositeCompression())}`)
if (await this.exists(incompatible)) throw this.encodingMismatch(incompatible)
}
}
}
private async rejectLegacyFlatArtifact(
project: string,
id: SessionId,
signal?: AbortSignal,
): Promise<void> {
signal?.throwIfAborted()
const encoded = encodeSegment(id)
for (const compression of ['zstd', 'none'] as const) {
const path = join(project, encoded + logSuffix(compression))
const artifactExists = await this.exists(path)
signal?.throwIfAborted()
if (artifactExists) throw this.legacyLayout(path)
}
}
private async rejectOppositeArtifact(cwd: string | undefined, id: SessionId): Promise<void> {
const path = logPath(this.root, cwd, id, this.oppositeCompression())
if (await this.exists(path)) throw this.encodingMismatch(path)
}
private oppositeCompression(): JsonlCompression {
return this.compression === 'zstd' ? 'none' : 'zstd'
}
private encodingMismatch(path: string): Error {
return new Error(
`session artifact ${JSON.stringify(path)} uses ${logSuffix(this.oppositeCompression())}, `
+ `but this backend is configured for compression ${JSON.stringify(this.compression)}; `
+ 'use a separate root or select the matching compression mode',
)
}
private legacyLayout(path: string): Error {
return new Error(
`session artifact ${JSON.stringify(path)} uses the unsupported flat-file layout; `
+ 'use a separate root or move it into a project/session directory before loading',
)
}
private async exists(path: string): Promise<boolean> {
try {
const handle = await open(path, 'r')
await handle.close()
return true
} catch (error) {
// Only ENOENT means absent. A permission/I/O error must surface rather
// than letting load or collision checks proceed under false absence.
// Windows reports ENOENT, not ENOTDIR, for `regular-file/child`; verify
// the immediate parent so a blocked session directory remains a storage fault.
/* v8 ignore else -- Windows reports file-valued parents as ENOENT; POSIX covers direct ENOTDIR. */
if (isENOENT(error)) {
await this.assertLogParentAllowsAbsence(path)
return false
}
/* v8 ignore next -- Windows repairs ENOTDIR from ENOENT above; POSIX covers direct ENOTDIR. */
throw error
}
}
/* v8 ignore start -- native Windows coverage exercises this repair; POSIX open reports ENOTDIR before this point. */
private async assertLogParentAllowsAbsence(path: string): Promise<void> {
try {
const parent = dirname(path)
const info = await stat(parent)
if (info.isDirectory()) return
const error = new Error(`ENOTDIR: parent path exists but is not a directory: ${parent}`) as NodeJS.ErrnoException
error.code = 'ENOTDIR'
error.path = parent
throw error
} catch (error) {
if (isENOENT(error)) return
throw error
}
}
/* v8 ignore stop */
}
export default SessionPersistenceJsonl

View File

@@ -0,0 +1,30 @@
/**
* Package-owned invariant companion for `@deepseek-ai/dsh-session-persistence-jsonl`.
* @module @deepseek-ai/dsh-session-persistence-jsonl/invariant
*/
/* jscpd:ignore-start */
import type { Context } from 'cordis'
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
const PACKAGE_NAME = '@deepseek-ai/dsh-session-persistence-jsonl'
/** Cordis companion plugin name. */
export const name = 'session-persistence-jsonl-invariant'
/** Service required before the companion can reserve package ownership. */
export const inject = ['invariants']
/**
* No runtime invariant: persistence correctness requires backend round-trip and crash-tail tests;
* this package exposes no continuously observable in-process relation.
*/
const install: InvariantInstaller = () => {}
/**
* Register this package's invariant companion.
* @param ctx - Cordis context carrying the invariant service.
* @returns the installed registration's disposer after setup succeeds.
*/
export const apply = (ctx: Context): Promise<() => void> =>
Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
/* jscpd:ignore-end */

View File

@@ -0,0 +1,152 @@
/**
* Windows durable namespace helpers for the JSONL backend.
*
* POSIX publishes a newly-created log by creating a directory entry and then
* fsyncing the parent directory. Windows does not expose that parent-directory
* fsync contract through Node, so the Windows path uses the native durable
* namespace primitive instead: create a staging object in the target directory
* and publish it with `MoveFileExW(..., MOVEFILE_WRITE_THROUGH)` without
* replacement or cross-volume copy fallback.
*
* @module dsh-session-persistence-jsonl/win32
*/
import { mkdtemp, rm, stat } from 'node:fs/promises'
import { join, parse, resolve, toNamespacedPath } from 'node:path'
type MoveFileExW = (existing: string, replacement: string, flags: number) => number
type GetLastError = () => number
interface Win32Bindings {
moveFileExW: MoveFileExW
getLastError: GetLastError
}
interface Win32ErrnoException extends NodeJS.ErrnoException {
win32Code: number
dest: string
}
const MOVEFILE_WRITE_THROUGH = 0x00000008
const ERROR_FILE_NOT_FOUND = 2
const ERROR_PATH_NOT_FOUND = 3
const ERROR_ACCESS_DENIED = 5
const ERROR_NOT_SAME_DEVICE = 17
const ERROR_FILE_EXISTS = 80
const ERROR_INVALID_NAME = 123
const ERROR_ALREADY_EXISTS = 183
let bindings: Win32Bindings | undefined
/** Load the small Win32 surface lazily so non-Windows processes never load Koffi. */
async function win32(): Promise<Win32Bindings> {
if (bindings !== undefined) return bindings
const koffi = (await import('koffi')).default
const kernel32 = koffi.load('kernel32.dll')
bindings = {
moveFileExW: kernel32.func('__stdcall', 'MoveFileExW', 'int', ['str16', 'str16', 'uint']) as MoveFileExW,
getLastError: kernel32.func('__stdcall', 'GetLastError', 'uint', []) as GetLastError,
}
return bindings
}
function errnoCode(win32Code: number): string {
switch (win32Code) {
case ERROR_FILE_NOT_FOUND:
case ERROR_PATH_NOT_FOUND:
return 'ENOENT'
case ERROR_ACCESS_DENIED:
return 'EACCES'
case ERROR_NOT_SAME_DEVICE:
return 'EXDEV'
case ERROR_FILE_EXISTS:
case ERROR_ALREADY_EXISTS:
return 'EEXIST'
case ERROR_INVALID_NAME:
return 'EINVAL'
default:
return 'EIO'
}
}
function win32Error(syscall: string, win32Code: number, path: string, dest: string): Win32ErrnoException {
const code = errnoCode(win32Code)
const error = new Error(`${syscall} ${code} (Win32 ${win32Code}): ${path} -> ${dest}`) as Win32ErrnoException
error.code = code
error.errno = win32Code
error.syscall = syscall
error.path = path
error.dest = dest
error.win32Code = win32Code
return error
}
function isENOENT(error: unknown): boolean {
return (error as NodeJS.ErrnoException | null)?.code === 'ENOENT'
}
function isEEXIST(error: unknown): boolean {
return (error as NodeJS.ErrnoException | null)?.code === 'EEXIST'
}
async function assertDirectory(path: string): Promise<boolean> {
try {
const info = await stat(path)
if (info.isDirectory()) return true
const error = new Error(`path exists but is not a directory: ${path}`) as NodeJS.ErrnoException
error.code = 'ENOTDIR'
error.path = path
throw error
} catch (error) {
if (isENOENT(error)) return false
throw error
}
}
/**
* Publish `existing` at `replacement` with Windows write-through rename
* semantics. The destination must not already exist; the move must stay within
* the volume (no copy fallback flag is set).
* @param existing - the synced staging path to move.
* @param replacement - the final path, which must not already exist.
*/
export async function publishNewFileWin32(existing: string, replacement: string): Promise<void> {
const api = await win32()
const ok = api.moveFileExW(toNamespacedPath(existing), toNamespacedPath(replacement), MOVEFILE_WRITE_THROUGH)
if (ok === 0) throw win32Error('MoveFileExW', api.getLastError(), existing, replacement)
}
/**
* Create `target` and its missing ancestors with durable Windows namespace
* publication. Each missing directory is first created as a random staging
* sibling, then moved to its final name with `MOVEFILE_WRITE_THROUGH`; races
* with another creator are accepted only after verifying the winner is a
* directory.
* @param target - the absolute directory path to create durably when absent.
*/
export async function ensureDurableDirectoryWin32(target: string): Promise<void> {
const absolute = resolve(target)
const root = parse(absolute).root
await assertDirectory(root)
const segments = absolute.slice(root.length).split(/[\\/]+/).filter(part => part.length > 0)
let current = root
for (const segment of segments) {
const next = join(current, segment)
if (!await assertDirectory(next)) await createLeafDirectoryWin32(current, next)
current = next
}
}
async function createLeafDirectoryWin32(parent: string, target: string): Promise<void> {
// Keep the staging component independent of the target basename so a legal
// 255-byte target component does not make mkdtemp's sibling name too long.
const staging = await mkdtemp(join(parent, '.dsh-mkdir-'))
try {
await publishNewFileWin32(staging, target)
} catch (error) {
await rm(staging, { recursive: true, force: true })
if (isEEXIST(error) && await assertDirectory(target)) return
throw error
}
}

View File

@@ -0,0 +1,178 @@
/**
* Node-private synchronous Zstandard frame decoder optimization.
* @module dsh-session-persistence-jsonl/zstd-private-decoder
*/
import { constants as bufferConstants } from 'node:buffer'
import { createZstdDecompress } from 'node:zlib'
import type { ZstdFrameDecoder, ZstdFrameRange } from './zstd.ts'
const DECODE_CHUNK_SIZE = 1024 * 1024
interface NodeZstdPrivateHandle {
writeSync(
flushFlag: number,
input: Buffer,
inputOffset: number,
inputLength: number,
output: Buffer,
outputOffset: number,
outputLength: number,
): void
}
type NodeZstdPrivateWriteState = Uint32Array & { 0: number; 1: number }
interface NodeZstdPrivateState {
[key: symbol]: unknown
_handle: NodeZstdPrivateHandle | null
_writeState: NodeZstdPrivateWriteState
_defaultFlushFlag: number
}
type NodeZstdPrivateStream = ReturnType<typeof createZstdDecompress> & NodeZstdPrivateState
/** Return the stream with its observed private Node contract, or reject that optimization. */
function privateZstdStream(
stream: ReturnType<typeof createZstdDecompress>,
): { stream: NodeZstdPrivateStream; errorKey: symbol } | undefined {
const candidate = stream as unknown as Partial<NodeZstdPrivateState>
const handle = candidate._handle
const errorKey = Reflect.ownKeys(stream).find((key): key is symbol => (
typeof key === 'symbol' && key.description === 'kError'
))
/* v8 ignore next -- one test runtime exposes one Node-private shape; the Node 22/24/26 matrix checks compatibility. */
if (
typeof handle !== 'object' || handle === null
|| typeof (handle as { writeSync?: unknown }).writeSync !== 'function'
|| !(candidate._writeState instanceof Uint32Array)
|| candidate._writeState.length < 2
|| typeof candidate._defaultFlushFlag !== 'number'
|| errorKey === undefined
|| candidate[errorKey] !== null
) return undefined
return { stream: stream as NodeZstdPrivateStream, errorKey }
}
/**
* Synchronous multi-frame decoder backed by one Node Zstd stream handle. Node
* exposes synchronous decoding only as a one-shot API, so this adapter uses
* the stream's private handle contract to reuse its native context and output
* chunks across frames.
*/
export class NodePrivateZstdFrameDecoder implements ZstdFrameDecoder {
private readonly output = Buffer.allocUnsafe(DECODE_CHUNK_SIZE)
private decoderError?: Error
private started = false
private closed = false
private constructor(
private readonly stream: NodeZstdPrivateStream,
private readonly errorKey: symbol,
) {
this.stream.on('error', (error: Error) => {
this.decoderError ??= error
})
}
/**
* Create the optimized decoder when this Node release exposes the expected
* private stream shape.
* @returns a shared decoder, or `undefined` when callers must use the public fallback.
*/
static create(): NodePrivateZstdFrameDecoder | undefined {
const stream = createZstdDecompress({ chunkSize: DECODE_CHUNK_SIZE })
const privateAccess = privateZstdStream(stream)
/* v8 ignore next -- reached only when a supported Node release changes its private stream shape. */
if (privateAccess !== undefined) {
return new NodePrivateZstdFrameDecoder(privateAccess.stream, privateAccess.errorKey)
}
/* v8 ignore next -- the active Node runtime passed the private-shape probe above. */
stream.close()
/* v8 ignore next -- the active Node runtime passed the private-shape probe above. */
return undefined
}
/** @inheritdoc */
public *decode(source: Buffer, frames: readonly ZstdFrameRange[]): Generator<Buffer, void, void> {
if (this.started) throw new Error('Zstandard frame decoder was already started')
if (this.closed) throw new Error('cannot start a closed Zstandard frame decoder')
this.started = true
try {
for (const frame of frames) {
try {
yield this.decodeFrame(source.subarray(frame.start, frame.end))
} catch (error) {
throw new Error(`corrupt Zstandard session log: frame at byte ${frame.start} failed validation`, {
cause: error,
})
}
}
} finally {
this.close()
}
}
/** Decode one frame; its returned scratch view remains valid until the next call. */
private decodeFrame(input: Buffer): Buffer {
const handle = this.stream._handle
/* v8 ignore next -- decode() rejects closed instances before entering this private frame operation. */
if (this.closed || handle === null) throw new Error('cannot decode with a closed Zstandard frame decoder')
let inputOffset = 0
let inputRemaining = input.length
let outputBytes = 0
const fullChunks: Buffer[] = []
for (;;) {
handle.writeSync(
this.stream._defaultFlushFlag,
input,
inputOffset,
inputRemaining,
this.output,
0,
this.output.length,
)
if (this.decoderError !== undefined) throw this.decoderError
const internalError = this.stream[this.errorKey]
if (internalError !== null) {
if (internalError instanceof Error) throw internalError
throw new Error('Zstandard decoder exposed a non-Error internal failure')
}
const outputAfter = this.stream._writeState[0]
const inputAfter = this.stream._writeState[1]
const consumed = inputRemaining - inputAfter
const produced = this.output.length - outputAfter
if (produced > 0) {
outputBytes += produced
/* v8 ignore next -- Buffer cannot materialize a frame beyond its own process-wide maximum length. */
if (outputBytes > bufferConstants.MAX_LENGTH) {
throw new Error(`Zstandard frame output exceeds ${bufferConstants.MAX_LENGTH} bytes`)
}
}
if (outputAfter !== 0) {
/* v8 ignore next -- structurally scanned ranges contain exactly one complete frame and no trailing bytes. */
if (inputAfter !== 0) throw new Error('Zstandard frame decoder left trailing input')
const finalChunk = this.output.subarray(0, produced)
if (fullChunks.length === 0) return finalChunk
if (produced > 0) fullChunks.push(Buffer.from(finalChunk))
const onlyChunk = fullChunks[0] as Buffer
return fullChunks.length === 1
? onlyChunk
: Buffer.concat(fullChunks, outputBytes)
}
fullChunks.push(Buffer.from(this.output))
inputOffset += consumed
inputRemaining = inputAfter
}
}
/** @inheritdoc */
close(): void {
if (this.closed) return
this.closed = true
this.stream.close()
}
}

View File

@@ -0,0 +1,40 @@
/**
* Public-API synchronous Zstandard frame decoder fallback.
* @module dsh-session-persistence-jsonl/zstd-public-decoder
*/
import { zstdDecompressSync } from 'node:zlib'
import type { ZstdFrameDecoder, ZstdFrameRange } from './zstd.ts'
/** Multi-frame adapter built exclusively from Node's supported one-shot API. */
export class PublicZstdFrameDecoder implements ZstdFrameDecoder {
private started = false
private closed = false
/** @inheritdoc */
public *decode(source: Buffer, frames: readonly ZstdFrameRange[]): Generator<Buffer, void, void> {
if (this.started) throw new Error('Zstandard frame decoder was already started')
if (this.closed) throw new Error('cannot start a closed Zstandard frame decoder')
this.started = true
try {
for (const { start, end } of frames) {
let decoded: Buffer
try {
decoded = zstdDecompressSync(source.subarray(start, end))
} catch (error) {
throw new Error(`corrupt Zstandard session log: frame at byte ${start} failed validation`, {
cause: error,
})
}
yield decoded
}
} finally {
this.close()
}
}
/** @inheritdoc */
close(): void {
this.closed = true
}
}

View File

@@ -0,0 +1,156 @@
/**
* Zstandard frame primitives for the JSONL persistence backend. The backend
* owns a concatenated-frame container so it can append and recover batches
* without exposing compression mechanics through the persistence seam.
* @module dsh-session-persistence-jsonl/zstd
*/
import {
constants, zstdCompress, zstdDecompress, type ZstdOptions,
} from 'node:zlib'
import { promisify } from 'node:util'
import { NodePrivateZstdFrameDecoder } from './zstd-private-decoder.ts'
import { PublicZstdFrameDecoder } from './zstd-public-decoder.ts'
const ZSTD_MAGIC = 0xFD2FB528
const zstdCompressAsync = promisify(zstdCompress)
const zstdDecompressAsync = promisify(zstdDecompress)
const CHECKSUM_OPTIONS: ZstdOptions = {
params: { [constants.ZSTD_c_checksumFlag]: 1 },
}
const INCOMPLETE_FRAME_OPTIONS: ZstdOptions = {
finishFlush: constants.ZSTD_e_flush,
}
/** Byte range occupied by one structurally complete Zstandard frame. */
export interface ZstdFrameRange {
/** Inclusive frame start. */
start: number
/** Exclusive frame end. */
end: number
}
/** Structural scan result for a concatenated Zstandard stream. */
export interface ZstdFrameScan {
/** Complete frames in file order. */
frames: ZstdFrameRange[]
/** Start of an incomplete final frame, when EOF interrupts one. */
tornStart?: number
}
/**
* Locate complete frames without decompressing their blocks. Invalid complete
* structure rejects; EOF inside the final frame returns its start for repair.
* @param buffer - complete bytes currently present in the session artifact.
* @param maxFrames - optional complete-frame limit for metadata-only readers.
* @returns complete frame ranges and an optional incomplete-final-frame start.
*/
export function scanZstdFrames(buffer: Buffer, maxFrames = Number.POSITIVE_INFINITY): ZstdFrameScan {
const frames: ZstdFrameRange[] = []
let offset = 0
while (offset < buffer.length) {
const start = offset
if (buffer.length - offset < 4) return { frames, tornStart: start }
if (buffer.readUInt32LE(offset) !== ZSTD_MAGIC) {
throw new Error(`corrupt Zstandard session log: invalid frame magic at byte ${offset}`)
}
offset += 4
if (offset === buffer.length) return { frames, tornStart: start }
const descriptor = buffer.readUInt8(offset)
offset += 1
if ((descriptor & 0x18) !== 0) {
throw new Error(`corrupt Zstandard session log: reserved frame-header bit at byte ${offset - 1}`)
}
const contentSizeFlag = descriptor >>> 6
const singleSegment = (descriptor & 0x20) !== 0
const checksum = (descriptor & 0x04) !== 0
const dictionaryFlag = descriptor & 0x03
const dictionaryBytes = dictionaryFlag === 3 ? 4 : dictionaryFlag
const contentSizeBytes = contentSizeFlag === 0
? (singleSegment ? 1 : 0)
: 1 << contentSizeFlag
const remainingHeaderBytes = (singleSegment ? 0 : 1) + dictionaryBytes + contentSizeBytes
if (buffer.length - offset < remainingHeaderBytes) return { frames, tornStart: start }
offset += remainingHeaderBytes
for (;;) {
if (buffer.length - offset < 3) return { frames, tornStart: start }
const blockHeader = buffer.readUIntLE(offset, 3)
offset += 3
const lastBlock = (blockHeader & 1) !== 0
const blockType = (blockHeader >>> 1) & 0x03
const blockSize = blockHeader >>> 3
if (blockType === 0x03) {
throw new Error(`corrupt Zstandard session log: reserved block type at byte ${offset - 3}`)
}
const payloadBytes = blockType === 0x01 ? 1 : blockSize
if (buffer.length - offset < payloadBytes) return { frames, tornStart: start }
offset += payloadBytes
if (lastBlock) break
}
if (checksum) {
if (buffer.length - offset < 4) return { frames, tornStart: start }
offset += 4
}
frames.push({ start, end: offset })
if (frames.length === maxFrames) return { frames }
}
return { frames }
}
/**
* Compress one independently decodable, checksummed Zstandard frame.
* @param input - JSONL bytes for a header or durable event batch.
* @returns the complete encoded frame.
*/
export async function compressZstdFrame(input: Buffer | string): Promise<Buffer> {
return zstdCompressAsync(input, CHECKSUM_OPTIONS)
}
/**
* Decompress one complete frame and validate its checksum.
* @param input - one structurally complete Zstandard frame.
* @returns the frame plaintext.
*/
export async function decompressZstdFrame(input: Buffer): Promise<Buffer> {
return zstdDecompressAsync(input)
}
/** Common lifecycle for interchangeable synchronous multi-frame decoders. */
export interface ZstdFrameDecoder {
/**
* Decode and checksum complete frames in source order. Each yielded buffer
* remains valid only until the iterator advances to the next frame.
* @param source - concatenated Zstandard frame bytes.
* @param frames - structurally complete ranges within `source`.
* @returns one plaintext buffer per frame.
*/
decode(source: Buffer, frames: readonly ZstdFrameRange[]): Generator<Buffer, void, void>
/** Release decoder-owned resources; repeated calls are harmless. */
close(): void
}
/**
* Select the shared private decoder when the running Node 22/24/26 shape is
* compatible, otherwise preserve correctness with the public one-shot API.
* @returns a synchronous decoder with an implementation-independent lifecycle.
*/
export function createZstdFrameDecoder(): ZstdFrameDecoder {
return NodePrivateZstdFrameDecoder.create() ?? new PublicZstdFrameDecoder()
}
/**
* Recover available plaintext from a structurally incomplete final frame.
* `ZSTD_e_flush` deliberately suppresses final-frame and checksum completion;
* callers must establish the torn frame boundary before using this helper.
* @param input - available bytes from a known incomplete Zstandard frame.
* @returns plaintext produced from the available input.
*/
export async function decompressZstdPrefix(input: Buffer): Promise<Buffer> {
return zstdDecompressAsync(input, INCOMPLETE_FRAME_OPTIONS)
}

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,178 @@
/**
* Unit tests for the Windows durable namespace helper with a mocked kernel32
* binding. The real JSONL suite exercises the helper on native Windows; these
* tests keep the Win32 error mapping and race handling covered on every host.
*/
import { afterEach, describe, expect, it, vi } from 'vitest'
import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from 'node:fs'
import { mkdtemp, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
const MOVEFILE_WRITE_THROUGH = 0x00000008
const ERROR_FILE_NOT_FOUND = 2
const ERROR_PATH_NOT_FOUND = 3
const ERROR_ACCESS_DENIED = 5
const ERROR_NOT_SAME_DEVICE = 17
const ERROR_FILE_EXISTS = 80
const ERROR_INVALID_NAME = 123
const ERROR_ALREADY_EXISTS = 183
type MoveFileExW = (existing: string, replacement: string, flags: number, setLastError: (code: number) => void) => number
const roots: string[] = []
function stripNamespace(path: string): string {
if (path.startsWith('\\\\?\\UNC\\')) return `\\\\${path.slice('\\\\?\\UNC\\'.length)}`
if (path.startsWith('\\\\?\\')) return path.slice('\\\\?\\'.length)
return path
}
async function tempRoot(): Promise<string> {
const dir = await mkdtemp(join(tmpdir(), 'dsh-jsonl-win32-'))
roots.push(dir)
return dir
}
async function importWithMove(moveFileExW: MoveFileExW): Promise<typeof import('../src/win32.ts')> {
vi.resetModules()
vi.doMock('koffi', () => {
let lastError = 0
const setLastError = (code: number): void => { lastError = code }
const move: MoveFileExW = (existing, replacement, flags, setError) => {
const ok = moveFileExW(existing, replacement, flags, setError)
lastError = ok === 0 ? lastError : 0
return ok
}
return {
default: {
load: () => ({
func: (_convention: string, name: string, result: string) => {
if (name === 'MoveFileExW') return (existing: string, replacement: string, flags: number) => {
expect(result).toBe('int')
const ok = move(existing, replacement, flags, setLastError)
return ok
}
return () => lastError
},
}),
},
}
})
return import('../src/win32.ts')
}
async function importWithError(code: number): Promise<typeof import('../src/win32.ts')> {
vi.resetModules()
vi.doMock('koffi', () => ({
default: {
load: () => ({
func: (_convention: string, name: string) => {
if (name === 'MoveFileExW') return () => 0
return () => code
},
}),
},
}))
return import('../src/win32.ts')
}
async function importWithFilesystemMove(): Promise<typeof import('../src/win32.ts')> {
return importWithMove((existing, replacement, flags, setLastError) => {
expect(flags).toBe(MOVEFILE_WRITE_THROUGH)
const from = stripNamespace(existing)
const to = stripNamespace(replacement)
if (!existsSync(from)) { setLastError(ERROR_FILE_NOT_FOUND); return 0 }
if (existsSync(to)) { setLastError(ERROR_ALREADY_EXISTS); return 0 }
renameSync(from, to)
return 1
})
}
afterEach(async () => {
vi.doUnmock('koffi')
vi.resetModules()
for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true })
})
describe('Windows durable namespace helpers', () => {
it('publishes a new file with write-through MoveFileExW semantics', async () => {
const { publishNewFileWin32 } = await importWithFilesystemMove()
const root = await tempRoot()
const tmp = join(root, 'log.tmp')
const final = join(root, 'log.jsonl')
await writeFile(tmp, 'content')
await publishNewFileWin32(tmp, final)
expect(existsSync(tmp)).toBe(false)
expect(readFileSync(final, 'utf8')).toBe('content')
})
it('maps Win32 publish failures to Node-style errno codes', async () => {
const cases = [
[ERROR_FILE_NOT_FOUND, 'ENOENT'],
[ERROR_PATH_NOT_FOUND, 'ENOENT'],
[ERROR_ACCESS_DENIED, 'EACCES'],
[ERROR_NOT_SAME_DEVICE, 'EXDEV'],
[ERROR_FILE_EXISTS, 'EEXIST'],
[ERROR_ALREADY_EXISTS, 'EEXIST'],
[ERROR_INVALID_NAME, 'EINVAL'],
[9999, 'EIO'],
] as const
for (const [win32Code, code] of cases) {
const { publishNewFileWin32 } = await importWithError(win32Code)
await expect(publishNewFileWin32('from', 'to')).rejects.toMatchObject({ code, win32Code, path: 'from', dest: 'to' })
}
})
it('creates missing directories through staging siblings and tolerates an already-created race', async () => {
const root = await tempRoot()
const raced = join(root, 'raced')
const { ensureDurableDirectoryWin32 } = await importWithMove((existing, replacement, flags, setLastError) => {
expect(flags).toBe(MOVEFILE_WRITE_THROUGH)
const from = stripNamespace(existing)
const to = stripNamespace(replacement)
if (to === raced) {
mkdirSync(to)
setLastError(ERROR_ALREADY_EXISTS)
return 0
}
if (!existsSync(from)) { setLastError(ERROR_FILE_NOT_FOUND); return 0 }
if (existsSync(to)) { setLastError(ERROR_ALREADY_EXISTS); return 0 }
renameSync(from, to)
return 1
})
await ensureDurableDirectoryWin32(join(root, 'a', 'b'))
expect(existsSync(join(root, 'a', 'b'))).toBe(true)
await ensureDurableDirectoryWin32(join(root, 'a', 'b'))
await ensureDurableDirectoryWin32(raced)
expect(existsSync(raced)).toBe(true)
})
it('keeps staging names valid for a maximum-length target component', async () => {
const { ensureDurableDirectoryWin32 } = await importWithFilesystemMove()
const root = await tempRoot()
const target = join(root, 'x'.repeat(255))
await ensureDurableDirectoryWin32(target)
expect(existsSync(target)).toBe(true)
})
it('surfaces directory publication failures other than an existing-target race', async () => {
const { ensureDurableDirectoryWin32 } = await importWithError(ERROR_ACCESS_DENIED)
const root = await tempRoot()
await expect(ensureDurableDirectoryWin32(join(root, 'denied'))).rejects.toMatchObject({ code: 'EACCES' })
})
it('rejects a non-directory component instead of treating it as missing', async () => {
const { ensureDurableDirectoryWin32 } = await importWithFilesystemMove()
const root = await tempRoot()
const blocked = join(root, 'blocked')
writeFileSync(blocked, 'x')
await expect(ensureDurableDirectoryWin32(join(blocked, 'child'))).rejects.toMatchObject({ code: 'ENOTDIR' })
})
})

View File

@@ -0,0 +1,39 @@
import { describe, expect, it } from 'vitest'
import {
compressZstdFrame, createZstdFrameDecoder, decompressZstdFrame, decompressZstdPrefix, scanZstdFrames,
} from '../src/zstd.ts'
import { NodePrivateZstdFrameDecoder } from '../src/zstd-private-decoder.ts'
import { PublicZstdFrameDecoder } from '../src/zstd-public-decoder.ts'
describe('JSONL Zstandard compatibility', () => {
it('round-trips concatenated checksummed frames through the built-in Node API', async () => {
const encoded = Buffer.concat([
await compressZstdFrame('{"type":"session","version":0,"id":"compat","createdAt":1}\n'),
await compressZstdFrame('{"type":"turn/start","seq":0,"turn":1}\n'),
])
const { frames, tornStart } = scanZstdFrames(encoded)
expect(tornStart).toBeUndefined()
expect(frames).toHaveLength(2)
expect(frames.map(frame => encoded.subarray(frame.start, frame.start + 4).toString('hex')))
.toEqual(['28b52ffd', '28b52ffd'])
const decoded = await Promise.all(frames.map(frame => decompressZstdFrame(encoded.subarray(frame.start, frame.end))))
expect(Buffer.concat(decoded).toString()).toContain('"type":"turn/start"')
const preferred = createZstdFrameDecoder()
expect(preferred).toBeInstanceOf(NodePrivateZstdFrameDecoder)
for (const decoder of [preferred, new PublicZstdFrameDecoder()]) {
try {
const plaintext = Array.from(decoder.decode(encoded, frames), chunk => Buffer.from(chunk))
expect(Buffer.concat(plaintext).toString()).toContain('"type":"turn/start"')
} finally {
decoder.close()
}
}
const eventFrame = encoded.subarray(frames[1]!.start, frames[1]!.end)
const missingChecksumByte = eventFrame.subarray(0, -1)
expect(scanZstdFrames(missingChecksumByte)).toEqual({ frames: [], tornStart: 0 })
expect((await decompressZstdPrefix(missingChecksumByte)).toString()).toContain('"type":"turn/start"')
})
})

View File

@@ -0,0 +1,717 @@
import { afterEach, describe, expect, it, vi } from 'vitest'
import { Context } from 'cordis'
import { appendFile, mkdir, mkdtemp, open, readFile, readdir, rm, stat, writeFile } from 'node:fs/promises'
import type { FileHandle } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { performance } from 'node:perf_hooks'
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
import type { SessionEvent } from '@deepseek-ai/dsh-session'
import SessionPersistenceJsonl from '@deepseek-ai/dsh-session-persistence-jsonl'
import { logPath, scanLog, sessionDir, toHeaderLine, type JsonlCompression } from '../src/format.ts'
import {
compressZstdFrame, createZstdFrameDecoder, decompressZstdFrame, decompressZstdPrefix, scanZstdFrames,
type ZstdFrameDecoder,
} from '../src/zstd.ts'
import { NodePrivateZstdFrameDecoder } from '../src/zstd-private-decoder.ts'
import { PublicZstdFrameDecoder } from '../src/zstd-public-decoder.ts'
import { runPersistenceContract, meta, oneTurnLog } from '../../session-persistence/tests/contract.ts'
import { runCoordinatorContract, type CoordinatorFixture } from '../../session-persistence/tests/coordinator-contract.ts'
const MAGIC = Buffer.from([0x28, 0xB5, 0x2F, 0xFD])
const roots: string[] = []
const contexts: Context[] = []
interface ZstdReaderInternals {
readZstdPrefix(buffer: Buffer, signal?: AbortSignal): Promise<{ events: SessionEvent[] }>
}
type HeaderRead = (
this: FileHandle,
buffer: Buffer,
offset: number,
length: number,
position: number | null,
) => Promise<{ bytesRead: number; buffer: Buffer }>
async function freshRoot(prefix = 'dsh-jsonl-zstd-'): Promise<string> {
const root = await mkdtemp(join(tmpdir(), prefix))
roots.push(root)
return root
}
async function mount(root: string, compression?: JsonlCompression): Promise<Context> {
const ctx = new Context()
contexts.push(ctx)
await ctx.plugin(SessionStore)
await ctx.plugin(SessionPersistenceJsonl, {
root,
...(compression === undefined ? {} : { compression }),
})
return ctx
}
async function decodeCompleteFrames(buffer: Buffer): Promise<Buffer> {
const { frames, tornStart } = scanZstdFrames(buffer)
expect(tornStart).toBeUndefined()
const plaintext: Buffer[] = []
for (const frame of frames) {
plaintext.push(await decompressZstdFrame(buffer.subarray(frame.start, frame.end)))
}
return Buffer.concat(plaintext)
}
async function tornFrame(
plaintext: string,
accepts: (decoded: string) => boolean,
): Promise<Buffer> {
const frame = await compressZstdFrame(plaintext)
const candidateEnds = [
frame.length - 1,
frame.length - 4,
...[0.9, 0.75, 0.6, 0.5, 0.4, 0.25].map(ratio => Math.floor(frame.length * ratio)),
]
for (const end of candidateEnds) {
const candidate = frame.subarray(0, end)
if (scanZstdFrames(candidate).tornStart !== 0) continue
try {
const decoded = (await decompressZstdPrefix(candidate)).toString('utf8')
if (accepts(decoded)) return candidate
} catch {
// Some early cuts precede the first decodable block; keep searching for
// a cut that exercises partial-plaintext recovery.
}
}
throw new Error('test fixture could not produce the requested torn Zstandard frame')
}
function deterministicNoise(length: number): string {
let state = 0x12345678
let output = ''
for (let index = 0; index < length; index++) {
state = (Math.imul(state, 1_664_525) + 1_013_904_223) >>> 0
output += String.fromCharCode(33 + (state % 90))
}
return output
}
function emptyStructuralFrame(descriptor: number): Buffer {
const contentSizeFlag = descriptor >>> 6
const singleSegment = (descriptor & 0x20) !== 0
const dictionaryBytes = [0, 1, 2, 4][descriptor & 0x03]!
const contentSizeBytes = contentSizeFlag === 0 ? (singleSegment ? 1 : 0) : 1 << contentSizeFlag
const variableHeader = Buffer.alloc((singleSegment ? 0 : 1) + dictionaryBytes + contentSizeBytes)
const lastEmptyRawBlock = Buffer.from([1, 0, 0])
const checksum = (descriptor & 0x04) === 0 ? Buffer.alloc(0) : Buffer.alloc(4)
return Buffer.concat([MAGIC, Buffer.from([descriptor]), variableHeader, lastEmptyRawBlock, checksum])
}
afterEach(async () => {
vi.restoreAllMocks()
for (const ctx of contexts.splice(0).reverse()) await ctx.fiber.dispose()
for (const root of roots.splice(0)) await rm(root, { recursive: true, force: true })
})
runPersistenceContract('jsonl-zstd', async () => {
const root = await mkdtemp(join(tmpdir(), 'dsh-jsonl-zstd-contract-'))
const ctx = new Context()
await ctx.plugin(SessionStore)
const fiber = await ctx.plugin(SessionPersistenceJsonl, { root })
return {
persistence: ctx.sessionPersistence,
dispose: async () => {
await fiber.dispose()
await rm(root, { recursive: true, force: true })
},
}
})
runCoordinatorContract('jsonl-zstd', async (): Promise<CoordinatorFixture> => {
const root = await mkdtemp(join(tmpdir(), 'dsh-jsonl-zstd-coordinator-'))
return {
mount: async ctx => ctx.plugin(SessionPersistenceJsonl, { root }),
corruptTail: async (id, cwd) => {
const line = JSON.stringify({
type: 'assistant/chunk',
seq: 8,
time: 9,
data: { turn: 2, step: 1, chunk: { type: 'text-delta', index: 0, text: deterministicNoise(300_000) } },
}) + '\n'
const partial = await tornFrame(line, decoded => decoded.length > 0 && !decoded.endsWith('\n'))
await appendFile(logPath(root, cwd, id, 'zstd'), partial)
},
cleanup: async () => { await rm(root, { recursive: true, force: true }) },
}
})
describe('Zstandard frame structure', () => {
it('scans concatenated checksummed frames and honors a frame limit', async () => {
const first = await compressZstdFrame('header\n')
const second = await compressZstdFrame('event\n')
const stream = Buffer.concat([first, second])
expect(scanZstdFrames(Buffer.alloc(0))).toEqual({ frames: [] })
expect(scanZstdFrames(stream)).toEqual({
frames: [{ start: 0, end: first.length }, { start: first.length, end: stream.length }],
})
expect(scanZstdFrames(stream, 1)).toEqual({ frames: [{ start: 0, end: first.length }] })
expect(first[4]! & 0x04).toBe(0x04)
expect(second[4]! & 0x04).toBe(0x04)
expect((await decompressZstdFrame(first)).toString()).toBe('header\n')
const decoder = createZstdFrameDecoder()
try {
const plaintext = Array.from(decoder.decode(stream, scanZstdFrames(stream).frames), chunk => Buffer.from(chunk))
expect(Buffer.concat(plaintext).toString()).toBe('header\nevent\n')
} finally {
decoder.close()
}
})
it('keeps the public and Node-private synchronous decoders interchangeable', async () => {
const frames = [await compressZstdFrame('first\n'), await compressZstdFrame('second\n')]
const stream = Buffer.concat(frames)
const ranges = scanZstdFrames(stream).frames
const privateDecoder = NodePrivateZstdFrameDecoder.create()
expect(privateDecoder).toBeDefined()
for (const decoder of [new PublicZstdFrameDecoder(), privateDecoder!]) {
try {
const plaintext = Array.from(decoder.decode(stream, ranges), chunk => Buffer.from(chunk))
expect(plaintext).toHaveLength(2)
expect(Buffer.concat(plaintext).toString()).toBe('first\nsecond\n')
} finally {
decoder.close()
}
}
})
it('falls back to the public decoder when the private Node contract is unavailable', () => {
vi.spyOn(NodePrivateZstdFrameDecoder, 'create').mockReturnValue(undefined)
const decoder = createZstdFrameDecoder()
expect(decoder).toBeInstanceOf(PublicZstdFrameDecoder)
decoder.close()
})
it('enforces decoder lifecycle and checksum errors through both implementations', async () => {
const frame = await compressZstdFrame('frame\n')
const range = [{ start: 0, end: frame.length }]
const corrupt = Buffer.from(frame)
corrupt[corrupt.length - 1] = corrupt[corrupt.length - 1]! ^ 0xFF
const factories: Array<() => ZstdFrameDecoder> = [
() => new PublicZstdFrameDecoder(),
() => NodePrivateZstdFrameDecoder.create()!,
]
for (const create of factories) {
const interrupted = create()
const iterator = interrupted.decode(frame, range)
expect(iterator.next().value?.toString()).toBe('frame\n')
iterator.return()
expect(() => Array.from(interrupted.decode(frame, range))).toThrow(/already started/)
interrupted.close()
const closed = create()
closed.close()
closed.close()
expect(() => Array.from(closed.decode(frame, range))).toThrow(/closed/)
const invalid = create()
expect(() => Array.from(invalid.decode(corrupt, range))).toThrow(/frame at byte 0 failed validation/)
}
})
it('assembles private-decoder output at and beyond its reusable chunk boundary', async () => {
for (const length of [8, 9]) {
const plaintext = Buffer.alloc(length, 0x61)
const frame = await compressZstdFrame(plaintext)
const decoder = NodePrivateZstdFrameDecoder.create()!
;(decoder as unknown as { output: Buffer }).output = Buffer.allocUnsafe(8)
const [decoded] = Array.from(
decoder.decode(frame, [{ start: 0, end: frame.length }]),
chunk => Buffer.from(chunk),
)
expect(decoded).toEqual(plaintext)
}
})
it('normalizes private decoder stream failures', async () => {
interface PrivateDecoderInternals {
stream: {
[key: symbol]: unknown
emit(event: string, error: Error): boolean
}
errorKey: symbol
}
const frame = await compressZstdFrame('frame\n')
const range = [{ start: 0, end: frame.length }]
const emitted = NodePrivateZstdFrameDecoder.create()!
const emittedInternals = emitted as unknown as PrivateDecoderInternals
const first = new Error('first emitted decoder failure')
emittedInternals.stream.emit('error', first)
emittedInternals.stream.emit('error', new Error('later emitted decoder failure'))
try {
Array.from(emitted.decode(frame, range))
throw new Error('expected emitted decoder failure')
} catch (error) {
expect((error as Error).cause).toBe(first)
}
for (const internalFailure of [new Error('internal decoder failure'), 'not an Error']) {
const decoder = NodePrivateZstdFrameDecoder.create()!
const internals = decoder as unknown as PrivateDecoderInternals
internals.stream[internals.errorKey] = internalFailure
try {
Array.from(decoder.decode(frame, range))
throw new Error('expected internal decoder failure')
} catch (error) {
const cause = (error as Error).cause
if (internalFailure instanceof Error) {
expect(cause).toBe(internalFailure)
} else {
expect(cause).toMatchObject({ message: 'Zstandard decoder exposed a non-Error internal failure' })
}
}
}
})
it('distinguishes incomplete frame regions from invalid complete structure', () => {
expect(scanZstdFrames(MAGIC.subarray(0, 2))).toEqual({ frames: [], tornStart: 0 })
expect(scanZstdFrames(MAGIC)).toEqual({ frames: [], tornStart: 0 })
expect(() => scanZstdFrames(Buffer.alloc(4))).toThrow(/invalid frame magic/)
expect(() => scanZstdFrames(Buffer.concat([MAGIC, Buffer.from([0x08])]))).toThrow(/reserved frame-header bit/)
// Non-single-segment descriptor with no window descriptor.
expect(scanZstdFrames(Buffer.concat([MAGIC, Buffer.from([0x00])]))).toEqual({ frames: [], tornStart: 0 })
// Single-segment header followed by only two bytes of the three-byte block header.
expect(scanZstdFrames(Buffer.concat([MAGIC, Buffer.from([0x20, 0x00, 0x01, 0x00])]))).toEqual({
frames: [],
tornStart: 0,
})
const rawFiveBytes = Buffer.from([(5 << 3) | 1, 0, 0])
expect(scanZstdFrames(Buffer.concat([
MAGIC,
Buffer.from([0x20, 0x00]),
rawFiveBytes,
Buffer.from([0x01, 0x02]),
]))).toEqual({ frames: [], tornStart: 0 })
const reservedBlock = Buffer.concat([
MAGIC,
Buffer.from([0x20, 0x00, 0x07, 0x00, 0x00]),
])
expect(() => scanZstdFrames(reservedBlock)).toThrow(/reserved block type/)
})
it('covers standard header variants, RLE blocks, multiple blocks, and checksums', () => {
for (const descriptor of [0x00, 0x21, 0x42, 0x83, 0xE3]) {
const frame = emptyStructuralFrame(descriptor)
expect(scanZstdFrames(frame)).toEqual({ frames: [{ start: 0, end: frame.length }] })
}
const rle = Buffer.concat([
MAGIC,
Buffer.from([0x20, 0x01]),
Buffer.from([(1 << 3) | (1 << 1) | 1, 0, 0]),
Buffer.from([0x41]),
])
expect(scanZstdFrames(rle)).toEqual({ frames: [{ start: 0, end: rle.length }] })
const twoBlocks = Buffer.concat([
MAGIC,
Buffer.from([0x20, 0x00]),
Buffer.from([0, 0, 0]),
Buffer.from([1, 0, 0]),
])
expect(scanZstdFrames(twoBlocks)).toEqual({ frames: [{ start: 0, end: twoBlocks.length }] })
const checksummed = emptyStructuralFrame(0x24)
expect(scanZstdFrames(checksummed.subarray(0, -1))).toEqual({ frames: [], tornStart: 0 })
expect(scanZstdFrames(checksummed)).toEqual({ frames: [{ start: 0, end: checksummed.length }] })
})
})
describe('SessionPersistenceJsonl: default Zstandard encoding', () => {
it('writes .jsonl.zstd by default with one header frame and one first-batch frame', async () => {
const root = await freshRoot()
const ctx = await mount(root)
const header = meta('default-zstd', '/work')
await ctx.sessionPersistence.create(header)
await ctx.sessionPersistence.append(header.id, oneTurnLog())
const path = logPath(root, header.cwd, header.id, 'zstd')
const buffer = await readFile(path)
expect(buffer.subarray(0, 4)).toEqual(MAGIC)
await expect(stat(logPath(root, header.cwd, header.id, 'none'))).rejects.toThrow()
expect(ctx.sessionPersistence.locate(header)).toEqual({ kind: 'jsonl', path })
const scan = scanZstdFrames(buffer)
expect(scan.frames).toHaveLength(2)
const plaintext = await decodeCompleteFrames(buffer)
expect(plaintext.toString()).toBe([
JSON.stringify(toHeaderLine(header)),
...oneTurnLog().map(e => JSON.stringify(e)),
'',
].join('\n'))
expect((await ctx.sessionPersistence.load(header.id)).events).toEqual(oneTurnLog())
})
it('resolves the default when a programmatic wrapper bypasses Loader schema normalization', async () => {
const root = await freshRoot()
const ctx = new Context()
contexts.push(ctx)
await ctx.plugin(SessionStore)
let backend!: SessionPersistenceJsonl
await ctx.plugin(Object.assign((inner: Context) => {
backend = new SessionPersistenceJsonl(inner, { root })
}, { inject: ['sessions'] }))
const header = meta('direct-default')
const path = logPath(root, header.cwd, header.id, 'zstd')
expect(backend.locate(header)).toEqual({
kind: 'jsonl',
path,
})
const base = oneTurnLog()
const events: SessionEvent[] = [
...base.slice(0, 3),
...Array.from({ length: 3 }, (_, index): SessionEvent => ({
type: 'assistant/chunk',
seq: 3 + index,
time: 4 + index,
data: { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: `part-${index}` } },
})),
...base.slice(3).map((event): SessionEvent => ({
...event,
seq: event.seq + 3,
time: event.time + 3,
})),
]
await backend.create(header)
await backend.append(header.id, events)
const plaintext = (await decodeCompleteFrames(await readFile(path))).toString()
const recordTypes = plaintext.trimEnd().split('\n')
.map(line => (JSON.parse(line) as { type: string }).type)
expect(recordTypes).toContain('text-chunks')
expect((await backend.load(header.id)).events).toEqual(events)
})
it('appends one frame per durable batch without rewriting prior bytes', async () => {
const root = await freshRoot()
const ctx = await mount(root)
const header = meta('append-frame')
await ctx.sessionPersistence.create(header)
await ctx.sessionPersistence.append(header.id, oneTurnLog())
const path = logPath(root, header.cwd, header.id, 'zstd')
const before = await readFile(path)
const secondTurn = [
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2 } },
{ type: 'turn/end', seq: 7, time: 8, data: { turn: 2, reason: { kind: 'completed' } } },
] as SessionEvent[]
await ctx.sessionPersistence.append(header.id, secondTurn)
const after = await readFile(path)
expect(after.subarray(0, before.length)).toEqual(before)
expect(scanZstdFrames(after).frames).toHaveLength(3)
expect((await ctx.sessionPersistence.load(header.id)).events).toEqual([...oneTurnLog(), ...secondTurn])
})
it('lists from a multi-chunk header frame without decoding a corrupt event frame', async () => {
const root = await freshRoot()
const ctx = await mount(root)
const header = meta('large-header', `/work/${'x'.repeat(24_000)}`)
await ctx.sessionPersistence.create(header)
await ctx.sessionPersistence.append(header.id, oneTurnLog())
const path = logPath(root, header.cwd, header.id, 'zstd')
const buffer = Buffer.from(await readFile(path))
const eventFrame = scanZstdFrames(buffer).frames[1]!
buffer[eventFrame.end - 1] = buffer[eventFrame.end - 1]! ^ 0xFF
await writeFile(path, buffer)
expect((await ctx.sessionPersistence.list()).map(item => item.id)).toEqual([header.id])
await expect(ctx.sessionPersistence.load(header.id)).rejects.toThrow(/frame at byte .* failed validation/)
})
it('stops multi-frame inspection when cancellation arrives at a slice deadline', async () => {
const root = await freshRoot()
const ctx = await mount(root)
const header = meta('cancel-zstd-frames')
const headerFrame = await compressZstdFrame(`${JSON.stringify(toHeaderLine(header))}\n`)
const eventFrame = await compressZstdFrame(`${JSON.stringify(oneTurnLog()[0])}\n`)
const laterFrame = await compressZstdFrame(`${JSON.stringify(oneTurnLog()[1])}\n`)
const stream = Buffer.concat([headerFrame, eventFrame, laterFrame])
const controller = new AbortController()
const reason = new Error('cancel after Zstandard decode starts')
const reader = ctx.sessionPersistence as unknown as ZstdReaderInternals
vi.spyOn(performance, 'now').mockReturnValueOnce(0).mockReturnValue(501)
const pending = reader.readZstdPrefix(stream, controller.signal)
queueMicrotask(() => { controller.abort(reason) })
await expect(pending).rejects.toBe(reason)
})
it('continues decoding every frame after a slice deadline yields', async () => {
const root = await freshRoot()
const ctx = await mount(root)
const header = meta('yield-zstd-frames')
const events = oneTurnLog().slice(0, 2)
const headerFrame = await compressZstdFrame(`${JSON.stringify(toHeaderLine(header))}\n`)
const eventFrames = await Promise.all(events.map(async event => (
compressZstdFrame(`${JSON.stringify(event)}\n`)
)))
const stream = Buffer.concat([headerFrame, ...eventFrames])
const reader = ctx.sessionPersistence as unknown as ZstdReaderInternals
vi.spyOn(performance, 'now').mockReturnValueOnce(0).mockReturnValue(501)
const prefix = await reader.readZstdPrefix(stream)
expect(prefix.events).toEqual(events)
})
it.each(['none', 'zstd'] as const)(
'observes cancellation after each async %s header read during listing',
async (compression) => {
const root = await freshRoot()
const ctx = await mount(root, compression)
const header = meta(`cancel-${compression}-header-read`, '/work')
await ctx.sessionPersistence.create(header)
await ctx.sessionPersistence.append(header.id, oneTurnLog())
await ctx.sessionPersistence.list()
const path = logPath(root, header.cwd, header.id, compression)
const probe = await open(path, 'r')
const prototype = Object.getPrototypeOf(probe) as { read: HeaderRead }
const originalRead = prototype.read
await probe.close()
const controller = new AbortController()
const reason = new Error(`cancel ${compression} header read`)
const read = vi.spyOn(prototype, 'read').mockImplementation(async function (
this: FileHandle,
buffer: Buffer,
offset: number,
length: number,
position: number | null,
) {
const result = await originalRead.call(this, buffer, offset, length, position)
controller.abort(reason)
return result
})
await expect(ctx.sessionPersistence.list(controller.signal)).rejects.toBe(reason)
expect(read).toHaveBeenCalledTimes(1)
},
)
it('preserves complete records from a torn frame and re-encodes them with crash closers', async () => {
const root = await freshRoot()
const ctx = await mount(root)
const header = meta('recover-torn', '/proj')
await ctx.sessionPersistence.create(header)
await ctx.sessionPersistence.append(header.id, oneTurnLog())
const path = logPath(root, header.cwd, header.id, 'zstd')
const committed = await readFile(path)
const openTurn = [
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2 } },
{ type: 'step/start', seq: 7, time: 8, data: { turn: 2, step: 1 } },
{ type: 'assistant/chunk', seq: 8, time: 9, data: { turn: 2, step: 1, chunk: { type: 'text-delta', index: 0, text: deterministicNoise(300_000) } } },
] as SessionEvent[]
const plaintext = openTurn.map(e => JSON.stringify(e)).join('\n') + '\n'
const partial = await tornFrame(plaintext, (decoded) => {
const newlines = decoded.match(/\n/g)?.length ?? 0
return newlines >= 2 && !decoded.endsWith('\n')
})
await appendFile(path, partial)
const loaded = await ctx.sessionPersistence.load(header.id)
expect(loaded.events.map(event => event.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
expect(loaded.events[6]).toEqual(openTurn[0])
expect(loaded.events[7]).toEqual(openTurn[1])
expect(loaded.events.some(event => event.type === 'assistant/chunk' && event.seq === 8)).toBe(false)
expect(loaded.events[8]?.type).toBe('step/end')
expect(loaded.events[9]?.type).toBe('turn/end')
const repaired = await readFile(path)
expect(repaired.subarray(0, committed.length)).toEqual(committed)
expect(scanZstdFrames(repaired).tornStart).toBeUndefined()
expect(scanLog(await decodeCompleteFrames(repaired)).events).toEqual(loaded.events)
})
it('drops a frame torn in its header before it has produced plaintext', async () => {
const root = await freshRoot()
const ctx = await mount(root)
const header = meta('partial-magic')
await ctx.sessionPersistence.create(header)
await ctx.sessionPersistence.append(header.id, oneTurnLog())
const path = logPath(root, header.cwd, header.id, 'zstd')
const committed = await readFile(path)
await appendFile(path, MAGIC.subarray(0, 2))
expect((await ctx.sessionPersistence.load(header.id)).events).toEqual(oneTurnLog())
expect(await readFile(path)).toEqual(committed)
})
it('recovers complete events when EOF tears only the final frame checksum', async () => {
const root = await freshRoot()
const ctx = await mount(root)
const header = meta('partial-checksum')
await ctx.sessionPersistence.create(header)
await ctx.sessionPersistence.append(header.id, oneTurnLog())
const path = logPath(root, header.cwd, header.id, 'zstd')
const secondTurn = [
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2 } },
{ type: 'turn/end', seq: 7, time: 8, data: { turn: 2, reason: { kind: 'completed' } } },
] as SessionEvent[]
const frame = await compressZstdFrame(secondTurn.map(e => JSON.stringify(e)).join('\n') + '\n')
await appendFile(path, frame.subarray(0, -1))
const loaded = await ctx.sessionPersistence.load(header.id)
expect(loaded.events).toEqual([...oneTurnLog(), ...secondTurn])
const repaired = await readFile(path)
expect(scanZstdFrames(repaired).tornStart).toBeUndefined()
expect(scanLog(await decodeCompleteFrames(repaired)).events).toEqual(loaded.events)
})
it('rejects a complete frame containing a torn JSONL record', async () => {
const root = await freshRoot()
const ctx = await mount(root)
const header = meta('complete-bad-jsonl')
await ctx.sessionPersistence.create(header)
await ctx.sessionPersistence.append(header.id, oneTurnLog())
await appendFile(
logPath(root, header.cwd, header.id, 'zstd'),
await compressZstdFrame('{"type":"turn/start"'),
)
await expect(ctx.sessionPersistence.load(header.id)).rejects.toThrow(/complete frame contains a torn JSONL record/)
})
it('rolls back a checksummed append frame when fsync fails', async () => {
const root = await freshRoot()
const ctx = await mount(root)
const header = meta('zstd-fsync-rollback')
await ctx.sessionPersistence.create(header)
await ctx.sessionPersistence.append(header.id, oneTurnLog())
const path = logPath(root, header.cwd, header.id, 'zstd')
const before = await readFile(path)
const handle = await open(path, 'r')
const prototype = Object.getPrototypeOf(handle) as { sync: () => Promise<void> }
await handle.close()
const realSync = prototype.sync
let failed = false
const spy = vi.spyOn(prototype, 'sync').mockImplementation(async function (this: FileHandle) {
if (!failed) {
failed = true
throw new Error('simulated Zstandard fsync failure')
}
return realSync.call(this)
})
const secondTurn = [
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2 } },
{ type: 'turn/end', seq: 7, time: 8, data: { turn: 2, reason: { kind: 'completed' } } },
] as SessionEvent[]
await expect(ctx.sessionPersistence.append(header.id, secondTurn)).rejects.toThrow(/simulated Zstandard fsync failure/)
expect(await readFile(path)).toEqual(before)
spy.mockRestore()
await ctx.sessionPersistence.append(header.id, secondTurn)
expect((await ctx.sessionPersistence.load(header.id)).events).toEqual([...oneTurnLog(), ...secondTurn])
})
it('skips empty, incomplete, and non-header compressed artifacts while rejecting malformed header frames', async () => {
const root = await freshRoot()
for (const [id, content] of [
['empty', Buffer.alloc(0)],
['partial', MAGIC],
['not-header', await compressZstdFrame('{"type":"turn/start"}\n')],
] as const) {
const sessionId = SessionId(id)
await mkdir(sessionDir(root, undefined, sessionId), { recursive: true })
await writeFile(logPath(root, undefined, sessionId, 'zstd'), content)
}
const ctx = await mount(root)
expect(await ctx.sessionPersistence.list()).toEqual([])
const twoLinesId = SessionId('two-lines')
await mkdir(sessionDir(root, undefined, twoLinesId), { recursive: true })
await writeFile(logPath(root, undefined, twoLinesId, 'zstd'), await compressZstdFrame([
JSON.stringify(toHeaderLine(meta('two-lines'))),
JSON.stringify({ type: 'turn/start' }),
'',
].join('\n')))
await expect(ctx.sessionPersistence.list()).rejects.toThrow(/first frame is not exactly one header line/)
await expect(ctx.sessionPersistence.load(SessionId('two-lines')))
.rejects.toThrow(/first frame is not exactly one header line/)
})
it('rejects missing, empty, and checksum-corrupt header frames on targeted reads', async () => {
const root = await freshRoot()
for (const id of ['partial-only', 'empty-header', 'bad-checksum']) {
await mkdir(sessionDir(root, undefined, SessionId(id)), { recursive: true })
}
await writeFile(logPath(root, undefined, SessionId('partial-only'), 'zstd'), MAGIC)
await writeFile(logPath(root, undefined, SessionId('empty-header'), 'zstd'), await compressZstdFrame(''))
const corruptHeader = Buffer.from(await compressZstdFrame(`${JSON.stringify(toHeaderLine(meta('bad-checksum')))}\n`))
corruptHeader[corruptHeader.length - 1] = corruptHeader[corruptHeader.length - 1]! ^ 0xFF
await writeFile(logPath(root, undefined, SessionId('bad-checksum'), 'zstd'), corruptHeader)
const ctx = await mount(root)
await expect(ctx.sessionPersistence.load(SessionId('partial-only')))
.rejects.toThrow(/empty or header-less Zstandard session log/)
await expect(ctx.sessionPersistence.load(SessionId('empty-header')))
.rejects.toThrow(/first frame is not exactly one header line/)
await expect(ctx.sessionPersistence.list()).rejects.toThrow(/header frame failed validation/)
})
})
describe('SessionPersistenceJsonl: encoding selection', () => {
it('rejects roots owned by the opposite encoding in both directions', async () => {
const rawRoot = await freshRoot('dsh-jsonl-raw-mismatch-')
const raw = await mount(rawRoot, 'none')
const rawHeader = meta('raw-log')
await raw.sessionPersistence.create(rawHeader)
await raw.sessionPersistence.append(rawHeader.id, oneTurnLog())
const defaultBackend = await mount(rawRoot)
await expect(defaultBackend.sessionPersistence.list()).rejects.toThrow(/configured for compression "zstd"/)
const zstdRoot = await freshRoot('dsh-jsonl-zstd-mismatch-')
const zstd = await mount(zstdRoot)
const zstdHeader = meta('zstd-log')
await zstd.sessionPersistence.create(zstdHeader)
await zstd.sessionPersistence.append(zstdHeader.id, oneTurnLog())
const rawBackend = await mount(zstdRoot, 'none')
await expect(rawBackend.sessionPersistence.list()).rejects.toThrow(/configured for compression "none"/)
})
it('rechecks targeted artifacts and listing after an initially empty root', async () => {
const root = await freshRoot()
const ctx = await mount(root)
expect(await ctx.sessionPersistence.list()).toEqual([])
const loadHeader = meta('late-raw-load', '/late')
await mkdir(sessionDir(root, loadHeader.cwd, loadHeader.id), { recursive: true })
await writeFile(logPath(root, loadHeader.cwd, loadHeader.id, 'none'), [
JSON.stringify(toHeaderLine(loadHeader)),
...oneTurnLog().map(e => JSON.stringify(e)),
'',
].join('\n'))
await expect(ctx.sessionPersistence.load(loadHeader.id)).rejects.toThrow(/uses \.jsonl/)
await expect((ctx.sessionPersistence as SessionPersistenceJsonl).loadStored(loadHeader.id))
.rejects.toThrow(/uses \.jsonl/)
await expect(ctx.sessionPersistence.list()).rejects.toThrow(/uses \.jsonl/)
})
it('refuses materialization when an opposite artifact appears after create', async () => {
const root = await freshRoot()
const ctx = await mount(root)
await ctx.sessionPersistence.list()
const header = meta('late-raw-materialize', '/late')
await ctx.sessionPersistence.create(header)
await mkdir(sessionDir(root, header.cwd, header.id), { recursive: true })
await writeFile(logPath(root, header.cwd, header.id, 'none'), [
JSON.stringify(toHeaderLine(header)),
...oneTurnLog().map(e => JSON.stringify(e)),
'',
].join('\n'))
await expect(ctx.sessionPersistence.append(header.id, oneTurnLog())).rejects.toThrow(/uses \.jsonl/)
expect((await readdir(sessionDir(root, header.cwd, header.id))).some(name => name.endsWith('.jsonl.zstd'))).toBe(false)
})
})

View File

@@ -0,0 +1,30 @@
{
"extends": "../../../tsconfig.base.json",
"compilerOptions": {
"rootDir": "src",
"outDir": "lib/types"
},
"include": [
"src"
],
"references": [
{
"path": "../../../vendor/cosmokit"
},
{
"path": "../../../vendor/cordis"
},
{
"path": "../../../vendor/schemastery"
},
{
"path": "../../core/session"
},
{
"path": "../session-persistence"
},
{
"path": "../../support/invariants"
}
]
}