feat(session): default JSONL writes to packed rows

This commit is contained in:
Tianyi Cui
2026-07-26 23:44:52 +08:00
parent cad2af4741
commit 4ff496c65c
47 changed files with 521 additions and 251 deletions

View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write
README.md: a0d718cf8bd0090df0409e7c60e6f7fd559b6f7d
README.zh.md: 307bef8efb506c2df7ef229e85b3224a8e7c29e1
README.md: ab6ecd28f12bd167aeac789d1565705e167d60f4
README.zh.md: 97d387a04fa4c658217e28619410a49b7e6d4ec0

View File

@@ -15,7 +15,7 @@ The JSONL durable session-persistence backend — a concrete `SessionPersistence
```
- The first logical line is the immutable `SessionHeader` tagged `{ type: 'session', version, id, cwd?, createdAt, parentSession?, seedLength?, delegationDepth }`. `delegationDepth` is required on disk and is `0` for a top-level session; a missing or invalid value rejects the log. Every subsequent logical line is one storage record; `assistant/chunk` events are never dropped, and `seq` stays contiguous across the decoded log (`events[i].seq === i`).
- A storage record is a `SessionEvent` JSON verbatim, or — written only under `packChunks` — a **packed chunk row** (`text-chunks` / `reasoning-chunks` / `tool-call-chunks`; bare slash-less tags like the header's `session`, so row tags cannot be confused with event types): one line holding a run of ≥3 consecutive same-block `assistant/chunk` delta events, `seq0`/`time0` plus per-member `dt` gaps reconstructing every member's `seq`/`time` exactly. The lossless codec lives in `@deepseek-ai/dsh-session` (`packChunkRuns`/`decodeStorageRecord`) and whitelists exact shapes — anything unrecognized stores verbatim. Reading is layout-blind: `load` always decodes rows, so packed, unpacked, and mixed files load identically.
- A storage record is a `SessionEvent` JSON verbatim, or — for an eligible run when `packChunks` is enabled — a **packed chunk row** (`text-chunks` / `reasoning-chunks` / `tool-call-chunks`; bare slash-less tags like the header's `session`, so row tags cannot be confused with event types): one line holding a run of ≥3 consecutive same-block `assistant/chunk` delta events, `seq0`/`time0` plus per-member `dt` gaps reconstructing every member's `seq`/`time` exactly. The lossless codec lives in `@deepseek-ai/dsh-session` (`packChunkRuns`/`decodeStorageRecord`) and whitelists exact shapes — anything unrecognized stores verbatim. Reading is layout-blind: `load` always decodes rows, so packed, unpacked, and mixed files load identically.
- The project directory keeps the normalized cwd readable for navigation and is bounded for filesystem component limits. Separator replacement and truncation are intentionally lossy, so cwd strings that normalize alike share a project directory; session ids still select distinct session directories. On a case-insensitive filesystem, identity validation accepts an alternate path spelling only when filesystem canonicalization resolves both spellings to the same transcript. The configured root remains deployment-controlled: it may be project-local, shared, temporary, or centralized. The [project-session directory decision](../../../.agents/notes/implemented/architecture/2026-07-24-project-session-directories.md) records this tradeoff.
- Session ids are unvalidated branded strings, so they are injectively escaped to a single safe path segment before use (no traversal, no collision). The resulting directory is reserved for additional session-owned artifacts; discovery reads only the fixed transcript filename.
@@ -24,7 +24,7 @@ The JSONL durable session-persistence backend — a concrete `SessionPersistence
| Key | Type | Notes |
|---|---|---|
| `root` | `string` (required) | Root directory for all session files. **No default** — a `process.cwd()` default would scatter files as the process's cwd changes (bash calls, subprocesses). An existing root must be a readable directory; an absent root is created on first materialization. |
| `packChunks` | `boolean` (default `false`) | Write delta-chunk runs as packed rows (~60% smaller logical logs measured on a real coding session). Off, the written logical layout is byte-identical to the pre-packing format; reading packed rows works regardless of this switch. Off by default while the snapshot goldens stay one-event-per-line — recording with packing on rewrites every fixture `session.jsonl`. |
| `packChunks` | `boolean` (default `true`) | Write eligible delta-chunk runs as packed rows (~60% smaller logical logs measured on a real coding session). Set `false` for one-event-per-line diagnostics; reading packed rows works regardless of this write-side switch. |
| `compression` | `'zstd' \| 'none'` | Defaults to `'zstd'`; `'none'` retains newline-delimited UTF-8 text. |
`locate(meta)` returns `{ kind: 'jsonl', path }` for the fixed transcript inside the resolved project/session directories. It performs no filesystem I/O: the target can be returned before the directory or file exists, and an existing file contains only the last flushed prefix.

View File

@@ -15,7 +15,7 @@ JSONL 持久会话持久化后端:一个具体 `SessionPersistence``dsh-ses
```
- 第一个逻辑行是不可变的 `SessionHeader`,标记为 `{ type: 'session', version, id, cwd?, createdAt, parentSession?, seedLength?, delegationDepth }``delegationDepth` 在磁盘上必需,顶层会话为 `0`;缺失或无效值会拒绝日志。后续每个逻辑行是一条存储记录;`assistant/chunk` 事件绝不丢弃,且 `seq` 在解码日志中保持连续(`events[i].seq === i`)。
- 存储记录是原样 `SessionEvent` JSON`packChunks` 写入的**打包分片行**`text-chunks` / `reasoning-chunks` / `tool-call-chunks`;像 header 的 `session` 一样不带斜杠,因此行 tag 不会与事件类型混淆):一行保存至少 3 个连续同 block `assistant/chunk` delta 事件,`seq0`/`time0` 和每成员 `dt` 间隔精确重建每个成员的 `seq`/`time`。无损 codec 位于 `@deepseek-ai/dsh-session``packChunkRuns`/`decodeStorageRecord`),并使用精确形态 allowlist任何未识别内容原样存储。读取与布局无关`load` 始终解码行,因此打包、非打包和混合文件加载结果一致。
- 存储记录是原样 `SessionEvent` JSON或在 `packChunks` 已启用且连续段符合条件时写入的**打包分片行**`text-chunks` / `reasoning-chunks` / `tool-call-chunks`;像 header 的 `session` 一样不带斜杠,因此行 tag 不会与事件类型混淆):一行保存至少 3 个连续同 block `assistant/chunk` delta 事件,`seq0`/`time0` 和每成员 `dt` 间隔精确重建每个成员的 `seq`/`time`。无损 codec 位于 `@deepseek-ai/dsh-session``packChunkRuns`/`decodeStorageRecord`),并使用精确形态 allowlist任何未识别内容原样存储。读取与布局无关`load` 始终解码行,因此打包、非打包和混合文件加载结果一致。
- 项目目录保留规范化 cwd 可读,并限制在文件系统组件上限内。分隔符替换和截断刻意有损,因此规范化相同的 cwd 字符串共享项目目录;会话 id 仍选择不同会话目录。在不区分大小写的文件系统上,只有文件系统规范化将两种写法解析到同一 transcript 时,身份验证才接受备选路径写法。配置根仍由部署控制:可以是项目本地、共享、临时或集中式。[项目会话目录决策](../../../.agents/notes/implemented/architecture/2026-07-24-project-session-directories.md) 记录这项取舍。
- 会话 id 是未验证的品牌化字符串,因此在使用前单射转义为一个安全路径段(无遍历、无冲突)。结果目录保留给其他会话自有产物;发现只读取固定 transcript 文件名。
@@ -24,7 +24,7 @@ JSONL 持久会话持久化后端:一个具体 `SessionPersistence``dsh-ses
| 键 | 类型 | 说明 |
|---|---|---|
| `root` | `string` (required) | 所有会话文件的根目录。**无默认值**`process.cwd()` 默认值会随进程 cwd 变更bash 调用、子进程)而分散文件。现有根必须是可读目录;缺失根在第一次实体化时创建。 |
| `packChunks` | `boolean` (default `false`) | 将 delta 分片运行写为打包行(在真实编码会话上测得逻辑日志约小 60%)。关闭时,写入逻辑布局与打包前格式字节相同;无论开关如何,都能读取打包行。快照预期输出仍是每事件一行时默认关闭:开启打包记录会重写每个 fixture `session.jsonl`。 |
| `packChunks` | `boolean` (default `true`) | 将符合条件的 delta 分片连续段写为打包行(在真实编码会话上测得逻辑日志约小 60%)。设为 `false` 可用于每事件一行诊断;无论该写入侧开关如何,都能读取打包行。 |
| `compression` | `'zstd' \| 'none'` | 默认 `'zstd'``'none'` 保留换行分隔 UTF-8 文本。 |
`locate(meta)` 返回已解析项目/会话目录内固定 transcript 的 `{ kind: 'jsonl', path }`。它不执行文件系统 I/O可以在目录或文件存在前返回目标现有文件也只包含最后 flush 前缀。

View File

@@ -48,10 +48,9 @@ export interface Config {
/**
* Write runs of consecutive `assistant/chunk` delta events as packed
* `text-chunks`/`reasoning-chunks`/`tool-call-chunks` rows (lossless,
* ~60% smaller logs measured on a real session). Off by default while
* snapshot fixtures stay in the one-event-per-line layout: recording with
* packing on rewrites every golden `session.jsonl`. READING packed rows is
* unconditional — a log's layout never depends on this switch.
* ~60% smaller logs measured on a real session). Defaults to true; false
* keeps one `SessionEvent` per line for diagnostics. Reading packed rows is
* unconditional: a log's layout never depends on this switch.
*/
packChunks?: boolean
/** Physical encoding; defaults to checksummed Zstandard frames. */
@@ -80,7 +79,7 @@ export class SessionPersistenceJsonl extends SessionPersistence implements Persi
static Config: z<Config> = z.object({
root: z.string().required(),
packChunks: z.boolean().default(false),
packChunks: z.boolean().default(true),
compression: JsonlCompressionSchema,
})

View File

@@ -724,7 +724,7 @@ describe('SessionPersistenceJsonl: scanLog unit', () => {
})
})
describe('SessionPersistenceJsonl: packed chunk rows (packChunks: true)', () => {
describe('SessionPersistenceJsonl: default packed chunk rows', () => {
let ctx: Context
beforeEach(async () => {
root = await freshRoot()
@@ -732,7 +732,7 @@ describe('SessionPersistenceJsonl: packed chunk rows (packChunks: true)', () =>
await ctx.plugin(SessionStore)
// compression: 'none' — these tests assert the textual storage-record layout
// (row tags per line); packing is orthogonal to the physical encoding.
await ctx.plugin(SessionPersistenceJsonl, { root, packChunks: true, compression: 'none' })
await ctx.plugin(SessionPersistenceJsonl, { root, compression: 'none' })
})
afterEach(async () => { await ctx.fiber.dispose() })
@@ -754,7 +754,7 @@ describe('SessionPersistenceJsonl: packed chunk rows (packChunks: true)', () =>
]
}
it('writes a delta run as one text-chunks row and loads back identical events', async () => {
it('writes a delta run as one text-chunks row by default and loads back identical events', async () => {
const m = meta('packed', '/work')
const log = chunkRunLog()
await ctx.sessionPersistence.create(m)
@@ -768,6 +768,32 @@ describe('SessionPersistenceJsonl: packed chunk rows (packChunks: true)', () =>
expect(loaded.events).toEqual(log)
})
it('packChunks: false writes one event per line and still loads identical events', async () => {
const unpackedRoot = await freshRoot()
const unpacked = new Context()
await unpacked.plugin(SessionStore)
await unpacked.plugin(SessionPersistenceJsonl, {
root: unpackedRoot,
packChunks: false,
compression: 'none',
})
try {
const m = meta('unpacked', '/work')
const log = chunkRunLog()
await unpacked.sessionPersistence.create(m)
await unpacked.sessionPersistence.append(m.id, log)
const records = (await readFile(rawLogPath(unpackedRoot, '/work', m.id), 'utf8'))
.split('\n').filter(Boolean).slice(1)
.map(line => JSON.parse(line) as { type: string })
expect(records.filter(record => record.type === 'assistant/chunk')).toHaveLength(5)
expect(records.some(record => record.type === 'text-chunks')).toBe(false)
expect((await unpacked.sessionPersistence.load(m.id)).events).toEqual(log)
} finally {
await unpacked.fiber.dispose()
}
})
it('loads a mixed file: verbatim lines from an unpacked writer, then packed appends', async () => {
const m = meta('mixed', '/work')
const log = chunkRunLog()