perf(jsonl): scan decoded frames incrementally
This commit is contained in:
imccyu
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-07-19-zstandard-jsonl-session-logs.md
|
||||
2026-07-19-zstandard-jsonl-session-logs.md: 287ec94a91101850e9343d36ffd27870daf1333b
|
||||
2026-07-19-zstandard-jsonl-session-logs.zh.md: 4e578432640651de1eb1977229b7cdd462766c24
|
||||
2026-07-19-zstandard-jsonl-session-logs.md: 93fc20f931c75552352834b9340e7d38680d4254
|
||||
2026-07-19-zstandard-jsonl-session-logs.zh.md: 061d7fcb55c775eed10e99bae47777d32cc8eee1
|
||||
|
||||
@@ -28,7 +28,7 @@ First materialization compresses the two initial frames before opening the tempo
|
||||
|
||||
### Read, listing, and crash recovery
|
||||
|
||||
A frame-boundary scanner reads the standard magic, variable header fields, block headers and payload sizes, and optional checksum trailer. It does not interpret compressed blocks. Complete frames are decompressed independently and sequentially with Node's default `ZSTD_e_end`, which requires frame completion and validates their checksums, and their plaintext is passed to the existing JSONL scanner. A checksum/decompression failure in any complete frame, a malformed complete-frame JSONL tail, or invalid frame structure is corruption and rejects.
|
||||
A frame-boundary scanner reads the standard magic, variable header fields, block headers and payload sizes, and optional checksum trailer. It does not interpret compressed blocks. Complete frames are independently checksum-validated and passed through the [large-session restore pipeline](2026-08-05-large-session-jsonl-restore-pipeline.md), which owns decoder reuse, cooperative yielding, and incremental JSONL scanning. A checksum/decompression failure in any complete frame, a malformed complete-frame JSONL tail, or invalid frame structure is corruption and rejects.
|
||||
|
||||
Listing reads in bounded chunks only until the first complete frame is available, validates and decompresses that header frame, and never reads an event frame. The dedicated header frame therefore preserves metadata-only listing even for very large session logs.
|
||||
|
||||
@@ -53,5 +53,5 @@ The shared persistence and coordinator contracts run against both encodings. Bac
|
||||
- Ordinary session roots store `.jsonl.zstd` and retain append-only, fsync, rollback, and interrupted-turn recovery semantics.
|
||||
- Raw JSONL remains a deliberate configuration, but changing encoding requires a fresh/separate root or selecting the mode that matches existing artifacts.
|
||||
- One frame per durable batch adds bounded framing/checksum overhead and allows header-only listing plus repair from an exact append boundary.
|
||||
- External tools must understand concatenated Zstandard frames or consume raw-mode artifacts; generic one-shot Node decompression reads only the first independent frame, so backend reads walk frames explicitly.
|
||||
- External tools must understand concatenated Zstandard frames or consume raw-mode artifacts; generic one-shot Node decompression reads only the first independent frame, so backend reads walk frames through the [restore pipeline](2026-08-05-large-session-jsonl-restore-pipeline.md).
|
||||
- The implementation depends on Node's experimental built-in Zstandard API without an npm dependency; the supported-version compatibility gate makes drift visible.
|
||||
|
||||
@@ -28,7 +28,7 @@ JSONL 持久化后端会逐字保留每个 `SessionEvent`，其中包括数量
|
||||
|
||||
### 读取、列举与崩溃恢复
|
||||
|
||||
帧边界扫描器会读取标准魔数、可变头字段、块头与负载长度，以及可选校验和尾部，但不会解释压缩块。后端使用 Node 默认的 `ZSTD_e_end` 独立且按顺序解压完整帧；该模式要求帧完整并验证各帧校验和，再把明文交给既有 JSONL 扫描器。任何完整帧的校验和或解压失败、完整帧中畸形的 JSONL 尾部，或者无效帧结构都属于损坏并拒绝加载。
|
||||
帧边界扫描器会读取标准魔数、可变头字段、块头与负载长度，以及可选校验和尾部，但不会解释压缩块。完整帧会独立验证校验和，再进入[大型会话恢复流水线](2026-08-05-large-session-jsonl-restore-pipeline.md)；该流水线负责复用解码器、协作式让出事件循环和增量扫描 JSONL。任何完整帧的校验和或解压失败、完整帧中畸形的 JSONL 尾部，或者无效帧结构都属于损坏并拒绝加载。
|
||||
|
||||
列举只按有界分片读取到第一个完整帧可用为止，验证并解压该头部帧，绝不读取事件帧。因此，即使会话日志很大，专用头部帧仍能维持仅元数据列举。
|
||||
|
||||
@@ -53,5 +53,5 @@ CLI、ACP 与 stdio 应用包公开对称的 `persistenceCompression` 透传配
|
||||
- 普通会话根目录存储 `.jsonl.zstd`，并保留仅追加、fsync、回滚与中断轮次恢复语义。
|
||||
- 原始 JSONL 仍是显式配置，但切换编码需要使用全新或单独根目录，或者选择与既有产物匹配的模式。
|
||||
- 每个持久批次一个帧会增加有界的帧与校验和开销，同时支持仅头部列举和从精确追加边界开始修复。
|
||||
- 外部工具必须理解串联的 Zstandard 帧，或者消费原始模式产物；Node 通用的一次性解压只读取第一个独立帧，因此后端读取会显式遍历各帧。
|
||||
- 外部工具必须理解串联的 Zstandard 帧，或者消费原始模式产物；Node 通用的一次性解压只读取第一个独立帧，因此后端读取会通过[恢复流水线](2026-08-05-large-session-jsonl-restore-pipeline.md)遍历各帧。
|
||||
- 实现依赖 Node 的实验性内置 Zstandard API，但不增加 NPM 依赖；受支持版本兼容性门禁会暴露 API 漂移。
|
||||
|
||||
Reference in New Issue
Block a user