feat(session): opt-in packed chunk rows in the JSONL log

Providers stream token-sized deltas, so a session log stores hundreds of
near-identical assistant/chunk lines whose JSON envelopes dwarf their
payloads (~56x measured on a real DeepSeek session, 73% of file bytes).

Add a lossless storage codec to dsh-session: packChunkRuns() folds each
run of >=3 consecutive same-block delta chunks into one storage row --
text-chunks / reasoning-chunks / tool-call-chunks, bare slash-less tags
like the header line's 'session' so rows cannot be confused with session
events -- and decodeStorageRecord() expands rows back to the exact
original events (seq0/time0 + dt gap array reconstruct every member's
seq/time; tool-call rows carry the run-constant id/name). The encoder
whitelists exact shapes and stores anything unrecognized verbatim; the
decoder validates row-tagged values and fails loud on malformation.

The JSONL backend gains a packChunks config (default false). Writing
packs only when enabled -- default-off output stays byte-identical to
the previous layout, so snapshot goldens are untouched. Reading is
layout-blind: scanLog always decodes rows and now checks seq contiguity
with a cursor instead of the line index, so packed, unpacked, and mixed
files all load identically. Fixture readers (llm-replay parseSessionLog,
acp-snapshot normalizeSessionLog) share the codec; the normalizer zeroes
a row's time0/dt exactly like an event's time. The two demo bundles
plumb packChunks from cordis.yml to the backend.

Measured on a real coding session: 105 KB -> 42 KB (-60%), 475 lines ->
74, with reasoning/tool-call heavy sessions saving the most. Covered by
example + fast-check round-trip codec tests, backend packed/mixed/torn-
tail specs, and an end-to-end demo run loading a packed log through a
default-config backend.
This commit is contained in:
kingwl
2026-07-15 21:26:36 +08:00
parent b045b553a9
commit 7ef21239ca
21 changed files with 816 additions and 51 deletions

View File

@@ -81,8 +81,10 @@ export function normalizeStdout(rawStdout: string, ctx: NormalizeContext): strin
* Normalize a session JSONL log into a stable golden: the header line's
* volatile fields (`createdAt`, `id`, `cwd`) and every event's `time` are
* zeroed/scrubbed, all volatile strings scrubbed, and `seq` is LEFT INTACT
* (deterministic by contract). Output is JSONL in the same shape as the input —
* one compact record per line.
* (deterministic by contract). A packed chunk row's timing (`time0`, the `dt`
* gaps) zeroes just like an event `time`; its `seq0` stays, like `seq`.
* Output is JSONL in the same shape as the input — one compact record per
* line.
*
* @param rawLog The raw session `.jsonl` content.
* @param ctx The run's volatile values to scrub.
@@ -95,6 +97,13 @@ export function normalizeSessionLog(rawLog: string, ctx: NormalizeContext): stri
// Header line: { type: 'session', createdAt, id, cwd, … }.
if (record.type === 'session') {
if ('createdAt' in record) record.createdAt = 0
} else if ('time0' in record) {
// Packed chunk row: zero the anchor timestamp and every member gap.
record.time0 = 0
const data = record.data
if (data !== null && typeof data === 'object' && Array.isArray((data as { dt?: unknown }).dt)) {
(data as { dt: unknown[] }).dt = (data as { dt: unknown[] }).dt.map(() => 0)
}
} else if ('time' in record) {
// Event line: zero the epoch-ms timestamp; keep seq (deterministic).
record.time = 0