Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(session): opt-in packed chunk rows in the JSONL log

Browse Source 
Providers stream token-sized deltas, so a session log stores hundreds of
near-identical assistant/chunk lines whose JSON envelopes dwarf their
payloads (~56x measured on a real DeepSeek session, 73% of file bytes).

Add a lossless storage codec to dsh-session: packChunkRuns() folds each
run of >=3 consecutive same-block delta chunks into one storage row --
text-chunks / reasoning-chunks / tool-call-chunks, bare slash-less tags
like the header line's 'session' so rows cannot be confused with session
events -- and decodeStorageRecord() expands rows back to the exact
original events (seq0/time0 + dt gap array reconstruct every member's
seq/time; tool-call rows carry the run-constant id/name). The encoder
whitelists exact shapes and stores anything unrecognized verbatim; the
decoder validates row-tagged values and fails loud on malformation.

The JSONL backend gains a packChunks config (default false). Writing
packs only when enabled -- default-off output stays byte-identical to
the previous layout, so snapshot goldens are untouched. Reading is
layout-blind: scanLog always decodes rows and now checks seq contiguity
with a cursor instead of the line index, so packed, unpacked, and mixed
files all load identically. Fixture readers (llm-replay parseSessionLog,
acp-snapshot normalizeSessionLog) share the codec; the normalizer zeroes
a row's time0/dt exactly like an event's time. The two demo bundles
plumb packChunks from cordis.yml to the backend.

Measured on a real coding session: 105 KB -> 42 KB (-60%), 475 lines ->
74, with reasoning/tool-call heavy sessions saving the most. Covered by
example + fast-check round-trip codec tests, backend packed/mixed/torn-
tail specs, and an end-to-end demo run loading a packed log through a
default-config backend.
This commit is contained in:
kingwl
2026-07-15 21:26:36 +08:00
parent b045b553a9
commit 7ef21239ca
21 changed files with 816 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files

17
docs/config-catalog.md
View File

@@ -50,6 +50,8 @@ export interface Config {
tools?: ToolsConfig
/** Directory the JSONL session backend writes under. Defaults to `./.sessions`. */
persistenceRoot?: string
/** Write delta-chunk runs as packed storage rows (the JSONL backend's `packChunks`). Defaults to `false`. */
packChunks?: boolean
/** Skill registry, local-provider, and model-facing consumer config forwarded to agent-spine-demo. */
skills?: agentCore.SkillConfig
}
@@ -417,7 +419,7 @@ export interface Config {
}
```
Source: [`packages/support/llm-replay/src/index.ts:306`](../packages/support/llm-replay/src/index.ts)
Source: [`packages/support/llm-replay/src/index.ts:308`](../packages/support/llm-replay/src/index.ts)
## `@deepseek-ai/dsh-mcp-client`
@@ -567,7 +569,7 @@ Source: [`packages/sandbox/sandbox-local/src/index.ts:20`](../packages/sandbox/s
Requires: `sessions`
```ts config-catalog
/** Plugin config: where the JSONL backend keeps its session logs (`root` is required — no default). */
/** Plugin config: where the JSONL backend keeps its session logs, and the packed-row write switch. */
export interface Config {
/**
* Root directory for all session files. Required (no default): a default of
@@ -575,6 +577,15 @@ export interface Config {
* (bash calls, subprocesses). Sessions group under per-cwd subdirectories.
*/
root: string
/**
* Write runs of consecutive `assistant/chunk` delta events as packed
* `text-chunks`/`reasoning-chunks`/`tool-call-chunks` rows (lossless,
* ~60% smaller logs measured on a real session). Off by default while
* snapshot fixtures stay in the one-event-per-line layout: recording with
* packing on rewrites every golden `session.jsonl`. READING packed rows is
* unconditional — a log's layout never depends on this switch.
*/
packChunks?: boolean
}
```
@@ -699,6 +710,8 @@ export interface Config {
tools?: ToolsConfig
/** Directory the JSONL session backend writes under. Defaults to `./.sessions`. */
persistenceRoot?: string
/** Write delta-chunk runs as packed storage rows (the JSONL backend's `packChunks`). Defaults to `false`. */
packChunks?: boolean
/** stdin-chat banner printed once on start. Defaults to `'ready.'`. */
welcome?: string
/** Skill registry, local-provider, and model-facing consumer config forwarded to agent-spine-demo. */

8
docs/cordis-catalog/events.md
View File

@@ -245,7 +245,7 @@ Creation announcement during session publication. A synchronous throw vetoes and
'session/created'(this: Scoped<Session>, session: Session): void
```
Source: [`packages/core/session/src/index.ts:47`](../../packages/core/session/src/index.ts)
Source: [`packages/core/session/src/index.ts:49`](../../packages/core/session/src/index.ts)
### `session/disposed` — emit
@@ -255,7 +255,7 @@ Emitted once when an announced session leaves the store, including publication r
'session/disposed'(this: Scoped<Session>, session: Session): void
```
Source: [`packages/core/session/src/index.ts:57`](../../packages/core/session/src/index.ts)
Source: [`packages/core/session/src/index.ts:59`](../../packages/core/session/src/index.ts)
### `session/event` — emit
@@ -267,7 +267,7 @@ Post-commit, fire-and-forget append feed. The listener snapshot resolves before
Types: [SessionEvent](../core-data-structures/core.md)
Source: [`packages/core/session/src/index.ts:69`](../../packages/core/session/src/index.ts)
Source: [`packages/core/session/src/index.ts:71`](../../packages/core/session/src/index.ts)
### `session/flush` — parallel
@@ -277,7 +277,7 @@ Awaited parallel durability checkpoint: every listener runs and the caller await
'session/flush'(this: Scoped<Session>, session: Session): Promise<void> | void
```
Source: [`packages/core/session/src/index.ts:79`](../../packages/core/session/src/index.ts)
Source: [`packages/core/session/src/index.ts:81`](../../packages/core/session/src/index.ts)
## `subagent/*`

2
docs/cordis-catalog/services.md
View File

@@ -200,7 +200,7 @@ list(): Session[]
fork(source: SessionForkSource, boundary?: number, childSessionId?: SessionId): Session
```
Source: [`packages/core/session/src/index.ts:564`](../../packages/core/session/src/index.ts)
Source: [`packages/core/session/src/index.ts:566`](../../packages/core/session/src/index.ts)
## `ctx.skills` — `SkillService`

2
docs/core-data-structures/session.md
View File

@@ -305,6 +305,6 @@ The hook bridges' `hook/invoked` / `hook/result` provenance pairs (from `@deepse
## Durability contract
What a persistence backend relies on: the durable log persists every event verbatim, **including** `assistant/chunk` — `seq` must stay contiguous, so chunks cannot be filtered out of the canonical log. All `event.data` must be JSON-serializable; `Session.append` enforces this at the source (throwing on non-serializable data), so a bad event never enters the log and `session.events` always equals what a backend can persist. Adding an event type that carries non-serializable data, or that breaks the turn/step nesting the invariants plugin checks, is a breaking change to the on-disk format.
What a persistence backend relies on: the durable log persists every event losslessly, **including** `assistant/chunk` — `seq` must stay contiguous, so chunks cannot be filtered out of the canonical log. A backend may choose its own storage encoding for an event batch as long as `load` returns the exact appended events (the JSONL backend's opt-in packed chunk rows are such an encoding — see [persistence.md](persistence.md)). All `event.data` must be JSON-serializable; `Session.append` enforces this at the source (throwing on non-serializable data), so a bad event never enters the log and `session.events` always equals what a backend can persist. Adding an event type that carries non-serializable data, or that breaks the turn/step nesting the invariants plugin checks, is a breaking change to the on-disk format.
The backends that consume this contract are on [persistence.md](persistence.md).

8
docs/event-producer-consumer.md
View File

@@ -25,10 +25,10 @@ This matrix shows which packages dispatch each harness-owned event and which pac
| `fs/observed` | `emit` | [`packages/fs/fs/src/index.ts:68`](../packages/fs/fs/src/index.ts) | [`tool-fs`](../packages/fs/tool-fs) (`emit`) | [`fs-policy`](../packages/fs/fs-policy) |
| `fs/write-intent` | `waterfall` | [`packages/fs/fs/src/index.ts:51`](../packages/fs/fs/src/index.ts) | [`tool-fs`](../packages/fs/tool-fs) (`waterfall`) | [`fs-policy`](../packages/fs/fs-policy) |
| `llm/stream` | `waterfall` | [`packages/llm/llm/src/index.ts:39`](../packages/llm/llm/src/index.ts) | [`llm`](../packages/llm/llm) (`waterfall`) | [`invariants`](../packages/support/invariants), [`llm-replay`](../packages/support/llm-replay) |
| `session/created` | `emit` | [`packages/core/session/src/index.ts:47`](../packages/core/session/src/index.ts) | [`session`](../packages/core/session) (`events.dispatch`) | [`invariants`](../packages/support/invariants), [`jsonrpc`](../packages/ui/jsonrpc), [`session-persistence`](../packages/session-persistence/session-persistence) |
| `session/disposed` | `emit` | [`packages/core/session/src/index.ts:57`](../packages/core/session/src/index.ts) | [`session`](../packages/core/session) (`events.dispatch`) | - |
| `session/event` | `emit` | [`packages/core/session/src/index.ts:69`](../packages/core/session/src/index.ts) | [`session`](../packages/core/session) (`events.dispatch`) | [`acp`](../packages/ui/acp), [`invariants`](../packages/support/invariants), [`jsonrpc`](../packages/ui/jsonrpc), [`session-persistence`](../packages/session-persistence/session-persistence), [`stdio`](../packages/ui/stdio) |
| `session/flush` | `parallel` | [`packages/core/session/src/index.ts:79`](../packages/core/session/src/index.ts) | [`session`](../packages/core/session) (`events.dispatch`) | [`session-persistence`](../packages/session-persistence/session-persistence) |
| `session/created` | `emit` | [`packages/core/session/src/index.ts:49`](../packages/core/session/src/index.ts) | [`session`](../packages/core/session) (`events.dispatch`) | [`invariants`](../packages/support/invariants), [`jsonrpc`](../packages/ui/jsonrpc), [`session-persistence`](../packages/session-persistence/session-persistence) |
| `session/disposed` | `emit` | [`packages/core/session/src/index.ts:59`](../packages/core/session/src/index.ts) | [`session`](../packages/core/session) (`events.dispatch`) | - |
| `session/event` | `emit` | [`packages/core/session/src/index.ts:71`](../packages/core/session/src/index.ts) | [`session`](../packages/core/session) (`events.dispatch`) | [`acp`](../packages/ui/acp), [`invariants`](../packages/support/invariants), [`jsonrpc`](../packages/ui/jsonrpc), [`session-persistence`](../packages/session-persistence/session-persistence), [`stdio`](../packages/ui/stdio) |
| `session/flush` | `parallel` | [`packages/core/session/src/index.ts:81`](../packages/core/session/src/index.ts) | [`session`](../packages/core/session) (`events.dispatch`) | [`session-persistence`](../packages/session-persistence/session-persistence) |
| `subagent/end` | `emit` | [`packages/subagent/subagent/src/index.ts:92`](../packages/subagent/subagent/src/index.ts) | [`subagent`](../packages/subagent/subagent) (`events.dispatch`) | [`hooks-claude`](../packages/hooks/hooks-claude), [`jsonrpc`](../packages/ui/jsonrpc) |
| `subagent/provider-added` | `emit` | [`packages/subagent/subagent/src/index.ts:66`](../packages/subagent/subagent/src/index.ts) | [`subagent`](../packages/subagent/subagent) (`emit`) | [`tool-subagent`](../packages/subagent/tool-subagent) |
| `subagent/provider-removed` | `emit` | [`packages/subagent/subagent/src/index.ts:72`](../packages/subagent/subagent/src/index.ts) | [`subagent`](../packages/subagent/subagent) (`events.dispatch`) | [`tool-subagent`](../packages/subagent/tool-subagent) |