Merge latest master into codex/llm-error-recovery-rfc
# Conflicts: # docs/architecture.md # docs/config-catalog.md # docs/cordis-catalog/events.md # docs/event-producer-consumer.md # docs/module-graph.md # docs/persistence-catalog.md # packages/cordis/tool-cordis/src/api-catalog.ts # packages/core/agent-loop/src/loop.ts # packages/core/agent/src/types.ts # packages/examples/agent-spine-demo/README.md # packages/examples/stdio-demo/README.md # packages/examples/stdio-demo/src/index.ts # packages/llm/llm-deepseek/README.md # packages/llm/llm-deepseek/src/adapter.ts # packages/llm/llm-deepseek/tests/adapter.spec.ts # packages/llm/llm/README.md # packages/llm/llm/src/error.ts # packages/llm/llm/tests/service.spec.ts # packages/sandbox/sandbox-policy/tsconfig.json # packages/ui/stdio/README.md # packages/ui/stdio/src/index.ts # packages/ui/stdio/tests/stdio.spec.ts # packages/ui/tui/src/index.ts # pnpm-lock.yaml # website/zh-CN/api/harness/events.md # website/zh-CN/api/harness/llm.md
This commit is contained in:
Tianyi Cui
@@ -10,7 +10,7 @@ The harness needs one internal language for messages that the loop, session log,
|
||||
|
||||
Own the vocabulary: messages are arrays of typed content blocks (`text`, `reasoning`, `tool-call`, `tool-result`), with the union derived from the merge-extensible `ContentBlockMap` so plugins add block types via declaration merging. The same merge-extensible-map pattern types every "stringly" field (`MessageSource`, `FinishReason`, `TurnTrigger`, `TurnEndReason`). Streaming is a raw chunk protocol; `BlockAssembler` is the single shared assembly implementation. Adapters translate to provider wire formats — mapping cost lives in adapters, where it belongs.
|
||||
|
||||
In-session context injection (`context/message`, `steering/message`) renders as tagged user-role envelopes (the system-reminder pattern) rather than a new role, so adapters carry zero burden. Live-adapter validation confirms this rendering for current DeepSeek behavior; a future provider-specific mismatch belongs in that adapter rather than a new canonical role.
|
||||
In-session context injection (`context/message`) and mid-turn steering (`steering/message`) originally rendered as tagged user-role envelopes (the system-reminder pattern) rather than a new role, so adapters carry zero burden. Both now project as plain user content with no wrapper; see [the injected-content-envelope Agent Note](../simplification/2026-07-20-unwrap-injected-content-envelopes.md). Live-adapter validation confirms this rendering for current DeepSeek behavior; a future provider-specific mismatch belongs in that adapter rather than a new canonical role.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@ The [event-sourced model](2026-06-11-event-sourced-sessions.md) makes the append
|
||||
Persistence is an abstract **capability seam** ([capability seams](2026-06-13-capability-seams.md), the `dsh-bash` template), not loop or core logic:
|
||||
|
||||
1. **Interface** (`dsh-session-persistence`, `ctx.sessionPersistence`) — an abstract `SessionPersistence` service: `create`/`append`/`load`/`list`. Its persisted unit IS the existing `SessionEvent` (`{ type, seq, time, data }`), reused verbatim — no conversion type.
|
||||
2. **Implementation** (`dsh-session-persistence-jsonl`) — an append-only JSONL log per session (a `SessionHeader` line then one `SessionEvent` per line, verbatim **including `assistant/chunk`**).
|
||||
2. **Implementation** (`dsh-session-persistence-jsonl`) — an append-only logical JSONL log per session (a `SessionHeader` line then one `SessionEvent` per line, verbatim **including `assistant/chunk`**), encoded as [checksummed Zstandard frames by default](2026-07-19-zstandard-jsonl-session-logs.md) or raw lines by configuration.
|
||||
|
||||
Key choices recorded here because they are durable, contested, and surprising:
|
||||
|
||||
|
||||
@@ -12,7 +12,7 @@ Three seams: the queue-aware cancel, the `AgentHandle` disposer, and the bash ow
|
||||
|
||||
### 1. Queue-aware `Agent.cancel(reason?)`
|
||||
|
||||
A new `cancel()` verb on the `Agent` interface — the single public stop primitive. (It originally shipped alongside a narrower step-only `abort()`; that verb was later removed as unused, leaving `cancel()` the only public way to stop work.) It clears the inbox's queued + steering FIFOs, aborts the in-flight step if any, and drives a **turn-scoped cancellation marker** the driver loop checks at every turn-decision point — so a prompt that is queued-but-not-yet-started never runs, a cancel landing in the pre-step / continuation window drops the about-to-run turn (ending it `aborted`), and a later prompt cannot be batched into the cancelled turn. `whenIdle()` reaches post-cancel quiescence. ACP `session/cancel` maps to `cancel()`. The marker is armed ONLY when there is something to cancel, so an idle no-op cancel cannot strand the next prompt.
|
||||
A new `cancel()` verb on the `Agent` interface — the single public stop primitive. (It originally shipped alongside a narrower step-only `abort()`; that verb was later removed as unused, leaving `cancel()` the only public way to stop work.) It clears the inbox's queued + steering FIFOs, aborts the in-flight step if any, and drives a **turn-scoped cancellation marker** the driver loop checks at every turn-decision point — so a prompt that is queued-but-not-yet-started never runs, a cancel landing in the pre-step / continuation window drops the about-to-run turn (ending it `aborted`), and a later accepted prompt remains an independent queued turn. `whenIdle()` reaches post-cancel quiescence. ACP `session/cancel` maps to `cancel()`. The marker is armed ONLY when there is something to cancel, so an idle no-op cancel cannot strand the next prompt.
|
||||
|
||||
### 2. `AgentHandle` async disposer
|
||||
|
||||
@@ -29,7 +29,7 @@ Background-task ownership moved from a `tool-bash` plugin-local `Map<string, Age
|
||||
These invariants hold and are pinned by tests:
|
||||
|
||||
- ACP disconnect/session close leaves no registered agent AND no session-store entry for that session, even when `session/load` races teardown.
|
||||
- `session/cancel` before a queued prompt starts prevents that prompt from running and cannot batch the next prompt into the cancelled turn.
|
||||
- `session/cancel` before a queued prompt starts prevents that prompt from running; a later accepted prompt remains an independent queued turn.
|
||||
- A `tool-bash` HMR reload does NOT make an existing background task readable or killable by a different session (ownership survives on the executor).
|
||||
- Existing non-ACP demos still work without managing handles explicitly; config-created agents remain owned by the `AgentLoop` plugin fiber.
|
||||
|
||||
|
||||
@@ -68,3 +68,5 @@ Every surface-eligible event must carry `surfaceOp` or it would disappear from d
|
||||
- **`packages/session-persistence/session-persistence`**: Abstract interface unchanged.
|
||||
|
||||
The surface is the foundation for future history manipulation. A compaction or tool-result-prune plugin appends one of the existing message-producing event types (a `user/message` carrying the summary, say) with `surfaceOp: { op: 'replace', start, end }` and `sourceEventSeqs` covering the shadowed entries — the new event takes the range's place on the surface while the plugin's own trace events (e.g. `compaction/start`, `compaction/end`) stay off it. Replay preserves the decision deterministically.
|
||||
|
||||
A `tool/result` replacement may rewrite exactly one current `tool/result` and must preserve every data field except `content`. Session acceptance enforces this rule together with positional range and provenance validation, independent of optional diagnostic plugins.
|
||||
|
||||
@@ -28,7 +28,7 @@ Six methods (five required + an optional lifecycle hook) — the only seam betwe
|
||||
|
||||
### The opaque torn marker
|
||||
|
||||
The single design choice that keeps the seam clean: the crash-repair "where is the torn tail" token is OPAQUE to the coordinator. The coordinator computes the synthetic closers (it owns `interruptedTurnClosers` from `dsh-session`), but it only ever tests `tornMarker !== undefined` and passes the value straight back to `commitRepair` — it never inspects it. Each backend picks its own marker type: JSONL uses the byte offset to truncate to, SQLite the seq to delete from (both happen to be `number`). The JSONL backend folds its `committedBytes < buffer.byteLength` comparison INSIDE the hook so the returned marker is already `number | undefined`; without that fold the coordinator would have to know about byte lengths.
|
||||
The single design choice that keeps the seam clean: the crash-repair "where is the torn tail" token is OPAQUE to the coordinator. The coordinator computes the synthetic closers (it owns `interruptedTurnClosers` from `dsh-session`), but it only ever tests `tornMarker !== undefined` and passes the value straight back to `commitRepair` — it never inspects it. Each backend picks its own marker type: JSONL carries the byte offset to truncate to plus any complete events decoded from an incomplete final frame, while SQLite carries the seq to delete from. The coordinator therefore knows neither byte lengths nor frame recovery state.
|
||||
|
||||
## Testing
|
||||
|
||||
|
||||
@@ -6,29 +6,28 @@ Status: implemented
|
||||
|
||||
An example folder is supposed to be *thin* — the variable wiring of a demo, not the demo's machinery. Before this change it was thick. Each example carried a hand-rolled `start.ts` boot bootstrap, an infra preamble (`timer`, and — for the stdio demos — `logger` + `hmr`), nested includes of three shared YAML fragments (`base.yml` / `base-core.yml` / `acp-agent/acp-tail.yml`), and per-example `agent-loop`/persistence/system-prompt config. The actual app — the spine of services every agent needs — was spread across the leaf and those includes.
|
||||
|
||||
The leaf configs also owned a coupled front door. ACP requires stdout purity and creates agents through `session/new`; stdio requires a console logger and a pre-created `main`. Prose warnings were the only guard against combining these incorrectly, while three `start.ts` files duplicated the Loader bootstrap and lifecycle code.
|
||||
The leaf configs also owned coupled front doors. ACP requires stdout purity and creates agents through `session/new`; terminal and Headless apps pre-create `main` but have different process I/O contracts. Prose warnings were the only guard against combining these incorrectly, while three `start.ts` files duplicated the Loader bootstrap and lifecycle code.
|
||||
|
||||
## Decision
|
||||
|
||||
Each example is now **mostly an invocation of an app package**, splitting the wiring along the existing [interface / implementation / consumer seam](2026-06-13-capability-seams.md): the **app package owns the composition**, the leaf `cordis.yml` owns only the **swappable choices** (which LLM adapter, which bash executor, model, prompt, persistence root).
|
||||
|
||||
- **`@deepseek-ai/dsh-agent-spine-demo`** ([packages/examples/agent-spine-demo](../../../../packages/examples/agent-spine-demo)) composes the providerless, executor-less, UI-less spine and forwards the loop's agent-list config. Its dependency on the concrete loop is intentional because this package composes the spine rather than extending it; swapping the loop means supplying another bundle.
|
||||
- **`@deepseek-ai/dsh-stdio-demo`** ([packages/examples/stdio-demo](../../../../packages/examples/stdio-demo)) and **`@deepseek-ai/dsh-acp-demo`** ([packages/examples/acp-demo](../../../../packages/examples/acp-demo)) bake in their front doors. Stdio includes `ui-stdio`, a console logger, and `main`; ACP includes the bridge and JSONL persistence but no stdout logger or pre-created agent. Leaves may add plugins, but the safe composition is now the default artifact.
|
||||
- **`start.ts` is gone.** Each app package exposes a `bin` (`dsh-stdio-demo` / `dsh-acp-demo`); the `demo:*` scripts invoke it (e.g. `dsh-stdio-demo ./cordis.yml`). The Loader-boot tail, `.env` loading, and fail-loud guards live in the shared [`@deepseek-ai/dsh-app-boot`](../../../../packages/ui/app-boot) package (unit-tested under the per-file coverage gate — see [share the app bins' boot glue](../simplification/2026-07-04-share-app-bin-boot-glue.md)); each bin is a thin self-executing composition over those helpers plus its app-specific lifecycle (the ACP bin: snapshot-mode selection and stdin-dispose). The `bin.ts` files themselves stay coverage-excluded (self-executing CLI entries, like the old `start.ts`) and are driven by the keyless Loader-path tests.
|
||||
- **Each leaf `cordis.yml` collapses** to backends + config: the LLM adapter (`llm-deepseek` with apiKey/models, or `llm-replay`), the bash executor (`bash-local`), `hmr` for the stdio demos (see the amendment below), and one app entry carrying the app's config (model, system prompt, persistence root — surfaced as the app package's own `Config`, which routes each value to wherever the app wires it: stdio onto its pre-created agent, acp onto the bridge plugin).
|
||||
- **echo-agent folds onto `dsh-stdio-demo`**, swapping the LLM backend to the local `mock-llm` and adding the local `echo-tool` (plus `bash-local`, which the spine's `tool-bash` injects) at the leaf — the clean demonstration of "swap the backend, keep the app". `mock-llm.ts` / `echo-tool.ts` stay as example-local teaching plugins.
|
||||
- **`@deepseek-ai/dsh-tui-demo`**, **`@deepseek-ai/dsh-cli-demo`**, and **`@deepseek-ai/dsh-acp-demo`** bake in their process roles. TUI includes the full-screen UI and a pre-created `main`; Headless includes the one-shot driver and a pre-created `main`; ACP includes the bridge and no pre-created agent. All three include JSONL persistence and omit stdout loggers.
|
||||
- **`start.ts` is gone.** Each app package exposes a bin; the `demo:*` scripts invoke it. Loader boot, `.env` loading, and fail-loud guards live in the shared [`@deepseek-ai/dsh-app-boot`](../../../../packages/ui/app-boot) package (unit-tested under the per-file coverage gate — see [share the app bins' boot glue](../simplification/2026-07-04-share-app-bin-boot-glue.md)); the thin self-executing entries are driven by keyless Loader-path tests.
|
||||
- **Each leaf `cordis.yml` collapses** to backends, optional product tools, and one app entry carrying the app config. TUI and Headless route model/session choices onto a pre-created agent; ACP routes the initial provider/model onto its bridge.
|
||||
- **`base.yml`, `base-core.yml`, and `acp-agent/acp-tail.yml` are retired** — the spine they shared now lives in `dsh-agent-spine-demo`.
|
||||
|
||||
`bash-local` and the LLM adapter stay **leaf choices**: the bundle ships `tool-bash` (the consumer schema), the leaf picks the executor implementation, so a sandboxed executor or replay adapter swaps in without touching the app.
|
||||
|
||||
### Amendment on implementation: `hmr` stays a leaf entry
|
||||
|
||||
The proposal listed `hmr` among the stdio app's baked-in front-door cluster. Validating against the code, baking `hmr` into the `dsh-stdio-demo` package fights cordis in two ways, so it ships as a **leaf `cordis.yml` entry** instead:
|
||||
The proposal listed `hmr` among the interactive app's baked-in front-door cluster. Validating against the code, baking `hmr` into the app package fights Cordis in two ways, so it ships as a **leaf `cordis.yml` entry** instead:
|
||||
|
||||
1. `@cordisjs/plugin-hmr` is a Loader-only, subprocess-only dev plugin — its constructor throws without `node --expose-internals` + a live `loader` service, so it can only run in the real `demo:*`/bin subprocess, never in the in-process unit/coverage tier.
|
||||
2. The in-process test tier (vitest) cannot even *import* the vendored `hmr` module (its class-decorator `@Inject` form fails under Vite's transform), so a package whose `apply` statically imported it could never satisfy the per-file 100% coverage gate on its headline function.
|
||||
|
||||
Crucially, `hmr` is **not** a stdout-purity footgun the way the console logger is — a stray `hmr` in the ACP config would not corrupt the JSON-RPC frames — so leaving it at the leaf costs none of the safety the coupling argument is about. The **logger** (the real coupling) stays baked in: the stdio app includes it, the ACP app omits it.
|
||||
Crucially, `hmr` is not a stdout-purity footgun: a stray entry in the ACP config does not corrupt JSON-RPC frames. Every shipped app omits a stdout console logger; the app or protocol driver alone owns stdout.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
@@ -39,13 +38,13 @@ The old `base*.yml`/`acp-tail.yml` includes already deduped the *config*, but a
|
||||
## Verification
|
||||
|
||||
- Example directories contain only their config, README, and tests: `start.ts`, the infrastructure preamble, and the shared YAML includes are gone.
|
||||
- `demo:echo`, `demo:repl`, and `demo:acp` invoke the app-package bins.
|
||||
- `demo:tui`, `demo:headless`, and `demo:acp` invoke the app-package bins.
|
||||
- Each new package has a README and per-file 100% coverage; each app package also has a keyless real-Loader-path bin smoke that catches export-shape failures described in [postmortem 0001](../../../../docs/postmortem/0001-acp-default-export-drops-inject.md).
|
||||
- The ACP replay transcript remains unchanged because the plugin set and load order did not change.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **The bare-plugin-tree pedagogy.** echo-agent's inlined `cordis.yml` showed every plugin at once; the spine now lives behind a bundle, so seeing the whole tree means opening `dsh-agent-spine-demo`. The app package's README carries that teaching weight.
|
||||
- **The bare-plugin-tree pedagogy.** The spine lives behind a bundle, so seeing the whole tree means opening `dsh-agent-spine-demo`. The app package's README carries that teaching weight.
|
||||
- **A layer of indirection.** "What does this demo load?" becomes a package read, not a single YAML scan.
|
||||
|
||||
## Related
|
||||
@@ -53,3 +52,4 @@ The old `base*.yml`/`acp-tail.yml` includes already deduped the *config*, but a
|
||||
- Supersedes [Make the shared example base providerless](../../rejected/architecture/2026-06-20-providerless-example-base.md): renaming `base.yml` to the providerless core is moot once the spine moves into `dsh-agent-spine-demo` and the `base*.yml` files are deleted.
|
||||
- Builds on the [capability-seams](2026-06-13-capability-seams.md) interface/implementation/consumer split — backends and presentation stay leaf choices; the spine is the shared bundle.
|
||||
- Complements [Reorganize packages into a modular hierarchy](2026-06-20-package-hierarchy.md): the new app/core packages slot into existing groups under that hierarchy (`core` for the reusable spine bundle, `ui` for the app-specific front doors).
|
||||
- The later [redundant-agent removal](../simplification/2026-07-20-remove-stdio-and-echo-agents.md) owns the final TUI/Headless split and removes the line-oriented and mock-only leaves.
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
Status: implemented
|
||||
|
||||
The later [fold-stdio-helper](../simplification/2026-07-04-fold-stdio-ui-helper.md) decision superseded the original `support/ui-stdio` placement, and the [redundant-agent removal](../simplification/2026-07-20-remove-stdio-and-echo-agents.md) subsequently removed that surface entirely. The uniform depth-two hierarchy remains the decision owned here.
|
||||
|
||||
## Problem
|
||||
|
||||
`packages/` was flat: 18 packages all sat at `packages/<name>/`, so a package's location said nothing about whether it was core product API, a swappable capability seam, a provider adapter, a product integration, or example/test support. The package README carried a `FIXME(package-hierarchy)` and `scripts/publint-all.ts` a `TODO(package-inventory)` flagging exactly this. Core packages, provider integrations, capability seams, example UI support, and snapshot-only replay support all looked equally foundational.
|
||||
|
||||
@@ -8,7 +8,7 @@ The assembled system prompt had four defects, all of one family: facts the harne
|
||||
|
||||
**The model could not know its own name.** `AgentOptions.model` drives every request, but no prompt text carried it — and nothing COULD carry it: sections in `dsh-system-prompt` were context-global while the model name is per-agent, and `assemble()` took no per-agent input at all.
|
||||
|
||||
**Tool guidance was hand-written prose in leaf YAML.** The bash/subagent/todo_write usage guidance lived in the `systemPrompt` strings of `examples/repl-agent/cordis.yml` and `examples/acp-agent/cordis.yml` — two drifting copies (the ACP one was already abridged) — while `dsh-tool-fs` and `dsh-tool-web` owned their guidance as `ctx.systemPrompt.section()` contributions. Loading or dropping a tool plugin meant editing every deployment's persona by hand; both YAMLs carried a `FIXME(config-comments)` apologizing for a symptom of the split, and the stdio welcome banner hand-enumerated the tool set too.
|
||||
**Tool guidance was hand-written prose in leaf YAML.** The bash/subagent/todo_write usage guidance lived in the coding-agent and ACP persona strings — two drifting copies (the ACP one was already abridged) — while `dsh-tool-fs` and `dsh-tool-web` owned their guidance as `ctx.systemPrompt.section()` contributions. Loading or dropping a tool plugin meant editing every deployment's persona by hand; both YAMLs carried a `FIXME(config-comments)` apologizing for a symptom of the split, and the old terminal welcome banner hand-enumerated the tool set too.
|
||||
|
||||
**The persona rendered after tool guidance.** The loop string-joined `agent.options.systemPrompt` AFTER the assembled sections, so the model read "Use the read tool…" before "You are a coding agent" — backwards relative to the identity-first convention (Claude Code, Codex) and a second composition path besides the section pipeline.
|
||||
|
||||
@@ -56,7 +56,7 @@ Per-tool semantics and selection guidance live in tool descriptions. Prompt sect
|
||||
|
||||
## Shipped invariants
|
||||
|
||||
- The repl-agent prompt renders identity, persona with the interpolated model, then fs/bash/web guidance through one assembly path.
|
||||
- The tui-agent prompt renders identity, persona with the interpolated model, then fs/bash/web guidance through one assembly path.
|
||||
- Fork and fresh subagent descriptions reflect whether the provider inherits completed conversation turns; the tool appears, disappears, and is reworded with provider lifecycle changes.
|
||||
- Unknown, valueless, malformed, or unbalanced variable references name the section and throw; duplicate section, variable, and tool registrations also throw.
|
||||
- Snapshot replay is prompt-independent: it keys recorded chunk streams by turn and step without comparing the outgoing request.
|
||||
|
||||
@@ -162,7 +162,7 @@ Those cases can consume `ctx.spillStore` directly in later work. They are not pa
|
||||
- `dsh-spill-local` unit tests cover `saveText`, `encodeSegment` sanitization (separators/tilde/whole-segment dots/empty), the session-hash directory, owner-only permissions, distinct paths per save, the configured/private root, and a storage-failure rejection.
|
||||
- `dsh-spill-policy` unit tests drive real tools through `ctx.tools.execute`: disabled-mode no-op, oversized-text replacement, small/non-text passthrough, `read` skip, best-effort fallback (save failure / no backend / no owner), and downstream-composition (bounding a replaced result, preserving `additionalContexts`).
|
||||
- `dsh-tool-web` integration drives `web_fetch` through `ctx.tools.execute` with the real `spill-local` backend + policy, proving the model-facing text changes only by the deliberate spill notice while the spill file holds the full formatted result.
|
||||
- The `repl-agent` example loads `spill-local` + `spill-policy`, so its keyless Loader smoke exercises the real load path (the namespace-plugin export shape + `inject`).
|
||||
- The `tui-agent` example loads `spill-local` + `spill-policy`, so its keyless Loader/PTY smoke exercises the real load path (the namespace-plugin export shape + `inject`).
|
||||
|
||||
## Consequences
|
||||
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-10-after-call-compaction-pressure-and-overflow-recovery.md: f1a1868cd00007fb24efb21779dcc94c098b54e2
|
||||
2026-07-10-after-call-compaction-pressure-and-overflow-recovery.zh.md: a4993de2830301610bb2a9b0d28e8bbdf0ed9c46
|
||||
2026-07-10-after-call-compaction-pressure-and-overflow-recovery.md: d470cceaff68229b3872d0ade93d5fabc2e10c3f
|
||||
2026-07-10-after-call-compaction-pressure-and-overflow-recovery.zh.md: b7753fb226638b16b0f244b681cd2b9bcc9f25c2
|
||||
|
||||
@@ -16,9 +16,9 @@ Successful calls are not the only pressure signal. A provider can reject a reque
|
||||
|
||||
`agent/pre-step` is narrowed to `(agent, turn, step, signal)`. It remains a generic serial checkpoint before `step/start`, but it carries no compaction-only prompt or prefix fields.
|
||||
|
||||
The loop fires awaited serial `agent/post-step(agent, turn, step, signal)` after assistant output, every dispatched or synthetic tool result, post-tool context, and steering are durable, but before `step/end`. This placement gives pressure policy the complete successful-call state without splitting an assistant tool call from its result. A listener failure is an ordinary turn failure; it never enters model-request recovery.
|
||||
The loop fires awaited serial `agent/post-step(agent, turn, step, signal)` after assistant output, every dispatched or synthetic tool result, post-tool context, and steering are durable, but before `step/end`. This placement gives pressure policy the complete successful-call state without splitting an assistant tool call from its result. A propagated listener failure is an ordinary turn failure; it never enters model-request recovery. Compact-basic contains its expected operational failures as described below.
|
||||
|
||||
`dsh-compact-basic` reads the exact latest routed model from the durable request header only to establish that a completed route exists, then asks the singleton `ctx.tokenMeter` to measure the canonical logged envelope and current surface. It does not fall back to `AgentOptions.model` for automatic pressure. A headerless session has no completed routed request to assess and produces no work; any durable non-empty model name uses the same estimator. Operational measurement or summarization failures warn and continue with full history.
|
||||
`dsh-compact-basic` reads the exact latest routed model from the durable request header only to establish that a completed route exists, then asks the singleton `ctx.tokenMeter` to measure the canonical logged envelope and current surface. It does not fall back to `AgentOptions.model` for automatic pressure. A headerless session has no completed routed request to assess and produces no work; any durable non-empty model name uses the same estimator. Operational measurement or summarization failures warn and continue from the latest durable surface: full history before any replacement, or the pruned surface if pruning already landed.
|
||||
|
||||
### Request recovery is limited to the final model boundary
|
||||
|
||||
@@ -32,17 +32,17 @@ If cancellation lands after assistant tool calls are durable but before all call
|
||||
|
||||
`CompactService.compactIfNeeded(agent, trigger, signal)` accepts `trigger: 'pressure' | 'context-overflow'`. The interface gains no estimation methods or token types; `ctx.tokenMeter` remains the reusable accounting owner.
|
||||
|
||||
For `pressure`, compact-basic applies the service-wide threshold and retained-tail policy to one unified `ctx.tokenMeter.measure()` result. The same singleton meter owns range pricing, provenance, shadowed token counts, and non-shrinking-summary rejection. The common defaults remain threshold ratio `0.8`, retained history `floor(contextWindow × 0.16)`, summarization provider/model `''`, `maxTokens: 8192`, `compactionRetries: 1`, and `auto: true`.
|
||||
For `pressure`, compact-basic applies the service-wide threshold and retained-tail policy to one unified `ctx.tokenMeter.measure()` result. Below pressure it returns without pruning. Once pressure qualifies, optional `ctx.toolResultPrune` rewrites oversized current results and compact-basic remeasures through the same meter; safe pressure skips the model call, while remaining pressure selects and summarizes from the pruned surface. The same singleton meter owns range pricing, provenance, shadowed token counts, and non-shrinking-summary rejection. The common defaults remain threshold ratio `0.8`, retained history `floor(contextWindow × 0.16)`, summarization provider/model `''`, `maxTokens: 8192`, `compactionRetries: 1`, and `auto: true`.
|
||||
|
||||
For canonical overflow, compact-basic bypasses scalar pressure and the normal retained-token budget. It chooses the maximal tool-balanced head range while leaving the newest indivisible unit, then attempts exactly one shrinking compaction under the same signal. The automatic listener snapshots `session.surface.replaceGeneration` and returns `{ action: 'retry' }` only when compaction succeeds and the generation increases. A backend returning a result without replacement cannot authorize retry.
|
||||
For canonical overflow, compact-basic bypasses scalar pressure and the normal retained-token budget. It prunes first, then chooses the maximal tool-balanced head range while leaving the newest indivisible unit and attempts one shrinking summary compaction under the same signal when a range exists. The automatic listener snapshots `session.surface.replaceGeneration` and returns `{ action: 'retry' }` whenever pruning or summarization increases it. This remains true when pruning lands before later summary work throws; cancellation still wins. A backend returning a result without replacement cannot authorize retry, while pruning-only progress can authorize a retry without a `CompactionResult`.
|
||||
|
||||
`maxOverflowRetries` is optional and defaults to `1`; `0` disables overflow recovery without disabling pressure. `auto: false` registers neither automatic listener. Noncanonical errors, exhausted attempts, an already-aborted signal, a missing routed model, no safe range, no generation change, and recovery throws all delegate to the next listener. With no later recovery, the loop reports the original provider error object and code. Cancellation or disposal remains authoritative even if recovery work completes concurrently.
|
||||
`maxOverflowRetries` is optional and defaults to `1`; `0` disables overflow recovery without disabling pressure. `auto: false` registers neither automatic listener. Noncanonical errors, exhausted attempts, an already-aborted signal, a missing routed model, no safe range, no generation change, and recovery throws before any replacement all delegate to the next listener. With no later recovery, the loop reports the original provider error object and code. A recovery throw after generation advances authorizes retry from durable progress; cancellation or disposal remains authoritative even if recovery work completes concurrently.
|
||||
|
||||
The default summarizer resolves explicit configuration, then the latest logged route, then agent options. Because direct `llm/stream` middleware may reroute that auxiliary call, `compact/summary.{provider, model}` records the final mutable `GenerateOptions` target observed after dispatch rather than the pre-waterfall candidate.
|
||||
|
||||
## Testing
|
||||
|
||||
Unit tests cover final-adapter failure provenance and identity, closed-step retry numbering and reset, cancellation and disposal, post-step ordering, routed-envelope pressure, balanced overflow reduction, generation proof, caps, delegation, and auxiliary-call routing. Real-loop tests cover thrown and in-band overflow through compaction to a reconstructed retry request.
|
||||
Unit tests cover final-adapter failure provenance and identity, closed-step retry numbering and reset, cancellation and disposal, post-step ordering, routed-envelope pressure, pressure-gated pruning, pruning-only relief, pruned-input summarization, balanced overflow reduction, durable prune progress before later failure, generation proof, caps, delegation, and auxiliary-call routing. Real-loop tests cover thrown and in-band overflow through pruning or summary compaction to a reconstructed retry request.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
@@ -54,8 +54,8 @@ Unit tests cover final-adapter failure provenance and identity, closed-step retr
|
||||
|
||||
## Consequences
|
||||
|
||||
Post-step pressure describes the completed routed request, including durable tool results and request-only prefix fields. Canonical overflow supplies the backstop when no successful usage anchor exists. Recovery is bounded, cancellation-owned, and monotonic: it retries only after a visible surface generation change.
|
||||
Post-step pressure describes the completed routed request, including durable tool results and request-only prefix fields. Optional model-free pruning removes predictable tool-output bulk before summary selection and can independently create retry-worthy progress. Canonical overflow supplies the backstop when no successful usage anchor exists. Recovery is bounded, cancellation-owned, and monotonic: it retries only after a visible surface generation change.
|
||||
|
||||
The cost is one additional serial checkpoint on successful steps and adapter-maintained overflow classification. Provider wording and heuristic character density remain maintenance risks. Surface compaction still cannot repair an envelope that alone exceeds the window or split one indivisible oversized message/tool unit.
|
||||
The cost is one additional serial checkpoint on successful steps and adapter-maintained overflow classification. Provider wording and heuristic character density remain maintenance risks. Surface compaction still cannot repair an envelope that alone exceeds the window, split an indivisible non-tool node, or repair a tool unit whose non-prunable remainder remains oversized. The optional pruner can repair an otherwise indivisible tool pair when removable text-bearing tool-result content is the bulk.
|
||||
|
||||
This Agent Note supersedes only the pre-step automatic-trigger portion of the [compaction capability-seam Agent Note](../feature/2026-06-18-compaction-capability-seam.md). The service split, standalone token meter, balanced range contract, log-recorded lock, summary replacement, and sole `summarize()` subclass hook remain unchanged.
|
||||
|
||||
@@ -16,9 +16,9 @@ Status: implemented
|
||||
|
||||
`agent/pre-step` 收窄为 `(agent, turn, step, signal)`。它仍是 `step/start` 之前的通用串行检查点，但不再携带压缩专用的提示词或前缀字段。
|
||||
|
||||
循环在 assistant 输出、所有已分发或合成的工具结果、工具后上下文与 steering 都持久化之后、`step/end` 之前，触发等待式串行 `agent/post-step(agent, turn, step, signal)`。该位置让压力策略看到完整的成功调用状态，同时不会拆开 assistant 工具调用与其结果。监听器失败属于普通 turn 失败，绝不会进入模型请求恢复。
|
||||
循环在 assistant 输出、所有已分发或合成的工具结果、工具后上下文与 steering 都持久化之后、`step/end` 之前，触发等待式串行 `agent/post-step(agent, turn, step, signal)`。该位置让压力策略看到完整的成功调用状态，同时不会拆开 assistant 工具调用与其结果。向外传播的监听器失败属于普通 turn 失败，绝不会进入模型请求恢复；compact-basic 会按下文所述在内部处理其预期的操作性失败。
|
||||
|
||||
`dsh-compact-basic` 从持久请求头读取精确的最新实际路由模型，只用它确认已经存在完整路由，随后让单例 `ctx.tokenMeter` 计量规范日志信封与当前表层。自动压力不会回退到 `AgentOptions.model`。没有请求头的会话尚无已完成路由请求可供判断，因此不执行工作；任意持久记录的非空模型名都使用同一个估算器。操作性的计量或摘要失败会发出警告，并继续使用完整历史。
|
||||
`dsh-compact-basic` 从持久请求头读取精确的最新实际路由模型，只用它确认已经存在完整路由，随后让单例 `ctx.tokenMeter` 计量规范日志信封与当前表层。自动压力不会回退到 `AgentOptions.model`。没有请求头的会话尚无已完成路由请求可供判断，因此不执行工作；任意持久记录的非空模型名都使用同一个估算器。操作性的计量或摘要失败会发出警告，并从最新持久表层继续：任何替换发生前使用完整历史；若剪枝已经落盘，则使用已剪枝表层。
|
||||
|
||||
### 请求恢复只覆盖最终模型边界
|
||||
|
||||
@@ -32,17 +32,17 @@ Status: implemented
|
||||
|
||||
`CompactService.compactIfNeeded(agent, trigger, signal)` 接收 `trigger: 'pressure' | 'context-overflow'`。接口不增加估算方法或 token 类型；`ctx.tokenMeter` 继续作为可复用的核算所有者。
|
||||
|
||||
对于 `pressure`，compact-basic 把服务级阈值与保留尾部策略应用到一次统一的 `ctx.tokenMeter.measure()` 结果。范围定价、来源、被遮蔽 token 数与非缩小摘要拒绝也由同一个单例 meter 完成。通用默认值保持为阈值比例 `0.8`、保留历史 `floor(contextWindow × 0.16)`、摘要提供方/模型 `''`、`maxTokens: 8192`、`compactionRetries: 1` 与 `auto: true`。
|
||||
对于 `pressure`，compact-basic 把服务级阈值与保留尾部策略应用到一次统一的 `ctx.tokenMeter.measure()` 结果。低于压力时直接返回，不执行剪枝。压力达到条件后，可选的 `ctx.toolResultPrune` 会改写当前表层中过大的工具结果，compact-basic 再通过同一个 meter 重新计量；若压力恢复安全则跳过模型调用，否则从已剪枝表层选择范围并生成摘要。范围定价、来源、被遮蔽 token 数与非缩小摘要拒绝也由同一个单例 meter 完成。通用默认值保持为阈值比例 `0.8`、保留历史 `floor(contextWindow × 0.16)`、摘要提供方/模型 `''`、`maxTokens: 8192`、`compactionRetries: 1` 与 `auto: true`。
|
||||
|
||||
对于规范化溢出，compact-basic 绕过标量压力与普通保留 token 预算。它在保留最新不可分割单元的同时，选择最大的工具配对平衡头部范围，并在同一 signal 下只尝试一次缩小压缩。自动监听器先记录 `session.surface.replaceGeneration`，只有压缩成功且 generation 增加时才返回 `{ action: 'retry' }`。后端若只返回结果但没有替换表层，不能授权重试。
|
||||
对于规范化溢出，compact-basic 绕过标量压力与普通保留 token 预算。它先执行剪枝，再在保留最新不可分割单元的同时选择最大的工具配对平衡头部范围；存在范围时，才在同一 signal 下尝试一次缩小摘要压缩。自动监听器先记录 `session.surface.replaceGeneration`，剪枝或摘要让 generation 增加时就返回 `{ action: 'retry' }`。即使剪枝先落盘而后续摘要工作抛错，这条规则仍然成立；取消依然优先。后端若只返回结果但没有替换表层，不能授权重试；只有剪枝取得进展时，即使没有 `CompactionResult` 也可以授权重试。
|
||||
|
||||
`maxOverflowRetries` 可选且默认为 `1`；`0` 只禁用溢出恢复，不会禁用压力检查。`auto: false` 不注册任何自动监听器。非规范化错误、尝试耗尽、已经中止的 signal、缺失路由模型、没有安全范围、generation 未变化，以及恢复抛错都会委托给下一个监听器。若没有后续恢复，循环报告原始提供方错误对象与代码。即使恢复工作并发完成，取消或销毁仍具有最终优先级。
|
||||
`maxOverflowRetries` 可选且默认为 `1`；`0` 只禁用溢出恢复，不会禁用压力检查。`auto: false` 不注册任何自动监听器。非规范化错误、尝试耗尽、已经中止的 signal、缺失路由模型、没有安全范围、generation 未变化，以及在任何替换之前恢复抛错，都会委托给下一个监听器。若没有后续恢复，循环报告原始提供方错误对象与代码。generation 增加后的恢复抛错会基于持久进展授权重试；即使恢复工作并发完成，取消或销毁仍具有最终优先级。
|
||||
|
||||
默认摘要器依次解析显式配置、最近记录的路由与 agent options。因为直接 `llm/stream` 中间件可以重新路由该辅助调用，`compact/summary.{provider, model}` 记录分发后最终可变的 `GenerateOptions` 目标，而不是 waterfall 之前的候选值。
|
||||
|
||||
## 测试
|
||||
|
||||
单元测试覆盖最终适配器失败的来源与身份、已关闭 step 的重试编号与重置、取消与销毁、post-step 顺序、已路由信封压力、平衡溢出缩减、generation 证明、上限、委托与辅助调用路由。真实循环测试覆盖抛出式和带内溢出，并验证压缩后的重试请求从替换表层重建。
|
||||
单元测试覆盖最终适配器失败的来源与身份、已关闭 step 的重试编号与重置、取消与销毁、post-step 顺序、已路由信封压力、压力门控剪枝、剪枝独立解除压力、从已剪枝输入生成摘要、平衡溢出缩减、后续失败前已落盘的剪枝进展、generation 证明、上限、委托与辅助调用路由。真实循环测试覆盖抛出式和带内溢出，并验证剪枝或摘要压缩后的重试请求从替换表层重建。
|
||||
|
||||
## 考虑过的替代方案
|
||||
|
||||
@@ -54,8 +54,8 @@ Status: implemented
|
||||
|
||||
## 后果
|
||||
|
||||
Post-step 压力描述已完成的路由请求，包括持久工具结果与仅请求前缀字段。当成功 usage 锚点不存在时，规范化溢出提供兜底路径。恢复有明确上限、以取消为准，并保持单调：只有模型可见的表层 generation 变化后才重试。
|
||||
Post-step 压力描述已完成的路由请求，包括持久工具结果与仅请求前缀字段。可选的无模型剪枝会在选择摘要前移除可预测的工具输出体积，也能独立产生足以重试的进展。当成功 usage 锚点不存在时，规范化溢出提供兜底路径。恢复有明确上限、以取消为准，并保持单调：只有模型可见的表层 generation 变化后才重试。
|
||||
|
||||
代价是成功 step 增加一个串行检查点，并需要适配器持续维护溢出分类。提供方措辞与启发式字符密度仍是维护风险。表层压缩依然无法修复仅信封本身就超出窗口的情况，也不能拆分单个不可分割的超大消息或工具单元。
|
||||
代价是成功 step 增加一个串行检查点，并需要适配器持续维护溢出分类。提供方措辞与启发式字符密度仍是维护风险。表层压缩依然无法修复仅信封本身就超出窗口的情况，也不能拆分不可分割的非工具节点，或修复非可剪枝剩余部分仍然过大的工具单元。若可移除的文本工具结果是主要体积，可选剪枝器仍可修复原本不可分割的工具配对。
|
||||
|
||||
本 Agent Note 只取代[压缩能力接缝 Agent Note](../feature/2026-06-18-compaction-capability-seam.md) 中的 pre-step 自动触发部分。服务拆分、独立 token meter、平衡范围契约、日志记录锁、摘要替换与唯一 `summarize()` 子类 hook 均保持不变。
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-19-zstandard-jsonl-session-logs.md: 09d30594fe31eed138a128dabc1947b15857808d
|
||||
2026-07-19-zstandard-jsonl-session-logs.zh.md: 131531d9dba7cb01407191bf937f8b0ee3c6860a
|
||||
@@ -0,0 +1,57 @@
|
||||
# Agent Note: Zstandard JSONL session logs
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-07-19-zstandard-jsonl-session-logs.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The JSONL persistence backend keeps every `SessionEvent` verbatim, including high-volume `assistant/chunk` records. Raw text makes logs inspectable but spends storage and I/O on repeated JSON keys and model text. Compression must retain the existing append/fsync commit boundary, collision-safe first materialization, crash repair, and metadata-only listing; rewriting a whole compressed file after every turn would discard those properties.
|
||||
|
||||
The encoding also has to remain explicit at the deployment boundary. Snapshot fixtures and external line readers require raw JSONL, while a backend cannot safely guess between compressed and raw artifacts in one root or silently migrate pre-release session data.
|
||||
|
||||
## Decision
|
||||
|
||||
### Configuration and suffix ownership
|
||||
|
||||
`dsh-session-persistence-jsonl` accepts `compression?: 'zstd' | 'none'` and explicitly resolves omission to `'zstd'`. Zstandard artifacts end in `.jsonl.zstd`; `'none'` retains the original newline-delimited UTF-8 `.jsonl` representation. `SessionLocation.kind` remains `'jsonl'`, because both encodings carry the same logical record format, and `SESSION_FORMAT_VERSION` remains `0` under the repository's pre-release reject-without-migration policy.
|
||||
|
||||
Each persistence root belongs to one encoding. A one-time discovery preflight rejects any opposite suffix, and targeted load, live-adoption, listing, and materialization paths repeat the relevant suffix check after an initially empty preflight. The error names the incompatible artifact and directs the deployment to the matching configuration or a separate root. There is no migration, dual read, dual write, or extension-based fallback.
|
||||
|
||||
### Frame and write path
|
||||
|
||||
The compressed artifact is a standard concatenation of independent [Zstandard frames](https://datatracker.ietf.org/doc/html/rfc8878): one checksummed frame containing exactly the header line, followed by one checksummed frame for every durable append batch. Normal loop batches are turn commits, so frame boundaries preserve the existing persistence checkpoint without making the storage layer depend on turn event types.
|
||||
|
||||
Compression uses Node's built-in [`zstdCompress` and `zstdDecompress`](https://nodejs.org/download/release/v22.19.0/docs/api/zlib.html), available at the repository's Node 22.19 floor. The backend enables `ZSTD_c_checksumFlag`, otherwise accepts Node's defaults, and exposes neither a compression-level knob nor a new dependency. The API is marked experimental by Node, so the Node 22.19, 24, and 26 compatibility gate exercises the exact helper.
|
||||
|
||||
First materialization compresses the two initial frames before opening the temporary file, then keeps the existing write, file `fsync`, collision-safe hard-link publication, and directory `fsync` sequence. Later batches are compressed before opening the destination and appended at EOF. A caught write or file-sync failure truncates to the prior byte length, syncs the rollback, and rethrows so the coordinator can retry the unchanged batch.
|
||||
|
||||
### Read, listing, and crash recovery
|
||||
|
||||
A frame-boundary scanner reads the standard magic, variable header fields, block headers and payload sizes, and optional checksum trailer. It does not interpret compressed blocks. Complete frames are decompressed independently and sequentially, which validates their checksums, and their plaintext is passed to the existing JSONL scanner. A checksum/decompression failure in any complete frame, a malformed complete-frame JSONL tail, or invalid frame structure is corruption and rejects.
|
||||
|
||||
Listing reads in bounded chunks only until the first complete frame is available, validates and decompresses that header frame, and never reads an event frame. The dedicated header frame therefore preserves metadata-only listing even for very large session logs.
|
||||
|
||||
EOF inside the final frame is a recoverable torn tail. Node's decoder is given the available frame prefix; every complete newline-terminated event it emits is retained. Repair truncates from that frame's starting byte and appends one new checksummed frame containing the recovered complete events followed by the coordinator's synthetic tool, step, and turn closers. If the tear occurs before any complete event is decodable, repair drops the partial frame and retains all prior complete frames.
|
||||
|
||||
### Consumers and verification
|
||||
|
||||
The CLI, ACP, and stdio app bundles expose symmetric `persistenceCompression` pass-through configuration. Snapshot recording and replay compositions select `'none'` explicitly because committed fixtures are raw JSONL inputs to replay and normalization; ordinary runtime compositions use the compressed default.
|
||||
|
||||
The shared persistence and coordinator contracts run against both encodings. Backend tests cover standard framing and checksum interoperability, header-only listing, append rollback, encoding mismatch rejection, complete-frame corruption, and final-frame tears through headers, blocks, and checksum trailers. Default runtime, built-bin, headless, ACP, and Python smokes assert the compressed suffix and Zstandard magic or decode the header; raw-content tests opt out explicitly.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **One frame per JSONL record** — rejected because it multiplies frame headers and checksums for high-volume chunk events and makes a physical boundary unrelated to the durable append batch.
|
||||
- **Rewrite one whole compressed stream after every append** — rejected because cost grows with log size and replacement would give up append/fsync rollback and the established collision-safe materialization mechanics.
|
||||
- **Use a streaming compressor across appends** — rejected because an interrupted encoder state does not leave independently checksummed append units, complicating bounded listing and frame-start repair.
|
||||
- **Add an external native Zstandard dependency** — rejected because the supported Node floor already provides the required codec; another native artifact would enlarge installation and executable-packaging risk without adding a required behavior.
|
||||
- **Expose compression level or keep raw JSONL as the default** — rejected because there is no deployment evidence for a second tuning policy, while `'none'` preserves the line-readable path for fixtures and integrations that need it.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Ordinary session roots store `.jsonl.zstd` and retain append-only, fsync, rollback, and interrupted-turn recovery semantics.
|
||||
- Raw JSONL remains a deliberate configuration, but changing encoding requires a fresh/separate root or selecting the mode that matches existing artifacts.
|
||||
- One frame per durable batch adds bounded framing/checksum overhead and allows header-only listing plus repair from an exact append boundary.
|
||||
- External tools must understand concatenated Zstandard frames or consume raw-mode artifacts; generic one-shot Node decompression reads only the first independent frame, so backend reads walk frames explicitly.
|
||||
- The implementation depends on Node's experimental built-in Zstandard API without an npm dependency; the supported-version compatibility gate makes drift visible.
|
||||
@@ -0,0 +1,57 @@
|
||||
# Agent Note: Zstandard JSONL 会话日志
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-07-19-zstandard-jsonl-session-logs.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
JSONL 持久化后端会逐字保留每个 `SessionEvent`，其中包括数量庞大的 `assistant/chunk` 记录。原始文本便于检查，但重复的 JSON 键和模型文本会增加存储与 I/O 开销。压缩编码必须保留既有的 append/fsync 提交边界、首次物化时的无冲突发布、崩溃修复以及仅元数据列举；如果每轮都重写整个压缩文件，就会失去这些属性。
|
||||
|
||||
编码还必须在部署边界上保持显式。快照 fixture 与外部逐行读取器需要原始 JSONL，而后端无法在同一根目录中安全猜测压缩产物与原始产物，也不能静默迁移预发布会话数据。
|
||||
|
||||
## 决策
|
||||
|
||||
### 配置与后缀归属
|
||||
|
||||
`dsh-session-persistence-jsonl` 接受 `compression?: 'zstd' | 'none'`，并将省略值显式解析为 `'zstd'`。Zstandard 产物使用 `.jsonl.zstd` 后缀；`'none'` 保留原有的换行分隔 UTF-8 `.jsonl` 表示。`SessionLocation.kind` 仍为 `'jsonl'`，因为两种编码承载同一逻辑记录格式；按照仓库的预发布拒绝且不迁移策略，`SESSION_FORMAT_VERSION` 仍为 `0`。
|
||||
|
||||
每个持久化根目录只归属于一种编码。一次性的发现预检会拒绝任何相反后缀，而针对性的加载、活跃采用、列举与物化路径会在最初空目录预检之后再次执行对应后缀检查。错误会指出不兼容产物，并要求部署选择匹配配置或单独根目录。系统不提供迁移、双重读取、双重写入或基于扩展名的兜底。
|
||||
|
||||
### 帧与写入路径
|
||||
|
||||
压缩产物是标准独立 [Zstandard 帧](https://datatracker.ietf.org/doc/html/rfc8878)的串联：第一个带校验和的帧只包含头部行，后续每个持久追加批次各占一个带校验和的帧。正常 agent loop 批次就是轮次提交，因此帧边界保留既有持久化检查点，同时不让存储层依赖轮次事件类型。
|
||||
|
||||
压缩使用 Node 内置的 [`zstdCompress` 与 `zstdDecompress`](https://nodejs.org/download/release/v22.19.0/docs/api/zlib.html)，仓库最低支持的 Node 22.19 已提供这些 API。后端启用 `ZSTD_c_checksumFlag`，其余采用 Node 默认值，不公开压缩级别调节项，也不增加依赖。Node 将该 API 标记为实验性，因此 Node 22.19、24 与 26 兼容性门禁会执行同一个辅助实现。
|
||||
|
||||
首次物化会在打开临时文件之前压缩两个初始帧，然后保留既有的写入、文件 `fsync`、避免冲突的硬链接发布与目录 `fsync` 顺序。后续批次也会先压缩，再打开目标并在 EOF 追加。捕获到写入或文件同步失败时，后端会截断到原有字节长度，同步回滚结果，再重新抛出错误，让协调器重试未变化的批次。
|
||||
|
||||
### 读取、列举与崩溃恢复
|
||||
|
||||
帧边界扫描器会读取标准魔数、可变头字段、块头与负载长度，以及可选校验和尾部，但不会解释压缩块。后端独立且按顺序解压完整帧，由此验证各帧校验和，再把明文交给既有 JSONL 扫描器。任何完整帧的校验和或解压失败、完整帧中畸形的 JSONL 尾部，或者无效帧结构都属于损坏并拒绝加载。
|
||||
|
||||
列举只按有界分片读取到第一个完整帧可用为止，验证并解压该头部帧，绝不读取事件帧。因此，即使会话日志很大，专用头部帧仍能维持仅元数据列举。
|
||||
|
||||
最终帧内部遇到 EOF 属于可恢复的撕裂尾部。后端把已有帧前缀交给 Node 解码器，并保留其产出的每个完整、以换行结束的事件。修复从该帧起始字节截断，再追加一个新的带校验和帧，其中依次包含恢复出的完整事件，以及协调器生成的工具、步骤与轮次闭合事件。如果撕裂位置尚不足以解码任何完整事件，修复会丢弃该不完整帧并保留此前全部完整帧。
|
||||
|
||||
### 消费方与验证
|
||||
|
||||
CLI、ACP 与 stdio 应用包公开对称的 `persistenceCompression` 透传配置。快照录制与回放组合显式选择 `'none'`，因为提交的 fixture 是回放与规范化过程使用的原始 JSONL 输入；普通运行时组合使用压缩默认值。
|
||||
|
||||
共享持久化契约与协调器契约会针对两种编码运行。后端测试覆盖标准帧与校验和互操作性、仅头部列举、追加回滚、编码不匹配拒绝、完整帧损坏，以及横跨头部、块和校验和尾部的最终帧撕裂。默认运行时、构建后二进制、headless、ACP 与 Python 冒烟测试会断言压缩后缀与 Zstandard 魔数，或解码头部；读取原始内容的测试则显式退出压缩。
|
||||
|
||||
## 考虑过的替代方案
|
||||
|
||||
- **每条 JSONL 记录一个帧**——不予采纳，因为它会让大量分片事件各自承担帧头与校验和开销，并让物理边界脱离持久追加批次。
|
||||
- **每次追加都重写一个完整压缩流**——不予采纳，因为成本会随日志大小增长，而且替换操作会放弃追加/fsync 回滚和既有的无冲突物化机制。
|
||||
- **跨追加使用流式压缩器**——不予采纳，因为编码器状态中断后不会留下可独立校验的追加单元，从而使有界列举与按帧起点修复更复杂。
|
||||
- **增加外部原生 Zstandard 依赖**——不予采纳，因为受支持的 Node 最低版本已经提供所需编解码器；另一个原生产物会增加安装与可执行文件打包风险，却不增加必需行为。
|
||||
- **公开压缩级别或继续默认使用原始 JSONL**——不予采纳，因为没有部署证据支持第二种调节策略，而 `'none'` 已为需要逐行读取的 fixture 与集成保留路径。
|
||||
|
||||
## 后果
|
||||
|
||||
- 普通会话根目录存储 `.jsonl.zstd`，并保留仅追加、fsync、回滚与中断轮次恢复语义。
|
||||
- 原始 JSONL 仍是显式配置，但切换编码需要使用全新或单独根目录，或者选择与既有产物匹配的模式。
|
||||
- 每个持久批次一个帧会增加有界的帧与校验和开销，同时支持仅头部列举和从精确追加边界开始修复。
|
||||
- 外部工具必须理解串联的 Zstandard 帧，或者消费原始模式产物；Node 通用的一次性解压只读取第一个独立帧，因此后端读取会显式遍历各帧。
|
||||
- 实现依赖 Node 的实验性内置 Zstandard API，但不增加 NPM 依赖；受支持版本兼容性门禁会暴露 API 漂移。
|
||||
Reference in New Issue
Block a user