fix(session): add semantic crash checkpoints

This commit is contained in:
Yichen Jiang
2026-07-21 14:50:06 +08:00
parent 9a5c81f9e5
commit 6d12e3ab41
56 changed files with 1016 additions and 61 deletions

View File

@@ -18,7 +18,7 @@ Persistence is an abstract **capability seam** ([capability seams](2026-06-13-ca
Key choices recorded here because they are durable, contested, and surprising:
- **The canonical durable log persists every `SessionEvent` verbatim, including `assistant/chunk`.** `deriveMessages()` skips chunks, and a chunk-filtered rollout (Codex's `policy.rs`) is tempting — but `seq = log.length` and the load-validation `events[i].seq === i` require a *contiguous* log; filtering chunks out would leave holes and break both the contract and resume. A chunk-filtered projection is possible later as a derived view with its own renumbering, but it is NOT the canonical log.
- **Append-only; a crashed turn is closed, never truncated.** Events through a flushed `turn/end` are never rewritten, and the loop flushes only at turn end. Because one interrupted turn may contain substantial valid work, `load` preserves its contiguous, parseable events and appends error results for unanswered tool calls, a missing `step/end`, and `turn/end` with `{ kind: 'interrupted' }`. The synthetic results keep resumed provider transcripts valid. Only an incomplete final record is discarded; a parse error or sequence gap at or before the last real `turn/end` is corruption and makes the session unloadable.
- **Append-only; a crashed turn is closed, never truncated.** Flushed events are never rewritten. The [semantic checkpoint policy](../bug-fix/2026-07-21-semantic-session-checkpoints.md) drains the request before model dispatch, a recorded top-level call before tool dispatch, and the complete response/result batch after a step; the loop drains the final turn boundary. Because one interrupted turn may contain substantial valid work, `load` preserves its contiguous, parseable events and appends risk-classified error results for unanswered assistant calls, a missing `step/end`, and `turn/end` with `{ kind: 'interrupted' }`. The synthetic results keep resumed provider transcripts valid. Only an incomplete final record is discarded; a parse error or sequence gap at or before the last real `turn/end` is corruption and makes the session unloadable.
- **File backend canonical, DB backend a proven drop-in.** `SessionEvent` maps 1:1 onto a row `(session_id, seq, type, time, data)``append` is INSERT (in a transaction asserting the contiguous-seq contract), `load` is SELECT … ORDER BY seq. `dsh-session-persistence-sqlite` is exactly this: a `SessionPersistence` subclass with no interface change (opencode runs this exact shape on SQLite/WAL), and it passes the same `runPersistenceContract` suite as the JSONL backend — so the contract holds both backends to identical semantics (lazy materialization, interrupted-turn close on load, contiguous-seq), expressed once over file bytes and once over rows.
- **Metadata is out-of-log.** Format version, cwd, and lineage are storage concerns, not replayable conversation state, so they live in a `SessionHeader` owned by `dsh-session` and attached to a `Session` via a new readonly `session.header` — never in `SessionEventMap`, never reaching `deriveMessages()`. The alternative (a merge-extensible `session/meta` event as log line 0) was rejected: an in-log event would ride along with a seeded/forked session for free, but metadata is not replayable state, so the explicit out-of-log header seam is the cleaner cost. (The header was originally split into an immutable `SessionHeader` plus a mutable `SessionSummary` whose union was `SessionMeta`; the mutable summary was later removed as dead state — see [Drop the mutable session summary](../simplification/2026-06-19-drop-mutable-session-summary.md).)
- **`ctx.agents.create()` and `ctx.agents.resume()` are async factories; resume additionally crosses the persistence boundary.** `ctx.agents.resume({ resumeSessionId })` awaits `ctx.sessionPersistence.load`, recreates the live session with the loaded events (so `lastTurnNumber`/`deriveMessages` continue), and registers the fresh agent under the exact resumed id. The agent-loop does NOT hard-inject `sessionPersistence` (that would pend non-persistent demos forever); `resume` rejects with a clear error when it is absent.

View File

@@ -0,0 +1,6 @@
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write
2026-07-21-semantic-session-checkpoints.md: e444e493bd0feface12088b7dba9227703fa147f
2026-07-21-semantic-session-checkpoints.zh.md: 503c768b5653e22c464c5892bd7f4336113d1e38

View File

@@ -0,0 +1,25 @@
# Agent Note: Semantic session checkpoints
Status: implemented
English | [中文](2026-07-21-semantic-session-checkpoints.zh.md)
## Problem
Persistence buffered every synchronous `session/event` until the loop's final turn checkpoint. A turn is the correct conversational transaction, but it is too coarse as the only crash-recovery point: a hard crash during a long model request or tool call could discard the whole in-flight turn, including the request envelope needed to identify what had been attempted. A tool call with no result was also repaired with one undifferentiated interruption error, so the resumed model could not tell whether execution had started and could retry a side effect blindly.
## Decision
`dsh-session-checkpoint-policy` owns semantic durability barriers as a zero-config plugin beside a persistence backend. It wraps `llm/stream` lazily and flushes the live session after `request/header` is logged but before the adapter stream is constructed. It wraps top-level `tools/execute` after ordered pre-execute policy and flushes the recorded `tool/call` before the tool body; nested dispatches reuse the outer model-visible call. It flushes at `agent/post-step` after the assistant message and ordered results are recorded. The loop's existing final `turn/end` checkpoint remains the closing boundary.
Checkpoint failure is fail-closed at effect boundaries. A rejected request checkpoint prevents adapter dispatch; a rejected tool checkpoint becomes an error result without invoking the tool body; a rejected post-step checkpoint stops continuation before another model request. Persistence serialization continues to belong to the coordinator, so concurrent tool checkpoints cannot duplicate event sequences.
Crash repair distinguishes durable evidence. An assistant tool request without a `tool/call` becomes `TOOL_NOT_STARTED` and may be retried if still needed. A durable `tool/call` without a result becomes `TOOL_OUTCOME_UNKNOWN`; its model-visible result permits retry only for read-only or idempotent operations and directs the model to verify external state or ask the user before deciding about side-effecting work. A provider that supports idempotency keys can receive the stable `callId`, but the Harness does not claim generic exactly-once effects.
## Alternatives considered
Flushing every event or streaming chunk minimizes loss but turns local append and `fsync` latency into the hot path and destabilizes streaming throughput. Moving the barriers into `agent-loop` centralizes the policy but makes one persistence strategy mandatory in the mechanism layer. Keeping turn-only flush preserves throughput but loses the request and execution intent needed for safe recovery. Automatically retrying every unmatched call is safe only for a subset of tools and can duplicate irreversible effects.
## Consequences
Hard-crash recovery retains the complete model request, durable tool intent, and complete settled step at the nearest semantic boundary while allowing partial streaming chunks since the previous boundary to remain lossy. Default CLI, TUI, ACP, headless persistence tests, and JSON-RPC compositions mount the policy with their persistence backend. Unit tests cover ordering, fail-closed behavior, nested dispatch, disposal, and Loader shape; a real child process killed with `SIGKILL` proves request and tool-intent recovery through JSONL, and the shared persistence contract proves both recovery classifications across backends. A keyless ACP snapshot loads a seeded unknown-outcome session through the shipped ACP example and proves that the retry-risk guidance reaches both resumed history and the next model turn.

View File

@@ -0,0 +1,25 @@
# Agent Note: 语义会话检查点
Status: implemented
[English](2026-07-21-semantic-session-checkpoints.md) | 中文
## 问题
持久化机制会缓冲所有同步 `session/event`,直到 agent loop智能体循环执行最后的轮次检查点才写入。一个轮次是正确的对话事务但作为唯一的崩溃恢复点过于粗粒度如果在耗时的模型请求或工具调用期间发生硬崩溃整个进行中的轮次都可能丢失其中包括识别已尝试操作所需的请求封套。系统还会使用同一种不作区分的中断错误修复没有结果的工具调用因此恢复运行的模型无法判断调用是否已经开始可能会盲目重试带有副作用的操作。
## 决策
`dsh-session-checkpoint-policy` 以零配置插件的形式与持久化后端共同加载,并负责语义持久性屏障。该插件惰性包装 `llm/stream`,在记录 `request/header` 之后、构造适配器流之前,刷新活动会话。该插件还在有序的执行前策略之后包装顶层 `tools/execute`,在进入工具主体前刷新已记录的 `tool/call`;嵌套分发则复用外层模型可见调用。它还会在 `agent/post-step` 时刷新会话,此时模型消息与按序结果都已记录。现有的最终 `turn/end` 检查点仍是轮次的收尾边界。
检查点失败在副作用边界上采取失败关闭策略。请求检查点被拒绝时,系统不会分发给适配器;工具检查点被拒绝时,系统会返回错误结果,不调用工具主体;步骤后检查点被拒绝时,系统会在发起下一个模型请求前停止继续执行。持久化写入的串行化仍由协调器负责,因此并发的工具检查点不会产生重复的事件序号。
崩溃修复会区分持久化证据。如果模型发出了工具请求,却没有 `tool/call`,系统会将其标记为 `TOOL_NOT_STARTED`;如果仍有需要,可以重试。如果持久化的 `tool/call` 没有结果,系统会将其标记为 `TOOL_OUTCOME_UNKNOWN`;对应的模型可见结果只允许重试只读或幂等操作,并指示模型在决定是否重试有副作用的工作前,先验证外部状态或询问用户。支持幂等键的模型提供方可以获取稳定的 `callId`,但 Harness 不承诺通用的副作用恰好执行一次保证。
## 考虑过的替代方案
刷新每个事件或流式分片虽能尽可能减少丢失,但会把本地追加与 `fsync` 延迟带入热路径,破坏流式输出的吞吐稳定性。将这些屏障放入 `agent-loop` 虽能集中管理策略,却会让某种持久化策略成为机制层的强制选项。仅保留轮次刷新可以维持吞吐量,但会丢失安全恢复所需的请求与执行意图。自动重试所有未匹配调用只对部分工具安全,可能会重复不可逆的副作用。
## 后果
发生硬崩溃时,崩溃恢复会在最近的语义边界保留完整的模型请求、持久化的工具意图与完整且已结束的步骤,但允许上一个边界之后的部分流式分片仍可能丢失。默认的 CLI命令行界面、TUI、ACPAgent Client Protocol、headless 持久化测试与 JSON-RPC 组合都会在持久化后端旁加载该策略。单元测试覆盖顺序、失败关闭行为、嵌套分发、dispose资源释放与 Loader 形状;一个被 `SIGKILL` 终止的真实子进程通过 JSONL 证明系统可以恢复请求与工具意图,共享持久化契约则证明各后端都支持这两种恢复分类。一项无密钥 ACP 快照通过已交付的 ACP 示例加载预置的结果未知会话,并证明重试风险指引会同时出现在恢复后的历史记录与下一个模型轮次中。