Merge branch 'master' into rfc/loop

This commit is contained in:
Tianyi Cui
2026-07-19 13:48:29 +08:00
committed by GitHub
925 changed files with 55463 additions and 13260 deletions

View File

@@ -0,0 +1,108 @@
# RFC: Recallable compaction — index checkpoints, a state checkpoint, and in-session history recall
Status: proposed
## Problem
Compaction is a one-way door. The summary the model sees carries no reference to what it shadows — the `shadowedRange` provenance lives only on the log-only `compact/summary` event — and no tool lets the model read a shadowed span back. Whatever the summarizer drops is gone from the model's reachable world, even though the append-only log holds every byte. Repeated compaction compounds this: the head checkpoint is rewritten every pass, so the request prefix takes a full prompt-cache miss each time, and earlier summaries are re-summarized generation after generation.
The root cause is one artifact playing two conflicting roles. An **index** wants to be frozen, chronological, and cheap; the model's **working memory** wants a global view, re-prioritization, and mutability. A single summary can be neither well.
No mainstream coding harness gives the model in-loop recall, and none of the surveyed implementations makes compaction prefix-cache-aware. An event-sourced session — originals durable, seq-addressable, replay-exact — is the natural substrate for both.
## Proposal
Split the checkpoint into two classes and make shadowed history reachable.
### Frozen index checkpoints
Newly stale history splits into chunks by deterministic policy: accumulate toward `chunkTokens`, snap edges with `toolPairingBalancedBefore` / `toolPairingBalancedAfter`, prefer turn boundaries, and place the final boundary as close to the retain boundary as balance allows, so the trailing slice shrinks to roughly one turn. Each chunk is compacted by one `compactRegion` call into an **index stub** (`stubTokens`, ~100–200 tokens):
- two or three lines of what happened;
- a keyword line of low-frequency literal anchors — exact error strings, values, config keys — grouped by kind;
- a code-composed footer: `[checkpoint c<summarySeq>: shadows conversation span #<start>–#<end>; originals retrievable via history_read]`. Pointers are assembled from provenance, never model-authored.
A committed stub is never rewritten and never re-enters a later compaction region. A stub call's input is layered: the fixed preamble and the byte-identical pass-start state checkpoint (the shared prefix across all calls in the phase), then the keyword lines of all previously committed stubs — so a new entry indexes what is distinctive to its chunk instead of repeating the directory — the one or two most recent committed stubs for chronological continuity, and the slice itself. Sibling stubs from the same pass are not inputs (the concurrent phase forbids it; turn-aligned boundaries carry local continuity instead), and the state checkpoint is background only, never material to summarize into the stub. A slice consisting of recalled content is stubbed by code alone — a pointer line, no LLM call. A failed stub call degrades the same way: its slice gets a code-only pointer stub and the pass continues, making the state rewrite the only hard LLM dependency in a pass.
### The state checkpoint
One mutable working-memory document (at most one; zero before the first pass), positioned after all stubs and before the retained tail. Each pass rewrites it from the previous state plus this pass's staled content — O(previous + new), under the merge-don't-restate rule already in the summarization prompt — covering decisions, current state, constraints, and next steps. It carries its own footer and a size cap at the scale of today's summary.
An inflation guard bounds the whole pass: if the post-compaction size is not strictly below the pre-compaction size, nothing commits and the turn proceeds; the attempt defers until more stale history accumulates. The guard compares one metric on both sides — provider-reported usage from the request path, falling back to the character estimator on both sides.
### Pass execution
- Chunk slices are surface position ranges. A pass runs two phases: all summarize calls execute concurrently, buffered off-surface; then regions commit strictly left to right — chunks first, trailing slice last — so the state checkpoint lands after every stub through contiguous single-node replaces. Wall-clock stays near one summarize call.
- The superseded state checkpoint folds into the next pass's first chunk as ordinary history: no tombstone, no new primitive. Its stub omits it, `history_read` renders it labeled `[prior state checkpoint]`, and its footer travels with the rendered text, keeping every trailing slice reachable through the two-hop chain.
- Range selection is frozen-aware: the compactable span begins after the last committed index checkpoint, at the surface head only when none exists. A legacy session's existing head checkpoint is adopted as state-class — its text the merge base, its node folded like any superseded state.
- A crash in the summarize phase commits nothing; a crash mid-commit leaves a left-to-right prefix committed, and the resumed pass reads its merge base from the log's latest state-class `compact/summary` event and commits the remaining regions unconditionally — restoring `[stubs…][state][tail]` outranks shrinking.
### The recall tools
A new package `@deepseek-ai/dsh-tool-recall` (consumer-only, over the `dsh-session` and `dsh-compact` vocabularies) registers two model-facing tools:
- `history_read(checkpoint, offset?)` — renders the shadowed span of any checkpoint in the log, including superseded ones, as `User:`/`Assistant:`/`Tool result:` transcript, paginated by a configured budget with a continuation cursor.
- `history_search(query, checkpoint?, limit?)` — case-insensitive literal scan over every shadowed span; returns snippets with checkpoint ids and coverage metadata (`scanned`/`matched`/`truncated`). The zero-match hint notes the scan is literal and points at direct `history_read` of a plausible checkpoint.
Both read `exec.agent.session.events` (the tool-todo access pattern; non-agent callers rejected), render only surface-type message events, and return ordinary `tool/result`s — recalled bytes land at the context tail, logged, so reconstructability holds with no special casing. There is no new storage and no sidecar index: the session log is the archive, `compact/summary` provenance is the index metadata, and the tools are a read path over both. The tool schemas and the package's one system-prompt section are static strings; checkpoint ids reach the model only through footers. The transcript renderer moves from `compact-basic` into `dsh-session`, shared by summarizer and tools.
### Cache and cost
The request prefix after a pass is `[system][stubs…][state][tail]`. Frozen stubs are byte-stable across passes, so the miss begins at the token replacing the previous state checkpoint and stays O(new chunks + state + tail) — against position zero today. Recall output lands at the tail, leaving the prefix untouched. Per-pass summarize input is roughly twice today's plus an m·S background term, bounded by a `chunkTokens` floor (a small multiple of the state cap) and a validated `stubTokens`/`chunkTokens` ratio ceiling; a shared-prefix input layout (preamble, then the byte-identical pass-start state, slice content in the tail) lets sibling calls earn cached-rate rereads.
### Packaging
The design ships as a new backend `dsh-compact-recallable` on the existing `ctx.compact` seam, enabled by default in the shipped example configs; `compact-basic` remains as the reference implementation and the seam's design twin, in the pattern of the paired LLM adapters. The seam JSDoc's "at most one auto-generated checkpoint, always at the head" clause is relaxed to name both backend behaviors.
### Relation to in-flight work
- **Tool-result pruning** (the in-flight pruning service): its replacement nodes carry `sourceEventSeqs`; the same registry fold lists pruned results as recallable. Follow-up scope; neither blocks the other.
- **Provider-usage token accounting** (the in-flight move of compaction pressure onto provider-reported usage): supplies the guard's accounting; the implementation stacks after it.
- **"Query sessions" backlog item**: the cross-session generalization; this RFC scopes to the live session with tool names and rendering chosen so that work extends rather than collides.
- **Training**: when to recall is a learned behavior. The deterministic footers and keyword anchors give training a stable target, and recall usage is fully visible in the session log for trajectory export; benchmark and RL design proceed with the post-training side.
### Follow-ups
Specified during review, deferred until observation calls for them:
- Guard degradation ladder (code-only rollup of the oldest stub prefix, footers preserved, rolled-up ids remain recall targets; then one summary after the frozen boundary) — on observed guard livelock or stub-region pressure.
- Echo detection on stub outputs (sentence-scale n-grams, short literals exempt, retry then strip) — on observed division-of-labor leakage.
- Periodic state refresh from chunk originals — on observed drift in the handoff probe.
- `stateFallbackThreshold` (full-detail state prompt below a stub count) — on short-session regression.
- Lazy registration of the recall tools — on measured context tax in never-compacting sessions.
- Amortized stub drafting at pre-step: as soon as stale-but-uncompacted content accumulates past `chunkTokens`, draft that chunk's stub at the next pre-step (a log-only draft event, written while the chunk's surrounding context is still live) and let the compaction pass commit drafts instead of summarizing in bulk — the deterministic, replay-exact equivalent of background compaction (the Claude Code session-memory pattern; OpenClaw demonstrates the synchronous semantics are identical). Trigger: observed pass latency, or stub-quality gains from drafting near-live proving out.
- Split summarizer models; model-chosen chunk boundaries; cross-session recall; semantic search fallback — each behind its own evidence.
- Richer `history_search` query forms — regex, and structured queries over logged JSON tool results (sql/jq-style, or agent-authored queries against an indexed store) — on demand from observed search misses; literal matching ships first because the recall path stays a pure function of the log.
## Alternatives considered
- **Staged delivery** (ship recall tools alone over today's backend; gate the checkpoint split on observed recall usage) — rejected: untrained models under-use any new tool, so the gate would measure training absence rather than design value, while the training side needs the complete mechanism to build environments against; the pre-release window is when persisted-format changes are cheapest; and the cache economics are first-party knowledge, not a hypothesis awaiting telemetry. The implementation still lands as stacked PRs with the recall tools first — construction order, not a decision gate.
- **All-frozen full-size summaries, no state checkpoint** — rejected: unbounded permanent-prefix growth, self-accelerating toward thrashing, with nothing left to re-prioritize.
- **Pure stubs, no state checkpoint** — rejected: presumes the model knows what it is missing; fails on unknown unknowns.
- **LLM aging/consolidation of frozen chunks** — rejected as a routine mechanism: summary-of-summary loss and frozen-prefix churn; the code-only rollup is its surviving form, deferred.
- **Full prefix as chunk-summarizer input** — rejected: O(N²); the state document gives the same background at O(state).
- **One summarize call emitting all outputs** — rejected: the summarize path has no structured-output enforcement; parsing one free-text response apart is the fragile seam the fail-closed design avoids.
- **Model-chosen chunk boundaries** — deferred: parse-and-validate cost against unproven value; chunk policy sits behind config.
- **Model-authored pointers** — rejected: pointers must be exact; deterministic assembly is.
- **FTS/vector index sidecar** — rejected in-session: the live log is in memory and bounded, a literal scan under budget suffices; an index earns its keep at cross-session scope.
- **Semantic search fallback / secondary-model extraction in the recall path** — rejected: an LLM or embedding call there breaks keyless replay determinism; recall stays a pure function of the log.
- **Raw events instead of rendered transcript** — rejected: leaks log-only vocabulary and chunk noise; the model reads what a model once saw.
- **Doing nothing (resume/fork as recovery)** — rejected: it makes recovery a human act.
## Acceptance criteria
- Auto-compaction over a long session yields `[stubs…][state][tail]` after every completed pass; prior stubs stay byte-identical across passes; committed stubs never fall inside a later region; the superseded state checkpoint folds without a tombstone, renders labeled, and stays reachable and searchable through the two-hop chain.
- Every checkpoint's surface text ends with the deterministic footer; footers round-trip through replay byte-identically; the state checkpoint's provenance records its wider input range.
- Nothing commits before all summaries exist and the guard passes on like-for-like accounting; a guard failure commits nothing and does not fail the turn; a mid-commit kill resumed at the next pre-step completes the pass with the state region committed unconditionally, merge base read from the log; a legacy head checkpoint is adopted as state-class.
- `history_read` renders any logged checkpoint's span under budget with a working cursor; `history_search` covers every shadowed span with checkpoint-id snippets and coverage metadata, asserted in particular by finding content that exists only in a span shadowed by a superseded state checkpoint — the regression pin for trailing-slice reachability; both reject non-agent callers and never-existing ids or orphaned `compact/start` with typed errors; recalled content appears as ordinary `tool/result`s; request-reconstruction invariants pass over sessions with compaction plus recall; one keyless snapshot scenario covers compact-then-recall end to end; tool schemas and the prompt section are byte-identical across passes.
- On the long-horizon bench suite: task success does not regress against `compact-basic` at equal budgets; a handoff-fidelity probe (restate K known decisions and constraints after a pass) scores no worse; recall usage frequency and hit usefulness are reported per run via the dsh bench report pipeline, alongside the stub-directory attention measurement and cache-hit telemetry.
- Seam JSDoc, the compaction capability-seam RFC, `architecture.md`, and the generated tool, config, persistence, and module-graph catalogs update in the same change; all budgets live in config; new source directories hold per-file 100% coverage with HMR disposal tests.
## Risks
- **Recall is a learned behavior**: untrained models will under-use it, and the bench report exists to track the gap while training closes it. Until then the state checkpoint keeps the floor at today's summary quality.
- **Unknown unknowns remain**: a detail absent from summaries and keywords draws no recall. Recall converts "unreachable even when suspected" into "reachable when suspected".
- **The stub directory occupies attention**: dozens of stable index cards per request may dilute focus; the bench measurement in the acceptance criteria tracks it against `compact-basic`.
- **Cost**: per-pass summarize input is roughly twice today's; short sessions sit near today's cost and quality, and the design pays off with session length.
- **State drift and division-of-labor leakage** are observable through the handoff probe and stub review; their counters are specified follow-ups.
- **Two backends** are a maintenance surface; the seam contract and the shared recall consumer bound it, and the bench comparison decides the default over time.

View File

@@ -4,7 +4,7 @@ Status: proposed
## Problem
Add isolated subagent providers for Claude Code and Codex. The existing [named-provider seam](../../implemented/feature/2026-06-21-subagent-capability-seam.md) and [ACP backend](../../implemented/feature/2026-06-22-acp-subagent-backend.md) establish the process-boundary shape. A harness turn should be able to delegate a self-contained task to either product and receive its final answer without exposing parent secrets or inheriting host configuration from `~/.claude` or `~/.codex`.
The subagent seam ([the seam RFC](../../implemented/feature/2026-06-21-subagent-capability-seam.md)) hosts multiple named providers on `ctx.subagents`, and the ACP backend ([the ACP backend RFC](../../implemented/feature/2026-06-22-acp-subagent-backend.md)) proved the seam generalizes across a process boundary; its Future-providers section explicitly named the Codex app-server and the Claude Code Agent SDK as mechanically similar siblings. Those two are the engines actually worth delegating to today: a harness turn should be able to hand a self-contained task to a real Claude Code or a real Codex — a separate product with its own model, tools, and sandbox — and get back one final answer, without the parent deployment leaking its secrets into the child or the child's behavior silently depending on whatever `~/.claude` / `~/.codex` state exists on the host machine.
## Proposal
@@ -12,9 +12,9 @@ Two sibling provider packages, structural variants of the ACP backend, plus one
- `@deepseek-ai/dsh-subagent-claude-code` — drives a Claude Code child through `@anthropic-ai/claude-agent-sdk`'s `query()` (the SDK runs in the parent process and spawns its bundled `claude` CLI as the subprocess). Provider name `claude-code`: the child is the Claude Code *product*, not an Anthropic model adapter — "claude" stays reserved for a future `dsh-llm` adapter.
- `@deepseek-ai/dsh-subagent-codex` — spawns `codex app-server` and drives one thread/turn over its JSON-RPC-over-stdio protocol with a hand-rolled newline-JSON client (~200–300 lines) in the package.
- `@deepseek-ai/dsh-subagent-process` — a pure library (the `subagent-inprocess` precedent) extracting what `dsh-subagent-acp` already carries and both new backends need: the credential env scrub (`SENSITIVE_ENV_PATTERN`/`buildChildEnv`), the EOF → SIGTERM → SIGKILL dispose ladder, and new isolated-config-dir helpers (`mkdtemp` create, best-effort remove). The ACP backend migrates onto it; `bash-local`'s sibling copy is left alone to bound the change.
- `@deepseek-ai/dsh-subagent-process` — a pure library (the `subagent-inprocess` precedent) extracting what `dsh-subagent-acp` already carries and both new backends need: the credential env scrub (`buildChildEnv`), the EOF → SIGTERM → SIGKILL dispose ladder, and new isolated-config-dir helpers (`mkdtemp` create, best-effort remove). The ACP backend migrates onto it; `bash-local`'s sibling copy is left alone to bound the change.
Both providers follow the ACP backend contract: a fresh child per `start`, one prompt round-trip, no inherited parent context or advertised optional capabilities, ignored `request.parent` and `request.agentOptions`, and a random branded agent id. `result` never rejects; child failures map to stop reasons while the original error reaches the logger. Each mounts `dsh-tool-subagent` under a distinct tool name. The tool result is the only new model-visible artifact, so no new session event is required; workspace mutations remain ambient side effects outside transcript replay.
Both providers copy the ACP backend's seam posture verbatim: fresh child per `start`, exactly one prompt round-trip, capabilities all `false`, `inheritsParentContext: false`, `request.parent`/`request.agentOptions` ignored, `id = SessionId(randomUUID())`, `result` never rejects — child-level failure flattens to a stop reason and the original error goes to `ctx.logger` via an `onError` spec callback. Model exposure is zero new code: `dsh-tool-subagent` is loaded once per provider with a distinct `toolName` (`subagent_claude_code`, `subagent_codex`). No new session events are needed — the only model-visible artifact is the tool result, so reconstructability holds exactly as it did for ACP. To be explicit about the boundary: the session log reconstructs the model-visible transcript, not workspace mutation history — a child granted write access mutates files as an ambient side effect outside the log, exactly as the bash tools and the ACP backend already do; replay reproduces requests, not the disk.
## Verified interface facts (pinned versions)
@@ -31,11 +31,11 @@ Both integration surfaces were verified against pinned implementations before th
## Isolation and credentials
Authentication is API-key-only. Each run uses a fresh config directory (`CLAUDE_CONFIG_DIR` with `settingSources: []`, or `CODEX_HOME`) that is removed best-effort on dispose; config may instead select a persistent directory. The shared child-env helper forwards ordinary values such as `PATH`, `HOME`, `TMPDIR`, locale, and proxy settings, removes credential-shaped names, and overlays explicit `config.env`. Claude Code receives its API key through that overlay, while Codex receives it through `account/login/start` rather than a hand-written auth file.
Deployments authenticate with API keys only, and the child must not see the host user's Claude Code / Codex configuration: behavior has to be a function of `cordis.yml` alone. Each run gets a fresh `mkdtemp` config dir — `CLAUDE_CONFIG_DIR` for Claude Code (paired with an explicit `settingSources: []`), `CODEX_HOME` for Codex — removed best-effort on dispose; a config field can pin a persistent dir instead. The child env reuses the ACP backend's `buildChildEnv` semantics verbatim via the extraction: the ambient env is forwarded MINUS credential-shaped vars (`/KEY|SECRET|TOKEN/i`), with `config.env` layered on top — so `PATH`, `HOME`, `TMPDIR`, locale, and proxy vars survive and the CLIs run normally, while only credential-shaped ambient vars are scrubbed (`ANTHROPIC_API_KEY` enters explicitly through `config.env` for Claude Code), and the Codex key travels via the `account/login/start` RPC into the isolated `CODEX_HOME` rather than a hand-written `auth.json`.
## Permission and approval policy
Each backend exposes its engine's native policy vocabulary. Claude Code defaults to `permissionMode: default` with `permission: reject`; Codex defaults to `sandboxMode: read-only`, `approvalPolicy: never`, and the same rejected fallback. Examples opt into `acceptEdits` or `workspace-write`. Known approval, user-input, and elicitation requests receive the configured answer; unknown methods receive method-not-found and unknown notifications are consumed. No prompt reaches a human, and no child can wait indefinitely for unavailable input.
Instead of collapsing to ACP's single `permission: allow|reject` knob, each backend exposes its engine's native vocabulary as config, with conservative defaults: Claude Code gets `permissionMode` (default `default`) plus `permission: allow|reject` (default `reject`) as the `canUseTool` auto-answer for whatever falls through; Codex gets `sandboxMode` (default `read-only`) and `approvalPolicy` (default `never`) plus the same `permission` fallback for approval requests that still arrive. Defaults are deliberately do-no-harm (the out-of-box child cannot write files); examples demonstrate opening up (`acceptEdits` / `workspace-write`). The mechanical rule: EVERY server-initiated request is settled programmatically and promptly — the enumerated approval/user-input/elicitation requests by the configured policy, an unknown request method with a JSON-RPC method-not-found error response (never left pending), unknown notifications consumed — so no child request can wedge a turn waiting on an answer that will never come. Prompts never reach a human in this cut, matching ACP.
## StopReason mapping
@@ -45,11 +45,11 @@ Liveness posture, stated explicitly: teardown timing is config, turn duration is
## Testing
Coverage is required at each applicable tier:
Named at every tier per the root AGENTS.md rule, and de-risked up front:
- **Keyless unit/integration:** drive a fake Claude CLI through the real SDK and a scripted Codex app-server through the real wire client. At per-file 100% coverage, exercise round trips, every stop mapping, both cancellation paths and pre-abort, permission policies, unknown messages, spawn failure, reload cleanup, export shape, scrubbed environments, temporary-directory removal, and Codex auth precheck failure.
- **With-key e2e:** each real engine performs file work under `acceptEdits` or `workspace-write`; skips name the missing binary or key and assert no child process remains.
- **Snapshot:** deferred as `TODO(claude-code-subagent-replay)` and `TODO(codex-subagent-replay)` pending the process-specific replay shape described by the [subagent replay RFC](../../implemented/testing/2026-06-22-subagent-snapshot-replay.md).
- **Keyless unit/integration**, mirroring the ACP spec list per backend (round-trip and output accumulation, every stop mapping, both cancel paths, already-aborted, permission auto-answer under both policies, unknown-message tolerance, bad-command spawn failure, HMR provider cleanup, export shape, isolation assertions on child env and temp-dir removal; Codex adds the auth-precheck failure path). Claude Code's harness is a scripted fake `claude` executable behind `pathToClaudeCodeExecutable` driven by the REAL SDK — a spike already passed end-to-end keyless in 24ms (the fake CLI answers one `control_request/initialize` and speaks plain stream-json, ~40 lines). Codex's harness is a scripted mock app-server subprocess speaking the verified wire protocol, the `mock-acp-server.ts` shape.
- **With-key e2e** per backend: the real engine does real file work verified on disk, under a pinned opened-up config so acceptance and the do-no-harm defaults don't collide — `permissionMode: 'acceptEdits'` for Claude Code, `sandboxMode: 'workspace-write'` + `approvalPolicy: 'never'` for Codex; self-skips report exactly what is missing (binary vs key). CI has no secrets, so these run locally per the with-key policy.
- **Snapshot**: deferred as `TODO(claude-code-subagent-replay)` / `TODO(codex-subagent-replay)` — the same distinct replay shape the ACP backend deferred ([the per-session replay RFC](../../implemented/testing/2026-06-22-subagent-snapshot-replay.md)); the keyless suites carry deterministic coverage meanwhile.
## Alternatives considered

View File

@@ -4,7 +4,7 @@ Status: proposed
## Problem
The exact-read `ctx.sessionQuery` service deliberately has no derived index. Large persisted histories need full-text search without scanning every event on every query, while current live sessions need an overlay newer than the last durability checkpoint. Search also needs concrete ranking, snippets, filters, pagination, cancellation, and rebuild behavior.
The exact-read `ctx.sessionQuery` service deliberately has no derived index. Large persisted histories need full-text search without scanning every event on every query, while current live sessions need an overlay newer than the last durability checkpoint. Search also needs concrete ranking, snippets, pagination, cancellation, and rebuild behavior.
Splitting those concerns across a speculative provider coordinator and a database implementation would create two coupled reconciliation state machines. The first real implementation should own the source observation, extraction, SQLite transaction, generation, and query as one lifecycle.
@@ -20,7 +20,7 @@ Persisted documents survive restarts. Live overrides are connection-local and sh
The implementation must define both cross-session and within-session scopes from executable use cases. Each searchable event is one document with session metadata, event metadata, surface classification, normalized semantic text, and a bounded plain-text snippet. Session results group by their strongest matching event; numeric backend scores remain private.
Filters compile to parameterized SQL before ranking. Query syntax is treated as data. Ordering includes stable tie fields. Opaque cursors bind to normalized request shape and the smallest relevant generation; unrelated session changes should not invalidate a within-session cursor. Cancellation must stop caller waiting and interrupt SQLite work where the runtime permits.
Search returns content-bearing result records rather than metadata-only headers. Chainable filters operate on that exact result shape and are designed and implemented with the search API instead of becoming a provider-specific pre-ranking contract. Query syntax is treated as data. Ordering includes stable tie fields. Opaque cursors bind to normalized request shape and the smallest relevant generation; unrelated session changes should not invalidate a within-session cursor. Cancellation must stop caller waiting and interrupt SQLite work where the runtime permits.
Tokenizer choice remains an implementation experiment. FTS5 trigram supports substring recall but rejects useful terms shorter than three characters and increases index size; the proposal must benchmark that tradeoff against the default Unicode tokenizer before making it contract.
@@ -41,7 +41,7 @@ Reconciliation may use stable fingerprints to avoid rewriting unchanged persiste
- Restart tests cover unchanged, new, changed, and deleted persisted sessions without rebuilding the whole index.
- Reopening preserves persisted rows and removes live rows; live rows shadow and then reveal their persisted base.
- Tests cover both search scopes, metadata filters, surface defaults, snippets, escaping, deterministic ties, pagination, scoped stale cursors, cancellation, dynamic persistence mount/unmount, and recovery after a failed transaction.
- Tests cover both search scopes, content-bearing results, chainable result filters, surface defaults, snippets, escaping, deterministic ties, pagination, scoped stale cursors, cancellation, dynamic persistence mount/unmount, and recovery after a failed transaction.
- A schema mismatch resets only the derived database.
- A keyless end-to-end test combines a real persistence backend with the real SQLite search package.
- The RFC is amended to the measured tokenizer and public API actually implemented before moving to `implemented/`.

View File

@@ -4,7 +4,7 @@ Status: proposed
## Problem
Package and gate inventories are repeated across TypeScript project references, package docs, CI prose, Knip overrides, and snapshot scenario metadata. Most restate package layout, manifest data, aggregate command contents, or fixture files. Each new package or scenario therefore creates avoidable synchronization points.
Package and gate inventories are repeated across TypeScript project references, package docs, CI prose, and Knip overrides. Most restate package layout, manifest data, or aggregate command contents. Each new package therefore creates avoidable synchronization points.
The [package hierarchy](../../implemented/architecture/2026-06-20-package-hierarchy.md) already removed several of these by hand: `scripts/publint-all.ts` now derives its list from the `packages/<group>/<pkg>` layout, and the two `tsconfig` `paths` maps collapsed to one `@deepseek-ai/dsh-*` wildcard. What remains is the inventory that cannot be globbed away — chiefly `tsconfig.build.json`'s project `references`, which TypeScript requires as an explicit array (no wildcard form).
@@ -16,7 +16,7 @@ Make the remaining package/gate inventories discoverable. A single canonical sou
The hierarchy does not need to encode every fact about a package, but it should encode the broad maintenance policy: core/product packages, integrations, capability seams, and support/test/example packages should not all require a hand-maintained exception list before scripts can tell them apart.
Two of the cataloged items need no generator at all: folding the e2e entry glob into knip's default stanza deletes the per-package restatements outright, and `childSessions` can be discovered from each scenario's fixture directory, leaving the scenario table to declare only policy (`recorded`, `hasModelTurn`, `comparesLog`) — and even those track fixture-derivable facts today (`comparesLog` ⟺ the committed log has entries beyond its header line; `recorded` ⟺ `hasModelTurn` with no `replay.override.json` sibling), so each new scenario class keeps adding knobs the fixture directory already answers.
One cataloged item needs no generator at all: folding the e2e entry glob into knip's default stanza deletes the per-package restatements outright.
## Acceptance criteria
@@ -25,7 +25,6 @@ Two of the cataloged items need no generator at all: folding the e2e entry glob
- Docs describe the source of truth rather than repeating generated inventories.
- CI invokes the aggregate commands and lets those commands own their sub-gate lists.
- `knip.json` carries a per-package override only where it encodes real information (an extra entry file, an ignored dependency), never a restatement of the default stanza.
- Snapshot scenarios declare policy, not facts discoverable from their fixture directories.
## Risks

View File

@@ -1,40 +0,0 @@
# RFC: Unify the agent id and the session id
Status: proposed
## Problem
The agent factory carries two ids for each live agent/session pair: `agentId`, the `AgentRegistry` routing handle, and `sessionId`, the event-sourced and persisted-log identity. `CreateAgentOptions` takes both; `ResumeAgentOptions` takes `agentId` plus `resumeSessionId`; in-process subagents mint two independent UUIDs despite recording lineage separately.
ACP already uses the same value for both identities. They diverge for config-created agents, resumed sessions, and in-process children, but no production path reattaches one live agent to several sessions or drives one session through several agent ids. Stdio keeps `labelBySession` only to recover an agent label from session events, and hooks expose both values for authors to reconcile.
The [agent-scope runtime](../../implemented/architecture/2026-07-12-agent-scope-runtime-design.md) has no identity-specific reservation state: create and resume use one `AgentCreationTransaction`, and both registry entries use the same final-entry collision rule. Separate ids do not duplicate liveness, rollback, or quiescence machinery. Unification deletes one caller-supplied id, one UUID per in-process child, and the remaining translation paths without changing the transaction lifecycle; it also makes the live-agent registry enforce the session identity used by background-task ownership.
`Session` separately exposes `Session.id` and `Session.header.id` even though construction requires them to match. The durable boundary must validate the duplicate, and consumers must choose between two homes for one fact.
## Proposal
Use one id for the agent registry entry and `session.header.id`. `CreateAgentOptions` accepts one identity for both final entries; resume registers the agent under the resumed session id; subagent creation mints one combined id; and `Session` keeps one identity home. Preserve the current transaction, final-entry collision checks, exact-entry detach, rollback, and quiescence; remove only maps and fields whose sole job is translating between the ids.
The config-driven path must first settle its resume-or-create policy. Today it uses a stable agent label and a fresh UUID-suffixed session id to avoid colliding with an existing durable log on the next run. Under unification it must deliberately resume a fixed id, mint a fresh combined id, or expose that policy; implementation must not choose silently.
`agent/created` and `agent/disposed` remain outside this proposal. They are publication lifecycle events rather than identity aliases; removing them requires a separate production-consumer audit and decision.
## Alternatives considered
**Keep separate routing and log identities.** A stable configured agent label paired with a fresh conversation is a real use of the distinction. If that display or routing identity is required, reject this proposal and enforce session-id uniqueness explicitly instead of hiding the translation in another map.
## Acceptance criteria
- Agent create/resume and subagent creation carry one identity; `Session` stores it in one place.
- The creation transaction retains final-entry collision, exact-entry detach, rollback, and quiescence guarantees without identity-specific lifecycle state.
- ACP, stdio, hooks, bash ownership, persistence, and lineage need no agent/session-id translation.
- The config-driven resume-or-create policy is explicit and covered across a durable restart.
- `agent/created` and `agent/disposed` change only after a separate production-consumer audit.
- Typecheck, coverage, snapshots, doc-sync, module-graph verification, build, and hygiene pass.
## Risks
Unification forecloses a stable actor identity spanning several session logs, including a future handoff or fork that preserves the actor while changing the session. Reintroducing that design would require a new explicit actor identity. It also makes a persisted, possibly client-chosen session id the registry handle and changes every create/resume call site and fixture.
The config restart policy is the blocking design decision: a fixed combined id may collide with its existing log, while a per-run id gives up the stable configured label. If either independent actor identity or the stable-label/fresh-session pairing is required, reject this proposal and retain the separate ids with an explicit uniqueness guard.

View File

@@ -25,6 +25,7 @@ The production corpus is `packages/*/*/src`, example sources/config, and runtime
| `CompactionResult.startSeq`, `summarySeq`, `endSeq`, and `summary` | The production consumer reads only shadowed range/seq/token accounting; the durable log owns summary and event identity. | Remove the four result echoes while keeping both shared transcript renderers. |
| `BasicCompactService` estimation/summarization visibility | No outside production caller invokes the five methods; the implemented RFC names only `estimateContentTokens()` and `summarize()` as subclass hooks. | Make those two `protected` and the three orchestration-only estimators private. |
| `CodeLogEntry.source`/`level` and `RunCodeMeta.dispatches` | Every production consumer maps logs to text; no presenter/model path reads the other fields or the persisted dispatch count. | Make code-runtime logs strings (or text-only entries) and remove result-meta dispatch plumbing; keep the local counter that mints deterministic dispatch ids. |
| `CodeRuntime.language` and `CodeRuntime.isolation` | The worker backend supplies the only production values, while Code Mode and every other production caller invoke only `run()`. | Remove the unread descriptors while preserving the worker's language, isolation, budgets, cancellation, and disposal behavior. |
| `ToolNotFoundError.toolName`, `SystemPrompt.config`, and `BashTask.command` | Each stored public value has no production reader. | Drop the unread field while retaining error messages, resolved configuration behavior, and task lifecycle. |
| Backend package-root implementation helpers | The exact inventory below is called only through relative same-package imports. Production namespace imports mount the retained plugin contract without reading these properties; named root consumers are tests. | Retain each adapter/provider/service and its config/error contract; stop exporting the listed helper functions/constants at package roots. |
| Consumer package-root implementation helpers | The exact inventory below has only same-package production callers. Production namespace imports mount plugin contracts without reading helper properties; named root consumers are tests. | Retain plugin contracts and stable error codes; move tests to package-local modules or public behavior and stop exporting the listed helpers at package roots. |

View File

@@ -1,36 +0,0 @@
# RFC: Simplify session-log representation
Status: proposed
## Problem
The session log maintains two representations that cost more machinery than their consumers require: a pseudo-linked surface and custom request-header deltas.
`SurfaceManager` stores the same order in an array, a seq map, and mutable `prev`/`next` links. Production never reads `prev`; compact's sole `next` read is the successor of an array position. Replacement already uses `indexOf`, so the links do not make its dominant operation constant-time. A seq array with linear replacement lookup has the same asymptotic replacement cost and one representation to validate.
The request-header subsystem implements a custom system/tool delta codec and transmission-decision layer even though its contract says deltas are an encoding optimization, not a reconstructability requirement. Retaining the initial/resume full snapshot at each loop-instance boundary, then writing a canonical full `request/header` whenever that instance's assembled header changes, preserves replay while deleting `SystemDelta`, `ToolsDelta`, round-trip fallback, and the durable `request/header-delta` variant. Codec-only vocabulary disappears with the codec, not because its individual arms were invalid.
This proposal deliberately retains append and replacement `sourceEventSeqs`, crash-repair provenance, and all `SessionStartSource` variants: implemented RFCs give those fields an audit/interception role that zero current readers does not overturn.
## Proposal
Make `SurfaceManager.nodes` a `readonly number[]` of event sequences and remove the public `SurfaceNode` shape. Keep the internal replace-generation signal; update tool-pairing balance and compaction callers to use array values/indices for predecessor, successor, and replacement ranges, removing node links and the seq-to-node map. Replace post-anchor header deltas with canonical full changed-header snapshots and remove the delta codec/event/tests; initial and resume anchors remain full snapshots even when the folded header is unchanged.
Amend the session-surface and reconstructable-request RFCs where they describe the removed encoding. Update event types/invariants, request logging/replay, persistence fixtures, generated catalogs, package docs, and snapshots. Replace the codec-only `fallback` reason with an explicit `change` reason for post-anchor full snapshots, distinguishing them from the retained `initial` and `resume` anchors.
`SESSION_FORMAT_VERSION` is deliberately pinned at `0`, so an old v0 log containing `request/header-delta` would otherwise pass the version check and silently lose header changes after the delta fold is deleted. Seed/load validation must reject that legacy event fail-loud at the format boundary; no compatibility fold or migration is added.
## Alternatives considered
**Keep linked nodes and compact deltas for possible scale.** Links could help a future cursor API, and deltas can reduce logs when large tool schemas change by a small amount. No shipped cursor uses the links, while full snapshots trade disk size for substantially simpler correctness. If header volume proves material, compression or a measured canonical-delta scheme can be designed around real traces.
## Acceptance criteria
- `SurfaceManager.nodes` is one ordered seq array with no `SurfaceNode`, link fields, or seq-to-node map; incremental append processing and the internal replace-generation signal remain.
- Replaying full changed-header snapshots reconstructs exactly the same requests; no header-delta event/type/codec remains.
- A v0 seed or persisted log containing legacy `request/header-delta` is rejected before replay, with coverage for JSONL and SQLite load paths.
- New-shape v0 JSONL/SQLite replay, provenance, crash repair, compaction, snapshots, invariants, typecheck, coverage, doc-sync, build, and hygiene pass.
## Risks
Full headers increase log volume, and linear replacement lookup could be slower on very large surfaces. Replacements are already linear because the implementation calls `indexOf`; benchmarks should be added only if real traces show the simpler array is a bottleneck. Because the format version remains `0`, forgetting the explicit legacy-event rejection would be silent data corruption rather than a type error; the fail-loud load test is therefore part of the proposal, not optional cleanup.

View File

@@ -0,0 +1,6 @@
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write
2026-07-19-make-jsonrpc-directional.md: 2a9579c53c111887e9d93cf02cc832304a496795
2026-07-19-make-jsonrpc-directional.zh.md: 4f3793f1b4de3b4f69c4219c10fe5b3b360c8692

View File

@@ -0,0 +1,46 @@
# RFC: Make JSON-RPC completion and transport directional
Status: proposed
English | [中文](2026-07-19-make-jsonrpc-directional.zh.md)
## Problem
The JSON-RPC bridge models both endpoints as symmetric peers although the shipped protocol is directional. The TypeScript server accepts requests and emits responses or notifications, but its transport also implements unused outbound requests and inbound notification dispatch. The Python SDK sends requests and receives responses or notifications, but it also queues unused inbound server requests and exposes response helpers.
`session/prompt` also reports one settled turn through two protocol shapes. The server emits `session.finished` and then returns the constant `{ accepted: true }`; the Python SDK discards that response and waits for the notification to recover the status. Because the response is written only after the handler returns, the notification necessarily precedes the constant response on the same stream.
The unused halves add pending-request maps, generated IDs, request queues, close-time rejection paths, response helpers, and a second completion waiter without serving a production caller.
## Proposal
Specialize each endpoint to its actual role. The TypeScript transport will retain inbound requests, outbound responses, and outbound notifications. The Python client will retain outbound requests and inbound responses or notifications. Delete the opposite-direction request machinery from each side.
Return the settled outcome directly from `session/prompt` as `{ status, reason }` after `agent.whenIdle()`. Delete `session.finished`, the constant acceptance response, and the Python post-response completion loop. `session.event` and subagent notifications still stream before the response, and durable session events remain the source for final-response reconstruction.
## Implementation plan
1. In `packages/ui/jsonrpc/src/server.ts`, replace `SessionPromptResult.accepted` with `status: 'ok' | 'error' | 'aborted'` and the captured `TurnEndReason`. `HarnessSdkServer.prompt()` will return `completed` as `ok`, `aborted` as `aborted`, and every other current or merge-extensible reason as `error`; reaching idle without a `turn/end` remains an invariant error. Remove only `session.finished`, leaving `session.event`, `subagent.started`, and `subagent.finished` unchanged.
2. In `packages/ui/jsonrpc/src/transport.ts`, replace `JsonRpcTransportPeer` with a server-side notification surface and retain `onRequest()`, `notify()`, `start()`, `flush()`, and `close()`. Remove generated request IDs, the pending-response map, outbound `request()`, inbound response and notification dispatch, and close-time pending-request rejection. Incoming response- and notification-shaped frames will be ignored, while request result, method-not-found, and handler-error responses retain their current behavior and remain ordered after notifications emitted by the awaited handler.
3. In `python/sdk/src/deepseek_harness/client.py`, `models.py`, and `__init__.py`, remove `IncomingRequest`, `_requests`, `notify()`, `next_request()`, `respond()`, and `respond_error()`. Add a public validated `SessionPromptResponse` carrying status and reason, return it from `session_prompt()`, and keep an explicit reader guard that ignores unexpected server-request frames instead of allowing them to match a response waiter.
4. In `python/sdk/src/deepseek_harness/api.py`, build `TurnResult.status` and a new `TurnResult.reason` from `SessionPromptResponse`, then delete the `session.finished` branch and second completion loop. Keep the subscription open during the request and preserve `_request_raw()`'s final notification drain so the last `turn/end` event and any subagent notification written before the response are collected before `Session.run()` reconstructs the final assistant message.
5. Replace the symmetric transport-pair cases in `packages/ui/jsonrpc/tests/transport.spec.ts` with raw client-input/server-output coverage, and update `server.spec.ts`, `plugin-apply.spec.ts`, and `built-scope-carrier.e2e.ts` for direct outcomes, ordering, overlap, shutdown, and the narrowed fake. Update `python/sdk/tests/test_client.py` for response-based settlement, unexpected-request-frame handling, callback and concurrency behavior, and the removed public helpers. Update the JSON-RPC and bilingual Python SDK READMEs, export JSDoc and declarations, `scripts/smoke-python-runtime.py`, and the Python single-executable snapshot.
## Alternatives considered
**Keep a generic symmetric JSON-RPC peer for future methods.** Server-initiated requests may eventually support interactive permissions, but no typed method or production consumer exists. The pre-release protocol can add the smallest required direction when that feature is designed instead of carrying an unexercised peer today.
**Keep `session.finished` for streaming clients.** Turn settlement is not incremental data: the request response already marks the same boundary and follows all earlier notifications on the ordered stream. A second terminal notification creates two representations that clients must reconcile.
## Acceptance criteria
- The TypeScript endpoint cannot originate requests or consume notifications.
- The Python endpoint cannot originate notifications or consume server requests.
- `session/prompt` returns the authoritative `ok`, `error`, or `aborted` outcome and reason after turn settlement.
- Session events and subagent lifecycle notifications emitted during the turn arrive before the response.
- Same-session overlap rejection, framing, multibyte input, handler errors, flush, shutdown ordering, and final-response reconstruction retain their behavior.
- TypeScript bridge tests, Python SDK tests, built JSON-RPC coverage, snapshots, and generated API documentation pass.
## Risks
This deliberately narrows the pre-release wire protocol. Raw clients listening only for `session.finished`, or embedders using the unused symmetric transport methods, must move to the prompt response. A future server-initiated request requires a new typed protocol addition rather than reusing generic dormant machinery.

View File

@@ -0,0 +1,46 @@
# RFC: 让 JSON-RPC 完成结果与传输方向单一化
Status: proposed
[English](2026-07-19-make-jsonrpc-directional.md) | 中文
## 问题
JSON-RPC 桥接层把两个端点都建模为对称的对等端,但实际协议具有固定方向。TypeScript 服务端接收请求并发出响应或通知,其传输层却还实现了未使用的出站请求和入站通知分发。Python SDK 发送请求并接收响应或通知,却还会把未使用的服务端入站请求放入队列,并公开响应辅助方法。
`session/prompt` 还会用两种协议结构报告同一个已结束轮次。服务端先发出 `session.finished`,再返回常量 `{ accepted: true }`;Python SDK 丢弃该响应,转而等待通知以取得状态。响应只有在处理函数返回后才会写入,因此在同一条有序流上,通知必然先于这个常量响应。
这些未使用的双向能力引入了待处理请求表、生成 ID、请求队列、关闭时的拒绝路径、响应辅助方法和第二套完成等待逻辑,却没有任何生产调用方使用。
## 提案
按实际角色收窄两个端点。TypeScript 传输层只保留入站请求、出站响应和出站通知。Python 客户端只保留出站请求以及入站响应或通知。删除两侧与实际方向相反的请求机制。
在 `agent.whenIdle()` 完成后,由 `session/prompt` 直接返回 `{ status, reason }` 作为轮次结果。删除 `session.finished`、常量接纳响应以及 Python 中响应后的完成等待循环。`session.event` 与 subagent 通知仍在响应前流式发出,持久会话事件仍是最终响应重建的真源。
## 实施计划
1. 在 `packages/ui/jsonrpc/src/server.ts` 中,用 `status: 'ok' | 'error' | 'aborted'` 和捕获的 `TurnEndReason` 替换 `SessionPromptResult.accepted`。`HarnessSdkServer.prompt()` 把 `completed` 映射为 `ok`,把 `aborted` 映射为 `aborted`,把其他当前或可合并扩展的原因映射为 `error`;进入空闲状态却没有 `turn/end` 仍视为不变量错误。只删除 `session.finished`,保持 `session.event`、`subagent.started` 和 `subagent.finished` 不变。
2. 在 `packages/ui/jsonrpc/src/transport.ts` 中,用服务端通知接口替换 `JsonRpcTransportPeer`,并保留 `onRequest()`、`notify()`、`start()`、`flush()` 和 `close()`。删除生成的请求 ID、待处理响应表、出站 `request()`、入站响应与通知分发,以及关闭时对待处理请求的拒绝逻辑。入站响应结构和通知结构将被忽略;请求结果、方法不存在与处理器错误响应保持原有行为,并继续排在被等待处理器发出的通知之后。
3. 在 `python/sdk/src/deepseek_harness/client.py`、`models.py` 和 `__init__.py` 中,删除 `IncomingRequest`、`_requests`、`notify()`、`next_request()`、`respond()` 和 `respond_error()`。新增公开且经过校验的 `SessionPromptResponse` 来携带状态与原因,由 `session_prompt()` 返回该对象,并保留明确的读取保护:忽略意外的服务端请求帧,避免它们命中响应等待器。
4. 在 `python/sdk/src/deepseek_harness/api.py` 中,根据 `SessionPromptResponse` 构造 `TurnResult.status` 和新增的 `TurnResult.reason`,再删除 `session.finished` 分支与第二个完成循环。请求期间保持订阅打开,并保留 `_request_raw()` 最后的通知排空步骤,确保写在响应前的最后一条 `turn/end` 事件与任何 subagent 通知,都会在 `Session.run()` 重建最终助手消息之前被收集。
5. 用原始客户端输入与服务端输出覆盖替换 `packages/ui/jsonrpc/tests/transport.spec.ts` 中的对称传输对用例,并更新 `server.spec.ts`、`plugin-apply.spec.ts` 和 `built-scope-carrier.e2e.ts`,覆盖直接结果、顺序、重叠、关闭和收窄后的伪实现。更新 `python/sdk/tests/test_client.py`,覆盖基于响应的结束流程、意外请求帧处理、回调与并发行为,以及已删除的公开辅助方法。同步更新 JSON-RPC README、双语 Python SDK README、导出 JSDoc 与声明、`scripts/smoke-python-runtime.py` 和 Python 单可执行文件快照。
## 备选方案
**为未来方法保留通用的对称 JSON-RPC 对等端。** 服务端发起的请求将来可能用于交互式权限,但当前没有类型化方法或生产消费方。该功能完成设计后,预发布协议可以增加所需的最小方向,无需提前保留未使用的对等端能力。
**为流式客户端保留 `session.finished`。** 轮次结束不是增量数据:请求响应已经标识同一个边界,并且在有序流中位于先前所有通知之后。第二条终止通知会产生两种结果表示,迫使客户端进行协调。
## 验收标准
- TypeScript 端点无法发起请求,也不消费通知。
- Python 端点无法发起通知,也不消费服务端请求。
- 轮次结束后,`session/prompt` 返回权威的 `ok`、`error` 或 `aborted` 状态及其原因。
- 轮次中发出的会话事件与 subagent 生命周期通知都先于响应到达。
- 同一会话的重叠拒绝、分帧、多字节输入、处理器错误、flush、关闭顺序与最终响应重建保持原有行为。
- TypeScript 桥接测试、Python SDK 测试、构建后 JSON-RPC 覆盖、快照和生成的 API 文档全部通过。
## 风险
本提案会刻意收窄预发布协议格式。仅监听 `session.finished` 的原始客户端,以及使用未使用对称传输方法的嵌入方,都必须改为读取请求响应。未来若需要服务端发起请求,应新增类型化协议,而不是复用休眠的通用机制。