Merge branch 'master' into rfc/loop
This commit is contained in:
108
docs/rfc/proposed/feature/2026-07-06-recallable-compaction.md
Normal file
108
docs/rfc/proposed/feature/2026-07-06-recallable-compaction.md
Normal file
@@ -0,0 +1,108 @@
|
||||
# RFC: Recallable compaction — index checkpoints, a state checkpoint, and in-session history recall
|
||||
|
||||
Status: proposed
|
||||
|
||||
## Problem
|
||||
|
||||
Compaction is a one-way door. The summary the model sees carries no reference to what it shadows — the `shadowedRange` provenance lives only on the log-only `compact/summary` event — and no tool lets the model read a shadowed span back. Whatever the summarizer drops is gone from the model's reachable world, even though the append-only log holds every byte. Repeated compaction compounds this: the head checkpoint is rewritten every pass, so the request prefix takes a full prompt-cache miss each time, and earlier summaries are re-summarized generation after generation.
|
||||
|
||||
The root cause is one artifact playing two conflicting roles. An **index** wants to be frozen, chronological, and cheap; the model's **working memory** wants a global view, re-prioritization, and mutability. A single summary can be neither well.
|
||||
|
||||
No mainstream coding harness gives the model in-loop recall, and none of the surveyed implementations makes compaction prefix-cache-aware. An event-sourced session — originals durable, seq-addressable, replay-exact — is the natural substrate for both.
|
||||
|
||||
## Proposal
|
||||
|
||||
Split the checkpoint into two classes and make shadowed history reachable.
|
||||
|
||||
### Frozen index checkpoints
|
||||
|
||||
Newly stale history splits into chunks by deterministic policy: accumulate toward `chunkTokens`, snap edges with `toolPairingBalancedBefore` / `toolPairingBalancedAfter`, prefer turn boundaries, and place the final boundary as close to the retain boundary as balance allows, so the trailing slice shrinks to roughly one turn. Each chunk is compacted by one `compactRegion` call into an **index stub** (`stubTokens`, ~100–200 tokens):
|
||||
|
||||
- two or three lines of what happened;
|
||||
- a keyword line of low-frequency literal anchors — exact error strings, values, config keys — grouped by kind;
|
||||
- a code-composed footer: `[checkpoint c<summarySeq>: shadows conversation span #<start>–#<end>; originals retrievable via history_read]`. Pointers are assembled from provenance, never model-authored.
|
||||
|
||||
A committed stub is never rewritten and never re-enters a later compaction region. A stub call's input is layered: the fixed preamble and the byte-identical pass-start state checkpoint (the shared prefix across all calls in the phase), then the keyword lines of all previously committed stubs — so a new entry indexes what is distinctive to its chunk instead of repeating the directory — the one or two most recent committed stubs for chronological continuity, and the slice itself. Sibling stubs from the same pass are not inputs (the concurrent phase forbids it; turn-aligned boundaries carry local continuity instead), and the state checkpoint is background only, never material to summarize into the stub. A slice consisting of recalled content is stubbed by code alone — a pointer line, no LLM call. A failed stub call degrades the same way: its slice gets a code-only pointer stub and the pass continues, making the state rewrite the only hard LLM dependency in a pass.
|
||||
|
||||
### The state checkpoint
|
||||
|
||||
One mutable working-memory document (at most one; zero before the first pass), positioned after all stubs and before the retained tail. Each pass rewrites it from the previous state plus this pass's staled content — O(previous + new), under the merge-don't-restate rule already in the summarization prompt — covering decisions, current state, constraints, and next steps. It carries its own footer and a size cap at the scale of today's summary.
|
||||
|
||||
An inflation guard bounds the whole pass: if the post-compaction size is not strictly below the pre-compaction size, nothing commits and the turn proceeds; the attempt defers until more stale history accumulates. The guard compares one metric on both sides — provider-reported usage from the request path, falling back to the character estimator on both sides.
|
||||
|
||||
### Pass execution
|
||||
|
||||
- Chunk slices are surface position ranges. A pass runs two phases: all summarize calls execute concurrently, buffered off-surface; then regions commit strictly left to right — chunks first, trailing slice last — so the state checkpoint lands after every stub through contiguous single-node replaces. Wall-clock stays near one summarize call.
|
||||
- The superseded state checkpoint folds into the next pass's first chunk as ordinary history: no tombstone, no new primitive. Its stub omits it, `history_read` renders it labeled `[prior state checkpoint]`, and its footer travels with the rendered text, keeping every trailing slice reachable through the two-hop chain.
|
||||
- Range selection is frozen-aware: the compactable span begins after the last committed index checkpoint, at the surface head only when none exists. A legacy session's existing head checkpoint is adopted as state-class — its text the merge base, its node folded like any superseded state.
|
||||
- A crash in the summarize phase commits nothing; a crash mid-commit leaves a left-to-right prefix committed, and the resumed pass reads its merge base from the log's latest state-class `compact/summary` event and commits the remaining regions unconditionally — restoring `[stubs…][state][tail]` outranks shrinking.
|
||||
|
||||
### The recall tools
|
||||
|
||||
A new package `@deepseek-ai/dsh-tool-recall` (consumer-only, over the `dsh-session` and `dsh-compact` vocabularies) registers two model-facing tools:
|
||||
|
||||
- `history_read(checkpoint, offset?)` — renders the shadowed span of any checkpoint in the log, including superseded ones, as `User:`/`Assistant:`/`Tool result:` transcript, paginated by a configured budget with a continuation cursor.
|
||||
- `history_search(query, checkpoint?, limit?)` — case-insensitive literal scan over every shadowed span; returns snippets with checkpoint ids and coverage metadata (`scanned`/`matched`/`truncated`). The zero-match hint notes the scan is literal and points at direct `history_read` of a plausible checkpoint.
|
||||
|
||||
Both read `exec.agent.session.events` (the tool-todo access pattern; non-agent callers rejected), render only surface-type message events, and return ordinary `tool/result`s — recalled bytes land at the context tail, logged, so reconstructability holds with no special casing. There is no new storage and no sidecar index: the session log is the archive, `compact/summary` provenance is the index metadata, and the tools are a read path over both. The tool schemas and the package's one system-prompt section are static strings; checkpoint ids reach the model only through footers. The transcript renderer moves from `compact-basic` into `dsh-session`, shared by summarizer and tools.
|
||||
|
||||
### Cache and cost
|
||||
|
||||
The request prefix after a pass is `[system][stubs…][state][tail]`. Frozen stubs are byte-stable across passes, so the miss begins at the token replacing the previous state checkpoint and stays O(new chunks + state + tail) — against position zero today. Recall output lands at the tail, leaving the prefix untouched. Per-pass summarize input is roughly twice today's plus an m·S background term, bounded by a `chunkTokens` floor (a small multiple of the state cap) and a validated `stubTokens`/`chunkTokens` ratio ceiling; a shared-prefix input layout (preamble, then the byte-identical pass-start state, slice content in the tail) lets sibling calls earn cached-rate rereads.
|
||||
|
||||
### Packaging
|
||||
|
||||
The design ships as a new backend `dsh-compact-recallable` on the existing `ctx.compact` seam, enabled by default in the shipped example configs; `compact-basic` remains as the reference implementation and the seam's design twin, in the pattern of the paired LLM adapters. The seam JSDoc's "at most one auto-generated checkpoint, always at the head" clause is relaxed to name both backend behaviors.
|
||||
|
||||
### Relation to in-flight work
|
||||
|
||||
- **Tool-result pruning** (the in-flight pruning service): its replacement nodes carry `sourceEventSeqs`; the same registry fold lists pruned results as recallable. Follow-up scope; neither blocks the other.
|
||||
- **Provider-usage token accounting** (the in-flight move of compaction pressure onto provider-reported usage): supplies the guard's accounting; the implementation stacks after it.
|
||||
- **"Query sessions" backlog item**: the cross-session generalization; this RFC scopes to the live session with tool names and rendering chosen so that work extends rather than collides.
|
||||
- **Training**: when to recall is a learned behavior. The deterministic footers and keyword anchors give training a stable target, and recall usage is fully visible in the session log for trajectory export; benchmark and RL design proceed with the post-training side.
|
||||
|
||||
### Follow-ups
|
||||
|
||||
Specified during review, deferred until observation calls for them:
|
||||
|
||||
- Guard degradation ladder (code-only rollup of the oldest stub prefix, footers preserved, rolled-up ids remain recall targets; then one summary after the frozen boundary) — on observed guard livelock or stub-region pressure.
|
||||
- Echo detection on stub outputs (sentence-scale n-grams, short literals exempt, retry then strip) — on observed division-of-labor leakage.
|
||||
- Periodic state refresh from chunk originals — on observed drift in the handoff probe.
|
||||
- `stateFallbackThreshold` (full-detail state prompt below a stub count) — on short-session regression.
|
||||
- Lazy registration of the recall tools — on measured context tax in never-compacting sessions.
|
||||
- Amortized stub drafting at pre-step: as soon as stale-but-uncompacted content accumulates past `chunkTokens`, draft that chunk's stub at the next pre-step (a log-only draft event, written while the chunk's surrounding context is still live) and let the compaction pass commit drafts instead of summarizing in bulk — the deterministic, replay-exact equivalent of background compaction (the Claude Code session-memory pattern; OpenClaw demonstrates the synchronous semantics are identical). Trigger: observed pass latency, or stub-quality gains from drafting near-live proving out.
|
||||
- Split summarizer models; model-chosen chunk boundaries; cross-session recall; semantic search fallback — each behind its own evidence.
|
||||
- Richer `history_search` query forms — regex, and structured queries over logged JSON tool results (sql/jq-style, or agent-authored queries against an indexed store) — on demand from observed search misses; literal matching ships first because the recall path stays a pure function of the log.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Staged delivery** (ship recall tools alone over today's backend; gate the checkpoint split on observed recall usage) — rejected: untrained models under-use any new tool, so the gate would measure training absence rather than design value, while the training side needs the complete mechanism to build environments against; the pre-release window is when persisted-format changes are cheapest; and the cache economics are first-party knowledge, not a hypothesis awaiting telemetry. The implementation still lands as stacked PRs with the recall tools first — construction order, not a decision gate.
|
||||
- **All-frozen full-size summaries, no state checkpoint** — rejected: unbounded permanent-prefix growth, self-accelerating toward thrashing, with nothing left to re-prioritize.
|
||||
- **Pure stubs, no state checkpoint** — rejected: presumes the model knows what it is missing; fails on unknown unknowns.
|
||||
- **LLM aging/consolidation of frozen chunks** — rejected as a routine mechanism: summary-of-summary loss and frozen-prefix churn; the code-only rollup is its surviving form, deferred.
|
||||
- **Full prefix as chunk-summarizer input** — rejected: O(N²); the state document gives the same background at O(state).
|
||||
- **One summarize call emitting all outputs** — rejected: the summarize path has no structured-output enforcement; parsing one free-text response apart is the fragile seam the fail-closed design avoids.
|
||||
- **Model-chosen chunk boundaries** — deferred: parse-and-validate cost against unproven value; chunk policy sits behind config.
|
||||
- **Model-authored pointers** — rejected: pointers must be exact; deterministic assembly is.
|
||||
- **FTS/vector index sidecar** — rejected in-session: the live log is in memory and bounded, a literal scan under budget suffices; an index earns its keep at cross-session scope.
|
||||
- **Semantic search fallback / secondary-model extraction in the recall path** — rejected: an LLM or embedding call there breaks keyless replay determinism; recall stays a pure function of the log.
|
||||
- **Raw events instead of rendered transcript** — rejected: leaks log-only vocabulary and chunk noise; the model reads what a model once saw.
|
||||
- **Doing nothing (resume/fork as recovery)** — rejected: it makes recovery a human act.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Auto-compaction over a long session yields `[stubs…][state][tail]` after every completed pass; prior stubs stay byte-identical across passes; committed stubs never fall inside a later region; the superseded state checkpoint folds without a tombstone, renders labeled, and stays reachable and searchable through the two-hop chain.
|
||||
- Every checkpoint's surface text ends with the deterministic footer; footers round-trip through replay byte-identically; the state checkpoint's provenance records its wider input range.
|
||||
- Nothing commits before all summaries exist and the guard passes on like-for-like accounting; a guard failure commits nothing and does not fail the turn; a mid-commit kill resumed at the next pre-step completes the pass with the state region committed unconditionally, merge base read from the log; a legacy head checkpoint is adopted as state-class.
|
||||
- `history_read` renders any logged checkpoint's span under budget with a working cursor; `history_search` covers every shadowed span with checkpoint-id snippets and coverage metadata, asserted in particular by finding content that exists only in a span shadowed by a superseded state checkpoint — the regression pin for trailing-slice reachability; both reject non-agent callers and never-existing ids or orphaned `compact/start` with typed errors; recalled content appears as ordinary `tool/result`s; request-reconstruction invariants pass over sessions with compaction plus recall; one keyless snapshot scenario covers compact-then-recall end to end; tool schemas and the prompt section are byte-identical across passes.
|
||||
- On the long-horizon bench suite: task success does not regress against `compact-basic` at equal budgets; a handoff-fidelity probe (restate K known decisions and constraints after a pass) scores no worse; recall usage frequency and hit usefulness are reported per run via the dsh bench report pipeline, alongside the stub-directory attention measurement and cache-hit telemetry.
|
||||
- Seam JSDoc, the compaction capability-seam RFC, `architecture.md`, and the generated tool, config, persistence, and module-graph catalogs update in the same change; all budgets live in config; new source directories hold per-file 100% coverage with HMR disposal tests.
|
||||
|
||||
## Risks
|
||||
|
||||
- **Recall is a learned behavior**: untrained models will under-use it, and the bench report exists to track the gap while training closes it. Until then the state checkpoint keeps the floor at today's summary quality.
|
||||
- **Unknown unknowns remain**: a detail absent from summaries and keywords draws no recall. Recall converts "unreachable even when suspected" into "reachable when suspected".
|
||||
- **The stub directory occupies attention**: dozens of stable index cards per request may dilute focus; the bench measurement in the acceptance criteria tracks it against `compact-basic`.
|
||||
- **Cost**: per-pass summarize input is roughly twice today's; short sessions sit near today's cost and quality, and the design pays off with session length.
|
||||
- **State drift and division-of-labor leakage** are observable through the handoff probe and stub review; their counters are specified follow-ups.
|
||||
- **Two backends** are a maintenance surface; the seam contract and the shared recall consumer bound it, and the bench comparison decides the default over time.
|
||||
@@ -4,7 +4,7 @@ Status: proposed
|
||||
|
||||
## Problem
|
||||
|
||||
Add isolated subagent providers for Claude Code and Codex. The existing [named-provider seam](../../implemented/feature/2026-06-21-subagent-capability-seam.md) and [ACP backend](../../implemented/feature/2026-06-22-acp-subagent-backend.md) establish the process-boundary shape. A harness turn should be able to delegate a self-contained task to either product and receive its final answer without exposing parent secrets or inheriting host configuration from `~/.claude` or `~/.codex`.
|
||||
The subagent seam ([the seam RFC](../../implemented/feature/2026-06-21-subagent-capability-seam.md)) hosts multiple named providers on `ctx.subagents`, and the ACP backend ([the ACP backend RFC](../../implemented/feature/2026-06-22-acp-subagent-backend.md)) proved the seam generalizes across a process boundary; its Future-providers section explicitly named the Codex app-server and the Claude Code Agent SDK as mechanically similar siblings. Those two are the engines actually worth delegating to today: a harness turn should be able to hand a self-contained task to a real Claude Code or a real Codex — a separate product with its own model, tools, and sandbox — and get back one final answer, without the parent deployment leaking its secrets into the child or the child's behavior silently depending on whatever `~/.claude` / `~/.codex` state exists on the host machine.
|
||||
|
||||
## Proposal
|
||||
|
||||
@@ -12,9 +12,9 @@ Two sibling provider packages, structural variants of the ACP backend, plus one
|
||||
|
||||
- `@deepseek-ai/dsh-subagent-claude-code` — drives a Claude Code child through `@anthropic-ai/claude-agent-sdk`'s `query()` (the SDK runs in the parent process and spawns its bundled `claude` CLI as the subprocess). Provider name `claude-code`: the child is the Claude Code *product*, not an Anthropic model adapter — "claude" stays reserved for a future `dsh-llm` adapter.
|
||||
- `@deepseek-ai/dsh-subagent-codex` — spawns `codex app-server` and drives one thread/turn over its JSON-RPC-over-stdio protocol with a hand-rolled newline-JSON client (~200–300 lines) in the package.
|
||||
- `@deepseek-ai/dsh-subagent-process` — a pure library (the `subagent-inprocess` precedent) extracting what `dsh-subagent-acp` already carries and both new backends need: the credential env scrub (`SENSITIVE_ENV_PATTERN`/`buildChildEnv`), the EOF → SIGTERM → SIGKILL dispose ladder, and new isolated-config-dir helpers (`mkdtemp` create, best-effort remove). The ACP backend migrates onto it; `bash-local`'s sibling copy is left alone to bound the change.
|
||||
- `@deepseek-ai/dsh-subagent-process` — a pure library (the `subagent-inprocess` precedent) extracting what `dsh-subagent-acp` already carries and both new backends need: the credential env scrub (`buildChildEnv`), the EOF → SIGTERM → SIGKILL dispose ladder, and new isolated-config-dir helpers (`mkdtemp` create, best-effort remove). The ACP backend migrates onto it; `bash-local`'s sibling copy is left alone to bound the change.
|
||||
|
||||
Both providers follow the ACP backend contract: a fresh child per `start`, one prompt round-trip, no inherited parent context or advertised optional capabilities, ignored `request.parent` and `request.agentOptions`, and a random branded agent id. `result` never rejects; child failures map to stop reasons while the original error reaches the logger. Each mounts `dsh-tool-subagent` under a distinct tool name. The tool result is the only new model-visible artifact, so no new session event is required; workspace mutations remain ambient side effects outside transcript replay.
|
||||
Both providers copy the ACP backend's seam posture verbatim: fresh child per `start`, exactly one prompt round-trip, capabilities all `false`, `inheritsParentContext: false`, `request.parent`/`request.agentOptions` ignored, `id = SessionId(randomUUID())`, `result` never rejects — child-level failure flattens to a stop reason and the original error goes to `ctx.logger` via an `onError` spec callback. Model exposure is zero new code: `dsh-tool-subagent` is loaded once per provider with a distinct `toolName` (`subagent_claude_code`, `subagent_codex`). No new session events are needed — the only model-visible artifact is the tool result, so reconstructability holds exactly as it did for ACP. To be explicit about the boundary: the session log reconstructs the model-visible transcript, not workspace mutation history — a child granted write access mutates files as an ambient side effect outside the log, exactly as the bash tools and the ACP backend already do; replay reproduces requests, not the disk.
|
||||
|
||||
## Verified interface facts (pinned versions)
|
||||
|
||||
@@ -31,11 +31,11 @@ Both integration surfaces were verified against pinned implementations before th
|
||||
|
||||
## Isolation and credentials
|
||||
|
||||
Authentication is API-key-only. Each run uses a fresh config directory (`CLAUDE_CONFIG_DIR` with `settingSources: []`, or `CODEX_HOME`) that is removed best-effort on dispose; config may instead select a persistent directory. The shared child-env helper forwards ordinary values such as `PATH`, `HOME`, `TMPDIR`, locale, and proxy settings, removes credential-shaped names, and overlays explicit `config.env`. Claude Code receives its API key through that overlay, while Codex receives it through `account/login/start` rather than a hand-written auth file.
|
||||
Deployments authenticate with API keys only, and the child must not see the host user's Claude Code / Codex configuration: behavior has to be a function of `cordis.yml` alone. Each run gets a fresh `mkdtemp` config dir — `CLAUDE_CONFIG_DIR` for Claude Code (paired with an explicit `settingSources: []`), `CODEX_HOME` for Codex — removed best-effort on dispose; a config field can pin a persistent dir instead. The child env reuses the ACP backend's `buildChildEnv` semantics verbatim via the extraction: the ambient env is forwarded MINUS credential-shaped vars (`/KEY|SECRET|TOKEN/i`), with `config.env` layered on top — so `PATH`, `HOME`, `TMPDIR`, locale, and proxy vars survive and the CLIs run normally, while only credential-shaped ambient vars are scrubbed (`ANTHROPIC_API_KEY` enters explicitly through `config.env` for Claude Code), and the Codex key travels via the `account/login/start` RPC into the isolated `CODEX_HOME` rather than a hand-written `auth.json`.
|
||||
|
||||
## Permission and approval policy
|
||||
|
||||
Each backend exposes its engine's native policy vocabulary. Claude Code defaults to `permissionMode: default` with `permission: reject`; Codex defaults to `sandboxMode: read-only`, `approvalPolicy: never`, and the same rejected fallback. Examples opt into `acceptEdits` or `workspace-write`. Known approval, user-input, and elicitation requests receive the configured answer; unknown methods receive method-not-found and unknown notifications are consumed. No prompt reaches a human, and no child can wait indefinitely for unavailable input.
|
||||
Instead of collapsing to ACP's single `permission: allow|reject` knob, each backend exposes its engine's native vocabulary as config, with conservative defaults: Claude Code gets `permissionMode` (default `default`) plus `permission: allow|reject` (default `reject`) as the `canUseTool` auto-answer for whatever falls through; Codex gets `sandboxMode` (default `read-only`) and `approvalPolicy` (default `never`) plus the same `permission` fallback for approval requests that still arrive. Defaults are deliberately do-no-harm (the out-of-box child cannot write files); examples demonstrate opening up (`acceptEdits` / `workspace-write`). The mechanical rule: EVERY server-initiated request is settled programmatically and promptly — the enumerated approval/user-input/elicitation requests by the configured policy, an unknown request method with a JSON-RPC method-not-found error response (never left pending), unknown notifications consumed — so no child request can wedge a turn waiting on an answer that will never come. Prompts never reach a human in this cut, matching ACP.
|
||||
|
||||
## StopReason mapping
|
||||
|
||||
@@ -45,11 +45,11 @@ Liveness posture, stated explicitly: teardown timing is config, turn duration is
|
||||
|
||||
## Testing
|
||||
|
||||
Coverage is required at each applicable tier:
|
||||
Named at every tier per the root AGENTS.md rule, and de-risked up front:
|
||||
|
||||
- **Keyless unit/integration:** drive a fake Claude CLI through the real SDK and a scripted Codex app-server through the real wire client. At per-file 100% coverage, exercise round trips, every stop mapping, both cancellation paths and pre-abort, permission policies, unknown messages, spawn failure, reload cleanup, export shape, scrubbed environments, temporary-directory removal, and Codex auth precheck failure.
|
||||
- **With-key e2e:** each real engine performs file work under `acceptEdits` or `workspace-write`; skips name the missing binary or key and assert no child process remains.
|
||||
- **Snapshot:** deferred as `TODO(claude-code-subagent-replay)` and `TODO(codex-subagent-replay)` pending the process-specific replay shape described by the [subagent replay RFC](../../implemented/testing/2026-06-22-subagent-snapshot-replay.md).
|
||||
- **Keyless unit/integration**, mirroring the ACP spec list per backend (round-trip and output accumulation, every stop mapping, both cancel paths, already-aborted, permission auto-answer under both policies, unknown-message tolerance, bad-command spawn failure, HMR provider cleanup, export shape, isolation assertions on child env and temp-dir removal; Codex adds the auth-precheck failure path). Claude Code's harness is a scripted fake `claude` executable behind `pathToClaudeCodeExecutable` driven by the REAL SDK — a spike already passed end-to-end keyless in 24ms (the fake CLI answers one `control_request/initialize` and speaks plain stream-json, ~40 lines). Codex's harness is a scripted mock app-server subprocess speaking the verified wire protocol, the `mock-acp-server.ts` shape.
|
||||
- **With-key e2e** per backend: the real engine does real file work verified on disk, under a pinned opened-up config so acceptance and the do-no-harm defaults don't collide — `permissionMode: 'acceptEdits'` for Claude Code, `sandboxMode: 'workspace-write'` + `approvalPolicy: 'never'` for Codex; self-skips report exactly what is missing (binary vs key). CI has no secrets, so these run locally per the with-key policy.
|
||||
- **Snapshot**: deferred as `TODO(claude-code-subagent-replay)` / `TODO(codex-subagent-replay)` — the same distinct replay shape the ACP backend deferred ([the per-session replay RFC](../../implemented/testing/2026-06-22-subagent-snapshot-replay.md)); the keyless suites carry deterministic coverage meanwhile.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@ Status: proposed
|
||||
|
||||
## Problem
|
||||
|
||||
The exact-read `ctx.sessionQuery` service deliberately has no derived index. Large persisted histories need full-text search without scanning every event on every query, while current live sessions need an overlay newer than the last durability checkpoint. Search also needs concrete ranking, snippets, filters, pagination, cancellation, and rebuild behavior.
|
||||
The exact-read `ctx.sessionQuery` service deliberately has no derived index. Large persisted histories need full-text search without scanning every event on every query, while current live sessions need an overlay newer than the last durability checkpoint. Search also needs concrete ranking, snippets, pagination, cancellation, and rebuild behavior.
|
||||
|
||||
Splitting those concerns across a speculative provider coordinator and a database implementation would create two coupled reconciliation state machines. The first real implementation should own the source observation, extraction, SQLite transaction, generation, and query as one lifecycle.
|
||||
|
||||
@@ -20,7 +20,7 @@ Persisted documents survive restarts. Live overrides are connection-local and sh
|
||||
|
||||
The implementation must define both cross-session and within-session scopes from executable use cases. Each searchable event is one document with session metadata, event metadata, surface classification, normalized semantic text, and a bounded plain-text snippet. Session results group by their strongest matching event; numeric backend scores remain private.
|
||||
|
||||
Filters compile to parameterized SQL before ranking. Query syntax is treated as data. Ordering includes stable tie fields. Opaque cursors bind to normalized request shape and the smallest relevant generation; unrelated session changes should not invalidate a within-session cursor. Cancellation must stop caller waiting and interrupt SQLite work where the runtime permits.
|
||||
Search returns content-bearing result records rather than metadata-only headers. Chainable filters operate on that exact result shape and are designed and implemented with the search API instead of becoming a provider-specific pre-ranking contract. Query syntax is treated as data. Ordering includes stable tie fields. Opaque cursors bind to normalized request shape and the smallest relevant generation; unrelated session changes should not invalidate a within-session cursor. Cancellation must stop caller waiting and interrupt SQLite work where the runtime permits.
|
||||
|
||||
Tokenizer choice remains an implementation experiment. FTS5 trigram supports substring recall but rejects useful terms shorter than three characters and increases index size; the proposal must benchmark that tradeoff against the default Unicode tokenizer before making it contract.
|
||||
|
||||
@@ -41,7 +41,7 @@ Reconciliation may use stable fingerprints to avoid rewriting unchanged persiste
|
||||
|
||||
- Restart tests cover unchanged, new, changed, and deleted persisted sessions without rebuilding the whole index.
|
||||
- Reopening preserves persisted rows and removes live rows; live rows shadow and then reveal their persisted base.
|
||||
- Tests cover both search scopes, metadata filters, surface defaults, snippets, escaping, deterministic ties, pagination, scoped stale cursors, cancellation, dynamic persistence mount/unmount, and recovery after a failed transaction.
|
||||
- Tests cover both search scopes, content-bearing results, chainable result filters, surface defaults, snippets, escaping, deterministic ties, pagination, scoped stale cursors, cancellation, dynamic persistence mount/unmount, and recovery after a failed transaction.
|
||||
- A schema mismatch resets only the derived database.
|
||||
- A keyless end-to-end test combines a real persistence backend with the real SQLite search package.
|
||||
- The RFC is amended to the measured tokenizer and public API actually implemented before moving to `implemented/`.
|
||||
|
||||
Reference in New Issue
Block a user