feat(session-query): checkpoint build round 1

This commit is contained in:
Hypatia May
2026-07-10 16:51:19 +08:00
parent 42ebbfdf8f
commit aa1dc0e2c7
40 changed files with 3174 additions and 102 deletions

View File

@@ -33,6 +33,7 @@ Composition is preferred over inheritance. `packages/core/` is a repository grou
| `ctx.subagents` | [`subagent/`](../packages/subagent/README.md) | named delegation providers |
| `ctx.workflows` | [`workflow/`](../packages/workflow/README.md) | script-driven multi-agent orchestration |
| `ctx.sessionPersistence` | [`session-persistence/`](../packages/session-persistence/README.md) | durable storage for session logs |
| `ctx.sessionQuery` | [`session-query/`](../packages/session-query/README.md) | live/persisted session retrieval and search-provider coordination |
## Event
@@ -156,4 +157,4 @@ The [extension cookbook](cookbook/extension-cookbook.md) carries plugin skeleton
- Exact event and service signatures in [events](cordis-catalog/events.md)
- [services](cordis-catalog/services.md) catalogs
- package contracts in the [package map](../packages/README.md)
- [RFCs](rfc/README.md)
- [RFCs](rfc/README.md)

View File

@@ -18,12 +18,14 @@ flowchart LR
svc_sessions["ctx.sessions<br/>In-memory session store"]
pkg_agent["agent"]
pkg_session_persistence["session-persistence"]
pkg_session_query["session-query"]
pkg_subagent_inprocess["subagent-inprocess"]
pkg_invariants["invariants"]
svc_sessionPersistence["ctx.sessionPersistence<br/>Durable session persistence seam"]
pkg_session_persistence_jsonl["session-persistence-jsonl"]
pkg_session_persistence_sqlite["session-persistence-sqlite"]
pkg_acp["acp"]
svc_sessionQuery["ctx.sessionQuery<br/>Session retrieval read model"]
pkg_system_prompt["system-prompt"]
svc_systemPrompt["ctx.systemPrompt<br/>System prompt assembly registry"]
pkg_tools["tools"]
@@ -90,6 +92,7 @@ flowchart LR
pkg_session_persistence --> svc_sessionPersistence
pkg_session_persistence_jsonl --> svc_sessionPersistence
pkg_session_persistence_sqlite --> svc_sessionPersistence
pkg_session_query --> svc_sessionQuery
pkg_stdio_agent --> svc_userInteraction
pkg_subagent --> svc_subagents
pkg_subagent_acp --> svc_subagents
@@ -122,10 +125,12 @@ flowchart LR
svc_llm --> pkg_compact_basic
svc_sessionPersistence --> pkg_acp
svc_sessionPersistence --> pkg_agent_loop
svc_sessionPersistence --> pkg_session_query
svc_sessions --> pkg_agent
svc_sessions --> pkg_agent_loop
svc_sessions --> pkg_invariants
svc_sessions --> pkg_session_persistence
svc_sessions --> pkg_session_query
svc_sessions --> pkg_subagent_inprocess
svc_subagents --> pkg_tool_subagent
svc_systemPrompt --> pkg_agent_loop
@@ -152,8 +157,9 @@ flowchart LR
| ctx key | Role | Owner | Implementations | Direct consumers | Companion plugins | Note |
| --- | --- | --- | --- | --- | --- | --- |
| `ctx.llm` | `seam` | [`llm`](../packages/llm/llm) | [`llm-deepseek`](../packages/llm/llm-deepseek), [`llm-pi-ai`](../packages/llm/llm-pi-ai), [`llm-replay`](../packages/support/llm-replay) | [`agent-loop`](../packages/core/agent-loop), [`compact-basic`](../packages/compact/compact-basic) | - | Adapters register provider implementations; the loop and compaction call the provider-neutral stream service. |
| `ctx.sessions` | `core` | [`session`](../packages/core/session) | - | [`agent-loop`](../packages/core/agent-loop), [`agent`](../packages/core/agent), [`session-persistence`](../packages/session-persistence/session-persistence), [`subagent-inprocess`](../packages/subagent/subagent-inprocess), [`invariants`](../packages/support/invariants) | - | Owns append-only Session instances and emits the durable session event feed. |
| `ctx.sessionPersistence` | `seam` | [`session-persistence`](../packages/session-persistence/session-persistence) | [`session-persistence-jsonl`](../packages/session-persistence/session-persistence-jsonl), [`session-persistence-sqlite`](../packages/session-persistence/session-persistence-sqlite) | [`agent-loop`](../packages/core/agent-loop), [`acp`](../packages/ui/acp) | - | Backends persist the same SessionEvent vocabulary; apps choose a backend at composition time. |
| `ctx.sessions` | `core` | [`session`](../packages/core/session) | - | [`agent-loop`](../packages/core/agent-loop), [`agent`](../packages/core/agent), [`session-persistence`](../packages/session-persistence/session-persistence), [`session-query`](../packages/session-query/session-query), [`subagent-inprocess`](../packages/subagent/subagent-inprocess), [`invariants`](../packages/support/invariants) | - | Owns append-only Session instances and emits the durable session event feed. |
| `ctx.sessionPersistence` | `seam` | [`session-persistence`](../packages/session-persistence/session-persistence) | [`session-persistence-jsonl`](../packages/session-persistence/session-persistence-jsonl), [`session-persistence-sqlite`](../packages/session-persistence/session-persistence-sqlite) | [`agent-loop`](../packages/core/agent-loop), [`acp`](../packages/ui/acp), [`session-query`](../packages/session-query/session-query) | - | Backends persist the same SessionEvent vocabulary; apps choose a backend at composition time. |
| `ctx.sessionQuery` | `seam` | [`session-query`](../packages/session-query/session-query) | - | - | - | Resolves live and optional persisted logs into one corpus and coordinates registered full-text providers. |
| `ctx.systemPrompt` | `core` | [`system-prompt`](../packages/core/system-prompt) | - | [`agent-loop`](../packages/core/agent-loop), [`tools`](../packages/core/tools), [`tool-fs`](../packages/fs/tool-fs), [`tool-web`](../packages/web/tool-web) | - | Collects prompt sections and model-facing tool schemas for each step. |
| `ctx.tools` | `core` | [`tools`](../packages/core/tools) | - | [`agent-loop`](../packages/core/agent-loop), [`tool-ask-user`](../packages/ui/tool-ask-user), [`tool-bash`](../packages/bash/tool-bash), [`tool-cordis`](../packages/cordis/tool-cordis), [`tool-fs`](../packages/fs/tool-fs), [`tool-subagent`](../packages/subagent/tool-subagent), [`tool-todo`](../packages/todo/tool-todo), [`tool-web`](../packages/web/tool-web), [`acp`](../packages/ui/acp) | - | Registers tool definitions, exposes schemas to the prompt, and routes calls through tools/pre-execute and tools/post-execute. |
| `ctx.userInteraction` | `seam` | [`user-interaction`](../packages/ui/user-interaction) | [`stdio-agent`](../packages/ui/stdio-agent), [`acp`](../packages/ui/acp) | [`tool-ask-user`](../packages/ui/tool-ask-user), [`stdio-agent`](../packages/ui/stdio-agent), [`acp`](../packages/ui/acp) | - | UI front doors provide the active human-answer provider; tool-ask-user pauses a tool call on the provider-neutral ask() promise. |

View File

@@ -483,6 +483,26 @@ export type JournalMode = 'wal' | 'delete' | 'truncate' | 'persist'
Source: [`packages/session-persistence/session-persistence-sqlite/src/index.ts:50`](../packages/session-persistence/session-persistence-sqlite/src/index.ts)
## `@deepseek-ai/dsh-session-query`
Requires: `sessions`
```ts config-catalog
/** Configuration for the provider-neutral session-query service. */
export interface Config {
/** Explicit provider id; omitted auto-selects exactly one usable provider. */
searchProvider?: string
/** Default search result page size. Defaults to 20. */
defaultLimit?: number
/** Maximum accepted search page size. Defaults to 100. */
maxLimit?: number
/** Maximum accepted raw read context on either side. Defaults to 50. */
readWindowMax?: number
}
```
Source: [`packages/session-query/session-query/src/config.ts:17`](../packages/session-query/session-query/src/config.ts)
## `@deepseek-ai/dsh-stdio-agent`
```ts config-catalog

View File

@@ -237,7 +237,7 @@ An event was appended to a session log (sync, fire-and-forget). This is the per-
Types: [SessionEvent](../core-data-structures/core.md)
Source: [`packages/core/session/src/index.ts:47`](../../packages/core/session/src/index.ts)
Source: [`packages/core/session/src/index.ts:55`](../../packages/core/session/src/index.ts)
### `session/flush` — parallel
@@ -247,7 +247,27 @@ Awaited durability checkpoint. The agent loop awaits `ctx.parallel('session/flus
'session/flush'(session: Session): Promise<void> | void
```
Source: [`packages/core/session/src/index.ts:57`](../../packages/core/session/src/index.ts)
Source: [`packages/core/session/src/index.ts:65`](../../packages/core/session/src/index.ts)
### `session/persisted` — parallel
A persistence backend committed a canonical session-log change. This is an observe-only notification for derived read models: the durable write has already succeeded, and listener failures are contained rather than propagated into append, load, flush, or teardown.
```ts cordis-catalog
'session/persisted'(header: SessionHeader, change: SessionPersistedChange): Promise<void> | void
```
Source: [`packages/session-persistence/session-persistence/src/index.ts:50`](../../packages/session-persistence/session-persistence/src/index.ts)
### `session/removed` — parallel
A session left the live store. The header is snapshotted after the store entry is removed; listener failures are contained and cannot break the owning fiber's teardown.
```ts cordis-catalog
'session/removed'(header: SessionHeader): Promise<void> | void
```
Source: [`packages/core/session/src/index.ts:47`](../../packages/core/session/src/index.ts)
## `subagent/*`

View File

@@ -167,7 +167,26 @@ abstract list(): Promise<SessionHeader[]>
Types: [SessionEvent](../core-data-structures/core.md)
Source: [`packages/session-persistence/session-persistence/src/index.ts:102`](../../packages/session-persistence/session-persistence/src/index.ts)
Source: [`packages/session-persistence/session-persistence/src/index.ts:125`](../../packages/session-persistence/session-persistence/src/index.ts)
## `ctx.sessionQuery` — `SessionQueryService`
Session-history retrieval and provider coordination service.
```ts cordis-catalog
listSessions(): Promise<SessionRecord[]>
async listEvents(sessionId: SessionId): Promise<SessionEventRecord[]>
async readEvent(request: SessionEventReadRequest): Promise<SessionEventWindow>
async traceSession(sessionId: SessionId): Promise<SessionLineageTrace>
async traceEvent(sessionId: SessionId, seq: number): Promise<SessionEventTrace>
registerSearchProvider(provider: SessionSearchProvider): () => void
registerEventTextExtractor<K extends SessionEventType>( type: K, extractor: SessionEventTextExtractor<K>, ): () => void
registerContentTextExtractor<K extends ContentBlockType>( type: K, extractor: SessionContentTextExtractor<K>, ): () => void
searchSessions( request: SessionSearchRequest, exec?: SessionQueryExecContext, ): Promise<SessionSearchPage<SessionSearchHit>>
searchEvents( request: SessionEventSearchRequest, exec?: SessionQueryExecContext, ): Promise<SessionSearchPage<SessionEventSearchHit>>
```
Source: [`packages/session-query/session-query/src/index.ts:59`](../../packages/session-query/session-query/src/index.ts)
## `ctx.sessions` — `SessionStore`
@@ -185,7 +204,7 @@ list(): Session[]
fork(source: SessionForkSource, boundary?: number, childSessionId?: SessionId): Session
```
Source: [`packages/core/session/src/index.ts:405`](../../packages/core/session/src/index.ts)
Source: [`packages/core/session/src/index.ts:413`](../../packages/core/session/src/index.ts)
## `ctx.subagents` — `SubagentService`

View File

@@ -24,8 +24,10 @@ This matrix shows which packages dispatch each harness-owned event and which pac
| `fs/write-intent` | `waterfall` | [`packages/fs/fs/src/index.ts:109`](../packages/fs/fs/src/index.ts) | [`tool-fs`](../packages/fs/tool-fs) (`waterfall`) | [`fs-policy`](../packages/fs/fs-policy) |
| `llm/stream` | `waterfall` | [`packages/llm/llm/src/index.ts:39`](../packages/llm/llm/src/index.ts) | [`llm`](../packages/llm/llm) (`waterfall`) | [`invariants`](../packages/support/invariants), [`llm-replay`](../packages/support/llm-replay) |
| `session/created` | `emit` | [`packages/core/session/src/index.ts:39`](../packages/core/session/src/index.ts) | [`session`](../packages/core/session) (`emit`) | [`invariants`](../packages/support/invariants), [`session-persistence`](../packages/session-persistence/session-persistence) |
| `session/event` | `emit` | [`packages/core/session/src/index.ts:47`](../packages/core/session/src/index.ts) | [`session`](../packages/core/session) (`emit`) | [`acp`](../packages/ui/acp), [`invariants`](../packages/support/invariants), [`session-persistence`](../packages/session-persistence/session-persistence), [`stdio-agent`](../packages/ui/stdio-agent) |
| `session/flush` | `parallel` | [`packages/core/session/src/index.ts:57`](../packages/core/session/src/index.ts) | [`agent-loop`](../packages/core/agent-loop) (`parallel`) | [`session-persistence`](../packages/session-persistence/session-persistence) |
| `session/event` | `emit` | [`packages/core/session/src/index.ts:55`](../packages/core/session/src/index.ts) | [`session`](../packages/core/session) (`emit`) | [`acp`](../packages/ui/acp), [`invariants`](../packages/support/invariants), [`session-persistence`](../packages/session-persistence/session-persistence), [`stdio-agent`](../packages/ui/stdio-agent) |
| `session/flush` | `parallel` | [`packages/core/session/src/index.ts:65`](../packages/core/session/src/index.ts) | [`agent-loop`](../packages/core/agent-loop) (`parallel`) | [`session-persistence`](../packages/session-persistence/session-persistence) |
| `session/persisted` | `parallel` | [`packages/session-persistence/session-persistence/src/index.ts:50`](../packages/session-persistence/session-persistence/src/index.ts) | [`session-persistence`](../packages/session-persistence/session-persistence) (`parallel`) | [`session-query`](../packages/session-query/session-query) |
| `session/removed` | `parallel` | [`packages/core/session/src/index.ts:47`](../packages/core/session/src/index.ts) | [`session`](../packages/core/session) (`parallel`) | - |
| `subagent/end` | `emit` | [`packages/subagent/subagent/src/index.ts:98`](../packages/subagent/subagent/src/index.ts) | [`subagent`](../packages/subagent/subagent) (`events.dispatch`) | [`hooks-claude`](../packages/hooks/hooks-claude) |
| `subagent/provider-added` | `emit` | [`packages/subagent/subagent/src/index.ts:72`](../packages/subagent/subagent/src/index.ts) | [`subagent`](../packages/subagent/subagent) (`emit`) | [`tool-subagent`](../packages/subagent/tool-subagent) |
| `subagent/provider-removed` | `emit` | [`packages/subagent/subagent/src/index.ts:83`](../packages/subagent/subagent/src/index.ts) | - | [`tool-subagent`](../packages/subagent/tool-subagent) |

View File

@@ -75,6 +75,9 @@ flowchart TD
pkg_session_persistence_jsonl["session-persistence-jsonl"]
pkg_session_persistence_sqlite["session-persistence-sqlite"]
end
subgraph group_session_query["packages/session-query"]
pkg_session_query["session-query"]
end
subgraph group_support["packages/support"]
pkg_acp_snapshot["acp-snapshot"]
pkg_invariants["invariants"]
@@ -145,6 +148,9 @@ flowchart TD
pkg_session_persistence_jsonl --> pkg_session_persistence
pkg_session_persistence_sqlite --> pkg_session
pkg_session_persistence_sqlite --> pkg_session_persistence
pkg_session_query --> pkg_llm
pkg_session_query --> pkg_session
pkg_session_query --> pkg_session_persistence
pkg_invariants --> pkg_agent
pkg_invariants --> pkg_llm
pkg_invariants --> pkg_session
@@ -297,6 +303,7 @@ flowchart TD
| [`compact-basic`](../packages/compact/compact-basic) | `compact` | [`agent`](../packages/core/agent), [`compact`](../packages/compact/compact), [`llm`](../packages/llm/llm), [`session`](../packages/core/session) |
| [`session-persistence-jsonl`](../packages/session-persistence/session-persistence-jsonl) | `session-persistence` | [`session`](../packages/core/session), [`session-persistence`](../packages/session-persistence/session-persistence) |
| [`session-persistence-sqlite`](../packages/session-persistence/session-persistence-sqlite) | `session-persistence` | [`session`](../packages/core/session), [`session-persistence`](../packages/session-persistence/session-persistence) |
| [`session-query`](../packages/session-query/session-query) | `session-query` | [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`session-persistence`](../packages/session-persistence/session-persistence) |
| [`invariants`](../packages/support/invariants) | `support` | [`agent`](../packages/core/agent), [`llm`](../packages/llm/llm), [`session`](../packages/core/session) |
| [`user-interaction`](../packages/ui/user-interaction) | `ui` | [`agent`](../packages/core/agent), [`llm`](../packages/llm/llm) |
| [`workflow`](../packages/workflow/workflow) | `workflow` | [`agent`](../packages/core/agent), [`brand`](../packages/util/brand), [`llm`](../packages/llm/llm) |

View File

@@ -12,6 +12,7 @@ Generated by `pnpm run gen-rfc-index` from the RFC tree — never edit by hand;
| [Multiplex concurrent ACP sessions over one connection](proposed/feature/2026-06-14-acp-multi-session.md) | 2026-06-14 |
| [Pre-tool input rewrite — a consistent design](proposed/feature/2026-06-30-pre-tool-input-rewrite.md) | 2026-06-30 |
| [Claude Code and Codex subagent backends (out-of-process delegation to external coding agents)](proposed/feature/2026-07-07-claude-code-and-codex-subagent-backends.md) | 2026-07-07 |
| [SQLite FTS5 session-query provider](proposed/feature/2026-07-10-sqlite-session-query-provider.md) | 2026-07-10 |
### Simplification
@@ -67,6 +68,7 @@ Generated by `pnpm run gen-rfc-index` from the RFC tree — never edit by hand;
| [The session prefix — request-only messages in front of the derived history](implemented/feature/2026-07-07-session-prefix.md) | 2026-07-07 |
| [Repeat-tool-call guard plugin](implemented/feature/2026-07-08-repeat-tool-guard.md) | 2026-07-08 |
| [The self-referential cordis toolset](implemented/feature/2026-07-08-self-referential-cordis-toolset.md) | 2026-07-08 |
| [Provider-neutral session query service](implemented/feature/2026-07-10-session-query-service.md) | 2026-07-10 |
### Simplification

View File

@@ -0,0 +1,61 @@
# RFC: Provider-neutral session query service
Status: implemented
## Problem
Session logs contain the harness's durable working memory, but the existing services expose them only as live objects or backend-specific persisted records. Consumers that want history search, compacted-event recall, lineage inspection, or another agent's status otherwise have to choose a storage backend, duplicate live-versus-persisted precedence, and reconstruct surface provenance independently. Live state also advances between persistence checkpoints, so treating durable storage as the only query source makes current-turn reads stale.
Search is only one operation in that read model. Metadata filtering must compose without another database round trip, event and session lineage need deterministic graph semantics, and an event read must return exact canonical content rather than a search snippet. Folding all of those responsibilities into one SQLite package would make storage technology the public API and would prevent live-only deployments from using the non-search capabilities.
## Decision
`@deepseek-ai/dsh-session-query` owns `ctx.sessionQuery`, a trusted provider-neutral read model over one logical corpus: live `SessionStore` entries plus an optional, dynamically mounted `SessionPersistence` service. Matching ids resolve to one record. Live events take precedence because they include appends after the latest checkpoint; the record still exposes independent `live` and `persisted` flags. The service compares immutable headers and fails with a typed source-conflict error when the two sources cannot represent the same session.
The service owns source observation, reconciliation, precedence, cloning, filters, tracing, extraction, and provider selection. It exposes lightweight session and event records, bounded exact-event reads, complete known session lineage, event surface/provenance traces, and two full-text scopes. A search backend owns only indexing, ranking, snippets, cursors, and backend-specific query validation.
Persistence is optional. Live-only reads and provider synchronization work without it. Unmounting persistence hides the provider's durable base rather than deleting derived cache rows, so remounting can reuse fingerprints. An installed but unreadable backend fails cross-session operations; a read of a known live session remains independent of that failure.
## Surface and lineage semantics
`dsh-session` exports `foldSurface(events)`, and `SurfaceManager` uses the same transition functions for its incremental cache. The fold returns detached current nodes and each replacement's actual removed seq range. Session-query derives `current`, `shadowed`, and `log-only` classifications and replacement chains from that result, so query and model-history derivation cannot disagree about positional replacement semantics.
Event traces accept any raw event. They return direct `sourceEventSeqs` references, reverse references, nodes directly shadowed by a replacement, its immediate replacer, and the transitive replacement chain toward the current surface. Related content is deliberately not embedded; exact content remains the job of the bounded event read.
Session traces walk parents nearest-first. A complete chain reports its root; a partial corpus reports the first unresolved parent id. Descendants form a complete known tree ordered by creation time and id. A cycle connected to the target is an invalid lineage error rather than a truncated result.
## Filters and public records
Serializable discriminated filter specs cover session identity, cwd, creation time, parent/root, availability, event seq/time/type, and surface status. Alternatives within one spec are OR; specs in a supplied array are AND. The exported generic transforms are pure, preserve order and item identity, and work on base records or richer hits. Search requests accept the same specs before ranking. Applying a transform to one materialized page never triggers a refill.
Public records are intentionally small. `SessionRecord` carries a cloned header and source flags. `SessionEventRecord` carries session id, seq, type, time, and surface status. Search adds a plain snippet to event hits and exactly one best event to session hits; numeric provider scores remain private. Search pages default to 20 and reject limits above 100. Exact event reads default to no neighbors and cap each side with the configurable `readWindowMax`, default 50.
## Lifecycle notifications
Two observe-only Cordis notifications keep derived read models current without joining the write transaction. `session/removed` fires after a live entry leaves `SessionStore`. `session/persisted` fires only after an ordinary append or load-time repair commits and carries the affected seq range. Both snapshot their payloads and contain synchronous dispatch errors and rejected listeners, so observers cannot fail session teardown or durability.
A persistence load preserves an existing live owner in coordinator state. HMR adoption of a torn durable prefix truncates only the uncommitted fragment while the live session remains authoritative; it does not publish a repair notification or synthesize an interrupted turn mid-turn. A later real append produces the ordinary committed notification.
## Provider and extractor contracts
A selected search provider receives separate persisted-base and live-override operations. Persisted reconciliation begins inactive, compares the provider inventory with SHA-256 fingerprints over canonicalized header/events and relevant extractor versions, replaces only changed sessions, removes proven-stale rows, and then activates the base. Live snapshots always replace the matching override; removal reveals an active persisted base. Search waits for relevant queued reconciliation, with corpus scope for session search and target scope for a live event search. A failed update stays retryable and fails affected searches with a typed derived-index error without affecting canonical writes. Caller cancellation stops waiting and reaches provider query work through `AbortSignal`.
Core extractors cover semantic messages, reasoning, tools, todos, blocked prompts, context and steering, and error/status detail. Chunks, request headers, and structural events add no document. Declaration-merged event and content-block owners can install one effect-scoped extractor per type with a stable version; unknown types stay non-searchable.
## Security boundary
The service is context-wide trusted infrastructure, not an authorization layer. A model-facing history tool or human UI applies explicit caller/session scope before invoking cross-session operations. This decision exposes no unscoped model tool and changes no transcript or snapshot surface.
## Alternatives considered
- **Put all query behavior in a SQLite implementation** — rejected because filters, exact reads, source precedence, lineage, and surface provenance are storage-independent, and live-only deployments still need them. It would also let backend details become the public service contract.
- **Query only persisted sessions** — rejected because persistence checkpoints occur at turn boundaries; a current live session would be stale precisely when an agent inspects its latest work.
- **Mirror every live append into persistence before querying** — rejected because query observation must not add durability latency or change the turn checkpoint contract. The live override is an ephemeral derived layer.
- **Express every chained filter as SQL** — rejected because post-filters operate over already materialized pages and must preserve item identity and caller-chosen composition. Serializable pure transforms also remain usable without a search provider.
- **Make session-query part of the compaction capability** — rejected because retrieval reads all session structure and has consumers beyond recall; compaction is one producer of replacement provenance, not the owner of the read model.
## Consequences
Consumers gain one coherent API for current and durable history, deterministic traces, and backend-neutral search. Derived index failures and optional persistence are isolated from canonical session writes, and unchanged persisted sessions can reuse provider rows across restarts.
The service carries non-trivial reconciliation state and performs canonical log loads to validate fingerprints. Cross-session search intentionally waits for whole-corpus synchronization, and live precedence means providers must implement a two-layer model. Authorization remains the responsibility of future consumers. Full-text search is unavailable until an implementation package registers a provider; that implementation is intentionally outside this decision's package.

View File

@@ -0,0 +1,50 @@
# RFC: SQLite FTS5 session-query provider
Status: proposed
## Problem
The provider-neutral session-query service defines full-text scopes and synchronization but deliberately ships no index. A first backend must search semantic event documents across large persisted histories without rebuilding unchanged sessions at every process start, while keeping unflushed live overrides current and disposable. It also needs deterministic ranking and pagination semantics strong enough for model tools and UI clients to continue a result set safely.
Using the canonical session-persistence database directly would couple two failure domains and schemas: query rows are derived and rebuildable, while session logs are authoritative. A query schema reset, corrupt index, or experimental tokenizer must never endanger durable conversation history.
## Proposal
Add an `@deepseek-ai/dsh-session-query-sqlite` implementation in a separate phase-two pull request after the provider-neutral phase is complete. It will register one `SessionSearchProvider` on `ctx.sessionQuery` and own a separate derived SQLite database. Persisted event documents survive provider restarts; live overrides remain connection-local and disappear when the provider closes.
The provider will use SQLite FTS5 with the trigram tokenizer. A query splits on whitespace and requires every term. Terms shorter than three characters fail with a typed provider error rather than silently changing matching semantics. Each searchable event is one document, including current, shadowed, and log-only states by default. Event search ranks documents within one session; session search groups by session and ranks it by exactly one strongest matching event. Ties are deterministic, public hits contain plain-text snippets, and numeric FTS scores remain internal.
## Storage and reconciliation
The database path, journal mode, page/result limits, and snippet length are validated configuration. Durable tables store provider schema version, persisted-session fingerprints, lightweight session metadata, event metadata, text, and the FTS virtual table. A provider-schema mismatch is the exceptional full reset; ordinary startup calls `persistedInventory()` and lets the service replace only new or changed sessions and remove canonical deletions.
The live layer uses temporary or connection-local tables with the same searchable shape. A live snapshot shadows every persisted document for that session. Removing the override reveals the active persisted base. `setPersistedActive(false)` excludes durable rows from results without deleting their fingerprint cache. Reopening the database proves that persisted rows remain and live rows do not.
## Query and cursor semantics
Search request filters compile to parameterized metadata predicates before FTS ranking. Query terms are escaped as data, never interpolated into FTS syntax. Snippets are plain text with bounded length and no provider-specific markup contract.
Opaque cursors bind to the normalized request shape and a generation. Session-search cursors bind to the global logical-corpus generation. Event-search cursors bind only to the target session generation. A relevant change makes the cursor stale and produces a typed error; unrelated session changes do not invalidate an inner-session cursor. Stable tie fields are encoded after rank so resumed pages neither duplicate nor skip hits.
Provider update operations are transactional. An index write failure leaves the prior committed generation queryable only after the owning service has successfully retried the dirty update; affected searches fail rather than returning a knowingly stale page. Abort signals interrupt waits and SQLite query work where the runtime permits.
## Alternatives considered
- **Use the session-persistence SQLite database and add FTS tables there** — rejected because derived-index schema churn, resets, and corruption recovery must not share the authoritative log's transaction or failure boundary.
- **Persist live overrides immediately** — rejected because live events are not canonical until the existing persistence checkpoint commits. Ephemeral overlay rows preserve read-your-writes without inventing a second durability path.
- **Use the default FTS5 unicode tokenizer** — rejected for the first backend because substring-oriented history recall is a core use case. Trigram search gives predictable mid-token matching at the accepted cost of rejecting sub-three-character terms.
- **Return raw BM25 scores** — rejected because scores are provider-specific and unstable across corpus changes. Ranking is observable; numeric scale is not part of the service API.
- **Keep cursors valid across index changes** — rejected because rank and grouping can move after a relevant write, making continued pages duplicate or omit hits.
## Acceptance criteria
- Restart tests prove an unchanged persisted fingerprint performs no FTS replacement, while new, changed, and deleted sessions reconcile correctly.
- Reopening proves persisted rows survive, live rows disappear, removing a live override reveals its persisted base, and the provider works with no persistence service.
- Tests cover both search scopes, all metadata filters, surface defaults, snippets, AND-term escaping, short-term rejection, deterministic ties, pagination, request-bound cursors, scoped stale generations, cancellation, and recovery after a failed index update.
- A provider-schema mismatch resets only the derived database. Normal source changes never trigger a full reset.
- A keyless end-to-end restart test combines a real persistence backend with the real SQLite query provider.
- The implementation, package wiring, and tests land only in the separate phase-two pull request; phase one contains this proposal but no SQLite query code.
## Risks
Trigram indexes use more space than word-token indexes, and loading canonical logs to recompute fingerprints still has startup I/O cost even when FTS replacement is skipped. FTS5 ranking and snippet behavior can differ across SQLite runtime versions, so deterministic tie fields and provider-owned snippet tests must pin only the contract the package controls. A global generation makes cross-session cursors conservative: any corpus change invalidates them. The separate derived database adds configuration and lifecycle work, but it preserves the authoritative store's safety boundary.