Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(session-query): add SQLite full-text search

Browse Source
This commit is contained in:
Hypatia May
2026-07-15 10:51:38 +08:00
parent e9f0c37745
commit ecf90ff382
38 changed files with 3181 additions and 120 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/architecture.md
View File

@@ -35,6 +35,7 @@ A harness is one [Cordis](cordis-primer.md) context. Packages contribute service
| `ctx.workflows` | [`workflow/`](../packages/workflow/README.md) | script-driven multi-agent orchestration |
| `ctx.sessionPersistence` | [`session-persistence/`](../packages/session-persistence/README.md) | durable storage for session logs |
| `ctx.sessionQuery` | [`session-query/`](../packages/session-query/README.md) | live-preferred logical-corpus and exact-event reads |
| `ctx.sessionSearch` | [`session-query/`](../packages/session-query/README.md) | SQLite full-text search |
## Event

13
docs/capability-seams.md
View File

@@ -19,6 +19,7 @@ flowchart LR
pkg_agent["agent"]
pkg_session_persistence["session-persistence"]
pkg_session_query["session-query"]
pkg_session_query_sqlite["session-query-sqlite"]
pkg_subagent_inprocess["subagent-inprocess"]
pkg_invariants["invariants"]
svc_sessionPersistence["ctx.sessionPersistence<br/>Durable session persistence seam"]
@@ -26,6 +27,7 @@ flowchart LR
pkg_session_persistence_sqlite["session-persistence-sqlite"]
pkg_acp["acp"]
svc_sessionQuery["ctx.sessionQuery<br/>Exact session-history reads"]
svc_sessionSearch["ctx.sessionSearch<br/>Full-text session search"]
pkg_system_prompt["system-prompt"]
svc_systemPrompt["ctx.systemPrompt<br/>System prompt assembly registry"]
pkg_tools["tools"]
@@ -108,6 +110,8 @@ flowchart LR
pkg_session_persistence_jsonl --> svc_sessionPersistence
pkg_session_persistence_sqlite --> svc_sessionPersistence
pkg_session_query --> svc_sessionQuery
pkg_session_query --> svc_sessionSearch
pkg_session_query_sqlite --> svc_sessionSearch
pkg_skill --> svc_skills
pkg_skill_local --> svc_skills
pkg_stdio_agent --> svc_userInteraction
@@ -146,11 +150,13 @@ flowchart LR
svc_sessionPersistence --> pkg_acp
svc_sessionPersistence --> pkg_agent_loop
svc_sessionPersistence --> pkg_session_query
svc_sessionPersistence --> pkg_session_query_sqlite
svc_sessions --> pkg_agent
svc_sessions --> pkg_agent_loop
svc_sessions --> pkg_invariants
svc_sessions --> pkg_session_persistence
svc_sessions --> pkg_session_query
svc_sessions --> pkg_session_query_sqlite
svc_sessions --> pkg_subagent_inprocess
svc_skills --> pkg_tool_skill
svc_subagents --> pkg_tool_subagent
@@ -179,9 +185,10 @@ flowchart LR
| ctx key | Role | Owner | Implementations | Direct consumers | Companion plugins | Note |
| --- | --- | --- | --- | --- | --- | --- |
| `ctx.llm` | `seam` | [`llm`](../packages/llm/llm) | [`llm-deepseek`](../packages/llm/llm-deepseek), [`llm-pi-ai`](../packages/llm/llm-pi-ai), [`llm-replay`](../packages/support/llm-replay) | [`agent-loop`](../packages/core/agent-loop), [`compact-basic`](../packages/compact/compact-basic) | - | Adapters register provider implementations; the loop and compaction call the provider-neutral stream service. |
| `ctx.sessions` | `core` | [`session`](../packages/core/session) | - | [`agent-loop`](../packages/core/agent-loop), [`agent`](../packages/core/agent), [`session-persistence`](../packages/session-persistence/session-persistence), [`session-query`](../packages/session-query/session-query), [`subagent-inprocess`](../packages/subagent/subagent-inprocess), [`invariants`](../packages/support/invariants) | - | Owns append-only Session instances and emits the durable session event feed. |
| `ctx.sessionPersistence` | `seam` | [`session-persistence`](../packages/session-persistence/session-persistence) | [`session-persistence-jsonl`](../packages/session-persistence/session-persistence-jsonl), [`session-persistence-sqlite`](../packages/session-persistence/session-persistence-sqlite) | [`agent-loop`](../packages/core/agent-loop), [`acp`](../packages/ui/acp), [`session-query`](../packages/session-query/session-query) | - | Backends persist the same SessionEvent vocabulary; apps choose a backend at composition time. |
| `ctx.sessionQuery` | `seam` | [`session-query`](../packages/session-query/session-query) | - | - | - | Resolves live and optional persisted logs into one logical corpus for exact reads. |
| `ctx.sessions` | `core` | [`session`](../packages/core/session) | - | [`agent-loop`](../packages/core/agent-loop), [`agent`](../packages/core/agent), [`session-persistence`](../packages/session-persistence/session-persistence), [`session-query`](../packages/session-query/session-query), [`session-query-sqlite`](../packages/session-query/session-query-sqlite), [`subagent-inprocess`](../packages/subagent/subagent-inprocess), [`invariants`](../packages/support/invariants) | - | Owns append-only Session instances and emits the durable session event feed. |
| `ctx.sessionPersistence` | `seam` | [`session-persistence`](../packages/session-persistence/session-persistence) | [`session-persistence-jsonl`](../packages/session-persistence/session-persistence-jsonl), [`session-persistence-sqlite`](../packages/session-persistence/session-persistence-sqlite) | [`agent-loop`](../packages/core/agent-loop), [`acp`](../packages/ui/acp), [`session-query`](../packages/session-query/session-query), [`session-query-sqlite`](../packages/session-query/session-query-sqlite) | - | Backends persist the same SessionEvent vocabulary; apps choose a backend at composition time. |
| `ctx.sessionQuery` | `seam` | [`session-query`](../packages/session-query/session-query) | - | - | - | Resolves live and optional persisted logs into one logical corpus for exact reads and semantic scans. |
| `ctx.sessionSearch` | `seam` | [`session-query`](../packages/session-query/session-query) | [`session-query-sqlite`](../packages/session-query/session-query-sqlite) | - | - | The concrete backend owns source reconciliation, ranking, snippets, and cursor generations as one lifecycle. |
| `ctx.systemPrompt` | `core` | [`system-prompt`](../packages/core/system-prompt) | - | [`agent-loop`](../packages/core/agent-loop), [`tools`](../packages/core/tools), [`tool-fs`](../packages/fs/tool-fs), [`tool-web`](../packages/web/tool-web) | - | Collects prompt sections and model-facing tool schemas for each step. |
| `ctx.tools` | `core` | [`tools`](../packages/core/tools) | - | [`agent-loop`](../packages/core/agent-loop), [`tool-ask-user`](../packages/ui/tool-ask-user), [`tool-bash`](../packages/bash/tool-bash), [`tool-cordis`](../packages/cordis/tool-cordis), [`tool-fs`](../packages/fs/tool-fs), [`tool-skill`](../packages/skill/tool-skill), [`tool-subagent`](../packages/subagent/tool-subagent), [`tool-todo`](../packages/todo/tool-todo), [`tool-web`](../packages/web/tool-web), [`acp`](../packages/ui/acp) | - | Registers capabilities, owns Code Mode transport, and routes calls through pre-policy, monotonic guards, around dispatch, post-policy, and final-result observation. |
| `ctx.userInteraction` | `seam` | [`user-interaction`](../packages/ui/user-interaction) | [`stdio-agent`](../packages/ui/stdio-agent), [`acp`](../packages/ui/acp) | [`tool-ask-user`](../packages/ui/tool-ask-user), [`stdio-agent`](../packages/ui/stdio-agent), [`acp`](../packages/ui/acp) | - | UI front doors provide the active human-answer provider; tool-ask-user pauses a tool call on the provider-neutral ask() promise. |

25
docs/config-catalog.md
View File

@@ -590,6 +590,31 @@ export interface Config {
Source: [`packages/session-query/session-query/src/config.ts:9`](../packages/session-query/session-query/src/config.ts)
## `@deepseek-ai/dsh-session-query-sqlite`
Requires: `sessions`
```ts config-catalog
/** SQLite session-search configuration. */
export interface Config {
/** Dedicated derived-index path; `:memory:` is supported for tests. */
path: string
/** SQLite journal mode. Defaults to `wal`. */
journalMode?: JournalMode
/** Page size when a request omits `limit`. Defaults to 20. */
defaultLimit?: number
/** Largest accepted page size. Defaults to 100. */
maxLimit?: number
/** Maximum snippet length in Unicode code points. Defaults to 240. */
snippetChars?: number
}
/** Supported SQLite journal modes. */
export type JournalMode = 'wal' | 'delete' | 'truncate' | 'persist'
```
Source: [`packages/session-query/session-query-sqlite/src/index.ts:58`](../packages/session-query/session-query-sqlite/src/index.ts)
## `@deepseek-ai/dsh-skill`
```ts config-catalog

16
docs/cordis-catalog/services.md
View File

@@ -208,10 +208,11 @@ Live-preferred logical-corpus and exact-event read service.
```ts cordis-catalog
listSessions(): Promise<SessionRecord[]>
async listEvents(sessionId: SessionId): Promise<SessionEventRecord[]>
async filterEvents( sessionId: SessionId, filters: readonly SessionEventResultFilter[], ): Promise<SessionEventSearchDocument[]>
async readEvent(request: SessionEventReadRequest): Promise<SessionEventWindow>
```
Source: [`packages/session-query/session-query/src/index.ts:35`](../../packages/session-query/session-query/src/index.ts)
Source: [`packages/session-query/session-query/src/index.ts:83`](../../packages/session-query/session-query/src/index.ts)
## `ctx.sessions` — `SessionStore`
@@ -232,6 +233,19 @@ fork(source: SessionForkSource, boundary?: number, childSessionId?: SessionId):
Source: [`packages/core/session/src/index.ts:590`](../../packages/core/session/src/index.ts)
## `ctx.sessionSearch` — `SessionSearchService` (abstract seam)
Abstract full-text search service implemented by one concrete backend.
The implementation owns source observation, reconciliation, cursor generations, ranking, and query execution as one lifecycle.
```ts cordis-catalog
abstract searchSessions( request: SessionSearchRequest, exec?: SessionSearchExecContext, ): Promise<SessionSearchPage<SessionSearchHit>>
abstract searchEvents( request: SessionEventSearchRequest, exec?: SessionSearchExecContext, ): Promise<SessionSearchPage<SessionEventSearchHit>>
```
Source: [`packages/session-query/session-query/src/index.ts:54`](../../packages/session-query/session-query/src/index.ts)
## `ctx.skills` — `SkillService`
Registry of skill providers. It merges provider catalogs with stable first-wins duplicate handling, exposes sorted model-visible summaries, and loads full skill bodies on demand.

2
docs/core-data-structures/core.md
View File

@@ -19,7 +19,7 @@ Everything else is documented on a **sub-page**, not here. The rule that draws t
| [scope.md](scope.md) | scoped registration identity, dispatch carriers, and the owned `Scope` context |
| [session.md](session.md) | the full `SessionEventMap` variant catalog, `TurnTrigger`/`TurnEndReason`, `deriveMessages()`, the turn-enclosure invariant |
| [persistence.md](persistence.md) | the durability seam: `SessionPersistence`, JSONL + SQLite backends, `session/flush`, crash recovery, `SessionHeader` |
| [session-query.md](session-query.md) | logical session/event records and bounded exact-event reads |
| [session-query.md](session-query.md) | logical records, semantic filters/documents, exact reads, and full-text result pages |
| [system-prompt.md](system-prompt.md) | per-assembly context, tool-provider results, prompt sections, and cooperative assembly |
| [tools.md](tools.md) | `ToolDefinition` full fields, the schema DSL, `ToolExecution`/`ToolResult`, tool-presentation UI types, and the guarded execution pipeline |
| [user-interaction.md](user-interaction.md) | the UI-backed human question/answer seam: `AskUserQuestionRequest`, answer/options vocabulary, provider API, error taxonomy |

82
docs/core-data-structures/session-query.md
View File

@@ -1,6 +1,6 @@
# Session Query
Exact reads over the live-preferred logical session corpus. The [package contract](../../packages/session-query/session-query) owns source precedence, dynamic optional persistence, cloning, surface classification, bounded windows, and typed failures. Full-text search is a separate proposed SQLite phase.
Query vocabulary over the live-preferred logical session corpus. The [interface package](../../packages/session-query/session-query) owns exact reads, source precedence, semantic extraction and provider-independent filters, while the [SQLite package](../../packages/session-query/session-query-sqlite) owns the concrete full-text index lifecycle.
Source: [`packages/session-query/session-query/src/types.ts`](../../packages/session-query/session-query/src/types.ts)
@@ -30,6 +30,79 @@ export interface SessionEventRecord {
}
```
## Provider-independent filters and documents
Session and event filter arrays are ANDed; values inside one list clause are ORed. Ranges are inclusive. The event `text` clause is a literal Unicode case-insensitive, whitespace-flexible regular-expression scan over extracted semantic text, independent of full-text providers.
```ts type-equiv
export type SessionResultFilter =
| { kind: 'id'; values: readonly SessionId[] }
| { kind: 'cwd'; values: readonly (string | null)[] }
| ({ kind: 'created-at' } & SessionResultRange)
| { kind: 'parent'; values: readonly (SessionId | null)[] }
| { kind: 'availability'; values: readonly SessionAvailability[] }
```
```ts type-equiv
export type SessionEventResultFilter =
| ({ kind: 'seq' } & SessionResultRange)
| ({ kind: 'time' } & SessionResultRange)
| { kind: 'type'; values: readonly SessionEventType[] }
| { kind: 'surface'; values: readonly SessionEventSurface[] }
| { kind: 'text'; text: string }
```
```ts type-equiv
export interface SessionEventSearchDocument extends SessionEventRecord {
text: string
}
```
`ctx.sessionQuery.filterEvents(sessionId, filters)` returns these documents in ascending seq order. Messages, reasoning, tool calls/results, blocked prompts, todos, and failure/status detail contribute semantic text; structural events and stream chunks do not.
## Full-text search pages
The independent `ctx.sessionSearch` seam has two scopes. `searchSessions()` groups the corpus by strongest matching event; `searchEvents()` searches one session. Requests bind an opaque cursor to the normalized query, metadata filters, and limit. The event text scan is intentionally absent from provider metadata filters.
```ts type-equiv
export interface SessionSearchRequest {
query: string
sessionFilters?: readonly SessionResultFilter[]
eventFilters?: readonly SessionEventMetadataFilter[]
limit?: number
cursor?: string
}
```
```ts type-equiv
export interface SessionEventSearchRequest {
sessionId: SessionId
query: string
filters?: readonly SessionEventMetadataFilter[]
limit?: number
cursor?: string
}
```
```ts type-equiv
export interface SessionSearchPage<T> {
items: readonly T[]
nextCursor?: string
}
```
```ts type-equiv
export interface SessionEventSearchHit extends SessionEventRecord {
snippet: string
}
```
```ts type-equiv
export interface SessionSearchHit extends SessionRecord {
bestMatch: SessionEventSearchHit
}
```
## Bounded event reads
The request addresses one raw seq and optional neighboring counts. The result carries a `SessionHeader` rather than availability flags so a known live target can remain independent of persistence health.
@@ -59,11 +132,18 @@ The closed code union distinguishes request validation, missing targets, malform
```ts type-equiv
export type SessionQueryErrorCode =
| 'SESSION_QUERY_ABORTED'
| 'SESSION_QUERY_EVENT_NOT_FOUND'
| 'SESSION_QUERY_INDEX_FAILED'
| 'SESSION_QUERY_INVALID_CONFIG'
| 'SESSION_QUERY_INVALID_CURSOR'
| 'SESSION_QUERY_INVALID_FILTER'
| 'SESSION_QUERY_INVALID_LIMIT'
| 'SESSION_QUERY_INVALID_QUERY'
| 'SESSION_QUERY_INVALID_SURFACE'
| 'SESSION_QUERY_INVALID_WINDOW'
| 'SESSION_QUERY_PERSISTENCE_FAILED'
| 'SESSION_QUERY_SESSION_NOT_FOUND'
| 'SESSION_QUERY_STALE_CURSOR'
| 'SESSION_QUERY_SOURCE_CONFLICT'
```

5
docs/module-graph.md
View File

@@ -84,6 +84,7 @@ flowchart TD
end
subgraph group_session_query["packages/session-query"]
pkg_session_query["session-query"]
pkg_session_query_sqlite["session-query-sqlite"]
end
subgraph group_support["packages/support"]
pkg_acp_snapshot["acp-snapshot"]
@@ -195,6 +196,9 @@ flowchart TD
pkg_bash_sandbox --> pkg_bash
pkg_bash_sandbox --> pkg_bash_local
pkg_bash_sandbox --> pkg_sandbox
pkg_session_query_sqlite --> pkg_session
pkg_session_query_sqlite --> pkg_session_persistence
pkg_session_query_sqlite --> pkg_session_query
pkg_agent_loop --> pkg_agent
pkg_agent_loop --> pkg_llm
pkg_agent_loop --> pkg_scope
@@ -372,6 +376,7 @@ flowchart TD
| [`workflow`](../packages/workflow/workflow) | `workflow` | [`agent`](../packages/core/agent), [`brand`](../packages/util/brand), [`llm`](../packages/llm/llm) |
| [`tools`](../packages/core/tools) | `core` | [`agent`](../packages/core/agent), [`code-runtime`](../packages/code-runtime/code-runtime), [`llm`](../packages/llm/llm), [`scope`](../packages/core/scope), [`session`](../packages/core/session), [`system-prompt`](../packages/core/system-prompt), [`user-approval`](../packages/ui/user-approval) |
| [`bash-sandbox`](../packages/bash/bash-sandbox) | `bash` | [`bash`](../packages/bash/bash), [`bash-local`](../packages/bash/bash-local), [`sandbox`](../packages/sandbox/sandbox) |
| [`session-query-sqlite`](../packages/session-query/session-query-sqlite) | `session-query` | [`session`](../packages/core/session), [`session-persistence`](../packages/session-persistence/session-persistence), [`session-query`](../packages/session-query/session-query) |
| [`agent-loop`](../packages/core/agent-loop) | `core` | [`agent`](../packages/core/agent), [`llm`](../packages/llm/llm), [`scope`](../packages/core/scope), [`session`](../packages/core/session), [`session-persistence`](../packages/session-persistence/session-persistence), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools) |
| [`tool-bash`](../packages/bash/tool-bash) | `bash` | [`agent`](../packages/core/agent), [`bash`](../packages/bash/bash), [`llm`](../packages/llm/llm), [`sandbox`](../packages/sandbox/sandbox), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools), [`user-approval`](../packages/ui/user-approval) |
| [`tool-fs`](../packages/fs/tool-fs) | `fs` | [`fs`](../packages/fs/fs), [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`system-prompt`](../packages/core/system-prompt), [`tools`](../packages/core/tools) |

2
docs/rfc/INDEX.md
View File

@@ -11,7 +11,6 @@ Generated by `pnpm run gen-rfc-index` from the RFC tree — never edit by hand;
| [Pre-tool input rewrite — a consistent design](proposed/feature/2026-06-30-pre-tool-input-rewrite.md) | 2026-06-30 |
| [Claude Code and Codex subagent backends (out-of-process delegation to external coding agents)](proposed/feature/2026-07-07-claude-code-and-codex-subagent-backends.md) | 2026-07-07 |
| [Interactive side sessions and merge-back](proposed/feature/2026-07-08-interactive-side-sessions.md) | 2026-07-08 |
| [SQLite FTS5 session search](proposed/feature/2026-07-10-sqlite-session-query-provider.md) | 2026-07-10 |
| [Stream workflow progress through tool calls](proposed/feature/2026-07-13-stream-workflow-progress-through-tool-calls.md) | 2026-07-13 |
### Simplification
@@ -77,6 +76,7 @@ Generated by `pnpm run gen-rfc-index` from the RFC tree — never edit by hand;
| [Repeat-tool-call guard plugin](implemented/feature/2026-07-08-repeat-tool-guard.md) | 2026-07-08 |
| [The self-referential cordis toolset](implemented/feature/2026-07-08-self-referential-cordis-toolset.md) | 2026-07-08 |
| [Exact session query service](implemented/feature/2026-07-10-session-query-service.md) | 2026-07-10 |
| [SQLite FTS5 session search](implemented/feature/2026-07-10-sqlite-session-query-provider.md) | 2026-07-10 |
| [Configure subagent persona, tool visibility, and depth](implemented/feature/2026-07-12-subagent-persona-tool-filter-and-depth.md) | 2026-07-12 |
### Simplification

12
docs/rfc/implemented/feature/2026-07-10-session-query-service.md
View File

@@ -6,11 +6,11 @@ Status: implemented
Session history exists in two places: current `SessionStore` objects and an optional persistence backend. Consumers that need exact inspection would otherwise duplicate live-versus-persisted precedence, persistence lifecycle handling, raw-event surface classification, and defensive cloning. Durable state can lag the live log between checkpoints, so persistence alone is not a truthful current source.
Full-text search is related but materially larger. Designing provider registration, extraction, synchronization, invalidation, ranking, and cursor contracts before a real backend exists creates two speculative state machines: one in the interface service and another in the eventual database package.
Full-text search is related but materially larger. Putting provider coordination, synchronization, invalidation, ranking, and cursor state into the exact-read service would create a second state machine beside the concrete database owner.
## Decision
`@deepseek-ai/dsh-session-query` owns `ctx.sessionQuery`, a small trusted exact-read service over one logical corpus. It exposes `listSessions()`, `listEvents(sessionId)`, and bounded `readEvent(request)`. It does not expose filters, lineage or provenance traversals, text extractors, search requests, provider registration, or derived-index synchronization.
`@deepseek-ai/dsh-session-query` owns `ctx.sessionQuery`, a small trusted exact-read service over one logical corpus. It exposes `listSessions()`, `listEvents(sessionId)`, provider-independent `filterEvents(sessionId, filters)`, and bounded `readEvent(request)`. The package also declares the separate abstract `ctx.sessionSearch` contract and shared semantic extraction used by the [SQLite search decision](2026-07-10-sqlite-session-query-provider.md), but `ctx.sessionQuery` does not coordinate providers or synchronize a derived index.
The service observes the optional `ctx.sessionPersistence` binding dynamically but retains no persisted cache or invalidation listener. Each cross-corpus list asks the active backend for authoritative metadata, then overlays a fresh live-store list. Matching ids become one `SessionRecord`: the live header wins and `live`/`persisted` independently report source availability. Immutable header disagreement is `SESSION_QUERY_SOURCE_CONFLICT`.
@@ -31,11 +31,11 @@ The service is context-wide trusted infrastructure, not an authorization layer.
- **Put logical-corpus resolution directly in every consumer** — rejected because source precedence, conflicts, optional-service lifecycle, cloning, and surface classification are shared correctness rules.
- **Query only persistence** — rejected because checkpoints can lag the current live log.
- **Cache persisted metadata and listen for writes/removals** — rejected because exact reads can ask the authoritative sources directly, while cache invalidation adds lifecycle and concurrency state before scale requires it.
- **Define a provider-neutral search protocol now** — rejected because no provider consumes it. The first SQLite FTS package should own one reconciliation/transaction state machine; a smaller shared seam can be extracted later only when a second implementation proves the boundary.
- **Include lineage, provenance, and generic filters in phase one** — rejected because no current consumer requires them and canonical logs remain sufficient to add them with evidence later.
- **Put provider registration into the exact-read service** — rejected because the SQLite package owns one reconciliation/transaction lifecycle; a registry would split that state without a second provider to justify it.
- **Include lineage and provenance traversal** — rejected because canonical logs remain sufficient to add those higher-level views when a concrete consumer requires them.
## Consequences
Phase one has one source-resolution state variable: the currently mounted persistence service. There are no provider queues, fingerprints, extractor registries, observation generations, or derived index updates. Exact reads remain usable in live-only deployments and deterministic when persistence is present.
Exact reads have one source-resolution state variable: the currently mounted persistence service. There are no provider queues, fingerprints, extractor registries, observation generations, or derived index updates in `ctx.sessionQuery`. Exact reads and semantic scans remain usable in live-only deployments and deterministic when persistence is present.
Cross-corpus listing and persisted exact reads perform backend I/O on each call. That is deliberate: correctness comes from current authoritative state, and scale-oriented search belongs to the phase-two database. Full-text search is unavailable until that package defines and implements its complete contract.
Cross-corpus listing and persisted exact reads perform backend I/O on each call. That is deliberate: correctness comes from current authoritative state, while scale-oriented full-text search uses the separately owned SQLite derived index.

57
docs/rfc/implemented/feature/2026-07-10-sqlite-session-query-provider.md Normal file
View File

@@ -0,0 +1,57 @@
# RFC: SQLite FTS5 session search
Status: implemented
## Problem
The exact-read `ctx.sessionQuery` service deliberately has no derived index. Large persisted histories need full-text search without scanning every event on every query, while current live sessions need an overlay newer than the last durability checkpoint. Search also needs concrete ranking, snippets, filters, pagination, cancellation, and rebuild behavior.
Splitting those concerns across a provider coordinator and a database implementation would create two coupled reconciliation state machines. The first implementation needs to own source observation, extraction, SQLite transactions, generations, and query execution as one lifecycle while still exposing a small provider-neutral call contract.
## Decision
`@deepseek-ai/dsh-session-query` declares an independent abstract `ctx.sessionSearch` service without changing the exact-read `ctx.sessionQuery` key. `searchSessions(request, exec?)` returns cursor-paginated `SessionSearchHit`s grouped by each session's strongest matching event; `searchEvents(request, exec?)` returns `SessionEventSearchHit`s within one logical session. Both requests require `query`, accept `limit` and an opaque `cursor`, and support an optional abort signal. Session search accepts `sessionFilters` plus event metadata filters; event search accepts event metadata filters. Results expose bounded plain-text snippets but no provider identifier or numeric relevance score.
`@deepseek-ai/dsh-session-query-sqlite` is the sole concrete owner of `ctx.sessionSearch`. It depends on live `ctx.sessions`, observes optional `ctx.sessionPersistence` dynamically, and owns a dedicated derived SQLite database. There is no search-provider registry, coordinator, persistence event, or agent-loop integration.
The interface package also owns shared first-party semantic extraction and provider-independent filtering. `SessionResultFilter` covers id, nullable cwd, created-at range, nullable parent, and availability. `SessionEventResultFilter` covers seq/time ranges, event type, surface, and literal semantic text. Arrays are ANDed and list values are ORed. The text clause escapes caller input into a Unicode case-insensitive regular expression whose whitespace runs match one or more whitespace characters; it is available through `ctx.sessionQuery.filterEvents()` and is not delegated to an FTS provider.
## Search semantics
Each semantic event is one FTS document carrying session metadata, event metadata, surface classification, and extracted text. All `current`, `shadowed`, and `log-only` documents participate unless a surface filter narrows them. Metadata filters compile to parameterized SQL before ranking. Session results partition matching documents by session and retain the strongest one.
Ordering is deterministic: BM25 ascending, event time descending, session id ascending for the cross-session scope, and seq descending. Provider scores remain private. Snippets normalize whitespace and are bounded by Unicode code points. Opaque cursors bind to the service instance, scope, canonical normalized request, offset, and relevant generation. Any corpus change invalidates cross-session cursors; a within-session cursor changes only when its target source/generation changes, so unrelated sessions do not invalidate it. Reopening creates a new service instance and invalidates old cursors.
Queries are trimmed, whitespace-normalized, and quoted as one literal FTS5 phrase. Embedded quotes are doubled before binding, so MATCH operators such as `OR`, `NEAR`, quotes, parentheses, and `*` remain data rather than executable query syntax. Phrase matching follows tokenizer tokens rather than arbitrary substrings.
## Tokenizer choice
Both persistent and live FTS5 tables use `unicode61`. The implementation experiment found that this tokenizer supports the two-character token `AI` and produces an index about 2.1× smaller than the trigram alternative. The accepted limitation is token/phrase recall: `AI` does not match the larger token `BRAID`, and arbitrary substring search uses the provider-independent text scan instead.
## Extraction and reconciliation
The shared extractor includes message text, reasoning, nested tool-call/result content, tool names and arguments, blocked-prompt reasons, todo status/content, and error or terminal status detail. Structural boundaries, stream chunks, request headers, successful completion markers, and unknown declaration-merged event/content variants produce no document. Surface classification reuses `foldSurface()` so search agrees with model-history derivation.
One serialized operation observes complete persisted and live sources, computes stable fingerprints, reconciles rows in one transaction, and executes the query. Unchanged persisted sessions retain their rows and generation. New, changed, and deleted persisted sessions update on the next search. A source or extraction failure cannot mark a row current, and a transaction failure rolls back so a later search retries.
Persisted documents survive restarts. Live sessions use connection-local TEMP tables, shadow the persisted base for the same id, and reveal that base on detach. Closing the database drops live rows. Unmounting persistence hides durable rows without treating absence as authoritative deletion; remounting observes and reconciles the backend again. Conflicting immutable live and durable headers fail rather than combining sources.
The derived schema has its own application id and monotonic schema version. A recognized incompatible version resets only this derived database. A database with a foreign application id or unrecognized user tables is refused, which prevents an accidentally configured canonical session database from being reset.
Cancellation rejects queued operations and caller waits around asynchronous source observation without committing an aborted observation. Node's synchronous `DatabaseSync` MATCH call cannot be interrupted once it is executing on the JavaScript thread, so the service checks the signal at serialized boundaries but does not promise mid-statement preemption.
## Alternatives considered
- **Add FTS tables to the canonical persistence database** — rejected because a rebuildable index must not share the authoritative log's schema, reset, or failure boundary.
- **Add a phase-one provider registry and coordinator** — rejected because one implementation provides no evidence for registration semantics and would split one reconciliation lifecycle across two owners.
- **Persist live overrides immediately** — rejected because live events are not canonical until the existing checkpoint commits.
- **Use the FTS5 trigram tokenizer** — rejected because it omits useful queries shorter than three characters and measured about 2.1× the index size of `unicode61`; literal substring filtering remains available through the scan path.
- **Return BM25 scores** — rejected because provider-specific numeric scales are unstable across corpus changes.
## Consequences
Search has a small provider-neutral API while its only backend owns every derived-index state transition. The separate database adds configuration and a reconciliation read before queries, but index corruption, reset, and tokenizer changes cannot endanger canonical logs. Persistent fingerprints avoid rewriting unchanged sessions; TEMP live overlays preserve current-session truth without making uncheckpointed events durable.
The chosen tokenizer supports short tokens with a smaller index but does not promise substring recall. Literal phrases make query syntax safe and predictable at the cost of excluding boolean/full MATCH expressions. Cancellation is effective while queued or awaiting sources, but synchronous SQLite execution remains a non-preemptible section.
Unit coverage pins extraction, filters, both search scopes, all default surfaces, metadata-before-ranking, snippets, literal escaping, deterministic ties, complete pagination, scoped cursor invalidation, dynamic persistence mount/unmount, restart reconciliation, live shadow/reveal/reopen, schema safety, rollback retry, and queued/in-flight source-wait cancellation. A keyless real-Loader-path test combines the package with the real SQLite persistence backend.

51
docs/rfc/proposed/feature/2026-07-10-sqlite-session-query-provider.md
View File

@@ -1,51 +0,0 @@
# RFC: SQLite FTS5 session search
Status: proposed
## Problem
The exact-read `ctx.sessionQuery` service deliberately has no derived index. Large persisted histories need full-text search without scanning every event on every query, while current live sessions need an overlay newer than the last durability checkpoint. Search also needs concrete ranking, snippets, filters, pagination, cancellation, and rebuild behavior.
Splitting those concerns across a speculative provider coordinator and a database implementation would create two coupled reconciliation state machines. The first real implementation should own the source observation, extraction, SQLite transaction, generation, and query as one lifecycle.
## Proposal
Add `@deepseek-ai/dsh-session-query-sqlite` beside the exact-read package. The package will expose a search service or extend the family with the smallest API required by its actual consumers; phase one does not pre-commit a provider-registration protocol. It will depend on `ctx.sessions` and optional `ctx.sessionPersistence`, own a separate derived SQLite database, and reuse the canonical `foldSurface()` classification.
The implementation owns one serialized reconciliation/DB transaction state machine. A transaction observes authoritative persisted metadata and live snapshots, extracts semantic documents, updates derived tables, advances relevant cursor generations, and executes or enables the corresponding query. No second service maintains parallel fingerprints, dirty flags, live-id sets, or invalidation generations.
Persisted documents survive restarts. Live overrides are connection-local and shadow the persisted rows for the same session, then disappear when the live owner or database closes. The derived database remains separate from canonical persistence so index reset, corruption, tokenizer changes, and schema churn cannot endanger durable conversation logs.
## Search semantics to decide with implementation
The implementation must define both cross-session and within-session scopes from executable use cases. Each searchable event is one document with session metadata, event metadata, surface classification, normalized semantic text, and a bounded plain-text snippet. Session results group by their strongest matching event; numeric backend scores remain private.
Filters compile to parameterized SQL before ranking. Query syntax is treated as data. Ordering includes stable tie fields. Opaque cursors bind to normalized request shape and the smallest relevant generation; unrelated session changes should not invalidate a within-session cursor. Cancellation must stop caller waiting and interrupt SQLite work where the runtime permits.
Tokenizer choice remains an implementation experiment. FTS5 trigram supports substring recall but rejects useful terms shorter than three characters and increases index size; the proposal must benchmark that tradeoff against the default Unicode tokenizer before making it contract.
## Extraction and reconciliation
The package starts with first-party semantic extraction for messages, reasoning, tool calls/results, blocked prompts, context, steering, todos, and error/status detail. Structural events and stream chunks contribute no document. Unknown declaration-merged event/content types remain non-searchable unless a real extension consumer demonstrates the need for a public extractor registry.
Reconciliation may use stable fingerprints to avoid rewriting unchanged persisted sessions, but the database package owns their calculation and storage. It must never report a row current when source observation or extraction failed. Provider-schema mismatch may reset only the derived database; ordinary source changes use transactional upsert/delete. Mounted but unreadable persistence fails affected searches without affecting canonical writes or known live exact reads.
## Alternatives considered
- **Add FTS tables to the canonical persistence database** — rejected because a rebuildable index must not share the authoritative log's schema/reset/failure boundary.
- **Reintroduce phase-one provider coordination** — rejected because there is one planned implementation and no evidence for a stable multi-provider seam.
- **Persist live overrides immediately** — rejected because live events are not canonical until the existing checkpoint commits.
- **Return BM25 scores** — rejected because provider-specific numeric scales are unstable across corpus changes.
## Acceptance criteria
- Restart tests cover unchanged, new, changed, and deleted persisted sessions without rebuilding the whole index.
- Reopening preserves persisted rows and removes live rows; live rows shadow and then reveal their persisted base.
- Tests cover both search scopes, metadata filters, surface defaults, snippets, escaping, deterministic ties, pagination, scoped stale cursors, cancellation, dynamic persistence mount/unmount, and recovery after a failed transaction.
- A schema mismatch resets only the derived database.
- A keyless end-to-end test combines a real persistence backend with the real SQLite search package.
- The RFC is amended to the measured tokenizer and public API actually implemented before moving to `implemented/`.
## Risks
A single owner is simpler but initially less reusable than a provider-neutral seam. That is intentional: a second real backend can reveal what to extract. SQLite runtime differences can affect FTS ranking and snippets, so tests must pin only contract-controlled ordering and presentation. The separate database adds configuration and lifecycle work, but preserves the canonical store's safety boundary.