Merge remote-tracking branch 'origin/master' into worktree/web-session-titles
This commit is contained in:
Tianyi Cui
@@ -8,9 +8,9 @@ Status: implemented
|
||||
|
||||
## Decision
|
||||
|
||||
Extract a backend-agnostic `PersistenceCoordinator` into `dsh-session-persistence`. The coordinator owns the orchestration once; each first-party backend composes one (`new PersistenceCoordinator(ctx, this)`), implements a small `PersistenceBackend` hook interface, and delegates its four public service methods (`create`/`append`/`load`/`list`) to it.
|
||||
Extract a backend-agnostic `PersistenceCoordinator` into `dsh-session-persistence`. The coordinator owns the orchestration once; each first-party backend composes one (`new PersistenceCoordinator(ctx, this)`), implements a small `PersistenceBackend` hook interface, and delegates its stateful public methods (`create`/`append`/`load`/`inspect`) to it. Backend-owned metadata and revision listing bypass the coordinator.
|
||||
|
||||
Composition, not inheritance. The coordinator is a concrete class the backend holds, not a base class the backend extends. The Agent Note's risk — "a coordinator must not make unusual backends fight an inheritance hierarchy" — is avoided: a backend exposes only the hooks; it cannot reach the coordinator's private orchestration state, and the public `SessionPersistence` service shape is unchanged, so a third-party backend MAY still implement the abstract service directly without the coordinator at all.
|
||||
Composition, not inheritance. The coordinator is a concrete class the backend holds, not a base class the backend extends. The Agent Note's risk — "a coordinator must not make unusual backends fight an inheritance hierarchy" — is avoided: a backend exposes only the hooks and cannot reach the coordinator's private orchestration state. A third-party backend MAY still implement the abstract service directly without the coordinator, including the non-mutating `inspect` contract used by read models.
|
||||
|
||||
The coordinator retires each live session from its `session/disposed` notification: it waits for that exact Session object's initialization, serializes a final drain, and then removes the owned state, buffer, and init entries. Failed drains retain their buffers for backend teardown to retry. Settled per-id chain tails remove themselves only when they are still the current tail, so a completion cannot erase a newer operation for the same id. Backend teardown unregisters the write-path listeners before awaiting all admitted retirements, remaining buffers, and chains, then closes the backend.
|
||||
|
||||
@@ -19,7 +19,7 @@ The coordinator retires each live session from its `session/disposed` notificati
|
||||
Five required members plus an optional lifecycle hook form the only boundary between the coordinator and storage:
|
||||
|
||||
- `name` — backend label for the dispose-failure `AggregateError`.
|
||||
- `loadStored(id)` — read one stored prefix by id across every storage scope (every JSONL cwd bucket; SQLite's id is globally unique). Resume/load, live adoption, and the create-collision probe share this lookup. The coordinator asserts the returned id and rejects a stored/live cwd mismatch before repair or state publication.
|
||||
- `loadStored(id)` — read one stored prefix by id across every storage scope (every JSONL cwd bucket; SQLite's id is globally unique). Resume/load, non-mutating inspection, live adoption, and the create-collision probe share this lookup. The coordinator asserts the returned id and rejects a stored/live cwd mismatch before repair or state publication.
|
||||
- `appendBatch(meta, events, isMaterialized)` — durably append a contiguous batch, lazily materializing the session ATOMICALLY when not yet materialized (the materialize-write and the first event batch must commit together — a crash between them must not leave a materialized-but-empty session; this is why there is no separate `materialize` hook).
|
||||
- `commitRepair(meta, tornMarker, closers)` — make a crash repair durable: truncate the torn tail (iff `tornMarker !== undefined`) and append `closers`. **NOT required to be atomic** — JSONL legitimately truncates-then-appends in two fsync'd steps, SQLite does DELETE+INSERT in one transaction. Used by `load` (truncate + synthetic closers) and live-adoption (truncate only, `closers = []`).
|
||||
- `list()` — list all stored metadata.
|
||||
@@ -31,7 +31,7 @@ The single design choice that keeps the seam clean: the crash-repair "where is t
|
||||
|
||||
## Testing
|
||||
|
||||
The shared `runPersistenceContract` (public-API contract) keeps running for every backend. `runCoordinatorContract` (`tests/coordinator-contract.ts`) holds the write-path orchestration — adoption, HMR, collision, session and backend disposal drains, and crash-tail repair — and runs once per backend through a `CoordinatorFixture` (an in-memory reference + jsonl + sqlite). Coordinator-specific tests pin retirement map cleanup, same-id chain-tail races, failed-drain retry, and close ordering. The per-backend specs retain storage mechanics only (JSONL: path safety, fsync rollback, bucket listing; SQLite: schema version, `scanRows`, transaction rollback). A through-coordinator torn-tail→load→`commitRepair` test per real backend (via a `corruptTail` fixture hook) keeps the coordinator's torn-marker repair branch covered under the 100% per-file gate — the contract crash test only produces synthetic closers, never a torn marker, so it could not reach that branch.
|
||||
The shared `runPersistenceContract` (public-API contract) keeps running for every backend and proves that `inspect` leaves interrupted logs and revisions unchanged before `load` performs recovery. `runCoordinatorContract` (`tests/coordinator-contract.ts`) holds the write-path orchestration — adoption, HMR, collision, session and backend disposal drains, and crash-tail repair — and runs once per backend through a `CoordinatorFixture` (an in-memory reference + jsonl + sqlite). Coordinator-specific tests pin retirement map cleanup, same-id chain-tail races, failed-drain retry, and close ordering. The per-backend specs retain storage mechanics only (JSONL: path safety, fsync rollback, bucket listing; SQLite: schema version, `scanRows`, transaction rollback). A through-coordinator torn-tail→load→`commitRepair` test per real backend (via a `corruptTail` fixture hook) keeps the coordinator's torn-marker repair branch covered under the 100% per-file gate — the contract crash test only produces synthetic closers, never a torn marker, so it could not reach that branch.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
@@ -40,4 +40,4 @@ The shared `runPersistenceContract` (public-API contract) keeps running for ever
|
||||
|
||||
## Consequences
|
||||
|
||||
The coordinator adds one indirection, an opaque torn marker, and detached session-retirement tasks, but centralizes correctness-heavy orchestration previously duplicated by every backend. Session disposal remains an observe-only event, so the session owner does not await persistence retirement; the coordinator contains failures, preserves uncommitted buffers, and makes backend teardown the quiescence boundary. Its hook surface stays narrow: identity, adoption, and collision checks reuse `loadStored`; materialization stays atomic inside `appendBatch`; and listing bypasses the coordinator. New backends implement storage primitives rather than copy the event-buffer-flush lifecycle.
|
||||
The coordinator adds one indirection, an opaque torn marker, and detached session-retirement tasks, but centralizes correctness-heavy orchestration previously duplicated by every backend. Session disposal remains an observe-only event, so the session owner does not await persistence retirement; the coordinator contains failures, preserves uncommitted buffers, and makes backend teardown the quiescence boundary. Its hook surface stays narrow: identity, adoption, collision checks, and non-mutating inspection reuse `loadStored`; materialization stays atomic inside `appendBatch`; and listing bypasses the coordinator. Read models use `inspect` rather than `load`, so observing a persisted open turn cannot race a new live owner by committing interruption closers. New backends implement storage primitives rather than copy the event-buffer-flush lifecycle.
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-23-unified-session-query-service.md: 0a466e1c36ff1796c858666b0eb36bbd0f480bb0
|
||||
2026-07-23-unified-session-query-service.zh.md: 448122b8e6951058b9f633cd56112b0391e1912e
|
||||
@@ -0,0 +1,35 @@
|
||||
# Agent Note: Unified session query service
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-07-23-unified-session-query-service.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
Exact reads, semantic filters, relationship traces, and full-text search operate on the same live-preferred session corpus. Exposing full-text search under a second context key makes consumers and app compositions treat one capability as two services, even though the SQLite implementation is the only backend-specific part.
|
||||
|
||||
The interface package already owns the shared record, filter, trace, search-request, cursor, and error contracts. A provider registry or coordinator would add runtime selection semantics unsupported by any current consumer.
|
||||
|
||||
## Decision
|
||||
|
||||
`SessionQueryService` is the single abstract service registered as `ctx.sessionQuery`. It concretely implements listing, title and event reads, surface reads, filtering, and relationship tracing through its backend-independent `SessionCorpus`. Its only abstract methods are `searchSessions()` and `searchEvents()`.
|
||||
|
||||
`SessionQuerySqlite` extends that service and is the sole concrete backend. One mounted instance therefore exposes every operation through `ctx.sessionQuery`; its inherited exact operations use the shared corpus implementation, while its SQLite-owned lifecycle observes sources, reconciles the derived FTS index, ranks matches, and owns cursor generations. The interface package has no standalone concrete plugin, search-provider registry, or second context key.
|
||||
|
||||
Backend configuration includes the inherited `readWindowMax` setting alongside its own index path, journal mode, page limits, and snippet limit. First-party apps that need session queries mount the SQLite backend and place its disposable index beside their configured persistence root.
|
||||
|
||||
This service topology supersedes the separate-key portion of the [exact query decision](../feature/2026-07-10-session-query-service.md) and [SQLite search decision](../feature/2026-07-10-sqlite-session-query-provider.md); their corpus, query, tokenizer, reconciliation, and safety decisions remain in force.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Keep `ctx.sessionQuery` and `ctx.sessionSearch` separate** — rejected because both expose operations over one logical corpus, force consumers to discover two keys, and let apps accidentally mount only a partial query surface.
|
||||
- **Keep a concrete base service and let the SQLite plugin register or mutate two search methods** — rejected because method availability would depend on plugin order and teardown, and the service would need a provider registration protocol for one implementation.
|
||||
- **Move every query implementation into the SQLite package** — rejected because exact reads, filters, and traces require no index and are shared behavior that belongs with their provider-independent contracts.
|
||||
|
||||
## Consequences
|
||||
|
||||
Consumers inject one service and can combine exact and full-text operations without a second capability lookup. A production composition must choose a concrete backend even when one consumer currently calls only inherited exact methods; tests may use a minimal subclass when backend behavior is outside their scope.
|
||||
|
||||
The unified object deliberately retains two internal observation strategies: exact operations read authoritative live/persisted sources per call, while full-text operations reconcile a disposable index. Sharing the context key does not make the derived index authoritative or couple exact-read availability to an FTS query.
|
||||
|
||||
Unit coverage pins inherited and abstract behavior on one key, SQLite coverage exercises both operation families on the concrete backend, and the real Loader path verifies that one exported plugin registers the combined service.
|
||||
@@ -0,0 +1,35 @@
|
||||
# Agent Note: 统一会话查询服务
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-07-23-unified-session-query-service.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
精确读取、语义过滤、关系追踪与全文搜索都作用于同一个实时源优先的会话语料库。将全文搜索暴露在第二个上下文键下，会让消费方与应用组合把同一项查询功能视为两个服务，尽管只有 SQLite 实现是后端特有的部分。
|
||||
|
||||
接口包已经拥有共享的记录、过滤、追踪、搜索请求、游标与错误契约。提供方注册表或协调器会引入运行时选择语义，而目前没有任何消费方支持这种语义。
|
||||
|
||||
## 决策
|
||||
|
||||
`SessionQueryService` 是注册为 `ctx.sessionQuery` 的唯一抽象服务。它通过后端无关的 `SessionCorpus` 具体实现列表查询、标题与事件读取、表层读取、过滤和关系追踪。仅有 `searchSessions()` 与 `searchEvents()` 两个方法为抽象方法。
|
||||
|
||||
`SessionQuerySqlite` 扩展该服务，并且是唯一的具体后端。因此，一个挂载实例便可通过 `ctx.sessionQuery` 暴露全部操作；其继承的精确操作使用共享的语料库实现，而由 SQLite 管理的生命周期负责观察数据源、对齐派生 FTS 索引、对匹配项排序并管理游标代际。接口包不提供独立的具体插件、搜索提供方注册表或第二个上下文键。
|
||||
|
||||
后端配置除了自身的索引路径、日志模式、分页限制与文本片段长度上限外，还包含继承的 `readWindowMax` 设置。需要会话查询的第一方应用挂载 SQLite 后端，并将其可丢弃索引放在已配置的持久化根目录旁。
|
||||
|
||||
这一服务拓扑取代了[精确查询决策](../feature/2026-07-10-session-query-service.md)和 [SQLite 搜索决策](../feature/2026-07-10-sqlite-session-query-provider.md)中关于分离上下文键的部分；其中关于语料库、查询、分词器、对齐与安全性的决策仍然有效。
|
||||
|
||||
## 已考虑的替代方案
|
||||
|
||||
- **保留相互独立的 `ctx.sessionQuery` 与 `ctx.sessionSearch`**：不予采纳，因为二者都针对同一逻辑语料库提供操作，迫使消费方识别两个键，还可能让应用误挂载一组不完整的查询接口。
|
||||
- **保留具体的基础服务，再由 SQLite 插件注册或修改两个搜索方法**：不予采纳，因为方法是否可用将取决于插件顺序与资源释放时机，而且该服务需要为唯一的实现定义一套提供方注册协议。
|
||||
- **将所有查询实现移入 SQLite 包**：不予采纳，因为精确读取、过滤与追踪不需要索引，并且都属于应与提供方无关契约放在一起的共享行为。
|
||||
|
||||
## 后果
|
||||
|
||||
消费方只需注入一个服务，无需再次查找其他功能，便可组合精确操作与全文操作。生产环境的组合必须选择一个具体后端，即使当前某个消费方只调用继承的精确方法；如果后端行为不在测试范围内，测试可以使用最小子类。
|
||||
|
||||
统一后的对象有意保留两种内部观察策略：精确操作在每次调用时读取权威的实时源或持久化源，全文操作则使可丢弃索引与数据源对齐。共用上下文键不会让派生索引成为权威来源，也不会使精确读取的可用性依赖 FTS 查询。
|
||||
|
||||
单元测试在同一个键上同时固定继承实现与抽象方法的契约，SQLite 测试在具体后端上覆盖两类操作，真实 Loader 路径则验证单个导出的插件能够注册组合后的服务。
|
||||
@@ -6,11 +6,11 @@ Status: implemented
|
||||
|
||||
Session history exists in two places: current `SessionStore` objects and an optional persistence backend. Consumers that need exact inspection would otherwise duplicate live-versus-persisted precedence, persistence lifecycle handling, raw-event surface classification, relationship tracing, and defensive cloning. Durable state can lag the live log between checkpoints, so persistence alone is not a truthful current source.
|
||||
|
||||
Full-text search is related but materially larger. Designing provider registration, extraction, synchronization, invalidation, ranking, and cursor contracts before a real backend exists creates two speculative state machines: one in the interface service and another in the eventual database package.
|
||||
Full-text search is related but materially larger. Putting provider coordination, synchronization, invalidation, ranking, and cursor state into the exact-read service would create a second state machine beside the concrete database owner.
|
||||
|
||||
## Decision
|
||||
|
||||
`@deepseek-ai/dsh-session-query` owns `ctx.sessionQuery`, a small trusted exact-inspection service over one logical corpus. It exposes `listSessions()`, `listEvents(sessionId)`, bounded `readEvent(request)`, `traceSession(sessionId)`, and `traceEvent(request)`. It does not expose filters, text extractors, search requests, provider registration, or derived-index synchronization. The separate [tracing decision](2026-07-13-session-query-tracing.md) owns lineage and event-relationship semantics.
|
||||
`@deepseek-ai/dsh-session-query` owns the single abstract `ctx.sessionQuery` service over one logical corpus. It concretely implements `listSessions()`, provider-independent `filterSessions(filters)`, `listEvents(sessionId)`, `filterEvents(sessionId, filters)`, bounded `readEvent(request)`, `traceSession(sessionId)`, and `traceEvent(request)`, while concrete backends implement its two full-text methods. The [unified service decision](../architecture/2026-07-23-unified-session-query-service.md) owns that topology, the [SQLite search decision](2026-07-10-sqlite-session-query-provider.md) owns search behavior, and the [tracing decision](2026-07-13-session-query-tracing.md) owns lineage and event-relationship semantics.
|
||||
|
||||
The service observes the optional `ctx.sessionPersistence` binding dynamically but retains no persisted cache or invalidation listener. Each cross-corpus list asks the active backend for authoritative metadata, then overlays a fresh live-store list. Matching ids become one `SessionRecord`: the live header wins and `live`/`persisted` independently report source availability. Immutable header disagreement is `SESSION_QUERY_SOURCE_CONFLICT`.
|
||||
|
||||
@@ -31,10 +31,10 @@ The service is context-wide trusted infrastructure, not an authorization layer.
|
||||
- **Put logical-corpus resolution directly in every consumer** — rejected because source precedence, conflicts, optional-service lifecycle, cloning, and surface classification are shared correctness rules.
|
||||
- **Query only persistence** — rejected because checkpoints can lag the current live log.
|
||||
- **Cache persisted metadata and listen for writes/removals** — rejected because exact reads can ask the authoritative sources directly, while cache invalidation adds lifecycle and concurrency state before scale requires it.
|
||||
- **Define a provider-neutral search protocol now** — rejected because no provider consumes it. The first SQLite FTS package should own one reconciliation/transaction state machine; a smaller shared seam can be extracted later only when a second implementation proves the boundary.
|
||||
- **Put provider registration into the exact-read service** — rejected because the SQLite package owns one reconciliation/transaction lifecycle; a registry would split that state without a second provider to justify it.
|
||||
|
||||
## Consequences
|
||||
|
||||
The service has one source-resolution state variable: the currently mounted persistence service. There are no provider queues, fingerprints, extractor registries, observation generations, or derived index updates. Exact reads and event traces remain usable in live-only deployments and deterministic when persistence is present.
|
||||
The inherited exact-read implementation has one source-resolution state variable: the currently mounted persistence service. It has no provider queues, fingerprints, extractor registries, observation generations, or derived index updates; a concrete backend owns its full-text state separately. Exact reads, semantic scans, and event traces remain usable in live-only deployments and deterministic when persistence is present.
|
||||
|
||||
Cross-corpus listing, lineage tracing, and persisted event operations perform backend I/O on each call. That is deliberate: correctness comes from current authoritative state, and scale-oriented search belongs to the proposed database package. Full-text search is unavailable until that package defines and implements its complete contract.
|
||||
Cross-corpus listing, lineage tracing, and persisted event operations perform backend I/O on each call. That is deliberate: correctness comes from current authoritative state, while scale-oriented full-text methods use the concrete backend's SQLite derived index.
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
# Agent Note: SQLite FTS5 session search
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
The exact-read `ctx.sessionQuery` service deliberately has no derived index. Large persisted histories need full-text search without scanning every event on every query, while current live sessions need an overlay newer than the last durability checkpoint. Search also needs concrete ranking, snippets, filters, pagination, cancellation, and rebuild behavior.
|
||||
|
||||
Splitting those concerns across a provider coordinator and a database implementation would create two coupled reconciliation state machines. The first implementation needs to own source observation, extraction, SQLite transactions, generations, and query execution as one lifecycle while still exposing a small provider-neutral call contract.
|
||||
|
||||
## Decision
|
||||
|
||||
`@deepseek-ai/dsh-session-query` declares one abstract `ctx.sessionQuery` service whose exact reads, filters, and traces are concrete and whose two full-text methods are abstract. `searchSessions(request, exec?)` returns cursor-paginated `SessionSearchHit`s grouped by each session's strongest matching event; `searchEvents(request, exec?)` returns `SessionEventSearchHit`s within one logical session. Both requests require `query`, accept `limit` and an owned branded `SessionSearchCursor`, and support an optional abort signal. Session search accepts `sessionFilters` plus event metadata filters; event search accepts event metadata filters. Results expose bounded plain-text snippets but no provider identifier or numeric relevance score. The [unified service decision](../architecture/2026-07-23-unified-session-query-service.md) owns the single-key topology.
|
||||
|
||||
`@deepseek-ai/dsh-session-query-sqlite` extends the interface service and is the sole concrete owner of `ctx.sessionQuery`. It depends on live `ctx.sessions`, observes optional `ctx.sessionPersistence` dynamically, and owns a dedicated derived SQLite database. There is no search-provider registry, coordinator, persistence event, or agent-loop integration.
|
||||
|
||||
The interface package also owns shared first-party semantic extraction and provider-independent filtering. `SessionResultFilter` covers id, nullable cwd, created-at range, nullable parent, and availability; `ctx.sessionQuery.filterSessions()` applies it without an FTS provider. `SessionEventResultFilter` covers seq/time ranges, event type, surface, and literal semantic text. Arrays are ANDed and list values are ORed. The text clause escapes caller input into a Unicode case-insensitive regular expression whose whitespace runs match one or more whitespace characters; it is available through `ctx.sessionQuery.filterEvents()` and is not delegated to an FTS provider.
|
||||
|
||||
## Search semantics
|
||||
|
||||
Each semantic event is one FTS document carrying session metadata, event metadata, surface classification, and extracted text. All `current`, `shadowed`, and `log-only` documents participate unless a surface filter narrows them. Metadata filters compile to parameterized SQL before ranking. Session results partition matching documents by session and retain the strongest one.
|
||||
|
||||
Ordering is deterministic and comparable across the persistent and TEMP FTS tables: actual FTS5 highlighted-match span count descending, indexed document code-point length ascending, event time descending, session id ascending for the cross-session scope, and seq descending. Snippets use those actual highlight positions, strip the reserved markers, normalize whitespace, and bound by Unicode code points. Opaque cursors bind to the service instance, scope, canonical normalized request, offset, and relevant generation. Any corpus change invalidates cross-session cursors; a within-session cursor changes only when its target source/generation changes, so unrelated sessions do not invalidate it. Reopening creates a new service instance and invalidates old cursors.
|
||||
|
||||
Queries are trimmed, whitespace-normalized, and quoted as one literal FTS5 phrase. Embedded quotes are doubled before binding, so MATCH operators such as `OR`, `NEAR`, quotes, parentheses, and `*` remain data rather than executable query syntax. NUL is rejected before SQLite execution. Reserved highlight noncharacters and NUL in documents are normalized before indexing, making inserted presentation markers collision-free. Phrase matching follows tokenizer tokens rather than arbitrary substrings.
|
||||
|
||||
## Tokenizer choice
|
||||
|
||||
Both persistent and live FTS5 tables use `unicode61`. The implementation experiment found that this tokenizer supports the two-character token `AI` and produces an index about 2.1× smaller than the trigram alternative. The accepted limitation is token/phrase recall: `AI` does not match the larger token `BRAID`, and arbitrary substring search uses the provider-independent text scan instead.
|
||||
|
||||
## Extraction and reconciliation
|
||||
|
||||
The shared extractor includes message text, reasoning, nested tool-call/result content, tool names and arguments, blocked-prompt reasons, todo status/content, and error or terminal status detail. Structural boundaries, stream chunks, request headers, successful completion markers, and unknown declaration-merged event/content variants produce no document. Surface classification reuses `foldSurface()` so search agrees with model-history derivation.
|
||||
|
||||
One serialized operation reads the provider-neutral `SessionPersistence` snapshot listing, compares each source-qualified opaque revision with the revision stored beside the indexed session, loads only new or changed logs, reconciles rows in one transaction, and executes the query. It never calls the backend's mutating `load()` for an id currently owned by `ctx.sessions`; the TEMP overlay records persisted availability, and the durable base refreshes after the live owner detaches. A revision identifies its backing persistence store as well as the backend-local log revision, so reopening against the same store reuses indexed rows while switching to an independent store cannot collide on a session id and local counter. Observation repeats when listing changes during a load; this incorporates a mutating load repair's refreshed revision before commit. Repeated queries and unchanged reopen load no full persisted logs. New, changed, and deleted sessions update on the next stable search. A source or extraction failure cannot mark a row current, and a transaction failure rolls back so a later search retries.
|
||||
|
||||
Persisted documents survive restarts. Live sessions use connection-local TEMP tables, shadow the persisted base for the same id, and reveal that base on detach. Closing the database drops live rows. Unmounting persistence hides durable rows without treating absence as authoritative deletion; remounting observes and reconciles the backend again. Conflicting immutable live and durable headers fail rather than combining sources.
|
||||
|
||||
The derived schema has its own application id and monotonic schema version. A recognized incompatible version resets only this derived database. A database with a foreign application id or unrecognized user tables is refused before journal-mode mutation, which prevents an accidentally configured canonical session database from being changed. On POSIX filesystems, missing directories and database files are created owner-only so new SQLite sidecars inherit that mode; existing modes are preserved. One service in one process exclusively owns a derived-index path; cross-process writers are unsupported because generations and live TEMP shadow state are connection-owned.
|
||||
|
||||
Cancellation rejects queued operations and caller waits around asynchronous source observation without committing an aborted observation. Node's synchronous `DatabaseSync` MATCH call cannot be interrupted once it is executing on the JavaScript thread, so the service checks the signal at serialized boundaries but does not promise mid-statement preemption.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Add FTS tables to the canonical persistence database** — rejected because a rebuildable index must not share the authoritative log's schema, reset, or failure boundary.
|
||||
- **Add a phase-one provider registry and coordinator** — rejected because one implementation provides no evidence for registration semantics and would split one reconciliation lifecycle across two owners.
|
||||
- **Persist live overrides immediately** — rejected because live events are not canonical until the existing checkpoint commits.
|
||||
- **Use the FTS5 trigram tokenizer** — rejected because it omits useful queries shorter than three characters and measured about 2.1× the index size of `unicode61`; literal substring filtering remains available through the scan path.
|
||||
- **Use FTS5 BM25 independently in each table** — rejected because scores from differently populated persistent and TEMP corpora are not comparable; actual matched spans and document length have one shared scale.
|
||||
|
||||
## Consequences
|
||||
|
||||
Search has a small provider-neutral API while its only backend owns every derived-index state transition. The separate database adds configuration and a lightweight snapshot read before queries, but index corruption, reset, and tokenizer changes cannot endanger canonical logs. Durable revisions avoid full-log reads and rewrites for unchanged sessions; TEMP live overlays preserve current-session truth without making uncheckpointed events durable.
|
||||
|
||||
The chosen tokenizer supports short tokens with a smaller index but does not promise substring recall. Literal phrases make query syntax safe and predictable at the cost of excluding boolean/full MATCH expressions. Cancellation is effective while queued or awaiting sources, but synchronous SQLite execution remains a non-preemptible section.
|
||||
|
||||
Unit coverage pins extraction, filters, both search scopes, all default surfaces, metadata-before-ranking, snippets, literal escaping, deterministic ties, complete pagination, scoped cursor invalidation, dynamic persistence mount/unmount, restart reconciliation, live shadow/reveal/reopen, schema safety, rollback retry, and queued/in-flight source-wait cancellation. A keyless real-Loader-path test combines the package with the real SQLite persistence backend.
|
||||
@@ -1,51 +0,0 @@
|
||||
# Agent Note: SQLite FTS5 session search
|
||||
|
||||
Status: proposed
|
||||
|
||||
## Problem
|
||||
|
||||
The exact-read `ctx.sessionQuery` service deliberately has no derived index. Large persisted histories need full-text search without scanning every event on every query, while current live sessions need an overlay newer than the last durability checkpoint. Search also needs concrete ranking, snippets, pagination, cancellation, and rebuild behavior.
|
||||
|
||||
Splitting those concerns across a speculative provider coordinator and a database implementation would create two coupled reconciliation state machines. The first real implementation should own the source observation, extraction, SQLite transaction, generation, and query as one lifecycle.
|
||||
|
||||
## Proposal
|
||||
|
||||
Add `@deepseek-ai/dsh-session-query-sqlite` beside the exact-read package. The package will expose a search service or extend the family with the smallest API required by its actual consumers; phase one does not pre-commit a provider-registration protocol. It will depend on `ctx.sessions` and optional `ctx.sessionPersistence`, own a separate derived SQLite database, and reuse the canonical `foldSurface()` classification.
|
||||
|
||||
The implementation owns one serialized reconciliation/DB transaction state machine. A transaction observes authoritative persisted metadata and live snapshots, extracts semantic documents, updates derived tables, advances relevant cursor generations, and executes or enables the corresponding query. No second service maintains parallel fingerprints, dirty flags, live-id sets, or invalidation generations.
|
||||
|
||||
Persisted documents survive restarts. Live overrides are connection-local and shadow the persisted rows for the same session, then disappear when the live owner or database closes. The derived database remains separate from canonical persistence so index reset, corruption, tokenizer changes, and schema churn cannot endanger durable conversation logs.
|
||||
|
||||
## Search semantics to decide with implementation
|
||||
|
||||
The implementation must define both cross-session and within-session scopes from executable use cases. Each searchable event is one document with session metadata, event metadata, surface classification, normalized semantic text, and a bounded plain-text snippet. Session results group by their strongest matching event; numeric backend scores remain private.
|
||||
|
||||
Search returns content-bearing result records rather than metadata-only headers. Chainable filters operate on that exact result shape and are designed and implemented with the search API instead of becoming a provider-specific pre-ranking contract. Query syntax is treated as data. Ordering includes stable tie fields. Opaque cursors bind to normalized request shape and the smallest relevant generation; unrelated session changes should not invalidate a within-session cursor. Cancellation must stop caller waiting and interrupt SQLite work where the runtime permits.
|
||||
|
||||
Tokenizer choice remains an implementation experiment. FTS5 trigram supports substring recall but rejects useful terms shorter than three characters and increases index size; the proposal must benchmark that tradeoff against the default Unicode tokenizer before making it contract.
|
||||
|
||||
## Extraction and reconciliation
|
||||
|
||||
The package starts with first-party semantic extraction for messages, reasoning, tool calls/results, blocked prompts, context, steering, todos, and error/status detail. Structural events and stream chunks contribute no document. Unknown declaration-merged event/content types remain non-searchable unless a real extension consumer demonstrates the need for a public extractor registry.
|
||||
|
||||
Reconciliation may use stable fingerprints to avoid rewriting unchanged persisted sessions, but the database package owns their calculation and storage. It must never report a row current when source observation or extraction failed. Provider-schema mismatch may reset only the derived database; ordinary source changes use transactional upsert/delete. Mounted but unreadable persistence fails affected searches without affecting canonical writes or known live exact reads.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Add FTS tables to the canonical persistence database** — rejected because a rebuildable index must not share the authoritative log's schema/reset/failure boundary.
|
||||
- **Reintroduce phase-one provider coordination** — rejected because there is one planned implementation and no evidence for a stable multi-provider seam.
|
||||
- **Persist live overrides immediately** — rejected because live events are not canonical until the existing checkpoint commits.
|
||||
- **Return BM25 scores** — rejected because provider-specific numeric scales are unstable across corpus changes.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Restart tests cover unchanged, new, changed, and deleted persisted sessions without rebuilding the whole index.
|
||||
- Reopening preserves persisted rows and removes live rows; live rows shadow and then reveal their persisted base.
|
||||
- Tests cover both search scopes, content-bearing results, chainable result filters, surface defaults, snippets, escaping, deterministic ties, pagination, scoped stale cursors, cancellation, dynamic persistence mount/unmount, and recovery after a failed transaction.
|
||||
- A schema mismatch resets only the derived database.
|
||||
- A keyless end-to-end test combines a real persistence backend with the real SQLite search package.
|
||||
- The Agent Note is amended to the measured tokenizer and public API actually implemented before moving to `implemented/`.
|
||||
|
||||
## Risks
|
||||
|
||||
A single owner is simpler but initially less reusable than a provider-neutral seam. That is intentional: a second real backend can reveal what to extract. SQLite runtime differences can affect FTS ranking and snippets, so tests must pin only contract-controlled ordering and presentation. The separate database adds configuration and lifecycle work, but preserves the canonical store's safety boundary.
|
||||
Reference in New Issue
Block a user