Merge remote-tracking branch 'origin/docs/i18n-batch-cds-postmortem' into worktree/pr345-retarget-stack
# Conflicts: # .agents/notes/proposed/feature/2026-07-10-sqlite-session-query-provider.md
This commit is contained in:
Tianyi Cui
@@ -1,6 +0,0 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-10-sqlite-session-query-provider.md: 828d73938e8b9de1a69d021cb10f86aa7b5cd576
|
||||
2026-07-10-sqlite-session-query-provider.zh.md: 2cb32238d388e3a1751451fe0bb2036dbac57809
|
||||
@@ -1,53 +0,0 @@
|
||||
# Agent Note: SQLite FTS5 session search
|
||||
|
||||
Status: proposed
|
||||
|
||||
English | [中文](2026-07-10-sqlite-session-query-provider.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The exact-read `ctx.sessionQuery` service deliberately has no derived index. Large persisted histories need full-text search without scanning every event on every query, while current live sessions need an overlay newer than the last durability checkpoint. Search also needs concrete ranking, snippets, pagination, cancellation, and rebuild behavior.
|
||||
|
||||
Splitting those concerns across a speculative provider coordinator and a database implementation would create two coupled reconciliation state machines. The first real implementation should own the source observation, extraction, SQLite transaction, generation, and query as one lifecycle.
|
||||
|
||||
## Proposal
|
||||
|
||||
Add `@deepseek-ai/dsh-session-query-sqlite` beside the exact-read package. The package will expose a search service or extend the family with the smallest API required by its actual consumers; phase one does not pre-commit a provider-registration protocol. It will depend on `ctx.sessions` and optional `ctx.sessionPersistence`, own a separate derived SQLite database, and reuse the canonical `foldSurface()` classification.
|
||||
|
||||
The implementation owns one serialized reconciliation/DB transaction state machine. A transaction observes authoritative persisted metadata and live snapshots, extracts semantic documents, updates derived tables, advances relevant cursor generations, and executes or enables the corresponding query. No second service maintains parallel fingerprints, dirty flags, live-id sets, or invalidation generations.
|
||||
|
||||
Persisted documents survive restarts. Live overrides are connection-local and shadow the persisted rows for the same session, then disappear when the live owner or database closes. The derived database remains separate from canonical persistence so index reset, corruption, tokenizer changes, and schema churn cannot endanger durable conversation logs.
|
||||
|
||||
## Search semantics to decide with implementation
|
||||
|
||||
The implementation must define both cross-session and within-session scopes from executable use cases. Each searchable event is one document with session metadata, event metadata, surface classification, normalized semantic text, and a bounded plain-text snippet. Session results group by their strongest matching event; numeric backend scores remain private.
|
||||
|
||||
Search returns content-bearing result records rather than metadata-only headers. Chainable filters operate on that exact result shape and are designed and implemented with the search API instead of becoming a provider-specific pre-ranking contract. Query syntax is treated as data. Ordering includes stable tie fields. Opaque cursors bind to normalized request shape and the smallest relevant generation; unrelated session changes should not invalidate a within-session cursor. Cancellation must stop caller waiting and interrupt SQLite work where the runtime permits.
|
||||
|
||||
Tokenizer choice remains an implementation experiment. FTS5 trigram supports substring recall but rejects useful terms shorter than three characters and increases index size; the proposal must benchmark that tradeoff against the default Unicode tokenizer before making it contract.
|
||||
|
||||
## Extraction and reconciliation
|
||||
|
||||
The package starts with first-party semantic extraction for messages, reasoning, tool calls/results, blocked prompts, context, steering, todos, and error/status detail. Structural events and stream chunks contribute no document. Unknown declaration-merged event/content types remain non-searchable unless a real extension consumer demonstrates the need for a public extractor registry.
|
||||
|
||||
Reconciliation may use stable fingerprints to avoid rewriting unchanged persisted sessions, but the database package owns their calculation and storage. It must never report a row current when source observation or extraction failed. Provider-schema mismatch may reset only the derived database; ordinary source changes use transactional upsert/delete. Mounted but unreadable persistence fails affected searches without affecting canonical writes or known live exact reads.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Add FTS tables to the canonical persistence database** — rejected because a rebuildable index must not share the authoritative log's schema/reset/failure boundary.
|
||||
- **Reintroduce phase-one provider coordination** — rejected because there is one planned implementation and no evidence for a stable multi-provider seam.
|
||||
- **Persist live overrides immediately** — rejected because live events are not canonical until the existing checkpoint commits.
|
||||
- **Return BM25 scores** — rejected because provider-specific numeric scales are unstable across corpus changes.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Restart tests cover unchanged, new, changed, and deleted persisted sessions without rebuilding the whole index.
|
||||
- Reopening preserves persisted rows and removes live rows; live rows shadow and then reveal their persisted base.
|
||||
- Tests cover both search scopes, content-bearing results, chainable result filters, surface defaults, snippets, escaping, deterministic ties, pagination, scoped stale cursors, cancellation, dynamic persistence mount/unmount, and recovery after a failed transaction.
|
||||
- A schema mismatch resets only the derived database.
|
||||
- A keyless end-to-end test combines a real persistence backend with the real SQLite search package.
|
||||
- The Agent Note is amended to the measured tokenizer and public API actually implemented before moving to `implemented/`.
|
||||
|
||||
## Risks
|
||||
|
||||
A single owner is simpler but initially less reusable than a provider-neutral seam. That is intentional: a second real backend can reveal what to extract. SQLite runtime differences can affect FTS ranking and snippets, so tests must pin only contract-controlled ordering and presentation. The separate database adds configuration and lifecycle work, but preserves the canonical store's safety boundary.
|
||||
@@ -1,53 +0,0 @@
|
||||
# Agent Note: SQLite FTS5 会话搜索
|
||||
|
||||
Status: proposed
|
||||
|
||||
[English](2026-07-10-sqlite-session-query-provider.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
精确读取的 `ctx.sessionQuery` 服务有意不维护派生索引。大规模持久化的历史记录需要全文搜索，而不是每次查询都扫描全部事件；当前的活跃会话则需要一个比上一次持久性检查点更新的覆盖层。搜索还需要具体的排序、摘要片段、分页、取消以及重建行为。
|
||||
|
||||
如果把这些关注点拆分到一个推测性的 provider 协调器和一个数据库实现之间，会产生两个耦合的协调状态机。第一个真实实现应当将源观察、提取、SQLite 事务、generation 管理和查询作为一个完整的生命周期来拥有。
|
||||
|
||||
## 提案
|
||||
|
||||
在精确读取包（exact-read package）旁新增 `@deepseek-ai/dsh-session-query-sqlite`。该包将暴露一个搜索服务，或以其实际消费方所需的最小 API 扩展服务族；第一阶段不预先承诺 provider 注册协议。它将依赖 `ctx.sessions` 和可选的 `ctx.sessionPersistence`，拥有一个独立的派生 SQLite 数据库，并复用规范的 `foldSurface()` 分类。
|
||||
|
||||
实现拥有一个串行化的协调/数据库事务状态机。一次事务观察权威的持久化元数据和活跃快照，提取语义文档，更新派生表，推进相关的游标 generation，并执行或启用对应的查询。没有第二个服务维护并行的指纹、脏标记、活跃 ID 集合或失效 generation。
|
||||
|
||||
持久化文档在重启后存活。活跃覆盖层是连接本地的，对同一会话的持久化行进行遮蔽，在活跃所有者或数据库关闭时消失。派生数据库与规范持久化分离，确保索引重置、损坏、分词器变更和 schema 变动不会危及持久化的对话日志。
|
||||
|
||||
## 随实现确定的搜索语义
|
||||
|
||||
实现必须从可执行的用例出发定义跨会话和会话内两种搜索范围。每个可搜索事件是一个文档，包含会话元数据、事件元数据、surface 分类、归一化的语义文本和有界的纯文本摘要片段。会话级结果按其最强匹配事件分组；数值化的后端分数保持私有。
|
||||
|
||||
搜索返回承载内容的结果记录，而非仅含元数据的 header。可链式过滤器作用于这一精确结果形状，并与搜索 API 一同设计和实现，而不会变成 provider 特有的预排序契约。查询语法被视为数据。排序包含稳定的平局字段。不透明游标绑定到归一化的请求形状和最小相关 generation；不相关的会话变更不应使会话内游标失效。取消操作必须停止调用方等待，并在运行时允许的范围内中断 SQLite 工作。
|
||||
|
||||
分词器选择仍是实现层面的实验。FTS5 trigram 支持子串召回，但会拒绝短于三个字符的有用词项并增大索引体积；提案在将其写入契约之前，必须对该权衡与默认 Unicode 分词器进行基准测试。
|
||||
|
||||
## 提取与协调
|
||||
|
||||
该包首先为以下内容提供第一方语义提取：消息、reasoning、工具调用/结果、被阻止的提示词、上下文、steering（中途引导）、待办事项和错误/状态详情。结构性事件和流式分片不贡献文档。未知的声明合并事件/内容类型保持不可搜索，除非有真实的扩展消费方证明需要公开的提取器注册表。
|
||||
|
||||
协调可以使用稳定指纹来避免重写未变更的持久化会话，但数据库包拥有指纹的计算和存储。当源观察或提取失败时，它绝不能报告某行为最新。provider-schema 不匹配只重置派生数据库；普通的源变更使用事务性 upsert/delete。已挂载但不可读的持久化层使受影响的搜索失败，但不影响规范写入或已知的活跃精确读取。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
- **将 FTS 表添加到规范持久化数据库中**：否决，因为可重建的索引不应与权威日志共享 schema/重置/故障边界。
|
||||
- **重新引入第一阶段的 provider 协调**：否决，因为只有一个计划中的实现，且没有证据表明存在稳定的多 provider seam。
|
||||
- **立即持久化活跃覆盖层**：否决，因为活跃事件在现有检查点提交之前不是规范的。
|
||||
- **返回 BM25 分数**：否决，因为 provider 特定的数值尺度在语料变化时不稳定。
|
||||
|
||||
## 验收标准
|
||||
|
||||
- 重启测试覆盖未变更、新增、变更和删除的持久化会话，且不重建整个索引。
|
||||
- 重新打开时保留持久化行并移除活跃行；活跃行先遮蔽、后显露其持久化基础。
|
||||
- 测试覆盖两种搜索范围、承载内容的结果、可链式结果过滤器、surface 默认值、摘要片段、转义、确定性平局、分页、范围内的陈旧游标、取消、动态持久化挂载/卸载，以及事务失败后的恢复。
|
||||
- schema 不匹配只重置派生数据库。
|
||||
- 一个 keyless 的端到端测试将真实的持久化后端与真实的 SQLite 搜索包组合使用。
|
||||
- 在移至 `implemented/` 之前，本 Agent Note 须修订为实际实现的分词器和公开 API。
|
||||
|
||||
## 风险
|
||||
|
||||
单一所有者比提供方无关的 seam 更简单，但初期可复用性较低。这是有意为之：第二个真实后端可以揭示应当抽取什么。SQLite 运行时差异可能影响 FTS 排序和摘要片段，因此测试只能固定契约控制的排序和呈现。独立数据库增加了配置和生命周期工作，但保全了规范存储的安全边界。
|
||||
Reference in New Issue
Block a user