Merge remote-tracking branch 'origin/master' into session-surface
Reconcile the session-surface feature with master's package reorg and simplifications: - Adopt master's folded usage (assistant/message.usage; standalone `usage` event dropped) and re-attach surface metadata (surfaceOp/sourceEventSeqs). - Add surface opts to master's new max-tokens assistant/message append. - Port surface columns onto the coordinator-refactored SQLite backend at its new path; drop the dead v1->v2 migration (bump-and-reject, no migration per pre-release policy). - Move the session-surface RFC into implemented/architecture/ and refresh its stale body (no migration, SESSION_FORMAT_VERSION=0, renamed package paths). - Update the core-data-structures catalog SessionEvent blocks for the two new surface fields; regenerate the cordis catalog. - Re-harvest ACP snapshot fixtures (keyless replay) to carry surface metadata.
This commit is contained in:
Hypatia May
@@ -1,33 +1,11 @@
|
||||
# @deepseek-ai/dsh-session-persistence
|
||||
# session-persistence/ — persistence capability family
|
||||
|
||||
The abstract durable session-persistence seam (`ctx.sessionPersistence`). Defines WHAT a persistence backend does — durably store, reload, list, and update sessions — without saying HOW. Mirrors the `dsh-bash` capability-seam template ([capability seams](../../docs/rfc/implemented/2026-06-13-capability-seams.md)): an abstract service here, a concrete implementation in a sibling package, consumers that inject the interface.
|
||||
The durable session-persistence seam and its storage backends. The interface package owns the abstract `SessionPersistence` service and the shared write coordinator; the backends are concrete implementations that register on `ctx.sessionPersistence`. All **product** packages.
|
||||
|
||||
The persisted unit IS the existing `SessionEvent` (event-sourced model — the log is the single source of truth), so there is no parallel "persisted message" type. Metadata that is NOT replayable conversation state (format version, cwd, lineage) travels separately as `SessionMeta`, owned by `dsh-session` and re-exported here.
|
||||
| Package | Role | ctx key |
|
||||
|---|---|---|
|
||||
| `session-persistence/` | Persistence seam + shared write coordinator | `ctx.sessionPersistence` |
|
||||
| `session-persistence-jsonl/` | JSONL-sidecar persistence backend | (registers `ctx.sessionPersistence`) |
|
||||
| `session-persistence-sqlite/` | SQLite persistence backend | (registers `ctx.sessionPersistence`) |
|
||||
|
||||
## Service API (`ctx.sessionPersistence`)
|
||||
|
||||
| Method | Contract |
|
||||
|---|---|
|
||||
| `create(meta): Promise<void>` | Register a new session's metadata. MAY defer the physical write until the first `append` (lazy materialization). |
|
||||
| `append(id, events): Promise<void>` | Durably persist a batch (from the `session/flush` drain). Append-only; first event `seq` == stored next-seq after any repair; rejects non-JSON-serializable data naming the offending type. |
|
||||
| `load(id): Promise<{ meta; events }>` | Reload meta + log. Preserves an interrupted (unclosed) final turn and closes it with synthetic closers — an error `tool/result` per unanswered `tool-call`, then `step/end?`+`turn/end {interrupted}` (a turn can be huge — never truncated); only a torn tail fragment is dropped. Events contiguous (`events[i].seq === i`); rejects a committed-region gap/parse error or unknown `version`. |
|
||||
| `list(): Promise<SessionMeta[]>` | Lightweight listing from metadata, no full-log parse. |
|
||||
| `has(id)` / `delete(id)` | Existence / removal. A zero-event lazily-materialized session is absent from `has`/`list`. |
|
||||
| `update(id, summary): Promise<void>` | Update mutable `SessionSummary` fields without touching the append-only log. |
|
||||
|
||||
## Invariants every backend must honor
|
||||
|
||||
- **Append-only; a crashed turn is closed, not truncated.** Committed events (at or below a flushed `turn/end`) are never rewritten. A crash can leave an unclosed final turn whose events are real and possibly large; `load` preserves them and durably appends synthetic closers (an error `tool/result` per unanswered `tool-call`, then `step/end?`+`turn/end {interrupted}`) to balance the log and keep the rehydrated history a valid provider transcript. Only a never-fully-written torn tail fragment is discarded.
|
||||
- **Contiguous seq.** `load` rejects a `seq` gap/parse error in the MIDDLE of the log; `append`'s first `seq` must equal the stored next-seq.
|
||||
- **JSON-serializable data.** `append` rejects non-serializable `event.data`; backends snapshot each event when buffering (the live `session.events` object is mutable).
|
||||
- **Durability.** `append` returns only once the batch is durable.
|
||||
|
||||
## Testing backends
|
||||
|
||||
Import `runPersistenceContract` from `tests/contract.ts` and call it with a factory that yields a fresh, empty backend plus a teardown. Every backend is held to the same append-only / contiguous-seq / lazy-materialization / serializability semantics; a backend's own spec adds implementation-specific tests (crash repair, path sanitization) on top.
|
||||
|
||||
Two backends run this suite: `dsh-session-persistence-jsonl` (append-only file log) and `dsh-session-persistence-sqlite` (`node:sqlite`, each `SessionEvent` one row `(session_id, seq, type, time, data)`). Both passing the same contract is the proof that the seam is genuinely backend-agnostic — lazy materialization, crash-tail-on-load, and contiguous-seq hold identically over file bytes and over a transactional store.
|
||||
|
||||
## Metadata types
|
||||
|
||||
Re-exported from `dsh-session`: `SessionHeader` (immutable: `version`, `id`, `createdAt`, `cwd?`, `parentSession?`), `SessionSummary` (mutable: `updatedAt`, `title?`, `firstPrompt?`), `SessionMeta` (their intersection).
|
||||
The interface lives at `session-persistence/session-persistence/`; backends are flat siblings. A new storage backend joins here and registers on `ctx.sessionPersistence`. See [session persistence](../../docs/rfc/implemented/architecture/2026-06-14-session-persistence.md).
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# @deepseek-ai/dsh-session-persistence-jsonl
|
||||
|
||||
The JSONL durable session-persistence backend — a concrete `SessionPersistence` (the `dsh-session-persistence` seam). One append-only `.jsonl` event log per session.
|
||||
|
||||
## On-disk layout
|
||||
|
||||
```
|
||||
<root>/
|
||||
cwd-<sha256(cwd)[:12]>/ # per-project bucket (or _no-cwd/ when no cwd)
|
||||
<encoded-id>.jsonl # header line + one SessionEvent per line (verbatim)
|
||||
```
|
||||
|
||||
- The first `.jsonl` line is the immutable `SessionHeader` tagged `{ type: 'session', version, id, cwd?, createdAt, parentSession? }`; every subsequent line is one `SessionEvent` JSON, **verbatim including `assistant/chunk`** so `seq` stays contiguous (`events[i].seq === i`).
|
||||
- Session ids are unvalidated branded strings, so they are percent-encoded to a single safe path segment before use (no traversal, no collision).
|
||||
|
||||
## Config
|
||||
|
||||
| Key | Type | Notes |
|
||||
|---|---|---|
|
||||
| `root` | `string` (required) | Root directory for all session files. **No default** — a `process.cwd()` default would scatter files as the process's cwd changes (bash calls, subprocesses). |
|
||||
|
||||
## Durability and crash semantics
|
||||
|
||||
- **Lazy materialization.** `create(meta)` writes nothing; the `.jsonl` (header + first batch) is written atomically (temp-write + `fsync` + rename) on the first `append`. A created-but-never-appended session leaves nothing on disk and is absent from `list`.
|
||||
- **Append-only.** Committed events (at or below a flushed `turn/end`) are never rewritten. Subsequent appends are line appends at EOF + `fsync`.
|
||||
- **Crash recovery — close, don't truncate.** A crash can leave a log whose final turn never closed (real events after the last `turn/end`). `load` PRESERVES those events (a turn can be huge — they are real work) and closes the orphaned turn by durably appending synthetic boundary events: an error `tool/result` for every `tool-call` the crash left unanswered (the loop logs the assistant message before running the tools, so a mid-tool crash leaves dangling calls — and `deriveMessages()` would replay an assistant tool-call with no result, which providers reject), then a `step/end` if a step was open, then `turn/end {kind:'interrupted'}`, returning a balanced log. Only a never-fully-written **torn tail fragment** (a final line with no newline / unparseable) is `ftruncate`d away before the closers are written. See [session persistence](../../../docs/rfc/implemented/architecture/2026-06-14-session-persistence.md).
|
||||
- **Contiguous-seq.** `load` rejects a mid-log parse error or `seq` gap (unloadable); `append` rejects a batch whose first `seq` does not continue the stored log, and rejects non-JSON-serializable `event.data` naming the offending event type.
|
||||
- **Format version.** Only the current `SESSION_FORMAT_VERSION` (v0) is supported; `load` rejects any other version. While the harness is unreleased the on-disk format is pre-release/unstable: a breaking format change is absorbed at v0 (no bump until the first tagged release) and non-current logs are rejected — there is no migration (no persisted user data to preserve).
|
||||
|
||||
## Write path
|
||||
|
||||
The plugin generalizes the example `session-jsonl.ts`: it subscribes to `session/created` (capture the header; persist a fork's seed once), `session/event` (snapshot each event when buffering — the live `session.events` object is mutable), and `session/flush`/dispose (drain the write-behind buffer through `append`). A per-session write cursor means a resumed session never re-appends already-stored events. Existing live sessions are seeded on plugin apply (HMR does not replay `session/created`). All backend operations for one session are serialized, and disposal awaits quiescence (every init + final drain) before returning, so no write lands after teardown.
|
||||
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-session-persistence-jsonl",
|
||||
"description": "JSONL durable session persistence backend for the DeepSeek Harness",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib",
|
||||
"src"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-session": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session-persistence": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
},
|
||||
"dependencies": {
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-persistence": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,249 @@
|
||||
/**
|
||||
* On-disk format helpers for the JSONL session-persistence backend: path
|
||||
* sanitization (a {@link SessionId} is an unvalidated branded string, so it
|
||||
* MUST be encoded before use in a path — no traversal, no collision), the
|
||||
* per-cwd directory layout, header-line (de)serialization, and the
|
||||
* truncation-repair offset computation.
|
||||
*
|
||||
* @module dsh-session-persistence-jsonl/format
|
||||
*/
|
||||
|
||||
import { createHash } from 'node:crypto'
|
||||
import { join } from 'node:path'
|
||||
import type { SessionEvent, SessionHeader, SessionId } from '@deepseek-ai/dsh-session'
|
||||
|
||||
/**
|
||||
* The first line of a session's `.jsonl` file: the immutable
|
||||
* {@link SessionHeader} tagged as a `session` record so a reader can tell it
|
||||
* apart from an event line.
|
||||
*/
|
||||
export interface HeaderLine {
|
||||
type: 'session'
|
||||
version: number
|
||||
id: SessionId
|
||||
createdAt: number
|
||||
cwd?: string
|
||||
parentSession?: SessionId
|
||||
}
|
||||
|
||||
/** Build the header line object from a {@link SessionHeader}. */
|
||||
export function toHeaderLine(header: SessionHeader): HeaderLine {
|
||||
return {
|
||||
type: 'session',
|
||||
version: header.version,
|
||||
id: header.id,
|
||||
createdAt: header.createdAt,
|
||||
...header.cwd !== undefined ? { cwd: header.cwd } : {},
|
||||
...header.parentSession !== undefined ? { parentSession: header.parentSession } : {},
|
||||
}
|
||||
}
|
||||
|
||||
/** Parse a header line back into a {@link SessionHeader}. */
|
||||
export function fromHeaderLine(line: HeaderLine): SessionHeader {
|
||||
return {
|
||||
version: line.version,
|
||||
id: line.id,
|
||||
createdAt: line.createdAt,
|
||||
...line.cwd !== undefined ? { cwd: line.cwd } : {},
|
||||
...line.parentSession !== undefined ? { parentSession: line.parentSession } : {},
|
||||
}
|
||||
}
|
||||
|
||||
/** Type guard: a parsed first line is a well-formed session header. */
|
||||
function isHeaderLine(value: unknown): value is HeaderLine {
|
||||
return (
|
||||
typeof value === 'object' && value !== null
|
||||
&& (value as { type?: unknown }).type === 'session'
|
||||
&& typeof (value as { version?: unknown }).version === 'number'
|
||||
&& typeof (value as { id?: unknown }).id === 'string'
|
||||
&& typeof (value as { createdAt?: unknown }).createdAt === 'number'
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* Encode an arbitrary string as a single safe path segment, injectively over
|
||||
* ALL JS (UTF-16) strings — including lone surrogates. A {@link SessionId} is
|
||||
* an unvalidated branded string, so this neutralizes `../`, absolute paths,
|
||||
* NUL, and separators before any filesystem use.
|
||||
*
|
||||
* Each UTF-16 code unit is either kept literal (the safe set `[A-Za-z0-9_-]`)
|
||||
* or escaped as `~XXXX` (its 4-hex-digit code unit). `~` is itself escaped, so
|
||||
* the mapping is injective and reversible: distinct inputs never collide. We
|
||||
* iterate code UNITS (`charCodeAt`), not code points, so a lone surrogate
|
||||
* escapes to a distinct `~XXXX` instead of being normalized to U+FFFD (which
|
||||
* `Buffer.from(…, 'utf8')` would do, breaking injectivity). `.` is in the safe
|
||||
* set for readability but the whole-segment tokens `.`/`..` are escaped so they
|
||||
* can never traverse.
|
||||
*/
|
||||
export function encodeSegment(raw: string): string {
|
||||
if (raw.length === 0) throw new Error('cannot encode an empty path segment')
|
||||
if (raw === '.') return '~002E'
|
||||
if (raw === '..') return '~002E~002E'
|
||||
let out = ''
|
||||
for (let i = 0; i < raw.length; i++) {
|
||||
const code = raw.charCodeAt(i)
|
||||
const ch = String.fromCharCode(code)
|
||||
if (ch !== '~' && /^[A-Za-z0-9._-]$/.test(ch)) {
|
||||
out += ch
|
||||
} else {
|
||||
out += '~' + code.toString(16).toUpperCase().padStart(4, '0')
|
||||
}
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
/**
|
||||
* The directory a session's files live in: the configured root, then a per-cwd
|
||||
* subdirectory so sessions group by project. The cwd subdir is a stable hash
|
||||
* (short, collision-resistant, filesystem-safe) plus an encoded suffix for
|
||||
* readability; sessions without a cwd go in a shared `_no-cwd` bucket.
|
||||
*/
|
||||
export function sessionDir(root: string, cwd: string | undefined): string {
|
||||
if (cwd === undefined) return join(root, '_no-cwd')
|
||||
const hash = createHash('sha256').update(cwd).digest('hex').slice(0, 12)
|
||||
return join(root, `cwd-${hash}`)
|
||||
}
|
||||
|
||||
/** The append-only event-log file path for a session. */
|
||||
export function logPath(root: string, cwd: string | undefined, id: SessionId): string {
|
||||
return join(sessionDir(root, cwd), `${encodeSegment(id)}.jsonl`)
|
||||
}
|
||||
|
||||
/** Serialize one event as a JSONL line (no trailing newline). */
|
||||
export function eventLine(event: SessionEvent): string {
|
||||
return JSON.stringify(event)
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse a JSONL log buffer into its preserved event prefix (the header is line
|
||||
* 0). Returns the longest prefix of complete, seq-contiguous events plus the
|
||||
* byte offset of the end of the last preserved line (`committedBytes`).
|
||||
*
|
||||
* A crash can leave a durable log whose final turn never closed: real,
|
||||
* fully-written events sit after the last `turn/end`. Those are PRESERVED (a
|
||||
* single turn can be huge in a long-horizon task — truncating it would destroy
|
||||
* real work); the backend closes the orphaned open turn with a synthetic
|
||||
* `turn/end {kind:'interrupted'}` on reload (the session-persistence RFC). Only a TORN trailing
|
||||
* fragment — a final line never fully flushed (no newline, unparseable, or a
|
||||
* seq gap) — is excluded; it bounds the preserved region. A parse error or seq
|
||||
* gap AT OR BEFORE the last committed `turn/end` is committed-data corruption
|
||||
* and makes the session unloadable (throws).
|
||||
*
|
||||
* This relies on the session-log invariant that every event lives inside a turn
|
||||
* (`Session.append` enforces it): only the final turn can be open, so the
|
||||
* preserved tail is at most one unclosed turn.
|
||||
*/
|
||||
export function scanLog(buffer: Buffer): { meta: SessionHeader; events: SessionEvent[]; committedBytes: number } {
|
||||
const text = buffer.toString('utf8')
|
||||
// Split into complete (newline-terminated) lines, tracking the byte offset of
|
||||
// each line's end so the truncation point is exact (multi-byte chars make the
|
||||
// char offset differ from the byte offset). A trailing line with no newline is
|
||||
// an uncommitted crash fragment and is ignored — it is below the last
|
||||
// turn/end by construction (the loop only flushes whole lines).
|
||||
//
|
||||
// Track the byte offset with a RUNNING accumulator (`endByte`), adding each
|
||||
// line's byte length as we go. Recomputing `Buffer.byteLength(text.slice(0, i))`
|
||||
// per newline would rescan the whole prefix every time — O(n²) over a long
|
||||
// log (one assistant/chunk line per token makes that pathological).
|
||||
const lines: { text: string; endByte: number }[] = []
|
||||
let start = 0
|
||||
let byteOffset = 0
|
||||
for (let i = 0; i < text.length; i++) {
|
||||
if (text[i] === '\n') {
|
||||
const lineText = text.slice(start, i)
|
||||
byteOffset += Buffer.byteLength(lineText, 'utf8') + 1 // +1 for the '\n' (a 1-byte char)
|
||||
lines.push({ text: lineText, endByte: byteOffset })
|
||||
start = i + 1
|
||||
}
|
||||
}
|
||||
|
||||
const [headerEntry, ...eventEntries] = lines
|
||||
if (headerEntry === undefined) throw new Error('empty or header-less session log')
|
||||
|
||||
// Line 0 is the header.
|
||||
let parsedHeader: unknown
|
||||
try {
|
||||
parsedHeader = JSON.parse(headerEntry.text)
|
||||
} catch {
|
||||
throw new Error('corrupt session log: header line is not valid JSON')
|
||||
}
|
||||
if (!isHeaderLine(parsedHeader)) {
|
||||
throw new Error('corrupt session log: first line is not a session header')
|
||||
}
|
||||
const headerLine = parsedHeader
|
||||
|
||||
// Find the committed region: the prefix up to and including the LAST complete
|
||||
// `turn/end` in the WHOLE log. Two passes so a crash tail after the last
|
||||
// turn/end is tolerated, but corruption/gaps AT OR BEFORE the last committed
|
||||
// turn/end make the log unloadable (committed data must never be silently
|
||||
// dropped).
|
||||
//
|
||||
// Pass 1: parse every line that parses, recording (parsedOk, seq, isTurnEnd,
|
||||
// endByte) per line index. Lines that fail to parse are holes.
|
||||
interface Parsed { ok: boolean; event?: SessionEvent; endByte: number }
|
||||
const parsed: Parsed[] = eventEntries.map((entry) => {
|
||||
try {
|
||||
return { ok: true, event: JSON.parse(entry.text) as SessionEvent, endByte: entry.endByte }
|
||||
} catch {
|
||||
return { ok: false, endByte: entry.endByte }
|
||||
}
|
||||
})
|
||||
|
||||
// The last index (into eventEntries) that is a valid `turn/end` — the last
|
||||
// fully-committed boundary (the loop flushes only at turn/end).
|
||||
let lastTurnEnd = -1
|
||||
for (let i = parsed.length - 1; i >= 0; i--) {
|
||||
const p = parsed[i]
|
||||
if (p?.ok && p.event?.type === 'turn/end') { lastTurnEnd = i; break }
|
||||
}
|
||||
|
||||
// Walk the longest PREFIX of complete, seq-contiguous, parseable event lines
|
||||
// (line i is a parsed event with seq === i). This is the preservable region:
|
||||
// it includes any fully-written events of an interrupted final turn AFTER the
|
||||
// last turn/end — those are real, durably-written work and must NOT be
|
||||
// truncated (a single turn can be huge in a long-horizon task; the orphaned
|
||||
// open turn is closed with a synthetic turn/end on reload, not discarded —
|
||||
// the session-persistence RFC). The walk stops at the first hole (unparseable line or seq gap):
|
||||
// - if that hole is AT OR BEFORE the last committed turn/end, committed data
|
||||
// was damaged → the session is unloadable (throw);
|
||||
// - if it is AFTER (or there is no committed turn/end yet), it is the
|
||||
// tolerated crash boundary — a torn final line never fully flushed — and
|
||||
// it simply bounds the preserved tail.
|
||||
const preserved: SessionEvent[] = []
|
||||
for (let i = 0; i < parsed.length; i++) {
|
||||
const p = parsed[i]
|
||||
if (!p?.ok || p.event === undefined) {
|
||||
if (i <= lastTurnEnd) throw new Error(`corrupt session log: unparsable committed event at line ${i + 1}`)
|
||||
break // torn tail fragment after the last turn/end — stop, tolerate
|
||||
}
|
||||
if (p.event.seq !== i) {
|
||||
if (i <= lastTurnEnd) throw new Error(`corrupt session log: seq gap in committed region at line ${i + 1} (expected ${i}, got ${p.event.seq})`)
|
||||
break // gap after the last turn/end — torn tail, stop
|
||||
}
|
||||
preserved.push(p.event)
|
||||
}
|
||||
|
||||
// committedBytes = end of the last PRESERVED line (header if none): the next
|
||||
// append truncates any torn bytes past this point before writing the
|
||||
// synthetic closers + new events.
|
||||
const lastPreserved = parsed[preserved.length - 1]
|
||||
const committedBytes = preserved.length > 0 && lastPreserved ? lastPreserved.endByte : headerEntry.endByte
|
||||
return { meta: fromHeaderLine(headerLine), events: preserved, committedBytes }
|
||||
}
|
||||
|
||||
/**
|
||||
* Parse just the header line of a log into a {@link SessionHeader}, or
|
||||
* `undefined` if it is missing/not a header. Used by `list()` to read session
|
||||
* metadata WITHOUT parsing the whole log: a session picker scales with the
|
||||
* number of sessions, not the total size of every conversation.
|
||||
*/
|
||||
export function parseHeaderMeta(firstLine: string): SessionHeader | undefined {
|
||||
let parsed: unknown
|
||||
try {
|
||||
parsed = JSON.parse(firstLine)
|
||||
} catch {
|
||||
return undefined
|
||||
}
|
||||
if (!isHeaderLine(parsed)) return undefined
|
||||
return fromHeaderLine(parsed)
|
||||
}
|
||||
@@ -0,0 +1,381 @@
|
||||
/**
|
||||
* JSONL durable session-persistence backend (`@deepseek-ai/dsh-session-persistence-jsonl`).
|
||||
*
|
||||
* One append-only `.jsonl` event log per session (a header line then one
|
||||
* `SessionEvent` per line, verbatim including `assistant/chunk` so `seq` stays
|
||||
* contiguous), with lazy materialization (no file until the first `append`),
|
||||
* atomic first write, and load-time repair of a never-committed crash tail.
|
||||
*
|
||||
* The backend supplies ONLY the file-bytes storage primitives (the
|
||||
* {@link PersistenceBackend} hooks below); all the write-path orchestration
|
||||
* (the `session/event` → buffer → `session/flush` drain, per-session
|
||||
* serialization, write cursors, fork-seed persistence, HMR live-adoption,
|
||||
* crash-repair sequencing, dispose quiescence) lives in the backend-agnostic
|
||||
* {@link PersistenceCoordinator} this class composes. The four public
|
||||
* {@link SessionPersistence} methods delegate to the coordinator.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-session-persistence-jsonl
|
||||
*/
|
||||
|
||||
import { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import { open, mkdir, readFile, readdir, link, rm, truncate } from 'node:fs/promises'
|
||||
import { dirname, resolve } from 'node:path'
|
||||
import { randomBytes } from 'node:crypto'
|
||||
import {
|
||||
SessionPersistence, PersistenceCoordinator,
|
||||
type PersistenceBackend, type StoredPrefix,
|
||||
} from '@deepseek-ai/dsh-session-persistence'
|
||||
import type { Session, SessionEvent, SessionId, SessionHeader } from '@deepseek-ai/dsh-session'
|
||||
import {
|
||||
encodeSegment, eventLine, logPath, parseHeaderMeta, scanLog, sessionDir, toHeaderLine,
|
||||
} from './format.ts'
|
||||
|
||||
export interface Config {
|
||||
/**
|
||||
* Root directory for all session files. Required (no default): a default of
|
||||
* `process.cwd()` would scatter session files as the process's cwd changes
|
||||
* (bash calls, subprocesses). Sessions group under per-cwd subdirectories.
|
||||
*/
|
||||
root: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether `error` is a "no such file/directory" (`ENOENT`) failure — the ONLY
|
||||
* filesystem error that legitimately means "this session/root is absent" for a
|
||||
* durable backend. Any OTHER error (`EACCES`, `ENOTDIR`, transient I/O) must
|
||||
* surface rather than be silently reported as absence. (A NodeJS filesystem
|
||||
* rejection carries a string `code`.)
|
||||
*/
|
||||
function isENOENT(error: unknown): boolean {
|
||||
return (error as NodeJS.ErrnoException | null)?.code === 'ENOENT'
|
||||
}
|
||||
|
||||
/**
|
||||
* The JSONL persistence backend. Load as a plugin; it registers as
|
||||
* `ctx.sessionPersistence` and (via the coordinator) installs the write-path
|
||||
* listeners. Its torn-tail marker is the byte offset to truncate the log to.
|
||||
*/
|
||||
export class SessionPersistenceJsonl extends SessionPersistence implements PersistenceBackend<number> {
|
||||
static inject = ['sessions']
|
||||
|
||||
static Config: z<Config> = z.object({
|
||||
root: z.string().required(),
|
||||
})
|
||||
|
||||
/**
|
||||
* Backend label for the coordinator's dispose-failure AggregateError and
|
||||
* effect name. NOTE: this intentionally shadows cordis `Service.name` (which
|
||||
* the base sets to `'sessionPersistence'`). The service is registered under the
|
||||
* fixed key the Service constructor captured (`reflect.provide('sessionPersistence', …)`),
|
||||
* not via `this.name`, so overwriting the instance field with the backend label
|
||||
* does not affect `ctx.sessionPersistence` resolution — it only relabels the
|
||||
* dispose diagnostics, which is exactly what {@link PersistenceBackend.name} is for.
|
||||
*/
|
||||
override readonly name = 'session-persistence-jsonl'
|
||||
|
||||
private root: string
|
||||
private coordinator: PersistenceCoordinator<number>
|
||||
|
||||
constructor(ctx: Context, public config: Config) {
|
||||
super(ctx)
|
||||
// Resolve the configured root to an ABSOLUTE path ONCE, here. A relative root
|
||||
// would otherwise re-resolve against `process.cwd()` at every later
|
||||
// readdir/open — so if any plugin or test changed cwd between create, append,
|
||||
// and load, one session's files could split across directories.
|
||||
this.root = resolve(config.root)
|
||||
this.coordinator = new PersistenceCoordinator<number>(this.ctx, this)
|
||||
}
|
||||
|
||||
// --- SessionPersistence service surface (delegated to the coordinator) ---
|
||||
|
||||
create(meta: SessionHeader): Promise<void> {
|
||||
return this.coordinator.create(meta)
|
||||
}
|
||||
|
||||
append(id: SessionId, events: readonly SessionEvent[]): Promise<void> {
|
||||
return this.coordinator.append(id, events)
|
||||
}
|
||||
|
||||
load(id: SessionId): Promise<{ meta: SessionHeader; events: SessionEvent[] }> {
|
||||
return this.coordinator.load(id)
|
||||
}
|
||||
|
||||
// `list` is BOTH the public service method and the PersistenceBackend hook —
|
||||
// one method, the bucket walk below. The coordinator adds no orchestration for
|
||||
// listing (no per-id serialization, no cursor), so it would just call back into
|
||||
// this same method; routing it through the coordinator would recurse. Defined
|
||||
// once, in the "PersistenceBackend hooks" section.
|
||||
|
||||
/**
|
||||
* The per-session init promises, exposed for white-box tests that await a
|
||||
* specific session's onCreated (there is no public API to await one init).
|
||||
*/
|
||||
get inits(): Map<Session, Promise<void>> {
|
||||
return this.coordinator.inits
|
||||
}
|
||||
|
||||
// --- PersistenceBackend hooks (the file-bytes storage primitives) ---
|
||||
|
||||
/** Read a stored prefix by id across ALL cwd buckets (cwd unknown). */
|
||||
async loadStored(id: SessionId): Promise<StoredPrefix<number> | undefined> {
|
||||
const file = await this.findLog(id)
|
||||
if (file === undefined) return undefined
|
||||
return this.readPrefix(file.path)
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a stored prefix SCOPED to `cwd` (HMR live-adoption must not cross cwd).
|
||||
* `undefined` is the DEFINITE "no-cwd" bucket, NOT "unknown" — a live session
|
||||
* with no cwd may only adopt a persisted no-cwd log, never a same-id log that
|
||||
* lives in some other cwd bucket. So this looks at exactly `logPath(cwd)`
|
||||
* (which maps `undefined` → the `_no-cwd` bucket), never the all-buckets scan.
|
||||
*/
|
||||
async loadLive(id: SessionId, cwd: string | undefined): Promise<StoredPrefix<number> | undefined> {
|
||||
const path = logPath(this.root, cwd, id)
|
||||
if (!await this.exists(path)) return undefined
|
||||
return this.readPrefix(path)
|
||||
}
|
||||
|
||||
/**
|
||||
* Read and scan a session's log file into a {@link StoredPrefix}. Folds the
|
||||
* torn-tail comparison HERE so the `tornMarker` is the byte offset to truncate
|
||||
* to (or `undefined` when nothing is torn) — the coordinator never sees the
|
||||
* raw byteLength.
|
||||
*/
|
||||
private async readPrefix(path: string): Promise<StoredPrefix<number>> {
|
||||
const buffer = await readFile(path)
|
||||
const { meta, events, committedBytes } = scanLog(buffer)
|
||||
return {
|
||||
meta,
|
||||
events,
|
||||
...committedBytes < buffer.byteLength ? { tornMarker: committedBytes } : {},
|
||||
}
|
||||
}
|
||||
|
||||
/** Durably append a batch, lazily materializing the file when not yet present. */
|
||||
async appendBatch(meta: SessionHeader, events: readonly SessionEvent[], isMaterialized: boolean): Promise<void> {
|
||||
if (isMaterialized) {
|
||||
await this.appendLines(meta, events)
|
||||
} else {
|
||||
await this.materialize(meta, events)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Make a crash repair durable: truncate the torn tail to `tornMarker` bytes (if
|
||||
* any), then append the synthetic `closers` (if any). Two fsync'd steps — the
|
||||
* seam does not require this to be atomic.
|
||||
*/
|
||||
async commitRepair(meta: SessionHeader, tornMarker: number | undefined, closers: readonly SessionEvent[]): Promise<void> {
|
||||
if (tornMarker !== undefined) await this.repair(meta, tornMarker)
|
||||
if (closers.length > 0) await this.appendLines(meta, closers)
|
||||
}
|
||||
|
||||
/** List all stored sessions' metadata (header line only — no full-log parse). */
|
||||
async list(): Promise<SessionHeader[]> {
|
||||
const metas: SessionHeader[] = []
|
||||
for (const dir of await this.listCwdDirs()) {
|
||||
for (const name of await this.listJsonl(dir)) {
|
||||
// Read ONLY the header line, not the whole log: a session picker must
|
||||
// scale with the number of sessions, not the total size of every
|
||||
// conversation (the log persists every assistant/chunk verbatim).
|
||||
const first = await this.readFirstLine(`${dir}/${name}`)
|
||||
if (first === undefined) continue // empty/half-written file
|
||||
const meta = parseHeaderMeta(first)
|
||||
if (meta === undefined) continue // not a session header
|
||||
metas.push(meta)
|
||||
}
|
||||
}
|
||||
return metas
|
||||
}
|
||||
|
||||
// --- materialization / append / repair (file mechanics) ---
|
||||
|
||||
/** Atomically write the header line + first batch (temp-write, fsync, rename). */
|
||||
private async materialize(meta: SessionHeader, events: readonly SessionEvent[]): Promise<void> {
|
||||
const dir = sessionDir(this.root, meta.cwd)
|
||||
await mkdir(this.root, { recursive: true, mode: 0o700 })
|
||||
await this.syncDir(dirname(this.root))
|
||||
await mkdir(dir, { recursive: true, mode: 0o700 })
|
||||
await this.syncDir(this.root)
|
||||
const finalPath = logPath(this.root, meta.cwd, meta.id)
|
||||
// Never rename over an existing committed log: materialize is the FIRST write
|
||||
// of a session the backend believes is new. A file here means a different
|
||||
// session shares this id on disk — reject loudly. (createCore already guards
|
||||
// the create path, so this is unreachable-in-practice TOCTOU defense.)
|
||||
/* v8 ignore next 3 -- createCore guards collisions before materialize; this is a TOCTOU backstop */
|
||||
if (await this.exists(finalPath)) {
|
||||
throw new Error(`refusing to materialize "${meta.id}": a log already exists on disk (load/resume it instead)`)
|
||||
}
|
||||
const header = JSON.stringify(toHeaderLine(meta))
|
||||
const body = events.map(eventLine).join('\n')
|
||||
const content = header + '\n' + body + '\n'
|
||||
|
||||
const tmp = `${finalPath}.${randomBytes(6).toString('hex')}.tmp`
|
||||
const handle = await open(tmp, 'wx', 0o600)
|
||||
try {
|
||||
await handle.writeFile(content)
|
||||
await handle.sync()
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
// Publish via link()+unlink(), NOT rename(): link fails with EEXIST if the
|
||||
// final path already exists, so two processes materializing the same id
|
||||
// concurrently cannot clobber each other. rename() would silently overwrite.
|
||||
let linked = false
|
||||
try {
|
||||
await link(tmp, finalPath)
|
||||
linked = true
|
||||
} finally {
|
||||
// If link FAILED, the temp is the only reference and must be removed before
|
||||
// the original error propagates. If it SUCCEEDED, defer temp cleanup to
|
||||
// AFTER the publish is durable (below) so a temp-rm failure can never reject
|
||||
// a session whose log already published.
|
||||
/* v8 ignore next -- link failure is the TOCTOU/IO race guarded above; not reachable in test */
|
||||
if (!linked) await rm(tmp, { force: true })
|
||||
}
|
||||
// link() succeeded — the log is published. fsync the directory so the new
|
||||
// entry survives a power loss: the new link is not crash-durable until the
|
||||
// parent directory's metadata is synced.
|
||||
await this.syncDir(dir)
|
||||
// Best-effort temp cleanup: the log is already published and durable, so a
|
||||
// failure to remove the (now-redundant) temp hard link must NOT reject the
|
||||
// append. Swallow only the rm failure; nothing else of consequence runs here.
|
||||
try {
|
||||
await rm(tmp, { force: true })
|
||||
} catch {
|
||||
/* v8 ignore next -- redundant temp link; publish already durable, rm failure is an unreachable IO edge */
|
||||
}
|
||||
}
|
||||
|
||||
/** fsync a directory so a just-created/renamed entry inside it is crash-durable. */
|
||||
private async syncDir(dir: string): Promise<void> {
|
||||
const handle = await open(dir, 'r')
|
||||
try {
|
||||
await handle.sync()
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Append event lines at EOF and fsync. On a write/sync failure AFTER the kernel
|
||||
* accepted some bytes (ENOSPC, an fsync error), truncate the file back to its
|
||||
* pre-append size before rethrowing: the cursor is unchanged, so the batch will
|
||||
* be retried, and without this rollback the retry would append AFTER the partial
|
||||
* bytes — producing duplicate seqs that make `scanLog` see a gap.
|
||||
*/
|
||||
private async appendLines(meta: SessionHeader, events: readonly SessionEvent[]): Promise<void> {
|
||||
const path = logPath(this.root, meta.cwd, meta.id)
|
||||
const handle = await open(path, 'a')
|
||||
try {
|
||||
const { size: before } = await handle.stat()
|
||||
try {
|
||||
await handle.writeFile(events.map(eventLine).join('\n') + '\n')
|
||||
await handle.sync()
|
||||
} catch (error) {
|
||||
// Roll back whatever bytes landed so a retry starts from a clean EOF.
|
||||
await handle.truncate(before)
|
||||
await handle.sync()
|
||||
throw error
|
||||
}
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
}
|
||||
|
||||
/** Truncate the log file to `offset` bytes and fsync (discard the crash tail). */
|
||||
private async repair(meta: SessionHeader, offset: number): Promise<void> {
|
||||
const path = logPath(this.root, meta.cwd, meta.id)
|
||||
await truncate(path, offset)
|
||||
const handle = await open(path, 'r+')
|
||||
try {
|
||||
await handle.sync()
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
}
|
||||
|
||||
// --- discovery helpers ---
|
||||
|
||||
/**
|
||||
* Read the first newline-terminated line of a file without loading the whole
|
||||
* file. Returns undefined if the file is empty or has no complete first line.
|
||||
* Reads in bounded chunks so a huge log costs only the header read.
|
||||
*/
|
||||
private async readFirstLine(path: string): Promise<string | undefined> {
|
||||
const handle = await open(path, 'r')
|
||||
try {
|
||||
const chunks: Buffer[] = []
|
||||
const buf = Buffer.alloc(8192)
|
||||
for (;;) {
|
||||
const { bytesRead } = await handle.read(buf, 0, buf.length, null)
|
||||
if (bytesRead === 0) return undefined // EOF with no newline → no complete line
|
||||
const slice = buf.subarray(0, bytesRead)
|
||||
const nl = slice.indexOf(0x0a)
|
||||
if (nl !== -1) {
|
||||
chunks.push(slice.subarray(0, nl))
|
||||
return Buffer.concat(chunks).toString('utf8')
|
||||
}
|
||||
chunks.push(Buffer.from(slice))
|
||||
}
|
||||
} finally {
|
||||
await handle.close()
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Find a session's log file by id across ALL cwd buckets — the any-cwd scan
|
||||
* for `loadStored` (resume identifies a session by id alone). The cwd-scoped
|
||||
* lookup (`loadLive`) does NOT use this; it goes straight to `logPath(cwd)` so
|
||||
* a no-cwd session can't match a real-cwd bucket.
|
||||
*/
|
||||
private async findLog(id: SessionId): Promise<{ path: string; cwd: string | undefined } | undefined> {
|
||||
const target = encodeSegment(id) + '.jsonl'
|
||||
for (const dir of await this.listCwdDirs()) {
|
||||
const path = `${dir}/${target}`
|
||||
if (await this.exists(path)) {
|
||||
// Recover the cwd from the header so the caller has the session's bucket.
|
||||
const { meta } = scanLog(await readFile(path))
|
||||
return { path, cwd: meta.cwd }
|
||||
}
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
/** The cwd-bucket directories under the root (absolute paths). */
|
||||
private async listCwdDirs(): Promise<string[]> {
|
||||
try {
|
||||
const entries = await readdir(this.root, { withFileTypes: true })
|
||||
return entries.filter(e => e.isDirectory()).map(e => `${this.root}/${e.name}`)
|
||||
} catch (error) {
|
||||
// ENOENT = the root has not been created yet → genuinely no sessions. Any
|
||||
// other error (EACCES, ENOTDIR, transient I/O) must NOT be reported as "no
|
||||
// sessions" — a durable backend cannot silently pretend state is absent.
|
||||
if (isENOENT(error)) return []
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
private async listJsonl(dir: string): Promise<string[]> {
|
||||
const entries = await readdir(dir)
|
||||
return entries.filter(n => n.endsWith('.jsonl'))
|
||||
}
|
||||
|
||||
private async exists(path: string): Promise<boolean> {
|
||||
try {
|
||||
const handle = await open(path, 'r')
|
||||
await handle.close()
|
||||
return true
|
||||
} catch (error) {
|
||||
// Only ENOENT means absent. A permission/I/O error must surface, not be
|
||||
// collapsed to `false` — otherwise load() reports "not found" and collision
|
||||
// checks proceed under a false absence assumption.
|
||||
if (isENOENT(error)) return false
|
||||
throw error
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export default SessionPersistenceJsonl
|
||||
@@ -0,0 +1,704 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import { appendFile, mkdtemp, mkdir, rm, readFile, writeFile, readdir, stat } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type { Session, SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import SessionPersistenceJsonl from '@deepseek-ai/dsh-session-persistence-jsonl'
|
||||
import { encodeSegment, logPath, scanLog, sessionDir } from '../src/format.ts'
|
||||
import { runPersistenceContract, meta, oneTurnLog } from '../../session-persistence/tests/contract.ts'
|
||||
import { runCoordinatorContract, type CoordinatorFixture } from '../../session-persistence/tests/coordinator-contract.ts'
|
||||
|
||||
let root: string
|
||||
const dirs: string[] = []
|
||||
|
||||
async function freshRoot(): Promise<string> {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'dsh-jsonl-'))
|
||||
dirs.push(dir)
|
||||
return dir
|
||||
}
|
||||
|
||||
afterEach(async () => {
|
||||
for (const d of dirs.splice(0)) await rm(d, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
// Run the shared backend contract against the real JSONL backend.
|
||||
runPersistenceContract('jsonl', async () => {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'dsh-jsonl-'))
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(SessionPersistenceJsonl, { root: dir })
|
||||
return {
|
||||
persistence: ctx.sessionPersistence,
|
||||
dispose: async () => {
|
||||
await fiber.dispose()
|
||||
await rm(dir, { recursive: true, force: true })
|
||||
},
|
||||
}
|
||||
})
|
||||
|
||||
// Run the shared coordinator orchestration suite against the real JSONL backend.
|
||||
// One temp root is the shared storage scope (two mounted instances over the same
|
||||
// root = HMR/reload). `corruptTail` appends a partial, newline-less fragment to
|
||||
// the session's .jsonl past the committed region — a never-committed torn tail
|
||||
// that drives the coordinator's commitRepair-with-tornMarker branch over real
|
||||
// file bytes.
|
||||
runCoordinatorContract('jsonl', async (): Promise<CoordinatorFixture> => {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'dsh-jsonl-coord-'))
|
||||
return {
|
||||
mount: async (ctx) => {
|
||||
const fiber = await ctx.plugin(SessionPersistenceJsonl, { root: dir })
|
||||
return fiber
|
||||
},
|
||||
corruptTail: async (id, cwd) => {
|
||||
// A half-written record with no trailing newline: scanLog treats it as an
|
||||
// uncommitted crash fragment and reports committedBytes < byteLength, so
|
||||
// the coordinator sees a tornMarker to truncate.
|
||||
await appendFile(logPath(dir, cwd, id), '{"type":"assistant/chunk","seq":8,"ti')
|
||||
},
|
||||
cleanup: async () => { await rm(dir, { recursive: true, force: true }) },
|
||||
}
|
||||
})
|
||||
|
||||
describe('SessionPersistenceJsonl: format helpers', () => {
|
||||
it('encodeSegment neutralizes traversal, separators, and absolute paths', () => {
|
||||
expect(encodeSegment('..')).toBe('~002E~002E')
|
||||
expect(encodeSegment('.')).toBe('~002E')
|
||||
expect(encodeSegment('a/b')).toBe('a~002Fb')
|
||||
expect(encodeSegment('/etc/passwd')).toBe('~002Fetc~002Fpasswd')
|
||||
expect(encodeSegment('a\u0000b')).toBe('a~0000b')
|
||||
expect(encodeSegment('plain-ID_1.2')).toBe('plain-ID_1.2') // safe chars pass through
|
||||
expect(encodeSegment('a~b')).toBe('a~007Eb') // ~ itself is escaped
|
||||
})
|
||||
|
||||
it('encodeSegment is injective over UTF-16, incl. lone surrogates', () => {
|
||||
// Distinct lone surrogates must NOT collide (Buffer.from would normalize
|
||||
// both to U+FFFD; code-unit escaping keeps them distinct).
|
||||
const hi = encodeSegment(String.fromCharCode(0xD800))
|
||||
const lo = encodeSegment(String.fromCharCode(0xDC00))
|
||||
expect(hi).toBe('~D800')
|
||||
expect(lo).toBe('~DC00')
|
||||
expect(hi).not.toBe(lo)
|
||||
// A literal "~002F" input cannot collide with the encoding of "/".
|
||||
expect(encodeSegment('~002F')).not.toBe(encodeSegment('/'))
|
||||
})
|
||||
|
||||
it('encodeSegment rejects an empty id', () => {
|
||||
expect(() => encodeSegment('')).toThrow(/empty/)
|
||||
})
|
||||
})
|
||||
|
||||
describe('SessionPersistenceJsonl: durability and crash semantics', () => {
|
||||
let ctx: Context
|
||||
beforeEach(async () => {
|
||||
root = await freshRoot()
|
||||
ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SessionPersistenceJsonl, { root })
|
||||
})
|
||||
afterEach(async () => { await ctx.fiber.dispose() })
|
||||
|
||||
it('lazy materialization: create() writes no file until the first append', async () => {
|
||||
const m = meta('lazy', '/work')
|
||||
await ctx.sessionPersistence.create(m)
|
||||
// nothing on disk yet
|
||||
const dir = sessionDir(root, '/work')
|
||||
await expect(stat(logPath(root, '/work', m.id))).rejects.toThrow()
|
||||
expect((await ctx.sessionPersistence.list()).map(h => h.id)).not.toContain(m.id)
|
||||
|
||||
await ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
// now materialized
|
||||
expect((await stat(logPath(root, '/work', m.id))).isFile()).toBe(true)
|
||||
expect((await ctx.sessionPersistence.list()).map(h => h.id)).toContain(m.id)
|
||||
void dir
|
||||
})
|
||||
|
||||
it('round-trip is byte-identical (incl. assistant/chunk verbatim)', async () => {
|
||||
const m = meta('chunks')
|
||||
const log: SessionEvent[] = [
|
||||
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'step/start', seq: 1, time: 2, data: { turn: 1, step: 1 } },
|
||||
{ type: 'assistant/chunk', seq: 2, time: 3, data: { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'he' } } },
|
||||
{ type: 'assistant/chunk', seq: 3, time: 4, data: { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'llo' } } },
|
||||
{ type: 'assistant/message', seq: 4, time: 5, data: { turn: 1, step: 1, content: [{ type: 'text', text: 'hello' }] } },
|
||||
{ type: 'step/end', seq: 5, time: 6, data: { turn: 1, step: 1 } },
|
||||
{ type: 'turn/end', seq: 6, time: 7, data: { turn: 1, reason: { kind: 'completed' } } },
|
||||
]
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, log)
|
||||
const loaded = await ctx.sessionPersistence.load(m.id)
|
||||
expect(loaded.events).toEqual(log) // chunks preserved, contiguous seqs
|
||||
})
|
||||
|
||||
it('crash recovery: load preserves the interrupted turn and closes it with a synthetic turn/end {interrupted}', async () => {
|
||||
const m = meta('crash', '/proj')
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, oneTurnLog()) // seqs 0..5, turn/end at 5
|
||||
|
||||
// Simulate a crash mid-second-turn: append raw lines that are NOT closed by
|
||||
// a turn/end (turn/start + step/start are fully written), plus a final
|
||||
// partial line with no newline (a torn fragment never fully flushed).
|
||||
const path = logPath(root, '/proj', m.id)
|
||||
await writeFile(path, [
|
||||
JSON.stringify({ type: 'turn/start', seq: 6, time: 8, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } }),
|
||||
JSON.stringify({ type: 'step/start', seq: 7, time: 9, data: { turn: 2, step: 1 } }),
|
||||
'{"type":"assistant/chunk","seq":8,"ti', // truncated partial line (no newline)
|
||||
].join('\n'), { flag: 'a' })
|
||||
|
||||
// load PRESERVES the interrupted turn's real events (turn/start 6, step/start
|
||||
// 7) — a turn can be huge, so they must not be truncated — and durably closes
|
||||
// the orphaned turn with synthetic step/end (8) + turn/end {interrupted} (9).
|
||||
const loaded = await ctx.sessionPersistence.load(m.id)
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
|
||||
const last = loaded.events.at(-1)!
|
||||
expect(last.type === 'turn/end' && last.data.reason).toEqual({ kind: 'interrupted' })
|
||||
const stepEnd = loaded.events[8]!
|
||||
expect(stepEnd.type).toBe('step/end')
|
||||
// the torn seq-8 chunk fragment did not survive
|
||||
expect(loaded.events.some(e => e.type === 'assistant/chunk' && e.seq === 8)).toBe(false)
|
||||
|
||||
// The next append continues at seq 10 (the balanced length).
|
||||
const turn3 = [
|
||||
{ type: 'turn/start', seq: 10, time: 11, data: { turn: 3, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'turn/end', seq: 11, time: 12, data: { turn: 3, reason: { kind: 'completed' } } },
|
||||
] as SessionEvent[]
|
||||
await ctx.sessionPersistence.append(m.id, turn3)
|
||||
const reloaded = await ctx.sessionPersistence.load(m.id)
|
||||
expect(reloaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
|
||||
})
|
||||
|
||||
it('committed events are never rewritten: only the crash tail is repaired', async () => {
|
||||
const m = meta('append-only')
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
const before = await readFile(logPath(root, undefined, m.id), 'utf8')
|
||||
const committedPrefix = before // the whole committed log
|
||||
|
||||
// A crash tail then a repair-append.
|
||||
await writeFile(logPath(root, undefined, m.id), '\n{"partial', { flag: 'a' })
|
||||
await ctx.sessionPersistence.load(m.id)
|
||||
await ctx.sessionPersistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 6, time: 9, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'turn/end', seq: 7, time: 10, data: { turn: 2, reason: { kind: 'completed' } } },
|
||||
] as SessionEvent[])
|
||||
const after = await readFile(logPath(root, undefined, m.id), 'utf8')
|
||||
// the committed prefix is byte-for-byte intact at the head of the file
|
||||
expect(after.startsWith(committedPrefix)).toBe(true)
|
||||
})
|
||||
|
||||
it('a failed appendLines truncates partial bytes so a retry has no seq gap', async () => {
|
||||
const m = meta('truncate-retry')
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, oneTurnLog()) // materialized, seqs 0..5
|
||||
const sizeBefore = (await stat(logPath(root, undefined, m.id))).size
|
||||
|
||||
// Force the NEXT fsync (inside appendLines) to fail once, AFTER writeFile
|
||||
// has already put bytes on disk — simulating an ENOSPC/fsync error
|
||||
// mid-append. The recovery truncate() also fsyncs, so allow that one.
|
||||
const handle = await (await import('node:fs/promises')).open(logPath(root, undefined, m.id), 'r')
|
||||
const proto = Object.getPrototypeOf(handle) as { sync: () => Promise<void> }
|
||||
await handle.close()
|
||||
const realSync = proto.sync
|
||||
let failed = false
|
||||
const spy = vi.spyOn(proto, 'sync').mockImplementation(async function (this: unknown) {
|
||||
if (!failed) { failed = true; throw new Error('simulated fsync ENOSPC') }
|
||||
return realSync.call(this)
|
||||
})
|
||||
|
||||
const turn2 = [
|
||||
{ type: 'turn/start', seq: 6, time: 9, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'turn/end', seq: 7, time: 10, data: { turn: 2, reason: { kind: 'completed' } } },
|
||||
] as SessionEvent[]
|
||||
// The append rejects, but the partial bytes are truncated back: the file is
|
||||
// its pre-append size and the cursor is unchanged.
|
||||
await expect(ctx.sessionPersistence.append(m.id, turn2)).rejects.toThrow(/ENOSPC/)
|
||||
expect((await stat(logPath(root, undefined, m.id))).size).toBe(sizeBefore)
|
||||
spy.mockRestore()
|
||||
|
||||
// The retry now succeeds with NO seq gap — the log is contiguous 0..7.
|
||||
await ctx.sessionPersistence.append(m.id, turn2)
|
||||
const loaded = await ctx.sessionPersistence.load(m.id)
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
})
|
||||
|
||||
it('load returns a meta copy: mutating it does not corrupt backend pathing', async () => {
|
||||
const m = meta('meta-copy', '/proj')
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
const loaded = await ctx.sessionPersistence.load(m.id)
|
||||
// A consumer mutates the returned meta's cwd. The backend's stored pathing
|
||||
// metadata must be unaffected, so a later append still finds the right log.
|
||||
loaded.meta.cwd = '/evil'
|
||||
await ctx.sessionPersistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 6, time: 9, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'turn/end', seq: 7, time: 10, data: { turn: 2, reason: { kind: 'completed' } } },
|
||||
] as SessionEvent[])
|
||||
// The append landed in the ORIGINAL /proj log, not beside an /evil path.
|
||||
const reloaded = await ctx.sessionPersistence.load(m.id)
|
||||
expect(reloaded.meta.cwd).toBe('/proj')
|
||||
expect(reloaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
})
|
||||
|
||||
it('rejects a re-append of an already-stored seq', async () => {
|
||||
const m = meta('reappend')
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
await expect(ctx.sessionPersistence.append(m.id, oneTurnLog())).rejects.toThrow(/seq mismatch/)
|
||||
})
|
||||
|
||||
it('path-traversal session ids are neutralized (no escape from root)', async () => {
|
||||
const evil = SessionId('../../etc/pwn')
|
||||
const m = { version: 0, id: evil, createdAt: 1 }
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(evil, oneTurnLog())
|
||||
// The file lives UNDER root, not at ../../etc.
|
||||
const all: string[] = []
|
||||
async function walk(dir: string): Promise<void> {
|
||||
for (const e of await readdir(dir, { withFileTypes: true })) {
|
||||
const p = join(dir, e.name)
|
||||
if (e.isDirectory()) await walk(p)
|
||||
else all.push(p)
|
||||
}
|
||||
}
|
||||
await walk(root)
|
||||
expect(all.length).toBeGreaterThan(0)
|
||||
expect(all.every(p => p.startsWith(root))).toBe(true)
|
||||
})
|
||||
})
|
||||
|
||||
describe('SessionPersistenceJsonl: write path (session/event → flush)', () => {
|
||||
it('concurrent sessions do not cross buffers', async () => {
|
||||
root = await freshRoot()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SessionPersistenceJsonl, { root })
|
||||
|
||||
const a = ctx.sessions.create(SessionId('sa'))
|
||||
const b = ctx.sessions.create(SessionId('sb'))
|
||||
a.append('user/message', { content: [{ type: 'text', text: 'A' }], source: { kind: 'user' } })
|
||||
b.append('user/message', { content: [{ type: 'text', text: 'B' }], source: { kind: 'user' } })
|
||||
a.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
b.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
await ctx.parallel('session/flush', a)
|
||||
await ctx.parallel('session/flush', b)
|
||||
|
||||
const la = await ctx.sessionPersistence.load(SessionId('sa'))
|
||||
const lb = await ctx.sessionPersistence.load(SessionId('sb'))
|
||||
expect(JSON.stringify(la.events)).toContain('"A"')
|
||||
expect(JSON.stringify(la.events)).not.toContain('"B"')
|
||||
expect(JSON.stringify(lb.events)).toContain('"B"')
|
||||
expect(JSON.stringify(lb.events)).not.toContain('"A"')
|
||||
await ctx.fiber.dispose()
|
||||
})
|
||||
|
||||
})
|
||||
|
||||
|
||||
describe('SessionPersistenceJsonl: scanLog unit', () => {
|
||||
it('rejects a header-less / empty log', () => {
|
||||
expect(() => scanLog(Buffer.from(''))).toThrow()
|
||||
})
|
||||
|
||||
it('rejects a corrupt header line', () => {
|
||||
expect(() => scanLog(Buffer.from('not json\n'))).toThrow(/header/)
|
||||
})
|
||||
|
||||
it('rejects a non-session first line', () => {
|
||||
expect(() => scanLog(Buffer.from('{"type":"event"}\n'))).toThrow(/session header/)
|
||||
})
|
||||
|
||||
it('a seq gap after the last turn/end bounds the preserved tail (torn fragment tolerated)', () => {
|
||||
const log = [
|
||||
JSON.stringify({ type: 'session', version: 0, id: 'g', createdAt: 1 }),
|
||||
JSON.stringify({ type: 'turn/start', seq: 0, time: 1, data: { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } } }),
|
||||
JSON.stringify({ type: 'step/start', seq: 2, time: 2, data: { turn: 1, step: 1 } }), // gap: missing seq 1
|
||||
].join('\n') + '\n'
|
||||
// No committed turn/end, so the gap is a tolerated crash boundary: scanLog
|
||||
// PRESERVES the contiguous prefix (turn/start seq 0) — real interrupted-turn
|
||||
// work, not discarded — and stops at the gap. The orphaned open turn is
|
||||
// closed by loadCore's synthetic turn/end, not here.
|
||||
expect(scanLog(Buffer.from(log)).events.map(e => e.seq)).toEqual([0])
|
||||
})
|
||||
|
||||
it('rejects a seq gap BEFORE a later committed turn/end (committed data damaged)', () => {
|
||||
const log = [
|
||||
JSON.stringify({ type: 'session', version: 0, id: 'g2', createdAt: 1 }),
|
||||
JSON.stringify({ type: 'turn/start', seq: 0, time: 1, data: { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } } }),
|
||||
JSON.stringify({ type: 'step/start', seq: 2, time: 2, data: { turn: 1, step: 1 } }), // gap: missing seq 1
|
||||
JSON.stringify({ type: 'turn/end', seq: 3, time: 3, data: { turn: 1, reason: { kind: 'completed' } } }),
|
||||
].join('\n') + '\n'
|
||||
// A turn/end exists, so the prefix up to it is committed — but it has a hole.
|
||||
// Truncating it would silently drop committed data → unloadable.
|
||||
expect(() => scanLog(Buffer.from(log))).toThrow(/seq gap in committed region/)
|
||||
})
|
||||
|
||||
it('rejects a corrupt line BEFORE a later committed turn/end (committed data damaged)', () => {
|
||||
const log = [
|
||||
JSON.stringify({ type: 'session', version: 0, id: 'c', createdAt: 1 }),
|
||||
'{not json', // corrupt, sits in the committed region (a turn/end follows)
|
||||
JSON.stringify({ type: 'turn/end', seq: 1, time: 2, data: { turn: 1, reason: { kind: 'completed' } } }),
|
||||
].join('\n') + '\n'
|
||||
expect(() => scanLog(Buffer.from(log))).toThrow(/unparsable committed event/)
|
||||
})
|
||||
|
||||
it('a header-only log (no event lines at all) preserves nothing — committedBytes is the header', () => {
|
||||
const log = JSON.stringify({ type: 'session', version: 0, id: 'h0', createdAt: 1 }) + '\n'
|
||||
const scanned = scanLog(Buffer.from(log))
|
||||
expect(scanned.events).toEqual([])
|
||||
// committedBytes falls back to the header line's end (no preserved events).
|
||||
expect(scanned.committedBytes).toBe(Buffer.byteLength(log, 'utf8'))
|
||||
})
|
||||
|
||||
it('a corrupt line after the last turn/end bounds the preserved tail', () => {
|
||||
const log = [
|
||||
JSON.stringify({ type: 'session', version: 0, id: 'c2', createdAt: 1 }),
|
||||
JSON.stringify({ type: 'turn/start', seq: 0, time: 1, data: { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } } }),
|
||||
'{not json', // corrupt crash fragment, no turn/end committed
|
||||
].join('\n') + '\n'
|
||||
// The contiguous prefix (turn/start seq 0) is preserved; the corrupt
|
||||
// fragment after it is the tolerated crash boundary.
|
||||
expect(scanLog(Buffer.from(log)).events.map(e => e.seq)).toEqual([0])
|
||||
})
|
||||
|
||||
it('tolerates a seq gap AFTER a turn/end (uncommitted tail)', () => {
|
||||
const log = [
|
||||
JSON.stringify({ type: 'session', version: 0, id: 't', createdAt: 1 }),
|
||||
JSON.stringify({ type: 'turn/start', seq: 0, time: 1, data: { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } } }),
|
||||
JSON.stringify({ type: 'turn/end', seq: 1, time: 2, data: { turn: 1, reason: { kind: 'completed' } } }),
|
||||
JSON.stringify({ type: 'step/start', seq: 9, time: 3, data: { turn: 2, step: 1 } }), // gap in uncommitted tail
|
||||
].join('\n') + '\n'
|
||||
const { events } = scanLog(Buffer.from(log))
|
||||
expect(events.map(e => e.seq)).toEqual([0, 1]) // tail dropped
|
||||
})
|
||||
})
|
||||
|
||||
describe('SessionPersistenceJsonl: edge cases', () => {
|
||||
let ctx: Context
|
||||
beforeEach(async () => {
|
||||
root = await freshRoot()
|
||||
ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SessionPersistenceJsonl, { root })
|
||||
})
|
||||
afterEach(async () => { await ctx.fiber.dispose() })
|
||||
|
||||
it('append rejects non-JSON-serializable undefined-producing data', async () => {
|
||||
const m = meta('undef')
|
||||
await ctx.sessionPersistence.create(m)
|
||||
// A value whose JSON.stringify yields undefined (a bare function as data).
|
||||
const bad = [{ type: 'user/message', seq: 0, time: 1, data: (() => 0) as unknown }] as unknown as SessionEvent[]
|
||||
await expect(ctx.sessionPersistence.append(m.id, bad)).rejects.toThrow(/non-JSON-serializable/)
|
||||
})
|
||||
|
||||
it('create snapshots its meta: mutating the caller object after the call is ignored', async () => {
|
||||
const m = meta('create-snap', '/orig')
|
||||
const p = ctx.sessionPersistence.create(m)
|
||||
// Mutate the caller's meta object immediately after calling create.
|
||||
m.cwd = '/mutated'
|
||||
await p
|
||||
await ctx.sessionPersistence.append(SessionId('create-snap'), oneTurnLog())
|
||||
// The log materialized under the ORIGINAL cwd, not the mutated one.
|
||||
expect((await stat(logPath(root, '/orig', SessionId('create-snap')))).isFile()).toBe(true)
|
||||
await expect(stat(logPath(root, '/mutated', SessionId('create-snap')))).rejects.toThrow()
|
||||
})
|
||||
|
||||
it('list discovers sessions across multiple cwd buckets', async () => {
|
||||
await ctx.sessionPersistence.create(meta('p1', '/projA'))
|
||||
await ctx.sessionPersistence.append(SessionId('p1'), oneTurnLog())
|
||||
await ctx.sessionPersistence.create(meta('p2', '/projB'))
|
||||
await ctx.sessionPersistence.append(SessionId('p2'), oneTurnLog())
|
||||
await ctx.sessionPersistence.create(meta('p3')) // no cwd → _no-cwd bucket
|
||||
await ctx.sessionPersistence.append(SessionId('p3'), oneTurnLog())
|
||||
|
||||
const ids = (await ctx.sessionPersistence.list()).map(x => x.id).sort()
|
||||
expect(ids).toEqual(['p1', 'p2', 'p3'])
|
||||
})
|
||||
|
||||
it('list on an empty root returns nothing', async () => {
|
||||
expect(await ctx.sessionPersistence.list()).toEqual([])
|
||||
})
|
||||
|
||||
it('list skips empty and non-header .jsonl files (metadata-only read)', async () => {
|
||||
// A real session…
|
||||
await ctx.sessionPersistence.create(meta('real', '/p'))
|
||||
await ctx.sessionPersistence.append(SessionId('real'), oneTurnLog())
|
||||
// …alongside two junk files in the _no-cwd bucket: an EMPTY file (readFirstLine
|
||||
// returns undefined) and a file whose first line is not a session header
|
||||
// (parseHeaderMeta returns undefined). Both are skipped, not listed.
|
||||
const bucket = join(root, '_no-cwd')
|
||||
await mkdir(bucket, { recursive: true })
|
||||
await writeFile(join(bucket, 'empty.jsonl'), '')
|
||||
await writeFile(join(bucket, 'notheader.jsonl'), '{"type":"turn/start"}\n')
|
||||
await writeFile(join(bucket, 'badjson.jsonl'), 'not json at all\n')
|
||||
|
||||
const ids = (await ctx.sessionPersistence.list()).map(x => x.id).sort()
|
||||
expect(ids).toEqual(['real'])
|
||||
})
|
||||
|
||||
it('list reads a header line longer than the 8KB read chunk', async () => {
|
||||
// readFirstLine accumulates across reads when the first line exceeds its
|
||||
// buffer. Plant a valid header whose line is > 8192 bytes (a long extra
|
||||
// field is tolerated by the header type guard) and confirm list() reads it.
|
||||
const bucket = join(root, '_no-cwd')
|
||||
await mkdir(bucket, { recursive: true })
|
||||
const bigHeader = JSON.stringify({ type: 'session', version: 0, id: 'big', createdAt: 1, pad: 'x'.repeat(9000) })
|
||||
await writeFile(join(bucket, 'big.jsonl'), bigHeader + '\n')
|
||||
const ids = (await ctx.sessionPersistence.list()).map(x => x.id)
|
||||
expect(ids).toContain('big')
|
||||
})
|
||||
|
||||
it('a DIFFERENT live session object reusing a disposed id gets its own init (no stale cache)', async () => {
|
||||
// Session A materializes a log under id "reuse".
|
||||
const sessFiberA = await ctx.plugin(Object.assign((inner: Context) => {
|
||||
const a = inner.sessions.create(SessionId('reuse'), { meta: { cwd: '/a' } })
|
||||
for (const e of oneTurnLog()) a.append(e.type, e.data)
|
||||
}, { inject: ['sessions'] }))
|
||||
// Drain A, then dispose ITS fiber (the live session A is gone) while the
|
||||
// backend stays loaded.
|
||||
for (const s of ctx.sessions.list()) await ctx.parallel('session/flush', s)
|
||||
await sessFiberA.dispose()
|
||||
|
||||
// A NEW live Session object reuses id "reuse". The init cache is keyed by
|
||||
// the Session OBJECT, so this gets its OWN onCreated (not A's stale promise)
|
||||
// — which detects the on-disk collision and rejects, rather than silently
|
||||
// appending the new session's events onto A's log under a stale cursor.
|
||||
const backend = ctx.sessionPersistence as unknown as { inits: Map<Session, Promise<void>> }
|
||||
let b!: Session
|
||||
await ctx.plugin(Object.assign((inner: Context) => {
|
||||
b = inner.sessions.create(SessionId('reuse'), { meta: { cwd: '/a' } })
|
||||
}, { inject: ['sessions'] }))
|
||||
await expect(backend.inits.get(b)).rejects.toThrow(/already bound to a different live session|already has a persisted log on disk/)
|
||||
})
|
||||
|
||||
it('a NO-CWD live session does NOT cross-cwd-adopt a same-id log from a real cwd bucket (loadLive is scope-exact)', async () => {
|
||||
// Backend 1: materialize a log under id "x" in the cwd "/w" bucket, then
|
||||
// dispose the WHOLE backend (so backend 2 mounts with an EMPTY states map —
|
||||
// the HMR/reload path where onCreated goes through loadLive, not a tracked
|
||||
// collision).
|
||||
await ctx.sessionPersistence.create(meta('x', '/w'))
|
||||
await ctx.sessionPersistence.append(SessionId('x'), oneTurnLog())
|
||||
await ctx.fiber.dispose()
|
||||
|
||||
// Backend 2 over the SAME root. A live no-cwd session reuses id "x". Because
|
||||
// loadLive(id, undefined) is the DEFINITE no-cwd bucket (NOT an all-buckets
|
||||
// scan), case-2 adoption does NOT match the "/w" log — so it would NOT
|
||||
// silently graft the no-cwd events onto the "/w" log with a mismatched cwd
|
||||
// (the bug a non-scope-exact loadLive caused). It falls through to the
|
||||
// new-session path, where createCore's any-cwd collision probe (loadStored)
|
||||
// catches the duplicate id and REJECTS — the id is taken in another bucket.
|
||||
const ctx2 = new Context()
|
||||
await ctx2.plugin(SessionStore)
|
||||
await ctx2.plugin(SessionPersistenceJsonl, { root })
|
||||
const backend = ctx2.sessionPersistence as unknown as { inits: Map<Session, Promise<void>> }
|
||||
let b!: Session
|
||||
await ctx2.plugin(Object.assign((inner: Context) => {
|
||||
b = inner.sessions.create(SessionId('x')) // no cwd
|
||||
}, { inject: ['sessions'] }))
|
||||
await expect(backend.inits.get(b)).rejects.toThrow(/already has a persisted log on disk/)
|
||||
|
||||
// The "/w" log is untouched — no no-cwd events were grafted onto it, and no
|
||||
// `_no-cwd` log for "x" was created.
|
||||
const inW = scanLog(await readFile(logPath(root, '/w', SessionId('x'))))
|
||||
expect(inW.meta.cwd).toBe('/w')
|
||||
expect(inW.events).toHaveLength(6)
|
||||
await expect(stat(logPath(root, undefined, SessionId('x')))).rejects.toThrow()
|
||||
await ctx2.fiber.dispose()
|
||||
})
|
||||
|
||||
it('a seed with matching seq/type/time but DIFFERENT data is rejected (deep prefix compare)', async () => {
|
||||
// Materialize and load (ownerless, cursor = 6).
|
||||
await ctx.sessionPersistence.create(meta('divergent', '/a'))
|
||||
await ctx.sessionPersistence.append(SessionId('divergent'), oneTurnLog())
|
||||
await ctx.sessionPersistence.load(SessionId('divergent'))
|
||||
|
||||
const backend = ctx.sessionPersistence as unknown as { inits: Map<Session, Promise<void>> }
|
||||
// A seed that keeps every seq/type/time but mutates a payload must NOT be
|
||||
// accepted as "the same session" — otherwise drain filters those seqs as
|
||||
// already persisted and the divergent payload is silently lost.
|
||||
const tampered = oneTurnLog()
|
||||
const userMsg = tampered[1]
|
||||
if (userMsg?.type === 'user/message') userMsg.data.content = [{ type: 'text', text: 'DIFFERENT' }]
|
||||
let bad!: Session
|
||||
await ctx.plugin(Object.assign((inner: Context) => {
|
||||
bad = inner.sessions.create(SessionId('divergent'), { seed: tampered, meta: { cwd: '/a' } })
|
||||
}, { inject: ['sessions'] }))
|
||||
await expect(backend.inits.get(bad)).rejects.toThrow(/do not match this live session|already has a persisted log/)
|
||||
})
|
||||
|
||||
it('a second live session reusing a bound id is rejected', async () => {
|
||||
// A live session materializes and owns the id.
|
||||
const firstFiber = await ctx.plugin(Object.assign((inner: Context) => {
|
||||
const a = inner.sessions.create(SessionId('bound'), { meta: { cwd: '/a' } })
|
||||
a.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
a.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
}, { inject: ['sessions'] }))
|
||||
for (const s of ctx.sessions.list()) await ctx.parallel('session/flush', s)
|
||||
await firstFiber.dispose()
|
||||
|
||||
const backend = ctx.sessionPersistence as unknown as { inits: Map<Session, Promise<void>> }
|
||||
let second!: Session
|
||||
await ctx.plugin(Object.assign((inner: Context) => {
|
||||
second = inner.sessions.create(SessionId('bound'), { meta: { cwd: '/a' } })
|
||||
}, { inject: ['sessions'] }))
|
||||
await expect(backend.inits.get(second))
|
||||
.rejects.toThrow(/already bound to a different live session|already has a persisted log|do not match/)
|
||||
})
|
||||
|
||||
it('list returns nothing when the root directory does not exist', async () => {
|
||||
const ctx2 = new Context()
|
||||
await ctx2.plugin(SessionStore)
|
||||
await ctx2.plugin(SessionPersistenceJsonl, { root: join(root, 'does-not-exist-yet') })
|
||||
expect(await ctx2.sessionPersistence.list()).toEqual([])
|
||||
await ctx2.fiber.dispose()
|
||||
})
|
||||
|
||||
it('list surfaces a non-ENOENT root error (ENOTDIR) instead of reporting no sessions', async () => {
|
||||
// A durable backend must NOT collapse a storage fault to "no sessions". Point
|
||||
// the root at a regular FILE: readdir then fails with ENOTDIR, which must
|
||||
// propagate rather than be swallowed as an empty listing.
|
||||
const filePath = join(root, 'not-a-dir')
|
||||
await writeFile(filePath, 'x')
|
||||
const ctx2 = new Context()
|
||||
await ctx2.plugin(SessionStore)
|
||||
await ctx2.plugin(SessionPersistenceJsonl, { root: filePath })
|
||||
await expect(ctx2.sessionPersistence.list()).rejects.toThrow(/ENOTDIR/)
|
||||
await ctx2.fiber.dispose()
|
||||
})
|
||||
|
||||
it('loadLive surfaces a non-ENOENT lookup error (ENOTDIR) instead of reporting absent', async () => {
|
||||
// A non-ENOENT error from the per-id open() must surface, not be collapsed to
|
||||
// "not found" (which would let live-adoption proceed under a false absence
|
||||
// assumption). A live session's onCreated reaches loadLive(id, cwd) →
|
||||
// exists(logPath). Make that cwd's bucket DIRECTORY a regular file: open()ing
|
||||
// `bucket/<id>.jsonl` under it then fails ENOTDIR.
|
||||
const cwd = '/x'
|
||||
const ctx2 = new Context()
|
||||
await ctx2.plugin(SessionStore)
|
||||
await ctx2.plugin(SessionPersistenceJsonl, { root })
|
||||
await writeFile(sessionDir(root, cwd), 'x') // bucket path is now a FILE
|
||||
const backend = ctx2.sessionPersistence as unknown as { inits: Map<Session, Promise<void>> }
|
||||
let s!: Session
|
||||
await ctx2.plugin(Object.assign((inner: Context) => {
|
||||
s = inner.sessions.create(SessionId('exists-fault'), { meta: { cwd } })
|
||||
}, { inject: ['sessions'] }))
|
||||
await expect(backend.inits.get(s)).rejects.toThrow(/ENOTDIR/)
|
||||
await ctx2.fiber.dispose()
|
||||
})
|
||||
|
||||
it('append() to a disk-only session adopts it and repairs a crash tail', async () => {
|
||||
// Persist a session, then corrupt its tail, all through ONE backend.
|
||||
const m = meta('disk-append', '/d')
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
await writeFile(logPath(root, '/d', m.id), '\n{"partial crash', { flag: 'a' })
|
||||
|
||||
// A FRESH backend with no in-memory state: append directly (no prior load)
|
||||
// → append must adopt from disk, and the adopt's load schedules a repair
|
||||
// that the same append then performs before writing.
|
||||
const ctx2 = new Context()
|
||||
await ctx2.plugin(SessionStore)
|
||||
await ctx2.plugin(SessionPersistenceJsonl, { root })
|
||||
await ctx2.sessionPersistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 6, time: 9, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'turn/end', seq: 7, time: 10, data: { turn: 2, reason: { kind: 'completed' } } },
|
||||
] as SessionEvent[])
|
||||
const loaded = await ctx2.sessionPersistence.load(m.id)
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
await ctx2.fiber.dispose()
|
||||
})
|
||||
|
||||
it('a header-only log (open turn, no turn/end) preserves the open turn on load and closes it', async () => {
|
||||
// A session whose only durable content is an unclosed first turn. scanLog
|
||||
// preserves the turn/start; loadCore closes it with a synthetic
|
||||
// turn/end {interrupted} so the returned log is balanced.
|
||||
const m = meta('open-turn', '/h')
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
] as SessionEvent[])
|
||||
const { events } = await ctx.sessionPersistence.load(m.id)
|
||||
expect(events.map(e => e.type)).toEqual(['turn/start', 'turn/end'])
|
||||
const end = events[1]!
|
||||
expect(end.type === 'turn/end' && end.data.reason).toEqual({ kind: 'interrupted' })
|
||||
})
|
||||
|
||||
|
||||
it('createCore rejects an id already on disk under a DIFFERENT cwd bucket', async () => {
|
||||
// Persist the id under cwd A.
|
||||
const a = meta('dup-id', '/projA')
|
||||
await ctx.sessionPersistence.create(a)
|
||||
await ctx.sessionPersistence.append(a.id, oneTurnLog())
|
||||
// A fresh backend creating the SAME id under cwd B must still refuse: load
|
||||
// identifies by id across all buckets, so a second log would make resume
|
||||
// nondeterministic. create scans every bucket, not just meta.cwd's.
|
||||
const ctx2 = new Context()
|
||||
await ctx2.plugin(SessionStore)
|
||||
await ctx2.plugin(SessionPersistenceJsonl, { root })
|
||||
await expect(ctx2.sessionPersistence.create(meta('dup-id', '/projB')))
|
||||
.rejects.toThrow(/already has a persisted log on disk/)
|
||||
await ctx2.fiber.dispose()
|
||||
})
|
||||
|
||||
it('flush keeps buffered events when the append fails (no silent loss)', async () => {
|
||||
root = await freshRoot()
|
||||
const ctx2 = new Context()
|
||||
await ctx2.plugin(SessionStore)
|
||||
await ctx2.plugin(SessionPersistenceJsonl, { root })
|
||||
const session = ctx2.sessions.create(SessionId('flush-fail'))
|
||||
// A full turn lands in the write-behind buffer.
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
// Make the durable materialize fail on the next flush.
|
||||
const backend = ctx2.sessionPersistence as unknown as { materialize: (...args: unknown[]) => Promise<void> }
|
||||
const origMat = backend.materialize.bind(backend)
|
||||
backend.materialize = () => Promise.reject(new Error('disk full'))
|
||||
await expect(ctx2.parallel('session/flush', session)).rejects.toThrow(/disk full/)
|
||||
// The events are STILL buffered (not silently dropped): a retry persists them.
|
||||
backend.materialize = origMat
|
||||
await ctx2.parallel('session/flush', session)
|
||||
const loaded = await ctx2.sessionPersistence.load(SessionId('flush-fail'))
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1])
|
||||
await ctx2.fiber.dispose()
|
||||
})
|
||||
|
||||
it('rejects non-JSON event data: BigInt, function, circular, Map, undefined property', async () => {
|
||||
const m = meta('serial')
|
||||
await ctx.sessionPersistence.create(m)
|
||||
const bad = (extra: unknown) => [{ type: 'user/message', seq: 0, time: 1, data: { content: [{ type: 'text', text: 'x' }], source: { kind: 'user' }, extra } }] as unknown as SessionEvent[]
|
||||
await expect(ctx.sessionPersistence.append(m.id, bad(1n))).rejects.toThrow(/non-JSON-serializable/)
|
||||
await expect(ctx.sessionPersistence.append(m.id, bad(() => 0))).rejects.toThrow(/non-JSON-serializable/)
|
||||
await expect(ctx.sessionPersistence.append(m.id, bad(Symbol('s')))).rejects.toThrow(/non-JSON-serializable/)
|
||||
await expect(ctx.sessionPersistence.append(m.id, bad(new Map()))).rejects.toThrow(/non-JSON-serializable/)
|
||||
await expect(ctx.sessionPersistence.append(m.id, bad(undefined))).rejects.toThrow(/non-JSON-serializable/)
|
||||
await expect(ctx.sessionPersistence.append(m.id, bad(Infinity))).rejects.toThrow(/non-JSON-serializable/)
|
||||
// a circular structure
|
||||
const circ: Record<string, unknown> = {}
|
||||
circ.self = circ
|
||||
await expect(ctx.sessionPersistence.append(m.id, bad(circ))).rejects.toThrow(/non-JSON-serializable/)
|
||||
// The session was never materialized by any of the rejected appends.
|
||||
expect((await ctx.sessionPersistence.list()).map(h => h.id)).not.toContain(m.id)
|
||||
})
|
||||
|
||||
it('accepts well-formed JSON values (null, booleans, nested arrays/objects)', async () => {
|
||||
const m = meta('json-ok')
|
||||
await ctx.sessionPersistence.create(m)
|
||||
const ev = [{ type: 'user/message', seq: 0, time: 1, data: { content: [{ type: 'text', text: 'x' }], source: { kind: 'user' }, extra: { a: null, b: true, c: [1, 2, { d: 'nested' }] } } }] as unknown as SessionEvent[]
|
||||
await ctx.sessionPersistence.append(m.id, ev)
|
||||
expect((await ctx.sessionPersistence.list()).map(h => h.id)).toContain(m.id)
|
||||
})
|
||||
|
||||
it('Session.append rejects a non-serializable event at the source (never enters the log)', () => {
|
||||
const session = ctx.sessions.create(SessionId('reject-bad'))
|
||||
// Serializability is enforced at the source: Session.append throws on a
|
||||
// BigInt-bearing event BEFORE it enters session.events, so the durable log
|
||||
// can never diverge from the live log. The throw surfaces at the caller's
|
||||
// append site, not asynchronously in a backend flush.
|
||||
expect(() => {
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'bad' }], source: { kind: 'user' }, bad: 1n } as never)
|
||||
}).toThrow(/non-JSON-serializable/)
|
||||
// The bad event was rejected, so the log stayed empty.
|
||||
expect(session.events.length).toBe(0)
|
||||
})
|
||||
|
||||
})
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
{
|
||||
"path": "../../session-persistence/session-persistence"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
# @deepseek-ai/dsh-session-persistence-sqlite
|
||||
|
||||
A SQLite durable session-persistence backend — a second `SessionPersistence` implementation ([session persistence](../../../docs/rfc/implemented/architecture/2026-06-14-session-persistence.md)), built to validate that the abstract seam and the shared `runPersistenceContract` suite are genuinely backend-agnostic. It satisfies the SAME contract as `dsh-session-persistence-jsonl` (append-only, contiguous-seq, lazy materialization, interrupted-turn close on load), expressed over `node:sqlite` rows instead of file bytes.
|
||||
|
||||
> **TODO:** this backend talks to `node:sqlite` directly. If a cordis database service (`cordis/db` / a `@cordisjs` SQL driver plugin) is adopted, route through that instead of holding a raw `DatabaseSync` here — the contract surface (`SessionPersistence`) would not change, only the storage driver.
|
||||
|
||||
## Storage model
|
||||
|
||||
Each `SessionEvent` maps 1:1 onto a row in an `events` table `(session_id, seq, type, time, data, source_event_seqs, surface_op)` — `data` is the event payload as JSON text, so the row shape is the event verbatim (including `assistant/chunk`, keeping `seq` contiguous). The two `TEXT` columns `source_event_seqs` and `surface_op` are nullable; they store the event's optional surface-metadata fields (see [session surface](../../../docs/rfc/implemented/architecture/2026-06-18-session-surface.md)). Out-of-log metadata (`SessionHeader`) lives in a `sessions` row. A `sessions` row is written only by the first `append` — its existence is the lazy-materialization signal (`list` reports exactly the sessions that have a row), so no separate column is needed.
|
||||
|
||||
The repo targets Node ≥ 24 (the root `engines` field), which includes the stable `node:sqlite` module. The database opens with `foreign_keys = ON` (so `ON DELETE CASCADE` drops a session's events with its row) and `journal_mode = WAL`. The table-layout version is stored in `PRAGMA user_version` and checked on open: a fresh database is stamped with the current `SCHEMA_VERSION`; a database written by any other, incompatible build (a non-current `user_version`, older or newer) is rejected rather than opened against an unknown layout — there is no migration (unreleased software).
|
||||
|
||||
## Contract semantics over rows
|
||||
|
||||
- **Append = a transaction.** `append` runs `BEGIN`/`COMMIT` around the batch: it materializes the `sessions` row (if still lazy) and INSERTs every event, asserting the contiguous-seq contract first (the first event's `seq` must equal the stored next-seq). A mid-batch failure (a UNIQUE violation on a duplicated seq) rolls back entirely, so the stored log and the in-memory cursor stay consistent. (`load()` already balanced the stored log, so `append` never has to repair a crash tail.)
|
||||
- **Lazy materialization.** `create()` records intent in memory only — no row is written until the first `append`. A created-but-never-appended session has no `sessions` row, so it is absent from `list()` (which reports exactly the sessions that have a row).
|
||||
- **Interrupted-turn close on load.** `load()` reads every stored event ordered by `seq` and finds the longest seq-contiguous, parseable prefix — INCLUDING the real events of an interrupted final turn after the last `turn/end` (the loop only flushes at `turn/end`, so a process killed mid-turn leaves real, fully-written rows past it). A single turn can be huge in a long-horizon task, so those events are **preserved, never truncated**: `load()` CLOSES the orphaned turn by durably appending the minimal synthetic boundary events (an error `tool/result` for every assistant tool call left unanswered, a `step/end` if a step was open, then a `turn/end` carrying `{ kind: 'interrupted' }`), inside one transaction that also DELETEs any never-fully-written torn tail row. `load()` is therefore mutating — after it the stored rows are balanced and the cursor is truthful, so the next `append` continues cleanly. The boundary (last `turn/end`, torn-tail detection) is computed from the `seq`/`type` columns so a malformed `data` in a torn tail row is never parsed (discarded, not unloadable). A parse error or `seq` gap inside the committed region (at or before the last real `turn/end`) makes the session unloadable. A session whose only turn never closed keeps its metadata row and stays present in `list()` — the same as the JSONL backend, whose file likewise survives a first append that never reached `turn/end`.
|
||||
|
||||
## Configuration (schemastery)
|
||||
|
||||
```ts
|
||||
interface Config {
|
||||
path: string // SQLite database file path, or ':memory:' for an in-process DB
|
||||
}
|
||||
```
|
||||
|
||||
## Write path
|
||||
|
||||
Like the JSONL backend, the plugin also installs the `session/event` → buffer → `session/flush` drain: it snapshots each event when buffered (the live `session.events` object is mutable), persists a fork's seed once on `session/created`, keeps a per-session write cursor so a resumed session never re-appends stored events, and seeds existing live sessions on apply (HMR does not replay `session/created`). Dispose awaits every in-flight init + final drain and then closes the database, so no write lands after teardown.
|
||||
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-session-persistence-sqlite",
|
||||
"description": "SQLite durable session persistence backend for the DeepSeek Harness",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib",
|
||||
"src"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-session": "^0.0.1",
|
||||
"@deepseek-ai/dsh-session-persistence": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
},
|
||||
"dependencies": {
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-persistence": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,261 @@
|
||||
/**
|
||||
* SQLite durable session-persistence backend (`@deepseek-ai/dsh-session-persistence-sqlite`).
|
||||
*
|
||||
* A SECOND {@link SessionPersistence} implementation, built to validate that the
|
||||
* abstract seam + the shared `runPersistenceContract` suite are genuinely
|
||||
* backend-agnostic: the same append-only / contiguous-seq / lazy-materialization
|
||||
* / interrupted-turn-close-on-load semantics the JSONL backend expresses over
|
||||
* file bytes, expressed here over `node:sqlite` rows. Each `SessionEvent` maps
|
||||
* 1:1 onto a row `(session_id, seq, type, time, data, source_event_seqs, surface_op)`.
|
||||
*
|
||||
* Like the JSONL backend it supplies ONLY the storage primitives (the
|
||||
* {@link PersistenceBackend} hooks below — INSERT/DELETE/SELECT inside
|
||||
* transactions); all the write-path orchestration lives in the backend-agnostic
|
||||
* {@link PersistenceCoordinator} this class composes. The four public
|
||||
* {@link SessionPersistence} methods delegate to the coordinator.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-session-persistence-sqlite
|
||||
*/
|
||||
|
||||
import { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import { DatabaseSync } from 'node:sqlite'
|
||||
import { mkdir } from 'node:fs/promises'
|
||||
import { dirname, resolve } from 'node:path'
|
||||
import {
|
||||
SessionPersistence, PersistenceCoordinator,
|
||||
type PersistenceBackend, type StoredPrefix,
|
||||
} from '@deepseek-ai/dsh-session-persistence'
|
||||
import type { Session, SessionEvent, SurfaceEventType, SessionId, SessionHeader } from '@deepseek-ai/dsh-session'
|
||||
import {
|
||||
openDatabase, rowToMeta, scanRows, type EventRow, type SessionRow,
|
||||
} from './schema.ts'
|
||||
|
||||
export { SCHEMA_VERSION } from './schema.ts'
|
||||
|
||||
/**
|
||||
* Serialize an event's surface-metadata fields for SQL binding. Both fields are
|
||||
* nullable TEXT columns — null when the event has no surface metadata (non-surface
|
||||
* events, events written before surface support).
|
||||
*/
|
||||
function surfaceBindings(event: SessionEvent): [string | null, string | null] {
|
||||
const se = event as SessionEvent<SurfaceEventType>
|
||||
return [
|
||||
se.sourceEventSeqs ? JSON.stringify(se.sourceEventSeqs) : null,
|
||||
se.surfaceOp !== undefined ? JSON.stringify(se.surfaceOp) : null,
|
||||
]
|
||||
}
|
||||
|
||||
/** Plugin configuration. */
|
||||
export interface Config {
|
||||
/**
|
||||
* Filesystem path to the SQLite database file. The special value `:memory:`
|
||||
* opens an in-process database (tests); a file path is created (with parent
|
||||
* dirs) on construction.
|
||||
*/
|
||||
path: string
|
||||
}
|
||||
|
||||
/**
|
||||
* The SQLite persistence backend. Load as a plugin; it registers as
|
||||
* `ctx.sessionPersistence` and (via the coordinator) installs the write-path
|
||||
* listeners. Its torn-tail marker is the seq to delete from.
|
||||
*/
|
||||
export class SessionPersistenceSqlite extends SessionPersistence implements PersistenceBackend<number> {
|
||||
static inject = ['sessions']
|
||||
|
||||
static Config: z<Config> = z.object({
|
||||
path: z.string().required(),
|
||||
})
|
||||
|
||||
/**
|
||||
* Backend label for the coordinator's dispose diagnostics. Intentionally
|
||||
* shadows cordis `Service.name` (set to `'sessionPersistence'` by the base);
|
||||
* see the JSONL backend for why this does not affect service resolution.
|
||||
*/
|
||||
override readonly name = 'session-persistence-sqlite'
|
||||
|
||||
private db!: DatabaseSync
|
||||
private ready: Promise<void>
|
||||
private coordinator: PersistenceCoordinator<number>
|
||||
|
||||
constructor(ctx: Context, public config: Config) {
|
||||
super(ctx)
|
||||
// Open the database asynchronously (the parent directory may need creating);
|
||||
// every hook awaits `ready` first. Opening synchronously would force a sync
|
||||
// mkdir and block plugin apply.
|
||||
this.ready = this.openDb(config.path)
|
||||
this.coordinator = new PersistenceCoordinator<number>(this.ctx, this)
|
||||
}
|
||||
|
||||
private async openDb(path: string): Promise<void> {
|
||||
if (path !== ':memory:') {
|
||||
const abs = resolve(path)
|
||||
await mkdir(dirname(abs), { recursive: true, mode: 0o700 })
|
||||
this.db = openDatabase(abs)
|
||||
} else {
|
||||
this.db = openDatabase(path)
|
||||
}
|
||||
}
|
||||
|
||||
// --- SessionPersistence service surface (delegated to the coordinator) ---
|
||||
|
||||
create(meta: SessionHeader): Promise<void> {
|
||||
return this.coordinator.create(meta)
|
||||
}
|
||||
|
||||
append(id: SessionId, events: readonly SessionEvent[]): Promise<void> {
|
||||
return this.coordinator.append(id, events)
|
||||
}
|
||||
|
||||
load(id: SessionId): Promise<{ meta: SessionHeader; events: SessionEvent[] }> {
|
||||
return this.coordinator.load(id)
|
||||
}
|
||||
|
||||
// `list` is BOTH the public service method and the PersistenceBackend hook —
|
||||
// one method (the SELECT below). The coordinator adds no orchestration for
|
||||
// listing, so routing it through the coordinator would just recurse. Defined
|
||||
// once, in the "PersistenceBackend hooks" section.
|
||||
|
||||
/**
|
||||
* The per-session init promises, exposed for white-box tests that await a
|
||||
* specific session's onCreated (there is no public API to await one init).
|
||||
*/
|
||||
get inits(): Map<Session, Promise<void>> {
|
||||
return this.coordinator.inits
|
||||
}
|
||||
|
||||
// --- PersistenceBackend hooks (the SQLite storage primitives) ---
|
||||
|
||||
/** Read a stored prefix by id (ids are globally unique — no scope to scan). */
|
||||
loadStored(id: SessionId): Promise<StoredPrefix<number> | undefined> {
|
||||
return this.readPrefix(id)
|
||||
}
|
||||
|
||||
/** Read a stored prefix; `cwd` is ignored (the id is globally unique in SQLite). */
|
||||
loadLive(id: SessionId, _cwd: string | undefined): Promise<StoredPrefix<number> | undefined> {
|
||||
return this.readPrefix(id)
|
||||
}
|
||||
|
||||
/**
|
||||
* Read a session's row + ordered events into a {@link StoredPrefix}. The
|
||||
* torn-tail marker is the seq from which a never-committed tail must be deleted
|
||||
* (`scanRows` already returns it as `number | undefined`).
|
||||
*/
|
||||
private async readPrefix(id: SessionId): Promise<StoredPrefix<number> | undefined> {
|
||||
await this.ready
|
||||
const row = this.rowFor(id)
|
||||
if (row === undefined) return undefined
|
||||
const meta = rowToMeta(row)
|
||||
const eventRows = this.db
|
||||
.prepare('SELECT seq, type, time, data, source_event_seqs, surface_op FROM events WHERE session_id = ? ORDER BY seq')
|
||||
.all(id) as unknown as EventRow[]
|
||||
const { preserved, tornFrom } = scanRows(eventRows)
|
||||
return { meta, events: preserved, ...tornFrom !== undefined ? { tornMarker: tornFrom } : {} }
|
||||
}
|
||||
|
||||
/**
|
||||
* Durably append a batch in ONE transaction: materialize the sessions row (if
|
||||
* lazy) and INSERT every event, or roll back entirely. The transaction is the
|
||||
* atomicity + durability boundary, so a mid-batch failure (a UNIQUE violation
|
||||
* on a duplicated seq) leaves the stored log untouched.
|
||||
*/
|
||||
async appendBatch(meta: SessionHeader, events: readonly SessionEvent[], isMaterialized: boolean): Promise<void> {
|
||||
await this.ready
|
||||
const insertEvent = this.db.prepare(
|
||||
'INSERT INTO events (session_id, seq, type, time, data, source_event_seqs, surface_op) VALUES (?, ?, ?, ?, ?, ?, ?)',
|
||||
)
|
||||
this.db.exec('BEGIN')
|
||||
try {
|
||||
if (!isMaterialized) this.writeRow(meta)
|
||||
for (const event of events) {
|
||||
const [surfaceSeqs, surfaceOp] = surfaceBindings(event)
|
||||
insertEvent.run(meta.id, event.seq, event.type, event.time, JSON.stringify(event.data), surfaceSeqs, surfaceOp)
|
||||
}
|
||||
this.db.exec('COMMIT')
|
||||
} catch (error) {
|
||||
this.db.exec('ROLLBACK')
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Make a crash repair durable in ONE transaction: DELETE the torn tail (from
|
||||
* `tornMarker`) and INSERT the synthetic `closers`. After COMMIT the stored rows
|
||||
* == the balanced log.
|
||||
*/
|
||||
async commitRepair(meta: SessionHeader, tornMarker: number | undefined, closers: readonly SessionEvent[]): Promise<void> {
|
||||
await this.ready
|
||||
this.db.exec('BEGIN')
|
||||
try {
|
||||
if (tornMarker !== undefined) {
|
||||
this.db.prepare('DELETE FROM events WHERE session_id = ? AND seq >= ?').run(meta.id, tornMarker)
|
||||
}
|
||||
if (closers.length > 0) {
|
||||
const insertEvent = this.db.prepare(
|
||||
'INSERT INTO events (session_id, seq, type, time, data, source_event_seqs, surface_op) VALUES (?, ?, ?, ?, ?, ?, ?)',
|
||||
)
|
||||
for (const event of closers) {
|
||||
const [surfaceSeqs, surfaceOp] = surfaceBindings(event)
|
||||
insertEvent.run(meta.id, event.seq, event.type, event.time, JSON.stringify(event.data), surfaceSeqs, surfaceOp)
|
||||
}
|
||||
}
|
||||
this.db.exec('COMMIT')
|
||||
} catch (error) {
|
||||
// The DELETE+INSERT cannot collide (a row at a closer's seq is preserved or
|
||||
// deleted as torn first); this rolls back a DB-level failure (disk full,
|
||||
// etc.), unreachable in test.
|
||||
/* v8 ignore start */
|
||||
this.db.exec('ROLLBACK')
|
||||
throw error
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
}
|
||||
|
||||
/** List all materialized sessions' metadata (every row is a materialized session). */
|
||||
async list(): Promise<SessionHeader[]> {
|
||||
await this.ready
|
||||
const rows = this.db
|
||||
.prepare('SELECT * FROM sessions')
|
||||
.all() as unknown as SessionRow[]
|
||||
return rows.map(rowToMeta)
|
||||
}
|
||||
|
||||
/** Close the database handle (awaited by the coordinator's dispose, post-drain). */
|
||||
async close(): Promise<void> {
|
||||
await this.ready
|
||||
this.db.close()
|
||||
}
|
||||
|
||||
// --- row helpers ---
|
||||
|
||||
/** Fetch a session's row, or undefined if absent. */
|
||||
private rowFor(id: SessionId): SessionRow | undefined {
|
||||
return this.db.prepare('SELECT * FROM sessions WHERE id = ?').get(id) as unknown as SessionRow | undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Insert-or-replace a session's metadata row. The only caller is the first
|
||||
* materializing `appendBatch`, so writing the row IS the materialization (its
|
||||
* existence is the signal `list` reads).
|
||||
*/
|
||||
private writeRow(meta: SessionHeader): void {
|
||||
this.db.prepare(`
|
||||
INSERT INTO sessions (id, version, created_at, cwd, parent_session)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT(id) DO UPDATE SET
|
||||
version = excluded.version,
|
||||
created_at = excluded.created_at,
|
||||
cwd = excluded.cwd,
|
||||
parent_session = excluded.parent_session
|
||||
`).run(
|
||||
meta.id,
|
||||
meta.version,
|
||||
meta.createdAt,
|
||||
meta.cwd ?? null,
|
||||
meta.parentSession ?? null,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
export default SessionPersistenceSqlite
|
||||
@@ -0,0 +1,195 @@
|
||||
/**
|
||||
* Schema + load-time helpers for the SQLite session-persistence backend: the
|
||||
* DDL (a `sessions` metadata table and a 1:1 `events` row per `SessionEvent`),
|
||||
* the database open/configure step, and the last-`turn/end` cut that gives the
|
||||
* SQLite backend the SAME crash-tail-on-load semantics as the JSONL backend.
|
||||
*
|
||||
* @module dsh-session-persistence-sqlite/schema
|
||||
*/
|
||||
|
||||
import { DatabaseSync } from 'node:sqlite'
|
||||
import type { SessionEvent, SessionId, SessionHeader, SurfaceOp } from '@deepseek-ai/dsh-session'
|
||||
|
||||
/**
|
||||
* The on-disk schema version. Bumped only on a breaking change to the table
|
||||
* layout; orthogonal to a session's own `version` (which versions the EVENT
|
||||
* vocabulary, stored per session in the `sessions` row).
|
||||
*/
|
||||
export const SCHEMA_VERSION = 2
|
||||
|
||||
/**
|
||||
* A row of the `sessions` table — the out-of-log metadata ({@link SessionHeader}).
|
||||
* The row's EXISTENCE is the materialization signal: it is written only by the
|
||||
* first `append` (lazy materialization), so a created-but-never-appended
|
||||
* session has no row and is absent from `list`, mirroring the JSONL
|
||||
* backend's "no file until first append".
|
||||
*/
|
||||
export interface SessionRow {
|
||||
id: string
|
||||
version: number
|
||||
created_at: number
|
||||
cwd: string | null
|
||||
parent_session: string | null
|
||||
}
|
||||
|
||||
/** An `events` table row: one `SessionEvent` mapped 1:1 (`data` is JSON text). */
|
||||
export interface EventRow {
|
||||
seq: number
|
||||
type: string
|
||||
time: number
|
||||
data: string
|
||||
/** JSON-encoded `number[]` — the event's sourceEventSeqs, or null. */
|
||||
source_event_seqs: string | null
|
||||
/** JSON-encoded `SurfaceOp` — how the event entered the surface, or null. */
|
||||
surface_op: string | null
|
||||
}
|
||||
|
||||
/**
|
||||
* Open the database at `path` and apply the schema + pragmas. `foreign_keys`
|
||||
* makes `ON DELETE CASCADE` drop a session's events with its row; `journal_mode
|
||||
* = WAL` matches the durability model the ADR records (the row shape maps 1:1
|
||||
* onto `SessionEvent`; opencode runs this exact shape on SQLite/WAL).
|
||||
*
|
||||
* The table-layout version is persisted in SQLite's `PRAGMA user_version` and
|
||||
* checked on open: a fresh database (user_version 0) is stamped with the
|
||||
* current {@link SCHEMA_VERSION}; an existing database whose version is NOT the
|
||||
* current one (written by a different, incompatible build — older or newer) is
|
||||
* REJECTED rather than opened against a layout this build does not understand.
|
||||
* There are no migrations: v1 had a different `sessions` layout and is not
|
||||
* upgraded in place.
|
||||
*/
|
||||
export function openDatabase(path: string): DatabaseSync {
|
||||
const db = new DatabaseSync(path)
|
||||
db.exec('PRAGMA foreign_keys = ON')
|
||||
db.exec('PRAGMA journal_mode = WAL')
|
||||
// `PRAGMA user_version` always returns exactly one row { user_version }.
|
||||
const { user_version: onDisk } = db.prepare('PRAGMA user_version').get() as { user_version: number }
|
||||
if (onDisk !== 0 && onDisk !== SCHEMA_VERSION) {
|
||||
db.close()
|
||||
throw new Error(`session database at "${path}" has schema version ${onDisk}, incompatible with this build (${SCHEMA_VERSION})`)
|
||||
}
|
||||
if (onDisk === 0) {
|
||||
// Fresh (or pre-versioning) database: stamp the current layout version.
|
||||
// PRAGMA does not accept bound parameters, so interpolate the integer
|
||||
// constant (SCHEMA_VERSION is a trusted in-code number, not user input).
|
||||
db.exec(`PRAGMA user_version = ${SCHEMA_VERSION}`)
|
||||
}
|
||||
db.exec(`
|
||||
CREATE TABLE IF NOT EXISTS sessions (
|
||||
id TEXT PRIMARY KEY,
|
||||
version INTEGER NOT NULL,
|
||||
created_at INTEGER NOT NULL,
|
||||
cwd TEXT,
|
||||
parent_session TEXT
|
||||
) STRICT
|
||||
`)
|
||||
db.exec(`
|
||||
CREATE TABLE IF NOT EXISTS events (
|
||||
session_id TEXT NOT NULL REFERENCES sessions(id) ON DELETE CASCADE,
|
||||
seq INTEGER NOT NULL,
|
||||
type TEXT NOT NULL,
|
||||
time INTEGER NOT NULL,
|
||||
data TEXT NOT NULL,
|
||||
source_event_seqs TEXT,
|
||||
surface_op TEXT,
|
||||
PRIMARY KEY (session_id, seq)
|
||||
) STRICT
|
||||
`)
|
||||
return db
|
||||
}
|
||||
|
||||
/** Reconstruct the {@link SessionHeader} from a `sessions` row. */
|
||||
export function rowToMeta(row: SessionRow): SessionHeader {
|
||||
return {
|
||||
version: row.version,
|
||||
id: row.id as SessionId,
|
||||
createdAt: row.created_at,
|
||||
...row.cwd !== null ? { cwd: row.cwd } : {},
|
||||
...row.parent_session !== null ? { parentSession: row.parent_session as SessionId } : {},
|
||||
}
|
||||
}
|
||||
|
||||
/** Reconstruct a {@link SessionEvent} from an `events` row (parses `data`). */
|
||||
export function rowToEvent(row: EventRow): SessionEvent {
|
||||
// Surface-metadata fields are conditional on the event type in the type
|
||||
// system; spread them so each variant gets only the fields it declares.
|
||||
const surfaceFields = {
|
||||
...row.source_event_seqs !== null ? { sourceEventSeqs: JSON.parse(row.source_event_seqs) as number[] } : {},
|
||||
...row.surface_op !== null ? { surfaceOp: JSON.parse(row.surface_op) as SurfaceOp } : {},
|
||||
}
|
||||
return {
|
||||
type: row.type as SessionEvent['type'],
|
||||
seq: row.seq,
|
||||
time: row.time,
|
||||
data: JSON.parse(row.data) as SessionEvent['data'],
|
||||
...surfaceFields,
|
||||
} as SessionEvent
|
||||
}
|
||||
|
||||
/**
|
||||
* The preserved prefix of an ordered event-row list (mirrors the JSONL
|
||||
* backend's `scanLog`): the longest prefix of complete, seq-contiguous,
|
||||
* parseable rows, PLUS the seq from which a never-committed torn tail must be
|
||||
* deleted (or `undefined` if the whole list is intact).
|
||||
*
|
||||
* A crash can leave a durable log whose final turn never closed: real,
|
||||
* fully-written rows sit after the last `turn/end`. Those are PRESERVED — a
|
||||
* single turn can be huge in a long-horizon task, so truncating it would
|
||||
* destroy real work; the backend closes the orphaned open turn with a synthetic
|
||||
* `turn/end {kind:'interrupted'}` on load (the session-persistence RFC). The ONLY thing excluded is
|
||||
* a torn trailing fragment — a row whose `data` never parses, or a seq gap —
|
||||
* AFTER the last committed `turn/end`; that bounds the preserved region and its
|
||||
* seq is returned as `tornFrom` so `load` can physically delete it.
|
||||
*
|
||||
* The last `turn/end` is computed from the `type` COLUMN (never parsing tail
|
||||
* `data`), so a malformed `data` in an uncommitted tail row is discarded rather
|
||||
* than making the session unloadable. A parse error or seq gap AT OR BEFORE the
|
||||
* last committed `turn/end` is committed-data corruption and throws.
|
||||
*
|
||||
* This relies on the session-log invariant that every event lives inside a turn
|
||||
* (`Session.append` enforces it): only the final turn can be open, so the
|
||||
* preserved tail is at most one unclosed turn.
|
||||
*/
|
||||
export function scanRows(rows: readonly EventRow[]): { preserved: SessionEvent[]; tornFrom?: number } {
|
||||
// Pass 1: parse each row's data; a row whose data is not valid JSON is a hole.
|
||||
// (The seq/type COLUMNS are always present even when `data` is corrupt.)
|
||||
interface Parsed { ok: boolean; event?: SessionEvent }
|
||||
const parsed: Parsed[] = rows.map((row) => {
|
||||
try {
|
||||
return { ok: true, event: rowToEvent(row) }
|
||||
} catch {
|
||||
return { ok: false }
|
||||
}
|
||||
})
|
||||
|
||||
// The last index that is a valid `turn/end` — the last fully-committed
|
||||
// boundary (the loop flushes only at turn/end).
|
||||
let lastTurnEnd = -1
|
||||
for (let i = parsed.length - 1; i >= 0; i--) {
|
||||
if (parsed[i]?.ok && rows[i]?.type === 'turn/end') { lastTurnEnd = i; break }
|
||||
}
|
||||
|
||||
// Walk the longest PREFIX of complete, seq-contiguous, parseable rows
|
||||
// (row i has seq === i). This includes the fully-written rows of an
|
||||
// interrupted final turn AFTER the last turn/end — real work, never
|
||||
// truncated. The walk stops at the first hole:
|
||||
// - at or before the last committed turn/end → committed corruption (throw);
|
||||
// - after it (or no committed turn/end) → tolerated torn tail (stop).
|
||||
const preserved: SessionEvent[] = []
|
||||
for (let i = 0; i < rows.length; i++) {
|
||||
const p = parsed[i]
|
||||
if (!p?.ok || p.event === undefined) {
|
||||
if (i <= lastTurnEnd) throw new Error(`corrupt session log: unparsable committed event at seq ${rows[i]?.seq}`)
|
||||
break // torn tail fragment after the last turn/end — stop, tolerate
|
||||
}
|
||||
if (p.event.seq !== i) {
|
||||
if (i <= lastTurnEnd) throw new Error(`corrupt session log: seq gap in committed region (expected ${i}, got ${p.event.seq})`)
|
||||
break // gap after the last turn/end — torn tail, stop
|
||||
}
|
||||
preserved.push(p.event)
|
||||
}
|
||||
|
||||
// Any rows past the preserved prefix are a never-committed torn tail; their
|
||||
// first seq is the deletion point for load's physical repair.
|
||||
return preserved.length < rows.length ? { preserved, tornFrom: preserved.length } : { preserved }
|
||||
}
|
||||
@@ -0,0 +1,451 @@
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import { mkdtemp, rm } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import SessionStore, { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type { Session, SessionEvent, SurfaceEvent, SurfaceEventType } from '@deepseek-ai/dsh-session'
|
||||
import SessionPersistenceSqlite, { SCHEMA_VERSION } from '@deepseek-ai/dsh-session-persistence-sqlite'
|
||||
import { openDatabase, rowToEvent, scanRows, type EventRow } from '../src/schema.ts'
|
||||
import { runPersistenceContract, meta, oneTurnLog } from '../../session-persistence/tests/contract.ts'
|
||||
import { runCoordinatorContract, type CoordinatorFixture } from '../../session-persistence/tests/coordinator-contract.ts'
|
||||
|
||||
const dirs: string[] = []
|
||||
afterEach(async () => { for (const d of dirs.splice(0)) await rm(d, { recursive: true, force: true }) })
|
||||
|
||||
async function freshDbPath(): Promise<string> {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'dsh-sqlite-'))
|
||||
dirs.push(dir)
|
||||
return join(dir, 'sessions.db')
|
||||
}
|
||||
|
||||
/** A context with the session store + SQLite backend, plus a teardown. */
|
||||
async function backend(path = ':memory:'): Promise<{ ctx: Context; dispose: () => Promise<void> }> {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(SessionPersistenceSqlite, { path })
|
||||
return { ctx, dispose: () => fiber.dispose() }
|
||||
}
|
||||
|
||||
// The payoff: the SAME backend-agnostic contract the JSONL backend runs, now
|
||||
// proving the SQLite backend satisfies identical semantics.
|
||||
runPersistenceContract('sqlite', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(SessionPersistenceSqlite, { path: ':memory:' })
|
||||
return {
|
||||
persistence: ctx.sessionPersistence,
|
||||
dispose: async () => { await fiber.dispose() },
|
||||
}
|
||||
})
|
||||
|
||||
// Run the shared coordinator orchestration suite against the real SQLite backend.
|
||||
// A FILE-backed db (not :memory:) is the shared storage scope so two mounted
|
||||
// instances see the same rows (HMR/reload). `corruptTail` INSERTs a row past the
|
||||
// committed seq whose `data` is invalid JSON — a never-committed torn tail that
|
||||
// drives the coordinator's commitRepair-with-tornMarker branch over real db rows.
|
||||
runCoordinatorContract('sqlite', async (): Promise<CoordinatorFixture> => {
|
||||
const dir = await mkdtemp(join(tmpdir(), 'dsh-sqlite-coord-'))
|
||||
const path = join(dir, 'sessions.db')
|
||||
return {
|
||||
mount: async ctx => ctx.plugin(SessionPersistenceSqlite, { path }),
|
||||
corruptTail: async (id) => {
|
||||
// A row past the committed region whose `data` does not parse: scanRows
|
||||
// bounds the preserved prefix at it and returns its seq as tornFrom, which
|
||||
// the backend surfaces to the coordinator as the tornMarker to delete from.
|
||||
const db = openDatabase(path)
|
||||
const next = (db.prepare('SELECT COALESCE(MAX(seq), -1) + 1 AS n FROM events WHERE session_id = ?')
|
||||
.get(id) as { n: number }).n
|
||||
db.prepare('INSERT INTO events (session_id, seq, type, time, data) VALUES (?, ?, ?, ?, ?)')
|
||||
.run(id, next, 'assistant/chunk', 99, '{not valid json')
|
||||
db.close()
|
||||
},
|
||||
cleanup: async () => { await rm(dir, { recursive: true, force: true }) },
|
||||
}
|
||||
})
|
||||
|
||||
describe('scanRows', () => {
|
||||
// scanRows works off EventRows (data is a JSON string column); build them from
|
||||
// SessionEvents so the unit tests read in terms of the event vocabulary.
|
||||
const rows = (events: SessionEvent[]): EventRow[] =>
|
||||
events.map(e => ({ seq: e.seq, type: e.type, time: e.time, data: JSON.stringify(e.data), source_event_seqs: null, surface_op: null }))
|
||||
|
||||
it('preserves the full log when it ends exactly on a turn/end (no torn tail)', () => {
|
||||
const { preserved, tornFrom } = scanRows(rows(oneTurnLog()))
|
||||
expect(preserved).toEqual(oneTurnLog())
|
||||
expect(tornFrom).toBeUndefined()
|
||||
})
|
||||
|
||||
it('PRESERVES the real events of an interrupted turn after the last turn/end', () => {
|
||||
// turn 1 committed (0..5) + a crashed turn 2 (turn/start 6, step/start 7, no
|
||||
// close): all 8 rows are intact, so the whole prefix is preserved and there
|
||||
// is no torn fragment to delete. (load() then synthesizes the closers.)
|
||||
const withOpenTurn: SessionEvent[] = [
|
||||
...oneTurnLog(),
|
||||
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'step/start', seq: 7, time: 8, data: { turn: 2, step: 1 } },
|
||||
]
|
||||
const { preserved, tornFrom } = scanRows(rows(withOpenTurn))
|
||||
expect(preserved.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
expect(tornFrom).toBeUndefined()
|
||||
})
|
||||
|
||||
it('preserves the contiguous prefix and flags a torn tail at a seq gap', () => {
|
||||
// A gap after seq 0 (no committed turn/end): seq 0 is the preserved
|
||||
// interrupted-turn event; the gap bounds it and marks the torn fragment.
|
||||
const gapped: SessionEvent[] = [
|
||||
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'step/start', seq: 2, time: 2, data: { turn: 1, step: 1 } }, // seq 1 missing
|
||||
]
|
||||
const { preserved, tornFrom } = scanRows(rows(gapped))
|
||||
expect(preserved.map(e => e.seq)).toEqual([0])
|
||||
expect(tornFrom).toBe(1)
|
||||
})
|
||||
|
||||
it('an empty log preserves nothing and has no torn tail', () => {
|
||||
expect(scanRows([])).toEqual({ preserved: [] })
|
||||
})
|
||||
|
||||
it('throws on a seq gap inside the committed region (before the last turn/end)', () => {
|
||||
const gapped: SessionEvent[] = [
|
||||
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'step/start', seq: 2, time: 2, data: { turn: 1, step: 1 } }, // seq 1 missing
|
||||
{ type: 'turn/end', seq: 3, time: 3, data: { turn: 1, reason: { kind: 'completed' } } },
|
||||
]
|
||||
expect(() => scanRows(rows(gapped))).toThrow(/seq gap in committed region/)
|
||||
})
|
||||
|
||||
it('throws on an unparsable row inside the committed region', () => {
|
||||
const withCorruptCommitted: EventRow[] = [
|
||||
{ seq: 0, type: 'turn/start', time: 1, data: '{not json', source_event_seqs: null, surface_op: null }, // corrupt, sits before a turn/end
|
||||
{ seq: 1, type: 'turn/end', time: 2, data: JSON.stringify({ turn: 1, reason: { kind: 'completed' } }), source_event_seqs: null, surface_op: null },
|
||||
]
|
||||
expect(() => scanRows(withCorruptCommitted)).toThrow(/unparsable committed event/)
|
||||
})
|
||||
|
||||
it('tolerates an unparsable torn-tail row after the last turn/end', () => {
|
||||
const withCorruptTail: EventRow[] = [
|
||||
...rows(oneTurnLog()),
|
||||
{ seq: 6, type: 'turn/start', time: 7, data: '{not json', source_event_seqs: null, surface_op: null }, // torn fragment, no committed turn/end after
|
||||
]
|
||||
const { preserved, tornFrom } = scanRows(withCorruptTail)
|
||||
expect(preserved).toEqual(oneTurnLog())
|
||||
expect(tornFrom).toBe(6)
|
||||
})
|
||||
})
|
||||
|
||||
describe('SessionPersistenceSqlite: durability and crash semantics', () => {
|
||||
it('an interrupted turn (rows after the last turn/end) is PRESERVED and closed during load', async () => {
|
||||
const path = await freshDbPath()
|
||||
const m = meta('crash')
|
||||
// Run 1: persist a complete turn, then a half-written second turn (no turn/end).
|
||||
const ctx1 = new Context()
|
||||
await ctx1.plugin(SessionStore)
|
||||
const fiber1 = await ctx1.plugin(SessionPersistenceSqlite, { path })
|
||||
await ctx1.sessionPersistence.create(m)
|
||||
await ctx1.sessionPersistence.append(m.id, oneTurnLog())
|
||||
await ctx1.sessionPersistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'step/start', seq: 7, time: 8, data: { turn: 2, step: 1 } },
|
||||
])
|
||||
await fiber1.dispose()
|
||||
|
||||
// Run 2: load PRESERVES the interrupted turn's real events (a turn can be huge
|
||||
// — never truncated) and closes the orphaned turn with synthetic boundary
|
||||
// events: step/end (the step was open) then turn/end {interrupted}.
|
||||
const ctx2 = new Context()
|
||||
await ctx2.plugin(SessionStore)
|
||||
const fiber2 = await ctx2.plugin(SessionPersistenceSqlite, { path })
|
||||
const loaded = await ctx2.sessionPersistence.load(m.id)
|
||||
expect(loaded.events.map(e => e.type)).toEqual([
|
||||
'turn/start', 'user/message', 'step/start', 'assistant/message', 'step/end', 'turn/end', // turn 1
|
||||
'turn/start', 'step/start', 'step/end', 'turn/end', // turn 2: real events + synthetic closers
|
||||
])
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
|
||||
const last = loaded.events.at(-1)!
|
||||
expect(last.type === 'turn/end' && last.data.reason).toEqual({ kind: 'interrupted' })
|
||||
|
||||
// load durably closed the turn, so the next append continues at the balanced
|
||||
// length (seq 10) and a reload round-trips identically.
|
||||
await ctx2.sessionPersistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 10, time: 9, data: { turn: 3, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'turn/end', seq: 11, time: 10, data: { turn: 3, reason: { kind: 'completed' } } },
|
||||
])
|
||||
const reloaded = await ctx2.sessionPersistence.load(m.id)
|
||||
expect(reloaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
|
||||
await fiber2.dispose()
|
||||
})
|
||||
|
||||
it('load() durably closes the interrupted turn: the synthetic closers are on disk after load', async () => {
|
||||
const path = await freshDbPath()
|
||||
const m = meta('load-closes')
|
||||
const b1 = await backend(path)
|
||||
await b1.ctx.sessionPersistence.create(m)
|
||||
await b1.ctx.sessionPersistence.append(m.id, oneTurnLog()) // seqs 0..5
|
||||
await b1.dispose()
|
||||
// Hand-write an interrupted turn (turn/start seq 6, no turn/end).
|
||||
const db = openDatabase(path)
|
||||
db.prepare('INSERT INTO events (session_id, seq, type, time, data) VALUES (?, 6, ?, 7, ?)')
|
||||
.run(m.id, 'turn/start', JSON.stringify({ turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } }))
|
||||
db.close()
|
||||
|
||||
const b2 = await backend(path)
|
||||
const loaded = await b2.ctx.sessionPersistence.load(m.id)
|
||||
// turn 2's real turn/start (seq 6) is preserved + a synthetic turn/end (seq 7).
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
expect(loaded.events.at(-1)!.type).toBe('turn/end')
|
||||
// load() is mutating: the synthetic turn/end MUST be on disk so the stored log
|
||||
// is balanced and the cursor is truthful (contract: load closes, not defers).
|
||||
const probe = openDatabase(path)
|
||||
const stored = probe.prepare('SELECT seq, type FROM events WHERE session_id = ? ORDER BY seq').all(m.id) as { seq: number; type: string }[]
|
||||
probe.close()
|
||||
expect(stored.map(r => r.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
expect(stored.at(-1)!.type).toBe('turn/end')
|
||||
await b2.dispose()
|
||||
})
|
||||
|
||||
it('all-tail load: a session whose only turn never closed is preserved and closed on load', async () => {
|
||||
const path = await freshDbPath()
|
||||
const m = meta('all-tail')
|
||||
const b1 = await backend(path)
|
||||
await b1.ctx.sessionPersistence.create(m)
|
||||
// A first turn that NEVER completed: turn/start + user/message, no turn/end.
|
||||
await b1.ctx.sessionPersistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 0, time: 1, data: { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'user/message', seq: 1, time: 2, data: { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } } },
|
||||
])
|
||||
await b1.dispose()
|
||||
|
||||
// A fresh backend loads it: the interrupted (only) turn's real events are
|
||||
// preserved and closed with a synthetic turn/end {interrupted} — NOT
|
||||
// truncated. The session was materialized, so list() reports it present.
|
||||
const b2 = await backend(path)
|
||||
const loaded = await b2.ctx.sessionPersistence.load(m.id)
|
||||
expect(loaded.events.map(e => e.type)).toEqual(['turn/start', 'user/message', 'turn/end'])
|
||||
expect(loaded.events.at(-1)!.type === 'turn/end' && loaded.events.at(-1)!.data).toMatchObject({ reason: { kind: 'interrupted' } })
|
||||
expect((await b2.ctx.sessionPersistence.list()).map(x => x.id)).toContain(m.id)
|
||||
await b2.dispose()
|
||||
})
|
||||
|
||||
it('rejects opening a database whose schema version is not the current build (newer OR older)', async () => {
|
||||
const path = await freshDbPath()
|
||||
openDatabase(path).close() // stamp user_version = SCHEMA_VERSION
|
||||
// Bump user_version past what this build supports.
|
||||
const dbNewer = openDatabase(path)
|
||||
dbNewer.exec(`PRAGMA user_version = ${SCHEMA_VERSION + 1}`)
|
||||
dbNewer.close()
|
||||
expect(() => openDatabase(path)).toThrow(/incompatible with this build/)
|
||||
|
||||
// A stale OLDER version (e.g. a pre-summary-drop v1 DB) is also rejected —
|
||||
// we do not migrate (unreleased software, no backward-compat).
|
||||
const olderPath = await freshDbPath()
|
||||
openDatabase(olderPath).close()
|
||||
const dbOlder = openDatabase(olderPath)
|
||||
dbOlder.exec('PRAGMA user_version = 1')
|
||||
dbOlder.close()
|
||||
expect(() => openDatabase(olderPath)).toThrow(/incompatible with this build/)
|
||||
})
|
||||
|
||||
it('a corrupt-JSON row in the uncommitted tail is discarded on load, not unloadable', async () => {
|
||||
const path = await freshDbPath()
|
||||
const m = meta('corrupt-tail')
|
||||
const b1 = await backend(path)
|
||||
await b1.ctx.sessionPersistence.create(m)
|
||||
await b1.ctx.sessionPersistence.append(m.id, oneTurnLog()) // committed: seqs 0..5
|
||||
await b1.dispose()
|
||||
|
||||
// Hand-insert a torn tail row (seq 6, no closing turn/end) whose `data` is
|
||||
// invalid JSON. The contract: only a parse error in the COMMITTED region is
|
||||
// unloadable; a torn tail must be discarded. scanRows finds the last
|
||||
// turn/end on the seq+type columns (never parsing tail `data`), so the
|
||||
// unparsable row after it bounds the preserved prefix and is deleted by load.
|
||||
const db = openDatabase(path)
|
||||
db.prepare('INSERT INTO events (session_id, seq, type, time, data) VALUES (?, 6, ?, 7, ?)')
|
||||
.run(m.id, 'turn/start', '{not valid json')
|
||||
db.close()
|
||||
|
||||
const b2 = await backend(path)
|
||||
const loaded = await b2.ctx.sessionPersistence.load(m.id)
|
||||
expect(loaded.events).toEqual(oneTurnLog()) // torn tail discarded, committed intact (turn 1 already balanced → no closers)
|
||||
// load physically deleted the corrupt tail row, so a fresh append continues.
|
||||
await b2.ctx.sessionPersistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 6, time: 8, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'turn/end', seq: 7, time: 9, data: { turn: 2, reason: { kind: 'completed' } } },
|
||||
])
|
||||
const reloaded = await b2.ctx.sessionPersistence.load(m.id)
|
||||
expect(reloaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
await b2.dispose()
|
||||
})
|
||||
|
||||
it('append rolls back the whole batch on a mid-batch seq collision (transaction)', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(SessionPersistenceSqlite, { path: ':memory:' })
|
||||
const m = meta('rollback')
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, oneTurnLog()) // seqs 0..5
|
||||
|
||||
// A batch that re-states an already-stored seq must be rejected and leave
|
||||
// the stored log unchanged (the UNIQUE (session_id, seq) constraint fires
|
||||
// inside the transaction → ROLLBACK).
|
||||
await expect(ctx.sessionPersistence.append(m.id, oneTurnLog())).rejects.toThrow()
|
||||
const loaded = await ctx.sessionPersistence.load(m.id)
|
||||
expect(loaded.events).toEqual(oneTurnLog()) // unchanged
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('persists across separate backend instances over the same file', async () => {
|
||||
const path = await freshDbPath()
|
||||
const m = meta('persist', '/proj')
|
||||
const ctx1 = new Context()
|
||||
await ctx1.plugin(SessionStore)
|
||||
const fiber1 = await ctx1.plugin(SessionPersistenceSqlite, { path })
|
||||
await ctx1.sessionPersistence.create(m)
|
||||
await ctx1.sessionPersistence.append(m.id, oneTurnLog())
|
||||
await fiber1.dispose()
|
||||
|
||||
const ctx2 = new Context()
|
||||
await ctx2.plugin(SessionStore)
|
||||
const fiber2 = await ctx2.plugin(SessionPersistenceSqlite, { path })
|
||||
expect((await ctx2.sessionPersistence.list()).map(x => x.id)).toContain(m.id)
|
||||
const loaded = await ctx2.sessionPersistence.load(m.id)
|
||||
expect(loaded.meta).toMatchObject({ id: m.id, cwd: '/proj' })
|
||||
expect(loaded.events).toEqual(oneTurnLog())
|
||||
await fiber2.dispose()
|
||||
})
|
||||
|
||||
it('exposes the schema version constant', () => {
|
||||
expect(SCHEMA_VERSION).toBe(2)
|
||||
})
|
||||
})
|
||||
|
||||
describe('SessionPersistenceSqlite: edge cases', () => {
|
||||
it('append rolls back and rethrows when an event INSERT fails inside the transaction', async () => {
|
||||
const path = await freshDbPath()
|
||||
const m = meta('rollback-insert')
|
||||
const b1 = await backend(path)
|
||||
await b1.ctx.sessionPersistence.create(m)
|
||||
await b1.ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
|
||||
// A SECOND backend over the same file loads the session first, so it adopts
|
||||
// cursor 6 (the committed length) into its OWN in-memory state.
|
||||
const b2 = await backend(path)
|
||||
await b2.ctx.sessionPersistence.load(m.id) // cursor 6 in b2
|
||||
const turn2: SessionEvent[] = [
|
||||
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'turn/end', seq: 7, time: 8, data: { turn: 2, reason: { kind: 'completed' } } },
|
||||
]
|
||||
// b1 commits seq 6..7 first.
|
||||
await b1.ctx.sessionPersistence.append(m.id, turn2)
|
||||
// b2 still thinks its cursor is 6, so this batch passes the contiguity check
|
||||
// but its INSERT of seq 6 hits the UNIQUE (session_id, seq) constraint
|
||||
// mid-transaction → ROLLBACK + rethrow.
|
||||
await expect(b2.ctx.sessionPersistence.append(m.id, turn2)).rejects.toThrow(/UNIQUE/)
|
||||
// b1's turn is intact; b2's rolled-back attempt left nothing extra.
|
||||
const loaded = await b1.ctx.sessionPersistence.load(m.id)
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
await b1.dispose()
|
||||
await b2.dispose()
|
||||
})
|
||||
|
||||
it('HMR: a DIFFERENT session colliding with a materialized on-disk id is rejected', async () => {
|
||||
const path = await freshDbPath()
|
||||
// Instance 1 materializes a session and disposes.
|
||||
const b1 = await backend(path)
|
||||
const s1 = b1.ctx.sessions.create(SessionId('hmr-collide'))
|
||||
for (const e of oneTurnLog()) s1.append(e.type, e.data)
|
||||
await b1.ctx.parallel('session/flush', s1)
|
||||
await b1.dispose()
|
||||
|
||||
// A fresh context with an UNRELATED live session reusing the id meets a
|
||||
// materialized row that is NOT a prefix of its events → reject.
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
let session!: Session
|
||||
await ctx.plugin(Object.assign((inner: Context) => {
|
||||
session = inner.sessions.create(SessionId('hmr-collide'))
|
||||
}, { inject: ['sessions'] }))
|
||||
session.append('turn/start', { turn: 9, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
await ctx.plugin(SessionPersistenceSqlite, { path })
|
||||
await expect(ctx.parallel('session/flush', session)).rejects.toThrow(/id collision/)
|
||||
await ctx.fiber.dispose()
|
||||
})
|
||||
})
|
||||
|
||||
describe('surface field round-trip', () => {
|
||||
it('rowToEvent parses surface fields from EventRow columns', () => {
|
||||
const row: EventRow = {
|
||||
seq: 0, type: 'assistant/message', time: 1,
|
||||
data: JSON.stringify({ turn: 1, step: 1, content: [] }),
|
||||
source_event_seqs: JSON.stringify([3, 5]),
|
||||
surface_op: JSON.stringify('append'),
|
||||
}
|
||||
const event = rowToEvent(row)
|
||||
expect((event as SurfaceEvent).sourceEventSeqs).toEqual([3, 5])
|
||||
expect((event as SurfaceEvent).surfaceOp).toBe('append')
|
||||
})
|
||||
|
||||
it('rowToEvent handles replace surfaceOp object', () => {
|
||||
const row: EventRow = {
|
||||
seq: 0, type: 'assistant/message', time: 1,
|
||||
data: JSON.stringify({ turn: 1, step: 1, content: [] }),
|
||||
source_event_seqs: JSON.stringify([0, 1]),
|
||||
surface_op: JSON.stringify({ op: 'replace', start: 0, end: 1 }),
|
||||
}
|
||||
const event = rowToEvent(row)
|
||||
expect((event as SurfaceEvent).sourceEventSeqs).toEqual([0, 1])
|
||||
expect((event as SurfaceEvent).surfaceOp).toEqual({ op: 'replace', start: 0, end: 1 })
|
||||
})
|
||||
|
||||
it('scanRows with surface columns reconstructs events with surface fields', () => {
|
||||
const rows: EventRow[] = [
|
||||
{ seq: 0, type: 'user/message', time: 1,
|
||||
data: JSON.stringify({ content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }),
|
||||
source_event_seqs: null, surface_op: '{"op":"replace","start":0,"end":0}' },
|
||||
{ seq: 1, type: 'turn/end', time: 2,
|
||||
data: JSON.stringify({ turn: 1, reason: { kind: 'completed' } }),
|
||||
source_event_seqs: null, surface_op: null },
|
||||
]
|
||||
const { preserved } = scanRows(rows)
|
||||
expect(preserved).toHaveLength(2)
|
||||
expect((preserved[0]! as SurfaceEvent).surfaceOp).toEqual({ op: 'replace', start: 0, end: 0 })
|
||||
expect((preserved[0]! as SurfaceEvent).sourceEventSeqs).toBeUndefined()
|
||||
expect((preserved[1] as SessionEvent<SurfaceEventType>).surfaceOp).toBeUndefined()
|
||||
})
|
||||
|
||||
it('append and load round-trips surface fields through SQLite', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(SessionPersistenceSqlite, { path: ':memory:' })
|
||||
const session = ctx.sessions.create(SessionId('roundtrip-surface'))
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } }, { surfaceOp: 'append' })
|
||||
session.append('assistant/message', { turn: 1, step: 1, content: [] }, { surfaceOp: 'append', sourceEventSeqs: [0] })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
await ctx.parallel('session/flush', session)
|
||||
const loaded = await ctx.sessionPersistence.load(SessionId('roundtrip-surface'))
|
||||
expect(loaded.events).toHaveLength(4)
|
||||
const um = loaded.events[1]!
|
||||
expect((um as SurfaceEvent).surfaceOp).toBe('append')
|
||||
expect((um as SurfaceEvent).sourceEventSeqs).toBeUndefined()
|
||||
const am = loaded.events[2]!
|
||||
expect((am as SurfaceEvent).surfaceOp).toBe('append')
|
||||
expect((am as SurfaceEvent).sourceEventSeqs).toEqual([0])
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('persists events with surfaceOp but no sourceEventSeqs (covers null branch in surfaceBindings)', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(SessionPersistenceSqlite, { path: ':memory:' })
|
||||
const session = ctx.sessions.create(SessionId('surface-noseq'))
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('steering/message', { turn: 1, content: [], source: { kind: 'user' } }, { surfaceOp: 'append' })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
await ctx.parallel('session/flush', session)
|
||||
const loaded = await ctx.sessionPersistence.load(SessionId('surface-noseq'))
|
||||
expect((loaded.events[1]! as SurfaceEvent).surfaceOp).toBe('append')
|
||||
expect((loaded.events[1]! as SurfaceEvent).sourceEventSeqs).toBeUndefined()
|
||||
await fiber.dispose()
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
{
|
||||
"path": "../../session-persistence/session-persistence"
|
||||
}
|
||||
]
|
||||
}
|
||||
51
packages/session-persistence/session-persistence/README.md
Normal file
51
packages/session-persistence/session-persistence/README.md
Normal file
@@ -0,0 +1,51 @@
|
||||
# @deepseek-ai/dsh-session-persistence
|
||||
|
||||
The abstract durable session-persistence seam (`ctx.sessionPersistence`). Defines WHAT a persistence backend does — durably store, reload, and list sessions — without saying HOW. Mirrors the `dsh-bash` capability-seam template ([capability seams](../../../docs/rfc/implemented/architecture/2026-06-13-capability-seams.md)): an abstract service here, a concrete implementation in a sibling package, consumers that inject the interface.
|
||||
|
||||
The persisted unit IS the existing `SessionEvent` (event-sourced model — the log is the single source of truth), so there is no parallel "persisted message" type. Metadata that is NOT replayable conversation state (format version, cwd, lineage) travels separately as `SessionHeader`, owned by `dsh-session` and re-exported here.
|
||||
|
||||
## Service API (`ctx.sessionPersistence`)
|
||||
|
||||
| Method | Contract |
|
||||
|---|---|
|
||||
| `create(meta): Promise<void>` | Register a new session's metadata. MAY defer the physical write until the first `append` (lazy materialization). |
|
||||
| `append(id, events): Promise<void>` | Durably persist a batch (from the `session/flush` drain). Append-only; first event `seq` == stored next-seq after any repair; rejects non-JSON-serializable data naming the offending type. |
|
||||
| `load(id): Promise<{ meta; events }>` | Reload meta + log. Preserves an interrupted (unclosed) final turn and closes it with synthetic closers — an error `tool/result` per unanswered `tool-call`, then `step/end?`+`turn/end {interrupted}` (a turn can be huge — never truncated); only a torn tail fragment is dropped. Events contiguous (`events[i].seq === i`); rejects a committed-region gap/parse error or unknown `version`. |
|
||||
| `list(): Promise<SessionHeader[]>` | Lightweight listing from metadata, no full-log parse. A zero-event lazily-materialized session is absent from `list`. |
|
||||
|
||||
## Invariants every backend must honor
|
||||
|
||||
- **Append-only; a crashed turn is closed, not truncated.** Committed events (at or below a flushed `turn/end`) are never rewritten. A crash can leave an unclosed final turn whose events are real and possibly large; `load` preserves them and durably appends synthetic closers (an error `tool/result` per unanswered `tool-call`, then `step/end?`+`turn/end {interrupted}`) to balance the log and keep the rehydrated history a valid provider transcript. Only a never-fully-written torn tail fragment is discarded.
|
||||
- **Contiguous seq.** `load` rejects a `seq` gap/parse error in the MIDDLE of the log; `append`'s first `seq` must equal the stored next-seq.
|
||||
- **JSON-serializable data.** `append` rejects non-serializable `event.data`; backends snapshot each event when buffering (the live `session.events` object is mutable).
|
||||
- **Durability.** `append` returns only once the batch is durable.
|
||||
|
||||
## The write coordinator
|
||||
|
||||
The two first-party backends were byte-identical (or same-algorithm) for ALL of their write-path orchestration — the in-memory bookkeeping (per-id state, write-behind buffers, per-id serialization chains, per-session init promises), the `session/event` → buffer → `session/flush` drain, lazy materialization, crash-tail repair on load, the four `session/created` adoption cases (new / HMR-adopt / collision / ownerless-claim), and dispose-time quiescence. Only the STORAGE primitives differed (write bytes vs. INSERT rows).
|
||||
|
||||
`PersistenceCoordinator` owns that orchestration once. A first-party backend composes one (`new PersistenceCoordinator(ctx, this)`), implements the small `PersistenceBackend` hook interface, and delegates its four public service methods to the coordinator. This keeps the duplicated, correctness-heavy orchestration in a single place (it used to receive the same fixes twice).
|
||||
|
||||
The `PersistenceBackend<TornMarker>` hooks (the only seam between the coordinator and storage):
|
||||
|
||||
| Hook | Role |
|
||||
|---|---|
|
||||
| `name` | Backend label for the dispose-failure `AggregateError`. |
|
||||
| `loadStored(id)` | Read a stored prefix by id, scanning ANY storage scope. Used by resume/load and, via `!== undefined`, the create-collision probe. Returns an opaque `tornMarker` iff a torn tail must be truncated. |
|
||||
| `loadLive(id, cwd)` | Read a stored prefix SCOPED to `cwd` (HMR live-adoption must only adopt a log at the SAME cwd; a same-id log elsewhere is a collision, not a resume). A globally-unique-id backend ignores `cwd`. |
|
||||
| `appendBatch(meta, events, isMaterialized)` | Durably append a contiguous batch, lazily materializing ATOMICALLY when not yet materialized. |
|
||||
| `commitRepair(meta, tornMarker, closers)` | Make a crash repair durable: truncate the torn tail (iff `tornMarker !== undefined` — a marker may be falsy, e.g. seq/offset `0`) and append `closers`. NOT required to be atomic. Used by load (truncate + closers) and live-adoption (truncate only). |
|
||||
| `list()` | List all stored metadata. |
|
||||
| `close?()` | Optional lifecycle teardown (e.g. close a db handle), awaited after the dispose drain. |
|
||||
|
||||
The `tornMarker` is fully OPAQUE: the coordinator only tests `!== undefined` and round-trips it to `commitRepair`, never inspecting its value (the JSONL backend uses the byte offset to truncate to, the SQLite backend the seq to delete from). The public `SessionPersistence` service shape is unchanged, so a third-party backend MAY still implement the abstract service directly without the coordinator. See [the write-coordinator RFC](../../../docs/rfc/implemented/architecture/2026-06-18-shared-persistence-write-coordinator.md).
|
||||
|
||||
## Testing backends
|
||||
|
||||
Import `runPersistenceContract` from `tests/contract.ts` (the public-API contract) and `runCoordinatorContract` from `tests/coordinator-contract.ts` (the shared write-path orchestration: adoption, HMR, collision, dispose-drain, crash-tail repair) and call each with a fixture for your backend. Every backend is held to the same append-only / contiguous-seq / lazy-materialization / serializability semantics AND the same orchestration, so a backend's own spec is left with only storage-mechanics tests (path sanitization, fsync rollback; schema version, transaction rollback) on top.
|
||||
|
||||
Three backends run these suites: an in-memory reference (in `tests/`), `dsh-session-persistence-jsonl` (append-only file log) and `dsh-session-persistence-sqlite` (`node:sqlite`, each `SessionEvent` one row `(session_id, seq, type, time, data)`). All passing the same contract + coordinator suite is the proof that the seam is genuinely backend-agnostic — lazy materialization, crash-tail-on-load, and contiguous-seq hold identically over file bytes and over a transactional store.
|
||||
|
||||
## Metadata types
|
||||
|
||||
Re-exported from `dsh-session`: `SessionHeader` (immutable session metadata: `version`, `id`, `createdAt`, `cwd?`, `parentSession?`).
|
||||
@@ -0,0 +1,549 @@
|
||||
/**
|
||||
* The backend-agnostic write-path orchestration shared by every first-party
|
||||
* {@link SessionPersistence} backend.
|
||||
*
|
||||
* Every durable backend needs the same orchestration: the in-memory bookkeeping
|
||||
* (the per-id state, the write-behind buffers, the per-id serialization chains,
|
||||
* the per-session init promises), the `session/event` → buffer → `session/flush`
|
||||
* drain, lazy materialization, crash-tail repair on load, the four
|
||||
* `session/created` adoption cases (new / HMR-adopt / collision /
|
||||
* ownerless-claim), and dispose-time quiescence. Only the STORAGE primitives are
|
||||
* backend-specific (file bytes for `dsh-session-persistence-jsonl`, `node:sqlite`
|
||||
* rows for `dsh-session-persistence-sqlite`). {@link PersistenceCoordinator} owns
|
||||
* the orchestration; a backend supplies the storage primitives as a small
|
||||
* {@link PersistenceBackend} hook object.
|
||||
*
|
||||
* The abstract {@link SessionPersistence} service's public API is independent of
|
||||
* this: a backend IS a `SessionPersistence` (its four public methods delegate to
|
||||
* a coordinator it composes), so a third-party backend MAY implement the service
|
||||
* directly without using the coordinator at all.
|
||||
*
|
||||
* See the write-coordinator RFC (docs/rfc/implemented/architecture/2026-06-18-shared-persistence-write-coordinator.md)
|
||||
* for the design rationale (composition over inheritance, the opaque torn marker).
|
||||
*
|
||||
* @module @deepseek-ai/dsh-session-persistence/coordinator
|
||||
*/
|
||||
|
||||
import { Context } from 'cordis'
|
||||
import { interruptedTurnClosers, SESSION_FORMAT_VERSION } from '@deepseek-ai/dsh-session'
|
||||
import type { Session, SessionEvent, SessionId, SessionHeader } from '@deepseek-ai/dsh-session'
|
||||
import { assertSerializable, seedCoversPrefix } from './index.ts'
|
||||
|
||||
/**
|
||||
* A stored session's durable prefix as read back from a backend: its
|
||||
* {@link SessionHeader}, the preserved (seq-contiguous, parseable) event prefix,
|
||||
* and an OPAQUE `tornMarker` that is present iff a never-committed torn tail must
|
||||
* be truncated before further writes.
|
||||
*
|
||||
* The coordinator NEVER inspects `tornMarker`'s value — it only tests
|
||||
* `!== undefined` (is there a tail to repair?) and passes the value back to
|
||||
* {@link PersistenceBackend.commitRepair}. Each backend chooses its own marker
|
||||
* type: the JSONL backend uses the byte offset to truncate to, the SQLite
|
||||
* backend uses the seq to delete from (both happen to be `number`).
|
||||
*/
|
||||
export interface StoredPrefix<TornMarker = unknown> {
|
||||
meta: SessionHeader
|
||||
events: SessionEvent[]
|
||||
tornMarker?: TornMarker
|
||||
}
|
||||
|
||||
/**
|
||||
* The storage seam between {@link PersistenceCoordinator} and a concrete
|
||||
* backend: the minimal set of durable primitives the orchestration calls. A
|
||||
* backend implements these (over files, rows, an object store, …); the
|
||||
* coordinator supplies everything else (buffering, serialization, cursors,
|
||||
* adoption, crash repair sequencing, dispose quiescence).
|
||||
*
|
||||
* @typeParam TornMarker - the backend's opaque torn-tail repair token (see
|
||||
* {@link StoredPrefix}). The coordinator treats it as fully opaque.
|
||||
*/
|
||||
export interface PersistenceBackend<TornMarker = unknown> {
|
||||
/** Human-readable backend name, used in the dispose-failure AggregateError. */
|
||||
readonly name: string
|
||||
|
||||
/**
|
||||
* Read a stored prefix by id, scanning ANY storage scope (for JSONL: every
|
||||
* cwd bucket). Returns `undefined` if no stored artifact exists. Used by
|
||||
* resume/load, and — via `!== undefined` — by the create-collision probe.
|
||||
* The returned `tornMarker` is present iff there is a torn tail to truncate.
|
||||
*/
|
||||
loadStored(id: SessionId): Promise<StoredPrefix<TornMarker> | undefined>
|
||||
|
||||
/**
|
||||
* Read a stored prefix SCOPED to `cwd`. Deliberately distinct from
|
||||
* {@link loadStored}: HMR live-adoption must only adopt a persisted log at the
|
||||
* SAME cwd as the live session (a same-id log at a different cwd is a
|
||||
* collision, not a resume) — conflating the two reintroduces a cross-cwd
|
||||
* adoption bug. For a globally-unique-id backend (SQLite) `cwd` is ignored.
|
||||
*/
|
||||
loadLive(id: SessionId, cwd: string | undefined): Promise<StoredPrefix<TornMarker> | undefined>
|
||||
|
||||
/**
|
||||
* Durably append a CONTIGUOUS batch, lazily materializing the session first
|
||||
* when `!isMaterialized`. The materialize-write and the first event batch MUST
|
||||
* commit ATOMICALLY (a crash between them must not leave a materialized-but-
|
||||
* empty session). Returns once the batch is durable.
|
||||
*/
|
||||
appendBatch(meta: SessionHeader, events: readonly SessionEvent[], isMaterialized: boolean): Promise<void>
|
||||
|
||||
/**
|
||||
* Make a crash repair durable: truncate the torn tail (iff
|
||||
* `tornMarker !== undefined`) and append `closers` (iff any). NOT required to
|
||||
* be atomic — a file backend may truncate-then-append in two fsync'd steps.
|
||||
* Used by load (truncate + synthetic closers) and by live-adoption (truncate
|
||||
* only, `closers = []`).
|
||||
*/
|
||||
commitRepair(meta: SessionHeader, tornMarker: TornMarker | undefined, closers: readonly SessionEvent[]): Promise<void>
|
||||
|
||||
/** List all stored (materialized) sessions' metadata. */
|
||||
list(): Promise<SessionHeader[]>
|
||||
|
||||
/**
|
||||
* Optional lifecycle teardown (e.g. close a database handle). Awaited by the
|
||||
* coordinator's dispose effect AFTER the quiescence drain. A stateless file
|
||||
* backend omits it.
|
||||
*/
|
||||
close?(): Promise<void>
|
||||
}
|
||||
|
||||
/** Per-session write state held by the coordinator's in-memory bookkeeping. */
|
||||
interface SessionState {
|
||||
meta: SessionHeader
|
||||
/** The next seq the backend expects to append (the stored log length). */
|
||||
cursor: number
|
||||
/**
|
||||
* Whether the backend has physically written this session (a JSONL file /
|
||||
* SQLite row exists). `create()` registers state LAZILY — cursor 0,
|
||||
* materialized false, nothing on disk — so an empty session leaves no
|
||||
* artifact and the FIRST `appendBatch` writes the header + its events in ONE
|
||||
* transaction (the "a row exists ⇔ it has events" invariant `list`
|
||||
* relies on; a separate up-front materialize could crash leaving a row with
|
||||
* zero events). The flag is the only signal that distinguishes a session
|
||||
* registered-but-never-written from one durably present, which the reclaim
|
||||
* path needs (an abandoned id with no artifact AND no buffered events is free
|
||||
* to reuse; a materialized one is a real collision).
|
||||
*/
|
||||
materialized: boolean
|
||||
/**
|
||||
* The live Session this state was bound to via `onCreated`, if any. State
|
||||
* created through the public `create()`/`load()` API has no owner; state bound
|
||||
* to a live session lets `onCreated` reject a second, unrelated session on the
|
||||
* same id (a collision) instead of silently no-opping.
|
||||
*/
|
||||
owner?: Session
|
||||
}
|
||||
|
||||
/** Collect the rejection reasons from a set of promises (none-throwing). */
|
||||
async function settledErrors(promises: Iterable<Promise<unknown>>): Promise<unknown[]> {
|
||||
const settled = await Promise.allSettled([...promises])
|
||||
const errors: unknown[] = []
|
||||
for (const result of settled) {
|
||||
if (result.status === 'rejected') errors.push(result.reason)
|
||||
}
|
||||
return errors
|
||||
}
|
||||
|
||||
/**
|
||||
* Owns the backend-agnostic session write-path orchestration. A backend
|
||||
* constructs one (`new PersistenceCoordinator(ctx, this)`), implements
|
||||
* {@link PersistenceBackend}, and delegates its four public service methods to
|
||||
* the matching coordinator methods.
|
||||
*
|
||||
* All per-id operations are serialized (a per-id promise chain) so concurrent
|
||||
* flushes / a flush racing a load never interleave storage writes. The
|
||||
* constructor installs the write-path listeners and the dispose effect.
|
||||
*
|
||||
* @typeParam TornMarker - the backend's opaque torn-tail repair token.
|
||||
*/
|
||||
export class PersistenceCoordinator<TornMarker = unknown> {
|
||||
/** Backend bookkeeping keyed by session id (NOT the live Session object). */
|
||||
private states = new Map<SessionId, SessionState>()
|
||||
/** Write-behind buffers keyed by the live Session (write path). */
|
||||
private buffers = new Map<Session, SessionEvent[]>()
|
||||
/**
|
||||
* Per-session serialization: every operation chains onto the prior one for the
|
||||
* same id, so writes for one session never interleave. Keyed by session id.
|
||||
*/
|
||||
private chains = new Map<SessionId, Promise<unknown>>()
|
||||
/**
|
||||
* Per-session init promise (onCreated). Keyed by the LIVE Session OBJECT, not
|
||||
* its id: a disposed fiber's session can be replaced by a different live
|
||||
* Session reusing the same id (HMR, an ACP reconnect), and an id-keyed cache
|
||||
* would hand the new object the old object's init promise.
|
||||
*
|
||||
* Public (readonly) so a backend can expose it for white-box tests that await
|
||||
* a specific session's init (there is no public API to await one init); the
|
||||
* coordinator itself only ever mutates it internally.
|
||||
*/
|
||||
readonly inits = new Map<Session, Promise<void>>()
|
||||
|
||||
constructor(private ctx: Context, private backend: PersistenceBackend<TornMarker>) {
|
||||
this.installWritePath()
|
||||
}
|
||||
|
||||
// --- public surface (the backend's service methods delegate here) ---
|
||||
|
||||
/**
|
||||
* Register a new session's metadata (lazy: no physical write until the first
|
||||
* {@link append}). Rejects if the id is already tracked or already persisted.
|
||||
*/
|
||||
create(meta: SessionHeader): Promise<void> {
|
||||
// Snapshot the metadata at call time: the op runs later (behind the
|
||||
// per-session chain) and the snapshot is stored as the lazy state, so keeping
|
||||
// the caller's object by reference would let a later mutation of `id`/`cwd`
|
||||
// register under one key but materialize under a different path/header.
|
||||
const snapshot: SessionHeader = { ...meta }
|
||||
return this.serialize(snapshot.id, () => this.createCore(snapshot))
|
||||
}
|
||||
|
||||
private async createCore(meta: SessionHeader): Promise<void> {
|
||||
// Do NOT clobber an existing session: the SessionId IS the identity.
|
||||
if (this.states.has(meta.id)) {
|
||||
throw new Error(`session "${meta.id}" already exists in this backend`)
|
||||
}
|
||||
// A persisted artifact under this id (in ANY scope) blocks creation: load/
|
||||
// resume identify a session by id alone, so a second artifact would make
|
||||
// resume nondeterministic.
|
||||
if (await this.backend.loadStored(meta.id) !== undefined) {
|
||||
throw new Error(`session "${meta.id}" already has a persisted log on disk; load/resume it instead of creating`)
|
||||
}
|
||||
// Pure lazy: record intent only. No artifact until the first append.
|
||||
this.states.set(meta.id, { meta, cursor: 0, materialized: false })
|
||||
}
|
||||
|
||||
// `async` so the synchronous validate/clone below reject (not throw) per the
|
||||
// Promise<void> contract — callers use `await expect(...).rejects`.
|
||||
/**
|
||||
* Durably persist a batch of events. Honors the append-only and contiguous-seq
|
||||
* contracts; rejects non-JSON-serializable `event.data`.
|
||||
*/
|
||||
async append(id: SessionId, events: readonly SessionEvent[]): Promise<void> {
|
||||
// Validate serializability BEFORE cloning so a bad event surfaces the typed
|
||||
// error rather than an opaque DataCloneError from structuredClone.
|
||||
assertSerializable(events)
|
||||
// Deep-snapshot the batch HERE, before the op waits behind the per-session
|
||||
// chain: a caller that mutates a live array (e.g. session.events) — or an
|
||||
// event inside it — before the op runs would otherwise have those changes
|
||||
// persisted. The clone is taken synchronously (at call time).
|
||||
const batch = events.map(e => structuredClone(e))
|
||||
return this.serialize(id, () => this.appendCore(id, batch))
|
||||
}
|
||||
|
||||
private async appendCore(id: SessionId, events: readonly SessionEvent[]): Promise<void> {
|
||||
if (events.length === 0) return
|
||||
let state = this.states.get(id)
|
||||
if (state === undefined) state = await this.adopt(id) // calls loadCore, not load
|
||||
|
||||
// Contiguity contract: each event's seq must continue the stored log.
|
||||
for (const [i, event] of events.entries()) {
|
||||
if (event.seq !== state.cursor + i) {
|
||||
throw new Error(`append seq mismatch for "${id}": expected ${state.cursor + i} at index ${i}, got ${event.seq}`)
|
||||
}
|
||||
}
|
||||
|
||||
await this.backend.appendBatch(state.meta, events, state.materialized)
|
||||
// The durable write is the transaction: mark materialized + advance the
|
||||
// cursor as soon as it commits (uniform across backends).
|
||||
state.materialized = true
|
||||
state.cursor += events.length
|
||||
}
|
||||
|
||||
/**
|
||||
* Reload a session: its {@link SessionHeader} plus the event log up to the last
|
||||
* durable checkpoint, with any interrupted final turn durably closed (synthetic
|
||||
* boundary events) during load.
|
||||
*/
|
||||
load(id: SessionId): Promise<{ meta: SessionHeader; events: SessionEvent[] }> {
|
||||
return this.serialize(id, () => this.loadCore(id))
|
||||
}
|
||||
|
||||
private async loadCore(id: SessionId): Promise<{ meta: SessionHeader; events: SessionEvent[] }> {
|
||||
const stored = await this.backend.loadStored(id)
|
||||
if (stored === undefined) throw new Error(`session "${id}" not found`)
|
||||
const { meta, events, tornMarker } = stored
|
||||
this.assertVersion(meta)
|
||||
|
||||
// Crash-recovery: if the log ended mid-turn (real, preserved events but no
|
||||
// closing turn/end), close it durably DURING load so disk, the returned log,
|
||||
// and the cursor all agree. The interrupted turn's real events are preserved,
|
||||
// never truncated (a turn can be huge — the session-persistence RFC); only a
|
||||
// never-fully-written torn tail fragment is discarded.
|
||||
const closers = interruptedTurnClosers(events)
|
||||
const balanced = [...events, ...closers]
|
||||
|
||||
// Make the repair durable (truncate the torn tail + append the synthetic
|
||||
// closers) BEFORE recording state — commitRepair takes `meta` directly, so
|
||||
// there is no state-path ordering dependency (uniform across backends).
|
||||
if (tornMarker !== undefined || closers.length > 0) {
|
||||
await this.backend.commitRepair(meta, tornMarker, closers)
|
||||
}
|
||||
// The state keeps its OWN copy of the meta; the returned value is separate so
|
||||
// a consumer mutating loaded.meta cannot corrupt the backend's metadata.
|
||||
this.states.set(id, { meta: { ...meta }, cursor: balanced.length, materialized: true })
|
||||
return { meta, events: balanced }
|
||||
}
|
||||
|
||||
// NOTE: there is deliberately no coordinator `list()`. Listing needs none of
|
||||
// the coordinator's orchestration (no per-id serialization, no cursor, no
|
||||
// in-memory state) — it is a pure read of stored metadata. A backend's public
|
||||
// `list()` IS the {@link PersistenceBackend.list} hook (one method); routing it
|
||||
// through the coordinator would only forward to that same hook, so the
|
||||
// coordinator stays out of the listing path entirely.
|
||||
|
||||
// --- per-id serialization + adoption helpers ---
|
||||
|
||||
/**
|
||||
* Run `op` after any in-flight operation for the same session id, so writes for
|
||||
* one session never interleave. Errors do not poison the chain. NOTE: serialized
|
||||
* public methods must NOT call each other (deadlock); they call the unserialized
|
||||
* `*Core` helpers instead.
|
||||
*/
|
||||
private serialize<T>(id: SessionId, op: () => Promise<T>): Promise<T> {
|
||||
const prior = this.chains.get(id) ?? Promise.resolve()
|
||||
const next = prior.then(op, op)
|
||||
// Keep the chain alive but swallow this op's rejection for the NEXT waiter
|
||||
// (the caller still sees the real rejection via `next`).
|
||||
this.chains.set(id, next.then(() => undefined, () => undefined))
|
||||
return next
|
||||
}
|
||||
|
||||
/** Build a state for a session discovered in storage but not yet in memory. */
|
||||
private async adopt(id: SessionId): Promise<SessionState> {
|
||||
// loadCore (NOT load) — adopt runs inside an already-serialized op, so
|
||||
// re-entering the chain via the public load() would deadlock.
|
||||
await this.loadCore(id)
|
||||
const state = this.states.get(id)
|
||||
/* v8 ignore next -- loadCore always sets the state for the id */
|
||||
if (!state) throw new Error(`failed to adopt session "${id}"`)
|
||||
return state
|
||||
}
|
||||
|
||||
private assertVersion(meta: SessionHeader): void {
|
||||
if (meta.version !== SESSION_FORMAT_VERSION) {
|
||||
throw new Error(`unsupported session format version ${meta.version} for "${meta.id}" (only v${SESSION_FORMAT_VERSION} is supported)`)
|
||||
}
|
||||
}
|
||||
|
||||
// --- write path (session/event → flush drain) ---
|
||||
|
||||
private installWritePath(): void {
|
||||
const ctx = this.ctx
|
||||
|
||||
// Capture the header on creation; persist a fork's seed once. Record the init
|
||||
// promise so flush/dispose can await it (onCreated is async).
|
||||
ctx.on('session/created', (session) => { void this.initFor(session) })
|
||||
|
||||
// Snapshot + buffer every event (the live object is mutable; clone so a later
|
||||
// in-place mutation cannot rewrite a buffered event). Serializability is
|
||||
// guaranteed at the source (Session.append), so structuredClone is safe.
|
||||
ctx.on('session/event', (session, event) => {
|
||||
let buffer = this.buffers.get(session)
|
||||
if (!buffer) this.buffers.set(session, buffer = [])
|
||||
buffer.push(structuredClone(event))
|
||||
})
|
||||
|
||||
// Drain to the backend at the durability checkpoint.
|
||||
ctx.on('session/flush', session => this.flush(session))
|
||||
|
||||
// Dispose must reach quiescence: await every init + final drain BEFORE
|
||||
// returning, then close the backend's own resources (AFTER the drain), so no
|
||||
// write lands after teardown and a close failure never MASKS a drain error.
|
||||
ctx.effect(() => async () => {
|
||||
let disposeError: unknown
|
||||
try {
|
||||
const errors = [
|
||||
...await settledErrors(this.inits.values()),
|
||||
...await settledErrors([...this.buffers.keys()].map(s => this.flush(s))),
|
||||
...await settledErrors(this.chains.values()),
|
||||
]
|
||||
if (errors.length > 0) {
|
||||
throw new AggregateError(errors, `${this.backend.name} dispose failed`)
|
||||
}
|
||||
} catch (error: unknown) {
|
||||
disposeError = error
|
||||
throw error
|
||||
} finally {
|
||||
try {
|
||||
await this.backend.close?.()
|
||||
} catch (closeError: unknown) {
|
||||
// A close failure can only add teardown context; keep the already-
|
||||
// captured drain AggregateError as the primary failure rather than
|
||||
// masking it. Only surface the close error if the drain succeeded.
|
||||
/* v8 ignore start -- close failure racing disposal is a defensive teardown edge */
|
||||
if (disposeError === undefined) throw closeError
|
||||
/* v8 ignore stop */
|
||||
}
|
||||
}
|
||||
}, `${this.backend.name} write path`)
|
||||
|
||||
// HMR: a hot reload does not replay session/created, so seed existing live
|
||||
// sessions (mirrors dsh-invariants).
|
||||
for (const session of ctx.sessions.list()) void this.initFor(session)
|
||||
}
|
||||
|
||||
/** Start (once) the async init for a session and remember its promise. */
|
||||
private initFor(session: Session): Promise<void> {
|
||||
const existing = this.inits.get(session)
|
||||
if (existing) return existing
|
||||
// Snapshot the seed SYNCHRONOUSLY — initFor runs inside the `session/created`
|
||||
// emit, before any later `append` adds non-seed events. A clone freezes it
|
||||
// against later mutation of the live event objects.
|
||||
const seed = session.events.map(e => structuredClone(e))
|
||||
const p = this.onCreated(session, seed)
|
||||
// Attach a no-op rejection handler so a failing init does not surface as an
|
||||
// unhandled rejection if no flush observes `p` before it rejects. The REAL
|
||||
// error is still delivered: flush/dispose await the same `p` from the map.
|
||||
p.catch(() => { /* observed by flush/dispose via the stored promise */ })
|
||||
this.inits.set(session, p)
|
||||
return p
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a live session's `seed` reproduces the first `cursor` persisted
|
||||
* events. A `cursor` of 0 (nothing persisted yet) trivially matches. Used when
|
||||
* a live session claims ownerless state left by a prior `load()`/`create()`.
|
||||
*/
|
||||
private async seedMatchesPersisted(id: SessionId, seed: readonly SessionEvent[], cursor: number): Promise<boolean> {
|
||||
if (cursor === 0) return true
|
||||
const stored = await this.backend.loadStored(id)
|
||||
/* v8 ignore next -- a cursor > 0 means the session was materialized, so it exists */
|
||||
if (stored === undefined) return false
|
||||
return seedCoversPrefix(seed, stored.events.slice(0, cursor))
|
||||
}
|
||||
|
||||
/**
|
||||
* On session/created: sync the backend's in-memory state to a live Session.
|
||||
*
|
||||
* Cases, by whether this backend tracks the id and whether an artifact exists:
|
||||
* 1. Already tracked → no-op (or claim ownerless state if the seed matches,
|
||||
* or reclaim a truly-abandoned id, else reject as a collision).
|
||||
* 2. Not tracked, an artifact EXISTS at this cwd and is a seq-aligned PREFIX
|
||||
* of the live events → ADOPT it (HMR/reload), persisting any live suffix.
|
||||
* 3. Not tracked, an artifact EXISTS but is NOT a prefix → REJECT (collision).
|
||||
* 4. Not tracked and NO artifact → a genuinely new session: register meta
|
||||
* (lazy) and persist its seed once.
|
||||
*/
|
||||
private async onCreated(session: Session, seed: readonly SessionEvent[]): Promise<void> {
|
||||
const id = session.header.id
|
||||
const tracked = this.states.get(id)
|
||||
if (tracked !== undefined) {
|
||||
// case 1: already tracked.
|
||||
/* v8 ignore next -- initFor dedupes per session object; same-object re-entry can't occur */
|
||||
if (tracked.owner === session) return
|
||||
if (tracked.owner === undefined) {
|
||||
// Ownerless state from the public create()/load() API. The FIRST live
|
||||
// session claims it — but ONLY if BOTH the cwd scope and the seed match.
|
||||
// The cwd guard mirrors case-2's cwd-scoped loadLive(): a same-id
|
||||
// ownerless artifact at a DIFFERENT cwd is a collision, not a claim
|
||||
// (claiming it would append the live cwd's events under the stored
|
||||
// header's cwd, the exact cross-cwd corruption the loadLive scope
|
||||
// prevents). The seed guard then ensures the live events reproduce the
|
||||
// persisted prefix (else a fresh, unrelated session reusing the id would
|
||||
// have its seq 0..cursor-1 events filtered as already-written and
|
||||
// grafted on).
|
||||
if (tracked.meta.cwd !== session.header.cwd) {
|
||||
throw new Error(`session "${id}" is already persisted at a different cwd (persisted: ${String(tracked.meta.cwd)}, live: ${String(session.header.cwd)}) (id collision)`)
|
||||
}
|
||||
if (!await this.seedMatchesPersisted(id, seed, tracked.cursor)) {
|
||||
throw new Error(`session "${id}" is already persisted with ${tracked.cursor} event(s) that do not match this live session (id collision)`)
|
||||
}
|
||||
tracked.owner = session
|
||||
// Persist the seed SUFFIX beyond the persisted prefix. Constructor seed
|
||||
// events never emit session/event, so the buffer never sees them.
|
||||
const suffix = seed.slice(tracked.cursor)
|
||||
if (suffix.length > 0) await this.append(id, suffix)
|
||||
return
|
||||
}
|
||||
// Owned by a DIFFERENT live session. Reclaim ONLY a truly-abandoned id
|
||||
// (never materialized, no pending buffer); else it is a real collision.
|
||||
const ownerBuffer = this.buffers.get(tracked.owner)
|
||||
if (!tracked.materialized && !ownerBuffer?.length) {
|
||||
this.states.delete(id)
|
||||
} else {
|
||||
throw new Error(`session "${id}" is already bound to a different live session in this backend (id collision)`)
|
||||
}
|
||||
}
|
||||
|
||||
// case 2/3: an artifact at THIS cwd is adopted as a live prefix (or rejected
|
||||
// as a collision inside adoptLivePrefix). cwd-scoped (loadLive), never
|
||||
// any-scope: a same-id artifact at a different cwd is a collision, not a
|
||||
// resume.
|
||||
const live = await this.backend.loadLive(id, session.header.cwd)
|
||||
if (live !== undefined) {
|
||||
// Do NOT route through loadCore(): that crash-repairs open turns as
|
||||
// interrupted, which is wrong for HMR while the live Session is still the
|
||||
// authority and may append the real step/turn end later.
|
||||
await this.serialize(id, () => this.adoptLivePrefix(session, seed, live))
|
||||
return
|
||||
}
|
||||
|
||||
// case 4: a genuinely new session. Register its meta (lazy), then persist its
|
||||
// seed (events present at creation time) once.
|
||||
const meta: SessionHeader = { ...session.header }
|
||||
await this.create(meta)
|
||||
// Bind this state to the live session so a later DIFFERENT session reusing
|
||||
// the id is detected as a collision (case 1) rather than silently no-opped.
|
||||
const created = this.states.get(id)
|
||||
/* v8 ignore next -- create() always sets the state for the id */
|
||||
if (created !== undefined) created.owner = session
|
||||
if (seed.length > 0) await this.append(id, seed)
|
||||
}
|
||||
|
||||
/**
|
||||
* Adopt a stored prefix as a live session's history (HMR/reload): verify the
|
||||
* seed covers the stored prefix, truncate any torn tail (NOT the open turn —
|
||||
* the live Session is still the authority), bind ownership, and persist the
|
||||
* live suffix that was ahead of the stored prefix.
|
||||
*/
|
||||
private async adoptLivePrefix(session: Session, seed: readonly SessionEvent[], stored: StoredPrefix<TornMarker>): Promise<void> {
|
||||
const { meta, events, tornMarker } = stored
|
||||
this.assertVersion(meta)
|
||||
if (!seedCoversPrefix(seed, events)) {
|
||||
throw new Error(`session "${session.header.id}" already has a persisted log on disk that does not match this live session (id collision)`)
|
||||
}
|
||||
// Truncate-only repair (no closers): the open turn is NOT closed here.
|
||||
if (tornMarker !== undefined) await this.backend.commitRepair(meta, tornMarker, [])
|
||||
this.states.set(session.header.id, {
|
||||
meta: { ...meta },
|
||||
cursor: events.length,
|
||||
materialized: true,
|
||||
owner: session,
|
||||
})
|
||||
const suffix = seed.slice(events.length)
|
||||
if (suffix.length > 0) await this.appendCore(session.header.id, suffix)
|
||||
}
|
||||
|
||||
private async flush(session: Session): Promise<void> {
|
||||
// Wait for the session's init (onCreated) so the state/cursor and any
|
||||
// fork-seed persistence are in place before draining. Awaiting the same
|
||||
// promise initFor stored also surfaces an init failure (e.g. a collision)
|
||||
// here, where the caller of session/flush observes it.
|
||||
await this.inits.get(session)
|
||||
// Serialize the WHOLE drain (read cursor → append → splice) on the per-session
|
||||
// chain so two concurrent flushes cannot both read the same cursor and
|
||||
// seq-mismatch on the second append.
|
||||
await this.serialize(session.header.id, () => this.drain(session))
|
||||
}
|
||||
|
||||
/** Drain a session's write buffer to the backend. Caller serializes this per id. */
|
||||
private async drain(session: Session): Promise<void> {
|
||||
const buffer = this.buffers.get(session)
|
||||
if (!buffer?.length) return
|
||||
// Copy WITHOUT removing: the buffer is the only durable-pending copy of these
|
||||
// events. Drain it only AFTER the append commits; events pushed during the
|
||||
// await sit past batch.length and survive the prefix splice, so a
|
||||
// retry/dispose re-drains the rest.
|
||||
const batch = buffer.slice()
|
||||
const state = this.states.get(session.header.id)
|
||||
// Only append events at or beyond the write cursor (a resumed session's seed
|
||||
// is already stored). flush awaits the init above, which always sets state,
|
||||
// so the `?? 0` fallback is a defensive guard that never fires in practice.
|
||||
/* v8 ignore next -- state is always set by the awaited init before flush */
|
||||
const cursor = state?.cursor ?? 0
|
||||
const fresh = batch.filter(e => e.seq >= cursor)
|
||||
// appendCore (NOT the serialized append) — drain already runs inside the
|
||||
// per-session chain, so re-entering via append() would deadlock.
|
||||
if (fresh.length > 0) await this.appendCore(session.header.id, fresh)
|
||||
buffer.splice(0, batch.length)
|
||||
}
|
||||
}
|
||||
@@ -1,7 +1,7 @@
|
||||
/**
|
||||
* The durable session-persistence seam (`ctx.sessionPersistence`): an abstract
|
||||
* service defining WHAT a persistence backend does — durably store, reload,
|
||||
* list, and update sessions — without saying HOW. Implementations subclass
|
||||
* and list sessions — without saying HOW. Implementations subclass
|
||||
* {@link SessionPersistence} and register themselves as the
|
||||
* `sessionPersistence` service; `@deepseek-ai/dsh-session-persistence-jsonl`
|
||||
* (an append-only JSONL log per session) is the first and
|
||||
@@ -15,17 +15,22 @@
|
||||
* parallel "persisted message" type the log must be converted to and from
|
||||
* (faithful to the event-sourced model: the log is the single source of
|
||||
* truth). Metadata that is NOT replayable conversation state (format version,
|
||||
* cwd, lineage) travels separately as {@link SessionMeta}, which is owned by
|
||||
* cwd, lineage) travels separately as {@link SessionHeader}, which is owned by
|
||||
* `dsh-session` and re-exported here.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-session-persistence
|
||||
*/
|
||||
|
||||
import { Context, Service } from 'cordis'
|
||||
import type { SessionEvent, SessionId, SessionMeta, SessionSummary } from '@deepseek-ai/dsh-session'
|
||||
import { isJsonValue } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionEvent, SessionId, SessionHeader } from '@deepseek-ai/dsh-session'
|
||||
|
||||
// Re-export the metadata vocabulary so consumers import it from the seam.
|
||||
export type { SessionHeader, SessionSummary, SessionMeta } from '@deepseek-ai/dsh-session'
|
||||
export type { SessionHeader } from '@deepseek-ai/dsh-session'
|
||||
|
||||
// The backend-agnostic write-path orchestration first-party backends compose.
|
||||
export { PersistenceCoordinator } from './coordinator.ts'
|
||||
export type { PersistenceBackend, StoredPrefix } from './coordinator.ts'
|
||||
|
||||
declare module 'cordis' {
|
||||
interface Context {
|
||||
@@ -33,6 +38,35 @@ declare module 'cordis' {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether a live session's seed reproduces a persisted prefix exactly. Backends
|
||||
* use this collision check to distinguish a legitimate resume/HMR rebind from a
|
||||
* different live session reusing an existing session id.
|
||||
*
|
||||
* The comparison includes the full event payload, not just seq/type/time, so a
|
||||
* mutated seed cannot be grafted onto a durable log with the same envelope.
|
||||
*/
|
||||
export function seedCoversPrefix(seed: readonly SessionEvent[], prefix: readonly SessionEvent[]): boolean {
|
||||
return prefix.length <= seed.length
|
||||
&& prefix.every((event, index) => {
|
||||
const seedEvent = seed[index]
|
||||
return seedEvent !== undefined && JSON.stringify(seedEvent) === JSON.stringify(event)
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Reject non-JSON-serializable event data before a backend serializes a batch.
|
||||
* Live session appends already enforce this; persistence append paths also
|
||||
* accept replay/fork batches that may bypass a live session instance.
|
||||
*/
|
||||
export function assertSerializable(events: readonly SessionEvent[]): void {
|
||||
for (const event of events) {
|
||||
if (!isJsonValue(event.data)) {
|
||||
throw new Error(`event "${event.type}" carries non-JSON-serializable data (seq ${event.seq})`)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Abstract durable session-persistence service. Subclass, implement the
|
||||
* abstract methods, and load the subclass as a plugin — it registers as
|
||||
@@ -69,10 +103,10 @@ export abstract class SessionPersistence extends Service {
|
||||
/**
|
||||
* Register a new session's metadata. A backend MAY defer the physical write
|
||||
* until the first {@link append} (lazy materialization), in which case a
|
||||
* created-but-never-appended session is absent from {@link has}/{@link list}
|
||||
* created-but-never-appended session is absent from {@link list}
|
||||
* — abandoned sessions leave nothing behind.
|
||||
*/
|
||||
abstract create(meta: SessionMeta): Promise<void>
|
||||
abstract create(meta: SessionHeader): Promise<void>
|
||||
|
||||
/**
|
||||
* Durably persist a batch of events (called from the write-behind drain at
|
||||
@@ -84,7 +118,7 @@ export abstract class SessionPersistence extends Service {
|
||||
abstract append(id: SessionId, events: readonly SessionEvent[]): Promise<void>
|
||||
|
||||
/**
|
||||
* Reload a session: its {@link SessionMeta} plus the event log up to the last
|
||||
* Reload a session: its {@link SessionHeader} plus the event log up to the last
|
||||
* durable checkpoint. Returns `meta` AND `events` so the live session is
|
||||
* reconstructed with its `cwd`/lineage, not just its log.
|
||||
*
|
||||
@@ -105,24 +139,10 @@ export abstract class SessionPersistence extends Service {
|
||||
* unloadable (reject). Rejects an unknown format `version`. See the session-persistence RFC for
|
||||
* the crash-recovery contract.
|
||||
*/
|
||||
abstract load(id: SessionId): Promise<{ meta: SessionMeta; events: SessionEvent[] }>
|
||||
abstract load(id: SessionId): Promise<{ meta: SessionHeader; events: SessionEvent[] }>
|
||||
|
||||
/** Lightweight listing from metadata, without a full-log parse. */
|
||||
abstract list(): Promise<SessionMeta[]>
|
||||
|
||||
/** Whether a session is durably present (materialized). */
|
||||
abstract has(id: SessionId): Promise<boolean>
|
||||
|
||||
/** Remove a session and all its persisted artifacts. */
|
||||
abstract delete(id: SessionId): Promise<void>
|
||||
|
||||
/**
|
||||
* Update mutable metadata ({@link SessionSummary}: `updatedAt`, `title`,
|
||||
* `firstPrompt`) WITHOUT touching the append-only event log. A backend
|
||||
* stores the summary beside the log (a sidecar file, a header row) and
|
||||
* rewrites only it.
|
||||
*/
|
||||
abstract update(id: SessionId, summary: Partial<SessionSummary>): Promise<void>
|
||||
abstract list(): Promise<SessionHeader[]>
|
||||
}
|
||||
|
||||
export default SessionPersistence
|
||||
@@ -9,8 +9,8 @@
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionEvent, SessionMeta } from '@deepseek-ai/dsh-session'
|
||||
import { SESSION_FORMAT_VERSION, SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionEvent, SessionHeader } from '@deepseek-ai/dsh-session'
|
||||
import { CallId } from '@deepseek-ai/dsh-llm'
|
||||
import type { SessionPersistence } from '../src/index.ts'
|
||||
|
||||
@@ -20,13 +20,12 @@ export interface ContractBackend {
|
||||
dispose: () => Promise<void>
|
||||
}
|
||||
|
||||
/** Build a minimal {@link SessionMeta} for a session id. */
|
||||
export function meta(id: string, cwd?: string): SessionMeta {
|
||||
/** Build a minimal {@link SessionHeader} for a session id. */
|
||||
export function meta(id: string, cwd?: string): SessionHeader {
|
||||
return {
|
||||
version: 1,
|
||||
version: SESSION_FORMAT_VERSION,
|
||||
id: SessionId(id),
|
||||
createdAt: 1000,
|
||||
updatedAt: 1000,
|
||||
...cwd !== undefined ? { cwd } : {},
|
||||
}
|
||||
}
|
||||
@@ -58,7 +57,7 @@ export function runPersistenceContract(name: string, make: () => Promise<Contrac
|
||||
await persistence.append(m.id, log)
|
||||
|
||||
const loaded = await persistence.load(m.id)
|
||||
expect(loaded.meta).toMatchObject({ version: 1, id: m.id, cwd: '/work' })
|
||||
expect(loaded.meta).toMatchObject({ version: SESSION_FORMAT_VERSION, id: m.id, cwd: '/work' })
|
||||
expect(loaded.events).toEqual(log)
|
||||
} finally {
|
||||
await dispose()
|
||||
@@ -143,24 +142,22 @@ export function runPersistenceContract(name: string, make: () => Promise<Contrac
|
||||
}
|
||||
})
|
||||
|
||||
it('has()/list() exclude a created-but-never-appended (zero-event) session', async () => {
|
||||
it('list() excludes a created-but-never-appended (zero-event) session', async () => {
|
||||
const { persistence, dispose } = await make()
|
||||
try {
|
||||
await persistence.create(meta('empty'))
|
||||
expect(await persistence.has(SessionId('empty'))).toBe(false)
|
||||
expect((await persistence.list()).map(m => m.id)).not.toContain(SessionId('empty'))
|
||||
} finally {
|
||||
await dispose()
|
||||
}
|
||||
})
|
||||
|
||||
it('has()/list() include a session once it has events', async () => {
|
||||
it('list() includes a session once it has events', async () => {
|
||||
const { persistence, dispose } = await make()
|
||||
try {
|
||||
const m = meta('s2')
|
||||
await persistence.create(m)
|
||||
await persistence.append(m.id, oneTurnLog())
|
||||
expect(await persistence.has(m.id)).toBe(true)
|
||||
expect((await persistence.list()).map(x => x.id)).toContain(m.id)
|
||||
} finally {
|
||||
await dispose()
|
||||
@@ -228,37 +225,5 @@ export function runPersistenceContract(name: string, make: () => Promise<Contrac
|
||||
await dispose()
|
||||
}
|
||||
})
|
||||
|
||||
it('delete removes a session', async () => {
|
||||
const { persistence, dispose } = await make()
|
||||
try {
|
||||
const m = meta('s6')
|
||||
await persistence.create(m)
|
||||
await persistence.append(m.id, oneTurnLog())
|
||||
expect(await persistence.has(m.id)).toBe(true)
|
||||
await persistence.delete(m.id)
|
||||
expect(await persistence.has(m.id)).toBe(false)
|
||||
} finally {
|
||||
await dispose()
|
||||
}
|
||||
})
|
||||
|
||||
it('update mutates summary fields without touching the event log', async () => {
|
||||
const { persistence, dispose } = await make()
|
||||
try {
|
||||
const m = meta('s7')
|
||||
const log = oneTurnLog()
|
||||
await persistence.create(m)
|
||||
await persistence.append(m.id, log)
|
||||
await persistence.update(m.id, { title: 'My session', firstPrompt: 'hi' })
|
||||
|
||||
const loaded = await persistence.load(m.id)
|
||||
expect(loaded.meta.title).toBe('My session')
|
||||
expect(loaded.meta.firstPrompt).toBe('hi')
|
||||
expect(loaded.events).toEqual(log) // log untouched
|
||||
} finally {
|
||||
await dispose()
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,775 @@
|
||||
/**
|
||||
* Reusable ORCHESTRATION suite for any backend that composes a
|
||||
* {@link PersistenceCoordinator}. Where {@link runPersistenceContract} (in
|
||||
* contract.ts) pins the public read/write SEMANTICS, this suite pins the
|
||||
* coordinator's WRITE-PATH ORCHESTRATION — the behavior that is identical across
|
||||
* every first-party backend because it lives in the shared coordinator, not in
|
||||
* the storage primitives: the `session/created` → `session/event` →
|
||||
* `session/flush` → dispose drain, lazy materialization, fork-seed persistence,
|
||||
* the four `onCreated` adoption cases (new / HMR-adopt / collision /
|
||||
* ownerless-claim), crash-tail repair on load, and dispose-time quiescence.
|
||||
*
|
||||
* A backend imports {@link runCoordinatorContract} and calls it with a
|
||||
* {@link CoordinatorFixture} factory that knows how to (a) mount the REAL
|
||||
* backend plugin on a {@link Context} over a SHARED storage scope (so HMR/reload
|
||||
* tests can dispose one instance and mount another over the same bytes/rows),
|
||||
* and (b) inject a never-committed torn tail for one session
|
||||
* ({@link CoordinatorFixture.corruptTail}) so the through-coordinator torn-tail
|
||||
* repair branch is exercised against real storage. The suite drives everything
|
||||
* through the PUBLIC {@link SessionPersistence} API + the cordis SessionStore
|
||||
* write path — never the storage primitives directly — so it runs unchanged for
|
||||
* every backend (memory / jsonl / sqlite).
|
||||
*
|
||||
* Each scenario lives here once and runs once per backend through the fixture;
|
||||
* the per-backend specs keep ONLY their storage-mechanics tests.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-session-persistence/tests/coordinator-contract
|
||||
*/
|
||||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { Context, type Fiber } from 'cordis'
|
||||
import SessionStore, { SESSION_FORMAT_VERSION, SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type { Session, SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionPersistence } from '../src/index.ts'
|
||||
import { meta, oneTurnLog } from './contract.ts'
|
||||
|
||||
/**
|
||||
* The backend-specific capabilities the orchestration suite needs beyond the
|
||||
* public service API. A fresh fixture is created per test (isolated storage);
|
||||
* the suite mounts/disposes backend instances on it and cleans it up at the end.
|
||||
*/
|
||||
export interface CoordinatorFixture {
|
||||
/**
|
||||
* Mount the REAL backend plugin (via `ctx.plugin`, the Loader path) on `ctx`,
|
||||
* over THIS fixture's shared storage scope. Returns the plugin fiber so the
|
||||
* suite can dispose a single instance (HMR/reload) while the storage — and any
|
||||
* still-live session in another fiber — survives. The caller has already
|
||||
* mounted `SessionStore` on `ctx`.
|
||||
*/
|
||||
mount: (ctx: Context) => Promise<Fiber>
|
||||
|
||||
/**
|
||||
* Inject a NEVER-COMMITTED torn tail into the backend's storage for `id` at
|
||||
* the given `cwd` (the cwd the session was created with): a half-written
|
||||
* record past the committed region (JSONL: a partial line with no newline;
|
||||
* SQLite: a row with invalid `data` JSON past the committed seq). This drives
|
||||
* the coordinator's `loadCore` `tornMarker !== undefined` → `commitRepair`
|
||||
* branch against real storage.
|
||||
*
|
||||
* OMITTED by a backend that structurally has no torn tails (memory): the
|
||||
* torn-tail scenario then self-skips (asserted explicitly in the suite).
|
||||
*/
|
||||
corruptTail?: (id: SessionId, cwd: string | undefined) => Promise<void>
|
||||
|
||||
/** Tear down the storage scope (remove the temp dir / file). */
|
||||
cleanup: () => Promise<void>
|
||||
}
|
||||
|
||||
/** A constant absolute cwd; jsonl keys directories off it, memory/sqlite ignore it. */
|
||||
const WORK = '/w'
|
||||
const OTHER = '/other'
|
||||
|
||||
/** The per-session init map a backend exposes for white-box init awaits. */
|
||||
function inits(persistence: SessionPersistence): Map<Session, Promise<void>> {
|
||||
return (persistence as unknown as { inits: Map<Session, Promise<void>> }).inits
|
||||
}
|
||||
|
||||
/** Append a whole event log to a live session, event by event (drives session/event). */
|
||||
function send(session: Session, events: readonly SessionEvent[]): void {
|
||||
for (const e of events) session.append(e.type, e.data)
|
||||
}
|
||||
|
||||
/** A live session created inside its OWN fiber, so it survives a backend reload. */
|
||||
async function liveSessionInFiber(
|
||||
ctx: Context, id: string, cwd: string | undefined,
|
||||
): Promise<Session> {
|
||||
let session!: Session
|
||||
await ctx.plugin(Object.assign((inner: Context) => {
|
||||
session = inner.sessions.create(SessionId(id), cwd !== undefined ? { meta: { cwd } } : undefined)
|
||||
}, { inject: ['sessions'] }))
|
||||
return session
|
||||
}
|
||||
|
||||
/**
|
||||
* Run the coordinator orchestration suite against a backend. `makeFixture()`
|
||||
* MUST return a fresh fixture (isolated storage) each call.
|
||||
*/
|
||||
export function runCoordinatorContract(name: string, makeFixture: () => Promise<CoordinatorFixture>): void {
|
||||
describe(`PersistenceCoordinator orchestration: ${name}`, () => {
|
||||
/** Mount SessionStore + a backend instance on a fresh context over the fixture's storage. */
|
||||
async function freshCtx(fix: CoordinatorFixture): Promise<{ ctx: Context; fiber: Fiber }> {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await fix.mount(ctx)
|
||||
return { ctx, fiber }
|
||||
}
|
||||
|
||||
// --- write path: live session → flush → reload ---
|
||||
|
||||
it('persists a live session driven through the store, surviving reload', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
const session = ctx.sessions.create(SessionId('live'), { meta: { cwd: WORK } })
|
||||
send(session, oneTurnLog())
|
||||
await ctx.parallel('session/flush', session)
|
||||
|
||||
const loaded = await ctx.sessionPersistence.load(SessionId('live'))
|
||||
expect(loaded.events).toHaveLength(6)
|
||||
expect(loaded.meta.cwd).toBe(WORK)
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('snapshot-on-buffer: mutating an event after session/event does not corrupt the persisted copy', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
const session = ctx.sessions.create(SessionId('mutate'), { meta: { cwd: WORK } })
|
||||
const ev = session.append('user/message', { content: [{ type: 'text', text: 'original' }], source: { kind: 'user' } })
|
||||
// Mutate the live event object AFTER it was buffered by session/event.
|
||||
;(ev.data as { content: { type: 'text'; text: string }[] }).content[0]!.text = 'HACKED'
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
await ctx.parallel('session/flush', session)
|
||||
|
||||
const loaded = await ctx.sessionPersistence.load(SessionId('mutate'))
|
||||
const first = loaded.events[0]
|
||||
expect(first?.type === 'user/message' && (first.data.content[0] as { text: string }).text).toBe('original')
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('append snapshots the batch: mutating the caller array/events after the call is ignored', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
const m = meta('snapshot', WORK)
|
||||
await ctx.sessionPersistence.create(m)
|
||||
const events = oneTurnLog() // seqs 0..5
|
||||
const userMsg = events[1] // the user/message event
|
||||
const p = ctx.sessionPersistence.append(m.id, events)
|
||||
// Mutate the caller's array AND an event object after the call but before
|
||||
// the queued op runs: the snapshot taken at call time must shield the copy.
|
||||
events.push({ type: 'turn/start', seq: 6, time: 99, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } })
|
||||
if (userMsg?.type === 'user/message') userMsg.data.content = [{ type: 'text', text: 'MUTATED' }]
|
||||
await p
|
||||
const loaded = await ctx.sessionPersistence.load(m.id)
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5]) // not 0..6
|
||||
const persisted = JSON.stringify(loaded.events)
|
||||
expect(persisted).toContain('hi') // original content
|
||||
expect(persisted).not.toContain('MUTATED')
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
// --- fork / resume ---
|
||||
|
||||
it('fork: a seeded new session persists its seed once (no double-write on a no-op flush)', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
const seed = oneTurnLog()
|
||||
// A fork: a brand-new id whose seed came from elsewhere.
|
||||
const forked = ctx.sessions.create(SessionId('forked'), { seed, meta: { cwd: WORK } })
|
||||
await inits(ctx.sessionPersistence).get(forked) // onCreated persisted the seed
|
||||
const loaded = await ctx.sessionPersistence.load(SessionId('forked'))
|
||||
expect(loaded.events).toEqual(seed)
|
||||
// A flush with no NEW events must not double-write.
|
||||
await ctx.parallel('session/flush', forked)
|
||||
const reloaded = await ctx.sessionPersistence.load(SessionId('forked'))
|
||||
expect(reloaded.events).toEqual(seed)
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('resume: a re-created session seeded with the loaded log does not re-append its seed and continues the seq', async () => {
|
||||
const fix = await makeFixture()
|
||||
const first = await freshCtx(fix)
|
||||
try {
|
||||
// First lifecycle: persist a session through the store.
|
||||
const s1 = first.ctx.sessions.create(SessionId('resumed'), { meta: { cwd: WORK } })
|
||||
send(s1, oneTurnLog())
|
||||
await first.ctx.parallel('session/flush', s1)
|
||||
} finally {
|
||||
await first.fiber.dispose()
|
||||
}
|
||||
|
||||
// Second lifecycle: a NEW backend instance + a session re-created with the
|
||||
// same id SEEDED with the loaded events. onCreated adopts the stored log
|
||||
// (does not re-persist the seed); a new turn appends at seq 6.
|
||||
const second = await freshCtx(fix)
|
||||
try {
|
||||
const loaded = await second.ctx.sessionPersistence.load(SessionId('resumed'))
|
||||
const s2 = second.ctx.sessions.create(SessionId('resumed'), { seed: loaded.events, meta: { cwd: WORK } })
|
||||
await inits(second.ctx.sessionPersistence).get(s2) // let onCreated adopt
|
||||
s2.append('turn/start', { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
s2.append('turn/end', { turn: 2, reason: { kind: 'completed' } })
|
||||
await second.ctx.parallel('session/flush', s2)
|
||||
|
||||
const reloaded = await second.ctx.sessionPersistence.load(SessionId('resumed'))
|
||||
// 6 original + 2 new, contiguous, no duplicated seed.
|
||||
expect(reloaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
} finally {
|
||||
await second.fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
// --- HMR ---
|
||||
|
||||
it('HMR: applying the plugin seeds existing live sessions', async () => {
|
||||
const fix = await makeFixture()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
// A session exists BEFORE the persistence plugin is applied.
|
||||
const session = ctx.sessions.create(SessionId('pre-existing'), { meta: { cwd: WORK } })
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
|
||||
const fiber = await fix.mount(ctx)
|
||||
try {
|
||||
// The plugin seeded it on apply; a subsequent flush persists its events.
|
||||
await ctx.parallel('session/flush', session)
|
||||
const loaded = await ctx.sessionPersistence.load(SessionId('pre-existing'))
|
||||
expect(loaded.events.length).toBeGreaterThanOrEqual(2)
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('HMR: dispose drains remaining buffers', async () => {
|
||||
const fix = await makeFixture()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await fix.mount(ctx)
|
||||
const session = await liveSessionInFiber(ctx, 'drain', WORK)
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'buffered' }], source: { kind: 'user' } })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
// No explicit flush — dispose must drain.
|
||||
await fiber.dispose()
|
||||
|
||||
// A fresh backend instance reads what the disposed one drained.
|
||||
const second = await freshCtx(fix)
|
||||
try {
|
||||
const loaded = await second.ctx.sessionPersistence.load(SessionId('drain'))
|
||||
expect(loaded.events.length).toBeGreaterThanOrEqual(2)
|
||||
} finally {
|
||||
await second.fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('HMR: reloading the backend adopts a still-live, already-materialized session', async () => {
|
||||
const fix = await makeFixture()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
// The session lives in its OWN fiber so it survives the backend reload.
|
||||
const session = await liveSessionInFiber(ctx, 'hmr-adopt', WORK)
|
||||
try {
|
||||
// Backend instance 1 materializes the session.
|
||||
const backend1 = await fix.mount(ctx)
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'hi' }], source: { kind: 'user' } })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
await ctx.parallel('session/flush', session)
|
||||
|
||||
// Hot-reload: dispose instance 1, mount instance 2 over the SAME storage
|
||||
// while the session stays live. Instance 2 has an empty states map but the
|
||||
// log is materialized and is a prefix of the live events — it must ADOPT
|
||||
// (not reject). A second turn appended after reload then persists.
|
||||
await backend1.dispose()
|
||||
await fix.mount(ctx)
|
||||
session.append('turn/start', { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'again' }], source: { kind: 'user' } })
|
||||
session.append('turn/end', { turn: 2, reason: { kind: 'completed' } })
|
||||
await expect(ctx.parallel('session/flush', session)).resolves.not.toThrow()
|
||||
|
||||
const loaded = await ctx.sessionPersistence.load(SessionId('hmr-adopt'))
|
||||
expect(loaded.events.filter(e => e.type === 'turn/start')).toHaveLength(2)
|
||||
} finally {
|
||||
await ctx.fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('HMR: adoption persists the live SUFFIX that was ahead of the stored prefix', async () => {
|
||||
const fix = await makeFixture()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const session = await liveSessionInFiber(ctx, 'hmr-suffix', WORK)
|
||||
try {
|
||||
// Instance 1 flushes turn 1.
|
||||
const backend1 = await fix.mount(ctx)
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
await ctx.parallel('session/flush', session)
|
||||
|
||||
// Append turn 2 to the LIVE session, then dispose instance 1 WITHOUT
|
||||
// flushing turn 2: it is now ONLY in the live session's events; the new
|
||||
// backend never buffered it via session/event.
|
||||
await backend1.dispose()
|
||||
session.append('turn/start', { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('turn/end', { turn: 2, reason: { kind: 'completed' } })
|
||||
|
||||
// Instance 2 adopts the stored prefix (turn 1) and MUST also persist the
|
||||
// live suffix (turn 2) carried in the session's events.
|
||||
await fix.mount(ctx)
|
||||
await ctx.parallel('session/flush', session)
|
||||
const loaded = await ctx.sessionPersistence.load(SessionId('hmr-suffix'))
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3])
|
||||
expect(loaded.events.filter(e => e.type === 'turn/start')).toHaveLength(2)
|
||||
} finally {
|
||||
await ctx.fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('HMR adoption does NOT crash-repair an active open turn as interrupted (truncate without closers)', async () => {
|
||||
const fix = await makeFixture()
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const session = await liveSessionInFiber(ctx, 'hmr-open', WORK)
|
||||
try {
|
||||
const first = await fix.mount(ctx)
|
||||
session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
await ctx.parallel('session/flush', session)
|
||||
|
||||
// Crash-tail a torn fragment past the (open) committed turn, then reload.
|
||||
await first.dispose()
|
||||
if (fix.corruptTail) await fix.corruptTail(SessionId('hmr-open'), WORK)
|
||||
const second = await fix.mount(ctx)
|
||||
// The live session is still the authority: it appends the REAL step/turn
|
||||
// end. Adoption must truncate the torn tail but NOT synthesize closers.
|
||||
session.append('step/end', { turn: 1, step: 1 })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
await ctx.parallel('session/flush', session)
|
||||
|
||||
const loaded = await ctx.sessionPersistence.load(SessionId('hmr-open'))
|
||||
expect(loaded.events.map(e => e.type)).toEqual(['turn/start', 'step/start', 'step/end', 'turn/end'])
|
||||
expect(loaded.events.at(-1)).toMatchObject({ type: 'turn/end', data: { reason: { kind: 'completed' } } })
|
||||
await second.dispose()
|
||||
} finally {
|
||||
await ctx.fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
// --- collision / id reuse ---
|
||||
|
||||
it('a NEW live session colliding on a persisted id is rejected, not silently adopted', async () => {
|
||||
const fix = await makeFixture()
|
||||
const first = await freshCtx(fix)
|
||||
try {
|
||||
const s1 = first.ctx.sessions.create(SessionId('collide'), { meta: { cwd: WORK } })
|
||||
send(s1, oneTurnLog())
|
||||
await first.ctx.parallel('session/flush', s1)
|
||||
} finally {
|
||||
await first.fiber.dispose()
|
||||
}
|
||||
|
||||
// A FRESH backend + a NEW live session with the same id but NO explicit
|
||||
// resume. onCreated treats it as new; create() rejects because a log already
|
||||
// exists. The rejection surfaces via the init promise (flush awaits it).
|
||||
const second = await freshCtx(fix)
|
||||
try {
|
||||
const s2 = second.ctx.sessions.create(SessionId('collide'), { meta: { cwd: WORK } })
|
||||
s2.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
await expect(inits(second.ctx.sessionPersistence).get(s2))
|
||||
.rejects.toThrow(/already has a persisted log|id collision/)
|
||||
} finally {
|
||||
await second.fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('an abandoned lazy session (never materialized) releases its id for reuse', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
// A live session created then disposed BEFORE its first append: cursor 0,
|
||||
// never materialized. A new live session reusing the id must reclaim it.
|
||||
let firstSession!: Session
|
||||
const firstFiber = await ctx.plugin(Object.assign((inner: Context) => {
|
||||
firstSession = inner.sessions.create(SessionId('abandoned'), { meta: { cwd: WORK } })
|
||||
}, { inject: ['sessions'] }))
|
||||
await inits(ctx.sessionPersistence).get(firstSession) // register the lazy state
|
||||
await firstFiber.dispose() // disposed before any append → never materialized
|
||||
|
||||
let reuse!: Session
|
||||
await ctx.plugin(Object.assign((inner: Context) => {
|
||||
reuse = inner.sessions.create(SessionId('abandoned'), { meta: { cwd: WORK } })
|
||||
}, { inject: ['sessions'] }))
|
||||
await expect(inits(ctx.sessionPersistence).get(reuse)).resolves.toBeUndefined()
|
||||
reuse.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
reuse.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
await ctx.parallel('session/flush', reuse)
|
||||
const loaded = await ctx.sessionPersistence.load(SessionId('abandoned'))
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1])
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('does NOT reclaim an id whose abandoned owner still has buffered (unflushed) events', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
let first!: Session
|
||||
const firstFiber = await ctx.plugin(Object.assign((inner: Context) => {
|
||||
first = inner.sessions.create(SessionId('buffered'), { meta: { cwd: WORK } })
|
||||
}, { inject: ['sessions'] }))
|
||||
await inits(ctx.sessionPersistence).get(first)
|
||||
// Append a turn but do NOT flush — events sit in the write-behind buffer.
|
||||
first.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } })
|
||||
first.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
await firstFiber.dispose() // disposed before flush; not materialized, buffer pending
|
||||
|
||||
let reuse!: Session
|
||||
await ctx.plugin(Object.assign((inner: Context) => {
|
||||
reuse = inner.sessions.create(SessionId('buffered'), { meta: { cwd: WORK } })
|
||||
}, { inject: ['sessions'] }))
|
||||
await expect(inits(ctx.sessionPersistence).get(reuse)).rejects.toThrow(/already bound to a different live session/)
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('initFor is idempotent: re-emitting session/created does not re-initialize', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
const session = ctx.sessions.create(SessionId('idem'), { meta: { cwd: WORK } })
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'x' }], source: { kind: 'user' } })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
await ctx.parallel('session/flush', session)
|
||||
// Re-emit session/created for the SAME live session (idempotent initFor).
|
||||
ctx.emit('session/created', session)
|
||||
await ctx.parallel('session/flush', session)
|
||||
const loaded = await ctx.sessionPersistence.load(SessionId('idem'))
|
||||
expect(loaded.events).toHaveLength(2) // not doubled
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
// --- ownerless-state claim (public create()/load() then a live session arrives) ---
|
||||
|
||||
it('a live session claims cursor-0 ownerless state created via the public API and persists its seed', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
// create() registers ownerless state with cursor 0 (lazy, nothing persisted).
|
||||
await ctx.sessionPersistence.create(meta('lazy-claim', WORK))
|
||||
// A live session with that id arrives and claims it (cursor 0 matches
|
||||
// trivially), persisting its seed.
|
||||
const live = ctx.sessions.create(SessionId('lazy-claim'), { seed: oneTurnLog(), meta: { cwd: WORK } })
|
||||
await expect(inits(ctx.sessionPersistence).get(live)).resolves.toBeUndefined()
|
||||
const loaded = await ctx.sessionPersistence.load(SessionId('lazy-claim'))
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5])
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('a fresh session reusing a previously-loaded id is rejected (ownerless guard)', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
// Materialize a log, then load() it WITHOUT a live session — ownerless
|
||||
// state, cursor at the persisted length.
|
||||
await ctx.sessionPersistence.create(meta('preview', WORK))
|
||||
await ctx.sessionPersistence.append(SessionId('preview'), oneTurnLog())
|
||||
await ctx.sessionPersistence.load(SessionId('preview'))
|
||||
|
||||
// A FRESH (empty-seed) live session reusing that id must be rejected: its
|
||||
// seq 0..cursor-1 events would otherwise be filtered as already-persisted.
|
||||
let fresh!: Session
|
||||
await ctx.plugin(Object.assign((inner: Context) => {
|
||||
fresh = inner.sessions.create(SessionId('preview'), { meta: { cwd: WORK } })
|
||||
}, { inject: ['sessions'] }))
|
||||
await expect(inits(ctx.sessionPersistence).get(fresh))
|
||||
.rejects.toThrow(/do not match this live session|already has a persisted log|id collision/)
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('a live session whose seed matches the loaded prefix claims ownerless state and persists the suffix', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
// Materialize and load (ownerless, cursor = 6).
|
||||
await ctx.sessionPersistence.create(meta('claim', WORK))
|
||||
await ctx.sessionPersistence.append(SessionId('claim'), oneTurnLog())
|
||||
const { events } = await ctx.sessionPersistence.load(SessionId('claim'))
|
||||
|
||||
// A live session SEEDED with the loaded log PLUS a new turn claims the
|
||||
// ownerless state and persists only the suffix.
|
||||
const cont = ctx.sessions.create(SessionId('claim'), { seed: [
|
||||
...events,
|
||||
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'turn/end', seq: 7, time: 8, data: { turn: 2, reason: { kind: 'completed' } } },
|
||||
], meta: { cwd: WORK } })
|
||||
await inits(ctx.sessionPersistence).get(cont)
|
||||
const loaded = await ctx.sessionPersistence.load(SessionId('claim'))
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('a live session at a DIFFERENT cwd cannot claim cursor-0 ownerless state (cwd scope)', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
// create() registers ownerless state at cwd /a (cursor 0 — claims would
|
||||
// otherwise match trivially on the seed).
|
||||
await ctx.sessionPersistence.create(meta('wrong-cwd-claim', OTHER))
|
||||
// A live session reusing the id but at cwd WORK must NOT claim it — the
|
||||
// cwd scope is the fence (without it, WORK events would append under the
|
||||
// OTHER header). Rejected as a collision.
|
||||
const live = ctx.sessions.create(SessionId('wrong-cwd-claim'), { seed: oneTurnLog(), meta: { cwd: WORK } })
|
||||
await expect(inits(ctx.sessionPersistence).get(live)).rejects.toThrow(/different cwd|id collision/)
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('a live session at a DIFFERENT cwd cannot claim loaded-prefix ownerless state (cwd scope)', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
// Materialize + load at cwd OTHER (ownerless, cursor = 6).
|
||||
await ctx.sessionPersistence.create(meta('wrong-cwd-load', OTHER))
|
||||
await ctx.sessionPersistence.append(SessionId('wrong-cwd-load'), oneTurnLog())
|
||||
const { events } = await ctx.sessionPersistence.load(SessionId('wrong-cwd-load'))
|
||||
// A live session whose SEED matches the loaded prefix but whose cwd is
|
||||
// WORK must still be rejected — the cwd guard runs before the seed check.
|
||||
const live = ctx.sessions.create(SessionId('wrong-cwd-load'), { seed: events, meta: { cwd: WORK } })
|
||||
await expect(inits(ctx.sessionPersistence).get(live)).rejects.toThrow(/different cwd|id collision/)
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('a no-cwd ownerless state cannot be claimed by a live session WITH a cwd (cwd scope, undefined side)', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
// Ownerless state created WITHOUT a cwd (the no-cwd bucket).
|
||||
await ctx.sessionPersistence.create(meta('no-cwd-state'))
|
||||
// A live session reusing the id but WITH cwd WORK is a cwd mismatch
|
||||
// (undefined vs WORK) and must be rejected.
|
||||
const live = ctx.sessions.create(SessionId('no-cwd-state'), { seed: oneTurnLog(), meta: { cwd: WORK } })
|
||||
await expect(inits(ctx.sessionPersistence).get(live)).rejects.toThrow(/different cwd|id collision/)
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
// --- append adopts a storage-only session (fresh instance, no prior create/load) ---
|
||||
|
||||
it('append adopts a storage-only session (fresh instance) and continues the seq', async () => {
|
||||
const fix = await makeFixture()
|
||||
const first = await freshCtx(fix)
|
||||
try {
|
||||
const m = meta('adopt-append', WORK)
|
||||
await first.ctx.sessionPersistence.create(m)
|
||||
await first.ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
} finally {
|
||||
await first.fiber.dispose()
|
||||
}
|
||||
|
||||
// A fresh instance appends a second turn WITHOUT a prior create/load: append
|
||||
// must adopt the stored session (cursor = stored length) and continue.
|
||||
const second = await freshCtx(fix)
|
||||
try {
|
||||
await second.ctx.sessionPersistence.append(SessionId('adopt-append'), [
|
||||
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'turn/end', seq: 7, time: 8, data: { turn: 2, reason: { kind: 'completed' } } },
|
||||
])
|
||||
const loaded = await second.ctx.sessionPersistence.load(SessionId('adopt-append'))
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7])
|
||||
} finally {
|
||||
await second.fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
// --- small public-API edges that the coordinator owns uniformly ---
|
||||
|
||||
it('append of an empty batch is a no-op (stays lazy)', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
const m = meta('empty-batch', WORK)
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, [])
|
||||
expect((await ctx.sessionPersistence.list()).map(h => h.id)).not.toContain(m.id)
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('load rejects a missing session', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
await expect(ctx.sessionPersistence.load(SessionId('nope'))).rejects.toThrow(/not found/)
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('create rejects a duplicate id (in memory and on a persisted log)', async () => {
|
||||
const fix = await makeFixture()
|
||||
const first = await freshCtx(fix)
|
||||
try {
|
||||
const m = meta('dup', WORK)
|
||||
await first.ctx.sessionPersistence.create(m)
|
||||
// Same in-memory state.
|
||||
await expect(first.ctx.sessionPersistence.create(m)).rejects.toThrow(/already exists in this backend/)
|
||||
await first.ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
} finally {
|
||||
await first.fiber.dispose()
|
||||
}
|
||||
|
||||
// A fresh instance over the same storage sees the persisted log.
|
||||
const second = await freshCtx(fix)
|
||||
try {
|
||||
await expect(second.ctx.sessionPersistence.create(meta('dup', WORK)))
|
||||
.rejects.toThrow(/already has a persisted log on disk/)
|
||||
} finally {
|
||||
await second.fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('rejects an unknown format version on load (assertVersion)', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
const m = { version: 99, id: SessionId('v99'), createdAt: 1, cwd: WORK }
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
await expect(ctx.sessionPersistence.load(m.id)).rejects.toThrow(/version/)
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('round-trips a header with parentSession (fork lineage)', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
const m = { version: SESSION_FORMAT_VERSION, id: SessionId('forked-child'), createdAt: 1, cwd: WORK, parentSession: SessionId('the-parent') }
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
const loaded = await ctx.sessionPersistence.load(m.id)
|
||||
expect(loaded.meta.parentSession).toBe('the-parent')
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
it('flush before init resolves uses cursor 0', async () => {
|
||||
const fix = await makeFixture()
|
||||
const { ctx, fiber } = await freshCtx(fix)
|
||||
try {
|
||||
// Append directly to a live session and flush IMMEDIATELY, before the
|
||||
// async onCreated init has necessarily set state (exercises the
|
||||
// state-undefined cursor path).
|
||||
const session = ctx.sessions.create(SessionId('flush-nostate'), { meta: { cwd: WORK } })
|
||||
session.append('user/message', { content: [{ type: 'text', text: 'q' }], source: { kind: 'user' } })
|
||||
session.append('turn/end', { turn: 1, reason: { kind: 'completed' } })
|
||||
await ctx.parallel('session/flush', session)
|
||||
const loaded = await ctx.sessionPersistence.load(SessionId('flush-nostate'))
|
||||
expect(loaded.events).toHaveLength(2)
|
||||
} finally {
|
||||
await fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
|
||||
// --- crash-tail repair THROUGH the coordinator (real storage torn tail) ---
|
||||
|
||||
it('torn-tail load: a never-committed tail is truncated and the open turn closed during load (commitRepair w/ tornMarker)', async () => {
|
||||
const fix = await makeFixture()
|
||||
if (!fix.corruptTail) {
|
||||
// A memory-style store has no torn tails (every write is atomic in RAM),
|
||||
// so there is no tornMarker path to exercise. Assert that explicitly
|
||||
// instead of silently skipping, then bail.
|
||||
expect(fix.corruptTail).toBeUndefined()
|
||||
await fix.cleanup()
|
||||
return
|
||||
}
|
||||
const first = await freshCtx(fix)
|
||||
try {
|
||||
const m = meta('torn', WORK)
|
||||
await first.ctx.sessionPersistence.create(m)
|
||||
await first.ctx.sessionPersistence.append(m.id, oneTurnLog()) // committed 0..5 (balanced)
|
||||
// A second turn whose real events are durable but never closed (open turn).
|
||||
await first.ctx.sessionPersistence.append(m.id, [
|
||||
{ type: 'turn/start', seq: 6, time: 7, data: { turn: 2, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'step/start', seq: 7, time: 8, data: { turn: 2, step: 1 } },
|
||||
])
|
||||
} finally {
|
||||
await first.fiber.dispose()
|
||||
}
|
||||
// Inject a torn fragment past the committed region (never-committed tail).
|
||||
await fix.corruptTail(SessionId('torn'), WORK)
|
||||
|
||||
// A FRESH instance loads: the torn tail is truncated (tornMarker !==
|
||||
// undefined) AND the open turn 2 is closed with synthetic step/end +
|
||||
// turn/end {interrupted} — commitRepair runs with BOTH a torn marker and
|
||||
// closers. The preserved real events (0..7) are never truncated.
|
||||
const second = await freshCtx(fix)
|
||||
try {
|
||||
const loaded = await second.ctx.sessionPersistence.load(SessionId('torn'))
|
||||
expect(loaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
|
||||
expect(loaded.events.map(e => e.type)).toEqual([
|
||||
'turn/start', 'user/message', 'step/start', 'assistant/message', 'step/end', 'turn/end', // turn 1
|
||||
'turn/start', 'step/start', 'step/end', 'turn/end', // turn 2: real + synthetic closers
|
||||
])
|
||||
const last = loaded.events.at(-1)!
|
||||
expect(last.type === 'turn/end' && last.data.reason).toEqual({ kind: 'interrupted' })
|
||||
|
||||
// The repair is durable: the next append continues at the balanced length
|
||||
// (seq 10) and a reload round-trips identically.
|
||||
await second.ctx.sessionPersistence.append(SessionId('torn'), [
|
||||
{ type: 'turn/start', seq: 10, time: 9, data: { turn: 3, trigger: { kind: 'message', source: { kind: 'user' } } } },
|
||||
{ type: 'turn/end', seq: 11, time: 10, data: { turn: 3, reason: { kind: 'completed' } } },
|
||||
])
|
||||
const reloaded = await second.ctx.sessionPersistence.load(SessionId('torn'))
|
||||
expect(reloaded.events.map(e => e.seq)).toEqual([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
|
||||
} finally {
|
||||
await second.fiber.dispose()
|
||||
await fix.cleanup()
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,196 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import SessionStore, { SessionId, isJsonValue } from '@deepseek-ai/dsh-session'
|
||||
import type { Session, SessionEvent, SessionHeader } from '@deepseek-ai/dsh-session'
|
||||
import {
|
||||
SessionPersistence, PersistenceCoordinator, assertSerializable, seedCoversPrefix,
|
||||
type PersistenceBackend, type StoredPrefix,
|
||||
} from '../src/index.ts'
|
||||
import { runPersistenceContract, meta, oneTurnLog } from './contract.ts'
|
||||
import { runCoordinatorContract, type CoordinatorFixture } from './coordinator-contract.ts'
|
||||
|
||||
/** The durable store shape: materialized sessions only (no lazy entries). */
|
||||
type MemoryStore = Map<string, { meta: SessionHeader; events: SessionEvent[] }>
|
||||
|
||||
/** Optional plugin config: an EXTERNAL store shared across backend instances. */
|
||||
interface MemoryConfig { store?: MemoryStore }
|
||||
|
||||
/**
|
||||
* A trivial in-memory {@link SessionPersistence} that composes a
|
||||
* {@link PersistenceCoordinator} over a dependency-free `Map`-backed
|
||||
* {@link PersistenceBackend}. It is BOTH the coordinator's reference vehicle
|
||||
* (the simplest possible storage — a `Map<id, {meta, events}>` with no torn
|
||||
* tails, so `tornMarker` is always undefined) and the cover for the abstract
|
||||
* base's constructor + service registration. The real durable backends are
|
||||
* `@deepseek-ai/dsh-session-persistence-jsonl` / `-sqlite`.
|
||||
*
|
||||
* The store can be supplied via config so two backend instances share one Map —
|
||||
* the in-RAM analogue of two backends over the same file/db, which the
|
||||
* coordinator orchestration suite's HMR/reload tests need (a fresh instance with
|
||||
* an empty in-memory states map adopting an already-materialized session).
|
||||
*/
|
||||
class MemoryPersistence extends SessionPersistence implements PersistenceBackend<never> {
|
||||
static inject = ['sessions']
|
||||
|
||||
override readonly name = 'session-persistence-memory'
|
||||
|
||||
/** The whole durable store: materialized sessions only (no lazy entries). */
|
||||
private store: MemoryStore
|
||||
private coordinator: PersistenceCoordinator<never>
|
||||
|
||||
constructor(ctx: Context, config?: MemoryConfig) {
|
||||
super(ctx)
|
||||
// Assign the store BEFORE constructing the coordinator: the coordinator's
|
||||
// constructor installs the write path and synchronously seeds existing live
|
||||
// sessions (onCreated → loadLive → this.store), so store must exist first.
|
||||
this.store = config?.store ?? new Map<string, { meta: SessionHeader; events: SessionEvent[] }>()
|
||||
this.coordinator = new PersistenceCoordinator<never>(this.ctx, this)
|
||||
}
|
||||
|
||||
// --- service surface (delegated to the coordinator) ---
|
||||
|
||||
create(m: SessionHeader): Promise<void> {
|
||||
return this.coordinator.create(m)
|
||||
}
|
||||
|
||||
append(id: SessionId, events: readonly SessionEvent[]): Promise<void> {
|
||||
return this.coordinator.append(id, events)
|
||||
}
|
||||
|
||||
load(id: SessionId): Promise<{ meta: SessionHeader; events: SessionEvent[] }> {
|
||||
return this.coordinator.load(id)
|
||||
}
|
||||
|
||||
/** White-box accessor: await a specific session's onCreated init. */
|
||||
get inits(): Map<Session, Promise<void>> {
|
||||
return this.coordinator.inits
|
||||
}
|
||||
|
||||
// --- PersistenceBackend hooks (the Map storage primitives) ---
|
||||
|
||||
// A Map-backed store has no torn tails, so `tornMarker` is never set. Ids are
|
||||
// globally unique, so loadStored and loadLive are identical (cwd is ignored).
|
||||
async loadStored(id: SessionId): Promise<StoredPrefix<never> | undefined> {
|
||||
const entry = this.store.get(id)
|
||||
if (!entry) return undefined
|
||||
return { meta: structuredClone(entry.meta), events: structuredClone(entry.events) }
|
||||
}
|
||||
|
||||
loadLive(id: SessionId, _cwd: string | undefined): Promise<StoredPrefix<never> | undefined> {
|
||||
return this.loadStored(id)
|
||||
}
|
||||
|
||||
async appendBatch(m: SessionHeader, events: readonly SessionEvent[], _isMaterialized: boolean): Promise<void> {
|
||||
// Defense-in-depth: the coordinator already validates serializability, but a
|
||||
// durable store must reject non-JSON data at its own boundary too.
|
||||
for (const e of events) {
|
||||
if (!isJsonValue(e.data)) throw new Error(`event "${e.type}" carries non-JSON-serializable data`)
|
||||
}
|
||||
const existing = this.store.get(m.id)
|
||||
if (!existing) {
|
||||
// First batch: `_isMaterialized` is false (the coordinator only omits
|
||||
// materialization on the first batch); writing the entry IS the materialization.
|
||||
this.store.set(m.id, { meta: structuredClone(m), events: structuredClone(events) as SessionEvent[] })
|
||||
} else {
|
||||
existing.events.push(...structuredClone(events) as SessionEvent[])
|
||||
}
|
||||
}
|
||||
|
||||
async commitRepair(m: SessionHeader, _tornMarker: undefined, closers: readonly SessionEvent[]): Promise<void> {
|
||||
// No torn tails in a Map store, so `_tornMarker` is always undefined; only the
|
||||
// synthetic closers are appended (the same DELETE+INSERT a DB backend does,
|
||||
// minus the truncate).
|
||||
const entry = this.store.get(m.id)
|
||||
/* v8 ignore next -- commitRepair only runs for a materialized (stored) session */
|
||||
if (!entry) return
|
||||
if (closers.length > 0) entry.events.push(...structuredClone(closers) as SessionEvent[])
|
||||
}
|
||||
|
||||
async list(): Promise<SessionHeader[]> {
|
||||
return [...this.store.values()].map(e => structuredClone(e.meta))
|
||||
}
|
||||
}
|
||||
|
||||
// Run the shared contract against the in-memory backend.
|
||||
runPersistenceContract('memory', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(MemoryPersistence)
|
||||
return {
|
||||
persistence: ctx.sessionPersistence,
|
||||
dispose: async () => { await fiber.dispose() },
|
||||
}
|
||||
})
|
||||
|
||||
// Run the shared coordinator orchestration suite against the in-memory backend.
|
||||
// A per-fixture Map is the shared "storage", so two mounted instances see the
|
||||
// same materialized sessions (HMR/reload). `corruptTail` is OMITTED: a Map store
|
||||
// writes atomically in RAM and has no torn tails, so the suite's torn-tail test
|
||||
// self-skips (and asserts the omission). The real torn-tail repair branch is
|
||||
// covered by the jsonl/sqlite fixtures, which CAN inject one.
|
||||
runCoordinatorContract('memory', async (): Promise<CoordinatorFixture> => {
|
||||
const store: MemoryStore = new Map()
|
||||
return {
|
||||
mount: async ctx => ctx.plugin(MemoryPersistence, { store }),
|
||||
cleanup: async () => { store.clear() },
|
||||
}
|
||||
})
|
||||
|
||||
describe('SessionPersistence service registration', () => {
|
||||
it('registers as ctx.sessionPersistence and is removed on fiber dispose (HMR safety)', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(MemoryPersistence)
|
||||
expect(ctx.sessionPersistence).toBeInstanceOf(SessionPersistence)
|
||||
|
||||
await fiber.dispose()
|
||||
expect(ctx.sessionPersistence).toBeUndefined()
|
||||
})
|
||||
|
||||
it('round-trips through the registered service instance', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
const fiber = await ctx.plugin(MemoryPersistence)
|
||||
const m = meta('reg')
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
const loaded = await ctx.sessionPersistence.load(m.id)
|
||||
expect(loaded.events).toHaveLength(6)
|
||||
await fiber.dispose()
|
||||
})
|
||||
})
|
||||
|
||||
describe('shared persistence helpers', () => {
|
||||
it('accepts a seed that reproduces the persisted prefix exactly', () => {
|
||||
const log = oneTurnLog()
|
||||
expect(seedCoversPrefix(log, log.slice(0, 3))).toBe(true)
|
||||
expect(seedCoversPrefix(log, [])).toBe(true)
|
||||
})
|
||||
|
||||
it('rejects a prefix longer than the seed', () => {
|
||||
const log = oneTurnLog()
|
||||
expect(seedCoversPrefix(log.slice(0, 2), log)).toBe(false)
|
||||
})
|
||||
|
||||
it('rejects a same-envelope event with mutated data', () => {
|
||||
const log = oneTurnLog()
|
||||
const tampered = structuredClone(log)
|
||||
const event = tampered[1]!
|
||||
tampered[1] = {
|
||||
...event,
|
||||
data: { ...event.data, content: [{ type: 'text', text: 'tampered' }] },
|
||||
} as SessionEvent
|
||||
expect(seedCoversPrefix(tampered, log.slice(0, 2))).toBe(false)
|
||||
})
|
||||
|
||||
it('accepts JSON-serializable event data', () => {
|
||||
expect(() => { assertSerializable(oneTurnLog()) }).not.toThrow()
|
||||
})
|
||||
|
||||
it('rejects non-JSON-serializable event data with type and seq context', () => {
|
||||
const bad = [
|
||||
{ type: 'user/message', seq: 0, time: 1, data: { content: 1n } },
|
||||
] as unknown as SessionEvent[]
|
||||
expect(() => { assertSerializable(bad) }).toThrow(/"user\/message".*seq 0/)
|
||||
})
|
||||
})
|
||||
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -1,106 +0,0 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import { SessionId, isJsonValue, interruptedTurnClosers } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionEvent, SessionMeta, SessionSummary } from '@deepseek-ai/dsh-session'
|
||||
import { SessionPersistence } from '../src/index.ts'
|
||||
import { runPersistenceContract, meta, oneTurnLog } from './contract.ts'
|
||||
|
||||
/**
|
||||
* A minimal in-memory {@link SessionPersistence} used to (a) cover the abstract
|
||||
* base's constructor + service registration and (b) validate the reusable
|
||||
* contract suite itself. The real durable backend is
|
||||
* `@deepseek-ai/dsh-session-persistence-jsonl`.
|
||||
*/
|
||||
class MemoryPersistence extends SessionPersistence {
|
||||
private store = new Map<string, { meta: SessionMeta; events: SessionEvent[] }>()
|
||||
private pending = new Map<string, SessionMeta>()
|
||||
|
||||
async create(m: SessionMeta): Promise<void> {
|
||||
// Lazy: record the intended meta, but stay absent from has/list until the
|
||||
// first append materializes the session.
|
||||
this.pending.set(m.id, m)
|
||||
}
|
||||
|
||||
async append(id: SessionId, events: readonly SessionEvent[]): Promise<void> {
|
||||
const existing = this.store.get(id)
|
||||
const nextSeq = existing ? existing.events.length : 0
|
||||
if (events.length > 0 && events[0]!.seq !== nextSeq) {
|
||||
throw new Error(`append seq mismatch for "${id}": expected ${nextSeq}, got ${events[0]!.seq}`)
|
||||
}
|
||||
for (let i = 0; i < events.length; i++) {
|
||||
const e = events[i]!
|
||||
if (e.seq !== nextSeq + i) throw new Error(`non-contiguous seq in batch for "${id}" at index ${i}`)
|
||||
if (!isJsonValue(e.data)) {
|
||||
throw new Error(`event "${e.type}" carries non-JSON-serializable data`)
|
||||
}
|
||||
}
|
||||
if (!existing) {
|
||||
const m = this.pending.get(id)
|
||||
if (!m) throw new Error(`append before create for "${id}"`)
|
||||
this.store.set(id, { meta: m, events: structuredClone(events) as SessionEvent[] })
|
||||
} else {
|
||||
existing.events.push(...structuredClone(events) as SessionEvent[])
|
||||
}
|
||||
}
|
||||
|
||||
async load(id: SessionId): Promise<{ meta: SessionMeta; events: SessionEvent[] }> {
|
||||
const entry = this.store.get(id)
|
||||
if (!entry) throw new Error(`session "${id}" not found`)
|
||||
// Honor the crash-recovery contract: if the stored log ends mid-turn, close
|
||||
// the orphaned turn durably with synthetic boundary events and continue from
|
||||
// the balanced length.
|
||||
const closers = interruptedTurnClosers(entry.events)
|
||||
if (closers.length > 0) entry.events.push(...structuredClone(closers))
|
||||
return { meta: structuredClone(entry.meta), events: structuredClone(entry.events) }
|
||||
}
|
||||
|
||||
async list(): Promise<SessionMeta[]> {
|
||||
return [...this.store.values()].map(e => structuredClone(e.meta))
|
||||
}
|
||||
|
||||
async has(id: SessionId): Promise<boolean> {
|
||||
return this.store.has(id)
|
||||
}
|
||||
|
||||
async delete(id: SessionId): Promise<void> {
|
||||
this.store.delete(id)
|
||||
this.pending.delete(id)
|
||||
}
|
||||
|
||||
async update(id: SessionId, summary: Partial<SessionSummary>): Promise<void> {
|
||||
const entry = this.store.get(id)
|
||||
if (entry) Object.assign(entry.meta, summary)
|
||||
}
|
||||
}
|
||||
|
||||
// Run the shared contract against the in-memory backend.
|
||||
runPersistenceContract('memory', async () => {
|
||||
const ctx = new Context()
|
||||
const fiber = await ctx.plugin(MemoryPersistence)
|
||||
return {
|
||||
persistence: ctx.sessionPersistence,
|
||||
dispose: async () => { await fiber.dispose() },
|
||||
}
|
||||
})
|
||||
|
||||
describe('SessionPersistence service registration', () => {
|
||||
it('registers as ctx.sessionPersistence and is removed on fiber dispose (HMR safety)', async () => {
|
||||
const ctx = new Context()
|
||||
const fiber = await ctx.plugin(MemoryPersistence)
|
||||
expect(ctx.sessionPersistence).toBeInstanceOf(SessionPersistence)
|
||||
|
||||
await fiber.dispose()
|
||||
expect(ctx.sessionPersistence).toBeUndefined()
|
||||
})
|
||||
|
||||
it('round-trips through the registered service instance', async () => {
|
||||
const ctx = new Context()
|
||||
const fiber = await ctx.plugin(MemoryPersistence)
|
||||
const m = meta('reg')
|
||||
await ctx.sessionPersistence.create(m)
|
||||
await ctx.sessionPersistence.append(m.id, oneTurnLog())
|
||||
const loaded = await ctx.sessionPersistence.load(m.id)
|
||||
expect(loaded.events).toHaveLength(6)
|
||||
await fiber.dispose()
|
||||
})
|
||||
})
|
||||
@@ -1,13 +0,0 @@
|
||||
{
|
||||
"extends": "../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib"
|
||||
},
|
||||
"include": ["src"],
|
||||
"references": [
|
||||
{ "path": "../../vendor/cosmokit" },
|
||||
{ "path": "../../vendor/cordis" },
|
||||
{ "path": "../session" }
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user