Merge remote-tracking branch 'origin/master' into session-surface
Reconcile the session-surface feature with master's package reorg and simplifications: - Adopt master's folded usage (assistant/message.usage; standalone `usage` event dropped) and re-attach surface metadata (surfaceOp/sourceEventSeqs). - Add surface opts to master's new max-tokens assistant/message append. - Port surface columns onto the coordinator-refactored SQLite backend at its new path; drop the dead v1->v2 migration (bump-and-reject, no migration per pre-release policy). - Move the session-surface RFC into implemented/architecture/ and refresh its stale body (no migration, SESSION_FORMAT_VERSION=0, renamed package paths). - Update the core-data-structures catalog SessionEvent blocks for the two new surface fields; regenerate the cordis catalog. - Re-harvest ACP snapshot fixtures (keyless replay) to carry surface metadata.
This commit is contained in:
Hypatia May
@@ -0,0 +1,21 @@
|
||||
# RFC: Provider-neutral content-block vocabulary owned by dsh-llm
|
||||
|
||||
Status: implemented (accepted 2026-06-11)
|
||||
|
||||
<!-- XXX: legacy ADR/RFC body format, not yet normalized to a unified RFC template. -->
|
||||
|
||||
## Context
|
||||
|
||||
The harness needs one internal language for messages that the loop, session log, and all plugins speak. Options: mirror the DeepSeek/OpenAI chat-completions shape (zero mapping for the first provider, awkward for rich content), adopt Anthropic's Messages block structure verbatim (battle-tested, but our canonical types would mirror a third-party API we don't target first), or own a vocabulary.
|
||||
|
||||
## Decision
|
||||
|
||||
Own it: messages are arrays of typed content blocks (`text`, `reasoning`, `tool-call`, `tool-result`, `image`), with the union derived from the merge-extensible `ContentBlockMap` so plugins add block types via declaration merging. The same merge-extensible-map pattern types every "stringly" field (`MessageSource`, `FinishReason`, `TurnTrigger`, `TurnEndReason`). Streaming is a raw chunk protocol; `BlockAssembler` is the single shared assembly implementation. Adapters translate to provider wire formats — mapping cost lives in adapters, where it belongs.
|
||||
|
||||
In-session context injection (`context/message`, `steering/message`) renders as tagged user-role envelopes (the system-reminder pattern) rather than a new role, so adapters carry zero burden. Live-adapter review has since validated the tagged-envelope rendering against current DeepSeek behavior; a future provider-specific mismatch should be handled in that adapter rather than by adding a new role to the canonical content vocabulary.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Reasoning, prefill, cache hints, and multimodal content all have a home without provider contortions.
|
||||
- Every adapter pays a translation cost; the first real adapters have since validated the streaming protocol, and new adapters should continue proving their provider-specific mapping in adapter-local tests.
|
||||
- IDs that cross package boundaries are branded (`CallId`, `SessionId`, `AgentId`) — nominal typing at zero runtime cost.
|
||||
@@ -0,0 +1,21 @@
|
||||
# RFC: Custom typed tool-schema DSL instead of schemastery
|
||||
|
||||
Status: implemented (accepted 2026-06-11)
|
||||
|
||||
<!-- XXX: legacy ADR/RFC body format, not yet normalized to a unified RFC template. -->
|
||||
|
||||
## Context
|
||||
|
||||
Tool parameters must reach the model as standard JSON Schema (the wire format), and tool authors deserve typed `execute(args)` without casts. The repo already vendors schemastery (used for plugin Config), so reusing it was the obvious candidate. The user also explicitly preferred per-property `required: true` booleans over JSON Schema's separate `required` array.
|
||||
|
||||
## Decision
|
||||
|
||||
A small custom DSL in dsh-tools: `SchemaSpec` (per-property specs with `required: true` booleans), type-level `InferArgs<S>` mapping a spec to the argument type (required keys non-optional, others genuinely optional via `?`), a runtime `schemaSpecToJsonSchema()` converter, and `defineTool()` tying them together. Raw JSON-Schema `ToolDefinition`s remain accepted by `ToolRegistry.register()` — that's how MCP-sourced tools arrive.
|
||||
|
||||
Schemastery was evaluated and rejected for this use: it targets validation / transformation against StandardSchema, not JSON Schema *generation*, so it would add indirection without producing the wire format cleanly.
|
||||
|
||||
## Consequences
|
||||
|
||||
- First-party tool authors get zero-cast typed args; the type gymnastics cost stays inside the core package (sanctioned by the AGENTS.md type-safety policy).
|
||||
- The DSL is deliberately small (string/number/boolean/object/array, enum, default, nested properties/items). Gaps vs full JSON Schema (unions, formats, constraints) are accepted until real tools demand them.
|
||||
- The InferArgs mapping is regression-tested at the type level (expectTypeOf) after an early optionality bug shipped and was caught by review.
|
||||
@@ -0,0 +1,29 @@
|
||||
# RFC: Dev-mode invariants over compile-time deep-readonly
|
||||
|
||||
Status: implemented (accepted 2026-06-13)
|
||||
|
||||
<!-- XXX: legacy ADR/RFC body format, not yet normalized to a unified RFC template. -->
|
||||
|
||||
## Context
|
||||
|
||||
The session log is append-only by contract, but the types don't enforce it: `session.events` returns `readonly SessionEvent[]` whose *elements* are mutable, and `deriveMessages()` handed the logged `content` arrays/blocks out by reference. The loop then passes those derived messages into the `agent/request` waterfall and on to adapters, where mutating the request is sanctioned — so a request middleware could reach back and rewrite history, silently breaking replay equivalence and the derived-history guarantee. Separately, the event taxonomy (turn/step nesting, seq monotonicity, tool-call/result pairing, legal status transitions) was asserted only where individual tests happened to look.
|
||||
|
||||
Two ways to defend the log: make immutability part of the type (`DeepReadonly<SessionEvent>` on the way out), or catch corruption at runtime in dev. The runtime-validation proposal took the runtime route; [the deep-readonly proposal](../../rejected/architecture/2026-06-11-immutable-public-surfaces.md) took the type route.
|
||||
|
||||
## Decision
|
||||
|
||||
Reject the pervasive `DeepReadonly<T>` type flip. Instead:
|
||||
|
||||
1. **Always-on:** `deriveMessages()` deep-clones the content it emits (one `structuredClone` per derived message). In-flight mutation of a request can no longer reach the log — this is the real fix, and it costs nothing meaningful next to a model call.
|
||||
2. **Dev-mode:** a new `dsh-invariants` plugin (pure listeners, off in production, on in tests and demos) asserts the event contract and `Object.freeze`s logged event data so any *other* code that mutates a logged event throws instead of corrupting silently. Seeded sessions are frozen and checked on `session/created` (the constructor copies the seed without emitting `session/event`).
|
||||
|
||||
The invariants encode the *real* contract, not an idealized one: a `tool/call` may have no `tool/result` (a thrown `tools/execute` waterfall ends the step), and both `idle→disposed` and `running→disposed` are legal.
|
||||
|
||||
`DeepReadonly` was rejected because it is compile-time only (a plugin casts straight through it), high type-noise across every log/message consumer and adapter, and would force readonly types through code where mutation is the sanctioned API. The clone draws the mutable/immutable boundary exactly at "logged vs in-flight" without any of that noise.
|
||||
|
||||
## Consequences
|
||||
|
||||
- History corruption is caught loudly in tests and demos, at zero production cost and zero type noise. The trade-off is that the guarantee is dynamic (a dev-mode tripwire) rather than static.
|
||||
- The invariants plugin doubles as executable documentation of the event taxonomy — the assertions are the contract.
|
||||
- `Session.events` keeps its `readonly SessionEvent[]` type; no consumer churn.
|
||||
- This folds in [the deep-readonly proposal](../../rejected/architecture/2026-06-11-immutable-public-surfaces.md) — there is no separate deep-readonly record; this records the decision to *not* pursue that approach. `InvariantError` is a plain `Error` with a `code` for now; a later taxonomy change can promote it.
|
||||
@@ -0,0 +1,24 @@
|
||||
# RFC: Event-sourced sessions with derived message history
|
||||
|
||||
Status: implemented (accepted 2026-06-11)
|
||||
|
||||
<!-- XXX: legacy ADR/RFC body format, not yet normalized to a unified RFC template. -->
|
||||
|
||||
## Context
|
||||
|
||||
The MVP requires strict event-based tracing with fully replayable sessions (严格的基于事件的trace、logging系统，session完全可回放). Two models were considered: a mutable message array with events fired as notifications (simpler, but state and log can diverge), or event-sourcing where the log IS the state.
|
||||
|
||||
## Decision
|
||||
|
||||
A `Session` is an append-only log of typed `SessionEvent`s — the single source of truth. The LLM message history is *derived* from the log (`deriveMessages()`); raw stream chunks are logged for token-level replay fidelity while the assembled `assistant/message` event is authoritative for derivation. Replay/fork = seed a new session with an existing log.
|
||||
|
||||
Appends are synchronous (the hot path never blocks on I/O); `session/event` is a sync notification; persistence plugins buffer write-behind and drain at the awaited `session/flush` checkpoint fired at every turn end.
|
||||
|
||||
Ordering contract: the loop appends to the session *before* emitting the corresponding Cordis event, and the `agent/step-result` waterfall runs before the `assistant/message` append so the log records what tool dispatch actually used (post-review fix; regression-tested).
|
||||
|
||||
## Consequences
|
||||
|
||||
- Replay, trace, and telemetry are structurally guaranteed, not bolted on.
|
||||
- Persistence stays a plugin concern; the in-memory store ships in dsh-session.
|
||||
- The event vocabulary is merge-extensible (plugins add e.g. compaction events); it carries a TODO(review) marker until the first persistence plugin and real adapter exercise it.
|
||||
- Derivation cost grows with log length — compaction (future plugin) is the intended mitigation, not log mutation.
|
||||
@@ -0,0 +1,26 @@
|
||||
# RFC: Microkernel — extension via Cordis event taxonomy, one concrete loop
|
||||
|
||||
Status: implemented (accepted 2026-06-11)
|
||||
|
||||
<!-- XXX: legacy ADR/RFC body format, not yet normalized to a unified RFC template. -->
|
||||
|
||||
## Context
|
||||
|
||||
The product principle (see the 微内核Harness实现思路 design doc) is "everything is a plugin": hooks, /goal, /loop, dynamic workflows, compaction, sandboxing, permissions, UI, persistence, MCP, skills must all be writable as plugins without modifying the core. Candidate mechanisms considered: a purpose-built middleware stack (koa-compose style), an explicit phase state machine plugins can insert into, or Cordis's native event system.
|
||||
|
||||
## Decision
|
||||
|
||||
Pure Cordis event taxonomy. The loop's extension seams are typed events with deliberate dispatch modes:
|
||||
|
||||
- **waterfall** (around-middleware) where plugins mutate or veto: `agent/request`, `agent/step-result`, `agent/turn-continuation`, `tools/execute`, `llm/stream`, `system-prompt/assemble`.
|
||||
- **emit** (sync fire-and-forget) for notifications: turn/step boundaries, stream chunks, lifecycle, errors.
|
||||
- **parallel** (awaited) for the one durability checkpoint: `session/flush`.
|
||||
|
||||
The event vocabulary lives in interface packages (dsh-agent declares the agent/* events); `@deepseek-ai/dsh-agent-loop` is the only concrete loop plugin and is itself swappable — nothing outside it may depend on it.
|
||||
|
||||
## Consequences
|
||||
|
||||
- Every MVP feature maps to a listener (the "plugin sanity checklist" in docs/architecture.md is the proof obligation, kept current).
|
||||
- HMR and disposal come free: listeners and registrations are Cordis effects.
|
||||
- Waterfall semantics (call `next()` or short-circuit) are non-obvious and must be taught — documented in AGENTS.md and covered by composition tests.
|
||||
- The loop must be defensive: plugin exceptions are contained at turn level, steering from any seam is never stranded (regression-tested).
|
||||
@@ -0,0 +1,22 @@
|
||||
# RFC: Runtime arg validation at the model boundary
|
||||
|
||||
Status: implemented (accepted 2026-06-13)
|
||||
|
||||
<!-- XXX: legacy ADR/RFC body format, not yet normalized to a unified RFC template. -->
|
||||
|
||||
## Context
|
||||
|
||||
`defineTool` ([the custom schema DSL](2026-06-11-custom-schema-dsl.md)) gives tool authors a typed `execute(args)` via the `InferArgs<S>` mapping. But that type is a compile-time claim about a value that arrives at runtime as model-generated JSON: nothing forced the model to honor the schema, so a malformed call — missing a required key, a string where a number was declared, an enum value outside the set — reached `execute` typed-in-name-only. The tool body then either crashed on the bad shape (a generic stack trace the model can't act on) or, worse, silently misbehaved. Meanwhile the converter already encodes the exact structure a validator would need to walk.
|
||||
|
||||
## Decision
|
||||
|
||||
`validateArgs(spec, args): string[]` interprets a `SchemaSpec` over a runtime value, returning human-readable violations (empty = valid), and is total (never throws). `defineTool` runs it before the typed body; on violations it throws `ToolArgsError` (`code: 'INVALID_ARGS'`, message listing the violations), which the registry's existing execute-waterfall catch turns into an `isError` result the model reads and self-corrects from.
|
||||
|
||||
The validator mirrors `schemaSpecToJsonSchema` semantics exactly — same structure walked, same rules: top level must be a non-array object; required keys come only from `required: true`; extra keys are allowed (no `additionalProperties: false`); `default` is not applied; an `object`/`array` prop without `properties`/`items` only type-checks; `enum` is membership. Raw-registered (MCP) tools are not touched — they validate their own input.
|
||||
|
||||
## Consequences
|
||||
|
||||
- The model gets actionable feedback on its own malformed calls instead of an opaque crash, closing the gap between `InferArgs`'s promise and runtime reality.
|
||||
- The validator and `InferArgs` must stay in agreement; that drift risk is to be closed by a property test ([property-based testing](../testing/2026-06-11-property-based-testing.md), not yet landed) generating args that satisfy `InferArgs` and asserting they pass `validateArgs`. Until then the agreement rests on the example tests and the shared converter structure.
|
||||
- `ToolArgsError` is a plain `Error` with a `code` field for now; if a harness-wide error taxonomy lands it becomes a subclass without changing callers that read `.message`.
|
||||
- Validation cost is negligible next to a model call.
|
||||
@@ -0,0 +1,26 @@
|
||||
# RFC: Structured error taxonomy
|
||||
|
||||
Status: implemented (accepted 2026-06-14)
|
||||
|
||||
<!-- XXX: legacy ADR/RFC body format, not yet normalized to a unified RFC template. -->
|
||||
|
||||
## Context
|
||||
|
||||
Failures crossed seams as bare strings. A tool error flattened to a text block — name, code, and stack lost — so a future sandbox/retry plugin couldn't tell ENOENT from EACCES, and the model got less actionable feedback than it could. A non-Error throw degraded further: the loop wrapped it in `new Error(String(x))`, dropping any code. And `LlmError` was the only typed error in the system, with no shared base, so there was nothing for a consumer to `instanceof` against generically.
|
||||
|
||||
This is the last of the runtime-validation / error-taxonomy pieces and the one the user was most skeptical of, so it was deliberately built **last and in isolation**: the earlier PRs (arg validation, dev invariants) threw plain `Error`s with a `code` field, decoupled from any shared base, so this change is a pure upgrade and is independently revertible without unpicking them.
|
||||
|
||||
## Decision
|
||||
|
||||
A single `HarnessError extends Error` base in `dsh-llm` (the leaf package every other imports — no new dependency edge): a stable `code` distinct from `message`, `cause` chaining via `ErrorOptions`, and `name` defaulting to the subclass. `isHarnessError` narrows at seams.
|
||||
|
||||
- `LlmError`, `ToolArgsError` (dsh-tools), and `InvariantError` (dsh-invariants) now extend it, keeping their existing codes.
|
||||
- `ToolExecutionResult` gains optional `error: { name, code }`, populated in the registry's catch when the thrown value is a `HarnessError`. The agent loop forwards it onto the `tool/result` session event (which gained the same optional field), so the structured failure survives into the log for retry/sandbox plugins and replay. The model-facing text block is unchanged.
|
||||
- The loop's `toError` wraps a non-Error throw in a `HarnessError` (`code: 'UNKNOWN'`, original chained as `cause`) instead of a bare `Error`, so even a bad throw carries a routable code into the session `error` event (which already surfaced `code`).
|
||||
|
||||
## Consequences
|
||||
|
||||
- Errors are machine-routable end-to-end: a plugin can branch on `error.code` rather than substring-matching a message.
|
||||
- One base class is imported widely, but it lives in the package everyone already depends on, so the cost is a single import, not a new edge.
|
||||
- `deriveMessages` does not surface `error` into model history — the model still sees the text block; the structured field is for code and replay.
|
||||
- Reverting this PR returns the earlier errors to plain `Error`+`code` form; nothing else in the stack depends on the shared base.
|
||||
@@ -0,0 +1,19 @@
|
||||
# RFC: Tool schemas are part of the system-prompt assembly
|
||||
|
||||
Status: implemented (accepted 2026-06-11)
|
||||
|
||||
<!-- XXX: legacy ADR/RFC body format, not yet normalized to a unified RFC template. -->
|
||||
|
||||
## Context
|
||||
|
||||
On the wire, tool schemas travel in a dedicated `tools` field of the model request, not in prompt text. Architecturally, though, "what the model is told it can do" is one coherent concern: prompt sections and the tool list are assembled from the same plugin contributions and consumed at the same moment. The alternative — the loop querying the tool registry separately from the prompt service — splits one concern across two seams.
|
||||
|
||||
## Decision
|
||||
|
||||
`PromptAssembly { sections, tools }`: the system-prompt service collects ordered text sections AND tool schemas (the tool registry auto-contributes a provider). The loop consumes one assembly per step; adapters map `sections` to the provider's system slot and `tools` to the wire `tools` field. The `system-prompt/assemble` waterfall is therefore a single interception point for everything the model is told up front — tool filtering (ToolSearch / progressive disclosure) is an assembly rewrite, same as prompt edits.
|
||||
|
||||
## Consequences
|
||||
|
||||
- One waterfall governs the model's standing context; plugins like plan mode can swap prompt text and visible tools in one listener.
|
||||
- The assembly interface is merge-extensible for future slots (no untyped `extras` bag — extension is declaration merging).
|
||||
- Slight conceptual surprise (schemas in a "prompt" service) is documented here and in the package README.
|
||||
@@ -0,0 +1,29 @@
|
||||
# RFC: Capability seams — interface / implementation / consumer split
|
||||
|
||||
Status: implemented (accepted 2026-06-13)
|
||||
|
||||
<!-- XXX: legacy ADR/RFC body format, not yet normalized to a unified RFC template. -->
|
||||
|
||||
## Context
|
||||
|
||||
The harness has swappable capabilities — bash execution today, sandboxed/remote executors and alternative model providers tomorrow. A capability has three concerns that change at different rates and for different reasons: the *contract* (what the capability is), the *implementation* (how it runs), and the *consumer surface* (what the model and other plugins program against). Bundling them in one package couples those rates of change — swapping a local executor for a sandboxed one would churn the tool schemas the model sees, even though the model-facing contract never changed.
|
||||
|
||||
This is distinct from "who provides vs. needs a capability at runtime", which Cordis already answers with services + `inject` (a provider registers `ctx.bash`; a consumer declares `inject: ['bash']` and its fiber pends until the service exists). That mechanism is necessary but doesn't dictate package boundaries; this RFC does.
|
||||
|
||||
## Decision
|
||||
|
||||
A swappable capability is **three packages**:
|
||||
|
||||
1. **Interface** — an abstract service + the vocabulary types, owning the `ctx.<key>` and depending only on cordis (e.g. `dsh-bash`: `BashExecutor`, `BashRunResult`, `BashTask`).
|
||||
2. **Implementation** — a concrete subclass loaded as a plugin (e.g. `dsh-bash-local`: subprocesses, process-group kills, spill-file truncation). Sandboxed/remote backends are sibling packages implementing the same interface.
|
||||
3. **Consumer** — what the model and plugins see (e.g. `dsh-tool-bash`: the `bash`/`bash_output`/`bash_kill` tool schemas). Consumers `inject` the interface key and never import implementation types.
|
||||
|
||||
Implementation and consumer then evolve independently: a sandboxed executor replaces `dsh-bash-local` without touching a tool schema.
|
||||
|
||||
Alternatives considered: **one combined package** — rejected because it recouples the three rates of change the split exists to separate (the whole point). **`@cordisjs/plugin-capability`** — a different axis entirely: it is a permission/capability-*security* service (named permissions with inheritance, tested against a session via `ctx.capability.test`), a candidate for the deferred permissions/sandbox work on the `tools/execute` veto seam, NOT a mechanism for swapping implementations. Confusing the two ("capability") is the trap this RFC names.
|
||||
|
||||
The split is not mandatory when the parts are genuinely one concern: the LLM seam folds interface + consumer into `dsh-llm` (the consumer is the loop itself, not a swappable schema surface) with adapters as the implementation packages. Don't split preemptively — a capability with one conceivable implementation and one consumer stays one package until a second appears.
|
||||
|
||||
## Consequences
|
||||
|
||||
More packages and more boilerplate per capability (a `package.json`/`tsconfig`/README trio, the inject wiring). Bought: implementations and consumers ship and version independently, and a new backend never risks the model-facing contract. The rule is documented in [AGENTS.md](../../../../AGENTS.md) § Conventions ("Capability seams are three packages") and [architecture.md](../../../architecture.md) § "Capability seams"; the bash trio is the reference template. When to fold vs. split is a judgment call the architecture doc spells out — this RFC records *why* the default is to split.
|
||||
@@ -0,0 +1,24 @@
|
||||
# RFC: Two LLM adapters as a design-verification twin
|
||||
|
||||
Status: implemented (accepted 2026-06-13)
|
||||
|
||||
<!-- XXX: legacy ADR/RFC body format, not yet normalized to a unified RFC template. -->
|
||||
|
||||
## Context
|
||||
|
||||
`dsh-llm` owns a provider-neutral streaming vocabulary — the `StreamChunk` protocol (`block-start`, `text-delta`, `reasoning-delta`, `tool-call-delta`, `block-end`, `usage`, `finish`) and the content-block types ([the content-block vocabulary](2026-06-11-content-block-vocabulary.md)). A vocabulary defined against a single adapter risks baking that adapter's quirks into the "neutral" contract: anything the one implementation happens to do becomes the de-facto spec, and the abstraction is unverified until a second provider arrives — by which point the leak is expensive to fix.
|
||||
|
||||
## Decision
|
||||
|
||||
Ship **two** adapters against the one contract from the start, deliberately built on different internals:
|
||||
|
||||
- `dsh-llm-deepseek` — hand-rolled `fetch` + SSE parsing against the DeepSeek API.
|
||||
- `dsh-llm-pi-ai` — the same endpoint through the `@earendil-works/pi-ai` library (its own event vocabulary).
|
||||
|
||||
The rule they enforce: **anything the StreamChunk vocabulary cannot express for BOTH implementations is a core-vocabulary bug**, caught immediately rather than at the next provider. The pair pinned down conventions now documented on `StreamChunk` in `dsh-llm/src/types.ts`: usage emitted before finish, nothing after finish, tool-call `arguments` as raw JSON strings end-to-end, and the two sanctioned error paths (throw from `stream()` *or* end with `finish {kind:'error'|'aborted'}`) that a consumer must handle on both sides — a divergence the library-backed adapter surfaced that a single hand-rolled adapter would have hidden.
|
||||
|
||||
Alternatives considered: **a single adapter** — less code and half the e2e cost, but leaves the "provider-neutral" claim unverified; the vocabulary would encode DeepSeek-via-fetch assumptions silently. **A mock second adapter** — cheaper but doesn't exercise a real provider's wire quirks, so it proves little. The twin is real-on-real.
|
||||
|
||||
## Consequences
|
||||
|
||||
Double the adapter maintenance and double the key-gated e2e surface (both adapters cover V4 Flash and Pro across representative thinking/effort modes). Bought: a continuously-verified neutrality guarantee for the most leak-prone abstraction in the codebase, and a worked second example for adapter authors. The two share the core Config shape (`apiKey`/`baseURL`/`models`) so a deployment swaps mostly one line, but the reasoning knob differs — `dsh-llm-deepseek` takes `thinking`/`reasoningEffort`, `dsh-llm-pi-ai` takes a single `reasoning` level — so a swap translates that field. If the maintenance cost ever outweighs the verification value (e.g. once conformance tests from [architectural conformance](../../proposed/process/2026-06-11-architectural-conformance.md) cover the contract mechanically), retiring the twin to a single adapter + the conformance kit would be a new RFC superseding this one.
|
||||
@@ -0,0 +1,34 @@
|
||||
# RFC: Session persistence as an abstract service over the existing `SessionEvent`
|
||||
|
||||
Status: implemented (proposed 2026-06-14, accepted 2026-06-15)
|
||||
|
||||
<!-- XXX: legacy ADR/RFC body format, not yet normalized to a unified RFC template. -->
|
||||
|
||||
> Merges the original proposal and the decision record for one topic. The proposal's full method-surface and write-path detail lives in git history; this records the decision and the durable, contested choices.
|
||||
|
||||
## Context
|
||||
|
||||
Sessions lived only in memory. The example `session-jsonl.ts` plugin (duplicated byte-for-byte in both examples) was write-only telemetry: it buffered `session/event` and appended JSON lines, with no read/replay path, no crash-safety (no fsync, no atomic write, a fire-and-forget dispose drain), no listing, and no format versioning. Nothing could rehydrate a past session from disk into a live agent, so durable resume ("continue yesterday's task"), durable forking, and the ACP `session/load` method ([ACP support](../../proposed/feature/2026-06-14-acp-agent-client-protocol.md)) were all impossible.
|
||||
|
||||
The [event-sourced model](2026-06-11-event-sourced-sessions.md) makes the append-only log the single source of truth and derives LLM history from it. Persistence had to stay faithful to that: persist the existing `SessionEvent` directly, with no parallel "persisted message" type that the log is converted to and from. The backend also had to be swappable — a file store now, a database store later — behind one interface.
|
||||
|
||||
## Decision
|
||||
|
||||
Persistence is an abstract **capability seam** ([capability seams](2026-06-13-capability-seams.md), the `dsh-bash` template), not loop or core logic:
|
||||
|
||||
1. **Interface** (`dsh-session-persistence`, `ctx.sessionPersistence`) — an abstract `SessionPersistence` service: `create`/`append`/`load`/`list`. Its persisted unit IS the existing `SessionEvent` (`{ type, seq, time, data }`), reused verbatim — no conversion type.
|
||||
2. **Implementation** (`dsh-session-persistence-jsonl`) — an append-only JSONL log per session (a `SessionHeader` line then one `SessionEvent` per line, verbatim **including `assistant/chunk`**).
|
||||
|
||||
Key choices recorded here because they are durable, contested, and surprising:
|
||||
|
||||
- **The canonical durable log persists every `SessionEvent` verbatim, including `assistant/chunk`.** `deriveMessages()` skips chunks, and a chunk-filtered rollout (Codex's `policy.rs`) is tempting — but `seq = log.length` and the load-validation `events[i].seq === i` require a *contiguous* log; filtering chunks out would leave holes and break both the contract and resume. A chunk-filtered projection is possible later as a derived view with its own renumbering, but it is NOT the canonical log.
|
||||
- **Append-only; a crashed turn is closed, never truncated.** Committed events — those at or below a flushed `turn/end` — are never rewritten. The loop only flushes at `turn/end`, so a crash can leave a durable log whose final turn never closed: real, fully-written events sit after the last `turn/end`. **A single turn can be huge in a long-horizon task** (many steps, large tool output spanning a long autonomous run), so discarding the interrupted turn would silently destroy a large amount of real work — truncating a turn is wrong. Instead, on reload `load` PRESERVES those events and CLOSES the orphaned turn by durably appending the minimal synthetic boundary events: an error `tool/result` for every `tool-call` the crash left unanswered, then a `step/end` if a step was still open, then a `turn/end` carrying the merge-extensible `{ kind: 'interrupted' }` reason (a marker that records the turn was cut short by a crash, not completed by the model — no loop ever emits it). The synthetic tool results matter for resume correctness: the loop logs the `assistant/message` (carrying the `tool-call` blocks) BEFORE running the tools, so a crash mid-tool leaves calls without results; `deriveMessages()` would then replay a dangling assistant tool-call, which every provider rejects as an invalid transcript on the next request. Answering each orphaned call with an error result keeps the rehydrated history valid. `load` returns the balanced log, so a resumed session is immediately usable. The ONLY thing discarded is a never-fully-written **torn tail fragment** — a final record whose bytes (JSONL) or row were never completely flushed; that fragment is not a valid event and is dropped before the synthetic closers are written. A parse error or `seq` gap in the COMMITTED region (at or before the last real `turn/end`) is genuine corruption and makes the session unloadable.
|
||||
- **File backend canonical, DB backend a proven drop-in.** `SessionEvent` maps 1:1 onto a row `(session_id, seq, type, time, data)` — `append` is INSERT (in a transaction asserting the contiguous-seq contract), `load` is SELECT … ORDER BY seq. `dsh-session-persistence-sqlite` is exactly this: a `SessionPersistence` subclass with no interface change (opencode runs this exact shape on SQLite/WAL), and it passes the same `runPersistenceContract` suite as the JSONL backend — so the contract holds both backends to identical semantics (lazy materialization, interrupted-turn close on load, contiguous-seq), expressed once over file bytes and once over rows.
|
||||
- **Metadata is out-of-log.** Format version, cwd, and lineage are storage concerns, not replayable conversation state, so they live in a `SessionHeader` owned by `dsh-session` and attached to a `Session` via a new readonly `session.header` — never in `SessionEventMap`, never reaching `deriveMessages()`. The alternative (a merge-extensible `session/meta` event as log line 0) was rejected: an in-log event would ride along with a seeded/forked session for free, but metadata is not replayable state, so the explicit out-of-log header seam is the cleaner cost. (The header was originally split into an immutable `SessionHeader` plus a mutable `SessionSummary` whose union was `SessionMeta`; the mutable summary was later removed as dead state — see [Drop the mutable session summary](../simplification/2026-06-19-drop-mutable-session-summary.md).)
|
||||
- **Resume is an async factory, not a change to synchronous create.** `ctx.agents.resume({ resumeSessionId })` awaits `ctx.sessionPersistence.load`, recreates the live session with the loaded events (so `lastTurnNumber`/`deriveMessages` continue), and starts a fresh agent on the resumed id (NOT `${agentId}-session`). The agent-loop does NOT hard-inject `sessionPersistence` (that would pend non-persistent demos forever); `resume` rejects with a clear error when it is absent.
|
||||
|
||||
Format versioning: the header carries a `version`; `load` rejects any non-current version (no migration — the pre-release session format is pinned at `SESSION_FORMAT_VERSION = 0` and absorbs shape churn, per the AGENTS.md pre-release stance). Stated honestly: append-only + flush is robust to partial trailing writes (tolerated on load) but not to fsync-less power loss mid-line; a DB/WAL backend is the stronger option later.
|
||||
|
||||
## Consequences
|
||||
|
||||
Two new packages and the metadata seam in `dsh-session` (`session.header`, the `create(id?, options?)` signature). Bought: durable resume/fork, a read/replay path, crash tolerance, and the foundation the ACP `session/load` ([ACP support](../../proposed/feature/2026-06-14-acp-agent-client-protocol.md)) needs — all over the existing event-sourced log, with the backend swappable behind one interface. The reusable `runPersistenceContract` suite holds every backend to the same append-only / contiguous-seq / lazy-materialization / serializability semantics. This completes [event-sourced sessions](2026-06-11-event-sourced-sessions.md)'s deferred "real persistence backend" and resolves its `TODO(review)` on the event vocabulary: persisting the log freezes its shape, and the `assistant/chunk` fidelity question is answered above (persist verbatim).
|
||||
@@ -0,0 +1,40 @@
|
||||
# RFC: Every session event is enclosed in a turn
|
||||
|
||||
Status: implemented (accepted 2026-06-15)
|
||||
|
||||
<!-- XXX: legacy ADR/RFC body format, not yet normalized to a unified RFC template. -->
|
||||
|
||||
## Context
|
||||
|
||||
A durable session-persistence backend (added in a companion change) uses the **turn** as its crash-recovery boundary: a crash can leave an unclosed final turn, which `load` closes with a synthetic `turn/end {kind:'interrupted'}` while preserving the turn's real events (see [session persistence](2026-06-14-session-persistence.md)). This recovery is only well-defined if nothing *legitimately* durable sits OUTSIDE a turn — between the last `turn/end` and the next `turn/start` — since such an event would be swept into the next turn's interrupted close.
|
||||
|
||||
That assumption did not hold. Two paths recorded events outside any turn:
|
||||
|
||||
1. **Queued user messages.** The loop drained queued messages and appended `user/message` *before* `turn/start` — so a turn's own prompt sat in the gap between the previous `turn/end` and the next `turn/start`.
|
||||
2. **Idle context injection.** `agent.inject()` appends a `context/message` directly. Its real production caller is `dsh-tool-bash`, which injects a background-task completion notice from `ctx.bash.onTaskDone` — a callback that fires whenever a background bash task finishes, frequently while the agent is **idle** (between turns).
|
||||
|
||||
In case 2, if the injected `context/message` is the last event before a flush/dispose (no later turn appends a `turn/end`), `scanLog` treats it as crash debris and **drops it on resume** — the injected context is durably on disk but silently lost on reload. Case 1 was benign in isolation (a `user/message` is always followed by the turn it triggered) but made the "what may appear outside a turn" rule fuzzy.
|
||||
|
||||
Two ways to fix it: relax the *reader* (let `scanLog` commit events that sit outside an open turn), or constrain the *producer* (make every event turn-enclosed so the reader's simple "last `turn/end`" rule is both correct and complete). We chose the producer-side invariant: a single, checkable rule beats a more permissive boundary scan that has to reason about partial turns *and* loose between-turn events.
|
||||
|
||||
## Decision
|
||||
|
||||
**Every session event lives inside a turn** — between a `turn/start` and its matching `turn/end`. Concretely:
|
||||
|
||||
- The loop appends queued `user/message` events **after** `turn/start` (inside the turn), not before it. `turn/end` is therefore owed the moment those messages are recorded, and the existing finalizer guarantees it.
|
||||
- An `agent.inject()` made while the agent is **running** appends its `context/message` into the already-open turn (unchanged).
|
||||
- An `agent.inject()` made while **idle** wraps its `context/message` in a one-shot turn: `turn/start{trigger:{kind:'injection'}}` → `context/message` → `turn/end{completed}`. A new `injection` variant joins the merge-extensible `TurnTriggerMap`.
|
||||
- The loop derives the next turn number from the log each iteration (`lastTurnNumber(session) + 1`) instead of keeping a private counter, so an idle injection's one-shot turn cannot collide with the next real turn's number.
|
||||
- The `dsh-invariants` plugin **enforces** the invariant in dev: a `user/message` / `context/message` / `steering/message` appended while no turn is open throws an `InvariantError`.
|
||||
|
||||
The serializability invariant is enforced at the same source boundary (`Session.append` throws on non-JSON-serializable data), so "what may enter the log" is now governed in one place rather than discovered downstream by whichever backend happens to be watching.
|
||||
|
||||
## Consequences
|
||||
|
||||
The turn is now the *single* durability/replay boundary, so [session persistence](2026-06-14-session-persistence.md)'s crash-recovery rule is complete, not merely sufficient: an interrupted final turn is closed (with a synthetic `turn/end {interrupted}`) and its real events preserved, with zero risk of conflating between-turn context into it, because there is no between-turn context. `scanLog` stays simple (one possibly-open final turn, never a loose between-turn event), and an idle background-task notice survives persist + resume.
|
||||
|
||||
Costs: `agent.inject()` while idle now writes three log lines instead of one, and the derived history gains a turn that carries only injected context (no assistant output) — `deriveMessages()` already derives purely by event type, so this renders identically. The `injection` trigger is a new on-disk vocabulary value; like every `SessionEventMap`/`TurnTriggerMap` addition it is part of the frozen format. Event ordering within a turn changed (`turn/start` now precedes `user/message`), which is observable to anything that asserted the old order — the loop's own tests were the only such consumers.
|
||||
|
||||
The rule is intentionally producer-enforced and dev-checked rather than reader-tolerated: a future backend (SQLite/WAL) inherits the same clean boundary for free, and a plugin that records an event outside a turn fails loudly in dev instead of silently losing data on the next reload.
|
||||
|
||||
The invariant also constrains where the loop may record an `error` event. A failure detected while a turn is open is appended INSIDE the turn (before `turn/end`); but a failure that surfaces once the turn is already closed — a rejecting `session/flush` (which runs as the post-`turn/end` durability checkpoint) or a throwing `agent/turn-end` listener (after `closeTurn` already appended `turn/end`) — has no in-turn position left. Appending an `error` there would land it past the last `turn/end`, exactly the crash-tail position a backend discards. So those post-turn failures are reported via the `agent/error` event and the logger only, never as a `SessionEvent`; the turn stays balanced and persistence keeps its buffered events for the next checkpoint. If durable operational diagnostics are ever needed, they belong on a separate telemetry channel, not the replayable session log.
|
||||
@@ -0,0 +1,42 @@
|
||||
# RFC: Agent lifecycle and ownership seams
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
Several ACP and tool-bash limitations were symptoms of the same missing seam: plugins could create or resume agents through `ctx.agents`, but they could not own and dispose one agent independently, and long-running bash tasks carried no stable owner in the executor itself. ACP aborted and awaited agents on disconnect but could not unregister just that session's agent; `session/cancel` could not cancel queued-but-not-yet-started work; and `tool-bash` kept task ownership in a plugin-local `Map`, so an HMR reload could make an old task look unowned.
|
||||
|
||||
## What was implemented
|
||||
|
||||
The three seams shipped across a stacked chain of PRs (the queue-aware cancel, the `AgentHandle` disposer, and the bash owner token), each converged independently.
|
||||
|
||||
### 1. Queue-aware `Agent.cancel(reason?)`
|
||||
|
||||
A new `cancel()` verb on the `Agent` interface — the single public stop primitive. (It originally shipped alongside a narrower step-only `abort()`; that verb was later removed as unused, leaving `cancel()` the only public way to stop work.) It clears the inbox's queued + steering FIFOs, aborts the in-flight step if any, and drives a **turn-scoped cancellation marker** the driver loop checks at every turn-decision point — so a prompt that is queued-but-not-yet-started never runs, a cancel landing in the pre-step / continuation window drops the about-to-run turn (ending it `aborted`), and a later prompt cannot be batched into the cancelled turn. `whenIdle()` reaches post-cancel quiescence. ACP `session/cancel` maps to `cancel()`. The marker is armed ONLY when there is something to cancel, so an idle no-op cancel cannot strand the next prompt.
|
||||
|
||||
### 2. `AgentHandle` async disposer
|
||||
|
||||
`ctx.agents.create`/`resume` (and the `AgentFactory` interface) return `AgentHandle = { agent: Agent; dispose(): Promise<void> }`. The disposer is a **capability** — only the holder can tear down exactly this agent: stop its loop, `await` the loop's exit (true quiescence, not just the `disposed` status flip), unregister it, and remove its session from the store. `ctx.agents.get(id)` still returns a bare `Agent`. Config-created agents stay owned by the `AgentLoop` fiber (the handle is discarded). ACP holds each session's disposer in its `SessionRecord` and runs it on disconnect/teardown, so a bare client disconnect leaves no registered agent and no session-store entry — even when `session/load` races teardown (the just-resumed handle is disposed before the closed-guard throw).
|
||||
|
||||
**Teardown ORDER is load-bearing for durability**, and the implementation folds the session lifecycle into the agent's SINGLE composite cordis effect (`SessionStore.prepare`/`enter`/`announce`, replacing a sibling-effect split). A fiber unload disposes sibling effects concurrently (`Promise.all`), which would race the session's `onAppend` detach against the loop's closing `session/flush` and drop the closing `turn/end`; inside one effect the disposers run as an ordered LIFO chain (loop stopped + `await agent.done` BEFORE the session detaches), so the loop's final flush is captured on BOTH the handle's `dispose()` and a fiber unload. The register disposer's `agent/disposed` emit is contained (a throwing listener must not reject the chain and skip the later session detach).
|
||||
|
||||
### 3. Bash owner token in the seam
|
||||
|
||||
Background-task ownership moved from a `tool-bash` plugin-local `Map<string, Agent>` into the executor. `BashExecRequest` gains an optional `owner?: string`; the resolved `BashExecSpec` carries it as required-but-nullable `owner: string | undefined` (a forgotten owner is a visible `undefined`, never a silently-absent property). The executor stores the token on its task and exposes it via a new `BashExecutor.ownerOf(id): string | undefined` seam (NOT on the public `BashTask` — one read path, no redundant API). `tool-bash` deletes its `Map` entirely: it stamps `exec.agent?.session.header.id` as the owner at `start`, and `bash_output`/`bash_kill` compare `ctx.bash.ownerOf(id)` to the caller's token with `!== undefined` semantics (an empty-string token is still a real owner). The completion notice finds the live agent by scanning `ctx.get('agents')?.list()` for `agent.session.header.id === ownerToken` (read via `ctx.get` — `onTaskDone` runs on the bash fiber, a foreign fiber, where the `ctx.agents` proxy would throw). Because ownership now lives on the task in the executor (disposed with the `dsh-bash` fiber), it SURVIVES a `tool-bash` HMR reload — closing the old `XXX(tool-bash-owner-hmr)` gap. (The `onTaskDone` listener is still effect-scoped to `tool-bash`'s `apply`, so a completion landing during the reload gap still drops its one notice — the pre-existing reload-gap drop — but the ownership fence itself is HMR-proof.)
|
||||
|
||||
## Acceptance Criteria (met)
|
||||
|
||||
- ACP disconnect/session close leaves no registered agent AND no session-store entry for that session, even when `session/load` races teardown.
|
||||
- `session/cancel` before a queued prompt starts prevents that prompt from running and cannot batch the next prompt into the cancelled turn.
|
||||
- A `tool-bash` HMR reload does NOT make an existing background task readable or killable by a different session (ownership survives on the executor).
|
||||
- Existing non-ACP demos still work without managing handles explicitly; config-created agents remain owned by the `AgentLoop` plugin fiber.
|
||||
|
||||
## Seam precondition (recorded)
|
||||
|
||||
The bash owner-token comparison relies on `session.header.id` being unique among live agents. The agent registry does NOT enforce this — it rejects a duplicate *agentId*, not a duplicate session id, and `createAgent` accepts an arbitrary `sessionId`. This is NOT reachable via ACP (UUID sessionId, `agentId === sessionId`, duplicate-load rejected), so it is not a live product hole, but a programmatic caller that registers two agents with the same session id would break bash isolation and mis-route the completion notice. The access *policy* (token comparison) stays in `tool-bash` (the consumer); the bash seam stores only an opaque `owner` string and never interprets it — the correct interface/impl/consumer split.
|
||||
|
||||
The planned resolution is to remove the precondition by construction — see [unify the agent id and the session id](../../proposed/simplification/2026-06-20-unify-agent-and-session-id.md): once an agent IS its session (one id), the registry's existing unique-`agentId` check is a unique-session-id guarantee and no two live agents can share a session token.
|
||||
|
||||
## Notes
|
||||
|
||||
This touched public interfaces (`Agent`, `AgentFactory`, the bash seam) deliberately, not as a local ACP patch. The simple synchronous `Agent.send()` ergonomics were preserved; the async lifecycle path is additive, for owners that need it.
|
||||
@@ -0,0 +1,63 @@
|
||||
# RFC: Session surface — a linked list over the event log for LLM message derivation
|
||||
|
||||
Status: implemented (accepted 2026-06-18)
|
||||
|
||||
## Context
|
||||
|
||||
The `Session` event log is the single source of truth ([event-sourced sessions](2026-06-11-event-sourced-sessions.md)), but the only view over it was `deriveMessages()` — a linear scan that filtered and transformed raw events into `Message[]`. This creates problems for session-history-manipulating plugins (compaction, tool-call result pruning, etc.). Without a central mechanism, each plugin would need to wrap `agent/request` to rewrite the message list — a pattern that suffers from listener-ordering fragility, provides no durable record of what was changed, and forces repeated changes to the core `deriveMessages()` whenever a new manipulation is added. A central hub in the `session` package, with a provenance-recording mechanism and enough flexibility for future plugins to manipulate session history through a stable API, lays a solid foundation for plugin development.
|
||||
|
||||
## Decision
|
||||
|
||||
Add a **surface** — a derived, cached linked list of "surface nodes" (the subset of events that produce LLM messages) — maintained by `surfaceOp` markers in the event log.
|
||||
|
||||
### Two new top-level fields on `SessionEvent`
|
||||
|
||||
Every `SessionEvent` gains two optional fields (structural metadata, like `seq`/`time`):
|
||||
|
||||
- **`sourceEventSeqs?: number[]`** — seq numbers of events that are provenance sources (e.g., the `assistant/chunk` seqs that built an `assistant/message`, or the surface nodes shadowed by a compaction marker). Provenance is a core design principle; without it, the replace-range operation cannot be validated on replay.
|
||||
- **`surfaceOp?: SurfaceOp`** — how this event entered the surface. Absent for non-surface events.
|
||||
|
||||
### SurfaceOp: two operations
|
||||
|
||||
```ts
|
||||
export type SurfaceOp =
|
||||
| 'append' // normal tail append
|
||||
| { op: 'replace'; start: number; end: number } // shadow [start, end] inclusive
|
||||
```
|
||||
|
||||
1. **Append** — add a new node to the tail. Used by `user/message`, `assistant/message`, `tool/result`, `context/message`, `steering/message`. The loop passes `surfaceOp: 'append'` on all such appends, and `sourceEventSeqs` where applicable (e.g., `assistant/message` records its `assistant/chunk` sources; `tool/result` records its `tool/call` source).
|
||||
|
||||
2. **Replace** — remove nodes from `start` through `end` (both inclusive) and insert a new node in their place. Both `start` and `end` must be valid surface node seqs in the current surface; `start === end` replaces a single node. The node's `sourceEventSeqs` must contain every shadowed surface node. The shadowed events remain in the log but are no longer on the surface.
|
||||
|
||||
The both-ends-inclusive design was chosen over half-open `[start, endExclusive)` because the surface is a doubly-linked list — both ends are naturally named by node seqs, and single-node replacement (`start === end`) is a common case that reads naturally with inclusive semantics.
|
||||
|
||||
### SurfaceManager: delta-based, not full rebuild
|
||||
|
||||
A `SurfaceManager` class (private to `Session`) maintains the cached linked list. It tracks `_lastProcessedSeq` and processes only the **delta** (new events since the last access) rather than rescanning the entire log. Because the log is append-only, prior events never change — full rebuild is only needed after a wholesale log replacement (e.g., seeding).
|
||||
|
||||
Why delta processing? The naive approach (a dirty flag + full rebuild on every access) would be O(N²) over a session's lifetime — every single-event append triggers a complete scan of all prior events. Delta processing is O(1) when no new events and O(new events) when new events arrive.
|
||||
|
||||
`deriveMessages()` uses the surface when surface markers exist, falling back to the existing linear scan for sessions without markers (backward compatibility).
|
||||
|
||||
### Persistence
|
||||
|
||||
The new fields are serialized as top-level JSON properties. The JSONL backend requires zero changes — `JSON.stringify`/`JSON.parse` preserve everything transparently. The SQLite backend's `events` table carries two nullable TEXT columns (`source_event_seqs`, `surface_op`). The on-disk `SCHEMA_VERSION` is bumped to reflect the column set, and — per the pre-release bump-and-reject policy — a database written by any other build is REJECTED on open rather than migrated (there is no persisted user data to upgrade). The session format `version` is pinned at `SESSION_FORMAT_VERSION = 0` (the "unstable / pre-release" stance): the optional surface fields are absorbed without bumping it.
|
||||
|
||||
### Crash recovery
|
||||
|
||||
The `repair.ts` module synthesizes `tool/result` closers for orphaned tool calls after a crash. These closers carry `surfaceOp: 'append'` and `sourceEventSeqs` pointing to the orphaned `tool/call` event, so the rehydrated surface is valid.
|
||||
|
||||
### Invariants
|
||||
|
||||
The dev-mode invariants plugin validates: `sourceEventSeqs` references (non-empty, no duplicates, references earlier events, references known seqs) and `surfaceOp` (replace start ≤ end).
|
||||
|
||||
## Consequences
|
||||
|
||||
- **`packages/core/session`**: New `surface.ts` (`SurfaceManager`), new types (`SurfaceOp`, `SurfaceAppendOpts`), new fields on `SessionEvent`, modified `append()` (third optional `SurfaceAppendOpts` param), refactored `deriveMessages()` (surface path + legacy fallback), surface-aware `repair.ts`.
|
||||
- **`packages/core/agent-loop`**: All surface-capable appends pass surface opts. Chunk seqs are collected for `assistant/message` provenance; `tool/call` seqs are captured for `tool/result` provenance.
|
||||
- **`packages/session-persistence/session-persistence-sqlite`**: Two new nullable TEXT columns (`source_event_seqs`, `surface_op`) on the `events` table; `SCHEMA_VERSION` bumped (bump-and-reject, no migration).
|
||||
- **`packages/support/invariants`**: Surface-related validation rules.
|
||||
- **`packages/session-persistence/session-persistence-jsonl`**: No changes required.
|
||||
- **`packages/session-persistence/session-persistence`**: Abstract interface unchanged.
|
||||
|
||||
The surface is the foundation for future compaction: a compaction plugin appends a new event (e.g., `compaction/marker`, added to `SessionEventMap` via declaration merging) with `surfaceOp: { op: 'replace', start, end }` and `sourceEventSeqs` covering the shadowed nodes. Replay preserves the compaction decision deterministically.
|
||||
@@ -0,0 +1,37 @@
|
||||
# RFC: Shared persistence write coordinator
|
||||
|
||||
Status: implemented (proposed and accepted 2026-06-18, implemented 2026-06-20)
|
||||
|
||||
## Problem
|
||||
|
||||
`dsh-session-persistence-jsonl` and `dsh-session-persistence-sqlite` intentionally prove the same `SessionPersistence` contract over different storage media, but their write-path orchestration was duplicated: per-session state, `session/created` adoption, backend-specific prefix reads, write-behind buffers, serialized flush chains, HMR seeding, and dispose drains. The pure seed-prefix collision and serializability guards had already moved into the seam package; the remaining orchestration was still correctness-heavy and received the same fixes twice. A code-level diff showed the two backends were byte-identical — or same-algorithm — for ALL of it: the four maps (`states`/`buffers`/`chains`/`inits`), `installWritePath`, `initFor`, `onCreated`'s four cases, `flush`, `drain`, `serialize`, `adopt`, `adoptLivePrefix`, `assertVersion`, and the `create`/`append`/`load` skeletons. Only the storage primitives (write bytes vs. INSERT rows) differed.
|
||||
|
||||
## Decision
|
||||
|
||||
Extract a backend-agnostic `PersistenceCoordinator` into `dsh-session-persistence`. The coordinator owns the orchestration once; each first-party backend composes one (`new PersistenceCoordinator(ctx, this)`), implements a small `PersistenceBackend` hook interface, and delegates its four public service methods (`create`/`append`/`load`/`list`) to it.
|
||||
|
||||
Composition, not inheritance. The coordinator is a concrete class the backend holds, not a base class the backend extends. The RFC's risk — "a coordinator must not make unusual backends fight an inheritance hierarchy" — is avoided: a backend exposes only the hooks; it cannot reach the coordinator's private orchestration state, and the public `SessionPersistence` service shape is unchanged, so a third-party backend MAY still implement the abstract service directly without the coordinator at all.
|
||||
|
||||
### The hook interface (`PersistenceBackend<TornMarker>`)
|
||||
|
||||
Six methods (five required + an optional lifecycle hook) — the only seam between the coordinator and storage:
|
||||
|
||||
- `name` — backend label for the dispose-failure `AggregateError`.
|
||||
- `loadStored(id)` — read a stored prefix by id, scanning ANY storage scope (every JSONL cwd bucket; SQLite's id is globally unique). Used by resume/load and, via `!== undefined`, the create-collision probe.
|
||||
- `loadLive(id, cwd)` — read a stored prefix SCOPED to `cwd`. **Deliberately distinct from `loadStored`**: HMR live-adoption must only adopt a persisted log at the SAME cwd as the live session; a same-id log at a different cwd is a collision, not a resume. Collapsing the two reintroduces a cross-cwd adoption bug. SQLite ignores `cwd`.
|
||||
- `appendBatch(meta, events, isMaterialized)` — durably append a contiguous batch, lazily materializing the session ATOMICALLY when not yet materialized (the materialize-write and the first event batch must commit together — a crash between them must not leave a materialized-but-empty session; this is why there is no separate `materialize` hook).
|
||||
- `commitRepair(meta, tornMarker, closers)` — make a crash repair durable: truncate the torn tail (iff `tornMarker !== undefined`) and append `closers`. **NOT required to be atomic** — JSONL legitimately truncates-then-appends in two fsync'd steps, SQLite does DELETE+INSERT in one transaction. Used by `load` (truncate + synthetic closers) and live-adoption (truncate only, `closers = []`).
|
||||
- `list()` — list all stored metadata.
|
||||
- `close?()` — optional lifecycle teardown (SQLite closes its db handle; JSONL omits it), awaited in the dispose effect AFTER the quiescence drain so a close failure never masks a drain error.
|
||||
|
||||
### The opaque torn marker
|
||||
|
||||
The single design choice that keeps the seam clean: the crash-repair "where is the torn tail" token is OPAQUE to the coordinator. The coordinator computes the synthetic closers (it owns `interruptedTurnClosers` from `dsh-session`), but it only ever tests `tornMarker !== undefined` and passes the value straight back to `commitRepair` — it never inspects it. Each backend picks its own marker type: JSONL uses the byte offset to truncate to, SQLite the seq to delete from (both happen to be `number`). The JSONL backend folds its `committedBytes < buffer.byteLength` comparison INSIDE the hook so the returned marker is already `number | undefined`; without that fold the coordinator would have to know about byte lengths.
|
||||
|
||||
## Testing
|
||||
|
||||
The shared `runPersistenceContract` (public-API contract) keeps running for every backend. A new `runCoordinatorContract` (`tests/coordinator-contract.ts`) holds the write-path orchestration — adoption, HMR, collision, dispose-drain, crash-tail repair — and runs once per backend through a `CoordinatorFixture` (an in-memory reference + jsonl + sqlite). The per-backend specs shrank to storage mechanics only (JSONL: path safety, fsync rollback, bucket listing; SQLite: schema version, `scanRows`, transaction rollback). A through-coordinator torn-tail→load→`commitRepair` test per real backend (via a `corruptTail` fixture hook) keeps the coordinator's torn-marker repair branch covered under the 100% per-file gate — the contract crash test only produces synthetic closers, never a torn marker, so it could not reach that branch.
|
||||
|
||||
## Risks and what we gave up
|
||||
|
||||
The pre-extraction duplication was verbose but explicit — each backend read top-to-bottom. The coordinator adds one indirection (the hook seam) and one new concept (the opaque torn marker). This clears the bar because the centralized logic is the correctness-heavy part that was already being fixed twice, and the hook set is narrow (six methods, no inheritance). The hook surface was deliberately held to the minimum: the create-collision probe is NOT a separate hook — it folds into `loadStored(id) !== undefined`; there is no separate `materialize` hook (folded into `appendBatch` for atomicity); `list()` stays a backend method with no coordinator pass-through (listing needs none of the orchestration). The net effect is a reduction: one orchestration copy instead of two, the backends shrank by ~1200 lines of duplicated churn, and a future backend implements a handful of small primitives instead of copying the entire `session/event` → buffer → flush machinery.
|
||||
68
docs/rfc/implemented/architecture/2026-06-20-branded-ids.md
Normal file
68
docs/rfc/implemented/architecture/2026-06-20-branded-ids.md
Normal file
@@ -0,0 +1,68 @@
|
||||
# RFC: Branded IDs everywhere they belong
|
||||
|
||||
Status: implemented (proposed and accepted 2026-06-20)
|
||||
|
||||
## Problem
|
||||
|
||||
The harness already brands three identifiers — `CallId` (`packages/llm/llm/src/brand.ts`), `SessionId` (`packages/core/session/src/types.ts`), and `AgentId` (`packages/core/agent/src/types.ts`) — using the `Branded<B> = string & { readonly [BRAND]: B }` machinery (owned by the type-only `@deepseek-ai/dsh-brand` package at `packages/util/brand/` — see its [README](../../../../packages/util/brand/README.md)) and a zero-cost cast factory per type. `dsh-brand` also states the governing policy: *"Branding is for ids that cross package boundaries and could plausibly be confused; not every string needs a brand."* That policy is right; the problem is that it is only half-applied. Two gaps let a structurally-identical-but-semantically-wrong string slip through the type checker today.
|
||||
|
||||
**Gap 1 — unbranded cross-boundary IDs in the bash seam.** The background-task id is a plain `string`: `BashTask.id: string` (`packages/bash/bash/src/types.ts`), carried as `string` through the whole executor seam (`BashExecutor.get`/`ownerOf`/`readOutput`/`kill(id: string)` in `packages/bash/bash/src/index.ts`) and validated/passed as `string` by the model-facing tools (`validateTaskId`, `assertTaskAccess`, the `task_id` schema arg in `packages/bash/tool-bash/src/index.ts`). It is generated by a per-executor counter — `` `bash-${this.nextTaskId++}` `` in `packages/bash/bash-local/src/index.ts` — which gives it **exactly the same `name-N` shape as `SessionId`'s default** (`` `session-${++counter}` `` in `packages/core/session/src/index.ts`). A bash task id and a session id are trivially swappable at a call site and the compiler says nothing. This is the headline case the user asked about, and it is a model-facing id (the model passes `task_id` back to `bash_output`/`bash_kill`), so a confusion here is reachable from untrusted input.
|
||||
|
||||
The bash **owner token** is the related sub-case: `BashExecRequest.owner?: string` and `BashExecSpec.owner: string | undefined` (`packages/bash/bash/src/types.ts`) are documented as a deliberately *opaque* isolation key, but in every live caller the value IS the owning agent's `session.header.id` (`callerToken = (exec) => exec.agent?.session.header.id` in `packages/bash/tool-bash/src/index.ts`) — i.e. a `SessionId` wearing a `string` disguise. It is compared for access control (`owner !== callerToken(exec)`), so a mismatched-but-well-typed string here is a cross-session isolation bug the type system currently cannot catch. This is the same `session.header.id`-as-owner alias that the [unify-the-agent-id-and-the-session-id](../../proposed/simplification/2026-06-20-unify-agent-and-session-id.md) proposal calls the "bash owner-token alias hole".
|
||||
|
||||
**Gap 2 — brand erosion at the seams of the *already-branded* IDs.** Even `CallId`/`SessionId`/`AgentId` decay back to bare `string` at exactly the places confusion is most likely: the registry/store `Map` key types and most public method params. Representative sites: `SessionStore.store = new Map<string, Session>()` and `create`/`prepare(id?: string)`/`get(id: string)` (`packages/core/session/src/index.ts`); `AgentRegistry.store = new Map<string, Agent>()` and `register`/`get(id: string)` (`packages/core/agent/src/index.ts`); `ToolPresenter.pending = new Map<string, …>()` keyed by call id and `call(callId: string)`/`result(callId: string)` (`packages/ui/acp/src/index.ts`); the ACP session-id surface beyond the store map — `SessionRecord.sessionId: string`, `bySession = new WeakMap<Agent, string>()`, `loadingIds = new Set<string>()`, `requireSession(sessionId: string)`, and the exported `streamSessionEventUpdate(sessionId: string, …)` (`packages/ui/acp/src/index.ts`); and the persistence coordinator's `Map<string, …>` keyed by session id (`packages/session-persistence/session-persistence/src/coordinator.ts`). A brand that is dropped at the `Map` key buys nothing on lookups — the value of the existing brands is partly unrealized.
|
||||
|
||||
## Proposal
|
||||
|
||||
A type-only change. Brands are zero-cost casts; nothing about runtime behavior, serialization, comparison, or the wire format changes. The work is in three parts, all honoring the existing "not every string" policy.
|
||||
|
||||
- **Brand the bash task id.** Add `BashTaskId = Branded<'BashTaskId'>` plus its same-named factory in `packages/bash/bash/src/types.ts` (the package that *owns* the id), importing `Branded` from `@deepseek-ai/dsh-brand` exactly as `SessionId`/`AgentId` already do. The brand primitive lives in the dependency-free `dsh-brand` utility package precisely so `dsh-bash` can brand its ids by depending on it alone — it never pulls in `dsh-llm` (or `dsh-session`) just to reach `Branded`. Thread it through `BashTask.id`, the `BashExecutor` seam methods (`get`/`ownerOf`/`readOutput`/`kill`), the generation site in `dsh-bash-local` (brand the counter output once, at creation), and the `dsh-tool-bash` validate/access surface (`validateTaskId` returns a `BashTaskId`; `task_id` is branded at the tool boundary where the model's string arrives).
|
||||
|
||||
- **Mint a distinct `OwnerToken` brand.** Add `OwnerToken = Branded<'OwnerToken'>` in `packages/bash/bash/src/types.ts`; type `BashExecRequest.owner` / `BashExecSpec.owner` / `BashExecutor.ownerOf` as `OwnerToken | undefined`. The `dsh-tool-bash` consumer casts the agent's `session.header.id` (a `SessionId`) into an `OwnerToken` at the boundary — the one place the two vocabularies meet. The bash seam never imports `dsh-session`. (Rationale in the next section.)
|
||||
|
||||
- **Stop the brand erosion.** Propagate the existing brands to the `Map` key types and public method params listed under Gap 2 — `Map<SessionId, Session>`, `get(id: SessionId)`, `Map<AgentId, Agent>`, `Map<CallId, …>`, the ACP `SessionRecord.sessionId: SessionId` surface, the coordinator's `Map<SessionId, …>`. This is the larger mechanical share of the diff and the part that makes the *existing* brands actually load-bearing on lookups, not just on the struct fields.
|
||||
|
||||
Illustrative shape (the factory pattern is identical to the three existing brands):
|
||||
|
||||
```ts ignore-check
|
||||
import type { Branded } from '@deepseek-ai/dsh-brand'
|
||||
|
||||
/** A background bash task handle (generated `bash-N` by the local executor). */
|
||||
export type BashTaskId = Branded<'BashTaskId'>
|
||||
export function BashTaskId(id: string): BashTaskId {
|
||||
return id as BashTaskId
|
||||
}
|
||||
|
||||
/** A bash task's opaque isolation key — the consumer's owner identity, NOT the bash seam's. */
|
||||
export type OwnerToken = Branded<'OwnerToken'>
|
||||
export function OwnerToken(id: string): OwnerToken {
|
||||
return id as OwnerToken
|
||||
}
|
||||
```
|
||||
|
||||
## Why a distinct OwnerToken brand (not SessionId)
|
||||
|
||||
The obvious shortcut is to type `owner` as `SessionId` directly — it always *is* one. We reject that. The bash executor seam is a capability seam (interface `dsh-bash`, implementation `dsh-bash-local`, consumer `dsh-tool-bash`) and its owner token is *documented as deliberately opaque*: the executor "never interprets it (no access policy lives in the seam — that is the consumer's job)" (`packages/bash/bash/src/types.ts`). Typing the seam's field as `SessionId` would import `dsh-session`'s vocabulary into a package that must not know what an owner token *means* — it would couple a generic execution backend to the session model and contradict the opaque-token design. A sandboxed or remote executor that replaces `dsh-bash-local` should not inherit a session dependency. The distinct `OwnerToken` brand keeps the seam decoupled: `dsh-bash` knows only "an owner is some opaque branded token," and the `dsh-tool-bash` consumer — which already decides the access policy — is the single boundary that casts its `SessionId` into an `OwnerToken`. The brand still delivers the safety win (you cannot pass a `BashTaskId` or a raw string where an owner is expected) without the coupling.
|
||||
|
||||
## Out of scope / possible extensions
|
||||
|
||||
Kept deliberately narrow per the "not every string needs a brand" policy. Each of these is a plausible future brand, deferred with a reason, not a commitment:
|
||||
|
||||
- **`ModelId`** (`GenerateOptions.model`, the `LlmService` adapter-registry key) — a real cross-package lookup key (config → agent → llm → adapter); a reasonable next brand, left out only to keep this RFC's blast radius focused.
|
||||
- **`ToolName`** (the `ToolRegistry` key) — author-defined, human-readable, and rarely confused with another id; the weakest candidate, likely not worth a brand.
|
||||
- **`ErrorCode`** (`HarnessError.code`) — a closed vocabulary (`ABORTED`, `NO_ADAPTER`, …), not a per-instance id; better served by a string-literal union than a brand, if anything.
|
||||
- **Numeric ordinals** — turn number, step number, and the event `seq` are `number`, not `string`, so `Branded<string>` does not apply; a parallel `number & { readonly [BRAND]: B }` variant could brand them, but they are positional ordinals rarely passed across boundaries, so the payoff is low.
|
||||
- **Validated construction** — the brand factories are pure casts with no runtime check, and every boundary (ACP `sessionId`, provider-issued `call.id`, the empty-string fallback in `dsh-llm-deepseek`) trusts the raw string today. A `SessionId.parse()` / `isValid()` companion that throws on malformed input at boundaries is a genuine gap, but it is a *runtime-behavior* change with its own design (what is "malformed"? what do we do on failure?) and belongs in its own RFC, not bundled into this type-only pass.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `BashTaskId` and `OwnerToken` are defined in `dsh-bash` and threaded end-to-end: the executor seam, the `dsh-bash-local` generation site, and the `dsh-tool-bash` model-facing surface all speak the brands; `dsh-bash` gains no dependency on `dsh-session`.
|
||||
- No collection keyed by an in-scope branded id (`CallId`/`SessionId`/`AgentId`/`BashTaskId`) is keyed by bare `string` — this covers `Map`, `WeakMap` value slots, and `Set` membership (e.g. the ACP `bySession`/`loadingIds`), not just `Map<string, …>`; the corresponding public method params and exported function signatures (e.g. `streamSessionEventUpdate`) take the brand, not `string`.
|
||||
- Brands are constructed via the cast factory at each boundary where a raw string enters (provider call id, ACP session id, model-supplied `task_id`); no `as` casts scattered at call sites.
|
||||
- `pnpm run typecheck` and `pnpm run doc-sync` are green; the change is observably type-only (no snapshot, no e2e behavioral diff).
|
||||
|
||||
## Risks / what we give up
|
||||
|
||||
- **Mechanical churn across two surfaces.** Propagating brands touches the bash seam (interface + impl + consumer) and the ACP session-id surface plus the persistence coordinator. The risk is broad but low-severity: a missed site is a compile error, not a silent bug. It ships as its own PR, converged with Codex, and stacks naturally near the [unify-the-agent-id-and-the-session-id](../../proposed/simplification/2026-06-20-unify-agent-and-session-id.md) work (both touch the session-id / owner-token boundary; if that proposal lands first, `OwnerToken` still stays distinct from the unified id for the decoupling reason above).
|
||||
- **Brands do not validate.** A brand is a confusability guard, not a correctness proof: a *wrong* session id that is still a well-formed string passes the type checker exactly as before. This RFC does not close that gap (see Out of scope) — it only stops the *category* error of passing the wrong *kind* of id.
|
||||
- **The "where to stop" line stays a judgment call.** Branding `BashTaskId` but not `ToolName`, `OwnerToken` but not `ModelId`, is a taste call about which strings "could plausibly be confused." Reasonable reviewers may want more or fewer; the policy in `brand.ts` is the tie-breaker, and this RFC errs toward the ids that are model-facing or used for access control.
|
||||
@@ -0,0 +1,53 @@
|
||||
# RFC: Extract example apps into packages
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
An example folder is supposed to be *thin* — the variable wiring of a demo, not the demo's machinery. Before this change it was thick. Each example carried a hand-rolled `start.ts` boot bootstrap, an infra preamble (`timer`, and — for the stdio demos — `logger` + `hmr`), nested includes of three shared YAML fragments (`base.yml` / `base-core.yml` / `acp-agent/acp-tail.yml`), and per-example `agent-loop`/persistence/system-prompt config. The actual app — the spine of services every agent needs — was spread across the leaf and those includes.
|
||||
|
||||
The deeper problem was a **coupled front-door cluster** that lived at the leaf with nothing enforcing it. Choosing the ACP bridge over `ui-stdio` was not one swappable line: an ACP server must **drop the stdout console logger** (stdout is the JSON-RPC channel — a stray log corrupts the frames) and pre-create **no** agents (ACP `session/new` creates them on demand), whereas the stdio app needs a console logger and a pre-created `main`. (`timer` is the one infra plugin common to both — it writes nothing to stdout — so it belongs in the shared spine, not the cluster.) That coupling was enforced only by prose warnings in the leaf YAML. A leaf that wired a console logger into the ACP config was a one-line, comment-only mistake away — exactly the [stdout-purity footgun](../feature/2026-06-18-acp-terminal-and-tool-rendering.md) the examples guarded by hand. The three `start.ts` files also duplicated the Loader-boot tail, the `.env` loader, and (for ACP) snapshot-mode branching and the stdin-dispose lifecycle.
|
||||
|
||||
## What shipped
|
||||
|
||||
Each example is now **mostly an invocation of an app package**, splitting the wiring along the existing [interface / implementation / consumer seam](2026-06-13-capability-seams.md): the **app package owns the composition**, the leaf `cordis.yml` owns only the **swappable choices** (which LLM adapter, which bash executor, model, prompt, persistence root).
|
||||
|
||||
- **`@deepseek-ai/dsh-agent-core`** ([packages/core/agent-core](../../../../packages/core/agent-core)) — a Cordis bundle plugin for the providerless, executor-less, UI-less spine: `timer` + `llm` + sessions + system-prompt + tools + agents + invariants + `tool-bash` + `agent-loop`, mounted as child plugins inside its `apply(ctx)` via `ctx.plugin(...)`. This is the old `base-core.yml` **minus** `bash-local`, **plus** `timer` and the loop, as code instead of a YAML include. The bundle **forwards** `agent-loop`'s `agents` list as its own config (`export const Config = AgentLoop.Config`, default `[]`, the existing `AgentLoop.Config` shape in [packages/core/agent-loop/src/index.ts](../../../../packages/core/agent-loop/src/index.ts)) — so each app supplies its own pre-created agents. This is precisely the reason the old `base-core.yml` gave for keeping `agent-loop` *out* of the shared core ("the examples disagree — stdio needs a pre-created `main`, acp needs none"); forwarding the config dissolves that objection — the loop is shared, the agents list is per-app. The bundle children register into the root service store, so a leaf-mounted sibling (the adapter, the executor) sees them exactly as a nested `plugin-include` subtree's services were seen before. Depending on the CONCRETE `dsh-agent-loop` (not just the `dsh-agent` interface) is deliberate and is the sanctioned exception to the "extension plugins depend on interfaces, never on the concrete loop" rule (packages/README.md, docs/architecture.md § Layering): the rule constrains plugins that EXTEND the system, whereas this bundle's whole job is to COMPOSE the concrete spine. Swapping the loop means publishing a different bundle, not rewiring every extension.
|
||||
- **`@deepseek-ai/dsh-stdio-agent`** ([packages/ui/stdio-agent](../../../../packages/ui/stdio-agent)) and **`@deepseek-ai/dsh-acp-agent`** ([packages/ui/acp-agent](../../../../packages/ui/acp-agent)) — app packages, each consuming `dsh-agent-core` and **baking in its coupled front-door cluster**: stdio = `ui-stdio` + console logger + a pre-created `main`; acp = the `acp` bridge + JSONL persistence + **no stdout logger** + no pre-created agents. The leaf no longer carries the cluster, so it has no logger entry to copy wrong by default — the common stdout-purity mistake loses its foothold. (A leaf can still *add* a sibling logger entry — a package cannot forbid what a leaf author writes — so the rule "never add a stdout logger to an ACP leaf" stays documented at the leaf; what changed is that the default leaf has nothing to get wrong.) They land under the existing `ui` group alongside `acp`, so no new package group (and no `tsconfig`/`packages/README` group plumbing) was needed.
|
||||
- **`start.ts` is gone.** Each app package exposes a `bin` (`dsh-stdio-agent` / `dsh-acp-agent`); the `demo:*` scripts invoke it (e.g. `dsh-stdio-agent ./cordis.yml`). The Loader-boot tail, `.env` loading, snapshot-mode selection, and stdin-dispose lifecycle moved into that bin, owned by the app. The `bin.ts` files are coverage-excluded (a self-executing CLI entry, like the old `start.ts`) and driven by the keyless Loader-path tests.
|
||||
- **Each leaf `cordis.yml` collapses** to backends + config: the LLM adapter (`llm-deepseek` with apiKey/models, or `llm-replay`), the bash executor (`bash-local`), `hmr` for the stdio demos (see the amendment below), and one app entry carrying the app's config (model, system prompt, persistence root — surfaced as the app package's own `Config`, which routes each value to wherever the app wires it: stdio onto its pre-created agent, acp onto the bridge plugin).
|
||||
- **echo-agent folds onto `dsh-stdio-agent`**, swapping the LLM backend to the local `mock-llm` and adding the local `echo-tool` (plus `bash-local`, which the spine's `tool-bash` injects) at the leaf — the clean demonstration of "swap the backend, keep the app". `mock-llm.ts` / `echo-tool.ts` stay as example-local teaching plugins.
|
||||
- **`base.yml`, `base-core.yml`, and `acp-agent/acp-tail.yml` are retired** — the spine they shared now lives in `dsh-agent-core`.
|
||||
|
||||
`bash-local` and the LLM adapter stay **leaf choices**: the bundle ships `tool-bash` (the consumer schema), the leaf picks the executor implementation, so a sandboxed executor or replay adapter swaps in without touching the app.
|
||||
|
||||
### Amendment on implementation: `hmr` stays a leaf entry
|
||||
|
||||
The proposal listed `hmr` among the stdio app's baked-in front-door cluster. Validating against the code, baking `hmr` into the `dsh-stdio-agent` package fights cordis in two ways, so it ships as a **leaf `cordis.yml` entry** instead:
|
||||
|
||||
1. `@cordisjs/plugin-hmr` is a Loader-only, subprocess-only dev plugin — its constructor throws without `node --expose-internals` + a live `loader` service, so it can only run in the real `demo:*`/bin subprocess, never in the in-process unit/coverage tier.
|
||||
2. The in-process test tier (vitest) cannot even *import* the vendored `hmr` module (its class-decorator `@Inject` form fails under Vite's transform), so a package whose `apply` statically imported it could never satisfy the per-file 100% coverage gate on its headline function.
|
||||
|
||||
Crucially, `hmr` is **not** a stdout-purity footgun the way the console logger is — a stray `hmr` in the ACP config would not corrupt the JSON-RPC frames — so leaving it at the leaf costs none of the safety the coupling argument is about. The **logger** (the real coupling) stays baked in: the stdio app includes it, the ACP app omits it.
|
||||
|
||||
## Why not keep the wiring in shared YAML includes?
|
||||
|
||||
The old `base*.yml`/`acp-tail.yml` includes already deduped the *config*, but a YAML include cannot **encapsulate** the front-door coupling — it can only describe it in a comment and trust every leaf to obey. It also cannot own a `bin`, so the boot glue stayed copied across three `start.ts` files. A package turns "the ACP app never logs to stdout" from a prose warning into a property of the artifact: there is no logger entry in the leaf to get wrong.
|
||||
|
||||
## Verification
|
||||
|
||||
- Each example directory is `cordis.yml` (+ the acp `cordis.snapshot.yml`) + `README.md` + tests only — no `start.ts`, no infra preamble; `base.yml`/`base-core.yml`/`acp-tail.yml` are gone.
|
||||
- `demo:echo` / `demo:coding` / `demo:acp` run via the app-package `bin`s.
|
||||
- The new packages carry the per-file 100% coverage gate and a README like every `@deepseek-ai/dsh-*`. Each app package has a keyless **real-load-path** smoke that boots it through its `bin` + the cordis Loader (not a hand-built `ctx.plugin({...})` mount), guarding the `unwrapExports` export-shape bug class ([postmortem 0001](../../../postmortem/0001-acp-default-export-drops-inject.md)).
|
||||
- The ACP snapshot **replay** transcript is unchanged: the boot restructuring preserved the plugin set + load order, so `pnpm run test:snapshot` stays green against the committed goldens with no re-record.
|
||||
|
||||
## What we give up
|
||||
|
||||
- **The bare-plugin-tree pedagogy.** echo-agent's inlined `cordis.yml` showed every plugin at once; the spine now lives behind a bundle, so seeing the whole tree means opening `dsh-agent-core`. The app package's README carries that teaching weight.
|
||||
- **A layer of indirection.** "What does this demo load?" becomes a package read, not a single YAML scan.
|
||||
|
||||
## Related
|
||||
|
||||
- Supersedes [Make the shared example base providerless](../../rejected/architecture/2026-06-20-providerless-example-base.md): renaming `base.yml` to the providerless core is moot once the spine moves into `dsh-agent-core` and the `base*.yml` files are deleted.
|
||||
- Builds on the [capability-seams](2026-06-13-capability-seams.md) interface/implementation/consumer split — backends and presentation stay leaf choices; the spine is the shared bundle.
|
||||
- Complements [Reorganize packages into a modular hierarchy](2026-06-20-package-hierarchy.md): the new app/core packages slot into existing groups under that hierarchy (`core` for the reusable spine bundle, `ui` for the app-specific front doors).
|
||||
@@ -0,0 +1,67 @@
|
||||
# RFC: Reorganize packages into a modular hierarchy
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
`packages/` was flat: 18 packages all sat at `packages/<name>/`, so a package's location said nothing about whether it was core product API, a swappable capability seam, a provider adapter, a product integration, or example/test support. The package README carried a `FIXME(package-hierarchy)` and `scripts/publint-all.ts` a `TODO(package-inventory)` flagging exactly this. Core packages, provider integrations, capability seams, example UI support, and snapshot-only replay support all looked equally foundational.
|
||||
|
||||
This was not just cosmetic. Because every top-level package looked like part of the same public surface, future removal was harder, and publish/lint/doc scripts had to encode intent through comments or hand-maintained static lists rather than reading it off the layout.
|
||||
|
||||
## What landed
|
||||
|
||||
Packages are grouped by modular role at a uniform `packages/<group>/<pkg>/` depth. Group directories are pure containers (no `package.json`); every package keeps its `@deepseek-ai/dsh-<pkg>` name — this is repo structure and maintenance policy, not package renaming.
|
||||
|
||||
```text
|
||||
packages/
|
||||
core/ (product API spine)
|
||||
session/
|
||||
system-prompt/
|
||||
tools/
|
||||
agent/
|
||||
agent-loop/
|
||||
llm/ (product — capability family)
|
||||
llm/
|
||||
llm-deepseek/
|
||||
llm-pi-ai/
|
||||
bash/ (product — capability family)
|
||||
bash/
|
||||
bash-local/
|
||||
tool-bash/
|
||||
session-persistence/ (product — capability family)
|
||||
session-persistence/
|
||||
session-persistence-jsonl/
|
||||
session-persistence-sqlite/
|
||||
ui/ (product integration)
|
||||
acp/
|
||||
support/ (dev/test/example infrastructure)
|
||||
invariants/
|
||||
ui-stdio/
|
||||
llm-replay/
|
||||
```
|
||||
|
||||
### Placement decisions
|
||||
|
||||
- **Same-name nesting for capability families.** A family's interface package sits at `packages/<group>/<group>/` (`llm/llm`, `bash/bash`, `session-persistence/session-persistence`), with implementations and consumers as flat siblings. There is no extra `adapters/`/`impls/` sub-tier — every package is exactly depth 2, which keeps the workspace glob a clean `packages/*/*` and lets one `@deepseek-ai/dsh-*` tsconfig wildcard resolve every package (unique dir names make first-on-disk-wins unambiguous).
|
||||
- **`session` stays in `core/`; persistence is its own family.** The session log is core product API. Its storage backends form a parallel capability family (`session-persistence/`) mirroring `llm/` and `bash/`, rather than nesting under `core/session/`.
|
||||
- **`agent-loop` is in `core/`.** It is the one concrete implementation of the `agent` seam, but it ships as the harness's default product loop, so it lives with the core spine. Plugins still depend on the `agent` vocabulary, never on `agent-loop`, so the loop stays swappable.
|
||||
- **`invariants` and `ui-stdio` are `support/`, not product.** `invariants` is dev-mode contract checking. `ui-stdio` was extracted from the examples for reuse and the coverage gate — it is example-coupled, so it sits in `support/` alongside `llm-replay` (the snapshot-test replay adapter). `acp` is the only `ui/` member because it is a real product surface (the ACP bridge an editor drives), structurally distinct from the readline demo helper.
|
||||
|
||||
### Deduplicating the package lists
|
||||
|
||||
The package list had been enumerated in five places. The uniform depth-2 layout lets most of them be derived instead:
|
||||
|
||||
- `tsconfig.base.json` and `tsconfig.typecheck.json` each map every package through a single `@deepseek-ai/dsh-*` `paths` wildcard listing one candidate per group, in place of 18 per-package entries. (One subtlety this introduced: a path candidate contains `/*/`, which a naive regex comment-stripper mistakes for a block comment — `scripts/doc-typecheck.ts` reads the `paths` map via the TypeScript JSONC API rather than stripping comments by hand for exactly this reason.)
|
||||
- `scripts/publint-all.ts` derives its list by reading the hierarchy (`packages/<group>/<pkg>`), resolving the `TODO(package-inventory)`.
|
||||
- `tsconfig.build.json`'s project `references` stay an explicit list — TypeScript project references have no wildcard form. Generating these from a manifest is left to a follow-up (see [discover package inventories](../../proposed/process/2026-06-20-discover-package-inventory.md)).
|
||||
|
||||
### Guardrails added
|
||||
|
||||
Two doc-sync/hygiene gates keep the structure and its references honest, so the manual checks this restructure required do not have to be repeated by hand:
|
||||
|
||||
- `scripts/verify-package-paths.ts` flags a `packages/<path>` reference (in Markdown or a `.ts` comment/string) that does not resolve **and** names a real package in a segment — i.e. a stale path to a moved package. A path naming a package that exists nowhere (a forward-looking proposal) is left alone, so the gate applies uniformly across proposed/implemented/rejected.
|
||||
- `scripts/check-workspace-constraints.ts` asserts the `packages/<group>/<pkg>` shape: group dirs carry no `package.json`, and no package sits flat at the root or nests deeper. Group names stay open — a new group may be added without editing the gate; only the depth-2 shape is fixed.
|
||||
|
||||
## What we gave up
|
||||
|
||||
The restructure churned imports, workspace globs, doc links, build references, and package paths in one coordinated move. That churn is acceptable pre-release (per the AGENTS.md foundation-over-blast-radius stance) because it stops the flat layout from fossilizing support packages as product contracts, and it is a one-time cost: the wildcard `paths`, the glob-derived publint list, and the shape gate mean a new package needs no further structural edits.
|
||||
Reference in New Issue
Block a user