Merge remote-tracking branch 'origin/master' into session-fork
# Conflicts: # docs/architecture.md # packages/README.md # tsconfig.build.json # tsconfig.json
This commit is contained in:
Hypatia May
@@ -24,6 +24,7 @@ For a catalog of the **data structures** this architecture moves around — the
|
||||
│ @deepseek-ai/dsh-agent-loop (the ONE concrete plugin) │
|
||||
│ @deepseek-ai/dsh-bash-local (bash impl) │
|
||||
│ @deepseek-ai/dsh-tool-bash (bash tool schemas) │
|
||||
│ @deepseek-ai/dsh-subagent-* (subagent providers) │
|
||||
│ @deepseek-ai/dsh-session-persistence-jsonl (persistence impl)│
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ @deepseek-ai/dsh-agent (vocabulary + registry) │
|
||||
@@ -33,6 +34,8 @@ For a catalog of the **data structures** this architecture moves around — the
|
||||
│ @deepseek-ai/dsh-session-persistence (persistence seam) │
|
||||
│ @deepseek-ai/dsh-llm (abstract model service) │
|
||||
│ @deepseek-ai/dsh-bash (abstract bash executor) │
|
||||
│ @deepseek-ai/dsh-compact (abstract compaction seam) │
|
||||
│ @deepseek-ai/dsh-subagent (provider registry seam) │
|
||||
├─────────────────────────────────────────────────────────────┤
|
||||
│ vendor/: cordis, loader, include, group, timer, hmr, │
|
||||
│ logger-console, cosmokit, schemastery │
|
||||
@@ -55,6 +58,7 @@ Dependency rule: **extension** plugins depend on interface packages, never on `d
|
||||
| `ctx.bash` | `BashExecutor` (abstract) | dsh-bash | bash execution seam: foreground runs + background tasks |
|
||||
| `ctx.compact` | `CompactService` (abstract) | dsh-compact | compaction seam: decide when history is too large, summarize an older range into a single surface node |
|
||||
| `ctx.sessionFork` | `SessionForkService` | dsh-session-fork | live-session fork seam: validate turn-boundary forks, snapshot seed events, create forked child sessions |
|
||||
| `ctx.subagents` | `SubagentService` | dsh-subagent | named provider registry for delegating a task to child agents |
|
||||
|
||||
All registrations (`registerAdapter`, `section`, `tools`, `register`, …) go through `ctx.effect()` and return disposers, so plugin hot-reload (vendored HMR) and fiber disposal clean up automatically.
|
||||
|
||||
@@ -87,11 +91,11 @@ A `Session` is an append-only log of typed `SessionEvent`s — the single source
|
||||
- `user/message` → user message
|
||||
- `assistant/message` → assistant message (raw `assistant/chunk` events are replay/UI data and are skipped in derivation; an empty-content `assistant/message`, which exists only to host a max-tokens step's `usage`, is skipped too)
|
||||
- `tool/result` → user message carrying a `tool-result` block
|
||||
- `context/message`, `steering/message` → user-role messages wrapped in a tagged envelope (`<context source="…">…</context>`) at their chronological position — the "system-reminder" pattern; models distinguish them from real user prompts by the envelope. **TODO(review)**: the real adapters now exist (the original precondition); the envelope still wants a deliberate review against live model behavior (`TODO(review)` in dsh-session).
|
||||
- `context/message`, `steering/message` → user-role messages wrapped in a tagged envelope (`<context source="…">…</context>`) at their chronological position — the "system-reminder" pattern; models distinguish them from real user prompts by the envelope. Live-adapter review has validated the tagged-envelope rendering against current DeepSeek behavior; provider-specific mismatches belong in that adapter.
|
||||
|
||||
Replay/fork = `ctx.sessions.create(id, { seed: seedEvents })`; user-facing live-session fork policy lives in the optional `ctx.sessionFork` service, which rejects non-boundary forks instead of changing the core store. Trace/telemetry = listen to `session/event`.
|
||||
|
||||
**Durability seam**: `session/event` is a synchronous notification; persistence plugins buffer (write-behind) and drain at the awaited `session/flush` checkpoint the loop fires at every turn end. The durable backend is a real **capability seam**: the abstract `SessionPersistence` service (`dsh-session-persistence`, `ctx.sessionPersistence`) defines create/append/load/list over the existing `SessionEvent` (no parallel persisted type), and `dsh-session-persistence-jsonl` is the first implementation — an append-only JSONL log per session with crash-safe atomic writes, crash recovery that PRESERVES an interrupted turn (closing it with a synthetic `turn/end {interrupted}` rather than truncating — a turn can be huge), and a read/replay path. Session metadata (format version, cwd, lineage) travels separately as `SessionHeader`, attached to a `Session` via `session.header`. Resuming a persisted session into a live agent is `ctx.agents.resume({ resumeSessionId })`. A second backend, `dsh-session-persistence-sqlite` (`node:sqlite`, one row per `SessionEvent` — the row shape `(session_id, seq, type, time, data)` maps 1:1 onto it), passes the same `runPersistenceContract` suite, proving the seam is genuinely backend-agnostic.
|
||||
**Durability seam**: `session/event` is a synchronous notification; persistence plugins buffer (write-behind) and drain at the awaited `session/flush` checkpoint the loop fires at every turn end. The durable backend is a real **capability seam**: the abstract `SessionPersistence` service (`dsh-session-persistence`, `ctx.sessionPersistence`) defines create/append/load/list over the existing `SessionEvent` (no parallel persisted type), and `dsh-session-persistence-jsonl` is the first implementation — an append-only JSONL log per session with crash-safe atomic writes, crash recovery that PRESERVES an interrupted turn (closing it with a synthetic `turn/end {interrupted}` rather than truncating — a turn can be huge), and a read/replay path. Session metadata (format version, cwd, lineage, seed boundary) travels separately as `SessionHeader`, attached to a `Session` via `session.header`. Resuming a persisted session into a live agent is `ctx.agents.resume({ resumeSessionId })`. A second backend, `dsh-session-persistence-sqlite` (`node:sqlite`, one row per `SessionEvent` — the row shape `(session_id, seq, type, time, data, source_event_seqs, surface_op)` maps 1:1 onto it), passes the same `runPersistenceContract` suite, proving the seam is genuinely backend-agnostic.
|
||||
|
||||
## Prompt assembly (dsh-system-prompt)
|
||||
|
||||
@@ -134,10 +138,11 @@ forever:
|
||||
drain queued → 'turn/start' → session('user/message'…) → emit agent/turn-start
|
||||
STEP loop:
|
||||
drain steering (late steering from previous step's listeners)
|
||||
session('step/start'); emit agent/step-start
|
||||
assembly = ctx.systemPrompt.assemble() ⟵ waterfall system-prompt/assemble
|
||||
await ctx.serial('agent/pre-step') ⟵ surface mutation (compaction) OUTSIDE the step
|
||||
session('step/start'); emit agent/step-start
|
||||
req = {model, system, tools, messages: session.deriveMessages(), signal}
|
||||
req = waterfall agent/request ⟵ hooks, compaction, model switch
|
||||
req = waterfall agent/request ⟵ hooks, model switch
|
||||
stream ctx.llm.stream(req) ⟵ waterfall llm/stream (raw chunks)
|
||||
session('assistant/chunk'); emit agent/stream-chunk
|
||||
if assembler.finish is error/aborted: throw ⟵ adapter's in-band error path →
|
||||
@@ -194,7 +199,7 @@ Every MVP feature (including the TODO-marked ones), with the mechanism that impl
|
||||
| `/loop` | on `agent/turn-end`, `send()` the next iteration; or force-continue |
|
||||
| Dynamic workflow | orchestrator plugin on `agent/turn-end` / `agent/step-end` driving `send`/`steer` (+ sub-agents later) |
|
||||
| Queued + steering messages | core `Agent.send()` / `Agent.steer()` |
|
||||
| Context compaction (auto + manual) | the `ctx.compact` seam ([dsh-compact](../packages/compact/compact)): a backend summarizes an older surface range into a single `user/message` `replace` op, bracketed by log-only `compact/*` events; auto = check token pressure at turn boundaries, manual = a `/compact` tool. See the [compaction capability-seam RFC](rfc/proposed/feature/2026-06-18-compaction-capability-seam.md) |
|
||||
| Context compaction (auto + manual) | the `dsh-compact` seam (`ctx.compact`) + a backend (`dsh-compact-basic`) on the serial `agent/pre-step` seam: a backend summarizes an older surface range into a single `user/message` `replace` op, bracketed by log-only `compact/*` events; auto = check token pressure before each step — runaway-turn survival, manual = a (deferred) `/compact` tool invoking the same `ctx.compact` routine. See the [compaction capability-seam RFC](rfc/implemented/feature/2026-06-18-compaction-capability-seam.md) |
|
||||
| Session fork | the `ctx.sessionFork` seam ([dsh-session-fork](../packages/session-fork/session-fork)): validate the source is at a turn boundary, snapshot its seed, and create a child session with `parentSession`/`seedLength` metadata. |
|
||||
| System prompt configurability | `ctx.systemPrompt.section()` with ordering |
|
||||
| AGENTS.md (root) | a section provider reading the file |
|
||||
@@ -204,7 +209,7 @@ Every MVP feature (including the TODO-marked ones), with the mechanism that impl
|
||||
| Tool sandbox (landlock / sandbox-exec) | wrap `tools/execute`, or implement a sandboxing `BashExecutor` (the dsh-bash seam) |
|
||||
| Permission system / AskUserQuestion | wrap `tools/execute` (veto or ask); register an ask tool |
|
||||
| Plan mode | wrap `tools/execute` (deny writes) + `agent/request` (inject mode prompt) |
|
||||
| Sub-agents (spawn / fork / steer) | TODO seam on `AgentLoop.create()`; fork = seed Session with parent events; `steer()` on the child handle |
|
||||
| Sub-agent delegation | Implemented as the `ctx.subagents` provider-registry seam: `dsh-subagent-spawn` starts a fresh in-process child, `dsh-subagent-fork` seeds a child from the parent's completed-turn prefix, `dsh-subagent-acp` drives an out-of-process child over ACP, and `dsh-tool-subagent` exposes one configured provider to the model |
|
||||
| MCP | one plugin per server: discover tools → `ctx.tools.register()` |
|
||||
| Skills | section + tool registration; `inject()` skill content on invocation |
|
||||
| Memory | section provider + tool |
|
||||
@@ -222,7 +227,7 @@ Code skeletons for the three plugin shapes (tool, hook/permission-gate, UI) and
|
||||
|
||||
Tracked here deliberately — each is designed-for but not implemented:
|
||||
|
||||
- **Sub-agent spawn/fork semantics** (seam: `AgentLoop.create()`); inter-agent channels beyond `send`/`steer`/events.
|
||||
- **Compaction implementation** (auto thresholds, summarization prompts) on the `agent/request` seam, with its session-event types added by declaration merging.
|
||||
- **Inter-agent channels beyond delegation** (shared state, streaming child output, background/poll semantics) remain out of scope for the current `ctx.subagents` seam.
|
||||
- **Compaction** — the `dsh-compact` seam (`ctx.compact`) and the `dsh-compact-basic` backend exist (auto thresholds, summarization on the serial `agent/pre-step` seam, `compact/*` session events via declaration merging). The model-facing `/compact` consumer tool is still deferred. See [the compaction capability-seam RFC](rfc/implemented/feature/2026-06-18-compaction-capability-seam.md).
|
||||
- **Parallel tool execution** (concurrency-safety hints on ToolDefinition).
|
||||
- **Session branching/tree** (pi-style entry tree) if needed beyond the current seed-based `ctx.sessionFork` service.
|
||||
|
||||
@@ -11,7 +11,7 @@ The **harness tier** below (the `@deepseek-ai/dsh-*` packages) is the vocabulary
|
||||
|
||||
## Events
|
||||
|
||||
Dispatch modes: **emit** (fire-and-forget), **waterfall** (each listener gets `next()` and may transform or veto — see [waterfall semantics](../architecture.md#cordis-waterfall-semantics-important)), **parallel** (awaited fan-out, no veto).
|
||||
Dispatch modes: **emit** (fire-and-forget), **waterfall** (each listener gets `next()` and may transform or veto — see [waterfall semantics](../architecture.md#cordis-waterfall-semantics-important)), **parallel** (awaited fan-out; all listeners run), **serial** (awaited in registration order until one returns a bail value — anything other than `null`, `false`, or `undefined`).
|
||||
|
||||
### `agent/*`
|
||||
|
||||
@@ -49,7 +49,21 @@ A step or turn errored. The loop reports a failure here (plus the logger) even w
|
||||
|
||||
Types: [Agent](../core-data-structures/core.md)
|
||||
|
||||
Source: [`packages/core/agent/src/types.ts:220`](../../packages/core/agent/src/types.ts)
|
||||
Source: [`packages/core/agent/src/types.ts:254`](../../packages/core/agent/src/types.ts)
|
||||
|
||||
#### `agent/pre-step` — serial
|
||||
|
||||
Awaited pre-step surface-mutation checkpoint, fired once per step AFTER `turn/start` (and after the prior step closed) but BEFORE this step's `step/start` — so anything a listener appends lands OUTSIDE the step, between `turn/start`/`step/end` and the upcoming `step/start`. `step` is the number of the step about to start. The loop awaits `ctx.serial('agent/pre-step', …)` after assembling the system prompt, then opens the step and derives the request history ONCE from whatever the surface now holds. This is where compaction belongs: it mutates the session surface in place (shadowing an older range with a summary node) with its log-only `compact/*` records cleanly outside any step, and the single subsequent derive reflects the mutation — so there is no double-derive and no listener can see (or be expected to act on) an assembled `messages` array that does not exist yet.
|
||||
|
||||
Serial (awaited in registration order), not a waterfall: a listener mutates the surface as a side effect; there is nothing to transform, but the loop must wait for the mutation to complete before opening the step and deriving. Cordis `serial` bails early if a listener returns a bail value; this event is typed and documented as `void`, so listeners must not return a semantic veto value. `fullSystemPrompt` is the assembled prompt a listener needs to measure pressure (the system prompt counts toward the budget). `signal` cancels any in-flight work a listener starts (e.g. a summarization model call).
|
||||
|
||||
```ts cordis-catalog
|
||||
'agent/pre-step'(agent: Agent, turn: number, step: number, fullSystemPrompt: string, signal: AbortSignal): Promise<void> | void
|
||||
```
|
||||
|
||||
Types: [Agent](../core-data-structures/core.md)
|
||||
|
||||
Source: [`packages/core/agent/src/types.ts:214`](../../packages/core/agent/src/types.ts)
|
||||
|
||||
#### `agent/queued` — emit
|
||||
|
||||
@@ -65,7 +79,7 @@ Source: [`packages/core/agent/src/types.ts:156`](../../packages/core/agent/src/t
|
||||
|
||||
#### `agent/request` — waterfall
|
||||
|
||||
Waterfall: mutate the fully-assembled GenerateOptions before the model call (hooks, compaction, model switching, tool filtering, …). Call `next()` to delegate, or return without it to short-circuit.
|
||||
Waterfall: mutate the fully-assembled GenerateOptions before the model call (hooks, model switching, tool filtering, …). Call `next()` to delegate, or return without it to short-circuit. For surface mutation that must precede history derivation (compaction), use agent/pre-step instead — by the time this fires, `options.messages` is already derived.
|
||||
|
||||
```ts cordis-catalog
|
||||
'agent/request'(agent: Agent, turn: number, step: number, options: GenerateOptions, next: () => Promise<GenerateOptions>): Promise<GenerateOptions>
|
||||
@@ -73,7 +87,7 @@ Waterfall: mutate the fully-assembled GenerateOptions before the model call (hoo
|
||||
|
||||
Types: [Agent](../core-data-structures/core.md) · [GenerateOptions](../core-data-structures/core.md)
|
||||
|
||||
Source: [`packages/core/agent/src/types.ts:189`](../../packages/core/agent/src/types.ts)
|
||||
Source: [`packages/core/agent/src/types.ts:223`](../../packages/core/agent/src/types.ts)
|
||||
|
||||
#### `agent/status` — emit
|
||||
|
||||
@@ -97,7 +111,7 @@ Steering content was injected into a running turn.
|
||||
|
||||
Types: [Agent](../core-data-structures/core.md) · [ContentBlock](../core-data-structures/core.md) · [MessageSource](../core-data-structures/core.md)
|
||||
|
||||
Source: [`packages/core/agent/src/types.ts:214`](../../packages/core/agent/src/types.ts)
|
||||
Source: [`packages/core/agent/src/types.ts:248`](../../packages/core/agent/src/types.ts)
|
||||
|
||||
#### `agent/step-end` — emit
|
||||
|
||||
@@ -121,7 +135,7 @@ Waterfall: post-process the assembled assistant Message before tool dispatch (va
|
||||
|
||||
Types: [Agent](../core-data-structures/core.md) · [Message](../core-data-structures/core.md)
|
||||
|
||||
Source: [`packages/core/agent/src/types.ts:195`](../../packages/core/agent/src/types.ts)
|
||||
Source: [`packages/core/agent/src/types.ts:229`](../../packages/core/agent/src/types.ts)
|
||||
|
||||
#### `agent/step-start` — emit
|
||||
|
||||
@@ -145,7 +159,7 @@ A raw StreamChunk arrived from the model (token-level UI/log feed).
|
||||
|
||||
Types: [Agent](../core-data-structures/core.md) · [StreamChunk](../core-data-structures/llm-streaming.md)
|
||||
|
||||
Source: [`packages/core/agent/src/types.ts:209`](../../packages/core/agent/src/types.ts)
|
||||
Source: [`packages/core/agent/src/types.ts:243`](../../packages/core/agent/src/types.ts)
|
||||
|
||||
#### `agent/turn-continuation` — waterfall
|
||||
|
||||
@@ -157,7 +171,7 @@ Waterfall: override the turn-continuation decision. The default (computed by the
|
||||
|
||||
Types: [Agent](../core-data-structures/core.md)
|
||||
|
||||
Source: [`packages/core/agent/src/types.ts:202`](../../packages/core/agent/src/types.ts)
|
||||
Source: [`packages/core/agent/src/types.ts:236`](../../packages/core/agent/src/types.ts)
|
||||
|
||||
#### `agent/turn-end` — emit
|
||||
|
||||
@@ -207,7 +221,7 @@ A session was created in the store.
|
||||
'session/created'(session: Session): void
|
||||
```
|
||||
|
||||
Source: [`packages/core/session/src/index.ts:33`](../../packages/core/session/src/index.ts)
|
||||
Source: [`packages/core/session/src/index.ts:34`](../../packages/core/session/src/index.ts)
|
||||
|
||||
#### `session/event` — emit
|
||||
|
||||
@@ -219,7 +233,7 @@ An event was appended to a session log (sync, fire-and-forget). This is the per-
|
||||
|
||||
Types: [SessionEvent](../core-data-structures/core.md)
|
||||
|
||||
Source: [`packages/core/session/src/index.ts:39`](../../packages/core/session/src/index.ts)
|
||||
Source: [`packages/core/session/src/index.ts:40`](../../packages/core/session/src/index.ts)
|
||||
|
||||
#### `session/flush` — parallel
|
||||
|
||||
@@ -229,7 +243,7 @@ Awaited durability checkpoint. The agent loop awaits `ctx.parallel('session/flus
|
||||
'session/flush'(session: Session): Promise<void> | void
|
||||
```
|
||||
|
||||
Source: [`packages/core/session/src/index.ts:48`](../../packages/core/session/src/index.ts)
|
||||
Source: [`packages/core/session/src/index.ts:49`](../../packages/core/session/src/index.ts)
|
||||
|
||||
### `subagent/*`
|
||||
|
||||
@@ -373,11 +387,11 @@ Implementations MUST honor:
|
||||
- **Blocking**: no compaction begins while another is in progress for the same session. The recommended mechanism is the log-recorded lock — append `compact/start` before the slow work and `compact/end` after (even on failure) — so the lock is visible to replay and crash recovery.
|
||||
|
||||
```ts cordis-catalog
|
||||
abstract compactIfNeeded( session: Session, systemPrompt?: string, model?: string, signal?: AbortSignal, ): Promise<CompactionResult | null>
|
||||
abstract compactRegion( session: Session, start: number, end: number, model: string, signal?: AbortSignal, ): Promise<CompactionResult>
|
||||
abstract compactIfNeeded( agent: CompactAgentContext, turn: number, step: number, fullSystemPrompt: string, signal: AbortSignal, ): Promise<CompactionResult | null>
|
||||
abstract compactRegion( session: Session, start: number, end: number, agent: CompactAgentContext, turn: number, step: number, signal?: AbortSignal, ): Promise<CompactionResult>
|
||||
```
|
||||
|
||||
Source: [`packages/compact/compact/src/index.ts:57`](../../packages/compact/compact/src/index.ts)
|
||||
Source: [`packages/compact/compact/src/index.ts:63`](../../packages/compact/compact/src/index.ts)
|
||||
|
||||
### `ctx.llm` — `LlmService`
|
||||
|
||||
@@ -441,7 +455,7 @@ get(id: SessionId): Session | undefined
|
||||
list(): Session[]
|
||||
```
|
||||
|
||||
Source: [`packages/core/session/src/index.ts:321`](../../packages/core/session/src/index.ts)
|
||||
Source: [`packages/core/session/src/index.ts:322`](../../packages/core/session/src/index.ts)
|
||||
|
||||
### `ctx.subagents` — `SubagentService`
|
||||
|
||||
@@ -508,7 +522,7 @@ The framework surface every plugin inherits, beyond the harness vocabulary above
|
||||
### Inherited `ctx` members
|
||||
|
||||
- `ctx.on / ctx.once` — Register an event listener (disposable). ([`vendor/cordis/src/events.ts:29`](../../vendor/cordis/src/events.ts))
|
||||
- `ctx.emit / ctx.parallel / ctx.serial / ctx.bail / ctx.waterfall` — Dispatch an event (sync / awaited / first-non-nullish / veto-chain). ([`vendor/cordis/src/events.ts:29`](../../vendor/cordis/src/events.ts))
|
||||
- `ctx.emit / ctx.parallel / ctx.serial / ctx.bail / ctx.waterfall` — Dispatch an event (sync / awaited / first-bail / veto-chain). ([`vendor/cordis/src/events.ts:29`](../../vendor/cordis/src/events.ts))
|
||||
- `ctx.plugin / ctx.inject` — Load a plugin / declare required services. ([`vendor/cordis/src/registry.ts:144`](../../vendor/cordis/src/registry.ts))
|
||||
- `ctx.effect` — Register a disposable side effect tied to the fiber. ([`vendor/cordis/src/fiber.ts:9`](../../vendor/cordis/src/fiber.ts))
|
||||
- `ctx.get / ctx.set / ctx.provide / ctx.accessor / ctx.mixin` — Low-level service-store access and binding. ([`vendor/cordis/src/reflect.ts:7`](../../vendor/cordis/src/reflect.ts))
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Compaction
|
||||
|
||||
The compaction seam — a [capability seam](../rfc/implemented/architecture/2026-06-13-capability-seams.md) split like bash: interface ([dsh-compact](../../packages/compact/compact), `ctx.compact`), implementation (a backend such as `dsh-compact-basic`, deferred), and consumer (a `/compact` tool, deferred). Compaction is **one optional capability**, not part of the agent-loop spine — so its vocabulary lives here, not in [core.md](core.md). A tokenizer- or template-based backend is a sibling package implementing the same interface. Unlike bash, the interface necessarily depends on `dsh-session` and `dsh-llm`: its verbs are defined over a `Session` and its output is the `ContentBlock` vocabulary (see the [compaction capability-seam RFC](../rfc/proposed/feature/2026-06-18-compaction-capability-seam.md)).
|
||||
The compaction seam — a [capability seam](../rfc/implemented/architecture/2026-06-13-capability-seams.md) split like bash: interface ([dsh-compact](../../packages/compact/compact), `ctx.compact`), implementation (a backend such as [dsh-compact-basic](../../packages/compact/compact-basic)), and consumer (a `/compact` tool, deferred). Compaction is **one optional capability**, not part of the agent-loop spine — so its vocabulary lives here, not in [core.md](core.md). A tokenizer- or template-based backend is a sibling package implementing the same interface. Unlike bash, the interface necessarily depends on `dsh-session` and `dsh-llm`: its verbs are defined over a `Session` and its output is the `ContentBlock` vocabulary (see the [compaction capability-seam RFC](../rfc/implemented/feature/2026-06-18-compaction-capability-seam.md)).
|
||||
|
||||
Source: [`packages/compact/compact/src/types.ts`](../../packages/compact/compact/src/types.ts)
|
||||
|
||||
@@ -11,7 +11,7 @@ Compaction extends [`SessionEventMap`](session.md) with three event types via de
|
||||
| Event | Payload | Role |
|
||||
|---|---|---|
|
||||
| `compact/start` | `{ turn }` | acquires the log-recorded lock |
|
||||
| `compact/summary` | `{ summary, shadowedRange, shadowedSeqs, shadowedTokenCount }` | provenance: the summary blocks, the shadowed seq range, and the estimated token count |
|
||||
| `compact/summary` | `{ summary, shadowedRange, shadowedSeqs, shadowedTokenCount }` | provenance: the summary blocks, the shadowed surface-boundary pair (`start`/`end` seqs — a position span, not a numeric interval), the shadowed seqs in surface order, and the estimated token count |
|
||||
| `compact/end` | `{ turn, error? }` | releases the lock (`error` set when summarization threw) |
|
||||
|
||||
The lock brackets the **whole** operation: `compact/start` is appended first, then summarization, the `compact/summary` provenance record, and the `user/message` replacement all land, and only then `compact/end`. Releasing the lock last turns a crash mid-operation into a detectable orphaned lock (a `compact/start` with no matching `compact/end`) rather than a `compact/end` that falsely claims compaction finished.
|
||||
@@ -32,9 +32,16 @@ interface CompactionResult {
|
||||
endSeq: number
|
||||
/** The summary content blocks produced by the backend. */
|
||||
summary: ContentBlock[]
|
||||
/** The seq range that was shadowed [start, end] inclusive. */
|
||||
/**
|
||||
* The surface-boundary pair that was shadowed: the seqs of the first
|
||||
* (`start`) and last (`end`) surface nodes of the replaced range. A
|
||||
* surface-POSITION span, not a numeric seq interval — after a prior replace
|
||||
* lands a fresh high-seq summary node at an older range's position, `start`
|
||||
* can be GREATER than `end`. {@link CompactionResult.shadowedSeqs} is the
|
||||
* authoritative set of shadowed nodes, in surface order.
|
||||
*/
|
||||
shadowedRange: { start: number; end: number }
|
||||
/** The seq numbers of all shadowed surface nodes. */
|
||||
/** The seqs of all shadowed surface nodes, in surface order. */
|
||||
shadowedSeqs: number[]
|
||||
/** Estimated token count of the shadowed content. */
|
||||
shadowedTokenCount: number
|
||||
@@ -43,4 +50,6 @@ interface CompactionResult {
|
||||
|
||||
## The service
|
||||
|
||||
`CompactService` (`ctx.compact`, abstract — defined in [`packages/compact/compact/src/index.ts`](../../packages/compact/compact/src/index.ts)) declares two abstract methods: `compactIfNeeded(session, systemPrompt?, model?, signal?)` checks token pressure and compacts an older range if the history is too large (returning `null` when nothing needs it), and `compactRegion(session, start, end, model, signal?)` forcibly summarizes surface nodes `[start, end]` into a single replacement node. Both take an optional `signal: AbortSignal` that a backend summarizing via `ctx.llm.stream()` must forward into the call's `GenerateOptions.signal`, so an abort or dispose tears down the in-flight summarization. The entire strategy — token estimation, retention policy, event sequencing, summarization — is a HOW decision owned by the implementation.
|
||||
`CompactService` (`ctx.compact`, abstract — defined in [`packages/compact/compact/src/index.ts`](../../packages/compact/compact/src/index.ts)) declares two abstract methods: `compactIfNeeded(agent, turn, step, fullSystemPrompt, signal)` checks token pressure and compacts an older range if the history is too large (returning `null` when nothing needs it), and `compactRegion(session, start, end, agent, turn, step, signal?)` forcibly summarizes surface nodes `[start, end]` into a single replacement node. `compactIfNeeded`'s parameters are all required — the loop's `agent/pre-step` checkpoint supplies the agent, lifecycle context, assembled `fullSystemPrompt`, and turn `signal`. A backend summarizing via `ctx.llm.stream()` must forward `signal` into the call's `GenerateOptions.signal`, so an abort or dispose tears down the in-flight summarization. The entire strategy — token estimation, retention policy, event sequencing, summarization — is a HOW decision owned by the implementation.
|
||||
|
||||
Auto-compaction runs on the serial `agent/pre-step` loop seam (fired once per step, after `turn/start` and BEFORE the step opens and its request history is derived), not the `agent/request` waterfall: compaction mutates the session surface in place — with its log-only `compact/*` records landing cleanly outside any step — and the loop derives the request from the already-compacted surface. Retention is turn-agnostic — the only structural guard is tool-pairing balance (a compacted region's edges are balanced cuts on the surface, so it never splits a step's tool-calls from their results), so a single runaway turn that alone exceeds the window compacts its own early closed steps rather than being retained verbatim. The backend that ships this (`dsh-compact-basic`) documents the retention walk, summary shrink validation, bounded re-compaction, and the crash/recoverable failure taxonomy.
|
||||
|
||||
@@ -206,7 +206,7 @@ type SessionEvent<T extends SessionEventType = SessionEventType> = {
|
||||
/**
|
||||
* Seq numbers of events that are provenance sources of this event
|
||||
* (e.g. the `assistant/chunk` seqs that built an `assistant/message`,
|
||||
* or the surface nodes shadowed by a compaction marker).
|
||||
* or the surface nodes shadowed by a compaction replace node).
|
||||
*/
|
||||
sourceEventSeqs?: number[]
|
||||
/** How this event entered the surface; absent for non-surface events. */
|
||||
@@ -307,7 +307,7 @@ interface Agent {
|
||||
}
|
||||
```
|
||||
|
||||
`AgentStatus` is `'idle' | 'running' | 'disposed'`. `AgentId` is a branded string. `AgentOptions` (`model?`, `systemPrompt?`) is merge-extensible — plugins add creation options by declaration merging. The `agent/*` event taxonomy (lifecycle, turn/step boundaries, the `agent/request`/`agent/step-result`/`agent/turn-continuation` waterfalls) is in [architecture.md § Event taxonomy](../architecture.md#event-taxonomy).
|
||||
`AgentStatus` is `'idle' | 'running' | 'disposed'`. `AgentId` is a branded string. `AgentOptions` (`model?`, `systemPrompt?`) is merge-extensible — plugins add creation options by declaration merging. The `agent/*` event taxonomy (lifecycle, turn/step boundaries, the serial `agent/pre-step` surface-mutation seam, and the `agent/request`/`agent/step-result`/`agent/turn-continuation` waterfalls) is in [architecture.md § Event taxonomy](../architecture.md#event-taxonomy).
|
||||
|
||||
## `ToolDefinition`
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ A backend that reloads a log crashed mid-turn finds an open `turn/start` with no
|
||||
|
||||
## `SessionHeader` — metadata beside the log
|
||||
|
||||
Per-session metadata travels **separately** from the event log: format version, cwd, and lineage are storage concerns, not conversation events, so they stay out of `SessionEventMap` and never reach `deriveMessages()`. The header is attached to a `Session` via `session.header`.
|
||||
Per-session metadata travels **separately** from the event log: format version, cwd, lineage, and the seed boundary are storage concerns, not conversation events, so they stay out of `SessionEventMap` and never reach `deriveMessages()`. The header is attached to a `Session` via `session.header`.
|
||||
|
||||
Source: [`packages/core/session/src/types.ts`](../../packages/core/session/src/types.ts)
|
||||
|
||||
@@ -78,6 +78,6 @@ Replay/fork is therefore `ctx.sessions.create(id, { seed: seedEvents })`; resumi
|
||||
Both implement the same abstract `SessionPersistence` (create/append/load/list over `SessionEvent`) and pass `runPersistenceContract`, proving the seam is genuinely backend-agnostic:
|
||||
|
||||
- **[dsh-session-persistence-jsonl](../../packages/session-persistence/session-persistence-jsonl)** — an append-only JSONL log per session with crash-safe atomic writes, the interrupted-turn crash recovery above, and a read/replay path.
|
||||
- **[dsh-session-persistence-sqlite](../../packages/session-persistence/session-persistence-sqlite)** — `node:sqlite`, one row per `SessionEvent`. The row shape `(session_id, seq, type, time, data)` maps 1:1 onto the event, so there is no parallel persisted schema to keep in sync.
|
||||
- **[dsh-session-persistence-sqlite](../../packages/session-persistence/session-persistence-sqlite)** — `node:sqlite`, one row per `SessionEvent`. The row shape `(session_id, seq, type, time, data, source_event_seqs, surface_op)` maps 1:1 onto the event, including optional surface metadata, so there is no parallel persisted schema to keep in sync.
|
||||
|
||||
Multiple backends sharing one on-disk session coordinate writes through the [shared persistence write-coordinator](../rfc/implemented/architecture/2026-06-18-shared-persistence-write-coordinator.md).
|
||||
|
||||
@@ -80,7 +80,7 @@ type SessionEvent<T extends SessionEventType = SessionEventType> = {
|
||||
/**
|
||||
* Seq numbers of events that are provenance sources of this event
|
||||
* (e.g. the `assistant/chunk` seqs that built an `assistant/message`,
|
||||
* or the surface nodes shadowed by a compaction marker).
|
||||
* or the surface nodes shadowed by a compaction replace node).
|
||||
*/
|
||||
sourceEventSeqs?: number[]
|
||||
/** How this event entered the surface; absent for non-surface events. */
|
||||
|
||||
@@ -61,7 +61,7 @@ lefthook is configured in `lefthook.yml` as an early local checkpoint before rev
|
||||
|
||||
The vendor manifest guard checks that changes under `vendor/*/src` are staged with the matching `vendor/README.md` manifest update. See `vendor/README.md` before editing vendored code.
|
||||
|
||||
These hooks do not exactly mirror CI. Notably, `pre-push` runs unit tests without coverage, while CI runs `pnpm run test:coverage`; CI also runs an echo-agent smoke test and exercises the matrix on Node 24 and 26.
|
||||
These hooks do not exactly mirror CI. Notably, `pre-push` runs unit tests without coverage, while CI runs `pnpm run test:coverage`; CI also runs echo-agent and built-bin smoke tests and exercises the matrix on Node 24 and 26.
|
||||
|
||||
## CI gates
|
||||
|
||||
@@ -78,6 +78,7 @@ The GitHub workflow runs these gates on each pull request:
|
||||
- `pnpm run build`
|
||||
- `pnpm run hygiene`
|
||||
- an echo-agent smoke test that checks the demo's tool call, tool result, and JSONL output
|
||||
- built-bin smoke tests that run the published `lib/bin.js` entrypoints under plain `node`
|
||||
|
||||
`pnpm run hygiene` is the local shorthand for `pnpm run knip && pnpm run publint && pnpm run constraints && pnpm run verify-node-next-types`; CI also runs `pnpm run constraints` as an earlier fail-fast step, then runs the full hygiene script after `pnpm run build`.
|
||||
|
||||
@@ -121,6 +122,12 @@ The coding-agent demo uses the real DeepSeek adapter and needs `DEEPSEEK_API_KEY
|
||||
pnpm run demo:coding
|
||||
```
|
||||
|
||||
The ACP server demo exposes the same coding agent over JSON-RPC stdio and also needs `DEEPSEEK_API_KEY`:
|
||||
|
||||
```sh
|
||||
pnpm run demo:acp
|
||||
```
|
||||
|
||||
## TODO markers
|
||||
|
||||
Use one of three comment tags to flag known issues in the code, ordered by urgency:
|
||||
|
||||
113
docs/i18n/terminology.md
Normal file
113
docs/i18n/terminology.md
Normal file
@@ -0,0 +1,113 @@
|
||||
# Terminology
|
||||
|
||||
本表约定本仓库的中英术语统一译法。
|
||||
|
||||
| English | 中文 | 备注 |
|
||||
|---|---|---|
|
||||
| ACP | ACP | 首次出现可写：ACP（Agent Client Protocol） |
|
||||
| AI | AI | 首次出现可写：人工智能（AI） |
|
||||
| API | API | |
|
||||
| CLI | CLI | 首次出现可写：命令行界面（CLI） |
|
||||
| Cordis | Cordis | 保留英文 |
|
||||
| Function Calling | Function Calling | 首次出现可写：Function Calling（函数调用） |
|
||||
| HMR | HMR | 首次出现可写：热模块替换（HMR） |
|
||||
| JSON Schema | JSON Schema | |
|
||||
| JSONL | JSONL | |
|
||||
| lint | lint | |
|
||||
| loader | loader | |
|
||||
| LLM | LLM | 首次出现可写：大语言模型（LLM） |
|
||||
| MCP | MCP | |
|
||||
| RAG | RAG | 首次出现可写：检索增强生成（RAG） |
|
||||
| SDK | SDK | |
|
||||
| SSE | SSE | 首次出现可写：SSE（Server-Sent Events） |
|
||||
| agent | agent | 首次出现可写：agent（智能体） |
|
||||
| agent loop | agent loop | |
|
||||
| fiber | fiber | 首次出现可写：fiber（插件运行时） |
|
||||
| fixture | fixture | 指测试前置数据或环境 |
|
||||
| fork | fork | 保留英文 |
|
||||
| harness | harness | 保留英文 |
|
||||
| manifest | manifest | 描述模块或工具元数据的文件 |
|
||||
| schema DSL | schema DSL | |
|
||||
| schema | schema | 保留英文 |
|
||||
| seam | seam | 首次出现可写：seam（扩展点） |
|
||||
| skill | skill | 首次出现可写：skill（技能） |
|
||||
| spawn | spawn | 保留英文 |
|
||||
| steering | steering | 首次出现可写：steering（中途引导） |
|
||||
| subagent | subagent | 首次出现可写：subagent（子 agent） |
|
||||
| transcript | transcript | 首次出现可写：transcript（文本记录）；指会话渲染给用户或编辑器的完整文本，区别于事件日志（event log） |
|
||||
| waterfall | waterfall | 首次出现可写：waterfall（瀑布式事件） |
|
||||
| wire format | 协议格式 | 首次出现可写：协议格式（wire format） |
|
||||
| adapter contract | 适配器契约 | 首次出现可写：适配器契约（adapter contract） |
|
||||
| adapter | 适配器 | |
|
||||
| append-only | 仅追加 | |
|
||||
| artifact | 产物 | |
|
||||
| block | 块 | |
|
||||
| background task | 后台任务 | |
|
||||
| backend | 后端 | |
|
||||
| capability | 能力 | |
|
||||
| cancel | 取消 | |
|
||||
| checkpoint | 检查点 | |
|
||||
| chunk | 分片 | |
|
||||
| compaction | compaction | 首次出现可写：compaction（上下文压缩）；正文优先保留英文 |
|
||||
| consumer | 消费方 | |
|
||||
| content block | 内容块 | |
|
||||
| config | 配置 | |
|
||||
| context | 上下文 | |
|
||||
| context compaction | 上下文压缩 | 首次出现可写：上下文压缩（context compaction） |
|
||||
| coverage | 覆盖率 | |
|
||||
| crash recovery | 崩溃恢复 | |
|
||||
| dispose | dispose | 首次出现可写：dispose（释放资源）；正文优先保留英文 |
|
||||
| durability | 持久性 | |
|
||||
| event log | 事件日志 | |
|
||||
| event | 事件 | |
|
||||
| event stream | 事件流 | |
|
||||
| executor | 执行器 | |
|
||||
| extension | 扩展 | |
|
||||
| finish reason | 结束原因 | |
|
||||
| foreground run | 前台运行 | |
|
||||
| hook | 钩子 | |
|
||||
| implementation | 实现 | |
|
||||
| inference | 推理（inference） | 每次提及时保留英文括注，避免与 reasoning 混淆 |
|
||||
| injection | 注入 | |
|
||||
| interface | 接口 | |
|
||||
| integration | 集成 | |
|
||||
| memory | memory / 记忆 / 内存 | 按上下文区分：agent memory 译为“记忆”；resource/memory usage 译为“内存” |
|
||||
| message | 消息 | |
|
||||
| mod | 模组 | 区别于 module（模块）；plugin 译作「插件」 |
|
||||
| model provider | 模型提供方 | |
|
||||
| module | 模块 | |
|
||||
| permission | 权限 | |
|
||||
| persistence | 持久化 | |
|
||||
| pipeline | 流水线 | |
|
||||
| plugin | 插件 | mod 对应“模组” |
|
||||
| prompt | 提示词 | |
|
||||
| provider | 提供方 | |
|
||||
| provider-neutral | 提供方无关 | |
|
||||
| quality gate | 质量门禁 | |
|
||||
| registry | 注册表 | |
|
||||
| reasoning | 推理（reasoning） | 需要和 inference 区分时保留英文括注；`reasoning_content` 译为“思考内容” |
|
||||
| replay | 回放 | |
|
||||
| resume | 恢复 | |
|
||||
| runtime | 运行时 | |
|
||||
| sandbox | 沙箱 | |
|
||||
| service | 服务 | |
|
||||
| session | 会话 | |
|
||||
| session event | 会话事件 | |
|
||||
| snapshot | 快照 | |
|
||||
| spine | 主干 | |
|
||||
| step | 步骤 | |
|
||||
| stream | 流 | |
|
||||
| streaming | 流式输出 | |
|
||||
| system prompt | 系统提示词 | |
|
||||
| taxonomy | 分类体系 | |
|
||||
| token usage | token 用量 | |
|
||||
| thinking | thinking | API 字段保留；模型模式译为“思考” |
|
||||
| tool | 工具 | |
|
||||
| tool call | 工具调用 | |
|
||||
| tool result | 工具结果 | |
|
||||
| tool schema | 工具 schema | |
|
||||
| toolkit | 工具包 | |
|
||||
| turn | 轮次 | |
|
||||
| typecheck | 类型检查 | |
|
||||
| vocabulary | 词汇 | |
|
||||
| workflow | 工作流 | |
|
||||
@@ -24,6 +24,10 @@ graph TD
|
||||
llm-replay --> session
|
||||
session-fork --> session
|
||||
session-persistence --> session
|
||||
compact-basic --> agent
|
||||
compact-basic --> compact
|
||||
compact-basic --> llm
|
||||
compact-basic --> session
|
||||
invariants --> agent
|
||||
invariants --> llm
|
||||
invariants --> session
|
||||
@@ -111,6 +115,7 @@ graph TD
|
||||
| `llm-replay` | `llm`, `session` |
|
||||
| `session-fork` | `session` |
|
||||
| `session-persistence` | `session` |
|
||||
| `compact-basic` | `agent`, `compact`, `llm`, `session` |
|
||||
| `invariants` | `agent`, `llm`, `session` |
|
||||
| `session-persistence-jsonl` | `session`, `session-persistence` |
|
||||
| `session-persistence-sqlite` | `session`, `session-persistence` |
|
||||
|
||||
@@ -44,7 +44,6 @@ Do NOT write one for a mechanical or local choice (a variable name, a one-file r
|
||||
| [Agent Client Protocol (ACP) support for external editors](proposed/feature/2026-06-14-acp-agent-client-protocol.md) | 2026-06-14 |
|
||||
| [Multiplex concurrent ACP sessions over one connection](proposed/feature/2026-06-14-acp-multi-session.md) | 2026-06-14 |
|
||||
| [Optional Code Mode — model writes TypeScript against an SDK of all tools](proposed/feature/2026-06-15-optional-code-mode.md) | 2026-06-15 |
|
||||
| [Compaction as a capability seam (abstract contract + basic backend)](proposed/feature/2026-06-18-compaction-capability-seam.md) | 2026-06-18 |
|
||||
|
||||
### Simplification
|
||||
|
||||
@@ -83,6 +82,7 @@ Do NOT write one for a mechanical or local choice (a variable name, a one-file r
|
||||
| Title | First proposed |
|
||||
|---|---|
|
||||
| [Rich ACP bash rendering — the terminal card (`_meta`) and command classification](implemented/feature/2026-06-18-acp-terminal-and-tool-rendering.md) | 2026-06-18 |
|
||||
| [Compaction as a capability seam (abstract contract + basic backend)](implemented/feature/2026-06-18-compaction-capability-seam.md) | 2026-06-18 |
|
||||
| [Subagent capability seam](implemented/feature/2026-06-21-subagent-capability-seam.md) | 2026-06-21 |
|
||||
| [ACP subagent backend (out-of-process delegation)](implemented/feature/2026-06-22-acp-subagent-backend.md) | 2026-06-22 |
|
||||
| [The `todo_write` tool — model task list as event-sourced session state](implemented/feature/2026-06-29-todo-write-tool.md) | 2026-06-29 |
|
||||
@@ -136,6 +136,7 @@ Do NOT write one for a mechanical or local choice (a variable name, a one-file r
|
||||
| [Core-data-structures catalog and the `ts type-equiv` drift gate](implemented/process/2026-06-20-core-data-structures-catalog.md) | 2026-06-20 |
|
||||
| [Generated cordis events + services catalog](implemented/process/2026-06-20-generated-cordis-catalog.md) | 2026-06-20 |
|
||||
| [Classify RFCs by kind via path-encoded subdirectories](implemented/process/2026-06-20-rfc-classification.md) | 2026-06-20 |
|
||||
| [Generated tool-schema catalog (boot-and-harvest)](implemented/process/2026-07-02-tool-schema-catalog.md) | 2026-07-02 |
|
||||
|
||||
### Testing
|
||||
|
||||
|
||||
@@ -0,0 +1,120 @@
|
||||
# RFC: Compaction as a capability seam (abstract contract + basic backend)
|
||||
|
||||
Status: implemented (2026-06-18; retention/seam reform 2026-06-26)
|
||||
|
||||
## Context
|
||||
|
||||
A long-running agent conversation grows without bound. As the event log accumulates turns, the derived message history eventually approaches the model's context window — the model then truncates mid-response (`max-tokens`) or degrades. **Compaction** is the mitigation: replace a run of older history with a concise summary, keeping recent context intact.
|
||||
|
||||
The [session surface](../../implemented/architecture/2026-06-18-session-surface.md) was built as the foundation for exactly this — a linked list over the event log with a `surfaceOp: { op: 'replace', start, end }` operation purpose-built to shadow a range of nodes and insert a replacement, with `sourceEventSeqs` recording provenance so the decision replays deterministically. What remained was the plugin that *decides what to compact and produces the summary*.
|
||||
|
||||
Two forces shape the design. First, compaction is **swappable**: token counting can be a char/4 heuristic or a real tokenizer, and summarization can be a model call, a template, or a remote service — these vary independently of *when* and *which range* to compact. Second, a later commit (`ce43c25`) closed `SurfaceEventType` to five event types (`user/message`, `assistant/message`, `tool/result`, `context/message`, `steering/message`); only those may carry `surfaceOp`. A bespoke `compaction/*` event therefore **cannot** itself appear on the surface — the compiler rejects `surfaceOp` on it and the invariants plugin rejects it at runtime.
|
||||
|
||||
## Decision
|
||||
|
||||
### Compaction is a capability seam, split interface / implementation
|
||||
|
||||
Per the [capability-seams RFC](../../implemented/architecture/2026-06-13-capability-seams.md), compaction ships as separate packages so the contract, the algorithm, and (later) the consumer surface evolve independently:
|
||||
|
||||
1. **Interface** — `@deepseek-ai/dsh-compact`: an abstract `CompactService` owning the `ctx.compact` key, the `CompactionResult` vocabulary, and the `compact/*` session events. It declares `compactIfNeeded()` and `compactRegion()` as **abstract** — the contract states *what* compaction does, not *how*.
|
||||
2. **Implementation** — `@deepseek-ai/dsh-compact-basic`: a concrete `BasicCompactService` that owns the entire algorithm — token estimation (char/4 + per-block overhead), the tail→head retention walk, summarization via `ctx.llm.stream()`, the surface replacement, the lock, and the `agent/pre-step` auto-compaction listener. A tokenizer-based or template-based backend is a sibling package (or a subclass overriding the two protected estimation/summarization hooks).
|
||||
3. **Consumer** — deferred. A `/compact` tool and slash command will `inject: ['compact']` and call the contract; they are intentionally out of scope here so the seam settles first.
|
||||
|
||||
### The contract depends on `dsh-session` and `dsh-llm` — a deliberate deviation
|
||||
|
||||
The capability-seams RFC states the interface package "depends only on cordis" (true of `dsh-bash`, whose vocabulary is self-contained). Compaction **cannot** honor that: its verbs are defined *over* a `Session` (`compactRegion(session, start, end)`) and its output *is* the content vocabulary (`CompactionResult.summary: ContentBlock[]`). There is no way to express the contract without naming `Session`/`SessionEvent` (from `dsh-session`) and `ContentBlock` (from `dsh-llm`).
|
||||
|
||||
This is not a coupling smell — it is the contract's domain. The "only cordis" guidance was always shorthand for "the interface depends only on what the contract genuinely names, and never on an implementation." `dsh-session` and `dsh-llm` are themselves interface/vocabulary packages, not implementations; `dsh-compact` still imports no backend. The seam's real invariant — *consumers and implementations evolve independently behind an abstract service* — holds intact.
|
||||
|
||||
### Abstract `compactIfNeeded` / `compactRegion`, algorithm in the backend
|
||||
|
||||
An earlier draft put the full algorithm (the retention walk, token-summing, text extraction) as concrete methods on the interface, with only `estimateContentTokens()` and `summarize()` abstract. That recouples the contract to one strategy: a backend that wants a different retention policy or a different event-sequencing would have to fight inherited concrete code. Making both core methods abstract puts every *how* decision in the backend, where it belongs, and keeps the interface a pure statement of *what*. The backend remains internally factored — `estimateContentTokens()` and `summarize()` are `protected` hooks a sub-backend can override without reimplementing the walk — but that factoring is the backend's private concern, not the contract's.
|
||||
|
||||
`compactIfNeeded(agent, turn, step, fullSystemPrompt, signal)` takes **required** parameters (not the original all-optional shape). The auto-compaction seam (below) always supplies the agent, lifecycle context, assembled system prompt (counted toward the estimate), and the turn's abort signal, so optionality would only invite a hidden default at the seam. The session being compacted comes from the agent context. `compactRegion(session, start, end, agent, turn, step, signal?)` keeps an optional signal (a manual caller may omit it). Passing lifecycle context rather than a concrete model keeps router agents honest: the backend's summarization request can run through `agent/request`, where model-routing plugins already choose the actual model.
|
||||
|
||||
### Auto-compaction runs on `agent/pre-step`, a dedicated surface-mutation seam
|
||||
|
||||
Compaction is a **surface mutation**, not a request transform — and that distinction is the seam it belongs on. The loop's request lifecycle, per step, is: assemble the system prompt → open the step → derive the message history from the surface → run the `agent/request` waterfall → call the model. An earlier cut wedged compaction into the `agent/request` waterfall, which forced two problems: (1) the loop had already derived `messages` from the *stale* surface, so the listener had to mutate the surface and then *re-derive* and overwrite `request.messages` — a double-derive whose only purpose was to undo the premature first derive; and (2) `agent/request` also carries downstream-injected context a listener might have added to `request.messages`, which compaction cannot act on (it can only compact the surface), inviting the confusion of measuring tokens compaction can't shed.
|
||||
|
||||
The fix is a dedicated loop seam, **`agent/pre-step`** (`@mode serial`), fired by the loop *after* system assembly and *before* the step opens (`step/start`):
|
||||
|
||||
```
|
||||
assembly = ctx.systemPrompt.assemble()
|
||||
await ctx.serial('agent/pre-step', agent, turn, step, system, signal) ⟵ compaction mutates the surface here
|
||||
session('step/start') ⟵ the step opens AFTER the seam
|
||||
messages = session.deriveMessages() ⟵ single derive, reflects the compaction
|
||||
request = waterfall agent/request ⟵ pure request transform (hooks, model switch)
|
||||
```
|
||||
|
||||
This makes the layering correct *by construction*: compaction mutates the surface, the loop derives **once** from the result (no double-derive), and at `pre-step` the assembled `messages` do not yet exist — so a listener structurally *cannot* see or be expected to act on downstream-injected context. `agent/request` reverts to a pure request transformer. Firing the seam **before** `step/start` (not inside the open step) is load-bearing for crash-safety: compaction's log-only `compact/*` records and its replacement node land *outside* any step, so the honest log structure a crash leaves (a dangling `compact/start` sitting before the synthetic `turn/end` that turn-repair appends) holds without a half-open step to reconcile. The seam is `serial` (awaited, in registration order), not `parallel`: a listener mutates the surface as a side effect — there is nothing to transform or return — and serial isolates listeners from each other so two surface-mutating listeners can never interleave their `session.append`s. Cordis `serial` does bail early if a listener returns a bail value, so `agent/pre-step` listeners are typed/documented to return `void` and must not use that bail channel as a semantic veto surface.
|
||||
|
||||
This **amends** the original RFC's claim of "NO changes to `dsh-agent-loop`; compaction is a pure plugin." That claim was load-bearing for a wrong design — reusing `agent/request` was the mistake. Per the pre-release "foundation over blast radius" stance, adding the correct seam (one event declaration in `dsh-agent`, one awaited emit in the loop) beats preserving a no-change boast that locked in the double-derive.
|
||||
|
||||
### Retention is turn-agnostic; tool-pairing balance is the only structural guard
|
||||
|
||||
Auto-compaction fires before **every** step, not once per turn. This is **load-bearing for runaway-turn survival**: a tool-heavy ReAct turn appends an `assistant/message` + a `tool/result` per step, so the surface grows *within* a turn. A single turn can grow past the window on its own (a "runaway turn") — and the only moment to rescue it before the next model call overflows is the next step's `pre-step` checkpoint. Gating compaction to a turn's first step (or, worse, retaining the whole in-flight turn verbatim) re-opens exactly the hole compaction exists to close: the harness would die when compaction is most needed.
|
||||
|
||||
So retention does **not** protect the in-flight turn, and turn boundaries play no role in it. `compactIfNeeded` walks the surface nodes tail→head, summing per-node token estimates, and retains the smallest tail-run of **whole units** whose total reaches `retainTokens`; everything older is compacted (head-anchored — see below). A *unit* is either a whole closed step (its `assistant/message` plus its `tool/result`s) or a single no-step node (a pre-step `user/message`, inter-step `steering/message`, or injection `context/message`). The walk rounds toward retaining *more*: when the raw token cutoff lands mid-step, it extends the retained side head-ward until the cut before the retained node is **tool-pairing balanced**. The single structural guard is therefore **tool-pairing balance** — a region's edges are balanced cuts on the *surface* (no unanswered `tool-call` crosses either edge), so a compacted region never splits a step's tool-calls from their `tool/result`s (which would produce a transcript every provider rejects). The check is decided over the surface linked list, **not** the log's `step/*` markers: a compaction lands a replacement node at a high log seq whose surface position is the head, so a log-position scan mis-reads its neighbours — `dsh-session` exports `isToolPairingBalanced(nodes, events, beforeSeq)` for the surface-anchored check. `compactRegion` enforces it strictly, throwing on a boundary that would split a step.
|
||||
|
||||
A runaway turn thus compacts exactly like any other history: its early *closed* steps get summarized while its recent steps stay verbatim. When the only compactable content left is an un-splittable open tail step (its tool-calls have no results yet), compaction declines (`null`) and retries once that step closes.
|
||||
|
||||
**Single-unit overflow is out of scope, by design.** If a single retained unit — one closed step, or a large free node such as a pasted `user/message` — *alone* exceeds the budget, compaction cannot help and the next model call may go out over-budget. Bounding an individual unit's size is a separate concern (output truncation), handled elsewhere; compaction makes no promise about it, and the harness without such a mechanism can still break on a single oversized unit. This is named honestly rather than papered over.
|
||||
|
||||
### Head-anchoring: one auto checkpoint, always at the head
|
||||
|
||||
`compactIfNeeded` always anchors the compacted range at the surface **head** (`nodes[0]`). After a first compaction lands a summary node at the head, the *second* compaction's range starts at that summary node and re-summarizes it together with the steps accumulated since — so the surface holds **at most one** auto-generated checkpoint, always at the head, re-consolidated each cycle (the backend's checkpoint-merge prompt makes this a cheap incremental merge — see below). This is *why* `CompactionResult.shadowedRange` is a **surface-position span, not a numeric seq interval**: after a replace lands a fresh high-seq summary node at an older range's position, `start` can be numerically **greater** than `end`. The range is resolved positionally (index into the ordered node list and slice), and `shadowedSeqs` is the authoritative set in surface order. (Manual `compactRegion` may target any aligned mid-range and so *can* leave several checkpoints; the checkpoint framing does not claim everything after it is recent.)
|
||||
|
||||
### Approximate convergence invariant
|
||||
|
||||
`resolveConfig` validates numeric knobs but does NOT reject based on a pretend summary-length invariant. Convergence is dynamic: provider output caps can be spent on hidden or surfaced reasoning tokens, and the model may emit a summary of unpredictable size. `maxTokens` is only the provider-side generation cap for the summarization call; reasoning blocks are stripped before the checkpoint is stored. If a compacted surface is still over threshold, `compactIfNeeded()` re-compacts the head checkpoint up to `compactionRetries` extra times, but each committed summary must be smaller than the content it shadows. The sole residual is the single-unit-overflow case above (a backward-rounded oversized step can push the retained tail over budget) — which is exactly the out-of-scope concern, not a thrash bug.
|
||||
|
||||
### Surface replacement: `compact/*` events are log-only; one `user/message` carries the summary
|
||||
|
||||
Because `SurfaceEventType` is closed, the summary cannot ride on a `compact/*` event. The backend instead appends a **single `user/message`** with `surfaceOp: { op: 'replace', start, end }` whose `content` is the (framed) summary and whose `sourceEventSeqs` covers the shadowed nodes *and* the bookkeeping events. The `compact/*` events are pure log records (lock + provenance). The surface mutation sits **inside** the lock — `compact/end` is the last event appended:
|
||||
|
||||
```
|
||||
compact/start → log-only. Acquires the lock.
|
||||
[summarize older range via the backend]
|
||||
compact/summary → log-only. Provenance: raw summary, range, shadowed seqs, token count.
|
||||
user/message → surfaceOp { op:'replace', start, end }. THE surface mutation (framed summary).
|
||||
deriveMessages() renders it as a user-role message.
|
||||
compact/end → log-only. Releases the lock (carries `error` on a recoverable failure).
|
||||
```
|
||||
|
||||
`deriveMessages()` then yields `[summary_as_user_message, ...retained_nodes]`. Reusing `user/message` is honest rather than a workaround: a summary genuinely *is* user-role context.
|
||||
|
||||
### Checkpoint framing + incremental merge (backend-private)
|
||||
|
||||
The landed `user/message` is not the raw summary: the backend wraps it in a checkpoint preamble (so a resuming model reads it as established background, not a fresh request) and `<compacted-summary>…</compacted-summary>` tags. The tags make a prior checkpoint detectable on the next cycle, and the summarization prompt then instructs the model to *merge it in place* (preserve still-true facts, drop stale) rather than re-summarize verbatim — a cheap incremental merge that needs no extra log/event machinery. The raw, unframed summary stays on the `compact/summary` provenance event. This framing is entirely a **backend HOW decision** — the contract only promises "a single replace `user/message` carries the (possibly framed) summary; the raw summary lives on `compact/summary`." A template or remote backend may frame differently or not at all.
|
||||
|
||||
### Blocking via a log-recorded lock, plus a crash/recoverable failure taxonomy
|
||||
|
||||
The `compact/start … compact/end` bracket is justified, in order of what now does the work:
|
||||
|
||||
1. **Crash-detectable orphan + provenance** (primary). Summarization is a slow model call persisted *after* `compact/start`. A crash mid-summarization leaves a `compact/start` with no matching `compact/end` — a detectable orphan. Releasing the lock last (rather than first) converts the crash window from *silent corruption* into that detectable orphan.
|
||||
2. **Prevents concurrent compaction.** `compactRegion` refuses to start if the current turn holds an unmatched `compact/start`. (The loop is single-threaded across the awaited `pre-step`, so this is also a re-entry tripwire — a thrown "already in progress" signals a real bug.)
|
||||
|
||||
Two failure paths, both documented:
|
||||
|
||||
- **Crash** (the loop dies mid-summarization): a dangling `compact/start`, no closer. Because `compact/*` are **log-only**, the orphan is **inert** — the surface replacement never landed, so the full, uncompacted history derives correctly. Generic turn-repair (`interruptedTurnClosers`) closes the turn with a synthetic `turn/end`; the orphan sits *before* that `turn/end`, so the turn-scoped in-progress check never sees it and a crash can't wedge future compaction. Compaction simply re-attempts at the next `pre-step`.
|
||||
- **Recoverable** (summarization throws but the loop survives): the backend appends `compact/end` with its **`error`** field set, leaving the surface untouched, and the model call proceeds with full history.
|
||||
|
||||
`compact/end` keeps its `error?` field (mirroring `tool/result`'s self-contained error — one event tells success from failure without correlating a sibling). There is no separate `compact/error` event.
|
||||
|
||||
**Core session repair stays compaction-agnostic — deliberately.** `interruptedTurnClosers` is never taught about `compact/*`. Teaching it would force every future `xxx/start … xxx/end` plugin pair to patch a core module — exactly the coupling the capability-seam architecture exists to avoid. Because the log-only orphan is inert, no special repair is needed: generic turn-repair plus the inertness of an un-landed surface mutation is sufficient.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **New packages**: `packages/compact/compact` (interface) and a sibling `compact-basic` (backend) under `packages/compact/`, wired into the root tsconfigs. The consumer tier is deferred.
|
||||
- **New loop seam**: `agent/pre-step` (`@mode serial`) declared in `dsh-agent` and emitted by `dsh-agent-loop` after system assembly and before `step/start`. This is a documented change to the loop — `docs/architecture.md` records it and the generated cordis catalog carries its signature.
|
||||
- **`SessionEventMap`** gains `compact/start` / `compact/summary` / `compact/end` by declaration merging (merge-extensible); `SurfaceEventType` is **not** touched. These are session events, not cordis `Events`, so the event-taxonomy gate needs no entry.
|
||||
- **`dsh-session`** gains the tool-pairing balance predicate (`isToolPairingBalanced`, in `tool-pairing.ts`, exported from the package index) that `compactRegion`/`compactIfNeeded` use to keep a collapsed region from splitting a step's tool-call/result pair. The surface `replace` op and the surface-metadata runtime guard already existed and are reused.
|
||||
- **`dsh-invariants`** drops its `surface replace: start must be <= end` assertion: a head-anchored compaction lands a high-seq replacement node at an older range's *position*, so `start > end` numerically is normal and valid (the range is positional, validated by the surface's `indexOf` checks that remain). The turn-enclosure invariant is reused unchanged.
|
||||
- **Wiring**: `dsh-compact-basic` is loaded in `examples/coding-agent`'s `cordis.yml`, so the seam ships in the real demo (it was previously loaded nowhere).
|
||||
|
||||
## Testing
|
||||
|
||||
- **Unit** (`dsh-compact-basic`): the whole-unit retention walk, the convergence-invariant throw, both failure paths (`compact/end` with/without `error`), head-anchoring producing a non-monotonic `shadowedRange`, decline-on-open-tail, crash-orphan inertness, and the **runaway-turn regression** — a single oversized open turn compacts its early closed steps (proven to fail on the layer-2 protection it replaced). Driven through the real `dsh-invariants` plugin and the real Loader/inject path.
|
||||
- **Loop** (`dsh-agent-loop`): `agent/pre-step` fires once per step, after `turn/start` and before `step/start`, awaited; a surface mutation in a `pre-step` listener lands outside the step and is reflected in the single derived request.
|
||||
- **With-key e2e** (`examples/coding-agent`): a real model + real bash session with a lowered `contextWindow`/`retainTokens` triggers compaction mid-session; the test verifies the WORLD (a `compact/start…end` pair landed, the surface shrank, the agent still completed the task after compaction). This is compaction's first real-world exercise and the runaway-survival net.
|
||||
- **Snapshot (deferred, named gap)**: a full-transcript snapshot of a runaway-turn compaction is NOT yet possible — `dsh-llm-replay` derives one model call per `(turn, step)` from `assistant/chunk` events, but the summarization call records no `assistant/chunk`s and carries no `sessionId` (it binds to the anonymous cursor and claims a non-existent extra script). Covering it needs net-new replay infrastructure (record/replay an interleaved summarization call) and is scheduled as a follow-up rather than discovered mid-build.
|
||||
@@ -20,7 +20,7 @@ Pure generation is correct here because the codebase is disciplined enough that
|
||||
|
||||
Specific choices:
|
||||
|
||||
- **`@mode` tag, cross-checked.** Each harness event's JSDoc carries an explicit `@mode emit|waterfall|parallel` tag; the generator hard-errors on a missing tag. Where the signature shape is conclusive — a trailing `next: () => …` parameter is structurally a waterfall — it asserts the tag agrees and hard-errors on a contradiction. The emit-vs-parallel distinction is not structurally visible (`session/flush` returns `Promise<void> | void` with no `next`), so it is trusted from the tag. The authoring rule lives in [AGENTS.md](../../../../AGENTS.md).
|
||||
- **`@mode` tag, cross-checked.** Each harness event's JSDoc carries an explicit `@mode emit|waterfall|parallel|serial` tag; the generator hard-errors on a missing tag. Where the signature shape is conclusive — a trailing `next: () => …` parameter is structurally a waterfall — it asserts the tag agrees and hard-errors on a contradiction. The emit/parallel/serial distinction is not structurally visible (`session/flush` returns `Promise<void> | void` with no `next`, as does the ordered `agent/pre-step` checkpoint), so it is trusted from the tag. The authoring rule lives in [AGENTS.md](../../../../AGENTS.md).
|
||||
- **Tiered scope.** The harness tier (the 8 `@deepseek-ai/dsh-*` services + their events) is rendered in full from source. The inherited tier (cordis-core `ctx.on/emit/effect/provide/…` + the `internal/*` events + loader/hmr/timer) is pinned vendor source a plugin also sees; it is rendered tersely (name + one-line + source pointer) from a curated table in the generator, NOT walked from the vendor AST — the cordis-core `Context` mixes true ctx members with non-service fields (`root`, `baseUrl`, `logger`), and the vendor surface changes only on a deliberate vendor sync.
|
||||
- **Cross-links to the data-structure catalog.** A type name in a signature (`GenerateOptions`, `StreamChunk`, `ToolDefinition`, …) links to the core-data-structures page that documents it. The map is a small hand-curated const in the generator — NOT `type-equiv.manifest.json`, which documents the `…Map` symbols while signatures reference the derived union names, and lists a few symbols on two pages.
|
||||
- **A dedicated fence.** Signature blocks use a ` ```ts cordis-catalog ` info string that `doc-typecheck` recognizes and skips (a bare signature fragment is not standalone-compilable), excluded from the opt-out ratio — the same treatment `type-equiv` blocks get.
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
# RFC: Generated tool-schema catalog (boot-and-harvest)
|
||||
|
||||
Status: implemented (accepted 2026-07-02)
|
||||
|
||||
## Context
|
||||
|
||||
A reader — a plugin author, a prompt engineer, someone auditing what the agent can do — has no single place that lists the model-facing tools the harness ships. The `name` / `description` / JSON-Schema `parameters` a tool contributes are what the model actually receives (via `ctx.systemPrompt.tools()` off `ctx.tools.schemas()`), but they are scattered across each `defineTool` call in each `packages/*/tool-*` package, buried in string concatenation and runtime spreads. The [cordis events & services catalog](../../../cordis-catalog/events-and-services.md) ([its RFC](2026-06-20-generated-cordis-catalog.md)) documents the *wiring* a plugin works against and the [core-data-structures catalog](../../../core-data-structures/core.md) documents the *vocabulary* those signatures move — but neither documents the *tools* the agent is offered. This RFC adds that third reference surface, `docs/tool-catalog/tools.md`, and a freshness gate so it cannot drift.
|
||||
|
||||
## Decision
|
||||
|
||||
Generate the catalog by **booting each tool plugin and reading its registered schemas**, not by parsing source. `scripts/gen-tool-catalog.ts` mounts each shipped tool package on a fresh cordis `Context` (with `SystemPrompt` + `ToolRegistry` and the injected seams the plugin's `apply` reads), calls `ctx.tools.schemas()` — exactly the `ToolSchema[]` the model is sent — disposes the context, and renders one `## <package>` section per package with a ` ```json ` `parameters` block per tool. It mirrors the `gen-cordis-catalog` / `gen-module-graph` CLI shape: default `--write` regenerates, `--check` fails if the committed copy is stale, output is deterministic (manifest-ordered, tools sorted by name). `verify-tool-catalog` (the `--check`) runs inside `doc-sync`, so the freshness gate fires in the same lefthook pre-push and CI paths as every other doc gate.
|
||||
|
||||
### Why boot, not parse (the crux)
|
||||
|
||||
The cordis catalog is a pure TypeScript-AST pass because every event/service name is a string literal that round-trips to a static declaration — the AST is the whole truth. **Tool schemas are not statically knowable**, so the same technique would produce a doc that lies:
|
||||
|
||||
- `tool-todo` writes `enum: [...STATUSES]` — a spread of a runtime `const`. The AST sees the spread expression, not `["pending","in_progress","completed"]`.
|
||||
- Every description is built by string **concatenation** (`'…' + '…'`). The AST sees concatenation nodes, not the final prose the model reads.
|
||||
- `tool-subagent`'s tool name is `config.toolName ?? 'subagent'` — chosen at load, not a literal.
|
||||
- An MCP plugin can register **raw JSON Schema** directly via `ctx.tools.register()` without `defineTool` at all, so enumerating `defineTool(` call sites structurally under-counts.
|
||||
|
||||
The only faithful source of truth is the schema the registry actually holds after the plugin loads. Booting is the [unit-test discipline](../../../../AGENTS.md) "verify the world, not a synthetic stand-in" applied to a doc generator: read the shipped artifact, not a re-derivation of it.
|
||||
|
||||
### Restoring "nothing silently omitted"
|
||||
|
||||
Booting has a cost the AST pass did not: there is no source declaration set to enumerate, so a new tool package could simply be forgotten. A **completeness guard** restores the guarantee — `assertManifestComplete` globs every `tool-*` package under `packages/` and hard-errors if any is absent from the generator's boot manifest. A new tool package fails the generator, and therefore `doc-sync`, until it is registered. This is the same structural property the cordis generator gets for free from enumerating source, re-created for a boot-based generator.
|
||||
|
||||
### A hand-maintained boot manifest is the irreducible policy
|
||||
|
||||
The boot manifest (`TOOL_PACKAGES`) is a hand-written list — in tension with the proposed [Discover package inventories instead of maintaining static lists](../../proposed/process/2026-06-20-discover-package-inventory.md). The tension is deliberate and resolved as follows: the *inventory* is discovered (the glob guard means no one maintains "the list of tool packages" — the filesystem is the source of truth, and drift fails the gate), but the *boot recipe* per package — which seams to plug (`bash-local` for `ctx.bash`, `subagent` + `subagent-mock` for `ctx.subagents`) and with what config (`{ provider: 'mock' }`) — is genuine policy that no layout fact encodes. Per that RFC's own "what we give up" ("stay boring: read manifests, filter on explicit fields, print the resolved list, and fail loud"), a recipe closure is the boring, explicit form; inferring seam wiring from injects would be the "too clever" path it warns against. So: discovered inventory, hand-written recipe, gate on completeness.
|
||||
|
||||
### Scope
|
||||
|
||||
Shipped product tool PACKAGES under `packages/*/tool-*`, each booted with its default config: `dsh-tool-bash` (`bash`, `bash_output`, `bash_kill`), `dsh-tool-todo` (`todo_write`), `dsh-tool-subagent` (`subagent`). The `examples/` demo tools (`echo`) are excluded, matching the cordis catalog's packages-only scope — a demo tool is not part of the product surface a reader is cataloguing.
|
||||
|
||||
The unit is the PACKAGE, not the deployed tool instance. A package's registered tool name can be a load-time config — `tool-subagent`'s `toolName` — so the same package surfaces as `subagent` (spawn backend) AND `subagent_fork` (fork backend) in the shipped `coding-agent` / `acp-agent` configs, with an identical schema. The generator boots each package once at its default and records such shipped aliases in a per-package note, rather than enumerating every deployment permutation. Cataloguing at the package level keeps the source of truth the package (what a plugin author reads) and avoids leaking example-app `cordis.yml` config into a packages-scoped generator; the note keeps the doc honest about the names a reader will actually see the model receive. The design deliberately does not attempt to catalog "every configured tool instance across every leaf config" — that is a deployment inventory, a different (and unbounded) surface.
|
||||
|
||||
### A plain `json` fence
|
||||
|
||||
Schema blocks use ` ```json `, not a bespoke `ts`-family fence. `doc-typecheck` only extracts `ts*` fences, so a JSON block is invisible to it — no `BlockKind` wiring is needed (unlike the cordis catalog's `ts cordis-catalog` fence, which had to be allowlisted so a bare signature fragment isn't compiled).
|
||||
|
||||
## Consequences
|
||||
|
||||
- The catalog cannot drift: a tool schema change the committed file doesn't reflect fails `verify-tool-catalog` in the pre-push hook and CI. A new `tool-*` package not added to the manifest fails the completeness guard outright.
|
||||
- Tool description prose has a single home — the `defineTool` `description` at the source — and the generated entry is only as good as it, the same forcing function the cordis catalog applies to event JSDoc.
|
||||
- The generator imports and executes workspace packages (the first repo script to do so; the others only read text). It runs under `tsx` via the root `tsconfig` `paths` map, the same unbuilt-source path the demos and tests use, so it needs no build step.
|
||||
- A new capability seam behind a future tool means a new manifest recipe entry (which seams to mount). This is the deliberate hand-written cost called out above; it changes only when a tool package is added.
|
||||
@@ -1,59 +0,0 @@
|
||||
# RFC: Compaction as a capability seam (abstract contract + basic backend)
|
||||
|
||||
Status: proposed (2026-06-18)
|
||||
|
||||
## Context
|
||||
|
||||
A long-running agent conversation grows without bound. As the event log accumulates turns, the derived message history eventually approaches the model's context window — the model then truncates mid-response (`max-tokens`) or degrades. **Compaction** is the mitigation: replace a run of older history with a concise summary, keeping recent context intact.
|
||||
|
||||
The [session surface](../../implemented/architecture/2026-06-18-session-surface.md) was built as the foundation for exactly this — a linked list over the event log with a `surfaceOp: { op: 'replace', start, end }` operation purpose-built to shadow a range of nodes and insert a replacement, with `sourceEventSeqs` recording provenance so the decision replays deterministically. What remained was the plugin that *decides what to compact and produces the summary*.
|
||||
|
||||
Two forces shape the design. First, compaction is **swappable**: token counting can be a char/4 heuristic or a real tokenizer, and summarization can be a model call, a template, or a remote service — these vary independently of *when* and *which range* to compact. Second, a later commit (`ce43c25`) closed `SurfaceEventType` to five event types (`user/message`, `assistant/message`, `tool/result`, `context/message`, `steering/message`); only those may carry `surfaceOp`. A bespoke `compaction/*` event therefore **cannot** itself appear on the surface — the compiler rejects `surfaceOp` on it and the invariants plugin rejects it at runtime.
|
||||
|
||||
## Decision
|
||||
|
||||
### Compaction is a capability seam, split interface / implementation
|
||||
|
||||
Per the [capability-seams RFC](../../implemented/architecture/2026-06-13-capability-seams.md), compaction ships as separate packages so the contract, the algorithm, and (later) the consumer surface evolve independently:
|
||||
|
||||
1. **Interface** — `@deepseek-ai/dsh-compact`: an abstract `CompactService` owning the `ctx.compact` key, the `CompactionResult` vocabulary, and the `compact/*` session events. It declares `compactIfNeeded()` and `compactRegion()` as **abstract** — the contract states *what* compaction does, not *how*.
|
||||
2. **Implementation** — `@deepseek-ai/dsh-compact-basic`: a concrete `BasicCompactService` that owns the entire algorithm — token estimation (char/4 + per-block overhead), the tail→head retention walk, summarization via `ctx.llm.generate()`, the surface replacement, the lock, and the `agent/request` auto-compaction listener. A tokenizer-based or template-based backend is a sibling package (or a subclass overriding the two protected estimation/summarization hooks).
|
||||
3. **Consumer** — deferred. A `/compact` tool and slash command will `inject: ['compact']` and call the contract; they are intentionally out of scope here so the seam settles first.
|
||||
|
||||
### The contract depends on `dsh-session` and `dsh-llm` — a deliberate deviation
|
||||
|
||||
The capability-seams RFC states the interface package "depends only on cordis" (true of `dsh-bash`, whose vocabulary is self-contained). Compaction **cannot** honor that: its verbs are defined *over* a `Session` (`compactRegion(session, start, end)`) and its output *is* the content vocabulary (`CompactionResult.summary: ContentBlock[]`). There is no way to express the contract without naming `Session`/`SessionEvent` (from `dsh-session`) and `ContentBlock` (from `dsh-llm`).
|
||||
|
||||
This is not a coupling smell — it is the contract's domain. The "only cordis" guidance was always shorthand for "the interface depends only on what the contract genuinely names, and never on an implementation." `dsh-session` and `dsh-llm` are themselves interface/vocabulary packages, not implementations; `dsh-compact` still imports no backend. The seam's real invariant — *consumers and implementations evolve independently behind an abstract service* — holds intact. We record the deviation here so a future reader doesn't mistake it for an accident or "fix" it by smuggling `Session` behind an opaque handle.
|
||||
|
||||
### Abstract `compactIfNeeded` / `compactRegion`, algorithm in the backend
|
||||
|
||||
An earlier draft put the full algorithm (the retention walk, token-summing, text extraction) as concrete methods on the interface, with only `estimateContentTokens()` and `summarize()` abstract. That recouples the contract to one strategy: a backend that wants a different retention policy (e.g. turn-count instead of token-budget) or a different event-sequencing would have to fight inherited concrete code. Making both core methods abstract puts every *how* decision in the backend, where it belongs, and keeps the interface a pure statement of *what*. The backend remains internally factored — `estimateContentTokens()` and `summarize()` are `protected` hooks a sub-backend can override without reimplementing the walk — but that factoring is the backend's private concern, not the contract's.
|
||||
|
||||
### Surface replacement: `compact/*` events are log-only; one `user/message` carries the summary
|
||||
|
||||
Because `SurfaceEventType` is closed, the summary cannot ride on a `compact/*` event. The backend instead appends a **single `user/message`** with `surfaceOp: { op: 'replace', start, end }` whose `content` is the summary `ContentBlock[]` and whose `sourceEventSeqs` covers the shadowed nodes *and* the bookkeeping events. The `compact/*` events are pure log records (lock + provenance), never on the surface. The surface mutation sits **inside** the lock — `compact/end` is the last event appended:
|
||||
|
||||
```
|
||||
compact/start → log-only. Acquires the lock.
|
||||
[summarize older range via the backend]
|
||||
compact/summary → log-only. Provenance: summary, range, shadowed seqs, token count.
|
||||
user/message → surfaceOp { op:'replace', start, end }. THE surface mutation.
|
||||
deriveMessages() renders it as a user-role message.
|
||||
compact/end → log-only. Releases the lock.
|
||||
```
|
||||
|
||||
Ordering the surface mutation **before** `compact/end` is deliberate: `session.append()` commits one event at a time, so there is no multi-event transaction to make the sequence atomic. Releasing the lock last converts the crash window from *silent corruption* (a `compact/end` that claims compaction finished while the surface was never shadowed) into a *detectable orphaned lock* (a `compact/start` with no matching `compact/end`), which a persistence backend already detects on reload. A `session/event` listener on `compact/end` likewise never sees the lock free before the replacement has landed.
|
||||
|
||||
`deriveMessages()` then yields `[summary_as_user_message, ...retained_nodes]`. An alternative — extending `SurfaceEventType` to admit a `compact/*` type — was rejected: the closed union is a deliberate safety boundary (only message-producing events reach the model), and a summary genuinely *is* user-role context, so reusing `user/message` is honest rather than a workaround.
|
||||
|
||||
### Blocking via a log-recorded lock, not a mutex
|
||||
|
||||
Compaction must be serialized: no second compaction starts before the first finishes, and no ordinary events interleave the slow summarization. Rather than an in-memory mutex (invisible to replay, lost on crash), the lock **is** the log: `compactRegion` refuses to start if the last `compact/start` has no matching `compact/end` after it. `compact/start` is appended first (fast, synchronous), the slow model call runs, then the `compact/summary` and `user/message` replacement land, and only then is `compact/end` appended — in a `catch` that records the error, so a failed summarization can never wedge the lock. Because the backend runs compaction synchronously inside the `agent/request` waterfall, the loop is single-threaded for that window; the lock additionally gives observability and lets a persistence backend detect an orphaned `compact/start` on reload.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **New packages**: `packages/compact/compact` (interface) and a sibling `compact-basic` (backend) under `packages/compact/`, wired into the three root tsconfigs. The consumer tier is deferred.
|
||||
- **`SessionEventMap`** gains `compact/start` / `compact/summary` / `compact/end` by declaration merging (merge-extensible); `SurfaceEventType` is **not** touched. These are session events, not cordis `Events`, so the event-taxonomy gate needs no entry.
|
||||
- **No changes** to `dsh-session`, `dsh-invariants`, or `dsh-agent-loop`: the surface replace op, the surface-metadata runtime guard, and the `agent/request` waterfall all already exist. Compaction is a pure plugin on documented seams.
|
||||
- The capability-seams convention gains a second reference beyond bash, and a documented case where "interface depends only on cordis" relaxes to "depends only on interface/vocabulary packages the contract genuinely names." On acceptance, [AGENTS.md](../../../../AGENTS.md) § Conventions and [architecture.md](../../../architecture.md) § "Capability seams" should note this relaxation.
|
||||
167
docs/tool-catalog/tools.md
Normal file
167
docs/tool-catalog/tools.md
Normal file
@@ -0,0 +1,167 @@
|
||||
<!-- Generated by scripts/gen-tool-catalog.ts — do not edit by hand.
|
||||
Run `pnpm run gen-tool-catalog` to regenerate. -->
|
||||
|
||||
# Tool Schema Catalog
|
||||
|
||||
Every model-facing tool a shipped plugin contributes to `ctx.tools`: the `name`, `description`, and JSON-Schema `parameters` the model receives via the system-prompt assembly. It complements the [cordis events & services catalog](../cordis-catalog/events-and-services.md) (the wiring a plugin listens to and calls) and [core-data-structures/](../core-data-structures/core.md) (the types those signatures move) — this page is the *tools* the agent is offered.
|
||||
|
||||
This file is GENERATED and verified fresh by `pnpm run verify-tool-catalog` (part of `doc-sync`) — do not edit it by hand. Unlike the cordis catalog (a pure source-AST pass), this generator BOOTS each tool plugin on a real context and reads `ctx.tools.schemas()`, because a tool schema is not statically knowable (runtime-spread enums, concatenated descriptions, config-driven names, raw-JSON-Schema MCP tools). A completeness guard globs `packages/*/tool-*` and fails if any package is missing from the generator's boot manifest, so a new tool cannot be silently undocumented. See [the tool-schema-catalog RFC](../rfc/implemented/process/2026-07-02-tool-schema-catalog.md).
|
||||
|
||||
Scope: shipped product tools under `packages/*/tool-*`, each booted with its DEFAULT config. The registered tool NAME can be a load-time config (e.g. `tool-subagent`'s `toolName`), so a deployment may surface a package under a different or additional name — a per-package note records those shipped aliases where they exist. The `examples/` demo tools (e.g. `echo`) are excluded, matching the cordis catalog's packages-only scope.
|
||||
|
||||
## `@deepseek-ai/dsh-tool-bash`
|
||||
|
||||
### `bash`
|
||||
|
||||
Execute a bash command (`bash -c`) and return its stdout/stderr. Each call runs in a fresh shell: no state (cwd, variables, functions) persists between calls — pass `workdir` instead of using `cd`. Non-zero exits are reported as `[exit code: N]`. Long output is truncated to its tail; the full output is saved to a file whose path is reported when available. Set `run_in_background: true` for long-running commands: the call returns a task id immediately; poll it with `bash_output` and stop it with `bash_kill`.
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"command": {
|
||||
"type": "string",
|
||||
"description": "The bash command to execute."
|
||||
},
|
||||
"description": {
|
||||
"type": "string",
|
||||
"description": "Clear, concise description of what this command does in active voice, 5-10 words (shown in the UI). Examples: \"ls\" → \"List files in current directory\"; \"git status\" → \"Show working tree status\"; \"npm install\" → \"Install package dependencies\"."
|
||||
},
|
||||
"timeoutMs": {
|
||||
"type": "number",
|
||||
"description": "Timeout in milliseconds. The executor applies its configured default and cap, and kills the command on expiry."
|
||||
},
|
||||
"workdir": {
|
||||
"type": "string",
|
||||
"description": "Working directory for this command. Defaults to the session workspace; a relative path is resolved against it."
|
||||
},
|
||||
"run_in_background": {
|
||||
"type": "boolean",
|
||||
"description": "Run in the background and return a task id immediately. No timeout applies."
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"command",
|
||||
"description"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Source: [`packages/bash/tool-bash/src/index.ts`](../../packages/bash/tool-bash/src/index.ts)
|
||||
|
||||
### `bash_kill`
|
||||
|
||||
Ask the executor to kill a running background bash task by task id.
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"task_id": {
|
||||
"type": "string",
|
||||
"description": "Task id returned by the bash tool."
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"task_id"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Source: [`packages/bash/tool-bash/src/index.ts`](../../packages/bash/tool-bash/src/index.ts)
|
||||
|
||||
### `bash_output`
|
||||
|
||||
Read new output from a background bash task started with `bash` + `run_in_background`. Returns only output produced since the previous bash_output call, plus the task status. Tasks keep running while you do other work; poll again later for more output.
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"task_id": {
|
||||
"type": "string",
|
||||
"description": "Task id returned by the bash tool."
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"task_id"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Source: [`packages/bash/tool-bash/src/index.ts`](../../packages/bash/tool-bash/src/index.ts)
|
||||
|
||||
## `@deepseek-ai/dsh-tool-subagent`
|
||||
|
||||
### `subagent`
|
||||
|
||||
Delegate a self-contained task to a subagent (a separate agent that works in its own context) and return its final result. Use this to offload focused, independent work — research, a scoped implementation, an analysis — so it does not consume this conversation's context. The subagent runs to completion and you receive only its final answer, not its intermediate steps. Give it a complete, standalone prompt: it does not see this conversation.
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"description": {
|
||||
"type": "string",
|
||||
"description": "A short (3-5 word) description of the delegated task, for display."
|
||||
},
|
||||
"prompt": {
|
||||
"type": "string",
|
||||
"description": "The complete, self-contained task for the subagent. It does not share this conversation's context, so include everything it needs."
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"description",
|
||||
"prompt"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Source: [`packages/subagent/tool-subagent/src/index.ts`](../../packages/subagent/tool-subagent/src/index.ts)
|
||||
|
||||
The registered tool name is the load-time `toolName` config (default `subagent`); the schema above is that default. The shipped example agents load this package once per subagent backend, so the model additionally sees `subagent_fork` (bound to the fork backend) with an identical schema — see `examples/coding-agent/cordis.yml` and `examples/acp-agent/cordis.yml`.
|
||||
|
||||
## `@deepseek-ai/dsh-tool-todo`
|
||||
|
||||
### `todo_write`
|
||||
|
||||
Record and update a structured task list for the current work. Send the ENTIRE list every call — it REPLACES the previous list (there are no partial updates, no per-item edits). Use it to plan multi-step work and show progress: add one todo per concrete step before you start. Keep AT MOST ONE todo `in_progress` at a time; while work remains, exactly one active task should be `in_progress`. Mark a todo `completed` the moment it is done (do not batch completions), and allow no `in_progress` item only once all work is complete. Skip the list for trivial single-step tasks. Statuses: `pending` (not started), `in_progress` (being worked on now), `completed` (finished).
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"todos": {
|
||||
"type": "array",
|
||||
"description": "The COMPLETE task list, replacing any previous list.",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"content": {
|
||||
"type": "string",
|
||||
"description": "What the task is — a short imperative line."
|
||||
},
|
||||
"status": {
|
||||
"type": "string",
|
||||
"description": "pending (not started) | in_progress (now) | completed (done).",
|
||||
"enum": [
|
||||
"pending",
|
||||
"in_progress",
|
||||
"completed"
|
||||
]
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"content",
|
||||
"status"
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"required": [
|
||||
"todos"
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Source: [`packages/todo/tool-todo/src/index.ts`](../../packages/todo/tool-todo/src/index.ts)
|
||||
Reference in New Issue
Block a user