Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(core-data-structures): catalog the core data structures

Browse Source 
A new docs/core-data-structures/ folder: a self-contained core.md defining what
counts as a "core" data structure (the agent-loop spine) and covering the spine
vocabulary, plus per-seam sub-pages (llm-streaming, session, persistence, tools,
bash). Type definitions are pasted verbatim via `ts type-equiv` blocks and
drift-checked by verify-type-equiv. Cross-linked from architecture.md; the
`ts type-equiv` mechanics are documented in development.md.
This commit is contained in:
Tianyi Cui
2026-06-20 16:24:56 +08:00
parent 07048983e0
commit 0f7abc9808
8 changed files with 797 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

123
docs/core-data-structures/bash.md Normal file
View File

@@ -0,0 +1,123 @@
# Bash Executor
The bash execution seam — the canonical [capability seam](../rfc/implemented/2026-06-13-capability-seams.md) example, split across three packages: interface ([dsh-bash](../../packages/bash), `ctx.bash`), implementation ([dsh-bash-local](../../packages/bash-local), local subprocesses), and consumer ([dsh-tool-bash](../../packages/tool-bash), the `bash`/`bash_output`/`bash_kill` tool schemas). Bash is **one optional capability**, not part of the agent-loop spine — so its vocabulary lives here, not in [core.md](core.md). A sandboxed, containerized, or remote backend is a sibling package implementing the same interface.
Source: [`packages/bash/src/types.ts`](../../packages/bash/src/types.ts)
## Request vs. spec: the `resolve()` split
The seam separates the **model-/plugin-facing request** (optional `workdir`/`timeoutMs`, filled from config) from the **fully-resolved spec** the executor acts on (those fields required). The tool layer calls `ctx.bash.resolve(request)` between them — this is the repo's "explicit > implicit at package seams" rule made concrete: the reader of a `BashExecSpec` never wonders where the working directory came from.
```ts type-equiv
interface BashExecRequest {
command: string
/** Working directory override (default: implementation-configured). */
workdir?: string | undefined
/** Timeout override in milliseconds (implementations cap it). */
timeoutMs?: number | undefined
/** Abort signal — implementations kill the command when it fires. */
signal?: AbortSignal | undefined
/**
* Opaque OWNER token for a background task — the consumer's isolation key
* (the tool layer passes the owning agent's `session.header.id`). The
* executor stores it on the task and exposes it via {@link BashExecutor.ownerOf};
* the executor itself NEVER interprets it (no access policy lives in the
* seam — that is the consumer's job). Absent for foreground runs and for an
* ownerless background start (a non-agent caller).
*/
owner?: string | undefined
}
```
```ts type-equiv
interface BashExecSpec {
command: string
workdir: string
timeoutMs: number
/** Abort signal — implementations kill the command when it fires. */
signal?: AbortSignal | undefined
/**
* Opaque owner token, REQUIRED-but-nullable (mirrors `workdir`/`timeoutMs`
* being required on the resolved spec): {@link BashExecutor.resolve} carries
* the request's `owner` through, defaulting a missing one to `undefined`. A
* required field makes a forgotten owner a VISIBLE `undefined` rather than a
* silently-absent property that yields an unowned (cross-session-readable)
* task. `start()` stores it; `run()` (foreground) ignores it.
*/
owner: string | undefined
}
```
The `owner` token is the isolation key: the executor stores it but never interprets it (access policy is the consumer's job), so a background task started by one agent isn't readable cross-session. A required-but-nullable field makes a forgotten owner a visible `undefined` rather than a silently-unowned task.
## Foreground runs: `BashRunResult`
The outcome of one completed (or killed) foreground run. Orthogonal outcomes are reported **independently** — a process can both time out AND exit 0 because it trapped the signal — so `timedOut`, `aborted`, `signal`, and `exitCode` are each their own field; a caller never reads a cut-short run as a clean success.
```ts type-equiv
interface BashRunResult {
/** Exit code; null when the process died from a signal. */
exitCode: number | null
/** Terminating signal (e.g. 'SIGTERM'); null on normal exit. */
signal: NodeJS.Signals | null
/** True when the executor's own timeout killed the command. */
timedOut: boolean
/** True when the caller's AbortSignal killed the command. */
aborted: boolean
/** The effective timeout applied to this run (after defaulting/capping). */
timeoutMs: number
stdout: CollectedOutput
stderr: CollectedOutput
}
```
Each stream is a `CollectedOutput` — the (possibly truncated) text plus recovery info. When truncated, `text` is the **tail** and the complete stream spills to a private file:
```ts type-equiv
interface CollectedOutput {
/** Collected text — the TAIL of the stream when truncated. */
text: string
/** True when bytes were dropped from `text`. */
truncated: boolean
/** Path to a file holding the COMPLETE stream, when truncated and available. */
spillPath?: string
}
```
## Background tasks: `BashTask`
A long-running command started with `start()` is tracked as a `BashTask`. `BashTaskStatus` is `'running' | 'completed' | 'killed'`; `done` resolves when the underlying process closes and never rejects.
```ts type-equiv
interface BashTask {
readonly id: string
readonly command: string
status: BashTaskStatus
/** Exit code once finished (null = killed by signal / still running). */
exitCode: number | null
/** Terminating signal name, when signal-killed. */
signal: NodeJS.Signals | null
/** Resolves when the underlying process closes (never rejects). */
readonly done: Promise<void>
}
```
`readOutput()` returns an incremental `BashTaskRead` — the output produced since the previous read, with a `lossy` flag when truncation dropped unread bytes:
```ts type-equiv
interface BashTaskRead {
task: BashTask
/** Output produced since the previous read (stderr in a marked section). */
delta: string
/** True when truncation dropped unread bytes the delta cannot include. */
lossy: boolean
/** Full stdout spill file, when stdout truncation occurred and a safe path is available. */
stdoutSpillPath?: string
/** Full stderr spill file, when stderr truncation occurred and a safe path is available. */
stderrSpillPath?: string
}
```
## The service
`BashExecutor` (`ctx.bash`, abstract — defined in [`packages/bash/src/index.ts`](../../packages/bash/src/index.ts)) mirrors the `LlmService`/`LlmAdapter` split: `resolve` (request → spec), `run` (foreground), `start` (background), `get`/`ownerOf`/`list`/`readOutput`/`kill`, and `onTaskDone` (a `BashTaskListener` completion callback). Spawned commands get a **scrubbed env** (dropping `*KEY*`/`*SECRET*`/`*TOKEN*`) and spill files use a private 0700 dir with random names and owner-only opens — model output never gets the ambient environment or a predictable path. The implementation that provides all this is `dsh-bash-local`; the model-facing `bash`/`bash_output`/`bash_kill` schemas that call it are in `dsh-tool-bash` (and present as terminals via the [tool-presentation vocabulary](tools.md#tool-presentation-ui-vocabulary)).

300
docs/core-data-structures/core.md Normal file
View File

@@ -0,0 +1,300 @@
# Core Data Structures
This folder catalogs the **data structures** of the DeepSeek Harness — what each core type represents, its literal shape, and where the full detail lives. It complements [architecture.md](../architecture.md), which describes *behavior* (the service map, the session/turn/step lifecycle, the event taxonomy); this page describes the *vocabulary* that behavior moves around.
## What counts as "core"
The harness is a microkernel: a tiny core plus many plugins. Most types belong to one plugin or one capability. A handful, though, are the **spine** — the language the agent loop and its events traffic in on *every* turn, no matter which optional plugins are loaded. Those are "core".
Precisely, a data structure is **core** if either:
1. it flows through the agent-loop spine — the loop holds it, derives it, streams it, or logs it on every turn (a `Message`, a `StreamChunk`, a `SessionEvent`, the `Agent` handle itself), independent of which plugins are present; **or**
2. it is the single headline type a plugin author writes against a pipeline — `ToolDefinition` (what every tool *is*).
Everything else is documented on a **sub-page**, not here. The rule that draws the line: *the type you write, hold, or receive is core; the machinery that types it, renders it, or persists it is a sub-page detail.* So `ToolDefinition` is core, but the `SchemaSpec`/`InferArgs` DSL that types it, the `ToolCallPresentation` vocabulary that renders it, and the `SessionPersistence` seam that stores the event log are not — they live on the sub-pages below.
| Sub-page | Owns |
|---|---|
| [llm-streaming.md](llm-streaming.md) | the `StreamChunk` wire protocol + adapter contract, `BlockAssembler`, the `LlmAdapter` seam |
| [session.md](session.md) | the full `SessionEventMap` variant catalog, `TurnTrigger`/`TurnEndReason`, `deriveMessages()`, the turn-enclosure invariant |
| [persistence.md](persistence.md) | the durability seam: `SessionPersistence`, JSONL + SQLite backends, `session/flush`, crash recovery, `SessionHeader` |
| [tools.md](tools.md) | `ToolDefinition` full fields, the schema DSL, `ToolExecution`/`ToolResult`, tool-presentation UI types, the `tools/execute` waterfall |
| [bash.md](bash.md) | the bash executor seam: `BashExecRequest`/`Spec`, `BashRunResult`, background `BashTask`s |
> Type definitions on this page are pasted **verbatim** from source and drift-checked by `pnpm run verify-type-equiv` (see [development.md](../development.md#documenting-types-verbatim-ts-type-equiv)). Inline JSDoc is omitted for readability; follow the source link for the full contracts.
## The `…Map → derived-union` pattern
Almost every extensible sum type in the harness follows one shape: an interface keyed by a discriminant tag (the `…Map`), from which the union is derived with `keyof`. Plugins add variants by **declaration merging** — no edit to the owning package.
```ts ignore-check
// The pattern, schematically:
interface ThingMap {
'a': { kind: 'a'; /* … */ }
'b': { kind: 'b'; /* … */ }
}
type ThingKind = keyof ThingMap // 'a' | 'b'
type Thing = ThingMap[keyof ThingMap] // the discriminated union
// A plugin extends it without touching the source package:
declare module '@deepseek-ai/dsh-llm' {
interface ThingMap {
'c': { kind: 'c'; /* … */ }
}
}
```
Six canonical maps use this pattern; a plugin author extends these:
| Map | Package | Derives | Catalog |
|---|---|---|---|
| `ContentBlockMap` | dsh-llm | `ContentBlock` | [below](#content-blocks-and-messages) |
| `MessageSourceMap` | dsh-llm | `MessageSource` | [below](#content-blocks-and-messages) |
| `FinishReasonMap` | dsh-llm | `FinishReason` | [below](#the-model-request-and-result) |
| `TurnTriggerMap` | dsh-session | `TurnTrigger` | [session.md](session.md) |
| `TurnEndReasonMap` | dsh-session | `TurnEndReason` | [session.md](session.md) |
| `SessionEventMap` | dsh-session | `SessionEvent` | [session.md](session.md) |
Two large discriminated unions are the ones consumers `switch` over most: **`StreamChunk`** (the streaming protocol) and **`SessionEvent`** (the log entry). Per the repo convention, `switch` on the tag — don't chain `if`s — so each arm narrows and a typo'd tag fails to compile.
## Branded IDs
IDs that cross package boundaries are **branded** — structurally strings, but non-interchangeable at the type level (an `AgentId` can't be passed where a `CallId` is expected). Construction goes through a per-type factory; comparison, logging, and JSON behave as ordinary strings.
Source: [`packages/llm/src/brand.ts`](../../packages/llm/src/brand.ts)
```ts type-equiv
type Branded<B extends string> = string & { readonly [BRAND]: B }
```
The three core IDs: `CallId` (correlates a tool call with its result; dsh-llm), `SessionId` (dsh-session), `AgentId` (dsh-agent). Each is `Branded<'CallId'>` etc. plus a same-named factory function.
## Content blocks and messages
A conversation is `Message`s; a message is an array of typed **content blocks**. The block union derives from `ContentBlockMap`.
Source: [`packages/llm/src/types.ts`](../../packages/llm/src/types.ts)
```ts type-equiv
interface ContentBlockMap {
'text': TextBlock
'reasoning': ReasoningBlock
'tool-call': ToolCallBlock
'tool-result': ToolResultBlock
'image': ImageBlock
}
```
The block interfaces (full fields in source): `TextBlock` (`text`), `ReasoningBlock` (thinking, distinct from visible text), `ToolCallBlock` (`id: CallId`, `name`, raw-JSON `arguments`), `ToolResultBlock` (`toolCallId`, nested `content: ContentBlock[]`, `isError?`), `ImageBlock` (`url`, `mimeType?`). `ContentBlock = ContentBlockMap[ContentBlockType]`.
A `Message` is a role plus blocks:
```ts type-equiv
interface Message {
role: 'system' | 'user' | 'assistant'
content: ContentBlock[]
}
```
Where a message came from is itself a merge-extensible sum type:
```ts type-equiv
interface MessageSourceMap {
user: { kind: 'user' }
plugin: { kind: 'plugin'; plugin: string }
agent: { kind: 'agent'; agentId: string }
}
```
## Streaming
Adapters emit a raw **chunk** protocol; the loop logs the chunks (replay fidelity) while feeding the same chunks through a `BlockAssembler` to rebuild blocks and messages. `StreamChunk` is a closed discriminated union over `type` — `block-start`, `text-delta`, `reasoning-delta`, `tool-call-delta`, `block-end`, `usage`, `finish`.
The full union, the adapter contract (usage-before-finish, raw-JSON tool arguments, the two sanctioned error paths), and `BlockAssembler` live on **[llm-streaming.md](llm-streaming.md)**.
## The model request and result
One model call is a fully-assembled `GenerateOptions`; the non-streaming result is `GenerateResult`.
Source: [`packages/llm/src/types.ts`](../../packages/llm/src/types.ts)
```ts type-equiv
interface GenerateOptions {
model: string
messages: Message[]
/** System prompt text (adapters map to the provider's system slot). */
system?: string
/** Tool schemas (adapters map to the provider's `tools` field). */
tools?: ToolSchema[]
/** Assistant prefix continuation (prefill). */
prefill?: ContentBlock[]
temperature?: number
maxTokens?: number
/**
* Stop sequences: generation halts as soon as the model produces any one of
* these strings (adapters map to the provider's stop field, e.g. OpenAI
* `stop`). The stop string itself is not included in the output.
*/
stop?: string[]
signal?: AbortSignal
}
```
```ts type-equiv
interface GenerateResult {
message: Message
usage?: TokenUsage
finish: FinishReason
}
```
Why a model response stopped is a merge-extensible reason:
```ts type-equiv
interface FinishReasonMap {
'stop': { kind: 'stop' }
'tool-calls': { kind: 'tool-calls' }
'max-tokens': { kind: 'max-tokens' }
'aborted': { kind: 'aborted' }
'error': { kind: 'error'; message: string; code?: string }
}
```
`FinishReason = FinishReasonMap[keyof FinishReasonMap]`. `TokenUsage` (per-call accounting with disjoint cache fields) is detailed on [llm-streaming.md](llm-streaming.md).
`GenerateOptions.tools` carries `ToolSchema` — the JSON-schema description of a tool, as sent to the model. It is declared in dsh-llm (not dsh-tools) precisely because it is part of the request the loop assembles every step:
```ts type-equiv
interface ToolSchema {
name: string
description: string
/** JSON Schema object for the arguments. */
parameters: Record<string, unknown>
strict?: boolean
}
```
The model-facing `ToolSchema` is the wire shape; the registered `ToolDefinition` that produces it (schema + `execute`) is on [tools.md](tools.md).
## Sessions
A `Session` is an **append-only log** of typed `SessionEvent`s — the single source of truth. The LLM message history is *derived* from the log (`deriveMessages()`), not stored separately. The event vocabulary derives from `SessionEventMap`:
Source: [`packages/session/src/types.ts`](../../packages/session/src/types.ts)
```ts type-equiv
type SessionEvent<T extends SessionEventType = SessionEventType> = {
[K in SessionEventType]: {
type: K
/** Monotonic sequence number within the session. */
seq: number
/** Unix epoch milliseconds. */
time: number
data: SessionEventMap[K]
}
}[T]
```
The thirteen event variants (`turn/start`, `turn/end`, `step/start`, `step/end`, `user/message`, `context/message`, `assistant/chunk`, `assistant/message`, `tool/call`, `tool/result`, `steering/message`, `usage`, `error`), the `deriveMessages()` projection rules, the `TurnTrigger`/`TurnEndReason` reasons, and the turn-enclosure invariant are on **[session.md](session.md)**. How the log is made durable — the `SessionPersistence` seam, JSONL/SQLite backends, the `session/flush` checkpoint, crash recovery, and `SessionHeader` — is on **[persistence.md](persistence.md)**.
## The agent handle
`Agent` is the surface every plugin (UI, hooks, orchestrators) programs against. The concrete implementation is `ReactLoopAgent` in dsh-agent-loop; nothing outside the loop depends on the implementation.
Source: [`packages/agent/src/types.ts`](../../packages/agent/src/types.ts)
```ts type-equiv
interface Agent {
readonly id: AgentId
readonly options: AgentOptions
readonly session: Session
readonly status: AgentStatus
/** Queue a user message. Starts a turn when idle; otherwise waits for the next turn. */
send(content: ContentBlock[], options?: SendOptions): void
/**
* Steer a running turn: content is injected between steps of the current
* turn. When idle, behaves like {@link send}.
*/
steer(content: ContentBlock[], options?: SendOptions): void
/**
* Inject in-session context (file-change notices, skill content, cron
* notifications, …): appends a `context/message` session event the next model
* request sees at its chronological position, rendered as tagged synthetic
* context rather than a user prompt. Does not run the model.
*
* Turn-enclosure (the turn-enclosure RFC): an inject while a turn is open joins that turn;
* an inject while idle wraps its `context/message` in a one-shot `injection`
* turn (`turn/start` → `context/message` → `turn/end`) and checkpoints it for
* durability, so every event stays inside a turn and a persistence backend
* never loses a between-turn notice. The idle checkpoint is fire-and-forget
* (inject is synchronous): a failing flush is reported via `agent/error`
* (step `0`) and the logger, never thrown into the caller.
*
* Live-adapter review has validated the tagged-envelope rendering against
* current DeepSeek behavior; provider-specific mismatches belong in that
* adapter, not in the canonical session vocabulary.
*/
inject(content: ContentBlock[], options?: SendOptions): void
/** Abort the in-flight step (if any); the turn ends with reason 'aborted'. */
abort(reason?: string): void
/**
* Cancel ALL pending work for the agent — the narrower {@link abort} kills
* only the in-flight step. `cancel()`:
*
* - clears the queued FIFO (un-started prompts never run) and the steering
* FIFO (steering for the cancelled turn is dropped, not re-enqueued);
* - aborts the in-flight step if one is running (the turn ends `aborted`);
* - drops a turn that is about to start (a `cancel()` landing in the
* pre-step window — after a `send()` queued but before the loop flips to
* `running`, or after `running` is emitted but before the first step) so
* that queued prompt does not run and cannot be batched into the cancelled
* turn.
*
* After `cancel()`, `whenIdle()` resolves on the post-cancel quiescent state.
* `cancel()` on an idle agent with nothing queued or running is a safe no-op
* — it does NOT arm anything that would drop a later legitimate prompt.
*/
cancel(reason?: string): void
/**
* Resolve once the agent has reached quiescence after settling out of
* `running`, or immediately if it is already idle with no queued work. The
* quiescence signal a teardown awaits: `agent.abort()` then
* `await agent.whenIdle()` guarantees queued/running work has fully stopped
* before the caller proceeds (a closing ACP connection, a disposing UI
* plugin), rather than returning while the driver is still streaming or about
* to start a queued turn.
*
* "Quiescence", not merely "status changed": a disposed agent emits
* `agent/status('disposed')` from inside its disposer, BEFORE the driver loop
* has unwound — so `whenIdle()` resolving on `disposed` must wait for the loop
* to actually exit (the implementation chains the loop-exit promise), not just
* observe the status flip. A mid-step disposal that never reaches `idle` still
* unblocks the await this way.
*
* Distinct from disposal: `whenIdle()` observes the transition WITHOUT tearing
* the agent down. A consumer that owns the agent's lifecycle disposes it
* separately.
*/
whenIdle(): Promise<void>
// TODO(sub-agents): spawn/fork seams — semantics deliberately deferred.
// The intended shape: a creation option referencing a parent agent
// (fork = seed the child Session with the parent's event log; spawn =
// fresh Session), with the child returned as an Agent handle so steer()
// and event subscription work uniformly. See docs/architecture.md.
}
```
`AgentStatus` is `'idle' | 'running' | 'disposed'`. `AgentId` is a branded string. `AgentOptions` (`model?`, `systemPrompt?`) is merge-extensible — plugins add creation options by declaration merging. The `agent/*` event taxonomy (lifecycle, turn/step boundaries, the `agent/request`/`agent/step-result`/`agent/turn-continuation` waterfalls) is in [architecture.md § Event taxonomy](../architecture.md#event-taxonomy).
## `ToolDefinition`
The one pipeline-authoring type that is core: what every registered tool *is* — a model-facing `ToolSchema` plus an `execute` function and optional UI presenters. A tool author rarely constructs it by hand (the `defineTool` DSL builds it with typed args), but it is the contract the registry holds and the loop dispatches through.
Its full fields, the `defineTool`/`SchemaSpec`/`InferArgs` typed schema DSL, the `ToolExecution`/`ToolExecutionResult` waterfall shapes, and the tool-presentation UI vocabulary are on **[tools.md](tools.md)**.

66
docs/core-data-structures/llm-streaming.md Normal file
View File

@@ -0,0 +1,66 @@
# LLM Streaming
The wire-level streaming vocabulary of [dsh-llm](../../packages/llm). [core.md](core.md) introduces `StreamChunk`, `Message`, and `ContentBlock`; this page owns the full chunk protocol, the adapter contract every adapter must obey, and the shared assembler.
Source: [`packages/llm/src/types.ts`](../../packages/llm/src/types.ts)
## `StreamChunk` — the raw protocol
A streaming response interleaves several typed blocks (text, reasoning, multiple tool calls). `index` ties each delta to its block; `block-end` carries the fully-assembled `ContentBlock` so consumers don't have to re-assemble deltas themselves. It is a **closed** discriminated union — a `switch` over `type` ends with `assertNever`, so adding a variant breaks compilation at every consumer that must handle it.
```ts type-equiv
type StreamChunk =
| { type: 'block-start'; index: number; blockType: ContentBlockType }
| { type: 'text-delta'; index: number; text: string }
| { type: 'reasoning-delta'; index: number; text: string }
| { type: 'tool-call-delta'; index: number; id: CallId; name?: string; argumentsDelta: string }
| { type: 'block-end'; index: number; block: ContentBlock }
| { type: 'usage'; usage: TokenUsage }
| { type: 'finish'; reason: FinishReason }
```
## The adapter contract
Every adapter MUST obey these, and every consumer may rely on them:
- **`usage` before `finish`, nothing after `finish`.** Defer both to the provider's end-of-stream marker so a trailing usage-only chunk can't violate the ordering.
- **Tool-call `arguments` stay raw JSON strings end-to-end.** Partial fragments stream via `argumentsDelta`; a provider that hands back parsed objects re-stringifies at `block-end`.
- **Two sanctioned error paths.** A failure may either THROW from `stream()` (transport/protocol errors) **or** end the stream with `finish {kind:'error'|'aborted'}` (provider in-band errors, for adapters that can't throw mid-stream). Consumers must handle *both*. The agent loop translates a finish-error/aborted into a turn error — it never logs a normal completed assistant message for a failed step.
This contract is why two adapters exist as a deliberate pair: `dsh-llm-deepseek` (hand-rolled fetch/SSE) and `dsh-llm-pi-ai` (the same endpoint through `@earendil-works/pi-ai`). Two independent internals over one contract is what pinned the protocol down — the library-backed adapter can't throw mid-stream, so it exercises the finish-chunk error path the hand-rolled one might not.
## `TokenUsage`
Per-call token accounting. Counts are **disjoint**: `inputTokens` is uncached input only; cached input is reported separately, and billed input is the sum of the three. Adapters whose providers fold cache hits into a single prompt total (DeepSeek's `prompt_tokens`) subtract them back out.
```ts type-equiv
interface TokenUsage {
inputTokens: number
outputTokens: number
cacheReadTokens?: number
cacheWriteTokens?: number
reasoningTokens?: number
}
```
## `BlockAssembler`
`BlockAssembler` ([`packages/llm/src/assembler.ts`](../../packages/llm/src/assembler.ts)) is the single shared implementation that folds a `StreamChunk` stream back into `ContentBlock`s and a final `Message`. The loop logs the raw chunks (for replay fidelity) while feeding the same chunks through an assembler — so the canonical log keeps token-level detail and the derived message is rebuilt deterministically. A consumer that needs the assembled result without re-implementing the fold uses this.
## The seam
`LlmAdapter` is the provider seam: subclass, implement `stream()`, register with `ctx.llm.registerAdapter(models, adapter)`. The `block-start` / `block-end` `index` correlation and the assembler together mean an adapter only has to emit well-formed chunks — block reassembly is not each adapter's problem. The consumer surface (`ctx.llm.stream()` / `streamBlocks()` / `generate()`) and the `llm/stream` waterfall are described in [architecture.md § The vocabulary](../architecture.md#the-vocabulary-dsh-llm).
`ContentBlockType` (the key set the `index`-correlated blocks carry) derives from `ContentBlockMap`:
```ts type-equiv
interface ContentBlockMap {
'text': TextBlock
'reasoning': ReasoningBlock
'tool-call': ToolCallBlock
'tool-result': ToolResultBlock
'image': ImageBlock
}
```
See [core.md § Content blocks and messages](core.md#content-blocks-and-messages) for the block interfaces.

63
docs/core-data-structures/persistence.md Normal file
View File

@@ -0,0 +1,63 @@
# Session Persistence
The **durability seam** for the event log. [session.md](session.md) describes the in-memory `Session` — the append-only `SessionEvent` log that is the source of truth. This page describes how that log is made durable: the abstract `SessionPersistence` service, its backends, the flush checkpoint, crash recovery, and the metadata header that travels alongside the log.
The seam is a textbook [capability seam](../rfc/implemented/2026-06-13-capability-seams.md): one abstract service ([dsh-session-persistence](../../packages/session-persistence), `ctx.sessionPersistence`) defining create/append/load/list over the existing `SessionEvent` — **no parallel persisted type** — and two interchangeable backends that pass the same `runPersistenceContract` suite. See the [session-persistence RFC](../rfc/implemented/2026-06-14-session-persistence.md).
## The flush checkpoint
`session/event` is a *synchronous* notification; persistence plugins buffer it (write-behind) and drain at the awaited `session/flush` checkpoint the loop fires at every turn end. Flush is `ctx.parallel` (awaited): a turn's events are durably committed before the next turn starts, and the turn boundary is the commit boundary. A rejecting flush is reported via `agent/error` and the logger — never as a session event (it would land past the commit boundary), so the backend keeps its buffered events for the next flush.
## Crash recovery preserves an interrupted turn
A backend that reloads a log crashed mid-turn finds an open `turn/start` with no `turn/end`. It does **not** truncate — a single turn can be huge in a long-horizon task (many steps, large tool output), and those events were durably appended before the crash. Instead it closes the orphaned turn with a synthetic `turn/end { reason: { kind: 'interrupted' } }`, keeping the log balanced and the turn-enclosure invariant intact. `interrupted` is the one `TurnEndReason` no loop emits (see [session.md](session.md#why-a-turn-ended-turnendreasonmap)).
## `SessionHeader` — metadata beside the log
Per-session metadata travels **separately** from the event log: format version, cwd, and lineage are storage concerns, not conversation events, so they stay out of `SessionEventMap` and never reach `deriveMessages()`. The header is attached to a `Session` via `session.header`.
Source: [`packages/session/src/types.ts`](../../packages/session/src/types.ts)
```ts type-equiv
interface SessionHeader {
/** On-disk format version; a persistence backend rejects unknown versions. */
version: number
/** The session's id (mirrors the {@link Session}'s id). */
id: SessionId
/** Unix epoch milliseconds when the session was created. */
createdAt: number
/** Absolute working directory the session was created in (if any). */
cwd?: string
/** The session this one was forked from (seed lineage), if any. */
parentSession?: SessionId
}
```
## `CreateSessionOptions` — seeding and metadata
Creating a `Session` through the store takes a `seed` (replay/fork an existing event log) and `meta` (the storage-level fields the store folds into a `SessionHeader`). The store fills in `version`/`id` and defaults `createdAt`; the caller supplies the validated absolute `cwd`, the `parentSession` lineage, and — only when reconstructing a persisted session — the original `createdAt` to preserve it.
```ts type-equiv
interface CreateSessionOptions {
/** Events to seed the new session with (replay/fork). */
seed?: SessionEvent[]
/**
* Creation metadata. The store fills in `version`/`id` and defaults
* `createdAt` to now; the caller supplies the storage-level fields (validated
* absolute `cwd`, `parentSession` lineage, and — when reconstructing a
* persisted session — the original `createdAt` to preserve it).
*/
meta?: { cwd?: string; parentSession?: SessionId; createdAt?: number }
}
```
Replay/fork is therefore `ctx.sessions.create(id, { seed: seedEvents })`; resuming a *persisted* session into a live agent is `ctx.agents.resume({ resumeSessionId })`.
## The backends
Both implement the same abstract `SessionPersistence` (create/append/load/list over `SessionEvent`) and pass `runPersistenceContract`, proving the seam is genuinely backend-agnostic:
- **[dsh-session-persistence-jsonl](../../packages/session-persistence-jsonl)** — an append-only JSONL log per session with crash-safe atomic writes, the interrupted-turn crash recovery above, and a read/replay path.
- **[dsh-session-persistence-sqlite](../../packages/session-persistence-sqlite)** — `node:sqlite`, one row per `SessionEvent`. The row shape `(session_id, seq, type, time, data)` maps 1:1 onto the event, so there is no parallel persisted schema to keep in sync.
Multiple backends sharing one on-disk session coordinate writes through the [shared persistence write-coordinator](../rfc/implemented/2026-06-18-shared-persistence-write-coordinator.md).

119
docs/core-data-structures/session.md Normal file
View File

@@ -0,0 +1,119 @@
# Sessions
The in-memory, event-sourced model of [dsh-session](../../packages/session). A `Session` is an **append-only log** of typed `SessionEvent`s — the single source of truth for an agent's whole interaction history. The LLM message history is *derived* from the log, never stored separately; replay is re-derivation from the same events. How the log is made **durable** (the persistence seam, backends, crash recovery) is the sibling concern on [persistence.md](persistence.md).
Source: [`packages/session/src/types.ts`](../../packages/session/src/types.ts)
## `SessionEventMap` — the event vocabulary
The append-only event types. Merge-extensible: a plugin (e.g. compaction) declares extra event types via declaration merging.
```ts type-equiv
interface SessionEventMap {
'turn/start': { turn: number; trigger: TurnTrigger }
'turn/end': { turn: number; reason: TurnEndReason }
'step/start': { turn: number; step: number }
'step/end': { turn: number; step: number }
/** A user-visible prompt (queued message drained at turn start). */
'user/message': { content: ContentBlock[]; source: MessageSource }
/**
* In-session context injection (file-change notices, subdir AGENTS.md,
* skill content, cron notifications, …). Rendered into the derived history
* as tagged synthetic context — NOT a user prompt.
*/
'context/message': { content: ContentBlock[]; source: MessageSource }
/** Raw stream chunk — token-level replay fidelity. */
'assistant/chunk': { turn: number; step: number; chunk: StreamChunk }
/** Assembled assistant message for one step (derived history uses this). */
'assistant/message': { turn: number; step: number; content: ContentBlock[] }
'tool/call': { turn: number; step: number; callId: CallId; name: string; arguments: string }
'tool/result': { turn: number; step: number; callId: CallId; content: ContentBlock[]; isError: boolean; error?: { name: string; code: string } }
/** Steering content injected between steps of a running turn. */
'steering/message': { turn: number; content: ContentBlock[]; source: MessageSource }
'usage': { turn: number; step: number; usage: TokenUsage }
'error': { turn: number; step: number; message: string; code?: string }
}
```
## `SessionEvent<T>` — one log entry
A proper discriminated union over `type` (not independent `type`/`data` unions), so `switch (event.type)` narrows `event.data` without casts. `seq` is the monotonic position in the log (`seq = log.length`); `time` is epoch ms.
```ts type-equiv
type SessionEvent<T extends SessionEventType = SessionEventType> = {
[K in SessionEventType]: {
type: K
/** Monotonic sequence number within the session. */
seq: number
/** Unix epoch milliseconds. */
time: number
data: SessionEventMap[K]
}
}[T]
```
`SessionEventType = keyof SessionEventMap`. Because `SessionEventMap` is merge-extensible, switches over `SessionEvent` must NOT use `assertNever` — a plugin-added variant is a valid unknown value; handle the known cases and fall through `default`.
## Derived history: `deriveMessages()`
`Session.deriveMessages()` projects the event log into the `Message[]` the model sees. The projection rules:
- `user/message` → a user message.
- `assistant/message` → an assistant message. Raw `assistant/chunk` events are replay/UI data and are **skipped** in derivation (the assembled message is authoritative).
- `tool/result` → a user message carrying a `tool-result` block.
- `context/message`, `steering/message` → user-role messages wrapped in a tagged envelope (`<context source="…">…</context>`) at their chronological position — the "system-reminder" pattern; the model distinguishes them from real prompts by the envelope.
Everything else (`turn/*`, `step/*`, `usage`, `error`) is structural/telemetry and does not project into a message.
## What started a turn: `TurnTriggerMap`
```ts type-equiv
interface TurnTriggerMap {
message: { kind: 'message'; source: MessageSource }
continuation: { kind: 'continuation' }
/**
* An out-of-band context injection (`agent.inject()`) made while the agent
* was idle. The loop wraps the injected `context/message` in a one-shot turn
* (`turn/start` → `context/message` → `turn/end`) so every event in the log
* stays turn-enclosed — the durability/replay boundary is the turn, and a
* bare event between turns would otherwise be indistinguishable from a crash
* tail on reload.
*/
injection: { kind: 'injection'; source: MessageSource }
}
```
## Why a turn ended: `TurnEndReasonMap`
```ts type-equiv
interface TurnEndReasonMap {
completed: { kind: 'completed' }
aborted: { kind: 'aborted'; reason?: string }
error: { kind: 'error'; message: string; code?: string }
disposed: { kind: 'disposed' }
'max-tokens': { kind: 'max-tokens' }
/**
* The turn never ended on its own: the process crashed mid-turn and a
* persistence backend later closed the orphaned (open) turn on reload so the
* log stays balanced. SYNTHESIZED by the backend's crash-recovery repair — no
* loop ever emits this. Its events are real (they were durably appended before
* the crash) and are PRESERVED, not discarded: a single turn can be huge in a
* long-horizon task (many steps, large tool output), so truncating it would
* lose real work. The marker records that the turn was cut short, not that the
* model completed it. See the session-persistence RFC.
*/
interrupted: { kind: 'interrupted' }
}
```
`max-tokens` mirrors the model-call `FinishReason` of the same name: any `max-tokens` step in a turn makes the whole turn end `max-tokens` (the cut-short fact wins over a later continuation), so a consumer can tell a clean stop from a truncated one. `interrupted` is the one reason no loop emits — it is synthesized by crash recovery (see [persistence.md](persistence.md)). Both maps are merge-extensible.
## The turn-enclosure invariant
Every session event lives **inside** a turn (between a `turn/start` and its `turn/end`). The loop appends queued `user/message` events *after* `turn/start`, and an idle `agent.inject()` wraps its `context/message` in a one-shot `injection` turn. This makes the turn the single durability/replay boundary: a backend can treat anything after the last `turn/end` as an interrupted-crash tail without risking the loss of legitimately-recorded between-turn context. The `dsh-invariants` plugin enforces it in dev (a message event outside an open turn throws). See [the turn-enclosure invariant RFC](../rfc/implemented/2026-06-15-turn-enclosure-invariant.md).
## Durability contract
What a persistence backend relies on: the durable log persists every event verbatim, **including** `assistant/chunk` — `seq` must stay contiguous, so chunks cannot be filtered out of the canonical log. All `event.data` must be JSON-serializable; `Session.append` enforces this at the source (throwing on non-serializable data), so a bad event never enters the log and `session.events` always equals what a backend can persist. Adding an event type that carries non-serializable data, or that breaks the turn/step nesting the invariants plugin checks, is a breaking change to the on-disk format.
The backends that consume this contract are on [persistence.md](persistence.md).

112
docs/core-data-structures/tools.md Normal file
View File

@@ -0,0 +1,112 @@
# Tools
The tool pipeline of [dsh-tools](../../packages/tools). [core.md](core.md) introduces `ToolDefinition` as the one pipeline-authoring type promoted to the spine and `ToolSchema` as the model-facing wire shape. This page owns the full `ToolDefinition`, the typed schema DSL that builds it, the waterfall execution shapes, and the UI-presentation vocabulary.
Source: [`packages/tools/src/index.ts`](../../packages/tools/src/index.ts) · [`packages/tools/src/schema.ts`](../../packages/tools/src/schema.ts)
## `ToolDefinition` — a registered tool
A `ToolSchema` (the model-facing fields) plus the `execute` function and optional UI presenters. The registry holds these; the loop dispatches calls through them. The registry's `schemas()` builds the model-facing `ToolSchema[]` by an explicit allowlist — `execute`/`presentCall`/`presentResult` must never leak into a model request.
```ts type-equiv
interface ToolDefinition extends ToolSchema {
execute(args: unknown, exec: ToolExecution): Promise<ContentBlock[]>
/**
* Optional: how to present the PENDING state of one call in a UI, derived
* from the call's `args` (parsed arguments, `unknown` — the tool validates/
* narrows its own input). Returning `undefined` (or omitting the method) tells
* a UI to fall back to a generic presentation (title = tool name, raw args as
* input). Pure and side-effect-free: a UI may call it during live streaming
* AND a session-log replay, so it must depend only on `args`.
*/
presentCall?(args: unknown): ToolCallPresentation | undefined
/**
* Optional: how to present the COMPLETED state, given the same `args` and the
* `result` (`execute`'s content + whether it errored). Returning `undefined`
* (or omitting the method) tells a UI to keep the pending title and render the
* raw result content. Pure and side-effect-free for the same replay reason.
*/
presentResult?(args: unknown, result: ToolResult): ToolResultPresentation | undefined
}
```
`execute` receives `args: unknown` — a raw `ToolDefinition` validates its own input. First-party tools don't write that by hand; they use `defineTool`, which validates and narrows for them.
## The typed schema DSL
Plugin authors write per-property specs with a boolean `required: true`, and a type-level helper maps the spec to the `execute` argument type — zero casts. The DSL is *machinery that types* `ToolDefinition`; it is intentionally a sub-page detail, not core.
Source: [`packages/tools/src/schema.ts`](../../packages/tools/src/schema.ts)
```ts type-equiv
interface SchemaProp {
type: SchemaType
/** Per-property required flag (NOT the JSON Schema top-level required array). */
required?: true
/** Human-readable description, surfaced in the JSON Schema as well. */
description?: string
/** Enum of allowed values (strings only). */
enum?: string[]
/** Default value. */
default?: unknown
/** Nested properties for type: 'object'. */
properties?: SchemaSpec
/** Items schema for type: 'array'. */
items?: SchemaProp
}
```
```ts type-equiv
type SchemaSpec = Record<string, SchemaProp>
```
`SchemaType` is the primitive union `'string' | 'number' | 'boolean' | 'object' | 'array'`. `InferArgs<S>` maps a `SchemaSpec` to the TS argument type — `required: true` props become required keys, everything else genuinely optional:
```ts type-equiv
type InferArgs<S extends SchemaSpec> = Simplify<
& { [K in RequiredKeys<S>]: InferPropValue<S[K]> }
& { [K in Exclude<keyof S, RequiredKeys<S>>]?: InferPropValue<S[K]> }
>
```
`defineTool({ name, description, parameters, execute, … })` ties it together: `parameters` is a `SchemaSpec`, `execute(args, exec)` gets `args: InferArgs<typeof parameters>`, and the helper converts the spec to JSON Schema (`schemaSpecToJsonSchema`) for the wire and validates model-generated args (`validateArgs`) before the typed body runs. A mismatch throws `ToolArgsError` (`code: 'INVALID_ARGS'`), which the registry turns into an `isError` result so the model can self-correct. Why a custom DSL and not schemastery: tool parameters need JSON Schema (the LLM wire format), not validation/transformation — the lightweight DSL gives the best authoring DX with the smallest surface.
## Execution: the `tools/execute` waterfall shapes
`ctx.tools.execute()` runs each call through the `tools/execute` waterfall — the single seam where sandbox, permission, hook, and plan-mode plugins wrap or veto. The pending call is a `ToolExecution`; the outcome is a `ToolExecutionResult`.
```ts type-equiv
interface ToolExecution {
callId: CallId
name: string
/** Parsed JSON arguments (unknown — tools validate their own input). */
arguments: unknown
/** The agent on whose behalf the call runs (set by the agent loop). */
agent?: Agent
signal?: AbortSignal
}
```
```ts type-equiv
interface ToolExecutionResult {
callId: CallId
content: ContentBlock[]
isError: boolean
/**
* Set when the call failed with a {@link HarnessError}: machine-routable
* `{ name, code }` for retry/sandbox plugins and replay. The model-facing
* text in `content` is always present; this is extra structure for code.
*/
error?: ToolErrorInfo
}
```
A waterfall listener receives `(exec, next)`: call `next()` to proceed (possibly around your own logic), or return a `ToolExecutionResult` without calling `next()` to veto. An unregistered tool routes through the same catch as a tool-thrown error, so both failure classes get a structured `{ name, code }` (`ToolNotFoundError` → `UNKNOWN_TOOL`) — the loop records a failed tool call instead of failing the whole turn.
## Tool-presentation UI vocabulary
How a tool wants its call shown in a UI (an editor tool-call card, a CLI log line), provider-neutral so a tool describes itself without depending on any client protocol. `presentCall` returns a `ToolCallPresentation` (pending state: `title`, `kind`, `rawInput`, `content`, optional `terminal`); `presentResult` returns a `ToolResultPresentation` (completed state: replacement `title`, reformatted `content`, terminal `output`/exit). `ToolCallKind` (`'read' | 'edit' | 'delete' | 'move' | 'search' | 'execute' | 'fetch' | 'other'`) picks an icon. A `ToolTerminal` asks a capable UI to render the call as a terminal card (cwd header, output, exit-status pill).
> These shapes carry a `FIXME(tool-presentation)` in source: they grew incrementally and the call-vs-result terminal split is muddy. Before more tools/UIs depend on them, they will be redesigned (a tagged union over card kinds) and pinned in an RFC, migrating `dsh-tool-bash` and the ACP bridge together. Treat the field-level shapes here as provisional; the source is authoritative.
The full presentation field docs live in [`packages/tools/src/index.ts`](../../packages/tools/src/index.ts). The bash tool's own schemas (`bash`/`bash_output`/`bash_kill`) and the executor they drive are on [bash.md](bash.md).