Add per-session snapshot replay for nested agents (PR2.5)

The snapshot tier was built single-session: dsh-llm-replay served calls from
one global positional cursor, and the harness harvested one session log. A
subagent runs as a second agent with its own session, so a parent→child
scenario could neither replay deterministically nor harvest the child's log.
This resolves the TODO(subagent-snapshots) deferral from the subagent RFC.

- Stamp the calling session id onto the model request: GenerateOptions.sessionId
  (typed Branded<'SessionId'> to avoid the dsh-llm↔dsh-session cycle), set by the
  agent loop from agent.session.id. Adapters ignore it; an llm/stream listener
  routes by it.
- Key replay per session: dsh-llm-replay loads the parent log plus one per child
  (childFiles / $DSH_SNAPSHOT_CHILD_FILES), derives a script per recorded session,
  and binds each live (freshly-random) session to a recorded script by first-call
  order — parent first (earliest createdAt, first to stream). Keys by WHO calls,
  so it survives a future concurrent/backgrounded subagent; a global cursor would
  not. An unrecorded extra session fails loud.
- Harvest every log: the harness collects all .jsonl across cwd buckets, ordered
  primary-first (top-level, then children by createdAt), and RunResult exposes the
  plural sessionLogs. The spec writes each back on record (session.jsonl +
  session.<n>.jsonl) and diffs each against its fixture on replay.
- Wire the subagent seam + spawn + fork + tool into the acp-agent example (both
  cordis configs) and add two nested scenarios recorded against the real API:
  subagent-spawn (parent + 1 child) and subagent-multi (parent + 2 children, 3
  sessions). Both replay keyless in the default gate.

A new RFC documents the design (docs/rfc/implemented/testing/). Single-session
replay is unchanged (a call with no sessionId is one anonymous primary session).

TODO follow-up: a dedicated branded-ids package could own the SessionId brand and
dissolve the cross-package cycle note; out of scope for this testing PR.
This commit is contained in:
Tianyi Cui
2026-06-22 08:39:36 +08:00
parent 9c1048f2b5
commit e68496fd79
22 changed files with 1392 additions and 88 deletions

View File

@@ -140,6 +140,20 @@ interface GenerateOptions {
*/
stop?: string[]
signal?: AbortSignal
/**
* The id of the session this request belongs to — stamped by the agent loop
* from `agent.session.id`. Adapters ignore it; it lets an `llm/stream` listener
* route a call by WHICH session issued it (the replay adapter keys its per-call
* cursor by session, so a parent and its in-process subagent — each with its
* own session on one context — replay from their own recorded scripts).
*
* Typed as `Branded<'SessionId'>` rather than importing `SessionId` from
* `dsh-session`: that package imports `Message` from here, so importing its
* `SessionId` back would cycle. `SessionId` IS `Branded<'SessionId'>`, so a
* real session id assigns with no cast. (A future ids package could own the
* brand and dissolve this note.)
*/
sessionId?: Branded<'SessionId'>
}
```

View File

@@ -139,6 +139,7 @@ Do NOT write one for a mechanical or local choice (a variable name, a one-file r
| [ACP snapshot tests — record-once / replay-deterministic](implemented/testing/2026-06-19-acp-snapshot-tests.md) | 2026-06-19 |
| [Real-API e2e in CI against the external DeepSeek API](implemented/testing/2026-06-19-real-api-e2e-ci.md) | 2026-06-19 |
| [Use `session.jsonl` as the only snapshot session-log artifact](implemented/testing/2026-06-20-remove-redundant-snapshot-log-goldens.md) | 2026-06-20 |
| [Per-session snapshot replay for nested agents](implemented/testing/2026-06-22-subagent-snapshot-replay.md) | 2026-06-22 |
## Rejected

View File

@@ -0,0 +1,52 @@
# RFC: Per-session snapshot replay for nested agents
Status: implemented
## Problem
The snapshot tier (`pnpm run test:snapshot`) boots the real `acp-agent` subprocess, replays a recorded session through [`dsh-llm-replay`](../../../../packages/support/llm-replay), and diffs the normalized stdout transcript + re-persisted session log against committed goldens. It is the only tier that exercises the full editor-facing transcript end to end.
It was built for ONE session per process, and that assumption is wired into two places:
- **`dsh-llm-replay` keyed nothing.** It served the Nth `llm/stream` call the Nth recorded entry from a single global cursor. With a parent agent AND an in-process subagent both streaming on one context, the calls interleave and the single cursor hands the child the parent's script (and vice versa).
- **The harness harvested one log.** `findSessionLog` walked the sessions root and returned the FIRST `.jsonl` it found. A subagent runs as a second `Session` with its own log in the same cwd bucket, so the child's transcript was silently dropped.
This was the `TODO(subagent-snapshots)` deferral recorded in the [subagent seam RFC](../../proposed/feature/2026-06-21-subagent-capability-seam.md): the in-process backends (PR2) shipped with unit + e2e coverage, but the full-transcript snapshot tier could not express a nested-agent shape until this infrastructure landed. This RFC is that stacked follow-up.
## Decision
Replay is keyed **per calling session**, and the harness harvests **every** session log.
### 1. The calling session id rides on the model request
`GenerateOptions` gains an optional `sessionId`, stamped by the agent loop from `agent.session.id` at request-assembly time (where the session is already in scope). Adapters ignore it; it exists so an `llm/stream` listener can route a call by WHICH session issued it. It is typed `Branded<'SessionId'>` (from `dsh-brand`) rather than importing `SessionId` from `dsh-session` — that package imports `Message` from `dsh-llm`, so importing its id back would cycle. `SessionId` IS `Branded<'SessionId'>`, so a real id assigns with no cast. (A future dedicated ids package could own the brand and dissolve the note; tracked separately — it touches every id import and does not belong in this testing PR.)
### 2. Replay binds live sessions to recorded scripts by first-call order
A nested scenario records more than one log: the parent (`session.jsonl`) plus one per subagent child (`session.1.jsonl`, …). `dsh-llm-replay` loads them all, derives one script per recorded session, and orders the scripts by header `createdAt` (the parent is created before its children).
Live session ids are freshly random every run and never equal the recorded ones, so a live session cannot bind to a script by id equality. Instead it binds by **first-call order**: the first live session to make any model call claims the first ordered script (the parent — earliest `createdAt`, and necessarily the first to stream, because it must run a turn before it can delegate), the next new live session claims the next script, and so on. Each session then advances its own cursor independently.
This keys by WHO calls, not by global call order — so it stays correct even if subagents ever run concurrently or in the background (a global cursor would interleave them). A call carrying no `sessionId` (a direct unit-test `stream()`) is treated as one anonymous session bound to the primary script, so the single-session path is byte-for-byte the old behavior. More distinct live sessions than recorded scripts is a fail-loud error (an unrecorded subagent appeared), never a silent mis-route.
The alternative considered and rejected was a **call-ordered merge of the parent and child logs** into one global script (sound only because in-process subagent execution is strictly nested — the parent blocks on the child). It is simpler for today's synchronous cut but bakes in the parent-blocks-on-child invariant that a future backgrounded/concurrent subagent would break; per-session keying does not.
### 3. The harness harvests every log, primary-first
`harvestSessionLogs` collects every `.jsonl` across every cwd bucket under the sessions root (the JSONL backend puts a parent and its same-cwd child in the same bucket), parses each header, and orders them primary-first: the top-level session (no `parentSession`) leads, then each child by ascending `createdAt`. `RunResult.sessionLogs` is the plural result; the spec writes each back to its fixture on record (`session.jsonl` + `session.<n>.jsonl`) and diffs each harvested log against its fixture on replay. The normalizer already accepted plural session ids and collapses any stray UUID, so no normalizer change was needed.
### 4. Scenarios
Two nested scenarios were added and recorded against the real API:
- **`subagent-spawn`** — the parent delegates one subtask via the `subagent` tool to a fresh spawn child (2 sessions).
- **`subagent-multi`** — the parent delegates two subtasks, each to its own spawn child (3 sessions), stressing the per-session keying with three concurrent scripts and the `createdAt` ordering of two children under one parent.
Both replay keyless in the default gate.
## Consequences
- The `TODO(subagent-snapshots)` deferral is resolved: nested-agent transcripts are now a first-class snapshot shape.
- `GenerateOptions.sessionId` is a small, honest core-seam addition useful beyond replay (telemetry, request routing).
- The `subagent` tool is bound to a single provider, so both children in `subagent-multi` are spawn (fresh). The fork backend is loaded in the example and exercised by PR2's unit tests; a mixed spawn+fork snapshot would need a second tool instance bound to `fork` (pure config) and is a trivial future addition, not a gap in the keying — the keying routes by session, not by backend.
- Out-of-process (ACP) subagents are a different replay shape entirely (each child is its own PROCESS with its own replay), tracked as `TODO(acp-subagent-replay)` in the PR3 plan.

View File

@@ -70,4 +70,4 @@ The `dsh-tool-subagent` consumer awaits `run.result` and returns the child's fin
- **Blocking the parent turn.** Synchronous collect holds the parent's `runStep` open for the child's full duration. This is acceptable for the first cut; **background / poll / spill semantics are deferred to a future redesign that unifies long-running-tool handling across subagents AND bash** (a sub-agent and a long `bash` background task pose the same "the model started something slow, how does it collect later" problem, and should share one mechanism rather than each inventing its own).
- **Live progress.** This cut surfaces only lifecycle + final result; a per-chunk child→parent update stream is deferred with the background redesign.
- **ACP client surface.** Proxying `fs`/`terminal` from the ACP child back to the parent (a shared-workspace mode) is future work; the first cut advertises neither, so the child self-serves in its own process.
- **Snapshot coverage of nested agents.** The snapshot tier (`pnpm run test:snapshot`) replays a recorded session through `dsh-llm-replay`, whose dispatch is a single GLOBAL positional cursor (the Nth `llm/stream` call serves the Nth recorded entry) and whose harness harvests a single session log file. A subagent runs as a *second* agent with its own session log, so a parent→child scenario needs per-session-keyed replay (or a call-ordered merge of both logs, sound because subagent execution is strictly nested/non-concurrent — the parent blocks on the child) plus harvest-all-logs and plural-session-id plumbing in the harness. This is self-contained infrastructure orthogonal to the backends, so it lands as a **dedicated stacked follow-up** rather than in the in-process-backends PR. Until it lands, in-process subagents are covered by real-loop unit tests (a parent driving a fork AND a spawn child) and a with-key e2e (a parent delegating to a child that writes a file), not by the snapshot transcript tier. Tracked by `TODO(subagent-snapshots)`.
- **Snapshot coverage of nested agents.** The snapshot tier (`pnpm run test:snapshot`) replays a recorded session through `dsh-llm-replay`. It was built single-session: a single GLOBAL positional cursor (the Nth `llm/stream` call serves the Nth recorded entry) and a harness that harvested a single session log file. A subagent runs as a *second* agent with its own session log, so a parent→child scenario needed per-session-keyed replay plus harvest-all-logs and plural-session-id plumbing self-contained infrastructure orthogonal to the backends, scheduled as a dedicated stacked follow-up rather than folded into the in-process-backends PR. That follow-up has **landed**: see [Per-session snapshot replay for nested agents](../../implemented/testing/2026-06-22-subagent-snapshot-replay.md). Replay now keys each call by its calling session (`GenerateOptions.sessionId`) and binds live sessions to recorded scripts by first-call order; the harness harvests every log; and two nested scenarios (`subagent-spawn`, `subagent-multi`) replay keyless in the default gate. In-process subagents remain covered by real-loop unit tests and a with-key e2e in addition to the snapshot tier.