Merge branch 'code-runtime-worker' into code-mode-tools
Brings in the refreshed base (master merged through the stack after #203 and #205 landed), including the acp-snapshot extraction (#204), and re-ports this PR's snapshot-suite extensions onto the extracted package: - dsh-acp-snapshot's Scenario gains headerClass and configPath; the suite factory pins the request header PER CLASS (construction rejects a missing or duplicated class pin), forwards a scenario's configPath to the harness (RunOptions.configPath overrides AgentUnderTest.configPath), and a new fixtures meta-test asserts every pinning fixture carries exactly one request/header and no deltas. - The acp-agent example's thin scenario table re-registers code-mode-turn and both-mode-turn with their overlay configs and per-class pins; the committed fixtures replay unchanged. - The package's synthetic suites cover the new surface (explicit headerClass on one suite, the default on the other, a configPath override through the fake bin, and the two construction throws).
This commit is contained in:
Tianyi Cui
@@ -889,6 +889,7 @@ Abstract service classes — a deployment loads a concrete implementation packag
|
||||
|
||||
Imported as libraries by other packages; a `cordis.yml` cannot load them.
|
||||
|
||||
- `@deepseek-ai/dsh-acp-snapshot` ([`packages/support/acp-snapshot/src/index.ts`](../packages/support/acp-snapshot/src/index.ts))
|
||||
- `@deepseek-ai/dsh-app-boot` ([`packages/ui/app-boot/src/index.ts`](../packages/ui/app-boot/src/index.ts))
|
||||
- `@deepseek-ai/dsh-brand` ([`packages/util/brand/src/index.ts`](../packages/util/brand/src/index.ts))
|
||||
- `@deepseek-ai/dsh-hook-protocol` ([`packages/hooks/hook-protocol/src/index.ts`](../packages/hooks/hook-protocol/src/index.ts))
|
||||
|
||||
@@ -68,6 +68,7 @@ flowchart TD
|
||||
pkg_session_persistence_sqlite["session-persistence-sqlite"]
|
||||
end
|
||||
subgraph group_support["packages/support"]
|
||||
pkg_acp_snapshot["acp-snapshot"]
|
||||
pkg_invariants["invariants"]
|
||||
pkg_llm_replay["llm-replay"]
|
||||
pkg_subagent_mock["subagent-mock"]
|
||||
@@ -216,6 +217,7 @@ flowchart TD
|
||||
| Package | Group | Depends on |
|
||||
| --- | --- | --- |
|
||||
| [`brand`](../packages/util/brand) | `util` | — |
|
||||
| [`acp-snapshot`](../packages/support/acp-snapshot) | `support` | — |
|
||||
| [`app-boot`](../packages/ui/app-boot) | `ui` | — |
|
||||
| [`code-runtime`](../packages/code-runtime/code-runtime) | `code-runtime` | — |
|
||||
| [`llm`](../packages/llm/llm) | `llm` | [`brand`](../packages/util/brand) |
|
||||
|
||||
@@ -11,6 +11,8 @@ Generated by `pnpm run gen-rfc-index` from the RFC tree — never edit by hand;
|
||||
| [Agent Client Protocol (ACP) support — drive the coding agent from external editors](proposed/feature/2026-06-14-acp-agent-client-protocol.md) | 2026-06-14 |
|
||||
| [Multiplex concurrent ACP sessions over one connection](proposed/feature/2026-06-14-acp-multi-session.md) | 2026-06-14 |
|
||||
| [Pre-tool input rewrite — a consistent design](proposed/feature/2026-06-30-pre-tool-input-rewrite.md) | 2026-06-30 |
|
||||
| [Claude Code and Codex subagent backends (out-of-process delegation to external coding agents)](proposed/feature/2026-07-07-claude-code-and-codex-subagent-backends.md) | 2026-07-07 |
|
||||
| [Repeat-tool-call guard plugin](proposed/feature/2026-07-08-repeat-tool-guard.md) | 2026-07-08 |
|
||||
|
||||
### Simplification
|
||||
|
||||
@@ -164,6 +166,7 @@ Generated by `pnpm run gen-rfc-index` from the RFC tree — never edit by hand;
|
||||
| [Hook snapshot matrix — end-to-end goldens for both bridges](implemented/testing/2026-07-04-hook-snapshot-matrix.md) | 2026-07-04 |
|
||||
| [Single-source the acp-agent replay config](implemented/testing/2026-07-04-single-source-acp-replay-config.md) | 2026-07-04 |
|
||||
| [Pin request-header content in one snapshot scenario](implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md) | 2026-07-06 |
|
||||
| [Extract the ACP snapshot suite into a support package](implemented/testing/2026-07-08-shared-acp-snapshot-package.md) | 2026-07-08 |
|
||||
|
||||
## Rejected
|
||||
|
||||
|
||||
@@ -32,4 +32,4 @@ Reviewers lose one artifact name that made the expected persisted log visually s
|
||||
|
||||
## Implementation note
|
||||
|
||||
The comparison normalizes BOTH sides, but each against its OWN volatile values, not a shared context. A raw harvested `session.jsonl` bakes in the recording run's session id, cwd, and timestamps; the replay run produces fresh ones. `normalizeSessionLog` scrubs cwd by exact string match, so normalizing the fixture against the *replay* run's cwd would leave the recorded cwd in the header unscrubbed and the compare would fail. The harness therefore derives the fixture's normalize context from its OWN header line (`{ type:'session', id, cwd }`) — `fixtureContext()` in `acp.snapshot.ts` — so both sides scrub to the same `{{sessionId}}`/`{{cwd}}` tokens. An authored fixture copied from the old golden already carries the normalized header (`id:'{{sessionId}}'`, `cwd:'{{cwd}}'`), which yields those tokens as the volatile values and scrubs idempotently. The session-log side uses a plain normalized-string `toEqual`, NOT `toMatchFileSnapshot`, so a run never overwrites the fixture.
|
||||
The comparison normalizes BOTH sides, but each against its OWN volatile values, not a shared context. A raw harvested `session.jsonl` bakes in the recording run's session id, cwd, and timestamps; the replay run produces fresh ones. `normalizeSessionLog` scrubs cwd by exact string match, so normalizing the fixture against the *replay* run's cwd would leave the recorded cwd in the header unscrubbed and the compare would fail. The harness therefore derives the fixture's normalize context from its OWN header line (`{ type:'session', id, cwd }`) — `fixtureContext()` in `dsh-acp-snapshot`'s suite module — so both sides scrub to the same `{{sessionId}}`/`{{cwd}}` tokens. An authored fixture copied from the old golden already carries the normalized header (`id:'{{sessionId}}'`, `cwd:'{{cwd}}'`), which yields those tokens as the volatile values and scrubs idempotently. The session-log side uses a plain normalized-string `toEqual`, NOT `toMatchFileSnapshot`, so a run never overwrites the fixture.
|
||||
|
||||
@@ -8,7 +8,7 @@ Every model-driving ACP snapshot fixture (`session.jsonl`) embedded the full com
|
||||
|
||||
## Decision
|
||||
|
||||
Exactly one scenario — `text-turn`, flagged `pinsHeader` in `acp.snapshot.ts` — commits and compares the full request-header content. Every other fixture stores and compares that content as stable tokens via the pure normalizer `scrubRequestHeaders` in `snapshot-normalize.ts`: a `request/header` event's `header.system` becomes `"{{system}}"` and `header.tools` becomes `"{{tools}}"`; a `request/header-delta` keeps its structural facts — the system delta's `keepStart`/`keepEnd` line positions with one `{{system}}` token per inserted line, the tools delta's added/removed/changed tool names — and tokenizes only the bulk (prompt text, schema bodies), so two different deltas still compare different. The scrub is composed in front of `normalizeSessionLog` on BOTH sides of a non-pinning scenario's log compare and applied to the harvested logs record mode writes, so a re-record cannot smuggle the content back. Absent fields stay absent — WHETHER a header carried a prompt or tools is behavior and stays visible — and `config`/`reason` stay verbatim: a model swap churns every fixture by design (it invalidates the recorded responses), while a prompt or schema edit churns none of them (replay derives model behavior exclusively from `assistant/chunk` events and never reads header content — see `dsh-llm-replay`).
|
||||
Exactly one scenario — `text-turn`, flagged `pinsHeader` in the `acp.snapshot.ts` scenario table — commits and compares the full request-header content; the pin mechanics live in [`dsh-acp-snapshot`](../../../../packages/support/acp-snapshot/README.md), whose suite factory enforces one pin per consuming suite. Every other fixture stores and compares that content as stable tokens via the pure normalizer `scrubRequestHeaders` in that package's `normalize.ts`: a `request/header` event's `header.system` becomes `"{{system}}"` and `header.tools` becomes `"{{tools}}"`; a `request/header-delta` keeps its structural facts — the system delta's `keepStart`/`keepEnd` line positions with one `{{system}}` token per inserted line, the tools delta's added/removed/changed tool names — and tokenizes only the bulk (prompt text, schema bodies), so two different deltas still compare different. The scrub is composed in front of `normalizeSessionLog` on BOTH sides of a non-pinning scenario's log compare and applied to the harvested logs record mode writes, so a re-record cannot smuggle the content back. Absent fields stay absent — WHETHER a header carried a prompt or tools is behavior and stays visible — and `config`/`reason` stay verbatim: a model swap churns every fixture by design (it invalidates the recorded responses), while a prompt or schema edit churns none of them (replay derives model behavior exclusively from `assistant/chunk` events and never reads header content — see `dsh-llm-replay`).
|
||||
|
||||
A system-prompt or tool-schema change therefore lands as exactly one committed-fixture diff — the pinned `text-turn` header line — updated by hand or by re-recording that one scenario (`pnpm run test:snapshot:record` with `-t text-turn`).
|
||||
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
# RFC: Extract the ACP snapshot suite into a support package
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
The ACP snapshot tier ([snapshot RFC](2026-06-19-acp-snapshot-tests.md)) was built from three modules living inside one example's test directory: `snapshot-harness.ts` (boot the real bin subprocess, drive it over ACP JSON-RPC, harvest the persisted logs), `snapshot-normalize.ts` (the pure golden normalizers), and the ~150-line scenario body plus fixture guards in `acp.snapshot.ts` (record/replay modes, the stdout-golden and log compares, the pinned-header uniformity guard, the orphan/required-file/single-pin meta-tests).
|
||||
|
||||
A second ACP example wanting snapshot coverage — the sandbox/approval composition is the immediate consumer — could only copy those modules, forking exactly the logic that must not drift: record write-back, header scrubbing, child-session harvest ordering. The spawn/client glue was already triplicated across `acp.e2e.ts`, `hooks.e2e.ts`, and the harness (`TODO(acp-test-harness)`). Location also decided test rigor: the per-file 100% coverage gate measures `packages/*/*/src` only, so none of this machinery was measured — the same gap that had moved `dsh-llm-replay` out of `examples/` into [packages/support](../../../../packages/support/README.md). And the harness's ACP client hardcoded `requestPermission → cancelled`, so an approval round-trip — the headline behavior of the sandbox composition — could not be expressed at the snapshot tier at all.
|
||||
|
||||
## Decision
|
||||
|
||||
The machinery lives in [`packages/support/acp-snapshot`](../../../../packages/support/acp-snapshot/README.md) (`@deepseek-ai/dsh-acp-snapshot`); an example's `*.snapshot.ts` is its scenario table, its agent paths, and one factory call, over its own `snapshots/` fixtures and `cordis.snapshot.yml` overlay ([single-source replay config](2026-07-04-single-source-acp-replay-config.md)). Reading `DSH_SNAPSHOT` stays at that edge — the library takes a resolved `mode`.
|
||||
|
||||
**`src/harness.ts`** — `runScenario` and the input-script/result types, parameterized by an `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath`; absolute paths the consuming suite resolves from its own `import.meta.url`). The client's `session/request_permission` handler consumes an optional `InputScript.permissionAnswers` FIFO queue, each entry selecting by option **kind** (ids are agent-issued randoms a committed script cannot know; kinds are the ACP-stable vocabulary, mapped to the offered `optionId` at answer time); an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run — the agent itself is answered `cancelled`, so the scenario bug fails the harness rather than being absorbed as an agent-side denial. This is what lets an approval suite drive allow/reject round-trips deterministically from `input.json`.
|
||||
|
||||
**`src/normalize.ts`** — the pure normalizers, hook-free by policy: when a future event carries a new volatile field (an approval duration, say), the shared normalizer learns it in the same change, keeping one home for what "normalized" means rather than per-suite scrub extensions.
|
||||
|
||||
**`src/suite.ts`** — the `Scenario` type and `defineAcpSnapshotSuite(options)`, registering the per-scenario compares, record-mode fixture write-back, the header pin with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin, non-pinning fixtures are `scrubRequestHeaders` fixed points). The pinned-header contract ([pinned-header RFC](2026-07-06-pin-request-header-content-in-one-scenario.md)) is per-suite: each suite flags exactly one `pinsHeader` scenario (the factory throws on zero, a meta-test rejects more than one; WHICH scenario pins is the table's reviewable choice), and the uniformity guard compares only that suite's sessions. The pure helpers (`childFixturePaths`, `fixtureContext`, `normalizedHeaders`, `headerDeltaCount`) are exported for direct unit coverage.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Copy the modules into each example** — the fork this RFC exists to prevent: the record/guard logic is exactly the code that must stay byte-identical across suites, and examples are outside the coverage gate, so each copy is also unmeasured.
|
||||
- **A shared module directory under `examples/`** — keeps the code outside the coverage gate and forces relative imports across example boundaries, against the package-name import convention; `examples/` leaves stay thin by design.
|
||||
- **A `/testing` subpath export of `dsh-acp-agent`** — couples test infrastructure into a product package's surface and dependency set; `packages/support/` exists precisely for real-but-lower-compatibility dev/test packages, with `dsh-llm-replay` as the precedent this package completes.
|
||||
- **Export raw test-body functions instead of a suite factory** — each example would re-own the `describe`/`it` skeleton (~80 lines of registration boilerplate per suite) for no flexibility gain; the factory keeps consumers to a scenario table plus one call, and the exported pure helpers preserve unit-testability inside the factory design.
|
||||
- **An injectable ACP `Client` factory instead of declarative `permissionAnswers`** — maximally flexible, but it leaks SDK client construction to every consumer and reopens per-example drift in exactly the layer being unified; a declarative queue keeps `input.json` the single scripting surface and stays golden-normalizable.
|
||||
- **Generalize beyond ACP (a transport-agnostic snapshot harness)** — no second transport exists; the harness is ACP-shaped end to end (SDK client, JSON-RPC frames, `session/update` waiters), and a speculative abstraction would be a seam split ahead of any consumer.
|
||||
|
||||
## Testing
|
||||
|
||||
Extraction parity was proven mechanically: after the move, `pnpm run test:snapshot` matched the base commit's result with zero byte changes under `examples/acp-agent/tests/snapshots/`. The package's `src/` holds per-file 100% statements/branches/functions/lines under the gating unit run, driven through the REAL spawn path by a scripted fake ACP bin (`tests/fixtures/fake-acp-agent.ts`, behavior scripted per scenario via a `behavior.json` beside the fixture): `harness.spec.ts` covers every step op, both expect-error arms, the permission queue (selection, fallback, impossible-click), env forwarding, workspace seeding, and the harvest ordering/noise/fallback branches; `suite.spec.ts` runs the factory for real at collection time — a replay suite over committed synthetic fixtures and a record suite over a temp copy (write-back never touches the committed tree; `ACP_SNAPSHOT_SPEC_BOOTSTRAP=1` re-bootstraps it) — plus direct cases for the pure helpers. Two structurally unreachable guards carry reasoned `v8 ignore` comments. The fake bin substitutes the `session/new` cwd, not `process.cwd()`, into scripted logs, matching what the real bin's header carries (darwin realpaths `/var/folders/…` to `/private/var/folders/…`).
|
||||
|
||||
## Consequences
|
||||
|
||||
A new example gets the whole snapshot tier from a scenario table plus fixtures — the sandbox branch merges master down and adds its own suite (own pin scenario, own overlay, fixtures via `test:snapshot:record`, approvals via `permissionAnswers`). The costs: `suite.ts` imports vitest, so the package is importable only inside a vitest run — a shape no other package has, stated in its README; each suite pins its own ~8 KB header fixture (a genuinely distinct composition deserves its own pin; an identical one would be caught by that suite's uniformity guard); and the e2e launcher duplication remains (`TODO(acp-test-harness)`) — the harness is the extraction target when that migration lands.
|
||||
@@ -0,0 +1,87 @@
|
||||
# RFC: Claude Code and Codex subagent backends (out-of-process delegation to external coding agents)
|
||||
|
||||
Status: proposed
|
||||
|
||||
## Problem
|
||||
|
||||
The subagent seam ([the seam RFC](../../implemented/feature/2026-06-21-subagent-capability-seam.md)) hosts multiple named providers on `ctx.subagents`, and the ACP backend ([the ACP backend RFC](../../implemented/feature/2026-06-22-acp-subagent-backend.md)) proved the seam generalizes across a process boundary; its Future-providers section explicitly named the Codex app-server and the Claude Code Agent SDK as mechanically similar siblings. Those two are the engines actually worth delegating to today: a harness turn should be able to hand a self-contained task to a real Claude Code or a real Codex — a separate product with its own model, tools, and sandbox — and get back one final answer, without the parent deployment leaking its secrets into the child or the child's behavior silently depending on whatever `~/.claude` / `~/.codex` state exists on the host machine.
|
||||
|
||||
## Proposal
|
||||
|
||||
Two sibling provider packages, structural variants of the ACP backend, plus one extraction:
|
||||
|
||||
- `@deepseek-ai/dsh-subagent-claude-code` — drives a Claude Code child through `@anthropic-ai/claude-agent-sdk`'s `query()` (the SDK runs in the parent process and spawns its bundled `claude` CLI as the subprocess). Provider name `claude-code`: the child is the Claude Code *product*, not an Anthropic model adapter — "claude" stays reserved for a future `dsh-llm` adapter.
|
||||
- `@deepseek-ai/dsh-subagent-codex` — spawns `codex app-server` and drives one thread/turn over its JSON-RPC-over-stdio protocol with a hand-rolled newline-JSON client (~200–300 lines) in the package.
|
||||
- `@deepseek-ai/dsh-subagent-process` — a pure library (the `subagent-inprocess` precedent) extracting what `dsh-subagent-acp` already carries and both new backends need: the credential env scrub (`SENSITIVE_ENV_PATTERN`/`buildChildEnv`), the EOF → SIGTERM → SIGKILL dispose ladder, and new isolated-config-dir helpers (`mkdtemp` create, best-effort remove). The ACP backend migrates onto it; `bash-local`'s sibling copy is left alone to bound the change.
|
||||
|
||||
Both providers copy the ACP backend's seam posture verbatim: fresh child per `start`, exactly one prompt round-trip, capabilities all `false`, `inheritsParentContext: false`, `request.parent`/`request.agentOptions` ignored, `id = AgentId(randomUUID())`, `result` never rejects — child-level failure flattens to a stop reason and the original error goes to `ctx.logger` via an `onError` spec callback. Model exposure is zero new code: `dsh-tool-subagent` is loaded once per provider with a distinct `toolName` (`subagent_claude_code`, `subagent_codex`). No new session events are needed — the only model-visible artifact is the tool result, so reconstructability holds exactly as it did for ACP. To be explicit about the boundary: the session log reconstructs the model-visible transcript, not workspace mutation history — a child granted write access mutates files as an ambient side effect outside the log, exactly as the bash tools and the ACP backend already do; replay reproduces requests, not the disk.
|
||||
|
||||
## Verified interface facts (pinned versions)
|
||||
|
||||
Both integration surfaces were verified against pinned implementations before this proposal — types and bundled source read, keyless spikes run — not from vendor docs alone. The pins are the verification baseline, not a runtime contract: the backends perform no runtime version probe (no `codex --version` gate, no SDK version sniffing). Compatibility is enforced at development time — every dependency bump re-runs the keyless suites against the real load path — and at runtime by failing loudly: a protocol-level surprise settles `error` via `onError`, never a silent misbehavior.
|
||||
|
||||
**`@anthropic-ai/claude-agent-sdk` 0.3.202.** `options.env` REPLACES the child environment (no merge with `process.env`), which is exactly what the scrub needs. `settingSources` defaults to loading ALL filesystem settings — isolation requires explicitly passing `[]`. Result subtypes are `success` | `error_during_execution` | `error_max_turns` | `error_max_budget_usd` | `error_max_structured_output_retries`. On abort the SDK escalates the CLI child itself: stdin EOF immediately, SIGTERM ~2s later if the child ignores it (observed; no leftover processes) — no bespoke kill fallback needed. `outputFormat: {type: 'json_schema'}` and an `agents` option exist, giving future landing points for the seam's `outputSchema` capability and named subagent types; both are out of scope here.
|
||||
|
||||
**codex CLI 0.142.5, `codex app-server` (v2 vocabulary).** LF-delimited JSON, JSON-RPC 2.0 shapes with the `"jsonrpc"` header omitted.
|
||||
|
||||
- Lifecycle: `initialize{clientInfo}` + `initialized` → `thread/start` (accepts `cwd`, `model`, `sandbox`, `approvalPolicy`, `ephemeral`; succeeds unauthenticated) → `turn/start{threadId, input:[{type:'text',text}]}` returns an `inProgress` turn immediately; the terminal signal is the `turn/completed` notification carrying `Turn{status: completed|interrupted|failed|inProgress, error}`.
|
||||
- Approvals are server-initiated requests — `item/commandExecution/requestApproval`, `item/fileChange/requestApproval`, `item/permissions/requestApproval`, `item/tool/requestUserInput`, `mcpServer/elicitation/request` — answered with `accept`/`decline`-family decisions.
|
||||
- Auth: `account/login/start{type:'apiKey', apiKey}` is a first-class RPC and `account/read` reports `requiresOpenaiAuth` — and an unauthenticated `turn/start` does NOT fail fast (it hangs in retry), so the backend MUST pre-check auth and settle `error` loudly instead of waiting on the turn.
|
||||
- Isolation: `CODEX_HOME` redirection is honored (the `initialize` response echoes it, so tests can assert isolation), and `ephemeral: true` threads leave no session files at all.
|
||||
|
||||
## Isolation and credentials
|
||||
|
||||
Deployments authenticate with API keys only, and the child must not see the host user's Claude Code / Codex configuration: behavior has to be a function of `cordis.yml` alone. Each run gets a fresh `mkdtemp` config dir — `CLAUDE_CONFIG_DIR` for Claude Code (paired with an explicit `settingSources: []`), `CODEX_HOME` for Codex — removed best-effort on dispose; a config field can pin a persistent dir instead. The child env reuses the ACP backend's `buildChildEnv` semantics verbatim via the extraction: the ambient env is forwarded MINUS credential-shaped vars (`/KEY|SECRET|TOKEN/i`), with `config.env` layered on top — so `PATH`, `HOME`, `TMPDIR`, locale, and proxy vars survive and the CLIs run normally, while only credential-shaped ambient vars are scrubbed (`ANTHROPIC_API_KEY` enters explicitly through `config.env` for Claude Code), and the Codex key travels via the `account/login/start` RPC into the isolated `CODEX_HOME` rather than a hand-written `auth.json`.
|
||||
|
||||
## Permission and approval policy
|
||||
|
||||
Instead of collapsing to ACP's single `permission: allow|reject` knob, each backend exposes its engine's native vocabulary as config, with conservative defaults: Claude Code gets `permissionMode` (default `default`) plus `permission: allow|reject` (default `reject`) as the `canUseTool` auto-answer for whatever falls through; Codex gets `sandboxMode` (default `read-only`) and `approvalPolicy` (default `never`) plus the same `permission` fallback for approval requests that still arrive. Defaults are deliberately do-no-harm (the out-of-box child cannot write files); examples demonstrate opening up (`acceptEdits` / `workspace-write`). The mechanical rule: EVERY server-initiated request is settled programmatically and promptly — the enumerated approval/user-input/elicitation requests by the configured policy, an unknown request method with a JSON-RPC method-not-found error response (never left pending), unknown notifications consumed — so no child request can wedge a turn waiting on an answer that will never come. Prompts never reach a human in this cut, matching ACP.
|
||||
|
||||
## StopReason mapping
|
||||
|
||||
Claude Code: `success` → `completed`; `error_max_turns`, `error_during_execution`, `error_max_budget_usd`, `error_max_structured_output_retries` → `error` (aligning with the ACP call on `max_turn_requests`: an unfinished task is not success); generator abort → `aborted`; anything unknown → `error`. Codex: `Turn.status` `completed` → `completed`; `interrupted` → `aborted`; `failed` with `codexErrorInfo: 'contextWindowExceeded'` → `max-tokens`, any other `failed` → `error`; transport/spawn/auth-precheck failure → `error` (or `aborted` if cancel was requested). In both, `cancel()` is the ACP shape: flag + abort/interrupt + a cancel-settled race arm so an uncooperative child cannot stall the result.
|
||||
|
||||
Liveness posture, stated explicitly: teardown timing is config, turn duration is not. Both backends take the dispose ladder's grace periods as defaulted validated config fields (the ACP backend's `disposeEofGraceMs`/`disposeGraceMs` shape, carried by the extraction), but there is deliberately NO turn-duration or startup timeout — matching ACP, liveness during a turn belongs to the caller via `cancel()`/the abort signal, a subagent turn is legitimately minutes long, and the Codex auth precheck removes the one verified guaranteed-hang; a deployment wanting a wall-clock bound cancels from the parent.
|
||||
|
||||
## Testing
|
||||
|
||||
Named at every tier per the root AGENTS.md rule, and de-risked up front:
|
||||
|
||||
- **Keyless unit/integration**, mirroring the ACP spec list per backend (round-trip and output accumulation, every stop mapping, both cancel paths, already-aborted, permission auto-answer under both policies, unknown-message tolerance, bad-command spawn failure, HMR provider cleanup, export shape, isolation assertions on child env and temp-dir removal; Codex adds the auth-precheck failure path). Claude Code's harness is a scripted fake `claude` executable behind `pathToClaudeCodeExecutable` driven by the REAL SDK — a spike already passed end-to-end keyless in 24ms (the fake CLI answers one `control_request/initialize` and speaks plain stream-json, ~40 lines). Codex's harness is a scripted mock app-server subprocess speaking the verified wire protocol, the `mock-acp-server.ts` shape.
|
||||
- **With-key e2e** per backend: the real engine does real file work verified on disk, under a pinned opened-up config so acceptance and the do-no-harm defaults don't collide — `permissionMode: 'acceptEdits'` for Claude Code, `sandboxMode: 'workspace-write'` + `approvalPolicy: 'never'` for Codex; self-skips report exactly what is missing (binary vs key). CI has no secrets, so these run locally per the with-key policy.
|
||||
- **Snapshot**: deferred as `TODO(claude-code-subagent-replay)` / `TODO(codex-subagent-replay)` — the same distinct replay shape the ACP backend deferred ([the per-session replay RFC](../../implemented/testing/2026-06-22-subagent-snapshot-replay.md)); the keyless suites carry deterministic coverage meanwhile.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
### Why not the official `@openai/codex-sdk` instead of a hand-rolled client?
|
||||
|
||||
The dispose ladder and env scrub require owning the child process (spawn args, env, signals, exit await); the SDK hides the process. The wire format is trivial to frame (LF JSON), the shapes are generatable per pinned version (`codex app-server generate-json-schema`), and the repo precedent (`hook-protocol`) is to own thin protocol cores rather than wrap someone's runtime. The SDK would save protocol-evolution maintenance but costs the exact control this backend exists to have.
|
||||
|
||||
### Why not a model-visible `subagent_type` parameter (one Task-style tool)?
|
||||
|
||||
Claude Code's own Task tool puts the subagent type in the model-facing schema, selecting a prompt-plus-toolset persona. Here the choice is between EXECUTION ENGINES, and only the deployer knows which engines have credentials configured — so selection stays deployment config, preserving `dsh-tool-subagent`'s documented one-provider-per-tool contract. A persona-style type selector would be a separate RFC against the tool, not the backends.
|
||||
|
||||
### Why not login-state credentials and the user's own config?
|
||||
|
||||
Inheriting `~/.claude` / `~/.codex` (subscription login, user settings, skills, MCP servers) would make child behavior depend on host-machine state and punch an implicit exception through the "credentials enter explicitly via `config.env`, never ambiently" rule the ACP backend and bash executor established. API-key-only plus forced config-dir isolation keeps runs reproducible; deployments wanting shared state can point the config-dir field at a persistent directory deliberately.
|
||||
|
||||
### Why not a driver-injection seam for the Claude Code keyless tests?
|
||||
|
||||
Injecting a fake `query()` would mock our own boundary and leave the real SDK load path untested (the real-over-mock policy in docs/testing.md). The risk that justified considering it — the SDK↔CLI stream-json control protocol being internal — was retired by the spike: the fake-CLI harness works against the real pinned SDK today. If an SDK upgrade breaks the mock, the keyless suite fails the upgrade PR, which is the gate working.
|
||||
|
||||
### Why not ACP adapters (e.g. `claude-code-acp`) reusing the existing backend?
|
||||
|
||||
Community shims wrap both engines in ACP, which would make them "just config" on `dsh-subagent-acp`. But that inserts an unofficial third-party layer between the harness and the engine, erases the native control surfaces this RFC exposes (permissionMode, sandboxMode/approvalPolicy, config-dir isolation, apiKey RPC), and trades first-party protocol stability for a shim's release cadence. First-party surfaces — the Agent SDK and the app-server — are the supported integration points.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
On a machine with both engines and keys configured: a REPL-driven model completes one real file task through `subagent_claude_code` and one through `subagent_codex`, the tool result being the child's final answer, with only `tool/call` + `tool/result` in the parent session log. Keyless suites pass at 100% per-file coverage in a credential-less environment, asserting isolation (scrubbed child env, no temp config dirs left after dispose) and that child behavior is unchanged by the presence or absence of `~/.claude` / `~/.codex`. Cancelling a parent turn quiesces both backends in bounded time with no leftover child processes. E2e suites self-skip cleanly, naming the missing prerequisite.
|
||||
|
||||
## Risks
|
||||
|
||||
- `codex app-server` is CLI-flagged experimental and its v1/v2 vocabularies coexist; the client pins 0.142.5, implements v2 only, and consumes unknown methods/notifications without crashing, but a future codex bump can still force rework (regenerate schemas and re-run the keyless suite on every bump — the development-time enforcement behind the no-runtime-version-probe stance above).
|
||||
- The Claude Code fake-CLI mock rides an internal protocol: any SDK upgrade must go through the keyless suite, and a breaking control-protocol change means reworking the mock (fallback: the driver-injection seam rejected above becomes the escape hatch).
|
||||
- The SDK's optionalDependencies weigh ~280MB per platform — accepted, and confined to the one backend package.
|
||||
- The SDK's SIGKILL branch beyond EOF→SIGTERM was not observed and is trusted; e2e keeps a no-leftover-process assertion.
|
||||
- Codex is a deployment prerequisite (no npm-bundled binary); a missing or incompatible binary surfaces as a loud spawn/protocol `error`, not a version probe.
|
||||
- Every run pays a fresh child process and only the final answer surfaces — thoughts, tool cards, and usage are consumed and dropped; pooling, intermediate-progress surfacing, `sendMessage`/`resume`, `outputSchema` via the SDK's `outputFormat`, and named subagent types via the SDK's `agents` option are all deliberate deferrals.
|
||||
79
docs/rfc/proposed/feature/2026-07-08-repeat-tool-guard.md
Normal file
79
docs/rfc/proposed/feature/2026-07-08-repeat-tool-guard.md
Normal file
@@ -0,0 +1,79 @@
|
||||
# RFC: Repeat-tool-call guard plugin
|
||||
|
||||
Status: proposed
|
||||
|
||||
## Problem
|
||||
|
||||
A model stuck in a loop re-issues the same tool call with byte-identical arguments — re-running a failing grep, re-reading an unchanged file, polling a command that already gave its answer — and each round trip burns tokens, wall-clock, and (for paid APIs) money without adding information. The harness has nothing that notices: the loop has no step budget, no plugin tracks call repetition, and the model only escapes when it happens to vary its own behavior. The failure mode is real and cheap to detect — [pi-repeat-tool-guard](https://github.com/Kingwl/pi-repeat-tool-guard) ships exactly this as a pi coding-agent extension: count consecutive identical calls and, past a threshold, append a `<system-reminder>` telling the model to stop repeating itself and change course.
|
||||
|
||||
The harness already has every seam the pi extension uses, and better ones: [the interception-seams RFC](../../implemented/feature/2026-06-30-interception-seams.md) gives `tools/post-execute` a sanctioned way to attach model-facing context to a finished call, the loop buffers and injects that context with call/result adjacency preserved, and injected context is a logged `context/message` — so a native guard satisfies the model-visible ⟺ logged rule with no new session event. What is missing is only the plugin itself.
|
||||
|
||||
## Proposal
|
||||
|
||||
The guard is a loop-hygiene plugin, not a model-facing tool: it never appears in the tool list, never vetoes or rewrites a call, and adds exactly one behavior — it watches each agent's stream of tool calls, counts runs of consecutive calls to the same tool with identical canonicalized arguments, and at configured run lengths injects an escalating advisory reminder telling the model to stop repeating itself, re-read the last result, and either change approach or conclude. The purpose is to break unproductive loops within a few wasted steps instead of letting them run to the turn's natural end — while leaving the decision (retry differently, gather more evidence, or finish) entirely with the model, so a legitimately repeated call is delayed by nothing and blocked by nothing.
|
||||
|
||||
The shape: one new leaf plugin package, `@deepseek-ai/dsh-repeat-tool-guard` at `packages/guard/repeat-tool-guard/`, opening a `guard/` group for loop-hygiene plugins (single-package groups have precedent: [the todo-write RFC](../../implemented/feature/2026-06-29-todo-write-tool.md) shipped `todo/tool-todo`). The plugin registers three listeners via `ctx.effect()` and holds all state in plugin-local maps keyed by `AgentId` — the tool registry is a context-level singleton whose waterfalls interleave every agent's calls (subagents run on the same context), so per-agent keying is correctness, not polish.
|
||||
|
||||
- **`tools/post-execute` (waterfall)** — the one detection point. The listener receives `(exec, result)` together, so counting and reminder delivery need no cross-event pending map (the pi extension needs one only because its `tool_call`/`tool_result` hooks are separate events). It always delegates via `next()` and, when a threshold is hit, folds a reminder onto the downstream decision's `additionalContext` — the observe-and-enrich posture [the hooks bridges](../../implemented/feature/2026-06-30-hook-bridges.md) already use, honoring the waterfall contract. Counting happens here rather than in `tools/pre-execute` because post-execute also runs for denied calls (`ToolRegistry.execute` routes a deny through the same pipeline), and a model hammering a denied call is exactly the loop worth breaking.
|
||||
- **`agent/prompt-submit` (waterfall)** — pure reset hook: delegate via `next()`, clear the submitting agent's chain. A user interjection changes the context; repetition across it is not a loop.
|
||||
- **`agent/status` (emit)** — on `disposed`, drop the agent's state, bounding the maps over harness lifetime.
|
||||
|
||||
### Detection semantics
|
||||
|
||||
The chain key is `(tool name, canonical arguments)`; a call identical to the previous tracked call increments the agent's consecutive counter, a different tracked call resets it to 1. Canonicalization is a deep key-sort plus `JSON.stringify`: `ToolExecution.arguments` is by construction the loop's `JSON.parse` output (or the raw string fallback for malformed argument JSON, which is itself a comparable value), so the pi original's bigint/circular/`undefined` handling has no inputs here and is deliberately dropped.
|
||||
|
||||
Two deliberate rules, both documented in the package README because they are behavior a reader would otherwise guess at:
|
||||
|
||||
- **Untracked calls are transparent to the chain.** A call excluded by `include`/`exclude` neither increments nor resets the counter, so `grep X → todo_write → grep X` still counts as two consecutive `grep X` when `todo_write` is excluded. This is what makes exclusion useful — bookkeeping tools interleaved into a loop must not launder it — and it is the pi extension's (undocumented) semantics, kept on purpose and written down.
|
||||
- **Calls without an agent are ignored.** A direct `ctx.tools.execute()` caller (tests, future non-loop consumers) has no model to remind and no `AgentId` to key on.
|
||||
|
||||
### Reminder delivery
|
||||
|
||||
Reminders ride `additionalContext` (source `{kind: 'plugin', plugin: 'repeat-tool-guard'}` — the label is load-bearing per `HookContext`), never a `content` replacement: the `tool/result` event stays the tool's own output for audit, and the loop already appends buffered context as `context/message`(s) after the step's results, which the session renders as the tagged synthetic-user envelope and derived history replays. Thresholds escalate: the first configured threshold gets a short "you are repeating yourself, analyze the previous result" nudge; each later threshold gets the detailed form naming the tool, the repeat count, and the canonical arguments, and stating that the calls made no progress. The pi original hardcodes the gentle text to the literal count 3; the guard keys it to `thresholds[0]`, fixing that bug in the port. When the downstream decision already carries `additionalContext` (a hook bridge on the same call), the guard folds content following the shared-merge precedent in `dsh-hook-protocol`.
|
||||
|
||||
### Config
|
||||
|
||||
```yaml
|
||||
- id: repeat-tool-guard
|
||||
name: '@deepseek-ai/dsh-repeat-tool-guard'
|
||||
config:
|
||||
thresholds: [3, 5, 8] # default; consecutive counts that trigger a reminder
|
||||
include: [] # tool-name patterns to track; empty ⇒ all tools
|
||||
exclude: [todo_write] # tool-name patterns transparent to the chain
|
||||
```
|
||||
|
||||
`thresholds` is validated at load and throws on an empty list, a non-integer, a value below 2, or a duplicate — misconfiguration fails loud, replacing the pi original's silent fall-back to defaults. `include`/`exclude` entries support `*` wildcards. Patterns are predicates over whatever tools exist at call time, not references to a registry entry, so an entry matching no currently registered tool is NOT an error — unlike `toolOrder`'s referent check, `exclude: [mcp_*]` must stay valid in a deployment that loads no MCP tools.
|
||||
|
||||
### Testing
|
||||
|
||||
Coverage named at plan time, per tier: **unit** — counting/reset semantics (identical, different-tracked, untracked-transparent, prompt-submit reset, disposal cleanup, per-agent isolation), canonicalization, threshold escalation including the `thresholds[0]` gentle-text rule, config fail-loud cases, and the fold-onto-downstream-decision path, to per-file 100% like every `packages/*/*/src` file. **Snapshot** — one scripted-replay scenario where the model repeats a call to threshold and the reminder `context/message` appears in the transcript, pinning the model-visible text and its envelope (this is a transcript-surface change; the ACP snapshot suite is the tier that owns it). **e2e** — none: the plugin is provider-independent and deterministic, and forcing a live model to repeat a call three times is not a stable test; the seam contracts it relies on are already e2e-covered by their owners.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Append the reminder into the tool result** (`accept` with replaced `content` — the pi extension's mechanism, which patches result content because that is the only channel its API offers) — rejected: it makes the logged `tool/result` lie about what the tool returned, and `additionalContext` exists precisely as the separate sanctioned channel for post-execute commentary, with loop-level buffering that preserves call/result adjacency.
|
||||
- **Count in `tools/pre-execute` with a pending-reminder map** (the pi two-phase shape) — rejected: post-execute alone sees `(exec, result)` together and also fires for denied calls, so one listener with no cross-event state covers strictly more attempts with less machinery.
|
||||
- **Escalate to `block` at the highest threshold** — rejected for the initial scope: a blocked call punishes legitimate identical repeats (polling a long-running terminal, re-checking a file the agent expects to change), and an advisory reminder keeps the model in control. Revisit with evidence; the decision shape (`PostToolDecision`) already supports it.
|
||||
- **A per-deployment external hook via the CC/Codex bridges** (a `PostToolUse` script) — rejected as the answer: it works today for one deployment, but a shipped, unit-tested, `cordis.yml`-configurable plugin is the harness-native form, without per-call subprocess cost.
|
||||
- **A loop-level step or repetition budget in `agent-loop`** — rejected: "plugins, not loop changes"; a hard step budget is a blunter, orthogonal control that would need its own proposal.
|
||||
- **Fuzzy/near-identical detection** (normalized paths, similar-but-not-equal arguments) — rejected: exact match after canonicalization is cheap, deterministic, and explainable to the model; similarity thresholds invite false positives and need evidence before they earn complexity.
|
||||
- **Placing the package in `core/`** — rejected: core is the product spine; a behavioral guard is an optional leaf plugin, and the `todo/` precedent is a small dedicated group per plugin family.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `packages/guard/repeat-tool-guard/` exists, registers all listeners through `ctx.effect()`, and is loadable from a `cordis.yml` with the config above; the config catalog regenerates with its entry.
|
||||
- Invalid `thresholds` (empty, non-integer, `< 2`, duplicate) throw at plugin load.
|
||||
- Unit suite covers the semantics list above at per-file 100%; a snapshot scenario replays a threshold-crossing repetition and pins the reminder `context/message` in the transcript on macOS and Linux.
|
||||
- The reminder is reconstructable from the session log alone (it is an ordinary `context/message` with a plugin source — no new session event).
|
||||
- The package README opens with the plugin's purpose — an advisory loop-breaker that is not a model-facing tool, never blocks or rewrites a call, and only injects reminders — then documents the transparency rule, the per-agent keying, and the in-memory-only state; `doc-sync` is green.
|
||||
|
||||
## Risks
|
||||
|
||||
- **False positives on legitimately repeated calls.** Idempotent polling patterns repeat identical calls on purpose; the reminder is advisory and thresholds/`exclude` are the pressure valves, but a badly tuned deployment adds noise to the transcript. Mitigation: conservative defaults and the reminder text explicitly allowing "finish the task if enough evidence has been gathered".
|
||||
- **Reminder tokens are model-visible cost.** Each trigger appends a paragraph to the next request; thresholds bound the frequency, but a pathological agent can hit 3/5/8 repeatedly across different keys.
|
||||
- **State is in-memory only.** A session resumed from persistence starts with a fresh chain, so a loop spanning a resume gets its reminders later than a live one — accepted: the guard is a heuristic nudge, not a logged invariant, and persisting counter state would buy little for real complexity.
|
||||
- **Multiple context producers on one call.** When a hook bridge and the guard both attach `additionalContext`, ordering follows listener registration order; the fold keeps both, but the combined envelope's readability depends on merge behavior that this RFC inherits rather than owns.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Should compaction reset chains? A compacted history changes what the model sees, but the repetition risk usually survives compaction; the initial answer is no.
|
||||
- Should subagents inherit the parent's thresholds via config only, or ever share chain state? Per-agent isolation is the proposed default; sharing looks like a smell until a concrete case appears.
|
||||
@@ -30,4 +30,4 @@ An e2e assertion re-runs the command or re-reads the file externally; a keyword
|
||||
|
||||
## When a snapshot test is required
|
||||
|
||||
Any change affecting the editor-facing transcript or end-to-end agent UX — the ACP bridge, the loop's observable output, tool presentation — adds or updates a scenario under `examples/acp-agent/tests/snapshots/` (or states in the PR why none applies). New capability seams, lifecycle shapes, or transcript surfaces name their coverage at every tier at plan time and verify the harness can express it — a harness gap is scheduled work, not a mid-build surprise.
|
||||
Any change affecting the editor-facing transcript or end-to-end agent UX — the ACP bridge, the loop's observable output, tool presentation — adds or updates a scenario in the owning example's snapshot suite (`examples/<name>/tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/support/acp-snapshot/README.md) suite factory; `examples/acp-agent` is the primary suite), or states in the PR why none applies. New capability seams, lifecycle shapes, or transcript surfaces name their coverage at every tier at plan time and verify the harness can express it — a harness gap is scheduled work, not a mid-build surprise.
|
||||
|
||||
Reference in New Issue
Block a user