docs: trim generated prose
This commit is contained in:
Tianyi Cui
@@ -1,19 +1,19 @@
|
||||
# AGENTS.md — Examples
|
||||
|
||||
Runnable demos showing how the harness is wired. **Examples are NOT workspaces** — each `examples/*/package.json` is a private, dependency-free stub, never built. They are booted as unbuilt `tsx` subprocesses via the cordis Loader reading a `cordis.yml`; the `@deepseek-ai/dsh-*` plugin names in those YAML files resolve through the root `tsconfig.json` `paths` map, not through `node_modules`.
|
||||
Runnable harness compositions. **Examples are not workspaces:** their private package stubs are not built; `tsx` and the Cordis Loader resolve package names through the root `tsconfig.json` paths.
|
||||
|
||||
Because examples are not under the `packages/*/src` coverage gate, an example that grows real, reusable *logic* should extract it into a `packages/` package (where it gets the per-file 100% gate and a README). Keep only example-specific glue here: the `cordis.yml` wiring, demo-only mocks/teaching artifacts, and the e2e/snapshot scenarios. There is no `start.ts` — the boot glue lives in each app package's `bin` (`@deepseek-ai/dsh-stdio-agent`, `@deepseek-ai/dsh-acp-agent`), which the `demo:*` scripts invoke against the leaf `cordis.yml`.
|
||||
Keep only wiring, demo-only fixtures, and e2e/snapshot scenarios here. Move reusable logic into `packages/`, where coverage and README requirements apply. App-package bins own bootstrapping; examples have no `start.ts`.
|
||||
|
||||
## Every example ships e2e smokes (keyless + with-key)
|
||||
|
||||
Each example must have **both** kinds of end-to-end smoke, because they catch different failures:
|
||||
Each example has both smoke tiers:
|
||||
|
||||
- **Keyless smoke** — boot the example through its real `cordis.yml` via the Loader (no API key), drive it, and assert the rendered output and a clean exit. This is the guard a hand-mounted unit test structurally cannot be: it exercises the REAL load path (`unwrapExports`, `inject`, the whole plugin tree), so a broken plugin export shape — e.g. a stray `export default` that collapses a namespace plugin and drops `inject` — fails here even when unit tests stay green (see [docs/postmortem/0001](../docs/postmortem/0001-acp-default-export-drops-inject.md)). It runs in the default e2e gate (CI has no secrets).
|
||||
- **With-key smoke** — send a real prompt against the live model and verify the WORLD (a file on disk, a non-empty assistant turn), not the agent's self-report. This proves the actual product works, which a mock/keyless run structurally cannot. Key-gated: it self-skips without `DEEPSEEK_API_KEY` (see [the testing policy](../docs/testing.md) — inference is cheap here, so write many).
|
||||
- **Keyless:** boot the real `cordis.yml` through the Loader, drive it, and assert output plus clean exit. This catches Loader/export-shape failures that hand-mounted tests miss ([postmortem](../docs/postmortem/0001-acp-default-export-drops-inject.md)).
|
||||
- **With-key:** send a live-model prompt and verify external state, not the model's claim. Self-skip without `DEEPSEEK_API_KEY`; see [testing.md](../docs/testing.md).
|
||||
|
||||
**Exception — keyless-by-nature examples.** An example whose model is itself a mock/deterministic stand-in (no real provider) has no meaningful with-key smoke; the keyless smoke is the complete requirement. State the exception inline in the test.
|
||||
Mock-only examples need only the keyless tier; state the exception in the test.
|
||||
|
||||
A keyless smoke that spawns the example from a temp cwd must set `TSX_TSCONFIG_PATH` to the repo-root tsconfig (the unbuilt `paths` map is found by searching UP from cwd), and pass `--expose-internals` when the `cordis.yml` loads the HMR plugin (mirror the `demo:*` script).
|
||||
A keyless smoke launched from a temporary cwd sets `TSX_TSCONFIG_PATH` to the root tsconfig and passes `--expose-internals` when loading HMR.
|
||||
|
||||
## Current state
|
||||
|
||||
|
||||
@@ -33,7 +33,7 @@ The editor sets each session's `cwd` to the project it opens; both the agent's b
|
||||
|
||||
## Snapshot tests (record-once / replay-deterministic)
|
||||
|
||||
This example is the home of the harness's **snapshot tests** — they boot this server as a real subprocess, drive it with a deterministic input script, and diff its normalized output against committed golden files. The model is made deterministic by `@deepseek-ai/dsh-llm-replay`, a function/namespace plugin that installs an `llm/stream` waterfall listener and short-circuits it, serving model streams reconstructed from a recorded **session JSONL** fixture (`<scenario>/session.jsonl`) — so replay needs no API key. The fixture IS the persisted session log: its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`"; use `pnpm run test:snapshot:record` when the model transcript itself should change, and `pnpm run test:snapshot:refresh` when the committed model transcript is still the right mock input and only the current replay output/goldens need to be rewritten. The two failure modes not expressible as logged chunks — a pure throw before any chunk, and cancel/hang — use an optional `<scenario>/replay.override.json` sidecar (a `ReplayEntry[]` that replaces the derived script). A scenario that needs the agent to operate on existing files ships an optional `<scenario>/workspace/` directory — the harness copies its contents into the temp cwd before the run (see `workspace-edit`). See [the ACP snapshot tests RFC](../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md) for the full design.
|
||||
This example hosts the ACP snapshot suite. `dsh-llm-replay` reconstructs model streams from `assistant/chunk` events in each scenario's session JSONL, so replay is keyless. Recording runs the real agent and harvests that log; refresh keeps the committed transcript as mock input and rewrites current replay outputs. `replay.override.json` covers throw and hang cases that chunks cannot express, and an optional `workspace/` seeds files. The [snapshot RFC](../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md) owns the full design.
|
||||
|
||||
## MVP limitations
|
||||
|
||||
|
||||
@@ -26,27 +26,12 @@ import {
|
||||
* WITHOUT a key, since it only needs the server to boot and answer initialize.
|
||||
*/
|
||||
|
||||
// The dsh-acp-agent bin (the demo:acp entry) and this example's cordis.yml. The
|
||||
// bin resolves its config-path arg from CWD; the subprocess runs from a temp
|
||||
// workdir, so pass the example config's ABSOLUTE path.
|
||||
// The dsh-acp-agent bin (the demo:acp entry) and this example's cordis.yml.
|
||||
const binScript = fileURLToPath(new URL('../../../packages/ui/acp-agent/src/bin.ts', import.meta.url))
|
||||
const configPath = fileURLToPath(new URL('../cordis.yml', import.meta.url))
|
||||
// Resolve tsx's loader to an ABSOLUTE path: the subprocess runs with cwd set to
|
||||
// a temp workdir (this test launches there and uses it as the session cwd; the
|
||||
// bridge no longer requires cwd === the launch dir, but a temp dir keeps the
|
||||
// test hermetic), where a bare `--import tsx` would not resolve from
|
||||
// node_modules. import.meta.resolve gives the worktree's tsx regardless of cwd.
|
||||
// Resolve tsx absolutely because the subprocess runs outside the repo.
|
||||
const tsxLoader = fileURLToPath(import.meta.resolve('tsx'))
|
||||
// Absolute path to the repo-root tsconfig. Dev/test/demo run UNBUILT: the
|
||||
// `@deepseek-ai/dsh-*` workspace imports resolve through the `paths` map in the
|
||||
// root tsconfig (tsx reads it), NOT through built `lib/` output. But tsx finds
|
||||
// that tsconfig by searching UP from the child's cwd — and the child's cwd is a
|
||||
// temp workdir OUTSIDE the repo, so the search misses and the dsh-* imports fail
|
||||
// (the child dies before writing a byte). Point tsx at the repo tsconfig
|
||||
// explicitly via TSX_TSCONFIG_PATH so resolution is cwd-independent. (Without
|
||||
// this the suite only passed by accident when a stale built `lib/` happened to
|
||||
// exist — exactly the contamination that masked the inject bug this suite now
|
||||
// guards.) The repo root is four levels up from this file (examples/acp-agent/tests).
|
||||
// Absolute path to the repo-root tsconfig.
|
||||
const repoTsconfig = fileURLToPath(new URL('../../../tsconfig.json', import.meta.url))
|
||||
|
||||
interface Spawned {
|
||||
@@ -155,9 +140,6 @@ describe('acp-agent over real stdio (no key required)', () => {
|
||||
it('emits only framed JSON-RPC on stdout', async () => {
|
||||
workdir = await mkdtemp(join(tmpdir(), 'acp-e2e-'))
|
||||
// Collect raw stdout bytes directly (bypass the SDK framing) to inspect.
|
||||
// A dummy key lets the deepseek adapter APPLY (it only checks the key is
|
||||
// present at boot, not valid — the key is used only on a real model call,
|
||||
// which this purity test never triggers). So this runs WITHOUT real creds.
|
||||
const child = spawn(process.execPath, ['--import', tsxLoader, binScript, configPath], {
|
||||
cwd: workdir,
|
||||
env: {
|
||||
@@ -196,17 +178,10 @@ describe('acp-agent over real stdio (no key required)', () => {
|
||||
}, 30_000)
|
||||
|
||||
it('session/new succeeds over real stdio (no model call)', async () => {
|
||||
// REGRESSION GUARD (this exact RPC crashed a real Zed session with
|
||||
// "cannot get property \"agents\" without inject"): `session/new` drives the
|
||||
// full bridge → `ctx.agents.create({sessionId, meta:{cwd}})` → AgentLoop →
|
||||
// registry/persistence path, ALL of which run from the JSON-RPC read loop
|
||||
// OUTSIDE the bridge plugin's injection scope. A lazy `ctx.<service>` read
|
||||
// on that path throws and the RPC fails with an Internal error — yet the
|
||||
// call never touches the model, so this reproduces WITHOUT a key. The
|
||||
// key-gated prompt test below never caught it (it needs real creds); the
|
||||
// initialize-only purity test never caught it (initialize does not reach
|
||||
// the factory). This closes that gap: boot the real subprocess and create a
|
||||
// session, asserting the RPC RESOLVES (not rejects with an inject error).
|
||||
// Regression guard (this exact RPC crashed a real Zed session with "cannot get property
|
||||
// \"agents\" without inject"): `session/new` drives the full bridge →
|
||||
// `ctx.agents.create({sessionId, meta:{cwd}})` → AgentLoop → registry/persistence path, ALL
|
||||
// of which run from the JSON-RPC read loop outside the bridge plugin's injection scope.
|
||||
workdir = await mkdtemp(join(tmpdir(), 'acp-e2e-'))
|
||||
// A dummy key lets the deepseek adapter boot (it only checks presence, not
|
||||
// validity, at apply time); no model call is made, so the key is never used.
|
||||
@@ -245,12 +220,9 @@ describe.skipIf(!process.env.DEEPSEEK_API_KEY)('acp-agent e2e: real prompt over
|
||||
const toolCalls = updates.filter(u => u.sessionUpdate === 'tool_call')
|
||||
expect(toolCalls.length).toBeGreaterThan(0)
|
||||
|
||||
// Tool-call UI quality (the tool owns its presentation): the bash tool's
|
||||
// `presentCall` sets the title to the exact command (an execute card hides
|
||||
// rawInput, so the command IS the title) — NOT the bare tool name "bash".
|
||||
// A `bash` call must therefore carry an execute kind, a non-"bash" title,
|
||||
// and a string rawInput (the command). `toolCalls` is already narrowed to
|
||||
// the `tool_call` shape by the filter above, so these fields are reachable.
|
||||
// Tool-call UI quality (the tool owns its presentation): the bash tool's `presentCall` sets
|
||||
// the title to the exact command (an execute card hides rawInput, so the command IS the
|
||||
// title) — not the bare tool name "bash".
|
||||
const bashCall = toolCalls.find(u => u.kind === 'execute')
|
||||
expect(bashCall).toBeDefined()
|
||||
if (bashCall === undefined) throw new Error('expected an execute tool_call')
|
||||
|
||||
@@ -75,31 +75,12 @@ const SCENARIOS: Scenario[] = [
|
||||
// child runs as a spawn subagent under the worker-thread engine (its session is the
|
||||
// child fixture), and the tool result carries the script's return value.
|
||||
{ name: 'workflow-run', hasModelTurn: true, recorded: true, childSessions: 1 },
|
||||
// Hook matrix — one scenario per hook point × its headline Decision outcome,
|
||||
// across BOTH bridges (Claude `hooks.json`, Codex `codex-hooks.json`, seeded in
|
||||
// workspace/). The block scenarios need no model call: a UserPromptSubmit hook
|
||||
// blocks the prompt before any step runs (keyless, authored — the derived
|
||||
// script is empty so no sidecar), yet persists a `rejected` turn carrying
|
||||
// `hook/*` events, so their logs ARE compared. Every other point fires a real
|
||||
// seam mid-turn, so its transcript is recorded WITH the hook active.
|
||||
// Hook matrix — one scenario per hook point × its headline Decision outcome, across BOTH
|
||||
// bridges (Claude `hooks.json`, Codex `codex-hooks.json`, seeded in workspace/).
|
||||
{ name: 'hook-cc-promptsubmit-block', hasModelTurn: false, comparesLog: true, recorded: false },
|
||||
{ name: 'hook-codex-promptsubmit-block', hasModelTurn: false, comparesLog: true, recorded: false },
|
||||
// The mid-turn seams fire during a real model turn, so each is recorded WITH
|
||||
// its hook active (the model's reaction to a deny/block/force-continue is part
|
||||
// of the captured transcript). The Codex bridge exercises the same seams in its
|
||||
// own snake_case dialect.
|
||||
//
|
||||
// Two hook points are deliberately NOT snapshotted, and stay on the bridges'
|
||||
// unit coverage (`bridge.spec.ts` / `coverage.spec.ts`) instead:
|
||||
// - SessionStart and SubagentStart inject context through a detached,
|
||||
// best-effort `void runPoint(...).then(agent.inject())` with no turn
|
||||
// binding, so the resulting `context/message` races the work it precedes
|
||||
// and lands at a nondeterministic log position — a recorded golden does not
|
||||
// even reproduce on its own replay.
|
||||
// - SubagentStop is observe-only with no turn and no injection, so it writes
|
||||
// NOTHING to the transcript — a golden would be byte-identical to the
|
||||
// no-hook run and could never be proven to fail.
|
||||
// See the hook-snapshot-matrix RFC for the full rationale.
|
||||
// The mid-turn seams fire during a real model turn, so each is recorded with its hook active
|
||||
// (the model's reaction to a deny/block/force-continue is part of the captured transcript).
|
||||
{ name: 'hook-cc-promptsubmit-context', hasModelTurn: true, recorded: true },
|
||||
{ name: 'hook-cc-pretool-deny', hasModelTurn: true, recorded: true },
|
||||
{ name: 'hook-cc-pretool-ask', hasModelTurn: true, recorded: true },
|
||||
@@ -114,11 +95,9 @@ const SCENARIOS: Scenario[] = [
|
||||
{ name: 'hook-codex-posttool-block', hasModelTurn: true, recorded: true },
|
||||
{ name: 'hook-codex-posttool-context', hasModelTurn: true, recorded: true },
|
||||
{ name: 'hook-codex-stop-continue', hasModelTurn: true, recorded: true },
|
||||
// Code Mode: the registry in `mode: code` — the wire tool list collapses to
|
||||
// [run_code], the tools:sdk section rides in the prompt, and the program's
|
||||
// tool calls land as tool/code-dispatch events. Each mode boots its own
|
||||
// overlay config, composes a different header by construction, and
|
||||
// therefore pins its own class.
|
||||
// Code Mode: the registry in `mode: code` — the wire tool list collapses to [run_code], the
|
||||
// tools:sdk section rides in the prompt, and the program's tool calls land as
|
||||
// tool/code-dispatch events.
|
||||
{ name: 'code-mode-turn', hasModelTurn: true, recorded: true, pinsHeader: true, headerClass: 'code', configPath: CODE_MODE_CONFIG },
|
||||
{ name: 'both-mode-turn', hasModelTurn: true, recorded: true, pinsHeader: true, headerClass: 'both', configPath: BOTH_MODE_CONFIG },
|
||||
]
|
||||
|
||||
@@ -17,21 +17,8 @@ import {
|
||||
} from '@agentclientprotocol/sdk'
|
||||
|
||||
/**
|
||||
* With-key e2e: the Claude Code hook bridge running against the REAL acp-agent
|
||||
* subprocess and the REAL model. The example `cordis.yml` loads `dsh-hooks-claude`
|
||||
* with a PROCESS-LEVEL `configPath` of `./hooks.json`, resolved once at load
|
||||
* against the ACP server's launch cwd (NOT per-session); this test sets that
|
||||
* launch cwd to the temp workspace and writes a `hooks.json` there with a
|
||||
* PreToolUse hook that BLOCKS every bash command, then asks the live model to
|
||||
* write a file — and verifies the WORLD (the file never appears on disk),
|
||||
* proving the hook actually intercepted execution rather than the agent merely
|
||||
* claiming it couldn't. (The hook itself then runs in the session cwd.)
|
||||
* Key-gated; owns and disposes its subprocess.
|
||||
*
|
||||
* A keyless companion lives in acp.e2e.ts (stdout purity + session/new); the
|
||||
* full hook-fires-end-to-end transcript is the keyless `hook-cc-promptsubmit-block`
|
||||
* snapshot scenario. This one closes the "green plumbing, broken product" gap:
|
||||
* only a real model deciding to call bash exercises the PreToolUse seam live.
|
||||
* With-key e2e: the Claude Code hook bridge running against the real acp-agent subprocess and
|
||||
* the real model.
|
||||
*/
|
||||
|
||||
const binScript = fileURLToPath(new URL('../../../packages/ui/acp-agent/src/bin.ts', import.meta.url))
|
||||
@@ -89,9 +76,7 @@ afterEach(async () => {
|
||||
describe.skipIf(!process.env.DEEPSEEK_API_KEY)('acp-agent e2e: a PreToolUse hook blocks bash (real model)', () => {
|
||||
it('denies every bash command, so the requested file is never written (verified on disk)', async () => {
|
||||
workdir = await mkdtemp(join(tmpdir(), 'acp-hooks-e2e-'))
|
||||
// A PreToolUse hook that blocks EVERY tool (exit 2, no matcher = match-all).
|
||||
// The session cwd is `workdir`, and the bridge resolves `./hooks.json` from
|
||||
// the process cwd (the launch dir = workdir), so this is the config it loads.
|
||||
// A PreToolUse hook that blocks every tool (exit 2, no matcher = match-all).
|
||||
await writeFile(join(workdir, 'hooks.json'), JSON.stringify({
|
||||
hooks: { PreToolUse: [{ hooks: [{ type: 'command', command: 'echo "bash blocked by policy" >&2; exit 2' }] }] },
|
||||
}))
|
||||
|
||||
@@ -6,17 +6,11 @@ import { fileURLToPath } from 'node:url'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
|
||||
/**
|
||||
* Keyless Loader-path smoke for the Code Mode overlay: boot the REAL
|
||||
* example through the `@deepseek-ai/dsh-stdio-agent` bin against
|
||||
* `code-mode.cordis.yml` (the cordis Loader, `unwrapExports`, the include
|
||||
* patches over ./cordis.yml, the worker-thread code runtime, and the
|
||||
* registry in `mode: code`), then close stdin with no prompt and assert
|
||||
* the Code Mode banner + a clean exit.
|
||||
*
|
||||
* No prompt is ever sent, so the model is NEVER called and no `run_code`
|
||||
* turn happens — a dummy key lets `llm-deepseek`'s key-PRESENT check boot
|
||||
* the tree. This is the export-shape guard (postmortem 0001) for the Code
|
||||
* Mode composition; the with-key proof lives in `code-mode.e2e.ts`.
|
||||
* Keyless Loader-path smoke for the Code Mode overlay: boot the real example through the
|
||||
* `@deepseek-ai/dsh-stdio-agent` bin against `code-mode.cordis.yml` (the cordis Loader,
|
||||
* `unwrapExports`, the include patches over ./cordis.yml, the worker-thread code runtime, and
|
||||
* the registry in `mode: code`), then close stdin with no prompt and assert the Code Mode
|
||||
* banner + a clean exit.
|
||||
*/
|
||||
|
||||
const binScript = fileURLToPath(new URL('../../../packages/ui/stdio-agent/src/bin.ts', import.meta.url))
|
||||
@@ -26,10 +20,7 @@ const tsxLoader = fileURLToPath(import.meta.resolve('tsx'))
|
||||
// `paths` map; tsx searches UP from cwd, and we spawn from a temp dir outside
|
||||
// the repo, so point it at the repo tsconfig.
|
||||
const repoTsconfig = fileURLToPath(new URL('../../../tsconfig.json', import.meta.url))
|
||||
// The real-API workflow runs up to 14 e2e files at once. Cold tsx/Loader
|
||||
// startup can therefore outlive a tight smoke-test deadline before the child
|
||||
// emits any output; 30s still detects a wedged process without confusing
|
||||
// bounded CI contention with a lifecycle failure.
|
||||
// The real-API workflow runs up to 14 e2e files at once.
|
||||
const PROCESS_TIMEOUT_MS = 30_000
|
||||
// Leave enough room for the process-owned timeout to report captured output
|
||||
// before Vitest aborts the test itself.
|
||||
|
||||
@@ -6,26 +6,8 @@ import type { Context } from 'cordis'
|
||||
import { AgentId } from '@deepseek-ai/dsh-agent'
|
||||
import { codingHarness, finalText, SYSTEM_PROMPT, waitForIdle } from './harness.ts'
|
||||
|
||||
/**
|
||||
* The compaction smoke test: a real model runs a multi-step bash task with a
|
||||
* deliberately tiny context window, so the auto-compaction listener fires
|
||||
* MID-SESSION and summarizes the older history into a checkpoint. This is the
|
||||
* first end-to-end exercise of the compaction seam (it is wired nowhere else),
|
||||
* and the runaway-survival regression net — it proves a session that grows past
|
||||
* the window keeps running rather than overflowing. Key-gated.
|
||||
*
|
||||
* Verifies the WORLD, not the agent's self-report: a compact/start…end pair
|
||||
* landed in the real session log, the surface actually shrank (a replace node
|
||||
* exists and shadowed older nodes), and the agent still produced a final answer
|
||||
* after compaction (so the summarized history did not break the conversation).
|
||||
*
|
||||
* FIXME(compaction-snapshot): this key-gated e2e is the ONLY coverage of runaway
|
||||
* compaction — there is no keyless full-transcript snapshot of it. dsh-llm-replay
|
||||
* reconstructs one model call per (turn, step) from `assistant/chunk` events, but
|
||||
* `summarize()` assembles its stream into a local BlockAssembler and appends no
|
||||
* `assistant/chunk`, so the interleaved summarization call is unreplayable. A
|
||||
* snapshot needs replay-harness work to serve that call; deferred as a follow-up.
|
||||
*/
|
||||
/** Key-gated smoke for mid-session compaction and continued agent progress. */
|
||||
// FIXME(compaction-snapshot): replay cannot serve the unlogged summarization model call.
|
||||
|
||||
let workdir: string | undefined
|
||||
let ctx: Context | undefined
|
||||
@@ -40,18 +22,11 @@ afterEach(async () => {
|
||||
describe.skipIf(!process.env.DEEPSEEK_API_KEY)('compaction: a long session compacts mid-flight and keeps running', () => {
|
||||
it('summarizes older history into a checkpoint without breaking the task', async () => {
|
||||
workdir = await mkdtemp(join(tmpdir(), 'dsh-compaction-'))
|
||||
// A handful of files for the model to read, so multiple bash steps
|
||||
// accumulate surface nodes (tool calls + results) and grow the history past
|
||||
// the (deliberately tiny) window.
|
||||
for (let i = 1; i <= 4; i++) {
|
||||
await writeFile(join(workdir, `file${i}.txt`), `This is file number ${i}. `.repeat(50))
|
||||
}
|
||||
|
||||
// Tiny window so a couple of steps crosses the threshold. The generation
|
||||
// cap is deliberately larger than the final checkpoint because
|
||||
// reasoning-capable APIs count reasoning tokens against the provider output
|
||||
// budget even though those blocks are stripped before the checkpoint is
|
||||
// stored.
|
||||
// Reasoning tokens require a larger generation cap than the retained checkpoint.
|
||||
ctx = await codingHarness(workdir, {
|
||||
persona: SYSTEM_PROMPT,
|
||||
compact: {
|
||||
|
||||
@@ -6,28 +6,14 @@ import { fileURLToPath } from 'node:url'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
|
||||
/**
|
||||
* Keyless Loader-path smoke for examples/coding-agent: boot the REAL example
|
||||
* through the `@deepseek-ai/dsh-stdio-agent` bin against its `cordis.yml` (the
|
||||
* cordis Loader, `unwrapExports`, the full plugin tree incl. the
|
||||
* `@deepseek-ai/dsh-agent-core` bundle and the app's in-package readline UI
|
||||
* module), then close stdin with no prompt and assert the
|
||||
* ready banner + a clean exit.
|
||||
*
|
||||
* No prompt is ever sent, so the model is NEVER called — this is why it runs
|
||||
* without a real key. coding-agent's `cordis.yml` loads `llm-deepseek`, whose
|
||||
* `apply()` only requires a key to be PRESENT (it does not validate it and only
|
||||
* uses it when a stream actually starts), so a dummy key lets the tree boot
|
||||
* while the absence of any prompt guarantees no network call. The value is the
|
||||
* real-Loader-path guard that the composed tree boots (see postmortem 0001;
|
||||
* the app carries no `inject`, so its export SHAPE is pinned by the stdio-agent
|
||||
* unit suite's unwrap assertion, not by a crash here),
|
||||
* complementing coding-agent's with-key e2e suites which prove the real
|
||||
* product.
|
||||
* Keyless Loader-path smoke for examples/coding-agent: boot the real example through the
|
||||
* `@deepseek-ai/dsh-stdio-agent` bin against its `cordis.yml` (the cordis Loader,
|
||||
* `unwrapExports`, the full plugin tree incl. the `@deepseek-ai/dsh-agent-core` bundle and the
|
||||
* app's in-package readline UI module), then close stdin with no prompt and assert the ready
|
||||
* banner + a clean exit.
|
||||
*/
|
||||
|
||||
// The dsh-stdio-agent bin (the demo:repl entry) and this example's cordis.yml.
|
||||
// The bin resolves its config-path arg from CWD; the test spawns from a temp
|
||||
// cwd, so we pass the example config's ABSOLUTE path.
|
||||
const binScript = fileURLToPath(new URL('../../../packages/ui/stdio-agent/src/bin.ts', import.meta.url))
|
||||
const configPath = fileURLToPath(new URL('../cordis.yml', import.meta.url))
|
||||
const tsxLoader = fileURLToPath(import.meta.resolve('tsx'))
|
||||
@@ -35,10 +21,7 @@ const tsxLoader = fileURLToPath(import.meta.resolve('tsx'))
|
||||
// `paths` map; tsx searches UP from cwd, and we spawn from a temp dir outside
|
||||
// the repo, so point it at the repo tsconfig (root is four levels up).
|
||||
const repoTsconfig = fileURLToPath(new URL('../../../tsconfig.json', import.meta.url))
|
||||
// The real-API workflow runs up to 14 e2e files at once. Cold tsx/Loader
|
||||
// startup can therefore outlive a tight smoke-test deadline before the child
|
||||
// emits any output; 30s still detects a wedged process without confusing
|
||||
// bounded CI contention with a lifecycle failure.
|
||||
// The real-API workflow runs up to 14 e2e files at once.
|
||||
const PROCESS_TIMEOUT_MS = 30_000
|
||||
// Leave enough room for the process-owned timeout to report captured output
|
||||
// before Vitest aborts the test itself.
|
||||
|
||||
@@ -78,10 +78,8 @@ describe.skipIf(!process.env.DEEPSEEK_API_KEY)('cordis tools: a real model modif
|
||||
}])
|
||||
await waitForIdle(ctx, agent)
|
||||
|
||||
// World checks: the tool exists in the registry, was invoked as a real
|
||||
// tool call, and its RESULT (the self-made execute actually running) is the
|
||||
// reversed string. The model's prose is not asserted — the tool result is
|
||||
// the world; the summary sentence is just the self-report.
|
||||
// World checks: the tool exists in the registry, was invoked as a real tool call, and its
|
||||
// RESULT (the self-made execute actually running) is the reversed string.
|
||||
expect(ctx.tools.get('reverse_text')).toBeDefined()
|
||||
const events = [...agent.session.events]
|
||||
const calls = events.filter(event => event.type === 'tool/call')
|
||||
|
||||
@@ -6,22 +6,15 @@ import { fileURLToPath } from 'node:url'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
|
||||
/**
|
||||
* Keyless Loader-path smoke for examples/cordis-agent: boot the REAL example
|
||||
* through the `@deepseek-ai/dsh-stdio-agent` bin against its `cordis.yml` —
|
||||
* the cordis Loader, `unwrapExports`, the full plugin tree INCLUDING the
|
||||
* `@deepseek-ai/dsh-tool-cordis` package resolved by name (whose `inject`
|
||||
* would crash a collapsed export shape at load, see docs/postmortem/0001) —
|
||||
* then close stdin with no prompt and assert the ready banner + a clean exit.
|
||||
*
|
||||
* No prompt is ever sent, so the model is NEVER called — that is why it runs
|
||||
* without a real key: `llm-deepseek`'s apply() only requires a key to be
|
||||
* PRESENT, and the absence of any prompt guarantees no network call. The
|
||||
* with-key product proof lives in cordis-tools.e2e.ts.
|
||||
* Keyless Loader-path smoke for examples/cordis-agent: boot the real example through the
|
||||
* `@deepseek-ai/dsh-stdio-agent` bin against its `cordis.yml` — the cordis Loader,
|
||||
* `unwrapExports`, the full plugin tree INCLUDING the `@deepseek-ai/dsh-tool-cordis` package
|
||||
* resolved by name (whose `inject` would crash a collapsed export shape at load, see
|
||||
* docs/postmortem/0001) — then close stdin with no prompt and assert the ready banner + a
|
||||
* clean exit.
|
||||
*/
|
||||
|
||||
// The dsh-stdio-agent bin (the demo:cordis entry) and this example's cordis.yml.
|
||||
// The bin resolves its config-path arg from CWD; the test spawns from a temp
|
||||
// cwd, so we pass the example config's ABSOLUTE path.
|
||||
const binScript = fileURLToPath(new URL('../../../packages/ui/stdio-agent/src/bin.ts', import.meta.url))
|
||||
const configPath = fileURLToPath(new URL('../cordis.yml', import.meta.url))
|
||||
const tsxLoader = fileURLToPath(import.meta.resolve('tsx'))
|
||||
@@ -29,10 +22,7 @@ const tsxLoader = fileURLToPath(import.meta.resolve('tsx'))
|
||||
// `paths` map; tsx searches UP from cwd, and we spawn from a temp dir outside
|
||||
// the repo, so point it at the repo tsconfig (root is three levels up).
|
||||
const repoTsconfig = fileURLToPath(new URL('../../../tsconfig.json', import.meta.url))
|
||||
// The real-API workflow runs up to 14 e2e files at once. Cold tsx/Loader
|
||||
// startup can therefore outlive a tight smoke-test deadline before the child
|
||||
// emits any output; 30s still detects a wedged process without confusing
|
||||
// bounded CI contention with a lifecycle failure.
|
||||
// The real-API workflow runs up to 14 e2e files at once.
|
||||
const PROCESS_TIMEOUT_MS = 30_000
|
||||
// Leave enough room for the process-owned timeout to report captured output
|
||||
// before Vitest aborts the test itself.
|
||||
|
||||
@@ -6,41 +6,20 @@ import { fileURLToPath } from 'node:url'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
|
||||
/**
|
||||
* Keyless Loader-path smoke for examples/echo-agent: boot the REAL example
|
||||
* through the `@deepseek-ai/dsh-stdio-agent` bin against this example's
|
||||
* `cordis.yml` (the cordis Loader, `unwrapExports`, the whole plugin tree),
|
||||
* pipe a script of stdin lines, and assert the rendered stdout.
|
||||
*
|
||||
* This is the guard the per-file unit suite structurally cannot be: it drives
|
||||
* the `@deepseek-ai/dsh-stdio-agent` app plugin, the `@deepseek-ai/dsh-agent-core`
|
||||
* bundle it loads, the app's in-package readline UI module, AND the
|
||||
* example-local `mock-llm.ts` / `echo-tool.ts` through their REAL load path
|
||||
* (see docs/postmortem/0001). The app itself carries no `inject`, so a stray
|
||||
* `export default` would boot rather than crash here — the export SHAPE is
|
||||
* pinned by the explicit unwrap assertion in the stdio-agent unit suite; this
|
||||
* smoke proves the composed tree actually runs. It needs no API key — the
|
||||
* `mock-echo` adapter never touches the network — so it runs in the default e2e
|
||||
* gate.
|
||||
*
|
||||
* Both branches of mock-llm.ts are exercised: an `echo …` line (the tool
|
||||
* round-trip → `ECHO: …`) and a plain line (the direct canned reply).
|
||||
* Keyless Loader-path smoke for examples/echo-agent: boot the real example through the
|
||||
* `@deepseek-ai/dsh-stdio-agent` bin against this example's `cordis.yml` (the cordis Loader,
|
||||
* `unwrapExports`, the whole plugin tree), pipe a script of stdin lines, and assert the
|
||||
* rendered stdout.
|
||||
*/
|
||||
|
||||
// The dsh-stdio-agent bin (the demo:echo entry) and this example's cordis.yml.
|
||||
// The bin resolves its config-path arg from CWD; the test spawns from a temp
|
||||
// cwd, so we pass the example config's ABSOLUTE path.
|
||||
const binScript = fileURLToPath(new URL('../../../packages/ui/stdio-agent/src/bin.ts', import.meta.url))
|
||||
const configPath = fileURLToPath(new URL('../cordis.yml', import.meta.url))
|
||||
const tsxLoader = fileURLToPath(import.meta.resolve('tsx'))
|
||||
// Dev/test run UNBUILT: `@deepseek-ai/dsh-*` imports resolve through the root
|
||||
// tsconfig `paths` map, which tsx finds by searching UP from cwd. We spawn from
|
||||
// a temp cwd OUTSIDE the repo, so point tsx at the repo tsconfig explicitly
|
||||
// (repo root is four levels up from examples/echo-agent/tests).
|
||||
// Dev/test run UNBUILT: `@deepseek-ai/dsh-*` imports resolve through the root tsconfig `paths`
|
||||
// map, which tsx finds by searching UP from cwd.
|
||||
const repoTsconfig = fileURLToPath(new URL('../../../tsconfig.json', import.meta.url))
|
||||
// The real-API workflow runs up to 14 e2e files at once. Cold tsx/Loader
|
||||
// startup can therefore outlive a tight smoke-test deadline before the child
|
||||
// emits any output; 30s still detects a wedged process without confusing
|
||||
// bounded CI contention with a lifecycle failure.
|
||||
// The real-API workflow runs up to 14 e2e files at once.
|
||||
const PROCESS_TIMEOUT_MS = 30_000
|
||||
// Leave enough room for the process-owned timeout to report captured output
|
||||
// before Vitest aborts the test itself.
|
||||
@@ -67,9 +46,8 @@ async function runEcho(lines: string[]): Promise<{ stdout: string; code: number
|
||||
return new Promise((resolve, reject) => {
|
||||
const proc = spawn(
|
||||
process.execPath,
|
||||
// --expose-internals: the example's cordis.yml loads the HMR plugin, which
|
||||
// requires it (mirrors the `demo:echo` script). The whole point is to boot
|
||||
// the example EXACTLY as it really runs, through the bin + Loader.
|
||||
// --expose-internals: the example's cordis.yml loads the HMR plugin, which requires it
|
||||
// (mirrors the `demo:echo` script).
|
||||
['--expose-internals', '--import', tsxLoader, binScript, configPath],
|
||||
{
|
||||
cwd,
|
||||
|
||||
@@ -13,4 +13,4 @@ Zed setup is the same as [acp-agent](../acp-agent/README.md) with this example's
|
||||
- **The write boundary is config-fixed**: an escalated `workspace-write` run may write under the launch directory (`workspaceRoot: process.cwd()`) plus the platform temp area — a per-session root is config-phase future work in the [sandbox RFC](../../docs/rfc/implemented/feature/2026-07-06-sandbox.md).
|
||||
- **No usable runner fails closed per command** (structured `SANDBOX_UNAVAILABLE`), and the filesystem tools stay unloaded for the same reason as `sandbox-agent`: they would bypass the bash sandbox.
|
||||
|
||||
Tests: `tests/escalation.e2e.ts` — keyless, it boots the real `cordis.yml` through the Loader as an ACP subprocess, proves the whole tree (sandbox executor + approval service + bridge) initializes and opens a session, and drives the config options end to end (both advertised with composition currents, switches honored and echoed as complete state, out-of-vocabulary values rejected); with a key and a usable runner, a scripted ACP client plays the human — the real model gets denied, escalates, the client answers `allow-once`, and the retried write must land on disk. `tests/acp.snapshot.ts` (the [shared snapshot kit](../../packages/support/acp-snapshot/) over this composition's `cordis.snapshot.yml` replay overlay) pins four scenarios as committed wire bytes: the keyless config-option exchange, the recorded `mode-switching` arc (the suite's pinned header — both switches, their prompt-section deltas, one "changed by the user" notice per knob, and a confined write landing under the switched mode), and both recorded escalation branches (`session/request_permission` answered allow-once / reject-once). Replay re-executes every recorded bash call under the host's real runner — Seatbelt works out of the box on macOS; on Linux install bubblewrap (or build the Landlock launcher) first, exactly what ci.yml's snapshot lane does. No fixture carries a real denial: denial stderr is backend dialect and would pin a fixture to its recording platform (the rationale comment atop the suite file).
|
||||
`tests/escalation.e2e.ts` boots the real composition keylessly and exercises config-option advertisement, updates, and validation; with a key and usable runner it also world-verifies an allowed escalation. `tests/acp.snapshot.ts` pins config exchange, mode switching, and allowed and rejected approval branches through the shared snapshot kit. Replay executes recorded bash calls on the host runner, so Linux needs bubblewrap or Landlock while macOS uses Seatbelt. Fixtures avoid real denial stderr because that dialect is platform-specific.
|
||||
|
||||
@@ -3,54 +3,20 @@ import { fileURLToPath } from 'node:url'
|
||||
import { defineAcpSnapshotSuite, type Scenario, type SnapshotSuiteOptions } from '@deepseek-ai/dsh-acp-snapshot'
|
||||
|
||||
/**
|
||||
* Snapshot suite for the SANDBOXED composition (`../cordis.yml`, swapped to
|
||||
* the sibling `cordis.snapshot.yml` replay overlay by the bin under
|
||||
* `DSH_SNAPSHOT=replay`). Replay swaps only the MODEL for the recorded
|
||||
* transcript — every bash call re-executes for real under the host's actual
|
||||
* runner (Seatbelt on macOS, bwrap on Linux CI: ci.yml's snapshot lane
|
||||
* installs bubblewrap for exactly this), so the recorded scenarios double as
|
||||
* cross-backend confinement regression: an allowed command a runner change
|
||||
* starts denying fails replay outright. Their commands are limited to
|
||||
* `cat`/`printf` shapes whose bytes are identical across those backends and
|
||||
* across GNU/BSD userlands.
|
||||
*
|
||||
* Deliberately ABSENT: a scenario whose transcript carries a real sandbox
|
||||
* DENIAL. The harness-authored `[sandbox: file access denied …]` marker is
|
||||
* byte-stable, but the denied command's own stderr is the backend's dialect
|
||||
* (bwrap EROFS "Read-only file system", Landlock EACCES "Permission
|
||||
* denied", Seatbelt EPERM "Operation not permitted", GNU vs BSD phrasing on
|
||||
* top), and stderr reaches both compared surfaces — such a fixture replays
|
||||
* only on the platform that recorded it. The denial→marker path stays on
|
||||
* dsh-tool-bash's unit tests and the real-kernel sandbox e2e legs
|
||||
* (.github/workflows/sandbox.yml); the escalation scenarios below sidestep
|
||||
* it by having the USER assert the prior denial, so the recorded model
|
||||
* escalates without a platform-variant denial in the log.
|
||||
* Snapshot suite for the sandboxed composition (`../cordis.yml`, swapped to the sibling
|
||||
* `cordis.snapshot.yml` replay overlay by the bin under `DSH_SNAPSHOT=replay`).
|
||||
*/
|
||||
const SCENARIOS: Scenario[] = [
|
||||
// Protocol-only (keyless, authored): the session config-option surface
|
||||
// this composition adds — both advertised selects on session/new, the
|
||||
// complete refreshed state every session/set_config_option answers with,
|
||||
// and both rejection shapes — as committed wire bytes. No bash runs, so
|
||||
// this one still replays on runner-less hosts.
|
||||
// Protocol-only (keyless, authored): the session config-option surface this composition adds
|
||||
// — both advertised selects on session/new, the complete refreshed state every
|
||||
// session/set_config_option answers with, and both rejection shapes — as committed wire
|
||||
// bytes.
|
||||
{ name: 'config-options', hasModelTurn: false, recorded: false },
|
||||
// The runtime mode-switching arc, and NECESSARILY the pinned-header
|
||||
// scenario: an approval-policy switch rewrites its prompt section, and the
|
||||
// resulting request/header-delta is legal only in the pinning scenario
|
||||
// (the factory's uniformity guard). The pin commits this composition's
|
||||
// full header — persona, tool schemas WITH the escalation fields — plus
|
||||
// the approval delta and its "changed by the user" notice verbatim. The
|
||||
// SANDBOX switch is deliberately silent (no section, no notice — the
|
||||
// sandbox RFC's visibility asymmetry): the recorded arc proves it by
|
||||
// BEHAVIOR, a confined write landing under the switched mode with no
|
||||
// header change.
|
||||
// The runtime mode-switching arc, and NECESSARILY the pinned-header scenario: an
|
||||
// approval-policy switch rewrites its prompt section, and the resulting request/header-delta
|
||||
// is legal only in the pinning scenario (the factory's uniformity guard).
|
||||
{ name: 'mode-switching', hasModelTurn: true, recorded: true, pinsHeader: true, expectedHeaderDeltas: 1 },
|
||||
// The approval wire end-to-end, under the DEFAULT read-only/ask (a switch
|
||||
// would emit a header-delta the uniformity guard forbids here): the
|
||||
// escalating bash call streams, session/request_permission attaches to it
|
||||
// (allow-once / reject-once), and the scripted answer drives each branch —
|
||||
// an approved run executes CONFINED under the granted workspace-write; a
|
||||
// rejected one executes nothing and fails with the deterministic
|
||||
// rejection text.
|
||||
// Pin both approval branches under the default read-only/ask policy.
|
||||
{ name: 'escalation-approved', hasModelTurn: true, recorded: true },
|
||||
{ name: 'escalation-rejected', hasModelTurn: true, recorded: true },
|
||||
]
|
||||
|
||||
@@ -18,21 +18,6 @@ import {
|
||||
|
||||
/**
|
||||
* examples/sandbox-acp-agent end to end.
|
||||
*
|
||||
* Keyless smoke: boot the REAL `cordis.yml` through the `dsh-acp-agent` bin as
|
||||
* an ACP subprocess and drive initialize + session/new — the real-Loader-path
|
||||
* guard (postmortem 0001) for THIS tree's export shapes, which now include the
|
||||
* sandbox executor AND the approval service. No prompt is sent, so neither the
|
||||
* model nor a sandbox runner is ever exercised.
|
||||
*
|
||||
* With-key escalation flow (self-skips without DEEPSEEK_API_KEY or a usable
|
||||
* platform runner): a scripted ACP client plays the human. The real model is
|
||||
* denied under `read-only`, escalates with `sandbox_permissions` +
|
||||
* `justification`, the bridge prompts THIS client over
|
||||
* `session/request_permission`, the client answers `allow-once`, and the
|
||||
* retried write must land ON DISK (world-verified). The session cwd is a temp
|
||||
* dir under the platform temp area, which `workspace-write` grants — so either
|
||||
* escalation target the model picks can land the write.
|
||||
*/
|
||||
|
||||
const binScript = fileURLToPath(new URL('../../../packages/ui/acp-agent/src/bin.ts', import.meta.url))
|
||||
@@ -42,10 +27,8 @@ const tsxLoader = fileURLToPath(import.meta.resolve('tsx'))
|
||||
// tsconfig so the unbuilt `paths` map resolves (see examples/AGENTS.md).
|
||||
const repoTsconfig = fileURLToPath(new URL('../../../tsconfig.json', import.meta.url))
|
||||
|
||||
// A usable confining runner, probed the same way the executor suites do:
|
||||
// bwrap on Linux, Seatbelt's sandbox-exec on macOS. Without one the strict
|
||||
// attempt would fail closed (SANDBOX_UNAVAILABLE) instead of producing the
|
||||
// denial this flow starts from.
|
||||
// A usable confining runner, probed the same way the executor suites do: bwrap on Linux,
|
||||
// Seatbelt's sandbox-exec on macOS.
|
||||
const hasBwrap = spawnSync('bwrap', ['--ro-bind', '/', '/', '--dev', '/dev', '--proc', '/proc', '--die-with-parent', '--', 'true'], {
|
||||
timeout: 5_000,
|
||||
stdio: 'ignore',
|
||||
@@ -121,9 +104,8 @@ describe('sandbox-acp-agent keyless smoke (real cordis.yml via the Loader)', ()
|
||||
workdir = await mkdtemp(join(tmpdir(), 'sandbox-acp-smoke-'))
|
||||
spawned = spawnSandboxAcpAgent(workdir, 'reject-once')
|
||||
const { client } = spawned
|
||||
// A dummy key boots the adapter; no prompt is ever sent, so no model call
|
||||
// and no sandbox runner probe happen. This drives the fiber tree the same
|
||||
// way an editor would, which is what catches a broken export/inject shape.
|
||||
// A dummy key boots the adapter; no prompt is ever sent, so no model call and no sandbox
|
||||
// runner probe happen.
|
||||
const init = await client.initialize({ protocolVersion: PROTOCOL_VERSION, clientCapabilities: {} })
|
||||
expect(init.protocolVersion).toBe(PROTOCOL_VERSION)
|
||||
const { sessionId } = await client.newSession({ cwd: workdir, mcpServers: [] })
|
||||
|
||||
Reference in New Issue
Block a user