Add examples/coding-agent and the docs cookbook
The first real agent wiring: DeepSeek V4 + the bash tool suite + stdio chat + JSONL persistence, runnable via yarn demo:coding (reads the gitignored repo-root .env through process.loadEnvFile). - examples/coding-agent: cordis.yml wiring both real plugin families (llm-deepseek with !!js env secrets; bash-local + tool-bash), a bash-only coding system prompt, a max-steps-guard plugin (bounds runaway turns via the agent/turn-continuation waterfall — abort() from step-end is a no-op by then), and a stdio UI with dimmed reasoning and exit-on-idle for piped stdin. - e2e (yarn test:e2e, key-gated): full-loop.e2e.ts runs a real model against the real bash tool; coding-task.e2e.ts is the swebench-style smoke — the model fixes a buggy add.js in a temp dir and the test re-runs node add.test.js itself rather than trusting the agent. - docs/cookbook: adding-a-package (the verified checklist), adding-a-tool (execute() contract, background pattern, seams), adding-an-llm-adapter (protocol obligations, mock-server testing, e2e policy). AGENTS.md layout/commands/secrets sections updated; architecture.md points at both examples and the cookbook. - vitest.e2e.config.ts: serialize test files + retry twice — parallel e2e files trip the shared internal key's concurrency quota. - fix: the !js YAML tag spelling in docs/JSDoc is actually !!js (js-yaml resolves custom tags under tag:yaml.org,2002:js).
This commit is contained in:
87
AGENTS.md
87
AGENTS.md
@@ -34,9 +34,13 @@ packages/ Harness packages, all named @deepseek-ai/dsh-<name>:
|
|||||||
tool-bash/ model-facing bash/bash_output/bash_kill tool schemas
|
tool-bash/ model-facing bash/bash_output/bash_kill tool schemas
|
||||||
examples/ Runnable demos (not workspaces). echo-agent = mock model + echo
|
examples/ Runnable demos (not workspaces). echo-agent = mock model + echo
|
||||||
tool + stdio UI + JSONL persistence, wired via cordis.yml.
|
tool + stdio UI + JSONL persistence, wired via cordis.yml.
|
||||||
|
coding-agent = the real thing: DeepSeek V4 + bash tools
|
||||||
|
(yarn demo:coding, needs DEEPSEEK_API_KEY).
|
||||||
docs/ architecture.md — the design doc. adr/ — decision records (the
|
docs/ architecture.md — the design doc. adr/ — decision records (the
|
||||||
why behind vendoring, event-sourcing, the schema DSL, …).
|
why behind vendoring, event-sourcing, the schema DSL, …).
|
||||||
rfc/ — proposals for substantial future work.
|
rfc/ — proposals for substantial future work.
|
||||||
|
cookbook/ — step-by-step guides: adding a package, a tool,
|
||||||
|
an LLM adapter.
|
||||||
scripts/ repo maintenance scripts (vendor-manifest guard, publint runner).
|
scripts/ repo maintenance scripts (vendor-manifest guard, publint runner).
|
||||||
JS bundling is tsdown (root tsdown.config.ts + two per-package
|
JS bundling is tsdown (root tsdown.config.ts + two per-package
|
||||||
overrides in vendor/).
|
overrides in vendor/).
|
||||||
@@ -53,6 +57,8 @@ yarn typecheck # tsc -b tsconfig.build.json (declarations only)
|
|||||||
yarn build # typecheck + tsdown JS bundles into each package's lib/
|
yarn build # typecheck + tsdown JS bundles into each package's lib/
|
||||||
yarn demo # run examples/echo-agent (needs --expose-internals, the
|
yarn demo # run examples/echo-agent (needs --expose-internals, the
|
||||||
# script passes it; type "echo hi" to see a tool call)
|
# script passes it; type "echo hi" to see a tool call)
|
||||||
|
yarn demo:coding # run examples/coding-agent — the real agent (needs
|
||||||
|
# DEEPSEEK_API_KEY; give it a coding task)
|
||||||
```
|
```
|
||||||
|
|
||||||
## Secrets / .env
|
## Secrets / .env
|
||||||
@@ -98,13 +104,21 @@ unresolved-type `no-unsafe-*` errors.
|
|||||||
- **Waterfall semantics**: `ctx.waterfall` listeners receive `(...args, next)`
|
- **Waterfall semantics**: `ctx.waterfall` listeners receive `(...args, next)`
|
||||||
and MUST call `next()` to delegate; returning without it short-circuits.
|
and MUST call `next()` to delegate; returning without it short-circuits.
|
||||||
This is the veto mechanism — use deliberately.
|
This is the veto mechanism — use deliberately.
|
||||||
|
- **Discriminated unions: match, don't chain**: branch on a tagged union
|
||||||
|
(`StreamChunk`, `FinishReason`, `SessionEvent`, …) with a `switch` on the
|
||||||
|
tag, not a chain of `if (x.kind === '…')`. The switch narrows each arm so
|
||||||
|
member-only fields (`finish.message`, `finish.code`) are reachable in the
|
||||||
|
right case and a typo'd tag fails to compile. Prefer extracting a small
|
||||||
|
typed helper (`finishError(finish: FinishReason)`) over inlining the
|
||||||
|
branches at the call site.
|
||||||
- **Switch exhaustiveness**: switches over CLOSED unions (e.g. `StreamChunk`)
|
- **Switch exhaustiveness**: switches over CLOSED unions (e.g. `StreamChunk`)
|
||||||
end with `default: assertNever(value, 'context')` (from dsh-llm) so adding a
|
end with `default: assertNever(value, 'context')` (from dsh-llm) so adding a
|
||||||
variant breaks compilation at every switch that must handle it. Switches
|
variant breaks compilation at every switch that must handle it. Switches
|
||||||
over MERGE-EXTENSIBLE unions (`SessionEventMap`, `ContentBlockMap`, …) must
|
over MERGE-EXTENSIBLE unions (`SessionEventMap`, `ContentBlockMap`,
|
||||||
NOT use assertNever — plugin-added variants are valid unknown values; handle
|
`FinishReason`, …) must NOT use assertNever — plugin-added variants are
|
||||||
known cases and fall through with a comment (the lint rule
|
valid unknown values; handle known cases and fall through `default` with a
|
||||||
`switch-exhaustiveness-check` makes the choice explicit either way).
|
comment (the lint rule `switch-exhaustiveness-check` makes the choice
|
||||||
|
explicit either way; a redundant disable directive is itself a lint error).
|
||||||
- **Plugins, not loop changes**: new behavior goes into a plugin on the
|
- **Plugins, not loop changes**: new behavior goes into a plugin on the
|
||||||
documented extension seams (see the plugin sanity checklist in
|
documented extension seams (see the plugin sanity checklist in
|
||||||
docs/architecture.md). Changing `agent-loop` requires updating that doc.
|
docs/architecture.md). Changing `agent-loop` requires updating that doc.
|
||||||
@@ -136,6 +150,11 @@ unresolved-type `no-unsafe-*` errors.
|
|||||||
`response.json()` parse in `dsh-llm-deepseek`'s adapter sets `code` + HTTP
|
`response.json()` parse in `dsh-llm-deepseek`'s adapter sets `code` + HTTP
|
||||||
`status` from the status line before the `try`, so a malformed provider body
|
`status` from the status line before the `try`, so a malformed provider body
|
||||||
can only cost a richer message, never the real error.
|
can only cost a richer message, never the real error.
|
||||||
|
- **Symmetry is usually more correct**: when two related values play parallel
|
||||||
|
roles (a test fixture and its expected output, a request shape and its
|
||||||
|
response shape, a buggy input and the test that checks the fix), give them
|
||||||
|
parallel form — both named consts, or both inline, not one each way. Asymmetry
|
||||||
|
is a smell that usually points at a missed extraction.
|
||||||
- **Tests**: vitest, colocated under `packages/<name>/tests/*.spec.ts`. Every
|
- **Tests**: vitest, colocated under `packages/<name>/tests/*.spec.ts`. Every
|
||||||
registry needs an HMR-safety test (dispose the contributing fiber, assert
|
registry needs an HMR-safety test (dispose the contributing fiber, assert
|
||||||
cleanup). **Excessive tests are welcome** — when in doubt, write the test;
|
cleanup). **Excessive tests are welcome** — when in doubt, write the test;
|
||||||
@@ -143,6 +162,66 @@ unresolved-type `no-unsafe-*` errors.
|
|||||||
concurrency races even if they seem unlikely. Review findings get regression
|
concurrency races even if they seem unlikely. Review findings get regression
|
||||||
tests (see `packages/agent-loop/tests/review-fixes.spec.ts`).
|
tests (see `packages/agent-loop/tests/review-fixes.spec.ts`).
|
||||||
|
|
||||||
|
## Defensive patterns (hard-won)
|
||||||
|
|
||||||
|
Each bullet is a bug class that bit us; the rule prevents the reoccurrence.
|
||||||
|
|
||||||
|
- **Report orthogonal outcomes independently.** A result can be several
|
||||||
|
things at once (a process can both time out AND exit 0 because it trapped
|
||||||
|
the signal). Don't nest the report of one flag inside the branch of
|
||||||
|
another. Surface each independent fact (`timedOut`, `signal`, `exitCode`)
|
||||||
|
on its own so a caller never reads a cut-short run as a clean success.
|
||||||
|
- **Honor cross-seam contracts on BOTH sides.** When an interface documents
|
||||||
|
two valid ways to signal something (e.g. an adapter may report a model
|
||||||
|
failure by THROWING from `stream()` *or* by ending the stream with a
|
||||||
|
`finish {kind:'error'|'aborted'}` chunk), the consumer must handle both —
|
||||||
|
not just the one the first implementation happened to use. A library-backed
|
||||||
|
adapter that can't throw mid-stream relies on the finish-chunk path; if the
|
||||||
|
loop only catches throws, a provider 401 becomes a normal completed turn.
|
||||||
|
Document the contract where the type is defined and exercise every branch
|
||||||
|
through the real consumer in tests.
|
||||||
|
- **Async state is not synchronous state.** `agent.send()` does not flip
|
||||||
|
status to `running` before it returns; a background task's completion races
|
||||||
|
turn boundaries; `reader.close()` fires for both EOF and disposal. Never
|
||||||
|
gate control flow on a status you only *just* requested. Drive lifecycle off
|
||||||
|
the events/promises that actually fire (`agent/status`, `task.done`), and
|
||||||
|
when "done" needs a settle signal, observe the transition (saw `running`
|
||||||
|
THEN `idle`) rather than counting actions you assume map 1:1 to turns —
|
||||||
|
the loop batches queued messages into one turn. But a settle-signal guard
|
||||||
|
cuts both ways: if the awaited transition can *never* occur (EOF with no
|
||||||
|
work submitted → no turn ever starts → never `running`), it hangs forever.
|
||||||
|
Always handle the "nothing to wait for" branch explicitly alongside the
|
||||||
|
"wait for the work" branch.
|
||||||
|
- **Dispose must reach quiescence, not just request it.** A teardown that
|
||||||
|
issues kills/aborts but returns before the work stops leaves orphans. Make
|
||||||
|
cleanup `async` and `await` the children's exit (kill → await `done`), and
|
||||||
|
close listener/notification registries *before* killing so late completions
|
||||||
|
stay silent. Tests must prove disposal *waited* (pid already gone right
|
||||||
|
after `await fiber.dispose()`), not merely that the process eventually dies.
|
||||||
|
- **Contain callback exceptions at the boundary.** A user-supplied listener
|
||||||
|
(`onTaskDone`, event handlers) that throws must not reject the promise it
|
||||||
|
runs inside or starve the listeners after it. Wrap the dispatch loop in
|
||||||
|
try/catch and log; never let one bad subscriber break core lifecycle.
|
||||||
|
- **Never hand untrusted/model output the ambient environment or predictable
|
||||||
|
paths.** Spawned commands get a scrubbed env (drop `*KEY*`/`*SECRET*`/
|
||||||
|
`*TOKEN*`) so the harness's own credentials can't leak into output, `env`,
|
||||||
|
or spill files. Temp/spill files use a private (0700) dir, random names,
|
||||||
|
and exclusive owner-only (`'wx'`, `0o600`) opens — predictable
|
||||||
|
world-readable paths invite symlink races and disclosure.
|
||||||
|
- **e2e tests own their resources.** Real-API/integration tests must create
|
||||||
|
the harness in the test and dispose it in `afterEach` (even on
|
||||||
|
failure/retry/timeout), so a flaky run doesn't leak processes or contexts.
|
||||||
|
Shared fixtures live in a plain `tests/harness.ts` module, NOT another
|
||||||
|
`*.e2e.ts` file — importing a spec file re-registers its `describe` and
|
||||||
|
duplicates real API calls. Verify the WORLD, not the agent's self-report:
|
||||||
|
re-run the command/check externally and assert files are byte-identical
|
||||||
|
where they should be unchanged (a keyword probe lets a cheating agent pass).
|
||||||
|
- **Tag spelling and EOF hygiene.** cordis.yml interpolates env via the
|
||||||
|
`!!js` tag (js-yaml resolves custom tags under `tag:yaml.org,2002:js`), not
|
||||||
|
`!js` — keep code, comments, and docs consistent. Files end with exactly
|
||||||
|
one trailing newline; `git diff --check` (a pre-push gate) rejects new
|
||||||
|
blank lines at EOF.
|
||||||
|
|
||||||
## Type Safety and Documentation
|
## Type Safety and Documentation
|
||||||
|
|
||||||
This codebase aims to be **very type-safe and well documented** for
|
This codebase aims to be **very type-safe and well documented** for
|
||||||
|
|||||||
@@ -371,14 +371,22 @@ export function apply(ctx: Context) {
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
A complete runnable wiring lives in [`examples/echo-agent`](../examples/echo-agent)
|
Two complete runnable wirings exist: [`examples/echo-agent`](../examples/echo-agent)
|
||||||
(mock model + echo tool + stdio UI + JSONL persistence, loaded from
|
(mock model + echo tool — the all-mock skeleton check) and
|
||||||
`cordis.yml` with HMR).
|
[`examples/coding-agent`](../examples/coding-agent) (DeepSeek V4 + the bash
|
||||||
|
tool suite — the real thing; `yarn demo:coding`). Both load from `cordis.yml`
|
||||||
|
with HMR.
|
||||||
|
|
||||||
|
Step-by-step guides live in [`docs/cookbook`](./cookbook): adding a package,
|
||||||
|
adding a tool, adding an LLM adapter.
|
||||||
|
|
||||||
## Deferred work (TODO)
|
## Deferred work (TODO)
|
||||||
|
|
||||||
Tracked here deliberately — each is designed-for but not implemented:
|
Tracked here deliberately — each is designed-for but not implemented:
|
||||||
|
|
||||||
|
- **Restructure this document** — it has grown long; split it into focused
|
||||||
|
sections (or per-area files) so readers can navigate it without scrolling
|
||||||
|
the whole thing.
|
||||||
- **Sub-agent spawn/fork semantics** (seam: `AgentLoop.create()`); inter-agent
|
- **Sub-agent spawn/fork semantics** (seam: `AgentLoop.create()`); inter-agent
|
||||||
channels beyond `send`/`steer`/events.
|
channels beyond `send`/`steer`/events.
|
||||||
- **Persistence backends** (JSONL session dirs, sqlite) on the
|
- **Persistence backends** (JSONL session dirs, sqlite) on the
|
||||||
|
|||||||
55
docs/cookbook/adding-a-package.md
Normal file
55
docs/cookbook/adding-a-package.md
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
# Cookbook: adding a workspace package
|
||||||
|
|
||||||
|
The file-by-file checklist for a new `@deepseek-ai/dsh-<name>` package.
|
||||||
|
(Verified by the bash and adapter packages; if it drifts, fix it here.)
|
||||||
|
|
||||||
|
## 1. Create the package
|
||||||
|
|
||||||
|
```
|
||||||
|
packages/<name>/
|
||||||
|
package.json # copy from packages/tools, adjust name/description/deps
|
||||||
|
tsconfig.json # extends ../../tsconfig.base.json, rootDir src, outDir lib,
|
||||||
|
# references: vendor/cosmokit, vendor/cordis (+ vendor/schemastery
|
||||||
|
# if you use Config, + ../<dep> for each dsh dependency)
|
||||||
|
src/index.ts # service default export or plugin (name/inject/apply/Config)
|
||||||
|
tests/<x>.spec.ts
|
||||||
|
README.md # service API, events, extension points, design notes
|
||||||
|
```
|
||||||
|
|
||||||
|
package.json invariants (enforced by `yarn constraints` / yarn.config.cjs):
|
||||||
|
`private: true`, `version: 0.0.1`, `type: module`, `cordis` in BOTH
|
||||||
|
peerDependencies and devDependencies (same range). Mirror every dsh peer
|
||||||
|
dependency in devDependencies. `schemastery` goes in `dependencies` (it is a
|
||||||
|
runtime validator), matching agent-loop.
|
||||||
|
|
||||||
|
## 2. Register it in the root configs
|
||||||
|
|
||||||
|
| File | Change |
|
||||||
|
|---|---|
|
||||||
|
| `tsconfig.base.json` | add `"@deepseek-ai/dsh-<name>": ["./packages/<name>/src"]` to `paths` |
|
||||||
|
| `tsconfig.typecheck.json` | same entry (this file overrides the map wholesale) |
|
||||||
|
| `tsconfig.build.json` | add `{ "path": "./packages/<name>" }` to `references` |
|
||||||
|
| `scripts/publint-all.ts` | add `'packages/<name>'` to the array |
|
||||||
|
| `knip.json` | only if the package has non-`*.spec.ts` entries (e.g. `*.e2e.ts` → add a per-workspace override like `packages/llm-deepseek`) |
|
||||||
|
|
||||||
|
Covered automatically by globs — no edits needed: root `package.json`
|
||||||
|
workspaces, `tsdown.config.ts`, `vitest.config.ts`, `eslint.config.mjs`.
|
||||||
|
|
||||||
|
## 3. Decide the package topology
|
||||||
|
|
||||||
|
For a swappable capability, split interface / implementation / consumer into
|
||||||
|
separate packages (see docs/architecture.md § "Capability seams" — the bash
|
||||||
|
trio is the template). A single-purpose plugin stays one package.
|
||||||
|
|
||||||
|
## 4. Verify
|
||||||
|
|
||||||
|
```sh
|
||||||
|
yarn install # registers the workspace
|
||||||
|
yarn constraints && yarn typecheck && yarn lint
|
||||||
|
yarn test:coverage # 100% per-file over src (types.ts exempt)
|
||||||
|
yarn build && yarn knip && yarn publint
|
||||||
|
```
|
||||||
|
|
||||||
|
Test expectations: every registry/registration needs an HMR-safety test
|
||||||
|
(register from a child fiber, dispose it, assert cleanup). Excessive tests
|
||||||
|
are welcome — see AGENTS.md.
|
||||||
77
docs/cookbook/adding-a-tool.md
Normal file
77
docs/cookbook/adding-a-tool.md
Normal file
@@ -0,0 +1,77 @@
|
|||||||
|
# Cookbook: adding a tool
|
||||||
|
|
||||||
|
How to give the model a new capability. Reference implementations:
|
||||||
|
`examples/echo-agent/src/echo-tool.ts` (minimal) and
|
||||||
|
`packages/tool-bash` (production-grade, three-package seam).
|
||||||
|
|
||||||
|
## The minimal shape
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import type { Context } from 'cordis'
|
||||||
|
import { defineTool } from '@deepseek-ai/dsh-tools'
|
||||||
|
|
||||||
|
export const name = 'my-tool'
|
||||||
|
export const inject = ['tools']
|
||||||
|
|
||||||
|
export function apply(ctx: Context) {
|
||||||
|
ctx.tools.register(defineTool({
|
||||||
|
name: 'read_file',
|
||||||
|
description: 'Read a file from disk.', // what the model sees
|
||||||
|
parameters: {
|
||||||
|
path: { type: 'string', required: true, description: 'Absolute path' },
|
||||||
|
limit: { type: 'number' }, // optional by default
|
||||||
|
},
|
||||||
|
async execute(args, exec) {
|
||||||
|
// args is TYPED from the schema: { path: string; limit?: number }
|
||||||
|
// exec carries { callId, name, arguments, agent?, signal? }
|
||||||
|
return [{ type: 'text', text: await readFile(args.path, 'utf8') }]
|
||||||
|
},
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Registration is effect-based: disposing the plugin fiber unregisters the
|
||||||
|
tool (write the HMR test). Schemas flow into the system-prompt assembly
|
||||||
|
automatically.
|
||||||
|
|
||||||
|
## Rules of the execute() contract
|
||||||
|
|
||||||
|
- **Validate args at runtime.** `defineTool`'s `InferArgs` typing is
|
||||||
|
compile-time only; at runtime `arguments` is whatever JSON the model
|
||||||
|
emitted. Check every field; throw a descriptive Error for bad input.
|
||||||
|
- **Throwing means isError.** The registry catches anything `execute()`
|
||||||
|
throws and returns `{isError: true}` to the model. Use that for
|
||||||
|
infrastructure failures (bad input, spawn errors, aborts) — but REPORT
|
||||||
|
domain failures in the result text instead (e.g. tool-bash returns
|
||||||
|
`[exit code: 9]` with `isError: false`: the model decides what a failing
|
||||||
|
command means).
|
||||||
|
- **Honor `exec.signal`.** Cancel in-flight work when it fires.
|
||||||
|
- **Use `exec.agent` for async notifications.** `agent.inject(content,
|
||||||
|
{source: {kind: 'plugin', plugin: '<name>'}})` appends durable context the
|
||||||
|
NEXT model request sees — it is not a wake-up (an idle agent stays idle).
|
||||||
|
Guard against disposed agents (try/catch).
|
||||||
|
|
||||||
|
## Long-running work
|
||||||
|
|
||||||
|
Follow tool-bash's background pattern: a `run_in_background` flag returns a
|
||||||
|
task id immediately; companion tools poll incrementally and kill; completion
|
||||||
|
notices arrive via `agent.inject()`. Bound buffers and spill full output to
|
||||||
|
disk so nothing is silently lost.
|
||||||
|
|
||||||
|
> TODO: each tool reimplements this background pattern by hand today. At some
|
||||||
|
> point we need a generic long-running-tool layer that handles task ids,
|
||||||
|
> incremental polling, kill, and completion notices uniformly.
|
||||||
|
|
||||||
|
## Permissions / sandboxing
|
||||||
|
|
||||||
|
Prefer not to build policy into the tool. The seam is the `tools/execute` waterfall
|
||||||
|
(veto or wrap — see the permission-gate example in docs/architecture.md), or
|
||||||
|
a sandboxing implementation behind the tool's executor seam.
|
||||||
|
|
||||||
|
## Tests every tool needs
|
||||||
|
|
||||||
|
Arg-validation rejections, result shaping for every outcome, the HMR
|
||||||
|
disposal test, and — for tools with side effects — an integration spec that
|
||||||
|
drives the tool through the agent loop with a scripted `MockAdapter`
|
||||||
|
(`packages/agent-loop/tests/mock-adapter.ts`), asserting the `tool/call` /
|
||||||
|
`tool/result` session events.
|
||||||
75
docs/cookbook/adding-an-llm-adapter.md
Normal file
75
docs/cookbook/adding-an-llm-adapter.md
Normal file
@@ -0,0 +1,75 @@
|
|||||||
|
# Cookbook: adding an LLM adapter
|
||||||
|
|
||||||
|
How to connect a new model provider. Reference implementations:
|
||||||
|
`packages/llm-deepseek` (hand-rolled HTTP/SSE) and `packages/llm-pi-ai`
|
||||||
|
(wrapping an LLM library). Read the `StreamChunk` doc in
|
||||||
|
`packages/llm/src/types.ts` first — it records the protocol conventions both
|
||||||
|
adapters were verified against.
|
||||||
|
|
||||||
|
## The shape
|
||||||
|
|
||||||
|
```ts
|
||||||
|
class MyAdapter extends LlmAdapter {
|
||||||
|
async * stream(options: GenerateOptions): AsyncIterable<StreamChunk> { … }
|
||||||
|
}
|
||||||
|
|
||||||
|
export const name = 'llm-myprovider'
|
||||||
|
export const inject = ['llm']
|
||||||
|
export const Config: z<Config> = z.object({ apiKey: z.string(), … })
|
||||||
|
|
||||||
|
export function apply(ctx: Context, config: Config) {
|
||||||
|
ctx.llm.registerAdapter(['model-a', 'model-b'], new MyAdapter(…))
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Registration is effect-based (HMR-safe); one adapter per model name —
|
||||||
|
duplicates throw. Secrets are cordis-native: schemastery Config with env
|
||||||
|
fallbacks, fed from cordis.yml via `!!js process.env.MY_KEY`. Never read
|
||||||
|
ad-hoc key files in code.
|
||||||
|
|
||||||
|
## Protocol obligations (the contract two implementations verified)
|
||||||
|
|
||||||
|
- Emit `usage` BEFORE `finish`; emit NOTHING after `finish`. The robust way:
|
||||||
|
buffer finish/usage until the provider's end-of-stream marker, then flush
|
||||||
|
(handles providers that send trailing usage-only chunks).
|
||||||
|
- Tool-call `arguments` are RAW JSON strings end-to-end; stream fragments as
|
||||||
|
`argumentsDelta`. If your provider hands back parsed objects, re-stringify
|
||||||
|
at `block-end`.
|
||||||
|
- Allocate block `index`es in first-seen stream order; reuse the index for
|
||||||
|
every delta of the same block.
|
||||||
|
- Errors have exactly two sanctioned paths: THROW from `stream()` (transport
|
||||||
|
and protocol failures — use `LlmError` with a stable code), or end the
|
||||||
|
stream with `finish {kind: 'error' | 'aborted'}` (provider in-band
|
||||||
|
failures). Consumers handle both; pick per failure class and document it.
|
||||||
|
- Honor `options.signal` (pass it to fetch / your SDK).
|
||||||
|
- `prefill` and other unsupported `GenerateOptions` fields: throw
|
||||||
|
`LlmError(..., 'UNSUPPORTED')` rather than silently dropping.
|
||||||
|
|
||||||
|
Provider-specific request knobs (thinking modes, effort levels) belong in
|
||||||
|
the ADAPTER's Config, not in `GenerateOptions` — the core vocabulary stays
|
||||||
|
provider-neutral.
|
||||||
|
|
||||||
|
## Structure that worked
|
||||||
|
|
||||||
|
Split the adapter into testable stages (llm-deepseek's layout): wire types
|
||||||
|
(`types.ts`, coverage-exempt) → request serializer → SSE/transport parser →
|
||||||
|
chunk-translation state machine → a thin adapter class wiring them. Each
|
||||||
|
stage gets its own unit suite.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
- **Unit: mock the provider, not the harness.** A scripted `node:http`
|
||||||
|
server speaking the provider's wire format covers happy paths, every error
|
||||||
|
status, malformed payloads, premature closes, and aborts — no network, and
|
||||||
|
it drives the 100% per-file coverage gate. Works for SDK-backed adapters
|
||||||
|
too (point the SDK's baseURL at the mock).
|
||||||
|
- **Hostile framing tests.** Split stream payloads at arbitrary byte
|
||||||
|
positions (including mid-UTF-8) — real networks do.
|
||||||
|
- **E2E: `tests/*.e2e.ts`** under `yarn test:e2e`, gated with
|
||||||
|
`describe.skipIf(!process.env.MY_KEY)` so CI (no secrets) stays green.
|
||||||
|
Cover each model × each provider mode you map (thinking on/off, effort
|
||||||
|
levels), a tool-call round trip INCLUDING the follow-up turn with results
|
||||||
|
in history, and loose assertions only (substring/structure, bounded
|
||||||
|
maxTokens — real models are nondeterministic).
|
||||||
|
- Register the e2e file pattern in `knip.json` (per-workspace `entry`
|
||||||
|
override) or knip flags it unused.
|
||||||
51
examples/coding-agent/README.md
Normal file
51
examples/coding-agent/README.md
Normal file
@@ -0,0 +1,51 @@
|
|||||||
|
# coding-agent
|
||||||
|
|
||||||
|
The first REAL agent wiring: DeepSeek V4 + the bash tool suite + stdio chat
|
||||||
|
+ JSONL persistence, loaded from `cordis.yml`. Where echo-agent proves the
|
||||||
|
skeleton with mocks, this example is a usable coding assistant.
|
||||||
|
|
||||||
|
## Run it
|
||||||
|
|
||||||
|
```sh
|
||||||
|
# repo root .env (gitignored) or exported env:
|
||||||
|
# DEEPSEEK_API_KEY=sk-…
|
||||||
|
# DEEPSEEK_BASE_URL=https://… # optional; defaults to the public API
|
||||||
|
yarn demo:coding
|
||||||
|
```
|
||||||
|
|
||||||
|
Type a coding task. The agent's only tools are `bash` (+ `bash_output` /
|
||||||
|
`bash_kill` for background tasks): file reads, writes, searches, and test
|
||||||
|
runs all happen through shell commands, each in a fresh `bash -c` (the
|
||||||
|
system prompt tells the model to pass `workdir` instead of `cd`). Reasoning
|
||||||
|
streams dimmed; tool calls/results render inline.
|
||||||
|
|
||||||
|
```
|
||||||
|
> fix the failing test in /path/to/project
|
||||||
|
[main turn 1] (reasoning…)
|
||||||
|
[tool call] bash({"command": "node --test", "workdir": "/path/to/project"})
|
||||||
|
[tool result] … [exit code: 1]
|
||||||
|
…
|
||||||
|
```
|
||||||
|
|
||||||
|
## What each plugin demonstrates
|
||||||
|
|
||||||
|
| Entry | Demonstrates |
|
||||||
|
|---|---|
|
||||||
|
| `llm-deepseek` | real `LlmAdapter` via config (`!!js process.env.…` secrets); swap one line to `@deepseek-ai/dsh-llm-pi-ai` for the library-backed twin |
|
||||||
|
| `bash` (`dsh-bash-local`) + `tool-bash` | the executor seam + tool schemas as separate plugins |
|
||||||
|
| `agent-loop` | agent created from config with a coding system prompt |
|
||||||
|
| `src/session-jsonl.ts` | write-behind persistence on `session/event` + `session/flush` (copied from echo-agent) |
|
||||||
|
| `src/stdio-chat.ts` | UI as a plugin; copied from echo-agent with reasoning-dimming and an exit-on-idle close handler for piped stdin. Example-local on purpose — extract a shared UI package when a third example needs it |
|
||||||
|
|
||||||
|
## End-to-end tests (`yarn test:e2e`, key-gated)
|
||||||
|
|
||||||
|
- `tests/full-loop.e2e.ts` — the canary: real model runs `echo e2e-ok`
|
||||||
|
through the real bash tool; asserts `tool/call`/`tool/result` session
|
||||||
|
events and the final answer.
|
||||||
|
- `tests/coding-task.e2e.ts` — the swebench-style smoke: a temp dir holds
|
||||||
|
`add.js` (with `a - b` where `a + b` belongs) and a failing
|
||||||
|
`add.test.js`; the agent must fix the bug and verify. The test re-runs
|
||||||
|
`node add.test.js` ITSELF and inspects the files — agent claims are not
|
||||||
|
trusted.
|
||||||
|
|
||||||
|
Both self-skip without `DEEPSEEK_API_KEY`.
|
||||||
77
examples/coding-agent/cordis.yml
Normal file
77
examples/coding-agent/cordis.yml
Normal file
@@ -0,0 +1,77 @@
|
|||||||
|
# The coding-agent plugin tree, loaded via @cordisjs/plugin-include.
|
||||||
|
# Core services first, then the real adapters/tools, then the agent itself.
|
||||||
|
#
|
||||||
|
# Requires DEEPSEEK_API_KEY (and optionally DEEPSEEK_BASE_URL) in the
|
||||||
|
# environment — start.ts loads the gitignored repo-root .env first.
|
||||||
|
|
||||||
|
- id: logger
|
||||||
|
name: '@cordisjs/plugin-logger-console'
|
||||||
|
|
||||||
|
- id: timer
|
||||||
|
name: '@cordisjs/plugin-timer'
|
||||||
|
|
||||||
|
- id: hmr
|
||||||
|
name: '@cordisjs/plugin-hmr'
|
||||||
|
config:
|
||||||
|
root: ['.']
|
||||||
|
|
||||||
|
- id: llm
|
||||||
|
name: '@deepseek-ai/dsh-llm'
|
||||||
|
|
||||||
|
- id: sessions
|
||||||
|
name: '@deepseek-ai/dsh-session'
|
||||||
|
|
||||||
|
- id: system-prompt
|
||||||
|
name: '@deepseek-ai/dsh-system-prompt'
|
||||||
|
|
||||||
|
- id: tools
|
||||||
|
name: '@deepseek-ai/dsh-tools'
|
||||||
|
|
||||||
|
- id: agents
|
||||||
|
name: '@deepseek-ai/dsh-agent'
|
||||||
|
|
||||||
|
# The DeepSeek adapter. Swap to '@deepseek-ai/dsh-llm-pi-ai' for the
|
||||||
|
# pi-ai-backed twin (same config shape; `reasoning: high` replaces
|
||||||
|
# thinking/reasoningEffort).
|
||||||
|
- id: llm-deepseek
|
||||||
|
name: '@deepseek-ai/dsh-llm-deepseek'
|
||||||
|
config:
|
||||||
|
apiKey: !!js process.env.DEEPSEEK_API_KEY
|
||||||
|
baseURL: !!js process.env.DEEPSEEK_BASE_URL
|
||||||
|
models:
|
||||||
|
- deepseek-v4-flash
|
||||||
|
- deepseek-v4-pro
|
||||||
|
|
||||||
|
# Bash execution: the local executor implementation + the tool schemas.
|
||||||
|
- id: bash
|
||||||
|
name: '@deepseek-ai/dsh-bash-local'
|
||||||
|
config:
|
||||||
|
timeoutMs: 60000
|
||||||
|
|
||||||
|
- id: tool-bash
|
||||||
|
name: '@deepseek-ai/dsh-tool-bash'
|
||||||
|
|
||||||
|
- id: agent-loop
|
||||||
|
name: '@deepseek-ai/dsh-agent-loop'
|
||||||
|
config:
|
||||||
|
agents:
|
||||||
|
- id: main
|
||||||
|
model: deepseek-v4-flash
|
||||||
|
systemPrompt: |
|
||||||
|
You are coding-agent, a CLI coding assistant.
|
||||||
|
|
||||||
|
Your only tools are bash (plus bash_output/bash_kill for background
|
||||||
|
tasks). Do ALL file operations through bash: read with cat/sed/head,
|
||||||
|
search with grep, write with heredocs (cat <<'EOF' > file), edit
|
||||||
|
with sed or a rewrite. Each bash call runs in a fresh shell — pass
|
||||||
|
workdir instead of cd, and never rely on shell state between calls.
|
||||||
|
|
||||||
|
Check the [exit code: N] marker on every command; investigate
|
||||||
|
failures before moving on. Verify your work by running the code or
|
||||||
|
tests. Keep answers brief and factual.
|
||||||
|
|
||||||
|
- id: session-jsonl
|
||||||
|
name: './src/session-jsonl.ts'
|
||||||
|
|
||||||
|
- id: stdio-chat
|
||||||
|
name: './src/stdio-chat.ts'
|
||||||
7
examples/coding-agent/package.json
Normal file
7
examples/coding-agent/package.json
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
{
|
||||||
|
"name": "coding-agent-example",
|
||||||
|
"private": true,
|
||||||
|
"version": "0.0.1",
|
||||||
|
"type": "module",
|
||||||
|
"description": "Runnable demo: a real coding agent — DeepSeek V4 + the bash tool suite"
|
||||||
|
}
|
||||||
36
examples/coding-agent/src/session-jsonl.ts
Normal file
36
examples/coding-agent/src/session-jsonl.ts
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
import { appendFile } from 'node:fs/promises'
|
||||||
|
import { join } from 'node:path'
|
||||||
|
import type { Context } from 'cordis'
|
||||||
|
import type { Session, SessionEvent } from '@deepseek-ai/dsh-session'
|
||||||
|
|
||||||
|
export const name = 'session-jsonl'
|
||||||
|
export const inject = ['sessions']
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Minimal persistence plugin: buffers session events (write-behind) and
|
||||||
|
* drains to a JSONL file at every `session/flush` checkpoint — the pattern a
|
||||||
|
* real JSONL/sqlite persistence plugin would follow.
|
||||||
|
*/
|
||||||
|
export function apply(ctx: Context) {
|
||||||
|
const buffers = new Map<Session, SessionEvent[]>()
|
||||||
|
const path = (session: Session) => join(import.meta.dirname, '..', `${session.id}.jsonl`)
|
||||||
|
|
||||||
|
ctx.on('session/event', (session, event) => {
|
||||||
|
let buffer = buffers.get(session)
|
||||||
|
if (!buffer) buffers.set(session, buffer = [])
|
||||||
|
buffer.push(event)
|
||||||
|
})
|
||||||
|
|
||||||
|
const flush = async (session: Session) => {
|
||||||
|
const buffer = buffers.get(session)
|
||||||
|
if (!buffer?.length) return
|
||||||
|
const lines = buffer.splice(0).map(event => JSON.stringify(event) + '\n').join('')
|
||||||
|
await appendFile(path(session), lines)
|
||||||
|
}
|
||||||
|
|
||||||
|
ctx.on('session/flush', flush)
|
||||||
|
ctx.effect(() => () => {
|
||||||
|
// drain remaining buffers on dispose
|
||||||
|
for (const session of buffers.keys()) void flush(session)
|
||||||
|
}, 'session-jsonl')
|
||||||
|
}
|
||||||
119
examples/coding-agent/src/stdio-chat.ts
Normal file
119
examples/coding-agent/src/stdio-chat.ts
Normal file
@@ -0,0 +1,119 @@
|
|||||||
|
import { createInterface } from 'node:readline'
|
||||||
|
import type { Context } from 'cordis'
|
||||||
|
import type {} from '@deepseek-ai/dsh-agent'
|
||||||
|
|
||||||
|
export const name = 'stdio-chat'
|
||||||
|
export const inject = ['agents']
|
||||||
|
|
||||||
|
// Copied from examples/echo-agent (welcome text + reasoning rendering
|
||||||
|
// adjusted). Deliberately example-local rather than a shared package — two
|
||||||
|
// examples don't justify the abstraction yet; revisit at the third.
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Minimal UI plugin: reads lines from stdin → agent.send(); renders the
|
||||||
|
* agent's stream chunks and tool activity to stdout. Demonstrates that a UI
|
||||||
|
* is "just a plugin" — it only consumes the agent/* event taxonomy.
|
||||||
|
*/
|
||||||
|
export function apply(ctx: Context) {
|
||||||
|
let inReasoning = false
|
||||||
|
ctx.on('agent/stream-chunk', (_agent, _turn, _step, chunk) => {
|
||||||
|
if (chunk.type === 'reasoning-delta') {
|
||||||
|
// Dim the chain-of-thought so the answer stands out.
|
||||||
|
if (!inReasoning) process.stdout.write('\x1B[2m')
|
||||||
|
inReasoning = true
|
||||||
|
process.stdout.write(chunk.text)
|
||||||
|
} else if (chunk.type === 'text-delta') {
|
||||||
|
if (inReasoning) process.stdout.write('\x1B[0m\n')
|
||||||
|
inReasoning = false
|
||||||
|
process.stdout.write(chunk.text)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
ctx.on('agent/turn-start', (agent, turn) => {
|
||||||
|
process.stdout.write(`\n[${agent.id} turn ${turn}] `)
|
||||||
|
})
|
||||||
|
|
||||||
|
ctx.on('agent/turn-end', () => {
|
||||||
|
if (inReasoning) process.stdout.write('\x1B[0m')
|
||||||
|
inReasoning = false
|
||||||
|
process.stdout.write('\n> ')
|
||||||
|
})
|
||||||
|
|
||||||
|
ctx.on('session/event', (_session, event) => {
|
||||||
|
if (event.type === 'tool/call') {
|
||||||
|
const { name: toolName, arguments: args } = event.data
|
||||||
|
if (inReasoning) process.stdout.write('\x1B[0m')
|
||||||
|
inReasoning = false
|
||||||
|
process.stdout.write(`\n [tool call] ${toolName}(${args})`)
|
||||||
|
} else if (event.type === 'tool/result') {
|
||||||
|
const { content } = event.data
|
||||||
|
const text = content.filter(b => b.type === 'text').map(b => b.text).join('')
|
||||||
|
process.stdout.write(`\n [tool result] ${text}\n `)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
ctx.effect(() => {
|
||||||
|
const reader = createInterface({ input: process.stdin })
|
||||||
|
// Piped-input exit, once stdin reaches EOF:
|
||||||
|
// - If no line ever submitted work (empty stdin, blank-only lines), exit
|
||||||
|
// immediately — no turn will ever start, so there is nothing to wait
|
||||||
|
// for. (Gating on an observed 'running' here would hang forever.)
|
||||||
|
// - If work WAS submitted, exit the next time the agent settles to idle
|
||||||
|
// AFTER having run. Two subtleties this handles: the loop batches
|
||||||
|
// several queued messages into ONE turn (one idle), so we don't count
|
||||||
|
// sends; and agent.send() does NOT synchronously flip status to
|
||||||
|
// 'running', so requiring an observed 'running' first (`sawRunning`)
|
||||||
|
// avoids exiting in the gap before the turn starts and dropping work.
|
||||||
|
let stdinClosed = false
|
||||||
|
let disposed = false
|
||||||
|
let submittedWork = false
|
||||||
|
let sawRunning = false
|
||||||
|
|
||||||
|
const maybeExit = (): void => {
|
||||||
|
if (disposed || !stdinClosed) return
|
||||||
|
// No work submitted: nothing will ever run, exit straight away.
|
||||||
|
// Work submitted: wait until a turn has run and the agent is idle.
|
||||||
|
if (submittedWork) {
|
||||||
|
if (!sawRunning) return
|
||||||
|
const agent = ctx.agents.get('main')
|
||||||
|
if (agent && agent.status !== 'idle') return // a turn is still running
|
||||||
|
}
|
||||||
|
// Let any final output flush, then exit.
|
||||||
|
setTimeout(() => process.exit(0), 200)
|
||||||
|
}
|
||||||
|
|
||||||
|
const disposeStatusListener = ctx.on('agent/status', (subject, status) => {
|
||||||
|
if (subject.id !== 'main') return
|
||||||
|
if (status === 'running') sawRunning = true
|
||||||
|
if (status === 'idle') maybeExit()
|
||||||
|
})
|
||||||
|
|
||||||
|
reader.on('line', (line) => {
|
||||||
|
const text = line.trim()
|
||||||
|
if (!text) return
|
||||||
|
const agent = ctx.agents.get('main')
|
||||||
|
if (!agent) {
|
||||||
|
console.error('agent "main" is not running')
|
||||||
|
return
|
||||||
|
}
|
||||||
|
submittedWork = true
|
||||||
|
if (agent.status === 'running') {
|
||||||
|
agent.steer([{ type: 'text', text }])
|
||||||
|
} else {
|
||||||
|
agent.send([{ type: 'text', text }])
|
||||||
|
}
|
||||||
|
})
|
||||||
|
reader.on('close', () => {
|
||||||
|
// Fires for BOTH stdin EOF and plugin disposal (reader.close() below);
|
||||||
|
// `disposed` guards teardown so HMR/dispose never exits the process.
|
||||||
|
stdinClosed = true
|
||||||
|
maybeExit()
|
||||||
|
})
|
||||||
|
process.stdout.write('coding-agent ready. Give it a coding task (bash is its only tool).\n> ')
|
||||||
|
return () => {
|
||||||
|
disposed = true
|
||||||
|
disposeStatusListener()
|
||||||
|
reader.close()
|
||||||
|
}
|
||||||
|
}, 'stdio-chat')
|
||||||
|
}
|
||||||
25
examples/coding-agent/start.ts
Normal file
25
examples/coding-agent/start.ts
Normal file
@@ -0,0 +1,25 @@
|
|||||||
|
import { pathToFileURL } from 'node:url'
|
||||||
|
import { Context } from 'cordis'
|
||||||
|
import Loader from '@cordisjs/plugin-loader'
|
||||||
|
|
||||||
|
// Load DEEPSEEK_API_KEY / DEEPSEEK_BASE_URL from a gitignored repo-root .env
|
||||||
|
// (Node >= 21.7 native). Absent file is fine — the environment may already
|
||||||
|
// carry the variables; cordis.yml reads them via the `!!js` tag.
|
||||||
|
try {
|
||||||
|
process.loadEnvFile(new URL('../../.env', import.meta.url).pathname)
|
||||||
|
} catch {
|
||||||
|
// no .env — rely on the ambient environment
|
||||||
|
}
|
||||||
|
|
||||||
|
// Boot a Cordis app from this example's cordis.yml — the same shape as the
|
||||||
|
// upstream `cordis` bin, pinned to this directory.
|
||||||
|
const ctx = new Context()
|
||||||
|
ctx.baseUrl = pathToFileURL(import.meta.dirname).href + '/'
|
||||||
|
|
||||||
|
await ctx.plugin(Loader)
|
||||||
|
await ctx.loader.create({
|
||||||
|
name: '@cordisjs/plugin-include',
|
||||||
|
config: {
|
||||||
|
path: './cordis.yml',
|
||||||
|
},
|
||||||
|
})
|
||||||
86
examples/coding-agent/tests/coding-task.e2e.ts
Normal file
86
examples/coding-agent/tests/coding-task.e2e.ts
Normal file
@@ -0,0 +1,86 @@
|
|||||||
|
import { spawnSync } from 'node:child_process'
|
||||||
|
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
|
||||||
|
import { tmpdir } from 'node:os'
|
||||||
|
import { join } from 'node:path'
|
||||||
|
import { afterEach, describe, expect, it } from 'vitest'
|
||||||
|
import type { Context } from 'cordis'
|
||||||
|
import { codingHarness, finalText, SYSTEM_PROMPT, waitForIdle } from './harness.ts'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The swebench-style smoke test: a real model fixes a real bug in a temp
|
||||||
|
* directory using only the bash tool, and the fix is verified OUTSIDE the
|
||||||
|
* agent by re-running the test script. Key-gated.
|
||||||
|
*/
|
||||||
|
|
||||||
|
const TEST_FILE = [
|
||||||
|
"const assert = require('node:assert');",
|
||||||
|
"const { add } = require('./add.js');",
|
||||||
|
'assert.strictEqual(add(2, 3), 5);',
|
||||||
|
'assert.strictEqual(add(-1, 1), 0);',
|
||||||
|
"console.log('PASS');",
|
||||||
|
'',
|
||||||
|
].join('\n')
|
||||||
|
|
||||||
|
const BUGGY_ADD = [
|
||||||
|
'// A tiny module with an obvious bug.',
|
||||||
|
'function add(a, b) {',
|
||||||
|
' return a - b;',
|
||||||
|
'}',
|
||||||
|
'module.exports = { add };',
|
||||||
|
'',
|
||||||
|
].join('\n')
|
||||||
|
|
||||||
|
let workdir: string | undefined
|
||||||
|
let ctx: Context | undefined
|
||||||
|
|
||||||
|
afterEach(async () => {
|
||||||
|
// Dispose the harness even on failure/retry: agent-loop teardown stops the
|
||||||
|
// loop and LocalBashExecutor teardown kills anything the model left running.
|
||||||
|
await ctx?.fiber.dispose()
|
||||||
|
ctx = undefined
|
||||||
|
if (workdir !== undefined) await rm(workdir, { recursive: true, force: true })
|
||||||
|
workdir = undefined
|
||||||
|
})
|
||||||
|
|
||||||
|
describe.skipIf(!process.env.DEEPSEEK_API_KEY)('coding task: fix a failing test via bash', () => {
|
||||||
|
it('repairs add.js so node add.test.js passes', async () => {
|
||||||
|
workdir = await mkdtemp(join(tmpdir(), 'dsh-coding-task-'))
|
||||||
|
await writeFile(join(workdir, 'add.js'), BUGGY_ADD)
|
||||||
|
await writeFile(join(workdir, 'add.test.js'), TEST_FILE)
|
||||||
|
|
||||||
|
// Confirm the fixture actually fails before the agent touches it.
|
||||||
|
const before = spawnSync('node', ['add.test.js'], { cwd: workdir })
|
||||||
|
expect(before.status).not.toBe(0)
|
||||||
|
|
||||||
|
ctx = await codingHarness(workdir)
|
||||||
|
const agent = ctx.agentLoop.create('e2e-task', {
|
||||||
|
model: 'deepseek-v4-flash',
|
||||||
|
systemPrompt: SYSTEM_PROMPT,
|
||||||
|
})
|
||||||
|
|
||||||
|
agent.send([{
|
||||||
|
type: 'text',
|
||||||
|
text: 'In the current directory, `node add.test.js` fails because add.js has a bug. '
|
||||||
|
+ 'Fix add.js so the test passes, run `node add.test.js` to verify, and report the result. '
|
||||||
|
+ 'Do not modify add.test.js.',
|
||||||
|
}])
|
||||||
|
await waitForIdle(ctx, agent)
|
||||||
|
|
||||||
|
// The agent claims success…
|
||||||
|
const summary = finalText([...agent.session.events]).toLowerCase()
|
||||||
|
expect(summary.length).toBeGreaterThan(0)
|
||||||
|
|
||||||
|
// …and the world agrees: the test passes when WE run it, and the test
|
||||||
|
// file is byte-identical (an agent that neutered the test instead of
|
||||||
|
// fixing the bug fails here, not just on a keyword probe).
|
||||||
|
const untouchedTest = await readFile(join(workdir, 'add.test.js'), 'utf8')
|
||||||
|
expect(untouchedTest).toBe(TEST_FILE)
|
||||||
|
|
||||||
|
const after = spawnSync('node', ['add.test.js'], { cwd: workdir, encoding: 'utf8' })
|
||||||
|
expect(after.stdout).toContain('PASS')
|
||||||
|
expect(after.status).toBe(0)
|
||||||
|
|
||||||
|
const fixed = await readFile(join(workdir, 'add.js'), 'utf8')
|
||||||
|
expect(fixed).not.toMatch(/a\s*-\s*b/)
|
||||||
|
}, 180_000)
|
||||||
|
})
|
||||||
43
examples/coding-agent/tests/full-loop.e2e.ts
Normal file
43
examples/coding-agent/tests/full-loop.e2e.ts
Normal file
@@ -0,0 +1,43 @@
|
|||||||
|
import { afterEach, describe, expect, it } from 'vitest'
|
||||||
|
import type { Context } from 'cordis'
|
||||||
|
import { codingHarness, finalText, SYSTEM_PROMPT, waitForIdle } from './harness.ts'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The first place a REAL model meets the REAL bash tool: the cheap canary
|
||||||
|
* before the coding-task e2e. Key-gated (see vitest.e2e.config.ts).
|
||||||
|
*/
|
||||||
|
|
||||||
|
let ctx: Context | undefined
|
||||||
|
|
||||||
|
afterEach(async () => {
|
||||||
|
// Always dispose the harness, even on failure/retry/timeout: agent-loop
|
||||||
|
// teardown stops the loop and LocalBashExecutor teardown kills any
|
||||||
|
// process the model left behind.
|
||||||
|
await ctx?.fiber.dispose()
|
||||||
|
ctx = undefined
|
||||||
|
})
|
||||||
|
|
||||||
|
describe.skipIf(!process.env.DEEPSEEK_API_KEY)('full loop: real model + real bash tool', () => {
|
||||||
|
it('runs a bash command on request and reports its output', async () => {
|
||||||
|
ctx = await codingHarness(process.cwd())
|
||||||
|
const agent = ctx.agentLoop.create('e2e-loop', {
|
||||||
|
model: 'deepseek-v4-flash',
|
||||||
|
systemPrompt: SYSTEM_PROMPT,
|
||||||
|
})
|
||||||
|
|
||||||
|
agent.send([{ type: 'text', text: 'Run `echo e2e-ok` with the bash tool and tell me its exact output.' }])
|
||||||
|
await waitForIdle(ctx, agent)
|
||||||
|
|
||||||
|
const events = [...agent.session.events]
|
||||||
|
const calls = events.filter(event => event.type === 'tool/call')
|
||||||
|
expect(calls.length).toBeGreaterThan(0)
|
||||||
|
expect(calls.some(event => event.data.name === 'bash')).toBe(true)
|
||||||
|
|
||||||
|
const results = events.filter(event => event.type === 'tool/result')
|
||||||
|
const resultTexts = results.flatMap(event =>
|
||||||
|
event.data.content.filter(block => block.type === 'text').map(block => block.text))
|
||||||
|
expect(resultTexts.some(text => text.includes('e2e-ok'))).toBe(true)
|
||||||
|
|
||||||
|
expect(finalText(events)).toContain('e2e-ok')
|
||||||
|
}, 120_000)
|
||||||
|
})
|
||||||
55
examples/coding-agent/tests/harness.ts
Normal file
55
examples/coding-agent/tests/harness.ts
Normal file
@@ -0,0 +1,55 @@
|
|||||||
|
import { Context } from 'cordis'
|
||||||
|
import LlmService from '@deepseek-ai/dsh-llm'
|
||||||
|
import SessionStore from '@deepseek-ai/dsh-session'
|
||||||
|
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||||
|
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
|
||||||
|
import ToolRegistry from '@deepseek-ai/dsh-tools'
|
||||||
|
import AgentRegistry from '@deepseek-ai/dsh-agent'
|
||||||
|
import AgentLoop, { LoopAgent } from '@deepseek-ai/dsh-agent-loop'
|
||||||
|
import { LocalBashExecutor } from '@deepseek-ai/dsh-bash-local'
|
||||||
|
import * as ToolBash from '@deepseek-ai/dsh-tool-bash'
|
||||||
|
import * as LlmDeepSeek from '@deepseek-ai/dsh-llm-deepseek'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Shared harness for the coding-agent e2e suites: the full plugin stack
|
||||||
|
* with the real DeepSeek adapter and the real bash tool. Lives outside the
|
||||||
|
* *.e2e.ts pattern so importing it never re-registers another file's tests.
|
||||||
|
*/
|
||||||
|
|
||||||
|
export const SYSTEM_PROMPT = 'You are a coding agent. Your only tool is bash; '
|
||||||
|
+ 'do file operations with cat/grep/heredocs, check [exit code: N] markers, '
|
||||||
|
+ 'and report results briefly.'
|
||||||
|
|
||||||
|
export async function codingHarness(workdir: string): Promise<Context> {
|
||||||
|
const ctx = new Context()
|
||||||
|
await ctx.plugin(LlmService)
|
||||||
|
await ctx.plugin(SessionStore)
|
||||||
|
await ctx.plugin(SystemPrompt)
|
||||||
|
await ctx.plugin(ToolRegistry)
|
||||||
|
await ctx.plugin(AgentRegistry)
|
||||||
|
await ctx.plugin(AgentLoop, { agents: [] })
|
||||||
|
await ctx.plugin(LlmDeepSeek, { models: ['deepseek-v4-flash'] })
|
||||||
|
await ctx.plugin(LocalBashExecutor, { cwd: workdir, timeoutMs: 30_000 })
|
||||||
|
await ctx.plugin(ToolBash)
|
||||||
|
return ctx
|
||||||
|
}
|
||||||
|
|
||||||
|
export function waitForIdle(ctx: Context, agent: LoopAgent): Promise<void> {
|
||||||
|
return new Promise((resolve) => {
|
||||||
|
const dispose = ctx.on('agent/status', (subject, status) => {
|
||||||
|
if (subject === agent && status === 'idle') {
|
||||||
|
dispose()
|
||||||
|
resolve()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
export function finalText(events: SessionEvent[]): string {
|
||||||
|
const message = events.findLast(event => event.type === 'assistant/message')
|
||||||
|
if (message?.type !== 'assistant/message') return ''
|
||||||
|
return message.data.content
|
||||||
|
.filter(block => block.type === 'text')
|
||||||
|
.map(block => block.text)
|
||||||
|
.join('')
|
||||||
|
}
|
||||||
@@ -4,7 +4,7 @@
|
|||||||
"ignoreWorkspaces": ["vendor/*"],
|
"ignoreWorkspaces": ["vendor/*"],
|
||||||
"workspaces": {
|
"workspaces": {
|
||||||
".": {
|
".": {
|
||||||
"entry": ["examples/echo-agent/src/*.ts"],
|
"entry": ["examples/echo-agent/src/*.ts", "examples/coding-agent/src/*.ts"],
|
||||||
"project": ["scripts/**/*.ts", "examples/**/*.ts"]
|
"project": ["scripts/**/*.ts", "examples/**/*.ts"]
|
||||||
},
|
},
|
||||||
"packages/*": {
|
"packages/*": {
|
||||||
|
|||||||
@@ -23,6 +23,7 @@
|
|||||||
"publint": "tsx scripts/publint-all.ts",
|
"publint": "tsx scripts/publint-all.ts",
|
||||||
"hygiene": "yarn knip && yarn publint && yarn constraints",
|
"hygiene": "yarn knip && yarn publint && yarn constraints",
|
||||||
"demo": "node --expose-internals --import tsx examples/echo-agent/start.ts",
|
"demo": "node --expose-internals --import tsx examples/echo-agent/start.ts",
|
||||||
|
"demo:coding": "node --expose-internals --import tsx examples/coding-agent/start.ts",
|
||||||
"postinstall": "lefthook install"
|
"postinstall": "lefthook install"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
|
|||||||
@@ -25,10 +25,15 @@ export default defineConfig({
|
|||||||
plugins: [tsconfigPaths({ projects: ['./tsconfig.test.json'] })],
|
plugins: [tsconfigPaths({ projects: ['./tsconfig.test.json'] })],
|
||||||
test: {
|
test: {
|
||||||
include: ['packages/*/tests/**/*.e2e.ts', 'examples/*/tests/**/*.e2e.ts'],
|
include: ['packages/*/tests/**/*.e2e.ts', 'examples/*/tests/**/*.e2e.ts'],
|
||||||
// Real model calls: generous timeouts, one retry for transient flakes,
|
// Real model calls: generous timeouts, and retries for transient flakes
|
||||||
// no coverage (unit suites own the coverage gate).
|
// (the shared internal key hits concurrency quotas). No coverage — the
|
||||||
|
// unit suites own the coverage gate.
|
||||||
testTimeout: 120_000,
|
testTimeout: 120_000,
|
||||||
hookTimeout: 30_000,
|
hookTimeout: 30_000,
|
||||||
retry: 1,
|
retry: 2,
|
||||||
|
// Run e2e files one at a time: the shared internal API key has a small
|
||||||
|
// concurrency quota, and parallel files issue enough simultaneous requests
|
||||||
|
// to trip it (manifesting as flaky rate-limit errors).
|
||||||
|
fileParallelism: false,
|
||||||
},
|
},
|
||||||
})
|
})
|
||||||
|
|||||||
Reference in New Issue
Block a user