docs: document package model experience

This commit is contained in:
Tianyi Cui
2026-07-12 02:12:36 +08:00
parent 3dca90261c
commit 027970043c
75 changed files with 561 additions and 5 deletions

View File

@@ -40,3 +40,9 @@ A scenario booting a differently-composed tree sets its own `configPath` (an ove
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config RFC](../../../docs/rfc/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log goldens, and each pin's Markdown prompt snapshot from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
Constraints: `suite.ts` imports vitest, so the package is importable only inside a vitest run (the harness and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the harness speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug). Session config options are scriptable too: the `setConfigOption` step switches a knob over `session/set_config_option`, and `setConfigOptionExpectError` asserts the bridge rejects an unknown id or out-of-vocabulary value (the error frame stays in the transcript).
## Model Experience
| Context surface | What the model sees | Token effect |
|---|---|---|
| None in production | The test harness records, normalizes, scrubs, and compares request headers and ACP transcripts but does not alter the agent's assembled context. Replay scenarios obtain assistant chunks from `dsh-llm-replay`; record mode uses the real composition. | Zero production tokens. Replay spends no provider tokens, while record mode pays the composition's ordinary model cost; golden scrubbing changes files only, never the live request. |

View File

@@ -4,6 +4,12 @@ Dev-mode event-contract invariants and session-log freeze. A pure-listener plugi
**Off in production.** Enable it in tests and the demos, where a contract violation should fail loudly. It costs nothing when not registered, and doubles as executable documentation of the event taxonomy — the assertions *are* the contract.
## Model Experience
| Context surface | What the model sees | Token effect |
|---|---|---|
| None | The plugin observes and validates session events, agent states, and frozen model requests; it does not rewrite a prompt, schema, message, or stream. An invariant failure aborts the faulty turn instead of adding guidance. | Zero model tokens when checks pass; a failure prevents or ends a request rather than contributing context. |
## Plugin
A functional plugin — register the module namespace (this is what loading by name in `cordis.yml` does):

View File

@@ -4,6 +4,12 @@ A replay LLM plugin for keyless snapshot tests. It installs a single `llm/stream
Its consumer is the ACP snapshot harness in `examples/acp-agent`, which loads this plugin (via `cordis.snapshot.yml`) in place of a real LLM adapter. The package exists so its derive/parse/replay logic falls under the per-file 100% coverage gate on `packages/*/src` (the same logic, while it lived under `examples/`, was outside the gate).
## Model Experience
| Context surface | What the model sees | Token effect |
|---|---|---|
| Keyless test stream | The real loop still assembles its normal system prompt, tools, prefix, and history, but no provider model receives them. Recorded assistant chunks are replayed as the response and then enter later history exactly like live output. | Zero billed or tokenizer-evaluated model tokens. Fixture output creates deterministic retained test context for later replay steps. |
## How the fixture works
The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`", done by the snapshot harness — this plugin does not record. A fixture may carry its `request/header` content tokenized to `{{system}}`/`{{tools}}` (the harness pins that content in one scenario and scrubs the rest); replay is indifferent — derivation reads only `assistant/chunk` events and the line-0 session header.

View File

@@ -4,6 +4,12 @@ A scripted `SubagentProvider` for testing the [subagent seam](../../subagent/sub
It lets a test drive `ctx.subagents` and the model-facing `dsh-tool-subagent` through the **real cordis Loader / export path**, exercising provider registration, start-time capability validation, the run lifecycle (`result` / `cancel` / `dispose`), and the structured-output branch — all deterministically and keylessly.
## Model Experience
| Context surface | What the model sees | Token effect |
|---|---|---|
| Deterministic test result | No child model runs. When exercised through `dsh-tool-subagent`, the parent sees the mock provider's configured reply or stop-reason error, and structured tests receive the configured object. | Zero child-model tokens. Only the scripted final result is added to the parent test history. |
## Usage
Load it as a plugin (functional shape: `name`/`inject`/`Config`/`apply`, no default). Config (all optional):