Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Pin request-header content in one snapshot scenario

Browse Source 
Every session.jsonl fixture embedded the full composed system prompt and
complete tool-schema list in its request/header event (~8 KB on one line,
identical across the suite), so any prompt or tool-schema edit forced a
re-record or hand-edit of every fixture — see the dynamic-workflows PR for
the churn pattern this removes.

Now exactly one scenario (text-turn, flagged pinsHeader) commits and
compares that content verbatim; every other fixture stores and compares it
as {{system}}/{{tools}} tokens via the new pure scrubRequestHeaders
normalizer (applied to both compare sides and to record-mode writes, so a
re-record cannot reintroduce the content). request/header-delta payloads
are scrubbed the same way; config/reason stay verbatim — a model swap
SHOULD churn every fixture, a prompt edit should not. Replay is unaffected:
script derivation reads only assistant/chunk events.

Fixture meta-guards enforce the split: non-pinning fixtures must be fixed
points of the scrub, the pinning fixture must not be, and exactly one
scenario pins. Committed fixtures migrated through the same function.

Docs: pinned-header RFC (implemented/testing), base snapshot RFC + testing
policy + llm-replay module doc/README updated.
This commit is contained in:
Tianyi Cui
2026-07-06 23:41:20 +08:00
parent 0606cd559c
commit 1fd9e2a9ab
43 changed files with 254 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
packages/support/llm-replay/README.md
View File

@@ -6,7 +6,7 @@ Its consumer is the ACP snapshot harness in `examples/acp-agent`, which loads th
## How the fixture works
The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`", done by the snapshot harness — this plugin does not record.
The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`", done by the snapshot harness — this plugin does not record. A fixture may carry its `request/header` content tokenized to `{{system}}`/`{{tools}}` (the harness pins that content in one scenario and scrubs the rest); replay is indifferent — derivation reads only `assistant/chunk` events and the line-0 session header.
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`: a `ReplayEntry[]`) that REPLACES the derived script.

6
packages/support/llm-replay/src/index.ts
View File

@@ -12,7 +12,11 @@
* `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model
* call per loop step — see packages/core/agent-loop/src/loop.ts). Recording is
* therefore "run the real agent once and harvest the `.jsonl`", done by the
* snapshot harness — this plugin does not record.
* snapshot harness — this plugin does not record. A fixture may carry its
* `request/header` content tokenized to `{{system}}`/`{{tools}}` (the harness
* pins that content in one scenario and scrubs the rest); replay is
* indifferent — derivation reads ONLY `assistant/chunk` events and the line-0
* session header.
*
* A NESTED-agent scenario records more than one log: the parent plus one per
* in-process subagent (each subagent runs as its own {@link Session} on the same