Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(replay): mark local compaction calls

Browse Source
This commit is contained in:
Tianyi Cui
2026-08-08 02:16:17 +08:00
parent d301dbdc7d
commit 8dcfe5c406
29 changed files with 147 additions and 50 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.i18n.yaml
View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md
2026-06-19-acp-snapshot-tests.md: c7b95bd68027705b99d850d596405e56eea0dfca
2026-06-19-acp-snapshot-tests.zh.md: 43d43262684920cae5feedb5f2eb109db00f9f8c
2026-06-19-acp-snapshot-tests.md: c6e81dc4a51083ed37ad6f837fe7dbb08c212e23
2026-06-19-acp-snapshot-tests.zh.md: 5a579c3373442dc4cfc7d661ea2f2f398b804dc7

7
.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md
View File

@@ -24,7 +24,7 @@ Every committed session-format fixture uses the canonical packed physical layout
### Replay derives the model script from the log
`llm-replay` short-circuits the provider-agnostic `llm/stream` waterfall. `deriveReplayScript()` groups recorded chunks by `(turn, step)` and serves one group per model call. The loop makes one stream call per step, so the grouping is exact and includes error finish chunks without special handling.
`llm-replay` short-circuits the provider-agnostic `llm/stream` waterfall. `deriveReplayScript()` splits recorded `assistant/chunk` events at terminal `finish` chunks and uses `(turn, step)` changes to reject an unterminated prior call. A `compact/summary` with `llmStreamCall: true` contributes one call at its durable log position: replay reconstructs canonical block boundaries from `rawOutput`, retains recorded usage when present, and supplies a terminal `stop`. The marker distinguishes that local call from template or remote summaries whose retained `rawOutput` did not consume this context's adapter.
### The in-memory replay entry honors the full LLM contract
@@ -36,7 +36,7 @@ Every committed session-format fixture uses the canonical packed physical layout
| { kind: 'hang' }
```
Logs derive chunk entries. Pre-stream throws and hangs have no reconstructable chunk representation, so those scenarios provide `replay.override.json`. A throw entry may include prefix chunks for mid-stream failure. Explicit overrides avoid inferring adapter behavior from lossy turn-end reasons.
Logs derive chunk entries from finished assistant streams and explicitly marked compaction calls. Pre-stream throws, hangs, and external summarizer calls have no reconstructable local chunk representation, so those scenarios provide `replay.override.json`. A throw entry may include prefix chunks for mid-stream failure. Explicit overrides avoid inferring adapter behavior from lossy turn-end reasons or provider output alone.
### Positional replay, one in-flight stream
@@ -74,12 +74,13 @@ Tool determinism comes from a generated cwd, scrubbed environment, fresh non-log
## Alternatives considered
- **A hand-authored `llm.json` of model chunks** — the earlier draft; reusing the real session log makes the fixture a genuine product of the system rather than a hand-built mock, and doubles it as a behavioral expected output.
- **A compulsory replay override for every compaction summary** — rejected: the durable summary event already fixes a successful local call's position, complete output, and optional usage. An explicit local-call marker preserves that single-source fixture without inventing a call for template or remote summarizers.
- **A byte-level HTTP-record library (Polly/nock/MSW)** — rejected: adapter-specific, awkward with streaming SSE, and lower-level than the thing under test.
- **Synthesizing throw/cancel entries from `turn/end {kind:'error'|'aborted'}`** — rejected: it couples `llm-replay` to loop-internal turn-closing semantics, and the `turn/end` reason is lossy (it cannot distinguish a thrown 401 from a finish-error); the explicit `replay.override.json` sidecar is the cleaner seam.
- **Copying both request-header sidecars beside every class pin** — rejected: prompt and tool-schema composition vary independently, so a change to one shared component would churn byte-identical files across unrelated class pins. Explicit per-component sources retain one structural pin per class without duplicating content.
## Consequences
The tier adds reviewed per-scenario input, session, stdout, optional override, and optional workspace fixtures, plus one file for each distinct pinned prompt and tool-schema sequence. Workspace seeds are copied into the generated cwd for both record and replay. In return the tier provides deterministic keyless coverage through the real Loader and tool composition. Most retained scenarios exercise the assembled backend rather than ACP; the [automation-only ACP decision](../simplification/2026-07-23-acp-automation-only-protocol.md#snapshot-boundary) keeps that corpus here until it can move to a transport-neutral headless suite without losing coverage.
The tier adds reviewed per-scenario input, session, stdout, optional override, and optional workspace fixtures, plus one file for each distinct pinned prompt and tool-schema sequence. Workspace seeds are copied into the generated cwd for both record and replay. In return the tier provides deterministic keyless coverage through the real Loader and tool composition, including an assembled context-overflow recovery whose marked compaction summary supplies the auxiliary call. Most retained scenarios exercise the assembled backend rather than ACP; the [automation-only ACP decision](../simplification/2026-07-23-acp-automation-only-protocol.md#snapshot-boundary) keeps that corpus here until it can move to a transport-neutral headless suite without losing coverage.
This Agent Note relates to but does not supersede the [proposed determinism Agent Note](../../proposed/testing/2026-06-11-deterministic-and-stress-testing.md): that proposal's "universal replay fixture" re-derives session *message history* after every test (an internal-consistency invariant), whereas these snapshots pin assembled behavior plus the external automation output. They are complementary until the backend corpus moves off ACP.

7
.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.zh.md
View File

@@ -24,7 +24,7 @@ Status: implemented
### 回放从日志推导模型脚本
`llm-replay` 短路了提供方无关的 `llm/stream` waterfall（瀑布式事件）。`deriveReplayScript()` 按 `(turn, step)` 对已录制的分片分组，每次模型调用服务一组。agent loop（智能体循环）每个步骤发起一次流调用，因此分组精确对应，错误结束分片也无需特殊处理。
`llm-replay` 短路了提供方无关的 `llm/stream` waterfall（瀑布式事件）。`deriveReplayScript()` 在终止的 `finish` 分片处切分已记录的 `assistant/chunk` 事件，并用 `(turn, step)` 变化拒绝前一条未终止的调用。携带 `llmStreamCall: true` 的 `compact/summary` 会在其持久日志位置贡献一次调用：回放根据 `rawOutput` 重建规范块边界，保留已记录的 usage（如有），并提供终止的 `stop`。该标记将这次本地调用与模板摘要或远程摘要区分开；后两者即使保留了 `rawOutput`，也未使用此上下文的适配器。
### 内存中的回放条目遵守完整的 LLM 契约
@@ -36,7 +36,7 @@ Status: implemented
| { kind: 'hang' }
```
日志推导出分片条目。流开始前的抛出和挂起没有可重建的分片表示，因此这些场景提供 `replay.override.json`。throw 条目可以包含前缀分片以模拟流中途失败。显式覆盖避免了从有损的轮次结束原因推断适配器行为。
日志从已结束的 assistant 流和显式标记的压缩（compaction）调用推导分片条目。流开始前的抛出、挂起和外部摘要器调用没有可重建的本地分片表示，因此这些场景提供 `replay.override.json`。throw 条目可以包含前缀分片以模拟流中途失败。显式覆盖避免了从有损的轮次结束原因或单独的提供方输出推断适配器行为。
### 位置式回放，单个在途流
@@ -74,12 +74,13 @@ Status: implemented
## 曾考虑的替代方案
- **手工编写包含模型分片的 `llm.json`**——早期草案；复用真实会话日志，使 fixture 成为系统的真实产物而非手工构建的 mock，并让它同时充当行为预期输出。
- **为每个压缩摘要强制提供回放 override**——否决：持久摘要事件已经固定成功本地调用的位置、完整输出与可选 usage。显式的本地调用标记保留了这份单一来源 fixture，而不会为模板摘要器或远程摘要器凭空构造调用。
- **字节级 HTTP 录制库（Polly/nock/MSW）**：否决。与适配器耦合，处理流式 SSE（Server-Sent Events）时笨拙，且层级低于被测对象。
- **从 `turn/end {kind:'error'|'aborted'}` 合成抛错/取消条目**：否决。这会将 `llm-replay` 耦合到 loop 内部的轮次关闭语义，且 `turn/end` 原因是有损的（无法区分抛出的 401 与 finish-error）；显式的 `replay.override.json` 伴随文件是更清晰的 seam。
- **在每个类别 pin 旁复制两个请求头伴随文件**：否决。提示词与工具 schema 的组合各自独立变化，因此一个共享组件发生变更，就会使不相关类别 pin 中字节完全相同的文件产生无意义改动。显式的分组件来源可在不重复内容的情况下，为每个类别保留一个结构性 pin。
## 后果
该测试层为每个场景增加经过评审的输入、会话、stdout、可选 override 和可选 workspace fixture，并为每个不同的已固定提示词序列、每个不同的已固定工具 schema 序列各增加一个文件。记录与回放都会把 workspace seed 复制到生成的 cwd。作为回报，该层通过真实 Loader 和工具组合提供确定性的无密钥覆盖。保留下来的大多数场景测试的是组装后的后端而非 ACP；[仅面向自动化的 ACP 决策](../simplification/2026-07-23-acp-automation-only-protocol.md#snapshot-boundary)将该语料保留在此处，直至它能够在不损失覆盖的情况下迁移到传输无关的 headless 套件。
该测试层为每个场景增加经过评审的输入、会话、stdout、可选 override 和可选 workspace fixture，并为每个不同的已固定提示词序列、每个不同的已固定工具 schema 序列各增加一个文件。记录与回放都会把 workspace seed 复制到生成的 cwd。作为回报，该层通过真实 Loader 和工具组合提供确定性的无密钥覆盖，其中包括一个组装后的上下文溢出恢复场景，其带标记的压缩摘要提供辅助调用。保留下来的大多数场景测试的是组装后的后端而非 ACP；[仅面向自动化的 ACP 决策](../simplification/2026-07-23-acp-automation-only-protocol.md#snapshot-boundary)将该语料保留在此处，直至它能够在不损失覆盖的情况下迁移到传输无关的 headless 套件。
本 Agent Note 与[拟议的确定性 Agent Note](../../proposed/testing/2026-06-11-deterministic-and-stress-testing.md)相关，但不取代它：该提案的“通用回放 fixture”在每次测试后重新派生会话*消息历史*（内部一致性不变量），而这些快照固定组装后的行为与外部自动化输出。在后端语料迁出 ACP 之前，两者相互补充。