docs: purge chain-of-thought leakage from prose
Delete design-session citations (decision/audit/plan ordinals, stack positions), change narration, review choreography, and reviewer-addressed justification from comments, JSDoc, docs, READMEs, Agent Notes, tests, and generator templates; restate every affected fact as current-state contract prose. Fix generated docs at their sources and regenerate the catalogs and cordis-surface regions; re-paste type-equiv blocks; update every bilingual counterpart and re-record the pairs. Record the citation rule in the committed-artifact-citations Agent Note.
This commit is contained in:
Tianyi Cui
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-06-11-property-based-testing.md
|
||||
2026-06-11-property-based-testing.md: a1bd4147a26a3d562899310e238096939fc2d01a
|
||||
2026-06-11-property-based-testing.zh.md: 062b2df76597ba16a7ad0682d57b35e54a07d9bd
|
||||
2026-06-11-property-based-testing.md: 7ab6f2daeb65e603d055ad6127a91f6b76ebc26e
|
||||
2026-06-11-property-based-testing.zh.md: 2fae02356a84b88e750ce1fd8c3003212395b084
|
||||
|
||||
@@ -4,7 +4,7 @@ Status: implemented
|
||||
|
||||
English | [中文](2026-06-11-property-based-testing.zh.md)
|
||||
|
||||
> Merges the original proposal and the decision record for one topic. It found a real BlockAssembler duplicate-`block-end` bug on first run.
|
||||
> The property suite found a real BlockAssembler duplicate-`block-end` bug on its first run.
|
||||
|
||||
## Problem
|
||||
|
||||
@@ -12,7 +12,7 @@ Example-based tests pin the cases we thought of. The harness's core is protocol-
|
||||
|
||||
## Decision
|
||||
|
||||
Adopt `fast-check` (a root devDependency) with one `tests/properties.spec.ts` per protocol-shaped package, generators tuned for *realistic-but-adversarial* inputs (not uniform noise) and `numRuns` kept so the suite stays well under ~10s locally. Failures print a reproducible seed. (The original proposal also sketched a nightly CI job running 100× the iterations; that was not shipped — the property suite runs only in the normal `push`/`pull_request` CI, and a scheduled high-iteration job remains possible future work.)
|
||||
`fast-check` (a root devDependency) powers one `tests/properties.spec.ts` per protocol-shaped package, with generators tuned for *realistic-but-adversarial* inputs (not uniform noise) and `numRuns` kept so the suite stays well under ~10s locally. Failures print a reproducible seed. (A nightly CI job running 100× the iterations is not shipped — the property suite runs only in the normal `push`/`pull_request` CI; a scheduled high-iteration job remains possible future work.)
|
||||
|
||||
- **dsh-llm / BlockAssembler:** arbitrary chunk streams (valid + malformed: duplicate indices, stragglers, missing block-start). Invariants: `blocks()` count ≤ distinct indices seen; re-assembly idempotent (`blocks()` is stable across repeated calls and `message().content` mirrors it); `blocks()` never throws and yields only valid content-block tags; `finish` reflects the last `finish` chunk, defaulting to `{kind:'stop'}` when none arrives.
|
||||
- **dsh-session:** arbitrary event logs. Invariants: `deriveMessages` deterministic; replay-from-seed identical; seq strictly monotonic; non-message events never affect derived history; derived content is decoupled from the log.
|
||||
|
||||
@@ -4,7 +4,7 @@ Status: implemented
|
||||
|
||||
[English](2026-06-11-property-based-testing.md) | 中文
|
||||
|
||||
> 将原始提案与同一主题的决策记录合并为一篇。首次运行即发现了 BlockAssembler 重复 `block-end` 的真实 bug。
|
||||
> 属性测试套件首次运行即发现了 BlockAssembler 重复 `block-end` 的真实 bug。
|
||||
|
||||
## 问题
|
||||
|
||||
@@ -12,7 +12,7 @@ Status: implemented
|
||||
|
||||
## 决策
|
||||
|
||||
引入 `fast-check`（作为根 devDependency），在每个协议形态的包中编写一个 `tests/properties.spec.ts`。生成器调优为*逼真但对抗性*的输入（而非均匀噪声），`numRuns` 控制在本地套件总耗时远低于约 10 秒。失败时打印可复现的 seed。（原始提案还草拟了一个夜间 CI job，以 100 倍迭代运行；该部分未交付。属性测试套件仅在常规的 `push`/`pull_request` CI 中运行，定时高迭代 job 仍属可能的后续工作。）
|
||||
`fast-check`（作为根 devDependency）为每个协议形态的包驱动一个 `tests/properties.spec.ts`。生成器调优为*逼真但对抗性*的输入（而非均匀噪声），`numRuns` 控制在本地套件总耗时远低于约 10 秒。失败时打印可复现的 seed。（以 100 倍迭代运行的夜间 CI job 未交付——属性测试套件仅在常规的 `push`/`pull_request` CI 中运行；定时高迭代 job 仍属可能的后续工作。）
|
||||
|
||||
- **dsh-llm / BlockAssembler：** 任意分片流（合法 + 畸形：重复索引、滞后分片、缺少 block-start）。不变式：`blocks()` 计数 ≤ 已见到的不同索引数；重组幂等（`blocks()` 在重复调用间稳定，且 `message().content` 与之一致）；`blocks()` 从不抛异常且仅产出合法的 content-block 标签；`finish` 反映最后一个 `finish` 分片，无此类分片时默认为 `{kind:'stop'}`。
|
||||
- **dsh-session：** 任意事件日志。不变式：`deriveMessages` 确定性；从 seed 回放结果一致；seq 严格单调递增；非消息事件不影响推导出的历史；推导出的内容与日志解耦。
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md
|
||||
2026-06-19-acp-snapshot-tests.md: c6e81dc4a51083ed37ad6f837fe7dbb08c212e23
|
||||
2026-06-19-acp-snapshot-tests.zh.md: 729e5645b097f8db5b23594e16e48dc5b4cf211f
|
||||
2026-06-19-acp-snapshot-tests.md: 9e03c3c5be7b15a1a34c954dca2f1d6e7a875ba2
|
||||
2026-06-19-acp-snapshot-tests.zh.md: 59033a2fd85bc08bfe4abe34e8fd9aff9a67c87e
|
||||
|
||||
@@ -8,9 +8,7 @@ English | [中文](2026-06-19-acp-snapshot-tests.zh.md)
|
||||
|
||||
Unit tests do not exercise the complete assembled-agent subprocess or its ACP automation wire, while real-API tests are nondeterministic and key-gated. Loader wiring, backend behavior, and protocol output can therefore regress despite green unit coverage, as the [default-export postmortem](../../../../docs/postmortem/0001-acp-default-export-drops-inject.md) demonstrated.
|
||||
|
||||
The blocker for a full-transcript test is the model: the agent's output is driven by a non-deterministic LLM, and a key-gated test that hits the real API on every run is neither deterministic nor CI-runnable. We want the fidelity of a real run with the determinism of a fixture.
|
||||
|
||||
This Agent Note records the decision to add a third test tier — **snapshot tests** — and the design choices that make it deterministic, keyless-in-CI, and cheap to maintain.
|
||||
The blocker for a full-transcript test is the model: the agent's output is driven by a non-deterministic LLM, and a key-gated test that hits the real API on every run is neither deterministic nor CI-runnable. The tier needs the fidelity of a real run with the determinism of a fixture.
|
||||
|
||||
## Decision
|
||||
|
||||
|
||||
@@ -8,9 +8,7 @@ Status: implemented
|
||||
|
||||
单元测试不会覆盖组装后的完整 agent（智能体）子进程及其 ACP（Agent Client Protocol）自动化协议格式，而真实 API 测试不具确定性且受密钥门控。因此，即使单元测试覆盖率检查通过，Loader 接线、后端行为和协议输出仍可能回归，[默认导出事故复盘（postmortem）](../../../../docs/postmortem/0001-acp-default-export-drops-inject.md)已经证明了这一点。
|
||||
|
||||
完整 transcript（文本记录）测试的阻塞因素在于模型：agent 的输出由非确定性的 LLM（大语言模型）驱动，而每次运行都命中真实 API 的密钥门控测试既不确定也无法在 CI 中运行。我们需要真实运行的保真度与 fixture（测试前置数据）的确定性兼得。
|
||||
|
||||
本 Agent Note 记下了新增第三层测试——**快照测试**——的决策，以及让它具备确定性、在 CI 中无需密钥、且维护成本低廉的设计选择。
|
||||
完整 transcript（文本记录）测试的阻塞因素在于模型：agent 的输出由非确定性的 LLM（大语言模型）驱动，而每次运行都命中真实 API 的密钥门控测试既不确定也无法在 CI 中运行。该测试层级需要真实运行的保真度与 fixture（测试前置数据）的确定性兼得。
|
||||
|
||||
## 决策
|
||||
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-06-19-real-api-e2e-ci.md
|
||||
2026-06-19-real-api-e2e-ci.md: 935664fd01df4844ee19be7b4f2f297ebf5bd29b
|
||||
2026-06-19-real-api-e2e-ci.zh.md: 4245cc9a3872cf2ab27633db5b23fc466d793934
|
||||
2026-06-19-real-api-e2e-ci.md: feb5d83f087f66dfec7e1540f6b32e33656409d4
|
||||
2026-06-19-real-api-e2e-ci.zh.md: c00aa6e962d9e86a9d5d5c1ccaf1001afa21ba67
|
||||
|
||||
@@ -10,11 +10,9 @@ The harness leans hard on real-API tests by policy: [docs/testing.md](../../../.
|
||||
|
||||
The default gate ([.github/workflows/ci.yml](../../../../.github/workflows/ci.yml)) is deliberately keyless: it carries no secret and runs for forks. `test:e2e` self-skips without a key (`describe.skipIf(!process.env.DEEPSEEK_API_KEY)`), so adding it there would report green without exercising the real suite. A separate secret-bearing workflow is required to make real-API coverage a merge signal.
|
||||
|
||||
This Agent Note records the decision to add a **second, secret-consuming workflow** that runs the real-API suite in CI, and — because introducing the first CI secret into a repo that may later go public is a security/isolation decision — the threat model it relies on and what changes when the repo becomes public.
|
||||
|
||||
## Decision
|
||||
|
||||
Add a dedicated workflow, [.github/workflows/e2e.yml](../../../../.github/workflows/e2e.yml), separate from ci.yml. It runs only `pnpm run test:e2e` against the external API using a repo secret, on trusted events, with a preflight that converts a missing secret into a loud failure instead of a false green. The keyless workflow remains separate so forkable quality gates and secret-consuming real-API gates keep different trigger and credential policies.
|
||||
A dedicated workflow, [.github/workflows/e2e.yml](../../../../.github/workflows/e2e.yml), separate from ci.yml, runs only `pnpm run test:e2e` against the external API using a repo secret, on trusted events, with a preflight that converts a missing secret into a loud failure instead of a false green. The keyless workflow remains separate so forkable quality gates and secret-consuming real-API gates keep different trigger and credential policies.
|
||||
|
||||
### A separate workflow, not a job in ci.yml
|
||||
|
||||
|
||||
@@ -10,11 +10,9 @@ Status: implemented
|
||||
|
||||
默认门禁（[.github/workflows/ci.yml](../../../../.github/workflows/ci.yml)）刻意无密钥：不携带 secret，可供 fork 运行。`test:e2e` 在无密钥时自动跳过（`describe.skipIf(!process.env.DEEPSEEK_API_KEY)`），因此将其加入该工作流只会报绿而不会真正执行真实套件。要让真实 API 覆盖率成为合并信号，需要一个独立的、携带 secret 的工作流。
|
||||
|
||||
本 Agent Note 记下了新增**第二条消费 secret 的工作流**以在 CI 中运行真实 API 套件的决策；由于向未来可能公开的仓库引入第一个 CI secret 属于安全/隔离决策，本文也记录其依赖的威胁模型，以及仓库公开时需要做出的变更。
|
||||
|
||||
## 决策
|
||||
|
||||
添加一个专用工作流 [.github/workflows/e2e.yml](../../../../.github/workflows/e2e.yml)，与 ci.yml 分离。它仅使用 repo secret 对外部 API 运行 `pnpm run test:e2e`，仅在可信事件上触发，并带有一个 preflight 检查：将缺失的 secret 转化为明确的失败而非虚假的绿色。无密钥工作流保持独立，使可 fork 的质量门禁与消费 secret 的真实 API 门禁各自拥有不同的触发和凭证策略。
|
||||
一个与 ci.yml 分离的专用工作流 [.github/workflows/e2e.yml](../../../../.github/workflows/e2e.yml) 仅使用 repo secret 对外部 API 运行 `pnpm run test:e2e`，仅在可信事件上触发，并带有一个 preflight 检查：将缺失的 secret 转化为明确的失败而非虚假的绿色。无密钥工作流保持独立，使可 fork 的质量门禁与消费 secret 的真实 API 门禁各自拥有不同的触发和凭证策略。
|
||||
|
||||
### 独立工作流，而非 ci.yml 中的一个 job
|
||||
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-06-22-subagent-snapshot-replay.md
|
||||
2026-06-22-subagent-snapshot-replay.md: 45034c29f608eb5b58f44b80438b801b6224cb82
|
||||
2026-06-22-subagent-snapshot-replay.zh.md: 1b90c1bda854b7cd8cd9eae1e8c9f34328477f15
|
||||
2026-06-22-subagent-snapshot-replay.md: 187831b2d3c2f08b32289983afbbe0bc03484c82
|
||||
2026-06-22-subagent-snapshot-replay.zh.md: b96be12df0ea1f1a6bb64dd787fec6880eb96243
|
||||
|
||||
@@ -13,7 +13,7 @@ It was built for ONE session per process, and that assumption is wired into two
|
||||
- **`dsh-llm-replay` keyed nothing.** It served the Nth `llm/stream` call the Nth recorded entry from a single global cursor. With a parent agent AND an in-process subagent both streaming on one context, the calls interleave and the single cursor hands the child the parent's script (and vice versa).
|
||||
- **The harness harvested one log.** `findSessionLog` walked the sessions root and returned the FIRST `.jsonl` it found. A subagent runs as a second `Session` with its own log, so the child's transcript was silently dropped.
|
||||
|
||||
This was the `TODO(subagent-snapshots)` deferral recorded in the [subagent seam Agent Note](../feature/2026-06-21-subagent-capability-seam.md): the in-process backends (PR2) shipped with unit + e2e coverage, but the full-transcript snapshot tier could not express a nested-agent shape until this infrastructure landed. This Agent Note is that stacked follow-up.
|
||||
This was the `TODO(subagent-snapshots)` deferral recorded in the [subagent seam Agent Note](../feature/2026-06-21-subagent-capability-seam.md): the in-process backends shipped with unit + e2e coverage, but the full-transcript snapshot tier could not express a nested-agent shape until this infrastructure landed.
|
||||
|
||||
## Decision
|
||||
|
||||
@@ -55,4 +55,4 @@ Both replay keyless in the default gate.
|
||||
- The `TODO(subagent-snapshots)` deferral is resolved: nested-agent transcripts are now a first-class snapshot shape.
|
||||
- `GenerateOptions.sessionId` is a small, honest core API addition useful beyond replay (telemetry, request routing).
|
||||
- The `subagent` tool is bound to a single provider, so both children in `subagent-multi` are spawn (fresh). The keying routes by session, not by backend, so it is already correct for fork. The script *derivation* was not: a fork child's log begins with the seeded parent prefix (the parent's `assistant/chunk` events), so deriving its script from the whole log would replay the parent's responses as the child's. That correctness gap is closed by persisting a seed boundary — see [Persist the seed boundary so fork-child replay routes correctly](2026-06-22-fork-child-replay-seed-boundary.md) — and recorded fork + mixed spawn+fork scenarios now exercise both transports through one transcript (see [Record fork and mixed spawn+fork snapshot scenarios](../../archived/testing/2026-06-22-fork-snapshot-scenarios.md)).
|
||||
- Out-of-process (ACP) subagents are a different replay shape entirely (each child is its own PROCESS with its own replay), tracked as `TODO(acp-subagent-replay)` in the PR3 plan.
|
||||
- Out-of-process (ACP) subagents are a different replay shape entirely (each child is its own PROCESS with its own replay), tracked as `TODO(acp-subagent-replay)` in `subagent-acp`.
|
||||
|
||||
@@ -13,7 +13,7 @@ Status: implemented
|
||||
- **`dsh-llm-replay` 没有做任何键控。** 它用一个全局游标，将第 N 次 `llm/stream` 调用对应到单一录制序列的第 N 条。当父 agent（智能体）和一个进程内 subagent 在同一个上下文上同时流式输出时，调用交错，单一游标会把子 agent 的脚本发给父 agent（反之亦然）。
|
||||
- **harness 只收集一份日志。** `findSessionLog` 遍历 sessions 根目录，返回找到的第一个 `.jsonl`。subagent 作为第二个 `Session` 运行并拥有自己的日志，因此子 agent 的 transcript（文本记录）被静默丢弃。
|
||||
|
||||
这就是 [subagent seam Agent Note](../feature/2026-06-21-subagent-capability-seam.md)中通过 `TODO(subagent-snapshots)` 推迟的工作：进程内后端（PR2）落地时已有单元 + e2e 覆盖，但在这套基础设施落地前，完整 transcript 快照层无法表达嵌套 agent 形状。本 Agent Note 就是该堆叠式后续工作。
|
||||
这就是 [subagent seam Agent Note](../feature/2026-06-21-subagent-capability-seam.md)中通过 `TODO(subagent-snapshots)` 推迟的工作：进程内后端落地时已有单元 + e2e 覆盖，但在这套基础设施落地前，完整 transcript 快照层无法表达嵌套 agent 形状。
|
||||
|
||||
## 决策
|
||||
|
||||
@@ -55,4 +55,4 @@ Status: implemented
|
||||
- `TODO(subagent-snapshots)` 延期项已解决：嵌套 agent 的 transcript 现在是快照层的一等形态。
|
||||
- `GenerateOptions.sessionId` 是一个小而诚实的 core API 新增，在回放之外同样有用（遥测、请求路由）。
|
||||
- `subagent` 工具绑定到单一提供方，因此 `subagent-multi` 中的两个子 agent 都是 spawn（全新创建）。键控按会话路由而非按后端路由，因此对 fork 同样正确。但脚本*派生*逻辑此前不正确：fork 子会话的日志以种子化的父前缀（父会话的 `assistant/chunk` 事件）开头，如果从完整日志派生脚本，就会把父 agent 的响应当作子 agent 的来回放。这一正确性缺口通过持久化种子边界来弥合——见[持久化 seed 边界以确保 fork 子会话回放正确路由](2026-06-22-fork-child-replay-seed-boundary.md)——录制的 fork 与混合 spawn+fork 场景现在通过一份 transcript 同时验证两种传输方式（见[记录 fork 与混合 spawn+fork 快照场景](../../archived/testing/2026-06-22-fork-snapshot-scenarios.md)）。
|
||||
- 进程外（ACP（Agent Client Protocol））subagent 是完全不同的回放形态（每个子 agent 是自己的进程、有自己的回放），作为 `TODO(acp-subagent-replay)` 记录在 PR3 计划中。
|
||||
- 进程外（ACP（Agent Client Protocol））subagent 是完全不同的回放形态（每个子 agent 是自己的进程、有自己的回放），作为 `TODO(acp-subagent-replay)` 记录在 `subagent-acp` 中。
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md
|
||||
2026-07-24-web-gui-browser-e2e-lane.md: aa289c9a70623904606e2d06b6492acb3a7d98a3
|
||||
2026-07-24-web-gui-browser-e2e-lane.zh.md: 9e26dcb53f0373ac967c627b95b895385784f925
|
||||
2026-07-24-web-gui-browser-e2e-lane.md: 84912895514f9dba4ff61bc8842444a81c5fbefa
|
||||
2026-07-24-web-gui-browser-e2e-lane.zh.md: 3c94b42d1b85190dfe6b3b8ebb14efb4624fd326
|
||||
|
||||
@@ -72,7 +72,7 @@ Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot
|
||||
|
||||
**Changing the wire protocol for testability.** Rejected: the contract already has a first-class keyless in-process path (`InProcessApiClient(toFetchHandler(api))`), the per-event unbatched SSE is exactly what makes replay observable in a browser, and testing a wire we no longer ship would invert the tier's purpose.
|
||||
|
||||
**Real-model browser tests as the keyless lane.** Rejected: nondeterministic by construction; the surveyed cautionary case (open-webui) grew unbounded timeouts and was deleted. The with-key W5 smoke stays as the live-model complement.
|
||||
**Real-model browser tests as the keyless lane.** Rejected: nondeterministic by construction; the surveyed cautionary case (open-webui) grew unbounded timeouts and was deleted. The with-key real-host smoke stays as the live-model complement.
|
||||
|
||||
**Running the high-cardinality performance case in the required browser gate.** Rejected: its fixture setup and full-history render add tens of seconds, while wall-clock and memory values vary with the host and cannot supply a stable correctness threshold. The required lane keeps deterministic behavior assertions; contributors run the diagnostic case when investigating or changing large-list and long-history rendering.
|
||||
|
||||
|
||||
@@ -72,7 +72,7 @@ Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bu
|
||||
|
||||
**为可测试性改 wire 协议。** 已否决：约定已有第一等的无密钥进程内路径（`InProcessApiClient(toFetchHandler(api))`），逐事件不合批的 SSE 恰是回放在浏览器中可观测的原因，测试一条不再交付的 wire 会颠倒该层的存在意义。
|
||||
|
||||
**以真实模型浏览器测试充当无密钥车道。** 已否决：按构造即不确定；被调研的前车之鉴（open-webui）长出无界超时后被删除。带密钥的 W5 冒烟仍是真实模型侧的补充。
|
||||
**以真实模型浏览器测试充当无密钥车道。** 已否决：按构造即不确定；被调研的前车之鉴（open-webui）长出无界超时后被删除。带密钥的真实 host 冒烟仍是真实模型侧的补充。
|
||||
|
||||
**在必需的浏览器门禁中运行高基数性能用例。** 已否决：其 fixture 设置和完整历史渲染会增加数十秒耗时，而壁钟时间和内存值随 host 不同而变化，无法提供稳定的正确性阈值。必需车道保留确定性行为断言；贡献者在调查或更改大列表和长历史渲染时运行该诊断用例。
|
||||
|
||||
|
||||
Reference in New Issue
Block a user