test(python): pin the minimal composition's model-visible surface
The Python lane never compared what the minimal composition shows the model: the mock model only asserted system-role messages, and the advanced snapshot tokenizes the assembled system prompt and tool schemas. The sdk-minimal scenario now records model-visible.json — every model request's advertised tool schemas verbatim and its message list, with system and user text kept and assistant/tool payloads reduced to call identity so the expected output replays on macOS and Linux. It excludes the dynamic runtime-context snapshot, which the same composition emits on macOS and not on Linux (#2488). AGENTS.md and the testing policy name both SDKs as independent projections of the agent loop, session lifecycle, and SessionEventMap.
This commit is contained in:
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write docs/testing.md
|
||||
testing.md: 8ed81412a954bcd1d4b5d84eb4133ed081092764
|
||||
testing.zh.md: 5662db51847ba2f4e297d0bd3d4af0d024fa29fb
|
||||
testing.md: bef73983e48365f6655e4d4242f4f73223d971ed
|
||||
testing.zh.md: 7a5935165a335b4f3885092ef7b7b29f08f53c51
|
||||
|
||||
@@ -46,4 +46,4 @@ An e2e assertion re-runs the command or re-reads the file externally; a keyword
|
||||
|
||||
## When a snapshot test is required
|
||||
|
||||
Every non-trivial model-, protocol-, or human-visible change adds or updates a keyless scenario in the same PR through a runnable example's owning snapshot suite. Package tests, e2e assertions, mock/test-only compositions, and PR rationale do not replace the assembled transcript; extend the harness when needed. ACP automation scenarios use `examples/<name>/tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/test-support/acp-snapshot/README.md) suite factory (`examples/acp-agent` is primary); `examples/headless-agent` owns the internal canonical-event JSONL snapshots and replay fixtures. The `pwsh-tool-turn` ACP scenario boots real `pwsh` and skips where it is absent. Completed interactive-terminal journeys use JSONL-driven scenarios under `apps/cli/tests/snapshots/`; transient presentation uses the package-local semantic matrix, with a PTY case when input, Loader selection, or terminal teardown changes. Browser-rendered web GUI journeys use `apps/web/tests/snapshots/`. New capability seams, lifecycle variants, or transcript surfaces name every coverage tier at plan time and verify the harness can express it before implementation.
|
||||
Every non-trivial model-, protocol-, or human-visible change adds or updates a keyless scenario in the same PR through a runnable example's owning snapshot suite. Package tests, e2e assertions, mock/test-only compositions, and PR rationale do not replace the assembled transcript; extend the harness when needed. ACP automation scenarios use `examples/<name>/tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/test-support/acp-snapshot/README.md) suite factory (`examples/acp-agent` is primary); `examples/headless-agent` owns the internal canonical-event JSONL snapshots and replay fixtures. The `pwsh-tool-turn` ACP scenario boots real `pwsh` and skips where it is absent. Completed interactive-terminal journeys use JSONL-driven scenarios under `apps/cli/tests/snapshots/`; transient presentation uses the package-local semantic matrix, with a PTY case when input, Loader selection, or terminal teardown changes. Browser-rendered web GUI journeys use `apps/web/tests/snapshots/`. The two SDKs project the agent loop, session lifecycle, and `SessionEventMap` independently, so changing any of those updates both: `examples/jsonrpc-agent/tests/snapshots/` owns the TypeScript client; `scripts/snapshots/python-sdk-single-exe/` owns the Python client, which only the required `python-runtime` CI job runs. New capability seams, lifecycle variants, or transcript surfaces name every coverage tier at plan time and verify the harness can express it before implementation.
|
||||
|
||||
@@ -46,4 +46,4 @@ e2e 断言应重新运行命令或从外部重新读取文件;对 agent 自身
|
||||
|
||||
## 何时需要快照测试
|
||||
|
||||
每项非平凡的模型可见、协议可见或人类可见变更,都必须在同一 PR 中,通过可运行示例所属的快照套件添加或更新无密钥场景。包测试、e2e 断言、mock 与仅测试组合、PR 理由都不能取代组装后的 transcript;必要时应扩展 harness。ACP 自动化场景使用 `examples/<name>/tests/snapshots/`,即基于 [`dsh-acp-snapshot`](../packages/test-support/acp-snapshot/README.md) 套件工厂的场景表(`examples/acp-agent` 为主套件);`examples/headless-agent` 拥有内部规范事件 JSONL 快照与回放 fixture。`pwsh-tool-turn` ACP 场景启动真实 `pwsh`,在无 `pwsh` 的主机上跳过。已完成的交互式终端旅程使用 `apps/cli/tests/snapshots/` 下由 JSONL 驱动的场景;瞬态呈现使用包内语义矩阵,输入、Loader 选择或终端清理发生变化时还要添加 PTY 用例。浏览器渲染的 Web GUI 旅程使用上述 Web 应用快照套件。新的能力 seam、生命周期变体或 transcript 呈现接口在计划阶段就要列出每个覆盖层级,并在实现前验证 harness 能够表达它们。
|
||||
每项非平凡的模型可见、协议可见或人类可见变更,都必须在同一 PR 中,通过可运行示例所属的快照套件添加或更新无密钥场景。包测试、e2e 断言、mock 与仅测试组合、PR 理由都不能取代组装后的 transcript;必要时应扩展 harness。ACP 自动化场景使用 `examples/<name>/tests/snapshots/`,即基于 [`dsh-acp-snapshot`](../packages/test-support/acp-snapshot/README.md) 套件工厂的场景表(`examples/acp-agent` 为主套件);`examples/headless-agent` 拥有内部规范事件 JSONL 快照与回放 fixture。`pwsh-tool-turn` ACP 场景启动真实 `pwsh`,在无 `pwsh` 的主机上跳过。已完成的交互式终端旅程使用 `apps/cli/tests/snapshots/` 下由 JSONL 驱动的场景;瞬态呈现使用包内语义矩阵,输入、Loader 选择或终端清理发生变化时还要添加 PTY 用例。浏览器渲染的 Web GUI 旅程使用上述 Web 应用快照套件。两个 SDK 各自独立地投影 agent loop、会话生命周期与 `SessionEventMap`,因此改动其中任何一项都要同时更新两者:`examples/jsonrpc-agent/tests/snapshots/` 拥有 TypeScript 客户端;`scripts/snapshots/python-sdk-single-exe/` 拥有 Python 客户端,且只有必需的 `python-runtime` CI 作业会运行它。新的能力 seam、生命周期变体或 transcript 呈现接口在计划阶段就要列出每个覆盖层级,并在实现前验证 harness 能够表达它们。
|
||||
|
||||
Reference in New Issue
Block a user