test(python): pin the minimal composition's model-visible surface
The Python lane never compared what the minimal composition shows the model: the mock model only asserted system-role messages, and the advanced snapshot tokenizes the assembled system prompt and tool schemas. The sdk-minimal scenario now records model-visible.json — every model request's advertised tool schemas verbatim and its message list, with system and user text kept and assistant/tool payloads reduced to call identity so the expected output replays on macOS and Linux. It excludes the dynamic runtime-context snapshot, which the same composition emits on macOS and not on Linux (#2488). AGENTS.md and the testing policy name both SDKs as independent projections of the agent loop, session lifecycle, and SessionEventMap.
This commit is contained in:
Yichen Jiang
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-08-12-required-python-runtime-pull-request-ci.md
|
||||
2026-08-12-required-python-runtime-pull-request-ci.md: 2f5dcac17262bd885049221620a4d9708b083faa
|
||||
2026-08-12-required-python-runtime-pull-request-ci.zh.md: 66c81f70d1bf25355425030883ccc4307efbb1ce
|
||||
2026-08-12-required-python-runtime-pull-request-ci.md: 61b1e832be6d29eafe5cb304d2bca3f0a59e3d84
|
||||
2026-08-12-required-python-runtime-pull-request-ci.zh.md: 92bf80688d8d152b26fdd893fe0f0b96553868e9
|
||||
|
||||
@@ -10,11 +10,11 @@ Ordinary pull-request CI runs the complete Python SDK pytest suite against fake
|
||||
|
||||
## Decision
|
||||
|
||||
Every pull request has a required `python-runtime` job in [CI](../../../../.github/workflows/ci.yml). It calls the shared [single-executable builder](../../../../.github/workflows/build-exe-for-python-sdk.yml) for `node24-linux-x64` without a path filter and participates in `all checks passed`. The called workflow builds the real executable, runs all keyless Python full-turn and direct-binary scenarios including the committed executable snapshot, builds the SDK and runtime wheels, installs them into a clean virtual environment, checks the executable and native addon's GLIBC requirements, and runs the installed wheels in a manylinux 2.28 container.
|
||||
Every pull request has a required `python-runtime` job in [CI](../../../../.github/workflows/ci.yml). It calls the shared [single-executable builder](../../../../.github/workflows/build-exe-for-python-sdk.yml) for `node24-linux-x64` without a path filter and participates in `all checks passed`. The called workflow builds the real executable, runs all keyless Python full-turn and direct-binary scenarios including both committed snapshots, builds the SDK and runtime wheels, installs them into a clean virtual environment, checks the executable and native addon's GLIBC requirements, and runs the installed wheels in a manylinux 2.28 container.
|
||||
|
||||
The required job and the [Python publication workflow](../process/2026-08-11-python-publication-workflow.md) use the same builder. Its concurrency key includes the caller workflow, so required CI and an explicit full release validation for the same ref do not cancel each other. The complete linux-x64, linux-arm64, and macos-arm64 matrix remains a release validation because platform-independent runtime, SDK, and snapshot behavior needs one merge-blocking native carrier, while architecture-specific executable, addon, wheel-tag, and deployment-target behavior still needs all release targets before publication.
|
||||
|
||||
The executable snapshot normalizes opaque session, message, subagent, and workflow-run identifiers before comparison. A newly persisted workflow event therefore changes the reviewed expected output without making a random run identifier part of that output.
|
||||
The advanced executable snapshot normalizes opaque session, message, subagent, and workflow-run identifiers before comparison. A newly persisted workflow event therefore changes the reviewed expected output without making a random run identifier part of that output. The minimal scenario's [model-visible snapshot](2026-08-13-python-minimal-model-visible-snapshot.md) covers the assembled system prompt, tool schemas, and message list that this one tokenizes.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
|
||||
@@ -10,11 +10,11 @@ Status: implemented
|
||||
|
||||
## 决策
|
||||
|
||||
每个拉取请求都在 [CI](../../../../.github/workflows/ci.yml) 中运行必需的 `python-runtime` 作业。该作业不使用路径过滤，调用共享的[单文件可执行程序构建器](../../../../.github/workflows/build-exe-for-python-sdk.yml)构建 `node24-linux-x64`，并参与 `all checks passed`。被调用的工作流会构建真实可执行文件，运行全部无密钥 Python 完整轮次和直接二进制场景（包括检入的 exe 快照），构建 SDK 与运行时 wheel 包，将二者安装进干净的虚拟环境，检查可执行文件与原生 addon 的 GLIBC 依赖，并在 manylinux 2.28 容器中运行已安装的 wheel 包。
|
||||
每个拉取请求都在 [CI](../../../../.github/workflows/ci.yml) 中运行必需的 `python-runtime` 作业。该作业不使用路径过滤，调用共享的[单文件可执行程序构建器](../../../../.github/workflows/build-exe-for-python-sdk.yml)构建 `node24-linux-x64`，并参与 `all checks passed`。被调用的工作流会构建真实可执行文件，运行全部无密钥 Python 完整轮次和直接二进制场景（包括两份检入的快照），构建 SDK 与运行时 wheel 包，将二者安装进干净的虚拟环境，检查可执行文件与原生 addon 的 GLIBC 依赖，并在 manylinux 2.28 容器中运行已安装的 wheel 包。
|
||||
|
||||
必需作业与 [Python 发布工作流](../process/2026-08-11-python-publication-workflow.md)共用同一构建器。其并发键包含调用方工作流，因此同一 ref 上的必需 CI 与显式完整发布验证不会互相取消。完整的 linux-x64、linux-arm64 和 macos-arm64 矩阵仍属于发布验证：平台无关的运行时、SDK 与快照行为只需要一个阻断合并的原生载体，而架构相关的可执行文件、addon、wheel 包标签与部署目标行为在发布前仍需要全部发布目标验证。
|
||||
|
||||
exe 快照会在比较前规范化不透明的会话、消息、subagent 和工作流运行标识符。因此，新增的持久化工作流事件会改变经过审阅的预期输出，但不会把随机运行标识符写入其中。
|
||||
进阶 exe 快照会在比较前规范化不透明的会话、消息、subagent 和工作流运行标识符。因此，新增的持久化工作流事件会改变经过审阅的预期输出，但不会把随机运行标识符写入其中。极简场景的[模型可见快照](2026-08-13-python-minimal-model-visible-snapshot.md)覆盖了这份快照所占位化的已组装系统提示词、工具 schema 与消息列表。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/testing/2026-08-13-python-minimal-model-visible-snapshot.md
|
||||
2026-08-13-python-minimal-model-visible-snapshot.md: 66cfa1ed667d9a60579b0d27ddca2667614d7e1c
|
||||
2026-08-13-python-minimal-model-visible-snapshot.zh.md: 40f596208f07532e68e382013b36e0d7ec46de3d
|
||||
@@ -0,0 +1,33 @@
|
||||
# Agent Note: Python minimal-composition model-visible snapshot
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-08-13-python-minimal-model-visible-snapshot.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The Python lane never compared what the minimal composition actually shows the model. Dynamic runtime context reaches history as a user message, so the mock model's assertion that system-role messages equal the deployment persona could not see it, and the advanced executable snapshot replaces each request header's assembled system prompt with a token and each tool schema with its name. The sandbox-policy runtime-context message therefore rode along in the checked-in [minimal composition](../../../../examples/jsonrpc-agent/minimal.cordis.yml) while `python-runtime` stayed green, and any plugin that adds a system section, a tool, or another context message could do the same.
|
||||
|
||||
## Decision
|
||||
|
||||
The `sdk-minimal` scenario in [the packaged-runtime smoke](../../../../scripts/smoke-python-runtime.py) records `scripts/snapshots/python-sdk-single-exe/minimal/model-visible.json`: for every model request of the turn, the advertised tool schemas verbatim and the message list. System and user messages keep their full text with the scenario's temporary directory tokenized; assistant and tool messages keep only call identity, because their PTY and filesystem text differs across the platforms the expected output replays on.
|
||||
|
||||
One model-visible message is excluded: the agent loop's dynamic runtime-context snapshot. The same composition emits it on macOS and not on Linux, which the required lane runs, so no single expected output can carry it. That difference is a defect in its own right ([#2488](https://github.com/deepseek-harness/deepseek-harness/issues/2488)) — this expected output covers every other model-visible message rather than waiting for it.
|
||||
|
||||
The mock model no longer asserts the minimal scenario's tools and system prompts — the snapshot owns that surface and reports a complete diff instead of the first mismatch. Snapshot comparison takes its directory and file set as arguments, so the `minimal` and `advanced` expected outputs use one implementation, and `--update-snapshots` accepts `sdk-minimal`.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Snapshot the minimal session log, like the advanced scenario.** The minimal turn drives a real PTY and editor, so persisted tool results carry platform-dependent text. The expected output would go red for reasons unrelated to model-visible assembly, and normalizing that text away leaves the log carrying little the model-visible file does not.
|
||||
|
||||
**Extend the mock model's inline assertions.** Every new model-visible contribution would need another hand-written expectation, and a failure names one mismatch rather than the whole surface. Tool descriptions would also be duplicated from the composition into the script.
|
||||
|
||||
**Rely on the TypeScript SDK snapshot.** Its `persistent-tools` scenario pins the same composition's system prompt, tool schemas, and runtime context, but through replayed model responses and a source or `lib` runtime, in a different required job. It cannot show what the deployed executable's closure assembles for a Python caller.
|
||||
|
||||
## Consequences
|
||||
|
||||
A change to the minimal composition's model-visible surface — a system section, a tool, a tool description, or an added user message — now fails `python-runtime` with the exact diff, and landing it means rerunning `--scenario sdk-minimal --update-snapshots` and reviewing that diff. The minimal composition's tool descriptions become reviewed expected output.
|
||||
|
||||
Assistant and tool message text is no longer compared, and the runtime-context snapshot is not compared at all. The scenario's own assertions continue to own persistent-shell state, editor output, and the final response; [#2488](https://github.com/deepseek-harness/deepseek-harness/issues/2488) owns the excluded message until its platform difference is resolved.
|
||||
|
||||
[AGENTS.md](../../../../AGENTS.md) and [the testing policy](../../../../docs/testing.md) now name both SDKs as independent projections of the agent loop, session lifecycle, and `SessionEventMap`, so a change to any of those carries updating both expected outputs rather than only the one a contributor happens to run.
|
||||
@@ -0,0 +1,33 @@
|
||||
# Agent Note：Python 极简组合的模型可见快照
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-08-13-python-minimal-model-visible-snapshot.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
Python 通道从未比对极简组合实际展示给模型的内容。动态运行时上下文以 user 消息进入历史，因此 mock 模型"system 角色消息等于部署 persona"的断言看不见它；而进阶可执行文件快照会把每个请求头中已组装的系统提示词换成占位符、把每个工具 schema 换成其名称。于是 sandbox-policy 的运行时上下文消息一直搭车留在签入的[极简组合](../../../../examples/jsonrpc-agent/minimal.cordis.yml)里，而 `python-runtime` 始终是绿的；任何新增系统分段、工具或其他上下文消息的插件都能照此蒙混过关。
|
||||
|
||||
## 决策
|
||||
|
||||
[打包运行时冒烟测试](../../../../scripts/smoke-python-runtime.py)的 `sdk-minimal` 场景会录制 `scripts/snapshots/python-sdk-single-exe/minimal/model-visible.json`：对该回合的每个模型请求，逐字记录对外公布的工具 schema 与消息列表。system 与 user 消息保留全文，仅将场景的临时目录替换为占位符；assistant 与 tool 消息只保留调用标识，因为它们的 PTY 与文件系统文本在期望输出需要重放的各平台上并不相同。
|
||||
|
||||
有一条模型可见消息被排除在外：agent loop 的动态运行时上下文快照。同一组合在 macOS 上会发出它，在必需车道所用的 Linux 上不会，因此任何单一期望输出都无法承载它。该差异本身就是缺陷（[#2488](https://github.com/deepseek-harness/deepseek-harness/issues/2488)）——这份期望输出覆盖其余全部模型可见消息，而不是等它先被修复。
|
||||
|
||||
mock 模型不再断言极简场景的工具与系统提示词——该面由快照拥有，并给出完整差异而非首个不匹配项。快照比对以目录与文件集合为参数，因此 `minimal` 与 `advanced` 两份期望输出共用一套实现，且 `--update-snapshots` 接受 `sdk-minimal`。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
**像进阶场景那样对极简会话日志做快照。** 极简回合驱动真实 PTY 与编辑器，持久化的工具结果带有平台相关文本。期望输出会因与模型可见组装无关的原因变红；而把这些文本归一化掉之后，日志所承载的内容也就所剩无几。
|
||||
|
||||
**扩展 mock 模型中的内联断言。** 每新增一项模型可见贡献都要再手写一条期望，且失败只会指出一处不匹配而非整个面。工具描述还会从组合复制进脚本，形成重复。
|
||||
|
||||
**依赖 TypeScript SDK 快照。** 其 `persistent-tools` 场景固定了同一组合的系统提示词、工具 schema 与运行时上下文，但走的是重放的模型响应与 source 或 `lib` 运行时，且位于另一个必需任务中。它无法体现已部署可执行文件的闭包为 Python 调用方组装出什么。
|
||||
|
||||
## 后果
|
||||
|
||||
极简组合模型可见面的改动——系统分段、工具、工具描述或新增的 user 消息——现在会让 `python-runtime` 带着精确差异失败；要让它落地，就必须重新运行 `--scenario sdk-minimal --update-snapshots` 并审阅该差异。极简组合的工具描述由此成为经过审阅的期望输出。
|
||||
|
||||
assistant 与 tool 消息文本不再参与比对，运行时上下文快照则完全不参与比对。持久 shell 状态、编辑器输出与最终响应仍由该场景自身的断言拥有；被排除的那条消息由 [#2488](https://github.com/deepseek-harness/deepseek-harness/issues/2488) 负责，直到其平台差异得到解决。
|
||||
|
||||
[AGENTS.md](../../../../AGENTS.md) 与[测试政策](../../../../docs/testing.md)现已点明两个 SDK 都是 agent loop、会话生命周期与 `SessionEventMap` 的独立投影，因此改动其中任何一项都要连带更新两侧的期望输出，而不只是贡献者恰好会运行的那一侧。
|
||||
Reference in New Issue
Block a user