ci: validate and publish Python runtime wheels

This commit is contained in:
Yichen Jiang
2026-07-13 16:34:16 +08:00
parent 283ac7f097
commit cdd11ac587
16 changed files with 556 additions and 33 deletions

View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write
2026-07-10-single-file-executable-sdk-runtime-distribution.md: 4fab5f89542753f2dd5977d4262e1310097f4545
2026-07-10-single-file-executable-sdk-runtime-distribution.zh.md: 56a93f2392bff655d32ec4947c357b33d9e6da5e
2026-07-10-single-file-executable-sdk-runtime-distribution.md: 0e463a23227bfb8f8eef5ee1a21bcddd9151c8e4
2026-07-10-single-file-executable-sdk-runtime-distribution.zh.md: 2ce32f9bba20e85fa67559389be9d2d5c7e60cee

View File

@@ -36,17 +36,19 @@ Config discovery has two channels and fails loudly when both are missing: the `D
Inside the exe's VFS sits a **real package tree in build-artifact form** (each package's `lib/` plus a real `node_modules`); the Loader resolves plugin names through standard dynamic `import()`: bare specifiers resolve upward along `node_modules` from the Loader's position inside the VFS, and land inside the VFS naturally. The closed set needs no allowlist code — the set is whatever the VFS has installed, and importing a name outside the set fails.
The deploy root is [`python/sdk-runtime/package.json`](../../../../python/sdk-runtime/package.json) (`dsh-jsonrpc-agent-pkg`, a pnpm workspace member and a zero-code pure dependency manifest) — the unified source of truth for "which plugins the exe ships" and "what the Python runtime distributes". Adding a plugin to the exe = adding one dependency line to the manifest and repackaging. Two closure-completeness rules learned from measurement: in-repo pure libraries that closure members reference as peerDependency (`dsh-brand`, `dsh-timeout`, `dsh-invariants`, `@cordisjs/plugin-timer`) must be listed explicitly in the manifest's `dependencies`; deploy packs by each package's `files`, so the shared chunks tsdown splits out must be covered by `files`.
The deploy root is [`python/sdk-runtime/package.json`](../../../../python/sdk-runtime/package.json) (`dsh-jsonrpc-agent-pkg`, a pnpm workspace member and a zero-code pure dependency manifest) — the unified source of truth for "which plugins the exe ships" and "what the Python runtime distributes". Adding a plugin to the exe = adding one dependency line to the manifest and repackaging. [`scripts/verify-runtime-closure.ts`](../../../../scripts/verify-runtime-closure.ts) traverses every workspace package covered by that manifest and requires every non-optional workspace peer at the runtime root, reporting the complete referencing-package → missing-peer chain; CI static, pre-push, and the single-exe build run it before packaging. Deploy also packs by each package's `files`, so the shared chunks tsdown splits out must be covered by `files`.
### Build pipeline and artifacts
[`scripts/build-exe-for-python-sdk.ts`](../../../../scripts/build-exe-for-python-sdk.ts): `pnpm run build` → (after clearing) `pnpm --filter dsh-jsonrpc-agent-pkg deploy --legacy --prod --config.node-linker=hoisted --config.auto-install-peers=false --config.link-workspace-packages=true` **directly into** `python/sdk-runtime/src/deepseek_harness_runtime/runtime/node/` → inject the pkg configuration (`bin` points at `node_modules/@deepseek-ai/dsh-jsonrpc-agent/lib/bin.js` inside the closure, `assets` is a full glob — dynamic import is invisible to pkg's static analysis, so everything must be packed in explicitly) → one `pkg --sea` per target → the artifacts `dsh-jsonrpc-agent-pkg-<platform>-<arch>` land in `dist-exe/` (the CI artifact) and are copied back into the runtime directory. All four deploy flags are grounded in measurement: `--legacy` is the mandatory path with inject-workspace-packages off; hoisted yields a zero-symlink file tree (most stable for the pkg VFS, physically guaranteeing a single cordis instance); disabling automatic peer installation keeps unpublished package names from triggering registry resolution; link-workspace-packages points the closure at workspace/vendor sources.
[`scripts/build-exe-for-python-sdk.ts`](../../../../scripts/build-exe-for-python-sdk.ts): runtime closure verification → `pnpm run build` → (after clearing) `pnpm --filter dsh-jsonrpc-agent-pkg deploy --legacy --prod --config.node-linker=hoisted --config.auto-install-peers=false --config.link-workspace-packages=true` **directly into** `python/sdk-runtime/src/deepseek_harness_runtime/runtime/node/` → inject the pkg configuration (`bin` points at `node_modules/@deepseek-ai/dsh-jsonrpc-agent/lib/bin.js` inside the closure, `assets` is a full glob — dynamic import is invisible to pkg's static analysis, so everything must be packed in explicitly) → one `pkg --sea` per target → the artifacts `dsh-jsonrpc-agent-pkg-<platform>-<arch>` land in `dist-exe/` (the CI artifact) and are copied back into the runtime directory. All four deploy flags are grounded in measurement: `--legacy` is the mandatory path with inject-workspace-packages off; hoisted yields a zero-symlink file tree (most stable for the pkg VFS, physically guaranteeing a single cordis instance); disabling automatic peer installation keeps unpublished package names from triggering registry resolution; link-workspace-packages points the closure at workspace/vendor sources.
CI: [`.github/workflows/build-exe-for-python-sdk.yml`](../../../../.github/workflows/build-exe-for-python-sdk.yml), triggered explicitly only — `workflow_dispatch`, or the `build-exe` label on a pull request; native builds on the three platforms linux-x64 / linux-arm64 (`ubuntu-24.04-arm`) / macos-arm64, with `~/.pkg-cache` cached and artifacts uploaded per platform; macOS ad-hoc signing is handled by pkg. Windows is a non-goal.
CI: [`.github/workflows/build-exe-for-python-sdk.yml`](../../../../.github/workflows/build-exe-for-python-sdk.yml), triggered explicitly only — `workflow_dispatch`, or the `build-exe` label on a pull request; native builds on the three platforms linux-x64 / linux-arm64 (`ubuntu-24.04-arm`) / macos-arm64, with `~/.pkg-cache` cached and artifacts uploaded per platform; macOS ad-hoc signing is handled by pkg. Each leg drives a mock SSE model through the SDK with the default config and a custom `cordis.yml`, drives the exe directly over NDJSON JSON-RPC, verifies the JSONL and final response, and installs release-shaped wheels into a clean venv without `runtime_bin`; Linux additionally inspects GLIBC requirements and runs in a manylinux 2.28 container. [`.gitlab-ci.yml`](../../../../.gitlab-ci.yml) accepts only `python-vX.Y.Z` tag pipelines, builds one SDK wheel and three native runtime wheels, then a single serialized job checks and publishes all four to the project PyPI registry. Windows is a non-goal.
### Python SDK distribution: two carriers, exe for production, node for development
The Python SDK lives at [`python/`](../../../../python/README.md): `python/sdk` (the client) + `python/sdk-runtime` (the runtime carrier package). The runtime package's data directory holds three kinds of content: the checked-in default `runtime/cordis.yml`, the build-injected platform exe, and the build-injected `runtime/node/` closure tree. `resolve_bundled_launch_args()` automatic resolution **finds the exe only**; the node carrier is enabled only by an explicit `DSH_RUNTIME_MODE=node` (running `runtime/node/node_modules/@deepseek-ai/dsh-jsonrpc-agent/lib/bin.js`, requiring a system node ≥22.19), positioned as the development-verification channel for members of this repo, and does not enter wheel/sdist distributions.
The Python SDK lives at [`python/`](../../../../python/README.md): `python/sdk` (the client) + `python/sdk-runtime` (the runtime carrier package). The runtime package's data directory holds three kinds of content: the checked-in default `runtime/cordis.yml`, the build-injected platform exe, and the build-injected `runtime/node/` closure tree. `resolve_bundled_launch_args()` automatic resolution **finds the exe only**; the node carrier is enabled only by an explicit `DSH_RUNTIME_MODE=node` (running `runtime/node/node_modules/@deepseek-ai/dsh-jsonrpc-agent/lib/bin.js`, requiring a system node ≥22.19), positioned as the development-verification channel for members of this repo, and does not enter wheel distributions.
[`scripts/build-python-release.py`](../../../../scripts/build-python-release.py) accepts only stable `python-vX.Y.Z` tags and stages both packages at `X.Y.Z`, with the SDK depending exactly on `deepseek-harness-runtime-bin==X.Y.Z`. The SDK is a `py3-none-any` wheel; the wheel-only runtime package contains exactly one exe and uses one of `py3-none-manylinux_2_28_x86_64`, `py3-none-manylinux_2_28_aarch64`, or `py3-none-macosx_11_0_arm64`. Its Hatch hook rejects sdists, universal tags, mixed executable payloads, and unsupported platforms.
The exe's "must be explicitly configured" hard semantic is unchanged; the zero-config experience is restored by the wrapper: when the caller gave no `cordis`, named no explicit runtime, and the environment has no `DSH_CORDIS_CONFIG`, the client explicitly injects the checked-in default `cordis.yml` (agent-core + preloaded llm-deepseek + JSONL persistence + bash-local + the `dsh-jsonrpc` serving entry, with `!!js` environment-variable fallbacks) via `DSH_CORDIS_CONFIG`.
@@ -60,7 +62,7 @@ The exe's "must be explicitly configured" hard semantic is unchanged; the zero-c
## Testing
The verification surface has three tiers. Mechanism tier: the measured conclusions for the `--sea` chain are embedded in the Decision sections (ESM dynamic import inside the VFS, single cordis instance, fail-loud config chain, `node:sqlite`, macOS ad-hoc signing runs). SDK tier: the 28 pytest cases in `python/sdk/tests` cover the client protocol (18 cases, keyless, against a fake runtime peer), dual-carrier launch (6 cases, each skipping independently when the artifact is missing), and the carrier resolution rules (4 cases). End-to-end tier: the acceptance material (five keyless items: env-channel launch, fail-loud on missing config, fail-loud on a bad plugin name, config channel, full turn against a mock endpoint + JSONL on disk; two with-key items: a real turn, and a world check where the bash tool writes a marker file + clean exit) drives the exe from a real Python SDK client. Current debts: running the above suites against the current artifacts, and a built-exe e2e. The JSON-RPC protocol is not part of the ACP snapshot system, so there is no snapshot tier (an explicitly named gap, not an oversight).
The verification surface has three tiers. Mechanism tier: the measured conclusions for the `--sea` chain are embedded in the Decision sections (ESM dynamic import inside the VFS, single cordis instance, fail-loud config chain, `node:sqlite`, macOS ad-hoc signing runs). SDK tier: the complete keyless pytest suite covers the client protocol against a fake runtime peer, subprocess cleanup, absolute cwd propagation, dual-carrier launch, and carrier resolution; root CI runs it on Python 3.10. End-to-end tier: every platform artifact completes a turn against a mock endpoint through the default SDK path, a custom config, and the direct binary protocol, with final text and JSONL checked; its platform wheel is then installed in a clean venv and run without `runtime_bin`. The JSON-RPC protocol is not part of the ACP snapshot system, so there is no snapshot tier (an explicitly named gap, not an oversight).
Manual-driving caveat: the bin treats stdin EOF as "the client is gone" and disposes immediately, so a short-lived pipe aborts an in-flight turn — pipe-driven runs must keep stdin open until the turn ends.
@@ -80,4 +82,4 @@ Manual-driving caveat: the bin treats stdin EOF as "the client is gone" and disp
**Bought**: zero-dependency single-file distribution on target platforms; plugin semantics strictly identical to running from source (the same real package tree, no transpilation, no registry); the serving surface, the plugin set, and the configuration all converge on two sources of truth — `cordis.yml` plus one dependency manifest; the exe and node carriers share one tree and one semantics, so development verification never waits for packaging; official Node binaries remove the patched-binary supply-chain concern.
**Paid**: artifacts on the order of 174MB with source entering the blob as-is (no bytecode obfuscation; a closed-source distribution requirement needs a separate evaluation); pkg's VFS/module-hook layer remains community-maintained (the build script pins `@yao-pkg/pkg@6.21.0`; upgrading is an explicit change); `--sea` is one invocation per target (matching CI's one leg per platform; local multi-platform builds are serial); worker-style plugins have undefined behavior inside the exe; acceptance and coverage carry catch-up debts (see Testing).
**Paid**: artifacts on the order of 174MB with source entering the blob as-is (no bytecode obfuscation; a closed-source distribution requirement needs a separate evaluation); pkg's VFS/module-hook layer remains community-maintained (the build script pins `@yao-pkg/pkg@6.21.0`; upgrading is an explicit change); `--sea` is one invocation per target (matching CI's one leg per platform; local multi-platform builds are serial); worker-style plugins have undefined behavior inside the exe.

View File

@@ -36,17 +36,19 @@ exe 用 [@yao-pkg/pkg](https://github.com/yao-pkg/pkg)vercel/pkg 归档后的
exe 的 VFS 内是**构建产物形态的真实包树**(各包 `lib/` + 真实 `node_modules`Loader 解析插件名走标准动态 `import()`:裸包名从 VFS 内 Loader 位置沿 `node_modules` 向上解析,天然落在 VFS 内。封闭集不需要白名单代码——集合就是 VFS 里装了什么,引用集合外的名字 import 失败。
deploy root 是 [`python/sdk-runtime/package.json`](../../../../python/sdk-runtime/package.json)`dsh-jsonrpc-agent-pkg`pnpm workspace 成员、零代码纯依赖清单——「exe 装什么插件」与「Python runtime 分发什么」的合一事实源。往 exe 加插件 = 清单加一行依赖再重打包。两条实测得来的闭包完整性规则:闭包成员以 peerDependency 引用的仓库内纯库(`dsh-brand``dsh-timeout``dsh-invariants``@cordisjs/plugin-timer`)必须显式列进清单 `dependencies`deploy 按各包 `files` 打包tsdown 拆出的共享 chunk 必须被 `files` 覆盖。
deploy root 是 [`python/sdk-runtime/package.json`](../../../../python/sdk-runtime/package.json)`dsh-jsonrpc-agent-pkg`pnpm workspace 成员、零代码纯依赖清单——「exe 装什么插件」与「Python runtime 分发什么」的合一事实源。往 exe 加插件 = 清单加一行依赖再重打包。[`scripts/verify-runtime-closure.ts`](../../../../scripts/verify-runtime-closure.ts) 遍历该清单覆盖的全部 workspace 包,要求每个非 optional workspace peer 显式列在 runtime root并报告「引用包 → 缺失 peer」的完整链路CI static、pre-push 与 single-exe 构建都会在打包前运行该门禁。deploy 还会按各包 `files` 打包,因此 tsdown 拆出的共享 chunk 必须被 `files` 覆盖。
### 构建管线与产物
[`scripts/build-exe-for-python-sdk.ts`](../../../../scripts/build-exe-for-python-sdk.ts)`pnpm run build` →(清空后)`pnpm --filter dsh-jsonrpc-agent-pkg deploy --legacy --prod --config.node-linker=hoisted --config.auto-install-peers=false --config.link-workspace-packages=true` **直落** `python/sdk-runtime/src/deepseek_harness_runtime/runtime/node/`→ 注入 pkg 配置(`bin` 指闭包内 `node_modules/@deepseek-ai/dsh-jsonrpc-agent/lib/bin.js``assets` 全量 glob——动态 import 对 pkg 静态分析不可见,必须显式全量打入)→ 每 target 一次 `pkg --sea` → 产物 `dsh-jsonrpc-agent-pkg-<platform>-<arch>``dist-exe/`CI artifact并拷回 runtime 目录。deploy 四 flag 均有实测依据:`--legacy` 是未开 inject-workspace-packages 时的必选路径hoisted 产出零符号链接文件树pkg VFS 最稳、物理保证 cordis 单实例);关 peer 自动安装避免未发布包名触发 registry 解析link-workspace-packages 让闭包指向 workspace/vendor 源。
[`scripts/build-exe-for-python-sdk.ts`](../../../../scripts/build-exe-for-python-sdk.ts)runtime 闭包校验 → `pnpm run build` →(清空后)`pnpm --filter dsh-jsonrpc-agent-pkg deploy --legacy --prod --config.node-linker=hoisted --config.auto-install-peers=false --config.link-workspace-packages=true` **直落** `python/sdk-runtime/src/deepseek_harness_runtime/runtime/node/`→ 注入 pkg 配置(`bin` 指闭包内 `node_modules/@deepseek-ai/dsh-jsonrpc-agent/lib/bin.js``assets` 全量 glob——动态 import 对 pkg 静态分析不可见,必须显式全量打入)→ 每 target 一次 `pkg --sea` → 产物 `dsh-jsonrpc-agent-pkg-<platform>-<arch>``dist-exe/`CI artifact并拷回 runtime 目录。deploy 四 flag 均有实测依据:`--legacy` 是未开 inject-workspace-packages 时的必选路径hoisted 产出零符号链接文件树pkg VFS 最稳、物理保证 cordis 单实例);关 peer 自动安装避免未发布包名触发 registry 解析link-workspace-packages 让闭包指向 workspace/vendor 源。
CI[`.github/workflows/build-exe-for-python-sdk.yml`](../../../../.github/workflows/build-exe-for-python-sdk.yml),仅显式触发——`workflow_dispatch` 手动派发,或给 PR 打 `build-exe` 标签linux-x64 / linux-arm64`ubuntu-24.04-arm`/ macos-arm64 三平台原生构建,`~/.pkg-cache` 缓存artifact 按平台上传macOS ad-hoc 签名由 pkg 处理。Windows 是非目标。
CI[`.github/workflows/build-exe-for-python-sdk.yml`](../../../../.github/workflows/build-exe-for-python-sdk.yml),仅显式触发——`workflow_dispatch` 手动派发,或给 PR 打 `build-exe` 标签linux-x64 / linux-arm64`ubuntu-24.04-arm`/ macos-arm64 三平台原生构建,`~/.pkg-cache` 缓存artifact 按平台上传macOS ad-hoc 签名由 pkg 处理。每个平台都以 mock SSE 模型分别通过默认配置和自定义 `cordis.yml` 驱动 SDK再以 NDJSON JSON-RPC 直接驱动 exe校验 JSONL 与最终响应,最后把 release 形态的 wheel 安装到干净 venv 中并在不传 `runtime_bin` 的情况下运行Linux 还检查 GLIBC 依赖并在 manylinux 2.28 容器中运行。[`.gitlab-ci.yml`](../../../../.gitlab-ci.yml) 只接受 `python-vX.Y.Z` tag 流水线,构建一个 SDK wheel 与 3 个原生 runtime wheel再由单个串行 job 校验并发布这 4 个文件到项目 PyPI 注册表。Windows 是非目标。
### Python SDK 分发双载体exe 为生产、node 为开发
Python SDK 位于 [`python/`](../../../../python/README.md)`python/sdk`(客户端)+ `python/sdk-runtime`运行时载体包。runtime 包数据目录三类内容:检入的默认 `runtime/cordis.yml`、构建注入的平台 exe、构建注入的 `runtime/node/` 闭包树。`resolve_bundled_launch_args()` 自动解析**只找 exe**node 载体仅显式 `DSH_RUNTIME_MODE=node` 启用(跑 `runtime/node/node_modules/@deepseek-ai/dsh-jsonrpc-agent/lib/bin.js`,需系统 node ≥22.19),定位是本仓库成员的开发验证通道,不进 wheel/sdist 分发物。
Python SDK 位于 [`python/`](../../../../python/README.md)`python/sdk`(客户端)+ `python/sdk-runtime`运行时载体包。runtime 包数据目录三类内容:检入的默认 `runtime/cordis.yml`、构建注入的平台 exe、构建注入的 `runtime/node/` 闭包树。`resolve_bundled_launch_args()` 自动解析**只找 exe**node 载体仅显式 `DSH_RUNTIME_MODE=node` 启用(跑 `runtime/node/node_modules/@deepseek-ai/dsh-jsonrpc-agent/lib/bin.js`,需系统 node ≥22.19),定位是本仓库成员的开发验证通道,不进 wheel 分发物。
[`scripts/build-python-release.py`](../../../../scripts/build-python-release.py) 只接受稳定的 `python-vX.Y.Z` tag并以 `X.Y.Z` 暂存两个包,同时让 SDK 精确依赖 `deepseek-harness-runtime-bin==X.Y.Z`。SDK 是 `py3-none-any` wheel只提供 wheel 的 runtime 包恰好包含一个 exetag 为 `py3-none-manylinux_2_28_x86_64``py3-none-manylinux_2_28_aarch64``py3-none-macosx_11_0_arm64`。其 Hatch 钩子拒绝 sdist、通用 tag、混合可执行载荷以及不支持的平台。
exe「必须显式配置」的硬语义不变零配置体验由 wrapper 恢复:调用方没给 `cordis`、没显式指定 runtime、环境无 `DSH_CORDIS_CONFIG` 时,客户端把检入的默认 `cordis.yml`agent-core + 预载 llm-deepseek + JSONL 持久化 + bash-local + `dsh-jsonrpc` serving 条目,`!!js` 环境变量兜底)显式注入 `DSH_CORDIS_CONFIG`
@@ -60,7 +62,7 @@ exe「必须显式配置」的硬语义不变零配置体验由 wrapper 恢
## 测试
验证面分三层。机制层:`--sea` 链路的实测结论内嵌在「决策」各节VFS 内 ESM 动态 import、cordis 单实例、fail-loud 配置链路、`node:sqlite`、macOS ad-hoc 签名可运行。SDK 层:`python/sdk/tests` 的 28 例 pytest 覆盖客户端协议18 例keyless、假运行时对端、双载体启动6 例,产物缺失时独立 skip与载体解析规则4 例。端到端层验收物料keyless 五项env 通道启动、缺配置 fail-loud、坏插件名 fail-loud、配置通道、mock 端点完整回合 + JSONL 落盘;带 key 两项真实回合、bash 工具写 marker 文件的世界校验 + 干净退出)由真实 Python SDK 客户端驱动 exe。当前欠账上述套件对当前产物的运行与 built-exe e2e。JSON-RPC 协议不在 ACP snapshot 体系内,无 snapshot 层(点名后的明确空缺,非遗漏)。
验证面分三层。机制层:`--sea` 链路的实测结论内嵌在「决策」各节VFS 内 ESM 动态 import、cordis 单实例、fail-loud 配置链路、`node:sqlite`、macOS ad-hoc 签名可运行。SDK 层:完整的 keyless pytest 套件以假运行时对端覆盖客户端协议、子进程清理、绝对 cwd 传递、双载体启动与载体解析;根 CI 在 Python 3.10 上运行全部用例。端到端层:每个平台产物都通过默认 SDK 路径、自定义配置和直接二进制协议对着 mock 端点完成一个轮次,并校验最终文本与 JSONL随后把平台 wheel 安装进干净 venv在不传 `runtime_bin` 的情况下运行。JSON-RPC 协议不在 ACP snapshot 体系内,无 snapshot 层(点名后的明确空缺,非遗漏)。
手工驱动注意bin 视 stdin EOF 为「客户端已走」并立即 dispose短命管道会中止在飞回合——管道驱动必须保持 stdin 打开到回合结束。
@@ -80,4 +82,4 @@ exe「必须显式配置」的硬语义不变零配置体验由 wrapper 恢
**买到的**目标平台零依赖单文件分发插件语义与源码运行严格一致同一棵真实包树无转译无注册表serving 面、插件集、配置三者全部收敛到 `cordis.yml` + 一份依赖清单两个事实源exe 与 node 双载体同树同语义,开发验证不必等打包;官方 Node 二进制消除了补丁二进制供应链顾虑。
**付出的**:产物 174MB 级且源码原样进 blob无字节码混淆闭源分发诉求需另行评估pkg 的 VFS/模块钩子层仍是社区维护(构建脚本钉死 `@yao-pkg/pkg@6.21.0`,升级走显式改动);`--sea` 单 target 单次调用(与 CI 每平台一腿匹配本地多平台构建串行worker 类插件在 exe 内行为未定义;验收与覆盖率存在补跑欠账(见「测试」)
**付出的**:产物 174MB 级且源码原样进 blob无字节码混淆闭源分发诉求需另行评估pkg 的 VFS/模块钩子层仍是社区维护(构建脚本钉死 `@yao-pkg/pkg@6.21.0`,升级走显式改动);`--sea` 单 target 单次调用(与 CI 每平台一腿匹配本地多平台构建串行worker 类插件在 exe 内行为未定义。