diff --git a/.agents/notes/implemented/process/2026-07-21-serial-cross-platform-ci-reference.i18n.yaml b/.agents/notes/implemented/process/2026-07-21-serial-cross-platform-ci-reference.i18n.yaml index 553e656805..51073f506e 100644 --- a/.agents/notes/implemented/process/2026-07-21-serial-cross-platform-ci-reference.i18n.yaml +++ b/.agents/notes/implemented/process/2026-07-21-serial-cross-platform-ci-reference.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-21-serial-cross-platform-ci-reference.md -2026-07-21-serial-cross-platform-ci-reference.md: 5eac1bc1c47c7309942b5615bc98a7fed893f346 -2026-07-21-serial-cross-platform-ci-reference.zh.md: 35fb761023fe7be081bf7d9591a53ed98b6e3abc +2026-07-21-serial-cross-platform-ci-reference.md: 9ecd7906d2f28461a8a6f3f2fe485580214b39f3 +2026-07-21-serial-cross-platform-ci-reference.zh.md: ed927012c5b0ba9d43444a129d10d966f5e7d923 diff --git a/.agents/notes/implemented/process/2026-07-21-serial-cross-platform-ci-reference.md b/.agents/notes/implemented/process/2026-07-21-serial-cross-platform-ci-reference.md index 5eac1bc1c4..9ecd7906d2 100644 --- a/.agents/notes/implemented/process/2026-07-21-serial-cross-platform-ci-reference.md +++ b/.agents/notes/implemented/process/2026-07-21-serial-cross-platform-ci-reference.md @@ -14,15 +14,15 @@ Reviewers also need a direct answer to a simpler question: what happens when the ## Decision -[CI](../../../../.github/workflows/ci.yml) gives pull-request and master-push events complementary responsibilities. Pull requests run consolidated Linux and Windows jobs plus the Node compatibility and Python contracts on standard GitHub-hosted capacity. A push to `master` skips those jobs and runs three explicit references named `serial / linux`, `serial / macos`, and `serial / windows`. They intentionally duplicate their short checkout, runtime setup, and immutable install sequences instead of hiding the operating systems behind a matrix or reusable workflow. `workflow_dispatch` is reserved for runner benchmarks. +[CI](../../../../.github/workflows/ci.yml) gives pull-request and master-push events complementary responsibilities. Pull requests run consolidated Linux and Windows jobs plus the Node compatibility and Python contracts on standard GitHub-hosted capacity. A push to `master` skips those jobs and runs four explicit references: `serial / linux`, `serial / macos`, and `serial / windows` on standard hosted runners, plus `serial / linux (self-hosted standby)` on the in-house `vm-backup` pool — the hot-standby drill that continuously re-proves the failover target described in the [failover runbook](2026-07-26-ci-failover-runbook.md). They intentionally duplicate their short checkout, runtime setup, and immutable install sequences instead of hiding the operating systems behind a matrix or reusable workflow. `workflow_dispatch` is reserved for runner benchmarks. -Each reference job runs `pnpm run check:ci` without any shard selector. `DSH_GATE_CONCURRENCY=1` makes the top-level aggregate execute one ready gate at a time; coverage, snapshot replay, built-bin smoke, and publication validation also receive worker counts of one. The three operating-system jobs may run beside one another, but each host's repository gates are serial and complete. Linux installs bubblewrap before replaying snapshots, and Windows enables Developer Mode before installing the symlinked workspace. +Each reference job runs `pnpm run check:ci` without any shard selector. `DSH_GATE_CONCURRENCY=1` makes the top-level aggregate execute one ready gate at a time; coverage, snapshot replay, built-bin smoke, and publication validation also receive worker counts of one. The reference jobs may run beside one another, but each host's repository gates are serial and complete. Linux installs bubblewrap before replaying snapshots, and Windows enables Developer Mode before installing the symlinked workspace. Platform ownership remains explicit inside that complete aggregate. `pty-local` supports Linux and macOS and therefore owns its unit and per-file coverage contract on POSIX rather than loading a backend that rejects `win32`; the Windows run still executes every portable package. Portable fixtures derive native paths through `node:path`, compare canonical identities with the same native realpath implementation as production, and use filenames legal on every host. ACP snapshot runs also pass both JavaScript and native realpath spellings of their generated cwd to the normalizer, which replaces aliases longest-first so Windows short and long paths cannot churn shared fixtures. The macOS reference runs the ordinary Vitest project in forked processes. Node 24 on macOS arm64 has aborted in its CJS lexer from a worker thread; the process boundary contains that external runtime failure without removing any test from the aggregate, while Linux and Windows retain the lower-overhead thread pool. Repository-owned races are fixed at their observation boundaries: dev bundle polling stages each candidate table, graph, and watch-baseline map before publishing a rescan, and a missing bundle remains dirty until a successful content hash. PTY readiness retains a prompt candidate while polling checks foreground ownership; the ordinary silence bound covers inherited markers from interactive children. Real PTY fixtures assemble synchronization tokens at runtime so the interactive shell's input echo cannot satisfy a child-readiness wait. The live-link package-manager e2e preserves the workflow-prepared Corepack home and pnpm metadata/store caches while isolating the other managers' mutable caches, so it does not discard reusable package-manager state before the install. -Master reference jobs are diagnostic and do not participate in the pull request's required `all checks passed` result. A pull request runs only its required jobs; a master push runs only the three serial references. Performance is evaluated from completed hosted-job timestamps and reported as a measurement; it is not encoded as a `timeout-minutes` value. +Master reference jobs are diagnostic and do not participate in the pull request's required `all checks passed` result. A pull request runs only its required jobs; a master push runs only the serial references. Performance is evaluated from completed hosted-job timestamps and reported as a measurement; it is not encoded as a `timeout-minutes` value. The portable reference uses GitHub's standard `ubuntu-latest`, `macos-latest`, and `windows-2025` labels; `serial / windows` is the one remaining native-Windows job, the complete-kernel oracle behind the Wine-hosted pull-request lane ([Wine lane decision](2026-07-27-wine-windows-gates-experiment.md)). Required pull-request jobs use portable standard capacity under the [required-CI decision](2026-07-23-portable-required-pull-request-ci.md). Higher-core hosted runners remain manual benchmarks because a correctness path must remain runnable without repository-external runner configuration. diff --git a/.agents/notes/implemented/process/2026-07-21-serial-cross-platform-ci-reference.zh.md b/.agents/notes/implemented/process/2026-07-21-serial-cross-platform-ci-reference.zh.md index 35fb761023..ed927012c5 100644 --- a/.agents/notes/implemented/process/2026-07-21-serial-cross-platform-ci-reference.zh.md +++ b/.agents/notes/implemented/process/2026-07-21-serial-cross-platform-ci-reference.zh.md @@ -14,15 +14,15 @@ Status: implemented ## 决策 -[CI](../../../../.github/workflows/ci.yml) 为拉取请求事件与 master 推送事件赋予互补的职责。拉取请求在 GitHub 标准托管容量上运行合并后的 Linux 和 Windows 作业，以及 Node 兼容性与 Python 契约。向 `master` 推送时会跳过这些作业，改为运行三个显式参考作业，名称分别为 `serial / linux`、`serial / macos` 和 `serial / windows`。这些作业有意分别重复简短的代码检出、运行时设置和依赖锁定的安装步骤，不用矩阵或可复用工作流把操作系统差异隐藏起来。`workflow_dispatch` 仅用于运行器基准测试。 +[CI](../../../../.github/workflows/ci.yml) 为拉取请求事件与 master 推送事件赋予互补的职责。拉取请求在 GitHub 标准托管容量上运行合并后的 Linux 和 Windows 作业，以及 Node 兼容性与 Python 契约。向 `master` 推送时会跳过这些作业，改为运行四个显式参考作业：在标准托管运行器上的 `serial / linux`、`serial / macos` 和 `serial / windows`，以及在公司自有 `vm-backup` 池上的 `serial / linux (self-hosted standby)`——后者是热备演练，持续验证[故障切换手册](2026-07-26-ci-failover-runbook.md)所描述的切换目标。这些作业有意分别重复简短的代码检出、运行时设置和依赖锁定的安装步骤，不用矩阵或可复用工作流把操作系统差异隐藏起来。`workflow_dispatch` 仅用于运行器基准测试。 -每个参考作业均在不设置任何分片选择器的情况下运行 `pnpm run check:ci`。`DSH_GATE_CONCURRENCY=1` 使顶层聚合每次只执行一个已经就绪的门禁；覆盖率、快照回放、built-bin 冒烟测试和发布验证的并发数也设为 1。三种操作系统的作业可以彼此并行，但每台主机上的仓库门禁都串行运行且完整执行。Linux 在回放快照前安装 bubblewrap，Windows 则在安装采用符号链接的工作区前启用开发人员模式。 +每个参考作业均在不设置任何分片选择器的情况下运行 `pnpm run check:ci`。`DSH_GATE_CONCURRENCY=1` 使顶层聚合每次只执行一个已经就绪的门禁；覆盖率、快照回放、built-bin 冒烟测试和发布验证的并发数也设为 1。各参考作业可以彼此并行，但每台主机上的仓库门禁都串行运行且完整执行。Linux 在回放快照前安装 bubblewrap，Windows 则在安装采用符号链接的工作区前启用开发人员模式。 该完整聚合流程仍明确划分平台归属。`pty-local` 支持 Linux 与 macOS，因此其单元测试和逐文件覆盖率契约由 POSIX 平台负责，而不会在 Windows 上加载一个明确拒绝 `win32` 的后端；Windows 仍会执行所有可移植包（package）。可移植 fixture（测试前置数据）通过 `node:path` 派生原生路径，使用与生产代码相同的原生 realpath 实现比较规范化后的路径标识，并采用所有宿主机均允许的文件名。ACP（Agent Client Protocol）快照运行还会把生成的 cwd 分别通过 realpath 的 JavaScript 实现与原生实现得到的两种表示一并传给规范化器；规范化器按长度从长到短替换这些别名，避免 Windows 的短路径与长路径表示差异导致共享 fixture 反复变化。 macOS 参考流程使用 fork 进程运行常规 Vitest 项目。macOS arm64 上的 Node 24 曾在工作线程中执行 CJS 词法分析器时异常终止；进程边界能够隔离这一外部运行时故障，且无需从聚合流程中删除任何测试，而 Linux 与 Windows 仍使用开销更低的线程池。仓库自身引入的竞态均在相应的观测边界修复：开发构建产物的轮询逻辑每次发布重新扫描结果前，都会先暂存候选表、候选图和候选监视基线映射；构建产物缺失后会一直保持脏状态，直到成功计算内容哈希。PTY 就绪检测会在轮询检查前台进程组归属期间保留提示符候选项；常规静默时限也适用于交互式子进程继承提示符标记的情况。真实 PTY fixture 会在运行时拼接同步标记，使就绪等待逻辑不会把交互式 shell 的输入回显误判为子进程已就绪。实时链接场景下的包管理器 e2e 会保留由工作流预先准备的 Corepack 主目录、pnpm 元数据缓存和 store 缓存，同时隔离其他包管理器的可变缓存，因此不会在安装前丢弃可复用的包管理器状态。 -master 分支的参考作业仅用于诊断，不参与拉取请求所要求的 `all checks passed` 结果。拉取请求只运行其必需作业；向 master 推送时只运行三个串行参考作业。系统根据已完成托管作业的时间戳评估性能，并将其报告为测量结果，而不是写成 `timeout-minutes` 值。 +master 分支的参考作业仅用于诊断，不参与拉取请求所要求的 `all checks passed` 结果。拉取请求只运行其必需作业；向 master 推送时只运行串行参考作业。系统根据已完成托管作业的时间戳评估性能，并将其报告为测量结果，而不是写成 `timeout-minutes` 值。 可移植的参考流程使用 GitHub 标准的 `ubuntu-latest`、`macos-latest` 和 `windows-2025` 标签；`serial / windows` 是仅存的原生 Windows 作业，是 Wine 托管拉取请求通道背后的完整内核标尺（[Wine 通道决策](2026-07-27-wine-windows-gates-experiment.md)）。依据[必需 CI 决策](2026-07-23-portable-required-pull-request-ci.md)，拉取请求必需作业使用可移植的标准容量。更高核心数的托管运行器仍仅用于手动基准测试，因为正确性路径必须无需仓库外部的运行器配置即可运行。 diff --git a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml index 3d8e6fc395..68b4098d4f 100644 --- a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml +++ b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml @@ -1,6 +1,6 @@ # Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: -# pnpm run verify-translation-pairing --write -2026-07-22-evidence-based-larger-hosted-runners.md: fe11e6929545923d27fbf41f5a39f7dd2b9c3fbf -2026-07-22-evidence-based-larger-hosted-runners.zh.md: 47879284532a537cbe7e78aa2c495c4ef0be26c4 +# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md +2026-07-22-evidence-based-larger-hosted-runners.md: 67fc7ded5cffc6a219665f135a4c9e1cc4752691 +2026-07-22-evidence-based-larger-hosted-runners.zh.md: 71c5c067361b57fab5aae9e9ffa3850a30609db3 diff --git a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md index fe11e69295..67fc7ded5c 100644 --- a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md +++ b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md @@ -52,6 +52,8 @@ The process-bound coverage project contains exactly five suite files. Thirty-two Complete serial Linux, macOS, and Windows references run only when `master` moves. Pull requests use the enterprise required path plus standard-hosted compatibility jobs, while other larger-runner sizes run only by manual dispatch. +An additional serial Linux reference runs on the in-house self-hosted pool (`vm-backup` label: a 64-core VM with six always-on systemd-managed runner instances) on every `master` push. It is a hot-standby drill, not a required check: each run re-proves that the persistent VM can execute the complete unsharded aggregate. The actual switch is pre-wired: the three required Linux jobs resolve their pool through the writer-manageable `DSH_CI_FAILOVER` repository variable, so an outage response is setting one variable and re-running — no merge, which would be deadlocked behind the failing checks themselves ([runbook](2026-07-26-ci-failover-runbook.md)). The standby lane is push-triggered, so it always executes the base branch's workflow definition. Under failover, however, `pull_request` jobs do reach these runners with the PR merge ref's own workflow definition — the trust boundary is repository membership (the repository is private with forking disabled, and the selectors exclude Dependabot), as the [failover runbook](2026-07-26-ci-failover-runbook.md) records. + ## Alternatives considered **Keep the three coarse primary Linux lanes.** The core, CPU, and production-site jobs met the latency targets, but they paid three setup waves and left primary Node work sharded after larger runners were available. The all-size trace showed that one unnecessary dependency, not a lack of host capacity, kept the single-box aggregate above one minute. diff --git a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md index 4787928453..71c5c06736 100644 --- a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md +++ b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md @@ -52,6 +52,8 @@ Windows 仓库工作在超过 16 核后收益很小，但 32 核池可以让完 只有在 `master` 移动时，才运行完整的 Linux、macOS 和 Windows 串行参考。拉取请求使用企业级运行器必需路径和标准托管兼容性作业，其他大型运行器规格仅通过手动触发运行。 +另有一条串行 Linux 参考在每次 `master` 推送时运行于公司自有的自托管池（`vm-backup` 标签：一台 64 核虚拟机，运行 6 个常驻的 systemd 管理运行器实例）。它是热备演练而非必需检查：每次运行都重新证明这台持久化虚拟机能够执行完整的未分片聚合流程。实际切换机制已预先布线：三个必需 Linux 作业通过写者可管理的仓库变量 `DSH_CI_FAILOVER` 解析运行器池，因此故障响应就是设置一个变量并重跑——无需合并（合并本身会被正在失败的检查死锁）（[切换手册](2026-07-26-ci-failover-runbook.md)）。该热备通道由 push 触发，执行的始终是基线分支自身的工作流定义。但需要注意：故障切换期间，`pull_request` 作业确实会带着 PR merge 引用自带的工作流定义到达这些运行器——信任边界是仓库成员资格（仓库为私有且禁用 fork，选择器排除 Dependabot），详见[故障切换手册](2026-07-26-ci-failover-runbook.md)的记录。 + ## 曾考虑的替代方案 **保留 3 个粗粒度 Linux 主流程通道。** 核心、CPU 和生产网站作业均达到延迟目标，但它们需要 3 轮设置，而且在大型运行器已经可用后仍对主 Node 工作进行分片。全规格运行轨迹表明，让单机聚合流程超过 1 分钟的是一项不必要的依赖，而非主机容量不足。 diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml new file mode 100644 index 0000000000..a7b9e827fa --- /dev/null +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md +2026-07-26-ci-failover-runbook.md: 4e4f8ea7fc60cf76fd8308147bbf7cc0bac74798 +2026-07-26-ci-failover-runbook.zh.md: bb7e43fe55c9cced51f042de6503978ec9349d6b diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md new file mode 100644 index 0000000000..4e4f8ea7fc --- /dev/null +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md @@ -0,0 +1,50 @@ +# Agent Note: CI failover runbook — hosted pools → in-house pool + +Status: implemented + +English | [中文](2026-07-26-ci-failover-runbook.zh.md) + +## Problem + +The three required Linux worker jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) run on the hosted enterprise 32-core pools; the required verdict job that aggregates them (`all checks passed`) runs on standard `ubuntu-latest`. When the enterprise pools degrade — jobs queue indefinitely or the enterprise labels vanish — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. **Scope: this switch recovers an enterprise Linux-pool outage.** The verdict's other required dependencies (`node-compat`, `python-sdk`, `windows`) stay on standard hosted runners by design (the portable boundary); in a broader GitHub-hosted capacity failure that also takes out the standard pools, those dependencies still block `all checks passed`, and only the Windows leg has no in-house substitute at all — during the 2026-07-27 outage the standard pools recovered first, which is the ordering this design bets on. An outage therefore needs a switch any responder with repository write access can throw without merging anything. + +## Decision + +Each of the three required Linux worker jobs — and the `all checks passed` verdict job, which would otherwise stay queued on the failed pool even after every worker passed — resolves its runner pool through the `DSH_CI_FAILOVER` repository variable. Unset (normal), they run on the hosted enterprise pools. Set to `selfhosted` by any repository writer, all four retarget onto the in-house self-hosted `vm-backup` pool, coverage and snapshot concurrency drop to shared-VM bounds, and the hosted-path pnpm cache restores are skipped. The switch is writer-manageable repository state, not a merge, so it works while every check is red. The in-house pool's readiness is continuously re-proven by the `serial / linux (self-hosted standby)` lane, which runs the complete unsharded aggregate on every master push. + +### What the in-house pool is + +`vm-backup`: one 64-core VM, six always-on systemd-managed runner instances. Check the latest `serial / linux (self-hosted standby)` run before switching: a green standby is verified-yesterday capacity. + +### Switch (any repository writer, ~1 minute, no merge) + +1. Repository **Settings → Secrets and variables → Actions → Variables → New repository variable**: name `DSH_CI_FAILOVER`, value `selfhosted`. +2. Retrigger the required jobs so they re-resolve their pool. Jobs already **queued** for the hosted labels do not retarget and cannot be re-run in place, so for the documented indefinite-queue outage, cancel the stuck run and re-run all jobs, or push a new commit; "Re-run failed jobs" only helps once a job has actually failed rather than queued. +3. That is the entire switch. Under failover the workflow also, automatically: drops `DSH_COVERAGE_MAX_WORKERS` to 8 and `DSH_SNAPSHOT_MAX_CONCURRENCY` to 12 (sized for six always-on instances: worst case 6 × 8 = 48 coverage workers on the 64-core VM) (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs). + +#**Dependabot exception.** All four selectors deliberately exclude `dependabot[bot]`: under failover, Dependabot PRs stay queued for the hosted pool rather than executing dependency-supplied code on the persistent VM. A Dependabot PR that remains queued during an outage is expected behavior, not a failed switch; it completes when the hosted pool recovers. + +**Who can flip the variable.** GitHub's API lets any collaborator with write access manage repository variables, so the switch is writer-level, not strictly admin-only. In this repository's trust model that is not an escalation: the runner group admits all workflows of this private, fork-disabled repository (a deliberate trade to make PR-ref failover possible at all), so any writer could already reach the VM by pushing a branch workflow. The boundary against untrusted code is repository membership; the variable only routes work for members. + +## Capacity during failover + +Six always-on instances absorb normal PR traffic (the pool's steady-state load is one serial standby job per master push, so failover capacity is effectively the full pool). If queues still build, register additional instances with an org registration token (org Settings → Actions → Runners → New runner). Clone an existing runner directory **excluding its identity files** — `rsync -a --exclude '.runner*' --exclude '.credentials*' --exclude '_diag' --exclude '_work' / /` (the globs also catch `.runner_migrated`/`.credentials_migrated`, which GitHub writes on migrated runners and which equally trigger the already-configured refusal) — then run `config.sh` (copying `.runner`/`.credentials` verbatim makes it refuse with "already configured"), and **start the listener**: `sudo ./svc.sh install ubuntu && sudo ./svc.sh start`. Registration alone leaves the runner offline; only a started service adds capacity. About a minute per instance. + + +### Switch back + +Delete the `DSH_CI_FAILOVER` variable (or set it to anything other than `selfhosted`). New runs resolve back to the hosted enterprise pools. Remove any extra instances that were registered during the incident. + +### Trust boundary + +The variable is writer-manageable repository state; a pull request event itself can neither set it nor read a different value into effect, and the selector expressions live in workflow definitions. Note that under failover, `pull_request` runs execute the PR merge ref's own workflow definition — the boundary against untrusted code is repository membership (private, forking disabled, Dependabot excluded by the selectors), not the variable. Note on runner-group policy: pinning the runner group to the master-ref workflow is **incompatible** with this failover — the four failover jobs are `pull_request` runs evaluated from PR merge refs, and a master-pinned group leaves them queued (observed live on 2026-07-27; the group was widened to all workflows of this repository to unblock the switch). A stricter runner-side policy therefore costs PR failover; the shipped posture accepts repository-scoped, all-workflow group access. + +## Alternatives considered + +**Merge a workflow change to switch pools.** Rejected because the outage that motivates the switch is exactly the state in which no PR can merge: the required checks are the ones failing. A repository variable is writer-manageable state that takes effect on re-run without a merge. + +**Keep the self-hosted pool always in the required path.** Rejected because it trades hosted-pool availability for the in-house VM's, moving a single point of failure rather than adding a fallback. The variable keeps the hosted pools primary and the self-hosted pool a proven, one-action standby. + +## Consequences + +Recovering from a hosted-pool outage is a single variable (any writer) plus a re-run, with no merge on the critical path. The cost is a second runner topology to keep working: the standby lane exercises it on every master push so the failover target never goes stale, and the concurrency and cache-restore branches in `ci.yml` carry a `selfhosted` leg that must stay in step with the hosted leg. diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md new file mode 100644 index 0000000000..bb7e43fe55 --- /dev/null +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md @@ -0,0 +1,50 @@ +# Agent Note: CI 故障切换手册 — 托管池 → 自有池 + +Status: implemented + +[English](2026-07-26-ci-failover-runbook.md) | 中文 + +## 问题 + +[CI](../../../../.github/workflows/ci.yml) 中三个必需的 Linux 工作作业（`node 24 / static`、`node 24 / coverage`、`node 24 / snapshots and artifacts`）运行在托管的企业级 32 核池上；聚合它们的必需判定作业（`all checks passed`）运行在标准 `ubuntu-latest` 上。当企业池发生故障——作业无限排队或企业标签消失——所有开启的拉取请求都无法合并，而"合并一个修复"这一常规恢复手段本身正被那些无法运行的必需检查死锁。**适用范围：本切换恢复的是企业级 Linux 池故障。**判定作业的其余必需依赖（`node-compat`、`python-sdk`、`windows`）按设计留在标准托管运行器上（可移植边界）；若更大范围的 GitHub 托管容量故障连标准池一并击倒，这些依赖仍会阻塞 `all checks passed`，且只有 Windows 这条腿完全没有自有替代——2026-07-27 的故障中标准池率先恢复，本设计押注的正是这一顺序。因此故障需要一个任何具备仓库写权限的响应者都能在不合并任何代码的情况下触发的开关。 + +## 决策 + +三个必需的 Linux 工作作业——以及 `all checks passed` 判定作业（若不随切换，即使全部工作作业通过，它仍会滞留在故障池的队列中）——各自通过仓库变量 `DSH_CI_FAILOVER` 解析运行器池。变量不存在（正常）时它们运行在托管企业池上；由任何具备写权限的协作者设为 `selfhosted` 时，四者全部切换到公司自有的自托管 `vm-backup` 池，coverage 与 snapshot 的并发降到共享虚拟机上限，并跳过托管路径的 pnpm 缓存恢复。这个开关是写者可管理的仓库状态而非一次合并，因此在所有检查都是红色时仍然有效。自有池的就绪状态由 `serial / linux (self-hosted standby)` 通道持续验证——每次 master 推送都在其上运行完整的未分片聚合流程。 + +### 自有池是什么 + +`vm-backup`：一台 64 核虚拟机，6 个常驻 systemd 管理的运行器实例。切换前先看 `serial / linux (self-hosted standby)` 最近一次运行：绿色 = 这套环境昨天刚被全量验证过。 + +### 切换步骤（任何具备写权限的协作者，约 1 分钟，无需合并） + +1. 仓库 **Settings → Secrets and variables → Actions → Variables → New repository variable**：名称 `DSH_CI_FAILOVER`，值 `selfhosted`。 +2. 重新触发必需作业，使其重新解析运行器池。已经为托管标签**排队**的作业不会重定向，也无法原地 re-run，因此对于本手册所述的无限排队故障，应取消卡住的运行并 re-run all jobs，或推送一个新提交；“Re-run failed jobs”只有在作业真正失败（而非仍在排队）时才有用。 +3. 切换到此完成。故障切换状态下工作流还会自动：把 `DSH_COVERAGE_MAX_WORKERS` 降为 8、`DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 12（按 6 个常驻实例定容：最坏 6 × 8 = 48 个覆盖率工作进程对 64 核）（共享虚拟机的争抢上限），并跳过托管路径的 pnpm 缓存恢复（虚拟机的持久 store 直接提供热安装）。 + +#**Dependabot 例外。**四个选择器都刻意排除了 `dependabot[bot]`：故障切换期间，Dependabot 拉取请求继续在托管池排队，而不是把依赖方提供的代码放到持久化虚拟机上执行。故障期间 Dependabot PR 持续排队是预期行为而非切换失败；托管池恢复后它会自行完成。 + +**谁能扳动这个变量。**GitHub 的 API 允许任何具有写权限的协作者管理仓库变量，因此该开关实际是写者级而非严格的管理员级。在本仓库的信任模型下这并不构成越权：runner group 接纳本私有、禁 fork 仓库的全部工作流（这是让 PR 引用的故障切换得以成立的刻意取舍），因此任何写者本就可以通过推送分支工作流触达这台虚拟机。抵御不可信代码的边界是仓库成员资格；变量只是为成员路由工作。 + +## 切换期间的容量 + +6 个常驻实例可承接正常 PR 流量（该池平时唯一的稳态负载是每次 master 推送一个串行热备作业，故障切换时几乎全池可用）。若仍出现排队，用组织级注册 token（组织 Settings → Actions → Runners → New runner）追加注册实例。复制现有 runner 目录时**必须排除身份文件**——`rsync -a --exclude '.runner*' --exclude '.credentials*' --exclude '_diag' --exclude '_work' / /`（通配同时排除 `.runner_migrated`/`.credentials_migrated`——GitHub 会在迁移过的运行器上写入这些文件，它们同样会触发 already-configured 拒绝）——再跑 `config.sh`（原样拷贝 `.runner`/`.credentials` 会使其以 "already configured" 拒绝），然后**启动监听器**：`sudo ./svc.sh install ubuntu && sudo ./svc.sh start`。仅注册不会上线；只有启动了服务的 runner 才会增加容量。每个约一分钟。 + + +### 切回 + +删除 `DSH_CI_FAILOVER` 变量（或改为 `selfhosted` 以外的任何值），新的运行即解析回托管企业池。若故障期间追加注册过实例，将其移除。 + +### 信任边界 + +该变量是写者可管理的仓库状态；`pull_request` 事件本身既不能设置它，也不能让不同的值生效，选择器表达式存在于工作流定义中。需要注意：故障切换期间，`pull_request` 运行执行的是 PR merge 引用自带的工作流定义——抵御不可信代码的边界是仓库成员资格（私有、禁 fork、选择器排除 Dependabot），而非该变量。关于 runner group 策略的说明：把 runner group 绑定到 master 引用的工作流与本故障切换机制**不兼容**——四个故障切换作业是从 PR merge 引用求值的 `pull_request` 运行，master 绑定的组会让它们持续排队（2026-07-27 实际故障中亲历；当时将组放宽为本仓库全部工作流才疏通了切换）。更严格的运行器侧策略以牺牲 PR 故障切换为代价；当前采用的形态是仓库范围、全工作流的组访问。 + +## 曾考虑的替代方案 + +**通过合并一次工作流改动来切换池。** 否决，因为触发切换的故障状态恰恰是任何 PR 都无法合并的状态：必需检查正是失败的那些。仓库变量是写者可管理的状态，重跑即生效，无需合并。 + +**让自托管池长期处于必需路径中。** 否决，因为这是拿托管池的可用性去换自有虚拟机的可用性，只是搬移了单点故障而非增加回退。该变量让托管池保持主路径，自托管池作为一个经过验证、一步即可启用的热备。 + +## 后果 + +从托管池故障中恢复只需一个变量（任何写者可设）加一次重跑，关键路径上没有合并。代价是要维护第二套运行器拓扑：热备通道在每次 master 推送时都运行它，使故障切换目标永不失效；而 `ci.yml` 中的并发与缓存恢复分支带有一条 `selfhosted` 支路，必须与托管支路保持同步。 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 19deb87034..e3b519130e 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -30,9 +30,24 @@ jobs: # Three enterprise jobs isolate coverage, static analysis, and the # build-backed consumer tail. The static job publishes its exact build so # consumers do not repeat the longest part of their critical path. + # + # FAILOVER: each Linux enterprise job resolves its pool through the + # DSH_CI_FAILOVER repository variable. Unset (normal), the expressions + # pick the hosted enterprise pools below. Setting the variable to + # 'selfhosted' (repo Settings → Actions → Variables; writer-manageable + # repository state — not PR-editable, no merge required) retargets all + # three onto the in-house + # vm-backup pool and re-running the failed jobs is the entire switch — + # see .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md. The + # in-house pool's readiness is re-proven on every master push by the + # serial-linux-selfhosted standby lane below. node-24: if: github.event_name == 'pull_request' - runs-on: dsh-enterprise-ubuntu-latest-32core-test + runs-on: >- + ${{ vars.DSH_CI_FAILOVER == 'selfhosted' + && github.event.pull_request.user.login != 'dependabot[bot]' + && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]') + || 'dsh-enterprise-ubuntu-latest-32core-test' }} name: node 24 / static env: DSH_GATE_CONCURRENCY: '8' @@ -44,8 +59,10 @@ jobs: persist-credentials: false # Pull requests consume the default-branch cache but do not put cache - # compression and upload on the paid latency-critical path. + # compression and upload on the paid latency-critical path. Skipped + # under failover — see the coverage lane's identical rationale. - uses: actions/cache/restore@v4 + if: vars.DSH_CI_FAILOVER != 'selfhosted' || github.event.pull_request.user.login == 'dependabot[bot]' with: path: /home/runner/.local/share/pnpm/store/v11 key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }} @@ -81,17 +98,30 @@ jobs: node-24-coverage: if: github.event_name == 'pull_request' - runs-on: dsh-enterprise-ubuntu-24-04-32core-test + runs-on: >- + ${{ vars.DSH_CI_FAILOVER == 'selfhosted' + && github.event.pull_request.user.login != 'dependabot[bot]' + && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]') + || 'dsh-enterprise-ubuntu-24-04-32core-test' }} name: node 24 / coverage env: - DSH_COVERAGE_MAX_WORKERS: '24' + # Failover shrinks the worker bound: the hosted 32-core runner is + # exclusive to one job, but the failover pool shares one 64-core VM + # across six always-on runner instances, and the timing-sensitive + # process suites have documented aggregate-contention failures. + # 8 × 6 instances = 48 workers worst case on 64 cores. + DSH_COVERAGE_MAX_WORKERS: ${{ vars.DSH_CI_FAILOVER == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '8' || '24' }} DSH_GATE_CONCURRENCY: '8' steps: - uses: actions/checkout@v6 with: persist-credentials: false + # Skipped under failover: the self-hosted VM's persistent pnpm store + # serves warm installs directly, and this hosted-path restore would + # spend ~52 s pulling ~180 MB into a path pnpm never reads there. - uses: actions/cache/restore@v4 + if: vars.DSH_CI_FAILOVER != 'selfhosted' || github.event.pull_request.user.login == 'dependabot[bot]' with: path: /home/runner/.local/share/pnpm/store/v11 key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }} @@ -122,7 +152,11 @@ jobs: node-24-consumers: needs: node-24 if: github.event_name == 'pull_request' - runs-on: dsh-enterprise-ubuntu-latest-32core-test + runs-on: >- + ${{ vars.DSH_CI_FAILOVER == 'selfhosted' + && github.event.pull_request.user.login != 'dependabot[bot]' + && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]') + || 'dsh-enterprise-ubuntu-latest-32core-test' }} name: node 24 / snapshots and artifacts env: DSH_ESLINT_CACHE: '1' @@ -130,7 +164,8 @@ jobs: DSH_GATE_CONCURRENCY: '8' DSH_NODE_COMPAT_SKIP_TYPECHECK: '1' DSH_PUBLINT_CONCURRENCY: '8' - DSH_SNAPSHOT_MAX_CONCURRENCY: '32' + # Failover halves snapshot concurrency for the shared 64-core VM. + DSH_SNAPSHOT_MAX_CONCURRENCY: ${{ vars.DSH_CI_FAILOVER == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '12' || '32' }} steps: - uses: actions/checkout@v6 with: @@ -144,7 +179,9 @@ jobs: - name: Restore built tree run: tar -xzf "$RUNNER_TEMP/node-24-built-tree.tar.gz" + # Skipped under failover — see the coverage lane's identical rationale. - uses: actions/cache/restore@v4 + if: vars.DSH_CI_FAILOVER != 'selfhosted' || github.event.pull_request.user.login == 'dependabot[bot]' with: path: /home/runner/.local/share/pnpm/store/v11 key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }} @@ -555,6 +592,56 @@ jobs: DSH_SNAPSHOT_MAX_CONCURRENCY: '1' run: pnpm run check:ci + # Hot-standby drill for the in-house self-hosted pool: every master move + # re-runs the complete unsharded aggregate on the persistent 64-core VM, + # continuously proving that environment can take over a required lane if + # the hosted pools degrade (the switch is then setting the writer-manageable + # DSH_CI_FAILOVER variable — see the failover runbook, no merge required). + # Push-triggered, so this lane always executes the base branch's own + # workflow definition. (Under failover, pull_request jobs do reach these + # runners with the PR merge ref's workflow — the boundary there is + # repository membership: private, forking disabled, Dependabot excluded.) + # Non-blocking for + # pull requests; no cache steps because the VM's persistent pnpm store and + # tool caches make them redundant (and saving here would poison the hosted + # cache namespace with self-hosted paths). + serial-linux-selfhosted: + if: github.event_name == 'push' && github.ref == 'refs/heads/master' + name: serial / linux (self-hosted standby) + runs-on: [self-hosted, linux, x64, vm-backup] + steps: + # Full history + DSH_ARCHIVE_BASE_REF below: same frozen-archive + # comparison as serial-linux. Depth 2 would miss github.event.before + # on multi-commit or force pushes; full fetch is cheap here because + # checkout resolves against the VM's local mirror. + - uses: actions/checkout@v6 + with: + fetch-depth: 0 + + - uses: actions/setup-node@v6 + with: + node-version: ${{ env.PRIMARY_NODE_VERSION }} + + - name: Enable corepack (pnpm) + run: corepack enable + + - name: Install (immutable) + run: pnpm install --frozen-lockfile + + - name: Prepare bubblewrap (unrestrict userns) + run: bash scripts/prepare-ci-bubblewrap.sh + + - name: Run complete unsharded primary Node CI serially + env: + DSH_ARCHIVE_BASE_REF: ${{ github.event.before }} + DSH_COVERAGE_MAX_WORKERS: '1' + DSH_E2E_MAX_WORKERS: '1' + DSH_ESLINT_CACHE: '1' + DSH_GATE_CONCURRENCY: '1' + DSH_PUBLINT_CONCURRENCY: '1' + DSH_SNAPSHOT_MAX_CONCURRENCY: '1' + run: pnpm run check:ci + serial-macos: if: github.event_name == 'push' && github.ref == 'refs/heads/master' name: serial / macos @@ -876,8 +963,16 @@ jobs: # 'cancelled' and 'skipped'. all-checks-passed: name: all checks passed - # This bookkeeping-only verdict must not depend on custom-pool provisioning. - runs-on: ubuntu-latest + # This bookkeeping-only verdict must not depend on custom-pool + # provisioning (master's #734 moved it to ubuntu-latest) — and under + # failover it follows the same selector as the worker jobs it + # aggregates, so a standard-hosted outage cannot strand the + # branch-protection verdict either (observed 2026-07-27). + runs-on: >- + ${{ vars.DSH_CI_FAILOVER == 'selfhosted' + && github.event.pull_request.user.login != 'dependabot[bot]' + && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]') + || 'ubuntu-latest' }} needs: [node-24, node-24-coverage, node-24-consumers, node-compat, python-sdk, windows] if: always() && github.event_name == 'pull_request' steps: