diff --git a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml index 1b14f5b689..cd3ae7a181 100644 --- a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml +++ b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write -2026-07-22-evidence-based-larger-hosted-runners.md: 6654f5eb3e21b48c6d33fd9d74ebd23cf3065d54 -2026-07-22-evidence-based-larger-hosted-runners.zh.md: 3fe5715b20d3b881f8fb439b61900bdb84e5a588 +2026-07-22-evidence-based-larger-hosted-runners.md: dd07280092565257f4b5324f997d5efd4c9c51cc +2026-07-22-evidence-based-larger-hosted-runners.zh.md: a9c034b643da0cb9148d08c0300f3e142eec31e6 diff --git a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md index 6654f5eb3e..dd07280092 100644 --- a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md +++ b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md @@ -48,7 +48,7 @@ The process-bound coverage project contains exactly five suite files. Thirty-two Complete serial Linux, macOS, and Windows references run only when `master` moves. Pull requests use the enterprise required path plus standard-hosted compatibility jobs, while other larger-runner sizes run only by manual dispatch. -An additional serial Linux reference runs on the in-house self-hosted pool (`vm-backup` label: a 64-core VM with four always-on systemd-managed runner instances plus four registered spares) on every `master` push. It is a hot-standby drill, not a required check: each run re-proves that the persistent VM can execute the complete unsharded aggregate, so if the enterprise pools degrade, a required lane can be retargeted with a one-line `runs-on` change onto an environment with continuously verified evidence. Because the lane is push-triggered, it always executes the base branch's workflow definition — no pull-request-editable path can route code to these runners, and the repository additionally keeps forking disabled. +An additional serial Linux reference runs on the in-house self-hosted pool (`vm-backup` label: a 64-core VM with four always-on systemd-managed runner instances plus four registered spares) on every `master` push. It is a hot-standby drill, not a required check: each run re-proves that the persistent VM can execute the complete unsharded aggregate. The actual switch is pre-wired: the three required Linux jobs resolve their pool through the admin-only `DSH_CI_FAILOVER` repository variable, so an outage response is setting one variable and re-running — no merge, which would be deadlocked behind the failing checks themselves ([runbook](ci-failover-runbook.md)). Because the standby lane is push-triggered, it always executes the base branch's workflow definition — no pull-request-editable path can route code to these runners, and the repository additionally keeps forking disabled. ## Alternatives considered diff --git a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md index 3fe5715b20..a9c034b643 100644 --- a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md +++ b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md @@ -48,7 +48,7 @@ Windows 仓库工作在超过 16 核后收益很小，但 32 核池可以让完 只有在 `master` 移动时，才运行完整的 Linux、macOS 和 Windows 串行参考。拉取请求使用企业级运行器必需路径和标准托管兼容性作业，其他大型运行器规格仅通过手动触发运行。 -另有一条串行 Linux 参考在每次 `master` 推送时运行于公司自有的自托管池（`vm-backup` 标签：一台 64 核虚拟机，运行 4 个常驻的 systemd 管理运行器实例，另有 4 个已注册备用位）。它是热备演练而非必需检查：每次运行都重新证明这台持久化虚拟机能够执行完整的未分片聚合流程，因此当企业池发生故障时，只需一行 `runs-on` 修改即可把必需通道切换到一个具有持续验证证据的环境上。该通道由 push 触发，执行的始终是基线分支自身的工作流定义——不存在任何可由拉取请求编辑的路径能把代码路由到这些运行器上；此外仓库继续保持禁用 fork。 +另有一条串行 Linux 参考在每次 `master` 推送时运行于公司自有的自托管池（`vm-backup` 标签：一台 64 核虚拟机，运行 4 个常驻的 systemd 管理运行器实例，另有 4 个已注册备用位）。它是热备演练而非必需检查：每次运行都重新证明这台持久化虚拟机能够执行完整的未分片聚合流程。实际切换机制已预先布线：三个必需 Linux 作业通过仅限管理员的仓库变量 `DSH_CI_FAILOVER` 解析运行器池，因此故障响应就是设置一个变量并重跑——无需合并（合并本身会被正在失败的检查死锁）（[切换手册](ci-failover-runbook.zh.md)）。该热备通道由 push 触发，执行的始终是基线分支自身的工作流定义——不存在任何可由拉取请求编辑的路径能把代码路由到这些运行器上；此外仓库继续保持禁用 fork。 ## 曾考虑的替代方案 diff --git a/.agents/notes/implemented/process/ci-failover-runbook.i18n.yaml b/.agents/notes/implemented/process/ci-failover-runbook.i18n.yaml new file mode 100644 index 0000000000..294ed38ddf --- /dev/null +++ b/.agents/notes/implemented/process/ci-failover-runbook.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write +ci-failover-runbook.md: d22c93fbedd3216e71bc24101dfa06dc606521c2 +ci-failover-runbook.zh.md: d7d26287191165cd3cb2666de4b1c7d6217ba71d diff --git a/.agents/notes/implemented/process/ci-failover-runbook.md b/.agents/notes/implemented/process/ci-failover-runbook.md new file mode 100644 index 0000000000..d22c93fbed --- /dev/null +++ b/.agents/notes/implemented/process/ci-failover-runbook.md @@ -0,0 +1,33 @@ +# Agent Note: CI failover runbook — hosted pools → in-house pool + +Status: implemented + +English | [中文](ci-failover-runbook.zh.md) + +## What this is + +The three required Linux jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) resolve their runner pool through the `DSH_CI_FAILOVER` repository variable. Normally the variable is unset and they run on the hosted enterprise 32-core pools. When the hosted pools are degraded (jobs queue indefinitely, the enterprise labels vanish, or GitHub-side capacity fails), a repository admin can retarget all three onto the in-house self-hosted pool without merging anything — merging would itself be blocked by the very checks that are failing. + +The in-house pool (`vm-backup`: one 64-core VM, four always-on systemd-managed runner instances, four registered spares) is continuously re-proven by the `serial / linux (self-hosted standby)` lane, which runs the complete unsharded aggregate on every master push. Check its latest run before switching: green standby = verified-yesterday capacity. + +## Switch (repo admin, ~1 minute, no merge) + +1. Repository **Settings → Secrets and variables → Actions → Variables → New repository variable**: name `DSH_CI_FAILOVER`, value `selfhosted`. +2. Re-run the failed/queued required jobs (Re-run failed jobs on affected PRs, or let new pushes pick it up). +3. That is the entire switch. Under failover the workflow also, automatically: halves `DSH_COVERAGE_MAX_WORKERS` to 12 and `DSH_SNAPSHOT_MAX_CONCURRENCY` to 16 (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs). + +## Capacity during failover + +Four always-on instances absorb normal PR traffic. If queues build, bring the four registered spares online on the VM (no token needed — they are already registered): + +```bash +for i in 7 8 9 10; do cd /data_local/actions-runner-$i && sudo ./svc.sh install ubuntu && sudo ./svc.sh start; done +``` + +## Switch back + +Delete the `DSH_CI_FAILOVER` variable (or set it to anything other than `selfhosted`). New runs resolve back to the hosted enterprise pools. Stop the spare instances if they were started. + +## Trust boundary + +The variable is repository-admin-only state: a pull request can neither set it nor read a different value into effect, and the expressions live in the base branch's workflow definition. This failover path therefore adds no PR-editable route to the self-hosted pool. (Runner-side enforcement — an org-level runner group restricting these runners to the master-ref workflow — is tracked separately and composes with this mechanism.) diff --git a/.agents/notes/implemented/process/ci-failover-runbook.zh.md b/.agents/notes/implemented/process/ci-failover-runbook.zh.md new file mode 100644 index 0000000000..d7d2628719 --- /dev/null +++ b/.agents/notes/implemented/process/ci-failover-runbook.zh.md @@ -0,0 +1,33 @@ +# Agent Note: CI 故障切换手册 — 托管池 → 自有池 + +Status: implemented + +[English](ci-failover-runbook.md) | 中文 + +## 这是什么 + +[CI](../../../../.github/workflows/ci.yml) 中三个必需的 Linux 作业（`node 24 / static`、`node 24 / coverage`、`node 24 / snapshots and artifacts`）通过仓库变量 `DSH_CI_FAILOVER` 解析运行器池。正常情况下该变量不存在，作业运行在托管的企业级 32 核池上。当托管池发生故障（作业无限排队、企业标签消失或 GitHub 侧容量故障）时，仓库管理员无需合并任何代码即可把三个作业整体切换到公司自有的自托管池——此时合并本身正被这些失败的检查阻塞，任何"先合 PR 再切换"的方案都是死锁。 + +自有池（`vm-backup`：一台 64 核虚拟机，4 个常驻 systemd 管理的运行器实例，另有 4 个已注册备用位）由 `serial / linux (self-hosted standby)` 通道持续验证——每次 master 推送都在其上运行完整的未分片聚合流程。切换前先看该通道最近一次运行：绿色 = 这套环境昨天刚被全量验证过。 + +## 切换步骤（仓库管理员，约 1 分钟，无需合并） + +1. 仓库 **Settings → Secrets and variables → Actions → Variables → New repository variable**：名称 `DSH_CI_FAILOVER`，值 `selfhosted`。 +2. 对受影响 PR 的失败/排队作业点 Re-run failed jobs（或等新推送自然触发）。 +3. 切换到此完成。故障切换状态下工作流还会自动：把 `DSH_COVERAGE_MAX_WORKERS` 降为 12、`DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 16（共享虚拟机的争抢上限），并跳过托管路径的 pnpm 缓存恢复（虚拟机的持久 store 直接提供热安装）。 + +## 切换期间的容量 + +4 个常驻实例可承接正常 PR 流量。若出现排队，在虚拟机上把 4 个已注册的备用位拉起（无需 token——它们已注册）： + +```bash +for i in 7 8 9 10; do cd /data_local/actions-runner-$i && sudo ./svc.sh install ubuntu && sudo ./svc.sh start; done +``` + +## 切回 + +删除 `DSH_CI_FAILOVER` 变量（或改为 `selfhosted` 以外的任何值），新的运行即解析回托管企业池。若启动过备用实例，将其停止。 + +## 信任边界 + +该变量是仅限仓库管理员的状态：拉取请求既不能设置它，也不能让不同的值生效，且表达式存在于基线分支的工作流定义中。因此这条故障切换路径没有增加任何可由 PR 编辑的自托管池访问途径。（运行器侧的强制约束——通过组织级 runner group 把这批运行器限定到 master 引用的工作流——另行跟踪，与本机制互补。） diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2666c93b8c..88722804f9 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -30,9 +30,22 @@ jobs: # Three enterprise jobs isolate coverage, static analysis, and the # build-backed consumer tail. The static job publishes its exact build so # consumers do not repeat the longest part of their critical path. + # + # FAILOVER: each Linux enterprise job resolves its pool through the + # DSH_CI_FAILOVER repository variable. Unset (normal), the expressions + # pick the hosted enterprise pools below. Setting the variable to + # 'selfhosted' (repo Settings → Actions → Variables; admin-only, not + # PR-editable, no merge required) retargets all three onto the in-house + # vm-backup pool and re-running the failed jobs is the entire switch — + # see .agents/notes/implemented/process/ci-failover-runbook.md. The + # in-house pool's readiness is re-proven on every master push by the + # serial-linux-selfhosted standby lane below. node-24: if: github.event_name == 'pull_request' - runs-on: dsh-enterprise-ubuntu-latest-32core-test + runs-on: >- + ${{ vars.DSH_CI_FAILOVER == 'selfhosted' + && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]') + || 'dsh-enterprise-ubuntu-latest-32core-test' }} name: node 24 / static env: DSH_GATE_CONCURRENCY: '8' @@ -77,17 +90,28 @@ jobs: node-24-coverage: if: github.event_name == 'pull_request' - runs-on: dsh-enterprise-ubuntu-24-04-32core-test + runs-on: >- + ${{ vars.DSH_CI_FAILOVER == 'selfhosted' + && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]') + || 'dsh-enterprise-ubuntu-24-04-32core-test' }} name: node 24 / coverage env: - DSH_COVERAGE_MAX_WORKERS: '24' + # Failover halves the worker bound: the hosted 32-core runner is + # exclusive to one job, but the failover pool shares one 64-core VM + # across four runner instances, and the timing-sensitive process + # suites have documented aggregate-contention failures. + DSH_COVERAGE_MAX_WORKERS: ${{ vars.DSH_CI_FAILOVER == 'selfhosted' && '12' || '24' }} DSH_GATE_CONCURRENCY: '8' steps: - uses: actions/checkout@v6 with: persist-credentials: false + # Skipped under failover: the self-hosted VM's persistent pnpm store + # serves warm installs directly, and this hosted-path restore would + # spend ~52 s pulling ~180 MB into a path pnpm never reads there. - uses: actions/cache/restore@v4 + if: vars.DSH_CI_FAILOVER != 'selfhosted' with: path: /home/runner/.local/share/pnpm/store/v11 key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }} @@ -118,7 +142,10 @@ jobs: node-24-consumers: needs: node-24 if: github.event_name == 'pull_request' - runs-on: dsh-enterprise-ubuntu-latest-32core-test + runs-on: >- + ${{ vars.DSH_CI_FAILOVER == 'selfhosted' + && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]') + || 'dsh-enterprise-ubuntu-latest-32core-test' }} name: node 24 / snapshots and artifacts env: DSH_ESLINT_CACHE: '1' @@ -126,7 +153,8 @@ jobs: DSH_GATE_CONCURRENCY: '8' DSH_NODE_COMPAT_SKIP_TYPECHECK: '1' DSH_PUBLINT_CONCURRENCY: '8' - DSH_SNAPSHOT_MAX_CONCURRENCY: '32' + # Failover halves snapshot concurrency for the shared 64-core VM. + DSH_SNAPSHOT_MAX_CONCURRENCY: ${{ vars.DSH_CI_FAILOVER == 'selfhosted' && '16' || '32' }} steps: - uses: actions/checkout@v6 with: @@ -140,7 +168,9 @@ jobs: - name: Restore built tree run: tar -xzf "$RUNNER_TEMP/node-24-built-tree.tar.gz" + # Skipped under failover — see the coverage lane's identical rationale. - uses: actions/cache/restore@v4 + if: vars.DSH_CI_FAILOVER != 'selfhosted' with: path: /home/runner/.local/share/pnpm/store/v11 key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}