ci: exempt only push from concurrency cancellation
The two self-hosted standby drills each run their complete unsharded aggregate with one gate worker, which takes longer than the interval between master merges, so unconditional cancel-in-progress supersedes a drill before it reaches a verdict and the lane yields no readiness evidence for the failover runbook to point a responder at. Exempt push and nothing else. This has to be decided at workflow level: cancellation applies to the whole superseded run, so a job-level concurrency group cannot exempt its job. The negated form is load-bearing — naming pull_request alone would also stop cancelling workflow_dispatch, and each runner benchmark fans out to twelve larger runners for up to fifteen minutes in this same group on master, so a re-dispatch would queue ahead of a drill instead of replacing a stale measurement. It does not promise every push run finishes: a newer pending run still displaces an older one, only that the lanes periodically reach a verdict. A master push carries only wine-apt-cache and the two drills; every other job is pull-request-gated, workflow_dispatch-gated, or if: false. The spec pins that set and classifies by exact condition, since a negated event test mentions the event it excludes.
This commit is contained in:
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md
|
||||
2026-07-26-ci-failover-runbook.md: 47901844f4ec581dff8500cc429c54a076b7642b
|
||||
2026-07-26-ci-failover-runbook.zh.md: 88a793e144a4d06718ffe323a94f8e81f8e62b82
|
||||
2026-07-26-ci-failover-runbook.md: 6f2b4bfba9255549b0887aab9061a6513cb3a68f
|
||||
2026-07-26-ci-failover-runbook.zh.md: 7bb1a025a71eb690d2d90fcdbddbdd56355786fc
|
||||
|
||||
@@ -12,6 +12,10 @@ The three required Linux worker jobs in [CI](../../../../.github/workflows/ci.ym
|
||||
|
||||
Each of the three required Linux worker jobs, the independent native Windows job, and the `all checks passed` verdict job — which would otherwise stay queued on the failed pool even after every worker passed — resolves its runner pool through the `DSH_CI_FAILOVER` repository variable. Unset (normal), they run on the hosted enterprise pools. Set to `selfhosted` by any repository writer, all five retarget onto the in-house self-hosted pools: the Linux jobs and verdict onto the `vm-backup` pool, coverage and snapshot concurrency drop to shared-VM bounds, and the hosted-path pnpm cache restores are skipped; the native Windows job onto the `dsh-win-ci` pool. The switch is writer-manageable repository state, not a merge, so it works while every check is red. The in-house pools' readiness is continuously re-proven by the `serial / linux (self-hosted standby)` and `serial / windows (self-hosted standby)` lanes, which run the complete unsharded aggregates on every master push.
|
||||
|
||||
`ci.yml` exempts exactly one event from `cancel-in-progress` (`${{ github.event_name != 'push' }}`), so a superseded pull-request run is still cancelled while a push run is left to run. Each drill runs its complete unsharded aggregate with one gate worker, which takes longer than the interval between master merges; under unconditional cancellation a drill is superseded before reaching a verdict and the lane yields no readiness evidence for a responder to check. This is not a guarantee that every push run finishes: GitHub keeps a single pending entry per group, so a newer pending run displaces an older one, and during busy periods intermediate push runs still end as `cancelled`. What the carve-out buys is that the lane periodically reaches a verdict at all, which is what makes it usable as evidence.
|
||||
|
||||
The decision belongs at workflow level because cancellation applies to the whole superseded run: a job-level `concurrency` group does not exempt its job. The negated form is load-bearing rather than cosmetic: naming `pull_request` alone would also stop cancelling `workflow_dispatch`, and each runner benchmark fans out to twelve larger runners for up to fifteen minutes inside this same group on master, so a re-dispatch would queue ahead of a drill instead of replacing a stale measurement. What bounds the cost is that a master push carries only `wine-apt-cache` and these two drills; every other job is pull-request-gated, `workflow_dispatch`-gated, or `if: false`, and `scripts/ci-workflow.spec.ts` pins that set — classifying by exact condition, since a negated event test mentions the event it excludes — so a new push-reachable job cannot quietly start accumulating uncancelled runs.
|
||||
|
||||
### What the in-house pool is
|
||||
|
||||
`vm-backup`: one 64-core VM, six always-on systemd-managed runner instances. Its image must preinstall Playwright Chromium's Linux system packages; CI downloads the lockfile-selected browser but never runs `apt` on this persistent shared host. Check the latest `serial / linux (self-hosted standby)` run before switching: its aggregate includes browser replay, so a green standby verifies both ordinary capacity and this browser prerequisite.
|
||||
|
||||
@@ -12,6 +12,10 @@ Status: implemented
|
||||
|
||||
三个必需的 Linux 工作作业、独立的原生 Windows 作业,以及 `all checks passed` 判定作业(若不随切换,即使全部工作作业通过,它仍会滞留在故障池的队列中)——各自通过仓库变量 `DSH_CI_FAILOVER` 解析运行器池。变量不存在(正常)时它们运行在托管企业池上;由任何具备写权限的协作者设为 `selfhosted` 时,五个作业全部切换到公司自有的自托管池:Linux 作业与判定作业切到 `vm-backup` 池,覆盖率与快照的并发降到共享虚拟机上限,并跳过托管路径的 pnpm 缓存恢复;原生 Windows 作业切到 `dsh-win-ci` 池。这个开关是写者可管理的仓库状态而非一次合并,因此在所有检查都是红色时仍然有效。自有池的就绪状态由 `serial / linux (self-hosted standby)` 与 `serial / windows (self-hosted standby)` 通道持续验证——每次 master 推送都在其上运行完整的未分片聚合流程。
|
||||
|
||||
`ci.yml` 只豁免一个事件不做取消(`${{ github.event_name != 'push' }}`),因此被取代的拉取请求运行仍会取消,而推送运行则留待执行。每次演练以单门禁工作进程执行完整的未分片聚合流程,耗时长于 master 合并的间隔;在无条件取消下,演练会在得出结论前被后续运行取代,该通道无法产出供响应者查看的就绪证据。这并不保证每次推送运行都能跑完:GitHub 每个组只保留一个待运行条目,更新的待运行条目会顶掉更早的,繁忙时段中间的推送运行仍会以 `cancelled` 结束。这项豁免换来的是该通道**周期性**地得出结论,而这正是它能作为证据的前提。
|
||||
|
||||
这个决定必须放在工作流级:取消作用于被取代的整个运行,作业级 `concurrency` 组并不能豁免其所属作业。采用否定式写法而非仅指名 `pull_request`,是有实质作用的:后者会连 `workflow_dispatch` 一起停止取消,而每次运行器基准测试会在 master 上的同一并发组内同时占用 12 台大规格运行器、最长 15 分钟,届时重复派发会排在演练之前,而不是替换掉已过时的测量。成本之所以可控,是因为一次 master 推送只承载 `wine-apt-cache` 和这两条演练;其余作业都受拉取请求门控、`workflow_dispatch` 门控或 `if: false`,并且 `scripts/ci-workflow.spec.ts` 会锁定这个集合——按条件精确匹配,因为否定式事件判断会包含它所排除的事件名——使新的推送可达作业无法悄悄开始累积未取消的运行。
|
||||
|
||||
### 自有池是什么
|
||||
|
||||
`vm-backup`:一台 64 核虚拟机,6 个常驻 systemd 管理的运行器实例。其镜像必须预装 Playwright Chromium 的 Linux 系统软件包;CI 会下载锁文件选定的浏览器,但绝不在这台持久化共享主机上运行 `apt`。切换前先看 `serial / linux (self-hosted standby)` 最近一次运行:其聚合流程包含浏览器回放,因此绿色热备同时验证常规容量和这项浏览器先决条件。
|
||||
|
||||
Reference in New Issue
Block a user