docs: narrow the push-exemption guarantee, unbreak two static gates

cancel-in-progress is evaluated against the newly triggered run, so
exempting push means one master merge does not cancel the drill still
running from the previous one — not that a drill always finishes. A
benchmark dispatched on master shares the group and does cancel a
mid-flight drill. Record that bound in the runbook and drop the
overstated wording from the workflow comment and the spec name.

Also repair two gates that fail on master and block every pull request:
the telemetry note referenced an SDK proposal deleted in 408721954a,
and the ui-settings-general README pair carried stale recorded hashes
after both sides were updated together in aa1ec02bc6.
This commit is contained in:
Chinesezjc
2026-08-12 18:16:24 +08:00
parent 0ef7f48802
commit 1caca08301
9 changed files with 26 additions and 20 deletions

View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md
2026-07-26-ci-failover-runbook.md: 6f2b4bfba9255549b0887aab9061a6513cb3a68f
2026-07-26-ci-failover-runbook.zh.md: 7bb1a025a71eb690d2d90fcdbddbdd56355786fc
2026-07-26-ci-failover-runbook.md: 90ef2905bec86911697e552c0c1eac96ea0d3a18
2026-07-26-ci-failover-runbook.zh.md: 5c8fd0e90ae31e3622ae6b49217591a54b84c351

View File

@@ -12,7 +12,9 @@ The three required Linux worker jobs in [CI](../../../../.github/workflows/ci.ym
Each of the three required Linux worker jobs, the independent native Windows job, and the `all checks passed` verdict job — which would otherwise stay queued on the failed pool even after every worker passed — resolves its runner pool through the `DSH_CI_FAILOVER` repository variable. Unset (normal), they run on the hosted enterprise pools. Set to `selfhosted` by any repository writer, all five retarget onto the in-house self-hosted pools: the Linux jobs and verdict onto the `vm-backup` pool, coverage and snapshot concurrency drop to shared-VM bounds, and the hosted-path pnpm cache restores are skipped; the native Windows job onto the `dsh-win-ci` pool. The switch is writer-manageable repository state, not a merge, so it works while every check is red. The in-house pools' readiness is continuously re-proven by the `serial / linux (self-hosted standby)` and `serial / windows (self-hosted standby)` lanes, which run the complete unsharded aggregates on every master push.
`ci.yml` exempts exactly one event from `cancel-in-progress` (`${{ github.event_name != 'push' }}`), so a superseded pull-request run is still cancelled while a push run is left to run. Each drill runs its complete unsharded aggregate with one gate worker, which takes longer than the interval between master merges; under unconditional cancellation a drill is superseded before reaching a verdict and the lane yields no readiness evidence for a responder to check. This is not a guarantee that every push run finishes: GitHub keeps a single pending entry per group, so a newer pending run displaces an older one, and during busy periods intermediate push runs still end as `cancelled`. What the carve-out buys is that the lane periodically reaches a verdict at all, which is what makes it usable as evidence.
`ci.yml` exempts exactly one event from `cancel-in-progress` (`${{ github.event_name != 'push' }}`), so one master push does not cancel the drill still running from the previous one. Each drill runs its complete unsharded aggregate with one gate worker, which takes longer than the interval between master merges; under unconditional cancellation a drill is superseded before reaching a verdict and the lane yields no readiness evidence for a responder to check.
The exemption is narrower than "a drill always finishes", in two ways. GitHub keeps a single pending entry per group, so a newer pending run displaces an older one and intermediate push runs still end as `cancelled` during busy periods. And the expression is evaluated against the *newly triggered* run, so a run whose own event is not `push` — a benchmark dispatched on master, sharing the group `CI-<ref>` — evaluates to `true` and does cancel a drill that is mid-flight. That is a rare manual action and the next master push restores the evidence, so it does not warrant further mechanism. What the carve-out buys is that the lane periodically reaches a verdict at all, which is what makes it usable as evidence.
The decision belongs at workflow level because cancellation applies to the whole superseded run: a job-level `concurrency` group does not exempt its job. The negated form is load-bearing rather than cosmetic: naming `pull_request` alone would also stop cancelling `workflow_dispatch`, and each runner benchmark fans out to twelve larger runners for up to fifteen minutes inside this same group on master, so a re-dispatch would queue ahead of a drill instead of replacing a stale measurement. What bounds the cost is that a master push carries only `wine-apt-cache` and these two drills; every other job is pull-request-gated, `workflow_dispatch`-gated, or `if: false`, and `scripts/ci-workflow.spec.ts` pins that set — classifying by exact condition, since a negated event test mentions the event it excludes — so a new push-reachable job cannot quietly start accumulating uncancelled runs.

View File

@@ -12,7 +12,9 @@ Status: implemented
三个必需的 Linux 工作作业、独立的原生 Windows 作业,以及 `all checks passed` 判定作业(若不随切换,即使全部工作作业通过,它仍会滞留在故障池的队列中)——各自通过仓库变量 `DSH_CI_FAILOVER` 解析运行器池。变量不存在(正常)时它们运行在托管企业池上;由任何具备写权限的协作者设为 `selfhosted`五个作业全部切换到公司自有的自托管池Linux 作业与判定作业切到 `vm-backup` 池,覆盖率与快照的并发降到共享虚拟机上限,并跳过托管路径的 pnpm 缓存恢复;原生 Windows 作业切到 `dsh-win-ci` 池。这个开关是写者可管理的仓库状态而非一次合并,因此在所有检查都是红色时仍然有效。自有池的就绪状态由 `serial / linux (self-hosted standby)``serial / windows (self-hosted standby)` 通道持续验证——每次 master 推送都在其上运行完整的未分片聚合流程。
`ci.yml` 只豁免一个事件不做取消(`${{ github.event_name != 'push' }}`),因此被取代的拉取请求运行仍会取消,而推送运行则留待执行。每次演练以单门禁工作进程执行完整的未分片聚合流程,耗时长于 master 合并的间隔;在无条件取消下,演练会在得出结论前被后续运行取代,该通道无法产出供响应者查看的就绪证据。这并不保证每次推送运行都能跑完GitHub 每个组只保留一个待运行条目,更新的待运行条目会顶掉更早的,繁忙时段中间的推送运行仍会以 `cancelled` 结束。这项豁免换来的是该通道**周期性**地得出结论,而这正是它能作为证据的前提。
`ci.yml` 只豁免一个事件不做取消(`${{ github.event_name != 'push' }}`),因此一次 master 推送不会取消上一次推送留下的、仍在运行的演练。每次演练以单门禁工作进程执行完整的未分片聚合流程,耗时长于 master 合并的间隔;在无条件取消下,演练会在得出结论前被后续运行取代,该通道无法产出供响应者查看的就绪证据。
这项豁免比「演练总能跑完」要窄有两点限制。其一GitHub 每个组只保留一个待运行条目,更新的待运行条目会顶掉更早的,繁忙时段中间的推送运行仍会以 `cancelled` 结束。其二,该表达式是针对**新触发的运行**求值的,因此自身事件不是 `push` 的运行——例如在 master 上派发的基准测试,与演练共用 `CI-<ref>` 组——求值为 `true`,会取消正在运行中的演练。这属于罕见的手动操作,且下一次 master 推送即可恢复证据,因此不值得为它再加机制。这项豁免换来的是该通道**周期性**地得出结论,而这正是它能作为证据的前提。
这个决定必须放在工作流级:取消作用于被取代的整个运行,作业级 `concurrency` 组并不能豁免其所属作业。采用否定式写法而非仅指名 `pull_request`,是有实质作用的:后者会连 `workflow_dispatch` 一起停止取消,而每次运行器基准测试会在 master 上的同一并发组内同时占用 12 台大规格运行器、最长 15 分钟,届时重复派发会排在演练之前,而不是替换掉已过时的测量。成本之所以可控,是因为一次 master 推送只承载 `wine-apt-cache` 和这两条演练;其余作业都受拉取请求门控、`workflow_dispatch` 门控或 `if: false`,并且 `scripts/ci-workflow.spec.ts` 会锁定这个集合——按条件精确匹配,因为否定式事件判断会包含它所排除的事件名——使新的推送可达作业无法悄悄开始累积未取消的运行。