ci: standby archive base ref; runbook — dependabot exception, clean bootstrap, variable trust model

- serial-linux-selfhosted now fetches depth 2 and passes
  DSH_ARCHIVE_BASE_REF=github.event.before, running the same
  frozen-archive comparison as serial-linux instead of diffing the
  new manifest against itself.
- Runbook (both languages): documents the deliberate dependabot
  exception (queued-on-hosted during failover is expected, not a
  failed switch); corrects the emergency-capacity bootstrap to
  exclude .runner/.credentials when cloning a runner directory; and
  replaces the 'admin-only' variable claim with the accurate
  trust-model statement — repository variables are writer-manageable,
  which in this private fork-disabled repo with an all-workflows
  runner group is routing among members, not an escalation.
Static gate green locally: 32 passed, 0 failed.
This commit is contained in:
Chinesezjc
2026-07-27 15:47:19 +08:00
parent caabf8f671
commit d606ab6877
4 changed files with 20 additions and 6 deletions

View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md
2026-07-26-ci-failover-runbook.md: ab1a727caa045d2074a9c577416f96f45efcd0aa
2026-07-26-ci-failover-runbook.zh.md: 5dfaca0c1c0f443307bea28bb6544385ebb68bb7
2026-07-26-ci-failover-runbook.md: 55c1350593562d62463e751451d50a79cf45a1d6
2026-07-26-ci-failover-runbook.zh.md: 13977b78244440a23722d089849ea7ff6b751aea

View File

@@ -22,9 +22,13 @@ Each of the three required Linux worker jobs — and the `all checks passed` ver
2. Retrigger the required jobs so they re-resolve their pool. Jobs already **queued** for the hosted labels do not retarget and cannot be re-run in place, so for the documented indefinite-queue outage, cancel the stuck run and re-run all jobs, or push a new commit; "Re-run failed jobs" only helps once a job has actually failed rather than queued.
3. That is the entire switch. Under failover the workflow also, automatically: drops `DSH_COVERAGE_MAX_WORKERS` to 8 and `DSH_SNAPSHOT_MAX_CONCURRENCY` to 12 (sized for six always-on instances: worst case 6 × 8 = 48 coverage workers on the 64-core VM) (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs).
### Capacity during failover
#**Dependabot exception.** All four selectors deliberately exclude `dependabot[bot]`: under failover, Dependabot PRs stay queued for the hosted pool rather than executing dependency-supplied code on the persistent VM. A Dependabot PR that remains queued during an outage is expected behavior, not a failed switch; it completes when the hosted pool recovers.
Six always-on instances absorb normal PR traffic (the pool's steady-state load is one serial standby job per master push, so failover capacity is effectively the full pool). If queues still build, register additional instances with an org registration token (org Settings → Actions → Runners → New runner) — cloning an existing runner directory and running `config.sh` takes about a minute per instance.
**Who can flip the variable.** GitHub's API lets any collaborator with write access manage repository variables, so the switch is writer-level, not strictly admin-only. In this repository's trust model that is not an escalation: the runner group admits all workflows of this private, fork-disabled repository (a deliberate trade to make PR-ref failover possible at all), so any writer could already reach the VM by pushing a branch workflow. The boundary against untrusted code is repository membership; the variable only routes work for members.
## Capacity during failover
Six always-on instances absorb normal PR traffic (the pool's steady-state load is one serial standby job per master push, so failover capacity is effectively the full pool). If queues still build, register additional instances with an org registration token (org Settings → Actions → Runners → New runner). Clone an existing runner directory **excluding its identity files**`rsync -a --exclude '.runner' --exclude '.credentials*' --exclude '_diag' --exclude '_work' <src>/ <dst>/` — then run `config.sh`; copying `.runner`/`.credentials` verbatim makes `config.sh` refuse with "already configured". About a minute per instance.
### Switch back

View File

@@ -22,9 +22,13 @@ Status: implemented
2. 重新触发必需作业,使其重新解析运行器池。已经为托管标签**排队**的作业不会重定向,也无法原地 re-run因此对于本手册所述的无限排队故障应取消卡住的运行并 re-run all jobs或推送一个新提交“Re-run failed jobs”只有在作业真正失败而非仍在排队时才有用。
3. 切换到此完成。故障切换状态下工作流还会自动:把 `DSH_COVERAGE_MAX_WORKERS` 降为 8、`DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 12按 6 个常驻实例定容:最坏 6 × 8 = 48 个覆盖率工作进程对 64 核)(共享虚拟机的争抢上限),并跳过托管路径的 pnpm 缓存恢复(虚拟机的持久 store 直接提供热安装)。
### 切换期间的容量
#**Dependabot 例外。**四个选择器都刻意排除了 `dependabot[bot]`故障切换期间Dependabot 拉取请求继续在托管池排队,而不是把依赖方提供的代码放到持久化虚拟机上执行。故障期间 Dependabot PR 持续排队是预期行为而非切换失败;托管池恢复后它会自行完成。
6 个常驻实例可承接正常 PR 流量(该池平时唯一的稳态负载是每次 master 推送一个串行热备作业,故障切换时几乎全池可用)。若仍出现排队,用组织级注册 token组织 Settings → Actions → Runners → New runner追加注册实例——复制现有 runner 目录再跑 `config.sh`,每个约一分钟
**谁能扳动这个变量。**GitHub 的 API 允许任何具有写权限的协作者管理仓库变量因此该开关实际是写者级而非严格的管理员级。在本仓库的信任模型下这并不构成越权runner group 接纳本私有、禁 fork 仓库的全部工作流(这是让 PR 引用的故障切换得以成立的刻意取舍),因此任何写者本就可以通过推送分支工作流触达这台虚拟机。抵御不可信代码的边界是仓库成员资格;变量只是为成员路由工作
## 切换期间的容量
6 个常驻实例可承接正常 PR 流量(该池平时唯一的稳态负载是每次 master 推送一个串行热备作业,故障切换时几乎全池可用)。若仍出现排队,用组织级注册 token组织 Settings → Actions → Runners → New runner追加注册实例。复制现有 runner 目录时**必须排除身份文件**——`rsync -a --exclude '.runner' --exclude '.credentials*' --exclude '_diag' --exclude '_work' <src>/ <dst>/`——再跑 `config.sh`;原样拷贝 `.runner`/`.credentials` 会使 `config.sh` 以 "already configured" 拒绝。每个约一分钟。
### 切回

View File

@@ -423,7 +423,12 @@ jobs:
name: serial / linux (self-hosted standby)
runs-on: [self-hosted, linux, x64, vm-backup]
steps:
# fetch-depth 2 + DSH_ARCHIVE_BASE_REF below: same frozen-archive
# comparison as serial-linux — without the prior commit the archive
# verifier defaults to HEAD and compares the new manifest with itself.
- uses: actions/checkout@v6
with:
fetch-depth: 2
- uses: actions/setup-node@v6
with:
@@ -440,6 +445,7 @@ jobs:
- name: Run complete unsharded primary Node CI serially
env:
DSH_ARCHIVE_BASE_REF: ${{ github.event.before }}
DSH_COVERAGE_MAX_WORKERS: '1'
DSH_E2E_MAX_WORKERS: '1'
DSH_ESLINT_CACHE: '1'