diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml index 26f7f23f85..7b8d08befe 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md -2026-07-26-ci-failover-runbook.md: ab1a727caa045d2074a9c577416f96f45efcd0aa -2026-07-26-ci-failover-runbook.zh.md: 5dfaca0c1c0f443307bea28bb6544385ebb68bb7 +2026-07-26-ci-failover-runbook.md: 55c1350593562d62463e751451d50a79cf45a1d6 +2026-07-26-ci-failover-runbook.zh.md: 13977b78244440a23722d089849ea7ff6b751aea diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md index ab1a727caa..55c1350593 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md @@ -22,9 +22,13 @@ Each of the three required Linux worker jobs — and the `all checks passed` ver 2. Retrigger the required jobs so they re-resolve their pool. Jobs already **queued** for the hosted labels do not retarget and cannot be re-run in place, so for the documented indefinite-queue outage, cancel the stuck run and re-run all jobs, or push a new commit; "Re-run failed jobs" only helps once a job has actually failed rather than queued. 3. That is the entire switch. Under failover the workflow also, automatically: drops `DSH_COVERAGE_MAX_WORKERS` to 8 and `DSH_SNAPSHOT_MAX_CONCURRENCY` to 12 (sized for six always-on instances: worst case 6 × 8 = 48 coverage workers on the 64-core VM) (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs). -### Capacity during failover +#**Dependabot exception.** All four selectors deliberately exclude `dependabot[bot]`: under failover, Dependabot PRs stay queued for the hosted pool rather than executing dependency-supplied code on the persistent VM. A Dependabot PR that remains queued during an outage is expected behavior, not a failed switch; it completes when the hosted pool recovers. -Six always-on instances absorb normal PR traffic (the pool's steady-state load is one serial standby job per master push, so failover capacity is effectively the full pool). If queues still build, register additional instances with an org registration token (org Settings → Actions → Runners → New runner) — cloning an existing runner directory and running `config.sh` takes about a minute per instance. +**Who can flip the variable.** GitHub's API lets any collaborator with write access manage repository variables, so the switch is writer-level, not strictly admin-only. In this repository's trust model that is not an escalation: the runner group admits all workflows of this private, fork-disabled repository (a deliberate trade to make PR-ref failover possible at all), so any writer could already reach the VM by pushing a branch workflow. The boundary against untrusted code is repository membership; the variable only routes work for members. + +## Capacity during failover + +Six always-on instances absorb normal PR traffic (the pool's steady-state load is one serial standby job per master push, so failover capacity is effectively the full pool). If queues still build, register additional instances with an org registration token (org Settings → Actions → Runners → New runner). Clone an existing runner directory **excluding its identity files** — `rsync -a --exclude '.runner' --exclude '.credentials*' --exclude '_diag' --exclude '_work' / /` — then run `config.sh`; copying `.runner`/`.credentials` verbatim makes `config.sh` refuse with "already configured". About a minute per instance. ### Switch back diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md index 5dfaca0c1c..13977b7824 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md @@ -22,9 +22,13 @@ Status: implemented 2. 重新触发必需作业,使其重新解析运行器池。已经为托管标签**排队**的作业不会重定向,也无法原地 re-run,因此对于本手册所述的无限排队故障,应取消卡住的运行并 re-run all jobs,或推送一个新提交;“Re-run failed jobs”只有在作业真正失败(而非仍在排队)时才有用。 3. 切换到此完成。故障切换状态下工作流还会自动:把 `DSH_COVERAGE_MAX_WORKERS` 降为 8、`DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 12(按 6 个常驻实例定容:最坏 6 × 8 = 48 个覆盖率工作进程对 64 核)(共享虚拟机的争抢上限),并跳过托管路径的 pnpm 缓存恢复(虚拟机的持久 store 直接提供热安装)。 -### 切换期间的容量 +#**Dependabot 例外。**四个选择器都刻意排除了 `dependabot[bot]`:故障切换期间,Dependabot 拉取请求继续在托管池排队,而不是把依赖方提供的代码放到持久化虚拟机上执行。故障期间 Dependabot PR 持续排队是预期行为而非切换失败;托管池恢复后它会自行完成。 -6 个常驻实例可承接正常 PR 流量(该池平时唯一的稳态负载是每次 master 推送一个串行热备作业,故障切换时几乎全池可用)。若仍出现排队,用组织级注册 token(组织 Settings → Actions → Runners → New runner)追加注册实例——复制现有 runner 目录再跑 `config.sh`,每个约一分钟。 +**谁能扳动这个变量。**GitHub 的 API 允许任何具有写权限的协作者管理仓库变量,因此该开关实际是写者级而非严格的管理员级。在本仓库的信任模型下这并不构成越权:runner group 接纳本私有、禁 fork 仓库的全部工作流(这是让 PR 引用的故障切换得以成立的刻意取舍),因此任何写者本就可以通过推送分支工作流触达这台虚拟机。抵御不可信代码的边界是仓库成员资格;变量只是为成员路由工作。 + +## 切换期间的容量 + +6 个常驻实例可承接正常 PR 流量(该池平时唯一的稳态负载是每次 master 推送一个串行热备作业,故障切换时几乎全池可用)。若仍出现排队,用组织级注册 token(组织 Settings → Actions → Runners → New runner)追加注册实例。复制现有 runner 目录时**必须排除身份文件**——`rsync -a --exclude '.runner' --exclude '.credentials*' --exclude '_diag' --exclude '_work' / /`——再跑 `config.sh`;原样拷贝 `.runner`/`.credentials` 会使 `config.sh` 以 "already configured" 拒绝。每个约一分钟。 ### 切回 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 8306b7e034..8c3b854ae6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -423,7 +423,12 @@ jobs: name: serial / linux (self-hosted standby) runs-on: [self-hosted, linux, x64, vm-backup] steps: + # fetch-depth 2 + DSH_ARCHIVE_BASE_REF below: same frozen-archive + # comparison as serial-linux — without the prior commit the archive + # verifier defaults to HEAD and compares the new manifest with itself. - uses: actions/checkout@v6 + with: + fetch-depth: 2 - uses: actions/setup-node@v6 with: @@ -440,6 +445,7 @@ jobs: - name: Run complete unsharded primary Node CI serially env: + DSH_ARCHIVE_BASE_REF: ${{ github.event.before }} DSH_COVERAGE_MAX_WORKERS: '1' DSH_E2E_MAX_WORKERS: '1' DSH_ESLINT_CACHE: '1'