ebb5bf4c19
config.sh registers the instance without starting a listener, so the procedure as written left the new runner offline and added no capacity.
6.5 KiB
Agent Note: CI failover runbook — hosted pools → in-house pool
Status: implemented
English | 中文
Problem
The three required Linux worker jobs in CI (node 24 / static, node 24 / coverage, node 24 / snapshots and artifacts) and the required verdict job that aggregates them (all checks passed) run on the hosted enterprise 32-core pools. When those pools degrade — jobs queue indefinitely, the enterprise labels vanish, or GitHub-side capacity fails — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. An outage therefore needs a switch a repository admin can throw without merging anything.
Decision
Each of the three required Linux worker jobs — and the all checks passed verdict job, which would otherwise stay queued on the failed pool even after every worker passed — resolves its runner pool through the DSH_CI_FAILOVER repository variable. Unset (normal), they run on the hosted enterprise pools. Set to selfhosted by any repository writer, all four retarget onto the in-house self-hosted vm-backup pool, coverage and snapshot concurrency drop to shared-VM bounds, and the hosted-path pnpm cache restores are skipped. The switch is writer-manageable repository state, not a merge, so it works while every check is red. The in-house pool's readiness is continuously re-proven by the serial / linux (self-hosted standby) lane, which runs the complete unsharded aggregate on every master push.
What the in-house pool is
vm-backup: one 64-core VM, six always-on systemd-managed runner instances. Check the latest serial / linux (self-hosted standby) run before switching: a green standby is verified-yesterday capacity.
Switch (repo admin, ~1 minute, no merge)
- Repository Settings → Secrets and variables → Actions → Variables → New repository variable: name
DSH_CI_FAILOVER, valueselfhosted. - Retrigger the required jobs so they re-resolve their pool. Jobs already queued for the hosted labels do not retarget and cannot be re-run in place, so for the documented indefinite-queue outage, cancel the stuck run and re-run all jobs, or push a new commit; "Re-run failed jobs" only helps once a job has actually failed rather than queued.
- That is the entire switch. Under failover the workflow also, automatically: drops
DSH_COVERAGE_MAX_WORKERSto 8 andDSH_SNAPSHOT_MAX_CONCURRENCYto 12 (sized for six always-on instances: worst case 6 × 8 = 48 coverage workers on the 64-core VM) (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs).
#Dependabot exception. All four selectors deliberately exclude dependabot[bot]: under failover, Dependabot PRs stay queued for the hosted pool rather than executing dependency-supplied code on the persistent VM. A Dependabot PR that remains queued during an outage is expected behavior, not a failed switch; it completes when the hosted pool recovers.
Who can flip the variable. GitHub's API lets any collaborator with write access manage repository variables, so the switch is writer-level, not strictly admin-only. In this repository's trust model that is not an escalation: the runner group admits all workflows of this private, fork-disabled repository (a deliberate trade to make PR-ref failover possible at all), so any writer could already reach the VM by pushing a branch workflow. The boundary against untrusted code is repository membership; the variable only routes work for members.
Capacity during failover
Six always-on instances absorb normal PR traffic (the pool's steady-state load is one serial standby job per master push, so failover capacity is effectively the full pool). If queues still build, register additional instances with an org registration token (org Settings → Actions → Runners → New runner). Clone an existing runner directory excluding its identity files — rsync -a --exclude '.runner' --exclude '.credentials*' --exclude '_diag' --exclude '_work' <src>/ <dst>/ — then run config.sh; copying .runner/.credentials verbatim makes config.sh refuse with "already configured". config.sh only registers the instance — it starts no listener, so a runner that stops there is registered and offline, adding no capacity. Install and start its service too: sudo ./svc.sh install <user> && sudo ./svc.sh start (this pool is systemd-managed; a foreground ./run.sh also works but dies with the shell). Confirm the instance reports Idle in org Settings → Actions → Runners before counting it. About a minute per instance.
Switch back
Delete the DSH_CI_FAILOVER variable (or set it to anything other than selfhosted). New runs resolve back to the hosted enterprise pools. Remove any extra instances that were registered during the incident.
Trust boundary
The variable is writer-manageable repository state; a pull request event itself can neither set it nor read a different value into effect, and the selector expressions live in workflow definitions. Note that under failover, pull_request runs execute the PR merge ref's own workflow definition — the boundary against untrusted code is repository membership (private, forking disabled, Dependabot excluded by the selectors), not the variable. Runner-side enforcement — an org-level runner group restricting these runners to the master-ref workflow — is tracked separately and composes with this mechanism.
Alternatives considered
Merge a workflow change to switch pools. Rejected because the outage that motivates the switch is exactly the state in which no PR can merge: the required checks are the ones failing. A repository variable is admin-controlled state that takes effect on re-run without a merge.
Keep the self-hosted pool always in the required path. Rejected because it trades hosted-pool availability for the in-house VM's, moving a single point of failure rather than adding a fallback. The variable keeps the hosted pools primary and the self-hosted pool a proven, one-action standby.
Consequences
Recovering from a hosted-pool outage is a single admin variable plus a re-run, with no merge on the critical path. The cost is a second runner topology to keep working: the standby lane exercises it on every master push so the failover target never goes stale, and the concurrency and cache-restore branches in ci.yml carry a selfhosted leg that must stay in step with the hosted leg.