Files

Chinesezjc 0fd6dc8924 ci: pre-wire admin-only failover from hosted pools to the in-house pool

The three required Linux jobs now resolve their pool through the
DSH_CI_FAILOVER repository variable. Unset, everything runs exactly as
today on the hosted enterprise pools. Setting it to 'selfhosted'
(repo-admin-only, not PR-editable, no merge required — a merge would be
deadlocked behind the failing checks themselves) retargets all three
onto the vm-backup pool, halves the coverage worker bound and snapshot
concurrency for the shared VM, and skips the hosted-path cache restores.

Adds a bilingual failover runbook (switch, capacity via the four
registered spare instances, switch-back, trust boundary) and links it
from the topology note. The push-triggered standby lane remains the
continuous proof that the failover target works.

2026-07-26 00:34:53 +08:00

2.6 KiB

Raw Blame History

Agent Note: CI failover runbook — hosted pools → in-house pool

Status: implemented

English | 中文

What this is

The three required Linux jobs in CI (node 24 / static, node 24 / coverage, node 24 / snapshots and artifacts) resolve their runner pool through the DSH_CI_FAILOVER repository variable. Normally the variable is unset and they run on the hosted enterprise 32-core pools. When the hosted pools are degraded (jobs queue indefinitely, the enterprise labels vanish, or GitHub-side capacity fails), a repository admin can retarget all three onto the in-house self-hosted pool without merging anything — merging would itself be blocked by the very checks that are failing.

The in-house pool (vm-backup: one 64-core VM, four always-on systemd-managed runner instances, four registered spares) is continuously re-proven by the serial / linux (self-hosted standby) lane, which runs the complete unsharded aggregate on every master push. Check its latest run before switching: green standby = verified-yesterday capacity.

Switch (repo admin, ~1 minute, no merge)

Repository Settings → Secrets and variables → Actions → Variables → New repository variable: name DSH_CI_FAILOVER, value selfhosted.
Re-run the failed/queued required jobs (Re-run failed jobs on affected PRs, or let new pushes pick it up).
That is the entire switch. Under failover the workflow also, automatically: halves DSH_COVERAGE_MAX_WORKERS to 12 and DSH_SNAPSHOT_MAX_CONCURRENCY to 16 (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs).

Capacity during failover

Four always-on instances absorb normal PR traffic. If queues build, bring the four registered spares online on the VM (no token needed — they are already registered):

for i in 7 8 9 10; do cd /data_local/actions-runner-$i && sudo ./svc.sh install ubuntu && sudo ./svc.sh start; done

Switch back

Delete the DSH_CI_FAILOVER variable (or set it to anything other than selfhosted). New runs resolve back to the hosted enterprise pools. Stop the spare instances if they were started.

Trust boundary

The variable is repository-admin-only state: a pull request can neither set it nor read a different value into effect, and the expressions live in the base branch's workflow definition. This failover path therefore adds no PR-editable route to the self-hosted pool. (Runner-side enforcement — an org-level runner group restricting these runners to the master-ref workflow — is tracked separately and composes with this mechanism.)

2.6 KiB Raw Blame History