0fd6dc8924
The three required Linux jobs now resolve their pool through the DSH_CI_FAILOVER repository variable. Unset, everything runs exactly as today on the hosted enterprise pools. Setting it to 'selfhosted' (repo-admin-only, not PR-editable, no merge required — a merge would be deadlocked behind the failing checks themselves) retargets all three onto the vm-backup pool, halves the coverage worker bound and snapshot concurrency for the shared VM, and skips the hosted-path cache restores. Adds a bilingual failover runbook (switch, capacity via the four registered spare instances, switch-back, trust boundary) and links it from the topology note. The push-triggered standby lane remains the continuous proof that the failover target works.
34 lines
2.6 KiB
Markdown
34 lines
2.6 KiB
Markdown
# Agent Note: CI failover runbook — hosted pools → in-house pool
|
|
|
|
Status: implemented
|
|
|
|
English | [中文](ci-failover-runbook.zh.md)
|
|
|
|
## What this is
|
|
|
|
The three required Linux jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) resolve their runner pool through the `DSH_CI_FAILOVER` repository variable. Normally the variable is unset and they run on the hosted enterprise 32-core pools. When the hosted pools are degraded (jobs queue indefinitely, the enterprise labels vanish, or GitHub-side capacity fails), a repository admin can retarget all three onto the in-house self-hosted pool without merging anything — merging would itself be blocked by the very checks that are failing.
|
|
|
|
The in-house pool (`vm-backup`: one 64-core VM, four always-on systemd-managed runner instances, four registered spares) is continuously re-proven by the `serial / linux (self-hosted standby)` lane, which runs the complete unsharded aggregate on every master push. Check its latest run before switching: green standby = verified-yesterday capacity.
|
|
|
|
## Switch (repo admin, ~1 minute, no merge)
|
|
|
|
1. Repository **Settings → Secrets and variables → Actions → Variables → New repository variable**: name `DSH_CI_FAILOVER`, value `selfhosted`.
|
|
2. Re-run the failed/queued required jobs (Re-run failed jobs on affected PRs, or let new pushes pick it up).
|
|
3. That is the entire switch. Under failover the workflow also, automatically: halves `DSH_COVERAGE_MAX_WORKERS` to 12 and `DSH_SNAPSHOT_MAX_CONCURRENCY` to 16 (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs).
|
|
|
|
## Capacity during failover
|
|
|
|
Four always-on instances absorb normal PR traffic. If queues build, bring the four registered spares online on the VM (no token needed — they are already registered):
|
|
|
|
```bash
|
|
for i in 7 8 9 10; do cd /data_local/actions-runner-$i && sudo ./svc.sh install ubuntu && sudo ./svc.sh start; done
|
|
```
|
|
|
|
## Switch back
|
|
|
|
Delete the `DSH_CI_FAILOVER` variable (or set it to anything other than `selfhosted`). New runs resolve back to the hosted enterprise pools. Stop the spare instances if they were started.
|
|
|
|
## Trust boundary
|
|
|
|
The variable is repository-admin-only state: a pull request can neither set it nor read a different value into effect, and the expressions live in the base branch's workflow definition. This failover path therefore adds no PR-editable route to the self-hosted pool. (Runner-side enforcement — an org-level runner group restricting these runners to the master-ref workflow — is tracked separately and composes with this mechanism.)
|