diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml index 0e3b0820c0..a7b9e827fa 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md -2026-07-26-ci-failover-runbook.md: 80dd7c4291e3de11c2f13b3247af56762396c720 -2026-07-26-ci-failover-runbook.zh.md: 7933a857f1559c540fccc2cd89352c4fe351dd7e +2026-07-26-ci-failover-runbook.md: 4e4f8ea7fc60cf76fd8308147bbf7cc0bac74798 +2026-07-26-ci-failover-runbook.zh.md: bb7e43fe55c9cced51f042de6503978ec9349d6b diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md index 80dd7c4291..4e4f8ea7fc 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md @@ -6,7 +6,7 @@ English | [中文](2026-07-26-ci-failover-runbook.zh.md) ## Problem -The three required Linux worker jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) and the required verdict job that aggregates them (`all checks passed`) run on the hosted enterprise 32-core pools. When those pools degrade — jobs queue indefinitely, the enterprise labels vanish, or GitHub-side capacity fails — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. An outage therefore needs a switch any responder with repository write access can throw without merging anything. +The three required Linux worker jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) run on the hosted enterprise 32-core pools; the required verdict job that aggregates them (`all checks passed`) runs on standard `ubuntu-latest`. When the enterprise pools degrade — jobs queue indefinitely or the enterprise labels vanish — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. **Scope: this switch recovers an enterprise Linux-pool outage.** The verdict's other required dependencies (`node-compat`, `python-sdk`, `windows`) stay on standard hosted runners by design (the portable boundary); in a broader GitHub-hosted capacity failure that also takes out the standard pools, those dependencies still block `all checks passed`, and only the Windows leg has no in-house substitute at all — during the 2026-07-27 outage the standard pools recovered first, which is the ordering this design bets on. An outage therefore needs a switch any responder with repository write access can throw without merging anything. ## Decision @@ -28,7 +28,7 @@ Each of the three required Linux worker jobs — and the `all checks passed` ver ## Capacity during failover -Six always-on instances absorb normal PR traffic (the pool's steady-state load is one serial standby job per master push, so failover capacity is effectively the full pool). If queues still build, register additional instances with an org registration token (org Settings → Actions → Runners → New runner). Clone an existing runner directory **excluding its identity files** — `rsync -a --exclude '.runner' --exclude '.credentials*' --exclude '_diag' --exclude '_work' / /` — then run `config.sh` (copying `.runner`/`.credentials` verbatim makes it refuse with "already configured"), and **start the listener**: `sudo ./svc.sh install ubuntu && sudo ./svc.sh start`. Registration alone leaves the runner offline; only a started service adds capacity. About a minute per instance. +Six always-on instances absorb normal PR traffic (the pool's steady-state load is one serial standby job per master push, so failover capacity is effectively the full pool). If queues still build, register additional instances with an org registration token (org Settings → Actions → Runners → New runner). Clone an existing runner directory **excluding its identity files** — `rsync -a --exclude '.runner*' --exclude '.credentials*' --exclude '_diag' --exclude '_work' / /` (the globs also catch `.runner_migrated`/`.credentials_migrated`, which GitHub writes on migrated runners and which equally trigger the already-configured refusal) — then run `config.sh` (copying `.runner`/`.credentials` verbatim makes it refuse with "already configured"), and **start the listener**: `sudo ./svc.sh install ubuntu && sudo ./svc.sh start`. Registration alone leaves the runner offline; only a started service adds capacity. About a minute per instance. ### Switch back diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md index 7933a857f1..bb7e43fe55 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md @@ -6,7 +6,7 @@ Status: implemented ## 问题 -[CI](../../../../.github/workflows/ci.yml) 中三个必需的 Linux 工作作业(`node 24 / static`、`node 24 / coverage`、`node 24 / snapshots and artifacts`)以及聚合它们的必需判定作业(`all checks passed`)运行在托管的企业级 32 核池上。当这些托管池发生故障——作业无限排队、企业标签消失或 GitHub 侧容量故障——所有开启的拉取请求都无法合并,而"合并一个修复"这一常规恢复手段本身正被那些无法运行的必需检查死锁。因此故障需要一个任何具备仓库写权限的响应者都能在不合并任何代码的情况下触发的开关。 +[CI](../../../../.github/workflows/ci.yml) 中三个必需的 Linux 工作作业(`node 24 / static`、`node 24 / coverage`、`node 24 / snapshots and artifacts`)运行在托管的企业级 32 核池上;聚合它们的必需判定作业(`all checks passed`)运行在标准 `ubuntu-latest` 上。当企业池发生故障——作业无限排队或企业标签消失——所有开启的拉取请求都无法合并,而"合并一个修复"这一常规恢复手段本身正被那些无法运行的必需检查死锁。**适用范围:本切换恢复的是企业级 Linux 池故障。**判定作业的其余必需依赖(`node-compat`、`python-sdk`、`windows`)按设计留在标准托管运行器上(可移植边界);若更大范围的 GitHub 托管容量故障连标准池一并击倒,这些依赖仍会阻塞 `all checks passed`,且只有 Windows 这条腿完全没有自有替代——2026-07-27 的故障中标准池率先恢复,本设计押注的正是这一顺序。因此故障需要一个任何具备仓库写权限的响应者都能在不合并任何代码的情况下触发的开关。 ## 决策 @@ -28,7 +28,7 @@ Status: implemented ## 切换期间的容量 -6 个常驻实例可承接正常 PR 流量(该池平时唯一的稳态负载是每次 master 推送一个串行热备作业,故障切换时几乎全池可用)。若仍出现排队,用组织级注册 token(组织 Settings → Actions → Runners → New runner)追加注册实例。复制现有 runner 目录时**必须排除身份文件**——`rsync -a --exclude '.runner' --exclude '.credentials*' --exclude '_diag' --exclude '_work' / /`——再跑 `config.sh`(原样拷贝 `.runner`/`.credentials` 会使其以 "already configured" 拒绝),然后**启动监听器**:`sudo ./svc.sh install ubuntu && sudo ./svc.sh start`。仅注册不会上线;只有启动了服务的 runner 才会增加容量。每个约一分钟。 +6 个常驻实例可承接正常 PR 流量(该池平时唯一的稳态负载是每次 master 推送一个串行热备作业,故障切换时几乎全池可用)。若仍出现排队,用组织级注册 token(组织 Settings → Actions → Runners → New runner)追加注册实例。复制现有 runner 目录时**必须排除身份文件**——`rsync -a --exclude '.runner*' --exclude '.credentials*' --exclude '_diag' --exclude '_work' / /`(通配同时排除 `.runner_migrated`/`.credentials_migrated`——GitHub 会在迁移过的运行器上写入这些文件,它们同样会触发 already-configured 拒绝)——再跑 `config.sh`(原样拷贝 `.runner`/`.credentials` 会使其以 "already configured" 拒绝),然后**启动监听器**:`sudo ./svc.sh install ubuntu && sudo ./svc.sh start`。仅注册不会上线;只有启动了服务的 runner 才会增加容量。每个约一分钟。 ### 切回