Rename RFCs to Agent Notes
This commit is contained in:
@@ -0,0 +1,31 @@
|
||||
# Agent Note: Deterministic tests, the replay invariant fixture, and race stress
|
||||
|
||||
Status: proposed
|
||||
|
||||
## Problem
|
||||
|
||||
Several loop tests synchronize with `setTimeout(30)` sleeps — flakiness debt that wastes agent cycles on retries and can mask ordering bugs. Separately, our core architectural promise (any session log replays to identical derived history) is asserted in two tests but is cheap to assert *everywhere*. And the inbox wakeup race was verified by hand exactly once; nothing re-verifies it continuously.
|
||||
|
||||
## Proposal
|
||||
|
||||
Three measures:
|
||||
|
||||
1. **No wall-clock sleeps in tests.** Replace `setTimeout(N)` waits with event-driven waits (the existing `waitForIdle` pattern, extended to `waitForStatus`, `waitForEvent(n)`) or vitest fake timers where time itself is under test. Enforce with a lint rule banning `setTimeout` in `packages/*/tests` outside an allowlisted helper module.
|
||||
2. **Universal replay fixture.** A shared test helper wraps the loop harness so that after every test, the agent's session log is replayed into a fresh Session and `deriveMessages()` equality is asserted automatically. The invariant then gets checked hundreds of times per CI run across every scenario the suite produces, not twice.
|
||||
3. **Nightly race stress.** A CI job running the agent-loop and inbox suites with `vitest --repeat=200` (and `--shuffle`) to flush scheduling-dependent failures; any flake found is a bug to fix, never a retry.
|
||||
|
||||
## Plan
|
||||
|
||||
Land 1 and 2 together (they touch the same helpers); add the nightly job after the suite is sleep-free so repeats are fast.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- No `setTimeout` remains in `packages/*/tests` outside the allowlisted helper module, enforced by the lint rule.
|
||||
- The shared harness replays every test's session log into a fresh `Session` and asserts `deriveMessages()` equality automatically, across the whole suite.
|
||||
- The nightly job runs the agent-loop and inbox suites with `--repeat` and `--shuffle`; a flake it finds is triaged as a bug, never retried away.
|
||||
|
||||
## Risks
|
||||
|
||||
Fake timers interact subtly with Promise scheduling in the loop — prefer event-driven waits; reserve fake timers for timer-service behavior itself.
|
||||
|
||||
<!-- agent-note-format: alternatives-not-recorded (pre-format Agent Note) -->
|
||||
@@ -0,0 +1,34 @@
|
||||
# Agent Note: Mutation testing as the coverage counterweight
|
||||
|
||||
Status: proposed
|
||||
|
||||
## Problem
|
||||
|
||||
The per-file 100% coverage gate ([the quality-gates decision](../../implemented/process/2026-06-11-quality-gates.md)) proves every line *executes* under test — not that any assertion would notice if the line were wrong. Under agent-written tests, coverage pressure can produce execution-without-assertion. Mutation testing measures what coverage cannot: whether the suite *kills* deliberately injected bugs.
|
||||
|
||||
## Proposal
|
||||
|
||||
Stryker (`@stryker-mutator/vitest-runner`) over `packages/*/src`:
|
||||
|
||||
- **PR-scoped incremental runs** (changed files only) as a CI job — fast enough to gate merges once tuned.
|
||||
- **Nightly full runs** with a tracked mutation score; start by recording, then set the threshold at the observed baseline and ratchet upward (same policy as coverage: thresholds only ever tighten).
|
||||
- Surviving mutants are work items: an agent picks a survivor, writes the killing test, repeats — a well-shaped autonomous loop.
|
||||
- Equivalent mutants (provably behavior-preserving) get annotated exclusions with reasons, mirroring the `/* v8 ignore */` policy.
|
||||
|
||||
## Plan
|
||||
|
||||
1. Add Stryker config scoped to one package (llm — smallest, most algorithmic) and measure runtime.
|
||||
2. Expand to all packages; record baseline scores in the config.
|
||||
3. Wire the nightly job; add the incremental PR job once runtime is acceptable.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- A Stryker config runs over `packages/*/src` with the vitest runner; a nightly job records the mutation score, and a ratcheting threshold fails the run when the score drops below the recorded baseline.
|
||||
- PR-scoped incremental runs gate merges once runtime is acceptable — or are explicitly kept nightly-only, with that outcome recorded here.
|
||||
- Equivalent mutants carry annotated exclusions with reasons, mirroring the `/* v8 ignore */` policy.
|
||||
|
||||
## Risks
|
||||
|
||||
Runtime: mutation testing is expensive; per-file 100% coverage helps (every mutant is at least reached). If PR-scoped runs stay too slow, keep them nightly-only and rely on the score ratchet.
|
||||
|
||||
<!-- agent-note-format: alternatives-not-recorded (pre-format Agent Note) -->
|
||||
Reference in New Issue
Block a user