Add RFCs for the remaining quality-proposal ideas
Eight proposals grouped by category, each with problem statement, concrete plan, and risks: property-based testing over the protocol-shaped core (chunk streams, event logs, schema DSL); mutation testing as the counterweight to the 100%-coverage gate; deterministic tests + a universal replay-invariant fixture + nightly race stress; architectural conformance (dependency-cruiser rules and the LlmAdapter conformance kit); runtime arg validation at the model boundary with a structured error taxonomy and dev-mode invariants; doc-sync enforcement (typechecked doc snippets, API reports); supply-chain checks and nightly vendor-drift verification against the manifest; and deep-readonly public surfaces (logged-vs-in-flight mutability boundary). AGENTS.md points at docs/adr and docs/rfc.
This commit is contained in:
40
docs/rfc/003-deterministic-and-stress-testing.md
Normal file
40
docs/rfc/003-deterministic-and-stress-testing.md
Normal file
@@ -0,0 +1,40 @@
|
||||
# RFC 003: Deterministic tests, the replay invariant fixture, and race stress
|
||||
|
||||
Status: proposed
|
||||
|
||||
## Problem
|
||||
|
||||
Several loop tests synchronize with `setTimeout(30)` sleeps — flakiness debt
|
||||
that wastes agent cycles on retries and can mask ordering bugs. Separately,
|
||||
our core architectural promise (any session log replays to identical derived
|
||||
history) is asserted in two tests but is cheap to assert *everywhere*. And
|
||||
the inbox wakeup race was verified by hand exactly once; nothing re-verifies
|
||||
it continuously.
|
||||
|
||||
## Proposal
|
||||
|
||||
Three measures:
|
||||
|
||||
1. **No wall-clock sleeps in tests.** Replace `setTimeout(N)` waits with
|
||||
event-driven waits (the existing `waitForIdle` pattern, extended to
|
||||
`waitForStatus`, `waitForEvent(n)`) or vitest fake timers where time
|
||||
itself is under test. Enforce with a lint rule banning `setTimeout` in
|
||||
`packages/*/tests` outside an allowlisted helper module.
|
||||
2. **Universal replay fixture.** A shared test helper wraps the loop harness
|
||||
so that after every test, the agent's session log is replayed into a fresh
|
||||
Session and `deriveMessages()` equality is asserted automatically. The
|
||||
invariant then gets checked hundreds of times per CI run across every
|
||||
scenario the suite produces, not twice.
|
||||
3. **Nightly race stress.** A CI job running the agent-loop and inbox suites
|
||||
with `vitest --repeat=200` (and `--shuffle`) to flush scheduling-dependent
|
||||
failures; any flake found is a bug to fix, never a retry.
|
||||
|
||||
## Plan
|
||||
|
||||
Land 1 and 2 together (they touch the same helpers); add the nightly job
|
||||
after the suite is sleep-free so repeats are fast.
|
||||
|
||||
## Risks
|
||||
|
||||
Fake timers interact subtly with Promise scheduling in the loop — prefer
|
||||
event-driven waits; reserve fake timers for timer-service behavior itself.
|
||||
Reference in New Issue
Block a user