test(web): pin every scenario end-state with an aria golden

Every spec now commits at least one golden and the interactive ones one
per distinct end-state (nine new .expected.md):

- live-interactions: cancel.expected.md (frozen partial + 已停止 marker),
  error-auth.expected.md (the prompt bubble alone — the committed artifact
  of the web-error-surface gap, the diff that flips when error rendering
  lands), retry.expected.md (indistinguishable from a clean completion —
  retries are deliberately invisible in the transcript).
- question-composer: answered.expected.md (the question resolved into its
  tool round trip plus the final reply, takeover gone) beside the existing
  waiting-state golden.
- steering: mid-steer.expected.md pins the accepted-but-INVISIBLE state
  (the loop drains steering only at the step boundary, so no interjection
  bubble exists while the question still blocks — if the client ever
  renders pending steers eagerly, this golden flips first) and
  settled.expected.md the badged bubble plus obeying reply.
- navigation-panes: waterfall.expected.md and details-open.expected.md
  (tool-name header, Input args, Output result) beside the trajectory one.
- lifecycle-chrome: reloaded.expected.md — rendering the same settled
  transcript from persistence alone IS the recovery claim.

Fixture inventories extended to the new closed sets; the Agent Note's
expected-outputs policy updated in both languages (per-end-state goldens
for interactive scenarios), pairing re-recorded.
This commit is contained in:
Tianyi Cui
2026-07-26 12:45:44 +08:00
parent 113f818084
commit ea1d8d06b3
17 changed files with 304 additions and 25 deletions

View File

@@ -20,13 +20,20 @@ import { deriveReplayScript, parseSessionLog } from '@deepseek-ai/dsh-llm-replay
import type { ReplayOverrideDoc } from '@deepseek-ai/dsh-llm-replay'
import type { SessionEvent } from '@deepseek-ai/dsh-session'
import {
assertFixtureInventory, fixtureUserPrompts, launchWebScaffold, recordFixture,
watchConsole, webSnapshotMode, type WebScaffold,
assertFixtureInventory, captureStableAria, compareOrRefreshGolden, fixtureUserPrompts,
launchWebScaffold, recordFixture, watchConsole, webSnapshotMode, type WebScaffold,
} from './scaffold.ts'
import { saveFailureShot } from './support.ts'
const SNAPSHOT_DIR = fileURLToPath(new URL('./snapshots/live-interactions', import.meta.url))
const FIXTURE = join(SNAPSHOT_DIR, 'session.jsonl')
// One golden per interactive end-state: what the user is left looking at
// after cancel, after a non-retryable failure (pins the FIXME(web-error-surface)
// gap as a reviewable artifact: NO error copy in the tree), and after retry
// recovery — three genuinely different terminal surfaces of one fixture.
const CANCEL_EXPECTED = join(SNAPSHOT_DIR, 'cancel.expected.md')
const ERROR_EXPECTED = join(SNAPSHOT_DIR, 'error-auth.expected.md')
const RETRY_EXPECTED = join(SNAPSHOT_DIR, 'retry.expected.md')
const MODE = webSnapshotMode()
// The recorded base: one text-only turn whose derived script the sidecars
@@ -123,6 +130,10 @@ describe('web e2e: live-turn interactions (cancel / error / retry)', () => {
// Composer recovered; no streaming node lingers.
await expect.poll(() => page.locator('textarea').first().isEnabled(), { timeout: 10_000 }).toBe(true)
expect(await page.locator('[data-streaming="true"]').count()).toBe(0)
// Golden of the aborted end-state: the prompt bubble plus the frozen
// partial ('partial' is the hang entry's replayed prefix) and no more.
const snapshot = await captureStableAria(page, '[class*="centerCol"]', scaffold!.workspaceCwd)
await compareOrRefreshGolden(CANCEL_EXPECTED, snapshot, MODE)
expect(tripwire.pageErrors).toEqual([])
}, 120_000)
@@ -144,6 +155,10 @@ describe('web e2e: live-turn interactions (cancel / error / retry)', () => {
// "no crash, composer recovers, turn logged as error".
await expect.poll(() => page.locator('textarea').first().isEnabled(), { timeout: 10_000 }).toBe(true)
expect(await page.locator('[data-streaming="true"]').count()).toBe(0)
// Golden of the same gap: the prompt bubble alone, no error copy in the
// tree — the diff that changes when web-error-surface lands.
const snapshot = await captureStableAria(page, '[class*="centerCol"]', scaffold!.workspaceCwd)
await compareOrRefreshGolden(ERROR_EXPECTED, snapshot, MODE)
expect(tripwire.pageErrors).toEqual([])
}, 120_000)
@@ -167,10 +182,16 @@ describe('web e2e: live-turn interactions (cancel / error / retry)', () => {
// only on change, so attempt count is invisible there).
expect(sessionEvents.filter(e => e.type === 'llm/retry').length).toBeGreaterThanOrEqual(1)
await expect.poll(() => page.getByText('event sourcing', { exact: false }).count(), { timeout: 10_000 }).toBeGreaterThan(0)
// Golden of the recovered end-state: indistinguishable from a clean
// completion — retries are deliberately invisible in the transcript.
const snapshot = await captureStableAria(page, '[class*="centerCol"]', scaffold!.workspaceCwd)
await compareOrRefreshGolden(RETRY_EXPECTED, snapshot, MODE)
expect(tripwire.pageErrors).toEqual([])
}, 120_000)
it.skipIf(MODE === 'record')('keeps the fixture inventory closed', async () => {
await assertFixtureInventory(SNAPSHOT_DIR, ['session.jsonl'])
await assertFixtureInventory(SNAPSHOT_DIR, [
'session.jsonl', 'cancel.expected.md', 'error-auth.expected.md', 'retry.expected.md',
])
})
})