test(web): stabilize and refresh the aria goldens for the speed readings
The TTFT and tok/s readings divide by measured wall time, so they are not
reproducible: the same replayed scenario yielded 69 and 70 tok/s on
consecutive local runs, and a 3 ms replayed stream reads 26333 tok/s. Baking
those into committed goldens made the Web lane flaky by construction, and the
goldens for the readings themselves were never refreshed.
Three fixes, then a refresh:
The footer's decorative dots are `aria-hidden`, so the readings concatenated
into one accessible string — `Ran for 13sTTFT 0.2s12 tok/s`. That is a real
defect on its own (a reader hears one run-on instead of three facts) and it
also denied `{{duration}}` the word boundary it matches on, so even the
previously-stable `Ran for` duration started leaking raw. The separators now
carry flanking spaces.
`normalizeAria` gains `{{throughput}}` beside `{{duration}}`, and its duration
alternation accepts the stats line's compact `2m42s` as well as the
message-chrome template's `2m 42s` — the compact form had no pattern at all,
which is why `LLM 382m39s` survived the first refresh.
Refreshed 17 goldens. They also record that the stats line's `LLM` group now
renders at all: it folds assistant `timing`, which the live transcript adapter
only began attaching in this branch, so the group was previously dead in Chat.
Verified by running the lane in replay three times after the refresh: 41/41
files green each time, goldens untouched. Before this change two consecutive
runs disagreed on both the values and the failure count.
This commit is contained in:
@@ -523,8 +523,14 @@ export async function seedSession(scaffold: WebScaffold, fixtureText: string, id
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize an aria snapshot: uuid, cwd, workspace-basename, and duration
|
||||
* volatility collapse to stable tokens.
|
||||
* Normalize an aria snapshot: uuid, cwd, workspace-basename, duration, and
|
||||
* decode-throughput volatility collapse to stable tokens.
|
||||
*
|
||||
* Throughput needs a token for the same reason durations do, and no fixture
|
||||
* can supply one: the figure divides a replayed step's output tokens by the
|
||||
* wall time the local run took to stream them, so it moves between two runs
|
||||
* on one machine (measured 69 → 70 tok/s) and swings wildly on a fast replay
|
||||
* (26333 tok/s for a 3 ms stream).
|
||||
*/
|
||||
function normalizeAria(snapshot: string, workspaceCwd: string): string {
|
||||
// The session heading renders the workspace's basename, not the full
|
||||
@@ -534,14 +540,17 @@ function normalizeAria(snapshot: string, workspaceCwd: string): string {
|
||||
.split(workspaceCwd).join('{{cwd}}')
|
||||
.split(base).join('{{workspace}}')
|
||||
.replace(/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi, '{{uuid}}')
|
||||
// The optional space in `\d+m ?\d+s` covers both minute spellings: the
|
||||
// stats line's compact `2m42s` and the message-chrome template's `2m 42s`.
|
||||
.replace(
|
||||
/~\d+(?:y(?: \d+mo)?|mo(?: \d+d)?)|\b(?:\d+d(?: \d+h(?: \d+m \d+s)?)?|\d+h \d+m \d+s|\d+m \d+s|\d+(?:\.\d+)?s|\d+(?:\.\d+)?ms)\b/g,
|
||||
/~\d+(?:y(?: \d+mo)?|mo(?: \d+d)?)|\b(?:\d+d(?: \d+h(?: \d+m \d+s)?)?|\d+h \d+m \d+s|\d+m ?\d+s|\d+(?:\.\d+)?s|\d+(?:\.\d+)?ms)\b/g,
|
||||
duration => duration.startsWith('~') ? duration : '{{duration}}',
|
||||
)
|
||||
.replace(
|
||||
/约\d+(?:年(?:\d+个月)?|个月(?:\d+天)?)|\d+(?:天(?:\d+小时(?:\d+分\d+秒)?)?|小时\d+分\d+秒|分\d+秒|(?:\.\d+)?秒)/g,
|
||||
duration => duration.startsWith('约') ? duration : '{{duration}}',
|
||||
)
|
||||
.replace(/\d+(?:\.\d+)?(?= tok\/s(?!\w))/g, '{{throughput}}')
|
||||
// Message IconActions clocks widen by calendar day/year; collapse every
|
||||
// shape so goldens stay stable across midnight and year boundaries.
|
||||
.replace(/\d{4}年\d{1,2}月\d{1,2}日 \d{2}:\d{2}/g, '{{clock}}')
|
||||
|
||||
@@ -30,4 +30,4 @@
|
||||
- text: Select model
|
||||
- img
|
||||
- button "Send message" [disabled]
|
||||
- text: 1 turns · 1 steps Tool call {{duration}} Cache hit 0% Input 10 tok · Output 10 tok
|
||||
- text: 1 turns · 1 steps LLM {{duration}} · Tool call {{duration}} TTFT avg {{duration}} · {{throughput}} tok/s Cache hit 0% Input 10 tok · Output 10 tok
|
||||
|
||||
@@ -36,7 +36,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: {{clock}}Ran for {{duration}}
|
||||
- text: {{clock}} Ran for {{duration}} TTFT {{duration}} {{throughput}} tok/s
|
||||
- textbox "Message the agent"
|
||||
- button "Commands":
|
||||
- img
|
||||
@@ -44,5 +44,6 @@
|
||||
- button "Select model, current DeepSeek-V4-Flash":
|
||||
- text: DeepSeek-V4-Flash
|
||||
- img
|
||||
- button "7% of context used"
|
||||
- button "Send message" [disabled]
|
||||
- text: 1 turns · 2 steps Tool call {{duration}} Context 7% of 128K Cache hit 52% Input 17.2K tok · Output 252 tok
|
||||
- text: 1 turns · 2 steps LLM {{duration}} · Tool call {{duration}} TTFT avg {{duration}} · {{throughput}} tok/s Cache hit 52% Input 17.2K tok · Output 252 tok
|
||||
|
||||
@@ -51,7 +51,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: {{clock}}Ran for {{duration}}
|
||||
- text: {{clock}} Ran for {{duration}} TTFT {{duration}} {{throughput}} tok/s
|
||||
- textbox "Message the agent"
|
||||
- button "Commands":
|
||||
- img
|
||||
@@ -59,5 +59,6 @@
|
||||
- button "Select model, current DeepSeek-V4-Flash":
|
||||
- text: DeepSeek-V4-Flash
|
||||
- img
|
||||
- button "13% of context used"
|
||||
- button "Send message" [disabled]
|
||||
- text: 1 turns · 4 steps Tool call {{duration}} Context 13% of 128K Cache hit 77% Input 66.5K tok · Output 312 tok
|
||||
- text: 1 turns · 4 steps LLM {{duration}} · Tool call {{duration}} TTFT avg {{duration}} · {{throughput}} tok/s Cache hit 77% Input 66.5K tok · Output 312 tok
|
||||
|
||||
@@ -31,7 +31,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: {{clock}}Ran for {{duration}}
|
||||
- text: {{clock}} Ran for {{duration}} TTFT {{duration}} {{throughput}} tok/s
|
||||
- textbox "Message the agent"
|
||||
- button "Commands":
|
||||
- img
|
||||
@@ -39,5 +39,6 @@
|
||||
- button "Select model, current DeepSeek-V4-Flash":
|
||||
- text: DeepSeek-V4-Flash
|
||||
- img
|
||||
- button "6% of context used"
|
||||
- button "Send message" [disabled]
|
||||
- text: 1 turns · 2 steps Tool call {{duration}} Context 6% of 128K Cache hit 99% Input 15.7K tok · Output 111 tok
|
||||
- text: 1 turns · 2 steps LLM {{duration}} · Tool call {{duration}} TTFT avg {{duration}} · {{throughput}} tok/s Cache hit 99% Input 15.7K tok · Output 111 tok
|
||||
|
||||
@@ -23,7 +23,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: {{clock}}Ran for {{duration}}
|
||||
- text: {{clock}} Ran for {{duration}} TTFT {{duration}} {{throughput}} tok/s
|
||||
- textbox "Message the agent"
|
||||
- button "Commands":
|
||||
- img
|
||||
@@ -31,5 +31,6 @@
|
||||
- button "Select model, current DeepSeek-V4-Flash":
|
||||
- text: DeepSeek-V4-Flash
|
||||
- img
|
||||
- button "6% of context used"
|
||||
- button "Send message" [disabled]
|
||||
- text: 1 turns · 1 steps Context 6% of 128K Cache hit 99% Input 7.8K tok · Output 21 tok
|
||||
- text: 1 turns · 1 steps LLM {{duration}} TTFT avg {{duration}} · {{throughput}} tok/s Cache hit 99% Input 7.8K tok · Output 21 tok
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: {{clock}}Ran for {{duration}}
|
||||
- text: {{clock}} Ran for {{duration}}
|
||||
- textbox "Message the agent"
|
||||
- button "Commands":
|
||||
- img
|
||||
|
||||
@@ -25,7 +25,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: {{clock}}Ran for {{duration}}
|
||||
- text: {{clock}} Ran for {{duration}} TTFT {{duration}} {{throughput}} tok/s
|
||||
- textbox "Message the agent"
|
||||
- button "Commands":
|
||||
- img
|
||||
@@ -33,5 +33,6 @@
|
||||
- button "Select model, current DeepSeek-V4-Flash":
|
||||
- text: DeepSeek-V4-Flash
|
||||
- img
|
||||
- button "6% of context used"
|
||||
- button "Send message" [disabled]
|
||||
- text: 1 turns · 1 steps Context 6% of 128K Cache hit 99% Input 7.8K tok · Output 79 tok
|
||||
- text: 1 turns · 1 steps LLM {{duration}} TTFT avg {{duration}} · {{throughput}} tok/s Cache hit 99% Input 7.8K tok · Output 79 tok
|
||||
|
||||
@@ -19,7 +19,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: {{clock}}Ran for {{duration}}
|
||||
- text: {{clock}} Ran for {{duration}}
|
||||
- textbox "Message the agent"
|
||||
- button "Commands":
|
||||
- img
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation" [disabled]:
|
||||
- img
|
||||
- text: Available only on the last message of a completed turn 7/25 {{clock}}Ran for {{duration}}
|
||||
- text: Available only on the last message of a completed turn 7/25 {{clock}} Ran for {{duration}} TTFT {{duration}} {{throughput}} tok/s
|
||||
- button "Read a.txt":
|
||||
- img
|
||||
- img
|
||||
@@ -46,7 +46,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: 7/25 {{clock}}Ran for {{duration}}
|
||||
- text: 7/25 {{clock}} Ran for {{duration}}
|
||||
- textbox "Message the agent"
|
||||
- button "Commands":
|
||||
- img
|
||||
@@ -55,4 +55,4 @@
|
||||
- text: Select model
|
||||
- img
|
||||
- button "Send message" [disabled]
|
||||
- text: 2 turns · 3 steps Tool call {{duration}} Cache hit 98% Input 7.8K tok · Output 103 tok
|
||||
- text: 2 turns · 3 steps LLM {{duration}} · Tool call {{duration}} TTFT avg {{duration}} · {{throughput}} tok/s Cache hit 98% Input 7.8K tok · Output 103 tok
|
||||
|
||||
@@ -36,7 +36,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: {{clock}}Ran for {{duration}}
|
||||
- text: {{clock}} Ran for {{duration}} TTFT {{duration}} {{throughput}} tok/s
|
||||
- textbox "Message the agent"
|
||||
- button "Commands":
|
||||
- img
|
||||
@@ -44,5 +44,6 @@
|
||||
- button "Select model, current DeepSeek-V4-Flash":
|
||||
- text: DeepSeek-V4-Flash
|
||||
- img
|
||||
- button "4% of context used"
|
||||
- button "Send message" [disabled]
|
||||
- text: 1 turns · 2 steps Tool call {{duration}} Context 4% of 128K Cache hit 51% Input 10.2K tok · Output 346 tok
|
||||
- text: 1 turns · 2 steps LLM {{duration}} · Tool call {{duration}} TTFT avg {{duration}} · {{throughput}} tok/s Cache hit 51% Input 10.2K tok · Output 346 tok
|
||||
|
||||
@@ -31,7 +31,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: {{clock}}Ran for {{duration}}
|
||||
- text: {{clock}} Ran for {{duration}} TTFT {{duration}} {{throughput}} tok/s
|
||||
- textbox "Message the agent"
|
||||
- button "Commands":
|
||||
- img
|
||||
@@ -39,5 +39,6 @@
|
||||
- button "Select model, current DeepSeek-V4-Flash":
|
||||
- text: DeepSeek-V4-Flash
|
||||
- img
|
||||
- button "3% of context used"
|
||||
- button "Send message" [disabled]
|
||||
- text: 1 turns · 2 steps Tool call {{duration}} Context 3% of 128K Cache hit 95% Input 8.6K tok · Output 180 tok
|
||||
- text: 1 turns · 2 steps LLM {{duration}} · Tool call {{duration}} TTFT avg {{duration}} · {{throughput}} tok/s Cache hit 95% Input 8.6K tok · Output 180 tok
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: {{clock}}Ran for {{duration}} Edited queue item {{clock}}
|
||||
- text: {{clock}} Ran for {{duration}} Edited queue item {{clock}}
|
||||
- button "Copy":
|
||||
- img
|
||||
- button "Branch into a new conversation" [disabled]:
|
||||
|
||||
@@ -33,7 +33,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: 7/25 {{clock}}Ran for {{duration}}
|
||||
- text: 7/25 {{clock}} Ran for {{duration}} TTFT {{duration}} {{throughput}} tok/s
|
||||
- button "Context compacted View compaction summary":
|
||||
- img
|
||||
- text: Context compacted View compaction summary
|
||||
@@ -51,4 +51,4 @@
|
||||
- text: Select model
|
||||
- img
|
||||
- button "Send message" [disabled]
|
||||
- text: 1 turns · 2 steps Tool call {{duration}} Cache hit 98% Input 15.8K tok · Output 135 tok
|
||||
- text: 1 turns · 2 steps LLM {{duration}} · Tool call {{duration}} TTFT avg {{duration}} · {{throughput}} tok/s Cache hit 98% Input 15.8K tok · Output 135 tok
|
||||
|
||||
@@ -33,7 +33,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: 7/25 {{clock}}Ran for {{duration}}
|
||||
- text: 7/25 {{clock}} Ran for {{duration}} TTFT {{duration}} {{throughput}} tok/s
|
||||
- button "Context compacted View compaction summary":
|
||||
- img
|
||||
- text: Context compacted View compaction summary
|
||||
@@ -49,4 +49,4 @@
|
||||
- text: Select model
|
||||
- img
|
||||
- button "Send message" [disabled]
|
||||
- text: 1 turns · 2 steps Tool call {{duration}} Cache hit 98% Input 15.8K tok · Output 135 tok
|
||||
- text: 1 turns · 2 steps LLM {{duration}} · Tool call {{duration}} TTFT avg {{duration}} · {{throughput}} tok/s Cache hit 98% Input 15.8K tok · Output 135 tok
|
||||
|
||||
@@ -37,7 +37,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: {{clock}}Ran for {{duration}}
|
||||
- text: {{clock}} Ran for {{duration}} TTFT {{duration}} {{throughput}} tok/s
|
||||
- textbox "Message the agent"
|
||||
- button "Commands":
|
||||
- img
|
||||
@@ -45,5 +45,6 @@
|
||||
- button "Select model, current DeepSeek-V4-Flash":
|
||||
- text: DeepSeek-V4-Flash
|
||||
- img
|
||||
- button "6% of context used"
|
||||
- button "Send message" [disabled]
|
||||
- text: 1 turns · 2 steps Tool call {{duration}} Context 6% of 128K Cache hit 98% Input 15.8K tok · Output 156 tok
|
||||
- text: 1 turns · 2 steps LLM {{duration}} · Tool call {{duration}} TTFT avg {{duration}} · {{throughput}} tok/s Cache hit 98% Input 15.8K tok · Output 156 tok
|
||||
|
||||
@@ -28,7 +28,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: {{clock}}Ran for {{duration}} Now give the same explanation to a human reader. {{clock}}
|
||||
- text: {{clock}} Ran for {{duration}} TTFT {{duration}} {{throughput}} tok/s Now give the same explanation to a human reader. {{clock}}
|
||||
- button "Copy":
|
||||
- img
|
||||
- button "Branch into a new conversation" [disabled]:
|
||||
@@ -43,10 +43,11 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: {{clock}}Ran for {{duration}}
|
||||
- text: {{clock}} Ran for {{duration}} TTFT {{duration}} {{throughput}} tok/s
|
||||
- textbox "Message the agent"
|
||||
- button "Commands":
|
||||
- img
|
||||
- 'button "Access mode, current: Workspace Write"': Workspace Write
|
||||
- button "6% of context used"
|
||||
- button "Send message" [disabled]
|
||||
- text: 2 turns · 2 steps Context 6% of 128K Cache hit 99% Input 15.6K tok · Output 158 tok
|
||||
- text: 2 turns · 2 steps LLM {{duration}} TTFT avg {{duration}} · {{throughput}} tok/s Cache hit 99% Input 15.6K tok · Output 158 tok
|
||||
|
||||
@@ -23,7 +23,7 @@
|
||||
- img
|
||||
- button "Branch into a new conversation":
|
||||
- img
|
||||
- text: {{clock}}Ran for {{duration}}
|
||||
- text: {{clock}} Ran for {{duration}} TTFT {{duration}} {{throughput}} tok/s
|
||||
- textbox "Message the agent"
|
||||
- button "Commands":
|
||||
- img
|
||||
@@ -31,5 +31,6 @@
|
||||
- button "Select model, current DeepSeek-V4-Flash":
|
||||
- text: DeepSeek-V4-Flash
|
||||
- img
|
||||
- button "0% of context used"
|
||||
- button "Send message" [disabled]
|
||||
- text: 1 turns · 2 steps Tool call {{duration}} Context 0% of 128K Cache hit 0% Input 22 tok · Output 7 tok
|
||||
- text: 1 turns · 2 steps LLM {{duration}} · Tool call {{duration}} TTFT avg {{duration}} · {{throughput}} tok/s Cache hit 0% Input 22 tok · Output 7 tok
|
||||
|
||||
Reference in New Issue
Block a user