c94f1563f5
Adds the first cut of snapshot scenarios, each asserting a normalized stdout
transcript golden and (for model turns) a re-persisted session-log golden:
- text-turn, tool-call-turn, multi-turn: RECORDED against the real API — the
committed session.jsonl is a genuine harvested log; replay derives the model
script from it and reproduces deterministically with no key. tool-call-turn
exercises the real bash executor (echo SNAPSHOT_OK → tool/call + tool/result
+ a post-tool answer step).
- error-finish, cancel: AUTHORED via a replay.override.json sidecar (the live
API can't be coaxed into a deterministic 401 or mid-stream cancel). error-
finish replays a {kind:throw} 401 → the bridge answers the prompt with a
JSON-RPC error and the log records turn/end{kind:error}; cancel replays a
{kind:hang} → stopReason:cancelled.
Two input-DSL ops support these: promptExpectError (awaits the prompt, asserts
it rejects — the editor's view of a failed turn — and swallows it) and
promptAndCancel (dispatches the prompt unawaited, waits until the client
OBSERVES the streamed agent_message_chunk, then cancels — pinning frame order
so the cancel transcript is deterministic; fixes a flake Codex caught where the
late chunk and the cancelled response could interleave either way).
Scenarios carry a `recorded` flag so test:snapshot:record only re-runs the
live-API ones. reasoning/max-tokens scenarios are deferred (hard to force
deterministically from the live model). Per docs/rfc/implemented/2026-06-19.
36 lines
5.0 KiB
JSON
36 lines
5.0 KiB
JSON
{"type":"session","version":1,"id":"dc42ad0b-4188-44c6-8d56-fd6f9cabe44f","createdAt":1781811940516,"cwd":"/tmp/acp-snap-cwd-9lipaP"}
|
|
{"type":"turn/start","seq":0,"time":1781811940519,"data":{"turn":1,"trigger":{"kind":"message","source":{"kind":"user"}}}}
|
|
{"type":"user/message","seq":1,"time":1781811940519,"data":{"content":[{"type":"text","text":"Reply with exactly the word: PONG. Do not use any tools."}],"source":{"kind":"user"}}}
|
|
{"type":"step/start","seq":2,"time":1781811940520,"data":{"turn":1,"step":1}}
|
|
{"type":"assistant/chunk","seq":3,"time":1781811940960,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":0,"blockType":"reasoning"}}}
|
|
{"type":"assistant/chunk","seq":4,"time":1781811940960,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"The"}}}
|
|
{"type":"assistant/chunk","seq":5,"time":1781811941102,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" user"}}}
|
|
{"type":"assistant/chunk","seq":6,"time":1781811941133,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" wants"}}}
|
|
{"type":"assistant/chunk","seq":7,"time":1781811941134,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" me"}}}
|
|
{"type":"assistant/chunk","seq":8,"time":1781811941134,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" to"}}}
|
|
{"type":"assistant/chunk","seq":9,"time":1781811941134,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" reply"}}}
|
|
{"type":"assistant/chunk","seq":10,"time":1781811941134,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" with"}}}
|
|
{"type":"assistant/chunk","seq":11,"time":1781811941134,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" exactly"}}}
|
|
{"type":"assistant/chunk","seq":12,"time":1781811941166,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" the"}}}
|
|
{"type":"assistant/chunk","seq":13,"time":1781811941167,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" word"}}}
|
|
{"type":"assistant/chunk","seq":14,"time":1781811941167,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" \""}}}
|
|
{"type":"assistant/chunk","seq":15,"time":1781811941167,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"P"}}}
|
|
{"type":"assistant/chunk","seq":16,"time":1781811941167,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"ONG"}}}
|
|
{"type":"assistant/chunk","seq":17,"time":1781811941167,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"\""}}}
|
|
{"type":"assistant/chunk","seq":18,"time":1781811941199,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" without"}}}
|
|
{"type":"assistant/chunk","seq":19,"time":1781811941200,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" using"}}}
|
|
{"type":"assistant/chunk","seq":20,"time":1781811941200,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" any"}}}
|
|
{"type":"assistant/chunk","seq":21,"time":1781811941200,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" tools"}}}
|
|
{"type":"assistant/chunk","seq":22,"time":1781811941200,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"."}}}
|
|
{"type":"assistant/chunk","seq":23,"time":1781811941233,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":1,"blockType":"text"}}}
|
|
{"type":"assistant/chunk","seq":24,"time":1781811941233,"data":{"turn":1,"step":1,"chunk":{"type":"text-delta","index":1,"text":"P"}}}
|
|
{"type":"assistant/chunk","seq":25,"time":1781811941234,"data":{"turn":1,"step":1,"chunk":{"type":"text-delta","index":1,"text":"ONG"}}}
|
|
{"type":"assistant/chunk","seq":26,"time":1781811941234,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":0,"block":{"type":"reasoning","text":"The user wants me to reply with exactly the word \"PONG\" without using any tools."}}}}
|
|
{"type":"assistant/chunk","seq":27,"time":1781811941234,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":1,"block":{"type":"text","text":"PONG"}}}}
|
|
{"type":"assistant/chunk","seq":28,"time":1781811941234,"data":{"turn":1,"step":1,"chunk":{"type":"usage","usage":{"inputTokens":117,"outputTokens":22,"cacheReadTokens":768,"reasoningTokens":19}}}}
|
|
{"type":"assistant/chunk","seq":29,"time":1781811941234,"data":{"turn":1,"step":1,"chunk":{"type":"finish","reason":{"kind":"stop"}}}}
|
|
{"type":"assistant/message","seq":30,"time":1781811941236,"data":{"turn":1,"step":1,"content":[{"type":"reasoning","text":"The user wants me to reply with exactly the word \"PONG\" without using any tools."},{"type":"text","text":"PONG"}]}}
|
|
{"type":"usage","seq":31,"time":1781811941236,"data":{"turn":1,"step":1,"usage":{"inputTokens":117,"outputTokens":22,"cacheReadTokens":768,"reasoningTokens":19}}}
|
|
{"type":"step/end","seq":32,"time":1781811941236,"data":{"turn":1,"step":1}}
|
|
{"type":"turn/end","seq":33,"time":1781811941236,"data":{"turn":1,"reason":{"kind":"completed"}}}
|