test(ralph): snapshot fresh-agent rounds in headless app

This commit is contained in:
Tianyi Cui
2026-07-20 18:18:08 +08:00
parent 94a6a95c8e
commit 9f6f87cf69
11 changed files with 157 additions and 4 deletions

View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write
2026-07-19-fresh-agent-ralph-workflow-tool.md: 159ad7e39602d8b84ffafdb396ad1083f9c73009
2026-07-19-fresh-agent-ralph-workflow-tool.zh.md: bb6025952ada35526459101d53b3ea1fea622163
2026-07-19-fresh-agent-ralph-workflow-tool.md: 46816b8cc06f0acf5615ffb44035c79ea13b6794
2026-07-19-fresh-agent-ralph-workflow-tool.zh.md: 188a0bf50a7a6a683cd6aa1004c4d78673d3631d

View File

@@ -46,7 +46,7 @@ ACP and terminal presentation use a generic `ralph` card whose raw input is the
Unit tests cover config and call-cap resolution, provider capability rejection, fixed start-request routing and child ceiling, all successful terminal outcomes, ordinary child-failure envelopes, malformed and oversized boundary values, exact successful-result truncation, abort timing, disposal, render intent, prompt lifecycle, and namespace-plugin shape at per-file 100% coverage. Worker-engine tests prove synchronous provider-route validation, per-run child ceilings below the deployment ceiling, and that a provider override selects every child without changing the configured default, including the built `lib/worker.cjs` under plain Node.
A keyless real-stack integration drives the fixed script through the actual worker-thread engine, spawn provider, structured-output runtime, and agent loop. It proves distinct child identities, absent `seedLength`, inherited cwd, no parent-history markers in either child request, exact previous-report handoff only in the following round, one phase event, terminal completion, and disposal of both children. The same real stack covers blocker and round-limit outcomes, unnormalized and semantically invalid reports, oversized handoffs, ordinary child failure with the last good handoff, and cancellation to child quiescence. Tool tests pin generic call/result presentation, while ACP replay header snapshots pin the shipped schema and prompt-guidance transcript surface.
A keyless real-stack integration drives the fixed script through the actual worker-thread engine, spawn provider, structured-output runtime, and agent loop. It proves distinct child identities, absent `seedLength`, inherited cwd, no parent-history markers in either child request, exact previous-report handoff only in the following round, one phase event, terminal completion, and disposal of both children. The same real stack covers blocker and round-limit outcomes, unnormalized and semantically invalid reports, oversized handoffs, ordinary child failure with the last good handoff, and cancellation to child quiescence. A shipped keyless headless snapshot additionally boots the real `examples/headless-agent` composition, invokes `ralph`, pins the parent stream transcript, and inspects persisted logs for two distinct unseeded child sessions and the round-one handoff appearing only in round two. Tool tests pin generic call/result presentation, while ACP replay header snapshots pin the shipped schema and prompt-guidance transcript surface.
## Alternatives considered

View File

@@ -46,7 +46,7 @@ ACP 和终端展示使用通用 `ralph` 卡片,并把目标作为原始输入
单元测试覆盖配置与调用上限解析、provider 能力拒绝、固定启动请求路由与子 agent 上限、全部成功终止结果、普通子 agent 失败外层值、畸形及过大边界值、成功结果精确截断、中止时序、处置、渲染意图、提示生命周期和命名空间插件形状,并达到逐文件 100% 覆盖率。工作流引擎测试证明 provider 路由会同步验证、每次运行的子 agent 上限可低于部署上限,并且 provider 覆盖会选择每个子 agent 且不改变配置默认值,其中包括普通 Node 下构建后的 `lib/worker.cjs`
一项无密钥真实栈集成测试通过实际工作线程引擎、spawn provider、结构化输出运行时和 agent loop 驱动固定脚本。它证明子 agent 标识不同、没有 `seedLength`、继承 cwd、两个子请求都不含父历史标记、上一份报告只精确出现在下一轮交接中、只产生一个阶段事件、终止完成以及两个子 agent 都被处置。同一真实栈还覆盖阻塞与轮次上限结果、未规范化及语义无效报告、过大交接、保留上一份有效交接的普通子 agent 失败,以及取消后达到子 agent 静止状态。工具测试固定通用调用/结果展示,而 ACP 重放请求头快照固定发布的 schema 与提示指导转录表面。
一项无密钥真实栈集成测试通过实际工作线程引擎、spawn provider、结构化输出运行时和 agent loop 驱动固定脚本。它证明子 agent 标识不同、没有 `seedLength`、继承 cwd、两个子请求都不含父历史标记、上一份报告只精确出现在下一轮交接中、只产生一个阶段事件、终止完成以及两个子 agent 都被处置。同一真实栈还覆盖阻塞与轮次上限结果、未规范化及语义无效报告、过大交接、保留上一份有效交接的普通子 agent 失败,以及取消后达到子 agent 静止状态。一项已发布的无密钥无头快照还会启动真实的 `examples/headless-agent` 组合、调用 `ralph`、固定父级流式转录,并检查持久化日志中存在两个不同且无种子的子会话,且第一轮交接只出现在第二轮。工具测试固定通用调用/结果展示,而 ACP 重放请求头快照固定发布的 schema 与提示指导转录表面。
## 考虑过的替代方案

View File

@@ -0,0 +1,12 @@
# Replay counterpart to cordis.yml for the shipped Ralph-loop snapshot.
- id: base
name: '@cordisjs/plugin-include'
config:
path: ./cordis.yml
patches:
- id: llm-deepseek
name: '@deepseek-ai/dsh-llm-deepseek'
disabled: true
- insert:
- id: llm-replay
name: '@deepseek-ai/dsh-llm-replay'

View File

@@ -17,6 +17,8 @@ const advancedStreamExpected = join(advancedScenarioDir, 'stream-json.expected.j
const advancedConfigPath = fileURLToPath(new URL('../advanced.cordis.snapshot.yml', import.meta.url))
const goalScenarioDir = join(snapshotsDir, 'goal-tools')
const goalConfigPath = fileURLToPath(new URL('../goal.cordis.snapshot.yml', import.meta.url))
const ralphScenarioDir = join(snapshotsDir, 'ralph-loop')
const ralphConfigPath = fileURLToPath(new URL('../ralph.cordis.snapshot.yml', import.meta.url))
const binScript = fileURLToPath(new URL('../../../packages/examples/cli-demo/src/bin.ts', import.meta.url))
const tsconfigPath = fileURLToPath(new URL('../../../tsconfig.json', import.meta.url))
const refreshing = process.env.DSH_SNAPSHOT === 'refresh'
@@ -216,4 +218,77 @@ describe('headless stream-json snapshots', () => {
if (refreshing) await writeFile(streamExpected, normalized)
expect(normalized).toBe(await readFile(streamExpected, 'utf8'))
}, LOADER_SMOKE_TEST_TIMEOUT_MS)
it('replays two fresh Ralph rounds through the one-shot app', async () => {
const prompt = await scenarioPrompt(ralphScenarioDir, 'ralph-loop')
const streamExpected = join(ralphScenarioDir, 'stream-json.expected.jsonl')
let runCwd = ''
const result = await runLoaderSmoke({
label: 'Ralph loop headless stream-json snapshot',
tempDirPrefix: 'headless-snapshot-ralph-loop-',
binScript,
configPath: ralphConfigPath,
binArgs: ['--config', ralphConfigPath, '--output-format', 'stream-json', prompt],
tsconfigPath,
env: {
DSH_SNAPSHOT: 'replay',
DSH_SNAPSHOT_FILE: join(ralphScenarioDir, 'session.jsonl'),
DSH_SNAPSHOT_OVERRIDE: join(ralphScenarioDir, 'replay.override.json'),
DSH_SNAPSHOT_CHILD_FILES: [
join(ralphScenarioDir, 'session.1.jsonl'),
join(ralphScenarioDir, 'session.2.jsonl'),
].join(delimiter),
NODE_OPTIONS: [process.env.NODE_OPTIONS, '--disable-warning=ExperimentalWarning'].filter(Boolean).join(' '),
},
prepare: (cwd) => { runCwd = cwd },
inspect: async (cwd) => {
const logs = await persistedLogs(cwd)
expect(logs).toHaveLength(3)
const parent = logs.find(log => typeof log.header.parentSession !== 'string')
if (parent === undefined) throw new Error('Ralph snapshot did not persist its parent session')
const parentId = parent.header.id
expect(typeof parentId).toBe('string')
const children = logs.filter(log => typeof log.header.parentSession === 'string')
.sort((left, right) => Number(left.header.createdAt) - Number(right.header.createdAt))
expect(children).toHaveLength(2)
expect(children.map(child => child.header.parentSession)).toEqual([parentId, parentId])
expect(children.map(child => child.header.cwd)).toEqual([parent.header.cwd, parent.header.cwd])
expect(children.map(child => child.header.seedLength)).toEqual([undefined, undefined])
expect(new Set(children.map(child => child.header.id)).size).toBe(2)
const parentRecords = parseJsonl(parent.content)
const parentCalls = parentRecords.filter(record => record.type === 'tool/call')
expect(parentCalls.map(record => (record.data as JsonObject | undefined)?.name)).toEqual(['ralph'])
const parentResult = parentRecords.find(record => record.type === 'tool/result')
const parentResultData = parentResult?.data as JsonObject | undefined
expect(parentResultData?.isError).toBe(false)
expect(JSON.stringify(parentResultData?.content)).toContain('reported completion after 2 rounds')
const childRecords = children.map(child => parseJsonl(child.content))
const childPrompts = childRecords.map((records) => {
const message = records.find(record => record.type === 'user/message')
return JSON.stringify((message?.data as JsonObject | undefined)?.content)
})
expect(childPrompts[0]).toContain('Ralph round: 1 of 2.')
expect(childPrompts[0]).toContain('(none — this is the first round)')
expect(childPrompts[0]).not.toContain('ROUND_ONE_HANDOFF')
expect(childPrompts[1]).toContain('Ralph round: 2 of 2.')
expect(childPrompts[1]).toContain('ROUND_ONE_HANDOFF')
for (const childPrompt of childPrompts) {
expect(childPrompt).toContain('Prove two fresh Ralph rounds through the shipped headless app.')
expect(childPrompt).not.toContain('Run a two-round fresh-agent Ralph loop')
}
for (const records of childRecords) {
const calls = records.filter(record => record.type === 'tool/call')
expect(calls.map(record => (record.data as JsonObject | undefined)?.name))
.toEqual(['structured_output'])
}
},
})
expect(result.stderr).toBe('')
const normalized = normalizeHeadlessStream(result.stdout, runCwd)
if (refreshing) await writeFile(streamExpected, normalized)
expect(normalized).toBe(await readFile(streamExpected, 'utf8'))
}, LOADER_SMOKE_TEST_TIMEOUT_MS)
})

View File

@@ -0,0 +1,8 @@
{
"steps": [
{
"op": "prompt",
"text": "Run a two-round fresh-agent Ralph loop to prove the shipped headless integration."
}
]
}

View File

@@ -0,0 +1,22 @@
[
{
"kind": "chunks",
"chunks": [
{ "type": "block-start", "index": 0, "blockType": "tool-call" },
{ "type": "tool-call-delta", "index": 0, "id": "call_ralph", "name": "ralph", "argumentsDelta": "{\"objective\":\"Prove two fresh Ralph rounds through the shipped headless app.\",\"maxRounds\":2}" },
{ "type": "block-end", "index": 0, "block": { "type": "tool-call", "id": "call_ralph", "name": "ralph", "arguments": "{\"objective\":\"Prove two fresh Ralph rounds through the shipped headless app.\",\"maxRounds\":2}" } },
{ "type": "usage", "usage": { "inputTokens": 20, "outputTokens": 8 } },
{ "type": "finish", "reason": { "kind": "tool-calls" } }
]
},
{
"kind": "chunks",
"chunks": [
{ "type": "block-start", "index": 0, "blockType": "text" },
{ "type": "text-delta", "index": 0, "text": "RALPH SNAPSHOT COMPLETE" },
{ "type": "block-end", "index": 0, "block": { "type": "text", "text": "RALPH SNAPSHOT COMPLETE" } },
{ "type": "usage", "usage": { "inputTokens": 30, "outputTokens": 4 } },
{ "type": "finish", "reason": { "kind": "stop" } }
]
}
]

View File

@@ -0,0 +1,6 @@
{"type":"session","version":0,"id":"42222222-2222-4222-8222-222222222222","createdAt":1783951001000,"cwd":"/tmp/ralph-headless","parentSession":"41111111-1111-4111-8111-111111111111"}
{"type":"assistant/chunk","seq":0,"time":1783951001001,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}}}
{"type":"assistant/chunk","seq":1,"time":1783951001002,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":0,"id":"round-one-report","name":"structured_output","argumentsDelta":"{\"status\":\"continue\",\"summary\":\"ROUND_ONE_HANDOFF\",\"evidence\":[\"Round one inspected the workspace.\"],\"nextSteps\":[\"Finish the snapshot objective.\"],\"blocker\":\"\"}"}}}
{"type":"assistant/chunk","seq":2,"time":1783951001003,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"round-one-report","name":"structured_output","arguments":"{\"status\":\"continue\",\"summary\":\"ROUND_ONE_HANDOFF\",\"evidence\":[\"Round one inspected the workspace.\"],\"nextSteps\":[\"Finish the snapshot objective.\"],\"blocker\":\"\"}"}}}}
{"type":"assistant/chunk","seq":3,"time":1783951001004,"data":{"turn":1,"step":1,"chunk":{"type":"usage","usage":{"inputTokens":30,"outputTokens":12}}}}
{"type":"assistant/chunk","seq":4,"time":1783951001005,"data":{"turn":1,"step":1,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}}

View File

@@ -0,0 +1,6 @@
{"type":"session","version":0,"id":"43333333-3333-4333-8333-333333333333","createdAt":1783951002000,"cwd":"/tmp/ralph-headless","parentSession":"41111111-1111-4111-8111-111111111111"}
{"type":"assistant/chunk","seq":0,"time":1783951002001,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}}}
{"type":"assistant/chunk","seq":1,"time":1783951002002,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":0,"id":"round-two-report","name":"structured_output","argumentsDelta":"{\"status\":\"complete\",\"summary\":\"The Ralph snapshot objective is complete.\",\"evidence\":[\"Two fresh rounds completed through the shipped app.\"],\"nextSteps\":[],\"blocker\":\"\"}"}}}
{"type":"assistant/chunk","seq":2,"time":1783951002003,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"round-two-report","name":"structured_output","arguments":"{\"status\":\"complete\",\"summary\":\"The Ralph snapshot objective is complete.\",\"evidence\":[\"Two fresh rounds completed through the shipped app.\"],\"nextSteps\":[],\"blocker\":\"\"}"}}}}
{"type":"assistant/chunk","seq":3,"time":1783951002004,"data":{"turn":1,"step":1,"chunk":{"type":"usage","usage":{"inputTokens":40,"outputTokens":12}}}}
{"type":"assistant/chunk","seq":4,"time":1783951002005,"data":{"turn":1,"step":1,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}}

View File

@@ -0,0 +1 @@
{"type":"session","version":0,"id":"41111111-1111-4111-8111-111111111111","createdAt":1783951000000,"cwd":"/tmp/ralph-headless"}

View File

@@ -0,0 +1,23 @@
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"turn/start","seq":0,"time":0,"data":{"turn":1,"trigger":{"kind":"message","source":{"kind":"user"}}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"user/message","seq":1,"time":0,"data":{"content":[{"type":"text","text":"Run a two-round fresh-agent Ralph loop to prove the shipped headless integration."}],"source":{"kind":"user"}},"surfaceOp":"append"}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"step/start","seq":2,"time":0,"data":{"turn":1,"step":1}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"request/header","seq":3,"time":0,"data":{"header":{"config":{"provider":"deepseek","model":"deepseek-v4-flash"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":4,"time":0,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":0,"blockType":"tool-call"}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":5,"time":0,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":0,"id":"call_ralph","name":"ralph","argumentsDelta":"{\"objective\":\"Prove two fresh Ralph rounds through the shipped headless app.\",\"maxRounds\":2}"}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":6,"time":0,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":0,"block":{"type":"tool-call","id":"call_ralph","name":"ralph","arguments":"{\"objective\":\"Prove two fresh Ralph rounds through the shipped headless app.\",\"maxRounds\":2}"}}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":7,"time":0,"data":{"turn":1,"step":1,"chunk":{"type":"usage","usage":{"inputTokens":20,"outputTokens":8}}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":8,"time":0,"data":{"turn":1,"step":1,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/message","seq":9,"time":0,"data":{"turn":1,"step":1,"content":[{"type":"tool-call","id":"call_ralph","name":"ralph","arguments":"{\"objective\":\"Prove two fresh Ralph rounds through the shipped headless app.\",\"maxRounds\":2}"}],"provenance":{"provider":"deepseek","model":"deepseek-v4-flash"},"usage":{"inputTokens":20,"outputTokens":8}},"sourceEventSeqs":[4,5,6,7,8],"surfaceOp":"append"}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"tool/call","seq":10,"time":0,"data":{"turn":1,"step":1,"callId":"call_ralph","name":"ralph","arguments":"{\"objective\":\"Prove two fresh Ralph rounds through the shipped headless app.\",\"maxRounds\":2}"}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"tool/result","seq":11,"time":0,"data":{"turn":1,"step":1,"callId":"call_ralph","content":[{"type":"text","text":"Ralph worker reported completion after 2 rounds.\nFinal report:\n{\n \"status\": \"complete\",\n \"summary\": \"The Ralph snapshot objective is complete.\",\n \"evidence\": [\n \"Two fresh rounds completed through the shipped app.\"\n ],\n \"nextSteps\": [],\n \"blocker\": \"\"\n}"}],"isError":false},"sourceEventSeqs":[10],"surfaceOp":"append"}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"step/end","seq":12,"time":0,"data":{"turn":1,"step":1}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"step/start","seq":13,"time":0,"data":{"turn":1,"step":2}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":14,"time":0,"data":{"turn":1,"step":2,"chunk":{"type":"block-start","index":0,"blockType":"text"}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":15,"time":0,"data":{"turn":1,"step":2,"chunk":{"type":"text-delta","index":0,"text":"RALPH SNAPSHOT COMPLETE"}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":16,"time":0,"data":{"turn":1,"step":2,"chunk":{"type":"block-end","index":0,"block":{"type":"text","text":"RALPH SNAPSHOT COMPLETE"}}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":17,"time":0,"data":{"turn":1,"step":2,"chunk":{"type":"usage","usage":{"inputTokens":30,"outputTokens":4}}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/chunk","seq":18,"time":0,"data":{"turn":1,"step":2,"chunk":{"type":"finish","reason":{"kind":"stop"}}}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"assistant/message","seq":19,"time":0,"data":{"turn":1,"step":2,"content":[{"type":"text","text":"RALPH SNAPSHOT COMPLETE"}],"provenance":{"provider":"deepseek","model":"deepseek-v4-flash"},"usage":{"inputTokens":30,"outputTokens":4}},"sourceEventSeqs":[14,15,16,17,18],"surfaceOp":"append"}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"step/end","seq":20,"time":0,"data":{"turn":1,"step":2}}}
{"type":"session_event","sessionId":"{{sessionId}}","event":{"type":"turn/end","seq":21,"time":0,"data":{"turn":1,"reason":{"kind":"completed"}}}}
{"type":"result","success":true,"sessionId":"{{sessionId}}","turn":1,"result":"RALPH SNAPSHOT COMPLETE","reason":{"kind":"completed"},"usage":{"inputTokens":50,"outputTokens":12}}