fix(ralph): harden execution boundaries
This commit is contained in:
@@ -4,11 +4,13 @@ The model-facing `ralph` tool runs a fixed foreground workflow that gives one im
|
||||
|
||||
## Contract
|
||||
|
||||
`ralph({ objective, maxRounds? })` waits for the entire run. The deployment config's `maxRounds` is both the default and a ceiling on a call override. Every Ralph round starts one child through `subagentProvider`; that provider must exist, support structured output, and report `inheritsParentContext: false`. The configured provider is carried as `WorkflowStartRequest.subagentProvider`, so the fixed script cannot inspect or change routing and the ordinary model-written `workflow` tool gains no provider selector.
|
||||
`ralph({ objective, maxRounds? })` waits for the entire run. The deployment config's `maxRounds` is both the default and a ceiling on a call override. Every Ralph round starts one child through `subagentProvider`; that provider must exist, support structured output, and report `inheritsParentContext: false`. The configured provider is carried as `WorkflowStartRequest.subagentProvider`, so the fixed script cannot inspect or change routing and the ordinary model-written `workflow` tool gains no provider selector. The resolved round cap is also carried as `WorkflowStartRequest.maxTotalAgents`, coordinating the fixed loop with the engine's total-child backstop; the engine rejects a Ralph cap above its deployment ceiling before publishing a run.
|
||||
|
||||
Each child receives only the immutable objective, its current Ralph round and cap, a shared-workspace-as-authority instruction, and the previous structured handoff. The workspace is long-term memory; parent conversation and prior child sessions are not seeded. Reports have `status: continue | complete | blocked`, a non-empty summary, evidence, next steps, and blocker text. Status-specific semantics and the serialized `maxHandoffChars` ceiling are validated inside the fixed workflow and again at the consumer boundary. Invalid, missing, or oversized reports fail the workflow instead of being truncated or mistaken for cap exhaustion.
|
||||
|
||||
The terminal tool result is `complete`, `blocked`, or `budget-limited`, with the last bounded report and number of rounds started. Child self-declaration determines completion in this cut. A workflow failure or cancellation is an error result; partial output is never success.
|
||||
The successful terminal tool result is `complete`, `blocked`, or `budget-limited`, with the last bounded report and number of rounds started. Completion and blocker labels explicitly say that a worker reported the outcome; they are not independent certification. `maxResultChars` bounds the complete successful text including its envelope and truncation marker, without altering the validated report used as a cross-round handoff.
|
||||
|
||||
An ordinary child failure produces an error naming the failed round and retaining the last successful handoff when one exists. Ralph does not retry that round. Fatal provider-start, transport, worker, or workflow failures remain workflow errors and may settle before the fixed script can return a handoff. Cancellation is also an error; partial output is never success.
|
||||
|
||||
## Lifecycle and cancellation
|
||||
|
||||
@@ -25,6 +27,7 @@ The pending call is a `generic` card titled `ralph`; the immutable objective is
|
||||
| `subagentProvider` | `spawn` | Fresh structured-output provider used for every round. |
|
||||
| `maxRounds` | `256` | Default and deployment ceiling for one Ralph run. |
|
||||
| `maxHandoffChars` | `16384` | Maximum serialized characters in one round report. |
|
||||
| `maxResultChars` | `16384` | Maximum characters in the complete successful parent result. |
|
||||
|
||||
All config values are normalized and validated when the plugin applies, including direct application outside Loader schema normalization. Provider capabilities are resolved immediately before each call because provider registration can change under plugin lifecycle and HMR.
|
||||
|
||||
@@ -39,7 +42,7 @@ Every parent request in this plugin's registration scope receives the fixed rout
|
||||
##### Ralph guidance
|
||||
|
||||
```markdown
|
||||
Use the ralph tool ONLY when the direct human explicitly asks for a Ralph loop or fresh-agent iterative execution. Each Ralph round starts a fresh child with no conversation seed and uses the shared workspace as durable memory. Use same-session goal tools for ordinary long-running objectives, and plain subagents or workflows for bounded delegation and fan-out.
|
||||
Use the ralph tool ONLY when the direct human explicitly asks for a Ralph loop or fresh-agent iterative execution. Each Ralph round starts a fresh child with no conversation seed and uses the shared workspace as durable memory. Completion and blockers are worker reports, not independent evaluation. Use same-session goal tools for ordinary long-running objectives, and plain subagents or workflows for bounded delegation and fan-out.
|
||||
```
|
||||
|
||||
#### Token effect
|
||||
@@ -68,11 +71,11 @@ Prefix-stable while the definition and visibility are unchanged.
|
||||
|
||||
#### What the model sees
|
||||
|
||||
Each child sees the standalone fixed round prompt plus the structured-output capture contract. The parent sees only the original call and one terminal result containing status, round count, and pretty-printed final report; intermediate child messages and reports do not enter the parent conversation.
|
||||
Each child sees the standalone fixed round prompt plus the structured-output capture contract. The parent sees only the original call and one terminal result containing a worker-reported status, round count, and pretty-printed final report; intermediate child messages and reports do not enter the parent conversation. A failed ordinary child instead yields an error with its round number and, after round one, the last successful handoff.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Every round pays for a fresh child context. The parent result is bounded indirectly by `maxHandoffChars`; child work remains outside the parent context.
|
||||
Every round pays for a fresh child context. `maxHandoffChars` bounds cross-round state and `maxResultChars` independently bounds the complete successful parent text; child work remains outside the parent context.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
@@ -84,4 +87,5 @@ Each fresh child has an independent request cache. The parent result appends aft
|
||||
- **Foreground only** — there is no task id, background collection, process-resume checkpoint, scheduler, or wall-clock start policy.
|
||||
- **The workspace is the only cross-round long-term memory** — one bounded report is the explicit handoff, and uncommitted conversational reasoning disappears with each child.
|
||||
- **One round is one fresh child** — there is no within-round fan-out, model/provider switching, fork context, or model-call-selected provider.
|
||||
- **Ordinary child failure is terminal for the run** — the fixed script reports the failed round and last successful handoff but does not retry; fatal workflow infrastructure failures can end before that state is returned.
|
||||
- **Only round count bounds aggregate effort** — token, price, and elapsed-time budgets are deferred.
|
||||
|
||||
@@ -26,6 +26,8 @@ export interface Config {
|
||||
maxRounds?: number
|
||||
/** Maximum serialized characters in one structured handoff (default 16384). */
|
||||
maxHandoffChars?: number
|
||||
/** Maximum characters in a successful parent-facing terminal text (default 16384). */
|
||||
maxResultChars?: number
|
||||
}
|
||||
|
||||
/** Schemastery configuration for the Ralph tool. */
|
||||
@@ -33,12 +35,14 @@ export const Config: z<Config> = z.object({
|
||||
subagentProvider: z.string().default('spawn'),
|
||||
maxRounds: z.number().step(1).min(1).max(Number.MAX_SAFE_INTEGER).default(256),
|
||||
maxHandoffChars: z.number().step(1).min(1).max(Number.MAX_SAFE_INTEGER).default(16_384),
|
||||
maxResultChars: z.number().step(1).min(1).max(Number.MAX_SAFE_INTEGER).default(16_384),
|
||||
})
|
||||
|
||||
interface ResolvedConfig {
|
||||
readonly subagentProvider: string
|
||||
readonly maxRounds: number
|
||||
readonly maxHandoffChars: number
|
||||
readonly maxResultChars: number
|
||||
}
|
||||
|
||||
type RalphRoundStatus = 'continue' | 'complete' | 'blocked'
|
||||
@@ -59,6 +63,14 @@ interface RalphRunResult {
|
||||
readonly report: RalphRoundReport
|
||||
}
|
||||
|
||||
interface RalphRoundFailure {
|
||||
readonly status: 'round-failed'
|
||||
readonly roundsStarted: number
|
||||
readonly lastReport?: RalphRoundReport
|
||||
}
|
||||
|
||||
type RalphTerminalResult = RalphRunResult | RalphRoundFailure
|
||||
|
||||
interface RalphCallArgs {
|
||||
objective: string
|
||||
maxRounds?: number
|
||||
@@ -136,8 +148,8 @@ function validateReport(report) {
|
||||
}
|
||||
|
||||
let previous
|
||||
phase('Fresh-agent rounds')
|
||||
for (let round = 1; round <= args.maxRounds; round += 1) {
|
||||
phase('Fresh-agent rounds')
|
||||
const prior = previous === undefined ? '(none — this is the first round)' : JSON.stringify(previous)
|
||||
const prompt = [
|
||||
'You are one fresh worker in a foreground Ralph loop. You receive no parent conversation and no prior child session. Do not call the ralph tool: this round already is its worker.',
|
||||
@@ -147,11 +159,15 @@ for (let round = 1; round <= args.maxRounds; round += 1) {
|
||||
'Previous structured handoff:\n' + prior,
|
||||
'Return one report with exact normalized strings. Use status continue with at least one nextSteps entry while useful work remains; complete only with concrete evidence and no nextSteps; blocked only when no meaningful progress is possible without human input or an external-state change. blocker must be empty unless blocked.',
|
||||
].join('\n\n')
|
||||
const report = validateReport(await agent(prompt, {
|
||||
const rawReport = await agent(prompt, {
|
||||
label: 'Ralph round ' + round,
|
||||
phase: 'Fresh-agent rounds',
|
||||
schema: reportSchema,
|
||||
}))
|
||||
})
|
||||
if (rawReport === null) {
|
||||
return { status: 'round-failed', roundsStarted: round, lastReport: previous ?? null }
|
||||
}
|
||||
const report = validateReport(rawReport)
|
||||
if (report.status === 'complete') return { status: 'complete', roundsStarted: round, report }
|
||||
if (report.status === 'blocked') return { status: 'blocked', roundsStarted: round, report }
|
||||
previous = report
|
||||
@@ -162,8 +178,8 @@ return { status: 'budget-limited', roundsStarted: args.maxRounds, report: previo
|
||||
const DESCRIPTION = 'Run a foreground fresh-agent Ralph loop toward one immutable objective. '
|
||||
+ 'Use only when the direct human explicitly asks for Ralph or fresh-agent iteration. Each round '
|
||||
+ 'opens a new child with no parent conversation or prior child session; the shared workspace is '
|
||||
+ 'long-term memory, and only a bounded structured report crosses rounds. The call returns on '
|
||||
+ 'completion, a concrete blocker, or the round limit. Ordinary long-running same-session work '
|
||||
+ 'long-term memory, and only a bounded structured report crosses rounds. The call returns when '
|
||||
+ 'a worker reports completion or a concrete blocker, or at the round limit. Ordinary long-running same-session work '
|
||||
+ 'belongs to goal tools.'
|
||||
|
||||
/** Validate defaults even when a caller invokes apply() without Loader normalization. */
|
||||
@@ -171,6 +187,7 @@ function resolveConfig(config: Config): ResolvedConfig {
|
||||
const subagentProvider = config.subagentProvider ?? 'spawn'
|
||||
const maxRounds = config.maxRounds ?? 256
|
||||
const maxHandoffChars = config.maxHandoffChars ?? 16_384
|
||||
const maxResultChars = config.maxResultChars ?? 16_384
|
||||
if (subagentProvider.length === 0 || subagentProvider !== subagentProvider.trim()) {
|
||||
throw new TypeError('subagentProvider must be a non-empty normalized string')
|
||||
}
|
||||
@@ -180,7 +197,10 @@ function resolveConfig(config: Config): ResolvedConfig {
|
||||
if (!Number.isSafeInteger(maxHandoffChars) || maxHandoffChars < 1) {
|
||||
throw new TypeError('maxHandoffChars must be a positive safe integer')
|
||||
}
|
||||
return { subagentProvider, maxRounds, maxHandoffChars }
|
||||
if (!Number.isSafeInteger(maxResultChars) || maxResultChars < 1) {
|
||||
throw new TypeError('maxResultChars must be a positive safe integer')
|
||||
}
|
||||
return { subagentProvider, maxRounds, maxHandoffChars, maxResultChars }
|
||||
}
|
||||
|
||||
/** Resolve one model-selected cap against the deployment ceiling. */
|
||||
@@ -259,9 +279,8 @@ function readReport(value: unknown, expectedStatus: RalphRoundStatus, maxChars:
|
||||
}
|
||||
|
||||
/** Defensively decode the fixed script's terminal value. */
|
||||
function readRunResult(value: unknown, maxRounds: number, maxHandoffChars: number): RalphRunResult {
|
||||
function readRunResult(value: unknown, maxRounds: number, maxHandoffChars: number): RalphTerminalResult {
|
||||
if (!isRecord(value)
|
||||
|| Object.keys(value).sort().join(',') !== 'report,roundsStarted,status'
|
||||
|| typeof value['roundsStarted'] !== 'number'
|
||||
|| !Number.isSafeInteger(value['roundsStarted'])
|
||||
|| value['roundsStarted'] < 1
|
||||
@@ -271,14 +290,42 @@ function readRunResult(value: unknown, maxRounds: number, maxHandoffChars: numbe
|
||||
const roundsStarted = value['roundsStarted']
|
||||
switch (value['status']) {
|
||||
case 'complete':
|
||||
if (Object.keys(value).sort().join(',') !== 'report,roundsStarted,status') {
|
||||
throw new Error('Ralph workflow returned a malformed terminal result')
|
||||
}
|
||||
return { status: 'complete', roundsStarted, report: readReport(value['report'], 'complete', maxHandoffChars) }
|
||||
case 'blocked':
|
||||
if (Object.keys(value).sort().join(',') !== 'report,roundsStarted,status') {
|
||||
throw new Error('Ralph workflow returned a malformed terminal result')
|
||||
}
|
||||
return { status: 'blocked', roundsStarted, report: readReport(value['report'], 'blocked', maxHandoffChars) }
|
||||
case 'budget-limited':
|
||||
if (Object.keys(value).sort().join(',') !== 'report,roundsStarted,status') {
|
||||
throw new Error('Ralph workflow returned a malformed terminal result')
|
||||
}
|
||||
if (roundsStarted !== maxRounds) {
|
||||
throw new Error('Ralph workflow returned budget-limited before the round limit')
|
||||
}
|
||||
return { status: 'budget-limited', roundsStarted, report: readReport(value['report'], 'continue', maxHandoffChars) }
|
||||
case 'round-failed': {
|
||||
if (Object.keys(value).sort().join(',') !== 'lastReport,roundsStarted,status') {
|
||||
throw new Error('Ralph workflow returned a malformed terminal result')
|
||||
}
|
||||
if (roundsStarted === 1) {
|
||||
if (value['lastReport'] !== null) {
|
||||
throw new Error('Ralph workflow returned an invalid first-round failure')
|
||||
}
|
||||
return { status: 'round-failed', roundsStarted }
|
||||
}
|
||||
if (value['lastReport'] === null) {
|
||||
throw new Error('Ralph workflow returned a round failure without its last handoff')
|
||||
}
|
||||
return {
|
||||
status: 'round-failed',
|
||||
roundsStarted,
|
||||
lastReport: readReport(value['lastReport'], 'continue', maxHandoffChars),
|
||||
}
|
||||
}
|
||||
default:
|
||||
throw new Error('Ralph workflow returned an unknown terminal status')
|
||||
}
|
||||
@@ -300,17 +347,40 @@ function stopReasonError(result: WorkflowResult): string | undefined {
|
||||
}
|
||||
}
|
||||
|
||||
/** Render the fixed terminal envelope without dropping the bounded report. */
|
||||
function renderResult(result: RalphRunResult): string {
|
||||
const TRUNCATION_NOTICE = '\n… [truncated]'
|
||||
|
||||
/** Bound complete parent-facing text, including its envelope and truncation marker. */
|
||||
function boundResult(text: string, maxChars: number): string {
|
||||
if (text.length <= maxChars) return text
|
||||
if (maxChars <= TRUNCATION_NOTICE.length) return TRUNCATION_NOTICE.slice(0, maxChars)
|
||||
return `${text.slice(0, maxChars - TRUNCATION_NOTICE.length)}${TRUNCATION_NOTICE}`
|
||||
}
|
||||
|
||||
/** Render the fixed terminal envelope without presenting self-report as certification. */
|
||||
function renderResult(result: RalphRunResult, maxChars: number): string {
|
||||
const rounds = `${result.roundsStarted} round${result.roundsStarted === 1 ? '' : 's'}`
|
||||
let text: string
|
||||
switch (result.status) {
|
||||
case 'complete':
|
||||
return `Ralph completed after ${rounds}.\nFinal report:\n${JSON.stringify(result.report, null, 2)}`
|
||||
text = `Ralph worker reported completion after ${rounds}.\nFinal report:\n${JSON.stringify(result.report, null, 2)}`
|
||||
break
|
||||
case 'blocked':
|
||||
return `Ralph blocked after ${rounds}.\nFinal report:\n${JSON.stringify(result.report, null, 2)}`
|
||||
text = `Ralph worker reported a blocker after ${rounds}.\nFinal report:\n${JSON.stringify(result.report, null, 2)}`
|
||||
break
|
||||
case 'budget-limited':
|
||||
return `Ralph reached its ${rounds} limit with work remaining.\nFinal report:\n${JSON.stringify(result.report, null, 2)}`
|
||||
text = `Ralph reached its ${rounds} limit; the worker reported work remaining.\nFinal report:\n${JSON.stringify(result.report, null, 2)}`
|
||||
break
|
||||
}
|
||||
return boundResult(text, maxChars)
|
||||
}
|
||||
|
||||
/** Render an ordinary child failure with the most recent durable handoff. */
|
||||
function renderRoundFailure(result: RalphRoundFailure, maxChars: number): string {
|
||||
const header = `Ralph round ${result.roundsStarted} child failed before producing a structured report.`
|
||||
const text = result.lastReport === undefined
|
||||
? `${header}\nNo previous handoff was available.`
|
||||
: `${header}\nLast successful handoff:\n${JSON.stringify(result.lastReport, null, 2)}`
|
||||
return boundResult(text, maxChars)
|
||||
}
|
||||
|
||||
function presentCall(args: RalphCallArgs): ToolCallView {
|
||||
@@ -329,7 +399,7 @@ export function apply(ctx: Context, config: Config): void {
|
||||
ctx.systemPrompt.section({
|
||||
name: 'tool:ralph',
|
||||
order: 116,
|
||||
text: 'Use the ralph tool ONLY when the direct human explicitly asks for a Ralph loop or fresh-agent iterative execution. Each Ralph round starts a fresh child with no conversation seed and uses the shared workspace as durable memory. Use same-session goal tools for ordinary long-running objectives, and plain subagents or workflows for bounded delegation and fan-out.',
|
||||
text: 'Use the ralph tool ONLY when the direct human explicitly asks for a Ralph loop or fresh-agent iterative execution. Each Ralph round starts a fresh child with no conversation seed and uses the shared workspace as durable memory. Completion and blockers are worker reports, not independent evaluation. Use same-session goal tools for ordinary long-running objectives, and plain subagents or workflows for bounded delegation and fan-out.',
|
||||
})
|
||||
ctx.tools.register(defineTool({
|
||||
name: 'ralph',
|
||||
@@ -360,6 +430,7 @@ export function apply(ctx: Context, config: Config): void {
|
||||
meta: RALPH_META,
|
||||
args: { objective, maxRounds, maxHandoffChars: resolved.maxHandoffChars },
|
||||
subagentProvider: resolved.subagentProvider,
|
||||
maxTotalAgents: maxRounds,
|
||||
parent,
|
||||
...exec.signal === undefined ? {} : { signal: exec.signal },
|
||||
})
|
||||
@@ -372,7 +443,8 @@ export function apply(ctx: Context, config: Config): void {
|
||||
const error = stopReasonError(settled)
|
||||
if (error !== undefined) throw new Error(error)
|
||||
const value = readRunResult(settled.value, maxRounds, resolved.maxHandoffChars)
|
||||
return [{ type: 'text', text: renderResult(value) }]
|
||||
if (value.status === 'round-failed') throw new Error(renderRoundFailure(value, resolved.maxResultChars))
|
||||
return [{ type: 'text', text: renderResult(value, resolved.maxResultChars) }]
|
||||
} finally {
|
||||
exec.signal?.removeEventListener('abort', onAbort)
|
||||
await run.dispose()
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import type { Agent } from '@deepseek-ai/dsh-agent'
|
||||
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
|
||||
@@ -10,9 +10,31 @@ import SubagentService from '@deepseek-ai/dsh-subagent'
|
||||
import { STRUCTURED_OUTPUT_TOOL } from '@deepseek-ai/dsh-subagent-inprocess'
|
||||
import * as spawn from '@deepseek-ai/dsh-subagent-spawn'
|
||||
import WorkerWorkflowEngine from '@deepseek-ai/dsh-workflow-workerthread'
|
||||
import { MockAdapter, textResponse, toolCallResponse } from '../../../core/agent-loop/tests/mock-adapter.ts'
|
||||
import { MockAdapter, maxTokensResponse, textResponse, toolCallResponse } from '../../../core/agent-loop/tests/mock-adapter.ts'
|
||||
import * as toolRalph from '../src/index.ts'
|
||||
|
||||
type MockScript = ConstructorParameters<typeof MockAdapter>[0]
|
||||
|
||||
/** Mount the shipped Ralph execution stack around one keyless model script. */
|
||||
async function mountRalph(script: MockScript, config: toolRalph.Config) {
|
||||
const ctx = new Context()
|
||||
const adapter = new MockAdapter(script)
|
||||
await mountAgentLoopTestDependencies(ctx)
|
||||
await ctx.plugin(Invariants)
|
||||
await ctx.plugin(AgentLoop, { agents: [] })
|
||||
await ctx.plugin(SubagentService)
|
||||
await ctx.plugin(spawn, { providerName: 'spawn' })
|
||||
await ctx.plugin(WorkerWorkflowEngine, {})
|
||||
await ctx.plugin(toolRalph, config)
|
||||
ctx.llm.registerAdapter(['mock'], adapter)
|
||||
const parentHandle = await ctx.agents.create({
|
||||
sessionId: SessionId('ralph-parent'),
|
||||
meta: { cwd: '/tmp/ralph-shared-workspace' },
|
||||
agentOptions: { provider: 'mock', model: 'mock' },
|
||||
})
|
||||
return { ctx, adapter, parentHandle, parent: parentHandle.agent }
|
||||
}
|
||||
|
||||
describe('dsh-tool-ralph over the real spawn and worker-thread stack', () => {
|
||||
it('uses distinct empty-seed children, shared cwd, and only the prior bounded handoff', async () => {
|
||||
const firstReport = {
|
||||
@@ -54,6 +76,8 @@ describe('dsh-tool-ralph over the real spawn and worker-thread stack', () => {
|
||||
await parent.whenIdle()
|
||||
|
||||
const children: Agent[] = []
|
||||
const phases: string[] = []
|
||||
ctx.on('workflow/phase', (_run, title) => { phases.push(title) })
|
||||
ctx.on('workflow/agent-start', (_run, child) => {
|
||||
const agent = ctx.agents.get(child.childId)
|
||||
expect(agent).toBeDefined()
|
||||
@@ -67,7 +91,9 @@ describe('dsh-tool-ralph over the real spawn and worker-thread stack', () => {
|
||||
})
|
||||
|
||||
expect(result.isError).toBe(false)
|
||||
expect((result.content[0] as { text: string }).text).toContain('Ralph completed after 2 rounds.')
|
||||
expect((result.content[0] as { text: string }).text)
|
||||
.toContain('Ralph worker reported completion after 2 rounds.')
|
||||
expect(phases).toEqual(['Fresh-agent rounds'])
|
||||
expect(children).toHaveLength(2)
|
||||
expect(new Set(children.map(child => child.id)).size).toBe(2)
|
||||
for (const child of children) {
|
||||
@@ -89,4 +115,151 @@ describe('dsh-tool-ralph over the real spawn and worker-thread stack', () => {
|
||||
|
||||
await parentHandle.dispose()
|
||||
})
|
||||
|
||||
it('reports the failed round and last good handoff when a child fails', async () => {
|
||||
const firstReport = {
|
||||
status: 'continue',
|
||||
summary: 'ROUND_ONE_HANDOFF',
|
||||
evidence: ['Created migration-a.ts.'],
|
||||
nextSteps: ['Finish migration-b.ts.'],
|
||||
blocker: '',
|
||||
}
|
||||
const { ctx, parent, parentHandle } = await mountRalph([
|
||||
toolCallResponse('round-1', STRUCTURED_OUTPUT_TOOL, firstReport),
|
||||
maxTokensResponse('unfinished child output'),
|
||||
], { maxRounds: 2 })
|
||||
const children: Agent[] = []
|
||||
ctx.on('workflow/agent-start', (_run, child) => {
|
||||
const agent = ctx.agents.get(child.childId)
|
||||
if (agent !== undefined) children.push(agent)
|
||||
})
|
||||
|
||||
const result = await ctx.tools.execute({
|
||||
callId: CallId('ralph-child-failure'),
|
||||
name: 'ralph',
|
||||
arguments: { objective: 'Complete both migration slices.', maxRounds: 2 },
|
||||
agent: parent,
|
||||
})
|
||||
|
||||
expect(result.isError).toBe(true)
|
||||
const text = (result.content[0] as { text: string }).text
|
||||
expect(text).toContain('Ralph round 2 child failed before producing a structured report.')
|
||||
expect(text).toContain('Last successful handoff:')
|
||||
expect(text).toContain('ROUND_ONE_HANDOFF')
|
||||
expect(children).toHaveLength(2)
|
||||
for (const child of children) expect(ctx.agents.get(child.id)).toBeUndefined()
|
||||
await parentHandle.dispose()
|
||||
})
|
||||
|
||||
it.each([
|
||||
{
|
||||
name: 'blocked',
|
||||
report: {
|
||||
status: 'blocked',
|
||||
summary: 'External authorization is required.',
|
||||
evidence: ['The local implementation is ready.'],
|
||||
nextSteps: ['Continue after authorization.'],
|
||||
blocker: 'The required external authorization is unavailable.',
|
||||
},
|
||||
config: { maxRounds: 2 },
|
||||
expectedError: false,
|
||||
expectedText: 'Ralph worker reported a blocker after 1 round.',
|
||||
},
|
||||
{
|
||||
name: 'budget-limited',
|
||||
report: {
|
||||
status: 'continue',
|
||||
summary: 'One slice is complete.',
|
||||
evidence: ['The first focused test passes.'],
|
||||
nextSteps: ['Implement the remaining slice.'],
|
||||
blocker: '',
|
||||
},
|
||||
config: { maxRounds: 1 },
|
||||
expectedError: false,
|
||||
expectedText: 'Ralph reached its 1 round limit; the worker reported work remaining.',
|
||||
},
|
||||
{
|
||||
name: 'unnormalized report',
|
||||
report: {
|
||||
status: 'continue',
|
||||
summary: ' padded summary ',
|
||||
evidence: ['A focused test passes.'],
|
||||
nextSteps: ['Continue implementation.'],
|
||||
blocker: '',
|
||||
},
|
||||
config: { maxRounds: 1 },
|
||||
expectedError: true,
|
||||
expectedText: 'summary must be non-empty and normalized',
|
||||
},
|
||||
{
|
||||
name: 'invalid continuing report',
|
||||
report: {
|
||||
status: 'continue',
|
||||
summary: 'Work remains.',
|
||||
evidence: ['A focused test passes.'],
|
||||
nextSteps: [],
|
||||
blocker: '',
|
||||
},
|
||||
config: { maxRounds: 1 },
|
||||
expectedError: true,
|
||||
expectedText: 'a continuing Ralph report needs nextSteps and an empty blocker',
|
||||
},
|
||||
{
|
||||
name: 'oversized report',
|
||||
report: {
|
||||
status: 'continue',
|
||||
summary: 'x'.repeat(300),
|
||||
evidence: ['A focused test passes.'],
|
||||
nextSteps: ['Continue implementation.'],
|
||||
blocker: '',
|
||||
},
|
||||
config: { maxRounds: 1, maxHandoffChars: 100 },
|
||||
expectedError: true,
|
||||
expectedText: 'Ralph round report exceeds maxHandoffChars',
|
||||
},
|
||||
])('enforces the fixed script for $name', async ({ report, config, expectedError, expectedText }) => {
|
||||
const { ctx, parent, parentHandle } = await mountRalph([
|
||||
toolCallResponse('round-report', STRUCTURED_OUTPUT_TOOL, report),
|
||||
], config)
|
||||
|
||||
const result = await ctx.tools.execute({
|
||||
callId: CallId('ralph-script-enforcement'),
|
||||
name: 'ralph',
|
||||
arguments: { objective: 'Complete the scoped work.', maxRounds: config.maxRounds },
|
||||
agent: parent,
|
||||
})
|
||||
|
||||
expect(result.isError).toBe(expectedError)
|
||||
expect((result.content[0] as { text: string }).text).toContain(expectedText)
|
||||
await parentHandle.dispose()
|
||||
})
|
||||
|
||||
it('cancels the real worker and fresh child to quiescence', async () => {
|
||||
const { ctx, parent, parentHandle } = await mountRalph(['hang'], { maxRounds: 2 })
|
||||
const children: Agent[] = []
|
||||
const outcomes: string[] = []
|
||||
ctx.on('workflow/agent-start', (_run, child) => {
|
||||
const agent = ctx.agents.get(child.childId)
|
||||
if (agent !== undefined) children.push(agent)
|
||||
})
|
||||
ctx.on('workflow/agent-end', (_run, child) => { outcomes.push(child.outcome) })
|
||||
const controller = new AbortController()
|
||||
const pending = ctx.tools.execute({
|
||||
callId: CallId('ralph-real-cancel'),
|
||||
name: 'ralph',
|
||||
arguments: { objective: 'Keep working until cancelled.', maxRounds: 2 },
|
||||
agent: parent,
|
||||
signal: controller.signal,
|
||||
})
|
||||
await vi.waitFor(() => { expect(children).toHaveLength(1) })
|
||||
|
||||
controller.abort()
|
||||
const result = await pending
|
||||
|
||||
expect(result.isError).toBe(true)
|
||||
expect((result.content[0] as { text: string }).text).toContain('Ralph workflow was cancelled')
|
||||
expect(outcomes).toEqual(['cancelled'])
|
||||
expect(ctx.agents.get(children[0]!.id)).toBeUndefined()
|
||||
await parentHandle.dispose()
|
||||
})
|
||||
})
|
||||
|
||||
@@ -82,6 +82,7 @@ async function setup(options?: SetupOptions) {
|
||||
if (options?.config?.subagentProvider !== undefined) config.subagentProvider = options.config.subagentProvider
|
||||
if (options?.config?.maxRounds !== undefined) config.maxRounds = options.config.maxRounds
|
||||
if (options?.config?.maxHandoffChars !== undefined) config.maxHandoffChars = options.config.maxHandoffChars
|
||||
if (options?.config?.maxResultChars !== undefined) config.maxResultChars = options.config.maxResultChars
|
||||
const fiber = await ctx.plugin(toolRalph, config)
|
||||
const parent = { id: SessionId('caller'), options: {} } as unknown as Agent
|
||||
return { ctx, engine: ctx.workflows as StubEngine, parent, fiber }
|
||||
@@ -145,6 +146,7 @@ describe('dsh-tool-ralph', () => {
|
||||
meta: { name: 'ralph-loop' },
|
||||
args: { objective: 'Finish the migration.', maxRounds: 4, maxHandoffChars: 9000 },
|
||||
subagentProvider: 'fresh',
|
||||
maxTotalAgents: 4,
|
||||
parent,
|
||||
})
|
||||
expect(engine.requests[0]!.script).toContain("status: 'budget-limited'")
|
||||
@@ -154,7 +156,8 @@ describe('dsh-tool-ralph', () => {
|
||||
report: COMPLETE,
|
||||
})
|
||||
expect(result.isError).toBe(false)
|
||||
expect((result.content[0] as { text: string }).text).toContain('Ralph completed after 1 round.')
|
||||
expect((result.content[0] as { text: string }).text)
|
||||
.toContain('Ralph worker reported completion after 1 round.')
|
||||
expect((result.content[0] as { text: string }).text).toContain('All required gates pass.')
|
||||
expect(engine.disposed).toBe(1)
|
||||
})
|
||||
@@ -167,7 +170,8 @@ describe('dsh-tool-ralph', () => {
|
||||
roundsStarted: 2,
|
||||
report: BLOCKED,
|
||||
}, 2)
|
||||
expect((blockedResult.content[0] as { text: string }).text).toContain('Ralph blocked after 2 rounds.')
|
||||
expect((blockedResult.content[0] as { text: string }).text)
|
||||
.toContain('Ralph worker reported a blocker after 2 rounds.')
|
||||
|
||||
const limited = execute(ctx, { objective: 'Ship it.' }, { agent: parent })
|
||||
await vi.waitFor(() => { expect(engine.requests).toHaveLength(2) })
|
||||
@@ -177,7 +181,54 @@ describe('dsh-tool-ralph', () => {
|
||||
report: CONTINUE,
|
||||
}, 2)
|
||||
expect((limitedResult.content[0] as { text: string }).text)
|
||||
.toContain('Ralph reached its 2 rounds limit with work remaining.')
|
||||
.toContain('Ralph reached its 2 rounds limit; the worker reported work remaining.')
|
||||
})
|
||||
|
||||
it('bounds the complete parent result and labels worker-reported completion', async () => {
|
||||
const { ctx, engine, parent } = await setup({ config: { maxResultChars: 160 } })
|
||||
const pending = execute(ctx, { objective: 'Ship it.' }, { agent: parent })
|
||||
const result = await settleCompleted(engine, pending, {
|
||||
status: 'complete',
|
||||
roundsStarted: 1,
|
||||
report: { ...COMPLETE, evidence: ['x'.repeat(500)] },
|
||||
})
|
||||
const text = (result.content[0] as { text: string }).text
|
||||
expect(text).toHaveLength(160)
|
||||
expect(text).toContain('Ralph worker reported completion after 1 round.')
|
||||
expect(text).toMatch(/… \[truncated\]$/)
|
||||
})
|
||||
|
||||
it('honors a result limit shorter than the truncation marker', async () => {
|
||||
const { ctx, engine, parent } = await setup({ config: { maxResultChars: 5 } })
|
||||
const result = await settleCompleted(engine, execute(ctx, { objective: 'Ship it.' }, { agent: parent }), {
|
||||
status: 'complete',
|
||||
roundsStarted: 1,
|
||||
report: COMPLETE,
|
||||
})
|
||||
expect((result.content[0] as { text: string }).text).toBe('\n… [t')
|
||||
})
|
||||
|
||||
it('reports an ordinary child failure with the failed round and last durable handoff', async () => {
|
||||
const { ctx, engine, parent } = await setup({ config: { maxRounds: 2 } })
|
||||
const first = execute(ctx, { objective: 'Ship it.', maxRounds: 2 }, { agent: parent })
|
||||
const firstResult = await settleCompleted(engine, first, {
|
||||
status: 'round-failed',
|
||||
roundsStarted: 1,
|
||||
lastReport: null,
|
||||
})
|
||||
expect(firstResult.isError).toBe(true)
|
||||
expect((firstResult.content[0] as { text: string }).text).toContain('Ralph round 1 child failed')
|
||||
expect((firstResult.content[0] as { text: string }).text).toContain('No previous handoff was available.')
|
||||
|
||||
const later = execute(ctx, { objective: 'Ship it.', maxRounds: 2 }, { agent: parent })
|
||||
const laterResult = await settleCompleted(engine, later, {
|
||||
status: 'round-failed',
|
||||
roundsStarted: 2,
|
||||
lastReport: CONTINUE,
|
||||
})
|
||||
expect(laterResult.isError).toBe(true)
|
||||
expect((laterResult.content[0] as { text: string }).text).toContain('Ralph round 2 child failed')
|
||||
expect((laterResult.content[0] as { text: string }).text).toContain('Implemented the first slice.')
|
||||
})
|
||||
|
||||
it('maps workflow error and cancellation reasons to tool errors and always disposes', async () => {
|
||||
@@ -251,6 +302,7 @@ describe('dsh-tool-ralph', () => {
|
||||
expect(() => { toolRalph.apply(new Context(), { subagentProvider: ' ' }) }).toThrow('non-empty normalized')
|
||||
expect(() => { toolRalph.apply(new Context(), { maxRounds: 0 }) }).toThrow('positive safe integer')
|
||||
expect(() => { toolRalph.apply(new Context(), { maxHandoffChars: 1.5 }) }).toThrow('positive safe integer')
|
||||
expect(() => { toolRalph.apply(new Context(), { maxResultChars: 0 }) }).toThrow('positive safe integer')
|
||||
})
|
||||
|
||||
it('turns malformed fixed-workflow terminal values and reports into errors', async () => {
|
||||
@@ -261,11 +313,18 @@ describe('dsh-tool-ralph', () => {
|
||||
{ value: { status: 'mystery', roundsStarted: 1, report: COMPLETE }, message: 'unknown terminal status' },
|
||||
{ value: { status: 'budget-limited', roundsStarted: 1, report: CONTINUE }, message: 'before the round limit', config: { maxRounds: 2 } },
|
||||
{ value: { status: 'complete', roundsStarted: 1, report: null }, message: 'malformed round report' },
|
||||
{ value: { status: 'complete', roundsStarted: 1, report: COMPLETE, extra: true }, message: 'malformed terminal result' },
|
||||
{ value: { status: 'blocked', roundsStarted: 1, report: BLOCKED, extra: true }, message: 'malformed terminal result' },
|
||||
{ value: { status: 'budget-limited', roundsStarted: 1, report: CONTINUE, extra: true }, message: 'malformed terminal result', config: { maxRounds: 1 } },
|
||||
{ value: { status: 'complete', roundsStarted: 1, report: { ...COMPLETE, status: 'continue' } }, message: 'malformed round report' },
|
||||
{ value: { status: 'budget-limited', roundsStarted: 1, report: { ...CONTINUE, nextSteps: [] } }, message: 'invalid continuing report', config: { maxRounds: 1 } },
|
||||
{ value: { status: 'complete', roundsStarted: 1, report: { ...COMPLETE, evidence: [] } }, message: 'invalid completion report' },
|
||||
{ value: { status: 'blocked', roundsStarted: 1, report: { ...BLOCKED, blocker: '' } }, message: 'invalid blocked report' },
|
||||
{ value: { status: 'complete', roundsStarted: 1, report: { ...COMPLETE, summary: 'x'.repeat(500) } }, message: 'oversized handoff', config: { maxHandoffChars: 100 } },
|
||||
{ value: { status: 'round-failed', roundsStarted: 1 }, message: 'malformed terminal result' },
|
||||
{ value: { status: 'round-failed', roundsStarted: 1, lastReport: CONTINUE }, message: 'invalid first-round failure' },
|
||||
{ value: { status: 'round-failed', roundsStarted: 2, lastReport: null }, message: 'without its last handoff', config: { maxRounds: 2 } },
|
||||
{ value: { status: 'round-failed', roundsStarted: 2, lastReport: { ...CONTINUE, nextSteps: [] } }, message: 'invalid continuing report', config: { maxRounds: 2 } },
|
||||
]
|
||||
for (const testCase of cases) {
|
||||
const { ctx, engine, parent } = await setup(
|
||||
@@ -294,7 +353,9 @@ describe('dsh-tool-ralph', () => {
|
||||
const { ctx, fiber } = await setup()
|
||||
const section = (await ctx.systemPrompt.assemble()).sections.find(candidate => candidate.name === 'tool:ralph')
|
||||
expect(section?.text).toContain('ONLY when the direct human explicitly asks')
|
||||
expect(section?.text).toContain('worker reports, not independent evaluation')
|
||||
const tool = ctx.tools.get('ralph')!
|
||||
expect(tool.description).toContain('worker reports completion')
|
||||
expect(tool.presentCall!({ objective: 'Finish it.' })).toEqual({
|
||||
card: 'generic',
|
||||
title: 'ralph',
|
||||
|
||||
Reference in New Issue
Block a user