Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(goal): require explained model blockers

Browse Source
This commit is contained in:
Tianyi Cui
2026-07-20 16:53:10 +08:00
parent 97e144fff6
commit 15640ca697
17 changed files with 289 additions and 287 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
packages/goal/tool-goal/README.md
View File

@@ -4,9 +4,9 @@ The model-facing control surface for [`ctx.goals`](../goal/README.md): `get_goal
## Tools
- `get_goal()` returns the current goal or `null`, including the compare-and-set id/revision, durable phase, admitted/capped goal rounds, and current process-local activation.
- `get_goal()` returns the current goal or `null`, including the compare-and-set id/revision, durable phase, admitted/capped goal rounds, any blocker reason, and current process-local activation.
- `create_goal(objective, max_goal_rounds?)` creates one goal from a direct top-level human turn. The model may infer long-running goal intent without an exact command phrase; non-human turns and subagents are rejected at execution.
- `update_goal(goal_id, revision, action, objective?, max_goal_rounds?)` supports `edit`, `pause`, `resume`, `complete`, and `blocked`. Replacements belong only to `edit`.
- `update_goal(goal_id, revision, action, objective?, max_goal_rounds?, blocked_reason?)` supports `edit`, `pause`, `resume`, `complete`, and `blocked`. Replacements belong only to `edit`; `blocked_reason` is required only for `blocked` and is persisted with the stable code `model-reported`.
All calls are exclusive, so a model-ordered batch observes earlier mutations and their new revisions. ACP and other clients receive pure generic cards: read for `get_goal`, other for mutations.
@@ -18,7 +18,7 @@ Execution requires the exact live `exec.agent`, its inherited `AgentRegistry` in
`{ kind: 'user' }` is a host attestation. `Agent.send()` and `steer()` assign it when their caller omits a source, so plugins, schedulers, and other non-human producers must pass their own source rather than inheriting human authority.
Complete and blocked also accept the exact current goal round: a goal-sourced `user/message` whose id, revision, and round equal the folded current goal. A goal-round blocked call is mechanically rejected until `blockedAfterConsecutiveRounds`; the model judges whether the same condition actually persisted. Direct human authority may stop a goal immediately.
Complete and blocked also accept the exact current goal round: a goal-sourced `user/message` whose id, revision, and round equal the folded current goal. A goal-round blocked call is mechanically rejected until `blockedAfterConsecutiveRounds`; the model judges whether the same condition actually persisted and must describe it in `blocked_reason`. Direct human authority may stop a goal immediately.
## Config
@@ -42,7 +42,7 @@ A fixed goal policy says when semantic human intent warrants creation, requires
##### Goal policy
```markdown
Use goal tools for one long-running completion objective in the current session. create_goal may infer goal intent from a direct human request in any language; do not create a goal for routine single-turn work. Call get_goal before update_goal and copy its exact goal_id and revision. After session resume or fork, an active goal is disarmed: when a human asks to continue or resume in any wording or language, use update_goal action resume to rearm it. Mark complete only when the objective is actually achieved. Mark blocked only after the same blocking condition persists for at least 3 consecutive goal rounds; difficulty, uncertainty, or useful remaining work is not blocked.
Use goal tools for one long-running completion objective in the current session. create_goal may infer goal intent from a direct human request in any language; do not create a goal for routine single-turn work. Call get_goal before update_goal and copy its exact goal_id and revision. After session resume or fork, an active goal is disarmed: when a human asks to continue or resume in any wording or language, use update_goal action resume to rearm it. Mark complete only when the objective is actually achieved. Mark blocked only after the same blocking condition persists for at least 3 consecutive goal rounds, and report that concrete condition in blocked_reason; difficulty, uncertainty, or useful remaining work is not blocked.
```
#### Token effect

34
packages/goal/tool-goal/src/index.ts
View File

@@ -51,7 +51,8 @@ const CREATE_DESCRIPTION =
const GET_DESCRIPTION =
'Read the current same-session goal, including its exact id/revision, objective, phase, completed '
+ 'continuation rounds, round limit, and whether another continuation is armed. Call this before updating a goal.'
+ 'continuation rounds, round limit, blocker reason when present, and whether another continuation is armed. '
+ 'Call this before updating a goal.'
/** Render policy guidance with its deployment-selected blocked threshold. */
function guidance(blockedAfter: number): string {
@@ -62,7 +63,8 @@ function guidance(blockedAfter: number): string {
+ 'a human asks to continue or resume in any wording or language, use update_goal action '
+ 'resume to rearm it. Mark complete only when the objective is actually achieved. Mark '
+ `blocked only after the same blocking condition persists for at least ${blockedAfter} `
+ 'consecutive goal rounds; difficulty, uncertainty, or useful remaining work is not blocked.'
+ 'consecutive goal rounds, and report that concrete condition in blocked_reason; difficulty, uncertainty, '
+ 'or useful remaining work is not blocked.'
}
/** Validate config even when apply is called directly outside Loader normalization. */
@@ -97,6 +99,7 @@ function renderGoal(goal: GoalView | undefined): string {
phase: goal.phase,
roundsStarted: goal.roundsStarted,
maxGoalRounds: goal.maxGoalRounds,
...goal.blockedReason === undefined ? {} : { blockedReason: goal.blockedReason },
},
activation: goal.activation,
})
@@ -183,7 +186,7 @@ export function apply(ctx: Context, config: Config): void {
description: 'Update the exact current goal revision. edit, pause, and resume require a direct '
+ 'top-level human request. During an automatic continuation of the current goal, complete '
+ 'and blocked are also allowed. blocked is rejected before the configured minimum round count; the model remains '
+ 'responsible for judging that the same condition persisted across those rounds.',
+ 'responsible for judging that the same condition persisted across those rounds and must explain it in blocked_reason.',
parameters: {
goal_id: { type: 'string', required: true, description: 'Exact id returned by get_goal.' },
revision: { type: 'number', required: true, description: 'Exact positive revision returned by get_goal.' },
@@ -195,6 +198,10 @@ export function apply(ctx: Context, config: Config): void {
},
objective: { type: 'string', description: 'Replacement objective; valid only with action edit.' },
max_goal_rounds: { type: 'number', description: 'Replacement cap; valid only with action edit.' },
blocked_reason: {
type: 'string',
description: 'Concrete blocking condition; required only with action blocked.',
},
},
execute(args, exec) {
const execution = goalToolExecution(ctx, exec)
@@ -205,6 +212,9 @@ export function apply(ctx: Context, config: Config): void {
}
if (args.action === 'edit') {
requireDirectHuman(ctx, execution)
if (args.blocked_reason !== undefined) {
throw new HarnessError('blocked_reason is valid only with action blocked', 'GOAL_TOOL_INVALID_UPDATE')
}
const goal = ctx.goals.edit(execution.agent, ref, replacements)
observeMutation(terminalTurns, execution, false)
return Promise.resolve([{
@@ -214,9 +224,9 @@ export function apply(ctx: Context, config: Config): void {
}
if (args.action === 'pause' || args.action === 'resume') {
requireDirectHuman(ctx, execution)
if (args.objective !== undefined || args.max_goal_rounds !== undefined) {
if (args.objective !== undefined || args.max_goal_rounds !== undefined || args.blocked_reason !== undefined) {
throw new HarnessError(
'objective and max_goal_rounds are valid only with action edit',
'objective and max_goal_rounds are valid only with action edit; blocked_reason is valid only with action blocked',
'GOAL_TOOL_INVALID_UPDATE',
)
}
@@ -233,6 +243,13 @@ export function apply(ctx: Context, config: Config): void {
'GOAL_TOOL_INVALID_UPDATE',
)
}
if (args.action === 'complete' && args.blocked_reason !== undefined) {
throw new HarnessError('blocked_reason is valid only with action blocked', 'GOAL_TOOL_INVALID_UPDATE')
}
if (args.action === 'blocked'
&& (args.blocked_reason === undefined || args.blocked_reason.trim().length === 0)) {
throw new HarnessError('blocked_reason is required with action blocked', 'GOAL_TOOL_INVALID_UPDATE')
}
if (args.action === 'blocked' && authority.kind === 'goal-round'
&& authority.goal.roundsStarted < resolved.blockedAfterConsecutiveRounds) {
throw new HarnessError(
@@ -243,14 +260,17 @@ export function apply(ctx: Context, config: Config): void {
}
const goal = args.action === 'complete'
? ctx.goals.complete(execution.agent, ref)
: ctx.goals.block(execution.agent, ref)
: ctx.goals.block(execution.agent, ref, {
code: 'model-reported',
message: args.blocked_reason as string,
})
observeMutation(terminalTurns, execution, authority.kind === 'goal-round')
return Promise.resolve([{ type: 'text', text: renderGoal(goal) }])
},
presentCall: args => present(
`${args.action === 'blocked' ? 'Mark' : args.action.charAt(0).toUpperCase() + args.action.slice(1)} goal`,
'other',
args.objective ?? args.goal_id,
args.blocked_reason ?? args.objective ?? args.goal_id,
),
}))
}

127
packages/goal/tool-goal/tests/tool-goal.e2e.ts
View File

@@ -1,127 +0,0 @@
import { spawn, type ChildProcessWithoutNullStreams } from 'node:child_process'
import { mkdtemp, readFile, readdir, rm } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { fileURLToPath } from 'node:url'
import { afterEach, describe, expect, it } from 'vitest'
import { decodeGoalChange } from '@deepseek-ai/dsh-goal'
import type { SessionEvent } from '@deepseek-ai/dsh-session'
import { resolveExampleLaunch } from '@deepseek-ai/dsh-loader-smoke'
const binScript = fileURLToPath(new URL('../../../examples/stdio-demo/src/bin.ts', import.meta.url))
const configPath = fileURLToPath(new URL(
'../../../../examples/echo-agent/tests/fixtures/goal/tool-goal/cordis.yml',
import.meta.url,
))
const repoTsconfig = fileURLToPath(new URL('../../../../tsconfig.json', import.meta.url))
const PROCESS_TIMEOUT_MS = 30_000
const TEST_TIMEOUT_MS = PROCESS_TIMEOUT_MS + 15_000
const PAUSED_RESULT = '"phase":"paused"'
let child: ChildProcessWithoutNullStreams | undefined
let workdir: string | undefined
afterEach(async () => {
if (child !== undefined && child.exitCode === null) child.kill('SIGKILL')
child = undefined
if (workdir !== undefined) await rm(workdir, { recursive: true, force: true })
workdir = undefined
})
async function jsonlFiles(dir: string): Promise<string[]> {
const entries = await readdir(dir, { withFileTypes: true })
const paths = await Promise.all(entries.map(async (entry) => {
const path = join(dir, entry.name)
if (entry.isDirectory()) return jsonlFiles(path)
return entry.isFile() && entry.name.endsWith('.jsonl') ? [path] : []
}))
return paths.flat()
}
async function runComposition(): Promise<{ stdout: string; stderr: string }> {
workdir = await mkdtemp(join(tmpdir(), 'goal-tools-e2e-'))
const cwd = workdir
return new Promise((resolve, reject) => {
const launch = resolveExampleLaunch({
srcBin: binScript,
configArgs: [configPath],
tsconfigPath: repoTsconfig,
exposeInternals: true,
env: {
DSH_HOME: join(cwd, '.dsh'),
DSH_AGENTS_HOME: join(cwd, '.agents'),
},
})
const proc = spawn(launch.command, launch.args, {
cwd,
env: { ...process.env, ...launch.env },
stdio: ['pipe', 'pipe', 'pipe'],
})
child = proc
let stdout = ''
let stderr = ''
let pauseSent = false
let inputClosed = false
proc.stdout.setEncoding('utf8')
proc.stdout.on('data', (chunk: string) => {
stdout += chunk
if (!pauseSent && stdout.includes('GOAL CREATED') && stdout.includes('\n> ')) {
pauseSent = true
proc.stdin.write('pause\n')
}
const pausedAt = stdout.indexOf(PAUSED_RESULT)
if (!inputClosed && pausedAt >= 0 && stdout.indexOf('\n> ', pausedAt) >= 0) {
inputClosed = true
proc.stdin.end()
}
})
proc.stderr.setEncoding('utf8')
proc.stderr.on('data', (chunk: string) => { stderr += chunk })
const timer = setTimeout(() => {
proc.kill('SIGKILL')
reject(new Error(
`goal-tools e2e did not exit within ${PROCESS_TIMEOUT_MS / 1_000}s. stdout:\n${stdout}\nstderr:\n${stderr}`,
))
}, PROCESS_TIMEOUT_MS)
proc.on('exit', (code) => {
clearTimeout(timer)
if (code === 0) resolve({ stdout, stderr })
else reject(new Error(`goal-tools e2e exited ${String(code)}. stdout:\n${stdout}\nstderr:\n${stderr}`))
})
proc.on('error', (error) => { clearTimeout(timer); reject(error) })
proc.stdin.write('start\n')
})
}
describe('goal tools through a real Loader, app, and stdio process', () => {
it('creates, reads, and pauses one root goal with durable tool and state records', async () => {
const { stdout, stderr } = await runComposition()
expect(stderr).not.toContain('UNHANDLED')
expect(stdout).toContain('goal-tools e2e ready.')
expect(stdout).toContain('GOAL CREATED')
expect(stdout).toContain(PAUSED_RESULT)
expect(stdout).toContain('GOAL PAUSED')
const logs = await jsonlFiles(join(workdir as string, '.sessions'))
expect(logs).toHaveLength(1)
const lines = (await readFile(logs[0] as string, 'utf8')).trimEnd().split('\n')
const events = lines.slice(1).map(line => JSON.parse(line) as SessionEvent)
const calls = events.filter(event => event.type === 'tool/call')
expect(calls.map(event => event.data.name)).toEqual(['create_goal', 'get_goal', 'update_goal'])
const results = events.filter(event => event.type === 'tool/result')
expect(results).toHaveLength(3)
expect(results.every(event => !event.data.isError)).toBe(true)
const changes = events
.filter(event => event.type === 'context/message' && event.data.source.kind === 'goal')
.map(event => event.type === 'context/message' ? decodeGoalChange(event.data.meta) : undefined)
expect(changes.map(change => change?.operation)).toEqual(['create', 'pause'])
expect(changes[1]).toMatchObject({ goal: { phase: 'paused', revision: 2, maxGoalRounds: 7 } })
expect(JSON.stringify(changes)).not.toContain('activation')
const headers = events.filter(event => event.type === 'request/header')
expect(JSON.stringify(headers)).toContain('infer goal intent')
expect(JSON.stringify(headers)).toContain('at least 3 consecutive goal rounds')
}, TEST_TIMEOUT_MS)
})

54
packages/goal/tool-goal/tests/tool-goal.spec.ts
View File

@@ -135,8 +135,8 @@ describe('goal tool registration and presentation', () => {
card: 'generic', title: 'Create goal', kind: 'other', rawInput: 'ship',
})
expect(ctx.tools.get('update_goal')?.presentCall?.({
goal_id: 'goal-1', revision: 2, action: 'blocked',
})).toEqual({ card: 'generic', title: 'Mark goal', kind: 'other', rawInput: 'goal-1' })
goal_id: 'goal-1', revision: 2, action: 'blocked', blocked_reason: 'Waiting for a human choice.',
})).toEqual({ card: 'generic', title: 'Mark goal', kind: 'other', rawInput: 'Waiting for a human choice.' })
expect(ctx.tools.get('update_goal')?.presentCall?.({
goal_id: 'goal-1', revision: 2, action: 'resume',
})).toEqual({ card: 'generic', title: 'Resume goal', kind: 'other', rawInput: 'goal-1' })
@@ -391,6 +391,26 @@ describe('goal tool state transitions', () => {
max_goal_rounds: 2,
}, root.agent)
expect(terminalUpdate.error?.code).toBe('GOAL_TOOL_INVALID_UPDATE')
const blockedWithoutReason = await execute(ctx, 'update_goal', {
goal_id: created.id, revision: created.revision, action: 'blocked',
}, root.agent)
expect(blockedWithoutReason.error?.code).toBe('GOAL_TOOL_INVALID_UPDATE')
const blockedWithEmptyReason = await execute(ctx, 'update_goal', {
goal_id: created.id, revision: created.revision, action: 'blocked', blocked_reason: ' ',
}, root.agent)
expect(blockedWithEmptyReason.error?.code).toBe('GOAL_TOOL_INVALID_UPDATE')
const completeWithReason = await execute(ctx, 'update_goal', {
goal_id: created.id, revision: created.revision, action: 'complete', blocked_reason: 'Not a blocker.',
}, root.agent)
expect(completeWithReason.error?.code).toBe('GOAL_TOOL_INVALID_UPDATE')
const editWithReason = await execute(ctx, 'update_goal', {
goal_id: created.id,
revision: created.revision,
action: 'edit',
objective: 'still valid',
blocked_reason: 'Not valid for edit.',
}, root.agent)
expect(editWithReason.error?.code).toBe('GOAL_TOOL_INVALID_UPDATE')
const malformedRef = await execute(ctx, 'update_goal', {
goal_id: '', revision: 0, action: 'edit', objective: 'x',
}, root.agent)
@@ -423,16 +443,26 @@ describe('goal tool state transitions', () => {
for (let round = 1; round <= 2; round += 1) {
turn = openTurn(root, { kind: 'goal', goalId: ref.id, revision: ref.revision, round })
const result = await execute(ctx, 'update_goal', {
goal_id: ref.id, revision: ref.revision, action: 'blocked',
goal_id: ref.id,
revision: ref.revision,
action: 'blocked',
blocked_reason: 'The required credential is still unavailable.',
}, root.agent)
expect(result.error?.code).toBe('GOAL_TOOL_BLOCK_THRESHOLD')
closeTurn(root, turn)
}
openTurn(root, { kind: 'goal', goalId: ref.id, revision: ref.revision, round: 3 })
const blocked = await execute(ctx, 'update_goal', {
goal_id: ref.id, revision: ref.revision, action: 'blocked',
goal_id: ref.id,
revision: ref.revision,
action: 'blocked',
blocked_reason: 'The required credential is still unavailable.',
}, root.agent)
expect(resultGoal(blocked)).toMatchObject({ phase: 'blocked', roundsStarted: 3 })
expect(resultGoal(blocked)).toMatchObject({
phase: 'blocked',
blockedReason: { code: 'model-reported', message: 'The required credential is still unavailable.' },
roundsStarted: 3,
})
})
it('lets direct human authority block before the model threshold', async () => {
@@ -440,8 +470,18 @@ describe('goal tool state transitions', () => {
openTurn(root, { kind: 'user' })
const created = ctx.goals.create(root.agent, { objective: 'human stop' })
const blocked = await execute(ctx, 'update_goal', {
goal_id: created.id, revision: created.revision, action: 'blocked',
goal_id: created.id,
revision: created.revision,
action: 'blocked',
blocked_reason: 'The user asked to stop until a prerequisite is available.',
}, root.agent)
expect(resultGoal(blocked)).toMatchObject({ phase: 'blocked', roundsStarted: 0 })
expect(resultGoal(blocked)).toMatchObject({
phase: 'blocked',
blockedReason: {
code: 'model-reported',
message: 'The user asked to stop until a prerequisite is available.',
},
roundsStarted: 0,
})
})
})