Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test: real-Loader dynamic composition, keyless onboarding snapshot, and .env-only e2e

Browse Source 
llm-deepseek gains a Loader+Include composition spec proving external
settings.yaml/.env edits reach the very next request, and a real-API e2e
where only a credentials-local document holds the key. The headless example
pins the first-run missing-credential UX as a keyless stream-json snapshot
(new credentials.cordis.snapshot.yml scenario); runLoaderSmoke learns
expectedExitCode so a designed failure surface can be pinned instead of
masked.
This commit is contained in:
Yichen Jiang
2026-07-29 13:44:24 +08:00
parent c0426142c5
commit d77db29f01
9 changed files with 290 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files

33
examples/headless-agent/tests/headless.snapshot.ts
View File

@@ -27,6 +27,8 @@ const goalScenarioDir = join(snapshotsDir, 'goal-tools')
const goalConfigPath = fileURLToPath(new URL('../goal.cordis.snapshot.yml', import.meta.url))
const retryScenarioDir = join(snapshotsDir, 'provider-retry')
const retryConfigPath = fileURLToPath(new URL('../retry.cordis.snapshot.yml', import.meta.url))
const credentialsScenarioDir = join(snapshotsDir, 'missing-credential')
const credentialsConfigPath = fileURLToPath(new URL('../credentials.cordis.snapshot.yml', import.meta.url))
const ralphScenarioDir = join(snapshotsDir, 'ralph-loop')
const ralphConfigPath = fileURLToPath(new URL('../ralph.cordis.snapshot.yml', import.meta.url))
const binScript = fileURLToPath(new URL('../../../packages/examples/cli-demo/src/bin.ts', import.meta.url))
@@ -168,6 +170,37 @@ describe('headless stream-json snapshots', () => {
expect(normalized).toBe(await readFile(streamExpected, 'utf8'))
}, LOADER_SMOKE_TEST_TIMEOUT_MS)
it('surfaces actionable missing-credential guidance through the one-shot app', async () => {
const streamExpected = join(credentialsScenarioDir, 'stream-json.expected.jsonl')
let runCwd = ''
const result = await runLoaderSmoke({
label: 'missing-credential headless stream-json snapshot',
tempDirPrefix: 'headless-snapshot-missing-credential-',
binScript,
configPath: credentialsConfigPath,
binArgs: ['--config', credentialsConfigPath, '--output-format', 'stream-json', 'say pong'],
tsconfigPath,
env: {
// First-run posture: no key in the environment, none under ./.dsh.
DEEPSEEK_API_KEY: '',
DEEPSEEK_BASE_URL: '',
NODE_OPTIONS: [process.env.NODE_OPTIONS, '--disable-warning=ExperimentalWarning'].filter(Boolean).join(' '),
},
// The designed failure surface: the one-shot app reports the failed turn.
expectedExitCode: 1,
prepare: (cwd) => { runCwd = cwd },
})
expect(result.stderr).toBe(
'dsh-cli-demo: turn 1 failed at step 1: llm-deepseek: no API key for provider route "deepseek";'
+ ' set the llm-deepseek "apiKey" setting, store DEEPSEEK_API_KEY with the credentials service,'
+ ' or export DEEPSEEK_API_KEY\n',
)
const normalized = normalizeHeadlessStream(result.stdout, runCwd)
if (refreshing) await writeFile(streamExpected, normalized)
expect(normalized).toBe(await readFile(streamExpected, 'utf8'))
}, LOADER_SMOKE_TEST_TIMEOUT_MS)
it('logs the model default and a dynamic next-step reasoning effort', async () => {
const result = await runLoaderSmoke({
label: 'reasoning effort headless stream-json snapshot',