Merge origin/master into codex/truncated-design

This commit is contained in:
Dudu-0223
2026-07-17 18:21:54 +08:00
1057 changed files with 42961 additions and 18526 deletions

View File

@@ -32,4 +32,54 @@ Each tool is registered independently; a product that wants only one disables th
Tool registration follows product **enablement**, not backend availability. A tool stays visible even when its selected provider is missing, misconfigured, ambiguous, or temporarily unavailable; the seam resolves the provider at execution time and execution fails with a structured `WebError` (e.g. `WEB_PROVIDER_UNAVAILABLE`, `WEB_PROVIDER_AMBIGUOUS`), which `ToolRegistry.execute()` turns into an error tool result the model can read and hooks/UI can route on. This keeps the model schema stable without making plugin load order, credential state, or HMR timing part of the model-facing contract. To remove a web tool entirely, disable it here in config.
The tool never calls a provider's `status()` and never enumerates providers — its only execution path is `ctx.web.search()` / `ctx.web.fetch()`, and provider unavailability reaches it as the structured `WebError` codes selection throws at execution time. Provider selection stays entirely inside the seam, with one owner.
The tool never calls a provider's `available()` and never enumerates providers — its only execution path is `ctx.web.search()` / `ctx.web.fetch()`, and provider unavailability reaches it as the structured `WebError` codes selection throws at execution time. Provider selection stays entirely inside the seam, with one owner.
## Model Experience
### System prompt
**What the model sees**: Search and fetch contribute the web-search and web-fetch guidance below. A scoped tool restriction does not remove these independently registered sections.
**Token effect**: Fixed guidance cost per request for each config-enabled tool, even when a restriction hides its schema.
#### Web search guidance
```markdown
Use the web_search tool to discover current information on the web. It returns an optional answer plus a list of source URLs. Follow up with web_fetch when you need the full content of a specific result, and cite the relevant URLs as markdown links.
```
#### Web fetch guidance
```markdown
Use the web_fetch tool to retrieve the content of a specific HTTP(S) URL (for example a result from web_search). It returns the page content decoded to text. Cite the URL as a markdown link when you use its content.
```
### Tool schemas
**What the model sees**: The model sees the generated [`web_search` and `web_fetch` schemas](../../../docs/tool-catalog.md#deepseek-aidsh-tool-web). Result-count and timeout budgets are deployment settings, not model arguments.
**Token effect**: Fixed schema cost per request; config disablement removes both schema and guidance, while a scoped restriction removes only the schema.
### Search result
**What the model sees**: The optional provider-owned answer is followed by `Sources:` and data-dependent lines shaped exactly `- [<title-or-url>](<url>)`, optionally suffixed ` — <snippet> (<publishedAt>)`. With neither answer nor sources the result says `No results found.` A capped list adds `(Showing the first <count> sources. Refine the query for more.)`; every result ends `Cite the relevant URLs above as markdown links in your answer.`
**Token effect**: Data-dependent results are resent until compaction and sources are capped by `searchMaxResults`.
### Fetch result
**What the model sees**: A successful fetch is exactly `Fetched <finalUrl> (HTTP <statusCode>)`, a blank line, and the provider-owned decoded body. Truncation adds a blank line and `(Content truncated. Fetch a more specific URL or section for the full text.)`; failures become `Error: <message>`. Queries and URLs remain in call history.
**Token effect**: Provider caps bound body size; retained call arguments and results are resent until compaction, and timeout policy can replace a late result with a short error.
### Argument errors
**What the model sees**: Blank inputs become exactly `Error: query must be a non-empty string` or `Error: url must be a non-empty string`.
**Token effect**: Only the failing call adds these retained tokens.
## Known Limitations and Deferred Work
- **`htmlToMarkdown` is a minimal regex converter, not an HTML parser** — it strips script/style/noscript, keeps headings/bullets/links, and decodes about a dozen named entities; tables, images, and nested formatting are lost.
- **The model-facing surface is minimal by design, with promotions deferred** — `max_results` stays a config bound (not a model argument), and `web_fetch` takes only `url` (no `format`/`prompt`/LLM-summarization mode); both are named later steps in [the seam RFC](../../../docs/rfc/implemented/architecture/2026-06-24-web-capability-seam.md).
- **No web-specific permission policy** — both tools execute without requesting `ctx.approval`; a deployment that needs confirmation must add a `tools/pre-execute` policy, and the package does not define persistent URL/domain grants.

View File

@@ -26,7 +26,7 @@
"@deepseek-ai/dsh-system-prompt": "^0.0.1",
"@deepseek-ai/dsh-tools": "^0.0.1",
"@deepseek-ai/dsh-web": "^0.0.1",
"cordis": "^4.0.0-rc.6"
"cordis": "^4.0.0-rc.7"
},
"dependencies": {
"schemastery": "^3.18.0"
@@ -43,6 +43,6 @@
"@deepseek-ai/dsh-web": "workspace:^",
"@deepseek-ai/dsh-web-fetch-local": "workspace:^",
"@deepseek-ai/dsh-web-search-exa": "workspace:^",
"cordis": "^4.0.0-rc.6"
"cordis": "^4.0.0-rc.7"
}
}

View File

@@ -1,15 +1,8 @@
/**
* The model-facing `web_fetch` tool: retrieve the content of a specific URL.
* Execution goes through `ctx.web` — this module owns the model-facing schema,
* argument validation, and PRESENTATION (HTML→markdown, truncation formatting),
* while the fetch provider owns safe retrieval (transport, redirects, caps).
*
* The model-facing schema exposes NO timeout knob: the tool-call budget is
* deployment policy DECLARED via this package's `fetchTimeoutMs` config (attached
* as `ToolDefinition.timeoutMs`) and ENFORCED by `@deepseek-ai/dsh-timeout-policy`
* (a `tools/execute` wrapper), matching the reference-agent `WebFetch` shape. This
* tool just forwards the (possibly deadline-derived) `exec.signal` to `ctx.web`;
* the provider keeps its own timeout only as a resource backstop for direct callers.
* The model-facing `web_fetch` tool. This module owns its schema, validation, and presentation;
* `ctx.web` owns retrieval. Timeout is deployment policy, not a model argument: config becomes
* `ToolDefinition.timeoutMs`, timeout policy enforces it, and this tool forwards the resulting
* signal. A provider timeout remains a backstop for direct seam callers.
*/
import type { Context } from 'cordis'
@@ -103,7 +96,7 @@ export function applyWebFetchTool(ctx: Context, timeoutMs: number): void {
const input = parseFetchArgs(args)
const result = await ctx.web.fetch(
{ url: input.url },
exec.signal ? { signal: exec.signal } : undefined,
exec.signal,
)
return [{ type: 'text', text: formatFetchOutput(result) }]
},

View File

@@ -1,11 +1,8 @@
/**
* Minimal, dependency-free HTML→markdown-ish text conversion for `web_fetch`
* presentation. This is intentionally NOT a full HTML parser: it strips
* script/style/noscript, drops tags, decodes the common named/numeric entities,
* and collapses whitespace into a readable plain-text approximation with a few
* markdown affordances (headings, list bullets, links). A heavier converter can
* replace this without touching the seam or the tool schema.
*
* Minimal dependency-free HTML-to-readable-text conversion for `web_fetch`, not a full parser. It
* removes non-content elements and tags, decodes common entities, collapses whitespace, and keeps
* basic headings, lists, and links. A richer converter can replace it without changing the seam or
* tool schema.
* @module @deepseek-ai/dsh-tool-web/html
*/

View File

@@ -1,19 +1,8 @@
/**
* The model-facing web tool suite (`web_search`, `web_fetch`) over the `ctx.web`
* seam. This root plugin registers the tools the product has ENABLED, composing
* the per-tool registration helpers (`applyWebSearchTool`, `applyWebFetchTool`).
*
* The package owns model-facing concerns only — tool names, JSON schemas,
* argument validation, prompt sections, result-cap constants, result formatting,
* HTML→markdown presentation. All web access goes through `ctx.web`; this
* package never imports a concrete provider package.
*
* Tool registration follows product/app ENABLEMENT, not backend availability: a
* tool stays visible even when its selected provider is missing/misconfigured,
* and execution fails with a structured `WebError` (resolved by the seam at call
* time). That keeps the model schema stable without making plugin load order,
* credential state, or HMR timing part of the model-facing contract.
*
* Model-facing `web_search` and `web_fetch` tools over `ctx.web`. This package owns schemas,
* validation, prompt guidance, limits, and presentation, never concrete providers. Enablement
* controls tool registration; an enabled tool remains visible when its provider is unavailable
* and fails with a structured error at execution time.
* @module @deepseek-ai/dsh-tool-web
*/

View File

@@ -113,7 +113,7 @@ export function applyWebSearchTool(ctx: Context, maxResults: number, timeoutMs:
const input = parseSearchArgs(args)
const result = await ctx.web.search(
{ query: input.query, maxResults },
exec.signal ? { signal: exec.signal } : undefined,
exec.signal,
)
return [{ type: 'text', text: formatSearchOutput(result) }]
},

View File

@@ -1,11 +1,9 @@
/**
* Integration: the real fetch backend (`dsh-web-fetch-local`) + a real search
* provider (`dsh-web-search-exa`) + the real seam (`dsh-web`) + the model tool
* (`dsh-tool-web`) + the tool-call timeout policy (`dsh-timeout-policy`),
* exercised through `ctx.tools.execute()` — nothing bypasses the tool registry.
* Fetch hits a real loopback HTTP server (verifying the WORLD); search runs the
* real Exa provider over a stubbed global `fetch` (the network is the one
* boundary we mock).
* Integration: the real fetch backend (`dsh-web-fetch-local`) + a real search provider
* (`dsh-web-search-exa`) + the real seam (`dsh-web`) + the model tool (`dsh-tool-web`) + the
* tool-call timeout policy (`dsh-timeout-policy`), exercised through `ctx.tools.execute()` —
* nothing bypasses the tool registry. Fetch verifies world effects against loopback HTTP; search
* uses the real Exa provider with only its network boundary stubbed.
*/
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
@@ -136,7 +134,7 @@ describe('tool-call timeout returns TOOL_TIMEOUT (deadline wins over a slow fetc
await tctx.plugin(ToolRegistry)
await tctx.plugin(WebService, { fetchProvider: WebFetchLocal.LOCAL_FETCH_PROVIDER_ID })
// Provider backstop well ABOVE the tool-call budget, so the policy wins.
await tctx.plugin(WebFetchLocal, { timeoutMs: 30_000, maxTimeoutMs: 60_000 })
await tctx.plugin(WebFetchLocal, { timeoutMs: 30_000 })
await tctx.plugin(TimeoutPolicy)
// The tool-call budget is declared by tool-web config, enforced by the policy.
tfiber = await tctx.plugin(ToolWeb, { fetchTimeoutMs: 50 })
@@ -158,12 +156,18 @@ describe('tool-call timeout returns TOOL_TIMEOUT (deadline wins over a slow fetc
expect(text).toContain('timed out after 50ms')
})
it('the provider backstop still protects a DIRECT ctx.web.fetch() call (no tool-call policy in that path)', async () => {
// A direct seam caller does not go through tools/execute, so the tool-call
// policy never applies; the provider's OWN timeout is the only budget. A
// short per-request hint proves the provider backstop is intact and classifies
// as WEB_FETCH_TIMEOUT (the provider-owned code), never TOOL_TIMEOUT.
const err = await tctx.web.fetch({ url: slowBase, timeoutMs: 50 }).then(
it('the provider backstop still protects a direct provider call (no tool-call policy in that path)', async () => {
// A direct provider caller bypasses tools/execute, so a short configured backstop
// must produce provider-owned WEB_FETCH_TIMEOUT rather than TOOL_TIMEOUT.
const direct = new WebFetchLocal.LocalFetchProvider({
maxUrlLength: 2048,
maxResponseBytes: 5_000_000,
maxBodyChars: 100_000,
timeoutMs: 50,
maxRedirects: 5,
userAgent: 'integration-test',
})
const err = await direct.fetch({ url: slowBase }).then(
() => undefined,
(e: unknown) => e as { code?: string },
)

View File

@@ -1,16 +1,8 @@
/**
* Real-load-path guard for @deepseek-ai/dsh-tool-web. `tool-web` is a NAMESPACE
* plugin with `inject` — so a stray `export default apply` would make the cordis
* Loader's `unwrapExports` (`exports.default ?? exports`) collapse the module to
* the bare `apply` function, DROPPING `inject`. The plugin would then read
* `ctx.web` without having injected it and throw `cannot get property … without
* inject` the moment it loads (postmortem 0001).
*
* A hand-built `ctx.plugin({ apply, inject })` mount CANNOT catch that — it
* bypasses `unwrapExports`. So this test unwraps the module through the REAL
* `Loader.prototype.unwrapExports` and mounts the result over `ctx.web`,
* exercising the exact path the Loader uses. Prove the guard bites: add
* `export default apply` to `src/index.ts`, watch this go red, revert.
* Real Loader-path guard for an injected namespace plugin. A default export would make
* `unwrapExports` collapse the namespace and drop `inject`, causing access to `ctx.web` to fail.
* Hand-built mounting bypasses that path, so this test unwraps through the real Loader first; see
* postmortem 0001.
*/
import { describe, expect, it } from 'vitest'
@@ -41,7 +33,7 @@ describe('dsh-tool-web real-load-path guard', () => {
const loader = Object.create(Loader.prototype) as Loader
const unwrapped = loader.unwrapExports(toolWeb) as Parameters<Context['plugin']>[0]
// A collapsed export shape (dropped inject) would throw "without inject" here.
// Mounting the collapsed shape would throw for missing injection here.
const fiber = await ctx.plugin(unwrapped)
expect(ctx.tools.schemas().map(s => s.name)).toEqual(expect.arrayContaining(['web_search', 'web_fetch']))
await fiber.dispose()

View File

@@ -4,7 +4,7 @@ import { CallId } from '@deepseek-ai/dsh-llm'
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
import ToolRegistry from '@deepseek-ai/dsh-tools'
import WebService from '@deepseek-ai/dsh-web'
import type { WebSearchProvider, WebSearchResult, WebProviderStatus } from '@deepseek-ai/dsh-web'
import type { WebSearchProvider, WebSearchResult } from '@deepseek-ai/dsh-web'
import * as ToolWeb from '@deepseek-ai/dsh-tool-web'
import {
formatSearchOutput,
@@ -18,10 +18,10 @@ import {
WEB_SEARCH_MAX_RESULTS,
} from '@deepseek-ai/dsh-tool-web'
const available: WebProviderStatus = { available: true }
const available = true
function searchProvider(result: WebSearchResult, status: WebProviderStatus = available): WebSearchProvider {
return { id: 'stub-search', status: () => status, search: () => Promise.resolve(result) }
function searchProvider(result: WebSearchResult, isAvailable = available): WebSearchProvider {
return { id: 'stub-search', available: () => isAvailable, search: () => Promise.resolve(result) }
}
/** Mount the real registry, seam, and tool-web; return an executor helper. */
@@ -46,7 +46,7 @@ async function mountTools(opts: {
describe('search formatting', () => {
it('renders content, sources with titles/hostnames, snippets, and a citation reminder', () => {
const out = formatSearchOutput({
providerId: 'p', query: 'q', content: 'an answer', truncated: false,
content: 'an answer', truncated: false,
sources: [
{ url: 'https://a.test/x', title: 'A', snippet: 'about a', publishedAt: '2026-01-01' },
{ url: 'https://b.test/y' },
@@ -59,19 +59,19 @@ describe('search formatting', () => {
})
it('reports no results when there is neither content nor sources', () => {
expect(formatSearchOutput({ providerId: 'p', query: 'q', sources: [], truncated: false }))
expect(formatSearchOutput({ sources: [], truncated: false }))
.toContain('No results found.')
})
it('renders content alone when there are no sources', () => {
const out = formatSearchOutput({ providerId: 'p', query: 'q', content: 'just an answer', sources: [], truncated: false })
const out = formatSearchOutput({ content: 'just an answer', sources: [], truncated: false })
expect(out).toContain('just an answer')
expect(out).not.toContain('No results found.')
expect(out).not.toContain('Sources:')
})
it('notes truncation', () => {
const out = formatSearchOutput({ providerId: 'p', query: 'q', sources: [{ url: 'https://a.test' }], truncated: true })
const out = formatSearchOutput({ sources: [{ url: 'https://a.test' }], truncated: true })
expect(out).toContain('Showing the first 1 sources')
})
@@ -88,7 +88,7 @@ describe('search formatting', () => {
describe('fetch formatting', () => {
it('renders an html body to markdown text with a status header', () => {
const out = formatFetchOutput({
providerId: 'p', url: 'https://a.test', statusCode: 200, truncated: false,
url: 'https://a.test', statusCode: 200, truncated: false,
body: { kind: 'html', content: '<h1>Title</h1><p>Body text</p>' },
})
expect(out).toContain('Fetched https://a.test (HTTP 200)')
@@ -98,7 +98,7 @@ describe('fetch formatting', () => {
it('passes a text body through and notes truncation', () => {
const out = formatFetchOutput({
providerId: 'p', url: 'https://a.test', statusCode: 200, truncated: true,
url: 'https://a.test', statusCode: 200, truncated: true,
body: { kind: 'text', content: 'plain' },
})
expect(out).toContain('plain')
@@ -155,7 +155,7 @@ describe('htmlToMarkdown', () => {
})
it('falls back to the raw URL as a source label when the URL is unparseable', () => {
const out = formatSearchOutput({ providerId: 'p', query: 'q', truncated: false, sources: [{ url: 'not a url' }] })
const out = formatSearchOutput({ truncated: false, sources: [{ url: 'not a url' }] })
expect(out).toContain('[not a url](not a url)')
})
})
@@ -209,7 +209,7 @@ describe('tool-web registration', () => {
describe('tool-web execution through the real registry', () => {
it('executes web_search and formats the result', async () => {
const result: WebSearchResult = {
providerId: 'stub-search', query: 'q', content: 'answer', truncated: false,
content: 'answer', truncated: false,
sources: [{ url: 'https://a.test', title: 'A', snippet: 'snip' }],
}
const { fiber, call } = await mountTools({ webConfig: { searchProvider: 'stub-search' }, search: searchProvider(result) })
@@ -228,8 +228,8 @@ describe('tool-web execution through the real registry', () => {
})
it('surfaces WEB_PROVIDER_AMBIGUOUS for multiple unconfigured providers', async () => {
const { ctx, fiber, call } = await mountTools({ search: searchProvider({ providerId: 'stub-search', query: 'q', sources: [], truncated: false }) })
ctx.web.registerSearchProvider({ id: 'other', status: () => available, search: () => Promise.resolve({ providerId: 'other', query: 'q', sources: [], truncated: false }) })
const { ctx, fiber, call } = await mountTools({ search: searchProvider({ sources: [], truncated: false }) })
ctx.web.registerSearchProvider({ id: 'other', available: () => available, search: () => Promise.resolve({ sources: [], truncated: false }) })
const out = await call('web_search', { query: 'q' })
expect(out.isError).toBe(true)
expect(out.error?.code).toBe('WEB_PROVIDER_AMBIGUOUS')
@@ -237,7 +237,7 @@ describe('tool-web execution through the real registry', () => {
})
it('rejects invalid arguments with a structured INVALID_ARGS error', async () => {
const { fiber, call } = await mountTools({ webConfig: { searchProvider: 'stub-search' }, search: searchProvider({ providerId: 'stub-search', query: 'q', sources: [], truncated: false }) })
const { fiber, call } = await mountTools({ webConfig: { searchProvider: 'stub-search' }, search: searchProvider({ sources: [], truncated: false }) })
const out = await call('web_search', { query: 123 })
expect(out.isError).toBe(true)
expect(out.error?.code).toBe('INVALID_ARGS')
@@ -249,14 +249,14 @@ describe('tool-web execution through the real registry', () => {
})
it('executes web_fetch, forwarding the url (no timeout param) and the abort signal to the seam', async () => {
const seen: { request?: { url: string; timeoutMs?: number }; signal?: AbortSignal | undefined } = {}
const seen: { request?: { url: string }; signal?: AbortSignal | undefined } = {}
const fetchProvider = {
id: 'stub-fetch',
status: () => available,
fetch: (request: { url: string; timeoutMs?: number }, exec?: { signal?: AbortSignal }) => {
available: () => available,
fetch: (request: { url: string }, signal?: AbortSignal) => {
seen.request = request
seen.signal = exec?.signal
return Promise.resolve({ providerId: 'stub-fetch', url: request.url, statusCode: 200, body: { kind: 'text' as const, content: 'ok' }, truncated: false })
seen.signal = signal
return Promise.resolve({ url: request.url, statusCode: 200, body: { kind: 'text' as const, content: 'ok' }, truncated: false })
},
}
const { ctx, fiber } = await mountTools({ webConfig: { fetchProvider: 'stub-fetch' }, fetchProvider })
@@ -271,21 +271,21 @@ describe('tool-web execution through the real registry', () => {
})
it('executes web_fetch with no caller signal (forwards undefined to the seam)', async () => {
const seen: { signal?: AbortSignal | undefined; passedExec?: boolean } = {}
const seen: { signal?: AbortSignal | undefined; passedSignal?: boolean } = {}
const fetchProvider = {
id: 'stub-fetch',
status: () => available,
fetch: (request: { url: string }, exec?: { signal?: AbortSignal }) => {
seen.passedExec = exec !== undefined
seen.signal = exec?.signal
return Promise.resolve({ providerId: 'stub-fetch', url: request.url, statusCode: 200, body: { kind: 'text' as const, content: 'ok' }, truncated: false })
available: () => available,
fetch: (request: { url: string }, signal?: AbortSignal) => {
seen.passedSignal = signal !== undefined
seen.signal = signal
return Promise.resolve({ url: request.url, statusCode: 200, body: { kind: 'text' as const, content: 'ok' }, truncated: false })
},
}
const { ctx, fiber } = await mountTools({ webConfig: { fetchProvider: 'stub-fetch' }, fetchProvider })
// No signal on the execution: the tool passes `undefined` (not `{ signal: undefined }`).
// No signal on the execution: the tool passes `undefined`.
const out = await ctx.tools.execute({ callId: CallId('fetch-2'), name: 'web_fetch', arguments: { url: 'https://a.test' } })
expect(out.isError).toBe(false)
expect(seen.passedExec).toBe(false)
expect(seen.passedSignal).toBe(false)
expect(seen.signal).toBeUndefined()
await fiber.dispose()
})
@@ -294,8 +294,8 @@ describe('tool-web execution through the real registry', () => {
const seen: { signal?: AbortSignal | undefined } = {}
const provider: WebSearchProvider = {
id: 'stub-search',
status: () => available,
search: (_request, exec) => { seen.signal = exec?.signal; return Promise.resolve({ providerId: 'stub-search', query: 'q', sources: [], truncated: false }) },
available: () => available,
search: (_request, signal) => { seen.signal = signal; return Promise.resolve({ sources: [], truncated: false }) },
}
const { ctx, fiber } = await mountTools({ webConfig: { searchProvider: 'stub-search' }, search: provider })
const controller = new AbortController()
@@ -310,8 +310,8 @@ describe('searchMaxResults is plugin config', () => {
const seen: { maxResults?: number | undefined } = {}
const provider: WebSearchProvider = {
id: 'stub-search',
status: () => available,
search: (request) => { seen.maxResults = request.maxResults; return Promise.resolve({ providerId: 'stub-search', query: 'q', sources: [], truncated: false }) },
available: () => available,
search: (request) => { seen.maxResults = request.maxResults; return Promise.resolve({ sources: [], truncated: false }) },
}
const { fiber, call } = await mountTools({ webConfig: { searchProvider: 'stub-search' }, search: provider })
await call('web_search', { query: 'q' })
@@ -323,8 +323,8 @@ describe('searchMaxResults is plugin config', () => {
const sources = Array.from({ length: 5 }, (_, i) => ({ url: `https://s${i}.test` }))
const provider: WebSearchProvider = {
id: 'stub-search',
status: () => available,
search: request => Promise.resolve({ providerId: 'stub-search', query: request.query, sources, truncated: false }),
available: () => available,
search: () => Promise.resolve({ sources, truncated: false }),
}
const { fiber, call } = await mountTools({ config: { searchMaxResults: 2 }, webConfig: { searchProvider: 'stub-search' }, search: provider })
const out = await call('web_search', { query: 'q' })