fix(tool-web): bound HTML conversion work
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
README.md: 44cb1ba2a2f4e1fba7e192d8b6645e0447ebf221
|
||||
README.zh.md: 35b390dd5407af16d84ab391dd8351f784c60035
|
||||
# pnpm run verify-translation-pairing --write packages/web/tool-web/README.md
|
||||
README.md: 9b78920b1b6c611118294421dec1e75e381ed5d6
|
||||
README.zh.md: 2152c40f1ccac2272fa0b2681514a712417c0ad3
|
||||
|
||||
@@ -26,9 +26,9 @@ The normalized seam results are also the canonical tool values: `WebSearchResult
|
||||
| `searchMaxResults` | `8` | Upper bound on sources returned by one `web_search` call (the seam truncates a longer provider list and flags it). |
|
||||
| `fetchTimeoutMs` | `30000` | Cooperative tool-call timeout budget (ms) for `web_fetch`. |
|
||||
| `searchTimeoutMs` | `30000` | Cooperative tool-call timeout budget (ms) for `web_search`. |
|
||||
| `fetchMaxOutputChars` | `200000` | Cap on one `web_fetch` output's characters — header, rendered body, and footer together; a cut body gets the truncation notice. |
|
||||
| `fetchMaxOutputChars` | `200000` | Cap on source characters converted synchronously and on one complete `web_fetch` output (header, rendered body, and footer); a cut body gets the truncation notice when it fits. |
|
||||
|
||||
`fetchTimeoutMs`/`searchTimeoutMs` declare each tool's cooperative timeout budget (attached as `ToolDefinition.timeoutMs`), enforced by [`@deepseek-ai/dsh-timeout-policy`](../../timeout/timeout-policy/README.md); the model-facing schema exposes no timeout argument. `fetchMaxOutputChars` bounds the complete rendered output because markdown escaping can expand converted HTML past a provider's body cap (worst case ~2×); the default is 2× the local provider's default 100,000-character body cap, so it never cuts what that bound already admits.
|
||||
`fetchTimeoutMs`/`searchTimeoutMs` declare each tool's cooperative timeout budget (attached as `ToolDefinition.timeoutMs`), enforced by [`@deepseek-ai/dsh-timeout-policy`](../../timeout/timeout-policy/README.md); the model-facing schema exposes no timeout argument. `fetchMaxOutputChars` bounds both synchronous conversion work and the complete rendered result: only that many source characters are converted, and the header, converted prefix, and truncation notice are then capped together. The default leaves headroom above the local provider's 100,000-character body cap, but rendered expansion can still make the final bound truncate the result.
|
||||
|
||||
```yaml
|
||||
- id: tool-web
|
||||
@@ -127,6 +127,6 @@ Append-only; newly visible content follows the reusable request prefix and does
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **HTML→markdown conversion falls back to raw HTML on pathological input** — [turndown](https://github.com/mixmark-io/turndown) (with GFM tables/strikethrough) converts fetched HTML through a real DOM, but the synchronous walk is superlinear on deep unclosed nesting, so bodies nested past a fixed 512-level preflight bound pass through unconverted (as does anything that still makes turndown throw) rather than stalling the event loop or erroring ([Agent Note](../../../.agents/notes/implemented/simplification/2026-07-26-turndown-for-tool-web-html-markdown.md)).
|
||||
- **HTML→markdown conversion degrades on inputs GFM cannot safely represent** — [turndown](https://github.com/mixmark-io/turndown) (with GFM tables/strikethrough) converts at most `fetchMaxOutputChars` source characters through a real DOM. A conservative 512-level lexical guard passes deeply or ambiguously nested bodies through as raw HTML, conversion exceptions do the same, and table `colspan` is ignored because GFM has no spanning-cell representation; these bounds avoid blocking the event loop or expanding output from an untrusted numeric attribute ([Agent Note](../../../.agents/notes/implemented/simplification/2026-07-26-turndown-for-tool-web-html-markdown.md)).
|
||||
- **The model-facing surface is minimal by design, with promotions deferred** — `max_results` stays a config bound (not a model argument), and `web_fetch` takes only `url` (no `format`/`prompt`/LLM-summarization mode); both are named later steps in [the seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md).
|
||||
- **No web-specific permission policy** — both tools execute without requesting `ctx.approval`; a deployment that needs confirmation must add a `tools/pre-execute` policy, and the package does not define persistent URL/domain grants.
|
||||
|
||||
@@ -26,9 +26,9 @@
|
||||
| `searchMaxResults` | `8` | 一次 `web_search` 调用返回的源数量上限(seam 截断更长的提供方列表并标记)。 |
|
||||
| `fetchTimeoutMs` | `30000` | `web_fetch` 的协作式工具调用超时预算(ms)。 |
|
||||
| `searchTimeoutMs` | `30000` | `web_search` 的协作式工具调用超时预算(ms)。 |
|
||||
| `fetchMaxOutputChars` | `200000` | 单次 `web_fetch` 输出的字符上限——状态头、渲染后的主体与页脚合并计算;被截断的主体带截断提示。 |
|
||||
| `fetchMaxOutputChars` | `200000` | 同步转换的源字符数与单次完整 `web_fetch` 输出的上限(状态头、渲染后的主体与页脚合并计算);主体被截断时,在能容纳的情况下附带截断提示。 |
|
||||
|
||||
`fetchTimeoutMs`/`searchTimeoutMs` 声明每个工具的协作式超时预算(附加为 `ToolDefinition.timeoutMs`),由 [`@deepseek-ai/dsh-timeout-policy`](../../timeout/timeout-policy/README.md) 强制执行;面向模型的 schema 不公开超时参数。`fetchMaxOutputChars` 对完整渲染输出设上限:markdown 转义可能让转换后的 HTML 超出提供方的主体上限(最坏约 2 倍);默认值取本地提供方默认 100,000 字符主体上限的 2 倍,因此绝不会削减该上限本已允许的内容。
|
||||
`fetchTimeoutMs`/`searchTimeoutMs` 声明每个工具的协作式超时预算(附加为 `ToolDefinition.timeoutMs`),由 [`@deepseek-ai/dsh-timeout-policy`](../../timeout/timeout-policy/README.md) 强制执行;面向模型的 schema 不公开超时参数。`fetchMaxOutputChars` 同时限制同步转换工作量和完整渲染结果:只转换至多该数量的源字符,随后对状态头、转换后的前缀和截断提示合并设限。默认值为本地提供方的 100,000 字符主体上限留出余量,但渲染膨胀仍可能使最终上限截断结果。
|
||||
|
||||
```yaml
|
||||
- id: tool-web
|
||||
@@ -127,6 +127,6 @@ Use the web_fetch tool to retrieve the content of a specific HTTP(S) URL (for ex
|
||||
|
||||
## 已知限制与暂缓事项
|
||||
|
||||
- **HTML→markdown 转换在病态输入上回退为原始 HTML**:[turndown](https://github.com/mixmark-io/turndown)(带 GFM 表格/删除线)通过真实 DOM 转换抓取到的 HTML,但同步遍历在深层未闭合嵌套上呈超线性,因此嵌套超过固定 512 层预检上限的主体不经转换原样通过(仍让 turndown 抛异常的输入同样如此),而非阻塞事件循环或报错([决策记录](../../../.agents/notes/implemented/simplification/2026-07-26-turndown-for-tool-web-html-markdown.md))。
|
||||
- **HTML→markdown 转换会在 GFM 无法安全表示的输入上降级**:[turndown](https://github.com/mixmark-io/turndown)(带 GFM 表格/删除线)通过真实 DOM 转换至多 `fetchMaxOutputChars` 个源字符。保守的 512 层词法守卫会将深层或嵌套有歧义的主体作为原始 HTML 直接透传,转换异常也会如此处理;表格的 `colspan` 会被忽略,因为 GFM 无法表示跨列单元格。这些限制可避免阻塞事件循环,也避免不受信任的数值属性使输出膨胀([决策记录](../../../.agents/notes/implemented/simplification/2026-07-26-turndown-for-tool-web-html-markdown.md))。
|
||||
- **面向模型的表层有意保持最小,提升项暂缓**:`max_results` 保持为配置上限(不是模型参数),`web_fetch` 只接受 `url`(没有 `format`/`prompt`/LLM 摘要模式);两项都列为 [seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md) 中的后续步骤。
|
||||
- **没有 web 专用权限策略**:两个工具都不会请求 `ctx.approval` 就直接执行;需要确认的部署必须添加 `tools/pre-execute` 策略,该包不定义持久 URL/domain 授权。
|
||||
|
||||
@@ -30,6 +30,52 @@ const turndown = new TurndownService({
|
||||
turndown.use(gfm)
|
||||
turndown.remove(['script', 'style', 'noscript'])
|
||||
|
||||
/** Render one GFM table cell without interpreting HTML span counts. */
|
||||
function renderTableCell(content: string, index: number): string {
|
||||
const prefix = index === 0 ? '| ' : ' '
|
||||
const escaped = content.trim().replace(/\n\r/g, '<br>').replace(/\n/g, '<br>').replace(/\|+/g, '\\|').padEnd(3, ' ')
|
||||
return `${prefix}${escaped} |`
|
||||
}
|
||||
|
||||
/** Whether a row is the table's Markdown heading row. */
|
||||
function isTableHeadingRow(row: HTMLTableRowElement): boolean {
|
||||
const cells = Array.from(row.cells)
|
||||
const section = row.parentElement as HTMLTableSectionElement
|
||||
const table = section.parentElement as HTMLTableElement
|
||||
return (section.nodeName === 'THEAD' || table.rows[0] === row)
|
||||
&& cells.every(cell => cell.nodeName === 'TH')
|
||||
}
|
||||
|
||||
/** Map an HTML table-cell alignment to the GFM separator marker. */
|
||||
function tableBorder(cell: HTMLTableCellElement): string {
|
||||
const alignment = (cell.getAttribute('align') || cell.style.textAlign || '').toLowerCase()
|
||||
if (alignment === 'left') return ':---'
|
||||
if (alignment === 'right') return '---:'
|
||||
if (alignment === 'center') return ':---:'
|
||||
return '---'
|
||||
}
|
||||
|
||||
turndown.addRule('tableCellWithoutSpanExpansion', {
|
||||
filter: ['th', 'td'],
|
||||
replacement(content, node) {
|
||||
const cell = node as HTMLTableCellElement
|
||||
const row = cell.parentNode as HTMLTableRowElement
|
||||
// GFM cannot represent spanning cells. Ignoring colspan keeps conversion
|
||||
// work and output proportional to the source instead of the numeric attribute.
|
||||
return renderTableCell(content, Array.prototype.indexOf.call(row.childNodes, cell))
|
||||
},
|
||||
})
|
||||
turndown.addRule('tableRowWithoutSpanExpansion', {
|
||||
filter: 'tr',
|
||||
replacement(content, node) {
|
||||
const row = node as HTMLTableRowElement
|
||||
const border = isTableHeadingRow(row)
|
||||
? Array.from(row.cells, (cell, index) => renderTableCell(tableBorder(cell), index)).join('')
|
||||
: ''
|
||||
return `\n${content}${border.length > 0 ? `\n${border}` : ''}`
|
||||
},
|
||||
})
|
||||
|
||||
/**
|
||||
* Validate value constraints the schema DSL can't express: a non-blank `url`.
|
||||
* Throws a plain `Error` otherwise. No timeout parameter — the tool-call budget
|
||||
@@ -55,63 +101,142 @@ export function parseFetchArgs(args: { url: string }): { url: string } {
|
||||
*/
|
||||
const MAX_CONVERSION_DEPTH = 512
|
||||
|
||||
/** Elements that never take a closing tag, so they must not count toward nesting depth. */
|
||||
/** Elements that never take a closing tag, so they do not grow the lexical stack. */
|
||||
const VOID_ELEMENTS = new Set([
|
||||
'area', 'base', 'br', 'col', 'embed', 'hr', 'img', 'input',
|
||||
'link', 'meta', 'param', 'source', 'track', 'wbr',
|
||||
])
|
||||
|
||||
/** Elements whose contents HTML parses as text until their matching end tag. */
|
||||
const RAW_TEXT_ELEMENTS = new Set(['script', 'style', 'noscript'])
|
||||
|
||||
/** Whether a character can occur after a raw-text end-tag name. */
|
||||
function isTagBoundary(char: string | undefined): boolean {
|
||||
return char === undefined || char === '>' || char === '/' || /\s/.test(char)
|
||||
}
|
||||
|
||||
/** Find the matching raw-text end tag without interpreting markup-like body text. */
|
||||
function findRawTextEnd(lowerHtml: string, name: string, from: number): number {
|
||||
const prefix = `</${name}`
|
||||
let candidate = lowerHtml.indexOf(prefix, from)
|
||||
while (candidate !== -1 && !isTagBoundary(lowerHtml[candidate + prefix.length])) {
|
||||
candidate = lowerHtml.indexOf(prefix, candidate + prefix.length)
|
||||
}
|
||||
return candidate
|
||||
}
|
||||
|
||||
/**
|
||||
* Estimate the maximum element nesting depth of an HTML string with one linear
|
||||
* tag scan. Overestimates when markup-like text sits inside `script`/`style`
|
||||
* bodies or comments (the scan does not parse those), which can only cause a
|
||||
* spurious raw-HTML fallback, never a missed bound.
|
||||
* Conservatively reject HTML whose lexical element stack crosses the conversion
|
||||
* depth ceiling. The single pass ignores closing tags inside comments, skips
|
||||
* raw-text bodies, respects quoted `>` characters, and only accepts a closing
|
||||
* tag for the current element; malformed input therefore over-counts rather
|
||||
* than hiding nesting.
|
||||
*
|
||||
* @param html - the decoded HTML body.
|
||||
* @returns the deepest open-element count the scan reaches.
|
||||
* @returns whether the body crosses {@link MAX_CONVERSION_DEPTH}.
|
||||
*/
|
||||
export function htmlNestingDepth(html: string): number {
|
||||
let depth = 0
|
||||
let max = 0
|
||||
for (const tag of html.matchAll(/<(\/?)([a-zA-Z][a-zA-Z0-9-]*)[^>]*?(\/?)>/g)) {
|
||||
const [, closing, rawName = '', selfClosing] = tag
|
||||
const name = rawName.toLowerCase()
|
||||
if (VOID_ELEMENTS.has(name) || selfClosing === '/') continue
|
||||
if (closing === '/') {
|
||||
if (depth > 0) depth -= 1
|
||||
} else {
|
||||
depth += 1
|
||||
if (depth > max) max = depth
|
||||
function exceedsConversionDepth(html: string): boolean {
|
||||
const lowerHtml = html.toLowerCase()
|
||||
const openElements: string[] = []
|
||||
let offset = 0
|
||||
let inComment = false
|
||||
|
||||
while (offset < html.length) {
|
||||
const start = html.indexOf('<', offset)
|
||||
if (inComment) {
|
||||
const end = html.indexOf('-->', offset)
|
||||
if (end !== -1 && (start === -1 || end < start)) {
|
||||
inComment = false
|
||||
offset = end + 3
|
||||
continue
|
||||
}
|
||||
}
|
||||
if (start === -1) break
|
||||
if (!inComment && html.startsWith('<!--', start)) {
|
||||
inComment = true
|
||||
offset = start + 4
|
||||
continue
|
||||
}
|
||||
|
||||
let cursor = start + 1
|
||||
const closing = html[cursor] === '/'
|
||||
if (closing) cursor += 1
|
||||
const nameStart = cursor
|
||||
while (/[a-zA-Z0-9-]/.test(html[cursor] ?? '')) cursor += 1
|
||||
if (cursor === nameStart || !/[a-zA-Z]/.test(html.charAt(nameStart))) {
|
||||
offset = start + 1
|
||||
continue
|
||||
}
|
||||
|
||||
const name = lowerHtml.slice(nameStart, cursor)
|
||||
let quote: '"' | "'" | undefined
|
||||
while (cursor < html.length) {
|
||||
const char = html[cursor]
|
||||
cursor += 1
|
||||
if (quote !== undefined) {
|
||||
if (char === quote) quote = undefined
|
||||
} else if (char === '"' || char === "'") {
|
||||
quote = char
|
||||
} else if (char === '>') {
|
||||
break
|
||||
}
|
||||
}
|
||||
if (html[cursor - 1] !== '>') break
|
||||
|
||||
if (closing) {
|
||||
if (!inComment && openElements.at(-1) === name) openElements.pop()
|
||||
} else {
|
||||
let last = cursor - 2
|
||||
while (/\s/.test(html.charAt(last))) last -= 1
|
||||
if (!VOID_ELEMENTS.has(name) && html[last] !== '/') {
|
||||
openElements.push(name)
|
||||
if (openElements.length > MAX_CONVERSION_DEPTH) return true
|
||||
if (!inComment && RAW_TEXT_ELEMENTS.has(name)) {
|
||||
const end = findRawTextEnd(lowerHtml, name, cursor)
|
||||
if (end === -1) break
|
||||
offset = end
|
||||
continue
|
||||
}
|
||||
}
|
||||
}
|
||||
offset = cursor
|
||||
}
|
||||
return max
|
||||
return false
|
||||
}
|
||||
|
||||
interface RenderedBody {
|
||||
/** Converted text, or raw HTML when conversion is unsafe or fails. */
|
||||
text: string
|
||||
/** Whether the source was cut before conversion to bound synchronous work. */
|
||||
sourceTruncated: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* Render a fetched body to model-facing markdown text.
|
||||
*
|
||||
* @param body - the decoded body; `html` is converted via turndown, `text`
|
||||
* passes through verbatim. HTML nested beyond {@link MAX_CONVERSION_DEPTH}
|
||||
* skips conversion up front (the synchronous walk over such trees is
|
||||
* superlinear and blocks the event loop past the cooperative timeout), and
|
||||
* when turndown itself throws the raw HTML passes through instead — a
|
||||
* degraded page beats an error for a body the provider already decoded.
|
||||
* @returns the text for the tool's output block.
|
||||
* passes through verbatim.
|
||||
* @param maxInputChars - maximum source characters processed synchronously.
|
||||
* @returns the rendered prefix and whether the source was cut. HTML nested
|
||||
* beyond {@link MAX_CONVERSION_DEPTH} or rejected by turndown passes through
|
||||
* raw; a degraded page beats an error for a body the provider decoded.
|
||||
*/
|
||||
export function renderBody(body: WebFetchBody): string {
|
||||
function renderBody(body: WebFetchBody, maxInputChars: number): RenderedBody {
|
||||
const content = body.content.slice(0, maxInputChars)
|
||||
const sourceTruncated = content.length !== body.content.length
|
||||
switch (body.kind) {
|
||||
case 'html':
|
||||
if (htmlNestingDepth(body.content) > MAX_CONVERSION_DEPTH) return body.content
|
||||
if (exceedsConversionDepth(content)) return { text: content, sourceTruncated }
|
||||
try {
|
||||
return turndown.turndown(body.content)
|
||||
return { text: turndown.turndown(content), sourceTruncated }
|
||||
} catch {
|
||||
// turndown's DOM walk recurses per element; malformed markup the depth
|
||||
// scan cannot see can still throw RangeError. Provider errors stay
|
||||
// turndown's DOM walk recurses per element; malformed markup the lexical
|
||||
// guard cannot model can still throw RangeError. Provider errors stay
|
||||
// structured WebErrors upstream; conversion failure downgrades to raw HTML.
|
||||
return body.content
|
||||
return { text: content, sourceTruncated }
|
||||
}
|
||||
case 'text':
|
||||
return body.content
|
||||
return { text: content, sourceTruncated }
|
||||
/* v8 ignore next 2 -- WebFetchBody is a closed union; this arm is unreachable and only makes adding a kind a compile error. */
|
||||
default:
|
||||
return assertNever(body, 'unhandled web fetch body kind')
|
||||
@@ -123,9 +248,8 @@ const TRUNCATION_FOOTER = '\n\n(Content truncated. Fetch a more specific URL or
|
||||
|
||||
/**
|
||||
* Format a fetch result as one model-facing text block, bounded as a whole.
|
||||
* Markdown escaping can expand converted HTML (worst case ~2× the provider's
|
||||
* body cap), so the bound applies here, where the complete output — header,
|
||||
* rendered body, and footer — is known.
|
||||
* The same cap limits the source prefix processed synchronously, then applies
|
||||
* again where the complete output — header, rendered body, and footer — is known.
|
||||
*
|
||||
* @param result - the seam's fetch outcome.
|
||||
* @param maxOutputChars - cap on the complete returned string; a cut body gets
|
||||
@@ -135,11 +259,13 @@ const TRUNCATION_FOOTER = '\n\n(Content truncated. Fetch a more specific URL or
|
||||
*/
|
||||
export function formatFetchOutput(result: WebFetchResult, maxOutputChars: number): string {
|
||||
const header = `Fetched ${result.url} (HTTP ${result.statusCode})\n\n`
|
||||
const body = renderBody(result.body)
|
||||
const full = `${header}${body}${result.truncated ? TRUNCATION_FOOTER : ''}`
|
||||
const rendered = renderBody(result.body, maxOutputChars)
|
||||
const prefix = `${header}${rendered.text}`
|
||||
const truncated = result.truncated || rendered.sourceTruncated || prefix.length > maxOutputChars
|
||||
const full = `${prefix}${truncated ? TRUNCATION_FOOTER : ''}`
|
||||
if (full.length <= maxOutputChars) return full
|
||||
const budget = Math.max(0, maxOutputChars - header.length - TRUNCATION_FOOTER.length)
|
||||
return `${header}${body.slice(0, budget)}${TRUNCATION_FOOTER}`
|
||||
if (maxOutputChars < TRUNCATION_FOOTER.length) return full.slice(0, maxOutputChars)
|
||||
return `${prefix.slice(0, maxOutputChars - TRUNCATION_FOOTER.length)}${TRUNCATION_FOOTER}`
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -160,8 +286,7 @@ export function presentFetchCall(args: { url: string }): GenericCallView {
|
||||
* @param timeoutMs - the cooperative tool-call budget (ms) attached as the tool's
|
||||
* `ToolDefinition.timeoutMs` for `@deepseek-ai/dsh-timeout-policy` to enforce.
|
||||
* @param maxOutputChars - cap on the complete rendered tool output (see
|
||||
* {@link formatFetchOutput}); markdown escaping can outgrow the provider's
|
||||
* body cap, so the model-context bound is enforced on the rendered result.
|
||||
* {@link formatFetchOutput}) and on source characters converted synchronously.
|
||||
*/
|
||||
export function applyWebFetchTool(ctx: Context, timeoutMs: number, maxOutputChars: number): void {
|
||||
ctx.systemPrompt.section({
|
||||
|
||||
@@ -13,7 +13,7 @@ import { applyWebSearchTool, WEB_SEARCH_MAX_RESULTS } from './search.ts'
|
||||
import { applyWebFetchTool } from './fetch.ts'
|
||||
|
||||
export { WEB_SEARCH_MAX_RESULTS, applyWebSearchTool, formatSearchOutput, parseSearchArgs, presentSearchCall } from './search.ts'
|
||||
export { applyWebFetchTool, formatFetchOutput, htmlNestingDepth, parseFetchArgs, presentFetchCall, renderBody } from './fetch.ts'
|
||||
export { applyWebFetchTool, formatFetchOutput, parseFetchArgs, presentFetchCall } from './fetch.ts'
|
||||
|
||||
/** Cordis plugin name used by loader diagnostics. */
|
||||
export const name = 'tool-web'
|
||||
@@ -25,11 +25,9 @@ export const inject = ['tools', 'web', 'systemPrompt']
|
||||
export const DEFAULT_WEB_TOOL_TIMEOUT_MS = 30_000
|
||||
|
||||
/**
|
||||
* Default cap on one `web_fetch` output's characters. Markdown escaping can
|
||||
* roughly double converted HTML, so this sits at 2× the local provider's
|
||||
* default 100,000-char body cap: it never cuts what that composition's
|
||||
* provider bound already admits, while restoring a model-context bound for
|
||||
* providers with larger or absent body caps.
|
||||
* Default cap on one `web_fetch` output and on source characters converted
|
||||
* synchronously. This leaves headroom above the local provider's default
|
||||
* 100,000-character body cap while bounding custom providers and rendered output.
|
||||
*/
|
||||
export const DEFAULT_FETCH_MAX_OUTPUT_CHARS = 200_000
|
||||
|
||||
@@ -45,7 +43,7 @@ export interface Config {
|
||||
fetchTimeoutMs?: number
|
||||
/** Cooperative timeout budget (ms) for `web_search`. Defaults to 30000. */
|
||||
searchTimeoutMs?: number
|
||||
/** Cap on one `web_fetch` output's characters (header, rendered body, and footer). Defaults to 200000. */
|
||||
/** Cap on source characters converted and complete `web_fetch` output characters. Defaults to 200000. */
|
||||
fetchMaxOutputChars?: number
|
||||
}
|
||||
|
||||
@@ -61,7 +59,7 @@ export const Config: z<Config> = z.object({
|
||||
/** The shape after schemastery applies its defaults to every field. */
|
||||
type ResolvedConfig = Required<Config>
|
||||
|
||||
/** The result cap must be a positive integer (it bounds a provider's source list). */
|
||||
/** Configured count, timeout, and character caps must be positive integers. */
|
||||
function assertPositiveInteger(name: string, value: number): void {
|
||||
if (!Number.isInteger(value) || value < 1) {
|
||||
throw new Error(`tool-web: ${name} must be a positive integer`)
|
||||
|
||||
@@ -10,12 +10,10 @@ import * as ToolWeb from '@deepseek-ai/dsh-tool-web'
|
||||
import {
|
||||
formatSearchOutput,
|
||||
formatFetchOutput,
|
||||
htmlNestingDepth,
|
||||
parseSearchArgs,
|
||||
parseFetchArgs,
|
||||
presentSearchCall,
|
||||
presentFetchCall,
|
||||
renderBody,
|
||||
WEB_SEARCH_MAX_RESULTS,
|
||||
} from '@deepseek-ai/dsh-tool-web'
|
||||
|
||||
@@ -95,6 +93,11 @@ describe('search formatting', () => {
|
||||
|
||||
describe('fetch formatting', () => {
|
||||
const NO_CAP = 1_000_000
|
||||
const HEADER = 'Fetched https://a.test (HTTP 200)\n\n'
|
||||
const renderHtml = (content: string) => formatFetchOutput({
|
||||
url: 'https://a.test', statusCode: 200, truncated: false,
|
||||
body: { kind: 'html', content },
|
||||
}, NO_CAP).slice(HEADER.length)
|
||||
|
||||
it('renders an html body to markdown text with a status header', () => {
|
||||
const out = formatFetchOutput({
|
||||
@@ -136,29 +139,36 @@ describe('fetch formatting', () => {
|
||||
url: 'https://a.test', statusCode: 200, truncated: true,
|
||||
body: { kind: 'text', content: 'abcdef' },
|
||||
}, 10)
|
||||
expect(tiny).toContain('Fetched https://a.test (HTTP 200)')
|
||||
expect(tiny).toContain('Content truncated')
|
||||
expect(tiny).not.toContain('abcdef')
|
||||
expect(tiny.length).toBeLessThanOrEqual(10)
|
||||
expect(tiny).toBe('Fetched ht')
|
||||
})
|
||||
|
||||
it('renderBody dispatches on kind', () => {
|
||||
expect(renderBody({ kind: 'text', content: 'x' })).toBe('x')
|
||||
expect(renderBody({ kind: 'html', content: '<p>y</p>' })).toBe('y')
|
||||
it('dispatches text and html bodies', () => {
|
||||
expect(formatFetchOutput({
|
||||
url: 'https://a.test', statusCode: 200, truncated: false,
|
||||
body: { kind: 'text', content: 'x' },
|
||||
}, NO_CAP)).toBe(`${HEADER}x`)
|
||||
expect(renderHtml('<p>y</p>')).toBe('y')
|
||||
})
|
||||
|
||||
it('converts html via turndown: entities, links, tables, nesting; drops script/style/noscript', () => {
|
||||
expect(renderBody({
|
||||
kind: 'html',
|
||||
content: '<style>.x{}</style><script>bad()</script><noscript>ns</noscript><p>Tom & Jerry © Résumé</p><a href="https://a.test">link</a>',
|
||||
})).toBe('Tom & Jerry © Résumé\n\n[link](https://a.test)')
|
||||
expect(renderBody({ kind: 'html', content: '<h2>Heading</h2><ul><li>one</li><li>two</li></ul>' }))
|
||||
expect(renderHtml('<style>.x{}</style><script>bad()</script><noscript>ns</noscript><p>Tom & Jerry © Résumé</p><a href="https://a.test">link</a>'))
|
||||
.toBe('Tom & Jerry © Résumé\n\n[link](https://a.test)')
|
||||
expect(renderHtml('<h2>Heading</h2><ul><li>one</li><li>two</li></ul>'))
|
||||
.toBe('## Heading\n\n- one\n- two')
|
||||
expect(renderBody({ kind: 'html', content: '<table><tr><th>A</th><th>B</th></tr><tr><td>1</td><td>2</td></tr></table>' }))
|
||||
expect(renderHtml('<table><tr><th>A</th><th>B</th></tr><tr><td>1</td><td>2</td></tr></table>'))
|
||||
.toBe('| A | B |\n| --- | --- |\n| 1 | 2 |')
|
||||
expect(renderBody({ kind: 'html', content: '<p><strong>bold <em>italic</em></strong></p><blockquote><p>quoted</p></blockquote>' }))
|
||||
expect(renderHtml('<table><thead><tr><th align="left">L</th><th align="right">R</th><th style="text-align:center">C</th></tr></thead><tbody><tr><td>1</td><td>2</td><td>3</td></tr></tbody></table>'))
|
||||
.toBe('| L | R | C |\n| :--- | ---: | :---: |\n| 1 | 2 | 3 |')
|
||||
expect(renderHtml('<p><strong>bold <em>italic</em></strong></p><blockquote><p>quoted</p></blockquote>'))
|
||||
.toBe('**bold _italic_**\n\n> quoted')
|
||||
})
|
||||
|
||||
it('does not expand numeric colspan attributes into unbounded output', () => {
|
||||
const table = '<table><thead><tr><th colspan="1000000">A</th></tr></thead><tbody><tr><td>B</td></tr></tbody></table>'
|
||||
expect(renderHtml(table)).toBe('| A |\n| --- |\n| B |')
|
||||
})
|
||||
|
||||
it('passes deeply nested html through raw without attempting conversion', () => {
|
||||
// Unclosed-tag nesting makes the synchronous conversion superlinear
|
||||
// (seconds at 20k levels, during which the cooperative timeout cannot
|
||||
@@ -167,27 +177,73 @@ describe('fetch formatting', () => {
|
||||
const depth = 20_000
|
||||
const pathological = '<div>'.repeat(depth) + 'x' + '</div>'.repeat(depth)
|
||||
const started = Date.now()
|
||||
expect(renderBody({ kind: 'html', content: pathological })).toBe(pathological)
|
||||
expect(formatFetchOutput({
|
||||
url: 'https://a.test', statusCode: 200, truncated: false,
|
||||
body: { kind: 'html', content: pathological },
|
||||
}, NO_CAP)).toBe(`${HEADER}${pathological}`)
|
||||
expect(Date.now() - started).toBeLessThan(2_000)
|
||||
})
|
||||
|
||||
it('htmlNestingDepth counts open elements, ignoring void and self-closing tags', () => {
|
||||
expect(htmlNestingDepth('<div><p>x</p></div>')).toBe(2)
|
||||
expect(htmlNestingDepth('<div><br><img src="x"><input/></div>')).toBe(1)
|
||||
expect(htmlNestingDepth('</div></div><p>x</p>')).toBe(1)
|
||||
expect(htmlNestingDepth('plain text, no tags')).toBe(0)
|
||||
expect(htmlNestingDepth('<div>'.repeat(600))).toBe(600)
|
||||
it('comments and mismatched closing tags cannot hide deep nesting from the preflight', () => {
|
||||
const pathological = '<div><!-- </div> --></span>'.repeat(600) + 'x'
|
||||
expect(formatFetchOutput({
|
||||
url: 'https://a.test', statusCode: 200, truncated: false,
|
||||
body: { kind: 'html', content: pathological },
|
||||
}, NO_CAP)).toBe(`${HEADER}${pathological}`)
|
||||
const abruptlyClosedComments = '<div><!-->'.repeat(600) + 'x'
|
||||
expect(formatFetchOutput({
|
||||
url: 'https://a.test', statusCode: 200, truncated: false,
|
||||
body: { kind: 'html', content: abruptlyClosedComments },
|
||||
}, NO_CAP)).toBe(`${HEADER}${abruptlyClosedComments}`)
|
||||
})
|
||||
|
||||
it('the preflight accepts ordinary closed, void, self-closing, quoted, and raw-text markup', () => {
|
||||
const paragraphs = '<p title=\'>\'>x<br ><img src="x"><input/></p>'.repeat(600)
|
||||
const script = `<script>const invalid = '</scriptx>'; const template = '${'<div>'.repeat(600)}'</script >`
|
||||
expect(renderHtml(`<!doctype html><?pi><1bad>${paragraphs}${script}`))
|
||||
.not.toContain('<p')
|
||||
expect(renderHtml('plain text')).toBe('plain text')
|
||||
expect(renderHtml('<p>x</p><!-- unfinished')).toBe('x')
|
||||
expect(renderHtml('<script>unclosed')).toBe('')
|
||||
expect(renderHtml('<script>closed by slash</script/>')).toBe('')
|
||||
expect(renderHtml('<script>closed at end</script')).toBe('')
|
||||
})
|
||||
|
||||
it('scans malformed unterminated tags in bounded time', () => {
|
||||
const malformed = '<a'.repeat(100_000)
|
||||
const started = Date.now()
|
||||
const out = formatFetchOutput({
|
||||
url: 'https://a.test', statusCode: 200, truncated: false,
|
||||
body: { kind: 'html', content: malformed },
|
||||
}, 200_000)
|
||||
expect(out.length).toBeLessThanOrEqual(200_000)
|
||||
expect(Date.now() - started).toBeLessThan(2_000)
|
||||
})
|
||||
|
||||
it('falls back to the raw html when turndown throws despite a shallow depth scan', () => {
|
||||
// Comments hide markup from the depth scan by design (it may only
|
||||
// over-count, never under-count real elements); simulate the residual
|
||||
// turndown failure path with a converter throw instead.
|
||||
const spy = vi.spyOn(TurndownService.prototype, 'turndown').mockImplementation(() => {
|
||||
throw new RangeError('Maximum call stack size exceeded')
|
||||
})
|
||||
try {
|
||||
expect(renderBody({ kind: 'html', content: '<p>x</p>' })).toBe('<p>x</p>')
|
||||
expect(formatFetchOutput({
|
||||
url: 'https://a.test', statusCode: 200, truncated: false,
|
||||
body: { kind: 'html', content: '<p>x</p>' },
|
||||
}, NO_CAP)).toBe(`${HEADER}<p>x</p>`)
|
||||
} finally {
|
||||
spy.mockRestore()
|
||||
}
|
||||
})
|
||||
|
||||
it('bounds source conversion work before rendering a custom provider body', () => {
|
||||
const spy = vi.spyOn(TurndownService.prototype, 'turndown').mockReturnValue('converted')
|
||||
try {
|
||||
const out = formatFetchOutput({
|
||||
url: 'https://a.test', statusCode: 200, truncated: false,
|
||||
body: { kind: 'html', content: `<p>${'x'.repeat(10_000)}</p>` },
|
||||
}, 500)
|
||||
expect(spy).toHaveBeenCalledWith(`<p>${'x'.repeat(497)}`)
|
||||
expect(out.length).toBeLessThanOrEqual(500)
|
||||
expect(out).toContain('Content truncated')
|
||||
} finally {
|
||||
spy.mockRestore()
|
||||
}
|
||||
@@ -436,3 +492,35 @@ describe('tool-call timeout budget is plugin config', () => {
|
||||
.rejects.toThrow(new RegExp(`tool-web: ${key} must be a positive integer`))
|
||||
})
|
||||
})
|
||||
|
||||
describe('fetchMaxOutputChars is plugin config', () => {
|
||||
it('bounds the rendered output of the registered web_fetch tool', async () => {
|
||||
const fetchProvider = {
|
||||
id: 'stub-fetch',
|
||||
available: () => available,
|
||||
fetch: (request: { url: string }) => Promise.resolve({
|
||||
url: request.url,
|
||||
statusCode: 200,
|
||||
body: { kind: 'html' as const, content: `<p>${'_'.repeat(1_000)}</p>` },
|
||||
truncated: false,
|
||||
}),
|
||||
}
|
||||
const { fiber, call } = await mountTools({
|
||||
config: { fetchMaxOutputChars: 100 },
|
||||
webConfig: { fetchProvider: 'stub-fetch' },
|
||||
fetchProvider,
|
||||
})
|
||||
const out = await call('web_fetch', { url: 'https://a.test' })
|
||||
expect(out.content.map(block => block.type === 'text' ? block.text : '').join('')).toHaveLength(100)
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it.each([0, -1, 1.5])('rejects an invalid fetchMaxOutputChars value %s at load', async (value) => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SystemPrompt)
|
||||
await ctx.plugin(ToolRegistry)
|
||||
await ctx.plugin(WebService, {})
|
||||
await expect(ctx.plugin(ToolWeb, { fetchMaxOutputChars: value }))
|
||||
.rejects.toThrow(/tool-web: fetchMaxOutputChars must be a positive integer/)
|
||||
})
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user