fix(tool-web): bound HTML conversion work

This commit is contained in:
Tianyi Cui
2026-07-27 16:40:02 +08:00
parent ae18216199
commit 9801a0d102
11 changed files with 314 additions and 100 deletions

View File

@@ -1,6 +1,6 @@
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write
README.md: 44cb1ba2a2f4e1fba7e192d8b6645e0447ebf221
README.zh.md: 35b390dd5407af16d84ab391dd8351f784c60035
# pnpm run verify-translation-pairing --write packages/web/tool-web/README.md
README.md: 9b78920b1b6c611118294421dec1e75e381ed5d6
README.zh.md: 2152c40f1ccac2272fa0b2681514a712417c0ad3

View File

@@ -26,9 +26,9 @@ The normalized seam results are also the canonical tool values: `WebSearchResult
| `searchMaxResults` | `8` | Upper bound on sources returned by one `web_search` call (the seam truncates a longer provider list and flags it). |
| `fetchTimeoutMs` | `30000` | Cooperative tool-call timeout budget (ms) for `web_fetch`. |
| `searchTimeoutMs` | `30000` | Cooperative tool-call timeout budget (ms) for `web_search`. |
| `fetchMaxOutputChars` | `200000` | Cap on one `web_fetch` output's characters — header, rendered body, and footer together; a cut body gets the truncation notice. |
| `fetchMaxOutputChars` | `200000` | Cap on source characters converted synchronously and on one complete `web_fetch` output (header, rendered body, and footer); a cut body gets the truncation notice when it fits. |
`fetchTimeoutMs`/`searchTimeoutMs` declare each tool's cooperative timeout budget (attached as `ToolDefinition.timeoutMs`), enforced by [`@deepseek-ai/dsh-timeout-policy`](../../timeout/timeout-policy/README.md); the model-facing schema exposes no timeout argument. `fetchMaxOutputChars` bounds the complete rendered output because markdown escaping can expand converted HTML past a provider's body cap (worst case ~2×); the default is 2× the local provider's default 100,000-character body cap, so it never cuts what that bound already admits.
`fetchTimeoutMs`/`searchTimeoutMs` declare each tool's cooperative timeout budget (attached as `ToolDefinition.timeoutMs`), enforced by [`@deepseek-ai/dsh-timeout-policy`](../../timeout/timeout-policy/README.md); the model-facing schema exposes no timeout argument. `fetchMaxOutputChars` bounds both synchronous conversion work and the complete rendered result: only that many source characters are converted, and the header, converted prefix, and truncation notice are then capped together. The default leaves headroom above the local provider's 100,000-character body cap, but rendered expansion can still make the final bound truncate the result.
```yaml
- id: tool-web
@@ -127,6 +127,6 @@ Append-only; newly visible content follows the reusable request prefix and does
## Known Limitations and Deferred Work
- **HTML→markdown conversion falls back to raw HTML on pathological input** — [turndown](https://github.com/mixmark-io/turndown) (with GFM tables/strikethrough) converts fetched HTML through a real DOM, but the synchronous walk is superlinear on deep unclosed nesting, so bodies nested past a fixed 512-level preflight bound pass through unconverted (as does anything that still makes turndown throw) rather than stalling the event loop or erroring ([Agent Note](../../../.agents/notes/implemented/simplification/2026-07-26-turndown-for-tool-web-html-markdown.md)).
- **HTML→markdown conversion degrades on inputs GFM cannot safely represent** — [turndown](https://github.com/mixmark-io/turndown) (with GFM tables/strikethrough) converts at most `fetchMaxOutputChars` source characters through a real DOM. A conservative 512-level lexical guard passes deeply or ambiguously nested bodies through as raw HTML, conversion exceptions do the same, and table `colspan` is ignored because GFM has no spanning-cell representation; these bounds avoid blocking the event loop or expanding output from an untrusted numeric attribute ([Agent Note](../../../.agents/notes/implemented/simplification/2026-07-26-turndown-for-tool-web-html-markdown.md)).
- **The model-facing surface is minimal by design, with promotions deferred** — `max_results` stays a config bound (not a model argument), and `web_fetch` takes only `url` (no `format`/`prompt`/LLM-summarization mode); both are named later steps in [the seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md).
- **No web-specific permission policy** — both tools execute without requesting `ctx.approval`; a deployment that needs confirmation must add a `tools/pre-execute` policy, and the package does not define persistent URL/domain grants.

View File

@@ -26,9 +26,9 @@
| `searchMaxResults` | `8` | 一次 `web_search` 调用返回的源数量上限seam 截断更长的提供方列表并标记)。 |
| `fetchTimeoutMs` | `30000` | `web_fetch` 的协作式工具调用超时预算ms。 |
| `searchTimeoutMs` | `30000` | `web_search` 的协作式工具调用超时预算ms。 |
| `fetchMaxOutputChars` | `200000` | 单次 `web_fetch` 输出的字符上限——状态头、渲染后的主体与页脚合并计算;被截断的主体带截断提示。 |
| `fetchMaxOutputChars` | `200000` | 同步转换的源字符数与单次完整 `web_fetch` 输出的上限(状态头、渲染后的主体与页脚合并计算);主体被截断时,在能容纳的情况下附带截断提示。 |
`fetchTimeoutMs``searchTimeoutMs` 声明每个工具的协作式超时预算(附加为 `ToolDefinition.timeoutMs`),由 [`@deepseek-ai/dsh-timeout-policy`](../../timeout/timeout-policy/README.md) 强制执行;面向模型的 schema 不公开超时参数。`fetchMaxOutputChars` 对完整渲染输出设上限markdown 转义可能让转换后的 HTML 超出提供方的主体上限(最坏约 2 倍);默认值本地提供方默认 100,000 字符主体上限的 2 倍,因此绝不会削减该上限本已允许的内容
`fetchTimeoutMs``searchTimeoutMs` 声明每个工具的协作式超时预算(附加为 `ToolDefinition.timeoutMs`),由 [`@deepseek-ai/dsh-timeout-policy`](../../timeout/timeout-policy/README.md) 强制执行;面向模型的 schema 不公开超时参数。`fetchMaxOutputChars` 同时限制同步转换工作量和完整渲染结果:只转换至多该数量的源字符,随后对状态头、转换后的前缀和截断提示合并设限。默认值本地提供方 100,000 字符主体上限留出余量,但渲染膨胀仍可能使最终上限截断结果
```yaml
- id: tool-web
@@ -127,6 +127,6 @@ Use the web_fetch tool to retrieve the content of a specific HTTP(S) URL (for ex
## 已知限制与暂缓事项
- **HTML→markdown 转换在病态输入上回退为原始 HTML**[turndown](https://github.com/mixmark-io/turndown)(带 GFM 表格/删除线)通过真实 DOM 转换抓取到的 HTML但同步遍历在深层未闭合嵌套上呈超线性因此嵌套超过固定 512 层预检上限的主体不经转换原样通过(仍让 turndown 抛异常的输入同样如此),而非阻塞事件循环或报错[决策记录](../../../.agents/notes/implemented/simplification/2026-07-26-turndown-for-tool-web-html-markdown.md))。
- **HTML→markdown 转换会在 GFM 无法安全表示的输入上降级**[turndown](https://github.com/mixmark-io/turndown)(带 GFM 表格/删除线)通过真实 DOM 转换至多 `fetchMaxOutputChars` 个源字符。保守的 512 层词法守卫会将深层或嵌套有歧义的主体作为原始 HTML 直接透传,转换异常也会如此处理;表格的 `colspan` 会被忽略,因为 GFM 无法表示跨列单元格。这些限制可避免阻塞事件循环,也避免不受信任的数值属性使输出膨胀[决策记录](../../../.agents/notes/implemented/simplification/2026-07-26-turndown-for-tool-web-html-markdown.md))。
- **面向模型的表层有意保持最小,提升项暂缓**`max_results` 保持为配置上限(不是模型参数),`web_fetch` 只接受 `url`(没有 `format``prompt`LLM 摘要模式);两项都列为 [seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md) 中的后续步骤。
- **没有 web 专用权限策略**:两个工具都不会请求 `ctx.approval` 就直接执行;需要确认的部署必须添加 `tools/pre-execute` 策略,该包不定义持久 URLdomain 授权。

View File

@@ -30,6 +30,52 @@ const turndown = new TurndownService({
turndown.use(gfm)
turndown.remove(['script', 'style', 'noscript'])
/** Render one GFM table cell without interpreting HTML span counts. */
function renderTableCell(content: string, index: number): string {
const prefix = index === 0 ? '| ' : ' '
const escaped = content.trim().replace(/\n\r/g, '<br>').replace(/\n/g, '<br>').replace(/\|+/g, '\\|').padEnd(3, ' ')
return `${prefix}${escaped} |`
}
/** Whether a row is the table's Markdown heading row. */
function isTableHeadingRow(row: HTMLTableRowElement): boolean {
const cells = Array.from(row.cells)
const section = row.parentElement as HTMLTableSectionElement
const table = section.parentElement as HTMLTableElement
return (section.nodeName === 'THEAD' || table.rows[0] === row)
&& cells.every(cell => cell.nodeName === 'TH')
}
/** Map an HTML table-cell alignment to the GFM separator marker. */
function tableBorder(cell: HTMLTableCellElement): string {
const alignment = (cell.getAttribute('align') || cell.style.textAlign || '').toLowerCase()
if (alignment === 'left') return ':---'
if (alignment === 'right') return '---:'
if (alignment === 'center') return ':---:'
return '---'
}
turndown.addRule('tableCellWithoutSpanExpansion', {
filter: ['th', 'td'],
replacement(content, node) {
const cell = node as HTMLTableCellElement
const row = cell.parentNode as HTMLTableRowElement
// GFM cannot represent spanning cells. Ignoring colspan keeps conversion
// work and output proportional to the source instead of the numeric attribute.
return renderTableCell(content, Array.prototype.indexOf.call(row.childNodes, cell))
},
})
turndown.addRule('tableRowWithoutSpanExpansion', {
filter: 'tr',
replacement(content, node) {
const row = node as HTMLTableRowElement
const border = isTableHeadingRow(row)
? Array.from(row.cells, (cell, index) => renderTableCell(tableBorder(cell), index)).join('')
: ''
return `\n${content}${border.length > 0 ? `\n${border}` : ''}`
},
})
/**
* Validate value constraints the schema DSL can't express: a non-blank `url`.
* Throws a plain `Error` otherwise. No timeout parameter — the tool-call budget
@@ -55,63 +101,142 @@ export function parseFetchArgs(args: { url: string }): { url: string } {
*/
const MAX_CONVERSION_DEPTH = 512
/** Elements that never take a closing tag, so they must not count toward nesting depth. */
/** Elements that never take a closing tag, so they do not grow the lexical stack. */
const VOID_ELEMENTS = new Set([
'area', 'base', 'br', 'col', 'embed', 'hr', 'img', 'input',
'link', 'meta', 'param', 'source', 'track', 'wbr',
])
/** Elements whose contents HTML parses as text until their matching end tag. */
const RAW_TEXT_ELEMENTS = new Set(['script', 'style', 'noscript'])
/** Whether a character can occur after a raw-text end-tag name. */
function isTagBoundary(char: string | undefined): boolean {
return char === undefined || char === '>' || char === '/' || /\s/.test(char)
}
/** Find the matching raw-text end tag without interpreting markup-like body text. */
function findRawTextEnd(lowerHtml: string, name: string, from: number): number {
const prefix = `</${name}`
let candidate = lowerHtml.indexOf(prefix, from)
while (candidate !== -1 && !isTagBoundary(lowerHtml[candidate + prefix.length])) {
candidate = lowerHtml.indexOf(prefix, candidate + prefix.length)
}
return candidate
}
/**
* Estimate the maximum element nesting depth of an HTML string with one linear
* tag scan. Overestimates when markup-like text sits inside `script`/`style`
* bodies or comments (the scan does not parse those), which can only cause a
* spurious raw-HTML fallback, never a missed bound.
* Conservatively reject HTML whose lexical element stack crosses the conversion
* depth ceiling. The single pass ignores closing tags inside comments, skips
* raw-text bodies, respects quoted `>` characters, and only accepts a closing
* tag for the current element; malformed input therefore over-counts rather
* than hiding nesting.
*
* @param html - the decoded HTML body.
* @returns the deepest open-element count the scan reaches.
* @returns whether the body crosses {@link MAX_CONVERSION_DEPTH}.
*/
export function htmlNestingDepth(html: string): number {
let depth = 0
let max = 0
for (const tag of html.matchAll(/<(\/?)([a-zA-Z][a-zA-Z0-9-]*)[^>]*?(\/?)>/g)) {
const [, closing, rawName = '', selfClosing] = tag
const name = rawName.toLowerCase()
if (VOID_ELEMENTS.has(name) || selfClosing === '/') continue
if (closing === '/') {
if (depth > 0) depth -= 1
} else {
depth += 1
if (depth > max) max = depth
function exceedsConversionDepth(html: string): boolean {
const lowerHtml = html.toLowerCase()
const openElements: string[] = []
let offset = 0
let inComment = false
while (offset < html.length) {
const start = html.indexOf('<', offset)
if (inComment) {
const end = html.indexOf('-->', offset)
if (end !== -1 && (start === -1 || end < start)) {
inComment = false
offset = end + 3
continue
}
}
if (start === -1) break
if (!inComment && html.startsWith('<!--', start)) {
inComment = true
offset = start + 4
continue
}
let cursor = start + 1
const closing = html[cursor] === '/'
if (closing) cursor += 1
const nameStart = cursor
while (/[a-zA-Z0-9-]/.test(html[cursor] ?? '')) cursor += 1
if (cursor === nameStart || !/[a-zA-Z]/.test(html.charAt(nameStart))) {
offset = start + 1
continue
}
const name = lowerHtml.slice(nameStart, cursor)
let quote: '"' | "'" | undefined
while (cursor < html.length) {
const char = html[cursor]
cursor += 1
if (quote !== undefined) {
if (char === quote) quote = undefined
} else if (char === '"' || char === "'") {
quote = char
} else if (char === '>') {
break
}
}
if (html[cursor - 1] !== '>') break
if (closing) {
if (!inComment && openElements.at(-1) === name) openElements.pop()
} else {
let last = cursor - 2
while (/\s/.test(html.charAt(last))) last -= 1
if (!VOID_ELEMENTS.has(name) && html[last] !== '/') {
openElements.push(name)
if (openElements.length > MAX_CONVERSION_DEPTH) return true
if (!inComment && RAW_TEXT_ELEMENTS.has(name)) {
const end = findRawTextEnd(lowerHtml, name, cursor)
if (end === -1) break
offset = end
continue
}
}
}
offset = cursor
}
return max
return false
}
interface RenderedBody {
/** Converted text, or raw HTML when conversion is unsafe or fails. */
text: string
/** Whether the source was cut before conversion to bound synchronous work. */
sourceTruncated: boolean
}
/**
* Render a fetched body to model-facing markdown text.
*
* @param body - the decoded body; `html` is converted via turndown, `text`
* passes through verbatim. HTML nested beyond {@link MAX_CONVERSION_DEPTH}
* skips conversion up front (the synchronous walk over such trees is
* superlinear and blocks the event loop past the cooperative timeout), and
* when turndown itself throws the raw HTML passes through instead — a
* degraded page beats an error for a body the provider already decoded.
* @returns the text for the tool's output block.
* passes through verbatim.
* @param maxInputChars - maximum source characters processed synchronously.
* @returns the rendered prefix and whether the source was cut. HTML nested
* beyond {@link MAX_CONVERSION_DEPTH} or rejected by turndown passes through
* raw; a degraded page beats an error for a body the provider decoded.
*/
export function renderBody(body: WebFetchBody): string {
function renderBody(body: WebFetchBody, maxInputChars: number): RenderedBody {
const content = body.content.slice(0, maxInputChars)
const sourceTruncated = content.length !== body.content.length
switch (body.kind) {
case 'html':
if (htmlNestingDepth(body.content) > MAX_CONVERSION_DEPTH) return body.content
if (exceedsConversionDepth(content)) return { text: content, sourceTruncated }
try {
return turndown.turndown(body.content)
return { text: turndown.turndown(content), sourceTruncated }
} catch {
// turndown's DOM walk recurses per element; malformed markup the depth
// scan cannot see can still throw RangeError. Provider errors stay
// turndown's DOM walk recurses per element; malformed markup the lexical
// guard cannot model can still throw RangeError. Provider errors stay
// structured WebErrors upstream; conversion failure downgrades to raw HTML.
return body.content
return { text: content, sourceTruncated }
}
case 'text':
return body.content
return { text: content, sourceTruncated }
/* v8 ignore next 2 -- WebFetchBody is a closed union; this arm is unreachable and only makes adding a kind a compile error. */
default:
return assertNever(body, 'unhandled web fetch body kind')
@@ -123,9 +248,8 @@ const TRUNCATION_FOOTER = '\n\n(Content truncated. Fetch a more specific URL or
/**
* Format a fetch result as one model-facing text block, bounded as a whole.
* Markdown escaping can expand converted HTML (worst case ~2× the provider's
* body cap), so the bound applies here, where the complete output — header,
* rendered body, and footer — is known.
* The same cap limits the source prefix processed synchronously, then applies
* again where the complete output — header, rendered body, and footer — is known.
*
* @param result - the seam's fetch outcome.
* @param maxOutputChars - cap on the complete returned string; a cut body gets
@@ -135,11 +259,13 @@ const TRUNCATION_FOOTER = '\n\n(Content truncated. Fetch a more specific URL or
*/
export function formatFetchOutput(result: WebFetchResult, maxOutputChars: number): string {
const header = `Fetched ${result.url} (HTTP ${result.statusCode})\n\n`
const body = renderBody(result.body)
const full = `${header}${body}${result.truncated ? TRUNCATION_FOOTER : ''}`
const rendered = renderBody(result.body, maxOutputChars)
const prefix = `${header}${rendered.text}`
const truncated = result.truncated || rendered.sourceTruncated || prefix.length > maxOutputChars
const full = `${prefix}${truncated ? TRUNCATION_FOOTER : ''}`
if (full.length <= maxOutputChars) return full
const budget = Math.max(0, maxOutputChars - header.length - TRUNCATION_FOOTER.length)
return `${header}${body.slice(0, budget)}${TRUNCATION_FOOTER}`
if (maxOutputChars < TRUNCATION_FOOTER.length) return full.slice(0, maxOutputChars)
return `${prefix.slice(0, maxOutputChars - TRUNCATION_FOOTER.length)}${TRUNCATION_FOOTER}`
}
/**
@@ -160,8 +286,7 @@ export function presentFetchCall(args: { url: string }): GenericCallView {
* @param timeoutMs - the cooperative tool-call budget (ms) attached as the tool's
* `ToolDefinition.timeoutMs` for `@deepseek-ai/dsh-timeout-policy` to enforce.
* @param maxOutputChars - cap on the complete rendered tool output (see
* {@link formatFetchOutput}); markdown escaping can outgrow the provider's
* body cap, so the model-context bound is enforced on the rendered result.
* {@link formatFetchOutput}) and on source characters converted synchronously.
*/
export function applyWebFetchTool(ctx: Context, timeoutMs: number, maxOutputChars: number): void {
ctx.systemPrompt.section({

View File

@@ -13,7 +13,7 @@ import { applyWebSearchTool, WEB_SEARCH_MAX_RESULTS } from './search.ts'
import { applyWebFetchTool } from './fetch.ts'
export { WEB_SEARCH_MAX_RESULTS, applyWebSearchTool, formatSearchOutput, parseSearchArgs, presentSearchCall } from './search.ts'
export { applyWebFetchTool, formatFetchOutput, htmlNestingDepth, parseFetchArgs, presentFetchCall, renderBody } from './fetch.ts'
export { applyWebFetchTool, formatFetchOutput, parseFetchArgs, presentFetchCall } from './fetch.ts'
/** Cordis plugin name used by loader diagnostics. */
export const name = 'tool-web'
@@ -25,11 +25,9 @@ export const inject = ['tools', 'web', 'systemPrompt']
export const DEFAULT_WEB_TOOL_TIMEOUT_MS = 30_000
/**
* Default cap on one `web_fetch` output's characters. Markdown escaping can
* roughly double converted HTML, so this sits at 2× the local provider's
* default 100,000-char body cap: it never cuts what that composition's
* provider bound already admits, while restoring a model-context bound for
* providers with larger or absent body caps.
* Default cap on one `web_fetch` output and on source characters converted
* synchronously. This leaves headroom above the local provider's default
* 100,000-character body cap while bounding custom providers and rendered output.
*/
export const DEFAULT_FETCH_MAX_OUTPUT_CHARS = 200_000
@@ -45,7 +43,7 @@ export interface Config {
fetchTimeoutMs?: number
/** Cooperative timeout budget (ms) for `web_search`. Defaults to 30000. */
searchTimeoutMs?: number
/** Cap on one `web_fetch` output's characters (header, rendered body, and footer). Defaults to 200000. */
/** Cap on source characters converted and complete `web_fetch` output characters. Defaults to 200000. */
fetchMaxOutputChars?: number
}
@@ -61,7 +59,7 @@ export const Config: z<Config> = z.object({
/** The shape after schemastery applies its defaults to every field. */
type ResolvedConfig = Required<Config>
/** The result cap must be a positive integer (it bounds a provider's source list). */
/** Configured count, timeout, and character caps must be positive integers. */
function assertPositiveInteger(name: string, value: number): void {
if (!Number.isInteger(value) || value < 1) {
throw new Error(`tool-web: ${name} must be a positive integer`)

View File

@@ -10,12 +10,10 @@ import * as ToolWeb from '@deepseek-ai/dsh-tool-web'
import {
formatSearchOutput,
formatFetchOutput,
htmlNestingDepth,
parseSearchArgs,
parseFetchArgs,
presentSearchCall,
presentFetchCall,
renderBody,
WEB_SEARCH_MAX_RESULTS,
} from '@deepseek-ai/dsh-tool-web'
@@ -95,6 +93,11 @@ describe('search formatting', () => {
describe('fetch formatting', () => {
const NO_CAP = 1_000_000
const HEADER = 'Fetched https://a.test (HTTP 200)\n\n'
const renderHtml = (content: string) => formatFetchOutput({
url: 'https://a.test', statusCode: 200, truncated: false,
body: { kind: 'html', content },
}, NO_CAP).slice(HEADER.length)
it('renders an html body to markdown text with a status header', () => {
const out = formatFetchOutput({
@@ -136,29 +139,36 @@ describe('fetch formatting', () => {
url: 'https://a.test', statusCode: 200, truncated: true,
body: { kind: 'text', content: 'abcdef' },
}, 10)
expect(tiny).toContain('Fetched https://a.test (HTTP 200)')
expect(tiny).toContain('Content truncated')
expect(tiny).not.toContain('abcdef')
expect(tiny.length).toBeLessThanOrEqual(10)
expect(tiny).toBe('Fetched ht')
})
it('renderBody dispatches on kind', () => {
expect(renderBody({ kind: 'text', content: 'x' })).toBe('x')
expect(renderBody({ kind: 'html', content: '<p>y</p>' })).toBe('y')
it('dispatches text and html bodies', () => {
expect(formatFetchOutput({
url: 'https://a.test', statusCode: 200, truncated: false,
body: { kind: 'text', content: 'x' },
}, NO_CAP)).toBe(`${HEADER}x`)
expect(renderHtml('<p>y</p>')).toBe('y')
})
it('converts html via turndown: entities, links, tables, nesting; drops script/style/noscript', () => {
expect(renderBody({
kind: 'html',
content: '<style>.x{}</style><script>bad()</script><noscript>ns</noscript><p>Tom &amp; Jerry &copy; R&eacute;sum&eacute;</p><a href="https://a.test">link</a>',
})).toBe('Tom & Jerry © Résumé\n\n[link](https://a.test)')
expect(renderBody({ kind: 'html', content: '<h2>Heading</h2><ul><li>one</li><li>two</li></ul>' }))
expect(renderHtml('<style>.x{}</style><script>bad()</script><noscript>ns</noscript><p>Tom &amp; Jerry &copy; R&eacute;sum&eacute;</p><a href="https://a.test">link</a>'))
.toBe('Tom & Jerry © Résumé\n\n[link](https://a.test)')
expect(renderHtml('<h2>Heading</h2><ul><li>one</li><li>two</li></ul>'))
.toBe('## Heading\n\n- one\n- two')
expect(renderBody({ kind: 'html', content: '<table><tr><th>A</th><th>B</th></tr><tr><td>1</td><td>2</td></tr></table>' }))
expect(renderHtml('<table><tr><th>A</th><th>B</th></tr><tr><td>1</td><td>2</td></tr></table>'))
.toBe('| A | B |\n| --- | --- |\n| 1 | 2 |')
expect(renderBody({ kind: 'html', content: '<p><strong>bold <em>italic</em></strong></p><blockquote><p>quoted</p></blockquote>' }))
expect(renderHtml('<table><thead><tr><th align="left">L</th><th align="right">R</th><th style="text-align:center">C</th></tr></thead><tbody><tr><td>1</td><td>2</td><td>3</td></tr></tbody></table>'))
.toBe('| L | R | C |\n| :--- | ---: | :---: |\n| 1 | 2 | 3 |')
expect(renderHtml('<p><strong>bold <em>italic</em></strong></p><blockquote><p>quoted</p></blockquote>'))
.toBe('**bold _italic_**\n\n> quoted')
})
it('does not expand numeric colspan attributes into unbounded output', () => {
const table = '<table><thead><tr><th colspan="1000000">A</th></tr></thead><tbody><tr><td>B</td></tr></tbody></table>'
expect(renderHtml(table)).toBe('| A |\n| --- |\n| B |')
})
it('passes deeply nested html through raw without attempting conversion', () => {
// Unclosed-tag nesting makes the synchronous conversion superlinear
// (seconds at 20k levels, during which the cooperative timeout cannot
@@ -167,27 +177,73 @@ describe('fetch formatting', () => {
const depth = 20_000
const pathological = '<div>'.repeat(depth) + 'x' + '</div>'.repeat(depth)
const started = Date.now()
expect(renderBody({ kind: 'html', content: pathological })).toBe(pathological)
expect(formatFetchOutput({
url: 'https://a.test', statusCode: 200, truncated: false,
body: { kind: 'html', content: pathological },
}, NO_CAP)).toBe(`${HEADER}${pathological}`)
expect(Date.now() - started).toBeLessThan(2_000)
})
it('htmlNestingDepth counts open elements, ignoring void and self-closing tags', () => {
expect(htmlNestingDepth('<div><p>x</p></div>')).toBe(2)
expect(htmlNestingDepth('<div><br><img src="x"><input/></div>')).toBe(1)
expect(htmlNestingDepth('</div></div><p>x</p>')).toBe(1)
expect(htmlNestingDepth('plain text, no tags')).toBe(0)
expect(htmlNestingDepth('<div>'.repeat(600))).toBe(600)
it('comments and mismatched closing tags cannot hide deep nesting from the preflight', () => {
const pathological = '<div><!-- </div> --></span>'.repeat(600) + 'x'
expect(formatFetchOutput({
url: 'https://a.test', statusCode: 200, truncated: false,
body: { kind: 'html', content: pathological },
}, NO_CAP)).toBe(`${HEADER}${pathological}`)
const abruptlyClosedComments = '<div><!-->'.repeat(600) + 'x'
expect(formatFetchOutput({
url: 'https://a.test', statusCode: 200, truncated: false,
body: { kind: 'html', content: abruptlyClosedComments },
}, NO_CAP)).toBe(`${HEADER}${abruptlyClosedComments}`)
})
it('the preflight accepts ordinary closed, void, self-closing, quoted, and raw-text markup', () => {
const paragraphs = '<p title=\'>\'>x<br ><img src="x"><input/></p>'.repeat(600)
const script = `<script>const invalid = '</scriptx>'; const template = '${'<div>'.repeat(600)}'</script >`
expect(renderHtml(`<!doctype html><?pi><1bad>${paragraphs}${script}`))
.not.toContain('<p')
expect(renderHtml('plain text')).toBe('plain text')
expect(renderHtml('<p>x</p><!-- unfinished')).toBe('x')
expect(renderHtml('<script>unclosed')).toBe('')
expect(renderHtml('<script>closed by slash</script/>')).toBe('')
expect(renderHtml('<script>closed at end</script')).toBe('')
})
it('scans malformed unterminated tags in bounded time', () => {
const malformed = '<a'.repeat(100_000)
const started = Date.now()
const out = formatFetchOutput({
url: 'https://a.test', statusCode: 200, truncated: false,
body: { kind: 'html', content: malformed },
}, 200_000)
expect(out.length).toBeLessThanOrEqual(200_000)
expect(Date.now() - started).toBeLessThan(2_000)
})
it('falls back to the raw html when turndown throws despite a shallow depth scan', () => {
// Comments hide markup from the depth scan by design (it may only
// over-count, never under-count real elements); simulate the residual
// turndown failure path with a converter throw instead.
const spy = vi.spyOn(TurndownService.prototype, 'turndown').mockImplementation(() => {
throw new RangeError('Maximum call stack size exceeded')
})
try {
expect(renderBody({ kind: 'html', content: '<p>x</p>' })).toBe('<p>x</p>')
expect(formatFetchOutput({
url: 'https://a.test', statusCode: 200, truncated: false,
body: { kind: 'html', content: '<p>x</p>' },
}, NO_CAP)).toBe(`${HEADER}<p>x</p>`)
} finally {
spy.mockRestore()
}
})
it('bounds source conversion work before rendering a custom provider body', () => {
const spy = vi.spyOn(TurndownService.prototype, 'turndown').mockReturnValue('converted')
try {
const out = formatFetchOutput({
url: 'https://a.test', statusCode: 200, truncated: false,
body: { kind: 'html', content: `<p>${'x'.repeat(10_000)}</p>` },
}, 500)
expect(spy).toHaveBeenCalledWith(`<p>${'x'.repeat(497)}`)
expect(out.length).toBeLessThanOrEqual(500)
expect(out).toContain('Content truncated')
} finally {
spy.mockRestore()
}
@@ -436,3 +492,35 @@ describe('tool-call timeout budget is plugin config', () => {
.rejects.toThrow(new RegExp(`tool-web: ${key} must be a positive integer`))
})
})
describe('fetchMaxOutputChars is plugin config', () => {
it('bounds the rendered output of the registered web_fetch tool', async () => {
const fetchProvider = {
id: 'stub-fetch',
available: () => available,
fetch: (request: { url: string }) => Promise.resolve({
url: request.url,
statusCode: 200,
body: { kind: 'html' as const, content: `<p>${'_'.repeat(1_000)}</p>` },
truncated: false,
}),
}
const { fiber, call } = await mountTools({
config: { fetchMaxOutputChars: 100 },
webConfig: { fetchProvider: 'stub-fetch' },
fetchProvider,
})
const out = await call('web_fetch', { url: 'https://a.test' })
expect(out.content.map(block => block.type === 'text' ? block.text : '').join('')).toHaveLength(100)
await fiber.dispose()
})
it.each([0, -1, 1.5])('rejects an invalid fetchMaxOutputChars value %s at load', async (value) => {
const ctx = new Context()
await ctx.plugin(SystemPrompt)
await ctx.plugin(ToolRegistry)
await ctx.plugin(WebService, {})
await expect(ctx.plugin(ToolWeb, { fetchMaxOutputChars: value }))
.rejects.toThrow(/tool-web: fetchMaxOutputChars must be a positive integer/)
})
})