feat(tool-web): replace the regex HTML-to-markdown converter with turndown

Implements the turndown Agent Note from the NIH dependency audit (full
variant, not the minimal entities-only fallback): dsh-tool-web's fetch
rendering now converts HTML through turndown + @joplin/turndown-plugin-gfm
(atx headings, fenced code, dash bullets, GFM tables/strikethrough) over
the real domino DOM, with script/style/noscript removed wholesale. The
hand-rolled ~86-line regex converter html.ts and its entity tables are
deleted; renderBody wraps the conversion in try/catch falling back to
the raw HTML body, because turndown's recursive DOM walk overflows with
a RangeError on pathological nesting (measured: 4k levels on the main
thread, 8k in a worker) where the regex version could never throw.

Closure weight, measured: tool-web IS in the single-exe runtime closure,
and the exe asset globs would pack ~7.9 MB of the three new packages —
but ~6 MB of that is domino's test corpus, with runtime lib/ at ~550 KB
against a ~174 MB artifact (<0.5% either way), so the swap wins.

Per testing policy the previously-missing keyless web_fetch snapshot
ships in the same change: the acp-agent `web-fetch` scenario boots a new
web.cordis.yml overlay (web seam + real dsh-web-fetch-local provider +
tool-web fetch-only + a loopback HTTP fixture server on a fixed port
serving deterministic HTML with entities, a GFM table, and nesting), so
recording and keyless replay both drive the real HTTP fetch and real
conversion end to end; the scenario pins the new `web` header class.

The Agent Note moves proposed -> implemented and is rewritten per the
lifecycle contract (Decision/Consequences/Testing, closure verdict and
alternatives recorded); tool-web and acp-agent READMEs updated in both
languages and pairs re-recorded.
This commit is contained in:
Tianyi Cui
2026-07-27 00:22:13 +08:00
parent 05adf5da4a
commit 45a5175e44
29 changed files with 962 additions and 210 deletions

View File

@@ -6,12 +6,29 @@
*/
import type { Context } from 'cordis'
import TurndownService from 'turndown'
import { gfm } from '@joplin/turndown-plugin-gfm'
import { defineTool } from '@deepseek-ai/dsh-tools'
import type { GenericCallView } from '@deepseek-ai/dsh-tools'
import type { WebFetchBody, WebFetchResult } from '@deepseek-ai/dsh-web'
import { assertNever } from '@deepseek-ai/dsh-llm'
import type {} from '@deepseek-ai/dsh-system-prompt'
import { htmlToMarkdown } from './html.ts'
/**
* The shared HTML→markdown converter: turndown over its bundled domino DOM,
* with GitHub-flavored tables/strikethrough (`@joplin/turndown-plugin-gfm`).
* The style options are fixed model-facing presentation (matching the repo's
* markdown conventions), not deployment tunables. `remove` drops non-content
* elements wholesale — turndown's default keeps their text. The instance is
* stateless across `turndown()` calls and safe to share.
*/
const turndown = new TurndownService({
headingStyle: 'atx',
codeBlockStyle: 'fenced',
bulletListMarker: '-',
})
turndown.use(gfm)
turndown.remove(['script', 'style', 'noscript'])
/**
* Validate value constraints the schema DSL can't express: a non-blank `url`.
@@ -30,14 +47,23 @@ export function parseFetchArgs(args: { url: string }): { url: string } {
/**
* Render a fetched body to model-facing markdown text.
*
* @param body - the decoded body; `html` is converted via
* {@link htmlToMarkdown}, `text` passes through verbatim.
* @param body - the decoded body; `html` is converted via turndown, `text`
* passes through verbatim. When turndown throws (deeply pathological HTML
* overflows its recursive DOM walk), the raw HTML passes through instead —
* a degraded page beats an error for a body the provider already decoded.
* @returns the text for the tool's output block.
*/
export function renderBody(body: WebFetchBody): string {
switch (body.kind) {
case 'html':
return htmlToMarkdown(body.content)
try {
return turndown.turndown(body.content)
} catch {
// turndown's DOM walk recurses per element; pathological nesting (a
// few thousand levels) throws RangeError. Provider errors stay
// structured WebErrors upstream; conversion failure downgrades to raw HTML.
return body.content
}
case 'text':
return body.content
/* v8 ignore next 2 -- WebFetchBody is a closed union; this arm is unreachable and only makes adding a kind a compile error. */