feat(tool-web): replace the regex HTML-to-markdown converter with turndown

Implements the turndown Agent Note from the NIH dependency audit (full
variant, not the minimal entities-only fallback): dsh-tool-web's fetch
rendering now converts HTML through turndown + @joplin/turndown-plugin-gfm
(atx headings, fenced code, dash bullets, GFM tables/strikethrough) over
the real domino DOM, with script/style/noscript removed wholesale. The
hand-rolled ~86-line regex converter html.ts and its entity tables are
deleted; renderBody wraps the conversion in try/catch falling back to
the raw HTML body, because turndown's recursive DOM walk overflows with
a RangeError on pathological nesting (measured: 4k levels on the main
thread, 8k in a worker) where the regex version could never throw.

Closure weight, measured: tool-web IS in the single-exe runtime closure,
and the exe asset globs would pack ~7.9 MB of the three new packages —
but ~6 MB of that is domino's test corpus, with runtime lib/ at ~550 KB
against a ~174 MB artifact (<0.5% either way), so the swap wins.

Per testing policy the previously-missing keyless web_fetch snapshot
ships in the same change: the acp-agent `web-fetch` scenario boots a new
web.cordis.yml overlay (web seam + real dsh-web-fetch-local provider +
tool-web fetch-only + a loopback HTTP fixture server on a fixed port
serving deterministic HTML with entities, a GFM table, and nesting), so
recording and keyless replay both drive the real HTTP fetch and real
conversion end to end; the scenario pins the new `web` header class.

The Agent Note moves proposed -> implemented and is rewritten per the
lifecycle contract (Decision/Consequences/Testing, closure verdict and
alternatives recorded); tool-web and acp-agent READMEs updated in both
languages and pairs re-recorded.

packages/web/tool-web/README.i18n.yaml

@@ -2,5 +2,5 @@
 # side as of the last confirmed-consistent state. Both languages carry equal authority;
 # after editing either side, bring the other along and re-record with:
 #   pnpm run verify-translation-pairing --write
-README.md: 5e567115c386d14b7e412ed2502e7290826a5e5e
-README.zh.md: b17fe4107908381806d4029481bbf03696c4f313
+README.md: 5fe48ced81a2cd02197cf8cc10a7d6567b17ffca
+README.zh.md: 34ad08e290166ee6db2cd7b836746541d18aad52

packages/web/tool-web/README.md

@@ -11,7 +11,7 @@ Each tool is registered independently; a product that wants only one disables th
 | Tool | Args | Behavior |
 |---|---|---|
 | `web_search` | `query` (string) | Discovery. Returns an optional answer plus source URLs. `max_results` is **not** model-facing — the tool sets the bound (the `searchMaxResults` config, default 8) and passes it to the seam. |
-| `web_fetch` | `url` (string) | Retrieves a specific URL. HTML bodies are rendered to markdown-ish text; text bodies pass through. A non-2xx status is reported, not an error. The tool-call timeout is deployment policy (`dsh-timeout-policy`), not a model argument. |
+| `web_fetch` | `url` (string) | Retrieves a specific URL. HTML bodies are rendered to markdown (turndown with GFM tables/strikethrough); text bodies pass through. A non-2xx status is reported, not an error. The tool-call timeout is deployment policy (`dsh-timeout-policy`), not a model argument. |
 
 Both tools opt into concurrent scheduling because provider reads return content without mutating parent-agent state.
 
@@ -126,6 +126,6 @@ Append-only; newly visible content follows the reusable request prefix and does
 
 ## Known Limitations and Deferred Work
 
-- **`htmlToMarkdown` is a minimal regex converter, not an HTML parser** — it strips script/style/noscript, keeps headings/bullets/links, and decodes about a dozen named entities; tables, images, and nested formatting are lost.
+- **HTML→markdown conversion falls back to raw HTML on pathological input** — [turndown](https://github.com/mixmark-io/turndown) (with GFM tables/strikethrough) converts fetched HTML through a real DOM, but its recursive walk overflows on absurdly deep nesting (thousands of levels); such a body passes through unconverted rather than erroring ([Agent Note](../../../.agents/notes/implemented/simplification/2026-07-26-turndown-for-tool-web-html-markdown.md)).
 - **The model-facing surface is minimal by design, with promotions deferred** — `max_results` stays a config bound (not a model argument), and `web_fetch` takes only `url` (no `format`/`prompt`/LLM-summarization mode); both are named later steps in [the seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md).
 - **No web-specific permission policy** — both tools execute without requesting `ctx.approval`; a deployment that needs confirmation must add a `tools/pre-execute` policy, and the package does not define persistent URL/domain grants.

packages/web/tool-web/README.zh.md

@@ -11,7 +11,7 @@
 | 工具 | 参数 | 行为 |
 |---|---|---|
 | `web_search` | `query`（string） | 发现。返回可选答案与源 URL。`max_results` **不** 面向模型：工具设置上限（`searchMaxResults` 配置，默认 8）并传给 seam。 |
-| `web_fetch` | `url`（string） | 获取特定 URL。HTML 主体渲染为近似 markdown 的文本；文本主体原样通过。非 2xx 状态会报告，而非报错。工具调用超时是部署策略（`dsh-timeout-policy`），不是模型参数。 |
+| `web_fetch` | `url`（string） | 获取特定 URL。HTML 主体渲染为 markdown（turndown，带 GFM 表格／删除线）；文本主体原样通过。非 2xx 状态会报告，而非报错。工具调用超时是部署策略（`dsh-timeout-policy`），不是模型参数。 |
 
 两个工具都选择并发调度，因为提供方读取会返回内容，不会修改父 agent 状态。
 
@@ -126,6 +126,6 @@ Use the web_fetch tool to retrieve the content of a specific HTTP(S) URL (for ex
 
 ## 已知限制与暂缓事项
 
-- **`htmlToMarkdown` 是最小正则转换器，不是 HTML parser**：它会移除 script/style/noscript，保留标题／项目符号／链接，并解码约十余个命名 entity；表格、图片与嵌套格式会丢失。
+- **HTML→markdown 转换在病态输入上回退为原始 HTML**：[turndown](https://github.com/mixmark-io/turndown)（带 GFM 表格／删除线）通过真实 DOM 转换抓取到的 HTML，但其递归遍历在极深嵌套（数千层）上会栈溢出；此类主体不经转换原样通过，而非报错（[决策记录](../../../.agents/notes/implemented/simplification/2026-07-26-turndown-for-tool-web-html-markdown.md)）。
 - **面向模型的表层有意保持最小，提升项暂缓**：`max_results` 保持为配置上限（不是模型参数），`web_fetch` 只接受 `url`（没有 `format`／`prompt`／LLM 摘要模式）；两项都列为 [seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md) 中的后续步骤。
 - **没有 web 专用权限策略**：两个工具都不会请求 `ctx.approval` 就直接执行；需要确认的部署必须添加 `tools/pre-execute` 策略，该包不定义持久 URL／domain 授权。

packages/web/tool-web/package.json

@@ -35,10 +35,13 @@
     "cordis": "^4.0.0-rc.7"
   },
   "dependencies": {
-    "schemastery": "^3.18.0"
+    "@joplin/turndown-plugin-gfm": "^1.0.67",
+    "schemastery": "^3.18.0",
+    "turndown": "^7.2.4"
   },
   "devDependencies": {
     "@deepseek-ai/dsh-agent": "workspace:^",
+    "@types/turndown": "^5.0.6",
     "@deepseek-ai/dsh-invariants": "workspace:^",
     "@deepseek-ai/dsh-llm": "workspace:^",
     "@deepseek-ai/dsh-session": "workspace:^",

packages/web/tool-web/src/fetch.ts

@@ -6,12 +6,29 @@
  */
 
 import type { Context } from 'cordis'
+import TurndownService from 'turndown'
+import { gfm } from '@joplin/turndown-plugin-gfm'
 import { defineTool } from '@deepseek-ai/dsh-tools'
 import type { GenericCallView } from '@deepseek-ai/dsh-tools'
 import type { WebFetchBody, WebFetchResult } from '@deepseek-ai/dsh-web'
 import { assertNever } from '@deepseek-ai/dsh-llm'
 import type {} from '@deepseek-ai/dsh-system-prompt'
-import { htmlToMarkdown } from './html.ts'
+
+/**
+ * The shared HTML→markdown converter: turndown over its bundled domino DOM,
+ * with GitHub-flavored tables/strikethrough (`@joplin/turndown-plugin-gfm`).
+ * The style options are fixed model-facing presentation (matching the repo's
+ * markdown conventions), not deployment tunables. `remove` drops non-content
+ * elements wholesale — turndown's default keeps their text. The instance is
+ * stateless across `turndown()` calls and safe to share.
+ */
+const turndown = new TurndownService({
+  headingStyle: 'atx',
+  codeBlockStyle: 'fenced',
+  bulletListMarker: '-',
+})
+turndown.use(gfm)
+turndown.remove(['script', 'style', 'noscript'])
 
 /**
  * Validate value constraints the schema DSL can't express: a non-blank `url`.
@@ -30,14 +47,23 @@ export function parseFetchArgs(args: { url: string }): { url: string } {
 /**
  * Render a fetched body to model-facing markdown text.
  *
- * @param body - the decoded body; `html` is converted via
- *   {@link htmlToMarkdown}, `text` passes through verbatim.
+ * @param body - the decoded body; `html` is converted via turndown, `text`
+ *   passes through verbatim. When turndown throws (deeply pathological HTML
+ *   overflows its recursive DOM walk), the raw HTML passes through instead —
+ *   a degraded page beats an error for a body the provider already decoded.
  * @returns the text for the tool's output block.
  */
 export function renderBody(body: WebFetchBody): string {
   switch (body.kind) {
     case 'html':
-      return htmlToMarkdown(body.content)
+      try {
+        return turndown.turndown(body.content)
+      } catch {
+        // turndown's DOM walk recurses per element; pathological nesting (a
+        // few thousand levels) throws RangeError. Provider errors stay
+        // structured WebErrors upstream; conversion failure downgrades to raw HTML.
+        return body.content
+      }
     case 'text':
       return body.content
     /* v8 ignore next 2 -- WebFetchBody is a closed union; this arm is unreachable and only makes adding a kind a compile error. */

packages/web/tool-web/src/html.ts

@@ -1,86 +0,0 @@
-/**
- * Minimal dependency-free HTML-to-readable-text conversion for `web_fetch`, not a full parser. It
- * removes non-content elements and tags, decodes common entities, collapses whitespace, and keeps
- * basic headings, lists, and links. A richer converter can replace it without changing the seam or
- * tool schema.
- * @module @deepseek-ai/dsh-tool-web/html
- */
-
-/** Decode the handful of HTML entities common in textual content. */
-function decodeEntities(text: string): string {
-  return text
-    .replace(/&(#[xX][0-9a-fA-F]+|#[0-9]+|[a-zA-Z]+);/g, (match, entity: string) => {
-      if (entity.startsWith('#x') || entity.startsWith('#X')) {
-        const code = Number.parseInt(entity.slice(2), 16)
-        return safeFromCodePoint(code, match)
-      }
-      if (entity.startsWith('#')) {
-        const code = Number.parseInt(entity.slice(1), 10)
-        return safeFromCodePoint(code, match)
-      }
-      return NAMED_ENTITIES[entity] ?? match
-    })
-}
-
-const NAMED_ENTITIES: Record<string, string> = {
-  amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ',
-  copy: '©', reg: '®', trade: '™', hellip: '…', mdash: '—', ndash: '–',
-}
-
-function safeFromCodePoint(code: number, fallback: string): string {
-  try {
-    return String.fromCodePoint(code)
-  } catch {
-    // An out-of-range code point (RangeError) is the only failure here; keep the
-    // original entity text rather than throwing out of pure presentation.
-    return fallback
-  }
-}
-
-/**
- * Convert an HTML document to a readable markdown-ish text approximation.
- * Best-effort and lossy by design — fidelity is the job of a future heavier
- * converter, not this fallback.
- *
- * @param html - the raw HTML source.
- * @returns plain text with markdown headings, list bullets, and links;
- *   whitespace collapsed to at most one blank line and trimmed.
- */
-export function htmlToMarkdown(html: string): string {
-  let text = html
-    // Drop non-content elements entirely (including their contents).
-    .replace(/<script\b[^>]*>[\s\S]*?<\/script>/gi, '')
-    .replace(/<style\b[^>]*>[\s\S]*?<\/style>/gi, '')
-    .replace(/<noscript\b[^>]*>[\s\S]*?<\/noscript>/gi, '')
-    .replace(/<!--[\s\S]*?-->/g, '')
-
-  // Convert links to markdown before stripping tags.
-  text = text.replace(/<a\b[^>]*\bhref\s*=\s*["']([^"']*)["'][^>]*>([\s\S]*?)<\/a>/gi, (_match, href: string, label: string) => {
-    const cleanLabel = label.replace(/<[^>]+>/g, '').trim()
-    return cleanLabel.length > 0 ? `[${cleanLabel}](${href})` : href
-  })
-
-  // Headings → markdown hashes.
-  text = text.replace(/<h([1-6])\b[^>]*>([\s\S]*?)<\/h\1>/gi, (_match, level: string, body: string) => {
-    const hashes = '#'.repeat(Number(level))
-    return `\n\n${hashes} ${body.replace(/<[^>]+>/g, '').trim()}\n\n`
-  })
-
-  // List items → bullets.
-  text = text.replace(/<li\b[^>]*>([\s\S]*?)<\/li>/gi, (_match, body: string) => `\n- ${body.replace(/<[^>]+>/g, '').trim()}`)
-
-  // Block-level breaks become paragraph breaks.
-  text = text
-    .replace(/<\/(p|div|section|article|header|footer|tr|table|ul|ol|blockquote)>/gi, '\n\n')
-    .replace(/<br\s*\/?>/gi, '\n')
-
-  // Drop all remaining tags, decode entities, collapse whitespace.
-  text = text.replace(/<[^>]+>/g, '')
-  text = decodeEntities(text)
-  text = text
-    .replace(/[ \t\f\v]+/g, ' ')
-    .replace(/ *\n */g, '\n')
-    .replace(/\n{3,}/g, '\n\n')
-    .trim()
-  return text
-}

packages/web/tool-web/src/index.ts

@@ -14,7 +14,6 @@ import { applyWebFetchTool } from './fetch.ts'
 
 export { WEB_SEARCH_MAX_RESULTS, applyWebSearchTool, formatSearchOutput, parseSearchArgs, presentSearchCall } from './search.ts'
 export { applyWebFetchTool, formatFetchOutput, parseFetchArgs, presentFetchCall, renderBody } from './fetch.ts'
-export { htmlToMarkdown } from './html.ts'
 
 /** Cordis plugin name used by loader diagnostics. */
 export const name = 'tool-web'

packages/web/tool-web/src/turndown-plugin-gfm.d.ts

@@ -0,0 +1,12 @@
+/**
+ * Ambient module declaration for `@joplin/turndown-plugin-gfm`, which ships no
+ * types and has no DefinitelyTyped package. Only the composite `gfm` plugin is
+ * declared; the package's individual plugins (`tables`, `strikethrough`, …)
+ * stay undeclared until something imports them.
+ */
+declare module '@joplin/turndown-plugin-gfm' {
+  import type TurndownService from 'turndown'
+
+  /** The composite GitHub-flavored-markdown plugin (tables, strikethrough, task lists, highlighted code blocks). */
+  export const gfm: TurndownService.Plugin
+}

packages/web/tool-web/tests/tool-web.spec.ts

@@ -14,7 +14,6 @@ import {
   presentSearchCall,
   presentFetchCall,
   renderBody,
-  htmlToMarkdown,
   WEB_SEARCH_MAX_RESULTS,
 } from '@deepseek-ai/dsh-tool-web'
 
@@ -82,6 +81,11 @@ describe('search formatting', () => {
     expect(parseSearchArgs({ query: 'hi' })).toEqual({ query: 'hi' })
   })
 
+  it('falls back to the raw URL as a source label when the URL is unparseable', () => {
+    const out = formatSearchOutput({ truncated: false, sources: [{ url: 'not a url' }] })
+    expect(out).toContain('[not a url](not a url)')
+  })
+
   it('presents a search call as a search-kind card titled by the query', () => {
     expect(presentSearchCall({ query: 'find me' })).toEqual({ card: 'generic', title: 'find me', kind: 'search', rawInput: 'find me' })
   })
@@ -112,6 +116,29 @@ describe('fetch formatting', () => {
     expect(renderBody({ kind: 'html', content: '<p>y</p>' })).toBe('y')
   })
 
+  it('converts html via turndown: entities, links, tables, nesting; drops script/style/noscript', () => {
+    expect(renderBody({
+      kind: 'html',
+      content: '<style>.x{}</style><script>bad()</script><noscript>ns</noscript><p>Tom &amp; Jerry &copy; R&eacute;sum&eacute;</p><a href="https://a.test">link</a>',
+    })).toBe('Tom & Jerry © Résumé\n\n[link](https://a.test)')
+    expect(renderBody({ kind: 'html', content: '<h2>Heading</h2><ul><li>one</li><li>two</li></ul>' }))
+      .toBe('## Heading\n\n-   one\n-   two')
+    expect(renderBody({ kind: 'html', content: '<table><tr><th>A</th><th>B</th></tr><tr><td>1</td><td>2</td></tr></table>' }))
+      .toBe('| A   | B   |\n| --- | --- |\n| 1   | 2   |')
+    expect(renderBody({ kind: 'html', content: '<p><strong>bold <em>italic</em></strong></p><blockquote><p>quoted</p></blockquote>' }))
+      .toBe('**bold _italic_**\n\n> quoted')
+  })
+
+  it('falls back to the raw html body when turndown throws on pathological nesting', { timeout: 60_000 }, () => {
+    // Nesting past V8's default stack overflows turndown/domino's recursive
+    // walk with a RangeError (measured: 4k levels throw on the main thread,
+    // 8k in a worker); 20k adds margin over either stack size. The raw body
+    // must pass through instead of throwing.
+    const depth = 20_000
+    const pathological = '<div>'.repeat(depth) + 'x' + '</div>'.repeat(depth)
+    expect(renderBody({ kind: 'html', content: pathological })).toBe(pathological)
+  })
+
   it('validates url (non-empty), no timeout parameter', () => {
     expect(() => parseFetchArgs({ url: ' ' })).toThrow('non-empty')
     expect(parseFetchArgs({ url: 'https://a.test' })).toEqual({ url: 'https://a.test' })
@@ -122,46 +149,6 @@ describe('fetch formatting', () => {
   })
 })
 
-describe('htmlToMarkdown', () => {
-  it('drops scripts/styles, keeps text, decodes entities, converts links', () => {
-    const md = htmlToMarkdown('<style>.x{}</style><script>bad()</script><p>Tom &amp; Jerry</p><a href="https://a.test">link</a>')
-    expect(md).not.toContain('bad()')
-    expect(md).not.toContain('.x{}')
-    expect(md).toContain('Tom & Jerry')
-    expect(md).toContain('[link](https://a.test)')
-  })
-
-  it('decodes numeric entities and collapses whitespace', () => {
-    expect(htmlToMarkdown('<p>a&#39;b</p>')).toBe("a'b")
-    expect(htmlToMarkdown('<div>x</div>\n\n\n<div>y</div>')).toBe('x\n\ny')
-  })
-
-  it('decodes hex entities and named entities, and leaves unknown/out-of-range ones intact', () => {
-    expect(htmlToMarkdown('<p>&#x41;&#X42;</p>')).toBe('AB')
-    expect(htmlToMarkdown('<p>&copy; &mdash;</p>')).toBe('© —')
-    expect(htmlToMarkdown('<p>&notareal;</p>')).toBe('&notareal;')
-    // An out-of-range code point keeps the original entity text (fromCodePoint fallback).
-    expect(htmlToMarkdown('<p>&#x110000;</p>')).toBe('&#x110000;')
-    expect(htmlToMarkdown('<p>&#1114112;</p>')).toBe('&#1114112;')
-  })
-
-  it('renders a link with an empty label as its bare href', () => {
-    expect(htmlToMarkdown('<a href="https://a.test"></a>')).toBe('https://a.test')
-  })
-
-  it('converts headings and list items to markdown', () => {
-    expect(htmlToMarkdown('<h2>Heading</h2><p>after</p>')).toContain('## Heading')
-    const list = htmlToMarkdown('<ul><li>one</li><li>two</li></ul>')
-    expect(list).toContain('- one')
-    expect(list).toContain('- two')
-  })
-
-  it('falls back to the raw URL as a source label when the URL is unparseable', () => {
-    const out = formatSearchOutput({ truncated: false, sources: [{ url: 'not a url' }] })
-    expect(out).toContain('[not a url](not a url)')
-  })
-})
-
 describe('tool-web registration', () => {
   it('registers both tools by default', async () => {
     const { fiber, ctx } = await mountTools()