docs: adopt dependencies-over-hand-rolling policy from NIH audit
A repo-wide Not Invented Here audit (ten parallel domain surveys covering every package group, scripts/, native/, vendor/ edges, python/, test infrastructure, and CI) asked of each hand-rolled surface whether a maintained external package or Node builtin deletes it with a net win. Policy: new implemented process note records that a dependency which genuinely deletes owned code is a preferred simplification (bar: net deletion, health, boundary fit, settled seams stay); root AGENTS.md carries the one-line rule and dsh-find-simplifications now surveys for hand-rolled-where-a-dependency-exists candidates. Findings, all bilingual from birth: - proposed/simplification: eventsource-parser for llm-deepseek SSE, node:timers/promises for three hand-rolled sleeps, turndown (or minimal 'entities') for tool-web HTML->markdown, gate-script consolidation onto mdast/parseArgs/globSync - proposed/testing: execa + parseArgs + loadEnvFile + vi.waitFor for hand-rolled test subprocess plumbing - proposed/process: pnpm/action-setup for symmetric CI caching - proposed/feature: evaluate landstrip before building a Windows sandbox launcher - rejected/simplification: ~30 swap verdicts recorded (vscode-jsonrpc, p-retry, Ajv, write-file-atomic, msw, hono, better-sqlite3, wireit, landstrip-for-linux, YAML consolidation, ...) so the survey is not re-litigated from scratch Also drops the stale prompt/ entry from the AGENTS.md layout map (workspace instructions live in packages/context/workspace-context).
This commit is contained in:
Tianyi Cui
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-26-builtin-timer-promises-for-hand-rolled-sleeps.md: 036e2f2906ca99aaab30a2164649f9c750b4ad21
|
||||
2026-07-26-builtin-timer-promises-for-hand-rolled-sleeps.zh.md: 15a6f0dd412d142647df2722335a454a028cb798
|
||||
@@ -0,0 +1,37 @@
|
||||
# Agent Note: Use node:timers/promises for hand-rolled cancellable sleeps
|
||||
|
||||
Status: proposed
|
||||
|
||||
English | [中文](2026-07-26-builtin-timer-promises-for-hand-rolled-sleeps.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
Three packages hand-roll promise-wrapped timers that the `node:timers/promises` builtin already provides, while other packages (`dsh-llm-mock-server` `pause()`, `dsh-lsp-local`, `dsh-acp-snapshot`) already use the builtin — so the hand-rolled copies are also a consistency gap:
|
||||
|
||||
- `packages/llm/llm-retry/src/index.ts` `cancellableDelay()` (~14 lines): `new Promise` + `setTimeout` + manual abort-listener add/remove, resolving `true` on elapse and `false` on abort, consumed once for the backoff wait.
|
||||
- `packages/workflow/workflow-workerthread/src/host.ts` `sleep()` (~7 lines): promise-wrapped unref'd `setTimeout` used as the dispose-grace bound.
|
||||
- `packages/pty/pty-local/src/session.ts` `delay()` (~4 lines): bare promise-wrapped `setTimeout` used in polling/teardown waits.
|
||||
|
||||
## Proposal
|
||||
|
||||
Replace both with `import { setTimeout } from 'node:timers/promises'`:
|
||||
|
||||
- llm-retry: `try { await setTimeout(delayMs, undefined, { signal }); /* retry */ } catch { /* abort → fail */ }` — with a signal, the promise rejects only with the abort error, and a pre-aborted signal rejects immediately; behavior is identical, including timer clearing on abort. The empty `catch` names the abort rejection per the repo's empty-catch rule.
|
||||
- workflow-workerthread: `setTimeout(ms, undefined, { ref: false })` — exact semantics including not holding the event loop open.
|
||||
- pty-local: `import { setTimeout as delay } from 'node:timers/promises'` — identical signature, call sites unchanged.
|
||||
|
||||
No dedicated tests pin the helpers themselves; the packages' behavior suites keep passing.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **`p-timeout`/`p-defer` style packages.** Rejected: the builtin covers both call sites exactly; an external package for a one-line await is negative-net.
|
||||
- **Leave them.** Rejected only weakly — the cost is small, but the repo already uses the builtin idiom elsewhere, and two hand-rolled variants of a builtin invite a third.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Neither package defines a promise-wrapped `setTimeout` helper; both import from `node:timers/promises`.
|
||||
- `llm-retry` and `workflow-workerthread` test suites pass unchanged (behavioral parity).
|
||||
|
||||
## Risks
|
||||
|
||||
Essentially none: no model-visible output, no platform concerns, no new dependency. The llm-retry rewrite changes a boolean-returning helper into try/catch control flow — a local readability judgment the implementing PR makes.
|
||||
@@ -0,0 +1,37 @@
|
||||
# Agent Note: 用 node:timers/promises 替代手写的可取消休眠
|
||||
|
||||
Status: proposed
|
||||
|
||||
[English](2026-07-26-builtin-timer-promises-for-hand-rolled-sleeps.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
三个包（package）手写了 promise 包装的定时器，而 `node:timers/promises` 内置模块早已提供同等能力；其他包（`dsh-llm-mock-server` 的 `pause()`、`dsh-lsp-local`、`dsh-acp-snapshot`）已经在使用该内置模块，因此这些手写副本同时也是一处一致性缺口：
|
||||
|
||||
- `packages/llm/llm-retry/src/index.ts` 的 `cancellableDelay()`（约 14 行）：`new Promise` + `setTimeout` + 手动添加/移除 abort 监听器，计时走完时 resolve 为 `true`、被中止时 resolve 为 `false`，仅在退避等待处消费一次。
|
||||
- `packages/workflow/workflow-workerthread/src/host.ts` 的 `sleep()`（约 7 行）：promise 包装、已 unref 的 `setTimeout`，用作 dispose（资源释放）宽限的时间上界。
|
||||
- `packages/pty/pty-local/src/session.ts` 的 `delay()`（约 4 行）：朴素的 promise 包装 `setTimeout`，用于轮询与拆除等待。
|
||||
|
||||
## 提案
|
||||
|
||||
用 `import { setTimeout } from 'node:timers/promises'` 替换上述实现：
|
||||
|
||||
- llm-retry：`try { await setTimeout(delayMs, undefined, { signal }); /* retry */ } catch { /* abort → fail */ }`。传入 signal 后，该 promise 只会以 abort 错误拒绝，已提前中止的 signal 则立即拒绝；行为完全一致，包括中止时清除定时器。按仓库的空 catch 规则，这个空 `catch` 注明其吞下的是 abort 拒绝。
|
||||
- workflow-workerthread：`setTimeout(ms, undefined, { ref: false })`，语义完全等价，包括不会让事件循环保持存活。
|
||||
- pty-local：`import { setTimeout as delay } from 'node:timers/promises'`，签名完全相同，调用点无需改动。
|
||||
|
||||
没有专属测试固定这些辅助函数本身；各包的行为测试套件继续通过。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
- **`p-timeout`/`p-defer` 一类的包。** 不予采纳：内置模块恰好精确覆盖这些调用点；为一行 await 引入外部包是负收益。
|
||||
- **维持现状。** 不予采纳，但理由较弱：成本确实很小，但仓库其他地方已经在用这一内置惯用法，而同一内置能力存在两个手写变体，就会招来第三个。
|
||||
|
||||
## 验收标准
|
||||
|
||||
- 上述包不再各自定义 promise 包装的 `setTimeout` 辅助函数，而是都从 `node:timers/promises` 导入。
|
||||
- `llm-retry` 与 `workflow-workerthread` 的测试套件原样通过（行为等价）。
|
||||
|
||||
## 风险
|
||||
|
||||
基本没有风险：不涉及模型可见的输出，没有平台顾虑，也不新增依赖。llm-retry 的改写把一个返回布尔值的辅助函数变成 try/catch 控制流，这是一项局部可读性判断，由实施 PR（Pull Request）裁量。
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-26-consolidate-gate-scripts-on-existing-deps.md: 2b6c2f80b4fc3d3bf818b6789b5f40bb7a61b654
|
||||
2026-07-26-consolidate-gate-scripts-on-existing-deps.zh.md: b20a5bd9ba1661321721c0c9d62de8dc63ec645b
|
||||
@@ -0,0 +1,38 @@
|
||||
# Agent Note: Consolidate gate scripts on already-present deps and builtins
|
||||
|
||||
Status: proposed
|
||||
|
||||
English | [中文](2026-07-26-consolidate-gate-scripts-on-existing-deps.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The `scripts/` gates mostly use the right tools (`node:fs` `globSync` in 15+ gates, mdast/micromark in the markdown gates), but a handful of stragglers hand-roll what a sibling gate already does with an existing dependency or builtin:
|
||||
|
||||
- **Duplicated fence scanners.** `scripts/md-fences.ts` (~55 lines, consumed by `doc-typecheck.ts`) and `extractEquivBlocks` in `scripts/verify-type-equiv.ts` (~39 lines) are two copies of the same regex line-scanner for fenced code blocks, while `scripts/verify-mermaid.ts` already extracts fences by visiting mdast `code` nodes via the shared `scripts/markdown.ts` helpers — and `markdownProseLines` in `markdown.ts` itself parses to mdast but then hand-tracks fence state with a second regex. The regex scanners only recognize backtick fences at column 0, so they silently disagree with the mdast-based gates on tilde and indented fences.
|
||||
- **Hand-rolled argv parsing.** `parseOptions` in `scripts/publint-all.ts` and its near-identical copy in `scripts/verify-built-package-invariants.mjs` (~26 lines) step argv indexes manually, while sibling scripts (`verify-runtime-closure.ts`, `build-exe-for-python-sdk.ts`, `packages/sdk/scripts/src/args.ts`) already use the `node:util` `parseArgs` builtin.
|
||||
- **Hand-rolled directory walks.** Five sites re-derive nested `readdirSync` walks that `globSync` covers: `verify-runtime-closure.ts` (packages + vendor manifests), `dev-web.ts` `discoverPluginDirs`, `verify-package-paths.ts` `realPackageNames`, `verify-client-domain-graph.ts` `listSources`, and `publint-all.ts` `addPath` (~55–65 lines total). `scripts/package-invariants.ts` shows the one-line `globSync` template.
|
||||
|
||||
No new dependency is needed anywhere; every replacement is an existing devDep or a Node builtin.
|
||||
|
||||
## Proposal
|
||||
|
||||
- Extract a shared ~10–15-line mdast fence helper (visiting `code` nodes for `lang`, `meta`, `value`, `position.start.line`) into `scripts/markdown.ts`; rewrite `doc-typecheck.ts` and `verify-type-equiv.ts` onto it; delete `md-fences.ts` and the duplicated scanner; drop the redundant fence regex in `markdownProseLines`.
|
||||
- Replace both `parseOptions` copies with `parseArgs`.
|
||||
- Replace the five straggler walks with `globSync`. Keep the walks in `check-workspace-constraints.ts` and `clean.ts`: they need dirent-level detail to diagnose malformed trees, which glob-by-pattern cannot report.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **A new glob/walking dependency (`tinyglobby`, `fdir`).** Rejected: the builtin already won repo-wide; these are stragglers, not a gap.
|
||||
- **`p-map` for `publint-all.ts`'s ~19-line ordered worker pool.** Deliberately left out: one new devDep for one small deletion is at the edge of the [dependency policy](../../implemented/process/2026-07-26-dependencies-over-hand-rolling.md) bar, and the pool's requirements (bounded workers, deterministic order, env override) are documented in the [parallel-gates note](../../implemented/process/2026-07-06-parallel-pre-push-gates.md). Fold it in only if `p-map` earns a second consumer.
|
||||
- **Leaving the fence scanners.** Rejected: two drifting copies of a parser beside a third correct implementation is exactly the duplication the shared `markdown.ts` helper exists to prevent, and the column-0-backtick-only limitation is a latent inconsistency between sibling gates.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `md-fences.ts` is gone; `doc-typecheck` and `verify-type-equiv` extract fences through `scripts/markdown.ts`; `pnpm run doc-sync` passes with unchanged results on the current tree (any delta traces to a fence shape the regex scanners mishandled).
|
||||
- Both CLIs parse via `parseArgs`; unknown options still fail loud.
|
||||
- The five walk sites use `globSync`; the gates they feed pass unchanged.
|
||||
|
||||
## Risks
|
||||
|
||||
- Behavioral deltas on pathological markdown: mdast honors tilde/indented fences the regex scanners ignored, so `doc-typecheck`'s opt-out ratio could shift if any stray fence shape exists in the docs tree; verify by running `doc-sync` before/after.
|
||||
- `parseArgs` keeps the last value of a duplicated option instead of erroring and consumes a `--`-prefixed next token as a value; both are dev-tool edge cases the tests don't pin.
|
||||
@@ -0,0 +1,38 @@
|
||||
# Agent Note: 把门禁脚本统一到已有依赖与内置模块上
|
||||
|
||||
Status: proposed
|
||||
|
||||
[English](2026-07-26-consolidate-gate-scripts-on-existing-deps.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
`scripts/` 下的门禁大多已经在用正确的工具（15 个以上的门禁使用 `node:fs` 的 `globSync`，markdown 门禁使用 mdast/micromark），但少数几个掉队的脚本仍在手写同类门禁早已用既有依赖或内置模块完成的事情：
|
||||
|
||||
- **重复的围栏扫描器。**`scripts/md-fences.ts`（约 55 行，由 `doc-typecheck.ts` 消费）和 `scripts/verify-type-equiv.ts` 中的 `extractEquivBlocks`（约 39 行）是同一个围栏代码块正则行扫描器的两份拷贝，而 `scripts/verify-mermaid.ts` 已经通过共享的 `scripts/markdown.ts` 辅助函数访问 mdast `code` 节点来提取代码围栏；`markdown.ts` 自己的 `markdownProseLines` 也是先解析成 mdast，再用第二个正则手工跟踪围栏状态。这两个正则扫描器只识别第 0 列的反引号围栏，因此在波浪线围栏和缩进围栏上与基于 mdast 的门禁悄悄不一致。
|
||||
- **手写的 argv 解析。**`scripts/publint-all.ts` 中的 `parseOptions` 和 `scripts/verify-built-package-invariants.mjs` 中与之几乎相同的拷贝（约 26 行）手工推进 argv 下标，而同类脚本（`verify-runtime-closure.ts`、`build-exe-for-python-sdk.ts`、`packages/sdk/scripts/src/args.ts`）已经在使用 `node:util` 的内置 `parseArgs`。
|
||||
- **手写的目录遍历。**五处代码各自重写了 `globSync` 已覆盖的嵌套 `readdirSync` 遍历：`verify-runtime-closure.ts` 对 packages 与 vendor manifest（元数据清单）的扫描、`dev-web.ts` 的 `discoverPluginDirs`、`verify-package-paths.ts` 的 `realPackageNames`、`verify-client-domain-graph.ts` 的 `listSources`，以及 `publint-all.ts` 的 `addPath`（合计约 55–65 行）。`scripts/package-invariants.ts` 展示了一行式的 `globSync` 模板。
|
||||
|
||||
所有替换都不需要引入新依赖；每一处替换用的都是既有的 devDependency 或 Node 内置模块。
|
||||
|
||||
## 提案
|
||||
|
||||
- 在 `scripts/markdown.ts` 中提取一个约 10–15 行的共享 mdast 围栏辅助函数（访问 `code` 节点，读取 `lang`、`meta`、`value`、`position.start.line`）；把 `doc-typecheck.ts` 和 `verify-type-equiv.ts` 改写到它上面；删除 `md-fences.ts` 和重复的扫描器；去掉 `markdownProseLines` 中冗余的围栏正则。
|
||||
- 用 `parseArgs` 替换两份 `parseOptions` 拷贝。
|
||||
- 用 `globSync` 替换那五处掉队的目录遍历。保留 `check-workspace-constraints.ts` 和 `clean.ts` 中的遍历：它们需要 dirent 级别的细节来诊断结构异常的目录树，按模式匹配的 glob 报告不了这些信息。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
- **新的 glob/目录遍历依赖（`tinyglobby`、`fdir`）。**不予采纳：内置模块已在全仓库范围内胜出；这几处只是掉队者，不是能力缺口。
|
||||
- **用 `p-map` 替换 `publint-all.ts` 中约 19 行的有序 worker 池。**刻意未纳入：为一次小删除引入一个新 devDependency，正处在[依赖策略](../../implemented/process/2026-07-26-dependencies-over-hand-rolling.md)门槛的边缘，而且该池的需求（worker 数量有界、确定性顺序、环境变量覆盖）已记录在[并行 pre-push 门禁决策记录](../../implemented/process/2026-07-06-parallel-pre-push-gates.md)中。仅当 `p-map` 赢得第二个消费方时再顺带纳入。
|
||||
- **保留这两个围栏扫描器。**不予采纳：在第三个正确实现旁边放着两份逐渐漂移的解析器拷贝，正是共享的 `markdown.ts` 辅助函数要防止的那种重复；「只认第 0 列反引号」的限制也是同类门禁之间的潜在不一致。
|
||||
|
||||
## 验收标准
|
||||
|
||||
- `md-fences.ts` 已删除；`doc-typecheck` 与 `verify-type-equiv` 通过 `scripts/markdown.ts` 提取代码围栏；`pnpm run doc-sync` 在当前代码树上通过且结果不变（如有差异，必须能追溯到正则扫描器处理有误的某种围栏形态）。
|
||||
- 两个 CLI 都改用 `parseArgs` 解析；未知选项仍然大声失败。
|
||||
- 五处遍历代码改用 `globSync`；它们供给的门禁保持原样通过。
|
||||
|
||||
## 风险
|
||||
|
||||
- 病态 markdown 上的行为差异：mdast 会承认正则扫描器忽略的波浪线围栏和缩进围栏，因此如果文档树中存在任何零散的此类围栏形态，`doc-typecheck` 的 opt-out 比例可能变化；应在改动前后分别运行 `doc-sync` 加以验证。
|
||||
- `parseArgs` 对重复出现的选项保留最后一个值而不报错，还会把下一个以 `--` 开头的 token 当作值消费；这两种情况都是测试未固定的开发工具边缘用例。
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-26-eventsource-parser-for-deepseek-sse.md: 8a93b7f6c7aa0d428f25e87c44e1d29e884ecc81
|
||||
2026-07-26-eventsource-parser-for-deepseek-sse.zh.md: b16109d9458f487c7e463cf02e6b2d22fbbde015
|
||||
@@ -0,0 +1,33 @@
|
||||
# Agent Note: Replace the hand-rolled SSE parser in llm-deepseek with eventsource-parser
|
||||
|
||||
Status: proposed
|
||||
|
||||
English | [中文](2026-07-26-eventsource-parser-for-deepseek-sse.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
`packages/llm/llm-deepseek/src/sse.ts` hand-implements Server-Sent Events parsing: a streaming `TextDecoder`, event-block splitting on `\r?\n\r?\n`, `data:` payload extraction and joining, comment/field skipping, the `[DONE]` sentinel, a `STREAM_CLOSED` error on EOF without it, and a flush of a final unterminated event block. The file is ~67 lines with ~108 lines of dedicated tests (`tests/sse.spec.ts`) re-proving SSE spec behavior — UTF-8 split across chunks, CRLF handling, multi-`data:` joining, no-space-after-colon — that a maintained parser already guarantees. Its only consumer is `adapter.ts` (`yield* translate(parseSse(response.body))`).
|
||||
|
||||
This is exactly the surface `eventsource-parser` owns: the de-facto standard SSE parser (it underlies the Vercel AI SDK and the MCP SDK), zero-dependency, actively maintained, and already present in this repo's lockfile transitively via `@modelcontextprotocol/sdk` — so adopting it directly adds no new supply-chain surface in practice.
|
||||
|
||||
## Proposal
|
||||
|
||||
Replace `sse.ts` with `EventSourceParserStream` from `eventsource-parser/stream`: `response.body.pipeThrough(new TextDecoderStream()).pipeThrough(new EventSourceParserStream())`, keeping only the DeepSeek protocol shim (~10–25 lines): yield each event's `data`, terminate on `[DONE]`, and throw `LlmError('STREAM_CLOSED')` when the stream ends without the sentinel. All required builtins (`TextDecoderStream`, `pipeThrough`, async-iterable `ReadableStream`) exist at the Node ^22.19 engine floor. Delete the spec-conformance tests; keep the `[DONE]`/`STREAM_CLOSED`/EOF contract tests. Add `eventsource-parser` to `llm-deepseek`'s dependencies (its second runtime dep after schemastery). Update the [twin-adapters note](../../implemented/architecture/2026-06-13-twin-llm-adapters.md) and the `dsh-llm` JSDoc that brand this adapter "hand-rolled fetch + SSE parsing" in the same PR.
|
||||
|
||||
The library also strips a leading BOM (the hand-rolled parser would fail to match `data:` after one) and offers `maxBufferSize` hardening the current parser lacks.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Keep the hand-rolled parser.** Defensible under the [twin-adapters decision](../../implemented/architecture/2026-06-13-twin-llm-adapters.md): the adapter is deliberately the hand-rolled design-verification twin of the pi-ai adapter. But the note's load-bearing distinction is owning the fetch/translate internals versus delegating to a full provider SDK; a ~700-byte SSE micro-parser is transport plumbing, not the design under verification. Whether that reading stands is the twin-note owner's call — this proposal explicitly needs their sign-off.
|
||||
- **`createParser({onEvent})` callback API instead of the stream.** Works fed by a manual `TextDecoder` loop, but the `pipeThrough` composition deletes more of the hand-rolled code.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `sse.ts`'s parsing internals are gone; the remaining shim only encodes the DeepSeek `[DONE]`/`STREAM_CLOSED` protocol.
|
||||
- `llm-deepseek` unit tests and the real-API e2e suite pass; keyless snapshots are unchanged (parsing is transport-internal and payload extraction is equivalent).
|
||||
- The twin-adapters note and `dsh-llm` JSDoc no longer claim hand-rolled SSE parsing.
|
||||
|
||||
## Risks
|
||||
|
||||
- One deliberate robustness deviation is lost: the hand-rolled parser flushes a final event block that lacks its terminating blank line, and `tests/sse.spec.ts` pins that a trailing `data: [DONE]` without `\n\n` still yields DONE. eventsource-parser is spec-strict and only dispatches on the blank line, so that shape becomes `STREAM_CLOSED`. Real providers and `dsh-llm-mock-server` always terminate events properly, so the pinned behavior is a robustness nicety, not an observed provider shape — drop the test, or keep a tiny buffer-tail check if the deviation is judged load-bearing.
|
||||
- Dilutes the documented "hand-rolled" identity of the twin adapter; mitigated by updating the note in the same change rather than leaving the claim stale.
|
||||
@@ -0,0 +1,33 @@
|
||||
# Agent Note: 用 eventsource-parser 替换 llm-deepseek 中手写的 SSE 解析器
|
||||
|
||||
Status: proposed
|
||||
|
||||
[English](2026-07-26-eventsource-parser-for-deepseek-sse.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
`packages/llm/llm-deepseek/src/sse.ts` 手写实现了 SSE（Server-Sent Events）解析：一个流式 `TextDecoder`、按 `\r?\n\r?\n` 切分事件块、提取并拼接 `data:` 载荷、跳过注释与其他字段、`[DONE]` 哨兵、在未见哨兵即 EOF 时抛出 `STREAM_CLOSED` 错误，以及对最后一个未终结事件块的 flush。该文件约 67 行，另有约 108 行专属测试（`tests/sse.spec.ts`）重复验证 SSE 规范行为——UTF-8 字符被切分到多个分片、CRLF 处理、多条 `data:` 拼接、冒号后无空格——而这些行为，持续维护的解析器早已有保证。它唯一的消费方是 `adapter.ts`（`yield* translate(parseSse(response.body))`）。
|
||||
|
||||
这恰好是 `eventsource-parser` 负责的接口面：事实标准的 SSE 解析器（Vercel AI SDK 和 MCP SDK 都构建在它之上），零依赖，持续维护，并且已通过 `@modelcontextprotocol/sdk` 作为传递依赖出现在本仓库的 lockfile 中——因此直接采用它实际上不增加新的供应链接触面。
|
||||
|
||||
## 提案
|
||||
|
||||
用 `eventsource-parser/stream` 的 `EventSourceParserStream` 替换 `sse.ts`：`response.body.pipeThrough(new TextDecoderStream()).pipeThrough(new EventSourceParserStream())`，只保留 DeepSeek 协议垫层（约 10–25 行）：逐个产出事件的 `data`，遇到 `[DONE]` 终止，流在未见哨兵时结束则抛出 `LlmError('STREAM_CLOSED')`。所需的全部内置能力（`TextDecoderStream`、`pipeThrough`、可异步迭代的 `ReadableStream`）在 Node ^22.19 引擎下限即已存在。删除规范符合性测试；保留 `[DONE]`/`STREAM_CLOSED`/EOF 契约测试。将 `eventsource-parser` 加入 `llm-deepseek` 的依赖（这是它继 schemastery 之后的第二个运行时依赖）。在同一个 PR（Pull Request）中更新[孪生适配器 Agent Note（agent 决策记录）](../../implemented/architecture/2026-06-13-twin-llm-adapters.md)以及 `dsh-llm` 中把该适配器标为「手写 fetch + SSE 解析」的 JSDoc。
|
||||
|
||||
该库还会剥离开头的 BOM（手写解析器在 BOM 之后会无法匹配 `data:`），并提供当前解析器缺少的 `maxBufferSize` 加固能力。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
- **保留手写解析器。** 依据[孪生适配器决策](../../implemented/architecture/2026-06-13-twin-llm-adapters.md)，这一选择有辩护余地：该适配器有意作为 pi-ai 适配器的手写设计验证孪生体。但那份 Agent Note 起支撑作用的区分在于「自行持有 fetch/translate 内部实现」与「委托给完整的提供方 SDK」；一个约 700 字节的 SSE 微型解析器属于传输层管道，不是被验证的设计本身。这一解读是否成立由孪生 Agent Note 的所有者裁定——本提案明确需要其签署确认。
|
||||
- **改用 `createParser({onEvent})` 回调 API 而非流。** 配合手动的 `TextDecoder` 循环可以工作，但 `pipeThrough` 组合方式能删除更多手写代码。
|
||||
|
||||
## 验收标准
|
||||
|
||||
- `sse.ts` 的解析内部实现消失；剩下的垫层只编码 DeepSeek 的 `[DONE]`/`STREAM_CLOSED` 协议。
|
||||
- `llm-deepseek` 单元测试与真实 API 的 e2e 套件通过；无密钥快照不变（解析属于传输层内部，载荷提取等价）。
|
||||
- 孪生适配器 Agent Note 与 `dsh-llm` 的 JSDoc 不再声称手写 SSE 解析。
|
||||
|
||||
## 风险
|
||||
|
||||
- 会失去一处有意为之的健壮性偏离：手写解析器会 flush 缺少终结空行的最后一个事件块，`tests/sse.spec.ts` 固定了「末尾的 `data: [DONE]` 即使没有 `\n\n` 也仍产出 DONE」这一行为。eventsource-parser 严格遵循规范，只在空行处分发事件，因此这种形态会变成 `STREAM_CLOSED`。真实提供方和 `dsh-llm-mock-server` 总是正确终结事件，所以被固定的行为只是健壮性上的锦上添花，并非实际观测到的提供方形态：可以删除该测试；若判定该偏离确有支撑作用，也可以保留一个小型的缓冲区尾部检查。
|
||||
- 稀释了孪生适配器有文档记录的「手写」身份；缓解方式是在同一次变更中更新那份 Agent Note，而不是让声明陈旧下去。
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-26-turndown-for-tool-web-html-markdown.md: 7f25e51bf6e6fc9313a880abee737bca80a472af
|
||||
2026-07-26-turndown-for-tool-web-html-markdown.zh.md: 3a59b08e13fd392e4f34ac543f32f5b4648f3c1c
|
||||
@@ -0,0 +1,32 @@
|
||||
# Agent Note: Replace tool-web's regex HTML-to-markdown converter with turndown
|
||||
|
||||
Status: proposed
|
||||
|
||||
English | [中文](2026-07-26-turndown-for-tool-web-html-markdown.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
`packages/web/tool-web/src/html.ts` (~86 lines, ~40 lines of dedicated tests) converts fetched HTML to markdown with regexes: strip script/style/noscript/comments, convert `<a>`/`<h1-6>`/`<li>`, decode numeric entities plus a 12-entry named-entity table, collapse whitespace. The module's own JSDoc says "A richer converter can replace it without changing the seam or tool schema", and the README's Known Limitations documents it as "a minimal regex converter, not an HTML parser — tables, images, and nested formatting are lost." The [web capability seam note](../../implemented/architecture/2026-06-24-web-capability-seam.md) assigns HTML→markdown to this package as presentation, so the swap point is exactly here. The converter's output is model-visible on every fetched HTML page; no keyless snapshot currently exercises `web_fetch`, so no expected outputs pin it.
|
||||
|
||||
## Proposal
|
||||
|
||||
Replace `htmlToMarkdown` with `turndown` (`new TurndownService().turndown(html)`), optionally with `turndown-plugin-gfm` for tables. The consumer switch in `fetch.ts` and the status-header/truncation-footer formatting stay. Wrap the call in try/catch falling back to the raw text path: the regex version could never throw; turndown on pathological HTML could. Delete `html.ts` and its conversion tests; keep tests for the fallback and the surrounding formatting. Update the README's Known Limitations to drop the regex-converter caveat.
|
||||
|
||||
If the "deliberately minimal fallback" stance is preferred instead, a minimal variant still deletes the worst part: replace the entity-decoding third of the file (~30 lines: `decodeEntities`, `NAMED_ENTITIES`, `safeFromCodePoint`) with the zero-dependency `entities` package (already in the lockfile transitively), erasing the documented "about a dozen entities" limitation at near-zero risk.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **`@mozilla/readability` + a DOM.** Solves a different problem (content extraction, not conversion) and drags a heavier DOM dependency; the seam only asks for markdown rendering of whatever the fetch returned.
|
||||
- **Keep the regex converter.** It was an explicit v1 placeholder per its own JSDoc; keeping it means model-visible quality (tables, images, nested formatting) stays lost for the cost of maintaining bespoke entity tables.
|
||||
- **The minimal `entities`-only variant.** Kept in the proposal as the fallback position; it deletes less but avoids the dependency-weight question entirely.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `web_fetch` renders tables/nested formatting via turndown (or, minimal variant: decodes all named entities), with the README limitation updated.
|
||||
- Unit tests cover the fallback path; `pnpm run test` passes for the package.
|
||||
- A keyless snapshot exercising `web_fetch` markdown rendering is added per testing policy (the missing snapshot coverage is part of the change, and it pins the new output).
|
||||
|
||||
## Risks
|
||||
|
||||
- Model-visible output changes on every fetched HTML page — transcript drift is acceptable pre-release, and nothing currently pins the old output.
|
||||
- Dependency weight: turndown's one dependency (`@mixmark-io/domino`) is a ~200 KB DOM that would enter the single-file-executable closure if tool-web ships in it ([single-exe note](../../implemented/architecture/2026-07-10-single-file-executable-sdk-runtime-distribution.md)); the minimal `entities` variant avoids this if closure size is the deciding factor.
|
||||
@@ -0,0 +1,32 @@
|
||||
# Agent Note: 用 turndown 替换 tool-web 的正则 HTML 转 markdown 转换器
|
||||
|
||||
Status: proposed
|
||||
|
||||
[English](2026-07-26-turndown-for-tool-web-html-markdown.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
`packages/web/tool-web/src/html.ts`（约 86 行，另有约 40 行专属测试）用正则表达式把抓取到的 HTML 转成 markdown：剥离 script、style、noscript 标签与注释，转换 `<a>`/`<h1-6>`/`<li>`，解码数字实体外加一张 12 项的命名实体表，并折叠空白。该模块自身的 JSDoc 写明「A richer converter can replace it without changing the seam or tool schema」，README 的 Known Limitations 章节也把它记载为「a minimal regex converter, not an HTML parser — tables, images, and nested formatting are lost」。[web 能力 seam 决策记录](../../implemented/architecture/2026-06-24-web-capability-seam.md)把 HTML 转 markdown 作为呈现职责划归本包（package），因此替换点恰好就在这里。每个抓取到的 HTML 页面上，该转换器的输出都对模型可见；当前没有任何无密钥快照执行到 `web_fetch`，因此没有预期输出固定它的行为。
|
||||
|
||||
## 提案
|
||||
|
||||
用 `turndown` 替换 `htmlToMarkdown`（`new TurndownService().turndown(html)`），可选择配合 `turndown-plugin-gfm` 支持表格。`fetch.ts` 中的消费方分支与状态头、截断页脚的格式化保持不变。把调用包在 try/catch 中，失败时回退到原始文本路径：正则版本从不可能抛异常，而 turndown 处理病态 HTML 时可能抛出。删除 `html.ts` 及其转换测试；保留回退路径与外围格式化的测试。更新 README 的 Known Limitations 章节，移除正则转换器的警示说明。
|
||||
|
||||
如果更倾向于「刻意保持最小回退实现」的立场，最小变体仍能删掉最糟的部分：用零依赖的 `entities` 包（已通过传递依赖存在于 lockfile 中）替换文件中占三分之一的实体解码部分（约 30 行：`decodeEntities`、`NAMED_ENTITIES`、`safeFromCodePoint`），以近乎为零的风险抹掉文档记载的「about a dozen entities」限制。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
- **`@mozilla/readability` 加一个 DOM。** 它解决的是另一个问题（内容提取，而非格式转换），还会拖入更重的 DOM 依赖；这个 seam 只要求把抓取返回的内容渲染成 markdown。
|
||||
- **保留正则转换器。** 按其自身 JSDoc 的说法，它本来就是明确的 v1 占位实现；保留它意味着模型可见的质量（表格、图片、嵌套格式）继续缺失，代价还是维护一套自制实体表。
|
||||
- **仅引入 `entities` 的最小变体。** 已作为退守方案保留在提案中；它删得更少，但完全避开了依赖体积问题。
|
||||
|
||||
## 验收标准
|
||||
|
||||
- `web_fetch` 经由 turndown 渲染表格与嵌套格式（或在最小变体下：解码全部命名实体），README 中的限制说明同步更新。
|
||||
- 单元测试覆盖回退路径；该包的 `pnpm run test` 通过。
|
||||
- 按测试政策补充一个执行 `web_fetch` markdown 渲染的无密钥快照（缺失的快照覆盖是本变更的一部分，它同时固定新输出）。
|
||||
|
||||
## 风险
|
||||
|
||||
- 模型可见的输出在每个抓取到的 HTML 页面上都会变化：预发布阶段的 transcript（文本记录）漂移可以接受，且当前没有任何东西固定旧输出。
|
||||
- 依赖体积：turndown 的唯一依赖（`@mixmark-io/domino`）是一个约 200 KB 的 DOM 实现，若 tool-web 进入单文件可执行文件，它会一并进入闭包（[single-exe 决策记录](../../implemented/architecture/2026-07-10-single-file-executable-sdk-runtime-distribution.md)）；若闭包体积是决定因素，最小的 `entities` 变体可以避开这一点。
|
||||
Reference in New Issue
Block a user