Merge remote-tracking branch 'origin/master' into feature/cordis-temporary-tools
This commit is contained in:
NI0317
@@ -1,6 +0,0 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-26-consolidate-gate-scripts-on-existing-deps.md: 0bf32e01ea407e2718f8ec39ca962587a37df9cc
|
||||
2026-07-26-consolidate-gate-scripts-on-existing-deps.zh.md: ba9f157a61ffdc415acf9b2a61857026fc2c8bf1
|
||||
@@ -1,38 +0,0 @@
|
||||
# Agent Note: Consolidate gate scripts on already-present deps and builtins
|
||||
|
||||
Status: proposed
|
||||
|
||||
English | [中文](2026-07-26-consolidate-gate-scripts-on-existing-deps.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The `scripts/` gates mostly use the right tools (`node:fs` `globSync` in 15+ gates, mdast/micromark in the markdown gates), but a handful of stragglers hand-roll what a sibling gate already does with an existing dependency or builtin:
|
||||
|
||||
- **Duplicated fence scanners.** `scripts/md-fences.ts` (~55 lines, consumed by `doc-typecheck.ts`) and `extractEquivBlocks` in `scripts/verify-type-equiv.ts` (~39 lines) are two copies of the same regex line-scanner for fenced code blocks, while `scripts/verify-mermaid.ts` already extracts fences by visiting mdast `code` nodes via the shared `scripts/markdown.ts` helpers — and `markdownProseLines` in `markdown.ts` itself parses to mdast but then hand-tracks fence state with a second regex. The regex scanners only recognize backtick fences at column 0, so they silently disagree with the mdast-based gates on tilde and indented fences.
|
||||
- **Hand-rolled argv parsing.** `parseOptions` in `scripts/publint-all.ts` and its near-identical copy in `scripts/verify-built-package-invariants.mjs` (~26 lines) step argv indexes manually, while sibling scripts (`verify-runtime-closure.ts`, `build-exe-for-python-sdk.ts`, `packages/sdk/scripts/src/args.ts`) already use the `node:util` `parseArgs` builtin.
|
||||
- **Hand-rolled directory walks.** Five sites re-derive nested `readdirSync` walks that `globSync` covers: `verify-runtime-closure.ts` (packages + vendor manifests), `dev-web.ts` `discoverPluginDirs`, `verify-package-paths.ts` `realPackageNames`, `verify-client-domain-graph.ts` `listSources`, and `publint-all.ts` `addPath` (~55–65 lines total). `scripts/package-invariants.ts` shows the one-line `globSync` template.
|
||||
|
||||
No new dependency is needed anywhere; every replacement is an existing devDep or a Node builtin.
|
||||
|
||||
## Proposal
|
||||
|
||||
- Extract a shared ~10–15-line mdast fence helper (visiting `code` nodes for `lang`, `meta`, `value`, `position.start.line`) into `scripts/markdown.ts`; rewrite `doc-typecheck.ts` and `verify-type-equiv.ts` onto it; delete `md-fences.ts` and the duplicated scanner; drop the redundant fence regex in `markdownProseLines`.
|
||||
- Replace both `parseOptions` copies with `parseArgs`.
|
||||
- Replace the five straggler walks with `globSync`. Keep the walks in `check-workspace-constraints.ts` and `clean.ts`: they need dirent-level detail to diagnose malformed trees, which glob-by-pattern cannot report.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **A new glob/walking dependency (`tinyglobby`, `fdir`).** Rejected: the builtin already won repo-wide; these are stragglers, not a gap.
|
||||
- **`p-map` for `publint-all.ts`'s ~19-line ordered worker pool.** Deliberately left out: one new devDep for one small deletion is at the edge of the [dependency policy](../../implemented/process/2026-07-26-dependencies-over-hand-rolling.md) bar, and the pool's requirements (bounded workers, deterministic order, env override) are documented in the [parallel-gates note](../../implemented/process/2026-07-06-parallel-pre-push-gates.md). Fold it in only if `p-map` earns a second consumer.
|
||||
- **Leaving the fence scanners.** Rejected: two drifting copies of a parser beside a third correct implementation is exactly the duplication the shared `markdown.ts` helper exists to prevent, and the column-0-backtick-only limitation is a latent inconsistency between sibling gates.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `md-fences.ts` is gone; `doc-typecheck` and `verify-type-equiv` extract fences through `scripts/markdown.ts`; `pnpm run doc-sync` passes with unchanged results on the current tree (any delta traces to a fence shape the regex scanners mishandled).
|
||||
- Both CLIs parse via `parseArgs`; unknown options still fail loud.
|
||||
- The five walk sites use `globSync`; the gates they feed pass unchanged.
|
||||
|
||||
## Risks
|
||||
|
||||
- Behavioral deltas on pathological markdown: mdast honors tilde/indented fences the regex scanners ignored, so `doc-typecheck`'s opt-out ratio could shift if any stray fence shape exists in the docs tree; verify by running `doc-sync` before/after.
|
||||
- `parseArgs` keeps the last value of a duplicated option instead of erroring — a dev-tool edge case the tests don't pin. (Strict mode still rejects a `--`-prefixed token where a value is expected, matching the current parsers.)
|
||||
@@ -1,38 +0,0 @@
|
||||
# Agent Note: 把门禁脚本统一到已有依赖与内置模块上
|
||||
|
||||
Status: proposed
|
||||
|
||||
[English](2026-07-26-consolidate-gate-scripts-on-existing-deps.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
`scripts/` 下的门禁大多已经在用正确的工具（15 个以上的门禁使用 `node:fs` 的 `globSync`，markdown 门禁使用 mdast/micromark），但少数几个掉队的脚本仍在手写同类门禁早已用既有依赖或内置模块完成的事情：
|
||||
|
||||
- **重复的围栏扫描器。**`scripts/md-fences.ts`（约 55 行，由 `doc-typecheck.ts` 消费）和 `scripts/verify-type-equiv.ts` 中的 `extractEquivBlocks`（约 39 行）是同一个围栏代码块正则行扫描器的两份拷贝，而 `scripts/verify-mermaid.ts` 已经通过共享的 `scripts/markdown.ts` 辅助函数访问 mdast `code` 节点来提取代码围栏；`markdown.ts` 自己的 `markdownProseLines` 也是先解析成 mdast，再用第二个正则手工跟踪围栏状态。这两个正则扫描器只识别第 0 列的反引号围栏，因此在波浪线围栏和缩进围栏上与基于 mdast 的门禁悄悄不一致。
|
||||
- **手写的 argv 解析。**`scripts/publint-all.ts` 中的 `parseOptions` 和 `scripts/verify-built-package-invariants.mjs` 中与之几乎相同的拷贝（约 26 行）手工推进 argv 下标，而同类脚本（`verify-runtime-closure.ts`、`build-exe-for-python-sdk.ts`、`packages/sdk/scripts/src/args.ts`）已经在使用 `node:util` 的内置 `parseArgs`。
|
||||
- **手写的目录遍历。**五处代码各自重写了 `globSync` 已覆盖的嵌套 `readdirSync` 遍历：`verify-runtime-closure.ts` 对 packages 与 vendor manifest（元数据清单）的扫描、`dev-web.ts` 的 `discoverPluginDirs`、`verify-package-paths.ts` 的 `realPackageNames`、`verify-client-domain-graph.ts` 的 `listSources`，以及 `publint-all.ts` 的 `addPath`（合计约 55–65 行）。`scripts/package-invariants.ts` 展示了一行式的 `globSync` 模板。
|
||||
|
||||
所有替换都不需要引入新依赖；每一处替换用的都是既有的 devDependency 或 Node 内置模块。
|
||||
|
||||
## 提案
|
||||
|
||||
- 在 `scripts/markdown.ts` 中提取一个约 10–15 行的共享 mdast 围栏辅助函数（访问 `code` 节点，读取 `lang`、`meta`、`value`、`position.start.line`）；把 `doc-typecheck.ts` 和 `verify-type-equiv.ts` 改写到它上面；删除 `md-fences.ts` 和重复的扫描器；去掉 `markdownProseLines` 中冗余的围栏正则。
|
||||
- 用 `parseArgs` 替换两份 `parseOptions` 拷贝。
|
||||
- 用 `globSync` 替换那五处掉队的目录遍历。保留 `check-workspace-constraints.ts` 和 `clean.ts` 中的遍历：它们需要 dirent 级别的细节来诊断结构异常的目录树，按模式匹配的 glob 报告不了这些信息。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
- **新的 glob/目录遍历依赖（`tinyglobby`、`fdir`）。**不予采纳：内置模块已在全仓库范围内胜出；这几处只是掉队者，不是能力缺口。
|
||||
- **用 `p-map` 替换 `publint-all.ts` 中约 19 行的有序 worker 池。**刻意未纳入：为一次小删除引入一个新 devDependency，正处在[依赖策略](../../implemented/process/2026-07-26-dependencies-over-hand-rolling.md)门槛的边缘，而且该池的需求（worker 数量有界、确定性顺序、环境变量覆盖）已记录在[并行 pre-push 门禁决策记录](../../implemented/process/2026-07-06-parallel-pre-push-gates.md)中。仅当 `p-map` 赢得第二个消费方时再顺带纳入。
|
||||
- **保留这两个围栏扫描器。**不予采纳：在第三个正确实现旁边放着两份逐渐漂移的解析器拷贝，正是共享的 `markdown.ts` 辅助函数要防止的那种重复；「只认第 0 列反引号」的限制也是同类门禁之间的潜在不一致。
|
||||
|
||||
## 验收标准
|
||||
|
||||
- `md-fences.ts` 已删除；`doc-typecheck` 与 `verify-type-equiv` 通过 `scripts/markdown.ts` 提取代码围栏；`pnpm run doc-sync` 在当前代码树上通过且结果不变（如有差异，必须能追溯到正则扫描器处理有误的某种围栏形态）。
|
||||
- 两个 CLI 都改用 `parseArgs` 解析；未知选项仍然大声失败。
|
||||
- 五处遍历代码改用 `globSync`；它们供给的门禁保持原样通过。
|
||||
|
||||
## 风险
|
||||
|
||||
- 病态 markdown 上的行为差异：mdast 会承认正则扫描器忽略的波浪线围栏和缩进围栏，因此如果文档树中存在任何零散的此类围栏形态，`doc-typecheck` 的 opt-out 比例可能变化；应在改动前后分别运行 `doc-sync` 加以验证。
|
||||
- `parseArgs` 对重复出现的选项保留最后一个值而不报错——一个测试未固定的开发工具边缘用例。（严格模式下，需要取值处遇到以 `--` 开头的 token 仍会拒绝，与现有解析器行为一致。）
|
||||
@@ -1,6 +0,0 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-26-turndown-for-tool-web-html-markdown.md: 7f25e51bf6e6fc9313a880abee737bca80a472af
|
||||
2026-07-26-turndown-for-tool-web-html-markdown.zh.md: 3a59b08e13fd392e4f34ac543f32f5b4648f3c1c
|
||||
@@ -1,32 +0,0 @@
|
||||
# Agent Note: Replace tool-web's regex HTML-to-markdown converter with turndown
|
||||
|
||||
Status: proposed
|
||||
|
||||
English | [中文](2026-07-26-turndown-for-tool-web-html-markdown.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
`packages/web/tool-web/src/html.ts` (~86 lines, ~40 lines of dedicated tests) converts fetched HTML to markdown with regexes: strip script/style/noscript/comments, convert `<a>`/`<h1-6>`/`<li>`, decode numeric entities plus a 12-entry named-entity table, collapse whitespace. The module's own JSDoc says "A richer converter can replace it without changing the seam or tool schema", and the README's Known Limitations documents it as "a minimal regex converter, not an HTML parser — tables, images, and nested formatting are lost." The [web capability seam note](../../implemented/architecture/2026-06-24-web-capability-seam.md) assigns HTML→markdown to this package as presentation, so the swap point is exactly here. The converter's output is model-visible on every fetched HTML page; no keyless snapshot currently exercises `web_fetch`, so no expected outputs pin it.
|
||||
|
||||
## Proposal
|
||||
|
||||
Replace `htmlToMarkdown` with `turndown` (`new TurndownService().turndown(html)`), optionally with `turndown-plugin-gfm` for tables. The consumer switch in `fetch.ts` and the status-header/truncation-footer formatting stay. Wrap the call in try/catch falling back to the raw text path: the regex version could never throw; turndown on pathological HTML could. Delete `html.ts` and its conversion tests; keep tests for the fallback and the surrounding formatting. Update the README's Known Limitations to drop the regex-converter caveat.
|
||||
|
||||
If the "deliberately minimal fallback" stance is preferred instead, a minimal variant still deletes the worst part: replace the entity-decoding third of the file (~30 lines: `decodeEntities`, `NAMED_ENTITIES`, `safeFromCodePoint`) with the zero-dependency `entities` package (already in the lockfile transitively), erasing the documented "about a dozen entities" limitation at near-zero risk.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **`@mozilla/readability` + a DOM.** Solves a different problem (content extraction, not conversion) and drags a heavier DOM dependency; the seam only asks for markdown rendering of whatever the fetch returned.
|
||||
- **Keep the regex converter.** It was an explicit v1 placeholder per its own JSDoc; keeping it means model-visible quality (tables, images, nested formatting) stays lost for the cost of maintaining bespoke entity tables.
|
||||
- **The minimal `entities`-only variant.** Kept in the proposal as the fallback position; it deletes less but avoids the dependency-weight question entirely.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `web_fetch` renders tables/nested formatting via turndown (or, minimal variant: decodes all named entities), with the README limitation updated.
|
||||
- Unit tests cover the fallback path; `pnpm run test` passes for the package.
|
||||
- A keyless snapshot exercising `web_fetch` markdown rendering is added per testing policy (the missing snapshot coverage is part of the change, and it pins the new output).
|
||||
|
||||
## Risks
|
||||
|
||||
- Model-visible output changes on every fetched HTML page — transcript drift is acceptable pre-release, and nothing currently pins the old output.
|
||||
- Dependency weight: turndown's one dependency (`@mixmark-io/domino`) is a ~200 KB DOM that would enter the single-file-executable closure if tool-web ships in it ([single-exe note](../../implemented/architecture/2026-07-10-single-file-executable-sdk-runtime-distribution.md)); the minimal `entities` variant avoids this if closure size is the deciding factor.
|
||||
@@ -1,32 +0,0 @@
|
||||
# Agent Note: 用 turndown 替换 tool-web 的正则 HTML 转 markdown 转换器
|
||||
|
||||
Status: proposed
|
||||
|
||||
[English](2026-07-26-turndown-for-tool-web-html-markdown.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
`packages/web/tool-web/src/html.ts`（约 86 行，另有约 40 行专属测试）用正则表达式把抓取到的 HTML 转成 markdown：剥离 script、style、noscript 标签与注释，转换 `<a>`/`<h1-6>`/`<li>`，解码数字实体外加一张 12 项的命名实体表，并折叠空白。该模块自身的 JSDoc 写明「A richer converter can replace it without changing the seam or tool schema」，README 的 Known Limitations 章节也把它记载为「a minimal regex converter, not an HTML parser — tables, images, and nested formatting are lost」。[web 能力 seam 决策记录](../../implemented/architecture/2026-06-24-web-capability-seam.md)把 HTML 转 markdown 作为呈现职责划归本包（package），因此替换点恰好就在这里。每个抓取到的 HTML 页面上，该转换器的输出都对模型可见；当前没有任何无密钥快照执行到 `web_fetch`，因此没有预期输出固定它的行为。
|
||||
|
||||
## 提案
|
||||
|
||||
用 `turndown` 替换 `htmlToMarkdown`（`new TurndownService().turndown(html)`），可选择配合 `turndown-plugin-gfm` 支持表格。`fetch.ts` 中的消费方分支与状态头、截断页脚的格式化保持不变。把调用包在 try/catch 中，失败时回退到原始文本路径：正则版本从不可能抛异常，而 turndown 处理病态 HTML 时可能抛出。删除 `html.ts` 及其转换测试；保留回退路径与外围格式化的测试。更新 README 的 Known Limitations 章节，移除正则转换器的警示说明。
|
||||
|
||||
如果更倾向于「刻意保持最小回退实现」的立场，最小变体仍能删掉最糟的部分：用零依赖的 `entities` 包（已通过传递依赖存在于 lockfile 中）替换文件中占三分之一的实体解码部分（约 30 行：`decodeEntities`、`NAMED_ENTITIES`、`safeFromCodePoint`），以近乎为零的风险抹掉文档记载的「about a dozen entities」限制。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
- **`@mozilla/readability` 加一个 DOM。** 它解决的是另一个问题（内容提取，而非格式转换），还会拖入更重的 DOM 依赖；这个 seam 只要求把抓取返回的内容渲染成 markdown。
|
||||
- **保留正则转换器。** 按其自身 JSDoc 的说法，它本来就是明确的 v1 占位实现；保留它意味着模型可见的质量（表格、图片、嵌套格式）继续缺失，代价还是维护一套自制实体表。
|
||||
- **仅引入 `entities` 的最小变体。** 已作为退守方案保留在提案中；它删得更少，但完全避开了依赖体积问题。
|
||||
|
||||
## 验收标准
|
||||
|
||||
- `web_fetch` 经由 turndown 渲染表格与嵌套格式（或在最小变体下：解码全部命名实体），README 中的限制说明同步更新。
|
||||
- 单元测试覆盖回退路径；该包的 `pnpm run test` 通过。
|
||||
- 按测试政策补充一个执行 `web_fetch` markdown 渲染的无密钥快照（缺失的快照覆盖是本变更的一部分，它同时固定新输出）。
|
||||
|
||||
## 风险
|
||||
|
||||
- 模型可见的输出在每个抓取到的 HTML 页面上都会变化：预发布阶段的 transcript（文本记录）漂移可以接受，且当前没有任何东西固定旧输出。
|
||||
- 依赖体积：turndown 的唯一依赖（`@mixmark-io/domino`）是一个约 200 KB 的 DOM 实现，若 tool-web 进入单文件可执行文件，它会一并进入闭包（[single-exe 决策记录](../../implemented/architecture/2026-07-10-single-file-executable-sdk-runtime-distribution.md)）；若闭包体积是决定因素，最小的 `entities` 变体可以避开这一点。
|
||||
Reference in New Issue
Block a user