Merge branch 'master' into nih-imp-gates
This commit is contained in:
Tianyi Cui
@@ -1,6 +0,0 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-26-eventsource-parser-for-deepseek-sse.md: 8a93b7f6c7aa0d428f25e87c44e1d29e884ecc81
|
||||
2026-07-26-eventsource-parser-for-deepseek-sse.zh.md: b16109d9458f487c7e463cf02e6b2d22fbbde015
|
||||
@@ -1,33 +0,0 @@
|
||||
# Agent Note: Replace the hand-rolled SSE parser in llm-deepseek with eventsource-parser
|
||||
|
||||
Status: proposed
|
||||
|
||||
English | [中文](2026-07-26-eventsource-parser-for-deepseek-sse.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
`packages/llm/llm-deepseek/src/sse.ts` hand-implements Server-Sent Events parsing: a streaming `TextDecoder`, event-block splitting on `\r?\n\r?\n`, `data:` payload extraction and joining, comment/field skipping, the `[DONE]` sentinel, a `STREAM_CLOSED` error on EOF without it, and a flush of a final unterminated event block. The file is ~67 lines with ~108 lines of dedicated tests (`tests/sse.spec.ts`) re-proving SSE spec behavior — UTF-8 split across chunks, CRLF handling, multi-`data:` joining, no-space-after-colon — that a maintained parser already guarantees. Its only consumer is `adapter.ts` (`yield* translate(parseSse(response.body))`).
|
||||
|
||||
This is exactly the surface `eventsource-parser` owns: the de-facto standard SSE parser (it underlies the Vercel AI SDK and the MCP SDK), zero-dependency, actively maintained, and already present in this repo's lockfile transitively via `@modelcontextprotocol/sdk` — so adopting it directly adds no new supply-chain surface in practice.
|
||||
|
||||
## Proposal
|
||||
|
||||
Replace `sse.ts` with `EventSourceParserStream` from `eventsource-parser/stream`: `response.body.pipeThrough(new TextDecoderStream()).pipeThrough(new EventSourceParserStream())`, keeping only the DeepSeek protocol shim (~10–25 lines): yield each event's `data`, terminate on `[DONE]`, and throw `LlmError('STREAM_CLOSED')` when the stream ends without the sentinel. All required builtins (`TextDecoderStream`, `pipeThrough`, async-iterable `ReadableStream`) exist at the Node ^22.19 engine floor. Delete the spec-conformance tests; keep the `[DONE]`/`STREAM_CLOSED`/EOF contract tests. Add `eventsource-parser` to `llm-deepseek`'s dependencies (its second runtime dep after schemastery). Update the [twin-adapters note](../../implemented/architecture/2026-06-13-twin-llm-adapters.md) and the `dsh-llm` JSDoc that brand this adapter "hand-rolled fetch + SSE parsing" in the same PR.
|
||||
|
||||
The library also strips a leading BOM (the hand-rolled parser would fail to match `data:` after one) and offers `maxBufferSize` hardening the current parser lacks.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Keep the hand-rolled parser.** Defensible under the [twin-adapters decision](../../implemented/architecture/2026-06-13-twin-llm-adapters.md): the adapter is deliberately the hand-rolled design-verification twin of the pi-ai adapter. But the note's load-bearing distinction is owning the fetch/translate internals versus delegating to a full provider SDK; a ~700-byte SSE micro-parser is transport plumbing, not the design under verification. Whether that reading stands is the twin-note owner's call — this proposal explicitly needs their sign-off.
|
||||
- **`createParser({onEvent})` callback API instead of the stream.** Works fed by a manual `TextDecoder` loop, but the `pipeThrough` composition deletes more of the hand-rolled code.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `sse.ts`'s parsing internals are gone; the remaining shim only encodes the DeepSeek `[DONE]`/`STREAM_CLOSED` protocol.
|
||||
- `llm-deepseek` unit tests and the real-API e2e suite pass; keyless snapshots are unchanged (parsing is transport-internal and payload extraction is equivalent).
|
||||
- The twin-adapters note and `dsh-llm` JSDoc no longer claim hand-rolled SSE parsing.
|
||||
|
||||
## Risks
|
||||
|
||||
- One deliberate robustness deviation is lost: the hand-rolled parser flushes a final event block that lacks its terminating blank line, and `tests/sse.spec.ts` pins that a trailing `data: [DONE]` without `\n\n` still yields DONE. eventsource-parser is spec-strict and only dispatches on the blank line, so that shape becomes `STREAM_CLOSED`. Real providers and `dsh-llm-mock-server` always terminate events properly, so the pinned behavior is a robustness nicety, not an observed provider shape — drop the test, or keep a tiny buffer-tail check if the deviation is judged load-bearing.
|
||||
- Dilutes the documented "hand-rolled" identity of the twin adapter; mitigated by updating the note in the same change rather than leaving the claim stale.
|
||||
@@ -1,33 +0,0 @@
|
||||
# Agent Note: 用 eventsource-parser 替换 llm-deepseek 中手写的 SSE 解析器
|
||||
|
||||
Status: proposed
|
||||
|
||||
[English](2026-07-26-eventsource-parser-for-deepseek-sse.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
`packages/llm/llm-deepseek/src/sse.ts` 手写实现了 SSE（Server-Sent Events）解析：一个流式 `TextDecoder`、按 `\r?\n\r?\n` 切分事件块、提取并拼接 `data:` 载荷、跳过注释与其他字段、`[DONE]` 哨兵、在未见哨兵即 EOF 时抛出 `STREAM_CLOSED` 错误，以及对最后一个未终结事件块的 flush。该文件约 67 行，另有约 108 行专属测试（`tests/sse.spec.ts`）重复验证 SSE 规范行为——UTF-8 字符被切分到多个分片、CRLF 处理、多条 `data:` 拼接、冒号后无空格——而这些行为，持续维护的解析器早已有保证。它唯一的消费方是 `adapter.ts`（`yield* translate(parseSse(response.body))`）。
|
||||
|
||||
这恰好是 `eventsource-parser` 负责的接口面：事实标准的 SSE 解析器（Vercel AI SDK 和 MCP SDK 都构建在它之上），零依赖，持续维护，并且已通过 `@modelcontextprotocol/sdk` 作为传递依赖出现在本仓库的 lockfile 中——因此直接采用它实际上不增加新的供应链接触面。
|
||||
|
||||
## 提案
|
||||
|
||||
用 `eventsource-parser/stream` 的 `EventSourceParserStream` 替换 `sse.ts`：`response.body.pipeThrough(new TextDecoderStream()).pipeThrough(new EventSourceParserStream())`，只保留 DeepSeek 协议垫层（约 10–25 行）：逐个产出事件的 `data`，遇到 `[DONE]` 终止，流在未见哨兵时结束则抛出 `LlmError('STREAM_CLOSED')`。所需的全部内置能力（`TextDecoderStream`、`pipeThrough`、可异步迭代的 `ReadableStream`）在 Node ^22.19 引擎下限即已存在。删除规范符合性测试；保留 `[DONE]`/`STREAM_CLOSED`/EOF 契约测试。将 `eventsource-parser` 加入 `llm-deepseek` 的依赖（这是它继 schemastery 之后的第二个运行时依赖）。在同一个 PR（Pull Request）中更新[孪生适配器 Agent Note（agent 决策记录）](../../implemented/architecture/2026-06-13-twin-llm-adapters.md)以及 `dsh-llm` 中把该适配器标为「手写 fetch + SSE 解析」的 JSDoc。
|
||||
|
||||
该库还会剥离开头的 BOM（手写解析器在 BOM 之后会无法匹配 `data:`），并提供当前解析器缺少的 `maxBufferSize` 加固能力。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
- **保留手写解析器。** 依据[孪生适配器决策](../../implemented/architecture/2026-06-13-twin-llm-adapters.md)，这一选择有辩护余地：该适配器有意作为 pi-ai 适配器的手写设计验证孪生体。但那份 Agent Note 起支撑作用的区分在于「自行持有 fetch/translate 内部实现」与「委托给完整的提供方 SDK」；一个约 700 字节的 SSE 微型解析器属于传输层管道，不是被验证的设计本身。这一解读是否成立由孪生 Agent Note 的所有者裁定——本提案明确需要其签署确认。
|
||||
- **改用 `createParser({onEvent})` 回调 API 而非流。** 配合手动的 `TextDecoder` 循环可以工作，但 `pipeThrough` 组合方式能删除更多手写代码。
|
||||
|
||||
## 验收标准
|
||||
|
||||
- `sse.ts` 的解析内部实现消失；剩下的垫层只编码 DeepSeek 的 `[DONE]`/`STREAM_CLOSED` 协议。
|
||||
- `llm-deepseek` 单元测试与真实 API 的 e2e 套件通过；无密钥快照不变（解析属于传输层内部，载荷提取等价）。
|
||||
- 孪生适配器 Agent Note 与 `dsh-llm` 的 JSDoc 不再声称手写 SSE 解析。
|
||||
|
||||
## 风险
|
||||
|
||||
- 会失去一处有意为之的健壮性偏离：手写解析器会 flush 缺少终结空行的最后一个事件块，`tests/sse.spec.ts` 固定了「末尾的 `data: [DONE]` 即使没有 `\n\n` 也仍产出 DONE」这一行为。eventsource-parser 严格遵循规范，只在空行处分发事件，因此这种形态会变成 `STREAM_CLOSED`。真实提供方和 `dsh-llm-mock-server` 总是正确终结事件，所以被固定的行为只是健壮性上的锦上添花，并非实际观测到的提供方形态：可以删除该测试；若判定该偏离确有支撑作用，也可以保留一个小型的缓冲区尾部检查。
|
||||
- 稀释了孪生适配器有文档记录的「手写」身份；缓解方式是在同一次变更中更新那份 Agent Note，而不是让声明陈旧下去。
|
||||
Reference in New Issue
Block a user