refactor: apply repository naming contract
Apply the accepted pre-release package, service, type, directory, and role renames as one repository-wide change.
This commit is contained in:
6
packages/web/web-fetch-http/README.i18n.yaml
Normal file
6
packages/web/web-fetch-http/README.i18n.yaml
Normal file
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/web/web-fetch-http/README.md
|
||||
README.md: 5589a8e8605a64ae9ef5f6d9978a9b63331d5b0d
|
||||
README.zh.md: e9cf98feb9065947af1321c87562a2ae9ac21315
|
||||
51
packages/web/web-fetch-http/README.md
Normal file
51
packages/web/web-fetch-http/README.md
Normal file
@@ -0,0 +1,51 @@
|
||||
# @deepseek-ai/dsh-web-fetch-http
|
||||
|
||||
English | [中文](README.zh.md)
|
||||
|
||||
An anonymous public HTTP(S) `WebFetchProvider` for the harness [web capability seam](../web/README.md) (`ctx.web`). It retrieves a concrete URL and returns a status code plus bounded decoded content.
|
||||
|
||||
This is an **implementation** package: it registers a provider into `ctx.web`, it does not own the key and it does not register a model-facing tool. It is a function/namespace plugin (`inject: ['web']`).
|
||||
|
||||
## Responsibility split
|
||||
|
||||
The provider owns **safe resource retrieval**: URL validation, HTTP transport, redirect policy, a resource-backstop timeout, abort propagation, byte caps, charset decoding, content-type classification, and binary rejection. `@deepseek-ai/dsh-tool-web` owns **presentation** (HTML→markdown, truncation formatting). A non-2xx HTTP response is a *result* (status code + decoded body), not an error; `WebError` is reserved for failures to safely retrieve or represent the resource.
|
||||
|
||||
The provider's `timeoutMs` is a resource backstop for direct `ctx.web.fetch()` callers and misconfigured deployments, not the model-facing tool-call budget. [`dsh-tool-call-timeout-policy`](../../guard/timeout-policy/README.md) owns the `web_fetch` tool-call budget by arming `exec.signal`.
|
||||
|
||||
A shipping web-tool deployment sets the provider backstop above the tool budget, so model calls normally return `TOOL_TIMEOUT`. If the outer deadline reaches the provider first, the provider reports `WEB_ABORTED` and the outer policy replaces it with `TOOL_TIMEOUT`. `WEB_FETCH_TIMEOUT` therefore identifies a direct service caller whose provider budget elapsed.
|
||||
|
||||
## Transport hygiene
|
||||
|
||||
- Accepts only `http:` and `https:` URLs; rejects credentials in URLs (`WEB_BLOCKED_URL`) and over-long/malformed URLs (`WEB_INVALID_URL`).
|
||||
- Enforces a max URL length, response byte cap (`WEB_FETCH_TOO_LARGE`), decoded body character cap, timeout (`WEB_FETCH_TIMEOUT`), and redirect hop cap.
|
||||
- Propagates the caller's abort signal (`WEB_ABORTED`) into the network request and the streaming read.
|
||||
- Follows only **same-origin** redirects; a cross-origin redirect fails with `WEB_REDIRECT_BLOCKED`, requiring a fresh tool call (the model of Claude Code's WebFetch).
|
||||
- Sends an explicit product `User-Agent`, never a browser disguise.
|
||||
- Rejects unsupported (e.g. binary) content types with `WEB_UNSUPPORTED_CONTENT_TYPE`.
|
||||
|
||||
## Config
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `maxUrlLength` | `2048` | Maximum accepted request URL length. |
|
||||
| `maxResponseBytes` | `5_000_000` | Maximum response body size in bytes. |
|
||||
| `maxBodyChars` | `100_000` | Maximum decoded body length in characters. |
|
||||
| `timeoutMs` | `30_000` | Fetch timeout within Node's timer range — a resource backstop for direct `ctx.web.fetch()` callers, not the model-facing tool-call budget (that is `dsh-tool-call-timeout-policy`). |
|
||||
| `maxRedirects` | `5` | Maximum same-origin redirect hops (`0` follows none). |
|
||||
| `userAgent` | `deepseek-harness/…` | `User-Agent` header. |
|
||||
|
||||
The numeric limits are validated at plugin construction: every cap except `maxRedirects` must be a positive finite number, and `maxRedirects` must be a non-negative integer. An invalid value throws rather than silently constructing a provider with nonsensical limits.
|
||||
|
||||
## Model Experience
|
||||
|
||||
Indirectly, through [`dsh-tool-web`](../tool-web/README.md), which places this provider's `maxBodyChars`-bounded decoded text or markdown-shaped HTML under its fetch-result wrapper and retains provider failures while redirects, headers, and transport mechanics remain hidden.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
No direct invalidation; the named consumer owns any request-prefix changes.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **SSRF / private-network protection is deferred** — no blocking of private, loopback, link-local, multicast, or otherwise non-public destinations, no DNS-resolve-then-validate, no per-hop re-validation (see [the web capability seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md)). Until it lands, this provider is an SSRF primitive and **must not be enabled** in a deployment that can reach sensitive internal network targets.
|
||||
- **Only textual content decodes** — html/xhtml and `text/*`-plus-JSON/XML families; a missing `Content-Type` or any binary type throws `WEB_UNSUPPORTED_CONTENT_TYPE`, and text-extractable PDF decoding is named deferred work.
|
||||
- **Charset comes only from the `Content-Type` header** (UTF-8 default) — an HTML `<meta charset>` declaration is ignored, and a declared-but-unrecognized charset label throws rather than falling back.
|
||||
51
packages/web/web-fetch-http/README.zh.md
Normal file
51
packages/web/web-fetch-http/README.zh.md
Normal file
@@ -0,0 +1,51 @@
|
||||
# @deepseek-ai/dsh-web-fetch-http
|
||||
|
||||
[English](README.md) | 中文
|
||||
|
||||
一个匿名公共 HTTP(S) `WebFetchProvider`,用于 harness [web 能力 seam](../web/README.md)(`ctx.web`)。它获取具体 URL,返回状态码和长度受限的解码内容。
|
||||
|
||||
这是一个**实现**包:它向 `ctx.web` 注册提供方,不拥有该键,也不注册面向模型的工具。它是函数/命名空间插件(`inject: ['web']`)。
|
||||
|
||||
## 职责拆分
|
||||
|
||||
提供方拥有**安全资源获取**:URL 验证、HTTP 传输、重定向策略、资源兜底超时、中止传播、字节上限、charset 解码、内容类型分类与二进制拒绝。`@deepseek-ai/dsh-tool-web` 拥有**呈现**(HTML→markdown、截断格式)。非 2xx HTTP 响应是*结果*(状态码 + 解码主体),不是错误;`WebError` 只用于无法安全获取或表示资源的失败。
|
||||
|
||||
提供方的 `timeoutMs` 是直接 `ctx.web.fetch()` 调用方和配置有误的部署所用的资源兜底,不是面向模型的工具调用预算。[`dsh-tool-call-timeout-policy`](../../guard/timeout-policy/README.md) 拥有 `web_fetch` 工具调用预算,并让 `exec.signal` 在超时时触发,以强制执行该预算。
|
||||
|
||||
已交付的 web 工具部署会把提供方兜底设为高于工具预算,因此模型调用通常返回 `TOOL_TIMEOUT`。如果外层截止期限先于提供方的兜底超时触发,提供方会报告 `WEB_ABORTED`,外层策略再将其替换为 `TOOL_TIMEOUT`。因此,`WEB_FETCH_TIMEOUT` 表明直接服务调用方的提供方预算已经耗尽。
|
||||
|
||||
## 传输卫生
|
||||
|
||||
- 只接受 `http:` 和 `https:` URL;拒绝 URL 中的凭据(`WEB_BLOCKED_URL`)以及过长/格式错误的 URL(`WEB_INVALID_URL`)。
|
||||
- 强制执行 URL 最大长度、响应字节上限(`WEB_FETCH_TOO_LARGE`)、解码主体字符上限、超时(`WEB_FETCH_TIMEOUT`)和重定向跳数上限。
|
||||
- 把调用方的中止信号(`WEB_ABORTED`)传播到网络请求与流式读取。
|
||||
- 只跟随**同源**重定向;跨源重定向以 `WEB_REDIRECT_BLOCKED` 失败,要求发起新的工具调用(沿用 Claude Code 的 WebFetch 模式)。
|
||||
- 发送显式的产品 `User-Agent`,绝不伪装成浏览器。
|
||||
- 不受支持的内容类型(例如二进制)以 `WEB_UNSUPPORTED_CONTENT_TYPE` 拒绝。
|
||||
|
||||
## 配置
|
||||
|
||||
| 配置键 | 默认值 | 含义 |
|
||||
|---|---|---|
|
||||
| `maxUrlLength` | `2048` | 接受的请求 URL 最大长度。 |
|
||||
| `maxResponseBytes` | `5_000_000` | 响应主体最大字节数。 |
|
||||
| `maxBodyChars` | `100_000` | 解码主体最大字符数。 |
|
||||
| `timeoutMs` | `30_000` | Node 定时器范围内的抓取超时:直接 `ctx.web.fetch()` 调用方的资源兜底,而非面向模型的工具调用预算(后者属于 `dsh-tool-call-timeout-policy`)。 |
|
||||
| `maxRedirects` | `5` | 同源重定向最大跳数(`0` 表示完全不跟随)。 |
|
||||
| `userAgent` | `deepseek-harness/…` | `User-Agent` 标头。 |
|
||||
|
||||
数值限制会在插件构造时验证:除 `maxRedirects` 外,每个上限都必须是正的有限数;`maxRedirects` 必须是非负整数。无效值会抛出异常,不会静默构造限制荒谬的提供方。
|
||||
|
||||
## 模型体验
|
||||
|
||||
通过 [`dsh-tool-web`](../tool-web/README.md) 间接影响;该工具把此提供方经 `maxBodyChars` 限制的解码文本或由 HTML 转换得到的 markdown 置于抓取结果包装层中,并保留提供方失败;重定向、标头与传输机制保持隐藏。
|
||||
|
||||
#### KV Cache 影响
|
||||
|
||||
不会直接导致 KV Cache 失效;请求前缀变更由上述消费方负责。
|
||||
|
||||
## 已知限制与暂缓事项
|
||||
|
||||
- **SSRF/私有网络防护暂缓**:不会阻止私有、loopback、link-local、multicast 或其他非公开目标,也不进行 DNS 解析后验证或逐跳重新验证(见 [web 能力 seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md))。在此功能落地前,该提供方是 SSRF 原语;能够访问敏感内部网络目标的部署**禁止启用它**。
|
||||
- **只解码文本内容**:包括 html/xhtml 与 `text/*` 加 JSON/XML 家族;缺少 `Content-Type` 或任何二进制类型都会抛出 `WEB_UNSUPPORTED_CONTENT_TYPE`,可提取文本的 PDF 解码属于明确的暂缓工作。
|
||||
- **charset 只来自 `Content-Type` 标头**(默认为 UTF-8):HTML `<meta charset>` 声明会被忽略;声明但无法识别的 charset 标签会抛出异常,而非回退。
|
||||
49
packages/web/web-fetch-http/package.json
Normal file
49
packages/web/web-fetch-http/package.json
Normal file
@@ -0,0 +1,49 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-web-fetch-http",
|
||||
"description": "Anonymous public HTTP(S) fetch provider for the DeepSeek Harness web capability seam (ctx.web)",
|
||||
"version": "0.0.1-rc.2",
|
||||
"publishConfig": {
|
||||
"access": "restricted"
|
||||
},
|
||||
"repository": {
|
||||
"type": "git",
|
||||
"url": "git+https://github.com/deepseek-ai/deepseek-harness.git",
|
||||
"directory": "packages/web/web-fetch-http"
|
||||
},
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/types/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./invariant": {
|
||||
"types": "./lib/types/invariant.d.ts",
|
||||
"default": "./lib/invariant.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib/index.js",
|
||||
"lib/invariant.js",
|
||||
"lib/types/**/*.d.ts"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-timeout": "workspace:^",
|
||||
"@deepseek-ai/dsh-web": "workspace:^",
|
||||
"@deepseek-ai/cordis": "workspace:^"
|
||||
},
|
||||
"dependencies": {
|
||||
"@deepseek-ai/schemastery": "workspace:^"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-timeout": "workspace:^",
|
||||
"@deepseek-ai/dsh-web": "workspace:^",
|
||||
"@deepseek-ai/cordis": "workspace:^"
|
||||
}
|
||||
}
|
||||
101
packages/web/web-fetch-http/src/index.ts
Normal file
101
packages/web/web-fetch-http/src/index.ts
Normal file
@@ -0,0 +1,101 @@
|
||||
/**
|
||||
* `@deepseek-ai/dsh-web-fetch-http`: registers an anonymous public HTTP(S)
|
||||
* `WebFetchProvider` with `ctx.web`. A function/namespace plugin (NOT a
|
||||
* default-export service): it registers INTO the seam's fetch registry, like the
|
||||
* search providers register into the search registry.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web-fetch-http
|
||||
*/
|
||||
|
||||
import type { Context } from '@deepseek-ai/cordis'
|
||||
import z from '@deepseek-ai/schemastery'
|
||||
import type {} from '@deepseek-ai/dsh-web'
|
||||
import { HttpFetchProvider } from './provider.ts'
|
||||
import type { HttpFetchLimits } from './provider.ts'
|
||||
|
||||
const MAX_NODE_TIMER_DELAY_MS = 2_147_483_647
|
||||
|
||||
export {
|
||||
LOCAL_FETCH_PROVIDER_ID,
|
||||
HttpFetchProvider,
|
||||
} from './provider.ts'
|
||||
export type { HttpFetchLimits } from './provider.ts'
|
||||
|
||||
/** Default `User-Agent`: an explicit product agent, never a browser disguise. */
|
||||
export const DEFAULT_USER_AGENT = 'deepseek-harness/0.0.1 (+https://github.com/deepseek-ai)'
|
||||
|
||||
/** Cordis plugin name used by loader diagnostics. */
|
||||
export const name = 'web-fetch-http'
|
||||
|
||||
/** The web seam this provider registers into. */
|
||||
export const inject = ['web']
|
||||
|
||||
/** Plugin config: the provider's transport and size limits plus its `User-Agent` (all defaulted). */
|
||||
export interface Config {
|
||||
/** Maximum accepted request URL length. */
|
||||
maxUrlLength?: number
|
||||
/** Maximum response body size in bytes. */
|
||||
maxResponseBytes?: number
|
||||
/** Maximum decoded body length in characters. */
|
||||
maxBodyChars?: number
|
||||
/** Default fetch timeout in milliseconds, within Node's timer range. */
|
||||
timeoutMs?: number
|
||||
/** Maximum number of same-origin redirect hops to follow. */
|
||||
maxRedirects?: number
|
||||
/** `User-Agent` header sent on every request. */
|
||||
userAgent?: string
|
||||
}
|
||||
|
||||
export const Config: z<Config> = z.object({
|
||||
maxUrlLength: z.number().default(2048),
|
||||
maxResponseBytes: z.number().default(5_000_000),
|
||||
maxBodyChars: z.number().default(100_000),
|
||||
timeoutMs: z.number().default(30_000),
|
||||
maxRedirects: z.number().default(5),
|
||||
userAgent: z.string().default(DEFAULT_USER_AGENT),
|
||||
})
|
||||
|
||||
/** Complete config after schemastery applies every field default. */
|
||||
type ResolvedConfig = Required<Config>
|
||||
|
||||
/** A resource limit (byte/char/length/timeout cap) must be a positive finite number. */
|
||||
function assertPositiveFinite(name: string, value: number): void {
|
||||
if (!Number.isFinite(value) || value <= 0) {
|
||||
throw new Error(`web-fetch-http: ${name} must be a positive finite number`)
|
||||
}
|
||||
}
|
||||
|
||||
/** Node coerces larger timer delays to 1 ms, so reject them at configuration time. */
|
||||
function assertTimeoutMs(value: number): void {
|
||||
assertPositiveFinite('timeoutMs', value)
|
||||
if (value > MAX_NODE_TIMER_DELAY_MS) {
|
||||
throw new Error(`web-fetch-http: timeoutMs must be no greater than ${MAX_NODE_TIMER_DELAY_MS}`)
|
||||
}
|
||||
}
|
||||
|
||||
/** The redirect hop cap must be a non-negative integer (0 follows no redirects). */
|
||||
function assertNonNegativeInteger(name: string, value: number): void {
|
||||
if (!Number.isInteger(value) || value < 0) {
|
||||
throw new Error(`web-fetch-http: ${name} must be a non-negative integer`)
|
||||
}
|
||||
}
|
||||
|
||||
/** Register the local HTTP(S) fetch provider with `ctx.web`. */
|
||||
export function apply(ctx: Context, config: Config): void {
|
||||
// schemastery (Config) has already filled every defaulted field.
|
||||
const resolved = config as ResolvedConfig
|
||||
assertPositiveFinite('maxUrlLength', resolved.maxUrlLength)
|
||||
assertPositiveFinite('maxResponseBytes', resolved.maxResponseBytes)
|
||||
assertPositiveFinite('maxBodyChars', resolved.maxBodyChars)
|
||||
assertTimeoutMs(resolved.timeoutMs)
|
||||
assertNonNegativeInteger('maxRedirects', resolved.maxRedirects)
|
||||
const limits: HttpFetchLimits = {
|
||||
maxUrlLength: resolved.maxUrlLength,
|
||||
maxResponseBytes: resolved.maxResponseBytes,
|
||||
maxBodyChars: resolved.maxBodyChars,
|
||||
timeoutMs: resolved.timeoutMs,
|
||||
maxRedirects: resolved.maxRedirects,
|
||||
userAgent: resolved.userAgent,
|
||||
}
|
||||
ctx.web.registerFetchProvider(new HttpFetchProvider(limits))
|
||||
}
|
||||
30
packages/web/web-fetch-http/src/invariant.ts
Normal file
30
packages/web/web-fetch-http/src/invariant.ts
Normal file
@@ -0,0 +1,30 @@
|
||||
/**
|
||||
* Package-owned invariant companion for `@deepseek-ai/dsh-web-fetch-http`.
|
||||
* @module @deepseek-ai/dsh-web-fetch-http/invariant
|
||||
*/
|
||||
|
||||
/* jscpd:ignore-start */
|
||||
import type { Context } from '@deepseek-ai/cordis'
|
||||
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
|
||||
|
||||
const PACKAGE_NAME = '@deepseek-ai/dsh-web-fetch-http'
|
||||
|
||||
/** Cordis companion plugin name. */
|
||||
export const name = 'web-fetch-http-invariant'
|
||||
/** Service required before the companion can reserve package ownership. */
|
||||
export const inject = ['invariants']
|
||||
|
||||
/**
|
||||
* No runtime invariant: this package exposes no independent event sequence or mutable data relation
|
||||
* beyond contracts enforced at its owning seam.
|
||||
*/
|
||||
const install: InvariantInstaller = () => {}
|
||||
|
||||
/**
|
||||
* Register this package's invariant companion.
|
||||
* @param ctx - Cordis context carrying the invariant service.
|
||||
* @returns the installed registration's disposer after setup succeeds.
|
||||
*/
|
||||
export const apply = (ctx: Context): Promise<() => void> =>
|
||||
Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
|
||||
/* jscpd:ignore-end */
|
||||
105
packages/web/web-fetch-http/src/policy.ts
Normal file
105
packages/web/web-fetch-http/src/policy.ts
Normal file
@@ -0,0 +1,105 @@
|
||||
/**
|
||||
* URL validation and content-type classification for the local HTTP(S) fetch
|
||||
* provider — the pure, network-free half. The provider's `fetch()` composes
|
||||
* these with transport (redirect following, byte caps, decoding).
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web-fetch-http/policy
|
||||
*/
|
||||
|
||||
import { WebError } from '@deepseek-ai/dsh-web'
|
||||
|
||||
/** The body kinds this provider decodes. */
|
||||
export type FetchableKind = 'html' | 'text'
|
||||
|
||||
/**
|
||||
* Validate a request URL against the basic transport hygiene the provider
|
||||
* enforces before any network access: http(s) only, no embedded credentials,
|
||||
* bounded length. Returns the parsed `URL`. Throws {@link WebError} otherwise.
|
||||
* (SSRF / private-network blocking is deferred — see the package Agent Note.)
|
||||
*
|
||||
* @param input - the raw URL string from the fetch request.
|
||||
* @param maxUrlLength - inclusive upper bound on `input`'s length.
|
||||
* @returns the parsed `URL`.
|
||||
*/
|
||||
export function validateFetchUrl(input: string, maxUrlLength: number): URL {
|
||||
if (input.length > maxUrlLength) {
|
||||
throw new WebError(`URL exceeds the maximum length of ${maxUrlLength}`, 'WEB_INVALID_URL')
|
||||
}
|
||||
let url: URL
|
||||
try {
|
||||
url = new URL(input)
|
||||
} catch (error: unknown) {
|
||||
throw new WebError(`invalid URL: ${input}`, 'WEB_INVALID_URL', { cause: error })
|
||||
}
|
||||
if (url.protocol !== 'http:' && url.protocol !== 'https:') {
|
||||
throw new WebError(`unsupported URL scheme "${url.protocol}" (only http and https are allowed)`, 'WEB_INVALID_URL')
|
||||
}
|
||||
if (url.username.length > 0 || url.password.length > 0) {
|
||||
throw new WebError('credentials in URLs are not allowed', 'WEB_BLOCKED_URL')
|
||||
}
|
||||
return url
|
||||
}
|
||||
|
||||
/**
|
||||
* Two URLs are same-origin when scheme, hostname, and port match. A redirect
|
||||
* that crosses origins is refused so each new origin requires a fresh tool call
|
||||
* (and thus a fresh provider/permission decision).
|
||||
*
|
||||
* @param a - one of the two URLs to compare.
|
||||
* @param b - the other URL to compare.
|
||||
* @returns true when `a` and `b` share scheme, hostname, and port.
|
||||
*/
|
||||
export function isSameOrigin(a: URL, b: URL): boolean {
|
||||
return a.protocol === b.protocol && a.hostname === b.hostname && a.port === b.port
|
||||
}
|
||||
|
||||
/**
|
||||
* Classify a response `Content-Type` into a decodable body kind, or `undefined`
|
||||
* for an unsupported (e.g. binary) type. `text/html` and `application/xhtml+xml`
|
||||
* are `html`; other `text/*` plus a few structured text types are `text`.
|
||||
*
|
||||
* @param contentType - the raw `Content-Type` header, or `null` when the
|
||||
* response carries none (unsupported).
|
||||
* @returns the decodable kind, or `undefined` for an unsupported type.
|
||||
*/
|
||||
export function classifyContentType(contentType: string | null): FetchableKind | undefined {
|
||||
const mime = (contentType ?? '').replace(/;.*$/s, '').trim().toLowerCase()
|
||||
if (mime === 'text/html' || mime === 'application/xhtml+xml') return 'html'
|
||||
if (mime.startsWith('text/')) return 'text'
|
||||
if (mime === 'application/json' || mime === 'application/xml' || mime.endsWith('+json') || mime.endsWith('+xml')) return 'text'
|
||||
return undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract the `charset` parameter from a response `Content-Type`, lower-cased,
|
||||
* or `undefined` when absent. The provider feeds this label to `TextDecoder`
|
||||
* so a non-UTF-8 response is decoded with its declared encoding rather than
|
||||
* silently mangled into replacement characters.
|
||||
*
|
||||
* @param contentType - the raw `Content-Type` header, or `null` when the
|
||||
* response carries none.
|
||||
* @returns the lower-cased charset label, or `undefined` when none is declared.
|
||||
*/
|
||||
export function parseCharset(contentType: string | null): string | undefined {
|
||||
const match = /;\s*charset\s*=\s*"?([^";]+)"?/i.exec(contentType ?? '')
|
||||
return match?.[1]?.trim().toLowerCase()
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a `TextDecoder` for the declared charset, falling back to UTF-8 when
|
||||
* none is declared. Throws {@link WebError} `WEB_UNSUPPORTED_CONTENT_TYPE` when
|
||||
* the label is present but not a charset `TextDecoder` recognizes — better to
|
||||
* fail loudly than return mojibake.
|
||||
*
|
||||
* @param charset - the declared charset label (from {@link parseCharset}), or
|
||||
* `undefined` to default to UTF-8.
|
||||
* @returns a decoder for the declared (or defaulted) encoding.
|
||||
*/
|
||||
export function decoderForCharset(charset: string | undefined): TextDecoder {
|
||||
if (charset === undefined) return new TextDecoder('utf-8')
|
||||
try {
|
||||
return new TextDecoder(charset)
|
||||
} catch (error: unknown) {
|
||||
throw new WebError(`unsupported charset "${charset}"`, 'WEB_UNSUPPORTED_CONTENT_TYPE', { cause: error })
|
||||
}
|
||||
}
|
||||
240
packages/web/web-fetch-http/src/provider.ts
Normal file
240
packages/web/web-fetch-http/src/provider.ts
Normal file
@@ -0,0 +1,240 @@
|
||||
/**
|
||||
* Safe HTTP(S) retrieval for `ctx.web`: validates URLs, follows only same-origin redirects,
|
||||
* enforces time and size limits, classifies and decodes text, and leaves presentation to
|
||||
* `@deepseek-ai/dsh-tool-web`. Requests carry no browser cookies or ambient credentials.
|
||||
*
|
||||
* Private-network and SSRF protection is not implemented; do not enable this provider where
|
||||
* it can reach sensitive internal targets.
|
||||
* @module @deepseek-ai/dsh-web-fetch-http/provider
|
||||
*/
|
||||
|
||||
import { WebError } from '@deepseek-ai/dsh-web'
|
||||
import type { WebFetchBody, WebFetchProvider, WebFetchRequest, WebFetchResult } from '@deepseek-ai/dsh-web'
|
||||
import { deadline, timeoutOf } from '@deepseek-ai/dsh-timeout'
|
||||
import { classifyContentType, decoderForCharset, isSameOrigin, parseCharset, validateFetchUrl } from './policy.ts'
|
||||
|
||||
/** Resolved provider limits (the plugin's schemastery Config supplies defaults). */
|
||||
export interface HttpFetchLimits {
|
||||
/** Maximum accepted request URL length. */
|
||||
maxUrlLength: number
|
||||
/** Maximum response body size in bytes (read is aborted past this). */
|
||||
maxResponseBytes: number
|
||||
/** Maximum decoded body length in characters (truncated past this). */
|
||||
maxBodyChars: number
|
||||
/** Default fetch timeout in milliseconds. */
|
||||
timeoutMs: number
|
||||
/** Maximum number of (same-origin) redirect hops to follow. */
|
||||
maxRedirects: number
|
||||
/** `User-Agent` header sent on every request. */
|
||||
userAgent: string
|
||||
}
|
||||
|
||||
/** Stable id this provider registers under. */
|
||||
export const LOCAL_FETCH_PROVIDER_ID = 'http'
|
||||
|
||||
/** The anonymous public HTTP(S) fetch provider. */
|
||||
export class HttpFetchProvider implements WebFetchProvider {
|
||||
readonly id = LOCAL_FETCH_PROVIDER_ID
|
||||
|
||||
constructor(private readonly limits: HttpFetchLimits) {}
|
||||
|
||||
/** No credentials to check — an anonymous public fetcher is always usable. */
|
||||
available(): boolean {
|
||||
return true
|
||||
}
|
||||
|
||||
async fetch(request: WebFetchRequest, signal?: AbortSignal): Promise<WebFetchResult> {
|
||||
if (signal?.aborted) throw new WebError('web fetch aborted', 'WEB_ABORTED')
|
||||
|
||||
// One signal stops both the request and body read. The deadline's TimeoutReason later
|
||||
// distinguishes this provider's timeout from caller or outer-deadline cancellation.
|
||||
using d = deadline(signal, this.limits.timeoutMs, 'WEB_FETCH_TIMEOUT')
|
||||
return await this.followAndRead(request.url, d.signal)
|
||||
}
|
||||
|
||||
/** Follow same-origin redirects up to the hop cap, then read the final response. */
|
||||
private async followAndRead(initialUrl: string, signal: AbortSignal): Promise<WebFetchResult> {
|
||||
let currentUrl = validateFetchUrl(initialUrl, this.limits.maxUrlLength)
|
||||
let redirectsFollowed = 0
|
||||
|
||||
for (;;) {
|
||||
const response = await this.requestOnce(currentUrl, signal)
|
||||
|
||||
if (isRedirectStatus(response.status)) {
|
||||
// Enforce the redirect budget before resolving or validating the next hop.
|
||||
if (redirectsFollowed >= this.limits.maxRedirects) {
|
||||
await response.body?.cancel()
|
||||
throw new WebError(`exceeded the maximum of ${this.limits.maxRedirects} redirects`, 'WEB_REDIRECT_BLOCKED')
|
||||
}
|
||||
const location = response.headers.get('location')
|
||||
if (location === null) {
|
||||
// A redirect status with no Location is not a usable resource. Cancel
|
||||
// the (possibly streaming) body before throwing so no socket leaks.
|
||||
await response.body?.cancel()
|
||||
throw new WebError(`redirect response (HTTP ${response.status}) without a Location header`, 'WEB_PROVIDER_ERROR')
|
||||
}
|
||||
const target = resolveRedirect(location, currentUrl)
|
||||
// Re-validate the target against the same transport hygiene a direct request gets: a
|
||||
// redirect must not be a back door to a credentialed, non-http(s), or over-long URL
|
||||
// that validateFetchUrl would reject.
|
||||
let validatedTarget: URL
|
||||
try {
|
||||
validatedTarget = validateFetchUrl(target.toString(), this.limits.maxUrlLength)
|
||||
if (!isSameOrigin(validatedTarget, currentUrl)) {
|
||||
throw new WebError(
|
||||
`cross-origin redirect to ${validatedTarget.origin} is not followed automatically; retry against that URL directly`,
|
||||
'WEB_REDIRECT_BLOCKED',
|
||||
)
|
||||
}
|
||||
} catch (error: unknown) {
|
||||
await response.body?.cancel()
|
||||
throw error
|
||||
}
|
||||
await response.body?.cancel()
|
||||
currentUrl = validatedTarget
|
||||
redirectsFollowed++
|
||||
continue
|
||||
}
|
||||
|
||||
return await this.readBody(response, currentUrl, signal)
|
||||
}
|
||||
}
|
||||
|
||||
private async requestOnce(url: URL, signal: AbortSignal): Promise<Response> {
|
||||
try {
|
||||
return await fetch(url, {
|
||||
method: 'GET',
|
||||
redirect: 'manual',
|
||||
headers: { 'user-agent': this.limits.userAgent, 'accept': 'text/html,application/xhtml+xml,text/*;q=0.9,application/json;q=0.8' },
|
||||
signal,
|
||||
})
|
||||
} catch (error: unknown) {
|
||||
throw translateAbortOrNetwork(error, signal)
|
||||
}
|
||||
}
|
||||
|
||||
/** Read, byte-cap, classify, and decode the final response body. */
|
||||
private async readBody(response: Response, finalUrl: URL, signal: AbortSignal): Promise<WebFetchResult> {
|
||||
const contentType = response.headers.get('content-type')
|
||||
const kind = classifyContentType(contentType)
|
||||
if (kind === undefined) {
|
||||
await response.body?.cancel()
|
||||
throw new WebError(`unsupported content type "${contentType ?? 'unknown'}"`, 'WEB_UNSUPPORTED_CONTENT_TYPE')
|
||||
}
|
||||
|
||||
// Resolve the decoder BEFORE reading the body so an unsupported charset
|
||||
// fails without consuming the stream — but cancel the body on that failure
|
||||
// so the socket does not leak (matching the unsupported-content-type path).
|
||||
let decoder: TextDecoder
|
||||
try {
|
||||
decoder = decoderForCharset(parseCharset(contentType))
|
||||
} catch (error: unknown) {
|
||||
await response.body?.cancel()
|
||||
throw error
|
||||
}
|
||||
const { bytes, truncatedByBytes } = await this.readCapped(response, signal)
|
||||
const decoded = decoder.decode(bytes)
|
||||
const truncatedByChars = decoded.length > this.limits.maxBodyChars
|
||||
const content = truncatedByChars ? decoded.slice(0, this.limits.maxBodyChars) : decoded
|
||||
const body: WebFetchBody = kind === 'html' ? { kind: 'html', content } : { kind: 'text', content }
|
||||
|
||||
return {
|
||||
url: finalUrl.toString(),
|
||||
statusCode: response.status,
|
||||
body,
|
||||
truncated: truncatedByBytes || truncatedByChars,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the response stream up to `maxResponseBytes`. A `Content-Length` over
|
||||
* the cap rejects immediately with `WEB_FETCH_TOO_LARGE`; a stream that grows
|
||||
* past the cap is cut short (`truncatedByBytes`) rather than rejected, so a
|
||||
* server that under-reports still yields a bounded usable body.
|
||||
*/
|
||||
private async readCapped(response: Response, signal: AbortSignal): Promise<{ bytes: Uint8Array; truncatedByBytes: boolean }> {
|
||||
const declared = response.headers.get('content-length')
|
||||
if (declared !== null) {
|
||||
const length = Number(declared)
|
||||
if (Number.isFinite(length) && length > this.limits.maxResponseBytes) {
|
||||
await response.body?.cancel()
|
||||
throw new WebError(`response exceeds the maximum of ${this.limits.maxResponseBytes} bytes`, 'WEB_FETCH_TOO_LARGE')
|
||||
}
|
||||
}
|
||||
|
||||
/* v8 ignore next -- a 2xx Response from fetch always exposes a body stream; the null guard is defensive. */
|
||||
if (response.body === null) return { bytes: new Uint8Array(0), truncatedByBytes: false }
|
||||
|
||||
const chunks: Uint8Array[] = []
|
||||
let total = 0
|
||||
let truncatedByBytes = false
|
||||
const reader = response.body.getReader()
|
||||
try {
|
||||
for (;;) {
|
||||
const { done, value } = await reader.read()
|
||||
if (done) break
|
||||
const remaining = this.limits.maxResponseBytes - total
|
||||
// Only DROPPED bytes count as truncation: a chunk that exactly fills the
|
||||
// remaining capacity keeps all its bytes and we read on to observe EOF,
|
||||
// so an exactly-at-cap body is not falsely flagged truncated.
|
||||
if (value.byteLength > remaining) {
|
||||
chunks.push(value.subarray(0, remaining))
|
||||
total += remaining
|
||||
truncatedByBytes = true
|
||||
break
|
||||
}
|
||||
chunks.push(value)
|
||||
total += value.byteLength
|
||||
}
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore next -- mid-stream read fault needs a network drop after headers; translate path covered by request-phase tests. */
|
||||
throw translateAbortOrNetwork(error, signal)
|
||||
} finally {
|
||||
/* v8 ignore next 4 -- cancel() after a completed/broken read settles without rejecting; unobserved best-effort cleanup. */
|
||||
await reader.cancel().catch(() => {
|
||||
// Cancel after a successful read (or after we broke past the cap) is
|
||||
// best-effort cleanup; the bytes we need are already collected.
|
||||
})
|
||||
}
|
||||
|
||||
const bytes = new Uint8Array(total)
|
||||
let offset = 0
|
||||
for (const chunk of chunks) {
|
||||
bytes.set(chunk, offset)
|
||||
offset += chunk.byteLength
|
||||
}
|
||||
return { bytes, truncatedByBytes }
|
||||
}
|
||||
}
|
||||
|
||||
/** HTTP redirect status codes that carry a `Location`. */
|
||||
function isRedirectStatus(status: number): boolean {
|
||||
return status === 301 || status === 302 || status === 303 || status === 307 || status === 308
|
||||
}
|
||||
|
||||
/** Resolve a (possibly relative) `Location` against the current URL. */
|
||||
function resolveRedirect(location: string, base: URL): URL {
|
||||
try {
|
||||
return new URL(location, base)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore next 2 -- URL resolution against a valid absolute base effectively never throws; defensive guard. */
|
||||
throw new WebError(`invalid redirect Location "${location}"`, 'WEB_PROVIDER_ERROR', { cause: error })
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Translate a thrown fetch/stream error into a `WebError`, classified by the
|
||||
* deadline signal rather than the thrown value (which differs by phase: the
|
||||
* request-phase `fetch` rejects with the abort reason, while the read-phase
|
||||
* reader surfaces a bare `AbortError`). `timeoutOf(signal, 'WEB_FETCH_TIMEOUT')`
|
||||
* recovering OUR reason means our timeout fired (`WEB_FETCH_TIMEOUT`); any other
|
||||
* abort — an upstream cancel, or a foreign/outer deadline's timeout under
|
||||
* nesting — is `WEB_ABORTED`; a throw with the signal NOT aborted is a
|
||||
* transport/network failure (`WEB_PROVIDER_ERROR`).
|
||||
*/
|
||||
function translateAbortOrNetwork(error: unknown, signal: AbortSignal): WebError {
|
||||
const timeout = timeoutOf(signal, 'WEB_FETCH_TIMEOUT')
|
||||
if (timeout !== undefined) return new WebError('web fetch timed out', 'WEB_FETCH_TIMEOUT', { cause: timeout })
|
||||
if (signal.aborted) return new WebError('web fetch aborted', 'WEB_ABORTED', { cause: error })
|
||||
return new WebError(`web fetch failed: ${String(error)}`, 'WEB_PROVIDER_ERROR', { cause: error })
|
||||
}
|
||||
429
packages/web/web-fetch-http/tests/fetch-http.spec.ts
Normal file
429
packages/web/web-fetch-http/tests/fetch-http.spec.ts
Normal file
@@ -0,0 +1,429 @@
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http'
|
||||
import { AddressInfo } from 'node:net'
|
||||
import { Context } from '@deepseek-ai/cordis'
|
||||
import WebRuntime from '@deepseek-ai/dsh-web'
|
||||
import { HttpFetchProvider, LOCAL_FETCH_PROVIDER_ID } from '@deepseek-ai/dsh-web-fetch-http'
|
||||
import type { HttpFetchLimits } from '@deepseek-ai/dsh-web-fetch-http'
|
||||
import * as fetchPlugin from '@deepseek-ai/dsh-web-fetch-http'
|
||||
import { classifyContentType, decoderForCharset, isSameOrigin, parseCharset, validateFetchUrl } from '../src/policy.ts'
|
||||
|
||||
const limits: HttpFetchLimits = {
|
||||
maxUrlLength: 2048,
|
||||
maxResponseBytes: 5_000_000,
|
||||
maxBodyChars: 100_000,
|
||||
timeoutMs: 5_000,
|
||||
maxRedirects: 5,
|
||||
userAgent: 'test-agent/1.0',
|
||||
}
|
||||
|
||||
type Handler = (req: IncomingMessage, res: ServerResponse) => void
|
||||
|
||||
let server: Server
|
||||
let base: string
|
||||
let handler: Handler
|
||||
|
||||
beforeEach(async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('default') }
|
||||
server = createServer((req, res) => { handler(req, res) })
|
||||
await new Promise<void>(resolve => server.listen(0, '127.0.0.1', resolve))
|
||||
const { port } = server.address() as AddressInfo
|
||||
base = `http://127.0.0.1:${port}`
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
vi.unstubAllGlobals()
|
||||
await new Promise<void>(resolve => server.close(() => { resolve() }))
|
||||
})
|
||||
|
||||
function provider(overrides: Partial<HttpFetchLimits> = {}): HttpFetchProvider {
|
||||
return new HttpFetchProvider({ ...limits, ...overrides })
|
||||
}
|
||||
|
||||
describe('policy helpers', () => {
|
||||
it('validates scheme, credentials, and length', () => {
|
||||
expect(validateFetchUrl('https://example.com/x', 2048).hostname).toBe('example.com')
|
||||
expect(() => validateFetchUrl('ftp://example.com', 2048)).toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' }))
|
||||
expect(() => validateFetchUrl('not a url', 2048)).toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' }))
|
||||
expect(() => validateFetchUrl('https://user:pass@example.com', 2048)).toThrow(expect.objectContaining({ code: 'WEB_BLOCKED_URL' }))
|
||||
expect(() => validateFetchUrl(`https://example.com/${'a'.repeat(3000)}`, 2048)).toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' }))
|
||||
})
|
||||
|
||||
it('classifies content types', () => {
|
||||
expect(classifyContentType('text/html; charset=utf-8')).toBe('html')
|
||||
expect(classifyContentType('application/xhtml+xml')).toBe('html')
|
||||
expect(classifyContentType('text/plain')).toBe('text')
|
||||
expect(classifyContentType('application/json')).toBe('text')
|
||||
expect(classifyContentType('image/png')).toBeUndefined()
|
||||
expect(classifyContentType(null)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('compares origins', () => {
|
||||
expect(isSameOrigin(new URL('https://a.com/x'), new URL('https://a.com/y'))).toBe(true)
|
||||
expect(isSameOrigin(new URL('https://a.com'), new URL('https://b.com'))).toBe(false)
|
||||
expect(isSameOrigin(new URL('http://a.com'), new URL('https://a.com'))).toBe(false)
|
||||
})
|
||||
|
||||
it('parses the charset parameter', () => {
|
||||
expect(parseCharset('text/html; charset=UTF-8')).toBe('utf-8')
|
||||
expect(parseCharset('text/plain; charset="iso-8859-1"')).toBe('iso-8859-1')
|
||||
expect(parseCharset('text/plain')).toBeUndefined()
|
||||
expect(parseCharset(null)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('builds a decoder for a charset and defaults to UTF-8', () => {
|
||||
expect(decoderForCharset(undefined).encoding).toBe('utf-8')
|
||||
expect(decoderForCharset('iso-8859-1').encoding).toBe('windows-1252')
|
||||
expect(() => decoderForCharset('not-a-charset')).toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' }))
|
||||
})
|
||||
})
|
||||
|
||||
describe('HttpFetchProvider success', () => {
|
||||
it('fetches a text body', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('hello world') }
|
||||
const result = await provider().fetch({ url: base })
|
||||
expect(provider().available()).toBe(true)
|
||||
expect(result.statusCode).toBe(200)
|
||||
expect(result.body).toEqual({ kind: 'text', content: 'hello world' })
|
||||
expect(result.truncated).toBe(false)
|
||||
})
|
||||
|
||||
it('fetches an html body and classifies it as html', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/html' }); res.end('<h1>hi</h1>') }
|
||||
const result = await provider().fetch({ url: base })
|
||||
expect(result.body).toEqual({ kind: 'html', content: '<h1>hi</h1>' })
|
||||
})
|
||||
|
||||
it('sends the configured user agent', async () => {
|
||||
let seen: string | undefined
|
||||
handler = (req, res) => { seen = req.headers['user-agent']; res.writeHead(200, { 'content-type': 'text/plain' }); res.end('ok') }
|
||||
await provider().fetch({ url: base })
|
||||
expect(seen).toBe('test-agent/1.0')
|
||||
})
|
||||
|
||||
it('returns a non-2xx response as a result, not an error', async () => {
|
||||
handler = (_req, res) => { res.writeHead(404, { 'content-type': 'text/plain' }); res.end('nope') }
|
||||
const result = await provider().fetch({ url: base })
|
||||
expect(result.statusCode).toBe(404)
|
||||
expect(result.body).toEqual({ kind: 'text', content: 'nope' })
|
||||
})
|
||||
})
|
||||
|
||||
describe('HttpFetchProvider caps', () => {
|
||||
it('rejects an over-cap Content-Length with WEB_FETCH_TOO_LARGE', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain', 'content-length': '999999' }); res.end('x'.repeat(999999)) }
|
||||
await expect(provider({ maxResponseBytes: 10 }).fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_FETCH_TOO_LARGE' }))
|
||||
})
|
||||
|
||||
it('truncates a stream that grows past the byte cap', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('abcdefghij') }
|
||||
const result = await provider({ maxResponseBytes: 4 }).fetch({ url: base })
|
||||
expect(result.body.content).toBe('abcd')
|
||||
expect(result.truncated).toBe(true)
|
||||
})
|
||||
|
||||
it('does not flag a body that exactly fills the byte cap as truncated', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('abcd') }
|
||||
const result = await provider({ maxResponseBytes: 4 }).fetch({ url: base })
|
||||
expect(result.body.content).toBe('abcd')
|
||||
expect(result.truncated).toBe(false)
|
||||
})
|
||||
|
||||
it('truncates a decoded body past the character cap', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('abcdefghij') }
|
||||
const result = await provider({ maxBodyChars: 3 }).fetch({ url: base })
|
||||
expect(result.body.content).toBe('abc')
|
||||
expect(result.truncated).toBe(true)
|
||||
})
|
||||
|
||||
it('rejects an unsupported content type', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'image/png' }); res.end('binary') }
|
||||
await expect(provider().fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' }))
|
||||
})
|
||||
|
||||
it('rejects a response with no content type at all', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200); res.end('no type') }
|
||||
await expect(provider().fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' }))
|
||||
})
|
||||
|
||||
it('accepts a declared content-length within the cap', async () => {
|
||||
handler = (_req, res) => { const body = 'sized'; res.writeHead(200, { 'content-type': 'text/plain', 'content-length': String(body.length) }); res.end(body) }
|
||||
const result = await provider().fetch({ url: base })
|
||||
expect(result.body.content).toBe('sized')
|
||||
})
|
||||
|
||||
it('decodes a non-UTF-8 declared charset', async () => {
|
||||
// 0xE9 is "é" in ISO-8859-1; decoded as UTF-8 it would be a replacement char.
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain; charset=iso-8859-1' }); res.end(Buffer.from([0x63, 0x61, 0x66, 0xE9])) }
|
||||
const result = await provider().fetch({ url: base })
|
||||
expect(result.body.content).toBe('café')
|
||||
})
|
||||
|
||||
it('rejects an unsupported declared charset', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain; charset=not-a-charset' }); res.end('x') }
|
||||
await expect(provider().fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' }))
|
||||
})
|
||||
})
|
||||
|
||||
describe('HttpFetchProvider redirects', () => {
|
||||
it('follows a same-origin redirect and reports the final URL', async () => {
|
||||
handler = (req, res) => {
|
||||
if (req.url === '/start') { res.writeHead(302, { location: '/end' }); res.end() }
|
||||
else { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('arrived') }
|
||||
}
|
||||
const result = await provider().fetch({ url: `${base}/start` })
|
||||
expect(result.body.content).toBe('arrived')
|
||||
expect(result.url).toBe(`${base}/end`)
|
||||
})
|
||||
|
||||
it('blocks a cross-origin redirect with WEB_REDIRECT_BLOCKED', async () => {
|
||||
handler = (_req, res) => { res.writeHead(302, { location: 'https://example.com/' }); res.end() }
|
||||
await expect(provider().fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_REDIRECT_BLOCKED' }))
|
||||
})
|
||||
|
||||
it('re-validates a redirect target, rejecting same-origin credentials in the Location', async () => {
|
||||
const { port } = server.address() as AddressInfo
|
||||
handler = (_req, res) => { res.writeHead(302, { location: `http://user:pass@127.0.0.1:${port}/` }); res.end() }
|
||||
await expect(provider().fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_BLOCKED_URL' }))
|
||||
})
|
||||
|
||||
it('rejects exceeding the redirect hop cap', async () => {
|
||||
handler = (req, res) => {
|
||||
const n = Number(new URL(req.url ?? '/', base).searchParams.get('n') ?? '0')
|
||||
res.writeHead(302, { location: `/?n=${n + 1}` })
|
||||
res.end()
|
||||
}
|
||||
await expect(provider({ maxRedirects: 2 }).fetch({ url: `${base}/?n=0` }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_REDIRECT_BLOCKED' }))
|
||||
})
|
||||
|
||||
it('follows exactly maxRedirects hops: a chain landing on the Nth redirect succeeds', async () => {
|
||||
// maxRedirects: 2 → /?n=0 → /?n=1 → /?n=2(200). Exactly 2 redirects + 1
|
||||
// final = 3 requests; the cap is inclusive of the landing request.
|
||||
let requests = 0
|
||||
handler = (req, res) => {
|
||||
requests++
|
||||
const n = Number(new URL(req.url ?? '/', base).searchParams.get('n') ?? '0')
|
||||
if (n >= 2) { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('landed') }
|
||||
else { res.writeHead(302, { location: `/?n=${n + 1}` }); res.end() }
|
||||
}
|
||||
const result = await provider({ maxRedirects: 2 }).fetch({ url: `${base}/?n=0` })
|
||||
expect(result.body.content).toBe('landed')
|
||||
expect(requests).toBe(3)
|
||||
})
|
||||
|
||||
it('makes exactly maxRedirects+1 requests before blocking an over-long chain', async () => {
|
||||
// maxRedirects: 2 on an infinite chain: requests at n=0,1,2 (the 3rd is the
|
||||
// over-limit redirect, refused before its Location is followed) = 3 total.
|
||||
let requests = 0
|
||||
handler = (req, res) => {
|
||||
requests++
|
||||
const n = Number(new URL(req.url ?? '/', base).searchParams.get('n') ?? '0')
|
||||
res.writeHead(302, { location: `/?n=${n + 1}` })
|
||||
res.end()
|
||||
}
|
||||
await expect(provider({ maxRedirects: 2 }).fetch({ url: `${base}/?n=0` }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_REDIRECT_BLOCKED', message: 'exceeded the maximum of 2 redirects' }))
|
||||
expect(requests).toBe(3)
|
||||
})
|
||||
|
||||
it('reports an over-limit redirect as "exceeded", not cross-origin, even when the over-limit hop points cross-origin', async () => {
|
||||
// The redirect budget is checked BEFORE the over-limit hop's target is
|
||||
// origin-validated, so the diagnosis is "exceeded", not "cross-origin".
|
||||
handler = (req, res) => {
|
||||
const n = Number(new URL(req.url ?? '/', base).searchParams.get('n') ?? '0')
|
||||
const location = n === 0 ? '/?n=1' : 'https://example.com/'
|
||||
res.writeHead(302, { location })
|
||||
res.end()
|
||||
}
|
||||
await expect(provider({ maxRedirects: 1 }).fetch({ url: `${base}/?n=0` }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_REDIRECT_BLOCKED', message: 'exceeded the maximum of 1 redirects' }))
|
||||
})
|
||||
|
||||
it('maxRedirects: 0 follows no redirect but still fetches a direct 200', async () => {
|
||||
handler = (req, res) => {
|
||||
if (req.url === '/r') { res.writeHead(302, { location: '/done' }); res.end() }
|
||||
else { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('direct') }
|
||||
}
|
||||
await expect(provider({ maxRedirects: 0 }).fetch({ url: `${base}/r` }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_REDIRECT_BLOCKED' }))
|
||||
const direct = await provider({ maxRedirects: 0 }).fetch({ url: `${base}/done` })
|
||||
expect(direct.body.content).toBe('direct')
|
||||
})
|
||||
|
||||
it('treats a redirect without a Location header as a provider error', async () => {
|
||||
handler = (_req, res) => { res.writeHead(302); res.end() }
|
||||
await expect(provider().fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' }))
|
||||
})
|
||||
|
||||
it('follows a relative same-origin redirect', async () => {
|
||||
handler = (req, res) => {
|
||||
if (req.url === '/a') { res.writeHead(301, { location: 'b' }); res.end() }
|
||||
else { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('landed') }
|
||||
}
|
||||
const result = await provider().fetch({ url: `${base}/a` })
|
||||
expect(result.body.content).toBe('landed')
|
||||
})
|
||||
})
|
||||
|
||||
describe('HttpFetchProvider invalid URLs and abort', () => {
|
||||
it('rejects a non-http scheme before any network access', async () => {
|
||||
await expect(provider().fetch({ url: 'ftp://example.com' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' }))
|
||||
})
|
||||
|
||||
it('rejects credentials in the URL', async () => {
|
||||
await expect(provider().fetch({ url: 'http://user:pass@127.0.0.1/' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_BLOCKED_URL' }))
|
||||
})
|
||||
|
||||
it('honors a pre-aborted signal', async () => {
|
||||
const controller = new AbortController()
|
||||
controller.abort()
|
||||
await expect(provider().fetch({ url: base }, controller.signal))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_ABORTED' }))
|
||||
})
|
||||
|
||||
it('aborts an in-flight fetch via the signal', async () => {
|
||||
handler = (_req, _res) => { /* never responds */ }
|
||||
const controller = new AbortController()
|
||||
const promise = provider().fetch({ url: base }, controller.signal)
|
||||
controller.abort()
|
||||
await expect(promise).rejects.toThrow(expect.objectContaining({ code: 'WEB_ABORTED' }))
|
||||
})
|
||||
|
||||
it('times out a slow response with WEB_FETCH_TIMEOUT', async () => {
|
||||
handler = (_req, _res) => { /* never responds */ }
|
||||
await expect(provider({ timeoutMs: 50 }).fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_FETCH_TIMEOUT' }))
|
||||
})
|
||||
|
||||
it('classifies a timeout DURING the body read as WEB_FETCH_TIMEOUT, not WEB_ABORTED', async () => {
|
||||
// Promise body that resolves headers (so fetch() returns) but a content-length
|
||||
// that outlasts the bytes sent, so readCapped()'s reader awaits more and the
|
||||
// timeout fires mid-read — the reader then surfaces a generic AbortError that
|
||||
// must still be recovered as the timeout reason via signal.reason.
|
||||
handler = (_req, res) => {
|
||||
res.writeHead(200, { 'content-type': 'text/plain', 'content-length': '100' })
|
||||
res.write('partial')
|
||||
// never send the remaining bytes nor end the response
|
||||
}
|
||||
await expect(provider({ timeoutMs: 80 }).fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_FETCH_TIMEOUT' }))
|
||||
})
|
||||
|
||||
it('maps a connection failure to WEB_PROVIDER_ERROR', async () => {
|
||||
// Port 1 on loopback is not listening: a real connection failure (not abort).
|
||||
await expect(provider().fetch({ url: 'http://127.0.0.1:1/' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' }))
|
||||
})
|
||||
|
||||
})
|
||||
|
||||
describe('HttpFetchProvider body cancellation on error paths', () => {
|
||||
/** A fake Response whose body.cancel is observable. */
|
||||
type FakeInit = { status: number; headers: Record<string, string>; location?: string }
|
||||
function fakeResponse(init: FakeInit): { response: Response; cancelled: () => boolean } {
|
||||
let cancelled = false
|
||||
const headers = new Headers(init.headers)
|
||||
if (init.location !== undefined) headers.set('location', init.location)
|
||||
const response = {
|
||||
status: init.status,
|
||||
headers,
|
||||
body: { cancel: () => { cancelled = true; return Promise.resolve() } },
|
||||
} as unknown as Response
|
||||
return { response, cancelled: () => cancelled }
|
||||
}
|
||||
|
||||
it('cancels the body when a cross-origin redirect is blocked', async () => {
|
||||
const { response, cancelled } = fakeResponse({ status: 302, headers: {}, location: 'https://elsewhere.test/' })
|
||||
vi.stubGlobal('fetch', vi.fn(async () => response))
|
||||
await expect(provider().fetch({ url: 'http://127.0.0.1:9/' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_REDIRECT_BLOCKED' }))
|
||||
expect(cancelled()).toBe(true)
|
||||
})
|
||||
|
||||
it('cancels the body when an unsupported charset is rejected', async () => {
|
||||
const { response, cancelled } = fakeResponse({ status: 200, headers: { 'content-type': 'text/plain; charset=not-a-charset' } })
|
||||
vi.stubGlobal('fetch', vi.fn(async () => response))
|
||||
await expect(provider().fetch({ url: 'http://127.0.0.1:9/' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' }))
|
||||
expect(cancelled()).toBe(true)
|
||||
})
|
||||
|
||||
it('cancels the body when a redirect has no Location header', async () => {
|
||||
const { response, cancelled } = fakeResponse({ status: 302, headers: {} })
|
||||
vi.stubGlobal('fetch', vi.fn(async () => response))
|
||||
await expect(provider().fetch({ url: 'http://127.0.0.1:9/' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' }))
|
||||
expect(cancelled()).toBe(true)
|
||||
})
|
||||
})
|
||||
|
||||
describe('web-fetch-http plugin registration', () => {
|
||||
it('registers the provider into ctx.web (HMR-safe)', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebRuntime, { fetchProvider: LOCAL_FETCH_PROVIDER_ID })
|
||||
const fiber = await ctx.plugin(fetchPlugin, {})
|
||||
await expect(ctx.web.fetch({ url: `${base}/` }))
|
||||
.resolves.toMatchObject({ statusCode: 200 })
|
||||
await fiber.dispose()
|
||||
await expect(ctx.web.fetch({ url: `${base}/` }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_CONFIGURED_MISSING' }))
|
||||
})
|
||||
|
||||
it('has no default export (namespace plugin export shape)', () => {
|
||||
expect('default' in fetchPlugin).toBe(false)
|
||||
})
|
||||
|
||||
it('rejects a non-positive resource limit at construction', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebRuntime, { fetchProvider: LOCAL_FETCH_PROVIDER_ID })
|
||||
await expect(ctx.plugin(fetchPlugin, { maxResponseBytes: -1 }))
|
||||
.rejects.toThrow(/maxResponseBytes must be a positive finite number/)
|
||||
})
|
||||
|
||||
it('rejects a zero timeout at construction', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebRuntime, { fetchProvider: LOCAL_FETCH_PROVIDER_ID })
|
||||
await expect(ctx.plugin(fetchPlugin, { timeoutMs: 0 }))
|
||||
.rejects.toThrow(/timeoutMs must be a positive finite number/)
|
||||
})
|
||||
|
||||
it('rejects a timeout beyond Node timer range at construction', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebRuntime, { fetchProvider: LOCAL_FETCH_PROVIDER_ID })
|
||||
await expect(ctx.plugin(fetchPlugin, { timeoutMs: 2_147_483_648 }))
|
||||
.rejects.toThrow(/timeoutMs must be no greater than 2147483647/)
|
||||
})
|
||||
|
||||
it('rejects a fractional redirect cap at construction', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebRuntime, { fetchProvider: LOCAL_FETCH_PROVIDER_ID })
|
||||
await expect(ctx.plugin(fetchPlugin, { maxRedirects: 1.5 }))
|
||||
.rejects.toThrow(/maxRedirects must be a non-negative integer/)
|
||||
})
|
||||
|
||||
it('rejects a negative redirect cap at construction', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebRuntime, { fetchProvider: LOCAL_FETCH_PROVIDER_ID })
|
||||
await expect(ctx.plugin(fetchPlugin, { maxRedirects: -1 }))
|
||||
.rejects.toThrow(/maxRedirects must be a non-negative integer/)
|
||||
})
|
||||
|
||||
it('accepts maxRedirects: 0 (follow no redirects) as valid config', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebRuntime, { fetchProvider: LOCAL_FETCH_PROVIDER_ID })
|
||||
const fiber = await ctx.plugin(fetchPlugin, { maxRedirects: 0 })
|
||||
await expect(ctx.web.fetch({ url: `${base}/` }))
|
||||
.resolves.toMatchObject({ statusCode: 200 })
|
||||
await fiber.dispose()
|
||||
})
|
||||
})
|
||||
30
packages/web/web-fetch-http/tsconfig.json
Normal file
30
packages/web/web-fetch-http/tsconfig.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib/types"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../../util/timeout"
|
||||
},
|
||||
{
|
||||
"path": "../web"
|
||||
},
|
||||
{
|
||||
"path": "../../runtime-diagnostics/invariants"
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user