Add web capability seam: ctx.web, search/fetch providers, web tools
Introduce web access as a first-class capability seam so the model-facing web tools stay stable while backends change. dsh-web owns ctx.web as a provider registry with registration-order-independent selection and the WebError taxonomy; dsh-web-search-exa, dsh-web-search-perplexity, and dsh-web-fetch-local register capabilities into it; dsh-tool-web is the sole owner of the model-facing web_search/web_fetch schemas, prompt sections, and HTML-to-markdown presentation. Search and fetch are deliberately one seam. Providers ship as namespace plugins that register into ctx.web (like an LlmAdapter into ctx.llm), not key-owning services, since multiple search providers cannot each own the key. Tool registration follows product enablement, not backend availability, so load order/credentials never enter the model contract; the seam resolves the provider at execution time and surfaces a structured WebError otherwise. Moves the RFC to implemented/ amended to match what shipped. Example/app configs are intentionally not wired yet (RFC migration step 6).
This commit is contained in:
34
packages/web/web-fetch-local/README.md
Normal file
34
packages/web/web-fetch-local/README.md
Normal file
@@ -0,0 +1,34 @@
|
||||
# @deepseek-ai/dsh-web-fetch-local
|
||||
|
||||
An anonymous public HTTP(S) `WebFetchProvider` for the harness [web capability seam](../web/README.md) (`ctx.web`). It retrieves a concrete URL and returns a status code plus bounded decoded content.
|
||||
|
||||
This is an **implementation** package: it registers a provider into `ctx.web`, it does not own the key and it does not register a model-facing tool. It is a function/namespace plugin (`inject: ['web']`).
|
||||
|
||||
## Responsibility split
|
||||
|
||||
The provider owns **safe resource retrieval**: URL validation, HTTP transport, redirect policy, timeout, abort propagation, byte caps, charset decoding, content-type classification, and binary rejection. `@deepseek-ai/dsh-tool-web` owns **presentation** (HTML→markdown, truncation formatting). A non-2xx HTTP response is a *result* (status code + decoded body), not an error; `WebError` is reserved for failures to safely retrieve or represent the resource.
|
||||
|
||||
## Transport hygiene
|
||||
|
||||
- Accepts only `http:` and `https:` URLs; rejects credentials in URLs (`WEB_BLOCKED_URL`) and over-long/malformed URLs (`WEB_INVALID_URL`).
|
||||
- Enforces a max URL length, response byte cap (`WEB_FETCH_TOO_LARGE`), decoded body character cap, timeout (`WEB_FETCH_TIMEOUT`), and redirect hop cap.
|
||||
- Propagates the caller's abort signal (`WEB_ABORTED`) into the network request and the streaming read.
|
||||
- Follows only **same-origin** redirects; a cross-origin redirect fails with `WEB_REDIRECT_BLOCKED`, requiring a fresh tool call (the model of Claude Code's WebFetch).
|
||||
- Sends an explicit product `User-Agent`, never a browser disguise.
|
||||
- Rejects unsupported (e.g. binary) content types with `WEB_UNSUPPORTED_CONTENT_TYPE`.
|
||||
|
||||
## Config
|
||||
|
||||
| Key | Default | Meaning |
|
||||
|---|---|---|
|
||||
| `maxUrlLength` | `2048` | Maximum accepted request URL length. |
|
||||
| `maxResponseBytes` | `5_000_000` | Maximum response body size in bytes. |
|
||||
| `maxBodyChars` | `100_000` | Maximum decoded body length in characters. |
|
||||
| `timeoutMs` | `30_000` | Default fetch timeout. |
|
||||
| `maxTimeoutMs` | `120_000` | Upper bound for a per-request timeout override. |
|
||||
| `maxRedirects` | `5` | Maximum same-origin redirect hops. |
|
||||
| `userAgent` | `deepseek-harness/…` | `User-Agent` header. |
|
||||
|
||||
## Security note
|
||||
|
||||
SSRF / private-network protection (blocking private, loopback, link-local, multicast, and otherwise non-public destinations, with DNS-resolve-then-validate and per-hop re-validation) is **deferred** — see the [web capability seam RFC](../../../docs/rfc/implemented/architecture/2026-06-24-web-capability-seam.md). Until it lands, this provider is an SSRF primitive and **must not be enabled** in a deployment that can reach sensitive internal network targets.
|
||||
33
packages/web/web-fetch-local/package.json
Normal file
33
packages/web/web-fetch-local/package.json
Normal file
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-web-fetch-local",
|
||||
"description": "Anonymous public HTTP(S) fetch provider for the DeepSeek Harness web capability seam (ctx.web)",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib",
|
||||
"src"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-web": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
},
|
||||
"dependencies": {
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-web": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
}
|
||||
}
|
||||
77
packages/web/web-fetch-local/src/index.ts
Normal file
77
packages/web/web-fetch-local/src/index.ts
Normal file
@@ -0,0 +1,77 @@
|
||||
/**
|
||||
* `@deepseek-ai/dsh-web-fetch-local`: registers an anonymous public HTTP(S)
|
||||
* `WebFetchProvider` with `ctx.web`. A function/namespace plugin (NOT a
|
||||
* default-export service): it registers INTO the seam's fetch registry, like the
|
||||
* search providers register into the search registry.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web-fetch-local
|
||||
*/
|
||||
|
||||
import type { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import type {} from '@deepseek-ai/dsh-web'
|
||||
import { LocalFetchProvider } from './provider.ts'
|
||||
import type { LocalFetchLimits } from './provider.ts'
|
||||
|
||||
export {
|
||||
LOCAL_FETCH_PROVIDER_ID,
|
||||
LocalFetchProvider,
|
||||
} from './provider.ts'
|
||||
export type { LocalFetchLimits } from './provider.ts'
|
||||
export { classifyContentType, isSameOrigin, validateFetchUrl } from './policy.ts'
|
||||
export type { FetchableKind } from './policy.ts'
|
||||
|
||||
/** Default `User-Agent`: an explicit product agent, never a browser disguise. */
|
||||
export const DEFAULT_USER_AGENT = 'deepseek-harness/0.0.1 (+https://github.com/deepseek-ai)'
|
||||
|
||||
/** Cordis plugin name used by loader diagnostics. */
|
||||
export const name = 'web-fetch-local'
|
||||
|
||||
/** The web seam this provider registers into. */
|
||||
export const inject = ['web']
|
||||
|
||||
export interface Config {
|
||||
/** Maximum accepted request URL length. */
|
||||
maxUrlLength?: number
|
||||
/** Maximum response body size in bytes. */
|
||||
maxResponseBytes?: number
|
||||
/** Maximum decoded body length in characters. */
|
||||
maxBodyChars?: number
|
||||
/** Default fetch timeout in milliseconds. */
|
||||
timeoutMs?: number
|
||||
/** Upper bound for a per-request timeout override. */
|
||||
maxTimeoutMs?: number
|
||||
/** Maximum number of same-origin redirect hops to follow. */
|
||||
maxRedirects?: number
|
||||
/** `User-Agent` header sent on every request. */
|
||||
userAgent?: string
|
||||
}
|
||||
|
||||
export const Config: z<Config> = z.object({
|
||||
maxUrlLength: z.number().default(2048),
|
||||
maxResponseBytes: z.number().default(5_000_000),
|
||||
maxBodyChars: z.number().default(100_000),
|
||||
timeoutMs: z.number().default(30_000),
|
||||
maxTimeoutMs: z.number().default(120_000),
|
||||
maxRedirects: z.number().default(5),
|
||||
userAgent: z.string().default(DEFAULT_USER_AGENT),
|
||||
})
|
||||
|
||||
/** The shape after schemastery applies its defaults to every field. */
|
||||
type ResolvedConfig = Required<Config>
|
||||
|
||||
/** Register the local HTTP(S) fetch provider with `ctx.web`. */
|
||||
export function apply(ctx: Context, config: Config): void {
|
||||
// schemastery (Config) has already filled every defaulted field.
|
||||
const resolved = config as ResolvedConfig
|
||||
const limits: LocalFetchLimits = {
|
||||
maxUrlLength: resolved.maxUrlLength,
|
||||
maxResponseBytes: resolved.maxResponseBytes,
|
||||
maxBodyChars: resolved.maxBodyChars,
|
||||
timeoutMs: resolved.timeoutMs,
|
||||
maxTimeoutMs: resolved.maxTimeoutMs,
|
||||
maxRedirects: resolved.maxRedirects,
|
||||
userAgent: resolved.userAgent,
|
||||
}
|
||||
ctx.web.registerFetchProvider(new LocalFetchProvider(limits))
|
||||
}
|
||||
59
packages/web/web-fetch-local/src/policy.ts
Normal file
59
packages/web/web-fetch-local/src/policy.ts
Normal file
@@ -0,0 +1,59 @@
|
||||
/**
|
||||
* URL validation and content-type classification for the local HTTP(S) fetch
|
||||
* provider — the pure, network-free half. The provider's `fetch()` composes
|
||||
* these with transport (redirect following, byte caps, decoding).
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web-fetch-local/policy
|
||||
*/
|
||||
|
||||
import { WebError } from '@deepseek-ai/dsh-web'
|
||||
|
||||
/** The body kinds this provider decodes. */
|
||||
export type FetchableKind = 'html' | 'text'
|
||||
|
||||
/**
|
||||
* Validate a request URL against the basic transport hygiene the provider
|
||||
* enforces before any network access: http(s) only, no embedded credentials,
|
||||
* bounded length. Returns the parsed `URL`. Throws {@link WebError} otherwise.
|
||||
* (SSRF / private-network blocking is deferred — see the package RFC.)
|
||||
*/
|
||||
export function validateFetchUrl(input: string, maxUrlLength: number): URL {
|
||||
if (input.length > maxUrlLength) {
|
||||
throw new WebError(`URL exceeds the maximum length of ${maxUrlLength}`, 'WEB_INVALID_URL')
|
||||
}
|
||||
let url: URL
|
||||
try {
|
||||
url = new URL(input)
|
||||
} catch (error: unknown) {
|
||||
throw new WebError(`invalid URL: ${input}`, 'WEB_INVALID_URL', { cause: error })
|
||||
}
|
||||
if (url.protocol !== 'http:' && url.protocol !== 'https:') {
|
||||
throw new WebError(`unsupported URL scheme "${url.protocol}" (only http and https are allowed)`, 'WEB_INVALID_URL')
|
||||
}
|
||||
if (url.username.length > 0 || url.password.length > 0) {
|
||||
throw new WebError('credentials in URLs are not allowed', 'WEB_BLOCKED_URL')
|
||||
}
|
||||
return url
|
||||
}
|
||||
|
||||
/**
|
||||
* Two URLs are same-origin when scheme, hostname, and port match. A redirect
|
||||
* that crosses origins is refused so each new origin requires a fresh tool call
|
||||
* (and thus a fresh provider/permission decision).
|
||||
*/
|
||||
export function isSameOrigin(a: URL, b: URL): boolean {
|
||||
return a.protocol === b.protocol && a.hostname === b.hostname && a.port === b.port
|
||||
}
|
||||
|
||||
/**
|
||||
* Classify a response `Content-Type` into a decodable body kind, or `undefined`
|
||||
* for an unsupported (e.g. binary) type. `text/html` and `application/xhtml+xml`
|
||||
* are `html`; other `text/*` plus a few structured text types are `text`.
|
||||
*/
|
||||
export function classifyContentType(contentType: string | null): FetchableKind | undefined {
|
||||
const mime = (contentType ?? '').replace(/;.*$/s, '').trim().toLowerCase()
|
||||
if (mime === 'text/html' || mime === 'application/xhtml+xml') return 'html'
|
||||
if (mime.startsWith('text/')) return 'text'
|
||||
if (mime === 'application/json' || mime === 'application/xml' || mime.endsWith('+json') || mime.endsWith('+xml')) return 'text'
|
||||
return undefined
|
||||
}
|
||||
233
packages/web/web-fetch-local/src/provider.ts
Normal file
233
packages/web/web-fetch-local/src/provider.ts
Normal file
@@ -0,0 +1,233 @@
|
||||
/**
|
||||
* `LocalFetchProvider`: a `WebFetchProvider` that retrieves a concrete public
|
||||
* HTTP(S) URL with the platform-native `fetch` (Node 24) and returns a status
|
||||
* code plus bounded decoded content. It owns SAFE RESOURCE RETRIEVAL — URL
|
||||
* validation, redirect policy, timeout, abort, byte caps, charset decoding,
|
||||
* content-type classification, binary rejection — but NOT presentation
|
||||
* (HTML→markdown lives in `@deepseek-ai/dsh-tool-web`).
|
||||
*
|
||||
* Redirects are followed manually (`redirect: 'manual'`) so the provider can
|
||||
* enforce a same-origin-only policy: a cross-origin redirect is refused with
|
||||
* `WEB_REDIRECT_BLOCKED`, requiring a fresh tool call (Claude Code's WebFetch
|
||||
* uses the same model). It does NOT carry browser cookies, editor/git
|
||||
* credentials, or implicit access to private services.
|
||||
*
|
||||
* SSRF / private-network protection is DEFERRED (see the package RFC); until it
|
||||
* lands this provider is an SSRF primitive and must not be enabled where it can
|
||||
* reach sensitive internal targets.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web-fetch-local/provider
|
||||
*/
|
||||
|
||||
import { WebError } from '@deepseek-ai/dsh-web'
|
||||
import type { WebFetchBody, WebFetchProvider, WebFetchRequest, WebFetchResult, WebProviderStatus } from '@deepseek-ai/dsh-web'
|
||||
import { classifyContentType, isSameOrigin, validateFetchUrl } from './policy.ts'
|
||||
|
||||
/** Resolved provider limits (the plugin's schemastery Config supplies defaults). */
|
||||
export interface LocalFetchLimits {
|
||||
/** Maximum accepted request URL length. */
|
||||
maxUrlLength: number
|
||||
/** Maximum response body size in bytes (read is aborted past this). */
|
||||
maxResponseBytes: number
|
||||
/** Maximum decoded body length in characters (truncated past this). */
|
||||
maxBodyChars: number
|
||||
/** Default fetch timeout in milliseconds. */
|
||||
timeoutMs: number
|
||||
/** Upper bound for a per-request timeout override. */
|
||||
maxTimeoutMs: number
|
||||
/** Maximum number of (same-origin) redirect hops to follow. */
|
||||
maxRedirects: number
|
||||
/** `User-Agent` header sent on every request. */
|
||||
userAgent: string
|
||||
}
|
||||
|
||||
/** Stable id this provider registers under. */
|
||||
export const LOCAL_FETCH_PROVIDER_ID = 'local-http'
|
||||
|
||||
/** The anonymous public HTTP(S) fetch provider. */
|
||||
export class LocalFetchProvider implements WebFetchProvider {
|
||||
readonly id = LOCAL_FETCH_PROVIDER_ID
|
||||
|
||||
constructor(private readonly limits: LocalFetchLimits) {}
|
||||
|
||||
/** No credentials to check — an anonymous public fetcher is always usable. */
|
||||
status(): WebProviderStatus {
|
||||
return { available: true }
|
||||
}
|
||||
|
||||
async fetch(request: WebFetchRequest, exec?: { readonly signal?: AbortSignal }): Promise<WebFetchResult> {
|
||||
const timeoutMs = request.timeoutMs !== undefined
|
||||
? Math.min(request.timeoutMs, this.limits.maxTimeoutMs)
|
||||
: this.limits.timeoutMs
|
||||
|
||||
// One controller drives both the caller's abort and our own timeout, so the
|
||||
// network request and the streaming read both stop on either.
|
||||
const controller = new AbortController()
|
||||
const onAbort = (): void => { controller.abort() }
|
||||
if (exec?.signal !== undefined) {
|
||||
if (exec.signal.aborted) throw new WebError('web fetch aborted', 'WEB_ABORTED')
|
||||
exec.signal.addEventListener('abort', onAbort, { once: true })
|
||||
}
|
||||
const timer = setTimeout(() => { controller.abort(new WebError('web fetch timed out', 'WEB_FETCH_TIMEOUT')) }, timeoutMs)
|
||||
|
||||
try {
|
||||
return await this.followAndRead(request.url, controller, timeoutMs)
|
||||
} finally {
|
||||
clearTimeout(timer)
|
||||
if (exec?.signal !== undefined) exec.signal.removeEventListener('abort', onAbort)
|
||||
}
|
||||
}
|
||||
|
||||
/** Follow same-origin redirects up to the hop cap, then read the final response. */
|
||||
private async followAndRead(initialUrl: string, controller: AbortController, timeoutMs: number): Promise<WebFetchResult> {
|
||||
let currentUrl = validateFetchUrl(initialUrl, this.limits.maxUrlLength)
|
||||
|
||||
for (let hop = 0; hop <= this.limits.maxRedirects; hop++) {
|
||||
const response = await this.requestOnce(currentUrl, controller, timeoutMs)
|
||||
|
||||
if (isRedirectStatus(response.status)) {
|
||||
const location = response.headers.get('location')
|
||||
if (location === null) {
|
||||
// A redirect status with no Location is not a usable resource.
|
||||
throw new WebError(`redirect response (HTTP ${response.status}) without a Location header`, 'WEB_PROVIDER_ERROR')
|
||||
}
|
||||
const target = resolveRedirect(location, currentUrl)
|
||||
if (!isSameOrigin(target, currentUrl)) {
|
||||
throw new WebError(
|
||||
`cross-origin redirect to ${target.origin} is not followed automatically; retry against that URL directly`,
|
||||
'WEB_REDIRECT_BLOCKED',
|
||||
)
|
||||
}
|
||||
await response.body?.cancel()
|
||||
currentUrl = target
|
||||
continue
|
||||
}
|
||||
|
||||
return await this.readBody(response, currentUrl)
|
||||
}
|
||||
|
||||
throw new WebError(`exceeded the maximum of ${this.limits.maxRedirects} redirects`, 'WEB_REDIRECT_BLOCKED')
|
||||
}
|
||||
|
||||
private async requestOnce(url: URL, controller: AbortController, _timeoutMs: number): Promise<Response> {
|
||||
try {
|
||||
return await fetch(url, {
|
||||
method: 'GET',
|
||||
redirect: 'manual',
|
||||
headers: { 'user-agent': this.limits.userAgent, 'accept': 'text/html,application/xhtml+xml,text/*;q=0.9,application/json;q=0.8' },
|
||||
signal: controller.signal,
|
||||
})
|
||||
} catch (error: unknown) {
|
||||
throw translateAbortOrNetwork(error)
|
||||
}
|
||||
}
|
||||
|
||||
/** Read, byte-cap, classify, and decode the final response body. */
|
||||
private async readBody(response: Response, finalUrl: URL): Promise<WebFetchResult> {
|
||||
const kind = classifyContentType(response.headers.get('content-type'))
|
||||
if (kind === undefined) {
|
||||
await response.body?.cancel()
|
||||
throw new WebError(`unsupported content type "${response.headers.get('content-type') ?? 'unknown'}"`, 'WEB_UNSUPPORTED_CONTENT_TYPE')
|
||||
}
|
||||
|
||||
const { bytes, truncatedByBytes } = await this.readCapped(response)
|
||||
const decoded = new TextDecoder('utf-8').decode(bytes)
|
||||
const truncatedByChars = decoded.length > this.limits.maxBodyChars
|
||||
const content = truncatedByChars ? decoded.slice(0, this.limits.maxBodyChars) : decoded
|
||||
const body: WebFetchBody = kind === 'html' ? { kind: 'html', content } : { kind: 'text', content }
|
||||
|
||||
return {
|
||||
providerId: this.id,
|
||||
url: finalUrl.toString(),
|
||||
statusCode: response.status,
|
||||
body,
|
||||
truncated: truncatedByBytes || truncatedByChars,
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the response stream up to `maxResponseBytes`. A `Content-Length` over
|
||||
* the cap rejects immediately with `WEB_FETCH_TOO_LARGE`; a stream that grows
|
||||
* past the cap is cut short (`truncatedByBytes`) rather than rejected, so a
|
||||
* server that under-reports still yields a bounded usable body.
|
||||
*/
|
||||
private async readCapped(response: Response): Promise<{ bytes: Uint8Array; truncatedByBytes: boolean }> {
|
||||
const declared = response.headers.get('content-length')
|
||||
if (declared !== null) {
|
||||
const length = Number(declared)
|
||||
if (Number.isFinite(length) && length > this.limits.maxResponseBytes) {
|
||||
await response.body?.cancel()
|
||||
throw new WebError(`response exceeds the maximum of ${this.limits.maxResponseBytes} bytes`, 'WEB_FETCH_TOO_LARGE')
|
||||
}
|
||||
}
|
||||
|
||||
/* v8 ignore next -- a 2xx Response from fetch always exposes a body stream; the null guard is defensive. */
|
||||
if (response.body === null) return { bytes: new Uint8Array(0), truncatedByBytes: false }
|
||||
|
||||
const chunks: Uint8Array[] = []
|
||||
let total = 0
|
||||
let truncatedByBytes = false
|
||||
const reader = response.body.getReader()
|
||||
try {
|
||||
for (;;) {
|
||||
const { done, value } = await reader.read()
|
||||
if (done) break
|
||||
const remaining = this.limits.maxResponseBytes - total
|
||||
if (value.byteLength >= remaining) {
|
||||
chunks.push(value.subarray(0, remaining))
|
||||
total += remaining
|
||||
truncatedByBytes = true
|
||||
break
|
||||
}
|
||||
chunks.push(value)
|
||||
total += value.byteLength
|
||||
}
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore next -- mid-stream read fault needs a network drop after headers; translate path covered by request-phase tests. */
|
||||
throw translateAbortOrNetwork(error)
|
||||
} finally {
|
||||
/* v8 ignore next 4 -- cancel() after a completed/broken read settles without rejecting; unobserved best-effort cleanup. */
|
||||
await reader.cancel().catch(() => {
|
||||
// Cancel after a successful read (or after we broke past the cap) is
|
||||
// best-effort cleanup; the bytes we need are already collected.
|
||||
})
|
||||
}
|
||||
|
||||
const bytes = new Uint8Array(total)
|
||||
let offset = 0
|
||||
for (const chunk of chunks) {
|
||||
bytes.set(chunk, offset)
|
||||
offset += chunk.byteLength
|
||||
}
|
||||
return { bytes, truncatedByBytes }
|
||||
}
|
||||
}
|
||||
|
||||
/** HTTP redirect status codes that carry a `Location`. */
|
||||
function isRedirectStatus(status: number): boolean {
|
||||
return status === 301 || status === 302 || status === 303 || status === 307 || status === 308
|
||||
}
|
||||
|
||||
/** Resolve a (possibly relative) `Location` against the current URL. */
|
||||
function resolveRedirect(location: string, base: URL): URL {
|
||||
try {
|
||||
return new URL(location, base)
|
||||
} catch (error: unknown) {
|
||||
/* v8 ignore next 2 -- URL resolution against a valid absolute base effectively never throws; defensive guard. */
|
||||
throw new WebError(`invalid redirect Location "${location}"`, 'WEB_PROVIDER_ERROR', { cause: error })
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Translate a thrown fetch/stream error into a `WebError`. Our own
|
||||
* `WEB_FETCH_TIMEOUT` (passed to `controller.abort(reason)`) and any other
|
||||
* already-typed `WebError` pass through; an `AbortError` becomes `WEB_ABORTED`;
|
||||
* anything else is a transport/network failure (`WEB_PROVIDER_ERROR`).
|
||||
*/
|
||||
function translateAbortOrNetwork(error: unknown): WebError {
|
||||
if (error instanceof WebError) return error
|
||||
if (error instanceof DOMException && error.name === 'AbortError') {
|
||||
return new WebError('web fetch aborted', 'WEB_ABORTED', { cause: error })
|
||||
}
|
||||
return new WebError(`web fetch failed: ${String(error)}`, 'WEB_PROVIDER_ERROR', { cause: error })
|
||||
}
|
||||
239
packages/web/web-fetch-local/tests/fetch-local.spec.ts
Normal file
239
packages/web/web-fetch-local/tests/fetch-local.spec.ts
Normal file
@@ -0,0 +1,239 @@
|
||||
import { afterEach, beforeEach, describe, expect, it } from 'vitest'
|
||||
import { createServer, type IncomingMessage, type Server, type ServerResponse } from 'node:http'
|
||||
import { AddressInfo } from 'node:net'
|
||||
import { Context } from 'cordis'
|
||||
import WebService from '@deepseek-ai/dsh-web'
|
||||
import { LocalFetchProvider, LOCAL_FETCH_PROVIDER_ID, classifyContentType, isSameOrigin, validateFetchUrl } from '@deepseek-ai/dsh-web-fetch-local'
|
||||
import type { LocalFetchLimits } from '@deepseek-ai/dsh-web-fetch-local'
|
||||
import * as fetchPlugin from '@deepseek-ai/dsh-web-fetch-local'
|
||||
|
||||
const limits: LocalFetchLimits = {
|
||||
maxUrlLength: 2048,
|
||||
maxResponseBytes: 5_000_000,
|
||||
maxBodyChars: 100_000,
|
||||
timeoutMs: 5_000,
|
||||
maxTimeoutMs: 10_000,
|
||||
maxRedirects: 5,
|
||||
userAgent: 'test-agent/1.0',
|
||||
}
|
||||
|
||||
type Handler = (req: IncomingMessage, res: ServerResponse) => void
|
||||
|
||||
let server: Server
|
||||
let base: string
|
||||
let handler: Handler
|
||||
|
||||
beforeEach(async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('default') }
|
||||
server = createServer((req, res) => { handler(req, res) })
|
||||
await new Promise<void>(resolve => server.listen(0, '127.0.0.1', resolve))
|
||||
const { port } = server.address() as AddressInfo
|
||||
base = `http://127.0.0.1:${port}`
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
await new Promise<void>(resolve => server.close(() => { resolve() }))
|
||||
})
|
||||
|
||||
function provider(overrides: Partial<LocalFetchLimits> = {}): LocalFetchProvider {
|
||||
return new LocalFetchProvider({ ...limits, ...overrides })
|
||||
}
|
||||
|
||||
describe('policy helpers', () => {
|
||||
it('validates scheme, credentials, and length', () => {
|
||||
expect(validateFetchUrl('https://example.com/x', 2048).hostname).toBe('example.com')
|
||||
expect(() => validateFetchUrl('ftp://example.com', 2048)).toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' }))
|
||||
expect(() => validateFetchUrl('not a url', 2048)).toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' }))
|
||||
expect(() => validateFetchUrl('https://user:pass@example.com', 2048)).toThrow(expect.objectContaining({ code: 'WEB_BLOCKED_URL' }))
|
||||
expect(() => validateFetchUrl(`https://example.com/${'a'.repeat(3000)}`, 2048)).toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' }))
|
||||
})
|
||||
|
||||
it('classifies content types', () => {
|
||||
expect(classifyContentType('text/html; charset=utf-8')).toBe('html')
|
||||
expect(classifyContentType('application/xhtml+xml')).toBe('html')
|
||||
expect(classifyContentType('text/plain')).toBe('text')
|
||||
expect(classifyContentType('application/json')).toBe('text')
|
||||
expect(classifyContentType('image/png')).toBeUndefined()
|
||||
expect(classifyContentType(null)).toBeUndefined()
|
||||
})
|
||||
|
||||
it('compares origins', () => {
|
||||
expect(isSameOrigin(new URL('https://a.com/x'), new URL('https://a.com/y'))).toBe(true)
|
||||
expect(isSameOrigin(new URL('https://a.com'), new URL('https://b.com'))).toBe(false)
|
||||
expect(isSameOrigin(new URL('http://a.com'), new URL('https://a.com'))).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('LocalFetchProvider success', () => {
|
||||
it('fetches a text body', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('hello world') }
|
||||
const result = await provider().fetch({ url: base })
|
||||
expect(result.providerId).toBe(LOCAL_FETCH_PROVIDER_ID)
|
||||
expect(result.statusCode).toBe(200)
|
||||
expect(result.body).toEqual({ kind: 'text', content: 'hello world' })
|
||||
expect(result.truncated).toBe(false)
|
||||
})
|
||||
|
||||
it('fetches an html body and classifies it as html', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/html' }); res.end('<h1>hi</h1>') }
|
||||
const result = await provider().fetch({ url: base })
|
||||
expect(result.body).toEqual({ kind: 'html', content: '<h1>hi</h1>' })
|
||||
})
|
||||
|
||||
it('sends the configured user agent', async () => {
|
||||
let seen: string | undefined
|
||||
handler = (req, res) => { seen = req.headers['user-agent']; res.writeHead(200, { 'content-type': 'text/plain' }); res.end('ok') }
|
||||
await provider().fetch({ url: base })
|
||||
expect(seen).toBe('test-agent/1.0')
|
||||
})
|
||||
|
||||
it('returns a non-2xx response as a result, not an error', async () => {
|
||||
handler = (_req, res) => { res.writeHead(404, { 'content-type': 'text/plain' }); res.end('nope') }
|
||||
const result = await provider().fetch({ url: base })
|
||||
expect(result.statusCode).toBe(404)
|
||||
expect(result.body).toEqual({ kind: 'text', content: 'nope' })
|
||||
})
|
||||
})
|
||||
|
||||
describe('LocalFetchProvider caps', () => {
|
||||
it('rejects an over-cap Content-Length with WEB_FETCH_TOO_LARGE', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain', 'content-length': '999999' }); res.end('x'.repeat(999999)) }
|
||||
await expect(provider({ maxResponseBytes: 10 }).fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_FETCH_TOO_LARGE' }))
|
||||
})
|
||||
|
||||
it('truncates a stream that grows past the byte cap', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('abcdefghij') }
|
||||
const result = await provider({ maxResponseBytes: 4 }).fetch({ url: base })
|
||||
expect(result.body.content).toBe('abcd')
|
||||
expect(result.truncated).toBe(true)
|
||||
})
|
||||
|
||||
it('truncates a decoded body past the character cap', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('abcdefghij') }
|
||||
const result = await provider({ maxBodyChars: 3 }).fetch({ url: base })
|
||||
expect(result.body.content).toBe('abc')
|
||||
expect(result.truncated).toBe(true)
|
||||
})
|
||||
|
||||
it('rejects an unsupported content type', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'image/png' }); res.end('binary') }
|
||||
await expect(provider().fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' }))
|
||||
})
|
||||
|
||||
it('rejects a response with no content type at all', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200); res.end('no type') }
|
||||
await expect(provider().fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_UNSUPPORTED_CONTENT_TYPE' }))
|
||||
})
|
||||
|
||||
it('accepts a declared content-length within the cap', async () => {
|
||||
handler = (_req, res) => { const body = 'sized'; res.writeHead(200, { 'content-type': 'text/plain', 'content-length': String(body.length) }); res.end(body) }
|
||||
const result = await provider().fetch({ url: base })
|
||||
expect(result.body.content).toBe('sized')
|
||||
})
|
||||
})
|
||||
|
||||
describe('LocalFetchProvider redirects', () => {
|
||||
it('follows a same-origin redirect and reports the final URL', async () => {
|
||||
handler = (req, res) => {
|
||||
if (req.url === '/start') { res.writeHead(302, { location: '/end' }); res.end() }
|
||||
else { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('arrived') }
|
||||
}
|
||||
const result = await provider().fetch({ url: `${base}/start` })
|
||||
expect(result.body.content).toBe('arrived')
|
||||
expect(result.url).toBe(`${base}/end`)
|
||||
})
|
||||
|
||||
it('blocks a cross-origin redirect with WEB_REDIRECT_BLOCKED', async () => {
|
||||
handler = (_req, res) => { res.writeHead(302, { location: 'https://example.com/' }); res.end() }
|
||||
await expect(provider().fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_REDIRECT_BLOCKED' }))
|
||||
})
|
||||
|
||||
it('rejects exceeding the redirect hop cap', async () => {
|
||||
handler = (req, res) => {
|
||||
const n = Number(new URL(req.url ?? '/', base).searchParams.get('n') ?? '0')
|
||||
res.writeHead(302, { location: `/?n=${n + 1}` })
|
||||
res.end()
|
||||
}
|
||||
await expect(provider({ maxRedirects: 2 }).fetch({ url: `${base}/?n=0` }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_REDIRECT_BLOCKED' }))
|
||||
})
|
||||
|
||||
it('treats a redirect without a Location header as a provider error', async () => {
|
||||
handler = (_req, res) => { res.writeHead(302); res.end() }
|
||||
await expect(provider().fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' }))
|
||||
})
|
||||
|
||||
it('follows a relative same-origin redirect', async () => {
|
||||
handler = (req, res) => {
|
||||
if (req.url === '/a') { res.writeHead(301, { location: 'b' }); res.end() }
|
||||
else { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('landed') }
|
||||
}
|
||||
const result = await provider().fetch({ url: `${base}/a` })
|
||||
expect(result.body.content).toBe('landed')
|
||||
})
|
||||
})
|
||||
|
||||
describe('LocalFetchProvider invalid URLs and abort', () => {
|
||||
it('rejects a non-http scheme before any network access', async () => {
|
||||
await expect(provider().fetch({ url: 'ftp://example.com' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_INVALID_URL' }))
|
||||
})
|
||||
|
||||
it('rejects credentials in the URL', async () => {
|
||||
await expect(provider().fetch({ url: 'http://user:pass@127.0.0.1/' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_BLOCKED_URL' }))
|
||||
})
|
||||
|
||||
it('honors a pre-aborted signal', async () => {
|
||||
const controller = new AbortController()
|
||||
controller.abort()
|
||||
await expect(provider().fetch({ url: base }, { signal: controller.signal }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_ABORTED' }))
|
||||
})
|
||||
|
||||
it('aborts an in-flight fetch via the signal', async () => {
|
||||
handler = (_req, _res) => { /* never responds */ }
|
||||
const controller = new AbortController()
|
||||
const promise = provider().fetch({ url: base }, { signal: controller.signal })
|
||||
controller.abort()
|
||||
await expect(promise).rejects.toThrow(expect.objectContaining({ code: 'WEB_ABORTED' }))
|
||||
})
|
||||
|
||||
it('times out a slow response with WEB_FETCH_TIMEOUT', async () => {
|
||||
handler = (_req, _res) => { /* never responds */ }
|
||||
await expect(provider({ timeoutMs: 50 }).fetch({ url: base }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_FETCH_TIMEOUT' }))
|
||||
})
|
||||
|
||||
it('maps a connection failure to WEB_PROVIDER_ERROR', async () => {
|
||||
// Port 1 on loopback is not listening: a real connection failure (not abort).
|
||||
await expect(provider().fetch({ url: 'http://127.0.0.1:1/' }))
|
||||
.rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_ERROR' }))
|
||||
})
|
||||
|
||||
it('caps the per-request timeout at maxTimeoutMs', async () => {
|
||||
handler = (_req, res) => { res.writeHead(200, { 'content-type': 'text/plain' }); res.end('ok') }
|
||||
const result = await provider({ maxTimeoutMs: 10_000 }).fetch({ url: base, timeoutMs: 999_999 })
|
||||
expect(result.statusCode).toBe(200)
|
||||
})
|
||||
})
|
||||
|
||||
describe('web-fetch-local plugin registration', () => {
|
||||
it('registers the provider into ctx.web (HMR-safe)', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebService, { fetchProvider: LOCAL_FETCH_PROVIDER_ID })
|
||||
const fiber = await ctx.plugin(fetchPlugin, {})
|
||||
expect(ctx.web.fetchStatus()).toEqual({ available: true, providerId: LOCAL_FETCH_PROVIDER_ID })
|
||||
await fiber.dispose()
|
||||
expect(ctx.web.fetchStatus()).toEqual({ available: false, reason: 'configured-missing' })
|
||||
})
|
||||
|
||||
it('has no default export (namespace plugin export shape)', () => {
|
||||
expect('default' in fetchPlugin).toBe(false)
|
||||
})
|
||||
})
|
||||
24
packages/web/web-fetch-local/tsconfig.json
Normal file
24
packages/web/web-fetch-local/tsconfig.json
Normal file
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../web"
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user