Add web capability seam: ctx.web, search/fetch providers, web tools
Introduce web access as a first-class capability seam so the model-facing web tools stay stable while backends change. dsh-web owns ctx.web as a provider registry with registration-order-independent selection and the WebError taxonomy; dsh-web-search-exa, dsh-web-search-perplexity, and dsh-web-fetch-local register capabilities into it; dsh-tool-web is the sole owner of the model-facing web_search/web_fetch schemas, prompt sections, and HTML-to-markdown presentation. Search and fetch are deliberately one seam. Providers ship as namespace plugins that register into ctx.web (like an LlmAdapter into ctx.llm), not key-owning services, since multiple search providers cannot each own the key. Tool registration follows product enablement, not backend availability, so load order/credentials never enter the model contract; the seam resolves the provider at execution time and surfaces a structured WebError otherwise. Moves the RFC to implemented/ amended to match what shipped. Example/app configs are intentionally not wired yet (RFC migration step 6).
This commit is contained in:
Dudu-0223
45
packages/web/web/README.md
Normal file
45
packages/web/web/README.md
Normal file
@@ -0,0 +1,45 @@
|
||||
# @deepseek-ai/dsh-web
|
||||
|
||||
The **web access seam**: an abstract `WebService` (`ctx.web`) defining WHAT web access the harness has — search the web, fetch a URL — over multiple providers, without binding the model contract to one vendor's API shape.
|
||||
|
||||
This package is the interface third of the web capability. Unlike bash/fs it spans two capabilities (search and fetch) on one seam, with potentially multiple providers each:
|
||||
|
||||
| Package | Role |
|
||||
|---|---|
|
||||
| `@deepseek-ai/dsh-web` (this) | the interface: the service, provider registries, selection policy, request/result vocabulary, the `WebError` taxonomy |
|
||||
| `@deepseek-ai/dsh-web-search-exa` | a search implementation: Exa |
|
||||
| `@deepseek-ai/dsh-web-search-perplexity` | a search implementation: Perplexity |
|
||||
| `@deepseek-ai/dsh-web-fetch-local` | a fetch implementation: anonymous public HTTP(S) |
|
||||
| `@deepseek-ai/dsh-tool-web` | the model-facing `web_search` / `web_fetch` tool schemas over `ctx.web` |
|
||||
|
||||
Search and fetch share no request schema and no business logic, but they are deliberately one seam: `ctx.web` is a single web-access middle layer with one provider-selection policy owner, one abort/error vocabulary, and one product-facing "how this harness reaches the web" config surface. The cost is the parallel `Search`/`Fetch` method pairs; that parallelism is intentional, not a missed extraction.
|
||||
|
||||
## Service API (`ctx.web`)
|
||||
|
||||
| Member | Semantics |
|
||||
|---|---|
|
||||
| `registerSearchProvider(provider)` / `registerFetchProvider(provider)` | Register a backend. Throws `WebError` `WEB_DUPLICATE_PROVIDER` on a duplicate id within that capability kind. Returns a disposer; emits `web/providers-change` on register and on dispose. Disposed with the calling fiber. |
|
||||
| `searchStatus()` / `fetchStatus()` | Derived (never stored) `WebCapabilityStatus`: whether the capability has a selected usable provider, or the broad category it fails in. Diagnostics + execution-resolution input. |
|
||||
| `search(request, exec?)` | Resolve the search provider and run one search. Enforces `request.maxResults` on the result (truncates `sources[]`, sets `truncated`). Throws `WebError` when the capability cannot run. |
|
||||
| `fetch(request, exec?)` | Resolve the fetch provider and retrieve one URL. A non-2xx response is a result, not a throw. Throws `WebError` for failures to safely retrieve or represent the resource. |
|
||||
|
||||
Providers register **capabilities**, not tools. `dsh-tool-web` is the only owner of model-facing names, descriptions, prompt guidance, JSON schemas, and presentation.
|
||||
|
||||
## Selection
|
||||
|
||||
Selection never depends on registration, config, or HMR order. A capability has an explicit provider id (config `searchProvider`/`fetchProvider`, or env `$DSH_WEB_SEARCH_PROVIDER`/`$DSH_WEB_FETCH_PROVIDER` feeding the same fields), or auto-selects when exactly one usable provider is registered:
|
||||
|
||||
| Situation | `WebCapabilityStatus` | Execution |
|
||||
|---|---|---|
|
||||
| configured id registered and `status().available` | `available` for it | runs |
|
||||
| configured id not registered | `configured-missing` | `WEB_PROVIDER_CONFIGURED_MISSING` |
|
||||
| configured id registered but unavailable | `configured-unavailable` | `WEB_PROVIDER_CONFIGURED_UNAVAILABLE` |
|
||||
| no id, exactly one registered usable provider | `available` for it | runs |
|
||||
| no id, no usable provider | `none` | `WEB_PROVIDER_UNAVAILABLE` |
|
||||
| no id, multiple usable providers | `ambiguous` | `WEB_PROVIDER_AMBIGUOUS` |
|
||||
|
||||
`WebCapabilityStatus` carries only `available` + a `reason` discriminant (plus the winning `providerId` on the available branch). The branchable per-reason detail lives in the thrown `WebError`, which is the surface callers route on — so the same fact never gets two homes that can disagree. A provider's own `status()` is a cheap local check (credential presence, parseable config) and **must not make network calls**; `dsh-tool-web` reads only the aggregated `searchStatus()`/`fetchStatus()`, never each provider's `status()` directly.
|
||||
|
||||
## Vocabulary
|
||||
|
||||
`WebSearchRequest` (`query`, `maxResults?`) → `WebSearchResult` (`providerId`, `query`, `content?`, `sources[]`, `truncated`); each `WebSearchSource` has a required `url` and optional `title`/`snippet`/`publishedAt` (Perplexity citations may be URL-only). `WebFetchRequest` (`url`, `timeoutMs?`) → `WebFetchResult` (`providerId`, final `url`, `statusCode`, `body`, `truncated`); `WebFetchBody` is a CLOSED discriminated union (`html` | `text`) owned here — consumers `switch` to exhaustiveness so a new kind breaks their compilation until handled. See `src/types.ts` for the full contracts and the `WebError` code taxonomy.
|
||||
33
packages/web/web/package.json
Normal file
33
packages/web/web/package.json
Normal file
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-web",
|
||||
"description": "Abstract web access capability seam (ctx.web) for the DeepSeek Harness — search/fetch provider registry, registration-order-independent selection, request/result vocabulary, and the WebError taxonomy",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/index.d.ts",
|
||||
"exports": {
|
||||
".": {
|
||||
"types": "./lib/index.d.ts",
|
||||
"default": "./lib/index.js"
|
||||
},
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": [
|
||||
"lib",
|
||||
"src"
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-llm": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
},
|
||||
"dependencies": {
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
}
|
||||
}
|
||||
270
packages/web/web/src/index.ts
Normal file
270
packages/web/web/src/index.ts
Normal file
@@ -0,0 +1,270 @@
|
||||
/**
|
||||
* The web access seam (`ctx.web`): a provider registry plus a provider-selecting
|
||||
* execution surface for two capabilities — search and fetch. Provider packages
|
||||
* register concrete backends with `registerSearchProvider` /
|
||||
* `registerFetchProvider`; the model-facing consumer
|
||||
* (`@deepseek-ai/dsh-tool-web`) reads capability status and executes through
|
||||
* `search()` / `fetch()`.
|
||||
*
|
||||
* The registry half stays close to `LlmService`: a `Map<id, provider>` per
|
||||
* capability kind, register methods that return disposers, duplicate ids that
|
||||
* throw, and execution-time resolution that throws when the selected provider is
|
||||
* absent or unusable. On top of that sits one small selection-status layer so
|
||||
* diagnostics and execution can explain why a capability can or cannot run,
|
||||
* independent of registration order.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web
|
||||
*/
|
||||
|
||||
import { Context, Service } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import type {
|
||||
WebCapabilityStatus,
|
||||
WebExecContext,
|
||||
WebFetchProvider,
|
||||
WebFetchRequest,
|
||||
WebFetchResult,
|
||||
WebProviderStatus,
|
||||
WebSearchProvider,
|
||||
WebSearchRequest,
|
||||
WebSearchResult,
|
||||
} from './types.ts'
|
||||
import { WebError } from './types.ts'
|
||||
|
||||
export {
|
||||
WebError,
|
||||
} from './types.ts'
|
||||
export type {
|
||||
WebCapabilityStatus,
|
||||
WebErrorCode,
|
||||
WebExecContext,
|
||||
WebFetchBody,
|
||||
WebFetchProvider,
|
||||
WebFetchRequest,
|
||||
WebFetchResult,
|
||||
WebProviderStatus,
|
||||
WebSearchProvider,
|
||||
WebSearchRequest,
|
||||
WebSearchResult,
|
||||
WebSearchSource,
|
||||
} from './types.ts'
|
||||
|
||||
declare module 'cordis' {
|
||||
interface Context {
|
||||
web: WebService
|
||||
}
|
||||
|
||||
interface Events {
|
||||
/**
|
||||
* Fired after the provider registry changes — a search or fetch provider was
|
||||
* registered or disposed. Carries no payload and no capability graph: it
|
||||
* means only "the provider registry changed; observers may recompute status
|
||||
* from `ctx.web`". `searchStatus()` / `fetchStatus()` stay derived, not
|
||||
* stored.
|
||||
* @mode emit
|
||||
*/
|
||||
'web/providers-change'(this: WebService): void
|
||||
}
|
||||
}
|
||||
|
||||
/** Selection inputs shared by the status query and execution resolution. */
|
||||
interface Selection<P> {
|
||||
/** The configured provider id for this capability, if any. */
|
||||
readonly configuredId?: string
|
||||
/** Providers registered for this capability kind. */
|
||||
readonly providers: ReadonlyMap<string, P>
|
||||
}
|
||||
|
||||
/**
|
||||
* Config for the web seam. `searchProvider` / `fetchProvider` pin which provider
|
||||
* wins for each capability; both are optional (a single registered usable
|
||||
* provider auto-selects). Operational overrides such as environment variables
|
||||
* must feed these same fields rather than introduce a hidden priority chain.
|
||||
*/
|
||||
export interface WebServiceConfig {
|
||||
/** Explicit search provider id. Omitted = auto-select when exactly one usable. */
|
||||
readonly searchProvider?: string
|
||||
/** Explicit fetch provider id. Omitted = auto-select when exactly one usable. */
|
||||
readonly fetchProvider?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* The web access service. Registered as `ctx.web` (one instance per context).
|
||||
*
|
||||
* Selection semantics (identical for status and execution, never order-
|
||||
* dependent):
|
||||
* - A configured id that is registered and `status().available` → that provider.
|
||||
* - A configured id not registered → `configured-missing` /
|
||||
* `WEB_PROVIDER_CONFIGURED_MISSING`.
|
||||
* - A configured id registered but unavailable → `configured-unavailable` /
|
||||
* `WEB_PROVIDER_CONFIGURED_UNAVAILABLE`.
|
||||
* - No id configured, exactly one registered usable provider → that provider.
|
||||
* - No id configured, multiple usable providers → `ambiguous` /
|
||||
* `WEB_PROVIDER_AMBIGUOUS`.
|
||||
* - No id configured, no usable provider → `none` / `WEB_PROVIDER_UNAVAILABLE`.
|
||||
*/
|
||||
export class WebService extends Service {
|
||||
/**
|
||||
* Provider selection config. Operational env overrides feed the SAME fields:
|
||||
* `$DSH_WEB_SEARCH_PROVIDER` / `$DSH_WEB_FETCH_PROVIDER` are equivalent to
|
||||
* `searchProvider` / `fetchProvider` and are NOT a hidden priority chain.
|
||||
*/
|
||||
static Config: z<WebServiceConfig> = z.object({
|
||||
searchProvider: z.string(),
|
||||
fetchProvider: z.string(),
|
||||
})
|
||||
|
||||
private searchProviders = new Map<string, WebSearchProvider>()
|
||||
private fetchProviders = new Map<string, WebFetchProvider>()
|
||||
private readonly searchProviderId: string | undefined
|
||||
private readonly fetchProviderId: string | undefined
|
||||
|
||||
constructor(ctx: Context, config: WebServiceConfig = {}) {
|
||||
super(ctx, 'web')
|
||||
this.searchProviderId = config.searchProvider ?? process.env.DSH_WEB_SEARCH_PROVIDER
|
||||
this.fetchProviderId = config.fetchProvider ?? process.env.DSH_WEB_FETCH_PROVIDER
|
||||
}
|
||||
|
||||
/**
|
||||
* Register a search provider. Throws {@link WebError} `WEB_DUPLICATE_PROVIDER`
|
||||
* if its id is already registered for search. Returns a disposer; emits
|
||||
* `web/providers-change` after a successful register and again on dispose.
|
||||
* Disposed with the calling fiber.
|
||||
*/
|
||||
registerSearchProvider(provider: WebSearchProvider): () => void {
|
||||
return this.registerProvider(this.searchProviders, provider)
|
||||
}
|
||||
|
||||
/**
|
||||
* Register a fetch provider. Throws {@link WebError} `WEB_DUPLICATE_PROVIDER`
|
||||
* if its id is already registered for fetch. Returns a disposer; emits
|
||||
* `web/providers-change` after a successful register and again on dispose.
|
||||
* Disposed with the calling fiber.
|
||||
*/
|
||||
registerFetchProvider(provider: WebFetchProvider): () => void {
|
||||
return this.registerProvider(this.fetchProviders, provider)
|
||||
}
|
||||
|
||||
private registerProvider<P extends { readonly id: string }>(store: Map<string, P>, provider: P): () => void {
|
||||
if (store.has(provider.id)) {
|
||||
throw new WebError(`a web provider with id "${provider.id}" is already registered`, 'WEB_DUPLICATE_PROVIDER')
|
||||
}
|
||||
const dispose = this.ctx.effect(function* (this: WebService) {
|
||||
store.set(provider.id, provider)
|
||||
// Yield the rollback BEFORE emitting `web/providers-change`: the generator
|
||||
// effect collects each yielded disposer before the next step runs, so a
|
||||
// throwing change listener removes the just-added provider instead of
|
||||
// leaking it into the registry.
|
||||
yield () => {
|
||||
store.delete(provider.id)
|
||||
this.ctx.emit('web/providers-change')
|
||||
}
|
||||
this.ctx.emit('web/providers-change')
|
||||
}.bind(this), 'web.registerProvider()')
|
||||
// ctx.effect's disposer returns Promise<void>; our disposer API is
|
||||
// synchronous fire-and-forget — discard the (always-resolved) promise.
|
||||
return () => void dispose()
|
||||
}
|
||||
|
||||
/** Search-capability selection status, derived live (never stored). */
|
||||
searchStatus(): WebCapabilityStatus {
|
||||
return resolveStatus({
|
||||
providers: this.searchProviders,
|
||||
...this.searchProviderId !== undefined ? { configuredId: this.searchProviderId } : {},
|
||||
})
|
||||
}
|
||||
|
||||
/** Fetch-capability selection status, derived live (never stored). */
|
||||
fetchStatus(): WebCapabilityStatus {
|
||||
return resolveStatus({
|
||||
providers: this.fetchProviders,
|
||||
...this.fetchProviderId !== undefined ? { configuredId: this.fetchProviderId } : {},
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Run one search through the selected provider. Resolves the provider at call
|
||||
* time with the selection rules above; throws {@link WebError} when the
|
||||
* capability cannot run. The seam enforces `request.maxResults` on the result:
|
||||
* if the provider over-returns, `sources[]` is truncated and `truncated` set.
|
||||
*/
|
||||
async search(request: WebSearchRequest, exec?: WebExecContext): Promise<WebSearchResult> {
|
||||
const provider = resolveProvider({
|
||||
providers: this.searchProviders,
|
||||
...this.searchProviderId !== undefined ? { configuredId: this.searchProviderId } : {},
|
||||
})
|
||||
const result = await provider.search(request, exec)
|
||||
return capSources(result, request.maxResults)
|
||||
}
|
||||
|
||||
/**
|
||||
* Retrieve one URL through the selected provider. Resolves the provider at
|
||||
* call time with the selection rules above; throws {@link WebError} when the
|
||||
* capability cannot run. A non-2xx response is a result, not a throw.
|
||||
*/
|
||||
async fetch(request: WebFetchRequest, exec?: WebExecContext): Promise<WebFetchResult> {
|
||||
const provider = resolveProvider({
|
||||
providers: this.fetchProviders,
|
||||
...this.fetchProviderId !== undefined ? { configuredId: this.fetchProviderId } : {},
|
||||
})
|
||||
return provider.fetch(request, exec)
|
||||
}
|
||||
}
|
||||
|
||||
interface ResolvableProvider {
|
||||
readonly id: string
|
||||
status(): WebProviderStatus
|
||||
}
|
||||
|
||||
/** Compute the capability status from configured id + registered providers. */
|
||||
function resolveStatus<P extends ResolvableProvider>(selection: Selection<P>): WebCapabilityStatus {
|
||||
const { configuredId, providers } = selection
|
||||
if (configuredId !== undefined) {
|
||||
const provider = providers.get(configuredId)
|
||||
if (!provider) return { available: false, reason: 'configured-missing' }
|
||||
if (!provider.status().available) return { available: false, reason: 'configured-unavailable' }
|
||||
return { available: true, providerId: configuredId }
|
||||
}
|
||||
const usable = [...providers.values()].filter(provider => provider.status().available)
|
||||
const [single] = usable
|
||||
if (single === undefined) return { available: false, reason: 'none' }
|
||||
if (usable.length > 1) return { available: false, reason: 'ambiguous' }
|
||||
return { available: true, providerId: single.id }
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve the selected provider or throw the matching {@link WebError}. Shares
|
||||
* the selection rules with {@link resolveStatus} so status and execution can
|
||||
* never disagree.
|
||||
*/
|
||||
function resolveProvider<P extends ResolvableProvider>(selection: Selection<P>): P {
|
||||
const { configuredId, providers } = selection
|
||||
if (configuredId !== undefined) {
|
||||
const provider = providers.get(configuredId)
|
||||
if (!provider) {
|
||||
throw new WebError(`configured web provider "${configuredId}" is not registered`, 'WEB_PROVIDER_CONFIGURED_MISSING')
|
||||
}
|
||||
if (!provider.status().available) {
|
||||
throw new WebError(`configured web provider "${configuredId}" is registered but unavailable`, 'WEB_PROVIDER_CONFIGURED_UNAVAILABLE')
|
||||
}
|
||||
return provider
|
||||
}
|
||||
const usable = [...providers.values()].filter(provider => provider.status().available)
|
||||
const [single] = usable
|
||||
if (single === undefined) {
|
||||
throw new WebError('no usable web provider is registered', 'WEB_PROVIDER_UNAVAILABLE')
|
||||
}
|
||||
if (usable.length > 1) {
|
||||
const ids = usable.map(provider => provider.id).join(', ')
|
||||
throw new WebError(`multiple usable web providers are registered (${ids}); configure one explicitly`, 'WEB_PROVIDER_AMBIGUOUS')
|
||||
}
|
||||
return single
|
||||
}
|
||||
|
||||
/** Enforce `maxResults` on a search result: truncate `sources[]` and flag it. */
|
||||
function capSources(result: WebSearchResult, maxResults: number | undefined): WebSearchResult {
|
||||
if (maxResults === undefined || result.sources.length <= maxResults) return result
|
||||
return { ...result, sources: result.sources.slice(0, maxResults), truncated: true }
|
||||
}
|
||||
|
||||
export default WebService
|
||||
225
packages/web/web/src/types.ts
Normal file
225
packages/web/web/src/types.ts
Normal file
@@ -0,0 +1,225 @@
|
||||
/**
|
||||
* Vocabulary for the web capability seam (`ctx.web`): the search/fetch
|
||||
* request/result shapes providers produce and consumers format, the provider
|
||||
* and capability status discriminants selection reports, the execution-control
|
||||
* context, and the typed error taxonomy.
|
||||
*
|
||||
* These types are shared by every provider backend
|
||||
* (`@deepseek-ai/dsh-web-search-exa`, `@deepseek-ai/dsh-web-search-perplexity`,
|
||||
* `@deepseek-ai/dsh-web-fetch-local`, and future backends) and by the
|
||||
* model-facing consumer (`@deepseek-ai/dsh-tool-web`). Search and fetch share no
|
||||
* request schema and no business logic, but they are deliberately one seam:
|
||||
* `ctx.web` is a single web-access middle layer with one provider-selection
|
||||
* policy, one abort/error vocabulary, and one product-facing configuration
|
||||
* point. The cost is the parallel `Search`/`Fetch` shapes below; that
|
||||
* parallelism is intentional.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-web/types
|
||||
*/
|
||||
|
||||
import { HarnessError } from '@deepseek-ai/dsh-llm'
|
||||
|
||||
/**
|
||||
* Execution control threaded from the tool layer through the seam into a
|
||||
* provider's network requests, stream readers, and expensive decoding. It is
|
||||
* NOT business input: the first version carries only `signal` so `tool-web` can
|
||||
* propagate turn cancellation, tool timeout, and agent disposal. It deliberately
|
||||
* does NOT carry `ToolExecution`, which would make `dsh-web` depend on
|
||||
* `dsh-tools`.
|
||||
*/
|
||||
export interface WebExecContext {
|
||||
/** Abort signal a provider must honor for its network/decoding work. */
|
||||
readonly signal?: AbortSignal
|
||||
}
|
||||
|
||||
/**
|
||||
* What one search-capable backend can return. The model-facing argument is just
|
||||
* a query; `maxResults` is a `dsh-tool-web`-layer bound passed through unchanged
|
||||
* and enforced on the way back by the seam (see {@link WebSearchResult}).
|
||||
*/
|
||||
export interface WebSearchRequest {
|
||||
readonly query: string
|
||||
/**
|
||||
* Upper bound on returned sources; the seam truncates to it. Omitted = no
|
||||
* bound. `dsh-tool-web` always sets it. A provider whose API supports a
|
||||
* result-count control (Exa's `numResults`) should apply it at the request
|
||||
* layer as a cost/latency optimization; the seam enforces the bound
|
||||
* regardless.
|
||||
*/
|
||||
readonly maxResults?: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalized search outcome. `content` is optional provider-generated answer
|
||||
* text or summary (Exa returns none; Perplexity returns a generated answer).
|
||||
* `sources[]` is the portable citation surface. `truncated` is set by the seam
|
||||
* when it cut `sources[]` down to `maxResults`.
|
||||
*/
|
||||
export interface WebSearchResult {
|
||||
/** Id of the provider that produced this result. */
|
||||
readonly providerId: string
|
||||
/** Echo of the query the provider answered. */
|
||||
readonly query: string
|
||||
/** Optional provider-generated answer text, search context, or summary. */
|
||||
readonly content?: string
|
||||
/** Citeable sources, already truncated to the request's `maxResults`. */
|
||||
readonly sources: readonly WebSearchSource[]
|
||||
/** True when the seam dropped sources to honor `maxResults`. */
|
||||
readonly truncated: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* One citeable source. A source always has a URL; `title`, `snippet`, and
|
||||
* `publishedAt` are optional because not every provider returns them — forcing
|
||||
* adapters to invent them would make the seam lie (Perplexity citations may be
|
||||
* URL-only). `dsh-tool-web` renders `title ?? hostname(url)` for display.
|
||||
*/
|
||||
export interface WebSearchSource {
|
||||
readonly url: string
|
||||
readonly title?: string
|
||||
readonly snippet?: string
|
||||
/** Publication/crawl timestamp as a provider-supplied ISO-8601 string. */
|
||||
readonly publishedAt?: string
|
||||
}
|
||||
|
||||
/**
|
||||
* What one fetch-capable backend is asked to retrieve. `timeoutMs` is an
|
||||
* optional positive hint the provider caps. The request deliberately omits
|
||||
* `format`, `prompt`, and extraction controls — those are presentation or
|
||||
* higher-level LLM concerns, not safe-retrieval inputs.
|
||||
*/
|
||||
export interface WebFetchRequest {
|
||||
readonly url: string
|
||||
readonly timeoutMs?: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalized fetch outcome. A successful network fetch of a non-2xx response is
|
||||
* a result, not an error: the status code is part of the fetched resource
|
||||
* state. {@link WebError} is reserved for failures to safely retrieve or
|
||||
* represent the resource.
|
||||
*/
|
||||
export interface WebFetchResult {
|
||||
/** Id of the provider that produced this result. */
|
||||
readonly providerId: string
|
||||
/** The final URL after allowed redirects (the request URL is in the request). */
|
||||
readonly url: string
|
||||
/** HTTP status code of the fetched response. */
|
||||
readonly statusCode: number
|
||||
/** Decoded body, classified by content kind. */
|
||||
readonly body: WebFetchBody
|
||||
/** True when the provider capped the decoded body. */
|
||||
readonly truncated: boolean
|
||||
}
|
||||
|
||||
/**
|
||||
* The decoded body of a fetched resource. A CLOSED discriminated union owned by
|
||||
* `dsh-web`: the provider decodes the kind and `dsh-tool-web` renders it, so a
|
||||
* new kind is a coordinated change across known packages, not a plugin
|
||||
* extension. Consumers `switch` on `kind` ending in `default: assertNever(...)`
|
||||
* so adding a kind breaks compilation at every consumer until handled. Each arm
|
||||
* stays its own object literal even where fields coincide today, leaving room
|
||||
* for arm-specific fields later (a `pdf` body's `pageCount`).
|
||||
*/
|
||||
export type WebFetchBody =
|
||||
| { readonly kind: 'html'; readonly content: string }
|
||||
| { readonly kind: 'text'; readonly content: string }
|
||||
|
||||
/**
|
||||
* Whether one concrete provider implementation is usable, by cheap local checks
|
||||
* only (credential presence, parseable endpoint config). A provider `status()`
|
||||
* must NOT make network calls. It is an input to selection, not a health system.
|
||||
*/
|
||||
export type WebProviderStatus =
|
||||
| { readonly available: true }
|
||||
| { readonly available: false; readonly reason: 'missing-credential' | 'misconfigured' }
|
||||
|
||||
/**
|
||||
* Whether a capability (search or fetch) has a selected usable provider, or the
|
||||
* broad category in which selection fails. Intentionally small: it carries the
|
||||
* winning `providerId` on the available branch (so diagnostics can report which
|
||||
* provider won) but NOT the per-reason payload (the missing id, the ambiguous
|
||||
* candidate set). That branchable detail lives in the {@link WebError} thrown at
|
||||
* execution time — the surface callers route on — so the same fact does not get
|
||||
* two homes that can disagree.
|
||||
*/
|
||||
export type WebCapabilityStatus =
|
||||
| { readonly available: true; readonly providerId: string }
|
||||
| { readonly available: false; readonly reason: 'none' | 'configured-missing' | 'configured-unavailable' | 'ambiguous' }
|
||||
|
||||
/**
|
||||
* A search-capable backend. Registered with `ctx.web.registerSearchProvider`.
|
||||
* `id` is a stable string, unique within the search capability kind.
|
||||
*/
|
||||
export interface WebSearchProvider {
|
||||
readonly id: string
|
||||
/** Cheap local usability check; must not make network calls. */
|
||||
status(): WebProviderStatus
|
||||
/** Run one search; honor `exec.signal` for cancellation. */
|
||||
search(request: WebSearchRequest, exec?: WebExecContext): Promise<WebSearchResult>
|
||||
}
|
||||
|
||||
/**
|
||||
* A fetch-capable backend. Registered with `ctx.web.registerFetchProvider`.
|
||||
* `id` is a stable string, unique within the fetch capability kind.
|
||||
*/
|
||||
export interface WebFetchProvider {
|
||||
readonly id: string
|
||||
/** Cheap local usability check; must not make network calls. */
|
||||
status(): WebProviderStatus
|
||||
/** Retrieve one URL; honor `exec.signal` for cancellation. */
|
||||
fetch(request: WebFetchRequest, exec?: WebExecContext): Promise<WebFetchResult>
|
||||
}
|
||||
|
||||
/**
|
||||
* Stable codes for {@link WebError}. Callers (hooks, tests, UI) route on these.
|
||||
*
|
||||
* - `WEB_PROVIDER_UNAVAILABLE`: no provider configured and none usable.
|
||||
* - `WEB_PROVIDER_CONFIGURED_MISSING`: a configured id is not registered.
|
||||
* - `WEB_PROVIDER_CONFIGURED_UNAVAILABLE`: a configured id is registered but its
|
||||
* `status()` reports unavailable.
|
||||
* - `WEB_PROVIDER_AMBIGUOUS`: no id configured and multiple usable providers
|
||||
* exist (selection refuses to pick by registration order).
|
||||
* - `WEB_DUPLICATE_PROVIDER`: a registration-time programming error — an id is
|
||||
* already registered for that capability kind.
|
||||
* - `WEB_INVALID_URL`: the fetch URL is malformed or not http(s).
|
||||
* - `WEB_BLOCKED_URL`: the fetch URL is rejected by policy (credentials in URL).
|
||||
* - `WEB_REDIRECT_BLOCKED`: a cross-origin redirect was refused.
|
||||
* - `WEB_FETCH_TOO_LARGE`: the response exceeded the byte/character cap.
|
||||
* - `WEB_FETCH_TIMEOUT`: the fetch exceeded its timeout.
|
||||
* - `WEB_ABORTED`: the operation was aborted via `WebExecContext.signal`.
|
||||
* - `WEB_UNSUPPORTED_CONTENT_TYPE`: the response content type cannot be decoded.
|
||||
* - `WEB_PROVIDER_ERROR`: catch-all for a provider's own failure surfaced through
|
||||
* the seam, including network/transport failure (DNS, connection refused, TLS).
|
||||
*/
|
||||
export type WebErrorCode =
|
||||
| 'WEB_PROVIDER_UNAVAILABLE'
|
||||
| 'WEB_PROVIDER_CONFIGURED_MISSING'
|
||||
| 'WEB_PROVIDER_CONFIGURED_UNAVAILABLE'
|
||||
| 'WEB_PROVIDER_AMBIGUOUS'
|
||||
| 'WEB_DUPLICATE_PROVIDER'
|
||||
| 'WEB_INVALID_URL'
|
||||
| 'WEB_BLOCKED_URL'
|
||||
| 'WEB_REDIRECT_BLOCKED'
|
||||
| 'WEB_FETCH_TOO_LARGE'
|
||||
| 'WEB_FETCH_TIMEOUT'
|
||||
| 'WEB_ABORTED'
|
||||
| 'WEB_UNSUPPORTED_CONTENT_TYPE'
|
||||
| 'WEB_PROVIDER_ERROR'
|
||||
|
||||
/**
|
||||
* Typed web error. Extends {@link HarnessError} so it carries a stable
|
||||
* {@link WebErrorCode} and chains `cause`. `dsh-web` owns this vocabulary so
|
||||
* providers, the seam, and the tool layer raise the same codes instead of each
|
||||
* inventing message strings. `ToolRegistry.execute()` converts a thrown
|
||||
* `WebError` into an error tool result whose structured metadata exposes the
|
||||
* code.
|
||||
*/
|
||||
export class WebError extends HarnessError {
|
||||
override readonly code: WebErrorCode
|
||||
|
||||
constructor(message: string, code: WebErrorCode, options?: ErrorOptions) {
|
||||
super(message, code, options)
|
||||
this.code = code
|
||||
}
|
||||
}
|
||||
263
packages/web/web/tests/web.spec.ts
Normal file
263
packages/web/web/tests/web.spec.ts
Normal file
@@ -0,0 +1,263 @@
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import WebService, {
|
||||
WebError,
|
||||
type WebFetchProvider,
|
||||
type WebFetchResult,
|
||||
type WebProviderStatus,
|
||||
type WebSearchProvider,
|
||||
type WebSearchRequest,
|
||||
type WebSearchResult,
|
||||
} from '@deepseek-ai/dsh-web'
|
||||
|
||||
/** A scripted search provider for contract tests. */
|
||||
function makeSearchProvider(
|
||||
id: string,
|
||||
status: WebProviderStatus,
|
||||
search: (request: WebSearchRequest) => Promise<WebSearchResult>,
|
||||
): WebSearchProvider {
|
||||
return { id, status: () => status, search: request => search(request) }
|
||||
}
|
||||
|
||||
function makeFetchProvider(id: string, status: WebProviderStatus, result: WebFetchResult): WebFetchProvider {
|
||||
return { id, status: () => status, fetch: () => Promise.resolve(result) }
|
||||
}
|
||||
|
||||
const available: WebProviderStatus = { available: true }
|
||||
const unavailable: WebProviderStatus = { available: false, reason: 'missing-credential' }
|
||||
|
||||
function searchResult(providerId: string, overrides: Partial<WebSearchResult> = {}): WebSearchResult {
|
||||
return { providerId, query: 'q', sources: [], truncated: false, ...overrides }
|
||||
}
|
||||
|
||||
function fetchResult(providerId: string): WebFetchResult {
|
||||
return { providerId, url: 'https://example.com', statusCode: 200, body: { kind: 'text', content: 'hi' }, truncated: false }
|
||||
}
|
||||
|
||||
/** Mount a WebService on a fresh root context with the given config. */
|
||||
async function mountWeb(config: ConstructorParameters<typeof WebService>[1] = {}): Promise<{ ctx: Context; web: WebService }> {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(WebService, config)
|
||||
return { ctx, web: ctx.web }
|
||||
}
|
||||
|
||||
describe('WebService registration', () => {
|
||||
it('registers and disposes a search provider, emitting providers-change each way', async () => {
|
||||
const { ctx, web } = await mountWeb()
|
||||
const changed = vi.fn()
|
||||
ctx.on('web/providers-change', changed)
|
||||
|
||||
const dispose = web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
expect(changed).toHaveBeenCalledTimes(1)
|
||||
expect(web.searchStatus()).toEqual({ available: true, providerId: 'exa' })
|
||||
|
||||
dispose()
|
||||
expect(changed).toHaveBeenCalledTimes(2)
|
||||
expect(web.searchStatus()).toEqual({ available: false, reason: 'none' })
|
||||
})
|
||||
|
||||
it('throws WEB_DUPLICATE_PROVIDER on a duplicate search id', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
expect(() => web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa')))))
|
||||
.toThrow(expect.objectContaining({ code: 'WEB_DUPLICATE_PROVIDER' }))
|
||||
})
|
||||
|
||||
it('keeps search and fetch id namespaces independent', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('shared', available, () => Promise.resolve(searchResult('shared'))))
|
||||
expect(() => web.registerFetchProvider(makeFetchProvider('shared', available, fetchResult('shared')))).not.toThrow()
|
||||
})
|
||||
|
||||
it('rolls back a registration when a providers-change listener throws', async () => {
|
||||
const { ctx, web } = await mountWeb()
|
||||
ctx.on('web/providers-change', () => { throw new Error('listener boom') })
|
||||
expect(() => web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa')))))
|
||||
.toThrow('listener boom')
|
||||
// The throwing listener must not leave the provider in the registry.
|
||||
expect(web.searchStatus()).toEqual({ available: false, reason: 'none' })
|
||||
})
|
||||
|
||||
it('disposes provider registrations when the contributing fiber is disposed (HMR safety)', async () => {
|
||||
const { ctx, web } = await mountWeb()
|
||||
const fiber = await ctx.plugin(Object.assign((inner: Context) => {
|
||||
inner.web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
}, { inject: ['web'] }))
|
||||
expect(web.searchStatus()).toEqual({ available: true, providerId: 'exa' })
|
||||
await fiber.dispose()
|
||||
expect(web.searchStatus()).toEqual({ available: false, reason: 'none' })
|
||||
})
|
||||
})
|
||||
|
||||
describe('WebService selection status', () => {
|
||||
it('reports none when nothing is registered', async () => {
|
||||
const { web } = await mountWeb()
|
||||
expect(web.searchStatus()).toEqual({ available: false, reason: 'none' })
|
||||
expect(web.fetchStatus()).toEqual({ available: false, reason: 'none' })
|
||||
})
|
||||
|
||||
it('auto-selects the single usable provider when no id is configured', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
expect(web.searchStatus()).toEqual({ available: true, providerId: 'exa' })
|
||||
})
|
||||
|
||||
it('reports ambiguous when multiple usable providers exist and none is configured', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
web.registerSearchProvider(makeSearchProvider('perplexity', available, () => Promise.resolve(searchResult('perplexity'))))
|
||||
expect(web.searchStatus()).toEqual({ available: false, reason: 'ambiguous' })
|
||||
})
|
||||
|
||||
it('ignores unusable providers when auto-selecting', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
web.registerSearchProvider(makeSearchProvider('perplexity', unavailable, () => Promise.resolve(searchResult('perplexity'))))
|
||||
expect(web.searchStatus()).toEqual({ available: true, providerId: 'exa' })
|
||||
})
|
||||
|
||||
it('reports none when providers exist but none are usable', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', unavailable, () => Promise.resolve(searchResult('exa'))))
|
||||
expect(web.searchStatus()).toEqual({ available: false, reason: 'none' })
|
||||
})
|
||||
|
||||
it('honors a configured id over a different registered provider', async () => {
|
||||
const { web } = await mountWeb({ searchProvider: 'perplexity' })
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
web.registerSearchProvider(makeSearchProvider('perplexity', available, () => Promise.resolve(searchResult('perplexity'))))
|
||||
expect(web.searchStatus()).toEqual({ available: true, providerId: 'perplexity' })
|
||||
})
|
||||
|
||||
it('reports configured-missing when the configured id is not registered', async () => {
|
||||
const { web } = await mountWeb({ searchProvider: 'perplexity' })
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
expect(web.searchStatus()).toEqual({ available: false, reason: 'configured-missing' })
|
||||
})
|
||||
|
||||
it('reports configured-unavailable when the configured id is registered but unusable', async () => {
|
||||
const { web } = await mountWeb({ searchProvider: 'exa' })
|
||||
web.registerSearchProvider(makeSearchProvider('exa', unavailable, () => Promise.resolve(searchResult('exa'))))
|
||||
expect(web.searchStatus()).toEqual({ available: false, reason: 'configured-unavailable' })
|
||||
})
|
||||
|
||||
it('does not let registration order change auto-selection', async () => {
|
||||
const a = await mountWeb()
|
||||
a.web.registerSearchProvider(makeSearchProvider('exa', unavailable, () => Promise.resolve(searchResult('exa'))))
|
||||
a.web.registerSearchProvider(makeSearchProvider('perplexity', available, () => Promise.resolve(searchResult('perplexity'))))
|
||||
expect(a.web.searchStatus()).toEqual({ available: true, providerId: 'perplexity' })
|
||||
|
||||
const b = await mountWeb()
|
||||
b.web.registerSearchProvider(makeSearchProvider('perplexity', available, () => Promise.resolve(searchResult('perplexity'))))
|
||||
b.web.registerSearchProvider(makeSearchProvider('exa', unavailable, () => Promise.resolve(searchResult('exa'))))
|
||||
expect(b.web.searchStatus()).toEqual({ available: true, providerId: 'perplexity' })
|
||||
})
|
||||
})
|
||||
|
||||
describe('WebService execution resolution', () => {
|
||||
it('throws WEB_PROVIDER_UNAVAILABLE when nothing is registered', async () => {
|
||||
const { web } = await mountWeb()
|
||||
await expect(web.search({ query: 'q' })).rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_UNAVAILABLE' }))
|
||||
})
|
||||
|
||||
it('throws WEB_PROVIDER_CONFIGURED_MISSING for an unregistered configured id', async () => {
|
||||
const { web } = await mountWeb({ searchProvider: 'perplexity' })
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
await expect(web.search({ query: 'q' })).rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_CONFIGURED_MISSING' }))
|
||||
})
|
||||
|
||||
it('throws WEB_PROVIDER_CONFIGURED_UNAVAILABLE for an unusable configured id', async () => {
|
||||
const { web } = await mountWeb({ searchProvider: 'exa' })
|
||||
web.registerSearchProvider(makeSearchProvider('exa', unavailable, () => Promise.resolve(searchResult('exa'))))
|
||||
await expect(web.search({ query: 'q' })).rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_CONFIGURED_UNAVAILABLE' }))
|
||||
})
|
||||
|
||||
it('throws WEB_PROVIDER_AMBIGUOUS rather than picking by order', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
web.registerSearchProvider(makeSearchProvider('perplexity', available, () => Promise.resolve(searchResult('perplexity'))))
|
||||
await expect(web.search({ query: 'q' })).rejects.toThrow(expect.objectContaining({ code: 'WEB_PROVIDER_AMBIGUOUS' }))
|
||||
})
|
||||
|
||||
it('runs the selected provider and returns its result', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(
|
||||
searchResult('exa', { content: 'answer', sources: [{ url: 'https://a' }] }),
|
||||
)))
|
||||
const result = await web.search({ query: 'q' })
|
||||
expect(result.providerId).toBe('exa')
|
||||
expect(result.content).toBe('answer')
|
||||
expect(result.sources).toEqual([{ url: 'https://a' }])
|
||||
})
|
||||
|
||||
it('propagates the abort signal to the provider', async () => {
|
||||
const { web } = await mountWeb()
|
||||
const seen: (AbortSignal | undefined)[] = []
|
||||
web.registerSearchProvider({
|
||||
id: 'exa',
|
||||
status: () => available,
|
||||
search: (_request, exec) => { seen.push(exec?.signal); return Promise.resolve(searchResult('exa')) },
|
||||
})
|
||||
const controller = new AbortController()
|
||||
await web.search({ query: 'q' }, { signal: controller.signal })
|
||||
expect(seen[0]).toBe(controller.signal)
|
||||
})
|
||||
})
|
||||
|
||||
describe('WebService maxResults enforcement', () => {
|
||||
it('truncates sources and sets truncated when a provider over-returns', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa', {
|
||||
sources: [{ url: 'https://1' }, { url: 'https://2' }, { url: 'https://3' }],
|
||||
}))))
|
||||
const result = await web.search({ query: 'q', maxResults: 2 })
|
||||
expect(result.sources).toHaveLength(2)
|
||||
expect(result.truncated).toBe(true)
|
||||
})
|
||||
|
||||
it('leaves truncated false when within the bound', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa', {
|
||||
sources: [{ url: 'https://1' }],
|
||||
}))))
|
||||
const result = await web.search({ query: 'q', maxResults: 8 })
|
||||
expect(result.sources).toHaveLength(1)
|
||||
expect(result.truncated).toBe(false)
|
||||
})
|
||||
|
||||
it('does not bound when maxResults is omitted', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa', {
|
||||
sources: [{ url: 'https://1' }, { url: 'https://2' }],
|
||||
}))))
|
||||
const result = await web.search({ query: 'q' })
|
||||
expect(result.sources).toHaveLength(2)
|
||||
expect(result.truncated).toBe(false)
|
||||
})
|
||||
})
|
||||
|
||||
describe('WebService fetch capability', () => {
|
||||
it('resolves and runs the fetch provider independently of search', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerFetchProvider(makeFetchProvider('local-http', available, fetchResult('local-http')))
|
||||
const result = await web.fetch({ url: 'https://example.com' })
|
||||
expect(result.providerId).toBe('local-http')
|
||||
expect(result.statusCode).toBe(200)
|
||||
})
|
||||
|
||||
it('throws WEB_PROVIDER_UNAVAILABLE for fetch when no fetch provider is registered', async () => {
|
||||
const { web } = await mountWeb()
|
||||
web.registerSearchProvider(makeSearchProvider('exa', available, () => Promise.resolve(searchResult('exa'))))
|
||||
await expect(web.fetch({ url: 'https://example.com' })).rejects.toThrow(
|
||||
expect.objectContaining({ code: 'WEB_PROVIDER_UNAVAILABLE' }),
|
||||
)
|
||||
})
|
||||
})
|
||||
|
||||
describe('WebError', () => {
|
||||
it('is a HarnessError carrying its code', () => {
|
||||
const error = new WebError('boom', 'WEB_INVALID_URL')
|
||||
expect(error.code).toBe('WEB_INVALID_URL')
|
||||
expect(error.name).toBe('WebError')
|
||||
})
|
||||
})
|
||||
24
packages/web/web/tsconfig.json
Normal file
24
packages/web/web/tsconfig.json
Normal file
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": {
|
||||
"rootDir": "src",
|
||||
"outDir": "lib"
|
||||
},
|
||||
"include": [
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../../vendor/cosmokit"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/schemastery"
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm"
|
||||
}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user