Merge branch 'codex/invariant-package-registration-gate' into codex/package-invariant-checks
# Conflicts: # .agents/notes/implemented/architecture/2026-07-19-package-invariant-runtime-contracts.i18n.yaml # .agents/notes/implemented/architecture/2026-07-19-package-invariant-runtime-contracts.md # .agents/notes/implemented/architecture/2026-07-19-package-invariant-runtime-contracts.zh.md # .agents/notes/implemented/architecture/2026-07-19-package-owned-invariant-service.i18n.yaml # docs/rfc/INDEX.md # packages/AGENTS.md
This commit is contained in:
Tianyi Cui
@@ -13,4 +13,4 @@ The web access capability seam: an abstract web interface, search/fetch provider
|
||||
|
||||
The interface lives at `web/web/`. Unlike bash/fs, the seam spans **two capabilities** (search and fetch) with potentially multiple providers each: `ctx.web` is one web-access middle layer with one provider-selection policy, one abort/error vocabulary, and one product-facing "how this harness reaches the web" config surface. Providers register **capabilities**, not tools; `tool-web` is the only owner of model-facing names, schemas, prompt guidance, and presentation. A search provider swap does not change how the model asks for a query, and a fetch implementation swap does not change how the model asks for a URL.
|
||||
|
||||
See the [web capability seam RFC](../../docs/rfc/implemented/architecture/2026-06-24-web-capability-seam.md) for the design rationale, including why search and fetch are deliberately one seam and why `web_fetch`'s SSRF protection is deferred.
|
||||
See the [web capability seam Agent Note](../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md) for the design rationale, including why search and fetch are deliberately one seam and why `web_fetch`'s SSRF protection is deferred.
|
||||
|
||||
@@ -40,48 +40,88 @@ The tool never calls a provider's `available()` and never enumerates providers
|
||||
|
||||
### System prompt
|
||||
|
||||
**What the model sees**: Search and fetch contribute the web-search and web-fetch guidance below. A scoped tool restriction does not remove these independently registered sections.
|
||||
#### What the model sees
|
||||
|
||||
**Token effect**: Fixed guidance cost per request for each config-enabled tool, even when a restriction hides its schema.
|
||||
Search and fetch contribute the web-search and web-fetch guidance below. A scoped tool restriction does not remove these independently registered sections.
|
||||
|
||||
#### Web search guidance
|
||||
##### Web search guidance
|
||||
|
||||
```markdown
|
||||
Use the web_search tool to discover current information on the web. It returns an optional answer plus a list of source URLs. Follow up with web_fetch when you need the full content of a specific result, and cite the relevant URLs as markdown links.
|
||||
```
|
||||
|
||||
#### Web fetch guidance
|
||||
##### Web fetch guidance
|
||||
|
||||
```markdown
|
||||
Use the web_fetch tool to retrieve the content of a specific HTTP(S) URL (for example a result from web_search). It returns the page content decoded to text. Cite the URL as a markdown link when you use its content.
|
||||
```
|
||||
|
||||
#### Token effect
|
||||
|
||||
Fixed guidance cost per request for each config-enabled tool, even when a restriction hides its schema.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
Prefix-stable while enabled tools, scope, and guidance text are unchanged. Config enablement or plugin lifecycle may invalidate reuse from the first changed prompt section; scoped schema restrictions do not remove it.
|
||||
|
||||
### Tool schemas
|
||||
|
||||
**What the model sees**: The model sees the generated [`web_search` and `web_fetch` schemas](../../../docs/tool-catalog.md#deepseek-aidsh-tool-web). Result-count and timeout budgets are deployment settings, not model arguments.
|
||||
#### What the model sees
|
||||
|
||||
**Token effect**: Fixed schema cost per request; config disablement removes both schema and guidance, while a scoped restriction removes only the schema.
|
||||
The model sees the generated [`web_search` and `web_fetch` schemas](../../../docs/tool-catalog.md#deepseek-aidsh-tool-web). Result-count and timeout budgets are deployment settings, not model arguments.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Fixed schema cost per request; config disablement removes both schema and guidance, while a scoped restriction removes only the schema.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
Prefix-stable while definitions and visibility are unchanged. Config enablement, plugin lifecycle, or scoped restrictions may invalidate reuse from the first changed schema token.
|
||||
|
||||
### Search result
|
||||
|
||||
**What the model sees**: The optional provider-owned answer is followed by `Sources:` and data-dependent lines shaped exactly `- [<title-or-url>](<url>)`, optionally suffixed ` — <snippet> (<publishedAt>)`. With neither answer nor sources the result says `No results found.` A capped list adds `(Showing the first <count> sources. Refine the query for more.)`; every result ends `Cite the relevant URLs above as markdown links in your answer.`
|
||||
#### What the model sees
|
||||
|
||||
**Token effect**: Data-dependent results are resent until compaction and sources are capped by `searchMaxResults`.
|
||||
The optional provider-owned answer is followed by `Sources:` and data-dependent lines shaped exactly `- [<title-or-url>](<url>)`, optionally suffixed ` — <snippet> (<publishedAt>)`. With neither answer nor sources the result says `No results found.` A capped list adds `(Showing the first <count> sources. Refine the query for more.)`; every result ends `Cite the relevant URLs above as markdown links in your answer.`
|
||||
|
||||
#### Token effect
|
||||
|
||||
Data-dependent results are resent until compaction and sources are capped by `searchMaxResults`.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
Append-only; newly visible content follows the reusable request prefix and does not invalidate existing KV-cache entries.
|
||||
|
||||
### Fetch result
|
||||
|
||||
**What the model sees**: A successful fetch is exactly `Fetched <finalUrl> (HTTP <statusCode>)`, a blank line, and the provider-owned decoded body. Truncation adds a blank line and `(Content truncated. Fetch a more specific URL or section for the full text.)`; failures become `Error: <message>`. Queries and URLs remain in call history.
|
||||
#### What the model sees
|
||||
|
||||
**Token effect**: Provider caps bound body size; retained call arguments and results are resent until compaction, and timeout policy can replace a late result with a short error.
|
||||
A successful fetch is exactly `Fetched <finalUrl> (HTTP <statusCode>)`, a blank line, and the provider-owned decoded body. Truncation adds a blank line and `(Content truncated. Fetch a more specific URL or section for the full text.)`; failures become `Error: <message>`. Queries and URLs remain in call history.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Provider caps bound body size; retained call arguments and results are resent until compaction, and timeout policy can replace a late result with a short error.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
Append-only; newly visible content follows the reusable request prefix and does not invalidate existing KV-cache entries.
|
||||
|
||||
### Argument errors
|
||||
|
||||
**What the model sees**: Blank inputs become exactly `Error: query must be a non-empty string` or `Error: url must be a non-empty string`.
|
||||
#### What the model sees
|
||||
|
||||
**Token effect**: Only the failing call adds these retained tokens.
|
||||
Blank inputs become exactly `Error: query must be a non-empty string` or `Error: url must be a non-empty string`.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Only the failing call adds these retained tokens.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
Append-only; newly visible content follows the reusable request prefix and does not invalidate existing KV-cache entries.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **`htmlToMarkdown` is a minimal regex converter, not an HTML parser** — it strips script/style/noscript, keeps headings/bullets/links, and decodes about a dozen named entities; tables, images, and nested formatting are lost.
|
||||
- **The model-facing surface is minimal by design, with promotions deferred** — `max_results` stays a config bound (not a model argument), and `web_fetch` takes only `url` (no `format`/`prompt`/LLM-summarization mode); both are named later steps in [the seam RFC](../../../docs/rfc/implemented/architecture/2026-06-24-web-capability-seam.md).
|
||||
- **The model-facing surface is minimal by design, with promotions deferred** — `max_results` stays a config bound (not a model argument), and `web_fetch` takes only `url` (no `format`/`prompt`/LLM-summarization mode); both are named later steps in [the seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md).
|
||||
- **No web-specific permission policy** — both tools execute without requesting `ctx.approval`; a deployment that needs confirmation must add a `tools/pre-execute` policy, and the package does not define persistent URL/domain grants.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
/**
|
||||
* Showcase integration: the real `web_fetch` tool + the real spill stack
|
||||
* (`dsh-spill-local` backend + `dsh-spill-policy`), exercised through
|
||||
* `ctx.tools.execute()`. Proves the RFC's default local-backend path — a large
|
||||
* `ctx.tools.execute()`. Proves the Agent Note's default local-backend path — a large
|
||||
* formatted fetch result is automatically retained and spilled with NO
|
||||
* tool-specific spill code, and the model-facing text changes ONLY by the
|
||||
* deliberate spill notice (the full formatted result lands in the spill file).
|
||||
@@ -48,7 +48,7 @@ beforeEach(async () => {
|
||||
await ctx.plugin(ToolRegistry)
|
||||
await ctx.plugin(WebService, { fetchProvider: WebFetchLocal.LOCAL_FETCH_PROVIDER_ID })
|
||||
// Provider cap generous so the tool returns a large formatted result; the
|
||||
// policy cap is what triggers the spill (the RFC's separation of concerns).
|
||||
// policy cap is what triggers the spill (the Agent Note's separation of concerns).
|
||||
await ctx.plugin(WebFetchLocal, { maxBodyChars: 500_000 })
|
||||
await ctx.plugin(LocalSpillStore, { root: spillRoot })
|
||||
await ctx.plugin(SpillPolicy, { maxInlineBytes: MAX_INLINE_BYTES })
|
||||
|
||||
@@ -38,8 +38,12 @@ The numeric limits are validated at plugin construction: every cap except `maxRe
|
||||
|
||||
Indirectly, through [`dsh-tool-web`](../tool-web/README.md), which places this provider's `maxBodyChars`-bounded decoded text or markdown-shaped HTML under its fetch-result wrapper and retains provider failures while redirects, headers, and transport mechanics remain hidden.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
No direct invalidation; the named consumer owns any request-prefix changes.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **SSRF / private-network protection is deferred** — no blocking of private, loopback, link-local, multicast, or otherwise non-public destinations, no DNS-resolve-then-validate, no per-hop re-validation (see [the web capability seam RFC](../../../docs/rfc/implemented/architecture/2026-06-24-web-capability-seam.md)). Until it lands, this provider is an SSRF primitive and **must not be enabled** in a deployment that can reach sensitive internal network targets.
|
||||
- **SSRF / private-network protection is deferred** — no blocking of private, loopback, link-local, multicast, or otherwise non-public destinations, no DNS-resolve-then-validate, no per-hop re-validation (see [the web capability seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md)). Until it lands, this provider is an SSRF primitive and **must not be enabled** in a deployment that can reach sensitive internal network targets.
|
||||
- **Only textual content decodes** — html/xhtml and `text/*`-plus-JSON/XML families; a missing `Content-Type` or any binary type throws `WEB_UNSUPPORTED_CONTENT_TYPE`, and text-extractable PDF decoding is named deferred work.
|
||||
- **Charset comes only from the `Content-Type` header** (UTF-8 default) — an HTML `<meta charset>` declaration is ignored, and a declared-but-unrecognized charset label throws rather than falling back.
|
||||
|
||||
@@ -15,7 +15,7 @@ export type FetchableKind = 'html' | 'text'
|
||||
* Validate a request URL against the basic transport hygiene the provider
|
||||
* enforces before any network access: http(s) only, no embedded credentials,
|
||||
* bounded length. Returns the parsed `URL`. Throws {@link WebError} otherwise.
|
||||
* (SSRF / private-network blocking is deferred — see the package RFC.)
|
||||
* (SSRF / private-network blocking is deferred — see the package Agent Note.)
|
||||
*
|
||||
* @param input - the raw URL string from the fetch request.
|
||||
* @param maxUrlLength - inclusive upper bound on `input`'s length.
|
||||
|
||||
@@ -43,15 +43,31 @@ Provider failures become `WEB_PROVIDER_ERROR`; caller cancellation becomes `WEB_
|
||||
|
||||
### Auxiliary DeepSeek search request
|
||||
|
||||
**What the model sees**: A separate DeepSeek model receives exactly `Perform a web search for the query: <query>` as its user text and one native `web_search` server-tool definition. This request is not part of the conversation model's context.
|
||||
#### What the model sees
|
||||
|
||||
**Token effect**: Separate provider input and output tokens are incurred for each search; `maxTokens` caps generated output and `maxUses` caps native search uses.
|
||||
A separate DeepSeek model receives exactly `Perform a web search for the query: <query>` as its user text and one native `web_search` server-tool definition. This request is not part of the conversation model's context.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Separate provider input and output tokens are incurred for each search; `maxTokens` caps generated output and `maxUses` caps native search uses.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
Independent of the conversation request cache. The auxiliary instruction and native tool definition can form a stable prefix, but each changed query or model route prevents reuse from its first difference.
|
||||
|
||||
### Conversation tool result, indirectly
|
||||
|
||||
**What the model sees**: Through [`dsh-tool-web`](../tool-web/README.md), the conversation model sees deduplicated URLs, titles, dates, and citation snippets from structured search blocks; provider prose is not trusted as an answer. This provider's exact failures are `DeepSeek search aborted`, `DeepSeek search request failed: <error>`, `DeepSeek returned no web_search_tool_result blocks; the request may not have triggered native web search`, and `DeepSeek returned an unprocessable response body: <error>`; HTTP failures preserve the provider message. The consumer owns the error wrapper.
|
||||
#### What the model sees
|
||||
|
||||
**Token effect**: Zero direct conversation tokens from registration. Result tokens scale with returned sources and snippets, then the seam enforces the requested source bound.
|
||||
Through [`dsh-tool-web`](../tool-web/README.md), the conversation model sees deduplicated URLs, titles, dates, and citation snippets from structured search blocks; provider prose is not trusted as an answer. This provider's exact failures are `DeepSeek search aborted`, `DeepSeek search request failed: <error>`, `DeepSeek returned no web_search_tool_result blocks; the request may not have triggered native web search`, and `DeepSeek returned an unprocessable response body: <error>`; HTTP failures preserve the provider message. The consumer owns the error wrapper.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Zero direct conversation tokens from registration. Result tokens scale with returned sources and snippets, then the seam enforces the requested source bound.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
Append-only; newly visible content follows the reusable request prefix and does not invalidate existing KV-cache entries.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
|
||||
@@ -29,8 +29,12 @@ Exa returns a flat `results[]` and no generated answer, so `content` is omitted.
|
||||
|
||||
Indirectly, through [`dsh-tool-web`](../tool-web/README.md), which retains this provider's `maxResults`-bounded URLs, titles, first highlights, and publication dates or its exact `Exa search aborted`, `Exa search request failed: <error>`, and `Exa returned an unprocessable response body: <error>` failures under the consumer's error wrapper while generated answers and provider-private fields remain outside context.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
No direct invalidation; the named consumer owns any request-prefix changes.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **A result with no non-blank highlight is dropped entirely** — no portable snippet to map, so fewer sources than the requested count can return.
|
||||
- **Only `searchType`/`numResults`/`highlightsPerResult` are exposed** — Exa's other controls (livecrawl, category, domain/date filters, full-text contents) wait on provider-neutral seam fields ([seam RFC](../../../docs/rfc/implemented/architecture/2026-06-24-web-capability-seam.md)).
|
||||
- **Only `searchType`/`numResults`/`highlightsPerResult` are exposed** — Exa's other controls (livecrawl, category, domain/date filters, full-text contents) wait on provider-neutral seam fields ([seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md)).
|
||||
- **Abort classification is error-shape-based** — only a `DOMException` named `AbortError` maps to `WEB_ABORTED`; an abort carrying a custom reason (e.g. `dsh-timeout`'s `TimeoutReason`) surfaces as `WEB_PROVIDER_ERROR`.
|
||||
|
||||
@@ -29,19 +29,35 @@ This is an **implementation** package: it registers a provider into `ctx.web`, i
|
||||
|
||||
### Auxiliary Perplexity request
|
||||
|
||||
**What the model sees**: A separate Perplexity model receives `<query>` verbatim as its sole user message through the chat-completions endpoint. This request is not part of the conversation model's context.
|
||||
#### What the model sees
|
||||
|
||||
**Token effect**: Separate provider tokens are incurred per search; `maxTokens` caps the generated answer.
|
||||
A separate Perplexity model receives `<query>` verbatim as its sole user message through the chat-completions endpoint. This request is not part of the conversation model's context.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Separate provider tokens are incurred per search; `maxTokens` caps the generated answer.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
Independent of the conversation request cache. An identical query under the same model route may reuse provider cache; a changed query or route establishes a different prefix.
|
||||
|
||||
### Conversation tool result, indirectly
|
||||
|
||||
**What the model sees**: Through [`dsh-tool-web`](../tool-web/README.md), the conversation model sees the generated answer plus structured result metadata or URL-only citations. This provider's exact failures are `Perplexity search aborted`, `Perplexity search request failed: <error>`, and `Perplexity returned an unprocessable response body: <error>`; HTTP failures preserve the provider message. The consumer owns the error wrapper.
|
||||
#### What the model sees
|
||||
|
||||
**Token effect**: Zero direct conversation tokens from registration. Answer and source tokens are data-dependent, source count is seam-bounded, and the retained result or error is resent until compaction.
|
||||
Through [`dsh-tool-web`](../tool-web/README.md), the conversation model sees the generated answer plus structured result metadata or URL-only citations. This provider's exact failures are `Perplexity search aborted`, `Perplexity search request failed: <error>`, and `Perplexity returned an unprocessable response body: <error>`; HTTP failures preserve the provider message. The consumer owns the error wrapper.
|
||||
|
||||
#### Token effect
|
||||
|
||||
Zero direct conversation tokens from registration. Answer and source tokens are data-dependent, source count is seam-bounded, and the retained result or error is resent until compaction.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
Append-only; newly visible content follows the reusable request prefix and does not invalidate existing KV-cache entries.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **Citation-fallback sources are URL-only** — when Perplexity omits structured `search_results[]`, sources carry no `title`/`snippet`/`publishedAt`, so the tool renders bare hostname labels.
|
||||
- **Over-returned sources still cost tokens and latency** — with no result-count control on the wire, `maxResults` is enforced only post-hoc by seam truncation.
|
||||
- **Only `model`/`maxTokens`/`searchRecency` are exposed** — Perplexity's other search controls (domain filters, `web_search_options` context size, images) wait on provider-neutral seam fields ([seam RFC](../../../docs/rfc/implemented/architecture/2026-06-24-web-capability-seam.md)).
|
||||
- **Only `model`/`maxTokens`/`searchRecency` are exposed** — Perplexity's other search controls (domain filters, `web_search_options` context size, images) wait on provider-neutral seam fields ([seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md)).
|
||||
- **Abort classification is error-shape-based** — only a `DOMException` named `AbortError` maps to `WEB_ABORTED`; an abort carrying a custom reason (e.g. `dsh-timeout`'s `TimeoutReason`) surfaces as `WEB_PROVIDER_ERROR`.
|
||||
|
||||
@@ -47,9 +47,13 @@ The failure branches throw `WebError`, whose structured code (plus message detai
|
||||
|
||||
Indirectly, through `dsh-tool-web`, which retains bounded normalized provider data or the exact configured-provider, unavailable-provider, no-provider, multiple-provider, and `Error: <message>` failures while this registry contributes no prompt or schema itself.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
No direct invalidation; the named consumer owns any request-prefix changes.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- **No observation surface** — no provider-change event and no capability-status query; availability is observed only by executing `search()`/`fetch()` and routing the thrown `WebError` codes, and the no-provider failure is the generic `WEB_PROVIDER_UNAVAILABLE` with no per-provider reason enumeration ([RFC](../../../docs/rfc/implemented/simplification/2026-07-04-drop-unconsumed-web-observation-surface.md)).
|
||||
- **`WebSearchRequest` carries only `query` + `maxResults`** — provider-neutral controls (recency, domain filters, regional hints, search depth) are deferred until Exa and Perplexity can both honor them honestly ([seam RFC](../../../docs/rfc/implemented/architecture/2026-06-24-web-capability-seam.md)).
|
||||
- **No observation surface** — no provider-change event and no capability-status query; availability is observed only by executing `search()`/`fetch()` and routing the thrown `WebError` codes, and the no-provider failure is the generic `WEB_PROVIDER_UNAVAILABLE` with no per-provider reason enumeration ([Agent Note](../../../.agents/notes/implemented/simplification/2026-07-04-drop-unconsumed-web-observation-surface.md)).
|
||||
- **`WebSearchRequest` carries only `query` + `maxResults`** — provider-neutral controls (recency, domain filters, regional hints, search depth) are deferred until Exa and Perplexity can both honor them honestly ([seam Agent Note](../../../.agents/notes/implemented/architecture/2026-06-24-web-capability-seam.md)).
|
||||
- **`WebFetchBody` has no `pdf` arm** — text-extractable PDF support is named deferred work; the closed union makes adding it a compile-enforced change across the three web packages.
|
||||
- **Provider-backed page extraction is out of scope of `fetch()`** — a Firecrawl/Tavily-style `web_extract` capability is deferred rather than widening the fetch seam.
|
||||
|
||||
Reference in New Issue
Block a user