|
|
|
|
@@ -12,7 +12,7 @@ A workflow capability family at `packages/workflow/` in the bash seam shape (int
|
|
|
|
|
|
|
|
|
|
### The script contract (Claude Code-compatible)
|
|
|
|
|
|
|
|
|
|
A script is `export const meta = {...}` (a PURE object literal: `name`, `description`, optional `whenToUse`/`phases`) followed by a plain-JS body with top-level `await`, ending in `return <json-value>`. The body sees exactly: `agent(prompt, {label, phase, schema, model})`, `parallel(thunks)`, `pipeline(items, ...stages)` (NO cross-stage barrier; `(prev, item, index)` callbacks), `phase(title)`, `log(message)`, and `args`. CC semantics are preserved where they matter to script authors: a failed child resolves `null` (scripts `.filter(Boolean)`); an ordinary stage throw nulls the ITEM and skips its remaining stages. CC's determinism bans (`Date.now()`/`Math.random()`/argless `new Date()` throwing) are NOT enforced — they exist for CC's journaling/resume, which this cut defers — so a CC-authored script runs unchanged while scripts written here may freely read the clock.
|
|
|
|
|
A workflow call is two parts: a `meta` JSON parameter (the identity block — `name`, `description`, optional `whenToUse`/`phases`; the field vocabulary matches Claude Code's meta block) and a `script` — a plain-JS body with top-level `await`, ending in `return <json-value>`. Meta is DATA, never code: the engine shape-validates it and evaluates no script text to obtain it (a body still opening with a CC-style `export const meta` statement is rejected with a pointed message). The body sees exactly: `agent(prompt, {label, phase, schema, model})`, `parallel(thunks)`, `pipeline(items, ...stages)` (NO cross-stage barrier; `(prev, item, index)` callbacks), `phase(title)`, `log(message)`, and `args`. CC semantics are preserved where they matter to script authors: a failed child resolves `null` (scripts `.filter(Boolean)`); an ordinary stage throw nulls the ITEM and skips its remaining stages. CC's determinism bans (`Date.now()`/`Math.random()`/argless `new Date()` throwing) are NOT enforced — they exist for CC's journaling/resume, which this cut defers — so a CC-authored BODY runs unchanged (its meta header moves into the parameter) while scripts written here may freely read the clock.
|
|
|
|
|
|
|
|
|
|
One deliberate strictness DIVERGENCE from CC: hook misuse — unknown or deferred options (`effort`/`isolation`/`agentType`), malformed arguments, schemas outside the supported subset, tripped caps, seam start failures — throws a `WorkflowError` with `fatal: true`, and the combinators RE-THROW fatal errors instead of nulling the item. Without this, a typo'd option dissolves into a `null` indistinguishable from a child failure — the accepted-then-ignored failure mode this repo bans. One addition: the tool's `args` parameter is a JSON OBJECT (a bare list is wrapped as a field) so the wire schema stays honest.
|
|
|
|
|
|
|
|
|
|
@@ -24,15 +24,15 @@ One deliberate strictness DIVERGENCE from CC: hook misuse — unknown or deferre
|
|
|
|
|
|
|
|
|
|
**Trust premise (governs every engine decision below)**: workflow scripts are MODEL-WRITTEN — the same trust level as the model's existing bash access — so the engine defends against BUGGY scripts, never hostile ones. In scope: `result` never rejects, no unhandled rejections from dropped hook promises, loud rejection of values JSON cannot carry, fatal-vs-null hook discipline, cancellation that always frees the caller. Out of scope, deliberately: adversarial values (throwing/spinning accessors, proxies with hostile traps, prototype forgery, `prepareStackTrace` hijack) AND Node-API escape from the script's context — the vm context shares object machinery with its surrounding realm, so a script can reach the `Function` constructor (`globalThis.constructor.constructor`) and from it `process` and every Node builtin; the absent globals are API surface, not containment, and a worker thread is NOT a security boundary (an escapee holds process-wide privileges — Node's permission model is per-process). Worker-side code MAY run script code while reading script values, and that is accepted: a synchronous spin costs the script its OWN thread (terminated at the post-cancel grace), never the host loop, so containing error VALUES would be cost without a threat model. Genuine sandboxing (isolated-vm, a separate process) remains an engine swap behind the seam, not incremental defenses here.
|
|
|
|
|
|
|
|
|
|
**Why node:worker_threads**: one run = one worker thread, no pooling — a run is heavyweight (many children), so thread spin-up (~tens of ms) is noise. The script runs in a vm context INSIDE the worker, keeping the script-visible surface exactly the hook contract above (a bare worker realm would leak `setTimeout`/`fetch`/`process` as accidental API), and every `agent()` bridges to `ctx.subagents` by message-port RPC — children are I/O-bound LLM loops and stay on the host loop; the thread isolates the SCRIPT, the only part that can spin. What the thread buys: `start()` never blocks the host (an in-process engine runs the initial synchronous slice inline and cannot kill a spin past the first await — it could only ABANDON such a script, leaving the spin on the host loop), the post-cancel grace ends in a REAL `worker.terminate()`, and the value boundary is serialization by construction. isolated-vm was rejected for actual sandboxing: maintenance mode, `--no-node-snapshot` on EVERY consumer process (including published bins) on Node ≥ 20, node-gyp source-build fallback. Key mechanics (details in the package README): meta extraction and a body pre-parse stay HOST-side (preserving the seam's synchronous throws), a ready→go handshake keeps a run cancelled before start from ever executing the body, `cancel()` drives both child-cancel channels host-side (the shared request signal AND each child's explicit `cancel()` — a wedged worker cannot relay its own cancel RPCs), a host-side child registry backs worker-death reaping and `dispose()` quiescence, the wire protocol is enum-keyed payload maps private to the package, and on a termination path `agentsStarted` degrades to the host-observed count. Coverage puts the worker-side session on an in-process `MessageChannel` (real-Worker code is invisible to main-process v8) and proves the built `lib/worker.js` — a second tsdown entry, sanctioned in the workspace-constraints gate by the `"./worker"` subpath export — under plain node in the built-bin smoke gate.
|
|
|
|
|
**Why node:worker_threads**: one run = one worker thread, no pooling — a run is heavyweight (many children), so thread spin-up (~tens of ms) is noise. The script runs in a vm context INSIDE the worker, keeping the script-visible surface exactly the hook contract above (a bare worker realm would leak `setTimeout`/`fetch`/`process` as accidental API), and every `agent()` bridges to `ctx.subagents` by message-port RPC — children are I/O-bound LLM loops and stay on the host loop; the thread isolates the SCRIPT, the only part that can spin. What the thread buys: `start()` never blocks the host (an in-process engine runs the initial synchronous slice inline and cannot kill a spin past the first await — it could only ABANDON such a script, leaving the spin on the host loop), the post-cancel grace ends in a REAL `worker.terminate()`, and the value boundary is serialization by construction. isolated-vm was rejected for actual sandboxing: maintenance mode, `--no-node-snapshot` on EVERY consumer process (including published bins) on Node ≥ 20, node-gyp source-build fallback. Key mechanics (details in the package README): meta shape-validation and a body pre-parse stay HOST-side (preserving the seam's synchronous throws), a ready→go handshake keeps a run cancelled before start from ever executing the body, `cancel()` drives both child-cancel channels host-side (the shared request signal AND each child's explicit `cancel()` — a wedged worker cannot relay its own cancel RPCs), a host-side child registry backs worker-death reaping and `dispose()` quiescence, the wire protocol is enum-keyed payload maps private to the package, and on a termination path `agentsStarted` degrades to the host-observed count. Coverage puts the worker-side session on an in-process `MessageChannel` (real-Worker code is invisible to main-process v8) and proves the built `lib/worker.js` — a second tsdown entry, sanctioned in the workspace-constraints gate by the `"./worker"` subpath export — under plain node in the built-bin smoke gate.
|
|
|
|
|
|
|
|
|
|
**Meta extraction**: a string/comment-aware brace scanner (template interpolation rejected) finds the literal; it is evaluated ALONE in an empty, timed vm context; the result must materialize to plain JSON data and pass shape validation (unknown fields rejected loud); the statement is blanked line-preservingly so stacks keep script line numbers.
|
|
|
|
|
**Meta as data, never evaluated**: the meta block reaches the seam as a plain JSON request field (the tool's schema-validated `meta` parameter) and the engine only shape-validates it, every violation named. This is a host-isolation invariant, not a convenience: evaluating a meta literal host-side — even one contractually "pure", in an empty timed vm context — hands script-controlled getters a host stack with no timeout the moment the result is READ, defeating the exact spin isolation the worker thread buys.
|
|
|
|
|
|
|
|
|
|
**Value boundary**: values leaving the script (meta, hook options, schemas, the return value) go through `materializeFromRealm` — a plain recursive walk that rejects loud everything JSON cannot carry (exotic prototypes, functions, symbols, cycles, sparse arrays, non-finite numbers, nested `undefined`), copying via `Object.defineProperty` so a `"__proto__"` key becomes a data property, never a prototype mutation; getters are read ordinarily and their RESULT crosses (a throwing read fails loud) — which is also what makes every later postMessage hop total. Values entering the realm (`args`, `agent()` results, hook promises and failures, combinator arrays) are handed over directly as worker-realm values — the script is trusted, so outer prototypes are not a leak; `args` rides the `workerData` structured clone (the caller-isolation copy) and is cloned once more so a script scribbling on it cannot mutate the session's init object. Hook failures are `WorkflowError`s built OUTSIDE the script's context: the combinators recognize fatality by `instanceof` against the engine's own class (unforgeable from the script), and the script-visible consequence — in-script `instanceof Error` is `false` for hook errors; branch on `e.name`/`e.code` — is documented in the engine README. Realm functions (stages, thunks) are called, never materialized. Thrown script values are rendered by a total renderer (stack → message → `String()`, fixed label if rendering throws), so `result` cannot reject. Caps (`maxConcurrentAgents` auto = `min(16, max(1, availableParallelism() - 2))`, `maxTotalAgents` 1000, `maxItemsPerCall` 4096) and timeouts are validated Config, not literals.
|
|
|
|
|
|
|
|
|
|
### The consumer (dsh-tool-workflow)
|
|
|
|
|
|
|
|
|
|
A `workflow` tool mirroring `dsh-tool-subagent`'s synchronous shape: start, await, `try/finally` dispose, abort-bridge `exec.signal`, non-`completed` → `isError`. Render intent: a `generic` card titled by a textual `meta.name` sniff (presentation is a pure function of args). The tool description IS the model-facing authoring spec. The usage policy ships with the tool as its own `tool:<toolName>` prompt section (explicit-ask-only guidance — tool guidance lives in tool plugins, never in the deployment persona); the harness has no ultracode-style effort gate.
|
|
|
|
|
A `workflow` tool mirroring `dsh-tool-subagent`'s synchronous shape: start, await, `try/finally` dispose, abort-bridge `exec.signal`, non-`completed` → `isError`. Render intent: a `generic` card titled by the call's `meta.name` parameter (presentation is a pure function of args). The tool description IS the model-facing authoring spec. The usage policy ships with the tool as its own `tool:<toolName>` prompt section (explicit-ask-only guidance — tool guidance lives in tool plugins, never in the deployment persona); the harness has no ultracode-style effort gate.
|
|
|
|
|
|
|
|
|
|
### The foundation: structured output on the subagent seam
|
|
|
|
|
|
|
|
|
|
@@ -55,7 +55,7 @@ A `workflow` tool mirroring `dsh-tool-subagent`'s synchronous shape: start, awai
|
|
|
|
|
- **In-process `node:vm` execution** (the first cut of this RFC shipped it): mechanically simplest — no RPC, no thread — but `start()` blocks the caller for the script's initial synchronous slice, a synchronous spin past the first await cannot be killed in-process (the vm `timeout` covers only that first slice), and `dispose()` could only ABANDON an unsettling script, leaving the spin on the host loop. Superseded by the worker-thread engine, which keeps the same vm-context script surface while unblocking the host and making termination real.
|
|
|
|
|
- **Background execution as the default** (CC's shape): deferred; foreground-synchronous matches `dsh-tool-subagent`'s cut, and background semantics should be designed ONCE across bash/subagent/workflow rather than per-tool.
|
|
|
|
|
- **Workflow-layer JSON parsing for `agent({schema})`**: duplicating a seam concern at one consumer while the seam's capability flag stayed dishonestly `false`.
|
|
|
|
|
- **Meta as tool parameters instead of `export const meta`**: zero parsing, but scripts stop being self-contained artifacts and CC-authored scripts stop being drop-in.
|
|
|
|
|
- **Meta embedded in the script as `export const meta = {...}`** (CC's exact format; the first cut shipped it): keeps scripts self-contained and CC scripts drop-in, but obtaining meta means evaluating model-written text on the HOST — the shipped extractor ran the literal in an empty timed vm context, yet reading the RESULT still executed script-controlled getters on the host stack outside any timeout, re-opening the host-spin hole the worker thread exists to close. A JSON parameter deletes the scanner, the evaluation, and the hole outright; the cost is that a CC script's meta header must move into the parameter (the body stays drop-in).
|
|
|
|
|
- **`SchemaSpec` as the outputSchema type**: the author-facing DSL cannot express what arrives as data and cannot be validated against without conversion loss.
|
|
|
|
|
- **A schema-object library (zod, or the repo's schemastery) for the structured-output subset**: the schema is wire data — plain JSON that crosses the vm realm boundary in `agent({schema})` and lands verbatim in the forced tool's parameters — exactly where live schema objects cannot sit; consuming raw JSON Schema at runtime would need a third-party converter on top (zod core only emits JSON Schema, not the reverse), and it would put a second schema language beside schemastery's config role.
|
|
|
|
|
- **ajv for value validation**: it validates FULL JSON Schema, so the subset gate — the module's actual point, since every accepted keyword must be one the harness enforces — would remain hand-written regardless; it compiles validators through `new Function`; and it would be dsh-tools' first runtime dependency, all to replace the ~70-line value walker while the path-qualified, every-violation error reporting stays custom either way.
|
|
|
|
|
|