diff --git a/examples/desktop/README.md b/examples/desktop/README.md index ed57191b02..00d451823e 100644 --- a/examples/desktop/README.md +++ b/examples/desktop/README.md @@ -364,16 +364,33 @@ becomes a visualiser of the runtime's actual state. that exact seq, edit the user turn, and let a different reply stream in. The child is anchored back in the session tree with a `⑂ forks from here (N)` card so the branching is legible. +- **Workflow cards — live.** When a composition mounts the workflow + engine, the runtime bridges its `workflow/*` Cordis lifecycle events + onto the wire as `workflow.event` notifications. The shell folds those + incremental frames (start / phase / log / agent-start / agent-end / + end, keyed by `runId`) into a live workflow card anchored to the + `workflow` tool call that spawned the run — each workflow-agent shows + as a step, `running` then `done`/`failed` as it settles, under a + `live · workflow.event` chip. On profiles/runtimes that never mount + `ctx.workflows` the notification simply never fires, so nothing + renders. (The Debug popover still mints the five fixture shapes + — seq / fan-out / dag / iter / branch — carrying a `mock` chip.) - **`{ }` Inspector — zero loss.** Every inspectable element in the chat stream — user + assistant bubbles, reasoning blocks, tool calls and results, compaction cards, context 📎 injections, subagent cards — carries an unobtrusive `{ }` badge that opens one unified - right-side Inspector with three tabs: **Pretty** (a readable, + right-side Inspector with four tabs: **Pretty** (a readable, type-specific view of that element), **Raw** (the verbatim `session.event` from the session log, with a seq/type/time header + - copy), and **JSON** (the same event through the recursive collapsible - Fields tree). The pretty renderer is a projection; the Raw/JSON tabs - are the source of truth, always one click away. + copy), **JSON** (the same event through the recursive collapsible + Fields tree), and **Feedback** (a per-event RL annotation: thumbs + up/down + a note + an optional rubric dimension drawn from the same + locked dimension set the Evals rubrics use). Annotated events show a + small ✓ on their `{ }` badge; the record — + `{sessionId, seq, verdict, note, rubricDim?, at}` — persists to + `~/.dsh-desktop/feedback-annotations.json` as the RL-annotation seed. + The pretty renderer is a projection; the Raw/JSON tabs are the source + of truth, always one click away. - **Compaction visualisation.** When the runtime emits a `session/compact` outcome, the compacted range renders as a collapsed banner inline in chat with the summary + token delta diff --git a/examples/desktop/docs/qa-wf-feedback/01-workflow-live.png b/examples/desktop/docs/qa-wf-feedback/01-workflow-live.png new file mode 100644 index 0000000000..e2a383de3c Binary files /dev/null and b/examples/desktop/docs/qa-wf-feedback/01-workflow-live.png differ diff --git a/examples/desktop/docs/qa-wf-feedback/02-feedback-tab.png b/examples/desktop/docs/qa-wf-feedback/02-feedback-tab.png new file mode 100644 index 0000000000..fd94998032 Binary files /dev/null and b/examples/desktop/docs/qa-wf-feedback/02-feedback-tab.png differ diff --git a/examples/desktop/docs/upstream-ledger.md b/examples/desktop/docs/upstream-ledger.md index 89e2845bf4..d14e589209 100644 --- a/examples/desktop/docs/upstream-ledger.md +++ b/examples/desktop/docs/upstream-ledger.md @@ -274,3 +274,91 @@ small wire addition that lets the workaround retire. **Reference.** PR #374 review comment by @ZiyaZhang — https://github.com/deepseek-harness/deepseek-harness/pull/374#issuecomment-5016306211 + +--- + +## L-5 Runtime should expose a mid-turn steer / context-inject RPC on the wire + +**Symptom.** The composer cannot send a "steer" while a turn is running. A +researcher watching the agent go down the wrong path mid-turn has no way to +nudge it ("actually, check the other file first") without waiting for the +turn to end. The receiving half of this feature already renders — steering +and context-injection events paint as 📎 cards in the stream and the context +rail — but there is no *send* path: the shell can start a turn +(`session/prompt`) and cancel a turn (`session/cancel`), and nothing in +between. + +**Root cause.** The kernel already has the seam; the wire never exposes it. + +- `packages/core/agent-loop/src/agent.ts:219` — `Agent.steer(content, + options)` exists: when a turn is running it accepts the message into the + inbox as a *steering* message (`steering: true`) rather than starting a new + turn. +- `packages/core/agent-loop/src/agent.ts:227` — `Agent.inject(content, + options)` exists: it appends a `context/message` event, turn-enclosed when + a turn is open (`agent.ts:235`) or wrapped in a one-shot turn when not + (`agent.ts:247`). This is the model-visible⟺logged injection primitive. +- `packages/ui/jsonrpc/src/protocol.ts:30-43` — the client→server `METHOD` + map exposes `session/prompt` (`:34`) and `session/cancel` (`:35`) but no + `session/steer` or `context/inject`. The one mid-turn channel that *does* + cross the wire goes the other direction: `HOST_METHOD.sessionInterrupt` + (`protocol.ts:57`) is server→client. There is no client→server verb that + reaches `agent.steer` / `agent.inject`. + +So the runtime can already *do* the thing (the kernel method is shipped and +tested — see `packages/core/agent-loop/tests/agent.spec.ts:196`, a balanced +`turn/start · context/message · turn/end`); the wire just never gave the +shell a way to *ask* for it. + +**Local workaround.** Message queue (merged, lane-msg-queue): a mid-turn +Enter doesn't drop the text and doesn't error on the wire's one-in-flight- +prompt rule — it parks the text in a per-session FIFO +(`src/renderer/msg-queue-model.js`) and auto-sends the head as a fresh +`session/prompt` when the current turn ends +(`src/renderer/renderer.js:107` + `drainMsgQueueOnce`). So a steer degrades +to a *queued next-turn message*: the intent survives, but it lands after the +turn instead of redirecting it mid-flight, and it arrives as an ordinary +user prompt rather than a `steering: true` inbox message. + +Spot the workaround in code review by `src/renderer/msg-queue-model.js` (the +whole file) and the `drainMsgQueueOnce` / enqueue-on-inflight path in +`renderer.js`. The receiving-side render that's already waiting for the real +feature is `src/renderer/context-rail.js:38` (`context/message`) and +`:46` (`steering/message`) — the 📎 cards light up today for injects the +runtime emits on its own; they'd light up for shell-originated steers the +moment the send path exists. + +**Upstream fix (needed).** Add a client→server RPC that rides the existing +kernel seam, e.g.: + +``` +Method: session/steer (or context/inject) +Params: { + sessionId: string, + content: ContentBlock[], // the steer/injection payload + mode?: 'steer' | 'inject', // steer → agent.steer (running turn only); + // inject → agent.inject (context/message) + source?: { kind, ... } // provenance, same shape context/message carries +} +Result: { + accepted: boolean, // false if the session id was unknown / disposed + applied: 'steered' | 'injected' | 'queued' +} +``` + +The handler in `HarnessSdkServer` routes to `agent.steer(content, options)` +when a turn is running and `agent.inject(content, options)` otherwise — both +already exist (`agent.ts:219,227`). The load-bearing invariant to preserve is +**model-visible ⟺ logged**: whatever the model sees mid-turn must appear on +the session event stream as a `context/message` / `steering/message` (which +`agent.inject` already guarantees via `session.append`), so replay and the +📎 cards stay faithful — no side-channel that reaches the model without a +log record, and no log record the model never saw. + +**Precedent in ledger.** Same shape as L-1 / L-2 / L-4: the shell paints +over a gap the runtime should own (here, the receiving half already renders; +only the send verb is missing), and the fix is a small wire addition that +lets the queue workaround retire — once `session/steer` lands, a mid-turn +Enter can route to it instead of the FIFO, and queued-message degradation +becomes the fallback for older runtimes only. + diff --git a/examples/desktop/scripts/qa-cdp-shoot-wf-feedback.mjs b/examples/desktop/scripts/qa-cdp-shoot-wf-feedback.mjs new file mode 100644 index 0000000000..f158d0aa62 --- /dev/null +++ b/examples/desktop/scripts/qa-cdp-shoot-wf-feedback.mjs @@ -0,0 +1,315 @@ +// QA verification script for lane-wf-feedback. Boots an isolated Electron on a +// private CDP port (≥9330 to dodge the live demo instance + sibling lanes), +// then captures two proof shots: +// +// 01-workflow-live A live workflow card, minted by feeding the REAL wire +// shape (workflow.event frames from runtime commit +// dd29d8631) through the __dshOnWorkflowEvent seam — the +// same onWorkflowEvent path the live notification uses. +// The card wears the "live · workflow.event" chip. +// 02-feedback-tab The inspector's 4th tab (Feedback) with a filled +// annotation (verdict + note + rubric dim) and the ✓ +// marker painted on the source event's { } badge. +// +// Isolation follows the 2026-07-18 postmortem (qa-cdp-shoot-p0-inspector.mjs): +// 1. --user-data-dir= isolates Chromium userdata. +// 2. DSH_DESKTOP_HOME= isolates the main-process config root so we +// never write into ~/.dsh-desktop (incl. the feedback-annotations.json +// this lane writes). +// 3. Own CDP port (≥9330) so we never attach to the user's live 9223. +// Electron binary comes from the PARENT repo (dsh-desktop-demo) per the brief. + +import { spawn } from 'node:child_process' +import { existsSync, mkdirSync, writeFileSync, rmSync, statSync, readFileSync } from 'node:fs' +import { resolve, join } from 'node:path' +import { setTimeout as sleep } from 'node:timers/promises' +import { tmpdir } from 'node:os' + +const WORKTREE = resolve(process.env.DSH_WORKTREE || process.cwd()) +const PARENT = resolve(process.env.DSH_REPO || '/Users/ziya/harness/dsh-desktop-demo') +const ELECTRON = join(PARENT, 'node_modules/.bin/electron') +const CDP_PORT = Number(process.env.DSH_WF_FEEDBACK_PORT || 9331) +const OUTDIR = join(WORKTREE, 'docs/qa-wf-feedback') + +if (!existsSync(ELECTRON)) { + console.error(`electron binary not found at ${ELECTRON}`) + process.exit(2) +} +mkdirSync(OUTDIR, { recursive: true }) + +function seedHome(dshHome) { + const seedOverlay = [ + '# QA wf-feedback-shoot seed overlay (tmp, per-run).', + 'plugins:', + ' - "@cordisjs/plugin-include":', + ` path: ${join(WORKTREE, 'config/daemon-echo.yml')}`, + '', + ].join('\n') + writeFileSync(join(dshHome, 'user-overlay.cordis.yml'), seedOverlay) + writeFileSync(join(dshHome, 'config.json'), JSON.stringify({ role: 'coding', approvalMode: 'never' }, null, 2)) + writeFileSync(join(dshHome, '.onboarded'), new Date().toISOString()) +} + +async function bootElectron(dshHome, userData, port) { + const child = spawn(ELECTRON, [ + `--remote-debugging-port=${port}`, + `--user-data-dir=${userData}`, + '--disable-gpu', + '--no-sandbox', + '.', + ], { + cwd: WORKTREE, + env: { + ...process.env, + DSH_DESKTOP_HOME: dshHome, + DSH_MAXIMIZE: '1', + DSH_QA: '1', // exposes window.__dshOnSessionEvent + __dshOnWorkflowEvent + }, + stdio: ['ignore', 'pipe', 'pipe'], + }) + const logs = [] + child.stdout.on('data', (d) => logs.push(String(d))) + child.stderr.on('data', (d) => logs.push(String(d))) + for (let i = 0; i < 40; i++) { + await sleep(500) + try { + const r = await fetch(`http://localhost:${port}/json/list`) + if (r.ok) return { child, logs } + } catch {} + } + child.kill('SIGKILL') + console.error('electron CDP did not come up in 20s. logs:\n' + logs.join('')) + process.exit(3) +} + +async function newCdp(port) { + const targets = await (await fetch(`http://localhost:${port}/json/list`)).json() + const target = targets.find((t) => t.type === 'page') + if (!target) throw new Error('no page target on port ' + port) + const ws = new WebSocket(target.webSocketDebuggerUrl) + await new Promise((ok, err) => { ws.onopen = ok; ws.onerror = (e) => err(e) }) + let id = 1 + const pending = new Map() + ws.onmessage = (ev) => { + const data = typeof ev.data === 'string' ? ev.data : String(ev.data) + let msg; try { msg = JSON.parse(data) } catch { return } + if (msg.id != null && pending.has(msg.id)) { + const [ok, err] = pending.get(msg.id); pending.delete(msg.id) + if (msg.error) err(new Error(msg.error.message)); else ok(msg.result) + } + } + const call = (m, p = {}, ms = 20000) => new Promise((ok, err) => { + const _id = id++ + const t = setTimeout(() => { pending.delete(_id); err(new Error('cdp timeout: ' + m)) }, ms) + pending.set(_id, [(v) => { clearTimeout(t); ok(v) }, (e) => { clearTimeout(t); err(e) }]) + ws.send(JSON.stringify({ id: _id, method: m, params: p })) + }) + const evj = async (expr) => { + const r = await call('Runtime.evaluate', { + expression: `(async()=>{try{return (${expr})}catch(e){return {__err:String(e)}}})()`, + returnByValue: true, awaitPromise: true, + }) + if (r.exceptionDetails) throw new Error(r.exceptionDetails.exception?.description || r.exceptionDetails.text) + return r.result?.value + } + return { ws, call, evj } +} + +const SESSION_ID = 'qa-wf-feedback-sess' + +// A minimal turn with a workflow tool/call so the live card has an anchor, +// plus the assistant bubble we'll annotate. +const TURN_EVENTS = [ + { type: 'turn/start', seq: 1, data: { turnId: 't0', trigger: 'user' } }, + { type: 'user/message', seq: 2, data: { text: 'Audit the three fs packages.' } }, + { type: 'tool/call', seq: 3, data: { callId: 'call-wf', name: 'workflow', arguments: { name: 'audit-flow', kind: 'seq' } } }, + { type: 'assistant/message', seq: 4, data: { content: [{ type: 'text', text: 'Kicking off the audit workflow across the three packages.' }], usage: { inputTokens: 210, outputTokens: 24, totalTokens: 234 } } }, + { type: 'turn/end', seq: 5, data: { turnId: 't0' } }, +] + +// The REAL workflow.event wire shape (runtime commit dd29d8631): incremental +// frames keyed by runId, meta{name,description}, per-kind payload. Named the +// same as the tool/call args so onWorkflowEvent anchors the card to the block. +const RUN_ID = 'run-audit-1' +const WF_META = { name: 'audit-flow', description: 'audit three fs packages' } +const WF_FRAMES = [ + { kind: 'workflow/start', runId: RUN_ID, meta: WF_META }, + { kind: 'workflow/phase', runId: RUN_ID, meta: WF_META, payload: 'Scan' }, + { kind: 'workflow/log', runId: RUN_ID, meta: WF_META, payload: 'starting with 3 packages' }, + { kind: 'workflow/agent-start', runId: RUN_ID, meta: WF_META, payload: { seq: 1, label: 'audit dsh-fs', phase: 'Scan', childId: 'c1' } }, + { kind: 'workflow/agent-end', runId: RUN_ID, meta: WF_META, payload: { seq: 1, label: 'audit dsh-fs', phase: 'Scan', childId: 'c1', outcome: 'completed' } }, + { kind: 'workflow/agent-start', runId: RUN_ID, meta: WF_META, payload: { seq: 2, label: 'audit dsh-bash', phase: 'Scan', childId: 'c2' } }, + { kind: 'workflow/agent-end', runId: RUN_ID, meta: WF_META, payload: { seq: 2, label: 'audit dsh-bash', phase: 'Scan', childId: 'c2', outcome: 'failed' } }, + { kind: 'workflow/agent-start', runId: RUN_ID, meta: WF_META, payload: { seq: 3, label: 'audit dsh-web', phase: 'Scan', childId: 'c3' } }, +] + +async function injectTurn(evj) { + return evj(` + (async () => { + const dispatch = window.__dshOnSessionEvent + const wf = window.__dshOnWorkflowEvent + if (typeof dispatch !== 'function') return { err: 'no __dshOnSessionEvent seam (DSH_QA=1?)' } + if (typeof wf !== 'function') return { err: 'no __dshOnWorkflowEvent seam (DSH_QA=1?)' } + if (window.__dshTabs && typeof window.__dshTabs.switchTo === 'function') window.__dshTabs.switchTo('chat') + // Clear the daemon-echo boot fixtures so our turn + live card is the only + // content on the stream (auto-follow otherwise re-pins to the boot noise + // and pushes our card below the fold — see qa-cdp-shoot-p0-inspector.mjs + // note on the flex/min-height:0 container). + const stream = document.getElementById('stream') + if (stream) stream.innerHTML = '' + for (const ev of ${JSON.stringify(TURN_EVENTS)}) dispatch('${SESSION_ID}', ev) + // Feed the REAL wire frames through the live path. + for (const f of ${JSON.stringify(WF_FRAMES)}) wf(f) + return { + ok: true, + liveCards: document.querySelectorAll('.workflow-card-live').length, + liveChip: document.querySelectorAll('.workflow-card-chip--live').length, + steps: document.querySelectorAll('.workflow-card-live .workflow-step').length, + toolBlocks: document.querySelectorAll('.tool-block[data-tool-name="workflow"]').length, + } + })() + `) +} + +async function screenshot(call, outName) { + let shot = null + let lastErr = null + const attempts = [ + { format: 'png' }, { format: 'png' }, + { format: 'jpeg', quality: 82 }, { format: 'jpeg', quality: 82 }, + ] + let usedFormat = 'png' + for (const a of attempts) { + try { + shot = await call('Page.captureScreenshot', { ...a, captureBeyondViewport: false }, 25000) + if (shot && shot.data) { usedFormat = a.format; break } + } catch (e) { lastErr = e; await new Promise((r) => setTimeout(r, 1200)) } + } + if (!shot || !shot.data) throw new Error('captureScreenshot failed for ' + outName + (lastErr ? ': ' + lastErr.message : '')) + const finalName = usedFormat === 'jpeg' ? outName.replace(/\.png$/, '.jpg') : outName + const outPath = join(OUTDIR, finalName) + writeFileSync(outPath, Buffer.from(shot.data, 'base64')) + const kb = Math.round(statSync(outPath).size / 1024) + console.log(` wrote ${outPath} (${kb} KB, ${usedFormat})`) + if (kb < 20) console.warn(` ⚠ ${outName} is only ${kb} KB (<20 KB) — likely a blank/trivial frame`) + return { path: outPath, kb } +} + +async function main() { + const dshHome = join(tmpdir(), 'dsh-wf-feedback-home') + const userData = join(tmpdir(), 'dsh-wf-feedback-userdata') + for (const dir of [dshHome, userData]) { + try { rmSync(dir, { recursive: true, force: true }) } catch {} + mkdirSync(dir, { recursive: true }) + } + seedHome(dshHome) + console.log(`[wf-feedback] booting on CDP port ${CDP_PORT}`) + const { child } = await bootElectron(dshHome, userData, CDP_PORT) + const results = [] + try { + await sleep(1500) + const { call, evj } = await newCdp(CDP_PORT) + await call('Page.enable') + await call('Runtime.enable') + + // Let the daemon-echo scripted boot session finish streaming before we + // inject — otherwise its late events re-append after our clear and + // auto-follow pins to them, burying our card. Poll the status dot for + // `idle` (bounded). + for (let i = 0; i < 16; i++) { + const st = await evj(`(document.querySelector('#status-text')||{}).textContent||''`) + if (String(st).toLowerCase().includes('idle')) break + await sleep(500) + } + await sleep(600) + + const injected = await injectTurn(evj) + console.log(' injected:', JSON.stringify(injected)) + if (!injected || !injected.ok) throw new Error('turn injection failed: ' + JSON.stringify(injected)) + if (!injected.liveCards) console.warn(' ⚠ no live workflow card minted — the seam may not be wired') + await sleep(400) + + // --- 01: the live workflow card (live chip + steps) --- + // Re-clear + re-inject atomically right before capture so the daemon-echo + // scripted session (which keeps re-rendering the stream) can't bury the + // card between inject and shot, then scroll it into view. + const scrolled = await evj(` + (() => { + const dispatch = window.__dshOnSessionEvent + const wf = window.__dshOnWorkflowEvent + const stream = document.getElementById('stream') + if (stream) stream.innerHTML = '' + for (const ev of ${JSON.stringify(TURN_EVENTS)}) dispatch('${SESSION_ID}', ev) + for (const f of ${JSON.stringify(WF_FRAMES)}) wf(f) + const card = document.querySelector('.workflow-card-live') + if (!card) return { err: 'no live card to scroll to' } + if (card.scrollIntoView) card.scrollIntoView({ block: 'center' }) + const chip = card.querySelector('.workflow-card-chip--live') + return { chip: chip ? chip.textContent : null, steps: card.querySelectorAll('.workflow-step').length } + })() + `) + console.log(' 01 scroll:', JSON.stringify(scrolled)) + await sleep(150) + results.push(await screenshot(call, '01-workflow-live.png')) + + // --- 02: inspector Feedback tab, filled annotation + ✓ marker --- + const fb = await evj(` + (async () => { + const ins = window.__dshInspector + if (!ins) return { err: 'no __dshInspector' } + // Open the inspector on the assistant event, Feedback tab. + const ev = { type: 'assistant/message', seq: 4, time: Date.now(), + data: { content: [{ type: 'text', text: 'Kicking off the audit workflow across the three packages.' }] } } + ins.open({ event: ev, tab: 'feedback', sessionId: '${SESSION_ID}' }) + const drawer = document.getElementById('inspector-drawer') + const panel = drawer.querySelector('.inspector-panel[data-panel="feedback"]') + // Fill: thumbs-up, a note, a rubric dim, then Save. + const up = panel.querySelector('[aria-label="Thumbs up"]') + if (up) up.click() + const note = panel.querySelector('.inspector-feedback-note-input') + if (note) note.value = 'Good — kicked off the right workflow for the audit.' + const sel = panel.querySelector('.inspector-feedback-dim-select') + if (sel && sel.options && sel.options.length > 1) sel.value = sel.options[1].value + const save = panel.querySelector('.inspector-feedback-save') + if (save) save.click() + await new Promise(r => setTimeout(r, 350)) + // Attach a badge on a stream bubble so the ✓ marker is visible, then + // refresh markers so the annotated (sessionId, seq) paints. + if (typeof ins.refreshInspectMarkers === 'function') ins.refreshInspectMarkers() + return { + open: drawer.classList.contains('open'), + verdictActive: !!panel.querySelector('.inspector-feedback-thumb.active'), + status: (panel.querySelector('.inspector-feedback-status') || {}).textContent, + markedBadges: document.querySelectorAll('.inspect-badge-annotated').length, + } + })() + `) + console.log(' 02 feedback:', JSON.stringify(fb)) + if (!fb || fb.open !== true) console.warn(' ⚠ Feedback tab did not open:', JSON.stringify(fb)) + await sleep(250) + results.push(await screenshot(call, '02-feedback-tab.png')) + + // Confirm the annotation actually persisted to the isolated home file. + let persisted = null + try { + const p = join(dshHome, 'feedback-annotations.json') + persisted = existsSync(p) ? JSON.parse(readFileSync(p, 'utf8')) : null + } catch (e) { persisted = { __err: String(e) } } + console.log(' persisted annotations:', JSON.stringify(persisted)) + + console.log('\n--- SUMMARY ---') + console.log('inject :', JSON.stringify(injected)) + console.log('02 :', JSON.stringify(fb)) + console.log('persist:', JSON.stringify(persisted)) + for (const r of results) console.log(`shot : ${r.path} (${r.kb} KB)`) + } finally { + try { child.kill('SIGKILL') } catch {} + for (let i = 0; i < 6; i++) { + await sleep(500) + try { await fetch(`http://localhost:${CDP_PORT}/json/list`) } catch { break } + } + } + process.exit(0) +} + +main().catch((err) => { console.error(err); process.exit(1) }) diff --git a/examples/desktop/src/main/feedback-annotations.js b/examples/desktop/src/main/feedback-annotations.js new file mode 100644 index 0000000000..d73e373737 --- /dev/null +++ b/examples/desktop/src/main/feedback-annotations.js @@ -0,0 +1,131 @@ +// feedback-annotations.js — per-event RL-annotation store (lane-wf-feedback). +// +// The inspector's Feedback tab lets a researcher annotate any single session +// event: thumbs up/down + a free-text note + an optional rubric dimension. +// This is the RL-annotation seed — the record shape is deliberately forward- +// compatible so a later trajectory-GRM pipeline can consume it as-is: +// +// { sessionId, seq, verdict, note, rubricDim?, at } +// +// Why a new side file rather than growth-v2? growth-v2 stores rubrics/errors +// keyed by COMPACT WINDOW (one file per window). Event-level annotations are +// keyed by (sessionId, seq) — a different grain that doesn't fit that shape. +// So this mirrors the growth-v2 IPC style (main-process module + preload +// namespace) but lands in its own append-safe JSON file under ~/.dsh-desktop, +// next to growth-log.jsonl / user-overlay.cordis.yml. +// +// Layout: `~/.dsh-desktop/feedback-annotations.json` — a single flat array of +// records. One file (not per-session) because the demo scale is small and a +// single file keeps the "export the whole RL seed" story a one-liner. Writes +// rewrite the whole file (no partial-write tearing). +// +// Upsert semantics: (sessionId, seq) is the identity. Re-annotating the same +// event overwrites its record (verdict/note/rubricDim), refreshing `at`. A +// verdict of null with an empty note clears the annotation (delete). + +'use strict' + +const fs = require('fs') +const path = require('path') +const os = require('os') + +function shellHome() { + return process.env.DSH_DESKTOP_HOME || path.join(os.homedir(), '.dsh-desktop') +} + +function annotationsPath() { + return path.join(shellHome(), 'feedback-annotations.json') +} + +const VALID_VERDICTS = new Set(['up', 'down']) + +function readAllRaw() { + try { + const text = fs.readFileSync(annotationsPath(), 'utf8') + const v = JSON.parse(text) + return Array.isArray(v) ? v : [] + } catch (err) { + if (err && err.code === 'ENOENT') return [] + console.debug(`feedback-annotations read failed: ${err.message}`) + return [] + } +} + +function writeAll(arr) { + const p = annotationsPath() + fs.mkdirSync(path.dirname(p), { recursive: true }) + fs.writeFileSync(p, JSON.stringify(arr, null, 2), 'utf8') +} + +// Return every annotation. The renderer indexes them by (sessionId, seq) to +// paint the ✓ marker + prefill the Feedback tab, so the wire is a flat list. +function list() { + return { ok: true, entries: readAllRaw() } +} + +// Normalize + validate one incoming annotation form. Returns a clean record +// or null when it isn't a usable annotation (no verdict AND no note → clear). +function normalize(form) { + if (!form || typeof form !== 'object') return null + const sessionId = String(form.sessionId || '').trim() + const seq = Number(form.seq) + if (!sessionId || !Number.isFinite(seq)) return null + const verdict = VALID_VERDICTS.has(form.verdict) ? form.verdict : null + const note = typeof form.note === 'string' ? form.note.trim() : '' + const rubricDim = (typeof form.rubricDim === 'string' && form.rubricDim.trim()) + ? form.rubricDim.trim() + : undefined + // No verdict and no note → nothing to store (treated as a clear upstream). + if (verdict === null && !note) return null + const rec = { sessionId, seq, verdict, note, at: Date.now() } + if (rubricDim) rec.rubricDim = rubricDim + return rec +} + +// Upsert one annotation keyed by (sessionId, seq). Returns { ok, entry } on a +// write, { ok:true, cleared:true } when the form clears an existing record, +// or { ok:false, reason } on a malformed form. +function upsert(form) { + const sessionId = form && String(form.sessionId || '').trim() + const seq = form && Number(form.seq) + if (!sessionId || !Number.isFinite(seq)) return { ok: false, reason: 'sessionId-and-seq-required' } + const arr = readAllRaw() + const idx = arr.findIndex((r) => r && r.sessionId === sessionId && Number(r.seq) === seq) + const rec = normalize(form) + if (rec === null) { + // Clear: drop any existing record for this (sessionId, seq). + if (idx >= 0) { + arr.splice(idx, 1) + writeAll(arr) + return { ok: true, cleared: true } + } + return { ok: false, reason: 'empty-annotation' } + } + if (idx >= 0) arr[idx] = rec + else arr.push(rec) + writeAll(arr) + return { ok: true, entry: rec } +} + +// Remove one annotation. Returns { ok, removed:boolean }. +function remove(form) { + const sessionId = form && String(form.sessionId || '').trim() + const seq = form && Number(form.seq) + if (!sessionId || !Number.isFinite(seq)) return { ok: false, reason: 'sessionId-and-seq-required' } + const arr = readAllRaw() + const idx = arr.findIndex((r) => r && r.sessionId === sessionId && Number(r.seq) === seq) + if (idx < 0) return { ok: true, removed: false } + arr.splice(idx, 1) + writeAll(arr) + return { ok: true, removed: true } +} + +module.exports = { + shellHome, + annotationsPath, + list, + upsert, + remove, + normalize, + VALID_VERDICTS, +} diff --git a/examples/desktop/src/main/main.js b/examples/desktop/src/main/main.js index 07786e5be0..bdcbe1c9b2 100644 --- a/examples/desktop/src/main/main.js +++ b/examples/desktop/src/main/main.js @@ -18,6 +18,7 @@ const M = require('./plugin-market.js') const GH = require('./gh-prs.js') const Growth = require('./growth-log.js') const GrowthV2 = require('./growth-v2.js') +const FeedbackAnnotations = require('./feedback-annotations.js') const { normalizeInterruptRequest } = require('./interrupt-normalize.js') const { classifyForkErrorMessage } = require('./fork-error-classify.js') const { revealWindow } = require('./window-reveal.js') @@ -830,6 +831,14 @@ app.whenReady().then(async () => { ipcMain.handle('growth:v2AddRubric', (_e, { compactWindowId, form } = {}) => GrowthV2.addRubric(compactWindowId, form)) ipcMain.handle('growth:v2AddError', (_e, { compactWindowId, form } = {}) => GrowthV2.addError(compactWindowId, form)) + // lane-wf-feedback: per-event RL-annotation store (inspector Feedback tab). + // Records land in ~/.dsh-desktop/feedback-annotations.json as a flat list of + // { sessionId, seq, verdict, note, rubricDim?, at }. Read on drawer open to + // prefill + paint the ✓ marker; upsert keyed by (sessionId, seq). + ipcMain.handle('feedback:list', () => FeedbackAnnotations.list()) + ipcMain.handle('feedback:upsert', (_e, { form } = {}) => FeedbackAnnotations.upsert(form)) + ipcMain.handle('feedback:remove', (_e, { form } = {}) => FeedbackAnnotations.remove(form)) + // --------------------------------------------------------------------- // Hub page (#186 + #190). // diff --git a/examples/desktop/src/preload/preload.js b/examples/desktop/src/preload/preload.js index 8b485e487c..0427b98519 100644 --- a/examples/desktop/src/preload/preload.js +++ b/examples/desktop/src/preload/preload.js @@ -164,6 +164,15 @@ contextBridge.exposeInMainWorld('dsh', { v2AddRubric: (compactWindowId, form) => ipcRenderer.invoke('growth:v2AddRubric', { compactWindowId, form }), v2AddError: (compactWindowId, form) => ipcRenderer.invoke('growth:v2AddError', { compactWindowId, form }), }, + // -- feedback annotations (inspector Feedback tab / RL seed) ---------------- + // Per-event annotation store at ~/.dsh-desktop/feedback-annotations.json. + // list() returns { ok, entries:[{sessionId,seq,verdict,note,rubricDim?,at}] }; + // upsert(form)/remove(form) key on (sessionId, seq) and return { ok, ... }. + feedback: { + list: () => ipcRenderer.invoke('feedback:list'), + upsert: (form) => ipcRenderer.invoke('feedback:upsert', { form }), + remove: (form) => ipcRenderer.invoke('feedback:remove', { form }), + }, // -- pull requests page ---------------------------------------------------- // `list` returns the 60s-cached payload; `refresh` bypasses the cache. // Both go through the same normalized shape: { rows, repo, source, viewer, diff --git a/examples/desktop/src/renderer/feedback-annotation-model.js b/examples/desktop/src/renderer/feedback-annotation-model.js new file mode 100644 index 0000000000..cd9984ff30 --- /dev/null +++ b/examples/desktop/src/renderer/feedback-annotation-model.js @@ -0,0 +1,132 @@ +// feedback-annotation-model.js — pure model for the inspector Feedback tab +// (lane-wf-feedback). The RL-annotation seed. +// +// The Feedback tab annotates a single session event: thumbs up/down + a free- +// text note + an optional rubric dimension. The persisted record is keyed by +// (sessionId, seq) and shaped for a downstream trajectory-GRM pipeline: +// +// { sessionId, seq, verdict: 'up'|'down'|null, note: string, +// rubricDim?: string, at: number } +// +// This module is the DATA + PURE HELPERS only — no DOM, no IPC. The renderer +// owns "read via window.dsh.feedback on open, write on submit"; this file: +// - derives the identity key from an inspector event, +// - normalizes a form into the forward-compatible record shape, +// - maintains an in-memory index (sessionId,seq → record) so the ✓ marker +// and prefill are synchronous (no round-trip on every badge render). +// +// The main-process feedback-annotations.js is the source of truth on disk; +// this index is a renderer-side cache hydrated from feedback.list() at boot +// and kept in step on each upsert/clear. + +'use strict' + +const VALID_VERDICTS = ['up', 'down'] + +// Stable string key for the (sessionId, seq) identity. Kept as one function so +// the index writer and the marker reader can never drift. +function keyFor(sessionId, seq) { + const sid = String(sessionId == null ? '' : sessionId) + const s = Number(seq) + if (!sid || !Number.isFinite(s)) return null + return `${sid}::${s}` +} + +// Pull (sessionId, seq) out of an inspector event + its owning session. The +// inspector opens on a session.event that carries `seq`; the sessionId comes +// from the caller (the active/owning session), since a reconstructed event may +// not carry it. Returns { sessionId, seq } | null. +function identityFor(event, sessionId) { + if (!event || typeof event !== 'object') return null + const seq = Number(event.seq) + if (!Number.isFinite(seq)) return null + const sid = String(sessionId == null ? '' : sessionId) + if (!sid) return null + return { sessionId: sid, seq } +} + +// Normalize a raw form ({ sessionId, seq, verdict, note, rubricDim }) into the +// persisted record shape, or null when there's nothing worth storing (no +// verdict AND no note → a clear). Mirrors the main-process normalize() so the +// renderer's optimistic cache matches what lands on disk. +function normalize(form) { + if (!form || typeof form !== 'object') return null + const id = identityFor({ seq: form.seq }, form.sessionId) + if (!id) return null + const verdict = VALID_VERDICTS.indexOf(form.verdict) >= 0 ? form.verdict : null + const note = typeof form.note === 'string' ? form.note.trim() : '' + const rubricDim = (typeof form.rubricDim === 'string' && form.rubricDim.trim()) + ? form.rubricDim.trim() + : undefined + if (verdict === null && !note) return null + const rec = { sessionId: id.sessionId, seq: id.seq, verdict, note } + if (rubricDim) rec.rubricDim = rubricDim + rec.at = Number.isFinite(form.at) ? form.at : Date.now() + return rec +} + +function createAnnotationIndex() { + /** @type {Map} */ + const byKey = new Map() + + // Hydrate from a flat list (feedback.list().entries). Ignores malformed + // records so a hand-edited file can't crash the marker pass. + function hydrate(entries) { + byKey.clear() + if (!Array.isArray(entries)) return + for (const e of entries) { + const k = keyFor(e && e.sessionId, e && e.seq) + if (k) byKey.set(k, e) + } + } + + // Look up the record for (sessionId, seq), or null. Drives the ✓ marker + + // Feedback-tab prefill. + function get(sessionId, seq) { + const k = keyFor(sessionId, seq) + return k ? (byKey.get(k) || null) : null + } + + function has(sessionId, seq) { + const k = keyFor(sessionId, seq) + return k ? byKey.has(k) : false + } + + // Apply a normalized record (or a clear) to the cache. Pass the FORM; a form + // that normalizes to null clears the (sessionId, seq) entry. Returns the + // stored record, or null on a clear/invalid. + function put(form) { + const id = identityFor({ seq: form && form.seq }, form && form.sessionId) + if (!id) return null + const k = keyFor(id.sessionId, id.seq) + const rec = normalize(form) + if (rec === null) { + byKey.delete(k) + return null + } + byKey.set(k, rec) + return rec + } + + function remove(sessionId, seq) { + const k = keyFor(sessionId, seq) + if (!k) return false + return byKey.delete(k) + } + + function size() { return byKey.size } + + function all() { return Array.from(byKey.values()) } + + return { hydrate, get, has, put, remove, size, all } +} + +const feedbackModelApi = { + VALID_VERDICTS, + keyFor, + identityFor, + normalize, + createAnnotationIndex, +} +if (typeof module !== 'undefined' && module.exports) module.exports = feedbackModelApi +if (typeof window !== 'undefined') window.__dshFeedbackModel = feedbackModelApi diff --git a/examples/desktop/src/renderer/index.html b/examples/desktop/src/renderer/index.html index f9a4eb0bae..662527bd96 100644 --- a/examples/desktop/src/renderer/index.html +++ b/examples/desktop/src/renderer/index.html @@ -1555,6 +1555,7 @@ + @@ -1566,7 +1567,8 @@ - + + @@ -1696,11 +1698,13 @@ +
+
diff --git a/examples/desktop/src/renderer/inspector-drawer.js b/examples/desktop/src/renderer/inspector-drawer.js index 8a939a1516..c23174ad21 100644 --- a/examples/desktop/src/renderer/inspector-drawer.js +++ b/examples/desktop/src/renderer/inspector-drawer.js @@ -32,7 +32,7 @@ ;(function () { const isBrowser = typeof window !== 'undefined' && typeof document !== 'undefined' - const TABS = ['pretty', 'raw', 'json'] + const TABS = ['pretty', 'raw', 'json', 'feedback'] // --- pure helpers ------------------------------------------------------ @@ -345,9 +345,186 @@ host.appendChild(pre) } + // The rubric dimension list the Feedback tab's dropdown offers. Reused from + // the SAME source Evals/rubrics-model.js uses (MULTI_TURN_DIMENSIONS) so a + // researcher annotates against the locked RL dimension vocabulary rather + // than an inspector-local copy. Falls back to an empty list pre-load. + function feedbackDimensions () { + const rm = (typeof window !== 'undefined') ? window.__dshRubricsModel : null + const dims = rm && Array.isArray(rm.MULTI_TURN_DIMENSIONS) ? rm.MULTI_TURN_DIMENSIONS : [] + return dims.map((d) => ({ id: d.id, label: d.label })) + } + + // Read the annotation index (renderer cache) for an event's (sessionId, seq) + // so the tab prefills + the marker paints without an IPC round-trip. Returns + // the stored record or null. + function currentAnnotation (sessionId, event) { + const fm = (typeof window !== 'undefined') ? window.__dshFeedbackModel : null + if (!fm || !fm.createAnnotationIndex) return null + const idx = feedbackIndex() + if (!idx) return null + const seq = event && Number(event.seq) + if (!Number.isFinite(seq) || !sessionId) return null + return idx.get(sessionId, seq) + } + + // Lazily-created singleton annotation index, hydrated from disk on first use. + let _idx = null + function feedbackIndex () { + const fm = (typeof window !== 'undefined') ? window.__dshFeedbackModel : null + if (!fm || !fm.createAnnotationIndex) return null + if (!_idx) _idx = fm.createAnnotationIndex() + return _idx + } + + // The session id the Feedback tab keys annotations to. Prefer the explicit + // `open({ sessionId })` value; fall back to the renderer's active session + // (the tool-card `{ }` path routes through openFromDrawer without a sessionId, + // so the active session is the honest owner of the event on screen). + function effectiveSessionId () { + if (state.sessionId != null && state.sessionId !== '') return state.sessionId + const chat = (typeof window !== 'undefined') ? window.__dshChat : null + if (chat && typeof chat.getActiveSessionId === 'function') { + const sid = chat.getActiveSessionId() + if (sid != null && sid !== '') return String(sid) + } + return null + } + + // Hydrate the annotation index from persisted records (feedback.list()). + // Called at boot by the renderer; safe to call repeatedly. + function hydrateFeedback (entries) { + const idx = feedbackIndex() + if (idx) idx.hydrate(entries) + } + + // Feedback tab renderer (doc-injected for tests). Builds the up/down verdict + // buttons, the rubric-dimension