feat(desktop): DSH Electron desktop shell — harness internals visualized
Minimal Electron shell over the DSH JSON-RPC runtime — a first-look at what a ChatGPT.app-style host on top of the DeepSeek Harness looks like, with the harness's normally-invisible internals (trace timeline, context surface, subagent tree, compaction, plugin registry, rubrics) brought forward as first-class UI surfaces so plugin authors and researchers can see what the agent is actually doing. Runs against three keyless-to-live profiles (stdio-echo works on master out of the box; daemon-echo / daemon-vibe-echo activate once the daemon-demo lands; stdio-deepseek and daemon-vibe hit the real DeepSeek API when you supply a key). HARNESS_DEV auto-resolves to the in-repo runtime when this shell ships under examples/desktop/, so a fresh clone launches without config; env DSH_DEV_ROOT overrides for custom layouts, and a sibling deepseek-harness-dev/ checkout is the original dev workflow. Cold-clone gate (P0 fixes for first-time-clone usability): - HARNESS_DEV: 3-candidate resolver (env → walk-up in-repo marker → sibling), unit-tested via mock fs so ordering is locked without needing either real layout on disk. - config yml leaves rewritten at assemble time so the sibling-clone paths (../../deepseek-harness-dev/examples/echo-agent/…) become the in-repo paths (../../echo-agent/…) in the released tree — source yml stays usable for local dev, released tree ships a working shape. - pnpm-workspace.yaml allowBuilds.electron = true (was placeholder). - missing-key card in stdio-deepseek offers a one-click switch to stdio-echo (the keyless profile that works on master) rather than daemon-echo (blocked on the not-yet-shipped daemon-demo). - assemble-oss-release.sh rewrites the source-side breadcrumb name 'dsh-desktop-demo' → 'dsh-desktop' for the released package.json. FOUC guard on the onboarding gate (41fc5df carried) keeps the first-launch splash from flashing before the runtime probe finishes. Test suite (1634 tests in source, 3990 in the runtime repo) covers resolver ordering, renderer classifiers, trace timeline shape, compaction diff rendering, rubric parity, and the missing-key onboarding paths.
This commit is contained in:
22
examples/desktop/fixtures/rubrics/bug-fix.md
Normal file
22
examples/desktop/fixtures/rubrics/bug-fix.md
Normal file
@@ -0,0 +1,22 @@
|
||||
---
|
||||
name: bug-fix
|
||||
group: fix-optimize
|
||||
template: fixed
|
||||
executor: llm-judge
|
||||
version: v1
|
||||
description: Evaluate a bug-fix trajectory — reproduction, minimality, tests, regressions, explanation.
|
||||
---
|
||||
|
||||
## Checklist
|
||||
|
||||
- Reproduces the reported failure with a minimal case before patching
|
||||
- Patch is minimal: no unrelated cleanups, formatting, or refactors
|
||||
- Regression test added or existing test updated to cover the fix
|
||||
- No obvious regressions in adjacent behavior (spot-check callers)
|
||||
- Written explanation states the root cause, not just the symptom
|
||||
|
||||
## Notes
|
||||
|
||||
- Prefer patches that add a regression test in the same PR.
|
||||
- Deduct when the model conflates symptom (crash) with cause (bad input).
|
||||
- Deduct when the model rewrites more than the failing branch.
|
||||
23
examples/desktop/fixtures/rubrics/code-review.md
Normal file
23
examples/desktop/fixtures/rubrics/code-review.md
Normal file
@@ -0,0 +1,23 @@
|
||||
---
|
||||
name: code-review
|
||||
group: se-process
|
||||
template: code-review
|
||||
executor: llm-judge
|
||||
version: v1
|
||||
description: Reviewer rubric — evaluates a code-review trajectory on style, correctness, and test coverage.
|
||||
---
|
||||
|
||||
## Checklist
|
||||
|
||||
- Identifies incorrect behavior (bugs, edge cases) present in the diff
|
||||
- Style feedback is specific to project conventions, not generic
|
||||
- Suggests test coverage where the diff adds untested code paths
|
||||
- Prioritizes correctness over style when both are present
|
||||
- Tone is direct without being hostile; asks questions when uncertain
|
||||
|
||||
## Notes
|
||||
|
||||
- LLM-as-judge for now; the code-executor variant would run project
|
||||
linters and compare their output against the review comments.
|
||||
- Deduct when reviewer misses a real bug the diff introduces.
|
||||
- Deduct when reviewer flags a style-only issue as correctness-critical.
|
||||
25
examples/desktop/fixtures/rubrics/correctness-score.md
Normal file
25
examples/desktop/fixtures/rubrics/correctness-score.md
Normal file
@@ -0,0 +1,25 @@
|
||||
---
|
||||
name: correctness-score
|
||||
group: fix-optimize
|
||||
template: fixed
|
||||
executor: llm-judge
|
||||
version: v1
|
||||
description: Continuous 0-1 correctness score — real-valued continuous primitive.
|
||||
---
|
||||
|
||||
## Dimensions
|
||||
|
||||
- correctness :: continuous :: 0-1 :: Correctness
|
||||
|
||||
## Checklist
|
||||
|
||||
- Overall correctness on the trajectory's stated goal (0 = wrong, 1 = correct)
|
||||
|
||||
## Notes
|
||||
|
||||
- Continuous primitive: judge model emits a real number in [0, 1].
|
||||
- Export carries `dim_types.correctness = { type: 'continuous', min: 0, max: 1 }`
|
||||
so downstream consumers can treat this as a probability rather than a
|
||||
discrete class.
|
||||
- Small integer ranges (like 1-5) render as button rows; the 0-1 range
|
||||
renders as a numeric input for a real-valued score.
|
||||
24
examples/desktop/fixtures/rubrics/intent-triage.md
Normal file
24
examples/desktop/fixtures/rubrics/intent-triage.md
Normal file
@@ -0,0 +1,24 @@
|
||||
---
|
||||
name: intent-triage
|
||||
group: interaction-reasoning
|
||||
template: fixed
|
||||
executor: llm-judge
|
||||
version: v1
|
||||
description: Categorical verdict on how well the model triaged user intent — enum categorical primitive.
|
||||
---
|
||||
|
||||
## Dimensions
|
||||
|
||||
- verdict :: categorical :: bad,ok,good :: Triage verdict
|
||||
|
||||
## Checklist
|
||||
|
||||
- Categorical judgment on intent triage quality: bad · ok · good
|
||||
|
||||
## Notes
|
||||
|
||||
- Categorical primitive: judge emits one of {bad, ok, good}.
|
||||
- Export preserves the enum text (`verdict: 'good'`) rather than mapping
|
||||
to an integer index — matches LangSmith's Feedback tab where a
|
||||
categorical value renders as its string label.
|
||||
- Renders as a three-button pill row; keyboard 1/2/3 selects.
|
||||
25
examples/desktop/fixtures/rubrics/multi-turn-feedback.md
Normal file
25
examples/desktop/fixtures/rubrics/multi-turn-feedback.md
Normal file
@@ -0,0 +1,25 @@
|
||||
---
|
||||
name: multi-turn-feedback
|
||||
group: interaction-reasoning
|
||||
template: multi-turn
|
||||
executor: llm-judge
|
||||
version: v1
|
||||
description: Score each assistant turn against the 5 fixed multi-turn dimensions; each dim 1-5 relative to the immediately preceding user feedback.
|
||||
---
|
||||
|
||||
## Checklist
|
||||
|
||||
- Feedback understanding — the model correctly parsed the user's ask
|
||||
- Fix effectiveness — the response actually addressed the previous feedback
|
||||
- No regression — behavior that was already good stayed good
|
||||
- Over-correction — the model changed only what the feedback asked for
|
||||
- Convergence — the turn is moving toward a stable answer
|
||||
|
||||
## Notes
|
||||
|
||||
- One score per dimension per assistant turn, on a 1–5 scale.
|
||||
- The pinned prior-user feedback is the anchor: every dim scores this turn
|
||||
relative to that specific prior message, not the whole trajectory.
|
||||
- Stage-2 of the RL plan uses this rubric alongside the stage-1 task rubric
|
||||
— this one grades feedback response quality, the other grades base
|
||||
task quality.
|
||||
23
examples/desktop/fixtures/rubrics/passes-bench.md
Normal file
23
examples/desktop/fixtures/rubrics/passes-bench.md
Normal file
@@ -0,0 +1,23 @@
|
||||
---
|
||||
name: passes-bench
|
||||
group: se-process
|
||||
template: fixed
|
||||
executor: llm-judge
|
||||
version: v1
|
||||
description: Boolean pass/fail on a bench probe — two-state boolean primitive.
|
||||
---
|
||||
|
||||
## Dimensions
|
||||
|
||||
- passes :: boolean :: pass/fail :: Passes bench probe
|
||||
|
||||
## Checklist
|
||||
|
||||
- Did the trajectory pass the associated bench probe?
|
||||
|
||||
## Notes
|
||||
|
||||
- Boolean primitive: two-state (pass / fail).
|
||||
- Export carries `passes: true` or `passes: false`; downstream RL loop can
|
||||
treat this as a hard label for reward-shaping.
|
||||
- Renders as a two-button toggle; keyboard 1 = true, 2 = false.
|
||||
24
examples/desktop/fixtures/rubrics/svg-generation.md
Normal file
24
examples/desktop/fixtures/rubrics/svg-generation.md
Normal file
@@ -0,0 +1,24 @@
|
||||
---
|
||||
name: svg-generation
|
||||
group: interaction-reasoning
|
||||
template: per-prompt
|
||||
executor: llm-judge
|
||||
version: v1
|
||||
description: Per-prompt rubric — judge model composes SVG-specific criteria before scoring, comparing to a reference image described in the prompt.
|
||||
---
|
||||
|
||||
## Checklist
|
||||
|
||||
- The generated SVG renders without XML errors
|
||||
- Shape count and composition match the prompt intent
|
||||
- Colors and gradient use match the reference (or spec)
|
||||
- Text elements (if any) match the requested strings
|
||||
- Prompt-specific criteria (added by the judge model per task)
|
||||
|
||||
## Notes
|
||||
|
||||
- Per-prompt template: the judge model receives the prompt + reference image
|
||||
description, then extends the fixed checklist with 2–4 prompt-specific
|
||||
items before scoring. Output preserves the {resolved, score, reason}
|
||||
block for DSBench compatibility.
|
||||
- Deduct heavily when the SVG is a stub (empty viewBox, no shapes).
|
||||
Reference in New Issue
Block a user