refactor(token-meter): fold the surface once for the meter and the projection

`contextBreakdown.messageTokens` and `measure().surfaceTokens` answer the same
question in the same heuristic vocabulary, and the panel's composition rows are
only honest while they agree. Each owner carried its own copy of the positional
fold — same pricing, same `{seq, tokens}` node list, same replace-range lookup
and guard, differing only in mutable versus immutable application — so an edit
to either one would have moved the panel away from `measure()` with both sides
still green. The duplication gate caught the shared 62 tokens.

`src/surface-fold.ts` now owns `foldSurfaceTokens`: total, allocation-fresh,
returning the event's price, the next surface, and the signed total delta. The
service assigns that result where it used to prepare a commit closure, which
keeps its validate-before-mutate replay transaction intact — the fold throws
before any state is touched, so a malformed event still fails identically on
every retry. `_prepareSurfaceMutation` and `_estimateSurfaceEvent` go away with
it, and the projection's apply drops to one call.

Covers the identity with a session that appends and then compacts, asserting
the projection figure equals the service surface at each boundary; the test
fails when either side of the fold is perturbed.

packages/llm/token-meter/README.i18n.yaml

@@ -2,5 +2,5 @@
 # side as of the last confirmed-consistent state. Both languages carry equal authority;
 # after editing either side, bring the other along and re-record with:
 #   pnpm run verify-translation-pairing --write packages/llm/token-meter/README.md
-README.md: 2b8320221dc314c9a18f93eab3d88a66d22d8342
-README.zh.md: ce2e352b8a4191bfbfcce4f317d21e3db512abfd
+README.md: dd751b7449fb4f75199af44781ad139f2952e9a4
+README.zh.md: 4ebd693f2a6197557ed5f0724020287c37f0a2fc

packages/llm/token-meter/README.md

@@ -29,7 +29,7 @@ When the composition provides `ctx.sessionProjections`, token-meter registers th
 
 `contextPressure` carries optional `pressureTokens` — the newest provider-reported prompt size, summing uncached input plus cache reads and writes — and optional `contextWindow` from the newest `request/context` record. Pressure stays absent until a provider reports usage; capacity stays absent for a route whose adapter advertises none. Output is excluded, so the numerator holds still while a turn streams and steps forward when the next request reports its usage.
 
-`contextBreakdown` carries heuristic `systemTokens`, `toolsTokens`, and `messageTokens` — the context's composition rather than its provider-billed size. The envelope figures reprice last-wins on every `request/header`; the message figure folds surface appends and positional replacements, so compaction shrinks it the same way it shrinks the next request. All three figures use the measurement service's fixed heuristic and are estimates: they do not reconcile with the provider-exact `pressureTokens`, and a UI should present them as approximations.
+`contextBreakdown` carries heuristic `systemTokens`, `toolsTokens`, and `messageTokens` — the context's composition rather than its provider-billed size. The envelope figures reprice last-wins on every `request/header`; the message figure replays `surface-fold.ts` — the same positional fold `measure()` runs — so it equals `measure().surfaceTokens` at every event boundary and compaction shrinks it the way it shrinks the next request. All three figures use the measurement service's fixed heuristic and are estimates: they do not reconcile with the provider-exact `pressureTokens`, and a UI should present them as approximations.
 
 All three units use the standard projection baseline, live frame, higher-seq-wins store, and JSON checkpoint paths. Unloading token-meter removes all three keys. A headless or TUI composition without the projection seam keeps the measurement service's existing behavior.
 

packages/llm/token-meter/README.zh.md

@@ -29,7 +29,7 @@ fold 跟踪完整请求标头快照、步骤边界、表层追加与替换、成
 
 `contextPressure` 携带可选的 `pressureTokens`（提供方报告的最新提示词规模，为未缓存输入加缓存读取与写入之和），以及来自最新一条 `request/context` 记录的可选 `contextWindow`。提供方报告用量前压力保持缺失；路由适配器未公布容量时容量也保持缺失。输出不计入其中，因此轮次流式输出期间分子保持不动，等到下一个请求报告用量时才前进。
 
-`contextBreakdown` 携带启发式的 `systemTokens`、`toolsTokens` 与 `messageTokens`，描述上下文的组成而非提供方计费规模。envelope 数字在每条 `request/header` 上按后者胜重新计价；消息数字折叠表层追加与位置替换，因此压缩会像缩小下一个请求那样缩小它。三个数字都使用测量服务的固定启发式规则，属于估算值：它们不会与提供方精确的 `pressureTokens` 对账，UI 应以近似值方式呈现。
+`contextBreakdown` 携带启发式的 `systemTokens`、`toolsTokens` 与 `messageTokens`，描述上下文的组成而非提供方计费规模。envelope 数字在每条 `request/header` 上按后者胜重新计价；消息数字重放 `surface-fold.ts`——与 `measure()` 运行的位置折叠是同一份——因此它在每个事件边界上都等于 `measure().surfaceTokens`，压缩会像缩小下一个请求那样缩小它。三个数字都使用测量服务的固定启发式规则，属于估算值：它们不会与提供方精确的 `pressureTokens` 对账，UI 应以近似值方式呈现。
 
 三个单元都使用标准的投影基线、实时帧、seq 高者胜值仓和 JSON 检查点路径。卸载 token-meter 会移除这三个键。不带投影 seam 的 headless 或 TUI 组合会保留测量服务的既有行为。
 

packages/llm/token-meter/src/breakdown-projection.ts

@@ -6,23 +6,20 @@
  */
 
 import { z } from 'zod'
-import { canonicalHeader, deriveEventMessage, isSurfaceEvent } from '@deepseek-ai/dsh-session'
+import { canonicalHeader, isSurfaceEvent } from '@deepseek-ai/dsh-session'
 import type { ProjectionDefinition } from '@deepseek-ai/dsh-session-projection'
-import { estimateMessage, estimateSystemTokens, estimateToolsTokens } from './estimate.ts'
+import type { TokenSurfaceNode } from './types.ts'
+import { estimateSystemTokens, estimateToolsTokens } from './estimate.ts'
+import { foldSurfaceTokens } from './surface-fold.ts'
 // Import for the `contextBreakdown` SessionProjectionMap key merge.
 import type {} from './projection.ts'
 
-/** One priced surface node (plain JSON for the persisted projection cache). */
-interface BreakdownSurfaceNode {
-  seq: number
-  tokens: number
-}
-
 interface ContextBreakdownState {
   systemTokens: number
   toolsTokens: number
   messageTokens: number
-  surface: BreakdownSurfaceNode[]
+  /** Priced surface nodes (plain JSON for the persisted projection cache). */
+  surface: TokenSurfaceNode[]
 }
 
 const breakdownSchema = z.object({
@@ -35,10 +32,9 @@ const breakdownSchema = z.object({
  * Token-meter's context-composition projection unit.
  *
  * Envelope figures are last-wins per `request/header`; the message figure
- * folds surface appends and positional replacements, so compaction shrinks it
- * the same way it shrinks the next request. Committed logs are
- * surface-validated at append time, so an unresolvable replace range here is
- * log corruption and fails loud rather than skipping the event.
+ * rides {@link foldSurfaceTokens} — the same fold the measurement service
+ * replays — so it equals `measure().surfaceTokens` at every event boundary and
+ * compaction shrinks it the way it shrinks the next request.
  */
 export const contextBreakdownProjectionDefinition:
 ProjectionDefinition<'contextBreakdown', ContextBreakdownState> = {
@@ -54,32 +50,11 @@ ProjectionDefinition<'contextBreakdown', ContextBreakdownState> = {
       return { ...state, systemTokens, toolsTokens }
     }
     if (!isSurfaceEvent(event)) return state
-    const message = deriveEventMessage(event)
-    const tokens = message === null ? 0 : estimateMessage(message)
-    const op = event.surfaceOp
-    if (op === 'append') {
-      return {
-        ...state,
-        messageTokens: state.messageTokens + tokens,
-        surface: [...state.surface, { seq: event.seq, tokens }],
-      }
-    }
-    const startIdx = state.surface.findIndex(node => node.seq === op.start)
-    const endIdx = state.surface.findIndex(node => node.seq === op.end)
-    if (startIdx === -1 || endIdx === -1 || startIdx > endIdx) {
-      throw new Error(
-        `context breakdown: replace at seq ${event.seq} has invalid current range ${op.start}-${op.end}`,
-      )
-    }
-    const removed = state.surface
-      .slice(startIdx, endIdx + 1)
-      .reduce((total, node) => total + node.tokens, 0)
-    const surface = [...state.surface]
-    surface.splice(startIdx, endIdx - startIdx + 1, { seq: event.seq, tokens })
+    const fold = foldSurfaceTokens(state.surface, event)
     return {
       ...state,
-      messageTokens: state.messageTokens + tokens - removed,
-      surface,
+      messageTokens: state.messageTokens + fold.deltaTokens,
+      surface: fold.nodes,
     }
   },
   view: ({ systemTokens, toolsTokens, messageTokens }) => ({ systemTokens, toolsTokens, messageTokens }),

packages/llm/token-meter/src/index.ts

@@ -8,7 +8,7 @@ import { Context, Service } from 'cordis'
 import z from 'schemastery'
 import { BlockAssembler, deepFreeze } from '@deepseek-ai/dsh-llm'
 import type { Message, TokenUsage } from '@deepseek-ai/dsh-llm'
-import type { EpochHeader, Session, SessionEvent, SurfaceEvent } from '@deepseek-ai/dsh-session'
+import type { EpochHeader, Session, SessionEvent } from '@deepseek-ai/dsh-session'
 import { canonicalHeader, headerEquals, isSurfaceEvent } from '@deepseek-ai/dsh-session'
 // Type-only: resolves the optional projection registry Context seam.
 import type {} from '@deepseek-ai/dsh-session-projection'
@@ -21,6 +21,7 @@ import type {
 import { contextBreakdownProjectionDefinition } from './breakdown-projection.ts'
 import { contextPressureProjectionDefinition, tokenUsageProjectionDefinition } from './usage-projection.ts'
 import { estimateContent, estimateHeader, estimateMessage, ROLE_OVERHEAD } from './estimate.ts'
+import { foldSurfaceTokens } from './surface-fold.ts'
 
 export type * from './types.ts'
 
@@ -39,11 +40,6 @@ interface ReplayState {
   anchor: MeasurementAnchor | undefined
 }
 
-interface PreparedSurfaceMutation {
-  readonly tokens: number
-  commit(state: ReplayState): void
-}
-
 /** Sum disjoint provider usage buckets without double-counting reasoning output. */
 function usageTokens(usage: TokenUsage): number {
   return usage.inputTokens
@@ -219,7 +215,7 @@ export class TokenMeterService extends Service {
     }
 
     const surface = isSurfaceEvent(event)
-      ? this._prepareSurfaceMutation(session, state, event)
+      ? foldSurfaceTokens(state.surface, event)
       : undefined
 
     if (event.type === 'assistant/message') {
@@ -266,53 +262,13 @@ export class TokenMeterService extends Service {
 
     state.header = nextHeader
     state.stepStart = nextStepStart
-    if (surface !== undefined) surface.commit(state)
+    if (surface !== undefined) {
+      state.surface = surface.nodes
+      state.surfaceTokens += surface.deltaTokens
+    }
     state.anchor = nextAnchor
   }
 
-  /** Validate one surface operation and return its allocation-light commit. */
-  private _prepareSurfaceMutation(
-    session: Session,
-    state: ReplayState,
-    event: SurfaceEvent,
-  ): PreparedSurfaceMutation {
-    const tokens = this._estimateSurfaceEvent(session, event)
-    const op = event.surfaceOp
-    if (op === 'append') {
-      return {
-        tokens,
-        commit(target) {
-          target.surface.push({ seq: event.seq, tokens })
-          target.surfaceTokens += tokens
-        },
-      }
-    }
-
-    const startIdx = state.surface.findIndex(node => node.seq === op.start)
-    const endIdx = state.surface.findIndex(node => node.seq === op.end)
-    if (startIdx === -1 || endIdx === -1 || startIdx > endIdx) {
-      throw new Error(
-        `token meter: replace at seq ${event.seq} has invalid current range ${op.start}-${op.end}`,
-      )
-    }
-    const removedTokens = state.surface
-      .slice(startIdx, endIdx + 1)
-      .reduce((total, node) => total + node.tokens, 0)
-    return {
-      tokens,
-      commit(target) {
-        target.surface.splice(startIdx, endIdx - startIdx + 1, { seq: event.seq, tokens })
-        target.surfaceTokens += tokens - removedTokens
-      },
-    }
-  }
-
-  /** Price one current surface event exactly as it projects to a request. */
-  private _estimateSurfaceEvent(session: Session, event: SurfaceEvent): number {
-    const message = session.deriveEventMessage(event)
-    return message === null ? 0 : this.estimateMessage(message)
-  }
-
   /**
    * Reassemble provider output from exact chunk provenance for a usage anchor.
    * Missing legacy provenance conservatively treats the durable output as the

packages/llm/token-meter/src/invariant.ts

@@ -17,9 +17,12 @@ export const inject = ['invariants']
 /**
  * No runtime invariant: token estimates are per-call outputs and the private
  * session cache is invalidated at its event mutation boundary. The package's
- * projection does expose an observation stream, but its schema fixes the JSON
- * payload and its pure fold replaces same-step samples; totals need not be
- * monotone when a final usage sample corrects an earlier chunk.
+ * three projections do expose observation streams, but their schemas fix the
+ * JSON payloads; the usage folds replace same-step samples, so totals need not
+ * be monotone when a final sample corrects an earlier chunk, and the
+ * composition fold shares `surface-fold.ts` with the measurement service,
+ * which makes its message figure equal `measure().surfaceTokens` by
+ * construction rather than by a relation worth observing at runtime.
  */
 const install: InvariantInstaller = () => {}
 

packages/llm/token-meter/src/surface-fold.ts

@@ -0,0 +1,63 @@
+/**
+ * The one positional surface fold, shared by the measurement service's replay
+ * state and the pure `contextBreakdown` projection. Both answer "what does the
+ * current model-visible conversation cost", so they MUST price and place every
+ * node identically: a private copy in either owner would let the panel's
+ * message figure drift away from `measure().surfaceTokens` with both sides
+ * still passing their own tests.
+ *
+ * @module @deepseek-ai/dsh-token-meter/surface-fold
+ */
+
+import { deriveEventMessage } from '@deepseek-ai/dsh-session'
+import type { SurfaceEvent } from '@deepseek-ai/dsh-session'
+import type { TokenSurfaceNode } from './types.ts'
+import { estimateMessage } from './estimate.ts'
+
+/** One surface event's placement and cost against the surface preceding it. */
+export interface SurfaceTokenFold {
+  /** Heuristic price of the event's own message; 0 when it derives none. */
+  readonly tokens: number
+  /** The surface after the event, detached from the input. */
+  readonly nodes: TokenSurfaceNode[]
+  /** Signed change in the surface total: `tokens` minus anything shadowed. */
+  readonly deltaTokens: number
+}
+
+/**
+ * Fold one surface event onto a priced surface.
+ *
+ * Total and allocation-fresh: the caller assigns the result rather than
+ * mutating in place, so a throw here leaves the caller's state untouched and
+ * the same malformed event fails identically on every retry.
+ * @param nodes - the priced surface preceding this event, in model-visible order.
+ * @param event - the surface event to place.
+ * @returns the event's price, the next surface, and the signed total delta.
+ * @throws when a replacement names a range absent from `nodes` — committed
+ *   logs are surface-validated at append time, so an unresolvable range is log
+ *   corruption and must fail loud rather than skip the event.
+ */
+export function foldSurfaceTokens(
+  nodes: readonly TokenSurfaceNode[],
+  event: SurfaceEvent,
+): SurfaceTokenFold {
+  const message = deriveEventMessage(event)
+  const tokens = message === null ? 0 : estimateMessage(message)
+  const op = event.surfaceOp
+  if (op === 'append') {
+    return { tokens, nodes: [...nodes, { seq: event.seq, tokens }], deltaTokens: tokens }
+  }
+  const startIdx = nodes.findIndex(node => node.seq === op.start)
+  const endIdx = nodes.findIndex(node => node.seq === op.end)
+  if (startIdx === -1 || endIdx === -1 || startIdx > endIdx) {
+    throw new Error(
+      `token surface: replace at seq ${event.seq} has invalid current range ${op.start}-${op.end}`,
+    )
+  }
+  const removed = nodes
+    .slice(startIdx, endIdx + 1)
+    .reduce((total, node) => total + node.tokens, 0)
+  const next = [...nodes]
+  next.splice(startIdx, endIdx - startIdx + 1, { seq: event.seq, tokens })
+  return { tokens, nodes: next, deltaTokens: tokens - removed }
+}

packages/llm/token-meter/tests/context-breakdown-projection.spec.ts

@@ -115,6 +115,47 @@ describe('contextBreakdown session projection', () => {
     expect(projected(ctx, session).messageTokens).toBe(estimateMessage(summary))
   })
 
+  it('keeps the message figure equal to the service surface across appends and a compaction', async () => {
+    const { ctx, session } = await harness()
+    // The panel's composition rows and `measure()` answer the same question in
+    // the same vocabulary; one shared fold is what makes that true.
+    const agree = (): number => {
+      const messageTokens = projected(ctx, session).messageTokens
+      expect(messageTokens).toBe(ctx.tokenMeter.measure(session).surfaceTokens)
+      return messageTokens
+    }
+    session.append('request/header', {
+      header: { config: CONFIG, system: 'You are terse.', tools: TOOLS },
+      reason: 'initial',
+    })
+    expect(agree()).toBe(0)
+
+    const question = appendUser(session, 'a first question, long enough to price above zero')
+    session.append('step/start', { turn: 1, step: 1 })
+    const answer = session.append('assistant/message', {
+      turn: 1,
+      step: 1,
+      message: createMessage({
+        role: 'assistant',
+        content: [{ type: 'text', text: 'a considered answer' }],
+        source: { kind: 'model', provider: 'mock', model: 'mock' },
+      }),
+      usage: { inputTokens: 40, outputTokens: 7 },
+    }, { surfaceOp: 'append', sourceEventSeqs: [] }).seq
+    session.append('step/end', { turn: 1, step: 1 })
+    const grown = agree()
+    expect(grown).toBeGreaterThan(0)
+
+    session.append('user/message', createUserMessage({
+      content: [{ type: 'text', text: 'summary' }],
+      source: { kind: 'plugin', plugin: 'test' },
+    }), {
+      surfaceOp: { op: 'replace', start: question, end: answer },
+      sourceEventSeqs: [question, answer],
+    })
+    expect(agree()).toBeLessThan(grown)
+  })
+
   it('fails loud on a replace range absent from the folded surface', () => {
     const definition = contextBreakdownProjectionDefinition
     const replace = (start: number, end: number): SessionEvent => ({