feat(web): turn speed metrics and composer context meter
Assistant footers and the stats line gain TTFT/tok-per-second readings folded from step timings; context occupancy moves off the stats line onto a composer ring whose panel shows a heuristic system/tools/messages breakdown from the new token-meter contextBreakdown session projection.
This commit is contained in:
Yif
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/llm/token-meter/README.md
|
||||
README.md: 701893b342f9a93a75bec175634b1054f3d17151
|
||||
README.zh.md: a5844e8788422bba669632ed587fb87e1e2a1e58
|
||||
README.md: 2b8320221dc314c9a18f93eab3d88a66d22d8342
|
||||
README.zh.md: 7c0a433146013e79034d5eb34ebb39e93caf30ff
|
||||
|
||||
@@ -23,13 +23,15 @@ Usage accounting sums disjoint input, cache-read, cache-write, and output bucket
|
||||
|
||||
## Session projections
|
||||
|
||||
When the composition provides `ctx.sessionProjections`, token-meter registers two units through an optional child fiber.
|
||||
When the composition provides `ctx.sessionProjections`, token-meter registers three units through an optional child fiber.
|
||||
|
||||
`tokenUsage` carries the complete durable log's `uncachedInputTokens`, `outputTokens`, `cacheReadTokens`, and `cacheWriteTokens`. Usage chunks are counted even when a request later fails; a final assistant-message usage for the same `(turn, step)` replaces that sample instead of double-counting it. Reasoning remains an output subdivision. The single last-sample slot relies on a session-log ordering property: once a later step reports usage, a legal log never reports usage for an earlier step again.
|
||||
|
||||
`contextPressure` carries optional `pressureTokens` — the newest provider-reported prompt size, summing uncached input plus cache reads and writes — and optional `contextWindow` from the newest `request/context` record. Pressure stays absent until a provider reports usage; capacity stays absent for a route whose adapter advertises none. Output is excluded, so the numerator holds still while a turn streams and steps forward when the next request reports its usage.
|
||||
|
||||
Both units use the standard projection baseline, live frame, higher-seq-wins store, and JSON checkpoint paths. Unloading token-meter removes both keys. A headless or TUI composition without the projection seam keeps the measurement service's existing behavior.
|
||||
`contextBreakdown` carries heuristic `systemTokens`, `toolsTokens`, and `messageTokens` — the context's composition rather than its provider-billed size. The envelope figures reprice last-wins on every `request/header`; the message figure folds surface appends and positional replacements, so compaction shrinks it the same way it shrinks the next request. All three figures use the measurement service's fixed heuristic and are estimates: they do not reconcile with the provider-exact `pressureTokens`, and a UI should present them as approximations.
|
||||
|
||||
All three units use the standard projection baseline, live frame, higher-seq-wins store, and JSON checkpoint paths. Unloading token-meter removes all three keys. A headless or TUI composition without the projection seam keeps the measurement service's existing behavior.
|
||||
|
||||
### Context occupancy is an approximation, by design
|
||||
|
||||
|
||||
@@ -23,13 +23,15 @@ fold 跟踪完整请求标头快照、步骤边界、表层追加与替换、成
|
||||
|
||||
## 会话投影
|
||||
|
||||
当组合提供 `ctx.sessionProjections` 时，token-meter 会通过一个可选子 fiber 注册两个单元。
|
||||
当组合提供 `ctx.sessionProjections` 时，token-meter 会通过一个可选子 fiber 注册三个单元。
|
||||
|
||||
`tokenUsage` 携带完整持久日志中的 `uncachedInputTokens`、`outputTokens`、`cacheReadTokens` 和 `cacheWriteTokens`。即使请求随后失败，用量分片仍会计入；同一 `(turn, step)` 的最终 assistant 消息用量会替换该样本，而不是重复计数。推理仍是输出的一个细分项。只保留单个最新样本，依赖的是会话日志的一条顺序性质：一旦某个更晚的步骤报告了用量，合法日志就绝不会再为更早的步骤报告用量。
|
||||
|
||||
`contextPressure` 携带可选的 `pressureTokens`（提供方报告的最新提示词规模，为未缓存输入加缓存读取与写入之和），以及来自最新一条 `request/context` 记录的可选 `contextWindow`。提供方报告用量前压力保持缺失；路由适配器未公布容量时容量也保持缺失。输出不计入其中，因此轮次流式输出期间分子保持不动，等到下一个请求报告用量时才前进。
|
||||
|
||||
两个单元都使用标准的投影基线、实时帧、seq 高者胜值仓和 JSON 检查点路径。卸载 token-meter 会移除这两个键。不带投影 seam 的 headless 或 TUI 组合会保留测量服务的既有行为。
|
||||
`contextBreakdown` 携带启发式的 `systemTokens`、`toolsTokens` 与 `messageTokens`，描述上下文的组成而非提供方计费规模。envelope 数字在每条 `request/header` 上按后者胜重新计价；消息数字折叠表层追加与位置替换，因此压缩会像缩小下一个请求那样缩小它。三个数字都使用测量服务的固定启发式规则，属于估算值：它们不会与提供方精确的 `pressureTokens` 对账，UI 应以近似值方式呈现。
|
||||
|
||||
三个单元都使用标准的投影基线、实时帧、seq 高者胜值仓和 JSON 检查点路径。卸载 token-meter 会移除这三个键。不带投影 seam 的 headless 或 TUI 组合会保留测量服务的既有行为。
|
||||
|
||||
### 上下文占用率是刻意为之的近似值
|
||||
|
||||
|
||||
87
packages/llm/token-meter/src/breakdown-projection.ts
Normal file
87
packages/llm/token-meter/src/breakdown-projection.ts
Normal file
@@ -0,0 +1,87 @@
|
||||
/**
|
||||
* Pure fold for the heuristic context-composition projection: system prompt
|
||||
* and tool schemas from the newest request envelope, conversation from the
|
||||
* live surface. Prices with the same shared estimator as the meter service,
|
||||
* so the three figures match `measure()`'s heuristic vocabulary exactly.
|
||||
*/
|
||||
|
||||
import { z } from 'zod'
|
||||
import { canonicalHeader, deriveEventMessage, isSurfaceEvent } from '@deepseek-ai/dsh-session'
|
||||
import type { ProjectionDefinition } from '@deepseek-ai/dsh-session-projection'
|
||||
import { estimateMessage, estimateSystemTokens, estimateToolsTokens } from './estimate.ts'
|
||||
// Import for the `contextBreakdown` SessionProjectionMap key merge.
|
||||
import type {} from './projection.ts'
|
||||
|
||||
/** One priced surface node (plain JSON for the persisted projection cache). */
|
||||
interface BreakdownSurfaceNode {
|
||||
seq: number
|
||||
tokens: number
|
||||
}
|
||||
|
||||
interface ContextBreakdownState {
|
||||
systemTokens: number
|
||||
toolsTokens: number
|
||||
messageTokens: number
|
||||
surface: BreakdownSurfaceNode[]
|
||||
}
|
||||
|
||||
const breakdownSchema = z.object({
|
||||
systemTokens: z.number().int().nonnegative(),
|
||||
toolsTokens: z.number().int().nonnegative(),
|
||||
messageTokens: z.number().int().nonnegative(),
|
||||
}).strict()
|
||||
|
||||
/**
|
||||
* Token-meter's context-composition projection unit.
|
||||
*
|
||||
* Envelope figures are last-wins per `request/header`; the message figure
|
||||
* folds surface appends and positional replacements, so compaction shrinks it
|
||||
* the same way it shrinks the next request. Committed logs are
|
||||
* surface-validated at append time, so an unresolvable replace range here is
|
||||
* log corruption and fails loud rather than skipping the event.
|
||||
*/
|
||||
export const contextBreakdownProjectionDefinition:
|
||||
ProjectionDefinition<'contextBreakdown', ContextBreakdownState> = {
|
||||
key: 'contextBreakdown',
|
||||
schema: breakdownSchema,
|
||||
init: () => ({ systemTokens: 0, toolsTokens: 0, messageTokens: 0, surface: [] }),
|
||||
apply: (state, event) => {
|
||||
if (event.type === 'request/header') {
|
||||
const header = canonicalHeader(event.data.header)
|
||||
const systemTokens = estimateSystemTokens(header)
|
||||
const toolsTokens = estimateToolsTokens(header)
|
||||
if (systemTokens === state.systemTokens && toolsTokens === state.toolsTokens) return state
|
||||
return { ...state, systemTokens, toolsTokens }
|
||||
}
|
||||
if (!isSurfaceEvent(event)) return state
|
||||
const message = deriveEventMessage(event)
|
||||
const tokens = message === null ? 0 : estimateMessage(message)
|
||||
const op = event.surfaceOp
|
||||
if (op === 'append') {
|
||||
return {
|
||||
...state,
|
||||
messageTokens: state.messageTokens + tokens,
|
||||
surface: [...state.surface, { seq: event.seq, tokens }],
|
||||
}
|
||||
}
|
||||
const startIdx = state.surface.findIndex(node => node.seq === op.start)
|
||||
const endIdx = state.surface.findIndex(node => node.seq === op.end)
|
||||
if (startIdx === -1 || endIdx === -1 || startIdx > endIdx) {
|
||||
throw new Error(
|
||||
`context breakdown: replace at seq ${event.seq} has invalid current range ${op.start}-${op.end}`,
|
||||
)
|
||||
}
|
||||
const removed = state.surface
|
||||
.slice(startIdx, endIdx + 1)
|
||||
.reduce((total, node) => total + node.tokens, 0)
|
||||
const surface = [...state.surface]
|
||||
surface.splice(startIdx, endIdx - startIdx + 1, { seq: event.seq, tokens })
|
||||
return {
|
||||
...state,
|
||||
messageTokens: state.messageTokens + tokens - removed,
|
||||
surface,
|
||||
}
|
||||
},
|
||||
view: ({ systemTokens, toolsTokens, messageTokens }) => ({ systemTokens, toolsTokens, messageTokens }),
|
||||
stateVersion: 1,
|
||||
}
|
||||
87
packages/llm/token-meter/src/estimate.ts
Normal file
87
packages/llm/token-meter/src/estimate.ts
Normal file
@@ -0,0 +1,87 @@
|
||||
/**
|
||||
* Fixed-density heuristic token pricing shared by the meter service and the
|
||||
* pure context-breakdown projection, so both surfaces price identical content
|
||||
* to identical numbers.
|
||||
*
|
||||
* @module @deepseek-ai/dsh-token-meter/estimate
|
||||
*/
|
||||
|
||||
import type { ContentBlock, Message } from '@deepseek-ai/dsh-llm'
|
||||
import type { EpochHeader } from '@deepseek-ai/dsh-session'
|
||||
|
||||
/** Fixed text-density estimate used until exact tokenization is needed. */
|
||||
const CHARS_PER_TOKEN = 4
|
||||
|
||||
/** Per-block structural overhead for JSON framing and type tags. */
|
||||
const BLOCK_OVERHEAD = 4
|
||||
|
||||
/** Role-field framing overhead added to every priced message. */
|
||||
export const ROLE_OVERHEAD = 4
|
||||
|
||||
/**
|
||||
* Price content blocks recursively under the fixed density heuristic.
|
||||
* @param blocks - content blocks to price without mutation.
|
||||
* @returns heuristic tokens including per-block structural overhead.
|
||||
*/
|
||||
export function estimateContent(blocks: readonly ContentBlock[]): number {
|
||||
let tokens = 0
|
||||
for (const block of blocks) {
|
||||
switch (block.type) {
|
||||
case 'text':
|
||||
case 'reasoning':
|
||||
tokens += Math.ceil(block.text.length / CHARS_PER_TOKEN) + BLOCK_OVERHEAD
|
||||
break
|
||||
case 'tool-call':
|
||||
tokens += Math.ceil(block.name.length / CHARS_PER_TOKEN)
|
||||
+ Math.ceil(block.arguments.length / CHARS_PER_TOKEN)
|
||||
+ BLOCK_OVERHEAD
|
||||
break
|
||||
case 'tool-result':
|
||||
tokens += estimateContent(block.content) + BLOCK_OVERHEAD
|
||||
break
|
||||
default:
|
||||
// ContentBlockMap is merge-extensible; unknown blocks retain a
|
||||
// conservative structural JSON price under the fixed heuristic.
|
||||
tokens += BLOCK_OVERHEAD + Math.ceil(JSON.stringify(block).length / CHARS_PER_TOKEN)
|
||||
}
|
||||
}
|
||||
return tokens
|
||||
}
|
||||
|
||||
/**
|
||||
* Heuristically price one model-visible message.
|
||||
* @param message - message to price without mutation.
|
||||
* @returns content and role-framing tokens under the fixed heuristic.
|
||||
*/
|
||||
export function estimateMessage(message: Message): number {
|
||||
return estimateContent(message.content) + ROLE_OVERHEAD
|
||||
}
|
||||
|
||||
/**
|
||||
* Price the system-prompt part of a canonical request envelope.
|
||||
* @param header - canonical envelope, or undefined before any request.
|
||||
* @returns heuristic system-prompt tokens; 0 when absent.
|
||||
*/
|
||||
export function estimateSystemTokens(header: EpochHeader | undefined): number {
|
||||
if (header?.system === undefined) return 0
|
||||
return Math.ceil(header.system.length / CHARS_PER_TOKEN) + ROLE_OVERHEAD
|
||||
}
|
||||
|
||||
/**
|
||||
* Price the tool-schema part of a canonical request envelope.
|
||||
* @param header - canonical envelope, or undefined before any request.
|
||||
* @returns heuristic tool-schema tokens; 0 when absent or empty.
|
||||
*/
|
||||
export function estimateToolsTokens(header: EpochHeader | undefined): number {
|
||||
if (header?.tools === undefined || header.tools.length === 0) return 0
|
||||
return Math.ceil(JSON.stringify(header.tools).length / CHARS_PER_TOKEN) + BLOCK_OVERHEAD
|
||||
}
|
||||
|
||||
/**
|
||||
* Price the complete non-surface request envelope.
|
||||
* @param header - canonical envelope, or undefined before any request.
|
||||
* @returns heuristic system plus tool tokens.
|
||||
*/
|
||||
export function estimateHeader(header: EpochHeader | undefined): number {
|
||||
return estimateSystemTokens(header) + estimateToolsTokens(header)
|
||||
}
|
||||
@@ -7,7 +7,7 @@
|
||||
import { Context, Service } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import { BlockAssembler, deepFreeze } from '@deepseek-ai/dsh-llm'
|
||||
import type { ContentBlock, Message, TokenUsage } from '@deepseek-ai/dsh-llm'
|
||||
import type { Message, TokenUsage } from '@deepseek-ai/dsh-llm'
|
||||
import type { EpochHeader, Session, SessionEvent, SurfaceEvent } from '@deepseek-ai/dsh-session'
|
||||
import { canonicalHeader, headerEquals, isSurfaceEvent } from '@deepseek-ai/dsh-session'
|
||||
// Type-only: resolves the optional projection registry Context seam.
|
||||
@@ -18,19 +18,12 @@ import type {
|
||||
TokenMeterConfig,
|
||||
TokenSurfaceNode,
|
||||
} from './types.ts'
|
||||
import { contextBreakdownProjectionDefinition } from './breakdown-projection.ts'
|
||||
import { contextPressureProjectionDefinition, tokenUsageProjectionDefinition } from './usage-projection.ts'
|
||||
import { estimateContent, estimateHeader, estimateMessage, ROLE_OVERHEAD } from './estimate.ts'
|
||||
|
||||
export type * from './types.ts'
|
||||
|
||||
/** Fixed text-density estimate used until exact tokenization is needed. */
|
||||
const CHARS_PER_TOKEN = 4
|
||||
|
||||
/** Per-block structural overhead for JSON framing and type tags. */
|
||||
const BLOCK_OVERHEAD = 4
|
||||
|
||||
/** Role-field framing overhead added to every priced message. */
|
||||
const ROLE_OVERHEAD = 4
|
||||
|
||||
interface MeasurementAnchor {
|
||||
readonly header: EpochHeader | undefined
|
||||
readonly surfaceTokens: number
|
||||
@@ -98,6 +91,7 @@ export class TokenMeterService extends Service {
|
||||
ctx.inject(['sessionProjections'], (projectionCtx) => {
|
||||
projectionCtx.sessionProjections.register(tokenUsageProjectionDefinition)
|
||||
projectionCtx.sessionProjections.register(contextPressureProjectionDefinition)
|
||||
projectionCtx.sessionProjections.register(contextBreakdownProjectionDefinition)
|
||||
})
|
||||
|
||||
// Readers catch up independently, while eager observation bounds ordinary
|
||||
@@ -141,7 +135,7 @@ export class TokenMeterService extends Service {
|
||||
} else {
|
||||
baseline = {
|
||||
kind: 'estimated',
|
||||
tokens: this._estimateHeader(header) + state.surfaceTokens,
|
||||
tokens: estimateHeader(header) + state.surfaceTokens,
|
||||
}
|
||||
surfaceDeltaTokens = 0
|
||||
}
|
||||
@@ -157,12 +151,13 @@ export class TokenMeterService extends Service {
|
||||
}
|
||||
|
||||
/**
|
||||
* Heuristically price one model-visible message.
|
||||
* Heuristically price one model-visible message (instance face of the pure
|
||||
* {@link estimateMessage}).
|
||||
* @param message - message to price without mutation.
|
||||
* @returns content and role-framing tokens under the fixed service heuristic.
|
||||
*/
|
||||
estimateMessage(message: Message): number {
|
||||
return this._estimateContent(message.content) + ROLE_OVERHEAD
|
||||
return estimateMessage(message)
|
||||
}
|
||||
|
||||
/** Catch one session's fold up to the current durable tail. */
|
||||
@@ -246,7 +241,7 @@ export class TokenMeterService extends Service {
|
||||
)
|
||||
const anchorSurfaceTokens = stepStart.surfaceTokens + providerAssistantTokens
|
||||
const providerTokens = usageTokens(event.data.usage)
|
||||
const estimatedAnchorTokens = this._estimateHeader(nextHeader) + anchorSurfaceTokens
|
||||
const estimatedAnchorTokens = estimateHeader(nextHeader) + anchorSurfaceTokens
|
||||
nextAnchor = {
|
||||
header: nextHeader,
|
||||
surfaceTokens: anchorSurfaceTokens,
|
||||
@@ -263,7 +258,7 @@ export class TokenMeterService extends Service {
|
||||
surfaceTokens: anchorSurfaceTokens,
|
||||
baseline: {
|
||||
kind: 'estimated',
|
||||
tokens: this._estimateHeader(nextHeader) + anchorSurfaceTokens,
|
||||
tokens: estimateHeader(nextHeader) + anchorSurfaceTokens,
|
||||
},
|
||||
}
|
||||
}
|
||||
@@ -355,46 +350,7 @@ export class TokenMeterService extends Service {
|
||||
assembler.push(sourceEvent.data.chunk)
|
||||
}
|
||||
const providerContent = assembler.blocks()
|
||||
return providerContent.length === 0 ? 0 : this._estimateContent(providerContent) + ROLE_OVERHEAD
|
||||
}
|
||||
|
||||
/** Price content blocks recursively under the fixed density heuristic. */
|
||||
private _estimateContent(blocks: readonly ContentBlock[]): number {
|
||||
let tokens = 0
|
||||
for (const block of blocks) {
|
||||
switch (block.type) {
|
||||
case 'text':
|
||||
case 'reasoning':
|
||||
tokens += Math.ceil(block.text.length / CHARS_PER_TOKEN) + BLOCK_OVERHEAD
|
||||
break
|
||||
case 'tool-call':
|
||||
tokens += Math.ceil(block.name.length / CHARS_PER_TOKEN)
|
||||
+ Math.ceil(block.arguments.length / CHARS_PER_TOKEN)
|
||||
+ BLOCK_OVERHEAD
|
||||
break
|
||||
case 'tool-result':
|
||||
tokens += this._estimateContent(block.content) + BLOCK_OVERHEAD
|
||||
break
|
||||
default:
|
||||
// ContentBlockMap is merge-extensible; unknown blocks retain a
|
||||
// conservative structural JSON price under the fixed heuristic.
|
||||
tokens += BLOCK_OVERHEAD + Math.ceil(JSON.stringify(block).length / CHARS_PER_TOKEN)
|
||||
}
|
||||
}
|
||||
return tokens
|
||||
}
|
||||
|
||||
/** Price the canonical non-surface request envelope. */
|
||||
private _estimateHeader(header: EpochHeader | undefined): number {
|
||||
if (header === undefined) return 0
|
||||
let tokens = 0
|
||||
if (header.system !== undefined) {
|
||||
tokens += Math.ceil(header.system.length / CHARS_PER_TOKEN) + ROLE_OVERHEAD
|
||||
}
|
||||
if (header.tools !== undefined && header.tools.length > 0) {
|
||||
tokens += Math.ceil(JSON.stringify(header.tools).length / CHARS_PER_TOKEN) + BLOCK_OVERHEAD
|
||||
}
|
||||
return tokens
|
||||
return providerContent.length === 0 ? 0 : estimateContent(providerContent) + ROLE_OVERHEAD
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -40,11 +40,30 @@ export interface ContextPressureProjection {
|
||||
contextWindow?: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Heuristic composition of the next request's context: what the prompt is
|
||||
* made of, not what it costs. All three figures use the meter's fixed
|
||||
* density estimate (they will not sum exactly to the provider-reported
|
||||
* `pressureTokens`, which is billing-grade and one request behind), and the
|
||||
* message figure tracks the live surface, so it moves as content is appended
|
||||
* or compacted while the provider number holds still.
|
||||
*/
|
||||
export interface ContextBreakdownProjection {
|
||||
/** Heuristic tokens of the newest request envelope's system prompt; 0 before any request. */
|
||||
systemTokens: number
|
||||
/** Heuristic tokens of the newest request envelope's tool schemas; 0 before any request. */
|
||||
toolsTokens: number
|
||||
/** Heuristic tokens of the current model-visible conversation surface. */
|
||||
messageTokens: number
|
||||
}
|
||||
|
||||
declare module '@deepseek-ai/dsh-session-projection/types' {
|
||||
interface SessionProjectionMap {
|
||||
/** Provider-reported usage accumulated across the complete durable log. */
|
||||
tokenUsage: TokenUsageProjection
|
||||
/** Newest request pressure paired with the newest known route capacity. */
|
||||
contextPressure: ContextPressureProjection
|
||||
/** Heuristic system/tools/message composition of the next request. */
|
||||
contextBreakdown: ContextBreakdownProjection
|
||||
}
|
||||
}
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
import type { TokenUsage } from '@deepseek-ai/dsh-llm'
|
||||
|
||||
export type { ContextPressureProjection, TokenUsageProjection } from './projection.ts'
|
||||
export type { ContextBreakdownProjection, ContextPressureProjection, TokenUsageProjection } from './projection.ts'
|
||||
|
||||
/** Token-meter plugin configuration; the fixed estimator has no settings. */
|
||||
export type TokenMeterConfig = Record<string, never>
|
||||
|
||||
@@ -0,0 +1,195 @@
|
||||
// contextBreakdown projection: heuristic system/tools/message composition,
|
||||
// plus the shared estimator's pricing branches.
|
||||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import { createMessage, createUserMessage } from '@deepseek-ai/dsh-llm'
|
||||
import type { ContentBlock, ToolSchema } from '@deepseek-ai/dsh-llm'
|
||||
import SessionStore from '@deepseek-ai/dsh-session'
|
||||
import type { Session, SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import SessionProjectionRegistry from '@deepseek-ai/dsh-session-projection'
|
||||
import TokenMeterService from '@deepseek-ai/dsh-token-meter'
|
||||
import type { ContextBreakdownProjection } from '@deepseek-ai/dsh-token-meter/client'
|
||||
import { contextBreakdownProjectionDefinition } from '../src/breakdown-projection.ts'
|
||||
import {
|
||||
estimateContent,
|
||||
estimateHeader,
|
||||
estimateMessage,
|
||||
estimateSystemTokens,
|
||||
estimateToolsTokens,
|
||||
} from '../src/estimate.ts'
|
||||
|
||||
const CONFIG = { provider: 'test', model: 'test-model' }
|
||||
|
||||
const TOOLS: ToolSchema[] = [{
|
||||
name: 'bash',
|
||||
description: 'run a command',
|
||||
parameters: { type: 'object', properties: {} },
|
||||
}]
|
||||
|
||||
async function harness(): Promise<{ ctx: Context; session: Session }> {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SessionProjectionRegistry)
|
||||
await ctx.plugin(TokenMeterService)
|
||||
return { ctx, session: ctx.sessions.create() }
|
||||
}
|
||||
|
||||
const projected = (ctx: Context, session: Session): ContextBreakdownProjection => {
|
||||
const value = ctx.sessionProjections.snapshot(session).values.contextBreakdown
|
||||
if (value === undefined) throw new Error('contextBreakdown projection is not registered')
|
||||
return value
|
||||
}
|
||||
|
||||
function appendUser(session: Session, text: string): number {
|
||||
return session.append('user/message', createUserMessage({
|
||||
content: [{ type: 'text', text }],
|
||||
source: { kind: 'user' },
|
||||
}), { surfaceOp: 'append' }).seq
|
||||
}
|
||||
|
||||
describe('contextBreakdown session projection', () => {
|
||||
it('serves zeros for an empty log', async () => {
|
||||
const { ctx, session } = await harness()
|
||||
expect(projected(ctx, session)).toEqual({ systemTokens: 0, toolsTokens: 0, messageTokens: 0 })
|
||||
})
|
||||
|
||||
it('prices the newest envelope last-wins and pushes no change for a restated one', async () => {
|
||||
const { ctx, session } = await harness()
|
||||
session.append('request/header', {
|
||||
header: { config: CONFIG, system: 'You are terse.', tools: TOOLS },
|
||||
reason: 'initial',
|
||||
})
|
||||
expect(projected(ctx, session)).toEqual({
|
||||
systemTokens: estimateSystemTokens({ config: CONFIG, system: 'You are terse.' }),
|
||||
toolsTokens: estimateToolsTokens({ config: CONFIG, tools: TOOLS }),
|
||||
messageTokens: 0,
|
||||
})
|
||||
|
||||
const changed: string[] = []
|
||||
ctx.sessionProjections.onChanged((_session, key) => { changed.push(key) })
|
||||
session.append('request/header', {
|
||||
header: { config: CONFIG, system: 'You are terse.', tools: TOOLS },
|
||||
reason: 'change',
|
||||
})
|
||||
session.append('todo/write', { todos: [] })
|
||||
expect(changed).not.toContain('contextBreakdown')
|
||||
|
||||
// A system-less, tool-less envelope prices back to zero.
|
||||
session.append('request/header', { header: { config: CONFIG }, reason: 'change' })
|
||||
expect(projected(ctx, session)).toEqual({ systemTokens: 0, toolsTokens: 0, messageTokens: 0 })
|
||||
})
|
||||
|
||||
it('sums surface appends and skips an empty-content assistant message', async () => {
|
||||
const { ctx, session } = await harness()
|
||||
appendUser(session, 'abcd')
|
||||
session.append('step/start', { turn: 1, step: 1 })
|
||||
session.append('assistant/message', {
|
||||
turn: 1,
|
||||
step: 1,
|
||||
message: createMessage({
|
||||
role: 'assistant',
|
||||
content: [],
|
||||
source: { kind: 'model', provider: 'mock', model: 'mock' },
|
||||
}),
|
||||
usage: { inputTokens: 9, outputTokens: 0 },
|
||||
}, { surfaceOp: 'append', sourceEventSeqs: [] })
|
||||
session.append('step/end', { turn: 1, step: 1 })
|
||||
// 'abcd' prices to 9 (1 text + 4 block + 4 role); the usage-only assistant
|
||||
// message derives to no transcript entry and adds nothing.
|
||||
expect(projected(ctx, session).messageTokens).toBe(9)
|
||||
})
|
||||
|
||||
it('shrinks the message figure when a replacement compacts the surface', async () => {
|
||||
const { ctx, session } = await harness()
|
||||
const first = appendUser(session, 'before compaction, a longer message')
|
||||
const second = appendUser(session, 'and a second entry')
|
||||
const summary = createUserMessage({
|
||||
content: [{ type: 'text', text: 'summary' }],
|
||||
source: { kind: 'plugin', plugin: 'test' },
|
||||
})
|
||||
session.append('user/message', summary, {
|
||||
surfaceOp: { op: 'replace', start: first, end: second },
|
||||
sourceEventSeqs: [first, second],
|
||||
})
|
||||
expect(projected(ctx, session).messageTokens).toBe(estimateMessage(summary))
|
||||
})
|
||||
|
||||
it('fails loud on a replace range absent from the folded surface', () => {
|
||||
const definition = contextBreakdownProjectionDefinition
|
||||
const replace = (start: number, end: number): SessionEvent => ({
|
||||
type: 'user/message',
|
||||
seq: 9,
|
||||
time: 0,
|
||||
data: createUserMessage({ content: [{ type: 'text', text: 'x' }], source: { kind: 'user' } }),
|
||||
surfaceOp: { op: 'replace', start, end },
|
||||
sourceEventSeqs: [start, end],
|
||||
} as unknown as SessionEvent)
|
||||
const append = (seq: number): SessionEvent => ({
|
||||
type: 'user/message',
|
||||
seq,
|
||||
time: 0,
|
||||
data: createUserMessage({ content: [{ type: 'text', text: 'x' }], source: { kind: 'user' } }),
|
||||
surfaceOp: 'append',
|
||||
} as unknown as SessionEvent)
|
||||
let state = definition.init()
|
||||
state = definition.apply(state, append(1))
|
||||
state = definition.apply(state, append(3))
|
||||
expect(() => definition.apply(state, replace(7, 3))).toThrow('invalid current range')
|
||||
expect(() => definition.apply(state, replace(1, 7))).toThrow('invalid current range')
|
||||
expect(() => definition.apply(state, replace(3, 1))).toThrow('invalid current range')
|
||||
})
|
||||
|
||||
it('restores from a JSON checkpoint and unregisters with the token-meter fiber', async () => {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SessionProjectionRegistry)
|
||||
const meterFiber = await ctx.plugin(TokenMeterService)
|
||||
const session = ctx.sessions.create()
|
||||
session.append('request/header', {
|
||||
header: { config: CONFIG, system: 'You are terse.' },
|
||||
reason: 'initial',
|
||||
})
|
||||
appendUser(session, 'abcd')
|
||||
const checkpoint = JSON.parse(JSON.stringify(
|
||||
ctx.sessionProjections.checkpoint(session),
|
||||
)) as ReturnType<typeof ctx.sessionProjections.checkpoint>
|
||||
|
||||
await meterFiber.dispose()
|
||||
expect(ctx.sessionProjections.snapshot(session).values).not.toHaveProperty('contextBreakdown')
|
||||
|
||||
await ctx.plugin(TokenMeterService)
|
||||
expect(ctx.sessionProjections.viewCheckpoint(checkpoint).contextBreakdown).toEqual({
|
||||
systemTokens: estimateSystemTokens({ config: CONFIG, system: 'You are terse.' }),
|
||||
toolsTokens: 0,
|
||||
messageTokens: 9,
|
||||
})
|
||||
})
|
||||
})
|
||||
|
||||
describe('shared estimator', () => {
|
||||
it('prices every content-block shape under the fixed heuristic', () => {
|
||||
expect(estimateContent([{ type: 'text', text: 'abcd' }])).toBe(5)
|
||||
expect(estimateContent([{ type: 'reasoning', text: 'abcdefgh' }] as ContentBlock[])).toBe(6)
|
||||
expect(estimateContent([{ type: 'tool-call', id: 'c' as never, name: 'bash', arguments: '{"a":1}' }])).toBe(7)
|
||||
expect(estimateContent([{
|
||||
type: 'tool-result', toolCallId: 'c' as never,
|
||||
content: [{ type: 'text', text: 'abcd' }],
|
||||
}])).toBe(9)
|
||||
const unknown = { type: 'mystery', payload: 'abc' } as unknown as ContentBlock
|
||||
expect(estimateContent([unknown])).toBe(4 + Math.ceil(JSON.stringify(unknown).length / 4))
|
||||
})
|
||||
|
||||
it('prices envelope parts independently and absent parts to zero', () => {
|
||||
expect(estimateSystemTokens(undefined)).toBe(0)
|
||||
expect(estimateSystemTokens({ config: CONFIG })).toBe(0)
|
||||
expect(estimateSystemTokens({ config: CONFIG, system: 'abcdefgh' })).toBe(6)
|
||||
expect(estimateToolsTokens(undefined)).toBe(0)
|
||||
expect(estimateToolsTokens({ config: CONFIG, tools: [] })).toBe(0)
|
||||
expect(estimateToolsTokens({ config: CONFIG, tools: TOOLS }))
|
||||
.toBe(Math.ceil(JSON.stringify(TOOLS).length / 4) + 4)
|
||||
expect(estimateHeader(undefined)).toBe(0)
|
||||
expect(estimateHeader({ config: CONFIG, system: 'abcdefgh', tools: TOOLS }))
|
||||
.toBe(6 + Math.ceil(JSON.stringify(TOOLS).length / 4) + 4)
|
||||
})
|
||||
})
|
||||
Reference in New Issue
Block a user