docs: document package KV cache effects

This commit is contained in:
Tianyi Cui
2026-07-19 17:39:50 +08:00
parent b3c4b732d8
commit bcc920369c
93 changed files with 366 additions and 34 deletions

View File

@@ -89,18 +89,24 @@ Every `SessionEvent` carries two optional top-level fields (structural metadata)
**Token effect**: Appended surface entries are resent on later steps. A `replace` surface operation removes the shadowed entries from future inputs without deleting their raw log records.
**KV Cache effect**: Appended surface entries preserve reusable prefixes. A `replace` operation invalidates reuse from the first shadowed message even though the underlying event log stays append-only.
### Crash-repair result
**What the model sees**: If a persisted turn ended with unanswered tool calls, each synthetic error result contains exactly `Tool call interrupted by a crash; no result was recorded.`
**Token effect**: Zero tokens in an intact session. Each repaired call adds this retained error text on resume.
**KV Cache effect**: Append-only; newly visible content follows the reusable request prefix and does not invalidate existing KV-cache entries.
### Logged request header
**What the model sees**: The session reconstructs the system prompt, tool schemas, call config, and session prefix that the loop actually sent. Header events do not add a second copy to message history; the prefix is prepended outside `deriveMessages()`.
**Token effect**: Zero duplicate tokens from logging. The reconstructed prefix, system text, and schemas still incur their normal per-request cost.
**KV Cache effect**: Logging causes no invalidation, and exact reconstruction preserves request-prefix identity. A later header with changed prefix, prompt, or schemas may invalidate reuse from its first difference.
## Known Limitations and Deferred Work
- **Session branching/tree** (pi-style entry tree) — deferred unless needed beyond boundary-based `fork()`.