review(tool-goal,llm-replay): address ds-review-bot round one
- fromRequest placeholder: the last two braces of a consecutive } run now
terminate the placeholder, so patterns may end with a brace quantifier
(bot warning; the truncated pattern could even silently mis-match since
an unclosed { is literal in JS regexes)
- document that derived JSONL entries pass through the same resolution
- widen ToolRunContext/deferContext seam docs beyond composite-only usage
(source JSDoc, README pair, core-data-structures type-equiv blocks)
- pin direct-human blocked as uninstructed, completing the
goal-round/direct-human x complete/blocked test quadrant
This commit is contained in:
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/support/llm-replay/README.md
|
||||
README.md: ea52525ee85aae58006c852afe93291ea70807d5
|
||||
README.zh.md: e8c0ec225df29fc6f5493776d75bc7f9e3e078de
|
||||
README.md: 85aa56705929e7630e4cfb6c2a3c9cbbd0d843a6
|
||||
README.zh.md: 751f75dea197ffb112cfa703e3a5dbfaffb8c0b2
|
||||
|
||||
@@ -12,7 +12,7 @@ The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assi
|
||||
|
||||
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`) that either replaces the derived script (a bare `ReplayEntry[]`) or augments it (`{ patches: [{ at, entry }] }`: keep every JSONL-derived call and swap the named 0-based call indexes; `at` equal to the derived length appends the retry attempt after an injected transient throw). Patch indexes must be unique. The override document, each patch and entry, and every chunk discriminant are validated when the file loads. A `hang` entry may name `readyFile`; replay writes that empty marker after its prefix chunks reach the loop and before it waits for cancellation, so an external driver can cancel deterministically without observing a presentation update.
|
||||
|
||||
A scripted string may embed `{{fromRequest:<regex>}}` to fill a value no static sidecar can know — for example a randomly minted goal id the model must echo back into `update_goal`. At stream time every placeholder resolves against the live request: the corpus is every string leaf of the request messages joined by newlines, the pattern's LAST corpus match wins, and its first capture group (or the whole match without one) substitutes in place. A pattern that matches nothing, an invalid pattern, and an unterminated placeholder each fail loud; the first `}}` ends the placeholder, so patterns cannot contain `}}`.
|
||||
A scripted string may embed `{{fromRequest:<regex>}}` to fill a value no static sidecar can know — for example a randomly minted goal id the model must echo back into `update_goal`. At stream time every placeholder resolves against the live request: the corpus is every string leaf of the request messages joined by newlines, the pattern's LAST corpus match wins, and its first capture group (or the whole match without one) substitutes in place. A pattern that matches nothing, an invalid pattern, and an unterminated placeholder each fail loud. The last two braces of a consecutive `}` run terminate the placeholder, so a pattern may end with a brace quantifier (`[0-9a-f]{4}`) but cannot contain `}}` followed by further pattern content. Resolution applies to every scripted entry, including ones derived from the recorded JSONL — a recorded fixture whose text legitimately contains the literal marker must be expressed through a sidecar without it.
|
||||
|
||||
## Nested agents: per-session keying
|
||||
|
||||
|
||||
@@ -12,7 +12,7 @@ fixture 就是持久化的会话日志(`<scenario>/session.jsonl`)。其 `as
|
||||
|
||||
有两种失败模式无法仅根据 `assistant/chunk` 重建:在产生任何分片前直接抛出异常(例如 HTTP 401,此时日志只有 `turn/end {error}` 而没有分片),以及取消或挂起(差异在时序,而非分片内容)。需要这些行为的场景可提供伴随文件(`<scenario>/replay.override.json`):它可以替换派生脚本(裸 `ReplayEntry[]`),也可以增补派生脚本(`{ patches: [{ at, entry }] }`:保留所有从 JSONL 派生的调用,只替换指定的从 0 开始计数的调用索引;当 `at` 等于派生长度时,则在注入瞬态异常后的重试位置追加一次调用)。补丁索引不得重复。文件加载时会校验覆写文档、每个补丁和条目,以及每个分片的判别标签。`hang` 条目可以指定 `readyFile`;当前缀分片到达循环后、开始等待取消前,回放会写入这个空标记,使外部驱动程序无需观察展示层更新即可确定性地取消。
|
||||
|
||||
脚本字符串可以内嵌 `{{fromRequest:<regex>}}`,用来填入静态伴随文件不可能预知的值——例如模型必须原样回填到 `update_goal` 的随机生成 goal id。回放时每个占位符针对实时请求解析:语料是请求消息的所有字符串叶子按换行拼接的结果,取该模式在语料中的最后一次匹配,用其第一个捕获组(无捕获组时用整个匹配)原位替换。模式匹配不到内容、模式非法、占位符未闭合都会明确报错;第一个 `}}` 即结束占位符,因此模式本身不能包含 `}}`。
|
||||
脚本字符串可以内嵌 `{{fromRequest:<regex>}}`,用来填入静态伴随文件不可能预知的值——例如模型必须原样回填到 `update_goal` 的随机生成 goal id。回放时每个占位符针对实时请求解析:语料是请求消息的所有字符串叶子按换行拼接的结果,取该模式在语料中的最后一次匹配,用其第一个捕获组(无捕获组时用整个匹配)原位替换。模式匹配不到内容、模式非法、占位符未闭合都会明确报错。连续右花括号串的最后两个花括号才是占位符结束符,因此模式可以以花括号量词收尾(如 `[0-9a-f]{4}`),但不能在 `}}` 之后还有后续模式内容。解析作用于所有脚本条目,包括从已记录 JSONL 派生的条目——若录制文本本身合法地含有该字面量标记,需改用不含标记的伴随文件表达。
|
||||
|
||||
## 嵌套 agent:每会话键控
|
||||
|
||||
|
||||
@@ -283,10 +283,13 @@ function substituteString(text: string, corpus: string): string {
|
||||
while (true) {
|
||||
const open = text.indexOf(FROM_REQUEST_OPEN, cursor)
|
||||
if (open === -1) return result + text.slice(cursor)
|
||||
const close = text.indexOf(FROM_REQUEST_CLOSE, open + FROM_REQUEST_OPEN.length)
|
||||
let close = text.indexOf(FROM_REQUEST_CLOSE, open + FROM_REQUEST_OPEN.length)
|
||||
if (close === -1) {
|
||||
throw new Error(`llm-replay: fromRequest placeholder is unterminated in ${JSON.stringify(text)}`)
|
||||
}
|
||||
// The last two braces of a consecutive `}` run terminate the placeholder,
|
||||
// so a pattern may end with a brace quantifier like `[0-9a-f]{4}`.
|
||||
while (text[close + FROM_REQUEST_CLOSE.length] === '}') close += 1
|
||||
const pattern = text.slice(open + FROM_REQUEST_OPEN.length, close)
|
||||
result += text.slice(cursor, open) + resolveFromRequest(pattern, corpus)
|
||||
cursor = close + FROM_REQUEST_CLOSE.length
|
||||
@@ -313,7 +316,10 @@ function substituteValue(value: unknown, corpus: string): unknown {
|
||||
* Scenario sidecars use this to script arguments no static file can know,
|
||||
* such as a randomly minted goal id the model must echo back. A pattern that
|
||||
* matches nothing, an invalid pattern, and an unterminated placeholder each
|
||||
* fail loud. Patterns cannot contain `}}` — the first `}}` ends the placeholder.
|
||||
* fail loud. The last two braces of a consecutive `}` run terminate the
|
||||
* placeholder, so a pattern may end with a brace quantifier but cannot
|
||||
* contain `}}` followed by further pattern content. Derived entries pass
|
||||
* through the same resolution as sidecar entries.
|
||||
* @param entry - the scripted entry about to replay.
|
||||
* @param messages - the live request messages searched by the placeholders.
|
||||
* @returns the entry itself when no placeholder appears, else a resolved deep copy.
|
||||
|
||||
@@ -350,6 +350,12 @@ describe('installLlmReplay (through the real LlmService)', () => {
|
||||
expect(delta).toMatchObject({ argumentsDelta: '{"goal_id":"goal-42ab"}' })
|
||||
})
|
||||
|
||||
it('keeps a trailing brace quantifier inside the pattern (terminator is the run tail)', async () => {
|
||||
const streamed = await streamScripted('{"goal_id":"{{fromRequest:goal-[0-9a-z]{4}}}"}')
|
||||
const delta = streamed.find(chunk => chunk.type === 'tool-call-delta')
|
||||
expect(delta).toMatchObject({ argumentsDelta: '{"goal_id":"goal-42ab"}' })
|
||||
})
|
||||
|
||||
it('fails loud when a placeholder matches nothing in the request', async () => {
|
||||
await expect(streamScripted('{"goal_id":"{{fromRequest:task-[0-9]+}}"}'))
|
||||
.rejects.toThrow(/fromRequest.*matched nothing/)
|
||||
|
||||
Reference in New Issue
Block a user