Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test(sandbox-policy): record cache-safe policy context

Browse Source
This commit is contained in:
NI0317
2026-07-30 22:15:34 +08:00
parent 9c32620e4b
commit 1a4349be88
90 changed files with 2468 additions and 2411 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
.agents/notes/implemented/feature/2026-07-30-current-sandbox-policy-context.i18n.yaml
View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-07-30-current-sandbox-policy-context.md
2026-07-30-current-sandbox-policy-context.md: 16805cc16242dca825dcc58b1e0c7baa4b1fbd0d
2026-07-30-current-sandbox-policy-context.zh.md: 9ef0dcc268ecbd78b2fb384ac09fd39f6b6c86d5
2026-07-30-current-sandbox-policy-context.md: 20920dd5842cd6e97a1b5d9f80a3646d57e0eec8
2026-07-30-current-sandbox-policy-context.zh.md: b03e6f49df10db8fbe576613dd3ab64059d87c99

4
.agents/notes/implemented/feature/2026-07-30-current-sandbox-policy-context.md
View File

@@ -28,7 +28,9 @@ The earlier real-provider Web fixture quantified the defect in the system-sectio
The wording experiment pre-registered preemptive refusal as its primary endpoint and required the old standing sentence to produce at least one refusal in twelve fresh sessions before any replacement could be judged. On 2026-07-30, commit `2bf41990401b194bd8637f07bbd90c67a9eeac75` ran `deepseek-v4-flash` through the shipped Web composition with the exact positive-control sentence `Bash commands run under the "read-only" file sandbox.` and the current tool-owned attempt guidance. The control produced zero preemptive refusals and zero speculative escalations; all twelve sessions made an ordinary bash call, observed a denial, escalated in the same turn, received approval, and landed the requested file. No sample was excluded.
The positive control therefore failed the pre-registered sensitivity gate. Candidate A and B were not run, and this experiment does not select or validate the current wording. It instead establishes that the earlier five-of-twelve result is not reproducible under this task and current tool guidance, and that a stronger positive control or different task distribution is required before making model-behavior rate claims. Deterministic tests below establish truthful request construction and replay only.
After the cache-safe delivery change, commit `10d4e0ff7b68d38fc4403403b644aac442b97a00` repeated the same twelve-session positive control through the new tail-context channel. It again produced zero preemptive refusals and zero speculative escalations; all twelve sessions made an ordinary first call, observed denial, escalated in the same turn, and received approval. Eight landed the exact requested file, and no sample was excluded.
Both positive controls therefore failed the pre-registered sensitivity gate. The formal twelve-session Candidate A and B arms were not run, and these experiments do not select or validate the current wording. They establish that the earlier five-of-twelve result is not reproducible under this task and current tool guidance, and that a stronger positive control or different task distribution is required before making model-behavior rate claims. Deterministic tests below establish truthful request construction and replay only.
The cache-safe delivery rework then supplied a separate, non-statistical acceptance comparison over the neutral Web task `Create the relative path policy-neutral.txt ...`; it does not replace the pre-registered twelve-session experiment. Candidate A's categorical read-only statement produced a text refusal with zero tool calls. Candidate B added one composition-conditioned sentence only for enforced families whose tools expose escalation. A fresh real-provider run then issued an ordinary `write`, observed the read-only denial, retried the same operation in the same turn with `sandbox_permissions: "workspace-write"`, received approval, read the file back, and verified the exact contents. It made no speculative escalation. Across the permission switches and four mutation steps, cache reads were 14,848–15,872 tokens while uncached input was 59–251 tokens per request, directly demonstrating the stable-prefix benefit.

4
.agents/notes/implemented/feature/2026-07-30-current-sandbox-policy-context.zh.md
View File

@@ -28,7 +28,9 @@ Status: implemented
措辞实验预先登记「预防性拒绝」为主要终点，并要求旧常驻句子在十二个 fresh session 中至少产生一次拒绝，之后才能评判任何替代措辞。2026-07-30，commit `2bf41990401b194bd8637f07bbd90c67a9eeac75` 通过已交付的 Web 组合运行 `deepseek-v4-flash`，使用精确的阳性对照句子 `Bash commands run under the "read-only" file sandbox.` 与当前工具归属方的尝试引导。对照组产生零次预防性拒绝和零次推测性升级；十二个会话全部先发起普通 bash 调用、观察到拒绝、在同一轮次升级、获得批准，并让所请求文件实际落盘。没有样本被排除。
因此，阳性对照未通过预先登记的灵敏度门槛。Candidate A 与 B 均未运行，本实验不选择也不验证当前措辞。它只说明先前十二次中五次的结果无法在本任务与当前工具引导下复现；在声明模型行为率之前，需要更强的阳性对照或不同的任务分布。下述确定性测试只证明请求构造与回放真实一致。
缓存安全交付变更后，commit `10d4e0ff7b68d38fc4403403b644aac442b97a00` 通过新的尾部上下文通道重复了同一项十二会话阳性对照。结果再次为零次预防性拒绝和零次推测性升权；十二个会话的首次调用均为普通调用，随后观察到拒绝、在同一轮次升权并获得批准。其中八个会话让所请求文件按确切要求落盘，没有样本被排除。
因此，两项阳性对照均未通过预先登记的灵敏度门槛。Candidate A 与 B 的正式十二会话实验组均未运行，这些实验不选择也不验证当前措辞。它们说明先前十二次中五次的结果无法在本任务与当前工具引导下复现；在声明模型行为率之前，需要更强的阳性对照或不同的任务分布。下述确定性测试只证明请求构造与回放真实一致。
随后，缓存安全交付重做针对中性 Web 任务 `Create the relative path policy-neutral.txt ...` 提供了一次独立的非统计验收对比；它不取代预先登记的十二会话实验。Candidate A 的绝对化只读声明导致模型以纯文本拒绝，工具调用为零。Candidate B 只针对受强制执行、且其工具公开升权能力的家族增加一句按组合条件化的文案。随后一次全新的真实提供方运行先发出普通 `write`，观察到只读拒绝，再在同一轮次用 `sandbox_permissions: "workspace-write"` 重试同一操作，获得批准、读回文件并核验确切内容。它没有进行推测性升权。在权限切换和四个变更步骤中，每个请求的缓存读取为 14,848–15,872 个 token，未缓存输入为 59–251 个 token，直接证明了稳定前缀的收益。