Coder/deepseek-harness
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(tools): widen the Unicode-skew obligation past isBareIdentifier

Browse Source 
The predicate is not the only reader of the engine's XID tables. camelCase
reads them through its split set and its head test, and the class name it
derives is emitted for EVERY tool -- including one the predicate rejected,
whose TypedDict is still declared and named. A tool named `zz-` + U+1E4D0
never reaches the skew in the predicate, since the `-` rejects it outright,
yet still emits `class Zz<U+1E4D0>xArgs`, which CPython 3.9.6 refuses the
same way. A backend PR executing "pin the predicate against tables for the
floor" literally would leave that path open, so the note and the docstring
now name all three read points.

Two corrections in the same paragraph. The failing direction is a character
added to XID_Start OR XID_Continue -- one added only to the latter passes
the trailing `\p{XID_Continue}*` in a tail position and fails identically.
And the safe direction routes a name to the subscript/`dict[str, Any]`
path: a rejected FIELD name degrades its whole enclosing object rather than
just itself, which the predicate's opening paragraph already said.

Also qualify the module header's "ONLY source" claim, which holds under
`mode: 'code'` but not `both`, where wireSchemas ships every native schema
alongside the SDK section; record the measured str.isidentifier()
equivalence (21 samples, zero divergence, Node 22.23.1 vs CPython 3.9.6)
where the versions it is relative to already live; and attribute the
`FInd` spelling in the ligature test to full case mapping rather than to
the NFKC step, which is the identity there.

Two tests. The fold-collision half of the childClassName fix: sibling joins
that are byte-distinct before NFKC and equal after, so `usedClassNames`
dedupes by raw bytes and the counter only sees the collision because the
join is normalized. And the argument-side oneOf-of-objects branch naming,
which reaches the same childClassName path the output side already pins.
This commit is contained in:
Chinesezjc
2026-08-05 21:08:24 +08:00
parent af51e68875
commit 2914a87eda
5 changed files with 110 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

66
packages/core/tools/src/py-types.ts
View File

@@ -5,11 +5,12 @@
* store, keyed by the loaded {@link @deepseek-ai/dsh-code-runtime#CodeRuntime.language | code
* runtime's language}.
*
* In Code Mode the native tool schemas are omitted from the request, so this generated SDK is
* the model's ONLY source for each tool's argument names, required fields, types, descriptions,
* and canonical output shapes. Object-shaped arguments and outputs therefore render as one named
* `TypedDict` per tool (and per nested object), not an opaque `dict[str, Any]`, so the shape
* survives into the program.
* Under `mode: 'code'` the native tool schemas are omitted from the request, so this generated
* SDK is the model's ONLY source for each tool's argument names, required fields, types,
* descriptions, and canonical output shapes; under `mode: 'both'` the native schemas ship
* alongside it and it is one of two. Object-shaped arguments and outputs therefore render as one
* named `TypedDict` per tool (and per nested object), not an opaque `dict[str, Any]`, so the
* shape survives into the program under the mode that has nothing else to carry it.
* @module @deepseek-ai/dsh-tools/src/py-types
*/
@@ -30,8 +31,8 @@ const IDENTIFIER = /^[\p{XID_Start}_]\p{XID_Continue}*$/u
*
* Python identifiers are not ASCII: `路径` is as legal a field name as `path`,
* and rejecting it would degrade the whole enclosing object, dropping every
* field's name, requiredness, and type — and in Code Mode the native schemas
* are omitted, so this text is the model's only source for them.
* field's name, requiredness, and type — which under `mode: 'code'` is the
* model's only source for them.
*
* NFKC stability is a second and separate condition, because CPython
* normalizes identifiers at compile time while JSON keys are compared as
@@ -40,23 +41,37 @@ const IDENTIFIER = /^[\p{XID_Start}_]\p{XID_Continue}*$/u
* that normalize together would collapse into one declaration. Those names
* take the subscript path, which carries their exact bytes.
*
* Both conditions are evaluated against the ENGINE's Unicode tables, and the
* two sides are versioned independently — `\p{XID_Start}` follows the running
* engine (Node 22.23.1 reports Unicode 17.0) while CPython follows its own
* (3.9.6 reports 13.0.0). The skew is not symmetric. A CPython older than the
* engine is the dangerous direction: a character added to `XID_Start` since its
* tables (U+1C89, U+10570, U+1E290, U+1E4D0 are all NFKC-stable and accepted
* here, and all rejected by that 3.9.6) is emitted bare and its tokenizer
* refuses the character, taking the whole SDK block down — the same
* parseability invariant {@link UNPRINTABLE}, {@link LONE_SURROGATE} and
* {@link MAX_LIST_NESTING} exist for. A CPython newer than the engine only
* routes a legal name to the subscript path: less readable, still correct. The
* NFKC condition reduces to the same skew, since normalization stability
* guarantees an assigned character's normalization never changes afterwards.
* The equivalence to `str.isidentifier()` was measured across 21 samples with
* zero divergence, on Node 22.23.1 against CPython 3.9.6 — the halves the two
* conditions are proxies for, both tested by that run.
*
* Closing the exposure needs the target interpreter's version, which the
* backend reporting `language: 'python'` owns and which is unpublished on this
* base; the note records it as that PR's decision.
* Both conditions are evaluated against the ENGINE's Unicode tables, and the
* two sides are versioned independently — `\p{XID_Start}`/`\p{XID_Continue}`
* follow the running engine (Node 22.23.1 reports Unicode 17.0) while CPython
* follows its own (3.9.6 reports 13.0.0). The skew is not symmetric. A CPython
* older than the engine is the dangerous direction: a character added to
* either property since its tables (U+1C89, U+10570, U+1E290, U+1E4D0 are all
* NFKC-stable and accepted here, and all rejected by that 3.9.6) is emitted
* bare and its tokenizer refuses the character, taking the whole SDK block
* down — the same parseability invariant {@link UNPRINTABLE},
* {@link LONE_SURROGATE} and {@link MAX_LIST_NESTING} exist for. Both
* properties carry it: a character added only to `XID_Continue` passes the
* trailing `\p{XID_Continue}*` in a tail position and fails the same way. A
* CPython newer than the engine only routes a legal name to the
* subscript/`dict[str, Any]` path: less readable, still correct. The NFKC
* condition reduces to the same skew, since normalization stability guarantees
* an assigned character's normalization never changes afterwards.
*
* This predicate is not the only reader of those tables. {@link camelCase}
* reads them too, through its split set and its head test, and its output is
* emitted for EVERY tool — including one this predicate rejected, whose
* `TypedDict` is still declared and named. A tool named `zz-\u{1E4D0}x` never
* reaches the skew here (the `-` rejects it outright) yet emits
* `class Zz\u{1E4D0}xArgs`, which that same 3.9.6 refuses. Closing the
* exposure therefore covers all three read points, not this predicate alone;
* it needs the target interpreter's version, which the backend reporting
* `language: 'python'` owns and which is unpublished on this base, so the note
* records it as that PR's decision.
*
* The `ts-types` sibling keeps its own ASCII rule rather than sharing this
* one: ECMAScript identifiers are a different set (`$`, ZWJ/ZWNJ) and are
@@ -221,6 +236,11 @@ function docLines(description: unknown, indent: number): string[] {
* a combining-mark head composes there (`U+0301` gives `Tooĺ`, U+013A), so
* normalizing only the un-prefixed part would emit a name CPython compiles to
* a different symbol. The second call is idempotent on the un-prefixed arm.
*
* The split set and the head test read the engine's Unicode tables, so this
* function carries the same version skew {@link isBareIdentifier} documents,
* by an independent path: a class name derived here is emitted for every tool,
* including one the predicate rejected.
* @param raw - the schema field or tool name to derive from.
* @returns a class-name segment safe to emit.
*/

65
packages/core/tools/tests/py-types.spec.ts
View File

@@ -453,8 +453,11 @@ describe('renderToolsSdkPy', () => {
// ligature name cannot, because `async def find` would define `find`. The
// subscript comment quotes the name, so its exact bytes survive, and its
// TypedDict is still named and referenced — the name is only unusable as a
// method, not as a class-name source (`camelCase` normalizes what it
// derives, since a generated name is never matched against a JSON key).
// method, not as a class-name source. The `FInd` spelling comes from `fi`'s
// multi-character full case mapping (`'fi'.toUpperCase()` is `'FI'`), not
// from `camelCase`'s NFKC step, which is the identity on `FInd`: the
// ligature is XID_Start, so the split set keeps it and only the
// capitalization of the head transforms it.
const of = (name: string): ToolSdkSchema => ({
name,
description: `Tool ${name}.`,
@@ -559,6 +562,64 @@ describe('renderToolsSdkPy', () => {
expect(text).toContain('class XArgs\uAC00\u1100(TypedDict):')
})
it('routes a fold collision through the counter that raw-byte dedup would miss', () => {
// The other half of the `childClassName` normalization: two joins that are
// byte-distinct before NFKC and identical after. Field `\uAC00` allocates
// `XArgs\uAC00`; the sibling `\u1100` allocates `XArgs\u1100`, and ITS child
// `\u1161` joins to `XArgs\u1100\u1161` — the same `XArgs\uAC00` once composed.
// Normalizing at the join is what lets `usedClassNames`, which dedupes by raw
// bytes, see the collision at all; unnormalized, both would be declared and
// CPython would compile the second as a shadow of the first.
const text = renderToolsSdkPy([
{
name: 'x',
description: 'Colliding jamo joins.',
parameters: {
type: 'object',
additionalProperties: false,
required: ['\uAC00', '\u1100'],
properties: {
'\uAC00': { type: 'object', additionalProperties: false, required: ['q'], properties: { q: { type: 'string' } } },
'\u1100': {
type: 'object',
additionalProperties: false,
required: ['\u1161'],
properties: {
'\u1161': { type: 'object', additionalProperties: false, required: ['q'], properties: { q: { type: 'string' } } },
},
},
},
},
output: { type: 'string' },
},
])
expect(text).toContain('class XArgs\uAC00(TypedDict):')
expect(text).toContain('class XArgs\uAC002(TypedDict):')
expect(text).toContain(' \u1161: XArgs\uAC002')
})
it('names both branches of a oneOf of objects on the argument side', () => {
// The output side is pinned elsewhere; arguments reach the same
// `childClassName(frame.className, index + 1)` path, and the annotation is
// the union of the two derived names rather than a degraded dict.
const text = renderToolsSdkPy([
{
name: 'x',
description: 'Union arguments.',
parameters: {
oneOf: [
{ type: 'object', additionalProperties: false, required: ['a'], properties: { a: { type: 'string' } } },
{ type: 'object', additionalProperties: false, required: ['b'], properties: { b: { type: 'number' } } },
],
},
output: { type: 'string' },
},
])
expect(text).toContain('class XArgs1(TypedDict):')
expect(text).toContain('class XArgs2(TypedDict):')
expect(text).toContain('async def x(self, args: XArgs1 | XArgs2) -> str:')
})
it('declares a closed empty object with omitted properties as an empty TypedDict, not dict[str, Any]', () => {
// `{ type: 'object', additionalProperties: false }` with no `properties`
// is a closed empty object — no key accepted — exactly as the validator