refactor(e2b): compose portable runtime consumers

This commit is contained in:
Tianyi Cui
2026-07-29 02:51:44 +08:00
parent 97496c3d00
commit 2abc9823e7
64 changed files with 1426 additions and 4229 deletions

View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write packages/e2b/fs-e2b/README.md
README.md: a505703fc764f8441fa54d2b1d922762eb3fafdf
README.zh.md: 626fdb52979d29f12d1bc1b67b0ea13730003af7
README.md: 86ad8720d3e4c7ce70ee0ac8c41713af03ed297a
README.zh.md: 90dff57a5b9fe786aea3315633125536277ecc20

View File

@@ -7,7 +7,9 @@ E2B implementation of the [`@deepseek-ai/dsh-fs`](../../fs/fs/README.md) provide
## Behavior
- **Remote identity and metadata** — relative paths resolve as POSIX paths against the caller cwd or `ctx.e2b.cwd`; `realpath -m` supplies canonical target identity without requiring the final file to exist. `stat`, no-follow `lstat`, and stable one-level directory listings project E2B metadata into the filesystem seam. Versions are opaque hashes of E2B metadata plus a per-write extended attribute.
- **Execution-world paths** — canonical targets expose absolute POSIX process paths, percent-encoded `file:` URIs, and provider-owned containment checks, so generic subprocess consumers never parse E2B target ids or apply host path rules.
- **UTF-8 reads** — whole reads and streamed reads preserve cross-chunk decoding, reject invalid UTF-8, and use the seam's 8192-byte NUL sample for binary detection. The model-facing tool still owns size selection and line windowing.
- **Stable bounded reads** — a dependency-free Node helper walks directory descriptors with no-follow opens and reads one held regular-file descriptor through the byte cap. Generic LSP queries therefore reject parent swaps, non-files, invalid UTF-8, and growth past the configured document limit before server startup.
- **Atomic mutations** — writes upload a mode-`0600` temporary sibling, preserve an existing file's POSIX mode, and publish through E2B's same-directory atomic rename. The rename response supplies the committed version, so no fallible metadata request follows the commit point. E2B creates missing parent directories. Literal edits LF-normalize for matching, restore dominant CRLF storage, and serialize mutations per canonical target within the host process. Optional create/version guards keep the base seam's observed-state semantics.
- **Failures and cancellation** — E2B not-found, permission, abort, and other controller failures map to the existing `FsError` vocabulary. Cancellation is best-effort at SDK request boundaries; a successful rename is the commit point.
@@ -26,4 +28,4 @@ No direct invalidation; the named consumer owns any request-prefix changes.
- **No host synchronization** — an empty E2B cwd stays empty until a tool, command, template, or external process populates it; local files are neither uploaded nor reflected back.
- **Mutation coordination is host-process-local** — another harness connection or remote command can race the adapter; version guards detect only metadata changes represented by E2B.
- **Whole-file mutation costs remain** — overwrite diffs and literal edits read complete files into host memory, and every operation incurs E2B controller latency.
- **Custom templates must support the used Linux and envd features** — `realpath`, `chmod`, `mv`, same-filesystem POSIX rename, streaming reads, and file metadata extended attributes are required; unsupported templates fail rather than degrade silently.
- **Custom templates must support the used Linux, Node, procfs, and envd features** — `realpath`, `chmod`, `mv`, same-filesystem POSIX rename, streaming reads, file metadata extended attributes, `/proc/self/fd`, and no-follow descriptor opens are required; unsupported templates fail rather than degrade silently.

View File

@@ -7,7 +7,9 @@
## 行为
- **远程身份与元数据**:相对路径以调用方 cwd 或 `ctx.e2b.cwd` 为基准,按照 POSIX 路径解析;`realpath -m` 提供规范化目标身份,且不要求最终文件存在。`stat`、不跟随链接的 `lstat` 和稳定的单层目录列表会把 E2B 元数据投影到文件系统 seam。版本是 E2B 元数据与每次写入设置的扩展属性所组成的不透明哈希。
- **执行世界路径**:规范化目标公开绝对 POSIX 进程路径、百分号编码的 `file:` URI以及由提供方负责的包含关系检查因此通用进程管理消费方无需解析 E2B 目标 ID也不会套用宿主路径规则。
- **UTF-8 读取**:完整读取和流式读取会保留跨分片解码、拒绝无效 UTF-8并使用 seam 的 8192 字节 NUL 样本检测二进制内容。面向模型的工具仍负责选择大小和行窗口。
- **稳定的有界读取**:一个零依赖 Node 辅助程序会以不跟随链接的方式逐级打开目录描述符,并通过一个持续持有的常规文件描述符读取至字节上限。因此,通用 LSP 查询会在服务器启动前拒绝父目录交换、非文件、无效 UTF-8以及增长后超出所配置文档上限的文件。
- **原子变更**:写入会上传 mode 为 `0600` 的同级临时文件,保留现有文件的 POSIX mode并通过 E2B 的同目录原子重命名发布。重命名响应会提供已提交的版本因此提交点之后不会再进行可能失败的元数据请求。E2B 会创建缺失的父目录。字面量编辑匹配时会规范化为 LF存储时恢复占主导的 CRLF并在宿主进程内按规范化目标串行执行变更。可选的创建版本防护会保留基础 seam 的已观察状态语义。
- **失败与取消**E2B 的未找到、权限、中止及其他控制器故障会映射到现有 `FsError` 词汇。取消在 SDK 请求边界上采用尽力而为语义;成功 rename 是提交点。
@@ -26,4 +28,4 @@
- **不提供宿主同步**:空的 E2B cwd 会一直为空,直到工具、命令、模板或外部进程填充它;本地文件既不会上传,也不会同步回本地。
- **变更协调仅限宿主进程内**:另一个 harness 连接或远程命令可能与适配器发生竞态;版本防护只能检测 E2B 元数据所体现的变更。
- **仍需承担完整文件变更成本**:覆盖差异和字面量编辑会把完整文件读入宿主内存,每项操作也都会产生 E2B 控制器延迟。
- **自定义模板必须支持所用的 Linux 与 envd 功能**:必须支持 `realpath``chmod``mv`、同一文件系统内的 POSIX rename、流式读取文件元数据扩展属性;不支持的模板会失败,而不会静默降级。
- **自定义模板必须支持所用的 Linux、Node、procfs 与 envd 功能**:必须支持 `realpath``chmod``mv`、同一文件系统内的 POSIX rename、流式读取文件元数据扩展属性`/proc/self/fd` 和不跟随链接的描述符打开操作;不支持的模板会失败,而不会静默降级。

View File

@@ -5,6 +5,7 @@
*/
import { createHash, randomUUID } from 'node:crypto'
import { Buffer } from 'node:buffer'
import { posix } from 'node:path'
import { FileSystem, FsError, FsTargetKey, FsVersion } from '@deepseek-ai/dsh-fs'
import type {
@@ -24,10 +25,18 @@ import {
quoteE2BShellArg,
} from '@deepseek-ai/dsh-e2b'
import type { EntryInfo, Sandbox } from '@deepseek-ai/dsh-e2b'
import { BOUNDED_READER_SOURCE } from './source-reader.ts'
const VERSION_METADATA_KEY = 'dsh-version'
const BINARY_SAMPLE_BYTES = 8192
type BoundedReadResponse =
| { kind: 'ok'; data: string }
| { kind: 'not-file' }
| { kind: 'oversize'; size: number }
| { kind: 'grew' }
| { kind: 'open-error'; message: string }
function assertNotAborted(signal: AbortSignal | undefined, operation: string): void {
if (signal?.aborted === true) throw new FsError(`${operation} aborted`, 'FS_ABORTED')
}
@@ -141,6 +150,21 @@ export class E2BFileSystem extends FileSystem {
}
}
override processPath(target: FsTarget): string {
return String(target.targetKey)
}
override fileUrl(target: FsTarget): string {
const path = this.processPath(target)
if (!posix.isAbsolute(path)) throw new Error(`fs-e2b: expected an absolute process path: ${JSON.stringify(path)}`)
return `file://${path.split('/').map(segment => encodeURIComponent(segment)).join('/')}`
}
override contains(parent: FsTarget, child: FsTarget): boolean {
const relative = posix.relative(this.processPath(parent), this.processPath(child))
return relative === '' || (relative !== '..' && !relative.startsWith('../') && !posix.isAbsolute(relative))
}
override async stat(target: FsTarget, signal?: AbortSignal): Promise<FsInfo | undefined> {
assertNotAborted(signal, 'stat')
const entry = await this.probe(String(target.targetKey), target.displayPath, signal)
@@ -184,6 +208,69 @@ export class E2BFileSystem extends FileSystem {
}
}
override async readTextBounded(target: FsTarget, maxBytes: number, signal?: AbortSignal): Promise<string> {
if (!Number.isSafeInteger(maxBytes) || maxBytes <= 0) {
throw new Error('bounded read maxBytes must be a positive safe integer')
}
assertNotAborted(signal, 'read')
const sandbox = await this.ctx.e2b.getSandbox()
try {
const node = await sandbox.commands.run('command -v -- node', signalOpts(signal))
const executable = node.stdout.trim()
if (!posix.isAbsolute(executable) || executable.includes('\n')) {
throw new Error('fs-e2b: bounded reader requires one absolute Node executable')
}
const command = [
quoteE2BShellArg(executable),
'--input-type=commonjs',
'-e',
quoteE2BShellArg(BOUNDED_READER_SOURCE),
quoteE2BShellArg(this.processPath(target)),
String(maxBytes),
].join(' ')
const result = await sandbox.commands.run(command, signalOpts(signal))
assertNotAborted(signal, 'read')
const response = this.parseBoundedRead(result.stdout, target)
if (response.kind === 'not-file') {
throw new FsError(`cannot read "${target.displayPath}": not a regular file`, 'FS_NOT_REGULAR_FILE')
}
if (response.kind === 'oversize' && Number.isSafeInteger(response.size)) {
throw new FsError(
`cannot read "${target.displayPath}": ${response.size} bytes exceeds the ${maxBytes}-byte limit`,
'FS_IO_ERROR',
)
}
if (response.kind === 'grew') {
throw new FsError(
`cannot read "${target.displayPath}": file grew past the ${maxBytes}-byte limit while reading`,
'FS_IO_ERROR',
)
}
if (response.kind === 'open-error' && typeof response.message === 'string') {
if (/ENOENT|no such file/i.test(response.message)) {
throw new FsError(`cannot read "${target.displayPath}": not found`, 'FS_NOT_FOUND')
}
if (/EACCES|EPERM|permission denied|operation not permitted/i.test(response.message)) {
throw new FsError(`cannot read "${target.displayPath}": permission denied`, 'FS_PERMISSION_DENIED')
}
throw new FsError(
`cannot read "${target.displayPath}" safely: ${response.message}`,
'FS_IO_ERROR',
)
}
if (response.kind !== 'ok' || typeof response.data !== 'string') {
throw new FsError(`cannot read "${target.displayPath}": bounded reader returned an invalid response`, 'FS_IO_ERROR')
}
const bytes = Buffer.from(response.data, 'base64')
if (bytes.toString('base64') !== response.data || bytes.length > maxBytes) {
throw new FsError(`cannot read "${target.displayPath}": bounded reader returned invalid bytes`, 'FS_IO_ERROR')
}
return decodeText(bytes, target.displayPath, BINARY_SAMPLE_BYTES)
} catch (error: unknown) {
throw mapError(error, 'read', target.displayPath, signal)
}
}
override async streamText(target: FsTarget, signal?: AbortSignal): Promise<AsyncIterable<string>> {
const sandbox = await this.ctx.e2b.getSandbox()
await this.requireRegular(target, signal)
@@ -336,6 +423,18 @@ export class E2BFileSystem extends FileSystem {
}
}
private parseBoundedRead(stdout: string, target: FsTarget): BoundedReadResponse {
try {
return JSON.parse(stdout) as BoundedReadResponse
} catch (error: unknown) {
throw new FsError(
`cannot read "${target.displayPath}": bounded reader returned invalid JSON`,
'FS_IO_ERROR',
{ cause: error },
)
}
}
private async probe(path: string, displayPath: string, signal?: AbortSignal): Promise<EntryInfo | undefined> {
assertNotAborted(signal, 'stat')
try {

View File

@@ -0,0 +1,63 @@
/** Dependency-free stable-handle bounded reader installed inside E2B. */
export const BOUNDED_READER_SOURCE = String.raw`
/* dsh-e2b-bounded-reader */
const fs = require('node:fs')
const target = process.argv[1]
const maxBytes = Number(process.argv[2])
const directoryFlags = fs.constants.O_RDONLY | fs.constants.O_DIRECTORY | fs.constants.O_NOFOLLOW | fs.constants.O_NONBLOCK
const fileFlags = fs.constants.O_RDONLY | fs.constants.O_NOFOLLOW | fs.constants.O_NONBLOCK
let directory
let descriptor
let response
const openChild = (parent, component, flags) => fs.openSync('/proc/self/fd/' + parent + '/' + component, flags)
const invalidComponent = component => component === '' || component === '.' || component === '..'
try {
if (typeof target !== 'string' || !target.startsWith('/') || !Number.isSafeInteger(maxBytes) || maxBytes <= 0) {
throw new Error('bounded reader requires an absolute target and positive byte limit')
}
const components = target === '/' ? [] : target.slice(1).split('/')
if (components.length === 0 || components.some(invalidComponent)) {
throw new Error('bounded reader received a non-canonical file path')
}
directory = fs.openSync('/', directoryFlags)
for (const component of components.slice(0, -1)) {
const child = openChild(directory, component, directoryFlags)
fs.closeSync(directory)
directory = child
}
descriptor = openChild(directory, components.at(-1), fileFlags)
const info = fs.fstatSync(descriptor)
if (!info.isFile()) response = { kind: 'not-file' }
else if (info.size > maxBytes) response = { kind: 'oversize', size: info.size }
else {
const chunks = []
let total = 0
while (total <= maxBytes) {
const chunk = Buffer.allocUnsafe(Math.min(65536, maxBytes - total + 1))
const bytesRead = fs.readSync(descriptor, chunk, 0, chunk.length, null)
if (bytesRead === 0) break
chunks.push(chunk.subarray(0, bytesRead))
total += bytesRead
}
response = total > maxBytes
? { kind: 'grew' }
: { kind: 'ok', data: Buffer.concat(chunks, total).toString('base64') }
}
} catch (error) {
response = { kind: 'open-error', message: error instanceof Error ? error.message : String(error) }
} finally {
for (const openDescriptor of [descriptor, directory]) {
if (openDescriptor === undefined) continue
try {
fs.closeSync(openDescriptor)
} catch (error) {
response = { kind: 'open-error', message: error instanceof Error ? error.message : String(error) }
}
}
}
process.stdout.write(JSON.stringify(response))
`

View File

@@ -1,3 +1,4 @@
import { Buffer } from 'node:buffer'
import { dirname, posix } from 'node:path'
import { Context } from 'cordis'
import {
@@ -8,7 +9,7 @@ import {
type Sandbox,
} from '@deepseek-ai/dsh-e2b'
import type E2BSandboxService from '@deepseek-ai/dsh-e2b'
import { FsVersion } from '@deepseek-ai/dsh-fs'
import { FsTargetKey, FsVersion } from '@deepseek-ai/dsh-fs'
import E2BFileSystem from '@deepseek-ai/dsh-fs-e2b'
import * as E2BFsInvariant from '../src/invariant.ts'
import InvariantService from '@deepseek-ai/dsh-invariants'
@@ -46,6 +47,10 @@ class FakeRemote {
nextReadError: unknown
nextRenameError: unknown
nextRemoveError: unknown
boundedOutput: string | undefined
boundedError: unknown
nodeExecutable = '/usr/bin/node\n'
abortAfterBoundedCommand: AbortController | undefined
abortAfterRename: AbortController | undefined
disappearOnInfo = new Set<string>()
private clock = 1
@@ -221,6 +226,14 @@ class FakeRemote {
const node = this.nodes.get(input)
return { exitCode: 0, stdout: `${node?.symlinkTarget ?? input}\n`, stderr: '' }
}
if (command === 'command -v -- node') {
return { exitCode: 0, stdout: this.nodeExecutable, stderr: '' }
}
if (command.includes('dsh-e2b-bounded-reader')) {
if (this.boundedError !== undefined) throw this.boundedError
this.abortAfterBoundedCommand?.abort('after bounded read')
return { exitCode: 0, stdout: this.boundedOutput ?? '{"kind":"ok","data":""}', stderr: '' }
}
const chmod = /^chmod ([0-7]+) -- '([^']+)'$/.exec(command)
if (chmod !== null) this.required(chmod[2]!).mode = Number.parseInt(chmod[1]!, 8)
const move = /^mv -f -- '([^']+)' '([^']+)'$/.exec(command)
@@ -289,6 +302,26 @@ describe('E2BFileSystem identity, metadata, and reads', () => {
expect(listed.some(entry => entry.name === 'nested.txt')).toBe(false)
})
it('projects canonical process paths, file URLs, and containment', async () => {
const remote = new FakeRemote()
remote.dir('/workspace/nested')
remote.file('/workspace/nested/multibyte # file.ts', 'text')
remote.file('/outside.ts', 'outside')
const { fs } = await setup(remote)
const workspace = await fs.resolve('/workspace')
const nested = await fs.resolve('/workspace/nested/multibyte # file.ts')
const outside = await fs.resolve('/outside.ts')
expect(fs.processPath(nested)).toBe('/workspace/nested/multibyte # file.ts')
expect(fs.fileUrl(nested)).toBe('file:///workspace/nested/multibyte%20%23%20file.ts')
expect(fs.contains(workspace, workspace)).toBe(true)
expect(fs.contains(workspace, nested)).toBe(true)
expect(fs.contains(nested, workspace)).toBe(false)
expect(fs.contains(workspace, outside)).toBe(false)
expect(() => fs.fileUrl({ targetKey: FsTargetKey('relative'), displayPath: 'relative' }))
.toThrow('expected an absolute process path')
})
it('reads whole and streamed UTF-8 across chunk boundaries', async () => {
const remote = new FakeRemote()
remote.file('/workspace/text.txt', 'A€B')
@@ -373,6 +406,78 @@ describe('E2BFileSystem identity, metadata, and reads', () => {
await expectCode(fs.readText(await fs.resolve('a')), 'FS_ABORTED')
})
it('performs stable bounded reads through the remote no-follow reader', async () => {
const remote = new FakeRemote()
remote.file('/workspace/a', 'unused')
const { fs } = await setup(remote)
const target = await fs.resolve('a')
remote.boundedOutput = JSON.stringify({ kind: 'ok', data: Buffer.from('hello 你好').toString('base64') })
await expect(fs.readTextBounded(target, 64)).resolves.toBe('hello 你好')
expect(remote.commands.some(command => command.includes('dsh-e2b-bounded-reader'))).toBe(true)
await expect(fs.readTextBounded(target, 0)).rejects.toThrow('positive safe integer')
await expect(fs.readTextBounded(target, 1.5)).rejects.toThrow('positive safe integer')
await expect(fs.readTextBounded(target, 64, AbortSignal.abort())).rejects.toMatchObject({ code: 'FS_ABORTED' })
})
it('maps bounded-reader file, size, and open failures', async () => {
const remote = new FakeRemote()
remote.file('/workspace/a', 'unused')
const { fs } = await setup(remote)
const target = await fs.resolve('a')
const cases: Array<[unknown, string]> = [
[{ kind: 'not-file' }, 'FS_NOT_REGULAR_FILE'],
[{ kind: 'oversize', size: 65 }, 'FS_IO_ERROR'],
[{ kind: 'grew' }, 'FS_IO_ERROR'],
[{ kind: 'open-error', message: 'ENOENT: no such file' }, 'FS_NOT_FOUND'],
[{ kind: 'open-error', message: 'EACCES: permission denied' }, 'FS_PERMISSION_DENIED'],
[{ kind: 'open-error', message: 'ELOOP: symbolic link' }, 'FS_IO_ERROR'],
[{ kind: 'oversize', size: 'large' }, 'FS_IO_ERROR'],
[{ kind: 'open-error', message: 7 }, 'FS_IO_ERROR'],
[{ kind: 'unknown' }, 'FS_IO_ERROR'],
]
for (const [response, code] of cases) {
remote.boundedOutput = JSON.stringify(response)
await expectCode(fs.readTextBounded(target, 64), code)
}
})
it('rejects malformed bounded-reader transports and bytes', async () => {
const remote = new FakeRemote()
remote.file('/workspace/a', 'unused')
const { fs } = await setup(remote)
const target = await fs.resolve('a')
remote.boundedOutput = 'not-json'
await expectCode(fs.readTextBounded(target, 4), 'FS_IO_ERROR')
remote.boundedOutput = JSON.stringify({ kind: 'ok', data: '!!!!' })
await expectCode(fs.readTextBounded(target, 4), 'FS_IO_ERROR')
remote.boundedOutput = JSON.stringify({ kind: 'ok', data: Buffer.from('12345').toString('base64') })
await expectCode(fs.readTextBounded(target, 4), 'FS_IO_ERROR')
remote.boundedOutput = JSON.stringify({ kind: 'ok', data: Buffer.from([0]).toString('base64') })
await expectCode(fs.readTextBounded(target, 4), 'FS_NOT_TEXT')
remote.boundedOutput = JSON.stringify({ kind: 'ok', data: Buffer.from([0xff]).toString('base64') })
await expectCode(fs.readTextBounded(target, 4), 'FS_NOT_TEXT')
remote.nodeExecutable = 'node\n'
await expectCode(fs.readTextBounded(target, 4), 'FS_IO_ERROR')
remote.nodeExecutable = '/usr/bin/node\n/other\n'
await expectCode(fs.readTextBounded(target, 4), 'FS_IO_ERROR')
remote.nodeExecutable = '/usr/bin/node\n'
remote.boundedError = new Error('reader transport failed')
await expectCode(fs.readTextBounded(target, 4), 'FS_IO_ERROR')
})
it('does not turn a post-read abort into successful source text', async () => {
const remote = new FakeRemote()
remote.file('/workspace/a', 'unused')
const controller = new AbortController()
remote.abortAfterBoundedCommand = controller
remote.boundedOutput = JSON.stringify({ kind: 'ok', data: Buffer.from('text').toString('base64') })
const { fs } = await setup(remote)
await expectCode(fs.readTextBounded(await fs.resolve('a'), 4, controller.signal), 'FS_ABORTED')
})
it('rejects empty paths and directory-listing type errors', async () => {
const remote = new FakeRemote()
remote.file('/workspace/file', 'x')