fix(e2b): harden SDK shell and cleanup boundaries
E2B starts command and PTY requests through login shells, so isolate each control shell behind a fresh randomized HOME and blank sandbox credential names before mutable profiles can run. Preserve the real remote HOME only for the requested argv. Collapse duplicate termination state, keep failed force cleanup retryable until quiescence is observed, and make terminal state allocation cancellable. Leave numeric PGID reuse as an explicit provider-level TODO because a userspace precheck would remain TOCTOU.
This commit is contained in:
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/e2b/subprocess-e2b/README.md
|
||||
README.md: aaac559e60162c3051cc6f06cbcfac011f5eb59a
|
||||
README.zh.md: 904e4fd1d9a13082ddfc8183c0e42abc25bd1caf
|
||||
README.md: 9b86af428533ebcc2c0da56339e6ea2a28170fd3
|
||||
README.zh.md: e3e8368534266e35ec04dcf5a2d6718829b4a016
|
||||
|
||||
@@ -8,8 +8,8 @@ E2B implementation of the [`@deepseek-ai/dsh-subprocess`](../../subprocess/subpr
|
||||
|
||||
- **Asynchronous remote start** — the synchronous seam returns a handle immediately while `Sandbox.commands.run(..., { background: true })` starts remotely. `pid` is `-1` until the wrapper publishes and the adapter validates its process-group id; stdin and ordinary observation wait for that publication. An owned startup signal aborts environment and private-state preparation before allocation; once allocation begins, cancellation waits for a provisional SDK handle it can clean.
|
||||
- **Execution-world coordinates** — `cwd` and private `runtimeRoot` come from the shared owner; executable lookup verifies absolute paths or resolves a bare name against the sandbox PATH plus explicit overrides.
|
||||
- **Linux process groups** — a quoted wrapper starts each argv under `exec setsid --wait` and records its actual process-group id plus private status files beneath `ctx.e2b.runtimeRoot/processes`. The handle waits for that file instead of treating the SDK command PID as its published identity. Termination signals the negative recorded id with `SIGTERM`, waits the caller's `graceMs`, then escalates to `SIGKILL` and the SDK kill fallback; TERM delivery or probe failures also force that escalation. Process-table probes treat groups containing only zombie or dead entries as quiescent. A failed transaction is observable through `waitForExit()` and may be retried, while any proven quiescence permanently fences later termination against PID reuse. Before publication, cancellation uses both the provisional group and SDK handle and proves the group empty before reporting quiescence; if publication fails, rollback applies the same proof before startup rejects. After publication, a monitoring failure also rolls back the group before rejecting. Service disposal rejects new starts, terminates and joins every retained process group, then awaits SDK settlement and private cleanup before the sandbox owner disposes.
|
||||
- **Environment boundary** — the sandbox command environment crosses the SDK callback boundary as base64 ASCII before one strict UTF-8 decode, then the wrapper removes ambient `DSH_*` and credential-shaped (`*KEY*`, `*SECRET*`, `*TOKEN*`) names and restores every valid `spec.env` entry as an explicit caller opt-in; empty names, `=`, and NUL framing violations reject before launch. Host ambient variables never enter the sandbox implicitly. Private environment files are removed after consumption, and failed command or terminal setup removes its private state before rejecting.
|
||||
- **Linux process groups** — a quoted wrapper starts each argv under `exec setsid --wait` and records its actual process-group id plus private status files beneath `ctx.e2b.runtimeRoot/processes`. The handle waits for that file instead of treating the SDK command PID as its published identity. Termination signals the negative recorded id with `SIGTERM`, waits the caller's `graceMs`, then escalates to `SIGKILL` and the SDK kill fallback; TERM delivery or probe failures also force that escalation. Process-table probes treat groups containing only zombie or dead entries as quiescent. Force cleanup succeeds only after a bounded probe finds the group empty; otherwise `waitForExit()` exposes a retryable failure, while proven quiescence makes later termination a no-op. Publication and monitoring failures apply the same cleanup transaction before rejecting. Service disposal rejects new starts, terminates and joins every retained process group, then awaits SDK settlement and private cleanup before the sandbox owner disposes.
|
||||
- **Environment boundary** — the sandbox command environment crosses the SDK callback boundary as base64 ASCII before one strict UTF-8 decode, then the wrapper removes ambient `DSH_*` and credential-shaped (`*KEY*`, `*SECRET*`, `*TOKEN*`) names and restores every valid `spec.env` entry as an explicit caller opt-in; empty names, `=`, and NUL framing violations reject before launch. E2B's fixed command and PTY login shells receive a fresh randomized root-level `HOME` plus empty overrides for every scrubbed ambient name before user profiles can run; the requested argv receives the serialized environment afterward. Host ambient variables never enter the sandbox implicitly. Private environment files are removed after consumption, and failed command or terminal setup removes its private state before rejecting.
|
||||
- **Stdio projection** — the remote wrapper branches raw bytes into optional bounded spill files, frames each live chunk as newline-delimited base64 ASCII, and the host incrementally restores bytes across arbitrary SDK callback boundaries. Pipe mode writes those bytes to host Node streams; inherit mode writes them to the harness process streams; collect mode retains a bounded host tail with offset reads. The wrapper publishes the direct command status before waiting for inherited writers. For collect or inherit output, the adapter disconnects an incomplete SDK stream after `graceMs`, withholds its partial spill, and returns that status while retaining the remote group for `waitForExit()` and termination. Natural raw-pipe completion instead awaits lossless transport and preserves backpressure; explicit termination destroys the host pipes and releases blocked output before remote cleanup. Batch and streaming stdin use the SDK handle.
|
||||
- **Terminal sessions** — `spawnTerminal()` uses E2B's byte PTY API, installs the exact argv and scrubbed environment through private mode-`0600` files, reports the foreground process group, sends real signals, and tears down every live group in the remote terminal session before settlement; zombie-only groups are already quiescent. A private random output boundary discards the E2B bootstrap shell's prompt and echoed runner command while preserving every requested-process byte, including its first prompt. PTY allocation is awaited through handle publication before cancellation is observed, so owned rollback can clean the published handle. Setup and teardown own the private state transaction, abort pending setup during service disposal, fence publication, and retain an unproven setup cleanup for disposal retry. Prompt detection, scrollback, readiness, and owner policy remain in `dsh-pty-local`.
|
||||
- **Sandbox disappearance** — `SandboxNotFoundError` during process or terminal liveness, termination, rollback, or disconnect proves the remote execution world cannot retain work, so cleanup treats it as quiescent; unrelated failures remain observable.
|
||||
|
||||
@@ -8,8 +8,8 @@
|
||||
|
||||
- **异步远程启动**:同步 seam 会立即返回一个句柄,同时由 `Sandbox.commands.run(..., { background: true })` 在远程启动进程。包装层发布进程组 ID 并由适配器完成验证之前,`pid` 为 `-1`;stdin 和常规观察会等待该发布。自有启动信号会在分配前中止环境和私有状态准备;分配开始后,取消会等待可清理的临时 SDK 句柄。
|
||||
- **执行世界坐标**:`cwd` 和私有 `runtimeRoot` 来自共享所有者;可执行文件查找会验证绝对路径,或根据沙箱 PATH 加显式覆盖来解析裸名称。
|
||||
- **Linux 进程组**:带引号保护的包装层会在 `exec setsid --wait` 下启动每组 argv,并在 `ctx.e2b.runtimeRoot/processes` 下记录实际进程组 ID 和私有状态文件。句柄会等待该文件,而不会把 SDK 命令 PID 当作已发布的身份。终止操作以记录的负数 ID 发送 `SIGTERM`,等待调用方的 `graceMs`,再升级到 `SIGKILL` 和 SDK kill 回退;TERM 信号发送或探测失败也会强制触发该升级。进程表探测会把仅含僵尸或已死亡条目的进程组视为完全停稳。失败的事务可通过 `waitForExit()` 观察,并可重试;任何已证明的完全停稳都会永久防止后续终止操作命中复用的 PID。发布前,取消操作会同时使用临时进程组与 SDK 句柄,并在报告完全停稳前证明进程组为空;如果发布失败,回滚会在启动拒绝前执行相同证明。发布后,监控失败也会在拒绝前回滚进程组。服务 dispose(资源释放)会拒绝新的启动请求、终止并等待每个保留进程组退出,再等待 SDK 结算和私有清理完成,之后沙箱所有者才会释放。
|
||||
- **环境边界**:沙箱命令环境会先以 base64 ASCII 跨越 SDK 回调边界,再进行一次严格 UTF-8 解码;随后包装层移除环境中的 `DSH_*` 和形似凭据的名称(`*KEY*`、`*SECRET*`、`*TOKEN*`),并把每个有效的 `spec.env` 条目恢复为调用方显式选择。空名称、`=` 和违反 NUL 分帧规则的条目会在启动前被拒绝。宿主环境变量绝不会隐式进入沙箱。私有环境文件在使用后会被删除;命令或终端设置失败时,会先删除其私有状态再拒绝。
|
||||
- **Linux 进程组**:带引号保护的包装层会在 `exec setsid --wait` 下启动每组 argv,并在 `ctx.e2b.runtimeRoot/processes` 下记录实际进程组 ID 和私有状态文件。句柄会等待该文件,而不会把 SDK 命令 PID 当作已发布的身份。终止操作以记录的负数 ID 发送 `SIGTERM`,等待调用方的 `graceMs`,再升级到 `SIGKILL` 和 SDK kill 回退;TERM 信号发送或探测失败也会强制触发该升级。进程表探测会把仅含僵尸或已死亡条目的进程组视为完全停稳。强制清理只有在有界探测发现进程组为空后才算成功;否则 `waitForExit()` 会公开可重试的失败,而已证明的完全停稳会让后续终止操作不再执行任何动作。发布失败与监控失败都会在拒绝前执行同一清理事务。服务 dispose(资源释放)会拒绝新的启动请求、终止并等待每个保留进程组退出,再等待 SDK 结算和私有清理完成,之后沙箱所有者才会释放。
|
||||
- **环境边界**:沙箱命令环境会先以 base64 ASCII 跨越 SDK 回调边界,再进行一次严格 UTF-8 解码;随后包装层移除环境中的 `DSH_*` 和形似凭据的名称(`*KEY*`、`*SECRET*`、`*TOKEN*`),并把每个有效的 `spec.env` 条目恢复为调用方显式选择。空名称、`=` 和违反 NUL 分帧规则的条目会在启动前被拒绝。在用户 profile 脚本运行前,E2B 固定的命令 shell 与 PTY 登录 shell 会获得位于根目录下、全新随机生成的 `HOME`,并为每个被清理的环境变量名设置空值覆盖;之后,请求的 argv 才会接收序列化环境。宿主环境变量绝不会隐式进入沙箱。私有环境文件在使用后会被删除;命令或终端设置失败时,会先删除其私有状态再拒绝。
|
||||
- **stdio 投影**:远程包装层先把原始字节分流到可选的有界 spill 文件,再把每个实时分片编码为换行分隔的 base64 ASCII 帧;宿主会跨任意 SDK 回调边界增量恢复字节。pipe 模式把这些字节写入宿主 Node 流;inherit 模式把字节写入 harness 进程流;collect 模式保留有界的宿主尾部,并支持基于偏移量读取。包装层会在等待继承管道的写入方之前发布直接命令状态。对于 collect 或 inherit 输出,超过 `graceMs` 后,适配器会断开未完成的 SDK 流,不公开其中不完整的 spill,并返回该状态,同时保留远程进程组供 `waitForExit()` 和终止操作使用。原始 pipe 自然完成时,会等待无损传输完成并保留背压;显式终止则会销毁宿主 pipe,并在远程清理前释放受阻的输出写入。批量 stdin 和流式 stdin 都使用 SDK 句柄。
|
||||
- **终端会话**:`spawnTerminal()` 使用 E2B 的字节 PTY API,以 mode 为 `0600` 的私有文件传入原样 argv 与清理后的环境,报告前台进程组,发送真实信号,并在结算前清理远程终端会话中仍存活的每个进程组;仅含僵尸进程的进程组已经完全停稳。私有随机输出边界会丢弃 E2B 引导 shell 的提示符和回显的 runner 命令,同时保留请求进程的每个字节,包括其第一个提示符。PTY 分配会一直等待到句柄发布后才观察取消,以便由承担清理责任的回滚清理已发布句柄。setup 与 teardown 负责私有状态事务,在服务 dispose 期间中止待处理的 setup、阻止发布,并保留未证明已完成的 setup 清理事务,供 dispose 重试。提示符检测、scrollback、就绪状态与所有者策略仍归 `dsh-pty-local` 所有。
|
||||
- **沙箱消失**:在进程或终端的存活探测、终止、回滚或断开连接期间出现 `SandboxNotFoundError`,证明远程执行环境无法保留工作,因此清理会将其视为完全停稳;其他故障仍可观察。
|
||||
|
||||
@@ -1,11 +1,24 @@
|
||||
/** Shared remote-environment scrubbing for E2B process and terminal launchers. */
|
||||
|
||||
import { Buffer } from 'node:buffer'
|
||||
import { posix } from 'node:path'
|
||||
import { e2bControlEnvs } from '@deepseek-ai/dsh-e2b'
|
||||
import type { Sandbox } from '@deepseek-ai/dsh-e2b'
|
||||
import { SENSITIVE_ENV_PATTERN } from '@deepseek-ai/dsh-subprocess'
|
||||
|
||||
const BASE64 = /^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/
|
||||
|
||||
function remoteEnvironmentEntries(raw: string): Array<readonly [string, string]> {
|
||||
const entries: Array<readonly [string, string]> = []
|
||||
for (const entry of raw.split('\0')) {
|
||||
if (entry.length === 0) continue
|
||||
const separator = entry.indexOf('=')
|
||||
if (separator <= 0) continue
|
||||
entries.push([entry.slice(0, separator), entry.slice(separator + 1)])
|
||||
}
|
||||
return entries
|
||||
}
|
||||
|
||||
/**
|
||||
* Read the remote environment through ASCII base64 so SDK callback chunking cannot corrupt UTF-8.
|
||||
* @param sandbox - shared E2B execution world.
|
||||
@@ -14,16 +27,29 @@ const BASE64 = /^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$
|
||||
*/
|
||||
export async function readRemoteEnvironment(sandbox: Sandbox, signal?: AbortSignal): Promise<string> {
|
||||
const result = await sandbox.commands.run(
|
||||
'set -o pipefail; env -0 | base64 -w 0',
|
||||
signal === undefined ? {} : { signal },
|
||||
'set -o pipefail; printf \'%s\' "$PWD" | base64 -w 0; printf \'\\n\'; env -0 | base64 -w 0',
|
||||
{ envs: e2bControlEnvs(), ...(signal === undefined ? {} : { signal }) },
|
||||
)
|
||||
const encoded = result.stdout.trim()
|
||||
if (!BASE64.test(encoded)) throw new Error('subprocess-e2b: remote environment transport returned invalid base64')
|
||||
const lines = result.stdout.trim().split('\n')
|
||||
if (lines.length !== 2 || !lines.every(line => BASE64.test(line))) {
|
||||
throw new Error('subprocess-e2b: remote environment transport returned invalid base64')
|
||||
}
|
||||
const [encodedHome, encodedEnvironment] = lines as [string, string]
|
||||
let home: string
|
||||
let raw: string
|
||||
try {
|
||||
return new TextDecoder('utf-8', { fatal: true }).decode(Buffer.from(encoded, 'base64'))
|
||||
const decoder = new TextDecoder('utf-8', { fatal: true })
|
||||
home = decoder.decode(Buffer.from(encodedHome, 'base64'))
|
||||
raw = decoder.decode(Buffer.from(encodedEnvironment, 'base64'))
|
||||
} catch (error: unknown) {
|
||||
throw new Error('subprocess-e2b: remote environment is not valid UTF-8', { cause: error })
|
||||
}
|
||||
if (!posix.isAbsolute(home) || home.includes('\0')) {
|
||||
throw new Error(`subprocess-e2b: remote login home is invalid: ${JSON.stringify(home)}`)
|
||||
}
|
||||
const environment = new Map(remoteEnvironmentEntries(raw))
|
||||
environment.set('HOME', home)
|
||||
return [...environment].map(([name, value]) => `${name}=${value}\0`).join('')
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -33,13 +59,22 @@ export async function readRemoteEnvironment(sandbox: Sandbox, signal?: AbortSign
|
||||
*/
|
||||
export function scrubRemoteEnvironment(raw: string): Map<string, string> {
|
||||
const environment = new Map<string, string>()
|
||||
for (const entry of raw.split('\0')) {
|
||||
if (entry.length === 0) continue
|
||||
const separator = entry.indexOf('=')
|
||||
if (separator <= 0) continue
|
||||
const name = entry.slice(0, separator)
|
||||
for (const [name, value] of remoteEnvironmentEntries(raw)) {
|
||||
if (name.startsWith('DSH_') || SENSITIVE_ENV_PATTERN.test(name)) continue
|
||||
environment.set(name, entry.slice(separator + 1))
|
||||
environment.set(name, value)
|
||||
}
|
||||
return environment
|
||||
}
|
||||
|
||||
/**
|
||||
* Isolate E2B's fixed login-shell bootstrap from user profiles and ambient credentials.
|
||||
* @param raw - The complete NUL-delimited remote environment.
|
||||
* @returns Explicit E2B command or PTY overrides for bootstrap-shell startup.
|
||||
*/
|
||||
export function bootstrapEnvironment(raw: string): Record<string, string> {
|
||||
const environment: Record<string, string> = { TERM: 'dumb' }
|
||||
for (const [name] of remoteEnvironmentEntries(raw)) {
|
||||
if (name.startsWith('DSH_') || SENSITIVE_ENV_PATTERN.test(name)) environment[name] = ''
|
||||
}
|
||||
return environment
|
||||
}
|
||||
|
||||
@@ -14,7 +14,7 @@ import type {
|
||||
SubprocessTerminalHandle,
|
||||
SubprocessTerminalSpawnSpec,
|
||||
} from '@deepseek-ai/dsh-subprocess'
|
||||
import { quoteE2BShellArg } from '@deepseek-ai/dsh-e2b'
|
||||
import { e2bControlEnvs, quoteE2BShellArg } from '@deepseek-ai/dsh-e2b'
|
||||
import { E2BSubprocessHandle } from './process.ts'
|
||||
import { spawnE2BTerminal } from './terminal.ts'
|
||||
|
||||
@@ -86,7 +86,7 @@ export class E2BSubprocessService extends SubprocessService {
|
||||
if (posix.isAbsolute(command)) {
|
||||
await sandbox.commands.run(
|
||||
`test -f ${quoteE2BShellArg(command)} -a -x ${quoteE2BShellArg(command)}`,
|
||||
signalOpts(signal),
|
||||
{ envs: e2bControlEnvs(), ...signalOpts(signal) },
|
||||
)
|
||||
signal?.throwIfAborted()
|
||||
return command
|
||||
@@ -95,7 +95,7 @@ export class E2BSubprocessService extends SubprocessService {
|
||||
const prefix = path === undefined ? '' : `PATH=${quoteE2BShellArg(path)} `
|
||||
const result = await sandbox.commands.run(
|
||||
`${prefix}command -v -- ${quoteE2BShellArg(command)}`,
|
||||
{ cwd: this.cwd, ...signalOpts(signal) },
|
||||
{ cwd: this.cwd, envs: e2bControlEnvs(), ...signalOpts(signal) },
|
||||
)
|
||||
signal?.throwIfAborted()
|
||||
const executable = result.stdout.trim()
|
||||
|
||||
@@ -5,6 +5,7 @@ import { PassThrough, Writable } from 'node:stream'
|
||||
import { posix } from 'node:path'
|
||||
import {
|
||||
CommandExitError,
|
||||
e2bControlEnvs,
|
||||
FileNotFoundError,
|
||||
SandboxNotFoundError,
|
||||
quoteE2BShellArg,
|
||||
@@ -18,7 +19,7 @@ import type {
|
||||
SubprocessSpawnSpec,
|
||||
} from '@deepseek-ai/dsh-subprocess'
|
||||
import type E2BSandboxService from '@deepseek-ai/dsh-e2b'
|
||||
import { readRemoteEnvironment, serializeRemoteEnvironment } from './environment.ts'
|
||||
import { bootstrapEnvironment, readRemoteEnvironment, serializeRemoteEnvironment } from './environment.ts'
|
||||
import { E2BBase64Decoder, E2B_OUTPUT_COMPLETE_FRAME, E2BOutputReader } from './output.ts'
|
||||
|
||||
const GROUP_POLL_MS = 20
|
||||
@@ -142,8 +143,11 @@ function commandText(spec: SubprocessSpawnSpec, paths: RemotePaths): string {
|
||||
return bootstrap
|
||||
}
|
||||
|
||||
function signalOpts(signal: AbortSignal | undefined): { signal?: AbortSignal } {
|
||||
return signal === undefined ? {} : { signal }
|
||||
function commandOpts(
|
||||
envs: Record<string, string>,
|
||||
signal: AbortSignal | undefined,
|
||||
): { envs: Record<string, string>; signal?: AbortSignal } {
|
||||
return { envs: e2bControlEnvs(envs), ...(signal === undefined ? {} : { signal }) }
|
||||
}
|
||||
|
||||
function isAborted(signal: AbortSignal | undefined): boolean {
|
||||
@@ -197,21 +201,18 @@ export class E2BSubprocessHandle implements SubprocessHandle {
|
||||
private readonly readyState = Promise.withResolvers<CommandHandle>()
|
||||
private readonly stdoutDecoder = new E2BBase64Decoder()
|
||||
private readonly stderrDecoder = new E2BBase64Decoder()
|
||||
private readonly outputTermination = new AbortController()
|
||||
private readonly startupController = new AbortController()
|
||||
private readonly terminationController = new AbortController()
|
||||
private readonly stdoutReader: E2BOutputReader | undefined
|
||||
private readonly stderrReader: E2BOutputReader | undefined
|
||||
private readonly paths: RemotePaths
|
||||
private controlEnvs: Record<string, string> = {}
|
||||
private remotePid = -1
|
||||
private commandHandle: CommandHandle | undefined
|
||||
private outputTransportError: Error | undefined
|
||||
private outputDrainExpired = false
|
||||
private stateDirectoryCreated = false
|
||||
private preparing = true
|
||||
private invalidHandleQuiescent = false
|
||||
private provisionalHandleQuiescent = false
|
||||
private terminationStarted = false
|
||||
private terminationFenced = false
|
||||
private quiescenceProven = false
|
||||
private terminationAttempt: Promise<void> | undefined
|
||||
private terminationFailure: Error | undefined
|
||||
@@ -264,20 +265,16 @@ export class E2BSubprocessHandle implements SubprocessHandle {
|
||||
|
||||
/** @inheritdoc */
|
||||
terminate(): void {
|
||||
if (this.terminationFenced || this.quiescenceProven || this.terminationAttempt !== undefined) return
|
||||
if (this.quiescenceProven || this.terminationAttempt !== undefined) return
|
||||
this.terminationStarted = true
|
||||
if (this.preparing) this.startupController.abort(new Error('subprocess-e2b: command terminated during startup'))
|
||||
this.outputTermination.abort()
|
||||
this.terminationController.abort(new Error('subprocess-e2b: command terminated'))
|
||||
this.stdout?.destroy()
|
||||
this.stderr?.destroy()
|
||||
this.terminationFailure = undefined
|
||||
const attempt = this.terminateRemote()
|
||||
this.terminationAttempt = attempt
|
||||
void attempt.then(
|
||||
() => {
|
||||
this.terminationFenced = true
|
||||
this.terminationAttempt = undefined
|
||||
},
|
||||
() => { this.terminationAttempt = undefined },
|
||||
(error: unknown) => {
|
||||
if (!this.quiescenceProven) this.terminationFailure = asError(error)
|
||||
this.terminationAttempt = undefined
|
||||
@@ -301,11 +298,8 @@ export class E2BSubprocessHandle implements SubprocessHandle {
|
||||
const attempt = this.terminationAttempt
|
||||
if (attempt !== undefined && await waitWithSignal(attempt, signal) === WAIT_ABORTED) return false
|
||||
this.throwTerminationFailure()
|
||||
/* v8 ignore else -- successful provisional cleanup always records one proof; failures throw above. */
|
||||
if (this.invalidHandleQuiescent || this.provisionalHandleQuiescent) {
|
||||
this.markQuiescent()
|
||||
return true
|
||||
}
|
||||
// Successful pre-publication termination records quiescence; its only other outcome is the failure above.
|
||||
return true
|
||||
}
|
||||
} else {
|
||||
try {
|
||||
@@ -361,6 +355,7 @@ export class E2BSubprocessHandle implements SubprocessHandle {
|
||||
{
|
||||
background: true,
|
||||
cwd: this.spec.cwd,
|
||||
envs: e2bControlEnvs(this.controlEnvs),
|
||||
stdin: this.spec.stdio.stdin !== 'ignore',
|
||||
timeoutMs: 0,
|
||||
onStdout: async (data) => { await this.dispatchOutput('stdout', data) },
|
||||
@@ -374,7 +369,7 @@ export class E2BSubprocessHandle implements SubprocessHandle {
|
||||
const invalidPid = new Error(`subprocess-e2b: E2B returned invalid command pid ${handle.pid}`)
|
||||
try {
|
||||
await handle.kill()
|
||||
this.invalidHandleQuiescent = true
|
||||
this.markQuiescent()
|
||||
this.commandHandle = undefined
|
||||
} catch (cleanupError: unknown) {
|
||||
this.terminationFailure = asError(cleanupError)
|
||||
@@ -412,7 +407,7 @@ export class E2BSubprocessHandle implements SubprocessHandle {
|
||||
} catch (error: unknown) {
|
||||
const canceledPreparation = this.preparing
|
||||
&& this.terminationStarted
|
||||
&& this.startupController.signal.aborted
|
||||
&& this.terminationController.signal.aborted
|
||||
let failure = await this.rollbackPublishedFailure(error)
|
||||
if (sandbox !== undefined && this.stateDirectoryCreated) {
|
||||
try {
|
||||
@@ -437,11 +432,15 @@ export class E2BSubprocessHandle implements SubprocessHandle {
|
||||
}
|
||||
|
||||
private async prepareState(sandbox: Sandbox): Promise<void> {
|
||||
const signal = this.startupController.signal
|
||||
const signal = this.terminationController.signal
|
||||
const ambient = await readRemoteEnvironment(sandbox, signal)
|
||||
this.controlEnvs = bootstrapEnvironment(ambient)
|
||||
await sandbox.files.makeDir(this.stateDir, { signal })
|
||||
this.stateDirectoryCreated = true
|
||||
await sandbox.commands.run(`chmod 700 -- ${quoteE2BShellArg(this.stateDir)}`, { signal })
|
||||
await sandbox.commands.run(
|
||||
`chmod 700 -- ${quoteE2BShellArg(this.stateDir)}`,
|
||||
commandOpts(this.controlEnvs, signal),
|
||||
)
|
||||
const files = [
|
||||
{ path: this.paths.pid, data: '' },
|
||||
{ path: this.paths.status, data: '' },
|
||||
@@ -450,7 +449,10 @@ export class E2BSubprocessHandle implements SubprocessHandle {
|
||||
...(hasSpill(this.spec.stdio.stderr) ? [{ path: this.paths.stderr, data: '' }] : []),
|
||||
]
|
||||
await sandbox.files.write(files, { signal })
|
||||
await sandbox.commands.run(`chmod 600 -- ${files.map(file => quoteE2BShellArg(file.path)).join(' ')}`, { signal })
|
||||
await sandbox.commands.run(
|
||||
`chmod 600 -- ${files.map(file => quoteE2BShellArg(file.path)).join(' ')}`,
|
||||
commandOpts(this.controlEnvs, signal),
|
||||
)
|
||||
signal.throwIfAborted()
|
||||
}
|
||||
|
||||
@@ -490,7 +492,7 @@ export class E2BSubprocessHandle implements SubprocessHandle {
|
||||
|
||||
private async writeOutput(pipe: PassThrough | undefined, inherited: NodeJS.WriteStream | undefined, data: Uint8Array): Promise<void> {
|
||||
const target = pipe ?? inherited
|
||||
if (target === undefined || data.length === 0 || this.outputTermination.signal.aborted) return
|
||||
if (target === undefined || data.length === 0 || this.terminationController.signal.aborted) return
|
||||
if (target.destroyed) throw new Error('subprocess output stream is closed')
|
||||
if (target.write(data)) return
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
@@ -502,13 +504,13 @@ export class E2BSubprocessHandle implements SubprocessHandle {
|
||||
target.removeListener('drain', onDrain)
|
||||
target.removeListener('close', onClose)
|
||||
target.removeListener('error', onError)
|
||||
this.outputTermination.signal.removeEventListener('abort', onTermination)
|
||||
this.terminationController.signal.removeEventListener('abort', onTermination)
|
||||
}
|
||||
target.once('drain', onDrain)
|
||||
target.once('close', onClose)
|
||||
target.once('error', onError)
|
||||
this.outputTermination.signal.addEventListener('abort', onTermination, { once: true })
|
||||
if (this.outputTermination.signal.aborted) onTermination()
|
||||
this.terminationController.signal.addEventListener('abort', onTermination, { once: true })
|
||||
if (this.terminationController.signal.aborted) onTermination()
|
||||
})
|
||||
}
|
||||
|
||||
@@ -596,12 +598,8 @@ export class E2BSubprocessHandle implements SubprocessHandle {
|
||||
// `setsid`, so E2B's command PID is the provisional group id even before the
|
||||
// private publication file can be trusted. Kill that group before the SDK-PID
|
||||
// fallback, then prove no group member survived before rejecting startup.
|
||||
try {
|
||||
await this.signalGroup(sandbox, handle.pid, 'KILL')
|
||||
} finally {
|
||||
await handle.kill().catch(() => false)
|
||||
}
|
||||
while (await this.groupAlive(sandbox, handle.pid)) await waitTick()
|
||||
await this.forceKillGroup(sandbox, handle, handle.pid)
|
||||
this.markQuiescent()
|
||||
}
|
||||
|
||||
private async terminateRemote(): Promise<void> {
|
||||
@@ -618,91 +616,76 @@ export class E2BSubprocessHandle implements SubprocessHandle {
|
||||
|
||||
private async terminateRemoteInSandbox(): Promise<void> {
|
||||
const handle = await this.commandState.promise
|
||||
if (handle === undefined) return
|
||||
if (handle === undefined) {
|
||||
this.markQuiescent()
|
||||
return
|
||||
}
|
||||
if (!isValidProcessId(handle.pid) && this.remotePid <= 0) {
|
||||
await handle.kill()
|
||||
this.invalidHandleQuiescent = true
|
||||
this.markQuiescent()
|
||||
this.commandHandle = undefined
|
||||
return
|
||||
}
|
||||
if (this.remotePid <= 0) {
|
||||
const sandbox = await this.runtime.getSandbox()
|
||||
this.terminationSignal = 'SIGTERM'
|
||||
try {
|
||||
const delivered = await this.signalGroup(sandbox, handle.pid, 'TERM')
|
||||
if (delivered) {
|
||||
const deadline = Date.now() + this.spec.graceMs
|
||||
while (Date.now() < deadline && await this.groupAlive(sandbox, handle.pid)) await waitTick()
|
||||
if (!await this.groupAlive(sandbox, handle.pid)) {
|
||||
this.provisionalHandleQuiescent = true
|
||||
return
|
||||
}
|
||||
}
|
||||
} catch (_gracefulTerminationFailure) {
|
||||
// A missing or unobservable provisional group still has the SDK handle fallback.
|
||||
}
|
||||
this.terminationSignal = 'SIGKILL'
|
||||
let groupDelivered = false
|
||||
let groupFailure: unknown
|
||||
try {
|
||||
groupDelivered = await this.signalGroup(sandbox, handle.pid, 'KILL')
|
||||
} catch (error: unknown) {
|
||||
groupFailure = error
|
||||
}
|
||||
let handleFailure: unknown
|
||||
try {
|
||||
if (!await handle.kill()) handleFailure = new Error('E2B SDK kill did not report command termination')
|
||||
} catch (error: unknown) {
|
||||
handleFailure = error
|
||||
}
|
||||
if (!groupDelivered && await this.groupAlive(sandbox, handle.pid)) {
|
||||
throw new AggregateError(
|
||||
[
|
||||
...(groupFailure === undefined ? [] : [groupFailure]),
|
||||
...(handleFailure === undefined
|
||||
? [new Error('E2B SDK kill did not quiesce the provisional process group')]
|
||||
: [handleFailure]),
|
||||
],
|
||||
'subprocess-e2b: force termination failed through both process-group and SDK transports',
|
||||
)
|
||||
}
|
||||
while (await this.groupAlive(sandbox, handle.pid)) await waitTick()
|
||||
this.provisionalHandleQuiescent = true
|
||||
return
|
||||
}
|
||||
const sandbox = await this.runtime.getSandbox()
|
||||
const processGroupId = this.remotePid
|
||||
const processGroupId = this.remotePid > 0 ? this.remotePid : handle.pid
|
||||
await this.terminateGroup(sandbox, handle, processGroupId)
|
||||
}
|
||||
|
||||
private async terminateGroup(sandbox: Sandbox, handle: CommandHandle, processGroupId: number): Promise<void> {
|
||||
this.terminationSignal = 'SIGTERM'
|
||||
try {
|
||||
await this.signalGroup(sandbox, processGroupId, 'TERM')
|
||||
const deadline = Date.now() + this.spec.graceMs
|
||||
while (Date.now() < deadline && await this.groupAlive(sandbox, processGroupId)) {
|
||||
await waitTick()
|
||||
if (await this.waitForGroupExit(sandbox, processGroupId)) {
|
||||
this.markQuiescent()
|
||||
return
|
||||
}
|
||||
if (!await this.groupAlive(sandbox, processGroupId)) return
|
||||
} catch (_gracefulTerminationFailure) {
|
||||
// Failed TERM delivery or observation cannot prove exit; force cleanup still owns the group.
|
||||
}
|
||||
this.terminationSignal = 'SIGKILL'
|
||||
await this.forceKillGroup(sandbox, handle, processGroupId)
|
||||
this.markQuiescent()
|
||||
}
|
||||
|
||||
private async forceKillGroup(sandbox: Sandbox, handle: CommandHandle, processGroupId: number): Promise<void> {
|
||||
let groupFailure: unknown
|
||||
let groupDelivered = false
|
||||
try {
|
||||
groupDelivered = await this.signalGroup(sandbox, processGroupId, 'KILL')
|
||||
if (!await this.signalGroup(sandbox, processGroupId, 'KILL')) {
|
||||
groupFailure = new Error('process-group KILL did not report delivery')
|
||||
}
|
||||
} catch (error: unknown) {
|
||||
groupFailure = error
|
||||
}
|
||||
let handleFailure: unknown
|
||||
try {
|
||||
await handle.kill()
|
||||
if (!await handle.kill()) handleFailure = new Error('E2B SDK kill did not report command termination')
|
||||
} catch (error: unknown) {
|
||||
handleFailure = error
|
||||
}
|
||||
if (!groupDelivered && handleFailure !== undefined && await this.groupAlive(sandbox, processGroupId)) {
|
||||
throw new AggregateError(
|
||||
[...(groupFailure === undefined ? [] : [groupFailure]), handleFailure],
|
||||
'subprocess-e2b: force termination failed through both process-group and SDK transports',
|
||||
)
|
||||
let proofFailure: unknown
|
||||
try {
|
||||
if (await this.waitForGroupExit(sandbox, processGroupId)) return
|
||||
proofFailure = new Error(`remote process group ${processGroupId} remained live after force termination`)
|
||||
} catch (error: unknown) {
|
||||
proofFailure = error
|
||||
}
|
||||
throw new AggregateError(
|
||||
[
|
||||
...(groupFailure === undefined ? [] : [groupFailure]),
|
||||
...(handleFailure === undefined ? [] : [handleFailure]),
|
||||
proofFailure,
|
||||
],
|
||||
'subprocess-e2b: force termination failed through both process-group and SDK transports',
|
||||
)
|
||||
}
|
||||
|
||||
private async waitForGroupExit(sandbox: Sandbox, processGroupId: number): Promise<boolean> {
|
||||
const deadline = Date.now() + this.spec.graceMs
|
||||
while (await this.groupAlive(sandbox, processGroupId)) {
|
||||
if (Date.now() >= deadline) return false
|
||||
await waitTick()
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
private throwTerminationFailure(): void {
|
||||
@@ -710,8 +693,13 @@ export class E2BSubprocessHandle implements SubprocessHandle {
|
||||
}
|
||||
|
||||
private async signalGroup(sandbox: Sandbox, pid: number, signal: 'TERM' | 'KILL'): Promise<boolean> {
|
||||
// TODO(e2b-pgid-identity): Prefer an atomic identity-bound group signal if E2B adds one;
|
||||
// a userspace identity precheck cannot close the numeric-PGID reuse race.
|
||||
try {
|
||||
await sandbox.commands.run(`kill -${signal} -- -${pid}`)
|
||||
await sandbox.commands.run(
|
||||
`kill -${signal} -- -${pid}`,
|
||||
commandOpts(this.controlEnvs, undefined),
|
||||
)
|
||||
return true
|
||||
} catch (error: unknown) {
|
||||
if (error instanceof CommandExitError || error instanceof SandboxNotFoundError) return false
|
||||
@@ -722,7 +710,7 @@ export class E2BSubprocessHandle implements SubprocessHandle {
|
||||
private async groupAlive(sandbox: Sandbox, pid: number, signal?: AbortSignal): Promise<boolean> {
|
||||
const result = await sandbox.commands.run(
|
||||
`set -o pipefail; ps -eo pgid=,stat= | awk '$1 == ${pid} && $2 !~ /^[ZXx]/ { live=1 } END { if (live) print "live" }'`,
|
||||
signalOpts(signal),
|
||||
commandOpts(this.controlEnvs, signal),
|
||||
).catch((error: unknown) => {
|
||||
if (signal?.aborted === true) return undefined
|
||||
if (error instanceof SandboxNotFoundError) return { exitCode: 0, stdout: '', stderr: '' }
|
||||
|
||||
@@ -6,6 +6,7 @@ import { PassThrough } from 'node:stream'
|
||||
import { posix } from 'node:path'
|
||||
import {
|
||||
CommandExitError,
|
||||
e2bControlEnvs,
|
||||
FileNotFoundError,
|
||||
SandboxNotFoundError,
|
||||
quoteE2BShellArg,
|
||||
@@ -20,7 +21,11 @@ import type {
|
||||
SubprocessTerminalSpawnSpec,
|
||||
} from '@deepseek-ai/dsh-subprocess'
|
||||
import type E2BSandboxService from '@deepseek-ai/dsh-e2b'
|
||||
import { readRemoteEnvironment, serializeRemoteEnvironment } from './environment.ts'
|
||||
import {
|
||||
bootstrapEnvironment,
|
||||
readRemoteEnvironment,
|
||||
serializeRemoteEnvironment,
|
||||
} from './environment.ts'
|
||||
|
||||
const POLL_MS = 20
|
||||
|
||||
@@ -54,6 +59,13 @@ function signalOpts(signal: AbortSignal | undefined): { signal?: AbortSignal } {
|
||||
return signal === undefined ? {} : { signal }
|
||||
}
|
||||
|
||||
function commandOpts(
|
||||
envs: Record<string, string>,
|
||||
signal?: AbortSignal,
|
||||
): { envs: Record<string, string>; signal?: AbortSignal } {
|
||||
return { envs: e2bControlEnvs(envs), ...signalOpts(signal) }
|
||||
}
|
||||
|
||||
function delay(ms: number): Promise<void> {
|
||||
return new Promise(resolve => setTimeout(resolve, ms))
|
||||
}
|
||||
@@ -143,8 +155,13 @@ function serializeValues(values: readonly string[], kind: string): string {
|
||||
return values.map(value => `${value}\0`).join('')
|
||||
}
|
||||
|
||||
async function terminalSessionId(sandbox: Sandbox, pid: number, signal?: AbortSignal): Promise<number> {
|
||||
const result = await sandbox.commands.run(`ps -o sid= -p ${pid}`, signalOpts(signal))
|
||||
async function terminalSessionId(
|
||||
sandbox: Sandbox,
|
||||
pid: number,
|
||||
envs: Record<string, string>,
|
||||
signal?: AbortSignal,
|
||||
): Promise<number> {
|
||||
const result = await sandbox.commands.run(`ps -o sid= -p ${pid}`, commandOpts(envs, signal))
|
||||
signal?.throwIfAborted()
|
||||
return parsePositiveId(result.stdout, `subprocess-e2b: cannot resolve process session for terminal ${pid}`)
|
||||
}
|
||||
@@ -169,11 +186,16 @@ async function waitUntilReady(
|
||||
}
|
||||
}
|
||||
|
||||
async function sessionProcessGroups(sandbox: Sandbox, sessionId: number): Promise<number[]> {
|
||||
async function sessionProcessGroups(
|
||||
sandbox: Sandbox,
|
||||
sessionId: number,
|
||||
envs: Record<string, string>,
|
||||
): Promise<number[]> {
|
||||
let result: CommandResult
|
||||
try {
|
||||
result = await sandbox.commands.run(
|
||||
`set -o pipefail; ps -eo sid=,pgid=,stat= | awk '$1 == ${sessionId} && $3 !~ /^[ZXx]/ { print $2 }'`,
|
||||
commandOpts(envs),
|
||||
)
|
||||
} catch (error: unknown) {
|
||||
if (error instanceof SandboxNotFoundError) return []
|
||||
@@ -194,9 +216,17 @@ async function sessionProcessGroups(sandbox: Sandbox, sessionId: number): Promis
|
||||
return [...groups]
|
||||
}
|
||||
|
||||
async function signalGroups(sandbox: Sandbox, groups: number[], signal: 'TERM' | 'KILL'): Promise<void> {
|
||||
async function signalGroups(
|
||||
sandbox: Sandbox,
|
||||
groups: number[],
|
||||
signal: 'TERM' | 'KILL',
|
||||
envs: Record<string, string>,
|
||||
): Promise<void> {
|
||||
try {
|
||||
await sandbox.commands.run(`kill -${signal} -- ${groups.map(group => `-${group}`).join(' ')}`)
|
||||
await sandbox.commands.run(
|
||||
`kill -${signal} -- ${groups.map(group => `-${group}`).join(' ')}`,
|
||||
commandOpts(envs),
|
||||
)
|
||||
} catch (error: unknown) {
|
||||
if (!(error instanceof CommandExitError) && !(error instanceof SandboxNotFoundError)) throw error
|
||||
}
|
||||
@@ -205,14 +235,15 @@ async function signalGroups(sandbox: Sandbox, groups: number[], signal: 'TERM' |
|
||||
async function awaitSessionEmpty(
|
||||
sandbox: Sandbox,
|
||||
sessionId: number,
|
||||
envs: Record<string, string>,
|
||||
graceMs: number,
|
||||
kill = false,
|
||||
): Promise<number[]> {
|
||||
const deadline = Date.now() + graceMs
|
||||
for (;;) {
|
||||
const groups = await sessionProcessGroups(sandbox, sessionId)
|
||||
const groups = await sessionProcessGroups(sandbox, sessionId, envs)
|
||||
if (groups.length === 0 || Date.now() >= deadline) return groups
|
||||
if (kill) await signalGroups(sandbox, groups, 'KILL')
|
||||
if (kill) await signalGroups(sandbox, groups, 'KILL', envs)
|
||||
await delay(Math.min(POLL_MS, Math.max(1, deadline - Date.now())))
|
||||
}
|
||||
}
|
||||
@@ -221,6 +252,7 @@ async function rollbackUnpublishedTerminal(
|
||||
sandbox: Sandbox,
|
||||
handle: CommandHandle,
|
||||
completion: Promise<CommandResult>,
|
||||
envs: Record<string, string>,
|
||||
graceMs: number,
|
||||
): Promise<void> {
|
||||
let topLevelExited = false
|
||||
@@ -234,20 +266,20 @@ async function rollbackUnpublishedTerminal(
|
||||
if (validPid) {
|
||||
sessionId = handle.pid
|
||||
try {
|
||||
sessionId = await terminalSessionId(sandbox, handle.pid)
|
||||
sessionId = await terminalSessionId(sandbox, handle.pid, envs)
|
||||
} catch (_sessionLookupFailure) {
|
||||
// E2B's PTY leader is also the provisional POSIX session leader, so its
|
||||
// PID remains usable after the setup lookup itself fails or is canceled.
|
||||
}
|
||||
try {
|
||||
let groups = await sessionProcessGroups(sandbox, sessionId)
|
||||
let groups = await sessionProcessGroups(sandbox, sessionId, envs)
|
||||
if (groups.length > 0) {
|
||||
await signalGroups(sandbox, groups, 'TERM')
|
||||
groups = await awaitSessionEmpty(sandbox, sessionId, graceMs)
|
||||
await signalGroups(sandbox, groups, 'TERM', envs)
|
||||
groups = await awaitSessionEmpty(sandbox, sessionId, envs, graceMs)
|
||||
}
|
||||
if (groups.length > 0) {
|
||||
await signalGroups(sandbox, groups, 'KILL')
|
||||
await awaitSessionEmpty(sandbox, sessionId, graceMs, true)
|
||||
await signalGroups(sandbox, groups, 'KILL', envs)
|
||||
await awaitSessionEmpty(sandbox, sessionId, envs, graceMs, true)
|
||||
}
|
||||
} catch (error: unknown) {
|
||||
attemptFailures.push(asError(error))
|
||||
@@ -279,7 +311,7 @@ async function rollbackUnpublishedTerminal(
|
||||
const proofFailures: Error[] = []
|
||||
if (sessionId !== undefined) {
|
||||
try {
|
||||
const groups = await awaitSessionEmpty(sandbox, sessionId, graceMs, true)
|
||||
const groups = await awaitSessionEmpty(sandbox, sessionId, envs, graceMs, true)
|
||||
if (groups.length > 0) {
|
||||
proofFailures.push(new Error(
|
||||
`subprocess-e2b: terminal setup rollback failed; surviving process groups: ${groups.join(', ')}`,
|
||||
@@ -322,6 +354,7 @@ export class E2BTerminalHandle implements SubprocessTerminalHandle {
|
||||
readonly output: PassThrough,
|
||||
private readonly completion: Promise<CommandResult>,
|
||||
private readonly sessionId: number,
|
||||
private readonly controlEnvs: Record<string, string>,
|
||||
private readonly stateDir: string,
|
||||
private readonly graceMs: number,
|
||||
signal?: AbortSignal,
|
||||
@@ -344,7 +377,10 @@ export class E2BTerminalHandle implements SubprocessTerminalHandle {
|
||||
/** @inheritdoc */
|
||||
async inspectForeground(): Promise<SubprocessTerminalForeground | undefined> {
|
||||
try {
|
||||
const result = await this.sandbox.commands.run(`ps -o tpgid= -p ${this.pid}`)
|
||||
const result = await this.sandbox.commands.run(
|
||||
`ps -o tpgid= -p ${this.pid}`,
|
||||
commandOpts(this.controlEnvs),
|
||||
)
|
||||
return {
|
||||
processGroupId: parsePositiveId(
|
||||
result.stdout,
|
||||
@@ -369,7 +405,10 @@ export class E2BTerminalHandle implements SubprocessTerminalHandle {
|
||||
if (signal === 'SIGKILL' && foreground.processGroupId === this.pid) {
|
||||
throw new Error('refusing to SIGKILL the terminal shell; terminate the terminal session instead')
|
||||
}
|
||||
await this.sandbox.commands.run(`kill -${signal.slice(3)} -- -${foreground.processGroupId}`)
|
||||
await this.sandbox.commands.run(
|
||||
`kill -${signal.slice(3)} -- -${foreground.processGroupId}`,
|
||||
commandOpts(this.controlEnvs),
|
||||
)
|
||||
return foreground.processGroupId
|
||||
}
|
||||
|
||||
@@ -402,11 +441,11 @@ export class E2BTerminalHandle implements SubprocessTerminalHandle {
|
||||
}
|
||||
|
||||
private async closeOnce(): Promise<void> {
|
||||
let groups = await sessionProcessGroups(this.sandbox, this.sessionId)
|
||||
let groups = await sessionProcessGroups(this.sandbox, this.sessionId, this.controlEnvs)
|
||||
if (groups.length > 0) {
|
||||
this.terminationSignal = 'SIGTERM'
|
||||
await signalGroups(this.sandbox, groups, 'TERM')
|
||||
groups = await awaitSessionEmpty(this.sandbox, this.sessionId, this.graceMs)
|
||||
await signalGroups(this.sandbox, groups, 'TERM', this.controlEnvs)
|
||||
groups = await awaitSessionEmpty(this.sandbox, this.sessionId, this.controlEnvs, this.graceMs)
|
||||
}
|
||||
if (groups.length === 0 && !this.topLevelExited) {
|
||||
await Promise.race([this.done.catch(() => undefined), delay(this.graceMs)])
|
||||
@@ -421,7 +460,7 @@ export class E2BTerminalHandle implements SubprocessTerminalHandle {
|
||||
throw error
|
||||
}
|
||||
}
|
||||
groups = await awaitSessionEmpty(this.sandbox, this.sessionId, this.graceMs, true)
|
||||
groups = await awaitSessionEmpty(this.sandbox, this.sessionId, this.controlEnvs, this.graceMs, true)
|
||||
if (!this.topLevelExited) await Promise.race([this.done.catch(() => undefined), delay(this.graceMs)])
|
||||
}
|
||||
if (groups.length > 0) {
|
||||
@@ -469,13 +508,18 @@ export async function spawnE2BTerminal(
|
||||
let handle: CommandHandle | undefined
|
||||
let completion: Promise<CommandResult> | undefined
|
||||
let stateDirectoryCreated = false
|
||||
let controlEnvs: Record<string, string> = {}
|
||||
try {
|
||||
const ambient = await readRemoteEnvironment(sandbox, spec.signal)
|
||||
controlEnvs = bootstrapEnvironment(ambient)
|
||||
const environment = serializeRemoteEnvironment(ambient, spec.env)
|
||||
const argv = serializeValues(spec.argv, 'argv')
|
||||
await sandbox.files.makeDir(stateDir)
|
||||
stateDirectoryCreated = true
|
||||
await sandbox.commands.run(`chmod 700 -- ${quoteE2BShellArg(stateDir)}`, signalOpts(spec.signal))
|
||||
await sandbox.files.makeDir(stateDir, signalOpts(spec.signal))
|
||||
await sandbox.commands.run(
|
||||
`chmod 700 -- ${quoteE2BShellArg(stateDir)}`,
|
||||
commandOpts(controlEnvs, spec.signal),
|
||||
)
|
||||
await sandbox.files.write([
|
||||
{ path: paths.runner, data: TERMINAL_RUNNER_SOURCE },
|
||||
{ path: paths.environment, data: environment },
|
||||
@@ -484,13 +528,13 @@ export async function spawnE2BTerminal(
|
||||
], signalOpts(spec.signal))
|
||||
await sandbox.commands.run(
|
||||
`chmod 600 -- ${quoteE2BShellArg(paths.runner)} ${quoteE2BShellArg(paths.environment)} ${quoteE2BShellArg(paths.argv)} ${quoteE2BShellArg(paths.outputMarker)}`,
|
||||
signalOpts(spec.signal),
|
||||
commandOpts(controlEnvs, spec.signal),
|
||||
)
|
||||
handle = await sandbox.pty.create({
|
||||
rows: spec.rows,
|
||||
cols: spec.cols,
|
||||
cwd: spec.cwd,
|
||||
envs: { TERM: 'dumb' },
|
||||
envs: e2bControlEnvs(controlEnvs),
|
||||
timeoutMs: 0,
|
||||
onData: (data) => { outputFilter.push(data) },
|
||||
})
|
||||
@@ -504,13 +548,14 @@ export async function spawnE2BTerminal(
|
||||
await sandbox.pty.sendInput(handle.pid, Buffer.from(command), signalOpts(spec.signal))
|
||||
await waitUntilReady(sandbox, paths, completion, spec.signal)
|
||||
await waitForBootstrapOutput(outputFilter.ready, completion, spec.signal)
|
||||
const sessionId = await terminalSessionId(sandbox, handle.pid, spec.signal)
|
||||
const sessionId = await terminalSessionId(sandbox, handle.pid, controlEnvs, spec.signal)
|
||||
return new E2BTerminalHandle(
|
||||
sandbox,
|
||||
handle,
|
||||
output,
|
||||
completion,
|
||||
sessionId,
|
||||
controlEnvs,
|
||||
stateDir,
|
||||
spec.graceMs,
|
||||
spec.signal,
|
||||
@@ -524,7 +569,7 @@ export async function spawnE2BTerminal(
|
||||
if (!terminalQuiescent && handle !== undefined) {
|
||||
try {
|
||||
if (completion === undefined) await handle.kill()
|
||||
else await rollbackUnpublishedTerminal(sandbox, handle, completion, spec.graceMs)
|
||||
else await rollbackUnpublishedTerminal(sandbox, handle, completion, controlEnvs, spec.graceMs)
|
||||
terminalQuiescent = true
|
||||
} catch (cleanupError: unknown) {
|
||||
if (cleanupError instanceof SandboxNotFoundError) terminalQuiescent = true
|
||||
|
||||
@@ -111,6 +111,7 @@ class FakeSandbox {
|
||||
alive = true
|
||||
zombieOnly = false
|
||||
ambient = 'PATH=/ambient/bin\0KEEP=safe\0UNICODE=你好\0NPM_TOKEN=secret\0DSH_STALE=old\0BROKEN\0=bad\0'
|
||||
environmentHome = '/home/user'
|
||||
environmentWire: string | undefined
|
||||
environmentRequest: ((signal: AbortSignal | undefined) => Promise<void>) | undefined
|
||||
processGroupId = '4242\n'
|
||||
@@ -241,7 +242,9 @@ class FakeSandbox {
|
||||
if (this.envError !== undefined) throw this.envError
|
||||
return {
|
||||
exitCode: 0,
|
||||
stdout: this.environmentWire ?? Buffer.from(this.ambient).toString('base64'),
|
||||
stdout: this.environmentWire ?? [this.environmentHome, this.ambient]
|
||||
.map(value => Buffer.from(value).toString('base64'))
|
||||
.join('\n'),
|
||||
stderr: '',
|
||||
}
|
||||
}
|
||||
@@ -396,7 +399,14 @@ describe('E2BSubprocessHandle', () => {
|
||||
expect(handle.pid).toBe(4343)
|
||||
expect(fake.handle.sent.map(value => String(value))).toEqual(['hello'])
|
||||
expect(fake.handle.closes).toBe(1)
|
||||
expect(fake.startOptions?.envs).toBeUndefined()
|
||||
const controlEnvs = fake.startOptions?.envs
|
||||
expect(controlEnvs?.HOME).toMatch(/^\/\.dsh-e2b-control-/)
|
||||
expect(controlEnvs).toEqual({
|
||||
TERM: 'dumb',
|
||||
NPM_TOKEN: '',
|
||||
DSH_STALE: '',
|
||||
HOME: controlEnvs?.HOME,
|
||||
})
|
||||
const command = fake.commandsSeen.find(value => value.includes('exec "$dsh_e2b_env_bin" -i'))!
|
||||
expect(command).toContain('"$dsh_e2b_setsid" --wait -- "$dsh_e2b_bash" -c')
|
||||
expect(command).not.toContain('DEEPSEEK_API_KEY')
|
||||
@@ -405,7 +415,9 @@ describe('E2BSubprocessHandle', () => {
|
||||
expect(command).not.toContain('explicit-secret')
|
||||
expect(command).not.toContain('hyphen-value')
|
||||
expect(command).not.toContain('${!dsh_e2b_name}')
|
||||
expect(fake.commandsSeen).toContain('set -o pipefail; env -0 | base64 -w 0')
|
||||
expect(fake.commandsSeen).toContain(
|
||||
'set -o pipefail; printf \'%s\' "$PWD" | base64 -w 0; printf \'\\n\'; env -0 | base64 -w 0',
|
||||
)
|
||||
expect(command).toContain('mapfile -d')
|
||||
expect(command).toContain('dsh_e2b_node="$(command -v node)"')
|
||||
expect(command).toContain('"$dsh_e2b_env_bin" -i "$dsh_e2b_node" -e')
|
||||
@@ -421,7 +433,7 @@ describe('E2BSubprocessHandle', () => {
|
||||
'/workspace/.dsh-e2b/processes/one/stderr.log',
|
||||
])
|
||||
expect(fake.writtenFileData.get('/workspace/.dsh-e2b/processes/one/environment')).toBe(
|
||||
'PATH=/bin\0KEEP=safe\0UNICODE=你好\0FOO-BAR=hyphen-value\0--split-string=literal-value\0DEEPSEEK_API_KEY=explicit-secret\0DSH_MODE=test\0',
|
||||
'PATH=/bin\0KEEP=safe\0UNICODE=你好\0HOME=/home/user\0FOO-BAR=hyphen-value\0--split-string=literal-value\0DEEPSEEK_API_KEY=explicit-secret\0DSH_MODE=test\0',
|
||||
)
|
||||
|
||||
let piped = ''
|
||||
@@ -560,6 +572,7 @@ describe('E2BSubprocessHandle', () => {
|
||||
await expect(handle.done).resolves.toEqual({ exitCode: 0, signal: null })
|
||||
expect(fake.handle.disconnects).toBe(1)
|
||||
fake.alive = false
|
||||
handle.terminate()
|
||||
await expect(handle.waitForExit()).resolves.toBe(true)
|
||||
})
|
||||
|
||||
@@ -783,7 +796,7 @@ describe('E2BSubprocessHandle', () => {
|
||||
expect(fake.handle.kills).toBe(1)
|
||||
})
|
||||
|
||||
it('does not treat successful termination transport as observed quiescence', async () => {
|
||||
it('keeps force cleanup retryable until quiescence is proven', async () => {
|
||||
const fake = new FakeSandbox()
|
||||
fake.trapsTerm = true
|
||||
fake.delaysKill = true
|
||||
@@ -793,16 +806,15 @@ describe('E2BSubprocessHandle', () => {
|
||||
await flush()
|
||||
handle.terminate()
|
||||
await vi.waitFor(() => { expect(fake.handle.kills).toBe(1) })
|
||||
await expect(handle.waitForExit()).rejects.toThrow('force termination failed through both')
|
||||
expect(fake.alive).toBe(true)
|
||||
|
||||
let quiescent = false
|
||||
const waiting = handle.waitForExit().then((value) => { quiescent = value })
|
||||
await new Promise(resolve => setTimeout(resolve, 10))
|
||||
expect(quiescent).toBe(false)
|
||||
|
||||
fake.alive = false
|
||||
fake.finish()
|
||||
await waiting
|
||||
expect(quiescent).toBe(true)
|
||||
fake.delaysKill = false
|
||||
fake.delaysKillCompletion = false
|
||||
fake.sdkKillStops = true
|
||||
handle.terminate()
|
||||
await expect(handle.waitForExit()).resolves.toBe(true)
|
||||
await expect(handle.done).resolves.toEqual({ exitCode: null, signal: 'SIGKILL' })
|
||||
})
|
||||
|
||||
it('honors termination requested before asynchronous startup finishes', async () => {
|
||||
@@ -870,14 +882,10 @@ describe('E2BSubprocessHandle', () => {
|
||||
|
||||
handle.terminate()
|
||||
await vi.waitFor(() => { expect(fake.handle.kills).toBe(1) })
|
||||
let quiescent = false
|
||||
const waiting = handle.waitForExit().then((value) => { quiescent = value })
|
||||
await new Promise(resolve => setTimeout(resolve, 10))
|
||||
expect(quiescent).toBe(false)
|
||||
|
||||
await expect(handle.waitForExit()).rejects.toThrow('force termination failed through both')
|
||||
fake.alive = false
|
||||
await waiting
|
||||
expect(quiescent).toBe(true)
|
||||
handle.terminate()
|
||||
await expect(handle.waitForExit()).resolves.toBe(true)
|
||||
fake.releaseProcessGroupRead()
|
||||
fake.finish()
|
||||
await handle.done
|
||||
@@ -1102,23 +1110,36 @@ describe('E2BSubprocessHandle', () => {
|
||||
await expect(envHandle.done).rejects.toThrow('ambient lookup failed')
|
||||
expect(envFailure.removed).toEqual([])
|
||||
|
||||
const malformedEnvironment = new FakeSandbox()
|
||||
malformedEnvironment.environmentWire = '%'
|
||||
const malformedEnvironmentHandle = new E2BSubprocessHandle(
|
||||
runtime(malformedEnvironment),
|
||||
spec(),
|
||||
'/runtime/malformed-environment',
|
||||
const expectEnvironmentFailure = async (name: string, wire: string, message: string): Promise<void> => {
|
||||
const fake = new FakeSandbox()
|
||||
fake.environmentWire = wire
|
||||
const failed = new E2BSubprocessHandle(runtime(fake), spec(), `/runtime/${name}`)
|
||||
await expect(failed.done).rejects.toThrow(message)
|
||||
}
|
||||
const encodedEnvironment = Buffer.from('PATH=/bin\0').toString('base64')
|
||||
const encodedHome = Buffer.from('/home/user').toString('base64')
|
||||
await expectEnvironmentFailure('malformed-frame', '%', 'invalid base64')
|
||||
await expectEnvironmentFailure('malformed-base64', `${encodedHome}\n%`, 'invalid base64')
|
||||
await expectEnvironmentFailure(
|
||||
'invalid-utf8-home',
|
||||
`${Buffer.from([0xff]).toString('base64')}\n${encodedEnvironment}`,
|
||||
'not valid UTF-8',
|
||||
)
|
||||
await expect(malformedEnvironmentHandle.done).rejects.toThrow('invalid base64')
|
||||
|
||||
const invalidUtf8Environment = new FakeSandbox()
|
||||
invalidUtf8Environment.environmentWire = Buffer.from([0xff]).toString('base64')
|
||||
const invalidUtf8EnvironmentHandle = new E2BSubprocessHandle(
|
||||
runtime(invalidUtf8Environment),
|
||||
spec(),
|
||||
'/runtime/invalid-utf8-environment',
|
||||
await expectEnvironmentFailure(
|
||||
'invalid-utf8-environment',
|
||||
`${encodedHome}\n${Buffer.from([0xff]).toString('base64')}`,
|
||||
'not valid UTF-8',
|
||||
)
|
||||
await expectEnvironmentFailure(
|
||||
'relative-home',
|
||||
`${Buffer.from('home/user').toString('base64')}\n${encodedEnvironment}`,
|
||||
'remote login home is invalid',
|
||||
)
|
||||
await expectEnvironmentFailure(
|
||||
'nul-home',
|
||||
`${Buffer.from('/home/user\0tail').toString('base64')}\n${encodedEnvironment}`,
|
||||
'remote login home is invalid',
|
||||
)
|
||||
await expect(invalidUtf8EnvironmentHandle.done).rejects.toThrow('not valid UTF-8')
|
||||
|
||||
const cleanupFailure = new FakeSandbox()
|
||||
cleanupFailure.backgroundError = new Error('start failed before credential consumption')
|
||||
@@ -1318,10 +1339,11 @@ describe('E2BSubprocessHandle', () => {
|
||||
const failures = Array.from(failure.errors as Iterable<unknown>)
|
||||
expect(failures).toHaveLength(2)
|
||||
expect(failures[0]).toBeInstanceOf(Error)
|
||||
expect(failures[1]).toBeInstanceOf(Error)
|
||||
if (!(failures[0] instanceof Error) || !(failures[1] instanceof Error)) throw new Error('expected nested errors')
|
||||
expect(failures[1]).toBeInstanceOf(AggregateError)
|
||||
if (!(failures[0] instanceof Error) || !(failures[1] instanceof AggregateError)) throw new Error('expected nested errors')
|
||||
expect(failures[0].message).toContain('invalid process-group id')
|
||||
expect(failures[1].message).toBe('rollback signal failed')
|
||||
expect(failures[1].message).toBe('subprocess-e2b: force termination failed through both process-group and SDK transports')
|
||||
expect(Array.from(failures[1].errors as Iterable<unknown>)).toContainEqual(new Error('rollback signal failed'))
|
||||
expect(fake.handle.kills).toBe(1)
|
||||
const bounded = new AbortController()
|
||||
const waiting = handle.waitForExit(bounded.signal)
|
||||
@@ -1330,6 +1352,15 @@ describe('E2BSubprocessHandle', () => {
|
||||
handle.terminate()
|
||||
await expect(handle.waitForExit()).resolves.toBe(true)
|
||||
expect(fake.commandsSeen).toContain('kill -TERM -- -4242')
|
||||
|
||||
const naturallyGone = new FakeSandbox()
|
||||
naturallyGone.processGroupId = 'not-a-pid\n'
|
||||
naturallyGone.signalError = new Error('rollback signal failed')
|
||||
naturallyGone.handle.killError = new Error('SDK kill failed')
|
||||
const observed = new E2BSubprocessHandle(runtime(naturallyGone), spec(), '/runtime/failed-rollback-observed')
|
||||
await expect(observed.done).rejects.toThrow('process-group publication failed')
|
||||
naturallyGone.alive = false
|
||||
await expect(observed.waitForExit()).resolves.toBe(true)
|
||||
})
|
||||
|
||||
it('waits for delayed process-group publication', async () => {
|
||||
@@ -1479,6 +1510,24 @@ describe('E2BSubprocessHandle', () => {
|
||||
raced.terminate()
|
||||
await expect(raced.waitForExit()).resolves.toBe(true)
|
||||
})
|
||||
|
||||
it('rejects an optimistic SDK kill while descendants survive a failed group KILL', async () => {
|
||||
const fake = new FakeSandbox()
|
||||
fake.trapsTerm = true
|
||||
fake.sdkKillStops = false
|
||||
fake.signalErrors.push(undefined, new Error('KILL transport failed'))
|
||||
const handle = new E2BSubprocessHandle(runtime(fake), spec({ graceMs: 1 }), '/runtime/optimistic-sdk-kill')
|
||||
await flush()
|
||||
|
||||
handle.terminate()
|
||||
await expect(handle.waitForExit()).rejects.toThrow('force termination failed through both')
|
||||
expect(fake.alive).toBe(true)
|
||||
|
||||
fake.sdkKillStops = true
|
||||
handle.terminate()
|
||||
await expect(handle.waitForExit()).resolves.toBe(true)
|
||||
await expect(handle.done).resolves.toEqual({ exitCode: null, signal: 'SIGKILL' })
|
||||
})
|
||||
})
|
||||
|
||||
describe('E2BSubprocessService', () => {
|
||||
|
||||
@@ -23,6 +23,7 @@ function commandError(exitCode: number): CommandExitError {
|
||||
interface CommandOptions {
|
||||
signal?: AbortSignal
|
||||
cwd?: string
|
||||
envs?: Record<string, string>
|
||||
}
|
||||
|
||||
class FakeTerminalCommandHandle {
|
||||
@@ -101,6 +102,7 @@ class FakeTerminalSandbox {
|
||||
writeError: unknown
|
||||
sendError: unknown
|
||||
commandFailure: unknown
|
||||
makeDirRequest: ((signal: AbortSignal | undefined) => Promise<void>) | undefined
|
||||
sessionGroupsFailure: unknown
|
||||
foregroundFailure: unknown
|
||||
termFailure: unknown
|
||||
@@ -129,8 +131,10 @@ class FakeTerminalSandbox {
|
||||
|
||||
readonly sandbox = {
|
||||
files: {
|
||||
makeDir: async (path: string): Promise<boolean> => {
|
||||
makeDir: async (path: string, options?: CommandOptions): Promise<boolean> => {
|
||||
this.directories.push(path)
|
||||
await this.makeDirRequest?.(options?.signal)
|
||||
options?.signal?.throwIfAborted()
|
||||
return true
|
||||
},
|
||||
write: async (files: Array<{ path: string; data: string }>): Promise<object[]> => {
|
||||
@@ -163,7 +167,11 @@ class FakeTerminalSandbox {
|
||||
throw error
|
||||
}
|
||||
if (command.includes('env -0 | base64')) {
|
||||
return { exitCode: 0, stdout: Buffer.from(this.ambient).toString('base64'), stderr: '' }
|
||||
return {
|
||||
exitCode: 0,
|
||||
stdout: ['/home/user', this.ambient].map(value => Buffer.from(value).toString('base64')).join('\n'),
|
||||
stderr: '',
|
||||
}
|
||||
}
|
||||
if (command.includes('command -v -- ')) {
|
||||
return { exitCode: 0, stdout: this.resolvedExecutable, stderr: '' }
|
||||
@@ -262,7 +270,15 @@ describe('E2B terminal allocation', () => {
|
||||
expect(output).toBe('requested-shell$ ')
|
||||
expect(output).not.toContain('buffered banner')
|
||||
expect(output).not.toContain('runner.bash')
|
||||
expect(fake.createOptions).toMatchObject({ rows: 24, cols: 80, cwd: '/workspace', timeoutMs: 0, envs: { TERM: 'dumb' } })
|
||||
expect(fake.createOptions).toMatchObject({ rows: 24, cols: 80, cwd: '/workspace', timeoutMs: 0 })
|
||||
const controlEnvs = fake.createOptions?.envs
|
||||
expect(controlEnvs?.HOME).toMatch(/^\/\.dsh-e2b-control-/)
|
||||
expect(controlEnvs).toEqual({
|
||||
TERM: 'dumb',
|
||||
NPM_TOKEN: '',
|
||||
DSH_STALE: '',
|
||||
HOME: controlEnvs?.HOME,
|
||||
})
|
||||
expect(fake.inputs[0]?.data.toString()).toContain("exec /bin/bash '/runtime/terminal-one/runner.bash'")
|
||||
expect(fake.writes.get('/runtime/terminal-one/environment')).toContain('KEEP=visible\0')
|
||||
expect(fake.writes.get('/runtime/terminal-one/environment')).toContain('UNICODE=你好\0')
|
||||
@@ -554,6 +570,7 @@ describe('E2B terminal lifecycle', () => {
|
||||
new PassThrough(),
|
||||
fake.handle.wait(),
|
||||
123,
|
||||
{ TERM: 'dumb' },
|
||||
'/runtime/pre-aborted',
|
||||
1,
|
||||
controller.signal,
|
||||
@@ -807,7 +824,10 @@ describe('E2B subprocess terminal service', () => {
|
||||
fake.resolvedExecutable = 'tools/bin/node\n'
|
||||
await expect(ctx.subprocess.resolveExecutable('node', { PATH: 'tools/bin' }))
|
||||
.resolves.toBe('/workspace/tools/bin/node')
|
||||
expect(fake.commandOptions.at(-1)).toMatchObject({ cwd: '/workspace' })
|
||||
const commandOptions = fake.commandOptions.at(-1)
|
||||
expect(commandOptions).toMatchObject({ cwd: '/workspace' })
|
||||
expect(commandOptions?.envs?.HOME).toMatch(/^\/\.dsh-e2b-control-/)
|
||||
expect(commandOptions?.envs).toEqual({ HOME: commandOptions?.envs?.HOME })
|
||||
expect((ctx.e2b)).toBeDefined()
|
||||
})
|
||||
|
||||
@@ -867,6 +887,29 @@ describe('E2B subprocess terminal service', () => {
|
||||
expect(fake.handle.disconnects).toBe(1)
|
||||
})
|
||||
|
||||
it('owns and cancels terminal state-directory creation during disposal', async () => {
|
||||
const fake = new FakeTerminalSandbox()
|
||||
fake.makeDirRequest = async (signal) => {
|
||||
await new Promise<never>((_resolve, reject) => {
|
||||
const onAbort = (): void => {
|
||||
const reason: unknown = signal?.reason
|
||||
reject(reason instanceof Error ? reason : new Error(String(reason)))
|
||||
}
|
||||
signal?.addEventListener('abort', onAbort, { once: true })
|
||||
if (signal?.aborted === true) onAbort()
|
||||
})
|
||||
}
|
||||
const { ctx, fiber } = await service(fake)
|
||||
const spawning = ctx.subprocess.spawnTerminal(spec())
|
||||
const rejected = expect(spawning).rejects.toThrow('service disposed during terminal setup')
|
||||
await vi.waitFor(() => { expect(fake.directories.some(path => path.includes('/terminals/'))).toBe(true) })
|
||||
|
||||
await fiber.dispose()
|
||||
await rejected
|
||||
expect(fake.removed.some(path => path.includes('/terminals/'))).toBe(true)
|
||||
expect(fake.createOptions).toBeUndefined()
|
||||
})
|
||||
|
||||
it('retains failed terminal setup cleanup for disposal retry', async () => {
|
||||
const fake = new FakeTerminalSandbox()
|
||||
fake.sendError = new Error('bootstrap failed')
|
||||
|
||||
Reference in New Issue
Block a user