Skip to content
17.7k

Windows Memory Growth

Some Windows users see the bun process behind opencodex grow to many gigabytes of RSS during long streaming sessions (reported as issue #314). This page explains what is actually happening and what you can do about it, honestly.

opencodex bundles the Bun runtime (currently 1.3.14). The memory growth is driven by known upstream Bun issues, not by JavaScript-level leaks in the proxy:

Bun issue State (checked 2026-07-23)
#28035 — fetch() receive backpressure not coupled to JS consumption Fixed by PR #29831; which release carries it is unverified — we assume the bundled 1.3.14 does not
#32111 — crash when a client aborts an async-pull stream Fix PR #32120 merged 2026-06-21; not assumed present in 1.3.14. Note: this crash is not Windows-specific (it also reproduced on macOS/Linux)
PR #31654 — node:net socket handle leak Still open upstream

On Windows, opencodex must keep streaming responses on a conservative code path to avoid the #32111 crash, and that path is the one most exposed to the backpressure issue: a slow or stalled client can leave the runtime buffering upstream data in native memory that JavaScript cannot bound.

Bounded mitigation and visibility — not a fix. On the bundled 1.3.14 runtime the leak itself remains an upstream problem:

  • Memory watchdog — the proxy samples its own memory every minute and logs a rate-limited warning when observed memory crosses 4 GiB. Observed memory is the largest of RSS, external, and arrayBuffers (not their sum), because Windows working-set/RSS counters can under-report committed external retention.

  • ocx doctor — a “Memory / runtime” section shows the service process’s Bun version, RSS, external/ArrayBuffers counters, JS-heap context, and stream-mode decision. On the bundled Bun 1.3.14 runtime, heapUsed / jscHeap alone are not a leak discriminator; compare observed memory with responseState and repeated samples before assigning an app-level leak.

  • GET /api/system/memory — the same data over the authenticated management API for dashboards or scripts. Alongside RSS/heap/external counters it reports a scalar responseState block (entry count, total/largest serialized bytes, oldest-entry age) for the proxy’s in-memory previous_response_id continuation store. This further attributes growth: a rising responseState.totalBytes under rising observed memory points at conversation retention (long store:false chains re-expanding each turn), whereas a flat responseState under rising observed memory points away from that store. The values are scalar-only — no request bodies, tokens, paths, or account identifiers — and the read is side-effect free (it never prunes or evicts). Spill-write health is reported separately inside that block: spillWriteStatus starts as initial, becomes degraded after a failed publication, and returns to healthy after the next successful publication. spillWriteConsecutiveFailures, the last failure time, and the last success time show whether failures are accumulating or recovering in the same process. The last failure is a fixed privacy-safe class such as EACCES, ENOSPC, ETIMEDOUT, or EACLRETRYEXHAUSTED; raw error messages and filesystem paths are never returned. spillLastWriteFailureOrigin adds a fixed origin or null: retry_returned_timeout means the existing second spill attempt returned a timeout; timeout_memo_refusal means the ACL helper refused through its remembered timeout state. Other failures use null. The cumulative spillAclRetryReturnedTimeouts and spillAclTimeoutMemoRefusals count terminal failed publications, not individual ACL commands or transient first attempts. When a new continuation does not fit under the 1 GiB spill ceiling, the proxy first evicts the oldest stored continuations to make room, as long as that can succeed; a continuation that cannot fit even then is refused. The cumulative spillHeadroomEvictions counts continuations evicted to make room, and spillCapacityRefusals counts continuations refused because the ceiling could not make room (each one is also a failed publication with class ECAPACITY). A rising spillCapacityRefusals while memory is healthy means the durable continuation set is full of space the proxy cannot reclaim — in-flight publications or files a failed delete left behind — rather than a memory problem. Success clears the failure streak but retains the last failure fields and cumulative counts; a later unrelated failure sets the last origin to null. These values are process-local, so compare snapshots from the same process. Neither origin identifies an OS command: the attempt budget can expire before a command starts, and an optional compliance inspection can run before a memo refusal. A separate process succeeding does not prove that the live process’s memo recovered. These observations do not add retries, clear memos, weaken required ACLs, or automatically restart the service.

    These diagnostics stay on the authenticated management endpoint and are intentionally absent from /healthz, which remains a liveness signal. The dashboard’s Memory observability card renders the memory and continuation-size fields from this endpoint and offers a confirm-gated Drain & restart action: it shows the current active-turn count, waits up to 60s for active turns (reusing the existing HTTP 503 JSON server_restarting + Retry-After: 5 drain), then aborts any remaining turns. The running proxy owns restart authorization and drain coordination, then exits; an installed service manager launches the replacement when applicable. The action reports success only after a different, identity-verified process is healthy on the same port, without tearing down Codex injection. That is a longer, informed recycle than the short drain on POST /api/stop.

    For a scriptable snapshot of the complete authenticated payload, run ocx observe memory --json; the CLI forwards the same response-state fields.

  • A gated alternative stream path — a bounded single-reader relay that removes the unbounded buffering shape entirely. On Windows it becomes the default automatically once a bundled Bun release verifiably carries the #32111 fix; today it is opt-in only (see below). On macOS it stays opt-in even after such a release — flipping macOS auto is a separate decision.

Catalog metadata reads use the selected runtime’s last confirmed bundled catalog. Cold or changed selections refresh through asynchronous version/model probes, so those probes do not hold the proxy event loop. Shared refreshes have a 45-second deadline; failed request-triggered refreshes retain matching rows and apply a 60-second retry cooldown. Request abort releases translator accounting even when a response body is unread. These liveness changes do not establish that Windows native stream retention (#6671) is resolved.

Real-world RSS improvement from these changes is awaiting verification by Windows users — we do not claim the leak is fixed.

Threshold-based auto-restart is deliberately not shipped. If the process crashes, the service managers (Task Scheduler/WinSW, launchd, systemd) already restart it.

  1. Wait for a bundled runtime update. Once a Bun release verifiably carries the fixes, opencodex will bump the bundled runtime and the safer stream path turns on automatically on Windows (macOS keeps requiring the explicit opt-in below).

  2. Run a Bun runtime you trust with OPENCODEX_BUN_PATH. Set the variable before launching ocx from that shell, then run ocx service repair to bake the selected runtime into the service definition. A later project .env does not change selection; changing the environment alone does not update an installed service.

  3. Opt into the bounded relay with streamMode: "eager-relay". Two ways: edit config.json (add "streamMode": "eager-relay"), or call the management API — a PUT /api/settings with {"streamMode":"eager-relay"} applies to new turns without a restart. Crash risk warning: on Bun 1.3.14 this uses the stream shape affected by #32111, which can crash the process mid-stream (on any OS, not just Windows). The service manager will restart it, but in-flight requests fail. "legacy-tee" pins the current default. On Windows, "auto" (default) lets the runtime gate decide. On macOS, "auto" always stays on tee; explicit "eager-relay" is the opt-in.

If you try any of these on a real Windows workload, please report the before and after ocx doctor memory sections on #314 — that is exactly the verification this mitigation is waiting on.

Windows service installation, repair and Codex shim installation/refresh check whether the selected executable can create and remove a directory inside the OpenCodex config root. This is a write-policy check, not a memory fix. Failure refuses before service stops, downloads, staging, token or launcher writes. Automatic shim repair is deferred with guidance and startup continues. Healthy or disabled shims do not probe. No alternate runtime is discovered. A first-time scheduler install registers its task and claims the config root before this check; on refusal that new registration is rolled back and the claimed root stays for a retry.

The selected Bun runtime could not create and remove a directory in the OpenCodex config directory. Windows application policy may deny writes from this executable location. Set OPENCODEX_BUN_PATH to a trusted Bun executable allowed by your policy before launching ocx, or reinstall opencodex with npm install -g @bitkyc08/opencodex, then retry. No alternate runtime was selected.