# Proof audit pass 1: terminal fallback process-table polling

PR [#24640](https://git.cafe/pingdotgg/t3code/pull/24640), branch `takeover/pr-16874`.
Reviewed head [cca46259f5](https://git.cafe/pingdotgg/t3code/commit/cca46259f59d82f31c7277cc2129b4a7185df890), before [945fd1df71](https://git.cafe/pingdotgg/t3code/commit/945fd1df71cf67311ffe50ef8f382ec812f83830). Full production and test diff inspected; only Manager.ts and Manager.test.ts change. This is a fixture specification, not a runtime result. No captures or tests were run for this pass. Existing older evidence is excluded, including purported supplementary proof unless freshly reproduced at these exact revisions.

## Findings first

1. **Proof is missing for this head.** Every row below is awaiting a before/after transcript. Existing unit tests were read, not executed; fake ProcessRunner / TestClock coverage does not prove real subprocess cancellation, native sidecar fallback selection, or application RPC publication. Windows remains `cannot-verify windows` on this Linux host.
2. **A PATH-only ps shim is insufficient on normal Linux.** Manager resolves `/bin/ps`, then `/usr/bin/ps`, before considering bare `ps` ([source](https://git.cafe/pingdotgg/t3code/blob/cca46259f59d82f31c7277cc2129b4a7185df890/apps/server/src/terminal/Manager.ts#L751)). Log the actual executable selected. Use an isolated absolute-command interceptor, not a claimed PATH override that silently leaves real ps untouched.
3. **Production and test cadence differ.** Production always supplies `nativeTelemetry.processTable`; native failure plus a successful fallback still sets `snapshotSucceeded=false`, increments failure count, and never earns quiet suppression. The new slow-runner tests omit `processTable`, so successful fallback resets failure count instead. Both branches need coverage; do not interpret their first 20-second slow-query gap as the perpetual production cadence.
4. **closeIdle is not an RPC method.** Its real callers are thread settlement (all terminals in a thread) and observed successful project-script completion (one terminal). Exercise those callers or clearly label a direct service invocation as harness evidence. A thread-settle RPC receipt alone does not prove asynchronous cleanup finished.
5. **Scope limit:** no confirmed implementation defect is asserted in this pass. Live forge status reports a merge conflict despite 38 successful checks; that is separate from this pinned-head proof assessment. Old links and green checks cannot satisfy these rows.

## Paths and observable shapes

- `make` supplies the native process table; `makeWithOptions` can instead omit it or supply a custom inspector. Custom inspector takes precedence. Native success reports elapsed=0; native failure invokes the platform fallback; absent native supplier uses fallback success/failure to decide failure count. Both polling and closeIdle call this acquisition path, with distinct fallback budgets.
- POSIX: resolve absolute ps, execute `-eo pid=,ppid=,comm=`, maximum stdout 524288 bytes; command failure, nonzero exit, timeout, or truncation makes a non-authoritative failed tick. Windows: `powershell.exe -NoProfile -NonInteractive -Command ... Get-CimInstance Win32_Process ...`, maximum stdout 262144 bytes, same rejection branches. Parsers themselves are unchanged.
- Poll: no running sessions -> no query; all eligible sessions quiet -> skip 29 ticks, then refresh; otherwise one snapshot shared by every running session with integer PID. Query failure logs and clears quiet bookkeeping, publishes no replacement activity. Successful acquisition inspects each captured terminal; inspector failure, exit, changed PID/process, removed or replaced session suppress stale publication. Changed busy/label emits activity and metadata upsert; unchanged data emits neither. Only trusted successful snapshots without a child and with unchanged activity mark earn quiet state after two snapshots.
- Poll completion updates failure count (reset or increment capped at 30) and sleeps. Default interval I=1000ms. New delay D=min(max(I*2^failureCount, 4*elapsedFallbackMs),60000). Before D=min(I*2^failureCount,60000). The measured fallback covers success AND typed failure. Successful results are applied before sleeping. Native request time, per-terminal processing and RPC delivery are not part of elapsedFallbackMs. Start-to-start spacing is query time + terminal processing + D + any following native query time; use end-to-next-start for comparisons, not start-to-start mislabeled as idle time.
- closeIdle holds the thread lock, selects running integer-PID sessions, optionally filters terminal ID, returns immediately if none, snapshots activity marks, inspects afresh, and retains busy or concurrently active terminals. It closes idle unchanged terminals. Acquisition/inspection failure logs and retains them. Its fallback budgets remain 1s POSIX / 1.5s Windows; native/custom query duration is not governed by those fallback budgets. `write` bypasses the thread lock; `resize` takes it, making resize a useful blocked-operation probe.
- Terminal create paths: open, initial attach / eligible attach restart, explicit restart, project scripts. Control paths: write, resize, close, exit, restart. Observe via `terminal.observe`, `terminal.attach`, terminal events and metadata subscriptions. `TerminalActivityEvent` has threadId, terminalId, sequence, hasRunningSubprocess, label. Metadata has snapshot/upsert/remove and per-terminal status/PID/busy/label. `terminal.open` returns a snapshot without the busy boolean; use activity or metadata to prove busy state.
- No component render, layout, count-dependent card, theme, width, viewport, provider adapter, wire schema, permission, desktop shell, or mobile-specific behavior changes. One RPC driver covers the shared server path used by web/desktop/mobile and remote clients. Do not claim native-client verification. Linux exercises the POSIX branch, not a real macOS host.

## Capture protocol and seeding

**Minimum dimensions:** before × after × each required row below; viewport=N/A, theme=N/A. All rows are **recording**: timestamped runtime JSONL/text sequences, including negative observation windows. No screenshot/video requirement. A continuous recording may contain several fixture phases if phase IDs, seed/reset boundaries and query identities remain unambiguous; each row still needs its own comparison. Parameterized variants explicitly named below require separate phases.

**Fresh state:** same driver and injected process wrapper bytes on each revision, separate new timestamped run directories and fresh databases under `/tmp/lead-24640-r1/`, explicit isolated home, no copied live user database, no real provider agents. Record full revision, clean worktree/source status, driver/wrapper hashes, launch argv (sanitized), Node/runtime/platform, DB path, server PID/port, run/phase identifiers, terminal count and IDs. Subscribe before creating terminals. Use RPC to create the project/thread needed for settlement; `terminal.open` gets threadId, terminalId, existing cwd, cols=100, rows=24. `terminal.write` starts a long-lived foreground Node program with a unique output readiness token, process PID, and controlled exit. Keep shell initialization deterministic and know whether real ps reports `node` or a wrapper shell as the immediate child. Commands and terminal output are public fixtures, never user data.

**Force real fallback:** start only the driver-owned server with `T3CODE_RESOURCE_MONITOR_PATH` pointing at a driver-owned, executable helper which exits immediately, and log sidecar health/failure. A nonexistent override alone is insufficient: the resolver can continue to a bundled real binary. Record repeated failure and actual fallback command launches. Native-success rows use the actual sidecar. Programmatic native fail/recover rows can use an explicit protocol helper, but label those native responses as synthetic; a direct injected service is a harness capture.

**Simulate slow ps without editing source or the host executable:** preferred options are (a) a private mount namespace that overlays only the server's resolved ps executable with a driver-owned wrapper and preserves an alias to the real executable, or (b) an external Node preload that intercepts only the exact ps invocation at the subprocess spawn boundary and redirects it to a driver-owned wrapper, preserving ProcessRunner, PTY, RPC, timeout and cancellation logic. For (b), patch/synchronize Node builtin exports before application imports, constrain interception to the server process/flag, and log requested command, exact arguments and replacement. Prove interception on a fast dry phase first. Such interception is a disclosed process-boundary fixture on a real server; direct fake ProcessRunner output is harness-only. If neither option works, mark the real-server rows missing rather than silently substituting unit tests. A PATH-only wrapper works only in an isolated filesystem where both absolute candidates are absent, or after explicitly changing command resolution (which would be harness evidence).

The wrapper uses one process with a timer (not a shell plus background sleep), logs monotonic query start/exit/signal and actual PID, then delegates to the preserved real ps for success rows. For delay-success capture, capture the process table at a documented instant, preferably just before output; keep the foreground Node child alive until results publish. Failure rows deliberately exit nonzero, exceed the output ceiling, or remain alive past deadline. Install a SIGTERM handler/log for hung fixtures and preserve normal kill semantics; record real exit, not just planned timeout. Do not intercept unrelated ps invocations. For concurrent closeIdle/background rows label launches by originating timeline/milestone and do not assume the one-background-query invariant also prohibits an independent closeIdle query.

**Every transcript records:** monotonic + UTC timestamps, RPC request/receipt and terminal readiness, raw subscribed activity/metadata/closed/output needed for that row, each query start/end/signal/exit and in-flight count, warnings/error fields, and final state. Before/after comparisons state what each side published, when, and measured idle gaps, not just pass/fail. Record actual ProcessRunner completion if available via existing logs/tracing; wrapper process-exit time is a lower-bound proxy, and cleanup/stream-drain/native request overhead must be identified. Scope teardown can extend the nominal 1s/15s timeout; do not insist real wall-clock equals that budget exactly. Keep signals and query completion observable.

**Negative claims:** observe until the relevant next-query/timeout milestone, then explicitly snapshot terminal metadata. Distinguish no duplicate activity (expected when busy/label unchanged) from missing the first busy event (failure). For low-count/many-terminal polling the maximum live background snapshot is 1; this does not mean one query per terminal. For teardown, log every spawned PID and prove none survives after driver cleanup. Never kill by process-name/path patterns. Do not upload tokens, pairing URLs, auth cookies or env dumps.

**Seeding closeIdle:** create a real local project and stopped thread via `projects.mutate` / orchestration RPC, open its terminals via terminal RPC, then dispatch `thread.settle` through `orchestration.dispatchCommand` and await terminal cleanup events/logs rather than only its receipt. The other caller requires a real configured project script with observed completion and exit code 0; identify the actual app workspace/setup action that consumes that completion effect before claiming it. If unavailable in the driver, use a labelled service harness for that caller and report this boundary. No imaginary `terminal.closeIdle` RPC is allowed. The two entry paths may share the same underlying query fixtures, but demonstrate both.

## Required fixture matrix

Status legend: `missing` = no capture yet; `harness missing` = this injectable branch cannot be selected through shipped RPC alone; `cannot-verify windows` = real Windows unavailable. Affected column: `yes` = changed timeout/pacing/result transport; `control` = refactored or adjoining behavior whose output must remain the same. Every row is recording. All Linux real-server rows are in the minimum capture set; harness and Windows rows remain explicit proof limits and cannot be silently counted as captured.

| ID | Named fixture / required data | Affected | Seed / capture surface | Required before → after transcript and acceptance | Status |
|---|---|---|---|---|---|
| R01 | Native success, one terminal, real foreground child then idle | control | Real sidecar; terminal open/write/exit, activity + metadata RPC | Both publish correct busy/label then idle, no fallback ps launches. No 4× fallback delay appears. Include unchanged-label phase with no duplicate activity. | missing |
| R02 | Native unavailable, fast valid ps, one busy terminal | yes | Failed sidecar helper; fast real-ps wrapper; RPC | Both publish busy/label; sequential shared queries. Production failures advance idle gaps about 2,4,8,16,32,60s (duration must be small enough not to dominate); do not expect perpetual 1s polling. At least first two gaps plus capped phase in R09. | missing |
| R03 | Native unavailable, fast valid ps, 20 busy terminals across two threads | yes | Same as R02; 20 unique IDs/readiness markers | All 20 receive correct busy labels and metadata from one query per polling round, not 20. Peak live background queries=1 and cadence matches one-terminal branch. | missing |
| R04 | Native unavailable, 0.75s valid ps, one busy terminal | yes | Wrapper delays 750ms, actual real ps result | Before accepts result with first failure-backoff idle gap about 2s; after accepts and first idle gap about 3s (=4×actual duration). Proves pacing changes even below old POSIX timeout; first successful event precedes that wait. | missing |
| R05 | Native unavailable, 5s valid ps, one busy terminal | yes | Wrapper delays 5s; keep child alive across query/result | Before terminates near 1s + cleanup and never publishes busy from that failed query; after completes near 5s, publishes true/expected label immediately, then next fallback begins after about 20s idle, not after 20s before publication. Observe at least next query. | missing |
| R06 | Native unavailable, 5s valid ps, 20 busy terminals | yes | Same as R05, 20 terminals/two threads | Before no busy results from timed-out query; after all 20 publish from shared result before pacing; peak live background query=1. Show per-terminal IDs, counts, and complete next-query gap. | missing |
| R07 | Native unavailable, 14s valid ps, one busy terminal | yes | Delay 14s (leave scheduling margin below 15s) | Before timeout; after successful near-budget table still applies before approximately 56s idle. Demonstrates near-deadline success distinct from timeout; measured elapsed determines prediction. | missing |
| R08 | Native unavailable, hung ps, one known-busy terminal | yes | Seed busy via prior fast query, then wrapper hangs beyond 15s | Before fallback times out near 1s; after near 15s plus measured teardown. Neither clears known busy/label on failure. After retry waits 60s from completed failed acquisition (4×elapsed cap); no overlap/orphan. Record at least timeout, wait and retry. | missing |
| R09 | Native unavailable, immediate nonzero failures through backoff cap, then native recovery | yes | Start busy; switch ps to exit 7; later restore real native sidecar via supported retry; if scripted native protocol used, disclose | Both preserve busy/label and warn; delays follow 2,4,8,16,32,60,60s without overflow. After recovery success resets failure count; subsequent first failure returns to 2s (subject to duration floor). Capture reset rather than inferring it. If real native recovery cannot be driven, split that segment into labelled H03 harness. | missing |
| R10 | Native unavailable, delayed 5s nonzero failure | yes | Start busy; wrapper waits 5s then exits 7 | Before times out; after reaches actual nonzero failure and logs correct failure kind. State remains busy; next delay reflects max(backoff,4×elapsed), first approximately 20s. Proves failed results are timed too, not only successes. | missing |
| R11 | Native unavailable, truncated process table | control | Start busy; wrapper emits >524288 bytes with valid-looking prefix quickly | Both reject whole result, warn with truncation, retain prior busy/label, advance failure backoff; never publish partial-table idle. No parser-change claim. | missing |
| R12 | Native unavailable, spawn failure | control | External interceptor targets a deliberately nonexistent executable for exact ps invocation | Both warn with spawn/cause information and retain prior state; after still advances normal failure backoff (near-zero elapsed). Restore wrapper and show recovery; no invented successful empty table. | missing |
| R13 | Zero running sessions: empty, all exited, then reopen | control | Fresh server, then open/exit all; observe metadata, reopen | No fallback while empty/exited; reopen resumes querying with failure count reset after an inactive loop. Allow already-started query to finish before claiming none. Starting/error/noninteger-PID synthetic states are H04 if needed, not real RPC seeds. | missing |
| R14 | Trusted idle native snapshots, quiet refresh, input/output wake | control | Real native sidecar; shell idle twice, then externally scheduled output/input and child; query telemetry/traces | Both earn quiet after two stable snapshots, skip 29 quiet ticks, refresh on 30th; input/output invalidates mark and refreshes at next eligible tick. Native-failed fallback does not enter this suppression (contrast R02). Do not claim 1s immediate wake during a previously scheduled long backoff. | missing |
| R15 | Close terminal while 5s query is in flight | yes | Gate on wrapper START, call terminal.close; keep another busy terminal for continued polling | Both close promptly independently of background query; before result times out, after result succeeds only for survivor. Metadata remove and closed precede any late result; no stale activity/upsert for closed ID. No query/PTY child survives cleanup. | missing |
| R16 | Restart same ID while 5s query is in flight; open a second new terminal | yes | Gate on START; terminal.restart same ID (new PID/process); terminal.open new ID | Both suppress old-process snapshot for restarted identity; new terminal is not accidentally populated from old candidate set. After next fresh query reports correct new identities. Capture old/new PIDs, event sequence and metadata to distinguish legitimate restart from stale upsert. | missing |
| R17 | closeIdle with no eligible sessions | control | Real settlement RPC for empty thread, then exited-only thread | Both cleanup paths perform no process-table query/closure when no eligible session. Await settlement cleanup milestone, not merely command receipt. | missing |
| R18 | closeIdle all-terminal selection: fast valid table, idle + busy + separate thread | control | Real thread settlement; one idle shell, one foreground Node, second thread busy | Both freshly query once and close only idle in target thread; busy survives and other thread unaffected. Empty child list is authoritative success; capture closed/remove and retained terminal output. No reliance on cached last busy flag. | missing |
| R19 | closeIdle single-terminal selection after script success | control | Real observed project-script completion exit 0; target idle, unrelated idle sibling | Both close only the script's targeted idle terminal, leaving idle sibling intact; log script completion, target ID and query. Exit nonzero must not call cleanup (paired control phase). If no reachable completion consumer can be driven, mark harness-only rather than R18 proving both callers. | missing |
| R20 | closeIdle hung POSIX fallback with queued same-thread resize | yes | Real settlement cleanup; gate on closeIdle query START then terminal.resize 80×30 | Both short budgets remain about 1s + real teardown, retain terminal, warn, and release thread lock so resize completes. Show queued resize request before query ends and receipt after cleanup; no 15s wait. Other-thread operation can complete during wait. | missing |
| R21 | Background slow query concurrent with closeIdle short query | yes | Start 5s background query, then settle thread and queue resize; identify both query starts | After background may run 5s/apply only still-live candidates, while closeIdle independently times out near 1s and releases resize; before both use short fallback budget. Two concurrent queries are allowed here (one background, one cleanup); do not present this as polling overlap. Repeat closeIdle on hung background if needed to separate ownership. | missing |
| R22 | closeIdle fast-enough valid table but input/output changes during query | control | Wrapper delay 500ms; use terminal.write during query and separate asynchronous PTY output phase; real settlement | Both preserve newly active terminal despite stale idle table, while truly unchanged idle sibling closes. Prove activity mark changed before result, no close/remove for active ID, and fresh writes still work. Writes are not the lock probe; resize is R20. | missing |
| H01 | No supplied processTable: fast/5s fallback, failure then successful reset | yes | Labelled makeWithOptions harness, real ProcessRunner preferred, subscribed manager events | Explicitly omit processTable. Successful fallback has snapshotSucceeded=true; fast success delay 1s, 5s success after=20s versus before timeout. After a failed tick, later success resets count. This is not the production native-failure path. | harness missing |
| H02 | Custom inspector success and per-terminal error | control | Labelled makeWithOptions harness with explicit subprocessInspector, processTable spy and registered process IDs | Both bypass process-table/native/fallback completely. Delayed success does not add 4× fallback wait; per-terminal typed failure warns/skips that terminal without corrupting others. Successful custom inspector contributes elapsed=0; include closeIdle invocation and its caught error. | harness missing |
| H03 | Native request slow success, failure + fallback, then recovery | control | Real sidecar if controllable; otherwise labelled processTable Effect harness / disclosed native protocol helper | Native-only delay does not count toward fallback elapsed and is not bounded by 15s/closeIdle's 1s fallback timeout. Failure triggers fallback, measuring only fallback time; later native success resets counter and quiet eligibility. Respect real native client's own request timeout; never call arbitrary injected 20s native success a shipped-sidecar result. | harness missing |
| H04 | Running status with no integer PID, and non-running candidate filtering | control | Labelled service/PTY harness; cannot force malformed PID through validated shipped RPC | Both skip noneligible candidates and do not query if none eligible; mixed valid/invalid set queries once and only inspects valid running PID. Covers internal filter branch without claiming real corrupt terminal state. | harness missing |
| W01 | Windows native unavailable, fast valid PowerShell, one and 20 terminals | yes | Real Windows host, failed sidecar, exact PowerShell process fixture; same RPC driver | Both publish normalized child labels; one shared query; successful fallback after native failure still uses failure backoff. No POSIX result accepted as Windows proof. | cannot-verify windows |
| W02 | Windows 5s valid PowerShell, one and 20 terminals | yes | Real Windows controlled PowerShell delay, valid Win32_Process result | Before 1.5s timeout/no busy result; after completes 5s, all labels apply before approximately 20s idle; maximum one background query. Fake win32 TestClock tests remain harness-only. | cannot-verify windows |
| W03 | Windows hung PowerShell and delayed nonzero/truncated result phases | yes | Real Windows; seed known busy, then controlled failures (ceiling 262144) | Hung query after nominal 15s + teardown and 60s retry cap, before 1.5s; no false idle. Delayed failure is timed; truncation/nonzero/spawn failures preserve cause/state. Distinguish timeout vs query failure. | cannot-verify windows |
| W04 | Windows close/restart during slow query | yes | Real Windows; close/restart RPC after logged query START | Correct metadata remove/restarted/new PID; no late old-process activity; surviving terminals still update; no orphan PowerShell/PTY process after tracked cleanup. | cannot-verify windows |
| W05 | Windows closeIdle fast mixed selection, then hung query + resize | yes | Real settlement and observed-script completion on Windows | Both retain 1.5s fallback budget despite 15s background budget, retain uncertain busy state, release thread lock for resize; fast valid table closes only selected idle terminals, active mark changes retain them. | cannot-verify windows |

## Comparison report acceptance

For each row include: run paths and SHA/hash manifest; seed/input summary; before published state and timestamps; after published state and timestamps; query duration/exit kind; completion-to-next-start gaps and predicted formula; applicable timeout tolerance with cleanup overhead; in-flight background count; cleanup evidence; observed/missing/cannot-verify result. Row grouping cannot hide missing count/platform/interaction variants.

The minimum Linux reproduction of the main fix is R05 (publish 5s result before pacing), R06 (shared many-terminal snapshot), R08 (15s bounded hang + 60s cap), and R20 (closeIdle still short). These four alone are NOT sufficient for the requested full path audit; the full required matrix above includes success/failure floor, reset/cap, native/absent/custom distinctions, stale identity and both cleanup callers. Extended backoff observations can be shared between R02/R09 to reduce runtime without losing phases.

No server, browser, emulator or lease was started by this audit. No code/PR state was changed. The lead owns the worktree and scratch root; only audit1 is removed after uploading this document. Free space at start: 18 GB.

FRICTION: brief | suggested PATH-only ps shim cannot intercept the manager's preferred absolute ps path | specify an isolated absolute-command interceptor and require executable-selection proof

VERDICT: INSUFFICIENT R01-R22, H01-H04, W01-W05