# PR #22996 runtime-evidence fixture matrix (proof auditor pass 1)

PR: https://github.com/pingdotgg/t3code/pull/22996 (GitCafe 22999), fixes the shutdown half of #22983.
Before = `8d3b0611` (merge-base on main). After = `bd63fded` (PR head); merged head `efd97c35`.

## What the diff changes (code paths)

| Path | File:line (after) | Before | After |
|---|---|---|---|
| `closeRun` with timeout | `apps/desktop/src/backend/DesktopBackendManager.ts:298-331` | Forks `close` (lazily) and waits up to `timeout`. On timeout it returns `false`, `stop` skips the state cleanup and **returns right away**, so the quit continues while the backend is still alive. | Forks `close` (`startImmediately: true`) and waits up to `timeout`. On timeout it logs `backend shutdown exceeded timeout; waiting for exit` {timeoutMs: 7000}, **then waits for `close` without a limit**, then always runs the state cleanup. |
| `closeRun` without timeout | same | `close` then `true` | `close` (the same in practice) |
| Quit threshold constant | `DesktopBackendManager.ts:44` `BACKEND_QUIT_TIMEOUT` = 5s grace + 2s = **7s** | hardcoded `Duration.seconds(5)` | 7s, used by both callers below |
| Normal / signal / relaunch quit | `apps/desktop/src/app/DesktopApp.ts:150-160` `stopAllPoolInstances` (scopedProgram finalizer `DesktopApp.ts:342-352`), reached from `DesktopLifecycle.ts` `handleBeforeQuit` (menu Quit, Cmd/Ctrl+Q, `window-all-closed` on non-darwin), `quitFromSignal` (SIGINT/SIGTERM, **non-win32 only**, `DesktopLifecycle.ts:256-263`), `relaunch` | 5s, then give up | 7s warning, then wait |
| Updater quit-and-install | `apps/desktop/src/updates/DesktopUpdates.ts:756-760` before `electronUpdater.quitAndInstall` | 5s, then give up and call `quitAndInstall` | 7s warning, then wait, then `quitAndInstall` |
| Process kill escalation (unchanged, but now always waited on) | `.repos/effect-smol/packages/platform/node-shared/src/NodeChildProcessSpawner.ts:473-494`, spawn options at `DesktopBackendManager.ts:457-458` | POSIX: SIGTERM to the process group `-pid`, wait 5s (`forceKillAfter`), SIGKILL the group, wait 1s, then **`Deferred.await(exitSignal)` with no limit**. win32: `taskkill` of the tree, only the leader's exit is awaited. | same mechanism; the app now always waits for it to finish |
| Pool scope close / unregister | `apps/desktop/src/backend/DesktopBackendPool.ts:252-300`; the only caller is `apps/desktop/src/wsl/DesktopWslBackend.ts:90` | — | Tests added (`DesktopBackendPool.test.ts`): unregister waits for exit, unregister during pool quit does not interrupt sibling stops, pool scope close stops a registered backend |

Lock under test: `acquireServerOwnership` (`apps/server/src/serverOwnership.ts:181-238`). It holds an exclusive sqlite transaction on `<stateDir>/server-owner.sqlite` for the server's lifetime. The OS releases it when the process exits, including on SIGKILL. Alongside it, `<stateDir>/server-runtime.json` (`apps/server/src/config.ts:146`) holds `ownerId`, and the server's release deletes it. A second server that finds the lock taken fails with `ServerAlreadyRunningError`.
With `T3CODE_HOME=$H` and no `VITE_DEV_SERVER_URL`, `stateDir = $H/userdata` (`apps/desktop/src/app/DesktopStatePaths.ts:23-31`). The backend inherits the same home.

## Fixtures

Legend: **Rec** = whether a screen recording is needed. The default is a transcript: timestamped `ps`, `lsof`, the lock probe, and log tail.

| ID | Scenario / code path | What it proves | Before (8d3b0611) | After (bd63fded) | Rec | How to drive it on Linux (headless-desktop) |
|---|---|---|---|---|---|---|
| **F1** | Normal quit, idle cooperative backend; signal path (`quitFromSignal` SIGTERM → `stopAllPoolInstances`) | Regression guard: the fast path is unchanged, there's no 7s stall, and SIGKILL isn't used | Backend exits on SIGTERM in well under 5s, Electron exits, the lock is free, an immediate relaunch acquires the lock | Same. No `exceeded timeout` warning. Quit takes about the same time as Before | No | Launch (recipe R1), wait for `server-runtime.json` and a `lsof` holder. `kill -TERM $ELECTRON_MAIN_PID`. Record `date +%s.%N` before the kill and after `wait`. Then run P1 (probe) and P2 (relaunch) |
| **F2** | Normal quit via `before-quit` (window close → `window-all-closed` → `app.quit` → `handleBeforeQuit`, non-darwin) | The UI quit path takes the same finalizer as F1 | Same as F1 | Same as F1 | Optional (only for showing the window disappear; the transcript is enough) | Launch with `--remote-debugging-port=9333`. Run `curl -s localhost:9333/json` to get the page WS URL, then evaluate `window.close()` over CDP (or use the `agent-browser` skill against the CDP port). xdotool is **not installed**, so Ctrl+Q / menu Quit can't be scripted |
| **F3** | Backend ignores SIGTERM (hung event loop, or a provider stuck in teardown). Core bug | Quit waits for SIGKILL escalation and the lock is released before Electron exits | `stop` gives up at 5s, the finalizer completes, and Electron exits while SIGKILL is never sent (the forked close dies with the process). The backend is detached in its own process group, so it **survives as an orphan in state `T`**, still holding `server-owner.sqlite`. P1 → BUSY. P2 → relaunch fails (`ServerAlreadyRunningError` / "already running" in the backend log) | Backend PID gets SIGTERM, no exit; at about 5s the spawner sends SIGKILL; it exits; Electron exits at about 5-6s. 7s isn't reached, so there's no warning. P1 → free, P2 → relaunch OK, no orphan in `ps` | No (transcript with timestamps) | Launch (R1). Find the backend PID as a child of the captured Electron main PID (R2). `kill -STOP $BACKEND_PID`. A stopped process doesn't act on SIGTERM, but SIGKILL still works. Then `kill -TERM $ELECTRON_MAIN_PID`. Sample `ps -o pid,pgid,stat,etime,cmd -g $BACKEND_PGID` every 0.5s until Electron exits. Then run P1 and P2. Before only: clean up the orphan with `kill -KILL $BACKEND_PID` (a PID you captured) |
| **F4** | Backend slow but cooperative (finishes graceful shutdown within the 5s grace, e.g. a busy turn tearing down provider sessions) | Quit waits for the graceful exit and there's no SIGKILL | Exit before 5s: same as After. (Before only breaks above 5s total, see F3) | Backend exits on the delayed SIGTERM, there's no SIGKILL, the app exits after the delay, the lock is free | No | Same as F3, but schedule `(sleep 3; kill -CONT $BACKEND_PID) &` right after sending SIGTERM to Electron. The pending SIGTERM is delivered on CONT and the server's own shutdown runs. Expected: the backend log shows a graceful-shutdown line and the exit happens at about 3s. Optionally use `sleep 4.5` to sit close to the grace boundary |
| **F5** | Quit with a real turn or long request in flight | A real workload's provider subprocesses (same process group) are torn down, and the app doesn't exit early | Turn teardown longer than 5s → orphaned backend and/or provider children, lock held (as in F3) | Waits through SIGTERM → grace → SIGKILL of the **whole process group**, so provider children die too, the lock is free | No | Needs a provider in the isolated home. Copy only provider credentials if the maintainers allow it, never `~/.t3/userdata`. Seed with `vp run migrate-dev-db`-style data copied into `$H/userdata`. Start a long turn, then quit as in F1. After exit, check `ps -o pid,pgid,stat,cmd -g $BACKEND_PGID` is empty. **If no provider is configurable, mark this partial; F3/F4 cover the timing mechanics** |
| **F6** | Warning branch: `close` exceeds 7s (SIGKILL + 1s wait not done by 7s). New `logBackendProcessWarning` + continued wait | The 7s timeout is only a warning; quit still waits for exit | No warning; at 5s it gives up (as F3) | `backend shutdown exceeded timeout; waiting for exit` {timeoutMs:7000} in the desktop log, then waits until exit | No | Hard to reach naturally: SIGKILL of a stopped process exits immediately. The only real trigger is a process the kernel can't kill quickly (D-state, e.g. a hung FUSE/NFS read) or a zombie descendant holding the group. On Linux without root FUSE: **UNVERIFIABLE-HERE in practice**. Covered by the unit test `the shared quit budget is strictly greater than terminate grace plus kill wait`. Note the risk: after SIGKILL the spawner's `Deferred.await(exitSignal)` has no limit, so a D-state backend now hangs quit **indefinitely** (Before: the app exited after 5s). Reviewers should accept this explicitly |
| **F7** | Updater quit-and-install (`DesktopUpdates.ts:756`) | Backends have exited before `autoUpdater.quitAndInstall` | `quitAndInstall` runs after 5s even if the backend is alive → orphan holds the lock across the relaunched new version | Waits for exit, then `quitAndInstall` | Yes, only if run (the install/relaunch is the visible step) | Needs a **packaged** build (`isPackaged`), `APPIMAGE` set (Linux requires AppImage or .deb, `DesktopUpdates.ts:323-331`), two versions served by `scripts/mock-update-server.ts` (`T3CODE_DESKTOP_MOCK_UPDATES=1`, `T3CODE_DESKTOP_MOCK_UPDATE_SERVER_PORT`, `T3CODE_DESKTOP_MOCK_UPDATE_SERVER_ROOT`) and `vp run dist:desktop:linux` twice with different versions. Feasible on Linux but costly (two AppImage builds). **UNVERIFIABLE-HERE within the 30-min budget**. The code path is the same `instance.stop({timeout: BACKEND_QUIT_TIMEOUT})` as F3, so F3 gives indirect evidence. Unit coverage only |
| **F8** | Relaunch (`DesktopLifecycle.relaunch` → `requestDesktopShutdownAndWait` → same finalizer, then `app.relaunch` + `exit(0)`) | The new instance can open the DB immediately | Slow backend → new instance's backend fails to acquire the lock | New instance starts cleanly | No | Triggered by app-internal relaunch reasons (settings changes such as Linux password store). Not cheaply scriptable from outside. F3 + P2 (manual relaunch right after exit) prove the same property. **Optional** |
| **F9** | Concurrent unregister + pool quit (WSL secondary backend) | A sibling stop isn't interrupted by unregister; unregister waits for exit | — | Covered by new `DesktopBackendPool.test.ts` cases | No | The only `pool.unregister` caller is `apps/desktop/src/wsl/DesktopWslBackend.ts:90` (WSL). **UNVERIFIABLE-HERE: Windows + WSL only** |
| **F10** | win32 kill branch (`taskkill` tree, only the leader awaited; no SIGINT/SIGTERM listener on win32, `DesktopLifecycle.ts:256`) | The quit wait works with taskkill | 5s give-up | Waits for the leader's exit | — | **UNVERIFIABLE-HERE: Windows only** |
| **F11** | macOS quit (Cmd+Q / Dock Quit → `before-quit`; `window-all-closed` doesn't quit on darwin; Squirrel.Mac updater relaunch) | Same finalizer on darwin | 5s give-up | Waits | — | **UNVERIFIABLE-HERE: macOS only.** Same `handleBeforeQuit` code as F2, which Linux exercises |
| **F12** | Error/edge: stop with no active run (`Option.none` run), backend already crashed before quit | No hang on the empty path | Returns immediately | Returns immediately | No | Launch, `kill -KILL $BACKEND_PID` (captured PID), wait for the desktop log to show the backend exited (desired-running restart may respawn it, so capture the new PID if it does), then quit as in F1. Expect a prompt exit and P1 free |

Recommended minimal set for pass 2: **F1, F3, F4** (Before and After each), plus F2 once on After and F12 once on After. Treat F5 as a stretch goal. F6, F7, F9, F10, and F11 are justified as unit-covered / UNVERIFIABLE-HERE.

## Driver recipe

All commands below were checked to exist on this machine: `headless-desktop` (`~/.local/bin`, wraps `xvfb-run -a` + `dbus-run-session` + an unlocked gnome-keyring), `xvfb-run`, `lsof`, `fuser`, `ffmpeg`, node v24.22.1 with `node:sqlite` (probe tested), Electron at `apps/desktop/node_modules/electron/dist/electron`. **Not installed: `xdotool`.**

### R0 — build each side (separate checkouts)
```bash
# After: ~/wt/pr-22996 is at efd97c35 (merge); for strict "after" use bd63fded in its own worktree
git -C ~/t3code worktree add /tmp/ev-22996/before 8d3b061121fe65669d5d43d011b48cd3027a4108   # pass 2, not done here (read-only)
(cd <checkout> && vp i && vp run build:desktop)   # root script: builds t3 (apps/server/dist/bin.mjs) + desktop (dist-electron/boot.cjs)
```

### R1 — launch with isolated state (never ~/.t3)
```bash
H=/tmp/ev-22996/<side>/home; X=/tmp/ev-22996/<side>/xdg; mkdir -p $H $X/config $X/data
cd <checkout>/apps/desktop
env -u VITE_DEV_SERVER_URL -u ELECTRON_RUN_AS_NODE -u T3CODE_PORT \
  T3CODE_HOME=$H XDG_CONFIG_HOME=$X/config XDG_DATA_HOME=$X/data \
  T3CODE_DISABLE_AUTO_UPDATE=1 ELECTRON_ENABLE_LOGGING=1 \
  headless-desktop ./node_modules/electron/dist/electron --no-sandbox --disable-gpu \
    [--remote-debugging-port=9333] dist-electron/boot.cjs > /tmp/ev-22996/<side>/stdout.log 2>&1 &
WRAPPER_PID=$!
```
- `T3CODE_HOME` sets the base dir. Production mode (no `VITE_DEV_SERVER_URL`) gives stateDir `$H/userdata`. Electron `userData` resolves under `XDG_CONFIG_HOME/t3code-v2` (`DesktopUserData.ts:42-45`).
- Desktop + server logs: `$H/userdata/logs/` (primary backend output: `server-child.log`, `DesktopObservability.ts:429-432`).
- Unset `T3CODE_PORT`: production picks a free port by scan.

### R2 — find PIDs without killing by pattern
```bash
ps -o pid,ppid,pgid,stat,etime,cmd --ppid $WRAPPER_PID --forest   # walk xvfb-run -> dbus-run-session -> bash -> electron
# ELECTRON_MAIN_PID = the electron process whose cmdline ends in dist-electron/boot.cjs and is a descendant of $WRAPPER_PID
# BACKEND_PID = the child of ELECTRON_MAIN_PID whose cmdline contains apps/server/dist/bin.mjs (ELECTRON_RUN_AS_NODE child)
cat /proc/$BACKEND_PID/environ | tr '\0' '\n' | grep '^T3CODE_HOME='   # confirm it is OUR $H before signalling
BACKEND_PGID=$(ps -o pgid= -p $BACKEND_PID)
```

### R3 — lock state
```bash
L=$H/userdata/server-owner.sqlite
lsof $L ; fuser -v $L                       # holder PID(s)
cat $H/userdata/server-runtime.json         # present while running; release deletes it if ownerId matches
```
**P1 — lock probe** (same primitive as the server; doesn't block, doesn't hold the lock):
```bash
node -e 'const {DatabaseSync}=require("node:sqlite");const d=new DatabaseSync(process.argv[1]);
try{d.exec("PRAGMA busy_timeout=0; BEGIN EXCLUSIVE;");d.exec("ROLLBACK");console.log("LOCK FREE")}
catch(e){console.log("LOCK HELD:",e.message);process.exitCode=1}' "$L"
```
**P2 — relaunch proof**: rerun R1 right after the quit returns (same `$H`). After: `server-runtime.json` reappears with a new `ownerId`, the window loads, `server-child.log` shows no `already running`. Before (F3): the backend log shows `ServerAlreadyRunningError` / "already running", and `lsof $L` still lists the orphan PID.

### R4 — quit triggers
- Signal: `kill -TERM $ELECTRON_MAIN_PID` → `quitFromSignal` (POSIX only).
- UI: CDP `window.close()` on the page target from `curl -s localhost:9333/json` → `window-all-closed` → `app.quit` → `handleBeforeQuit`.
- Slow backend: `kill -STOP $BACKEND_PID` before the quit (F3). For F4, also add `(sleep 3; kill -CONT $BACKEND_PID)&`.
- Timing: `t0=$(date +%s.%N); kill -TERM $ELECTRON_MAIN_PID; while kill -0 $ELECTRON_MAIN_PID 2>/dev/null; do ps -o pid,pgid,stat,etime,cmd -g $BACKEND_PGID; sleep 0.5; done; t1=$(date +%s.%N)`.

### Expected After transcript for F3
`t≈0` backend stat `T`, Electron alive → `t≈5s` the group gets SIGKILL, the backend disappears → Electron exits at `t≈5-6s` → `lsof $L` empty → P1 `LOCK FREE` → P2 relaunch OK.
Before: Electron exits at `t≈5s` while the backend is still listed with stat `T`, ppid 1 → P1 `LOCK HELD` → P2 fails.

## Risks for the reviewer
1. After SIGKILL, `terminateProcessGroup` ends with an unbounded `Deferred.await(exitSignal)`. The PR removes the only remaining limit on quit, so a backend in uninterruptible sleep, or one that never reports exit, now blocks app exit forever. Before, the app exited after 5s and left an orphan. That trade-off is the intent, but it should be stated explicitly. A hard cap (for example, exit anyway after N more seconds with an error log) is worth considering.
2. The comment says the threshold is 7s, but F3's expected exit is about 5-6s; 7s only matters for the warning (F6).
3. **The Before result for F3 can be flaky.** On the Before side, the outer 5s timeout and the spawner's 5s `forceKillAfter` are the same length. The fork is lazy, so the outer timer starts slightly before SIGTERM goes out and usually wins. Electron's remaining quit work then has to finish before the spawner's SIGKILL at about 5s. If Electron is slow, SIGKILL can still land and Before looks fixed. Run F3 Before **at least 3 times** and report every run. Hold the backend's exit longer to remove the race: SIGSTOP plus a wrapper isn't possible, because SIGKILL always works. So use timestamps to show Electron's exit happened *before* the backend vanished. (Before: Electron gone while `ps` still shows the backend with stat `T` and ppid 1.)
4. Effect's spawner only signals the process group again when the *child* exits, not when the parent exits (`NodeChildProcessSpawner.ts:563-567`). So on Before, nothing kills an orphan left behind by Electron's exit. That's why the lock stays held.