# #22830 (GitCafe #22829) runtime evidence for #15250: native Codex child model lookup retries **Environment:** - Machine `fs-24`, Linux 6.1.102 x86_64, node v26.12.1. - BEFORE is `8d3b061121` (merge-base, detached worktree). AFTER is `aa57d7d3ba` (the fix). The evidence tooling was committed on top as `89d72c1ff2`, and it changes no server code. - **Codex is a STUBBED Codex app-server** (`apps/server/scripts/evidence/issue-15250/stub-codex-app-server.mjs`), not real Codex. - Real `t3 start` servers ran with isolated `--base-dir` homes under `/tmp/ev-22830/raw/{before,after}/home`, on 127.0.0.1:43832 (before) and 43833 (after). **How it works:** - Each server's `providerInstances.codex.config` = `{setupMode:"existing", binaryPath:}`. - Each row is one `t3 thread start . "fixture= …" --provider codex --model gpt-5.4`. - For each thread, the stub emits `turn/started`, then `item/completed{subAgentActivity kind:"started", agentThreadId:child--}`. That registers a null-model child and starts the lookup. - The stub scripts `thread/read` and `thread/resume` replies per attempt, and timestamps every frame it sees. - The parent turn is held open for 80 s, then completes normally. - After the last thread starts, the driver watches for 100 s. It then reads `orchestration_v2_projection_subagents.payload_json` (`$.model`, `$.modelSelection`) and the `subagent.updated` rows of `orchestration_events`. Both come from the isolated `statev2.sqlite`, opened **read-only** (node:sqlite `readOnly`). - Client value is that projection row: the WS snapshot the web and mobile clients render through `resolveSubagentMetadata`, where null shows as "Not reported". It was read from the DB, not over a WS socket. ## Matrix Times are seconds after the child was registered (the stub's `subAgentActivity started` frame). R = `thread/read`, S = `thread/resume`. | Row | Fixture | Before (8d3b061121) | After (aa57d7d3ba) | Client value, before → after | |---|---|---|---|---| | **R1** first lookup OK | `first-ok` | 1R 0S (R@0.04) | 1R 0S (R@0.04) | `gpt-6-luna`, `options=[reasoningEffort:high]` → same | | **R2a** attempt 0 R+S fail, retry OK (**the bug**) | `retry-ok` | 1R 1S (R@0.01 S@0.02), then nothing | 2R 1S (R@0.00 S@0.04 **R@1.04 ok**) | **`null` ("Not reported")** → **`gpt-6-luna`** `{instanceId:codex, model:gpt-6-luna, options:[reasoningEffort:high]}`, written at +1.01 s | | **R3** always failing | `always-fail` | 1R 1S, then quiet for ~99 s | **6R 6S** at 0.01/0.04, 1.04/1.04, 3.04/3.04, 7.05/7.05, 15.05/15.05, 31.05/31.05. Then **quiet for 68 s** (watched to about +99 s; the parent turn completed at +79.9 s with no new lookups) | `null` → `null`. One `subagent.updated` (registration) at both SHAs | | **R4a** child interrupted after attempt 0 | `interrupted` | 1R 1S; interrupt at +0.23 | 1R 1S; interrupt at +0.25; **no attempt at +1 s or later** (compare R3 after, which retries at +1.04) | status `interrupted`, model `null`, at both SHAs | | **R5** `model/rerouted` mid-retry | `rerouted` | 1R 1S; reroute at +0.23 | 1R 1S; reroute at +0.24; **no further reads** | `gpt-5.6-sol` at both SHAs. Exactly one model-bearing `subagent.updated` (2 total: registration with null, then reroute) | | R6 read empty, resume OK | `read-empty-resume-ok` | 1R 1S, model reported | 1R 1S, model reported | `gpt-6-luna` → `gpt-6-luna` | | R3b read always hangs | `always-hang` | 1R (hang), 0S: the 5 s chain timeout ends the lookup | **6R, 0S** at 0.00, 6.01, 13.01, 22.01, 35.02, 56.02 (each 5 s timeout plus 2^(k-1) s sleep) | `null` → `null`. As the audit noted, there is no resume fallback when the read hangs | Notes: - R4a and R5 stop after 1R+1S at both SHAs. BEFORE stops anyway, since it never retries. The AFTER evidence is the absence of the +1.04 s retry that R3 shows under the same build and timing. - The projection statuses above were snapshotted while the servers were running. The `projection_subagents.txt` dumps in the raw tarball were taken after shutdown, so they show `cancelled` for still-running children. That is session teardown, not part of this change. - Recheck: after lint and format fixes, the driver was re-run on AFTER for R2a and R3 (`raw/after-recheck.driver.log`). It gave R2a 2R/1S → `gpt-6-luna` and R3 6R/6S → `null` again. ## Transcript excerpts (stub request log, `dir reply` = what the stub answered) R2a before: ``` + 0.00s out item/completed parent-1654117-2 started + 0.01s reply thread/read child-retry-ok-1654117 err:rollout is empty + 0.02s reply thread/resume child-retry-ok-1654117 err:temporary worktree fault + 79.90s out turn/completed parent-1654117-2 projection: running model=None modelSelection=None ``` R2a after: ``` + 0.00s reply thread/read child-retry-ok-1654203 err:rollout is empty + 0.04s reply thread/resume child-retry-ok-1654203 err:temporary worktree fault + 1.04s reply thread/read child-retry-ok-1654203 ok subagent.updated 11:14:22.815 running model=null subagent.updated 11:14:23.826 running model=gpt-6-luna ``` R3 after: ``` + 0.01s R err / +0.04s S err + 1.04s R err / +1.04s S err + 3.04s R err / +3.04s S err + 7.05s R err / +7.05s S err + 15.05s R err / +15.05s S err + 31.05s R err / +31.05s S err + 79.90s out turn/completed parent-1654203-7 (no child requests after 31.05; DB read at ~+99s) ``` R4a after: ``` + 0.00s reply thread/read err +0.05s reply thread/resume err + 0.25s out item/completed subAgentActivity interrupted (no further child requests; status=interrupted, model=null) ``` R5 after: ``` + 0.01s reply thread/read err +0.03s reply thread/resume err + 0.24s out model/rerouted child-rerouted-1654203 gpt-5.6-sol subagent.updated 11:14:34.182 running model=null subagent.updated 11:14:34.395 running model=gpt-5.6-sol (no further reads) ``` R3b after: `thread/read` hangs at +0.00, 6.01, 13.01, 22.01, 35.02 and 56.02, with no `thread/resume`. ## What else changed between before and after `git diff --stat 8d3b061121 89d72c1ff2`: ``` apps/server/scripts/evidence/issue-15250/README.md | 3 + apps/server/scripts/evidence/issue-15250/run.mjs | 218 +++ .../evidence/issue-15250/stub-codex-app-server.mjs | 303 +++ .../Adapters/CodexAdapterV2.test.ts | 127 +++ .../orchestration-v2/Adapters/CodexAdapterV2.ts | 109 +++--- ``` That is only the adapter, its test and these evidence files. `aa57d7d3ba` is a merge of `origin/main` whose tree differs from the merge-base only in those two adapter files. The evidence scripts are loaded from the PR worktree for both runs, so both SHAs ran the identical driver and stub. ## Exact commands ```sh export PATH="$HOME/.bun/bin:$HOME/.local/bin:$HOME/.local/share/vite-plus/bin:$PATH" git -C /home/ubuntu/t3code worktree add --detach /tmp/ev-22830/before 8d3b061121 (cd /tmp/ev-22830/before && pnpm install --frozen-lockfile) S=/home/ubuntu/wt/pr-22830/apps/server/scripts/evidence/issue-15250/run.mjs node $S --checkout /tmp/ev-22830/before --out /tmp/ev-22830/raw/before --port 43832 > raw/before.driver.log 2>&1 & node $S --checkout /home/ubuntu/wt/pr-22830 --out /tmp/ev-22830/raw/after --port 43833 > raw/after.driver.log 2>&1 & # recheck after lint/format edits: node $S --checkout /home/ubuntu/wt/pr-22830 --out /tmp/ev-22830/dry --port 43831 --rows R2a,R3 --window-ms 35000 vp lint apps/server/scripts/evidence/issue-15250/ # clean ``` The driver spawns `node apps/server/src/bin.ts start --base-dir /home --no-browser --host 127.0.0.1 --port

--auto-bootstrap-project-from-cwd` and records the PID. Rows run through `bin.ts thread start . "fixture=…" --provider codex --model gpt-5.4 --json`. At the end it SIGTERMs only that PID. Every server I started exited (before pid 1652425, after pid 1652417, recheck pid 1660206, plus two dry runs). ## Rows not captured - **R2b, R2c, R4b–R4e, R6b, R7.** These were not run in the 75-minute budget; none were required. The stub's fixture table can add each one in a few lines. - **R4f and R5b** (in-flight races). These were not run. The audit flags them as timing-sensitive; R5b is covered by the existing unit test. - **No web or mobile screenshot of "Not reported".** No browser was authorised. The client value comes from the projection row the client renders.