chore(wolf): mark bug-1678 fixed; log bug-1679 (live job-status pushes never fire, discovered via bug-1678 verification)

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AgYRU3zDTqNGV5Jk8aiKLg

.wolf/buglog.json

@@ -18555,7 +18555,7 @@
       "error_message": "Gitea issue #43 (\"\u91cd\u542fdaemon\u540e\u4f5c\u4e1a\u8c03\u5ea6/\u4fe1\u606f\u4e0d\u5bf9\"): after restarting ALL daemons (controller, scheduler, worker, apiserver), previously-RUN jobs come back with STATUS=RUN correctly recovered but EXEC_HOST permanently blank (\"-\"), even though the underlying job is genuinely still running. CONFIRMED LIVE on branch scheduler-framework, corrected finding: an earlier verification pass on this same issue (2026-08-24, same session) restarted only controller+scheduler+worker and left the apiserver running, and did NOT reproduce the symptom -- that was a false negative, caused by the still-running apiserver simply continuing to display its stale-but-correct in-memory job cache from before the restart, never refreshed. Repeating the test with the apiserver ALSO restarted (matching the original report's \"restart all daemons\" exactly) reproduces the bug immediately and durably: 3 RUN jobs with EXEC_HOST=1*worker-1 before the restart; killed all 4 daemons; ran start-all.sh; EXEC_HOST was blank on every poll (15 polls over 30s, first poll ~6s after apiserver came back up) while STATUS stayed correctly RUN throughout. Controller log corroborates the mechanism: ctrl-1.log's recovery trace shows apply_job_update_transition logging 'duplicate UPDATE to state 0x2 -- no-op' for every job during WAL replay -- consistent with the WAL_JOB_DISPATCH fact applying SILENTLY first (PEND->RUN, no log line on the APPLIED path), immediately followed by each job's WAL_JOB_START fact (worker-confirmed start, also state=RUN) correctly detecting the now-redundant transition and logging the no-op. Neither replay step ever touches job.allocs.",
       "file": "src/controller/job.c:1369-1370, src/controller/persist.c:3708-3757, src/controller/job.c:288-351 (restore_job), src/controller/persist.c:6690-6713 (persist_submit_job_fact)",
       "root_cause": "dispatch_a_job_() (job.c:1343-1372) sets j->job.allocs directly in memory (lines 1346-1353, the host/resource assignment the scheduler decided) and transitions j->job.state = JOB_STAT_RUN (line 1355), but the durable record of this transition -- persist_submit_job_fact(&c->persist, WAL_JOB_DISPATCH, alloc->jobid, JOB_STAT_RUN) at line 1369-1370 -- only carries {jobid, state} (persist_submit_job_fact()'s own signature, persist.c:6690-6694, has no allocation parameter at all; its payload type wal_jobstate_t is exactly {jobid, state}). The resource allocation itself is never written to the WAL anywhere. On a full-process restart, job objects are reconstructed purely from replay: restore_job() (job.c:288-351) decodes a job's ONLY WAL_JOB_SUBMIT entry (written at initial submission, strictly before dispatch -- confirmed via persist_job_hook_fun's own comment that WAL_JOB_SUBMIT durability is written once, before any job object exists -- so its capnp payload's allocs field is genuinely empty/NULL at encode time). apply_committed_entry()'s WAL_JOB_DISPATCH case (persist.c:3708-3757) then replays the dispatch fact via apply_job_update_transition(), which by explicit design and its own doc comment 'only ever touches state' -- job->job.allocs is never set anywhere in the replay path. Result: after restart, job.state is correctly RUN (both WAL_JOB_DISPATCH and the later WAL_JOB_START fact carry state=RUN and replay correctly) but job.allocs is permanently NULL for the remaining life of that job -- nothing ever re-derives it, since scheduler-dispatch-commit's design treats a job's dispatch as a one-time PEND->RUN event with no re-dispatch/re-reconciliation path for an already-RUN job (see bug-1631's confirmation that no live caller ever re-adds a RUN job back to the scheduler's dispatch pool). EXEC_HOST is rendered by the apiserver from this now-NULL allocation once the recovered, allocation-less job is pushed to it -- which is why restarting the apiserver alongside the controller (forcing a fresh push) reveals the bug, while leaving the apiserver running (as an earlier verification pass on this same issue did) masks it behind stale cached data.",
-      "fix": "NOT FIXED -- reproduced live and root-caused during a Gitea-issue verification pass; supersedes this same session's earlier NOT-REPRODUCED finding for issue #43 (that finding's test methodology had a gap: it never restarted the apiserver, so it could not observe the controller's actual post-recovery state, only the apiserver's stale pre-restart cache). Fix direction (per user 2026-08-24 review of this exact finding): the WAL_JOB_DISPATCH fact needs to carry the resource allocation the scheduler committed at dispatch time, not just {jobid, state} -- e.g. a richer payload (encode j->job.allocs via a capnp Job-alloc-shaped message, or reuse capnp_encode_job's existing xce_job_t encoder scoped to just the allocs field) written alongside or instead of the current wal_jobstate_t shape, plus a corresponding decode-and-apply step in apply_committed_entry()'s WAL_JOB_DISPATCH case (currently routed through the shared apply_job_update_transition() core that intentionally only touches state -- this fact type would need its own richer apply path, mirroring how WAL_JOB_FAIL/WAL_JOB_UPDATE already get dedicated apply-time-recomputed handling instead of joining the shared {jobid,state}-only group). Needs a design pass: touches the scheduler-dispatch-commit spec's own WAL_JOB_DISPATCH wire format (wal_types.h:117), which is implementation-complete and already has test coverage assuming the current {jobid,state}-only shape.",
+      "fix": "FIXED (2026-08-25). No capnp schema change needed -- JobAlloc (xce_job_alloc_t) was already a full top-level capnp struct (the existing Job.allocs field) with generated encode/decode functions already present in the build. New glue only: src/controller/persist.c gains capnp_encode_joballoc()/decode_joballoc() (mirroring the existing capnp_encode_worker()/restore_job() patterns) and persist_submit_job_dispatch_fact(persist_t*, jobid, state, alloc) -- WAL_JOB_DISPATCH's payload is now a wal_jobstate_t header plus trailing capnp-encoded JobAlloc bytes, declared in src/include/persist.h. src/controller/job.c: dispatch_a_job_()'s success path now calls persist_submit_job_dispatch_fact(..., j->job.allocs) instead of persist_submit_job_fact(...), reading j->job.allocs (the freshly-constructed canonical struct) under the same write lock. Both apply seams updated: apply_committed_entry() (primary's own boot replay) gets one added else-if branch inside its existing WAL_JOB_START/WAL_JOB_FINISH stamp-chain (minimal diff, same shared plumbing) that decodes and adopts trailing allocation bytes when present. apply_wal_entry() (standby catch-up / replicated-apply -- also required, since a promoted standby would hit the exact same bug) has WAL_JOB_DISPATCH split into its own case, calling new restore_job_dispatch_state() (job.c, restore_job_state()'s sibling, takes ownership of an optional decoded allocation). A payload of exactly sizeof(wal_jobstate_t) (any WAL_JOB_DISPATCH entry written before this fix) decodes no allocation and behaves exactly as before -- full backward compatibility, verified by the 3 pre-existing test_wal_job_dispatch_fact.c tests passing completely unchanged.\n\nPrimary regression test: test_dispatch_commit.c's test_dc_success_appends_wal_job_dispatch_durably (already drove the REAL dispatch_a_job_() production path through a real WAL-write + persist_recover() restart cycle, previously only checking j2->job.state) extended to also assert j2->job.allocs matches what was dispatched (host, resource name, value) -- the user's exact acceptance criterion, 'after recover the job should get the same as it's state before shutdown.' RED->GREEN independently reproduced (not just narrated): git-stashed job.c/persist.c/persist.h with the test extension kept, rebuilt, confirmed FAIL at exactly this assertion; restored, rebuilt, confirmed PASS. Added 3 new tests to test_wal_job_dispatch_fact.c covering the richer payload directly on both apply seams plus a full restart-replay variant -- all pass. Full controller_unit_test suite independently re-run: 1136/1136 pass, zero regressions.\n\nLive-verified at the WAL level via xce_test's waldump: a real dispatch_a_job_() call produced a WAL_JOB_DISPATCH entry with PLEN=176 (vs the old bare wal_jobstate_t shape's PLEN=16), confirming the richer payload is genuinely written on the production path.\n\nIMPORTANT: full xcecli-visible E2E verification (submit -> restart -> check EXEC_HOST live) was blocked by discovering bug-1679 (see that entry) -- a separate, more severe, pre-existing bug where live job-status pushes to the apiserver never fire at all, restart or not. This fix's correctness rests on the unit-test evidence above (which inspects the controller's actual recovered in-memory state directly, with no dependency on the broken apiserver-notification path) and the direct WAL-payload inspection, not on an xcecli-visible live check.",
       "tags": [
         "gitea-issue-43",
         "controller",
@@ -18570,11 +18570,37 @@
         "wal-jobdispatch"
       ],
       "related_bugs": [
-        "bug-1631"
+        "bug-1631",
+        "bug-1679"
       ],
       "occurrences": 1,
-      "last_seen": "2026-08-24T13:55:00Z",
+      "last_seen": "2026-08-25T03:15:00Z",
       "gitea_issue": 43
+    },
+    {
+      "id": "bug-1679",
+      "timestamp": "2026-08-25T03:15:00Z",
+      "error_message": "A job's live dispatch/start/finish transitions are never pushed to the apiserver -- xcecli jobs list shows a job stuck at PEND forever (or, in one observed case, disappearing from the default query entirely) even though the controller's own WAL and in-memory state correctly progress the job all the way through RUN to EXIT/DONE. Discovered as a side effect while live-verifying bug-1678 (restart-recovery of job.allocs) -- this is a SEPARATE, unrelated, more severe bug: it reproduces with NO restart involved at all, on a single, freshly-submitted job, every time. Confirmed reproducible on the STOCK branch (git-stashed all of bug-1678's changes, rebuilt, redeployed fresh) -- NOT a regression introduced by bug-1678 or any other fix landed this session (bug-1673/1674/1675/1676/1677 touch only Go/apiserver code, a Python script, or job_submit_data_t's from_host field + jobj.c's dedup identity check -- none touch the state-transition/hook-firing logic implicated here).",
+      "file": "src/controller/job.c (dispatch_a_job_, on_job_start, and likely other direct-mutation call sites), src/controller/persist.c (apply_committed_entry's WAL_JOB_START/.../DISPATCH shared case, apply_job_update_transition)",
+      "root_cause": "dispatch_a_job_() (job.c, scheduler-dispatch-commit's task 3.1 restructuring) mutates j->job.state DIRECTLY in memory (job.c ~1355, `j->job.state = JOB_STAT_RUN;`) BEFORE calling persist_submit_job_dispatch_fact()/persist_submit_job_fact() to durably record the transition. On a live, single-node, self-committing controller, the commit-then-apply pipeline runs synchronously enough that apply_committed_entry()'s own WAL_JOB_DISPATCH handling (via apply_job_update_transition(j, JOB_STAT_RUN)) sees `cur == state == JOB_STAT_RUN` -- because the state was ALREADY set moments earlier by the same call, under the same lock -- and returns JOB_TRANSITION_NOOP_DUPLICATE. The shared case body's `if (result != JOB_TRANSITION_APPLIED) return 0; /* rejected-backward / no-op-duplicate -- no hook */` then unconditionally skips xce_run_hooks(HOOK_AFTER_EVENT, HOOK_DETAIL_JOB_UPDATE, ...) -- the ONLY mechanism (via apiserver_job_hook_fun/apiserver_push_joblist in apiserver.c, registered against XCE_CONTROLLER_HOOK_JOB) that pushes an updated job record to the apiserver. Live-confirmed via controller log: `apply_job_update_transition: job <id>: duplicate UPDATE to state 0x2 -- no-op` fires for BOTH the WAL_JOB_DISPATCH commit AND the subsequent WAL_JOB_START commit (on_job_start(), job.c) on the exact same job, back to back -- meaning the job NEVER gets a single HOOK_DETAIL_JOB_UPDATE fire through its entire dispatch. apiserver_push_joblist() itself only ever fires ONCE per job, at creation time (apiserver_job_new_fun, subscribed to HOOK_DETAIL_JOB_NEW, itself fired once by apply_committed_entry's WAL_JOB_SUBMIT case) -- there is no periodic/polling full-refresh fallback, confirmed by reading apiserver_push_joblist's only call site. So a job's apiserver-visible record is permanently frozen at whatever it looked like at creation (JOB_STAT_PEND, no exec_host) for its entire lifetime, regardless of what actually happens to it afterward, UNLESS some other code path elsewhere in job.c legitimately fires the hook directly (mutate-then-notify, like mark_job_update()/persist_job_hook_fun's 'still-legitimate append' pattern) rather than relying on the apply-time hook-fire -- not confirmed either way for on_job_exit/mark_job_fail/other terminal-transition call sites; only dispatch_a_job_() and on_job_start were directly traced. This is very likely a structural gap introduced by (or newly exposed by) the raft-apply-decoupling / scheduler-dispatch-commit redesigns moving job-state mutation to happen synchronously ahead of persistence for several call sites, while the apiserver-notification hook-fire mechanism was designed under an assumption (state hasn't already changed by the time apply runs) that held for the OLD architecture but no longer holds for these newer, direct-mutation call sites.",
+      "fix": "NOT FIXED -- discovered and root-caused as a side effect of bug-1678 verification, out of that fix's scope (bug-1678 is specifically about restart/replay recovery of job.allocs, verified independently via unit tests that inspect controller-internal state directly, bypassing this broken notification path entirely). Needs its own investigation: (1) confirm the full list of live call sites with this same mutate-then-persist-then-rely-on-apply-hook shape (dispatch_a_job_, on_job_start confirmed; on_job_exit/mark_job_fail/others not yet checked), (2) decide the fix architecture -- either make each such call site fire HOOK_DETAIL_JOB_UPDATE directly/synchronously itself (mirroring mark_job_update()'s existing mutate-then-notify pattern, which the persist_job_hook_fun's Hazard-1 dual-origin guard already anticipates and handles), or change apply_committed_entry's shared case to fire the hook even on a same-value transition specifically for these WAL types (riskier -- could reintroduce duplicate-notification issues the NOOP_DUPLICATE check was presumably added to prevent elsewhere), (3) full regression + live E2E verification that a freshly-submitted job's status visibly progresses through xcecli jobs list without any restart involved.",
+      "tags": [
+        "controller",
+        "job.c",
+        "persist.c",
+        "apiserver-notification",
+        "hook-suppression",
+        "scheduler-dispatch-commit",
+        "live-status",
+        "data-loss",
+        "high-severity",
+        "discovered-via-bug-1678"
+      ],
+      "related_bugs": [
+        "bug-1678"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-08-25T03:15:00Z"
     }
   ]
 }