myxce/.wolf/bugs/bug-1679.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	legacy
bug-1679	fixed	2026-09-24		controller	unknown	2026-08-25T03:15:00Z	2026-09-19T09:00:00Z	1	src/controller/job.c (dispatch_a_job_ on_job_start and likely other direct-mutation call sites) src/controller/persist.c (apply_committed_entry's WAL_JOB_START/.../DISPATCH shared case apply_job_update_transition)	controller job.c persist.c apiserver-notification hook-suppression scheduler-dispatch-commit live-status data-loss high-severity discovered-via-bug-1678	bug-1678 bug-1680	true

bug-1679: A job's live dispatch/start/finish transitions are never pushed to the apiserver -- xcecli jobs…

Description

A job's live dispatch/start/finish transitions are never pushed to the apiserver -- xcecli jobs list shows a job stuck at PEND forever (or, in one observed case, disappearing from the default query entirely) even though the controller's own WAL and in-memory state correctly progress the job all the way through RUN to EXIT/DONE. Discovered as a side effect while live-verifying bug-1678 (restart-recovery of job.allocs) -- this is a SEPARATE, unrelated, more severe bug: it reproduces with NO restart involved at all, on a single, freshly-submitted job, every time. Confirmed reproducible on the STOCK branch (git-stashed all of bug-1678's changes, rebuilt, redeployed fresh) -- NOT a regression introduced by bug-1678 or any other fix landed this session (bug-1673/1674/1675/1676/1677 touch only Go/apiserver code, a Python script, or job_submit_data_t's from_host field + jobj.c's dedup identity check -- none touch the state-transition/hook-firing logic implicated here).

Environment

Not recorded (migrated from buglog.json).

Reproduction

Not recorded (migrated from buglog.json).

Root cause

dispatch_a_job_() (job.c, scheduler-dispatch-commit's task 3.1 restructuring) mutates j->job.state DIRECTLY in memory (job.c ~1355, j->job.state = JOB_STAT_RUN;) BEFORE calling persist_submit_job_dispatch_fact()/persist_submit_job_fact() to durably record the transition. On a live, single-node, self-committing controller, the commit-then-apply pipeline runs synchronously enough that apply_committed_entry()'s own WAL_JOB_DISPATCH handling (via apply_job_update_transition(j, JOB_STAT_RUN)) sees cur == state == JOB_STAT_RUN -- because the state was ALREADY set moments earlier by the same call, under the same lock -- and returns JOB_TRANSITION_NOOP_DUPLICATE. The shared case body's if (result != JOB_TRANSITION_APPLIED) return 0; /* rejected-backward / no-op-duplicate -- no hook */ then unconditionally skips xce_run_hooks(HOOK_AFTER_EVENT, HOOK_DETAIL_JOB_UPDATE, ...) -- the ONLY mechanism (via apiserver_job_hook_fun/apiserver_push_joblist in apiserver.c, registered against XCE_CONTROLLER_HOOK_JOB) that pushes an updated job record to the apiserver. Live-confirmed via controller log: apply_job_update_transition: job <id>: duplicate UPDATE to state 0x2 -- no-op fires for BOTH the WAL_JOB_DISPATCH commit AND the subsequent WAL_JOB_START commit (on_job_start(), job.c) on the exact same job, back to back -- meaning the job NEVER gets a single HOOK_DETAIL_JOB_UPDATE fire through its entire dispatch. apiserver_push_joblist() itself only ever fires ONCE per job, at creation time (apiserver_job_new_fun, subscribed to HOOK_DETAIL_JOB_NEW, itself fired once by apply_committed_entry's WAL_JOB_SUBMIT case) -- there is no periodic/polling full-refresh fallback, confirmed by reading apiserver_push_joblist's only call site. So a job's apiserver-visible record is permanently frozen at whatever it looked like at creation (JOB_STAT_PEND, no exec_host) for its entire lifetime, regardless of what actually happens to it afterward, UNLESS some other code path elsewhere in job.c legitimately fires the hook directly (mutate-then-notify, like mark_job_update()/persist_job_hook_fun's 'still-legitimate append' pattern) rather than relying on the apply-time hook-fire -- not confirmed either way for on_job_exit/mark_job_fail/other terminal-transition call sites; only dispatch_a_job_() and on_job_start were directly traced. This is very likely a structural gap introduced by (or newly exposed by) the raft-apply-decoupling / scheduler-dispatch-commit redesigns moving job-state mutation to happen synchronously ahead of persistence for several call sites, while the apiserver-notification hook-fire mechanism was designed under an assumption (state hasn't already changed by the time apply runs) that held for the OLD architecture but no longer holds for these newer, direct-mutation call sites.

Fix

FIXED (2026-08-25). Root cause confirmed exactly as filed: dispatch_a_job_() (job.c) mutates job.state/job.allocs directly then persists, firing no hooks at all -- apply_committed_entry()'s later apply of that same WAL_JOB_DISPATCH entry sees state already matches and skips its would-be notify (JOB_TRANSITION_NOOP_DUPLICATE).

New src/controller/persist.c function persist_notify_job_updated_already_persisted(cluster_t*, job_t*) (declared in src/include/persist.h), called from dispatch_a_job_()'s success branch immediately after unlock(j). Internally guards with the exact same t_apply_seam_notifying flag apply_committed_entry()'s own shared WAL_JOB_START/.../DISPATCH case already uses around its identical hook fire (same guard, same hook -- HOOK_DETAIL_JOB_UPDATE's AFTER_EVENT -- same detail) -- without it, persist_job_hook_fun (also subscribed to this hook) would redundantly re-append the same transition as a second, duplicate WAL_JOB_UPDATE entry. The flag stays file-private to persist.c; job.c reaches it only through this one new function. dispatch_a_job_()'s own mutation/persistence logic is otherwise untouched.

TDD: new test_dc_success_notifies_job_update_hook_exactly_once (test_dispatch_commit.c) -- a spy hook counting HOOK_AFTER_EVENT/HOOK_DETAIL_JOB_UPDATE fires specifically. RED->GREEN independently reproduced: temporarily disabled the new call (production code unchanged otherwise), rebuilt, confirmed FAIL (Expected 1 Was 0); restored, rebuilt, confirmed PASS. Verified the two existing tests that would catch a broken guard (double WAL append): test_jobdispatch_receipt_ack_no_wal_append (asserts exactly one WAL entry for a successful dispatch) and test_handle_jobdispatch_matches_dispatch_a_job_notify_count -- both pass unchanged, and the former's fixture genuinely wires persist.cluster with persist_job_hook_fun registered, so it actually exercises the guard. Full controller_unit_test: 1137/1137 (2 pre-existing test_config_repl.c failures -- the known overlayfs fdatasync errno=5 flake, bug-1561 -- confirmed identical via git-stash on stock code, not a regression).

Live-verified on a fresh single-node deploy (rebuilt controller/scheduler/worker/xcecli): submitted a job, xcecli jobs list showed PEND at t+1s then RUN with EXEC_HOST=1worker from t+2s onward continuously -- no restart involved, matching the exact acceptance criterion. Also confirmed record_job_finish_fact's existing notification path needed no fix: on_job_finish fired correctly and xcecli jobs list -d showed DONE with EXEC_HOST intact for both test jobs, exactly as the root-cause analysis predicted (only dispatch_a_job_()'s broken hybrid pattern was ever the problem -- the durable-append-only pattern record_job_start_fact/record_job_finish_fact use was already correct).

Verification

Not recorded (migrated from buglog.json).

References

• Related: bug-1678, bug-1680