---
id: "bug-1741"
status: "fixed"
status_updated: "2026-09-27"
status_note: "0cad6850: OPCode_notifyAcked -- job start/finish/fail acknowledged after commit, resent until acked; FF fallback for old controllers"
component: "lib"
severity: "unknown"
timestamp: "2026-08-31T23:45:00Z"
last_seen: "2026-09-19T09:00:00Z"
occurrences: 1
files: ["src/worker/src/controller_connector.c (xcecd_job_finished)", "src/lib/client.c (xcecli_notify", "xcecli_notify_jobfinish)", "src/lib/client_internal.h (XCECLI_FF_CALL)", "src/controller/job.c (on_job_finish", "never invoked for the 6 lost reports)", "src/controller/cluster.c (do_notify/handle_notify)"]
tags: ["controller", "worker", "job-lifecycle", "job-finish", "notify", "xcecli_notify", "XCECLI_FF_CALL", "fire-and-forget", "burst-load", "array-job", "phantom-run-job", "stuck-job", "data-loss", "found-live", "partially-root-caused", "needs-live-repro", "needs-owner", "related-to-bug-1548", "distinct-from-bug-1731", "distinct-from-bug-1698", "possibly-same-root-cause-as-bug-1698"]
related: ["bug-1548", "bug-1731", "bug-1737", "bug-1738", "bug-1739", "bug-1698"]
fix_commits: ["0cad6850"]
legacy: true
---

# bug-1741: User-reported live deployment (/tmp/xce-ha-deploy, 3-node HA cluster + xcecd_virtual worker…

## Description
User-reported live deployment (/tmp/xce-ha-deploy, 3-node HA cluster + xcecd_virtual worker, build.dbg): after submitting a 50-element array job (jobsub -a 50 -c hostname) immediately following a fresh primary election, `xce_test`'s `joblist` shows array elements [6] through [12] (7 consecutive elements) permanently stuck in state RUN (FinishTime=0) while every other element (0-5, 13-49) correctly reached DONE with real FinishTimes, including elements submitted/dispatched both before and after the stuck range and well after the cluster had fully stabilized (up to ~46s post-submit). The user confirmed via repeated `joblist` checks over several minutes that elements 6-12 never progress -- not a transient lag.

## Environment
_Not recorded (migrated from buglog.json)._

## Reproduction
_Not recorded (migrated from buglog.json)._

## Root cause
PARTIALLY ROOT-CAUSED via direct log correlation across worker, controller, and scheduler logs from the live incident (not yet independently live-reproduced in isolation this session -- found while investigating the unrelated bug-1698 HA-hang occurrence in the same incident's data, but happened ~19s BEFORE ctrl-1's hang began and is confirmed unrelated to it: ctrl-1 was fully healthy and primary at the time these 7 finish reports were lost). Timeline reconstructed from three independent logs for elements 6-12: xcecd_worker.log shows the xcecd_virtual master daemon spawned one subprocess per array element (its own established per-job-process architecture), each connecting to the master via a local unix-socket IPC (uv_ipc.c) and reporting 'Completed' for all 7 elements within a ~600ms window (16:25:00.943 to 16:25:01.535). ctrl-1.log shows dispatch_a_job_/on_job_start logged correctly for all 7 elements, but job.c's on_job_finish() (traced via 'job finish jobid ... exit code' TRACE, its own first log line) fired for element [6] ONLY -- elements [7] through [12]'s finish reports never reached on_job_finish at all, i.e. the loss is upstream of job.c entirely, not a hook-firing gap (distinct from bug-1731's fix, which addressed apply_committed_entry() firing HOOK_DETAIL_JOB_START/EXIT -- that fix only helps once on_job_finish/WAL_JOB_FINISH already happened; here it never happens for 6 of 7 jobs). scheduler.log's later repeated 'sched_add_job failed ... rc=-17, merging state into existing tracked job' WARNs for exactly elements 6-12 (recurring at 16:25:23, 16:25:41, 16:25:46 across multiple post-failover JOBS_PUSH resyncs from the new primary ctrl-2) confirm the loss is durable, not just delayed: ctrl-2 inherited ctrl-1's own WAL/replicated state, which never recorded a finish for these 6 elements either, so every resync just re-merges the same stale RUN state -- there is no surviving source of truth anywhere in the cluster that ever learned these jobs finished. Traced the worker-side send path for the likely drop mechanism: xcecd_job_finished() (controller_connector.c:1680) -> xcecli_notify_jobfinish() -> xcecli_notify() (client.c:610, explicitly commented 'Fire-and-forget chokepoint') -> XCECLI_FF_CALL (client_internal.h:290-302), whose own doc comment states plainly: 'single-shot, NOT a retry loop... evaluate EXPR exactly ONCE... A lost send rc is surfaced as-is, never retried.' All 7 near-simultaneous finish notifications for this burst go out over the SAME single xcecli_ctx/network connection (one xcecd master daemon process per worker host, matching this codebase's one-connection-per-daemon model) via this explicitly best-effort, non-retrying path -- exactly the kind of rapid same-connection burst most likely to hit whatever failure mode XCECLI_FF_CALL's own comment already acknowledges as a known risk ('A failed reconnect is non-fatal -- the send below is simply best-effort'). This is the SAME XCECLI_FF_CALL mechanism bug-1548 already flagged for a related-but-distinct reliability gap (silently self-healing the transport across ticks, masking a needed rejoin) -- this occurrence is a second, independent real-world manifestation of the same primitive's fire-and-forget design causing silent data loss, this time for job-finish reports specifically. NOT YET CONFIRMED: the exact internal reason xce_notify()'s single-shot send fails/drops 6 of 7 calls in this specific burst (whether it's a genuine transport error being silently discarded because nothing checks xcecd_job_finished()'s return value, some reentrancy/buffer-reuse issue in xce_notify()'s own encode/send path when called back-to-back without waiting for the prior call's completion, or contention on ac->failover.lock combined with a narrow race) -- would need either a live gdb/strace-instrumented repro of this exact burst pattern, or a close read of xce_notify()'s implementation and whether xcecd_job_finished()'s callers check/log its return code at all (not yet done this session). USER-PROMPTED REFINEMENT (2026-08-31, same session): the user hypothesized this might be a stale-primary artifact (worker kept reporting to the old primary and never resent to the new one). That exact framing doesn't fit the millisecond timing (these 7 finishes/sends happened at 16:25:00.9-01.5, before any failover was detected by anyone), but chasing it surfaced an important adjacent fact from the worker daemon's OWN controller-connection log (xcecd.log, root-owned -- required sudo to read, distinct from xcecd_worker.log's per-job-subprocess-only content used for the rest of this investigation): the WORKER's client-side failover logic logged its own first sign of trouble with ctrl-1 at 16:25:12.164 ('xcecli failover: attempt 1 -> localhost:9901') -- 8 SECONDS BEFORE ctrl-1 itself logged anything wrong ('standby[0] silent for 12s ... evicting', 16:25:20 per bug-1698's own timeline), and about 11-19 seconds AFTER the lost job-finish notifies for elements 6-12 were sent (16:25:00.943-01.535). The worker successfully failed over to the new primary at 16:25:22.332 ('rejoin after transparent reconnect succeeded', following attempt 2 -> localhost:9902 at 16:25:22.165). This reframes the likely relationship between this bug and bug-1698: rather than two unrelated defects that happened to occur in the same incident, the timing is consistent with a SINGLE underlying trigger -- ctrl-1's own I/O/connection handling apparently began degrading measurably before ctrl-1 itself noticed (silently failing to process/ack the worker's job-finish sends circa 16:25:00-01, then dropping the worker's connection entirely by ~16:25:12) -- with ctrl-1's own self-diagnosis (the 12s standby-liveness timeout) simply being the slowest of the three symptoms to surface. If that shared-trigger theory holds, this bug's fix should NOT rely solely on 'the worker eventually fails over correctly' (bug-1698's territory, and confirmed here to work correctly once detected, ~10s after the fact) -- XCECLI_FF_CALL's fire-and-forget send has zero resend/replay mechanism, so a notify lost during exactly this kind of covert pre-failover degradation window is gone forever regardless of how well the subsequent failover behaves. Still not proven whether ctrl-1's degradation genuinely started as early as 16:25:00 (matching the lost notifies) or only at 16:25:12 (matching the worker's own first detected symptom) with the 16:25:00-01 notify loss being a coincidentally-timed SEPARATE glitch -- distinguishing these needs the same live gdb/strace capture bug-1698's own next-steps already call for, ideally instrumented to also watch ctrl-1's handling of inbound OPCode_notify requests specifically, not just its outbound peer reconnect attempts.

## Fix
NOT YET FIXED -- found and root-caused-to-this-depth this session while investigating bug-1698 in the same incident's log data; not yet independently reproduced or fixed. Next steps: (1) confirm whether xcecd_job_finished()'s callers (worker_job_callback / process_exit_callback path in xcecd_worker.c) check and log its return code -- if not, that alone is worth fixing regardless of the deeper cause, so a real send failure is at least loud instead of silent; (2) read xce_notify()'s actual implementation (xceapi.c, shared with every other XCECLI_FF_CALL user) to determine whether it is safe to call back-to-back in a tight loop on the same api_ctx_t without waiting for a prior call's underlying I/O to fully complete, or whether it has an internal single-buffer/reentrancy hazard; (3) attempt a targeted live repro (a single worker host completing 5+ jobs within under a second, e.g. a small array of near-instant jobs against one host with concurrency >= the array size) with the worker daemon under gdb/strace to catch xce_notify()'s return code and underlying syscall result for the calls that get lost; (4) once root-caused, likely fix directions include making job-finish notifications durable/retried (not fire-and-forget) given their consequence (a permanently stuck job holding worker slots/resources forever is a correctness issue, not just a display one -- unlike heartbeats, a job-finish report has no self-healing follow-up message the way bug-1737's DONE-vs-EXIT divergence did), or serializing/queuing bursts of notify calls from the same worker daemon so they cannot race or overwrite each other on the shared connection.

## Verification
_Not recorded (migrated from buglog.json)._

## References
- Related: bug-1548, bug-1731, bug-1737, bug-1738, bug-1739, bug-1698