test(scheduler-dispatch-commit): terminal dispatch failure e2e coverage (task 5.3)

Adds test_handle_jobdispatch_terminal_no_notify_no_leak to
test_jobdispatch_no_double_notify.c, closing a gap neither existing
test fully covered: test_dispatch_fail_terminal.c (task 3.2) bypasses
handle_jobdispatch() entirely via dispatch_a_job_test(), so it can't
exercise the double-notify race at all; test_jobdispatch_no_double_notify.c
(task 4.1) only proves an aggregate 3-job-batch notify count, not a
direct per-job zero-notify proof.

The new test drives a single terminal-classified job through the real
handle_jobdispatch() entry point and asserts, in one run: JOB_STAT_EXIT
with allocs left NULL, a direct (non-aggregate) zero-count read from
the notify-capture spy proving no jobDispatchFail was produced for this
job, and (via task 3.2's canary-reference technique, armed before the
call under test) no reference over-release.

Test-only; job.c/handler.c are untouched. RED phase verified via an
actual revert-rebuild-rerun of handle_jobdispatch() to its pre-task-4.1
shape. Independently reviewed and approved -- the reviewer notably
declined to trust a relayed build report and re-ran verification
first-hand before approving.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN

.kiro/specs/scheduler-dispatch-commit/tasks.md

@@ -83,7 +83,7 @@
   - _Requirements: 3.1, 3.2, 4.1, 4.3, 4.4, 4.5_
   - _Depends: 2.2, 3.3_
 
-- [ ] 5.3 End-to-end test: terminal dispatch failure resolves without duplicate reporting or leaked references
+- [x] 5.3 End-to-end test: terminal dispatch failure resolves without duplicate reporting or leaked references
   - Simulate a locally-invalid dispatch decision and confirm the job resolves to permanently failed, that no retry-eligible report is also produced for the same failure, and that the job record's reference count returns to its pre-attempt value
   - Observable completion: the assertions above all pass in the same test run
   - _Requirements: 4.2, 4.5_
@@ -118,3 +118,4 @@
 - Task 5.1: no controller-side test was added for this task — Requirements 2.1/2.3 (controller transitions to RUN independent of confirmation) are already covered by task 3.1's `test_dc_success_commits_alloc_state_and_resets_counter`. No live scheduler+controller wire-protocol integration harness exists anywhere in this codebase; building one was judged disproportionate to what this task verifies, and the reviewer agreed.
 - Task 5.2: **bug-1631 (logged, NOT fixed, deliberately deferred)** — `dispatch_a_job_()`'s four retryable-failure branches (job.c, tasks 3.1/3.3) never revert `job.state`/`job.allocs` when a retryable failure occurs while the job's controller-side state is CURRENTLY RUN (they were only ever designed for a job whose every dispatch attempt starts from PEND, which is all that's reachable today). A kiro-debug investigation (HIGH confidence), independently re-verified even more exhaustively by this task's own reviewer (full trace of every `pendJobs`-reinsertion and every controller-side `job.state`-mutation call site), confirmed this scenario is genuinely UNREACHABLE via any real caller in the current, fully-implemented system: the scheduler's picker permanently removes a job from its PEND candidate pool the instant it dispatches it (task 2.1), and nothing already-implemented ever re-adds a successfully-dispatched (RUN) job back to that pool. The only mechanism design.md itself anticipates for that is `scheduler-preemption`, confirmed still unimplemented (`requirements-generated`, no approvals). **Revisit bug-1631 and add the state/allocs reversal to `dispatch_a_job_()`'s four retryable branches only once `scheduler-preemption` (or an equivalent mechanism) actually creates a real RUN→retryable-redispatch path** — not before, since there's no reachable caller to regression-test against today. `test_rpc_failure_bound_resets_after_intervening_success` (`test_dispatch_fail_retry.c`) proves Requirement 4.3's actual, literal ask (the counter genuinely resets on an intervening success) without asserting the unreachable state/allocs behavior.
 - Task 5.2: design.md's "vacuously by construction" reasoning (`Controller Dispatch Commit` and `Controller Dispatch-Failure Classification` sections — "on any of the six classified failure paths there is nothing to release") silently assumes every `dispatch_a_job_()` invocation starts from a job that's never had anything committed (i.e., a first-ever dispatch attempt) — true today (see bug-1631 above) but not stated as a scoped assumption in the doc text itself. Worth correcting design.md's wording if it's ever revisited for the `scheduler-preemption` follow-up.
+- Task 5.3: `test_dispatch_fail_terminal.c` (task 3.2) and `test_jobdispatch_no_double_notify.c` (task 4.1) each covered PART of this task but not the full combination in one run — 3.2's test bypasses `handle_jobdispatch()` entirely (via `dispatch_a_job_test()`), so it can't exercise the double-notify race at all; 4.1's test only proves an *aggregate* batch notify count, not a *direct* per-job zero-notify proof. `test_handle_jobdispatch_terminal_no_notify_no_leak` (extending 4.1's file, the one with the right entry point) closes this by combining EXIT resolution + a direct single-job zero-notify assertion + the canary-reference no-leak proof, all through the real `handle_jobdispatch()` entry point in one test. Pattern worth remembering: a batch-aggregate test and a bypassed-entry-point test can each look like they cover a scenario while neither actually does, in combination, what a later task's literal wording asks for ("all pass IN THE SAME TEST RUN").

src/controller/testsuite/test_jobdispatch_no_double_notify.c

@@ -46,6 +46,21 @@
  * for the full rationale -- this binary's sole notify() definition lives in
  * test_commit_barrier_reply.c, exposed read-only via its
  * cbrp_notify_capture_*_ext() accessors (added by task 3.3).
+ *
+ * scheduler-dispatch-commit task 5.3 (Req 4.2, 4.5) adds
+ * test_handle_jobdispatch_terminal_no_notify_no_leak, below. It is a
+ * sharper-focused sibling of this file's original (task 4.1) 3-job-batch
+ * test: that test proves the BATCH's total notify_dispatch() count is 1,
+ * attributable to the retryable job (an aggregate proof the terminal job
+ * contributed zero calls only by subtraction/arithmetic against the other
+ * two jobs' known outcomes) and never exercises reference-count cleanliness
+ * for any job. The new test instead drives a SINGLE terminal-classified job
+ * through this same real handle_jobdispatch() entry point (not
+ * dispatch_a_job_test()'s bypass, unlike test_dispatch_fail_terminal.c's
+ * task-3.2 tests) and combines, in ONE test run: (a) JOB_STAT_EXIT
+ * resolution, (b) a direct (not subtracted) zero-notify assertion for this
+ * job's own batch, and (c) test_dispatch_fail_terminal.c's canary-reference
+ * technique proving the reference count returns to its pre-attempt value.
  */
 
 #include <rtti.h>
@@ -347,6 +362,115 @@ static void test_handle_jobdispatch_matches_dispatch_a_job_notify_count(void)
           "PASS\n");
 }
 
+#define JND_JOBID_TERMINAL_SOLO 92201ULL
+
+/* Canary-reference technique, reused verbatim from
+ * test_dispatch_fail_terminal.c's dft_arm_ref_canary()/dft_assert_ref_
+ * not_over_released() (task 3.2): armed immediately BEFORE the call under
+ * test (not after -- arming after would rely incidentally on other
+ * assertions crashing first, which doesn't independently prove anything),
+ * released and the job re-touched immediately after. An over-release during
+ * the call would free j while the canary reference is still held, making
+ * the post-call touch a real, ASAN-caught heap-use-after-free. job_t's real
+ * refcount lives in jobj.c's file-local job_private_t with no accessor
+ * exposed to a test, so this indirect technique is the only available
+ * proof. */
+static void jnd_arm_ref_canary(job_t* j)
+{
+    reference(j);
+}
+
+static void jnd_assert_ref_not_over_released(job_t* j, uint64_t expect_jobid)
+{
+    dereference(j);
+
+    TEST_ASSERT_EQUAL_UINT64_MESSAGE(expect_jobid, j->job.jobid,
+        "Req 4.5: j must still be safely readable after releasing the "
+        "canary reference -- an over-release inside "
+        "handle_jobdispatch()/dispatch_a_job_() would have freed j while "
+        "the canary was still held, making this a heap-use-after-free "
+        "(this terminal site already dereferences correctly even before "
+        "bug-1627's fix, which was scoped to the four retryable sites -- "
+        "this check guards the same class of risk, not that specific bug)");
+}
+
+/* Task 5.3's focused proof (Req 4.2, 4.5) -- see this file's header for how
+ * this differs from test_handle_jobdispatch_matches_dispatch_a_job_notify_
+ * count above. A single-job batch, terminal-classified (nhosts == 0),
+ * driven through the real handle_jobdispatch() entry point. */
+static void test_handle_jobdispatch_terminal_no_notify_no_leak(void)
+{
+    xce_job_alloc_t*   allocs[1];
+    xce_job_dispatch_t dispatchmsg;
+    xce_request_t      req;
+    job_t*             jterm;
+    int                rc;
+
+    jnd_setup();
+
+    jterm = jnd_seed_job(JND_JOBID_TERMINAL_SOLO, "jnd-q-solo");
+
+    /* nhosts == 0: XCE_DISPATCH_FAIL_SITE_INVALID_ALLOC, terminal --
+     * mirrors this file's own JND_JOBID_TERMINAL allocation and
+     * test_dispatch_fail_terminal.c's test_invalid_alloc_resolves_job_
+     * to_exit, but reached this time through the real handle_jobdispatch()
+     * entry point instead of dispatch_a_job_test()'s bypass. */
+    allocs[0] = make_job_alloc(JND_JOBID_TERMINAL_SOLO, 0, NULL);
+    TEST_ASSERT_NOT_NULL(allocs[0]);
+
+    memset(&dispatchmsg, 0, sizeof(dispatchmsg));
+    dispatchmsg.njobs  = 1;
+    dispatchmsg.allocs = allocs;
+
+    memset(&req, 0, sizeof(req));
+    req.opcode           = OPCode_jobdispatch;
+    req.seq              = 43;
+    req.from             = &g_jnd_req_from;
+    req.body_which       = Request_body_dispatchmsg;
+    req.body.dispatchmsg = &dispatchmsg;
+
+    cbrp_notify_capture_reset_ext();
+
+    jnd_arm_ref_canary(jterm);
+
+    /* Act: handle_jobdispatch() consumes/frees allocs[0] itself via
+     * MOVE_PTR + free_JobAlloc_ptr() in its own loop -- nothing left for
+     * this test to free afterward. */
+    rc = handle_jobdispatch(&g_jnd, &g_jnd_req_ep, &req);
+    TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc, "handle_jobdispatch's own return "
+        "is unconditionally 0, unaffected by any per-job outcome");
+
+    TEST_ASSERT_EQUAL_UINT32_MESSAGE(JOB_STAT_EXIT, jterm->job.state,
+        "Req 4.2: a terminally-classified dispatch failure must resolve "
+        "the job to EXIT, reached this time through the real "
+        "handle_jobdispatch() entry point rather than the "
+        "dispatch_a_job_test() bypass test_dispatch_fail_terminal.c uses");
+    TEST_ASSERT_NULL_MESSAGE(jterm->job.allocs,
+        "Req 4.5: no partial allocation data may be left behind");
+
+    /* THE assertion this task exists for: a terminal dispatch failure must
+     * not ALSO produce a retry-eligible jobDispatchFail report for the same
+     * failure -- mark_job_fail()'s own schedFailJob hook chain (already
+     * fired above) is the sole notification. Unlike test_handle_jobdispatch_
+     * matches_dispatch_a_job_notify_count's 3-job batch (where the terminal
+     * job's zero contribution is inferred by the total equaling exactly the
+     * retryable job's own 1 call), this single-job batch makes the zero
+     * count direct: any notify_dispatch() call surfacing here can only have
+     * come from this one terminal job, whether from handle_jobdispatch()'s
+     * old per-job loop call (task 4.1's regression) or from dispatch_a_
+     * job_() itself misclassifying/misrouting (task 3.2's regression). */
+    TEST_ASSERT_EQUAL_INT_MESSAGE(0, cbrp_notify_capture_count_ext(),
+        "a terminal dispatch failure must not also produce a "
+        "notify_dispatch() report for this same failure (regression test "
+        "for the round-2 double-notify finding, design.md's Integration "
+        "Tests third bullet)");
+
+    jnd_assert_ref_not_over_released(jterm, JND_JOBID_TERMINAL_SOLO);
+
+    jnd_teardown();
+    printf("test_handle_jobdispatch_terminal_no_notify_no_leak: PASS\n");
+}
+
 void test_jobdispatch_no_double_notify(void)
 {
     UnitySetTestFile(__FILE__);
@@ -354,4 +478,5 @@ void test_jobdispatch_no_double_notify(void)
           "4.2)\n\n");
 
     RUN_TEST(test_handle_jobdispatch_matches_dispatch_a_job_notify_count);
+    RUN_TEST(test_handle_jobdispatch_terminal_no_notify_no_leak);
 }