feat(scheduler-dispatch-commit): controller resolves terminal dispatch failures to EXIT (task 3.2)

The two terminal-classified failure sites in dispatch_a_job_() (invalid
allocation, queue-exec-info lookup failure) now actually resolve the job
instead of leaving it stuck at PEND forever with a dangling reference to
nowhere. Each site: lock(j, RTTI_LOCK_WRITE) -> genuinely calls
xce_classify_dispatch_fail(site) and branches on the result (not a
hardcoded assumption) -> mark_job_fail() under the lock (stage=0 routes
apply_job_fail_transition() to JOB_STAT_EXIT) -> unlock(j) -> the existing
dereference(j) (kept exactly where it was, not duplicated) -> the existing
return code, unchanged. No notify_dispatch() added -- mark_job_fail()'s
own hook chain is the sole notification for a terminal outcome per
design.md; handle_jobdispatch()'s still-unconditional notify_dispatch()
call is untouched (task 4.1's job).

New test_dispatch_fail_terminal.c proves both sites resolve to
JOB_STAT_EXIT with allocs left NULL, using a canary-reference technique
to prove no reference over-release (job_t's real refcount has no test
accessor). Independently reviewed and approved; applied the reviewer's
one non-blocking fast-follow (arm the canary before the call under test,
not after, so it's an independent proof rather than relying on other
assertions to crash first) and re-verified with a clean incremental build.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Cxa1PmqAB5PyXfJaNvj8TN
This commit is contained in:
Rongsong Shen 2026-08-23 16:36:39 +00:00
parent d9aedc4b34
commit 8a4266ada4
No known key found for this signature in database
GPG key ID: 5BCDAAE0123E9FE2
5 changed files with 436 additions and 1 deletions

View file

@ -49,7 +49,7 @@
- _Requirements: 2.1, 2.2, 2.3, 2.4, 5.1_
- _Depends: 1.2, 1.3_
- [ ] 3.2 Controller resolves a terminal (locally-invalid) dispatch failure
- [x] 3.2 Controller resolves a terminal (locally-invalid) dispatch failure
- When a dispatch attempt fails for a reason the classification (Task 1.1) marks terminal, the controller resolves the job to a permanently failed state immediately, protected against concurrent access to the job record, and releases any resource-allocation data tentatively associated with the attempt
- This covers the two failure sites that already release the job reference they hold today — confirm that behavior is preserved, not duplicated, by this new resolution step
- Observable completion: a terminally-classified dispatch failure leaves the job permanently failed, with no partial allocation data left behind and the job's reference count unchanged from before the attempt
@ -105,3 +105,5 @@
- Task 3.1: reconfirmed **bug-1429** (`controller_unit_test` TIMEOUT/hang, pre-existing, unrooted) a 3rd time, now hanging inside `test_commit_barrier_reply.c`'s `test_jobsubmit_repeated_truncation_forced_failures_no_phantom_jobs` — a 3rd distinct hang location across 3 sessions, confirmed unrelated to `dispatch_a_job_()` (that test never calls it) and independently reproduced twice more (standalone binary run past the meson harness's 300s cap, and a fully separate rebuild). Because this hang sits earlier in `controller_test.c`'s registration order than most of this feature's own new test files, a full un-filtered `meson test` run will NOT reach them — **task 5.4's full-suite validation must run with the hanging group filtered out (`controller_unit_test -t <group>`) or budget for the 300s timeout**, not treat a bare `meson test` TIMEOUT as this feature's own failure.
- Task 3.1: confirmed pre-existing (not introduced by this task, not in this task's boundary to fix) — `dispatch_a_job_()`'s success path never calls `dereference(j)` on the reference taken at lookup; identical in the diff's before/after, and the sole caller `handle_jobdispatch()` never dereferences either. A candidate future bug-log entry, out of scope for this spec.
- Task 3.1: when validating a build against this suite's ASAN leak output, always replicate the project's own `ASAN_OPTIONS=detect_leaks=0` test env (`meson.build:229`) — a standalone binary run without it produces spurious leak reports (a red herring, not a regression) that can waste review time.
- Task 3.2: `job_t`'s real reference count lives in `jobj.c`'s file-local `job_private_t`, with no accessor exposed anywhere in the codebase — a test proving no reference-count regression can't do a direct numeric comparison. The working pattern (used by `test_dispatch_fail_terminal.c`) is a "canary reference": take one extra manual `reference()` on the job **immediately before** the call under test (not after — arming it after only relies incidentally on other assertions crashing first, which doesn't independently prove anything), then `dereference()` it after the call and confirm the job is still safely readable — an over-release during the call frees the job while the canary is still held, so the post-call touch is a real ASAN-caught heap-use-after-free.
- Task 3.2: reconfirmed bug-1429's hang again (still `test_commit_barrier_reply.c`, still unrelated to any change in this feature) — third+ reproduction, still not root-caused. Continues to require `controller_unit_test -t <group>` filtering for a clean isolated regression signal.