test(scheduler-allocation): end-to-end multi-host parallel job proof

Drive a real multi-slot parallel job through the actual production
dispatch path -- real sched_init(), real queue/worker/job ingest via
xce_core_push_events(), a real scheduling session (exercising task 5.2's
cache-clear wiring), the real sched_allocator() thread, cache lookup,
parjob_allocator's real my_alloc(), host_iter, and commit -- and confirm
the resulting job allocation genuinely spans 2 hosts with correct
per-host slot counts (Req 4.1, 4.2, 4.3).

Discovered and logged bug-1770: parse_resreq_primary() unconditionally
hardcodes primary->minTasks/maxTasks to 1 regardless of what a job
actually requests, and nothing in the real ingest pipeline ever copies
the job's own submitted task count (correctly threaded from the
apiserver's num_processors/maxnum_processors all the way to
xce_sched_job_t.mintasks/maxtasks) onto the nested xce_primary_resreq_t
fields the allocator dispatch actually reads. This predates the feature
by ~7 months and is out of this spec's scope to fix, but it means
parjob_allocator has zero live production path until a dedicated
follow-up lands. The test compensates with one clearly-labeled patch of
the real, already-ingested resreq's task-count fields immediately after
real parsing; everything else in the pipeline is unmodified production
code.

Also logged bug-1771 (xce_sched_fcfs's fcfs_del_job() dereferences
job->queue->name with no NULL guard) discovered while building this
task's fixture; worked around by registering the job's queue before
push, mirroring the apiserver's own pre-submission queue validation.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg

.wolf/buglog.json

@@ -20947,6 +20947,51 @@
       ],
       "occurrences": 1,
       "last_seen": "2026-09-04T00:00:00Z"
+    },
+    {
+      "id": "bug-1770",
+      "timestamp": "2026-09-04T00:00:00Z",
+      "error_message": "OPEN, NOT A CRASH -- pre-existing, out-of-scope gap discovered during scheduler-allocation task 5.3's end-to-end integration test. The REAL job-ingest path can NEVER produce a primary resource request with maxTasks > 1, regardless of what task count a real job submission specifies -- parjob_allocator.c's my_alloc() (maxTasks > 1) is therefore UNREACHABLE from any real production job submission today.",
+      "root_cause": "CONFIRMED via direct code read and a genuine RED-phase run of test_parjob_e2e_multislot.c (src/scheduler/testsuite/): parse_resreq_primary() (src/lib/resreq.c:431-432) unconditionally hardcodes `primary->minTasks = 1; primary->maxTasks = 1;` after parsing a resreq string's select/rusage/span/same/affinity clauses, regardless of their content -- there is no resreq-string syntax for task count at all (LSF-style task count is a job-submission-level parameter, `job_submit_data_t.mintasks/maxtasks`, sibling to the resreq string, not nested inside it). Neither of this codebase's two job-ingest constructors -- core_build_job()/core_build_primary_schedobj() (src/scheduler/src/core_events.c, the real production JOBS_PUSH path) nor make_sched_job() (src/scheduler/src/schedobj.c, the sibling non-event construction path) -- ever copies the job's own submitted job_submit_data_t.mintasks/maxtasks (correctly propagated onto the TOP-LEVEL xce_sched_job_t.mintasks/maxtasks by both functions, confirmed working) onto the NESTED xce_primary_resreq_t.minTasks/maxTasks that sched_make_primary_alloc_()'s maxTasks>1 dispatch check (scheduler.c:290) and parjob_allocator.c's my_alloc() actually read. Net effect: every job's primary resreq that parses successfully has maxTasks pinned to 1, so sched_make_primary_alloc_() always routes it to seqjob_allocator instead of parjob_allocator (scheduler.c:290-294) -- parjob_allocator's real combinatorial multi-host allocation (scheduler-allocation spec, tasks 4.1-4.3) is fully implemented and unit-tested but cannot be exercised by any real job submission as the system stands today. Live-confirmed: test_parjob_e2e_multislot.c's own RED-phase run (maxTasks patch temporarily disabled) showed the job routing to seqjob and dispatching onto exactly 1 host instead of the intended 2 (`Expected 2 Was 1`).",
+      "fix": "NOT FIXED -- discovered and characterized during scheduler-allocation task 5.3's review (2026-09-04), deliberately left out of that task's boundary (an end-to-end TEST task, not scoped to touch resreq-parsing/job-submission plumbing). Fixing requires a real design decision: parse_resreq_primary()'s signature would need the job's task count threaded in from BOTH real call sites (core_build_job()/core_build_primary_schedobj() and make_sched_job()), plus a decision about how a COMPOUND resreq's several nested primaries should each receive a single job-level task count (today's model has exactly one job-level mintasks/maxtasks pair, but a COMPOUND can nest multiple independent primaries). Well outside scheduler-allocation's own Boundary Context (the iterator/allocator rework, not resreq-parsing/job-submission wiring) -- exactly the same class of pre-existing, out-of-scope gap tasks 5.1/5.2 already found and deferred in this same feature (bug-1766, bug-1768, bug-1769). test_parjob_e2e_multislot.c's own fixture compensates with one minimal, explicitly-labeled patch of the real, already-ingested xce_primary_resreq_t's task-count fields (PATCH_MAXTASKS_FOR_BUG_1770 in that file) immediately after real JOBS_PUSH ingest and before the dispatch round runs -- everything downstream of that point (cache lookup, parjob dispatch, plan iteration, host claiming, multi-host commit, decision production) is exercised through the unmodified real dispatch path. Needs its own dedicated follow-up task/spec given the severity (parjob_allocator is effectively dead code in production without this fix) and the cross-cutting design decision required for COMPOUND.",
+      "tags": [
+        "scheduler",
+        "resreq-parsing",
+        "job-submission",
+        "parjob-allocator",
+        "pre-existing",
+        "severity-high",
+        "scheduler-allocation-discovery",
+        "task-5.3"
+      ],
+      "related_bugs": [
+        "bug-1766",
+        "bug-1768",
+        "bug-1769"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-09-04T00:00:00Z"
+    },
+    {
+      "id": "bug-1771",
+      "timestamp": "2026-09-04T00:00:00Z",
+      "error_message": "ASAN SEGV (READ of address 0x0) in fcfs_del_job() (src/scheduler/src/plugins/fcfs.c:318, `job->queue->name`), reached via fcfs_commit() <- plugin_commit_decision() <- sched_commit_decision() <- commit_decision() <- sched_allocator(), the very first time a real DISPATCH decision was committed through the real, live sched_allocator() thread in this test directory. Discovered live during scheduler-allocation task 5.3's own end-to-end integration test construction (test_parjob_e2e_multislot.c) -- full ASAN backtrace captured.",
+      "root_cause": "CONFIRMED via direct code read and the live ASAN backtrace. plugin_commit_decision() (src/scheduler/src/plugin.c:442-475) unconditionally fans a committed decision out to EVERY registered plugin's own `.hooks.commit_hook`, regardless of which plugin is the currently-ACTIVE job picker -- this includes the legacy 'xce_sched_fcfs' plugin (src/scheduler/src/plugins/fcfs.c), still registered alongside the modern default 'builtin_fcfs' by sched_register_plugins() (scheduler.c) even when it is never the resolved picker. fcfs_commit() (fcfs.c:326-337) calls fcfs_del_job(plugin, decision->u.alloc->jobid) for every XCE_DECISION_DISPATCH, and fcfs_del_job() (fcfs.c:303-324) unconditionally dereferences `job->queue->name` with no NULL check on `job->queue` -- sched_get_queue() (core.c:257-271) is get-ONLY (never auto-creates), so any job whose xce_sched_job_t.queue is NULL (i.e. its submitted queue name was never registered via a real QUEUE_UPDATE/CONFIG_PUSH event before the job reached DISPATCH) crashes this unrelated legacy plugin's own housekeeping, even though that plugin never elected or tracked the job at all. Every prior real-scheduler_srcs test in this directory that reaches a real sched_commit_decision() (test_scheduler_cache_dispatch.c, test_scheduler_cache_dispatch_compound.c, test_snapshot_load_allocator_regression.c) only ever produces PEND decisions (fcfs_commit()'s `default: break;` branch, never touching job->queue), so this NULL-deref had never been triggered before. Not reachable in real production as the system stands today (a real job's queue is always registered via controller-pushed config before any job can target it), but IS a genuine, real, unguarded NULL dereference in a code path that runs unconditionally for every real DISPATCH decision regardless of test/production context.",
+      "fix": "NOT FIXED -- discovered during scheduler-allocation task 5.3's end-to-end test construction (2026-09-04). test_parjob_e2e_multislot.c's own fixture was corrected instead (added a real XCE_CORE_EVENT_QUEUE_UPDATE push registering the job's queue before the JOBS_PUSH event, matching real production sequencing where queues are always configured before any job can target them) -- this is the correct fix for that specific test, not a production defect this task's own boundary covers. The underlying hardening gap (fcfs_del_job() missing a `job->queue == NULL` guard before dereferencing `job->queue->name`) remains open; recommended minimal fix for a dedicated follow-up: `if ((job == NULL) || (job->queue == NULL)) { return 0; }` in fcfs_del_job(), mirroring the function's own existing `if (job == NULL) { return 0; }` early-return style.",
+      "tags": [
+        "scheduler",
+        "fcfs",
+        "legacy-plugin",
+        "null-deref",
+        "asan",
+        "pre-existing",
+        "severity-low",
+        "scheduler-allocation-discovery",
+        "task-5.3"
+      ],
+      "related_bugs": [],
+      "occurrences": 1,
+      "last_seen": "2026-09-04T00:00:00Z"
     }
   ]
 }