feat(resreq-maxtasks-wiring): prove array-job element spans hosts too

Extends xce-test-multihost with a second scenario: a 2-element array
job (jobsub -a 2) where each element independently requests the same
nproc+1 task count as the plain-job scenario. Drives element [0]
through the identical RUN-then-DONE, both-hosts-started proof, and
confirms its jobinfo Slots: field matches the plain job's exactly
(Requirement 5.5). Traced the array-expansion path
(handler.c's sched_on_job_array()/clone_job_submit_data()) to confirm
no array-specific shortcut bypasses apply_job_task_count() -- each
element flows through the same core_build_job() ingest as a
standalone job.

Fixes a real bug (bug-1775) found while building this: an array
element's bracketed jobid (e.g. "<base>[0]") was misparsed by
grep -E's bracket-expression syntax, causing a false negative in the
per-host verification. Added ere_escape(), a no-op for plain
(non-bracketed) jobids.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XYdtrEWWZfKRRGED7raEYg

.wolf/buglog.json

@@ -20992,6 +20992,93 @@
       "related_bugs": [],
       "occurrences": 1,
       "last_seen": "2026-09-04T00:00:00Z"
+    },
+    {
+      "id": "bug-1772",
+      "timestamp": "2026-09-05T00:00:00Z",
+      "error_message": "OPEN, NOT A CRASH -- pre-existing, out-of-scope gap discovered during requirements scoping for the resreq-maxtasks-wiring spec (bug-1770's fix). snapshot_build_primary_schedobj() (src/scheduler/src/snapshot_load.c:81-96) never computes primary->resreq_fp at all, unlike core_build_primary_schedobj() (src/scheduler/src/core_events.c:130-135), which sets resreq_fp.min_tasks/max_tasks/select_node_id/rusage_hash from the parsed xce_primary_resreq_t immediately after parsing. Every job reconstructed from a snapshot (post-restart or post-failover restore) therefore gets a zeroed resreq_fp -- the same class of gap as bug-1767's broader 'recurring schedobj-reconstruction-duplication' pattern (snapshot_load.c independently re-implements core_events.c's schedobj construction rather than sharing it, and each independent copy has repeatedly drifted field-by-field). TRIGGER CORRECTION (2026-09-05, resreq-maxtasks-wiring design review): this entry's original phrasing ('post-restart or post-failover restore') overstates the live trigger. xce_sched_core_load_snapshot()'s ONLY real (non-test) caller anywhere in src/ is sim.c's sim_locate_and_load_snapshot(), reachable only via main.c's --sim FILE offline replay flag. Normal-mode main.c has no snapshot-load call or flag at all; a live scheduler restart (including post-failover, e.g. run-ha-soak.sh's restart_scheduler_for_new_primary()) starts with empty in-memory state and rejoins via on_scheduler_join() -> the same core_build_job() path fresh ingest uses -- never touching snapshot_load.c. The accurate trigger is: a job in a .replay-recorded snapshot loaded by the --sim simulation/replay driver, not any live-cluster crash-recovery or failover scenario. This does not change the bug's validity or fix (the gap is real for --sim/replay fidelity, which matters per this project's driver-equivalence invariant since --sim drives the identical allocator dispatch) -- it only corrects when it's actually reachable, which matters for scoping test coverage (an HA soak does NOT exercise this path at all).",
+      "root_cause": "snapshot_load.c's snapshot_build_primary_schedobj() is a hand-duplicated, field-for-field port of core_events.c's core_build_primary_schedobj() (kept independent because core_build_primary_schedobj() is static and the two files are compiled as separate objects -- see bug-1767's own root-cause text for the deliberate-duplication rationale). The port was written before task 1.2 (scheduler-allocation feature) added resreq_fp/hash_rusages() wiring to the core_events.c original, and the snapshot_load.c copy was never updated to match. Effect: a snapshot-restored job's unsatisfiable-resreq cache fingerprint (used by scheduler-allocation's alloc_cache.c) is always the zero value, which is a valid-looking but WRONG fingerprint -- it could produce a false cache hit against an unrelated job that also happens to have a zeroed fingerprint (extremely unlikely today since real jobs always populate resreq_fp correctly at first ingest, but a snapshot-restored job's own fingerprint stays wrong for its entire post-restore lifetime, and any code that later relies on comparing/keying by resreq_fp for a restored job silently gets 0/0/0/0). Discovered while scoping resreq-maxtasks-wiring (bug-1770's fix spec) because that fix will make resreq_fp carry real, meaningful, non-degenerate maxTasks values for the first time -- increasing the practical consequence of this gap versus its current all-zero-everywhere baseline.",
+      "fix": "NOT FIXED -- logged for its own follow-up, deliberately kept out of the resreq-maxtasks-wiring spec's scope (per that spec's own requirements-phase scoping decision, 2026-09-05) to avoid absorbing an unrelated schedobj-reconstruction-duplication gap into a narrowly-scoped bug fix. Fix should add the same resreq_fp computation core_build_primary_schedobj() does (or better, per bug-1767's own recommended remediation, export core_build_primary_schedobj()/core_build_schedobj() and have snapshot_load.c call the real function instead of maintaining a third independent duplicate).",
+      "tags": [
+        "scheduler",
+        "snapshot-restore",
+        "resreq-fingerprint",
+        "cache",
+        "pre-existing",
+        "schedobj-duplication",
+        "severity-medium"
+      ],
+      "related_bugs": [
+        "bug-1767",
+        "bug-1770"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-09-05T00:00:00Z"
+    },
+    {
+      "id": "bug-1773",
+      "timestamp": "2026-09-05T00:00:00Z",
+      "error_message": "OPEN, NOT A CRASH -- pre-existing, out-of-scope gap discovered during design review for the resreq-maxtasks-wiring spec (bug-1770's fix). A job's resource-request UPDATE (JOB_UPDATE core event / sched_update_existing_job()) never takes effect for scheduling purposes, for ANY field -- not just task count. sched_update_existing_job() (src/scheduler/src/schedobj.c:597-656) and core_events.c's JOB_UPDATE handler (src/scheduler/src/core_events.c:1200-1234) both do `free(job->resreq); job->resreq = newreq;` (a shallow free via FREEUP, utils.h:114) but never touch job->schedobj. ADDENDUM (2026-09-05, independent follow-up investigation): sched_update_existing_job() itself is dead code (unreachable -- its sole caller is scheduler.c's sched_update(), which has zero live callers, same root architecture as bug-1740). core_events.c's JOB_UPDATE handler IS live (traced main.c -> driver.c's xce_sched_driver_translate_pending_events() -> xce_core_push_events() -> the JOB_UPDATE case), reached routinely by ordinary job-lifecycle notifications (state/rusage/postdone), not only by a genuine resource-request change -- but grepping the whole controller finds no code path that ever mutates an existing job's submit->resreq after initial submission, so this live handler's parse_resreq() branch always re-parses the job's original, unchanged resreq string today. No real 'update a job's resource request' feature exists yet in this codebase; this bug describes what WOULD happen if one were added without also fixing job->schedobj reconstruction.",
+      "root_cause": "CONFIRMED via direct code read: the scheduler's actual dispatch reads job->schedobj (scheduler.c:483, sched_make_allocation(sched, job->schedobj)), not job->resreq. sched_make_primary_allocation() (scheduler.c:315) reads xce_primary_resreq_t* primary = obj->u.primary->resreq -- i.e. job->schedobj->u.primary->resreq, a pointer set exactly once, at ingest, by core_build_primary_schedobj()/make_primary_schedobj() (primary->resreq = resreq, aliasing job->resreq->u.primary at that moment). job->schedobj is built ONLY in core_build_job() (core_events.c:259-265) and make_sched_job() (schedobj.c:194-202) -- confirmed via grepping every `job->schedobj =` assignment in the scheduler; neither update path is among them. So after any resreq update, job->schedobj->u.primary->resreq keeps pointing at the ORIGINAL, ingest-time xce_primary_resreq_t forever, completely disconnected from the freshly-parsed newreq the update path installed into job->resreq. Any change an operator makes to a pending job's select/rusage/task-count via a resource-request update is silently invisible to the scheduler's own dispatch decision -- only a job's original, at-submission resreq is ever actually evaluated.",
+      "fix": "NOT FIXED -- discovered during resreq-maxtasks-wiring's design review (2026-09-05); deliberately kept out of that spec's scope (it is not about task-count wiring, and fixing it changes long-standing update semantics far beyond this feature's boundary). Needs its own dedicated bugfix task/spec: the update paths must rebuild (or re-point) job->schedobj's primary->resreq to the newly-parsed newreq, mirroring how core_build_job()/make_sched_job() construct it at ingest, probably by extracting a shared 'build/refresh primary schedobj from a resreq' helper both ingest and update can call.",
+      "tags": [
+        "scheduler",
+        "job-update",
+        "resreq",
+        "schedobj",
+        "pre-existing",
+        "severity-high",
+        "resreq-maxtasks-wiring-discovery"
+      ],
+      "related_bugs": [
+        "bug-1770"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-09-05T00:00:00Z"
+    },
+    {
+      "id": "bug-1774",
+      "timestamp": "2026-09-05T00:00:00Z",
+      "error_message": "OPEN, NOT A CRASH, DEAD-CODE-ONLY -- pre-existing gap discovered during design review for the resreq-maxtasks-wiring spec, sibling to bug-1772, CORRECTED 2026-09-05: this entry originally claimed make_sched_job()'s ingest path was \"confirmed live, not legacy\" -- that claim was WRONG and has been retracted (see root_cause). make_primary_schedobj() (src/scheduler/src/schedobj.c:45-63, backing make_sched_job()) never computes primary->resreq_fp, unlike core_build_primary_schedobj() (src/scheduler/src/core_events.c:130-135) which sets resreq_fp.min_tasks/max_tasks/select_node_id/rusage_hash from the parsed xce_primary_resreq_t immediately after parsing -- but make_sched_job() itself is unreachable in production today (see root_cause), so this gap currently has ZERO live impact.",
+      "root_cause": "CONFIRMED via direct code read that make_primary_schedobj() calloc's the primary schedobj and leaves resreq_fp all-zero, with no equivalent computation anywhere in the function -- identical failure shape to bug-1772 (snapshot_load.c's snapshot_build_primary_schedobj() has the same gap), a third instance of the recurring schedobj-reconstruction-duplication pattern already tracked by bug-1767. HOWEVER, independently re-verified 2026-09-05 (two separate investigations, exhaustive grep of every call site of sched_update(), sched_update_jobs(), and make_sched_job() across src/ outside testsuite/): sched_update(xce_scheduler_t*) (scheduler.c:1200) has ZERO live callers anywhere in this codebase -- only its own declaration/definition and doc-comments describing it as dead/superseded exist (matching bug-1740's already-established finding for the sibling sched_apply_job_rusage() case, same root architecture). sched_update_jobs() (scheduler.c:1109) is called only from inside sched_update()'s switch, and make_sched_job() (schedobj.c:153) is called only from sched_update_jobs() -- so the ENTIRE chain is unreachable in production. The real, live production job-ingest paths are exactly two: core_build_job() (core_events.c, fresh submission via the driver-translated XCE_CORE_EVENT_JOBS_PUSH path) and snapshot_load.c's restore path -- both confirmed live by the same grep rigor. This entry's original \"confirmed live, not legacy\" parenthetical was an unverified, incorrect aside and is hereby retracted; make_sched_job()'s resreq_fp gap is real but currently dead-code-only, with no production consequence unless/until something makes this path reachable again.",
+      "fix": "NOT FIXED, and NOT CURRENTLY WORTH FIXING given zero live impact -- discovered during resreq-maxtasks-wiring's design review (2026-09-05), corrected same day once the dead-code status was established. resreq-maxtasks-wiring's own Requirement 5.4 does NOT need an exception for this path (it can be excluded from consideration entirely, not merely excluded-and-documented, since it never ingests any real job) -- that spec's actual Req 5.4 exception is for bug-1772's snapshot-restore gap, which IS live. If make_sched_job()'s dead code path is ever revived or repurposed in a future change, revisit this fix then: add the same resreq_fp computation core_build_primary_schedobj() does to make_primary_schedobj(), or better, per bug-1767's broader recommended remediation, share one real schedobj-construction implementation across all three call sites instead of maintaining three independent, drift-prone copies (two live, one dead).",
+      "tags": [
+        "scheduler",
+        "resreq-fingerprint",
+        "dead-code",
+        "schedobj-duplication",
+        "pre-existing",
+        "severity-low",
+        "resreq-maxtasks-wiring-discovery",
+        "corrected"
+      ],
+      "related_bugs": [
+        "bug-1767",
+        "bug-1770",
+        "bug-1772"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-09-05T00:00:00Z"
+    },
+    {
+      "id": "bug-1775",
+      "timestamp": "2026-09-06T00:00:00Z",
+      "error_message": "xce-test-multihost's run-multihost-verify.sh (resreq-maxtasks-wiring task 4.3, array-job scenario): verify_task_started_on_host()'s grep -E pattern embedded a raw jobid string directly in an extended-regex alternation. A bracketed array-element jobid (e.g. \"<base>[0]\") has its \"[0]\" misparsed by grep -E as a one-character bracket-expression class (matching a literal '0') instead of the literal 3-character substring \"[0]\" -- causing a false negative: `grep -qE 'Job base[0] ntasks is:'` does not match the literal log line `\"Job base[0] ntasks is: 5\"`.",
+      "root_cause": "Extended regular expressions treat unescaped `[...]` as a bracket-expression (character class), not a literal substring. Array-element jobids introduced by task 4.3 are the first jobid format in this harness containing ERE metacharacters (plain jobids are digits/hyphens only); the pre-existing plain-job code path never hit this because grep -E on a digits-and-hyphens-only string has no metacharacters to misparse.",
+      "fix": "FIXED (2026-09-06, resreq-maxtasks-wiring task 4.3): added ere_escape() to run-multihost-verify.sh, applied to any jobid string before use in a grep -E pattern (escapes '[' and ']', the only metacharacters a jobid can contain per fmt_jobid_ex/REGEX_JOBARRAY_ID's digit/hyphen/bracket-only format). Confirmed a byte-identical no-op for plain (non-array) jobids. Reproduced the false negative standalone before the fix and confirmed the escaped pattern matches correctly after, independently re-verified by review.",
+      "tags": [
+        "test-infrastructure",
+        "shell-script",
+        "regex",
+        "array-job",
+        "xce-test-multihost",
+        "severity-low"
+      ],
+      "related_bugs": [
+        "bug-1770"
+      ],
+      "occurrences": 1,
+      "last_seen": "2026-09-06T00:00:00Z"
     }
   ]
 }