myxce/.wolf/bugs/bug-1883.md

id	status	status_updated	status_note	component	severity	timestamp	last_seen	occurrences	files	tags	related	fix_commits	gitea_issue	gitea_synced_status	gitea_synced_hash
bug-1883	fixed	2026-09-29		scheduler	wrong-result	2026-09-29T04:07:19Z	2026-09-29T04:07:19Z	1	src/scheduler/src/plugins/seqjob_allocator.c src/scheduler/src/plugin.c src/scheduler/src/schedobj.c src/scheduler/src/host_iter.c	scheduler allocation compound seqjob	bug-1768 bug-1766	c10f32cc	334	fixed	91f00d9aa29c

bug-1883: xce_scheduler seqjob reuses a stale per-host want from an earlier COMPOUND/ALTER primary

Description

Observed: (by code reading; no failing run captured yet) When a COMPOUND or ALTER job has a maxTasks == 1 primary after another primary in the same attempt, seqjob's host search for the later primary starts from the per-host want (ctx->alloc_req.host) left behind by the earlier primary, with slots:1 appended rather than replacing it:

.host on entry to seqjob host loop (2nd+ primary, shared ctx)
  = <previous primary's last per-host want, e.g. slots:N + rusage> + slots:1

• plugin_prepare_allocation() resets only alloc_req.shared in the global pass (src/scheduler/src/plugin.c:617-619), never .host.
• seqjob_prepare_allocation() appends slots=1 (src/scheduler/src/plugins/seqjob_allocator.c:118), and xvec_idval_store appends without dedup (src/lib/xvec.c:247-270).
• The host iterator pre-filters candidates against this stale vector (src/scheduler/src/host_iter.c:136,158), silently dropping hosts that would satisfy the later primary's real {slots:1} want.
• A global limit on slots reads the first match via get_resval_by_id (src/scheduler/src/resource.c:951-968), so it sees the stale N, not 1.

parjob does not have this problem because it clears .host per bucket (src/scheduler/src/plugins/parjob_allocator.c:478-479).

Expected: every primary's per-host want is computed from that primary's own resource request only (the bug-1768 principle), so the later seqjob primary is pre-filtered and limit-checked against {slots:1} plus its own rusage. Impact: wrong result. A later single-task primary of a COMPOUND/ALTER job can pend (or pick a later host) although a host that fits it is free; pend reasons can list hosts filtered on resources the primary never asked for. Plain single-primary jobs are unaffected (fresh ctx). Found by: review. Discovery/requirements research for spec scheduler-allocator-runner (2026-09-28). That spec deliberately keeps this behaviour (Requirement 3 AC 6) so the refactor stays behaviour-preserving; fix here, separately, with its own test.

Environment

• OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
• libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
• Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
• Build: asan (xce.build)
• Code: scheduler-framework @ 51736fa4-dirty
• Deployment: n/a; found by code reading, not on a deployment. Affects any topology where the scheduler dispatches a COMPOUND/ALTER job with a maxTasks == 1 primary after another primary.

Reproduction

1. git checkout 51736fa4 (branch scheduler-framework).
2. Not yet reproduced by a test. Proposed white-box unit test (same harness as src/scheduler/testsuite/test_parjob_e2e_multislot.c's compound tests): one shared xce_sched_ctx_t; primary A maxTasks=4 dispatched through parjob onto host-1 (4 slots) so ctx->alloc_req.host ends as slots:4; then sched_ctx_set_primary() for primary B maxTasks=1, and run seqjob against host-2 with exactly 1 free slot.
3. Expected RED: host-2 is pre-filtered out (stale slots:4) and B pends. Expected GREEN after a fix: B is placed on host-2.

• Frequency: deterministic (1/1 runs of the test before the fix)
• Evidence: not kept (no run yet). Code-reading trace in the research summary for spec .kiro/specs/scheduler-allocator-runner/ (requirements Requirement 3 AC 6).

Root cause

plugin_prepare_allocation() (src/scheduler/src/plugin.c) reset only ctx->alloc_req.shared in its job-level pass (ctx->worker == NULL), while seqjob's and parjob's prepare hooks append this primary's job-level slots want into ctx->alloc_req.host with the append-only xvec_idval_store(). A COMPOUND/ALTER job's primaries share one ctx (sched_ctx_set_primary() runs the job-level pass per primary), so .host still held the previous primary's last per-host want. The host iterator pre-filtered on it (host_iter.c), and a global limit on slots read its first slots entry.

Fix

plugin_prepare_allocation()'s job-level pass now resets .host as well as .shared, so each primary's pass starts from its own want only (c10f32cc). The per-host pass already reset .host; parjob's per-bucket reset is unchanged.

Verification

• Test: test_compound_later_seqjob_primary_ignores_previous_want in src/scheduler/testsuite/test_parjob_e2e_multislot.c (hosts A=4 slots, B=1 slot; COMPOUND 4*{..} + 1*{..} through the real scheduler dispatch): RED before the fix (PEND, "Expected 1 Was 2"), GREEN after (DISPATCH, 4 slots on A + 1 on B)
• Suites: ASAN debug build (src/xce.build.discrete), meson test --no-suite xceadmin: 117/117 OK, 0 Unity :FAIL lines, 0 sanitizer reports (2026-09-29)
• Live: n/a -- no soak or deploy script submits COMPOUND/ALTER jobs; the test drives the real picker/allocator dispatch path end to end

References

• Related: bug-1768 (per-primary sizing), bug-1766 (COMPOUND publish-only-if-all-succeed)
• Spec: .kiro/specs/scheduler-allocator-runner/ (keeps this behaviour; out of scope there)