107 lines
5.7 KiB
Markdown
107 lines
5.7 KiB
Markdown
---
|
|
id: "bug-1883"
|
|
status: "fixed"
|
|
status_updated: "2026-09-29"
|
|
status_note: ""
|
|
component: "scheduler"
|
|
severity: "wrong-result"
|
|
timestamp: "2026-09-29T04:07:19Z"
|
|
last_seen: "2026-09-29T04:07:19Z"
|
|
occurrences: 1
|
|
files: ["src/scheduler/src/plugins/seqjob_allocator.c", "src/scheduler/src/plugin.c", "src/scheduler/src/schedobj.c", "src/scheduler/src/host_iter.c"]
|
|
tags: ["scheduler", "allocation", "compound", "seqjob"]
|
|
related: ["bug-1768", "bug-1766"]
|
|
fix_commits: ["c10f32cc"]
|
|
gitea_issue: 334
|
|
gitea_synced_status: "fixed"
|
|
gitea_synced_hash: "91f00d9aa29c"
|
|
---
|
|
|
|
# bug-1883: xce_scheduler seqjob reuses a stale per-host want from an earlier COMPOUND/ALTER primary
|
|
|
|
## Description
|
|
**Observed:** (by code reading; no failing run captured yet)
|
|
When a COMPOUND or ALTER job has a `maxTasks == 1` primary after another
|
|
primary in the same attempt, seqjob's host search for the later primary
|
|
starts from the per-host want (`ctx->alloc_req.host`) left behind by the
|
|
earlier primary, with `slots:1` appended rather than replacing it:
|
|
|
|
```
|
|
.host on entry to seqjob host loop (2nd+ primary, shared ctx)
|
|
= <previous primary's last per-host want, e.g. slots:N + rusage> + slots:1
|
|
```
|
|
|
|
- `plugin_prepare_allocation()` resets only `alloc_req.shared` in the
|
|
global pass (`src/scheduler/src/plugin.c:617-619`), never `.host`.
|
|
- `seqjob_prepare_allocation()` appends `slots=1`
|
|
(`src/scheduler/src/plugins/seqjob_allocator.c:118`), and
|
|
`xvec_idval_store` appends without dedup (`src/lib/xvec.c:247-270`).
|
|
- The host iterator pre-filters candidates against this stale vector
|
|
(`src/scheduler/src/host_iter.c:136,158`), silently dropping hosts that
|
|
would satisfy the later primary's real `{slots:1}` want.
|
|
- A global limit on `slots` reads the first match via `get_resval_by_id`
|
|
(`src/scheduler/src/resource.c:951-968`), so it sees the stale N, not 1.
|
|
|
|
parjob does not have this problem because it clears `.host` per bucket
|
|
(`src/scheduler/src/plugins/parjob_allocator.c:478-479`).
|
|
|
|
**Expected:** every primary's per-host want is computed from that primary's
|
|
own resource request only (the bug-1768 principle), so the later seqjob
|
|
primary is pre-filtered and limit-checked against `{slots:1}` plus its own
|
|
rusage.
|
|
**Impact:** wrong result. A later single-task primary of a COMPOUND/ALTER job
|
|
can pend (or pick a later host) although a host that fits it is free; pend
|
|
reasons can list hosts filtered on resources the primary never asked for.
|
|
Plain single-primary jobs are unaffected (fresh ctx).
|
|
**Found by:** review. Discovery/requirements research for spec
|
|
`scheduler-allocator-runner` (2026-09-28). That spec deliberately keeps this
|
|
behaviour (Requirement 3 AC 6) so the refactor stays behaviour-preserving;
|
|
fix here, separately, with its own test.
|
|
|
|
## Environment
|
|
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
|
- libc: glibc 2.42-47 (interpreter /nix/store/wb6rhpznjfczwlwx23zmdrrw74bayxw4-glibc-2.42-47/lib/ld-linux-x86-64.so.2)
|
|
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
|
- Build: asan (xce.build)
|
|
- Code: scheduler-framework @ 51736fa4-dirty
|
|
- Deployment: n/a; found by code reading, not on a deployment. Affects any topology where the scheduler dispatches a COMPOUND/ALTER job with a `maxTasks == 1` primary after another primary.
|
|
|
|
## Reproduction
|
|
1. `git checkout 51736fa4` (branch `scheduler-framework`).
|
|
2. Not yet reproduced by a test. Proposed white-box unit test (same
|
|
harness as `src/scheduler/testsuite/test_parjob_e2e_multislot.c`'s
|
|
compound tests): one shared `xce_sched_ctx_t`; primary A `maxTasks=4`
|
|
dispatched through parjob onto host-1 (4 slots) so `ctx->alloc_req.host`
|
|
ends as `slots:4`; then `sched_ctx_set_primary()` for primary B
|
|
`maxTasks=1`, and run seqjob against host-2 with exactly 1 free slot.
|
|
3. Expected RED: host-2 is pre-filtered out (stale `slots:4`) and B pends.
|
|
Expected GREEN after a fix: B is placed on host-2.
|
|
- Frequency: deterministic (1/1 runs of the test before the fix)
|
|
- Evidence: not kept (no run yet). Code-reading trace in the research
|
|
summary for spec `.kiro/specs/scheduler-allocator-runner/` (requirements
|
|
Requirement 3 AC 6).
|
|
|
|
## Root cause
|
|
`plugin_prepare_allocation()` (src/scheduler/src/plugin.c) reset only
|
|
`ctx->alloc_req.shared` in its job-level pass (`ctx->worker == NULL`), while
|
|
seqjob's and parjob's prepare hooks append this primary's job-level slots
|
|
want into `ctx->alloc_req.host` with the append-only `xvec_idval_store()`.
|
|
A COMPOUND/ALTER job's primaries share one ctx (`sched_ctx_set_primary()`
|
|
runs the job-level pass per primary), so `.host` still held the previous
|
|
primary's last per-host want. The host iterator pre-filtered on it
|
|
(`host_iter.c`), and a global limit on slots read its first `slots` entry.
|
|
|
|
## Fix
|
|
`plugin_prepare_allocation()`'s job-level pass now resets `.host` as well
|
|
as `.shared`, so each primary's pass starts from its own want only
|
|
(`c10f32cc`). The per-host pass already reset `.host`; parjob's per-bucket
|
|
reset is unchanged.
|
|
|
|
## Verification
|
|
- Test: test_compound_later_seqjob_primary_ignores_previous_want in src/scheduler/testsuite/test_parjob_e2e_multislot.c (hosts A=4 slots, B=1 slot; COMPOUND `4*{..} + 1*{..}` through the real scheduler dispatch): RED before the fix (PEND, "Expected 1 Was 2"), GREEN after (DISPATCH, 4 slots on A + 1 on B)
|
|
- Suites: ASAN debug build (src/xce.build.discrete), `meson test --no-suite xceadmin`: 117/117 OK, 0 Unity :FAIL lines, 0 sanitizer reports (2026-09-29)
|
|
- Live: n/a -- no soak or deploy script submits COMPOUND/ALTER jobs; the test drives the real picker/allocator dispatch path end to end
|
|
|
|
## References
|
|
- Related: bug-1768 (per-primary sizing), bug-1766 (COMPOUND publish-only-if-all-succeed)
|
|
- Spec: `.kiro/specs/scheduler-allocator-runner/` (keeps this behaviour; out of scope there)
|