docs: log bug-1931/1932, init parjob-capacity-partitioning spec
- bug-1931: the parjob allocator ignores minTasks (a tasks 2..5 job pends although 2 slots are free). - bug-1932: the unsatisfiable-request cache drops pend reasons on a hit and is unused in the simulator. - .kiro/specs/parjob-capacity-partitioning: initialized spec with the candidate heuristics for capacity-driven plan generation. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
a64d0846c7
commit
cff3304f5d
GPG key ID:
80533536B0EB651C
5 changed files with 270 additions and 0 deletions
53
.kiro/specs/parjob-capacity-partitioning/requirements.md
Normal file
53
.kiro/specs/parjob-capacity-partitioning/requirements.md
Normal file
|
|
@ -0,0 +1,53 @@
|
|||
# Requirements Document
|
||||
|
||||
## Project Description (Input)
|
||||
**Who has the problem:** users submitting parallel (multi-task, `maxTasks > 1`) jobs, and the operators who must explain why such jobs pend.
|
||||
|
||||
**Current situation:** `parjob_plan_iter_next()` (`src/scheduler/src/plugins/parjob_allocator.c`) generates exactly one plan per host count k = 1..maxTasks, always the balanced ceil/floor partition (e.g. N=5, k=2 → only `[3,2]`). Plan shape is a pure function of `maxTasks` and never looks at worker state (`scheduler-allocation` requirements.md, Req 4 AC 1 and "Data Structure Changes Summary", v1 balanced default). Unbalanced splits such as `[4,1]` or `[3,1,1]` are never generated, so a job pends even when the cluster's free capacity could hold it (e.g. two hosts with 4 and 1 free slots for a 5-task job). The per-position first-fit host claim in `sched_claim_bucket_()` (`alloc_runner.c`) is already complete for balanced plans (nested per-bucket requests, largest bucket first); the coverage gap is only the partition shape. Exhaustive partition enumeration is infeasible (p(100) ≈ 1.9×10⁸).
|
||||
|
||||
**What should change:** derive plan shape from the candidate hosts' free capacity instead of from `maxTasks` alone — order the candidate hosts (by preference / free slots) and greedily give each host min(what fits, tasks remaining) until every task is placed or the hosts run out. Requirements to settle in discussion:
|
||||
- keep the fewest-hosts-first preference and bounded cost (~O(hosts) per attempt);
|
||||
- keep the decision trace meaningful (plan / position / host verdicts, plan counts) and pend reasons recorded once per job (`add_job_preason_once`, bug-1927);
|
||||
- relationship to the existing balanced plans (replace, or try balanced first and fall back to capacity-driven);
|
||||
- this supersedes the "pure function of maxTasks" constraint in the `scheduler-allocation` spec, which must be amended accordingly;
|
||||
- multi-dimensional requests (slots + mem etc.): "what fits" per host is the max task count satisfying every resource;
|
||||
- interaction with snapshot/equivalence tests (`test_alloc_equivalence_parjob.c`) and the sim decision trace.
|
||||
|
||||
## Candidate Approaches (discussion notes, 2026-10-09)
|
||||
|
||||
**Groundwork shared by every heuristic — per-host fit.** For each candidate host that passes the existing host-selection pre-filter:
|
||||
|
||||
`fit(h) = min over requested resources r of floor(free_r(h) / per_task_r)`
|
||||
|
||||
Fit values only shape plans. Every host still goes through the authoritative per-host check in `sched_try_alloc_on_host_()`, and the first-fit position filling in `sched_claim_bucket_()` (`alloc_runner.c`) stays valid: a plan's parts are still ordered largest first, and a host that fits a larger part also fits a smaller one, so first-fit loses nothing.
|
||||
|
||||
### Heuristics considered
|
||||
|
||||
1. **Largest-fit greedy (fewest hosts).** Sort hosts by fit, largest first; take hosts until their fits add up to at least N, giving each min(fit, remaining). Always finds the fewest possible hosts (the smallest k whose k largest fits cover N). O(H log H). Produces `[4,1]` for hosts with fits 4 and 1 and N=5.
|
||||
2. **Balanced split under the caps.** Once k is known, spread the N tasks as evenly as each host's fit allows: each host gets min(fit, level), with level chosen so the total is N. Keeps the even split the balanced plans aim for (good for tightly synchronized tasks, e.g. MPI) while respecting capacity.
|
||||
3. **Best-fit to limit fragmentation.** Among ways to cover N with k hosts, prefer hosts whose fit is close to the share they receive. Simple version: run greedy, then swap the last chosen host for the smallest-fit host that still covers the remainder. Keeps large free holes for later big jobs, at the cost of a slightly less even split.
|
||||
4. **Today's balanced plans as a fallback.** Keep the existing per-k balanced plans as last resorts, so current behaviour is a subset and existing equivalence tests keep passing on clusters where balanced plans already succeed.
|
||||
5. **Locality scoring (later).** Rank hosts by rack/switch affinity before applying 1–3; plugs into the `span`/`same`/`affinity` fields the `scheduler-allocation` spec already reserves.
|
||||
|
||||
**Rejected: search-based methods** (local search, simulated annealing, beam search). The plan generator runs every scheduling cycle for every pending parallel job, so its cost must be small, bounded and deterministic; determinism also matters for the simulator's decision trace and the equivalence tests.
|
||||
|
||||
### Recommended direction (to confirm)
|
||||
|
||||
Plan list, in order:
|
||||
1. largest-fit greedy plan (fewest hosts);
|
||||
2. balanced split under the caps on those same k hosts;
|
||||
3. today's balanced plans as a fallback.
|
||||
|
||||
At most about two plans plus the existing N. Configurable: the preference between balance and fewest hosts (or fragmentation), and whether the fallback plans are kept at all.
|
||||
|
||||
### Points the requirements must settle
|
||||
|
||||
- **Reading live worker state.** The plan generator would read worker state, breaking the "plan shape depends only on maxTasks" rule (`scheduler-allocation` Req 4 AC 1) — that spec must be amended — and must take the same read lock the host iterator uses.
|
||||
- **Stale fit values.** A fit computed at plan time can be out of date by the time a host is checked; the per-host check stays authoritative and a failed plan moves on to the next one.
|
||||
- **Decision trace.** The trace should name the heuristic that produced each plan, e.g. `plan 1 (min-hosts) [4,1]`, so pend explanations stay readable.
|
||||
- **Pend reasons.** Still recorded once per job (`add_job_preason_once`, bug-1927).
|
||||
|
||||
Status: initialized only; requirements to be discussed with the user before `/kiro-spec-requirements`.
|
||||
|
||||
## Requirements
|
||||
<!-- Will be generated in /kiro-spec-requirements phase -->
|
||||
24
.kiro/specs/parjob-capacity-partitioning/spec.json
Normal file
24
.kiro/specs/parjob-capacity-partitioning/spec.json
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
{
|
||||
"feature_name": "parjob-capacity-partitioning",
|
||||
"created_at": "2026-10-09T07:06:43Z",
|
||||
"updated_at": "2026-10-09T07:08:41Z",
|
||||
"language": "en",
|
||||
"phase": "initialized",
|
||||
"approvals": {
|
||||
"requirements": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"design": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
},
|
||||
"tasks": {
|
||||
"generated": false,
|
||||
"approved": false
|
||||
}
|
||||
},
|
||||
"ready_for_implementation": false
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -17666,6 +17666,54 @@
|
|||
"record": ".wolf/bugs/bug-1930.md",
|
||||
"gitea_issue": 355,
|
||||
"gitea_synced_status": "fixed"
|
||||
},
|
||||
{
|
||||
"id": "bug-1931",
|
||||
"status": "open",
|
||||
"status_updated": "2026-10-09",
|
||||
"status_note": "newly logged",
|
||||
"component": "scheduler",
|
||||
"severity": "wrong-result",
|
||||
"timestamp": "2026-10-09T07:23:49Z",
|
||||
"last_seen": "2026-10-09T07:23:49Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "xce_scheduler parjob allocator ignores minTasks: a tasks 2..5 job pends though 2 slots are free\n\n**Observed:** in a `xce_scheduler --sim` session, element `[5]` of an array job requesting `tasks 2..5` pends although its only candidate host, pc13v, has 2 free slots (22 total, 20 in use by elements [1]-[4], 5 tasks each). The decision trace shows that only 5-task partitions are tried:\n```\n[trace] job 0-5-1791529946-0[5] session 23 position 5 level full\n requirement single tasks 2..5 requests mem=409600.000000 (400.00 KB)\n plan 1 hosts 1 tasks per position [5]\n host pc13v rejected (per-host check): resource shortfall\n slots total 22 in-use 20 reserved 0 requested 5\n plan 2 hosts 2 tasks per position [3, 2]\n host pc13v rejected (per-host check): resource shortfall\n slots total 22 in-use 20 reserved 0 requested 3\n plan 3 hosts 3 tasks per position [2, 2, 1]\n host pc13v accepted\n ...\n plans: 5 generated, 5 tried, 0 discarded, 0 not tried; no plan could be placed\n outcome: PEND\n host pc13v: resource slots: NO_RESOURCE\n```\n**Expected:** a job with `minTasks=2, maxTasks=5` is placed with any task count in 2..5 that fits, preferring the largest; here 2 tasks on pc13v (plan `[2]`). If even `minTasks` tasks cannot be placed, the job pends.\n**Impact:** every parallel job submitted with a task range (`minTasks < maxTasks`) is scheduled as if `minTasks == maxTasks`: it pends until `maxTasks` tasks fit, wasting free capacity and delaying the job indefinitely on a busy cluster. Wrong scheduling result, no crash.\n**Found by:** user review of the sim decision trace (2026-10-09).",
|
||||
"file": "src/scheduler/src/plugins/parjob_allocator.c, src/scheduler/src/alloc_runner.c",
|
||||
"root_cause": "TODO: required for fixed/partial",
|
||||
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"parjob",
|
||||
"allocation",
|
||||
"mintasks"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1931.md"
|
||||
},
|
||||
{
|
||||
"id": "bug-1932",
|
||||
"status": "open",
|
||||
"status_updated": "2026-10-09",
|
||||
"status_note": "newly logged",
|
||||
"component": "scheduler",
|
||||
"severity": "wrong-result",
|
||||
"timestamp": "2026-10-09T07:23:52Z",
|
||||
"last_seen": "2026-10-09T07:23:52Z",
|
||||
"occurrences": 1,
|
||||
"error_message": "xce_scheduler unsatisfiable-request cache drops pend reasons on a hit and is unused in sim\n\n**Observed:** in a `xce_scheduler --sim` session, array elements `[5]` and `[6]` have identical requests (`tasks 2..5`, mem 400 KB, same user/queue/project). `[5]` exhausts all 5 plans and pends; `[6]` then repeats the exact same search instead of reusing `[5]`'s verdict:\n```\n[sim] PEND jobid=0-5-1791529946-0[5]\n ...\n plans: 5 generated, 5 tried, 0 discarded, 0 not tried; no plan could be placed\n outcome: PEND\n main reason: none\n host pc13v: resource slots: NO_RESOURCE\n[sim] PEND jobid=0-5-1791529946-0[6]\n ... (same 5 plans, same host verdicts)\n plans: 5 generated, 5 tried, 0 discarded, 0 not tried; no plan could be placed\n```\n**Expected:** a request already proved unsatisfiable in this session pends at once, with the same pend reasons (main and per-host) the original attempt recorded, and the trace says it was skipped because of that earlier verdict. The live scheduler and the simulator should behave alike.\n**Impact:** the defect is in the normal (production) scheduler; the simulator's decision trace is what exposed it. Two defects in the session-level unsatisfiable-resreq cache (`alloc_cache.c`, scheduler-allocation Req 7):\n1. (normal scheduler, primary) see 2 below. (simulator) The simulator never registers the cache hooks (`main.c:527-528` registers them only in normal mode; `scheduler.c:247-256` treats NULL hooks as \"always compute fresh\"), so sim cost and sim decision traces differ from production: each identical pending element pays a full plan/host search.\n2. (normal scheduler) A cache hit reports only the stored main-reason category: `sched_pend_from_cached_reason_()` (`scheduler.c:173-184`) calls `add_job_preason(reasons, category, NULL, NULL)`, which erases host/resource reasons and sets `main_reason`. When the original attempt's main reason is 0 (\"none\") and its real cause is a per-host reason (as here: `pc13v: resource slots: NO_RESOURCE`), every later identical job pends with an empty reason, misleading operators (contrary to Req 7.5, \"report the same reason category\").\nWrong result (missing pend reasons) plus wasted scheduling time; no crash.\n**Found by:** user review of the sim decision trace (2026-10-09); defect 2 found by code review while investigating it.\n\nUPDATE (2026-10-09): per the user, the fix must cover the normal scheduler, not just the simulator: a cache hit must carry the original attempt's full pend reason (main, per-host and per-resource) in production, and the simulator must use the same cache so its traces match production.\n\nUPDATE (2026-10-09): production evidence already existed. `sim_verify.c`'s bug-1845 comment records a real captured `.replay` in which a live cache-hit PEND was recorded as `{main_reason=0, nhost_reasons=0, nres_reasons=0}` while the full computation for the same job gave `{main_reason=0, nres_reasons=2}`: the empty-reason hit of defect 2, on a real deployment.",
|
||||
"file": "src/scheduler/src/scheduler.c, src/scheduler/src/alloc_cache.c, src/scheduler/src/main.c",
|
||||
"root_cause": "Two separate gaps around the session-level unsatisfiable-resreq cache (`alloc_cache.c`, scheduler-allocation Req 7):\n\n1. **Only the main reason was cached.** `sched_make_primary_alloc_()` (`scheduler.c`) populated the cache with `ctx->reason->reasons->main_reason` (an `int`), and a hit went through `sched_pend_from_cached_reason_()`, i.e. `add_job_preason(reasons, category, NULL, NULL)`, whose main-reason branch erases the lists and sets only `main_reason`. The allocators record shortfalls as per-host/per-resource entries and leave `main_reason` at 0, so every hit pended its job with an entirely empty reason. Present since the cache was introduced (scheduler-allocation task 2.2/5.1); affects every normal-mode scheduler.\n2. **The simulator had no cache.** The table lived on `xce_scheduler_t` (`scheduler.h`), which a simulation core does not have, and the scheduler-sim-live-allocation spec made \"simulation isolation\" a non-goal: `sim_loop.c`/`sim.c` registered none of the hooks. So `xce_scheduler --sim` searched every identical pending job again, and its decision traces and pend reasons differed from production. This also forced the main-only `--sim-verify` PEND compare (bug-1845).",
|
||||
"fix": "- **Cache entries hold the full reason.** A cache value is now an owned entry (`cache_entry_t`, `alloc_cache.c`): a deep copy of the exhausted attempt's whole `xce_job_preason_t` (main, per-host, per-resource; strings `strdup`ed via `add_job_preason`, main reason first because that branch erases) plus the job id that proved it. The table's datakiller frees entries on clear/destroy; a failed `key_add` frees the rejected copy. `cache_lookup()` fills a caller-supplied reason; the scheduler looks up into a local and, only on a hit, moves it into `ctx->reason`, so a miss leaves `ctx->reason` exactly as before.\n- **The table lives on the core.** `xce_sched_core_t.unsatisfiable_cache` replaces `xce_scheduler_t.unsatisfiable_cache`; `cache_lookup/insert/clear` and the three hook adapters take `xce_sched_core_t*` with no cast to a driver struct (the round-1 cast is what once clobbered `alt_events_file`). `sched_init()` builds it, `cache_clear()` rebuilds it every session, `xce_sched_core_fini()` destroys it.\n- **Both drivers register the hooks.** `main.c` registers them on the live core as before and passes the same three adapters to the simulator through new `sched_sim_args_t` fields; `sched_sim_driver_start()` sets them on the simulation core (plain field writes, since white-box tests `#include` `sim.c` without `core_session.c`/`alloc_cache.c` and leave them NULL).\n- **Hook signatures** (`core.h`): lookup returns the reason and the source job id; populate takes the reason and job id. `sched_allocator()` (`sched_election.c`) sets `core->alloc_jobid` around `sched_make_allocation()` so the populate knows the job.\n- **Decision trace:** a hit adds a `NO_PLAN` entry with the new reason `XCE_TRACE_NOPLAN_KNOWN_UNSAT` and the source job (`sched_trace_known_unsat()`), rendered as `no plan generated: skipped, same request as <job> proved unsatisfiable earlier this session`; the outcome lines then show the restored reasons. `trace.h` stays at 796 lines.\n- `sim_verify.c` keeps its main-only PEND compare (old recordings still carry main-only hits, and a hit carries the proving job's reasons); its comment, `preason.h`'s, `core.h`'s and the scheduler-sim-live-allocation design.md (superseded note) were updated.\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"sim",
|
||||
"alloc-cache",
|
||||
"pend-reason"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"fix_commits": [],
|
||||
"record": ".wolf/bugs/bug-1932.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
67
.wolf/bugs/bug-1931.md
Normal file
67
.wolf/bugs/bug-1931.md
Normal file
|
|
@ -0,0 +1,67 @@
|
|||
---
|
||||
id: "bug-1931"
|
||||
status: "open"
|
||||
status_updated: "2026-10-09"
|
||||
status_note: "newly logged"
|
||||
component: "scheduler"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-10-09T07:23:49Z"
|
||||
last_seen: "2026-10-09T07:23:49Z"
|
||||
occurrences: 1
|
||||
files: ["src/scheduler/src/plugins/parjob_allocator.c", "src/scheduler/src/alloc_runner.c"]
|
||||
tags: ["scheduler", "parjob", "allocation", "mintasks"]
|
||||
related: []
|
||||
---
|
||||
|
||||
# bug-1931: xce_scheduler parjob allocator ignores minTasks: a tasks 2..5 job pends though 2 slots are free
|
||||
|
||||
## Description
|
||||
**Observed:** in a `xce_scheduler --sim` session, element `[5]` of an array job requesting `tasks 2..5` pends although its only candidate host, pc13v, has 2 free slots (22 total, 20 in use by elements [1]-[4], 5 tasks each). The decision trace shows that only 5-task partitions are tried:
|
||||
```
|
||||
[trace] job 0-5-1791529946-0[5] session 23 position 5 level full
|
||||
requirement single tasks 2..5 requests mem=409600.000000 (400.00 KB)
|
||||
plan 1 hosts 1 tasks per position [5]
|
||||
host pc13v rejected (per-host check): resource shortfall
|
||||
slots total 22 in-use 20 reserved 0 requested 5
|
||||
plan 2 hosts 2 tasks per position [3, 2]
|
||||
host pc13v rejected (per-host check): resource shortfall
|
||||
slots total 22 in-use 20 reserved 0 requested 3
|
||||
plan 3 hosts 3 tasks per position [2, 2, 1]
|
||||
host pc13v accepted
|
||||
...
|
||||
plans: 5 generated, 5 tried, 0 discarded, 0 not tried; no plan could be placed
|
||||
outcome: PEND
|
||||
host pc13v: resource slots: NO_RESOURCE
|
||||
```
|
||||
**Expected:** a job with `minTasks=2, maxTasks=5` is placed with any task count in 2..5 that fits, preferring the largest; here 2 tasks on pc13v (plan `[2]`). If even `minTasks` tasks cannot be placed, the job pends.
|
||||
**Impact:** every parallel job submitted with a task range (`minTasks < maxTasks`) is scheduled as if `minTasks == maxTasks`: it pends until `maxTasks` tasks fit, wasting free capacity and delaying the job indefinitely on a busy cluster. Wrong scheduling result, no crash.
|
||||
**Found by:** user review of the sim decision trace (2026-10-09).
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: not recorded: the failing `xce_scheduler` binary is not in this checkout (no build dir); host ldd reports glibc 2.40. The defect is in allocator logic and does not depend on libc
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: profile not recorded (asan, debug or release unknown; user's sim session); the defect is profile-independent allocator logic
|
||||
- Code: scheduler-framework @ a64d0846-dirty
|
||||
- Deployment: `xce_scheduler --sim` (sim session 23), one simulated worker pc13v (22 slots), queue xce, policy builtin_fcfs
|
||||
|
||||
## Reproduction
|
||||
1. In `xce_scheduler --sim`, set up one worker with 22 slots and submit an array job of at least 5 elements, each with `minTasks=2, maxTasks=5` (any small mem request, e.g. 400 KB).
|
||||
2. `trace on all`, then run a scheduling session: elements [1]-[4] start with 5 tasks each (20 slots used).
|
||||
3. Element [5] pends; its trace lists only partitions of 5 tasks, never `[2]`, `[3]` or `[4]`.
|
||||
4. Code-level check: `parjob_plan_iter_init()`/`parjob_plan_iter_next()` (`src/scheduler/src/plugins/parjob_allocator.c:202-288`) read only `resreq->maxTasks`; no allocator reads `minTasks` (`grep -rn minTasks src/scheduler/src` shows only the trace and cache-fingerprint fields).
|
||||
- Frequency: deterministic
|
||||
- Evidence: decision trace quoted above (from the user's sim session); session not saved
|
||||
## Root cause
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Fix
|
||||
TODO: required for fixed/partial
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
|
||||
## References
|
||||
- Related: bug-1770 (wired real minTasks/maxTasks into the primary resreq; the allocator never consumed minTasks), bug-1932 (same trace; unsatisfiable cache not used in sim)
|
||||
- Docs: .kiro/specs/scheduler-allocation/requirements.md (Req 4 AC 1), .kiro/specs/resreq-maxtasks-wiring/, .kiro/specs/parjob-capacity-partitioning/
|
||||
78
.wolf/bugs/bug-1932.md
Normal file
78
.wolf/bugs/bug-1932.md
Normal file
|
|
@ -0,0 +1,78 @@
|
|||
---
|
||||
id: "bug-1932"
|
||||
status: "open"
|
||||
status_updated: "2026-10-09"
|
||||
status_note: "newly logged"
|
||||
component: "scheduler"
|
||||
severity: "wrong-result"
|
||||
timestamp: "2026-10-09T07:23:52Z"
|
||||
last_seen: "2026-10-09T07:23:52Z"
|
||||
occurrences: 1
|
||||
files: ["src/scheduler/src/scheduler.c", "src/scheduler/src/alloc_cache.c", "src/scheduler/src/main.c"]
|
||||
tags: ["scheduler", "sim", "alloc-cache", "pend-reason"]
|
||||
related: []
|
||||
---
|
||||
|
||||
# bug-1932: xce_scheduler unsatisfiable-request cache drops pend reasons on a hit and is unused in sim
|
||||
|
||||
## Description
|
||||
**Observed:** in a `xce_scheduler --sim` session, array elements `[5]` and `[6]` have identical requests (`tasks 2..5`, mem 400 KB, same user/queue/project). `[5]` exhausts all 5 plans and pends; `[6]` then repeats the exact same search instead of reusing `[5]`'s verdict:
|
||||
```
|
||||
[sim] PEND jobid=0-5-1791529946-0[5]
|
||||
...
|
||||
plans: 5 generated, 5 tried, 0 discarded, 0 not tried; no plan could be placed
|
||||
outcome: PEND
|
||||
main reason: none
|
||||
host pc13v: resource slots: NO_RESOURCE
|
||||
[sim] PEND jobid=0-5-1791529946-0[6]
|
||||
... (same 5 plans, same host verdicts)
|
||||
plans: 5 generated, 5 tried, 0 discarded, 0 not tried; no plan could be placed
|
||||
```
|
||||
**Expected:** a request already proved unsatisfiable in this session pends at once, with the same pend reasons (main and per-host) the original attempt recorded, and the trace says it was skipped because of that earlier verdict. The live scheduler and the simulator should behave alike.
|
||||
**Impact:** the defect is in the normal (production) scheduler; the simulator's decision trace is what exposed it. Two defects in the session-level unsatisfiable-resreq cache (`alloc_cache.c`, scheduler-allocation Req 7):
|
||||
1. (normal scheduler, primary) see 2 below. (simulator) The simulator never registers the cache hooks (`main.c:527-528` registers them only in normal mode; `scheduler.c:247-256` treats NULL hooks as "always compute fresh"), so sim cost and sim decision traces differ from production: each identical pending element pays a full plan/host search.
|
||||
2. (normal scheduler) A cache hit reports only the stored main-reason category: `sched_pend_from_cached_reason_()` (`scheduler.c:173-184`) calls `add_job_preason(reasons, category, NULL, NULL)`, which erases host/resource reasons and sets `main_reason`. When the original attempt's main reason is 0 ("none") and its real cause is a per-host reason (as here: `pc13v: resource slots: NO_RESOURCE`), every later identical job pends with an empty reason, misleading operators (contrary to Req 7.5, "report the same reason category").
|
||||
Wrong result (missing pend reasons) plus wasted scheduling time; no crash.
|
||||
**Found by:** user review of the sim decision trace (2026-10-09); defect 2 found by code review while investigating it.
|
||||
|
||||
UPDATE (2026-10-09): per the user, the fix must cover the normal scheduler, not just the simulator: a cache hit must carry the original attempt's full pend reason (main, per-host and per-resource) in production, and the simulator must use the same cache so its traces match production.
|
||||
|
||||
UPDATE (2026-10-09): production evidence already existed. `sim_verify.c`'s bug-1845 comment records a real captured `.replay` in which a live cache-hit PEND was recorded as `{main_reason=0, nhost_reasons=0, nres_reasons=0}` while the full computation for the same job gave `{main_reason=0, nres_reasons=2}`: the empty-reason hit of defect 2, on a real deployment.
|
||||
## Environment
|
||||
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
|
||||
- libc: not recorded: the failing `xce_scheduler` binary is not in this checkout (no build dir); host ldd reports glibc 2.40. The defect is in scheduler logic and does not depend on libc
|
||||
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
|
||||
- Build: profile not recorded (asan, debug or release unknown; user's sim session); the defect is profile-independent scheduler logic
|
||||
- Code: scheduler-framework @ a64d0846-dirty
|
||||
- Deployment: `xce_scheduler --sim` (sim session 23), one simulated worker pc13v (22 slots), queue xce, policy builtin_fcfs; defect 2 is in the normal-mode scheduler (any deployment with two or more pending jobs sharing a request, user, queue and project in one session)
|
||||
|
||||
## Reproduction
|
||||
1. In `xce_scheduler --sim`, set up one worker with 22 slots and submit an array job of at least 6 elements with identical requests (e.g. `minTasks=2, maxTasks=5`, 400 KB mem).
|
||||
2. `trace on all`, then run one scheduling session: elements [1]-[4] start (20 slots used).
|
||||
3. Elements [5] and [6] both pend, and the trace shows a full 5-plan search for each (defect 1).
|
||||
4. Defect 2 (live, code-level): a cache hit goes through `sched_pend_from_cached_reason_()` (`src/scheduler/src/scheduler.c:173-184`) → `add_job_preason(reasons, category, NULL, NULL)` (`src/lib/preason.c:146-163`), whose main-reason branch calls `erase_job_preason()` and stores only `category`; with `category == 0` the second job's pend reason is empty. A unit test with the hooks registered and a host-only original reason shows it.
|
||||
- Frequency: deterministic
|
||||
- Evidence: decision trace quoted above (from the user's sim session); session not saved
|
||||
|
||||
## Root cause
|
||||
Two separate gaps around the session-level unsatisfiable-resreq cache (`alloc_cache.c`, scheduler-allocation Req 7):
|
||||
|
||||
1. **Only the main reason was cached.** `sched_make_primary_alloc_()` (`scheduler.c`) populated the cache with `ctx->reason->reasons->main_reason` (an `int`), and a hit went through `sched_pend_from_cached_reason_()`, i.e. `add_job_preason(reasons, category, NULL, NULL)`, whose main-reason branch erases the lists and sets only `main_reason`. The allocators record shortfalls as per-host/per-resource entries and leave `main_reason` at 0, so every hit pended its job with an entirely empty reason. Present since the cache was introduced (scheduler-allocation task 2.2/5.1); affects every normal-mode scheduler.
|
||||
2. **The simulator had no cache.** The table lived on `xce_scheduler_t` (`scheduler.h`), which a simulation core does not have, and the scheduler-sim-live-allocation spec made "simulation isolation" a non-goal: `sim_loop.c`/`sim.c` registered none of the hooks. So `xce_scheduler --sim` searched every identical pending job again, and its decision traces and pend reasons differed from production. This also forced the main-only `--sim-verify` PEND compare (bug-1845).
|
||||
|
||||
## Fix
|
||||
- **Cache entries hold the full reason.** A cache value is now an owned entry (`cache_entry_t`, `alloc_cache.c`): a deep copy of the exhausted attempt's whole `xce_job_preason_t` (main, per-host, per-resource; strings `strdup`ed via `add_job_preason`, main reason first because that branch erases) plus the job id that proved it. The table's datakiller frees entries on clear/destroy; a failed `key_add` frees the rejected copy. `cache_lookup()` fills a caller-supplied reason; the scheduler looks up into a local and, only on a hit, moves it into `ctx->reason`, so a miss leaves `ctx->reason` exactly as before.
|
||||
- **The table lives on the core.** `xce_sched_core_t.unsatisfiable_cache` replaces `xce_scheduler_t.unsatisfiable_cache`; `cache_lookup/insert/clear` and the three hook adapters take `xce_sched_core_t*` with no cast to a driver struct (the round-1 cast is what once clobbered `alt_events_file`). `sched_init()` builds it, `cache_clear()` rebuilds it every session, `xce_sched_core_fini()` destroys it.
|
||||
- **Both drivers register the hooks.** `main.c` registers them on the live core as before and passes the same three adapters to the simulator through new `sched_sim_args_t` fields; `sched_sim_driver_start()` sets them on the simulation core (plain field writes, since white-box tests `#include` `sim.c` without `core_session.c`/`alloc_cache.c` and leave them NULL).
|
||||
- **Hook signatures** (`core.h`): lookup returns the reason and the source job id; populate takes the reason and job id. `sched_allocator()` (`sched_election.c`) sets `core->alloc_jobid` around `sched_make_allocation()` so the populate knows the job.
|
||||
- **Decision trace:** a hit adds a `NO_PLAN` entry with the new reason `XCE_TRACE_NOPLAN_KNOWN_UNSAT` and the source job (`sched_trace_known_unsat()`), rendered as `no plan generated: skipped, same request as <job> proved unsatisfiable earlier this session`; the outcome lines then show the restored reasons. `trace.h` stays at 796 lines.
|
||||
- `sim_verify.c` keeps its main-only PEND compare (old recordings still carry main-only hits, and a hit carries the proving job's reasons); its comment, `preason.h`'s, `core.h`'s and the scheduler-sim-live-allocation design.md (superseded note) were updated.
|
||||
|
||||
## Verification
|
||||
- Test: TODO: name + file; RED before the fix, GREEN after
|
||||
- Suites: TODO: suite results and build profile
|
||||
- Live: TODO: soak/deploy run, or n/a and why
|
||||
|
||||
## References
|
||||
- Related: bug-1931 (same trace; parjob ignores minTasks), bug-1927 (pend reasons recorded once per job)
|
||||
- Docs: .kiro/specs/scheduler-allocation/requirements.md (Req 7), .kiro/specs/scheduler-sim-live-allocation/ (task 2.2, cache callback seam)
|
||||
Loading…
Add table
Add a link
Reference in a new issue