docs: log bug-1931/1932, init parjob-capacity-partitioning spec

- bug-1931: the parjob allocator ignores minTasks (a tasks 2..5 job pends
  although 2 slots are free).
- bug-1932: the unsatisfiable-request cache drops pend reasons on a hit
  and is unused in the simulator.
- .kiro/specs/parjob-capacity-partitioning: initialized spec with the
  candidate heuristics for capacity-driven plan generation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-09 08:02:00 +00:00
parent a64d0846c7
commit cff3304f5d
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
5 changed files with 270 additions and 0 deletions

View file

@ -17666,6 +17666,54 @@
"record": ".wolf/bugs/bug-1930.md",
"gitea_issue": 355,
"gitea_synced_status": "fixed"
},
{
"id": "bug-1931",
"status": "open",
"status_updated": "2026-10-09",
"status_note": "newly logged",
"component": "scheduler",
"severity": "wrong-result",
"timestamp": "2026-10-09T07:23:49Z",
"last_seen": "2026-10-09T07:23:49Z",
"occurrences": 1,
"error_message": "xce_scheduler parjob allocator ignores minTasks: a tasks 2..5 job pends though 2 slots are free\n\n**Observed:** in a `xce_scheduler --sim` session, element `[5]` of an array job requesting `tasks 2..5` pends although its only candidate host, pc13v, has 2 free slots (22 total, 20 in use by elements [1]-[4], 5 tasks each). The decision trace shows that only 5-task partitions are tried:\n```\n[trace] job 0-5-1791529946-0[5] session 23 position 5 level full\n requirement single tasks 2..5 requests mem=409600.000000 (400.00 KB)\n plan 1 hosts 1 tasks per position [5]\n host pc13v rejected (per-host check): resource shortfall\n slots total 22 in-use 20 reserved 0 requested 5\n plan 2 hosts 2 tasks per position [3, 2]\n host pc13v rejected (per-host check): resource shortfall\n slots total 22 in-use 20 reserved 0 requested 3\n plan 3 hosts 3 tasks per position [2, 2, 1]\n host pc13v accepted\n ...\n plans: 5 generated, 5 tried, 0 discarded, 0 not tried; no plan could be placed\n outcome: PEND\n host pc13v: resource slots: NO_RESOURCE\n```\n**Expected:** a job with `minTasks=2, maxTasks=5` is placed with any task count in 2..5 that fits, preferring the largest; here 2 tasks on pc13v (plan `[2]`). If even `minTasks` tasks cannot be placed, the job pends.\n**Impact:** every parallel job submitted with a task range (`minTasks < maxTasks`) is scheduled as if `minTasks == maxTasks`: it pends until `maxTasks` tasks fit, wasting free capacity and delaying the job indefinitely on a busy cluster. Wrong scheduling result, no crash.\n**Found by:** user review of the sim decision trace (2026-10-09).",
"file": "src/scheduler/src/plugins/parjob_allocator.c, src/scheduler/src/alloc_runner.c",
"root_cause": "TODO: required for fixed/partial",
"fix": "TODO: required for fixed/partial\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
"tags": [
"scheduler",
"parjob",
"allocation",
"mintasks"
],
"related_bugs": [],
"fix_commits": [],
"record": ".wolf/bugs/bug-1931.md"
},
{
"id": "bug-1932",
"status": "open",
"status_updated": "2026-10-09",
"status_note": "newly logged",
"component": "scheduler",
"severity": "wrong-result",
"timestamp": "2026-10-09T07:23:52Z",
"last_seen": "2026-10-09T07:23:52Z",
"occurrences": 1,
"error_message": "xce_scheduler unsatisfiable-request cache drops pend reasons on a hit and is unused in sim\n\n**Observed:** in a `xce_scheduler --sim` session, array elements `[5]` and `[6]` have identical requests (`tasks 2..5`, mem 400 KB, same user/queue/project). `[5]` exhausts all 5 plans and pends; `[6]` then repeats the exact same search instead of reusing `[5]`'s verdict:\n```\n[sim] PEND jobid=0-5-1791529946-0[5]\n ...\n plans: 5 generated, 5 tried, 0 discarded, 0 not tried; no plan could be placed\n outcome: PEND\n main reason: none\n host pc13v: resource slots: NO_RESOURCE\n[sim] PEND jobid=0-5-1791529946-0[6]\n ... (same 5 plans, same host verdicts)\n plans: 5 generated, 5 tried, 0 discarded, 0 not tried; no plan could be placed\n```\n**Expected:** a request already proved unsatisfiable in this session pends at once, with the same pend reasons (main and per-host) the original attempt recorded, and the trace says it was skipped because of that earlier verdict. The live scheduler and the simulator should behave alike.\n**Impact:** the defect is in the normal (production) scheduler; the simulator's decision trace is what exposed it. Two defects in the session-level unsatisfiable-resreq cache (`alloc_cache.c`, scheduler-allocation Req 7):\n1. (normal scheduler, primary) see 2 below. (simulator) The simulator never registers the cache hooks (`main.c:527-528` registers them only in normal mode; `scheduler.c:247-256` treats NULL hooks as \"always compute fresh\"), so sim cost and sim decision traces differ from production: each identical pending element pays a full plan/host search.\n2. (normal scheduler) A cache hit reports only the stored main-reason category: `sched_pend_from_cached_reason_()` (`scheduler.c:173-184`) calls `add_job_preason(reasons, category, NULL, NULL)`, which erases host/resource reasons and sets `main_reason`. When the original attempt's main reason is 0 (\"none\") and its real cause is a per-host reason (as here: `pc13v: resource slots: NO_RESOURCE`), every later identical job pends with an empty reason, misleading operators (contrary to Req 7.5, \"report the same reason category\").\nWrong result (missing pend reasons) plus wasted scheduling time; no crash.\n**Found by:** user review of the sim decision trace (2026-10-09); defect 2 found by code review while investigating it.\n\nUPDATE (2026-10-09): per the user, the fix must cover the normal scheduler, not just the simulator: a cache hit must carry the original attempt's full pend reason (main, per-host and per-resource) in production, and the simulator must use the same cache so its traces match production.\n\nUPDATE (2026-10-09): production evidence already existed. `sim_verify.c`'s bug-1845 comment records a real captured `.replay` in which a live cache-hit PEND was recorded as `{main_reason=0, nhost_reasons=0, nres_reasons=0}` while the full computation for the same job gave `{main_reason=0, nres_reasons=2}`: the empty-reason hit of defect 2, on a real deployment.",
"file": "src/scheduler/src/scheduler.c, src/scheduler/src/alloc_cache.c, src/scheduler/src/main.c",
"root_cause": "Two separate gaps around the session-level unsatisfiable-resreq cache (`alloc_cache.c`, scheduler-allocation Req 7):\n\n1. **Only the main reason was cached.** `sched_make_primary_alloc_()` (`scheduler.c`) populated the cache with `ctx->reason->reasons->main_reason` (an `int`), and a hit went through `sched_pend_from_cached_reason_()`, i.e. `add_job_preason(reasons, category, NULL, NULL)`, whose main-reason branch erases the lists and sets only `main_reason`. The allocators record shortfalls as per-host/per-resource entries and leave `main_reason` at 0, so every hit pended its job with an entirely empty reason. Present since the cache was introduced (scheduler-allocation task 2.2/5.1); affects every normal-mode scheduler.\n2. **The simulator had no cache.** The table lived on `xce_scheduler_t` (`scheduler.h`), which a simulation core does not have, and the scheduler-sim-live-allocation spec made \"simulation isolation\" a non-goal: `sim_loop.c`/`sim.c` registered none of the hooks. So `xce_scheduler --sim` searched every identical pending job again, and its decision traces and pend reasons differed from production. This also forced the main-only `--sim-verify` PEND compare (bug-1845).",
"fix": "- **Cache entries hold the full reason.** A cache value is now an owned entry (`cache_entry_t`, `alloc_cache.c`): a deep copy of the exhausted attempt's whole `xce_job_preason_t` (main, per-host, per-resource; strings `strdup`ed via `add_job_preason`, main reason first because that branch erases) plus the job id that proved it. The table's datakiller frees entries on clear/destroy; a failed `key_add` frees the rejected copy. `cache_lookup()` fills a caller-supplied reason; the scheduler looks up into a local and, only on a hit, moves it into `ctx->reason`, so a miss leaves `ctx->reason` exactly as before.\n- **The table lives on the core.** `xce_sched_core_t.unsatisfiable_cache` replaces `xce_scheduler_t.unsatisfiable_cache`; `cache_lookup/insert/clear` and the three hook adapters take `xce_sched_core_t*` with no cast to a driver struct (the round-1 cast is what once clobbered `alt_events_file`). `sched_init()` builds it, `cache_clear()` rebuilds it every session, `xce_sched_core_fini()` destroys it.\n- **Both drivers register the hooks.** `main.c` registers them on the live core as before and passes the same three adapters to the simulator through new `sched_sim_args_t` fields; `sched_sim_driver_start()` sets them on the simulation core (plain field writes, since white-box tests `#include` `sim.c` without `core_session.c`/`alloc_cache.c` and leave them NULL).\n- **Hook signatures** (`core.h`): lookup returns the reason and the source job id; populate takes the reason and job id. `sched_allocator()` (`sched_election.c`) sets `core->alloc_jobid` around `sched_make_allocation()` so the populate knows the job.\n- **Decision trace:** a hit adds a `NO_PLAN` entry with the new reason `XCE_TRACE_NOPLAN_KNOWN_UNSAT` and the source job (`sched_trace_known_unsat()`), rendered as `no plan generated: skipped, same request as <job> proved unsatisfiable earlier this session`; the outcome lines then show the restored reasons. `trace.h` stays at 796 lines.\n- `sim_verify.c` keeps its main-only PEND compare (old recordings still carry main-only hits, and a hit carries the proving job's reasons); its comment, `preason.h`'s, `core.h`'s and the scheduler-sim-live-allocation design.md (superseded note) were updated.\n\nVerification:\n- Test: TODO: name + file; RED before the fix, GREEN after\n- Suites: TODO: suite results and build profile\n- Live: TODO: soak/deploy run, or n/a and why",
"tags": [
"scheduler",
"sim",
"alloc-cache",
"pend-reason"
],
"related_bugs": [],
"fix_commits": [],
"record": ".wolf/bugs/bug-1932.md"
}
]
}

67
.wolf/bugs/bug-1931.md Normal file
View file

@ -0,0 +1,67 @@
---
id: "bug-1931"
status: "open"
status_updated: "2026-10-09"
status_note: "newly logged"
component: "scheduler"
severity: "wrong-result"
timestamp: "2026-10-09T07:23:49Z"
last_seen: "2026-10-09T07:23:49Z"
occurrences: 1
files: ["src/scheduler/src/plugins/parjob_allocator.c", "src/scheduler/src/alloc_runner.c"]
tags: ["scheduler", "parjob", "allocation", "mintasks"]
related: []
---
# bug-1931: xce_scheduler parjob allocator ignores minTasks: a tasks 2..5 job pends though 2 slots are free
## Description
**Observed:** in a `xce_scheduler --sim` session, element `[5]` of an array job requesting `tasks 2..5` pends although its only candidate host, pc13v, has 2 free slots (22 total, 20 in use by elements [1]-[4], 5 tasks each). The decision trace shows that only 5-task partitions are tried:
```
[trace] job 0-5-1791529946-0[5] session 23 position 5 level full
requirement single tasks 2..5 requests mem=409600.000000 (400.00 KB)
plan 1 hosts 1 tasks per position [5]
host pc13v rejected (per-host check): resource shortfall
slots total 22 in-use 20 reserved 0 requested 5
plan 2 hosts 2 tasks per position [3, 2]
host pc13v rejected (per-host check): resource shortfall
slots total 22 in-use 20 reserved 0 requested 3
plan 3 hosts 3 tasks per position [2, 2, 1]
host pc13v accepted
...
plans: 5 generated, 5 tried, 0 discarded, 0 not tried; no plan could be placed
outcome: PEND
host pc13v: resource slots: NO_RESOURCE
```
**Expected:** a job with `minTasks=2, maxTasks=5` is placed with any task count in 2..5 that fits, preferring the largest; here 2 tasks on pc13v (plan `[2]`). If even `minTasks` tasks cannot be placed, the job pends.
**Impact:** every parallel job submitted with a task range (`minTasks < maxTasks`) is scheduled as if `minTasks == maxTasks`: it pends until `maxTasks` tasks fit, wasting free capacity and delaying the job indefinitely on a busy cluster. Wrong scheduling result, no crash.
**Found by:** user review of the sim decision trace (2026-10-09).
## Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: not recorded: the failing `xce_scheduler` binary is not in this checkout (no build dir); host ldd reports glibc 2.40. The defect is in allocator logic and does not depend on libc
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: profile not recorded (asan, debug or release unknown; user's sim session); the defect is profile-independent allocator logic
- Code: scheduler-framework @ a64d0846-dirty
- Deployment: `xce_scheduler --sim` (sim session 23), one simulated worker pc13v (22 slots), queue xce, policy builtin_fcfs
## Reproduction
1. In `xce_scheduler --sim`, set up one worker with 22 slots and submit an array job of at least 5 elements, each with `minTasks=2, maxTasks=5` (any small mem request, e.g. 400 KB).
2. `trace on all`, then run a scheduling session: elements [1]-[4] start with 5 tasks each (20 slots used).
3. Element [5] pends; its trace lists only partitions of 5 tasks, never `[2]`, `[3]` or `[4]`.
4. Code-level check: `parjob_plan_iter_init()`/`parjob_plan_iter_next()` (`src/scheduler/src/plugins/parjob_allocator.c:202-288`) read only `resreq->maxTasks`; no allocator reads `minTasks` (`grep -rn minTasks src/scheduler/src` shows only the trace and cache-fingerprint fields).
- Frequency: deterministic
- Evidence: decision trace quoted above (from the user's sim session); session not saved
## Root cause
TODO: required for fixed/partial
## Fix
TODO: required for fixed/partial
## Verification
- Test: TODO: name + file; RED before the fix, GREEN after
- Suites: TODO: suite results and build profile
- Live: TODO: soak/deploy run, or n/a and why
## References
- Related: bug-1770 (wired real minTasks/maxTasks into the primary resreq; the allocator never consumed minTasks), bug-1932 (same trace; unsatisfiable cache not used in sim)
- Docs: .kiro/specs/scheduler-allocation/requirements.md (Req 4 AC 1), .kiro/specs/resreq-maxtasks-wiring/, .kiro/specs/parjob-capacity-partitioning/

78
.wolf/bugs/bug-1932.md Normal file
View file

@ -0,0 +1,78 @@
---
id: "bug-1932"
status: "open"
status_updated: "2026-10-09"
status_note: "newly logged"
component: "scheduler"
severity: "wrong-result"
timestamp: "2026-10-09T07:23:52Z"
last_seen: "2026-10-09T07:23:52Z"
occurrences: 1
files: ["src/scheduler/src/scheduler.c", "src/scheduler/src/alloc_cache.c", "src/scheduler/src/main.c"]
tags: ["scheduler", "sim", "alloc-cache", "pend-reason"]
related: []
---
# bug-1932: xce_scheduler unsatisfiable-request cache drops pend reasons on a hit and is unused in sim
## Description
**Observed:** in a `xce_scheduler --sim` session, array elements `[5]` and `[6]` have identical requests (`tasks 2..5`, mem 400 KB, same user/queue/project). `[5]` exhausts all 5 plans and pends; `[6]` then repeats the exact same search instead of reusing `[5]`'s verdict:
```
[sim] PEND jobid=0-5-1791529946-0[5]
...
plans: 5 generated, 5 tried, 0 discarded, 0 not tried; no plan could be placed
outcome: PEND
main reason: none
host pc13v: resource slots: NO_RESOURCE
[sim] PEND jobid=0-5-1791529946-0[6]
... (same 5 plans, same host verdicts)
plans: 5 generated, 5 tried, 0 discarded, 0 not tried; no plan could be placed
```
**Expected:** a request already proved unsatisfiable in this session pends at once, with the same pend reasons (main and per-host) the original attempt recorded, and the trace says it was skipped because of that earlier verdict. The live scheduler and the simulator should behave alike.
**Impact:** the defect is in the normal (production) scheduler; the simulator's decision trace is what exposed it. Two defects in the session-level unsatisfiable-resreq cache (`alloc_cache.c`, scheduler-allocation Req 7):
1. (normal scheduler, primary) see 2 below. (simulator) The simulator never registers the cache hooks (`main.c:527-528` registers them only in normal mode; `scheduler.c:247-256` treats NULL hooks as "always compute fresh"), so sim cost and sim decision traces differ from production: each identical pending element pays a full plan/host search.
2. (normal scheduler) A cache hit reports only the stored main-reason category: `sched_pend_from_cached_reason_()` (`scheduler.c:173-184`) calls `add_job_preason(reasons, category, NULL, NULL)`, which erases host/resource reasons and sets `main_reason`. When the original attempt's main reason is 0 ("none") and its real cause is a per-host reason (as here: `pc13v: resource slots: NO_RESOURCE`), every later identical job pends with an empty reason, misleading operators (contrary to Req 7.5, "report the same reason category").
Wrong result (missing pend reasons) plus wasted scheduling time; no crash.
**Found by:** user review of the sim decision trace (2026-10-09); defect 2 found by code review while investigating it.
UPDATE (2026-10-09): per the user, the fix must cover the normal scheduler, not just the simulator: a cache hit must carry the original attempt's full pend reason (main, per-host and per-resource) in production, and the simulator must use the same cache so its traces match production.
UPDATE (2026-10-09): production evidence already existed. `sim_verify.c`'s bug-1845 comment records a real captured `.replay` in which a live cache-hit PEND was recorded as `{main_reason=0, nhost_reasons=0, nres_reasons=0}` while the full computation for the same job gave `{main_reason=0, nres_reasons=2}`: the empty-reason hit of defect 2, on a real deployment.
## Environment
- OS: no /etc/os-release; Docker container; NixOS kernel; kernel Linux 6.18.33 x86_64
- libc: not recorded: the failing `xce_scheduler` binary is not in this checkout (no build dir); host ldd reports glibc 2.40. The defect is in scheduler logic and does not depend on libc
- Toolchain: gcc (GCC) 15.2.0, meson 1.9.1
- Build: profile not recorded (asan, debug or release unknown; user's sim session); the defect is profile-independent scheduler logic
- Code: scheduler-framework @ a64d0846-dirty
- Deployment: `xce_scheduler --sim` (sim session 23), one simulated worker pc13v (22 slots), queue xce, policy builtin_fcfs; defect 2 is in the normal-mode scheduler (any deployment with two or more pending jobs sharing a request, user, queue and project in one session)
## Reproduction
1. In `xce_scheduler --sim`, set up one worker with 22 slots and submit an array job of at least 6 elements with identical requests (e.g. `minTasks=2, maxTasks=5`, 400 KB mem).
2. `trace on all`, then run one scheduling session: elements [1]-[4] start (20 slots used).
3. Elements [5] and [6] both pend, and the trace shows a full 5-plan search for each (defect 1).
4. Defect 2 (live, code-level): a cache hit goes through `sched_pend_from_cached_reason_()` (`src/scheduler/src/scheduler.c:173-184`) → `add_job_preason(reasons, category, NULL, NULL)` (`src/lib/preason.c:146-163`), whose main-reason branch calls `erase_job_preason()` and stores only `category`; with `category == 0` the second job's pend reason is empty. A unit test with the hooks registered and a host-only original reason shows it.
- Frequency: deterministic
- Evidence: decision trace quoted above (from the user's sim session); session not saved
## Root cause
Two separate gaps around the session-level unsatisfiable-resreq cache (`alloc_cache.c`, scheduler-allocation Req 7):
1. **Only the main reason was cached.** `sched_make_primary_alloc_()` (`scheduler.c`) populated the cache with `ctx->reason->reasons->main_reason` (an `int`), and a hit went through `sched_pend_from_cached_reason_()`, i.e. `add_job_preason(reasons, category, NULL, NULL)`, whose main-reason branch erases the lists and sets only `main_reason`. The allocators record shortfalls as per-host/per-resource entries and leave `main_reason` at 0, so every hit pended its job with an entirely empty reason. Present since the cache was introduced (scheduler-allocation task 2.2/5.1); affects every normal-mode scheduler.
2. **The simulator had no cache.** The table lived on `xce_scheduler_t` (`scheduler.h`), which a simulation core does not have, and the scheduler-sim-live-allocation spec made "simulation isolation" a non-goal: `sim_loop.c`/`sim.c` registered none of the hooks. So `xce_scheduler --sim` searched every identical pending job again, and its decision traces and pend reasons differed from production. This also forced the main-only `--sim-verify` PEND compare (bug-1845).
## Fix
- **Cache entries hold the full reason.** A cache value is now an owned entry (`cache_entry_t`, `alloc_cache.c`): a deep copy of the exhausted attempt's whole `xce_job_preason_t` (main, per-host, per-resource; strings `strdup`ed via `add_job_preason`, main reason first because that branch erases) plus the job id that proved it. The table's datakiller frees entries on clear/destroy; a failed `key_add` frees the rejected copy. `cache_lookup()` fills a caller-supplied reason; the scheduler looks up into a local and, only on a hit, moves it into `ctx->reason`, so a miss leaves `ctx->reason` exactly as before.
- **The table lives on the core.** `xce_sched_core_t.unsatisfiable_cache` replaces `xce_scheduler_t.unsatisfiable_cache`; `cache_lookup/insert/clear` and the three hook adapters take `xce_sched_core_t*` with no cast to a driver struct (the round-1 cast is what once clobbered `alt_events_file`). `sched_init()` builds it, `cache_clear()` rebuilds it every session, `xce_sched_core_fini()` destroys it.
- **Both drivers register the hooks.** `main.c` registers them on the live core as before and passes the same three adapters to the simulator through new `sched_sim_args_t` fields; `sched_sim_driver_start()` sets them on the simulation core (plain field writes, since white-box tests `#include` `sim.c` without `core_session.c`/`alloc_cache.c` and leave them NULL).
- **Hook signatures** (`core.h`): lookup returns the reason and the source job id; populate takes the reason and job id. `sched_allocator()` (`sched_election.c`) sets `core->alloc_jobid` around `sched_make_allocation()` so the populate knows the job.
- **Decision trace:** a hit adds a `NO_PLAN` entry with the new reason `XCE_TRACE_NOPLAN_KNOWN_UNSAT` and the source job (`sched_trace_known_unsat()`), rendered as `no plan generated: skipped, same request as <job> proved unsatisfiable earlier this session`; the outcome lines then show the restored reasons. `trace.h` stays at 796 lines.
- `sim_verify.c` keeps its main-only PEND compare (old recordings still carry main-only hits, and a hit carries the proving job's reasons); its comment, `preason.h`'s, `core.h`'s and the scheduler-sim-live-allocation design.md (superseded note) were updated.
## Verification
- Test: TODO: name + file; RED before the fix, GREEN after
- Suites: TODO: suite results and build profile
- Live: TODO: soak/deploy run, or n/a and why
## References
- Related: bug-1931 (same trace; parjob ignores minTasks), bug-1927 (pend reasons recorded once per job)
- Docs: .kiro/specs/scheduler-allocation/requirements.md (Req 7), .kiro/specs/scheduler-sim-live-allocation/ (task 2.2, cache callback seam)