feat(scheduler): PEND reason classification — main_reason + limit/queue/worker reasons (POC "why is my job waiting")

P0 (done, cluster-verified):
- main_reason was always 0: add_job_preason only set it when BOTH host and res
  were NULL, but the allocator always attaches host/res detail rows. Now the
  first detail add wins as main_reason, so the API's main_reason_name is
  meaningful (was "unknown(0)"). Verified live: PEND jobs now report
  main_reason=3/match_fail + per-host reason_name.
- limit plugin attach: host-based quota hit (hg1_slots_per_host) -> LIMIT_HOST
  + limit name; global quota hit (ug1_slots, ...) -> LIMIT_GLOBAL + limit name.
  Previously both blocks were silent (job just stayed PEND).
- Extended the PREASON_* enum (was 3 codes) with LSF-style classes:
  LIMIT_GLOBAL/LIMIT_HOST/QUEUE_CLOSED/QUEUE_DRAINING/TIME_WINDOW/HOST_UNAVAIL.
  apiserver pendReasonName maps them; host/res detail rows now carry reason_name.

P1 (implemented):
- queue status: sched_queue_is_ready no longer a stub — closed/inactivated
  queues are not electable; the fcfs election attaches QUEUE_CLOSED /
  QUEUE_DRAINING to every PEND job in a skipped queue.
- worker filter: sched_try_alloc_on_host_ skips UNAVAIL/CLOSED workers and
  attaches HOST_UNAVAIL instead of trying to allocate on them.

Unit tests: preason suite updated for the main_reason change + 3 new tests
(limit host/global, first-add-wins); 295 lib tests pass under ASAN.
apiserver backend tests pass.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-08-14 09:43:21 +00:00
parent 62778930e2
commit 4e2ac861f4
10 changed files with 211 additions and 23 deletions

View file

@ -133,7 +133,8 @@ func cJobToModel(job *C.xce_job_t) *models.JobInfo {
hr := C.job_host_reason(pr, C.int(i))
if hr != nil && hr.host != nil {
r.pendReason.HostReasons = append(r.pendReason.HostReasons,
rawHostReason{Host: goString(hr.host), Reason: int(hr.reason)})
rawHostReason{Host: goString(hr.host), Reason: int(hr.reason),
ReasonName: pendReasonName(int(hr.reason))})
}
}
for i := 0; i < int(pr.nres_reasons); i++ {
@ -141,7 +142,8 @@ func cJobToModel(job *C.xce_job_t) *models.JobInfo {
if rr != nil && rr.resource != nil {
r.pendReason.ResReasons = append(r.pendReason.ResReasons,
rawResReason{Resource: goString(rr.resource),
Host: goString(rr.host), Reason: int(rr.reason)})
Host: goString(rr.host), Reason: int(rr.reason),
ReasonName: pendReasonName(int(rr.reason))})
}
}
}