scheduler audit P1: picker hook validation + rejection WARNs + sim bootstrap by name

P1-1: sched_resolve_job_picker() now validates all nine MUST hooks --
the four state-change hooks (on_queue_update/on_job_push/
on_job_state_change/on_job_update) were called unconditionally by
core_events.c, scheduler.c and snapshot_load.c but never validated at
resolution time, so a picker omitting one segfaulted on its first event
instead of failing startup validation with -EINVAL. Doc comment on
core.h's declaration updated to match.

P1-2: handle the picker hooks' return values instead of dropping them.
The known real producer of a rejection is builtin_fcfs's "queue not
(yet) in the candidate registry" -EINVAL: the job stayed tracked in
core->jobs but was never a scheduling candidate, with no log at all --
invisible forever (no dispatch, no PEND reason, nothing). Five sites in
core_events.c now WARN on rc < 0: the JOBS_PUSH fresh-job site plus a
root-cause WARN in core_upsert_queue (a rejected queue makes every job
in it fail too), both JOBS_PUSH resync-merge on_job_state_change sites,
JOB_STATE_CHANGE and JOB_UPDATE. All non-fatal: the push itself still
succeeds and the job stays tracked.

P1-3: sim_bootstrap_picker() is now name-driven with an explicit WARN
and graceful degrade for any picker name that has no sim-viable builtin
picker -- including the legacy "xce_sched_fcfs", whose job_elect_fn
reads plugin->sched (an xce_scheduler_t* the sim driver never has, and
which plugin_register_core() deliberately leaves NULL with no legacy
wrapper), so bootstrapping it would crash the rebuild walk on
picker_iface == NULL. Previously an unknown name silently returned
while sim still warned downstream. A short experiment with bootstrapping
the legacy plugin via an fcfs_new_plugin() factory was reverted: making
that plugin core-only-viable is a refactor of a plugin slated for
removal (audit P2 #18), not a bootstrap fix.

Tests:
- sched_picker_wiring_test: new test_picker_missing_state_change_hooks_
  fails_with_einval wires the five ORIGINAL MUST hooks and leaves ONLY
  the four state-change hooks NULL, so the asserted -EINVAL can only
  come from the new checks.
- sched_core_events_test: new test_jobs_push_picker_rejection_still_
  tracks_job_and_succeeds proves the push succeeds, the hook fires
  once, and sched_get_job() still resolves the job after a rejection.

P2 items from the same audit are recorded (not fixed) in gaps.md
section 七 (items 18-21): legacy fcfs zombie, dead scaffolding fields,
preemption without a producer, job-priority chain.

134/134 tests pass.

Co-Authored-By: Claude Code <noreply@anthropic.com>
This commit is contained in:
lei.chen 2026-10-22 14:05:38 +00:00
parent c194973d42
commit d16954ad10
8 changed files with 268 additions and 21 deletions

24
gaps.md
View file

@ -5,6 +5,8 @@
> 约定:本文档只做记录,未做任何代码改动。
>
> **2026-10-20 增补**:openmpi 回归(OS openmpi 4.1.6 跨机跑通,见 memory xce-mpi-regression)与全框架评审后新增"五、架构层改进项"(11–17)与"六、构建/分发系统",原有 1–10 维持不变。
>
> **2026-10-22 增补**:调度框架 picker 子系统专项审计(本次 P1 修复见 git log;本节新增审计中**未修**的 P2 项,编号顺延为 18–21,见文末"七")。
---
@ -161,6 +163,28 @@ main_reason 契约在两个代码库各自演化,scheduler-framework 合并时
---
## 七、调度框架 picker 子系统审计遗留项(2026-10-22 增补,P2 未修)
> 来源:调度框架专项审计("作业调度除了 FCFS 还有哪些?调度框架本身有什么要修补的?")。P1 三项已在本次修复:sched_resolve_job_picker 九钩子全量校验、core_events.c 五处 on_job_push/on_job_state_change/on_job_update/on_queue_update 拒绝 WARN、sim_picker.c 按名字 bootstrap(仅 builtin_fcfs;legacy "xce_sched_fcfs" 明确不支持——其 job_elect_fn 读 plugin->sched,sim 无 xce_scheduler_t 可给,见 sim_picker.c 文件头)。以下为 P2,**只记录未动**。
### 18. legacy fcfs 插件(僵尸)——与"四"中"legacy fcfs 插件仍注册"同一主体
- register_fcfs() 每次启动注册 "xce_sched_fcfs"(deprecated job_elect_fn 钩子),注册后不选中就无用,选中则走 fcfs_elect_job 的 sleep(1) 死循环。
- 本次审计确认它还有一层问题:插件只认 plugin->sched(xce_scheduler_t*),plugin_register_core() 不做 legacy 包装也不设 sched——sim 模式/任何 core-only 调用方都无法安全运行它。sim_picker.c 的 bootstrap 因此明确拒绝该名字。
- **处置建议**:builtin_fcfs 已完整覆盖 FCFS 语义,可直接删 register_fcfs() 启动注册与整个 fcfs.c(连带 plugin.c 的 legacy_picker_wrapper_iface 与 sched_election.c 的 job_elect_fn 委托分支一起收缩)。
### 19. 死脚手架字段:queue->job_pick_plugin / queue->project_name / queue->resreq
- core_build_queue()/make_sched_queue() 都把这些字段显式置 NULL(core_events.c:385-387、schedobj.c),注释自认"无 wire-config 来源,后续任务接线"。
- 全库无生产者无消费者(仅测试断言 NULL)。builtin_fcfs 的队列级 per-queue picker 定制(如果有此需求)就挂在这个字段上。
- **处置建议**:要么接 config(queue conf 加字段一路穿到 wire schema),要么删字段。留着会让下一个读代码的人以为 per-queue picker 已可用。
### 20. 抢占决策类型有实现无生产者(即"一、2"同一条,从 picker 框架角度再记一笔)
- make_preempt_decision()(decision.c:35)+ XCE_DECISION_PREEMPT 类型链路完整,仅 testsuite 调用。picker 的 iface 里也没有任何"抢占回调"设计位——真做抢占时 picker/allocator 契约需要扩。
### 21. 作业级优先级(即"一、1",审计复核确认范围不变)
- job_submit_data_t 无 priority 字段(types.h)、convert.go modelJobToSubmit 丢弃 JobPriority、core_build_job 硬编码 job->priority=0(core_events.c:353 FIXME)、builtin_fcfs 的 priority 降序比较器因此是死代码。修复面=wire schema + Go API 提交路径 + core_events/schedobj 填充 + (可选)replay/snapshot 携带 priority,四段都要动。
---
## 附:排查方法论备忘(可复用)
- **gdb 活体结构遍历**:`static xce_scheduler_t sched`(main.c:202)二进制缺完整 DWARF 时,按裸地址 `x/64gx` dump,手工映射 kvmgr {key_type@0, nslots@4, kvtab@8, data@16} 与 xzarray_t {count@0, caps@8, delta@16, pos@24, data@32, killer@40}(zarray.c:392 实际布局,非 include/zarray.h 旧布局),用已知常量(queues nslots=59、workers 5381、delta=64)校验。