From c90e63ec0fffdeda3886262a70ff1fc03a2cc1f8 Mon Sep 17 00:00:00 2001 From: "lei.chen" Date: Fri, 7 Aug 2026 07:35:51 +0000 Subject: [PATCH] fix(frontend): job control no longer returns HTTP 500 (bug-1490) Two stacked defects in the apiserver job-control link, both of which kept hold/release/cancel from ever reaching the controller: 1. controllers/jobs.go: validateJobControl and foundJob asserted jobinfo.Data["array_idx"].(int64), but StoreToOrderMaps produces array_idx as int (models.JobInfo.ArrayIdx is declared int) -> panic -> Gin Recovery -> HTTP 500 before backend.ControlJob ran. Add normalizeJobInt64 / jobArrayIdx helpers and use them in both functions (job_status already had this fix via normalizeJobStatus; array_idx was missed). 2. backend/clib.go: backend.ControlJob set xce_jobctrl_t.action to the models.JOB_* constant (JOB_STOP=1 / JOB_RESUME=2 / JOB_KILL=3 / JOB_CHKPNT=8), but that field is the capnp JobControl union discriminant (JobControl_u_siginfo=0 / JobControl_u_ckptoption=1). The controller's job_ctrl() selected the wrong union branch (or none, since encode_JobControl drops the union pointer for non-0/1 discriminants) and rejected every control with -EINVAL. Add jobCtrlDiscriminant() and use it. Also: resolveExecUser was defined in the cgo-only convert.go but is exercised by a non-cgo test, so `make test` (CGO_ENABLED=0) failed to build the backend package. Move it to convert_pure.go so it compiles in both modes. Tests: 3 controller regression tests (validateJobControl allowed / not-allowed branches, foundJob array-child branch) + 1 backend jobCtrlDiscriminant test. make test fully green. Cluster E2E verified: hold -> PSUSP, release -> PEND, cancel -> EXIT on a PEND job, no 500, no -EINVAL, all daemons healthy. Co-Authored-By: Claude --- docs/left_scheduler_work.md | 196 ++++++++++++++++++ src/frontend/apiserver/backend/clib.go | 7 +- src/frontend/apiserver/backend/convert.go | 13 -- .../apiserver/backend/convert_pure.go | 30 +++ .../apiserver/backend/convert_test.go | 27 +++ src/frontend/apiserver/controllers/jobs.go | 52 +++-- .../apiserver/controllers/jobs_test.go | 72 +++++++ 7 files changed, 368 insertions(+), 29 deletions(-) create mode 100644 docs/left_scheduler_work.md diff --git a/docs/left_scheduler_work.md b/docs/left_scheduler_work.md new file mode 100644 index 0000000..50ed749 --- /dev/null +++ b/docs/left_scheduler_work.md @@ -0,0 +1,196 @@ +# Scheduler 剩余工作清单（left_scheduler_work） + +- 状态：持续更新 +- 日期：2026-08-07 +- 来源：项目自带 MD 设计文档分析（.kiro/steering、.kiro/specs、docs/、CLAUDE.md） + +本文汇总 scheduler 相关的一切剩余/未完成/已知损坏的工作，按优先级排序。每项标注来源文档与状态。 + +--- + +## 一、最紧急：当前影响集群的 bug + +### 1. bug-1494 — 作业全部 PEND 不调度【根因已定位，2026-08-07】 +- **状态**：未解决（根因已定位） +- **来源**：`CLAUDE.md`、`docs/Scheduler-Agent-设计文档.md` §10.2b +- **根因（实证确认）**：controller 重启后，worker + scheduler **都不能自愈重连**，导致 scheduler 资源视图 slots=0 → 无法派发： + 1. **worker 不重连**：controller 重启后 t1 保持 `UNAVAIL`（NCPUS/MAX_SLOT=-，HOST_TYPE=UNKNOWN），每 15s `Failed to send load indices error -1`，无 rejoin 尝试（worker 连接监控 `checking_connection` 未检测到 controller 重启） + 2. **scheduler 无重连逻辑**：controller 重启后 scheduler 连接断开，重启后重新 join 时看到 t1 UNAVAIL、0 slots（`resource 0 (slots) total 0.000000 shortfall 1.0`） + 3. scheduler 无法分配 → 全部 PEND（`main decision = (nil)`） +- **已排除**：不是 wbuf 污染（bug-1508 修复无效）；不是 resources.yaml（缺 slots 定义但 slots 是内置资源，来自 host config `mxj`/`maxcpus`） +- **关联**：component-failover-adoption（scheduler 重连）+ worker 连接监控健壮性——**这是 P2.5/component-failover-adoption 的核心，尚未实现** +- **临时恢复**：全量重启 daemon（clean restart）可恢复，但集群无自愈能力 + +### 2. bug-1490 — apiserver control HTTP 500【已修复，2026-08-07】 +- **状态**：已修复 +- **来源**：`CLAUDE.md`、`Scheduler-Agent-设计文档.md` §10.2b +- **根因（两个叠加缺陷）**： + 1. **array_idx 类型断言 panic → HTTP 500**：`validateJobControl`/`foundJob` 用 `jobinfo.Data["array_idx"].(int64)`，但 `StoreToOrderMaps` 产出的是 `int`（`JobInfo.ArrayIdx` 声明为 `int`）→ panic → Gin Recovery → HTTP 500 → 请求到不了 controller。 + 2. **jobctrl 联合判别符错误 → -EINVAL**：`backend.ControlJob` 把 `xce_jobctrl_t.action` 设成 `models.JOB_*`（1/2/3/8），但该字段是 capnp `JobControl` union 判别符（`JobControl_u_siginfo=0`/`JobControl_u_ckptoption=1`）→ controller `job_ctrl()` 走错分支（default）→ 每个动作都 -EINVAL。 +- **修复**：`normalizeJobInt64`/`jobArrayIdx` 辅助函数 + `jobCtrlDiscriminant` 映射。单测 5 个（controllers 3 + backend 1 + 顺带修了 pre-existing 的 backend cgo 测试编译）。集群 E2E 验证：hold→PSUSP、release→PEND、cancel→EXIT，无 500 无 -EINVAL，全部 daemon 健康。 +- **顺带修复**：`resolveExecUser` 被放在 `//go:build cgo` 的 convert.go 里导致 `make test`（CGO_ENABLED=0）下 backend 包编译失败——移到纯 Go 的 convert_pure.go。 + +### 3. bug-1489 / bug-1491 — 已修复（上下文） +- bug-1491 OrderMap 列格式解析修复（2026-08-01，httpclient.go） +- bug-1489 whitelist 缺 `/xce/v1/apikeys` 修复（2026-08-01，middleware/whitelist.go） + +--- + +## 二、scheduler-controller-msg-gap：已实现但有已知缺陷 + +spec 声称 36/36 任务完成，但 re-audit（research.md）发现缺陷。来源：`.kiro/specs/scheduler-controller-msg-gap/research.md`。 + +| 项 | 严重度 | 描述 | +|----|--------|------| +| **G-8** | HIGH | `sched_update_existing_job` 是死代码。notify 驱动的作业更新走 bulk `sched_update_jobs()` 的 CREATE 路径（make_sched_job 新建），损坏/复制作业状态。**很可能与 bug-1494 相关**。Req 3 UB-3.1/3.2 未满足。 | +| **G-3** | MED | 5 个新 op 的 payload（UPDATE_QUEUE/UPDATE_RESOURCES/UPDATE_RESVALS_FULL/UPDATE_RESVALS/JOB_RUSAGE）在 `sched_pend_update_killer` 里 `break;` 从不释放——长驻 daemon 慢速泄漏。 | +| **G-5** | MED | `xce_sched_job_t` 无 rusage 字段。JOB_RUSAGE case 只调插件 hook 就丢 payload。UB-6.1 未满足。 | +| **G-4** | LOW | 即使修好 G-8 让 `sched_update_existing_job` 可达，它也只在作业上应用 state——priority/queue/resreq 未应用。 | +| **G-6** | TODO | 任务 9.2/9.3/9.4/9.6/9.7 勾了 `[x]` 但测试缺失/部分；9.5 测的是本地 stub 非真 schedobj.c。 | +| **G-7** | TODO | 文档漂移：design 说 `xce_sched_op_name` 在 lib/helper，实际是 scheduler.c 里 static `sched_op_name`。 | +| **R-3** | 休眠 | Req 1 处理器前瞻实现，但 controller 当前不推送 Data_data_resources/resvals 表给 scheduler——生产未触发。 | + +--- + +## 三、核心调度引擎重构（scheduler-framework 家族：spec 有、零实现） + +来源：`.kiro/specs/scheduler-framework/`、`scheduler-core/`、`scheduler-allocation/`、`scheduler-preemption/`（均只有 requirements/brief，无 design/tasks）。roadmap 显示 "Next: /kiro-spec-requirements scheduler-framework"。 + +### scheduler-core（Phase 1-3） +| 项 | 描述 | +|----|------| +| 可插拔 picker 接口 | 替代硬编码 `"xce_sched_fcfs"` 字符串，`job_pick_plugin` 字段 + config 加载 + 每 queue 覆盖。核心循环调 `picker->next_job()`。 | +| `builtin_fcfs` 默认实现 | MUST/SHOULD hook 全实现，私有 ordq_t 状态，排序同现状（跨 queue priority desc，队内 priority desc + queue_time asc）。 | +| 无回归保证 | HA soak 必须验证作业挑选顺序零偏差。 | +| core/driver 分离 | 提取 `xce_sched_core_t`，无 mode flag，虚拟时钟 `xce_core_now()`，决策回调。Phase 2。 | +| 模拟模式 | JSONL/.replay 重放 + 虚拟时钟 + snapshot/restore + 确定性重放 + `xceadmin scheduler record` + `--sim/--sim-verify/--interactive`。Phase 3。 | +| 等价保证 | "记录实跑 → 模拟重放 → 决策字节级一致"（AC 7.1）。 | + +### scheduler-allocation +| 项 | 描述 | +|----|------| +| 惰性 plan 迭代器 | `xce_alloc_plan_iter_t`：init(resreq)/next()->plan/fini()，cancel-safe。 | +| 惰性 host 迭代器 | `xce_host_iter_t`：init(plan, workers)/next()->worker，复用现有 select 表达式求值器。 | +| parjob 组合分配 | 替换 stub `return 0`：1host×N、2hosts×ceil(N/2)、…、Nhosts×1。 | +| 会话级 resreq 缓存 | keyed on {minTasks,maxTasks,select_node_id,rusage_hash,user,queue,project}，作业 ingest 时预计算。 | +| seqjob 重包 | 1host/1slot 走同一迭代器接口，验证抽象。 | + +### scheduler-preemption +| 项 | 描述 | +|----|------| +| `XCE_DECISION_PREEMPT` | decision.h 新 opcode + `xce_preempt_decision_t`（victim_jobids, SIGSTP, preempting_jobid）。 | +| victim 选择策略 | 默认 `"lowest_priority_first"`，LSF 式跨 queue（preemptible_queues），最小足够 victim 集，安全护栏（不抢占同/高优先级、非抢占 queue）。 | +| SIGSTP 管线 | scheduler→controller→worker→SIGSTP 进程组，作业→JOB_SUSPENDED，复用 HOOK_DETAIL_JOB_STOP。每 queue `preemptible/preempt_policy/preemptible_queues` 字段。 | +| 核心循环集成 | 分配失败后、pending 前：can_preempt → 选 victim → 发 PREEMPT → preempting 作业下一轮再派（异步抢占）。 | +| 后续推迟项 | SIGKILL+requeue 升级、JOB_SUSPENDED resume 机制。 | + +--- + +## 四、故障转移 / 重连 + +### component-failover-adoption（14 任务全未勾选） +- **来源**：`.kiro/specs/component-failover-adoption/` +- **关键结论**：requirements.md Background 明确 "**scheduler 没有任何运行时重连/rejoin 逻辑**——连的 controller 死后，scheduler 无法切换到其他候选"。 +- 任务 3.1：把 scheduler 的 per-candidate 循环换成单一 library 驱动的 connect/join + rejoin 回调。 +- 任务 3.2：scheduler 多 controller join E2E 测试。 +- Req 3：重连重试/backoff 不能阻塞 picker 线程（保持现有线程拆分）。 +- Req 5：per-candidate 重连诊断日志（成功/不可达/standby 重定向）。 + +### P2.5 +- **来源**：`product.md:35`、`docs/superpowers/plans/2026-05-07-p25-discovery.md` +- product.md 标 P2.5 "In progress"；P2.5-6 描述 scheduler `--peers`/`--ca-cert` CLI 更新；P2.5-7（CGo bridge）推迟。是否全完成文档不确定。 +- **注意**：P2.5 与 component-failover-adoption 用了两套技术路径（xce_connect_cluster 裸 endpoint vs xcecli_ctx_t/XCECLI_FAILOVER_CALL），文档未调和。 + +--- + +## 五、scheduler agent / MCP server（Phase 2-4） + +来源：`docs/Scheduler-Agent-设计文档.md` + +| 项 | 状态 | 描述 | +|----|------|------| +| Phase 2：SSE 事件流 | 计划 | apiserver SSE endpoint + EventBroker，schedulerclient `WatchJobs()`，mcp-server 接入。ADR-004 设计完成待实施。估 2-3 天。 | +| Phase 3：LSF adapter | 计划 | CLI 解析优先，Python Sidecar 后续。估 +1 月。 | +| Phase 4：Slurm + K8s | 计划 | slurmrestd + CLI fallback；client-go。估 +1 月。 | +| modify 工具 | 推迟 | apiserver 当前无 modify 端点（control 仅 kill/stop/resume/checkpoint），需新增后端端点。 | +| 未决决策 | 未解决 | LLM 模型选型（Claude API/自建/混合）；多集群 URM 源识别；air-gapped LLM 访问。 | + +--- + +## 六、roadmap / 加固项 + +| 项 | 状态 | 描述 | +|----|------|------| +| 调度决策过 commit barrier | 计划 | DISPATCH/PEND/PREEMPT 需过 majority commit barrier。多文档提及为独立加固项，从 scheduler-framework scope 推迟。 | +| worker-rejoin-resync | backlog | worker 重连重报运行/完成作业状态（bug-396 兜底）。依赖 ha-consistency-invariants。scheduler 间接受影响（恢复作业需调度状态核对）。 | +| entity-bearing snapshots | backlog | 解锁 config-upgrade-snapshot、基于快照的状态传输。影响 scheduler 模拟（snapshot/restore）。COW 机制存在但休眠。 | +| project-resource-ownership | 未来 spec | 层级项目树 + 资源配额，project-aware picker 插件，queue↔project 映射。核心循环零改动（只加新 picker）。 | +| perf-tuning-extraction / clusterconfig-schema-split | 计划 | 配置 schema 变更触及 scheduler 配置摄取。 | +| bulk job-array query/cancel | 推迟 | 依赖 job-array-single-wal-record。 | + +--- + +## 七、文档空白（设计层面没人写过） + +1. **调度决策 commit barrier 细节**——roadmap 只提了标题，无 spec/design。 +2. **scheduler 状态在 controller failover 后如何同步**——全量重推？增量？文档沉默。roadmap:67 提到 lightweight rejoin re-attach（当前是 full-state rejoin，"大但罕见"）。 +3. **scheduler 模拟模式 vs controller WAL replay 关系**——failover 后 scheduler 从 controller WAL 重放还是收新状态推送？未讨论。 +4. **core/driver 分离下的对象内存/生命周期**——三线程（picker/allocator/driver）async 生产者/消费者模型下借用指针的归属未细化。 +5. **无 scheduler 专属 HA soak 测试**——多次提到 "HA soak regression" 但无 `/xce-test-ha` 变体专门跑 scheduler failover/reconnect。 +6. **`src/scheduler/CLAUDE.md` 太薄**——约 25 行，无 spec 状态、无已知 bug 清单、无剩余工作节。建议补全（对照 `src/controller/CLAUDE.md`）。 +7. **scheduler 与 raft/TLA+ 架构整合**——raft 系列落地后 scheduler 与 log/commit 模型的交互未讨论。 + +--- + +## 八、scheduler-agent（chatbot / AI 辅助）剩余工作 + +- 状态：持续更新 +- 日期：2026-08-07 +- 来源：`docs/Scheduler-Agent-设计文档.md`（v1.2，权威设计文档）、`src/docs/chatbot-verification.md`、`src/frontend/CLAUDE.md`、`CLAUDE.md` +- **注意**：feature 直接从设计文档实现，**未走 .kiro spec 流程**（`.kiro/specs/` 无 scheduler-agent/chatbot spec）。`/home/lchen/code/chatbot/` 是 XCE 完整镜像工作树，代码两份一致。 + +### 架构与现状 + +用户自然语言 → Claude Code（LLM）→ MCP stdio → mcp-server → schedulerclient → apiserver REST → controller（C 后端）。**不是独立聊天组件，而是自然语言调度管理 Agent 框架**。 + +**已完成**： +| Phase | 内容 | 状态 | +|-------|------|------| +| Phase 0 | 只读 MCP Server：6 读工具、schedulerclient、`mcp_readonly` 角色、API Key 认证 | ✅ 2026-07-26 | +| Phase 1 | 4 写工具（submit/cancel/hold/release）、`general_serial` 模板、`mcp_operator`、`dry_run` 审查 | ✅ 2026-08-01，submit E2E 验证，control 被 bug-1490 阻塞 | + +代码：`src/frontend/mcp-server/`（10 工具，~1300 行）、`src/frontend/schedulerclient/`（45 测试）、apiserver API Key + RBAC 支持。 + +### 1. 阻塞性 bug（影响 chatbot E2E 验证） +| Bug | 影响 | +|-----|------| +| **bug-1490** | control（hold/release/cancel）HTTP 500，apiserver control 链路预存 bug，请求到不了 controller。**写工具 E2E 验证的头号拦路虎** | +| **bug-1494** | 作业全 PEND 不调度，Agent 提交流程无法完成。根因已定位（见第一节） | + +### 2. Phase 2：监控 + 纠错闭环（未开始，估 2-3 天） +- **SSE 事件流**（ADR-004 设计已完、未实现）：apiserver SSE endpoint + EventBroker，schedulerclient `WatchJobs()`，mcp-server 事件接入。Agent 工作流：submit → watch → analyze → auto-handle +- 3 个高级工具：`scheduler_analyze_failure`、`scheduler_suggest_resources`、`scheduler_estimate_wait_time` + +### 3. Phase 3 / Phase 4：后端适配器（未开始） +- Phase 3：**LSF adapter**（CLI 解析优先，Python Sidecar 后续，估 +1 月） +- Phase 4：**Slurm**（slurmrestd）+ **K8s**（client-go）（估 +1 月） + +### 4. 推迟 / 未决 +- `scheduler_modify` 工具推迟——apiserver 当前无 modify 端点（control 仅 kill/stop/resume/checkpoint），需新增后端端点 +- 3 个未决决策：LLM 模型选型（Claude API/自建/混合）、多集群 URM 来源识别、air-gapped LLM 访问 + +### 5. 设计层加固（文档已写、代码零实现） +- **Auto-pilot 模式**：用户显式授权范围内的自主行为 +- **AI 操作审计日志**：每条操作记 `[timestamp] [user] [action] [params] [confirmed?]` +- **Golden test suite**：同一自然语言输入 → 同一工具调用序列的回归测试 + +--- + +## 建议优先级 + +1. **bug-1490**（apiserver control HTTP 500）——阻塞 chatbot 写工具 E2E，小而明确，能立即修。 +2. **bug-1494**（作业 PEND）——根因已定位（worker+scheduler 不自愈）。G-8 死代码（scheduler 作业更新路径损坏）很可能是元凶。 +3. **msg-gap G-8/G-3/G-5**——小而明确，能立即修。 +4. **component-failover-adoption**（scheduler 重连）——与 bug-1494 的 join 链路疑点直接相关。 +5. scheduler-agent Phase 2（SSE 事件流）——依赖 bug-1490/bug-1494 先修好。 +6. scheduler-framework 重构——大规划，建议等前 5 项稳定后再动。 diff --git a/src/frontend/apiserver/backend/clib.go b/src/frontend/apiserver/backend/clib.go index b0dfe64..f99aa5e 100644 --- a/src/frontend/apiserver/backend/clib.go +++ b/src/frontend/apiserver/backend/clib.go @@ -394,7 +394,12 @@ func SubmitJob(job *models.JobInfo, arraySize int) error { func ControlJob(jobid uint64, action int, content string) error { var ctrl C.xce_jobctrl_t ctrl.jobid = C.uint64_t(jobid) - ctrl.action = C.int(action) + // ctrl.action is the capnp JobControl union DISCRIMINANT (0=siginfo, + // 1=ckptoption), NOT the models.JOB_* constant. bug-1490: setting it to + // JOB_STOP/RESUME/KILL/CHKPNT (1/2/3/8) made the controller's job_ctrl() + // select the wrong union branch (or none) and reject every control with + // -EINVAL, so hold/release/cancel never reached the job. + ctrl.action = C.int(jobCtrlDiscriminant(action)) // For signal-based actions set signo; for checkpoint set ckpt options. // xce_jobctrl_t.u is a C union — we use a static C helper to set the diff --git a/src/frontend/apiserver/backend/convert.go b/src/frontend/apiserver/backend/convert.go index ac865be..1bbc29c 100644 --- a/src/frontend/apiserver/backend/convert.go +++ b/src/frontend/apiserver/backend/convert.go @@ -249,19 +249,6 @@ func cClusterConfToClusterInfo(conf *C.cluster_config_t) *models.ClusterInfo { // modelJobToSubmit converts a Go JobInfo to C xce_job_submit_t heap structs. // The caller must free all returned C memory by calling freeJobSubmit. -// resolveExecUser returns the user a job should run as: an explicit exec_user -// wins; otherwise the job runs as the submitting user. bug-1500: the submit -// conversion previously dropped this entirely, so the worker received a NULL -// execuser and ran every job as "nobody". Kept as a plain-Go function so it is -// unit-testable without cgo (the Go toolchain rejects `import "C"` in _test.go). -func resolveExecUser(job *models.JobInfo) string { - execUser := job.ExecUser - if execUser == "" { - execUser = job.SubmitUser - } - return execUser -} - func modelJobToSubmit(job *models.JobInfo) (*C.xce_job_submit_t, *C.job_submit_data_t, *C.job_startinfo_t) { si := (*C.job_startinfo_t)(C.calloc(1, C.sizeof_job_startinfo_t)) si.cmdline = C.CString(job.Command) diff --git a/src/frontend/apiserver/backend/convert_pure.go b/src/frontend/apiserver/backend/convert_pure.go index 0ba0daf..4e5be98 100644 --- a/src/frontend/apiserver/backend/convert_pure.go +++ b/src/frontend/apiserver/backend/convert_pure.go @@ -7,6 +7,36 @@ import ( "fmt" ) +// resolveExecUser returns the user a job should run as: an explicit exec_user +// wins; otherwise the job runs as the submitting user. bug-1500: the submit +// conversion previously dropped this entirely, so the worker received a NULL +// execuser and ran every job as "nobody". Plain Go so it is unit-testable +// without cgo — keep it OUT of the cgo-only convert.go. +func resolveExecUser(job *models.JobInfo) string { + execUser := job.ExecUser + if execUser == "" { + execUser = job.SubmitUser + } + return execUser +} + +// jobCtrlDiscriminant maps a models.JOB_* action to the capnp JobControl union +// discriminant carried in xce_jobctrl_t.action: JobControl_u_siginfo (0) for +// signal-based actions (kill/stop/resume), JobControl_u_ckptoption (1) for +// checkpoint. The models.JOB_* constants are a DIFFERENT enum — using them as +// the discriminant made the controller's job_ctrl() select the wrong union +// branch (or none) and reject every control request with -EINVAL (bug-1490). +// Returns -1 for an unknown action (clib.go never calls ControlJob with one). +func jobCtrlDiscriminant(action int) int { + switch action { + case models.JOB_STOP, models.JOB_RESUME, models.JOB_KILL: + return 0 // JobControl_u_siginfo + case models.JOB_CHKPNT: + return 1 // JobControl_u_ckptoption + } + return -1 +} + // JobID bit layout constants — mirrors src/include/config.h. // // |63 62|61 56|55 52|51 20|19 15|14 0| diff --git a/src/frontend/apiserver/backend/convert_test.go b/src/frontend/apiserver/backend/convert_test.go index a0a4bc8..6d60574 100644 --- a/src/frontend/apiserver/backend/convert_test.go +++ b/src/frontend/apiserver/backend/convert_test.go @@ -5,6 +5,33 @@ import ( "testing" ) +// TestJobCtrlDiscriminant is a regression test for bug-1490: backend.ControlJob +// used to set xce_jobctrl_t.action to the models.JOB_* constant +// (JOB_STOP=1/JOB_RESUME=2/JOB_KILL=3/JOB_CHKPNT=8), but that field is the +// capnp JobControl union DISCRIMINANT (JobControl_u_siginfo=0, +// JobControl_u_ckptoption=1). The controller's job_ctrl() interpreted the wrong +// union branch (or none) and rejected every control request with -EINVAL, so +// hold/release/cancel never reached the job. +func TestJobCtrlDiscriminant(t *testing.T) { + cases := []struct { + name string + action int + want int + }{ + {"stop is a signal action", models.JOB_STOP, 0}, // JobControl_u_siginfo + {"resume is a signal action", models.JOB_RESUME, 0}, + {"kill is a signal action", models.JOB_KILL, 0}, + {"checkpoint is a ckptoption action", models.JOB_CHKPNT, 1}, // JobControl_u_ckptoption + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := jobCtrlDiscriminant(tc.action); got != tc.want { + t.Errorf("jobCtrlDiscriminant(%d) = %d, want %d", tc.action, got, tc.want) + } + }) + } +} + func TestRawJobToModel_BasicFields(t *testing.T) { r := rawJobData{ jobID: 123, diff --git a/src/frontend/apiserver/controllers/jobs.go b/src/frontend/apiserver/controllers/jobs.go index 1e20c80..44976d6 100644 --- a/src/frontend/apiserver/controllers/jobs.go +++ b/src/frontend/apiserver/controllers/jobs.go @@ -3025,14 +3025,14 @@ func validateJobControl(jobinfo utils.OrderMap, action string, failedJobs = append(failedJobs, map[string]interface{}{ "unique_job_id": jobinfo.Data["unique_job_id"].(int64), "job_id": jobinfo.Data["job_id"].(int64), - "array_idx": int(jobinfo.Data["array_idx"].(int64)), + "array_idx": jobArrayIdx(jobinfo), "msg": fmt.Sprintf("job cannot be or "), }) } else { successJobs = append(successJobs, map[string]interface{}{ "unique_job_id": jobinfo.Data["unique_job_id"].(int64), "job_id": jobinfo.Data["job_id"].(int64), - "array_idx": int(jobinfo.Data["array_idx"].(int64)), + "array_idx": jobArrayIdx(jobinfo), "msg": fmt.Sprintf("job is being <%s>", action), }) } @@ -3041,7 +3041,7 @@ func validateJobControl(jobinfo utils.OrderMap, action string, successJobs = append(successJobs, map[string]interface{}{ "unique_job_id": jobinfo.Data["unique_job_id"].(int64), "job_id": jobinfo.Data["job_id"].(int64), - "array_idx": int(jobinfo.Data["array_idx"].(int64)), + "array_idx": jobArrayIdx(jobinfo), "msg": fmt.Sprintf("job is being <%s>", action), }) } else { @@ -3049,7 +3049,7 @@ func validateJobControl(jobinfo utils.OrderMap, action string, failedJobs = append(failedJobs, map[string]interface{}{ "unique_job_id": jobinfo.Data["unique_job_id"].(int64), "job_id": jobinfo.Data["job_id"].(int64), - "array_idx": int(jobinfo.Data["array_idx"].(int64)), + "array_idx": jobArrayIdx(jobinfo), "msg": fmt.Sprintf("job cannot be <%s>", action), }) } @@ -3058,7 +3058,7 @@ func validateJobControl(jobinfo utils.OrderMap, action string, failedJobs = append(failedJobs, map[string]interface{}{ "unique_job_id": jobinfo.Data["unique_job_id"].(int64), "job_id": jobinfo.Data["job_id"].(int64), - "array_idx": int(jobinfo.Data["array_idx"].(int64)), + "array_idx": jobArrayIdx(jobinfo), "msg": fmt.Sprintf("Invalid job_status"), }) } @@ -3067,7 +3067,7 @@ func validateJobControl(jobinfo utils.OrderMap, action string, failedJobs = append(failedJobs, map[string]interface{}{ "unique_job_id": jobinfo.Data["unique_job_id"].(int64), "job_id": jobinfo.Data["job_id"].(int64), - "array_idx": int(jobinfo.Data["array_idx"].(int64)), + "array_idx": jobArrayIdx(jobinfo), "msg": fmt.Sprintf("job_status not found for job"), }) } @@ -3083,7 +3083,7 @@ func foundJob(jobid int64, array_idx int, jobinfos []utils.OrderMap) ([]utils.Or for _, jobinfo := range jobinfos { if array_idx > 0 { if jobinfo.Data["job_id"].(int64) == jobid && - int(jobinfo.Data["array_idx"].(int64)) == array_idx { + jobArrayIdx(jobinfo) == array_idx { jobs = append(jobs, jobinfo) } } else { @@ -3099,26 +3099,48 @@ func foundJob(jobid int64, array_idx int, jobinfos []utils.OrderMap) ([]utils.Or } } -// normalizeJobStatus converts a job_status value (int, int64, or float64 from -// JSON decode) into a plain int. Returns ok=false for unsupported types. -func normalizeJobStatus(v any) (int, bool) { +// normalizeJobInt64 converts a job field value (int, int32, int64, float64, or +// json.Number — whichever representation the store or a JSON decode path +// produced) into a plain int64. Returns ok=false for unsupported types. +func normalizeJobInt64(v any) (int64, bool) { switch s := v.(type) { case int: - return s, true + return int64(s), true case int32: - return int(s), true + return int64(s), true case int64: - return int(s), true + return s, true case float64: - return int(s), true + return int64(s), true case json.Number: if n, err := s.Int64(); err == nil { - return int(n), true + return n, true } } return 0, false } +// normalizeJobStatus converts a job_status value (int, int64, or float64 from +// JSON decode) into a plain int. Returns ok=false for unsupported types. +func normalizeJobStatus(v any) (int, bool) { + n, ok := normalizeJobInt64(v) + return int(n), ok +} + +// jobArrayIdx extracts the array_idx field from a job row's Data map, tolerating +// any numeric representation the store produces. Defaults to 0 when absent. +// Regression for bug-1490: the old code asserted .(int64) on a value the store +// produces as int, panicking every control request into an HTTP 500 before it +// reached the controller. +func jobArrayIdx(row utils.OrderMap) int { + if v, ok := row.Data["array_idx"]; ok { + if n, ok := normalizeJobInt64(v); ok { + return int(n) + } + } + return 0 +} + func processJobData(rows []utils.OrderMap, pagetoken string) utils.QueryResult { logger.Debug("Enter processJobData() function", zap.Any("rows", rows)) for _, row := range rows { diff --git a/src/frontend/apiserver/controllers/jobs_test.go b/src/frontend/apiserver/controllers/jobs_test.go index 26727c9..375f16b 100644 --- a/src/frontend/apiserver/controllers/jobs_test.go +++ b/src/frontend/apiserver/controllers/jobs_test.go @@ -74,3 +74,75 @@ func TestPreCheckJobRequest_EmptyQueue(t *testing.T) { t.Errorf("empty queue should not trigger reserved-name error: %v", err) } } + +// jobRowFromStore builds an OrderMap exactly as store.StoreToOrderMaps +// produces it for a JobInfo: job_id/unique_job_id are int64, but array_idx and +// job_status are int (models.JobInfo.ArrayIdx/JobStatus are declared int). +func jobRowFromStore(jobID int64, arrayIdx int, status int) utils.OrderMap { + return utils.OrderMap{ + Data: map[string]interface{}{ + "unique_job_id": jobID, + "job_id": jobID, + "array_idx": arrayIdx, + "submit_user": "tester", + "queue_name": "normal", + "job_status": status, + }, + } +} + +// TestValidateJobControl_ArrayIdxInt_Allowed is a regression test for bug-1490: +// job control (hold/release/cancel) returned HTTP 500 because validateJobControl +// asserted jobinfo.Data["array_idx"].(int64), but the store produces array_idx +// as int. That panic was caught by Gin's Recovery middleware → HTTP 500, before +// the control request ever reached the controller. An allowed action must now +// classify the job as a success without panicking. +func TestValidateJobControl_ArrayIdxInt_Allowed(t *testing.T) { + resetJobSubmitState() + + jobInfo := jobRowFromStore(2001, 0, models.JOBSTAT["RUN"]) + + success, failed := validateJobControl(jobInfo, "stop", "TSTP") + if len(success) != 1 { + t.Fatalf("expected 1 success job, got %d (failed=%v)", len(success), failed) + } + if got := success[0]["array_idx"].(int); got != 0 { + t.Errorf("array_idx = %d, want 0", got) + } + if got := success[0]["unique_job_id"].(int64); got != 2001 { + t.Errorf("unique_job_id = %d, want 2001", got) + } +} + +// TestValidateJobControl_ArrayIdxInt_NotAllowed covers the "action not allowed" +// failure branch of validateJobControl, which asserted array_idx the same way +// (bug-1490). Resume on a RUN job is not allowed and must report a failed job +// without panicking. +func TestValidateJobControl_ArrayIdxInt_NotAllowed(t *testing.T) { + resetJobSubmitState() + + jobInfo := jobRowFromStore(2001, 0, models.JOBSTAT["RUN"]) + + success, failed := validateJobControl(jobInfo, "resume", "") + if len(failed) != 1 { + t.Fatalf("expected 1 failed job, got %d (success=%v)", len(failed), success) + } + if got := failed[0]["array_idx"].(int); got != 0 { + t.Errorf("array_idx = %d, want 0", got) + } +} + +// TestFoundJob_ArrayIdxInt covers foundJob's array-child branch, which also +// asserted array_idx as int64 (bug-1490) and panicked for array jobs with an +// explicit index range. +func TestFoundJob_ArrayIdxInt(t *testing.T) { + row := jobRowFromStore(2001, 2, models.JOBSTAT["PEND"]) + + jobs, err := foundJob(2001, 2, []utils.OrderMap{row}) + if err != nil { + t.Fatalf("foundJob returned error: %v", err) + } + if len(jobs) != 1 { + t.Fatalf("expected 1 matching job, got %d", len(jobs)) + } +}