From ac6131dbd72370eaee2fa8e246030305efbaafdb Mon Sep 17 00:00:00 2001 From: "lei.chen" Date: Thu, 13 Aug 2026 09:16:13 +0000 Subject: [PATCH] =?UTF-8?q?feat(apiserver):=20expose=20PEND=20reason=20end?= =?UTF-8?q?-to-end=20(scheduler=20=E2=86=92=20API)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pending_reason field was declared on models.JobInfo but never populated: scheduler computed PEND reasons and the controller stored them on job->pend_reasons, yet the data stopped there. Two gaps closed: 1. controller/apiserver.c: apiserver_job_hook_fun now handles HOOK_DETAIL_SETPREASON by pushing the updated job to API servers (reusing apiserver_job_update_fun). Without this, a job whose PEND reasons change never reached the API server — only JOB_NEW/UPDATE/START/EXIT events did, and setpreason fired none of those. 2. backend/convert.go + convert_pure.go: cJobToModel / rawJobToModel now extract xce_job_t.pend_reasons (main_reason + host/res reasons) and marshal them into models.JobInfo.PendingReason as JSONB. models/jobs.go: pending_reason added to JOB_DEFAULT_COLUMN so default job queries return it (it was only in JSONBJFields, never selected). Verified live on t1/t2/t3: a job with resreq "type == nosuchhost" stays PEND and returns "pending_reason":{"main_reason":0,"host_reasons":[{"host":"t1","reason":3}, {"host":"t2","reason":3},{"host":"t3","reason":3}]} (main_reason=3 PREASON_MATCH_FAIL per host) via both /jobs list, /jobs/:id and the SSE stream. 4 new unit tests (rawJobToModel pendReason marshal + name mapping), all apiserver backend tests pass. Co-Authored-By: Claude --- docs/aibot_poc_investiagting.md | 66 ++++++++++++++++ src/controller/apiserver.c | 8 ++ src/frontend/apiserver/backend/convert.go | 28 +++++++ .../apiserver/backend/convert_pure.go | 47 ++++++++++++ .../apiserver/backend/convert_test.go | 75 +++++++++++++++++++ src/frontend/apiserver/models/jobs.go | 1 + 6 files changed, 225 insertions(+) create mode 100644 docs/aibot_poc_investiagting.md diff --git a/docs/aibot_poc_investiagting.md b/docs/aibot_poc_investiagting.md new file mode 100644 index 0000000..be4c817 --- /dev/null +++ b/docs/aibot_poc_investiagting.md @@ -0,0 +1,66 @@ +# AI Bot 对外 POC 能力建议（scheduler-agent MCP） + +> 状态：建议稿（2026-08-13）。讨论稿，未实施。 +> 目标：对外 POC 演示 XCE 的 AI agent（scheduler-agent MCP）能力。出彩的关键不是"更多查询工具"，而是**闭环 + 叙事**——让观众看到 agent 不只是读状态，还能诊断、能动手、能解释。 + +## 一、现状：已注册 14 个 MCP 工具 + +| 类别 | 工具 | +|------|------| +| 只读·状态（6） | `scheduler_list_jobs` / `get_job` / `list_queues` / `list_nodes` / `job_logs` / `cluster_metrics` | +| 只读·高级（4） | `wait_job` / `analyze_failure` / `suggest_resources` / `estimate_wait_time` | +| 写（4，需确认） | `submit`（dry_run 审查 + general_serial 模板）/ `cancel` / `hold` / `release` | + +## 二、推荐增强方向（按 POC 性价比排序） + +### 1. 失败作业「自愈闭环」— 首选，对外 POC 的 money shot +现状 `analyze_failure` 只是"把数据打包给你看"。升级为 **诊断 → 归因 → 行动 → 复跑验证** 的闭环： +- **归因分类**：按 exit code / signal / 日志特征分类——OOM、被信号杀（segfault/killed）、命令不存在、超时、基础设施失败（节点掉了）；每类给一句人话解释 + 证据链 +- **自动重试策略**：OOM → 建议加内存重交；被信号杀 → 检查资源限值；脚本错误 → 不改资源、指出错在哪行 +- **自愈动作**：`rerunable` 作业 → 用调整后的规格 re-submit → `wait_job` 验证 → 报告"第一次 OOM 失败 → 加大内存后成功" + +数据已齐（failinfo code、日志、submit 全参数）；主要是把 `analyze_failure` + `submit` + `wait_job` 编排成智能逻辑。 +演示剧本："提交一个会 OOM 的作业 → agent 自己把它修好跑通"。 + +### 2. 作业旅程「实时叙事」— 最便宜、最抓眼球 +SSE 事件流已建好（PEND→RUN→DONE），缺把事件转成有人味的实时解说： +> "作业 #12 已排队（第 3 位）→ 预计 40s 后开始 → 已分配到 t2 节点（负载 30%）→ 运行中，耗时 2m10s → 完成 ✅" + +配合 `wait_job` 即"提交后 agent 全程盯着并讲解"。live demo 连续跑几个作业画面感极强。工程量最小（prompt/模板活）。 + +### 3. 调度决策「解释」（PEND reason）— XCE 差异化，别家抄不走 +**"为什么我的作业等了 10 分钟 / 为什么没排到更空的节点？"** HPC 特有。 +scheduler 侧已有 PEND 原因（`PREASON_NO_RESOURCE`、`PREASON_MATCH_FAIL`、资源不满足 slots/mem…），agent 若能读取并翻译： +> "作业 PEND 是因为 8 槽节点全满，你的 2 核需求需等 t3 释放 1 个作业（约 3 分钟）" + +**前置检查（2026-08-13 完成）**：PEND reason 全链路**只差一环**——apiserver Go 转换层没填充 `PendingReason` 字段。详见下方「五、PEND reason 现状调查」。 + +### 4. 集群调优（数据可行，次优先级） +- **热点/碎片化分析**：扫节点 + 资源分布 → "t1 长期 90% 满载、t3 只有 20%，建议放宽 big 队列 affinity"（`resources/distribution` + hosts 数据已可做） +- **排空时间预测**：估算当前队列多久清空（`estimate_wait_time` 的队列维度扩展） +- **配置变更 what-if**：模拟改队列限制/加节点对等待时间的影响——最"调优"但需调度模型，工程量最大，POC 后置 + +## 三、POC 演示剧本建议 +按感染力排序：**①自愈闭环（故事核心）→ ②实时叙事（视觉）→ ③PEND 解释（差异化）→ ④调优（收尾卖点）**。 + +## 四、待办 +- [x] 检查 PEND reason 全链路（scheduler → apiserver）是否暴露齐全（2026-08-13 完成，结论：只差 apiserver Go 转换层） +- [ ] 补 apiserver 转换层：`cJobToModel` / `rawJobToModel` 填充 `PendingReason`（convert.go / convert_pure.go） +- [ ] ① 自愈闭环：失败归因分类器 + 自愈复跑流程 +- [ ] ② 实时叙事 prompt/模板 + +## 五、PEND reason 现状调查（2026-08-13） + +**结论：全链路只差一环——apiserver Go 转换层没把 C 的 `pend_reasons` 映射到 `models.JobInfo.PendingReason`，API 的 `pending_reason` 恒为空。** 上游全部就绪： + +| 环节 | 状态 | 位置 | +|------|------|------| +| scheduler 计算并 publish PEND 原因 | ✅ | `sched_make_allocation` 返回 0 → `make_pend_decision(reason)`（scheduler.c:267）；main.c:189 `sched_publish_pend_reason` → `setpreason` opcode | +| 原因类型 | ✅ | `PREASON_BAD_RESREQ`/`NO_RESOURCE`/`MATCH_FAIL` + host/res 明细（preason.h；resource.c allocator `add_job_preason`） | +| controller 存储 | ✅ | `do_set_preason` → `job->pend_reasons`（controller/job.c:2585-2613）；`do_clr_preason` 清除 | +| capnp wire 编解码 | ✅ | `JobInfo.pendreason @10` encode/decode（xce.build/lib/controller.capnp.c:6152/6168） | +| C 结构体字段 | ✅ | `xce_job_t.pend_reasons`（src/include/xce/types.h:479） | +| apiserver model | ✅ | `JobInfo.PendingReason json:"pending_reason"`（models/jobs.go:553）+ 在 `JSONBJFields`（:140），填充后 processJobData 会自动解析浮现（jobs.go:3208） | +| **apiserver Go 转换** | ❌ | `cJobToModel`（backend/convert.go:74）与 `rawJobToModel`（backend/convert_pure.go:210）**均未读取 `pend_reasons`** → PendingReason 恒 nil | + +**修复量**：小。在两个转换函数各加一段把 C `pend_reasons`（main_reason + host/res reasons）编码为 JSONB 塞进 `PendingReason`，配单测即可。 diff --git a/src/controller/apiserver.c b/src/controller/apiserver.c index 08f9f2a..58425c2 100644 --- a/src/controller/apiserver.c +++ b/src/controller/apiserver.c @@ -414,6 +414,14 @@ static int apiserver_job_hook_fun(int type, int situation, int detail, return apiserver_job_fail_fun(type, situation, data, ap); case HOOK_DETAIL_JOB_RUSAGE_UPDATE: return apiserver_job_rusage_update_fun(type, situation, data, ap); + case HOOK_DETAIL_SETPREASON: + /* The scheduler pushed new PEND reasons via setpreason and the + * controller just stored them on the job (set_preason). Push the + * updated job to the API servers the same way a JOB_UPDATE does — + * otherwise the API servers keep the job without pend_reasons for + * its whole PEND lifetime (the job's own state never changed, so no + * JOB_UPDATE fires) and the pending_reason field stays empty. */ + return apiserver_job_update_fun(type, situation, data, ap); } return 0; diff --git a/src/frontend/apiserver/backend/convert.go b/src/frontend/apiserver/backend/convert.go index 1bbc29c..ddf459f 100644 --- a/src/frontend/apiserver/backend/convert.go +++ b/src/frontend/apiserver/backend/convert.go @@ -9,6 +9,10 @@ package backend // Helper to index into xce_host_alloc_t** (CGO cannot dereference C pointer-to-pointer arrays). static xce_host_alloc_t *job_alloc_host(xce_job_alloc_t *a, int i) { return a->allocs[i]; } + +// Helpers to index into the reason pointer arrays of xce_job_preason_t. +static xce_host_reason_t *job_host_reason(xce_job_preason_t *p, int i) { return p->host_based_reasons[i]; } +static xce_res_reason_t *job_res_reason(xce_job_preason_t *p, int i) { return p->res_based_reasons[i]; } */ import "C" @@ -117,6 +121,30 @@ func cJobToModel(job *C.xce_job_t) *models.JobInfo { } } } + // PEND reason: the scheduler pushes these via setpreason; the controller + // stores them on job->pend_reasons. Maps to models.JobInfo.PendingReason. + if job.pend_reasons != nil { + pr := job.pend_reasons + r.pendReason = &rawPendReason{ + MainReason: int(pr.main_reason), + MainReasonName: pendReasonName(int(pr.main_reason)), + } + for i := 0; i < int(pr.nhost_reasons); i++ { + hr := C.job_host_reason(pr, C.int(i)) + if hr != nil && hr.host != nil { + r.pendReason.HostReasons = append(r.pendReason.HostReasons, + rawHostReason{Host: goString(hr.host), Reason: int(hr.reason)}) + } + } + for i := 0; i < int(pr.nres_reasons); i++ { + rr := C.job_res_reason(pr, C.int(i)) + if rr != nil && rr.resource != nil { + r.pendReason.ResReasons = append(r.pendReason.ResReasons, + rawResReason{Resource: goString(rr.resource), + Host: goString(rr.host), Reason: int(rr.reason)}) + } + } + } return rawJobToModel(r) } diff --git a/src/frontend/apiserver/backend/convert_pure.go b/src/frontend/apiserver/backend/convert_pure.go index 4e5be98..ce6cb0f 100644 --- a/src/frontend/apiserver/backend/convert_pure.go +++ b/src/frontend/apiserver/backend/convert_pure.go @@ -159,6 +159,47 @@ type rawSubmitData struct { startInfo *rawStartInfo } +// rawHostReason mirrors the C xce_host_reason_t (one host-level PEND reason). +type rawHostReason struct { + Host string `json:"host"` + Reason int `json:"reason"` +} + +// rawResReason mirrors the C xce_res_reason_t (one resource-level PEND reason). +type rawResReason struct { + Resource string `json:"resource"` + Host string `json:"host"` + Reason int `json:"reason"` +} + +// rawPendReason mirrors the C xce_job_preason_t attached to a PEND job by the +// controller (scheduler → setpreason → job->pend_reasons). JSON-tagged so the +// marshaled value lands in models.JobInfo.PendingReason as a JSONB object — +// processJobData's JSONBJFields pass then surfaces it in the API response. +type rawPendReason struct { + MainReason int `json:"main_reason"` + MainReasonName string `json:"main_reason_name"` + HostReasons []rawHostReason `json:"host_reasons,omitempty"` + ResReasons []rawResReason `json:"res_reasons,omitempty"` +} + +// pendReasonName maps a PREASON_* main_reason code to a readable name. The +// codes are the PEND reason bitfield of xce_job_preason_t (PREASON_BAD_RESREQ/ +// NO_RESOURCE/MATCH_FAIL, src/include/preason.h). Unknown codes are reported +// verbatim so a new scheduler reason never silently disappears. +func pendReasonName(code int) string { + switch code { + case 1: // PREASON_BAD_RESREQ + return "bad_resreq" + case 2: // PREASON_NO_RESOURCE + return "no_resource" + case 3: // PREASON_MATCH_FAIL + return "match_fail" + default: + return fmt.Sprintf("unknown(%d)", code) + } +} + // rawJobData mirrors the C xce_job_t fields used by the Go model. type rawJobData struct { jobID int64 @@ -170,6 +211,7 @@ type rawJobData struct { finishTime int64 submit *rawSubmitData execHosts []string + pendReason *rawPendReason } // rawWorkerData mirrors the C xce_worker_t fields used by the Go model. @@ -258,6 +300,11 @@ func rawJobToModel(r rawJobData) *models.JobInfo { data, _ := json.Marshal(execHostsJSON{HostSlots: slots, NumHosts: len(slots)}) info.ExecHosts = models.Jsonb{String: string(data)} } + if r.pendReason != nil { + if data, err := json.Marshal(r.pendReason); err == nil { + info.PendingReason = models.Jsonb{String: string(data)} + } + } return info } diff --git a/src/frontend/apiserver/backend/convert_test.go b/src/frontend/apiserver/backend/convert_test.go index a3ff57e..e8079b2 100644 --- a/src/frontend/apiserver/backend/convert_test.go +++ b/src/frontend/apiserver/backend/convert_test.go @@ -2,6 +2,7 @@ package backend import ( "apiserver/models" + "encoding/json" "testing" ) @@ -96,6 +97,80 @@ func TestRawJobToModel_NilSubmit(t *testing.T) { } } +// PEND reason: rawJobData.pendReason must marshal into +// JobInfo.PendingReason as a JSONB string (processJobData parses it via +// JSONBJFields). No pendReason → PendingReason stays empty. +func TestRawJobToModel_PendReason(t *testing.T) { + r := rawJobData{ + jobID: 7, + state: 1, // JOB_STAT_PEND + pendReason: &rawPendReason{ + MainReason: 2, // PREASON_NO_RESOURCE + MainReasonName: "no_resource", + HostReasons: []rawHostReason{{Host: "t1", Reason: 2}}, + ResReasons: []rawResReason{{Resource: "slots", Host: "t1", Reason: 2}}, + }, + } + + info := rawJobToModel(r) + + if info.PendingReason.String == "" { + t.Fatal("PendingReason should be populated for a PEND job, got empty") + } + var decoded struct { + MainReason int `json:"main_reason"` + MainReasonName string `json:"main_reason_name"` + HostReasons []struct { + Host string `json:"host"` + Reason int `json:"reason"` + } `json:"host_reasons"` + ResReasons []struct { + Resource string `json:"resource"` + Host string `json:"host"` + Reason int `json:"reason"` + } `json:"res_reasons"` + } + if err := json.Unmarshal([]byte(info.PendingReason.String), &decoded); err != nil { + t.Fatalf("PendingReason should be valid JSON, got err: %v", err) + } + if decoded.MainReason != 2 || decoded.MainReasonName != "no_resource" { + t.Errorf("main_reason: got %d/%q, want 2/no_resource", + decoded.MainReason, decoded.MainReasonName) + } + if len(decoded.HostReasons) != 1 || decoded.HostReasons[0].Host != "t1" { + t.Errorf("host_reasons: got %+v, want [t1]", decoded.HostReasons) + } + if len(decoded.ResReasons) != 1 || decoded.ResReasons[0].Resource != "slots" { + t.Errorf("res_reasons: got %+v, want [slots]", decoded.ResReasons) + } +} + +func TestRawJobToModel_PendReasonNil(t *testing.T) { + info := rawJobToModel(rawJobData{jobID: 8, state: 1}) + if info.PendingReason.String != "" { + t.Errorf("PendingReason should be empty without a pendReason, got %q", + info.PendingReason.String) + } +} + +// pendReasonName maps PREASON_* codes; unknown codes are surfaced verbatim. +func TestPendReasonName(t *testing.T) { + cases := []struct { + code int + want string + }{ + {1, "bad_resreq"}, + {2, "no_resource"}, + {3, "match_fail"}, + {42, "unknown(42)"}, + } + for _, c := range cases { + if got := pendReasonName(c.code); got != c.want { + t.Errorf("pendReasonName(%d): got %q, want %q", c.code, got, c.want) + } + } +} + func TestRawJobToModel_Timestamps(t *testing.T) { r := rawJobData{ jobID: 1, diff --git a/src/frontend/apiserver/models/jobs.go b/src/frontend/apiserver/models/jobs.go index e8eaee3..8a5a969 100644 --- a/src/frontend/apiserver/models/jobs.go +++ b/src/frontend/apiserver/models/jobs.go @@ -206,6 +206,7 @@ var JOB_DEFAULT_COLUMN = []string{ "job_name", "command", "submit_time", + "pending_reason", } var ARRAY_JOB_DEFAULT_COLUMN = []string{