fix(configs): add worker-builtin load indices missing from resources.yaml
worker/src/resource_sync.c init_builtin_load_indices registers 11 load indices (r15s/r1m/r15m/ut/pg/ls/it/swp/mem/tmp/io) but resources.yaml only defined r1m/r15s/mem/tmp. The 7 missing ones (r15m/ut/pg/ls/it/swp/io) caused controller "update_resval: Success" ERRORs and scheduler "unknown resource in per-worker resvals update" warnings, leaving the worker resource view incomplete. Add the 7 definitions matching the existing r1m/r15s pattern. Docs: log bug-1494 (all jobs PEND, scheduler never dispatches; after this config fix jobs still PEND — remaining suspect is the scheduler→controller join path after restart), update design doc + verification manual known-issue tables and root CLAUDE.md cluster-verification status. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
lei.chen
parent
226259a52b
commit
62dee0edc1
6 changed files with 130 additions and 5 deletions
|
|
@ -14564,5 +14564,64 @@
|
|||
],
|
||||
"occurrences": 4,
|
||||
"last_seen": "2026-08-02"
|
||||
},
|
||||
{
|
||||
"id": "bug-1492",
|
||||
"timestamp": "2026-08-02T12:20:00+08:00",
|
||||
"error_message": "scheduler_list_nodes/cluster_metrics memory_bytes_total=8224821248000000 (~8.2 PB) \u2014 unrealistically large; expected ~8.2 GB",
|
||||
"file": "src/frontend/schedulerclient/httpclient.go",
|
||||
"root_cause": "PENDING USER JUDGMENT. C hostattrs_t.maxmem (src/include/xce/types.h:191) is a double carrying 8224821248 in live data. If that value is already bytes (~8.2GB), schedulerclient's mbToBytes(1e6) multiply in ListNodes makes it 8.2 PB. Could also be debug-info or another cause. User will decide next step.",
|
||||
"fix": "NOT DECIDED \u2014 awaiting user judgment on whether this is a real bug, debug artifact, or other cause.",
|
||||
"tags": [
|
||||
"schedulerclient",
|
||||
"memory-units",
|
||||
"host",
|
||||
"observation"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1491"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-02T12:20:00+08:00"
|
||||
},
|
||||
{
|
||||
"id": "bug-1493",
|
||||
"timestamp": "2026-08-02T12:35:00+08:00",
|
||||
"error_message": "scheduler_get_job returns the first job in the full list regardless of requested ID. Requested 73929981243162624 (sleep666) and 73929907581747200 (verify3) both returned verify1 (73929906127372288).",
|
||||
"file": "src/frontend/apiserver/controllers/jobs.go (GetJobByIDs) + src/frontend/schedulerclient/httpclient.go GetJob",
|
||||
"root_cause": "GET /xce/v1/jobs/:jobids calls GetJobByIDs which builds job_id eq filter, but live response returns ALL jobs (count:5) \u2014 the filter is not applied by the QueryData/store layer. schedulerclient.GetJob then takes jobs[0], masking it as 'returns wrong job'. Filter application on job_id appears broken in the query layer.",
|
||||
"fix": "NOT YET FIXED \u2014 awaiting user judgment. Likely fix in query/store layer to honor job_id filter, or client-side post-filter by ID.",
|
||||
"tags": [
|
||||
"apiserver",
|
||||
"schedulerclient",
|
||||
"get-job",
|
||||
"filter",
|
||||
"query-layer"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1491"
|
||||
],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-08-02T12:35:00+08:00"
|
||||
},
|
||||
{
|
||||
"id": "bug-1494",
|
||||
"timestamp": "2026-08-02",
|
||||
"error_message": "All jobs stay PEND \u2014 scheduler never dispatches. New jobs (submitted while scheduler running) also stay PEND; scheduler log has 0 job/alloc lines.",
|
||||
"file": "src/controller/sched.c, src/configs/resources.yaml",
|
||||
"root_cause": "WORKER-HARDCODED LOAD INDICES missing from resources.yaml (worker/src/resource_sync.c init_builtin_load_indices registers r15m/ut/pg/ls/it/swp/io; resources.yaml only had r1m/r15s/mem/tmp) \u2192 controller \"update_resval: Success\" ERROR + scheduler \"unknown resource in per-worker resvals update\". Fixed resources, rebuilt cluster.bin (a7824e0586d0), restarted controller+scheduler+apiserver \u2014 jobs STILL PEND. NEW scheduler (292698) went completely silent (no sched_update, no job log), unlike the old scheduler which got worker resvals every 15s. tcpdump earlier proved controller DOES push a 11720B payload (jobs) to scheduler on join. So the remaining suspect is the scheduler\u2192controller join/connection path after restart, not resource mismatch. NOT ROOT-CAUSED.",
|
||||
"fix": "NOT FIXED. Partial: added 7 missing resources to resources.yaml + rebuilt cluster.bin (a7824e0586d0) + restarted components. Remaining: investigate why the restarted scheduler does not connect/join the controller (new scheduler silent).",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"pend",
|
||||
"dispatch",
|
||||
"join",
|
||||
"open"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1492"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-02"
|
||||
}
|
||||
]
|
||||
|
|
@ -7290,3 +7290,12 @@
|
|||
|------|--------|---------|---------|--------|
|
||||
| 12:05 | 查询集群机器: MCP 只读工具(list_nodes/queues/jobs)返回空字段 → 根因: apiserver 返回 OrderMap 列式格式 {Fields,Data}, schedulerclient 用扁平 struct 解析 | src/frontend/schedulerclient/httpclient.go | 记 bug-1491, 未修 | ~1k tok |
|
||||
| 05:00 | Fixed bug-1491: schedulerclient read clients now parse apiserver OrderMap columnar rows (apiRow + rowStr/rowInt/rowInt64/rowFloat). TDD: 3 ordermap_test.go red→green; live-verified on t1 (nodes/queues/jobs fields all correct) | src/frontend/schedulerclient/httpclient.go, ordermap_test.go | green | ~1k tok |
|
||||
| 12:20 | 修复生效(list_nodes 出真实数据); 发现 memory_bytes_total ~8.2PB 异常 → 已记 bug-1492, 用户自判是否真 bug/调试残留 | .wolf/buglog.json | recorded, no fix | ~0.3k tok |
|
||||
| 12:35 | 提交 sleep666 作业成功(73929981243162624, PEND); 发现 get_job 不按 ID 过滤(恒返第一条) → 记 bug-1493, apiserver GetJobByIDs job_id filter 未生效 | .wolf/buglog.json | logged, no fix | ~0.5k tok |
|
||||
| 07:10 | PEND 根因调查：tcpdump 证明 controller 在 scheduler join 后推送了 11720B 大包（=push_jobs 含作业），作业确实发出；但 scheduler 侧 unknown resource（worker 上报 r15m/ut/pg/ls/it/swp/io 在 resources.yaml 未定义）→ 疑资源不匹配阻塞调度。授权修复 | t1 诊断 | 进行中 | ~1k tok |
|
||||
| 07:11 | 修复第一步：本地 resources.yaml 补全 worker 硬编码的 7 个负载指标（r15m/ut/pg/ls/it/swp/io），yaml 校验通过 | src/configs/resources.yaml | done | ~0.5k tok |
|
||||
| 07:12 | 修复第二步：t1 resources.yaml 已替换 + cluster.bin 重建（新 sha a7824e0586d0，旧 044f1b3a），待部署到 controller 并重启 | t1 /opt/newxce/config | done | ~0.5k tok |
|
||||
| 07:12 | 修复第三步：controller 已停，新 cluster.bin.a7824e0586d0 已部署到 t1/conf + current 指针更新 | t1 controller conf | done | ~0.3k tok |
|
||||
| 07:13 | 修复第四步：controller 重启成功（新 config tag=a7824e0586d0，CTRL_PRIMARY term=2），待重启 scheduler | t1 controller | done | ~0.3k tok |
|
||||
| 07:16 | 修复第五步：controller 重启导致 apiserver(8080) 退出（无监听）；新 scheduler(292698) 启动后暂未报 unknown resource（新配置疑似生效）。待重启 apiserver | t1 | 进行中 | ~0.3k tok |
|
||||
| 07:35 | 收尾：bug-1494 记录（作业 PEND + scheduler join 链路疑点）；resources.yaml 补全 7 个负载指标已提交本地；更新设计文档/验证手册/CLAUDE.md 已知问题表 | 多处 | 进行中 | ~1k tok |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue