fix(schedulerclient): node memory_bytes_total no longer ×1e6 (bug-1514)
The apiserver's host max_mem is already in bytes (the worker reads sysinfo().totalram × mem_unit → bytes and passes it through unchanged via controller.capnp), but ListNodes treated it as MB and multiplied by mbToBytes, inflating each node's reported memory to ~8.2e15 (~7.3 PiB). Now reports the raw bytes (~7.7 GiB per node). - httpclient.go: drop ×mbToBytes for node max_mem; clarify mbToBytes is job-submit-path only (job max_mem is never populated → stays 0) - ordermap_test.go: correct assertion that had encoded the buggy value - httpclient_test.go: regression test TestListNodeMemoryBytesKeepsByteUnit - buglog.json: bug-1514 + close-tags for the bug-1494 family (prior session) Also add MCP demo deliverables: - docs/claude-code-demo-guide.md: live Claude Code demo guide (7 scenes) - docs/mcp-demo-design.md: leadership demo design - docs/mcp-demo-run.py: add --slow/--color flags for screen recording Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
lei.chen
parent
bbe8ed3335
commit
06887985b4
7 changed files with 423 additions and 36 deletions
|
|
@ -14612,17 +14612,19 @@
|
|||
"root_cause": "WORKER-HARDCODED LOAD INDICES missing from resources.yaml (worker/src/resource_sync.c init_builtin_load_indices registers r15m/ut/pg/ls/it/swp/io; resources.yaml only had r1m/r15s/mem/tmp) → controller \"update_resval: Success\" ERROR + scheduler \"unknown resource in per-worker resvals update\". Fixed resources, rebuilt cluster.bin (a7824e0586d0), restarted controller+scheduler+apiserver — jobs STILL PEND. NEW scheduler (292698) went completely silent (no sched_update, no job log), unlike the old scheduler which got worker resvals every 15s. tcpdump earlier proved controller DOES push a 11720B payload (jobs) to scheduler on join. So the remaining suspect is the scheduler→controller join/connection path after restart, not resource mismatch. NOT ROOT-CAUSED.",
|
||||
"fix": "FIXED (via bug-1495 root cause): put_resvals used idxs[i] \"<= 0\" which treated the VALID first resource index 0 (slots) as invalid, skipping its val fill and then SEGV on the write. Changed to \"idx < 0\" + NULL-guard on the write loop. After fix, a locally-built scheduler connected to t1 controller and dispatched all 5 jobs to node t1 (exec_hosts populated); all jobs ran and exited (status EXIT). Full scheduling chain verified working.",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"pend",
|
||||
"closed",
|
||||
"dispatch",
|
||||
"fixed",
|
||||
"join",
|
||||
"fixed"
|
||||
"pend",
|
||||
"scheduler"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1492"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-03"
|
||||
"last_seen": "2026-08-03",
|
||||
"fix_validation": "FIXED + closed 2026-08-17. Root family (all-jobs-PEND) fully resolved: put_resvals idx<0 (bug-1495), dispatch-fail re-PEND (bug-1510), worker-rejoin slots in-place update (bug-1511), code-0 ack over-dispatch (bug-1512). Cluster-verified 08-12: 24 long jobs -> 8/8/8 across 3 workers, no duplicates, all DONE. Later PEND observations were the limit1 3-slot global quota (config), not this bug."
|
||||
},
|
||||
{
|
||||
"id": "bug-1495",
|
||||
|
|
@ -14632,17 +14634,19 @@
|
|||
"root_cause": "put_resvals (resource.c:89-91) writes xvals[i]->u.fvalue unconditionally, but the first loop only fills xvals[i] for idxs[i] > 0. When accounts[i]->residx is invalid (<=0, i.e. the resource name was NOT found in sched->resmgr at resource.c:422-431, so residx stays 0/uninitialized), the second loop dereferences a NULL xvals[i] -> SEGV. Stack: sched_allocator -> commit_decision -> sched_commit_decision -> sched_commit_job_alloc -> sched_charge_account -> sched_update_account -> sched_update_res_accounts -> put_resvals. Root trigger: a job/account references a resource name absent from the loaded resources.yaml (resource-convention mismatch between job and scheduler config).",
|
||||
"fix": "FIXED: resource.c put_resvals idxs[i] \"<= 0\" -> \"< 0\" (idx 0 = valid, e.g. slots), plus NULL-guard on xvals[i] write loop. Verified: scheduler no longer SEGVs and dispatched all 5 test jobs to t1 (exec_hosts populated, jobs EXIT).",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"segfault",
|
||||
"resource",
|
||||
"account",
|
||||
"fixed"
|
||||
"closed",
|
||||
"fixed",
|
||||
"resource",
|
||||
"scheduler",
|
||||
"segfault"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1494"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-03"
|
||||
"last_seen": "2026-08-03",
|
||||
"fix_validation": "FIXED + closed 2026-08-17 (sub-bug of bug-1494). put_resvals idxs[i] \"<= 0\" -> \"< 0\" (idx 0 is the built-in slots resource). Verified by 07ab056 fix + lib tests."
|
||||
},
|
||||
{
|
||||
"id": "bug-1496",
|
||||
|
|
@ -14692,17 +14696,19 @@
|
|||
"root_cause": "1) sched_job_is_ready 恒真 stub: pendJobs 里任何作业都被 elect; 2) 派发后 scheduler 内作业状态仍为 PEND (controller 从不发 schedStartJob, record_job_start_fact 的 AFTER_EVENT 通知被推迟到未实现的 apply pump), fcfs_update_job 在每次 job update 时把仍 PEND 的作业重新塞回 pendJobs; 3) fcfs_elect_job 每秒重扫重新 elect 已派发作业 → 重复 dispatch → worker 被 SIGTERM 杀",
|
||||
"fix": "FIXED: 1) sched_job_is_ready() 从恒真 stub 改为检查 JOB_STAT_PEND 且非挂起; 2) sched_commit_decision dispatch 分支把作业标记为 JOB_STAT_RUN（派发即离开 PEND 集合）; 3) fcfs_update_job 用 ordq_peek 去重防重复入队。新增 test_redispatch.c 回归测试 (8 用例)。实机验证: sleep 60 作业只 elect 1 次, 跑满 60s, exit 0 completed successfully (对比修复前 ~11s 被杀 143)。注: 作业上报状态仍受 controller apply-pump 未实现影响, 属独立 bug。",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"re-dispatch",
|
||||
"job-lifecycle",
|
||||
"closed",
|
||||
"fcfs",
|
||||
"fixed"
|
||||
"fixed",
|
||||
"job-lifecycle",
|
||||
"re-dispatch",
|
||||
"scheduler"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1494"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-03"
|
||||
"last_seen": "2026-08-03",
|
||||
"fix_validation": "FIXED + closed 2026-08-17. sched_job_is_ready() from always-true stub -> checks JOB_STAT_PEND; fcfs dedup never enqueues same job twice. Verified by test_handle_dispatch_fail_requeues_only_on_negative_code + cluster 8/8/8 run."
|
||||
},
|
||||
{
|
||||
"id": "bug-1499",
|
||||
|
|
@ -14959,11 +14965,12 @@
|
|||
"root_cause": "scheduler 的 sched_handle_notify switch 没有 NotifyData_data_jobDispatchFail 分支（default 丢弃）。controller 在 dispatch_a_job_ 拒绝派发（如 worker UNAVAIL/ep nil）时发 jobDispatchFail 通知，scheduler 却已通过 sched_commit_decision 把作业标为 RUN；通知被丢弃后作业保持 RUN 视图，sched_job_is_ready 不再选举它。与此前 bug-1494（put_resvals 修复）是同一 PEND 症状的另一个根因。",
|
||||
"fix": "handler.c 增加 NotifyData_data_jobDispatchFail case：日志 + 调用新函数 sched_requeue_pend_job()（释放 allocs/accounts、重置 JOB_STAT_PEND、经 plugin_job_update 重新入 fcfs pendJobs 队列），下一选举会话重新派发。测试 TU（test_handlers/test_job_array_expand/test_job_array_wire_e2e，白盒 include handler.c）加 stub。集群验证：worker down 时提交 job → 持续重试派发（code -10002）→ worker 恢复后派发成功 → EXIT。commit 6928f74。",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"pend",
|
||||
"closed",
|
||||
"dispatch",
|
||||
"fixed",
|
||||
"jobDispatchFail",
|
||||
"fixed"
|
||||
"pend",
|
||||
"scheduler"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1494",
|
||||
|
|
@ -14971,7 +14978,8 @@
|
|||
"bug-1498"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-11"
|
||||
"last_seen": "2026-08-11",
|
||||
"fix_validation": "FIXED + closed 2026-08-17 (sub-bug of bug-1494). Added NotifyData_data_jobDispatchFail case -> sched_requeue_pend_job() re-PENDs on real dispatch failure (commit 6928f74). Verified by unit test + cluster."
|
||||
},
|
||||
{
|
||||
"id": "bug-1511",
|
||||
|
|
@ -14981,19 +14989,21 @@
|
|||
"root_cause": "worker 死亡时 mark_worker_unavail 正确清零 slots=0；worker 重新 join 时 controller 推送新 worker（slots=8），scheduler 的 sched_update_workers 创建新 worker 后 sched_add_worker 用 key_add 遇 EEXIST 丢弃新对象，scheduler 永久保留旧的 slots=0 worker → 分配失败（unsatisfy slots total 0）→ 全部 PEND。与 bug-1510（dispatch-fail 不重试）同属 bug-1494 症状的另一根因。",
|
||||
"fix": "sched_update_workers 对已存在 worker 就地更新：交换新 resvals/resInUse/status 到现有 worker，释放被丢弃对象（注意 w->name 是 makeString 驻留串不能 free，否则 ASAN bad-free）；NULL-guard make_sched_worker。集群验证：scheduler 在 worker 死亡时启动（slots=0）→ worker 重新 join（slots=8）→ 卡住 PEND 作业全部派发完成，无 ASAN。commit 1a2bd3f。",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"pend",
|
||||
"slots",
|
||||
"worker-rejoin",
|
||||
"closed",
|
||||
"fixed",
|
||||
"key_add-eexist",
|
||||
"fixed"
|
||||
"pend",
|
||||
"scheduler",
|
||||
"slots",
|
||||
"worker-rejoin"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1494",
|
||||
"bug-1510"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-11"
|
||||
"last_seen": "2026-08-11",
|
||||
"fix_validation": "FIXED + closed 2026-08-17 (sub-bug of bug-1494). sched_update_workers updates existing worker in place (swap resvals/resInUse/status) so rejoin slots=8 is not EEXIST-dropped (commit 1a2bd3f). Note: makeString returns interned string, must not free. Verified by cluster rejoin test."
|
||||
},
|
||||
{
|
||||
"id": "bug-1512",
|
||||
|
|
@ -15003,21 +15013,23 @@
|
|||
"root_cause": "controller handle_jobdispatch (controller/job.c) 对每个派发的作业都发 NotifyData_data_jobDispatchFail：失败分支 code<0，但成功分支 else 也调 notify_dispatch(jobid, 0)——code=0 是成功 ack。scheduler sched_handle_notify 之前（bug-1510 修复 6928f74）无 code 判断，把成功 ack 也当失败 sched_requeue_pend_job → 每个成功派发的作业立即 re-PEND → 下次 session 重新选举 → 再派发，原实例继续跑 → 重复实例/过派发。是 bug-1510 修复引入的回归。",
|
||||
"fix": "sched_handle_notify 的 jobDispatchFail case 加 code 判断：fail->code >= 0 是成功 ack，当 no-op 消费（作业已在 commit 时标记 RUN）；只有 code < 0 才 sched_requeue_pend_job。加白盒单测 test_handle_dispatch_fail_requeues_only_on_negative_code（code=0 保持 RUN、code<0 re-PEND），原 15 测试全过。集群验证（新 scheduler 部署 t1）：24 个 sleep-300 作业均匀 8/8/8 分布、零重复、全部 DONE；dispatchFail 日志从 38 降到 0。commit e4aaef4。",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"over-dispatch",
|
||||
"duplicate",
|
||||
"multi-worker",
|
||||
"dispatchfail",
|
||||
"closed",
|
||||
"code-0-ack",
|
||||
"dispatchfail",
|
||||
"duplicate",
|
||||
"fixed",
|
||||
"multi-worker",
|
||||
"over-dispatch",
|
||||
"regression",
|
||||
"fixed"
|
||||
"scheduler"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1494",
|
||||
"bug-1510"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-13"
|
||||
"last_seen": "2026-08-13",
|
||||
"fix_validation": "FIXED + closed 2026-08-17 (sub-bug of bug-1494, regression from 1510 fix). Only re-PEND when fail->code < 0; code=0 is the success ack (no-op). Prevents multi-worker over-dispatch/duplicate instances (commit e4aaef4). Verified: 24 jobs -> 8/8/8 no duplicates."
|
||||
},
|
||||
{
|
||||
"id": "bug-1513",
|
||||
|
|
@ -15037,5 +15049,17 @@
|
|||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-13"
|
||||
},
|
||||
{
|
||||
"id": "bug-1514",
|
||||
"timestamp": "2026-08-22",
|
||||
"error_message": "MCP scheduler_list_nodes 的 memory_bytes_total 每节点报 8.22e15 字节（~7.3 PiB），实际节点仅 ~7.7 GiB——数值被放大约 1e6 倍；Claude Code 演示里节点内存显示成 ~7.5 TB",
|
||||
"file": "src/frontend/schedulerclient/httpclient.go",
|
||||
"root_cause": "apiserver 的 max_mem 字段本来就是字节（worker hardware_monitor.c:157 sysinfo().totalram×mem_unit → bytes，经 hostattrs_t.maxmem / controller.capnp / convert.go 原样传递），但 schedulerclient ListNodes 误当 MB，×mbToBytes(1e6) 后存为 bytes → 8.22e9→8.22e15",
|
||||
"fix": "httpclient.go:198 去掉 ×mbToBytes，直接 int64(rowInt(\"max_mem\"))；修正 :396 常量注释（mbToBytes 仅用于作业提交路径）。新增 httpclient_test.go TestListNodeMemoryBytesKeepsByteUnit + 修正 ordermap_test.go 固化旧 bug 行为的断言。已重建 /tmp/xce-mcp 并集群验证：t1/t2/t3 均 8224825344B（7.7 GiB），cluster_metrics 总内存 23 GiB。作业路径 :352 读取的 job max_mem 在 apiserver 从不赋值（恒 0），实际无害，未改动。",
|
||||
"tags": ["units", "memory", "schedulerclient", "mcp", "fix_validation", "fixed"],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-22"
|
||||
}
|
||||
]
|
||||
]
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue