forked from lei.chen/myxce
fix(configs): add worker-builtin load indices missing from resources.yaml
worker/src/resource_sync.c init_builtin_load_indices registers 11 load indices (r15s/r1m/r15m/ut/pg/ls/it/swp/mem/tmp/io) but resources.yaml only defined r1m/r15s/mem/tmp. The 7 missing ones (r15m/ut/pg/ls/it/swp/io) caused controller "update_resval: Success" ERRORs and scheduler "unknown resource in per-worker resvals update" warnings, leaving the worker resource view incomplete. Add the 7 definitions matching the existing r1m/r15s pattern. Docs: log bug-1494 (all jobs PEND, scheduler never dispatches; after this config fix jobs still PEND — remaining suspect is the scheduler→controller join path after restart), update design doc + verification manual known-issue tables and root CLAUDE.md cluster-verification status. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
226259a52b
commit
62dee0edc1
6 changed files with 130 additions and 5 deletions
|
|
@ -14564,5 +14564,64 @@
|
|||
],
|
||||
"occurrences": 4,
|
||||
"last_seen": "2026-08-02"
|
||||
},
|
||||
{
|
||||
"id": "bug-1492",
|
||||
"timestamp": "2026-08-02T12:20:00+08:00",
|
||||
"error_message": "scheduler_list_nodes/cluster_metrics memory_bytes_total=8224821248000000 (~8.2 PB) \u2014 unrealistically large; expected ~8.2 GB",
|
||||
"file": "src/frontend/schedulerclient/httpclient.go",
|
||||
"root_cause": "PENDING USER JUDGMENT. C hostattrs_t.maxmem (src/include/xce/types.h:191) is a double carrying 8224821248 in live data. If that value is already bytes (~8.2GB), schedulerclient's mbToBytes(1e6) multiply in ListNodes makes it 8.2 PB. Could also be debug-info or another cause. User will decide next step.",
|
||||
"fix": "NOT DECIDED \u2014 awaiting user judgment on whether this is a real bug, debug artifact, or other cause.",
|
||||
"tags": [
|
||||
"schedulerclient",
|
||||
"memory-units",
|
||||
"host",
|
||||
"observation"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1491"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-02T12:20:00+08:00"
|
||||
},
|
||||
{
|
||||
"id": "bug-1493",
|
||||
"timestamp": "2026-08-02T12:35:00+08:00",
|
||||
"error_message": "scheduler_get_job returns the first job in the full list regardless of requested ID. Requested 73929981243162624 (sleep666) and 73929907581747200 (verify3) both returned verify1 (73929906127372288).",
|
||||
"file": "src/frontend/apiserver/controllers/jobs.go (GetJobByIDs) + src/frontend/schedulerclient/httpclient.go GetJob",
|
||||
"root_cause": "GET /xce/v1/jobs/:jobids calls GetJobByIDs which builds job_id eq filter, but live response returns ALL jobs (count:5) \u2014 the filter is not applied by the QueryData/store layer. schedulerclient.GetJob then takes jobs[0], masking it as 'returns wrong job'. Filter application on job_id appears broken in the query layer.",
|
||||
"fix": "NOT YET FIXED \u2014 awaiting user judgment. Likely fix in query/store layer to honor job_id filter, or client-side post-filter by ID.",
|
||||
"tags": [
|
||||
"apiserver",
|
||||
"schedulerclient",
|
||||
"get-job",
|
||||
"filter",
|
||||
"query-layer"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1491"
|
||||
],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-08-02T12:35:00+08:00"
|
||||
},
|
||||
{
|
||||
"id": "bug-1494",
|
||||
"timestamp": "2026-08-02",
|
||||
"error_message": "All jobs stay PEND \u2014 scheduler never dispatches. New jobs (submitted while scheduler running) also stay PEND; scheduler log has 0 job/alloc lines.",
|
||||
"file": "src/controller/sched.c, src/configs/resources.yaml",
|
||||
"root_cause": "WORKER-HARDCODED LOAD INDICES missing from resources.yaml (worker/src/resource_sync.c init_builtin_load_indices registers r15m/ut/pg/ls/it/swp/io; resources.yaml only had r1m/r15s/mem/tmp) \u2192 controller \"update_resval: Success\" ERROR + scheduler \"unknown resource in per-worker resvals update\". Fixed resources, rebuilt cluster.bin (a7824e0586d0), restarted controller+scheduler+apiserver \u2014 jobs STILL PEND. NEW scheduler (292698) went completely silent (no sched_update, no job log), unlike the old scheduler which got worker resvals every 15s. tcpdump earlier proved controller DOES push a 11720B payload (jobs) to scheduler on join. So the remaining suspect is the scheduler\u2192controller join/connection path after restart, not resource mismatch. NOT ROOT-CAUSED.",
|
||||
"fix": "NOT FIXED. Partial: added 7 missing resources to resources.yaml + rebuilt cluster.bin (a7824e0586d0) + restarted components. Remaining: investigate why the restarted scheduler does not connect/join the controller (new scheduler silent).",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"pend",
|
||||
"dispatch",
|
||||
"join",
|
||||
"open"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1492"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-02"
|
||||
}
|
||||
]
|
||||
|
|
@ -7290,3 +7290,12 @@
|
|||
|------|--------|---------|---------|--------|
|
||||
| 12:05 | 查询集群机器: MCP 只读工具(list_nodes/queues/jobs)返回空字段 → 根因: apiserver 返回 OrderMap 列式格式 {Fields,Data}, schedulerclient 用扁平 struct 解析 | src/frontend/schedulerclient/httpclient.go | 记 bug-1491, 未修 | ~1k tok |
|
||||
| 05:00 | Fixed bug-1491: schedulerclient read clients now parse apiserver OrderMap columnar rows (apiRow + rowStr/rowInt/rowInt64/rowFloat). TDD: 3 ordermap_test.go red→green; live-verified on t1 (nodes/queues/jobs fields all correct) | src/frontend/schedulerclient/httpclient.go, ordermap_test.go | green | ~1k tok |
|
||||
| 12:20 | 修复生效(list_nodes 出真实数据); 发现 memory_bytes_total ~8.2PB 异常 → 已记 bug-1492, 用户自判是否真 bug/调试残留 | .wolf/buglog.json | recorded, no fix | ~0.3k tok |
|
||||
| 12:35 | 提交 sleep666 作业成功(73929981243162624, PEND); 发现 get_job 不按 ID 过滤(恒返第一条) → 记 bug-1493, apiserver GetJobByIDs job_id filter 未生效 | .wolf/buglog.json | logged, no fix | ~0.5k tok |
|
||||
| 07:10 | PEND 根因调查:tcpdump 证明 controller 在 scheduler join 后推送了 11720B 大包(=push_jobs 含作业),作业确实发出;但 scheduler 侧 unknown resource(worker 上报 r15m/ut/pg/ls/it/swp/io 在 resources.yaml 未定义)→ 疑资源不匹配阻塞调度。授权修复 | t1 诊断 | 进行中 | ~1k tok |
|
||||
| 07:11 | 修复第一步:本地 resources.yaml 补全 worker 硬编码的 7 个负载指标(r15m/ut/pg/ls/it/swp/io),yaml 校验通过 | src/configs/resources.yaml | done | ~0.5k tok |
|
||||
| 07:12 | 修复第二步:t1 resources.yaml 已替换 + cluster.bin 重建(新 sha a7824e0586d0,旧 044f1b3a),待部署到 controller 并重启 | t1 /opt/newxce/config | done | ~0.5k tok |
|
||||
| 07:12 | 修复第三步:controller 已停,新 cluster.bin.a7824e0586d0 已部署到 t1/conf + current 指针更新 | t1 controller conf | done | ~0.3k tok |
|
||||
| 07:13 | 修复第四步:controller 重启成功(新 config tag=a7824e0586d0,CTRL_PRIMARY term=2),待重启 scheduler | t1 controller | done | ~0.3k tok |
|
||||
| 07:16 | 修复第五步:controller 重启导致 apiserver(8080) 退出(无监听);新 scheduler(292698) 启动后暂未报 unknown resource(新配置疑似生效)。待重启 apiserver | t1 | 进行中 | ~0.3k tok |
|
||||
| 07:35 | 收尾:bug-1494 记录(作业 PEND + scheduler join 链路疑点);resources.yaml 补全 7 个负载指标已提交本地;更新设计文档/验证手册/CLAUDE.md 已知问题表 | 多处 | 进行中 | ~1k tok |
|
||||
|
|
|
|||
|
|
@ -49,7 +49,7 @@ For details on any topic below, read the corresponding steering file in `.kiro/s
|
|||
- `src/worker/` — See `src/worker/CLAUDE.md`
|
||||
- `src/scheduler/` — See `src/scheduler/CLAUDE.md`
|
||||
- `src/lib/` — See `src/lib/CLAUDE.md`
|
||||
- `src/frontend/schedulerclient/`, `src/frontend/mcp-server/` — **Phase 0-1 已完成**(Phase 0: 2026-07-26,Phase 1: 2026-08-01)。`schedulerclient` 是独立 Go library,提供读 + 写 Client(6 只读 + 4 写:SubmitJob/CancelJob/HoldJob/ReleaseJob,resty HTTP 实现,33+12 单测)。`mcp-server` 是 MCP stdio 二进制(`github.com/mark3labs/mcp-go@v0.57.0`),通过 `schedulerclient` 调用 `apiserver` REST API,暴露 10 个 MCP Tool(6 只读 + 4 写,写工具含 `dry_run` review 模式 + `general_serial` 作业模板 registry)。Pure Go, `CGO_ENABLED=0`。Phase 1 需 `mcp_operator` RBAC 角色(`config/rbac.yaml`)。总体架构与分阶段计划见 `docs/Scheduler-Agent-设计文档.md`。**集群验证(2026-08-01)**:t1 单节点 controller 集群上,读操作 + submit 已通过 mcp_operator API key 验证;hold/release/cancel 因 apiserver control 链路预存 bug(HTTP 500)待修(bug-1490)。
|
||||
- `src/frontend/schedulerclient/`, `src/frontend/mcp-server/` — **Phase 0-1 已完成**(Phase 0: 2026-07-26,Phase 1: 2026-08-01)。`schedulerclient` 是独立 Go library,提供读 + 写 Client(6 只读 + 4 写:SubmitJob/CancelJob/HoldJob/ReleaseJob,resty HTTP 实现,33+12 单测)。`mcp-server` 是 MCP stdio 二进制(`github.com/mark3labs/mcp-go@v0.57.0`),通过 `schedulerclient` 调用 `apiserver` REST API,暴露 10 个 MCP Tool(6 只读 + 4 写,写工具含 `dry_run` review 模式 + `general_serial` 作业模板 registry)。Pure Go, `CGO_ENABLED=0`。Phase 1 需 `mcp_operator` RBAC 角色(`config/rbac.yaml`)。总体架构与分阶段计划见 `docs/Scheduler-Agent-设计文档.md`。**集群验证(2026-08-01/02)**:t1 单节点 controller 集群上,读操作(含 OrderMap 解析修复 bug-1491)+ submit 已通过 mcp_operator API key 验证;**作业全部 PEND 不调度**(bug-1494,疑 scheduler→controller join 链路,已补 resources.yaml 但未解决);hold/release/cancel 因 apiserver control 链路预存 bug(HTTP 500)待修(bug-1490)。
|
||||
|
||||
# Agentic SDLC and Spec-Driven Development
|
||||
|
||||
|
|
|
|||
|
|
@ -551,11 +551,11 @@ Agent loop 跨多轮调度器轮次保持上下文:
|
|||
|
||||
| Bug | 现象 | 修复 |
|
||||
|-----|------|------|
|
||||
| whitelist 缺 `/xce/v1/apikeys` | API Key 生成端点 403 "URL is not allowed" | `middleware/whitelist.go` 加 `/xce/v1/apikeys`(bug-1489) |
|
||||
| control 的 `job_status` int64 断言 | control 返回 "Invalid job_status" | `controllers/jobs.go` 加 `normalizeJobStatus()` 兼容 int/int64/float64 |
|
||||
| whitelist 缺 `/xce/v1/apikeys` | API Key 生成端点 403 "URL is not allowed" | `middleware/whitelist.go` 加 `/xce/v1/apikeys`(bug-1489,已修) |
|
||||
| control 的 `job_status` int64 断言 | control 返回 "Invalid job_status" | `controllers/jobs.go` 加 `normalizeJobStatus()` 兼容 int/int64/float64(已修) |
|
||||
| control 链路 500(未修) | hold/release/cancel 返回 500,请求未达 controller | 待排查(bug-1490) |
|
||||
|
||||
### 10.3 未决决策项
|
||||
| 读接口 OrderMap 解析(已修) | `scheduler_list_nodes/queues/jobs` 实体数量对但字段全空 | `httpclient.go` 用 `apiRow` 解析列式格式(bug-1491,已修 + 真实验证) |
|
||||
| 作业全部 PEND(未修) | 提交的作业从不被调度(scheduler 日志 0 job 行;新作业也 PEND) | 已补 resources.yaml 缺失的 worker 负载指标 + 重建 cluster.bin;剩余疑 scheduler→controller join 链路(bug-1494) |
|
||||
|
||||
### 10.3 未决决策项
|
||||
|
||||
|
|
|
|||
|
|
@ -31,6 +31,61 @@ resources:
|
|||
order: decrease
|
||||
description: "15-second load average (built-in dynamic load index)"
|
||||
|
||||
# Remaining worker-builtin load indices (worker/src/resource_sync.c
|
||||
# init_builtin_load_indices): without these definitions, every worker
|
||||
# resval update for them is dropped (controller "update_resval: Success"
|
||||
# ERROR + scheduler "unknown resource ... in per-worker resvals update"),
|
||||
# leaving the worker's resource view incomplete so the scheduler cannot
|
||||
# allocate any job (all stay PEND). bug-1492.
|
||||
- name: r15m
|
||||
type: numeric
|
||||
interval: 60
|
||||
consumable: false
|
||||
order: decrease
|
||||
description: "15-minute load average (built-in dynamic load index)"
|
||||
|
||||
- name: ut
|
||||
type: numeric
|
||||
interval: 60
|
||||
consumable: false
|
||||
order: decrease
|
||||
description: "user-mode CPU time ratio (built-in dynamic load index)"
|
||||
|
||||
- name: pg
|
||||
type: numeric
|
||||
interval: 60
|
||||
consumable: false
|
||||
order: decrease
|
||||
description: "page rate (built-in dynamic load index)"
|
||||
|
||||
- name: ls
|
||||
type: numeric
|
||||
interval: 60
|
||||
consumable: false
|
||||
order: decrease
|
||||
description: "login session count (built-in dynamic load index)"
|
||||
|
||||
- name: it
|
||||
type: numeric
|
||||
interval: 60
|
||||
consumable: false
|
||||
order: decrease
|
||||
description: "idle time ratio (built-in dynamic load index)"
|
||||
|
||||
- name: swp
|
||||
type: numeric
|
||||
interval: 60
|
||||
consumable: false
|
||||
order: decrease
|
||||
description: "swap usage (built-in dynamic load index)"
|
||||
|
||||
- name: io
|
||||
type: numeric
|
||||
interval: 60
|
||||
consumable: false
|
||||
order: decrease
|
||||
description: "I/O wait ratio (built-in dynamic load index)"
|
||||
|
||||
- name: tmp
|
||||
type: numeric
|
||||
interval: 60
|
||||
|
|
|
|||
|
|
@ -338,7 +338,9 @@ go run /tmp/demo.go
|
|||
| 问题 | 状态 | Bug |
|
||||
|------|------|-----|
|
||||
| **job control (cancel/hold/release) HTTP 500** | 未修。请求未达 controller(controller 日志无 control 事件)。已修 `job_status` int64 断言(从 "Invalid job_status" 变为 500),500 本体待排查 | bug-1490 |
|
||||
| **作业全部 PEND(不调度)** | 未修。scheduler 日志 0 job 行。已补 resources.yaml 缺失的 7 个 worker 负载指标(r15m/ut/pg/ls/it/swp/io)+ 重建 cluster.bin + 重启组件;作业仍 PEND。剩余疑 scheduler→controller join 链路(新 scheduler 重启后静默,无 sched_update) | bug-1494 |
|
||||
| **3 节点 HA 选举死锁** | 遗留 term 状态(`ha-term.bin` 残留 `voted_for=t1, term=1`)+ standby 无 pool 导致所有节点 STANDBY。单节点绕过 | 记录在会话 |
|
||||
| **读接口字段全空(已修)** | `scheduler_list_nodes/queues/jobs` 数量对但字段空(OrderMap 列式格式未解析) | bug-1491(已修) |
|
||||
| **apikeys 路由 whitelist** | 已修:`WhitelistPOSTURL` 加 `/xce/v1/apikeys` | bug-1489(已修) |
|
||||
| **mcp_operator 缺 queue:control** | 已修:rbac.yaml 加 `queue:control`(job-control handler 也检查它) | — |
|
||||
| **HA 3 节点需 xceadmin push** | `xceadmin config push` 有 discovery.py 解包 bug(deployment.md 已知) | 已知 |
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue