forked from lei.chen/myxce
fix(configs): add worker-builtin load indices missing from resources.yaml
worker/src/resource_sync.c init_builtin_load_indices registers 11 load indices (r15s/r1m/r15m/ut/pg/ls/it/swp/mem/tmp/io) but resources.yaml only defined r1m/r15s/mem/tmp. The 7 missing ones (r15m/ut/pg/ls/it/swp/io) caused controller "update_resval: Success" ERRORs and scheduler "unknown resource in per-worker resvals update" warnings, leaving the worker resource view incomplete. Add the 7 definitions matching the existing r1m/r15s pattern. Docs: log bug-1494 (all jobs PEND, scheduler never dispatches; after this config fix jobs still PEND — remaining suspect is the scheduler→controller join path after restart), update design doc + verification manual known-issue tables and root CLAUDE.md cluster-verification status. Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
226259a52b
commit
62dee0edc1
6 changed files with 130 additions and 5 deletions
|
|
@ -14564,5 +14564,64 @@
|
|||
],
|
||||
"occurrences": 4,
|
||||
"last_seen": "2026-08-02"
|
||||
},
|
||||
{
|
||||
"id": "bug-1492",
|
||||
"timestamp": "2026-08-02T12:20:00+08:00",
|
||||
"error_message": "scheduler_list_nodes/cluster_metrics memory_bytes_total=8224821248000000 (~8.2 PB) \u2014 unrealistically large; expected ~8.2 GB",
|
||||
"file": "src/frontend/schedulerclient/httpclient.go",
|
||||
"root_cause": "PENDING USER JUDGMENT. C hostattrs_t.maxmem (src/include/xce/types.h:191) is a double carrying 8224821248 in live data. If that value is already bytes (~8.2GB), schedulerclient's mbToBytes(1e6) multiply in ListNodes makes it 8.2 PB. Could also be debug-info or another cause. User will decide next step.",
|
||||
"fix": "NOT DECIDED \u2014 awaiting user judgment on whether this is a real bug, debug artifact, or other cause.",
|
||||
"tags": [
|
||||
"schedulerclient",
|
||||
"memory-units",
|
||||
"host",
|
||||
"observation"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1491"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-02T12:20:00+08:00"
|
||||
},
|
||||
{
|
||||
"id": "bug-1493",
|
||||
"timestamp": "2026-08-02T12:35:00+08:00",
|
||||
"error_message": "scheduler_get_job returns the first job in the full list regardless of requested ID. Requested 73929981243162624 (sleep666) and 73929907581747200 (verify3) both returned verify1 (73929906127372288).",
|
||||
"file": "src/frontend/apiserver/controllers/jobs.go (GetJobByIDs) + src/frontend/schedulerclient/httpclient.go GetJob",
|
||||
"root_cause": "GET /xce/v1/jobs/:jobids calls GetJobByIDs which builds job_id eq filter, but live response returns ALL jobs (count:5) \u2014 the filter is not applied by the QueryData/store layer. schedulerclient.GetJob then takes jobs[0], masking it as 'returns wrong job'. Filter application on job_id appears broken in the query layer.",
|
||||
"fix": "NOT YET FIXED \u2014 awaiting user judgment. Likely fix in query/store layer to honor job_id filter, or client-side post-filter by ID.",
|
||||
"tags": [
|
||||
"apiserver",
|
||||
"schedulerclient",
|
||||
"get-job",
|
||||
"filter",
|
||||
"query-layer"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1491"
|
||||
],
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-08-02T12:35:00+08:00"
|
||||
},
|
||||
{
|
||||
"id": "bug-1494",
|
||||
"timestamp": "2026-08-02",
|
||||
"error_message": "All jobs stay PEND \u2014 scheduler never dispatches. New jobs (submitted while scheduler running) also stay PEND; scheduler log has 0 job/alloc lines.",
|
||||
"file": "src/controller/sched.c, src/configs/resources.yaml",
|
||||
"root_cause": "WORKER-HARDCODED LOAD INDICES missing from resources.yaml (worker/src/resource_sync.c init_builtin_load_indices registers r15m/ut/pg/ls/it/swp/io; resources.yaml only had r1m/r15s/mem/tmp) \u2192 controller \"update_resval: Success\" ERROR + scheduler \"unknown resource in per-worker resvals update\". Fixed resources, rebuilt cluster.bin (a7824e0586d0), restarted controller+scheduler+apiserver \u2014 jobs STILL PEND. NEW scheduler (292698) went completely silent (no sched_update, no job log), unlike the old scheduler which got worker resvals every 15s. tcpdump earlier proved controller DOES push a 11720B payload (jobs) to scheduler on join. So the remaining suspect is the scheduler\u2192controller join/connection path after restart, not resource mismatch. NOT ROOT-CAUSED.",
|
||||
"fix": "NOT FIXED. Partial: added 7 missing resources to resources.yaml + rebuilt cluster.bin (a7824e0586d0) + restarted components. Remaining: investigate why the restarted scheduler does not connect/join the controller (new scheduler silent).",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"pend",
|
||||
"dispatch",
|
||||
"join",
|
||||
"open"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1492"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-02"
|
||||
}
|
||||
]
|
||||
Loading…
Add table
Add a link
Reference in a new issue