fix(worker): real maxswap/maxtmp detection; feat(scheduler): annotate multi-slot assigned jobs
bug-1554: init_host_attrs() replaces hardcoded placeholder stubs with real
detection -- maxswap from sysinfo()'s totalswap (reusing the same syscall
snapshot already taken for maxmem), maxtmp from a fresh statvfs("/tmp")
total-capacity measurement. No synthetic fallback on detection failure, so
a system with swap genuinely disabled correctly reports 0.
bug-1776: the scheduler simulator's "info worker" assigned-jobs listing now
annotates a job consuming more than one "slots" unit (e.g. "jobid*2"),
using per-host allocation data that was already being walked but discarded.
Falls back to the pre-existing per-job listing when jobs on the same worker
have differing slot counts, since a single suffix on a compressed range
would be ambiguous.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Hb6Edtj4KbS8CrgEWYdA2E
This commit is contained in:
parent
a6b735b6f2
commit
a35f597e06
9 changed files with 660 additions and 28 deletions
|
|
@ -15694,18 +15694,18 @@
|
|||
"error_message": "workerlist / resval show maxswap=1024 and maxtmp=2048 for every host regardless of actual hardware, unless overridden by a per-host config resource entry",
|
||||
"file": "src/worker/src/xcecd.c",
|
||||
"root_cause": "init_host_attrs() hardcodes host_attrs.maxswap = 1024 and host_attrs.maxtmp = 2048 as placeholder constants (lines ~349-350) -- unlike maxcpus/maxmem, which come from real hardware_monitor_* detection, these two fields have no real detection implemented yet. Confirmed live: a deploy's maxtmp showed exactly 2048 (the hardcoded stub) while maxswap showed 10000 (a per-host config override in hosts.yaml resource list, taking precedence via apply_join_state's mark_host_res loop -- proves the stub IS reachable/live whenever no override exists).",
|
||||
"fix": "Not fixed -- user explicitly deferred this (2026-08-13): real swap/tmp-disk detection is a bigger feature (needs actual sysinfo swap totals + statvfs on the configured tmp path) worth discussing design for, not a one-line fix. Flagging only per user's request.",
|
||||
"fix": "FIXED (2026-09-06, worker hardware detection). Real detection replaces both placeholder stubs: maxswap now comes from sysinfo()'s totalswap field (hardware_monitor_get_total_swap(), the same sysinfo() snapshot hardware_monitor_update() already takes for maxmem -- no extra syscall), and maxtmp now comes from a fresh statvfs(\"/tmp\") total-capacity measurement (hardware_monitor_get_disk_info()), mirroring node_information.c's node_info_get_tmp() own statvfs(\"/tmp\") convention but reporting total blocks*frsize instead of free space. init_host_attrs() (xcecd.c) wires both in, with NO synthetic fallback on detection failure (left at 0, matching maxcpus/maxmem's own existing no-fallback behavior) -- a system with swap genuinely disabled reports 0 correctly instead of a fabricated non-zero placeholder. New tests: test_hardware_monitor.c (7 tests, cross-checks each function's output against an independent sysinfo()/statvfs() call made directly in the test) and a new integration test in test_xcecd_startup_order.c pinning that init_host_attrs() picks up whatever these two functions report. Full worker suite: 9/9 pass.",
|
||||
"tags": [
|
||||
"fixed",
|
||||
"worker",
|
||||
"hardware-detection",
|
||||
"stub",
|
||||
"not-fixed",
|
||||
"maxswap",
|
||||
"maxtmp"
|
||||
"maxtmp",
|
||||
"hardware-detection",
|
||||
"maxswap"
|
||||
],
|
||||
"related_bugs": [],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-08-13T07:36:00Z"
|
||||
"occurrences": 2,
|
||||
"last_seen": "2026-09-06T21:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1634",
|
||||
|
|
@ -21079,6 +21079,51 @@
|
|||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-06T00:00:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1776",
|
||||
"timestamp": "2026-09-06T21:05:00Z",
|
||||
"error_message": "User improvement request (simulator REPL feedback): 'info worker's assigned jobs' listing gives no indication of how many 'slots' each assigned job actually consumes on that worker -- every assigned job looks identical regardless of whether it occupies 1 slot or many (e.g. a fully-packed 22-slot worker running one 2-slot job shows the exact same line shape as a 1-slot job).",
|
||||
"file": "src/scheduler/src/repl_info.c",
|
||||
"root_cause": "NOT A BUG -- feature gap. repl_info_worker_one()'s assigned-jobs loop already walks each job's xce_host_alloc_t (per-host allocation) to match it against the current worker's name, but only extracted job->jobid and discarded the xce_host_alloc_t itself (`break`) without ever inspecting its `allocs[]` array, which already carries a per-resource allocation list (same struct/pattern the controller's own get_nslots_in_alloc(), job.c, already scans for the identical 'slots' resname) -- so the slot count was always available with no new wire/plumbing needed, just never read at this call site.",
|
||||
"fix": "FIXED (2026-09-06). Added repl_worker_alloc_nslots() (repl_info.c) mirroring get_nslots_in_alloc()'s resname==\"slots\" scan (duplicated rather than shared, since that helper is file-static to job.c and this is a read-only REPL display concern). The assigned-jobs loop now also collects a parallel assigned_slots[] array (repl_append_int(), new int-array counterpart to the existing repl_append_u64()). When every assigned job on a worker consumed the SAME slot count and that count is >1, the existing compressed jobid-range line gets a trailing '*N' (e.g. 'assigned jobs: 0-14-1788751435-0*2'); a single-slot-per-job worker (the common case) renders identically to before. When assigned jobs on the SAME worker have DIFFERING slot counts, a single '*N' suffix on a compressed range would be ambiguous, so the listing falls back to the pre-existing uncompressed per-job lines (same fallback tier a too-small compression buffer already used), each annotated with its own count. New tests in test_repl_resource_table.c: test_info_worker_assigned_job_listing_appends_slot_count_when_uniform, test_info_worker_assigned_job_listing_falls_back_per_job_when_slots_differ. Full scheduler suite: 76/77 pass (the one failure, test_job_rusage_reconciles_mem_reservation_against_usage in test_core_events.c, is a pre-existing unrelated mem-reservation-subsystem failure already documented in bug-1762, untouched by this change).",
|
||||
"tags": [
|
||||
"scheduler",
|
||||
"repl",
|
||||
"sim",
|
||||
"assigned-jobs",
|
||||
"ux-improvement",
|
||||
"fixed"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1601",
|
||||
"bug-1591"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-06T21:05:00Z"
|
||||
},
|
||||
{
|
||||
"id": "bug-1777",
|
||||
"timestamp": "2026-09-06T21:10:00Z",
|
||||
"error_message": "xce_test workerlist's MaxSwap column can permanently disagree with the scheduler's own resmgr/replay view for the same worker: workerlist always shows the join-time hardware_monitor stub/detected value, never a hosts.yaml per-host maxswap override, while the scheduler always uses the override. User noticed this while comparing a scheduler simulator replay (maxswap=10000) against xce_test workerlist (MaxSwap=1.00 GB) for the same host and asked whether the two should be consistent.",
|
||||
"file": "src/controller/testsuite/test.c",
|
||||
"root_cause": "CONFIRMED via code trace (not yet reproduced against a live pc13v/replay artifact -- none existed on disk at investigation time). xce_worker_t carries two separate copies of a host-attrs-style resource: attrs->maxswap (hostattrs_t, written ONLY by xcecd.c's init_host_attrs() at join time -- rejoin_worker(), worker.c:440, just re-copies the same join-time value) and resvals[idx(\"maxswap\")] (resmgr-indexed zarray, seeded from attrs->maxswap by add_hostattrs_resvals() at apply_join_state() start, worker.c, then OVERWRITTEN by mark_host_res()/update_val_res() if a hosts.yaml resource override exists for that host, worker.c:79 `array = &(w->host.resvals)`). cmd_workerlist() (test.c:1478) reads attrs->maxswap directly -- structurally incapable of ever reflecting a config override, since nothing writes an override back into attrs. The scheduler-facing path (collect_resvals_workers() -> resvals_to_sched -> xcesched_push_workers(), dataex.c/sched.c) reads resvals, which DOES pick up the override. So whenever a maxswap override is configured, the scheduler correctly uses it (and a replay/snapshot reflects it) while xce_test workerlist permanently shows the stale join-time value regardless of timing/staleness -- not a caching/race artifact, a structural gap. The Go apiserver/xcecli path (convert.go) also just casts attrs.maxswap with no override-awareness, and xcecli's hosts.yaml model doesn't even expose a MaxSwap column today.",
|
||||
"fix": "NOT FIXED -- user explicitly deferred (2026-09-06) to prioritize bug-1554 (real swap/tmp hardware detection, now fixed) and the assigned-jobs slot-count display improvement first. Two candidate fix directions discussed: (a) make cmd_workerlist()/the Go apiserver path read the effective resmgr/resvals value instead of attrs directly, or (b) make mark_host_res() write the override back into attrs too, so every attrs consumer (maxcpus/maxmem/maxswap/maxtmp alike) stays in sync at the source -- (b) was the recommended direction since it fixes the whole class rather than patching each display path individually, but neither is implemented yet.",
|
||||
"tags": [
|
||||
"controller",
|
||||
"worker",
|
||||
"resource",
|
||||
"maxswap",
|
||||
"workerlist",
|
||||
"config-override",
|
||||
"not-fixed",
|
||||
"deferred"
|
||||
],
|
||||
"related_bugs": [
|
||||
"bug-1554"
|
||||
],
|
||||
"occurrences": 1,
|
||||
"last_seen": "2026-09-06T21:10:00Z"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue