From e61fa2b3a83bc5e387d852f4d086fcb952a6feb0 Mon Sep 17 00:00:00 2001 From: Rongsong Shen Date: Thu, 20 Aug 2026 03:03:12 +0000 Subject: [PATCH] docs(roadmap): add resource-distribution-mapping (existed since 2026-07-23, never tracked here) Surfaced while investigating a user request to show total/current values for GLOBAL/SHARED resources in the scheduler REPL's "info resource" command -- researched and confirmed that concept doesn't exist anywhere in this codebase (resourcemap config field dropped at build time, capnp fields commented out, worker.c actively rejects XCE_RESF_SHARED resources today). This spec already owns closing that gap; it just never made it into the roadmap's backlog list. --- .kiro/steering/roadmap.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.kiro/steering/roadmap.md b/.kiro/steering/roadmap.md index 26a608b9..0a66a72c 100644 --- a/.kiro/steering/roadmap.md +++ b/.kiro/steering/roadmap.md @@ -93,6 +93,7 @@ Smaller items that likely don't warrant a full spec; pick up opportunistically o - [x] scheduler framework enhancement — **PROMOTED to spec 2026-07-06, then DECOMPOSED 2026-07-27** into `scheduler-core` (**IMPLEMENTATION-COMPLETE**), `scheduler-allocation`, and `scheduler-preemption` — see **In Flight** above for current status. Scope notes from the original brief still apply: commit barrier wiring deferred (separate hardening item); GPU/affinity allocation deferred (pluggable via new allocator interface); new job selection policies deferred (framework enables them, ships separately). No longer tracked here. - [→] **perf-tuning-extraction** — **PROMOTED to spec 2026-07-20** (`clusterconfig-schema-split`, `.kiro/specs/clusterconfig-schema-split/`). Originally recorded 2026-07-06 from admin-config-tool discovery as a documentation/audit item; the audit (2026-07-20) found the fields are load-bearing enough to need real code changes, not just a doc pass — `candidates`/`vcpus`/hash-tuning fields (`jobmgr`/`workermgr`/`qmgr`/`resmgr`/`xstrings`) are already dead weight in the `ClusterConfig` capnp blob (superseded by `--peers`/`--vcpus` CLI flags or never read into runtime state — see `src/docs/cluster-config-split-analysis.md`). **`port` is NOT dead** — a second review pass (2026-07-20) found `cluster_open_listener` (`cluster.c:1216-1258`) falls back to the blob's `port` field, and single-node mode (no `--peers` flag at all, `main.c:293-294`) depends on that fallback entirely; the spec now requires a replacement port-sourcing mechanism before the field can be removed (`clusterconfig-schema-split` Requirement 1.7), not a bare deletion. `heartbeatInterval`/`inactiveHeartbeatCount` (controller→worker liveness, NOT the HA/Raft election heartbeat) were investigated and deliberately kept IN `ClusterConfig` for operational uniformity despite passing the independence rule — see `.kiro/steering/config-design.md`. `fetchJobInterval`/`netioTimeout` left unresolved, flagged out-of-scope in the new spec for a future pass. - [ ] **Raft/HA heartbeat interval+timeout as CLI parameters** — **NEW 2026-07-20 from clusterconfig-schema-split discovery.** The controller-to-controller HA/Raft election heartbeat (`ha->config.heartbeat_interval_ms`/`heartbeat_timeout_ms` — the one where a cross-node mismatch risks spurious failover) is 100% hardcoded today (`HA_DEFAULT_HEARTBEAT_INTERVAL_MS`/`HA_DEFAULT_HEARTBEAT_TIMEOUT_MS`, `cluster.c:1131`) — there is no `ClusterConfig` field for it, and per user decision (2026-07-20) there should never be one: these values must be available to a controller BEFORE it has validated/received cluster configuration (a controller needs its own election-timing parameters to even participate in the election that delivers config in the first place — a bootstrap/chicken-egg constraint, not an independence/frequency judgment call). Target: `--heartbeat-interval-ms`/`--heartbeat-timeout-ms` CLI flags following the exact `--vcpus` pattern (`OPT_*` + `struct arguments` + `myconf_t` field, `0` sentinel = use hardcoded default). Kept hardcoded for now; this is a future item, not blocking `clusterconfig-schema-split`. +- [ ] **resource-distribution-mapping** — **created 2026-07-23, never added here (found 2026-08-19, itself a small doc gap).** Physical resources have no "configured total vs. currently-consumed" concept anywhere in this codebase, live or dead — `resources.yaml`'s `resourcemap` field (which expresses exactly this: a value shared across N hosts) is deliberately dropped by `xceadmin config build` (`build.py:41-43`), and the matching `controller.capnp` fields are commented out (`controller.capnp:208-209`). `mark_host_res()` (`worker.c`) actively rejects any resource flagged `XCE_RESF_SHARED` today, so "shared pool" and "per-host value" are mutually exclusive by construction in the live system right now. A working total/current pooling mechanism DOES exist in the scheduler (`resource.c`'s `sched_check_shared_satisfy_()`), but it exclusively serves job-limit virtual resources (queue `MAX_RUNNING`-style), never physical shared resources — architecturally reusable, never connected. This spec (research.md done, 3 scoped options, requirements.md never generated — sitting at the decision point) is the existing owner of closing this gap; surfaced again 2026-08-19 when scheduler-core's REPL `info resource` command couldn't show a total for a GLOBAL/SHARED resource because there genuinely isn't one to show. Next: pick a scope option in research.md, then `/kiro-spec-requirements resource-distribution-mapping`. - [ ] **scheduler-sim-diagnostics** — **NEW 2026-08-10 (`/kiro-discovery`).** Open-ended, continuously-extended spec for scheduler simulation-mode debug/troubleshoot capability — added to over time as new investigation needs surface from discussion or from other in-flight scheduler specs (`scheduler-allocation`, `scheduler-preemption`). Extends `scheduler-core`'s REPL (`info job/queue/worker/resource/summary`, breakpoints, snapshots). Two seed capabilities identified so far: (1) **limit inspection** — `xce_limit_t`/`xce_vres_t` config + live usage currently live only in the `limit` plugin's private kvmgr tables, invisible to the REPL; needs a new `info limit ` command + accessor plumbing. (2) **decision-reasoning trace** — `xce_pendreason_t` is captured on pend decisions but never printed by the REPL, and dispatch decisions have no reasoning trail at all (no record of which limit/resource checks passed) — needs new instrumentation in the allocation path (`resource.c`'s `sched_try_alloc_on_host_`, `plugin.c`'s `limit_prepare_allocation`) plus a REPL command to surface it. Explicitly simulation-mode only (no live/production controller-connected diagnostics) and read-only (no state mutation) per user decision 2026-08-10. Brief: `.kiro/specs/scheduler-sim-diagnostics/brief.md`. Not started — requirements to be added per-capability, not fully speced upfront. Next: `/kiro-spec-init scheduler-sim-diagnostics` when ready to begin. - [ ] **xcecd_virtual concurrent-job support (MXJ > 1)** — **NEW 2026-08-19, recorded during bug-1589 investigation** (`.wolf/buglog.json`). Live testing found that when a job array (multiple elements) is dispatched to a single `xcecd_virtual` worker in quick succession, only the first element's start ever gets confirmed back to the controller (`on_job_start`) — the rest are dispatched but never confirmed. Root-caused (or being root-caused — see bug-1589 for the confirmed mechanism once the in-flight investigation lands) to `xcecd_virtual`'s job-simulation path: unlike the real `xcecd` binary, which launches an actual dbus-managed worker process per job (tracked by real pid, real IPC status reports), `xcecd_virtual` (`src/worker/src/xcecd.c`'s `xcecd_process_command()`, `VIRTUAL_CLUSTER` branch) never launches a real process at all — it assigns a bare incrementing fake pid (`static int virtual_pid`, `xcecd.c:27`) and simulates job execution via a sleep-timer mechanism (`xcecd_worker_utils.h`'s `VIRTUAL_CLUSTER`-only `xcecd_worker_t` fields `start_time`/`sleep_time`/`killed_signal`, `check_worker_sleep_time_elapsed()`). The immediate bug-1589 fix (once landed) addresses the specific reported symptom; this item is the broader follow-up: **properly support N concurrent simulated jobs per `xcecd_virtual` worker**, since MXJ (max jobs per host) > 1 is a legitimate, common configuration to want to exercise in virtual-cluster-mode testing/CI (this project's own dev workflow relies on `xcecd_virtual` for HA soak tests, resource-value verification, and now scheduler-core's live validation — several `.wolf/buglog.json` entries this session alone used it), not just a single-job-at-a-time edge case. Scope sketch (to refine once bug-1589's root cause is fully confirmed): audit every VIRTUAL_CLUSTER-only code path in `src/worker/` (`xcecd.c`, `main.c`, `xcecd_worker_utils.h`/`.c`, `process_worker_manager.c`) for single-job assumptions (a lone poll/early-return that only processes one pending simulated job per tick, a shared mutable scratch struct reused across concurrent jobs, etc.) and generalize each to iterate the full set of in-flight simulated jobs, matching the pattern the real-process path (`process_manager.c`'s `process_info_t manager->processes[]`, a proper per-pid array) already establishes correctly. Dependencies: bug-1589 (must land first — its root-cause trace is the concrete map of what needs generalizing). Promote via `/kiro-discovery` or `/kiro-spec-init` once bug-1589 is closed and the exact mechanism is confirmed.