# Implementation Plan - [x] 1. Foundation: test scaffolding and skill directory setup - [x] 1.1 Extract resupdate's type-resolution logic and write failing unit tests for its fallback behavior - Move `set_resval()`'s resource-name → `Value_u_which` table lookup into a new, small, dependency-light `resval_typeresolve.c`/`.h` module with no networking/REPL/global-state dependency — `test.c` (4298 lines, its own `main()`, heavy `libedit`/network dependencies) cannot itself be white-box included into a unit test, so this extraction is a hard prerequisite for testing this logic in isolation at all - Create `test_resval_typeresolve.c` (Unity) and wire both new files into `src/controller/meson.build`: add `testsuite/resval_typeresolve.c` to `xce_test_src` (line 57) so `xce_test` still links, and add a new `test()` target for `test_resval_typeresolve.c` following the existing `controller_unit_test`/`sched_push_unit_test` pattern (lines 86, 215, 222) — these are hand-maintained explicit lists, not globs, so the module and test will not build without this step - Write tests covering: a currently-known name (e.g. `mem`) still resolves numeric exactly as today; a currently-known string name (`type`) still resolves string exactly as today; an unrecognized name (e.g. `r15m`) defaults to numeric; an unrecognized name with an explicit type-override flag resolves to the requested type - Observable completion: the new module and test target both compile and link via `meson`, and the new test target fails (RED) because the fallback-to-numeric and type-override behavior don't exist yet - _Requirements: 1.1, 1.4_ - [x] 1.2 (P) Scaffold the xce-test-resources skill directory - Create the skill directory with a `SKILL.md` stub describing the intended usage (`/xce-test-resources [resource=value ...]`) and the VERDICT line format, following the `xce-test-ha`/`xce-deploy-ha` SKILL.md convention - Observable completion: the skill is discoverable/invocable (even though its underlying script is not yet functional) and its documented contract matches design.md's Batch/Job Contract for the skill - _Requirements: 6.1_ - _Boundary: xce-test-resources skill_ - [x] 2. Core: resupdate resource-name coverage fix - [x] 2.1 Wire the extracted resolver into resupdate and add the type-override flag - Extend `resval_typeresolve.c`'s fallback path so any resource name not in the existing table defaults to numeric (`Value_u_fvalue`), while `type`/`model` and all currently-supported names keep their exact existing behavior - Make `test.c`'s `set_resval()` a thin wrapper delegating to `resval_typeresolve.c` (for type resolution) plus its own existing `.host` tagging (unchanged, still needs REPL state) - Add an optional `-t ` flag to `cmd_resupdate()`, applied per `-n`/`-v` pair - Observable completion: the unit tests from 1.1 pass (GREEN); running `resupdate -n r15m -v 1.5` in the worker context (against a live controller) now succeeds where it previously silently failed - _Requirements: 1.1, 1.4_ - _Depends: 1.1_ - [x] 2.2 Implement per-name accept/reject verification in resupdate - After `cmd_resupdate()` sends the update RPC, issue one follow-up `xcecli_resval()` query per submitted `(resource, worker)` pair (the same client-library call `resval`/`cmd_resval` already uses) and compare the returned value against what was just sent - Report "accepted" for a matching value, "rejected: unknown resource" for a not-found result, with a short bounded retry before concluding rejection (guards against a genuinely-registered resource whose value hasn't propagated to the query path yet) - Report every submitted name's individual accept/reject outcome when a single invocation carries multiple `-n`/`-v` pairs - Observable completion: against a live controller, `resupdate -n mem -v 5 -n not-a-real-resource -v 1` reports `mem` as accepted and `not-a-real-resource` as rejected, in the same invocation - _Requirements: 1.2, 1.3_ - _Depends: 2.1_ - [x] 3. Core: apiserver local deployment capability - [x] 3.1 (P) Extend `xce-deploy-ha`'s `deploy-ha.sh` with a conditional apiserver step and a combined `start-all.sh` - In `deploy-ha.sh` Step 3, add a new conditional branch in the same *opt-in-by-presence* spirit as the existing worker/scheduler branch: check for a pre-built apiserver binary at its own Go-build output path (e.g. `/tmp/apiserver`, `make build-server`'s default) — NOT alongside `$BINARY`, since apiserver is a separate build system, not the same meson build dir worker/scheduler share; `deploy-ha.sh` does not invoke `make build-server` itself, building it remains a caller prerequisite (document in SKILL.md's Prerequisites section, same pattern as the existing `/xce-build controller` note). If present, generate `$TOP/scripts/start-apiserver.sh` plus a minimal `logger.yaml` (modeled on `make test-integration`'s `XCE_BIN`/`SRV_URL` convention) — functional auth/RBAC is NOT required (verification is trace-log-based, task 6.4; no REST query is ever issued against this instance), but `logger.yaml` MUST set `logging.level: debug` or task 4.2's trace line is silently never written; absent the binary, generate nothing, so existing callers are unaffected - Also in Step 3, always generate `$TOP/scripts/start-all.sh`: runs `start.sh` for the controller node(s), waits on Step 5's existing health-check, then runs whichever of `start-worker.sh`/`start-scheduler.sh`/`start-apiserver.sh` were actually generated for this deploy (skipping any that weren't) — start the scheduler with `-vvv` specifically (verbosity 3, matching `tlog_debug3`/`TRACE`), overriding SKILL.md's existing `-v`-only example for this feature's needs, or task 4.1's trace line is silently never written - Update `.claude/skills/xce-deploy-ha/SKILL.md` with a new "Step 4d" documenting the apiserver step (following the existing Step 4b pattern) and documenting `start-all.sh` as a combined shortcut for Steps 4/4b/4c - Write apiserver/scheduler logs under the caller's scratch directory, matching the existing `logs/` convention — these are the logs tasks 6.3/6.4 grep - Observable completion: running `deploy-ha.sh` against a build that includes an apiserver binary, then `start-all.sh`, leaves a running controller, scheduler (at `-vvv`), and apiserver (with debug-level logging) with no further manual steps; running `deploy-ha.sh` against a build without an apiserver binary (e.g. `xce-test-ha`'s existing usage) is unaffected — no `start-apiserver.sh` is generated, and `start-all.sh` skips that step - _Requirements: 5.3, 6.1_ - _Boundary: xce-deploy-ha (deploy-ha.sh, SKILL.md)_ - [x] 4. Core: scheduler and apiserver trace-log instrumentation - [x] 4.1 (P) Add a trace log line to the scheduler's live resource-update handling - In `xce_core_push_events()`'s `XCE_CORE_EVENT_RESVALS_DELTA` case (`src/scheduler/src/core_events.c`), add one `TRACE`-level log line (this component's existing `tinylog` convention) immediately after the resolved worker's value is written, logging the resolved host, resource name, and applied value in a fixed, greppable format - Confirm the existing scheduler testsuite (`test_core_events.c` and friends) still passes unchanged — this is a pure log addition with no control-flow change, so no new unit test is required (consistent with this codebase's existing precedent: other `TRACE`/`WARN` calls in this area, e.g. `apiserver_worker_change`, have no dedicated unit test coverage either); the log line's actual content/format is confirmed live in task 8.1 - Observable completion: a live scheduler daemon's log file contains one line per applied resource update, in the format task 6.3 will grep for; existing scheduler tests are unaffected - _Requirements: 4.1, 4.2, 4.3_ - [x] 4.2 (P) Add a trace log line to the apiserver's resource-update storage handling - In `OnResourceLocationUpdated` (`src/frontend/apiserver/backend/handler.go:92-95`), add one `logger.Debug(...)` call (zap, per this component's existing convention) immediately after `store.ResourceLocations.Put(key, *res)`, logging the resource name, location/host, and stored value - Confirm existing apiserver backend tests still pass unchanged — same rationale as 4.1, no new unit test required for a pure log addition; content/format confirmed live in task 8.1 - Observable completion: a live apiserver's log file contains one line per stored resource-location update, in the format task 6.4 will grep for; existing backend tests are unaffected - _Requirements: 5.1, 5.2, 5.4_ - [x] 5. Integration: full-stack deployment orchestration - [x] 5.1 Wire controller, scheduler, and apiserver deployment into the skill via `start-all.sh` - Invoke `xce-deploy-ha`'s `deploy-ha.sh` (task 3.1) into a fresh scratch directory, then its generated `start-all.sh` — no manual per-component sequencing lives in this skill; `start-all.sh` already gates scheduler/apiserver startup on the controller health-check and handles the `-vvv`/debug-level logging needed for tasks 4.1/4.2's trace lines - Register a `trap`-based cleanup (or equivalent) before invoking `start-all.sh`, not just at the end of the script — this must cover a mid-deployment abort (e.g. apiserver fails to start after controller+scheduler are already up), not only a completed PASS/FAIL run; deliberately stronger than `xce-deploy-ha`'s own convention (a separate, manually-invoked `cleanup.sh`) because Requirement 6.5 requires cleanup-by-default on every path unless the operator passed the "keep running" flag - Observable completion: a single invocation of the orchestration script leaves a running controller, live scheduler daemon (built with task 4.1's trace log, started at `-vvv`), and apiserver (built with task 4.2's trace log, debug-level logging enabled), all logging under the same scratch directory, with no manual steps; killing the script mid-deployment (e.g. before the apiserver step completes) still removes every process it had already started, unless "keep running" was passed - _Requirements: 5.3, 6.1, 6.5_ - _Depends: 1.2, 3.1, 4.1, 4.2_ - [x] 6. Core: resource injection and per-layer verification checks - [x] 6.1 Implement worker join and resource injection - Join a synthetic worker through the `xce_test` worker context (no real worker daemon process), then run `resupdate` for one or more resource names/values - Observable completion: after this step runs, `resupdate` has been accepted for every requested resource name with zero real worker daemon processes running - _Requirements: 2.1, 2.2, 6.1, 6.4_ - _Depends: 2.1, 2.2, 5.1_ - [x] 6.2 Implement the controller-side verification check - Query the controller via `resval` for each injected resource/worker pair and compare against the injected value - Observable completion: the check reports, per resource, whether the controller's observed value matches the injected value, including both values when it does not - _Requirements: 3.1, 3.2_ - _Depends: 6.1_ - _Boundary: xce-test-resources skill (controller check) — same file as 6.3/6.4 (`run-resource-verify.sh`), sequential, not parallel-safe_ - [x] 6.3 Implement the scheduler-side verification check via trace-log tail - Tail/grep the live scheduler daemon's log file (task 4.1's trace line) for each injected resource/worker pair, polling with a short interval up to a generous, explicit bounded wait — no record, no replay, no REPL - Compare the scheduler's observed value against the injected value, reported distinctly from the controller-side result; a timeout with no matching line is reported as "unreachable," distinct from a value mismatch - Observable completion: the check reports, per resource, whether the scheduler's observed value matches the injected value within the bounded wait, attributable specifically to the scheduler layer - _Requirements: 4.1, 4.2, 4.3_ - _Depends: 6.1, 6.2_ - _Boundary: xce-test-resources skill (scheduler check) — same file as 6.2/6.4, sequential_ - [x] 6.4 Implement the apiserver-side verification check via trace-log tail - Tail/grep the live apiserver's log file (task 4.2's trace line) for each injected resource/worker pair, polling with a short interval up to a generous, explicit bounded wait — no `xcecli` query, no REST call - Compare the apiserver's observed value against the injected value, reported distinctly from the controller-side and scheduler-side results; a timeout with no matching line is reported as "unreachable," distinct from a value mismatch - Observable completion: the check reports, per resource, whether the apiserver's observed value matches the injected value within the bounded wait, attributable specifically to the apiserver layer - _Requirements: 5.1, 5.2, 5.4_ - _Depends: 6.1, 6.3_ - _Boundary: xce-test-resources skill (apiserver check) — same file as 6.2/6.3, sequential_ - [x] 7. Integration: verdict aggregation, reporting, and cleanup - [x] 7.1 Aggregate layer results, emit the verdict, and clean up - Combine the controller/scheduler/apiserver check results into a single per-resource, per-layer comparison, implemented as a self-contained comparison step that accepts a set of (resource, injected value, per-layer observed value) records — so it can be invoked directly with synthetic records for testing (task 8.2), not only reachable from a live run - Emit a machine-parseable VERDICT line (following `xce-test-ha`'s convention) plus the standard terminal PASS/FAIL oracle line, naming every layer that mismatched or was unreachable (timed out) - Support checking multiple resource names in one run, defaulting to the resource names added in task 2.1 when none are supplied - Clean up every process and runtime/data-directory artifact this run created, unless the operator passed a flag requesting the deployment be left running — this is the normal-completion firing of the same trap-based cleanup task 5.1 registered for the abort path, not a separate mechanism. **Log files under the scratch directory are always preserved**, regardless of the keep-running flag, so a FAIL verdict's underlying scheduler/apiserver trace-log evidence remains inspectable after the run - Observable completion: a full run against the default resource set produces exactly one VERDICT line and a matching process exit code, no controller/scheduler/apiserver processes remain afterward (unless the keep-running flag was used), and the run's log files are still present on disk regardless - _Requirements: 6.2, 6.3, 6.4, 6.5_ - _Depends: 6.2, 6.3, 6.4_ - [x] 8. Validation: end-to-end live verification runs - [x] 8.1 Run the full flow live, including one deliberately-unregistered resource name - Execute the skill end-to-end against a freshly deployed controller/scheduler/apiserver, using the default resource set (task 2.1's previously-missing names) plus one resource name that does not exist in the cluster's registered set - Observable completion: every genuinely-registered resource individually reports matching values at all three layers (confirming tasks 4.1/4.2's trace-log lines are correctly formatted and greppable, not just present), AND the unregistered name is reported as a clear, specifically-identified rejection (not a silent omission) with correct per-name accept/reject accounting across the whole batch; per design.md's Error Handling (an unrecognized name is a run-level failure), the run's single overall VERDICT is FAIL, attributable specifically to the unregistered name — not a false overall PASS - _Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2, 4.3, 5.1, 5.2, 5.3, 5.4, 6.1, 6.2, 6.4_ - _Depends: 7.1_ - [x] 8.2 Verify the verdict logic correctly reports FAIL on a genuine mismatch - Invoke task 7.1's comparison step directly with a synthetic set of per-layer observed-value records where one layer's value deliberately does not match the corresponding injected value (no live deployment required for this check, since the comparison step is self-contained per task 7.1) - Observable completion: the comparison step reports an overall FAIL verdict that correctly names the specific layer whose synthetic record was mismatched, and reports PASS for every other, matching layer/resource in the same input set — proving the verdict logic distinguishes pass from fail rather than always reporting PASS - _Requirements: 6.3_ - _Depends: 7.1_ ## Implementation Notes - Task 8.1 (validation-only, no code changes) confirmed the full spec end-to-end live: `VERDICT resources=10 layers_checked=3 mismatches=0 unreachable=0 rejected=1 result=FAIL` (default 9 + 1 fake name) and `VERDICT resources=9 ... rejected=0 result=PASS` (defaults only, positive control). Found (but correctly did not fix, out of boundary) **bug-1583**: the apiserver's `OnResourceLocationUpdated` performs zero name validation and stores ANY resource-location update pushed to it — including a genuinely-unregistered name the controller/scheduler both reject — because `do_update_resvals()` fires the `HOOK_DETAIL_RESOURCE_UPDATE` hook with the raw, unfiltered submitted list; the scheduler happens to independently re-validate and reject, the apiserver does not. Confirmed this does NOT affect this feature's own correctness (accept/reject is determined entirely by `cmd_resupdate()`'s own controller-side follow-up query, before any layer is checked) — but it's a real, previously-undocumented data-integrity gap in production code, worth a dedicated follow-up outside this feature. - Task 7.1 (round 1 REJECTED, round 2 fixed): `aggregate_verdict()`/`build_verdict_record()` (VERDICT_FIELD_SEP=`$'\x1f'`, ASCII Unit Separator — bash's IFS tab-splitting silently collapses adjacent empty fields even with IFS set to only a tab, bug-1581) are the self-contained, synthetic-testable comparison step task 8.2 needs. `main()`'s live-data wiring is `build_verdict_records_from_tsv()` — derive a layer's "ok" status from `SCHEDULER_CHECK_RESULTS[$name]`/`APISERVER_CHECK_RESULTS[$name]`'s STRING CONTENT (`UNREACHABLE:*` prefix check), never from array-key existence — tasks 6.3/6.4 always set the key, even on a genuine timeout (just with an empty value), so key-existence can never distinguish UNREACHABLE from MISMATCH (bug-1582). `CONTROLLER_OBSERVED`'s key-existence check IS correct for the controller layer specifically (task 6.2 has no separate UNREACHABLE concept — a query failure there aborts the whole run). VERDICT line format: `[xce-test-resources] VERDICT resources= layers_checked=3 mismatches= unreachable= rejected= result=PASS|FAIL` — `rejected=` is an intentional addition beyond design.md's illustrative template (Requirement 6.3 needs rejected names attributable in the machine-parseable line itself); `layers_checked=3` is a fixed constant, not a per-run computed count. SKILL.md fully rewritten to describe the now-complete, functional tool. - Task 6.4: `poll_apiserver_log()`/`query_apiserver_resvals()` mirror task 6.3's scheduler-check pattern exactly (same bounded-wait/batched-polling shape). The apiserver's zap `console` encoder renders structured fields as a trailing JSON object (`resource-location stored {"resource": "r15m", "host": "...", "value": 1.5}`), not classic `key=value` text — grep for the message string, then parse the trailing JSON block. All three layers (controller/scheduler/apiserver) confirmed simultaneously MATCH on the full default set including `cpuf`, proving the bug-1579 fix holds end-to-end. - **bug-1579 is now FULLY fixed** (retrofit on top of tasks 6.1/6.2, not a numbered task — user-requested follow-up). `inject_resources()`'s worker-context `xce_test` script now inserts a `!` line between `resupdate` and `quit`: `! bash