myxce/.kiro/specs/resource-value-verification/tasks.md

Implementation Plan

• 1. Foundation: test scaffolding and skill directory setup

• 1.1 Extract resupdate's type-resolution logic and write failing unit tests for its fallback behavior

  • Move set_resval()'s resource-name → Value_u_which table lookup into a new, small, dependency-light resval_typeresolve.c/.h module with no networking/REPL/global-state dependency — test.c (4298 lines, its own main(), heavy libedit/network dependencies) cannot itself be white-box included into a unit test, so this extraction is a hard prerequisite for testing this logic in isolation at all
  • Create test_resval_typeresolve.c (Unity) and wire both new files into src/controller/meson.build: add testsuite/resval_typeresolve.c to xce_test_src (line 57) so xce_test still links, and add a new test() target for test_resval_typeresolve.c following the existing controller_unit_test/sched_push_unit_test pattern (lines 86, 215, 222) — these are hand-maintained explicit lists, not globs, so the module and test will not build without this step
  • Write tests covering: a currently-known name (e.g. mem) still resolves numeric exactly as today; a currently-known string name (type) still resolves string exactly as today; an unrecognized name (e.g. r15m) defaults to numeric; an unrecognized name with an explicit type-override flag resolves to the requested type
  • Observable completion: the new module and test target both compile and link via meson, and the new test target fails (RED) because the fallback-to-numeric and type-override behavior don't exist yet
  • Requirements: 1.1, 1.4

• 1.2 (P) Scaffold the xce-test-resources skill directory

  • Create the skill directory with a SKILL.md stub describing the intended usage (/xce-test-resources [resource=value ...]) and the VERDICT line format, following the xce-test-ha/xce-deploy-ha SKILL.md convention
  • Observable completion: the skill is discoverable/invocable (even though its underlying script is not yet functional) and its documented contract matches design.md's Batch/Job Contract for the skill
  • Requirements: 6.1
  • Boundary: xce-test-resources skill

• 2. Core: resupdate resource-name coverage fix

• 2.1 Wire the extracted resolver into resupdate and add the type-override flag

  • Extend resval_typeresolve.c's fallback path so any resource name not in the existing table defaults to numeric (Value_u_fvalue), while type/model and all currently-supported names keep their exact existing behavior
  • Make test.c's set_resval() a thin wrapper delegating to resval_typeresolve.c (for type resolution) plus its own existing .host tagging (unchanged, still needs REPL state)
  • Add an optional -t <numeric|string> flag to cmd_resupdate(), applied per -n/-v pair
  • Observable completion: the unit tests from 1.1 pass (GREEN); running resupdate -n r15m -v 1.5 in the worker context (against a live controller) now succeeds where it previously silently failed
  • Requirements: 1.1, 1.4
  • Depends: 1.1

• 2.2 Implement per-name accept/reject verification in resupdate

  • After cmd_resupdate() sends the update RPC, issue one follow-up xcecli_resval() query per submitted (resource, worker) pair (the same client-library call resval/cmd_resval already uses) and compare the returned value against what was just sent
  • Report "accepted" for a matching value, "rejected: unknown resource" for a not-found result, with a short bounded retry before concluding rejection (guards against a genuinely-registered resource whose value hasn't propagated to the query path yet)
  • Report every submitted name's individual accept/reject outcome when a single invocation carries multiple -n/-v pairs
  • Observable completion: against a live controller, resupdate -n mem -v 5 -n not-a-real-resource -v 1 reports mem as accepted and not-a-real-resource as rejected, in the same invocation
  • Requirements: 1.2, 1.3
  • Depends: 2.1

• 3. Core: apiserver local deployment capability

• 3.1 (P) Extend xce-deploy-ha's deploy-ha.sh with a conditional apiserver step and a combined start-all.sh

  • In deploy-ha.sh Step 3, add a new conditional branch in the same opt-in-by-presence spirit as the existing worker/scheduler branch: check for a pre-built apiserver binary at its own Go-build output path (e.g. /tmp/apiserver, make build-server's default) — NOT alongside $BINARY, since apiserver is a separate build system, not the same meson build dir worker/scheduler share; deploy-ha.sh does not invoke make build-server itself, building it remains a caller prerequisite (document in SKILL.md's Prerequisites section, same pattern as the existing /xce-build controller note). If present, generate $TOP/scripts/start-apiserver.sh plus a minimal logger.yaml (modeled on make test-integration's XCE_BIN/SRV_URL convention) — functional auth/RBAC is NOT required (verification is trace-log-based, task 6.4; no REST query is ever issued against this instance), but logger.yaml MUST set logging.level: debug or task 4.2's trace line is silently never written; absent the binary, generate nothing, so existing callers are unaffected
  • Also in Step 3, always generate $TOP/scripts/start-all.sh: runs start.sh for the controller node(s), waits on Step 5's existing health-check, then runs whichever of start-worker.sh/start-scheduler.sh/start-apiserver.sh were actually generated for this deploy (skipping any that weren't) — start the scheduler with -vvv specifically (verbosity 3, matching tlog_debug3/TRACE), overriding SKILL.md's existing -v-only example for this feature's needs, or task 4.1's trace line is silently never written
  • Update .claude/skills/xce-deploy-ha/SKILL.md with a new "Step 4d" documenting the apiserver step (following the existing Step 4b pattern) and documenting start-all.sh as a combined shortcut for Steps 4/4b/4c
  • Write apiserver/scheduler logs under the caller's scratch directory, matching the existing logs/ convention — these are the logs tasks 6.3/6.4 grep
  • Observable completion: running deploy-ha.sh against a build that includes an apiserver binary, then start-all.sh, leaves a running controller, scheduler (at -vvv), and apiserver (with debug-level logging) with no further manual steps; running deploy-ha.sh against a build without an apiserver binary (e.g. xce-test-ha's existing usage) is unaffected — no start-apiserver.sh is generated, and start-all.sh skips that step
  • Requirements: 5.3, 6.1
  • Boundary: xce-deploy-ha (deploy-ha.sh, SKILL.md)

• 4. Core: scheduler and apiserver trace-log instrumentation

• 4.1 (P) Add a trace log line to the scheduler's live resource-update handling

  • In xce_core_push_events()'s XCE_CORE_EVENT_RESVALS_DELTA case (src/scheduler/src/core_events.c), add one TRACE-level log line (this component's existing tinylog convention) immediately after the resolved worker's value is written, logging the resolved host, resource name, and applied value in a fixed, greppable format
  • Confirm the existing scheduler testsuite (test_core_events.c and friends) still passes unchanged — this is a pure log addition with no control-flow change, so no new unit test is required (consistent with this codebase's existing precedent: other TRACE/WARN calls in this area, e.g. apiserver_worker_change, have no dedicated unit test coverage either); the log line's actual content/format is confirmed live in task 8.1
  • Observable completion: a live scheduler daemon's log file contains one line per applied resource update, in the format task 6.3 will grep for; existing scheduler tests are unaffected
  • Requirements: 4.1, 4.2, 4.3

• 4.2 (P) Add a trace log line to the apiserver's resource-update storage handling

  • In OnResourceLocationUpdated (src/frontend/apiserver/backend/handler.go:92-95), add one logger.Debug(...) call (zap, per this component's existing convention) immediately after store.ResourceLocations.Put(key, *res), logging the resource name, location/host, and stored value
  • Confirm existing apiserver backend tests still pass unchanged — same rationale as 4.1, no new unit test required for a pure log addition; content/format confirmed live in task 8.1
  • Observable completion: a live apiserver's log file contains one line per stored resource-location update, in the format task 6.4 will grep for; existing backend tests are unaffected
  • Requirements: 5.1, 5.2, 5.4

• 5. Integration: full-stack deployment orchestration

• 5.1 Wire controller, scheduler, and apiserver deployment into the skill via start-all.sh

  • Invoke xce-deploy-ha's deploy-ha.sh (task 3.1) into a fresh scratch directory, then its generated start-all.sh — no manual per-component sequencing lives in this skill; start-all.sh already gates scheduler/apiserver startup on the controller health-check and handles the -vvv/debug-level logging needed for tasks 4.1/4.2's trace lines
  • Register a trap-based cleanup (or equivalent) before invoking start-all.sh, not just at the end of the script — this must cover a mid-deployment abort (e.g. apiserver fails to start after controller+scheduler are already up), not only a completed PASS/FAIL run; deliberately stronger than xce-deploy-ha's own convention (a separate, manually-invoked cleanup.sh) because Requirement 6.5 requires cleanup-by-default on every path unless the operator passed the "keep running" flag
  • Observable completion: a single invocation of the orchestration script leaves a running controller, live scheduler daemon (built with task 4.1's trace log, started at -vvv), and apiserver (built with task 4.2's trace log, debug-level logging enabled), all logging under the same scratch directory, with no manual steps; killing the script mid-deployment (e.g. before the apiserver step completes) still removes every process it had already started, unless "keep running" was passed
  • Requirements: 5.3, 6.1, 6.5
  • Depends: 1.2, 3.1, 4.1, 4.2

• 6. Core: resource injection and per-layer verification checks

• 6.1 Implement worker join and resource injection

  • Join a synthetic worker through the xce_test worker context (no real worker daemon process), then run resupdate for one or more resource names/values
  • Observable completion: after this step runs, resupdate has been accepted for every requested resource name with zero real worker daemon processes running
  • Requirements: 2.1, 2.2, 6.1, 6.4
  • Depends: 2.1, 2.2, 5.1

• 6.2 Implement the controller-side verification check

  • Query the controller via resval for each injected resource/worker pair and compare against the injected value
  • Observable completion: the check reports, per resource, whether the controller's observed value matches the injected value, including both values when it does not
  • Requirements: 3.1, 3.2
  • Depends: 6.1
  • Boundary: xce-test-resources skill (controller check) — same file as 6.3/6.4 (run-resource-verify.sh), sequential, not parallel-safe

• 6.3 Implement the scheduler-side verification check via trace-log tail

  • Tail/grep the live scheduler daemon's log file (task 4.1's trace line) for each injected resource/worker pair, polling with a short interval up to a generous, explicit bounded wait — no record, no replay, no REPL
  • Compare the scheduler's observed value against the injected value, reported distinctly from the controller-side result; a timeout with no matching line is reported as "unreachable," distinct from a value mismatch
  • Observable completion: the check reports, per resource, whether the scheduler's observed value matches the injected value within the bounded wait, attributable specifically to the scheduler layer
  • Requirements: 4.1, 4.2, 4.3
  • Depends: 6.1, 6.2
  • Boundary: xce-test-resources skill (scheduler check) — same file as 6.2/6.4, sequential

• 6.4 Implement the apiserver-side verification check via trace-log tail

  • Tail/grep the live apiserver's log file (task 4.2's trace line) for each injected resource/worker pair, polling with a short interval up to a generous, explicit bounded wait — no xcecli query, no REST call
  • Compare the apiserver's observed value against the injected value, reported distinctly from the controller-side and scheduler-side results; a timeout with no matching line is reported as "unreachable," distinct from a value mismatch
  • Observable completion: the check reports, per resource, whether the apiserver's observed value matches the injected value within the bounded wait, attributable specifically to the apiserver layer
  • Requirements: 5.1, 5.2, 5.4
  • Depends: 6.1, 6.3
  • Boundary: xce-test-resources skill (apiserver check) — same file as 6.2/6.3, sequential

• 7. Integration: verdict aggregation, reporting, and cleanup

• 7.1 Aggregate layer results, emit the verdict, and clean up

  • Combine the controller/scheduler/apiserver check results into a single per-resource, per-layer comparison, implemented as a self-contained comparison step that accepts a set of (resource, injected value, per-layer observed value) records — so it can be invoked directly with synthetic records for testing (task 8.2), not only reachable from a live run
  • Emit a machine-parseable VERDICT line (following xce-test-ha's convention) plus the standard terminal PASS/FAIL oracle line, naming every layer that mismatched or was unreachable (timed out)
  • Support checking multiple resource names in one run, defaulting to the resource names added in task 2.1 when none are supplied
  • Clean up every process and runtime/data-directory artifact this run created, unless the operator passed a flag requesting the deployment be left running — this is the normal-completion firing of the same trap-based cleanup task 5.1 registered for the abort path, not a separate mechanism. Log files under the scratch directory are always preserved, regardless of the keep-running flag, so a FAIL verdict's underlying scheduler/apiserver trace-log evidence remains inspectable after the run
  • Observable completion: a full run against the default resource set produces exactly one VERDICT line and a matching process exit code, no controller/scheduler/apiserver processes remain afterward (unless the keep-running flag was used), and the run's log files are still present on disk regardless
  • Requirements: 6.2, 6.3, 6.4, 6.5
  • Depends: 6.2, 6.3, 6.4

• 8. Validation: end-to-end live verification runs

• 8.1 Run the full flow live, including one deliberately-unregistered resource name

  • Execute the skill end-to-end against a freshly deployed controller/scheduler/apiserver, using the default resource set (task 2.1's previously-missing names) plus one resource name that does not exist in the cluster's registered set
  • Observable completion: every genuinely-registered resource individually reports matching values at all three layers (confirming tasks 4.1/4.2's trace-log lines are correctly formatted and greppable, not just present), AND the unregistered name is reported as a clear, specifically-identified rejection (not a silent omission) with correct per-name accept/reject accounting across the whole batch; per design.md's Error Handling (an unrecognized name is a run-level failure), the run's single overall VERDICT is FAIL, attributable specifically to the unregistered name — not a false overall PASS
  • Requirements: 1.1, 1.2, 1.3, 1.4, 2.1, 2.2, 3.1, 3.2, 4.1, 4.2, 4.3, 5.1, 5.2, 5.3, 5.4, 6.1, 6.2, 6.4
  • Depends: 7.1

• 8.2 Verify the verdict logic correctly reports FAIL on a genuine mismatch

  • Invoke task 7.1's comparison step directly with a synthetic set of per-layer observed-value records where one layer's value deliberately does not match the corresponding injected value (no live deployment required for this check, since the comparison step is self-contained per task 7.1)
  • Observable completion: the comparison step reports an overall FAIL verdict that correctly names the specific layer whose synthetic record was mismatched, and reports PASS for every other, matching layer/resource in the same input set — proving the verdict logic distinguishes pass from fail rather than always reporting PASS
  • Requirements: 6.3
  • Depends: 7.1

Implementation Notes

• Task 1.1: resval_typeresolve.c's new function is int resval_resolve_type(const char* name, int type_override), returning a Value_u_which tag or RESVAL_TYPE_UNKNOWN (-1, doesn't collide with the real 0-6 enum range). Task 2.1 should build on this exact signature rather than inventing a new one. test.c's set_resval()/cmd_resupdate() were deliberately left untouched by task 1.1 — task 2.1 is the one that makes set_resval() a thin wrapper.
• controller_unit_test (the big legacy Unity binary) has a pre-existing, unrelated flaky failure in test_config_repl.c (bug-1561, overlayfs fdatasync errno=5 sandbox artifact) that shows up on every /xce-build controller run in this environment. It is NOT caused by this feature's changes — do not chase it in later tasks; cross-check .wolf/buglog.json's bug-1561 entry before treating a controller_unit_test failure as real, and confirm via git status that test_config_repl.c/config_repl.c are untouched by the current task's diff.
• Task 2.1 (round 1 REJECTED, round 2 fixed): refactoring a getopt-loop's flush/finalization timing (needed to let a trailing flag like -t apply to its preceding -n/-v pair) can silently drop an unrelated existing guard (here: duplicate--v rejection) that depended on the old immediate-finalization timing. When touching cmd_resupdate()-shaped CLI parsers in this codebase again, explicitly re-trace EVERY branch of the old getopt loop against the new one, not just the branches directly related to the new flag — a "traced every path" claim is only as good as the list of paths actually enumerated. xce_test also needs ASAN_OPTIONS=detect_leaks=0 for reliable non-interactive stdin-piped smoke testing (its default leak-detection exit path can swallow buffered stdout).