myxce/.claude/skills/xce-test-resources/SKILL.md

name	description	model
xce-test-resources	End-to-end resource-value propagation verification: inject a synthetic worker resource value via xce_test's resupdate and confirm it is correctly visible at the controller, live scheduler, and Go apiserver. Usage: /xce-test-resources [resource=value ...]	haiku

XCE Resource-Value Verification

Status: complete and functional (tasks 1.1-7.1 all done)

run-resource-verify.sh implements the full flow described below: deploy controller + scheduler + apiserver, join a synthetic worker, inject one or more resource values, verify all three layers, emit a VERDICT line plus the standard terminal PASS/FAIL oracle line, and clean up. Invoking /xce-test-resources runs the whole thing end-to-end and its exit code reflects the real overall verdict (0 = PASS, non-zero = FAIL) -- not just whether the deployment stood up.

Bug-1569 required deploying real worker/controller/scheduler/apiserver daemons and waiting on periodic reporting cycles just to confirm a resource-value routing fix. This skill turns that multi-minute manual investigation into one command.

What this skill does

1. Deploy controller + scheduler + apiserver into a fresh scratch directory by invoking xce-deploy-ha's deploy-ha.sh (Step 3) and then its generated start-all.sh -- no manual per-component sequencing lives in this skill; start-all.sh already gates scheduler/apiserver startup on xce-deploy-ha's existing controller health-check. A trap-based cleanup is registered before the first process starts, so it fires on every exit path -- a completed PASS/FAIL run, a mid-deployment abort, or an external SIGINT/SIGTERM -- not just a successful completion.
2. Inject -- join a synthetic worker through xce_test's worker context (no real worker daemon process) and run resupdate -n <name> -v <value> for one or more resource names in a single RPC round trip. Every submitted name gets a definitive "accepted" or "rejected: <reason>" outcome.
3. Verify all three layers, for every accepted name:
   • Controller -- query via xce_test's client-context resval and compare against the injected value (run nested inside the still-open injecting worker session, so host-attrs resources like cpuf are never reset by a premature disconnect -- see bug-1579).
   • Scheduler -- tail/grep the live scheduler daemon's own log file (resval applied host=<H> resource=<R> value=<V>, -vvv) with a short poll interval up to a 15s bounded wait (no record/replay/REPL).
   • Apiserver -- tail/grep the apiserver's own log file (zap "resource-location stored" with a trailing JSON field block), same bounded-wait posture. Each layer's observed value is reported distinctly (MATCH / MISMATCH / UNREACHABLE), so a failure is attributable to a specific layer, not just "something failed" -- UNREACHABLE (bounded wait timed out, no trace line ever appeared) is always reported distinctly from MISMATCH (a trace line appeared with the wrong value).
4. Aggregate and report -- combine the injection accept/reject outcome and all three layers' results into one machine-parseable VERDICT line plus the standard terminal PASS/FAIL oracle line (see below). A rejected resource name is a run-level failure in its own right, independent of whatever the (nonexistent, since a rejected name is never actually applied anywhere) per-layer results would say.
5. Clean up every process and runtime/data-directory artifact this run created -- unconditionally, on PASS, FAIL, or a mid-deployment abort alike -- unless the operator explicitly asked for the deployment to be left running (--keep-running). Log files under the scratch directory are always preserved, independent of that flag, so a FAIL verdict's underlying trace-log evidence stays inspectable after the run (a deliberate divergence from xce-deploy-ha's own cleanup.sh, which wipes logs/*.log).

Idempotency & recovery: every invocation is safe to re-run -- a fresh scratch directory per run, following xce-deploy-ha's existing "every invocation is a fresh, self-contained deploy" convention -- with no shared mutable state carried across runs.

Usage

run-resource-verify.sh [--keep-running] [--dir DIR] [resource=value ...]

(the /xce-test-resources [resource=value ...] form above dispatches to this script.)

• No arguments: defaults to DEFAULT_RESOURCE_ARGS -- Requirement 1's gap-resource set (r15m, pg, ls, it, swp, io, tmp) plus two already-supported NUMERIC names for regression coverage (mem, cpuf), so a default run also proves the resupdate fix didn't disturb existing behavior. Deliberately excludes type/model (string-typed) from the default set -- see bug-1574 below.
• One or more resource=value pairs: checks exactly those, e.g. /xce-test-resources r15m=1.5 pg=3. A name the cluster doesn't recognize is reported as a rejection and drives the overall verdict to FAIL, identifying the specific name -- it is never silently dropped or ignored.
• --dir DIR -- scratch/deploy directory. Default: a fresh directory generated via mktemp -d (xce-deploy-ha's "every invocation is a fresh, self-contained deploy" convention), or $XCE_RESVERIFY_TOP when set and --dir is not given (same precedence pattern as xce-test-ha's $XCE_HA_TOP).
• --keep-running -- skip PROCESS teardown in cleanup, leaving the deployment running under DIR for further inspection. Log files under DIR are ALWAYS preserved regardless of this flag; it only controls whether controller/scheduler/apiserver processes are killed. The verdict is still computed and reported normally either way -- --keep-running does not change PASS/FAIL or the exit code, only what happens to the processes afterward. Tear the deployment down manually afterward (its PIDs are visible via ps/pgrep scoped to DIR).
• Other overridable env vars (all optional, default to this repo's own meson build dir / /tmp/apiserver): $XCE_RESVERIFY_BUILD_DIR, $XCE_RESVERIFY_BIN (controller), $XCE_RESVERIFY_SCHED_BIN, $XCE_RESVERIFY_APISERVER_BIN, $XCE_RESVERIFY_XT (the xce_test binary), $XCE_RESVERIFY_PEERS (defaults to a single-node localhost:9991 -- this skill verifies value propagation, not raft/HA failover, so a single node is deliberate and avoids election-timing flakiness; xce-test-ha remains the tool for HA testing).

Must be invoked under an environment with nix on PATH -- the script shells out to nix develop itself for the steps that need it, so it does not need to be pre-wrapped by the caller.

VERDICT line and PASS/FAIL oracle

Following xce-test-ha's machine-parseable VERDICT convention, emitted by aggregate_verdict():

[xce-test-resources] VERDICT resources=9 layers_checked=3 mismatches=0 unreachable=0 rejected=0 result=PASS

• resources -- total count of resource names checked this run (accepted and rejected alike).
• layers_checked -- always 3 (controller, scheduler, apiserver).
• mismatches -- count of (resource, layer) pairs where a trace line/query DID return a value but it did not match the injected value (within numeric tolerance; exact string match for non-numeric values).
• unreachable -- count of (resource, layer) pairs where the bounded wait timed out with NO trace line ever observed -- distinct from mismatches per Requirement 4.3/5.4 (the two point at different bugs: a value-propagation defect vs. a layer that never came up or never logged).
• rejected -- count of resource names the injection step itself rejected (Requirement 1.2) -- never checked at any layer, since nothing was ever applied anywhere for a rejected name.
• result -- PASS iff mismatches == 0 AND unreachable == 0 AND rejected == 0; FAIL otherwise.

Immediately above the VERDICT line, one FAILURE: line is printed per failing (resource, layer) pair or rejected name, naming the specific resource and layer (or layer=injection for a rejection) plus the injected and observed values -- e.g.:

[xce-test-resources] FAILURE: resource=r15m layer=scheduler status=MISMATCH injected=1.5 observed=1.2
[xce-test-resources] FAILURE: resource=not-a-real-resource layer=injection status=REJECTED detail="rejected: unknown resource"

Then exactly one terminal oracle line:

[xce-test-resources] PASS: 9 resource(s), 3 layers checked, zero mismatches, zero unreachable, zero rejected
[xce-test-resources] FAIL: 2 resource(s), 3 layers checked, 0 mismatch(es), 0 unreachable, 1 rejected -- see FAILURE lines above

PASS ⇒ exit 0. Any FAIL condition ⇒ exit 1. A structural failure before the aggregation step is reached (deployment failure, worker-join failure, a malformed --dir/argument, etc.) still exits non-zero via this script's fail()/usage() (1, or 2 for a usage/argument error) -- unrelated to the verdict's own result= field, since aggregation never ran.

A full run against the default resource set produces exactly one VERDICT line and a matching process exit code; no controller/scheduler/apiserver processes remain afterward (unless --keep-running was passed); the run's log files are present on disk regardless.

The comparison/aggregation step is self-contained and independently testable

aggregate_verdict() (and its build_verdict_record() formatting helper) takes plain records as function arguments -- no live command, no file I/O, no global deployment state -- so it can be invoked directly with hand-written synthetic records, with no live deployment required. This is what task 8.2's dedicated test harness exercises; source the script (source run-resource-verify.sh) to get aggregate_verdict(), build_verdict_record(), and controller_values_match() in scope without triggering main() or any live deployment -- the bottom of the file is guarded ([[ "${BASH_SOURCE[0]}" == "${0}" ]]) specifically so sourcing it has no side effects.

Error handling this skill implements

Per design.md's Error Handling section:

• Unrecognized resource name -- resupdate's own follow-up xcecli_resval() verification (task 2.2) reports it as rejected; the aggregation step (task 7.1) treats this as a run-level failure and names the specific rejected resource(s) in its FAILURE lines.
• Layer mismatch -- reported with both the injected and observed value, attributed to the specific layer (controller/scheduler/apiserver).
• Layer unreachable (e.g. apiserver never came up, or the expected trace line never appears before the bounded wait elapses) -- treated the same as a mismatch for verdict purposes (FAIL), but labeled UNREACHABLE distinctly from MISMATCH in the report, never a silent skip.
• Deployment failure (controller/scheduler/apiserver fails to start) -- the skill aborts before attempting injection, with a clear "deployment failed at step X" message. Cleanup still runs on this path: a trap registered before the first process starts covers every deployment step from the first process start onward, not just the post-injection verification path.

Known pre-existing bugs relevant to this skill (not fixed here)

• bug-1572 (open): the live scheduler daemon can double-free/crash if the SAME resource is re-pushed to it a second time while it's running (e.g. left alive for a while under --keep-running, or a controller periodic resync). A single run's own injection (one resupdate per resource, one verification pass) does not trigger this, but don't rely on a --keep-running deployment staying up indefinitely once left running.
• bug-1574 (open): xce_test can non-deterministically SEGFAULT on a string-typed resupdate (-n type/-n model) -- uninitialized-memory UB, not this feature's own bug. This is why the default resource set is numeric-only; avoid adding type/model to a custom resource=value argument until this is fixed.
• bug-1579 (fixed): host-attrs resources (cpuf, maxcpus, etc.) used to false-MISMATCH at the controller layer because the synthetic worker disconnected before the controller-side check ran. Fixed by nesting the controller check inside the still-open injecting worker session.

Not using context: fork for the orchestration script

Neither this skill's frontmatter nor the invocation instructions above set context: fork, for the same reason xce-test-ha and xce-deploy-ha both call out explicitly in their own SKILL.mds: run-resource-verify.sh starts real backgrounded controller/scheduler/apiserver processes for the duration of the run, and context: fork's own teardown sends SIGHUP through the whole process group on return -- the same confirmed-broken mechanism xce-test-ha's soak driver and xce-deploy-ha both hit -- which would silently kill those processes mid-run, before cleanup or the verdict line ever fires. Dispatch it directly via Bash, or via the Agent tool (empirically verified safe for this exact pattern -- see xce-test-ha's and xce-deploy-ha's own SKILL.mds), not via a forked Skill-tool context.

Dependencies

• xce_test's worker-context resupdate/join and client-context resval -- resource-name table gap closed (tasks 2.1/2.2).
• xce-deploy-ha's deploy-ha.sh / start-all.sh -- apiserver step and combined start-all.sh (task 3.1).
• Scheduler and apiserver trace-log lines (tasks 4.1 and 4.2).
• This skill's own run-resource-verify.sh -- deployment (task 5.1), injection + per-layer checks (tasks 6.1-6.4), and verdict aggregation + cleanup-on-every-path (task 7.1) -- all done.

Requirements traceability

This document describes design.md's "xce-test-resources skill" component (Components and Interfaces → Test Tooling / Bash), covering Requirements 3.1, 3.2, 4.1, 4.2, 4.3, 5.1, 5.2, 5.4, 6.1, 6.2, 6.3, 6.4, 6.5. See .kiro/specs/resource-value-verification/requirements.md and design.md for the full text.