myxce/.kiro/specs/resource-value-verification/design.md

# Design Document

## Overview
This feature closes a coverage gap in `xce_test`'s worker-context `resupdate` command (its resource-name table is missing 7 real dynamic resources) and adds a repeatable, self-contained skill that injects a synthetic resource value through `resupdate` and confirms it is correctly received and stored at the controller, the live scheduler, and the Go apiserver — the exact three-layer propagation path that took a multi-hour live-deployment investigation to verify manually during bug-1569.

**Purpose**: Turn a class of bug (resource-value routing/propagation defects) that previously required deploying real worker/controller/scheduler/apiserver daemons and waiting on periodic reporting cycles into a command that reproduces and confirms the fix in seconds.
**Users**: XCE engineers debugging or regression-testing resource-value propagation across the controller/scheduler/apiserver boundary.
**Impact**: Extends `xce_test` (test.c) with a small, backward-compatible change; extends `xce-deploy-ha`'s `deploy-ha.sh`/`SKILL.md` with an opt-in, conditionally-generated apiserver deployment step (see Design Decisions below — no change to existing callers that don't need an apiserver); adds two single-line trace-log additions (scheduler, apiserver); adds one new skill. No production runtime *behavior* changes — the trace-log additions are inert log-only output at debug/trace level, and the `xce-deploy-ha` extension only activates when the apiserver binary is present, mirroring its existing worker/scheduler conditional-generation pattern.

### Goals
- `resupdate` accepts any resource name the cluster has registered (Requirement 1).
- Confirm the existing `join` + `resupdate` worker-context flow needs no real worker daemon (Requirement 2).
- A single command deploys controller + scheduler + apiserver, injects one or more resource values, and reports PASS/FAIL per layer (Requirements 3–6).

### Non-Goals
- Renaming `xce_test`, per-command help, reference documentation, JSON output mode, non-interactive batch mode, argument-convention unification — explicitly deferred to a future "enhance xce_test" project (see requirements.md Future Considerations).
- A new live scheduler admin query RPC — rejected in favor of trace-log observation, which needs no new RPC at all (see research.md Design Decisions).
- The record+replay+REPL flow and the apiserver REST/`xcecli` query path — both considered and superseded during design validation in favor of live trace-log observation at each layer's point of application; see research.md Design Decisions for why (this feature's own originating bugs, bug-1568 and bug-1569A, were both storage-layer defects found via exactly this technique, and the REPL's step/continue requirement had already produced one false negative live).
- General-purpose apiserver production deployment/HA — the new `xce-deploy-ha` apiserver step is a minimal single-node instance for local verification only, and no longer needs functional auth/RBAC since no REST query is issued against it.
- A new, standalone apiserver deploy script — considered and superseded in favor of extending `xce-deploy-ha` itself, reusing its already-precedented conditional (binary-present-only) generation pattern instead of re-implementing the same conventions in parallel (see research.md Design Decisions).
- CI/automated-pipeline integration.
- Fixing unrelated bugs the verification run happens to surface — those are reported, not silently patched, by this feature.

## Boundary Commitments

### This Spec Owns
- `set_resval()`/`cmd_resupdate` in `src/controller/testsuite/test.c`: accepting any registered resource name, with an explicit type-override escape hatch.
- A single trace/debug log line in `xce_core_push_events()`'s `XCE_CORE_EVENT_RESVALS_DELTA` case (`src/scheduler/src/core_events.c`), emitted at the point the live scheduler daemon applies a resource update — the only scheduler-side change this spec makes.
- A single trace/debug log line in `OnResourceLocationUpdated` (`src/frontend/apiserver/backend/handler.go:92-95`), emitted at the point the apiserver stores a received resource update — the only apiserver-side change this spec makes.
- An opt-in, conditionally-generated apiserver deployment step added to `xce-deploy-ha`'s `src/tools/deploy-ha.sh` and documented in its `SKILL.md` (Step 4d, alongside the existing Step 4b worker/scheduler steps) — generates `start-apiserver.sh` only when a pre-built apiserver binary already exists at its own Go-build output path (same "caller opts in by having already built it" spirit as the worker/scheduler conditional, though checked at a different path since apiserver is a separate build system — see Components for the precise distinction). Existing `xce-deploy-ha` callers that never build an apiserver (e.g. `xce-test-ha`) see zero behavior change.
- A new skill (`.claude/skills/xce-test-resources/`) that orchestrates deploy + inject + three-layer verify + PASS/FAIL report.

### Out of Boundary
- `resource_sync.c`'s real, production periodic load-index collection — unchanged.
- The controller's resval-forwarding hooks (`do_update_resvals`, `sched_resource_hook_fun`, `apiserver_resource_hook_fun`) — unchanged; this design only consumes them.
- The scheduler's REPL (`repl.c`/`repl_info.c`), recording admin RPCs, and the apiserver's REST API/`xcecli` query path — none of these are touched or driven by this design (superseded by the trace-log decision; see research.md Design Decisions). They remain available as a future follow-up if a query-layer-specific check is ever needed.
- All other scheduler/apiserver logic beyond the two single-line trace additions listed under "This Spec Owns" above — no behavior change, no new code paths, purely additive log output.
- `xce-deploy-ha`'s existing controller/worker/scheduler steps (1–4c) — unmodified; only a new, conditionally-triggered step is added alongside them.
- The broader "enhance xce_test" ideas listed in requirements.md's Future Considerations.

### Allowed Dependencies
- `xce-deploy-ha`'s controller deployment conventions and its existing worker/scheduler conditional-generation pattern (directory layout, TLS cert generation, `cluster.bin` build via `xceadmin`, Step 3's "generate the extra start script only if the binary exists" rule) — reused directly, by extending the same script, not duplicated in a parallel one (see research.md Design Decisions).
- `xce_test`'s existing worker context (`join`, `resupdate`) and client context (`resval`) — unmodified except for Requirement 1's table extension.
- `make build-server` (`src/frontend/Makefile`) to build the apiserver binary; `make test-integration`'s `XCE_BIN`/`SRV_URL` convention as the closest existing precedent for pointing it at a controller.
- Each daemon's own existing logging infrastructure (`tinylog` in C for the scheduler, zap `logger.Debug` in Go for the apiserver — per `error-handling.md`) as the channel the new trace lines are written through; no new logging mechanism is introduced.

### Revalidation Triggers
- The wire shape of `xce_resval_t` / `resupdate`'s CLI flags changes.
- `xce_core_push_events()`'s `XCE_CORE_EVENT_RESVALS_DELTA` handling, or `OnResourceLocationUpdated`'s call site, is refactored such that the trace log's format/location moves.
- `xce-deploy-ha`'s Step 3 conditional-generation pattern (or its directory layout) changes shape — this feature's apiserver step is now part of that same script, so a shape change there is a direct, first-party revalidation trigger rather than a drift risk between two separate scripts.
- If a future bug turns out to live specifically in the query/REST/serialization path rather than storage, revisit adding a real query-based check for that layer (not precluded by this design, just not built here).

## Architecture

### Existing Architecture Analysis
A resource value reaches all three layers through **one** controller-side write, not three independent paths (see research.md): `do_update_resvals()` applies it to the controller's own `w->host.resvals` (queried via `resval`) and fires `HOOK_DETAIL_RESOURCE_UPDATE`, which fans out to both `sched_resource_hook_fun` (scheduler) and `apiserver_resource_hook_fun` (apiserver). This design's job is purely to exercise this existing path with a synthetic worker and observe the result at each of the three existing points of application — it introduces no new production data path, and (per the trace-log decision) reads state at the storage point rather than through a query surface.

### Architecture Pattern & Boundary Map

```mermaid
graph TB
    Skill[xce-test-resources skill]
    WorkerCtx[xce_test worker context]
    ClientCtx[xce_test client context]
    Controller[xce_controller]
    Scheduler[xce_scheduler daemon]
    SchedLog[scheduler log]
    Apiserver[apiserver]
    ApiLog[apiserver log]

    Skill --> WorkerCtx
    WorkerCtx -->|join and resupdate| Controller
    Skill --> ClientCtx
    ClientCtx -->|resval query| Controller
    Controller -->|resource update notify| Scheduler
    Controller -->|resource update push| Apiserver
    Scheduler -->|trace log on apply| SchedLog
    Apiserver -->|trace log on apply| ApiLog
    Skill -->|tail and grep| SchedLog
    Skill -->|tail and grep| ApiLog
```

**Architecture Integration**:
- Selected pattern: black-box orchestration over existing interfaces plus two minimal, inert trace-log additions at each layer's point of application — this feature adds a coordinating layer (the skill), one small compatibility fix (`resupdate`'s resource table), and the two log lines; it does not add a new data path or query surface.
- Domain boundaries: `resupdate` table fix (C, test.c) is independent of the skill (bash orchestration); the `xce-deploy-ha` apiserver extension is independent of both and reusable by any `xce-deploy-ha` caller, not just this skill; the two trace-log additions are independent single-line changes in their respective daemons.
- Existing patterns preserved: `xce-deploy-ha`'s scratch-directory deploy convention and worker/scheduler conditional-generation pattern (extended, not duplicated, for apiserver), `xce-test-ha`'s VERDICT-line reporting convention, each daemon's own existing logging infrastructure (no new logging mechanism).
- New components rationale: the `xce-test-resources` skill is new because no existing tool exercises all three layers together; the `resupdate` fix is new because the current table is demonstrably incomplete; the `xce-deploy-ha` apiserver step and `start-all.sh` extend an existing tool rather than introduce a new one, following its own established conditional-generation precedent; the two trace-log lines are new because neither layer currently logs the value it applies (confirmed by direct inspection — see research.md).
- Steering compliance: production code changes are two single-line, debug/trace-level log additions with no behavior change, plus `test.c`'s test-tooling scope; TDD applies to the `set_resval()` change (see Testing Strategy).

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|------------------|-------|
| Test client (C, gnu17) | `xce_test` (existing binary, `src/controller/testsuite/test.c`) | Inject via worker context; query controller via client context | Requirements 1, 2, 3 |
| Scheduler (C, gnu17) | Live `xce_scheduler` daemon (existing, one new trace-log line) | Applies the resource update and logs it at the point of application | Requirement 4; see research.md Design Decisions |
| API server (Go) | `apiserver` binary via `make build-server` (one new trace-log line in `backend/handler.go`) | Receives and stores the resource update, logs it at the point of application | Requirement 5; see research.md Design Decisions |
| Orchestration | Bash skill script, `xce-deploy-ha`/`xce-test-ha` conventions | Deploy, inject, tail/grep each layer's log, verify, report, clean up | Requirement 6 |

## File Structure Plan

### Directory Structure
```
src/
└── controller/testsuite/
    ├── resval_typeresolve.c            # NEW: set_resval()'s name→Value_u_which resolution, extracted for isolated unit testing
    ├── resval_typeresolve.h            # NEW: its declaration
    ├── test_resval_typeresolve.c       # NEW: Unity unit tests for resval_typeresolve.c (4 cases, see Testing Strategy)
    └── test.c                          # MODIFIED: set_resval() delegates to resval_typeresolve.c; cmd_resupdate() gains -t flag + post-update accept/reject verification
src/controller/meson.build             # MODIFIED: add resval_typeresolve.c to xce_test_src (both xce_test and test_resval_typeresolve.c link against it); add test_resval_typeresolve.c + a new controller_unit_test-style test() target — these are hand-maintained explicit file lists, not globs
src/scheduler/src/core_events.c        # MODIFIED: one new trace/debug log line in xce_core_push_events()'s XCE_CORE_EVENT_RESVALS_DELTA case
src/frontend/apiserver/backend/handler.go  # MODIFIED: one new trace/debug log line in OnResourceLocationUpdated
src/tools/
└── deploy-ha.sh                         # MODIFIED: Step 3 gains (a) a conditional apiserver-config-generation branch mirroring the existing worker/scheduler conditional, generating start-apiserver.sh (minimal boot-only config + start command — no functional auth/RBAC required, see research.md Design Decisions), and (b) an always-generated start-all.sh that sequences start.sh + health-check + whichever of start-worker.sh/start-scheduler.sh/start-apiserver.sh exist for this deploy — automates the manual recipe SKILL.md's Steps 4/4b/4c already document
.claude/skills/
├── xce-deploy-ha/
│   └── SKILL.md                         # MODIFIED: new Step 4d documenting the opt-in apiserver step, alongside existing Step 4b (worker/scheduler)
└── xce-test-resources/
    ├── SKILL.md                        # NEW: skill description, usage, VERDICT format (xce-test-ha convention)
    └── run-resource-verify.sh          # NEW: orchestrates deploy + inject + 3-layer verify (log-tail/grep for scheduler+apiserver) + verdict + cleanup
```

### Modified Files
- `src/controller/testsuite/test.c` — `set_resval()` becomes a thin wrapper around the new `resval_typeresolve.c` module; `cmd_resupdate()` gains an optional `-t <numeric|string>` flag per `-n`/`-v` pair, and, after sending the update, a follow-up per-name verification query (see Components below — Requirement 1.2/1.3's mechanism).
- `src/controller/meson.build` — `xce_test_src` (line 57) gains `testsuite/resval_typeresolve.c` so both `xce_test` and the new unit test binary can link it; a new Unity test source list/target (following the `controller_unit_test`/`sched_push_unit_test` pattern already in this file, e.g. lines 86, 215, 222) gains `testsuite/test_resval_typeresolve.c`. Both lists are hand-maintained and explicit, not glob-based — this file must be edited or the new module/test never builds.
- `src/scheduler/src/core_events.c` — `xce_core_push_events()`'s `XCE_CORE_EVENT_RESVALS_DELTA` case gains one trace/debug-level log line (via this component's existing `tinylog`-based logging, e.g. `TRACE`) emitting the resolved worker host, resource name, and applied value, immediately after the value is written into the worker's live state. Debug/trace level only — never fires at WARN/ERROR, never changes control flow. This is the exact site bug-1568 was root-caused at via ad hoc instrumentation; this spec makes that instrumentation permanent and machine-greppable.
- `src/frontend/apiserver/backend/handler.go` — `OnResourceLocationUpdated` (line 92) gains one `logger.Debug(...)` call (zap, per this component's existing convention) logging the resource name, host/location, and stored value, immediately after `store.ResourceLocations.Put()`.
- `src/tools/deploy-ha.sh` — Step 3 gains two additions:
  1. A new conditional branch, same *opt-in-by-presence* spirit as the existing worker/scheduler branch (but checking a pre-built binary at the Go build's own output path, e.g. `/tmp/apiserver`, rather than alongside `$BINARY` — a different build system, see Components below): if present, generate `$TOP/scripts/start-apiserver.sh` (minimal boot-only config) and a `logger.yaml` with `logging.level: debug`; absent, generates nothing, so existing callers are unaffected.
  2. An always-generated `$TOP/scripts/start-all.sh`: runs `start.sh` for the controller node(s), waits on Step 5's existing health-check, then runs whichever of `start-worker.sh`/`start-scheduler.sh`/`start-apiserver.sh` were actually generated for this deploy (skips any that weren't) — the same backgrounding pattern (`&`, or `setsid nohup … & disown` per SKILL.md's own noted improvement) Steps 4/4b/4c already document doing by hand.
- `.claude/skills/xce-deploy-ha/SKILL.md` — documents the new opt-in apiserver step as "Step 4d" (following the existing Step 4b worker/scheduler pattern) and `start-all.sh` as a documented shortcut for Steps 4/4b/4c combined.

### New Files
- `src/controller/testsuite/resval_typeresolve.c`/`.h` — `set_resval()`'s existing `resinfo[]` table lookup plus the new numeric-by-default fallback, moved out of `test.c` into a small, dependency-light module (no networking, no REPL state) so it is independently unit-testable. `test.c` (4298 lines, its own `main()`, `libedit`/network/REPL dependencies) cannot itself be white-box `#include`d into a Unity test the way smaller single-purpose files in this codebase are — extraction is required, not optional, to satisfy Requirement 1.1/1.4's testability.
- `src/controller/testsuite/test_resval_typeresolve.c` — Unity test program for `resval_typeresolve.c`, registered as its own `meson.build` `test()` target (see Modified Files above); this is the file task 1.1's RED tests and task 2.1's GREEN tests live in.

## System Flows

```mermaid
sequenceDiagram
    participant Skill as xce-test-resources
    participant Worker as xce_test (worker ctx, ONE session)
    participant Nested as xce_test (nested ! client ctx)
    participant Ctrl as controller
    participant Sched as scheduler daemon
    participant Api as apiserver

    Skill->>Worker: join synthetic-worker
    Skill->>Worker: resupdate -n <res> -v <value>
    Worker->>Ctrl: xce_update_resvals (wire)
    Ctrl->>Ctrl: apply to w.host.resvals
    Ctrl->>Sched: notify schedUpdateResvals
    Ctrl->>Api: notify schedUpdateResvals
    Sched->>Sched: apply + trace-log resource, host, value
    Api->>Api: store.ResourceLocations.Put + debug-log resource, host, value
    Worker->>Nested: !<shell-cmd> --internal-controller-check (worker STILL connected)
    Nested->>Ctrl: resval -n <res> -m synthetic-worker (query)
    Ctrl-->>Nested: controller-observed value
    Nested-->>Worker: result written to a marker file (nested exit code ignored)
    Worker->>Ctrl: quit (worker disconnects HERE -- not before)
    Skill->>Skill: parse marker file for controller-observed value
    Skill->>Sched: tail/grep scheduler log for expected line (bounded wait)
    Sched-->>Skill: scheduler-observed value (or timeout)
    Skill->>Api: tail/grep apiserver log for expected line (bounded wait)
    Api-->>Skill: apiserver-observed value (or timeout)
    Skill->>Skill: compare all three vs injected value, emit verdict
```

**Load-bearing constraint the diagram makes explicit (as-built, bug-1579)**: the controller-layer check is NOT an independently-launched client session run after the worker disconnects. Any host-attrs-derived resource (`type`/`model`/`cpuf`/`maxcpus`/`maxmem`/`maxswap`/`maxtmp`/`ndisks` — `worker.c`'s `N_HOST_ATTRS`) is reset to baseline by the controller the instant the injecting worker disconnects — so a controller-layer check run as a separate, later session against any host-attrs name would ALWAYS false-MISMATCH, independent of whether the underlying propagation is actually correct (discovered as bug-1579, task 6.2, round-1 REJECTED). The fix nests the controller-layer `resval` query INSIDE the injecting worker's own still-open session, via `xce_test`'s `!<shell-command>` escape (`preprocess_and_run()` — works in any REPL context, including `worker`, with no `leave_ctx()`/disconnect) — the query genuinely runs before `quit`, not merely before some unrelated timeout. The scheduler and apiserver legs have no such constraint: both are pure log-tail/grep against a file, entirely independent of whether the worker connection is still open, so they run after the worker session ends like any other step.

Key simplification from the superseded design: no `record`/`replay`/REPL round-trip and no REST/`xcecli` query are needed — the scheduler and apiserver legs both poll their own live daemon's log file for the trace line the injection is expected to produce, within a bounded wait, exactly like the controller leg's own bounded-wait posture. This removes the REPL's `step`/`continue` pitfall entirely (there is no replay step in this flow) and the apiserver's auth/RBAC config from the critical path (see research.md Design Decisions).

## Requirements Traceability

| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1–1.4 | `resupdate` accepts any registered resource name | `set_resval()`/`cmd_resupdate` (test.c) | CLI flags `-n`/`-v`/`-t` | — |
| 2.1–2.2 | Inject without a real worker daemon | `xce_test` worker context (`join`, `resupdate`, existing) | CLI | Sequence diagram: join → resupdate → wire update |
| 3.1–3.2 | Controller-side visibility check | `xce-test-resources` skill, `xce_test` client context (`resval`, existing) | CLI | Sequence diagram: `resval` query → controller-observed value |
| 4.1–4.3 | Scheduler-side visibility check | `xce-test-resources` skill, scheduler trace log (`core_events.c`, new) | Log line, bounded-wait grep | Sequence diagram: scheduler applies + trace-logs → skill tails/greps scheduler log |
| 5.1, 5.2, 5.4 | Apiserver-side visibility check | `xce-test-resources` skill, `xce-deploy-ha` apiserver step, apiserver trace log (`handler.go`, new) | Log line, bounded-wait grep | Sequence diagram: apiserver stores + debug-logs → skill tails/greps apiserver log |
| 5.3 | Apiserver local deployment capability | `xce-deploy-ha` apiserver step + `start-all.sh` | Bash script | — |
| 6.1–6.5 | Repeatable run, PASS/FAIL verdict, cleanup | `xce-test-resources` skill (`run-resource-verify.sh`) | CLI, VERDICT line | Sequence diagram: final compare + verdict step |

## Components and Interfaces

| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|-----------|--------------|--------|---------------|---------------------------|-----------|
| `set_resval()` / `cmd_resupdate` | Test Tooling / C | Accept any registered resource name for injection | 1.1–1.4 | `xcecli_update_resvals` (P0), `xcecli_resval` (P0) | Batch |
| Scheduler resval trace log | Scheduler Core / C | Log the resolved host/resource/value at the point the live daemon applies an update | 4.1–4.3 | `xce_core_push_events()` (P0), `tinylog` (P0) | Event |
| Apiserver resval trace log | API Server / Go | Log the resource/host/value at the point the apiserver stores a received update | 5.1, 5.2, 5.4 | `OnResourceLocationUpdated` (P0), zap `logger` (P0) | Event |
| `xce-deploy-ha` apiserver step | Deployment Tooling / Bash | Extend `deploy-ha.sh` to conditionally stand up a minimal, boot-only single-node apiserver for local verification | 5.3 | `make build-server` (P0), a running controller (P0), `deploy-ha.sh`'s existing Step 3 (P0) | Batch |
| `xce-test-resources` skill | Test Tooling / Bash | Orchestrate deploy, inject, three-layer verify (log-tail/grep for scheduler+apiserver), report, clean up | 3.1–3.2, 4.1–4.3, 5.1, 5.2, 5.4, 6.1–6.5 | `xce-deploy-ha` (P0, extended), `xce_test` (P0), scheduler/apiserver trace logs (P0) | Batch |

### Test Tooling / C

#### `set_resval()` / `cmd_resupdate`

| Field | Detail |
|-------|--------|
| Intent | Resolve a `-n <name> -v <value>` pair into a wire `xce_resval_t`, for any resource the cluster has registered |
| Requirements | 1.1, 1.2, 1.3, 1.4 |

**Responsibilities & Constraints**
- Resolve a resource name to a `Value_u_which` tag (`resval_typeresolve.c`): known non-numeric names (`type`, `model`) resolve to `Value_u_svalue`; every other name — including ones not seen before — defaults to `Value_u_fvalue`, overridable via a new optional `-t <numeric|string>` flag.
- Preserve existing behavior byte-for-byte for every currently-supported resource name (Requirement 1.4) — this is a strictly additive change to the fallback path, not a rewrite of the existing table lookup.
- Report per-name accept/reject outcome when a single invocation carries multiple `-n`/`-v` pairs (Requirement 1.3). **Confirmed via direct source inspection**: the controller's synchronous RPC reply cannot carry this signal — `update_resval_list()` (`src/controller/worker.c:185-198`) unconditionally `return 0` regardless of any per-item failure, so `do_update_resvals()`'s `reply.status` is always success even when every submitted name is unregistered. `cmd_resupdate()` therefore verifies acceptance itself: after the `xcecli_update_resvals()` call, it issues one follow-up `xcecli_resval()` query per submitted `(resource, worker)` pair — the same client-library call `cmd_resval`/`resval` already uses — and compares the returned value against what was just sent. A match is "accepted"; a not-found result or a mismatched value is "rejected", satisfying Requirement 1.2's clear per-name error without any controller-side change.

**Dependencies**
- Outbound: `xcecli_update_resvals` (`src/lib/client.c`) — unchanged, existing wire call (P0). `xcecli_resval` (`src/lib/client.c`) — unchanged, existing wire call, reused here as the accept/reject verification signal (P0).

**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [x] / State [ ]

##### Batch / Job Contract
- Trigger: operator (or the `xce-test-resources` skill) runs `resupdate -n <name> -v <value> [-t <numeric|string>] [-n <name2> -v <value2> ...]` in the `xce_test` worker context.
- Input / validation: resource name (string, required per pair), value (string, parsed per resolved type), optional type override.
- Output / destination: one `xce_update_resvals` RPC to the controller carrying every resolved pair; then one `xcecli_resval` query per pair to confirm it landed; per-pair accept/reject reported to stdout based on that follow-up query, not the update RPC's own reply (which cannot distinguish per-item outcomes — see Responsibilities above).
- Idempotency & recovery: re-running with the same name/value is safe (matches existing `resupdate` semantics — a plain overwrite, no accumulation).

**Implementation Notes**
- Integration: the type-resolution table lookup moves into `resval_typeresolve.c` unchanged in behavior; `cmd_resupdate()` gains the new post-update verification loop described above.
- Validation: the follow-up `xcecli_resval()` query adds one RPC round-trip per submitted resource name — acceptable for a test/verification tool, not a production hot path.
- Risks: a resource that is genuinely registered but whose value legitimately hasn't propagated yet (a race, not a rejection) could be misreported as "rejected" if queried immediately; mitigate with a short bounded retry on the follow-up query before concluding rejection (this codebase's existing resval-propagation timing, confirmed live during the bug-1569 investigation this spec originated from, is sub-second for a controller-local query).

### Scheduler Core / C

#### Scheduler resval trace log

| Field | Detail |
|-------|--------|
| Intent | Make the live scheduler daemon's application of a resource update observable via its own log, without a query RPC or replay |
| Requirements | 4.1, 4.2, 4.3 |

**Responsibilities & Constraints**
- Emit one trace/debug-level log line inside `xce_core_push_events()`'s `XCE_CORE_EVENT_RESVALS_DELTA` case (`src/scheduler/src/core_events.c`), immediately after the resolved worker's live resource value is written — including the resolved host name, resource name, and applied value, in a fixed, greppable format.
- Debug/trace level only (`TRACE`, which expands to `tlog_debug3` — `src/include/utils.h:93`) — never WARN/ERROR, never on any decision path, so normal operation is unaffected; this is purely additive observability.
- Fires on every real resource update in production too (not just this feature's synthetic injections) — acceptable, since it is the same log-volume category as this component's other existing `TRACE` calls (e.g. `apiserver_worker_change`'s `TRACE` in `src/controller/apiserver.c:446`).
- **Load-bearing operational requirement**: `tlog_debug3`/`TRACE` output is gated by the scheduler's own runtime log level (`src/scheduler/src/main.c`'s `-v` flag, cumulative — confirmed by reading `main.c`: `arguments->verbose++` per `-v`, then `verbose_to_loglevel(arguments.verbose)`). At the daemon's default (unset `-v`) log level, this line is silently never written, and the verification skill's log-grep would time out on every run — a false FAIL, not a bug in the feature under test. The skill's scheduler-deploy step MUST start the scheduler daemon with `-vvv` (verbosity 3, matching `tlog_debug3`) — the same convention `xce-deploy-ha`'s generated `start.sh` already uses for the controller (`start.sh ctrl-1 -vvv`) — see task 5.1.

**Dependencies**
- Inbound: fires as part of the existing `XCE_CORE_EVENT_RESVALS_DELTA` handling path (P0) — no new caller, no new event type.
- Outbound: `tinylog` (P0) — existing logging infrastructure, unchanged.

**Contracts**: Service [ ] / API [ ] / Event [x] / Batch [ ] / State [ ]

##### Event Contract
- Trigger: the live scheduler daemon processes a `RESVALS_DELTA` event carrying a `resupdate`-injected (or real) value.
- Output / destination: one line to the scheduler daemon's own log file.
- Idempotency & recovery: log-only; no state changes, safe to fire on every event including retries/duplicates.

### API Server / Go

#### Apiserver resval trace log

| Field | Detail |
|-------|--------|
| Intent | Make the apiserver's storage of a received resource update observable via its own log, without a REST/CLI query |
| Requirements | 5.1, 5.2, 5.4 |

**Responsibilities & Constraints**
- Add one `logger.Debug(...)` call (zap, per `error-handling.md`'s existing convention) inside `OnResourceLocationUpdated` (`src/frontend/apiserver/backend/handler.go:92-95`), immediately after `store.ResourceLocations.Put(key, *res)`, logging the resource name, location/host, and stored value.
- Debug level only — never `Error`/`Warn`, no behavior change to the store write itself.
- **Load-bearing operational requirement**: the apiserver's log level is config-driven (`config/logger.yaml`'s `logging.level`, loaded by `loadLoggerConfig()` in `src/frontend/apiserver/utils/logger.go`). Zap's default level is not Debug, so unless this config explicitly sets `level: debug`, `logger.Debug(...)` output is silently suppressed and the skill's log-grep would time out on every run — a false FAIL, not a bug in the feature under test. `xce-deploy-ha`'s apiserver step (task 3.1) MUST generate a `logger.yaml` with `logging.level: debug`.

**Dependencies**
- Inbound: fires as part of the existing `OnResourceLocationUpdated` notification-handling path (P0) — no new caller.
- Outbound: zap `logger` (P0) — existing logging infrastructure, unchanged.

**Contracts**: Service [ ] / API [ ] / Event [x] / Batch [ ] / State [ ]

##### Event Contract
- Trigger: the live apiserver process receives and stores a resource-location update pushed from the controller.
- Output / destination: one line to the apiserver's own log file.
- Idempotency & recovery: log-only; no state changes.

### Deployment Tooling / Bash

#### `xce-deploy-ha` apiserver step + `start-all.sh`

| Field | Detail |
|-------|--------|
| Intent | Extend `deploy-ha.sh` to (a) conditionally stand up a minimal, boot-only single-node apiserver against an already-deployed controller, and (b) generate a single `start-all.sh` that sequences controller → (worker/scheduler/apiserver, whichever were generated) so callers need no manual multi-script sequencing |
| Requirements | 5.3, 6.1 |

**Responsibilities & Constraints**
- Apiserver step: generate the minimal config needed for the process to **boot and connect to the controller**, plus a `logger.yaml` with `logging.level: debug` (required for task 4.2's trace line to actually be emitted — see that component's Load-bearing operational requirement), modeled on `make test-integration`'s existing `XCE_BIN`/`SRV_URL` convention. Because verification is trace-log-based (Requirement 5, see research.md Design Decisions), this config no longer needs to produce *functional* auth/RBAC — no REST query is ever issued against this instance — only enough to start the process and complete the apiserver↔controller connection that `apiserver_resource_hook_fun`/`OnResourceLocationUpdated` depends on.
- Generated **only if** a pre-built apiserver binary is already present at its conventional output path (e.g. `/tmp/apiserver`, `make build-server`'s default). **Not an exact mirror of the worker/scheduler conditional**: those binaries live in the same meson build directory as the controller and are simple presence-checked; the apiserver is a separate Go build (`make build-server`, `src/frontend/Makefile`) with its own output path, so `deploy-ha.sh` does **not** invoke `make build-server` itself — building the apiserver remains the caller's prerequisite, exactly like the existing "run `/xce-build controller` first if stale" convention in SKILL.md's Prerequisites section, just naming a different build command. This preserves the same "caller opts in by having already built the optional component" semantics the worker/scheduler conditional relies on for backward compatibility — a caller who never runs `make build-server` sees no apiserver step generated, same as one who never builds worker/scheduler today.
- `start-all.sh`: always generated (independent of which optional components exist for a given deploy). Runs `start.sh` for the controller node(s), waits on Step 5's existing health-check, then runs whichever of `start-worker.sh`/`start-scheduler.sh`/`start-apiserver.sh` were actually generated — skipping any that weren't — with the scheduler started at `-vvv` (verbosity 3) specifically so task 4.1's `TRACE`/`tlog_debug3` line is actually emitted (see that component's Load-bearing operational requirement).
- Does not attempt HA/multi-instance apiserver deployment — single instance only, sufficient for this feature's verification needs.

**Dependencies**
- Inbound: `xce-test-resources` skill (P0) — the primary consumer motivating this extension; also available to any other `xce-deploy-ha` caller by construction (same reusability posture as the existing worker/scheduler steps).
- Outbound: `make build-server` (P0), a running controller (P0), `deploy-ha.sh`'s existing Step 3/Step 5 (P0).

**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [x] / State [ ]

##### Batch / Job Contract
- Trigger: `deploy-ha.sh` invoked normally (apiserver/`start-all.sh` generation happens automatically as part of Step 3, not a separate opt-in flag the caller must pass); `start-all.sh` then invoked directly by the `xce-test-resources` skill, or by an operator.
- Input / validation: target controller host:port, a deploy/scratch directory (same as existing `xce-deploy-ha` inputs — no new required parameters).
- Output / destination: a running apiserver process connected to the controller, logs under the scratch directory (same convention as `xce-deploy-ha`'s `logs/`) — this is the log task 6.4 greps for Requirement 5's trace line.
- Idempotency & recovery: each invocation deploys fresh into a caller-specified scratch directory, matching `xce-deploy-ha`'s existing "every invocation is a fresh, self-contained deploy" convention — no in-place upgrade semantics.

**Implementation Notes**
- Integration: exact minimal-boot config shape is confirmed during implementation, not fully pinned here — see research.md Risks; scope is narrower than originally planned since functional auth/RBAC is no longer required.
- Validation: `start-all.sh`'s own controller health-check gate (reusing Step 5) must complete before it proceeds to start worker/scheduler/apiserver — the skill must additionally confirm the apiserver process has registered with the controller (connection-level, not HTTP-listener) before injecting and grepping for Requirement 5's trace line.
- Risks: apiserver startup/config requirements were previously this design's least-verified part; narrowed by dropping the REST/auth requirement, but boot-time config shape is still unconfirmed until implementation.

### Test Tooling / Bash

#### `xce-test-resources` skill

| Field | Detail |
|-------|--------|
| Intent | Deploy controller+scheduler+apiserver, inject one or more resource values via `resupdate`, verify all three layers, report PASS/FAIL, clean up |
| Requirements | 3.1, 3.2, 4.1, 4.2, 4.3, 5.1, 5.2, 5.4, 6.1, 6.2, 6.3, 6.4, 6.5 |

**Responsibilities & Constraints**
- Deploy controller, scheduler, and apiserver into a fresh scratch directory by invoking `xce-deploy-ha`'s `deploy-ha.sh` (Step 3) then its generated `start-all.sh` — no manual per-component sequencing lives in this skill; `start-all.sh` already handles controller→health-check→scheduler/apiserver ordering (see the `xce-deploy-ha` apiserver step + `start-all.sh` component above).
- Join a synthetic worker and inject one or more resource values via `resupdate` (default set: the 7 names Requirement 1 adds, plus a couple of already-supported names for regression coverage).
- Query the controller (`resval`), and tail/grep the scheduler daemon's and apiserver's own log files (bounded wait each) for the trace lines their respective resval-update handlers now emit; compare all three against the injected value and attribute any mismatch or timeout to a specific layer (Requirement 4.3's distinct-reporting requirement generalizes to all three layers).
- Emit a single machine-parseable VERDICT line, following `xce-test-ha`'s existing convention (e.g. `[xce-test-resources] VERDICT resources=7 layers_checked=3 mismatches=0 result=PASS`), plus the usual terminal PASS/FAIL oracle line.
- Clean up all deployed **processes** and runtime/data state (PIDs, sockets, controller/scheduler data directories) unconditionally — on PASS, on FAIL, and on a mid-deployment abort alike — unless the operator passes a "keep running" flag (Requirement 6.5; see Error Handling's Deployment failure entry for why this must cover the abort path too, not just a completed run). **Log files under the scratch directory are always preserved**, independent of the "keep running" flag — this is a deliberate divergence from `xce-deploy-ha`'s own `cleanup.sh` (which wipes `logs/*.log`), because the Monitoring section below promises logs survive "for post-run inspection," and this feature's own trace-log verdict evidence (the exact scheduler/apiserver lines a FAIL was based on) would otherwise be unrecoverable after a completed run.

**Dependencies**
- Inbound: none (top-level entry point, invoked directly by an operator).
- Outbound: `xce-deploy-ha` (P0, extended with the apiserver step + `start-all.sh`), `xce_test` worker/client contexts (P0), the scheduler daemon's and apiserver's own log files (P0, via the two new trace-log additions).

**Contracts**: Service [ ] / API [ ] / Event [ ] / Batch [x] / State [ ]

##### Batch / Job Contract
- Trigger: `/xce-test-resources [resource=value ...]` (defaults to Requirement 1's gap-resource set if none given).
- Input / validation: optional list of `resource=value` pairs; optional scratch directory; optional "keep running" flag.
- Output / destination: per-layer pass/fail detail (injected vs. observed value, per resource, per layer) plus one VERDICT line and the standard terminal PASS/FAIL exit code, matching `xce-test-ha`'s reporting contract closely enough to be immediately familiar to this codebase's engineers.
- Idempotency & recovery: safe to re-run — fresh scratch directory per run (`xce-deploy-ha` convention), no shared mutable state across runs.

**Implementation Notes**
- Integration: invokes `deploy-ha.sh` then its generated `start-all.sh`, which already gates scheduler/apiserver startup on the controller health-check internally — this skill does not re-implement that sequencing.
- Integration: registers cleanup via a `trap` (or equivalent) as soon as `start-all.sh` is invoked, not just at the end of a successful run — deliberately stronger than `xce-deploy-ha`'s own convention (a separate, manually-invoked `cleanup.sh`), because Requirement 6.5 requires cleanup-by-default on every path including a mid-deployment abort (see Error Handling).
- Validation: the scheduler and apiserver checks are both a bounded-wait poll of a live log file (no replay, no step/continue) — poll with a short interval up to a generous timeout, consistent with `xce-test-ha`'s existing soak-test conventions; a match found before timeout short-circuits the wait.
- Risks: timing-sensitive (bounded waits for controller→scheduler and controller→apiserver propagation); use generous, explicit timeouts, not tight polling that could flake.

## Error Handling

### Error Strategy
This feature is a verification tool, not a production service — "errors" are primarily reportable verification failures (a layer did not observe the injected value) rather than exceptions to recover from mid-run.

### Error Categories and Responses
- **Unrecognized resource name** (Requirement 1.2): `resupdate` reports a clear per-name error, sourced from its own follow-up `xcecli_resval()` verification query (see Components — the controller's update RPC reply cannot carry this signal); the skill treats this as a run-level failure with the specific rejected name(s) named in its report.
- **Layer mismatch** (Requirements 3.2, 4.2, 5.2): the skill reports which layer(s) failed, with both the injected and observed value, per Requirement 6.3.
- **Layer unreachable** (e.g. apiserver never came up, or the expected trace line never appears in the scheduler/apiserver log before the bounded wait elapses): treated the same as a mismatch for verdict purposes (FAIL, layer identified, "timeout" distinguished from "mismatch" in the report), not a silent skip.
- **Deployment failure** (controller/scheduler/apiserver fails to start): the skill aborts before attempting injection, with a clear "deployment failed at step X" message. **Cleanup still runs on this path** — this is a deliberate strengthening of `xce-deploy-ha`'s precedent, not a mirror of it: `xce-deploy-ha`'s `deploy-ha.sh` has no trap-based cleanup and leaves already-started processes running on a mid-deploy failure, requiring the caller to run its separate `cleanup.sh` manually (confirmed against `src/tools/deploy-ha.sh`). Requirement 6.5 promises cleanup by default regardless of outcome, unless the operator explicitly requested the deployment be left running — a deployment failure is not that request, so the skill installs a trap (or equivalent) covering every deployment step from the first process start onward, not just the post-injection verification path.
- **Custom host-attrs resource name false-MISMATCH (known residual risk, not fully closed)**: the default resource set avoids every host-attrs-derived name (`type`/`model`/`cpuf`/`maxcpus`/`maxmem`/`maxswap`/`maxtmp`/`ndisks` — `worker.c`'s `N_HOST_ATTRS`) specifically to route around this; the System Flows section above documents the nested-`!`-escape fix and why it closes the controller-layer check for that default set. If an operator passes a CUSTOM `resource=value` argument naming a host-attrs resource, the controller-layer check will still report a false MISMATCH — not because propagation is wrong, but because that resource is reset to baseline the instant the worker session ends, and no code path today re-checks it before that reset for an arbitrary caller-supplied name. This is a known, documented gap (tasks.md task 6.2), not silently swallowed: a genuine fix needs either a C-level `xce_test` change (no REPL path from `worker` context to `client` context without an equivalent disconnect) or a bash coprocess restructuring to keep the worker connected across every layer's check — out of scope for the current implementation. The skill should warn (not silently mismatch) when a caller-supplied resource name matches a known host-attrs name.

### Monitoring
Not applicable — this is an on-demand, operator-invoked CLI tool, not a running service. Logs are written under the run's scratch directory, following `xce-deploy-ha`/`xce-test-ha`'s existing `logs/` convention, for post-run inspection.

## Testing Strategy

- **Unit Tests**: `resval_typeresolve.c`'s type-resolution fallback — 4 cases: a currently-known numeric name (e.g. `mem`) unchanged; a currently-known string name (`type`) unchanged; an unrecognized name defaults to numeric; an unrecognized name with `-t string` override resolves to string — directly compilable and testable in isolation since the extracted module has no networking, REPL, or `test.c`-global dependency (unlike `test.c` itself, which cannot be white-box included — see File Structure Plan). The skill's verdict-aggregation/comparison step (Requirement 6) is implemented as a self-contained function over (resource, injected value, per-layer observed value) records specifically so its negative path — a genuine per-layer mismatch correctly producing an overall FAIL naming the right layer — can be verified the same way, with synthetic records, without requiring a live deployment.
- **Integration Tests**: the skill's own live run against a real controller+scheduler+apiserver deployment *is* this feature's primary integration/e2e coverage, following this codebase's established precedent for `xce-test-ha`-class tooling (soak/verification tools in this project are validated live, not via `meson test()`). At minimum, one live run per implementation change verifying: (1) a previously-missing resource name (e.g. `r15m`) is now accepted and correctly injected, (2) all three layers' trace lines report the correct value within the bounded wait, (3) a genuinely-unregistered resource name included in the same batch is reported as a specific, clearly-identified rejection (not silently dropped) and correctly drives the run's overall verdict to FAIL, per this design's Error Handling.
- **E2E Path**: the full flow in System Flows' sequence diagram, run end-to-end at least once with the default resource set before this feature is considered done.