myxce/.kiro/specs/resource-value-verification/research.md

# Research & Design Decisions

## Summary
- **Feature**: `resource-value-verification`
- **Discovery Scope**: Extension (existing systems: `xce_test`, controller, scheduler, apiserver)
- **Key Findings**:
  - `xce_test`'s worker-context `resupdate` command already exists, already sends updates through the identical wire path a real worker daemon's periodic report uses (`xcecli_update_resvals` → `xce_update_resvals` → controller's `do_update_resvals`), and already tags `.host` correctly via `get_setting("myname")`. It is the right injection primitive for this feature — no new wire path is needed.
  - `resupdate`'s resource-name coverage is gated by a hardcoded table (`set_resval()`, `src/controller/testsuite/test.c:3267`) that is missing 7 real dynamic resources (`r15m`, `pg`, `ls`, `it`, `swp`, `io`, `tmp`) present in `resource_sync.c`'s load-index set.
  - The scheduler has no live (non-replay) query mechanism for its own resource state today. The only existing *query* path is `xceadmin scheduler record start/stop` + `xce_scheduler -i -s <file> --interactive` (the REPL). The REPL starts **paused** at the initial snapshot and requires an explicit `step`/`continue` before any subsequently-recorded event (including a resource update) is reflected in `info worker` output — discovered directly during this session's bug-1569 investigation, where querying immediately after load repeatedly and misleadingly looked like a bug. **This path was the original design choice and is now superseded — see Design Decisions below**: the scheduler applies the update to its own live state regardless of whether anything ever queries it, and a trace log at that point of application is both simpler and a closer match to how this feature's own originating bugs were actually found.
  - No apiserver deployment mechanism exists in this repo for local/skill-driven testing today. `xce-deploy-ha` only deploys the C controller. The Go apiserver is built via `make build-server` (`src/frontend/Makefile`) → `/tmp/apiserver`, and `make test-integration` already establishes a `XCE_BIN`/`SRV_URL` convention for pointing Go-side tooling at a running controller — the closest existing precedent for this feature's apiserver deploy step.
  - The apiserver already exposes a query this feature could use: `xcecli resources distribution list` → `GET /xce/v1/resources/distribution`, backed by Go's `store.ResourceLocations`. **Superseded — see Design Decisions**: the final design observes `store.ResourceLocations`'s write directly via a trace log in `OnResourceLocationUpdated`, not through this REST/CLI path, so the apiserver deploy step no longer needs functional auth/RBAC.
  - On the controller, `apiserver_resource_hook_fun` (`src/controller/apiserver.c`) forwards the same raw wire resval payload to the apiserver that `sched_resource_hook_fun` forwards to the scheduler — both are fed by the same `HOOK_DETAIL_RESOURCE_UPDATE` hook `do_update_resvals()` fires. This confirms a `resupdate` injection is visible to all three layers through one single controller-side write, not three independent paths.
  - **Final approach (see Design Decisions)**: both the scheduler and apiserver checks observe each layer's own point of application via a new one-line trace/debug log (`core_events.c` for the scheduler, `handler.go`'s `OnResourceLocationUpdated` for the apiserver), greppable from the skill within a bounded wait — not the REPL and not a REST/CLI query. This directly matches how bug-1568 and bug-1569(A), the two real bugs that motivated this feature, were actually root-caused.

## Research Log

### How does a resource value actually reach the scheduler and apiserver today?
- **Context**: Needed to confirm `resupdate` is a faithful stand-in for a real worker's periodic report, and to identify what each verification layer should query.
- **Sources Consulted**: `src/controller/testsuite/test.c` (`cmd_resupdate`, `set_resval`), `src/lib/client.c` (`xcecli_update_resvals`), `src/controller/opupd.c` (`do_update_resvals`), `src/controller/sched.c` (`sched_resource_hook_fun`), `src/controller/apiserver.c` (`apiserver_worker_change`, `apiserver_resource_hook_fun`), `src/scheduler/src/core_events.c` (`XCE_CORE_EVENT_RESVALS_DELTA` handling) — all read directly (not summarized) during this session's bug-1569 investigation and the follow-up research for this spec.
- **Findings**: One controller-side apply (`do_update_resvals` → `update_resval_list`) plus one hook firing (`HOOK_DETAIL_RESOURCE_UPDATE`) fans out to both the scheduler (`sched_resource_hook_fun` → `notify()` → `XCE_CORE_EVENT_RESVALS_DELTA`) and the apiserver (`apiserver_resource_hook_fun` → `notify()` to each connected apiserver instance). The controller's own `w->host.resvals` (queried via `resval` in the client context) is updated synchronously in the same call, independent of either forward.
- **Implications**: All three verification layers (Requirements 3–5) are downstream of the exact same single injection call — the design does not need three separate injection paths, only three separate read/verify steps.

### Does a live scheduler resource query already exist?
- **Context**: Requirements left "how to confirm scheduler-side visibility" as an open design decision (extend the replay/REPL path vs. add a new live admin query).
- **Sources Consulted**: `src/scheduler/src/handler.c` (`sched_handle_admin_request`, `SchedAdminAction_*`), `src/scheduler/src/repl.c`/`repl_info.c` (the `--interactive` REPL), `xceadmin`'s `scheduler record start/stop`/`scheduler dump` commands.
- **Findings**: The only scheduler-admin RPC surface today (`OPCode_schedAdmin`) covers recording control (`recordStart`/`recordStop`/`recordStatus`) — nothing exposes live resource/worker state over the wire. The only way to inspect scheduler-side state is to record a `.replay` file and load it into the offline `--interactive` REPL, which requires explicit `step`/`continue` before it reflects anything beyond the initial snapshot.
- **Implications**: See Design Decision below — adopting the existing REPL path (documented pitfall and all) instead of building a new live-query RPC.

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Trace log at point of application (scheduler + apiserver) | One debug/trace log line each in `core_events.c`'s resval-delta handling and `handler.go`'s `OnResourceLocationUpdated`; skill greps each live process's log | Matches how bug-1568/1569A were actually found; no record/replay round-trip; no REPL step/continue pitfall; no apiserver auth/RBAC functional requirement | Small, deliberate production-code touch (one log line each) in scheduler and apiserver; does not exercise the real query/REST/CLI path, so a query-layer-only regression wouldn't be caught | **Selected** |
| Record+replay+REPL for scheduler; deploy+REST-query for apiserver | Record a short window, replay it, `step`/`continue`, `info worker`; separately, deploy apiserver with working auth and query via `xcecli` | Exercises the real client-facing query path; zero scheduler/apiserver code changes | REPL step/continue pitfall already produced one false negative live; apiserver auth/RBAC config previously flagged as this design's least-verified part | Original choice; superseded — see Design Decisions |
| New live scheduler admin query RPC | Add a new `SchedAdminAction` (e.g. `infoWorker`) returning live resource state | Instant, no record/replay round-trip, exercises a real query path | New wire schema (capnp) + new scheduler handler + new `xceadmin` verb — real production-code surface for a test-tooling spec | Rejected; possible future follow-up, not built here |

## Design Decisions

### Decision (SUPERSEDED — see below): Adopt the existing record+replay+REPL flow for scheduler-side verification
- **Context**: Requirement 4 needs to confirm scheduler-side visibility; requirements.md deliberately left the mechanism to design.
- **Alternatives Considered**:
  1. Add a new live scheduler admin query (e.g. `SchedAdminAction_infoWorker`) that returns current resource state directly, without recording.
  2. Reuse the existing `xceadmin scheduler record start/stop` + `xce_scheduler -i -s <file>` REPL flow, with explicit `step`/`continue` before querying.
- **Selected Approach (at the time)**: (2) — reuse the existing REPL flow.
- **Why superseded**: re-examined during design validation against this spec's own originating bug history (bug-1568, bug-1569A) and replaced by the trace-log decision immediately below — see that entry for the full rationale. This entry is kept for the record of what was considered and rejected the first time; do not implement against it.

### Decision: Verify scheduler- and apiserver-side visibility via live trace-log observation, not REPL replay or a REST/CLI query
- **Context**: Requirement 4 (scheduler) and Requirement 5 (apiserver) need a mechanism to confirm an injected value is visible at each layer; the record+replay+REPL approach above was the original design-phase choice, revisited after the user asked (during design validation) whether scheduler-sim-mode dependence could be avoided.
- **Key finding that changed the decision**: this feature's own originating bugs were re-examined for what kind of defect they actually were. **Bug-1568** (`.wolf/buglog.json`) — "`info worker` shows 0.00 for every resource" — was root-caused as a genuine *storage*-layer bug: `sched_add_worker()` (`src/scheduler/src/core.c:270`) is a plain `key_add()` (ADD-ONLY; fails `-EEXIST` on a worker that already exists), so a second `XCE_CORE_EVENT_WORKERS_PUSH` carrying the real values silently loses to the first, zero-valued push — the scheduler's actual in-memory state was wrong, not merely mis-displayed. It was root-caused by adding **live TRACE instrumentation directly at the point of application** (`core_events.c`), not by querying through the REPL. **Bug-1569(A)** was likewise a storage-layer bug (`resource_sync.c` tagging periodic reports with the wrong `.host` key) — and the REPL path's step/continue requirement itself produced a *false negative* during that investigation (looked broken, wasn't). Neither of the two real bugs this feature exists to catch was a query/display-layer defect; both were "the stored value is wrong," and both were actually found via a trace log at the storage site, not via REPL/REST querying.
- **Alternatives Considered**:
  1. (original) Record+replay+REPL for scheduler; deploy apiserver + `xcecli resources distribution list` REST query for apiserver.
  2. Add a debug/trace log line at each layer's actual point of application — scheduler: `xce_core_push_events()`'s `XCE_CORE_EVENT_RESVALS_DELTA` case in `src/scheduler/src/core_events.c` (adjacent to `sched_get_worker`); apiserver: `OnResourceLocationUpdated` in `src/frontend/apiserver/backend/handler.go:92-95` (where `store.ResourceLocations.Put()` is called). The skill greps each layer's own live log file for the expected line within a bounded wait, instead of recording/replaying (scheduler) or deploying-and-querying (apiserver).
- **Selected Approach**: (2), for both layers.
- **Rationale**: Directly matches how the actual defects in this bug class were found and is simpler on every axis — no `record`/`replay` round-trip, no REPL `step`/`continue` pitfall (already proven capable of a false negative once), no apiserver auth/RBAC config to get right (previously flagged as this design's least-verified, riskiest part), no dependency on the REST/JSON stack. The apiserver still needs to be deployed and connected to the controller (Requirement 5.3 — `OnResourceLocationUpdated` only fires on a live, connected instance), but no longer needs *functional* auth/RBAC — only enough config to boot and connect, which substantially de-risks `deploy-apiserver.sh`.
- **Trade-offs (explicitly accepted by the user)**: this checks that the value was correctly received and stored at each layer, not that a real client can retrieve it through the actual query/REST/CLI path. A future bug in the query/serialization layer specifically (as opposed to the storage layer) would not be caught by this design. Requirement 5's acceptance criteria are reworded in requirements.md to stop mandating "query the API server" as the specific mechanism, so this is a requirements-level acceptance of the trade-off, not just a design-level shortcut.
- **Follow-up**: if a future bug turns out to be in the query/serialization path rather than storage, re-add a real query-based check for that layer specifically — this decision does not preclude it.

### Decision: Default unrecognized `resupdate` resource names to numeric, keep a small explicit list for non-numeric ones
- **Context**: Requirement 1 needs `resupdate` to accept any registered resource name, not just the current hardcoded 14-entry table.
- **Alternatives Considered**:
  1. Query the controller/resmgr live for each resource's registered value type before sending the update.
  2. Extend the hardcoded table to enumerate every resource name across every possible cluster config.
  3. Default any unrecognized name to numeric (`Value_u_fvalue`), keep only the currently-known non-numeric names (`type`, `model`) in a small explicit list, and add an optional override flag for a caller that needs a non-numeric type.
- **Selected Approach**: (3).
- **Rationale**: (1) adds a network round-trip and new query dependency to a simple CLI command for a case (wrong type guess) that is rare and already recoverable via the override flag. (2) does not scale — resource sets are cluster-config-defined, not fixed, so any static enumeration will eventually be incomplete again (as demonstrated by the 7 already-missing names). (3) is the minimal change that satisfies "accept any resource name" without new runtime dependencies, is backward compatible (existing behavior for `type`/`model` and all currently-numeric names is unchanged), and gives an explicit escape hatch instead of guessing silently wrong.
- **Trade-offs**: A brand-new non-numeric resource requires the caller to pass the override flag once; this is judged acceptable given non-numeric resources are rare in this codebase (currently only 2 of ~30+ registered resources).
- **Follow-up**: None required before implementation.

### Decision: Verify per-name accept/reject via a follow-up `xcecli_resval()` query, not the update RPC's own reply status
- **Context**: Requirement 1.2 (clear error for a genuinely unregistered resource name) and 1.3 (per-name accept/reject reporting for multi-pair invocations) need a real signal distinguishing per-item success from failure. The original design draft assumed the controller's synchronous RPC reply (`do_update_resvals`'s `reply.status`) could carry this, flagged as unconfirmed in this file's Risks — that assumption was checked directly against source during a final spec review and found to be false.
- **Confirmed finding**: `update_resval_list()` (`src/controller/worker.c:185-198`) loops over the whole batch, logs each per-item failure via `log_update_resval_failure()`, but unconditionally `return 0`. `do_update_resvals()` (`src/controller/opupd.c`) sets `reply.status` from that single return value, so the RPC reply is always "success" for the whole batch regardless of how many individual names were rejected — there is no per-name signal on this path at all, and no batch-level failure signal either.
- **Alternatives Considered**:
  1. Modify `update_resval_list()`/`do_update_resvals()` to return/report per-item status on the wire.
  2. Have `cmd_resupdate()` verify each submitted name itself, via a follow-up `xcecli_resval()` query (the same client-library call `resval`/`cmd_resval` already uses) per `(resource, worker)` pair, comparing the returned value against what was just sent.
- **Selected Approach**: (2).
- **Rationale**: (1) requires modifying production controller code (`worker.c`, `opupd.c`, and the RPC reply wire shape) — explicitly out of this spec's boundary, and disproportionate scope for a test-tooling fix. (2) uses only existing, unmodified client-library functions, stays entirely inside `test.c` (this spec's boundary), and gives a genuinely observable per-name result (a live query confirming — or failing to confirm — the value actually landed), which is a stronger correctness signal than a wire-level status flag would have been anyway.
- **Trade-offs**: Adds one RPC round-trip per submitted resource name (acceptable for a test tool, not a production hot path). A resource that is genuinely registered but whose value hasn't propagated to the query path yet could theoretically be misreported as rejected — mitigated with a short bounded retry before concluding rejection.
- **Follow-up**: None required before implementation; this fully resolves what was previously an open risk (see below).

### Decision: Extend `xce-deploy-ha`'s `deploy-ha.sh` with an opt-in apiserver step, instead of a new standalone `deploy-apiserver.sh`
- **Context**: this feature needs an apiserver instance connected to the controller for local verification; the original design (see design.md history) planned a brand-new, standalone `src/tools/deploy-apiserver.sh` that separately re-implemented `xce-deploy-ha`'s directory/scratch/health-check conventions "modeled on" the original rather than reusing it directly. Revisited during design validation after the user asked whether the deploy skill itself should be enhanced to support apiserver.
- **Existing precedent found**: `deploy-ha.sh` (Step 3, `.claude/skills/xce-deploy-ha/SKILL.md`) already has exactly this shape of extension for worker and scheduler — it generates `start-worker.sh`/`start-scheduler.sh` **conditionally**, only if the `xcecd_virtual`/`xce_scheduler` binaries exist alongside the controller binary at deploy time. Callers that never build worker/scheduler (e.g. `xce-test-ha`) get no such scripts and are completely unaffected. This is an established, low-risk, additive pattern for exactly this kind of "optional extra daemon" need.
- **Alternatives Considered**:
  1. (original) A new, standalone `src/tools/deploy-apiserver.sh`, independently modeling `xce-deploy-ha`'s conventions.
  2. Extend `deploy-ha.sh` itself: generate `start-apiserver.sh` under the same "only if the binary exists" rule already used for worker/scheduler, and document it as Step 4d in `xce-deploy-ha`'s own `SKILL.md`.
- **Selected Approach**: (2).
- **Rationale**: avoids re-implementing the same directory/cert/health-check conventions in parallel (a reuse/duplication smell the original design already half-acknowledged via "modeled on"/"mirrors" language instead of actual reuse); removes the Revalidation Trigger risk of the two scripts' conventions drifting apart over time, since there is now only one script; matches an already-precedented conditional-generation pattern (worker/scheduler), so this is idiomatic to the codebase rather than novel; and requirements.md's own "Adjacent expectations" note already floated "hand off" to `xce-deploy-ha` as a legitimate option at requirements time.
- **Trade-offs**: this feature's implementation now touches a shared, actively-used skill (`xce-deploy-ha`) instead of staying fully isolated in a new file — real blast radius, mitigated by making the apiserver step strictly opt-in/conditional (existing callers that don't build/need an apiserver see zero behavior change, exactly like the worker/scheduler precedent) and additive-only (no changes to existing controller/worker/scheduler steps).
- **Follow-up**: None required before implementation.
- **Extension (raised by the user during the same design-validation pass)**: also generate a combined `start-all.sh` in Step 3, alongside the existing per-component scripts (`start.sh`, `start-worker.sh`, `start-scheduler.sh`, and the new `start-apiserver.sh`) — it starts the controller node(s) via `start.sh`, health-checks (Step 5's existing check), then starts whichever of worker/scheduler/apiserver scripts were actually generated for this deploy, in the same order and with the same backgrounding SKILL.md's Step 4b already documents doing by hand. This is not new sequencing logic — it automates the exact manual recipe `xce-deploy-ha`'s own SKILL.md Steps 4/4b/4c already instruct a human to run — and it directly serves Requirement 6.1 ("stand up... without further manual steps") by letting the `xce-test-resources` skill call one generated script instead of re-implementing controller→scheduler/apiserver sequencing itself. Always generated (regardless of which optional components are present for a given deploy), so it has one consistent name and behavior across all callers.

## Risks & Mitigations
- No apiserver deployment mechanism exists yet, and this session never exercised the Go apiserver live. Now that verification no longer depends on a functional REST query (trace-log decision above), the remaining unknown is narrower: just enough config for the binary to boot and register with the controller (no working auth/RBAC required). Mitigation: model the new `xce-deploy-ha` apiserver step on `make test-integration`'s existing `XCE_BIN`/`SRV_URL` convention, reusing `deploy-ha.sh`'s own directory/config-generation pattern directly (see the "extend xce-deploy-ha" decision above) rather than a separate script; confirm the minimal-boot config shape during implementation.
- The scheduler's `TRACE`/`tlog_debug3` output and the apiserver's `logger.Debug` output are both gated by runtime log-level configuration (`-vvv` for the scheduler, `logger.yaml`'s `logging.level: debug` for the apiserver) — if `start-all.sh` fails to set either, the corresponding verification check times out on every run with no indication the trace line was simply never written (looks identical to a genuine propagation failure). Mitigation: this is called out explicitly as a load-bearing requirement in design.md's two trace-log component sections and in task 3.1/4.1/4.2, not left implicit.
- A trace log line added to production code (`core_events.c`, `handler.go`) must be genuinely inert in normal operation — gated at debug/trace level (never WARN/ERROR, never on the hot path for every event), consistent with each component's existing logging conventions (`tinylog` in C, zap `logger.Debug` in Go per `error-handling.md`).
- The chosen approach verifies storage-layer correctness, not the real query/REST/CLI path — accepted trade-off (see Design Decisions); a future query-layer bug would need a separate check to catch.
- Live-deployment verification (this feature's core value) is inherently timing-sensitive (bounded waits per Requirements 4/6). Mitigation: use generous, explicit timeouts consistent with this codebase's existing soak-test conventions (`xce-test-ha`), not tight polling that could flake.
- `test.c` (4298 lines, its own `main()`, heavy `libedit`/network/REPL dependencies) cannot be white-box `#include`d into a Unity test the way smaller single-purpose files in this codebase are — confirmed by direct inspection during final spec review. Mitigation: extract `set_resval()`'s type-resolution logic into a new, small, dependency-light `resval_typeresolve.c`/`.h` module that both `test.c` and a new unit test can compile against independently (see design.md File Structure Plan).

## References
- `.wolf/buglog.json` bug-1569 — the investigation that surfaced every finding in this research log, including the REPL step/continue pitfall and the incomplete `resupdate` resource table.
- `.claude/skills/xce-test-ha/SKILL.md`, `.claude/skills/xce-deploy-ha/SKILL.md` — existing skill conventions this design follows (VERDICT line format, scratch-directory deploy pattern, `setsid nohup … & disown` process backgrounding).