myxce/.kiro/specs/resource-value-verification/research.md

Research & Design Decisions

Summary

• Feature: resource-value-verification
• Discovery Scope: Extension (existing systems: xce_test, controller, scheduler, apiserver)
• Key Findings:
  • xce_test's worker-context resupdate command already exists, already sends updates through the identical wire path a real worker daemon's periodic report uses (xcecli_update_resvals → xce_update_resvals → controller's do_update_resvals), and already tags .host correctly via get_setting("myname"). It is the right injection primitive for this feature — no new wire path is needed.
  • resupdate's resource-name coverage is gated by a hardcoded table (set_resval(), src/controller/testsuite/test.c:3267) that is missing 7 real dynamic resources (r15m, pg, ls, it, swp, io, tmp) present in resource_sync.c's load-index set.
  • The scheduler has no live (non-replay) query mechanism for its own resource state today. The only existing query path is xceadmin scheduler record start/stop + xce_scheduler -i -s <file> --interactive (the REPL). The REPL starts paused at the initial snapshot and requires an explicit step/continue before any subsequently-recorded event (including a resource update) is reflected in info worker output — discovered directly during this session's bug-1569 investigation, where querying immediately after load repeatedly and misleadingly looked like a bug. This path was the original design choice and is now superseded — see Design Decisions below: the scheduler applies the update to its own live state regardless of whether anything ever queries it, and a trace log at that point of application is both simpler and a closer match to how this feature's own originating bugs were actually found.
  • No apiserver deployment mechanism exists in this repo for local/skill-driven testing today. xce-deploy-ha only deploys the C controller. The Go apiserver is built via make build-server (src/frontend/Makefile) → /tmp/apiserver, and make test-integration already establishes a XCE_BIN/SRV_URL convention for pointing Go-side tooling at a running controller — the closest existing precedent for this feature's apiserver deploy step.
  • The apiserver already exposes a query this feature could use: xcecli resources distribution list → GET /xce/v1/resources/distribution, backed by Go's store.ResourceLocations. Superseded — see Design Decisions: the final design observes store.ResourceLocations's write directly via a trace log in OnResourceLocationUpdated, not through this REST/CLI path, so the apiserver deploy step no longer needs functional auth/RBAC.
  • On the controller, apiserver_resource_hook_fun (src/controller/apiserver.c) forwards the same raw wire resval payload to the apiserver that sched_resource_hook_fun forwards to the scheduler — both are fed by the same HOOK_DETAIL_RESOURCE_UPDATE hook do_update_resvals() fires. This confirms a resupdate injection is visible to all three layers through one single controller-side write, not three independent paths.
  • Final approach (see Design Decisions): both the scheduler and apiserver checks observe each layer's own point of application via a new one-line trace/debug log (core_events.c for the scheduler, handler.go's OnResourceLocationUpdated for the apiserver), greppable from the skill within a bounded wait — not the REPL and not a REST/CLI query. This directly matches how bug-1568 and bug-1569(A), the two real bugs that motivated this feature, were actually root-caused.

Research Log

How does a resource value actually reach the scheduler and apiserver today?

• Context: Needed to confirm resupdate is a faithful stand-in for a real worker's periodic report, and to identify what each verification layer should query.
• Sources Consulted: src/controller/testsuite/test.c (cmd_resupdate, set_resval), src/lib/client.c (xcecli_update_resvals), src/controller/opupd.c (do_update_resvals), src/controller/sched.c (sched_resource_hook_fun), src/controller/apiserver.c (apiserver_worker_change, apiserver_resource_hook_fun), src/scheduler/src/core_events.c (XCE_CORE_EVENT_RESVALS_DELTA handling) — all read directly (not summarized) during this session's bug-1569 investigation and the follow-up research for this spec.
• Findings: One controller-side apply (do_update_resvals → update_resval_list) plus one hook firing (HOOK_DETAIL_RESOURCE_UPDATE) fans out to both the scheduler (sched_resource_hook_fun → notify() → XCE_CORE_EVENT_RESVALS_DELTA) and the apiserver (apiserver_resource_hook_fun → notify() to each connected apiserver instance). The controller's own w->host.resvals (queried via resval in the client context) is updated synchronously in the same call, independent of either forward.
• Implications: All three verification layers (Requirements 3–5) are downstream of the exact same single injection call — the design does not need three separate injection paths, only three separate read/verify steps.

Does a live scheduler resource query already exist?

• Context: Requirements left "how to confirm scheduler-side visibility" as an open design decision (extend the replay/REPL path vs. add a new live admin query).
• Sources Consulted: src/scheduler/src/handler.c (sched_handle_admin_request, SchedAdminAction_*), src/scheduler/src/repl.c/repl_info.c (the --interactive REPL), xceadmin's scheduler record start/stop/scheduler dump commands.
• Findings: The only scheduler-admin RPC surface today (OPCode_schedAdmin) covers recording control (recordStart/recordStop/recordStatus) — nothing exposes live resource/worker state over the wire. The only way to inspect scheduler-side state is to record a .replay file and load it into the offline --interactive REPL, which requires explicit step/continue before it reflects anything beyond the initial snapshot.
• Implications: See Design Decision below — adopting the existing REPL path (documented pitfall and all) instead of building a new live-query RPC.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Trace log at point of application (scheduler + apiserver)	One debug/trace log line each in core_events.c's resval-delta handling and handler.go's OnResourceLocationUpdated; skill greps each live process's log	Matches how bug-1568/1569A were actually found; no record/replay round-trip; no REPL step/continue pitfall; no apiserver auth/RBAC functional requirement	Small, deliberate production-code touch (one log line each) in scheduler and apiserver; does not exercise the real query/REST/CLI path, so a query-layer-only regression wouldn't be caught	Selected
Record+replay+REPL for scheduler; deploy+REST-query for apiserver	Record a short window, replay it, step/continue, info worker; separately, deploy apiserver with working auth and query via xcecli	Exercises the real client-facing query path; zero scheduler/apiserver code changes	REPL step/continue pitfall already produced one false negative live; apiserver auth/RBAC config previously flagged as this design's least-verified part	Original choice; superseded — see Design Decisions
New live scheduler admin query RPC	Add a new SchedAdminAction (e.g. infoWorker) returning live resource state	Instant, no record/replay round-trip, exercises a real query path	New wire schema (capnp) + new scheduler handler + new xceadmin verb — real production-code surface for a test-tooling spec	Rejected; possible future follow-up, not built here

Design Decisions

Decision (SUPERSEDED — see below): Adopt the existing record+replay+REPL flow for scheduler-side verification

• Context: Requirement 4 needs to confirm scheduler-side visibility; requirements.md deliberately left the mechanism to design.
• Alternatives Considered:
  1. Add a new live scheduler admin query (e.g. SchedAdminAction_infoWorker) that returns current resource state directly, without recording.
  2. Reuse the existing xceadmin scheduler record start/stop + xce_scheduler -i -s <file> REPL flow, with explicit step/continue before querying.
• Selected Approach (at the time): (2) — reuse the existing REPL flow.
• Why superseded: re-examined during design validation against this spec's own originating bug history (bug-1568, bug-1569A) and replaced by the trace-log decision immediately below — see that entry for the full rationale. This entry is kept for the record of what was considered and rejected the first time; do not implement against it.

Decision: Verify scheduler- and apiserver-side visibility via live trace-log observation, not REPL replay or a REST/CLI query

• Context: Requirement 4 (scheduler) and Requirement 5 (apiserver) need a mechanism to confirm an injected value is visible at each layer; the record+replay+REPL approach above was the original design-phase choice, revisited after the user asked (during design validation) whether scheduler-sim-mode dependence could be avoided.
• Key finding that changed the decision: this feature's own originating bugs were re-examined for what kind of defect they actually were. Bug-1568 (.wolf/buglog.json) — "info worker shows 0.00 for every resource" — was root-caused as a genuine storage-layer bug: sched_add_worker() (src/scheduler/src/core.c:270) is a plain key_add() (ADD-ONLY; fails -EEXIST on a worker that already exists), so a second XCE_CORE_EVENT_WORKERS_PUSH carrying the real values silently loses to the first, zero-valued push — the scheduler's actual in-memory state was wrong, not merely mis-displayed. It was root-caused by adding live TRACE instrumentation directly at the point of application (core_events.c), not by querying through the REPL. Bug-1569(A) was likewise a storage-layer bug (resource_sync.c tagging periodic reports with the wrong .host key) — and the REPL path's step/continue requirement itself produced a false negative during that investigation (looked broken, wasn't). Neither of the two real bugs this feature exists to catch was a query/display-layer defect; both were "the stored value is wrong," and both were actually found via a trace log at the storage site, not via REPL/REST querying.
• Alternatives Considered:
  1. (original) Record+replay+REPL for scheduler; deploy apiserver + xcecli resources distribution list REST query for apiserver.
  2. Add a debug/trace log line at each layer's actual point of application — scheduler: xce_core_push_events()'s XCE_CORE_EVENT_RESVALS_DELTA case in src/scheduler/src/core_events.c (adjacent to sched_get_worker); apiserver: OnResourceLocationUpdated in src/frontend/apiserver/backend/handler.go:92-95 (where store.ResourceLocations.Put() is called). The skill greps each layer's own live log file for the expected line within a bounded wait, instead of recording/replaying (scheduler) or deploying-and-querying (apiserver).
• Selected Approach: (2), for both layers.
• Rationale: Directly matches how the actual defects in this bug class were found and is simpler on every axis — no record/replay round-trip, no REPL step/continue pitfall (already proven capable of a false negative once), no apiserver auth/RBAC config to get right (previously flagged as this design's least-verified, riskiest part), no dependency on the REST/JSON stack. The apiserver still needs to be deployed and connected to the controller (Requirement 5.3 — OnResourceLocationUpdated only fires on a live, connected instance), but no longer needs functional auth/RBAC — only enough config to boot and connect, which substantially de-risks deploy-apiserver.sh.
• Trade-offs (explicitly accepted by the user): this checks that the value was correctly received and stored at each layer, not that a real client can retrieve it through the actual query/REST/CLI path. A future bug in the query/serialization layer specifically (as opposed to the storage layer) would not be caught by this design. Requirement 5's acceptance criteria are reworded in requirements.md to stop mandating "query the API server" as the specific mechanism, so this is a requirements-level acceptance of the trade-off, not just a design-level shortcut.
• Follow-up: if a future bug turns out to be in the query/serialization path rather than storage, re-add a real query-based check for that layer specifically — this decision does not preclude it.

Decision: Default unrecognized resupdate resource names to numeric, keep a small explicit list for non-numeric ones

• Context: Requirement 1 needs resupdate to accept any registered resource name, not just the current hardcoded 14-entry table.
• Alternatives Considered:
  1. Query the controller/resmgr live for each resource's registered value type before sending the update.
  2. Extend the hardcoded table to enumerate every resource name across every possible cluster config.
  3. Default any unrecognized name to numeric (Value_u_fvalue), keep only the currently-known non-numeric names (type, model) in a small explicit list, and add an optional override flag for a caller that needs a non-numeric type.
• Selected Approach: (3).
• Rationale: (1) adds a network round-trip and new query dependency to a simple CLI command for a case (wrong type guess) that is rare and already recoverable via the override flag. (2) does not scale — resource sets are cluster-config-defined, not fixed, so any static enumeration will eventually be incomplete again (as demonstrated by the 7 already-missing names). (3) is the minimal change that satisfies "accept any resource name" without new runtime dependencies, is backward compatible (existing behavior for type/model and all currently-numeric names is unchanged), and gives an explicit escape hatch instead of guessing silently wrong.
• Trade-offs: A brand-new non-numeric resource requires the caller to pass the override flag once; this is judged acceptable given non-numeric resources are rare in this codebase (currently only 2 of ~30+ registered resources).
• Follow-up: None required before implementation.

Decision: Verify per-name accept/reject via a follow-up xcecli_resval() query, not the update RPC's own reply status

• Context: Requirement 1.2 (clear error for a genuinely unregistered resource name) and 1.3 (per-name accept/reject reporting for multi-pair invocations) need a real signal distinguishing per-item success from failure. The original design draft assumed the controller's synchronous RPC reply (do_update_resvals's reply.status) could carry this, flagged as unconfirmed in this file's Risks — that assumption was checked directly against source during a final spec review and found to be false.
• Confirmed finding: update_resval_list() (src/controller/worker.c:185-198) loops over the whole batch, logs each per-item failure via log_update_resval_failure(), but unconditionally return 0. do_update_resvals() (src/controller/opupd.c) sets reply.status from that single return value, so the RPC reply is always "success" for the whole batch regardless of how many individual names were rejected — there is no per-name signal on this path at all, and no batch-level failure signal either.
• Alternatives Considered:
  1. Modify update_resval_list()/do_update_resvals() to return/report per-item status on the wire.
  2. Have cmd_resupdate() verify each submitted name itself, via a follow-up xcecli_resval() query (the same client-library call resval/cmd_resval already uses) per (resource, worker) pair, comparing the returned value against what was just sent.
• Selected Approach: (2).
• Rationale: (1) requires modifying production controller code (worker.c, opupd.c, and the RPC reply wire shape) — explicitly out of this spec's boundary, and disproportionate scope for a test-tooling fix. (2) uses only existing, unmodified client-library functions, stays entirely inside test.c (this spec's boundary), and gives a genuinely observable per-name result (a live query confirming — or failing to confirm — the value actually landed), which is a stronger correctness signal than a wire-level status flag would have been anyway.
• Trade-offs: Adds one RPC round-trip per submitted resource name (acceptable for a test tool, not a production hot path). A resource that is genuinely registered but whose value hasn't propagated to the query path yet could theoretically be misreported as rejected — mitigated with a short bounded retry before concluding rejection.
• Follow-up: None required before implementation; this fully resolves what was previously an open risk (see below).

Decision: Extend xce-deploy-ha's deploy-ha.sh with an opt-in apiserver step, instead of a new standalone deploy-apiserver.sh

• Context: this feature needs an apiserver instance connected to the controller for local verification; the original design (see design.md history) planned a brand-new, standalone src/tools/deploy-apiserver.sh that separately re-implemented xce-deploy-ha's directory/scratch/health-check conventions "modeled on" the original rather than reusing it directly. Revisited during design validation after the user asked whether the deploy skill itself should be enhanced to support apiserver.
• Existing precedent found: deploy-ha.sh (Step 3, .claude/skills/xce-deploy-ha/SKILL.md) already has exactly this shape of extension for worker and scheduler — it generates start-worker.sh/start-scheduler.sh conditionally, only if the xcecd_virtual/xce_scheduler binaries exist alongside the controller binary at deploy time. Callers that never build worker/scheduler (e.g. xce-test-ha) get no such scripts and are completely unaffected. This is an established, low-risk, additive pattern for exactly this kind of "optional extra daemon" need.
• Alternatives Considered:
  1. (original) A new, standalone src/tools/deploy-apiserver.sh, independently modeling xce-deploy-ha's conventions.
  2. Extend deploy-ha.sh itself: generate start-apiserver.sh under the same "only if the binary exists" rule already used for worker/scheduler, and document it as Step 4d in xce-deploy-ha's own SKILL.md.
• Selected Approach: (2).
• Rationale: avoids re-implementing the same directory/cert/health-check conventions in parallel (a reuse/duplication smell the original design already half-acknowledged via "modeled on"/"mirrors" language instead of actual reuse); removes the Revalidation Trigger risk of the two scripts' conventions drifting apart over time, since there is now only one script; matches an already-precedented conditional-generation pattern (worker/scheduler), so this is idiomatic to the codebase rather than novel; and requirements.md's own "Adjacent expectations" note already floated "hand off" to xce-deploy-ha as a legitimate option at requirements time.
• Trade-offs: this feature's implementation now touches a shared, actively-used skill (xce-deploy-ha) instead of staying fully isolated in a new file — real blast radius, mitigated by making the apiserver step strictly opt-in/conditional (existing callers that don't build/need an apiserver see zero behavior change, exactly like the worker/scheduler precedent) and additive-only (no changes to existing controller/worker/scheduler steps).
• Follow-up: None required before implementation.
• Extension (raised by the user during the same design-validation pass): also generate a combined start-all.sh in Step 3, alongside the existing per-component scripts (start.sh, start-worker.sh, start-scheduler.sh, and the new start-apiserver.sh) — it starts the controller node(s) via start.sh, health-checks (Step 5's existing check), then starts whichever of worker/scheduler/apiserver scripts were actually generated for this deploy, in the same order and with the same backgrounding SKILL.md's Step 4b already documents doing by hand. This is not new sequencing logic — it automates the exact manual recipe xce-deploy-ha's own SKILL.md Steps 4/4b/4c already instruct a human to run — and it directly serves Requirement 6.1 ("stand up... without further manual steps") by letting the xce-test-resources skill call one generated script instead of re-implementing controller→scheduler/apiserver sequencing itself. Always generated (regardless of which optional components are present for a given deploy), so it has one consistent name and behavior across all callers.

Risks & Mitigations

• No apiserver deployment mechanism exists yet, and this session never exercised the Go apiserver live. Now that verification no longer depends on a functional REST query (trace-log decision above), the remaining unknown is narrower: just enough config for the binary to boot and register with the controller (no working auth/RBAC required). Mitigation: model the new xce-deploy-ha apiserver step on make test-integration's existing XCE_BIN/SRV_URL convention, reusing deploy-ha.sh's own directory/config-generation pattern directly (see the "extend xce-deploy-ha" decision above) rather than a separate script; confirm the minimal-boot config shape during implementation.
• The scheduler's TRACE/tlog_debug3 output and the apiserver's logger.Debug output are both gated by runtime log-level configuration (-vvv for the scheduler, logger.yaml's logging.level: debug for the apiserver) — if start-all.sh fails to set either, the corresponding verification check times out on every run with no indication the trace line was simply never written (looks identical to a genuine propagation failure). Mitigation: this is called out explicitly as a load-bearing requirement in design.md's two trace-log component sections and in task 3.1/4.1/4.2, not left implicit.
• A trace log line added to production code (core_events.c, handler.go) must be genuinely inert in normal operation — gated at debug/trace level (never WARN/ERROR, never on the hot path for every event), consistent with each component's existing logging conventions (tinylog in C, zap logger.Debug in Go per error-handling.md).
• The chosen approach verifies storage-layer correctness, not the real query/REST/CLI path — accepted trade-off (see Design Decisions); a future query-layer bug would need a separate check to catch.
• Live-deployment verification (this feature's core value) is inherently timing-sensitive (bounded waits per Requirements 4/6). Mitigation: use generous, explicit timeouts consistent with this codebase's existing soak-test conventions (xce-test-ha), not tight polling that could flake.
• test.c (4298 lines, its own main(), heavy libedit/network/REPL dependencies) cannot be white-box #included into a Unity test the way smaller single-purpose files in this codebase are — confirmed by direct inspection during final spec review. Mitigation: extract set_resval()'s type-resolution logic into a new, small, dependency-light resval_typeresolve.c/.h module that both test.c and a new unit test can compile against independently (see design.md File Structure Plan).

References

• .wolf/buglog.json bug-1569 — the investigation that surfaced every finding in this research log, including the REPL step/continue pitfall and the incomplete resupdate resource table.
• .claude/skills/xce-test-ha/SKILL.md, .claude/skills/xce-deploy-ha/SKILL.md — existing skill conventions this design follows (VERDICT line format, scratch-directory deploy pattern, setsid nohup … & disown process backgrounding).