myxce/.kiro/specs/resource-value-verification/requirements.md

Requirements Document

Introduction

XCE currently has no fast, self-contained way to verify that a worker's resource value change propagates correctly through the full pipeline: controller → scheduler → API server. During a recent investigation (bug-1569), engineers had to deploy full worker daemons and wait on periodic reporting cycles to reproduce and verify resource-value routing bugs, when xce_test's existing worker context already offers a resupdate command capable of injecting synthetic resource values directly — but its resource-name whitelist is incomplete, and no tooling exists to confirm those values are correctly visible downstream at the scheduler or the Go API server. This feature closes the resupdate gap and adds an end-to-end resource-value verification capability so this class of bug can be reproduced and confirmed fixed in seconds instead of through multi-minute live-deployment investigations.

Boundary Context (Optional)

• In scope: extending xce_test's worker-context resupdate command to support any resource the cluster has registered (not just a fixed subset); a repeatable verification flow/tool that injects a resource value and confirms it is visible at the controller, scheduler, and API server layers; any deployment scripting needed to stand up a Go API server instance for this verification (none currently exists for local testing).
• Out of scope: changing production resource-collection behavior (resource_sync.c's real load-index collection); building CI/automated-pipeline integration; fixing unrelated bugs surfaced incidentally during verification (they should be reported, not silently fixed as part of this feature); broader xce_test tooling improvements (per-command help, reference documentation, structured output, batch mode, argument-convention unification, renaming) — deliberately deferred to a separate future project, see Future Considerations below.
• Adjacent expectations: this feature depends on a working controller/scheduler/API-server deployment mechanism (the existing xce-deploy-ha skill covers controller deployment only; API-server deployment is a new capability this feature must provide or explicitly hand off). How scheduler-side (and apiserver-side) visibility is confirmed is left to the design phase, not fixed here — design.md/research.md settle on live trace-log observation at each layer's point of application, a third option not enumerated at requirements time (neither "extend the REPL flow" nor "add a new live query RPC").

Requirements

Requirement 1: Complete resource-name coverage in resupdate

Objective: As a developer testing XCE's resource-value pipeline, I want xce_test's resupdate command to accept any resource name the cluster has registered, so that I can inject a synthetic value for any resource without being blocked by an incomplete hardcoded list.

Acceptance Criteria

1. When an operator runs resupdate -n <resource-name> -v <value> for a resource name currently missing from the command's supported list (including r15m, pg, ls, it, swp, io, tmp), the worker test context shall accept the update and report success.
2. When an operator runs resupdate -n <resource-name> -v <value> for a resource name that does not exist in the cluster's registered resource set, the worker test context shall report a clear error identifying the unrecognized name.
3. The worker test context shall report which resource names were accepted and which were rejected when a single resupdate invocation includes multiple -n/-v pairs.
4. Where a resource name is recognized, the worker test context shall preserve the resource's existing value-type behavior (numeric vs. string) exactly as before this change, for every currently-supported resource.

Requirement 2: Inject a resource value without a real worker daemon

Objective: As a developer reproducing a resource-value bug, I want to set a worker's resource value through xce_test's worker context alone, so that I don't need to deploy and wait on a real worker daemon's periodic reporting cycle.

Acceptance Criteria

1. When an operator joins a synthetic worker through xce_test's worker context and then runs resupdate, the system shall apply the update as if it came from a real, currently-connected worker daemon, with no other worker process required.
2. The system shall tag every resupdate-injected value with the joined worker's own identity, consistently with how a real worker daemon's periodic report is identified.

Requirement 3: Confirm the value is visible at the controller

Objective: As a developer verifying a resource-value fix, I want confirmation that an injected value is visible through the controller's own resource query, so that I know the controller-side portion of the pipeline is correct.

Acceptance Criteria

1. When a resource value has been injected via resupdate, the verification capability shall query the controller for that resource/worker pair and shall report whether the returned value matches the injected value.
2. If the controller's returned value does not match the injected value, the verification capability shall report this as a failure and shall include both the injected and observed values in its report.

Requirement 4: Confirm the value is visible at the scheduler

Objective: As a developer verifying a resource-value fix, I want confirmation that an injected value is visible in the scheduler's own live resource state, so that I know the scheduler-side portion of the pipeline is correct.

Acceptance Criteria

1. When a resource value has been injected via resupdate, the verification capability shall determine, within a bounded wait time, whether that value is visible in the scheduler's own resource state for the corresponding worker.
2. If the scheduler's resource state does not reflect the injected value within the bounded wait time, the verification capability shall report this as a failure and shall include both the injected and observed values in its report.
3. The verification capability shall report the scheduler-side observed value distinctly from the controller-side observed value, so a mismatch can be attributed to a specific layer.

Requirement 5: Confirm the value is visible at the API server

Objective: As a developer verifying a resource-value fix, I want confirmation that an injected value is correctly received and stored by the Go API server, so that I know the apiserver-side portion of the pipeline is correct.

Acceptance Criteria

1. When a resource value has been injected via resupdate and a Go API server instance is running against the same cluster, the verification capability shall determine, within a bounded wait time, whether that value is visible in the API server's own resource state for the corresponding worker.
2. If the API server's observed value does not match the injected value within the bounded wait time, the verification capability shall report this as a failure and shall include both the injected and observed values in its report.
3. Where no Go API server deployment mechanism currently exists for local testing, the verification capability shall provide one as part of standing up the test environment.
4. The verification capability shall report the apiserver-side observed value distinctly from the controller-side and scheduler-side observed values, so a mismatch can be attributed to a specific layer.

Design note: this requirement is intentionally worded like Requirement 4 (implementation-agnostic — "visible in the apiserver's own resource state," not "returned by a query") because the chosen mechanism observes the apiserver's storage layer directly (a trace log at the point of ingestion) rather than exercising its REST/CLI query path — see design.md/research.md for the rationale and the accepted trade-off (a query/serialization-layer-only regression would not be caught by this design).

Requirement 6: Repeatable, reportable verification run

Objective: As a developer diagnosing or regression-testing resource-value propagation, I want a single repeatable command that runs the full injection-and-verification flow and reports a clear pass/fail result, so that I can confirm a fix (or catch a regression) without manual multi-step investigation.

Acceptance Criteria

1. When an operator invokes the verification capability, the system shall stand up (or reuse) a controller + scheduler + API server test deployment, inject one or more resource values, and check all three layers (controller, scheduler, API server) without further manual steps.
2. When every checked layer's observed value matches the injected value, the verification capability shall report an overall PASS.
3. When any checked layer's observed value does not match the injected value (or a layer is unreachable), the verification capability shall report an overall FAIL and shall identify which layer(s) failed.
4. The verification capability shall support checking multiple resource names (including the ones named in Requirement 1) in a single run.
5. The system shall clean up any deployment or process state it created for the verification run, unless the operator explicitly requests it be left running for further inspection.

Future Considerations (explicitly out of scope for this project)

Raised while gathering requirements. These are real, worth doing, but the user has explicitly deferred all of them to a separate future "enhance xce_test" project — none are implemented as part of this spec:

• Renaming xce_test: as its role grows from an ad-hoc test client into a serious tool for verifying XCE controller/scheduler behavior and core architecture correctness, a more descriptive name could better communicate that purpose. Carries real blast radius (invoked by name in scripts, the xce-test-ha/xce-deploy-ha skills, and developer muscle memory) — needs its own explicit decision (candidate name, rename-vs-alias) when picked up.
• Discoverable per-command help: today's help only lists command names with no per-command usage/arguments/examples. A future project should add detailed help for every command, in every context (client/worker/scheduler).
• Reference documentation: a maintained written reference describing every xce_test context and command, its purpose, arguments, and an example invocation.
• Structured (JSON) output mode for query commands (resval, joblist, workerlist, etc.) so scripts/skills can parse results reliably instead of scraping human-readable tables — this session's own investigation relied entirely on fragile table-text grep.
• Non-interactive batch/script mode with clear per-command success/failure exit codes, for more robust use from skills and CI (this session drove xce_test via heredoc-piped stdin, which works but gives no clean per-command pass/fail signal).
• Consistent argument conventions across contexts (e.g. resupdate's -n/-v flags vs. jobsub's positional arguments) — a larger, more invasive cleanup with real backward-compat risk for existing scripts/skills.