myxce/.kiro/specs/controller-peers-env-fallback/design.md

Design Document

Overview

Purpose: This feature lets an operator set one environment variable, XCE_CONTROLLER_PEERS, once per session or deployment, instead of retyping --peers (controller, scheduler, worker, harness/xce_test) or --controller (xceadmin) on every invocation. It changes nothing about what any of these components do once their connection target is resolved — only where that target's default value comes from when the operator did not name one explicitly.

Users: Operators running local/interactive HPC cluster tools, and deploy/test scripts (e.g. HA soak setups) that currently either hardcode --peers/--controller into every generated command or accept the single-node/no-controller default they didn't actually want.

Impact: Five independently-built binaries each gain one additional, lowest-priority source for their existing connection-target resolution. No existing explicit-flag behavior, no existing "omitted, no env var" behavior, and no existing multi-candidate failover behavior changes.

Goals

• One new environment variable, consulted only when the corresponding CLI option is not given, across controller, scheduler, worker, harness/xce_test (via --peers) and xceadmin (via --controller).
• Zero behavior change when the CLI option is given, and zero behavior change when neither the option nor the variable is set.
• Reuse xceadmin's existing multi-candidate failover (resolve_controllers/follow_redirects_any, bug-1821) and its existing XCEADMIN_CONTROLLER fallback unmodified.

Non-Goals

• apiserver (YAML-config-based) and xcecli's --url (a different target) — untouched.
• Any change to src/tools/deploy-ha.sh's generated scripts' own behavior.
• Any change to what "no --peers/--controller and no env var" means today for any component.
• A TSAN/new-test-infrastructure investment for main()-level argument parsing where none exists today (see Testing Strategy).

Boundary Commitments

This Spec Owns

• A single shared C primitive (xce_controller_peers_env(), libxce) that reads XCE_CONTROLLER_PEERS, treating an unset or empty value as absent.
• The one-line integration of that primitive into controller, scheduler, and worker's existing --peers-gating if, and into harness's settings-table default computation at startup.
• Extending xceadmin's existing resolve_controllers() to strip an optional name= prefix per candidate entry.
• Adding "XCE_CONTROLLER_PEERS" ahead of the existing "XCEADMIN_CONTROLLER" in each --controller option's envvar= list.
• Help-text and documentation updates naming the new variable (CLI --peers/--controller help text, build-deployment.md, xce-deploy-ha/xce-test-multihost skill docs).
• A repeatable regression script (src/tools/verify-controller-peers-env.sh) proving the above across all 5 components, intended for ongoing re-use, not a one-time implementation check.

Out of Boundary

• apiserver's YAML-based peer configuration.
• xcecli's --url option.
• deploy-ha.sh's generated scripts' own invocations (documentation may mention the variable; the generated scripts keep passing --peers/--controller explicitly).
• Any change to xce_parse_peers()'s or resolve_controllers()/follow_redirects_any()'s existing parsing/failover semantics beyond the one name=-stripping addition.
• Refactoring any of the four C main() functions to become unit-testable in general — only the new, genuinely independent primitive is unit-tested (see Testing Strategy).

Allowed Dependencies

• All four C targets already link libxce (verified per-component meson.build) — the shared primitive may live there with no new build-graph edge.
• xceadmin's existing resolve_controllers()/follow_redirects_any() (discovery.py) and its existing --controller click options (cli.py) — reused and extended in place, not duplicated.

Project Invariant Check

This feature's controller/scheduler changes touch only pre-startup argument resolution (main.c, before sched_init()/myapp_run()/any daemon thread, event loop, or Raft/HA state machine is created) — it does not intersect xce-audit-invariants' controller dimensions (async I/O thread-role placement, fail-loud/never-self-exit process lifecycle, consensus/ replication safety) or its scheduler dimension (driver/simulation state-equivalence). No audit re-check is required beyond this design.

Revalidation Triggers

• Any change to xce_parse_peers()'s accepted value format (e.g. a new candidate syntax) must be mirrored in resolve_controllers()'s parsing if the two are to stay consistent for XCE_CONTROLLER_PEERS values shared between C daemons and xceadmin.
• Any future change to harness's settings[] initialization order (e.g. rc-file loading moved earlier than today) must preserve "the env-var-derived default is computed before any explicit set masters can run."
• If deploy-ha.sh is ever updated to adopt XCE_CONTROLLER_PEERS (explicitly deferred here), its generated scripts' current behavior baseline (documented in this design's Out of Boundary) is the thing to re-check against.

Architecture

Existing Architecture Analysis

Each of the five components already resolves its own connection target independently, with no shared code path today:

Component	Mechanism	Where "omitted" is decided	Existing fallback today
controller	argp, OPT_PEERS → arguments.peers_str	if (arguments.peers_str) (main.c ~L455)	Single-node primary (bug-1457)
scheduler	argp, 'P'/--peers → arguments.peers	if (arguments.peers) (main.c ~L403); --sim returns before this point is reached	"single-controller path" (logged)
worker	getopt_long, OPT_PEERS → peers_str	if (peers_str) (main.c ~L470)	Single-controller path via config.remote_host/config.remote_port (logged; same shape as controller/scheduler's own single-controller fallback, not an error path)
harness/xce_test	settings[] table, key "masters", no CLI flag at all	get_setting()'s value ?: defval (test.c ~L991)	defval = "localhost:9990"
xceadmin	click --controller option, envvar="XCEADMIN_CONTROLLER"	click's own parameter resolution	_DEFAULT_CONTROLLER = "localhost:9991"

All three C daemons that gate on a --peers argument already treat an invalid value as a hard startup error (xce_parse_peers() != 0 → message + return 1), never a silent fallback — this feature preserves that for both an invalid explicit flag and an invalid XCE_CONTROLLER_PEERS value.

Architecture Integration

• Selected pattern: one shared, stateless C primitive consulted independently by four call sites, plus one Python parsing extension consulted by click's own (unmodified) resolution order — no new service, process, or inter-component call is introduced.
• Domain/feature boundaries: each component keeps deciding its own resolved value; the new variable is only ever a source, never a coordinator.
• Existing patterns preserved: xce_parse_peers()'s name=-prefix convention; xceadmin's resolve_controllers/follow_redirects_any failover; each component's existing hard-error-on-invalid-value behavior.
• New components rationale: exactly one new C function, because harness's settings-table shape cannot reuse a "given cli_value, return resolved" signature the other three could share (see research.md, Design Decisions).
• Steering compliance: no new build dependency; C standard/build system (Meson) and Python (click) usage unchanged.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
C CLI / libxce	gnu17, existing libxce static lib	New xce_controller_peers_env() primitive; consumed at 4 existing call sites	No new dependency
Python CLI	click (already a xceadmin dependency)	Native multi-envvar list resolution (no new click usage pattern)	Confirmed via direct read of the installed click version's resolve_envvar_value()

File Structure Plan

One new file (the regression script below); every other change is to an existing file — no new directories.

New Files

• src/tools/verify-controller-peers-env.sh — repeatable regression script (see Testing Strategy → Scripted Regression Verification) driving the real built binaries of all 4 C components plus harness/xce_test, asserting on startup log/settings-dump output only (no live cluster required).

Modified Files

• src/include/xce/cluster.h — declare const char *xce_controller_peers_env(void); alongside the existing xce_parse_peers() declaration.
• src/lib/cluster.c — implement xce_controller_peers_env(): read getenv("XCE_CONTROLLER_PEERS"), return it if non-empty, else NULL.
• src/lib/testsuite/test_cluster.c — new unit tests for xce_controller_peers_env() (unset, empty, non-empty).
• src/controller/main.c — at the existing if (arguments.peers_str) site (~L455), resolve arguments.peers_str against xce_controller_peers_env() before the check; on a parse failure, name the source (flag vs. environment variable) in the existing error message; add XCE_CONTROLLER_PEERS to the --peers option's help text (~L85-87).
• src/scheduler/src/main.c — identical shape at the existing if (arguments.peers) site (~L403); help text update (~L84). No change anywhere in the --sim/--interactive path, which already returns before this point.
• src/worker/src/main.c — identical shape at the existing if (peers_str) site (~L470); update the -h/usage printf (~L443-446) to mention XCE_CONTROLLER_PEERS.
• src/harness/test.c — at startup, before the REPL loop and before rc-file loading, compute the "masters" entry's defval from xce_controller_peers_env() if set, else leave it at today's literal "localhost:9990"; extend the USAGE/set-command help text (~L409-421) to mention the new variable.
• src/xceadmin/src/xceadmin/discovery.py — extend resolve_controllers()'s per-entry parsing to strip an optional name= prefix (mirrors xce_parse_peers()) before its existing rsplit(":", 1).
• src/xceadmin/src/xceadmin/cli.py — change each of the 4 --controller options' envvar="XCEADMIN_CONTROLLER" to envvar=["XCE_CONTROLLER_PEERS", "XCEADMIN_CONTROLLER"]; update each option's help string.
• src/xceadmin/tests/test_discovery.py — new unit tests for resolve_controllers()'s name=-stripping.
• src/xceadmin/tests/test_cli.py — new unit tests for the new env-var precedence (mirrors the existing test_controller_option_falls_back_to_xceadmin_controller_env_var/test_controller_flag_overrides_xceadmin_controller_env_var tests).
• .kiro/steering/build-deployment.md — document XCE_CONTROLLER_PEERS: components, value format, precedence.
• .claude/skills/xce-deploy-ha/SKILL.md — informational note only (no generated-script behavior change).
• .claude/skills/xce-test-multihost/SKILL.md — same note, at its three existing --peers-mentioning locations (single-node deploy step, xcecd_virtual flag-table note, Prerequisites' generated-script list).

System Flows

flowchart TD
    Start[Resolution point reached] --> HasFlag{Explicit flag or\nset command given?}
    HasFlag -- yes --> UseFlag[Use that value\nnever consult env var]
    HasFlag -- no --> HasNewEnv{XCE_CONTROLLER_PEERS\nnon-empty?}
    HasNewEnv -- yes --> UseNewEnv[Use it\nxceadmin strips name prefix]
    HasNewEnv -- no --> IsAdmin{Component is xceadmin?}
    IsAdmin -- yes --> HasLegacyEnv{XCEADMIN_CONTROLLER\nnon-empty?}
    IsAdmin -- no --> UseTodayDefault[Today's unchanged default\nstandalone / built-in literal]
    HasLegacyEnv -- yes --> UseLegacyEnv[Use it, unchanged behavior]
    HasLegacyEnv -- no --> UseTodayDefault

Both branches that resolve to a real candidate list (UseFlag, UseNewEnv, UseLegacyEnv) still pass through each component's existing, unchanged validation (xce_parse_peers()/resolve_controllers()); an invalid value is a hard error either way, distinguished by source only in the error message text (1.5).

This resolution point is reached exactly once, at process startup, for controller/scheduler/worker/xceadmin. For harness/xce_test it is reached on every connection attempt (enter_ctx()), since set masters ... can run at any point in an interactive/scripted session — the diagram represents one such evaluation, not a startup-only decision, for that component.

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1	Recognize XCE_CONTROLLER_PEERS	controller/scheduler/worker/harness main.c/test.c	xce_controller_peers_env()	Flowchart: HasNewEnv
1.2	Explicit --peers wins, env var not consulted	controller/scheduler/worker main.c	existing if (peers) gate	Flowchart: HasFlag = yes
1.3	Env var used when flag omitted	controller/scheduler/worker main.c, harness test.c	xce_controller_peers_env()	Flowchart: UseNewEnv
1.4	Unset/empty env var ⇒ unchanged today's behavior	all 4	same call sites	Flowchart: UseTodayDefault
1.5	Parse-error message names the source	controller/scheduler/worker main.c, harness test.c	existing error fprintf/enter_ctx message	n/a
1.6	--peers help text documents the variable	controller/scheduler/worker main.c, harness test.c	argp/getopt/settings help strings	n/a
2.1	Derive controller list from env var	xceadmin cli.py, discovery.py	click envvar= list	Flowchart: UseNewEnv
2.2	Strip name= prefix	xceadmin discovery.py	resolve_controllers()	n/a
2.3	Try each candidate in order	xceadmin discovery.py	resolve_controllers/follow_redirects_any (unmodified)	n/a
2.4	Explicit --controller wins	xceadmin cli.py	click parameter resolution (unmodified)	Flowchart: HasFlag = yes
2.5	Legacy XCEADMIN_CONTROLLER fallback unchanged	xceadmin cli.py	click envvar= list order	Flowchart: HasLegacyEnv
2.6	Both unset ⇒ hardcoded default unchanged	xceadmin cli.py	_DEFAULT_CONTROLLER (unmodified)	Flowchart: UseTodayDefault
2.7	--controller help text documents the variable	xceadmin cli.py	click option help strings	n/a
3.1	Steering doc describes the variable	build-deployment.md	n/a	n/a
3.2	HA deploy guidance notes availability	xce-deploy-ha/SKILL.md	n/a	n/a
3.3	Multi-host guidance notes availability	xce-test-multihost/SKILL.md	n/a	n/a

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies	Contracts
xce_controller_peers_env()	libxce / C	Read XCE_CONTROLLER_PEERS, empty-as-unset	1.1, 1.3, 1.4	none (P0)	Service
controller/scheduler/worker --peers call sites	C daemon CLI	Combine flag + env var, preserve existing validation/help	1.2-1.6	xce_controller_peers_env (P0)	—
harness masters setting default	C CLI (settings table)	Compute defval from env var at startup	1.1, 1.3, 1.4, 1.5, 1.6	xce_controller_peers_env (P0)	State
resolve_controllers() extension	xceadmin / Python	Strip name= prefix per entry	2.1, 2.2	none (P0)	Service
xceadmin --controller options	xceadmin / Python (click)	Add XCE_CONTROLLER_PEERS ahead of legacy envvar	2.1, 2.4-2.7	resolve_controllers (P0), click (P0, external)	—
Documentation updates	Docs	Describe/announce the variable	3.1-3.3	none	—

Only the two components introducing genuinely new logic — xce_controller_peers_env() and the resolve_controllers() extension — warrant a full contract block; the remaining rows are thin call-site wiring around existing, unchanged behavior.

libxce

xce_controller_peers_env

Field	Detail
Intent	Sole read point for XCE_CONTROLLER_PEERS, normalizing empty-string to absent
Requirements	1.1, 1.3, 1.4

Responsibilities & Constraints

• Pure function: no side effects, no caching (the environment can only change between process invocations for these components, so no staleness concern).
• Treats an empty string identically to an unset variable, matching the same convention resolve_controllers()'s per-entry blank-skip already uses.

Dependencies

• Inbound: controller/scheduler/worker main.c --peers call sites (P0), harness test.c startup masters-default computation (P0).
• Outbound: none.

Contracts: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Service Interface

/* src/include/xce/cluster.h */
const char *xce_controller_peers_env(void);

• Preconditions: none.
• Postconditions: returns a pointer into the process environment (valid for the process lifetime) or NULL.
• Invariants: never returns a non-NULL empty string.

xceadmin

resolve_controllers (extended)

Field	Detail
Intent	Parse a --controller-shaped value into an ordered candidate list, now tolerant of name= prefixes
Requirements	2.1, 2.2

Responsibilities & Constraints

• Existing behavior (whitespace stripping, blank-entry skipping, bare-host default port) is unchanged.
• New: for each comma-separated entry, if it contains =, discard everything up to and including the first = before applying the existing rsplit(":", 1).

Dependencies

• Inbound: xceadmin --controller options (P0), any future caller expecting name=host:port tolerance.
• Outbound: none new.

Contracts: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Service Interface

def resolve_controllers(controller: str) -> list[tuple[str, int]]: ...

• Preconditions: controller is a comma-separated string of [name=]host[:port] entries.
• Postconditions: returns an ordered list of (host, port) tuples, name= prefixes (if any) stripped.
• Invariants: a bare host:port entry (today's only supported shape) parses identically to before.

Implementation Notes

• Integration: follow_redirects_any() is untouched — it only ever consumes resolve_controllers()'s output.
• Validation: existing test_discovery.py coverage for bare-host/list parsing must stay green; new tests add name=-prefixed and mixed-prefix cases.
• Risks: none identified (see research.md Risks & Mitigations for the =-in-hostname non-issue).

Error Handling

Error Strategy

Every existing hard-error-on-invalid-value behavior is preserved exactly; the only addition is naming the value's source in the message text, per 1.5.

Error Categories and Responses

• Invalid candidate list, from an explicit --peers/--controller/set masters value: unchanged existing message and exit behavior.
• Invalid candidate list, from XCE_CONTROLLER_PEERS: same parse failure, same exit behavior, message additionally names the environment variable as the source (controller/scheduler/worker: adjust the existing fprintf; harness: enter_ctx()'s existing "invalid 'masters' setting" message, extended to check settings[i].value == NULL for the "masters" entry at the point of failure — reusing get_setting()'s own existing value-before-defval precedence check, not a new tracking variable — to say "default (from XCE_CONTROLLER_PEERS)" instead of "explicit set masters" when true).
• Invalid candidate list, from legacy XCEADMIN_CONTROLLER: unchanged (out of scope — 1.5/2.x do not require source-naming for the legacy variable, only for XCE_CONTROLLER_PEERS and the explicit flag).

Monitoring

No new logging/metrics infrastructure; each component's existing startup log line (e.g. scheduler's INFO("HA cluster config: ...") / INFO("No --peers flag; ...")) is extended only to remain accurate about which source supplied the value it's reporting.

Testing Strategy

Unit Tests

• xce_controller_peers_env() (src/lib/testsuite/test_cluster.c): unset → NULL; empty string → NULL; non-empty → returned unchanged. (1.1, 1.3, 1.4)
• resolve_controllers() extension (src/xceadmin/tests/test_discovery.py): single name=host:port; mixed bare/named multi-candidate list; existing bare-host cases still pass unchanged. (2.2)
• xceadmin --controller option precedence (src/xceadmin/tests/test_cli.py): XCE_CONTROLLER_PEERS set alone; both XCE_CONTROLLER_PEERS and XCEADMIN_CONTROLLER set (new wins); explicit --controller overrides both; neither set (existing default). (2.1, 2.4-2.6) All existing test_cli.py cases that assert XCEADMIN_CONTROLLER-only or no-env-var behavior (e.g. test_controller_option_falls_back_to_xceadmin_controller_env_var) must explicitly monkeypatch.delenv("XCE_CONTROLLER_PEERS", raising=False), so the new variable's introduction cannot make them order-dependent on ambient environment state.

Scripted Regression Verification

None of controller/scheduler/worker/harness's main()-level argument or settings-table resolution has existing unit-test coverage today (confirmed via source search; test_cluster_parse.c tests a standalone duplicated reference parser, not live main() wiring) — refactoring any of the four into a unit-testable shape is out of this feature's boundary (see Non-Goals). Rather than one-time manual checking, this is verified by a new repeatable script, src/tools/verify-controller-peers-env.sh, following xce-test-multihost's established pattern of driving real built binaries and asserting on their output — but deliberately lightweight: it asserts on each component's startup log/settings-dump line, not on a live, reachable cluster, since resolving the candidate list happens, and is observable, entirely before any connection attempt succeeds or fails:

• controller/scheduler/worker: with XCE_CONTROLLER_PEERS set (e.g. a=host1:1,b=host2:2) and no --peers, start briefly and grep the log for the existing "N candidate(s)" line already logged today for an explicit --peers value, asserting N and the candidate identity match the env var's content — no reachable target required. (1.1, 1.3)
• Same three, with both XCE_CONTROLLER_PEERS set to one value and --peers set to a different one; assert the log reflects the --peers value, not the env var's. (1.2)
• Same three, with neither set; assert the existing "No --peers flag" (or worker's "single-controller path") log line, unchanged. (1.4)
• scheduler with --sim <file> and XCE_CONTROLLER_PEERS set: assert the process never logs an "HA cluster config"/"No --peers flag" line at all (the --sim path returns before that code is reached — this is a by-construction guarantee already, not new logic; the script asserts it stays true). (Non-Goals cross-check)
• harness/xce_test: run non-interactively with a script containing only set (no arguments) — cmd_set's existing argc==1 branch prints every setting's current value/default with no connection attempt — first with XCE_CONTROLLER_PEERS set and no prior set masters, asserting the printed masters default reflects the env var; then with an explicit set masters ... line before the bare set, asserting the printed value (not default) reflects the explicit override. (1.1, 1.3)
• Trigger a parse error via each of an invalid explicit flag and an invalid XCE_CONTROLLER_PEERS value on one component (e.g. controller); assert the two error messages are distinguishable by source. (1.5)

This script is a new addition to the File Structure Plan and is intended to be re-run on any future change to these five components' startup argument handling, not a one-time implementation check.