myxce/.kiro/specs/controller-peers-env-fallback/design.md

Design Document

Overview

Purpose: This feature lets an operator set one environment variable, XCE_CONTROLLER_PEERS, once per session or deployment, instead of retyping --peers (controller, scheduler, worker), set masters (harness/xce_test) or --controller (xceadmin) on every invocation. It changes nothing about what any of these components do once their connection target is resolved — only where that target's default value comes from when the operator did not name one explicitly.

Users: Operators running local/interactive HPC cluster tools, and deploy/test scripts (e.g. HA soak setups) that currently either hardcode --peers/--controller into every generated command or accept the single-node/no-controller default they didn't actually want.

Impact: Five independently-built binaries each gain one additional, lowest-priority source for their existing connection-target resolution. Explicit --peers/set masters behavior, "omitted, no env var" behavior (apart from a log line's wording, 1.8), and xceadmin's multi-candidate failover order are unchanged; xceadmin's parsing of any --controller value, explicit or not, gains name= stripping and a clean malformed-entry error (2.2, 2.8). Two guards come with the new source: the controller refuses to start, early and with a message naming the variable, when an env-supplied list does not contain its own name (that list is Raft membership, not just a target list), and deploy-ha.sh's generated scripts/xceadmin wrapper exports the new variable so an ambient value cannot redirect it.

Goals

• One new environment variable, used only when no explicit value is given, across controller, scheduler and worker (via --peers), harness/xce_test (via set masters) and xceadmin (via --controller).
• An explicit value (--peers, set masters, --controller) always wins over the variable. Explicit --peers/set masters behave exactly as today; an explicit --controller gains only name= stripping and a clean malformed-entry error (2.2, 2.8).
• Zero behavior change when neither an explicit value nor the variable is set (apart from the startup log line's wording, 1.8).
• Reuse xceadmin's existing multi-candidate failover (follow_redirects_any, bug-1821, unmodified) and its existing XCEADMIN_CONTROLLER fallback; resolve_controllers() is extended only as 2.2/2.8 require.

Non-Goals

• apiserver (YAML-config-based) and xcecli's --url (a different target) — untouched.
• Any change to src/tools/deploy-ha.sh's generated daemon scripts' behavior (the scripts/xceadmin wrapper's added export, Requirement 4, is the one exception).
• Any change to what "no explicit value and no env var" does today for any component (only the log wording changes, 1.8).
• A TSAN/new-test-infrastructure investment for main()-level argument parsing where none exists today (see Testing Strategy).

Boundary Commitments

This Spec Owns

• A single shared C primitive (xce_controller_peers_env(), libxce) that reads XCE_CONTROLLER_PEERS, treating an unset or empty value as absent.
• The one-line integration of that primitive into controller, scheduler, and worker's existing --peers-gating if, and into harness's settings-table default computation at startup.
• A second shared C primitive, xce_peers_find_identity() (libxce), holding the one rule for "which candidate is me" (candidate name if set, else hostname, compared with the controller's name). The controller's two existing copies of that loop — the my_priority assignment in cluster_init_phase1() (src/controller/cluster.c ~L1784-1791) and the listen-port lookup in cluster_open_listener() (~L1890-1906) — are switched to call it, each keeping its own existing no-match handling, so the new startup check and both existing lookups can never disagree.
• The controller's startup identity check for an env-supplied candidate list (1.6).
• One added line in deploy-ha.sh's generated scripts/xceadmin wrapper: export XCE_CONTROLLER_PEERS="$PEERS_STR" (Requirement 4).
• Extending xceadmin's existing resolve_controllers() to strip an optional name= prefix per candidate entry.
• Adding "XCE_CONTROLLER_PEERS" ahead of the existing "XCEADMIN_CONTROLLER" in each --controller option's envvar= list.
• Help-text and documentation updates naming the new variable (CLI --peers/--controller help text, build-deployment.md, xce-deploy-ha/xce-test-multihost skill docs).
• A repeatable regression script (src/tools/verify-controller-peers-env.sh) proving the above across all 5 components, intended for ongoing re-use, not a one-time implementation check.

Out of Boundary

• apiserver's YAML-based peer configuration.
• xcecli's --url option.
• deploy-ha.sh's generated daemon scripts' own invocations (they keep passing --peers explicitly). The scripts/xceadmin wrapper's one added export (Requirement 4) is the only deploy-ha.sh change.
• Changing what the controller does when an explicit --peers list lacks its name. Today cluster_open_listener() refuses (ERROR in the log, myapp_run() returns before any network I/O), and main() ignores myapp_run()'s result, so the process exits with status 0. That behavior is kept; see Revalidation Triggers.
• Any change to xce_parse_peers()'s or resolve_controllers()/follow_redirects_any()'s existing parsing/failover semantics beyond resolve_controllers()'s two additions: name= stripping (2.2) and a clean error for a malformed entry (2.8). Both apply to an explicit --controller value too.
• Refactoring any of the four C main() functions to become unit-testable in general — only the two new, genuinely independent primitives are unit-tested (see Testing Strategy).

Allowed Dependencies

• All four C targets already link libxce (verified per-component meson.build) — the shared primitives may live there with no new build-graph edge.
• xceadmin's existing resolve_controllers()/follow_redirects_any() (discovery.py) and its existing --controller click options (cli.py) — reused and extended in place, not duplicated.

Project Invariant Check

Walked against xce-audit-invariants' controller and scheduler dimensions:

• Consensus/replication safety (controller) — applies. For the controller, --peers is the Raft membership, and my_priority doubles as self_idx (ha.c ~L633, ~L2964). If the controller's name matches no candidate, cluster_init_phase1() (cluster.c ~L1783) falls back to the last index, which would give the node another peer's self_idx. That is already contained today: cluster_open_listener() (~L1902) refuses the same no-match case in myapp_run() step 3, before the listener, iomgr or any election starts, so no Raft traffic is sent under the wrong identity. An inherited value makes the case far more likely, though (a controller started without --myname, so its name is the hostname, or one meant to run standalone), and the existing refusal then blames "--peers", logs only to the log file, and exits with status 0. Guard (1.6): when the list came from XCE_CONTROLLER_PEERS and xce_peers_find_identity() finds no match, main.c writes the error to stderr naming the variable and the remedies and returns 1, before myapp_run(). Because all three lookups now share one function, the new check cannot pass where the listener check would refuse, or the reverse. A bare single host:port value follows the same rule: its key is the hostname, so a controller joins a one-node HA cluster only when its name is literally that hostname; otherwise it refuses, rather than silently changing from standalone to HA mode.
• Fail-loud, never self-exit (controller) — not affected. The new exit happens in main() during argument validation, the same place and shape as the existing --peers parse error and even-peer-count rejection; no daemon thread, event loop or HA state exists yet.
• Async I/O thread roles (controller) — not affected. All reads of the environment happen in main() before any thread starts; the two cluster.c refactors are pure function calls in existing pre-election startup code (cluster_init_phase1(), cluster_open_listener()), with no blocking added.
• Driver/simulation state-equivalence (scheduler) — not affected. --sim (with or without --interactive) returns before the --peers block, so the variable can never reach replay mode; --interactive without --sim is a live run and takes the normal --peers path.

Revalidation Triggers

• xce_peers_find_identity() is now the only "which candidate is me" rule for the controller (the 1.6 check, my_priority, listen port). Any future identity rule (e.g. matching on host:port) changes there, never in a re-added inline loop.
• If the explicit---peers no-match case is ever moved to the same early main() check with a non-zero exit (a reasonable follow-up, out of scope here), drop the env-only condition on the 1.6 check instead of adding a second check.
• Any change to xce_parse_peers()'s accepted value format (e.g. a new candidate syntax) must be mirrored in resolve_controllers()'s parsing if the two are to stay consistent for XCE_CONTROLLER_PEERS values shared between C daemons and xceadmin.
• Any future change to harness's settings[] initialization order (e.g. rc-file loading moved earlier than today) must preserve "the env-var-derived default is computed before any explicit set masters can run."
• If deploy-ha.sh's generated daemon scripts are ever updated to rely on XCE_CONTROLLER_PEERS instead of an explicit --peers (explicitly deferred here), re-check them against the baseline in this design's Out of Boundary, and re-check the 1.6 identity rule for each generated controller script.

Architecture

Existing Architecture Analysis

Each of the five components already resolves its own connection target independently, with no shared code path today:

Component	Mechanism	Where "omitted" is decided	Existing fallback today
controller	argp, OPT_PEERS → arguments.peers_str	if (arguments.peers_str) (main.c ~L455)	Single-node primary (bug-1457). With a list, own entry found by name twice in cluster.c (~L1784 my_priority, ~L1890 listen port); no match is refused in cluster_open_listener() before any network I/O, exit status 0
scheduler	argp, 'P'/--peers → arguments.peers	if (arguments.peers) (main.c ~L411); --sim returns before this point is reached	"single-controller path" (logged)
worker	getopt_long, OPT_PEERS → peers_str	if (peers_str) (main.c ~L470)	Single-controller path via config.remote_host/config.remote_port (logged; same shape as controller/scheduler's own single-controller fallback, not an error path)
harness/xce_test	settings[] table, key "masters", no CLI flag at all	get_setting()'s value ?: defval (test.c ~L991)	defval = "localhost:9990"
xceadmin	click --controller option, envvar="XCEADMIN_CONTROLLER"	click's own parameter resolution	_DEFAULT_CONTROLLER = "localhost:9991"

All three C daemons that gate on a --peers argument already treat an invalid value as a hard startup error (xce_parse_peers() != 0 → message + return 1), never a silent fallback — this feature preserves that for both an invalid explicit flag and an invalid XCE_CONTROLLER_PEERS value.

Architecture Integration

• Selected pattern: two shared, stateless C primitives (an env reader used by four call sites, and an identity lookup used by three controller sites), plus one Python parsing extension consulted by click's own (unmodified) resolution order — no new service, process, or inter-component call is introduced.
• Domain/feature boundaries: each component keeps deciding its own resolved value; the new variable is only ever a source, never a coordinator.
• Existing patterns preserved: xce_parse_peers()'s name=-prefix convention; xceadmin's resolve_controllers/follow_redirects_any failover; each component's existing hard-error-on-invalid-value behavior.
• New components rationale: two new C functions. xce_controller_peers_env() is a bare env reader because harness's settings-table shape cannot reuse a "given cli_value, return resolved" signature the other three could share (see research.md, Design Decisions). xce_peers_find_identity() exists so the controller's new startup check and its two existing lookups (my_priority, listen port) share one matching rule.
• Steering compliance: no new build dependency; C standard/build system (Meson) and Python (click) usage unchanged.

Technology Stack

Layer	Choice / Version	Role in Feature	Notes
C CLI / libxce	gnu17, existing libxce static lib	New xce_controller_peers_env() (4 call sites) and xce_peers_find_identity() (controller main.c + two cluster.c sites)	No new dependency
Python CLI	click (already a xceadmin dependency)	Native multi-envvar list resolution (no new click usage pattern)	Confirmed via direct read of the installed click version's resolve_envvar_value()

File Structure Plan

One new file (the regression script below); every other change is to an existing file — no new directories.

New Files

• src/tools/verify-controller-peers-env.sh — repeatable regression script (see Testing Strategy → Scripted Regression Verification) driving the real built binaries of controller, scheduler, worker and harness/xce_test, plus deploy-ha.sh's generated xceadmin wrapper, asserting on startup log/settings-dump/printed-environment output only (no live cluster required).

Modified Files

• src/include/xce/cluster.h — declare const char *xce_controller_peers_env(void); and int xce_peers_find_identity(const xce_cluster_config_t *cfg, const char *name); alongside the existing xce_parse_peers() declaration.
• src/lib/cluster.c — implement xce_controller_peers_env(): read getenv("XCE_CONTROLLER_PEERS"), return it if non-empty, else NULL. Implement xce_peers_find_identity() with the matching rule moved verbatim from src/controller/cluster.c (identical at ~L1784-1791 and ~L1890-1900).
• src/lib/testsuite/test_cluster.c — new unit tests for xce_controller_peers_env() (unset, empty, non-empty) and xce_peers_find_identity() (named match, bare-hostname match, named entry not matched by its hostname, no match → -1, first of duplicate names wins, NULL name → -1, empty name matching an unnamed empty-hostname entry exactly as today).
• src/controller/main.c — at the existing if (arguments.peers_str) site (~L455), resolve arguments.peers_str against xce_controller_peers_env() before the check, remembering in a local peers_from_env whether the env var supplied it; on a parse or even-count failure, name the source (flag vs. environment variable) in the existing error message; after parsing, when peers_from_env is true and xce_peers_find_identity(&myconf.peers_cfg, arguments.myname) < 0, print the 1.6 error and return 1; extend the "No --peers flag" INFO (~L486) per 1.8; call mylog_fini() before every early return after mylog_init() (~L393) — the get_myname() failure (~L403), the mylock_init() failure (~L452), the --peers parse and even-count errors, the new 1.6 check, and the cert pre-flight (~L489-496); it is a no-op before mylog_init() and safe to call twice — because tlog writes the log file from a background thread with no atexit flush, so today those returns can lose already-logged lines (the scheduler already drains explicitly for the same reason); add XCE_CONTROLLER_PEERS and the "unset it to run standalone" note (1.7) to the --peers option's help text (~L85-87).
• src/controller/cluster.c — replace both inline identity loops with xce_peers_find_identity(pcfg, c->myconf->name): in cluster_init_phase1() (~L1784-1791), keep the existing if (pcfg->candidates_count > 0) guard (~L1782; standalone mode leaves my_priority = 0) and replace only the loop inside it with my_priority = idx >= 0 ? idx : pcfg->candidates_count - 1;; in cluster_open_listener() (~L1890-1906), matched = idx >= 0; my_port = matched ? pcfg->candidates[idx].port : 0; followed by the existing refusal and default-port code. Same result for every input.
• src/tools/deploy-ha.sh — in the generated scripts/xceadmin wrapper heredoc (~L689-694), add export XCE_CONTROLLER_PEERS="$PEERS_STR" next to the existing XCEADMIN_CONTROLLER export, and extend the comment above it to say why (Requirement 4).
• src/scheduler/src/main.c — identical shape at the existing if (arguments.peers) site (~L411), without the identity check; name the source in both the parse-error and the xce_cluster_config_validate() failure messages (~L412-424, 1.4); "No --peers flag" INFO (~L435) per 1.8; help text update with 1.7's note (~L88). No change anywhere in the --sim path, which already returns before this point.
• src/worker/src/main.c — identical shape at the existing if (peers_str) site (~L470), without the identity check; name the source in both the parse-error and the xce_cluster_config_validate() failure messages (~L472-485, 1.4); "No --peers flag" log line (~L501) per 1.8; update the -h/usage printf (~L443-446) to mention XCE_CONTROLLER_PEERS, 1.7's note, and that sudo drops it unless preserved.
• src/harness/test.c — at startup, before the REPL loop and before rc-file loading, compute the "masters" entry's defval from xce_controller_peers_env() if set, else leave it at today's literal "localhost:9990"; extend the USAGE/set-command help text (~L403-415) to mention the new variable and 1.7's note.
• src/xceadmin/src/xceadmin/discovery.py — extend resolve_controllers()'s per-entry parsing to strip an optional name= prefix (mirrors xce_parse_peers()) before its existing rsplit(":", 1); turn today's uncaught ValueError from int(port_str) (~L78) into a new ControllerValueError naming the bad entry (2.8).
• src/xceadmin/src/xcecommon/errors.py — add ControllerValueError(XceAdminError) next to the existing controller-reachability errors (ConnectionTimeoutError, TooManyRedirectsError, NoPrimaryError).
• src/xceadmin/src/xceadmin/cli.py — change _CONTROLLER_ENV_VAR (~L109), which all 4 --controller options use, to the list ["XCE_CONTROLLER_PEERS", "XCEADMIN_CONTROLLER"]; rewrite each option's help string, which today interpolates _CONTROLLER_ENV_VAR with an f-string and would print a raw Python list once it becomes one, and the precedence comment above it (~L102-108). Add a helper _controller_source_note() that returns " (from $NAME)" when click.get_current_context().get_parameter_source("controller") is ParameterSource.ENVIRONMENT (NAME = the first non-empty variable in the list, the same rule click applies) and "" otherwise; append it to the two existing cannot connect to {controller} messages (~L172, ~L265) only when the OSError is not a local file error (e.filename is None: socket errors carry no filename, while read_bytes() of a missing or unreadable cluster-bin file in push_config()/probe_config() (push.py ~L89, ~L143) raises a FileNotFoundError/PermissionError with filename set, which reaches the same except OSError branch today and must not be blamed on the variable), and, in the existing except XceAdminError handlers of the --controller-accepting commands, to the message when the error is one of _CONTROLLER_ERRORS = (ControllerValueError, ConnectionTimeoutError, TooManyRedirectsError, NoPrimaryError) (2.8). Errors unrelated to the controller value (e.g. a missing cluster-bin file, or ClusterBinFileError from config build) keep their text unchanged.
• src/tools/test-version-startup-consistency.sh, src/tools/test-dirty-fingerprint-startup-log-unchanged.sh — add unset XCE_CONTROLLER_PEERS near the top, so their scheduler run keeps taking the no-peers path they were written for (see Testing Strategy, ambient-variable exposure).
• src/xceadmin/tests/test_discovery.py — new unit tests for resolve_controllers()'s name=-stripping and its ControllerValueError on a non-numeric port.
• src/xceadmin/tests/test_cli.py — new unit tests for the new env-var precedence (mirrors the existing test_controller_option_falls_back_to_xceadmin_controller_env_var/test_controller_flag_overrides_xceadmin_controller_env_var tests) and for 2.8's messages.
• .kiro/steering/build-deployment.md — document XCE_CONTROLLER_PEERS: components, value format, precedence, the "unset it to run standalone" opt-out, the controller's own-name rule (1.6), and that sudo (default env_reset) and systemd services do not inherit it (sudo --preserve-env=XCE_CONTROLLER_PEERS, or Environment= in the unit).
• .claude/skills/xce-deploy-ha/SKILL.md — note the variable's availability, that the generated daemon scripts still pass --peers, and that scripts/xceadmin now also exports XCE_CONTROLLER_PEERS (update the existing ~L330 description and the ~L518 troubleshooting row's "check $TOP/scripts/xceadmin" hint).
• .claude/skills/xce-test-multihost/SKILL.md — same note, at its three existing --peers-mentioning locations (single-node deploy step, xcecd_virtual flag-table note, Prerequisites' generated-script list).

System Flows

flowchart TD
    Start[Resolution point reached] --> HasFlag{Explicit flag or\nset command given?}
    HasFlag -- yes --> UseFlag[Use that value\nenv var not used as list]
    HasFlag -- no --> HasNewEnv{XCE_CONTROLLER_PEERS\nnon-empty?}
    HasNewEnv -- yes --> IsCtrl{Component is controller?}
    IsCtrl -- no --> UseNewEnv[Use it\nxceadmin strips name prefix]
    IsCtrl -- yes --> SelfListed{Own name matches\na candidate?}
    SelfListed -- yes --> UseNewEnv
    SelfListed -- no --> Refuse[Refuse to start\nname source and remedies]
    HasNewEnv -- no --> IsAdmin{Component is xceadmin?}
    IsAdmin -- yes --> HasLegacyEnv{XCEADMIN_CONTROLLER\nnon-empty?}
    IsAdmin -- no --> UseTodayDefault[Today's unchanged default\nstandalone / built-in literal]
    HasLegacyEnv -- yes --> UseLegacyEnv[Use it, unchanged behavior]
    HasLegacyEnv -- no --> UseTodayDefault

All three branches that resolve to a real candidate list (UseFlag, UseNewEnv, UseLegacyEnv) still pass through each component's existing validation (xce_parse_peers() and the controller's odd-count rule, unchanged; resolve_controllers(), extended per 2.2/2.8); an invalid value is a hard error either way, distinguished by source only in the error message text (1.4, 2.8).

This resolution point is reached exactly once, at process startup, for controller/scheduler/worker/xceadmin. For harness/xce_test it is reached on every connection attempt (enter_ctx()), since set masters ... can run at any point in an interactive/scripted session — the diagram represents one such evaluation, not a startup-only decision, for that component.

Requirements Traceability

Requirement	Summary	Components	Interfaces	Flows
1.1	Explicit candidate list wins, env var not used as candidate list	controller/scheduler/worker main.c; harness test.c (explicit set masters, interactive or rc-file)	existing if (peers) gate; get_setting() value-before-defval	Flowchart: HasFlag = yes
1.2	Env var used when no explicit list (harness: at each connection attempt)	controller/scheduler/worker main.c, harness test.c	xce_controller_peers_env(); harness masters defval	Flowchart: HasNewEnv → UseNewEnv
1.3	Unset/empty env var ⇒ unchanged today's behavior	all 4	same call sites	Flowchart: UseTodayDefault
1.4	Parse/validation error (incl. controller even count) names the source	controller/scheduler/worker main.c, harness test.c	existing error fprintfs (parse, controller even-count)/enter_ctx message	n/a
1.5	--peers (harness: set) help text documents the variable	controller/scheduler/worker main.c, harness test.c	argp/getopt/settings help strings	n/a
1.6	Controller refuses an env-supplied list without its own name	controller main.c	xce_peers_find_identity()	Flowchart: SelfListed = no
1.7	Opt-out is "unset the variable"; help says so	controller/scheduler/worker main.c, harness test.c	help strings; xce_controller_peers_env() empty-as-unset	Flowchart: HasNewEnv = no
1.8	"No --peers" log line says the variable was unset or empty	controller/scheduler/worker main.c	existing INFO/log line text	Flowchart: UseTodayDefault
2.1	Derive controller list from env var	xceadmin cli.py, discovery.py	click envvar= list	Flowchart: UseNewEnv
2.2	Strip name= prefix, any source incl. explicit --controller	xceadmin discovery.py	resolve_controllers()	n/a
2.3	Try each candidate in order	xceadmin discovery.py	resolve_controllers/follow_redirects_any (unmodified)	n/a
2.4	Explicit --controller wins	xceadmin cli.py	click parameter resolution (unmodified)	Flowchart: HasFlag = yes
2.5	Legacy XCEADMIN_CONTROLLER fallback unchanged	xceadmin cli.py	click envvar= list order	Flowchart: HasLegacyEnv
2.6	Both unset ⇒ hardcoded default unchanged	xceadmin cli.py	_DEFAULT_CONTROLLER (unmodified)	Flowchart: UseTodayDefault
2.7	--controller help text documents the variable	xceadmin cli.py	click option help strings	n/a
2.8	Clean error for unreachable/malformed; names the env var source	xceadmin cli.py, discovery.py	ControllerValueError; _controller_source_note()	n/a
3.1	Steering doc describes the variable: components, format, precedence, opt-out (1.7), own-name rule (1.6), sudo/systemd not inheriting it	build-deployment.md	n/a	n/a
3.2	HA deploy guidance notes availability and the wrapper export	xce-deploy-ha/SKILL.md	n/a	n/a
3.3	Multi-host guidance notes availability	xce-test-multihost/SKILL.md	n/a	n/a
4.1	Wrapper exports the deployment's own list	deploy-ha.sh wrapper heredoc	export XCE_CONTROLLER_PEERS="$PEERS_STR"	n/a
4.2	Wrapper keeps XCEADMIN_CONTROLLER	deploy-ha.sh wrapper heredoc	existing export, unchanged	n/a
4.3	Explicit --controller still wins in the wrapper	xceadmin cli.py	click parameter resolution (unmodified)	Flowchart: HasFlag = yes

Components and Interfaces

Component	Domain/Layer	Intent	Req Coverage	Key Dependencies	Contracts
xce_controller_peers_env()	libxce / C	Read XCE_CONTROLLER_PEERS, empty-as-unset	1.2, 1.3, 1.7	none (P0)	Service
xce_peers_find_identity()	libxce / C	One rule for "which candidate is me"	1.6	none (P0)	Service
controller/scheduler/worker --peers call sites	C daemon CLI	Combine flag + env var, preserve existing validation/help; controller adds the 1.6 check	1.1-1.8	xce_controller_peers_env (P0), xce_peers_find_identity (P0, controller only)	—
harness masters setting default	C CLI (settings table)	Compute defval from env var at startup	1.1-1.5, 1.7	xce_controller_peers_env (P0)	State
resolve_controllers() extension	xceadmin / Python	Strip name= prefix per entry; clean error on a malformed entry	2.1, 2.2, 2.8	none (P0)	Service
xceadmin --controller options	xceadmin / Python (click)	Add XCE_CONTROLLER_PEERS ahead of legacy envvar; name the env source in errors	2.1, 2.4-2.8	resolve_controllers (P0), click (P0, external)	—
scripts/xceadmin wrapper export	deploy-ha.sh	Pin the wrapper to its own deployment	4.1-4.3	none	—
Documentation updates	Docs	Describe/announce the variable	3.1-3.3	none	—

Only the three components introducing genuinely new logic — xce_controller_peers_env(), xce_peers_find_identity() and the resolve_controllers() extension — warrant a full contract block; the remaining rows are thin call-site wiring around existing, unchanged behavior.

libxce

xce_controller_peers_env

Field	Detail
Intent	Sole read point for XCE_CONTROLLER_PEERS, normalizing empty-string to absent
Requirements	1.2, 1.3, 1.7

Responsibilities & Constraints

• Pure function: no side effects, no caching (the environment can only change between process invocations for these components, so no staleness concern).
• Treats an empty string identically to an unset variable, matching the same convention resolve_controllers()'s per-entry blank-skip already uses.

Dependencies

• Inbound: controller/scheduler/worker main.c --peers call sites (P0), harness test.c startup masters-default computation (P0).
• Outbound: none.

Contracts: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Service Interface

/* src/include/xce/cluster.h */
const char *xce_controller_peers_env(void);

• Preconditions: none.
• Postconditions: returns a pointer into the process environment (valid for the process lifetime) or NULL.
• Invariants: never returns a non-NULL empty string.

xce_peers_find_identity

Field	Detail
Intent	Find the controller's own candidate in a parsed peers list
Requirements	1.6

Responsibilities & Constraints

• Pure function over an already-parsed xce_cluster_config_t; no environment or global access.
• Matching rule, moved unchanged from src/controller/cluster.c (~L1784-1791 and ~L1890-1900, identical today): a candidate's key is its name when non-empty, else its hostname; the first candidate whose key equals name (compared with strncmp(key, name, XCE_MAXNAME_LEN - 1)) wins.
• Does not apply any default: "no match → last index" stays at the my_priority call site, "no match → refuse" at cluster_open_listener(), and the 1.6 check treats no match as an error.

Dependencies

• Inbound: controller main.c startup check (P0), controller cluster.c my_priority assignment and listen-port lookup (P0).
• Outbound: none.

Contracts: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Service Interface

/* src/include/xce/cluster.h */
int xce_peers_find_identity(const xce_cluster_config_t *cfg, const char *name);

• Preconditions: cfg non-NULL.
• Postconditions: returns the index (0..candidates_count-1) of the first matching candidate, or -1 when name is NULL or nothing matches. An empty name is compared like any other (today's loops do the same: it matches an unnamed entry with an empty hostname, e.g. from :5), so the invariant below holds for every input.
• Invariants: for every input with candidates_count > 0, idx >= 0 ? idx : candidates_count - 1 equals what cluster.c computes for my_priority today (with no candidates, the call site's existing guard keeps my_priority = 0, and the helper returns -1); and for every input, idx >= 0 equals the listener's matched today.

xceadmin

resolve_controllers (extended)

Field	Detail
Intent	Parse a --controller-shaped value into an ordered candidate list, now tolerant of name= prefixes
Requirements	2.1, 2.2, 2.8

Responsibilities & Constraints

• Existing behavior (whitespace stripping, blank-entry skipping, bare-host default port) is unchanged.
• New: for each comma-separated entry, if it contains =, discard everything up to and including the first = before applying the existing rsplit(":", 1).
• New: a port that is not an integer raises ControllerValueError (an XceAdminError subclass) naming the entry, instead of today's uncaught ValueError.
• Both changes apply to every caller, an explicit --controller value included.

Dependencies

• Inbound: xceadmin --controller options (P0), any future caller expecting name=host:port tolerance.
• Outbound: none new.

Contracts: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

Service Interface

def resolve_controllers(controller: str) -> list[tuple[str, int]]: ...

• Preconditions: controller is a comma-separated string of [name=]host[:port] entries.
• Postconditions: returns an ordered list of (host, port) tuples, name= prefixes (if any) stripped; raises ControllerValueError on a malformed entry.
• Invariants: a bare host:port entry (today's only supported shape) parses identically to before.

Implementation Notes

• Integration: follow_redirects_any() is untouched — it only ever consumes resolve_controllers()'s output.
• Validation: existing test_discovery.py coverage for bare-host/list parsing must stay green; new tests add name=-prefixed and mixed-prefix cases.
• Risks: none identified (see research.md Risks & Mitigations for the =-in-hostname non-issue).

Error Handling

Error Strategy

Every existing hard-error-on-invalid-value behavior is preserved exactly, with the value's source named in the message text (1.4). The one new error is the controller's own-name check for an env-supplied list (1.6).

Error Categories and Responses

• Invalid candidate list, from an explicit --peers value: unchanged exit behavior; message names --peers as the source (most already do; scheduler's and worker's xce_cluster_config_validate() messages gain it).
• Invalid candidate list, from an explicit set masters: unchanged behavior; enter_ctx()'s message names the source as described in the next bullet.
• Invalid or unreachable value, from an explicit --controller: today's unreachable messages unchanged; a malformed entry now gives the 2.8 clean error instead of a traceback. No source note is added, since the operator typed the value.
• Invalid candidate list, from XCE_CONTROLLER_PEERS: same parse or validation failure (including the controller's even-count rejection), same exit behavior, message additionally names the environment variable as the source (controller/scheduler/worker: adjust the existing fprintf; harness: enter_ctx()'s existing "invalid 'masters' setting" message, extended to say "default (from XCE_CONTROLLER_PEERS)" when the "masters" entry's settings[i].value == NULL (no set masters ran, reusing get_setting()'s own value-before-defval precedence) and xce_controller_peers_env() != NULL; "explicit set masters" when value != NULL; and today's wording otherwise (the built-in localhost:9990 literal, which cannot fail to parse today, is then never blamed on the variable).
• Controller name not in an env-supplied list (1.6): main.c prints, to stderr, e.g. xce_controller: XCE_CONTROLLER_PEERS (a=h1:1,b=h2:2) has no candidate named 'node7'; pass --myname <one of: a, b> or unset XCE_CONTROLLER_PEERS to run standalone, then return 1 before the cert pre-flight and myapp_run(). This replaces, for the env-var case only, the later cluster_open_listener() refusal, which blames "--peers", writes only to the log file, and exits with status 0. Candidates without a name= are listed by hostname.
• xceadmin, unreachable candidates or a malformed entry (2.8): click.ClickException (exit 1) with today's text (cannot connect to {controller}: ... for OSError, or the e.message of ConnectionTimeoutError/TooManyRedirectsError/NoPrimaryError), or invalid --controller entry '...' for ControllerValueError, followed by (from $XCE_CONTROLLER_PEERS) or (from $XCEADMIN_CONTROLLER) when the value came from the environment. Today a malformed port is an uncaught ValueError traceback.

Monitoring

No new logging/metrics infrastructure; each component's existing startup log line (e.g. scheduler's INFO("HA cluster config: ...") / INFO("No --peers flag; ...")) is extended only to remain accurate about which source supplied the value it's reporting. The three "No --peers flag" lines (controller main.c ~L486, scheduler ~L435, worker ~L501) become "No --peers flag and XCE_CONTROLLER_PEERS unset or empty; ..." (1.8). This is the operator's only clue when sudo's default env_reset or a systemd unit drops the variable — the main risk for the production worker, which check_user() requires to run as root (deploy-ha.sh ~L470 starts it with sudo).

Testing Strategy

Unit Tests

• xce_controller_peers_env() (src/lib/testsuite/test_cluster.c): unset → NULL; empty string → NULL; non-empty → returned unchanged. (1.2, 1.3, 1.7)
• xce_peers_find_identity() (src/lib/testsuite/test_cluster.c): the cases listed in File Structure Plan, each written first as a failing test. The existing controller tests that go through my_priority and cluster_open_listener() must stay green after cluster.c switches to the helper. (1.6)
• resolve_controllers() extension (src/xceadmin/tests/test_discovery.py): single name=host:port; mixed bare/named multi-candidate list; existing bare-host cases still pass unchanged. (2.2)
• xceadmin --controller option precedence (src/xceadmin/tests/test_cli.py): XCE_CONTROLLER_PEERS set alone; both XCE_CONTROLLER_PEERS and XCEADMIN_CONTROLLER set (new wins); explicit --controller overrides both; neither set (existing default). (2.1, 2.4-2.6) For 2.3: with XCE_CONTROLLER_PEERS="a=127.0.0.1:1,b=127.0.0.1:2,127.0.0.1:3" and no --controller, patch follow_redirects (the per-candidate call inside follow_redirects_any) so the first candidate raises a connection error and the second returns a canned reply, and assert the candidates arrive in env order with name= stripped and that the second one is tried next. (2.3) For 2.8: an unreachable value from each of XCE_CONTROLLER_PEERS, XCEADMIN_CONTROLLER and an explicit --controller (message names the variable in the first two only), and a non-numeric port giving a clean error rather than a traceback — run with a readable cluster-bin file too, since config push reads the file before resolving controllers: once as an explicit --controller a=h:x (message quotes the entry, no source note) and once as XCE_CONTROLLER_PEERS=a=h:x with no --controller (message quotes the entry and ends (from $XCE_CONTROLLER_PEERS)). Run the unreachable cases through config push with a real, readable cluster-bin file (push_config() reads it before resolving controllers, so a missing file raises FileNotFoundError first — add one case asserting that error gets no source note even with XCE_CONTROLLER_PEERS set), or patch out the retry sleep, since the scheduler record commands retry NoPrimaryError 8 times with a 2 s pause (scheduler.py ~L131-132). All existing test_cli.py cases that assert XCEADMIN_CONTROLLER-only or no-env-var behavior (e.g. test_controller_option_falls_back_to_xceadmin_controller_env_var) must explicitly monkeypatch.delenv("XCE_CONTROLLER_PEERS", raising=False), so the new variable's introduction cannot make them order-dependent on ambient environment state.

Scripted Regression Verification

None of controller/scheduler/worker/harness's main()-level argument or settings-table resolution has existing unit-test coverage today (confirmed via source search; test_cluster_parse.c tests a standalone duplicated reference parser, not live main() wiring) — refactoring any of the four into a unit-testable shape is out of this feature's boundary (see Non-Goals). Rather than one-time manual checking, this is verified by a new repeatable script, src/tools/verify-controller-peers-env.sh, following xce-test-multihost's established pattern of driving real built binaries and asserting on their output — but deliberately lightweight: it asserts on each component's startup log/settings-dump line, not on a live, reachable cluster, since resolving the candidate list happens, and is observable, entirely before any connection attempt succeeds or fails. The worker cases use xcecd_virtual, since the production xcecd requires root (check_user()), and grep the worker's captured stdout, not xcecd-<name>.log: its "HA cluster config" / "No --peers flag" lines (worker/src/main.c ~L493-503) are logged before its log file is set up (~L529-544), so the logging library still writes them to stdout. The controller and scheduler set up their log files earlier (controller main.c ~L393, scheduler ~L332), so their cases grep the log file. Controller cases always use an odd candidate count and pass --myname naming a candidate unless the case is about the 1.6 check, so the even-count rule and 1.6 do not mask what the case is testing. Every daemon run is isolated in a fresh scratch directory: the controller always gets --workdir, --certdir and --logfile there (otherwise it defaults to /tmp/xce and /tmp/xce_controller.log, src/include/config.h:22, main.c:312, and could touch a local cluster's state), the scheduler --logfile, and the worker -l. With an empty certificate directory the controller exits at the certificate pre-flight (main.c ~L489), right after logging the line under test (~L476/~L486) and after the 1.6 check, so it never reaches myapp_run(); the mylog_fini() added before that return (File Structure Plan, main.c) is what guarantees the line is in the log file when the script reads it. Scheduler and worker runs are killed after a short timeout once the asserted line has appeared. Two things a scratch directory cannot isolate: with no peers list the scheduler and worker still try their default controller (localhost:9990), and the worker always uses the hardcoded /var/run/.xcecd.sock. The script therefore checks first that nothing listens on port 9990 and that the socket does not exist, and fails with a message (rather than running) when a local cluster is up.

• controller/scheduler/worker: with XCE_CONTROLLER_PEERS set (e.g. a=localhost:1,b=localhost:2,c=localhost:3; controller also --myname a) and no --peers, start briefly and grep the log (worker: stdout) for the line each already logs today for an explicit --peers value — controller HA mode: N controller candidates, identity=... (main.c ~L476), scheduler and worker HA cluster config: N candidate(s), ... (scheduler ~L431, worker ~L493) — asserting N matches the env var's content — no reachable target required. (1.2)
• Same three, with both XCE_CONTROLLER_PEERS set to one value (3 candidates) and --peers set to a different one (1 candidate, which the controller's --myname names); assert the log reflects the --peers value, not the env var's (worker: stdout). (1.1)
• Same three, with neither set; assert the "No --peers flag" (or worker's "single-controller path", on stdout) log line now also says XCE_CONTROLLER_PEERS was unset or empty, and that behavior is otherwise unchanged. (1.3, 1.8)
• scheduler with --sim <nonexistent file> and XCE_CONTROLLER_PEERS set: assert stderr shows xce_scheduler: --sim startup failed, which only the --sim branch prints (proving that branch ran, so the case cannot pass by failing earlier), and that the process never logs an "HA cluster config"/"No --peers flag" line at all (the --sim path returns before that code is reached — this is a by-construction guarantee already, not new logic; the script asserts it stays true). (Non-Goals cross-check)
• harness/xce_test (always with --norc, so a developer's own rc file cannot change the result): run non-interactively with a script containing only set (no arguments) — cmd_set's existing argc==1 branch prints every setting's current value/default with no connection attempt — first with XCE_CONTROLLER_PEERS set and no prior set masters, asserting the printed masters default reflects the env var; then with an explicit set masters ... line before the bare set, asserting the printed value (not default) reflects the explicit override. (1.1, 1.2)
• For each of controller, scheduler and worker, trigger a parse error once with an invalid explicit --peers and once with an invalid XCE_CONTROLLER_PEERS and no --peers; assert on stderr (all three write these errors with fprintf(stderr, ...)) that the two messages name different sources. For the controller, repeat with a 2-candidate value from each source (even-count message); for scheduler and worker, repeat with a=localhost:0, which parses but fails xce_cluster_config_validate() (port 0). (1.4)
• harness/xce_test --norc, each script ending in a connecting command (client): (a) set masters bad with XCE_CONTROLLER_PEERS unset → message says explicit set masters; (b) no set masters, XCE_CONTROLLER_PEERS=bad → message says default from XCE_CONTROLLER_PEERS; (c) set masters bad with XCE_CONTROLLER_PEERS set to a valid value → still says explicit set masters (explicit wins, 1.1). (1.1, 1.4)
• controller with XCE_CONTROLLER_PEERS=a=localhost:1,b=localhost:2,c=localhost:3: with --myname zz, assert a non-zero exit and the 1.6 message (names zz, the variable, and the remedies); with --myname b, assert the "HA mode: 3 controller candidates" line and no 1.6 message; with --myname zz --peers a=localhost:1,b=localhost:2,c=localhost:3, assert no 1.6 message (explicit-flag behavior unchanged). (1.1, 1.6)
• controller with XCE_CONTROLLER_PEERS="" and no --peers: assert the "No --peers flag ... running as single-node primary" line. (1.7)
• Help text (1.5, 1.7): run xce_controller --help, xce_scheduler --help and xcecd_virtual -h, and feed help set to xce_test --norc; assert each output names XCE_CONTROLLER_PEERS and says that unsetting or emptying it opts out for one invocation. For xceadmin, test_cli.py asserts the same name in each --controller option's --help output (2.7).
• deploy-ha.sh wrapper: run deploy-ha.sh with the built controller binary and a peers list on free high ports (it refuses ports already in use, deploy-ha.sh ~L204-225) into a scratch $TOP (generation only, nothing started), copy $TOP/scripts/xceadmin with its final exec line replaced by printenv XCE_CONTROLLER_PEERS XCEADMIN_CONTROLLER, and run the copy with XCE_CONTROLLER_PEERS=bogus:1 in the caller's environment; assert it prints the deployment's own list in deploy-ha.sh's PEERS_STR form (ctrl-1=host:port,ctrl-2=..., not the raw peers argument) and its first candidate (PEER_ARRAY[0], bare host:port). If the xceadmin wheel is missing (so no wrapper is generated), this case fails with a message naming /xce-build xceadmin, rather than being skipped. (4.1, 4.2)
• Ambient-variable exposure of existing tests: the live xceadmin tests and the xce-test-* skill scripts pass --peers, --controller or set masters explicitly (e.g. test_e2e_live_ha_push.py ~L361/~L488), so a developer's XCE_CONTROLLER_PEERS cannot change them (1.1). Two existing scripts start xce_scheduler with only --logfile: src/tools/test-version-startup-consistency.sh (~L302) and src/tools/test-dirty-fingerprint-startup-log-unchanged.sh. With the variable set, their scheduler takes the HA path instead of the single-controller path; they still pass, because they match the startup line by its version text, which the "HA cluster config" line also carries. Implementation adds unset XCE_CONTROLLER_PEERS at the top of both so they keep testing the path they were written for. Any new test that relies on the no-value default must do the same.
• xceadmin (test_cli.py): with XCE_CONTROLLER_PEERS and XCEADMIN_CONTROLLER both set and an explicit --controller, the flag wins. (4.3; this runs the CLI directly rather than through the generated wrapper, which is sufficient because the wrapper only exports variables and passes its arguments through unchanged)

This script is a new addition to the File Structure Plan and is intended to be re-run on any future change to these five components' startup argument handling, not a one-time implementation check.