myxce/.kiro/specs/controller-peers-env-fallback/brief.md

Brief: controller-peers-env-fallback

Problem

Operators and deploy/test scripts must retype the full --peers candidate list (or, for xceadmin, --controller) on every single invocation of every XCE component -- there is no way to set a default once per session/ environment. xceadmin already solved this for its own --controller flag (XCEADMIN_CONTROLLER env var, bug-1566), but the four --peers-based daemons (controller, scheduler, worker, harness/xce_test) have no equivalent: --peers is optional (omitting it runs standalone/non-HA), but there is no way to supply a default candidate list via the environment for someone who wants the convenience of not retyping it, as opposed to someone deliberately omitting it to opt out of HA.

Current State

• xceadmin (Python, click): --controller already falls back to $XCEADMIN_CONTROLLER, then a hardcoded localhost:9991 default. Fully implemented and tested (bug-1566), used consistently by every --controller-accepting subcommand (config push, config probe, scheduler record start/stop/status) -- confirmed via source read, src/xceadmin/src/xceadmin/cli.py.
• Correction, discovered mid-session (bug-1821, 2026-09-17): --controller is NOT single-target-only -- it already accepts a comma-separated list of bare host:port candidates (xceadmin.discovery.resolve_controllers() -> follow_redirects_any()), tried in order until one is reachable, mirroring the client-side master-rotation failover xcecli/xce_test already use. resolve_controllers() does a plain part.rsplit(":", 1) per comma-separated entry with no name= prefix handling -- a name=host:port-shaped entry (the --peers/xce_parse_peers() format) would mis-parse as a literal (unresolvable) hostname "name=host". This directly shapes the merge design below: xceadmin cannot consume a name=host:port,... value as-is, but stripping the name= prefixes and handing the WHOLE resulting candidate list to its own existing resolve_controllers/follow_redirects_any gets proper multi-candidate failover for free, rather than arbitrarily trusting just one entry.
• controller, scheduler, worker (C, argp/getopt_long) and harness/xce_test (C, custom flagspec.c parser) each take an optional --peers <name=host:port,...> flag with no environment-variable fallback at all -- confirmed via source grep, zero getenv() calls for this anywhere in these four components.
• All four of these C binaries already link libxce (src/lib/) -- confirmed via each component's meson.build (libxce_dep).
• apiserver (Go) takes its controller peer list from a YAML config file (app.yaml's backend.peers), not a CLI flag -- no --peers/ --controller equivalent exists there.
• xcecli (Go) takes --url to reach the apiserver -- a different target ("which apiserver") than "which controller(s)" entirely.

Desired Outcome

• A single, merged environment variable, XCE_CONTROLLER_PEERS, provides the default target for all five components. Its value format is exactly what xce_parse_peers() already accepts (src/lib/cluster.c): either a bare host:port (single candidate, no name) or a full name=host:port,name=host:port,... candidate list -- confirmed by reading xce_parse_peers() itself, which already tolerates both shapes with no name required (a strict superset relationship, not an assumption).
• controller, scheduler, worker, harness/xce_test: when --peers is not passed, use $XCE_CONTROLLER_PEERS as-is (no extra parsing needed -- it's already exactly xce_parse_peers()'s native input shape). Precedence: --peers flag > $XCE_CONTROLLER_PEERS > today's no-peers behavior (standalone/non-HA).
• xceadmin: when --controller is not passed, read the same $XCE_CONTROLLER_PEERS, strip any name= prefix from each comma-separated entry (producing the bare host:port,host:port,... shape resolve_controllers() already natively accepts), and feed the WHOLE resulting string through xceadmin's existing --controller value path -- i.e. functionally as if the user had typed --controller "host1:port1,host2:port2,..." themselves. This inherits xceadmin's own already-tested multi-candidate failover (resolve_controllers + follow_redirects_any, bug-1821) for free, rather than arbitrarily trusting a single entry regardless of whether it's alive. The existing XCEADMIN_CONTROLLER env var (bare host:port or comma-list, unchanged) keeps working as a lower-priority legacy fallback -- no breaking change to the already-shipped, tested, deploy-ha.sh-integrated behavior. Precedence: --controller flag > $XCE_CONTROLLER_PEERS (name= stripped, full candidate list) > $XCEADMIN_CONTROLLER (legacy) > localhost:9991 bare default.
• This merges what were originally proposed as two separate new env vars (XCE_CONTROLLER_PEERS and XCE_CONTROLLER) into one, since the peers-list format is a strict superset of the single-target format -- reduces the operator-facing surface to one variable to set, shared by every component.

Approach

Add one small shared helper in src/lib/ (all four C targets already link libxce) that resolves "the CLI-provided value, or else $XCE_CONTROLLER_PEERS" -- called once by each of controller/main.c, scheduler/src/main.c, worker/src/main.c, and harness/test.c right after their own (unchanged) argp/getopt_long/flagspec parsing of --peers, replacing four near-identical inline duplications with one shared, TDD-covered implementation. This helper does no format-specific parsing itself -- it just returns whichever raw string wins precedence, which then flows into each component's existing xce_parse_peers() call exactly as --peers would have.

xceadmin's click-based --controller option gains a new small conversion helper (Python): given $XCE_CONTROLLER_PEERS's raw value, strip an optional name= prefix from EVERY comma-delimited entry (not just the first) and return the resulting bare host:port,host:port,... string unchanged in shape from what --controller/resolve_controllers() already parses today. Wired in ahead of the existing envvar="XCEADMIN_CONTROLLER" in click's precedence chain (click's envvar= only reads a raw string per name, so the name-stripping conversion needs a small callback=/default resolution wrapper around the option rather than relying on click's built-in multi-envvar list mechanism, since the two source vars now have different value shapes and only one of them needs transformation). No change to resolve_controllers()/follow_redirects_any() themselves -- they already do exactly the right thing once handed a bare-host candidate list, regardless of whether that list came from an explicit --controller flag or this new env-var conversion.

Scope

• In:
  • New XCE_CONTROLLER_PEERS env var fallback for --peers in controller, scheduler, worker, harness/xce_test (used as-is, no transformation).
  • The same XCE_CONTROLLER_PEERS env var, name=-stripped and handed to xceadmin's existing resolve_controllers/follow_redirects_any multi-candidate failover, as a new, higher-precedence fallback for xceadmin's --controller option, ahead of the existing XCEADMIN_CONTROLLER (kept working, noted as legacy).
  • TDD test coverage for all five call sites (4 C components' own testsuites + xceadmin's existing click-based test file), including the name=-stripping conversion specifically (bare host:port input, name=host:port input, multi-candidate input with a mix of named and bare entries, and that the resulting failover still tries every candidate in order).
• Out:
  • apiserver (YAML-config-based, no CLI flag for this) -- explicit user decision.
  • xcecli's --url (different target: apiserver, not controller) -- explicit user decision.
  • src/tools/deploy-ha.sh and its generated wrapper scripts -- stay exactly as they are today (explicit --peers/--controller baked into every generated script); this feature is purely additive, available for other use, not adopted into the deploy skill itself -- explicit user decision.
  • Any change to what happens today when --peers/--controller is omitted and no env var is set either (standalone/bare-default behavior is unchanged).

Boundary Candidates

• Shared libxce helper -- one implementation, one seam all 4 C daemons call through.
• xceadmin's click option definitions -- already-existing seam in cli.py, just add an env var name to the existing list.
• Per-component test coverage -- controller/scheduler/worker/harness each already have their own testsuite; this feature adds one test per component to each, not new test infrastructure.

Out of Boundary

• apiserver and xcecli, per user decision (different flag shapes/targets, not part of this feature).
• deploy-ha.sh adoption, per user decision (kept purely additive; no change to existing deploy/test skill behavior).
• Any redesign of what "no --peers and no env var" means today (standalone mode semantics untouched).

Upstream / Downstream

• Upstream: none -- this is additive to existing, independent argument-parsing in each of the 5 components.
• Downstream: future deploy/test tooling (including, later, an opt-in update to deploy-ha.sh, explicitly deferred here) could adopt XCE_CONTROLLER_PEERS once this ships; not a requirement of this spec.

Existing Spec Touchpoints

• Extends: none directly -- admin-config-tool is where xceadmin's CLI (including XCEADMIN_CONTROLLER) originally shipped, but this change is additive/small enough not to require reopening that spec; noted here for cross-reference only. xceadmin.discovery.resolve_controllers()/ follow_redirects_any() (bug-1821, 2026-09-17, not tied to any open spec) is reused as-is, not modified -- this feature only needs to produce a value in the shape those functions already accept.
• Adjacent: structure.md/build-deployment.md steering (component build/CLI conventions) -- no changes expected, just awareness.

Constraints

• TDD mandatory per project rules: a test in each of the 4 C components' testsuite/ dirs, and an addition to xceadmin's existing click-based test coverage, before/alongside the implementation.
• Must not change behavior when --peers/--controller IS explicitly passed on the command line (env var is purely a fallback, never an override).
• Open question, not resolved in discovery: what happens for an operator who has XCE_CONTROLLER_PEERS set globally in their shell but wants ONE specific invocation to run standalone/non-HA? Needs an explicit answer at requirements/design time (e.g. is "unset the env var for that one invocation" considered sufficient, or does a component need an explicit opt-out?).
• Naming: a single XCE_CONTROLLER_PEERS env var shared by all five components (merged from an earlier two-variable proposal, per explicit user decision, once xce_parse_peers()'s existing bare-host:port tolerance confirmed the peers-list format is a strict superset of the single-target format); XCEADMIN_CONTROLLER stays supported as a legacy, lower-priority fallback, not removed.
• The name=-stripping conversion (xceadmin side only) must be exercised by tests covering: bare host:port, name=host:port, a multi-candidate list with a mix of named and bare entries, and that the converted value still drives resolve_controllers/follow_redirects_any's existing in-order failover across every candidate (not just the first) -- this is new logic (not just a new envvar= name) and is the one place this design has room for a subtle bug (e.g. mis-splitting on an = inside a hostname, though none of xce_parse_peers()'s own accepted characters make that ambiguous in practice).