myxce/.kiro/specs/controller-peers-env-fallback/design.md

# Design Document

## Overview

**Purpose**: This feature lets an operator set one environment variable, `XCE_CONTROLLER_PEERS`, once per session or deployment, instead of retyping `--peers` (controller, scheduler, worker), `set masters` (harness/xce_test) or `--controller` (xceadmin) on every invocation. It changes nothing about what any of these components do once their connection target is resolved — only where that target's default value comes from when the operator did not name one explicitly.

**Users**: Operators running local/interactive HPC cluster tools, and deploy/test scripts (e.g. HA soak setups) that currently either hardcode `--peers`/`--controller` into every generated command or accept the single-node/no-controller default they didn't actually want.

**Impact**: Five independently-built binaries each gain one additional, lowest-priority source for their existing connection-target resolution. Explicit `--peers`/`set masters` behavior, "omitted, no env var" behavior (apart from a log line's wording, 1.8), and xceadmin's multi-candidate failover order are unchanged; xceadmin's parsing of any `--controller` value, explicit or not, gains `name=` stripping and a clean malformed-entry error (2.2, 2.8). Two guards come with the new source: the controller refuses to start, early and with a message naming the variable, when an env-supplied list does not contain its own name (that list is Raft membership, not just a target list), and `deploy-ha.sh`'s generated `scripts/xceadmin` wrapper exports the new variable so an ambient value cannot redirect it.

### Goals
- One new environment variable, used only when no explicit value is given, across controller, scheduler and worker (via `--peers`), harness/xce_test (via `set masters`) and xceadmin (via `--controller`).
- An explicit value (`--peers`, `set masters`, `--controller`) always wins over the variable. Explicit `--peers`/`set masters` behave exactly as today; an explicit `--controller` gains only `name=` stripping and a clean malformed-entry error (2.2, 2.8).
- Zero behavior change when neither an explicit value nor the variable is set (apart from the startup log line's wording, 1.8).
- Reuse xceadmin's existing multi-candidate failover (`follow_redirects_any`, bug-1821, unmodified) and its existing `XCEADMIN_CONTROLLER` fallback; `resolve_controllers()` is extended only as 2.2/2.8 require.

### Non-Goals
- apiserver (YAML-config-based) and xcecli's `--url` (a different target) — untouched.
- Any change to `src/tools/deploy-ha.sh`'s generated daemon scripts' behavior (the `scripts/xceadmin` wrapper's added export, Requirement 4, is the one exception).
- Any change to what "no explicit value and no env var" does today for any component (only the log wording changes, 1.8).
- A TSAN/new-test-infrastructure investment for `main()`-level argument parsing where none exists today (see Testing Strategy).

## Boundary Commitments

### This Spec Owns
- A single shared C primitive (`xce_controller_peers_env()`, `libxce`) that reads `XCE_CONTROLLER_PEERS`, treating an unset or empty value as absent.
- The one-line integration of that primitive into controller, scheduler, and worker's existing `--peers`-gating `if`, and into harness's settings-table default computation at startup.
- A second shared C primitive, `xce_peers_find_identity()` (`libxce`), holding the one rule for "which candidate is me" (candidate `name` if set, else `hostname`, compared with the controller's name). The controller's two existing copies of that loop — the `my_priority` assignment in `cluster_init_phase1()` (`src/controller/cluster.c` ~L1784-1791) and the listen-port lookup in `cluster_open_listener()` (~L1890-1906) — are switched to call it, each keeping its own existing no-match handling, so the new startup check and both existing lookups can never disagree.
- The controller's startup identity check for an env-supplied candidate list (1.6).
- One added line in `deploy-ha.sh`'s generated `scripts/xceadmin` wrapper: `export XCE_CONTROLLER_PEERS="$PEERS_STR"` (Requirement 4).
- Extending xceadmin's existing `resolve_controllers()` to strip an optional `name=` prefix per candidate entry.
- Adding `"XCE_CONTROLLER_PEERS"` ahead of the existing `"XCEADMIN_CONTROLLER"` in each `--controller` option's `envvar=` list.
- Help-text and documentation updates naming the new variable (CLI `--peers`/`--controller` help text, `build-deployment.md`, `xce-deploy-ha`/`xce-test-multihost` skill docs).
- A repeatable regression script (`src/tools/verify-controller-peers-env.sh`) proving the above across all 5 components, intended for ongoing re-use, not a one-time implementation check.

### Out of Boundary
- apiserver's YAML-based peer configuration.
- xcecli's `--url` option.
- `deploy-ha.sh`'s generated daemon scripts' own invocations (they keep passing `--peers` explicitly). The `scripts/xceadmin` wrapper's one added export (Requirement 4) is the only `deploy-ha.sh` change.
- Changing what the controller does when an *explicit* `--peers` list lacks its name. Today `cluster_open_listener()` refuses (`ERROR` in the log, `myapp_run()` returns before any network I/O), and `main()` ignores `myapp_run()`'s result, so the process exits with status 0. That behavior is kept; see Revalidation Triggers.
- Any change to `xce_parse_peers()`'s or `resolve_controllers()`/`follow_redirects_any()`'s existing parsing/failover semantics beyond `resolve_controllers()`'s two additions: `name=` stripping (2.2) and a clean error for a malformed entry (2.8). Both apply to an explicit `--controller` value too.
- Refactoring any of the four C `main()` functions to become unit-testable in general — only the two new, genuinely independent primitives are unit-tested (see Testing Strategy).

### Allowed Dependencies
- All four C targets already link `libxce` (verified per-component `meson.build`) — the shared primitives may live there with no new build-graph edge.
- xceadmin's existing `resolve_controllers()`/`follow_redirects_any()` (`discovery.py`) and its existing `--controller` click options (`cli.py`) — reused and extended in place, not duplicated.

### Project Invariant Check
Walked against `xce-audit-invariants`' controller and scheduler dimensions:

- **Consensus/replication safety (controller) — applies.** For the
  controller, `--peers` is the Raft membership, and `my_priority` doubles
  as `self_idx` (`ha.c` ~L633, ~L2964). If the controller's name matches
  no candidate, `cluster_init_phase1()` (`cluster.c` ~L1783) falls back to
  the last index, which would give the node another peer's `self_idx`.
  That is already contained today: `cluster_open_listener()` (~L1902)
  refuses the same no-match case in `myapp_run()` step 3, before the
  listener, iomgr or any election starts, so no Raft traffic is sent
  under the wrong identity. An inherited value makes the case far more
  likely, though (a controller started without `--myname`, so its name is
  the hostname, or one meant to run standalone), and the existing refusal
  then blames "--peers", logs only to the log file, and exits with status
  0. Guard (1.6): when the list came from `XCE_CONTROLLER_PEERS` and
  `xce_peers_find_identity()` finds no match, `main.c` writes the error to
  stderr naming the variable and the remedies and returns 1, before
  `myapp_run()`. Because all three lookups now share one function, the
  new check cannot pass where the listener check would refuse, or the
  reverse. A bare single `host:port` value follows the same rule: its key
  is the hostname, so a controller joins a one-node HA cluster only when
  its name is literally that hostname; otherwise it refuses, rather than
  silently changing from standalone to HA mode.
- **Fail-loud, never self-exit (controller) — not affected.** The new exit
  happens in `main()` during argument validation, the same place and shape
  as the existing `--peers` parse error and even-peer-count rejection;
  no daemon thread, event loop or HA state exists yet.
- **Async I/O thread roles (controller) — not affected.** All reads of the
  environment happen in `main()` before any thread starts; the two
  `cluster.c` refactors are pure function calls in existing pre-election
  startup code (`cluster_init_phase1()`, `cluster_open_listener()`), with
  no blocking added.
- **Driver/simulation state-equivalence (scheduler) — not affected.**
  `--sim` (with or without `--interactive`) returns before the `--peers`
  block, so the variable can never reach replay mode; `--interactive`
  without `--sim` is a live run and takes the normal `--peers` path.

### Revalidation Triggers
- `xce_peers_find_identity()` is now the only "which candidate is me" rule for the controller (the 1.6 check, `my_priority`, listen port). Any future identity rule (e.g. matching on host:port) changes there, never in a re-added inline loop.
- If the explicit-`--peers` no-match case is ever moved to the same early `main()` check with a non-zero exit (a reasonable follow-up, out of scope here), drop the env-only condition on the 1.6 check instead of adding a second check.
- Any change to `xce_parse_peers()`'s accepted value format (e.g. a new candidate syntax) must be mirrored in `resolve_controllers()`'s parsing if the two are to stay consistent for `XCE_CONTROLLER_PEERS` values shared between C daemons and xceadmin.
- Any future change to harness's `settings[]` initialization order (e.g. rc-file loading moved earlier than today) must preserve "the env-var-derived default is computed before any explicit `set masters` can run."
- If `deploy-ha.sh`'s generated daemon scripts are ever updated to rely on `XCE_CONTROLLER_PEERS` instead of an explicit `--peers` (explicitly deferred here), re-check them against the baseline in this design's Out of Boundary, and re-check the 1.6 identity rule for each generated controller script.

## Architecture

### Existing Architecture Analysis

Each of the five components already resolves its own connection target independently, with no shared code path today:

| Component | Mechanism | Where "omitted" is decided | Existing fallback today |
|---|---|---|---|
| controller | `argp`, `OPT_PEERS` → `arguments.peers_str` | `if (arguments.peers_str)` (`main.c` ~L455) | Single-node primary (bug-1457). With a list, own entry found by name twice in `cluster.c` (~L1784 `my_priority`, ~L1890 listen port); no match is refused in `cluster_open_listener()` before any network I/O, exit status 0 |
| scheduler | `argp`, `'P'`/`--peers` → `arguments.peers` | `if (arguments.peers)` (`main.c` ~L411); **`--sim` returns before this point is reached** | "single-controller path" (logged) |
| worker | `getopt_long`, `OPT_PEERS` → `peers_str` | `if (peers_str)` (`main.c` ~L470) | Single-controller path via `config.remote_host`/`config.remote_port` (logged; same shape as controller/scheduler's own single-controller fallback, not an error path) |
| harness/xce_test | `settings[]` table, key `"masters"`, no CLI flag at all | `get_setting()`'s `value ?: defval` (`test.c` ~L991) | `defval = "localhost:9990"` |
| xceadmin | click `--controller` option, `envvar="XCEADMIN_CONTROLLER"` | click's own parameter resolution | `_DEFAULT_CONTROLLER = "localhost:9991"` |

All three C daemons that gate on a `--peers` argument already treat an invalid value as a hard startup error (`xce_parse_peers() != 0` → message + `return 1`), never a silent fallback — this feature preserves that for both an invalid explicit flag and an invalid `XCE_CONTROLLER_PEERS` value.

### Architecture Integration
- **Selected pattern**: two shared, stateless C primitives (an env reader used by four call sites, and an identity lookup used by three controller sites), plus one Python parsing extension consulted by click's own (unmodified) resolution order — no new service, process, or inter-component call is introduced.
- **Domain/feature boundaries**: each component keeps deciding its own resolved value; the new variable is only ever a *source*, never a coordinator.
- **Existing patterns preserved**: `xce_parse_peers()`'s `name=`-prefix convention; xceadmin's `resolve_controllers`/`follow_redirects_any` failover; each component's existing hard-error-on-invalid-value behavior.
- **New components rationale**: two new C functions. `xce_controller_peers_env()` is a bare env reader because harness's settings-table shape cannot reuse a "given `cli_value`, return resolved" signature the other three could share (see `research.md`, Design Decisions). `xce_peers_find_identity()` exists so the controller's new startup check and its two existing lookups (`my_priority`, listen port) share one matching rule.
- **Steering compliance**: no new build dependency; C standard/build system (Meson) and Python (click) usage unchanged.

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| C CLI / libxce | gnu17, existing `libxce` static lib | New `xce_controller_peers_env()` (4 call sites) and `xce_peers_find_identity()` (controller `main.c` + two `cluster.c` sites) | No new dependency |
| Python CLI | click (already a `xceadmin` dependency) | Native multi-`envvar` list resolution (no new click usage pattern) | Confirmed via direct read of the installed click version's `resolve_envvar_value()` |

## File Structure Plan

One new file (the regression script below); every other change is to an existing file — no new directories.

### New Files
- `src/tools/verify-controller-peers-env.sh` — repeatable regression script (see Testing Strategy → Scripted Regression Verification) driving the real built binaries of controller, scheduler, worker and harness/`xce_test`, plus `deploy-ha.sh`'s generated xceadmin wrapper, asserting on startup log/settings-dump/printed-environment output only (no live cluster required).

### Modified Files
- `src/include/xce/cluster.h` — declare `const char *xce_controller_peers_env(void);` and `int xce_peers_find_identity(const xce_cluster_config_t *cfg, const char *name);` alongside the existing `xce_parse_peers()` declaration.
- `src/lib/cluster.c` — implement `xce_controller_peers_env()`: read `getenv("XCE_CONTROLLER_PEERS")`, return it if non-empty, else `NULL`. Implement `xce_peers_find_identity()` with the matching rule moved verbatim from `src/controller/cluster.c` (identical at ~L1784-1791 and ~L1890-1900).
- `src/lib/testsuite/test_cluster.c` — new unit tests for `xce_controller_peers_env()` (unset, empty, non-empty) and `xce_peers_find_identity()` (named match, bare-hostname match, named entry not matched by its hostname, no match → -1, first of duplicate names wins, NULL name → -1, empty name matching an unnamed empty-hostname entry exactly as today).
- `src/controller/main.c` — at the existing `if (arguments.peers_str)` site (~L455), resolve `arguments.peers_str` against `xce_controller_peers_env()` before the check, remembering in a local `peers_from_env` whether the env var supplied it; on a parse or even-count failure, name the source (flag vs. environment variable) in the existing error message; after parsing, when `peers_from_env` is true and `xce_peers_find_identity(&myconf.peers_cfg, arguments.myname) < 0`, print the 1.6 error and `return 1`; extend the "No --peers flag" `INFO` (~L486) per 1.8; call `mylog_fini()` before every early `return` after `mylog_init()` (~L393) — the `get_myname()` failure (~L403), the `mylock_init()` failure (~L452), the `--peers` parse and even-count errors, the new 1.6 check, and the cert pre-flight (~L489-496); it is a no-op before `mylog_init()` and safe to call twice — because tlog writes the log file from a background thread with no `atexit` flush, so today those returns can lose already-logged lines (the scheduler already drains explicitly for the same reason); add `XCE_CONTROLLER_PEERS` and the "unset it to run standalone" note (1.7) to the `--peers` option's help text (~L85-87).
- `src/controller/cluster.c` — replace both inline identity loops with `xce_peers_find_identity(pcfg, c->myconf->name)`: in `cluster_init_phase1()` (~L1784-1791), keep the existing `if (pcfg->candidates_count > 0)` guard (~L1782; standalone mode leaves `my_priority = 0`) and replace only the loop inside it with `my_priority = idx >= 0 ? idx : pcfg->candidates_count - 1;`; in `cluster_open_listener()` (~L1890-1906), `matched = idx >= 0; my_port = matched ? pcfg->candidates[idx].port : 0;` followed by the existing refusal and default-port code. Same result for every input.
- `src/tools/deploy-ha.sh` — in the generated `scripts/xceadmin` wrapper heredoc (~L689-694), add `export XCE_CONTROLLER_PEERS="$PEERS_STR"` next to the existing `XCEADMIN_CONTROLLER` export, and extend the comment above it to say why (Requirement 4).
- `src/scheduler/src/main.c` — identical shape at the existing `if (arguments.peers)` site (~L411), without the identity check; name the source in both the parse-error and the `xce_cluster_config_validate()` failure messages (~L412-424, 1.4); "No --peers flag" `INFO` (~L435) per 1.8; help text update with 1.7's note (~L88). No change anywhere in the `--sim` path, which already returns before this point.
- `src/worker/src/main.c` — identical shape at the existing `if (peers_str)` site (~L470), without the identity check; name the source in both the parse-error and the `xce_cluster_config_validate()` failure messages (~L472-485, 1.4); "No --peers flag" log line (~L501) per 1.8; update the `-h`/usage `printf` (~L443-446) to mention `XCE_CONTROLLER_PEERS`, 1.7's note, and that `sudo` drops it unless preserved.
- `src/harness/test.c` — at startup, before the REPL loop and before rc-file loading, compute the `"masters"` entry's `defval` from `xce_controller_peers_env()` if set, else leave it at today's literal `"localhost:9990"`; extend the `USAGE`/`set`-command help text (~L403-415) to mention the new variable and 1.7's note.
- `src/xceadmin/src/xceadmin/discovery.py` — extend `resolve_controllers()`'s per-entry parsing to strip an optional `name=` prefix (mirrors `xce_parse_peers()`) before its existing `rsplit(":", 1)`; turn today's uncaught `ValueError` from `int(port_str)` (~L78) into a new `ControllerValueError` naming the bad entry (2.8).
- `src/xceadmin/src/xcecommon/errors.py` — add `ControllerValueError(XceAdminError)` next to the existing controller-reachability errors (`ConnectionTimeoutError`, `TooManyRedirectsError`, `NoPrimaryError`).
- `src/xceadmin/src/xceadmin/cli.py` — change `_CONTROLLER_ENV_VAR` (~L109), which all 4 `--controller` options use, to the list `["XCE_CONTROLLER_PEERS", "XCEADMIN_CONTROLLER"]`; rewrite each option's help string, which today interpolates `_CONTROLLER_ENV_VAR` with an f-string and would print a raw Python list once it becomes one, and the precedence comment above it (~L102-108). Add a helper `_controller_source_note()` that returns `" (from $NAME)"` when `click.get_current_context().get_parameter_source("controller")` is `ParameterSource.ENVIRONMENT` (NAME = the first non-empty variable in the list, the same rule click applies) and `""` otherwise; append it to the two existing `cannot connect to {controller}` messages (~L172, ~L265) only when the `OSError` is not a local file error (`e.filename is None`: socket errors carry no filename, while `read_bytes()` of a missing or unreadable cluster-bin file in `push_config()`/`probe_config()` (`push.py` ~L89, ~L143) raises a `FileNotFoundError`/`PermissionError` with `filename` set, which reaches the same `except OSError` branch today and must not be blamed on the variable), and, in the existing `except XceAdminError` handlers of the `--controller`-accepting commands, to the message when the error is one of `_CONTROLLER_ERRORS = (ControllerValueError, ConnectionTimeoutError, TooManyRedirectsError, NoPrimaryError)` (2.8). Errors unrelated to the controller value (e.g. a missing cluster-bin file, or `ClusterBinFileError` from `config build`) keep their text unchanged.
- `src/tools/test-version-startup-consistency.sh`, `src/tools/test-dirty-fingerprint-startup-log-unchanged.sh` — add `unset XCE_CONTROLLER_PEERS` near the top, so their scheduler run keeps taking the no-peers path they were written for (see Testing Strategy, ambient-variable exposure).
- `src/xceadmin/tests/test_discovery.py` — new unit tests for `resolve_controllers()`'s `name=`-stripping and its `ControllerValueError` on a non-numeric port.
- `src/xceadmin/tests/test_cli.py` — new unit tests for the new env-var precedence (mirrors the existing `test_controller_option_falls_back_to_xceadmin_controller_env_var`/`test_controller_flag_overrides_xceadmin_controller_env_var` tests) and for 2.8's messages.
- `.kiro/steering/build-deployment.md` — document `XCE_CONTROLLER_PEERS`: components, value format, precedence, the "unset it to run standalone" opt-out, the controller's own-name rule (1.6), and that `sudo` (default `env_reset`) and systemd services do not inherit it (`sudo --preserve-env=XCE_CONTROLLER_PEERS`, or `Environment=` in the unit).
- `.claude/skills/xce-deploy-ha/SKILL.md` — note the variable's availability, that the generated daemon scripts still pass `--peers`, and that `scripts/xceadmin` now also exports `XCE_CONTROLLER_PEERS` (update the existing ~L330 description and the ~L518 troubleshooting row's "check `$TOP/scripts/xceadmin`" hint).
- `.claude/skills/xce-test-multihost/SKILL.md` — same note, at its three existing `--peers`-mentioning locations (single-node deploy step, `xcecd_virtual` flag-table note, Prerequisites' generated-script list).

## System Flows

```mermaid
flowchart TD
    Start[Resolution point reached] --> HasFlag{Explicit flag or\nset command given?}
    HasFlag -- yes --> UseFlag[Use that value\nenv var not used as list]
    HasFlag -- no --> HasNewEnv{XCE_CONTROLLER_PEERS\nnon-empty?}
    HasNewEnv -- yes --> IsCtrl{Component is controller?}
    IsCtrl -- no --> UseNewEnv[Use it\nxceadmin strips name prefix]
    IsCtrl -- yes --> SelfListed{Own name matches\na candidate?}
    SelfListed -- yes --> UseNewEnv
    SelfListed -- no --> Refuse[Refuse to start\nname source and remedies]
    HasNewEnv -- no --> IsAdmin{Component is xceadmin?}
    IsAdmin -- yes --> HasLegacyEnv{XCEADMIN_CONTROLLER\nnon-empty?}
    IsAdmin -- no --> UseTodayDefault[Today's unchanged default\nstandalone / built-in literal]
    HasLegacyEnv -- yes --> UseLegacyEnv[Use it, unchanged behavior]
    HasLegacyEnv -- no --> UseTodayDefault
```

All three branches that resolve to a real candidate list (`UseFlag`, `UseNewEnv`, `UseLegacyEnv`) still pass through each component's existing validation (`xce_parse_peers()` and the controller's odd-count rule, unchanged; `resolve_controllers()`, extended per 2.2/2.8); an invalid value is a hard error either way, distinguished by source only in the error message text (1.4, 2.8).

This resolution point is reached exactly once, at process startup, for controller/scheduler/worker/xceadmin. For harness/`xce_test` it is reached on *every* connection attempt (`enter_ctx()`), since `set masters ...` can run at any point in an interactive/scripted session — the diagram represents one such evaluation, not a startup-only decision, for that component.

## Requirements Traceability

| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1 | Explicit candidate list wins, env var not used as candidate list | controller/scheduler/worker main.c; harness test.c (explicit `set masters`, interactive or rc-file) | existing `if (peers)` gate; `get_setting()` value-before-defval | Flowchart: `HasFlag` = yes |
| 1.2 | Env var used when no explicit list (harness: at each connection attempt) | controller/scheduler/worker main.c, harness test.c | `xce_controller_peers_env()`; harness `masters` `defval` | Flowchart: `HasNewEnv` → `UseNewEnv` |
| 1.3 | Unset/empty env var ⇒ unchanged today's behavior | all 4 | same call sites | Flowchart: `UseTodayDefault` |
| 1.4 | Parse/validation error (incl. controller even count) names the source | controller/scheduler/worker main.c, harness test.c | existing error `fprintf`s (parse, controller even-count)/`enter_ctx` message | n/a |
| 1.5 | `--peers` (harness: `set`) help text documents the variable | controller/scheduler/worker main.c, harness test.c | argp/getopt/settings help strings | n/a |
| 1.6 | Controller refuses an env-supplied list without its own name | controller main.c | `xce_peers_find_identity()` | Flowchart: `SelfListed` = no |
| 1.7 | Opt-out is "unset the variable"; help says so | controller/scheduler/worker main.c, harness test.c | help strings; `xce_controller_peers_env()` empty-as-unset | Flowchart: `HasNewEnv` = no |
| 1.8 | "No --peers" log line says the variable was unset or empty | controller/scheduler/worker main.c | existing `INFO`/log line text | Flowchart: `UseTodayDefault` |
| 2.1 | Derive controller list from env var | xceadmin cli.py, discovery.py | click `envvar=` list | Flowchart: `UseNewEnv` |
| 2.2 | Strip `name=` prefix, any source incl. explicit `--controller` | xceadmin discovery.py | `resolve_controllers()` | n/a |
| 2.3 | Try each candidate in order | xceadmin discovery.py | `resolve_controllers`/`follow_redirects_any` (unmodified) | n/a |
| 2.4 | Explicit `--controller` wins | xceadmin cli.py | click parameter resolution (unmodified) | Flowchart: `HasFlag` = yes |
| 2.5 | Legacy `XCEADMIN_CONTROLLER` fallback unchanged | xceadmin cli.py | click `envvar=` list order | Flowchart: `HasLegacyEnv` |
| 2.6 | Both unset ⇒ hardcoded default unchanged | xceadmin cli.py | `_DEFAULT_CONTROLLER` (unmodified) | Flowchart: `UseTodayDefault` |
| 2.7 | `--controller` help text documents the variable | xceadmin cli.py | click option help strings | n/a |
| 2.8 | Clean error for unreachable/malformed; names the env var source | xceadmin cli.py, discovery.py | `ControllerValueError`; `_controller_source_note()` | n/a |
| 3.1 | Steering doc describes the variable: components, format, precedence, opt-out (1.7), own-name rule (1.6), `sudo`/systemd not inheriting it | `build-deployment.md` | n/a | n/a |
| 3.2 | HA deploy guidance notes availability and the wrapper export | `xce-deploy-ha/SKILL.md` | n/a | n/a |
| 3.3 | Multi-host guidance notes availability | `xce-test-multihost/SKILL.md` | n/a | n/a |
| 4.1 | Wrapper exports the deployment's own list | `deploy-ha.sh` wrapper heredoc | `export XCE_CONTROLLER_PEERS="$PEERS_STR"` | n/a |
| 4.2 | Wrapper keeps `XCEADMIN_CONTROLLER` | `deploy-ha.sh` wrapper heredoc | existing export, unchanged | n/a |
| 4.3 | Explicit `--controller` still wins in the wrapper | xceadmin cli.py | click parameter resolution (unmodified) | Flowchart: `HasFlag` = yes |

## Components and Interfaces

| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|--------------|--------|---------------|-------------------|-----------|
| `xce_controller_peers_env()` | libxce / C | Read `XCE_CONTROLLER_PEERS`, empty-as-unset | 1.2, 1.3, 1.7 | none (P0) | Service |
| `xce_peers_find_identity()` | libxce / C | One rule for "which candidate is me" | 1.6 | none (P0) | Service |
| controller/scheduler/worker `--peers` call sites | C daemon CLI | Combine flag + env var, preserve existing validation/help; controller adds the 1.6 check | 1.1-1.8 | `xce_controller_peers_env` (P0), `xce_peers_find_identity` (P0, controller only) | — |
| harness `masters` setting default | C CLI (settings table) | Compute `defval` from env var at startup | 1.1-1.5, 1.7 | `xce_controller_peers_env` (P0) | State |
| `resolve_controllers()` extension | xceadmin / Python | Strip `name=` prefix per entry; clean error on a malformed entry | 2.1, 2.2, 2.8 | none (P0) | Service |
| xceadmin `--controller` options | xceadmin / Python (click) | Add `XCE_CONTROLLER_PEERS` ahead of legacy envvar; name the env source in errors | 2.1, 2.4-2.8 | `resolve_controllers` (P0), click (P0, external) | — |
| `scripts/xceadmin` wrapper export | `deploy-ha.sh` | Pin the wrapper to its own deployment | 4.1-4.3 | none | — |
| Documentation updates | Docs | Describe/announce the variable | 3.1-3.3 | none | — |

Only the three components introducing genuinely new logic — `xce_controller_peers_env()`, `xce_peers_find_identity()` and the `resolve_controllers()` extension — warrant a full contract block; the remaining rows are thin call-site wiring around existing, unchanged behavior.

### libxce

#### xce_controller_peers_env

| Field | Detail |
|-------|--------|
| Intent | Sole read point for `XCE_CONTROLLER_PEERS`, normalizing empty-string to absent |
| Requirements | 1.2, 1.3, 1.7 |

**Responsibilities & Constraints**
- Pure function: no side effects, no caching (the environment can only change between process invocations for these components, so no staleness concern).
- Treats an empty string identically to an unset variable, matching the same convention `resolve_controllers()`'s per-entry blank-skip already uses.

**Dependencies**
- Inbound: controller/scheduler/worker `main.c` `--peers` call sites (P0), harness `test.c` startup `masters`-default computation (P0).
- Outbound: none.

**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

##### Service Interface
```c
/* src/include/xce/cluster.h */
const char *xce_controller_peers_env(void);
```
- Preconditions: none.
- Postconditions: returns a pointer into the process environment (valid for the process lifetime) or `NULL`.
- Invariants: never returns a non-NULL empty string.

#### xce_peers_find_identity

| Field | Detail |
|-------|--------|
| Intent | Find the controller's own candidate in a parsed peers list |
| Requirements | 1.6 |

**Responsibilities & Constraints**
- Pure function over an already-parsed `xce_cluster_config_t`; no environment or global access.
- Matching rule, moved unchanged from `src/controller/cluster.c` (~L1784-1791 and ~L1890-1900, identical today): a candidate's key is its `name` when non-empty, else its `hostname`; the first candidate whose key equals `name` (compared with `strncmp(key, name, XCE_MAXNAME_LEN - 1)`) wins.
- Does not apply any default: "no match → last index" stays at the `my_priority` call site, "no match → refuse" at `cluster_open_listener()`, and the 1.6 check treats no match as an error.

**Dependencies**
- Inbound: controller `main.c` startup check (P0), controller `cluster.c` `my_priority` assignment and listen-port lookup (P0).
- Outbound: none.

**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

##### Service Interface
```c
/* src/include/xce/cluster.h */
int xce_peers_find_identity(const xce_cluster_config_t *cfg, const char *name);
```
- Preconditions: `cfg` non-NULL.
- Postconditions: returns the index (0..`candidates_count`-1) of the first matching candidate, or -1 when `name` is NULL or nothing matches. An empty `name` is compared like any other (today's loops do the same: it matches an unnamed entry with an empty hostname, e.g. from `:5`), so the invariant below holds for every input.
- Invariants: for every input with `candidates_count > 0`, `idx >= 0 ? idx : candidates_count - 1` equals what `cluster.c` computes for `my_priority` today (with no candidates, the call site's existing guard keeps `my_priority = 0`, and the helper returns -1); and for every input, `idx >= 0` equals the listener's `matched` today.

### xceadmin

#### resolve_controllers (extended)

| Field | Detail |
|-------|--------|
| Intent | Parse a `--controller`-shaped value into an ordered candidate list, now tolerant of `name=` prefixes |
| Requirements | 2.1, 2.2, 2.8 |

**Responsibilities & Constraints**
- Existing behavior (whitespace stripping, blank-entry skipping, bare-host default port) is unchanged.
- New: for each comma-separated entry, if it contains `=`, discard everything up to and including the first `=` before applying the existing `rsplit(":", 1)`.
- New: a port that is not an integer raises `ControllerValueError` (an `XceAdminError` subclass) naming the entry, instead of today's uncaught `ValueError`.
- Both changes apply to every caller, an explicit `--controller` value included.

**Dependencies**
- Inbound: xceadmin `--controller` options (P0), any future caller expecting `name=host:port` tolerance.
- Outbound: none new.

**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

##### Service Interface
```python
def resolve_controllers(controller: str) -> list[tuple[str, int]]: ...
```
- Preconditions: `controller` is a comma-separated string of `[name=]host[:port]` entries.
- Postconditions: returns an ordered list of `(host, port)` tuples, `name=` prefixes (if any) stripped; raises `ControllerValueError` on a malformed entry.
- Invariants: a bare `host:port` entry (today's only supported shape) parses identically to before.

**Implementation Notes**
- Integration: `follow_redirects_any()` is untouched — it only ever consumes `resolve_controllers()`'s output.
- Validation: existing `test_discovery.py` coverage for bare-host/list parsing must stay green; new tests add `name=`-prefixed and mixed-prefix cases.
- Risks: none identified (see `research.md` Risks & Mitigations for the `=`-in-hostname non-issue).

## Error Handling

### Error Strategy
Every existing hard-error-on-invalid-value behavior is preserved exactly, with the value's source named in the message text (1.4). The one new error is the controller's own-name check for an env-supplied list (1.6).

### Error Categories and Responses
- **Invalid candidate list, from an explicit `--peers` value**: unchanged exit behavior; message names `--peers` as the source (most already do; scheduler's and worker's `xce_cluster_config_validate()` messages gain it).
- **Invalid candidate list, from an explicit `set masters`**: unchanged behavior; `enter_ctx()`'s message names the source as described in the next bullet.
- **Invalid or unreachable value, from an explicit `--controller`**: today's unreachable messages unchanged; a malformed entry now gives the 2.8 clean error instead of a traceback. No source note is added, since the operator typed the value.
- **Invalid candidate list, from `XCE_CONTROLLER_PEERS`**: same parse or validation failure (including the controller's even-count rejection), same exit behavior, message additionally names the environment variable as the source (controller/scheduler/worker: adjust the existing `fprintf`; harness: `enter_ctx()`'s existing "invalid 'masters' setting" message, extended to say "default (from XCE_CONTROLLER_PEERS)" when the `"masters"` entry's `settings[i].value == NULL` (no `set masters` ran, reusing `get_setting()`'s own value-before-defval precedence) **and** `xce_controller_peers_env() != NULL`; "explicit `set masters`" when `value != NULL`; and today's wording otherwise (the built-in `localhost:9990` literal, which cannot fail to parse today, is then never blamed on the variable).
- **Controller name not in an env-supplied list (1.6)**: `main.c` prints, to stderr, e.g. `xce_controller: XCE_CONTROLLER_PEERS (a=h1:1,b=h2:2) has no candidate named 'node7'; pass --myname <one of: a, b> or unset XCE_CONTROLLER_PEERS to run standalone`, then `return 1` before the cert pre-flight and `myapp_run()`. This replaces, for the env-var case only, the later `cluster_open_listener()` refusal, which blames "--peers", writes only to the log file, and exits with status 0. Candidates without a `name=` are listed by hostname.
- **xceadmin, unreachable candidates or a malformed entry (2.8)**: `click.ClickException` (exit 1) with today's text (`cannot connect to {controller}: ...` for `OSError`, or the `e.message` of `ConnectionTimeoutError`/`TooManyRedirectsError`/`NoPrimaryError`), or `invalid --controller entry '...'` for `ControllerValueError`, followed by ` (from $XCE_CONTROLLER_PEERS)` or ` (from $XCEADMIN_CONTROLLER)` when the value came from the environment. Today a malformed port is an uncaught `ValueError` traceback.

### Monitoring
No new logging/metrics infrastructure; each component's existing startup log line (e.g. scheduler's `INFO("HA cluster config: ...")` / `INFO("No --peers flag; ...")`) is extended only to remain accurate about which source supplied the value it's reporting. The three "No --peers flag" lines (controller `main.c` ~L486, scheduler ~L435, worker ~L501) become "No --peers flag and XCE_CONTROLLER_PEERS unset or empty; ..." (1.8). This is the operator's only clue when `sudo`'s default `env_reset` or a systemd unit drops the variable — the main risk for the production worker, which `check_user()` requires to run as root (`deploy-ha.sh` ~L470 starts it with `sudo`).

## Testing Strategy

### Unit Tests
- `xce_controller_peers_env()` (`src/lib/testsuite/test_cluster.c`): unset → `NULL`; empty string → `NULL`; non-empty → returned unchanged. (1.2, 1.3, 1.7)
- `xce_peers_find_identity()` (`src/lib/testsuite/test_cluster.c`): the cases listed in File Structure Plan, each written first as a failing test. The existing controller tests that go through `my_priority` and `cluster_open_listener()` must stay green after `cluster.c` switches to the helper. (1.6)
- `resolve_controllers()` extension (`src/xceadmin/tests/test_discovery.py`): single `name=host:port`; mixed bare/named multi-candidate list; existing bare-host cases still pass unchanged. (2.2)
- xceadmin `--controller` option precedence (`src/xceadmin/tests/test_cli.py`): `XCE_CONTROLLER_PEERS` set alone; both `XCE_CONTROLLER_PEERS` and `XCEADMIN_CONTROLLER` set (new wins); explicit `--controller` overrides both; neither set (existing default). (2.1, 2.4-2.6) For 2.3: with
`XCE_CONTROLLER_PEERS="a=127.0.0.1:1,b=127.0.0.1:2,127.0.0.1:3"` and no
`--controller`, patch `follow_redirects` (the per-candidate call inside
`follow_redirects_any`) so the first candidate raises a connection error
and the second returns a canned reply, and assert the candidates arrive
in env order with `name=` stripped and that the second one is tried next.
(2.3) For 2.8: an unreachable value from each of `XCE_CONTROLLER_PEERS`, `XCEADMIN_CONTROLLER` and an explicit `--controller` (message names the variable in the first two only), and a non-numeric port giving a clean error rather than a traceback — run with a readable cluster-bin file too, since `config push` reads the file before resolving controllers: once as an explicit `--controller a=h:x` (message quotes the entry, no source note) and once as `XCE_CONTROLLER_PEERS=a=h:x` with no `--controller` (message quotes the entry and ends `(from $XCE_CONTROLLER_PEERS)`). Run the unreachable cases through `config push` with a real, readable cluster-bin file (`push_config()` reads it before resolving controllers, so a missing file raises `FileNotFoundError` first — add one case asserting that error gets no source note even with `XCE_CONTROLLER_PEERS` set), or patch out the retry sleep, since the `scheduler record` commands retry `NoPrimaryError` 8 times with a 2 s pause (`scheduler.py` ~L131-132). All existing `test_cli.py` cases that assert `XCEADMIN_CONTROLLER`-only or no-env-var behavior (e.g. `test_controller_option_falls_back_to_xceadmin_controller_env_var`) must explicitly `monkeypatch.delenv("XCE_CONTROLLER_PEERS", raising=False)`, so the new variable's introduction cannot make them order-dependent on ambient environment state.

### Scripted Regression Verification
None of controller/scheduler/worker/harness's `main()`-level argument or
settings-table resolution has existing unit-test coverage today (confirmed
via source search; `test_cluster_parse.c` tests a standalone duplicated
reference parser, not live `main()` wiring) — refactoring any of the four
into a unit-testable shape is out of this feature's boundary (see
Non-Goals). Rather than one-time manual checking, this is verified by a new
repeatable script, `src/tools/verify-controller-peers-env.sh`, following
`xce-test-multihost`'s established pattern of driving real built binaries
and asserting on their output — but deliberately lightweight: it asserts on
each component's **startup log/settings-dump line**, not on a live,
reachable cluster, since resolving the candidate list happens, and is
observable, entirely before any connection attempt succeeds or fails.
The worker cases use `xcecd_virtual`, since the production `xcecd`
requires root (`check_user()`), and grep the worker's captured **stdout**,
not `xcecd-<name>.log`: its "HA cluster config" / "No --peers flag" lines
(`worker/src/main.c` ~L493-503) are logged before its log file is set up
(~L529-544), so the logging library still writes them to stdout. The
controller and scheduler set up their log files earlier (controller
`main.c` ~L393, scheduler ~L332), so their cases grep the log file. Controller cases always use an odd
candidate count and pass `--myname` naming a candidate unless the case
is about the 1.6 check, so the even-count rule and 1.6 do not mask what
the case is testing.
Every daemon run is isolated in a fresh scratch directory: the controller
always gets `--workdir`, `--certdir` and `--logfile` there (otherwise it
defaults to `/tmp/xce` and `/tmp/xce_controller.log`, `src/include/config.h:22`,
`main.c:312`, and could touch a local cluster's state), the scheduler
`--logfile`, and the worker `-l`. With an empty certificate directory the
controller exits at the certificate pre-flight (`main.c` ~L489), right
after logging the line under test (~L476/~L486) and after the 1.6 check,
so it never reaches `myapp_run()`; the `mylog_fini()` added before that
return (File Structure Plan, `main.c`) is what guarantees the line is in
the log file when the script reads it. Scheduler and worker runs are killed
after a short timeout once the asserted line has appeared. Two things a
scratch directory cannot isolate: with no peers list the scheduler and
worker still try their default controller (`localhost:9990`), and the
worker always uses the hardcoded `/var/run/.xcecd.sock`. The script
therefore checks first that nothing listens on port 9990 and that the
socket does not exist, and fails with a message (rather than running)
when a local cluster is up.
- controller/scheduler/worker: with `XCE_CONTROLLER_PEERS` set (e.g.
  `a=localhost:1,b=localhost:2,c=localhost:3`; controller also
  `--myname a`) and no `--peers`, start briefly and grep the log (worker:
  stdout) for the line each already logs today for an explicit `--peers`
  value — controller `HA mode: N controller candidates, identity=...`
  (`main.c` ~L476), scheduler and worker `HA cluster config: N
  candidate(s), ...` (scheduler ~L431, worker ~L493) — asserting `N`
  matches the env var's content — no reachable target required. (1.2)
- Same three, with both `XCE_CONTROLLER_PEERS` set to one value (3
  candidates) and `--peers` set to a different one (1 candidate, which the
  controller's `--myname` names); assert the log reflects the `--peers`
  value, not the env var's (worker: stdout). (1.1)
- Same three, with neither set; assert the "No --peers flag" (or
  worker's "single-controller path", on stdout) log line now also says
  `XCE_CONTROLLER_PEERS` was unset or empty, and that behavior is otherwise
  unchanged. (1.3, 1.8)
- scheduler with `--sim <nonexistent file>` and `XCE_CONTROLLER_PEERS`
  set: assert stderr shows `xce_scheduler: --sim startup failed`, which only
  the `--sim` branch prints (proving that branch ran, so the case cannot
  pass by failing earlier), and that the process never logs an
  "HA cluster config"/"No --peers flag" line at all
  (the `--sim` path returns before that code is reached — this is a
  by-construction guarantee already, not new logic; the script asserts it
  stays true). (Non-Goals cross-check)
- harness/`xce_test` (always with `--norc`, so a developer's own rc file
  cannot change the result): run non-interactively with a script containing only
  `set` (no arguments) — `cmd_set`'s existing argc==1 branch prints every
  setting's current value/default with **no connection attempt** — first
  with `XCE_CONTROLLER_PEERS` set and no prior `set masters`, asserting the
  printed `masters` default reflects the env var; then with an explicit
  `set masters ...` line before the bare `set`, asserting the printed
  *value* (not default) reflects the explicit override. (1.1, 1.2)
- For each of controller, scheduler and worker, trigger a parse error
  once with an invalid explicit `--peers` and once with an invalid
  `XCE_CONTROLLER_PEERS` and no `--peers`; assert on **stderr** (all three
  write these errors with `fprintf(stderr, ...)`) that the two messages
  name different sources. For the controller, repeat with a 2-candidate
  value from each source (even-count message); for scheduler and worker,
  repeat with `a=localhost:0`, which parses but fails
  `xce_cluster_config_validate()` (port 0). (1.4)
- harness/`xce_test --norc`, each script ending in a connecting command
  (`client`): (a) `set masters bad` with `XCE_CONTROLLER_PEERS` unset →
  message says explicit `set masters`; (b) no `set masters`,
  `XCE_CONTROLLER_PEERS=bad` → message says default from
  `XCE_CONTROLLER_PEERS`; (c) `set masters bad` with
  `XCE_CONTROLLER_PEERS` set to a valid value → still says explicit
  `set masters` (explicit wins, 1.1). (1.1, 1.4)
- controller with `XCE_CONTROLLER_PEERS=a=localhost:1,b=localhost:2,c=localhost:3`:
  with `--myname zz`, assert a non-zero exit and the 1.6 message (names
  `zz`, the variable, and the remedies); with `--myname b`, assert the
  "HA mode: 3 controller candidates" line and no 1.6 message; with
  `--myname zz --peers a=localhost:1,b=localhost:2,c=localhost:3`, assert
  no 1.6 message (explicit-flag behavior unchanged). (1.1, 1.6)
- controller with `XCE_CONTROLLER_PEERS=""` and no `--peers`: assert the
  "No --peers flag ... running as single-node primary" line. (1.7)
- Help text (1.5, 1.7): run `xce_controller --help`, `xce_scheduler --help`
  and `xcecd_virtual -h`, and feed `help set` to `xce_test --norc`; assert
  each output names `XCE_CONTROLLER_PEERS` and says that unsetting or
  emptying it opts out for one invocation. For xceadmin, `test_cli.py`
  asserts the same name in each `--controller` option's `--help` output
  (2.7).
- `deploy-ha.sh` wrapper: run `deploy-ha.sh` with the built controller
  binary and a peers list on free high ports (it refuses ports already in
  use, `deploy-ha.sh` ~L204-225) into a scratch `$TOP` (generation only,
  nothing started), copy
  `$TOP/scripts/xceadmin` with its final `exec` line replaced by
  `printenv XCE_CONTROLLER_PEERS XCEADMIN_CONTROLLER`, and run the copy
  with `XCE_CONTROLLER_PEERS=bogus:1` in the caller's environment; assert
  it prints the deployment's own list in `deploy-ha.sh`'s `PEERS_STR` form (`ctrl-1=host:port,ctrl-2=...`, not the raw peers argument) and its first candidate (`PEER_ARRAY[0]`, bare `host:port`). If
  the xceadmin wheel is missing (so no wrapper is generated), this case
  fails with a message naming `/xce-build xceadmin`, rather than being
  skipped. (4.1, 4.2)
- Ambient-variable exposure of existing tests: the live xceadmin tests
  and the `xce-test-*` skill scripts pass `--peers`, `--controller` or
  `set masters` explicitly (e.g. `test_e2e_live_ha_push.py` ~L361/~L488),
  so a developer's `XCE_CONTROLLER_PEERS` cannot change them (1.1). Two
  existing scripts start `xce_scheduler` with only `--logfile`:
  `src/tools/test-version-startup-consistency.sh` (~L302) and
  `src/tools/test-dirty-fingerprint-startup-log-unchanged.sh`. With the
  variable set, their scheduler takes the HA path instead of the
  single-controller path; they still pass, because they match the
  startup line by its version text, which the "HA cluster config" line
  also carries. Implementation adds `unset XCE_CONTROLLER_PEERS` at the top
  of both so they keep testing the path they were written for. Any new
  test that relies on the no-value default must do the same.
- xceadmin (`test_cli.py`): with `XCE_CONTROLLER_PEERS` and
  `XCEADMIN_CONTROLLER` both set and an explicit `--controller`, the flag
  wins. (4.3; this runs the CLI directly rather than through the
  generated wrapper, which is sufficient because the wrapper only exports
  variables and passes its arguments through unchanged)

This script is a new addition to the File Structure Plan and is intended to
be re-run on any future change to these five components' startup argument
handling, not a one-time implementation check.