myxce/.kiro/specs/controller-peers-env-fallback/design.md

# Design Document

## Overview

**Purpose**: This feature lets an operator set one environment variable, `XCE_CONTROLLER_PEERS`, once per session or deployment, instead of retyping `--peers` (controller, scheduler, worker, harness/xce_test) or `--controller` (xceadmin) on every invocation. It changes nothing about what any of these components do once their connection target is resolved — only where that target's default value comes from when the operator did not name one explicitly.

**Users**: Operators running local/interactive HPC cluster tools, and deploy/test scripts (e.g. HA soak setups) that currently either hardcode `--peers`/`--controller` into every generated command or accept the single-node/no-controller default they didn't actually want.

**Impact**: Five independently-built binaries each gain one additional, lowest-priority source for their existing connection-target resolution. No existing explicit-flag behavior, no existing "omitted, no env var" behavior, and no existing multi-candidate failover behavior changes.

### Goals
- One new environment variable, consulted only when the corresponding CLI option is not given, across controller, scheduler, worker, harness/xce_test (via `--peers`) and xceadmin (via `--controller`).
- Zero behavior change when the CLI option is given, and zero behavior change when neither the option nor the variable is set.
- Reuse xceadmin's existing multi-candidate failover (`resolve_controllers`/`follow_redirects_any`, bug-1821) and its existing `XCEADMIN_CONTROLLER` fallback unmodified.

### Non-Goals
- apiserver (YAML-config-based) and xcecli's `--url` (a different target) — untouched.
- Any change to `src/tools/deploy-ha.sh`'s generated scripts' own behavior.
- Any change to what "no `--peers`/`--controller` and no env var" means today for any component.
- A TSAN/new-test-infrastructure investment for `main()`-level argument parsing where none exists today (see Testing Strategy).

## Boundary Commitments

### This Spec Owns
- A single shared C primitive (`xce_controller_peers_env()`, `libxce`) that reads `XCE_CONTROLLER_PEERS`, treating an unset or empty value as absent.
- The one-line integration of that primitive into controller, scheduler, and worker's existing `--peers`-gating `if`, and into harness's settings-table default computation at startup.
- Extending xceadmin's existing `resolve_controllers()` to strip an optional `name=` prefix per candidate entry.
- Adding `"XCE_CONTROLLER_PEERS"` ahead of the existing `"XCEADMIN_CONTROLLER"` in each `--controller` option's `envvar=` list.
- Help-text and documentation updates naming the new variable (CLI `--peers`/`--controller` help text, `build-deployment.md`, `xce-deploy-ha`/`xce-test-multihost` skill docs).
- A repeatable regression script (`src/tools/verify-controller-peers-env.sh`) proving the above across all 5 components, intended for ongoing re-use, not a one-time implementation check.

### Out of Boundary
- apiserver's YAML-based peer configuration.
- xcecli's `--url` option.
- `deploy-ha.sh`'s generated scripts' own invocations (documentation may *mention* the variable; the generated scripts keep passing `--peers`/`--controller` explicitly).
- Any change to `xce_parse_peers()`'s or `resolve_controllers()`/`follow_redirects_any()`'s existing parsing/failover semantics beyond the one `name=`-stripping addition.
- Refactoring any of the four C `main()` functions to become unit-testable in general — only the new, genuinely independent primitive is unit-tested (see Testing Strategy).

### Allowed Dependencies
- All four C targets already link `libxce` (verified per-component `meson.build`) — the shared primitive may live there with no new build-graph edge.
- xceadmin's existing `resolve_controllers()`/`follow_redirects_any()` (`discovery.py`) and its existing `--controller` click options (`cli.py`) — reused and extended in place, not duplicated.

### Project Invariant Check
This feature's controller/scheduler changes touch only pre-startup argument
resolution (`main.c`, before `sched_init()`/`myapp_run()`/any daemon thread,
event loop, or Raft/HA state machine is created) — it does not intersect
`xce-audit-invariants`' controller dimensions (async I/O thread-role
placement, fail-loud/never-self-exit process lifecycle, consensus/
replication safety) or its scheduler dimension (driver/simulation
state-equivalence). No audit re-check is required beyond this design.

### Revalidation Triggers
- Any change to `xce_parse_peers()`'s accepted value format (e.g. a new candidate syntax) must be mirrored in `resolve_controllers()`'s parsing if the two are to stay consistent for `XCE_CONTROLLER_PEERS` values shared between C daemons and xceadmin.
- Any future change to harness's `settings[]` initialization order (e.g. rc-file loading moved earlier than today) must preserve "the env-var-derived default is computed before any explicit `set masters` can run."
- If `deploy-ha.sh` is ever updated to adopt `XCE_CONTROLLER_PEERS` (explicitly deferred here), its generated scripts' current behavior baseline (documented in this design's Out of Boundary) is the thing to re-check against.

## Architecture

### Existing Architecture Analysis

Each of the five components already resolves its own connection target independently, with no shared code path today:

| Component | Mechanism | Where "omitted" is decided | Existing fallback today |
|---|---|---|---|
| controller | `argp`, `OPT_PEERS` → `arguments.peers_str` | `if (arguments.peers_str)` (`main.c` ~L455) | Single-node primary (bug-1457) |
| scheduler | `argp`, `'P'`/`--peers` → `arguments.peers` | `if (arguments.peers)` (`main.c` ~L403); **`--sim` returns before this point is reached** | "single-controller path" (logged) |
| worker | `getopt_long`, `OPT_PEERS` → `peers_str` | `if (peers_str)` (`main.c` ~L470) | Single-controller path via `config.remote_host`/`config.remote_port` (logged; same shape as controller/scheduler's own single-controller fallback, not an error path) |
| harness/xce_test | `settings[]` table, key `"masters"`, no CLI flag at all | `get_setting()`'s `value ?: defval` (`test.c` ~L991) | `defval = "localhost:9990"` |
| xceadmin | click `--controller` option, `envvar="XCEADMIN_CONTROLLER"` | click's own parameter resolution | `_DEFAULT_CONTROLLER = "localhost:9991"` |

All three C daemons that gate on a `--peers` argument already treat an invalid value as a hard startup error (`xce_parse_peers() != 0` → message + `return 1`), never a silent fallback — this feature preserves that for both an invalid explicit flag and an invalid `XCE_CONTROLLER_PEERS` value.

### Architecture Integration
- **Selected pattern**: one shared, stateless C primitive consulted independently by four call sites, plus one Python parsing extension consulted by click's own (unmodified) resolution order — no new service, process, or inter-component call is introduced.
- **Domain/feature boundaries**: each component keeps deciding its own resolved value; the new variable is only ever a *source*, never a coordinator.
- **Existing patterns preserved**: `xce_parse_peers()`'s `name=`-prefix convention; xceadmin's `resolve_controllers`/`follow_redirects_any` failover; each component's existing hard-error-on-invalid-value behavior.
- **New components rationale**: exactly one new C function, because harness's settings-table shape cannot reuse a "given `cli_value`, return resolved" signature the other three could share (see `research.md`, Design Decisions).
- **Steering compliance**: no new build dependency; C standard/build system (Meson) and Python (click) usage unchanged.

### Technology Stack

| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| C CLI / libxce | gnu17, existing `libxce` static lib | New `xce_controller_peers_env()` primitive; consumed at 4 existing call sites | No new dependency |
| Python CLI | click (already a `xceadmin` dependency) | Native multi-`envvar` list resolution (no new click usage pattern) | Confirmed via direct read of the installed click version's `resolve_envvar_value()` |

## File Structure Plan

One new file (the regression script below); every other change is to an existing file — no new directories.

### New Files
- `src/tools/verify-controller-peers-env.sh` — repeatable regression script (see Testing Strategy → Scripted Regression Verification) driving the real built binaries of all 4 C components plus harness/`xce_test`, asserting on startup log/settings-dump output only (no live cluster required).

### Modified Files
- `src/include/xce/cluster.h` — declare `const char *xce_controller_peers_env(void);` alongside the existing `xce_parse_peers()` declaration.
- `src/lib/cluster.c` — implement `xce_controller_peers_env()`: read `getenv("XCE_CONTROLLER_PEERS")`, return it if non-empty, else `NULL`.
- `src/lib/testsuite/test_cluster.c` — new unit tests for `xce_controller_peers_env()` (unset, empty, non-empty).
- `src/controller/main.c` — at the existing `if (arguments.peers_str)` site (~L455), resolve `arguments.peers_str` against `xce_controller_peers_env()` before the check; on a parse failure, name the source (flag vs. environment variable) in the existing error message; add `XCE_CONTROLLER_PEERS` to the `--peers` option's help text (~L85-87).
- `src/scheduler/src/main.c` — identical shape at the existing `if (arguments.peers)` site (~L403); help text update (~L84). No change anywhere in the `--sim`/`--interactive` path, which already returns before this point.
- `src/worker/src/main.c` — identical shape at the existing `if (peers_str)` site (~L470); update the `-h`/usage `printf` (~L443-446) to mention `XCE_CONTROLLER_PEERS`.
- `src/harness/test.c` — at startup, before the REPL loop and before rc-file loading, compute the `"masters"` entry's `defval` from `xce_controller_peers_env()` if set, else leave it at today's literal `"localhost:9990"`; extend the `USAGE`/`set`-command help text (~L409-421) to mention the new variable.
- `src/xceadmin/src/xceadmin/discovery.py` — extend `resolve_controllers()`'s per-entry parsing to strip an optional `name=` prefix (mirrors `xce_parse_peers()`) before its existing `rsplit(":", 1)`.
- `src/xceadmin/src/xceadmin/cli.py` — change each of the 4 `--controller` options' `envvar="XCEADMIN_CONTROLLER"` to `envvar=["XCE_CONTROLLER_PEERS", "XCEADMIN_CONTROLLER"]`; update each option's help string.
- `src/xceadmin/tests/test_discovery.py` — new unit tests for `resolve_controllers()`'s `name=`-stripping.
- `src/xceadmin/tests/test_cli.py` — new unit tests for the new env-var precedence (mirrors the existing `test_controller_option_falls_back_to_xceadmin_controller_env_var`/`test_controller_flag_overrides_xceadmin_controller_env_var` tests).
- `.kiro/steering/build-deployment.md` — document `XCE_CONTROLLER_PEERS`: components, value format, precedence.
- `.claude/skills/xce-deploy-ha/SKILL.md` — informational note only (no generated-script behavior change).
- `.claude/skills/xce-test-multihost/SKILL.md` — same note, at its three existing `--peers`-mentioning locations (single-node deploy step, `xcecd_virtual` flag-table note, Prerequisites' generated-script list).

## System Flows

```mermaid
flowchart TD
    Start[Resolution point reached] --> HasFlag{Explicit flag or\nset command given?}
    HasFlag -- yes --> UseFlag[Use that value\nnever consult env var]
    HasFlag -- no --> HasNewEnv{XCE_CONTROLLER_PEERS\nnon-empty?}
    HasNewEnv -- yes --> UseNewEnv[Use it\nxceadmin strips name prefix]
    HasNewEnv -- no --> IsAdmin{Component is xceadmin?}
    IsAdmin -- yes --> HasLegacyEnv{XCEADMIN_CONTROLLER\nnon-empty?}
    IsAdmin -- no --> UseTodayDefault[Today's unchanged default\nstandalone / built-in literal]
    HasLegacyEnv -- yes --> UseLegacyEnv[Use it, unchanged behavior]
    HasLegacyEnv -- no --> UseTodayDefault
```

Both branches that resolve to a real candidate list (`UseFlag`, `UseNewEnv`, `UseLegacyEnv`) still pass through each component's existing, unchanged validation (`xce_parse_peers()`/`resolve_controllers()`); an invalid value is a hard error either way, distinguished by source only in the error message text (`1.5`).

This resolution point is reached exactly once, at process startup, for controller/scheduler/worker/xceadmin. For harness/`xce_test` it is reached on *every* connection attempt (`enter_ctx()`), since `set masters ...` can run at any point in an interactive/scripted session — the diagram represents one such evaluation, not a startup-only decision, for that component.

## Requirements Traceability

| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1 | Recognize `XCE_CONTROLLER_PEERS` | controller/scheduler/worker/harness main.c/test.c | `xce_controller_peers_env()` | Flowchart: `HasNewEnv` |
| 1.2 | Explicit `--peers` wins, env var not consulted | controller/scheduler/worker main.c | existing `if (peers)` gate | Flowchart: `HasFlag` = yes |
| 1.3 | Env var used when flag omitted | controller/scheduler/worker main.c, harness test.c | `xce_controller_peers_env()` | Flowchart: `UseNewEnv` |
| 1.4 | Unset/empty env var ⇒ unchanged today's behavior | all 4 | same call sites | Flowchart: `UseTodayDefault` |
| 1.5 | Parse-error message names the source | controller/scheduler/worker main.c, harness test.c | existing error `fprintf`/`enter_ctx` message | n/a |
| 1.6 | `--peers` help text documents the variable | controller/scheduler/worker main.c, harness test.c | argp/getopt/settings help strings | n/a |
| 2.1 | Derive controller list from env var | xceadmin cli.py, discovery.py | click `envvar=` list | Flowchart: `UseNewEnv` |
| 2.2 | Strip `name=` prefix | xceadmin discovery.py | `resolve_controllers()` | n/a |
| 2.3 | Try each candidate in order | xceadmin discovery.py | `resolve_controllers`/`follow_redirects_any` (unmodified) | n/a |
| 2.4 | Explicit `--controller` wins | xceadmin cli.py | click parameter resolution (unmodified) | Flowchart: `HasFlag` = yes |
| 2.5 | Legacy `XCEADMIN_CONTROLLER` fallback unchanged | xceadmin cli.py | click `envvar=` list order | Flowchart: `HasLegacyEnv` |
| 2.6 | Both unset ⇒ hardcoded default unchanged | xceadmin cli.py | `_DEFAULT_CONTROLLER` (unmodified) | Flowchart: `UseTodayDefault` |
| 2.7 | `--controller` help text documents the variable | xceadmin cli.py | click option help strings | n/a |
| 3.1 | Steering doc describes the variable | `build-deployment.md` | n/a | n/a |
| 3.2 | HA deploy guidance notes availability | `xce-deploy-ha/SKILL.md` | n/a | n/a |
| 3.3 | Multi-host guidance notes availability | `xce-test-multihost/SKILL.md` | n/a | n/a |

## Components and Interfaces

| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|--------------|--------|---------------|-------------------|-----------|
| `xce_controller_peers_env()` | libxce / C | Read `XCE_CONTROLLER_PEERS`, empty-as-unset | 1.1, 1.3, 1.4 | none (P0) | Service |
| controller/scheduler/worker `--peers` call sites | C daemon CLI | Combine flag + env var, preserve existing validation/help | 1.2-1.6 | `xce_controller_peers_env` (P0) | — |
| harness `masters` setting default | C CLI (settings table) | Compute `defval` from env var at startup | 1.1, 1.3, 1.4, 1.5, 1.6 | `xce_controller_peers_env` (P0) | State |
| `resolve_controllers()` extension | xceadmin / Python | Strip `name=` prefix per entry | 2.1, 2.2 | none (P0) | Service |
| xceadmin `--controller` options | xceadmin / Python (click) | Add `XCE_CONTROLLER_PEERS` ahead of legacy envvar | 2.1, 2.4-2.7 | `resolve_controllers` (P0), click (P0, external) | — |
| Documentation updates | Docs | Describe/announce the variable | 3.1-3.3 | none | — |

Only the two components introducing genuinely new logic — `xce_controller_peers_env()` and the `resolve_controllers()` extension — warrant a full contract block; the remaining rows are thin call-site wiring around existing, unchanged behavior.

### libxce

#### xce_controller_peers_env

| Field | Detail |
|-------|--------|
| Intent | Sole read point for `XCE_CONTROLLER_PEERS`, normalizing empty-string to absent |
| Requirements | 1.1, 1.3, 1.4 |

**Responsibilities & Constraints**
- Pure function: no side effects, no caching (the environment can only change between process invocations for these components, so no staleness concern).
- Treats an empty string identically to an unset variable, matching the same convention `resolve_controllers()`'s per-entry blank-skip already uses.

**Dependencies**
- Inbound: controller/scheduler/worker `main.c` `--peers` call sites (P0), harness `test.c` startup `masters`-default computation (P0).
- Outbound: none.

**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

##### Service Interface
```c
/* src/include/xce/cluster.h */
const char *xce_controller_peers_env(void);
```
- Preconditions: none.
- Postconditions: returns a pointer into the process environment (valid for the process lifetime) or `NULL`.
- Invariants: never returns a non-NULL empty string.

### xceadmin

#### resolve_controllers (extended)

| Field | Detail |
|-------|--------|
| Intent | Parse a `--controller`-shaped value into an ordered candidate list, now tolerant of `name=` prefixes |
| Requirements | 2.1, 2.2 |

**Responsibilities & Constraints**
- Existing behavior (whitespace stripping, blank-entry skipping, bare-host default port) is unchanged.
- New: for each comma-separated entry, if it contains `=`, discard everything up to and including the first `=` before applying the existing `rsplit(":", 1)`.

**Dependencies**
- Inbound: xceadmin `--controller` options (P0), any future caller expecting `name=host:port` tolerance.
- Outbound: none new.

**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]

##### Service Interface
```python
def resolve_controllers(controller: str) -> list[tuple[str, int]]: ...
```
- Preconditions: `controller` is a comma-separated string of `[name=]host[:port]` entries.
- Postconditions: returns an ordered list of `(host, port)` tuples, `name=` prefixes (if any) stripped.
- Invariants: a bare `host:port` entry (today's only supported shape) parses identically to before.

**Implementation Notes**
- Integration: `follow_redirects_any()` is untouched — it only ever consumes `resolve_controllers()`'s output.
- Validation: existing `test_discovery.py` coverage for bare-host/list parsing must stay green; new tests add `name=`-prefixed and mixed-prefix cases.
- Risks: none identified (see `research.md` Risks & Mitigations for the `=`-in-hostname non-issue).

## Error Handling

### Error Strategy
Every existing hard-error-on-invalid-value behavior is preserved exactly; the only addition is naming the value's source in the message text, per 1.5.

### Error Categories and Responses
- **Invalid candidate list, from an explicit `--peers`/`--controller`/`set masters` value**: unchanged existing message and exit behavior.
- **Invalid candidate list, from `XCE_CONTROLLER_PEERS`**: same parse failure, same exit behavior, message additionally names the environment variable as the source (controller/scheduler/worker: adjust the existing `fprintf`; harness: `enter_ctx()`'s existing "invalid 'masters' setting" message, extended to check `settings[i].value == NULL` for the `"masters"` entry at the point of failure — reusing `get_setting()`'s own existing value-before-defval precedence check, not a new tracking variable — to say "default (from XCE_CONTROLLER_PEERS)" instead of "explicit `set masters`" when true).
- **Invalid candidate list, from legacy `XCEADMIN_CONTROLLER`**: unchanged (out of scope — 1.5/2.x do not require source-naming for the legacy variable, only for `XCE_CONTROLLER_PEERS` and the explicit flag).

### Monitoring
No new logging/metrics infrastructure; each component's existing startup log line (e.g. scheduler's `INFO("HA cluster config: ...")` / `INFO("No --peers flag; ...")`) is extended only to remain accurate about which source supplied the value it's reporting.

## Testing Strategy

### Unit Tests
- `xce_controller_peers_env()` (`src/lib/testsuite/test_cluster.c`): unset → `NULL`; empty string → `NULL`; non-empty → returned unchanged. (1.1, 1.3, 1.4)
- `resolve_controllers()` extension (`src/xceadmin/tests/test_discovery.py`): single `name=host:port`; mixed bare/named multi-candidate list; existing bare-host cases still pass unchanged. (2.2)
- xceadmin `--controller` option precedence (`src/xceadmin/tests/test_cli.py`): `XCE_CONTROLLER_PEERS` set alone; both `XCE_CONTROLLER_PEERS` and `XCEADMIN_CONTROLLER` set (new wins); explicit `--controller` overrides both; neither set (existing default). (2.1, 2.4-2.6) All existing `test_cli.py` cases that assert `XCEADMIN_CONTROLLER`-only or no-env-var behavior (e.g. `test_controller_option_falls_back_to_xceadmin_controller_env_var`) must explicitly `monkeypatch.delenv("XCE_CONTROLLER_PEERS", raising=False)`, so the new variable's introduction cannot make them order-dependent on ambient environment state.

### Scripted Regression Verification
None of controller/scheduler/worker/harness's `main()`-level argument or
settings-table resolution has existing unit-test coverage today (confirmed
via source search; `test_cluster_parse.c` tests a standalone duplicated
reference parser, not live `main()` wiring) — refactoring any of the four
into a unit-testable shape is out of this feature's boundary (see
Non-Goals). Rather than one-time manual checking, this is verified by a new
repeatable script, `src/tools/verify-controller-peers-env.sh`, following
`xce-test-multihost`'s established pattern of driving real built binaries
and asserting on their output — but deliberately lightweight: it asserts on
each component's **startup log/settings-dump line**, not on a live,
reachable cluster, since resolving the candidate list happens, and is
observable, entirely before any connection attempt succeeds or fails:
- controller/scheduler/worker: with `XCE_CONTROLLER_PEERS` set (e.g.
  `a=host1:1,b=host2:2`) and no `--peers`, start briefly and grep the log
  for the existing "N candidate(s)" line already logged today for an
  explicit `--peers` value, asserting `N` and the candidate identity match
  the env var's content — no reachable target required. (1.1, 1.3)
- Same three, with both `XCE_CONTROLLER_PEERS` set to one value and
  `--peers` set to a different one; assert the log reflects the `--peers`
  value, not the env var's. (1.2)
- Same three, with neither set; assert the existing "No --peers flag"
  (or worker's "single-controller path") log line, unchanged. (1.4)
- scheduler with `--sim <file>` and `XCE_CONTROLLER_PEERS` set: assert the
  process never logs an "HA cluster config"/"No --peers flag" line at all
  (the `--sim` path returns before that code is reached — this is a
  by-construction guarantee already, not new logic; the script asserts it
  stays true). (Non-Goals cross-check)
- harness/`xce_test`: run non-interactively with a script containing only
  `set` (no arguments) — `cmd_set`'s existing argc==1 branch prints every
  setting's current value/default with **no connection attempt** — first
  with `XCE_CONTROLLER_PEERS` set and no prior `set masters`, asserting the
  printed `masters` default reflects the env var; then with an explicit
  `set masters ...` line before the bare `set`, asserting the printed
  *value* (not default) reflects the explicit override. (1.1, 1.3)
- Trigger a parse error via each of an invalid explicit flag and an invalid
  `XCE_CONTROLLER_PEERS` value on one component (e.g. controller); assert
  the two error messages are distinguishable by source. (1.5)

This script is a new addition to the File Structure Plan and is intended to
be re-run on any future change to these five components' startup argument
handling, not a one-time implementation check.