docs(controller-peers-env-fallback): design review fixes and requirements revision

Design review (NO-GO -> GO) and iterated requirements review until clean:

- Controller refuses to start (non-zero exit, stderr, before any network
  I/O) when an env-supplied XCE_CONTROLLER_PEERS lacks its own name (1.6);
  new libxce xce_peers_find_identity() replaces both identity loops in
  controller/cluster.c so the check and my_priority/listen port agree.
- deploy-ha.sh's generated scripts/xceadmin wrapper exports
  XCE_CONTROLLER_PEERS so an ambient value cannot redirect it (Req 4).
- Opt-out is unsetting/emptying the variable (1.7); "No --peers" log
  lines say it was unset or empty (1.8); docs cover sudo/systemd not
  inheriting it (3.1).
- Requirement 1 defines explicit candidate list / resolution point /
  own name so harness (set masters) is covered precisely; error messages
  name the value's source incl. validation failures (1.4).
- xceadmin: name= stripping and a clean malformed-entry error apply to
  every --controller source; errors name the env var source (2.2, 2.8).
- Test plan fixed against real code (stderr vs stdout, log texts,
  xcecd_virtual, --norc, free ports, failover order test).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rongsong Shen 2026-10-03 04:07:49 +00:00
parent 5c4d101b15
commit bd1f5175a1
Signed by untrusted user: rshen
GPG key ID: 80533536B0EB651C
4 changed files with 417 additions and 153 deletions

View file

@ -2,21 +2,22 @@
## Overview
**Purpose**: This feature lets an operator set one environment variable, `XCE_CONTROLLER_PEERS`, once per session or deployment, instead of retyping `--peers` (controller, scheduler, worker, harness/xce_test) or `--controller` (xceadmin) on every invocation. It changes nothing about what any of these components do once their connection target is resolved — only where that target's default value comes from when the operator did not name one explicitly.
**Purpose**: This feature lets an operator set one environment variable, `XCE_CONTROLLER_PEERS`, once per session or deployment, instead of retyping `--peers` (controller, scheduler, worker), `set masters` (harness/xce_test) or `--controller` (xceadmin) on every invocation. It changes nothing about what any of these components do once their connection target is resolved — only where that target's default value comes from when the operator did not name one explicitly.
**Users**: Operators running local/interactive HPC cluster tools, and deploy/test scripts (e.g. HA soak setups) that currently either hardcode `--peers`/`--controller` into every generated command or accept the single-node/no-controller default they didn't actually want.
**Impact**: Five independently-built binaries each gain one additional, lowest-priority source for their existing connection-target resolution. No existing explicit-flag behavior, no existing "omitted, no env var" behavior, and no existing multi-candidate failover behavior changes.
**Impact**: Five independently-built binaries each gain one additional, lowest-priority source for their existing connection-target resolution. Explicit `--peers`/`set masters` behavior, "omitted, no env var" behavior (apart from a log line's wording, 1.8), and xceadmin's multi-candidate failover order are unchanged; xceadmin's parsing of any `--controller` value, explicit or not, gains `name=` stripping and a clean malformed-entry error (2.2, 2.8). Two guards come with the new source: the controller refuses to start, early and with a message naming the variable, when an env-supplied list does not contain its own name (that list is Raft membership, not just a target list), and `deploy-ha.sh`'s generated `scripts/xceadmin` wrapper exports the new variable so an ambient value cannot redirect it.
### Goals
- One new environment variable, consulted only when the corresponding CLI option is not given, across controller, scheduler, worker, harness/xce_test (via `--peers`) and xceadmin (via `--controller`).
- Zero behavior change when the CLI option is given, and zero behavior change when neither the option nor the variable is set.
- Reuse xceadmin's existing multi-candidate failover (`resolve_controllers`/`follow_redirects_any`, bug-1821) and its existing `XCEADMIN_CONTROLLER` fallback unmodified.
- One new environment variable, used only when no explicit value is given, across controller, scheduler and worker (via `--peers`), harness/xce_test (via `set masters`) and xceadmin (via `--controller`).
- An explicit value (`--peers`, `set masters`, `--controller`) always wins over the variable. Explicit `--peers`/`set masters` behave exactly as today; an explicit `--controller` gains only `name=` stripping and a clean malformed-entry error (2.2, 2.8).
- Zero behavior change when neither an explicit value nor the variable is set (apart from the startup log line's wording, 1.8).
- Reuse xceadmin's existing multi-candidate failover (`follow_redirects_any`, bug-1821, unmodified) and its existing `XCEADMIN_CONTROLLER` fallback; `resolve_controllers()` is extended only as 2.2/2.8 require.
### Non-Goals
- apiserver (YAML-config-based) and xcecli's `--url` (a different target) — untouched.
- Any change to `src/tools/deploy-ha.sh`'s generated scripts' own behavior.
- Any change to what "no `--peers`/`--controller` and no env var" means today for any component.
- Any change to `src/tools/deploy-ha.sh`'s generated daemon scripts' behavior (the `scripts/xceadmin` wrapper's added export, Requirement 4, is the one exception).
- Any change to what "no explicit value and no env var" does today for any component (only the log wording changes, 1.8).
- A TSAN/new-test-infrastructure investment for `main()`-level argument parsing where none exists today (see Testing Strategy).
## Boundary Commitments
@ -24,6 +25,9 @@
### This Spec Owns
- A single shared C primitive (`xce_controller_peers_env()`, `libxce`) that reads `XCE_CONTROLLER_PEERS`, treating an unset or empty value as absent.
- The one-line integration of that primitive into controller, scheduler, and worker's existing `--peers`-gating `if`, and into harness's settings-table default computation at startup.
- A second shared C primitive, `xce_peers_find_identity()` (`libxce`), holding the one rule for "which candidate is me" (candidate `name` if set, else `hostname`, compared with the controller's name). The controller's two existing copies of that loop — the `my_priority` assignment in `cluster_init_phase1()` (`src/controller/cluster.c` ~L1784-1791) and the listen-port lookup in `cluster_open_listener()` (~L1890-1906) — are switched to call it, each keeping its own existing no-match handling, so the new startup check and both existing lookups can never disagree.
- The controller's startup identity check for an env-supplied candidate list (1.6).
- One added line in `deploy-ha.sh`'s generated `scripts/xceadmin` wrapper: `export XCE_CONTROLLER_PEERS="$PEERS_STR"` (Requirement 4).
- Extending xceadmin's existing `resolve_controllers()` to strip an optional `name=` prefix per candidate entry.
- Adding `"XCE_CONTROLLER_PEERS"` ahead of the existing `"XCEADMIN_CONTROLLER"` in each `--controller` option's `envvar=` list.
- Help-text and documentation updates naming the new variable (CLI `--peers`/`--controller` help text, `build-deployment.md`, `xce-deploy-ha`/`xce-test-multihost` skill docs).
@ -32,27 +36,58 @@
### Out of Boundary
- apiserver's YAML-based peer configuration.
- xcecli's `--url` option.
- `deploy-ha.sh`'s generated scripts' own invocations (documentation may *mention* the variable; the generated scripts keep passing `--peers`/`--controller` explicitly).
- Any change to `xce_parse_peers()`'s or `resolve_controllers()`/`follow_redirects_any()`'s existing parsing/failover semantics beyond the one `name=`-stripping addition.
- Refactoring any of the four C `main()` functions to become unit-testable in general — only the new, genuinely independent primitive is unit-tested (see Testing Strategy).
- `deploy-ha.sh`'s generated daemon scripts' own invocations (they keep passing `--peers` explicitly). The `scripts/xceadmin` wrapper's one added export (Requirement 4) is the only `deploy-ha.sh` change.
- Changing what the controller does when an *explicit* `--peers` list lacks its name. Today `cluster_open_listener()` refuses (`ERROR` in the log, `myapp_run()` returns before any network I/O), and `main()` ignores `myapp_run()`'s result, so the process exits with status 0. That behavior is kept; see Revalidation Triggers.
- Any change to `xce_parse_peers()`'s or `resolve_controllers()`/`follow_redirects_any()`'s existing parsing/failover semantics beyond `resolve_controllers()`'s two additions: `name=` stripping (2.2) and a clean error for a malformed entry (2.8). Both apply to an explicit `--controller` value too.
- Refactoring any of the four C `main()` functions to become unit-testable in general — only the two new, genuinely independent primitives are unit-tested (see Testing Strategy).
### Allowed Dependencies
- All four C targets already link `libxce` (verified per-component `meson.build`) — the shared primitive may live there with no new build-graph edge.
- All four C targets already link `libxce` (verified per-component `meson.build`) — the shared primitives may live there with no new build-graph edge.
- xceadmin's existing `resolve_controllers()`/`follow_redirects_any()` (`discovery.py`) and its existing `--controller` click options (`cli.py`) — reused and extended in place, not duplicated.
### Project Invariant Check
This feature's controller/scheduler changes touch only pre-startup argument
resolution (`main.c`, before `sched_init()`/`myapp_run()`/any daemon thread,
event loop, or Raft/HA state machine is created) — it does not intersect
`xce-audit-invariants`' controller dimensions (async I/O thread-role
placement, fail-loud/never-self-exit process lifecycle, consensus/
replication safety) or its scheduler dimension (driver/simulation
state-equivalence). No audit re-check is required beyond this design.
Walked against `xce-audit-invariants`' controller and scheduler dimensions:
- **Consensus/replication safety (controller) — applies.** For the
controller, `--peers` is the Raft membership, and `my_priority` doubles
as `self_idx` (`ha.c` ~L633, ~L2964). If the controller's name matches
no candidate, `cluster_init_phase1()` (`cluster.c` ~L1783) falls back to
the last index, which would give the node another peer's `self_idx`.
That is already contained today: `cluster_open_listener()` (~L1902)
refuses the same no-match case in `myapp_run()` step 3, before the
listener, iomgr or any election starts, so no Raft traffic is sent
under the wrong identity. An inherited value makes the case far more
likely, though (a controller started without `--myname`, so its name is
the hostname, or one meant to run standalone), and the existing refusal
then blames "--peers", logs only to the log file, and exits with status
0. Guard (1.6): when the list came from `XCE_CONTROLLER_PEERS` and
`xce_peers_find_identity()` finds no match, `main.c` writes the error to
stderr naming the variable and the remedies and returns 1, before
`myapp_run()`. Because all three lookups now share one function, the
new check cannot pass where the listener check would refuse, or the
reverse. A bare single `host:port` value follows the same rule: its key
is the hostname, so a controller joins a one-node HA cluster only when
its name is literally that hostname; otherwise it refuses, rather than
silently changing from standalone to HA mode.
- **Fail-loud, never self-exit (controller) — not affected.** The new exit
happens in `main()` during argument validation, the same place and shape
as the existing `--peers` parse error and even-peer-count rejection;
no daemon thread, event loop or HA state exists yet.
- **Async I/O thread roles (controller) — not affected.** All reads of the
environment happen in `main()` before any thread starts; the two
`cluster.c` refactors are pure function calls in existing pre-election
startup code (`cluster_init_phase1()`, `cluster_open_listener()`), with
no blocking added.
- **Driver/simulation state-equivalence (scheduler) — not affected.**
`--sim`/`--interactive` return before the `--peers` block, so the
variable can never reach replay mode.
### Revalidation Triggers
- `xce_peers_find_identity()` is now the only "which candidate is me" rule for the controller (the 1.6 check, `my_priority`, listen port). Any future identity rule (e.g. matching on host:port) changes there, never in a re-added inline loop.
- If the explicit-`--peers` no-match case is ever moved to the same early `main()` check with a non-zero exit (a reasonable follow-up, out of scope here), drop the env-only condition on the 1.6 check instead of adding a second check.
- Any change to `xce_parse_peers()`'s accepted value format (e.g. a new candidate syntax) must be mirrored in `resolve_controllers()`'s parsing if the two are to stay consistent for `XCE_CONTROLLER_PEERS` values shared between C daemons and xceadmin.
- Any future change to harness's `settings[]` initialization order (e.g. rc-file loading moved earlier than today) must preserve "the env-var-derived default is computed before any explicit `set masters` can run."
- If `deploy-ha.sh` is ever updated to adopt `XCE_CONTROLLER_PEERS` (explicitly deferred here), its generated scripts' current behavior baseline (documented in this design's Out of Boundary) is the thing to re-check against.
- If `deploy-ha.sh`'s generated daemon scripts are ever updated to rely on `XCE_CONTROLLER_PEERS` instead of an explicit `--peers` (explicitly deferred here), re-check them against the baseline in this design's Out of Boundary, and re-check the 1.6 identity rule for each generated controller script.
## Architecture
@ -62,8 +97,8 @@ Each of the five components already resolves its own connection target independe
| Component | Mechanism | Where "omitted" is decided | Existing fallback today |
|---|---|---|---|
| controller | `argp`, `OPT_PEERS` → `arguments.peers_str` | `if (arguments.peers_str)` (`main.c` ~L455) | Single-node primary (bug-1457) |
| scheduler | `argp`, `'P'`/`--peers` → `arguments.peers` | `if (arguments.peers)` (`main.c` ~L403); **`--sim` returns before this point is reached** | "single-controller path" (logged) |
| controller | `argp`, `OPT_PEERS` → `arguments.peers_str` | `if (arguments.peers_str)` (`main.c` ~L455) | Single-node primary (bug-1457). With a list, own entry found by name twice in `cluster.c` (~L1784 `my_priority`, ~L1890 listen port); no match is refused in `cluster_open_listener()` before any network I/O, exit status 0 |
| scheduler | `argp`, `'P'`/`--peers` → `arguments.peers` | `if (arguments.peers)` (`main.c` ~L411); **`--sim` returns before this point is reached** | "single-controller path" (logged) |
| worker | `getopt_long`, `OPT_PEERS` → `peers_str` | `if (peers_str)` (`main.c` ~L470) | Single-controller path via `config.remote_host`/`config.remote_port` (logged; same shape as controller/scheduler's own single-controller fallback, not an error path) |
| harness/xce_test | `settings[]` table, key `"masters"`, no CLI flag at all | `get_setting()`'s `value ?: defval` (`test.c` ~L991) | `defval = "localhost:9990"` |
| xceadmin | click `--controller` option, `envvar="XCEADMIN_CONTROLLER"` | click's own parameter resolution | `_DEFAULT_CONTROLLER = "localhost:9991"` |
@ -71,17 +106,17 @@ Each of the five components already resolves its own connection target independe
All three C daemons that gate on a `--peers` argument already treat an invalid value as a hard startup error (`xce_parse_peers() != 0` → message + `return 1`), never a silent fallback — this feature preserves that for both an invalid explicit flag and an invalid `XCE_CONTROLLER_PEERS` value.
### Architecture Integration
- **Selected pattern**: one shared, stateless C primitive consulted independently by four call sites, plus one Python parsing extension consulted by click's own (unmodified) resolution order — no new service, process, or inter-component call is introduced.
- **Selected pattern**: two shared, stateless C primitives (an env reader used by four call sites, and an identity lookup used by three controller sites), plus one Python parsing extension consulted by click's own (unmodified) resolution order — no new service, process, or inter-component call is introduced.
- **Domain/feature boundaries**: each component keeps deciding its own resolved value; the new variable is only ever a *source*, never a coordinator.
- **Existing patterns preserved**: `xce_parse_peers()`'s `name=`-prefix convention; xceadmin's `resolve_controllers`/`follow_redirects_any` failover; each component's existing hard-error-on-invalid-value behavior.
- **New components rationale**: exactly one new C function, because harness's settings-table shape cannot reuse a "given `cli_value`, return resolved" signature the other three could share (see `research.md`, Design Decisions).
- **New components rationale**: two new C functions. `xce_controller_peers_env()` is a bare env reader because harness's settings-table shape cannot reuse a "given `cli_value`, return resolved" signature the other three could share (see `research.md`, Design Decisions). `xce_peers_find_identity()` exists so the controller's new startup check and its two existing lookups (`my_priority`, listen port) share one matching rule.
- **Steering compliance**: no new build dependency; C standard/build system (Meson) and Python (click) usage unchanged.
### Technology Stack
| Layer | Choice / Version | Role in Feature | Notes |
|-------|------------------|-----------------|-------|
| C CLI / libxce | gnu17, existing `libxce` static lib | New `xce_controller_peers_env()` primitive; consumed at 4 existing call sites | No new dependency |
| C CLI / libxce | gnu17, existing `libxce` static lib | New `xce_controller_peers_env()` (4 call sites) and `xce_peers_find_identity()` (controller `main.c` + two `cluster.c` sites) | No new dependency |
| Python CLI | click (already a `xceadmin` dependency) | Native multi-`envvar` list resolution (no new click usage pattern) | Confirmed via direct read of the installed click version's `resolve_envvar_value()` |
## File Structure Plan
@ -89,22 +124,25 @@ All three C daemons that gate on a `--peers` argument already treat an invalid v
One new file (the regression script below); every other change is to an existing file — no new directories.
### New Files
- `src/tools/verify-controller-peers-env.sh` — repeatable regression script (see Testing Strategy → Scripted Regression Verification) driving the real built binaries of all 4 C components plus harness/`xce_test`, asserting on startup log/settings-dump output only (no live cluster required).
- `src/tools/verify-controller-peers-env.sh` — repeatable regression script (see Testing Strategy → Scripted Regression Verification) driving the real built binaries of controller, scheduler, worker and harness/`xce_test`, plus `deploy-ha.sh`'s generated xceadmin wrapper, asserting on startup log/settings-dump/printed-environment output only (no live cluster required).
### Modified Files
- `src/include/xce/cluster.h` — declare `const char *xce_controller_peers_env(void);` alongside the existing `xce_parse_peers()` declaration.
- `src/lib/cluster.c` — implement `xce_controller_peers_env()`: read `getenv("XCE_CONTROLLER_PEERS")`, return it if non-empty, else `NULL`.
- `src/lib/testsuite/test_cluster.c` — new unit tests for `xce_controller_peers_env()` (unset, empty, non-empty).
- `src/controller/main.c` — at the existing `if (arguments.peers_str)` site (~L455), resolve `arguments.peers_str` against `xce_controller_peers_env()` before the check; on a parse failure, name the source (flag vs. environment variable) in the existing error message; add `XCE_CONTROLLER_PEERS` to the `--peers` option's help text (~L85-87).
- `src/scheduler/src/main.c` — identical shape at the existing `if (arguments.peers)` site (~L403); help text update (~L84). No change anywhere in the `--sim`/`--interactive` path, which already returns before this point.
- `src/worker/src/main.c` — identical shape at the existing `if (peers_str)` site (~L470); update the `-h`/usage `printf` (~L443-446) to mention `XCE_CONTROLLER_PEERS`.
- `src/harness/test.c` — at startup, before the REPL loop and before rc-file loading, compute the `"masters"` entry's `defval` from `xce_controller_peers_env()` if set, else leave it at today's literal `"localhost:9990"`; extend the `USAGE`/`set`-command help text (~L409-421) to mention the new variable.
- `src/xceadmin/src/xceadmin/discovery.py` — extend `resolve_controllers()`'s per-entry parsing to strip an optional `name=` prefix (mirrors `xce_parse_peers()`) before its existing `rsplit(":", 1)`.
- `src/xceadmin/src/xceadmin/cli.py` — change each of the 4 `--controller` options' `envvar="XCEADMIN_CONTROLLER"` to `envvar=["XCE_CONTROLLER_PEERS", "XCEADMIN_CONTROLLER"]`; update each option's help string.
- `src/xceadmin/tests/test_discovery.py` — new unit tests for `resolve_controllers()`'s `name=`-stripping.
- `src/xceadmin/tests/test_cli.py` — new unit tests for the new env-var precedence (mirrors the existing `test_controller_option_falls_back_to_xceadmin_controller_env_var`/`test_controller_flag_overrides_xceadmin_controller_env_var` tests).
- `.kiro/steering/build-deployment.md` — document `XCE_CONTROLLER_PEERS`: components, value format, precedence.
- `.claude/skills/xce-deploy-ha/SKILL.md` — informational note only (no generated-script behavior change).
- `src/include/xce/cluster.h` — declare `const char *xce_controller_peers_env(void);` and `int xce_peers_find_identity(const xce_cluster_config_t *cfg, const char *name);` alongside the existing `xce_parse_peers()` declaration.
- `src/lib/cluster.c` — implement `xce_controller_peers_env()`: read `getenv("XCE_CONTROLLER_PEERS")`, return it if non-empty, else `NULL`. Implement `xce_peers_find_identity()` with the matching rule moved verbatim from `src/controller/cluster.c` (identical at ~L1784-1791 and ~L1890-1900).
- `src/lib/testsuite/test_cluster.c` — new unit tests for `xce_controller_peers_env()` (unset, empty, non-empty) and `xce_peers_find_identity()` (named match, bare-hostname match, named entry not matched by its hostname, no match → -1, first of duplicate names wins, NULL name → -1, empty name matching an unnamed empty-hostname entry exactly as today).
- `src/controller/main.c` — at the existing `if (arguments.peers_str)` site (~L455), resolve `arguments.peers_str` against `xce_controller_peers_env()` before the check, remembering in a local `peers_from_env` whether the env var supplied it; on a parse or even-count failure, name the source (flag vs. environment variable) in the existing error message; after parsing, when `peers_from_env` is true and `xce_peers_find_identity(&myconf.peers_cfg, arguments.myname) < 0`, print the 1.6 error and `return 1`; extend the "No --peers flag" `INFO` (~L486) per 1.8; add `XCE_CONTROLLER_PEERS` and the "unset it to run standalone" note (1.7) to the `--peers` option's help text (~L85-87).
- `src/controller/cluster.c` — replace both inline identity loops with `xce_peers_find_identity(pcfg, c->myconf->name)`: in `cluster_init_phase1()` (~L1784-1791), `my_priority = idx >= 0 ? idx : pcfg->candidates_count - 1;`; in `cluster_open_listener()` (~L1890-1906), `matched = idx >= 0; my_port = matched ? pcfg->candidates[idx].port : 0;` followed by the existing refusal and default-port code. Same result for every input.
- `src/tools/deploy-ha.sh` — in the generated `scripts/xceadmin` wrapper heredoc (~L689-694), add `export XCE_CONTROLLER_PEERS="$PEERS_STR"` next to the existing `XCEADMIN_CONTROLLER` export, and extend the comment above it to say why (Requirement 4).
- `src/scheduler/src/main.c` — identical shape at the existing `if (arguments.peers)` site (~L411), without the identity check; name the source in both the parse-error and the `xce_cluster_config_validate()` failure messages (~L412-424, 1.4); "No --peers flag" `INFO` (~L435) per 1.8; help text update with 1.7's note (~L88). No change anywhere in the `--sim`/`--interactive` path, which already returns before this point.
- `src/worker/src/main.c` — identical shape at the existing `if (peers_str)` site (~L470), without the identity check; name the source in both the parse-error and the `xce_cluster_config_validate()` failure messages (~L472-485, 1.4); "No --peers flag" log line (~L501) per 1.8; update the `-h`/usage `printf` (~L443-446) to mention `XCE_CONTROLLER_PEERS`, 1.7's note, and that `sudo` drops it unless preserved.
- `src/harness/test.c` — at startup, before the REPL loop and before rc-file loading, compute the `"masters"` entry's `defval` from `xce_controller_peers_env()` if set, else leave it at today's literal `"localhost:9990"`; extend the `USAGE`/`set`-command help text (~L403-415) to mention the new variable and 1.7's note.
- `src/xceadmin/src/xceadmin/discovery.py` — extend `resolve_controllers()`'s per-entry parsing to strip an optional `name=` prefix (mirrors `xce_parse_peers()`) before its existing `rsplit(":", 1)`; turn today's uncaught `ValueError` from `int(port_str)` (~L78) into a new `ControllerValueError` naming the bad entry (2.8).
- `src/xceadmin/src/xcecommon/errors.py` — add `ControllerValueError(XceAdminError)` next to the existing controller-reachability errors (`ConnectionTimeoutError`, `TooManyRedirectsError`, `NoPrimaryError`).
- `src/xceadmin/src/xceadmin/cli.py` — change `_CONTROLLER_ENV_VAR` (~L109), which all 4 `--controller` options use, to the list `["XCE_CONTROLLER_PEERS", "XCEADMIN_CONTROLLER"]`; rewrite each option's help string, which today interpolates `_CONTROLLER_ENV_VAR` with an f-string and would print a raw Python list once it becomes one, and the precedence comment above it (~L102-108). Add a helper `_controller_source_note()` that returns `" (from $NAME)"` when `click.get_current_context().get_parameter_source("controller")` is `ParameterSource.ENVIRONMENT` (NAME = the first non-empty variable in the list, the same rule click applies) and `""` otherwise; append it to the two existing `cannot connect to {controller}` messages (~L172, ~L265), and, in the existing `except XceAdminError` handlers of the `--controller`-accepting commands, to the message when the error is one of `_CONTROLLER_ERRORS = (ControllerValueError, ConnectionTimeoutError, TooManyRedirectsError, NoPrimaryError)` (2.8). Errors unrelated to the controller value (e.g. `ClusterBinFileError`) keep their text unchanged.
- `src/xceadmin/tests/test_discovery.py` — new unit tests for `resolve_controllers()`'s `name=`-stripping and its `ControllerValueError` on a non-numeric port.
- `src/xceadmin/tests/test_cli.py` — new unit tests for the new env-var precedence (mirrors the existing `test_controller_option_falls_back_to_xceadmin_controller_env_var`/`test_controller_flag_overrides_xceadmin_controller_env_var` tests) and for 2.8's messages.
- `.kiro/steering/build-deployment.md` — document `XCE_CONTROLLER_PEERS`: components, value format, precedence, the "unset it to run standalone" opt-out, the controller's own-name rule (1.6), and that `sudo` (default `env_reset`) and systemd services do not inherit it (`sudo --preserve-env=XCE_CONTROLLER_PEERS`, or `Environment=` in the unit).
- `.claude/skills/xce-deploy-ha/SKILL.md` — note the variable's availability, that the generated daemon scripts still pass `--peers`, and that `scripts/xceadmin` now also exports `XCE_CONTROLLER_PEERS` (update the existing ~L330 description and the ~L518 troubleshooting row's "check `$TOP/scripts/xceadmin`" hint).
- `.claude/skills/xce-test-multihost/SKILL.md` — same note, at its three existing `--peers`-mentioning locations (single-node deploy step, `xcecd_virtual` flag-table note, Prerequisites' generated-script list).
## System Flows
@ -112,9 +150,13 @@ One new file (the regression script below); every other change is to an existing
```mermaid
flowchart TD
Start[Resolution point reached] --> HasFlag{Explicit flag or\nset command given?}
HasFlag -- yes --> UseFlag[Use that value\nnever consult env var]
HasFlag -- yes --> UseFlag[Use that value\nenv var not used as list]
HasFlag -- no --> HasNewEnv{XCE_CONTROLLER_PEERS\nnon-empty?}
HasNewEnv -- yes --> UseNewEnv[Use it\nxceadmin strips name prefix]
HasNewEnv -- yes --> IsCtrl{Component is controller?}
IsCtrl -- no --> UseNewEnv[Use it\nxceadmin strips name prefix]
IsCtrl -- yes --> SelfListed{Own name matches\na candidate?}
SelfListed -- yes --> UseNewEnv
SelfListed -- no --> Refuse[Refuse to start\nname source and remedies]
HasNewEnv -- no --> IsAdmin{Component is xceadmin?}
IsAdmin -- yes --> HasLegacyEnv{XCEADMIN_CONTROLLER\nnon-empty?}
IsAdmin -- no --> UseTodayDefault[Today's unchanged default\nstandalone / built-in literal]
@ -122,7 +164,7 @@ flowchart TD
HasLegacyEnv -- no --> UseTodayDefault
```
Both branches that resolve to a real candidate list (`UseFlag`, `UseNewEnv`, `UseLegacyEnv`) still pass through each component's existing, unchanged validation (`xce_parse_peers()`/`resolve_controllers()`); an invalid value is a hard error either way, distinguished by source only in the error message text (`1.5`).
All three branches that resolve to a real candidate list (`UseFlag`, `UseNewEnv`, `UseLegacyEnv`) still pass through each component's existing validation (`xce_parse_peers()` and the controller's odd-count rule, unchanged; `resolve_controllers()`, extended per 2.2/2.8); an invalid value is a hard error either way, distinguished by source only in the error message text (1.4, 2.8).
This resolution point is reached exactly once, at process startup, for controller/scheduler/worker/xceadmin. For harness/`xce_test` it is reached on *every* connection attempt (`enter_ctx()`), since `set masters ...` can run at any point in an interactive/scripted session — the diagram represents one such evaluation, not a startup-only decision, for that component.
@ -130,35 +172,43 @@ This resolution point is reached exactly once, at process startup, for controlle
| Requirement | Summary | Components | Interfaces | Flows |
|-------------|---------|------------|------------|-------|
| 1.1 | Recognize `XCE_CONTROLLER_PEERS` | controller/scheduler/worker/harness main.c/test.c | `xce_controller_peers_env()` | Flowchart: `HasNewEnv` |
| 1.2 | Explicit `--peers` wins, env var not consulted | controller/scheduler/worker main.c | existing `if (peers)` gate | Flowchart: `HasFlag` = yes |
| 1.3 | Env var used when flag omitted | controller/scheduler/worker main.c, harness test.c | `xce_controller_peers_env()` | Flowchart: `UseNewEnv` |
| 1.4 | Unset/empty env var ⇒ unchanged today's behavior | all 4 | same call sites | Flowchart: `UseTodayDefault` |
| 1.5 | Parse-error message names the source | controller/scheduler/worker main.c, harness test.c | existing error `fprintf`/`enter_ctx` message | n/a |
| 1.6 | `--peers` help text documents the variable | controller/scheduler/worker main.c, harness test.c | argp/getopt/settings help strings | n/a |
| 1.1 | Explicit candidate list wins, env var not used as candidate list | controller/scheduler/worker main.c; harness test.c (explicit `set masters`, interactive or rc-file) | existing `if (peers)` gate; `get_setting()` value-before-defval | Flowchart: `HasFlag` = yes |
| 1.2 | Env var used when no explicit list (harness: at each connection attempt) | controller/scheduler/worker main.c, harness test.c | `xce_controller_peers_env()`; harness `masters` `defval` | Flowchart: `HasNewEnv` → `UseNewEnv` |
| 1.3 | Unset/empty env var ⇒ unchanged today's behavior | all 4 | same call sites | Flowchart: `UseTodayDefault` |
| 1.4 | Parse/validation error (incl. controller even count) names the source | controller/scheduler/worker main.c, harness test.c | existing error `fprintf`s (parse, controller even-count)/`enter_ctx` message | n/a |
| 1.5 | `--peers` (harness: `set`) help text documents the variable | controller/scheduler/worker main.c, harness test.c | argp/getopt/settings help strings | n/a |
| 1.6 | Controller refuses an env-supplied list without its own name | controller main.c | `xce_peers_find_identity()` | Flowchart: `SelfListed` = no |
| 1.7 | Opt-out is "unset the variable"; help says so | controller/scheduler/worker main.c, harness test.c | help strings; `xce_controller_peers_env()` empty-as-unset | Flowchart: `HasNewEnv` = no |
| 1.8 | "No --peers" log line says the variable was unset or empty | controller/scheduler/worker main.c | existing `INFO`/log line text | Flowchart: `UseTodayDefault` |
| 2.1 | Derive controller list from env var | xceadmin cli.py, discovery.py | click `envvar=` list | Flowchart: `UseNewEnv` |
| 2.2 | Strip `name=` prefix | xceadmin discovery.py | `resolve_controllers()` | n/a |
| 2.2 | Strip `name=` prefix, any source incl. explicit `--controller` | xceadmin discovery.py | `resolve_controllers()` | n/a |
| 2.3 | Try each candidate in order | xceadmin discovery.py | `resolve_controllers`/`follow_redirects_any` (unmodified) | n/a |
| 2.4 | Explicit `--controller` wins | xceadmin cli.py | click parameter resolution (unmodified) | Flowchart: `HasFlag` = yes |
| 2.5 | Legacy `XCEADMIN_CONTROLLER` fallback unchanged | xceadmin cli.py | click `envvar=` list order | Flowchart: `HasLegacyEnv` |
| 2.6 | Both unset ⇒ hardcoded default unchanged | xceadmin cli.py | `_DEFAULT_CONTROLLER` (unmodified) | Flowchart: `UseTodayDefault` |
| 2.7 | `--controller` help text documents the variable | xceadmin cli.py | click option help strings | n/a |
| 3.1 | Steering doc describes the variable | `build-deployment.md` | n/a | n/a |
| 3.2 | HA deploy guidance notes availability | `xce-deploy-ha/SKILL.md` | n/a | n/a |
| 2.8 | Clean error for unreachable/malformed; names the env var source | xceadmin cli.py, discovery.py | `ControllerValueError`; `_controller_source_note()` | n/a |
| 3.1 | Steering doc describes the variable: components, format, precedence, opt-out (1.7), own-name rule (1.6), `sudo`/systemd not inheriting it | `build-deployment.md` | n/a | n/a |
| 3.2 | HA deploy guidance notes availability and the wrapper export | `xce-deploy-ha/SKILL.md` | n/a | n/a |
| 3.3 | Multi-host guidance notes availability | `xce-test-multihost/SKILL.md` | n/a | n/a |
| 4.1 | Wrapper exports the deployment's own list | `deploy-ha.sh` wrapper heredoc | `export XCE_CONTROLLER_PEERS="$PEERS_STR"` | n/a |
| 4.2 | Wrapper keeps `XCEADMIN_CONTROLLER` | `deploy-ha.sh` wrapper heredoc | existing export, unchanged | n/a |
| 4.3 | Explicit `--controller` still wins in the wrapper | xceadmin cli.py | click parameter resolution (unmodified) | Flowchart: `HasFlag` = yes |
## Components and Interfaces
| Component | Domain/Layer | Intent | Req Coverage | Key Dependencies | Contracts |
|-----------|--------------|--------|---------------|-------------------|-----------|
| `xce_controller_peers_env()` | libxce / C | Read `XCE_CONTROLLER_PEERS`, empty-as-unset | 1.1, 1.3, 1.4 | none (P0) | Service |
| controller/scheduler/worker `--peers` call sites | C daemon CLI | Combine flag + env var, preserve existing validation/help | 1.2-1.6 | `xce_controller_peers_env` (P0) | — |
| harness `masters` setting default | C CLI (settings table) | Compute `defval` from env var at startup | 1.1, 1.3, 1.4, 1.5, 1.6 | `xce_controller_peers_env` (P0) | State |
| `resolve_controllers()` extension | xceadmin / Python | Strip `name=` prefix per entry | 2.1, 2.2 | none (P0) | Service |
| xceadmin `--controller` options | xceadmin / Python (click) | Add `XCE_CONTROLLER_PEERS` ahead of legacy envvar | 2.1, 2.4-2.7 | `resolve_controllers` (P0), click (P0, external) | — |
| `xce_controller_peers_env()` | libxce / C | Read `XCE_CONTROLLER_PEERS`, empty-as-unset | 1.2, 1.3, 1.7 | none (P0) | Service |
| `xce_peers_find_identity()` | libxce / C | One rule for "which candidate is me" | 1.6 | none (P0) | Service |
| controller/scheduler/worker `--peers` call sites | C daemon CLI | Combine flag + env var, preserve existing validation/help; controller adds the 1.6 check | 1.1-1.8 | `xce_controller_peers_env` (P0), `xce_peers_find_identity` (P0, controller only) | — |
| harness `masters` setting default | C CLI (settings table) | Compute `defval` from env var at startup | 1.1-1.5, 1.7 | `xce_controller_peers_env` (P0) | State |
| `resolve_controllers()` extension | xceadmin / Python | Strip `name=` prefix per entry; clean error on a malformed entry | 2.1, 2.2, 2.8 | none (P0) | Service |
| xceadmin `--controller` options | xceadmin / Python (click) | Add `XCE_CONTROLLER_PEERS` ahead of legacy envvar; name the env source in errors | 2.1, 2.4-2.8 | `resolve_controllers` (P0), click (P0, external) | — |
| `scripts/xceadmin` wrapper export | `deploy-ha.sh` | Pin the wrapper to its own deployment | 4.1-4.3 | none | — |
| Documentation updates | Docs | Describe/announce the variable | 3.1-3.3 | none | — |
Only the two components introducing genuinely new logic — `xce_controller_peers_env()` and the `resolve_controllers()` extension — warrant a full contract block; the remaining rows are thin call-site wiring around existing, unchanged behavior.
Only the three components introducing genuinely new logic — `xce_controller_peers_env()`, `xce_peers_find_identity()` and the `resolve_controllers()` extension — warrant a full contract block; the remaining rows are thin call-site wiring around existing, unchanged behavior.
### libxce
@ -167,7 +217,7 @@ Only the two components introducing genuinely new logic — `xce_controller_peer
| Field | Detail |
|-------|--------|
| Intent | Sole read point for `XCE_CONTROLLER_PEERS`, normalizing empty-string to absent |
| Requirements | 1.1, 1.3, 1.4 |
| Requirements | 1.2, 1.3, 1.7 |
**Responsibilities & Constraints**
- Pure function: no side effects, no caching (the environment can only change between process invocations for these components, so no staleness concern).
@ -188,6 +238,33 @@ const char *xce_controller_peers_env(void);
- Postconditions: returns a pointer into the process environment (valid for the process lifetime) or `NULL`.
- Invariants: never returns a non-NULL empty string.
#### xce_peers_find_identity
| Field | Detail |
|-------|--------|
| Intent | Find the controller's own candidate in a parsed peers list |
| Requirements | 1.6 |
**Responsibilities & Constraints**
- Pure function over an already-parsed `xce_cluster_config_t`; no environment or global access.
- Matching rule, moved unchanged from `src/controller/cluster.c` (~L1784-1791 and ~L1890-1900, identical today): a candidate's key is its `name` when non-empty, else its `hostname`; the first candidate whose key equals `name` (compared with `strncmp(key, name, XCE_MAXNAME_LEN - 1)`) wins.
- Does not apply any default: "no match → last index" stays at the `my_priority` call site, "no match → refuse" at `cluster_open_listener()`, and the 1.6 check treats no match as an error.
**Dependencies**
- Inbound: controller `main.c` startup check (P0), controller `cluster.c` `my_priority` assignment and listen-port lookup (P0).
- Outbound: none.
**Contracts**: Service [x] / API [ ] / Event [ ] / Batch [ ] / State [ ]
##### Service Interface
```c
/* src/include/xce/cluster.h */
int xce_peers_find_identity(const xce_cluster_config_t *cfg, const char *name);
```
- Preconditions: `cfg` non-NULL.
- Postconditions: returns the index (0..`candidates_count`-1) of the first matching candidate, or -1 when `name` is NULL or nothing matches. An empty `name` is compared like any other (today's loops do the same: it matches an unnamed entry with an empty hostname, e.g. from `:5`), so the invariant below holds for every input.
- Invariants: for every input, `idx >= 0 ? idx : candidates_count - 1` equals what `cluster.c` computes for `my_priority` today, and `idx >= 0` equals the listener's `matched` today.
### xceadmin
#### resolve_controllers (extended)
@ -195,11 +272,13 @@ const char *xce_controller_peers_env(void);
| Field | Detail |
|-------|--------|
| Intent | Parse a `--controller`-shaped value into an ordered candidate list, now tolerant of `name=` prefixes |
| Requirements | 2.1, 2.2 |
| Requirements | 2.1, 2.2, 2.8 |
**Responsibilities & Constraints**
- Existing behavior (whitespace stripping, blank-entry skipping, bare-host default port) is unchanged.
- New: for each comma-separated entry, if it contains `=`, discard everything up to and including the first `=` before applying the existing `rsplit(":", 1)`.
- New: a port that is not an integer raises `ControllerValueError` (an `XceAdminError` subclass) naming the entry, instead of today's uncaught `ValueError`.
- Both changes apply to every caller, an explicit `--controller` value included.
**Dependencies**
- Inbound: xceadmin `--controller` options (P0), any future caller expecting `name=host:port` tolerance.
@ -212,7 +291,7 @@ const char *xce_controller_peers_env(void);
def resolve_controllers(controller: str) -> list[tuple[str, int]]: ...
```
- Preconditions: `controller` is a comma-separated string of `[name=]host[:port]` entries.
- Postconditions: returns an ordered list of `(host, port)` tuples, `name=` prefixes (if any) stripped.
- Postconditions: returns an ordered list of `(host, port)` tuples, `name=` prefixes (if any) stripped; raises `ControllerValueError` on a malformed entry.
- Invariants: a bare `host:port` entry (today's only supported shape) parses identically to before.
**Implementation Notes**
@ -223,22 +302,31 @@ def resolve_controllers(controller: str) -> list[tuple[str, int]]: ...
## Error Handling
### Error Strategy
Every existing hard-error-on-invalid-value behavior is preserved exactly; the only addition is naming the value's source in the message text, per 1.5.
Every existing hard-error-on-invalid-value behavior is preserved exactly, with the value's source named in the message text (1.4). The one new error is the controller's own-name check for an env-supplied list (1.6).
### Error Categories and Responses
- **Invalid candidate list, from an explicit `--peers`/`--controller`/`set masters` value**: unchanged existing message and exit behavior.
- **Invalid candidate list, from `XCE_CONTROLLER_PEERS`**: same parse failure, same exit behavior, message additionally names the environment variable as the source (controller/scheduler/worker: adjust the existing `fprintf`; harness: `enter_ctx()`'s existing "invalid 'masters' setting" message, extended to check `settings[i].value == NULL` for the `"masters"` entry at the point of failure — reusing `get_setting()`'s own existing value-before-defval precedence check, not a new tracking variable — to say "default (from XCE_CONTROLLER_PEERS)" instead of "explicit `set masters`" when true).
- **Invalid candidate list, from legacy `XCEADMIN_CONTROLLER`**: unchanged (out of scope — 1.5/2.x do not require source-naming for the legacy variable, only for `XCE_CONTROLLER_PEERS` and the explicit flag).
- **Invalid candidate list, from an explicit `--peers` value**: unchanged exit behavior; message names `--peers` as the source (most already do; scheduler's and worker's `xce_cluster_config_validate()` messages gain it).
- **Invalid candidate list, from an explicit `set masters`**: unchanged behavior; `enter_ctx()`'s message names the source as described in the next bullet.
- **Invalid or unreachable value, from an explicit `--controller`**: today's unreachable messages unchanged; a malformed entry now gives the 2.8 clean error instead of a traceback. No source note is added, since the operator typed the value.
- **Invalid candidate list, from `XCE_CONTROLLER_PEERS`**: same parse or validation failure (including the controller's even-count rejection), same exit behavior, message additionally names the environment variable as the source (controller/scheduler/worker: adjust the existing `fprintf`; harness: `enter_ctx()`'s existing "invalid 'masters' setting" message, extended to say "default (from XCE_CONTROLLER_PEERS)" when the `"masters"` entry's `settings[i].value == NULL` (no `set masters` ran, reusing `get_setting()`'s own value-before-defval precedence) **and** `xce_controller_peers_env() != NULL`; "explicit `set masters`" when `value != NULL`; and today's wording otherwise (the built-in `localhost:9990` literal, which cannot fail to parse today, is then never blamed on the variable).
- **Controller name not in an env-supplied list (1.6)**: `main.c` prints, to stderr, e.g. `xce_controller: XCE_CONTROLLER_PEERS (a=h1:1,b=h2:2) has no candidate named 'node7'; pass --myname <one of: a, b> or unset XCE_CONTROLLER_PEERS to run standalone`, then `return 1` before the cert pre-flight and `myapp_run()`. This replaces, for the env-var case only, the later `cluster_open_listener()` refusal, which blames "--peers", writes only to the log file, and exits with status 0. Candidates without a `name=` are listed by hostname.
- **xceadmin, unreachable candidates or a malformed entry (2.8)**: `click.ClickException` (exit 1) with today's text (`cannot connect to {controller}: ...` for `OSError`, or the `e.message` of `ConnectionTimeoutError`/`TooManyRedirectsError`/`NoPrimaryError`), or `invalid --controller entry '...'` for `ControllerValueError`, followed by ` (from $XCE_CONTROLLER_PEERS)` or ` (from $XCEADMIN_CONTROLLER)` when the value came from the environment. Today a malformed port is an uncaught `ValueError` traceback.
### Monitoring
No new logging/metrics infrastructure; each component's existing startup log line (e.g. scheduler's `INFO("HA cluster config: ...")` / `INFO("No --peers flag; ...")`) is extended only to remain accurate about which source supplied the value it's reporting.
No new logging/metrics infrastructure; each component's existing startup log line (e.g. scheduler's `INFO("HA cluster config: ...")` / `INFO("No --peers flag; ...")`) is extended only to remain accurate about which source supplied the value it's reporting. The three "No --peers flag" lines (controller `main.c` ~L486, scheduler ~L435, worker ~L501) become "No --peers flag and XCE_CONTROLLER_PEERS unset or empty; ..." (1.8). This is the operator's only clue when `sudo`'s default `env_reset` or a systemd unit drops the variable — the main risk for the production worker, which `check_user()` requires to run as root (`deploy-ha.sh` ~L470 starts it with `sudo`).
## Testing Strategy
### Unit Tests
- `xce_controller_peers_env()` (`src/lib/testsuite/test_cluster.c`): unset → `NULL`; empty string → `NULL`; non-empty → returned unchanged. (1.1, 1.3, 1.4)
- `xce_controller_peers_env()` (`src/lib/testsuite/test_cluster.c`): unset → `NULL`; empty string → `NULL`; non-empty → returned unchanged. (1.2, 1.3, 1.7)
- `xce_peers_find_identity()` (`src/lib/testsuite/test_cluster.c`): the cases listed in File Structure Plan, each written first as a failing test. The existing controller tests that go through `my_priority` and `cluster_open_listener()` must stay green after `cluster.c` switches to the helper. (1.6)
- `resolve_controllers()` extension (`src/xceadmin/tests/test_discovery.py`): single `name=host:port`; mixed bare/named multi-candidate list; existing bare-host cases still pass unchanged. (2.2)
- xceadmin `--controller` option precedence (`src/xceadmin/tests/test_cli.py`): `XCE_CONTROLLER_PEERS` set alone; both `XCE_CONTROLLER_PEERS` and `XCEADMIN_CONTROLLER` set (new wins); explicit `--controller` overrides both; neither set (existing default). (2.1, 2.4-2.6) All existing `test_cli.py` cases that assert `XCEADMIN_CONTROLLER`-only or no-env-var behavior (e.g. `test_controller_option_falls_back_to_xceadmin_controller_env_var`) must explicitly `monkeypatch.delenv("XCE_CONTROLLER_PEERS", raising=False)`, so the new variable's introduction cannot make them order-dependent on ambient environment state.
- xceadmin `--controller` option precedence (`src/xceadmin/tests/test_cli.py`): `XCE_CONTROLLER_PEERS` set alone; both `XCE_CONTROLLER_PEERS` and `XCEADMIN_CONTROLLER` set (new wins); explicit `--controller` overrides both; neither set (existing default). (2.1, 2.4-2.6) For 2.3: with
`XCE_CONTROLLER_PEERS="a=127.0.0.1:1,b=127.0.0.1:2,127.0.0.1:3"` and no
`--controller`, mock the send function under `follow_redirects_any`, make
the first candidate raise a connection error, and assert the candidates
arrive in env order with `name=` stripped and that the second one is
tried next. (2.3) For 2.8: an unreachable value from each of `XCE_CONTROLLER_PEERS`, `XCEADMIN_CONTROLLER` and an explicit `--controller` (message names the variable in the first two only), and a non-numeric port giving a clean error rather than a traceback. Run the unreachable cases through `config push` with a real, readable cluster-bin file (`push_config()` reads it before resolving controllers, so a missing file raises `ClusterBinFileError` first), or patch out the retry sleep, since the `scheduler record` commands retry `NoPrimaryError` 8 times with a 2 s pause (`scheduler.py` ~L131-132). All existing `test_cli.py` cases that assert `XCEADMIN_CONTROLLER`-only or no-env-var behavior (e.g. `test_controller_option_falls_back_to_xceadmin_controller_env_var`) must explicitly `monkeypatch.delenv("XCE_CONTROLLER_PEERS", raising=False)`, so the new variable's introduction cannot make them order-dependent on ambient environment state.
### Scripted Regression Verification
None of controller/scheduler/worker/harness's `main()`-level argument or
@ -252,32 +340,91 @@ repeatable script, `src/tools/verify-controller-peers-env.sh`, following
and asserting on their output — but deliberately lightweight: it asserts on
each component's **startup log/settings-dump line**, not on a live,
reachable cluster, since resolving the candidate list happens, and is
observable, entirely before any connection attempt succeeds or fails:
observable, entirely before any connection attempt succeeds or fails.
The worker cases use `xcecd_virtual`, since the production `xcecd`
requires root (`check_user()`), and grep the worker's captured **stdout**,
not `xcecd-<name>.log`: its "HA cluster config" / "No --peers flag" lines
(`worker/src/main.c` ~L493-503) are logged before its log file is set up
(~L529-544), so the logging library still writes them to stdout. The
controller and scheduler set up their log files earlier (controller
`main.c` ~L393, scheduler ~L332), so their cases grep the log file. Controller cases always use an odd
candidate count and pass `--myname` naming a candidate unless the case
is about the 1.6 check, so the even-count rule and 1.6 do not mask what
the case is testing.
- controller/scheduler/worker: with `XCE_CONTROLLER_PEERS` set (e.g.
`a=host1:1,b=host2:2`) and no `--peers`, start briefly and grep the log
for the existing "N candidate(s)" line already logged today for an
explicit `--peers` value, asserting `N` and the candidate identity match
the env var's content — no reachable target required. (1.1, 1.3)
- Same three, with both `XCE_CONTROLLER_PEERS` set to one value and
`--peers` set to a different one; assert the log reflects the `--peers`
value, not the env var's. (1.2)
- Same three, with neither set; assert the existing "No --peers flag"
(or worker's "single-controller path") log line, unchanged. (1.4)
`a=localhost:1,b=localhost:2,c=localhost:3`; controller also
`--myname a`) and no `--peers`, start briefly and grep the log (worker:
stdout) for the line each already logs today for an explicit `--peers`
value — controller `HA mode: N controller candidates, identity=...`
(`main.c` ~L476), scheduler and worker `HA cluster config: N
candidate(s), ...` (scheduler ~L431, worker ~L493) — asserting `N`
matches the env var's content — no reachable target required. (1.2)
- Same three, with both `XCE_CONTROLLER_PEERS` set to one value (3
candidates) and `--peers` set to a different one (1 candidate, which the
controller's `--myname` names); assert the log reflects the `--peers`
value, not the env var's (worker: stdout). (1.1)
- Same three, with neither set; assert the "No --peers flag" (or
worker's "single-controller path", on stdout) log line now also says
`XCE_CONTROLLER_PEERS` was unset or empty, and that behavior is otherwise
unchanged. (1.3, 1.8)
- scheduler with `--sim <file>` and `XCE_CONTROLLER_PEERS` set: assert the
process never logs an "HA cluster config"/"No --peers flag" line at all
(the `--sim` path returns before that code is reached — this is a
by-construction guarantee already, not new logic; the script asserts it
stays true). (Non-Goals cross-check)
- harness/`xce_test`: run non-interactively with a script containing only
- harness/`xce_test` (always with `--norc`, so a developer's own rc file
cannot change the result): run non-interactively with a script containing only
`set` (no arguments) — `cmd_set`'s existing argc==1 branch prints every
setting's current value/default with **no connection attempt** — first
with `XCE_CONTROLLER_PEERS` set and no prior `set masters`, asserting the
printed `masters` default reflects the env var; then with an explicit
`set masters ...` line before the bare `set`, asserting the printed
*value* (not default) reflects the explicit override. (1.1, 1.3)
- Trigger a parse error via each of an invalid explicit flag and an invalid
`XCE_CONTROLLER_PEERS` value on one component (e.g. controller); assert
the two error messages are distinguishable by source. (1.5)
*value* (not default) reflects the explicit override. (1.1, 1.2)
- For each of controller, scheduler and worker, trigger a parse error
once with an invalid explicit `--peers` and once with an invalid
`XCE_CONTROLLER_PEERS` and no `--peers`; assert on **stderr** (all three
write these errors with `fprintf(stderr, ...)`) that the two messages
name different sources. For the controller, repeat with a 2-candidate
value from each source (even-count message); for scheduler and worker,
repeat with `a=localhost:0`, which parses but fails
`xce_cluster_config_validate()` (port 0). (1.4)
- harness/`xce_test --norc`, each script ending in a connecting command
(`client`): (a) `set masters bad` with `XCE_CONTROLLER_PEERS` unset →
message says explicit `set masters`; (b) no `set masters`,
`XCE_CONTROLLER_PEERS=bad` → message says default from
`XCE_CONTROLLER_PEERS`; (c) `set masters bad` with
`XCE_CONTROLLER_PEERS` set to a valid value → still says explicit
`set masters` (explicit wins, 1.1). (1.1, 1.4)
- controller with `XCE_CONTROLLER_PEERS=a=localhost:1,b=localhost:2,c=localhost:3`:
with `--myname zz`, assert a non-zero exit and the 1.6 message (names
`zz`, the variable, and the remedies); with `--myname b`, assert the
"HA mode: 3 controller candidates" line and no 1.6 message; with
`--myname zz --peers a=localhost:1,b=localhost:2,c=localhost:3`, assert
no 1.6 message (explicit-flag behavior unchanged). (1.1, 1.6)
- controller with `XCE_CONTROLLER_PEERS=""` and no `--peers`: assert the
"No --peers flag ... running as single-node primary" line. (1.7)
- Help text (1.5, 1.7): run `xce_controller --help`, `xce_scheduler --help`
and `xcecd_virtual -h`, and feed `help set` to `xce_test --norc`; assert
each output names `XCE_CONTROLLER_PEERS` and says that unsetting or
emptying it opts out for one invocation. For xceadmin, `test_cli.py`
asserts the same name in each `--controller` option's `--help` output
(2.7).
- `deploy-ha.sh` wrapper: run `deploy-ha.sh` with the built controller
binary and a peers list on free high ports (it refuses ports already in
use, `deploy-ha.sh` ~L204-225) into a scratch `$TOP` (generation only,
nothing started), copy
`$TOP/scripts/xceadmin` with its final `exec` line replaced by
`printenv XCE_CONTROLLER_PEERS XCEADMIN_CONTROLLER`, and run the copy
with `XCE_CONTROLLER_PEERS=bogus:1` in the caller's environment; assert
it prints the deployment's own `--peers` list and first candidate. If
the xceadmin wheel is missing (so no wrapper is generated), this case
fails with a message naming `/xce-build xceadmin`, rather than being
skipped. (4.1, 4.2)
- xceadmin (`test_cli.py`): with `XCE_CONTROLLER_PEERS` and
`XCEADMIN_CONTROLLER` both set and an explicit `--controller`, the flag
wins. (4.3; this runs the CLI directly rather than through the
generated wrapper, which is sufficient because the wrapper only exports
variables and passes its arguments through unchanged)
This script is a new addition to the File Structure Plan and is intended to
be re-run on any future change to these five components' startup argument