# Requirements Document ## Introduction Operators and deploy/test scripts must retype the full controller candidate list on every invocation of controller, scheduler and worker (their `--peers` option) and of harness/xce_test (its `set masters` setting) — there is no way to set a default once per session or environment for these four peers-based components today. The candidate list is optional for all four, and leaving it out intentionally selects each component's existing standalone/single-controller behavior (confirmed: the controller, scheduler and worker startup logs already name this an intentional path, not an error) — that behavior must keep working unchanged, apart from added wording in its startup log line (1.8). The fifth tool, xceadmin, already solved the equivalent problem for its own `--controller` option via the `XCEADMIN_CONTROLLER` environment variable (bug-1566), and, separately, xceadmin's `--controller` option can already accept a comma-separated list of candidates and will try each in order until one is reachable (bug-1821), rather than only a single target. This feature introduces one new environment variable, `XCE_CONTROLLER_PEERS`, read as a fallback default by all five components (controller, scheduler, worker, harness/xce_test, and xceadmin) whenever no explicit value is given (`--peers`, `set masters`, or `--controller`) — and only then; an explicit value always wins, and the environment variable is never used as the candidate list when one is given. Explicit `--peers` and `set masters` values behave exactly as today, apart from the error wording in 1.4; an explicit `--controller` value gains the two parsing changes described under Adjacent expectations (2.2, 2.8). xceadmin's existing `XCEADMIN_CONTROLLER` variable keeps working as a lower-priority fallback beneath the new variable, unchanged apart from the parsing changes in 2.2 and 2.8 (which apply to every source of the `--controller` value). For the controller specifically, `--peers` is not just a list of servers to try: it defines the HA (Raft) cluster membership, and the controller finds its own place in that list by matching its name. A value inherited from the environment, rather than typed next to `--myname`, must therefore never let a controller join a cluster it cannot identify itself in. ## Boundary Context - **In scope**: A single new environment variable, `XCE_CONTROLLER_PEERS`, providing the default connection target for controller, scheduler and worker (via their existing `--peers` option), harness/xce_test (via its existing `masters` setting) and xceadmin (via its existing `--controller` option), used only when no explicit value is given. - **Out of scope**: - The apiserver (its controller connection comes from a configuration file, not a command-line option) and xcecli's `--url` option (a different target — the apiserver, not a controller). Both are unaffected. - Any change to `src/tools/deploy-ha.sh`'s generated daemon scripts, which continue to pass `--peers` explicitly exactly as they do today. One exception: the generated `scripts/xceadmin` wrapper selects its deployment's controllers only through the `XCEADMIN_CONTROLLER` environment variable, which the new variable outranks, so it also exports `XCE_CONTROLLER_PEERS` (Requirement 4). - Any change to what each component does today when no explicit value is given and no environment variable applies either (today's standalone/single-controller behavior is preserved, apart from the log wording in 1.8). - **Adjacent expectations**: xceadmin's existing in-order, try-until-reachable failover for `--controller` (bug-1821) and its existing `XCEADMIN_CONTROLLER` fallback (bug-1566) keep working as they do today, apart from the parsing changes below. xceadmin's parsing of a `--controller`-shaped value changes in two ways, whichever source the value comes from (an explicit `--controller` included): a `name=` prefix on an entry is removed (2.2), and a malformed entry is reported as an error rather than a crash (2.8). Today an explicit `--controller ctrl-1=host:port` fails as an unresolvable hostname; after this feature it reaches `host:port`. ## Requirements ### Requirement 1: Default candidate list for peers-based components **Objective:** As an operator running controller, scheduler, worker, or harness/xce_test, I want a default controller candidate list available via an environment variable, so that I don't have to retype it on every invocation while still being able to run any single invocation without it. **Terms used below:** - *Peers-based component*: controller, scheduler, worker, or harness/xce_test. - *Explicit candidate list*: for controller, scheduler and worker, the `--peers` argument; for harness/xce_test, a `set masters ...` command, whether typed interactively, in a script, or in its rc file. - *Resolution point*: for controller, scheduler and worker, process startup; for harness/xce_test, each connection attempt (`client`, `worker` and similar commands), since `set masters` can run at any time in a session. - *Own name* (controller only): the `--myname` value, or the host name the controller uses when `--myname` is not given. #### Acceptance Criteria 1. When the peers-based component reaches its resolution point with an explicit candidate list, the peers-based component shall use that list and shall not use `XCE_CONTROLLER_PEERS` as its candidate list (it may still read the variable, e.g. to show it as harness/xce_test's `masters` default in `set` output). 2. When the peers-based component reaches its resolution point without an explicit candidate list and `XCE_CONTROLLER_PEERS` is set to a non-empty value, the peers-based component shall use that value as its candidate list, exactly as if it had been given as an explicit candidate list, except for the error-source wording in 1.4, the controller's own-name check in 1.6, and, for harness/xce_test, that the value is shown as the `masters` setting's default rather than its value in `set` output. 3. When the peers-based component reaches its resolution point without an explicit candidate list and `XCE_CONTROLLER_PEERS` is unset or empty, the peers-based component shall behave exactly as it does today with no explicit candidate list (controller: standalone single-node primary; scheduler and worker: their single-controller path; harness/xce_test: its built-in `masters` default), apart from the log wording in 1.8. 4. If the candidate list in use, from either an explicit candidate list or `XCE_CONTROLLER_PEERS`, fails parsing or the component's own validation of it (including the controller's rule that the number of candidates must be odd), the peers-based component shall report an error message that identifies whether the value came from the explicit candidate list (`--peers`, or `set masters` for harness/xce_test) or from the `XCE_CONTROLLER_PEERS` environment variable, and shall keep today's failure behavior for that error. 5. The peers-based component's help/usage text for its candidate list (`--peers` for controller, scheduler and worker; the `masters` setting in harness/xce_test's `set` help) shall document `XCE_CONTROLLER_PEERS` as an environment-variable fallback for it. 6. If the controller takes its candidate list from `XCE_CONTROLLER_PEERS` and its own name matches no candidate in that list, the controller shall exit with a non-zero status before opening any network connection, writing an error message to stderr that states the controller's own name, names `XCE_CONTROLLER_PEERS` as the source, and the two remedies: pass `--myname` naming one of the candidates, or unset `XCE_CONTROLLER_PEERS` to run standalone. When the candidate list comes from an explicit `--peers` argument, the controller's identity handling is unchanged. 7. The peers-based component shall treat an empty `XCE_CONTROLLER_PEERS` the same as an unset one, so that unsetting or emptying the variable for one invocation gives that invocation the behavior in 1.3; no separate opt-out option is added, and the help text from 1.5 shall describe this way of opting out. 8. When the controller, scheduler or worker starts with neither an explicit `--peers` argument nor a non-empty `XCE_CONTROLLER_PEERS`, its existing startup log line for that case shall also say that `XCE_CONTROLLER_PEERS` was unset or empty, so an operator whose variable was dropped (e.g. by `sudo`) can tell from the log why the standalone or single-controller path was taken. ### Requirement 2: Default controller candidate list for xceadmin **Objective:** As an xceadmin operator, I want `--controller` to fall back to the same environment variable the other XCE components use, so that one variable configures every tool's default connection target. #### Acceptance Criteria 1. When an xceadmin subcommand that accepts `--controller` runs without an explicit `--controller` option and `XCE_CONTROLLER_PEERS` is set to a non-empty value, xceadmin shall derive its controller candidate list from `XCE_CONTROLLER_PEERS`. 2. Where a candidate entry carries a `name=` prefix, xceadmin shall remove the prefix before treating that entry as a controller candidate. This applies to every source of the `--controller` value, an explicit `--controller` included. 3. When xceadmin derives its controller candidate list from `XCE_CONTROLLER_PEERS`, xceadmin shall attempt each resulting candidate in the same order the variable lists them, moving to the next candidate only when the current one is unreachable — the same behavior xceadmin already provides for an explicit multi-candidate `--controller` value. 4. When an xceadmin subcommand that accepts `--controller` runs with an explicit `--controller` option, xceadmin shall use that option's value (parsed as in 2.2 and 2.8) and shall not consult `XCE_CONTROLLER_PEERS` or `XCEADMIN_CONTROLLER`. 5. When an xceadmin subcommand that accepts `--controller` runs without an explicit `--controller` option, and `XCE_CONTROLLER_PEERS` is unset or empty, and `XCEADMIN_CONTROLLER` is set to a non-empty value, xceadmin shall use `XCEADMIN_CONTROLLER` exactly as it does today, apart from the parsing changes in 2.2 and 2.8. 6. When an xceadmin subcommand that accepts `--controller` runs without an explicit `--controller` option, and both `XCE_CONTROLLER_PEERS` and `XCEADMIN_CONTROLLER` are unset or empty, xceadmin shall fall back to its existing hardcoded default controller target, unchanged. 7. The `--controller` option's help text shall document `XCE_CONTROLLER_PEERS` as its preferred environment-variable fallback, alongside the existing `XCEADMIN_CONTROLLER` mention. 8. If xceadmin cannot reach any controller candidate, or a candidate entry is malformed (e.g. a non-numeric port), xceadmin shall exit with an error message rather than an unhandled exception. When the `--controller` value came from `XCE_CONTROLLER_PEERS` or `XCEADMIN_CONTROLLER` rather than an explicit `--controller` option, that message shall name the environment variable it came from. For a malformed entry, whatever the source, the message shall quote the malformed entry. ### Requirement 3: Documentation of the new environment variable **Objective:** As an operator or future contributor, I want the project's build/deployment documentation to describe `XCE_CONTROLLER_PEERS`, so that I can discover and use it without reading source code. #### Acceptance Criteria 1. The project's build/deployment steering documentation shall describe `XCE_CONTROLLER_PEERS`: which components read it, its value format, its precedence relative to each component's explicit value (`--peers`, `set masters`, or `--controller`), how to opt out for one invocation (1.7), the controller's own-name rule (1.6), and that it is not inherited by a process started through `sudo` (default `env_reset`) or by a systemd service, with the way to pass it in each case (e.g. `sudo --preserve-env=XCE_CONTROLLER_PEERS ...`, or `Environment=` in the unit). This matters most for the production worker, which must run as root. 2. The HA cluster deployment guidance shall note that `XCE_CONTROLLER_PEERS` is available as an alternative to passing `--peers`/`--controller` explicitly, that the generated daemon scripts still pass `--peers` explicitly, and that the generated `scripts/xceadmin` wrapper exports `XCE_CONTROLLER_PEERS` for its own deployment (Requirement 4). 3. The multi-host verification guidance shall note the same availability wherever it currently documents driving deployment via `--peers`, without stating or implying any change to that verification's own generated invocations. ### Requirement 4: Generated xceadmin wrapper is not shadowed by an ambient variable **Objective:** As an operator who has `XCE_CONTROLLER_PEERS` set in my shell for another cluster, I want a deployment's generated `scripts/xceadmin` wrapper to keep reaching that deployment's own controllers, so that the new variable does not bring back bug-1566's wrong-controller symptom. #### Acceptance Criteria 1. The `scripts/xceadmin` wrapper generated by `deploy-ha.sh` shall export `XCE_CONTROLLER_PEERS` set to the deployment's own full `--peers` candidate list before starting xceadmin, overriding any value inherited from the caller's environment. 2. The generated wrapper shall keep exporting `XCEADMIN_CONTROLLER` exactly as it does today. 3. An explicit `--controller` option passed to the generated wrapper shall still take precedence over both exported variables.