docs(controller-peers-env-fallback): design review fixes and requirements revision

Design review (NO-GO -> GO) and iterated requirements review until clean:

- Controller refuses to start (non-zero exit, stderr, before any network
  I/O) when an env-supplied XCE_CONTROLLER_PEERS lacks its own name (1.6);
  new libxce xce_peers_find_identity() replaces both identity loops in
  controller/cluster.c so the check and my_priority/listen port agree.
- deploy-ha.sh's generated scripts/xceadmin wrapper exports
  XCE_CONTROLLER_PEERS so an ambient value cannot redirect it (Req 4).
- Opt-out is unsetting/emptying the variable (1.7); "No --peers" log
  lines say it was unset or empty (1.8); docs cover sudo/systemd not
  inheriting it (3.1).
- Requirement 1 defines explicit candidate list / resolution point /
  own name so harness (set masters) is covered precisely; error messages
  name the value's source incl. validation failures (1.4).
- xceadmin: name= stripping and a clean malformed-entry error apply to
  every --controller source; errors name the env var source (2.2, 2.8).
- Test plan fixed against real code (stderr vs stdout, log texts,
  xcecd_virtual, --norc, free ports, failover order test).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

.kiro/specs/controller-peers-env-fallback/requirements.md

@@ -2,15 +2,17 @@
 
 ## Introduction
 
-Operators and deploy/test scripts must retype the full `--peers` candidate
-list on every invocation of controller, scheduler, worker, and
-harness/xce_test, and the `--controller` target on every invocation of
-xceadmin — there is no way to set a default once per session or
-environment for the four `--peers`-based components today. `--peers` is
-optional for all four, and omitting it intentionally selects each
-component's existing standalone/single-controller behavior (confirmed:
-each component's own startup log already names this an intentional path,
-not an error) — that behavior must keep working unchanged. xceadmin
+Operators and deploy/test scripts must retype the full controller
+candidate list on every invocation of controller, scheduler and worker
+(their `--peers` option) and of harness/xce_test (its `set masters`
+setting) — there is no way to set a default once per session or
+environment for these four peers-based components today. The candidate
+list is optional for all four,
+and leaving it out intentionally selects each component's existing
+standalone/single-controller behavior (confirmed: the controller,
+scheduler and worker startup logs already name this an intentional path,
+not an error) — that behavior must keep working unchanged, apart from
+added wording in its startup log line (1.8). The fifth tool, xceadmin,
 already solved the equivalent problem for its own `--controller` option via
 the `XCEADMIN_CONTROLLER` environment variable (bug-1566), and, separately,
 xceadmin's `--controller` option can already accept a comma-separated list
@@ -20,69 +22,129 @@ rather than only a single target.
 This feature introduces one new environment variable,
 `XCE_CONTROLLER_PEERS`, read as a fallback default by all five components
 (controller, scheduler, worker, harness/xce_test, and xceadmin) whenever
-their respective connection-target option (`--peers` or `--controller`) is
-not explicitly given on the command line — and only then; an explicit
-option value is always used exactly as it is today, with the environment
-variable never consulted. xceadmin's existing `XCEADMIN_CONTROLLER`
-variable keeps working exactly as it does today, as a lower-priority
-fallback beneath the new variable.
+no explicit value is given (`--peers`, `set masters`, or `--controller`)
+— and only then; an explicit value always wins, and the environment
+variable is never used as the candidate list when one is given. Explicit `--peers` and
+`set masters` values behave exactly as today, apart from the error wording
+in 1.4; an explicit `--controller`
+value gains the two parsing changes described under Adjacent expectations
+(2.2, 2.8). xceadmin's existing `XCEADMIN_CONTROLLER` variable keeps
+working as a lower-priority fallback beneath the new variable, unchanged
+apart from the parsing changes in 2.2 and 2.8 (which apply to every source
+of the `--controller` value).
+
+For the controller specifically, `--peers` is not just a list of servers to
+try: it defines the HA (Raft) cluster membership, and the controller finds
+its own place in that list by matching its name. A value inherited from the
+environment, rather than typed next to `--myname`, must therefore never let
+a controller join a cluster it cannot identify itself in.
 
 ## Boundary Context
 
 - **In scope**: A single new environment variable, `XCE_CONTROLLER_PEERS`,
-  providing the default connection target for controller, scheduler,
-  worker, harness/xce_test (via their existing `--peers` option) and
-  xceadmin (via its existing `--controller` option), consulted only when
-  the corresponding command-line option is omitted.
-- **Out of scope**: The apiserver (its controller connection comes from a
-  configuration file, not a command-line option, and is unaffected) and
-  xcecli's `--url` option (a different target — the apiserver, not a
-  controller — and unaffected). Any change to `src/tools/deploy-ha.sh`'s
-  generated scripts, which continue to pass `--peers`/`--controller`
-  explicitly exactly as they do today. Any change to what each component
-  does today when its connection-target option is omitted and no
-  environment variable applies either (today's standalone/single-controller
-  behavior is preserved exactly).
-- **Adjacent expectations**: xceadmin's existing multi-candidate,
-  try-until-reachable behavior for `--controller` (bug-1821) and its
-  existing `XCEADMIN_CONTROLLER` fallback (bug-1566) are both reused
-  unmodified — this feature only changes what value feeds into that
-  already-existing behavior, never how that behavior itself works.
+  providing the default connection target for controller, scheduler and
+  worker (via their existing `--peers` option), harness/xce_test (via its
+  existing `masters` setting) and xceadmin (via its existing `--controller`
+  option), used only when no explicit value is given.
+- **Out of scope**:
+  - The apiserver (its controller connection comes from a configuration
+    file, not a command-line option) and xcecli's `--url` option (a
+    different target — the apiserver, not a controller). Both are
+    unaffected.
+  - Any change to `src/tools/deploy-ha.sh`'s generated daemon scripts,
+    which continue to pass `--peers` explicitly exactly as they do today.
+    One exception: the generated `scripts/xceadmin` wrapper selects its
+    deployment's controllers only through the `XCEADMIN_CONTROLLER`
+    environment variable, which the new variable outranks, so it also
+    exports `XCE_CONTROLLER_PEERS` (Requirement 4).
+  - Any change to what each component does today when no explicit value
+    is given and no environment variable applies either (today's
+    standalone/single-controller behavior is preserved, apart from the log
+    wording in 1.8).
+- **Adjacent expectations**: xceadmin's existing in-order,
+  try-until-reachable failover for `--controller` (bug-1821) and its
+  existing `XCEADMIN_CONTROLLER` fallback (bug-1566) keep working as they
+  do today, apart from the parsing changes below. xceadmin's parsing of a `--controller`-shaped value changes in
+  two ways, whichever source the value comes from (an explicit
+  `--controller` included): a `name=` prefix on an entry is removed (2.2),
+  and a malformed entry is reported as an error rather than a crash (2.8).
+  Today an explicit `--controller ctrl-1=host:port` fails as an
+  unresolvable hostname; after this feature it reaches `host:port`.
 
 ## Requirements
 
-### Requirement 1: Default peers candidate list for --peers-based components
+### Requirement 1: Default candidate list for peers-based components
 
 **Objective:** As an operator running controller, scheduler, worker, or
-harness/xce_test, I want a default `--peers` candidate list available via
+harness/xce_test, I want a default controller candidate list available via
 an environment variable, so that I don't have to retype it on every
 invocation while still being able to run any single invocation without it.
 
+**Terms used below:**
+- *Peers-based component*: controller, scheduler, worker, or
+  harness/xce_test.
+- *Explicit candidate list*: for controller, scheduler and worker, the
+  `--peers` argument; for harness/xce_test, a `set masters ...` command,
+  whether typed interactively, in a script, or in its rc file.
+- *Resolution point*: for controller, scheduler and worker, process
+  startup; for harness/xce_test, each connection attempt (`client`,
+  `worker` and similar commands), since `set masters` can run at any time
+  in a session.
+- *Own name* (controller only): the `--myname` value, or the host name
+  the controller uses when `--myname` is not given.
+
 #### Acceptance Criteria
-1. The peers-based component (controller, scheduler, worker, or
-   harness/xce_test) shall recognize `XCE_CONTROLLER_PEERS` as the
-   environment variable providing its default `--peers` candidate list.
-2. When the peers-based component starts with an explicit `--peers`
-   argument, the peers-based component shall use that argument's value and
-   shall not consult `XCE_CONTROLLER_PEERS`.
-3. When the peers-based component starts without an explicit `--peers`
-   argument and `XCE_CONTROLLER_PEERS` is set to a non-empty value, the
-   peers-based component shall use that value as its `--peers` candidate
-   list, exactly as if it had been supplied via `--peers`.
-4. When the peers-based component starts without an explicit `--peers`
-   argument and `XCE_CONTROLLER_PEERS` is unset or empty, the peers-based
-   component shall behave exactly as it does today when started with no
-   `--peers` argument (today's standalone/single-controller behavior,
-   unchanged).
-5. If the candidate-list value in use — from either `--peers` or
-   `XCE_CONTROLLER_PEERS` — fails to parse as a valid candidate list, the
-   peers-based component shall report a parse-error message, and that
-   message shall identify whether the invalid value came from the
-   `--peers` argument or from the `XCE_CONTROLLER_PEERS` environment
-   variable.
-6. The peers-based component's `--peers` help/usage text shall document
-   `XCE_CONTROLLER_PEERS` as an environment-variable fallback for that
-   option.
+1. When the peers-based component reaches its resolution point with an
+   explicit candidate list, the peers-based component shall use that list
+   and shall not use `XCE_CONTROLLER_PEERS` as its candidate list (it may
+   still read the variable, e.g. to show it as harness/xce_test's `masters`
+   default in `set` output).
+2. When the peers-based component reaches its resolution point without an
+   explicit candidate list and `XCE_CONTROLLER_PEERS` is set to a
+   non-empty value, the peers-based component shall use that value as its
+   candidate list, exactly as if it had been given as an explicit
+   candidate list, except for the error-source wording in 1.4, the
+   controller's own-name check in 1.6, and, for harness/xce_test, that the
+   value is shown as the `masters` setting's default rather than its value
+   in `set` output.
+3. When the peers-based component reaches its resolution point without an
+   explicit candidate list and `XCE_CONTROLLER_PEERS` is unset or empty,
+   the peers-based component shall behave exactly as it does today with no
+   explicit candidate list (controller: standalone single-node primary;
+   scheduler and worker: their single-controller path; harness/xce_test:
+   its built-in `masters` default), apart from the log wording in 1.8.
+4. If the candidate list in use, from either an explicit candidate list
+   or `XCE_CONTROLLER_PEERS`, fails parsing or the component's own
+   validation of it (including the controller's rule that the number of
+   candidates must be odd), the peers-based component shall report an
+   error message that identifies whether the value came from the explicit
+   candidate list (`--peers`, or `set masters` for harness/xce_test) or
+   from the `XCE_CONTROLLER_PEERS` environment variable, and shall keep
+   today's failure behavior for that error.
+5. The peers-based component's help/usage text for its candidate list
+   (`--peers` for controller, scheduler and worker; the `masters` setting
+   in harness/xce_test's `set` help) shall document `XCE_CONTROLLER_PEERS`
+   as an environment-variable fallback for it.
+6. If the controller takes its candidate list from `XCE_CONTROLLER_PEERS`
+   and its own name matches no candidate in that list, the controller
+   shall exit with a non-zero status before opening any network
+   connection, writing an error message to stderr that states the
+   controller's own name, names `XCE_CONTROLLER_PEERS` as the source, and the two remedies: pass
+   `--myname` naming one of the candidates, or unset
+   `XCE_CONTROLLER_PEERS` to run standalone. When the candidate list comes
+   from an explicit `--peers` argument, the controller's identity handling
+   is unchanged.
+7. The peers-based component shall treat an empty `XCE_CONTROLLER_PEERS`
+   the same as an unset one, so that unsetting or emptying the variable for
+   one invocation gives that invocation the behavior in 1.3; no separate
+   opt-out option is added, and the help text from 1.5 shall describe this
+   way of opting out.
+8. When the controller, scheduler or worker starts with neither an
+   explicit `--peers` argument nor a non-empty `XCE_CONTROLLER_PEERS`, its
+   existing startup log line for that case shall also say that
+   `XCE_CONTROLLER_PEERS` was unset or empty, so an operator whose variable was
+   dropped (e.g. by `sudo`) can tell from the log why the standalone or
+   single-controller path was taken.
 
 ### Requirement 2: Default controller candidate list for xceadmin
 
@@ -95,9 +157,10 @@ variable configures every tool's default connection target.
    explicit `--controller` option and `XCE_CONTROLLER_PEERS` is set to a
    non-empty value, xceadmin shall derive its controller candidate list
    from `XCE_CONTROLLER_PEERS`.
-2. Where a candidate entry in `XCE_CONTROLLER_PEERS` carries a `name=`
-   prefix, xceadmin shall remove the prefix before treating that entry as a
-   controller candidate.
+2. Where a candidate entry carries a `name=` prefix, xceadmin shall remove
+   the prefix before treating that entry as a controller candidate. This
+   applies to every source of the `--controller` value, an explicit
+   `--controller` included.
 3. When xceadmin derives its controller candidate list from
    `XCE_CONTROLLER_PEERS`, xceadmin shall attempt each resulting candidate
    in the same order the variable lists them, moving to the next candidate
@@ -105,11 +168,13 @@ variable configures every tool's default connection target.
    already provides for an explicit multi-candidate `--controller` value.
 4. When an xceadmin subcommand that accepts `--controller` runs with an
    explicit `--controller` option, xceadmin shall use that option's value
-   and shall not consult `XCE_CONTROLLER_PEERS` or `XCEADMIN_CONTROLLER`.
+   (parsed as in 2.2 and 2.8) and shall not consult `XCE_CONTROLLER_PEERS`
+   or `XCEADMIN_CONTROLLER`.
 5. When an xceadmin subcommand that accepts `--controller` runs without an
    explicit `--controller` option, and `XCE_CONTROLLER_PEERS` is unset or
    empty, and `XCEADMIN_CONTROLLER` is set to a non-empty value, xceadmin
-   shall use `XCEADMIN_CONTROLLER` exactly as it does today.
+   shall use `XCEADMIN_CONTROLLER` exactly as it does today, apart from
+   the parsing changes in 2.2 and 2.8.
 6. When an xceadmin subcommand that accepts `--controller` runs without an
    explicit `--controller` option, and both `XCE_CONTROLLER_PEERS` and
    `XCEADMIN_CONTROLLER` are unset or empty, xceadmin shall fall back to
@@ -117,6 +182,14 @@ variable configures every tool's default connection target.
 7. The `--controller` option's help text shall document
    `XCE_CONTROLLER_PEERS` as its preferred environment-variable fallback,
    alongside the existing `XCEADMIN_CONTROLLER` mention.
+8. If xceadmin cannot reach any controller candidate, or a candidate entry
+   is malformed (e.g. a non-numeric port), xceadmin shall exit with an
+   error message rather than an unhandled exception. When the
+   `--controller` value came from `XCE_CONTROLLER_PEERS` or
+   `XCEADMIN_CONTROLLER` rather than an explicit `--controller` option,
+   that message shall name the environment variable it came from. For a
+   malformed entry, whatever the source, the message shall quote the
+   malformed entry.
 
 ### Requirement 3: Documentation of the new environment variable
 
@@ -126,15 +199,41 @@ I can discover and use it without reading source code.
 
 #### Acceptance Criteria
 1. The project's build/deployment steering documentation shall describe
-   `XCE_CONTROLLER_PEERS`: which components read it, its value format, and
-   its precedence relative to each component's connection-target option
-   (`--peers` or `--controller`).
+   `XCE_CONTROLLER_PEERS`: which components read it, its value format, its
+   precedence relative to each component's explicit value (`--peers`,
+   `set masters`, or `--controller`), how to opt out for one invocation
+   (1.7), the controller's own-name rule (1.6), and that it is not
+   inherited by a
+   process started through `sudo` (default `env_reset`) or by a systemd
+   service, with the way to pass it in each case (e.g.
+   `sudo --preserve-env=XCE_CONTROLLER_PEERS ...`, or `Environment=` in
+   the unit). This matters most for the production worker, which must run
+   as root.
 2. The HA cluster deployment guidance shall note that
    `XCE_CONTROLLER_PEERS` is available as an alternative to passing
-   `--peers`/`--controller` explicitly on every generated script
-   invocation, without stating or implying any change to what those
-   generated scripts themselves pass.
+   `--peers`/`--controller` explicitly, that the generated daemon scripts
+   still pass `--peers` explicitly, and that the generated
+   `scripts/xceadmin` wrapper exports `XCE_CONTROLLER_PEERS` for its own
+   deployment (Requirement 4).
 3. The multi-host verification guidance shall note the same availability
    wherever it currently documents driving deployment via `--peers`,
    without stating or implying any change to that verification's own
    generated invocations.
+
+### Requirement 4: Generated xceadmin wrapper is not shadowed by an ambient variable
+
+**Objective:** As an operator who has `XCE_CONTROLLER_PEERS` set in my
+shell for another cluster, I want a deployment's generated
+`scripts/xceadmin` wrapper to keep reaching that deployment's own
+controllers, so that the new variable does not bring back bug-1566's
+wrong-controller symptom.
+
+#### Acceptance Criteria
+1. The `scripts/xceadmin` wrapper generated by `deploy-ha.sh` shall export
+   `XCE_CONTROLLER_PEERS` set to the deployment's own full `--peers`
+   candidate list before starting xceadmin, overriding any value inherited
+   from the caller's environment.
+2. The generated wrapper shall keep exporting `XCEADMIN_CONTROLLER` exactly
+   as it does today.
+3. An explicit `--controller` option passed to the generated wrapper shall
+   still take precedence over both exported variables.