myxce/.kiro/specs/controller-peers-env-fallback/tasks.md

Implementation Plan

• 1. Foundation: shared libxce primitives and the regression script

• 1.1 Add the environment-variable reader for the default candidate list

  • Write the failing libxce unit tests first: variable unset gives no value, set to an empty string gives no value, set to a non-empty string returns it unchanged
  • Add the reader to the shared cluster module and declare it next to the existing peers parser, so every C daemon and the harness can call it with no new build dependency
  • Each test restores the variable to unset afterwards, so later libxce tests are unaffected
  • Done when the libxce suite passes with the three new cases and every existing libxce test still passes
  • Requirements: 1.2, 1.3, 1.7

• 1.2 Add the shared "which candidate is me" lookup

  • Write the failing libxce unit tests first: named match, unnamed entry matched by hostname, named entry not matched by its hostname, no match, first of duplicate names wins, no name given, and an empty name matching an unnamed empty-hostname entry exactly as the controller's existing loops do
  • Implement the lookup with the matching rule moved unchanged from the controller (candidate name when set, otherwise hostname; first match wins); it applies no default of its own
  • Done when the libxce suite passes with all seven cases
  • Requirements: 1.6

• 1.3 Create the regression script skeleton

  • Test infrastructure for the scripted cases in later tasks; it satisfies no acceptance criterion by itself
  • Create the repeatable regression script that later tasks add their cases to: take the build directory as an input and locate the built binaries in it, create a fresh scratch directory per daemon run, and provide helpers that run a binary with a given environment and arguments, wait for a line in a log file or captured stdout/stderr with a timeout, kill the process, and record pass/fail
  • Pre-flight: fail with a message, rather than run, when a local cluster is up (something listening on port 9990, or the worker's fixed socket present) or a needed binary is missing
  • All binaries the script uses must sit in one build directory: build each component with /xce-build <component> --incremental (without --incremental a component build wipes the directory, removing the others); every later "after rebuilding" step means the same
  • Done when, after /xce-build (controller, scheduler, worker, harness) on a machine with no local cluster, the script runs with no cases and exits 0, and exits non-zero with the pre-flight message while a listener is bound to port 9990 or when pointed at a build directory missing a binary

• 2. Controller

• 2.1 Route the controller's two existing identity loops through the shared lookup

  • Replace the loop that picks the controller's election priority, keeping the existing "at least one candidate" guard so a standalone controller still gets priority 0, and keeping "no match gives the last index" at that call site
  • Replace the loop that picks the listen port, keeping the existing "no match with a peers list refuses to start" and "port 0 falls back to the default port" behavior
  • Pure refactor: no behavior change for any input
  • Done when the full controller suite (including the HA and listener tests) passes unchanged
  • Requirements: 1.6

• 2.2 Flush the controller's log before every early exit

  • Add a regression-script case first: a controller run with scratch work, certificate and log paths stops at the certificate pre-flight, and its last startup line (today's "No --peers flag" line) must already be in the log file. The loss is a race with the log's background writer, so the case repeats the run 20 times and fails if any run is missing the line; run it before the fix and record whether it failed (if no run fails, note that the race did not reproduce here and keep the case as a regression guard)
  • Flush the log before every early return after the log is opened: name lookup failure, lock init failure, --peers parse error, even candidate count, and the certificate pre-flight; flushing is a no-op before the log is opened and safe to repeat
  • Done when, after rebuilding the controller, the controller suite passes and that case shows the line in the log file in all 20 runs
  • Requirements: 1.3

• 2.3 Fall back to the environment variable in the controller and name the value's source

  • Add the controller cases to the regression script first and see them fail: env list used with no --peers, an odd candidate count and --myname naming a candidate ("HA mode: N controller candidates" in the log), so neither the even-count rule nor the own-name check in 2.4 masks the result; --peers winning; neither set (extended standalone line); empty variable (standalone); parse and even-count errors from each source naming that source on stderr; --help naming the variable and the opt-out. Each run gets scratch work, certificate and log paths, so it stops at the certificate pre-flight
  • Use the variable as the peers list only when no --peers is given, and remember whether it supplied the list
  • Name the source in the parse-error and even-count messages; extend the "No --peers flag" line to say the variable was unset or empty; add the variable and the opt-out note to the --peers help
  • Done when, after rebuilding the controller, the controller suite passes and every controller case in the regression script passes
  • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.7, 1.8

• 2.4 Refuse an env-supplied peers list that does not contain the controller's own name

  • Add the cases to the regression script first and see them fail: with a three-candidate variable, a name not in it exits non-zero with the message; a name in it logs "HA mode: 3 controller candidates" with no such message; the same list given as explicit --peers with a name not in it prints no such message
  • When the list came from the variable and the shared lookup finds no candidate for the controller's own name (its --myname, or the host name when not given), print to stderr the controller's name, the variable as the source, the candidates (by name, or hostname when unnamed) and the two remedies (pass --myname naming a candidate, or unset the variable); flush the log and exit non-zero before the certificate pre-flight and any network activity
  • Leave the explicit---peers path untouched (its later refusal in the listener stays as it is)
  • Done when, after rebuilding the controller, the controller suite passes and the own-name cases in the regression script pass
  • Requirements: 1.6

• 3. Scheduler, worker and harness

• 3.1 Fall back to the environment variable in the scheduler

  • Add the scheduler cases to the regression script first and see them fail: env list used ("HA cluster config: N candidate(s)" in the log); --peers winning; neither set (extended no-peers line); parse error and port-0 validation error from each source naming that source on stderr; --help naming the variable and the opt-out; simulation mode with a missing replay file and the variable set showing the simulation start-up failure on stderr and no peers-related log line
  • Use the variable only when no --peers is given; leave the simulation path, which returns earlier, untouched
  • Name the source in the parse-error and cluster-config-validation messages; extend the no-peers line; update the help text
  • Done when, after rebuilding the scheduler, the scheduler suite passes and every scheduler case in the regression script passes
  • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.7, 1.8

• 3.2 Fall back to the environment variable in the worker

  • Add the worker cases to the regression script first and see them fail, using the virtual worker: env list used ("HA cluster config: N candidate(s)" on stdout, since these lines are logged before the worker's log file exists); --peers winning; neither set (extended no-peers line); parse error and port-0 validation error from each source naming that source on stderr; -h naming the variable, the opt-out, and that sudo drops it unless preserved
  • Use the variable only when no --peers is given; name the source in both error messages; extend the no-peers line; update the usage text
  • Done when, after rebuilding the worker, the worker suite passes and every worker case in the regression script passes
  • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.7, 1.8

• 3.3 Default the harness masters setting from the environment variable

  • Add the harness cases to the regression script first and see them fail, always with --norc: a bare set shows the env value as the masters default; after set masters ... the explicit value is shown; on a connecting command, the invalid-value message says explicit set masters when one ran, default from the variable when none ran and the variable is set, and an explicit invalid value still blames set masters with a valid variable set; help set names the variable and the opt-out
  • At startup, before rc-file loading and the command loop, set the masters default from the variable when it is set, otherwise keep today's built-in default; an explicit set masters still wins
  • Make the invalid-masters message name the source as above (today's wording when neither applies); update the set help text
  • Done when, after rebuilding the harness, the harness suite (including the rc-file integration test) passes and every harness case in the regression script passes
  • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.7

• 4. xceadmin

• 4.1 (P) Accept name= prefixes and report malformed entries cleanly when parsing controller candidates

  • Write the failing tests first: a single name=host:port; a mixed named/bare list keeping its order; a non-numeric port raising the new controller-value error that quotes the entry; existing bare-host and list cases unchanged
  • Add the controller-value error next to the existing controller-reachability errors
  • Strip everything up to the first = in each entry before the existing host/port split; turn the failing port conversion into the new error
  • Done when the xceadmin discovery tests pass with the new cases and every existing discovery test still passes
  • Requirements: 2.2, 2.8
  • Boundary: xceadmin discovery and errors

• 4.2 (P) Read the new variable ahead of the legacy one in every --controller option

  • Write the failing CLI tests first: the variable alone; both variables set (new one wins); explicit --controller overriding both; neither set (built-in default); each option's --help naming both variables
  • Add monkeypatch.delenv for the new variable to every existing test that asserts legacy-variable or no-variable behavior, so ambient state cannot affect them
  • Make the shared env-var setting an ordered list (new variable first, legacy second) and rewrite each option's help string and the precedence comment, since the help strings interpolate the setting today
  • Done when the xceadmin CLI tests pass with the new precedence and help cases, and the existing legacy-variable tests still pass with the new variable set in the test process's environment
  • Requirements: 2.1, 2.4, 2.5, 2.6, 2.7, 4.3
  • Boundary: xceadmin CLI

• 4.3 Name the environment source in xceadmin's controller errors

  • Write the failing CLI tests first: in-order failover over an env-derived named list (patching the per-candidate request so the first fails and the second answers); unreachable values from each variable and from an explicit --controller (source note only for the two variables); a missing cluster-bin file with the variable set (no source note); a malformed entry from an explicit --controller and from the variable (entry quoted, note only for the variable). Use config push with a real cluster-bin file, or patch out the record commands' retry pause
  • Add a helper that returns "(from $NAME)" when the value came from the environment, using the same first-non-empty rule click applies
  • Append it to the "cannot connect" messages only for network errors (not local file errors), and to the controller-reachability and controller-value errors in the existing handlers; leave other errors unchanged
  • Done when the xceadmin CLI tests pass with all the new cases and no command shows a traceback for a malformed entry
  • Requirements: 2.3, 2.8
  • Depends: 4.1, 4.2

• 5. Integration

• 5.1 Pin the generated xceadmin wrapper to its own deployment

  • Add the wrapper case to the regression script first and see it fail: generate a deployment on free high ports into a scratch directory, run a copy of the generated wrapper whose final line prints the two variables, with a bogus value of the new variable in the caller's environment, and expect the deployment's own ctrl-N=host:port list and first candidate; fail with a message naming the xceadmin build when the wheel is missing
  • In the deploy script's generated wrapper, export the new variable set to the deployment's full peers list next to the existing legacy export, and extend the comment above it to say why
  • Done when, with the xceadmin wheel built (/xce-build xceadmin), the wrapper case in the regression script passes
  • Requirements: 4.1, 4.2

• 5.2 Keep existing no-peers test scripts on the path they test

  • Both scripts delete their scheduler log on exit, so each must check the path itself: first add a check that greps its scheduler log for the "No --peers flag ... XCE_CONTROLLER_PEERS unset or empty" line and fails the script if it is missing, and see it fail with the variable set in the calling shell
  • Then add unset XCE_CONTROLLER_PEERS near the top of the version-startup-consistency and dirty-fingerprint-startup-log scripts, which start the scheduler without --peers
  • Done when, with the variable set in the calling shell, both scripts pass, including their new no-peers-line check
  • Requirements: 1.3

• 5.3 Full build and test pass

  • Run the full debug + ASAN build with every component's test suite, the xceadmin test suite, both scripts from 5.2, and the whole regression script
  • Done when all suites and scripts pass with no ASAN reports
  • Requirements: 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 4.1, 4.2, 4.3

• 6. Documentation

• 6.1 (P) Document the variable in the build/deployment steering

  • Describe which components read it, its value format, its precedence against each explicit value, the one-invocation opt-out, the controller's own-name rule, and that sudo and systemd do not pass it on (with --preserve-env and Environment=)
  • Done when the steering document has a section covering every item above
  • Requirements: 3.1
  • Boundary: steering docs

• 6.2 (P) Note the variable in the HA-deploy and multi-host skill docs

  • HA-deploy skill: the variable's availability, that generated daemon scripts still pass --peers, and that the xceadmin wrapper now exports the variable (description and troubleshooting row)
  • Multi-host skill: the same availability note at its three --peers locations, without implying any change to its generated invocations
  • Done when both skill documents carry the notes at the listed locations
  • Requirements: 3.2, 3.3
  • Boundary: skill docs

Implementation Notes

• (1.3) verify-controller-peers-env.sh unsets XCE_CONTROLLER_PEERS at startup; every case sets it explicitly. Exit 2 = usage/pre-flight refusal, 1 = a case failed. BusyBox timeout is in use here: a timed-out run exits 124 (coreutils) or 143 (BusyBox) — accept either.
• (1.3 → 3.3) start_bg/run_to_completion feed stdin from /dev/null, but xce_test reads its script only from stdin; add an optional stdin file to run_to_completion (e.g. RUN_STDIN) rather than the sh -c 'exec "$0" --norc <"$1"' workaround.
• (1.3 → 3.2) The worker's early "HA cluster config"/"No --peers flag" lines go to stdout before its log file is set up; confirm they are not stdio-buffered when stdout is a file (else wait_for_line on stdout times out while it runs) — wait for exit or use line buffering if so.
• (1.3) Drop the file-wide shellcheck disable=SC2329,SC2034 once cases call the helpers, so real unused-variable warnings surface.
• (2.2) The tlog early-exit race did not reproduce on this machine (0/20 before the fix, 0/200 ad hoc); case_controller_log_flushed_on_early_exit is kept as a regression guard, not proof of the fix.
• (2.2 → 2.3) That case matches the text "No --peers flag; running as single-node primary"; when 2.3 extends the line (1.8), keep that text intact or update the case's pattern.
• (2.3) The 1.8 wording cannot contain the old "No --peers flag; running ..." substring, so the 2.2 flush case now matches the full new line. Env-sourced "HA mode" lines end " (from XCE_CONTROLLER_PEERS)"; explicit---peers lines and error texts are unchanged apart from naming the source. Reword the script header's "while the case list is empty" comment in 3.x when the shellcheck disable is dropped.
• (3.1 → 3.2) The scheduler's validation message now quotes the value, names the source and hints "(empty host or port 0?)" (the old "missing identity?" was wrong: identity is always set there, and xce_parse_peers caps the candidate count itself). Use the same shape for the worker's xce_cluster_config_validate() message.
• (3.1 → 5.2) On this machine test-version-startup-consistency.sh and test-dirty-fingerprint-startup-log-unchanged.sh exit 1 for an environmental reason: their apiserver go build fails with "module lookup disabled by GOPROXY=off" (no Go module cache). Their scheduler checks pass. Task 5.2's "both scripts pass" needs a machine with the Go module cache, or the result must be reported as environment-limited with the scheduler checks shown passing.
• (3.2) The worker's early stdout lines are not buffered (visible within 0.3s while it runs). xcecd_virtual ignores SIGTERM (pre-existing), so start_worker SIGKILLs it after the asserted line; the virtual worker leaves only its scratch log. Its peers parse/validation early returns now free logdir (they leaked under ASAN). Out of scope, pre-existing: -l <dir> combined with -h/--version still leaks logdir.
• (3.2 → 3.3) The file-wide shellcheck disable=SC2329 must stay (cases and helpers run via the "$fn" registry and trap); the SC2034 disable now covers only the HARNESS_BIN line — remove it once the harness cases use it.