myxce/.kiro/specs/controller-peers-env-fallback/research.md

# Research & Design Decisions

## Summary
- **Feature**: `controller-peers-env-fallback`
- **Discovery Scope**: Extension (existing CLI argument-parsing across 5 already-shipped binaries)
- **Key Findings**:
  - Finding 1: harness/`xce_test` does **not** take a `--peers` CLI flag at all — it reads a `masters` *setting* (a `{key, value, defval}` table entry, set only via the interactive/scripted `set masters ...` command or rc-file loading). This is architecturally different from controller/scheduler/worker and requires a different insertion point.
  - Finding 2: click's `envvar=` parameter already accepts an ordered list of names and natively returns "the first non-empty value," treating an empty string exactly like unset — verified directly in the installed click library's `Parameter.resolve_envvar_value()`. This eliminates the need for any custom default-resolution code on the xceadmin side.
  - Finding 3: `xce_parse_peers()` (`src/lib/cluster.c`) already strips an optional `name=` prefix per comma-separated entry before parsing `host:port` — `xceadmin.discovery.resolve_controllers()` does not. Porting the same, already-proven convention into `resolve_controllers()` is a minimal, consistent, backward-compatible extension rather than a new parsing scheme.

## Research Log

### Does `--peers`-omitted behavior differ by component, and is it safe to default it via an env var?
- **Context**: Requirement 1.3 requires that omitting `--peers` with no env var set behaves exactly as today; needed to confirm this "today" behavior is genuinely a supported mode (not an error) for all four components, and that an env-var-supplied default cannot unsafely interact with an unrelated mode.
- **Sources Consulted**: `src/controller/main.c` (~L403-429), `src/scheduler/src/main.c` (~L380-429), `src/worker/src/main.c` (~L389-475), buglog.json bug-1457.
- **Findings**:
  - controller: `if (arguments.peers_str) {...}` — omitting it is controller's own supported single-node/standalone primary path (bug-1457 fixed a startup-ordering defect in this exact path in 2026-07; the path itself is intentional, not an error).
  - scheduler: `if (arguments.peers) {...} else { INFO("No --peers flag; using single-controller path, ...") }` — same shape, explicitly logged as intentional. Critically, scheduler's `--sim` mode `return`s **before** this block is ever reached (`main.c` ~L380-400) — an env-var-supplied peers default therefore cannot affect `--sim` replay mode (with or without `--interactive`) at all, by construction, with no special-casing needed.
  - worker: `if (peers_str) {...}` — identical shape.
  - All three: an invalid `--peers` value is a hard parse-error (`xce_parse_peers() != 0` → `fprintf(stderr, ...); return 1;`), never a silent fallback.
- **Implications**: The env-var fallback only needs to be consulted at the single point each component already gates on `peers_str`/`peers` being non-NULL — no new mode-interaction risk exists, and `--sim` is unaffected without any extra guard.

### harness/xce_test's actual connection-target mechanism
- **Context**: Requirements assumed harness took a `--peers`-shaped CLI argument like the other three; needed to verify before designing its integration point.
- **Sources Consulted**: `src/harness/test.c` (`settings[]` ~L265-273, `get_setting()` ~L991-1005, `enter_ctx()` ~L1007-1035), `src/harness/CLAUDE.md`.
- **Findings**: harness has no `--peers`/`--masters` top-level CLI flag. `settings[]` is a static `{key, value, defval}` array; the `"masters"` entry ships `value=NULL, defval="localhost:9990"`. `get_setting()` returns `value` if the operator has run `set masters ...` (interactively or via an rc-file, at any point before `enter_ctx()`'s connection attempt) — otherwise it falls through to `defval`. `enter_ctx()` parses whatever `get_setting("masters")` returns through the same `xce_parse_peers()` used everywhere else, and hard-errors on an invalid value.
- **Implications**: harness's integration point is computing `defval` once at process startup (before the REPL loop and rc-file loading) from the env var, not gating on a CLI argument. This preserves the existing `value`-before-`defval` precedence untouched — an explicit `set masters ...` (interactive or rc-file) still always wins over the env-var-derived default, satisfying the same "explicit override always wins" property the other three components have, just through a different existing seam.

### click's multi-envvar precedence (xceadmin side)
- **Context**: Needed to confirm whether xceadmin's `--controller` option can prioritize `XCE_CONTROLLER_PEERS` over the existing `XCEADMIN_CONTROLLER` without custom callback code.
- **Sources Consulted**: Installed click library, `click/core.py`, `Parameter.resolve_envvar_value()` (read directly, not from memory).
- **Findings**: `envvar` accepts `str | Sequence[str]`. When given a sequence, click iterates in order and "return[s] the first non-empty value of the list" — an empty string is treated identically to the variable being absent. This is already exactly the "new var first, legacy var second, empty-or-unset falls through" precedence Requirement 2 (2.4-2.6) specifies.
- **Implications**: No custom `default=`/`callback=` wiring is needed. Each `--controller` option's existing `envvar="XCEADMIN_CONTROLLER"` becomes `envvar=["XCE_CONTROLLER_PEERS", "XCEADMIN_CONTROLLER"]`. This is strictly simpler than the two-separate-variable design considered during discovery.

### Where to strip a `name=` prefix for xceadmin
- **Context**: `XCE_CONTROLLER_PEERS` may carry `name=host:port` entries (the daemons' native format); `xceadmin.discovery.resolve_controllers()` (bug-1821) does a plain `part.rsplit(":", 1)` with no such handling, so a named entry would mis-parse as a literal unresolvable hostname.
- **Sources Consulted**: `src/xceadmin/src/xceadmin/discovery.py` (`resolve_controllers()`, `follow_redirects_any()`), `src/lib/cluster.c` (`xce_parse_peers()`'s own `name=` stripping, for convention parity).
- **Findings**: `xce_parse_peers()` already strips everything before the first `=` in each entry, unconditionally, before its own `host:port` parse. No production hostname can legally contain `=` (not permitted in DNS names or IP literals), so this is unambiguous and safe.
- **Implications**: Extend `resolve_controllers()` itself with the same one-line convention, rather than adding a separate conversion function ahead of it. This keeps the `name=`-tolerant behavior available to *any* caller of `resolve_controllers()` (not just the new env-var path) and mirrors an already-proven pattern instead of inventing a second one.

## Architecture Pattern Evaluation

| Option | Description | Strengths | Risks / Limitations | Notes |
|--------|-------------|-----------|---------------------|-------|
| Single shared C helper (`xce_controller_peers_env()`) in `libxce`, called individually by each daemon's existing `--peers`-gating `if` | One pure function (read env var, treat empty as unset); each call site keeps its own existing control flow | Minimal surface, trivially unit-testable, no new abstraction | None identified — all 4 C targets already link `libxce` | **Selected** |
| Per-component duplicate `getenv()` calls, no shared helper | Simplest possible per-file change | No shared bug surface to fix once | Duplicates the exact same 2-line logic 4 times; harder to keep the "empty string == unset" rule consistent if one copy drifts | Rejected — violates Simplification/DRY with zero offsetting benefit given the shared helper is trivial |
| Custom click `default=`/`callback=` wrapper for xceadmin's `--controller` | Full control over resolution order and transformation | None over the adopted option | Reinvents what click's own multi-`envvar` list already does natively | Rejected once click's native behavior was confirmed (see Research Log) |
| Extend `resolve_controllers()` to strip `name=` prefixes | One-line change to an existing, already-tested function | Reuses existing, well-tested `follow_redirects_any` failover; no new function; benefits any future caller | None identified | **Selected** |

## Design Decisions

### Decision: One shared C primitive, not a per-component copy or a `--peers`-value-transforming helper
- **Context**: controller/scheduler/worker each already gate on `peers_str`/`peers` being non-NULL at one call site; harness has no such call site (see settings-table finding above).
- **Alternatives Considered**:
  1. A helper taking `(cli_value)` and returning the resolved value, changing each call site's `if (arguments.peers)` to `if ((resolved = xce_resolve_peers(arguments.peers)))`.
  2. A bare `xce_controller_peers_env(void)` returning the raw env value (or NULL), left for each call site to combine with its own local variable using its own existing `if`.
- **Selected Approach**: Option 2 — `const char *xce_controller_peers_env(void)` in `src/lib/cluster.c`, declared in `src/include/xce/cluster.h` alongside `xce_parse_peers()`.
- **Rationale**: harness has no `cli_value` to pass in at all (its "explicit override" is a settings-table write, not a function argument), so a helper shaped around "given a CLI value, return the resolved one" doesn't fit uniformly across all 4 components. A bare env-reader fits every call site's own existing control flow without forcing a shape mismatch.
- **Trade-offs**: Each of the 4 call sites writes its own 1-3 line combination (`arguments.peers_str ? arguments.peers_str : xce_controller_peers_env()`, or, for harness, a startup-time `defval` computation) instead of one identical call everywhere — a small amount of repetition, accepted because the alternative (option 1) does not actually fit harness's shape.

### Decision: Reuse click's native multi-`envvar` list instead of a custom default resolver
- **Context**: See Research Log.
- **Alternatives Considered**:
  1. A new `controller_peers_from_env()` Python helper performing `name=` stripping, wired in via a custom `default=` callable ahead of click's own `envvar=` chain.
  2. Extend `resolve_controllers()` to tolerate `name=` prefixes directly, and add `"XCE_CONTROLLER_PEERS"` to each option's existing `envvar=` list.
- **Selected Approach**: Option 2.
- **Rationale**: Click already implements exactly the required precedence (first non-empty of an ordered list, empty treated as unset) — reinventing it in a custom `default=` callable would duplicate already-correct, already-tested library behavior for no benefit.
- **Trade-offs**: None identified; strictly less code than the alternative.

### Decision: Controller refuses an env-supplied peers list that lacks its own name (design review, 2026-10-02)
- **Context**: For the controller, `--peers` is the Raft membership. `src/controller/cluster.c` ~L1780-1791 finds `my_priority` (which is also Raft `self_idx`) by matching the controller's name against each candidate's `name` (or `hostname` when unnamed), and falls back to the last index when nothing matches. `cluster_open_listener()` (~L1902) already refuses that no-match case before any network I/O, so Raft safety is not at risk, but the refusal blames "--peers", goes only to the log file, and the process exits with status 0 (`main()` ignores `myapp_run()`'s result). An inherited `XCE_CONTROLLER_PEERS` makes the case far more likely: a controller started without `--myname` (name = hostname), or one meant to run standalone.
- **Alternatives Considered**:
  1. The controller ignores `XCE_CONTROLLER_PEERS`; only scheduler, worker, harness and xceadmin read it.
  2. The controller reads it, but refuses to start when its own name is not a candidate.
  3. Make the no-match case an error for explicit `--peers` too.
- **Selected Approach**: Option 2, with the matching rule moved into a shared `xce_peers_find_identity()` that both existing `cluster.c` lookups also call, so the three cannot drift.
- **Rationale**: Keeps one variable for every component (the brief's goal) and turns the most likely new misconfiguration into an immediate, clear, non-zero-exit error instead of a log-only refusal that blames the wrong source. Option 3 changes shipped explicit-flag behavior, which the requirements rule out; it is recorded as a follow-up in the design's Revalidation Triggers.
- **Opt-out for one invocation** (the brief's open question): unset the variable or set it empty for that run; no separate flag. `xce_controller_peers_env()` already treats empty as unset.

### Decision: Generated `scripts/xceadmin` wrapper exports `XCE_CONTROLLER_PEERS` (design review, 2026-10-02)
- **Context**: `deploy-ha.sh` ~L692 points the wrapper at its own deployment only through `XCEADMIN_CONTROLLER`. Because the new variable outranks it, an operator with `XCE_CONTROLLER_PEERS` set for another cluster would see the wrapper reach the wrong controller (bug-1566's symptom), while the generated daemon scripts stay safe because they pass `--peers` explicitly.
- **Alternatives Considered**:
  1. Keep `deploy-ha.sh` untouched and document the shadowing.
  2. Add `export XCE_CONTROLLER_PEERS="$PEERS_STR"` to the wrapper.
- **Selected Approach**: Option 2. One line, and the wrapper gains failover across the deployment's full candidate list instead of only the first candidate.
- **Trade-offs**: Narrows the brief's "deploy-ha unchanged" scope decision to "generated daemon scripts unchanged".

## Risks & Mitigations
- **Risk**: A future `--peers`/`--controller`/`masters` value could legitimately contain `=` in a way that collides with the `name=` stripping convention. **Mitigation**: Not possible in practice — DNS hostnames and IP literals cannot contain `=`; both `xce_parse_peers()` (existing, proven) and the extended `resolve_controllers()` share the identical, narrow "split on the first `=`" rule.
- **Risk**: harness's `defval` mutation could be perceived as "changing settings without the operator asking," surprising a `set` command's own listing output. **Mitigation**: `defval` has always been the table's documented fallback slot (`get_setting()` already falls through to it); computing it from an explicit, documented environment variable at startup is more transparent than silently keeping a stale hardcoded literal while an env var the operator deliberately set is ignored.

## References
- `src/lib/cluster.c` — `xce_parse_peers()` (existing `name=` convention, reused).
- `src/xceadmin/src/xceadmin/discovery.py` — `resolve_controllers()`/`follow_redirects_any()` (bug-1821).
- `src/xceadmin/src/xceadmin/cli.py` — existing `--controller` option definitions (bug-1566).
- click `core.py`, `Parameter.resolve_envvar_value()` — multi-`envvar` precedence, read directly from the installed library.