myxce/.kiro/specs/controller-peers-env-fallback/research.md

Research & Design Decisions

Summary

• Feature: controller-peers-env-fallback
• Discovery Scope: Extension (existing CLI argument-parsing across 5 already-shipped binaries)
• Key Findings:
  • Finding 1: harness/xce_test does not take a --peers CLI flag at all — it reads a masters setting (a {key, value, defval} table entry, set only via the interactive/scripted set masters ... command or rc-file loading). This is architecturally different from controller/scheduler/worker and requires a different insertion point.
  • Finding 2: click's envvar= parameter already accepts an ordered list of names and natively returns "the first non-empty value," treating an empty string exactly like unset — verified directly in the installed click library's Parameter.resolve_envvar_value(). This eliminates the need for any custom default-resolution code on the xceadmin side.
  • Finding 3: xce_parse_peers() (src/lib/cluster.c) already strips an optional name= prefix per comma-separated entry before parsing host:port — xceadmin.discovery.resolve_controllers() does not. Porting the same, already-proven convention into resolve_controllers() is a minimal, consistent, backward-compatible extension rather than a new parsing scheme.

Research Log

Does --peers-omitted behavior differ by component, and is it safe to default it via an env var?

• Context: Requirement 1.3 requires that omitting --peers with no env var set behaves exactly as today; needed to confirm this "today" behavior is genuinely a supported mode (not an error) for all four components, and that an env-var-supplied default cannot unsafely interact with an unrelated mode.
• Sources Consulted: src/controller/main.c (~L403-429), src/scheduler/src/main.c (~L380-429), src/worker/src/main.c (~L389-475), buglog.json bug-1457.
• Findings:
  • controller: if (arguments.peers_str) {...} — omitting it is controller's own supported single-node/standalone primary path (bug-1457 fixed a startup-ordering defect in this exact path in 2026-07; the path itself is intentional, not an error).
  • scheduler: if (arguments.peers) {...} else { INFO("No --peers flag; using single-controller path, ...") } — same shape, explicitly logged as intentional. Critically, scheduler's --sim mode returns before this block is ever reached (main.c ~L380-400) — an env-var-supplied peers default therefore cannot affect --sim replay mode (with or without --interactive) at all, by construction, with no special-casing needed.
  • worker: if (peers_str) {...} — identical shape.
  • All three: an invalid --peers value is a hard parse-error (xce_parse_peers() != 0 → fprintf(stderr, ...); return 1;), never a silent fallback.
• Implications: The env-var fallback only needs to be consulted at the single point each component already gates on peers_str/peers being non-NULL — no new mode-interaction risk exists, and --sim is unaffected without any extra guard.

harness/xce_test's actual connection-target mechanism

• Context: Requirements assumed harness took a --peers-shaped CLI argument like the other three; needed to verify before designing its integration point.
• Sources Consulted: src/harness/test.c (settings[] ~L265-273, get_setting() ~L991-1005, enter_ctx() ~L1007-1035), src/harness/CLAUDE.md.
• Findings: harness has no --peers/--masters top-level CLI flag. settings[] is a static {key, value, defval} array; the "masters" entry ships value=NULL, defval="localhost:9990". get_setting() returns value if the operator has run set masters ... (interactively or via an rc-file, at any point before enter_ctx()'s connection attempt) — otherwise it falls through to defval. enter_ctx() parses whatever get_setting("masters") returns through the same xce_parse_peers() used everywhere else, and hard-errors on an invalid value.
• Implications: harness's integration point is computing defval once at process startup (before the REPL loop and rc-file loading) from the env var, not gating on a CLI argument. This preserves the existing value-before-defval precedence untouched — an explicit set masters ... (interactive or rc-file) still always wins over the env-var-derived default, satisfying the same "explicit override always wins" property the other three components have, just through a different existing seam.

click's multi-envvar precedence (xceadmin side)

• Context: Needed to confirm whether xceadmin's --controller option can prioritize XCE_CONTROLLER_PEERS over the existing XCEADMIN_CONTROLLER without custom callback code.
• Sources Consulted: Installed click library, click/core.py, Parameter.resolve_envvar_value() (read directly, not from memory).
• Findings: envvar accepts str | Sequence[str]. When given a sequence, click iterates in order and "return[s] the first non-empty value of the list" — an empty string is treated identically to the variable being absent. This is already exactly the "new var first, legacy var second, empty-or-unset falls through" precedence Requirement 2 (2.4-2.6) specifies.
• Implications: No custom default=/callback= wiring is needed. Each --controller option's existing envvar="XCEADMIN_CONTROLLER" becomes envvar=["XCE_CONTROLLER_PEERS", "XCEADMIN_CONTROLLER"]. This is strictly simpler than the two-separate-variable design considered during discovery.

Where to strip a name= prefix for xceadmin

• Context: XCE_CONTROLLER_PEERS may carry name=host:port entries (the daemons' native format); xceadmin.discovery.resolve_controllers() (bug-1821) does a plain part.rsplit(":", 1) with no such handling, so a named entry would mis-parse as a literal unresolvable hostname.
• Sources Consulted: src/xceadmin/src/xceadmin/discovery.py (resolve_controllers(), follow_redirects_any()), src/lib/cluster.c (xce_parse_peers()'s own name= stripping, for convention parity).
• Findings: xce_parse_peers() already strips everything before the first = in each entry, unconditionally, before its own host:port parse. No production hostname can legally contain = (not permitted in DNS names or IP literals), so this is unambiguous and safe.
• Implications: Extend resolve_controllers() itself with the same one-line convention, rather than adding a separate conversion function ahead of it. This keeps the name=-tolerant behavior available to any caller of resolve_controllers() (not just the new env-var path) and mirrors an already-proven pattern instead of inventing a second one.

Architecture Pattern Evaluation

Option	Description	Strengths	Risks / Limitations	Notes
Single shared C helper (xce_controller_peers_env()) in libxce, called individually by each daemon's existing --peers-gating if	One pure function (read env var, treat empty as unset); each call site keeps its own existing control flow	Minimal surface, trivially unit-testable, no new abstraction	None identified — all 4 C targets already link libxce	Selected
Per-component duplicate getenv() calls, no shared helper	Simplest possible per-file change	No shared bug surface to fix once	Duplicates the exact same 2-line logic 4 times; harder to keep the "empty string == unset" rule consistent if one copy drifts	Rejected — violates Simplification/DRY with zero offsetting benefit given the shared helper is trivial
Custom click default=/callback= wrapper for xceadmin's --controller	Full control over resolution order and transformation	None over the adopted option	Reinvents what click's own multi-envvar list already does natively	Rejected once click's native behavior was confirmed (see Research Log)
Extend resolve_controllers() to strip name= prefixes	One-line change to an existing, already-tested function	Reuses existing, well-tested follow_redirects_any failover; no new function; benefits any future caller	None identified	Selected

Design Decisions

Decision: One shared C primitive, not a per-component copy or a --peers-value-transforming helper

• Context: controller/scheduler/worker each already gate on peers_str/peers being non-NULL at one call site; harness has no such call site (see settings-table finding above).
• Alternatives Considered:
  1. A helper taking (cli_value) and returning the resolved value, changing each call site's if (arguments.peers) to if ((resolved = xce_resolve_peers(arguments.peers))).
  2. A bare xce_controller_peers_env(void) returning the raw env value (or NULL), left for each call site to combine with its own local variable using its own existing if.
• Selected Approach: Option 2 — const char *xce_controller_peers_env(void) in src/lib/cluster.c, declared in src/include/xce/cluster.h alongside xce_parse_peers().
• Rationale: harness has no cli_value to pass in at all (its "explicit override" is a settings-table write, not a function argument), so a helper shaped around "given a CLI value, return the resolved one" doesn't fit uniformly across all 4 components. A bare env-reader fits every call site's own existing control flow without forcing a shape mismatch.
• Trade-offs: Each of the 4 call sites writes its own 1-3 line combination (arguments.peers_str ? arguments.peers_str : xce_controller_peers_env(), or, for harness, a startup-time defval computation) instead of one identical call everywhere — a small amount of repetition, accepted because the alternative (option 1) does not actually fit harness's shape.

Decision: Reuse click's native multi-envvar list instead of a custom default resolver

• Context: See Research Log.
• Alternatives Considered:
  1. A new controller_peers_from_env() Python helper performing name= stripping, wired in via a custom default= callable ahead of click's own envvar= chain.
  2. Extend resolve_controllers() to tolerate name= prefixes directly, and add "XCE_CONTROLLER_PEERS" to each option's existing envvar= list.
• Selected Approach: Option 2.
• Rationale: Click already implements exactly the required precedence (first non-empty of an ordered list, empty treated as unset) — reinventing it in a custom default= callable would duplicate already-correct, already-tested library behavior for no benefit.
• Trade-offs: None identified; strictly less code than the alternative.

Decision: Controller refuses an env-supplied peers list that lacks its own name (design review, 2026-10-02)

• Context: For the controller, --peers is the Raft membership. src/controller/cluster.c ~L1780-1791 finds my_priority (which is also Raft self_idx) by matching the controller's name against each candidate's name (or hostname when unnamed), and falls back to the last index when nothing matches. cluster_open_listener() (~L1902) already refuses that no-match case before any network I/O, so Raft safety is not at risk, but the refusal blames "--peers", goes only to the log file, and the process exits with status 0 (main() ignores myapp_run()'s result). An inherited XCE_CONTROLLER_PEERS makes the case far more likely: a controller started without --myname (name = hostname), or one meant to run standalone.
• Alternatives Considered:
  1. The controller ignores XCE_CONTROLLER_PEERS; only scheduler, worker, harness and xceadmin read it.
  2. The controller reads it, but refuses to start when its own name is not a candidate.
  3. Make the no-match case an error for explicit --peers too.
• Selected Approach: Option 2, with the matching rule moved into a shared xce_peers_find_identity() that both existing cluster.c lookups also call, so the three cannot drift.
• Rationale: Keeps one variable for every component (the brief's goal) and turns the most likely new misconfiguration into an immediate, clear, non-zero-exit error instead of a log-only refusal that blames the wrong source. Option 3 changes shipped explicit-flag behavior, which the requirements rule out; it is recorded as a follow-up in the design's Revalidation Triggers.
• Opt-out for one invocation (the brief's open question): unset the variable or set it empty for that run; no separate flag. xce_controller_peers_env() already treats empty as unset.

Decision: Generated scripts/xceadmin wrapper exports XCE_CONTROLLER_PEERS (design review, 2026-10-02)

• Context: deploy-ha.sh ~L692 points the wrapper at its own deployment only through XCEADMIN_CONTROLLER. Because the new variable outranks it, an operator with XCE_CONTROLLER_PEERS set for another cluster would see the wrapper reach the wrong controller (bug-1566's symptom), while the generated daemon scripts stay safe because they pass --peers explicitly.
• Alternatives Considered:
  1. Keep deploy-ha.sh untouched and document the shadowing.
  2. Add export XCE_CONTROLLER_PEERS="$PEERS_STR" to the wrapper.
• Selected Approach: Option 2. One line, and the wrapper gains failover across the deployment's full candidate list instead of only the first candidate.
• Trade-offs: Narrows the brief's "deploy-ha unchanged" scope decision to "generated daemon scripts unchanged".

Risks & Mitigations

• Risk: A future --peers/--controller/masters value could legitimately contain = in a way that collides with the name= stripping convention. Mitigation: Not possible in practice — DNS hostnames and IP literals cannot contain =; both xce_parse_peers() (existing, proven) and the extended resolve_controllers() share the identical, narrow "split on the first =" rule.
• Risk: harness's defval mutation could be perceived as "changing settings without the operator asking," surprising a set command's own listing output. Mitigation: defval has always been the table's documented fallback slot (get_setting() already falls through to it); computing it from an explicit, documented environment variable at startup is more transparent than silently keeping a stale hardcoded literal while an env var the operator deliberately set is ignored.

References

• src/lib/cluster.c — xce_parse_peers() (existing name= convention, reused).
• src/xceadmin/src/xceadmin/discovery.py — resolve_controllers()/follow_redirects_any() (bug-1821).
• src/xceadmin/src/xceadmin/cli.py — existing --controller option definitions (bug-1566).
• click core.py, Parameter.resolve_envvar_value() — multi-envvar precedence, read directly from the installed library.