chore(controller): clusterconfig-schema-split task 5.3 — single-node/peers-override/EINVAL live-verified

Live-verified all three cluster_open_listener code paths not exercised by
the 3-node soak: no --peers binds hardcoded default 9990; a single
self-referencing --peers entry binds its custom port instead; --peers
given but local name absent still refuses to start with -EINVAL, unchanged.
Independently reproduced by the reviewer. This closes the actual
regression risk this spec's design was built around.

Review also found (bug-1457, logged, not fixed here) that pre-spec
single-node startup was already broken with -EINVAL before this spec
began, due to an unrelated myapp_run listener-open/config-load ordering
issue -- task 2.1's rewrite incidentally repaired it. research.md updated
to record this against the spec's original preservation premise.

.kiro/specs/clusterconfig-schema-split/research.md

@@ -88,3 +88,9 @@ The task 1.1 implementer deliberately left `HashZArrayConfig` and `ResmgrConfig`
 - **Root cause**: this is the same class of gap Requirement 1.6 already closed for `MasterCandidate` (used only by the removed `candidates` field) — just missed for the tunables side during the original requirements pass, since `MasterCandidate` was called out explicitly by name while the tunables' supporting struct types were not.
 - **Decision**: extend the same treatment — remove `HashZArrayConfig`/`ResmgrConfig` from `controller.capnp`. Verified via grep (same method as the original `MasterCandidate` verification) that neither struct is referenced anywhere else in the schema. New Requirement 2.5 added (mirrors 1.6's phrasing); design.md's File Structure Plan and Requirements Traceability updated; `tasks.md` gained task 1.3 (Schema Component, same boundary as 1.1) to close this before task 4's full build.
 - **Why this wasn't caught in the requirements-phase discovery pass**: the original `cluster-config-split-analysis.md` and this spec's requirements-phase grep swept for the FIELDS being removed, not for supporting/nested struct TYPES those fields' capnp types depended on. A useful pattern for future field-removal specs: after identifying fields to remove, also check whether any of their types are struct types used nowhere else in the schema.
+
+## Correction: Pre-Spec Single-Node Startup Was Already Broken (discovered during task 5.3's review, 2026-07-20)
+
+Task 5.3's reviewer, verifying the live single-node scenarios, built and ran the actual pre-spec baseline (commit `94f50ce`, HEAD before this spec's first commit) for comparison — and found single-node startup (`xce_controller` with no `--peers`) was **already exiting with `-EINVAL`**, unconditionally, before this spec began. Root cause: `myapp_run()` (`src/controller/myapp.c`) calls `cluster_open_listener()` before the primary config (`cluster_config`) is loaded from disk, so the pre-spec fallback branch (`cluster_config->port`, only reachable when `cluster_config != NULL`) always hit its `else` branch. Logged as `bug-1457` (unrelated to this spec, not fixed as its own change — task 2.1's rewrite incidentally repaired it, verified live in task 5.3).
+
+**This invalidates part of this spec's own premise, stated in the Introduction and Requirement 1.7's original correction**: "`cluster_open_listener`... has a live fallback to `cluster_config->port`... the sole port-sourcing mechanism for single-node deployments" and "single-node controller startup SHALL continue to work exactly as it does today." In reality, it did not work today (pre-spec) — for a reason entirely unrelated to `ClusterConfig`'s schema. The spec's actual behavior (task 2.1's hardcoded-default fallback, independent of `cluster_config` load state) is still correct and was still the right thing to build — it just fixed a real bug as a side effect, rather than merely "preserving" already-working behavior as originally framed. Not correcting the historical Introduction/Requirement 1.7 text itself (it accurately reflects what was believed and reviewed at design time); this note is the record of what live verification actually found.

.kiro/specs/clusterconfig-schema-split/tasks.md

@@ -84,7 +84,7 @@
   - _Requirements: 3.3, 5.4_
   - _Depends: 4, 5.1_
 
-- [ ] 5.3 Single-node startup, self-referencing-peers port override, and preserved error-path verification
+- [x] 5.3 Single-node startup, self-referencing-peers port override, and preserved error-path verification
   - Start a controller with no `--peers` flag and confirm it binds its listener to the hardcoded default port (9990) from task 2.1
   - Start a controller with a single self-referencing `--peers <name>=<host>:<port>` entry (own `-n`/`--myname` matching, custom port) and confirm it binds to that port instead of the default — this is the sole replacement mechanism for a non-default single-node port under this spec
   - Start a controller with a `--peers` list that does not contain the controller's own name and confirm it still refuses to start with the existing `-EINVAL` error, unchanged from pre-spec behavior

.wolf/buglog.json

@@ -13171,7 +13171,7 @@
     "error_message": "lib/controller.capnp.h:4170:43: error: unknown type name 'LimitCon'; lib/controller.capnp.h:1: error: unterminated #ifndef \u2014 during ninja compile of lib/libxce.a.p/utils.c.o, cascading into ~440 bogus errors (glibc math.h storage-class errors, utils.h ectx_t redefinition) for the rest of that one translation unit",
     "file": "src/xce.build/lib/controller.capnp.h (generated by [5/190] Generating lib/controllercapnp.[ch] custom command)",
     "root_cause": "SUSPECTED (not confirmed): parallel-build race \u2014 gcc appears to have read lib/controller.capnp.h while the custom-command generation step was still writing it (non-atomic write, no write-to-tmp+rename). Evidence: re-inspecting the SAME on-disk file immediately after the failed build shows correct, well-formed content at line 4170 (get_LimitConfig(struct LimitConfig*, LimitConfig_list, int i);) and balanced #ifndef/#endif (9/9) \u2014 i.e. the finished file is NOT actually corrupt, only what gcc read mid-write was. Not yet reproduced a second time to confirm.",
-    "fix": "NONE APPLIED \u2014 this run was build+report only (nix develop ./src -- xce-build.sh controller), not a fix task. Build failed via set -euo pipefail before any tests ran; no controller/xce_controller, controller_unit_test, or xce_test binaries were produced. Next step if it recurs: check meson.build custom_target() for lib/controller.capnp.[ch] has correct output declarations / add explicit depends on all consumers, or file upstream against capnpc-c invocation for atomic output. RECURRENCE 2026-07-16 (2nd occurrence, CONFIRMS the race, different symptom): clean build (rm -rf xce.build) of `nix develop ./src --command bash src/tools/xce-build.sh controller asan` at HEAD 2a76940. This time gcc failed with `fatal error: controller.capnp.h: No such file or directory` (not corrupted content) for hook.c.o, threshold.c.o, xlist.c.o, utils.c.o, window.c.o, xceapi.c.o, wrap.c.o \u2014 all 7 consumers of utils.h#include controller.capnp.h. Critically, ninja's own completion log shows `[12/191] Generating lib/controllercapnp.[ch]` printed AFTER jobs [5..11] (the 7 failed compiles) had already completed/failed. This proves the custom_target that generates lib/controller.capnp.h has no dependency edge forcing it to run before these 7 specific compile units \u2014 confirms the suspected root cause in the original entry (missing depends()/output declaration in meson.build), not just a non-atomic-write timing fluke. Still NOT fixed (build+report only task, no source edits authorized). controller_unit_test and xce_test were never produced this run (clean build wiped the prior binaries and the new build failed before relinking). RECURRENCE 2026-07-17 (3rd occurrence, kiro-validate-impl smoke-boot build for job-array-single-wal-record): fresh (non-clean) build of `nix develop ./src --command bash src/tools/xce-build.sh controller` at HEAD ea8d9d3 failed on the FIRST invocation with yet another symptom of the same race: `fatal error: opening dependency file controller/controller_unit_test.p/ha.c.o.d: No such file or directory` AND simultaneously `fatal error: opening dependency file lib/libxce.a.p/meson-generated_.._controller.capnp.c.o.d: No such file or directory` for two unrelated targets at once ([187/192] and [188/192]) -- gcc could not open its OWN .o.d dependency-file output path, not an input header this time, reinforcing this is a general ninja/meson parallel-build filesystem race in this sandbox, not specific to controller.capnp.h generation. WORKAROUND CONFIRMED AGAIN: simply re-running the identical build command with no source changes succeeded cleanly (85/85 targets, controller_unit_test 3/3 OK), producing fresh xce_controller/xce_test binaries. Do-not-repeat: if xce-build.sh fails with any \"fatal error: opening dependency file ... .o.d: No such file or directory\" or \"controller.capnp.h: No such file or directory\", just retry the same command once before treating it as a real build break -- do not start debugging source code.",
+    "fix": "NONE APPLIED \u2014 this run was build+report only (nix develop ./src -- xce-build.sh controller), not a fix task. Build failed via set -euo pipefail before any tests ran; no controller/xce_controller, controller_unit_test, or xce_test binaries were produced. Next step if it recurs: check meson.build custom_target() for lib/controller.capnp.[ch] has correct output declarations / add explicit depends on all consumers, or file upstream against capnpc-c invocation for atomic output. RECURRENCE 2026-07-16 (2nd occurrence, CONFIRMS the race, different symptom): clean build (rm -rf xce.build) of `nix develop ./src --command bash src/tools/xce-build.sh controller asan` at HEAD 2a76940. This time gcc failed with `fatal error: controller.capnp.h: No such file or directory` (not corrupted content) for hook.c.o, threshold.c.o, xlist.c.o, utils.c.o, window.c.o, xceapi.c.o, wrap.c.o \u2014 all 7 consumers of utils.h#include controller.capnp.h. Critically, ninja's own completion log shows `[12/191] Generating lib/controllercapnp.[ch]` printed AFTER jobs [5..11] (the 7 failed compiles) had already completed/failed. This proves the custom_target that generates lib/controller.capnp.h has no dependency edge forcing it to run before these 7 specific compile units \u2014 confirms the suspected root cause in the original entry (missing depends()/output declaration in meson.build), not just a non-atomic-write timing fluke. Still NOT fixed (build+report only task, no source edits authorized). controller_unit_test and xce_test were never produced this run (clean build wiped the prior binaries and the new build failed before relinking). RECURRENCE 2026-07-17 (3rd occurrence, kiro-validate-impl smoke-boot build for job-array-single-wal-record): fresh (non-clean) build of `nix develop ./src --command bash src/tools/xce-build.sh controller` at HEAD ea8d9d3 failed on the FIRST invocation with yet another symptom of the same race: `fatal error: opening dependency file controller/controller_unit_test.p/ha.c.o.d: No such file or directory` AND simultaneously `fatal error: opening dependency file lib/libxce.a.p/meson-generated_.._controller.capnp.c.o.d: No such file or directory` for two unrelated targets at once ([187/192] and [188/192]) -- gcc could not open its OWN .o.d dependency-file output path, not an input header this time, reinforcing this is a general ninja/meson parallel-build filesystem race in this sandbox, not specific to controller.capnp.h generation. WORKAROUND CONFIRMED AGAIN: simply re-running the identical build command with no source changes succeeded cleanly (85/85 targets, controller_unit_test 3/3 OK), producing fresh xce_controller/xce_test binaries. Do-not-repeat: if xce-build.sh fails with any \"fatal error: opening dependency file ... .o.d: No such file or directory\" or \"controller.capnp.h: No such file or directory\", just retry the same command once before treating it as a real build break -- do not start debugging source code. RECURRENCE 2026-07-20 (4th occurrence, xce-build.sh all asan build+report task validating bug-1453/1454 C-side changes, HEAD after commit 619f41f): clean build failed on FIRST invocation with fatal error: controller.capnp.h: No such file or directory for xlist.c.o, htab.c.o, window.c.o, xstring.c.o, hook.c.o, wrap.c.o, threshold.c.o, xceapi.c.o, utils.c.o, zarray.c.o (10 consumers of utils.h#include controller.capnp.h) -- ninja log again shows [16/353] Generating lib/controllercapnp.[ch] printed AFTER the failed compiles. WORKAROUND CONFIRMED AGAIN (4th time): identical retry with zero source changes succeeded cleanly, 353/353 ninja targets, all 9 meson test suites OK (0 Fail), test_cluster (part of libxce test suite) and scheduler tests all passed. Still unfixed at the meson.build dependency-edge level -- do-not-repeat guidance holds: always retry once before treating this as a real regression.",
     "tags": [
       "ninja-race",
       "controller",
@@ -13185,8 +13185,8 @@
       "capnp"
     ],
     "related_bugs": [],
-    "occurrences": 3,
-    "last_seen": "2026-07-17T09:39:00Z"
+    "occurrences": 4,
+    "last_seen": "2026-07-20T00:00:00Z"
   },
   {
     "id": "bug-1440",
@@ -13518,5 +13518,112 @@
     ],
     "occurrences": 1,
     "last_seen": "2026-07-19"
+  },
+  {
+    "id": "bug-1453",
+    "timestamp": "2026-07-20T00:00:00.000Z",
+    "error_message": "Unnecessary CA certificate plumbing across C daemons and Go CLI \u2014 ca_cert_path fields, --ca-cert flags, and CACert config are dead or misleading; TLS verification is SSL_VERIFY_NONE everywhere",
+    "file": "multiple \u2014 see root_cause",
+    "root_cause": "XCE design uses server-side TLS only (encryption without peer certificate verification). The CA cert plumbing was intended for future P3 mTLS hardening but was never implemented: (1) C-side: zero SSL_CTX_load_verify_locations calls in entire codebase \u2014 ca_cert_path in xce/cluster.h:35 is set by controller/worker/scheduler but never loaded into any SSL_CTX; do_connect() production path in cluster.c:73 is a NULL-returning stub; (2) Go-side: HTTPS is disabled by default; xcecli uses trust-on-first-use pinning of the server leaf cert (not a CA) stored via a misleadingly-named CACert field; falls back to InsecureSkipVerify:true. Design docs (persistence-and-ha-design.md:1700) explicitly tag peer verification as optional P3 hardening, confirming current encryption-only posture is intentional.",
+    "fix": "DONE. C-side: removed ca_cert_path field from xce_cluster_config_t (src/include/xce/cluster.h); do_connect()/xce_connect_fn_t dropped the ca_cert param (src/lib/cluster.c); removed ca_cert_path non-empty check from xce_cluster_config_validate(); removed --ca-cert flag from worker (src/worker/src/main.c) and scheduler (src/scheduler/src/main.c); removed ca_cert_path population from src/controller/main.c:279-281; updated src/lib/testsuite/test_cluster.c mock_connect signature + fixture. Go-side: deleted xcecli/cmd/auth/cert/{cert.go,parser.go} (self-signed cert TOFU-pinning helper); removed CACert field from config.go APIServerInfo and models/logon.go LogonResponse; removed SetRootCAs/loadRootCAs from common/request/request.go and certPath param from Get/Post/DeleteRequestFromAPIServer; updated all ~18 xcecli cmd/* call sites; updated cmd/auth/logon.go to drop the https cert-fetch block; removed TestSetRootCAs_*/TestBuildTLSLogic from request_test.go. Verified: full C build (all components, ASAN) 353/353 targets, meson test suite 9/9 OK; xcecli go build/vet/test all PASS; grep -rn 'CACert|SetRootCAs|loadRootCAs|ca_cert' src/ clean (only historical design-doc mentions remain in src/docs/*.md, left as-is).",
+    "tags": [
+      "dead-code",
+      "tls",
+      "ca-cert",
+      "cleanup",
+      "controller",
+      "worker",
+      "scheduler",
+      "xcecli",
+      "apiserver",
+      "fixed"
+    ],
+    "related_bugs": [],
+    "occurrences": 1,
+    "last_seen": "2026-07-20"
+  },
+  {
+    "id": "bug-1454",
+    "timestamp": "2026-07-20T00:00:00.000Z",
+    "error_message": "Scheduler --peers correctly wired for initial controller discovery; minor issues: nmasters=1 (no EndpointRotator) and leaked ctx on HA path",
+    "file": "src/scheduler/src/scheduler.c",
+    "root_cause": "VERIFICATION: --peers IS correctly parsed via xce_parse_peers() (main.c:129), stored via sched_set_cluster_cfg() (main.c:156), and iterated in sched_init() HA loop (scheduler.c:462-493) BEFORE sched_init \u2014 init ordering is correct. Narrowed findings: (1) nmasters=1 per attempt (line 468) \u2014 manual candidate loop instead of filling config->masters[] from all candidates so libxce EndpointRotator never engages. Functional but suboptimal. (2) Leaked apictx on HA path \u2014 xcecli_ctx_init at line 454 allocates sched->apictx, then HA loop at line 472 re-inits without xcecli_ctx_fini on the first allocation. On standby-failure branches (lines 480, 492) cleanup is correct. OUT OF SCOPE: runtime reconnect/rejoin after controller death belongs to .kiro/specs/component-failover-adoption/ (requirements + design approved, tasks 3.1/3.2 not yet approved).",
+    "fix": "DONE (leak fix only). Fixed the confirmed defect: sched_init() called xcecli_ctx_init(&sched->apictx, config) unconditionally, then on the HA path re-called xcecli_ctx_init() inside the candidate loop without freeing the first ctx -> leaked one xcecli_ctx_t per sched_init() call. Fix: xcecli_ctx_fini(sched->apictx) added immediately before the HA loop starts (src/scheduler/src/scheduler.c), since the loop's first iteration always replaces it. Verified via full ASAN build: all 5 scheduler meson test suites + libxce suite pass, no regression. REJECTED the nmasters>1+EndpointRotator suggestion: memory obs 11825 (this session) confirms scheduler/worker deliberately use the candidates[]-loop path instead of filling config->masters[], so changing that would contradict the established design, not fix a bug. Runtime reconnect/rejoin gap remains out of scope, deferred to component-failover-adoption spec (unchanged from original triage).",
+    "tags": [
+      "scheduler",
+      "peers",
+      "verified-working",
+      "minor-ctx-leak",
+      "nmasters",
+      "fixed"
+    ],
+    "related_bugs": [],
+    "occurrences": 1,
+    "last_seen": "2026-07-20"
+  },
+  {
+    "id": "bug-1455",
+    "timestamp": "2026-07-20T00:00:00.000Z",
+    "error_message": "Scheduler --peers silently ignored: sched_init() memset(sched,0,...) wipes sched->cluster_cfg that main.c populated via sched_set_cluster_cfg() just before calling sched_init(), so the HA candidate loop's candidates_count check always reads 0 and falls back to the hardcoded localhost:9990 single-host path regardless of --peers.",
+    "file": "src/scheduler/src/scheduler.c, src/scheduler/src/main.c, src/scheduler/include/scheduler.h",
+    "root_cause": "sched_set_cluster_cfg(&sched, &cluster_cfg) at main.c was called before sched_init(&sched, &config, handler); sched_init's first action is memset(sched, 0, sizeof(*sched)), which zeroes the cluster_cfg field just written. sched_init's own internal HA-vs-single-host branch reads sched->cluster_cfg.candidates_count immediately after, so it always sees 0. User-reported regression found via code review after bug-1454's leak fix landed in the same function.",
+    "fix": "Restructured instead of patched around: sched_init() signature changed to take `const xce_cluster_config_t*` directly (assigned to sched->cluster_cfg AFTER the memset, inside sched_init, not before via a separate setter). Deleted sched_set_cluster_cfg() (no other callers). main.c hoists cluster_cfg/cluster_cfg_ptr to function scope and passes cluster_cfg_ptr (NULL or &cluster_cfg) straight into sched_init(). Also fixed an adjacent smell in the same HA loop: it used to mutate the caller's xce_client_config_t* config in place per candidate, leaving it in a surprising mutated state after sched_init returned \u2014 changed to a per-iteration local copy. Verified: full ASAN build (all components) 353/353 targets, meson 9/9 OK, manual scheduler testsuite 15/15, Go tests all pass. No test exercises sched_init's HA path with real --peers input (pre-existing gap, not introduced by this fix).",
+    "tags": [
+      "scheduler",
+      "peers",
+      "ha",
+      "config",
+      "api-design",
+      "fixed"
+    ],
+    "related_bugs": [
+      "bug-1454"
+    ],
+    "occurrences": 1,
+    "last_seen": "2026-07-20"
+  },
+  {
+    "id": "bug-1456",
+    "timestamp": "2026-07-20T00:00:00.000Z",
+    "error_message": "Dangling pointer risk in sched_init() HA candidate loop: a later refactor (following bug-1454) changed the loop to build a per-iteration stack-local xce_client_config_t candidate_config and pass &candidate_config into xcecli_ctx_init().",
+    "file": "src/scheduler/src/scheduler.c",
+    "root_cause": "xcecli_ctx_init() (src/lib/client.c) stores the raw pointer it is given as ctx->conf \u2014 it does not copy the struct. That pointer is dereferenced later, long after sched_init() returns, by client_failover.c runtime reconnect/rotation logic (e.g. ac->conf->masters[next]). When the candidate loop used a stack-local xce_client_config_t per iteration, ctx->conf pointed at a stack frame that no longer existed once sched_init() returned \u2014 a classic dangling pointer that only manifests later during HA reconnect/rotation, making it easy to miss in a build+unit-test pass.",
+    "fix": "Reverted to mutating the caller-supplied xce_client_config_t* config parameter in place (config->nmasters, config->masters[0], config->master_ports[0]) instead of a stack-local copy. config is owned by main()'s stack frame, which lives for the whole process, so storing &config-derived state in ctx->conf is lifetime-safe. Added a comment in scheduler.c explaining why the stack-local-copy pattern is unsafe here, to prevent it from being reintroduced.",
+    "tags": [
+      "scheduler",
+      "ha",
+      "dangling-pointer",
+      "lifetime",
+      "client-config",
+      "fixed",
+      "regression-of-bug-1454"
+    ],
+    "related_bugs": [
+      "bug-1454",
+      "bug-1455"
+    ],
+    "occurrences": 1,
+    "last_seen": "2026-07-20"
+  },
+  {
+    "id": "bug-1457",
+    "timestamp": "2026-07-20T22:56:00Z",
+    "error_message": "xce_controller with no --peers flag exits with -EINVAL from cluster_open_listener instead of starting as single-node primary",
+    "file": "src/controller/myapp.c",
+    "root_cause": "Discovered while independently reviewing clusterconfig-schema-split task 5.3 by building and running the actual pre-spec baseline (commit 94f50ce) for comparison. Single-node startup (no --peers) was ALREADY broken before this spec began: myapp_run() calls cluster_open_listener() before the primary config (cluster_config) is loaded from disk, so the pre-spec fallback branch (cluster_config->port, only reachable when cluster_config != NULL) always hit its else branch and returned -EINVAL, unconditionally, for every zero-flag single-node start. This invalidates part of clusterconfig-schema-split's premise (requirements.md's Introduction claimed cluster_config->port was 'the sole live port-sourcing mechanism for single-node deployments' and 'single-node controller startup SHALL continue to work exactly as it does today' — in reality it did not work today). Not caused by, or a regression from, clusterconfig-schema-split's changes — task 2.1's port-resolution rewrite (commit 1229f9a) incidentally fixed this as a side effect, since its hardcoded-default fallback no longer depends on cluster_config being loaded.",
+    "fix": "Not fixed as its own change — already incidentally repaired by clusterconfig-schema-split task 2.1 (cluster_open_listener now falls back to a hardcoded XCE_CONTROLLER_DEFAULT_PORT constant instead of an unloaded cluster_config->port). Live-verified post-fix: single-node startup with no --peers now correctly binds port 9990 and reaches CTRL_PRIMARY. No further action needed unless a future audit wants to correct the historical claim in clusterconfig-schema-split's requirements.md/design.md that pre-spec single-node startup was previously working — it was not, for an unrelated reason (myapp_run's listener-open/config-load ordering).",
+    "tags": [
+      "controller",
+      "startup",
+      "single-node",
+      "pre-existing",
+      "myapp_run",
+      "cluster_open_listener",
+      "discovered-during-review",
+      "incidentally-fixed"
+    ],
+    "related_bugs": [],
+    "occurrences": 1,
+    "last_seen": "2026-07-20"
   }
 ]