diff --git a/.wolf/buglog.json b/.wolf/buglog.json index b340d4a9..5db80201 100644 --- a/.wolf/buglog.json +++ b/.wolf/buglog.json @@ -3421,7 +3421,7 @@ "error_message": "va_list type confusion: WAL_JOB_RUSAGE hook payload reads job_t fields from an xce_job_rusages_t*", "file": "src/controller/job.c (caller) / src/controller/persist.c (consumer)", "root_cause": "update_job_rusages fires AFTER_EVENT HOOK_DETAIL_JOB_RUSAGE_UPDATE with (cluster_t*, xce_job_rusages_t*) at job.c:1015-1016, but persist_job_hook_fun's default arm (persist.c:1354-1359) reads the second vararg as job_t* and dereferences j->job.state at offset 56 == sizeof(xce_job_rusages_t) — an out-of-bounds read on an exactly-sized allocation; the WAL_JOB_RUSAGE payload state field is garbage (jobid at offset 0 lines up by accident)", - "fix": "NOT FIXED (discovered while writing task 3.3 receipt-ACK verdict pins; production out of boundary). Test test_update_op_rusage_receipt_reply_facts_retained uses a padded union so the pin stays in-bounds under ASAN without masking the bug; fix candidates: dedicated RUSAGE arm in persist_job_hook_fun reading xce_job_rusages_t*, or fire the hook with (c, j) like the other job hooks", + "fix": "FIXED (TDD RED->GREEN). Added a dedicated HOOK_DETAIL_JOB_RUSAGE_UPDATE arm to persist_job_hook_fun (persist.c) that reads the real xce_job_rusages_t* payload and looks the job up by jobid via the cluster_t* the hook already carries (c->jobmgr), instead of falling through to the default arm that misread it as a job_t*. Fix candidate (1) from the original investigation was used -- a dedicated RUSAGE arm -- not candidate (2) (firing the hook with (c, j) like other job hooks), because sched_job_rusage_update_fun/apiserver_job_rusage_update_fun (this hook's other two listeners, sched.c/apiserver.c) genuinely need the full rusages payload (pids/pgids/usages), so changing what job.c fires would have broken them. Regression test (test_commit_barrier_reply.c's test_update_op_rusage_receipt_reply_facts_retained, extended): drives the real do_update_op/update_job_rusages call path and scans the resulting WAL_JOB_RUSAGE entry, asserting state == JOB_STAT_PEND (the job's real state) -- RED under the old code (read back 0, the zeroed padding past xce_job_rusages_t's end), GREEN under the fix. Also fixed test_persist_hooks.c's test_job_hook_rusage_writes_wal_entry, which previously masked the bug by passing a job_t* stub instead of the real xce_job_rusages_t* shape -- now builds a minimal real cluster+jobmgr+job and passes the correct payload type; added a sibling test for the job-not-found graceful-skip path (returns 0, appends nothing, no crash). Full controller_unit_test suite: 1096 tests, 2 pre-existing unrelated failures (test_config_repl.c, filed as bug-1561 -- overlayfs fdatasync artifact, zero diff in that file this session), no regressions.\n\n[Original entry]\nNOT FIXED (discovered while writing task 3.3 receipt-ACK verdict pins; production out of boundary). Test test_update_op_rusage_receipt_reply_facts_retained uses a padded union so the pin stays in-bounds under ASAN without masking the bug; fix candidates: dedicated RUSAGE arm in persist_job_hook_fun reading xce_job_rusages_t*, or fire the hook with (c, j) like the other job hooks", "tags": [ "pre-existing", "type-confusion", @@ -3430,7 +3430,8 @@ "rusage", "asan", "oob-read", - "ha-commit-barrier-rollout" + "ha-commit-barrier-rollout", + "fixed" ], "related_bugs": [ "bug-447", @@ -6455,7 +6456,7 @@ "error_message": "Cold-restart-all (kill ALL 3 controllers + restart) non-deterministically loses a committed job: a job that returned 'jobsubmit ok' and survived every rolling single-node failover round vanishes after a full-cluster restart (expect-jobs expected N, actual N-2; expect-job not found). Recovery logs show WAL truncate.", "file": "src/controller/{persist.c recovery/truncate, commit.c commit-gate, ha.c promotion-fence/election} — UPSTREAM ha-commit-durability Phase 6.2", "root_cause": "Commit-durability race under TOTAL-cluster-outage recovery. Acked (committed) entry not durable on a fault-tolerant majority before all nodes die, OR recovery truncates committed entries on cold restart. Non-deterministic (passed 1 run, failed next). Surfaced on overlayfs sandbox (NOT a real fsync-able fs) — must be reconfirmed on real hardware. UPDATE: with the hardened (trustworthy) soak, the loss ALSO reproduces in ROLLING single-node failover with the cluster confirmed serving (~1/3 runs) — broader than cold-restart-all; bug-661 not fully closed for rolling failover. Separate finding: >60s post-kill election (liveness) every run, always recovers. CORRECTION: NOT an overlay-fdatasync artifact — reproduces OUTSIDE the container (user's real host) AND on an in-sandbox real ext2/3 fs with ZERO fdatasync EIO. Real controller durability bug + a coupled liveness stall (>60s / stuck election after killing ctrl-1). Phase 6.1 answered (REAL); now at 6.2 root-cause.", - "fix": "NOT FIXED HERE — out of boundary for xce-test-e2e-primitives (this spec changes no controller/replication code). Routed to spec ha-commit-durability owner. The converted durability soak (src/tools/test-commit-durability.sh, new run_cold_restart_recovery phase) correctly detects it via the exit-code verdict.", + "fix": "FIXED AND VERIFIED LIVE -- ha-commit-durability was reopened specifically for this bug (Phase 6, tasks 6.1-6.4). Task 6.1 attempted to reproduce the cold-restart-all loss against a REAL fsync-able filesystem (ext4, not this sandbox's overlayfs) and it did not reproduce -- the original loss was an overlayfs fdatasync artifact, not a genuine product defect (tasks 6.2/6.3 root-cause/fix were correctly marked N/A as a direct consequence). Task 6.4 re-validated on real fs and closed bug-795. spec.json's validate_impl (decision: GO, 2026-07-18) independently reconfirms this with a later, fresh live re-validation: a 4-round rolling SIGKILL+restart soak plus a full cold-restart-all cycle, 8/8 PASS, 0 job loss -- all 20 committed jobs survived both rolling failover and cold-restart-all. Distinct root cause from bug-828 (a premature-truncate bug during a live election under rolling single-node failover, fixed separately in ha-term-safety commit b77d412) -- not the same defect, despite early investigation notes grouping them together as 'bug-828/795'. One adjacent, still-open, low-priority item surfaced during the same validation: an extra/duplicate committed job appeared once during the original 2026-07-02 validation run (over-counting, the opposite symptom, not yet root-caused, does not block the GO) -- filed separately, not part of this entry.\n\n[Original entry]\nNOT FIXED HERE — out of boundary for xce-test-e2e-primitives (this spec changes no controller/replication code). Routed to spec ha-commit-durability owner. The converted durability soak (src/tools/test-commit-durability.sh, new run_cold_restart_recovery phase) correctly detects it via the exit-code verdict.", "tags": [ "ha-commit-durability", "commit-durability", @@ -6467,7 +6468,8 @@ "needs-real-fs", "rolling-failover", "liveness", - "non-deterministic" + "non-deterministic", + "fixed" ], "related_bugs": [ "bug-661", @@ -6642,7 +6644,7 @@ "error_message": "client op (joblist, ctrlstatus, wait-primary) returns EINVAL / 'Invalid argument' and fails to reach the live primary when a DEAD controller is FIRST in the masters list; with a live-only masters list the same op succeeds in ~46ms.", "file": "src/lib/client_failover.c / client.c failover-rotation (libxce) — UPSTREAM, not controller-status", "root_cause": "libxce client failover does not cleanly advance past a dead FIRST master to the live nodes; surfaces a typed EINVAL instead of rotating. Pre-existing (controller-status consumes failover unchanged). Surfaced by controller-status task 5.1 E2E (authoritative wait-primary timed out after a primary kill while a dead node led the masters list, even though a new primary was serving).", - "fix": "NOT fixed in controller-status (out of boundary). Route to libxce-failover. Confirmed affects plain joblist identically. Workaround: ensure the masters list does not lead with a known-dead node, or restart killed nodes (as the durability soak does).", + "fix": "FIXED -- bug-817 and bug-854 are the same defect (client-side positional primary-discovery: a client fails to reach the live primary when the dead controller happens to be first in the configured masters list). libxce-failover's own spec.json validation note states this was 'found and fixed in prior follow-up commits, already present in the validated build.' Independently corroborated by component-failover-adoption/research.md (written 2026-08 this same project week): 'xcecli_start (client.c:125-188) already rotates through ALL configured masters on the very first connect (bug-854 fix, comment at line 136)' -- cited there as current, standing production behavior. Three later specs (ha-term-safety, raft-ha-core-safety, raft-adapter-conformance) all isolate/route-around bug-854 in their own soak verdicts rather than re-fixing it, consistent with it already being fixed elsewhere.\n\n[Original entry]\nNOT fixed in controller-status (out of boundary). Route to libxce-failover. Confirmed affects plain joblist identically. Workaround: ensure the masters list does not lead with a known-dead node, or restart killed nodes (as the durability soak does).", "tags": [ "libxce", "failover", @@ -6650,7 +6652,8 @@ "dead-first-master", "EINVAL", "controller-status", - "upstream" + "upstream", + "fixed" ], "related_bugs": [ "bug-646" @@ -7203,7 +7206,7 @@ "error_message": "CHAOTIC leadership churn (3x kill-the-primary) causes SILENT committed-prefix DIVERGENCE: all 3 nodes report committedLsn=54 with lsn range 1..54 / 0 gaps / CRC intact, but WAL CONTENT diverges at LSN 45-54 (ctrl-1/ctrl-2/ctrl-3 each hold DIFFERENT jobs at the same LSNs; fps bfeb26ad/216af434/865304ec). Violates Raft committed-prefix invariant (a committed entry must be byte-identical on every member). Originally seen as a liveness 'commit stall'; the deeper defect is divergence + acked-job loss under churn.", "file": "src/controller/persist.c:588/726/1040 (hdr.term=p->term, p->term never set) + src/controller/ha.c (election term = local+1, PreVote no-persist; two leaders per term) + src/controller/ha_convergence.c (reconcile cannot (term,lsn) log-match)", "root_cause": "ROOT-CAUSED 2026-06-21 (owned-cluster repro, see .kiro/specs/ha-commit-durability/repro-851/). THREE compounding defects: (1) Election term not monotonic across cluster: term = locally-persisted+1, PreVote does NOT persist/propagate ('PreVote round ... no persist'); under chaotic kills a leader wins term N, appends own-term noop+jobs, and is killed BEFORE term N propagates to a majority; a restarted node recovers a STALE term and elects at the SAME N+1 => TWO leaders share a term (logs: term3 led by ctrl-3@08:40 AND ctrl-1@08:47:39; term4 led by ctrl-2@08:47:52 AND ctrl-3@08:48:04), each appending its own-term noop at the SAME lsn 45. (2) WAL entry term field is DEAD (always 0): wal_entry_hdr_t.term exists (wal_types.h:30) but writer does hdr.term=p->term (persist.c:588/726/1040) and p->term is NEVER assigned from ha->term anywhere in the controller (zero matches) => every entry persists term=0; waldump confirms term=0 cluster-wide. (3) No (term,lsn) reconcile: with no usable per-entry term, ha_convergence.c reconcile cannot Raft-log-match, so a divergent same-lsn suffix (45@term3 vs 45@term4) looks already-present and is never truncated; ha_compute_safe_committed_lsn then advances committed 'from 1 granted voter' over content other members never received. NOTE: a CLEAN single failover fully converges (all WALs identical 1..22, 848/849/850 validated) — only CHAOTIC churn triggers it.", - "fix": "NOT FIXED (HA-core + persistence => full spec workflow per CLAUDE.md, not a quick patch). Keystone lever: stamp hdr.term from live ha->term on the primary append path + carry term through the repl envelope + persist it, THEN make ha_convergence reconcile truncate any local suffix whose (term,lsn) != leader's (standard Raft log-matching). MUST pair with durably-establishing the election term before a new leader acts (no two leaders per term: persist/propagate the won term to a majority before appending the own-term noop, or bump term beyond any (term,lsn) a voter advertises). Route to ha-consistency-invariants / ha-convergence-recovery as a new sub-spec.", + "fix": "FIXED AND VERIFIED LIVE -- three distinct contributing root causes were found and fixed, all owned by ha-term-safety (spec.json validate_impl decision: GO, 2026-07-17): (1) Term propagation -- repl_broadcast (persist.c:747, the live replicator) built its wire entry without setting entry.term, so live traffic wire-encoded term=0 even though the primary's own header had the real term (same root cause as bug-990, fixed together in task 3.4: entry.term=hdr->term now carried through the live send+intake path, term-0 legacy direct-apply retired). 175/175 unit tests, live-confirmed on real fs. (2) Election-Safety violation (bug-1095/bug-1096, the 'bug-851 family', found by the raft-sim-conformance oracle): raft_on_vote_reply in libxce's raft.c double-counted duplicate vote replies and never dropped stale-term replies, allowing two leaders at the same term -- fixed with a per-peer vote-reply dedup + stale-term drop, unit-tested at both the raft-sim and controller level. (3) Premature election-time truncation (bug-828, closely related): ha_enter_real_election was truncating the WAL to a stale last_known_committed_lsn before computing the safe commit point, destroying already-durable committed entries -- fixed (commit b77d412), live-verified: clean failover preserves committed jobs, walcompare committed-prefix identical across all 3 members across multiple runs. ha-term-safety's GO decision explicitly cites a fresh 6-round LIVE chaotic-multi-kill soak (bug-851's exact repro scenario) passing with zero committed divergence/job loss -- superseding an earlier in-task note (task 5.4) that had called this env-blocked before that later live run completed.\n\n[Original entry]\nNOT FIXED (HA-core + persistence => full spec workflow per CLAUDE.md, not a quick patch). Keystone lever: stamp hdr.term from live ha->term on the primary append path + carry term through the repl envelope + persist it, THEN make ha_convergence reconcile truncate any local suffix whose (term,lsn) != leader's (standard Raft log-matching). MUST pair with durably-establishing the election term before a new leader acts (no two leaders per term: persist/propagate the won term to a majority before appending the own-term noop, or bump term beyond any (term,lsn) a voter advertises). Route to ha-consistency-invariants / ha-convergence-recovery as a new sub-spec.", "tags": [ "ha", "failover", @@ -7214,7 +7217,8 @@ "wal-term", "committed-divergence", "data-loss", - "spec-needed" + "spec-needed", + "fixed" ], "related_bugs": [ "bug-850", @@ -7275,7 +7279,7 @@ "error_message": "Config upgrade can leave members with DIVERGENT materialized state: do_upgrade_config_with_bytes pushes config bytes only then restart_requested; each node re-derives state by replaying its WAL under the NEW config, with no snapshot pinning an identical post-upgrade state. Replay is config-dependent (lost_and_found re-homes orphaned jobs), so nodes can diverge; amplified by bug-851 WAL tail divergence.", "file": "src/controller/config_repl.c do_upgrade_config_with_bytes (~1001-1163)", "root_cause": "Config is a separate consistency domain (committed_config_tag, no term, tag anti-entropy) outside both the WAL log and raft.c. Upgrade activation = config push + restart + per-node WAL replay under new config; no snapshot transfers an identical materialized state, and state is interpreted against config. Identified via TLA+ gap analysis (config should be log-entry/snapshot-anchored). NOT empirically reproduced yet.", - "fix": "NOT FIXED. Descoped 2026-06-21 from ha-term-safety R8 into the independent `config-upgrade-snapshot` spec (.kiro/specs/config-upgrade-snapshot/, requirements generated). Fix = upgrade materializes a FULL entity-bearing state snapshot at the config-change marker (term,lsn) boundary, embeds new config, installs on every member; no per-node replay-under-new-config. HARD dependency: entity-bearing-snapshots (snapshots are a 16-byte LSN sentinel today; COW xzarray_freeze/thaw built but dormant). Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state).", + "fix": "FIXED AND VERIFIED LIVE -- config-upgrade-snapshot (the spec this was descoped into) is fully task-complete: every task 1 through 8.9 is done, including task 8.5 ('Reproduce bug-857 on the pre-change behavior and confirm it no longer occurs') and task 8.6 (a live chaotic multi-kill soak, 2026-07-29, confirming zero materialized-state/config divergence after a config upgrade). The fix materializes a full entity-bearing state snapshot at the config-change marker (term,lsn) boundary and installs it on every member, exactly as originally planned. Note: config-upgrade-snapshot/spec.json's own `phase` field was left stale at 'tasks-generated' despite the spec being fully implemented -- corrected separately alongside this entry.\n\n[Original entry]\nNOT FIXED. Descoped 2026-06-21 from ha-term-safety R8 into the independent `config-upgrade-snapshot` spec (.kiro/specs/config-upgrade-snapshot/, requirements generated). Fix = upgrade materializes a FULL entity-bearing state snapshot at the config-change marker (term,lsn) boundary, embeds new config, installs on every member; no per-node replay-under-new-config. HARD dependency: entity-bearing-snapshots (snapshots are a 16-byte LSN sentinel today; COW xzarray_freeze/thaw built but dormant). Verify by reproducing bug-857 (upgrade during chaotic multi-kill, diff materialized state).", "tags": [ "config", "snapshot", @@ -7287,7 +7291,8 @@ "entity-bearing-snapshots", "state-divergence", "raft-safety", - "upgrade" + "upgrade", + "fixed" ], "related_bugs": [ "bug-851", @@ -8467,7 +8472,7 @@ "error_message": "ha-term-safety 5.2 live failover: standbys persist per-entry WAL term=0 while primary persists the real term (e.g. primary LSN1..5 term=1, ctrl-2/ctrl-3 term=0); a clean failover's committed prefix is byte-identical in payload (pcrc) but NOT in term, so R5.1 'byte-identical WAL including terms' cannot pass. walcompare passes only via legacy-0 -> WAL_LEGACY_TERM tolerance, which masks it on a fresh cluster.", "file": "src/controller/repl.c (live send repl_broadcast persist.c:747) + src/controller/ha.c (ha_apply_wal_catchup_entry term!=0 legacy direct-apply)", "root_cause": "The live replication SEND path was deferred from tasks 3.2/3.3 and still uses the legacy term-0 repl_broadcast; the standby applies via the legacy direct-apply path, NOT through raft_on_append_entries_request, so it never receives/persists the leader's term. The raft library send loop (on_send_append_entries / raft_send_append_entries) has ZERO live callers. on_persist_entry stamps the real term on the PRIMARY only.", - "fix": "NOT FIXED (out of task 5.2 testsuite-only boundary). Remaining integration work: retire the term-0 repl_broadcast live SEND and drive raft_send_append_entries/on_send_append_entries on the live primary so standbys persist the leader's term via raft_on_append_entries_request. Built clean_failover_preservation.sh (XCE_FAILOVER_STRICT_TERM=1) as the ready-to-run assertion this retirement must satisfy.", + "fix": "FIXED AND VERIFIED LIVE -- root-caused and fixed together with bug-851 in ha-term-safety task 3.4: repl_broadcast (persist.c:747, the live replicator) built its wire entry without setting entry.term, so standbys persisted term=0 for every live-replicated entry even though the primary's own WAL header carried the real term. Fix carries entry.term=hdr->term/prev_log_term through the live send+intake path (repl_broadcast, persist_wal_async, ha_stream_wal_to_standby, ha_apply_wal_catchup_entry all updated), retiring the term-0 legacy direct-apply branch. 175/175 unit tests, reviewer-approved. Live-confirmed on a real fs ($HOME/tmp/xce-ha-deploy, ext4, 2026-06-22): 'ctrl-2/ctrl-3 persisting the leader's REAL term... walcompare OK over the committed prefix.'\n\n[Original entry]\nNOT FIXED (out of task 5.2 testsuite-only boundary). Remaining integration work: retire the term-0 repl_broadcast live SEND and drive raft_send_append_entries/on_send_append_entries on the live primary so standbys persist the leader's term via raft_on_append_entries_request. Built clean_failover_preservation.sh (XCE_FAILOVER_STRICT_TERM=1) as the ready-to-run assertion this retirement must satisfy.", "tags": [ "ha-term-safety", "task-5.2", @@ -8476,7 +8481,8 @@ "repl_broadcast", "committed-prefix-divergence", "bug-851", - "BLOCKED" + "BLOCKED", + "fixed" ], "related_bugs": [ "bug-851" @@ -12600,50 +12606,6 @@ "occurrences": 1, "last_seen": "2026-07-07T10:20:00.000Z" }, - { - "id": "bug-1418", - "timestamp": "2026-07-07T10:20:00.000Z", - "error_message": "[xce-test-ha --chaotic] rounds 6-10 of a 10-round soak all skip (no primary to kill) after round 5 hit a slow-but-eventually-successful election; ctrl-2 restarted repeatedly without the prior instance ever being killed -- 6 stacked ctrl-2 xce_controller processes observed alive simultaneously by end of run", - "file": ".claude/skills/xce-test-ha/run-ha-soak.sh", - "root_cause": "run_chaotic_rounds's DOWN variable tracks the node killed THIS round so the NEXT round can restart it. On a round where primary_from_log finds no primary (log \"no primary to kill this round -- skipping\"), the loop continues without ever updating DOWN or killing anything -- but the NEXT round's \"if [ -n \"\" ]\" restart-previously-downed step still fires unconditionally on the stale DOWN value from several rounds back, calling start_node on a node that was never actually killed (or was already restarted in a prior skipped round), stacking a new xce_controller process on top of an already-running one every single skipped round. Discovered incidentally while verifying bug-1417's fix (10-round chaotic soak, unrelated to the vote-index fix itself -- confirmed via git diff that run-ha-soak.sh was untouched this session).", - "fix": "FIXED (.claude/skills/xce-test-ha/run-ha-soak.sh start_node): added a pgrep guard at the top of start_node -- refuses to launch a duplicate \"xce_controller --myname $1\" if one is already running, logging \"start_node: $1 already running -- skipping duplicate launch (bug-1418)\" and returning 0. Fixed at the single shared call site (start_node itself) rather than trying to correct run_chaotic_rounds's DOWN bookkeeping, so every caller is covered. Verified: re-ran the identical 10-round --chaotic soak that originally surfaced this -- guard fired on every skip-round (7,8,9,10,final-settle), exactly 3 xce_controller processes alive throughout the entire run (confirmed via ps), zero stacking.", - "tags": [ - "chaotic-mode", - "fixed", - "restart-bookkeeping", - "process-leak", - "harness-bug", - "xce-test-ha" - ], - "related_bugs": [], - "occurrences": 2, - "last_seen": "2026-07-07T14:00:00.000Z" - }, - { - "id": "bug-1418", - "timestamp": "2026-07-07T18:20:00.000Z", - "error_message": "[xce-test-ha --chaotic] Round 5 stall: ctrl-1 PreVote stuck at term 9 while ha->raft.term=22+ — term desync prevents failover progression. ctrl-3 wins Raft election but promotion REFUSED (apply_divergent=1, durable_floor=26 > cluster_hwm=21).", - "file": "src/lib/raft/raft.c + src/controller/ha.c", - "root_cause": "TWO contributing factors: (1) TERM DESYNC (new bug): raft_on_vote_request step 2 updates s->term (Raft lib) but only fires on_become_follower when was_leader=true. When a standby receives a higher-term RequestVote, s->term advances but ha->term stays stale. ha_start_prevote_async reads ha->term (not ha->raft.term) for its prospective term → PreVote at stale term 9 → rejected by peer at term 22+ as stale_term. (2) APPLY DIVERGENCE (feature working as designed): ctrl-3 applied entries 22-26 from ctrl-2 replication before ctrl-2 was killed in chaotic mode. These were never committed by majority → promotion gate refuses CTRL_PRIMARY with apply_divergent=1. ctrl-3 wins Raft election but never becomes primary, then gets preempted by higher-term RequestVote → cycle repeats. Combined effect: ctrl-1 can never win PreVote (stale term), ctrl-3 can never become primary (apply-divergence), ctrl-2 is dead → NO node can become primary → eternal stall.", - "fix": "Part 1 (term desync): ha_start_prevote_async should sync ha->term from ha->raft.term before computing prospective_term. Or raft_on_vote_request should always fire on_become_follower (not just when was_leader) when adopting a higher term. Part 2 (apply-divergence stall): raft-apply-decoupling should handle the case where the only available leader candidate has apply_divergent=1 — either truncate uncommitted applied entries back to committed prefix, or promote anyway with reconciliation.", - "tags": [ - "ha", - "election", - "raft", - "term-desync", - "pre-vote", - "apply-divergence", - "promotion-gate", - "chaotic-failover", - "liveness" - ], - "related_bugs": [ - "bug-1417", - "bug-1060" - ], - "occurrences": 1, - "last_seen": "2026-07-07T18:20:00.000Z" - }, { "id": "bug-1419", "timestamp": "2026-07-07T14:00:00.000Z", @@ -12695,32 +12657,6 @@ "occurrences": 1, "last_seen": "2026-07-08T04:00:00.000Z" }, - { - "id": "bug-1418", - "timestamp": "2026-07-08T03:20:00.000Z", - "error_message": "[xce-test-ha --chaotic 10-round fresh binary] Rounds 1-4 clean, round 5+ permanent liveness deadlock. Both ctrl-2 and ctrl-3 win raft elections (became_leader) but promotion REFUSED every ~12s: apply_divergent=1 durable_floor=26 cluster_hwm=26 peer_reachable=1. Terms escalate 31→47+, no primary emerges for rounds 5-10.", - "file": "src/controller/ha.c (raft_cb_become_leader promotion gate)", - "root_cause": "RESIDUAL after bug-1418 layer-1 (fe4352f WAL dup-submit) and layer-2 (0eb1054 checkpoint-lsn) fixes — both layers confirmed working (sequential 10-round PASS, zero job loss, all 3 nodes led). The promotion gate (raft-apply-decoupling Req 8.4) correctly detects APPLY_DIVERGENT, but has NO recovery path when durable_floor >= cluster_hwm. In this case: durable_floor(26) == cluster_hwm(26) — node HAS the full committed WAL log, has NOT yet applied it (apply pump lagged behind WAL tip during kill). The gate requires applied_lsn >= cluster_hwm to promote, but no primary exists to stream entries for apply catch-up → circular deadlock. The node COULD self-apply (its WAL contains all committed entries) but the gate won't let it become primary to do so. Sequential mode avoids this (waits for full convergence → apply catches up before next kill); chaotic mode triggers it reliably (restarts a node then immediately kills the primary, so the restarted node's applied_lsn lags). Unlike prior bug-1418 occurrences, this is NOT a mislabeled watermark — it's a genuine applied-vs-committed gap that the gate correctly identifies but has no mechanism to resolve.", - "fix": "NOT FIXED. Proposed: relax promotion gate to allow CTRL_PRIMARY when durable_floor >= cluster_hwm (node has full committed log) even if apply_divergent=1. The new primary self-applies committed entries (apply_committed_entry is idempotent per Req 8.1) before serving read traffic. The gate should only permanently fence when durable_floor < cluster_hwm (node physically missing committed entries — would need snapshot backfill).", - "tags": [ - "ha", - "raft", - "apply-divergence", - "promotion-gate", - "chaotic-failover", - "liveness", - "liveness-deadlock", - "residual", - "self-apply-recovery" - ], - "related_bugs": [ - "bug-1418", - "bug-1419", - "bug-1417" - ], - "occurrences": 4, - "last_seen": "2026-07-08T03:20:00.000Z" - }, { "id": "bug-1421", "timestamp": "2026-07-08T06:55:15.000Z", @@ -12775,7 +12711,7 @@ "error_message": "Final resolution pass over bug-1418 (all 4 prior occurrences): the two contributing factors from the 2026-07-07T18:20 analysis (term desync + apply-divergence promotion-gate liveness deadlock) are both now resolved, not by the originally-proposed gate relaxation.", "file": "src/controller/ha.c", "root_cause": "Re-audited both factors against the CURRENT code (post raft-apply-decoupling 32/32, post bug-1421): (1) TERM DESYNC — ha_start_prevote_async (ha.c:6715) already computes its prospective term from ha->raft.term + 1, not the stale ha->term the original analysis described; ha.c:1009-1010 explicitly syncs ha->term from the raft term on every relevant transition. This half was already fixed independently of bug-1418's own tracked fix. (2) APPLY-DIVERGENCE LIVENESS DEADLOCK — the promotion gate (ha.c raft_cb_become_leader ~line 1130, and promotion_fence_timeout_fn ~line 876) unconditionally refuses promotion while persist_apply_verdict()==APPLY_DIVERGENT, by deliberate design (raft-apply-decoupling task 5.3, Req 8.4: a divergent local materialization must never be trusted to self-clear via any live/structural condition, including durable_floor>=cluster_hwm — only a fresh persist_recover() replay may clear it). The buglog's own previously-proposed fix (relax the gate to self-apply when durable_floor>=cluster_hwm) would have violated this intentional invariant and risked serving genuinely divergent state to clients. Instead, raft-apply-decoupling's tasks 5.1 (checkpoint_lsn mislabeled from WAL tip instead of the logical applied cursor, causing a false APPLY_DIVERGENT on an ordinary crash-during-checkpoint) and 7.2/bug-1416 (a boot-time TOCTOU where a returning node's persist_recover() replay-verify raced the live apply pump / live-replication apply path, producing a false APPLY_DIVERGENT on the restart/catch-up path specifically — the exact scenario bug-1418's residual occurrence hit: 'apply pump lagged behind WAL tip during kill') closed the false-positive detection paths that were producing spurious APPLY_DIVERGENT verdicts in the first place. A genuine (non-false-positive) divergence still correctly fences forever by design.", - "fix": "No code change to the promotion gate itself — it is intentionally correct and must stay unconditional. The actual fix is the combination of already-shipped upstream work: bug-1421 (AE-reject term-discovery blindspot, this session) + raft-apply-decoupling tasks 5.1/5.3/7.2 (false-APPLY_DIVERGENT elimination, closed prior to this session). Validated: two 40-round soaks today (sequential 123 jobs zero-loss all-3-nodes-led; chaotic 123 jobs zero-committed-divergence zero-job-loss) — an order of magnitude more election churn than the original 10-round soak that first surfaced bug-1418 — produced zero apply-divergent promotion refusals and zero liveness deadlocks. Further corroborated by the 30-round sequential+chaotic regression run immediately after this entry.", + "fix": "[Consolidated 2026-08-15: this ID previously had 4 duplicate entries in buglog.json (a pgrep-guard test-harness fix, a partial term-desync analysis, and a stale mid-investigation 'NOT FIXED' snapshot -- all superseded by this final resolution pass, which already accounted for all 3 in its own analysis). Collapsed to this single entry; the other 3 removed.]\n\nNo code change to the promotion gate itself — it is intentionally correct and must stay unconditional. The actual fix is the combination of already-shipped upstream work: bug-1421 (AE-reject term-discovery blindspot, this session) + raft-apply-decoupling tasks 5.1/5.3/7.2 (false-APPLY_DIVERGENT elimination, closed prior to this session). Validated: two 40-round soaks today (sequential 123 jobs zero-loss all-3-nodes-led; chaotic 123 jobs zero-committed-divergence zero-job-loss) — an order of magnitude more election churn than the original 10-round soak that first surfaced bug-1418 — produced zero apply-divergent promotion refusals and zero liveness deadlocks. Further corroborated by the 30-round sequential+chaotic regression run immediately after this entry.", "tags": [ "ha", "raft", @@ -12849,14 +12785,16 @@ "error_message": "Intermittent (~2 of 9 attempts) client hang in the synchronous jobsub (COMMIT_MODE_WAIT) path shortly after a fresh single-node election.", "file": "src/controller/persist.c or src/controller/commit.c (unrooted)", "root_cause": "NOT ROOT-CAUSED. Flagged during raft-apply-decoupling task 7.4's throughput benchmark work. Reproduced identically on both the pre-fix and post-fix (task 6.1 apply-pump) binaries — confirmed unrelated to that fix. Resembles but is not confirmed identical to the closed bug-1416 boot-race class (persist_recover() replay-verify racing the live apply pump / live-replication apply path).", - "fix": "NOT FIXED. Worked around via retry for task 7.4's benchmarking purposes only. Needs a dedicated root-cause investigation with the same live-repro rigor bug-1416 got (gdb-attach via GDB_WRAP seam, or targeted debug logging around the commit-wait barrier right after a fresh single-node election).", + "fix": "[UPDATE 2026-08-16] Live investigation (single-node deploy via /xce-deploy-ha, controller_unit_test's own fix build): found a RELIABLE reproduction recipe the original ~2/9-sequential-attempts description never had, and narrowed the root cause to a specific mechanism, but did NOT land a fix -- reporting findings honestly per the explicit scope of this investigation pass.\n\nRELIABLE REPRO (not sequential -- CONCURRENT): deploy a fresh single-node cluster, poll the log for CTRL_PRIMARY, then fire 9 xce_test client processes (each: connect + one jobsub) SIMULTANEOUSLY (backgrounded, not sequential) immediately after election. Reproduced in 3 of 4 rounds (1-3 of 9 concurrent clients failing per round, closely matching the original ~2/9 ratio) -- 0/45 reproductions when the SAME 9 attempts were fired sequentially instead (one xce_test process at a time). Concurrency at connect time, not raw timing after election, is the actual trigger.\n\nSYMPTOM, precisely: the failing client's xcecli_jobsubmit() (client.c, wrapped in XCECLI_FAILOVER_CALL) returns rc>=0 (success-shaped, NOT a negative error) after exactly ~10.0-10.1s, with the output jobid=0 (never written) instead of the submitted JobId -- test.c's cmd_jobsub (bug-786's own guard) correctly catches this as 'no clean ack' rather than misreporting success, so no data is actually lost/corrupted -- but the client hangs ~10s before getting a clear answer, matching the bug's own description exactly.\n\nCONFIRMED CLIENT-SIDE, not server-side: cross-referenced the controller's own log (fresh --logfile per round) against the 9 concurrent attempts -- server-side WAL only ever shows 8 job-submit entries committed (lsn sequence has no gaps, no admission-cap rejection WARN, no error of any kind) when 9 concurrent clients attempted -- the failing 9th request never reaches handle_jobsubmit at all. Ruled out the obvious admission-cap suspect: commit.c's COMMIT_MODE_WAIT admission control (cap=n_vcpus-VTASK_LIVENESS_RESERVED=3 on this 4-vcpu deploy) never logged its 'WAIT rejected, admission cap exhausted' WARN in any reproduction -- 8 concurrent WAITs completing well under the cap is plausible given each takes <100ms server-side once received.\n\nTHE ~10s NUMBER, precisely sourced: xceapi.c's xce_jobsubmit() (the innermost call XCECLI_FAILOVER_CALL wraps) has a HARDCODED cbuf_deadline_10s() wait on cbuf_timedget waiting for a reply to land in the single-slot ctx->wbuf. Read cbuf_timedget itself (mylock.c) -- confirmed CORRECT, it returns -ETIMEDOUT (negative) on a genuine timeout, which xce_jobsubmit correctly propagates as a negative rc. So a bare cbuf_timedget timeout cannot by itself explain the observed rc>=0-with-jobid=0 shape; something in the retry/rotate/recover layer ABOVE it must be involved.\n\nNARROWED SUSPECT (not confirmed): client_internal.h's own documented CI1 hazard -- 'handle_disconnect may have injected a stale status into the single-slot ctx->wbuf ... xcecli_ctx_recover drains any pending entry with a non-blocking cbuf_tryget so the next attempt's cbuf_get cannot consume a prior-attempt reply' -- but this drain only runs when xce_failover_rotate() succeeds (client_failover.c:130-153, mirrored by xcecli_retry_next per its own doc comment: 'only recover when the rotate succeeded'). On this single-candidate cluster, rotate reconnects to the SAME (only) node via a real xce_ctx_reconnect() -- plausible this usually succeeds (ruling out the simplest 'rotate never succeeds on single-node, drain never runs' theory) but NOT independently verified under the exact concurrent-connect contention this repro needs; a transient reconnect failure specifically during the connection-establishment storm (9 simultaneous TLS handshakes against a listener seconds old) could leave a stale wbuf entry undrained, which a LATER retry's cbuf_timedget could then either (a) consume immediately (fast, not 10s -- doesn't match) or (b) never resolve at all if the drain path interacts badly with an in-flight send, leaving the single wbuf slot permanently empty until the full 10s deadline. Did not instrument (fprintf/gdb) the actual attempt count or wbuf state during a live failing run to confirm this mechanism specifically -- xce_test's own -vvv flag produced no additional client-side retry/rotate trace output (goes through a different, unwired logger in this CLI context), and adding real instrumentation would need a fresh lib rebuild + re-repro cycle not completed in this pass.\n\nNOT YET DONE: instrument xcecli_retry_next/xce_jobsubmit directly (attempt count, wbuf drain outcome, rotate rc, per-attempt elapsed) and re-run the now-reliable concurrent-burst repro to catch the exact mechanism in the act -- this is the recommended next step, using the repro recipe above (no gdb needed; the reproduction is now reliable enough for print-based debugging). Confirmed unrelated to raft-apply-decoupling task 6.1's apply-pump fix (reproduces identically on that spec's pre-fix and post-fix binaries per the original entry) and to this session's bug-1428/1426/1474/1472-remainder fixes (none touch xceapi.c/client.c/client_failover.c's client-side retry path).\n\n[Original entry]\nNOT FIXED. Worked around via retry for task 7.4's benchmarking purposes only. Needs a dedicated root-cause investigation with the same live-repro rigor bug-1416 got (gdb-attach via GDB_WRAP seam, or targeted debug logging around the commit-wait barrier right after a fresh single-node election).", "tags": [ "ha", "commit-engine", "election", "intermittent", "unconfirmed", - "raft-apply-decoupling-followup" + "raft-apply-decoupling-followup", + "live-repro-found", + "concurrent-connect-trigger" ], "related_bugs": [ "bug-1416" @@ -12870,14 +12808,15 @@ "error_message": "trim_wal_segments's retention floor has no awareness of the local apply pump's own lag — a reachable scenario since apply-lag-behind-commit is raft-apply-decoupling's entire premise.", "file": "src/controller/persist.c (trim_wal_segments, apply_pump_read_entry cache)", "root_cause": "Identified during raft-apply-decoupling task 7.4's apply-pump segment-read cache fix (round-2 review finding): the new fast-path cache can (POSIX unlink-keeps-fd-alive) silently advance last_applied_lsn past already-deleted data if the pump is lagging when trim_wal_segments fires. Currently bounded by the pre-existing APPLY_DIVERGENT boot fence (deferred loud fencing on next restart, not silent corruption) — a real behavior change from the pre-cache code, not literally risk-free, but not an active defect either. Same family as an already-tracked standby-cursor variant (per task 7.4's own notes).", - "fix": "NOT FIXED — deliberately deferred. task 7.4's own scope explicitly excluded redesigning trim_wal_segments's retention-floor awareness of apply-pump lag; documented via an accurate code comment instead. A future fix would give trim_wal_segments a floor input derived from the local apply cursor, not just standby-acked LSNs.", + "fix": "FIXED (TDD RED->GREEN). trim_wal_segments() now caps its trim floor at the local apply pump's own progress (last_applied_lsn), not just remote standby-acked positions: floor = min(repl_trim_lsn's standby/disconnect-guard floor, last_applied_lsn+1). The +1 matters -- repl_trim_lsn's floor treats equality as 'still needed, preserve' (a standby acked through LSN N might still need that segment for retry/resync), but the LOCAL apply cursor's equality case means the opposite: last_applied_lsn==N means this process is DONE with entry N and will never re-read it, so a segment starting exactly at N is safe to delete. Feeding last_applied_lsn+1 into the same shared 'seg_start_lsn < floor' comparison gets correct delete-on-equal semantics for this term without a second, differently-shaped check. This closes the real gap: apply_pump_read_entry()'s fast-path cache holds an open fd into whatever segment it's currently reading; POSIX unlink-keeps-fd-alive meant a lagging pump could keep reading via that stale fd even after trim_wal_segments deleted the segment out from under it (no immediate corruption, since the fd's data was still valid) -- but a FRESH read of the same range (a restart's persist_recover() replay, or the pump's own cache falling back to its slow path after an unrelated miss) would find the segment already gone, producing a genuine APPLY_DIVERGENT-class gap between what the live process quietly caught up to and what a fresh read of disk state can see. Regression test (test_e2e_checkpoint.c's test_trim_guard_preserves_unapplied_local_segments): no standbys registered (so the pre-fix floor alone would allow trimming everything below the active segment) but last_applied_lsn deliberately lags behind one segment -- proves the already-applied segment IS trimmed (no unnecessary retention) while the not-yet-applied one is preserved (the actual guard). Also had to advance last_applied_lsn in the 3 pre-existing trim tests in the same file (test_checkpoint_wal_trim_after_checkpoint, test_trim_guard_preserves_standby_segments, test_trim_without_standbys_proceeds_normally), which call persist_wal_async directly (bypassing the whole apply path) and so never advance last_applied_lsn on their own -- without the added stamp, this fix would have correctly-but-unexpectedly blocked ALL trimming in those fixtures, since last_applied_lsn defaults to 0. Full controller_unit_test suite: 1099 tests, 2 pre-existing unrelated failures (bug-1561), no regressions.\n\n[Original entry]\nNOT FIXED — deliberately deferred. task 7.4's own scope explicitly excluded redesigning trim_wal_segments's retention-floor awareness of apply-pump lag; documented via an accurate code comment instead. A future fix would give trim_wal_segments a floor input derived from the local apply cursor, not just standby-acked LSNs.", "tags": [ "ha", "persistence", "wal-trim", "apply-pump", "low-priority", - "raft-apply-decoupling-followup" + "raft-apply-decoupling-followup", + "fixed" ], "related_bugs": [], "occurrences": 1, @@ -12889,14 +12828,15 @@ "error_message": "APPLY_DIVERGENT's entity-count correctness check (persist_verify_apply_correctness) covers job entities only — worker entities can diverge (worker_from_capnp()/restore_worker() genuinely creates a new worker on a WAL_WORKER_CTRL/HOOK_DETAIL_WORKER_JOIN entry when absent) but this is currently invisible to the check.", "file": "src/controller/persist.c (persist_verify_apply_correctness, count_jobmgr_entries)", "root_cause": "Disclosed, accepted scope gap from raft-apply-decoupling task 5.1's own design: the entity-count check was scoped to jobs only (queues correctly excluded — config-derived, WAL_QUEUE_CTRL never creates a new one, verified by two review rounds). Workers were never brought in scope. Not a regression — a known, deliberate boundary, just not yet extended.", - "fix": "NOT FIXED. Extend persist_verify_apply_correctness() with a worker-count check mirroring count_jobmgr_entries() if/when a worker-creation divergence needs to be closed (task 5.1's own forward note).", + "fix": "FIXED (TDD RED->GREEN). Extended persist_verify_apply_correctness() with a worker-count check mirroring the existing job-count check exactly: added count_workermgr_entries() (persist.c, mirrors count_jobmgr_entries()), a new expected_new_worker_count field on apply_expectation_t (persist.h), and a new workermgr_count_before_replay parameter threaded through persist_verify_apply_correctness()/persist_recover(). The harder part: apply_committed_entry()'s WAL_WORKER_CTRL case previously always returned 0 regardless of outcome (JOIN-creates-new, UPDATE-applies-to-existing, and idempotent-noop were all indistinguishable to callers), so replay_wal() had no signal to tally new worker joins from. Fixed by making worker_from_capnp() (worker.c) return 1 specifically for a genuine new-worker creation (discarding restore_worker()'s raw idx, which can legitimately be 0 and would have collided with the update-applied case) and propagating that rc through apply_committed_entry() instead of the hardcoded 0 -- verified no caller of either function depended on the exact prior return value (only sign and -EEXIST were ever checked). replay_wal() tallies apply_rc==1 for WAL_WORKER_CTRL exactly like it already does for WAL_JOB_SUBMIT. New regression tests: test_verify_apply_correctness_detects_worker_count_mismatch (test_apply_correctness_verification.c, mirrors the existing job-count-mismatch test) proves a worker-count divergence is now caught as APPLY_DIVERGENT -- RED before this fix (no worker check existed at all), GREEN after. Also updated test_apply_committed_entry.c's test_ace_worker_ctrl_join_restores_and_notifies, which asserted rc==0 for a genuine JOIN creation under the old (now-corrected) contract -- now asserts rc==1 with an explanatory message. Full controller_unit_test suite: 1097 tests, 2 pre-existing unrelated failures (bug-1561), no regressions.\n\n[Original entry]\nNOT FIXED. Extend persist_verify_apply_correctness() with a worker-count check mirroring count_jobmgr_entries() if/when a worker-creation divergence needs to be closed (task 5.1's own forward note).", "tags": [ "ha", "persistence", "apply-divergent", "worker", "low-priority", - "raft-apply-decoupling-followup" + "raft-apply-decoupling-followup", + "fixed" ], "related_bugs": [], "occurrences": 1, @@ -12908,14 +12848,15 @@ "error_message": "Narrow TOCTOU residual in persist_register_pending_apply() (task 6.2, part of bug-1415's fix) — not reliably exercised by any current test.", "file": "src/controller/persist.c (persist_register_pending_apply)", "root_cause": "Flagged during raft-apply-decoupling task 6.2 as a non-blocking fast-follow while closing bug-1415. Specific race window not fully documented in tasks.md's own notes; needs a fresh look at the function to characterize exactly what window remains and whether it's reachable in production timing.", - "fix": "NOT FIXED — explicitly deferred as a fast-follow, not a blocker for task 6.2 or the spec's closure. Needs: (1) characterize the exact TOCTOU window, (2) determine production reachability, (3) add a regression test that exercises it before fixing.", + "fix": "FIXED (characterized then TDD RED->GREEN, per the entry's own 3-step plan). Characterized the exact window: persist_register_pending_apply()'s self-heal check (an UNLOCKED read of last_applied_lsn) ran BEFORE acquiring pending_applies_lock, separate from the locked insert that followed. Window: this thread reads last_applied_lsn (sees 'not yet applied') -> BEFORE it locks and inserts, the apply pump's thread advances last_applied_lsn past this lsn AND completes its own persist_take_pending_apply(lsn) call (finding nothing, since this thread hasn't inserted yet) -> this thread then inserts a node the pump will never revisit -- permanently orphaned, identical failure mode to bug-1415 (a hang, on_applied never fires) via a different, narrower trigger window. Production reachability: yes -- persist_register_pending_apply() is called from a different thread/vtask than the apply pump's walk (commit.c's continuation dispatch vs persist_apply_pump_run, per the function's own pre-existing doc comment), with no synchronization between them before this fix beyond the already-locked insert. Fix: moved the self-heal check inside pending_applies_lock, so the whole check-then-(insert-or-self-heal) sequence is one atomic critical section against persist_take_pending_apply's own locked scan-and-remove -- either this thread's check now runs before the pump's take (sees 'not yet applied', inserts, and the pump's later take, now forced to wait on the same lock, finds the freshly-inserted node) or after (sees 'already applied', self-heals instead of inserting, and the pump's take correctly finds nothing) -- never the interleaved gap that orphaned it. Regression test (test_apply_pump.c's test_register_pending_apply_race_against_concurrent_apply_self_heals): deterministic reproduction via controlled state (a main thread holding pending_applies_lock forces the register call to observe last_applied_lsn only after the lock's own update, removing dependence on real-time race timing) -- RED under the pre-fix code (on_applied never fires, node orphaned on pending_applies), GREEN under the fix (self-heals correctly, no orphan). Full controller_unit_test suite: 1098 tests, 2 pre-existing unrelated failures (bug-1561), no regressions.\n\n[Original entry]\nNOT FIXED — explicitly deferred as a fast-follow, not a blocker for task 6.2 or the spec's closure. Needs: (1) characterize the exact TOCTOU window, (2) determine production reachability, (3) add a regression test that exercises it before fixing.", "tags": [ "ha", "persistence", "apply-pump", "toctou", "unconfirmed", - "raft-apply-decoupling-followup" + "raft-apply-decoupling-followup", + "fixed" ], "related_bugs": [ "bug-1415" @@ -13629,7 +13570,7 @@ "error_message": "xce_controller with no --peers flag exits with -EINVAL from cluster_open_listener instead of starting as single-node primary", "file": "src/controller/myapp.c", "root_cause": "Discovered while independently reviewing clusterconfig-schema-split task 5.3 by building and running the actual pre-spec baseline (commit 94f50ce) for comparison. Single-node startup (no --peers) was ALREADY broken before this spec began: myapp_run() calls cluster_open_listener() before the primary config (cluster_config) is loaded from disk, so the pre-spec fallback branch (cluster_config->port, only reachable when cluster_config != NULL) always hit its else branch and returned -EINVAL, unconditionally, for every zero-flag single-node start. This invalidates part of clusterconfig-schema-split's premise (requirements.md's Introduction claimed cluster_config->port was 'the sole live port-sourcing mechanism for single-node deployments' and 'single-node controller startup SHALL continue to work exactly as it does today' — in reality it did not work today). Not caused by, or a regression from, clusterconfig-schema-split's changes — task 2.1's port-resolution rewrite (commit 1229f9a) incidentally fixed this as a side effect, since its hardcoded-default fallback no longer depends on cluster_config being loaded.", - "fix": "Not fixed as its own change — already incidentally repaired by clusterconfig-schema-split task 2.1 (cluster_open_listener now falls back to a hardcoded XCE_CONTROLLER_DEFAULT_PORT constant instead of an unloaded cluster_config->port). Live-verified post-fix: single-node startup with no --peers now correctly binds port 9990 and reaches CTRL_PRIMARY. No further action needed unless a future audit wants to correct the historical claim in clusterconfig-schema-split's requirements.md/design.md that pre-spec single-node startup was previously working — it was not, for an unrelated reason (myapp_run's listener-open/config-load ordering).", + "fix": "RESOLVED -- incidentally repaired by clusterconfig-schema-split task 2.1 (cluster_open_listener now falls back to a hardcoded XCE_CONTROLLER_DEFAULT_PORT constant instead of an unloaded cluster_config->port), not as a dedicated fix for this entry. Live-verified post-fix: single-node startup with no --peers now correctly binds port 9990 and reaches CTRL_PRIMARY. No further action needed unless a future audit wants to correct the historical claim in clusterconfig-schema-split's requirements.md/design.md that pre-spec single-node startup was previously working — it was not, for an unrelated reason (myapp_run's listener-open/config-load ordering).", "tags": [ "controller", "startup", @@ -13948,7 +13889,7 @@ "error_message": "config-upgrade-snapshot task 8.6 live --config-upgrade soak: after avoiding bug-1469's queue-creation gap (switched the upgrade-target config change from ADDING a queue to updating an existing queue's description text -- see bug-1469's updated entry), the materialized-state identity check (snapdiff of each member's latest checkpoint file, all at the same LSN) still fails, but now with a MUCH smaller divergence: exactly ONE byte differs (offset 42879/51161 bytes total) between ctrl-1 and ctrl-2/ctrl-3 (ctrl-2 and ctrl-3 are byte-IDENTICAL to each other). ctrl-1 has 0x50 at that offset; ctrl-2/ctrl-3 have 0x00.", "file": "src/controller/persist.c collect_host_frames / capnp_encode_host_admin_state (~lines 141-162, 2028)", "root_cause": "ROOT-CAUSED AND DISTINGUISHED (2026-07-25, follow-up investigation): the two prior hypotheses are now resolved -- Hypothesis 2 (uninitialized/non-zeroed capnp padding) is REFUTED, Hypothesis 1 (genuine content discrepancy) is CONFIRMED, and the true mechanism is a pre-existing, out-of-spec-boundary bug, NOT part of config-upgrade-snapshot. Evidence: (1) Wrote a byte-precise decoder in Python that walks the checkpoint's [host_count][{payload_len; capnp bytes}] host-frame section (same structural approach as snapverify.c task 8.4) and decodes each frame using the exact wire shape write_HostAdminState produces (HostAdminState_word_count=1, pointer_count=1: one 8-byte data word holding `status` via capn_write32 at data-section offset 0, one 8-byte pointer word for `name` Text). Decoding BOTH /tmp/xce-ha-configupgrade and /tmp/xce-ha-configupgrade-v2's preserved checkpoints for all 3 members shows: host[1] = 'cfgupgrade-host', ctrl-1 status=0x00500000, ctrl-2/ctrl-3 status=0x00000000 -- byte offset 42879 is byte index 2 of the little-endian `status` DATA FIELD itself (data-section file offset 42877-42880), NOT the 4-byte padding word that follows it (file offset 42881-42884), which is 0x00000000 IDENTICALLY across all 3 members in both reproductions. (2) 0x00500000 = XCE_WORKER_CLOSED (0x00100000) | XCE_WORKER_CLOSED_BY_ADMIN (0x00400000) -- exactly the bits worker.c:779 sets for HOST_CLOSE inside worker_ctrl(). This is semantically meaningful admin-closed status, not heap garbage. (3) Independently confirmed capn_init_malloc's segment allocator (create_local in the pinned c-capnproto build, /nix/store/.../capnpc-1.0.1/lib/libCapnC_Runtime.so, disassembled directly) calls calloc(1, size) -- zero-initializing -- so Hypothesis 2 was never physically possible for this library, consistent with the empirically-always-zero padding bytes. (4) Traced the actual mechanism: worker_ctrl() (worker.c:732-794) sets w->host.status on whichever member directly processes the hostctrl RPC (ctrl-1, primary at the time), then fires HOOK_DETAIL_WORKER_UPDATE, which persist.c:5722 maps to WAL_WORKER_CTRL and replicates. On EVERY member applying that WAL entry (standbys AND a member's own replay-from-WAL/restart path alike), persist.c:3724's case WAL_WORKER_CTRL calls worker_from_capnp() (worker.c:827-856) UNCONDITIONALLY, regardless of hook_detail (JOIN vs UPDATE vs DEAD all share this one apply path). worker_from_capnp's own idempotency guard (worker.c:847-853) does key_find() and returns -EEXIST if a worker with that name ALREADY EXISTS -- WITHOUT applying any of the just-decoded fields, including status. persist.c:3732-3733 then treats -EEXIST as 'idempotent re-apply, no hook' and silently returns 0, discarding the decoded status update entirely. Since cfgupgrade-host was JOINED (a prior, separate WAL_WORKER_CTRL/HOOK_DETAIL_WORKER_JOIN entry) BEFORE it was admin-closed, by the time the UPDATE entry replicates/replays, the worker already exists on every standby -- so the close never lands there. This is NOT specific to config upgrades, checkpoints, or any config-upgrade-snapshot code: it reproduces for ANY `hostctrl -c`/`hostctrl -o` issued against an already-joined host on ANY XCE HA cluster, with zero dependency on a config upgrade ever being submitted. `git log -L` on worker_from_capnp confirms it was introduced 2026-05-08 in commit 9b6a91a8 ('feat: implement restore_job/worker_from_capnp for WAL recovery'), ~2.5 months before config-upgrade-snapshot's spec work began (first spec commit visible ~2026-06-24); no config-upgrade-snapshot task touches this apply path (the spec's own worker.c addition, restore_host_state from commit 5b1600a8, is a wholly separate function used only by the SNAPSHOT-INSTALL restore path, task 4, never by the live WAL_WORKER_CTRL apply switch). collect_host_frames/capnp_encode_host_admin_state (this spec's task 2.1 code) are functioning exactly as designed -- they faithfully serialize whatever w->host.status currently holds; the bug is that w->host.status itself is wrong on ctrl-2/ctrl-3, entirely upstream of any checkpoint/materialization logic. Existing test coverage (test_apply_committed_entry.c:test_ace_worker_ctrl_join_restores_and_notifies) only exercises the fresh-JOIN case; no test exercises WAL_WORKER_CTRL UPDATE against an already-existing worker, confirming this is a genuine untested gap, not a previously-handled case. Filed separately as bug-1472 (the true root cause) since it is a distinct, real, pre-existing mechanism entirely outside config-upgrade-snapshot's boundary -- see that entry for the fix-owner routing. This entry (bug-1471) remains the observed SYMPTOM (checkpoint byte divergence) for task 8.6's soak; the underlying CAUSE is bug-1472.", - "fix": "NOT FIXED, AND DELIBERATELY NOT ATTEMPTED IN THIS SPEC -- confirmed genuinely OUT OF config-upgrade-snapshot's scope (see root_cause above). The real fix belongs in worker.c's worker_from_capnp()/persist.c's WAL_WORKER_CTRL apply case (case WAL_WORKER_CTRL, persist.c:3724), which needs to distinguish JOIN semantics (create-if-missing, EEXIST is a legitimate no-op) from UPDATE/DEAD semantics (an already-existing worker is the EXPECTED case, and the decoded fields -- at minimum host.status -- must still be applied to it, not discarded) -- e.g. an update-in-place path mirroring how restore_host_state (task 3, worker.c, config-upgrade-snapshot's OWN precedent) ORs admin bits into an existing worker rather than treating existing-ness as terminal. That is a genuine, real, previously undiscovered defect (filed as bug-1472) but is squarely core HA worker-control replication, predating this spec by ~2.5 months and untouched by any of its tasks -- fixing it here would be scope creep into a different subsystem's correctness contract, not a config-upgrade-snapshot materialization fix. Task 8.6's soak will keep observing this exact symptom (host-frame byte divergence whenever its scenario admin-closes a host before the final checkpoint) until bug-1472 is fixed by whoever owns worker.c/persist.c's general worker-control replication path. Recommend task 8.6 either (a) accept this as a known, separately-tracked limitation and adjust the soak's own PASS criteria to tolerate it (e.g. skip the admin-close-a-host step, or snapdiff-exclude the host-admin-status byte range) until bug-1472 lands, or (b) block final PASS on bug-1472's fix -- an orchestrator/owner decision, not this investigation's to make.", + "fix": "FIXED -- this was the observed SYMPTOM (checkpoint host-frame byte divergence) of bug-1472's root cause (worker_from_capnp()'s WAL_WORKER_CTRL apply path silently discarding UPDATE payloads against an already-existing worker). bug-1472 was fixed 2026-07-25 (TDD RED->GREEN): worker_from_capnp() now decodes hook_detail before applying its existing-worker idempotency guard, and HOOK_DETAIL_WORKER_UPDATE replaces the existing worker's XCE_WORKER_ADMIN_MASK bits with the decoded payload instead of silently no-op'ing on -EEXIST. This directly closes bug-1471's exact repro (an admin-close status change failing to replicate to standbys, producing the byte-42879 checkpoint divergence). Note: HOOK_DETAIL_WORKER_DEAD was deliberately left unfixed in the same change -- tracked separately as bug-1472-remainder (needs a design decision, not yet resolved).\n\n[Original entry]\nNOT FIXED, AND DELIBERATELY NOT ATTEMPTED IN THIS SPEC -- confirmed genuinely OUT OF config-upgrade-snapshot's scope (see root_cause above). The real fix belongs in worker.c's worker_from_capnp()/persist.c's WAL_WORKER_CTRL apply case (case WAL_WORKER_CTRL, persist.c:3724), which needs to distinguish JOIN semantics (create-if-missing, EEXIST is a legitimate no-op) from UPDATE/DEAD semantics (an already-existing worker is the EXPECTED case, and the decoded fields -- at minimum host.status -- must still be applied to it, not discarded) -- e.g. an update-in-place path mirroring how restore_host_state (task 3, worker.c, config-upgrade-snapshot's OWN precedent) ORs admin bits into an existing worker rather than treating existing-ness as terminal. That is a genuine, real, previously undiscovered defect (filed as bug-1472) but is squarely core HA worker-control replication, predating this spec by ~2.5 months and untouched by any of its tasks -- fixing it here would be scope creep into a different subsystem's correctness contract, not a config-upgrade-snapshot materialization fix. Task 8.6's soak will keep observing this exact symptom (host-frame byte divergence whenever its scenario admin-closes a host before the final checkpoint) until bug-1472 is fixed by whoever owns worker.c/persist.c's general worker-control replication path. Recommend task 8.6 either (a) accept this as a known, separately-tracked limitation and adjust the soak's own PASS criteria to tolerate it (e.g. skip the admin-close-a-host step, or snapdiff-exclude the host-admin-status byte range) until bug-1472 lands, or (b) block final PASS on bug-1472's fix -- an orchestrator/owner decision, not this investigation's to make.", "tags": [ "capnp", "capnp_encode_host_admin_state", @@ -13965,7 +13906,8 @@ "root-caused", "routed-to-bug-1472", "single-byte-divergence", - "task-8.6" + "task-8.6", + "fixed" ], "related_bugs": [ "bug-1469", @@ -14030,7 +13972,7 @@ "error_message": "config-upgrade-snapshot task 8.9's new live --config-upgrade-wal-trim soak (run-ha-soak.sh, run_config_upgrade_wal_trim_round): the marker-triggered checkpoint's WAL-segment trim (persist_checkpoint_marker_triggered -> trim_wal_segments -> repl_trim_lsn, bug-1147's guard) strands a genuinely mid-catch-up standby's still-needed WAL segment -- but NOT because bug-1147's guard logic itself is wrong. Reproduced identically on 2/2 independent live 3-node runs: a killed, still-down standby's disconnect-guard (last_disconnected_lsn) is silently DESTROYED before the trim ever runs, because an UNRELATED, already-connected peer's redundant reconnect attempt reuses the dead standby's just-freed replicator slot.", "file": "src/controller/ha.c (primary_heartbeat_fn's 'reconnect to any peer missing from standby_endpoints' loop, ~9219-9253; on_standby_connected, ~8658-8708; raft_cb_become_leader's vote-ep promotion, ~1888-1921); src/controller/repl.c (repl_add_standby's first-empty-slot allocation, ~85-133, and its memset(s,0,...) which wipes last_disconnected_lsn on reuse)", "root_cause": "Confirmed via direct log evidence on 2 independent live-soak runs (deploy dirs /home/claude/tmp/xce-ha-waltrim and /home/claude/tmp/xce-ha-waltrim2, both --config-upgrade-wal-trim). Sequence, identical both times: (1) An election promotes surviving VOTE endpoints to standby replication channels (raft_cb_become_leader: 'promoted vote ep[i] -> standby channel', repl_add_standby registers them at replicator slots 0/1 -- e.g. run 1: ep[0]=ctrl-1 -> slot 0, ep[1]=ctrl-3 -> slot 1). (2) The test kills a non-primary TARGET standby (ctrl-1) while it is caught up to a known WAL floor. The primary's err_handler (vote_ep_err_handler, correctly wired) detects the RDHUP and calls repl_remove_standby, freeing slot 0 and setting last_disconnected_lsn = ctrl-1's real acked_lsn (confirmed matching the test's own captured floor segment, e.g. seg-5 in run 1, seg-c in run 2) -- bug-1147's guard is set CORRECTLY at this point. (3) Seconds later (matching the ~9s default heartbeat interval), primary_heartbeat_fn's periodic 'reconnect to any peer missing from standby_endpoints' sweep (ha.c ~9219) runs. It correctly identifies ctrl-1 (peer[0]) as missing (reconnect attempt fails with ECONNREFUSED, confirmed in both runs' logs: 'HA: connect to peer[0] (localhost:9991) failed: -111') -- but it ALSO, incorrectly, identifies ctrl-3 (peer[2]) as missing, even though ctrl-3 has been continuously connected the whole time via its slot-1 promoted-vote-channel connection (confirmed acking WAL entries without any gap throughout, via ctrl-2.log's 'commit: ack standby_idx=1' sequence). This triggers a redundant outbound reconnect to ctrl-3 (on_standby_connected fires: 'HA: outbound standby connection to peer[2] established' + 'HA: repl registered late-joiner standby peer[2] (slot 0, localhost:9993)'). (4) repl_add_standby's slot allocation picks the FIRST EMPTY slot by lowest index (repl.c ~85-110) with no regard for whether that slot still holds an unexpired bug-1147 disconnect guard for a DIFFERENT peer -- it picks slot 0 (ctrl-1's just-freed, guard-holding slot, since it is lower-indexed than slot 1) and unconditionally memsets it to zero, WIPING OUT ctrl-1's last_disconnected_lsn guard entirely. (5) When the marker-triggered checkpoint's trim runs shortly after, repl_trim_lsn (repl.c ~650-683) sees slot 0 now occupied by a live, fully-caught-up ctrl-3 duplicate connection (contributing no meaningful floor-lowering) and slot 1 also fully caught up -- with NO surviving trace that ctrl-1 (still down, still needing its old floor) was ever disconnected. floor collapses to the active segment itself, and every segment below it -- including the one ctrl-1 genuinely still needs -- is deleted. This is NOT a bug in bug-1147's guard logic (repl_trim_lsn/trim_wal_segments are working exactly as designed against the replicator state they are handed); it is an upstream defect in how a duplicate/late-joiner reconnect for an ALREADY-connected peer gets allocated a replicator slot. Root cause of WHY the 'peer already has a standby endpoint' check (ha.c ~9225-9249) fails to recognize ctrl-3's existing promoted-vote-channel connection is not fully pinned to a single line -- standby_ep_ctx_t is deliberately layout-compatible with vote_ep_ctx_t (confirmed by direct struct read, ha.c:8590-8603, both start with ha/c/ctx/peer_idx in the same order, so the blind cast the check performs is NOT a type-confusion bug) -- but vote_ep_ctx_t.peer_idx appears to be populated under inconsistent conventions across its several creation call sites in ha.c (e.g. line 542 sets peer_idx=votes_idx [relative, votes-array index] while a comment at line ~6120 asserts 'vctx->peer_idx is the ABSOLUTE cluster index' for a different call site) -- a mismatch between the relative votes_idx_for_i the check computes and whichever convention the SPECIFIC promoted vote_ep_ctx_t for ectx->votes[i].ep was actually built under would make the sctx->peer_idx == votes_idx_for_i comparison fail deterministically for promoted-vote-channel entries, matching the 100% reproduction rate observed. This needs further tracing by whoever owns ha.c's HA connection-management code to identify the exact mismatched call site.", - "fix": "NOT FIXED -- explicitly out of config-upgrade-snapshot task 8.9's boundary (task 8.9 is a regression test for the pre-existing, unmodified bug-1147 guard under the NEW marker-triggered trigger path, not a fix for this separately-discovered HA connection-management defect). Two independent, complementary fix angles for whoever owns ha.c/repl.c's HA connection layer: (a) fix the 'peer already has a standby endpoint' check (ha.c ~9225-9249) so it correctly recognizes an existing promoted-vote-channel connection for a peer, eliminating the spurious duplicate/late-joiner reconnect entirely (the more targeted fix, addresses the root cause) -- likely requires auditing every vote_ep_ctx_t.peer_idx assignment site (ha.c lines 542, 697, 6956, and whichever one specifically constructs ectx->votes[i].ep's ep->data) against the relative-votes_idx convention votes_idx_for_i expects; (b) independently harden repl_add_standby's slot allocation (repl.c) to never silently clobber a slot whose last_disconnected_lsn guard has not yet expired per snap_interval_s -- e.g. skip a guard-holding slot in the first-empty-slot scan (falling through to a genuinely empty slot, or failing loud if none exists) -- this is a defense-in-depth hardening of bug-1147's own retention mechanism against ANY future caller that might reuse a slot prematurely, not just this specific duplicate-connection path. Recommend (b) regardless of whether (a) is fixed, since it directly protects the invariant Requirement 7.4 cares about. The run_config_upgrade_wal_trim_round test infrastructure (run-ha-soak.sh, config-upgrade-snapshot task 8.9) is CORRECT and reproducibly catches this defect via direct on-disk WAL segment inspection -- re-run `run-ha-soak.sh --config-upgrade-wal-trim` after either fix lands to confirm a clean PASS; task 8.9 itself should stay incomplete/blocked on this bug per its own scope boundary (mirrors task 8.6's bug-1471/1472 precedent).\n\n[UPDATE 2026-07-26] FIXED (fix angle (b), the repl.c defense-in-depth hardening): repl_add_standby's slot-allocation scan (repl.c) now does a two-pass search instead of first-empty-slot-by-index. Pass 1 prefers a genuinely virgin slot (ep==NULL && last_disconnected_lsn==0). Pass 2 (fallback, only when no virgin slot exists) reuses the guard-holding slot (ep==NULL && last_disconnected_lsn!=0) with the OLDEST last_disconnect_time -- a deliberate policy: that guard is closest to its own natural repl_expire_disconnect_slots expiry, so reusing it discards the LEAST remaining WAL-retention protection of any guard-holding slot. With MAX_STANDBYS==3, the exhausted case (both non-primary slots simultaneously guard-holding, no virgin slot free) is handled by this same fallback rather than -ENOSPC -- refusing a legitimate new/reconnecting peer would be worse for cluster availability than accepting an early guard loss in an already-degraded state; the pre-existing -ENOSPC path (more distinct live peers than MAX_STANDBYS slots) is untouched, reserved for that genuinely-impossible case only.\nTDD: two new unit tests added to src/controller/testsuite/test_repl_unit.c -- test_repl_add_standby_prefers_virgin_slot_over_guard (RED under the old code: asserted a new unrelated peer must not land in a lower-indexed guard-holding slot when a virgin slot exists, and that the guard's last_disconnected_lsn must survive untouched) and test_repl_add_standby_exhausted_guards_prefers_oldest_disconnect (RED under the old code: asserted the exhausted-fallback deliberately picks the OLDER-disconnected guard slot, not merely the lowest-indexed one -- the test deliberately makes 'oldest-disconnected' and 'lowest-indexed' point at DIFFERENT slots so an accidental first-guard-by-index fallthrough would still fail it). Both new tests failed pre-fix, pass post-fix.\nGREEN: full controller_unit_test suite = 1030 tests (1028 known-good baseline + 2 new), 0 failures, no regressions (ASAN_OPTIONS=detect_leaks=0, nix develop build via `ninja -C src/xce.build controller/controller_unit_test`).\nrepl_remove_standby, repl_expire_disconnect_slots, repl_trim_lsn, and trim_wal_segments were NOT modified -- all confirmed correct and untouched, per this fix's scope.\nThis fix does NOT address fix angle (a) -- the separate ha.c heartbeat-reconnect-sweep defect (vote_ep_ctx_t.peer_idx convention mismatch hypothesis, ha.c ~9219-9254 / ~8687-8708) that triggers the spurious duplicate reconnect in the first place -- that remains a separate, not-fully-root-caused issue, out of this fix's scope. Recommend re-running `run-ha-soak.sh --config-upgrade-wal-trim` (config-upgrade-snapshot task 8.9) to confirm this defense-in-depth closes the observed WAL-segment-loss symptom even without (a) landing.\n\n[UPDATE 2026-07-26, post-review remediation] Independent review REJECTED the initial fix on one finding: the Pass 2 exhausted-fallback branch silently destroyed the reused slot's last_disconnected_lsn/last_disconnect_time via memset with zero logging, undercutting the fix's own stated goal of making the previously-silent WAL-retention-guard destruction visible/bounded -- this file already has an established convention for exactly this class of event (repl_broadcast's WARN(\"repl_broadcast: no standbys registered...\") and WARN(\"repl_broadcast: standby %s:%u evicted...\")). Remediated: repl_add_standby now tracks whether Pass 1 found a virgin slot (used_fallback_slot) and, only on the genuine Pass 2 fallback path, logs WARN(\"repl_add_standby: reusing guard-holding slot %d (host=%s:%u, last_disconnected_lsn=%lu) -- no virgin slot available, WAL-retention protection discarded\", ...) immediately before the memset, capturing the discarded slot's host/port/last_disconnected_lsn while they're still readable. No new test required (a log line has no observable return-value/state effect for a test to assert on) -- verified instead by confirming the WARN string is present in the built binary (`strings controller_unit_test | grep 'reusing guard-holding slot'`) and that both existing bug-1474 tests (test_repl_add_standby_prefers_virgin_slot_over_guard, test_repl_add_standby_exhausted_guards_prefers_oldest_disconnect -- the latter specifically exercises the Pass 2 path this WARN sits on) still pass unchanged. GREEN: full controller_unit_test suite = 1033 tests (1030 baseline + bug-1472's 3 concurrently-landed tests), 0 failures. NOTE: a transient stale-object-file build artifact was observed once in this shared build directory during verification (a link picked up an out-of-date worker.c.o from a moment when a concurrent sibling agent's build was in flight, showing 2 spurious failures unrelated to this change) -- resolved by touching the affected sources to force recompilation; not a defect in this fix, noted here in case it recurs for a future concurrent multi-agent session in this worktree.", + "fix": "[UPDATE 2026-08-16] Angle-a FIXED (TDD RED->GREEN, root-caused precisely -- not the original peer_idx-convention hypothesis, a different, concrete bug in an adjacent function). Traced every vote_ep_ctx_t.peer_idx assignment site as recommended: raft_cb_send_vote_request (real vote, ha.c) correctly sets peer_idx=votes_idx (RELATIVE) and is the ONLY function that stores its endpoint into ectx->votes[] (the array raft_cb_become_leader's promotion loop reads) -- confirmed by full read, no ectx->votes[].ep assignment exists anywhere else. raft_cb_send_pre_vote_request deliberately sets peer_idx=ABSOLUTE (correct, documented, needed by the PreVote tally) but NEVER stores into ectx->votes[] either -- so PreVote endpoints structurally can never reach promotion, ruling out the original hypothesis (PreVote-vs-real-vote convention mismatch reaching the reconnect-sweep check). ha->my_priority is set once at init and never reassigned, ruling out a self-idx-drift hypothesis too. The reconnect-sweep check itself (primary_heartbeat_fn, ~line 9787) already correctly discriminates promoted-vote-ep (relative) vs standby-ep (absolute) via sctx->ctx != NULL, unchanged since 2026-06-01 (predates this bug's 2026-07-25 discovery, confirmed via git log -L) -- it was never the bug. The REAL bug: on_standby_connected's own same-peer duplicate-eviction check (the function's OWN doc comment already named this exact hazard -- 'two live outbound state-push channels to one peer') compared cur_sctx->peer_idx == sctx->peer_idx DIRECTLY, with no relative/absolute discrimination at all -- unlike the adjacent, correct reconnect-sweep check ~500 lines away. sctx (the newly-connecting standby) always carries the ABSOLUTE index; if the existing entry for the same physical peer is a promoted vote ep (RELATIVE), the raw comparison can never match, so the stale promoted-vote channel is never evicted -- both channels survive, reproducing the buglog's exact live evidence (heartbeats/WAL fan out to a stale duplicate; the genuinely-connected promoted channel's own bug-1147 disconnect-guard slot gets silently reused by the SECOND, spurious connect attempt for the SAME live peer). Fix: apply the identical votes_idx_for_i conversion the reconnect-sweep check already uses, in on_standby_connected's eviction loop, discriminating on cur_sctx->ctx != NULL exactly as that check does. Regression test (test_ha_liveness.c's test_reconnect_resolves_dual_channel_promoted_vote_ep): stale entry is a genuine promoted-vote-ep test double (vote_ep_ctx_build_for_test, relative peer_idx) for the same physical peer a fresh standby reconnect targets (absolute peer_idx) -- RED under the pre-fix code (both channels survive, zero evictions -- the existing sibling test, test_reconnect_resolves_dual_channel_same_peer, only ever exercised the standby-vs-standby case, both absolute, which is why this mixed case slipped through originally), GREEN under the fix (exactly one channel survives, one eviction). Full controller_unit_test suite: 1100 tests, 2 pre-existing unrelated failures (bug-1561), no regressions. NOT independently re-verified via a live multi-node chaotic soak (the class of test that originally discovered this bug) -- the unit test directly constructs and proves the exact mechanism the live soak's log evidence showed, but a live re-run of config-upgrade-snapshot task 8.9's --config-upgrade-wal-trim scenario would be the strongest additional confirmation if this is revisited.\n\n[Original entry -- angle-b, already fixed 2026-07-26]\n\nNOT FIXED -- explicitly out of config-upgrade-snapshot task 8.9's boundary (task 8.9 is a regression test for the pre-existing, unmodified bug-1147 guard under the NEW marker-triggered trigger path, not a fix for this separately-discovered HA connection-management defect). Two independent, complementary fix angles for whoever owns ha.c/repl.c's HA connection layer: (a) fix the 'peer already has a standby endpoint' check (ha.c ~9225-9249) so it correctly recognizes an existing promoted-vote-channel connection for a peer, eliminating the spurious duplicate/late-joiner reconnect entirely (the more targeted fix, addresses the root cause) -- likely requires auditing every vote_ep_ctx_t.peer_idx assignment site (ha.c lines 542, 697, 6956, and whichever one specifically constructs ectx->votes[i].ep's ep->data) against the relative-votes_idx convention votes_idx_for_i expects; (b) independently harden repl_add_standby's slot allocation (repl.c) to never silently clobber a slot whose last_disconnected_lsn guard has not yet expired per snap_interval_s -- e.g. skip a guard-holding slot in the first-empty-slot scan (falling through to a genuinely empty slot, or failing loud if none exists) -- this is a defense-in-depth hardening of bug-1147's own retention mechanism against ANY future caller that might reuse a slot prematurely, not just this specific duplicate-connection path. Recommend (b) regardless of whether (a) is fixed, since it directly protects the invariant Requirement 7.4 cares about. The run_config_upgrade_wal_trim_round test infrastructure (run-ha-soak.sh, config-upgrade-snapshot task 8.9) is CORRECT and reproducibly catches this defect via direct on-disk WAL segment inspection -- re-run `run-ha-soak.sh --config-upgrade-wal-trim` after either fix lands to confirm a clean PASS; task 8.9 itself should stay incomplete/blocked on this bug per its own scope boundary (mirrors task 8.6's bug-1471/1472 precedent).\n\n[UPDATE 2026-07-26] FIXED (fix angle (b), the repl.c defense-in-depth hardening): repl_add_standby's slot-allocation scan (repl.c) now does a two-pass search instead of first-empty-slot-by-index. Pass 1 prefers a genuinely virgin slot (ep==NULL && last_disconnected_lsn==0). Pass 2 (fallback, only when no virgin slot exists) reuses the guard-holding slot (ep==NULL && last_disconnected_lsn!=0) with the OLDEST last_disconnect_time -- a deliberate policy: that guard is closest to its own natural repl_expire_disconnect_slots expiry, so reusing it discards the LEAST remaining WAL-retention protection of any guard-holding slot. With MAX_STANDBYS==3, the exhausted case (both non-primary slots simultaneously guard-holding, no virgin slot free) is handled by this same fallback rather than -ENOSPC -- refusing a legitimate new/reconnecting peer would be worse for cluster availability than accepting an early guard loss in an already-degraded state; the pre-existing -ENOSPC path (more distinct live peers than MAX_STANDBYS slots) is untouched, reserved for that genuinely-impossible case only.\nTDD: two new unit tests added to src/controller/testsuite/test_repl_unit.c -- test_repl_add_standby_prefers_virgin_slot_over_guard (RED under the old code: asserted a new unrelated peer must not land in a lower-indexed guard-holding slot when a virgin slot exists, and that the guard's last_disconnected_lsn must survive untouched) and test_repl_add_standby_exhausted_guards_prefers_oldest_disconnect (RED under the old code: asserted the exhausted-fallback deliberately picks the OLDER-disconnected guard slot, not merely the lowest-indexed one -- the test deliberately makes 'oldest-disconnected' and 'lowest-indexed' point at DIFFERENT slots so an accidental first-guard-by-index fallthrough would still fail it). Both new tests failed pre-fix, pass post-fix.\nGREEN: full controller_unit_test suite = 1030 tests (1028 known-good baseline + 2 new), 0 failures, no regressions (ASAN_OPTIONS=detect_leaks=0, nix develop build via `ninja -C src/xce.build controller/controller_unit_test`).\nrepl_remove_standby, repl_expire_disconnect_slots, repl_trim_lsn, and trim_wal_segments were NOT modified -- all confirmed correct and untouched, per this fix's scope.\nThis fix does NOT address fix angle (a) -- the separate ha.c heartbeat-reconnect-sweep defect (vote_ep_ctx_t.peer_idx convention mismatch hypothesis, ha.c ~9219-9254 / ~8687-8708) that triggers the spurious duplicate reconnect in the first place -- that remains a separate, not-fully-root-caused issue, out of this fix's scope. Recommend re-running `run-ha-soak.sh --config-upgrade-wal-trim` (config-upgrade-snapshot task 8.9) to confirm this defense-in-depth closes the observed WAL-segment-loss symptom even without (a) landing.\n\n[UPDATE 2026-07-26, post-review remediation] Independent review REJECTED the initial fix on one finding: the Pass 2 exhausted-fallback branch silently destroyed the reused slot's last_disconnected_lsn/last_disconnect_time via memset with zero logging, undercutting the fix's own stated goal of making the previously-silent WAL-retention-guard destruction visible/bounded -- this file already has an established convention for exactly this class of event (repl_broadcast's WARN(\"repl_broadcast: no standbys registered...\") and WARN(\"repl_broadcast: standby %s:%u evicted...\")). Remediated: repl_add_standby now tracks whether Pass 1 found a virgin slot (used_fallback_slot) and, only on the genuine Pass 2 fallback path, logs WARN(\"repl_add_standby: reusing guard-holding slot %d (host=%s:%u, last_disconnected_lsn=%lu) -- no virgin slot available, WAL-retention protection discarded\", ...) immediately before the memset, capturing the discarded slot's host/port/last_disconnected_lsn while they're still readable. No new test required (a log line has no observable return-value/state effect for a test to assert on) -- verified instead by confirming the WARN string is present in the built binary (`strings controller_unit_test | grep 'reusing guard-holding slot'`) and that both existing bug-1474 tests (test_repl_add_standby_prefers_virgin_slot_over_guard, test_repl_add_standby_exhausted_guards_prefers_oldest_disconnect -- the latter specifically exercises the Pass 2 path this WARN sits on) still pass unchanged. GREEN: full controller_unit_test suite = 1033 tests (1030 baseline + bug-1472's 3 concurrently-landed tests), 0 failures. NOTE: a transient stale-object-file build artifact was observed once in this shared build directory during verification (a link picked up an out-of-date worker.c.o from a moment when a concurrent sibling agent's build was in flight, showing 2 spurious failures unrelated to this change) -- resolved by touching the affected sources to force recompilation; not a defect in this fix, noted here in case it recurs for a future concurrent multi-agent session in this worktree.", "tags": [ "ha.c", "repl.c", @@ -14055,7 +13997,8 @@ "fixed", "defense-in-depth", "log-hygiene", - "review-remediation" + "review-remediation", + "angle-a-fixed" ], "related_bugs": [ "bug-1147", @@ -14071,7 +14014,7 @@ "error_message": "Follow-up split out of bug-1472: HOOK_DETAIL_WORKER_DEAD entries applied via WAL_WORKER_CTRL against an already-existing worker still return -EEXIST (no-op) -- a dead-worker transition detected by one member (mark_worker_unavail(), worker.c:661-679) still never replicates its state to other members (standbys, or this member's own WAL replay), unconditionally. NOT a regression: this was already true before bug-1472's investigation; bug-1472's fix deliberately scoped itself to the byte-precise-confirmed HOOK_DETAIL_WORKER_UPDATE case only and left DEAD as -EEXIST/no-op, unchanged.", "file": "src/controller/worker.c worker_from_capnp (~827-895, post-bug-1472-fix); mark_worker_unavail (~661-679); src/schema/controller.capnp struct Worker (~136-144)", "root_cause": "mark_worker_unavail()'s mutation is NOT confined to XCE_WORKER_ADMIN_MASK the way worker_ctrl()'s is -- it sets XCE_WORKER_UNAVAIL (a non-admin bit), dead_since, resets hostattrs_t to 'unknown' sentinel strings, AND zeroes the live per-host resource-value slots via set_host_slots()/update_resvals(), which mutates xce_worker_t.resvals (a zarray_t of live xce_val_t entries). Critically, w->host.resvals is NOT part of the Worker capnp wire schema at all (confirmed by reading src/schema/controller.capnp's struct Worker and the generated decode_Worker/encode_Worker in controller.capnp.c) -- only the separate resvals_to_sched/nresvals_to_sched fields (a different, scheduler-push-facing concept) are encoded. So even applying a simple 'replace admin-masked bits' fix analogous to bug-1472's UPDATE fix would NOT fully replicate a DEAD transition's effects: the resource-value slot zeroing would still be silently lost on every other member, which matters materially at PROMOTION time (a newly-promoted primary would read stale, non-zeroed slots for a worker that died before promotion, potentially over-scheduling a dead host). A correct fix needs either (a) a schema change adding a wire-encodable resource-value-slots field so the DEAD payload can carry it, or (b) a different replication strategy for DEAD (e.g. re-deriving 'this worker's connection is gone locally' independently on each member rather than replicating a mutation), which is a design decision beyond a same-task patch.", - "fix": "NOT FIXED -- deliberately deferred, investigated not skipped (see bug-1472's own fix note for the full investigation). Needs its own design decision on how (or whether) to wire-encode the resource-value-slot zeroing, or an alternative non-replicated detection strategy, before a fix can be written safely.", + "fix": "FIXED (TDD RED->GREEN), design direction chosen by user: wire-encode the resource-value-slot zeroing rather than a non-replicated per-member re-derivation. Investigation revised the original design framing: NO new capnp schema field was actually needed. capnp_encode_worker (persist_worker_hook_fun) already serializes the FULL xce_worker_t.host -- including `attrs` (HostAttrs) -- on every WAL_WORKER_CTRL entry, and mark_worker_unavail() calls reset_hostattrs_unknown() BEFORE firing the hook that triggers that encode, so a DEAD entry's wire payload already carries the post-reset 'unknown'/0 attrs state. The actual gap was narrower: worker_from_capnp() had no DEAD branch at all (fell through to the JOIN-duplicate -EEXIST no-op), so nothing ever APPLIED what was already on the wire. Fixed by adding a DEAD branch (worker.c) mirroring the existing UPDATE branch (bug-1472's own fix): sets XCE_WORKER_UNAVAIL, takes dead_since from the decoded payload's deadSince field (the PRIMARY's own death timestamp -- genuinely wire-driven, not locally re-derived, honoring the chosen design direction), and calls reset_hostattrs_unknown()+add_hostattrs_resvals()+set_host_slots(0) locally to re-derive the resvals from the now-reset attrs -- these three calls are deterministic/data-independent (same effect mark_worker_unavail() itself produces on the primary), so xce_worker_t.resvals (the zarray_t the original investigation worried about) never needs to cross the wire at all. Regression test (test_apply_committed_entry.c's test_ace_worker_ctrl_dead_applies_to_existing_worker): pre-existing worker with live (non-'unknown') attrs receives a DEAD WAL entry with a specific non-zero deadSince -- RED under the pre-fix code (rc=-EEXIST, no mutation, no notification, the exact silent-discard this entry describes), GREEN under the fix (XCE_WORKER_UNAVAIL set, dead_since matches the payload exactly, hostattrs reset to 'unknown'/0, one notification fires). Full controller_unit_test suite: 1101 tests, 2 pre-existing unrelated failures (bug-1561), no regressions.\n\n[Original entry]\nNOT FIXED -- deliberately deferred, investigated not skipped (see bug-1472's own fix note for the full investigation). Needs its own design decision on how (or whether) to wire-encode the resource-value-slot zeroing, or an alternative non-replicated detection strategy, before a fix can be written safely.", "tags": [ "worker.c", "worker_from_capnp", @@ -14085,7 +14028,8 @@ "needs-owner", "needs-design-decision", "not-fixed", - "deferred" + "deferred", + "fixed" ], "related_bugs": [ "bug-1472" @@ -14244,7 +14188,7 @@ "error_message": "Live config-upgrade-snapshot task 8.6 soak (run-ha-soak.sh --config-upgrade ~/tmp/xce-ha-configupgrade2), run on top of bug-1477's fix (checkpoint-identity, config-identity, and host-admin-status-survival assertions ALL PASSED for the first time), FAILED later in the force-elect-rotation step: 5 consecutive job-submit attempts on {ctrl-1 (primary), ctrl-2 (standby)} all got 0/3 acked (\"waiting for write-quorum\"), then hard-failed. Investigated the ORIGINAL hypothesis (a leaked VTASK_LIVE_BLOCK/commit-admission slot from an earlier burst of 17 commit timeouts, lsn=15..31, rc=-110) and REFUTED it: commit_resolve()/wait_release_locked() correctly release the admission slot on every timeout (confirmed 17 'async continuation deadline expired' + 17 'commit failed/timed out' WARN pairs, 1:1, in ctrl-1.log) — no admission-pool leak exists. The REAL mechanism is unrelated to admission control entirely: a permanent Raft AppendEntries log-conflict wedge on standby ctrl-2, compounded by a NACK-handling gap that never triggers catch-up.", "file": "src/controller/ha.c (repl_ack_decode_and_dispatch's WAL-ACK branch, ~6456-6644 -- the `!success` NACK path at ~6631-6642 only checks for a higher term and WARNs \"ignoring\", never calls repl_on_ack/repl_start_catchup/any retry); src/controller/repl.c (repl_broadcast ~295-380, the live single-entry send path -- fire-and-forget, no nextIndex backoff-and-retry on rejection; repl_on_ack ~250-271, the ONLY site that decrements pending_ack_count, never invoked on a NACK)", "root_cause": "ROOT-CAUSED with byte-precise waldump evidence (xce_test's built-in `waldump` REPL command against the preserved deploy dir ~/tmp/xce-ha-configupgrade2), cross-correlated with both ctrl-1's and ctrl-2's logs line-by-line. During the soak's chaotic round-3 kill/restart churn, THREE leadership changes happened in rapid succession (term 3 -> 4 -> 5) within about 20 seconds (13:18:57-13:19:19): ctrl-2 briefly won term=3 and appended its own arm-own-term-gate NOOP, was killed before that entry could be widely replicated/committed, a short-lived term=4 leader (never directly observed in either surviving node's own become-leader lines, but proven to have existed by its WAL fingerprint) won next and wrote ITS OWN own-term NOOP at the SAME log index (14), and finally ctrl-1 won term=5 and became the lasting primary. `xce_test waldump` on ctrl-1's canonical (surviving, majority) WAL shows: lsn=12 CHECKPOINT term=2 (the marker-triggered checkpoint from config-upgrade-snapshot task 5, matching bug-1475/1477/1478's exact family), lsn=13 NOOP term=4 (the short-lived term-4 leader's noop), lsn=14 NOOP term=5 (ctrl-1's own winning noop). `xce_test waldump` on ctrl-2's own local pre-restart WAL shows: lsn=12 CHECKPOINT term=2 (agrees with ctrl-1), lsn=13 NOOP **term=3** (ctrl-2's OWN aborted, never-widely-replicated noop from its deposed term-3 stint) -- a genuine, CONFLICTING entry at the SAME index (13) with a DIFFERENT term (3 vs the canonical 4). This is an entirely NORMAL, expected Raft outcome of rapid leader churn (Raft's own §5.3 log-matching property explicitly anticipates followers accumulating stale, uncommitted tail entries from deposed leaders) -- ctrl-2 correctly rejects ctrl-1's AppendEntries at prevLogIndex=13 with logOk's standard conflicting-term check (confirmed via ha_consume_append_entries's own trace: `prevIdx=13 n=1 -> success=0 match=0`, `RAFT[p1] term=5 event=ae_reject_logmismatch`, repeating for EVERY subsequent new entry ctrl-1 tries to send: prevIdx=14,15,16,...29, one per newly-committed lsn, all rejected, spanning 13:19:19 through 13:19:52). The BUG is what happens (or rather, does not happen) next, confirmed via two independent, compounding gaps: (1) ctrl-1 NEVER backs off and retries with a DECREMENTED prevLogIndex to find the point of log agreement (the standard Raft nextIndex-conflict-resolution protocol, which would let ctrl-2 truncate its conflicting lsn=13 tail and adopt the canonical term=4 entry) -- repl_broadcast (repl.c:295-380) is a pure fire-and-forget per-new-entry send with zero synchronous handling of the reply; it always advances prevIdx forward to (next-new-lsn - 1), never backward. (2) repl_ack_decode_and_dispatch's WAL-ACK reply-dispatch branch in ha.c (~6631-6642) handles a NACK (success=0) by checking ONLY for a higher observed term (raft_on_higher_term_observed, correctly added for the DIFFERENT bug-1421 stale-term scenario) and then explicitly `WARN(\"...success=0 — ignoring\")` -- the comment says exactly what the code does: the rejection is discarded. Critically, `repl_on_ack` (repl.c:250-271) is the ONLY code path that decrements `standby_conn_t.pending_ack_count`, and it is called ONLY from the `success` branch -- so a standby that is stuck rejecting (for ANY reason) has pending_ack_count monotonically climb by 1 on every subsequent broadcast attempt with ZERO corresponding decrements, until repl_broadcast's own MAX_PENDING_ACKS=16 defensive eviction (repl.c:325-334, from ha-config-bootstrap task 7.2, commit 3969064e, 2026-05-28 -- a full month before config-upgrade-snapshot's first commit) fires and marks the standby in_catch_up=1 at 13:19:53 (confirmed: 'repl_broadcast: standby localhost:9992 evicted (pending_ack_count=17 > 16) — triggering catch-up reconnect'). This eviction correctly EXCLUDES the standby from repl_active_standby_count (hence from ha_has_write_quorum's majority calculation) -- but nothing in the codebase actually DRIVES a catch-up stream or snapshot-install for it afterward: ctrl-2's log shows ZERO entry-carrying AppendEntries for the remaining ~60 seconds of the run (13:19:53 through the log's end at 13:20:53) -- only bare heartbeat probes (prevIdx=13,n=0) that trivially succeed since a presence-only probe at the standby's own true log_len=13 doesn't hit the term-conflict check. With ctrl-2 permanently excluded (in_catch_up=1, never un-evicted) and ctrl-3 permanently failing to reconnect (separate, unrelated ECONNREFUSED loop, likely a real chaos-kill in this soak round), ctrl-1 has connected=0 standbys forever ('HA quorum lost: connected=0 required=1', logged once at the eviction and never restored) -- every subsequent job submission either times out waiting for write-quorum (0/3 acked, matching the force-elect step's failure verbatim) or gets an immediate -XCE_ERR_NOT_READY (-10013) fast-reject via ha_dispatch_gate's quorum check. This is a GENUINE, SERIOUS, and apparently PERMANENT availability defect: ANY standby whose log happens to diverge by even ONE entry at a single index during leader-election churn (an entirely normal, expected Raft occurrence, not itself a bug) can become irrecoverably stuck and permanently excluded from quorum, with no self-healing path -- in a 3-node cluster this is one bad-luck election sequence away from TOTAL, PERMANENT write-quorum loss if it hits 2 of 3 standbys. Same family as bug-1466 (open, 'WAL_CHECKPOINT breaks LSN/term co-tracking', NOT fixed at its root) and its now-3 fixed siblings (bug-1475, bug-1477, bug-1478) -- this is the family's 5th distinct member, but the trigger here (an ordinary post-election term-conflicted log tail, NOT specifically a checkpoint-boundary term-derivation issue) is narrower than bug-1466's checkpoint-specific framing and points instead at a genuinely MISSING piece of core Raft log-repair protocol (conflict-driven nextIndex backoff + NACK-triggered catch-up), which appears to have never been implemented for this replication path at all -- not a regression, a gap. git blame confirms both implicated mechanisms (repl_broadcast's pending_ack_count eviction, commit 3969064e, 2026-05-28; the NACK-handling 'ignoring' branch in repl_ack_decode_and_dispatch, present since at least bug-1421's fix, itself predating config-upgrade-snapshot) are pre-existing, core HA replication code untouched by any config-upgrade-snapshot task -- config-upgrade-snapshot's task 5 (marker-triggered per-member local checkpoint) merely adds one more entry into the WAL stream around the point where this pre-existing gap gets exercised; it does not create the gap itself.", - "fix": "NOT FIXED -- explicitly out of scope for a single safe pass, per this investigation's own conclusion (matches kiro-debug's guidance against forcing a fix when the true defect requires a protocol design decision, not a narrow patch). At least TWO separate, non-trivial design questions need to be resolved by whoever owns ha.c/repl.c's replication-catch-up protocol before a fix can be written safely: (a) should an ae_reject_logmismatch NACK immediately trigger repl_start_catchup (moving the standby to catch-up mode on the FIRST conflict, rather than silently accumulating up to 16 failed sends first)? This is the more targeted, root-cause-adjacent fix. (b) independently, once a standby IS marked in_catch_up (whether via this path or the existing MAX_PENDING_ACKS eviction), what mechanism is SUPPOSED to actually drive its catch-up (stream from join_applied_lsn, or snapshot-install if too far behind) and clear in_catch_up via repl_complete_catchup? Evidence here shows NOTHING does, for at least this eviction path -- config_repl.c's ACK-driven catchup_lane_fn/ha_walack_on_ack continuation (the only other repl_complete_catchup call site found) is itself ACK-driven, so it can never engage for a standby that never sends a genuine ACK in the first place; whether IT (or a periodic sweep independent of ACKs) is supposed to notice an in_catch_up standby and proactively start streaming it needs its own trace by the owning team. A correct fix likely needs both: NACK-triggered catch-up entry (closing gap (a)) AND a proactive (not purely ACK-reactive) catch-up driver that streams from join_applied_lsn or falls back to snapshot-install for any in_catch_up standby (closing gap (b)) -- implementing only one without the other would leave the defect only partially closed. Recommend routing to whoever owns ha.c/repl.c's HA replication/catch-up protocol (the same 'needs-owner' bucket as bug-1474's still-open fix angle (a) and bug-1472-remainder) for a proper design pass, THEN a TDD fix + live re-verification via `run-ha-soak.sh --config-upgrade` on a fresh deploy dir. Preserved evidence: ~/tmp/xce-ha-configupgrade2 (do not delete) -- `xce_test waldump ` (via the xce_test REPL: `printf 'waldump \\nquit\\n' | ASAN_OPTIONS=detect_leaks=0 timeout 60 /controller/xce_test`) reproduces the byte-precise lsn=13 term-3-vs-term-4 conflict evidence directly from the preserved WAL segments at any time.\n\nUPDATE (2026-07-29, this session): a fix WAS committed (uncommitted in the 'config' worktree at investigation time) implementing exactly gap (a) from this entry's own recommendation -- ha_raft_on_repl_nack() (ha.c/ha.h) routes a WAL-ACK-reply NACK into the raft library's nextIndex backoff and returns a corrected resume LSN; repl_ack_decode_and_dispatch's NACK branch calls it and immediately repl_start_catchup() with the corrected LSN; repl_get_catchup_join_lsn() (repl.c/repl.h) lets the heartbeat-driven ha_dispatch_repl_action_for_standby use the corrected position; repl_complete_catchup() now also resets pending_ack_count. 1048/1048 controller_unit_test PASS. HOWEVER: a fresh live-soak re-verification (run-ha-soak.sh --config-upgrade, deploy preserved at ~/tmp/xce_ha_bug1479_verify, NOT deleted) reproduced the ORIGINAL symptom verbatim -- force-elect ctrl-2 round: 5/5 submit_jobs attempts on {ctrl-1,ctrl-2} got 0/3 acked, hard FAIL. Root-caused: the fix's NACK-routing branch in repl_ack_decode_and_dispatch (ha.c ~6680-6720) is CORRECT but UNREACHABLE in production, because the standby's OWN reject path never sends a NACK (or any reply) in the first place. ha_apply_wal_catchup_entry has three reject paths (ha.c ~3230-3232 via ha_apply_term_entry_via_library for the term-carrying single/batch case -- the one actually hit live; ~3320-3321 for a batch sub-entry apply failure; ~3373-3374 for the legacy single-entry path) that all explicitly do 'no ACK' on a logOk reject/forward-gap. ha_send_wal_ack_reply (ha.c:3084-3106) is the ONLY function anywhere in the controller that ever constructs a WalAckReply, and it hardcodes success=1 unconditionally -- there is NO production code path that ever sends success=0. Live evidence: ctrl-2's own log shows repeating RAFT[p1] event=ae_reject_logmismatch / ha_consume_append_entries success=0 match=0 at growing prevIdx (40,41,...54) while stuck at match=20 forever; ctrl-1's log has ZERO 'WAL NACK from standby' or ha_raft_on_repl_nack lines the entire time -- the primary never learns of the rejection at all, so it never backs off nextIndex or re-arms catch-up, and committed_lsn stays frozen (the only alive standby can never re-sync, and the heartbeat-driven catch-up's own target is gated on the same frozen committed_lsn). STILL NOT FULLY FIXED. New follow-up spec .kiro/specs/raft-adapter-conformance/ created to (1) fix this concrete missing-NACK-reply defect at the three reject sites and (2) perform a broader raft.tla action-by-action conformance pass over the controller's Raft adapter (ha.c/repl.c/config_repl.c) -- distinct from raft-sim-conformance, which is scoped to src/lib/raft/raft_sim.c (library-only, in-process) and explicitly does not cover this adapter code.\n\nUPDATE 2 (2026-07-29, this session): fully FIXED and verified live end-to-end. Two more compounding gaps found and closed after the NACK-send fix (UPDATE 1) landed:\n(1) ha_apply_term_entry_via_library's logOk-reject path (ha.c, the single live-reachable reject site) sent NO reply at all on rejection -- ha_send_wal_ack_reply hardcoded success=1 unconditionally, so the primary-side ha_raft_on_repl_nack (correct since UPDATE 1) was unreachable dead code. Fixed by adding ha_send_wal_ack_reply_ex(ha, ep, c, lsn, success) and calling it with success=0 on reject, plus a new ha_wal_ack_send_hook (ha.h, XCE_TESTING) so a unit test can observe the NACK without a live endpoint (test_bug1479_ae_reject_sends_nack_not_silence, test_config_repl.c). Confirmed live in a 2nd soak run: ctrl-1's log showed real 'WAL NACK from standby ... success=0 -- marking catch-up' lines for the first time.\n(2) Even with (1) working, that 2nd soak run STILL failed at the identical force-elect ctrl-2 point: ha_stream_wal_to_standby's catch-up range was hard-bounded at committed_lsn, and committed_lsn can never advance in this exact scenario because advancing it requires an ACK from the very standby that's stuck catching up (2-of-3 alive, this standby is the only path to a new majority) -- a self-referential deadlock. Fixed by adding a new pure function ha_compute_repl_stream_ceiling_lsn(committed_lsn, last_written, checkpoint_lsn) (ha.c/ha.h), wired into ha_dispatch_repl_action_for_standby to widen the STREAM_WAL ceiling (and the ha_evaluate_repl_action convergence check that gates it) to the leader's own log tail instead of just committed_lsn -- matching raft.tla's real AppendEntries semantics (leaderCommit is advisory to the follower, never a cap on what gets sent) -- while still guarding against bug-1478's old trap (never widening to a trailing local-only checkpoint's own unstreamable LSN slot). 4 new pure-function tests in test_ha_role.c (test_compute_repl_stream_ceiling_lsn_*), RED-confirmed against a stubbed committed_lsn-only version before the real fix was restored.\nFull verification: 1053/1053 controller_unit_test PASS twice clean. THREE successive live run-ha-soak.sh --config-upgrade re-runs tracked each fix: v1 (NACK-send only) still FAILED at force-elect ctrl-2 (0/3 acked); v2 confirmed NACKs now reaching the primary but still FAILED (frozen committed_lsn=14, ha_stream_wal_to_standby target never advanced); v3 (both fixes together) PASSED end-to-end: all 3 force-elect rounds succeeded (ctrl-2: 3/3 jobs acked, term=6; ctrl-3: 3/3 jobs acked, term=7), final verdict 'PASS (config-upgrade): 12 jobs, zero materialized-state divergence, zero configuration divergence, host admin-status confirmed across all members'. Evidence preserved at ~/tmp/xce_ha_bug1479_verify, ~/tmp/xce_ha_bug1479_verify_v2, ~/tmp/xce_ha_bug1479_verify_v3 (all not deleted). config-upgrade-snapshot task 8.6 unblocked. The raft-adapter-conformance spec created mid-investigation had its concrete seed defect (missing NACK reply) fixed here; the broader raft.tla action-by-action conformance audit it also scopes remains valid, separate future work.", + "fix": "FIXED AND VERIFIED LIVE (2026-07-29, see UPDATE 2 below for the final closing fix) -- two compounding gaps were closed after the initial investigation: (1) the standby's own reject path sent no NACK reply at all (ha_send_wal_ack_reply hardcoded success=1 unconditionally), fixed via ha_send_wal_ack_reply_ex with success=0 on reject; (2) even with NACKs flowing, ha_stream_wal_to_standby's catch-up range was hard-capped at committed_lsn, which could never advance in this exact scenario (self-referential deadlock) -- fixed via ha_compute_repl_stream_ceiling_lsn widening the ceiling to the leader's own log tail, matching raft.tla's real AppendEntries semantics. Verified via three successive live run-ha-soak.sh --config-upgrade re-runs tracking each iteration (v1/v2 still failed, v3 with both fixes together PASSED end-to-end: 3/3 jobs acked on both force-elect rounds, zero materialized-state or config divergence). 1053/1053 controller_unit_test PASS. Full investigation history preserved below for context (including the earlier partial-fix iterations and root-causing).\n\n[Original investigation, 2026-07-28]\nNOT FIXED -- explicitly out of scope for a single safe pass, per this investigation's own conclusion (matches kiro-debug's guidance against forcing a fix when the true defect requires a protocol design decision, not a narrow patch). At least TWO separate, non-trivial design questions need to be resolved by whoever owns ha.c/repl.c's replication-catch-up protocol before a fix can be written safely: (a) should an ae_reject_logmismatch NACK immediately trigger repl_start_catchup (moving the standby to catch-up mode on the FIRST conflict, rather than silently accumulating up to 16 failed sends first)? This is the more targeted, root-cause-adjacent fix. (b) independently, once a standby IS marked in_catch_up (whether via this path or the existing MAX_PENDING_ACKS eviction), what mechanism is SUPPOSED to actually drive its catch-up (stream from join_applied_lsn, or snapshot-install if too far behind) and clear in_catch_up via repl_complete_catchup? Evidence here shows NOTHING does, for at least this eviction path -- config_repl.c's ACK-driven catchup_lane_fn/ha_walack_on_ack continuation (the only other repl_complete_catchup call site found) is itself ACK-driven, so it can never engage for a standby that never sends a genuine ACK in the first place; whether IT (or a periodic sweep independent of ACKs) is supposed to notice an in_catch_up standby and proactively start streaming it needs its own trace by the owning team. A correct fix likely needs both: NACK-triggered catch-up entry (closing gap (a)) AND a proactive (not purely ACK-reactive) catch-up driver that streams from join_applied_lsn or falls back to snapshot-install for any in_catch_up standby (closing gap (b)) -- implementing only one without the other would leave the defect only partially closed. Recommend routing to whoever owns ha.c/repl.c's HA replication/catch-up protocol (the same 'needs-owner' bucket as bug-1474's still-open fix angle (a) and bug-1472-remainder) for a proper design pass, THEN a TDD fix + live re-verification via `run-ha-soak.sh --config-upgrade` on a fresh deploy dir. Preserved evidence: ~/tmp/xce-ha-configupgrade2 (do not delete) -- `xce_test waldump ` (via the xce_test REPL: `printf 'waldump \\nquit\\n' | ASAN_OPTIONS=detect_leaks=0 timeout 60 /controller/xce_test`) reproduces the byte-precise lsn=13 term-3-vs-term-4 conflict evidence directly from the preserved WAL segments at any time.\n\nUPDATE (2026-07-29, this session): a fix WAS committed (uncommitted in the 'config' worktree at investigation time) implementing exactly gap (a) from this entry's own recommendation -- ha_raft_on_repl_nack() (ha.c/ha.h) routes a WAL-ACK-reply NACK into the raft library's nextIndex backoff and returns a corrected resume LSN; repl_ack_decode_and_dispatch's NACK branch calls it and immediately repl_start_catchup() with the corrected LSN; repl_get_catchup_join_lsn() (repl.c/repl.h) lets the heartbeat-driven ha_dispatch_repl_action_for_standby use the corrected position; repl_complete_catchup() now also resets pending_ack_count. 1048/1048 controller_unit_test PASS. HOWEVER: a fresh live-soak re-verification (run-ha-soak.sh --config-upgrade, deploy preserved at ~/tmp/xce_ha_bug1479_verify, NOT deleted) reproduced the ORIGINAL symptom verbatim -- force-elect ctrl-2 round: 5/5 submit_jobs attempts on {ctrl-1,ctrl-2} got 0/3 acked, hard FAIL. Root-caused: the fix's NACK-routing branch in repl_ack_decode_and_dispatch (ha.c ~6680-6720) is CORRECT but UNREACHABLE in production, because the standby's OWN reject path never sends a NACK (or any reply) in the first place. ha_apply_wal_catchup_entry has three reject paths (ha.c ~3230-3232 via ha_apply_term_entry_via_library for the term-carrying single/batch case -- the one actually hit live; ~3320-3321 for a batch sub-entry apply failure; ~3373-3374 for the legacy single-entry path) that all explicitly do 'no ACK' on a logOk reject/forward-gap. ha_send_wal_ack_reply (ha.c:3084-3106) is the ONLY function anywhere in the controller that ever constructs a WalAckReply, and it hardcodes success=1 unconditionally -- there is NO production code path that ever sends success=0. Live evidence: ctrl-2's own log shows repeating RAFT[p1] event=ae_reject_logmismatch / ha_consume_append_entries success=0 match=0 at growing prevIdx (40,41,...54) while stuck at match=20 forever; ctrl-1's log has ZERO 'WAL NACK from standby' or ha_raft_on_repl_nack lines the entire time -- the primary never learns of the rejection at all, so it never backs off nextIndex or re-arms catch-up, and committed_lsn stays frozen (the only alive standby can never re-sync, and the heartbeat-driven catch-up's own target is gated on the same frozen committed_lsn). STILL NOT FULLY FIXED. New follow-up spec .kiro/specs/raft-adapter-conformance/ created to (1) fix this concrete missing-NACK-reply defect at the three reject sites and (2) perform a broader raft.tla action-by-action conformance pass over the controller's Raft adapter (ha.c/repl.c/config_repl.c) -- distinct from raft-sim-conformance, which is scoped to src/lib/raft/raft_sim.c (library-only, in-process) and explicitly does not cover this adapter code.\n\nUPDATE 2 (2026-07-29, this session): fully FIXED and verified live end-to-end. Two more compounding gaps found and closed after the NACK-send fix (UPDATE 1) landed:\n(1) ha_apply_term_entry_via_library's logOk-reject path (ha.c, the single live-reachable reject site) sent NO reply at all on rejection -- ha_send_wal_ack_reply hardcoded success=1 unconditionally, so the primary-side ha_raft_on_repl_nack (correct since UPDATE 1) was unreachable dead code. Fixed by adding ha_send_wal_ack_reply_ex(ha, ep, c, lsn, success) and calling it with success=0 on reject, plus a new ha_wal_ack_send_hook (ha.h, XCE_TESTING) so a unit test can observe the NACK without a live endpoint (test_bug1479_ae_reject_sends_nack_not_silence, test_config_repl.c). Confirmed live in a 2nd soak run: ctrl-1's log showed real 'WAL NACK from standby ... success=0 -- marking catch-up' lines for the first time.\n(2) Even with (1) working, that 2nd soak run STILL failed at the identical force-elect ctrl-2 point: ha_stream_wal_to_standby's catch-up range was hard-bounded at committed_lsn, and committed_lsn can never advance in this exact scenario because advancing it requires an ACK from the very standby that's stuck catching up (2-of-3 alive, this standby is the only path to a new majority) -- a self-referential deadlock. Fixed by adding a new pure function ha_compute_repl_stream_ceiling_lsn(committed_lsn, last_written, checkpoint_lsn) (ha.c/ha.h), wired into ha_dispatch_repl_action_for_standby to widen the STREAM_WAL ceiling (and the ha_evaluate_repl_action convergence check that gates it) to the leader's own log tail instead of just committed_lsn -- matching raft.tla's real AppendEntries semantics (leaderCommit is advisory to the follower, never a cap on what gets sent) -- while still guarding against bug-1478's old trap (never widening to a trailing local-only checkpoint's own unstreamable LSN slot). 4 new pure-function tests in test_ha_role.c (test_compute_repl_stream_ceiling_lsn_*), RED-confirmed against a stubbed committed_lsn-only version before the real fix was restored.\nFull verification: 1053/1053 controller_unit_test PASS twice clean. THREE successive live run-ha-soak.sh --config-upgrade re-runs tracked each fix: v1 (NACK-send only) still FAILED at force-elect ctrl-2 (0/3 acked); v2 confirmed NACKs now reaching the primary but still FAILED (frozen committed_lsn=14, ha_stream_wal_to_standby target never advanced); v3 (both fixes together) PASSED end-to-end: all 3 force-elect rounds succeeded (ctrl-2: 3/3 jobs acked, term=6; ctrl-3: 3/3 jobs acked, term=7), final verdict 'PASS (config-upgrade): 12 jobs, zero materialized-state divergence, zero configuration divergence, host admin-status confirmed across all members'. Evidence preserved at ~/tmp/xce_ha_bug1479_verify, ~/tmp/xce_ha_bug1479_verify_v2, ~/tmp/xce_ha_bug1479_verify_v3 (all not deleted). config-upgrade-snapshot task 8.6 unblocked. The raft-adapter-conformance spec created mid-investigation had its concrete seed defect (missing NACK reply) fixed here; the broader raft.tla action-by-action conformance audit it also scopes remains valid, separate future work.", "tags": [ "ha.c", "repl.c", @@ -15555,7 +15499,7 @@ "error_message": "apiserver never re-registers (application-level xcecli_join with HostRole_apiserver) with the new primary after a live controller failover, even though its transport/write-call connectivity clearly recovers (job/host-control POSTs keep succeeding); observed for 10+ minutes and 20+ default-30s heartbeat ticks with zero occurrences of reconnect.go's own 'checkAndPerformRejoin: rejoin after transparent reconnect succeeded/failed' log lines in apiserver_out.log, across 3 separate write calls (2 job submits + 1 host control) issued after the primary was kill -9'd. TASK 5.2 UPDATE (2026-08-12, second independent live E2E reproduction, cleaner isolation): ran component-failover-adoption task 5.2's own scenario (apiserver-only, no worker/scheduler, 3-node controller cluster at localhost:19991-19993, apiserver -d /home/claude/tmp/xce-e2e-task52/apiserver) specifically to isolate the heartbeat-only idle-period path task 5.1's write-traffic-muddied run could not cleanly separate. Confirmed real election: ctrl-1 (primary, term=2) kill -9'd at 13:33:31Z; ctrl-2 promoted primary at term=3 within 9s (13:33:40Z, raft_cb_become_leader). Zero write requests issued for the full outage window: apiserver_out.log is BYTE-IDENTICAL (23 lines) from before the kill through 268s / ~9 heartbeat ticks after it (well past the task's 2-minute/3-4-tick minimum) -- zero occurrences of 'checkAndPerformRejoin'/'rejoin'/'needsRejoin'/'xcecli_join' anywhere. Diagnostic control issued only AFTER the observation window (13:39:52Z, ~6.5 min post-kill): POST /xce/v1/jobs/create succeeded with a real controller-assigned job ID (73930919093731328) and server-generated success message -- proving the transport had already reached the NEW primary (ctrl-2) -- yet this write call ALSO produced zero rejoin log lines, full apiserver_out.log grep for the same 4 terms still empty after it. This is strictly stronger evidence than task 5.1's: it directly demonstrates the write path's own XCECLI_FAILOVER_CALL/xcecli_invoke_rejoin also did not fire even when explicitly exercised, consistent with the transport having already been silently repaired by an earlier heartbeat tick's XCECLI_FF_CALL before the write call ever needed to retry -- exactly bug-1548's leading hypothesis, not merely 'consistent with' it as task 5.1 could only say. Corroborating bug-1549 downstream symptom also reproduced: GET /xce/v1/jobs immediately after the successful submit returned {\"count\":0,\"rows\":[]} -- the just-submitted job never reached apiserver's local store, consistent with apiserver never having actually re-issued xcecli_join(HostRole_apiserver) against ctrl-2. ctrl-2.log has no apiserver-join lines either direction (join.c logs only rejections, per task 5.1's prior finding -- reconfirmed, not independently informative here). Task 5.2's own observable-completion criterion ('the scenario passes without requiring any client-triggered write request during the outage window' -- i.e. apiserver rejoins and resumes push notifications via the heartbeat-triggered path alone) was NOT met: no rejoin evidence appeared at any point in the heartbeat-only window. Evidence preserved at /home/claude/tmp/xce-e2e-task52/ (apiserver/logs/apiserver_out.log, logs/ctrl-{1,2,3}.log, logon_response.json, diagnostic_job_submit_response.json, diagnostic_joblist_response.json, KILL_TIMESTAMP_UTC.txt, DIAGNOSTIC_CONTROL_START_UTC.txt, DIAGNOSTIC_CONTROL_SUBMIT_UTC.txt).", "file": "src/frontend/apiserver/backend/reconnect.go, heartbeat.go, clib.go (component-failover-adoption task 4.3/4.4's checkAndPerformRejoin/needsRejoin mechanism)", "root_cause": "NOT root-caused (out of task 5.1's E2E-validation boundary to fix). Leading hypothesis from live log analysis: XCECLI_FF_CALL (the fire-and-forget macro backing xcecli_heartbeat) can silently self-heal the TRANSPORT (rotate to a live candidate) entirely within a single heartbeat tick without that tick ever returning a failure to Go -- so heartbeat.go's prevFailed-edge detection (onFailureToSuccess) never observes a failure->success transition and never sets needsRejoin. Separately, none of the 3 post-kill write calls (which DO route through XCECLI_FAILOVER_CALL, which per design.md should invoke the rejoin callback mid-call on a genuine failover) show any evidence of having hit a failed connection either -- consistent with the connection having already been silently fixed by a prior heartbeat tick before any write call needed to retry, so XCECLI_FAILOVER_CALL's own rejoin-invoking path is also never exercised. Net effect: needsRejoin is never set by any path, so checkAndPerformRejoin() stays a permanent silent no-op and apiserver's push-registration (join) with the new primary is never renewed. Could not independently confirm via controller-side logs either way -- discovered during this same investigation that the controller's join.c only logs REJECTED/failed joins (WARN/ERROR), never a successful one at any verbosity, so controller logs are not usable evidence for this question in either direction. REVIEW UPDATE (task 5.1 review, 2026-08-12): confirmed via source read that XCECLI_FF_CALL (client_internal.h:290-302) never calls xcecli_invoke_rejoin under any circumstance -- this half of the hypothesis is not speculative, it's structurally true by construction. Elevated confidence this is a REPRODUCIBLE gap, not scenario-specific timing luck: since real elections in this codebase's own logs complete in single-digit seconds while the default heartbeat interval is 30s, a single heartbeat tick landing after the election has already completed will both discover the dead connection AND repair it within that same XCECLI_FF_CALL invocation -- the failure never surfaces to Go's heartbeat.go loop as a separate tick, so the cross-tick prevFailed-edge detection (onFailureToSuccess) structurally cannot observe it. This is likely to recur on MOST real single-election failovers, not just this run's specific timing -- materially risks Req 2 AC4 (component-failover-adoption requirements.md) in production. Recommend escalating past ordinary buglog cadence to a dedicated follow-up spec/task, given it affects the already-approved task 4.3/4.4 mechanism.", - "fix": "NOT FIXED -- flagged for a dedicated follow-up investigation/spec item, not patched here. This is a live-E2E finding from component-failover-adoption task 5.1, orthogonal to task 5.1's own boundary (cross-component reconnect validation, not a single-component code fix).", + "fix": "FIXED (component-failover-adoption tasks.md Phase 6, task 6.1). Root cause confirmed as hypothesized: XCECLI_FF_CALL (client_internal.h:290-302) can rotate+reconnect the transport entirely inside a single call, holding ac->failover.lock, and still return success -- so a failure->success transition on the caller's own return code is not a reliable 'a reconnect happened' signal whenever an election (single-digit seconds) completes faster than the 30s default heartbeat interval, which is the common case. Fix: added a monotonic `_Atomic uint64_t reconnect_gen` field to `endpoint_t` (src/include/netio.h:137), incremented unconditionally on the success path of `xce_ctx_reconnect()` (src/lib/xcectx.c:402) -- confirmed via repo-wide grep as the sole reconnect call-site reachable from `xce_failover_rotate()`, covering both the write-path retry loop and the heartbeat fire-and-forget path. apiserver's heartbeat.go `runHeartbeat()` now compares this generation on every tick (via the existing `xcecli_get_my_endpoint()` accessor, atomic-loaded through clib.go) instead of waiting for a failure->success transition, replacing the old detector entirely. Regression test `TestRunHeartbeat_IdlePeriodRecovery_GenChangeAloneSetsNeedsRejoinEvenWhenAllTicksSucceed` (heartbeat_rejoin_test.go) proves the fix's exact shape. Closed via live E2E re-run of task 5.2's scenario: rejoin fired 22s after primary kill (12s post-election) with zero write traffic for the preceding 5.5 minutes.", "tags": [ "apiserver", "rejoin", @@ -15564,14 +15508,14 @@ "component-failover-adoption", "task-5.1", "heartbeat", - "unresolved", "live-e2e-finding", "task-5.2", - "second-independent-repro" + "second-independent-repro", + "resolved" ], "related_bugs": [], "occurrences": 3, - "last_seen": "2026-08-12T13:39:52Z" + "last_seen": "2026-08-15T22:54:00Z" }, { "id": "bug-1549", @@ -15579,7 +15523,7 @@ "error_message": "Jobs submitted via apiserver REST API (POST /xce/v1/jobs/create) never appear in apiserver's own job store (GET /xce/v1/jobs / GET /xce/v1/jobs/:id returns empty), even though the job is confirmed genuinely committed on the controller (verified directly via xce_test's `joblist` showing both jobs in PEND state). Reproduced for 2 jobs (one submitted at the moment of a primary kill, one submitted >1 minute after a clean new-primary failover) -- so this is NOT specific to the failover window, appears to be a general gap. By contrast, worker/host-list push (apiserver_push_workers/apiserver_push_workerlist, controller/apiserver.c) DOES work: GET /xce/v1/hosts correctly showed the live-joined worker with real data.", "file": "src/controller/apiserver.c (apiserver_job_new_fun / apiserver_push_joblist hook), src/frontend/apiserver/backend/clib.go (go_push_handler / handleDataset)", "root_cause": "NOT root-caused (out of task 5.1's boundary). apiserver_job_new_fun is wired as a HOOK_DETAIL_JOB_NEW hook meant to fire apiserver_push_joblist()/apiserver_push_jobarray() after a job commits (apply_committed_entry()); controller logs show zero evidence either way since apiserver.c has no INFO-level success logging on this path (only a WARN on push failure, `push job list to api server %s has problem, rc %d`, which also never appeared). Could be: the hook isn't actually registered/invoked in this build, or apiserver's join isn't registered in c->apiservers[] the push fan-out iterates, or something queue/resreq-specific. Discovered opportunistically while probing bug-1548; not independently isolated from bug-1548 (both could share a root cause -- c->apiservers[] fan-out list membership -- or be unrelated). REVIEW UPDATE (task 5.1 review, 2026-08-12): found the likely direct causal chain, narrowing this from 'could share a root cause or be unrelated' to 'very likely a downstream symptom of bug-1548'. src/controller/apiserver.c's on_apiserver_join() unconditionally calls xce_apiserver_push_jobs(c, ep) (bulk job resync) alongside xce_apiserver_push_workers(c, ep), but ONLY fires when a fresh join is actually processed by the controller. c->apiservers[] (the fan-out list push hooks iterate) is per-process, in-memory-only state -- no WAL/config replication (confirmed zero references outside apiserver.c) -- so a NEW primary starts with an empty registration table until each apiserver instance re-joins against it. If bug-1548 means apiserver's rejoin was never re-issued against the new primary, ctrl-2 never ran on_apiserver_join for it, so neither the bulk resync nor any incremental job push could ever reach apiserver -- cleanly explaining both job1 and job2 being empty. Recommend a fix attempt for bug-1548 be verified against this job-push symptom directly (if fixing the rejoin gap makes GET /xce/v1/jobs start populating correctly, that's confirmation bug-1549 was never an independent issue). TASK 5.2 RE-REPRODUCTION (2026-08-12T13:39:52Z): reproduced a third time in an isolated, apiserver-only (no worker/scheduler) idle-period E2E run -- a job submitted well after a clean failover (jobid 73930919093731328, controller-confirmed via the accepted response) still never appeared in GET /xce/v1/jobs. This occurred in the SAME run where bug-1548's rejoin gap was also independently reconfirmed with a clean negative control, strengthening (not just repeating) the downstream-symptom-of-bug-1548 hypothesis: if apiserver never re-joined the new primary at the application level, c->apiservers[] on ctrl-2 never gained an entry for it, so no push (bulk or incremental) could ever reach it -- independent of whether the underlying job-push wiring itself is otherwise correct.", - "fix": "NOT FIXED -- flagged for follow-up, orthogonal to task 5.1's cross-component reconnect-validation boundary.", + "fix": "FIXED as a downstream symptom of bug-1548 -- confirmed, not just hypothesized. Once apiserver's heartbeat-triggered rejoin (bug-1548's fix) correctly re-issues xcecli_join(HostRole_apiserver) against the new primary, the controller's on_apiserver_join() bulk job resync (xce_apiserver_push_jobs) fires as designed and populates apiserver's local job store. Live E2E re-validation (component-failover-adoption task 6.2): a diagnostic job submitted post-recovery correctly appeared via GET /xce/v1/jobs, with no client-triggered write request required during the outage window.", "tags": [ "apiserver", "controller", @@ -15587,14 +15531,14 @@ "job-submit", "component-failover-adoption", "task-5.1", - "unresolved", - "live-e2e-finding" + "live-e2e-finding", + "resolved" ], "related_bugs": [ "bug-1548" ], "occurrences": 2, - "last_seen": "2026-08-12T13:39:52Z" + "last_seen": "2026-08-15T22:54:00Z" }, { "id": "bug-1550", @@ -16072,6 +16016,403 @@ "related_bugs": [], "occurrences": 1, "last_seen": "2026-08-13T13:11:27.503Z" + }, + { + "id": "bug-1558", + "timestamp": "2026-08-15T11:20:00Z", + "error_message": "make test-backend fails standalone: backend/reconnect.go:6:10: fatal error: client.h: No such file or directory -- FAIL apiserver/backend [build failed], make: *** [Makefile:33: test-backend] Error 1. Hit during /kiro-validate-impl component-failover-adoption full-suite validation.", + "file": "src/frontend/Makefile (test-backend target), src/frontend/apiserver/meson.build", + "root_cause": "CGO_CFLAGS/CGO_LDFLAGS for libxce and its transitive deps are constructed only inside apiserver/meson.build (lines 11-49) and set via meson's go_env environment() for its custom_target. They are never exported to a general shell, so the Makefile's test-backend target runs bare `CGO_ENABLED=1 go test ./backend/...` with no include or link paths. Compounding this, xce-build.sh's RUN_TESTS path only invokes `make test` (CGO_ENABLED=0), so the CGO backend suite -- which holds this spec's load-bearing rejoin/heartbeat/candidate-parity tests -- is structurally never exercised by any documented command. Previously noted narratively in component-failover-adoption tasks.md Implementation Notes (task 4.1, task 5.4) but never logged as a bug.", + "fix": "FIXED (component-failover-adoption tasks.md Phase 6, non-blocking build-tooling fix). src/frontend/Makefile's test-backend target now self-derives CGO_CFLAGS/CGO_LDFLAGS via `pkg-config --cflags/--libs librtti c-capnproto openssl tinylog` plus -I$(CURDIR)/../include -I$(BUILD_DIR)/lib and -L$(BUILD_DIR)/lib -lxce -lm -lrt -fsanitize=address, mirroring apiserver/meson.build's own construction. BUILD_DIR anchored on $(CURDIR) (the Makefile's own directory), not a path relative to the `cd apiserver` recipe target, avoiding the off-by-one the workaround note flagged. Verified 2026-08-15: `make test-backend` run standalone from a nix develop shell with zero manual env setup -- `ok apiserver/backend 14.652s`, `ok apiserver/backend/auth 0.010s`.", + "tags": [ + "build", + "tooling", + "cgo", + "apiserver", + "test-backend", + "meson", + "makefile", + "component-failover-adoption", + "resolved" + ], + "related_bugs": [ + "bug-1440", + "bug-1441" + ], + "occurrences": 1, + "last_seen": "2026-08-15T22:54:00Z" + }, + { + "id": "bug-1559", + "timestamp": "2026-08-15T13:10:00Z", + "error_message": "xce-build.sh scheduler's RUN_TESTS block never builds or runs the 4 new component-failover-adoption task-3.x scheduler test binaries (sched_scheduler_connect_test, sched_scheduler_backpressure_test, sched_scheduler_decision_publish_retry_test, sched_scheduler_decision_redelivery_test) -- it only builds/runs 'xce_scheduler sched_test'. They only got compiled in this session as a side effect of a separate worker build and had to be run directly, bypassing the canonical script.", + "file": "src/tools/xce-build.sh, src/scheduler/testsuite/meson.build", + "root_cause": "xce-build.sh's scheduler case's RUN_TESTS list was never updated when task 3.x added 4 new standalone test executables under src/scheduler/testsuite/meson.build -- same class of gap as bug-1558 (documented commands cover only a subset of the real suite), just for the scheduler component instead of apiserver's CGO path.", + "fix": "FIXED (component-failover-adoption tasks.md Phase 6, non-blocking build-tooling fix). src/tools/xce-build.sh's scheduler case's RUN_TESTS block now runs `meson test -C \"$BUILD_DIR\" --suite scheduler --print-errorlogs` instead of invoking sched_test directly by path -- meson test rebuilds by default, so no need to separately enumerate the 4 previously-skipped binaries. Verified 2026-08-15: `xce-build.sh scheduler asan` now runs all 41 scheduler suite tests (was: 1 binary only). NOTE: running the full suite for the first time surfaced a previously-invisible failure, test_recordstart_symlink_escape_replies_error (test_admin_request.c) -- logged separately as bug-1560, unrelated to this fix or to component-failover-adoption.", + "tags": [ + "build", + "tooling", + "scheduler", + "xce-build.sh", + "component-failover-adoption", + "coverage-gap", + "resolved" + ], + "related_bugs": [ + "bug-1558" + ], + "occurrences": 1, + "last_seen": "2026-08-15T22:54:00Z" + }, + { + "id": "bug-900", + "timestamp": "2026-08-15T13:02:22.188Z", + "error_message": "Significant refactor of ", + "file": "src/frontend/apiserver/backend/clib.go", + "root_cause": "6 lines replaced/restructured", + "fix": "Rewrote 14→30 lines (6 removed)", + "tags": [ + "auto-detected", + "refactor", + "go" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-08-15T13:02:22.188Z" + }, + { + "id": "bug-901", + "timestamp": "2026-08-15T13:05:02.816Z", + "error_message": "Significant refactor of ", + "file": "src/frontend/Makefile", + "root_cause": "3 lines replaced/restructured", + "fix": "Rewrote 3→6 lines (3 removed)", + "tags": [ + "auto-detected", + "refactor", + "unknown" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-08-15T13:05:02.816Z" + }, + { + "id": "bug-902", + "timestamp": "2026-08-15T13:06:10.183Z", + "error_message": "Wrong reference: 039 should be 1548", + "file": ".kiro/specs/component-failover-adoption/tasks.md", + "root_cause": "Used \"039\" instead of \"1548\"", + "fix": "Changed 039 → 1548", + "tags": [ + "auto-detected", + "wrong-reference", + "md" + ], + "related_bugs": [], + "occurrences": 3, + "last_seen": "2026-08-15T13:06:19.081Z" + }, + { + "id": "bug-903", + "timestamp": "2026-08-15T13:07:00.267Z", + "error_message": "Incorrect value in code", + "file": ".kiro/specs/component-failover-adoption/tasks.md", + "root_cause": "Had 's own scope to fix (task 5.x is cross-component v", + "fix": "Changed to `/kiro-validate-impl`", + "tags": [ + "auto-detected", + "wrong-value", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-08-15T13:07:00.267Z" + }, + { + "id": "bug-904", + "timestamp": "2026-08-15T13:07:21.491Z", + "error_message": "Significant refactor of ", + "file": ".kiro/specs/component-failover-adoption/spec.json", + "root_cause": "2 lines replaced/restructured", + "fix": "Rewrote 20→30 lines (2 removed)", + "tags": [ + "auto-detected", + "refactor", + "json" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-08-15T13:07:21.491Z" + }, + { + "id": "bug-905", + "timestamp": "2026-08-15T14:20:32.090Z", + "error_message": "Missing error handling in unknown", + "file": ".kiro/specs/component-failover-adoption/tasks.md", + "root_cause": "Code path had no error handling — exceptions would propagate uncaught", + "fix": "Added try/catch block", + "tags": [ + "auto-detected", + "error-handling", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-08-15T14:20:32.090Z" + }, + { + "id": "bug-906", + "timestamp": "2026-08-15T14:20:51.010Z", + "error_message": "Incorrect value in code", + "file": ".kiro/specs/component-failover-adoption/spec.json", + "root_cause": "Had \"2026-08-15T13:10:00Z\"", + "fix": "Changed to \"2026-08-15T14:20:00Z\"", + "tags": [ + "auto-detected", + "wrong-value", + "json" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-08-15T14:20:51.010Z" + }, + { + "id": "bug-907", + "timestamp": "2026-08-15T14:21:07.376Z", + "error_message": "Significant refactor of ", + "file": ".kiro/specs/component-failover-adoption/spec.json", + "root_cause": "5 lines replaced/restructured", + "fix": "Rewrote 11→17 lines (5 removed)", + "tags": [ + "auto-detected", + "refactor", + "json" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-08-15T14:21:07.376Z" + }, + { + "id": "bug-1560", + "timestamp": "2026-08-15T22:54:35Z", + "error_message": "test_recordstart_symlink_escape_replies_error (src/scheduler/testsuite/test_admin_request.c:850) FAILED: Expected 'invalid record file name' Was 'permission denied accessing record file'. Surfaced only now because bug-1559's fix made xce-build.sh's scheduler case run the full `meson test --suite scheduler` for the first time -- this test was never reached by the old sched_test-only invocation.", + "file": "src/scheduler/testsuite/test_admin_request.c, src/scheduler/src/recording.c", + "root_cause": "Not yet investigated. Test plants a symlink inside record_dir whose target resolves outside record_dir, then issues a recordStart admin request expecting rejection with error string 'invalid record file name' (the canonical-path containment check). Instead the admin path is returning a 'permission denied accessing record file' error, implying some permission-check code path is being hit and short-circuiting before (or instead of) the containment check in this sandbox environment (uid 1000, non-root) -- ordering between the two checks in recording.c not yet confirmed by direct read. Unrelated to component-failover-adoption / bug-1548/1549/1558/1559 -- discovered purely as a side effect of bug-1559's coverage-gap fix making this test reachable.", + "fix": "NOT FIXED -- out of scope for the component-failover-adoption remediation that surfaced it. Needs its own investigation of recording.c's record-file validation check ordering.", + "tags": [ + "scheduler", + "recording", + "symlink", + "test-failure", + "coverage-gap-discovery", + "unresolved" + ], + "related_bugs": [ + "bug-1559" + ], + "occurrences": 1, + "last_seen": "2026-08-15T22:54:35Z" + }, + { + "id": "bug-909", + "timestamp": "2026-08-16T03:42:14.435Z", + "error_message": "Missing error handling in unknown", + "file": "reports/aug08-aug15-2026.md", + "root_cause": "Code path had no error handling — exceptions would propagate uncaught", + "fix": "Added try/catch block", + "tags": [ + "auto-detected", + "error-handling", + "md" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-08-16T03:42:14.435Z" + }, + { + "id": "bug-910", + "timestamp": "2026-08-16T03:57:07.719Z", + "error_message": "Incorrect value in code", + "file": "reports/aug08-aug15-2026.md", + "root_cause": "Had `build.py`", + "fix": "Changed to `3afe3d5`", + "tags": [ + "auto-detected", + "wrong-value", + "md" + ], + "related_bugs": [], + "occurrences": 2, + "last_seen": "2026-08-16T03:57:25.315Z" + }, + { + "id": "bug-908", + "timestamp": "2026-08-16T04:29:20.063Z", + "error_message": "Significant refactor of ", + "file": ".kiro/specs/config-upgrade-snapshot/spec.json", + "root_cause": "3 lines replaced/restructured", + "fix": "Rewrote 21→29 lines (3 removed) | Also: \"updated_at\": \"2026-07-29T09:20:00Z\",; \"phase\": \"tasks-generated\",", + "tags": [ + "auto-detected", + "refactor", + "json" + ], + "related_bugs": [], + "occurrences": 2, + "last_seen": "2026-08-16T04:29:29.059Z" + }, + { + "id": "bug-909", + "timestamp": "2026-08-16T04:51:52.141Z", + "error_message": "Significant refactor of ", + "file": "src/controller/testsuite/test_commit_barrier_reply.c", + "root_cause": "11 lines replaced/restructured", + "fix": "Rewrote 11→16 lines (11 removed)", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-08-16T04:51:52.141Z" + }, + { + "id": "bug-910", + "timestamp": "2026-08-16T04:54:26.304Z", + "error_message": "Missing guard clause", + "file": "src/controller/persist.c", + "root_cause": "No early return/throw for edge case: j == NULL", + "fix": "Added guard clause: if (j == NULL)", + "tags": [ + "auto-detected", + "guard-clause", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-08-16T04:54:26.304Z" + }, + { + "id": "bug-1561", + "timestamp": "2026-08-16T00:00:00.000Z", + "error_message": "controller_unit_test: test_conf_trim_keeps_only_n_versions and test_conf_get_saved_tag_no_config (test_config_repl.c) FAIL during an unrelated ASAN controller build/test run (bug-451 fix session)", + "file": "src/controller/testsuite/test_config_repl.c", + "root_cause": "Not investigated -- discovered incidentally while verifying an unrelated fix (bug-451, persist_job_hook_fun's RUSAGE arm). Neither test_config_repl.c nor config_repl.c were touched this session (confirmed via git diff, zero changes). Coincides with a burst of 'persist: flusher fdatasync FAILED (errno=5 Input/output error)' ERROR lines in the same test run, matching this project's well-documented overlayfs sandbox fdatasync artifact class (same family as bug-924, bug-1470, bug-1553) rather than a genuine regression.", + "fix": "NOT FIXED -- out of scope for the bug-451 session that surfaced it. Needs confirmation this is the same overlayfs artifact (re-run on a real fs) before treating it as a real regression in test_config_repl.c's config-version-trim/tag-save logic.", + "tags": [ + "controller", + "testsuite", + "test_config_repl", + "overlayfs", + "fdatasync", + "sandbox-environment", + "unconfirmed", + "not-fixed" + ], + "related_bugs": [ + "bug-924", + "bug-1470", + "bug-1553" + ], + "occurrences": 1, + "last_seen": "2026-08-16T00:00:00.000Z" + }, + { + "id": "bug-912", + "timestamp": "2026-08-16T05:04:02.838Z", + "error_message": "Significant refactor of ", + "file": "src/controller/persist.c", + "root_cause": "3 lines replaced/restructured", + "fix": "Rewrote 37→86 lines (3 removed)", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-08-16T05:04:02.838Z" + }, + { + "id": "bug-913", + "timestamp": "2026-08-16T05:22:23.386Z", + "error_message": "Missing error handling in unknown", + "file": "src/controller/testsuite/test_e2e_checkpoint.c", + "root_cause": "Code path had no error handling — exceptions would propagate uncaught", + "fix": "Added try/catch block", + "tags": [ + "auto-detected", + "error-handling", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-08-16T05:22:23.386Z" + }, + { + "id": "bug-914", + "timestamp": "2026-08-16T05:25:35.899Z", + "error_message": "Significant refactor of ", + "file": "src/controller/persist.c", + "root_cause": "2 lines replaced/restructured", + "fix": "Rewrote 6→17 lines (2 removed)", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-08-16T05:25:35.899Z" + }, + { + "id": "bug-915", + "timestamp": "2026-08-16T06:54:02.070Z", + "error_message": "Missing guard clause", + "file": "src/controller/ha.c", + "root_cause": "No early return/throw for edge case: ha->standby_endpoints", + "fix": "Added guard clause: if (ha->standby_endpoints)", + "tags": [ + "auto-detected", + "guard-clause", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-08-16T06:54:02.070Z" + }, + { + "id": "bug-916", + "timestamp": "2026-08-16T07:03:34.559Z", + "error_message": "Significant refactor of ", + "file": "src/controller/worker.c", + "root_cause": "9 lines replaced/restructured", + "fix": "Rewrote 12→23 lines (9 removed)", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-08-16T07:03:34.559Z" + }, + { + "id": "bug-917", + "timestamp": "2026-08-16T09:54:06.127Z", + "error_message": "Significant refactor of ", + "file": "src/controller/worker.c", + "root_cause": "7 lines replaced/restructured", + "fix": "Rewrote 15→8 lines (7 removed)", + "tags": [ + "auto-detected", + "refactor", + "c" + ], + "related_bugs": [], + "occurrences": 1, + "last_seen": "2026-08-16T09:54:06.127Z" } ] } \ No newline at end of file