Rongsong Shen
20e59d0b42
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
99 lines
14 KiB
Text
99 lines
14 KiB
Text
[2026-06-21 08:37:36,163] [1217460] [ INFO] [ main] My name is ctrl-2
|
|
[2026-06-21 08:37:36,164] [1217460] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-2
|
|
[2026-06-21 08:37:36,164] [1217460] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2)
|
|
[2026-06-21 08:37:36,164] [1217460] [ INFO] [ cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin
|
|
[2026-06-21 08:37:36,173] [1217460] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
|
[2026-06-21 08:37:36,173] [1217460] [ INFO] [ load_primary_config] No config at '/tmp/xce-deploy/controller/ctrl-2/conf/current' (rc=-2) — will fetch from primary after election
|
|
[2026-06-21 08:37:36,173] [1217460] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=0)
|
|
[2026-06-21 08:37:36,673] [1217460] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
|
[2026-06-21 08:37:36,673] [1217460] [ INFO] [ha_start_election_async] ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only, not starting own election
|
|
[2026-06-21 08:37:36,673] [1217460] [ ERROR] [ cluster_init_phase2] cluster_init_phase2: cluster_config not loaded
|
|
[2026-06-21 08:37:36,673] [1217460] [ WARN] [ myapp_run] myapp_run: cluster_init_phase2 on standby rejoin failed (rc=-22) — WAL not recovered; falling back to election-path recovery
|
|
[2026-06-21 08:37:36,673] [1217460] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
|
[2026-06-21 08:37:36,673] [1217460] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
|
[2026-06-21 08:37:46,177] [1217460] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
|
[2026-06-21 08:37:46,177] [1217460] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
|
[2026-06-21 08:37:46,177] [1217460] [ INFO] [ha_on_primary_ctrl_fd_rdhup] ha_on_primary_ctrl_fd_rdhup: no local config (tag=0, n_peers=3) — voter only, not starting PreVote/RDHUP re-election; awaiting configured primary (ha-config-bootstrap Req 2.8)
|
|
[2026-06-21 08:37:50,743] [1217460] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
|
[2026-06-21 08:37:50,743] [1217460] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
|
[2026-06-21 08:37:50,743] [1217460] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
|
[2026-06-21 08:37:50,743] [1217460] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
|
[2026-06-21 08:40:02,029] [1217460] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
|
[2026-06-21 08:40:02,029] [1217460] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
|
[2026-06-21 08:40:02,029] [1217460] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
|
[2026-06-21 08:40:02,029] [1217460] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 585ms (priority=1) to avoid split vote (bug-606)
|
|
[2026-06-21 08:40:02,629] [1217460] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=1, n_peers=3) — staying CTRL_STANDBY, no persist
|
|
[2026-06-21 08:40:03,270] [1217460] [ INFO] [ha_on_heartbeat_received] HA: primary heartbeat arrived mid-PreVote — cancelling the round and re-attaching as CTRL_STANDBY (leader alive, no term change) (Req 4.2)
|
|
[2026-06-21 08:40:03,273] [1217460] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
|
[2026-06-21 08:40:03,273] [1217460] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
|
[2026-06-21 08:40:03,273] [1217460] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
|
[2026-06-21 08:40:03,273] [1217460] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=9)
|
|
[2026-06-21 08:46:07,141] [1227110] [ INFO] [ main] My name is ctrl-2
|
|
[2026-06-21 08:46:07,141] [1227110] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-2
|
|
[2026-06-21 08:46:07,141] [1227110] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2)
|
|
[2026-06-21 08:46:07,141] [1227110] [ INFO] [ cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin
|
|
[2026-06-21 08:46:07,149] [1227110] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
|
[2026-06-21 08:46:07,149] [1227110] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
|
[2026-06-21 08:46:07,149] [1227110] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
|
[2026-06-21 08:46:07,149] [1227110] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=3)
|
|
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-3, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election
|
|
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
|
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ xce_config_init] xce controller is running
|
|
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
|
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
|
[2026-06-21 08:46:07,650] [1227110] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=22 current_lsn=23; applied_lsn=22 advertised for catch-up
|
|
[2026-06-21 08:46:07,650] [1227110] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
|
[2026-06-21 08:46:07,650] [1227110] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
|
[2026-06-21 08:46:09,319] [1227110] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
|
[2026-06-21 08:46:09,319] [1227110] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=6)
|
|
[2026-06-21 08:47:38,988] [1227110] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
|
[2026-06-21 08:47:38,988] [1227110] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
|
[2026-06-21 08:47:38,988] [1227110] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
|
[2026-06-21 08:47:38,988] [1227110] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 556ms (priority=1) to avoid split vote (bug-606)
|
|
[2026-06-21 08:47:39,276] [1227110] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
|
[2026-06-21 08:47:39,276] [1227110] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
|
[2026-06-21 08:47:39,277] [1227110] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
|
[2026-06-21 08:47:39,277] [1227110] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=8)
|
|
[2026-06-21 08:47:39,588] [1227110] [ INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606)
|
|
[2026-06-21 08:47:51,992] [1227110] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
|
[2026-06-21 08:47:51,992] [1227110] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
|
[2026-06-21 08:47:51,992] [1227110] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
|
[2026-06-21 08:47:51,992] [1227110] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 662ms (priority=1) to avoid split vote (bug-606)
|
|
[2026-06-21 08:47:52,793] [1227110] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=4, n_peers=3) — staying CTRL_STANDBY, no persist
|
|
[2026-06-21 08:47:52,840] [1227110] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 4)
|
|
[2026-06-21 08:47:52,843] [1227110] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
|
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
|
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=44 safe_config_tag=5298030106630493228 (from 1 granted voters)
|
|
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=4, first_boot=0)
|
|
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
|
|
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9991)
|
|
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
|
|
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
|
|
[2026-06-21 08:47:52,850] [1227110] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
|
|
[2026-06-21 08:47:52,850] [1227110] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=45, writable_lsn gate armed
|
|
[2026-06-21 08:47:53,395] [1227110] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=48 < target=49 → streaming [48+1..49]
|
|
[2026-06-21 08:47:53,395] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=49 target_lsn=49
|
|
[2026-06-21 08:47:53,395] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [48+1..49] to standby
|
|
[2026-06-21 08:47:53,595] [1227110] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=49 < target=53 → streaming [49+1..53]
|
|
[2026-06-21 08:47:53,595] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=50 target_lsn=53
|
|
[2026-06-21 08:47:53,596] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [49+1..53] to standby
|
|
[2026-06-21 08:47:59,844] [1228672] [ INFO] [ main] My name is ctrl-2
|
|
[2026-06-21 08:47:59,844] [1228672] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-2
|
|
[2026-06-21 08:47:59,844] [1228672] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2)
|
|
[2026-06-21 08:47:59,845] [1228672] [ INFO] [ cluster_init_phase1] HA: restored term=4 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin
|
|
[2026-06-21 08:47:59,854] [1228672] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
|
[2026-06-21 08:47:59,854] [1228672] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
|
[2026-06-21 08:47:59,854] [1228672] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
|
[2026-06-21 08:47:59,854] [1228672] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=4)
|
|
[2026-06-21 08:48:00,354] [1228672] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-2, voted_term=4, raft_term=4, primary_fd=-1) — skipping own election
|
|
[2026-06-21 08:48:00,354] [1228672] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
|
[2026-06-21 08:48:00,355] [1228672] [ INFO] [ xce_config_init] xce controller is running
|
|
[2026-06-21 08:48:00,355] [1228672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
|
[2026-06-21 08:48:00,355] [1228672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
|
[2026-06-21 08:48:00,357] [1228672] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=53 current_lsn=54; applied_lsn=53 advertised for catch-up
|
|
[2026-06-21 08:48:00,357] [1228672] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
|
[2026-06-21 08:48:00,357] [1228672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
|
[2026-06-21 08:48:04,312] [1228672] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
|
[2026-06-21 08:48:04,312] [1228672] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
|
[2026-06-21 08:48:04,312] [1228672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
|
[2026-06-21 08:48:04,312] [1228672] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|