lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/ha-commit-durability/repro-851/ctrl-1.log
Rongsong Shen 20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family 
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00

164 lines
22 KiB
Text
Raw Blame History

[2026-06-21 08:37:35,663] [1217413] [ INFO] [ main] My name is ctrl-1
[2026-06-21 08:37:35,663] [1217413] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-1
[2026-06-21 08:37:35,663] [1217413] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1)
[2026-06-21 08:37:35,663] [1217413] [ INFO] [ cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
[2026-06-21 08:37:35,671] [1217413] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
[2026-06-21 08:37:35,671] [1217413] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
[2026-06-21 08:37:35,671] [1217413] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
[2026-06-21 08:37:35,671] [1217413] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=0)
[2026-06-21 08:37:35,671] [1217413] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
[2026-06-21 08:37:35,674] [1217413] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
[2026-06-21 08:37:40,873] [1217413] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ xce_config_init] xce controller is running
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=0 current_lsn=1; applied_lsn=0 advertised for catch-up
[2026-06-21 08:37:40,874] [1217413] [ INFO] [raft_cb_become_follower] HA FSM (raft): → CTRL_STANDBY (term=1, first_boot=1)
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
[2026-06-21 08:37:50,478] [1217413] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
[2026-06-21 08:37:50,479] [1217413] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
[2026-06-21 08:37:50,479] [1217413] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
[2026-06-21 08:37:50,479] [1217413] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 8ms (priority=0) to avoid split vote (bug-606)
[2026-06-21 08:37:50,679] [1217413] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=2, n_peers=3) — staying CTRL_STANDBY, no persist
[2026-06-21 08:37:50,728] [1217413] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 2)
[2026-06-21 08:37:50,728] [1217413] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=0 safe_config_tag=5298030106630493228 (from 1 granted voters)
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=2, first_boot=0)
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992)
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
[2026-06-21 08:37:50,738] [1217413] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
[2026-06-21 08:37:50,738] [1217413] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=1, writable_lsn gate armed
[2026-06-21 08:37:53,924] [1217413] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=2 (committed_tag=5298030106630493228 committed_lsn=1)
[2026-06-21 08:37:53,925] [1217413] [ INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=2 (committed_tag=5298030106630493228 committed_lsn=1)
[2026-06-21 08:39:27,321] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=2 < target=5 → streaming [2+1..5]
[2026-06-21 08:39:27,321] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=3 target_lsn=5
[2026-06-21 08:39:27,322] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [2+1..5] to standby
[2026-06-21 08:39:27,522] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=3 < target=7 → streaming [3+1..7]
[2026-06-21 08:39:27,522] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=4 target_lsn=7
[2026-06-21 08:39:27,522] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [3+1..7] to standby
[2026-06-21 08:39:27,722] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=7 < target=11 → streaming [7+1..11]
[2026-06-21 08:39:27,722] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=8 target_lsn=11
[2026-06-21 08:39:27,722] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [7+1..11] to standby
[2026-06-21 08:39:27,922] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=4 < target=13 → streaming [4+1..13]
[2026-06-21 08:39:27,922] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=5 target_lsn=13
[2026-06-21 08:39:27,923] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [4+1..13] to standby
[2026-06-21 08:39:28,123] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=8 < target=13 → streaming [8+1..13]
[2026-06-21 08:39:28,123] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=9 target_lsn=13
[2026-06-21 08:39:28,123] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [8+1..13] to standby
[2026-06-21 08:39:28,323] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=5 < target=13 → streaming [5+1..13]
[2026-06-21 08:39:28,323] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=6 target_lsn=13
[2026-06-21 08:39:28,323] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [5+1..13] to standby
[2026-06-21 08:39:28,523] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=9 < target=13 → streaming [9+1..13]
[2026-06-21 08:39:28,523] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=10 target_lsn=13
[2026-06-21 08:39:28,524] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [9+1..13] to standby
[2026-06-21 08:39:28,724] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=6 < target=13 → streaming [6+1..13]
[2026-06-21 08:39:28,724] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=7 target_lsn=13
[2026-06-21 08:39:28,724] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [6+1..13] to standby
[2026-06-21 08:39:28,924] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=10 < target=13 → streaming [10+1..13]
[2026-06-21 08:39:28,924] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=11 target_lsn=13
[2026-06-21 08:39:28,924] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [10+1..13] to standby
[2026-06-21 08:39:29,125] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=7 < target=13 → streaming [7+1..13]
[2026-06-21 08:39:29,125] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=8 target_lsn=13
[2026-06-21 08:39:29,125] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [7+1..13] to standby
[2026-06-21 08:39:29,325] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=11 < target=13 → streaming [11+1..13]
[2026-06-21 08:39:29,325] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=12 target_lsn=13
[2026-06-21 08:39:29,325] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [11+1..13] to standby
[2026-06-21 08:39:29,525] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=8 < target=13 → streaming [8+1..13]
[2026-06-21 08:39:29,525] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=9 target_lsn=13
[2026-06-21 08:39:29,526] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [8+1..13] to standby
[2026-06-21 08:39:29,726] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=12 < target=13 → streaming [12+1..13]
[2026-06-21 08:39:29,726] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=13 target_lsn=13
[2026-06-21 08:39:29,726] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [12+1..13] to standby
[2026-06-21 08:39:29,926] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=9 < target=13 → streaming [9+1..13]
[2026-06-21 08:39:29,926] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=10 target_lsn=13
[2026-06-21 08:39:29,926] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [9+1..13] to standby
[2026-06-21 08:39:30,326] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=10 < target=13 → streaming [10+1..13]
[2026-06-21 08:39:30,326] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=11 target_lsn=13
[2026-06-21 08:39:30,327] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [10+1..13] to standby
[2026-06-21 08:39:30,527] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=11 < target=13 → streaming [11+1..13]
[2026-06-21 08:39:30,527] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=12 target_lsn=13
[2026-06-21 08:39:30,527] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [11+1..13] to standby
[2026-06-21 08:39:30,727] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=12 < target=13 → streaming [12+1..13]
[2026-06-21 08:39:30,727] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=13 target_lsn=13
[2026-06-21 08:39:30,727] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [12+1..13] to standby
[2026-06-21 08:43:44,321] [1224672] [ INFO] [ main] My name is ctrl-1
[2026-06-21 08:43:44,321] [1224672] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-1
[2026-06-21 08:43:44,321] [1224672] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1)
[2026-06-21 08:43:44,321] [1224672] [ INFO] [ cluster_init_phase1] HA: restored term=2 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
[2026-06-21 08:43:44,327] [1224672] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=2)
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-1, voted_term=2, raft_term=2, primary_fd=-1) — skipping own election
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ xce_config_init] xce controller is running
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=13 current_lsn=14; applied_lsn=13 advertised for catch-up
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
[2026-06-21 08:43:45,250] [1224672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
[2026-06-21 08:43:45,250] [1224672] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=6)
[2026-06-21 08:47:39,017] [1224672] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
[2026-06-21 08:47:39,017] [1224672] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
[2026-06-21 08:47:39,017] [1224672] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
[2026-06-21 08:47:39,017] [1224672] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 122ms (priority=0) to avoid split vote (bug-606)
[2026-06-21 08:47:39,217] [1224672] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=3, n_peers=3) — staying CTRL_STANDBY, no persist
[2026-06-21 08:47:39,262] [1224672] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 3)
[2026-06-21 08:47:39,265] [1224672] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=44 safe_config_tag=5298030106630493228 (from 1 granted voters)
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=3, first_boot=0)
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992)
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
[2026-06-21 08:47:39,272] [1224672] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
[2026-06-21 08:47:39,272] [1224672] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=45, writable_lsn gate armed
[2026-06-21 08:47:41,419] [1224672] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=46 < target=48 → streaming [46+1..48]
[2026-06-21 08:47:41,419] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=47 target_lsn=48
[2026-06-21 08:47:41,420] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [46+1..48] to standby
[2026-06-21 08:47:41,620] [1224672] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=48 < target=53 → streaming [48+1..53]
[2026-06-21 08:47:41,620] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=49 target_lsn=53
[2026-06-21 08:47:41,620] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [48+1..53] to standby
[2026-06-21 08:47:47,832] [1228504] [ INFO] [ main] My name is ctrl-1
[2026-06-21 08:47:47,832] [1228504] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-1
[2026-06-21 08:47:47,832] [1228504] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1)
[2026-06-21 08:47:47,832] [1228504] [ INFO] [ cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
[2026-06-21 08:47:47,839] [1228504] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=3)
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-1, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ xce_config_init] xce controller is running
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
[2026-06-21 08:47:47,841] [1228504] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=53 current_lsn=54; applied_lsn=53 advertised for catch-up
[2026-06-21 08:47:47,841] [1228504] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
[2026-06-21 08:47:47,841] [1228504] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
[2026-06-21 08:47:52,855] [1228504] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
[2026-06-21 08:47:52,855] [1228504] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
[2026-06-21 08:47:52,855] [1228504] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
[2026-06-21 08:47:52,855] [1228504] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=8)
[2026-06-21 08:48:04,047] [1228504] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
[2026-06-21 08:48:04,047] [1228504] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
[2026-06-21 08:48:04,047] [1228504] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
[2026-06-21 08:48:04,047] [1228504] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 159ms (priority=0) to avoid split vote (bug-606)
[2026-06-21 08:48:04,248] [1228504] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=4, n_peers=3) — staying CTRL_STANDBY, no persist
[2026-06-21 08:48:04,295] [1228504] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 4)
[2026-06-21 08:48:04,295] [1228504] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=53 safe_config_tag=5298030106630493228 (from 1 granted voters)
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=4, first_boot=0)
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992)
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
[2026-06-21 08:48:04,305] [1228504] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
[2026-06-21 08:48:04,307] [1228504] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=54, writable_lsn gate armed
[2026-06-21 08:48:07,493] [1228504] [ INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=55 (committed_tag=5298030106630493228 committed_lsn=54)
[2026-06-21 08:48:07,493] [1228504] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=55 (committed_tag=5298030106630493228 committed_lsn=54)
Reference in a new issue View git blame Copy permalink