lei.chen/myxce
1
0
Fork 1
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
myxce/.kiro/specs/ha-commit-durability/repro-851/ctrl-3.log
Rongsong Shen 20e59d0b42 checkpoint: bug-851 root cause + raft TLA+ conformance spec family 
Investigation & fixes (failover committed-job loss):
- root-cause bug-851 on an owned 3-node cluster: chaotic failover causes
  silent committed-prefix divergence (WAL hdr.term always 0; two leaders
  per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION.
- live-validated fixes 848/849/850 (recovery-on-promotion, advertise real
  applied_lsn, persist-layer contiguity guard); add waldump WAL inspector.

Specs (strict raft.tla conformance — library owns the log):
- TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6.
- raft-log-replication, raft-commit-safety, raft-sim-conformance: extend
  src/lib/raft to own log/commit/verification via callbacks (specs generated,
  cross-spec review passed).
- ha-term-safety: controller-side adapter + config marker (R1-R7); design done.
- config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857).
- roadmap + buglog (851/854/857) + .wolf knowledge updated.
2026-06-21 14:02:15 +00:00

213 lines
30 KiB
Text
Raw Blame History

[2026-06-21 08:37:36,665] [1217472] [ INFO] [ main] My name is ctrl-3
[2026-06-21 08:37:36,665] [1217472] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-3
[2026-06-21 08:37:36,666] [1217472] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=2 (name=ctrl-3)
[2026-06-21 08:37:36,666] [1217472] [ INFO] [ cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-3/data/ha-term.bin
[2026-06-21 08:37:36,672] [1217472] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-3/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
[2026-06-21 08:37:36,672] [1217472] [ INFO] [ load_primary_config] No config at '/tmp/xce-deploy/controller/ctrl-3/conf/current' (rc=-2) — will fetch from primary after election
[2026-06-21 08:37:36,672] [1217472] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=2, term=0)
[2026-06-21 08:37:37,672] [1217472] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
[2026-06-21 08:37:37,672] [1217472] [ INFO] [ha_start_election_async] ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only, not starting own election
[2026-06-21 08:37:37,672] [1217472] [ ERROR] [ cluster_init_phase2] cluster_init_phase2: cluster_config not loaded
[2026-06-21 08:37:37,672] [1217472] [ WARN] [ myapp_run] myapp_run: cluster_init_phase2 on standby rejoin failed (rc=-22) — WAL not recovered; falling back to election-path recovery
[2026-06-21 08:37:37,672] [1217472] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
[2026-06-21 08:37:37,672] [1217472] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
[2026-06-21 08:37:47,276] [1217472] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
[2026-06-21 08:37:47,276] [1217472] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
[2026-06-21 08:37:47,276] [1217472] [ INFO] [ha_on_primary_ctrl_fd_rdhup] ha_on_primary_ctrl_fd_rdhup: no local config (tag=0, n_peers=3) — voter only, not starting PreVote/RDHUP re-election; awaiting configured primary (ha-config-bootstrap Req 2.8)
[2026-06-21 08:37:50,743] [1217472] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
[2026-06-21 08:37:50,743] [1217472] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
[2026-06-21 08:37:50,743] [1217472] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
[2026-06-21 08:37:50,743] [1217472] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
[2026-06-21 08:40:01,930] [1217472] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
[2026-06-21 08:40:01,930] [1217472] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
[2026-06-21 08:40:01,930] [1217472] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
[2026-06-21 08:40:01,930] [1217472] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 1150ms (priority=2) to avoid split vote (bug-606)
[2026-06-21 08:40:02,636] [1217472] [ INFO] [ myreq_handler] HA: PreVote DENIED candidate='ctrl-2' term=1 reason=stale_term (my_term=0, floor tag=5298030106630493228 lsn=13)
[2026-06-21 08:40:03,130] [1217472] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=3, n_peers=3) — staying CTRL_STANDBY, no persist
[2026-06-21 08:40:03,176] [1217472] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 3)
[2026-06-21 08:40:03,176] [1217472] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
[2026-06-21 08:40:03,180] [1217472] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=13 safe_config_tag=5298030106630493228 (from 1 granted voters)
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ xce_config_init] xce controller is running
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
[2026-06-21 08:40:03,224] [1217472] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=13 current_lsn=14; applied_lsn=14 advertised for catch-up
[2026-06-21 08:40:03,224] [1217472] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=3, first_boot=1)
[2026-06-21 08:40:03,224] [1217472] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
[2026-06-21 08:40:03,224] [1217472] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 0, localhost:9992)
[2026-06-21 08:40:03,229] [1217472] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
[2026-06-21 08:40:03,230] [1217472] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=14, writable_lsn gate armed
[2026-06-21 08:40:06,333] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 1)
[2026-06-21 08:40:06,333] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
[2026-06-21 08:40:06,374] [1217472] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=15 (committed_tag=5298030106630493228 committed_lsn=14)
[2026-06-21 08:40:33,348] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 10)
[2026-06-21 08:40:33,348] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
[2026-06-21 08:41:03,364] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 20)
[2026-06-21 08:41:03,364] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
[2026-06-21 08:41:33,382] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 30)
[2026-06-21 08:41:33,382] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
[2026-06-21 08:42:03,397] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 40)
[2026-06-21 08:42:03,397] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
[2026-06-21 08:42:33,411] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 50)
[2026-06-21 08:42:33,411] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
[2026-06-21 08:43:03,425] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 60)
[2026-06-21 08:43:03,425] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
[2026-06-21 08:43:33,240] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 70)
[2026-06-21 08:43:33,240] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
[2026-06-21 08:43:45,247] [1217472] [ INFO] [ ha_connect_to_peer] HA: connecting outbound to peer[0] (localhost:9991)
[2026-06-21 08:43:45,250] [1217472] [ INFO] [ on_standby_connected] HA: outbound standby connection to peer[0] established, ctrlhello sent
[2026-06-21 08:43:45,250] [1217472] [ INFO] [ on_standby_connected] HA: repl registered late-joiner standby peer[0] (slot 1, localhost:9991)
[2026-06-21 08:43:48,290] [1217472] [ INFO] [ha_dispatch_repl_action_for_standby] HA repl: standby[0] lsn=13 < committed_lsn=22 → stream WAL batches
[2026-06-21 08:43:48,290] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=14 target_lsn=22
[2026-06-21 08:43:48,290] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [13+1..22] to standby
[2026-06-21 08:43:48,449] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=14 < target=22 → streaming [14+1..22]
[2026-06-21 08:43:48,449] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=15 target_lsn=22
[2026-06-21 08:43:48,449] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [14+1..22] to standby
[2026-06-21 08:43:48,649] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=15 < target=22 → streaming [15+1..22]
[2026-06-21 08:43:48,649] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=16 target_lsn=22
[2026-06-21 08:43:48,649] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [15+1..22] to standby
[2026-06-21 08:43:48,849] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=16 < target=22 → streaming [16+1..22]
[2026-06-21 08:43:48,849] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=17 target_lsn=22
[2026-06-21 08:43:48,850] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [16+1..22] to standby
[2026-06-21 08:43:49,050] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=17 < target=22 → streaming [17+1..22]
[2026-06-21 08:43:49,050] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=18 target_lsn=22
[2026-06-21 08:43:49,050] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [17+1..22] to standby
[2026-06-21 08:43:49,250] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=18 < target=22 → streaming [18+1..22]
[2026-06-21 08:43:49,250] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=19 target_lsn=22
[2026-06-21 08:43:49,251] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [18+1..22] to standby
[2026-06-21 08:43:49,451] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=19 < target=22 → streaming [19+1..22]
[2026-06-21 08:43:49,451] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=20 target_lsn=22
[2026-06-21 08:43:49,451] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [19+1..22] to standby
[2026-06-21 08:43:49,651] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=20 < target=22 → streaming [20+1..22]
[2026-06-21 08:43:49,651] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=21 target_lsn=22
[2026-06-21 08:43:49,651] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [20+1..22] to standby
[2026-06-21 08:43:49,851] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=21 < target=22 → streaming [21+1..22]
[2026-06-21 08:43:49,851] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=22 target_lsn=22
[2026-06-21 08:43:49,852] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [21+1..22] to standby
[2026-06-21 08:43:51,294] [1217472] [ INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=23 (committed_tag=5298030106630493228 committed_lsn=22)
[2026-06-21 08:45:24,291] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 1)
[2026-06-21 08:45:24,292] [1217472] [ INFO] [ ha_connect_to_peer] HA: connecting outbound to peer[0] (localhost:9991)
[2026-06-21 08:45:24,292] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[1] (localhost) not connected — initiating outbound connect (attempt 1)
[2026-06-21 08:45:24,292] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[1] (localhost:9992) failed: -111
[2026-06-21 08:45:24,294] [1217472] [ INFO] [ on_standby_connected] HA: outbound standby connection to peer[0] established, ctrlhello sent
[2026-06-21 08:45:24,294] [1217472] [ INFO] [ on_standby_connected] HA: repl registered late-joiner standby peer[0] (slot 0, localhost:9991)
[2026-06-21 08:45:51,305] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[1] (localhost) not connected — initiating outbound connect (attempt 10)
[2026-06-21 08:45:51,305] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[1] (localhost:9992) failed: -111
[2026-06-21 08:46:07,913] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=35 < target=36 → streaming [35+1..36]
[2026-06-21 08:46:07,913] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=36 target_lsn=36
[2026-06-21 08:46:07,913] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [35+1..36] to standby
[2026-06-21 08:46:09,314] [1217472] [ INFO] [ ha_connect_to_peer] HA: connecting outbound to peer[1] (localhost:9992)
[2026-06-21 08:46:09,319] [1217472] [ INFO] [ on_standby_connected] HA: outbound standby connection to peer[1] established, ctrlhello sent
[2026-06-21 08:46:09,319] [1217472] [ INFO] [ on_standby_connected] HA: repl registered late-joiner standby peer[1] (slot 2, localhost:9992)
[2026-06-21 08:46:12,358] [1217472] [ INFO] [ha_dispatch_repl_action_for_standby] HA repl: standby[1] lsn=22 < committed_lsn=44 → stream WAL batches
[2026-06-21 08:46:12,358] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=23 target_lsn=44
[2026-06-21 08:46:12,358] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [22+1..44] to standby
[2026-06-21 08:46:12,516] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=23 < target=44 → streaming [23+1..44]
[2026-06-21 08:46:12,516] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=24 target_lsn=44
[2026-06-21 08:46:12,516] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [23+1..44] to standby
[2026-06-21 08:46:12,716] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=24 < target=44 → streaming [24+1..44]
[2026-06-21 08:46:12,716] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=25 target_lsn=44
[2026-06-21 08:46:12,717] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [24+1..44] to standby
[2026-06-21 08:46:12,917] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=25 < target=44 → streaming [25+1..44]
[2026-06-21 08:46:12,917] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=26 target_lsn=44
[2026-06-21 08:46:12,917] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [25+1..44] to standby
[2026-06-21 08:46:13,117] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=26 < target=44 → streaming [26+1..44]
[2026-06-21 08:46:13,118] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=27 target_lsn=44
[2026-06-21 08:46:13,118] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [26+1..44] to standby
[2026-06-21 08:46:13,318] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=27 < target=44 → streaming [27+1..44]
[2026-06-21 08:46:13,318] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=28 target_lsn=44
[2026-06-21 08:46:13,319] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [27+1..44] to standby
[2026-06-21 08:46:13,519] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=28 < target=44 → streaming [28+1..44]
[2026-06-21 08:46:13,519] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=29 target_lsn=44
[2026-06-21 08:46:13,519] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [28+1..44] to standby
[2026-06-21 08:46:13,719] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=29 < target=44 → streaming [29+1..44]
[2026-06-21 08:46:13,719] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=30 target_lsn=44
[2026-06-21 08:46:13,720] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [29+1..44] to standby
[2026-06-21 08:46:13,920] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=30 < target=44 → streaming [30+1..44]
[2026-06-21 08:46:13,920] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=31 target_lsn=44
[2026-06-21 08:46:13,920] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [30+1..44] to standby
[2026-06-21 08:46:14,121] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=31 < target=44 → streaming [31+1..44]
[2026-06-21 08:46:14,121] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=32 target_lsn=44
[2026-06-21 08:46:14,121] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [31+1..44] to standby
[2026-06-21 08:46:14,321] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=32 < target=44 → streaming [32+1..44]
[2026-06-21 08:46:14,321] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=33 target_lsn=44
[2026-06-21 08:46:14,321] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [32+1..44] to standby
[2026-06-21 08:46:14,522] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=33 < target=44 → streaming [33+1..44]
[2026-06-21 08:46:14,522] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=34 target_lsn=44
[2026-06-21 08:46:14,522] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [33+1..44] to standby
[2026-06-21 08:46:14,722] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=34 < target=44 → streaming [34+1..44]
[2026-06-21 08:46:14,722] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=35 target_lsn=44
[2026-06-21 08:46:14,722] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [34+1..44] to standby
[2026-06-21 08:46:14,922] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=35 < target=44 → streaming [35+1..44]
[2026-06-21 08:46:14,922] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=36 target_lsn=44
[2026-06-21 08:46:14,923] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [35+1..44] to standby
[2026-06-21 08:46:15,123] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=36 < target=44 → streaming [36+1..44]
[2026-06-21 08:46:15,123] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=37 target_lsn=44
[2026-06-21 08:46:15,123] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [36+1..44] to standby
[2026-06-21 08:46:15,323] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=37 < target=44 → streaming [37+1..44]
[2026-06-21 08:46:15,323] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=38 target_lsn=44
[2026-06-21 08:46:15,323] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [37+1..44] to standby
[2026-06-21 08:46:15,565] [1217472] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=45 (committed_tag=5298030106630493228 committed_lsn=44)
[2026-06-21 08:46:15,724] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=38 < target=44 → streaming [38+1..44]
[2026-06-21 08:46:15,724] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=39 target_lsn=44
[2026-06-21 08:46:15,724] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [38+1..44] to standby
[2026-06-21 08:46:15,924] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=39 < target=44 → streaming [39+1..44]
[2026-06-21 08:46:15,924] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=40 target_lsn=44
[2026-06-21 08:46:15,924] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [39+1..44] to standby
[2026-06-21 08:46:16,124] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=40 < target=44 → streaming [40+1..44]
[2026-06-21 08:46:16,124] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=41 target_lsn=44
[2026-06-21 08:46:16,125] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [40+1..44] to standby
[2026-06-21 08:46:16,325] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=41 < target=44 → streaming [41+1..44]
[2026-06-21 08:46:16,325] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=42 target_lsn=44
[2026-06-21 08:46:16,325] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [41+1..44] to standby
[2026-06-21 08:46:16,525] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=42 < target=44 → streaming [42+1..44]
[2026-06-21 08:46:16,525] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=43 target_lsn=44
[2026-06-21 08:46:16,526] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [42+1..44] to standby
[2026-06-21 08:46:16,726] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=43 < target=44 → streaming [43+1..44]
[2026-06-21 08:46:16,726] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=44 target_lsn=44
[2026-06-21 08:46:16,726] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [43+1..44] to standby
[2026-06-21 08:47:29,557] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=45 < target=48 → streaming [45+1..48]
[2026-06-21 08:47:29,557] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=46 target_lsn=48
[2026-06-21 08:47:29,558] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [45+1..48] to standby
[2026-06-21 08:47:29,758] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=46 < target=51 → streaming [46+1..51]
[2026-06-21 08:47:29,758] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=47 target_lsn=51
[2026-06-21 08:47:29,758] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [46+1..51] to standby
[2026-06-21 08:47:36,064] [1228337] [ INFO] [ main] My name is ctrl-3
[2026-06-21 08:47:36,064] [1228337] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-3
[2026-06-21 08:47:36,065] [1228337] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=2 (name=ctrl-3)
[2026-06-21 08:47:36,065] [1228337] [ INFO] [ cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-3/data/ha-term.bin
[2026-06-21 08:47:36,071] [1228337] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-3/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
[2026-06-21 08:47:36,072] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
[2026-06-21 08:47:36,072] [1228337] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
[2026-06-21 08:47:36,072] [1228337] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=2, term=3)
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-3, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ xce_config_init] xce controller is running
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
[2026-06-21 08:47:37,074] [1228337] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=52 current_lsn=53; applied_lsn=52 advertised for catch-up
[2026-06-21 08:47:37,074] [1228337] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
[2026-06-21 08:47:37,074] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
[2026-06-21 08:47:39,277] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
[2026-06-21 08:47:39,277] [1228337] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
[2026-06-21 08:47:39,277] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
[2026-06-21 08:47:39,277] [1228337] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
[2026-06-21 08:47:52,078] [1228337] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
[2026-06-21 08:47:52,078] [1228337] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
[2026-06-21 08:47:52,078] [1228337] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
[2026-06-21 08:47:52,078] [1228337] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 1088ms (priority=2) to avoid split vote (bug-606)
[2026-06-21 08:47:52,855] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
[2026-06-21 08:47:52,855] [1228337] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
[2026-06-21 08:47:52,855] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
[2026-06-21 08:47:52,855] [1228337] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
[2026-06-21 08:47:53,279] [1228337] [ INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606)
[2026-06-21 08:48:04,083] [1228337] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
[2026-06-21 08:48:04,083] [1228337] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
[2026-06-21 08:48:04,083] [1228337] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
[2026-06-21 08:48:04,083] [1228337] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 1092ms (priority=2) to avoid split vote (bug-606)
[2026-06-21 08:48:04,312] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
[2026-06-21 08:48:04,312] [1228337] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
[2026-06-21 08:48:04,312] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
[2026-06-21 08:48:04,312] [1228337] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
[2026-06-21 08:48:05,284] [1228337] [ INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606)
Reference in a new issue View git blame Copy permalink