checkpoint: bug-851 root cause + raft TLA+ conformance spec family
Investigation & fixes (failover committed-job loss): - root-cause bug-851 on an owned 3-node cluster: chaotic failover causes silent committed-prefix divergence (WAL hdr.term always 0; two leaders per term; no (term,lsn) reconcile). Evidence in repro-851/ + INVESTIGATION. - live-validated fixes 848/849/850 (recovery-on-promotion, advertise real applied_lsn, persist-layer contiguity guard); add waldump WAL inspector. Specs (strict raft.tla conformance — library owns the log): - TLA-gap-analysis.md: impl vs raft/raft.tla, gaps G1-G6. - raft-log-replication, raft-commit-safety, raft-sim-conformance: extend src/lib/raft to own log/commit/verification via callbacks (specs generated, cross-spec review passed). - ha-term-safety: controller-side adapter + config marker (R1-R7); design done. - config-upgrade-snapshot: R8 descoped here (deps entity-bearing-snapshots, bug-857). - roadmap + buglog (851/854/857) + .wolf knowledge updated.
This commit is contained in:
parent
7d70820aac
commit
20e59d0b42
85 changed files with 10525 additions and 1993 deletions
164
.kiro/specs/ha-commit-durability/repro-851/ctrl-1.log
Normal file
164
.kiro/specs/ha-commit-durability/repro-851/ctrl-1.log
Normal file
|
|
@ -0,0 +1,164 @@
|
|||
[2026-06-21 08:37:35,663] [1217413] [ INFO] [ main] My name is ctrl-1
|
||||
[2026-06-21 08:37:35,663] [1217413] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-1
|
||||
[2026-06-21 08:37:35,663] [1217413] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1)
|
||||
[2026-06-21 08:37:35,663] [1217413] [ INFO] [ cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
|
||||
[2026-06-21 08:37:35,671] [1217413] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:37:35,671] [1217413] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:37:35,671] [1217413] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:37:35,671] [1217413] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=0)
|
||||
[2026-06-21 08:37:35,671] [1217413] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:37:35,674] [1217413] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:37:40,873] [1217413] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=0 current_lsn=1; applied_lsn=0 advertised for catch-up
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [raft_cb_become_follower] HA FSM (raft): → CTRL_STANDBY (term=1, first_boot=1)
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:37:50,478] [1217413] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
||||
[2026-06-21 08:37:50,479] [1217413] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:37:50,479] [1217413] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:37:50,479] [1217413] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 8ms (priority=0) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:37:50,679] [1217413] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=2, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:37:50,728] [1217413] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 2)
|
||||
[2026-06-21 08:37:50,728] [1217413] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=0 safe_config_tag=5298030106630493228 (from 1 granted voters)
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=2, first_boot=0)
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992)
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
|
||||
[2026-06-21 08:37:50,738] [1217413] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
|
||||
[2026-06-21 08:37:50,738] [1217413] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=1, writable_lsn gate armed
|
||||
[2026-06-21 08:37:53,924] [1217413] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=2 (committed_tag=5298030106630493228 committed_lsn=1)
|
||||
[2026-06-21 08:37:53,925] [1217413] [ INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=2 (committed_tag=5298030106630493228 committed_lsn=1)
|
||||
[2026-06-21 08:39:27,321] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=2 < target=5 → streaming [2+1..5]
|
||||
[2026-06-21 08:39:27,321] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=3 target_lsn=5
|
||||
[2026-06-21 08:39:27,322] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [2+1..5] to standby
|
||||
[2026-06-21 08:39:27,522] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=3 < target=7 → streaming [3+1..7]
|
||||
[2026-06-21 08:39:27,522] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=4 target_lsn=7
|
||||
[2026-06-21 08:39:27,522] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [3+1..7] to standby
|
||||
[2026-06-21 08:39:27,722] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=7 < target=11 → streaming [7+1..11]
|
||||
[2026-06-21 08:39:27,722] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=8 target_lsn=11
|
||||
[2026-06-21 08:39:27,722] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [7+1..11] to standby
|
||||
[2026-06-21 08:39:27,922] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=4 < target=13 → streaming [4+1..13]
|
||||
[2026-06-21 08:39:27,922] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=5 target_lsn=13
|
||||
[2026-06-21 08:39:27,923] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [4+1..13] to standby
|
||||
[2026-06-21 08:39:28,123] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=8 < target=13 → streaming [8+1..13]
|
||||
[2026-06-21 08:39:28,123] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=9 target_lsn=13
|
||||
[2026-06-21 08:39:28,123] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [8+1..13] to standby
|
||||
[2026-06-21 08:39:28,323] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=5 < target=13 → streaming [5+1..13]
|
||||
[2026-06-21 08:39:28,323] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=6 target_lsn=13
|
||||
[2026-06-21 08:39:28,323] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [5+1..13] to standby
|
||||
[2026-06-21 08:39:28,523] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=9 < target=13 → streaming [9+1..13]
|
||||
[2026-06-21 08:39:28,523] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=10 target_lsn=13
|
||||
[2026-06-21 08:39:28,524] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [9+1..13] to standby
|
||||
[2026-06-21 08:39:28,724] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=6 < target=13 → streaming [6+1..13]
|
||||
[2026-06-21 08:39:28,724] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=7 target_lsn=13
|
||||
[2026-06-21 08:39:28,724] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [6+1..13] to standby
|
||||
[2026-06-21 08:39:28,924] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=10 < target=13 → streaming [10+1..13]
|
||||
[2026-06-21 08:39:28,924] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=11 target_lsn=13
|
||||
[2026-06-21 08:39:28,924] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [10+1..13] to standby
|
||||
[2026-06-21 08:39:29,125] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=7 < target=13 → streaming [7+1..13]
|
||||
[2026-06-21 08:39:29,125] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=8 target_lsn=13
|
||||
[2026-06-21 08:39:29,125] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [7+1..13] to standby
|
||||
[2026-06-21 08:39:29,325] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=11 < target=13 → streaming [11+1..13]
|
||||
[2026-06-21 08:39:29,325] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=12 target_lsn=13
|
||||
[2026-06-21 08:39:29,325] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [11+1..13] to standby
|
||||
[2026-06-21 08:39:29,525] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=8 < target=13 → streaming [8+1..13]
|
||||
[2026-06-21 08:39:29,525] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=9 target_lsn=13
|
||||
[2026-06-21 08:39:29,526] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [8+1..13] to standby
|
||||
[2026-06-21 08:39:29,726] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=12 < target=13 → streaming [12+1..13]
|
||||
[2026-06-21 08:39:29,726] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=13 target_lsn=13
|
||||
[2026-06-21 08:39:29,726] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [12+1..13] to standby
|
||||
[2026-06-21 08:39:29,926] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=9 < target=13 → streaming [9+1..13]
|
||||
[2026-06-21 08:39:29,926] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=10 target_lsn=13
|
||||
[2026-06-21 08:39:29,926] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [9+1..13] to standby
|
||||
[2026-06-21 08:39:30,326] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=10 < target=13 → streaming [10+1..13]
|
||||
[2026-06-21 08:39:30,326] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=11 target_lsn=13
|
||||
[2026-06-21 08:39:30,327] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [10+1..13] to standby
|
||||
[2026-06-21 08:39:30,527] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=11 < target=13 → streaming [11+1..13]
|
||||
[2026-06-21 08:39:30,527] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=12 target_lsn=13
|
||||
[2026-06-21 08:39:30,527] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [11+1..13] to standby
|
||||
[2026-06-21 08:39:30,727] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=12 < target=13 → streaming [12+1..13]
|
||||
[2026-06-21 08:39:30,727] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=13 target_lsn=13
|
||||
[2026-06-21 08:39:30,727] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [12+1..13] to standby
|
||||
[2026-06-21 08:43:44,321] [1224672] [ INFO] [ main] My name is ctrl-1
|
||||
[2026-06-21 08:43:44,321] [1224672] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-1
|
||||
[2026-06-21 08:43:44,321] [1224672] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1)
|
||||
[2026-06-21 08:43:44,321] [1224672] [ INFO] [ cluster_init_phase1] HA: restored term=2 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
|
||||
[2026-06-21 08:43:44,327] [1224672] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=2)
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-1, voted_term=2, raft_term=2, primary_fd=-1) — skipping own election
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=13 current_lsn=14; applied_lsn=13 advertised for catch-up
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:43:45,250] [1224672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:43:45,250] [1224672] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=6)
|
||||
[2026-06-21 08:47:39,017] [1224672] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
||||
[2026-06-21 08:47:39,017] [1224672] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:47:39,017] [1224672] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:47:39,017] [1224672] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 122ms (priority=0) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:47:39,217] [1224672] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=3, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:47:39,262] [1224672] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 3)
|
||||
[2026-06-21 08:47:39,265] [1224672] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=44 safe_config_tag=5298030106630493228 (from 1 granted voters)
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=3, first_boot=0)
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992)
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
|
||||
[2026-06-21 08:47:39,272] [1224672] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
|
||||
[2026-06-21 08:47:39,272] [1224672] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=45, writable_lsn gate armed
|
||||
[2026-06-21 08:47:41,419] [1224672] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=46 < target=48 → streaming [46+1..48]
|
||||
[2026-06-21 08:47:41,419] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=47 target_lsn=48
|
||||
[2026-06-21 08:47:41,420] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [46+1..48] to standby
|
||||
[2026-06-21 08:47:41,620] [1224672] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=48 < target=53 → streaming [48+1..53]
|
||||
[2026-06-21 08:47:41,620] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=49 target_lsn=53
|
||||
[2026-06-21 08:47:41,620] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [48+1..53] to standby
|
||||
[2026-06-21 08:47:47,832] [1228504] [ INFO] [ main] My name is ctrl-1
|
||||
[2026-06-21 08:47:47,832] [1228504] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-1
|
||||
[2026-06-21 08:47:47,832] [1228504] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1)
|
||||
[2026-06-21 08:47:47,832] [1228504] [ INFO] [ cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
|
||||
[2026-06-21 08:47:47,839] [1228504] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=3)
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-1, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:47:47,841] [1228504] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=53 current_lsn=54; applied_lsn=53 advertised for catch-up
|
||||
[2026-06-21 08:47:47,841] [1228504] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:47:47,841] [1228504] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:52,855] [1228504] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:52,855] [1228504] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:47:52,855] [1228504] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:52,855] [1228504] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=8)
|
||||
[2026-06-21 08:48:04,047] [1228504] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:48:04,047] [1228504] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:48:04,047] [1228504] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:48:04,047] [1228504] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 159ms (priority=0) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:48:04,248] [1228504] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=4, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:48:04,295] [1228504] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 4)
|
||||
[2026-06-21 08:48:04,295] [1228504] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=53 safe_config_tag=5298030106630493228 (from 1 granted voters)
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=4, first_boot=0)
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992)
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
|
||||
[2026-06-21 08:48:04,305] [1228504] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
|
||||
[2026-06-21 08:48:04,307] [1228504] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=54, writable_lsn gate armed
|
||||
[2026-06-21 08:48:07,493] [1228504] [ INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=55 (committed_tag=5298030106630493228 committed_lsn=54)
|
||||
[2026-06-21 08:48:07,493] [1228504] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=55 (committed_tag=5298030106630493228 committed_lsn=54)
|
||||
99
.kiro/specs/ha-commit-durability/repro-851/ctrl-2.log
Normal file
99
.kiro/specs/ha-commit-durability/repro-851/ctrl-2.log
Normal file
|
|
@ -0,0 +1,99 @@
|
|||
[2026-06-21 08:37:36,163] [1217460] [ INFO] [ main] My name is ctrl-2
|
||||
[2026-06-21 08:37:36,164] [1217460] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-2
|
||||
[2026-06-21 08:37:36,164] [1217460] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2)
|
||||
[2026-06-21 08:37:36,164] [1217460] [ INFO] [ cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin
|
||||
[2026-06-21 08:37:36,173] [1217460] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:37:36,173] [1217460] [ INFO] [ load_primary_config] No config at '/tmp/xce-deploy/controller/ctrl-2/conf/current' (rc=-2) — will fetch from primary after election
|
||||
[2026-06-21 08:37:36,173] [1217460] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=0)
|
||||
[2026-06-21 08:37:36,673] [1217460] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:37:36,673] [1217460] [ INFO] [ha_start_election_async] ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only, not starting own election
|
||||
[2026-06-21 08:37:36,673] [1217460] [ ERROR] [ cluster_init_phase2] cluster_init_phase2: cluster_config not loaded
|
||||
[2026-06-21 08:37:36,673] [1217460] [ WARN] [ myapp_run] myapp_run: cluster_init_phase2 on standby rejoin failed (rc=-22) — WAL not recovered; falling back to election-path recovery
|
||||
[2026-06-21 08:37:36,673] [1217460] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:37:36,673] [1217460] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:37:46,177] [1217460] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
||||
[2026-06-21 08:37:46,177] [1217460] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:37:46,177] [1217460] [ INFO] [ha_on_primary_ctrl_fd_rdhup] ha_on_primary_ctrl_fd_rdhup: no local config (tag=0, n_peers=3) — voter only, not starting PreVote/RDHUP re-election; awaiting configured primary (ha-config-bootstrap Req 2.8)
|
||||
[2026-06-21 08:37:50,743] [1217460] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:37:50,743] [1217460] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:37:50,743] [1217460] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:37:50,743] [1217460] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
[2026-06-21 08:40:02,029] [1217460] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:40:02,029] [1217460] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:40:02,029] [1217460] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:40:02,029] [1217460] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 585ms (priority=1) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:40:02,629] [1217460] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=1, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:40:03,270] [1217460] [ INFO] [ha_on_heartbeat_received] HA: primary heartbeat arrived mid-PreVote — cancelling the round and re-attaching as CTRL_STANDBY (leader alive, no term change) (Req 4.2)
|
||||
[2026-06-21 08:40:03,273] [1217460] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:40:03,273] [1217460] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:40:03,273] [1217460] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:40:03,273] [1217460] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=9)
|
||||
[2026-06-21 08:46:07,141] [1227110] [ INFO] [ main] My name is ctrl-2
|
||||
[2026-06-21 08:46:07,141] [1227110] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-2
|
||||
[2026-06-21 08:46:07,141] [1227110] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2)
|
||||
[2026-06-21 08:46:07,141] [1227110] [ INFO] [ cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin
|
||||
[2026-06-21 08:46:07,149] [1227110] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:46:07,149] [1227110] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:46:07,149] [1227110] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:46:07,149] [1227110] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=3)
|
||||
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-3, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election
|
||||
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:46:07,650] [1227110] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=22 current_lsn=23; applied_lsn=22 advertised for catch-up
|
||||
[2026-06-21 08:46:07,650] [1227110] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:46:07,650] [1227110] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:46:09,319] [1227110] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:46:09,319] [1227110] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=6)
|
||||
[2026-06-21 08:47:38,988] [1227110] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
||||
[2026-06-21 08:47:38,988] [1227110] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:47:38,988] [1227110] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:47:38,988] [1227110] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 556ms (priority=1) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:47:39,276] [1227110] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:39,276] [1227110] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:47:39,277] [1227110] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:39,277] [1227110] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=8)
|
||||
[2026-06-21 08:47:39,588] [1227110] [ INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606)
|
||||
[2026-06-21 08:47:51,992] [1227110] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:47:51,992] [1227110] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:47:51,992] [1227110] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:47:51,992] [1227110] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 662ms (priority=1) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:47:52,793] [1227110] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=4, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:47:52,840] [1227110] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 4)
|
||||
[2026-06-21 08:47:52,843] [1227110] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=44 safe_config_tag=5298030106630493228 (from 1 granted voters)
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=4, first_boot=0)
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9991)
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
|
||||
[2026-06-21 08:47:52,850] [1227110] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
|
||||
[2026-06-21 08:47:52,850] [1227110] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=45, writable_lsn gate armed
|
||||
[2026-06-21 08:47:53,395] [1227110] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=48 < target=49 → streaming [48+1..49]
|
||||
[2026-06-21 08:47:53,395] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=49 target_lsn=49
|
||||
[2026-06-21 08:47:53,395] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [48+1..49] to standby
|
||||
[2026-06-21 08:47:53,595] [1227110] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=49 < target=53 → streaming [49+1..53]
|
||||
[2026-06-21 08:47:53,595] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=50 target_lsn=53
|
||||
[2026-06-21 08:47:53,596] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [49+1..53] to standby
|
||||
[2026-06-21 08:47:59,844] [1228672] [ INFO] [ main] My name is ctrl-2
|
||||
[2026-06-21 08:47:59,844] [1228672] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-2
|
||||
[2026-06-21 08:47:59,844] [1228672] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2)
|
||||
[2026-06-21 08:47:59,845] [1228672] [ INFO] [ cluster_init_phase1] HA: restored term=4 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin
|
||||
[2026-06-21 08:47:59,854] [1228672] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:47:59,854] [1228672] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:59,854] [1228672] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:47:59,854] [1228672] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=4)
|
||||
[2026-06-21 08:48:00,354] [1228672] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-2, voted_term=4, raft_term=4, primary_fd=-1) — skipping own election
|
||||
[2026-06-21 08:48:00,354] [1228672] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:48:00,355] [1228672] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:48:00,355] [1228672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:48:00,355] [1228672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:48:00,357] [1228672] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=53 current_lsn=54; applied_lsn=53 advertised for catch-up
|
||||
[2026-06-21 08:48:00,357] [1228672] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:48:00,357] [1228672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:48:04,312] [1228672] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:48:04,312] [1228672] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:48:04,312] [1228672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:48:04,312] [1228672] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
213
.kiro/specs/ha-commit-durability/repro-851/ctrl-3.log
Normal file
213
.kiro/specs/ha-commit-durability/repro-851/ctrl-3.log
Normal file
|
|
@ -0,0 +1,213 @@
|
|||
[2026-06-21 08:37:36,665] [1217472] [ INFO] [ main] My name is ctrl-3
|
||||
[2026-06-21 08:37:36,665] [1217472] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-3
|
||||
[2026-06-21 08:37:36,666] [1217472] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=2 (name=ctrl-3)
|
||||
[2026-06-21 08:37:36,666] [1217472] [ INFO] [ cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-3/data/ha-term.bin
|
||||
[2026-06-21 08:37:36,672] [1217472] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-3/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:37:36,672] [1217472] [ INFO] [ load_primary_config] No config at '/tmp/xce-deploy/controller/ctrl-3/conf/current' (rc=-2) — will fetch from primary after election
|
||||
[2026-06-21 08:37:36,672] [1217472] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=2, term=0)
|
||||
[2026-06-21 08:37:37,672] [1217472] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:37:37,672] [1217472] [ INFO] [ha_start_election_async] ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only, not starting own election
|
||||
[2026-06-21 08:37:37,672] [1217472] [ ERROR] [ cluster_init_phase2] cluster_init_phase2: cluster_config not loaded
|
||||
[2026-06-21 08:37:37,672] [1217472] [ WARN] [ myapp_run] myapp_run: cluster_init_phase2 on standby rejoin failed (rc=-22) — WAL not recovered; falling back to election-path recovery
|
||||
[2026-06-21 08:37:37,672] [1217472] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:37:37,672] [1217472] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:37:47,276] [1217472] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
||||
[2026-06-21 08:37:47,276] [1217472] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:37:47,276] [1217472] [ INFO] [ha_on_primary_ctrl_fd_rdhup] ha_on_primary_ctrl_fd_rdhup: no local config (tag=0, n_peers=3) — voter only, not starting PreVote/RDHUP re-election; awaiting configured primary (ha-config-bootstrap Req 2.8)
|
||||
[2026-06-21 08:37:50,743] [1217472] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:37:50,743] [1217472] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:37:50,743] [1217472] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:37:50,743] [1217472] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
[2026-06-21 08:40:01,930] [1217472] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:40:01,930] [1217472] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:40:01,930] [1217472] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:40:01,930] [1217472] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 1150ms (priority=2) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:40:02,636] [1217472] [ INFO] [ myreq_handler] HA: PreVote DENIED candidate='ctrl-2' term=1 reason=stale_term (my_term=0, floor tag=5298030106630493228 lsn=13)
|
||||
[2026-06-21 08:40:03,130] [1217472] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=3, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:40:03,176] [1217472] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 3)
|
||||
[2026-06-21 08:40:03,176] [1217472] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:40:03,180] [1217472] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=13 safe_config_tag=5298030106630493228 (from 1 granted voters)
|
||||
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:40:03,224] [1217472] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=13 current_lsn=14; applied_lsn=14 advertised for catch-up
|
||||
[2026-06-21 08:40:03,224] [1217472] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=3, first_boot=1)
|
||||
[2026-06-21 08:40:03,224] [1217472] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
|
||||
[2026-06-21 08:40:03,224] [1217472] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 0, localhost:9992)
|
||||
[2026-06-21 08:40:03,229] [1217472] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
|
||||
[2026-06-21 08:40:03,230] [1217472] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=14, writable_lsn gate armed
|
||||
[2026-06-21 08:40:06,333] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 1)
|
||||
[2026-06-21 08:40:06,333] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:40:06,374] [1217472] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=15 (committed_tag=5298030106630493228 committed_lsn=14)
|
||||
[2026-06-21 08:40:33,348] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 10)
|
||||
[2026-06-21 08:40:33,348] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:41:03,364] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 20)
|
||||
[2026-06-21 08:41:03,364] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:41:33,382] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 30)
|
||||
[2026-06-21 08:41:33,382] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:42:03,397] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 40)
|
||||
[2026-06-21 08:42:03,397] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:42:33,411] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 50)
|
||||
[2026-06-21 08:42:33,411] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:43:03,425] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 60)
|
||||
[2026-06-21 08:43:03,425] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:43:33,240] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 70)
|
||||
[2026-06-21 08:43:33,240] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:43:45,247] [1217472] [ INFO] [ ha_connect_to_peer] HA: connecting outbound to peer[0] (localhost:9991)
|
||||
[2026-06-21 08:43:45,250] [1217472] [ INFO] [ on_standby_connected] HA: outbound standby connection to peer[0] established, ctrlhello sent
|
||||
[2026-06-21 08:43:45,250] [1217472] [ INFO] [ on_standby_connected] HA: repl registered late-joiner standby peer[0] (slot 1, localhost:9991)
|
||||
[2026-06-21 08:43:48,290] [1217472] [ INFO] [ha_dispatch_repl_action_for_standby] HA repl: standby[0] lsn=13 < committed_lsn=22 → stream WAL batches
|
||||
[2026-06-21 08:43:48,290] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=14 target_lsn=22
|
||||
[2026-06-21 08:43:48,290] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [13+1..22] to standby
|
||||
[2026-06-21 08:43:48,449] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=14 < target=22 → streaming [14+1..22]
|
||||
[2026-06-21 08:43:48,449] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=15 target_lsn=22
|
||||
[2026-06-21 08:43:48,449] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [14+1..22] to standby
|
||||
[2026-06-21 08:43:48,649] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=15 < target=22 → streaming [15+1..22]
|
||||
[2026-06-21 08:43:48,649] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=16 target_lsn=22
|
||||
[2026-06-21 08:43:48,649] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [15+1..22] to standby
|
||||
[2026-06-21 08:43:48,849] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=16 < target=22 → streaming [16+1..22]
|
||||
[2026-06-21 08:43:48,849] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=17 target_lsn=22
|
||||
[2026-06-21 08:43:48,850] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [16+1..22] to standby
|
||||
[2026-06-21 08:43:49,050] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=17 < target=22 → streaming [17+1..22]
|
||||
[2026-06-21 08:43:49,050] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=18 target_lsn=22
|
||||
[2026-06-21 08:43:49,050] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [17+1..22] to standby
|
||||
[2026-06-21 08:43:49,250] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=18 < target=22 → streaming [18+1..22]
|
||||
[2026-06-21 08:43:49,250] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=19 target_lsn=22
|
||||
[2026-06-21 08:43:49,251] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [18+1..22] to standby
|
||||
[2026-06-21 08:43:49,451] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=19 < target=22 → streaming [19+1..22]
|
||||
[2026-06-21 08:43:49,451] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=20 target_lsn=22
|
||||
[2026-06-21 08:43:49,451] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [19+1..22] to standby
|
||||
[2026-06-21 08:43:49,651] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=20 < target=22 → streaming [20+1..22]
|
||||
[2026-06-21 08:43:49,651] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=21 target_lsn=22
|
||||
[2026-06-21 08:43:49,651] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [20+1..22] to standby
|
||||
[2026-06-21 08:43:49,851] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=21 < target=22 → streaming [21+1..22]
|
||||
[2026-06-21 08:43:49,851] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=22 target_lsn=22
|
||||
[2026-06-21 08:43:49,852] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [21+1..22] to standby
|
||||
[2026-06-21 08:43:51,294] [1217472] [ INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=23 (committed_tag=5298030106630493228 committed_lsn=22)
|
||||
[2026-06-21 08:45:24,291] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 1)
|
||||
[2026-06-21 08:45:24,292] [1217472] [ INFO] [ ha_connect_to_peer] HA: connecting outbound to peer[0] (localhost:9991)
|
||||
[2026-06-21 08:45:24,292] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[1] (localhost) not connected — initiating outbound connect (attempt 1)
|
||||
[2026-06-21 08:45:24,292] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[1] (localhost:9992) failed: -111
|
||||
[2026-06-21 08:45:24,294] [1217472] [ INFO] [ on_standby_connected] HA: outbound standby connection to peer[0] established, ctrlhello sent
|
||||
[2026-06-21 08:45:24,294] [1217472] [ INFO] [ on_standby_connected] HA: repl registered late-joiner standby peer[0] (slot 0, localhost:9991)
|
||||
[2026-06-21 08:45:51,305] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[1] (localhost) not connected — initiating outbound connect (attempt 10)
|
||||
[2026-06-21 08:45:51,305] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[1] (localhost:9992) failed: -111
|
||||
[2026-06-21 08:46:07,913] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=35 < target=36 → streaming [35+1..36]
|
||||
[2026-06-21 08:46:07,913] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=36 target_lsn=36
|
||||
[2026-06-21 08:46:07,913] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [35+1..36] to standby
|
||||
[2026-06-21 08:46:09,314] [1217472] [ INFO] [ ha_connect_to_peer] HA: connecting outbound to peer[1] (localhost:9992)
|
||||
[2026-06-21 08:46:09,319] [1217472] [ INFO] [ on_standby_connected] HA: outbound standby connection to peer[1] established, ctrlhello sent
|
||||
[2026-06-21 08:46:09,319] [1217472] [ INFO] [ on_standby_connected] HA: repl registered late-joiner standby peer[1] (slot 2, localhost:9992)
|
||||
[2026-06-21 08:46:12,358] [1217472] [ INFO] [ha_dispatch_repl_action_for_standby] HA repl: standby[1] lsn=22 < committed_lsn=44 → stream WAL batches
|
||||
[2026-06-21 08:46:12,358] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=23 target_lsn=44
|
||||
[2026-06-21 08:46:12,358] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [22+1..44] to standby
|
||||
[2026-06-21 08:46:12,516] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=23 < target=44 → streaming [23+1..44]
|
||||
[2026-06-21 08:46:12,516] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=24 target_lsn=44
|
||||
[2026-06-21 08:46:12,516] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [23+1..44] to standby
|
||||
[2026-06-21 08:46:12,716] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=24 < target=44 → streaming [24+1..44]
|
||||
[2026-06-21 08:46:12,716] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=25 target_lsn=44
|
||||
[2026-06-21 08:46:12,717] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [24+1..44] to standby
|
||||
[2026-06-21 08:46:12,917] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=25 < target=44 → streaming [25+1..44]
|
||||
[2026-06-21 08:46:12,917] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=26 target_lsn=44
|
||||
[2026-06-21 08:46:12,917] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [25+1..44] to standby
|
||||
[2026-06-21 08:46:13,117] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=26 < target=44 → streaming [26+1..44]
|
||||
[2026-06-21 08:46:13,118] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=27 target_lsn=44
|
||||
[2026-06-21 08:46:13,118] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [26+1..44] to standby
|
||||
[2026-06-21 08:46:13,318] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=27 < target=44 → streaming [27+1..44]
|
||||
[2026-06-21 08:46:13,318] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=28 target_lsn=44
|
||||
[2026-06-21 08:46:13,319] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [27+1..44] to standby
|
||||
[2026-06-21 08:46:13,519] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=28 < target=44 → streaming [28+1..44]
|
||||
[2026-06-21 08:46:13,519] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=29 target_lsn=44
|
||||
[2026-06-21 08:46:13,519] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [28+1..44] to standby
|
||||
[2026-06-21 08:46:13,719] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=29 < target=44 → streaming [29+1..44]
|
||||
[2026-06-21 08:46:13,719] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=30 target_lsn=44
|
||||
[2026-06-21 08:46:13,720] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [29+1..44] to standby
|
||||
[2026-06-21 08:46:13,920] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=30 < target=44 → streaming [30+1..44]
|
||||
[2026-06-21 08:46:13,920] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=31 target_lsn=44
|
||||
[2026-06-21 08:46:13,920] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [30+1..44] to standby
|
||||
[2026-06-21 08:46:14,121] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=31 < target=44 → streaming [31+1..44]
|
||||
[2026-06-21 08:46:14,121] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=32 target_lsn=44
|
||||
[2026-06-21 08:46:14,121] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [31+1..44] to standby
|
||||
[2026-06-21 08:46:14,321] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=32 < target=44 → streaming [32+1..44]
|
||||
[2026-06-21 08:46:14,321] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=33 target_lsn=44
|
||||
[2026-06-21 08:46:14,321] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [32+1..44] to standby
|
||||
[2026-06-21 08:46:14,522] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=33 < target=44 → streaming [33+1..44]
|
||||
[2026-06-21 08:46:14,522] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=34 target_lsn=44
|
||||
[2026-06-21 08:46:14,522] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [33+1..44] to standby
|
||||
[2026-06-21 08:46:14,722] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=34 < target=44 → streaming [34+1..44]
|
||||
[2026-06-21 08:46:14,722] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=35 target_lsn=44
|
||||
[2026-06-21 08:46:14,722] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [34+1..44] to standby
|
||||
[2026-06-21 08:46:14,922] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=35 < target=44 → streaming [35+1..44]
|
||||
[2026-06-21 08:46:14,922] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=36 target_lsn=44
|
||||
[2026-06-21 08:46:14,923] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [35+1..44] to standby
|
||||
[2026-06-21 08:46:15,123] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=36 < target=44 → streaming [36+1..44]
|
||||
[2026-06-21 08:46:15,123] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=37 target_lsn=44
|
||||
[2026-06-21 08:46:15,123] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [36+1..44] to standby
|
||||
[2026-06-21 08:46:15,323] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=37 < target=44 → streaming [37+1..44]
|
||||
[2026-06-21 08:46:15,323] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=38 target_lsn=44
|
||||
[2026-06-21 08:46:15,323] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [37+1..44] to standby
|
||||
[2026-06-21 08:46:15,565] [1217472] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=45 (committed_tag=5298030106630493228 committed_lsn=44)
|
||||
[2026-06-21 08:46:15,724] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=38 < target=44 → streaming [38+1..44]
|
||||
[2026-06-21 08:46:15,724] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=39 target_lsn=44
|
||||
[2026-06-21 08:46:15,724] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [38+1..44] to standby
|
||||
[2026-06-21 08:46:15,924] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=39 < target=44 → streaming [39+1..44]
|
||||
[2026-06-21 08:46:15,924] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=40 target_lsn=44
|
||||
[2026-06-21 08:46:15,924] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [39+1..44] to standby
|
||||
[2026-06-21 08:46:16,124] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=40 < target=44 → streaming [40+1..44]
|
||||
[2026-06-21 08:46:16,124] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=41 target_lsn=44
|
||||
[2026-06-21 08:46:16,125] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [40+1..44] to standby
|
||||
[2026-06-21 08:46:16,325] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=41 < target=44 → streaming [41+1..44]
|
||||
[2026-06-21 08:46:16,325] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=42 target_lsn=44
|
||||
[2026-06-21 08:46:16,325] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [41+1..44] to standby
|
||||
[2026-06-21 08:46:16,525] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=42 < target=44 → streaming [42+1..44]
|
||||
[2026-06-21 08:46:16,525] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=43 target_lsn=44
|
||||
[2026-06-21 08:46:16,526] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [42+1..44] to standby
|
||||
[2026-06-21 08:46:16,726] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=43 < target=44 → streaming [43+1..44]
|
||||
[2026-06-21 08:46:16,726] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=44 target_lsn=44
|
||||
[2026-06-21 08:46:16,726] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [43+1..44] to standby
|
||||
[2026-06-21 08:47:29,557] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=45 < target=48 → streaming [45+1..48]
|
||||
[2026-06-21 08:47:29,557] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=46 target_lsn=48
|
||||
[2026-06-21 08:47:29,558] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [45+1..48] to standby
|
||||
[2026-06-21 08:47:29,758] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=46 < target=51 → streaming [46+1..51]
|
||||
[2026-06-21 08:47:29,758] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=47 target_lsn=51
|
||||
[2026-06-21 08:47:29,758] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [46+1..51] to standby
|
||||
[2026-06-21 08:47:36,064] [1228337] [ INFO] [ main] My name is ctrl-3
|
||||
[2026-06-21 08:47:36,064] [1228337] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-3
|
||||
[2026-06-21 08:47:36,065] [1228337] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=2 (name=ctrl-3)
|
||||
[2026-06-21 08:47:36,065] [1228337] [ INFO] [ cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-3/data/ha-term.bin
|
||||
[2026-06-21 08:47:36,071] [1228337] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-3/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:47:36,072] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:36,072] [1228337] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:47:36,072] [1228337] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=2, term=3)
|
||||
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-3, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election
|
||||
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:47:37,074] [1228337] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=52 current_lsn=53; applied_lsn=52 advertised for catch-up
|
||||
[2026-06-21 08:47:37,074] [1228337] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:47:37,074] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:39,277] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:39,277] [1228337] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:47:39,277] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:39,277] [1228337] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
[2026-06-21 08:47:52,078] [1228337] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:47:52,078] [1228337] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:47:52,078] [1228337] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:47:52,078] [1228337] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 1088ms (priority=2) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:47:52,855] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:52,855] [1228337] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:47:52,855] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:52,855] [1228337] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
[2026-06-21 08:47:53,279] [1228337] [ INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606)
|
||||
[2026-06-21 08:48:04,083] [1228337] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:48:04,083] [1228337] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:48:04,083] [1228337] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:48:04,083] [1228337] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 1092ms (priority=2) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:48:04,312] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:48:04,312] [1228337] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:48:04,312] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:48:04,312] [1228337] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
[2026-06-21 08:48:05,284] [1228337] [ INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606)
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
1 | 0 | NOOP | 0 | 00000000 |
|
||||
2 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
3 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
4 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
5 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
6 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
7 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
8 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
9 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
10 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
11 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
12 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
13 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
14 | 0 | NOOP | 0 | 00000000 |
|
||||
15 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
16 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
17 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
18 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
19 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
20 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
21 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
22 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
23 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
24 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
25 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
26 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
27 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
28 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
29 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
30 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
31 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
32 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
33 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
34 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
35 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
36 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
37 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
38 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
39 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
40 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
41 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
42 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
43 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
44 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
45 | 0 | NOOP | 0 | 00000000 |
|
||||
46 | 0 | JOB_SUBMIT | 3952 | faead246 | jobid=1868595630964736 subtime=1782031661 state=1
|
||||
47 | 0 | JOB_SUBMIT | 3952 | faead246 | jobid=1868595630964736 subtime=1782031661 state=1
|
||||
48 | 0 | JOB_SUBMIT | 3952 | be0059bf | jobid=1868595630964737 subtime=1782031661 state=1
|
||||
49 | 0 | JOB_SUBMIT | 3952 | be0059bf | jobid=1868595630964737 subtime=1782031661 state=1
|
||||
50 | 0 | JOB_SUBMIT | 3952 | fe7a3d0b | jobid=1868595630964738 subtime=1782031661 state=1
|
||||
51 | 0 | JOB_SUBMIT | 3952 | fe7a3d0b | jobid=1868595630964738 subtime=1782031661 state=1
|
||||
52 | 0 | JOB_SUBMIT | 3952 | 37d54e4d | jobid=1868595630964739 subtime=1782031661 state=1
|
||||
53 | 0 | JOB_SUBMIT | 3952 | 37d54e4d | jobid=1868595630964739 subtime=1782031661 state=1
|
||||
54 | 0 | NOOP | 0 | 00000000 |
|
||||
segments: 3 entries: 54 submits: 50
|
||||
lsn range: 1..54 recovered tip: 54
|
||||
gaps: 0 crc chain: intact
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
1 | 0 | NOOP | 0 | 00000000 |
|
||||
2 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
3 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
4 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
5 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
6 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
7 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
8 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
9 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
10 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
11 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
12 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
13 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
14 | 0 | NOOP | 0 | 00000000 |
|
||||
15 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
16 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
17 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
18 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
19 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
20 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
21 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
22 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
23 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
24 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
25 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
26 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
27 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
28 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
29 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
30 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
31 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
32 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
33 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
34 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
35 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
36 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
37 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
38 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
39 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
40 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
41 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
42 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
43 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
44 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
45 | 0 | NOOP | 0 | 00000000 |
|
||||
46 | 0 | JOB_SUBMIT | 3952 | d32ac747 | jobid=1868595643547648 subtime=1782031673 state=1
|
||||
47 | 0 | JOB_SUBMIT | 3952 | d32ac747 | jobid=1868595643547648 subtime=1782031673 state=1
|
||||
48 | 0 | JOB_SUBMIT | 3952 | 97c04cbe | jobid=1868595643547649 subtime=1782031673 state=1
|
||||
49 | 0 | JOB_SUBMIT | 3952 | 97c04cbe | jobid=1868595643547649 subtime=1782031673 state=1
|
||||
50 | 0 | JOB_SUBMIT | 3952 | d7ba280a | jobid=1868595643547650 subtime=1782031673 state=1
|
||||
51 | 0 | JOB_SUBMIT | 3952 | d7ba280a | jobid=1868595643547650 subtime=1782031673 state=1
|
||||
52 | 0 | JOB_SUBMIT | 3952 | 1e155b4c | jobid=1868595643547651 subtime=1782031673 state=1
|
||||
53 | 0 | JOB_SUBMIT | 3952 | 1e155b4c | jobid=1868595643547651 subtime=1782031673 state=1
|
||||
54 | 0 | NOOP | 0 | 00000000 |
|
||||
segments: 3 entries: 54 submits: 50
|
||||
lsn range: 1..54 recovered tip: 54
|
||||
gaps: 0 crc chain: intact
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
1 | 0 | NOOP | 0 | 00000000 |
|
||||
2 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
3 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
4 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
5 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
6 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
7 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
8 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
9 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
10 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
11 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
12 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
13 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
14 | 0 | NOOP | 0 | 00000000 |
|
||||
15 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
16 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
17 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
18 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
19 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
20 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
21 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
22 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
23 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
24 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
25 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
26 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
27 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
28 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
29 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
30 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
31 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
32 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
33 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
34 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
35 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
36 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
37 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
38 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
39 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
40 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
41 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
42 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
43 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
44 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
45 | 0 | JOB_SUBMIT | 3952 | 4c6d24e6 | jobid=1868595618381824 subtime=1782031649 state=1
|
||||
46 | 0 | JOB_SUBMIT | 3952 | 4c6d24e6 | jobid=1868595618381824 subtime=1782031649 state=1
|
||||
47 | 0 | JOB_SUBMIT | 3952 | 0887af1f | jobid=1868595618381825 subtime=1782031649 state=1
|
||||
48 | 0 | JOB_SUBMIT | 3952 | 0887af1f | jobid=1868595618381825 subtime=1782031649 state=1
|
||||
49 | 0 | JOB_SUBMIT | 3952 | 48fdcbab | jobid=1868595618381826 subtime=1782031649 state=1
|
||||
50 | 0 | JOB_SUBMIT | 3952 | 48fdcbab | jobid=1868595618381826 subtime=1782031649 state=1
|
||||
51 | 0 | JOB_SUBMIT | 3952 | 8152b8ed | jobid=1868595618381827 subtime=1782031649 state=1
|
||||
52 | 0 | JOB_SUBMIT | 3952 | 8152b8ed | jobid=1868595618381827 subtime=1782031649 state=1
|
||||
53 | 0 | JOB_SUBMIT | 3952 | 37d54e4d | jobid=1868595630964739 subtime=1782031661 state=1
|
||||
54 | 0 | NOOP | 0 | 00000000 |
|
||||
segments: 2 entries: 54 submits: 51
|
||||
lsn range: 1..54 recovered tip: 54
|
||||
gaps: 0 crc chain: intact
|
||||
Loading…
Add table
Add a link
Reference in a new issue