merge-fix: drop Rongsong bookkeeping missed in merge (.kiro/ reports/ CHANGES.md)
First git rm during merge was refused for worktree-modified files; only .wolf/ got removed. Clean up the remaining OpenWolf/spec bookkeeping and weekly reports so the merged branch carries source fixes only.
This commit is contained in:
lei.chen
parent
3b6f53bb65
commit
3dbb96d4e6
343 changed files with 0 additions and 56290 deletions
|
|
@ -1,123 +0,0 @@
|
|||
# Investigation: committed jobs lost across controller failover (bug-795 family)
|
||||
|
||||
> Live investigation 2026-06-20. Status: **root-cause located, fix not yet applied.** This is the long-standing data-loss bug (PREDATES libxce-failover — confirmed by the user; libxce-failover + xce-test-e2e-primitives were *test-infrastructure* improvements, not the durability fix). Owning spec: **ha-commit-durability** (REOPENED for bug-795). Adjacent: ha-convergence-recovery, ha-single-leader-role, ha-peer-connection-reuse.
|
||||
|
||||
## Symptom
|
||||
Running the **controller failover test** (kill primary / restart nodes while a client submits jobs), committed jobs that were ACKed (`jobsubmit ok`) and visible in `joblist` **disappear** after enough kill/restart cycles. In the captured run, jobs survived terms 2→4 then vanished at the term-4→5 transition; `ctrlstatus` then showed them absent. NOT a `committedLsn` display bug — the data is genuinely gone.
|
||||
|
||||
## DECISIVE EVIDENCE — it is real data loss, not a commit-index issue
|
||||
Decoded the raw WAL bytes at `/tmp/xce-ha-deploy/controller/ctrl-{1,2,3}/data/wal/seg-0000000000000001.wal` (format: `wal_entry_hdr_t` 40B `{term,lsn,timestamp,type,payload_len,crc,prev_crc}` + payload; `src/include/wal_types.h`):
|
||||
|
||||
| Node | WAL entries (physical) | JOB_SUBMITs | State |
|
||||
|------|------------------------|-------------|-------|
|
||||
| ctrl-1 | LSN **1–15** (starts NOOP@1) | **12** | full history — but **DOWN** (`connect peer failed -111`) |
|
||||
| ctrl-2 | LSN **16–18** (starts NOOP@16) | 2 | live, **missing 1–15** |
|
||||
| ctrl-3 | LSN **13–18** (starts NOOP@13) | 4 | live, **missing 1–12** |
|
||||
|
||||
The committed prefix (LSN 2–12 = the lost jobs) is **physically absent** from both live nodes; it exists only on the down node. `snap/` dirs are **empty** on all nodes (no snapshot).
|
||||
|
||||
## DECISIVE EVIDENCE 2 — catch-up streaming DROPS payloads (bug-847, REPRODUCED 2026-06-20)
|
||||
A live restart of ctrl-1 (kill ctrl-1 → restart) showed it rejoin and catch up to the cluster length `[1..14]` with **correct LSN/type/CRC chain but EMPTY payloads** — `waldump` reports every JOB_SUBMIT as `PLEN=0 PCRC=00000000 jobid=?`, while ctrl-2/ctrl-3 still hold full 872-byte payloads. So the WAL is **structurally intact but content-empty** — silent corruption that passes every check that doesn't decode the payload (the `PCRC` column is what exposed it). This, not just the missing-prefix gap, is a concrete committed-job-loss mechanism: if the empty-payload node ever leads, the empty entries become authoritative.
|
||||
|
||||
**Root cause (reproduced in a unit test):** the catch-up batch wire path (`xce_wal_batch_encode_raw`/`wal_entry_to_wire` in `src/lib/request.c`) builds a `WalCatchupEntry` struct-list whose per-element `payload :Data` is allocated via `capn_new_list8(cs=root_seg,…)`. When the batch outgrows its first capnp segment, the per-element payload `Data` pointer is **lost across the segment boundary** → on decode `extract_data_bytes(we.payload)` reads `CAPN_NULL` → `raw_payload_len=0` → the receiver writes a payload-less entry. **`test_wal_batch_roundtrip_large_payloads` (14×872B) FAILS "Expected 872 Was 0"; the 2-entry round-trip PASSES** — the threshold is message SIZE (segment overflow), not entry count. **Live replication (`repl_broadcast`) carries payloads fine — only the catch-up/rejoin path corrupts**, which is exactly why loss surfaces only after a node restarts and catches up. Candidate fix: carry all sub-entry payloads as ONE top-level capnp `Data` blob + lengths (single `Data` round-trips correctly), sidestepping per-element cross-segment pointers. RED test is in `src/lib/testsuite/test_wal_wire.c`.
|
||||
|
||||
## ROOT CAUSE (mechanism, code-grounded)
|
||||
A node that (re)joins with an **empty WAL** is brought up to the cluster's **current committed watermark by WAL streaming `[watermark..committed]` — NOT from LSN 1 — and there is no snapshot to backfill `[1..watermark-1]`.** So it ends up with a permanent prefix gap. When the failover test eventually has the only full-log node down while the others rejoined at high watermarks, the committed jobs are gone for good.
|
||||
|
||||
Why the `seg-...001` file contains LSN 16: a node booting empty defaults `current_lsn=1` (`persist.c:429`); the first write lazily calls `open_new_segment` which names the file by `current_lsn` → `seg-...001` and **`O_TRUNC`s** it (`persist.c:145`); catch-up then writes entries carrying explicit high LSNs (16…) from offset 0. So "writes from the beginning of the file" = **booted empty + caught up from the watermark**, not an in-place overwrite of recovered data. (`replay_wal` at `persist.c:1422` is correct — it opens existing segments **read-only** and advances `current_lsn` past what it recovers.)
|
||||
|
||||
## OPEN FORKS (what to settle next)
|
||||
1. **Test procedure — clean vs preserve.** `scripts/cleanup.sh` does `rm -rf .../data/*`; `scripts/start.sh` only creates `data/` if missing (preserves). The on-disk files (named `001`, starting at 16/13) match the **cleaned** case. **Confirm with the user whether the failover test cleans data dirs before restart or restarts with data intact.** (Still worth confirming for the exact trigger ordering, but no longer the architectural question — see VERDICT.)
|
||||
- If preserved + node still returns gapped → **recovery bug** (`replay_wal` failing to restore).
|
||||
- If cleaned → **catch-up never backfills the prefix + no snapshot** = the architectural gap (CONFIRMED below).
|
||||
2. **Catch-up start point — ✅ SETTLED (2026-06-20).** Traced. `handle_ctrlhello` (`config_repl.c:1436`) is connection-setup ONLY; catch-up is driven by the heartbeat decision loop. The standby reports its **REAL `applied_lsn`** — `ha_process_standby_state(ha, peer_idx, sr.config_tag, sr.applied_lsn, …)` at `ha.c:3518`, where `sr.applied_lsn` is the standby's own state-report value (an empty standby reports 0). The decision `ha_evaluate_repl_action` (`ha.c:1811`) is CORRECT: `standby_lsn+1 < oldest_wal_lsn → SEND_SNAPSHOT`, `else standby_lsn < committed → STREAM_WAL`. So advertise is NOT the bug — a fresh standby correctly asks for a snapshot when the primary's WAL no longer starts at 1.
|
||||
3. **Snapshots — ✅ SETTLED: THIS IS THE ROOT CAUSE (2026-06-20).** `write_snapshot` (`persist.c:1051`) writes a **16-byte SENTINEL**: `"XCESNAP\0"` (8) + `checkpoint_lsn` (8) — its own comment says *"Full Cap'n Proto serialization is Task 11."* It carries **ZERO job/queue entities.** `load_snapshot` (`persist.c:1267`) just does `p->checkpoint_lsn = p->current_lsn = stored_lsn` — it **advances the watermark past entries it never received, restoring nothing.** And `persist_checkpoint` → `trim_wal_segments` (`persist.c:1090`) **deletes WAL segments below the checkpoint.** So checkpoint = "trim the committed prefix + write an empty backup": a WAL-eating loop with no recoverable safety net. This is the "entity-bearing snapshots" gap (ha-convergence-recovery backlog) — and it makes any empty-rejoin/prefix-trim below committed **unrecoverable BY DESIGN.**
|
||||
|
||||
## VERDICT (2026-06-20) — architectural root confirmed, code-grounded end-to-end
|
||||
The loss is **not** an advertise/start-point bug and **not** a recovery bug. It is the **absence of entity-bearing snapshots** combined with **prefix-trimming checkpoints**. Full mechanism:
|
||||
1. A fresh/empty member (applied_lsn=0), or any member catching up against a primary whose own `oldest_wal_lsn > 1` (its prefix was trimmed by a prior checkpoint).
|
||||
2. `ha_evaluate_repl_action` → `SEND_SNAPSHOT` (when `1 < oldest_wal_lsn`) or `STREAM_WAL` from the high watermark.
|
||||
3. SEND_SNAPSHOT path: `persist_checkpoint` writes the **sentinel** (no entities) + **trims the prefix WAL**, then ships the sentinel; receiver `persist_load_snapshot` sets `current_lsn = checkpoint_lsn` with **no entity restore**.
|
||||
4. STREAM_WAL path: `ha_stream_wal_to_standby` streams `[standby_lsn+1 .. committed]`; `ha_apply_wal_catchup_entry`/`ha_apply_single_entry` **physically persist** those entries (consistent with ctrl-3's on-disk seg holding LSN 13–18) — but the prefix `[1..oldest-1]` is **never streamed and has no snapshot to backfill it.**
|
||||
5. Net: the member's watermark reaches `committed` while the committed-prefix job entities are gone. **Permanent loss, BY DESIGN.**
|
||||
|
||||
**Fix layer (recommendation — option (b), question-the-architecture):** implement **entity-bearing snapshots** — `write_snapshot` Cap'n-Proto-serializes all live job/queue entities at `checkpoint_lsn`; `load_snapshot` deserializes + `restore_job`/restore_queue them BEFORE advancing `current_lsn`; and `trim_wal_segments` must NOT trim below the LSN covered by a durable entity-bearing snapshot (install-snapshot-before-stream on the receive side). Option (a) "stream `[1..committed]`" only works while some node still retains LSN 1 (false after any trim), so it is a partial mitigation, not the fix. This is the **ha-convergence-recovery** "entity-bearing snapshots" item; route the fix there (or a new `ha-entity-snapshots` spec) under the ha-commit-durability umbrella.
|
||||
|
||||
## SECONDARY FINDINGS
|
||||
- **Double `WAL_JOB_SUBMIT` per job (bug-845, 2026-06-20, via waldump).** Every committed job appears as TWO consecutive entries with the SAME jobid (ctrl-1: 2,3 / 4,5 / … ; ctrl-3: 14,15 ; ctrl-2: 17,18). **Ruled out array expansion:** `job_submit` assigns array members DISTINCT ids (`base_id+i`, job.c) and the `Job` capnp has a single jobid field, so identical-jobid pairs are NOT two array tasks — a single submit is double-appending. **CONFIRMED a literal byte-for-byte double-append** by the enhanced waldump (payload-CRC + subtime columns, 2026-06-20): each pair is byte-identical — same PCRC (LSN 2,3 both `6d79074e`; 4,5 `c25ac163`; …) and same `subtime` (e.g. 1781959730), so all 12 ctrl-1 submits = 6 jobs × 2. A failover retry would re-stamp `subtime=time(NULL)` and differ → **retry definitively ruled out; it's the SAME encoded `job_t` written twice in one submit flow.** It is replicated *consistently* (F=…555020288 at 14,15 on BOTH ctrl-1 and ctrl-3; G at 17,18 on BOTH ctrl-2/3), so not itself divergence, but it doubles LSN consumption and is a latent cross-node divergence risk if the duplicate count ever varies by primary. Next: trace `handle_jobsubmit`→`persist_job_hook_fun` AFTER_EVENT loop (jlist holding the job twice, or a double hook-fire).
|
||||
- **Raft cross-node consistency check (user request, 2026-06-20).** Per-LSN comparison of the three dumps: at every OVERLAPPING index the entries AGREE — ctrl-1∩ctrl-3 = {13:NOOP, 14:F, 15:F}; ctrl-2∩ctrl-3 = {16:NOOP, 17:G, 18:G}. So **State Machine Safety (no two members hold different entries at the same index) is NOT violated** in this snapshot; no job sits at different LSNs on different nodes. The broken guarantee is **Leader Completeness / durability**: jobid E=…394588160 was committed+acked at LSN 11–12 (present on ctrl-1) but is ABSENT from the surviving majority (ctrl-2/3) — a committed entry lost (exactly the root-cause mechanism: it was below the survivors' empty-rejoin catch-up start, with no entity snapshot to backfill it). Conversely G=…630517760 is on the survivors but absent on ctrl-1, which was already down before G — losing an entry a crashed minority-node never had is Raft-legal. So the cluster ended with two committed histories sharing only the F segment: ctrl-1's {…,E@11-12,F@14-15} vs survivors' {F@14-15,…,G@17-18}.
|
||||
- **`term=0` on every WAL entry.** `p->term` (the `hdr.term` source) is **never assigned** the election term anywhere (grep for `p->term =` / `->term =` in persist.c/ha.c returns nothing). So per-entry term is dead. XCE matches logs via the `prev_crc` chain, so possibly benign — but it means **no term-based divergence detection**. Worth a hard look (could let divergent same-LSN entries be treated as matching).
|
||||
- **Cluster cannot hold a primary** under the test: peer connect `-111` (ECONNREFUSED), RDHUP, rapid term churn 2→5 in ~7 min, elections logged `safe_committed_lsn=X (from 1 granted voters)`. Logged separately in `ha-convergence-recovery/research.md` (2026-06-20 live finding).
|
||||
- **Crash fixed en-route (unrelated):** `xce_strerror_r` global-buffer-overflow on HA error codes — bug-827, fixed in `src/lib/utils.c` (commit 1aa0761).
|
||||
|
||||
## Is the ha-heartbeat-committed-propagation change implicated?
|
||||
The loss **predates** it, so it is **not the origin**. But two real concerns it introduces (it writes `last_known_committed_lsn` per-heartbeat, decoupled from WAL possession):
|
||||
- `ha_enter_real_election` (`ha.c:4137`) truncates WAL above `last_known_committed_lsn` — a heartbeat-fed value now feeds a truncation path (violates the Req 3.1 "advisory-only" boundary I claimed; guarded by `if (committed>0)` so 0 is safe, but still a real boundary leak).
|
||||
- If the rejoin/catch-up advertised position is derived from `last_known_committed_lsn`, the per-heartbeat freshness would **widen** the prefix gap (fresh node learns committed=13 via heartbeat → advertises 13 → streams from 14). **This is exactly Open Fork #2 — read the CtrlHello advertise path; it also settles aggravation.**
|
||||
- Candidate-advertise is SAFE: `ha_candidate_advertise_lsn` (`ha.c:2598`) returns bare `persist_durable_floor`, NOT last_known — so a behind node cannot fake election freshness.
|
||||
|
||||
## REPRO / ARTIFACTS (on this host)
|
||||
- Live deploy with the failed run: `/tmp/xce-ha-deploy/` — controller logs `logs/ctrl-{1,2,3}.log`, WAL `controller/ctrl-N/data/wal/seg-...001.wal`, empty `snap/`.
|
||||
- Offline WAL decoder (Python, ad-hoc) — walks `wal_entry_hdr_t`, lists `lsn|term|type|payload_len`, flags submits by header timestamp. (To be productized as `waldump`, below.)
|
||||
- Deployed binary includes controller-status (responderTime) + ha-heartbeat-committed (the `ha.c:4967 "learned committed_lsn=X from heartbeat"` logs are the consume path).
|
||||
|
||||
## NEXT STEPS
|
||||
1. **Build the WAL/snapshot inspection tooling in `xce_test`** — ✅ `waldump` DONE (2026-06-20). `snapdump`/`walcompare` still TODO. Gap-location is now a one-liner per run.
|
||||
2. **Read the CtrlHello catch-up advertise/start-point** (`config_repl.c` handle_ctrlhello + the standby's CtrlHelloRequest build) — Open Fork #2.
|
||||
3. **Confirm whether snapshot creation exists** or is a stub — Open Fork #3 (likely the architectural root: no entity-bearing snapshot ⇒ no prefix backfill).
|
||||
4. Decide fix layer: (a) full-WAL stream `[1..committed]` when a node has no snapshot, (b) implement entity-bearing snapshots + install-before-stream, or (c) prevent empty-rejoin from advancing committed without holding the entries. Likely a **question-the-architecture** moment (catch-up assumes a snapshot safety net that does not exist).
|
||||
|
||||
## DECISIVE EVIDENCE 3 — bug-851 REPRODUCED & ROOT-CAUSED on an owned cluster (2026-06-21)
|
||||
|
||||
Stood up a fresh local 3-node cluster (`/xce-deploy-ha`, binary built with the 848/849/850 fixes) and drove two scenarios with full log access. Artifacts: `repro-851/{ctrl-1,ctrl-2,ctrl-3}.log` + `waldump-ctrl-{1,2,3}.txt`.
|
||||
|
||||
**Scenario 1 — clean single failover: PASS (848/849/850 validated end-to-end).**
|
||||
6 jobs → identical `1..13` on all 3 (matching PCRCs). Kill primary ctrl-1 → ctrl-3 promoted: `cluster_init_phase2: persist_recover done — recovered durable_floor=13 … applied_lsn=14 advertised` (bug-849 recovery-on-promotion + bug-848 advertise-real-LSN both fire). 4 jobs on new primary → ctrl-2 converges `14..22`. Restart old primary → recovers `1..13`, catches up `14..22`. **Final: all three WALs byte-identical `1..22`, identical PCRC fingerprint `cc5d361e…`, 0 gaps.**
|
||||
|
||||
**Scenario 2 — chaotic leadership churn (3× kill-the-primary): FAIL — silent committed-prefix divergence.**
|
||||
After 3 rounds, all three report `committedLsn=54`, all `lsn range 1..54`, 0 gaps, CRC intact — **but the logs DIVERGE at LSN 45–54**:
|
||||
- ctrl-1: `45=NOOP`, `46..53 = faead246/be0059bf/fe7a3d0b/37d54e4d` → fp `bfeb26ad…`, 25 jobids
|
||||
- ctrl-2: `45=NOOP`, `46..53 = d32ac747/97c04cbe/d7ba280a/1e155b4c` (DIFFERENT jobs) → fp `216af434…`, 25 jobids
|
||||
- ctrl-3: `45 = 4c6d24e6` (a JOB, no noop), `45..52 = 4c6d24e6/0887af1f/48fdcbab/8152b8ed` → fp `865304ec…`, 26 jobids
|
||||
|
||||
Three different "committed" tails, each node reporting them committed. This violates the Raft committed-prefix invariant (a committed entry must be byte-identical on every member).
|
||||
|
||||
**ROOT CAUSE (code-grounded, three compounding defects):**
|
||||
1. **Term not monotonic across the cluster.** Election term = *locally-persisted* term + 1; PreVote explicitly does NOT persist/propagate ("starting PreVote round … no persist"). Under chaotic kills a leader wins term N, appends its own-term noop + jobs, and is killed **before term N is durably propagated to a majority**. A restarted node recovers a *stale* term and elects at the **same** N+1. Logs show **two leaders at term 3** (ctrl-3 @08:40, ctrl-1 @08:47:39) and **two leaders at term 4** (ctrl-2 @08:47:52, ctrl-3 @08:48:04) — each appending its own-term noop at the **same LSN 45**. Persisted terms ended split: ctrl-1=4, ctrl-2=4, ctrl-3=3.
|
||||
2. **The WAL entry term field is DEAD (always 0).** `wal_entry_hdr_t.term` exists (wal_types.h:30) and the replication envelope carries `term` (typemap.h:381), but the writer does `hdr.term = p->term` (persist.c:588/726/1040) and **`p->term` is NEVER assigned from `ha->term` anywhere in the controller** (zero matches). Every entry, every term, persists `term=0`. waldump confirms term=0 cluster-wide.
|
||||
3. **No (term,lsn) reconcile → divergent tails never truncated.** Because the WAL has no usable per-entry term, rejoin-reconcile (`ha_convergence.c`) cannot do Raft log-matching: entry 45@ctrl-1 (term 3) and 45@ctrl-2 (term 4) look identical (both term 0, both JOB_SUBMIT/NOOP), so the divergent suffix is taken as already-present and is never overwritten. `ha_compute_safe_committed_lsn` then advances committed (44, 53) "from 1 granted voter" over content that other members never received.
|
||||
|
||||
Net: under leadership churn, acked entries written by a short-lived leader become a permanent divergent tail, and `committed` advances over it → silent loss + cross-node divergence. This is distinct from the catch-up payload-drop (bug-847) and the empty-rejoin gap (bug-828); it is a **Raft-safety gap in term establishment + WAL term recording**.
|
||||
|
||||
**Keystone fix candidate (smallest correct lever):** stamp `hdr.term` from the live raft term (`ha->term`) on the primary append path + carry it through the replication envelope + persist it, THEN make reconcile truncate any local suffix whose (term,lsn) does not match the leader's — standard Raft log-matching. Must be paired with making the election term durably-established-before-acting (no two leaders per term). **HA-core + persistence → full spec workflow required (per CLAUDE.md); not a quick patch.**
|
||||
|
||||
**Secondary (separate) finding — client primary-discovery is positional, not failover-correct.** With `masters = [m0, m1, …]`, the client (`xcecli_ctrlstatus_primary`, client.c:403) only reaches the primary if `m0` IS the primary: a standby answers `primaryName=""` ("unknown" by design, ctrlstatus.c:104) so no redirect hint is captured, and the inner `xcecli_ctrlstatus` SUCCEEDS on the reachable non-primary so transport failover never rotates. Confirmed: `9993`(primary-first)→ok 45ms; `9992,9993`(standby-first)→"no primary"; `9991,9993`(dead-first)→"no primary"; `9993,9991`→ok. Also the dead node returns `ctrlstatus failed: Invalid argument` (bug-846 EINVAL). This is what made earlier remote sessions read as "committed frozen" — the client was pinned to a stale/standby node. Fix options: (a) standby tracks+reports the primary name it heartbeats with (enable redirect), or (b) client rotates to the next master on a reachable-but-non-authoritative reply.
|
||||
|
||||
## TOOLING PLAN (xce_test enhancement — requested by user)
|
||||
Read-only debug commands in `xce_test`. **NOTE (correction):** `xce_test` does NOT link `persist.c`/`job.c` — it builds from `test.c` only + `libxce`. But `read_Job` is generated into `controller.capnp.c` which IS compiled into `libxce`, so the capnp decode is reachable; `waldump` adds `c_capnproto_dep`+`zlib_dep` to `xce_test_deps` and ships its own zero-alloc jobid decode (reads `struct Job.jobid` after `read_Job`, no `decode_Job`).
|
||||
- `waldump <data_dir|wal_dir|seg_file>` — ✅ **DONE 2026-06-20.** Per node/segment, lists every entry `lsn | term | type | payload_len | jobid`, flags **LSN gaps** (non-contiguous / missing prefix) + the CRC recovered tip; nonzero exit on gap/CRC-break so it doubles as an E2E assertion. Impl: `src/controller/testsuite/waldump.{c,h}` (module), `cmd_waldump` wired into `top_commands` in `test.c`; unit test `test_waldump.c` (5 cases, hand-built raw segments). **Validated live** against `/tmp/xce-ha-deploy/controller/ctrl-{1,2,3}` — reproduced the exact bug table (ctrl-1 full 1–15; ctrl-2 missing prefix 1–15; ctrl-3 missing prefix 1–12). Also surfaced a NEW anomaly: each JOB_SUBMIT appears as a **consecutive duplicate pair** (same jobid at LSN N and N+1) — worth a follow-up.
|
||||
- `snapdump <snap_dir|snap_file>` — TODO. Decode the `XCESNAP` header (lsn) and report whether it carries entities (it likely does NOT — confirms Fork #3).
|
||||
- `walcompare <dirA> <dirB>` — TODO. Diff two nodes' logs by `lsn → crc/jobid` to flag divergence/gaps instantly.
|
||||
|
||||
## UPDATE 2026-06-30 — SHARPENED ROOT CAUSE for the chaotic-soak loss: `xce_run_hooks` clobbers the durability `-ETIMEDOUT`
|
||||
|
||||
The 10-round chaotic soak (`/home/claude/tmp/ha-1296`) loss is **NOT** the prefix-gap/entity-snapshot mechanism above. Decisive WAL evidence (waldump):
|
||||
- All three nodes are **gapless, CRC-intact, full 6504B payloads** (so NOT bug-847 payload-drop, NOT a trimmed prefix — `seg-...001` present on all three, `snap/` empty, no trim fired).
|
||||
- Lost job `0-0-1782854622-0` exists **only on ctrl-1 @ LSN 30, term 9**; ctrl-2/ctrl-3 hold a **term-10 NOOP @ LSN 30** (the new leader's first entry). Classic divergent uncommitted tail at the term 9→10 boundary.
|
||||
|
||||
ctrl-1's controller log (PID 406309, term-9 leader) shows the durable-majority gate **worked correctly**:
|
||||
```
|
||||
commit: WAIT deadline expired without commit (lsn=29 committed_lsn=0 writable_lsn=28 timeout_ms=500)
|
||||
commit: WAIT deadline expired without commit (lsn=30 ...) <- the lost job
|
||||
commit: WAIT deadline expired without commit (lsn=31 ...)
|
||||
```
|
||||
The standby was still catching up (`standby[1] lsn=23 < committed_lsn=29`), so LSN 30 never reached majority → `repl_wait_majority_ack` → `commit_submit` returned **`-ETIMEDOUT`**, and `persist_wal_sync_append` propagated it (persist.c:2550-2551), and `persist_job_hook_fun` returned it (persist.c:2849).
|
||||
|
||||
**The leak is in `xce_run_hooks` (src/lib/hook.c:134-157).** It runs all hooks for the type but `rc` is **overwritten every iteration** and it `return rc` — i.e. it returns ONLY the LAST-registered hook's value (the `break` is `BEFORE_EVENT`-only). Multiple JOB `AFTER_EVENT` hooks are registered (`apiserver_job_hook_fun` apiserver.c:545, `sched_job_hook_fun` sched.c:651, `persist_job_hook_fun` persist.c:615 + re-added on promotion ha.c:3436). When `persist_job_hook_fun` returns `-ETIMEDOUT` but a later-registered hook returns 0, **the `-ETIMEDOUT` is clobbered to 0**. `handle_jobsubmit` (job.c:593-611) then sees `hook_rc == 0`, skips the reject path (job.c:603 — confirmed: NO "rejecting job" line in ctrl-1's log), and replies `jobsubmit ok` (job.c:613-618) for an entry that was never majority-committed. A term-10 leader then legally overwrites LSN 30 → **committed-job-loss** + the harness's `committed_job_loss` count.
|
||||
|
||||
This VIOLATES the CLAUDE.md **Commit Durability** invariant (reply-success ⇒ majority-durable). `handle_jobsubmit`'s gate is correct; the durability signal is destroyed one layer down in the shared hook runner.
|
||||
|
||||
### Fix (surgical, libxce)
|
||||
`xce_run_hooks` must not let a later hook's success clobber an earlier hook's failure on `AFTER_EVENT`: fold to a **sticky-negative** rc (run every AFTER_EVENT hook, but once any returns < 0 preserve it; return the first/any negative). `BEFORE_EVENT` keeps veto/break semantics. This restores "AFTER_EVENT fails if ANY hook failed," which is exactly what `handle_jobsubmit` already assumes. RED: register two AFTER_EVENT hooks where the first returns `-ETIMEDOUT` and the second returns 0; assert `xce_run_hooks` returns `-ETIMEDOUT`. Blast radius: shared by controller/worker/scheduler — audit any consumer relying on the *last* hook's positive rc (none known; consumers treat non-zero as failure).
|
||||
|
||||
NOTE: the entity-bearing-snapshot / prefix-trim gap documented above is a SEPARATE, still-real defect (it bites a node that rejoins after a checkpoint trims the prefix). It is just not the mechanism this particular soak hit — confirm with a soak that forces a checkpoint+trim.
|
||||
|
|
@ -1,73 +0,0 @@
|
|||
# Brief: ha-commit-durability
|
||||
|
||||
## Problem
|
||||
A 3-node HA controller failover test **permanently lost committed jobs**. The client submitted 10 jobs across several forced primary kills; the 7 earliest (JobIds `0-0-1781708993..1781709035`, WAL LSN ≤20) — which had been replicated and majority-ACKed, and survived the *first* failover — were gone by the end, while only the 3 latest survived. Losing an acknowledged, committed job is a correctness (data-loss) failure, the most severe class for this system.
|
||||
|
||||
## Current State
|
||||
The HA consistency machinery is implemented (commit engine, WAL replication, convergence/truncate-on-demotion, single-leader own-term gate, PreVote, vote-freshness bug-339). But **"committed" does not imply "durable on a majority"**:
|
||||
|
||||
- **WAL fsync is asynchronous** — a timer thread (`persist.c:173 fsync_worker`) fdatasyncs every `wal_sync_ms`. `persist_append` does **not** fsync before returning; the standby does **not** fsync before sending its `WalAck`.
|
||||
- **The commit engine advances `committed_lsn` on majority *ACK*** (`commit.c:303 commit_on_ack`, `repl_wait_majority_ack`). An ACK only proves the entry reached the OS page cache — it can be durable on **zero** disks at the moment it is declared committed.
|
||||
- **Recovery silently truncates to nothing.** When killed nodes restart, `persist_recover` hits a CRC chain break at a zeroed/never-fsync'd region (`ctrl-3.log:1597` `prev_crc=00000000` at offset 40) and stops replay at ~0 entries — but the node still becomes eligible.
|
||||
- **A hollow node wins and overwrites the good timeline.** With durability lost on the killed nodes and failover quorum collapsed to `required=1`, the freshness gate cannot fence the stale candidate. ctrl-3 became PRIMARY term=4 advertising `committed_lsn=21` over an empty job table (`ctrl-3.log:1612`); the node that *did* hold the data (ctrl-1, applied_lsn=22) converged **down** to it (`ctrl-3.log:1618`). LSN ≤20 discarded forever.
|
||||
|
||||
This violates the Leader-Completeness assumption documented in `src/controller/CLAUDE.md` ("committed entries are on a majority" presumes majority-ACK ⇒ durable-on-a-majority) and the TLA+ mapping `log[i] = WAL on disk`.
|
||||
|
||||
## Desired Outcome
|
||||
1. An operation reported committed is **durable on a majority of nodes** before it is declared committed — surviving simultaneous crash + restart of any minority.
|
||||
2. A node that cannot recover its WAL to a valid committed tip is **fenced from leadership** (fail-loud), never silently truncating to zero and then leading; the cluster never converges onto a timeline below its known committed high-water mark.
|
||||
|
||||
## Approach
|
||||
Two coupled requirements (chosen scope: *Durability + promotion fence*):
|
||||
|
||||
- **R1 Durable commit (fsync-before-ACK):** the standby `fdatasync`s the appended WAL bytes **before** emitting `WalAck`; the primary counts its own vote toward majority only after its local `fdatasync`. `committed_lsn` advances only on a **durable** majority. Preserve throughput via group-commit / batched fdatasync (amortize the sync across pending appends) rather than one sync per entry. Keep `wal_sync_ms` async path only for non-committing/best-effort writes, if any remain.
|
||||
- **R2 Recovery & promotion fence:** a WAL CRC/replay failure that prevents reconstructing a contiguous committed tip makes the node **ineligible to become primary** until it has caught up from a healthy peer (FATAL or fence, never silent truncate-to-zero-then-lead). A node must not win election or self-promote advertising a `committed_lsn` below the cluster's known committed high-water mark, even under degraded single-voter quorum.
|
||||
|
||||
Honor the user's fail-loud preference: surface the real durability/recovery failure rather than masking it.
|
||||
|
||||
## Scope
|
||||
- **In**: fsync-before-ACK on the standby apply path and primary self-count; durable-majority gating of `committed_lsn`; group-commit batching to bound the latency cost; fencing an un-recovered/corrupt-WAL node from promotion; preventing a sub-high-water-mark leader from being elected under degraded quorum; unit tests for each; the 3-node live E2E that reproduces this incident (kill-primary-under-load, assert zero committed-job loss).
|
||||
- **Out**: WAL wire format / ACK message shape (`ha-wal-replication`); per-LSN majority math (`ha-commit-engine` — consumed, the *durability precondition* of its ACK input is what changes); demotion truncate/self-restart and rejoin reconciliation of an *intact* log (`ha-convergence-recovery`); promotion no-op / own-term gate (`ha-single-leader-role`); raw vote-grant arithmetic (`raft-library` — this spec adds the durability+high-water *precondition*, not new vote math); entity-bearing snapshots (roadmap); client-side failover (`libxce-failover`).
|
||||
|
||||
## Boundary Candidates
|
||||
- Durable-commit seam: standby `fdatasync`-before-`WalAck` + primary self-fdatasync before majority self-count (the "ACK ⇒ durable" guarantee).
|
||||
- Promotion-eligibility seam: recovery-integrity + committed-high-water check gating `raft_cb_become_leader` / PreVote eligibility.
|
||||
|
||||
## Out of Boundary
|
||||
- Tuning the async `fsync_worker` cadence as a "fix" (it is the defect for committing writes, not a knob to turn).
|
||||
- Changing how a majority is *counted* (commit-engine) or how votes are *granted* (raft-library) beyond adding the durability/high-water precondition.
|
||||
- Worker-reported fact resync across restart (`worker-rejoin-resync`, roadmap backstop).
|
||||
|
||||
## Upstream / Downstream
|
||||
- **Upstream**: `ha-commit-engine` (committed_lsn semantics it now gates on durable ACKs), `ha-wal-replication` (ACK/catch-up machinery this makes durability-correct), `persist.c` fsync infrastructure, `ha-prevote` + vote-freshness (bug-339) eligibility path it strengthens.
|
||||
- **Downstream**: `ha-convergence-recovery` E2E (its single-outcome guarantee depends on this), `component-failover-adoption` and any real failover deployment.
|
||||
|
||||
## Existing Spec Touchpoints
|
||||
- **Extends/strengthens**: `ha-wal-replication` (durability precondition on ACK), `ha-commit-engine` (durable-majority), `ha-convergence-recovery` (assumes intact persisted committed state — this spec makes that assumption true).
|
||||
- **Adjacent (avoid overlapping)**: `ha-single-leader-role`, `raft-library`, `ha-prevote`.
|
||||
|
||||
## Constraints
|
||||
- C / gnu17, `src/controller/persist.c`, `commit.c`, `repl.c`, `ha.c`, `cluster.c`; TDD mandatory (tests in `testsuite/` before C changes).
|
||||
- Async-I/O rule: never block the iomgr thread — fdatasync must not stall heartbeat/vote handling; use the dedicated persist/fsync thread + group-commit and signal the commit waiter on completion.
|
||||
- Must verify against `raft/raft.tla` and the `src/controller/CLAUDE.md` Protocol Verification Checklist (commitIndex monotonic; leader holds all committed entries).
|
||||
- Live E2E needs `/xce-deploy-ha` (cluster at `/tmp/xce-ha-deploy`); reproduction harness = `xce_test` driving jobsub + primary kills. An automated soak reproduction now exists at `src/tools/test-commit-durability.sh` (rolling SIGKILL+restart under load, asserts every acked job survives).
|
||||
- Performance: durable-majority adds sync latency to the submit path — group-commit is required, not optional.
|
||||
|
||||
## Reproduction Findings (2026-06-17, from building the E2E)
|
||||
- **bug-661 is a RACE, not a deterministic failure.** `wal_sync_ms` is hardcoded to `DEFAULT_WAL_SYNC_MS = 10` ms (`config.h:86`) with **no CLI/env/config override**. The committed-but-not-fsync'd window is ~10ms, so a single-node SIGKILL+restart only loses a committed job when the kill lands inside that window AND the truncated node re-leads before catch-up repairs its WAL. The soak reproduced real loss once (4 committed jobs) but passed an 8-round run clean — confirming the rarity.
|
||||
- **Required test seam (NEW spec item): a configurable fsync interval** (e.g. `--wal-sync-ms`, plumbed into `persist_config_t.wal_sync_ms`). Setting it to several seconds makes every committed entry provably un-fsync'd at kill time, turning the racy soak into a **deterministic** RED. This seam is also the natural way to unit-test fsync-before-ACK. None exists today — add it as part of Requirement 1's testability (cf. the convergence spec's XCE_TESTING restart seam).
|
||||
- **Acceptance asymmetry**: a soak FAIL proves the bug; a soak PASS does not prove durability (the race may simply not have fired). After the fix, the soak must pass *deterministically regardless of timing* — that, plus the deterministic `--wal-sync-ms` test, is the real acceptance bar.
|
||||
- **Amplifier confirmed**: loss needs a killed+restarted node to re-lead with a truncated WAL *before* replication catch-up repairs it. The fix must therefore also fence a node that has not caught up to the committed high-water mark from leading (Requirement 2).
|
||||
|
||||
## Design Decision — durability mechanism (2026-06-17, with user)
|
||||
|
||||
**Invariant (user-stated, authoritative):** when the controller replies **ok** to a client, the operation's WAL entry MUST already be persisted (fdatasync'd) on a fault-tolerant majority — self + ⌊N/2⌋ standbys.
|
||||
|
||||
**Exact defect located:** `commit.c:majority_candidate_locked` counts the primary's own contribution as `cap = highest_appended_lsn(p)` — every *appended* (page-cache) entry, not every *fdatasync'd* one. On 3 nodes a commit needs self + 1 standby; the standby ACK is durable (`persist_wal_write_replicated:515` fdatasyncs before replying) but the primary's copy is **not** (it relies on the `fsync_worker` timer). So a "committed" entry is durable on **exactly 1 node** → zero durability fault-tolerance. That is bug-661.
|
||||
|
||||
**Chosen direction (user): synchronous persistence — eliminate `fsync_worker` as the durability mechanism.**
|
||||
- The submit path **already blocks off the iomgr thread** waiting for majority replication (`job.c:568-569`; `persist_commit_barrier → commit_submit(COMMIT_MODE_WAIT)`), so adding the primary's own `fdatasync` on that path is liveness-safe — it does not stall heartbeats.
|
||||
- Minimal correctness change: (1) primary `fdatasync`s its entry before its self-contribution covers that LSN (symmetric with the standby path at `persist.c:515`); (2) `commit.c` self-contribution uses a `durable_lsn` (highest fdatasync'd), not `highest_appended_lsn`; (3) reply stays gated on the commit barrier → reply-ok ⇒ durable-majority.
|
||||
- **Throughput**: per-entry fdatasync. The dominant amplifier (a job array fanning out to N entries → N fsyncs) is removed by the roadmap item **job-array-single-wal-record** (one WAL record per array → one fsync). Cross-reference that item; this spec assumes it lands or is acceptable interim.
|
||||
- **Group-commit IS in scope (revised 2026-06-18 at design-review).** Initially deferred, but `/kiro-validate-design` found the highest-volume WAL path is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage) — not array-collapsible, not covered by job-array-single-record — so a pure per-entry synchronous design penalizes the dominant writes. Decision (with user): repurpose `fsync_worker` into a demand-driven group-commit flusher (batch `fdatasync`, advance `durable_lsn`, wake commit-eval) as part of this spec. The roadmap "WAL group-commit batching" item is absorbed here.
|
||||
- **Open for design:** whether to delete the `fsync_worker` thread entirely or retain a minimal best-effort flush for any non-client-acked / local-only writes (e.g. checkpoint markers already fsync inline; audit `wal_classify` for any write that gates no reply). Confirm no client-acked op bypasses the commit barrier.
|
||||
|
|
@ -1,498 +0,0 @@
|
|||
# Design Document — ha-commit-durability
|
||||
|
||||
## Overview
|
||||
|
||||
**Purpose:** Make acknowledgment imply durability in the XCE HA controller. Before this spec, the controller advanced its commit position on a majority of replication **acknowledgments**, but the primary counted its own contribution from a page-cache write (`commit.c:majority_candidate_locked` used `highest_appended_lsn`), while background `fdatasync` ran on a timer (`fsync_worker`). A "committed" operation could therefore be durable on as few as one member — bug-661, observed as permanent job loss across failover. (Post-implementation, `majority_candidate_locked` caps the self-contribution at `persist_durable_lsn(ce->p)` — verified current in `src/controller/commit.c`.)
|
||||
|
||||
**Users:** Cluster operators and every client whose acknowledged job/queue/worker operations must survive controller failover.
|
||||
|
||||
**Impact:** Strengthens the meaning of "committed" from "majority-ACKed (volatile)" to "persisted to stable storage on a fault-tolerant majority." The client reply path is already gated on the commit barrier, so once `committed_lsn` implies a durable majority, the invariant holds with no change to the reply path. Durability becomes **group-commit**: `fsync_worker` is repurposed from a fire-and-forget timer into a demand-driven flusher that, on append, batches `fdatasync` across all pending entries, advances `durable_lsn`, and wakes the commit evaluation. The commit majority counts `durable_lsn` (not appended). This amortizes one `fdatasync` across many entries — important because the highest-volume WAL traffic is `WAL_MODE_RETAIN` facts (job start/finish/fail/rusage), which are not client-latency-bound but must be durable before `committed_lsn` passes them. A second concern — a member with a corrupt or truncated log becoming authoritative — is closed by a recovery-integrity verdict and a promotion fence.
|
||||
|
||||
### Goals
|
||||
- **Durable commit:** when the controller replies *ok*, the operation's log entry is `fdatasync`'d on the acknowledging member plus ⌊N/2⌋ others (Req 1).
|
||||
- **Recovery integrity & promotion fence:** a member that cannot reconstruct its committed state, or that lags the cluster's committed high-water mark, is barred from leadership until caught up — fail-loud when it cannot recover safely (Req 2).
|
||||
- **Deterministic verification:** durability timing is controllable in a test mode so the guarantee is provable without racing the clock (Req 3).
|
||||
- Preserve cluster liveness: durability work never stalls heartbeat/election processing.
|
||||
|
||||
### Non-Goals
|
||||
- Single-record job-array persistence (`job-array-single-wal-record`, roadmap) — complements this spec by bounding per-array fsync count; independent of the group-commit flusher introduced here.
|
||||
- Per-entry synchronous fsync as the steady-state mechanism — superseded by group-commit (decision: the high-volume `WAL_MODE_RETAIN` path needs batching; see Components/DurableCommit).
|
||||
- Replication/ACK wire format (`ha-wal-replication`), per-entry majority arithmetic (`ha-commit-engine`), demotion truncate / rejoin reconciliation of an intact log (`ha-convergence-recovery`), promotion no-op / own-term gate (`ha-single-leader-role`), raw vote-grant arithmetic (`raft-library`).
|
||||
|
||||
## Boundary Commitments
|
||||
|
||||
### This Spec Owns
|
||||
- The **durability precondition** on the commit majority: a member contributes to the commit majority for an LSN only once that LSN is `fdatasync`'d locally. Introduces and is authoritative for `persist_t.durable_lsn`.
|
||||
- **Group-commit durability**: `fsync_worker` repurposed from a timer into a demand-driven flusher that batches `fdatasync`, advances `durable_lsn`, and wakes commit evaluation; `persist_wal_async` signals the flusher instead of deferring to a timer. Benefits all members on the high-volume `WAL_MODE_RETAIN` path.
|
||||
- **Recovery integrity verdict**: `replay_wal`/`persist_recover` distinguish a clean log end from a corruption-truncated end, and re-derive the member's durable floor from the recovered WAL.
|
||||
- **Promotion eligibility fence**: a recovery-truncated member, or one whose durable position is below the cluster committed high-water mark, is ineligible to become primary until caught up; re-eligible on catch-up.
|
||||
- **Durable-backed freshness advertisement**: a candidate's advertised log position and a voter's freshness floor derive from durable WAL state, not from volatile in-memory positions that survive a truncating restart.
|
||||
- **Durability verification seam**: a configurable durability cadence / background-flush control plus an `XCE_TESTING` hook.
|
||||
|
||||
### Out of Boundary
|
||||
- Counting the majority or advancing `committed_lsn` arithmetic (`ha-commit-engine` — consumed; this spec changes only the *durability precondition* of its inputs).
|
||||
- The reply ordering / classification sweep (`ha-commit-barrier-rollout` — reused unchanged).
|
||||
- Demotion truncate + self-restart and rejoin reconciliation of an **intact** log (`ha-convergence-recovery`). This spec fences **promotion**; convergence-recovery handles **demotion** and **rejoin**.
|
||||
- Vote-grant arithmetic and term handling (`raft-library`); this spec adds only the durability/high-water **precondition** to the existing freshness gate.
|
||||
- WAL group-commit batching; job-array single-record persistence.
|
||||
|
||||
### Allowed Dependencies
|
||||
- `ha-commit-engine`: `commit_engine_t`, `commit_submit`, `majority_candidate_locked` (modify self-contribution input only).
|
||||
- `ha-wal-replication`: standby ACK path (`persist_wal_write_replicated`) and catch-up streaming (consumed; `in_catch_up` reused).
|
||||
- `ha-commit-barrier-rollout`: `persist_commit_barrier` reply gate (consumed unchanged).
|
||||
- `persist_t` internals (`current_lsn`, `committed_lsn`, `last_crc`, segment I/O), `ha_state_t` (`applied_lsn`, `last_known_committed_lsn`, `in_catch_up`, `term`).
|
||||
- Constraint: must not write/persist `committed_lsn` to disk (Raft keeps the commit index volatile); durability is about the **log**, not the commit index.
|
||||
|
||||
### Revalidation Triggers
|
||||
- Change to the `durable_lsn` semantics or to `majority_candidate_locked`'s self-contribution → re-check `ha-commit-engine` and `ha-convergence-recovery` (which assume durable committed state).
|
||||
- Change to the promotion-eligibility gate or freshness-floor source → re-check `ha-single-leader-role`, `ha-prevote`, `raft-library`, and the `CLAUDE.md` TLA+ checklist.
|
||||
- Removal of `fsync_worker` → re-check any caller that assumed deferred fsync (audit `wal_classify`).
|
||||
- New `wal_sync` override surface → re-check `main.c` flag conventions and deployment scripts.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Existing Architecture Analysis
|
||||
|
||||
- **Commit gate already off the iomgr thread:** the AFTER_EVENT hook fires before the reply and `persist_commit_barrier → commit_submit(COMMIT_MODE_WAIT)` blocks on a condvar (`job.c:2214`, `do_jobcontrol_op`; AFTER_EVENT hook itself e.g. `job.c:2189`). (Note: citation corrected — file has grown substantially since this spec closed; original cited `job.c:568-569`.) Adding `fdatasync` to this path is liveness-safe.
|
||||
- **Standby already persists-before-ACK:** `persist_wal_write_replicated` `fdatasync`s before returning (`persist.c:1417`, function defined at `persist.c:1193`). The primary is the asymmetric gap. (Note: citation corrected from the original `persist.c:515`.)
|
||||
- **Recovery is silent-truncating:** `replay_wal` (`persist.c:4624`) `break`s on CRC chain break / mismatch / short read (now around `persist.c:4704-4784`, several distinct break sites) and returns a verdict; `persist_recover` never sets `committed_lsn` (stays 0). (Note: citation corrected — original cited `persist.c:1021/1050/1042`; `replay_wal` has since moved substantially within the file.)
|
||||
- **Promotion has no integrity gate:** `raft_cb_become_leader` (`ha.c:1615`) computes `safe_lsn` from votes (`ha.c:1641-1644`) then `persist_truncate_wal_above(safe_lsn)` (`ha.c:1662`) with no check that the local log is intact or ≥ `safe_lsn`. (Note: this describes the pre-spec baseline; citation corrected from the original `ha.c:572/577`. The gate this spec adds is now implemented immediately after, at `ha.c:1699-1822`.)
|
||||
- **Freshness floor is volatile:** `ha_vote_self_freshness = max(applied_lsn, last_known_committed_lsn)` (now `ha.c:5310`, corrected from the original `ha.c:2127`), in-memory atomics not re-derived from durable WAL.
|
||||
- **Test-seam convention:** `#ifdef XCE_TESTING` hook shims are now spread across dozens of sites in `ha.c` (the file has roughly doubled since this spec closed); `wal_sync_ms` was a hard constant (`config.h:86`, still `DEFAULT_WAL_SYNC_MS = 10`) with no override — this spec added the `--wal-sync-ms` override (confirmed live in `main.c`/`persist.h`). (Note: the original `ha.c:2270-2335` range no longer corresponds to a single representative hook-shim cluster.)
|
||||
|
||||
### Architecture Pattern & Boundary Map
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph RequestPath
|
||||
Hook[AFTER_EVENT hook off iomgr]
|
||||
Barrier[persist_commit_barrier]
|
||||
end
|
||||
subgraph Persistence
|
||||
Append[persist_wal_async sync fsync]
|
||||
DurLsn[durable_lsn]
|
||||
Recover[persist_recover integrity verdict]
|
||||
ReplApply[persist_wal_write_replicated standby fsync]
|
||||
end
|
||||
subgraph Commit
|
||||
Majority[majority_candidate_locked durable self]
|
||||
Committed[committed_lsn]
|
||||
end
|
||||
subgraph Election
|
||||
Fresh[freshness floor from durable WAL]
|
||||
Promote[raft_cb_become_leader promotion fence]
|
||||
end
|
||||
Hook --> Append
|
||||
Append --> DurLsn
|
||||
Append --> ReplApply
|
||||
DurLsn --> Majority
|
||||
ReplApply --> Majority
|
||||
Majority --> Committed
|
||||
Committed --> Barrier
|
||||
Barrier --> Reply[reply ok to client]
|
||||
Recover --> Fresh
|
||||
Recover --> Promote
|
||||
Fresh --> Promote
|
||||
DurLsn --> Promote
|
||||
```
|
||||
|
||||
**Architecture Integration**
|
||||
- **Selected pattern:** extend-in-place for the durability precondition (Req 1, 3) plus a small testable fence helper for promotion eligibility (Req 2). No new module — `durable_lsn` is one atomic on `persist_t`.
|
||||
- **Boundaries:** persistence owns `durable_lsn` and the recovery verdict; the commit engine consumes `durable_lsn`; election consumes the recovery verdict + durable-backed floor. No shared ownership of the commit index.
|
||||
- **Patterns preserved:** commit-barrier reply gate, `in_catch_up` fence concept, `XCE_TESTING` hooks, the standby persist-before-ACK already in place.
|
||||
- **Steering compliance:** never block the iomgr thread (fsync lands on the off-iomgr commit path); re-verify against `raft/raft.tla` (commit rule, leader completeness) per `CLAUDE.md`.
|
||||
|
||||
### Technology Stack
|
||||
|
||||
| Layer | Choice / Version | Role in Feature | Notes |
|
||||
|-------|------------------|-----------------|-------|
|
||||
| Backend / Services | C (gnu17), controller daemon | Durable commit, recovery integrity, promotion fence | Extends `persist.c`, `commit.c`, `ha.c`, `cluster.c` |
|
||||
| Data / Storage | Local WAL segments + `fdatasync` | Synchronous durability; integrity-verified recovery | `posix_fallocate`'d segments; `fdatasync` (data + pre-allocated metadata) |
|
||||
| Infrastructure / Runtime | `--wal-sync-ms` (new) + `XCE_TESTING` seam | Deterministic durability verification | Follows `--sync-timeout-ms` (`OPT_SYNC_TIMEOUT`) precedent |
|
||||
|
||||
## File Structure Plan
|
||||
|
||||
### Modified Files
|
||||
- `src/controller/persist.c` — `persist_wal_async`: append then **signal the flusher** (no inline per-entry fsync); convert `fsync_worker` from a timer wait into a **demand-driven group-commit flusher** (wake on append-signal, `fdatasync`, advance `durable_lsn` to the highest synced LSN, broadcast the commit-eval condvar); keep its thread/mutex/cond lifecycle in `persist_start`/`persist_stop`/`persist_fini`. `replay_wal`/`persist_recover`: return an integrity verdict; re-derive durable floor from the recovered WAL tip. `persist_wal_sync_append`: **resolved (task 2.4)** — re-routed rather than deleted. It now calls the guarded `sync_append_legacy_commit_store` (`persist.c:5617`), which only advances `committed_lsn` when `written_lsn <= durable_lsn` (regression-pinned by `test_legacy_sync_append_holds_committed_at_durable`). (Note: current state corrected — as of this audit `persist_wal_sync_append` (`persist.c:5639`) DOES have a live production caller, `submit_job_wal_entry_and_commit` (`persist.c:5990-6004`, wired by `raft-apply-decoupling`), contradicting this section's original "dormant, no production caller" framing; the durable-precondition guard was added as designed, so the resolution is correct even though the "dormant" premise no longer holds.)
|
||||
- `src/include/persist.h` — add `_Atomic uint64_t durable_lsn` and a recovery-verdict type to `persist_t`/the recover signature; keep `fsync_*` fields (now driving the group-commit flusher); add an override field to `persist_config_t` (or its setter). Confirmed current: `durable_lsn` at `persist.h:275`, `RECOVER_CLEAN`/`RECOVER_TRUNCATED` enum, `recover_decision_t`, `recovery_unrecoverable` flag all present as designed.
|
||||
- `src/controller/commit.c` — `majority_candidate_locked`: use `durable_lsn` (not `highest_appended_lsn`) for the self contribution `cap`. Confirmed current (`commit.c:135`, `persist_durable_lsn(ce->p)`).
|
||||
- `src/controller/ha.c` — `raft_cb_become_leader`: insert the pre-promotion eligibility gate; `ha_vote_self_freshness`: derive floor from durable WAL state; fence-release on catch-up completion. (Note: the durability-verification `XCE_TESTING` seam actually lives in `src/controller/persist.c` — `persist_test_set_force_fsync_fail` at `persist.c:1547`, `persist_test_durable_lsn_on_disk` at `persist.c:1610` — not in `ha.c` as this section originally stated.)
|
||||
- `src/controller/cluster.c` — vote / PreVote freshness comparison consume the durable-backed floor; promotion path honors the fence. (Note: citations corrected — real-vote comparison now at `cluster.c:544-557`, PreVote at `cluster.c:643-651`; original cited `cluster.c:380-396`/`467-474`.)
|
||||
- `src/include/config.h` — keep `DEFAULT_WAL_SYNC_MS`; reconcile the stale `50` in `persistence-and-ha-design.md`.
|
||||
- `src/controller/main.c` — parse `--wal-sync-ms` (and optionally `--wal-sync-disable`) into `persist_config_t`.
|
||||
- `src/controller/myapp.c` / `cluster.c` (`persist_config_defaults` callers) — apply the override before `persist_init`.
|
||||
|
||||
### New Files
|
||||
- `src/controller/testsuite/test_commit_durability.c` — unit/white-box tests for `durable_lsn` advancement, self-counts-durable, recovery verdict, promotion fence (Req 1, 2, 3).
|
||||
- `src/tools/test-commit-durability.sh` — **exists** (soak E2E); extend to drive the new `--wal-sync-*` seam for a deterministic mode.
|
||||
|
||||
> Each modified function keeps one responsibility: persistence owns durability + recovery verdict; commit owns majority math (consuming `durable_lsn`); election owns eligibility (consuming the verdict + floor).
|
||||
|
||||
## System Flows
|
||||
|
||||
### Durable commit (reply ok ⇒ persisted on a majority)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Client
|
||||
participant Primary
|
||||
participant Standby
|
||||
participant Flusher
|
||||
Client->>Primary: submit op
|
||||
Primary->>Primary: persist_wal_async pwrite then signal flusher
|
||||
Primary->>Standby: replicate entry
|
||||
Flusher->>Flusher: batched fdatasync then durable_lsn advance then wake commit
|
||||
Standby->>Standby: persist_wal_write_replicated fdatasync
|
||||
Standby-->>Primary: WalAck acked_lsn
|
||||
Primary->>Primary: majority_candidate_locked self equals durable_lsn plus standby acks
|
||||
Primary->>Primary: committed_lsn advance then commit barrier releases
|
||||
Primary-->>Client: reply ok
|
||||
```
|
||||
|
||||
Key decisions: the reply is released only after `committed_lsn` covers the entry, and `committed_lsn` now requires a **durable** self plus a **durable** standby majority. The `fdatasync` runs on the dedicated flusher thread (the repurposed `fsync_worker`), batched across all pending appends; the AFTER_EVENT caller blocks on the commit barrier (off iomgr) until `durable_lsn` + standby acks cover its entry, so heartbeats are unaffected (Req 1.6) and one `fdatasync` amortizes across the high-volume RETAIN-fact stream.
|
||||
|
||||
### Recovery integrity & promotion fence
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
Start[startup persist_recover] --> Replay[replay_wal]
|
||||
Replay --> Verdict{integrity verdict}
|
||||
Verdict -->|clean eof| Floor[set durable floor from WAL tip]
|
||||
Verdict -->|corruption truncated| Truncated[mark recovery_truncated]
|
||||
Floor --> Eligible[eligible to lead]
|
||||
Truncated --> CanCatchUp{healthy peer reachable}
|
||||
CanCatchUp -->|yes| Fence[fence from promotion until caught up]
|
||||
CanCatchUp -->|no| LoudFence[LOUD PERMANENT FENCE: non-promotable, STAYS ALIVE, keeps retrying catch-up — NEVER self-exit]
|
||||
LoudFence --> Caught
|
||||
Fence --> Caught{caught up past last committed boundary}
|
||||
Caught -->|yes| Eligible
|
||||
```
|
||||
|
||||
Key decisions: corruption is never silently accepted (Req 2.1). A truncated member is fenced from promotion and catches up from a healthy primary (Req 2.2, 2.6); if it cannot reach a consistent committed state, the controller logs a LOUD operator-visible alert and holds itself PERMANENTLY non-promotable while STAYING ALIVE and continuing to retry catch-up (Req 2.4) — it refuses to serve divergent state by refusing leadership, NOT by terminating. **A controller MUST NEVER self-exit** (`exit`/`abort`) as a designed response to an operational condition (user invariant, 2026-06-18): a self-exited controller needs manual admin recovery and degrades quorum meanwhile, whereas one that stays alive self-heals when a peer returns (Req 2.6). A candidate never advertises or leads beyond its durable WAL tip (Req 2.3), so a hollow node cannot win even under degraded quorum (Req 2.5).
|
||||
|
||||
## Requirements Traceability
|
||||
|
||||
| Requirement | Summary | Components | Interfaces | Flows |
|
||||
|-------------|---------|------------|------------|-------|
|
||||
| 1.1 | reply ok ⇒ durable on majority | DurableCommit | `commit barrier`, `durable_lsn` | Durable commit |
|
||||
| 1.2 | self counts only when durable | DurableCommit | `majority_candidate_locked` | Durable commit |
|
||||
| 1.3 | standby persists before ACK | DurableCommit (pin) | `persist_wal_write_replicated` | Durable commit |
|
||||
| 1.4 | no premature ok | DurableCommit | `persist_commit_barrier` | Durable commit |
|
||||
| 1.5 | survives minority crash+restart | DurableCommit | durable-majority property | Durable commit / soak |
|
||||
| 1.6 | durability work preserves liveness | DurableCommit | off-iomgr placement | Durable commit |
|
||||
| 2.1 | no silent truncate-accept | RecoveryIntegrity | `replay_wal` verdict | Recovery fence |
|
||||
| 2.2 | fence until caught up | PromotionFence | promotion gate, `in_catch_up` | Recovery fence |
|
||||
| 2.3 | never lead below high-water | PromotionFence + FreshnessFloor | `raft_cb_become_leader`, `ha_vote_self_freshness` | Recovery fence |
|
||||
| 2.4 | loud permanent non-promotable fence when unrecoverable (stays alive, NEVER self-exit) | RecoveryIntegrity | loud-fence policy + sticky `recovery_unrecoverable` flag | Recovery fence |
|
||||
| 2.5 | degraded quorum preserves invariant | DurableCommit + PromotionFence | durable-majority + fence | Both |
|
||||
| 2.6 | re-eligibility on catch-up | PromotionFence | fence-release hook | Recovery fence |
|
||||
| 3.1 | configurable durability cadence | DurabilityConfig | `--wal-sync-ms` | — |
|
||||
| 3.2 | recoverable after ungraceful kill in verify mode | DurabilityTestSeam | `XCE_TESTING` hook | soak deterministic mode |
|
||||
| 3.3 | identical external semantics in/out of mode | DurabilityTestSeam | seam affects timing only | — |
|
||||
|
||||
## Components and Interfaces
|
||||
|
||||
| Component | Layer | Intent | Req Coverage | Key Dependencies (P0/P1) | Contracts |
|
||||
|-----------|-------|--------|--------------|--------------------------|-----------|
|
||||
| DurableCommit | Persistence + Commit | Synchronous persist + self-counts-durable | 1.1–1.6, 2.5 | persist_t (P0), commit_engine (P0), repl ACK (P1) | State |
|
||||
| RecoveryIntegrity | Persistence | Verdict on recovery; durable floor; loud permanent non-promotable fence policy (stays alive, NEVER self-exit) | 2.1, 2.4 | replay_wal (P0) | State |
|
||||
| PromotionFence | Election | Eligibility gate + fence-release | 2.2, 2.3, 2.6, 2.5 | RecoveryIntegrity (P0), in_catch_up (P1) | State |
|
||||
| FreshnessFloor | Election | Durable-backed vote floor & advertisement | 2.3 | durable_lsn (P0), ha_state (P1) | State |
|
||||
| DurabilityConfig + TestSeam | Runtime / Test | Override cadence; verification hook | 3.1–3.3 | persist_config (P0), XCE_TESTING (P1) | State |
|
||||
|
||||
### Persistence + Commit
|
||||
|
||||
#### DurableCommit
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Make the primary persist before it contributes to the commit majority |
|
||||
| Requirements | 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 2.5 |
|
||||
|
||||
- `persist_wal_async` appends and signals the flusher; it does **not** fsync inline per entry.
|
||||
- The **group-commit flusher** (repurposed `fsync_worker`) wakes on the append-signal, performs one `fdatasync`, advances `durable_lsn` to the highest synced LSN, and broadcasts the commit-eval condvar — amortizing one sync across all entries pending at wake time (the high-volume RETAIN stream benefits most).
|
||||
- `majority_candidate_locked` caps the self contribution at `durable_lsn`, not `highest_appended_lsn`.
|
||||
- Standby persist-before-ACK is preserved and regression-pinned (`persist_wal_write_replicated`, defined at `persist.c:1193`, `fdatasync` call at `persist.c:1417`; note corrected from the original `persist.c:515`); the standby path may later adopt the same flusher (out of scope here).
|
||||
- Checkpoint markers retain their existing inline `fdatasync` (`persist.c:1761`; note corrected from the original `persist.c:573`, which now falls inside unrelated term-index code added by `ha-term-safety`).
|
||||
- `durable_lsn` is monotonic, never exceeds the live segment tip; on `persist_truncate_wal_above`, clamp `durable_lsn` to the new tip.
|
||||
|
||||
**Dependencies**
|
||||
- Inbound: AFTER_EVENT hooks (`job.c`, `queue.c`, `worker.c`, `join.c`), `ha.c` WAL_NOOP — purpose: append path (P0).
|
||||
- Outbound: `commit_engine_t` — purpose: self contribution input (P0).
|
||||
|
||||
**Contracts**: State [x]
|
||||
|
||||
##### State Management
|
||||
- **State model:** `_Atomic uint64_t durable_lsn` on `persist_t`; invariant `durable_lsn <= current_lsn - 1`, monotonic non-decreasing except clamp-on-truncate.
|
||||
- **Persistence & consistency:** `durable_lsn` advances only after a successful `fdatasync` in the flusher; the commit engine reads it for self-coverage so `committed_lsn` ⇒ durable-majority.
|
||||
- **Concurrency strategy:** the flusher takes `p->lock` to read the appended tip and `fdatasync`, advances `durable_lsn`, releases, then signals commit-eval; `majority_candidate_locked` reads `durable_lsn` via atomic load. Define the append-signal → flush → commit-wake ordering precisely so a WAIT caller cannot miss a wake (lost-wakeup guard).
|
||||
|
||||
**Implementation Notes**
|
||||
- Integration: the WAIT-mode caller blocks on the commit barrier off the iomgr thread; only the flusher thread blocks on `fdatasync`. Heartbeats unaffected.
|
||||
- Validation: assert `committed_lsn <= durable_lsn` on the local member after every commit advance (debug build); verify the flusher batches (one fsync advances `durable_lsn` past multiple LSNs under load).
|
||||
- Risks: **lost-wakeup / batching-window correctness** between `persist_wal_async` signal and the flusher (an entry must never be considered durable before its `fdatasync`); design the signal as level-triggered on the appended-vs-synced gap, not edge-triggered. Bounded flush latency vs throughput is a tunable (`--wal-sync-ms` becomes the max batching delay, not a durability gap).
|
||||
|
||||
### Persistence — RecoveryIntegrity
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Distinguish clean log end from corruption; re-derive durable floor; loud permanent non-promotable fence when unrecoverable (stays alive, NEVER self-exit) |
|
||||
| Requirements | 2.1, 2.4 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- `replay_wal`/`persist_recover` return a verdict: `RECOVER_CLEAN` (reached a clean entry boundary) vs `RECOVER_TRUNCATED` (stopped at CRC break / mismatch / short read).
|
||||
- On `RECOVER_TRUNCATED`, the member is marked recovery-truncated and is NOT treated as a clean recovery; it must not advertise or lead beyond the last intact LSN.
|
||||
- The recovered durable floor (highest intact replayed LSN) is published so the freshness advertisement is durable-backed.
|
||||
- Loud-fence policy (NEVER self-exit, user invariant 2026-06-18): if a recovery-truncated / behind member cannot reach a healthy peer to catch up, the controller logs a LOUD operator-visible alert (truncation boundary vs cluster HWM) via `persist_recover_loud_fence_if_unrecoverable`, sets a sticky `recovery_unrecoverable` flag, and holds itself PERMANENTLY non-promotable while STAYING ALIVE and continuing to retry catch-up. It refuses to serve divergent state by refusing leadership — it does NOT `exit`/`abort`. The flag clears (and the fence releases → eligible, Req 2.6) automatically when a peer returns and catch-up reaches a committed-consistent state.
|
||||
|
||||
**Contracts**: State [x]
|
||||
|
||||
##### State Management
|
||||
- **State model:** verdict enum + `recovery_truncated` flag (on `ha_state_t` or `persist_t`); recovered durable floor = highest intact LSN.
|
||||
- **Persistence & consistency:** does not persist `committed_lsn`; only re-derives the durable log tip from on-disk segments.
|
||||
- **Concurrency strategy:** runs single-threaded during `cluster_init_phase2` before the node participates.
|
||||
|
||||
**Implementation Notes**
|
||||
- Integration: composes with `ha-convergence-recovery` — that spec truncates an **intact** log on demotion/rejoin; this spec refuses to accept a **corrupt** log silently.
|
||||
- Validation: unit test injects a CRC break at LSN 1 and asserts verdict `RECOVER_TRUNCATED` + non-promotability.
|
||||
- Risks: must not loud-fence on a legitimately empty first-boot WAL (no entries ≠ corruption).
|
||||
|
||||
### Boot Policy Decision Table (task 3.2)
|
||||
|
||||
> **Status:** DECISION recorded (task 3.2); **terminal action corrected by the never-self-exit rework (2026-06-18).** The original table's `FATAL` outcome (a self-`abort`) was REPLACED: the unrecoverable terminal is now a **LOUD PERMANENT FENCE** — non-promotable, **stays alive**, keeps retrying catch-up. This table is the contract that task **3.3** (the loud-fence terminal) and task **4.3** (the promotion fence + release) implement.
|
||||
|
||||
**Decision (Req 2.2, 2.4):** A member that boots with a structurally clean log but below the cluster's committed high-water mark, or with a recovery-**truncated** log, is **fenced from leadership** at boot (never promoted while behind/truncated). It then **attempts catch-up** from a healthy peer. It becomes eligible **only** once catch-up brings its durable position to a committed-consistent state at/above the cluster boundary. If it cannot reach that state (no healthy peer reachable / cannot reach a committed-consistent state), it enters a **LOUD PERMANENT FENCE**: it logs a loud, operator-visible alert (truncation boundary vs cluster HWM), holds itself **permanently non-promotable**, and **stays alive while continuing to retry catch-up** — it **NEVER self-terminates**. The fence releases (→ ELIGIBLE) automatically once catch-up reaches a committed-consistent state.
|
||||
|
||||
**Invariant (authoritative):** A truncated/behind member always resolves to exactly one of two outcomes: it **catches up → ELIGIBLE**, or it enters a **LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying catch-up — NEVER self-exit)**. It is never silently divergent (it refuses to lead), and the process never self-`exit`s/`abort`s as a designed response to this operational condition (user invariant, 2026-06-18) — a self-exited controller needs manual admin recovery and degrades quorum, whereas one that stays alive self-heals when a peer returns (Req 2.6). Snapshot-sentinel recovery is treated as CLEAN-but-behind (catch-up), never the unrecoverable terminal on a legitimate sentinel.
|
||||
|
||||
**Inputs (evaluated at boot / promotion-eligibility):**
|
||||
- **verdict** — recovery integrity verdict from task 3.1: `CLEAN` vs `TRUNCATED`.
|
||||
- **durable-vs-HWM** — local durable position relative to the cluster committed high-water mark: `at/above` vs `below`.
|
||||
- **peer** — a healthy peer reachable for catch-up: `yes` vs `no`.
|
||||
- **catch-up** — catch-up reaches a committed-consistent state within the bounded window: `succeeds` vs `fails` (n/a when no catch-up is attempted).
|
||||
|
||||
**Outcomes:**
|
||||
- **ELIGIBLE** — may promote (no fence).
|
||||
- **FENCE+CATCH-UP → ELIGIBLE** — ineligible to promote; catch up from a healthy peer; regain eligibility on success (Req 2.6).
|
||||
- **LOUD PERMANENT FENCE** — log a loud operator-visible alert (truncation/boundary), set the sticky `recovery_unrecoverable` flag, and hold the member permanently non-promotable while it **stays alive** and keeps retrying catch-up — **NEVER self-exit**; releases → ELIGIBLE on catch-up (Req 2.4, 2.6).
|
||||
|
||||
| # | verdict | durable-vs-HWM | healthy peer | catch-up result | Outcome | Rationale / Req |
|
||||
|---|---------|----------------|--------------|-----------------|---------|-----------------|
|
||||
| 1 | CLEAN | at/above | n/a | n/a | **ELIGIBLE** | Log intact and not behind — normal promotable member. (Req 2.3) |
|
||||
| 2 | CLEAN | below | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Intact but behind the HWM; must not lead below the boundary; catch up then promote. (Req 2.2, 2.3, 2.6) |
|
||||
| 3 | TRUNCATED | (below — a truncated log is by definition not provably at/above) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | Structurally truncated but a healthy peer exists; fence, catch up, regain eligibility. Never the unrecoverable terminal when a peer is reachable. (Req 2.2, 2.6) |
|
||||
| 4 | TRUNCATED | below | no | fails (no committed-consistent state reachable) | **LOUD PERMANENT FENCE** (non-promotable, stays alive, keeps retrying catch-up — NEVER self-exit) | Cannot recover locally AND no peer to catch up from ⇒ log loudly + hold non-promotable + alive; releases → ELIGIBLE when a peer returns and it catches up. NEVER `exit`/`abort`. (Req 2.4, 2.6) |
|
||||
| 4b | CLEAN | below | no | fails (no peer to advance from) | **LOUD PERMANENT FENCE** (non-promotable, stays alive, keeps retrying catch-up — NEVER self-exit) | Behind the HWM and no healthy peer to catch up from — same non-terminating loud-fence as row 4; NEVER self-exit. (Req 2.4, 2.2, 2.6) |
|
||||
| 5 | CLEAN (snapshot-sentinel: committed at sentinel LSN, no sub-sentinel entries) | below (no entries below the sentinel) | yes | succeeds | **FENCE+CATCH-UP → ELIGIBLE** | A legitimate sentinel snapshot is CLEAN-but-behind, NOT corruption; treat as must-catch-up, **never the unrecoverable terminal** on a legitimate sentinel. (Req 2.1, 2.2) |
|
||||
| 6 | CLEAN | at/above (empty first-boot WAL, fresh cold start: durable=0, HWM=0) | n/a | n/a | **ELIGIBLE** | An empty first-boot WAL is not behind anything — cold-start members are promotable. (Req 2.1, 2.3) |
|
||||
|
||||
**Notes on the dimensions:**
|
||||
- A `TRUNCATED` verdict is never simultaneously "provably at/above the HWM": truncation means the log stops below its last intact boundary, so a truncated member is treated as behind for fence purposes regardless of the volatile in-memory position (this is why 4.2/3.4 derive the freshness floor from the **durable** WAL tip, not a volatile value).
|
||||
- the catch-up retry is **indefinite** (the member stays alive and keeps trying) and **loud** (a periodic operator-visible re-alert, controlled by `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS`); it is never silently divergent because the member refuses leadership the entire time. The previous "bounded window before FATAL" deadline was REMOVED by the never-self-exit rework — there is no deadline-driven termination.
|
||||
- The `healthy peer = no` + `at/above` combination does not need catch-up and is covered by row 1 (ELIGIBLE) — a member already at/above the HWM is promotable without a peer.
|
||||
|
||||
### Election — PromotionFence + FreshnessFloor
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Bar a truncated/behind member from leadership until caught up; advertise/vote on durable WAL state |
|
||||
| Requirements | 2.2, 2.3, 2.5, 2.6 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- Pre-promotion gate in `raft_cb_become_leader`: if recovery-truncated, or local durable position < cluster committed high-water mark, refuse promotion (set/extend `in_catch_up`-style fence) and instead catch up from a healthy primary.
|
||||
- `ha_vote_self_freshness`: floor derives from the **durable** WAL tip (re-derived on recovery), so a truncated member cannot over-advertise a volatile `last_known_committed_lsn`. Honors `CLAUDE.md`: the post-election safe-commit point still comes from voter replies, never from a stale ctrlhello.
|
||||
- Fence-release: when catch-up brings the member past its last-known committed boundary, restore eligibility (hook the existing catch-up completion / re-inclusion path).
|
||||
- Degraded quorum: because a candidate cannot lead beyond its durable tip, a hollow node cannot win even at minimal quorum — preserving Req 1's invariant across leadership changes.
|
||||
|
||||
**Contracts**: State [x]
|
||||
|
||||
**Implementation Notes**
|
||||
- Integration: reuses `in_catch_up` (`ha.c:917`, `cluster.c:713`; note corrected from the original `ha.c:954`/`cluster.c:529`) and the PreVote/vote freshness comparison (`cluster.c:544-557` real-vote, `cluster.c:643-651` PreVote; note corrected from the original `cluster.c:380-396`/`467-474`); extends the fence from "streaming standby" to "recovery-truncated / behind-high-water promoter."
|
||||
- Validation: simulate a node restarting truncated; assert it is denied promotion until catch-up, then re-enabled (Req 2.2/2.6).
|
||||
- Risks: interaction with `ha-single-leader-role` own-term gate and `ha-prevote` — re-verify the combined eligibility predicate against `raft.tla`.
|
||||
|
||||
### Freshness-Floor Formula (task 4.1)
|
||||
|
||||
> **Status:** PINNED (task 4.1, gating). This section is the authoritative contract that tasks **4.2** (implement the floor in `ha.c`/`cluster.c`) and **4.3** (promotion gate) implement against. **Verification method:** this is a **manual conceptual-alignment + Protocol Verification Checklist** argument, NOT a TLC run — there is no TLA+ toolchain in this environment (no `tla2tools.jar`/TLC/java). The argument maps the formula to `raft/raft.tla`'s `mlastLogIndex`/`LastTerm` semantics and to the `src/controller/CLAUDE.md` checklist. Per the spec scope note, `raft.tla` validates the **election/freshness logic** this change touches; it does **not** validate the *durability* of the floor — that is verified by tests 3.4 / 2.x, because `raft.tla` assumes a durable log (see Non-Goals and §5.3 of the broader Raft model).
|
||||
|
||||
#### 1. The pinned formula
|
||||
|
||||
Let, on member `M`:
|
||||
- `durable_floor(M) = persist_durable_floor(M)` — the durable WAL tip (task 3.4): the highest LSN whose bytes are `fdatasync`'d locally, re-derived on recovery as the **highest intact replayed LSN** (the truncation boundary on `RECOVER_TRUNCATED`, the clean tip on `RECOVER_CLEAN`). This is a durable **last-log index**, not the commit position.
|
||||
- `learned_committed(M) = last_known_committed_lsn(M)` — the volatile learned cluster commit high-water (a hint, possibly stale across a truncating restart).
|
||||
- `learned_tag(M) = committed_config_tag(M)`, `local_tag(M) = config_version_tag` — the config epoch (compared first; unchanged by this task).
|
||||
|
||||
**(a) What a candidate ADVERTISES (`mlastLogIndex` analog):**
|
||||
|
||||
```
|
||||
cand_lsn = durable_floor(M) // NOT max(applied_lsn, learned_committed)
|
||||
cand_tag = max(local_tag, learned_tag) // config epoch, unchanged
|
||||
```
|
||||
|
||||
The candidate advertises **exactly its durable WAL tip**. It does **not** advertise `applied_lsn` (which can exceed the durable tip — entries applied to volatile state but not yet `fdatasync`'d) and does **not** advertise `learned_committed` (a hint that, after a truncating restart, can exceed the durable tip the node actually holds). This is the core change: a candidate may only claim a log position it can durably back. `cand_tag` is unchanged — config-epoch freshness is orthogonal to log durability.
|
||||
|
||||
**(b) What a voter uses as its freshness FLOOR (the reconciliation):**
|
||||
|
||||
```
|
||||
voter_floor_lsn = max( durable_floor(M), learned_committed(M) )
|
||||
voter_floor_tag = max( local_tag(M), learned_tag(M) ) // unchanged
|
||||
```
|
||||
|
||||
The voter floor is the **max of its own durable tip and its learned cluster-committed high-water**. This is the precise reconciliation of the durable-derived floor against the existing `max(applied_lsn, last_known_committed_lsn)`:
|
||||
- `applied_lsn` is **replaced** by `durable_floor` (a node must not demand of a candidate a position it itself only holds volatilely; and symmetrically must not under-advertise — see §3/§4). Note `applied_lsn` and `durable_floor` use **different conventions and are not directly comparable as raw counters**: `applied_lsn` is advanced to `last_good_lsn` — the last contiguously-held LSN — via `ha_advance_applied_lsn` (`ha.c:3050`, called at `ha.c:3426-3432`), whereas `durable_floor`/`durable_lsn` is the actual durable LSN *watermark* (the highest LSN whose bytes are `fdatasync`'d). (Note: this audit found the convention itself has since changed — the code comment at `ha.c:3426-3429` cites a later fix, bug-991, that moved `applied_lsn` from `last_good_lsn + 1` [the *next-expected* index this section originally described, cited at the now-stale `ha.c:1234/1253`] to bare `last_good_lsn`, "not `last_good_lsn + 1`," because the `+1` wedged the leader's catch-up streamer. This does not affect the argument here — the two quantities were already established as not directly comparable, and the safety/liveness proof in §3/§4 does not rely on the exact `applied_lsn` convention.) We therefore do **not** rely on a raw `durable_floor ≤ applied_lsn` inequality. The claim that this term never raises the floor beyond what is reachable rests entirely on the `D*` argument in §4: the voter floor's local-state contribution `durable_floor(v)` is bounded above by `D*` (the most-durable reachable candidate's tip) by definition of `D*`, so swapping in `durable_floor` cannot push the floor above some reachable candidate ⇒ cannot newly induce livelock (§4).
|
||||
- `learned_committed` is **retained** as the safety backstop established by bug-339: a voter must never grant to a candidate below what the cluster has already **committed**, even if this voter's own durable tip is colder. Committed entries are now durable on a majority (Req 1/2.x), so at least a majority of voters carry that committed LSN in their `durable_floor` *and* every voter that learned the commit carries it in `learned_committed` — keeping the floor at the committed high-water on the voter side closes the window where a lagging voter would otherwise grant to a sub-committed candidate.
|
||||
|
||||
**(c) The exact comparison (candidate ≥ voter floor — `logOk` analog):**
|
||||
|
||||
```
|
||||
grant_freshness(cand, voter) ==
|
||||
cand_tag > voter_floor_tag
|
||||
\/ ( cand_tag == voter_floor_tag /\ cand_lsn >= voter_floor_lsn )
|
||||
```
|
||||
|
||||
i.e. deny iff `cand_tag < voter_floor_tag`, or `cand_tag == voter_floor_tag && cand_lsn < voter_floor_lsn`. Term-then-index ordering, matching raft's `LastTerm`-then-`mlastLogIndex` lexicographic order; here the config epoch (`tag`) is the high-order key (raft's `LastTerm`) and the durable LSN is the low-order key (raft's `mlastLogIndex`). Equality at both ⇒ grant (a node at least as fresh, including exactly as fresh, is grantable — preserves the existing `>=` semantics and idempotent re-grant).
|
||||
|
||||
**Candidate and voter use DIFFERENT quantities — this is the load-bearing distinction, and the implementation must keep the two paths separate:**
|
||||
|
||||
> **Citation note (this audit):** the site line numbers below are corrected to their current locations; `ha.c` and `cluster.c` have grown substantially since this spec closed (2026-07-18) from later work (`ha-term-safety`, `raft-apply-decoupling`, `job-array-single-wal-record`, and unlisted in-flight specs). The mechanism itself is unchanged and verified current — a dedicated helper `ha_candidate_advertise_lsn(persist_t *p)` (`ha.c:5347-5349`, added since this spec's original text, which described inline snapshotting instead) now wraps the bare `persist_durable_floor(p)` call for both candidate sites.
|
||||
|
||||
| Role | Site(s) | Quantity it must produce | How |
|
||||
|------|---------|--------------------------|-----|
|
||||
| **Candidate** (advertised `cand_lsn`) | `ha.c:7938` (real-vote snapshot), `ha.c:8017` (PreVote snapshot) — corrected from the original `ha.c:3660-3666`/`3735-3741` | **bare `durable_floor(M)`** — NOT folded with `learned_committed` | calls `ha_candidate_advertise_lsn(&c->persist)` (`ha.c:5347-5349`), a thin wrapper over `persist_durable_floor(p)`; do **NOT** route the advertised LSN through `ha_vote_self_freshness` |
|
||||
| **Voter** (freshness floor `voter_floor_lsn`) | `cluster.c:544-557` (real-vote), `cluster.c:643-651` (PreVote), `ha.c:5436` (`handle_request_vote`, the `XCE_TESTING`-only legacy test gate) — corrected from the original `cluster.c:388-395`/`471-475`/`ha.c:2243` | **`max(durable_floor(M), learned_committed(M))`** — the backstop is retained | keep calling `ha_vote_self_freshness`, changing only its `local_lsn` input from `applied_lsn` to `durable_floor(M)`; the function still folds in `last_known_committed_lsn` |
|
||||
|
||||
**Why the candidate must NOT reuse `ha_vote_self_freshness`.** That function (`ha.c:5310-5324`; note corrected from the original `ha.c:2127-2142`) *unconditionally* computes `*out_lsn = max(local_applied_lsn, last_known_committed_lsn)`. Feeding it `durable_floor` as the local input therefore yields `max(durable_floor, learned_committed)` — **not** the bare `durable_floor` §1a/§3 require. A truncated node with `durable_floor = 4` but a stale volatile `learned_committed = 8` would then advertise `cand_lsn = 8` and **win** — exactly the truncated-candidate-wins hole this task closes (§5a). So 4.2 must give the candidate sites a **candidate-only path** that calls `persist_durable_floor(p)` directly and assigns it to `cand_lsn`, bypassing the `learned_committed` fold. (The candidate's `cand_tag` is still `max(local_tag, learned_tag)`; only the **LSN axis** must bypass the fold.) The two PreVote/real-vote candidate snapshots currently call `ha_vote_self_freshness` for both fields — 4.2 splits the LSN field off onto the bare-`durable_floor` path while leaving the tag field on the existing max.
|
||||
|
||||
**Why safety lives on the advertisement axis and the backstop lives on the voter axis.** The candidate must not **over-advertise** beyond durable state — that is the safety requirement (§3): pinning `cand_lsn = durable_floor` is what denies a truncated candidate. The `learned_committed` backstop belongs on the **voter** axis (§3 dual guard): a voter must refuse any candidate below the cluster-committed boundary it learned, even when the voter's own durable tip momentarily lags. Folding `learned_committed` into the *candidate* advertisement would defeat the very safety property it protects on the *voter* side; folding it into the *voter* floor is exactly the bug-339 backstop. raft.tla confirms the split: `RequestVote` sends `mlastLogIndex |-> Len(log[i])` — the **bare candidate log length**, with no committed term folded into the candidate's advertisement; the committed-high-water guard lives only on the voter's `logOk` floor. This recipe is now internally consistent with §1a (bare candidate `durable_floor`), §3 (quorum-intersection on the advertised `durable_floor`), and §4 (voter floor bounded by `D*`).
|
||||
|
||||
**(d) `safe_committed_lsn` is UNCHANGED and still from voter replies.** This floor governs only **vote granting / advertisement**. The post-election `safe_committed_lsn` is still computed by `ha_compute_safe_committed_lsn` (`ha.c:1144`; note corrected from the original `ha.c:480/562`) from the `peerLastLogLsn` values in the `RequestVoteReply` set, never from this floor and never from a stale ctrlhello (CLAUDE.md "Post-Election Safe Commit Point"). The reply's `peer_last_log_lsn` field (assigned at `cluster.c:596` real-vote, `cluster.c:735` PreVote; note corrected from the original `cluster.c:426`) now carries the durable-backed `voter_floor_lsn`, which is *more* conservative (never above a durable position), so the elected leader's `safe_committed_lsn` is if anything tightened, never loosened — `persist_truncate_wal_above(safe_committed_lsn)` still only removes entries strictly above a majority-backed point.
|
||||
|
||||
#### 2. raft.tla alignment
|
||||
|
||||
`raft.tla` (`HandleRequestVoteRequest`, lines 284-289):
|
||||
|
||||
```
|
||||
logOk == \/ m.mlastLogTerm > LastTerm(log[i])
|
||||
\/ /\ m.mlastLogTerm = LastTerm(log[i])
|
||||
/\ m.mlastLogIndex >= Len(log[i])
|
||||
```
|
||||
|
||||
Mapping:
|
||||
- `Len(log[i])` (voter's last-log **index**) ↔ `voter_floor_lsn` on the durable axis. Our durable WAL tip `persist_durable_floor` is the faithful analog of `Len(log[i])`: in `raft.tla` the log is **restored intact from stable storage** on restart (the model has no volatile-vs-durable distinction — every entry in `log[i]` is durable by assumption). By making committed entries durable on a majority (2.x) and deriving the advertised/floor index from `persist_durable_floor` (the on-disk tip) rather than a volatile `applied_lsn`, our `cand_lsn`/`voter_floor_lsn` become the genuine "length of the durable log," so the `mlastLogIndex >= Len(log[i])` comparison holds with the same meaning the model assigns it.
|
||||
- `m.mlastLogIndex` (candidate's advertised index) ↔ `cand_lsn = durable_floor(candidate)` — exactly raft's `mlastLogIndex |-> Len(log[i])` in `RequestVote` (line 196): the candidate advertises *its own log length*, which in our durable-log mapping is its durable tip. The pre-change code advertised `max(applied_lsn, learned_committed)`, which could exceed `Len(durable log)` — an over-advertisement raft.tla cannot express (in the model you cannot advertise an index beyond `Len(log[i])`). The fix restores faithfulness to the model.
|
||||
- `LastTerm(log[i])` / `m.mlastLogTerm` ↔ the config epoch `tag`. XCE folds per-entry Raft terms into a monotone config-version tag (single-writer log; term changes ride the config epoch); the high-order lexicographic key is preserved. This task does not change tag handling.
|
||||
|
||||
**Caveat (restated):** `raft.tla` does **not** model durability — it assumes `log[i]` survives restart intact, so it cannot itself prove that `persist_durable_floor` *is* durable. That property (the WAL-tip derivation is genuinely on stable storage and survives an ungraceful kill) is the subject of tests 3.4 and 2.x, not this alignment argument. What `raft.tla` *does* validate — and what this section argues we faithfully implement — is the election/freshness **logic**: with our durable tip standing in for `Len(log)`, the `logOk` predicate and the Leader-Completeness proof carry over.
|
||||
|
||||
#### 3. Leader Completeness (safety) — a truncated/behind candidate cannot win
|
||||
|
||||
Claim: under this formula, any candidate that wins an election holds **all committed entries** (raft's Leader Completeness).
|
||||
|
||||
Argument:
|
||||
1. By Req 1/2.x, an entry committed at LSN `L` is `fdatasync`'d on a **majority** `Q_L` of members; each member in `Q_L` has `durable_floor ≥ L` (durability is a contiguous prefix watermark — CLAUDE.md Commit Ordering — so a durable tip ≥ L implies all entries ≤ L are durable on that member).
|
||||
2. A candidate `C` needs votes from a **majority** `Q_v`. Any two majorities intersect: `Q_L ∩ Q_v ≠ ∅`, so at least one voter `v` has `durable_floor(v) ≥ L`, hence `voter_floor_lsn(v) ≥ durable_floor(v) ≥ L`.
|
||||
3. For `v` to grant, the comparison requires `cand_lsn ≥ voter_floor_lsn(v) ≥ L`. Since `cand_lsn = durable_floor(C)` (the candidate cannot advertise above its durable tip), `durable_floor(C) ≥ L` — `C` durably holds entry `L` (again by the prefix-watermark property, all entries ≤ L).
|
||||
4. Therefore a candidate whose durable tip is **below** the committed high-water `L` (a truncated or behind node) is denied by every voter in the intersection and **cannot assemble a majority**. The pre-change bug was that `C` could advertise `learned_committed` (or volatile `applied_lsn`) above its durable tip and thereby satisfy step 3 *without durably holding `L`* — winning, then truncating committed work away. Pinning `cand_lsn = durable_floor(C)` closes exactly that hole.
|
||||
|
||||
The retained `learned_committed` term in the *voter* floor (§1b) is the dual guard: even a voter whose own durable tip momentarily lags `L` (e.g. just rejoined) still refuses a candidate below the cluster-committed `L` it learned, so the safety margin does not depend on which majority voted.
|
||||
|
||||
#### 4. Liveness (no election livelock) — the most-durable candidate can always win
|
||||
|
||||
Claim: the formula is not "too high" — among any available majority, the member holding the **longest durable log** can always be elected; the floor is always satisfiable by some reachable candidate, so there is no bug-339-style infinite election loop.
|
||||
|
||||
Argument:
|
||||
1. Let `D* = max{ durable_floor(m) : m in an available majority A }`, achieved by member `C*`. `C*` advertises `cand_lsn = D*`.
|
||||
2. The committed high-water `L` satisfies `L ≤ D*`: `L` is durable on a majority, that majority intersects `A`, so some member of `A` has `durable_floor ≥ L`, hence `D* ≥ L`.
|
||||
3. For any voter `v ∈ A`: `voter_floor_lsn(v) = max(durable_floor(v), learned_committed(v))`. We have `durable_floor(v) ≤ D*` by definition of `D*`. And `learned_committed(v) ≤ L ≤ D*` — a member never *learns* a committed LSN above the true committed high-water (`learned_committed` is a lower-or-equal estimate of `L`, advanced only from majority-acked commits / leader heartbeats). Therefore `voter_floor_lsn(v) ≤ D* = cand_lsn(C*)`, so the comparison `cand_lsn ≥ voter_floor_lsn(v)` **holds for every** `v ∈ A`, and `C*` is grantable by the whole majority.
|
||||
4. Hence the floor is always satisfiable by the most-durable reachable candidate ⇒ an election can always complete ⇒ no livelock.
|
||||
|
||||
Why this avoids the bug-339 pattern specifically: bug-339 was an *infinite* loop because the RDHUP failover path advertised a **stale `committed_lsn`** that no longer matched the floor the (other) voters demanded — the advertiser claimed a position it could not satisfy against the floor, so it kept losing and retrying. Under the pinned formula **no node advertises a position it cannot durably back** (`cand_lsn = durable_floor`), and the voter floor is bounded above by `D*` (step 3, because both `durable_floor(v) ≤ D*` and `learned_committed(v) ≤ L ≤ D*`). The floor is therefore always met by `C*`. The earlier bug-339 fix pinned the voter floor to `max(applied_lsn, last_known_committed_lsn)`; this task swaps the local-state term from the volatile `applied_lsn` to the durable `durable_floor`. Per §1b we do **not** rest on a raw `durable_floor ≤ applied_lsn` comparison (the two use different counter conventions); instead, the no-livelock guarantee rests solely on the `D*` bound of §4 — `durable_floor(v) ≤ D*` for every reachable voter `v` — so the swapped-in term can never push the voter floor above the most-durable reachable candidate `C*`. It therefore cannot reintroduce a too-high (livelock) floor while it closes the too-low (safety) hole on the advertisement side.
|
||||
|
||||
> **The safety/liveness tension is resolved, not papered over.** The "too low ⇒ truncated node wins" failure lives on the **advertisement** axis (what a candidate may claim) and is fixed by `cand_lsn = durable_floor` (§3). The "too high ⇒ livelock" failure lives on the **voter-floor** axis (what a voter demands) and is avoided because the voter floor is bounded above by `D*`, the most-durable reachable candidate's tip (§4). The two fixes act on different sides of the same comparison, so there is no contradiction: §3 raises the *floor for under-durable candidates* without raising it *beyond the best available candidate*.
|
||||
|
||||
#### 5. The two required cases
|
||||
|
||||
**(a) Truncated-candidate DENIED.** Node `T` restarts; its WAL hit a CRC break at LSN 5, so `replay_wal` returns `RECOVER_TRUNCATED` with `durable_floor(T) = 4` (highest intact replayed LSN). Cluster committed high-water `L = 8` (entries 5–8 are durable on the surviving majority `Q`). `T` becomes a candidate and advertises `cand_lsn = durable_floor(T) = 4` (it **cannot** advertise the stale `learned_committed = 8` it may still hold in volatile state — that is exactly the over-advertisement the formula forbids). Any voter `v ∈ Q` has `durable_floor(v) ≥ 8`, so `voter_floor_lsn(v) ≥ 8 > 4 = cand_lsn` ⇒ every such voter **denies** (§1c). `T` cannot reach a majority ⇒ not elected (Leader Completeness preserved, §3). Independently, the promotion fence (task 4.3) bars `T` from `raft_cb_become_leader` because `persist_recovery_truncated(T)` is true and `durable_floor(T) < L` — so even a degenerate quorum cannot promote it. `T` instead catches up from a healthy peer to `durable_floor ≥ 8`, then becomes both grantable and promotable (Req 2.6 / Boot Policy Decision Table row 3).
|
||||
|
||||
**(b) Longest-durable-log STILL ELECTS (no livelock).** Three nodes; the primary committed through **LSN 8**, so `L = 8`. Because committed ⇒ durable on a majority (Req 1/2.x), at least two of the three members hold `durable_floor ≥ 8`; the primary then died. The two survivors are `A` and `B`.
|
||||
|
||||
- *Sub-case L=8 is consistent with both survivors:* the committing majority for `L = 8` was some 2-of-3 set. If the dead primary was **not** in that majority, then both survivors `A` and `B` were in it ⇒ both have `durable_floor ≥ 8`. If the dead primary **was** in it, then exactly one of the two acking members survives; say `A` has `durable_floor = 8` while `B` ended cleanly one entry behind at `durable_floor = 7` (it appended/acked 8 against an earlier *uncommitted* round but its flusher had not advanced past 7 at crash — `B` is `RECOVER_CLEAN`, just behind, not truncated). Either way **some survivor has `durable_floor ≥ 8`**, so `D* ≥ 8`.
|
||||
|
||||
Take the tighter sub-case (`A=8`, `B=7`), where `D* = 8` achieved by `A`. `A` advertises bare `cand_lsn = durable_floor(A) = 8`. Voter `B`: `voter_floor_lsn(B) = max(durable_floor=7, learned_committed(B) ≤ L = 8)`. Here `learned_committed(B) ≤ 8`; if `B` had learned the `L = 8` commit then `voter_floor_lsn(B) = 8 = cand_lsn(A)` ⇒ `B` grants (equality grants, §1c); if `B` only learned up to 7 then `voter_floor_lsn(B) = 7 < 8` ⇒ `B` also grants. **Either way `B` grants `A`**, `A` self-grants ⇒ majority ⇒ **`A` is elected** (no livelock — consistent with §4 step 3, `voter_floor_lsn(v) ≤ D*`). Symmetrically `B` cannot win if `A` participates: `B` advertises bare `cand_lsn = 7`, and `A`'s `voter_floor_lsn = max(8, learned_committed_A ≤ 8) = 8 > 7` ⇒ `A` denies `B`, steering the election to the most-durable node `A` (§4).
|
||||
|
||||
**The dangerous-looking livelock state is UNREACHABLE.** The only way `A` could fail to be grantable by a majority is if some voter `v` had `voter_floor_lsn(v) > cand_lsn(A) = D*` — which by §4 step 3 requires `learned_committed(v) > D*`. But `learned_committed(v) ≤ L` (a node never learns a committed LSN above the true high-water) and `L ≤ D*` (committed ⇒ durable on a majority that intersects the survivors ⇒ some survivor has `durable_floor ≥ L`, so `D* ≥ L`). Hence `learned_committed(v) ≤ L ≤ D*` for every survivor `v`, so no survivor can demand a floor above `D*`, and a two-equal-tip-but-high-`learned_committed` deadlock (both survivors `durable_floor = 5`, both `learned_committed = 8`) **cannot arise when `L = 8` is genuinely committed**: `L = 8` committed forces some survivor to `durable_floor ≥ 8 > 5`, contradicting "both tips = 5". This is precisely the property §4 needs the case to demonstrate.
|
||||
|
||||
The elected `A` then computes `safe_committed_lsn` from the vote replies (durable-backed `peer_last_log_lsn` of `A`=8, `B`=7 ⇒ majority-backed safe point), unchanged by this task.
|
||||
|
||||
#### 6. Protocol Verification Checklist pass (`src/controller/CLAUDE.md`)
|
||||
|
||||
Walking the relevant items under the pinned formula:
|
||||
|
||||
- **Freshness gate — candidate `lastLogTerm >= voter` AND `lastLogIndex >= voter`:** HOLDS. The comparison (§1c) is `cand_tag >= voter_floor_tag` (term axis) AND, at equal tag, `cand_lsn >= voter_floor_lsn` (index axis) — exactly the checklist's two-key gate, now with `lastLogIndex` meaning the **durable** tip on both sides.
|
||||
- **`safe_committed_lsn` derived from voter replies, not stale ctrlhello:** HOLDS, UNCHANGED. This task does not touch `ha_compute_safe_committed_lsn`; it still reads `votes[].peer_last_log_lsn` (§1d). The floor change makes those replies durable-backed (more conservative), never sourced from a ctrlhello.
|
||||
- **`commitIndex` / `committed_lsn` monotonic, contiguous prefix:** HOLDS, UNCHANGED. The floor affects election only; commit advancement is untouched here (it is task 2.x). The freshness floor never writes `committed_lsn`.
|
||||
- **WAL truncation on promotion only removes entries above the safe commit point:** HOLDS. `persist_truncate_wal_above(safe_committed_lsn)` is unchanged; the durable-backed (tightened) `safe_committed_lsn` means truncation is, if anything, more conservative.
|
||||
- **`durable_lsn` is a prefix watermark (never a per-entry set):** HOLDS and is RELIED UPON — the Leader-Completeness argument (§3) and liveness argument (§4) both use "durable tip ≥ L ⇒ all entries ≤ L durable," which is exactly the prefix-watermark property the checklist asserts. `cand_lsn`/`voter_floor_lsn` are single LSNs (watermarks), not sets.
|
||||
- **(post `ha-commit-durability`) a member counts toward the commit majority only at its durably-persisted LSN; acknowledged ⇒ durable on a majority:** the *election-side* analog HOLDS here — a member contributes to a **vote-grant** (and advertises) only at its durably-persisted LSN. The commit-majority arithmetic side is task 2.x; this task makes the freshness/advertisement side durable-backed so the two are consistent.
|
||||
- **`votedFor` updated atomically before granting; `currentTerm` monotonic:** UNCHANGED by this task (term/vote bookkeeping in `raft_on_vote_request` is untouched); the floor change only gates *whether* `grant` is reached.
|
||||
|
||||
**Concern / caveat (restated):** `raft.tla` validates the freshness/election **logic** above; it does **not** validate that `persist_durable_floor` is genuinely durable (the model assumes a durable log). That durability is verified by tests 3.4 (durable-floor re-derivation on recovery) and 2.x (acknowledged ⇒ durable). No safety/liveness contradiction was found — the formula simultaneously satisfies Leader Completeness (§3) and no-livelock (§4) because the two pressures act on opposite sides of the comparison.
|
||||
|
||||
### Runtime / Test — DurabilityConfig + TestSeam
|
||||
|
||||
| Field | Detail |
|
||||
|-------|--------|
|
||||
| Intent | Deterministic, configurable durability timing for verification |
|
||||
| Requirements | 3.1, 3.2, 3.3 |
|
||||
|
||||
**Responsibilities & Constraints**
|
||||
- `--wal-sync-ms` plumbed `main.c → persist_config_t` before `persist_init`; default unchanged (`DEFAULT_WAL_SYNC_MS`). Under group-commit this bounds the **maximum batching delay** (a flush fires on the append-signal or when this elapses), never a durability gap — an acked op is always durable regardless of the value.
|
||||
- The seam's verification role: an `XCE_TESTING` hook (and/or a large `--wal-sync-ms` to widen the batching window) lets a test assert an acknowledged op is recoverable after an immediate ungraceful termination — proving durability comes from the commit gate, not the flush cadence.
|
||||
- External acknowledgment semantics are identical with or without the mode (timing controllability only).
|
||||
|
||||
**Contracts**: State [x]
|
||||
|
||||
**Implementation Notes**
|
||||
- Integration: follows `--sync-timeout-ms` (`OPT_SYNC_TIMEOUT`, `main.c:32`) convention; the actual durability test-seam functions live in `src/controller/persist.c` (`persist_test_set_force_fsync_fail` at `persist.c:1547`, `persist_test_durable_lsn_on_disk` at `persist.c:1610`), not at the originally-cited `ha.c:2270-2335`.
|
||||
- Validation: deterministic E2E mode in `test-commit-durability.sh` drives the seam so the soak becomes a deterministic RED on the unfixed build and GREEN after.
|
||||
- Risks: ensure the seam is compiled out of production (`#ifdef XCE_TESTING`).
|
||||
|
||||
## Error Handling
|
||||
|
||||
### Error Strategy
|
||||
- **Durability failure** (`fdatasync` error in the flusher): `durable_lsn` does not advance, so the entry is NOT counted durable, the commit does not advance, and the client does not receive *ok* (Req 1.4). The flusher surfaces the I/O error loudly (fail-loud); a persistent local sync failure is treated as a member fault, not silently retried into a false-durable state.
|
||||
- **Recovery corruption** (`RECOVER_TRUNCATED`): never silently accepted (Req 2.1). Fence from promotion + catch up; when catch-up is impossible (no peer) the node enters a LOUD PERMANENT FENCE — non-promotable, **stays alive**, keeps retrying catch-up, NEVER self-exits (Req 2.4, 2.6).
|
||||
- **Promotion while behind**: refused at the gate (Req 2.3); logged with the local durable tip vs the cluster high-water mark.
|
||||
|
||||
### Monitoring
|
||||
- Log every `durable_lsn` regression-clamp on truncate, every recovery verdict (with the truncation boundary), every promotion refusal (durable tip vs high-water), and every fence set/release — operator-visible, once per transition (consistent with the quorum-transition logging already in the HA stack).
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Tests
|
||||
- `durable_lsn` advances only after `fdatasync`; `majority_candidate_locked` self-cap equals `durable_lsn` not `highest_appended_lsn` (1.2).
|
||||
- `replay_wal` returns `RECOVER_TRUNCATED` on an injected CRC break at LSN 1 and `RECOVER_CLEAN` on an intact log (2.1).
|
||||
- Promotion gate denies a recovery-truncated / behind-high-water member and re-enables it after catch-up (2.2, 2.6).
|
||||
- `ha_vote_self_freshness` floor reflects the durable WAL tip, not a stale in-memory `last_known_committed_lsn` (2.3).
|
||||
- Standby `persist_wal_write_replicated` `fdatasync`s before ACK — regression pin (1.3).
|
||||
|
||||
### Integration Tests
|
||||
- Commit advances `committed_lsn` only when self is durable + a standby ACK is durable; reply released only after (1.1, 1.4).
|
||||
- Kill the sole durable contributor mid-commit; acknowledged op survives on the majority (1.5).
|
||||
- Durability work does not delay heartbeat handling beyond tolerance (1.6).
|
||||
|
||||
### E2E / Failover
|
||||
- `src/tools/test-commit-durability.sh` soak: rolling SIGKILL+restart under load asserts every acknowledged job survives (1.1, 1.5, 2.x). With the `--wal-sync-*` seam, run a **deterministic** variant: widen/disable background timing, kill a durable contributor in-window, assert survival — RED on the current build, GREEN after the fix (3.1, 3.2).
|
||||
|
||||
### Performance
|
||||
- Measure submit latency delta from synchronous `fdatasync` vs the existing replication round-trip; confirm it is within the round-trip already paid before deciding group-commit is needed (research item).
|
||||
|
||||
## Performance & Scalability
|
||||
- **Group-commit** amortizes one `fdatasync` across all entries pending at flush time, so the per-entry sync cost falls as load rises — critical for the high-volume `WAL_MODE_RETAIN` fact stream (job start/finish/fail/rusage) which is not client-latency-bound. `--wal-sync-ms` bounds the maximum batching delay (a latency/throughput tunable), never a durability gap. The `job-array-single-wal-record` roadmap item independently cuts per-array entry count. Target: zero acknowledged-job loss across failover (correctness) takes precedence over per-submit latency; group-commit keeps the throughput cost sub-linear in write rate.
|
||||
|
||||
## Open Questions / Gating Items (resolve in `/kiro-spec-tasks`)
|
||||
- **Freshness-floor formula (gating, election-safety):** pin the exact reconciliation of the durable-WAL-derived floor vs the existing `max(applied_lsn, last_known_committed_lsn)`. Too low ⇒ a truncated node still wins (the bug); too high ⇒ election livelock. Make a `raft/raft.tla` Leader-Completeness **and liveness** re-check a blocking sub-task before the freshness change merges (Req 2.3).
|
||||
- **Unrecoverable-terminal decision point (gating):** a booting truncated member cannot know it is "required for quorum" at `cluster_init`. Concrete policy: fence at boot, attempt catch-up from a healthy primary, and on a failure to reach a committed-consistent state (no peer) enter a **LOUD PERMANENT FENCE** — non-promotable, stays alive, keeps retrying catch-up, **NEVER self-exit** (Req 2.2, 2.4, 2.6). **RESOLVED (task 3.2; terminal corrected by the never-self-exit rework 2026-06-18 — the original self-`abort` FATAL was replaced by the loud permanent fence):** see the **Boot Policy Decision Table (task 3.2)** above; tasks 3.3 (loud-fence terminal) and 4.3 (fence + release) implement it.
|
||||
- **Flusher lost-wakeup proof:** the append-signal → flush → commit-wake path must be shown free of lost wakeups (level-triggered on the appended-vs-synced gap) before relying on it for the durability invariant (Req 1.1, 1.2).
|
||||
- **Snapshot-recovery edge:** current snapshots are 16-byte LSN sentinels (no entity data), so a member that recovers "from snapshot" has a committed position at the sentinel LSN but **no log entries below it**. The recovery verdict (3.1) must treat snapshot-sentinel + clean replay as CLEAN (not TRUNCATED), and the promotion fence (4.3) must treat such a member as behind-and-must-catch-up (never the unrecoverable terminal on a legitimate sentinel snapshot). Interacts with the roadmap `entity-bearing snapshots` item; pin the classification before implementing 3.1/4.3 (Req 2.1, 2.2).
|
||||
- **Synchronous-append path — RESOLVED (task 2.4), re-verified current:** `persist_wal_sync_append` (`persist.c:5639`) no longer advances the commit position unguarded — it routes through `sync_append_legacy_commit_store` (`persist.c:5617`), which only stores past `committed_lsn` when `written_lsn <= durable_lsn`, closing the ACK-without-durability hole (Req 1.1, 1.2; regression-pinned by `test_legacy_sync_append_holds_committed_at_durable`). (Note: this audit found the path is NOT dormant as originally described here — it now has a live production caller, `submit_job_wal_entry_and_commit` (`persist.c:5990-6004`), introduced by `raft-apply-decoupling`. The guard makes this safe regardless, so no further fix is needed, but the "no production caller today" premise in this section's original text no longer holds.)
|
||||
|
|
@ -1,164 +0,0 @@
|
|||
[2026-06-21 08:37:35,663] [1217413] [ INFO] [ main] My name is ctrl-1
|
||||
[2026-06-21 08:37:35,663] [1217413] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-1
|
||||
[2026-06-21 08:37:35,663] [1217413] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1)
|
||||
[2026-06-21 08:37:35,663] [1217413] [ INFO] [ cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
|
||||
[2026-06-21 08:37:35,671] [1217413] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:37:35,671] [1217413] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:37:35,671] [1217413] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:37:35,671] [1217413] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=0)
|
||||
[2026-06-21 08:37:35,671] [1217413] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:37:35,674] [1217413] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:37:40,873] [1217413] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=0 current_lsn=1; applied_lsn=0 advertised for catch-up
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [raft_cb_become_follower] HA FSM (raft): → CTRL_STANDBY (term=1, first_boot=1)
|
||||
[2026-06-21 08:37:40,874] [1217413] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:37:50,478] [1217413] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
||||
[2026-06-21 08:37:50,479] [1217413] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:37:50,479] [1217413] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:37:50,479] [1217413] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 8ms (priority=0) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:37:50,679] [1217413] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=2, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:37:50,728] [1217413] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 2)
|
||||
[2026-06-21 08:37:50,728] [1217413] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=0 safe_config_tag=5298030106630493228 (from 1 granted voters)
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=2, first_boot=0)
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992)
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
|
||||
[2026-06-21 08:37:50,732] [1217413] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
|
||||
[2026-06-21 08:37:50,738] [1217413] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
|
||||
[2026-06-21 08:37:50,738] [1217413] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=1, writable_lsn gate armed
|
||||
[2026-06-21 08:37:53,924] [1217413] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=2 (committed_tag=5298030106630493228 committed_lsn=1)
|
||||
[2026-06-21 08:37:53,925] [1217413] [ INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=2 (committed_tag=5298030106630493228 committed_lsn=1)
|
||||
[2026-06-21 08:39:27,321] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=2 < target=5 → streaming [2+1..5]
|
||||
[2026-06-21 08:39:27,321] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=3 target_lsn=5
|
||||
[2026-06-21 08:39:27,322] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [2+1..5] to standby
|
||||
[2026-06-21 08:39:27,522] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=3 < target=7 → streaming [3+1..7]
|
||||
[2026-06-21 08:39:27,522] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=4 target_lsn=7
|
||||
[2026-06-21 08:39:27,522] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [3+1..7] to standby
|
||||
[2026-06-21 08:39:27,722] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=7 < target=11 → streaming [7+1..11]
|
||||
[2026-06-21 08:39:27,722] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=8 target_lsn=11
|
||||
[2026-06-21 08:39:27,722] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [7+1..11] to standby
|
||||
[2026-06-21 08:39:27,922] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=4 < target=13 → streaming [4+1..13]
|
||||
[2026-06-21 08:39:27,922] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=5 target_lsn=13
|
||||
[2026-06-21 08:39:27,923] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [4+1..13] to standby
|
||||
[2026-06-21 08:39:28,123] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=8 < target=13 → streaming [8+1..13]
|
||||
[2026-06-21 08:39:28,123] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=9 target_lsn=13
|
||||
[2026-06-21 08:39:28,123] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [8+1..13] to standby
|
||||
[2026-06-21 08:39:28,323] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=5 < target=13 → streaming [5+1..13]
|
||||
[2026-06-21 08:39:28,323] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=6 target_lsn=13
|
||||
[2026-06-21 08:39:28,323] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [5+1..13] to standby
|
||||
[2026-06-21 08:39:28,523] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=9 < target=13 → streaming [9+1..13]
|
||||
[2026-06-21 08:39:28,523] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=10 target_lsn=13
|
||||
[2026-06-21 08:39:28,524] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [9+1..13] to standby
|
||||
[2026-06-21 08:39:28,724] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=6 < target=13 → streaming [6+1..13]
|
||||
[2026-06-21 08:39:28,724] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=7 target_lsn=13
|
||||
[2026-06-21 08:39:28,724] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [6+1..13] to standby
|
||||
[2026-06-21 08:39:28,924] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=10 < target=13 → streaming [10+1..13]
|
||||
[2026-06-21 08:39:28,924] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=11 target_lsn=13
|
||||
[2026-06-21 08:39:28,924] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [10+1..13] to standby
|
||||
[2026-06-21 08:39:29,125] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=7 < target=13 → streaming [7+1..13]
|
||||
[2026-06-21 08:39:29,125] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=8 target_lsn=13
|
||||
[2026-06-21 08:39:29,125] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [7+1..13] to standby
|
||||
[2026-06-21 08:39:29,325] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=11 < target=13 → streaming [11+1..13]
|
||||
[2026-06-21 08:39:29,325] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=12 target_lsn=13
|
||||
[2026-06-21 08:39:29,325] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [11+1..13] to standby
|
||||
[2026-06-21 08:39:29,525] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=8 < target=13 → streaming [8+1..13]
|
||||
[2026-06-21 08:39:29,525] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=9 target_lsn=13
|
||||
[2026-06-21 08:39:29,526] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [8+1..13] to standby
|
||||
[2026-06-21 08:39:29,726] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=12 < target=13 → streaming [12+1..13]
|
||||
[2026-06-21 08:39:29,726] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=13 target_lsn=13
|
||||
[2026-06-21 08:39:29,726] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [12+1..13] to standby
|
||||
[2026-06-21 08:39:29,926] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=9 < target=13 → streaming [9+1..13]
|
||||
[2026-06-21 08:39:29,926] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=10 target_lsn=13
|
||||
[2026-06-21 08:39:29,926] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [9+1..13] to standby
|
||||
[2026-06-21 08:39:30,326] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=10 < target=13 → streaming [10+1..13]
|
||||
[2026-06-21 08:39:30,326] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=11 target_lsn=13
|
||||
[2026-06-21 08:39:30,327] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [10+1..13] to standby
|
||||
[2026-06-21 08:39:30,527] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=11 < target=13 → streaming [11+1..13]
|
||||
[2026-06-21 08:39:30,527] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=12 target_lsn=13
|
||||
[2026-06-21 08:39:30,527] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [11+1..13] to standby
|
||||
[2026-06-21 08:39:30,727] [1217413] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=12 < target=13 → streaming [12+1..13]
|
||||
[2026-06-21 08:39:30,727] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=13 target_lsn=13
|
||||
[2026-06-21 08:39:30,727] [1217413] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [12+1..13] to standby
|
||||
[2026-06-21 08:43:44,321] [1224672] [ INFO] [ main] My name is ctrl-1
|
||||
[2026-06-21 08:43:44,321] [1224672] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-1
|
||||
[2026-06-21 08:43:44,321] [1224672] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1)
|
||||
[2026-06-21 08:43:44,321] [1224672] [ INFO] [ cluster_init_phase1] HA: restored term=2 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
|
||||
[2026-06-21 08:43:44,327] [1224672] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=2)
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-1, voted_term=2, raft_term=2, primary_fd=-1) — skipping own election
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=13 current_lsn=14; applied_lsn=13 advertised for catch-up
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:43:44,328] [1224672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:43:45,250] [1224672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:43:45,250] [1224672] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=6)
|
||||
[2026-06-21 08:47:39,017] [1224672] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
||||
[2026-06-21 08:47:39,017] [1224672] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:47:39,017] [1224672] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:47:39,017] [1224672] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 122ms (priority=0) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:47:39,217] [1224672] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=3, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:47:39,262] [1224672] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 3)
|
||||
[2026-06-21 08:47:39,265] [1224672] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=44 safe_config_tag=5298030106630493228 (from 1 granted voters)
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=3, first_boot=0)
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992)
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
|
||||
[2026-06-21 08:47:39,267] [1224672] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
|
||||
[2026-06-21 08:47:39,272] [1224672] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
|
||||
[2026-06-21 08:47:39,272] [1224672] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=45, writable_lsn gate armed
|
||||
[2026-06-21 08:47:41,419] [1224672] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=46 < target=48 → streaming [46+1..48]
|
||||
[2026-06-21 08:47:41,419] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=47 target_lsn=48
|
||||
[2026-06-21 08:47:41,420] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [46+1..48] to standby
|
||||
[2026-06-21 08:47:41,620] [1224672] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=48 < target=53 → streaming [48+1..53]
|
||||
[2026-06-21 08:47:41,620] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=49 target_lsn=53
|
||||
[2026-06-21 08:47:41,620] [1224672] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [48+1..53] to standby
|
||||
[2026-06-21 08:47:47,832] [1228504] [ INFO] [ main] My name is ctrl-1
|
||||
[2026-06-21 08:47:47,832] [1228504] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-1
|
||||
[2026-06-21 08:47:47,832] [1228504] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=0 (name=ctrl-1)
|
||||
[2026-06-21 08:47:47,832] [1228504] [ INFO] [ cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-1/data/ha-term.bin
|
||||
[2026-06-21 08:47:47,839] [1228504] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-1/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=0, term=3)
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-1, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:47:47,840] [1228504] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:47:47,841] [1228504] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=53 current_lsn=54; applied_lsn=53 advertised for catch-up
|
||||
[2026-06-21 08:47:47,841] [1228504] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:47:47,841] [1228504] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:52,855] [1228504] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:52,855] [1228504] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:47:52,855] [1228504] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:52,855] [1228504] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=8)
|
||||
[2026-06-21 08:48:04,047] [1228504] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:48:04,047] [1228504] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:48:04,047] [1228504] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:48:04,047] [1228504] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 159ms (priority=0) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:48:04,248] [1228504] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=4, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:48:04,295] [1228504] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 4)
|
||||
[2026-06-21 08:48:04,295] [1228504] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=53 safe_config_tag=5298030106630493228 (from 1 granted voters)
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=4, first_boot=0)
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9992)
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
|
||||
[2026-06-21 08:48:04,300] [1228504] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
|
||||
[2026-06-21 08:48:04,305] [1228504] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
|
||||
[2026-06-21 08:48:04,307] [1228504] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=54, writable_lsn gate armed
|
||||
[2026-06-21 08:48:07,493] [1228504] [ INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=55 (committed_tag=5298030106630493228 committed_lsn=54)
|
||||
[2026-06-21 08:48:07,493] [1228504] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=55 (committed_tag=5298030106630493228 committed_lsn=54)
|
||||
|
|
@ -1,99 +0,0 @@
|
|||
[2026-06-21 08:37:36,163] [1217460] [ INFO] [ main] My name is ctrl-2
|
||||
[2026-06-21 08:37:36,164] [1217460] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-2
|
||||
[2026-06-21 08:37:36,164] [1217460] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2)
|
||||
[2026-06-21 08:37:36,164] [1217460] [ INFO] [ cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin
|
||||
[2026-06-21 08:37:36,173] [1217460] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:37:36,173] [1217460] [ INFO] [ load_primary_config] No config at '/tmp/xce-deploy/controller/ctrl-2/conf/current' (rc=-2) — will fetch from primary after election
|
||||
[2026-06-21 08:37:36,173] [1217460] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=0)
|
||||
[2026-06-21 08:37:36,673] [1217460] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:37:36,673] [1217460] [ INFO] [ha_start_election_async] ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only, not starting own election
|
||||
[2026-06-21 08:37:36,673] [1217460] [ ERROR] [ cluster_init_phase2] cluster_init_phase2: cluster_config not loaded
|
||||
[2026-06-21 08:37:36,673] [1217460] [ WARN] [ myapp_run] myapp_run: cluster_init_phase2 on standby rejoin failed (rc=-22) — WAL not recovered; falling back to election-path recovery
|
||||
[2026-06-21 08:37:36,673] [1217460] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:37:36,673] [1217460] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:37:46,177] [1217460] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
||||
[2026-06-21 08:37:46,177] [1217460] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:37:46,177] [1217460] [ INFO] [ha_on_primary_ctrl_fd_rdhup] ha_on_primary_ctrl_fd_rdhup: no local config (tag=0, n_peers=3) — voter only, not starting PreVote/RDHUP re-election; awaiting configured primary (ha-config-bootstrap Req 2.8)
|
||||
[2026-06-21 08:37:50,743] [1217460] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:37:50,743] [1217460] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:37:50,743] [1217460] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:37:50,743] [1217460] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
[2026-06-21 08:40:02,029] [1217460] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:40:02,029] [1217460] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:40:02,029] [1217460] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:40:02,029] [1217460] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 585ms (priority=1) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:40:02,629] [1217460] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=1, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:40:03,270] [1217460] [ INFO] [ha_on_heartbeat_received] HA: primary heartbeat arrived mid-PreVote — cancelling the round and re-attaching as CTRL_STANDBY (leader alive, no term change) (Req 4.2)
|
||||
[2026-06-21 08:40:03,273] [1217460] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:40:03,273] [1217460] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:40:03,273] [1217460] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:40:03,273] [1217460] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=9)
|
||||
[2026-06-21 08:46:07,141] [1227110] [ INFO] [ main] My name is ctrl-2
|
||||
[2026-06-21 08:46:07,141] [1227110] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-2
|
||||
[2026-06-21 08:46:07,141] [1227110] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2)
|
||||
[2026-06-21 08:46:07,141] [1227110] [ INFO] [ cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin
|
||||
[2026-06-21 08:46:07,149] [1227110] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:46:07,149] [1227110] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:46:07,149] [1227110] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:46:07,149] [1227110] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=3)
|
||||
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-3, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election
|
||||
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:46:07,649] [1227110] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:46:07,650] [1227110] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=22 current_lsn=23; applied_lsn=22 advertised for catch-up
|
||||
[2026-06-21 08:46:07,650] [1227110] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:46:07,650] [1227110] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:46:09,319] [1227110] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:46:09,319] [1227110] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=6)
|
||||
[2026-06-21 08:47:38,988] [1227110] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
||||
[2026-06-21 08:47:38,988] [1227110] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:47:38,988] [1227110] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:47:38,988] [1227110] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 556ms (priority=1) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:47:39,276] [1227110] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:39,276] [1227110] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:47:39,277] [1227110] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:39,277] [1227110] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=8)
|
||||
[2026-06-21 08:47:39,588] [1227110] [ INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606)
|
||||
[2026-06-21 08:47:51,992] [1227110] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:47:51,992] [1227110] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:47:51,992] [1227110] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:47:51,992] [1227110] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 662ms (priority=1) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:47:52,793] [1227110] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=4, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:47:52,840] [1227110] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 4)
|
||||
[2026-06-21 08:47:52,843] [1227110] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=44 safe_config_tag=5298030106630493228 (from 1 granted voters)
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=4, first_boot=0)
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[0] → standby channel
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[0] (slot 0, localhost:9991)
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
|
||||
[2026-06-21 08:47:52,845] [1227110] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 1, localhost:9993)
|
||||
[2026-06-21 08:47:52,850] [1227110] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
|
||||
[2026-06-21 08:47:52,850] [1227110] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=45, writable_lsn gate armed
|
||||
[2026-06-21 08:47:53,395] [1227110] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=48 < target=49 → streaming [48+1..49]
|
||||
[2026-06-21 08:47:53,395] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=49 target_lsn=49
|
||||
[2026-06-21 08:47:53,395] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [48+1..49] to standby
|
||||
[2026-06-21 08:47:53,595] [1227110] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=49 < target=53 → streaming [49+1..53]
|
||||
[2026-06-21 08:47:53,595] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=50 target_lsn=53
|
||||
[2026-06-21 08:47:53,596] [1227110] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [49+1..53] to standby
|
||||
[2026-06-21 08:47:59,844] [1228672] [ INFO] [ main] My name is ctrl-2
|
||||
[2026-06-21 08:47:59,844] [1228672] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-2
|
||||
[2026-06-21 08:47:59,844] [1228672] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=1 (name=ctrl-2)
|
||||
[2026-06-21 08:47:59,845] [1228672] [ INFO] [ cluster_init_phase1] HA: restored term=4 from /tmp/xce-deploy/controller/ctrl-2/data/ha-term.bin
|
||||
[2026-06-21 08:47:59,854] [1228672] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-2/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:47:59,854] [1228672] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:59,854] [1228672] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:47:59,854] [1228672] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=1, term=4)
|
||||
[2026-06-21 08:48:00,354] [1228672] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-2, voted_term=4, raft_term=4, primary_fd=-1) — skipping own election
|
||||
[2026-06-21 08:48:00,354] [1228672] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:48:00,355] [1228672] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:48:00,355] [1228672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:48:00,355] [1228672] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:48:00,357] [1228672] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=53 current_lsn=54; applied_lsn=53 advertised for catch-up
|
||||
[2026-06-21 08:48:00,357] [1228672] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:48:00,357] [1228672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:48:04,312] [1228672] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:48:04,312] [1228672] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:48:04,312] [1228672] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:48:04,312] [1228672] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
|
|
@ -1,213 +0,0 @@
|
|||
[2026-06-21 08:37:36,665] [1217472] [ INFO] [ main] My name is ctrl-3
|
||||
[2026-06-21 08:37:36,665] [1217472] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-3
|
||||
[2026-06-21 08:37:36,666] [1217472] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=2 (name=ctrl-3)
|
||||
[2026-06-21 08:37:36,666] [1217472] [ INFO] [ cluster_init_phase1] HA: restored term=0 from /tmp/xce-deploy/controller/ctrl-3/data/ha-term.bin
|
||||
[2026-06-21 08:37:36,672] [1217472] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-3/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:37:36,672] [1217472] [ INFO] [ load_primary_config] No config at '/tmp/xce-deploy/controller/ctrl-3/conf/current' (rc=-2) — will fetch from primary after election
|
||||
[2026-06-21 08:37:36,672] [1217472] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=2, term=0)
|
||||
[2026-06-21 08:37:37,672] [1217472] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:37:37,672] [1217472] [ INFO] [ha_start_election_async] ha_start_election_async: no local config (tag=0, n_peers=3) — participating as voter only, not starting own election
|
||||
[2026-06-21 08:37:37,672] [1217472] [ ERROR] [ cluster_init_phase2] cluster_init_phase2: cluster_config not loaded
|
||||
[2026-06-21 08:37:37,672] [1217472] [ WARN] [ myapp_run] myapp_run: cluster_init_phase2 on standby rejoin failed (rc=-22) — WAL not recovered; falling back to election-path recovery
|
||||
[2026-06-21 08:37:37,672] [1217472] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:37:37,672] [1217472] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:37:47,276] [1217472] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 10s (timeout=10s) → failover
|
||||
[2026-06-21 08:37:47,276] [1217472] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:37:47,276] [1217472] [ INFO] [ha_on_primary_ctrl_fd_rdhup] ha_on_primary_ctrl_fd_rdhup: no local config (tag=0, n_peers=3) — voter only, not starting PreVote/RDHUP re-election; awaiting configured primary (ha-config-bootstrap Req 2.8)
|
||||
[2026-06-21 08:37:50,743] [1217472] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:37:50,743] [1217472] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:37:50,743] [1217472] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:37:50,743] [1217472] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
[2026-06-21 08:40:01,930] [1217472] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:40:01,930] [1217472] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:40:01,930] [1217472] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:40:01,930] [1217472] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 1150ms (priority=2) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:40:02,636] [1217472] [ INFO] [ myreq_handler] HA: PreVote DENIED candidate='ctrl-2' term=1 reason=stale_term (my_term=0, floor tag=5298030106630493228 lsn=13)
|
||||
[2026-06-21 08:40:03,130] [1217472] [ INFO] [ha_start_prevote_async] HA: starting PreVote round (prospective term=3, n_peers=3) — staying CTRL_STANDBY, no persist
|
||||
[2026-06-21 08:40:03,176] [1217472] [ INFO] [pre_vote_reply_dispatch] HA: PreVote round WON — entering real election (term will be 3)
|
||||
[2026-06-21 08:40:03,176] [1217472] [ INFO] [ha_start_election_async] HA FSM: → CTRL_ELECTING (async, n_peers=3)
|
||||
[2026-06-21 08:40:03,180] [1217472] [ INFO] [ha_start_election_async] ha_start_election_async: returning (n_peers=3, raft election started)
|
||||
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ha_compute_safe_committed_lsn] HA: safe_committed_lsn=13 safe_config_tag=5298030106630493228 (from 1 granted voters)
|
||||
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:40:03,223] [1217472] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:40:03,224] [1217472] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=13 current_lsn=14; applied_lsn=14 advertised for catch-up
|
||||
[2026-06-21 08:40:03,224] [1217472] [ INFO] [ raft_cb_become_leader] HA FSM (raft): → CTRL_PRIMARY (term=3, first_boot=1)
|
||||
[2026-06-21 08:40:03,224] [1217472] [ INFO] [ raft_cb_become_leader] HA: promoted vote ep[1] → standby channel
|
||||
[2026-06-21 08:40:03,224] [1217472] [ INFO] [ raft_cb_become_leader] HA: repl registered standby ep[1] (slot 0, localhost:9992)
|
||||
[2026-06-21 08:40:03,229] [1217472] [ INFO] [ha_start_primary_heartbeat] HA: primary heartbeat armed (interval=3000ms, rc=0)
|
||||
[2026-06-21 08:40:03,230] [1217472] [ INFO] [ ha_arm_own_term_gate] HA: own-term noop appended lsn=14, writable_lsn gate armed
|
||||
[2026-06-21 08:40:06,333] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 1)
|
||||
[2026-06-21 08:40:06,333] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:40:06,374] [1217472] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=15 (committed_tag=5298030106630493228 committed_lsn=14)
|
||||
[2026-06-21 08:40:33,348] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 10)
|
||||
[2026-06-21 08:40:33,348] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:41:03,364] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 20)
|
||||
[2026-06-21 08:41:03,364] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:41:33,382] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 30)
|
||||
[2026-06-21 08:41:33,382] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:42:03,397] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 40)
|
||||
[2026-06-21 08:42:03,397] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:42:33,411] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 50)
|
||||
[2026-06-21 08:42:33,411] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:43:03,425] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 60)
|
||||
[2026-06-21 08:43:03,425] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:43:33,240] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 70)
|
||||
[2026-06-21 08:43:33,240] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[0] (localhost:9991) failed: -111
|
||||
[2026-06-21 08:43:45,247] [1217472] [ INFO] [ ha_connect_to_peer] HA: connecting outbound to peer[0] (localhost:9991)
|
||||
[2026-06-21 08:43:45,250] [1217472] [ INFO] [ on_standby_connected] HA: outbound standby connection to peer[0] established, ctrlhello sent
|
||||
[2026-06-21 08:43:45,250] [1217472] [ INFO] [ on_standby_connected] HA: repl registered late-joiner standby peer[0] (slot 1, localhost:9991)
|
||||
[2026-06-21 08:43:48,290] [1217472] [ INFO] [ha_dispatch_repl_action_for_standby] HA repl: standby[0] lsn=13 < committed_lsn=22 → stream WAL batches
|
||||
[2026-06-21 08:43:48,290] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=14 target_lsn=22
|
||||
[2026-06-21 08:43:48,290] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [13+1..22] to standby
|
||||
[2026-06-21 08:43:48,449] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=14 < target=22 → streaming [14+1..22]
|
||||
[2026-06-21 08:43:48,449] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=15 target_lsn=22
|
||||
[2026-06-21 08:43:48,449] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [14+1..22] to standby
|
||||
[2026-06-21 08:43:48,649] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=15 < target=22 → streaming [15+1..22]
|
||||
[2026-06-21 08:43:48,649] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=16 target_lsn=22
|
||||
[2026-06-21 08:43:48,649] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [15+1..22] to standby
|
||||
[2026-06-21 08:43:48,849] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=16 < target=22 → streaming [16+1..22]
|
||||
[2026-06-21 08:43:48,849] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=17 target_lsn=22
|
||||
[2026-06-21 08:43:48,850] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [16+1..22] to standby
|
||||
[2026-06-21 08:43:49,050] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=17 < target=22 → streaming [17+1..22]
|
||||
[2026-06-21 08:43:49,050] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=18 target_lsn=22
|
||||
[2026-06-21 08:43:49,050] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [17+1..22] to standby
|
||||
[2026-06-21 08:43:49,250] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=18 < target=22 → streaming [18+1..22]
|
||||
[2026-06-21 08:43:49,250] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=19 target_lsn=22
|
||||
[2026-06-21 08:43:49,251] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [18+1..22] to standby
|
||||
[2026-06-21 08:43:49,451] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=19 < target=22 → streaming [19+1..22]
|
||||
[2026-06-21 08:43:49,451] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=20 target_lsn=22
|
||||
[2026-06-21 08:43:49,451] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [19+1..22] to standby
|
||||
[2026-06-21 08:43:49,651] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=20 < target=22 → streaming [20+1..22]
|
||||
[2026-06-21 08:43:49,651] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=21 target_lsn=22
|
||||
[2026-06-21 08:43:49,651] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [20+1..22] to standby
|
||||
[2026-06-21 08:43:49,851] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=21 < target=22 → streaming [21+1..22]
|
||||
[2026-06-21 08:43:49,851] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=22 target_lsn=22
|
||||
[2026-06-21 08:43:49,852] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [21+1..22] to standby
|
||||
[2026-06-21 08:43:51,294] [1217472] [ INFO] [ha_process_standby_state] HA: standby[0] converged: config_tag=5298030106630493228 applied_lsn=23 (committed_tag=5298030106630493228 committed_lsn=22)
|
||||
[2026-06-21 08:45:24,291] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[0] (localhost) not connected — initiating outbound connect (attempt 1)
|
||||
[2026-06-21 08:45:24,292] [1217472] [ INFO] [ ha_connect_to_peer] HA: connecting outbound to peer[0] (localhost:9991)
|
||||
[2026-06-21 08:45:24,292] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[1] (localhost) not connected — initiating outbound connect (attempt 1)
|
||||
[2026-06-21 08:45:24,292] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[1] (localhost:9992) failed: -111
|
||||
[2026-06-21 08:45:24,294] [1217472] [ INFO] [ on_standby_connected] HA: outbound standby connection to peer[0] established, ctrlhello sent
|
||||
[2026-06-21 08:45:24,294] [1217472] [ INFO] [ on_standby_connected] HA: repl registered late-joiner standby peer[0] (slot 0, localhost:9991)
|
||||
[2026-06-21 08:45:51,305] [1217472] [ INFO] [ ha_connect_to_peer] HA: peer[1] (localhost) not connected — initiating outbound connect (attempt 10)
|
||||
[2026-06-21 08:45:51,305] [1217472] [ WARN] [ ha_connect_to_peer] HA: connect to peer[1] (localhost:9992) failed: -111
|
||||
[2026-06-21 08:46:07,913] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=35 < target=36 → streaming [35+1..36]
|
||||
[2026-06-21 08:46:07,913] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=36 target_lsn=36
|
||||
[2026-06-21 08:46:07,913] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [35+1..36] to standby
|
||||
[2026-06-21 08:46:09,314] [1217472] [ INFO] [ ha_connect_to_peer] HA: connecting outbound to peer[1] (localhost:9992)
|
||||
[2026-06-21 08:46:09,319] [1217472] [ INFO] [ on_standby_connected] HA: outbound standby connection to peer[1] established, ctrlhello sent
|
||||
[2026-06-21 08:46:09,319] [1217472] [ INFO] [ on_standby_connected] HA: repl registered late-joiner standby peer[1] (slot 2, localhost:9992)
|
||||
[2026-06-21 08:46:12,358] [1217472] [ INFO] [ha_dispatch_repl_action_for_standby] HA repl: standby[1] lsn=22 < committed_lsn=44 → stream WAL batches
|
||||
[2026-06-21 08:46:12,358] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=23 target_lsn=44
|
||||
[2026-06-21 08:46:12,358] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [22+1..44] to standby
|
||||
[2026-06-21 08:46:12,516] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=23 < target=44 → streaming [23+1..44]
|
||||
[2026-06-21 08:46:12,516] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=24 target_lsn=44
|
||||
[2026-06-21 08:46:12,516] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [23+1..44] to standby
|
||||
[2026-06-21 08:46:12,716] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=24 < target=44 → streaming [24+1..44]
|
||||
[2026-06-21 08:46:12,716] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=25 target_lsn=44
|
||||
[2026-06-21 08:46:12,717] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [24+1..44] to standby
|
||||
[2026-06-21 08:46:12,917] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=25 < target=44 → streaming [25+1..44]
|
||||
[2026-06-21 08:46:12,917] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=26 target_lsn=44
|
||||
[2026-06-21 08:46:12,917] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [25+1..44] to standby
|
||||
[2026-06-21 08:46:13,117] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=26 < target=44 → streaming [26+1..44]
|
||||
[2026-06-21 08:46:13,118] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=27 target_lsn=44
|
||||
[2026-06-21 08:46:13,118] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [26+1..44] to standby
|
||||
[2026-06-21 08:46:13,318] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=27 < target=44 → streaming [27+1..44]
|
||||
[2026-06-21 08:46:13,318] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=28 target_lsn=44
|
||||
[2026-06-21 08:46:13,319] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [27+1..44] to standby
|
||||
[2026-06-21 08:46:13,519] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=28 < target=44 → streaming [28+1..44]
|
||||
[2026-06-21 08:46:13,519] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=29 target_lsn=44
|
||||
[2026-06-21 08:46:13,519] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [28+1..44] to standby
|
||||
[2026-06-21 08:46:13,719] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=29 < target=44 → streaming [29+1..44]
|
||||
[2026-06-21 08:46:13,719] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=30 target_lsn=44
|
||||
[2026-06-21 08:46:13,720] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [29+1..44] to standby
|
||||
[2026-06-21 08:46:13,920] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=30 < target=44 → streaming [30+1..44]
|
||||
[2026-06-21 08:46:13,920] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=31 target_lsn=44
|
||||
[2026-06-21 08:46:13,920] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [30+1..44] to standby
|
||||
[2026-06-21 08:46:14,121] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=31 < target=44 → streaming [31+1..44]
|
||||
[2026-06-21 08:46:14,121] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=32 target_lsn=44
|
||||
[2026-06-21 08:46:14,121] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [31+1..44] to standby
|
||||
[2026-06-21 08:46:14,321] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=32 < target=44 → streaming [32+1..44]
|
||||
[2026-06-21 08:46:14,321] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=33 target_lsn=44
|
||||
[2026-06-21 08:46:14,321] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [32+1..44] to standby
|
||||
[2026-06-21 08:46:14,522] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=33 < target=44 → streaming [33+1..44]
|
||||
[2026-06-21 08:46:14,522] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=34 target_lsn=44
|
||||
[2026-06-21 08:46:14,522] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [33+1..44] to standby
|
||||
[2026-06-21 08:46:14,722] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=34 < target=44 → streaming [34+1..44]
|
||||
[2026-06-21 08:46:14,722] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=35 target_lsn=44
|
||||
[2026-06-21 08:46:14,722] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [34+1..44] to standby
|
||||
[2026-06-21 08:46:14,922] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=35 < target=44 → streaming [35+1..44]
|
||||
[2026-06-21 08:46:14,922] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=36 target_lsn=44
|
||||
[2026-06-21 08:46:14,923] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [35+1..44] to standby
|
||||
[2026-06-21 08:46:15,123] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=36 < target=44 → streaming [36+1..44]
|
||||
[2026-06-21 08:46:15,123] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=37 target_lsn=44
|
||||
[2026-06-21 08:46:15,123] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [36+1..44] to standby
|
||||
[2026-06-21 08:46:15,323] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=37 < target=44 → streaming [37+1..44]
|
||||
[2026-06-21 08:46:15,323] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=38 target_lsn=44
|
||||
[2026-06-21 08:46:15,323] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [37+1..44] to standby
|
||||
[2026-06-21 08:46:15,565] [1217472] [ INFO] [ha_process_standby_state] HA: standby[1] converged: config_tag=5298030106630493228 applied_lsn=45 (committed_tag=5298030106630493228 committed_lsn=44)
|
||||
[2026-06-21 08:46:15,724] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=38 < target=44 → streaming [38+1..44]
|
||||
[2026-06-21 08:46:15,724] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=39 target_lsn=44
|
||||
[2026-06-21 08:46:15,724] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [38+1..44] to standby
|
||||
[2026-06-21 08:46:15,924] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=39 < target=44 → streaming [39+1..44]
|
||||
[2026-06-21 08:46:15,924] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=40 target_lsn=44
|
||||
[2026-06-21 08:46:15,924] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [39+1..44] to standby
|
||||
[2026-06-21 08:46:16,124] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=40 < target=44 → streaming [40+1..44]
|
||||
[2026-06-21 08:46:16,124] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=41 target_lsn=44
|
||||
[2026-06-21 08:46:16,125] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [40+1..44] to standby
|
||||
[2026-06-21 08:46:16,325] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=41 < target=44 → streaming [41+1..44]
|
||||
[2026-06-21 08:46:16,325] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=42 target_lsn=44
|
||||
[2026-06-21 08:46:16,325] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [41+1..44] to standby
|
||||
[2026-06-21 08:46:16,525] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=42 < target=44 → streaming [42+1..44]
|
||||
[2026-06-21 08:46:16,525] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=43 target_lsn=44
|
||||
[2026-06-21 08:46:16,526] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [42+1..44] to standby
|
||||
[2026-06-21 08:46:16,726] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=43 < target=44 → streaming [43+1..44]
|
||||
[2026-06-21 08:46:16,726] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=44 target_lsn=44
|
||||
[2026-06-21 08:46:16,726] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [43+1..44] to standby
|
||||
[2026-06-21 08:47:29,557] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[0] acked=45 < target=48 → streaming [45+1..48]
|
||||
[2026-06-21 08:47:29,557] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=46 target_lsn=48
|
||||
[2026-06-21 08:47:29,558] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [45+1..48] to standby
|
||||
[2026-06-21 08:47:29,758] [1217472] [ INFO] [ha_catchup_continuation_fn] HA catch-up continuation: standby[1] acked=46 < target=51 → streaming [46+1..51]
|
||||
[2026-06-21 08:47:29,758] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: starting WAL catch-up start_lsn=47 target_lsn=51
|
||||
[2026-06-21 08:47:29,758] [1217472] [ INFO] [ha_stream_wal_to_standby] ha_stream_wal_to_standby: streamed WAL [46+1..51] to standby
|
||||
[2026-06-21 08:47:36,064] [1228337] [ INFO] [ main] My name is ctrl-3
|
||||
[2026-06-21 08:47:36,064] [1228337] [ INFO] [ main] HA mode: 3 controller candidates, identity=ctrl-3
|
||||
[2026-06-21 08:47:36,065] [1228337] [ INFO] [ cluster_init_phase1] HA peers: 3 candidates, my_priority=2 (name=ctrl-3)
|
||||
[2026-06-21 08:47:36,065] [1228337] [ INFO] [ cluster_init_phase1] HA: restored term=3 from /tmp/xce-deploy/controller/ctrl-3/data/ha-term.bin
|
||||
[2026-06-21 08:47:36,071] [1228337] [ WARN] [ persist_init] WAL/snapshot stored unencrypted in /tmp/xce-deploy/controller/ctrl-3/data/wal — ensure filesystem is encrypted if job payloads contain sensitive data
|
||||
[2026-06-21 08:47:36,072] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:36,072] [1228337] [ INFO] [ myapp_run] Pre-election config loaded: tag=4986624fd309142c
|
||||
[2026-06-21 08:47:36,072] [1228337] [ INFO] [ ha_init_raft] HA: raft initialized (n_peers=3, self=2, term=3)
|
||||
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ myapp_run] myapp_run: leader already known after startup delay (voted_for=ctrl-3, voted_term=3, raft_term=3, primary_fd=-1) — skipping own election
|
||||
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ cluster_init_phase2] commit engine wired (n_peers=3, n_vcpus=4)
|
||||
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ xce_config_init] xce controller is running
|
||||
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' queue
|
||||
[2026-06-21 08:47:37,072] [1228337] [ INFO] [ create_lost_and_found] Created synthetic 'lost_and_found' host
|
||||
[2026-06-21 08:47:37,074] [1228337] [ INFO] [ cluster_init_phase2] HA: persist_recover done — recovered durable_floor=52 current_lsn=53; applied_lsn=52 advertised for catch-up
|
||||
[2026-06-21 08:47:37,074] [1228337] [ INFO] [ myapp_run] myapp_run: CTRL_STANDBY — waiting for primary to connect and push config
|
||||
[2026-06-21 08:47:37,074] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:39,277] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:39,277] [1228337] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:47:39,277] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:39,277] [1228337] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
[2026-06-21 08:47:52,078] [1228337] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:47:52,078] [1228337] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:47:52,078] [1228337] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:47:52,078] [1228337] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 1088ms (priority=2) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:47:52,855] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:47:52,855] [1228337] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:47:52,855] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:47:52,855] [1228337] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
[2026-06-21 08:47:53,279] [1228337] [ INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606)
|
||||
[2026-06-21 08:48:04,083] [1228337] [ WARN] [ liveness_poll_fn] HA: liveness poll: no primary activity for 11s (timeout=10s) → failover
|
||||
[2026-06-21 08:48:04,083] [1228337] [ INFO] [ha_on_primary_ctrl_fd_rdhup] HA: primary_ctrl_fd RDHUP → begin PreVote failover (staying CTRL_STANDBY until PreVote wins) (Req 1.1)
|
||||
[2026-06-21 08:48:04,083] [1228337] [ INFO] [ ha_begin_failover] HA: beginning failover via PreVote (no truncate / no term bump until PreVote wins) — staying CTRL_STANDBY
|
||||
[2026-06-21 08:48:04,083] [1228337] [ INFO] [ ha_begin_failover] HA: staggering PreVote start by 1092ms (priority=2) to avoid split vote (bug-606)
|
||||
[2026-06-21 08:48:04,312] [1228337] [ INFO] [ load_primary_config] Primary config loaded: tag=4986624fd309142c gen=1
|
||||
[2026-06-21 08:48:04,312] [1228337] [ INFO] [handle_repl_config_push] handle_repl_config_push: config saved and loaded (tag=4986624fd309142c)
|
||||
[2026-06-21 08:48:04,312] [1228337] [ INFO] [ha_arm_heartbeat_timeout_vtask] HA: standby liveness poll armed (timeout=10000ms, poll=1500ms)
|
||||
[2026-06-21 08:48:04,312] [1228337] [ INFO] [ handle_ctrlhello] HA: primary control channel established (fd=7)
|
||||
[2026-06-21 08:48:05,284] [1228337] [ INFO] [ha_failover_stagger_fn] HA: failover stagger elapsed but a primary is alive — skipping PreVote (bug-606)
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
1 | 0 | NOOP | 0 | 00000000 |
|
||||
2 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
3 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
4 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
5 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
6 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
7 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
8 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
9 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
10 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
11 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
12 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
13 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
14 | 0 | NOOP | 0 | 00000000 |
|
||||
15 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
16 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
17 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
18 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
19 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
20 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
21 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
22 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
23 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
24 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
25 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
26 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
27 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
28 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
29 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
30 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
31 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
32 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
33 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
34 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
35 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
36 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
37 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
38 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
39 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
40 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
41 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
42 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
43 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
44 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
45 | 0 | NOOP | 0 | 00000000 |
|
||||
46 | 0 | JOB_SUBMIT | 3952 | faead246 | jobid=1868595630964736 subtime=1782031661 state=1
|
||||
47 | 0 | JOB_SUBMIT | 3952 | faead246 | jobid=1868595630964736 subtime=1782031661 state=1
|
||||
48 | 0 | JOB_SUBMIT | 3952 | be0059bf | jobid=1868595630964737 subtime=1782031661 state=1
|
||||
49 | 0 | JOB_SUBMIT | 3952 | be0059bf | jobid=1868595630964737 subtime=1782031661 state=1
|
||||
50 | 0 | JOB_SUBMIT | 3952 | fe7a3d0b | jobid=1868595630964738 subtime=1782031661 state=1
|
||||
51 | 0 | JOB_SUBMIT | 3952 | fe7a3d0b | jobid=1868595630964738 subtime=1782031661 state=1
|
||||
52 | 0 | JOB_SUBMIT | 3952 | 37d54e4d | jobid=1868595630964739 subtime=1782031661 state=1
|
||||
53 | 0 | JOB_SUBMIT | 3952 | 37d54e4d | jobid=1868595630964739 subtime=1782031661 state=1
|
||||
54 | 0 | NOOP | 0 | 00000000 |
|
||||
segments: 3 entries: 54 submits: 50
|
||||
lsn range: 1..54 recovered tip: 54
|
||||
gaps: 0 crc chain: intact
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
1 | 0 | NOOP | 0 | 00000000 |
|
||||
2 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
3 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
4 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
5 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
6 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
7 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
8 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
9 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
10 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
11 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
12 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
13 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
14 | 0 | NOOP | 0 | 00000000 |
|
||||
15 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
16 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
17 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
18 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
19 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
20 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
21 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
22 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
23 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
24 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
25 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
26 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
27 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
28 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
29 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
30 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
31 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
32 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
33 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
34 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
35 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
36 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
37 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
38 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
39 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
40 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
41 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
42 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
43 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
44 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
45 | 0 | NOOP | 0 | 00000000 |
|
||||
46 | 0 | JOB_SUBMIT | 3952 | d32ac747 | jobid=1868595643547648 subtime=1782031673 state=1
|
||||
47 | 0 | JOB_SUBMIT | 3952 | d32ac747 | jobid=1868595643547648 subtime=1782031673 state=1
|
||||
48 | 0 | JOB_SUBMIT | 3952 | 97c04cbe | jobid=1868595643547649 subtime=1782031673 state=1
|
||||
49 | 0 | JOB_SUBMIT | 3952 | 97c04cbe | jobid=1868595643547649 subtime=1782031673 state=1
|
||||
50 | 0 | JOB_SUBMIT | 3952 | d7ba280a | jobid=1868595643547650 subtime=1782031673 state=1
|
||||
51 | 0 | JOB_SUBMIT | 3952 | d7ba280a | jobid=1868595643547650 subtime=1782031673 state=1
|
||||
52 | 0 | JOB_SUBMIT | 3952 | 1e155b4c | jobid=1868595643547651 subtime=1782031673 state=1
|
||||
53 | 0 | JOB_SUBMIT | 3952 | 1e155b4c | jobid=1868595643547651 subtime=1782031673 state=1
|
||||
54 | 0 | NOOP | 0 | 00000000 |
|
||||
segments: 3 entries: 54 submits: 50
|
||||
lsn range: 1..54 recovered tip: 54
|
||||
gaps: 0 crc chain: intact
|
||||
|
|
@ -1,57 +0,0 @@
|
|||
1 | 0 | NOOP | 0 | 00000000 |
|
||||
2 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
3 | 0 | JOB_SUBMIT | 3952 | 558f45c3 | jobid=1868595112968192 subtime=1782031167 state=1
|
||||
4 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
5 | 0 | JOB_SUBMIT | 3952 | 1165ce3a | jobid=1868595112968193 subtime=1782031167 state=1
|
||||
6 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
7 | 0 | JOB_SUBMIT | 3952 | 511faa8e | jobid=1868595112968194 subtime=1782031167 state=1
|
||||
8 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
9 | 0 | JOB_SUBMIT | 3952 | 98b0d9c8 | jobid=1868595112968195 subtime=1782031167 state=1
|
||||
10 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
11 | 0 | JOB_SUBMIT | 3952 | 5cae9b59 | jobid=1868595112968196 subtime=1782031167 state=1
|
||||
12 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
13 | 0 | JOB_SUBMIT | 3952 | 184410a0 | jobid=1868595112968197 subtime=1782031167 state=1
|
||||
14 | 0 | NOOP | 0 | 00000000 |
|
||||
15 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
16 | 0 | JOB_SUBMIT | 3960 | 9500c01f | jobid=1868595360432128 subtime=1782031403 state=1
|
||||
17 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
18 | 0 | JOB_SUBMIT | 3960 | 3c740aaf | jobid=1868595360432129 subtime=1782031403 state=1
|
||||
19 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
20 | 0 | JOB_SUBMIT | 3960 | f184343c | jobid=1868595360432130 subtime=1782031403 state=1
|
||||
21 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
22 | 0 | JOB_SUBMIT | 3960 | 98c7a13f | jobid=1868595360432131 subtime=1782031403 state=1
|
||||
23 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
24 | 0 | JOB_SUBMIT | 3960 | d9d22050 | jobid=1868595531350016 subtime=1782031566 state=1
|
||||
25 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
26 | 0 | JOB_SUBMIT | 3960 | 1bf6e2e0 | jobid=1868595531350017 subtime=1782031566 state=1
|
||||
27 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
28 | 0 | JOB_SUBMIT | 3960 | 2636be93 | jobid=1868595531350018 subtime=1782031566 state=1
|
||||
29 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
30 | 0 | JOB_SUBMIT | 3960 | 44ce61c1 | jobid=1868595531350019 subtime=1782031566 state=1
|
||||
31 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
32 | 0 | JOB_SUBMIT | 3960 | fd6a1b97 | jobid=1868595531350020 subtime=1782031566 state=1
|
||||
33 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
34 | 0 | JOB_SUBMIT | 3960 | 04591e02 | jobid=1868595532398592 subtime=1782031567 state=1
|
||||
35 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
36 | 0 | JOB_SUBMIT | 3960 | c67ddcb2 | jobid=1868595532398593 subtime=1782031567 state=1
|
||||
37 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
38 | 0 | JOB_SUBMIT | 3960 | fbbd80c1 | jobid=1868595532398594 subtime=1782031567 state=1
|
||||
39 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
40 | 0 | JOB_SUBMIT | 3960 | 11d8be08 | jobid=1868595533447168 subtime=1782031568 state=1
|
||||
41 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
42 | 0 | JOB_SUBMIT | 3960 | 83927249 | jobid=1868595533447169 subtime=1782031568 state=1
|
||||
43 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
44 | 0 | JOB_SUBMIT | 3960 | ee3c20cb | jobid=1868595533447170 subtime=1782031568 state=1
|
||||
45 | 0 | JOB_SUBMIT | 3952 | 4c6d24e6 | jobid=1868595618381824 subtime=1782031649 state=1
|
||||
46 | 0 | JOB_SUBMIT | 3952 | 4c6d24e6 | jobid=1868595618381824 subtime=1782031649 state=1
|
||||
47 | 0 | JOB_SUBMIT | 3952 | 0887af1f | jobid=1868595618381825 subtime=1782031649 state=1
|
||||
48 | 0 | JOB_SUBMIT | 3952 | 0887af1f | jobid=1868595618381825 subtime=1782031649 state=1
|
||||
49 | 0 | JOB_SUBMIT | 3952 | 48fdcbab | jobid=1868595618381826 subtime=1782031649 state=1
|
||||
50 | 0 | JOB_SUBMIT | 3952 | 48fdcbab | jobid=1868595618381826 subtime=1782031649 state=1
|
||||
51 | 0 | JOB_SUBMIT | 3952 | 8152b8ed | jobid=1868595618381827 subtime=1782031649 state=1
|
||||
52 | 0 | JOB_SUBMIT | 3952 | 8152b8ed | jobid=1868595618381827 subtime=1782031649 state=1
|
||||
53 | 0 | JOB_SUBMIT | 3952 | 37d54e4d | jobid=1868595630964739 subtime=1782031661 state=1
|
||||
54 | 0 | NOOP | 0 | 00000000 |
|
||||
segments: 2 entries: 54 submits: 51
|
||||
lsn range: 1..54 recovered tip: 54
|
||||
gaps: 0 crc chain: intact
|
||||
|
|
@ -1,61 +0,0 @@
|
|||
# Requirements Document
|
||||
|
||||
## Project Description (Input)
|
||||
|
||||
**Who has the problem:** Cluster operators running the XCE HA controller cluster — and every client whose acknowledged job/queue/worker operations must survive controller failover.
|
||||
|
||||
**Current situation:** In a 3-node failover test, committed jobs were permanently lost (`joblist` dropped 10→3; the 7 earliest, already acknowledged, vanished). Root cause (bug-661): **acknowledgment does not imply durability.** The controller advances its commit position on a majority of replication **acknowledgments**, but the primary counts its own contribution as soon as the entry is written to volatile buffers, not after it is persisted to stable storage — background persistence happens on a timer. A standby already persists before acknowledging, but the primary does not. So a "committed" operation can be durable on as few as one member; a single ill-timed crash + restart, or a restarted member with a truncated log winning leadership, discards it. This violates the documented assumption that a committed operation is held durably on a majority.
|
||||
|
||||
**What should change (chosen scope — durable commit + promotion fence):**
|
||||
1. **Durable commit:** when the controller acknowledges an operation to a client, that operation's log entry must already be persisted to stable storage on a fault-tolerant majority (the acknowledging member plus ⌊N/2⌋ others). The controller must count its own copy toward the commit majority only once it is durably persisted.
|
||||
2. **Recovery integrity and leadership fence (fail-loud):** a member that cannot reconstruct its committed state from persisted storage must be barred from leadership until it has caught up, and must never become authoritative while behind the cluster's known committed high-water mark — surfacing the failure loudly rather than silently discarding committed operations.
|
||||
|
||||
The mechanism (synchronous persistence vs. later group-commit batching) is a design decision; group-commit batching is deferred to the roadmap as a contract-preserving throughput optimization. See `.kiro/specs/ha-commit-durability/brief.md` for the full discovery analysis, evidence trail, and design decision.
|
||||
|
||||
## Introduction
|
||||
|
||||
This spec closes the durability gap beneath the HA consistency machinery: it makes acknowledgment imply durability. After this spec, any operation the controller reports as successful is guaranteed to be persisted on a majority of members at the moment of acknowledgment, and therefore survives the loss of any minority of members and any subsequent leadership change. It also fences a member that cannot recover its committed state — or that lags the cluster's committed high-water mark — from becoming the authoritative leader, so a truncated or corrupted log can never overwrite acknowledged operations. The client reply path already waits on the commit barrier; this spec strengthens the meaning of "committed" to "durably persisted on a majority," so the existing reply gate enforces the invariant. A testability provision makes the guarantee deterministically verifiable without depending on race timing.
|
||||
|
||||
## Boundary Context
|
||||
|
||||
- **In scope**: the durable-commit invariant on the primary's commit path (self contributes to the commit majority only when durably persisted); pinning the standby's persist-before-acknowledge behavior; the no-premature-acknowledgment guarantee; survival of acknowledged operations across the crash+restart of any minority; preserving cluster liveness (heartbeat/election) while performing durability work; recovery-integrity and leadership fencing for members that cannot reconstruct committed state or lag the committed high-water mark; degraded-quorum preservation of the invariant; a deterministic durability-verification provision; the soak E2E (`src/tools/test-commit-durability.sh`) plus unit coverage.
|
||||
- **Out of scope**: the replication/acknowledgment wire format (`ha-wal-replication`); per-entry majority arithmetic and commit-index advancement (`ha-commit-engine` — consumed, this spec changes only the durability *precondition* on its inputs); demotion truncate/self-restart and rejoin reconciliation of an intact log (`ha-convergence-recovery`); promotion no-op and own-term gate (`ha-single-leader-role`); raw vote-grant arithmetic (`raft-library` — this spec adds a durability/high-water *precondition*, not new vote logic); WAL group-commit batching (roadmap — a contract-preserving throughput optimization, deferred); single-record job-array persistence (`job-array-single-wal-record`, roadmap); entity-bearing snapshots (roadmap); client-side controller failover (`libxce-failover`).
|
||||
- **Adjacent expectations**: relies on `ha-commit-engine`'s commit-position semantics (monotonic, per-entry majority) and on `ha-wal-replication`'s acknowledgment and catch-up machinery; the client reply is already gated on the commit barrier (`ha-commit-barrier-rollout`), so strengthening "committed" to "durable-majority" enforces the invariant without changing the reply path. Downstream, `ha-convergence-recovery`'s single-outcome convergence depends on this spec making "persisted committed state" genuinely durable; a returning member that this spec fences from leadership until caught up complements that spec's truncate-and-restart reconciliation.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Requirement 1: Durable Commit — Acknowledgment Implies Majority Persistence
|
||||
|
||||
**Objective:** As a cluster operator, on behalf of every client, I want any operation the controller acknowledges as successful to already be durably persisted on a majority of controllers, so that no acknowledged work is ever lost to a controller failure or failover.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the controller acknowledges a state-changing operation to a client as successful, the controller shall have durably persisted that operation's log entry to stable storage on a majority of cluster members — the acknowledging member plus at least ⌊N/2⌋ other members.
|
||||
2. While evaluating whether an operation is committed, the controller shall count its own log entry toward the commit majority only after that entry is durably persisted to its local stable storage, not when it is merely written to volatile buffers.
|
||||
3. When a standby acknowledges a replicated log entry, the standby shall durably persist that entry to its local stable storage before sending the acknowledgment.
|
||||
4. While a state-changing operation's log entry is not yet durably persisted on a majority of members, the controller shall not report that operation as successful to the requester.
|
||||
5. If any minority of cluster members that contributed to an operation's commit are simultaneously lost and restarted, the acknowledged operation shall remain present and recoverable on the surviving cluster.
|
||||
6. While performing durability persistence for a commit, the controller shall continue to service heartbeat and election processing without stalling, so that durability work does not induce spurious failover.
|
||||
|
||||
### Requirement 2: Recovery Integrity and Leadership Fence
|
||||
|
||||
**Objective:** As a cluster operator, I want a controller that cannot reconstruct its committed state to be barred from leadership until it has caught up, so that a member with a truncated or corrupted log can never become authoritative and discard acknowledged operations.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. When the controller recovers its persisted log at startup, if the log cannot be reconstructed contiguously up to its last known committed boundary, the controller shall not silently discard committed entries and continue as if its committed state were complete.
|
||||
2. If a controller's recovered log does not reach its last known committed boundary, the controller shall be ineligible to become primary until it has caught up to the cluster's committed boundary from a healthy member.
|
||||
3. While a controller's committed position is below the cluster's known committed high-water mark, the controller shall not assume leadership in a way that would discard already-committed operations.
|
||||
4. If a controller cannot recover to a consistent committed state and cannot catch up from a healthy member, the controller shall surface the condition as a loud, operator-visible alert and hold itself permanently non-promotable while staying alive and continuing to retry catch-up — refusing to serve potentially divergent state by refusing leadership, NOT by terminating the process. A controller MUST NEVER self-terminate (`exit`/`abort`) as a designed response to an operational condition: a self-exited controller requires manual admin recovery and degrades quorum meanwhile, whereas one that stays alive and keeps retrying self-heals automatically when a peer returns (see Requirement 2.6). (A genuine programming-invariant `assert`, or an external admin/supervisor kill, is not such a designed response.)
|
||||
5. While the cluster operates with reduced redundancy or minimal quorum, the controller shall preserve the durable-commit guarantee of Requirement 1 across leadership changes, so that no acknowledged operation is lost even when quorum is temporarily degraded.
|
||||
6. When a fenced member finishes catching up to the cluster's committed boundary, the controller shall restore that member's eligibility to participate in leadership.
|
||||
|
||||
### Requirement 3: Deterministic Durability Verification
|
||||
|
||||
**Objective:** As a developer verifying durability, I want the durability behavior to be deterministically controllable in a verification mode, so that the durable-before-acknowledge guarantee can be confirmed without depending on race timing.
|
||||
|
||||
#### Acceptance Criteria
|
||||
|
||||
1. Where a durability-verification mode is enabled, the controller shall make the durability behavior deterministically controllable (for example, by configuring or disabling background persistence) so that an acknowledged operation's durability can be asserted without relying on timing.
|
||||
2. Where durability-verification mode removes reliance on background persistence, an operation that the controller acknowledged as successful shall be recoverable after an immediate, ungraceful termination of a contributing member.
|
||||
3. The controller shall preserve the same externally observable acknowledgment semantics whether or not durability-verification mode is enabled, so that the mode changes timing controllability only, not the durability contract.
|
||||
|
|
@ -1,220 +0,0 @@
|
|||
# Gap Analysis — ha-commit-durability
|
||||
|
||||
> Brownfield analysis against the XCE controller (`src/controller/`). Informs `/kiro-spec-design`.
|
||||
> Line numbers from the working tree at analysis time (2026-06-18); re-confirm before editing.
|
||||
|
||||
## Current-State Summary
|
||||
|
||||
The HA stack is mature and well-factored: per-LSN majority commit engine (`commit.c`), fan-out WAL replication with ACK/catch-up (`repl.c`, `config_repl.c`), a commit barrier that gates client replies (`persist_commit_barrier`), demotion truncate + rejoin reconciliation (`ha_convergence.c`), single-leader own-term gate, PreVote, and a vote-freshness gate. The durability gap is **beneath** all of it: "committed" (majority-ACKed) does not imply "durable on a majority", and recovery/promotion trust in-memory position rather than the durable log.
|
||||
|
||||
The submit reply path is already commit-gated and runs **off the iomgr thread** (`job.c:568-569` — "the AFTER_EVENT hook writes WAL and waits for majority replication"), so durability work can be added to that path without stalling heartbeats.
|
||||
|
||||
## Requirement-to-Asset Map
|
||||
|
||||
### Requirement 1 — Durable Commit (acknowledgment ⇒ majority persistence)
|
||||
|
||||
| Need | Existing asset | Status |
|
||||
|------|----------------|--------|
|
||||
| Reply gated on commit | `persist_commit_barrier` → `commit_submit(COMMIT_MODE_WAIT)` (persist.c); callers `job.c:1024`, `queue.c:144`, `worker.c:713`, `join.c:130` | ✅ Exists — reuse; strengthen meaning of "committed" |
|
||||
| Standby persists before ACK | `persist_wal_write_replicated` `fdatasync` before return (persist.c:515) | ✅ Exists — **pin with regression test** (Req 1.3) |
|
||||
| Primary self-counts only when durable | `commit.c:majority_candidate_locked` sets `cap = highest_appended_lsn(ce->p)` (appended, not durable) | ❌ **Missing** — core defect (bug-661) |
|
||||
| Primary persists before self-count | `persist_wal_async` (persist.c:340) appends via `pwrite`, broadcasts, returns **without fsync**; durability deferred to `fsync_worker` timer (persist.c:173) | ❌ **Missing** |
|
||||
| Track highest durable LSN | none — `persist_t` has `committed_lsn`/`current_lsn` but no `durable_lsn` | ❌ **Missing** (new field) |
|
||||
| Liveness during durability work | commit-wait path already off iomgr (`job.c:568-569`) | ✅ Constraint satisfied by placement |
|
||||
|
||||
**Minimal change shape:** add `durable_lsn` to `persist_t`, advanced when the entry's bytes are `fdatasync`'d; `persist_wal_async` fsyncs (inline, synchronous design — see brief decision) and sets `durable_lsn` before returning; `majority_candidate_locked` uses `min(durable_lsn, …)` for the self contribution instead of `highest_appended_lsn`. Reply gate then enforces the invariant unchanged.
|
||||
|
||||
### Requirement 2 — Recovery Integrity + Leadership Fence
|
||||
|
||||
| Need | Existing asset | Status |
|
||||
|------|----------------|--------|
|
||||
| No silent truncate-to-zero on corrupt recovery | `replay_wal` breaks on CRC chain break / CRC mismatch / short read (persist.c:1021, 1050, 1042) and returns the replayed count as **success**; only `-ENOMEM` propagates (persist.c:1107, 1152) | ❌ **Missing** — silent truncate-accept is the amplifier |
|
||||
| Durable floor re-derived on restart | `persist_recover` **never sets `committed_lsn`** — it stays 0 after restart; floor is reconstructed only at promotion (persist.c:1114) | ❌ **Missing** — restart advertises in-memory floor unbacked by WAL |
|
||||
| Pre-promotion integrity / caught-up gate | `raft_cb_become_leader` (ha.c:551) computes `safe_lsn` from votes (ha.c:572) then `persist_truncate_wal_above(safe_lsn)` (ha.c:577) — **no check that local WAL is intact or ≥ safe_lsn** | ❌ **Missing** — hollow node can promote, truncate collapses tip without backfill |
|
||||
| Freshness floor from durable WAL | `ha_vote_self_freshness` = `max(applied_lsn, last_known_committed_lsn)` — **in-memory atomics**, not re-derived from durable WAL (ha.c:2127; used cluster.c:380-396 vote, cluster.c:467-474 PreVote) | ❌ **Missing** — truncated candidate advertises a floor it can't back |
|
||||
| Fence "ineligible until caught up" | `in_catch_up` atomic exists but only a **standby** uses it to deny its **own** outgoing vote (ha.c:325, cluster.c:529); no fence for a freshly-recovered/promoting node | ⚠️ Partial — extend concept |
|
||||
| Re-eligibility on catch-up | standby catch-up apply `ha_apply_wal_catchup_entry` (ha.c:1192); re-inclusion exists | ✅ Exists — hook the fence release here |
|
||||
|
||||
### Requirement 3 — Deterministic Durability Verification
|
||||
|
||||
| Need | Existing asset | Status |
|
||||
|------|----------------|--------|
|
||||
| Configurable durability cadence / disable background flush | `wal_sync_ms` = `DEFAULT_WAL_SYNC_MS` (config.h:86, value 10); `persist_config_defaults` (persist.h:30); **no env/CLI/config override** — callers `myapp.c:164`, `cluster.c:1267` pass defaults unmutated | ❌ **Missing** — must add an override |
|
||||
| Test-mode seam convention | `#ifdef XCE_TESTING` hook shims (ha.c:2270-2335), test-only `handle_request_vote` (ha.c:2161); cluster.c not linked under XCE_TESTING (cluster.c:67) | ✅ Pattern exists — follow it |
|
||||
|
||||
> Doc drift noted: `persistence-and-ha-design.md:4178` says `wal_sync_ms=50`; code is 10 (config.h:86). Reconcile in design.
|
||||
|
||||
## Implementation Approach Options
|
||||
|
||||
**Option A — Extend existing components (recommended).** All change sites are inside existing, well-scoped functions (`commit.c`, `persist.c`, `ha.c`, `cluster.c`). R1 = `durable_lsn` field + 2 edits (`persist_wal_async` fsync, `majority_candidate_locked` self cap). R2 = make `replay_wal` return an integrity verdict + a pre-promotion gate in `raft_cb_become_leader` + floor-from-durable-WAL in `ha_vote_self_freshness`. R3 = config override + an `XCE_TESTING` seam.
|
||||
- ✅ Leverages the commit-barrier and `in_catch_up` machinery already present; smallest surface; matches TLA+ mapping in `CLAUDE.md`.
|
||||
- ❌ Touches correctness-critical election/commit code — demands rigorous TDD and TLA+ re-verification.
|
||||
|
||||
**Option B — New durability module.** Extract a `durability`/`durable_lsn` tracker as a new component.
|
||||
- ✅ Isolated unit testing. ❌ Over-engineered — `durable_lsn` is one atomic on `persist_t`; a new module adds indirection without separation benefit. Not recommended.
|
||||
|
||||
**Option C — Hybrid (recommended for R2 only).** Extend in place for R1/R3; add a small new **recovery-integrity + promotion-eligibility helper** (e.g. `persist_recovered_intact()` / `ha_promotion_durable_ok()`) for R2 so the integrity verdict and the fence predicate are unit-testable and don't bloat `raft_cb_become_leader`.
|
||||
- ✅ Keeps the election hot path readable; testable fence predicate. Recommended: **A for R1+R3, C for R2.**
|
||||
|
||||
## Effort & Risk
|
||||
|
||||
| Req | Effort | Risk | Justification |
|
||||
|-----|--------|------|---------------|
|
||||
| R1 Durable commit | **M** (3–7d) | **Medium-High** | Localized (`durable_lsn` + 2 edits) but correctness-critical; needs TLA+ re-check (commit rule, leader completeness) and durability TDD. Liveness already safe (off-iomgr path). |
|
||||
| R2 Recovery + fence | **L** (1–2wk) | **High** | Touches recovery, promotion, and the vote-freshness floor; must **compose with `ha-convergence-recovery`** (truncate/rejoin) and `ha-single-leader-role` (own-term gate) without double-handling, and honor `CLAUDE.md`'s "never truncate from stale ctrlhello" rule. Election-safety sensitive. |
|
||||
| R3 Verification seam | **S** (1–3d) | **Low** | Config override + `XCE_TESTING` hook following an established pattern. |
|
||||
|
||||
## Recommendations for Design Phase
|
||||
|
||||
**Preferred approach:** Option A (extend) for R1 + R3; Option C (extend + small fence helper) for R2.
|
||||
|
||||
**Key design decisions to resolve:**
|
||||
1. **Durability mechanism** — already decided (brief): synchronous persist-before-self-count, eliminate `fsync_worker` as the durability mechanism; group-commit batching deferred to roadmap. Design must confirm `fsync_worker`'s fate: delete entirely vs retain a best-effort flush for any non-client-acked / local-only write (audit `wal_classify` for writes that gate no reply; checkpoint markers already fsync inline at persist.c:573).
|
||||
2. **`durable_lsn` semantics** — single atomic on `persist_t`; advanced under `p->lock` after `fdatasync`; consumed by `majority_candidate_locked`. Confirm interaction with `persist_truncate_wal_above` (must not let `durable_lsn` exceed a post-truncation tip).
|
||||
3. **Recovery integrity verdict** — `replay_wal`/`persist_recover` must distinguish "clean EOF" from "stopped at corruption". Decide: FATAL on corruption (fail-loud, simplest, honors the user's fail-loud preference) vs flag-and-fence (recover what's intact, fence from leadership until caught up). The fail-loud preference (`[[feedback_fail_loud_over_timeout]]`) leans FATAL unless the node can provably catch up.
|
||||
4. **Freshness floor source** — derive the candidate's advertised floor from the **durable WAL tip**, not in-memory `applied_lsn`/`last_known_committed_lsn` alone, so a truncated node cannot advertise an unbacked floor. Reconcile with `CLAUDE.md`: never truncate using a stale ctrlhello `last_known_committed_lsn`; the safe-commit point still comes from voter replies at promotion.
|
||||
5. **Compose with convergence-recovery** — R2's pre-promotion fence must not duplicate `ha_convergence.c`'s demotion-truncate/rejoin logic; define the seam (this spec fences *promotion*; convergence handles *demotion* and *rejoin reconciliation* of an intact log).
|
||||
|
||||
**Research items to carry forward:**
|
||||
- Measure `fdatasync` latency on the target storage vs the existing replication round-trip, to confirm the synchronous-design latency add is acceptable before group-commit is needed.
|
||||
- Confirm no client-acked operation class bypasses the commit barrier (`wal_classify` audit) — any that do would violate the invariant.
|
||||
- Decide the `wal_sync_ms` override surface (env var vs CLI flag vs config field) consistent with existing `--sync-timeout-ms` precedent (main.c `OPT_SYNC_TIMEOUT`).
|
||||
- Re-verify against `raft/raft.tla` and the `CLAUDE.md` Protocol Verification Checklist after the self-count change (commit rule, leader completeness).
|
||||
|
||||
---
|
||||
|
||||
## Design Synthesis (2026-06-18)
|
||||
|
||||
Decisions resolved into `design.md`:
|
||||
- **Durability mechanism = synchronous.** Add `persist_t.durable_lsn` (atomic, advanced after `fdatasync`); `persist_wal_async` persists before return; `commit.c:majority_candidate_locked` self-contribution uses `durable_lsn` not `highest_appended_lsn`. **`fsync_worker` removed** as the durability mechanism (checkpoint markers keep their inline fsync). Group-commit batching deferred to roadmap as a strict, contract-preserving optimization.
|
||||
- **No new module (Option A) for R1/R3; small fence helper (Option C) for R2.** `durable_lsn` is one atomic — a module would be over-abstraction (build-vs-adopt: extend).
|
||||
- **Recovery: fail-loud + fence.** `replay_wal` returns a verdict (`RECOVER_CLEAN` vs `RECOVER_TRUNCATED`); a truncated member is fenced from promotion and catches up from a healthy peer; FATAL only when it cannot reach a consistent committed state and is required for quorum. Balances `[[feedback_fail_loud_over_timeout]]` with availability.
|
||||
- **Freshness floor from durable WAL**, not volatile in-memory `last_known_committed_lsn`; safe-commit point at promotion still from voter replies (honors `CLAUDE.md`).
|
||||
- **Compose with convergence-recovery:** this spec fences *promotion* + verifies recovery integrity; convergence-recovery owns *demotion* truncate and *rejoin* of an intact log. No shared ownership of the commit index (kept volatile per Raft).
|
||||
- **Liveness:** the added `fdatasync` lands on the already-off-iomgr commit path (`job.c:568-569`), so heartbeats are unaffected — Option A's only real risk is retired.
|
||||
|
||||
Generalizations/simplifications: reuse the existing `in_catch_up` fence concept (extend from streaming-standby to recovery-truncated promoter) rather than inventing a new fence; reuse `persist_commit_barrier` reply gate unchanged; follow `--sync-timeout-ms` and `XCE_TESTING` hook conventions for R3.
|
||||
|
||||
---
|
||||
|
||||
## Gap Re-Audit (2026-06-18)
|
||||
|
||||
> **Post-implementation gap audit.** All 17 tasks (1.1–5.3) + the never-self-exit
|
||||
> rework are marked `[x]` (landed 2026-06-18). This pass re-verifies each EARS
|
||||
> acceptance criterion against the on-disk code (codegraph + grep), tags each
|
||||
> MET / Missing / Unknown / Constraint, and records the known residual
|
||||
> test-thoroughness items called out in the Implementation Notes. Line numbers
|
||||
> from the working tree at audit time; re-confirm before editing.
|
||||
|
||||
### Requirement-to-Asset Map (verified against code)
|
||||
|
||||
#### Requirement 1 — Durable Commit (acknowledgment ⇒ majority persistence)
|
||||
|
||||
| Crit | Asset (file:line) | Tag |
|
||||
|------|-------------------|-----|
|
||||
| 1.1 ack ⇒ durable on majority | Reply gated on commit barrier (unchanged) + self durable-capped (`commit.c:93`) + standby persist-before-ACK (`persist.c:761`); end-to-end test `test_durable_majority_commit_end_to_end` | **MET** |
|
||||
| 1.2 self counts only when durable | `majority_candidate_locked` self-cap `cap = persist_durable_lsn(ce->p)` (`commit.c:93`, was `highest_appended_lsn`); debug invariant `assert(cand <= persist_durable_lsn)` (`commit.c:162`) | **MET** |
|
||||
| 1.3 standby persists before ACK | `persist_wal_write_replicated` inline `fdatasync` rc-gated, `return -EIO` on failure so the standby does NOT ACK (`persist.c:761-773`); caller `ha_apply_single_entry` ACKs only on rc==0 (no-apply/no-ACK on negative). Pinned by `test_replicated_persist_syncs_before_return` / `test_replicated_sync_failure_no_ack` (counter `test_replicated_fdatasync_count`) | **MET** |
|
||||
| 1.4 not reported until durable-majority | committed_lsn cannot advance over a non-durable entry (1.2 cap) → existing WAIT-mode reply gate enforces it unchanged | **MET** |
|
||||
| 1.5 minority loss survives | Property test `test_sole_durable_contributor_loss_survives` (committed ⇒ ≥⌊N/2⌋+1 durable copies, drop one); live failover proof via soak (5.2) | **MET** (live N=2 fault is decision/property-level — see residual G1) |
|
||||
| 1.6 liveness during durability work | Group-commit flusher (`fsync_worker`) syncs OFF the iomgr/caller thread; `fdatasync` under `p->lock` only (never with `fsync_mutex`); `test_durability_work_off_iomgr_preserves_liveness` asserts flush on a non-caller thread + heartbeat reads <50ms | **MET** |
|
||||
|
||||
Mechanism note: `persist_wal_async` no longer fsyncs inline (`persist.c:643-652` signals the flusher); durability watermark `persist_t.durable_lsn` (`_Atomic uint64_t`) advanced ONLY after a successful `fdatasync` (flusher `persist.c:281-311`, `persist_flush` `persist.c:801-806`). `fsync_worker` repurposed from fixed-interval timer to demand-driven group-commit flusher; checkpoint markers retain their inline `fdatasync` (`persist.c:162`). Legacy `sync_append` ACK-commit store now durable-gated (`sync_append_legacy_commit_store`, store only when `written_lsn <= durable_lsn`).
|
||||
|
||||
#### Requirement 2 — Recovery Integrity + Leadership Fence
|
||||
|
||||
| Crit | Asset (file:line) | Tag |
|
||||
|------|-------------------|-----|
|
||||
| 2.1 no silent truncate-accept | `replay_wal(p, from_lsn, recover_verdict_t*)` classifies CRC chain break / CRC mismatch / short header / short payload → `RECOVER_TRUNCATED`; clean EOF / pre-alloc zero tail / empty WAL → `RECOVER_CLEAN` (`persist.c:1411-1605`); verdict stored on `persist_t.recovery_verdict`, accessors `persist_recovery_verdict`/`persist_recovery_truncated` (production, not XCE_TESTING-gated) | **MET** |
|
||||
| 2.2 ineligible until caught up | Pre-promotion gate in `raft_cb_become_leader` (`ha.c:952-1010`): REFUSE if `persist_recovery_truncated` OR `durable_floor < promotion_safe_lsn` (read AFTER truncate-clamp), set `in_catch_up`, return WITHOUT becoming primary | **MET** |
|
||||
| 2.3 below-HWM never authoritative | Durable-backed freshness floor: candidate advertises bare `ha_candidate_advertise_lsn = persist_durable_floor` (`ha.c:2587`), voter floor `max(durable_floor, learned)`; promotion gate compares clamped `durable_floor` vs voter-reply `promotion_safe_lsn`; floor re-derived from recovered WAL tip in `persist_recover` (assign, not max) | **MET** |
|
||||
| 2.4 loud alert, hold non-promotable, STAY ALIVE, never self-exit | `persist_recover_loud_fence_if_unrecoverable` (`persist.c:1787`): loud ERROR with truncation boundary + sticky `_Atomic bool recovery_unrecoverable`, **NO abort/exit/tlog_exit**; `RECOVER_DECISION_FENCE_UNRECOVERABLE` (non-terminating). grep confirms zero abort/exit on the recovery path; `persist_fatal_hook` removed from production (only a test comment remains) | **MET** |
|
||||
| 2.5 preserve durable-commit at degraded/minimal quorum | Hollow node refused regardless of quorum size (gate is unconditional on truncated/behind) + 4.2 freshness floor; `test_promotion_gate_hollow_refused_at_minimal_quorum` (decision-level, Req 2.5) | **MET** (decision-level; no dedicated live N=2 fault-injection — residual G1) |
|
||||
| 2.6 re-eligibility on catch-up | Fence/flag clear automatically when `!truncated && durable_floor >= HWM`: `ha_evaluate_promotion_fence` ELIGIBLE branch clears `in_catch_up` (`ha.c:592-597`); `promotion_fence_timeout_fn` releases + `persist_clear_recovery_unrecoverable` on catch-up (`ha.c:755-763`); gate ELIGIBLE branch clears + cancels timer (`ha.c:986-993`) | **MET** |
|
||||
|
||||
Bounded-window catch-up (task 4.4): `raft_cb_become_leader` NEVER FATALs/aborts synchronously — on FENCE/would-be-unrecoverable it arms a one-shot `promotion_fence_timeout_fn` (`DEFAULT_RECOVERY_CATCHUP_WINDOW_MS=30000`) that re-reads live state, releases on catch-up, re-emits the loud alert + re-arms if still hollow with no peer — never exits (`ha.c:721-807`). UAF-safe cancel; cancelled in `ha_fini`.
|
||||
|
||||
#### Requirement 3 — Deterministic Durability Verification
|
||||
|
||||
| Crit | Asset (file:line) | Tag |
|
||||
|------|-------------------|-----|
|
||||
| 3.1 configurable durability cadence | `--wal-sync-ms` flag (`main.c:67,118`) → `myconf.wal_sync_ms` → `persist_config_set_wal_sync_ms` (`persist.h:130`) applied at BOTH persist_init sites (`myapp.c:169` production, `cluster.c:1284` phase2); 0-sentinel = keep `DEFAULT_WAL_SYNC_MS`. Test seam `persist_test_durable_lsn_on_disk` (read-only on-disk CRC-validated durable LSN) + `persist_test_set_force_fsync_fail` fault injection, all XCE_TESTING-gated (`persist.h:445-506`) | **MET** |
|
||||
| 3.2 ack ⇒ recoverable after ungraceful kill in verify-mode | Deterministic E2E (`test-commit-durability.sh --deterministic`): with `--wal-sync-ms 30000`, in-window primary `fdatasync >= 1` (=2) precedes ack ("durable-before-ack"), flipped RED→GREEN stable 3/3 / 9-9 probes (5.2) | **MET** |
|
||||
| 3.3 same observable ack semantics with/without verify-mode | Documented as a property in the test header (`test_commit_durability.c:33-34`): verify-mode "changes nothing observable about the acknowledgment contract" — but there is **NO dedicated A/B assertion** running the same op with mode on vs off and asserting identical observable acknowledgment | **PARTIAL** (residual G2) |
|
||||
|
||||
Constraint compliance: `safe_committed_lsn` arithmetic UNTOUCHED (voter-reply-sourced, honors `CLAUDE.md` "never truncate from stale ctrlhello"). raft.tla election/freshness logic maps faithfully; raft.tla does **NOT** model durability (restores `log` intact on restart, `raft.tla:165-174`) — the commit-durability invariant is verified by tests 2.x/5.1/5.2 + the manual Protocol Verification Checklist, NOT by raft.tla. (Tagged **Constraint** — by-design deferral, see residual G3.)
|
||||
|
||||
### Residual Gaps
|
||||
|
||||
| ID | Gap | Severity | Notes |
|
||||
|----|-----|----------|-------|
|
||||
| **G1** | Req 2.5 has no dedicated **minimal-quorum live fault-injection** test (N=2, lose the sole durable contributor across a real failover). Current coverage = decision/property-level (`test_promotion_gate_hollow_refused_at_minimal_quorum`, `test_sole_durable_contributor_loss_survives`) + the rolling-SIGKILL soak. | **LOW** | Invariant is enforced by the unconditional gate + freshness floor and the soak observed zero loss; a targeted N=2 case would harden the assertion against a future regression but is not a correctness gap. |
|
||||
| **G2** | Req 3.3 has no **verify-mode A/B observable-equivalence** assertion (same op, mode on vs off, identical externally observable acknowledgment). Property is asserted only by header comment + the structural fact that the seam is read-only / production-excluded. | **LOW** | The seam (`persist_test_*`, `--wal-sync-ms`) demonstrably changes only timing, not the reply path (reply gate unchanged). An explicit A/B test would close the letter of 3.3. |
|
||||
| **G3** | raft.tla does **not** model durable-vs-volatile log; the formal model cannot prove the commit-durability invariant. | **LOW (by design)** | Explicitly deferred (task 5.3 note, requirements out-of-scope). Durability is test-backed. A durable-log TLA+ extension is out of scope unless explicitly opted in. |
|
||||
|
||||
No CRITICAL/HIGH residuals. No criterion is **Missing** or **Unknown**: 16/16 verifiable EARS criteria are MET (1.1–1.6, 2.1–2.6, 3.1–3.2) with 3.3 PARTIAL (property held but not A/B-asserted). The never-self-exit invariant (Req 2.4) is upheld — grep/nm confirm no abort/exit/tlog_exit on the recovery/promotion path.
|
||||
|
||||
### Integration Risks
|
||||
|
||||
- **Group-commit flusher latency on a degrading disk** (Implementation Note 2.1 follow-up): the flusher tight-spins on SUSTAINED `fdatasync` failure (releases `p->lock` each iter so liveness holds, but CPU/log burn). A HANGING (not erroring) `fdatasync` blocks appends since it runs under `p->lock` — pre-existing, not introduced here. Non-blocking; carry to roadmap with group-commit batching.
|
||||
- **Test-fixture durable priming (bug-704 class)**: any commit-engine / election fixture that primes `current_lsn`/`applied_lsn`/`last_known_committed_lsn` but NOT `durable_lsn` will see floor/advertise=0 and committed-stall, because both the self-cap (2.2) and the freshness floor (4.2) now read durable state. Future tests MUST prime `durable_lsn` to the modeled tip. (4 fixtures already corrected; bug-760 caught one adjacent real-flusher test.)
|
||||
- **fsync-able filesystem for validation**: `/tmp` overlay in some sandboxes EIOs on `fdatasync`; because committed_lsn is now durable-gated, an EIO correctly HOLDS committed_lsn and surfaces as adjacent-test failures. Final validation MUST run on a real fsync-able fs. Gate on ANSI-stripped `:FAIL:` count (meson `Fail:0` masks Unity sub-group failures).
|
||||
|
||||
### Options for the Open Test-Thoroughness Items (G1, G2)
|
||||
|
||||
**Option A — Close both with targeted unit/integration tests (recommended).**
|
||||
- G1: add an N=2 minimal-quorum integration test reusing the `test_sole_durable_contributor_loss_survives` two-persist-dir harness, but drive it through a real failover-style commit + drop-the-durable-contributor and assert the acked op is present on the survivor.
|
||||
- G2: add an A/B test that submits the same op with verify-mode (`--wal-sync-ms` slow / forced) and without, asserting byte-identical externally observable acknowledgment (same rc, same reply gate release semantics).
|
||||
- **Effort: S (1–3d).** **Risk: Low** — test-only, no production change; reuses existing harnesses/seams.
|
||||
|
||||
**Option B — Cover via the soak/E2E harness only.**
|
||||
- Extend `test-commit-durability.sh` with an N=2 deterministic round (G1) and a mode-on/mode-off ack-equivalence probe (G2).
|
||||
- **Effort: S (1–2d).** **Risk: Low-Medium** — E2E is less deterministic than unit; the strace-gap probe is the established deterministic surface, so prefer it over joblist-loss for G2.
|
||||
|
||||
**Option C — Accept as documented residuals (status quo).**
|
||||
- Record G1/G2 as LOW-severity known limitations; rely on the gate/floor invariants + the zero-loss soak already in place.
|
||||
- **Effort: none.** **Risk: Low** — acceptable given no CRITICAL/HIGH gap, but leaves the letter of Req 3.3 (A/B) and a dedicated Req 2.5 live case unasserted, so a future regression in those exact paths could go uncaught until soak.
|
||||
|
||||
**Recommendation:** Option A — both items are S-effort, test-only, and turn two LOW residuals into fully-asserted criteria; G3 stays a by-design deferral.
|
||||
|
||||
---
|
||||
|
||||
## REOPENED 2026-06-19 — cold-restart-all (total-cluster-outage) committed-job loss (bug-795)
|
||||
|
||||
**Trigger:** the `xce-test-e2e-primitives` spec converted `test-commit-durability.sh` to exit-code primitives and ADDED a new `run_cold_restart_recovery` phase (kill ALL 3 controllers simultaneously, then restart ALL from disk, then assert every acknowledged job recovers). Running it live on a 3-node cluster surfaced a NON-DETERMINISTIC committed-job loss that this spec's original validation never exercised.
|
||||
|
||||
**What was observed (3 live soak runs, sandbox 3-node cluster):**
|
||||
- Rolling **single-node** SIGKILL + restart: ZERO loss every round (consistent with this spec's 5.2 result — single-host loss is non-deterministic and standbys fdatasync-before-WalAck, so a committed job is durable off the primary).
|
||||
- **Cold-restart-all** (kill ALL 3, restart ALL): a job that returned `jobsubmit ok` AND survived every rolling round VANISHED after the full restart (`expect-jobs expected N, actual N-2`; `expect-job <id> not found`). PASSED in one run, FAILED in another → a **race**. Recovery logs show WAL `truncate` on the rejoining nodes.
|
||||
|
||||
**Why this spec did not catch it:** 5.2's soak was rolling **single-node** kills only; the durability guarantee was validated (and CLAUDE.md flipped to "GUARANTEED") against that scenario. The **total simultaneous outage** path — where ALL members lose their volatile/page-cache tail at once and recovery must reconstruct committed state from on-disk WAL alone — was never tested. When every node dies together, the "durable off the primary" assumption that makes single-host loss non-deterministic no longer applies: only what was `fdatasync`'d on a majority before the ack can survive.
|
||||
|
||||
**Open question — REAL durability gap vs OVERLAYFS artifact (must resolve first):**
|
||||
- This sandbox's `/tmp` is **overlay** (NOT a real fsync-able fs). This spec's own notes repeatedly flag that overlay `/tmp` can EIO on `fdatasync` and that "final validation must run on a real fsync-able fs."
|
||||
- If `fdatasync` on overlay is weak (returns rc=0 without true durability) OR if recovery's truncation (`replay_wal` / `persist_truncate_wal_above` / durable-floor re-derivation, tasks 3.1/3.4) is too aggressive after a total outage, a committed entry's tail could be lost or truncated on cold restart.
|
||||
- The durable-gating (2.2/2.4) should make `committed_lsn` HOLD if `fdatasync` failed — so an un-fdatasync'd job should NOT have returned `jobsubmit ok`. Yet these jobs DID get `jobsubmit ok`. So either `fdatasync` returned success on overlay without real durability, or a non-durable-gated ack path exists under the total-outage timing, or recovery truncates committed entries.
|
||||
|
||||
**Disposition:** the `xce-test-e2e-primitives` soak is the reproduction surface and is working correctly (it detects the loss via exit codes). The fix (if real) belongs HERE. See Phase 6.
|
||||
|
||||
### UPDATE 2026-06-19b — trustworthy soak sharpens the finding (rolling-failover loss + liveness)
|
||||
|
||||
After the `xce-test-e2e-primitives` soak was HARDENED (commit d82cc78: it no longer conflates liveness with durability — durability is asserted ONLY against a confirmed-serving cluster, with a catch-up-lag confirm re-check; a non-serving cluster is INCONCLUSIVE, never "DATA LOSS"), three 6–10-round live runs give a cleaner signal:
|
||||
|
||||
1. **The committed-job loss reproduces in ROLLING single-node failover too — not only cold-restart-all.** One run lost a committed job at round 6 with the cluster CONFIRMED SERVING (`DATA LOSS — persistently absent from a SERVING cluster`); two other runs had zero loss. So the durability race is NON-DETERMINISTIC (~1 in 3 runs here) and broader than the cold-restart-all framing in the original reopen note — it is a general acked-but-not-durable-on-majority race that a kill can expose during normal rolling failover. **This is exactly the bug-661 invariant this spec claimed CLOSED for rolling failover (task 5.2) — it is not fully closed.** (Caveat still applies: sandbox `/tmp` is overlayfs; 6.1 must confirm on a real fsync-able fs.)
|
||||
2. **Separate LIVENESS concern:** every run shows 2–3 rounds where, after killing the current primary (consistently `ctrl-1`), NO new primary served within 60s (the cluster always eventually recovered). A >60s failover for a single-node kill in a 3-node cluster is poor availability and may share a root cause with the durability race (a behind/just-rejoined node interacting with the promotion fence / freshness floor). Worth a dedicated liveness check in 6.2.
|
||||
|
||||
**Phase 6 scope note:** 6.1's reproduction MUST include BOTH the cold-restart-all step AND the rolling-failover rounds (the hardened soak runs both and reports trustworthy per-round verdicts), and should capture the >60s-election cases for the liveness angle.
|
||||
|
||||
### UPDATE 2026-06-19c — 6.1 ANSWERED: the loss is REAL, not (only) an overlay artifact
|
||||
|
||||
The cold-restart-all / rolling durability loss was initially suspected to be an overlay-`/tmp` `fdatasync`-EIO artifact of the dev sandbox. That is now ruled out as the primary cause:
|
||||
- **User runs the soak OUTSIDE the container** (real host, real fsync-able fs, NO overlay `fdatasync` EIO) and still observes the DATA LOSS + the liveness failures. So they are genuine, not sandbox `fdatasync` noise.
|
||||
- In-sandbox confirmation on a **real ext2/3 deploy dir** (`fdatasync` succeeds, ZERO EIO): a 3-round run PASSED clean, but an 8-round run reproduced BOTH a liveness cascade (after killing `ctrl-1` in round 1 the cluster never returned to 3-healthy) AND a cold-restart `expect-jobs` shortfall with a PERSISTENTLY-absent committed job — with **zero `fdatasync` EIO**. (Caveat: the 8-round run's cold-restart followed a degraded run, so the cleanest real-fs evidence is the user's outside-container host.)
|
||||
|
||||
**Verdict for 6.1:** the durability loss and the liveness stall reproduce off overlay → REAL controller findings, not an `fdatasync`-EIO artifact. The overlay EIO remains a SEPARATE, sandbox-only confound (it makes the dev-box `controller_unit_test` flaky — see bug-760 — and inflates loss on overlay deploys), but it is not the root cause of the soak loss.
|
||||
|
||||
**Phase 6 is now at 6.2 (root-cause).** Two real, non-deterministic findings to root-cause:
|
||||
- **Durability:** an acked (`jobsubmit ok`) job is PERSISTENTLY absent from a SERVING cluster after failover / cold restart — i.e. bug-661 is NOT fully closed (Phase 5 only exercised rolling single-node kills with the racy soak). Buckets: acked-without-durable-majority (commit-gate timing) / recovery truncates committed entries / behind-then-killed node re-leads with a truncated log.
|
||||
- **Liveness:** after killing the current primary (consistently `ctrl-1` in these runs), the cluster sometimes takes >60s — or fails entirely — to elect a serving primary, and a restarted node sometimes never returns to a healthy 3-node serving set. Possibly coupled to the promotion-fence / freshness-floor interaction with a just-rejoined node.
|
||||
|
||||
The hardened `xce-test-e2e-primitives` soak (trustworthy: liveness vs durability separated, confirm-settle, quiet wait-primary) is the reproduction surface for both.
|
||||
File diff suppressed because one or more lines are too long
|
|
@ -1,176 +0,0 @@
|
|||
# Implementation Plan
|
||||
|
||||
> TDD is mandatory (project rule): write the failing test in `testsuite/` before the C change (red → green → refactor).
|
||||
> Re-verify HA changes against `raft/raft.tla` and the `src/controller/CLAUDE.md` Protocol Verification Checklist.
|
||||
> Most core tasks edit the same correctness-critical files (`persist.c`, `commit.c`, `ha.c`, `cluster.c`) and are therefore sequential by design; only genuinely file-disjoint analysis is marked `(P)`.
|
||||
|
||||
- [x] 1. Foundation: durability verification seam and deterministic reproduction
|
||||
- [x] 1.1 Add a configurable flush-cadence override
|
||||
- Parse a `--wal-sync-ms` controller flag (following the existing `--sync-timeout-ms` convention) into the persistence config before persistence starts; default behavior unchanged.
|
||||
- Observable: starting a controller with `--wal-sync-ms <N>` sets the flush cadence to N; omitting it preserves the current default.
|
||||
- _Requirements: 3.1_
|
||||
- _Boundary: DurabilityConfig_
|
||||
|
||||
- [x] 1.2 Add the test-only durability-verification hook
|
||||
- Add an `XCE_TESTING`-gated seam that lets a test assert an acknowledged operation is recoverable after an immediate ungraceful termination; compiled out of production builds.
|
||||
- Observable: in a test build the hook is callable and the production build links without it.
|
||||
- _Requirements: 3.2, 3.3_
|
||||
- _Boundary: DurabilityTestSeam_
|
||||
|
||||
- [x] 1.3 Drive a deterministic durability reproduction from the soak harness
|
||||
- Extend `test-commit-durability.sh` with a deterministic mode that widens the batching window / uses the seam and kills a durable contributor in-window.
|
||||
- Observable: on the current (unfixed) build the deterministic mode reports DATA LOSS (RED), proving bug-661 without relying on race timing.
|
||||
- _Requirements: 3.1, 3.2_
|
||||
- _Depends: 1.1, 1.2_
|
||||
|
||||
- [x] 2. Durable commit mechanism (acknowledgment ⇒ majority persistence)
|
||||
- [x] 2.1 Introduce durable-LSN tracking via a group-commit flusher
|
||||
- Repurpose the background fsync thread from a timer into a demand-driven flusher: an append signals it; it performs one `fdatasync` over all pending appends, advances a durable-LSN watermark, and wakes the commit evaluation.
|
||||
- Guard against lost wakeups: the signal is level-triggered on the appended-vs-synced gap so an entry is never considered durable before its `fdatasync` (gating item).
|
||||
- Observable: a unit test shows one `fdatasync` advances the durable-LSN past several appended LSNs under load, and the durable-LSN never exceeds the synced tip.
|
||||
- _Requirements: 1.1, 1.6_
|
||||
- _Boundary: DurableCommit_
|
||||
|
||||
- [x] 2.2 Count self toward the commit majority only when durable
|
||||
- Change the commit-majority evaluation so the local member's self-contribution is capped at the durable-LSN watermark, not the appended tip; clamp the watermark on WAL truncation.
|
||||
- Observable: a unit test shows `committed_lsn` does not advance over an entry until that entry is durable locally, and the self-cap equals the durable-LSN.
|
||||
- _Requirements: 1.2, 1.4_
|
||||
- _Boundary: DurableCommit_
|
||||
|
||||
- [x] 2.3 Pin standby persist-before-acknowledge with a regression test
|
||||
- Add a regression test asserting the standby apply path `fdatasync`s a replicated entry before sending its acknowledgment.
|
||||
- Observable: the test fails if the standby ACK is emitted before the durable write and passes against the current behavior.
|
||||
- _Requirements: 1.3_
|
||||
- _Boundary: DurableCommit_
|
||||
|
||||
- [x] 2.4 Reconcile the dormant synchronous-append commit path
|
||||
- The synchronous-append path advances the commit position directly on majority acknowledgment and is built on the premise that the append already fsync'd inline — a premise group-commit removes. Confirm it has no production caller and delete it, or route it through the same durable-majority evaluation so it can never advance the commit position on a non-durable entry.
|
||||
- Observable: a code search shows no path advances the commit position on majority ACK without the durable precondition; the dormant path is removed or re-routed, and a test guards against a future caller reintroducing an ACK-without-durability commit.
|
||||
- _Requirements: 1.1, 1.2_
|
||||
- _Boundary: DurableCommit_
|
||||
- _Depends: 2.1_
|
||||
|
||||
- [x] 3. Recovery integrity (no silent truncate; durable floor)
|
||||
- [x] 3.1 Return a recovery integrity verdict and mark a truncated member non-promotable
|
||||
- Make WAL replay distinguish a clean log end from a corruption-stopped end (CRC chain break / mismatch / short read) and surface that verdict; a corruption-truncated recovery is never reported as a clean recovery and sets a member-level "recovery-truncated" flag that the promotion gate (4.3) consumes.
|
||||
- Observable: a unit test injecting a CRC break at the first entry yields a TRUNCATED verdict and sets the non-promotable flag, while an intact log yields CLEAN; an empty first-boot WAL is CLEAN (not corruption).
|
||||
- _Requirements: 2.1_
|
||||
- _Boundary: RecoveryIntegrity_
|
||||
|
||||
- [x] 3.2 Pin the fail-loud vs fence boot policy as a decision with a verification table
|
||||
- Pin the concrete policy (gating item, decision only — no behavior change yet): a truncated member fences from leadership at boot, attempts catch-up from a healthy peer, and terminates loudly only on a bounded failure to reach a committed-consistent state — never a silent indefinite fence. Encode the decision as a table of input states → expected outcome.
|
||||
- Observable: the policy and its decision table are recorded, and skeleton tests assert each row's expected outcome (initially failing), establishing the contract 3.3 and 4.3 implement.
|
||||
- _Requirements: 2.2, 2.4_
|
||||
- _Boundary: RecoveryIntegrity_
|
||||
|
||||
- [x] 3.3 Implement loud FATAL on bounded-unrecoverable recovery
|
||||
- Implement the FATAL branch of the 3.2 policy: when a recovery-truncated member cannot reach a committed-consistent state within the bounded catch-up window, terminate loudly with the truncation boundary logged; a reachable healthy peer never triggers FATAL.
|
||||
- Observable: the 3.2 decision-table rows for the FATAL/no-FATAL cases pass; logs show the truncation boundary on FATAL.
|
||||
- _Requirements: 2.4_
|
||||
- _Depends: 3.1, 3.2_
|
||||
- _Boundary: RecoveryIntegrity_
|
||||
|
||||
- [x] 3.4 Re-derive the durable floor from the recovered WAL tip
|
||||
- On recovery, publish the member's durable floor as the highest intact replayed LSN so later advertisement is backed by durable state rather than a volatile in-memory position.
|
||||
- Precision: this floor is the durable **last-log index** (WAL tip), used by the Raft freshness gate — not the commit position; the post-election safe-commit point continues to come from voter replies, not from this floor.
|
||||
- Observable: a unit test shows that after a truncating restart the published durable floor equals the highest intact LSN (the WAL tip), not a stale higher in-memory value or the commit position.
|
||||
- _Requirements: 2.3_
|
||||
- _Boundary: RecoveryIntegrity_
|
||||
|
||||
- [x] 4. Promotion fence and durable-backed freshness
|
||||
- [x] 4.1 (P) Pin the freshness-floor formula and re-verify election safety and liveness
|
||||
- Define the exact reconciliation of the durable-derived freshness floor against the existing in-memory floor — the durable quantity is the durable **last-log index** (WAL tip), not the commit position — then re-check it against `raft/raft.tla` for Leader Completeness AND liveness (no election livelock) plus the Protocol Verification Checklist (gating item; analysis only, no shared source files).
|
||||
- Scope of the raft.tla check: it validates the **election/freshness logic** (the `mlastLogIndex`/`LastTerm` semantics this change touches), which raft.tla models directly. It does NOT validate the *durability* of that floor — raft.tla assumes a durable log (see 5.3). The durability of the WAL-tip derivation is verified by tests (3.4), not by raft.tla.
|
||||
- Observable: the pinned formula is recorded and the model/checklist verification passes for both a truncated-candidate-denied case and a longest-durable-log-still-elects case.
|
||||
- _Requirements: 2.3_
|
||||
- _Boundary: FreshnessFloor_
|
||||
|
||||
- [x] 4.2 Derive the vote and PreVote freshness floor from durable WAL state
|
||||
- Implement the pinned formula so a candidate cannot advertise, and a voter cannot grant against, a position beyond durable WAL state; the post-election safe-commit point continues to come from voter replies, never a stale ctrlhello.
|
||||
- Observable: a node restarted with a truncated log is denied (vote and PreVote), while the node holding the longest durable log still wins — no livelock.
|
||||
- Note: 4.2 and 4.3 both edit the same election code (`cluster.c` vote/PreVote and `ha.c` promotion); they are sequential (4.2 before 4.3), not parallel.
|
||||
- _Requirements: 2.3_
|
||||
- _Depends: 4.1, 3.4_
|
||||
|
||||
- [x] 4.3 Add the pre-promotion eligibility gate and fence release
|
||||
- Before a member becomes primary, refuse promotion if it carries the recovery-truncated flag (from 3.1) or its durable position is below the cluster committed high-water mark; extend the existing catch-up fence to cover the promoting/recovered member and release it when catch-up passes the last committed boundary. Reuse the durable-LSN truncate-clamp from 2.2 so the gate evaluates a clamped durable position at the promotion-path truncate site.
|
||||
- Observable: a recovery-truncated or behind member is denied promotion until catch-up completes, then regains eligibility; a hollow node cannot win even at minimal quorum.
|
||||
- _Requirements: 2.2, 2.5, 2.6_
|
||||
- _Depends: 4.2, 3.1, 3.2, 2.2_
|
||||
|
||||
- [x] 4.4 Bounded-window catch-up before FATAL (fence does not FATAL immediately)
|
||||
- The promotion gate (4.3) currently FATALs a recovery-truncated/behind member immediately at the promotion instant when no granting voter has a live endpoint AT THAT INSTANT — a transient no-peer snapshot during aggressive failover. Per the 3.2 decision table (rows 3/5) a member with a (momentarily) reachable peer must FENCE→catch-up→ELIGIBLE, NEVER immediate-FATAL; FATAL is reached ONLY after a BOUNDED failure to reach a committed-consistent state. Implement the deferred bounded-window catch-up: on FENCE (or would-be-FATAL no-peer), keep the member alive in `in_catch_up`, schedule a one-shot delaymgr timer for `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS`, and reach the 3.3 FATAL ONLY from that timer if the member is STILL recovery-truncated AND durable_floor < cluster HWM AND no healthy peer / no catch-up progress when it fires. Re-derive eligibility (release the fence) when catch-up brings durable_floor ≥ HWM. Never FATAL synchronously in `raft_cb_become_leader`; never fence indefinitely.
|
||||
- Observable: a recovery-truncated node whose peers are transiently unreachable FENCES (stays alive) and is NOT FATAL'd; once a healthy peer is reachable and catch-up passes the committed boundary it regains eligibility; FATAL fires only after the bounded window with no peer/progress (unit-tested via the `persist_fatal_hook` + a short window). The rolling SIGKILL soak (task 5.2) keeps the cluster available (always re-elects) with zero acknowledged-job loss.
|
||||
- _Requirements: 2.2, 2.4, 2.6_
|
||||
- _Depends: 4.3, 3.3_
|
||||
- _Boundary: PromotionFence_
|
||||
|
||||
- [x] 5. Integration and validation
|
||||
- [x] 5.1 Wire durable-majority commit end to end
|
||||
- Connect the durable-LSN self-count and the commit barrier so the client reply is released only after the entry is durable on a majority; assert liveness (the flusher does not stall heartbeat/election handling).
|
||||
- Observable: an integration test confirms reply-ok only after durable-majority, killing the sole durable contributor mid-commit leaves the acknowledged op present on the survivors, and the full controller suite passes under ASAN.
|
||||
- _Requirements: 1.1, 1.4, 1.5, 1.6, 2.5_
|
||||
- _Depends: 2.2, 4.2_
|
||||
|
||||
- [x] 5.2 Prove zero acknowledged-job loss across failover
|
||||
- Run the deterministic durability E2E and the rolling SIGKILL+restart soak; both must show every acknowledged job surviving every failover.
|
||||
- Observable: the deterministic mode that was RED in 1.3 now passes (GREEN), and the soak reports zero DATA LOSS across rounds.
|
||||
- _Requirements: 1.1, 1.5, 2.2, 2.3, 2.6_
|
||||
- _Depends: 5.1, 4.3, 1.3_
|
||||
|
||||
- [x] 5.3 Re-verify the formal safety checklist and flip the durability gap note
|
||||
- Re-run the `raft/raft.tla` alignment and the Protocol Verification Checklist (commit rule, leader completeness, `commitIndex` monotonic) against the integrated implementation.
|
||||
- IMPORTANT — raft.tla does NOT model durability: it restores `log` intact from "stable storage" on restart (`raft.tla:165-174`), i.e. it assumes what bug-661 violates. A passing TLC/alignment run on the unmodified spec therefore does NOT prove the commit-durability invariant. That invariant is verified by the unit/integration/E2E tests (2.x, 5.1, 5.2) plus the manual checklist — not by raft.tla. Do not conclude durability is proven from raft.tla alone. Formal durability proof would require extending raft.tla with a durable-vs-volatile log model (out of scope unless explicitly opted in).
|
||||
- Update `src/controller/CLAUDE.md`: convert the "Commit Durability — KNOWN GAP (bug-661)" note into the now-true guarantee and tick the post-`ha-commit-durability` checklist item.
|
||||
- Observable: the checklist passes with the durable self-count and the durable-backed freshness floor in place; the durability invariant is backed by passing tests (not by raft.tla); the CLAUDE.md gap note is removed; any deviation is recorded.
|
||||
- _Requirements: 1.2, 2.3_
|
||||
- _Depends: 5.1_
|
||||
|
||||
- [x] 6. REOPENED 2026-06-19 — total-cluster-outage (cold-restart-all) committed-job loss (bug-795) <!-- CLOSED 2026-07-02: 6.1 found the loss does NOT reproduce on a real fs — overlayfs fdatasync artifact confirmed. 6.2/6.3 not applicable (conditional on 6.1=REAL, which it wasn't). See Implementation Notes. -->
|
||||
> Surfaced by the `xce-test-e2e-primitives` soak's new cold-restart-all step (kill ALL 3 + restart ALL). Phase 5 only validated rolling SINGLE-node failover; the total simultaneous outage path was never tested. Sequential by design (each task gates the next). DO NOT start a fix before 6.1 resolves real-vs-artifact.
|
||||
- [x] 6.1 Reproduce cold-restart-all loss on a REAL fsync-able fs (rule out the overlayfs artifact)
|
||||
- Run the converted `test-commit-durability.sh` (incl. its `run_cold_restart_recovery` phase) against a live 3-node cluster on a REAL fsync-able filesystem (NOT container overlay `/tmp`) — e.g. a tmpfs-free ext4/xfs deploy dir. Repeat enough rounds to characterize the race (it is non-deterministic: ~1-in-2 in sandbox runs). Capture controller recovery logs (WAL `truncate`, durable-floor, `committed_lsn`) for a losing run.
|
||||
- Observable: a definitive verdict — EITHER the loss reproduces on a real fs (⇒ genuine total-outage durability gap, proceed to 6.2) OR it does NOT reproduce on a real fs and only occurs on overlay `fdatasync` (⇒ overlayfs artifact; record as an environment limitation, close Phase 6 with a note, and the `xce-test-e2e-primitives` task 7.2 unblocks on a real-fs run).
|
||||
- _Requirements: 1.1, 1.5_
|
||||
- _Boundary: ValidationOnly (no production change) — `src/tools/test-commit-durability.sh` is owned by xce-test-e2e-primitives; consume it as-is._
|
||||
- [x] 6.2 Root-cause the total-outage recovery loss (conditional on 6.1 = REAL) <!-- N/A — 6.1 found no loss on real fs -->
|
||||
- Instrument the recovery path for a simultaneous all-node SIGKILL + restart: determine whether (a) a job returned `jobsubmit ok` without being `fdatasync`'d on a fault-tolerant majority (a Req-1 commit-gate hole specific to the total-outage timing), (b) recovery (`replay_wal` / `persist_truncate_wal_above` / durable-floor re-derivation, tasks 3.1/3.4) truncates committed entries on cold restart, or (c) a just-rejoined-then-immediately-cold-killed BEHIND node re-leads with a truncated log and overwrites the good timeline (a promotion-fence/freshness-floor hole under cold restart, tasks 4.2/4.3).
|
||||
- Observable: the exact mechanism is identified with a captured trace; classified as commit-gate / recovery-truncate / promotion-fence; mapped to the specific Requirement(s) and code path it violates.
|
||||
- _Requirements: 1.1, 1.4, 1.5, 2.3, 2.6_
|
||||
- _Boundary: RecoveryIntegrity / DurableCommit (controller persist/commit/ha)_
|
||||
- _Depends: 6.1_
|
||||
- [x] 6.3 Fix the total-outage durability/recovery path + add a deterministic regression (conditional on 6.2) <!-- N/A — 6.2 not applicable -->
|
||||
- Repair the identified mechanism so an acknowledged job survives a simultaneous all-node kill+restart, honoring the never-self-terminate invariant (loud fence, stay alive, keep retrying — see `src/controller/CLAUDE.md` Process Lifecycle). Add a deterministic cold-restart-all regression (prefer a unit/integration harness over the racy soak; the soak's `run_cold_restart_recovery` step stays as the E2E backstop).
|
||||
- Observable: the cold-restart-all loss no longer reproduces (real fs); the new regression is RED on the unfixed build and GREEN after; the full controller suite passes under ASAN with `:FAIL:`==0.
|
||||
- _Requirements: 1.1, 1.4, 1.5, 2.3, 2.6_
|
||||
- _Boundary: DurableCommit / RecoveryIntegrity / PromotionFence_
|
||||
- _Depends: 6.2_
|
||||
- [x] 6.4 Re-validate and reconcile the durability claim
|
||||
- Re-run the deterministic regression + the cold-restart-all soak on a real fs (GREEN); reconcile `src/controller/CLAUDE.md` "Commit Durability — GUARANTEED" to state the total-outage scenario is now covered (or, if 6.1 found an artifact, qualify the overlay-fs caveat). Unblock `xce-test-e2e-primitives` task 7.2 and re-run its soak to close it.
|
||||
- Observable: durability guarantee holds for BOTH rolling single-node failover AND total-cluster-outage cold restart, backed by the new regression; CLAUDE.md is accurate; bug-795 closed.
|
||||
- _Requirements: 1.1, 1.2_
|
||||
- _Boundary: DurableCommit + docs_
|
||||
- _Depends: 6.3_
|
||||
|
||||
## Implementation Notes
|
||||
- 1.1: Config plumbing routes through `myconf_t.wal_sync_ms` + `persist_config_set_wal_sync_ms()` (0-sentinel = keep `DEFAULT_WAL_SYNC_MS`), NOT a new config.h constant (design File Structure Plan keeps `DEFAULT_WAL_SYNC_MS`). The override is applied at BOTH `persist_init` sites — `myapp_run` step-6 (production path) and `cluster_init_phase2` (guarded by `wal_path[0]=='\0'`, skipped in production). They are mutually exclusive; apply any new persist_config override at both sites or it is silently dropped. Pre-existing latent bug (do NOT fix here): `--sync-timeout-ms` is applied only in phase2, so it is ignored on the production startup path — relevant if a later task relies on `sync_timeout_ms`.
|
||||
- 1.2: Test seam = `persist_test_durable_lsn_on_disk(persist_t*)` (decl `persist.h`, impl `persist.c`, all `#ifdef XCE_TESTING`) — re-opens WAL segment files from disk and CRC-validates the chain (read-only, state-free) to return the highest fully-persisted LSN; new test `testsuite/test_commit_durability.c` (registered in `meson.build` + `controller_test.c`). Production `xce_controller` links without it (proven via `nm`). DRY NOTE: ~115 lines of segment-scan/CRC-chain logic are duplicated from the static `replay_wal`; tasks 3.1 (recovery verdict) and 3.4 (durable floor from WAL tip) touch the SAME scan/CRC logic — consider extracting a shared read-only `wal_scan_durable_lsn`-style helper there and routing this seam through it (avoid a third copy). `XCE_TESTING` is defined only for `controller_unit_test` (`src/controller/meson.build:167`).
|
||||
- 1.3: Deterministic mode = `bash src/tools/test-commit-durability.sh --deterministic` proves the durability GAP via strace, NOT joblist data-loss. SCOPE CLARIFICATION (review-ratified): single-host SIGKILL committed-job loss is physically NON-deterministic — `persist_wal_async` pwrites inline (page cache survives process death) and standbys fdatasync-before-WalAck (persist.c:515), so a committed job is always durable off the primary; a data-loss run flips GREEN/RED/RED. The deterministic slice: with `--wal-sync-ms 30000` the primary acks "jobsubmit ok" with ZERO primary fdatasync in the submit→ack window (exec-wrap `strace -f -tt -e trace=fdatasync,fsync`; PTRACE_ATTACH `-p` is yama-blocked). RED = in-window primary syncs==0 (15/15 across 5 runs; primary=0 vs standby=2). This is a valid RED→GREEN gate: the task-2.x fsync-before-ack fix makes the primary count ≥1 ⇒ GREEN. Original racy soak retained as the default mode. Boundary: only `src/tools/test-commit-durability.sh` (flag plumbed via deploy-ha.sh start.sh `$*` passthrough — no C edits). For task 5.2's E2E, prefer this strace gap-proof over joblist-loss for a deterministic durability assertion.
|
||||
- 2.1: `persist_t.durable_lsn` (`_Atomic uint64_t`, accessor `persist_durable_lsn()`) = prefix watermark advanced ONLY after a successful `fdatasync`. `fsync_worker` is now a demand-driven group-commit flusher: `persist_wal_async` pwrites under `p->lock`, releases, then signals `fsync_cond`. Lost-wakeup guard is LEVEL-triggered on `flusher_work_pending_locked` (`durable_lsn < current_lsn-1`), re-checked under `p->lock` before wait and after each flush (reviewer-confirmed sound). Invariants: monotonic, `durable_lsn <= current_lsn-1`, clamped on truncate (`clamp_durable_lsn_locked` at both truncate paths). fail-loud rc-gate: `durable_lsn` does NOT advance on `fdatasync` error (all 4 sites: flusher main/fallback, `persist_stop`, `persist_flush`) — logs ERROR. Test fault-injection seam: `#ifdef XCE_TESTING` toggle `persist_test_set_force_fsync_fail()` (production-excluded, nm-verified). NOT yet consumed by commit (that is 2.2). FOLLOW-UP (non-blocking, out of scope for 2.1): bound the flusher retry on SUSTAINED fdatasync failure (backoff / rate-limited logging) — currently tight-spins (releases `p->lock` each iter, so liveness OK, but CPU/log burn on a degrading disk). NOTE: a HANGING (not erroring) fdatasync blocks appends since it runs under `p->lock` — pre-existing, not 2.1's boundary.
|
||||
- 2.2: `majority_candidate_locked` (commit.c:93) self-cap = `persist_durable_lsn(ce->p)` (was `highest_appended_lsn`); appended tip kept only for the `assert(cand <= highest_appended_lsn)` never-beyond-log check. So `committed_lsn` cannot advance over an entry until it is durable locally (Req 1.2/1.4). Durable-advance WAKE (the wire 2.1 deferred): persist defines a callback hook `on_durable_advance(void*)`/`durable_cb_ctx` + `persist_set_durable_cb()`; commit registers `commit_on_durable_advance` (ctx=engine) in `commit_engine_init`, unregisters in `commit_engine_fini`. Flusher invokes it ONLY when `durable_lsn` advanced, AFTER releasing `p->lock` → callback takes `ce->lock` + broadcasts `ce->cond`. Respects commit→persist dependency direction (persist never includes commit). Lock order p->lock→ce->lock only (cycle-free). Lost-wakeup-free: broadcast follows the release-store; waiter re-reads durable under ce->lock; timedwait backstop. Shutdown UAF closed: `persist_stop` JOINS the flusher BEFORE `commit_engine_fini` (cluster.c:1675 vs 1695); final stop-flush does NOT fire the callback; fini also unregisters defensively. GOTCHA (bug-704): self-cap broke 4 existing commit-path fixtures (test_repl_majority/coldstart_lsn0/commit_barrier_rollout/demotion_convergence) that primed `current_lsn` but not `durable_lsn` — fix = prime `durable_lsn = appended tip` in any in-memory fixture that sets current_lsn without writing real WAL bytes (the flusher can't advance durable for a fake log). Any future commit-engine test must prime durable_lsn too.
|
||||
- 2.3: Standby persist-before-ACK pinned + Req-1.3 hole closed. `persist_wal_write_replicated` (persist.c:741) now rc-gates the inline `fdatasync`: on failure (or `flusher_fsync_forced_fail`) it logs ERROR + `return -EIO` (symmetric with the 2.1 flusher gate) — the standby will NOT ACK a non-durable entry. Caller `ha_apply_single_entry` (ha.c:1135-1166) ALREADY routes a negative return to no-apply/no-ACK (`ha_send_wal_ack_reply` reached only on rc==0), so NO ha.c change was needed (boundary stayed persistence-only). Ordering pin = `#ifdef XCE_TESTING atomic_uint test_replicated_fdatasync_count` incremented at the sync site; `test_replicated_persist_syncs_before_return` asserts it increments inside the call (page-cache means a disk re-read can't prove fsync — use the counter). `test_replicated_sync_failure_no_ack` REDs without the gate. Seam production-excluded (nm-verified). HOUSEKEEPING: controller tests `mkdtemp` into `/tmp/xce_*` and don't clean up — thousands accumulate and eventually make `mkdtemp` fail → misleading suite-wide "Expected Non-NULL" cascade. Run `rm -rf /tmp/xce_*` if that appears (pre-existing test pattern, not a regression).
|
||||
- 2.4: SPEC INACCURACY corrected — `persist_wal_sync_append` is NOT dormant; it is the LIVE job-submit path (`persist_job_hook_fun` calls it at persist.c:1911). Took the "route through durable-majority eval" option (delete infeasible). Production path (ce!=NULL, lsn!=0) was already durable-safe via 2.2's `majority_candidate_locked` self-cap. Closed the hazard in the LEGACY branch: the former unconditional `atomic_store(committed_lsn, written_lsn)` on majority ACK is now `sync_append_legacy_commit_store()` which advances ONLY when `written_lsn <= persist_durable_lsn(p)` (LSN-0 edge: store of 0 valid). Added the design's debug invariant `committed_lsn <= durable_lsn` (`#ifdef XCE_TESTING` assert at commit.c:162 + inside the legacy store) as the future-proof guard. Fixed the stale "after local fsync via persist_wal_async" comment (2.1 deferred the fsync). CODE-SEARCH (authoritative): the ONLY `committed_lsn` stores are — engine advance (commit.c, durable-capped by 2.2), legacy sync_append (now gated), and the two `persist_truncate_wal_above` cutoff stores (NOT ACK-driven; each immediately `clamp_durable_lsn_locked`). ha.c/ha_convergence.c stores are `last_known_committed_lsn` (DIFFERENT field). So no path advances commit on ACK without the durable precondition. Seam `persist_test_sync_append_legacy_commit_store` is production-excluded.
|
||||
- 3.1: `recover_verdict_t {RECOVER_CLEAN, RECOVER_TRUNCATED}` (persist.h) + `persist_t.recovery_verdict` field + accessors `persist_recovery_verdict()`/`persist_recovery_truncated()` (NOT XCE_TESTING-gated — 4.3 consumes them in production). `replay_wal` classifies: CRC chain break / CRC mismatch / short header/payload read → TRUNCATED; clean entry boundary / pre-allocated-zero tail / empty WAL / snapshot-sentinel+clean-replay → CLEAN. Verdict is monotonic (init CLEAN once; only ever set TRUNCATED; a later clean segment can't un-set it). Verdict = STRUCTURAL integrity, NOT "behind the cluster" (the behind-high-water case is 4.2/4.3). 3.1 ONLY sets state + WARNs — does NOT FATAL (3.3) or fence (4.3). DRY: did NOT extract a shared scanner (replay_wal mutates persist_t; the 1.2 seam is read-only) — no third copy added. ENV FINDING (see cerebrum 2026-06-18): this sandbox's /tmp can be an overlay that EIOs on fdatasync; because committed_lsn is now durable-gated (2.x), an fdatasync EIO correctly HOLDS committed_lsn, which makes `test_fact_hook_append_is_retained_and_nonblocking` (commit_barrier_rollout) FAIL on overlay /tmp — confirmed pre-existing (fails identically on the clean pre-spec baseline), NOT a regression. Final validation (5.x) must run on a real fsync-able fs.
|
||||
- 3.2: DECISION-ONLY (no production change). Boot Policy Decision Table recorded in design.md §"Boot Policy Decision Table (task 3.2)" — 7 rows over {verdict CLEAN/TRUNCATED × at-above/below HWM × peer reachable × bounded-catch-up succeeds/fails} + snapshot-sentinel + empty-first-boot. Outcomes: ELIGIBLE / FENCE+CATCH-UP→ELIGIBLE / bounded FATAL. AUTHORITATIVE INVARIANT: never a silent indefinite fence — every truncated/behind member resolves to exactly one terminal (catch-up→ELIGIBLE, or BOUNDED FATAL). Snapshot-sentinel is NEVER FATAL on a legitimate sentinel. 7 skeleton tests in test_commit_durability.c (row1,2,3,4,4b,5,6) are `TEST_IGNORE_MESSAGE`-skipped (PENDING 3.3/4.3) so the suite stays GREEN — they are the RED contract 3.3 (FATAL rows) and 4.3 (fence/eligible rows) turn green. NOTE: pre-existing ASAN leaks in unrelated tests (test_demotion_convergence, ha_start_election_async, ha_rdhup_integration) make the STANDALONE controller_unit_test exit=1, but meson reports Fail:0 — don't be fooled by standalone exit code.
|
||||
- 3.3: FATAL branch = pure decision `persist_recover_decision(recover_policy_inputs_t) → recover_decision_t {ELIGIBLE, FENCE, FATAL}` (persist.c) + action `persist_recover_fatal_if_unrecoverable()` which logs the truncation boundary (`durable_floor_lsn`/`cluster_hwm_lsn` + window ms) at ERROR then terminates via `persist_fatal_hook` (extern `void(*)(const char*)`, NULL in prod → `tlog_exit(); abort()`; mirrors `ha_fatal_hook`; tests install it to capture instead of abort). FATAL ONLY for bounded-unrecoverable + NO healthy peer (rows 4/4b); a reachable peer NEVER FATALs (gated behind `if (!healthy_peer_reachable)`); eligibility rows 1/2/6 never FATAL. Bound = FINITE `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS=30000` (persist.h) — keys off the `catch_up_succeeded` bounded-window result, never an indefinite wait. Turned 3.2 skeleton rows 4/4b GREEN + added `test_boot_policy_peer_reachable_never_fatal`; rows 1/2/3/5/6 stay IGNORE (FENCE→ELIGIBLE is 4.3). DEFERRED to 4.3/5.x: wiring this decision into the live boot/promotion path (3.3 is the unit-tested decision + terminate only; no ha.c/cluster.c edits).
|
||||
- 3.4: On recovery, `persist_recover` publishes the durable floor = highest intact replayed LSN (recovered WAL tip = `current_lsn-1` after `replay_wal`) into `durable_lsn` via `atomic_store` (ASSIGN, NOT max — so a truncating restart lands at the LOWER intact tip, never over-advertising; this is the crux of Req 2.3). Added `persist_durable_floor()` (NULL-safe thin accessor) as the seam task 4.2 consumes. Floor is the durable LAST-LOG INDEX (WAL tip), NOT the commit position (`committed_lsn`) and NOT a volatile in-memory value. Invariant-safe (recovery/init-time; recovered entries are CRC-validated on-disk = genuinely durable; preserves `durable_lsn <= current_lsn-1`). 4.2 (ha.c/cluster.c) CONSUMES this floor in `ha_vote_self_freshness` — 3.4 only PUBLISHES (persistence-only). Phase 3 (recovery integrity) COMPLETE.
|
||||
- 4.1: ANALYSIS-ONLY (no .c/.h). Pinned freshness-floor formula in design.md §"Freshness-Floor Formula (task 4.1)" — THE CONTRACT 4.2 implements. CANDIDATE advertises BARE `cand_lsn = persist_durable_floor(p)` (durable WAL tip) — sites ha.c:3660-3666 (real-vote) + ha.c:3735-3741 (PreVote) must call `persist_durable_floor` DIRECTLY, BYPASSING `ha_vote_self_freshness` on the LSN axis (that fn unconditionally folds in `last_known_committed_lsn` → `max(durable_floor, learned)` = the truncated-candidate-wins bug). VOTER floor = `max(durable_floor, last_known_committed_lsn)` — sites cluster.c:388-395/471-475 + ha.c:2243 keep `ha_vote_self_freshness` with `local_lsn=durable_floor` (the learned_committed backstop stays on the VOTER axis only). Grant iff `cand_tag > floor_tag OR (==tag AND cand_lsn >= floor_lsn)` (term-then-index, matches raft.tla `LastTerm`/`mlastLogIndex |-> Len(log)`). `cand_tag`/`floor_tag` = `max(local_tag, learned_tag)`. SAFETY (Leader Completeness): quorum intersection Q_L∩Q_v ⇒ truncated candidate (durable tip < committed L) denied. LIVENESS (no bug-339 livelock): rests on D* (max durable tip in reachable majority); `durable_floor(v) ≤ D*` and `learned_committed(v) ≤ L ≤ D*`, so the most-durable candidate is always grantable. `safe_committed_lsn` STAYS voter-reply-sourced (unchanged). CAVEAT: raft.tla validates election/freshness LOGIC only, NOT durability (no TLC run — no toolchain; conceptual alignment) — durability verified by 2.x/3.4 tests. (1 remediation: candidate-vs-voter contradiction + §5b example fixed.)
|
||||
- 4.2: Implements the 4.1 formula. New `ha_candidate_advertise_lsn(persist*)` = bare `persist_durable_floor`. CANDIDATE sites advertise bare durable_floor for the LSN (tag still via `ha_vote_self_freshness(...,&tag,NULL)`): ha.c:~3690 (real election), ha.c:~3771 (PreVote), AND myapp.c:~234 (boot election — a THIRD candidate site the brief didn't name; fixed too, else a boot-time truncated candidate over-advertises). VOTER floor: cluster.c:~383 (real) + cluster.c:~471 (PreVote) feed `local_lsn = persist_durable_floor(&c->persist)` into `ha_vote_self_freshness` → floor = `max(durable_floor, learned_committed)`. Comparison unchanged (term-then-index). `ha.c:2243` legacy `handle_request_vote` is XCE_TESTING-only/param-driven — left unchanged. `safe_committed_lsn` UNTOUCHED (voter-reply-sourced). 3 tests in test_ha_vote_gate.c prove: candidate advertises bare durable_floor (not stale learned); voter durable-floor denies truncated; longest-durable grantable / no livelock. bug-339 regression test fixture primed `durable_lsn=10` (model-preserving). INFO: truncated-deny proven at the freshness-comparison-expression level shared by both vote+PreVote paths (no dedicated live PreVote-handler truncation driver — adequate; nice-to-have for 5.x). GOTCHA for future election tests: the freshness floor now reads `persist_durable_floor`, so any vote/election fixture that sets `applied_lsn`/`last_known_committed_lsn` but not durable state will see floor/advertise=0 — prime `durable_lsn` to the modeled tip.
|
||||
- 4.3: Pre-promotion eligibility gate in `raft_cb_become_leader` (ha.c, post-truncate / pre-CTRL_PRIMARY, returns on refuse). `ha_promotion_gate_decision` (pure, delegates to 3.3's `persist_recover_decision` — table is single source of truth) + `ha_evaluate_promotion_fence` (live: ELIGIBLE clears `in_catch_up`, FENCE sets it, FATAL → 3.3 `persist_recover_fatal_if_unrecoverable`). REFUSE if `persist_recovery_truncated` OR `persist_durable_floor < promotion_safe_lsn` (the existing voter-reply-derived `safe_committed_lsn`, read AFTER `persist_truncate_wal_above` so it reflects 2.2's clamp). `peer_reachable` = some granting voter has a live endpoint (`ectx->votes[i].vote_granted && .ep != NULL`) — so a multi-node election winner ALWAYS has a peer ⇒ FENCE (never premature FATAL, honoring rows 3/5); FATAL only fires no-peer (single-node corrupt = genuinely unrecoverable, row 4). Reuses `in_catch_up` (NOT a parallel fence); release on a later promotion attempt once caught up (durable_floor ≥ HWM, not truncated) → ELIGIBLE clears the fence (Req 2.6, re-evaluation model). `safe_committed_lsn` arithmetic UNTOUCHED. Turned 3.2 rows 1/2/3/5/6 GREEN (0 ignored boot-policy rows remain) + minimal-quorum test (Req 2.5). DEFERRED to 5.1: the live bounded-window catch-up TIMER (grant `DEFAULT_RECOVERY_CATCHUP_WINDOW_MS` to find a peer before FATAL) — here `catch_up_succeeded=false` at the gate, so it FENCEs (peer) or FATALs (no peer) immediately; fail-loud-conservative, never silently leads hollow. NOTE: reviewed in MAIN CONTEXT (review subagent hit a spend limit) — suite green ×1 (Fail:0, all 9 boot-policy/gate tests PASS, test_fact_hook_append passed too since this shell's /tmp is fsync-able). Phase 4 COMPLETE.
|
||||
- 5.1: INTEGRATION verification — the end-to-end durable-majority path was ALREADY wired by 2.1/2.2/2.4/4.x (no production gap found); 5.1 adds 3 integration tests + a tiny `#ifdef XCE_TESTING` liveness-observability seam (`persist_test_last_fdatasync_thread`, nm-excluded from prod). `test_durable_majority_commit_end_to_end`: holds durability off (forced fsync-fail), delivers BOTH standby ACKs, asserts committed_lsn/WAIT do NOT release (ACK-majority alone insufficient) — then flush makes self durable ⇒ committed advances + WAIT releases rc=0; asserts `committed_lsn <= durable_lsn`. RED proof: injecting the pre-2.2 `highest_appended_lsn` cap trips `assert(cand <= persist_durable_lsn)`. `test_sole_durable_contributor_loss_survives` (Req 1.5/2.5): property-asserted (committed ⇒ ≥⌊N/2⌋+1 durable copies, two persist dirs, drop one) — LIVE failover proof is 5.2. `test_durability_work_off_iomgr_preserves_liveness` (Req 1.6): seam records the flusher thread id; asserts fdatasync runs off the caller/iomgr thread + heartbeat reads stay <50ms during flushes. GOTCHA: a real persist with registered standbys needs the `XCE_TESTING repl_broadcast_send_hook` (drop) installed before append, else dummy endpoints trip an ASAN overflow in `dataq_enq`. CAVEATS (for 5.3/validation): (1) meson `Fail:0` masks Unity sub-group failures — gate on `grep -c :FAIL: testlog.txt`==0; (2) run on a fsync-able fs (the pre-existing test_fact_hook_append EIO failure is /tmp-overlay-only). 4.3 catch-up-timer-before-FATAL still a follow-up. Reviewed in MAIN CONTEXT (subagent spend limit).
|
||||
- 4.4: Closed the 5.2-soak liveness gap. `raft_cb_become_leader` NO LONGER FATALs synchronously — on FENCE / would-be-FATAL (truncated or durable_floor < HWM) it calls `ha_arm_promotion_fence_timer` (sets `in_catch_up`, schedules a one-shot `delay_task_ms(&c->delaymgr, ha_promotion_fence_window_ms(), 1, promotion_fence_timeout_fn, ctx)`) and RETURNS without becoming primary. The timer callback (`promotion_fence_timeout_fn`, ha.c:702) re-reads LIVE state after the window: caught-up (`!truncated && durable_floor>=HWM`) → release fence, NO FATAL (Req 2.6); peer reachable (`repl_active_standby_count>0`) OR durable_floor advanced past the arm-time baseline → progress possible, leave fenced (existing catch-up machinery / next promotion-gate ELIGIBLE releases it), NO FATAL; ONLY still-truncated/behind + no-peer + no-progress → 3.3 `persist_recover_fatal_if_unrecoverable` (Req 2.4). UAF-safe cancel (`ha_cancel_promotion_fence_timer`: cancelled flag + atomic-exchange the stored ctx; handle cleared before the cancelled-check, mirrors `raft_election_timeout_fn`); cancelled in `ha_fini`; fail-loud if `delay_task_ms` itself fails (never fence indefinitely). Test window seam `ha_set_promotion_fence_window_ms` + driver `ha_arm_promotion_fence_timer_for_test`. 3 tests (defers-fatal-to-timer, released-on-catchup-no-fatal, cancel-prevents-fatal) with RED proof (re-injecting sync FATAL → 3 fails). Reviewed MAIN CONTEXT (subagent spend limit): suite :FAIL:==0, /tmp fsync-able this run. NOTE: this is the bounded-window timer 4.3/5.1 deferred; with it, 5.2's soak should keep the cluster available (truncated rejoiners catch up, not die).
|
||||
- REWORK (2026-06-18, user invariant — a controller MUST NEVER self-terminate): tasks 3.3/4.3/4.4 had implemented Req 2.4 "fail loud" as a SELF-ABORT (`persist_recover_fatal_if_unrecoverable → tlog_exit(); abort()` + `persist_fatal_hook`). That violated the invariant (a self-exited controller needs MANUAL admin recovery + leaves quorum degraded; a live retrying node self-heals when a peer returns) and was the true root cause of the 5.2 soak quorum collapse. CORRECTED: removed the abort/exit terminal entirely. New: `persist_recover_loud_fence_if_unrecoverable(persist_t*)` (NO abort) logs a loud operator-visible ERROR (truncation boundary vs cluster HWM) + sets sticky `_Atomic bool recovery_unrecoverable` (accessors `persist_recovery_unrecoverable`/`persist_clear_recovery_unrecoverable`); the member STAYS ALIVE, permanently non-promotable/fenced (`in_catch_up` + the sticky flag both consulted by the promotion gate), and KEEPS RETRYING catch-up. Decision enum `RECOVER_DECISION_FATAL` → `RECOVER_DECISION_FENCE_UNRECOVERABLE` (non-terminating). 4.4 timer no longer aborts — it re-emits the loud operator alert + re-arms (operator visibility), never exits. Flag + fence CLEAR automatically on catch-up (not truncated AND durable_floor ≥ HWM) → re-eligible (Req 2.6), fully automatic. `persist_fatal_hook` removed. Verified: grep/nm show NO abort/exit/tlog_exit on the recovery path; ASAN suite :FAIL:==0; reworked tests (row4/4b → loud_fence, unrecoverable-stays-alive-loud, released-on-catchup, cancel-stops-realert) PASS. Spec updated: requirements.md Req 2.4 + design.md (decision table rows 4/4b, RecoveryIntegrity, traceability, FATAL-vs-fence open question) reworded "FATAL/terminate" → "LOUD PERMANENT FENCE (non-promotable, stays alive, keeps retrying — NEVER self-exit)". PROJECT-WIDE GUIDANCE added: `src/controller/CLAUDE.md` §"Process Lifecycle (DESIGN INVARIANT)" + `.kiro/steering/ha-async-io.md` — controllers never self-terminate. (`ha_self_restart` execv-restart in convergence/demotion is a SEPARATE primitive, untouched.)
|
||||
- 5.2: PASS on the reworked (no-self-exit) build. (a) DETERMINISTIC E2E flipped RED→GREEN, stable 3/3 runs / 9-9 probes — in-window primary fdatasync ≥1 (=2) precedes the ack ("durable-before-ack"), the inverse of 1.3's unfixed RED. (b) ROLLING SIGKILL soak (6 rounds) exit 0 = ZERO acknowledged-job loss, cluster stayed continuously available, 0 self-terminations (the no-self-exit rework removed the abort that collapsed quorum in the first attempt). Validation-only (no production code). NOTE: ran in-sandbox via background+poll on a worktree deploy dir (avoids /tmp overlay EIO); the earlier collapse was the abort terminal, now corrected — a host re-run is still good practice but the in-sandbox soak now passes with the fix.
|
||||
- 5.3: Formal re-verification (conceptual — no TLA+ toolchain in env; project practice = manual alignment + the CLAUDE.md Protocol Verification Checklist). Checklist walked against the integrated impl, all hold: votedFor/currentTerm (untouched); `commitIndex`/`committed_lsn` monotonic + never exceeds `durable_lsn` (2.2/2.4 debug-assert); leader commits via per-entry majority ack (unchanged); `safe_committed_lsn` from voter replies (untouched, 4.2/4.3); WAL truncation only above the safe point + `durable_lsn` clamped on truncate; freshness gate term-then-index with durable-backed LSN (4.1 pinned vs raft.tla `mlastLogIndex`/`LastTerm`; candidate bare `durable_floor`, voter floor `max(durable_floor, learned)`); contiguous-prefix commit; `durable_lsn` prefix watermark; causal-order append; and the NEW item — a member counts toward the commit majority only at its durable LSN, acknowledged⇒durable-on-majority — now TRUE. raft.tla alignment: the election/freshness logic maps faithfully; raft.tla does NOT model durability (assumes a durable log, restores `log` intact on restart `raft.tla:165-174`) so it CANNOT prove the commit-durability invariant — that invariant is backed by tests 2.x/5.1 + the deterministic RED→GREEN E2E + the zero-loss/available soak (5.2), NOT by raft.tla. `src/controller/CLAUDE.md`: "Commit Durability — KNOWN GAP (bug-661)" flipped to "GUARANTEED / CLOSED (landed 2026-06-18)" with the mechanism list (incl. the no-self-exit loud fence); the post-`ha-commit-durability` checklist item ticked `[x]`. ALL 17 tasks (1.1–5.3) + the no-self-exit rework COMPLETE.
|
||||
- 6.1-6.4 (2026-07-02): Ran `src/tools/test-commit-durability.sh ~/tmp/xce-durability-deploy 8` (default 8 rolling rounds + `run_cold_restart_recovery`) 5 times against a live 3-node cluster deployed to `~/tmp` — confirmed via `mount`/`df -T` to be genuine ext4 on `/dev/nvme0n1p2`, NOT the container's overlay `/`. **Verdict: the cold-restart-all committed-job LOSS does NOT reproduce on a real fs.** Across 5 full runs (40 rolling-failover rounds + 5 cold-restart-all cycles total): zero instances of a committed job vanishing after a full cold-stop-all + cold-start-all cycle. Confirms bug-795's original sighting was an overlayfs `fdatasync` artifact, not a genuine total-outage durability gap — the branch predicted in task 6.1's own Observable. 6.2/6.3 (conditional on 6.1=REAL) are therefore N/A. One SEPARATE, unrelated anomaly surfaced in run 5/5: `expect-jobs` reported `actual 20` vs `expected 19` after cold-restart-all — an EXTRA committed job, not a missing one (the opposite of bug-795's symptom), root cause not yet investigated (deprioritized — doesn't match the loss scenario this phase was probing for). Logged as **bug-1396** (open, low-priority, unconfirmed whether harness-side ACKED_FILE race or a genuine controller duplication). `src/controller/CLAUDE.md`'s "Commit Durability" caveat updated to reflect the closed investigation (see that file).
|
||||
- VALIDATION FIX (2026-06-18, `/kiro-validate-impl` finding bug-760): the feature-level validation caught a deterministic 6/6 FAIL of the ADJACENT-spec test `test_fact_hook_append_is_retained_and_nonblocking` (`test_commit_barrier_rollout.c:446`, "Expected 1 Was 0"). ROOT CAUSE = task 2.2's durable-gating: committed_lsn now requires the async group-commit flusher to advance durable_lsn, but this real-persist test checked committed_lsn SYNCHRONOUSLY the instant after the covering ACK (before the flusher fdatasync'd). Task 2.2 had primed durable_lsn in the in-memory `barrier_setup_vcpus` stubs but MISSED this real-flusher test. CORRECTION of the earlier impl-notes (3.1/5.1) claim that this was a pre-existing `/tmp`-overlay fdatasync-EIO artifact: that is FALSE in this env — fdatasync returns rc=0 on /tmp here and the test passed pre-spec; it was a genuine durable-gating regression. FIX: insert `persist_flush(g_fp)` + assert `persist_durable_lsn(g_fp) >= fact_lsn` BEFORE `repl_on_ack` (commit_on_ack runs synchronously inside repl_on_ack, so the now-durable self completes the majority deterministically). Production was never affected (RETAIN facts have no client waiter; WAIT-mode replies block on the commit barrier `commit.c:334`). Verified: full controller suite 0 failures across 4 runs + meson `Ok:1 Fail:0`. GATING NOTE: meson `Fail:0` masks Unity sub-group failures — gate on the ANSI-stripped `:FAIL:` count from the test binary stdout.
|
||||
Loading…
Add table
Add a link
Reference in a new issue