Guards at the top of mark_job_start/finish/update under the caller's
job lock, before BEFORE_EVENT: duplicate (FINISH on terminal) → INFO
no-op skipping the AFTER_EVENT hook (no WAL append, lsn frozen);
terminal→non-terminal backward (START/UPDATE) → WARN reject preserving
state. Shared job_state_is_backward predicate provably admits every
production forward merge (PDONE/PERR/suspend bit-OR) while rejecting
overwrites. mark_job_fail intentionally unguarded (its merges are
always terminal-forward; job_ctrl pre-gates terminal). 5/5 guard tests
green; full job-lifecycle suite green; census 0 FAIL.
3 RED carriers (duplicate FINISH on DONE appends a 2nd WAL entry +
overwrites state; START on terminal clobbers it; replay non-equivalent)
+ 2 preservation pins (forward postdone PDONE-merge, fail-after-finish
PERR-merge) that 6.2 must not break. Reviewer-confirmed: the
UPDATE-clears-terminal arm is production-unreachable (pinned, not
driven). 22 tests / 3 RED; census 508 distinct triples, only those 3
fail.
configAck branch: per-push majority (acked_config_lsn >= push_tag,
1+count > n_peers/2 over r->ce->n_peers, n_peers<=1 immediate,
ce==NULL -EINVAL); legacy >0 counting and n_active==0 shortcut dead.
push_config_to_standbys returns the barrier result; upgrade failure ->
send_upgrade_error (rc forwarded), no success ACK, no restart,
committed_config_tag success-only.
Remediation r1 (two CRITICAL review findings):
- F1: wire production config-ACK ingestion — replconfigpushreplymsg
decode branch in repl_ack_decode_and_dispatch -> repl_on_config_ack
on the promoted-vote ep; + defensive cluster.c OPCode_reply case.
Closes the unlanded half of ha-config-bootstrap 8.1; without this the
barrier would brick every multi-member config upgrade. No schema change.
- F2: keep the barrier off the iomgr thread — export
push_config_transport_only (non-blocking) for the 5 ha.c + join.c
streaming sites; barrier path now vtask-only (do_upgrade_config).
Wire-path test drives real capnp ACK bytes through the production decode
branch. 30/30 rollout, 6/6 repl_majority, 46/46 config_repl; census 0
FAIL; pre-existing fixture leak only (less than baseline).
Green integration proof: 4 facts driven during total quorum loss append
RETAIN with rc 0 in <1000ms (vs 5000ms timeout — wrongly-applied WAIT
tripwire), pending with retained-depth=4 logged, present exactly once
on disk (real WAL header scan); reconnect + one covering ACK commits
the suffix with current_lsn unchanged (no redelivery). Completes the
bug-396 fix evidence (umbrella Req 2.8). Census 495/495.
notify/update/setpreason/clrpreason admitted in CTRL_PRIMARY without
write quorum (Req 3.2 / umbrella 2.8 / bug-396): all four terminate in
receipt-style RETAIN fact paths — reviewer-traced no-block proof (no
admitted opcode reaches a barrier/WAIT path; request opcodes stay
rejected). 21/21 group; census 494/494 zero FAIL; RED→GREEN
independently reproduced via stash-rebuild.
4 RED carriers (notify/update/setpreason/clrpreason must be admitted
without quorum — currently NOT_READY, the bug-396 hole) + 2 pins (six
request opcodes stay rejected; quorum-present admits all ten). Census
494 = 488 + 6; exactly 4 FAILs.
4 green pins: handle_jobdispatch replies receipt-only with zero WAL
appends through a real 2-alloc dispatch cycle (notify outcomes intact,
reply-precedes-processing); set/clr_preason and do_update_op rusage
stay receipt-style with RETAIN-pending fact appends ACK-committed.
Umbrella Reply-Ordering Contract cited per pin. DISCOVERY bug-451
(reviewer-verified): RUSAGE hook va_list type confusion — 4-byte OOB
read + garbage state in WAL_JOB_RUSAGE payloads; fix deferred to 7.1.
Census 488; zero failures.
7 cases: two covered short-circuits, timeout with committed_lsn
unmoved, covering-ACK release, sync_timeout_ms fallback (bounded both
directions), n_peers=1 standalone, NOT_READY passthrough via cap
saturation. Watermark resolved to current_lsn - 1 (design's literal
phrasing unsatisfiable per persist.h:71 invariant) with off-by-one
killing power both ways. 15 tests / 7 RED; suite baseline zero.
Verified table implemented (STOP/RESUME/UPDATE/FAIL RETAIN per
supersession; WORKER detail switch; CHECKPOINT LOCAL); unknown type =
ERROR + XCE_TESTING abort + production RETAIN fallback. All three
hooks route through wal_hook_append_classified (zero bare async
appends); RETAIN registers with the engine lock-free; facts propagate
local append rc only. 8/8 group tests green; full suite 465/465.
Fixed the 3 bug-411 baseline failures honestly: stale timeout
expectation updated to the reject-on-timeout contract (engine-wired
fixture, killing power retained); two healthy-primary gate fixtures
populated with standby_endpoints (gate logic untouched). Census: 46
groups / 467 tests / 0 failures standalone under default ASAN.
Audit (reviewer-confirmed, recorded in tasks.md): no production walAck
path reaches the deprecated acked_lsn>0 counting or n_active==0
short-circuit except the documented LSN-0 first entry. Leak inventory
for 6 groups documented as follow-up.
cluster_t embeds commit_engine_t; init in cluster_init_phase2 after the
vtaskmgr guard (idempotence-guarded — phase2 can run twice), wiring
repl_set_commit_engine + ha.repl back-pointer before any standby can
register; fini after iomgr/vtask shutdown. persist_wal_sync_append
passes written_lsn; engine-wired path drops -ENODEV and the
committed_lsn store (engine owns advancement); LSN-0 first-entry
routes to legacy per the accepted interim. p2 -ENODEV pin updated to
the Req 6.3 contract. 34/34 + 6/6 green through honest delegation;
census exactly at the 3 validated baselines; production binary links.
replicator_t gains ce pointer (set-once-before-traffic contract).
repl_on_ack feeds commit_on_ack after releasing r->lock (iomgr-safe);
four membership mutators feed commit_on_membership_change post-unlock,
gated on actual mutation. walAck branch delegates to commit_submit(WAIT)
when ce wired AND entry_lsn != 0 (LSN-0 interim guard until 5.3);
legacy counting kept for ce==NULL, marked DEPRECATED. commit.c linked
into production target (inert until 5.3 wiring). Tests 30-31 + all
repl_majority walAck cases green; 33/34 remain RED for 5.3; baseline
unchanged.
repl_active_standby_count: registry truth (ep != NULL && !in_catch_up)
under r->lock. ha_state_t gains repl back-pointer (wired at init by
task 5.3); ha_has_write_quorum uses registry truth when wired, exact
legacy fallback when NULL; formula unchanged. Test 29 (wired gate,
catch-up-only standby → NOT_READY, admits after clear) captured RED
pre-fix. 29/29 group tests; suite at the 3-failure baseline.
Tests 21-28: repl_active_standby_count contract (5 RED vs inert -1
stub) + 3 gate-level pins (n_peers<=1 always-true, zero-connected
reject, stateless re-accept) that 4.2 must preserve. Declaration in
repl.h; ha.c/ha.h untouched (4.2 boundary). 28 tests / 5 RED; suite
at the 3-failure baseline with unchanged signatures.
Tests 17-20: catch-up ACK exclusion vs pending waiter, membership-change
wake of covered waiter, removal-below-majority fails at own deadline
(no forced early-fail), non-blocking bound. Landed GREEN: task 1.2's
commit_on_membership_change already implements the full 3.2 contract,
so RED was unachievable; reviewer confirmed mutation-killing power and
zero production changes. 20/20 group tests; suite at baseline.
commit_submit(WAIT): n_vcpus-relative admission cap with fast retriable
NOT_READY (admission-cap-only per corrected contract), CLOCK_REALTIME
timedwait loop on the per-LSN + gate predicate, post-timeout predicate
re-check, n_peers<=1 self-quorum through the single advancement site.
Inflight counter accurate under engine lock for the full blocking
interval with XCE_TESTING asserts. 16/16 group tests green unmodified;
full suite at the 3-failure baseline.
9 new cases (8-16): block-until-majority, deadline timeout, standalone
fast path, zero-ACK regression, n_vcpus-relative cap with floor, ACK
progress at saturation, gate as blocking predicate (NOT_READY is
admission-cap-only per corrected contract). Main-thread-only assertions
after joins; 16 tests / 9 RED failures; baseline unchanged.
Add test_repl_registration_lifecycle covering the complete 3-step standby
lifecycle: register (nstandbys==1, ep pointer verified), duplicate
(idempotent, same slot), deregister (ep==NULL, nstandbys==0 — the
previously-untested slot-zeroing invariant). Tasks 5.1/5.2 were already
covered by test_broadcast_one_standby_sends_entry, test_wal_wire
round-trip, and test_wal_ack_reply_updates_acked_lsn_and_signals.
18 repl unit tests pass under ASAN.
Fix defective assertion in test_committed_lsn_advances_after_majority_ack
(before/after delta instead of >0 which fails for lsn=0 first write).
Add test_single_node_committed_lsn_unchanged documenting -ENODEV degraded
mode for no-standby path. Fix NACK test expected return from -ETIMEDOUT to
0: with 1 registered standby, quorum=1 is met by primary self-count alone.
308 tests pass, 0 fail under ASAN.
Handle incoming WalAckReply in repl_ack_decode_and_dispatch: extract
lsn+success, call repl_on_ack on success to update standby acked_lsn
and signal majority-ack waiter. NACK is WARN-logged with no state
change. Two unit tests verify positive path (acked_lsn updated, waiter
unblocked) and negative path (NACK leaves acked_lsn=0, wait times out).
Standby sends WalAckReply via xce_reply after each successfully
applied WAL entry. ha_apply_wal_catchup_entry signature extended
with endpoint_t* and cluster_t* parameters. walackreplymsg union
arm added to RequestReply capnp schema and xce_reply_t. Error
entries skip ACK. Dispatch gate test confirms OPCode_walCatchupEntry
allowed in CTRL_STANDBY/ELECTING/INIT.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Election protocol fixes validated against raft.tla invariants:
- ha_init_raft(): one-time raft state init at startup (re-init would lose
durable election progress); n_peers = full cluster size including self
- handle_request_vote(): higher-term step-down moved BEFORE the role gate
(split-brain fix); standby vote-deny now requires a live primary control
channel (primary_ctrl_fd >= 0) AND a fresh heartbeat
- ha_heartbeat_is_fresh(): derive freshness from ha->last_activity (the
field production actually writes); the orphaned last_heartbeat timespec
had no production writer, leaving the vote-deny gate inert (bug-337);
field removed from ha_state_t
- handle_request_vote(): restore same-candidate idempotent re-grant —
deny only a DIFFERENT candidate in the same term, matching raft.c and
Raft §5.2 votedFor ∈ {Nil, candidateId} (bug-338)
- raft_init(): restore voted_for_term alongside a persisted vote; leaving
it 0 allowed a double vote in the same term across a restart (bug-339)
- myapp_run(): leader_known check treats a vote recorded under this node's
own name as a self-vote — production persists the real candidate name,
the "self" placeholder only appears in test mode
- raft_on_vote_request(): leader steps down on higher-term vote request
instead of denying it
Tests: test_proto_H6_restored_vote_blocks_second_vote added; H1 restore
assertion strengthened; test_ha_vote_gate.c driven through last_activity +
primary_ctrl_fd (the real production fields). Full /xce-build all green
(lib + controller + scheduler + Go suites, 0 failures, ASAN clean).
deploy-ha.sh: start.sh now forwards extra args to the controller binary.
Root cause: task_cmp in delay.c compared nextrun (always 0 for ms-tasks)
instead of nextrun_ts, making all ms-tasks appear equal in the ordered
queue. The heartbeat task was re-enqueued correctly but stuck behind the
liveness poll task in the broken ordering.
Changes:
- delay.c: fix task_cmp to compare nextrun_ts for ms-tasks (ns precision)
- ha.c: use runcount=-1 (infinite repeat) instead of one-shot+re-arm;
simpler and avoids the broken re-arm path entirely
- ha.c: add static atomic idempotency guard to prevent duplicate heartbeat
tasks on re-election
- ha_state_t.n_peers now matches raft_config_t.n_peers convention:
total cluster size (including self), consistently across both layers
- ha_start_election_async: ha->n_peers = n_peers (candidates_count)
- Convergence/counting loops: i < n_peers - 1 (skip self for standby arrays)
- Single-node: n_peers=1, threshold=0 (same behavior, clear semantics)
- ha_elect_ctx_t.n_peers unchanged (separate field for vote array sizing)
- Tests: 3-node n_peers=2->3, single-node 0->1, loop bounds updated
E2E test 6.7: 3-node cluster lifecycle verified
- Initial: ctrl-1 PRIMARY via majority convergence, no degraded mode
- Kill primary: RDHUP detected on both standbys, ctrl-2 elected at term=4
- No data loss: config tag preserved across failover
- Full ASAN build green
- Fix ha_start_election_async: ha->n_peers = n_other (other peers),
not candidates_count (total including self). The standby_converged[]
array is indexed by peer index, so the bound must be n_other.
- Fix ha_compute_safe_committed_lsn: return self_lsn for n_peers==0
(prevents WAL truncation on single-node re-election).
- Integration test: 3-node cluster reaches CTRL_PRIMARY via majority
convergence within ~20s; no degraded-mode WARN in logs.
6.3: Fix single-node bug — ha_compute_safe_committed_lsn now returns
self_lsn instead of 0 for n_peers=0 (prevents WAL truncation on
single-node re-election). Add self_lsn/self_tag to ha_elect_ctx_t,
populated unconditionally in ha_start_election_async. All 3 test
scenarios pass under ASAN.
6.4: Add test_snapshot_recv_128kb_two_chunks (128KB snapshot in
2x64KB chunks, verifies size and applied_lsn); 4 other test cases
already existed from tasks 3.5c/3.5d. All ASAN-clean.
