feat(ha-commit-barrier-rollout): config barrier with production ACK ingestion (task 5.2 + r1)
configAck branch: per-push majority (acked_config_lsn >= push_tag, 1+count > n_peers/2 over r->ce->n_peers, n_peers<=1 immediate, ce==NULL -EINVAL); legacy >0 counting and n_active==0 shortcut dead. push_config_to_standbys returns the barrier result; upgrade failure -> send_upgrade_error (rc forwarded), no success ACK, no restart, committed_config_tag success-only. Remediation r1 (two CRITICAL review findings): - F1: wire production config-ACK ingestion — replconfigpushreplymsg decode branch in repl_ack_decode_and_dispatch -> repl_on_config_ack on the promoted-vote ep; + defensive cluster.c OPCode_reply case. Closes the unlanded half of ha-config-bootstrap 8.1; without this the barrier would brick every multi-member config upgrade. No schema change. - F2: keep the barrier off the iomgr thread — export push_config_transport_only (non-blocking) for the 5 ha.c + join.c streaming sites; barrier path now vtask-only (do_upgrade_config). Wire-path test drives real capnp ACK bytes through the production decode branch. 30/30 rollout, 6/6 repl_majority, 46/46 config_repl; census 0 FAIL; pre-existing fixture leak only (less than baseline).
This commit is contained in:
parent
9cdeba5816
commit
5a2fae1d37
12 changed files with 1034 additions and 399 deletions
|
|
@ -72,7 +72,7 @@
|
|||
- Observable completion: new cases FAIL (red) against the current logged-only result (config_repl.c:916-929) and unconditional success ACK (config_repl.c:1062-1073)
|
||||
- _Requirements: 4.1, 4.2, 4.3, 4.4_
|
||||
|
||||
- [ ] 5.2 Implement the config barrier to green
|
||||
- [x] 5.2 Implement the config barrier to green (+ remediation r1: wired production config-ACK ingestion, kept barrier off iomgr thread)
|
||||
- repl.c: configAck branch takes the in-flight push tag (signature change in src/include/repl.h), counts `acked_config_lsn >= push_tag`, majority `1 + count > n_peers / 2` over configured members, `n_peers <= 1` immediate success
|
||||
- config_repl.c: `push_config_to_standbys` gains `int *acked_out` and returns the barrier result, passing `new_tag` to the wait; `do_upgrade_config_with_bytes` (config_repl.c:1056-1094 anchor) on non-zero return sends `send_upgrade_error`, skips the success ACK and skips the restart request; `committed_config_tag` advance stays success-branch-only
|
||||
- Observable completion: all task 5.1 tests PASS; a failed upgrade leaves the admin tool with an explicit error and the primary running on the old committed tag
|
||||
|
|
@ -112,6 +112,7 @@
|
|||
- (1.1) wal_entry_type has 13 values (wal_types.h:44-58), no WAL_NOOP yet (lands in ha-single-leader-role, which must add a classification row + sweep entry — the totality sweep will catch it).
|
||||
- (1.1) Suite baseline is ZERO failures since ha-commit-engine 6.1. detect_leaks=0 remains the harness norm (6 inventoried pre-existing leak groups).
|
||||
- (1.2) CENSUS METHOD: only ~9 of 48 groups emit Unity summary blocks; the rest report per-test PASS lines only — count distinct Unity results (~465) and grep FAIL lines, never sum summary blocks. commit_submit(RETAIN) early-returns BEFORE taking ce->lock (commit.c:233-239) — registration is lock-free, safe with entity write locks held. wal_hook_append_classified is the single classified-append seam; appended_lsn read = current_lsn - 1 under p->lock (benign race, commented).
|
||||
- (5.2) **REJECTED then remediated (r1)** — two CRITICAL findings the first pass missed, both fixed: (F1) the config-ACK signal was never produced in production (repl_on_config_ack had zero callers; the standby's replconfigpushreplymsg reply was dropped on the primary) so the new authoritative barrier would have BRICKED every multi-member config upgrade — this was the unlanded half of ha-config-bootstrap task 8.1. Fixed by a replconfigpushreplymsg decode branch in repl_ack_decode_and_dispatch (ha.c) → repl_on_config_ack on the repl_add_standby-registered promoted-vote ep (production-normal path), + a defensive cluster.c OPCode_reply ingestion for inbound standbys (no-ops against unregistered slots, forward-looking). Schema unchanged (ReplConfigPushReply already carried versionTag). (F2) the 500ms barrier wait reached the iomgr epoll thread via 5 ha.c push sites. Fixed by exporting push_config_transport_only (non-blocking fan-out) for all best-effort ha.c/join.c streaming sites; push_config_to_standbys (transport+barrier) now has exactly ONE caller — do_upgrade_config_with_bytes in vtask context. LESSON: the design's "handlers run on vtask threads" claim was only true for cluster.c-dispatched opcodes, not the ha.c peer-channel push sites the design never analyzed (ha.c was outside its modified-files list). Wire-path test drives real capnp ACK bytes through the production decode branch.
|
||||
- (3.3) **bug-451 DISCOVERED (reviewer-verified byte-precise, NOT fixed — boundary)**: update_job_rusages fires HOOK_DETAIL_JOB_RUSAGE_UPDATE with (cluster_t*, xce_job_rusages_t*) (job.c:1014-1015) but persist_job_hook_fun's default arm (persist.c:1355-1359) reads vararg 2 as job_t* and dereferences j->job.state at offset 56 — one past the end of the exactly-56-byte xce_job_rusages_t → 4-byte OOB read + garbage state field in every WAL_JOB_RUSAGE payload (jobid lines up by accident). Production fix candidate for task 7.1's sweep or a follow-up: persist arm needs a RUSAGE-specific va_list signature. Census now 488.
|
||||
- (3.2) bug-447 FIXED via rc normalization inside the three ctrl fns (real errors/misses preserved; reviewer-verified per-fn). Semantics change shipped intentionally: idempotent no-op control actions now return 0 (was leaked 1); queue/host mutations now append WAL + replicate. Join barrier failure → error reply + on_worker_join SKIPPED (coherent reading of design; worker retries via idempotent rejoin; documented at join.c:108-113). PRE-EXISTING ref leak: job_ctrl takes reference(j) (job.c:317) but normal exit never dereferences (bug-448) — out of scope, future fix.
|
||||
- (3.1) **bug-447 CONFIRMED (reviewer-verified to librtti disassembly)**: key_find returns 1 on found (unnormalized from xkvmgr_find/kvtab_find); queue_ctrl (queue.c:61, gate :106) and worker_ctrl (worker.c:644, gate :678) leak rc=1 → AFTER_EVENT never fires on the found path → **queue/host control silently unpersisted+unreplicated in production today**; job_ctrl appends via mark_job_* hooks but still replies status 1. Fix point = 3.2's handler rc normalization (key_find is in binary librtti, out of repo). RED tests pin it.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue