Commit graph

12 commits

Author SHA1 Message Date
5a2fae1d37 feat(ha-commit-barrier-rollout): config barrier with production ACK ingestion (task 5.2 + r1)
configAck branch: per-push majority (acked_config_lsn >= push_tag,
1+count > n_peers/2 over r->ce->n_peers, n_peers<=1 immediate,
ce==NULL -EINVAL); legacy >0 counting and n_active==0 shortcut dead.
push_config_to_standbys returns the barrier result; upgrade failure ->
send_upgrade_error (rc forwarded), no success ACK, no restart,
committed_config_tag success-only.

Remediation r1 (two CRITICAL review findings):
- F1: wire production config-ACK ingestion — replconfigpushreplymsg
  decode branch in repl_ack_decode_and_dispatch -> repl_on_config_ack
  on the promoted-vote ep; + defensive cluster.c OPCode_reply case.
  Closes the unlanded half of ha-config-bootstrap 8.1; without this the
  barrier would brick every multi-member config upgrade. No schema change.
- F2: keep the barrier off the iomgr thread — export
  push_config_transport_only (non-blocking) for the 5 ha.c + join.c
  streaming sites; barrier path now vtask-only (do_upgrade_config).

Wire-path test drives real capnp ACK bytes through the production decode
branch. 30/30 rollout, 6/6 repl_majority, 46/46 config_repl; census 0
FAIL; pre-existing fixture leak only (less than baseline).
2026-06-13 03:46:29 +00:00
732e836868 test(ha-commit-barrier-rollout): fact-retention-across-quorum-loss proof (task 4.3)
Green integration proof: 4 facts driven during total quorum loss append
RETAIN with rc 0 in <1000ms (vs 5000ms timeout — wrongly-applied WAIT
tripwire), pending with retained-depth=4 logged, present exactly once
on disk (real WAL header scan); reconnect + one covering ACK commits
the suffix with current_lsn unchanged (no redelivery). Completes the
bug-396 fix evidence (umbrella Req 2.8). Census 495/495.
2026-06-12 15:29:47 +00:00
7103fa643d test(ha-commit-barrier-rollout): receipt-ACK verdict pins (task 3.3)
4 green pins: handle_jobdispatch replies receipt-only with zero WAL
appends through a real 2-alloc dispatch cycle (notify outcomes intact,
reply-precedes-processing); set/clr_preason and do_update_op rusage
stay receipt-style with RETAIN-pending fact appends ACK-committed.
Umbrella Reply-Ordering Contract cited per pin. DISCOVERY bug-451
(reviewer-verified): RUSAGE hook va_list type confusion — 4-byte OOB
read + garbage state in WAL_JOB_RUSAGE payloads; fix deferred to 7.1.
Census 488; zero failures.
2026-06-12 11:16:51 +00:00
facdb90917 feat(ha-commit-barrier-rollout): commit-first-then-reply in all four handlers (task 3.2)
Uniform pattern (ctrl fn -> persist_commit_barrier -> reply) in
do_jobcontrol_op/do_queuecontrol_op/do_hostcontrol_op; join barrier
between AFTER_EVENT and success reply (failure -> error reply, skip
on_worker_join). FIXES bug-447: key_find found-rc normalized inside
job_ctrl/queue_ctrl/worker_ctrl — queue/host control now actually
append WAL and replicate (was silently unpersisted); idempotent no-ops
now return 0. handle_jobsubmit untouched. 12/12 + 15/15; census zero
failures.
2026-06-12 10:28:25 +00:00
e1c3c173f4 test(ha-commit-barrier-rollout): RED commit-first-then-reply tests per opcode (task 3.1)
New test_commit_barrier_reply.c drives do_jobcontrol_op /
do_queuecontrol_op / do_hostcontrol_op / handle_worker_join directly
(pthread harness, ep->outq reply capture, wired engine n_peers=3).
12 cases: 8 RED (reply-before-commit + bug-447 pins), 4 validation
pins green. DISCOVERY bug-447 (reviewer-confirmed): key_find found-rc
leak means queue/host control never append WAL in production — 3.2's
rc normalization is the fix point. Census 484 = 472 + 12; zero
baseline regressions.
2026-06-12 10:00:38 +00:00
95a10cedf3 test(ha-commit-engine): full regression green + legacy-path audit (task 6.1)
Fixed the 3 bug-411 baseline failures honestly: stale timeout
expectation updated to the reject-on-timeout contract (engine-wired
fixture, killing power retained); two healthy-primary gate fixtures
populated with standby_endpoints (gate logic untouched). Census: 46
groups / 467 tests / 0 failures standalone under default ASAN.
Audit (reviewer-confirmed, recorded in tasks.md): no production walAck
path reaches the deprecated acked_lsn>0 counting or n_active==0
short-circuit except the documented LSN-0 first entry. Leak inventory
for 6 groups documented as follow-up.
2026-06-12 03:32:19 +00:00
d5a0f9c64d test(ha-commit-engine): RED tests for per-LSN majority evaluation (task 1.1)
commit.h umbrella-verbatim interface + inert stub commit.c (test target
only); 7 Unity tests covering per-LSN vs acked_lsn>0 counting, majority
median, monotonicity, append clamp, lagging standby, RETAIN immediacy.
All 7 fail RED as intended. Documented 3 pre-existing masked suite
failures (bug-411) and the per-group exit-code masking root cause.
2026-06-11 14:17:09 +00:00
06135954ff chore: baseline snapshot — prior-session work + ha-consistency-invariants spec family
Pre-existing uncommitted work: ha-repl-hardening payload data plane (7.x),
heartbeat liveness fix (Req 1.3), connection-reuse ASAN fixes, doc cleanup.
New: ha-consistency-invariants umbrella (requirements/design/research) and
its four implementation sub-specs (specs generated via batch, cross-spec
review applied), steering roadmap + product.md Raft+CP decision.
2026-06-11 13:21:50 +00:00
18e351ee1b feat(ha-wal-replication): repl_broadcast TCP send (task 1.1)
Replace no-send stub with real WAL entry transmission via
xce_wal_batch_req_raw. For each non-catch-up registered standby,
serialize the entry header+payload into a wal_catchup_entry_t
and enqueue. Keep the MAX_PENDING_ACKS eviction guard.

5 new tests: no-standbys, one-standby-send, skip-catch-up,
evict-overloaded, send-failure-continues. Controller suite green.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-06-10 08:11:05 +00:00
cdd446fc17 docs(specs): gap-analysis validation pass across all 9 HA-family specs
Appended "Gap Analysis — Validation Pass (2026-06-09)" to each spec's
research.md with requirement-to-asset maps, status tags, and effort/risk.

Key outcomes:
- ha-repl-hardening: REOPENED with task series 7.1-7.7 — replication
  payload data plane is missing (snapshot chunks and WAL catch-up ship
  header-only; standbys converge on LSN without holding entity state)
- worker-join-refactor, resval-id-decoupling: marked implemented
  (all criteria re-verified, full test suite green)
- buglog: 335/336 (scheduler dequeue, open), 337/338/339 (fixed in
  previous commit); cerebrum Decision Log corrected (term-only vote
  deny was recorded as intentional but was a regression)
2026-06-10 01:26:33 +00:00
fa66d1ba9c finalize configuration replication 2026-06-09 16:40:13 +08:00
e31673dd88 chore: checkpoint all prior work for clean state
Captures all uncommitted work from ha-config-bootstrap,
ha-heartbeat-liveness, and associated spec/tooling updates:

- src: ha.c, ha.h, cluster.c, config_repl.c, myapp.c, raft.c —
  heartbeat liveness, RDHUP integration, committing phase fixes
- tests: test_ha_role.c, test_config_repl.c, test_ha_rdhup_integration.c,
  test_p1_integration.c, test_p3_hardening.c, scheduler test
- specs: election-iomgr-refactor, ha-heartbeat-liveness,
  ha-peer-connection-reuse, raft-library (new); ha-config-bootstrap
  and election-iomgr-refactor task/research updates
- tooling: .wolf/ state, ha/data cluster fixtures, raft/raft.tla,
  src/build symlink, xce-build skill update
2026-06-04 13:04:16 +00:00