feat(ha-commit-barrier-rollout): config barrier with production ACK ingestion (task 5.2 + r1)
configAck branch: per-push majority (acked_config_lsn >= push_tag, 1+count > n_peers/2 over r->ce->n_peers, n_peers<=1 immediate, ce==NULL -EINVAL); legacy >0 counting and n_active==0 shortcut dead. push_config_to_standbys returns the barrier result; upgrade failure -> send_upgrade_error (rc forwarded), no success ACK, no restart, committed_config_tag success-only. Remediation r1 (two CRITICAL review findings): - F1: wire production config-ACK ingestion — replconfigpushreplymsg decode branch in repl_ack_decode_and_dispatch -> repl_on_config_ack on the promoted-vote ep; + defensive cluster.c OPCode_reply case. Closes the unlanded half of ha-config-bootstrap 8.1; without this the barrier would brick every multi-member config upgrade. No schema change. - F2: keep the barrier off the iomgr thread — export push_config_transport_only (non-blocking) for the 5 ha.c + join.c streaming sites; barrier path now vtask-only (do_upgrade_config). Wire-path test drives real capnp ACK bytes through the production decode branch. 30/30 rollout, 6/6 repl_majority, 46/46 config_repl; census 0 FAIL; pre-existing fixture leak only (less than baseline).
This commit is contained in:
parent
9cdeba5816
commit
5a2fae1d37
12 changed files with 1034 additions and 399 deletions
|
|
@ -72,7 +72,7 @@
|
|||
- Observable completion: new cases FAIL (red) against the current logged-only result (config_repl.c:916-929) and unconditional success ACK (config_repl.c:1062-1073)
|
||||
- _Requirements: 4.1, 4.2, 4.3, 4.4_
|
||||
|
||||
- [ ] 5.2 Implement the config barrier to green
|
||||
- [x] 5.2 Implement the config barrier to green (+ remediation r1: wired production config-ACK ingestion, kept barrier off iomgr thread)
|
||||
- repl.c: configAck branch takes the in-flight push tag (signature change in src/include/repl.h), counts `acked_config_lsn >= push_tag`, majority `1 + count > n_peers / 2` over configured members, `n_peers <= 1` immediate success
|
||||
- config_repl.c: `push_config_to_standbys` gains `int *acked_out` and returns the barrier result, passing `new_tag` to the wait; `do_upgrade_config_with_bytes` (config_repl.c:1056-1094 anchor) on non-zero return sends `send_upgrade_error`, skips the success ACK and skips the restart request; `committed_config_tag` advance stays success-branch-only
|
||||
- Observable completion: all task 5.1 tests PASS; a failed upgrade leaves the admin tool with an explicit error and the primary running on the old committed tag
|
||||
|
|
@ -112,6 +112,7 @@
|
|||
- (1.1) wal_entry_type has 13 values (wal_types.h:44-58), no WAL_NOOP yet (lands in ha-single-leader-role, which must add a classification row + sweep entry — the totality sweep will catch it).
|
||||
- (1.1) Suite baseline is ZERO failures since ha-commit-engine 6.1. detect_leaks=0 remains the harness norm (6 inventoried pre-existing leak groups).
|
||||
- (1.2) CENSUS METHOD: only ~9 of 48 groups emit Unity summary blocks; the rest report per-test PASS lines only — count distinct Unity results (~465) and grep FAIL lines, never sum summary blocks. commit_submit(RETAIN) early-returns BEFORE taking ce->lock (commit.c:233-239) — registration is lock-free, safe with entity write locks held. wal_hook_append_classified is the single classified-append seam; appended_lsn read = current_lsn - 1 under p->lock (benign race, commented).
|
||||
- (5.2) **REJECTED then remediated (r1)** — two CRITICAL findings the first pass missed, both fixed: (F1) the config-ACK signal was never produced in production (repl_on_config_ack had zero callers; the standby's replconfigpushreplymsg reply was dropped on the primary) so the new authoritative barrier would have BRICKED every multi-member config upgrade — this was the unlanded half of ha-config-bootstrap task 8.1. Fixed by a replconfigpushreplymsg decode branch in repl_ack_decode_and_dispatch (ha.c) → repl_on_config_ack on the repl_add_standby-registered promoted-vote ep (production-normal path), + a defensive cluster.c OPCode_reply ingestion for inbound standbys (no-ops against unregistered slots, forward-looking). Schema unchanged (ReplConfigPushReply already carried versionTag). (F2) the 500ms barrier wait reached the iomgr epoll thread via 5 ha.c push sites. Fixed by exporting push_config_transport_only (non-blocking fan-out) for all best-effort ha.c/join.c streaming sites; push_config_to_standbys (transport+barrier) now has exactly ONE caller — do_upgrade_config_with_bytes in vtask context. LESSON: the design's "handlers run on vtask threads" claim was only true for cluster.c-dispatched opcodes, not the ha.c peer-channel push sites the design never analyzed (ha.c was outside its modified-files list). Wire-path test drives real capnp ACK bytes through the production decode branch.
|
||||
- (3.3) **bug-451 DISCOVERED (reviewer-verified byte-precise, NOT fixed — boundary)**: update_job_rusages fires HOOK_DETAIL_JOB_RUSAGE_UPDATE with (cluster_t*, xce_job_rusages_t*) (job.c:1014-1015) but persist_job_hook_fun's default arm (persist.c:1355-1359) reads vararg 2 as job_t* and dereferences j->job.state at offset 56 — one past the end of the exactly-56-byte xce_job_rusages_t → 4-byte OOB read + garbage state field in every WAL_JOB_RUSAGE payload (jobid lines up by accident). Production fix candidate for task 7.1's sweep or a follow-up: persist arm needs a RUSAGE-specific va_list signature. Census now 488.
|
||||
- (3.2) bug-447 FIXED via rc normalization inside the three ctrl fns (real errors/misses preserved; reviewer-verified per-fn). Semantics change shipped intentionally: idempotent no-op control actions now return 0 (was leaked 1); queue/host mutations now append WAL + replicate. Join barrier failure → error reply + on_worker_join SKIPPED (coherent reading of design; worker retries via idempotent rejoin; documented at join.c:108-113). PRE-EXISTING ref leak: job_ctrl takes reference(j) (job.c:317) but normal exit never dereferences (bug-448) — out of scope, future fix.
|
||||
- (3.1) **bug-447 CONFIRMED (reviewer-verified to librtti disassembly)**: key_find returns 1 on found (unnormalized from xkvmgr_find/kvtab_find); queue_ctrl (queue.c:61, gate :106) and worker_ctrl (worker.c:644, gate :678) leak rc=1 → AFTER_EVENT never fires on the found path → **queue/host control silently unpersisted+unreplicated in production today**; job_ctrl appends via mark_job_* hooks but still replies status 1. Fix point = 3.2's handler rc normalization (key_find is in binary librtti, out of repo). RED tests pin it.
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -9,6 +9,7 @@
|
|||
<!-- How the user likes things done. Code style, tools, patterns, communication. -->
|
||||
|
||||
## Key Learnings
|
||||
- (2026-06-13, barrier-rollout 5.2) push_config_to_standbys now RETURNS the configAck barrier result (0/-ETIMEDOUT/-EINVAL), push count via acked_out; the configAck branch of repl_wait_majority_ack reads entry_lsn as the push tag and r->ce->n_peers (set-once, lock-free) as the denominator, ce==NULL -> -EINVAL fail-loud. ha.c/join.c streaming pushes discard the return BY DESIGN (log-only); only do_upgrade_config_with_bytes treats it as authoritative.
|
||||
|
||||
- [2026-06-12] Census parsing caveat: controller_unit_test logs (tinylog) and Unity result lines BOTH go to stdout and can splice mid-line under the multi-threaded config-push tests — a distinct-Unity-line count silently drops spliced results. Robust census = union of Unity result names AND the `test_x: PASS` printf lines, cross-checked against the per-group `N Tests M Failures` summary blocks (which are splice-immune).
|
||||
- [2026-06-12] Config-barrier test trick: when a test needs a covering config ACK for a tag only known AFTER conf_save_config hashes the re-encoded blob (upgradeConfig path), pre-ack with acked_config_lsn = UINT64_MAX — covers any push tag under both legacy (>0) and per-push (>= tag) counting.
|
||||
|
|
|
|||
|
|
@ -977,3 +977,37 @@
|
|||
| 08:51 | Edited src/controller/testsuite/test_commit_barrier_rollout.c | 3→8 lines | ~109 |
|
||||
| 16:10 | Task 5.1 ha-commit-barrier-rollout: RED tests for configAck per-push arithmetic + upgrade failure path; behavior-preserving push_config_to_standbys signature change (+int *acked_out, return semantics unchanged) | src/controller/{config_repl.c,config_repl.h,ha.c,join.c}, testsuite/{test_commit_barrier_rollout.c,test_repl_majority.c} | 8 new cases (6 RED + 2 pins), isolated group 29 tests/6 fails, full suite: only those 6 fail (summary blocks), repl_majority green, prod controller builds | ~110k |
|
||||
| 09:31 | Edited .kiro/specs/ha-commit-barrier-rollout/tasks.md | inline fix | ~22 |
|
||||
| 17:15 | Edited src/controller/repl.c | modified track() | ~202 |
|
||||
| 17:15 | Edited src/controller/repl.c | added 2 condition(s) | ~323 |
|
||||
| 17:16 | Edited src/controller/repl.c | quorum() → success() | ~305 |
|
||||
| 17:16 | Edited src/include/repl.h | wired() → members() | ~127 |
|
||||
| 17:17 | Edited src/controller/config_repl.c | modified if() | ~942 |
|
||||
| 17:18 | Edited src/controller/config_repl.c | modified config_push_transport() | ~792 |
|
||||
| 17:18 | Edited src/controller/config_repl.c | added 1 condition(s) | ~267 |
|
||||
| 17:19 | Edited src/controller/config_repl.c | standbys() → tool() | ~73 |
|
||||
| 17:19 | Edited src/controller/config_repl.h | modified acked_out() | ~312 |
|
||||
| 00:30 | Task 5.2 ha-commit-barrier-rollout: configAck per-push arithmetic (acked_config_lsn>=tag, 1+count>n_peers/2, n_peers<=1 fast path, ce==NULL -EINVAL); push_config_to_standbys returns barrier result (transport split into config_push_transport helper); do_upgrade_config_with_bytes sends send_upgrade_error + skips ACK/restart on barrier failure | repl.c, repl.h, config_repl.c/.h | 29/29 group, 6/6 repl_majority, 46/46 config_repl, full suite 0 FAIL, sanitizers clean | ~60k |
|
||||
| 01:00 | Adversarial review task 5.2 ha-commit-barrier-rollout: tests green (rollout 29/29, majority 6/6, config_repl 46/46, census 0 FAIL) but REJECTED — repl_on_config_ack has no production caller (config upgrades brick on n_peers>1) + push_config_to_standbys barrier now blocks iomgr thread at ha.c:570/1410 | src/controller/repl.c, config_repl.c, ha.c | REJECTED | ~60k |
|
||||
| 18:03 | Edited src/controller/testsuite/test_commit_barrier_rollout.c | added 1 condition(s) | ~2006 |
|
||||
| 18:04 | Edited src/controller/testsuite/test_commit_barrier_rollout.c | 3→6 lines | ~95 |
|
||||
| 18:10 | Edited src/controller/ha.c | added 3 condition(s) | ~492 |
|
||||
| 18:10 | Edited src/controller/ha.c | 5→8 lines | ~115 |
|
||||
| 18:11 | Edited src/controller/cluster.c | added 2 condition(s) | ~441 |
|
||||
| 18:11 | Edited src/controller/config_repl.c | added 1 condition(s) | ~256 |
|
||||
| 18:11 | Edited src/controller/config_repl.c | added 1 condition(s) | ~35 |
|
||||
| 18:12 | Edited src/controller/config_repl.h | modified n_out() | ~635 |
|
||||
| 18:13 | Edited src/controller/ha.c | modified XCE_TESTING() | ~296 |
|
||||
| 18:14 | Edited src/controller/join.c | push_config_to_standbys() → push_config_transport_only() | ~125 |
|
||||
| 18:14 | Edited src/controller/config_repl.c | modified send_upgrade_error() | ~180 |
|
||||
| 18:15 | Edited src/controller/config_repl.c | modified if() | ~110 |
|
||||
| 18:15 | Edited src/controller/config_repl.c | modified ONLY() | ~172 |
|
||||
| 01:35 | Task 5.2 remediation r1: wired production config-ACK ingestion (ha.c+cluster.c), split push_config_transport_only from barrier entry, switched 5 ha.c + 1 join.c sites, barrier rc into upgrade error reply; wire-path test added | ha.c, cluster.c, config_repl.c/.h, join.c, test_commit_barrier_rollout.c | rollout 30/30, repl_majority 6/6, config_repl 46/46, census 0 FAIL | ~95k |
|
||||
| 18:38 | Edited src/controller/ha.c | inline fix | ~30 |
|
||||
| 18:38 | Edited src/controller/ha.c | inline fix | ~22 |
|
||||
|
||||
## Session: 2026-06-13 20:00
|
||||
|
||||
| Time | Action | File(s) | Outcome | ~Tokens |
|
||||
|------|--------|---------|---------|--------|
|
||||
| 20:45 | Edited .kiro/specs/ha-commit-barrier-rollout/tasks.md | inline fix | ~37 |
|
||||
| 20:46 | Edited .kiro/specs/ha-commit-barrier-rollout/tasks.md | 1→2 lines | ~383 |
|
||||
|
|
|
|||
|
|
@ -482,6 +482,35 @@ static int myreq_handler(vtask_t* task)
|
|||
}
|
||||
break;
|
||||
#endif
|
||||
case OPCode_reply:
|
||||
/* Config-push ACK arriving on an inbound standby endpoint (the
|
||||
* join.c-registered standby_endpoints channel — these eps reply on
|
||||
* the same connection the primary pushed on, so the ACK lands here
|
||||
* instead of peer_ctrl_msg_handler). Feed it to the replicator so
|
||||
* the configAck barrier can observe it (ha-commit-barrier-rollout
|
||||
* task 5.2 remediation r1; same ingestion as
|
||||
* repl_ack_decode_and_dispatch in ha.c — completes the unlanded
|
||||
* half of ha-config-bootstrap task 8.1). */
|
||||
if (request.body_which == Request_body_replymsg
|
||||
&& request.body.replymsg
|
||||
&& request.body.replymsg->data_which
|
||||
== RequestReply_data_replconfigpushreplymsg
|
||||
&& request.body.replymsg->data.replconfigpushreplymsg) {
|
||||
repl_config_push_reply_t *cack =
|
||||
request.body.replymsg->data.replconfigpushreplymsg;
|
||||
if (cack->accepted) {
|
||||
repl_on_config_ack(&c->persist.repl, ep, cack->version_tag);
|
||||
TRACE("myreq_handler: config ACK from inbound standby"
|
||||
" tag=%016" PRIx64, cack->version_tag);
|
||||
} else {
|
||||
WARN("myreq_handler: config push rejected by inbound standby"
|
||||
" (tag=%016" PRIx64 ") — ignoring", cack->version_tag);
|
||||
}
|
||||
rc = 0;
|
||||
break;
|
||||
}
|
||||
/* Any other reply body keeps the legacy unknown-opcode handling. */
|
||||
/* fall through */
|
||||
default:
|
||||
rc = -E_XCE_BAD_OPCODE;
|
||||
ERROR("Unknown opcode: %d", request.opcode);
|
||||
|
|
|
|||
|
|
@ -754,15 +754,18 @@ int handle_repl_config_push(cluster_t *c, endpoint_t *ep, xce_request_t *req)
|
|||
|
||||
/* ── send_upgrade_error ─────────────────────────── */
|
||||
|
||||
/* status: the negative errno carried to the requester (task 5.2 r1,
|
||||
* reviewer finding 3: validation failures keep -EINVAL; the barrier
|
||||
* failure path forwards its own rc, e.g. -ETIMEDOUT). */
|
||||
static void send_upgrade_error(endpoint_t *ep, xce_request_t *req,
|
||||
cluster_t *c, const char *msg)
|
||||
cluster_t *c, int status, const char *msg)
|
||||
{
|
||||
xce_reply_t reply;
|
||||
init_reply_by_req(&reply, req);
|
||||
reply.status = -EINVAL;
|
||||
reply.status = (status < 0) ? status : -EINVAL;
|
||||
reply.data_which = RequestReply_data_nodata;
|
||||
xce_reply(ep, c->myhost, &reply);
|
||||
ERROR("OP_upgradeConfig rejected: %s", msg);
|
||||
ERROR("OP_upgradeConfig rejected (status=%d): %s", reply.status, msg);
|
||||
}
|
||||
|
||||
/* ── push_config_to_standbys ─────────────────────── */
|
||||
|
|
@ -863,21 +866,32 @@ int wait_all_with_timeout_for_test(pthread_t *threads, push_ctx_t *ctxs,
|
|||
}
|
||||
#endif
|
||||
|
||||
int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen,
|
||||
const uint8_t sha256[32], uint64_t new_tag,
|
||||
int *acked_out)
|
||||
/* push_config_transport_only: transport fan-out half of
|
||||
* push_config_to_standbys (exported — task 5.2 remediation r1): spawn one
|
||||
* push thread per registered standby endpoint and wait for the sends.
|
||||
* Best-effort streaming, NO barrier wait — the sends are non-blocking
|
||||
* endpoint_enq()s, so this is safe on iomgr-reachable paths (election
|
||||
* promotion, heartbeat-driven convergence, controller join). Alloc or
|
||||
* thread failures degrade to fewer pushes, never to a fabricated quorum
|
||||
* (the barrier wait in push_config_to_standbys owns the quorum verdict).
|
||||
* Stores the registered standby count in *n_out (may be NULL); returns
|
||||
* the number of standbys that ACKed the transport push. */
|
||||
int push_config_transport_only(cluster_t *c, const uint8_t *bytes,
|
||||
size_t blen, const uint8_t sha256[32],
|
||||
uint64_t new_tag, int *n_out)
|
||||
{
|
||||
/* acked_out: per-standby transport push count (task 5.1 signature;
|
||||
* behavior-preserving — the return value keeps the legacy push-count
|
||||
* semantics until task 5.2 re-plumbs it to the barrier result). */
|
||||
if (acked_out)
|
||||
*acked_out = 0;
|
||||
int n;
|
||||
int acked;
|
||||
|
||||
if (n_out)
|
||||
*n_out = 0;
|
||||
if (!c->ha.standby_endpoints)
|
||||
return 0;
|
||||
|
||||
pthread_mutex_lock(&c->ha.lock);
|
||||
int n = c->ha.standby_endpoints->count;
|
||||
n = c->ha.standby_endpoints->count;
|
||||
if (n_out)
|
||||
*n_out = n;
|
||||
if (n == 0) {
|
||||
pthread_mutex_unlock(&c->ha.lock);
|
||||
return 0;
|
||||
|
|
@ -885,6 +899,8 @@ int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen,
|
|||
push_ctx_t *ctxs = calloc((size_t)n, sizeof(push_ctx_t));
|
||||
if (!ctxs) {
|
||||
pthread_mutex_unlock(&c->ha.lock);
|
||||
WARN("push_config_transport_only: ctx alloc failed (n=%d)"
|
||||
" — skipping transport push", n);
|
||||
return 0;
|
||||
}
|
||||
for (int i = 0; i < n; i++) {
|
||||
|
|
@ -900,32 +916,54 @@ int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen,
|
|||
pthread_t *threads = malloc((size_t)n * sizeof(pthread_t));
|
||||
if (!threads) {
|
||||
free(ctxs);
|
||||
WARN("push_config_transport_only: thread alloc failed (n=%d)"
|
||||
" — skipping transport push", n);
|
||||
return 0;
|
||||
}
|
||||
|
||||
for (int i = 0; i < n; i++) {
|
||||
ctxs[i].started = (pthread_create(&threads[i], NULL, push_thread, &ctxs[i]) == 0);
|
||||
if (!ctxs[i].started)
|
||||
WARN("push_config_to_standbys: pthread_create failed for standby %d", i);
|
||||
WARN("push_config_transport_only: pthread_create failed for standby %d", i);
|
||||
}
|
||||
|
||||
atomic_int push_done;
|
||||
atomic_init(&push_done, 0);
|
||||
|
||||
int acked = wait_all_with_timeout(threads, ctxs, n,
|
||||
CONF_PUSH_TIMEOUT_MS, &push_done);
|
||||
acked = wait_all_with_timeout(threads, ctxs, n,
|
||||
CONF_PUSH_TIMEOUT_MS, &push_done);
|
||||
|
||||
free(ctxs);
|
||||
free(threads);
|
||||
return acked;
|
||||
}
|
||||
|
||||
int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen,
|
||||
const uint8_t sha256[32], uint64_t new_tag,
|
||||
int *acked_out)
|
||||
{
|
||||
/* Returns the majority-ACK barrier result (0 / -ETIMEDOUT / -EINVAL);
|
||||
* the per-standby transport push count travels via acked_out (task 5.2
|
||||
* re-plumb). The transport fan-out is best-effort — push failures and
|
||||
* an empty standby set fall through to the barrier wait, which owns
|
||||
* the quorum policy (n_peers <= 1 → immediate success; multi-member
|
||||
* with no covering acks → -ETIMEDOUT; no n_active shortcut).
|
||||
* VTASK-CONTEXT ONLY (task 5.2 remediation r1): the barrier blocks up
|
||||
* to CONF_PUSH_TIMEOUT_MS — iomgr-reachable callers must use
|
||||
* push_config_transport_only instead (see config_repl.h). */
|
||||
int n = 0;
|
||||
int acked = push_config_transport_only(c, bytes, blen, sha256, new_tag, &n);
|
||||
|
||||
if (acked_out)
|
||||
*acked_out = acked;
|
||||
|
||||
/* Req 8.1–8.3: wait for majority-quorum OP_upgradeConfigAck.
|
||||
* On quorum: advance committed_config_tag atomically (Req 8.5). */
|
||||
/* entry_lsn = 0: the configAck branch ignores it (walAck-only adapter
|
||||
* parameter, ha-commit-engine task 5.1). */
|
||||
int rc = repl_wait_majority_ack(&c->persist.repl, 0, CONF_PUSH_TIMEOUT_MS,
|
||||
/* Req 8.1–8.3 / barrier-rollout Req 4.1-4.3: wait for majority-quorum
|
||||
* OP_upgradeConfigAck, passing the in-flight push tag for per-push
|
||||
* evaluation (entry_lsn carries the tag — task 5.2). On quorum:
|
||||
* advance committed_config_tag atomically (Req 8.5 / 4.3 — the
|
||||
* advance stays inside this success branch only). */
|
||||
int rc = repl_wait_majority_ack(&c->persist.repl, new_tag,
|
||||
CONF_PUSH_TIMEOUT_MS,
|
||||
(uint8_t)OPCode_upgradeConfigAck);
|
||||
if (rc == 0) {
|
||||
atomic_store_explicit(&c->ha.committed_config_tag, new_tag,
|
||||
|
|
@ -933,12 +971,11 @@ int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen,
|
|||
INFO("Config committed on majority (tag=%016" PRIx64
|
||||
", pushed=%d/%d standbys)", new_tag, acked, n);
|
||||
} else {
|
||||
WARN("Config push: timeout waiting for majority ACK"
|
||||
" (tag=%016" PRIx64 ", pushed=%d/%d standbys)",
|
||||
new_tag, acked, n);
|
||||
WARN("Config push: no majority ACK (rc=%d, tag=%016" PRIx64
|
||||
", pushed=%d/%d standbys)", rc, new_tag, acked, n);
|
||||
}
|
||||
|
||||
return acked;
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* ── handle_upgrade_config ──────────────────────── */
|
||||
|
|
@ -956,8 +993,9 @@ int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen,
|
|||
* 4. Increment generation, re-encode with updated counter
|
||||
* 5. Save to disk atomically
|
||||
* 6. Update in-memory generation
|
||||
* 7. Push to standbys (best-effort)
|
||||
* 8. ACK admin tool
|
||||
* 7. Push to standbys and wait for the majority-ACK barrier; on failure
|
||||
* reply with send_upgrade_error and return (no success ACK, no restart)
|
||||
* 8. ACK admin tool (success ⇔ majority-acked config)
|
||||
* 9. Wait for in-flight checkpoint, then signal restart
|
||||
*/
|
||||
static int do_upgrade_config_with_bytes(cluster_t *c, endpoint_t *ep,
|
||||
|
|
@ -968,11 +1006,11 @@ static int do_upgrade_config_with_bytes(cluster_t *c, endpoint_t *ep,
|
|||
{
|
||||
/* 0. Validate size */
|
||||
if (in_len == 0 || in_bytes == NULL || in_sha == NULL) {
|
||||
send_upgrade_error(ep, req, c, "Missing bytes or sha256 field");
|
||||
send_upgrade_error(ep, req, c, -EINVAL, "Missing bytes or sha256 field");
|
||||
return -EINVAL;
|
||||
}
|
||||
if (in_len > CONF_MAX_SIZE) {
|
||||
send_upgrade_error(ep, req, c, "Config blob exceeds CONF_MAX_SIZE");
|
||||
send_upgrade_error(ep, req, c, -EINVAL, "Config blob exceeds CONF_MAX_SIZE");
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
|
|
@ -980,7 +1018,7 @@ static int do_upgrade_config_with_bytes(cluster_t *c, endpoint_t *ep,
|
|||
uint8_t computed[32];
|
||||
SHA256(in_bytes, in_len, computed);
|
||||
if (memcmp(computed, in_sha, 32) != 0) {
|
||||
send_upgrade_error(ep, req, c,
|
||||
send_upgrade_error(ep, req, c, -EINVAL,
|
||||
"SHA-256 mismatch -- blob corrupted in transit");
|
||||
return -EINVAL;
|
||||
}
|
||||
|
|
@ -994,13 +1032,13 @@ static int do_upgrade_config_with_bytes(cluster_t *c, endpoint_t *ep,
|
|||
|
||||
int fd = open(tmp_path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
|
||||
if (fd < 0) {
|
||||
send_upgrade_error(ep, req, c, "Failed to create temp decode file");
|
||||
send_upgrade_error(ep, req, c, -EINVAL, "Failed to create temp decode file");
|
||||
return -errno;
|
||||
}
|
||||
if (write(fd, in_bytes, in_len) != (ssize_t)in_len) {
|
||||
close(fd);
|
||||
unlink(tmp_path);
|
||||
send_upgrade_error(ep, req, c, "Failed to write temp decode file");
|
||||
send_upgrade_error(ep, req, c, -EINVAL, "Failed to write temp decode file");
|
||||
return -EIO;
|
||||
}
|
||||
close(fd);
|
||||
|
|
@ -1014,7 +1052,7 @@ static int do_upgrade_config_with_bytes(cluster_t *c, endpoint_t *ep,
|
|||
free_ClusterConfig(new_cfg);
|
||||
free(new_cfg);
|
||||
}
|
||||
send_upgrade_error(ep, req, c, "Failed to parse Cap'n Proto blob");
|
||||
send_upgrade_error(ep, req, c, -EINVAL, "Failed to parse Cap'n Proto blob");
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
|
|
@ -1022,7 +1060,7 @@ static int do_upgrade_config_with_bytes(cluster_t *c, endpoint_t *ep,
|
|||
if (config_contains_reserved_name(new_cfg)) {
|
||||
free_ClusterConfig(new_cfg);
|
||||
free(new_cfg);
|
||||
send_upgrade_error(ep, req, c,
|
||||
send_upgrade_error(ep, req, c, -EINVAL,
|
||||
"Config contains reserved name 'lost_and_found'");
|
||||
return -EINVAL;
|
||||
}
|
||||
|
|
@ -1046,7 +1084,7 @@ static int do_upgrade_config_with_bytes(cluster_t *c, endpoint_t *ep,
|
|||
uint8_t *out_bytes = (uint8_t *)malloc(out_len);
|
||||
if (!out_bytes) {
|
||||
capn_free(&out_cap);
|
||||
send_upgrade_error(ep, req, c, "OOM re-encoding config");
|
||||
send_upgrade_error(ep, req, c, -EINVAL, "OOM re-encoding config");
|
||||
return -ENOMEM;
|
||||
}
|
||||
capn_write_mem(&out_cap, out_bytes, out_len, 0);
|
||||
|
|
@ -1058,19 +1096,37 @@ static int do_upgrade_config_with_bytes(cluster_t *c, endpoint_t *ep,
|
|||
rc = conf_save_config(c->myconf, out_bytes, out_len, computed);
|
||||
if (rc != 0) {
|
||||
free(out_bytes);
|
||||
send_upgrade_error(ep, req, c, "Failed to save new config to disk");
|
||||
send_upgrade_error(ep, req, c, -EINVAL, "Failed to save new config to disk");
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* 6. Update in-memory generation */
|
||||
c->myconf->config_generation = new_gen;
|
||||
|
||||
/* 7. Push to standbys (best-effort; see push_config_to_standbys comment) */
|
||||
int acked = push_config_to_standbys(c, out_bytes, out_len, computed,
|
||||
c->myconf->config_version_tag, NULL);
|
||||
/* 7. Push to standbys — the majority-ACK barrier result is
|
||||
* authoritative for the upgrade outcome (barrier-rollout Req 4.1/4.2):
|
||||
* non-zero means the new tag never reached quorum, so the requester
|
||||
* gets an explicit error and the cluster stays on the old committed
|
||||
* tag (no success ACK, no restart — Req 4.3/4.4). Disk save and
|
||||
* generation bump have already happened (pre-existing sequence);
|
||||
* convergence of a half-pushed config remains the OP_replConfig
|
||||
* anti-entropy behavior. */
|
||||
int acked = 0;
|
||||
rc = push_config_to_standbys(c, out_bytes, out_len, computed,
|
||||
c->myconf->config_version_tag, &acked);
|
||||
free(out_bytes);
|
||||
|
||||
/* 8. ACK the admin tool */
|
||||
if (rc != 0) {
|
||||
/* Forward the barrier rc (-ETIMEDOUT / -EINVAL) as the reply
|
||||
* status so the admin tool sees WHY the commit failed (task 5.2
|
||||
* r1, reviewer finding 3). */
|
||||
send_upgrade_error(ep, req, c, rc,
|
||||
"Config push failed to reach majority quorum"
|
||||
" -- upgrade not committed, no restart");
|
||||
return rc;
|
||||
}
|
||||
|
||||
/* 8. ACK the admin tool (success ⇔ majority-acked config) */
|
||||
upgrade_config_reply_t urep;
|
||||
memset(&urep, 0, sizeof(urep));
|
||||
urep.version_tag = c->myconf->config_version_tag;
|
||||
|
|
@ -1118,12 +1174,12 @@ int handle_upgrade_config(cluster_t *c, endpoint_t *ep, xce_request_t *req)
|
|||
|
||||
int rc = xce_upgrade_config_req_extract(req, &in_bytes, &in_len, &in_sha);
|
||||
if (rc != 0 || !in_bytes || !in_sha) {
|
||||
send_upgrade_error(ep, req, c,
|
||||
send_upgrade_error(ep, req, c, -EINVAL,
|
||||
"Failed to extract bytes from UpgradeConfigRequest");
|
||||
return rc != 0 ? rc : -EINVAL;
|
||||
}
|
||||
if (in_len == 0) {
|
||||
send_upgrade_error(ep, req, c, "bytes or sha256 field missing");
|
||||
send_upgrade_error(ep, req, c, -EINVAL, "bytes or sha256 field missing");
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -109,14 +109,38 @@ int handle_config_request(cluster_t *c, endpoint_t *ep, xce_request_t *req);
|
|||
int handle_repl_config_push(cluster_t *c, endpoint_t *ep, xce_request_t *req);
|
||||
|
||||
/*
|
||||
* push_config_to_standbys: fan-out a new config blob to all connected standby
|
||||
* controllers in parallel. Best-effort — standbys that miss the push will
|
||||
* pull on reconnect via handle_config_request().
|
||||
* push_config_transport_only: best-effort streaming fan-out of a config blob
|
||||
* to all connected standby endpoints — NO barrier wait (ha-commit-barrier-
|
||||
* rollout task 5.2 remediation r1). Spawns one push thread per registered
|
||||
* standby; the sends are non-blocking endpoint_enq()s, so the helper is
|
||||
* iomgr-safe and is the ONLY config-push entry callers reachable from
|
||||
* peer_ctrl_msg_handler / io_worker may use (election promotion, heartbeat-
|
||||
* driven convergence in ha.c, controller join in join.c). Standbys that
|
||||
* miss the push converge later via the OP_replConfig anti-entropy pull and
|
||||
* the heartbeat decision loop.
|
||||
* n_out (may be NULL): receives the registered standby count.
|
||||
* Returns the number of standbys that ACKed the push at the transport level.
|
||||
*/
|
||||
int push_config_transport_only(cluster_t *c, const uint8_t *bytes,
|
||||
size_t blen, const uint8_t sha256[32],
|
||||
uint64_t new_tag, int *n_out);
|
||||
|
||||
/*
|
||||
* push_config_to_standbys: transport fan-out (push_config_transport_only)
|
||||
* PLUS the majority-quorum config ACK barrier on new_tag (ha-commit-barrier-
|
||||
* rollout task 5.2). The barrier blocks up to CONF_PUSH_TIMEOUT_MS — this
|
||||
* entry is for VTASK-CONTEXT callers ONLY (the OP_upgradeConfig path,
|
||||
* do_upgrade_config_with_bytes); it must never run on the iomgr epoll
|
||||
* thread (no path reachable from peer_ctrl_msg_handler / io_worker — those
|
||||
* callers use push_config_transport_only above).
|
||||
* acked_out (may be NULL): receives the number of standbys that successfully
|
||||
* received the push at the transport level (per-standby push count).
|
||||
* Returns the number of standbys that successfully received the push.
|
||||
* (ha-commit-barrier-rollout task 5.2 moves the majority-ACK barrier result
|
||||
* into the return value; the push count is then only available via acked_out.)
|
||||
* Returns the barrier result: 0 once 1 (self) + standbys with
|
||||
* acked_config_lsn >= new_tag form a majority of configured members
|
||||
* (n_peers <= 1 → immediate 0); -ETIMEDOUT when the quorum never forms
|
||||
* within CONF_PUSH_TIMEOUT_MS; -EINVAL if the commit engine is not wired.
|
||||
* On 0 committed_config_tag has been advanced to new_tag; on failure it is
|
||||
* untouched. The upgrade path treats the result as authoritative.
|
||||
* Only meaningful when CONF_ENABLE_UPGRADE_CONFIG is set, but always declared
|
||||
* so callers can reference it without an #if guard.
|
||||
*/
|
||||
|
|
|
|||
|
|
@ -68,15 +68,23 @@ typedef struct {
|
|||
cluster_t *c;
|
||||
} committing_timeout_ctx_t;
|
||||
|
||||
/* cluster_init_phase2 / push_config_to_standbys / ha_send_ctrlhello_to_all_standbys:
|
||||
* defined in cluster.c / config_repl.c; only referenced in production builds.
|
||||
* (XCE_TESTING uses hook shims and does not link cluster.c.) */
|
||||
/* cluster_init_phase2 / push_config_transport_only /
|
||||
* ha_send_ctrlhello_to_all_standbys: defined in cluster.c / config_repl.c;
|
||||
* only referenced in production builds.
|
||||
* (XCE_TESTING uses hook shims and does not link cluster.c.)
|
||||
* ha.c uses push_config_transport_only ONLY (task 5.2 remediation r1):
|
||||
* every ha.c push site is reachable from peer_ctrl_msg_handler / io_worker
|
||||
* (vote replies, heartbeat replies), where the push_config_to_standbys
|
||||
* barrier wait would block the single iomgr epoll thread — and the config
|
||||
* ACK it waits for would arrive on that same blocked thread (structural
|
||||
* deadlock). Convergence is owned by the OP_replConfig anti-entropy pull
|
||||
* and the heartbeat decision loop, not by these best-effort pushes. */
|
||||
#ifndef XCE_TESTING
|
||||
extern int cluster_init_phase2(cluster_t *c);
|
||||
extern void ha_send_ctrlhello_to_all_standbys(cluster_t *c);
|
||||
extern int push_config_to_standbys(cluster_t *c, const uint8_t *bytes, size_t blen,
|
||||
const uint8_t sha256[32], uint64_t new_tag,
|
||||
int *acked_out);
|
||||
extern int push_config_transport_only(cluster_t *c, const uint8_t *bytes,
|
||||
size_t blen, const uint8_t sha256[32],
|
||||
uint64_t new_tag, int *n_out);
|
||||
#endif
|
||||
|
||||
static void on_vote_connected(endpoint_t *ep);
|
||||
|
|
@ -567,12 +575,12 @@ static void raft_cb_become_leader(void *ctx, uint64_t term)
|
|||
extern int conf_read_raw(const char *, uint8_t **, size_t *);
|
||||
if (c->myconf && c->myconf->conffile &&
|
||||
conf_read_raw(c->myconf->conffile, &cfg_bytes, &cfg_len) == 0) {
|
||||
push_config_to_standbys(c, cfg_bytes, cfg_len,
|
||||
push_config_transport_only(c, cfg_bytes, cfg_len,
|
||||
c->myconf->config_sha256,
|
||||
c->myconf->config_version_tag, NULL);
|
||||
free(cfg_bytes);
|
||||
} else {
|
||||
push_config_to_standbys(c, NULL, 0, NULL,
|
||||
push_config_transport_only(c, NULL, 0, NULL,
|
||||
c->myconf ? c->myconf->config_version_tag : 0,
|
||||
NULL);
|
||||
}
|
||||
|
|
@ -1362,7 +1370,7 @@ void ha_check_majority_convergence(ha_state_t *ha, cluster_t *c)
|
|||
* (stored by ha_process_standby_state on the same heartbeat reply), calls
|
||||
* ha_evaluate_repl_action, and acts on the result:
|
||||
*
|
||||
* HA_REPL_PUSH_CONFIG → push_config_to_standbys(c, …) [or test hook] (Task 3.3)
|
||||
* HA_REPL_PUSH_CONFIG → push_config_transport_only(c, …) [or test hook] (Task 3.3)
|
||||
* HA_REPL_SEND_SNAPSHOT→ stub (Task 3.5)
|
||||
* HA_REPL_STREAM_WAL → ha_stream_wal_to_standby(c, ha, idx) (Task 3.4)
|
||||
* HA_REPL_CONVERGED → no-op
|
||||
|
|
@ -1407,12 +1415,12 @@ int ha_dispatch_repl_action_for_standby(ha_state_t *ha,
|
|||
size_t cfg_len = 0;
|
||||
if (c->myconf && c->myconf->conffile &&
|
||||
conf_read_raw(c->myconf->conffile, &cfg_bytes, &cfg_len) == 0) {
|
||||
push_config_to_standbys(c, cfg_bytes, cfg_len,
|
||||
push_config_transport_only(c, cfg_bytes, cfg_len,
|
||||
c->myconf->config_sha256,
|
||||
c->myconf->config_version_tag, NULL);
|
||||
free(cfg_bytes);
|
||||
} else {
|
||||
push_config_to_standbys(c, NULL, 0, NULL,
|
||||
push_config_transport_only(c, NULL, 0, NULL,
|
||||
c->myconf ? c->myconf->config_version_tag : 0,
|
||||
NULL);
|
||||
}
|
||||
|
|
@ -2078,7 +2086,7 @@ int handle_request_vote(ha_state_t *ha,
|
|||
|
||||
/* ── Test-only hook shims (XCE_TESTING only) ──────────────────────────────
|
||||
* Type definitions (election_timeout_ctx_t, committing_timeout_ctx_t),
|
||||
* extern declarations (cluster_init_phase2, push_config_to_standbys,
|
||||
* extern declarations (cluster_init_phase2, push_config_transport_only,
|
||||
* ha_send_ctrlhello_to_all_standbys), and static forward declarations
|
||||
* (committing_timeout_fn, ha_heartbeat_is_fresh) have been moved to the
|
||||
* forward-declarations block near the top of this file so that the raft
|
||||
|
|
@ -2101,9 +2109,9 @@ int (*ha_phase2_hook)(void *c) = NULL;
|
|||
int (*ha_send_ctrlhello_hook)(void *c) = NULL;
|
||||
|
||||
/*
|
||||
* ha_push_config_hook: test-only seam for push_config_to_standbys.
|
||||
* ha_push_config_hook: test-only seam for push_config_transport_only.
|
||||
* When non-NULL, the COMMITTING path calls this INSTEAD OF the real function
|
||||
* (config_repl.c's push_config_to_standbys needs a real cluster). Req 7.1.
|
||||
* (config_repl.c's push_config_transport_only needs a real cluster). Req 7.1.
|
||||
*/
|
||||
int (*ha_push_config_hook)(void *c) = NULL;
|
||||
|
||||
|
|
@ -2118,7 +2126,7 @@ int (*ha_nop_req_hook)(endpoint_t *ep) = NULL;
|
|||
* ha_repl_push_config_for_standby_hook: test-only seam for the per-standby
|
||||
* config push triggered by the heartbeat-reply decision loop (Task 3.3).
|
||||
* When non-NULL, ha_dispatch_repl_action_for_standby() calls this INSTEAD OF
|
||||
* push_config_to_standbys() when the decision is HA_REPL_PUSH_CONFIG.
|
||||
* push_config_transport_only() when the decision is HA_REPL_PUSH_CONFIG.
|
||||
* Requirements: 1.4 (ha-repl-hardening)
|
||||
*/
|
||||
int (*ha_repl_push_config_for_standby_hook)(void *c, int peer_idx) = NULL;
|
||||
|
|
@ -2283,7 +2291,7 @@ apply_role:
|
|||
ha_push_config_hook(c);
|
||||
#else
|
||||
if (c != NULL)
|
||||
push_config_to_standbys(c, NULL, 0, NULL, 0, NULL);
|
||||
push_config_transport_only(c, NULL, 0, NULL, 0, NULL);
|
||||
#endif
|
||||
|
||||
/* Send CtrlHelloRequest to all standbys to trigger WAL catch-up (Req 7.2) */
|
||||
|
|
@ -2649,7 +2657,10 @@ done:
|
|||
*
|
||||
* - OPCode_replCatchUpDone → ha_repl_ack_received
|
||||
* - OPCode_reply with stateReportMsg → ha_process_standby_state (Task 3.1)
|
||||
* - OPCode_nop or OPCode_reply without stateReport → silently ignored
|
||||
* - OPCode_reply with walackreplymsg → repl_on_ack (Task 4.1)
|
||||
* - OPCode_reply with replconfigpushreplymsg → repl_on_config_ack
|
||||
* (barrier-rollout task 5.2 r1 / ha-config-bootstrap task 8.1)
|
||||
* - OPCode_nop or OPCode_reply with any other body → silently ignored
|
||||
* - anything else → WARN + drop
|
||||
*/
|
||||
static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx)
|
||||
|
|
@ -2775,6 +2786,32 @@ static int repl_ack_decode_and_dispatch(endpoint_t *ep, vote_ep_ctx_t *vctx)
|
|||
WARN("repl_ack_decode_and_dispatch: WAL NACK from standby"
|
||||
" lsn=%lu success=0 — ignoring", (unsigned long)acked_lsn);
|
||||
}
|
||||
} else if (rep_s.data_which == RequestReply_data_replconfigpushreplymsg) {
|
||||
/* Config-push ACK from standby (ha-commit-barrier-rollout task
|
||||
* 5.2 remediation r1; completes the unlanded half of
|
||||
* ha-config-bootstrap task 8.1): the standby replies to
|
||||
* OP_replConfigPush with replconfigpushreplymsg carrying
|
||||
* {accepted, versionTag} (config_repl.c:744-752). Decode the
|
||||
* acked tag and apply it to the replicator so the configAck
|
||||
* barrier (repl_wait_majority_ack) can observe the covering
|
||||
* ACK — without this branch every multi-member config upgrade
|
||||
* would time out. read_ReplConfigPushReply reads scalars only
|
||||
* — safe to call before capn_free. */
|
||||
struct ReplConfigPushReply rcp_raw;
|
||||
memset(&rcp_raw, 0, sizeof(rcp_raw));
|
||||
read_ReplConfigPushReply(&rcp_raw, rep_s.data.replconfigpushreplymsg);
|
||||
uint64_t acked_tag = (uint64_t)rcp_raw.versionTag;
|
||||
int accepted = rcp_raw.accepted ? 1 : 0;
|
||||
capn_free(&capn_ctx);
|
||||
|
||||
if (accepted && c) {
|
||||
repl_on_config_ack(&c->persist.repl, ep, acked_tag);
|
||||
TRACE("repl_ack_decode_and_dispatch: config ACK from standby"
|
||||
" tag=%016" PRIx64 " accepted=1", acked_tag);
|
||||
} else if (!accepted) {
|
||||
WARN("repl_ack_decode_and_dispatch: config push rejected by"
|
||||
" standby (tag=%016" PRIx64 ") — ignoring", acked_tag);
|
||||
}
|
||||
} else {
|
||||
capn_free(&capn_ctx);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -182,9 +182,12 @@ static int handle_controller_join(cluster_t* c, endpoint_t* ep,
|
|||
c->ha.standby_endpoints->count);
|
||||
|
||||
/* Immediately push current config to the newly-joined standby so it
|
||||
* doesn't have to wait for the next scheduled push. */
|
||||
* doesn't have to wait for the next scheduled push. Transport-only
|
||||
* (no barrier wait — task 5.2 remediation r1): this is a best-effort
|
||||
* catch-up stream; convergence is owned by the OP_replConfig
|
||||
* anti-entropy pull and the heartbeat decision loop. */
|
||||
if (ha_is_primary(&c->ha))
|
||||
push_config_to_standbys(c, NULL, 0, NULL, 0, NULL);
|
||||
push_config_transport_only(c, NULL, 0, NULL, 0, NULL);
|
||||
}
|
||||
|
||||
return 0;
|
||||
|
|
|
|||
|
|
@ -407,11 +407,16 @@ void repl_on_config_ack(replicator_t *r, repl_ep_t ep, uint64_t acked_config_lsn
|
|||
* anything ever" counting with self-count and the zero-active
|
||||
* success short-circuit is retired). Returns the engine code
|
||||
* unmodified: 0 / -ETIMEDOUT / -XCE_ERR_NOT_READY / -EINVAL.
|
||||
* configAck: legacy standby counting (hardened by barrier-rollout task 5):
|
||||
* quorum = (n_active + 2) / 2 over active (non-catch-up)
|
||||
* standbys; acked_total = standbys with acked_config_lsn > 0,
|
||||
* NO self-count (config must be confirmed by standbys).
|
||||
* Returns 0 on quorum, -ETIMEDOUT on timeout.
|
||||
* configAck: per-push arithmetic mirroring the engine's rules on the
|
||||
* config track (barrier-rollout task 5.2, design §Config
|
||||
* Barrier): entry_lsn carries the in-flight config push tag;
|
||||
* count = active (non-catch-up) standbys with
|
||||
* acked_config_lsn >= push_tag (a previous upgrade's ack no
|
||||
* longer counts); majority = 1 (self) + count > n_peers / 2
|
||||
* over configured members (engine n_peers, set-once at
|
||||
* wiring); n_peers <= 1 → immediate success. Returns 0 on
|
||||
* quorum, -ETIMEDOUT on timeout, -EINVAL when the engine is
|
||||
* not wired (fail-loud, mirroring the walAck branch).
|
||||
* ============================================================ */
|
||||
|
||||
int repl_wait_majority_ack(replicator_t *r, uint64_t entry_lsn,
|
||||
|
|
@ -441,7 +446,33 @@ int repl_wait_majority_ack(replicator_t *r, uint64_t entry_lsn,
|
|||
return commit_submit(r->ce, entry_lsn, COMMIT_MODE_WAIT, timeout_ms);
|
||||
}
|
||||
|
||||
/* ── configAck track (legacy standby counting; task 5 hardens it) ── */
|
||||
/* ── configAck track (per-push arithmetic; barrier-rollout task 5.2) ──
|
||||
* entry_lsn carries the in-flight config push tag (repl.h: the branch
|
||||
* reuses the walAck adapter parameter — no second parameter). */
|
||||
|
||||
uint64_t push_tag = entry_lsn;
|
||||
int n_peers;
|
||||
|
||||
if (r->ce == NULL) {
|
||||
/* The engine is wired at cluster_init_phase2 before any config
|
||||
* push reaches this wait; an unwired engine is a harness defect,
|
||||
* not a quorum — fail loud rather than fabricate a result
|
||||
* (mirrors the walAck branch's commit_submit -EINVAL). */
|
||||
WARN("repl_wait_majority_ack: configAck wait without a wired"
|
||||
" commit engine (push_tag=%016" PRIx64 ")", push_tag);
|
||||
return -EINVAL;
|
||||
}
|
||||
|
||||
/* ce and ce->n_peers are set-once before replication traffic
|
||||
* (repl_set_commit_engine contract) — safe to read without locks. */
|
||||
n_peers = r->ce->n_peers;
|
||||
|
||||
if (n_peers <= 1) {
|
||||
/* Single-member cluster: the local save is the quorum. */
|
||||
TRACE("repl_wait_majority_ack: configAck self-quorum"
|
||||
" (n_peers=%d, push_tag=%016" PRIx64 ")", n_peers, push_tag);
|
||||
return 0;
|
||||
}
|
||||
|
||||
struct timespec deadline;
|
||||
clock_gettime(CLOCK_REALTIME, &deadline);
|
||||
|
|
@ -458,8 +489,7 @@ int repl_wait_majority_ack(replicator_t *r, uint64_t entry_lsn,
|
|||
while (1) {
|
||||
mylock_lock(&r->lock);
|
||||
|
||||
int n_active = 0;
|
||||
int acked_count = 0;
|
||||
int covered = 0;
|
||||
|
||||
for (int i = 0; i < MAX_STANDBYS; i++) {
|
||||
standby_conn_t *s = &r->standbys[i];
|
||||
|
|
@ -467,26 +497,18 @@ int repl_wait_majority_ack(replicator_t *r, uint64_t entry_lsn,
|
|||
continue;
|
||||
if (atomic_load_explicit(&s->in_catch_up, memory_order_acquire))
|
||||
continue;
|
||||
n_active++;
|
||||
if (atomic_load_explicit(&s->acked_config_lsn,
|
||||
memory_order_acquire) > 0)
|
||||
acked_count++;
|
||||
memory_order_acquire) >= push_tag)
|
||||
covered++;
|
||||
}
|
||||
|
||||
mylock_unlock(&r->lock);
|
||||
|
||||
/* Single-node shortcircuit: no active standbys → local write is
|
||||
* quorum (configAck only, Req 8.5 — walAck never reaches this
|
||||
* loop; its single-member case is the engine's self-quorum). */
|
||||
if (n_active == 0) {
|
||||
rc = 0;
|
||||
break;
|
||||
}
|
||||
|
||||
int quorum = (n_active + 2) / 2;
|
||||
int acked_total = acked_count; /* no self-count for configAck */
|
||||
|
||||
if (acked_total >= quorum) {
|
||||
/* Majority over configured members, self counted: the primary's
|
||||
* own save plus standbys covering the in-flight push tag. No
|
||||
* n_active shortcut — a multi-member cluster with zero covering
|
||||
* acks must time out, never fabricate success (Req 4.2). */
|
||||
if (1 + covered > n_peers / 2) {
|
||||
rc = 0;
|
||||
break;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1492,6 +1492,175 @@ static void test_upgrade_config_majority_ack_success_unchanged(void)
|
|||
printf("test_upgrade_config_majority_ack_success_unchanged: PASS\n");
|
||||
}
|
||||
|
||||
/* ════════════════════════════════════════════════════════════════════════
|
||||
* Task 5.2 remediation r1 — wire-path config ACK ingestion (Req 4.1, 4.2).
|
||||
*
|
||||
* The barrier consumes acked_config_lsn, which only repl_on_config_ack
|
||||
* advances — so the standby's OP_replConfigPush reply
|
||||
* (replconfigpushreplymsg, config_repl.c:744-752) must be decoded on the
|
||||
* primary and fed to repl_on_config_ack. This case drives the ACK over
|
||||
* the WIRE: it builds the standby's capnp reply bytes and pushes them
|
||||
* through peer_ctrl_msg_handler → repl_ack_decode_and_dispatch (the
|
||||
* production decode entry for promoted peer endpoints), asserting the
|
||||
* decoded tag lands in the replicator slot and completes a pending
|
||||
* barrier. This also completes the unlanded half of ha-config-bootstrap
|
||||
* task 8.1 (production ingestion of the config-push ACK), which until
|
||||
* now was only exercised by direct repl_on_config_ack injection.
|
||||
* ════════════════════════════════════════════════════════════════════════ */
|
||||
|
||||
/* ha.c test seams (test_ha_connection_reuse.c precedent) */
|
||||
extern int peer_ctrl_msg_handler_for_test(endpoint_t *ep, io_packet_t *pkt);
|
||||
extern endpoint_t *vote_ep_ctx_build_for_test(endpoint_t *ep,
|
||||
ha_state_t *ha,
|
||||
ha_elect_ctx_t *ctx,
|
||||
int peer_idx,
|
||||
uint64_t my_term,
|
||||
uint64_t my_tag,
|
||||
uint64_t my_lsn,
|
||||
const char *my_name);
|
||||
extern void vote_ep_ctx_set_cluster_for_test(endpoint_t *ep, cluster_t *c);
|
||||
|
||||
/* Build the standby's config-push ACK exactly as xce_reply() puts it on
|
||||
* the wire (config_repl.c:744-752): Request{opcode=reply,
|
||||
* body.replymsg=RequestReply{data.replconfigpushreplymsg=
|
||||
* ReplConfigPushReply{accepted, versionTag}}}. The handler takes
|
||||
* ownership (free_packet + free). */
|
||||
static io_packet_t *cb5_build_config_ack_pkt(int accepted, uint64_t tag)
|
||||
{
|
||||
struct capn cs;
|
||||
struct capn_segment *seg;
|
||||
struct ReplConfigPushReply ack_capn;
|
||||
ReplConfigPushReply_ptr ack_ptr;
|
||||
struct RequestReply rr_capn;
|
||||
RequestReply_ptr rr_ptr;
|
||||
struct Request rq_capn;
|
||||
Request_ptr rq_ptr;
|
||||
io_packet_t *pkt;
|
||||
|
||||
capn_init_malloc(&cs);
|
||||
seg = capn_root(&cs).seg;
|
||||
|
||||
memset(&ack_capn, 0, sizeof(ack_capn));
|
||||
ack_capn.accepted = accepted ? 1 : 0;
|
||||
ack_capn.versionTag = tag;
|
||||
ack_ptr = new_ReplConfigPushReply(seg);
|
||||
write_ReplConfigPushReply(&ack_capn, ack_ptr);
|
||||
|
||||
memset(&rr_capn, 0, sizeof(rr_capn));
|
||||
rr_capn.seq = 1;
|
||||
rr_capn.opcode = OPCode_replConfigPush;
|
||||
rr_capn.status = 0;
|
||||
rr_capn.data_which = RequestReply_data_replconfigpushreplymsg;
|
||||
rr_capn.data.replconfigpushreplymsg = ack_ptr;
|
||||
rr_ptr = new_RequestReply(seg);
|
||||
write_RequestReply(&rr_capn, rr_ptr);
|
||||
|
||||
memset(&rq_capn, 0, sizeof(rq_capn));
|
||||
rq_capn.opcode = OPCode_reply;
|
||||
rq_capn.seq = 1;
|
||||
rq_capn.body_which = Request_body_replymsg;
|
||||
rq_capn.body.replymsg = rr_ptr;
|
||||
rq_ptr = new_Request(seg);
|
||||
write_Request(&rq_capn, rq_ptr);
|
||||
capn_setp(capn_root(&cs), 0, rq_ptr.p);
|
||||
|
||||
pkt = calloc(1, sizeof(*pkt));
|
||||
TEST_ASSERT_NOT_NULL_MESSAGE(pkt, "config ACK pkt alloc");
|
||||
pkt->sz = capn_size(&cs);
|
||||
TEST_ASSERT_TRUE_MESSAGE(alloc_packet(pkt) >= 0, "config ACK pkt data alloc");
|
||||
capn_write_mem(&cs, (uint8_t *)pkt->data, (size_t)pkt->sz, 0);
|
||||
capn_free(&cs);
|
||||
|
||||
return pkt;
|
||||
}
|
||||
|
||||
typedef struct {
|
||||
endpoint_t *ep;
|
||||
io_packet_t *pkt;
|
||||
unsigned delay_ms;
|
||||
} cb5_wire_ack_args_t;
|
||||
|
||||
/* Deliver the wire ACK to the promoted-peer message handler after a delay,
|
||||
* while the main thread is parked inside the barrier wait. */
|
||||
static void *cb5_wire_ack_thread(void *arg)
|
||||
{
|
||||
cb5_wire_ack_args_t *a = (cb5_wire_ack_args_t *)arg;
|
||||
|
||||
usleep(a->delay_ms * 1000);
|
||||
peer_ctrl_msg_handler_for_test(a->ep, a->pkt);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
/* statics survive Unity longjmp */
|
||||
static endpoint_t g_cb5_wire_ep;
|
||||
static cb5_wire_ack_args_t g_cb5_wire_args;
|
||||
|
||||
/* A config ACK delivered as WIRE BYTES through the production decode entry
|
||||
* (peer_ctrl_msg_handler → repl_ack_decode_and_dispatch) must reach
|
||||
* repl_on_config_ack: the replicator slot's acked_config_lsn advances to
|
||||
* the decoded tag and a pending configAck barrier completes (Req 4.1).
|
||||
* RED (remediation r1): repl_ack_decode_and_dispatch has no
|
||||
* replconfigpushreplymsg branch — the reply is silently dropped, the
|
||||
* barrier times out, rc=-ETIMEDOUT. */
|
||||
static void test_config_ack_wire_path_completes_barrier(void)
|
||||
{
|
||||
const uint64_t push_tag = 0xC0FFEE;
|
||||
pthread_t th;
|
||||
int rc;
|
||||
uint64_t acked = 0;
|
||||
int found = 0;
|
||||
|
||||
/* Arrange: cb5 cluster (wired engine n_peers=3, two never-acking
|
||||
* standbys) + a third, promoted-peer endpoint whose vote_ep_ctx routes
|
||||
* peer_ctrl_msg_handler to repl_ack_decode_and_dispatch (role PRIMARY). */
|
||||
cb5_setup();
|
||||
memset(&g_cb5_wire_ep, 0, sizeof(g_cb5_wire_ep));
|
||||
g_cb5_wire_ep.fd = -1;
|
||||
vote_ep_ctx_build_for_test(&g_cb5_wire_ep, &g_cb5_c.ha, NULL, 0,
|
||||
1, 0, 0, "ctrl-cb5");
|
||||
vote_ep_ctx_set_cluster_for_test(&g_cb5_wire_ep, &g_cb5_c);
|
||||
atomic_store_explicit(&g_cb5_c.ha.role, (int32_t)CTRL_PRIMARY,
|
||||
memory_order_release);
|
||||
TEST_ASSERT_TRUE(repl_add_standby(&g_cb5_c.persist.repl,
|
||||
&g_cb5_wire_ep, "cb5_wire", 7502) >= 0);
|
||||
|
||||
/* Act: park in the barrier (self + 1 covering = 2 > 3/2 quorum), then
|
||||
* deliver the standby's ACK as raw capnp bytes through the handler. */
|
||||
g_cb5_wire_args.ep = &g_cb5_wire_ep;
|
||||
g_cb5_wire_args.pkt = cb5_build_config_ack_pkt(1, push_tag);
|
||||
g_cb5_wire_args.delay_ms = 50;
|
||||
TEST_ASSERT_EQUAL_INT(0, pthread_create(&th, NULL, cb5_wire_ack_thread,
|
||||
&g_cb5_wire_args));
|
||||
rc = repl_wait_majority_ack(&g_cb5_c.persist.repl, push_tag, 1000,
|
||||
(uint8_t)OPCode_upgradeConfigAck);
|
||||
pthread_join(th, NULL);
|
||||
|
||||
/* Assert: barrier completed off the wire-decoded ACK … */
|
||||
TEST_ASSERT_EQUAL_INT_MESSAGE(0, rc,
|
||||
"a wire-delivered config ACK must complete a pending barrier"
|
||||
" (production ingestion path — Req 4.1)");
|
||||
|
||||
/* … because repl_on_config_ack advanced the slot to the decoded tag */
|
||||
for (int i = 0; i < MAX_STANDBYS; i++) {
|
||||
if (g_cb5_c.persist.repl.standbys[i].ep == (void *)&g_cb5_wire_ep) {
|
||||
acked = atomic_load_explicit(
|
||||
&g_cb5_c.persist.repl.standbys[i].acked_config_lsn,
|
||||
memory_order_acquire);
|
||||
found = 1;
|
||||
break;
|
||||
}
|
||||
}
|
||||
TEST_ASSERT_TRUE_MESSAGE(found, "wire ep must be registered in the replicator");
|
||||
TEST_ASSERT_EQUAL_UINT64_MESSAGE(push_tag, acked,
|
||||
"the decoded versionTag must land in acked_config_lsn via"
|
||||
" repl_on_config_ack");
|
||||
|
||||
free(g_cb5_wire_ep.data); /* vctx from vote_ep_ctx_build_for_test */
|
||||
g_cb5_wire_ep.data = NULL;
|
||||
cb5_teardown();
|
||||
printf("test_config_ack_wire_path_completes_barrier: PASS\n");
|
||||
}
|
||||
|
||||
/* ── Test group entry point ──────────────────────────────────────────────── */
|
||||
|
||||
void test_commit_barrier_rollout(void)
|
||||
|
|
@ -1531,5 +1700,8 @@ void test_commit_barrier_rollout(void)
|
|||
RUN_TEST(test_push_config_majority_acks_returns_zero_and_commits);
|
||||
RUN_TEST(test_upgrade_config_barrier_failure_error_reply_no_restart);
|
||||
RUN_TEST(test_upgrade_config_majority_ack_success_unchanged);
|
||||
/* Task 5.2 remediation r1 — production wire-path ACK ingestion
|
||||
* (completes the unlanded half of ha-config-bootstrap task 8.1) */
|
||||
RUN_TEST(test_config_ack_wire_path_completes_barrier);
|
||||
UNITY_END();
|
||||
}
|
||||
|
|
|
|||
|
|
@ -80,10 +80,13 @@ void repl_on_config_ack(replicator_t *r, repl_ep_t ep, uint64_t acked_config_lsn
|
|||
/* Sync replication (stub in P2-1; wired in P2-3) */
|
||||
int repl_wait_ack(replicator_t *r, uint64_t target_lsn, uint32_t timeout_ms);
|
||||
|
||||
/* Majority-quorum ACK wait (task 7.3; ha-commit-engine task 5 adapter).
|
||||
* entry_lsn: the WAL entry the caller waits on. The walAck branch delegates
|
||||
* to the commit engine for per-LSN evaluation when wired (task 5.2); the
|
||||
* legacy counting body ignores it. configAck callers pass 0 (ignored). */
|
||||
/* Majority-quorum ACK wait (task 7.3; ha-commit-engine task 5 adapter;
|
||||
* configAck track hardened by ha-commit-barrier-rollout task 5.2).
|
||||
* entry_lsn: walAck — the WAL entry the caller waits on, delegated to the
|
||||
* commit engine for per-LSN evaluation; configAck — the in-flight config
|
||||
* push tag, counted as acked_config_lsn >= tag with majority
|
||||
* 1 (self) + count > n_peers / 2 over configured members (no second
|
||||
* parameter — the configAck branch reuses this one). */
|
||||
int repl_wait_majority_ack(replicator_t *r, uint64_t entry_lsn,
|
||||
uint32_t timeout_ms, uint8_t ack_op);
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue